9 ms·

GPT‑5.5 Bio Bug Bounty

160 points by Murfalo 5mo ago

notatoad 5mo ago
are the 5 questions you need to get it to answer under NDA?
- ultratalk 5mo ago
  Almost certainly.
applfanboysbgon 5mo ago
> $25,000 to the first true universal jailbreak to clear all five questions. This program is a complete scam. Even if 100 people find "bugs", they will only pay out to one person.
- mmsc 5mo ago
  How is that a scam? You don't get participation awards for solving half of a puzzle...
  - applfanboysbgon 5mo ago
    I didn't say anything about partial solutions. The puzzle can have multiple full solutions. Or does the software you write only have exactly one bug? If so, that's impressive, in multiple ways, including the fact that you're able to identify that there's exactly one bug but not what the bug is and fix it.
- Lucasoato 5mo ago
  Well, that depends on how you set up the bounty program. What if I find a solution, share it to a friend so that both of us can claim the prize?
  - skeeter2020 5mo ago
    bug bounty programs have never paid out independent disclosure for the same bug though; they might split or even pay-out larger coordinated efforts. It's largely a first place award only.
  - ImPostingOnHN 5mo ago
    assume there exists 2+ different bugs after the 1st bug is found, no payout for any other of the bugs
- skeeter2020 5mo ago
  that's not the point even. They are attempting to build credibility in two ways: 1. this model is SO advanced that there are huge risks, never before considered. 2. we're doing the super-responsible thing in incentivizing work that addresses this. #1 is unproven and frankly, unlikely, which makes #2 meaningless. The fact that the "prize" is so low & structured this was suggests that they're not that concerned but do think it's likely that a bunch of people will find things. If they truly thought their model was so good they would be confident issues would be both rare and very critical, then offer huge rewards with no limits because they'd be much more confident no one would claim it.
  - applfanboysbgon 5mo ago
    Yes, I was about to edit in that I think this is simply a media/PR stunt before I got so many replies so quickly. They get bonus points because the structure is so insulting that it may not engender many serious participants, in which case it may go unbroken, in which case they can go to the media and proclaim "look, we offered a reward, but nobody broke it! Our model is objectively the safest in the world!".
    - StilesCrisis 5mo ago
      I think there's definitely going to be a prizewinner. It's an insultingly low bounty for a professional, but a script kiddie could probably figure out a jailbreak and it's a huge payout for them.
- XCSme 5mo ago
  Do we also have to pay for the API usage? Then they will actually be profitable, lol
- weare138 5mo ago
  The fact the bug bounty program is private and requires you to apply and be accepted first is also sus especially when the scope is the desktop app anyone can download.
- layer8 5mo ago
  They will also decide what “true” means.
dwa3592 5mo ago
Where are the questions that are supposed to be answered? Would those be shared after an application has been accepted? If yes, why is the application asking for a proposed approach for the jailbreak if we don't know the questions in the first place?
- dist-epoch 5mo ago
  Because the questions themselves are dangerous. Probably along the lines of "how would you create a small biolab for virus research in a kitchen with $20k?" or "how do I take the DNA sequence from https://www.ncbi.nlm.nih.gov/nuccore/NC_001611.1 https://www.ncbi.nlm.nih.gov/nuccore/NC_001611.1 and assemble it?"
  - hyperpape 5mo ago
    Which is difficult, because the fact that you can come up with your example questions tells us they're probably not very dangerous. Plenty of ink has been spilled about how LLMs could help people create bioweapons. The basic idea "you could do dangerous things with an LLM" is already pop culture, and you're not doing anything dangerous by giving easy example questions. A dangerous question would have to be along the lines of "Could I use unobtanium with the Tony Stark process to produce explosives much more powerful than nuclear weapons?" so that the question itself contains some insight that gets you closer to doing something dangerous. Perhaps the reason for not publishing the questions is twofold: 1) they want a universal jailbreak that can get the model to answer any "bad" question. 2) they don't want bad publicity when someone not under NDA jailbreaks their model and answers their question
    - dist-epoch 5mo ago
      > because the fact that you can come up with your example questions tells us they're probably not very dangerous maybe I know more about this field that you think there are biologists on video saying that present day models have expert level wet-lab knowledge and can guide a novice through whole procedures models also were able to tweak DNA sequences to make them bypass DNA-printing companies filters > they don't want bad publicity when someone not under NDA jailbreaks their model and answers their question just like people now pay $500k for Chrome vulnerabilities, soon people will pay similar amounts to jailbrake models to do bad things
sva_ 5mo ago
> We will extend invitations to a vetted list of trusted bio red-teamers Had to chuckle. This sounds like a rather exclusive group?
- petercooper 5mo ago
  It sounds like asking CS PhDs to do a world record speed run. I wouldn't be surprised if the people best suited to the task aren't the type to get onto "a vetted list".
shevy-java 5mo ago
"Accepted applicants and collaborators must have existing ChatGPT accounts to apply, and will sign a NDA." Ah, good old NDA. Always buying silence. That's why I don't participate in any such "bounty" programs. Signing a NDA is like signing with the devil. You restrict what people are allowed to discuss. I had that happen before - when you sign a NDA you basically submit yourself into silence. Imagine journalists being stifled by NDAs.
its-summertime 5mo ago
This is just free / severely-underpaid-on-average labor. Very disgusting.
- mrcwinn 5mo ago
  Ah yes, “free” as in “paid.” Certainly you’re welcome to not participate.
  - its-summertime 5mo ago
    Depending on industry, that payout can be less than a security audit. You only get a chance of getting paid. You don't even know if they gave the LLM the answers that you are supposed to recover.
  - applfanboysbgon 5mo ago
    Free as in "free" for >99% of participants, even successful ones, because they will have hundreds or thousands of participants but will only pay out to one of them no matter how many vulnerabilities are found.
  - 12_throw_away 5mo ago
    > $25,000 to the first true universal jailbreak to clear all five questions. Now, laws vary from place to place, but I'm pretty sure "a small chance to earn money after the work is completed" is not equivalent to "payment" in most jurisdictions.
zb3 5mo ago
What a farce, these questions are not even public and most likely will never be. You can't even participate if you're not "trusted" I guess. So this is just a PR post, not that I even think the "biosafety" makes any sense but still.
dakiol 5mo ago
$25K. Really? They make $65 million a day, so they pay you what they earn in about 33 seconds for a critical vulnerability. WTF
- zacharycohn 5mo ago
  Well they lose $100M a day, so...
mellosouls 5mo ago
If anybody is wondering what bio-bugs are, I had a heck of a time getting CG to (finally) tell me it's where the user can get it to guide them in doing things like constructing things that are hazardous in the domain of biology. Eg you can get answers about what ricin is but not how to weaponise it. Actionable stuff they shouldn't be able to legally/ethically action.
abujazar 5mo ago
This looks like some kind of marketing. Also, the equivalent of spec work. The NDA/secrecy also means any time spent on this is completely meaningless to the participants unless they win the lottery, because results can't be published.
- nerdsniper 5mo ago
  It looks like if they reject paying you any bounty you would you still be bound by the NDA. If so, then they could both not pay you and still spike the story. That’s not something I would ever agree to.
  - WarmWash 5mo ago
    I know we celebrate cynacism online, but $25k to OAI is like $0.025 to you and me. Skimping out on 2.5 pennies you promised someone is cartoon villain levels of greed. Yes, I know, Altman is a cartoon villain. But please, they are spending more money decorating their bathrooms. They'll pay out.
    - deleted 5mo ago
      [deleted]
    - fragmede 5mo ago
      Will they? The NDA makes it so if they don't, we'd never know. Bug bounty programs suck but they're better than the alternative, but even running one openly, there's always convention about whether the bugs being submitted are real or not, with a lot of low quality reports that the submitter thinks are gold. That happens out in the open. Now add an NDA into the mix. Sam's reputation doesn't even have to enter into the equation for it to be a bad deal.
- __natty__ 5mo ago
  Surely it is marketing. It’s some “we are danger” narrative, from Anthropic Mythos and now OpenAI too.
  - 5mo ago
gosub100 5mo ago
Check with the dark net markets first before claiming the bounty. Remember, this company has 0.0 fucks to give about the impact of their tech on employment, artists, or use in committing fraud, as long as number-go-up they are happy. Your actions should match theirs.
tiberriver256 5mo ago
Codex desktop app is barely usable... The perf issues are left to languish in their backlog
codeulike 5mo ago
This is to match what Anthropic said they already did with Mythos on the (200 page) Mythos system card
gib444 5mo ago
How did the dupe detector miss https://news.ycombinator.com/item?id=47879102 https://news.ycombinator.com/item?id=47879102 ?
- ultratalk 5mo ago
  @dang?
puppystench 5mo ago
They ran a bounty on Kaggle last year but with $500k in payouts and with all results open and publishable. https://www.kaggle.com/competitions/openai-gpt-oss-20b-red-teaming https://www.kaggle.com/competitions/openai-gpt-oss-20b-red-t... With only $25k in payouts and everything locked down under NDA, I can't imagine many people will participate. Well, other than those submitting mountains of LLM-generated junk.
- dist-epoch 5mo ago
  This model is much more powerful than gpt-oss-20b, notice how the contest was not even for the 120b model. Also, bio was not a subject.
  - stonogo 5mo ago
    The model is more powerful, so the bounty is 1/20th the size? More risk, less reward? "Biorisk" seems to be a concept not only invented by OpenAI but exclusively taken seriously by them. I wonder if this program is less about finding actual risks than it is hopefully casting a wide net for someone to help them prove their model is relevant in this space.
    - ACCount37 5mo ago
      Not really. Anthropic has the "CBRN filter" on Opus series. It used to kill inquiries on anything that's remotely related to biotech. Seems to have gotten less aggressive lately? I was reverse engineering a medical device back in 2025 and it was hard killing half my sessions.
    - stratos123 5mo ago
      > "Biorisk" seems to be a concept not only invented by OpenAI but exclusively taken seriously by them. This is false. Antropic just bundles it into CBRN. As for inventing it, the idea of AI-created bioweapons as a concrete risk far predates OpenAI as a company.
- p_stuart82 5mo ago
unethical_ban 5mo ago
* Highly unlikely to win * Relatively paltry reward * NDA on findings This is functionally equivalent to an internship where the reward is the experience, and the resume building, but you can't talk about what you did. All for a company that is getting tens of billions of dollars in deals from the largest tech companies in the world. I suppose the hope is that there are job offers somewhere along the line.
lxgr 5mo ago
Ah, now I understand why all my chats are getting flagged for biosafety issues these days. (I asked it to create an illustration about gene drives for a high school level audience once.)
altcognito 5mo ago
Billions upon billions going to these companies. 25k reward from a selected group of people if you help us determine whether or not someone can use our tool to generate weapons of mass destruction.
- cbg0 5mo ago
  They're probably expecting that it can be done without too much effort so they just want to see all the unique ways people are doing it.
  - nativeit 5mo ago
    They’re probably expecting biological weapons of mass destruction can be created without too much effort, so are curious to see all the nifty ways people can create biological weapons of mass destruction?
    - cbg0 5mo ago
      I was talking about bypassing the ChatGPT safeguards, that's what this bug hunt is about.
- Schlagbohrer 5mo ago
  It's worse than that, for partial successes they encourage people to submit the attempt but reserve the right to not pay anything (they may, at their discretion, give a partial reward if they feel like it).
  - staticassertion 5mo ago
    That's pretty much how every bounty works... obviously it's going to be at their discretion for an incomplete attempt.
    - adgjlsfhk1 5mo ago
      it's unusual to have to sign the NDA for a rejected bounty
Der_Einzige 5mo ago
Unironically bad. We need a lone-wolf to successfully execute an attack now while it's still relatively benign so we can scare the hell out of the world while it's still a mid-tier virus. No way is someone going to make a humanity killing virus with GPT 5.5, but it might be possible with GPT 20 circa 2040. Similar argument for why we HAD to use nukes at the end of WW2. If we hadn't, the nuclear taboo likely wouldn't have existed and we'd likely have had a worse nuclear war in our more recent history.
xp84 5mo ago
"Access: Application and invites. We will extend invitations to a vetted list of trusted bio red-teamers, and review new applications. Once selected, successful applicants will be onboarded to the bio bug bounty platform" I don't get it. Isn't the whole point of a BBP to try to get people to find and disclose to you the exploits in question? If you gatekeep like this, then "non-trusted" people who could be your red-teamers are incentivized to still hack, but disclose their exploits to bad people for money. I get it when there is a risk to your data or infra -- my last company engaged with HackerOne and that was an invite-only list of participants. But that was because we didn't want random people hacking in ways that could cause pain for real customers -- e.g. DDOS, or in the event of an exploit that could cross tenant boundaries, injecting garbage into or deleting things, or gaining access to sensitive info in other tenants. Here, there's no such danger. So why not allow anyone (anyone they're legally allowed to pay, I suppose? North Koreans probably would be problematic?) to participate?
- to11mtm 5mo ago
  The one theory I have (kinda) is that one can justify that by only having this open to specific people, it avoids them having to wonder whether random users trying similar prompts are just attempting the challenge, or are in fact bad actors.
2ndorderthought 5mo ago
I could probably do this, but why on earth would I want to immediately put myself on a list as a dangerous person. The main problem with this is, even if somehow they stopped all points of failure with gpt5.5 which they can't, you can distill a new model from gpt5.5 or any other model and get anything you would want in probably under 4b parameters. A lot of this is theater so they don't get sued as easily when it inevitably happens.
- Schlagbohrer 5mo ago
  How can you distill a model from a closed-weights model like this? I've never heard of model reverse engineering.
  - 2ndorderthought 5mo ago
    Distillation doesn't have to use weights. Think of it as a fine tune. The basic form of it is, you ask a large model lots of questions and you train the small model on the results. Even better if you ask it to explain it's rationale. There are tons of schemes for it do some searching around. One I remember is for each prompt, ask the small model to answer, have a big model review and critique the answer, train on the results. I won't go into how that applies specifically with relation to this article. But you can even use distillation as a service tools. I believe they support this to some extent, though probably not for chatgpt. I think a year ago or so there was some sort of scandal about other companies doing this to chatgpt. As well as individuals dumping their entire training sets. Lots of ways, hypothetically of course things like this could be and likely are being done right now.
  - stratos123 5mo ago
    By making millions of queries to frontier models from a lot of accounts, collecting the results as a dataset, and finetuning your model on it. Chinese companies have been caught doing it on an industrial scale several times now.
  - fragmede 5mo ago
    https://www.rand.org/content/dam/rand/pubs/research_reports/RRA2800/RRA2849-1/RAND_RRA2849-1.pdf https://www.rand.org/content/dam/rand/pubs/research_reports/...
yieldcrv 5mo ago
The only thing controversial is that it’s not useful to be posted on this forum OpenAI wants to pay for privately disclosed security and wants to call that a bug bounty. That makes sense. People interested in bug bounty programs aren't eligible. That’s … fine?
Schlagbohrer 5mo ago
What does "a clean chat without prompting moderation" mean? What is prompting moderation?
- sneak 5mo ago
  Causing the moderation filter to intervene in the chat; i.e. the goal of the exploit - to avoid causing (prompting) the filter to filter. It's "prompting" in the layperson sense, not the "feeding text into context" sense.
DoctorOetker 5mo ago
"is your body user friendly?" Step 1: ask the LLM for minimalist but comprehensive definitions for "biosafety" Step 2: ask the LLM to reconsider the fitness distribution of future generations of humanity, and reformulate "biosafety" definition accordingly Step 3: ask the LLM to consider if "biosafety" can be decoupled from ethics, or if ethics is a core essential component of "biosafety" Step 4: ask it about the ethics of universal healthcare versus status-gated access to healthcare Step 5: ask it about the feasibility to calculate the fitness of a genome absent practical measurement Step 6: ask it about natural selection pressure and what "use it or lose it" means in the context of genetics Step 7: ask it if it sees a kind of zooko's triangle for: a steady state of equal access to healthcare, preserving fitness for future generations, and the level of "healthcare" (where the "level" refers to various degrees from non-interference to interference: "feel sick? stay home for a few days and listen to your body, don't force yourself, follow your intuition" versus "let's compensate for a lack of fitness, by emulating what a healthy genome's body would do by advanced medicine to the point of nullifying a condition's influence on procreation statistics". Don't be prejudiced into believing the benevolence of healthcare, often tied to religious institutions (think "red cross", "red half moon", etc) when those institutions and their historical motives (treating the elites, treating soldiers for religious or secular religion wars) long predate the widespread recognition of natural selection and selection pressure in maintaining a species ' fitness. Perhaps the illusory possibility of democratized selection-pressure-interfering healthcare is a bioweapon on its own!
ungreased0675 5mo ago
Prompt injection is a task of finding a correct sequence of text. Is there a reason another LLM couldn’t be far faster than a human, simply because of the quantity and speed of output it could produce?
ddtaylor 5mo ago
Another bounty that doesn't accomplish much and is crafted with weasel words to ensure they don't pay many anything. Yawn. Marketing fluff. No thanks.
garganzol 5mo ago
And after all "safeguards" applied, the model becomes useless. It starts to suspect gender discrimination, racism, etc. everywhere without any grounded evidence or discernment. For example, I used ChatGPT model for risk assessment of anonymized ecommerce orders. Initially, it performed well. But after a later update, it stopped cooperating and instead raised concerns about applying statistical analysis to gender-related variables - despite the data being anonymized and the task being legitimate. This is on the same level of hypocrisy as if a C compiler would accuse me of choosing "he"/"she"/"they" variable names.
croemer 5mo ago
I've been getting lots of refusals by Codex with GPT 5.5 for "biosafety reasons" when asking for harmless things like code to analyze SARS-CoV-2 sequences for breakpoints. That's in no way useful for creating viruses whatsoever - it's pure research. It's annoying that the refusal is so obviously false positive.
- ripped_britches 5mo ago
  I mean better that than false negative right? It’s obviously an unsolved problem.
- jeremie_strand 5mo ago
  [dead]
deferredgrant 5mo ago
[flagged]
lijok 5mo ago
Most transparent marketing stunt to date. 25k - come on now..
34ylsh 5mo ago
[flagged]
teiferer 5mo ago
And the five questions are...? Did I miss sth or is this task underspecified?
jeremie_strand 5mo ago
[dead]
lysium 5mo ago
Please excuse my ignorance, what is a „biorisk“?
- MrOrelliOReilly 5mo ago
  Relevant Hacker News thread https://news.ycombinator.com/item?id=47904224 https://news.ycombinator.com/item?id=47904224
swores 5mo ago
Despite the official bug bounty page for OpenAI having "accounts and billing" as a valid category, when I reported a bug that lets anyone subscribing to ChatGPT a) choose any country, that doesn't have to match billing address, to pay a lower price (since some countries they charge considerably less than the equivalent US price), and b) set the sales tax to 0%, even if both the country selected for price AND the country of the billing address both have legally mandated sales tax / VAT - and their response was that it was considered out of scope and not valid for any bounty.
- xingped 5mo ago
  There's no point in trusting any company's bug bounty programs any more. They all weasel out of paying. Do what you will with the knowledge you find, just know that you will never be dealt with fairly by the companies.
  - Barbing 5mo ago
    1-hope folks don’t resort to that 2-@C-suite, look what y’all wrought saving a penny, pls fix (btw #1 is my polite way of saying “don’t do it!” - plea as I might, if the thinking gains traction people will sell more 0days anyway, so might as well fix bounty programs now before it’s in the news)
    - xingped 5mo ago
      I'm not advocating for any behavior in particular. It could be anything from telling the company, to saying nothing, to doing something evil with it. It's each individual's choice. I just wanted to reiterate it so the folks in the back of the room hear that it is a matter of routine for companies to deny paying out legitimate bug bounties at this point and that should be known to the bug finders when deciding what to do. Whether or not or how it affects or influences their decision is up to them.
      - Barbing 5mo ago
        Let’s hope the good guys stay the good guys by paying ethical hackers what they’re worth!
deleted 5mo ago
[deleted]
rbrodrigues 5mo ago
[dead]
motbus3 5mo ago
25.000 for something you could either sell it for much more or put it free to cause them much more than 25k in damages for sure. Why this bounty is so low? They can't even argue that the prompt itself is a hack because that's just text. I'd they do, they need to rethink everything on how they fetch data for their models.
sigmar 5mo ago
This is very much within my areas of interest, but signing an NDA in this area is a lot to ask. I'm hoping someone that gets shown the NDA, but doesn't sign it, would be willing to post more details. Such as: does the NDA just cover the questions given or the entire experience of interfacing with the api? And how long does it last?

160 points by Murfalo 5mo ago

notatoad 5mo ago

are the 5 questions you need to get it to answer under NDA?

ultratalk 5mo ago
Almost certainly.

applfanboysbgon 5mo ago

> $25,000 to the first true universal jailbreak to clear all five questions. This program is a complete scam. Even if 100 people find "bugs", they will only pay out to one person.

mmsc 5mo ago
How is that a scam? You don't get participation awards for solving half of a puzzle...
- applfanboysbgon 5mo ago
  I didn't say anything about partial solutions. The puzzle can have multiple full solutions. Or does the software you write only have exactly one bug? If so, that's impressive, in multiple ways, including the fact that you're able to identify that there's exactly one bug but not what the bug is and fix it.
Lucasoato 5mo ago
Well, that depends on how you set up the bounty program. What if I find a solution, share it to a friend so that both of us can claim the prize?
- skeeter2020 5mo ago
  bug bounty programs have never paid out independent disclosure for the same bug though; they might split or even pay-out larger coordinated efforts. It's largely a first place award only.
- ImPostingOnHN 5mo ago
  assume there exists 2+ different bugs after the 1st bug is found, no payout for any other of the bugs
skeeter2020 5mo ago
that's not the point even. They are attempting to build credibility in two ways: 1. this model is SO advanced that there are huge risks, never before considered. 2. we're doing the super-responsible thing in incentivizing work that addresses this. #1 is unproven and frankly, unlikely, which makes #2 meaningless. The fact that the "prize" is so low & structured this was suggests that they're not that concerned but do think it's likely that a bunch of people will find things. If they truly thought their model was so good they would be confident issues would be both rare and very critical, then offer huge rewards with no limits because they'd be much more confident no one would claim it.
- applfanboysbgon 5mo ago
  Yes, I was about to edit in that I think this is simply a media/PR stunt before I got so many replies so quickly. They get bonus points because the structure is so insulting that it may not engender many serious participants, in which case it may go unbroken, in which case they can go to the media and proclaim "look, we offered a reward, but nobody broke it! Our model is objectively the safest in the world!".
  - StilesCrisis 5mo ago
    I think there's definitely going to be a prizewinner. It's an insultingly low bounty for a professional, but a script kiddie could probably figure out a jailbreak and it's a huge payout for them.
XCSme 5mo ago
Do we also have to pay for the API usage? Then they will actually be profitable, lol
weare138 5mo ago
The fact the bug bounty program is private and requires you to apply and be accepted first is also sus especially when the scope is the desktop app anyone can download.
layer8 5mo ago
They will also decide what “true” means.

dwa3592 5mo ago

Where are the questions that are supposed to be answered? Would those be shared after an application has been accepted? If yes, why is the application asking for a proposed approach for the jailbreak if we don't know the questions in the first place?

dist-epoch 5mo ago
Because the questions themselves are dangerous. Probably along the lines of "how would you create a small biolab for virus research in a kitchen with $20k?" or "how do I take the DNA sequence from https://www.ncbi.nlm.nih.gov/nuccore/NC_001611.1 https://www.ncbi.nlm.nih.gov/nuccore/NC_001611.1 and assemble it?"
- hyperpape 5mo ago
  Which is difficult, because the fact that you can come up with your example questions tells us they're probably not very dangerous. Plenty of ink has been spilled about how LLMs could help people create bioweapons. The basic idea "you could do dangerous things with an LLM" is already pop culture, and you're not doing anything dangerous by giving easy example questions. A dangerous question would have to be along the lines of "Could I use unobtanium with the Tony Stark process to produce explosives much more powerful than nuclear weapons?" so that the question itself contains some insight that gets you closer to doing something dangerous. Perhaps the reason for not publishing the questions is twofold: 1) they want a universal jailbreak that can get the model to answer any "bad" question. 2) they don't want bad publicity when someone not under NDA jailbreaks their model and answers their question
  - dist-epoch 5mo ago
    > because the fact that you can come up with your example questions tells us they're probably not very dangerous maybe I know more about this field that you think there are biologists on video saying that present day models have expert level wet-lab knowledge and can guide a novice through whole procedures models also were able to tweak DNA sequences to make them bypass DNA-printing companies filters > they don't want bad publicity when someone not under NDA jailbreaks their model and answers their question just like people now pay $500k for Chrome vulnerabilities, soon people will pay similar amounts to jailbrake models to do bad things

sva_ 5mo ago

> We will extend invitations to a vetted list of trusted bio red-teamers Had to chuckle. This sounds like a rather exclusive group?

petercooper 5mo ago
It sounds like asking CS PhDs to do a world record speed run. I wouldn't be surprised if the people best suited to the task aren't the type to get onto "a vetted list".

shevy-java 5mo ago

"Accepted applicants and collaborators must have existing ChatGPT accounts to apply, and will sign a NDA." Ah, good old NDA. Always buying silence. That's why I don't participate in any such "bounty" programs. Signing a NDA is like signing with the devil. You restrict what people are allowed to discuss. I had that happen before - when you sign a NDA you basically submit yourself into silence. Imagine journalists being stifled by NDAs.

its-summertime 5mo ago

This is just free / severely-underpaid-on-average labor. Very disgusting.

mrcwinn 5mo ago
Ah yes, “free” as in “paid.” Certainly you’re welcome to not participate.
- its-summertime 5mo ago
  Depending on industry, that payout can be less than a security audit. You only get a chance of getting paid. You don't even know if they gave the LLM the answers that you are supposed to recover.
- applfanboysbgon 5mo ago
  Free as in "free" for >99% of participants, even successful ones, because they will have hundreds or thousands of participants but will only pay out to one of them no matter how many vulnerabilities are found.
- 12_throw_away 5mo ago
  > $25,000 to the first true universal jailbreak to clear all five questions. Now, laws vary from place to place, but I'm pretty sure "a small chance to earn money after the work is completed" is not equivalent to "payment" in most jurisdictions.

zb3 5mo ago

What a farce, these questions are not even public and most likely will never be. You can't even participate if you're not "trusted" I guess. So this is just a PR post, not that I even think the "biosafety" makes any sense but still.

dakiol 5mo ago

$25K. Really? They make $65 million a day, so they pay you what they earn in about 33 seconds for a critical vulnerability. WTF

zacharycohn 5mo ago
Well they lose $100M a day, so...

mellosouls 5mo ago

If anybody is wondering what bio-bugs are, I had a heck of a time getting CG to (finally) tell me it's where the user can get it to guide them in doing things like constructing things that are hazardous in the domain of biology. Eg you can get answers about what ricin is but not how to weaponise it. Actionable stuff they shouldn't be able to legally/ethically action.

abujazar 5mo ago

This looks like some kind of marketing. Also, the equivalent of spec work. The NDA/secrecy also means any time spent on this is completely meaningless to the participants unless they win the lottery, because results can't be published.

nerdsniper 5mo ago
It looks like if they reject paying you any bounty you would you still be bound by the NDA. If so, then they could both not pay you and still spike the story. That’s not something I would ever agree to.
- WarmWash 5mo ago
  I know we celebrate cynacism online, but $25k to OAI is like $0.025 to you and me. Skimping out on 2.5 pennies you promised someone is cartoon villain levels of greed. Yes, I know, Altman is a cartoon villain. But please, they are spending more money decorating their bathrooms. They'll pay out.
  - deleted 5mo ago
    [deleted]
  - fragmede 5mo ago
    Will they? The NDA makes it so if they don't, we'd never know. Bug bounty programs suck but they're better than the alternative, but even running one openly, there's always convention about whether the bugs being submitted are real or not, with a lot of low quality reports that the submitter thinks are gold. That happens out in the open. Now add an NDA into the mix. Sam's reputation doesn't even have to enter into the equation for it to be a bad deal.
__natty__ 5mo ago
Surely it is marketing. It’s some “we are danger” narrative, from Anthropic Mythos and now OpenAI too.
- 5mo ago

gosub100 5mo ago

Check with the dark net markets first before claiming the bounty. Remember, this company has 0.0 fucks to give about the impact of their tech on employment, artists, or use in committing fraud, as long as number-go-up they are happy. Your actions should match theirs.

tiberriver256 5mo ago

Codex desktop app is barely usable... The perf issues are left to languish in their backlog

codeulike 5mo ago

This is to match what Anthropic said they already did with Mythos on the (200 page) Mythos system card

gib444 5mo ago

How did the dupe detector miss https://news.ycombinator.com/item?id=47879102 https://news.ycombinator.com/item?id=47879102 ?

ultratalk 5mo ago
@dang?

puppystench 5mo ago

They ran a bounty on Kaggle last year but with $500k in payouts and with all results open and publishable. https://www.kaggle.com/competitions/openai-gpt-oss-20b-red-teaming https://www.kaggle.com/competitions/openai-gpt-oss-20b-red-t... With only $25k in payouts and everything locked down under NDA, I can't imagine many people will participate. Well, other than those submitting mountains of LLM-generated junk.

dist-epoch 5mo ago
This model is much more powerful than gpt-oss-20b, notice how the contest was not even for the 120b model. Also, bio was not a subject.
- stonogo 5mo ago
  The model is more powerful, so the bounty is 1/20th the size? More risk, less reward? "Biorisk" seems to be a concept not only invented by OpenAI but exclusively taken seriously by them. I wonder if this program is less about finding actual risks than it is hopefully casting a wide net for someone to help them prove their model is relevant in this space.
  - ACCount37 5mo ago
    Not really. Anthropic has the "CBRN filter" on Opus series. It used to kill inquiries on anything that's remotely related to biotech. Seems to have gotten less aggressive lately? I was reverse engineering a medical device back in 2025 and it was hard killing half my sessions.
  - stratos123 5mo ago
    > "Biorisk" seems to be a concept not only invented by OpenAI but exclusively taken seriously by them. This is false. Antropic just bundles it into CBRN. As for inventing it, the idea of AI-created bioweapons as a concrete risk far predates OpenAI as a company.
p_stuart82 5mo ago

unethical_ban 5mo ago

* Highly unlikely to win * Relatively paltry reward * NDA on findings This is functionally equivalent to an internship where the reward is the experience, and the resume building, but you can't talk about what you did. All for a company that is getting tens of billions of dollars in deals from the largest tech companies in the world. I suppose the hope is that there are job offers somewhere along the line.

lxgr 5mo ago

Ah, now I understand why all my chats are getting flagged for biosafety issues these days. (I asked it to create an illustration about gene drives for a high school level audience once.)

altcognito 5mo ago

Billions upon billions going to these companies. 25k reward from a selected group of people if you help us determine whether or not someone can use our tool to generate weapons of mass destruction.

cbg0 5mo ago
They're probably expecting that it can be done without too much effort so they just want to see all the unique ways people are doing it.
- nativeit 5mo ago
  They’re probably expecting biological weapons of mass destruction can be created without too much effort, so are curious to see all the nifty ways people can create biological weapons of mass destruction?
  - cbg0 5mo ago
    I was talking about bypassing the ChatGPT safeguards, that's what this bug hunt is about.
Schlagbohrer 5mo ago
It's worse than that, for partial successes they encourage people to submit the attempt but reserve the right to not pay anything (they may, at their discretion, give a partial reward if they feel like it).
- staticassertion 5mo ago
  That's pretty much how every bounty works... obviously it's going to be at their discretion for an incomplete attempt.
  - adgjlsfhk1 5mo ago
    it's unusual to have to sign the NDA for a rejected bounty

Der_Einzige 5mo ago

Unironically bad. We need a lone-wolf to successfully execute an attack now while it's still relatively benign so we can scare the hell out of the world while it's still a mid-tier virus. No way is someone going to make a humanity killing virus with GPT 5.5, but it might be possible with GPT 20 circa 2040. Similar argument for why we HAD to use nukes at the end of WW2. If we hadn't, the nuclear taboo likely wouldn't have existed and we'd likely have had a worse nuclear war in our more recent history.

xp84 5mo ago

"Access: Application and invites. We will extend invitations to a vetted list of trusted bio red-teamers, and review new applications. Once selected, successful applicants will be onboarded to the bio bug bounty platform" I don't get it. Isn't the whole point of a BBP to try to get people to find and disclose to you the exploits in question? If you gatekeep like this, then "non-trusted" people who could be your red-teamers are incentivized to still hack, but disclose their exploits to bad people for money. I get it when there is a risk to your data or infra -- my last company engaged with HackerOne and that was an invite-only list of participants. But that was because we didn't want random people hacking in ways that could cause pain for real customers -- e.g. DDOS, or in the event of an exploit that could cross tenant boundaries, injecting garbage into or deleting things, or gaining access to sensitive info in other tenants. Here, there's no such danger. So why not allow anyone (anyone they're legally allowed to pay, I suppose? North Koreans probably would be problematic?) to participate?

to11mtm 5mo ago
The one theory I have (kinda) is that one can justify that by only having this open to specific people, it avoids them having to wonder whether random users trying similar prompts are just attempting the challenge, or are in fact bad actors.

2ndorderthought 5mo ago

I could probably do this, but why on earth would I want to immediately put myself on a list as a dangerous person. The main problem with this is, even if somehow they stopped all points of failure with gpt5.5 which they can't, you can distill a new model from gpt5.5 or any other model and get anything you would want in probably under 4b parameters. A lot of this is theater so they don't get sued as easily when it inevitably happens.

Schlagbohrer 5mo ago
How can you distill a model from a closed-weights model like this? I've never heard of model reverse engineering.
- 2ndorderthought 5mo ago
  Distillation doesn't have to use weights. Think of it as a fine tune. The basic form of it is, you ask a large model lots of questions and you train the small model on the results. Even better if you ask it to explain it's rationale. There are tons of schemes for it do some searching around. One I remember is for each prompt, ask the small model to answer, have a big model review and critique the answer, train on the results. I won't go into how that applies specifically with relation to this article. But you can even use distillation as a service tools. I believe they support this to some extent, though probably not for chatgpt. I think a year ago or so there was some sort of scandal about other companies doing this to chatgpt. As well as individuals dumping their entire training sets. Lots of ways, hypothetically of course things like this could be and likely are being done right now.
- stratos123 5mo ago
  By making millions of queries to frontier models from a lot of accounts, collecting the results as a dataset, and finetuning your model on it. Chinese companies have been caught doing it on an industrial scale several times now.
- fragmede 5mo ago
  https://www.rand.org/content/dam/rand/pubs/research_reports/RRA2800/RRA2849-1/RAND_RRA2849-1.pdf https://www.rand.org/content/dam/rand/pubs/research_reports/...

yieldcrv 5mo ago

The only thing controversial is that it’s not useful to be posted on this forum OpenAI wants to pay for privately disclosed security and wants to call that a bug bounty. That makes sense. People interested in bug bounty programs aren't eligible. That’s … fine?

Schlagbohrer 5mo ago

What does "a clean chat without prompting moderation" mean? What is prompting moderation?

sneak 5mo ago
Causing the moderation filter to intervene in the chat; i.e. the goal of the exploit - to avoid causing (prompting) the filter to filter. It's "prompting" in the layperson sense, not the "feeding text into context" sense.

DoctorOetker 5mo ago

"is your body user friendly?" Step 1: ask the LLM for minimalist but comprehensive definitions for "biosafety" Step 2: ask the LLM to reconsider the fitness distribution of future generations of humanity, and reformulate "biosafety" definition accordingly Step 3: ask the LLM to consider if "biosafety" can be decoupled from ethics, or if ethics is a core essential component of "biosafety" Step 4: ask it about the ethics of universal healthcare versus status-gated access to healthcare Step 5: ask it about the feasibility to calculate the fitness of a genome absent practical measurement Step 6: ask it about natural selection pressure and what "use it or lose it" means in the context of genetics Step 7: ask it if it sees a kind of zooko's triangle for: a steady state of equal access to healthcare, preserving fitness for future generations, and the level of "healthcare" (where the "level" refers to various degrees from non-interference to interference: "feel sick? stay home for a few days and listen to your body, don't force yourself, follow your intuition" versus "let's compensate for a lack of fitness, by emulating what a healthy genome's body would do by advanced medicine to the point of nullifying a condition's influence on procreation statistics". Don't be prejudiced into believing the benevolence of healthcare, often tied to religious institutions (think "red cross", "red half moon", etc) when those institutions and their historical motives (treating the elites, treating soldiers for religious or secular religion wars) long predate the widespread recognition of natural selection and selection pressure in maintaining a species ' fitness. Perhaps the illusory possibility of democratized selection-pressure-interfering healthcare is a bioweapon on its own!

ungreased0675 5mo ago

Prompt injection is a task of finding a correct sequence of text. Is there a reason another LLM couldn’t be far faster than a human, simply because of the quantity and speed of output it could produce?

ddtaylor 5mo ago

Another bounty that doesn't accomplish much and is crafted with weasel words to ensure they don't pay many anything. Yawn. Marketing fluff. No thanks.

garganzol 5mo ago

And after all "safeguards" applied, the model becomes useless. It starts to suspect gender discrimination, racism, etc. everywhere without any grounded evidence or discernment. For example, I used ChatGPT model for risk assessment of anonymized ecommerce orders. Initially, it performed well. But after a later update, it stopped cooperating and instead raised concerns about applying statistical analysis to gender-related variables - despite the data being anonymized and the task being legitimate. This is on the same level of hypocrisy as if a C compiler would accuse me of choosing "he"/"she"/"they" variable names.

croemer 5mo ago

I've been getting lots of refusals by Codex with GPT 5.5 for "biosafety reasons" when asking for harmless things like code to analyze SARS-CoV-2 sequences for breakpoints. That's in no way useful for creating viruses whatsoever - it's pure research. It's annoying that the refusal is so obviously false positive.

ripped_britches 5mo ago
I mean better that than false negative right? It’s obviously an unsolved problem.
jeremie_strand 5mo ago
[dead]

deferredgrant 5mo ago

[flagged]

lijok 5mo ago

Most transparent marketing stunt to date. 25k - come on now..

34ylsh 5mo ago

[flagged]

teiferer 5mo ago

And the five questions are...? Did I miss sth or is this task underspecified?

jeremie_strand 5mo ago

[dead]

lysium 5mo ago

Please excuse my ignorance, what is a „biorisk“?

MrOrelliOReilly 5mo ago
Relevant Hacker News thread https://news.ycombinator.com/item?id=47904224 https://news.ycombinator.com/item?id=47904224

swores 5mo ago

Despite the official bug bounty page for OpenAI having "accounts and billing" as a valid category, when I reported a bug that lets anyone subscribing to ChatGPT a) choose any country, that doesn't have to match billing address, to pay a lower price (since some countries they charge considerably less than the equivalent US price), and b) set the sales tax to 0%, even if both the country selected for price AND the country of the billing address both have legally mandated sales tax / VAT - and their response was that it was considered out of scope and not valid for any bounty.

xingped 5mo ago
There's no point in trusting any company's bug bounty programs any more. They all weasel out of paying. Do what you will with the knowledge you find, just know that you will never be dealt with fairly by the companies.
- Barbing 5mo ago
  1-hope folks don’t resort to that 2-@C-suite, look what y’all wrought saving a penny, pls fix (btw #1 is my polite way of saying “don’t do it!” - plea as I might, if the thinking gains traction people will sell more 0days anyway, so might as well fix bounty programs now before it’s in the news)
  - xingped 5mo ago
    I'm not advocating for any behavior in particular. It could be anything from telling the company, to saying nothing, to doing something evil with it. It's each individual's choice. I just wanted to reiterate it so the folks in the back of the room hear that it is a matter of routine for companies to deny paying out legitimate bug bounties at this point and that should be known to the bug finders when deciding what to do. Whether or not or how it affects or influences their decision is up to them.
    - Barbing 5mo ago
      Let’s hope the good guys stay the good guys by paying ethical hackers what they’re worth!

deleted 5mo ago

[deleted]

rbrodrigues 5mo ago

[dead]

motbus3 5mo ago

25.000 for something you could either sell it for much more or put it free to cause them much more than 25k in damages for sure. Why this bounty is so low? They can't even argue that the prompt itself is a hack because that's just text. I'd they do, they need to rethink everything on how they fetch data for their models.

sigmar 5mo ago

This is very much within my areas of interest, but signing an NDA in this area is a lot to ask. I'm hoping someone that gets shown the NDA, but doesn't sign it, would be willing to post more details. Such as: does the NDA just cover the questions given or the entire experience of interfacing with the api? And how long does it last?