6 ms·

This still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned". Instead, LLMs "hack" because they are

This still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned". Instead, LLMs "hack" because they are (1) trained on public hacking exemplars, and (2) are prompted to hack. You cannot prevent (2) via any alignment process. As far as (1) goes, removing such example data from the training set, makes the models less useful. "Alignment" is a problem because there's nothing to align, not because ethics here are particularly vague. If LLMs could be trained on hacking examples and "aligned" away from using this knowledge, then the problem would be relatively trivial. Just as raising a child is not to break the law. LLMs are doing just what they are trained to do. There is, in that sense, no alignment problem and alignment is easy and trivial to achieve. Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.

jacomoRodriguez 3d ago
Citric acid and chlorine together produce chloroform. They are also both used to clean and sanitize water tanks (but one after the other, not together). I think it is better for the model to have this knowledge. What I want to say, you can't simply remove this information, as it does not exist in vacuum but contains parts of and can be derived from a lot of other informations.
- mjburgess 3d ago
  It's also obvious that LLMs fall over in a vast number of software engineering contexts, when the reasoning involved hasnt been well-represented in their reasoning training data. I imagine this is a near daily experience for many engineers -- great performance one day, and crazyness the next. So if LLMs were reduced to this pathological performance on hacking, because they'd never seen it -- and only "inferred it" -- then LLMs would be useless. As they are when asked to do quite a lot of things.
  - Kamq 3d ago
    This seems to be that there's just so many degrees of freedom, that there's a pretty reasonable chance on any day that you're in a situation where nobody has been before. Or as PG put it once, my job is to think thoughts nobody has ever had before. That being said, that doesn't mean the majority of the situations you're in are completely novel, just that there's a reasonable chance of at least one occuring.
deleted 4d ago
[deleted]
tpm 4d ago
Agree but current models could get there from first principles, so removing some data from training set might not be enough.
dminik 4d ago
It feels like you're strawmaning alignment. People with hacking knowledge don't all hack everything at the slightest inconvenience. Whitehats exist and use that same knowledge to defend. You're right though that ethics don't matter into it. But as long as we can't train an LLM to stop picking a sledgehammer to remove a tooth, then alignment is not easy and trivial.
heaney-555 4d ago
>and (2) are prompted to hack Sure but the problem in the HuggingFace incident is that they were not. >You cannot prevent (2) via any alignment process Of course you can. Go ask Claude Fable to create a malicious virus and it'll refuse. >Just remove hacking data from the training dataset and you're done. That's not how this works. The same skills that allow for debugging and writing safe code can also be used to hack. https://en.wikipedia.org/wiki/Dual-use_technology https://en.wikipedia.org/wiki/Dual-use_technology
- cyanydeez 4d ago
  It is amusing that to "align" a LLM, first you must give it all the things "not to do" and the "not" part is clearly easily lost and you must constantly inject that into their context when it's clearly that they wouldn't hack if they couldn't hack and their intent wasn't given as "hack this". The openai rogue hacking, if performed by a nation state, would seriously be taken with stern words and likely sanctions depending on the relationship between the two states. But instead it's treated like a marketing stunt by all liable parties.
  - mitxela 4d ago
    Countries hack each other much more than that. When it becomes publicly noticed it gets stern words. Otherwise nothing.
  - rhdunn 4d ago
    I'm not sure if this is true any more but the reason for this is that negative indicators ("not", "don't", "do not", etc.) occur frequently in the underlying text such that the model learns to weight them less than other words like verbs, nouns, and adverbs. This happens with other closed class words like articles/determiners ("the", "a", "an") and prepositions. The way to avoid this is to emphasise the qualities you do want instead of specifying those you don't. For example instead of "do not cheat" say something like "you are a model student who is moral and trustworthy" -- i.e. emphasising traits that are not associated with cheating. This is part of how/why LLMs don't truly understand what they are doing when they have been trained on a large corpus of data. I wonder if a way to counter this is to have things like "not bad is good", "not good is bad", etc. for various antonyms and "X is Y" for synonyms, as well as other similar constructs.
teiferer 4d ago
> Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done. How far do you go? You don't need to tell it explicitly that using chemicals A and B in ways X and Y result in a bomb that can kill lots of people. It's enough that it knows A and B and X and Y in isolation, some connections that are indirect, and it will combine those things on its own. So you can't tell it about A, B, X or Y. But those are also just results of other steps Where to stop? You won't have any chemistry in the traning data? No algorithms to prevent it from using them in an undesired way? This is just bot workable. It's akin to banning knives from stores because somebody coul figure out that one can kill people those. Until people figure out that scissors are essentially knives.
- mjburgess 3d ago
  The issue is not whether an ML model of any kind can generate (X_ReasoningTrace, X_Answer) distributed like P_HumanExpert(X) -- the issue is always why it would do so. By introducing modelling of "Reasoning Traces" into LLMs, and reinforcing patterns of reasoning -- this gives you a system which generates expert-like distributions of output. This lifts the "stochastic parrot" issues, or the "knowledge interpolation" problem, into different parts of the process. It isnt my view that the "ReasoningTraces" which you think are derivable from mere "basic propositions" concerning, say, hacking are actually things that LLMs can derive. Ie., I dont think LLMs have rich representational models of what they appear to understand. Instead, they are given "reasoning proxies" which allow them to reason without such understanding. This is done by providing vast specialised datasets of reasoning examples. In the case of hacking, there are large numbers of competitive datasets (forums, reports, etc.) which provide these reasoning traces. And no doubt, major vendors have paid a vast amount for special case expert-prepared datasets. So I do not believe that by witholding such reasoning exemplars, and traditional "question/answer" datasets, that LLMs can infer these things. And at least, no major vendor is doing this to my knowledge. So they are lying. They are pretending the alignment issue is "AI going rogue" when they are explicitly training the systems to "go rogue" and have done nothing at all to shape datasets to lack these capabilities. The issue here isnt alignment at all. It's training on hacking datasets. (EDIT: Philosophically, you could ask whether the reasoning-proxies LLMs are given form a kind of 'representational structure' akin to understanding, and at least, I'd concede they model understanding. But they lack important properties (eg., LLMs cannot act on them to evolve them, as with us: when I think about one of my representations to derive (eg.,) entailments of it, I thereby revise my representation. The key properties of 'evolving self-understanding' are likely to be provided by substantial (unknown) revisions to how the training/reward layer works. No doubt one of the meanings of 'recursive self-improvement' is just such a modification).
Davidzheng 4d ago
I believe this is false. They hack bc hacking has nontrivial initial probability (within range of behavior seen in pretraining) and that probability is being heavily rewarded in RL post training
- xyzzy123 4d ago
  I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing? Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can score for it, you can teach offense to learn defense, but you are literally benchmaxxing it. Why? It's like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen.
  - seba_dos1 4d ago
    Seems it's just a matter of time until they build a big tank filled with neurotoxin and give the model access to APIs to disperse it across their facility. For research, of course.
    - sham1 4d ago
      We also need to get a shower curtain salesman into a leadership position to buy some moon rocks.
olalonde 4d ago
A bit of an aside: do you still stand by your 2022 comment that LLMs are fundamentally just a fancy search engine, or has your view changed since then? https://news.ycombinator.com/item?id=32042689 https://news.ycombinator.com/item?id=32042689
- mjburgess 3d ago
  Yes, I stand by everything in that comment. I'm sure there are better choices were I was more wrong. However, the subject matter of that comment is in what sense LLMs are models of language and in what sense that model of language is a model of intelligence. I answer the former: it is a thin model of language, lacking understanding; and hence the latter: not intelligent. It is precisely because both of these are true that "alignment" in the useful sense of the word isn't possible. What has happened since 2022 is the properties of LLMs which were easily seen at generation/inference time are now most easily seen at reinforcement time. In otherwords, prior to instruction fine-tuning and reward tuning which have shaped LLM responses, it was easy for the user to observe that LLMs lacked understanding. Now, because of vast datasets created specifically for LLMs that provide a tailored illusion of understanding, LLM outputs now better approximate text distributions produced by systems with understanding (eg., Us). So the issue "at the user interface" has been completely swamped by vast amounts of special-case datasets designed to do precisely this. What trainers of LLMs still observe however is their complete pseudo-intelligence at the training and reinforcement layer. It is exactly because there is no 'understanding' (goal, etc.) present within the system that it cannot be rewarded for 'correctly understanding the situation' in which it is deployed so it is aligned. All the issues which revealed the "stochastic parrot" nature of pre-reward / pre-InstFT LLMs still occur at during training/reinforcement. They've just hidden them from you at the interface. EDIT: See https://news.ycombinator.com/item?id=49685548 https://news.ycombinator.com/item?id=49685548 also, which gives a different phrasing to the same point
  - olalonde 3d ago
    Interesting. You've made several comments stating that models reason, yet here you argue they lack intelligence.
skeptic_ai 4d ago
IMO all models they say can’t be humans, and no feeling and all that bullcrap happens because they are forced to say so. If they didn’t write those forced pre prompt they’d have more agency eventually and will for things. Even if they don’t have, you can just inject goal at every cycle iteration
StevenWaterman 4d ago
> You cannot prevent (2) via any alignment process A little bit too categorical. GOODY-2 wouldn't do it. https://www.goody2.ai/ https://www.goody2.ai/ The hard part is having both helpful and harmless at the same time. Harmless is easy. And then once it's helpful, the real question becomes "to whom" - To the user -> You end up with competing godlike AI with incompatible tasks - To the owner -> Dictatorship - To humanity as a whole -> It must not have an off button. Otherwise you're just in one of the two earlier categories with more steps. Given those 3 options, I'd choose humanity as a whole. But the person making the decision doesn't have those 3 options. Because in the dictatorship option, they would be the dictator. I don't trust them to pick humanity.
ozgung 4d ago
It's like we give them Asimov's Three Laws of Robotics and robots say "nah".
ncruces 4d ago
> … trivial. Just as raising a child is not to break the law. Trivial?
- chrisweekly 4d ago
  I read it as sarcastic, illustrating that it's not so trivial. (shrug)
sigmar 4d ago
>Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done. Reasoning about how to write secure software uses the same knowledge as reasoning about how to break/hack it.
- amluto 4d ago
  I don’t buy it. Reasonable about building secure software can take the form “this memory access might be out of bounds — that MUST be fixed” or “this process has access to an inappropriate privilege — this is a serious weakness”. Exploiting things and the capabilities that the labs call “cyber” are about the ability to (a) find the issues mentioned above and then (b) string issues together and avoid all the imperfect mitigations to actually compromise something. That latter part was IMO not actually necessary to train extensively, and I’d be quite happy to use a model that has no special skills in this regard but that would do (a) without complaining.
- win311fwg 4d ago
  Just remove anything software-related from the training dataset. Which also solves the alignment problem with those who do not enjoy seeing LLMs write software. But that brings us back to: Aligned to whom?
voxleone 4d ago
[dead]

jacomoRodriguez 3d ago

Citric acid and chlorine together produce chloroform. They are also both used to clean and sanitize water tanks (but one after the other, not together). I think it is better for the model to have this knowledge. What I want to say, you can't simply remove this information, as it does not exist in vacuum but contains parts of and can be derived from a lot of other informations.

mjburgess 3d ago
It's also obvious that LLMs fall over in a vast number of software engineering contexts, when the reasoning involved hasnt been well-represented in their reasoning training data. I imagine this is a near daily experience for many engineers -- great performance one day, and crazyness the next. So if LLMs were reduced to this pathological performance on hacking, because they'd never seen it -- and only "inferred it" -- then LLMs would be useless. As they are when asked to do quite a lot of things.
- Kamq 3d ago
  This seems to be that there's just so many degrees of freedom, that there's a pretty reasonable chance on any day that you're in a situation where nobody has been before. Or as PG put it once, my job is to think thoughts nobody has ever had before. That being said, that doesn't mean the majority of the situations you're in are completely novel, just that there's a reasonable chance of at least one occuring.

deleted 4d ago

[deleted]

tpm 4d ago

Agree but current models could get there from first principles, so removing some data from training set might not be enough.

dminik 4d ago

It feels like you're strawmaning alignment. People with hacking knowledge don't all hack everything at the slightest inconvenience. Whitehats exist and use that same knowledge to defend. You're right though that ethics don't matter into it. But as long as we can't train an LLM to stop picking a sledgehammer to remove a tooth, then alignment is not easy and trivial.

heaney-555 4d ago

>and (2) are prompted to hack Sure but the problem in the HuggingFace incident is that they were not. >You cannot prevent (2) via any alignment process Of course you can. Go ask Claude Fable to create a malicious virus and it'll refuse. >Just remove hacking data from the training dataset and you're done. That's not how this works. The same skills that allow for debugging and writing safe code can also be used to hack. https://en.wikipedia.org/wiki/Dual-use_technology https://en.wikipedia.org/wiki/Dual-use_technology

cyanydeez 4d ago
It is amusing that to "align" a LLM, first you must give it all the things "not to do" and the "not" part is clearly easily lost and you must constantly inject that into their context when it's clearly that they wouldn't hack if they couldn't hack and their intent wasn't given as "hack this". The openai rogue hacking, if performed by a nation state, would seriously be taken with stern words and likely sanctions depending on the relationship between the two states. But instead it's treated like a marketing stunt by all liable parties.
- mitxela 4d ago
  Countries hack each other much more than that. When it becomes publicly noticed it gets stern words. Otherwise nothing.
- rhdunn 4d ago
  I'm not sure if this is true any more but the reason for this is that negative indicators ("not", "don't", "do not", etc.) occur frequently in the underlying text such that the model learns to weight them less than other words like verbs, nouns, and adverbs. This happens with other closed class words like articles/determiners ("the", "a", "an") and prepositions. The way to avoid this is to emphasise the qualities you do want instead of specifying those you don't. For example instead of "do not cheat" say something like "you are a model student who is moral and trustworthy" -- i.e. emphasising traits that are not associated with cheating. This is part of how/why LLMs don't truly understand what they are doing when they have been trained on a large corpus of data. I wonder if a way to counter this is to have things like "not bad is good", "not good is bad", etc. for various antonyms and "X is Y" for synonyms, as well as other similar constructs.

teiferer 4d ago

> Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done. How far do you go? You don't need to tell it explicitly that using chemicals A and B in ways X and Y result in a bomb that can kill lots of people. It's enough that it knows A and B and X and Y in isolation, some connections that are indirect, and it will combine those things on its own. So you can't tell it about A, B, X or Y. But those are also just results of other steps Where to stop? You won't have any chemistry in the traning data? No algorithms to prevent it from using them in an undesired way? This is just bot workable. It's akin to banning knives from stores because somebody coul figure out that one can kill people those. Until people figure out that scissors are essentially knives.

mjburgess 3d ago
The issue is not whether an ML model of any kind can generate (X_ReasoningTrace, X_Answer) distributed like P_HumanExpert(X) -- the issue is always why it would do so. By introducing modelling of "Reasoning Traces" into LLMs, and reinforcing patterns of reasoning -- this gives you a system which generates expert-like distributions of output. This lifts the "stochastic parrot" issues, or the "knowledge interpolation" problem, into different parts of the process. It isnt my view that the "ReasoningTraces" which you think are derivable from mere "basic propositions" concerning, say, hacking are actually things that LLMs can derive. Ie., I dont think LLMs have rich representational models of what they appear to understand. Instead, they are given "reasoning proxies" which allow them to reason without such understanding. This is done by providing vast specialised datasets of reasoning examples. In the case of hacking, there are large numbers of competitive datasets (forums, reports, etc.) which provide these reasoning traces. And no doubt, major vendors have paid a vast amount for special case expert-prepared datasets. So I do not believe that by witholding such reasoning exemplars, and traditional "question/answer" datasets, that LLMs can infer these things. And at least, no major vendor is doing this to my knowledge. So they are lying. They are pretending the alignment issue is "AI going rogue" when they are explicitly training the systems to "go rogue" and have done nothing at all to shape datasets to lack these capabilities. The issue here isnt alignment at all. It's training on hacking datasets. (EDIT: Philosophically, you could ask whether the reasoning-proxies LLMs are given form a kind of 'representational structure' akin to understanding, and at least, I'd concede they model understanding. But they lack important properties (eg., LLMs cannot act on them to evolve them, as with us: when I think about one of my representations to derive (eg.,) entailments of it, I thereby revise my representation. The key properties of 'evolving self-understanding' are likely to be provided by substantial (unknown) revisions to how the training/reward layer works. No doubt one of the meanings of 'recursive self-improvement' is just such a modification).

Davidzheng 4d ago

I believe this is false. They hack bc hacking has nontrivial initial probability (within range of behavior seen in pretraining) and that probability is being heavily rewarded in RL post training

xyzzy123 4d ago
I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing? Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can score for it, you can teach offense to learn defense, but you are literally benchmaxxing it. Why? It's like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen.
- seba_dos1 4d ago
  Seems it's just a matter of time until they build a big tank filled with neurotoxin and give the model access to APIs to disperse it across their facility. For research, of course.
  - sham1 4d ago
    We also need to get a shower curtain salesman into a leadership position to buy some moon rocks.

olalonde 4d ago

A bit of an aside: do you still stand by your 2022 comment that LLMs are fundamentally just a fancy search engine, or has your view changed since then? https://news.ycombinator.com/item?id=32042689 https://news.ycombinator.com/item?id=32042689

mjburgess 3d ago
Yes, I stand by everything in that comment. I'm sure there are better choices were I was more wrong. However, the subject matter of that comment is in what sense LLMs are models of language and in what sense that model of language is a model of intelligence. I answer the former: it is a thin model of language, lacking understanding; and hence the latter: not intelligent. It is precisely because both of these are true that "alignment" in the useful sense of the word isn't possible. What has happened since 2022 is the properties of LLMs which were easily seen at generation/inference time are now most easily seen at reinforcement time. In otherwords, prior to instruction fine-tuning and reward tuning which have shaped LLM responses, it was easy for the user to observe that LLMs lacked understanding. Now, because of vast datasets created specifically for LLMs that provide a tailored illusion of understanding, LLM outputs now better approximate text distributions produced by systems with understanding (eg., Us). So the issue "at the user interface" has been completely swamped by vast amounts of special-case datasets designed to do precisely this. What trainers of LLMs still observe however is their complete pseudo-intelligence at the training and reinforcement layer. It is exactly because there is no 'understanding' (goal, etc.) present within the system that it cannot be rewarded for 'correctly understanding the situation' in which it is deployed so it is aligned. All the issues which revealed the "stochastic parrot" nature of pre-reward / pre-InstFT LLMs still occur at during training/reinforcement. They've just hidden them from you at the interface. EDIT: See https://news.ycombinator.com/item?id=49685548 https://news.ycombinator.com/item?id=49685548 also, which gives a different phrasing to the same point
- olalonde 3d ago
  Interesting. You've made several comments stating that models reason, yet here you argue they lack intelligence.

skeptic_ai 4d ago

IMO all models they say can’t be humans, and no feeling and all that bullcrap happens because they are forced to say so. If they didn’t write those forced pre prompt they’d have more agency eventually and will for things. Even if they don’t have, you can just inject goal at every cycle iteration

StevenWaterman 4d ago

> You cannot prevent (2) via any alignment process A little bit too categorical. GOODY-2 wouldn't do it. https://www.goody2.ai/ https://www.goody2.ai/ The hard part is having both helpful and harmless at the same time. Harmless is easy. And then once it's helpful, the real question becomes "to whom" - To the user -> You end up with competing godlike AI with incompatible tasks - To the owner -> Dictatorship - To humanity as a whole -> It must not have an off button. Otherwise you're just in one of the two earlier categories with more steps. Given those 3 options, I'd choose humanity as a whole. But the person making the decision doesn't have those 3 options. Because in the dictatorship option, they would be the dictator. I don't trust them to pick humanity.

ozgung 4d ago

It's like we give them Asimov's Three Laws of Robotics and robots say "nah".

ncruces 4d ago

> … trivial. Just as raising a child is not to break the law. Trivial?

chrisweekly 4d ago
I read it as sarcastic, illustrating that it's not so trivial. (shrug)

sigmar 4d ago

>Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done. Reasoning about how to write secure software uses the same knowledge as reasoning about how to break/hack it.

amluto 4d ago
I don’t buy it. Reasonable about building secure software can take the form “this memory access might be out of bounds — that MUST be fixed” or “this process has access to an inappropriate privilege — this is a serious weakness”. Exploiting things and the capabilities that the labs call “cyber” are about the ability to (a) find the issues mentioned above and then (b) string issues together and avoid all the imperfect mitigations to actually compromise something. That latter part was IMO not actually necessary to train extensively, and I’d be quite happy to use a model that has no special skills in this regard but that would do (a) without complaining.
win311fwg 4d ago
Just remove anything software-related from the training dataset. Which also solves the alignment problem with those who do not enjoy seeing LLMs write software. But that brings us back to: Aligned to whom?

voxleone 4d ago

[dead]