6 ms·

Why dont they train LLMs not to speak like that? Is it some tragedy of the commons here?

schubidubiduba 10d ago
They are certainly trying to
tudelo 10d ago
You ever wonder why recent claude models speak in riddles? I dunno, maybe all those "rare" books? They may have been rare for a reason.
- knollimar 10d ago
  Rare like a steak not like a treasure.
rcxdude 10d ago
LLMs get a lot of finetuning, but I suspect there are two things that can cause this kind of writing: Firstly, some parts of the RLHF involve human graders on the LLM's performance. I suspect their general bias towards a punchy, persuasive writing style could come from what biases the graders towards preferring that response, especially in shorter segments and when the grader is not focused on writing style Secondly, later parts of the finetuning involve reinforcement learning on achieving certain tasks which are automatically graded: stuff like coding tasks. I think this can create a kind of feedback loop where the style drifts further, and you get the kind of LLM tics which are even more extreme (it might be that they incidentally help somehow with the actual tasks, or it might be a drift that comes from the grader also now being an LLM or some of this finetuning happening on output from other models). The more recent claude models seem to suffer from this a lot, moreso than earlier ones.
- valleyer 10d ago
  I'm not an expert, but... might this indicate the need for "sub-models", meant to be invoked by the general model to write good prose for it? Those sub-models would not be RLed on code (or any algorithmic-feedback tasks that might reinforce bad writing), and could specifically be tuned for prose, even at the cost of general intelligence (which would be provided by the worse-at-writing more general model).
ahepp 10d ago
A bigger question I'm interested in is why do LLMs speak like that in the first place? Is that really what you get if you took the average of the English language? It would be difficult for me to believe that. Is there something about tuning for desirable qualities that forces LLMs to have this voice?
- Ygg2 10d ago
  Not average of English. Average of all written text. Which probably includes lots of marketing and hypetexts.
  - AnimalMuppet 10d ago
    And The New Yorker - writing that is written to sound impressive, and takes forever to get to the point. I hated that kind of writing in The New Yorker long before LLMs made it cool to hate that.
- gosub100 10d ago
  It wouldn't surprise me if it takes more effort to be succinct. There's some famous quote from I think Mark Twain "this letter would have been shorter but I ran out of time". Just my guess

They are certainly trying to

You ever wonder why recent claude models speak in riddles? I dunno, maybe all those "rare" books? They may have been rare for a reason.

knollimar 10d ago
Rare like a steak not like a treasure.

rcxdude 10d ago

LLMs get a lot of finetuning, but I suspect there are two things that can cause this kind of writing: Firstly, some parts of the RLHF involve human graders on the LLM's performance. I suspect their general bias towards a punchy, persuasive writing style could come from what biases the graders towards preferring that response, especially in shorter segments and when the grader is not focused on writing style Secondly, later parts of the finetuning involve reinforcement learning on achieving certain tasks which are automatically graded: stuff like coding tasks. I think this can create a kind of feedback loop where the style drifts further, and you get the kind of LLM tics which are even more extreme (it might be that they incidentally help somehow with the actual tasks, or it might be a drift that comes from the grader also now being an LLM or some of this finetuning happening on output from other models). The more recent claude models seem to suffer from this a lot, moreso than earlier ones.

valleyer 10d ago
I'm not an expert, but... might this indicate the need for "sub-models", meant to be invoked by the general model to write good prose for it? Those sub-models would not be RLed on code (or any algorithmic-feedback tasks that might reinforce bad writing), and could specifically be tuned for prose, even at the cost of general intelligence (which would be provided by the worse-at-writing more general model).

ahepp 10d ago

A bigger question I'm interested in is why do LLMs speak like that in the first place? Is that really what you get if you took the average of the English language? It would be difficult for me to believe that. Is there something about tuning for desirable qualities that forces LLMs to have this voice?

Ygg2 10d ago
Not average of English. Average of all written text. Which probably includes lots of marketing and hypetexts.
- AnimalMuppet 10d ago
  And The New Yorker - writing that is written to sound impressive, and takes forever to get to the point. I hated that kind of writing in The New Yorker long before LLMs made it cool to hate that.
gosub100 10d ago
It wouldn't surprise me if it takes more effort to be succinct. There's some famous quote from I think Mark Twain "this letter would have been shorter but I ran out of time". Just my guess