8 ms·

Why Claude's Comment Paper Is a Poor Rebuttal

50 points by vectorhacker 1y ago

low_tech_love 1y ago
A fundamental problem that we’re still far away from solving is not necessarily that LLMs/LRMs cannot reason the same way that we do (which I guess should be clear by now); but that they might not have to. They generate slop so fast that, if one can benefit a little bit from each output, i.e. if you can find a little bit of use hidden beneath the mountain of meaningless text they’ll create, then this might still be more valuable than preemptively taking the time to create something more meaningful to begin with. I can’t say for sure what is the reward system behind LLM use in general, but given how much money people are willing to spend with models even in their current deeply flawed state, I’d say it’s clear that the time savings are outweighing the mistakes and shallowness. Take the comment paper, for example. Since Claude Opus is the first author, I’m assuming that the human author took a backseat and let the AI build the reasoning and most of the writing. Unsurprisingly, it is full of errors and contradictions, to a point where it looks like the human author didn’t bother too much to check what was being published. One might say that the human author, in trying to build some reputation by showing that their model could answer a scientific criticism, actually did the opposite: it provided more evidence that its model cannot reason deeply, and maybe hurt their reputation even more. But the real question is, did they really? How much backlash will they possibly get from submitting this to arxiv without checking? Would that backlash keep them from submitting 10 more papers next week with Claude as the first author? If one puts in a balance the amount of slop you can put out (with a slight benefit) vs. the bad reputation one gets from it, I cannot say that “human thinking” is actually worth it anymore.
- iLoveOncall 1y ago
  Mediocre people produce mediocre work. Using AI might make those mediocre people produce even worse work, but I don't think it'll affect competent people who have standards regardless of the available tooling. If anything the outcome will be good: mediocre people will produce even worse work and will weed themselves out. Cause in point: the author of the rebuttal made basic and obvious mistakes that make his work even easier to dismiss and no further paper of his will be considered seriously.
  - drsim 1y ago
    I think the pull will be hard to resist even for competent people. Like the obesity crisis driven by sugar highs, the overall population will be affected, and overall quality will suffer, at least for a while.
  - Arainach 1y ago
    >mediocre people will produce even worse work and will weed themselves out. [[Citation needed]] I don't believe anyone who has experienced working with other people - in the workspace, in school, whatever - believes that people get weeded out for mediocre output.
    - delusional 1y ago
      You can also be mediocre in a lot of different ways. Some people are mediocre thinkers, but fantastic hype men. Some people are fantastic at thinking, but suck at playing the political games you have to play in an office. Personally I find that I need some of all of those aspects to have success in a project, the amount varies by the work and external collaborators. Intelligence isn't just one measure you can have less or more of. I thought we figured this out 20 years ago.
    - Muromec 1y ago
      Weeded out to where anyway? Doing some silly thing, like being a cashier or taxi driver?
  - bananapub 1y ago
    > Mediocre people produce mediocre work. Using AI might make those mediocre people produce even worse work, but I don't think it'll affect competent people who have standards regardless of the available tooling. this is clearly not the case, given: - mass layoffs in the tech industry to force more use of such things - extremely strong pressure from management to use it, rarely framed as "please use this tooling as you see fit" - extremely low quality bars in all sorts of things, e.g. getting your dumb "We wrote a 200 word prompt then stuck that and some web scraped data in to an LLM run by Google/OpenAI/Anthropic" site to the top of hacker news, or most of VC funding in the tech world - extremely large swathes of (at least) the western power structures not giving a shit about doing anything well, e.g. the entire US Federal government leadership now, or the UK government's endless idiocy about "AI Policy development", lawyers getting caught in court having just not even read the documents they put their name on, etc - actual strong desire from many people to outsource their toxic plans to "AI", e.g. the US's machine learning probation or sentencing stuff I don't think any of us are ready for the tsunami of garbage that's going to be thrown in to every facet of our lives, from government policy to sending people to jail to murdering people with robots to spamming open source projects with useless code and bug reports etc etc etc
- practice9 1y ago
  The human is a bad co-author here really. I deployed lots of high performance, clean, well documented etc code generated by Claude or o3. I reviewed it wrt requirements, added tests and so on. Even with that in mind it allowed me to work 3x faster. But it required conscious effort on my part to point out issues and inefficiencies on LLMs part. It is a collaborative type of work where LLMs shine (even in so called agentic flows)
roenxi 1y ago
Has anyone come up with a definition of AGI where humans are near-universally capable of GI? These articles seem to be slowly pushing the boundaries past the point where slower humans are disbarred from intelligence. Many years ago I bumped in to Towers of Hanoi in a computer game and failed to solve it algorithmicly, so I suppose I'm lucky I only work a knowledge job rather than an intelligence-based one.
- parodysbird 1y ago
  The original Turing Test was one of the more interesting standards... An expert judge talks with two subjects in order to determine which is the human: one is a human who knows the point of the test, and one is machine trying to fool the judge into being no better than a coin flip at correctly choosing who was human. Allow for many judges and experience in each etc. The brilliance of the test, which was strangely lost on Turing, is that the test is doubtful to be passed with any enduring consistency. Intelligence is actually more of a social description. Solving puzzles, playing tricky games, etc is only intelligent if we agree that the actor involved faces normal human constraints or more. We don't actually think machines fulfill that (they obviously do not, that's why we build them: to overcome our own constraints), and so this is why calculating logarithms or playing chess ultimately do not end up counting as actual intelligence when a machine does them.
- James_K 1y ago
  It may genuinely be the case that slower humans are not generally intelligent. But that sounds rather snobbish so it's not an opinion I'd like to express frequently. I think the complaint made by apple is quite logical though and you mischaracterise it here. The question asked in the Apple study was "if I give you the algorithm that solves a puzzle, can you solve that puzzle?" The answer for most humans should be yes. Indeed, the answer is yes for computers which are not generally intelligent. Models failed to execute the algorithm. This suggests that the models are far inferior to the human mind in terms of their computational ability, which precedes general intelligence if you ask me. It seems to indicate that the models are using more of a "guess and check" approach than actually thinking. (A specifically interesting result was that model performance did not substantially improve between a puzzle with the solution algorithm given, and one where no algorithm was given.) You can sort of imagine the human mind as the head of a Turing Machine which operates on language tokens, and the goal of an LLM is to imitate the internal logic of that head. This paper seems to demonstrate that they are not very good at doing that. It makes a lot of sense when you think about it, because the models work by consuming their entire input at once where the human mind operates with only a small working memory. A fundamental architectural difference which I suspect is the cause of the collapse noted in the Apple paper.
amelius 1y ago
Is there anything falsifiable in Apple's paper?
- deleted 1y ago
  [deleted]
djoldman 1y ago
> I would consider this a death blow paper to the current push for using LLMs and LRMs as the basis for AGI. Anytime I see "Artificial General Intelligence," "AGI," "ASI," etc., I mentally replace it with "something no one has defined meaningfully." Or the long version: "something about which no conclusions can be drawn because the proposed definitions lack sufficient precision and completeness." Or the short versions: "Skippetyboop," "plipnikop," and "zingybang."
- coffeefirst 1y ago
  “The Messiah.” The believers know it’s coming and will transform the world in ways that don’t even make sense to outsiders. They do not change their mind when it doesn’t happen as foreseen.
- chrsw 1y ago
  One vague definition I see tossed around a lot "something can replace almost any human knowledge/white collar worker". What does that mean in concrete terms? I'm not sure. Many of these models can already pass bar exams but how many can be lawyers? Probably none. What's missing?
  - thaumasiotes 1y ago
    > Probably none. The qualification is unnecessary; we know the answer is "none". There's a steady stream of lawyers getting penalized for submitting LLM output to judges.
    - chrsw 1y ago
      You're right. I should have said "can ever". Both in terms of permitted to and in terms of have the capacity to. And I'm only referring to current machine learning architectures.
  - 542354234235 1y ago
    Tests designed for humans are not good at testing LLMs because the failure modes are different. Humans don’t have eidetic memories, so they can’t just ingest random facts and recall them at will. Memorizing the relevant facts and figures in a subject and recalling them on a test shows at least some sort of study and likely some sort of overall understanding of the subject. AI, not so much. A driving test that shows a red octagon with white letters and asks what it means, is a good indicator whether a driver will know that out in the real world, regardless of if it is half covered by a tree, has graffiti on it, is hanging upside down, etc. We found that self driving cars can’t “generalize” like that and failed to recognize a stop sign when it was upside down (and had to be retrained). What is missing is a well functions "judgement" or "reasoning" part that can generalize knowledge, experience, and context. Some models can do something like that for a specific task, but nothing that works long term.
Herring 1y ago
Apple's tune will completely change the second they get a leading LLM - Look at all the super important and useful things you can do with "Apple General Intelligence"!
- jmsdnns 1y ago
  No, it wont. This comment essentially says science doesnt matter for anyone, only whether or not they're leading in marketing.
  - jstanley 1y ago
    I think the comment says science doesn't matter for Apple, specifically.
  - Lerc 1y ago
    I think it's closer to saying that those who are falling behind declare that the science doesn't matter.
    - jmsdnns 1y ago
      one hasnt fallen behind when those in "front" are on the wrong path
      - Lerc 1y ago
        Perhaps, but that remains to be seen. On the other hand future events have no bearing on what Herring was saying, only the accuracy of it.
- throwaway287391 1y ago
  As someone who used to write academic ML papers, it's funny to me that people are treating this academic style paper written by a few Apple researchers as Apple's official company-wide stance, especially given the first author was an intern. I suppose it's "fair" since it's published on the Apple website with the authors' Apple affiliations, but historically speaking, at least in ML where publication is relatively fast-paced and low-overhead, academic papers by small teams of individual researchers have in no way reflected the opinions of e.g. the executives of a large company. I would not be particularly surprised to see another team of Apple researchers publishing a paper in the coming weeks with the opposite take, for example.
robertk 1y ago
The Apple paper does not look at its own data — the model outputs become short past some thresholds because the models reflectively realize they do not have the context to respond in the steps as requested, and suggest a Python program instead, just as a human would. One of the penalized environments is proven impossible to solve in the literature for n>6, seemingly unaware to the authors. I consider this and more the definitive rebuttal of the sloppiness of the paper: https://www.alignmentforum.org/posts/5uw26uDdFbFQgKzih/beware-general-claims-about-generalizable-reasoning https://www.alignmentforum.org/posts/5uw26uDdFbFQgKzih/bewar...
- suddenlybananas 1y ago
  [flagged]
pu_pe 1y ago
The author's main point is that output token constraints should not be the root cause for poor performance in reasoning tests, as in many cases the LLMs did not even come close to exceeding their token budgets before giving up. While that may be true, do we understand how LLMs behave according to token budget constraints? This might impact much simpler tasks as well. If we give them a task to list the names of all cities in the world according to population, do they spit out a python script if we give them a 4k output token budget but a full list if we give them 100k?
- vectorhacker 1y ago
  Makes me wonder if LLMs get tired. /s
gjm11 1y ago
This rebuttal-of-a-rebuttal looks to me as if it gets one (fairly important) thing right but pretty much everything else wrong. (Not all in the same direction; the rebuttal^2 fails to point out what seems to me to be the single biggest deficiency in the rebuttal.) The thing it gets right: the "Illusion of illusion" rebuttal claims that in the original "Illusion of Thinking" paper's version of the Towers of Hanoi problem, "The authors’ evaluation format requires outputting the full sequence of moves at each step, leading to quadratic token growth"; this doesn't seem to be true at all, and this "Beyond Token Limits" rebuttal^2 is correct to point it out. (This implies, in particular, that there's something fishy in the IoI rebuttal's little table showing where 5(2^n-1)^2 exceeds the token budget, which they claim explains the alleged "collapse" at roughly those points.) Things it gets wrong: "The rebuttal conflates solution length with computational difficulty". This is just flatly false. The IoI rebuttal explicitly makes pretty much the same points as the BTL rebuttal^2 does here. "The rebuttal paper’s own data contradicts its thesis. Its own data shows that models can generate long sequences when they choose to, but in the findings of the original Apple paper, it finds that models systematically choose NOT to generate longer reasoning traces on harder problems, effectively just giving up." I don't see anything in the rebuttal that "shows that models can generate long sequences when they choose to". What the rebuttal finds is that (specifically for the ToH problem) if you allow the models to answer by describing the procedure rather than enumerating all its steps, they can do it. The original paper didn't allow them to do this. There's no contradiction here. "It instead completely ignores this finding [that once solutions reach a certain level of difficulty the models give up trying to give complete answers] and offers no explanation as to why models would systematically reduce computational effort when faced with harder problems." The rebuttal doesn't completely ignore this finding. That little table of alleged ToH token counts is precisely targeted at this finding. (It seems like it's wrong, which is important, but the problem here isn't that the paper ignores this issue, it's that it has a mistake that invalidates how it addresses the issue.) Things that a good rebuttal^2 should point out but this rebuttal completely ignores: The most glaring one, to me, is that the rebuttal focuses almost entirely on the Tower of Hanoi, where there's a plausible "the only problem is that there aren't enough tokens" issue, and largely ignores the other problems that the original paper also claims to find "collapse" problems with. Maybe token-limit issues are also sufficient explanation for the problems with other models (e.g., if something is effectively only solvable by exhaustive search, then maybe there aren't enough tokens for the model to do that search in) but the rebuttal never actually makes that argument (e.g., by estimating how many tokens are needed to do the relevant exhaustive search). The rebuttal does point out what if correct is a serious problem with the original paper's treatment of the "River Crossing" problem (apparently the problem they asked the AI to solve is literally unsolvable for many of the cases they put to it), but the unsolvability starts at N=6 and the original paper finds that the models were unable to solve the problem starting at N=3. (Anecdata: I had a go at solving the River Crossing problem for N=3 myself. I made a stupid mistake that stopped me finding a solution and didn't have sufficient patience to track it down. My guess is that if you could spawn many independent copies of me and ask them all to solve it, probably about 2/3 would solve it and 1/3 would screw up in something like the way actual-me did. If I actually needed to solve it for larger N I'd write some code, which I suspect the AI models could do about as well as I could. For what it's worth, I think the amount of text-editor scribbling I did while not solving the puzzle was quite a bit less than the thinking-token limits these models had.) The rebuttal^2 does complain about the "narrow focus" of the rebuttal, but it means something else by that.
vectorhacker 1y ago
So it turns out the comment paper was a joke: https://lawsen.substack.com/p/when-your-joke-paper-goes-viral?triedRedirect=true https://lawsen.substack.com/p/when-your-joke-paper-goes-vira...

50 points by vectorhacker 1y ago

A fundamental problem that we’re still far away from solving is not necessarily that LLMs/LRMs cannot reason the same way that we do (which I guess should be clear by now); but that they might not have to. They generate slop so fast that, if one can benefit a little bit from each output, i.e. if you can find a little bit of use hidden beneath the mountain of meaningless text they’ll create, then this might still be more valuable than preemptively taking the time to create something more meaningful to begin with. I can’t say for sure what is the reward system behind LLM use in general, but given how much money people are willing to spend with models even in their current deeply flawed state, I’d say it’s clear that the time savings are outweighing the mistakes and shallowness. Take the comment paper, for example. Since Claude Opus is the first author, I’m assuming that the human author took a backseat and let the AI build the reasoning and most of the writing. Unsurprisingly, it is full of errors and contradictions, to a point where it looks like the human author didn’t bother too much to check what was being published. One might say that the human author, in trying to build some reputation by showing that their model could answer a scientific criticism, actually did the opposite: it provided more evidence that its model cannot reason deeply, and maybe hurt their reputation even more. But the real question is, did they really? How much backlash will they possibly get from submitting this to arxiv without checking? Would that backlash keep them from submitting 10 more papers next week with Claude as the first author? If one puts in a balance the amount of slop you can put out (with a slight benefit) vs. the bad reputation one gets from it, I cannot say that “human thinking” is actually worth it anymore.

iLoveOncall 1y ago
Mediocre people produce mediocre work. Using AI might make those mediocre people produce even worse work, but I don't think it'll affect competent people who have standards regardless of the available tooling. If anything the outcome will be good: mediocre people will produce even worse work and will weed themselves out. Cause in point: the author of the rebuttal made basic and obvious mistakes that make his work even easier to dismiss and no further paper of his will be considered seriously.
- drsim 1y ago
  I think the pull will be hard to resist even for competent people. Like the obesity crisis driven by sugar highs, the overall population will be affected, and overall quality will suffer, at least for a while.
- Arainach 1y ago
  >mediocre people will produce even worse work and will weed themselves out. [[Citation needed]] I don't believe anyone who has experienced working with other people - in the workspace, in school, whatever - believes that people get weeded out for mediocre output.
  - delusional 1y ago
    You can also be mediocre in a lot of different ways. Some people are mediocre thinkers, but fantastic hype men. Some people are fantastic at thinking, but suck at playing the political games you have to play in an office. Personally I find that I need some of all of those aspects to have success in a project, the amount varies by the work and external collaborators. Intelligence isn't just one measure you can have less or more of. I thought we figured this out 20 years ago.
  - Muromec 1y ago
    Weeded out to where anyway? Doing some silly thing, like being a cashier or taxi driver?
- bananapub 1y ago
  > Mediocre people produce mediocre work. Using AI might make those mediocre people produce even worse work, but I don't think it'll affect competent people who have standards regardless of the available tooling. this is clearly not the case, given: - mass layoffs in the tech industry to force more use of such things - extremely strong pressure from management to use it, rarely framed as "please use this tooling as you see fit" - extremely low quality bars in all sorts of things, e.g. getting your dumb "We wrote a 200 word prompt then stuck that and some web scraped data in to an LLM run by Google/OpenAI/Anthropic" site to the top of hacker news, or most of VC funding in the tech world - extremely large swathes of (at least) the western power structures not giving a shit about doing anything well, e.g. the entire US Federal government leadership now, or the UK government's endless idiocy about "AI Policy development", lawyers getting caught in court having just not even read the documents they put their name on, etc - actual strong desire from many people to outsource their toxic plans to "AI", e.g. the US's machine learning probation or sentencing stuff I don't think any of us are ready for the tsunami of garbage that's going to be thrown in to every facet of our lives, from government policy to sending people to jail to murdering people with robots to spamming open source projects with useless code and bug reports etc etc etc
practice9 1y ago
The human is a bad co-author here really. I deployed lots of high performance, clean, well documented etc code generated by Claude or o3. I reviewed it wrt requirements, added tests and so on. Even with that in mind it allowed me to work 3x faster. But it required conscious effort on my part to point out issues and inefficiencies on LLMs part. It is a collaborative type of work where LLMs shine (even in so called agentic flows)

roenxi 1y ago

Has anyone come up with a definition of AGI where humans are near-universally capable of GI? These articles seem to be slowly pushing the boundaries past the point where slower humans are disbarred from intelligence. Many years ago I bumped in to Towers of Hanoi in a computer game and failed to solve it algorithmicly, so I suppose I'm lucky I only work a knowledge job rather than an intelligence-based one.

parodysbird 1y ago
The original Turing Test was one of the more interesting standards... An expert judge talks with two subjects in order to determine which is the human: one is a human who knows the point of the test, and one is machine trying to fool the judge into being no better than a coin flip at correctly choosing who was human. Allow for many judges and experience in each etc. The brilliance of the test, which was strangely lost on Turing, is that the test is doubtful to be passed with any enduring consistency. Intelligence is actually more of a social description. Solving puzzles, playing tricky games, etc is only intelligent if we agree that the actor involved faces normal human constraints or more. We don't actually think machines fulfill that (they obviously do not, that's why we build them: to overcome our own constraints), and so this is why calculating logarithms or playing chess ultimately do not end up counting as actual intelligence when a machine does them.
James_K 1y ago
It may genuinely be the case that slower humans are not generally intelligent. But that sounds rather snobbish so it's not an opinion I'd like to express frequently. I think the complaint made by apple is quite logical though and you mischaracterise it here. The question asked in the Apple study was "if I give you the algorithm that solves a puzzle, can you solve that puzzle?" The answer for most humans should be yes. Indeed, the answer is yes for computers which are not generally intelligent. Models failed to execute the algorithm. This suggests that the models are far inferior to the human mind in terms of their computational ability, which precedes general intelligence if you ask me. It seems to indicate that the models are using more of a "guess and check" approach than actually thinking. (A specifically interesting result was that model performance did not substantially improve between a puzzle with the solution algorithm given, and one where no algorithm was given.) You can sort of imagine the human mind as the head of a Turing Machine which operates on language tokens, and the goal of an LLM is to imitate the internal logic of that head. This paper seems to demonstrate that they are not very good at doing that. It makes a lot of sense when you think about it, because the models work by consuming their entire input at once where the human mind operates with only a small working memory. A fundamental architectural difference which I suspect is the cause of the collapse noted in the Apple paper.

amelius 1y ago

Is there anything falsifiable in Apple's paper?

deleted 1y ago
[deleted]

djoldman 1y ago

> I would consider this a death blow paper to the current push for using LLMs and LRMs as the basis for AGI. Anytime I see "Artificial General Intelligence," "AGI," "ASI," etc., I mentally replace it with "something no one has defined meaningfully." Or the long version: "something about which no conclusions can be drawn because the proposed definitions lack sufficient precision and completeness." Or the short versions: "Skippetyboop," "plipnikop," and "zingybang."

coffeefirst 1y ago
“The Messiah.” The believers know it’s coming and will transform the world in ways that don’t even make sense to outsiders. They do not change their mind when it doesn’t happen as foreseen.
chrsw 1y ago
One vague definition I see tossed around a lot "something can replace almost any human knowledge/white collar worker". What does that mean in concrete terms? I'm not sure. Many of these models can already pass bar exams but how many can be lawyers? Probably none. What's missing?
- thaumasiotes 1y ago
  > Probably none. The qualification is unnecessary; we know the answer is "none". There's a steady stream of lawyers getting penalized for submitting LLM output to judges.
  - chrsw 1y ago
    You're right. I should have said "can ever". Both in terms of permitted to and in terms of have the capacity to. And I'm only referring to current machine learning architectures.
- 542354234235 1y ago
  Tests designed for humans are not good at testing LLMs because the failure modes are different. Humans don’t have eidetic memories, so they can’t just ingest random facts and recall them at will. Memorizing the relevant facts and figures in a subject and recalling them on a test shows at least some sort of study and likely some sort of overall understanding of the subject. AI, not so much. A driving test that shows a red octagon with white letters and asks what it means, is a good indicator whether a driver will know that out in the real world, regardless of if it is half covered by a tree, has graffiti on it, is hanging upside down, etc. We found that self driving cars can’t “generalize” like that and failed to recognize a stop sign when it was upside down (and had to be retrained). What is missing is a well functions "judgement" or "reasoning" part that can generalize knowledge, experience, and context. Some models can do something like that for a specific task, but nothing that works long term.

Herring 1y ago

Apple's tune will completely change the second they get a leading LLM - Look at all the super important and useful things you can do with "Apple General Intelligence"!

jmsdnns 1y ago
No, it wont. This comment essentially says science doesnt matter for anyone, only whether or not they're leading in marketing.
- jstanley 1y ago
  I think the comment says science doesn't matter for Apple, specifically.
- Lerc 1y ago
  I think it's closer to saying that those who are falling behind declare that the science doesn't matter.
  - jmsdnns 1y ago
    one hasnt fallen behind when those in "front" are on the wrong path
    - Lerc 1y ago
      Perhaps, but that remains to be seen. On the other hand future events have no bearing on what Herring was saying, only the accuracy of it.
throwaway287391 1y ago
As someone who used to write academic ML papers, it's funny to me that people are treating this academic style paper written by a few Apple researchers as Apple's official company-wide stance, especially given the first author was an intern. I suppose it's "fair" since it's published on the Apple website with the authors' Apple affiliations, but historically speaking, at least in ML where publication is relatively fast-paced and low-overhead, academic papers by small teams of individual researchers have in no way reflected the opinions of e.g. the executives of a large company. I would not be particularly surprised to see another team of Apple researchers publishing a paper in the coming weeks with the opposite take, for example.

robertk 1y ago

The Apple paper does not look at its own data — the model outputs become short past some thresholds because the models reflectively realize they do not have the context to respond in the steps as requested, and suggest a Python program instead, just as a human would. One of the penalized environments is proven impossible to solve in the literature for n>6, seemingly unaware to the authors. I consider this and more the definitive rebuttal of the sloppiness of the paper: https://www.alignmentforum.org/posts/5uw26uDdFbFQgKzih/beware-general-claims-about-generalizable-reasoning https://www.alignmentforum.org/posts/5uw26uDdFbFQgKzih/bewar...

suddenlybananas 1y ago
[flagged]

pu_pe 1y ago

The author's main point is that output token constraints should not be the root cause for poor performance in reasoning tests, as in many cases the LLMs did not even come close to exceeding their token budgets before giving up. While that may be true, do we understand how LLMs behave according to token budget constraints? This might impact much simpler tasks as well. If we give them a task to list the names of all cities in the world according to population, do they spit out a python script if we give them a 4k output token budget but a full list if we give them 100k?

vectorhacker 1y ago
Makes me wonder if LLMs get tired. /s

gjm11 1y ago

This rebuttal-of-a-rebuttal looks to me as if it gets one (fairly important) thing right but pretty much everything else wrong. (Not all in the same direction; the rebuttal^2 fails to point out what seems to me to be the single biggest deficiency in the rebuttal.) The thing it gets right: the "Illusion of illusion" rebuttal claims that in the original "Illusion of Thinking" paper's version of the Towers of Hanoi problem, "The authors’ evaluation format requires outputting the full sequence of moves at each step, leading to quadratic token growth"; this doesn't seem to be true at all, and this "Beyond Token Limits" rebuttal^2 is correct to point it out. (This implies, in particular, that there's something fishy in the IoI rebuttal's little table showing where 5(2^n-1)^2 exceeds the token budget, which they claim explains the alleged "collapse" at roughly those points.) Things it gets wrong: "The rebuttal conflates solution length with computational difficulty". This is just flatly false. The IoI rebuttal explicitly makes pretty much the same points as the BTL rebuttal^2 does here. "The rebuttal paper’s own data contradicts its thesis. Its own data shows that models can generate long sequences when they choose to, but in the findings of the original Apple paper, it finds that models systematically choose NOT to generate longer reasoning traces on harder problems, effectively just giving up." I don't see anything in the rebuttal that "shows that models can generate long sequences when they choose to". What the rebuttal finds is that (specifically for the ToH problem) if you allow the models to answer by describing the procedure rather than enumerating all its steps, they can do it. The original paper didn't allow them to do this. There's no contradiction here. "It instead completely ignores this finding [that once solutions reach a certain level of difficulty the models give up trying to give complete answers] and offers no explanation as to why models would systematically reduce computational effort when faced with harder problems." The rebuttal doesn't completely ignore this finding. That little table of alleged ToH token counts is precisely targeted at this finding. (It seems like it's wrong, which is important, but the problem here isn't that the paper ignores this issue, it's that it has a mistake that invalidates how it addresses the issue.) Things that a good rebuttal^2 should point out but this rebuttal completely ignores: The most glaring one, to me, is that the rebuttal focuses almost entirely on the Tower of Hanoi, where there's a plausible "the only problem is that there aren't enough tokens" issue, and largely ignores the other problems that the original paper also claims to find "collapse" problems with. Maybe token-limit issues are also sufficient explanation for the problems with other models (e.g., if something is effectively only solvable by exhaustive search, then maybe there aren't enough tokens for the model to do that search in) but the rebuttal never actually makes that argument (e.g., by estimating how many tokens are needed to do the relevant exhaustive search). The rebuttal does point out what if correct is a serious problem with the original paper's treatment of the "River Crossing" problem (apparently the problem they asked the AI to solve is literally unsolvable for many of the cases they put to it), but the unsolvability starts at N=6 and the original paper finds that the models were unable to solve the problem starting at N=3. (Anecdata: I had a go at solving the River Crossing problem for N=3 myself. I made a stupid mistake that stopped me finding a solution and didn't have sufficient patience to track it down. My guess is that if you could spawn many independent copies of me and ask them all to solve it, probably about 2/3 would solve it and 1/3 would screw up in something like the way actual-me did. If I actually needed to solve it for larger N I'd write some code, which I suspect the AI models could do about as well as I could. For what it's worth, I think the amount of text-editor scribbling I did while not solving the puzzle was quite a bit less than the thinking-token limits these models had.) The rebuttal^2 does complain about the "narrow focus" of the rebuttal, but it means something else by that.

vectorhacker 1y ago

So it turns out the comment paper was a joke: https://lawsen.substack.com/p/when-your-joke-paper-goes-viral?triedRedirect=true https://lawsen.substack.com/p/when-your-joke-paper-goes-vira...