17 ms·

Model intelligence is no longer the constraint for automation

113 points by drivian 1y ago

neom 1y ago
Same same human problems. Regardless of their inherent intelligence...humans perform well only when given decent context and clear specifications/data. If you place a brilliant executive into a scenario without meaningful context.... an unfamiliar board meeting where they have no idea of the company’s history, prior strategic discussions, current issues, personel dynamics...expectations..etc etc, they will struggle just as a model does surly. They may still manage something reasonably insightful, leveraging general priors, common sense, and inferential reasoning... their performance will never match their potential had they been fully informed of all context and clearly data/objectives. I think context is the primary primitive property of intelligent systems in general?
- mrlongroots 1y ago
  > they will struggle just as a model does surly A human will struggle, but they will recognize the things they need to know, and seek out people who may have the relevant information. If asked "how are things going" they will reliably be able to say "badly, I don't have anything I need".
  - KoolKat23 1y ago
    That's just additional context.
    - Jensson 1y ago
      That the person go and get themselves. If a model could to that we wouldn't need you to drive them. Basically every human is self going that way, you don't need to go and pick them up since they got stuck in a loop of unknowns at a grocery store etc.
      - KoolKat23 1y ago
        Yes, like search or operator. It's at the early stages but they are doing this. Currently counting in minutes rather than hours or days.
- getnormality 1y ago
  This comparison may make sense on short-horizon tasks for which there is no possibility of preparation. Given some weeks to prepare, a good human executive will get the context, while today's best AI systems will completely fail to do so.
  - crazylogger 1y ago
    Today’s AI systems probably won’t excel, but they won’t completely fail either. Basically give the LLM a computer to do all kinds of stuff against the real world, kick it off with a high level goal like “build a startup”. The key is to instruct it to manage its own memory in its computer, and when context limit inevitably approaches, programmatically interrupt the LLM loop and instruct it to jot down everything it has for its future self. It already kinda works today, and I believe AI systems a year from now will excel at this: https://dwyer.co.za/static/claude-code-is-all-you-need.html https://dwyer.co.za/static/claude-code-is-all-you-need.html https://www.anthropic.com/research/project-vend-1 https://www.anthropic.com/research/project-vend-1
- simoncion 1y ago
  > I think context is the primary primitive property of intelligent systems in general? What do you mean by 'context' in this context? As written, I believe that I could knock down your claim by pointing out that there exist humans who would do catastrophically poorly at a task that other humans would excel at, even if both humans have been fully informed of all of the same context.
  - simoncion 1y ago
    To clarify what I'm thinking here by analogy... Imagine that someone said: > I think wood is the primary primitive property of sawmills in general. An obvious observation would be that it is dreadfully difficult to produce the expected product of a sawmill without tools to cut or sand or otherwise shape the wood into the desired shapes. One might also notice that while a sawmill with no wood to work on will not produce any output, a sawmill with wood but without woodworking tools is vanishingly unlikely to produce any output... and any it does manage to produce is not going to be good enough for any real industrial purpose.
    - neom 1y ago
      My perspective ("context as primary primitive") was about context as the foundational prerequisite of intelligent performance. I'm discussing a scenario with the minimum conditions for any intelligent action, whether small scale or large scale. At risk of talking past each other due to nuance methinks and I'm a bit lazy to think it through properly but... I think there is something in saw vs sawmill? Like a scale thing? Either way I wasn't trying to be profound or anything, I was just saying I think context abilities is likely the first prerequisite for any minimally intelligent thing (maybe I shouldn't have used the word system in my original comment).
- whattheheckheck 1y ago
  An intelligent system would know how to get that information without getting spoon fed it
- Greenpants 1y ago
  I really like this analogy! Many real-world tasks that we'd like to use AI for seem infinitely more complex than can be captured in a simple question/prompt. The main challenge going forward, in my opinion, is how to let LLMs ask the right questions – query for the right information – given a task to perform. Tool use with MCPs might be a good start, though it still feels hacky to have to define custom tools for LLMs first, as opposed to how humans effectively browse and skim lots of documentation to find actually relevant bits.
threecheese 1y ago
Author IMO correctly recognizes that access to context needs to scale (“latent intent” which I love), but I’m not sure I’m convinced that current models will be effective even if given access to all priors needed for a complex task. The ability to discriminate valuable from extraneous context will need to scale with size of available context, it will be pulling needles from haystacks that aren’t straightforward similarity. I think we will need to steer these things.
- jondwillis 1y ago
  We’re already steering, during pre-training (e.g. reasoning RLHF), as well as test-time (structured outputs, tool calls, agents…)
jefftitan 1y ago
Providing more context is difficult for a number of reasons. If you do it RAG style you need to know which context is relevant. LLMs are notorious for knowing that a factor is relevant if directly asked about that factor, but not bringing it up if it's implicit. In business things like people's feelings on things, historical business dealings, relevance to trending news can all be factors. If you fine tune... well... there have been articles recently about fine tuning on specific domains causing overall misalignment. The more you fine tune, the riskier.
miller24 1y ago
It 100% is still intelligence. GPT-5 with Thinking still can't win at tic-tac-toe.
- dismalaf 1y ago
  Tic-tac-toe is solved and a draw can be forced 100% of the time...
  - HappMacDonald 1y ago
    .. and you can "program" a neural network — so simple it can be implemented by boxes full of marbles and simple rules about how to interact with the boxes — to learn by playing tictactoe until it always plays perfect games. This is frequently chosen as a lesson in how neural network training even works. But I have a different challenge for you: train a human to play tictactoe, but never allow them to see the game visually, even in examples. You have to train them to play only by spoken words. Point being that tictactoe is a visual game and when you're only teaching a model to learn from the vast sea of stream-of-tokens (similar to stream-of-phonemes) language, visual games like this aren't going to be well covered in the training set, nor is it going to be easy to generalize to playing them.
    - miller24 1y ago
      Well whatever your story is, I know with near certainty that no amount of scaffolding is going to get you from an LLM that can't figure out tic-tac-toe (but will confidently make bad moves) to something that can replace a human in an economically important job.
    - bwfan123 1y ago
      llm maximalists' apologies: - but tokens are not letters - but humans fail too - just wait, we are on an S curve to AGI - but your prompt was incorrect - but I tried and here it works Meanwhile, their claims: - LLMs are performing at PhD levels. - AGI is around the corner - humanity will be wiped out - situational awareness report
    - 1y ago
Kuinox 1y ago
It's specific model that run for maths. GPT-5 and Gemini 2.5 still cannot compute an arbitrary length sum of whole number without a calculator. I have a proceduraly generated benchmark of basic operations, LLMs gets better at it with time, but they cant still solve basic maths or logic problems. BTW I'm open to selling it, my email is on my hn profile.
- bt1a 1y ago
  i'd wager your benchmark problems require cumbersome arithmetic or are poorly worded / inadequately described. or, you're mislabeling them as basic math and logic (a domain within which LLMs have proven their strengths!) i only call this out because you're selling it and don't hypothesize* on why they fail your simple problems. i suppose an easily aced bench wouldn't be very marketable
  - Kuinox 1y ago
    This is a simple sum of 2 whole number, the number are simply big. Most of the time they make a correct summation table but fail to copy correctly the sum result into a final result. That is not a tokenisation problem (you can change the output format to make sure of it). I have a separated benchmark that test specifically this, when the input is too large, the LLMs fails to accuratly copy the correct token. I suppose the positional embedding, are not perfectly learned and it sometimes cause a mistake. The prompt is quite short, it use structured output, and I can generate a nice graph of % of good response accross difficulity of the question (which is just the total digit count of the input numbers. LLMs have 100% success rate on theses sum until they reach a frontier, past that their accuracy collapse at various speed depending of the model.
    - energy123 1y ago
      Have you tried greedy decoding (temp 0) in aistudio? The temp 0.7-1.0 defaults are not designed for reconstructing context with perfect accuracy.
      - 1y ago
thorum 1y ago
This article is insightful, but I blinked when I saw the headline “Reducing the human bottleneck” used without any apparent irony. At some point we should probably take a step back and ask “Why do we want to solve this problem?” Is a world where AI systems are highly intelligent tools, but humans are needed to manage the high level complexity of the real world… supposed to be a disappointing outcome?
- fnordpiglet 1y ago
  Assuming you buy the idea of a post scarcity society and assuming we can separate our long ingrained notion that spending your existence in toil to survive is a moral imperative and not working is deserving of punishment if not death, I personally look forward to a time we can get off the hamster wheel. Most buttons that get pushed by people are buttons not worth spending your existence pushing. This includes an awful lot of “knowledge work,” which is often better paid but more insidious in that it requires not just your presence but capturing your entire attention and mind inside and outside work. I would also be hopeful that fertility rates would decline and there would simply be far fewer humans. In Asimov’s robots stories the spacers are long lived and low population because robots do most everything. He presents this as a dead end, that stops us from conquering the galaxy. This to me sounds like a feature not a bug. I think human existence could be quite good with large scale automation, fewer people, and less suffering due to the necessity for everyone to be employed. Note I recognize you’re not saying exactly the same thing as I’m saying. I think humans will never cede full executive control by choice at some level. But I suspect, sadly, power will be confined to those few who do get to manage the high level complexity of the real world.
  - nradov 1y ago
    We will never have a post scarcity society. Automation can make certain foodstuffs and manufactured goods somewhat cheaper but the things that people really want will always be in short supply, for example real estate in geographically favorable areas.
    - lll-o-lll 1y ago
etler 1y ago
I think the framing of these models are being "intelligent" is not the right way to go. They've gotten better at recall and association. They can recall prior reasoning from text they are trained on which allows them to handle complex tasks that have been solved before, but when working on complex, novel, or nuanced tasks there is no high quality relevant training data to recall. Intelligence has always been a fraught word to define and I don't think what LLMs do is the right attribute for defining it. I agree with a good deal of the article but because it keeps using loaded works like "intelligent" and "smarter", it has a hard time explaining what's missing.
mrlongroots 1y ago
I very much disagree. To attempt a proof by contradiction: Let us assume that the author's premise is correct, and LLMs are plenty powerful given the right context. Can an LLM recognize the context deficit and frame the right questions to ask? They can not: LLMs have no ability to understand when to stop and ask for directions. They routinely produce contradictions, fail simple tasks like counting the letters in a word etc. etc. They can not even reliably execute my "ok modify this text in canvas" vs "leave canvas alone, provide suggestions in chat, apply an edit once approved" instructions.
- beering 1y ago
  It feels crazy to keep arguing about LLMs being able to do this or that, but not mention the specific model? The post author only mentions the IMO gold-medal model. And your post could be about anything. Am I to believe that the two of you are talking about the same thing? This discussion is not useful if that’s not the case.
- bobbylarrybobby 1y ago
  Claude routinely stops and asks me clarifying questions before continuing, especially when the given extended thinking or doing research.
  - diggan 1y ago
    Indeed, the ability to do so seems to depend more on how well your system prompt is laying out that workflow, than how "intelligent" the model is.
    - Jensson 1y ago
      Prompting it to ask clarifying questions will make it ask questions it has seen before, not ask questions it needs you to clarify. So that doesn't solve the problem, it just causes other problems. If it actually did solve the problem then they would train the models to act that way by default, so anything that you need to make smart prompts for has to be dumb.
    - 1y ago
stephc_int13 1y ago
This is because we tend to use a human-centric reference to evaluate the difficulty of a task : playing chess at grand master level is a lot harder than folding laundry, except that it is the opposite, and this weird bias is well known as Moravec’s Paradox. Intelligence is the bottleneck, but not the kind of intelligence you need to solve puzzles.
- mdaniel 1y ago
  For others who also hadn't heard of that: https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox
visarga 1y ago
Verification is the bottleneck, not ideation. LLMs can generate anything on tap, but solving any non-trivial problem requires iteration between thinking, doing and observing outcomes. The real world is too complex to be simulated by AI or humans. The scientific method works the same way, we are not exempt from having to validate our ideas. But as humans we have better feedback and access to context and we can assume risks on our own. AI has no skin and bears no responsibility. So the missing ingredient for AI is access to environment for feedback learning. It has little to do with AI architecture or datasets. I think a huge source of such data is our human-LLM chat logs. We act as LLM eyes, hands and feed on the ground. We carry the tacit knowledge and social context. OpenAI reports billions of tasks per day, probably trillions of tokens of interactive language combining human, AI and feedback from the environment. Maybe this is how AI can inch towards learning how to solve real world problems, it is part of the loop of problem solving, and benefits from having this data for training.
- bwfan123 1y ago
  > Verification is the bottleneck In my use of cursor as a coding assistant, this is the primary problem. The code is 90% on the mark, but still buggy, and needs verification, and the feedback it gets from me is not with full fidelity as something is lost in translation. But, a bigger issue is that AI has only some solution templates for problems that it is trained on, and being able to generate new templates is beyond its capability as that requires training on datasets of higher levels of abstration.
ankit219 1y ago
The bottleneck for automation is verification. With human work, verification was fast(er) because you know where to look with certain assumptions that your upstream tasker would not have made trivial mistakes. For automation, AI needs to verify it's own work, review, and self correct to be able to automate any given work. Where this works, it will also change the abstraction layer compared to what it is today. The problem is same with every automation promise - it needs to work reliably at say 95% or 99% times and when it doesn't, there should be human contingency in terms of what to look for. Considering coding as the first example: it's already underway. AI generates the code, the test cases, and then verifies if the code works as intended. Code has a built in verification layer (both compiler and unit tests). High probablity the other domains move towards something similar too. I would also say the model needs to be intelligent to course correct when the output isn't validated[1]. Verification solves the human in the loop dependency both for AI and human tasks. All the places where we could automate in the past, there were clearly quality checks which ensured the machinery were working as expected. Same thing will be replicated with AI too. Disclaimer: I have been working on building a universal verifier for AI tasks. The way it works is you give it a set of rules (policy) + AI output (could be human output too) and it outputs a scalar score + clause level citations. So I have been thinking about the problem space and might be over rating this. Would welcome contrarian ideas. (no, it's not llm as a judge) [1]: Some people may call it environment based learning, but in ML terms i feel it's different. That woudl be another example of sv startups using technical terms to market themselves when they dont do what they say.
- saint_yossarian 1y ago
  One thing that comes to mind: You still have to verify that the tests are exhaustive, and that the code isn't just gaming specific test scenarios. I guess fuzzing and property-based testing could mitigate this to some extent.
  - ankit219 1y ago
    Yes, we are getting there. I think compiler is a bigger problem than unit tests given most verticals don't even have that. With unit tests, there would be some reward hacking but would be controlled at the model level + tests. (this is one of the reason i dont believe in transformer based llm as a judge for a verifier)
fmbb 1y ago
> It’s because the bottleneck isn’t in intelligence, but in human tasks: specifying intent and context engineering. So the bottleneck is intelligence. Junior engineers are intelligent enough to understand when they don't understand. They interrogate the intent and context of the tasks they are given. This is intelligence. Solving math questions is not intelligence, computers have been better than humans at that for like 100 years, as long as you first do the intelligent part as a human: specifying the task formally. Now we just have computer programs with another kind of input in natural language, and which require dozens of gigabytes of video ram and millions of cores to execute. And we still have to have humans to the intelligent part, figure out how to describe the problem so the dumb but very very fast machine can answer the question.
- pvtmert 1y ago
  I truly love this comment, which essentially says: LLMs are glorified calculators, with ambiguous grammar. :)
  - handzhiev 1y ago
    Computers are glorified calculators, yet they power most of our lives
    - giantrobot 1y ago
      Many computers and interfaces are deterministic. LLMs are by nature not deterministic and not even non-deterministic the same way on any two invocations given the same prompt and context. Natural language is ambiguous and for many languages very context dependent. It's not the greatest interface for a calculator from which we're expecting deterministic accurate answers. WolframAlpha is a more impressive front end to a calculator than I've seen out of LLMs. Not only does it show me how it translated my natural-ish language query but it shows me potential alternative interpretations to my question. LLMs by the nature of how training works can't necessarily tell me why and how they interpreted my prompt. The thinking models are better but still not great.
  - bamboozled 1y ago
fifteen1506 1y ago
As a human, I'd also appreciate the specifications, documentation and meetings were not inaccessible to me.
graycat 1y ago
For the square covering problem, answer, 2.
Paratoner 1y ago
I'm endlessly fascinated by the way these Humans (probably) speak of their follow peers as though they are a problem to solve for. > Longer term, we can reduce the human bottleneck by Thank God we have ways to remove the thorn in our(?) side for good. The world can finally heal when the pursuit of fulfillment becomes inaccessible to the masses.
darepublic 1y ago
Model capability is absolutely the main constraint.
dash2 1y ago
I'm not sure about the assumption that science is context-free. Maths maybe, but a lot of practical science has tons of unformalized contextual knowledge that is "handed down" by practitioners. It's one reason why replication can be so hard. OTOH, I also think a lot of science is like 1% inspiration, 99% very mundane tasks like data cleaning. So no reason the AI can't help with that. And scientists write terrible code, so the bar is low :-)
aledalgrande 1y ago
I don't agree with the author. Where is the part about nondeterminism and hallucinations? Drawing a pretty chart doesn't make the argument true. All these benchmarks and competitions are on problems that have a _right answer_. I write most my code entirely through Claude at work and have Claude Max for personal, and I can see every day that even with the right context, it's not certain that the model is going to converge to a decent answer on complex real life issues. At least one thing I do agree on: model growth is not an exponential, like everyone thought when we were on the first leg of it, but a logarithmic.

113 points by drivian 1y ago

neom 1y ago

Same same human problems. Regardless of their inherent intelligence...humans perform well only when given decent context and clear specifications/data. If you place a brilliant executive into a scenario without meaningful context.... an unfamiliar board meeting where they have no idea of the company’s history, prior strategic discussions, current issues, personel dynamics...expectations..etc etc, they will struggle just as a model does surly. They may still manage something reasonably insightful, leveraging general priors, common sense, and inferential reasoning... their performance will never match their potential had they been fully informed of all context and clearly data/objectives. I think context is the primary primitive property of intelligent systems in general?

mrlongroots 1y ago
> they will struggle just as a model does surly A human will struggle, but they will recognize the things they need to know, and seek out people who may have the relevant information. If asked "how are things going" they will reliably be able to say "badly, I don't have anything I need".
- KoolKat23 1y ago
  That's just additional context.
  - Jensson 1y ago
    That the person go and get themselves. If a model could to that we wouldn't need you to drive them. Basically every human is self going that way, you don't need to go and pick them up since they got stuck in a loop of unknowns at a grocery store etc.
    - KoolKat23 1y ago
      Yes, like search or operator. It's at the early stages but they are doing this. Currently counting in minutes rather than hours or days.
getnormality 1y ago
This comparison may make sense on short-horizon tasks for which there is no possibility of preparation. Given some weeks to prepare, a good human executive will get the context, while today's best AI systems will completely fail to do so.
- crazylogger 1y ago
  Today’s AI systems probably won’t excel, but they won’t completely fail either. Basically give the LLM a computer to do all kinds of stuff against the real world, kick it off with a high level goal like “build a startup”. The key is to instruct it to manage its own memory in its computer, and when context limit inevitably approaches, programmatically interrupt the LLM loop and instruct it to jot down everything it has for its future self. It already kinda works today, and I believe AI systems a year from now will excel at this: https://dwyer.co.za/static/claude-code-is-all-you-need.html https://dwyer.co.za/static/claude-code-is-all-you-need.html https://www.anthropic.com/research/project-vend-1 https://www.anthropic.com/research/project-vend-1
simoncion 1y ago
> I think context is the primary primitive property of intelligent systems in general? What do you mean by 'context' in this context? As written, I believe that I could knock down your claim by pointing out that there exist humans who would do catastrophically poorly at a task that other humans would excel at, even if both humans have been fully informed of all of the same context.
- simoncion 1y ago
  To clarify what I'm thinking here by analogy... Imagine that someone said: > I think wood is the primary primitive property of sawmills in general. An obvious observation would be that it is dreadfully difficult to produce the expected product of a sawmill without tools to cut or sand or otherwise shape the wood into the desired shapes. One might also notice that while a sawmill with no wood to work on will not produce any output, a sawmill with wood but without woodworking tools is vanishingly unlikely to produce any output... and any it does manage to produce is not going to be good enough for any real industrial purpose.
  - neom 1y ago
    My perspective ("context as primary primitive") was about context as the foundational prerequisite of intelligent performance. I'm discussing a scenario with the minimum conditions for any intelligent action, whether small scale or large scale. At risk of talking past each other due to nuance methinks and I'm a bit lazy to think it through properly but... I think there is something in saw vs sawmill? Like a scale thing? Either way I wasn't trying to be profound or anything, I was just saying I think context abilities is likely the first prerequisite for any minimally intelligent thing (maybe I shouldn't have used the word system in my original comment).
whattheheckheck 1y ago
An intelligent system would know how to get that information without getting spoon fed it
Greenpants 1y ago
I really like this analogy! Many real-world tasks that we'd like to use AI for seem infinitely more complex than can be captured in a simple question/prompt. The main challenge going forward, in my opinion, is how to let LLMs ask the right questions – query for the right information – given a task to perform. Tool use with MCPs might be a good start, though it still feels hacky to have to define custom tools for LLMs first, as opposed to how humans effectively browse and skim lots of documentation to find actually relevant bits.

threecheese 1y ago

Author IMO correctly recognizes that access to context needs to scale (“latent intent” which I love), but I’m not sure I’m convinced that current models will be effective even if given access to all priors needed for a complex task. The ability to discriminate valuable from extraneous context will need to scale with size of available context, it will be pulling needles from haystacks that aren’t straightforward similarity. I think we will need to steer these things.

jondwillis 1y ago
We’re already steering, during pre-training (e.g. reasoning RLHF), as well as test-time (structured outputs, tool calls, agents…)

jefftitan 1y ago

Providing more context is difficult for a number of reasons. If you do it RAG style you need to know which context is relevant. LLMs are notorious for knowing that a factor is relevant if directly asked about that factor, but not bringing it up if it's implicit. In business things like people's feelings on things, historical business dealings, relevance to trending news can all be factors. If you fine tune... well... there have been articles recently about fine tuning on specific domains causing overall misalignment. The more you fine tune, the riskier.

miller24 1y ago

It 100% is still intelligence. GPT-5 with Thinking still can't win at tic-tac-toe.

dismalaf 1y ago
Tic-tac-toe is solved and a draw can be forced 100% of the time...
- HappMacDonald 1y ago
  .. and you can "program" a neural network — so simple it can be implemented by boxes full of marbles and simple rules about how to interact with the boxes — to learn by playing tictactoe until it always plays perfect games. This is frequently chosen as a lesson in how neural network training even works. But I have a different challenge for you: train a human to play tictactoe, but never allow them to see the game visually, even in examples. You have to train them to play only by spoken words. Point being that tictactoe is a visual game and when you're only teaching a model to learn from the vast sea of stream-of-tokens (similar to stream-of-phonemes) language, visual games like this aren't going to be well covered in the training set, nor is it going to be easy to generalize to playing them.
  - miller24 1y ago
    Well whatever your story is, I know with near certainty that no amount of scaffolding is going to get you from an LLM that can't figure out tic-tac-toe (but will confidently make bad moves) to something that can replace a human in an economically important job.
  - bwfan123 1y ago
    llm maximalists' apologies: - but tokens are not letters - but humans fail too - just wait, we are on an S curve to AGI - but your prompt was incorrect - but I tried and here it works Meanwhile, their claims: - LLMs are performing at PhD levels. - AGI is around the corner - humanity will be wiped out - situational awareness report
  - 1y ago

Kuinox 1y ago

It's specific model that run for maths. GPT-5 and Gemini 2.5 still cannot compute an arbitrary length sum of whole number without a calculator. I have a proceduraly generated benchmark of basic operations, LLMs gets better at it with time, but they cant still solve basic maths or logic problems. BTW I'm open to selling it, my email is on my hn profile.

bt1a 1y ago
i'd wager your benchmark problems require cumbersome arithmetic or are poorly worded / inadequately described. or, you're mislabeling them as basic math and logic (a domain within which LLMs have proven their strengths!) i only call this out because you're selling it and don't hypothesize* on why they fail your simple problems. i suppose an easily aced bench wouldn't be very marketable
- Kuinox 1y ago
  This is a simple sum of 2 whole number, the number are simply big. Most of the time they make a correct summation table but fail to copy correctly the sum result into a final result. That is not a tokenisation problem (you can change the output format to make sure of it). I have a separated benchmark that test specifically this, when the input is too large, the LLMs fails to accuratly copy the correct token. I suppose the positional embedding, are not perfectly learned and it sometimes cause a mistake. The prompt is quite short, it use structured output, and I can generate a nice graph of % of good response accross difficulity of the question (which is just the total digit count of the input numbers. LLMs have 100% success rate on theses sum until they reach a frontier, past that their accuracy collapse at various speed depending of the model.
  - energy123 1y ago
    Have you tried greedy decoding (temp 0) in aistudio? The temp 0.7-1.0 defaults are not designed for reconstructing context with perfect accuracy.
    - 1y ago

thorum 1y ago

This article is insightful, but I blinked when I saw the headline “Reducing the human bottleneck” used without any apparent irony. At some point we should probably take a step back and ask “Why do we want to solve this problem?” Is a world where AI systems are highly intelligent tools, but humans are needed to manage the high level complexity of the real world… supposed to be a disappointing outcome?

fnordpiglet 1y ago
Assuming you buy the idea of a post scarcity society and assuming we can separate our long ingrained notion that spending your existence in toil to survive is a moral imperative and not working is deserving of punishment if not death, I personally look forward to a time we can get off the hamster wheel. Most buttons that get pushed by people are buttons not worth spending your existence pushing. This includes an awful lot of “knowledge work,” which is often better paid but more insidious in that it requires not just your presence but capturing your entire attention and mind inside and outside work. I would also be hopeful that fertility rates would decline and there would simply be far fewer humans. In Asimov’s robots stories the spacers are long lived and low population because robots do most everything. He presents this as a dead end, that stops us from conquering the galaxy. This to me sounds like a feature not a bug. I think human existence could be quite good with large scale automation, fewer people, and less suffering due to the necessity for everyone to be employed. Note I recognize you’re not saying exactly the same thing as I’m saying. I think humans will never cede full executive control by choice at some level. But I suspect, sadly, power will be confined to those few who do get to manage the high level complexity of the real world.
- nradov 1y ago
  We will never have a post scarcity society. Automation can make certain foodstuffs and manufactured goods somewhat cheaper but the things that people really want will always be in short supply, for example real estate in geographically favorable areas.
  - lll-o-lll 1y ago

etler 1y ago

I think the framing of these models are being "intelligent" is not the right way to go. They've gotten better at recall and association. They can recall prior reasoning from text they are trained on which allows them to handle complex tasks that have been solved before, but when working on complex, novel, or nuanced tasks there is no high quality relevant training data to recall. Intelligence has always been a fraught word to define and I don't think what LLMs do is the right attribute for defining it. I agree with a good deal of the article but because it keeps using loaded works like "intelligent" and "smarter", it has a hard time explaining what's missing.

mrlongroots 1y ago

I very much disagree. To attempt a proof by contradiction: Let us assume that the author's premise is correct, and LLMs are plenty powerful given the right context. Can an LLM recognize the context deficit and frame the right questions to ask? They can not: LLMs have no ability to understand when to stop and ask for directions. They routinely produce contradictions, fail simple tasks like counting the letters in a word etc. etc. They can not even reliably execute my "ok modify this text in canvas" vs "leave canvas alone, provide suggestions in chat, apply an edit once approved" instructions.

beering 1y ago
It feels crazy to keep arguing about LLMs being able to do this or that, but not mention the specific model? The post author only mentions the IMO gold-medal model. And your post could be about anything. Am I to believe that the two of you are talking about the same thing? This discussion is not useful if that’s not the case.
bobbylarrybobby 1y ago
Claude routinely stops and asks me clarifying questions before continuing, especially when the given extended thinking or doing research.
- diggan 1y ago
  Indeed, the ability to do so seems to depend more on how well your system prompt is laying out that workflow, than how "intelligent" the model is.
  - Jensson 1y ago
    Prompting it to ask clarifying questions will make it ask questions it has seen before, not ask questions it needs you to clarify. So that doesn't solve the problem, it just causes other problems. If it actually did solve the problem then they would train the models to act that way by default, so anything that you need to make smart prompts for has to be dumb.
  - 1y ago

stephc_int13 1y ago

This is because we tend to use a human-centric reference to evaluate the difficulty of a task : playing chess at grand master level is a lot harder than folding laundry, except that it is the opposite, and this weird bias is well known as Moravec’s Paradox. Intelligence is the bottleneck, but not the kind of intelligence you need to solve puzzles.

mdaniel 1y ago
For others who also hadn't heard of that: https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox

visarga 1y ago

Verification is the bottleneck, not ideation. LLMs can generate anything on tap, but solving any non-trivial problem requires iteration between thinking, doing and observing outcomes. The real world is too complex to be simulated by AI or humans. The scientific method works the same way, we are not exempt from having to validate our ideas. But as humans we have better feedback and access to context and we can assume risks on our own. AI has no skin and bears no responsibility. So the missing ingredient for AI is access to environment for feedback learning. It has little to do with AI architecture or datasets. I think a huge source of such data is our human-LLM chat logs. We act as LLM eyes, hands and feed on the ground. We carry the tacit knowledge and social context. OpenAI reports billions of tasks per day, probably trillions of tokens of interactive language combining human, AI and feedback from the environment. Maybe this is how AI can inch towards learning how to solve real world problems, it is part of the loop of problem solving, and benefits from having this data for training.

bwfan123 1y ago
> Verification is the bottleneck In my use of cursor as a coding assistant, this is the primary problem. The code is 90% on the mark, but still buggy, and needs verification, and the feedback it gets from me is not with full fidelity as something is lost in translation. But, a bigger issue is that AI has only some solution templates for problems that it is trained on, and being able to generate new templates is beyond its capability as that requires training on datasets of higher levels of abstration.

ankit219 1y ago

The bottleneck for automation is verification. With human work, verification was fast(er) because you know where to look with certain assumptions that your upstream tasker would not have made trivial mistakes. For automation, AI needs to verify it's own work, review, and self correct to be able to automate any given work. Where this works, it will also change the abstraction layer compared to what it is today. The problem is same with every automation promise - it needs to work reliably at say 95% or 99% times and when it doesn't, there should be human contingency in terms of what to look for. Considering coding as the first example: it's already underway. AI generates the code, the test cases, and then verifies if the code works as intended. Code has a built in verification layer (both compiler and unit tests). High probablity the other domains move towards something similar too. I would also say the model needs to be intelligent to course correct when the output isn't validated[1]. Verification solves the human in the loop dependency both for AI and human tasks. All the places where we could automate in the past, there were clearly quality checks which ensured the machinery were working as expected. Same thing will be replicated with AI too. Disclaimer: I have been working on building a universal verifier for AI tasks. The way it works is you give it a set of rules (policy) + AI output (could be human output too) and it outputs a scalar score + clause level citations. So I have been thinking about the problem space and might be over rating this. Would welcome contrarian ideas. (no, it's not llm as a judge) [1]: Some people may call it environment based learning, but in ML terms i feel it's different. That woudl be another example of sv startups using technical terms to market themselves when they dont do what they say.

saint_yossarian 1y ago
One thing that comes to mind: You still have to verify that the tests are exhaustive, and that the code isn't just gaming specific test scenarios. I guess fuzzing and property-based testing could mitigate this to some extent.
- ankit219 1y ago
  Yes, we are getting there. I think compiler is a bigger problem than unit tests given most verticals don't even have that. With unit tests, there would be some reward hacking but would be controlled at the model level + tests. (this is one of the reason i dont believe in transformer based llm as a judge for a verifier)

fmbb 1y ago

> It’s because the bottleneck isn’t in intelligence, but in human tasks: specifying intent and context engineering. So the bottleneck is intelligence. Junior engineers are intelligent enough to understand when they don't understand. They interrogate the intent and context of the tasks they are given. This is intelligence. Solving math questions is not intelligence, computers have been better than humans at that for like 100 years, as long as you first do the intelligent part as a human: specifying the task formally. Now we just have computer programs with another kind of input in natural language, and which require dozens of gigabytes of video ram and millions of cores to execute. And we still have to have humans to the intelligent part, figure out how to describe the problem so the dumb but very very fast machine can answer the question.

pvtmert 1y ago
I truly love this comment, which essentially says: LLMs are glorified calculators, with ambiguous grammar. :)
- handzhiev 1y ago
  Computers are glorified calculators, yet they power most of our lives
  - giantrobot 1y ago
    Many computers and interfaces are deterministic. LLMs are by nature not deterministic and not even non-deterministic the same way on any two invocations given the same prompt and context. Natural language is ambiguous and for many languages very context dependent. It's not the greatest interface for a calculator from which we're expecting deterministic accurate answers. WolframAlpha is a more impressive front end to a calculator than I've seen out of LLMs. Not only does it show me how it translated my natural-ish language query but it shows me potential alternative interpretations to my question. LLMs by the nature of how training works can't necessarily tell me why and how they interpreted my prompt. The thinking models are better but still not great.
- bamboozled 1y ago

fifteen1506 1y ago

As a human, I'd also appreciate the specifications, documentation and meetings were not inaccessible to me.

graycat 1y ago

For the square covering problem, answer, 2.

Paratoner 1y ago

I'm endlessly fascinated by the way these Humans (probably) speak of their follow peers as though they are a problem to solve for. > Longer term, we can reduce the human bottleneck by Thank God we have ways to remove the thorn in our(?) side for good. The world can finally heal when the pursuit of fulfillment becomes inaccessible to the masses.

darepublic 1y ago

Model capability is absolutely the main constraint.

dash2 1y ago

I'm not sure about the assumption that science is context-free. Maths maybe, but a lot of practical science has tons of unformalized contextual knowledge that is "handed down" by practitioners. It's one reason why replication can be so hard. OTOH, I also think a lot of science is like 1% inspiration, 99% very mundane tasks like data cleaning. So no reason the AI can't help with that. And scientists write terrible code, so the bar is low :-)

aledalgrande 1y ago

I don't agree with the author. Where is the part about nondeterminism and hallucinations? Drawing a pretty chart doesn't make the argument true. All these benchmarks and competitions are on problems that have a _right answer_. I write most my code entirely through Claude at work and have Claude Max for personal, and I can see every day that even with the right context, it's not certain that the model is going to converge to a decent answer on complex real life issues. At least one thing I do agree on: model growth is not an exponential, like everyone thought when we were on the first leg of it, but a logarithmic.