7 ms·
> The ‘give up after ten attempts’ threshold aims to prevent Claude from wasting tokens when further progress is unlikely. It was only partially successful, as
by t_mann 10mo ago
> The ‘give up after ten attempts’ threshold aims to prevent Claude from wasting tokens when further progress is unlikely. It was only partially successful, as Claude would still sometimes make dozens of attempts.
Not what I would have expected from a 'one-shot'. Maybe self-supervised would be a more suitable term?
- wavemode 10mo ago"one-shot" usually just means, one example and its correct answer was provided in the prompt. See also, "zero-shot" / "few-shot" etc.
- simonw 10mo agoI've seen one-shot used to mean two different things in LLMs: 1. Getting an LLM to do something based on a single example 2. Getting an LLM to achieve a goal from a single prompt with no follow-ups I think both are equally valid.
- baq 10mo agoOne-shot as in ‘given one example’ is the ML term. One-shot as in ‘in a single prompt’ is the colloquial meaning. Both are useful, but it can be confusing when discussing LLMs in ML topics.
- t_mann 10mo agoThe article says that having decompiled some functions helps with decompiling others, so it seems like more than one example could be provided in the context. I think the OP was referring to the fact that only a single prompt created by a human was used. But then it goes off into what appears to be an agentic loop with no hard stopping conditions outside of what the agent decides. We're essentially trying to map 'traditional' ML terminology to LLMs, it's natural that it'll take some time to get settled. I just thought that one-shot isn't an ideal name for something that might go off into an arbitrarily long loop.
- hombre_fatal 10mo agoMeh, the main idea of one-shot is that you prompted it once and got a good impl when it decided it was done. As opposed to having to workshop yourself with additional prompts to fix things. It doesn't do it in one-shot on the GPU either. It feeds outputs back into inputs over and over. By the time you see tokens as an end-user, the clanker has already made a bunch of iterations.
- johnfn 10mo agoOne shot just means one prompt. What Claude decides to do during that prompt is up to it.
- voiper1 10mo agoI definitely didn't expect one-shot to mean "let it run itself in an indefinite loop"