9 ms·

i'm not sure why people expect agents to one shot everything to perfection with just a prompt. There's a reason why we talk about software development lifecycl

• 0x457 24d ago
  > i'm not sure why people expect agents to one shot everything to perfection with just a prompt. because that's how agents are marketed.
  • addandsubtract 23d ago
    Not just marketed, but also benchmarked (and benchmaxxed).
  • uproarchat 24d ago
    I've never seen model providers marketing like that. What examples have you seen?
    • Lalabadie 24d ago
      I don't really think they advertise "Create your app idea in one weekend night" and assume the general public will mentally add "... but hire an experienced developer to supervise the process".
    • 0x457 24d ago
      Literally any coding agent marking material: - https://cognition.com/ https://cognition.com/ - https://openai.com/index/introducing-the-codex-app/ https://openai.com/index/introducing-the-codex-app/ - https://www.anthropic.com/news/claude-3-7-sonnet https://www.anthropic.com/news/claude-3-7-sonnet anthropic specifically brags about how good claude code is every annoucement of a new model. I will surrender that none of them claim its "to perfection", but IMO its implied because no one would claim that their model one-shots any issue to dog shit quality.
      • hombre_fatal 24d ago
        Seems like motte and bailey fallacy. They say their models are good (the motte), therefore their models must one-shot everything to perfection (the bailey). Besides, other people's claims about something doesn't give you license to abandon all critical thinking. Though it's evident they don't claim what you say they are.
        • 0x457 24d ago
          That's irrelevant. Questions was why people assume somthing, and answer is because that's how it advertised. To be clear that's not what I'm thinking, even Fable 5 produces some hilariously bad results under some conditions and sonnet 5 produced great results under others.
          • hombre_fatal 24d ago
            > because that's how it advertised. But you didn't provide the evidence for that. You shared some links and then admitted they didn't claim it. It kinda seems like "because I think they're a little too positive about their product, I can set my expectations to anything I want and la-la-la it's their fault." And I don't see the problem with agents building test scaffolding as they go. It might be too defensive at times, like testing a shell script you don't run often, but big deal. It's kinda cool imo, and it's trivial to make it stop.
      • infinite_spin 24d ago
        - https://openai.com/index/introducing-the-codex-app/ https://openai.com/index/introducing-the-codex-app/ no where does this document suggest that codex can "one shot everything to perfection with just a prompt". It describes using a prompt plus agent skills (which are essentially many other prompts) to develop a playable game.. nothing about it being perfect or anything more than being in a playable state.
    • mrheosuper 24d ago
      I've seen a lot of cursor app, about some PO that has an idea for an app in the morning, then asking her agent to make it when her commuting, when arrive at office the app is done
    • onion2k 24d ago
      Claude has a /goal function that explicitly says it'll carry on working until it's done what you prompted it to do. That's exactly what a 'AI will zero-shot anything' believer is looking for. Behind the scenes it's really multi-shotting with generated prompts, but the user won't care.
  • deleted 24d ago
    [deleted]
  • vanuatu 24d ago
    we should exercise critical thinking then heaps of people on this site expect them to be omnipotent then claim it’s fake when it doesn’t read minds
    • joshribakoff 24d ago
      You’re right — people really should think critically, but the issue remains, that many do not.
    • bigstrat2003 24d ago
      I think it's perfectly fair to evaluate the tools based on how well they live up to the hype that is being pumped out by the sellers of said tools. If they want us to compare their products to a more measured, reasonable take then they can advertise them as that.
      • sfn42 23d ago
        While some people are busy bickering about this, the rest of us are using these awesome new tools to get more work done in less time with higher quality than ever. I don't care what the company claims, I just use the tool the way I want to. I work very closely with the AI. I'll tell it to plan a change, review the plan, then execute. Then I'll test the changes and have it fix whatever I'm not happy with one thing at a time. I'll specify in detail both what to do and loosely describe how to do it or if I'm not sure I'll ask it to plan the change then review the plan and ask for changes if I want them etc. I also review my own PRs before I submit them to colleagues. This way I maintain full control of everything, it just saves me hours of googling, planning and typing code - which I do miss a bit but I can't really justify writing code myself when I can achieve the same thing just by loosely describing my idea instead. It also saves a lot of time debugging, I think I'm generally a pretty good programmer but the AI makes fewer mistakes than me. It'll often catch some logic error I made during planning and suggest a good alternative. A lot of developers seem to give up control entirely and then complain that they're no longer in control. Trying for that 10-100x speedup doing weeks of work in a day. I'm happy doing one week of work in a day. There's a limit to how much I can oversee without compromising quality.
    • 24d ago
• jimmaswell 24d ago
  > i'm not sure why people expect agents to one shot everything to perfection with just a prompt. They do often enough that it's not a surprising event, depending on prompt quality, context available, ability for the result to be objectively judged and iterate on by the agent, etc. For frontiers on very high settings at least.
• redox99 24d ago
  > i'm not sure why people expect agents to one shot everything to perfection with just a prompt. because that's the end goal? and for simple small stuff they're already there?
  • sfn42 23d ago
    It's an impossible goal. "Perfection" is subjective and situational. It's basically impossible to specify a non-trivial task perfectly, and without a perfect specification you can't have a consistently perfect result. Personally I find it much more efficient to give vague instructions and refine on the way, rather than trying to specify everything up front. With this workflow there is no such thing as a one-shot, I don't even know all the details of my intended result until I reach it. Sure it can one-shot many small things, but for a larger feature it has to be a incremental process. Even when I have a Figma design to work from it never contains all the details like semantics of how various interactive elements work, edge cases etc.
• perarneng 24d ago
  If you explicitly ask the agent to make the perfect architecture for the problem and write it down in to a spec and have the developer agents follow it they will. Its just that coding agents have a hard time coding at think about architecture at the same time.
• JoshTriplett 24d ago
  > i'm not sure why people expect agents to one shot everything to perfection with just a prompt. Every time you see a benchmark for "how long the agent can go without asking for human intervention", that's encouraging vibe coding.
• giancarlostoro 24d ago
  I can one shot a prompt if I write down a nice spec file, Claude can do a lot in one shot. I test it every few months. With enough detail Claude will know what to do.
  • drdo 24d ago
    So we're coding in an ill-defined, ambiguous and error prone language. Sweet, I can't believe some people don't love this.
  • polnoner 24d ago
    Opus 5 one shot an access virus B synth clone for me as a single page index.html that is more impressive than anything I have seen as a VST synth. That is also because I have been obsessed with this synth for almost 30 years. I built clones of it 20 years ago in reaktor. I know how to spec out every aspect of this synth and I gave Claude a 150 page pdf on digital filter design too. The results are far different than someone who has never used a virus prompting "make me an access virus B synth as a single html page". We are calling both of these processes "one shot" but this is not even close to the same process. I suspect this is the LLM discourse in a nutshell. People are using the same vocabulary for wildly different processes.
    • giancarlostoro 24d ago
      100% that is part of the problem. :)
• CoolestBeans 24d ago
  Using AI is kayfabe. What I mean is, you create interaction patterns that resemble how humans work. This is because it is what the models are trained on but also because we've all been trained to interact in this way. So it manipulates you into providing more useful prompts. But I don't really want to play a part in a simulation, trying to cajole my scene partners into saying the lines I need them to say. I want to use a tool the same way I would use any other tool. If this is AI it should just do the thing. Anything else is an imperfection of the technology. But at the same time, language is a vague communication medium. We have a precise language for describing forms of computation, but that's code so we're back at square one. We still haven't nailed the right amount of follow up and correction and interrupt-ability of these coding agents. And we may never figure it out. It may simply be impossible. But it doesn't mean this weird anthropomorphization of AI is something I want to do. If I wanted to be a manager, I would be a manager.
  • sfn42 23d ago
    > language is a vague communication medium. We have a precise language for describing forms of computation, but that's code so we're back at square one. That's why you tell it how to write the code, and then review the code to ensure it is what you wanted. > We still haven't nailed the right amount of follow up and correction I feel like I've got it under control. It's not really a problem at all to me. I just work closely with the AI. I do small tasks, I don't just have it generate thousands of lines at once. I tell it what to do and how, or give it some vague guidance and ask it to make a plan. Then review the plan, ask for some changes if necessary and execute. Then I go over all the changes, test them to ensure they work properly, have it fix any issues I find and so on. I think the main problem with AI coding is people try to do too much. You can't keep a tight leash on it while also having it do a week's worth of work in an hour. I do one task at a time and I am heavily involved in it, deciding exactly how it's done. I micromanage the crap out of that thing. I write commits myself and I always review my own PR before submitting it to colleagues. Works great. I get things done much faster than I used to, with better quality than before.