12 ms·
Agent mania setting in It's also pretty funny sometimes how it gives weird future roadmap estimates ("part 2 - 3 weeks, part 3 - 2 months", etc.) and when you
by throwaway67678 3mo ago
Agent mania setting in
It's also pretty funny sometimes how it gives weird future roadmap estimates ("part 2 - 3 weeks, part 3 - 2 months", etc.) and when you tell it to actually do those changes it's pretty much done in half an hour
- smith7018 3mo agoI've long believed those numbers were faked by Anthropic/OpenAI to serve as a form of advertisement. The estimates are impossible to verify and their ability to do "2 days of work" in 10 minutes will presumably make the user go "Wow, I just saved SO much time!" Plus, the unnecessary text eats up the users' tokens so it helps the companies on the backend, as well.
- leodavi 3mo agoI agree with you that labs are benefiting from those outputs but I'm skeptical that labs are purposefully training the models to produce those outputs. Raw pre-training data includes plenty of conversations between professional builders and some of those include estimates. I believe the outputs are a training coincidence with consequences that are opportunitistic for the labs.
- AgentMasterRace 3mo agoAll the models have broken estimates. They're trained heavily on jira and GitHub tasks and issues, that's why their estimates are human.
- esperent 3mo agoEven for humans the estimates are way off, unless it's based on data that has some serious padding. That said, it'll often say "2 days of work" and then complete the coding in 30 minutes, and while that's amusing, afterwards, I'll need to manually test, or send to other people for review, or realize the agent only actually did half the work and I need to do a second pass (or a third etc.) and then often getting the feature in does genuinely take two days.
- dizhn 3mo agoAll models do it. It's their training. They didn't have "a person does this in a week but an LLM could in a minute" in their training yet. They also don't have the concept of elapsed time unless you ask them how long something has taken.
- Terretta 3mo ago> the estimates It doesn't estimate. It generates tokens that read like estimates associated with the context in its training material. What would you expect the generator to output instead?
- ghshephard 3mo agoI think people are continuing to view these systems as pure LLMs - when that ship sailed 6+ months ago. Between being able to review memory, using agent harnesses and sub agents and skills to go out and discover information - modern systems (Codex, Claude Code, Cursor) - use LLMs - but the LLM is only a small component of it. Compare what you get from sending a request to a chatbot like ChatGPT - to what you can from a modern harness. The output is influenced by the LLM, but it's no longer a "model making a token prediction based on training material and RLHF" - that's a very 2025 way of looking at these systems. Even Gary Marcus is starting to come around and realize that his priors are no longer as relevant as they once were.
- irthomasthomas 3mo agoNo one is bitter lesson pilled anymore. Everyone is pivoting to neurosymbolic systems. It looks like Gary Marcus was right.
- nl 3mo ago> No one is bitter lesson pilled anymore. Will the 10T parameter Mythos model be released this month or next month? They better soon because it is generally accepted that one of the reasons GPT 5.5 is better at hard tasks than Opus is because of its parameter size - and that Opus 4.8 remains competitive only be scaling test-time compute (see how many more tokens it uses than GPT 5.5) https://www.reddit.com/r/LLM/comments/1sz8bjz/parameter_estimate/ https://www.reddit.com/r/LLM/comments/1sz8bjz/parameter_esti...
- irthomasthomas 3mo agoWhy ask me? Anyway, Mythos is not 10T. Anthropic confirmed the training run was under 10^26 flops. You can't train 10T to chincilla and stay under 10^26. Anthropic also confirmed they will not release Mythos, only a "Mythos-class" model, whatever that means.
- KronisLV 3mo agoI mean in general I'd rather take slightly inflated estimates than the odd sprint poker stuff where other devs and PMs negotiate hours down and before you know it you're also stuck fixing nitpicky reviewer comments on code that is already good enough and have to send a release at like 7 PM, ofc also without enough tests or even enough manual checks and testing, cause people repeatedly act against their self-interest and try to compress timelines, thinking that that's somehow good for them. At least with AI that actually does things more quickly, there is a bit more breathing room (introducing AI is easier than changing a given environment). Aside from that, I wonder how much variety there is in practice: between "Oh yeah, I added that new button while we were in the meeting" and "The new button feature will be ready in Q3 according to the roadmap, once we have sign-off from all the stakeholders."
- Narciss 3mo agoNah it’s all from the pretraining data
- overgard 3mo agoI tend to be cynical about AI companies, but I'm guessing the bad estimates more just come from a complete lack of actual data it could use for that so it's more or less a hallucination.
- BobbyTables2 3mo agoThat’s right up there with Scotty in the classic Star Trek always multiplying time estimates by 4 so he looks like a “miracle worker”
- throw1234567891 3mo agoIt repeats what it has seen in the training data. Expecting it to reason about the complexity of a task is a pipe dream. The best is to tell it not to come back with estimates, and when it does, remove them anyway.
- andai 3mo agoI added "you can do anything, believe in yourself" to system prompt, and task completion increased significantly.
- andai 3mo agoI heard an anecdote. Guy spent several days trying to convince his AI agent to build a feature. Kept saying it was crazy complicated, would take weeks. Finally he convinced it to try. It one shotted it in 30 seconds. Turns out the agents' idea of what is hard and easy also comes from Common Crawl.
- wild_egg 3mo agoWhy on earth would you spend any time at all convincing an agent of anything? You say "just do it" and off it goes.
- dr_dshiv 3mo agoYa, but “doit” is 2x more efficient
- brianwawok 3mo agoUh Claude tries real hard to dodge work. Talks about how it’s really hard 10 PRs. Finally convince it to do as 1. It stops 10% through and says ok done with PR 1, we can work on the last 9 tomorrow. Ugh.
- handfuloflight 3mo agoMaybe we shouldn't have AI mimic humans too closely?
- g8oz 3mo agoYou need to assert dominance.
- znpy 3mo ago> It's also pretty funny sometimes how it gives weird future roadmap estimates ("part 2 - 3 weeks, part 3 - 2 months", etc.) those estimates are based on previous human estimates (the datasets it's been trained on). unironically, when your comments will become part of a dataset, LLMs will likely get much better at estimating. now that i think about it, all these writings about LLMs will give LLMs something much like meta-cognition.
- jimbokun 3mo agoWell how else could I keep my reputation as a miracle worker Captain?