8 ms·
>If you were trying and failing to use an LLM for code 6 months ago †, you’re not doing what most serious LLM-assisted coders are doing. Here’s the thing from
by davidclark 1y ago
>If you were trying and failing to use an LLM for code 6 months ago †, you’re not doing what most serious LLM-assisted coders are doing.
Here’s the thing from the skeptic perspective: This statement keeps getting made on a rolling basis. 6 months ago if I wasn’t using the life-changing, newest LLM at the time, I was also doing it wrong and being a luddite.
It creates a never ending treadmill of boy-who-cried-LLM. Why should I believe anything outlined in the article is transformative now when all the same vague claims about productivity increases were being made about the LLMs from 6 months ago which we now all agree are bad?
I don’t really know what would actually unseat this epistemic prior at this point for me.
In six months, I predict the author will again think the LLM products of 6 month ago (now) were actually not very useful and didn’t live up to the hype.
- simonw 1y agotptacek wasn't making this argument six months ago. LLMs get better over time. In doing so they occasionally hit points where things that didn't work start working. "Agentic" coding tools that run commands in a loop hit that point within the past six months. If your mental model is "people say they got better every six months, therefore I'll never take them seriously because they'll say it again in six months time" you're hurting your own ability to evaluate this (and every other) technology.
- JohnKemeny 1y agoBut they say "yes, it didn't work 6 months ago, but it does now", and they say this every month. They're constantly resetting the goal post. Today it works, it didn't in the past, but it does now. Rinse and repeat.
- Sebguer 1y agothis is only a compelling counter-argument if you are referring to a single, individual person who is saying this repeatedly. and there probably are! but the author of this article is not that person, and is also speaking to a very specific loop that only first truly became prevalent 6-9 months ago.
- jasonfarnon 1y ago"they say this every month" But I think the commenter is saying "they" comprises many different people, and they can each honestly say, at different times, "LLMs just started working". I had been loving LLMs for solving NLP since they came out, and playing with them all the time, but in my field I've only found them to improve productivity earlier this year (gemini 2.5).
- conception 1y agoI don’t think this is true actually. There was a huge shift of llm coding ability with the release of sonnet 2.5. That was a real shift in how people started using LMS for coding. Before that it was more of a novelty not something people used a lot for real work. As someone who is not a software engineer, as of about November 2024, I “write” hundreds of lines of code a day for meaningful work to get done.
- ipaddr 1y agoHow did you manage before?
- conception 1y agoThe work just wasn’t done. Or it took enough time for me to go and learn how to do it.
- nsonha 1y agoWhy focus on the 6 months or however long you think the cycle is. The milestones of AI coding are self-explanatory: autocomplete (shit) -> multi-files edit (useful for simple cases) -> agent (feedback loop with rag & tool use), this is where we are. Really think about it and ask yourself if it's possible that AI can make any, ANY work a little more efficient?
- jasonfarnon 1y agoI don't really get this argument. Technology can be improving can't it? You're just saying that people saying it's improving, isn't a great signal. Maybe not, but you still don't conclude that the tech isn't improving, right? If you're old enough, remember the internet was very much hyped. Al gore was involved. But it's probably been every bit as transformative as promised.
- prmph 1y agoTechnology improving is not the issue. 1. LLM fanboy: "LLMs are awesome, they can do x, y, and z really well." 2. LLM skeptic: "OK, but I tried them and found them wanting for doing x, y, and z" 3. LLM fanboy: "You're doing it wrong. Do it this way ..." 4. The LLM skeptic goes to try it that way, still finds it unsatisfactory. A few months pass.... 5. LLM fanboy: "Hey, have you tried model a.b.c-new? The problems with doing x, y, and z have now been fixed" (implicitly now agrees that the original complaints were valid) 6. LLM skeptic: "What the heck, I though you denied there were problems with LLMs doing x, y, and z? And I still have problems getting them to do it well" 7. Goto 3
- deleted 1y ago[deleted]
- skwirl 1y agoIt doesn’t really matter what this or that person said six months ago or what they are saying today. This morning I used cursor to write something in under an hour that previously would have taken me a couple of days. That is what matters to me. I gain nothing from posting about my experience here. I’ve got nothing to sell and nothing to prove. You write like this is some grand debate you are engaging in and trying to win. But to people on what you see as the other side, there is no debate. The debate is over. You drag your feet at your own peril.
- deadbabe 1y agoThe thing about people making claims like “An LLM did something for me in an hour that would take me days” is that people conveniently leave out what their own skill level is. I’ve definitely seen humans do stuff in an hour that takes others days to do. In fact, I see it all the time. And sometimes, I know people who have skills to do stuff very quickly but they choose not to because they’d rather procrastinate and not get pressured to pick up even more work. And some people waste even more time writing stuff from scratch when libraries exist for whatever they’re trying to do, which could get them up and running quickly. So really I don’t think these bold claims of LLMs being so much faster than humans hit as hard as some people think they do. And here’s the thing: unless you’re using the time you save to fill yourself up with even more work, you’re not really making productivity gains, you’re just using an LLM to acquire more free time on the company dime.
- esperent 1y agoI stopped paying attention for a few days so I'm way out of date. What is the state of the art for agentic coding now? I've been using Cline and it can do a few of the things suggested as "agentic", but I'd have no idea how to leave it writing and then running tests in a VM and creating a PR for me to review. Or let it roam around in the file tree and create new files as needed. How does that work? Are there better tools for this? Or do I need to configure Cline in some way?
- simonw 1y agotptacek is using Zed, which I've not tried myself. I actually do most of my "agentic coding" (not a fan of the term, but whatever) in ChatGPT Code Interpreter, which hasn't changed much in two years other than massive upgrades to the model it uses - I run that mainly via o4-mini-high or o3 these days. OpenAI's Codex is a leading new thing, but only if you pay $200/month for it. Google's equivalent https://jules.google/ https://jules.google/ is currently free. GitHub Copilot gained an "agent mode" recently: https://github.blog/ai-and-ml/github-copilot/agent-mode-101-all-about-github-copilots-powerful-mode/ https://github.blog/ai-and-ml/github-copilot/agent-mode-101-... There's also Copilot Coding Agent, which is confusingly an entirely different product: https://github.blog/changelog/2025-05-19-github-copilot-coding-agent-in-public-preview/ https://github.blog/changelog/2025-05-19-github-copilot-codi...
- garblegarble 1y agoI'd be quite interested in a more formal post with a detailed analysis of the effectiveness of the different agent impls, including Claude Code and Jetbrains Junie. Do you use ChatGPT Code Interpreter because it's better, or is it just something you're more familiar with and you're sticking with it for convenience? Of course, I don't know how one would structure a suitable test, since doing it sequentially would likely bias the later agents with clearer descriptions & feedback on the tasks. I imagine familiarity with how to prompt each particular model is also a factor.
- simonw 1y agoI like Code Interpreter because I'm deeply familiar with it. I don't have to worry about safety at all because it's running in OpenAI's kubernetes container, not on my own laptop. I can control exactly what it can see by selectively uploading files to it. I know it can't make outbound network requests.
- dingnuts 1y ago[flagged]
- simonw 1y agoHow is this "appeal to authority"? I'm countering the argument that "people say the same thing every six months" by arguing that the individual in question did NOT say the same thing six months ago.
- whoisthemachine 1y agoHave the models significantly improved, or have we just developed new programs that take better advantage of them?
- jppittma 1y agoBoth.
- simonw 1y agoYeah, definitely both. New models come out all the time. One of the most interesting signals to look out for is when they tip over the quality boundary from "not useful at task X" to "useful at task X". It happened for coding about a year ago. It happened for search-based research assistants just two months ago, in my opinion - I wrote about that here: https://simonwillison.net/2025/Apr/21/ai-assisted-search/ https://simonwillison.net/2025/Apr/21/ai-assisted-search/
- dcre 1y agoBoth, but it’s mostly the models. The programs like Claude Code are actually simpler than the ones from before because of this.
- cmdli 1y ago> tptacek wasn't making this argument six months ago. Yes, but other smart people were making this argument six months ago. Why should we trust the smart person we don't know now if we (looking back) shouldn't have trusted the smart person before? Part of evaluating a claim is evaluating the source of the claim. For basically everybody, the source of these claim is always "the AI crowd", because those outside the AI space have no way of telling who is trustworthy and who isn't.
- simonw 1y agoPart of being on Hacker News is learning that there are people in this community - like tptacek - who are worth listening to. In general, part of being an effective member of human society is getting good at evaluating who you should listen to and who is just hot air. I collect people who I consider to be credible and who have provided me with useful information in the past. If they start spouting junk I quietly drop them from my "pay attention to these people" list.
- bgwalter 1y ago[flagged]
- kalkin 1y agoIf you automatically lump anyone who makes an argument that AI is capable - not even good for the world on net, just useful in some tasks - into "the AI crowd", you will tautologically never hear that argument from anywhere else. But if you've been paying attention to software development discussion online for a few years, you've plausibly heard of tptacek and kentonv, eg, from prior work. If you haven't heard of them in particular, no judgement, but you gotta have someone you can classify as credible independently of their AI take if you want to be able to learn anything at all from other people on the subject.
- secalex 1y agoThomas is one of the pickier, crankier, least faddish technologists I've ever met. If he has gone fanboy that holds a lot of weight with me.
- anxoo 1y agoname 5 tasks which you think current AIs can't do. then go and spend 30 minutes seeing how current AIs can do on them. write it on a sticky note and put it somewhere that you'll see it. otherwise, yes, you'll continue to be irritated by AI hype, maybe up until the point where our civilization starts going off the rails
- apwell23 1y ago> name 5 tasks which you think current AIs can't do. For coding it seems to back itself into a corner and never recover from it until i "reset" it . AI can't write software without an expert guiding it. I cannot open a non trivial PR to postgres tonight using AI.
- simonw 1y ago"AI can't write software without an expert guiding it. I cannot open a non trivial PR to postgres tonight using AI." 100% true, but is that really what it would take for this to be useful today?
- deleted 1y ago[deleted]
- TheRoque 1y agoWell, I'll try to do a sticky note here: - they can't be aware of the latest changes in the frameworks I use, and so force me to use older features, sometimes less efficient - they fail at doing clean DRY practices even though they are supposed to skim through the codebase much faster than me - they bait me into inexisting apis, or hallucinate solutions or issues - they cannot properly pick the context and the files to read in a mid-size app - they suggest to download some random packages, sometimes low quality ones, or unmaintained ones
- travisjungroth 1y agoThose aren't tasks.
- 1y ago
- dolebirchwood 1y ago> Here’s the thing from the skeptic perspective: This statement keeps getting made on a rolling basis. Dude, just try the things out. It's just undeniable in my day-to-day life that I've been able to rely on Sonnet (first 3.7 and now 4.0) and Gemini 2.5 to absolutely crush code. I've done 3 side projects in the past 6 months that I would have been way too lazy to build without these tools. They work. Never going back.
- ryandrake 1y agoWhy can't reviews of AI be somewhere in the middle between "useless" and "the second coming"? I tried Copilot a few months ago just to give it a shot and so I could discuss it with at least a shred of experience with the tool, and yea, it's a neat feature. I wouldn't call it a gimmick--it deserves a little more than that, but I didn't exactly cream my pants over it like a lot of people seem to be doing. It's kind of convenient, like a smart autocomplete. Will it fundamentally change how I write software? No way. But it's cool.
- dolebirchwood 1y agoMy anecdotes are all I've got, bro. If they come a little creamy, well, YMMV.
- deleted 1y ago[deleted]
- idlewords 1y agoAn exponential curve looks locally the same at all points in time. For a very long period of time, computers were always vastly better than they were a year ago, and that wasn't because the computer you'd bought the year before was junk. Consider that what you're reacting to is a symptom of genuine, rapid progress.
- Retr0id 1y agoI don't think anyone's contesting that LLMs are better now than they were previously.
- Nevermark 1y ago> It creates a never ending treadmill of boy-who-cried-LLM. The crying wolf reference only makes sense as a soft claim that LLM’s better or not, are not getting better in important ways. Not a view I hold.
- Retr0id 1y agoThe implicit claim is just that they're still not good enough (for whatever the use cases the claimant had in mind)
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- pera 1y agoA flatline also looks locally the same at all points in time.
- EA-3167 1y agoNor does local flatness imply direction, the curve could be descending for all that "looks locally flat" matters. It also isn't on the skeptics to disprove that "AI" is a transformative, exponentially growing miracle, it's on the people selling it.
- stouset 1y agoI saw this article and thought, now's the time to try again! Using Claude Sonnet 4, I attempted to add some better configuration to my golang project. An hour later, I was unable to get it to produce a usable configuration, apparently due to a recent v1-to-v2 config format migration. It took less time to hand-edit one based on reading the docs. I keep getting told that this time agents are ready. Every time I decide to use them they fall flat on their face. Guess I'll try again in six months.
- Yiin 1y agoyou can add links to docs to llm agents instead of letting them work blindfolded with hardcoded assumptions
- maleldil 1y agoClaude Code will even request access documentation on its own sometimes. I caught it asking to run a `pydoc` command the other day. I'm not sure if it has access to web search, but it should.
- stouset 1y agoIt reached out to the Internet and pulled the docs. Repeatedly. I even linked to the docs directly to be helpful.
- simonw 1y agoIf you share your conversation (with the share link in Claude) I'd be happy to see if there are any tweaks I can suggest to how you prompted it.
- porridgeraisin 1y agoYes. I made the mistake of procrastinating on one part of a project thinking "Oh, that is easily LLMable". By God, was I proven wrong. Was quite the rush before the deadline. On the flip side, I'm happy I don't have to write the code for a matplotlib scatterplot for the 10000th time, it mostly gets the variables in the current scope that I intended to plot. But I've really not had that much success on larger tasks. The "information retrieval" part of the tech is beautiful though. Hallucinations are avoided only if you provide an information bank in the context in my experience. If it needs to use the search tool itself, it's not as good. Personally, I haven't seen any improvement from the "RLd on math problems" models onward (I don't care for benchmarks). However, I agree that deepseek-r1-zero was a cool result. Pure RL (plain R1 used a few examples) automatically leading to longer responses. A lot of the improvements suggested in this thread are related to the infra around LLMs such as tool use. These are much more well organised these days with MCP and what not, enabling you to provide it the aforementioned information bank easily. But all of it is built on top of the same fragile next-token generator we know and love.
- someothherguyy 1y agoAlso, professional programmers have varying needs. These people are coding in different languages, with varying complexity, domains, existing code bases and so on. People making arguments based on sweeping generalizations to a wide audience are often going to be perceived as delusional, as their statements do not apply universally to everyone. To me, thinking LLMs can code generally because you have success with them and then telling others they are wrong in how they use them is making a gigantic assumptive leap.
- spacemadness 1y agoI just assume every blog post in HN starts with “As a web dev, TITLE”
- libraryofbabel 1y agoThis isn't a particularly useful filter, because it applies to many very successful technologies as well. Early automobiles generated a lot of hype and excitement, but they were not very good (unreliable, loud, and dangerous, and generally still worse than horses). They got steadily better until eventually they hit an inflection point where the skeptics were dug in repeating the same increasingly old complaints, while Henry Ford was building the Model T.
- orionsbelt 1y agoAt what point would you be impressed by a human being if you asked it to help you with a task every 6 months from birth until it was 30 years old? If you ask different people the above question, and if you vary it based on type of task, or which human, you would get different answers. But as time goes on, more and more people would become impressed with what the human can do. I don't know when LLMs will stop progressing, but all I know is they continue to progress at what is to me a similar astounding rate as to a growing child. For me personally, I never used LLMs for anything, and since o3 and Gemini 2.5 Pro, I use them all the time for all sorts of stuff. You may be smarter than me and still not impressed, but I'd try the latest models and play around, and if you aren't impressed yet, I'd bet money you will be within 3 years max (likely much earlier).
- Velorivox 1y ago> At what point would you be impressed by a human being if you asked it to help you with a task every 6 months from birth until it was 30 years old? In this context, never. Especially because the parent knows you will always ask 2+2 and can just teach the child to say “four” as their first and only word. You’ll be on to them, too.
- Velorivox 1y agoTo be clear, I’m just saying the analogy isn’t great, not that one can never be impressed by an LLM (or a person for that matter)!
- x-complexity 1y ago> In this context, never. Especially because the parent knows you will always ask 2+2 and can just teach the child to say “four” as their first and only word. You’ll be on to them, too. On the assumption that you'll always only ask it "what's 2+2?" Keywords being "always" & "you". In aggregate, the set of questions will continuously expand as a non-zero percentage of people will ask new questions. The set of questions asked will continue to expand, and the LLMs will continue to be trained to fill in the last 20%. Even under the best interpretations, this is the detractors continuously moving goalposts, because the last 20% will never be filled: New tasks will continuously be found, and critics will point to them as "oh, see, they can't do that". By the time that the LLMs can do those tasks, the goalpost will be moved to a new point and they'll continue to be hypocrites. ------ > > At what point would you be impressed by a human being if you asked it to help you with a task every 6 months from birth until it was 30 years old? Taking GP's question seriously: When a task consisting of more than 20 non-decomposable (atomic) sub-tasks is completed above 1 standard deviation of the human average in that given task. (much more likely) OR When an advancement is made in a field by that person. (statistically much rarer)
- carpo 1y agoI think they just meant it hit an inflection point. Some people were copying pasting to ChatGPT and saying it was crap and others were using agents that could see the context of the code and worked much, much better. It's the workflow used not just the specific LLM.
- mathgorges 1y agoIn my experience it's less about the latest generation of LLMs being better, and more about the tooling around them for integration into a programmer's workflow being waaaay better. The article doesn't explicitly spell it out until several paragraphs later, but I think what your quoted sentence is alluding to is that Cursor, Cline et al can be pretty revolutionary in terms of removing toil from the development process. Need to perform a gnarly refactor that's easy to describe but difficult to implement because it's spread far and wide across the codebase? Let the LLM handle it and then check its work. Stuck in dependency hell because you updated one package due to a CVE? The LLM can (often) sort that out for you. Heck, did the IDE's refactor tool fail at renaming a function again? LLM. I'm remain skeptical of LLM-based development insofar as I think the enshitification will inevitably come when the Magic Money Machine breaks down. And I don't think I would hire a programmer that needs LLM assistance in order to program. But it's hard to deny that it has made me a lot more productive. At the current price it's a no-brainer to use it.
- tho23j4o3j4324 1y agoIt's great when it works, but half the time IME it's so stupid that it can't even use the edit/path tools properly even when given line numbers prepended inputs. (I should know since I've created half-a-dozen tools for this with gptel. Cline hasn't been any better on my codebase.)
- karthink 1y agoDo Cursor and co have better tools than the ones we write ourselves for lower-level interfaces like gptel? Or do they work better because they add post-processing layers that verify the state of the repo after the tool call?
- tho23j4o3j4324 1y agoCursor is proprietary, but is known to index code for doing queries etc. Cline is closer in spirit to GPTel, but since CLINE is an actual business, it does seem to do well off the bat. That said, I haven't found it to be "hugely better" compared to whatever you can hack in GPTel. Quite frankly being able to hack the tools on the go in Elisp, makes GPTel far far better (for some of us anyway). (Thanks for creating GPTel BTW!)
- deleted 1y ago[deleted]
- killerstorm 1y agoBullshit. We have absolute numbers, not just vibes. The top of SWE-bench Verified leaderboard was at around 20% in mid-2024, i.e. AI was failing at most tasks. Now it's at 70%. Clearly it's objectively better at tackling typical development tasks. And it's not like it went from 2% to 7%.
- lexandstuff 1y agoIsn't SWE-bench based on public Github issues? Wouldn't the increase in performance also be explained by continuing to train on newer scraped Github data, aka training on the test set? The pressure for AI companies to release a new SOTA model is real, as the technology rapidly become commoditised. I think people have good reason to be skeptical of these benchmark results.
- killerstorm 1y agoThat sounds like a conspiracy theory. If it was just some mysterious benchmark and nothing else then sure, you have reasons to be skeptical. But there's a plenty of people who actually tried LLMs for actual work and swear they work now. Do you think they are all lying?.. Many people with good reputation, not just noobs.
- deleted 1y ago[deleted]