6 ms·
When I hear this I do wonder to myself how they're using AI. For me, as long as I'm properly RPI looping it and not blindly pressing "yes" then it will nearly
by JauntyHatAngle 12d ago
When I hear this I do wonder to myself how they're using AI.
For me, as long as I'm properly RPI looping it and not blindly pressing "yes" then it will nearly always reach the solution, usually a fair bit quicker, because it effectively becomes an ideation machine that can keep more thoughts and knowledge in it than my brain can.
However, I'm using it through giving it the context, it has access to the repos, access to the sytems, I'm telling it where the logs are, I have docs to show it what each part of the system is for.
Along with that, it does depend on my own instincts/knowledge for me to read its response and for me to say "no, you've over-thought/engineered this and this is actually the better solution", but its very are nowadays for me that it can't find the 1 liner, as long as I've fed in the right context, mostly pre-done because I've spent a bit of time building out the context tree for the repos/systems in a way that allows it to know what it needs to know.
I'm not doubting that people spin their wheels and couldn't find the one liner, but if its as simple as you say, that does seem like people who aren't great at LLMs along with a lack of instinct/experience.
I guess in essence, I think you can use LLMs in an old school troubleshooting way, and I find it still speeds that up the majority of the time. Its basically how I use it most of the time. And like old-school troubleshooting, if you build out the LLMs context over time, it also grows in capability, as long as its being used as a tool and not blindly trusted.
Should I not assume that most senior developers who are serious about LLMs do this?
- nekoashide 12d agoIn my experience at a large bank with unlimited AI, my spend is in the top 5% and I'm leveraging AI just like you. I was in a meeting with someone who had a bug in an application that I don't own myself on Friday so I told claude, "I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed" 15 minutes later the developer ask me if I want it fixed today or Tuesday. That user could have done the same thing as me, had access to all the same systems and tools as I have, and also received the same AI training I had. The difference is that some users are just not, for lack of better words, AI native.
- piker 12d agoOften though working through a bug teaches the developer something about the relevant abstraction. That knowledge can get lost in this new process.
- nottorp 12d agoClaude will happily add a null guard without checking why the null is there in the first place...
- JauntyHatAngle 11d agoWhile you're right, and I'm constantly thinking/worrying about my own processes giving me AI apathy, you can use AI in a way that develops your understanding of a bug. It really depends on your workflow. When I use AI to debug a problem, I'm constantly questioning the specifics because I want to learn it, and often I'm better than the AI at quickly recognising patterns that point to a specific issue, where the AI will spin for some time trying to work out where to start - even if it has full context. So I guess I'm half with you, but an LLM used right can still teach you effectively.
- piker 11d agoYes. I agree with that as well. If you can intuit that it's (for example) an off-by-one error somewhere in your logic, asking an LLM to pinpoint that issue can save you tons of time that is better spent thinking about your abstractions. Also, I tend to ask high-level questions to the frontier models about best practices with common abstractions. But "the screen flickers here, please fix" is a bad pattern, IMO. Understanding why the screen is flickering is likely to educate you about where your abstractions are leaking.
- sidhsjsjhz 11d ago> "I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed" 15 minutes later the developer ask me if I want it fixed today or Tuesday. Did you find out if it actually fixed the issue? I was on the other end of this last week. Reported an issue to another team, they asked Claude to diagnose, and when they gave me the response back I pushed back cause it didn’t make sense given the behavior we were seeing. Turns out Claude had hallucinated reading a log (it said it did but it didn’t). Sent it down a hole rabbit hole from there. I’ve had the one shots you describe and they’re great, but they’re the happy path and require almost 0 skill to find. I’ve yet to see a case where developing an expert level knowledge of your domain isn’t the best way to get good at LLMs. Knowledge of how to interface with these tools is helpful but changes (and depreciates) rapidly as the models get smarter and/or other people commoditize it. Deep subject knowledge is still the best way to get a lot out of these tools. This applies to development and other areas I have a deep knowledge of. “LLM training” is kinda snake oil. Learn your craft deeply and you’ll be able to catch up on LLM training in a few days, but you’ll be light years ahead of a person who’s only expertise is different ways to prompt.
- oblio 12d ago> RPI loop What's that?
- JauntyHatAngle 11d agoResearch Plan Implement. At the risk of you already knowing this or similar - in essence for a slice of a piece of work, you get the LLM to research the information required into (usually) a research file, which you go back and forth with in-line or via conversation until you're happy. You then build the plan for the piece of work, again going back and forth with the LLM until you agree with the plan, with the key here being you have a specific set of concrete steps with very limited vagueness. Again this is quite often a plan document. You then implement the plan with your LLM, following the concrete steps and monitor/evaluate the result. For me its the 101 mandatory first way to use LLM that turns it from guesswork to genuinely useful. (that or some other similar method, SDD or whatever) Because without it or something similar you're fully dependent on AI's weaknesses not coming to the fore (context/scoping failures that are silent unless you demand visibility up front and ruthlessly chop/clarify), and you often ascribe bad AI slop as model limitations instead of context/scoping failures.