5 ms·

I stumbled into Agentic Coding in VS Code Nightlys with co-pilot using Claude Sonnet 4 and I've been silly productive. Even when half my day is meetings, you wo

I stumbled into Agentic Coding in VS Code Nightlys with co-pilot using Claude Sonnet 4 and I've been silly productive. Even when half my day is meetings, you wouldn't be able to tell from my git history. My thinking now is removed from the gory details and is a step or two up. How can I validate the changes are working? Can I understand this code? How should it be structured so I can better understand it? Is there more we can add to the AI conventions markdown in the repo to guide the Agent to make fewer mistaken assumptions? Last night I had a file with 38 mypy errors. I turned it over to the agent and went and had a conversation with my wife for 15 minutes. I came back, it summarized the changes it made and why, I debated one of the changes with it but ultimately decided it was right. Mypy passed. Good to go. I'm currently trying to get my team to really understand the power here. There's a lot of skeptics and the AI still isn't perfect and people who are against the AI era will latch onto that as validation but it's exactly opposite the correct reaction. It's really validation because as a friend of mine says "Today is the worst day you will have with this technology for the rest of your life."

GardenLetter27 1y ago
I trust it more with Rust than Python tbh, because with Python you need to make sure it runs every code path as the static analysis isn't as good as clippy + rust-analyzer.
- diggan 1y ago
  I agree, had more luck with various models writing Rust than Python, but only in the case where they have tools available so one way or another it can run `cargo check` and see the nice errors, otherwise it's pretty equal between the two. I think the excellent error messages in Rust also help as much humans as it does LLMs, but some of the weaker models get misdirected by some of the "helpful" tips, like some error message suggest "Why don't you try .clone here?" when the actual way to address the issue was something else.
- redman25 1y ago
  That's true typed languages seem to handle the slop better. One thing I've noticed specifically with rust is that agents tend to overcomplicate things though. They tend to start digging into the gnarlier bits of the language much quicker than they probably need to.
km144 1y ago
> Is there more we can add to the AI conventions markdown in the repo to guide the Agent to make fewer mistaken assumptions? Forgive my ignorance, but is this just a file you're adding to the context of every agent turn or this a formal convention in the VS code copilot agent? And I'm curious if there's any resources you used to determine the structure of that document or if it was just a refinement over time based on mistakes the AI was repeating?
- jnwatson 1y ago
  I just finished writing one. It is essentially the onboarding doc for your project. It is the same stuff you'd tell a new developer on your team: here are the design docs, here are the tools, the code, and this is how you build and test, and here are the parts you might get hung up on. In hindsight, it is the doc I should have already written.
apwell23 1y ago
> you wouldn't be able to tell from my git history. I can easily tell from git history which commits were heavily AI generated
ajdidbdbsgs 1y ago
> Last night I had a file with 38 mypy errors Fixing type checker errors should be one the least time consuming things you do. This was previously consuming a lot of your time? A lot of the AI discourse would be more effective if we could all see the actual work one another is doing with it (similar to the cloudflare post).
- diggan 1y ago
  > AI discourse would be more effective if we could all see the actual work one another is doing with it Yes, this is a frequent problem both here and everywhere else. The discussions need to include things like exact model version, inference parameters, what system prompt you used, what user prompt, what code you gave it, what exactly it replied and so much more details, as currently almost every comment is "Well, I used Sonnet last week and it worked great" without any details. Not to mention discussions around local models missing basic stuff like what quantization (if any) and what hardware you're running it on. People just write out "Wow fast model" or stuff like that, and call it a day. Although I understand why, every comment be huge if everyone always add sufficient context. I don't know the solution to this, but it does frustrate me.
  - square_usual 1y ago
    There's many examples of exactly what you're asking for, such as Kenton Varda's Cloudlfare oauth provider [1] and Simon Willison's tools [2]. I see a new blog post like this with detailed explanations of what they did pretty frequently, like Steve Klabnik's recent post [3], which while it isn't as detailed has a lot of very concrete facts. There's even more blog posts from prominent devs like antirez who talk about other things they're doing with AI like rubber ducking [4], if you're curious about how some people who say "I used Sonnet last week and it was great" are working, because not everyone uses it to write code - I personally don't because I care a lot about code style. [1]: https://github.com/cloudflare/workers-oauth-provider/ https://github.com/cloudflare/workers-oauth-provider/ [2]: https://tools.simonwillison.net/ https://tools.simonwillison.net/ [3]: https://steveklabnik.com/writing/a-tale-of-two-claudes/ https://steveklabnik.com/writing/a-tale-of-two-claudes/ [4]: https://antirez.com/news/153 https://antirez.com/news/153
andnand 1y ago
Whats your workflow? Ive been playing with Claude Code for personal use. Usually new projects for experimentation. We have Copilot licenses through work so I've been playing around with VS Code agent mode for the last week. Usually using 3.5, 3.7 Sonnet or 04-mini. This is in a large Go project. Its been abysmal at everything other than tests. I've been trying to figure out if I'm just using the tooling wrong but I feel like I've tried all the "best practices" currently. Contexts, switching models for planning and coding, rules, better prompting. Nothings worked so far.
- polskibus 1y ago
  My experiments with copilot and Claude desktop via mcp on the same codebase suggest that copilot is trimming the context much more than desktop. Using the same model the outputs are just less informed.
- SparkyMcUnicorn 1y ago
  Switch to using Sonnet 4 (it's available in VS Code Insiders for me at least). I'm not 100% sure but a Github org admin and/or you might need to enable this model in the Github web interface. Write good base instructions for your agent[0][1] and keep them up to date. Have your agent help you write and critique it. Start tasks by planning with your agent (e.g. "do not write any code."), and have your agent propose 2-3 ways to implement what you want. Jumping straight into something with a big prompt is hit or miss, especially with increased task complexity. Planning also gives your agent a chance to read and understand the context/files/code involved. Apologies if I'm giving you info you're already aware of. [0] https://code.visualstudio.com/docs/copilot/copilot-customization#_custom-instructions https://code.visualstudio.com/docs/copilot/copilot-customiza... [1] Claude Code `/init`
  - andnand 1y ago
    This is exactly what I was looking for. Thanks! Im trying to give these tools a fair shot before I judge them. Ive had success with detailed prompts and letting the agent jump straight in when working on small/new projects. Ill give more planning prompts a shot. Do you change models between planning and implementation? I've seen that recommended but it's been hard to judge if that's made a difference.
namaria 1y ago
> "Today is the worst day you will have with this technology for the rest of your life." Why do we trust corporations to keep making things better all of a sudden? The most jarring effect of this hype cycle is that all appear to refers to some imaginary set of corporate entities.
sitkack 1y ago
> Even when half my day is meetings, you wouldn't be able to tell from my git history. Your employer, if it is not you, will now expect this level of output.

GardenLetter27 1y ago

I trust it more with Rust than Python tbh, because with Python you need to make sure it runs every code path as the static analysis isn't as good as clippy + rust-analyzer.

diggan 1y ago
I agree, had more luck with various models writing Rust than Python, but only in the case where they have tools available so one way or another it can run `cargo check` and see the nice errors, otherwise it's pretty equal between the two. I think the excellent error messages in Rust also help as much humans as it does LLMs, but some of the weaker models get misdirected by some of the "helpful" tips, like some error message suggest "Why don't you try .clone here?" when the actual way to address the issue was something else.
redman25 1y ago
That's true typed languages seem to handle the slop better. One thing I've noticed specifically with rust is that agents tend to overcomplicate things though. They tend to start digging into the gnarlier bits of the language much quicker than they probably need to.

km144 1y ago

> Is there more we can add to the AI conventions markdown in the repo to guide the Agent to make fewer mistaken assumptions? Forgive my ignorance, but is this just a file you're adding to the context of every agent turn or this a formal convention in the VS code copilot agent? And I'm curious if there's any resources you used to determine the structure of that document or if it was just a refinement over time based on mistakes the AI was repeating?

jnwatson 1y ago
I just finished writing one. It is essentially the onboarding doc for your project. It is the same stuff you'd tell a new developer on your team: here are the design docs, here are the tools, the code, and this is how you build and test, and here are the parts you might get hung up on. In hindsight, it is the doc I should have already written.

apwell23 1y ago

> you wouldn't be able to tell from my git history. I can easily tell from git history which commits were heavily AI generated

ajdidbdbsgs 1y ago

> Last night I had a file with 38 mypy errors Fixing type checker errors should be one the least time consuming things you do. This was previously consuming a lot of your time? A lot of the AI discourse would be more effective if we could all see the actual work one another is doing with it (similar to the cloudflare post).

diggan 1y ago
> AI discourse would be more effective if we could all see the actual work one another is doing with it Yes, this is a frequent problem both here and everywhere else. The discussions need to include things like exact model version, inference parameters, what system prompt you used, what user prompt, what code you gave it, what exactly it replied and so much more details, as currently almost every comment is "Well, I used Sonnet last week and it worked great" without any details. Not to mention discussions around local models missing basic stuff like what quantization (if any) and what hardware you're running it on. People just write out "Wow fast model" or stuff like that, and call it a day. Although I understand why, every comment be huge if everyone always add sufficient context. I don't know the solution to this, but it does frustrate me.
- square_usual 1y ago
  There's many examples of exactly what you're asking for, such as Kenton Varda's Cloudlfare oauth provider [1] and Simon Willison's tools [2]. I see a new blog post like this with detailed explanations of what they did pretty frequently, like Steve Klabnik's recent post [3], which while it isn't as detailed has a lot of very concrete facts. There's even more blog posts from prominent devs like antirez who talk about other things they're doing with AI like rubber ducking [4], if you're curious about how some people who say "I used Sonnet last week and it was great" are working, because not everyone uses it to write code - I personally don't because I care a lot about code style. [1]: https://github.com/cloudflare/workers-oauth-provider/ https://github.com/cloudflare/workers-oauth-provider/ [2]: https://tools.simonwillison.net/ https://tools.simonwillison.net/ [3]: https://steveklabnik.com/writing/a-tale-of-two-claudes/ https://steveklabnik.com/writing/a-tale-of-two-claudes/ [4]: https://antirez.com/news/153 https://antirez.com/news/153

andnand 1y ago

Whats your workflow? Ive been playing with Claude Code for personal use. Usually new projects for experimentation. We have Copilot licenses through work so I've been playing around with VS Code agent mode for the last week. Usually using 3.5, 3.7 Sonnet or 04-mini. This is in a large Go project. Its been abysmal at everything other than tests. I've been trying to figure out if I'm just using the tooling wrong but I feel like I've tried all the "best practices" currently. Contexts, switching models for planning and coding, rules, better prompting. Nothings worked so far.

polskibus 1y ago
My experiments with copilot and Claude desktop via mcp on the same codebase suggest that copilot is trimming the context much more than desktop. Using the same model the outputs are just less informed.
SparkyMcUnicorn 1y ago
Switch to using Sonnet 4 (it's available in VS Code Insiders for me at least). I'm not 100% sure but a Github org admin and/or you might need to enable this model in the Github web interface. Write good base instructions for your agent[0][1] and keep them up to date. Have your agent help you write and critique it. Start tasks by planning with your agent (e.g. "do not write any code."), and have your agent propose 2-3 ways to implement what you want. Jumping straight into something with a big prompt is hit or miss, especially with increased task complexity. Planning also gives your agent a chance to read and understand the context/files/code involved. Apologies if I'm giving you info you're already aware of. [0] https://code.visualstudio.com/docs/copilot/copilot-customization#_custom-instructions https://code.visualstudio.com/docs/copilot/copilot-customiza... [1] Claude Code `/init`
- andnand 1y ago
  This is exactly what I was looking for. Thanks! Im trying to give these tools a fair shot before I judge them. Ive had success with detailed prompts and letting the agent jump straight in when working on small/new projects. Ill give more planning prompts a shot. Do you change models between planning and implementation? I've seen that recommended but it's been hard to judge if that's made a difference.

namaria 1y ago

> "Today is the worst day you will have with this technology for the rest of your life." Why do we trust corporations to keep making things better all of a sudden? The most jarring effect of this hype cycle is that all appear to refers to some imaginary set of corporate entities.

sitkack 1y ago

> Even when half my day is meetings, you wouldn't be able to tell from my git history. Your employer, if it is not you, will now expect this level of output.