5 ms·

I'm so happy someone else says this, because I'm doing exactly the same. I tried to use agent mode in vs code and the output was still bad. You read simple thin

I'm so happy someone else says this, because I'm doing exactly the same. I tried to use agent mode in vs code and the output was still bad. You read simple things like: "We use it to write tests". I gave it a very simple repository, said to write tests, and the result wasn't usable at all. Really wonder if I'm doing it wrong.

embedding-shape 8mo ago
You didn't actually just say "write tests" though right? What was the actual prompt you used? I feel like that matters more than the tooling at this point. I can't really understand letting LLMs decide what to test or not, they seem to completely miss the boat when it comes to testing. Half of them are useless because they duplicate what they test, and the other half doesn't test what they should be testing. So many shortcuts, and LLMs require A LOT of hand-holding when writing tests, more so than other code I'd wager.
- Balinares 8mo ago
  There are a lot of comments on HN and other places breathlessly gushing about agents totally doing everything end to end, so I couldn't blame someone new to this space for naively assuming that agents would be able to handle a well-bounded problem such as test coverage reasonably well.
  - embedding-shape 8mo ago
    > naively assuming that agents would be able to handle a well-bounded problem such as test coverage reasonably well. We haven't figured out a way for humans to do that well :P I still see people arguing about "80% test coverage is obviously better than 70%" and similar dumb sentiments that completely misses the point. But agree with the first part, LLMs are massively oversold and it's hard to blame users for believing them. Tempered expectations as always win.
- prettygood 8mo ago
  No, that was an exaggeration. The prompt was decent. I explained the point of the repository, that I wanted full coverage with tests, that it could keep going until it worked. Maybe that was still not enough. With how others talk about it, I must be missing something.
  - embedding-shape 8mo ago
    For tests, you need to be precise about what it should test, how it should test it, and what the assertions should be, otherwise you'll mostly get trash, they're exceptionally horrible at writing tests. Which makes sense, most programmers are too, but given the importance of correct tests, it's probably the part that needs to most human handholding right now.
agumonkey 8mo ago
you need to write a test suite to check his test generation (soft /s)
torben-friis 8mo ago
I’m not particularly proAI but I struggle with the mentality some engineers seem to apply to trying. If you read someone say “I don’t know what’s the big deal with vim, I ran it and pressed some keys and it didn’t write text at all” they’d be mocked for it. But with these tools there seems to be an attitude of “if I don’t get results straight away it’s bad”. Why the difference?
- Macha 8mo ago
  There isn't a bunch of managers metaphorically asking people if they're using vim enough, and not so many blog posts proclaiming vim as the only future for building software
  - torben-friis 8mo ago
    I’d argue that, if we accept that AI is relevant enough to at least be worth checking, then dismissing it with minimal effort is just as bad as mindlessly hyping the tech.
  - dist-epoch 8mo ago
    You must be new here. "I use vim between", "you don't use vim, you use Visual Studio, your opinion doesn't count" is a thing in programming circles.
    - Macha 8mo ago
      Internet commenters, sure. It never broke into the workplace like measuring AI use among your employees. Nobody's asked me about how I've used vim keybinds to improve the company's growth in a performance review.
- galaxyLogic 8mo ago
  Well one could say that since it's AI, AI should be able to tell us what we're doing wrong. No? AI is supposed to make our work easier.
  - 8mo ago
sixtyj 8mo ago
No, you have similar experience as a lot of people have. LLMs just fail (hallucinate) in less known fields of expertise. Funny: Today I have asked Claude to give me syntax how to run Claude Code. And its answer was totally wrong :) So you go to documentation… and its parts are obsolete as well. LLM development is in style “move fast and break things”. So in few years there will be so many repos with gibberish code because “everybody is coder now” even basketball players or taxi drivers (no offense, ofc, just an example). It is like giving F1 car to me :)
tasuki 8mo ago
> I gave it a very simple repository, said to write tests, and the result wasn't usable at all. Really wonder if I'm doing it wrong. I think so. The humans should be writing the spec. The AI can then (try to) make the tests pass.
threecheese 8mo ago
“Write tests“ may not be enough; provide it with a test harness, and instruct it to “write tests until they pass “. Next would be “your feature isn’t complete without N% coverage”. These require the ‘agentic’ piece, which is at its simplest some prompts run in a loop until an exit condition is met.

embedding-shape 8mo ago

You didn't actually just say "write tests" though right? What was the actual prompt you used? I feel like that matters more than the tooling at this point. I can't really understand letting LLMs decide what to test or not, they seem to completely miss the boat when it comes to testing. Half of them are useless because they duplicate what they test, and the other half doesn't test what they should be testing. So many shortcuts, and LLMs require A LOT of hand-holding when writing tests, more so than other code I'd wager.

Balinares 8mo ago
There are a lot of comments on HN and other places breathlessly gushing about agents totally doing everything end to end, so I couldn't blame someone new to this space for naively assuming that agents would be able to handle a well-bounded problem such as test coverage reasonably well.
- embedding-shape 8mo ago
  > naively assuming that agents would be able to handle a well-bounded problem such as test coverage reasonably well. We haven't figured out a way for humans to do that well :P I still see people arguing about "80% test coverage is obviously better than 70%" and similar dumb sentiments that completely misses the point. But agree with the first part, LLMs are massively oversold and it's hard to blame users for believing them. Tempered expectations as always win.
prettygood 8mo ago
No, that was an exaggeration. The prompt was decent. I explained the point of the repository, that I wanted full coverage with tests, that it could keep going until it worked. Maybe that was still not enough. With how others talk about it, I must be missing something.
- embedding-shape 8mo ago
  For tests, you need to be precise about what it should test, how it should test it, and what the assertions should be, otherwise you'll mostly get trash, they're exceptionally horrible at writing tests. Which makes sense, most programmers are too, but given the importance of correct tests, it's probably the part that needs to most human handholding right now.

agumonkey 8mo ago

you need to write a test suite to check his test generation (soft /s)

torben-friis 8mo ago

I’m not particularly proAI but I struggle with the mentality some engineers seem to apply to trying. If you read someone say “I don’t know what’s the big deal with vim, I ran it and pressed some keys and it didn’t write text at all” they’d be mocked for it. But with these tools there seems to be an attitude of “if I don’t get results straight away it’s bad”. Why the difference?

Macha 8mo ago
There isn't a bunch of managers metaphorically asking people if they're using vim enough, and not so many blog posts proclaiming vim as the only future for building software
- torben-friis 8mo ago
  I’d argue that, if we accept that AI is relevant enough to at least be worth checking, then dismissing it with minimal effort is just as bad as mindlessly hyping the tech.
- dist-epoch 8mo ago
  You must be new here. "I use vim between", "you don't use vim, you use Visual Studio, your opinion doesn't count" is a thing in programming circles.
  - Macha 8mo ago
    Internet commenters, sure. It never broke into the workplace like measuring AI use among your employees. Nobody's asked me about how I've used vim keybinds to improve the company's growth in a performance review.
galaxyLogic 8mo ago
Well one could say that since it's AI, AI should be able to tell us what we're doing wrong. No? AI is supposed to make our work easier.
- 8mo ago

sixtyj 8mo ago

No, you have similar experience as a lot of people have. LLMs just fail (hallucinate) in less known fields of expertise. Funny: Today I have asked Claude to give me syntax how to run Claude Code. And its answer was totally wrong :) So you go to documentation… and its parts are obsolete as well. LLM development is in style “move fast and break things”. So in few years there will be so many repos with gibberish code because “everybody is coder now” even basketball players or taxi drivers (no offense, ofc, just an example). It is like giving F1 car to me :)

tasuki 8mo ago

> I gave it a very simple repository, said to write tests, and the result wasn't usable at all. Really wonder if I'm doing it wrong. I think so. The humans should be writing the spec. The AI can then (try to) make the tests pass.

threecheese 8mo ago

“Write tests“ may not be enough; provide it with a test harness, and instruct it to “write tests until they pass “. Next would be “your feature isn’t complete without N% coverage”. These require the ‘agentic’ piece, which is at its simplest some prompts run in a loop until an exit condition is met.