5 ms·

> How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage? This is such a salient question. Sometimes (definitely not al

> How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage? This is such a salient question. Sometimes (definitely not always) the test suites produced by LLMs are so trivial it's scary. Coverage can be an illusion for sure.

coldtea 2mo ago
I have been seeing such trivial shit getting submitted and deleting them from the test suites, but it's disheartening to see that the devs didn't have the better taste to see those LLM "tests" are so trivial they border on noop before submitting them...
- westurner 2mo ago
  What new or existing metric would better assess test coverage than line coverage and branch coverage? I wrote a tool called tert - I guess it's called an agent harness now - to run various test runners and log test output and coverage output to disk. FWIU stripping spaces from JSON does save tokens. It seems like feeding coverage lines-missing maps into the prompt results in better output, better LLM-authored tests. "Refactor these tests for maintainability and coverage. Use fixtures, mocks, and parametrization"
  - coldtea 2mo ago
    >What new or existing metric would better assess test coverage than line coverage and branch coverage? Substance coverage - testing the actual logic, edge cases, etc. Not mere lines.
  - what 2mo ago
    You can write tests that cover every line/branch without actually verifying the code does what it’s supposed to. Relying on (unreviewed?) LLM written tests that provide coverage may not be a great idea.
    - westurner 2mo ago
      So, a project can have 100% test coverage (the tests run each line or branch at least once without any errors or failures) but fail acceptance tests. If there is prompt insufficiency, there is probably acceptance test insufficiency. A more assuming agent could automatically develop a plan that includes presumptive acceptance tests and request feedback before spending tokens
- mkw5053 2mo ago
  We try to use types as much as possible. And if we can't, then lint rules, etc. Basically, get everything as far left in the SDLC as we can. And, if/where we need tests, we write the source so they are few, high value, and complementary. Like actual unit tests, not complex with stuff like mocks just to generate trivial coverage.

coldtea 2mo ago

I have been seeing such trivial shit getting submitted and deleting them from the test suites, but it's disheartening to see that the devs didn't have the better taste to see those LLM "tests" are so trivial they border on noop before submitting them...

westurner 2mo ago
What new or existing metric would better assess test coverage than line coverage and branch coverage? I wrote a tool called tert - I guess it's called an agent harness now - to run various test runners and log test output and coverage output to disk. FWIU stripping spaces from JSON does save tokens. It seems like feeding coverage lines-missing maps into the prompt results in better output, better LLM-authored tests. "Refactor these tests for maintainability and coverage. Use fixtures, mocks, and parametrization"
- coldtea 2mo ago
  >What new or existing metric would better assess test coverage than line coverage and branch coverage? Substance coverage - testing the actual logic, edge cases, etc. Not mere lines.
- what 2mo ago
  You can write tests that cover every line/branch without actually verifying the code does what it’s supposed to. Relying on (unreviewed?) LLM written tests that provide coverage may not be a great idea.
  - westurner 2mo ago
    So, a project can have 100% test coverage (the tests run each line or branch at least once without any errors or failures) but fail acceptance tests. If there is prompt insufficiency, there is probably acceptance test insufficiency. A more assuming agent could automatically develop a plan that includes presumptive acceptance tests and request feedback before spending tokens
mkw5053 2mo ago
We try to use types as much as possible. And if we can't, then lint rules, etc. Basically, get everything as far left in the SDLC as we can. And, if/where we need tests, we write the source so they are few, high value, and complementary. Like actual unit tests, not complex with stuff like mocks just to generate trivial coverage.