5 ms·
Indeed, much of the scariness is how fearlessly and confidently it writes them with little regard to their actual usefulness or value. When I find it adding a l
by aroman 2mo ago
Indeed, much of the scariness is how fearlessly and confidently it writes them with little regard to their actual usefulness or value. When I find it adding a lot of tests, I often say something like: "audit each test carefully, and consider whether the test is testing a meaningful boundary or is more ceremonial. delete low-value tests and add new tests to cover meaningful boundaries not exercised by the gaps you identify". Without fail, this always produces some decent results.
Having said that, in truth, I almost never read the unit tests. Before AI, we had almost none (see: several person game studio) so the tradeoff is not "AI-generated tests" vs "human written ones", it's whether we have tests at all. So, I take them for what they're worth - not much - but if it catches an extra regression before it ships every now and then, it was worth it for the price (~free).
- pkulak 2mo agoI notice that the tests are very often for its own benefit. Like, you’ll ask it to stop doing something, literally to just remove code, and it will write a test to verify the behavior is not there. I can’t imagine a more useless test, but I get that it wants to verify that it actually made the change. I just delete the test when it’s done.
- stpedgwdgfhgdd 2mo agoLots of unit test do not add value in the traditional sense, but they do help the llm to understand the code.