8 ms·
Using LLMs to enhance our testing practices
- iambateman 2y agoI did this for Laravel a few months ago and it’s great. It’s basically the same as the article describes, and it has definitely increased the number of tests I write. Happy to open source if anyone is interested.
- frays 2y agoI'd certainly be interested to read more about your experience!
- deleted 2y ago[deleted]
- simonw 2y agoIf you add "white-space: pre-wrap" to the elements containing those prompt examples you'll avoid the horizontal scrollbar (which I'm getting even on desktop) and make them easier to read.
- johnjwang 2y agoThanks for the suggestion -- I'll take a look into adding this!
- satisfice 2y agoLike nearly all the articles about AI doing "testing" or any other skilled activity, the last part of it admits that it is an unreliable method. What I don't see in this article-- which I suspect is because they haven't done any-- is any description of a competent and reasonably complete testing process of this method of writing "tests." What they probably did is to try this, feel good about it (because testing is not their passion, so they are easily impressed), and then mark it off in their minds as a solved problem. The retort by AI fanboys is always "humans are unreliable, too." Yes, they are. But they have other important qualities: accountability, humility, legibility, and the ability to learn experientially as well as conceptually. LLM's are good at instantiating typical or normal patterns (based on its training data). Skilled testing cannot be limited to typicality, although that's a start. What I'd say is that this is an interesting idea that has an important hazard associated with it: complacency on the part of the developer who uses this method, which turns things that COULD be missed by a skilled tester into things that are GUARANTEED to be missed.
- johnjwang 2y agoAuthor here: Yes, there are certain functions where writing good tests will be difficult for an LLM, but in my experience I've found that the majority of functions that I write don't need anything out of the ordinary and are relatively straightforward. Using LLMs allows us to have much higher coverage than if we didn't use it. To me and our engineering team, this is a pretty good thing because in the time prioritization matrix, if I can get a higher quality code base with higher test coverage with minimal extra work, I will definitely take it (and in fact it's something I encourage our engineering teams to do). Most of the base tests that we use were created originally by some of our best engineers. The patterns they developed are used throughout our code base and LLMs can take these and make our code very consistent, which I also view as a plus. re: Complacency: We actually haven't found this to be the case. In fact, we've seen more tests being written with this method. Just think about how much easier it is to review a PR and make edits vs write a PR. You can actually spend your time enforcing higher quality tests because you don't have to do most of the boilerplate for writing a test.
- youoy 2y agoI would say that the complacency part is identifying good test with good coverage. I agree that writing test is one of the best use cases for LLMs, and it definitely saves engineers a lot of time. But if you follow them to blindly it is easy to get carried away by how easy it is to write tests that focus on coverage instead of actually testing more quality things. Which is what the previous comment was pointing at: > which turns things that COULD be missed by a skilled tester into things that are GUARANTEED to be missed.
- satisfice 2y agoHave you systematically tested this approach? It sounds like you are reporting on your good vibes. Your writing is strictly anecdotal. I’ve been working with AI, too. I see what I’m guessing is the same unreliability that you admit in the last part of your article. For some reason, you are sanguine about it, whereas I see it as a serious problem. You say you aren’t complacent, but your words don’t seem to address the complacency issue. “More tests” does not mean better testing, or even good enough testing. Google “automation bias” and tell me what policies and procedures or training is in place to avoid it.
- renegade-otter 2y agoIn every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test code than the actual code (probably normal). Also, I feel like people would be inclined to go with whatever the LLM gives them, as opposed to really sitting down and thinking about all the unhappy paths and edge cases of UX. Using an autocomplete to "bang it out" seems foolish.
- swatcoder 2y agoFully agreed. It's bad enough when human team members are submitting useless, brittle tests with their PR's just to satisfy some org pressure to write them. The lazy ones provide a false sense of security even though they neglect critical scenarios, the unstable ones undermine trust in the test output because they intermittently raise false negatives that nobody has time to debug, and the pointless ones do nothing but reify architecture so it becomes too laborious to refactor anything. As contextually aware generators, there are doubtless good uses for LLM's in test developement, but (as with many other domains) they threaten to amplify an already troubling problem with low-quality, high-volume content spam.
- viraptor 2y agoPretty much this and I prefer the opposite. "Here's the new test case from me, make the code pass it" is a decent workflow with Aider. I get that occasionally there are some really trivial but important tests that take time and would be nice to automate. But that's a minority in my experience.
- skissane 2y ago> "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Are there ways we can measure this? One idea that I’ve had, is collect code coverage separately for each test. If a test isn’t covering any unique code or branches, maybe it is superfluous - although not necessarily, it can make sense to separately test all the boundary conditions of a function, even if doing so doesn’t hit any unique branches. Maybe prefer a smaller test which covers the same code to a bigger one. However, sometimes if a test is very DRY, it can be more brittle, since it can be non-obvious how to update it to handle a code change. A repetitive test, updating it can be laborious, but at least reasonably obvious how to do so. Could an LLM evaluate test quality, if you give it a prompt containing some expert advice on good and bad testing practices?
- apwell23 2y agoi would love to used to use it change code in ways that compiles and see if test fails. Coverage metric sometimes doesn't really tell you if some piece of code is covered or not.
- sesm 2y agoCoverage metric can tell if lines of code were executed, but they can't tell if execution result was checked.
- taberiand 2y agoI believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea
- rgmerk 2y agoLLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.
- taberiand 2y agoSet aside LLMs, why does mutation testing need to be fast? It would be fine to have mutation tests run slowly, out-of-band of the main CI pipeline. They aren't mission critical, they're smoke tests for your unit tests. Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.
- nazgul17 2y agoShould we not, instead, write tests ourselves and have LLMs write the code to make them pass?
- jayd16 2y agoJust ask it to do both.
- sdesol 2y agoAnd remember to always challenge the response with both the same and different models. No joke. Just continue the conversation for the example in the blog and ask the LLM "Do you see anything wrong with the code?" and it will spit out "Yes" and explain why.
- mastersummoner 2y agoI actually tested Claude Sonnet to see how it would fare at writing a test suite for a background worker. My previous experience was with some version of GPT via Copilot, and it was... not good. I was, however, extremely impressed with Claude this time around. Not only did it do a great job off the bat, but it taught me some techniques and tricks available in the language/framework (Ruby, Rspec) which I wasn't familiar with. I'm certain that it helped having a decent prompt, asking it to consider all the potential user paths and edge cases, and also having a very good understanding of the code myself. Still, this was the first time for me I could honestly say that an LLM actually saved me time as a developer.
- shadowmanifold 2y agoThis latest update to Sonnet is super impressive. We are really already past the point of being able to discuss these matters though in large groups. The herd speaks as if all LLMs on all programming languages are basically the same. It is an absurdity. Talking to the herd is mostly for entertainment at this point. If I actually want to learn something, I will ask Sonnet.
- throwa5456435 2y agoAll this makes me think making software engineers redundant is really the "killer app" of LLM's. This is where the AI labs are spending most of the effort - its the best marketing after all for their product - fear sells better than greed (loss aversion) making engineers notice and unable to dismiss it. Despite some of the comments on this thread, despite it not wanting to be true, I must admit LLM's are impressive. Software engineers and ML specialists have finally invented the thing which disrupts their own jobs substantially either via large reduction in hours and/or reduction in staff. As the hours a software engineer spends coding diminishes by large factors so too especially in this economy will hours spent required paying an engineer will fall up to the point where anyone can create code and learn from an LLM as you have just done. Once everybody is special, no one is and fundamentally employment, and value of things created from software, comes from scarcity just like everything else in our current system. I think there's probably only a few years left where software engineers are around - or at least seen as a large part of an organization with large teams, etc. Yes AI software will have bugs, and yes it won't be perfect but you can get away with just one or two for a whole org to fix the odd blip of an LLM. It feels like people are picking on minor things at this point, which while true, for a business those costs are "meh" while the gains of removing engineers are substantial. I want to be wrong; but every time I see someone "learning from LLM's", saving lots of time doing stuff, saving 100's of hours, etc I think its only 2-3 years in and already its come this far.
- mkleczek 2y agoI am very sceptical of LLM (or any AI) code generation usefulness and it does not really have anything to do with AI itself. In the past I've been involved in several projects deeply using MDA (Model Driven Architecture) techniques which used various code generation methods to develop software. One of the main obstacles was the problem of maintaining the generated code. IOW: how should we treat generated code? If we treat it in the same way as code produced by humans (ie. we maintain it) then the maintenance cost grows (super-linearly) with the amount of code we generate. To make matters worse for LLM: since the code it generates is buggy it means we have more buggy code to maintain. Code review is not the answer because code review power in finding bugs is very weak. This is unlike compilers (that also generate code) because we don't maintain code generated by compilers - we regenerate it anytime we need. The fundamental issue is: for a given set of requirements the goal is to produce less code, not more. _Any_ code generation (however smart it might be) goes against this goal. EDIT: typos
- mvdtnz 2y agoYou should NEVER modify generated code. All of our generated code is pretended with a big comment that says "GENERATED CODE DO NOT MODIFY. This code could be regenerated at any time and any changes will be lost." If you need to change behaviour of generated code you need to change your generator to provide the right hooks. Obviously none of this applies to "AI" generated code because the "AI" generator is not deterministic and will hallucinate different bugs from run to run. You must treat "AI" generated code as if it was written by the dumbest person you've ever worked with.
- 2y ago
- deleted 2y ago[deleted]
- DeathArrow 2y agoIt's hard to generate tests for typical C# code. Or for any context where you have external dependencies. If you have injected services in your current service, the LLM doesn't know anything about those so it makes poor guesses. You have to bring those in context, so they can be mocked properly. You end up spending a lot of time guiding the LLM, so it's not measurably faster than writing test by hand. I want my prompt to be: "write unit tests for XYZ method" without having to accurately describe it the prompt what the method does, how it does it and why it does it. Writing too many details in the prompt takes the same time as writing the code myself. Github Copilot should be better since it's supposed to have access to you entire code base. But somehow it doesn't look at dependencies and it just uses the knowledge of the codebase for stylistic purposes. It's probably my fault, there are for sure better ways to use LLMs for code, but I am probably not the only one who struggles.
- deleted 2y ago[deleted]
- tsv_ 2y agoEach time a new LLM version comes out, I give it another try at generating tests. However, even with the latest models, tailored GPTs, and well-crafted prompts with code examples, the same issues keep surfacing: - The models often create several tests within the same equivalence class, which barely expands test coverage - They either skip parameterization, creating multiple redundant tests, or go overboard with 5+ parameters that make tests hard to read and maintain - The model seems focused on "writing a test at any cost" often resorting to excessive mocking or monkey-patching without much thought - The models don’t leverage existing helper functions or classes in the project, requiring me to upload the whole project context each time or customize GPTs for every individual project Given these limitations, I primarily use LLMs for refactoring tests where IDE isn’t as efficient: - Extracting repetitive code in tests into helpers or fixtures - Merging multiple tests into a single parameterized test - Breaking up overly complex parameterized tests for readability - Renaming tests to maintain a consistent style across a module, without getting stuck on names
- deeviant 2y agoAll of the points you raise I find common in human written tests.
- dfilppi 2y ago[dead]
- gengstrand 2y agoI went with a more clinical approach and used models that were available a half year ago but I also was interested in using LLMs to write unit tests. You can learn the details of that experiment at https://www.infoq.com/articles/llm-productivity-experiment/ https://www.infoq.com/articles/llm-productivity-experiment/ but the net of what I found was that LLMs improve developer productivity in the form of unit test creation but only marginally. Perhaps I find myself a bit skeptical on the claims from that Assembled blog on significant improvement.