7 ms·

Hypothesis, Antithesis, synthesis

285 points by alpaylan 6mo ago

DRMacIver 6mo ago
Post author here btw, happy to take questions, whether they're about Hegel in particular, property-based testing in general, or some variant on "WTF do you mean you wrote rust bindings to a python library?"
- anentropic 6mo ago
  TBH reading the first few words of that section I was definitely expecting it to continue "so we used Claude to rewrite Hypothesis in Rust..." so that was quite a surprise!
  - DRMacIver 6mo ago
    It's on the agenda! We definitely want to rewrite the Hegel core server in rust, but not as much as we wanted to get it working well first. My personal hope is that we can port most of the Hypothesis test suite to hegel-rust, then point Claude at all the relevant code and tell it to write us a hegel-core in rust with that as its test harness. Liam thinks this isn't going to work, I think it's like... 90% likely to get us close enough to working that we can carry it over the finish line. It's not a small project though. There are a lot of fiddly bits in Hypothesis, and the last time I tried to get Claude to port it to Rust the result was better than I expected but still not good enough to use.
    - tybug 6mo ago
      To put it on the record: my position is current models can't get us there, and neither can the next iteration of models, but in two model iterations this will be worth doing. There's a lot of fiddly details in Hypothesis that are critical to get right. You can get a plausible 80% port with agents today but find they've structured it in a way to make it impossible to get to 100%.
- Chinjut 6mo ago
  You mention in the post that there are design differences between Hegel/Hypothesis and QuickCheck, partly due to attitude differences between Python/non-Haskell programmers and Haskell programmers. As someone coming from the Haskell world (though by no means considering Haskell a perfect language), could you expand on what kinds of differences these are?
  - DRMacIver 6mo ago
    So I think a short list of big API differences are something like: * Hypothesis/Hegel are very much focused on using test assertions rather than a single property that can be true or false. This naturally drives a style that is much more like "normal" testing, but also has the advantage that you can distinguish between different types of failing test. We don't go too hard on this, but both Hegel and Hypothesis will report multiple distinct failures if your test can fail in multiple ways. * Hegelothesis's data generation and how it interacts with testing is much more flexible and basically fully imperative. You can basically generate whatever data you like wherever in your test you like, freely interleaving data generation and test execution. * QuickCheck is very much type-first and explicit generators as an afterthought. I think this is mostly a mistake even in Haskell, but in languages where "just wrap your thing in a newtype and define a custom implementation for it" will get you a "did you just tell me to go fuck myself?" response, it's a nonstarter. Hygel is generator first, and you can get the default generator for a type if you want but it's mostly a convenience function with the assumption that you're going to want a real generator specification at some point soon. From an implementation point of view, and what enables the big conveniences, Hypothesis has a uniform underlying representation of test cases and does all its operations on them. This means you get: * Test caching (if you rerun a failing test, it will immediately fail in the same way with the previously shrunk example) * Validity guarantees on shrinking (your shrunk test case will always be ones your generators could have produced. It's a huge footgun in QuickCheck that you can shrink to an invalid test case) * Automatically improving the quality of your generators, never having to write your own shrinkers, and a whole bunch of other quality of life improvements that the universal representation lets us implement once and users don't have to care about. The validity thing in particular is a huge pain point for a lot of users of PBT, and is what drove a lot of the core Hypothesis model to make sure that this problem could never happen. The test caching is because I personally hated rerunning tests and not knowing whether it was just a coincidence that they were passing this time or that the test case had changed.
- peytongreen_dev 6mo ago
  [flagged]
- mullr 6mo ago
  Why would I use this over the existing Proptest library in Rust?
  - DRMacIver 6mo ago
    Answered this over here: https://news.ycombinator.com/item?id=47506274 https://news.ycombinator.com/item?id=47506274
- nz 6mo ago
  Not really a question. Just wanted to express my gratitude for Hypothesis. I use it regularly. A few years back, I had to build a semi-formally-verified fund and account management service, and used the state-based-testing of Hypothesis to validate its correctness. Cannot express how invaluable this little framework has been. A little while after that, I spoke to someone in the pharma-adjacent-space who was looking at Antithesis to validate their product. At the time, Antithesis (the company) told him that it was a bad fit. I suggested something akin to my previous approach (which did not include antithesis). No clue what they ended up doing, but it is nice to see that Hypothesis and Antithesis have finally joined forces.
  - DRMacIver 6mo ago
    You're very welcome! I'm glad it's been useful for you.
- iainmerrick 6mo ago
  One thing I'm curious about, which I couldn't figure out from a skim of your post, is whether the generated test inputs are random, sequential, or adversarial. IIRC there are fuzz testers that will analyze the branches of the code to look for edge cases that might break it -- that seems like something that would be wonderful to have in a property tester, but it also seems very difficult to do, especially in a language agnostic way. How long does it take to find breaking cases like "0/0" or "ß"? Do they pop up immediately, or does it only happen after hundreds or thousands of runs?
  - DRMacIver 6mo ago
    They're random but with a lot of tweaks to the distribution that makes weird edge cases pop up with fairly high probability, and with some degree of internal mutation, followed by shrinking to turn them into nice tidy test cases. In Python we do a little bit of code analysis to find interesting constants, but Hegel doesn't do that, it's just tuned to common edge cases. I think all the examples I had in the post are typically found in the first 100 test cases and reliably found in the first 1000, but I wouldn't swear that that's the case without double checking. We don't do any coverage-guidance in Hegel or Hypothesis, because for unit testing style workflows it's rarely worth it - it's very hard to do good coverage guidance in under like... 10k test runs at a minimum, 100k is more likely. You don't have enough time to get really good at exploring the state space, and you haven't hit the point where pure random testing has exhausted itself enough that you have to do something smarter to win. It's been a long-standing desire of mine to figure out a way to use coverage to do better even on short runs, and there are some kinda neat things you can do with it, but we've not found anything really compelling.
tybug 6mo ago
As possibly the one community on earth where it's actually better to post the code than the blog post: TL;DR this is a universal property-based testing protocol (https://github.com/hegeldev/hegel-core https://github.com/hegeldev/hegel-core) and family of libraries (https://github.com/hegeldev/hegel-rust https://github.com/hegeldev/hegel-rust, more to come later). I've talked with lots of people in the PBT world who have always seen something like this as the end goal of the PBT ecosystem. It seemed like a thing that would happen eventually, someone just had to do it. I'm super excited to actually be doing it and bringing great PBT to every and any language. It doesn't hurt that this is coming right as great PBT in every language is suddenly a lot more important thanks to AI code!
hugeBirb 6mo ago
Not that it matters at this point but the hegelian dialectic is not thesis, antithesis and synthesis. Usually attributed to Hegel but as I understand it he actually pushed back on this mechanical view of it all and his views on these transitory states was much more nuanced.
- DRMacIver 6mo ago
  Conversation with Will (Antithesis CEO) a couple months ago, heavily paraphrased: Will: "Apparently Hegel actually hated the whole Hegelian dialectic and it's falsely attributed to him." Me: "Oh, hm. But the name is funny and I'm attached to it now. How much of a problem is that?" Will: "Well someone will definitely complain about it on hacker news." Me: "That's true. Is that a problem?" Will: "No, probably not." (Which is to say: You're entirely right. But we thought the name was funny so we kept it. Sorry for the philosophical inaccuracy)
  - wwilson 6mo ago
    If I had been wearing my fiendish CEO hat at the time, I might have even said something like: "somebody pointing this out will be a great way to jumpstart discussion in the comments." One of the evilest tricks in marketing to developers is to ensure your post contains one small inaccuracy so somebody gets nerdsniped... not that I have ever done that.
    - 1-more 6mo ago
      A sort of broadening of Cunningham's Law (the fastest way to get an answer online is not by posting the question, but by posting the wrong answer—very true in my experience). If there's no issue of fact at hand, then you end up getting some engagement about the intentional malapropism/misattribution/mistake/whatever and then the forum rules tend to herd participants back to discussing the matter at hand: your company. https://meta.wikimedia.org/wiki/Cunningham%27s_Law https://meta.wikimedia.org/wiki/Cunningham%27s_Law
    - jpadkins 6mo ago
      Seth Godin made the case that its more important for people to make remarks than to be favorable (https://en.wikipedia.org/wiki/Purple_Cow:_Transform_Your_Business_by_Being_Remarkable https://en.wikipedia.org/wiki/Purple_Cow:_Transform_Your_Bus...) Trump did this a lot with the legacy media in his first term. He would make inaccurate statements to the media on the topic he wanted to be in the spotlight, and the media would jump to "fact check" him. Guess what, now everyone is talking about illegal immigration, tariffs, or whatever subject Trump thought was to their advantage.
pron 6mo ago
> property-based testing is going to be a huge part of how we make AI-agent-based software development not go terribly. There's no doubt, I think, testing will remain important and possibly become more important with more AI use, and so better testing is helpful, PBT included. But the problem remains verifying that the tests actually test what they're supposed to. Mutation tests can allow agents to get good coverage with little human intervention, and PBT can make tests better and more readable. But still, people have to read them and understand them, and I suspect that many people who claim to generate thousands of LOC per day don't. And even if the tests were great and people carefully reviewed them, that's not enough to make sure things don't go terribly wrong. Anthropic's C compiler experiment didn't fail because of bad testing. Not only were the tests good, it took humans years to write the tests by hand, and the agents still failed to converge. I think good tests are a necessary condition for AI not generating terrible software, but we're clearly not yet at a point where they're a sufficient one. So "a huge part" - possibly, but there are other huge parts still missing.
- tybug 6mo ago
  I actually think there's another angle here where PBT helps, which wasn't explored in the blog post. That angle is legibility. How do you know your AI-written slop software is doing the right thing? One would normally read all the code. Bad news: that's not much less labor intensive as not using AI at all. But, if one has comprehensive property-based tests, they can instead read only the property-based tests to convince themselves the software is doing the right thing. By analogy: one doesn't need to see the machine-checked proof to know the claim is correct. One only needs to check the theorem statement is saying the right thing.
  - pron 6mo ago
    Right, I said that property based tests are easier to read, and that's good. But people still have to actually read them. Also, because they still work best at the "unit" level, to understand them, the people reading them need to know how all the units are connected (e.g. a single person cannot review even PBTs required for 10KLOC per day [1]). My point isn't so much about PBT, but about how we don't yet know just how much agents help write real software (and how to get the most help from them). [1]: I'm only using that number because Garry Tan, CEO of YC, claimed to generate 10K lines of text per day that he believes to be working code and developers working with AI agents know they can't be.
rdevilla 6mo ago
This is the first time in my HN membership where I was excited to read about the dialectic, only to be disappointed upon finding out the article is about Rust. PBT is for sure the future - which is apparently now? 10 years ago when I was talking about QuickCheck [0] all the JS and Ruby programmers in my city just looked at me like I had two heads. [0] https://github.com/ryandv/chesskell/blob/master/test/Test/Chess/BitboardSpec.hs#L134 https://github.com/ryandv/chesskell/blob/master/test/Test/Ch...
- DRMacIver 6mo ago
  TBF PBT has been the present in Python for a while now. 10 years ago might have been a little early (Hypothesis 1.0 came out 11 years ago this coming Thursday), but we had pretty wide adoption by year two and it's only been growing. It's just that the other languages have all lagged behind. It's by no means universally adopted, but it's not a weird rare thing that nobody has heard of.
lwhsiao 6mo ago
DRMacIver, can you comment on how this fits into the existing property-based testing ecosystems for various languages? E.g., if I use proptest in Rust, why would/should I switch to Hegel?
- DRMacIver 6mo ago
  The short answer to how it fits into existing ecosystems is... in competition I suppose. We've got a lot of respect for the people working on these libraries, but we think the Hypothesis-based approach is better than the various approaches people have adopted. I don't love that the natural languages for us to start with are ones where there are already pretty good property-based testing libraries whose toes we're stepping on, but it ended up being the right choice because those are the languages people care about writing correct software in, and also the ones we most want the tools in ourselves! I think right now if you're a happy proptest user it's probably not clear that you should switch to Hegel. I'd love to hear about people trying, but I can't hand on my heart say that it's clearly the correct thing for you to do given its early state, even though I believe it will eventually be. But roughly the things that I think are clearly better about the Hegel approach and why it might be worth trying Hegel if you're starting greenfield are: * Much better generator language than proptest (I really dislike proptest's choices here. This is partly personal aesthetic preferences, but I do think the explicitly constructed generators work better as an approach and I think this has been borne out in Hypothesis). Hegel has a lot of flexible tooling for generating the data you want. * Hegel gets you great shrinking out of the box which always respects the validity requirements of your data. If you've written a generator to always ensure something is true, that should also be true of your shrunk data. This is... only kindof true in proptest at best. It's not got quite as many footguns in this space as original quickcheck and its purely type-based shrinking, but you will often end up having to make a choice between shrinking that produces good results and shrinking that you're sure will give you valid data. * Hegel's test replay is much better than seed saving. If you have a failing test and you rerun it, it will almost immediately fail again in exactly the same way. With approaches that don't use the Hypothesis model, the best you can hope for is to save a random seed, then rerun shrinking from that failing example, which is a lot slower. There are probably a bunch of other quality of life improvements, but these are the things that have stood out to me when I've used proptest, and are in general the big contrast between the Hypothesis model and the more classic QuickCheck-derived ones.
skybrian 6mo ago
It isn't used by anyone besides me, but I wrote a property-testing library for Deno [1] that has a form of "sometimes" assertions (inspired by Antithesis) and uses "internal shrinking" (inspired by Hypothesis). But it's still a "blind" fuzzer and it would be nice to write one that gets feedback from code coverage somehow. Instead, you have to run code coverage yourself and figure out how to change test data generation to improve it. [1] https://jsr.io/@skybrian/repeat-test https://jsr.io/@skybrian/repeat-test
- chriswarbo 6mo ago
  > But it's still a "blind" fuzzer and it would be nice to write one that gets feedback from code coverage somehow There have been simplistic attempts at this, e.g. instead of performing 100 tests, just keep going as long as coverage increases. The Choice Gradient Sampling algorithm from https://arxiv.org/pdf/2203.00652 https://arxiv.org/pdf/2203.00652 feels like a nice way to steer generators in a more nuanced way. That paper uses it to avoid discards when rejection-sampling; but I have a feeling it could be repurposed to "reward" based on new coverage instead/as-well.
  - skybrian 6mo ago
    It's not like how it's done in that paper, but oddly enough I did end up implementing some conversions in both directions, from an array of choices to a JavaScript object and back again. https://jsr.io/@skybrian/repeat-test/doc/core/~/Domain https://jsr.io/@skybrian/repeat-test/doc/core/~/Domain
seamossfet 6mo ago
Oh my god, the rust developers are writing tests with Hegelian dialects.
sunshowers 6mo ago
Hi David, congratulations on the release! I'm excited to play around with Hypothesis's bitstream-based shrinking. As you're aware, prop_flat_map is a pain to deal with, and I'd love to replace some of my proptest-based tests with Hegel. I spent a little time looking at Hegel last week and it wasn't quite clear to me how I'd go about having something like a canonical generator for a type (similar to proptest's Arbitrary). I've found that to be very helpful while generating large structures to test something like serialization roundtripping against — in particular, the test-strategy library has derive macros that work very well for business logic types with, say, 10-15 enum variants each of which may have 0-10 subfields. I'm curious if that is supported today, or if you have plans to support this kind of composition in the future. edit: oh I completely missed the macro to derive DefaultGenerator! Whoops
- tybug 6mo ago
  Yep, `#[derive(DefaultGenerator)]` and `generators::default<T>()` are the right tools here. This is one of the areas we've dogfooded the least, so we'd definitely be happy to get feedback on any sharp corners here! I think `from_type` is one of Hypothesis's most powerful and ergonomic strategies, and that while we probably can't get quite to that level in rust, we can still get something that's pretty great.
  - sunshowers 6mo ago
    Thank you! I have some particularly annoying proptest-based tests that I'll try porting over to Hegel soon. (Thanks for writing the Claude skill to do this.)
    - DRMacIver 6mo ago
      Please let us know how it goes! As Liam says, the derive generator is not very well dogfooded at present. The claude skill is a bit better, but we've only been through a few iterations of using it and getting Claude to improve it, and porting from proptest is one of the less well tested areas (because we don't use proptest much ourselves). I expect all of this works, but I'd like to know ways that it works less well than it could. Or, you know, to bask in the glow of praise of it working perfectly if that turns out to be an option.
shrubby 6mo ago
In here just for the Hegel joke.
LoganDark 6mo ago
Using Python from other languages is terrible. I love this kind of testing but this implementation is not for me. I was so excited before learning it depends on Python.
- epage 6mo ago
  Yeah, having `cargo test` require another binary like `uv` is not idiomatic. 99% of the time, I should be able to walk up to a Rust project and run `cargo test` and it should just work.
  - LoganDark 6mo ago
    `uv` isn't even a Python interpreter. I prefer to build my systems without Python support.
keeganpoppen 6mo ago
love the hegel reference. i know hypothesis is awesome, and so im certain this is as well. this is no real complaint about the post, because it is a personal skill issue, but for someone who is more of a PL nerd than… most people on earth… i find Ruat code to be some of the hardest to actually parse in any meaningful way. even to get a sense of how the property defs were being set up, i kinda just took it at face value that it is cool… i guess it really comes down to personally not having much interest in rust compared to most newfangled languages out there. like i said, skipl issue, not a conplaint. wish you all the best, and lots of success!
ruuda 6mo ago
Property-based testing is nice, but making it coverage-driven is a game changer. It will explore code paths that naive random inputs will not trigger in a thousand years. In Rust this works very well with libFuzzer and the Arbitrary crate to derive the generators.
- wwilson 6mo ago
  [Disclaimer: Started Antithesis] If you run your Hegel tests in Antithesis, you get this for free (along with various sorts of “non-local” assertions, perfect reproducibility even for concurrent or distributed code, etc.). But yeah, not hard to hack together basic coverage guidance outside Antithesis. That works well for large classes of programs, just not a majority of them.
Weebs 6mo ago
"I thought this would be about Hegel... oh, wait"
ahf8Aithaex7Nai 6mo ago
To all you amateur Hegel enthusiasts out there: there is no synthesis in Hegel. Otherwise: Congratulations on the QuickCheck-style testing in Rust. At work, I’m always surprised that property-based testing is so little known and so rarely used outside of functional programming.
- DiscourseFan 6mo ago
  “Synthesis” is the purposiveness in the force of the Concept.
- dr_dshiv 6mo ago
  yeah where did that come from? It's like attributing the cartesian coordinate plane to Descartes -- when actually it was a textbook author who came up with the modern one...
- DRMacIver 6mo ago
  > To all you amateur Hegel enthusiasts out there: there is no synthesis in Hegel. Looks like the mods deleted the last long thread about this, so best not to relitigate, but short version: Yes, we know. We liked the name and thought it was funny so we kept it. > Otherwise: Congratulations on the QuickCheck-style testing in Rust. At work, I’m always surprised that property-based testing is so little known and so rarely used outside of functional programming. Actually, it's Hypothesis-style testing in Rust. There was already QuickCheck style. Property-based testing is in fact far more widely used in Python than in functional programming (probably not as a percentage of users, but in terms of raw numbers), which I'm always surprised that the functional programming community seems mostly unaware of.
  - ahf8Aithaex7Nai 6mo ago
    > We liked the name and thought it was funny so we kept it. It is funny, and I really like the reference. > ... , which I'm always surprised that the functional programming community seems mostly unaware of. Oh, I should have clicked on the Hypothesis link in the first paragraph. Thanks for pointing that out! Edit: And it makes me smile that there was a long thread about it.
fede_dp 6mo ago
[dead]

285 points by alpaylan 6mo ago

Post author here btw, happy to take questions, whether they're about Hegel in particular, property-based testing in general, or some variant on "WTF do you mean you wrote rust bindings to a python library?"

anentropic 6mo ago
TBH reading the first few words of that section I was definitely expecting it to continue "so we used Claude to rewrite Hypothesis in Rust..." so that was quite a surprise!
- DRMacIver 6mo ago
  It's on the agenda! We definitely want to rewrite the Hegel core server in rust, but not as much as we wanted to get it working well first. My personal hope is that we can port most of the Hypothesis test suite to hegel-rust, then point Claude at all the relevant code and tell it to write us a hegel-core in rust with that as its test harness. Liam thinks this isn't going to work, I think it's like... 90% likely to get us close enough to working that we can carry it over the finish line. It's not a small project though. There are a lot of fiddly bits in Hypothesis, and the last time I tried to get Claude to port it to Rust the result was better than I expected but still not good enough to use.
  - tybug 6mo ago
    To put it on the record: my position is current models can't get us there, and neither can the next iteration of models, but in two model iterations this will be worth doing. There's a lot of fiddly details in Hypothesis that are critical to get right. You can get a plausible 80% port with agents today but find they've structured it in a way to make it impossible to get to 100%.
Chinjut 6mo ago
You mention in the post that there are design differences between Hegel/Hypothesis and QuickCheck, partly due to attitude differences between Python/non-Haskell programmers and Haskell programmers. As someone coming from the Haskell world (though by no means considering Haskell a perfect language), could you expand on what kinds of differences these are?
- DRMacIver 6mo ago
  So I think a short list of big API differences are something like: * Hypothesis/Hegel are very much focused on using test assertions rather than a single property that can be true or false. This naturally drives a style that is much more like "normal" testing, but also has the advantage that you can distinguish between different types of failing test. We don't go too hard on this, but both Hegel and Hypothesis will report multiple distinct failures if your test can fail in multiple ways. * Hegelothesis's data generation and how it interacts with testing is much more flexible and basically fully imperative. You can basically generate whatever data you like wherever in your test you like, freely interleaving data generation and test execution. * QuickCheck is very much type-first and explicit generators as an afterthought. I think this is mostly a mistake even in Haskell, but in languages where "just wrap your thing in a newtype and define a custom implementation for it" will get you a "did you just tell me to go fuck myself?" response, it's a nonstarter. Hygel is generator first, and you can get the default generator for a type if you want but it's mostly a convenience function with the assumption that you're going to want a real generator specification at some point soon. From an implementation point of view, and what enables the big conveniences, Hypothesis has a uniform underlying representation of test cases and does all its operations on them. This means you get: * Test caching (if you rerun a failing test, it will immediately fail in the same way with the previously shrunk example) * Validity guarantees on shrinking (your shrunk test case will always be ones your generators could have produced. It's a huge footgun in QuickCheck that you can shrink to an invalid test case) * Automatically improving the quality of your generators, never having to write your own shrinkers, and a whole bunch of other quality of life improvements that the universal representation lets us implement once and users don't have to care about. The validity thing in particular is a huge pain point for a lot of users of PBT, and is what drove a lot of the core Hypothesis model to make sure that this problem could never happen. The test caching is because I personally hated rerunning tests and not knowing whether it was just a coincidence that they were passing this time or that the test case had changed.
peytongreen_dev 6mo ago
[flagged]
mullr 6mo ago
Why would I use this over the existing Proptest library in Rust?
- DRMacIver 6mo ago
  Answered this over here: https://news.ycombinator.com/item?id=47506274 https://news.ycombinator.com/item?id=47506274
nz 6mo ago
Not really a question. Just wanted to express my gratitude for Hypothesis. I use it regularly. A few years back, I had to build a semi-formally-verified fund and account management service, and used the state-based-testing of Hypothesis to validate its correctness. Cannot express how invaluable this little framework has been. A little while after that, I spoke to someone in the pharma-adjacent-space who was looking at Antithesis to validate their product. At the time, Antithesis (the company) told him that it was a bad fit. I suggested something akin to my previous approach (which did not include antithesis). No clue what they ended up doing, but it is nice to see that Hypothesis and Antithesis have finally joined forces.
- DRMacIver 6mo ago
  You're very welcome! I'm glad it's been useful for you.
iainmerrick 6mo ago
One thing I'm curious about, which I couldn't figure out from a skim of your post, is whether the generated test inputs are random, sequential, or adversarial. IIRC there are fuzz testers that will analyze the branches of the code to look for edge cases that might break it -- that seems like something that would be wonderful to have in a property tester, but it also seems very difficult to do, especially in a language agnostic way. How long does it take to find breaking cases like "0/0" or "ß"? Do they pop up immediately, or does it only happen after hundreds or thousands of runs?
- DRMacIver 6mo ago
  They're random but with a lot of tweaks to the distribution that makes weird edge cases pop up with fairly high probability, and with some degree of internal mutation, followed by shrinking to turn them into nice tidy test cases. In Python we do a little bit of code analysis to find interesting constants, but Hegel doesn't do that, it's just tuned to common edge cases. I think all the examples I had in the post are typically found in the first 100 test cases and reliably found in the first 1000, but I wouldn't swear that that's the case without double checking. We don't do any coverage-guidance in Hegel or Hypothesis, because for unit testing style workflows it's rarely worth it - it's very hard to do good coverage guidance in under like... 10k test runs at a minimum, 100k is more likely. You don't have enough time to get really good at exploring the state space, and you haven't hit the point where pure random testing has exhausted itself enough that you have to do something smarter to win. It's been a long-standing desire of mine to figure out a way to use coverage to do better even on short runs, and there are some kinda neat things you can do with it, but we've not found anything really compelling.

tybug 6mo ago

As possibly the one community on earth where it's actually better to post the code than the blog post: TL;DR this is a universal property-based testing protocol (https://github.com/hegeldev/hegel-core https://github.com/hegeldev/hegel-core) and family of libraries (https://github.com/hegeldev/hegel-rust https://github.com/hegeldev/hegel-rust, more to come later). I've talked with lots of people in the PBT world who have always seen something like this as the end goal of the PBT ecosystem. It seemed like a thing that would happen eventually, someone just had to do it. I'm super excited to actually be doing it and bringing great PBT to every and any language. It doesn't hurt that this is coming right as great PBT in every language is suddenly a lot more important thanks to AI code!

hugeBirb 6mo ago

Not that it matters at this point but the hegelian dialectic is not thesis, antithesis and synthesis. Usually attributed to Hegel but as I understand it he actually pushed back on this mechanical view of it all and his views on these transitory states was much more nuanced.

DRMacIver 6mo ago
Conversation with Will (Antithesis CEO) a couple months ago, heavily paraphrased: Will: "Apparently Hegel actually hated the whole Hegelian dialectic and it's falsely attributed to him." Me: "Oh, hm. But the name is funny and I'm attached to it now. How much of a problem is that?" Will: "Well someone will definitely complain about it on hacker news." Me: "That's true. Is that a problem?" Will: "No, probably not." (Which is to say: You're entirely right. But we thought the name was funny so we kept it. Sorry for the philosophical inaccuracy)
- wwilson 6mo ago
  If I had been wearing my fiendish CEO hat at the time, I might have even said something like: "somebody pointing this out will be a great way to jumpstart discussion in the comments." One of the evilest tricks in marketing to developers is to ensure your post contains one small inaccuracy so somebody gets nerdsniped... not that I have ever done that.
  - 1-more 6mo ago
    A sort of broadening of Cunningham's Law (the fastest way to get an answer online is not by posting the question, but by posting the wrong answer—very true in my experience). If there's no issue of fact at hand, then you end up getting some engagement about the intentional malapropism/misattribution/mistake/whatever and then the forum rules tend to herd participants back to discussing the matter at hand: your company. https://meta.wikimedia.org/wiki/Cunningham%27s_Law https://meta.wikimedia.org/wiki/Cunningham%27s_Law
  - jpadkins 6mo ago
    Seth Godin made the case that its more important for people to make remarks than to be favorable (https://en.wikipedia.org/wiki/Purple_Cow:_Transform_Your_Business_by_Being_Remarkable https://en.wikipedia.org/wiki/Purple_Cow:_Transform_Your_Bus...) Trump did this a lot with the legacy media in his first term. He would make inaccurate statements to the media on the topic he wanted to be in the spotlight, and the media would jump to "fact check" him. Guess what, now everyone is talking about illegal immigration, tariffs, or whatever subject Trump thought was to their advantage.

pron 6mo ago

> property-based testing is going to be a huge part of how we make AI-agent-based software development not go terribly. There's no doubt, I think, testing will remain important and possibly become more important with more AI use, and so better testing is helpful, PBT included. But the problem remains verifying that the tests actually test what they're supposed to. Mutation tests can allow agents to get good coverage with little human intervention, and PBT can make tests better and more readable. But still, people have to read them and understand them, and I suspect that many people who claim to generate thousands of LOC per day don't. And even if the tests were great and people carefully reviewed them, that's not enough to make sure things don't go terribly wrong. Anthropic's C compiler experiment didn't fail because of bad testing. Not only were the tests good, it took humans years to write the tests by hand, and the agents still failed to converge. I think good tests are a necessary condition for AI not generating terrible software, but we're clearly not yet at a point where they're a sufficient one. So "a huge part" - possibly, but there are other huge parts still missing.

tybug 6mo ago
I actually think there's another angle here where PBT helps, which wasn't explored in the blog post. That angle is legibility. How do you know your AI-written slop software is doing the right thing? One would normally read all the code. Bad news: that's not much less labor intensive as not using AI at all. But, if one has comprehensive property-based tests, they can instead read only the property-based tests to convince themselves the software is doing the right thing. By analogy: one doesn't need to see the machine-checked proof to know the claim is correct. One only needs to check the theorem statement is saying the right thing.
- pron 6mo ago
  Right, I said that property based tests are easier to read, and that's good. But people still have to actually read them. Also, because they still work best at the "unit" level, to understand them, the people reading them need to know how all the units are connected (e.g. a single person cannot review even PBTs required for 10KLOC per day [1]). My point isn't so much about PBT, but about how we don't yet know just how much agents help write real software (and how to get the most help from them). [1]: I'm only using that number because Garry Tan, CEO of YC, claimed to generate 10K lines of text per day that he believes to be working code and developers working with AI agents know they can't be.

rdevilla 6mo ago

This is the first time in my HN membership where I was excited to read about the dialectic, only to be disappointed upon finding out the article is about Rust. PBT is for sure the future - which is apparently now? 10 years ago when I was talking about QuickCheck [0] all the JS and Ruby programmers in my city just looked at me like I had two heads. [0] https://github.com/ryandv/chesskell/blob/master/test/Test/Chess/BitboardSpec.hs#L134 https://github.com/ryandv/chesskell/blob/master/test/Test/Ch...

DRMacIver 6mo ago
TBF PBT has been the present in Python for a while now. 10 years ago might have been a little early (Hypothesis 1.0 came out 11 years ago this coming Thursday), but we had pretty wide adoption by year two and it's only been growing. It's just that the other languages have all lagged behind. It's by no means universally adopted, but it's not a weird rare thing that nobody has heard of.

lwhsiao 6mo ago

DRMacIver, can you comment on how this fits into the existing property-based testing ecosystems for various languages? E.g., if I use proptest in Rust, why would/should I switch to Hegel?

DRMacIver 6mo ago
The short answer to how it fits into existing ecosystems is... in competition I suppose. We've got a lot of respect for the people working on these libraries, but we think the Hypothesis-based approach is better than the various approaches people have adopted. I don't love that the natural languages for us to start with are ones where there are already pretty good property-based testing libraries whose toes we're stepping on, but it ended up being the right choice because those are the languages people care about writing correct software in, and also the ones we most want the tools in ourselves! I think right now if you're a happy proptest user it's probably not clear that you should switch to Hegel. I'd love to hear about people trying, but I can't hand on my heart say that it's clearly the correct thing for you to do given its early state, even though I believe it will eventually be. But roughly the things that I think are clearly better about the Hegel approach and why it might be worth trying Hegel if you're starting greenfield are: * Much better generator language than proptest (I really dislike proptest's choices here. This is partly personal aesthetic preferences, but I do think the explicitly constructed generators work better as an approach and I think this has been borne out in Hypothesis). Hegel has a lot of flexible tooling for generating the data you want. * Hegel gets you great shrinking out of the box which always respects the validity requirements of your data. If you've written a generator to always ensure something is true, that should also be true of your shrunk data. This is... only kindof true in proptest at best. It's not got quite as many footguns in this space as original quickcheck and its purely type-based shrinking, but you will often end up having to make a choice between shrinking that produces good results and shrinking that you're sure will give you valid data. * Hegel's test replay is much better than seed saving. If you have a failing test and you rerun it, it will almost immediately fail again in exactly the same way. With approaches that don't use the Hypothesis model, the best you can hope for is to save a random seed, then rerun shrinking from that failing example, which is a lot slower. There are probably a bunch of other quality of life improvements, but these are the things that have stood out to me when I've used proptest, and are in general the big contrast between the Hypothesis model and the more classic QuickCheck-derived ones.

skybrian 6mo ago

It isn't used by anyone besides me, but I wrote a property-testing library for Deno [1] that has a form of "sometimes" assertions (inspired by Antithesis) and uses "internal shrinking" (inspired by Hypothesis). But it's still a "blind" fuzzer and it would be nice to write one that gets feedback from code coverage somehow. Instead, you have to run code coverage yourself and figure out how to change test data generation to improve it. [1] https://jsr.io/@skybrian/repeat-test https://jsr.io/@skybrian/repeat-test

chriswarbo 6mo ago
> But it's still a "blind" fuzzer and it would be nice to write one that gets feedback from code coverage somehow There have been simplistic attempts at this, e.g. instead of performing 100 tests, just keep going as long as coverage increases. The Choice Gradient Sampling algorithm from https://arxiv.org/pdf/2203.00652 https://arxiv.org/pdf/2203.00652 feels like a nice way to steer generators in a more nuanced way. That paper uses it to avoid discards when rejection-sampling; but I have a feeling it could be repurposed to "reward" based on new coverage instead/as-well.
- skybrian 6mo ago
  It's not like how it's done in that paper, but oddly enough I did end up implementing some conversions in both directions, from an array of choices to a JavaScript object and back again. https://jsr.io/@skybrian/repeat-test/doc/core/~/Domain https://jsr.io/@skybrian/repeat-test/doc/core/~/Domain

seamossfet 6mo ago

Oh my god, the rust developers are writing tests with Hegelian dialects.

sunshowers 6mo ago

Hi David, congratulations on the release! I'm excited to play around with Hypothesis's bitstream-based shrinking. As you're aware, prop_flat_map is a pain to deal with, and I'd love to replace some of my proptest-based tests with Hegel. I spent a little time looking at Hegel last week and it wasn't quite clear to me how I'd go about having something like a canonical generator for a type (similar to proptest's Arbitrary). I've found that to be very helpful while generating large structures to test something like serialization roundtripping against — in particular, the test-strategy library has derive macros that work very well for business logic types with, say, 10-15 enum variants each of which may have 0-10 subfields. I'm curious if that is supported today, or if you have plans to support this kind of composition in the future. edit: oh I completely missed the macro to derive DefaultGenerator! Whoops

tybug 6mo ago
Yep, `#[derive(DefaultGenerator)]` and `generators::default<T>()` are the right tools here. This is one of the areas we've dogfooded the least, so we'd definitely be happy to get feedback on any sharp corners here! I think `from_type` is one of Hypothesis's most powerful and ergonomic strategies, and that while we probably can't get quite to that level in rust, we can still get something that's pretty great.
- sunshowers 6mo ago
  Thank you! I have some particularly annoying proptest-based tests that I'll try porting over to Hegel soon. (Thanks for writing the Claude skill to do this.)
  - DRMacIver 6mo ago
    Please let us know how it goes! As Liam says, the derive generator is not very well dogfooded at present. The claude skill is a bit better, but we've only been through a few iterations of using it and getting Claude to improve it, and porting from proptest is one of the less well tested areas (because we don't use proptest much ourselves). I expect all of this works, but I'd like to know ways that it works less well than it could. Or, you know, to bask in the glow of praise of it working perfectly if that turns out to be an option.

shrubby 6mo ago

In here just for the Hegel joke.

LoganDark 6mo ago

Using Python from other languages is terrible. I love this kind of testing but this implementation is not for me. I was so excited before learning it depends on Python.

epage 6mo ago
Yeah, having `cargo test` require another binary like `uv` is not idiomatic. 99% of the time, I should be able to walk up to a Rust project and run `cargo test` and it should just work.
- LoganDark 6mo ago
  `uv` isn't even a Python interpreter. I prefer to build my systems without Python support.

keeganpoppen 6mo ago

love the hegel reference. i know hypothesis is awesome, and so im certain this is as well. this is no real complaint about the post, because it is a personal skill issue, but for someone who is more of a PL nerd than… most people on earth… i find Ruat code to be some of the hardest to actually parse in any meaningful way. even to get a sense of how the property defs were being set up, i kinda just took it at face value that it is cool… i guess it really comes down to personally not having much interest in rust compared to most newfangled languages out there. like i said, skipl issue, not a conplaint. wish you all the best, and lots of success!

ruuda 6mo ago

Property-based testing is nice, but making it coverage-driven is a game changer. It will explore code paths that naive random inputs will not trigger in a thousand years. In Rust this works very well with libFuzzer and the Arbitrary crate to derive the generators.

wwilson 6mo ago
[Disclaimer: Started Antithesis] If you run your Hegel tests in Antithesis, you get this for free (along with various sorts of “non-local” assertions, perfect reproducibility even for concurrent or distributed code, etc.). But yeah, not hard to hack together basic coverage guidance outside Antithesis. That works well for large classes of programs, just not a majority of them.

Weebs 6mo ago

"I thought this would be about Hegel... oh, wait"

ahf8Aithaex7Nai 6mo ago

To all you amateur Hegel enthusiasts out there: there is no synthesis in Hegel. Otherwise: Congratulations on the QuickCheck-style testing in Rust. At work, I’m always surprised that property-based testing is so little known and so rarely used outside of functional programming.

DiscourseFan 6mo ago
“Synthesis” is the purposiveness in the force of the Concept.
dr_dshiv 6mo ago
yeah where did that come from? It's like attributing the cartesian coordinate plane to Descartes -- when actually it was a textbook author who came up with the modern one...
DRMacIver 6mo ago
> To all you amateur Hegel enthusiasts out there: there is no synthesis in Hegel. Looks like the mods deleted the last long thread about this, so best not to relitigate, but short version: Yes, we know. We liked the name and thought it was funny so we kept it. > Otherwise: Congratulations on the QuickCheck-style testing in Rust. At work, I’m always surprised that property-based testing is so little known and so rarely used outside of functional programming. Actually, it's Hypothesis-style testing in Rust. There was already QuickCheck style. Property-based testing is in fact far more widely used in Python than in functional programming (probably not as a percentage of users, but in terms of raw numbers), which I'm always surprised that the functional programming community seems mostly unaware of.
- ahf8Aithaex7Nai 6mo ago
  > We liked the name and thought it was funny so we kept it. It is funny, and I really like the reference. > ... , which I'm always surprised that the functional programming community seems mostly unaware of. Oh, I should have clicked on the Hypothesis link in the first paragraph. Thanks for pointing that out! Edit: And it makes me smile that there was a long thread about it.

fede_dp 6mo ago

[dead]