11 ms·

I don't want to knock you down as most have already did. In-fact it's a useful exercise going forward in exploring how to work with AI. It's here, we're all goi

I don't want to knock you down as most have already did. In-fact it's a useful exercise going forward in exploring how to work with AI. It's here, we're all going to use it one way or the other. Zero issues with that, in-fact kudos to going through the pain of it all. Now, having gone through several such endeavors originally myself, albeit with internal tools and systems (as an exercise), I've noticed that while all my tests passed with flying colors the rewrite itself was broken even on basic functionality or missed a ton of details. It was in effect useless when I dived into it. Initial tests also showed massive gain in performance, and I know people who were involved aren't really dumb so something smelled funny. Turns out all those things left out and honestly... moments were the key ingredients. What I did learn from those beginning explorations though was that one-shotting, grand architecture or source up-front, master plans up-front.. all these do not yield good results - YET. Who know what we'll see in few years though. What I did found that works (FOR ME, nota bene) is to keep the design and checklists for myself, written by myself and then do a small piece by piece.. as if you would if you were coding alone or if you would waterfalling a small team of talented juniors. Then, suddenly super happy results come out, but then it's mostly you driving all the way where llm writes code and offers advice (which for the most part you ignore). It's a happy place for myself at least. It's then truly unlocking yourself to the mythical 10x. Rewriting a large proven system with decades of ultra expertise behind it, which I don't have, is guaranteed not to end up the same 1:1 replacement. If you found a recipe for that - please do share.

ramijames 2mo ago
I'm curious. Do you attribute this to weak and/or incomplete tests? How granular should tests be to have complete coverage so that an AI won't create a converted codebase that "passes tests" but is still functionally inaccurate?
- Keyframe 2mo ago
  That's the million dollar question. Do you/we/us have tests that cover everything which covers QA as well? If such a mythical beast exists, maybe from remnants of ye olde TDD past and hasn't been modified as such.. then maybe this would be possible to do as such.
  - ramijames 2mo ago
    I've always played the game that tests are only as useful as you have resources to attribute to them. Mostly all modern development is a compromise between new features and stuff that supports those new features (tests included, but also reviews, code maintenance, docs, etc.). If LLMs can be utilized to quickly make deep testing possible, I think that's probably a net-positive.
- NohatCoder 2mo ago
  There is no such thing as a complete test suite, there will always be some possible bug that it doesn't catch. In particular, if you put an LLM in an automated loop of "this test fails, please fix it", there is a pretty good chance that it will simply special case all of the tests, possibly in some contrived way that makes it not at all obvious when you read the code.
  - aeonfox 2mo ago
    This is where fuzzing would be useful. We have an at-least-parity-bug-level oracle with the reference PostgreSQL implementation. Just build a generator of queries (both invalid and valid) and ensure the output matches. The yardstick is how many log10(queries) it can go on average before a discrepancy is found.
    - apf6 2mo ago
      It’s 2026, so another basic technique that every team should add to their testing strategy (in addition to proven techniques like fuzzing) is agentic user simulation. Set up an AI agent with access to the product, and prompt it to use the thing in hundreds of realistic use cases, and to report any possible bugs. It will catch a lot of ‘blind spot’ bugs that were previously things that were only caught by humans.
      - aeonfox 2mo ago
        Debug yourself of the AI hype^. Besides config, SQL implementations have a very limited textual API surface of accepting queries. Putting whole layers of UI and agents around it is really inefficient way of finding edge cases. I wouldn't call that basic, nor the most comprehensive option. Fuzzing at the query layer is far more computationally efficient and effective. A truly stochastic method is more likely to hit against edge cases, rather than an agent that tends to towards idiomatic solutions and that is trained against a corpus of existing software, and burns millions of tokens/watts spinning its wheels. ^ There's plenty of business value to be found in agentic AI without reaching for it for every solution. I'd even posit agentic AI is even better when paired with focused old-fashioned squishy-brained software engineering in the loop.
- cookiengineer 2mo ago
  Unit tests don't test for branch coverage. That's the culprit, because LLMs tend to forget and remove a lot of branch logic in these kinds of tasks. If unit tests don't cover these specific if/elseif/else cases, then they'll just disappear. They'll also disappear if the LLM is allowed to modify the unit tests, because they sure like to cheat their way around into greenlit test suites. The agentic environment must disallow write access to the unit test files for the agent that writes the code. If you implement that in your tools, you'll see quickly how the models will try to rewrite the unit tests at all cost, no matter what kind of prompting you've done. Tool policies are the only boundary to successfully guarantee this. Source: Am building my own agentic environment because of that behavior
jakswa 2mo ago
what model did you try it with? I agree and also push back a bit: How will we know when the LLMs reach the point of handling it if no one takes the leap? I applaud more people sludging through the slop and hauling their slop buckets around. An example is Fable being released. I felt like the most complex thing I was willing to sludge through was having it clone llama-server's web UI with my own opinions (I really like the original, kudos to them). And the initial skeleton was working so well I felt like I had sunk the tokens and committed to getting it the rest of the way: https://inkcap.click https://inkcap.click
- Keyframe 2mo ago
  Last time it was Opus 4.8 to no avail. Fable is too expen..precious to try that yet :)
gulugawa 2mo ago
I don't code with LLMs, and think you might be right. However, Postgres is a tool with clearly defined functionality and doesn't have ambiguous requirements that are seen in user facing software. As a result, it is entirely plausible that the author created a working Postgres replacement for certain use cases. I personally want to see more evidence about the quality of the tool after it is in a finished state.

ramijames 2mo ago

I'm curious. Do you attribute this to weak and/or incomplete tests? How granular should tests be to have complete coverage so that an AI won't create a converted codebase that "passes tests" but is still functionally inaccurate?

Keyframe 2mo ago
That's the million dollar question. Do you/we/us have tests that cover everything which covers QA as well? If such a mythical beast exists, maybe from remnants of ye olde TDD past and hasn't been modified as such.. then maybe this would be possible to do as such.
- ramijames 2mo ago
  I've always played the game that tests are only as useful as you have resources to attribute to them. Mostly all modern development is a compromise between new features and stuff that supports those new features (tests included, but also reviews, code maintenance, docs, etc.). If LLMs can be utilized to quickly make deep testing possible, I think that's probably a net-positive.
NohatCoder 2mo ago
There is no such thing as a complete test suite, there will always be some possible bug that it doesn't catch. In particular, if you put an LLM in an automated loop of "this test fails, please fix it", there is a pretty good chance that it will simply special case all of the tests, possibly in some contrived way that makes it not at all obvious when you read the code.
- aeonfox 2mo ago
  This is where fuzzing would be useful. We have an at-least-parity-bug-level oracle with the reference PostgreSQL implementation. Just build a generator of queries (both invalid and valid) and ensure the output matches. The yardstick is how many log10(queries) it can go on average before a discrepancy is found.
  - apf6 2mo ago
    It’s 2026, so another basic technique that every team should add to their testing strategy (in addition to proven techniques like fuzzing) is agentic user simulation. Set up an AI agent with access to the product, and prompt it to use the thing in hundreds of realistic use cases, and to report any possible bugs. It will catch a lot of ‘blind spot’ bugs that were previously things that were only caught by humans.
    - aeonfox 2mo ago
      Debug yourself of the AI hype^. Besides config, SQL implementations have a very limited textual API surface of accepting queries. Putting whole layers of UI and agents around it is really inefficient way of finding edge cases. I wouldn't call that basic, nor the most comprehensive option. Fuzzing at the query layer is far more computationally efficient and effective. A truly stochastic method is more likely to hit against edge cases, rather than an agent that tends to towards idiomatic solutions and that is trained against a corpus of existing software, and burns millions of tokens/watts spinning its wheels. ^ There's plenty of business value to be found in agentic AI without reaching for it for every solution. I'd even posit agentic AI is even better when paired with focused old-fashioned squishy-brained software engineering in the loop.
cookiengineer 2mo ago
Unit tests don't test for branch coverage. That's the culprit, because LLMs tend to forget and remove a lot of branch logic in these kinds of tasks. If unit tests don't cover these specific if/elseif/else cases, then they'll just disappear. They'll also disappear if the LLM is allowed to modify the unit tests, because they sure like to cheat their way around into greenlit test suites. The agentic environment must disallow write access to the unit test files for the agent that writes the code. If you implement that in your tools, you'll see quickly how the models will try to rewrite the unit tests at all cost, no matter what kind of prompting you've done. Tool policies are the only boundary to successfully guarantee this. Source: Am building my own agentic environment because of that behavior

jakswa 2mo ago

what model did you try it with? I agree and also push back a bit: How will we know when the LLMs reach the point of handling it if no one takes the leap? I applaud more people sludging through the slop and hauling their slop buckets around. An example is Fable being released. I felt like the most complex thing I was willing to sludge through was having it clone llama-server's web UI with my own opinions (I really like the original, kudos to them). And the initial skeleton was working so well I felt like I had sunk the tokens and committed to getting it the rest of the way: https://inkcap.click https://inkcap.click

Keyframe 2mo ago
Last time it was Opus 4.8 to no avail. Fable is too expen..precious to try that yet :)

gulugawa 2mo ago

I don't code with LLMs, and think you might be right. However, Postgres is a tool with clearly defined functionality and doesn't have ambiguous requirements that are seen in user facing software. As a result, it is entirely plausible that the author created a working Postgres replacement for certain use cases. I personally want to see more evidence about the quality of the tool after it is in a finished state.