5 ms·

I think the reverse direction is more important: taking a massive complex problem/codebase and decomposing it to short pseudocode. Then you could edit the pseud

by avaer 27d ago

I think the reverse direction is more important: taking a massive complex problem/codebase and decomposing it to short pseudocode. Then you could edit the pseudocode and compile it back into the system. That's the way software engineers working on large projects work anyway: you first gather context on the state of the system and read it at a level you can understand. Then you propose a change on the simplified representation, and then holistically update the machine-runnable format ("implementation"). I'd be interested in tools that formalize/automate this process more.

igravious 27d ago
Clever! :) Difficult! :(
- avaer 27d ago
  I don't think it's particularly difficult, or clever. You can prompt your way into this with frontier models, especially if your codebase is designed for it from the ground up (architecture well factored, documented, tested, harnessable, so that it even has a pseudocode representation in the first place). The biggest issue is you end up leaning heavily on the quality of the model. Lower fidelity models tend to make a mess and add tech debt that you must frequently repay with intentional cleanup passes from a higher quality model, or else the rate of useful progress will fall off a cliff. At least that's my experience.
  - mym1990 27d ago
    Massive, complex codebase…short pseudocode. The fact that you don’t think this is particularly difficult makes me questions everything after that statement.
ianhorn 27d ago
I'm working on a bidirectional version of this! The big challenge is keeping the various versions of the thing in sync through edits.
- egonschiele 27d ago
  Care to share a link? This sounds neat
a2ff6eeb0 27d ago
This is the entire domain of programming language design, except with the goal of doing it reliably and deterministically enough that you didn't need to look past the short version of the code at the compiled output. We all know how badly that failed once we started coming up with AI, and could outsource dealing with all that bullshit. Nobody wants this -- they ran screaming as soon as it was viable.
danielvaughn 27d ago
Yeah I'm not opposed to bi-directionality, though right now there are a ton of projects attempting to do some variation on that theme. Extracting specs from code, producing short summaries, etc. I've tried a few ideas at the company I work for, and they never seem to pan out. At the end of the day, you need something that was simply written by human hands.
andai 27d ago
I had a LLM based prototype called "cleanroom" which would convert a program into a spec and then back into a program. The results were disgusting: the spec would encode all sorts of irrelevant implementation details, and then the new version would reimplement them faithfully, and be 3x more bloated than the original. The exact opposite of what I was going for! I didn't put much effort into it, maybe it was solvable with prompting (or more likely, more human effort on the spec phase), but it looks like the LLM has the same problem as the human, it can't know what the intention was, and it can't know what's relevant, what's essential and incidental. But basically, what I needed wasn't a spec but user stories. (And probably multiple prototype outputs to choose from...) I should definitely give it another crack though... --- P.S., Spoiler for next ten years: software as biology (esp. crossbreeding, mutation, selection pressure...)
- danielvaughn 27d ago
  This largely matches my experience, and was one of the motivators behind this project. For some reason, LLMs are still very bad at distillation. We had a really large and complex codebase at my job that I had to quickly ramp up on. I asked an LLM to generate product docs from it, and it barely helped me. I came to the conclusion that something in the codebase needs a hard boundary, where the team agrees that only human hands touch it. Otherwise the entire repo becomes untrustworthy as far as discovering intent goes.
rpunkfu 27d ago
I think you might be interested in what I’m working on: https://ctx.company/blog/introducing-ctx-traits/ https://ctx.company/blog/introducing-ctx-traits/

by avaer 27d ago

igravious 27d ago

Clever! :) Difficult! :(

avaer 27d ago
I don't think it's particularly difficult, or clever. You can prompt your way into this with frontier models, especially if your codebase is designed for it from the ground up (architecture well factored, documented, tested, harnessable, so that it even has a pseudocode representation in the first place). The biggest issue is you end up leaning heavily on the quality of the model. Lower fidelity models tend to make a mess and add tech debt that you must frequently repay with intentional cleanup passes from a higher quality model, or else the rate of useful progress will fall off a cliff. At least that's my experience.
- mym1990 27d ago
  Massive, complex codebase…short pseudocode. The fact that you don’t think this is particularly difficult makes me questions everything after that statement.

ianhorn 27d ago

I'm working on a bidirectional version of this! The big challenge is keeping the various versions of the thing in sync through edits.

egonschiele 27d ago
Care to share a link? This sounds neat

a2ff6eeb0 27d ago

This is the entire domain of programming language design, except with the goal of doing it reliably and deterministically enough that you didn't need to look past the short version of the code at the compiled output. We all know how badly that failed once we started coming up with AI, and could outsource dealing with all that bullshit. Nobody wants this -- they ran screaming as soon as it was viable.

danielvaughn 27d ago

Yeah I'm not opposed to bi-directionality, though right now there are a ton of projects attempting to do some variation on that theme. Extracting specs from code, producing short summaries, etc. I've tried a few ideas at the company I work for, and they never seem to pan out. At the end of the day, you need something that was simply written by human hands.

andai 27d ago

I had a LLM based prototype called "cleanroom" which would convert a program into a spec and then back into a program. The results were disgusting: the spec would encode all sorts of irrelevant implementation details, and then the new version would reimplement them faithfully, and be 3x more bloated than the original. The exact opposite of what I was going for! I didn't put much effort into it, maybe it was solvable with prompting (or more likely, more human effort on the spec phase), but it looks like the LLM has the same problem as the human, it can't know what the intention was, and it can't know what's relevant, what's essential and incidental. But basically, what I needed wasn't a spec but user stories. (And probably multiple prototype outputs to choose from...) I should definitely give it another crack though... --- P.S., Spoiler for next ten years: software as biology (esp. crossbreeding, mutation, selection pressure...)

danielvaughn 27d ago
This largely matches my experience, and was one of the motivators behind this project. For some reason, LLMs are still very bad at distillation. We had a really large and complex codebase at my job that I had to quickly ramp up on. I asked an LLM to generate product docs from it, and it barely helped me. I came to the conclusion that something in the codebase needs a hard boundary, where the team agrees that only human hands touch it. Otherwise the entire repo becomes untrustworthy as far as discovering intent goes.

rpunkfu 27d ago

I think you might be interested in what I’m working on: https://ctx.company/blog/introducing-ctx-traits/ https://ctx.company/blog/introducing-ctx-traits/