12 ms·
The unexpected effectiveness of one-shot decompilation with Claude
- knackers 10mo agoI've been experimenting with running Claude in headless mode + a continuous loop to decompile N64 functions and the results have been pretty incredible. (This is despite already using Claude in my decompilation workflow). I hope that others find this similarly useful.
- garrettjoecox 10mo agoWhat game are you working on?
- wk_end 10mo agoLast sentence of the first paragraph says it’s Snowboard Kids 2.
- deleted 10mo ago[deleted]
- rat9988 10mo agoFor his defense, it is missing a "Tell HN"
- dpkirchner 10mo agoAnd it isn't always obvious when the commenter is the submitter (no [S] tag like you see on other sites).
- garrettjoecox 10mo agowhoops, I did indeed miss that this was OP
- plastic-enjoyer 10mo agoThis sounds interesting! Do you have some good introduction to N64 decompiliation? Would you recommend using Claude right from the start or rather try to get to know the ins and outs of N64 decomp?
- turnsout 10mo agoThis is super cool! I would be curious to see how Gemini 3 fares… I've found it to be even more effective than Opus 4.5 at technical analysis (in another domain).
- deleted 10mo ago[deleted]
- viraptor 10mo agoOne thing I don't annoying in really old sources is that sometimes you can't go function by function, because the code will occasionally just use a random register to pass results. Passing the whole file works better at that point.
- djmips 10mo agoThanks, this is very cool! I've started to dip my toes into this and it's good to see it has potential.
- ACCount37 10mo agoIf you aren't using LLMs for your reverse engineering tasks, you're missing out, big time. Claude kicks ass. It's good at cleaning up decompiled code, at figuring out what functions do, at uncovering weird assembly tricks and more.
- amelius 10mo agoMakes sense because LLMs are quite good at translating between natural languages. Anyway, we're reaching the point where documentation can be generated by LLMs and this is great news for developers.
- monsieurbanana 10mo agoMaybe documentation meant for other llms to ingest. Their documentation is like their code, it might work, but I don't want to have to be the one to read it. Although of course if you don't vibe document but instead just use them as a tool, with significant human input, then yes go ahead.
- dunham 10mo agoAlthough with code it's implementing functions that don't exist yet and with documentation, it's describing functions that don't exist yet.
- james_marks 10mo agoI stumbled across a fun trick this week. After making some API changes, I had CC “write a note to the FE team with the changes”. I then pasted this to another CC instance running the FE app, and it made the counter part. Yes, I could have CC running against both repos and sometimes do, but I often run separate instances when tasks are complex.
- saagarjha 10mo agoDocumentation is one place where humans should have input. If an LLM can generate documentation, why would I want you to generate it when I can do so myself (probably with a better, newer model)?
- butz 10mo agoAre there any similar specialized decompilation LLM models available to be used locally?
- jamesbelchamber 10mo agoThis is a refreshingly practical demonstration of an LLM adding value. More of this please.
- rlili 10mo agoMakes me wonder if decompilation could eventually become so trivial that everything would become de-facto open source.
- Xmd5a 10mo agoThis deserves a discussion
- ronsor 10mo agoI've used LLMs to help with decompilation since the original release of GPT-4. They're excellent at recognizing the purpose of functions and refactoring IDA or Ghidra pseudo-C into readable code.
- galangalalgol 10mo agoHow does it do on things that were originally written in assembly?
- saagarjha 10mo agoThis is typically easier because the code was written for humans already.
- euroderf 10mo agoSomeone please try this on an original (early 1980s) IBM-PC BIOS.
- saagarjha 10mo agoIt's worth noting here that the author came up with a handful of good heuristics to guide Claude and a very specific goal, and the LLM did a good job given those constraints. Most seasoned reverse engineers I know have found similar wins with those in place. What LLMs are (still?) not good at is one-shot reverse engineering for understanding by a non-expert. If that's your goal, don't blindly use an LLM. People already know that you getting an LLM to write prose or code is bad, but it's worth remembering that doing this for decompilation is even harder :)
- ph4evers 10mo agoAre they not performing well because they are trained to be more generic, or is the task too complex? It seems like a cheap problem to fine-tune.
- pixl97 10mo agoSounds like a more agentic pipeline task. Decompile, assess, explain.
- motoboi 10mo agoThe knowledge probably is o the pre-training data (the internet documenta the LLM is trained at to get a good grasp), but probably very poorly represented in the reinforcement learning phase. Which is to say that probably antropic don’t have good training documents and evals to teach the model how to do that. Well they didn’t. But now they have some. If the author want to improve his efficiency even more, I’d suggest he starts creating tools that allow a human to create a text trace of a good run on decompilating this project. Those traces can be hosted in a place Antropic can see and then after the next model pre-training there will be a good chance the model become even better at this task.
- saagarjha 10mo agoYou need a lot of context to get the correct answer and it’s difficult to know you’ve got the correct answer among the many options.
- zdware 10mo ago
- VikingCoder 10mo agoI've been waiting for decompilation to show up in this space.
- simonw 10mo agoFor anyone else who was initially confused by this, useful context is that Snowboard Kids 2 is an N64 game. I also wasn't familiar with this terminology: > You hand it a function; it tries to match it, and you move on. In decompilation "matching" means you found a function block in the machine code, wrote some C, then confirmed that the C produces the exact same binary machine code once it is compiled. The author's previous post explains this all in a bunch more detail: https://blog.chrislewis.au/using-coding-agents-to-decompile-nintendo-64-games/ https://blog.chrislewis.au/using-coding-agents-to-decompile-...
- elitan 10mo agohelpful
- your_sweetpea 10mo agoI'd like to see this given a bit more structure, honestly. What occurs to me is constraining the grammar for LLM inference to ensure valid C89 (or close-to, as much can be checked without compilation), then perhaps experimentally switching to a permuter once/if a certain threshold is reached for accuracy of the decompiled function. Eventually some or many of these attempts would, of course, fail, and require programmer intervention, but I suspect we might be surprised how far it could go.
- ACCount37 10mo agoI don't expect constraining the grammar to do all that much for modern LLMs - they're pretty good at constraining themselves. Having it absorb the 1% of failures that's caused by grammar issues is not worth the engineering effort. The modern approach is: feed the errors back to the LLM and have it fix them.
- Animats 10mo agoIn decompilation "matching" means you found a function block in the machine code, wrote some C, then confirmed that the C produces the exact same binary machine code once it is compiled. They had access to the same C compiler used by Nintendo in 1999? And the register allocation on a MIPS CPU is repeatable enough to get an exact match? That's impressive.
- DrNosferatu 10mo agoMore than an overview, a step by step tutorial on this would be awesome!
- t_mann 10mo ago> The ‘give up after ten attempts’ threshold aims to prevent Claude from wasting tokens when further progress is unlikely. It was only partially successful, as Claude would still sometimes make dozens of attempts. Not what I would have expected from a 'one-shot'. Maybe self-supervised would be a more suitable term?
- wavemode 10mo ago"one-shot" usually just means, one example and its correct answer was provided in the prompt. See also, "zero-shot" / "few-shot" etc.
- simonw 10mo agoI've seen one-shot used to mean two different things in LLMs: 1. Getting an LLM to do something based on a single example 2. Getting an LLM to achieve a goal from a single prompt with no follow-ups I think both are equally valid.
- baq 10mo agoOne-shot as in ‘given one example’ is the ML term. One-shot as in ‘in a single prompt’ is the colloquial meaning. Both are useful, but it can be confusing when discussing LLMs in ML topics.
- t_mann 10mo agoThe article says that having decompiled some functions helps with decompiling others, so it seems like more than one example could be provided in the context. I think the OP was referring to the fact that only a single prompt created by a human was used. But then it goes off into what appears to be an agentic loop with no hard stopping conditions outside of what the agent decides. We're essentially trying to map 'traditional' ML terminology to LLMs, it's natural that it'll take some time to get settled. I just thought that one-shot isn't an ideal name for something that might go off into an arbitrarily long loop.
- hombre_fatal 10mo ago
- benmccann 10mo agoI used Gemini to compare the minimized output of the Rollup vs Rolldown JavaScript bundlers to find locations where the latter was not yet at the same degree of optimization. It was astoundingly good and I'm not sure how I would have been able to accomplish the task without an LLM as an available tool.
- deleted 10mo ago[deleted]
- sehugg 10mo agoI ran Node with --print-opt-code and had Opus look at Turbofan's output. It was able to add comments to the JIT'ed code and give suggestions on how to improve the JavaScript for better optimization.
- heavyset_go 10mo agoAm I just wrong in thinking doing decompilation of copyrighted code via the cloud is a bad idea? Like, if it ever leaks, or you were planning on releasing it, literally every step you took in your crime is uploaded to the cloud ready to send you to prison. It's what's stopped me from using hosted LLMs for DMCA-legal RE. All it takes is for a prosecutor/attorney to spin a narrative based on uploaded evidence and your ass is in court.
- Juliate 10mo agoIt wouldn't fit most of the current LLM cloud providers narrative about privacy and copyright either, so, not sure they would be as cooperative with a prosecutor as they are today with lawmakers and right holders.
- viraptor 10mo agoYeah, it works great for porting as well. I tried it on the assembler sources of Prince of Persia for Apple ii and went from nothing to basics being playable (with a few bugs but still) on modern Mac with SDL graphics within a day.
- djmips 10mo agoThat's impressive. Did you convert POP from 6502 to C?
- viraptor 10mo agoYup. Still fighting some collision bugs, but it mostly works. I'll post it when it's complete. What I actually wanted to do is try to put fluid movement into it - something closer to Dead Cells, just for fun to see how it would change the feel of it.
- ukuina 10mo agoYou might enjoy Lost Crown or Rogue Prince. They both have dashes and zippy moves.
- lomase 10mo agoI can also make a port of Prince of Persia, and faster than a LLM: https://github.com/NagyD/SDLPoP https://github.com/NagyD/SDLPoP
- xnx 10mo agoGreat use case. Curious to see how Gemini fares when tested.
- wiz21c 10mo agoLast day I asked Claude to estimate a loop of a dozen 6502 instructions. It failed but his estimate was not bad at all. Amazing!
- grim_io 10mo agoI need to try using a frontier LLM for deobfuscation. That's a huge pain in the ass for a noob like me.
- Nevermark 10mo agoThere are quite a few comments here on code obfuscation. The hardest form of code obfuscation is called homomorphic computing, which is code transformed to act on encrypted data isomorphically to regular code on regular data. The homomorphic code is hard obfuscated by this transformation. Now create a homomorphic virtual machine, that operates on encrypted code over encrypted data. Very hard to understand. Now add data encryption/decryption algorithms, both homomorphically encrypted to be run by the virtual machine, to prepare and recover inputs, outputs or effects of any data or event information, for the homomorphic application code. Now that all data within the system is encrypted by means which are hard obfuscated, running on code which is hard obfuscated, the entire system becomes hard^2 (not a formal measure) opaque. This isn't realistic in practice. Homomorphic implementations of even simple functions are extremely inefficient for the time being. But it is possible, and improvements in efficiency have not been exhausted. Equivalent but different implementations of homomorphic code can obviously be made. However, given the only credible explanations for design decisions of the new code are, to exactly match the original code, this precludes any "clean room" defenses. -- Implementing software with neural network models wouldn't stop replication, but would decompile as source that was clearly not developed independent from the original implementation. Even distilling (training a new model on the "decompiled" model) would be dead giveaway that it was derived directly from the source, not a clean room implementation. -- I have wondered, if quantum computing wouldn't enable an efficient version of homomorphic computing over classical data. Just some wild thoughts.
- YesBox 10mo agoIm a encryption noob. Less than a noob. But something I've been wondering about is how can homomorphic computing be opaque/unencryptable? If you are able to monitor what happens to encrypted data being processed by an LLM, could you not match that with the same patterns created by unencrypted data? Real simple example, let's say I have a program that sums numbers. One sends the data to an LLM or w/e unencrypted, the other encrypted. Wouldn't the same part of the LLM/compute machine "light up" so to speak?
- saagarjha 10mo agoI don’t actually think an LLM is a good way to sum numbers, but it is a pretty good example to explain the phenomenon you’re interested in. When you run an LLM, you essentially take your input and matrix multiply it with the weights to get an output. This happens regardless of what the input is or what the model “needs” to do. So, to some extent, the same part of the machine is being used every time even though the results seem very different. (Of course, the reality is much more complicated than this; you can trace things like power and in theory track individual activations with great difficulty and then do interpretability to see what the model is doing. But hopefully it illustrates that you can usually take some sort of special operation and turn it into a process that does the same operation on different data.)
- reactordev 10mo agoI’ve been having fun sending Claude down the old school MUD route, giving it access to a SMAUG derivative and once it’s mastered the play, give it admin powers to create new play experiences. I stayed away from decompilation and reverse engineering, for legal reasons. Claude is amazing. It can sometimes get stuck in a reason loop but will break away, reassess, and continue on until it finds its way. Claude was murdered in a dark instance dungeon when it managed to defeat the dragon but ran out of lamp oil and torches to find its way out. Because of the light system it kept getting “You can’t seem to see anything in the darkness” and randomly walked into a skeleton lair. Super fun to watch from an observer. Super terrifying that this will replace us at the office.
- thelittleone 10mo agoI'd enjoy watching. A new category of eSports.
- Rastonbury 10mo agonot really a game but check out AI Village by aidigest, where LLMs work together to achieve goals with a computer, the stuff they/say do can be pretty amusing
- knallfrosch 10mo agoWe're wasting Energy reverse-engineering code, which, by definition, already exists now. Oh god. Have you tried asking them to simply open source the code?
- looperhacks 10mo agoHave you ever tried to get a game developer to open source a game? And a Japanese one at that? Even if they were willing to (they're not) and if they still have the code (they don't), it will contain proprietary code from Nintendo and you'll never get your hands on that (legally)
- djmips 10mo agofrom 1999! Plus they probably don't even have the source anymore! A lot of game companies just never kept it!
- hombre_fatal 10mo agoWow, I haven't thought of this game since I played it as a kid. My friend would bring it over all the time for sleep overs. I'm going to try to emulate it right now for old time's sake. I loved this game.
- throwaway81523 10mo agoRather than insisting on byte perfect matches, sometimes you can prove code equivalence of machine code sequences using SAT solvers. That might be an interesting extension, maybe giving clearer code output and/or solution to difficult functions in some cases.