9 ms·

This is why round-tripping the code is important. If you decompile the binary to source, then compile the source back to binary you should get the original bin

by Eager 3y ago

This is why round-tripping the code is important. If you decompile the binary to source, then compile the source back to binary you should get the original binary. You just need to do this enough times until the loss drops to some acceptable amount. It's a great task for reinforcement learning, which is known to be unreasonably effective for these types of problems.

thfuran 3y ago
>If you decompile the binary to source, then compile the source back to binary you should get the original binary. You really can't expect that if you're not using exactly the same version of exactly the same compiler with exactly the same flags, and often not even then.
- Eager 3y ago
  You try your best, and if you provide enough examples, it will undoubtedly get figured out.
  - deleted 3y ago
    [deleted]
  - thfuran 3y ago
    What exactly are you suggesting will get figured out?
    - spqrr 3y ago
      The mapping from binary to source code.
      - layer8 3y ago
        The question was about the reverse mapping.
      - thfuran 3y ago
        Even ignoring all sources of irreproducibility, there does not exist a bijection between source and binary artifact irrespective of tool chain. Two different toolchains could compile the same source to different binaries or different sources to the same binary. And you absolutely shouldn't be ignoring sources of irreproducibility in this context, since they'll cause even the same toolchain to keep producing different binaries given the same source.
        achrono 3y ago
        Exactly, but neither the source nor the binary is what's truly important here. The real question is: can the LLM generate the functionally valid source equivalent of the binary at hand? If I disassemble Microsoft Paint, can I get code that will result in a mostly functional version of Microsoft Paint, or will I just get 515 compile errors instead?
        Brian_K_White 3y ago
        This is what I thought the question was really about. I assume that an llm will simply see patterns that look similar to other patterns and make assosciations and assume ewuivalences on that level, meanwhile real code is full of things where the programmer, especially assembly programmers, modify something by a single instruction or offset value etc to get a very specific and functionally important result. Often the result is code that not only isn't obvious, it's nominaly flatly wrong, violating standards, specs, intended function, datasheet docs, etc. If all you knew were the rules written in the docs, the code is broken and invalid. Is the llm really going to see or understand the intent of that? They find matching patterns in other existing stuff, and to the user who can not see the infinite body of that other stuff the llm pulled from, it looks like the llm understood the intent of a question, but I say it just found the prior work of some human who understood a similar intent somewhere else. Maybe an llm or some other flavor of ai can operate some other way like actually playing out the binary like executing in a debugger and map out the results not just look at the code as fuzzy matching patterns. Can that take the place of understanding the intents the way a human would reading the decompiled assembly? Guess we'll be finding out sooner of later since of course it will all be tried.
  - fao_ 3y ago
    Except LLMs cannot reason.
    - LoganDark 3y ago
      LLMs can mimic past examples of reasoning from the dataset. So, it can re-use reasoning that it has already been trained on. If the network manages to generalize well enough across its training data, then it can get close to reproducing general reasoning. But it can't yet fully get there, of course.
      - mrtesthah 3y ago
        Do you have evidence LLMs can indeed generalize outside their training data distribution? https://twitter.com/abacaj/status/1721223737729581437/photo/1 https://twitter.com/abacaj/status/1721223737729581437/photo/...
        LoganDark 3y ago
        No. I know only that they can generalize within it, and only to a limited degree, but don't have solid evidence of even that.
        fao_ 2y ago
        So what you're saying is there's tenuous-at-best, non-"solid" evidence that LLMs can reason even within their training data. And yet I'm currently sitting at -1 for stating the blisteringly obvious. Lmao
        LoganDark 2y ago
        Yes, that's basically what I'm saying. Just less bluntly. It's slightly more nuanced than "LLMs cannot reason" because lines of reasoning are often in their dataset and can sometimes be used by the model. It's just that the model can't be relied on to know the correct reasoning to use in a given situation.
  - lolinder 3y ago
    I think you're misunderstanding OP's objection. It's not simply a matter of going back and forth with the LLM until eventually (infinite monkeys on typewriters style) it gets the same binary as before: Even if you got the exact same source code as the original there's still no automated way to tell that you're done because the bits you get back out of the recompile step will almost certainly not be the same, even if your decompiled source were identical in every way. They might even vary quite substantially depending on a lot of different environmental factors. Reproducible builds are hard to pull off cooperatively, when you control the pipeline that built the original binary and can work to eliminate all sources of variation. It's simply not going to happen in a decompiler like this.
    - blagie 3y ago
      Well, no, but yes. The critical piece is that this can be done in training. If I collect a large number of C programs from github, compile them (in a deterministic fashion), I can use that as a training, test, and validation set. The output of the ML ought to compile to the same way given the same environment. Indeed, I can train over multiple deterministic build environments (e.g. different compilers, different compiler flags) to be even more robust. The second critical piece is that for something like a GAN, it doesn't need to be identical. You have two ML algorithms competing: - One is trying to identify generated versus ground-truth source code - One is trying to generate source code Virtually all ML tasks are trained this way, and it doesn't matter. I have images and descriptions, and all the ML needs to do is generate an indistinguishable description. So if I give the poster a lot more benefit of the doubt on what they wanted to say, it can make sense.
      - lolinder 3y ago
        Oh, I was assuming that Eager was responding to klik99's question about how we could identify hallucinations in the output—round tripping doesn't help with that. If what they're actually saying is that it's possible to train a model to low loss and then you just have to trust the results, yes, what you say makes sense.
        3y ago
  - junon 3y ago
    Err, no, sorry, it won't. Compilers don't work that way. There's a lot of ways to compile down source to machine code and the output changes from compiler version to compiler version. The LLM would have to know exactly how the compiler worked at which version to do this. So the idea is technically possible but not technically feasible.
- dheera 3y ago
  Maybe we then need an LLM to tell us if two pieces of compiled code are equivalent in an input-output mapping sense (ignoring execution time). I'm actually serious; it would be exceedingly easy to get training data for this just by running the same source code through a bunch of different compiler versions and optimization flags.
  - thfuran 3y ago
    Why would an llm be the tool for that job?
    - dheera 3y ago
      Without analytical thinking how else would you come to conviction that two functions are identical, for a computationally unfeasible number of possible inputs?
      - kolinko 3y ago
        Formal logic / formal proofs. We have good systems for verifying that. The proper flow is that you use LLM to generate decompilation steps, along with potential proofs, and then use old algorithms from 1970s that verify that the steps are correct. Source: I built a decompiler for EVM, arguably the best one on the market, and to some extent it was how it worked (and others comparable in class). The issue was always the exploration of possible transformations of code, once you manage to find the right ones (which LLMs can propose way better than old hard coded rules and SMT solvers), it's simple to verify that the transformations are correct.
  - saagarjha 3y ago
    An LLM cannot do this. I don’t even mean this in a formal sense, because your problem is addressed by Rice’s Theorem, which places bounds on what any system (LLM or not) can do here; I mean it in the sense that an LLM isn’t even appropriate to use here because the best it can possibly do is provide you with its best guess at the answer. And while this might be a useful property for decompilation in general that’s not what was being discussed here.
- vasvir 3y ago
  Right. A less formidable problem with higher chances of succeeding is from a given binary to figure out first compiler, compiler-version, compiler-flags. From there you could have a model for every combination or at least a model for the compiler variant and use the other info (version, flags) as input to the model.
- moyix 3y ago
  Yes, that's a limitation of trying to ensure exact binary reconstruction. Luckily there is also a separate line of work on detecting the compiler version and optimization flags based on a binary – it turns out this is not that hard and it's easy to get a bunch of labeled data for a classifier. If folks are interested in reading more there's a nice paper by Grammatech on the idea: https://eschulte.github.io/data/bed.pdf https://eschulte.github.io/data/bed.pdf (though it's pre-LLM and uses evolutionary algorithms on the initial decompilation to search for a version that recompiles exactly).
codethief 3y ago
> you should get the original binary According to the project's README, they only seem to be checking mere "re-compilability" and "re-executability" of the decompiled code, though.
1024core 3y ago
> If you decompile the binary to source, then compile the source back to binary you should get the original binary. Doesn't that depend on the compiler's version though? Or, for that matter, even the sub-version. Every compiler does things differently.
- fulafel 3y ago
  From the README: > By re-compiling the decompiled output and running the test cases, we assess if the decompilation preserved the program logic and behavior. As this is in the metrics section, I guess fully automating this is not part of the research.

by Eager 3y ago

thfuran 3y ago

>If you decompile the binary to source, then compile the source back to binary you should get the original binary. You really can't expect that if you're not using exactly the same version of exactly the same compiler with exactly the same flags, and often not even then.

Eager 3y ago
You try your best, and if you provide enough examples, it will undoubtedly get figured out.
- deleted 3y ago
  [deleted]
- thfuran 3y ago
  What exactly are you suggesting will get figured out?
  - spqrr 3y ago
    The mapping from binary to source code.
    - layer8 3y ago
      The question was about the reverse mapping.
    - thfuran 3y ago
      Even ignoring all sources of irreproducibility, there does not exist a bijection between source and binary artifact irrespective of tool chain. Two different toolchains could compile the same source to different binaries or different sources to the same binary. And you absolutely shouldn't be ignoring sources of irreproducibility in this context, since they'll cause even the same toolchain to keep producing different binaries given the same source.
      - achrono 3y ago
        Exactly, but neither the source nor the binary is what's truly important here. The real question is: can the LLM generate the functionally valid source equivalent of the binary at hand? If I disassemble Microsoft Paint, can I get code that will result in a mostly functional version of Microsoft Paint, or will I just get 515 compile errors instead?
        Brian_K_White 3y ago
        This is what I thought the question was really about. I assume that an llm will simply see patterns that look similar to other patterns and make assosciations and assume ewuivalences on that level, meanwhile real code is full of things where the programmer, especially assembly programmers, modify something by a single instruction or offset value etc to get a very specific and functionally important result. Often the result is code that not only isn't obvious, it's nominaly flatly wrong, violating standards, specs, intended function, datasheet docs, etc. If all you knew were the rules written in the docs, the code is broken and invalid. Is the llm really going to see or understand the intent of that? They find matching patterns in other existing stuff, and to the user who can not see the infinite body of that other stuff the llm pulled from, it looks like the llm understood the intent of a question, but I say it just found the prior work of some human who understood a similar intent somewhere else. Maybe an llm or some other flavor of ai can operate some other way like actually playing out the binary like executing in a debugger and map out the results not just look at the code as fuzzy matching patterns. Can that take the place of understanding the intents the way a human would reading the decompiled assembly? Guess we'll be finding out sooner of later since of course it will all be tried.
- fao_ 3y ago
  Except LLMs cannot reason.
  - LoganDark 3y ago
    LLMs can mimic past examples of reasoning from the dataset. So, it can re-use reasoning that it has already been trained on. If the network manages to generalize well enough across its training data, then it can get close to reproducing general reasoning. But it can't yet fully get there, of course.
    - mrtesthah 3y ago
      Do you have evidence LLMs can indeed generalize outside their training data distribution? https://twitter.com/abacaj/status/1721223737729581437/photo/1 https://twitter.com/abacaj/status/1721223737729581437/photo/...
      - LoganDark 3y ago
        No. I know only that they can generalize within it, and only to a limited degree, but don't have solid evidence of even that.
        fao_ 2y ago
        So what you're saying is there's tenuous-at-best, non-"solid" evidence that LLMs can reason even within their training data. And yet I'm currently sitting at -1 for stating the blisteringly obvious. Lmao
        LoganDark 2y ago
        Yes, that's basically what I'm saying. Just less bluntly. It's slightly more nuanced than "LLMs cannot reason" because lines of reasoning are often in their dataset and can sometimes be used by the model. It's just that the model can't be relied on to know the correct reasoning to use in a given situation.
- lolinder 3y ago
  I think you're misunderstanding OP's objection. It's not simply a matter of going back and forth with the LLM until eventually (infinite monkeys on typewriters style) it gets the same binary as before: Even if you got the exact same source code as the original there's still no automated way to tell that you're done because the bits you get back out of the recompile step will almost certainly not be the same, even if your decompiled source were identical in every way. They might even vary quite substantially depending on a lot of different environmental factors. Reproducible builds are hard to pull off cooperatively, when you control the pipeline that built the original binary and can work to eliminate all sources of variation. It's simply not going to happen in a decompiler like this.
  - blagie 3y ago
    Well, no, but yes. The critical piece is that this can be done in training. If I collect a large number of C programs from github, compile them (in a deterministic fashion), I can use that as a training, test, and validation set. The output of the ML ought to compile to the same way given the same environment. Indeed, I can train over multiple deterministic build environments (e.g. different compilers, different compiler flags) to be even more robust. The second critical piece is that for something like a GAN, it doesn't need to be identical. You have two ML algorithms competing: - One is trying to identify generated versus ground-truth source code - One is trying to generate source code Virtually all ML tasks are trained this way, and it doesn't matter. I have images and descriptions, and all the ML needs to do is generate an indistinguishable description. So if I give the poster a lot more benefit of the doubt on what they wanted to say, it can make sense.
    - lolinder 3y ago
      Oh, I was assuming that Eager was responding to klik99's question about how we could identify hallucinations in the output—round tripping doesn't help with that. If what they're actually saying is that it's possible to train a model to low loss and then you just have to trust the results, yes, what you say makes sense.
      - 3y ago
- junon 3y ago
  Err, no, sorry, it won't. Compilers don't work that way. There's a lot of ways to compile down source to machine code and the output changes from compiler version to compiler version. The LLM would have to know exactly how the compiler worked at which version to do this. So the idea is technically possible but not technically feasible.
dheera 3y ago
Maybe we then need an LLM to tell us if two pieces of compiled code are equivalent in an input-output mapping sense (ignoring execution time). I'm actually serious; it would be exceedingly easy to get training data for this just by running the same source code through a bunch of different compiler versions and optimization flags.
- thfuran 3y ago
  Why would an llm be the tool for that job?
  - dheera 3y ago
    Without analytical thinking how else would you come to conviction that two functions are identical, for a computationally unfeasible number of possible inputs?
    - kolinko 3y ago
      Formal logic / formal proofs. We have good systems for verifying that. The proper flow is that you use LLM to generate decompilation steps, along with potential proofs, and then use old algorithms from 1970s that verify that the steps are correct. Source: I built a decompiler for EVM, arguably the best one on the market, and to some extent it was how it worked (and others comparable in class). The issue was always the exploration of possible transformations of code, once you manage to find the right ones (which LLMs can propose way better than old hard coded rules and SMT solvers), it's simple to verify that the transformations are correct.
- saagarjha 3y ago
  An LLM cannot do this. I don’t even mean this in a formal sense, because your problem is addressed by Rice’s Theorem, which places bounds on what any system (LLM or not) can do here; I mean it in the sense that an LLM isn’t even appropriate to use here because the best it can possibly do is provide you with its best guess at the answer. And while this might be a useful property for decompilation in general that’s not what was being discussed here.
vasvir 3y ago
Right. A less formidable problem with higher chances of succeeding is from a given binary to figure out first compiler, compiler-version, compiler-flags. From there you could have a model for every combination or at least a model for the compiler variant and use the other info (version, flags) as input to the model.
moyix 3y ago
Yes, that's a limitation of trying to ensure exact binary reconstruction. Luckily there is also a separate line of work on detecting the compiler version and optimization flags based on a binary – it turns out this is not that hard and it's easy to get a bunch of labeled data for a classifier. If folks are interested in reading more there's a nice paper by Grammatech on the idea: https://eschulte.github.io/data/bed.pdf https://eschulte.github.io/data/bed.pdf (though it's pre-LLM and uses evolutionary algorithms on the initial decompilation to search for a version that recompiles exactly).

codethief 3y ago

> you should get the original binary According to the project's README, they only seem to be checking mere "re-compilability" and "re-executability" of the decompiled code, though.

1024core 3y ago

> If you decompile the binary to source, then compile the source back to binary you should get the original binary. Doesn't that depend on the compiler's version though? Or, for that matter, even the sub-version. Every compiler does things differently.

fulafel 3y ago
From the README: > By re-compiling the decompiled output and running the test cases, we assess if the decompilation preserved the program logic and behavior. As this is in the metrics section, I guess fully automating this is not part of the research.