5 ms·
What are you aiming to reverse-engineer? Applications written in C/C++? Java/.NET? Aged firmware (8/16 bit asm)? ARM code? I have some experience with the C/C+
by DCoder 16y ago
What are you aiming to reverse-engineer? Applications written in C/C++? Java/.NET? Aged firmware (8/16 bit asm)? ARM code?
I have some experience with the C/C++ part, so I'll try to answer that. Though this is more of a "what to learn in what order" post than a "how to learn it" one. Sorry, I'm mostly self-taught so can't really recommend exhaustive resources.
0: Intel's manuals are great, but mind that they're a reference, not beginner's books. Also, Intel used to give away dead tree versions of their manuals for free, unfortunately I don't think they do that anymore.
1: I don't think you need to learn all the ins and outs of assembly at first - knowing the basic flow control and math/logic operations will be enough. Don't worry about interrupts, descriptor tables, and other magic yet.
2: Once you learn the basics, practice: try to write simple programs in C and step through them in an IDE while looking at the disassembly view. Add inline assembly and practice writing it too.
3: After that, you'll probably need to learn how C++ constructs get translated to assembly code, which is a really large topic. Stanley Lippman's "C++ Object Model" covers quite a lot of it by showing how C++ gets transformed into C-like code, which you should already be familiar with. Mind that each compiler does some things differently than the others, so you should look at at least G++ and MSVC++.
When you get here, it's a good idea to browse around blogs like http://www.nynaeve.net/ http://www.nynaeve.net/ and http://www.pagetable.com/ http://www.pagetable.com/ .
4: OS specifics and abstractions. Learn what the OS won't let you do in user mode, why, and so on.
5: Code injection. If you want to add/change code in an existing program, you'll either need to patch its binaries or use an existing code injection program to do it automatically. The second approach is nicer when your changes need to be propagated to other installations of that program, and they don't require modifications to the actual program, which can be nice when it does checksumming/updates itself. Several "gray" methods exist to do this, something like [1] might be a good start.
Tools: Aside from a C/C++ toolchain (no assembler as such - I don't think raw assembly programs are a good idea, and prefer pieces of inline assembly in a C++ program, YMMV), you'll definitely need a disassembler and a debugger. Personally, I love IDA Pro (it does both) with its graphing features and extendability. Their decompiler is a fabulous piece of work too, even though it's not perfect.
(In fact, my friends and I are currently reverse-engineering a certain game and fixing/extending it. This is a really fun and enlightening hobby - we need to work out what the functions/objects do, map out their signatures in C++ , fix any inconsistencies with inline ASM, resulting in interesting macro collections like [2], and then write code using them without breaking stuff. Fun, for certain #define of fun.)
[1]http://www.memoryhacking.com/Misc/Tut/Injection%20Manager.htm http://www.memoryhacking.com/Misc/Tut/Injection%20Manager.ht...
[2]http://svn.renegadeprojects.com/filedetails.php?repname=YR%2B%2B&path=%2Ftrunk%2FASMMacros.h http://svn.renegadeprojects.com/filedetails.php?repname=YR%2...