8 ms·
The Journey Before main()
- hagbard_c 11mo agoOn the subject of symbols: > Yeah, that’s it. Now, 2308 may be slightly bloated because we link against musl instead of glibc, but the point still stands: There’s a lot of stuff going on behind the scenes here. Slightly bloated is a slight understatement. The same program linked to glibc tops at 36 symbols in .symtab: $ readelf -a hello|grep "'.symtab'" Symbol table '.symtab' contains 36 entries:
- amitprasad 11mo agoAh I should have taken the time to verify; It might also have something to do with the way I was compiling / cross-compiling for RISC-V! More generally, I'm not surprised at the symtab bloat from statically-linking given the absolute size increase of the binary.
- vbezhenar 11mo agoI wonder how many C projects prefer to avoid standard library, just invoking Linux syscalls directly. Much more fun to write software this way, IMO.
- forrestthewoods 11mo agoYou had me with “avoid C standard library” but lost me at “incoming Linux syscalls directly”. Windows support is a requirement, and no WSL2 doesn’t count. C standard library is pretty bad and it’d be great if not using it was a little easier and more common.
- pmc00 11mo agoYou can do this in Windows too, useful if you want tiny executables that use minimum resources. I wrote this little systemwide mute utility for Windows that way, annoying to be missing some parts of the CRT but not bad, code here: https://github.com/pablocastro/minimute https://github.com/pablocastro/minimute
- gpm 11mo agoI thought windows had an unstable syscall interface?
- LegionMammal978 11mo agoIt looks like that project does link against the usual Windows DLLs, it just doesn't use a static or dynamic C runtime.
- pmc00 11mo agoWindows isn’t quite like Linux in that typically apps don’t make syscalls directly. Maybe you could say what’s in ntdll is the system call contract, but in practice you call the subsystem specific API, typically the Win32 API, which is huge compared to the Linux syscall list because it includes all sorts of things like UI, COM (!), etc. The project has some of the properties discussed above such as not having a typical main() (or winmain), because there’s no CRT to call it.
- Dwedit 11mo agoPretty much yeah. You have your usual Win32 API functions found in libraries like Kernel32, User32, and GDI32, but since after Windows XP, those don't actually make system calls. The actual system calls are found in NTDLL and Win32U. Lots of functions you can import, and they're basically one instruction long. Just SYSENTER for the native version, or a switch back to 64-bit mode for a WOW64 DLL. The names of the function always begin with Nt, like NtCreateFile. There's a corresponding Kernel mode call that starts with Zw instead, so in Kernel mode you have ZwCreateFile. But the system call numbers used with SYSENTER are indeed reordered every time there's a major version change to Windows, so you just call into NTDLL or Win32U instead if you want to directly make a system call.
- antihero 11mo ago> Windows support is a requirement Why, exactly?
- AnimalMuppet 11mo ago> Windows support is a requirement... For what? There is some software for which Windows support is required. There are others for which it is not, and never will be. (And for an article about running ELF files on RiscV with a Linux OS, the "Windows support" complaint seems a bit odd...)
- throwawaysoxjje 11mo agoA requirement from whom? To do what?
- rfl890 11mo agoYou can make CRT-free Win32 programs, read this guide[1] and you're all set. I've written a couple CLI utilities which are completely CRT-free and weigh just under a few kilobytes. [1]: https://nullprogram.com/blog/2023/02/15/ https://nullprogram.com/blog/2023/02/15/
- forrestthewoods 11mo agoGreat post!
- matheusmoreira 11mo agoAlmost freestanding. It still requires you to link against kernel32 and use the functions it provides. This is because issuing system calls directly to the Windows kernel is not supported. The kernel developers reserve the right to change things like system call numbers, so they can't be hardcoded into the application.
- forrestthewoods 11mo ago> Almost freestanding. It still requires you to link against kernel32 Nitpick: the phrase “link against kernel32” feels like a Linux-ism. If you’re only calling a few function you need to load kernel32.dll and call some functions in it. But that’s a slightly different operation than linking against it. At least how I’ve always used the term link. You’re not wrong in principle. But Linux and Windows do a lot of things differently wrt linking and loading libs. (I think Windows does it waaay better but ymmv)
- matheusmoreira 11mo agoLoading means creating a memory image of the library. Linking means resolving the symbols to addresses within that memory image. Loading a library and calling some functions from it is linking. The function pointer you receive is your link to the library function.
- forrestthewoods 11mo ago
- WJW 11mo agoObviously only a requirement if you intend your software to run under windows. But if you don't, why bother. Not all software is intended to be distributed to users far and wide. Some of it is just for yourself, and some of it will only ever run on linux servers.
- forrestthewoods 11mo ago> some of it will only ever run on linux servers. I’ve spent quite a lot of time dealing with code that will ever run on Linux which did not in fact only ever run on Linux! Obviously for hobby projects anyone can do what they want. But adult projects should support Windows imho and consider Windows support from the start. Cross-platform is super easy unless you choose to make it hard.
- deleted 11mo ago[deleted]
- WJW 11mo agoI don't think we are talking about the same type of software? The type I was talking about will only ever run on Linux because it's a (HTTP-ish) server that will only ever run on Linux. Probably a server that is only ever run by a single company on a single CPU type. That company will have complete control of the OS stack, so if it says no Windows, then no Windows has to be supported.
- forrestthewoods 11mo agocool
- matheusmoreira 11mo ago> But adult projects should support Windows imho and consider Windows support from the start. Hope whatever "adult" is working on the project this is getting paid handsomely. They'd certainly need to pay me big bucks to care about Windows support. In any case, Linux system call ABI is becoming a lingua franca of systems programming. BSDs have implemented Linux system calls. Windows has straight up included Linux in the system. It looks like simply targeting Linux can easily result in a binary that actually does run anywhere.
- jjmarr 11mo agoTons of driver code does this.
- 1718627440 11mo agoI generally try to stay portable, but file descriptors are just to nice, to not use them.
- Retr0id 11mo agoFile descriptors are part of the linux syscall API, not libc. Are you thinking of FILE?
- 1718627440 11mo agoI did mean file descriptors.
- Retr0id 11mo agoThen I'm confused by what you meant, because you can use fds with or without libc.
- 1718627440 11mo agoI don't want to bypass libc in general, because I care about portability, but fds are just a nice interface, so I still use them instead of FILE, which would be the portable choice. My calls are still subject to OS choices, that differ from the kernel, since I don't bypass libc.
- ajross 11mo agoThe "syscall API" is part of libc too. The read syscall is a trap, you put arguments in the right registers and issue the correct instruction[1] to enter the kernel. That's not something that can be expressed in C. The read() function that your C code actually uses is a C function provided by the C library. [1] "svc 0" on ARM, "int 0x80" on i386, etc...
- Retr0id 11mo agosyscalls are an implementation detail of some libc impls on some platforms, but the C spec does not mention syscalls.
- electroly 11mo agoNot exactly the same, but on Windows if you use entirely Win32 calls you can avoid linking any C runtime library. Win32 is below the C standard library on Windows and the C runtime is optional.
- okanat 11mo agoThis is one of the cornerstones that guarantee Windows can easily upgrade the C runtime and make performance and security upgrades. Win32 APIs have a different function calling ABI too. So only part of that gets "bloated" is Win32 API itself (which is spread across multiple DLLs and don't actually bloat RAM usage). Most of the time even those functions and structures are carefully designed to have some future-proofness but it is usual to see APIs like CreateFile, CreateFile2, CreateFile3. Internally the earlier versions are upgraded to call the latest version. So not so much bloating there either. When the C runtime and the OS system calls are combined into the single binary like POSIX, it creates the ABI hell we're in with the modern Unix-likes. Either the OSes have to regularly break the C ABI compatibility for the updates or we have to live with terrible implementations. GNU libc and Linux combo is particularly bad. On GNU/Linux (or any other current libc replacements), the dynamic loading is also provided by the C library. This makes "forever" binary file compatibility particularly tricky to achieve. Glibc broke certain games / Steam by removing some parts of their ELF implementation: https://sourceware.org/bugzilla/show_bug.cgi?id=32653 https://sourceware.org/bugzilla/show_bug.cgi?id=32653 . They backed due to huge backlash from the community. If "the year of Linux desktop" would ever happen, they need to either do an Android and change the definition of what a software package is, or split Glibc into 3 parts: syscalls, dynamic loader and the actual C library. PS: There is actually a catch to your " C runtime is optional." argument. Microsoft still intentionally holds back the ability of compiling native ABI Windows programs without Visual Studio. The structured exception handlers (equivalent of Windows for SIGILL, SIGBUS etc.. not for SIGINT or SIGTERM though) are populated by the object files from the C runtime libraries (called VCRuntime/VCStartup). So it is actually not possible to have official Windows binaries without MSVC or any other C runtime like Mingw-64 that provides those symbols. It looks like some developers in Microsoft wanted to open-source VCRuntime / VCStartup but it was ~vetoed~ not fully approved by some people: https://github.com/microsoft/STL/issues/4560#issuecomment-2348622212 https://github.com/microsoft/STL/issues/4560#issuecomment-23... , https://www.reddit.com/r/cpp/comments/1l8mqlv/is_msvc_ever_going_open_source/mxa2m8k/ https://www.reddit.com/r/cpp/comments/1l8mqlv/is_msvc_ever_g...
- matheusmoreira 11mo agoI once wrote a liblinux project just for this!! It was indeed extremely fun. Details in my other comment: https://news.ycombinator.com/item?id=45709141 https://news.ycombinator.com/item?id=45709141 I abandoned it because Linux itself now has a rich set of nolibc headers. Now I'm working on a whole programming language based around this concept. A freestanding lisp interpreter targeting Linux directly with builtin system call support. The idea is to complete the interpreter and then write the standard library and Linux user space in lisp using the system calls. It's been an amazing journey. It's incredible how far one can take this.
- codedokode 11mo agoI think using syscalls directly is a worse idea than loading shared libraries, and new kernel features, like ALSA (audio playback), DRM (graphics rendering) and other use libraries instead of documenting syscalls and ioctls. This is better because it allows intercepting and subverting the calls, adding support for features even if the kernel doesn't support it, makes it easier to port code to other OSes, support different architectures (32-bit code on 64-bit kernel), and allows changing kernel interface without breaking anything. So Windows-style approach with system libraries is better in every aspect.
- mmsc 11mo agoIt's also possible to pack a whole codebase into "before main()" - or with no main() at all. I was recently experimenting doing this, as well as a whole codebase that only uses main() and calls itself over and over. Good fun: https://joshua.hu/packing-codebase-into-single-function-disrupt-reverse-engineering https://joshua.hu/packing-codebase-into-single-function-disr...
- 1718627440 11mo agoThat is a really fun read and honestly doesn't even seem to be complicated and brittle. Just rename every function to main(100+n, ...).
- thatxliner 11mo agoJust wondering, how did you get that domain name? I’ve been looking for registrars offering .hu
- hashstring 11mo agowhois data points to https://www.domain.hu https://www.domain.hu.
- slater 11mo agohttps://nic.hu/index_en.html https://nic.hu/index_en.html ?
- khaledh 11mo ago> A note on interpreters: If the executable file starts with a shebang (#!), the kernel will use the shebang-specified interpreter to run the program. For example, #!/usr/bin/python3 will run the program using the Python interpreter, #!/bin/bash will run the program using the Bash shell, etc. This caused me a lot of pain while trying to debug a 3rd party Java application that was trying to launch an executable script, and throwing an IO error "java.io.IOException: error=2, No such file or directory." I was puzzled because I know the script is right there (using its full path) and it had the executable bit set. It turns out that the shebang in the script was wrong, so the OS was complaining (actual error from a shell would be "The file specified the interpreter '/foo/bar', which is not an executable command."), but the Java error was completely misleading :| Note: If you wonder why I didn't see this error by running the script myself: I did, and it ran fine locally. But the application was running on a remote host that had a different path for the interpreter.
- mscdex 11mo agoAlso be aware that kernel support for shebangs depends on CONFIG_BINFMT_SCRIPT=y being in the kernel config.
- 1718627440 11mo agoNote, that this is not a Java specific problem, it can occur with other programs as well. "No such file or directory" is just the nice description for ENOENT, which can occur in a lot of syscalls. I typically just run the program through strace, then you will quickly see what the program did.
- gjf 11mo agoFor those interested, I did a breakdown of the hashbang: https://blog.foletta.net/post/2021-04-19-what-the/ https://blog.foletta.net/post/2021-04-19-what-the/
- itopaloglu83 11mo agoI like doing this with old microcontrollers like PIC16 series etc. You said see how to stack pointer, timers, and variables etc. all are configured.
- fweimer 11mo ago> The ELF file contains a dynamic section which tells the kernel which shared libraries to load, and another section which tells the kernel to dynamically “relocate” pointers to those functions, so everything checks out. This is not how dynamic linking works on GNU/Linux. The kernel processes the program headers for the main program (mapping the PT_LOAD segments, without relocating them) and notices the PT_INTERP program interpreter (the path to the dynamic linker) among the program headers. The kernel then loads the dynamic linker in much the same way as the main program (again without relocation) and transfers control to its entry point. It's up to the dynamic linker to self-relocate, load the referenced share objects (this time using plain mmap and mprotect, the kernel ELF loader is not used for that), relocate them and the main program, and then transfer control to the main program. The scheme is not that dissimilar to the #! shebang lines, with the dynamic linker taking the role of the script interpreter, except that ELF is a binary format.
- amitprasad 11mo agoYou’re right, and I knew this back in February when I wrote most of this post. I must have revised it down incorrectly before posting; will correct. Bit of a facepalm from my side.
- mkoubaa 11mo agoI've always wondered why there weren't more popular loaders to choose from given that on Linux loaders are user-space
- BobbyTables2 11mo agoI suspect it is because they get really hairy. Loading ELFs and processing relocations is actually not too bad. It’s fun after the initial learning curve. Then one has to worry about handling of “dlopen” and the loader creating the data structures it cares about. Yuck!!! It’s kinda a shame because the glibc loader is a bit bloated with all the audit and preload handling. Great for flexibility, not for security.
- matheusmoreira 11mo ago
- turbert 11mo agoIts been a while since I've touched this stuff but my recollection is the ELF interpreter (ldso, not the kernel) is responsible for everything after mapping the initial ELF's segments. iirc execve maps pt_load segments from the program header, populates the aux vector on the stack, and jump straight to the ELF interpreter's entry point. Any linked objects are loaded in userspace by the elf interpreter. The kernel has no knowledge of the PLT/GOT.
- matheusmoreira 11mo agoThat's right! https://lwn.net/Articles/631631/ https://lwn.net/Articles/631631/ https://github.com/torvalds/linux/blob/master/fs/binfmt_elf.c https://github.com/torvalds/linux/blob/master/fs/binfmt_elf.... Especially relevant for dynamic linkers is the AT_PHDR and AT_BASE auxiliary vector entries which provide the address of the executable's program header table and the address of the interpreter, respectively. https://lwn.net/Articles/519085/ https://lwn.net/Articles/519085/
- archmaster 11mo agoThis is awesome! To anyone interested in learning more about this, I wrote https://cpu.land/ https://cpu.land/ a couple years ago. It doesn't go as in-depth into e.g. memory layout as OP does but does cover multitasking and how the code is loaded in the first place.
- fuzzy_biscuit 11mo agoI love cpu.land! Thanks for creating such a fun resource.
- bignerd_95 11mo agoAs someone who teaches this stuff at university, I see students getting confused every single year by how textbooks draw memory. The problem is mostly visual, not conceptual. Most diagrams in books and slides use an old hardware-centric convention: they draw higher addresses at the top of the page and lower addresses at the bottom. People sometimes justify this with an analogy like “floors in a building go up,” so address 0x7fffffffe000 is drawn “higher” than 0x400000. But this is backwards from how humans read almost everything today. When you look at code in VS Code or any other IDE, line 1 is at the top, then line 2 is below it, then 3, 4, etc. Numbers go up as you go down. Your brain learns: “down = bigger index.” Memory in a real Linux process actually matches the VS Code model much more closely than the textbook diagrams suggest. You can see it yourself with: cat /proc/$$/maps (pick any PID instead of $$). ... [0x00000000] lower addresses ... [0x00620000] HEAP start [0x00643000] HEAP extended ↓ (more allocations => higher addresses) ... [0x7ffd8c3f7000] STACK top (<- stack pointer) ↑ the stack pointer starts here and moves upward (toward lower addresses) when you push [0x7ffd8c418000] STACK start ... [0xffffffffff600000] higher addresses ... The output is printed from low addresses to high addresses. At the top of the output you'll usually see the binary, shared libs, heap, etc. Those all live at lower virtual addresses. Farther down in the output you'll eventually see the stack, which lives at a higher virtual address. In other words: as you scroll down, the addresses get bigger. Exactly like scrolling down in an editor gives you bigger line numbers. The phrases “the heap grows up” and “the stack grows down” aren't wrong. They're just describing what happens to the numeric addresses: the heap expands toward higher addresses, and the stack moves into lower addresses. The real problem is how we draw it. We label “up” on the page as “higher address,” which is the opposite of how people read code or even how /proc/<pid>/maps is printed. So students have to mentally flip the diagram before they can even think about what the stack and heap are doing. If we just drew memory like an editor (low addresses at the top, high addresses further down) it would click instantly. Scroll down, addresses go up, and the stack sits at the bottom. At that point it’s no longer “the stack grows down”: it’s just the stack pointer being decremented, moving to lower addresses (which, in the diagram, means moving upward).
- 1718627440 11mo agoThat's how stacks on my desk grow and how everything grows in reality. I wouldn't numerate stacked things on my desk from the top, since this constantly changes. You also wouldn't name the first branch of a tree (the plant) to be the top-most one. In your example "the stack grows down", seems to be wrong in the image.
- ramanvarma 11mo agodid you see the relocations for the main binary applied before or after the linker resolves its own symbols? the ordering always feels like black magic when you step through it in a debugger
- matheusmoreira 11mo agoHacking this stuff is so fun!! > Depending on your program, _start may be the only thing between the entrypoint and your main function I once developed a liblinux project entirely built around this idea. I wanted to get rid of libc and all of its initialization, complexity and global state. The C library is so complex it has a primitive form of package management built into it: https://blogs.oracle.com/solaris/post/init-and-fini-processing-who-designed-this https://blogs.oracle.com/solaris/post/init-and-fini-processi... So I made _start functions which did nothing but pass argc, argv, envp and auxv to the actual main function: https://github.com/matheusmoreira/liblinux/blob/master/start/x86_64/_start.S https://github.com/matheusmoreira/liblinux/blob/master/start... https://github.com/matheusmoreira/liblinux/blob/master/start/x86_64/liblinux_start.c https://github.com/matheusmoreira/liblinux/blob/master/start... You can get surprisingly far with just this, and it's actually possible to understand what's going on. Biggest pain point was the lack of C library utility functions like number/string conversion. I simply wrote my own. https://github.com/matheusmoreira/liblinux/tree/master/examples https://github.com/matheusmoreira/liblinux/tree/master/examp... Linux is the only operating system that lets us do this. In other systems, the C library is part of the kernel interface. Bypassing it like this can and does break things. Go developers once discovered this the hard way. https://www.matheusmoreira.com/articles/linux-system-calls https://www.matheusmoreira.com/articles/linux-system-calls The kernel has their own nolibc infrastructure now, no doubt much better than my project. https://github.com/torvalds/linux/tree/master/tools/include/nolibc https://github.com/torvalds/linux/tree/master/tools/include/... I encourage everyone to use it. Note also that _start is an arbitrary symbol. The name is not special at all. It's just some linker default. The ELF header contains a pointer to the entry point, not a symbol. Feel free to choose a nice name!
- yawpitch 11mo agoYou’ve got a broken link in your markdown, round about the phrase “lang_start function (defined here)”.
- Animats 11mo agoFrom the title, I thought this was going to be about the parts of a program that run before the main function is entered. Static objects have to be constructed. Quite a bit of code can run. Order of initialization can be a problem. What happens if you try to do I/O from a static constructor? Does that even work?
- amitprasad 11mo agoThis is heavily language runtime dependent — there’s nothing that fundamentally stops you from doing anything during the phase between jumping to an entry point and the main()
- abnercoimbre 11mo agoIndeed the craziest among us occasionally abuse this fact, so long as the compiler implementation lets us.
- Animats 11mo agoRight. This tends to come up with packages, which just by virtue of being loaded, set up to do something such as log, print, catch errors, or phone home to something.
- nneonneo 11mo agoFor a fun example of a crash that can occur before main() even starts: https://stackoverflow.com/questions/12570374/floating-point-exception-sigfpe-on-int-main-return0 https://stackoverflow.com/questions/12570374/floating-point-... The poster was receiving a SIGFPE (floating point exception) on a C program that is simply “int main() { return 0; }”. A fun little mystery to dive into!