9 ms·

It’s a simple and practical solution. It irks me for some reason but I can’t think of something better.

TheSoftwareGuy 7y ago
It irks you because storing source code directly in the final binary is bonkers, but unfortuantely it is the only way to reveal all the complexity that can be exposed in a C header file. This isn't really just a C problem though. Even rust has a similar problem for propagating macros. The reason for this limitation is really that making an ABI for metaprograms (macros) is exceedingly difficult
- colechristensen 7y ago
  Are there other languages that provide interfaces as cleanly as exposing C header files?
  - java-man 7y ago
    java interfaces. I think the concept of coding to an interface (or an API specification) goes beyond what java did. In my professional life, I see people constantly downplaying and ignoring importance of a formal and stable API specification, just to suffer consequences later.
    - colechristensen 7y ago
      If you just give me a .jar do I get those interfaces in a consumable form? Said differently, can I download your .jar and write my own code to interface with it while on a desert island without any other resources?
      - Mister_Snuggles 7y ago
        Yes. If you have a random JAR file, an IDE can introspect it to see what classes are in it and what methods are in those classes. The only exception is if you run it through some kind of obfuscator first.
        Asooka 7y ago
        Yes but JAR files are basically equivalent to source code parsed and serialised as bytecode with only the most rudimentary optimisations applied. I don't really see a difference between that and carrying gzipped headers with the kernel. Yes, the .class format is cleaner, but it's still very close to what the programmer wrote. Another plus to carrying the header with the kernel is that you can carry the preprocessed header to make sure the user doesn't set the wrong defines, and you can add compiler checks to check that the user isn't using a too old or too new compiler with incompatible ABI.
        java-man 7y ago
        I suppose the real difference is that with java platform, you get 1. 1/10 or 1/100 compile time 2. full introspection in regards to the interfaces (the topic of this discussion). the implementation classes could be obfuscated, but the point of an interface (and an API in general) that it is not. 3. one could possibly allow multiple versions of the interface be present on the same machine at the same time. for example, when SomeInterfaceV1 is replaced (or extended) by SomeInterfaceV2, it might be possible to provide an adapter that publishes the older interface for backward compatibility. these are just random thoughts, because the difference between current systems written in c/c++ and a new architecture that uses interface-based approach is too great, I don't think I can name a real system that uses it in practice. done the API versioning myself several times with complex applications though.
mikepurvis 7y ago
It's too bad the squashfs idea was dropped. Having to download and unpack the archive somewhere seems like an extra step that should be unnecessary vs just loading the module and having the tree appear in /sys for you to point your compiler directly at. Also a bummer that it looks like you have to be running the kernel to get access to this— perhaps the archive could be marked off in the binary somehow so that there's a way to access it for non-running kernels (eg, the dkms use case of wanting to build the module for that kernel you just installed ahead of booting it for the first time). Overall binary size is a legit concern, but if this is going to be in an optionally-loaded module anyway, it seems weird to make that call just based on memory usage. I imagine most distro kernel builds will just disable this, since they already ship the headers in separate packages.
- tych0 7y ago
  squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me. Hopefully someone picks it up :)
  - hawski 7y ago
    Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for my use, but I think it would be generally useful. Sort list file format is funky. It's: "filename_string priority_integer", so it will not work for files with spaces. Also new lines, but that's far less common. It was not yet a problem for me and I can always patch it more and one can say, that people having files with spaces deserve it, though it's entirely different issue.
ori_b 7y ago
Debug symbols, which can be resolved at load time. This would also make ebpf bytecode more or less kernel version independent.
JoshTriplett 7y ago
Doesn't necessarily work for the kernel, but GIR and typelibs provide machine-readable descriptions of C APIs: https://github.com/GNOME/gobject-introspection https://github.com/GNOME/gobject-introspection
- comex 7y ago
  From what I know about it, gobject-introspection has some nice properties, but one killer drawback: it's incompatible with cross compilation. This is apparently because it requires running binaries compiled for the target system as part of the build process. You actually can cross compile if you have an emulator for that system handy [1], but that's horrible. Apparently the reason it requires running the compiled binaries is that GObject types are only registered at runtime, within so-called "_get_type" functions. For more typical systems, everything needed can be determined at compile time. Too bad there's no portable way to ask a C compiler to dump what it knows about a source file, but if you just want things like struct sizes and field offsets, you can compile a C file that embeds them as global variables, and then extract the variable values. For more advanced introspection there are many less-portable options including Clang's API, GCC-XML, parsing debug info, or writing your own compiler (easier for C; it seems that parts of gobject-introspection work like this). Anyway, another interesting comparison is DTrace's CTF (Compact Type Format) [2], a simple binary format that describes the kernel's C struct layouts, function signatures, etc. This information is simply converted from compiler-generated debug info [3], but it's stripped down enough that it can be embedded into every kernel without too much size overhead. When the DTrace compiler is invoked to compile a user hook, it parses the CTF data and exposes the types and functions to the user's code (which is written in a custom C-like language). Ironically, BPF has BTF, which is a very similar-looking format that encodes very similar kinds of data – but is used for a completely different purpose. Specifically, it's only used to encode types and functions defined by BPF programs, to allow the kernel to pretty-print things. But in theory BTF could be repurposed to work like CTF: you would need to generate BTF information for the kernel itself, and then Clang could be extended to support "including" BTF files in place of C headers. However, this option was apparently discussed and rejected [4]. I haven't read the original threads to find out why, but I suspect it might involve: - Lack of existing tooling to do the above; - Lower expressivity compared to C headers, e.g. the inability to encode macros (although this could be fixed); - Desire to use the information for building not just BPF hooks but also full-fledged kernel modules. [1] https://maxice8.github.io/8-cross-the-gir/ https://maxice8.github.io/8-cross-the-gir/ [2] https://github.com/oracle/libdtrace-ctf/blob/master/include/sys/ctf.h https://github.com/oracle/libdtrace-ctf/blob/master/include/... [3] https://www.freebsd.org/cgi/man.cgi?query=ctfconvert&sektion=1&apropos=0&manpath=FreeBSD+10.0-RELEASE https://www.freebsd.org/cgi/man.cgi?query=ctfconvert&sektion... [4] https://lwn.net/Articles/783832/ https://lwn.net/Articles/783832/

It irks you because storing source code directly in the final binary is bonkers, but unfortuantely it is the only way to reveal all the complexity that can be exposed in a C header file. This isn't really just a C problem though. Even rust has a similar problem for propagating macros. The reason for this limitation is really that making an ABI for metaprograms (macros) is exceedingly difficult

colechristensen 7y ago
Are there other languages that provide interfaces as cleanly as exposing C header files?
- java-man 7y ago
  java interfaces. I think the concept of coding to an interface (or an API specification) goes beyond what java did. In my professional life, I see people constantly downplaying and ignoring importance of a formal and stable API specification, just to suffer consequences later.
  - colechristensen 7y ago
    If you just give me a .jar do I get those interfaces in a consumable form? Said differently, can I download your .jar and write my own code to interface with it while on a desert island without any other resources?
    - Mister_Snuggles 7y ago
      Yes. If you have a random JAR file, an IDE can introspect it to see what classes are in it and what methods are in those classes. The only exception is if you run it through some kind of obfuscator first.
      - Asooka 7y ago
        Yes but JAR files are basically equivalent to source code parsed and serialised as bytecode with only the most rudimentary optimisations applied. I don't really see a difference between that and carrying gzipped headers with the kernel. Yes, the .class format is cleaner, but it's still very close to what the programmer wrote. Another plus to carrying the header with the kernel is that you can carry the preprocessed header to make sure the user doesn't set the wrong defines, and you can add compiler checks to check that the user isn't using a too old or too new compiler with incompatible ABI.
        java-man 7y ago
        I suppose the real difference is that with java platform, you get 1. 1/10 or 1/100 compile time 2. full introspection in regards to the interfaces (the topic of this discussion). the implementation classes could be obfuscated, but the point of an interface (and an API in general) that it is not. 3. one could possibly allow multiple versions of the interface be present on the same machine at the same time. for example, when SomeInterfaceV1 is replaced (or extended) by SomeInterfaceV2, it might be possible to provide an adapter that publishes the older interface for backward compatibility. these are just random thoughts, because the difference between current systems written in c/c++ and a new architecture that uses interface-based approach is too great, I don't think I can name a real system that uses it in practice. done the API versioning myself several times with complex applications though.

mikepurvis 7y ago

It's too bad the squashfs idea was dropped. Having to download and unpack the archive somewhere seems like an extra step that should be unnecessary vs just loading the module and having the tree appear in /sys for you to point your compiler directly at. Also a bummer that it looks like you have to be running the kernel to get access to this— perhaps the archive could be marked off in the binary somehow so that there's a way to access it for non-running kernels (eg, the dkms use case of wanting to build the module for that kernel you just installed ahead of booting it for the first time). Overall binary size is a legit concern, but if this is going to be in an optionally-loaded module anyway, it seems weird to make that call just based on memory usage. I imagine most distro kernel builds will just disable this, since they already ship the headers in separate packages.

tych0 7y ago
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me. Hopefully someone picks it up :)
- hawski 7y ago
  Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for my use, but I think it would be generally useful. Sort list file format is funky. It's: "filename_string priority_integer", so it will not work for files with spaces. Also new lines, but that's far less common. It was not yet a problem for me and I can always patch it more and one can say, that people having files with spaces deserve it, though it's entirely different issue.

ori_b 7y ago

Debug symbols, which can be resolved at load time. This would also make ebpf bytecode more or less kernel version independent.

JoshTriplett 7y ago

Doesn't necessarily work for the kernel, but GIR and typelibs provide machine-readable descriptions of C APIs: https://github.com/GNOME/gobject-introspection https://github.com/GNOME/gobject-introspection

comex 7y ago
From what I know about it, gobject-introspection has some nice properties, but one killer drawback: it's incompatible with cross compilation. This is apparently because it requires running binaries compiled for the target system as part of the build process. You actually can cross compile if you have an emulator for that system handy [1], but that's horrible. Apparently the reason it requires running the compiled binaries is that GObject types are only registered at runtime, within so-called "_get_type" functions. For more typical systems, everything needed can be determined at compile time. Too bad there's no portable way to ask a C compiler to dump what it knows about a source file, but if you just want things like struct sizes and field offsets, you can compile a C file that embeds them as global variables, and then extract the variable values. For more advanced introspection there are many less-portable options including Clang's API, GCC-XML, parsing debug info, or writing your own compiler (easier for C; it seems that parts of gobject-introspection work like this). Anyway, another interesting comparison is DTrace's CTF (Compact Type Format) [2], a simple binary format that describes the kernel's C struct layouts, function signatures, etc. This information is simply converted from compiler-generated debug info [3], but it's stripped down enough that it can be embedded into every kernel without too much size overhead. When the DTrace compiler is invoked to compile a user hook, it parses the CTF data and exposes the types and functions to the user's code (which is written in a custom C-like language). Ironically, BPF has BTF, which is a very similar-looking format that encodes very similar kinds of data – but is used for a completely different purpose. Specifically, it's only used to encode types and functions defined by BPF programs, to allow the kernel to pretty-print things. But in theory BTF could be repurposed to work like CTF: you would need to generate BTF information for the kernel itself, and then Clang could be extended to support "including" BTF files in place of C headers. However, this option was apparently discussed and rejected [4]. I haven't read the original threads to find out why, but I suspect it might involve: - Lack of existing tooling to do the above; - Lower expressivity compared to C headers, e.g. the inability to encode macros (although this could be fixed); - Desire to use the information for building not just BPF hooks but also full-fledged kernel modules. [1] https://maxice8.github.io/8-cross-the-gir/ https://maxice8.github.io/8-cross-the-gir/ [2] https://github.com/oracle/libdtrace-ctf/blob/master/include/sys/ctf.h https://github.com/oracle/libdtrace-ctf/blob/master/include/... [3] https://www.freebsd.org/cgi/man.cgi?query=ctfconvert&sektion=1&apropos=0&manpath=FreeBSD+10.0-RELEASE https://www.freebsd.org/cgi/man.cgi?query=ctfconvert&sektion... [4] https://lwn.net/Articles/783832/ https://lwn.net/Articles/783832/