7 ms·

Primary author and maintainer here! Happy to answer any questions anyone has.

zbhojkiuy 8y ago
I'm considering using the library for an embedded project that communicates with a Linux host and I have some questions. - Does serialization/de-serialization require dynamic memory allocation? From what I've seen it looks like it doesn't. - How are you handling endianness? floating point numbers? - How complete/tested is the library? Are you aware of projects using it?
- arcticbull 8y ago
  I suggest looking at nanopb to compare, that’s how I’ve solved this problem in the past. It’s just a couple kB on an AVR.
  - zbhojkiuy 8y ago
    I've used nanopb in the past. It's a great library. The only drawback is that it requires a type compilation step that generates the sources.
- coreytabaka 8y ago
  Thanks for considering using libnop! This library works well in an embedded context. I've personally used it on Cortex-M class micro controller firmware. The serializer/deserializer does not require dynamic memory allocation. Whether or not dynamic allocation happens depends on how you use the library. If you avoid using data types that perform dynamic allocation (e.g. std::vector) and use (or write) Reader/Writer types that use static memory (e.g. nop::BufferReader/nop::BufferWriter) then you should be fine. There are some nice tricks you can use to permit protocols that have convenient dynamic containers on the host and static versions on the embedded device: https://github.com/google/libnop/blob/master/docs/getting-started.md#logical-buffers https://github.com/google/libnop/blob/master/docs/getting-st... https://github.com/google/libnop/blob/master/docs/getting-started.md#fungibility https://github.com/google/libnop/blob/master/docs/getting-st... https://github.com/google/libnop/blob/master/examples/shared.cpp#L59 https://github.com/google/libnop/blob/master/examples/shared... Endianness is assumed to be little because the vast majority of hardware is little endian. There are utilities to convert here: https://github.com/google/libnop/blob/master/include/nop/utility/endian.h https://github.com/google/libnop/blob/master/include/nop/uti... These are not currently used by the Reader and Writer types included with libnop for efficiency, but are available for you to use in your own Reader and Writer types if you really need it. Floating point is a much stickier problem due to lack of standardization across hardware. The library does not address this automatically and just packs floating point types in machine order. However, the library provides tools to help address the issue. One approach that works well is to use a fixed point representation for the wire type -- a value wrapper type is convenient for automating this: https://github.com/google/libnop/blob/master/docs/getting-started.md#user-defined-value-wrappers https://github.com/google/libnop/blob/master/docs/getting-st... See the Fixed template in the example. This is especially handy if you have a micro controller without floating point support or you want to minimize the size of the payload at the cost of range and/or precision. The library has a complete suite of tests and 97.9% line coverage according to GCOV, with particular attention to conditional and error paths. We use the library for a few internal embedded prototypes. I have not tracked its usage outside of Google since its recent release. Best, C
  - zbhojkiuy 8y ago
    Thanks for the detailed answer. Looks like it is a good fit for an embedded platform.
    - coreytabaka 8y ago
      Supporting embedded platforms is one of the objectives of the library. Feel free to open a ticket on github if you have any further questions.
  - kevincox 8y ago
    > Endianness is assumed to be little because the vast majority of hardware is little endian ... libnop for efficiency Do you have any evidence of this? I would be flabbergasted if a modern compiler couldn't optimize out a no-op endiannes conversion.
    - coreytabaka 8y ago
      For the most part the optimization works well. The more subtle issue is that the union trick used by the endian utilities is not compatible with constexpr expressions. There are some interesting use cases for constexpr serialization that fail if the conversions are interposed in the Reader/Writer types. The alternative is to use reinterpret_cast, which is also incompatible with constexpr expressions. Moreover, endian conversion templates are unusually frustrating in the current C++ standard. I wish there were a better way, but one does not currently exist.
Game_Ender 8y ago
Have you done any speed comparisons to other libraries? Since there are so many of these libraries that is a big deciding point, especially in C++.
sytelus 8y ago
Why C++14 dependency? A lot of other projects are still stuck on C++11 because of variety of reasons. It's sad to see this dependency :(.
- coreytabaka 8y ago
  I tried to keep it C++11, but generalized lambdas are critical in important use cases (see nop::Variant). Besides, GCC and Clang have solid C++14 support. C++17 is another story... ;-)
  - sytelus 8y ago
    The problem is not compiler support but the fact library is going to get used with other code base that is still C++11. Typically projects build system will enforce this and it’s hard to change without full evaluation of all dependencies. Is it possible to add support for C++11 and turn of these additional features that might not get used?
sytelus 8y ago
Serialization is part of the story. It would be great if someone can also write header-only dependency-free cross-platform RPC library :).
- coreytabaka 8y ago
  Since you mention it, check out the experimental RPC support: https://github.com/google/libnop/blob/master/examples/interface.cpp https://github.com/google/libnop/blob/master/examples/interf... https://github.com/google/libnop/blob/master/include/nop/rpc/interface.h https://github.com/google/libnop/blob/master/include/nop/rpc... I haven't documented it yet since it's not fully baked, but it's quite functional nonetheless. I have a working prototype of RPC over USB between a host PC and a Cortex-M micro controller. The ability to define constexpr dispatch tables is very convenient in a micro controller environment.
  - sytelus 8y ago
    What transport do you use? http? Do you use any dependency for handling transport layer? I haven’t seen a good clean dependency less headed-only approach for this yet.
    - coreytabaka 8y ago
      The library is transport agnostic. There is a Reader/Writer abstraction to adapt to any transport you like. https://github.com/google/libnop/blob/master/docs/getting-started.md#readerwriter-interfaces https://github.com/google/libnop/blob/master/docs/getting-st...

I'm considering using the library for an embedded project that communicates with a Linux host and I have some questions. - Does serialization/de-serialization require dynamic memory allocation? From what I've seen it looks like it doesn't. - How are you handling endianness? floating point numbers? - How complete/tested is the library? Are you aware of projects using it?

arcticbull 8y ago
I suggest looking at nanopb to compare, that’s how I’ve solved this problem in the past. It’s just a couple kB on an AVR.
- zbhojkiuy 8y ago
  I've used nanopb in the past. It's a great library. The only drawback is that it requires a type compilation step that generates the sources.
coreytabaka 8y ago
Thanks for considering using libnop! This library works well in an embedded context. I've personally used it on Cortex-M class micro controller firmware. The serializer/deserializer does not require dynamic memory allocation. Whether or not dynamic allocation happens depends on how you use the library. If you avoid using data types that perform dynamic allocation (e.g. std::vector) and use (or write) Reader/Writer types that use static memory (e.g. nop::BufferReader/nop::BufferWriter) then you should be fine. There are some nice tricks you can use to permit protocols that have convenient dynamic containers on the host and static versions on the embedded device: https://github.com/google/libnop/blob/master/docs/getting-started.md#logical-buffers https://github.com/google/libnop/blob/master/docs/getting-st... https://github.com/google/libnop/blob/master/docs/getting-started.md#fungibility https://github.com/google/libnop/blob/master/docs/getting-st... https://github.com/google/libnop/blob/master/examples/shared.cpp#L59 https://github.com/google/libnop/blob/master/examples/shared... Endianness is assumed to be little because the vast majority of hardware is little endian. There are utilities to convert here: https://github.com/google/libnop/blob/master/include/nop/utility/endian.h https://github.com/google/libnop/blob/master/include/nop/uti... These are not currently used by the Reader and Writer types included with libnop for efficiency, but are available for you to use in your own Reader and Writer types if you really need it. Floating point is a much stickier problem due to lack of standardization across hardware. The library does not address this automatically and just packs floating point types in machine order. However, the library provides tools to help address the issue. One approach that works well is to use a fixed point representation for the wire type -- a value wrapper type is convenient for automating this: https://github.com/google/libnop/blob/master/docs/getting-started.md#user-defined-value-wrappers https://github.com/google/libnop/blob/master/docs/getting-st... See the Fixed template in the example. This is especially handy if you have a micro controller without floating point support or you want to minimize the size of the payload at the cost of range and/or precision. The library has a complete suite of tests and 97.9% line coverage according to GCOV, with particular attention to conditional and error paths. We use the library for a few internal embedded prototypes. I have not tracked its usage outside of Google since its recent release. Best, C
- zbhojkiuy 8y ago
  Thanks for the detailed answer. Looks like it is a good fit for an embedded platform.
  - coreytabaka 8y ago
    Supporting embedded platforms is one of the objectives of the library. Feel free to open a ticket on github if you have any further questions.
- kevincox 8y ago
  > Endianness is assumed to be little because the vast majority of hardware is little endian ... libnop for efficiency Do you have any evidence of this? I would be flabbergasted if a modern compiler couldn't optimize out a no-op endiannes conversion.
  - coreytabaka 8y ago
    For the most part the optimization works well. The more subtle issue is that the union trick used by the endian utilities is not compatible with constexpr expressions. There are some interesting use cases for constexpr serialization that fail if the conversions are interposed in the Reader/Writer types. The alternative is to use reinterpret_cast, which is also incompatible with constexpr expressions. Moreover, endian conversion templates are unusually frustrating in the current C++ standard. I wish there were a better way, but one does not currently exist.

Game_Ender 8y ago

Have you done any speed comparisons to other libraries? Since there are so many of these libraries that is a big deciding point, especially in C++.

sytelus 8y ago

Why C++14 dependency? A lot of other projects are still stuck on C++11 because of variety of reasons. It's sad to see this dependency :(.

coreytabaka 8y ago
I tried to keep it C++11, but generalized lambdas are critical in important use cases (see nop::Variant). Besides, GCC and Clang have solid C++14 support. C++17 is another story... ;-)
- sytelus 8y ago
  The problem is not compiler support but the fact library is going to get used with other code base that is still C++11. Typically projects build system will enforce this and it’s hard to change without full evaluation of all dependencies. Is it possible to add support for C++11 and turn of these additional features that might not get used?

sytelus 8y ago

Serialization is part of the story. It would be great if someone can also write header-only dependency-free cross-platform RPC library :).

coreytabaka 8y ago
Since you mention it, check out the experimental RPC support: https://github.com/google/libnop/blob/master/examples/interface.cpp https://github.com/google/libnop/blob/master/examples/interf... https://github.com/google/libnop/blob/master/include/nop/rpc/interface.h https://github.com/google/libnop/blob/master/include/nop/rpc... I haven't documented it yet since it's not fully baked, but it's quite functional nonetheless. I have a working prototype of RPC over USB between a host PC and a Cortex-M micro controller. The ability to define constexpr dispatch tables is very convenient in a micro controller environment.
- sytelus 8y ago
  What transport do you use? http? Do you use any dependency for handling transport layer? I haven’t seen a good clean dependency less headed-only approach for this yet.
  - coreytabaka 8y ago
    The library is transport agnostic. There is a Reader/Writer abstraction to adapt to any transport you like. https://github.com/google/libnop/blob/master/docs/getting-started.md#readerwriter-interfaces https://github.com/google/libnop/blob/master/docs/getting-st...