9 ms·
Compile-time JSON deserialization in C++
- fsloth 2y agoWhat a beautiful example of abuse of C++ templates. I love it. But please don’t do this in production. What ever you need to do, use C++ templates as the last resort because you’ve figured out all other approaches suck even more. Maintaining template heavy code is absolutely horrible and wasteful (and if it’s C++ production code we measure it’s lifetime in decades). And no, there is no way ”to do it correctly so it doesn’t suck”. Templates belong to the lowest abstraction levels - as stl mostly does. Anyhting more prevalent is an abomination. If the schema is fixed, have types with the data and if you have a default data, provide it using initializer lists. Ie. have a struct or structs with explicit serializeToJson and deserializeFromJson functions. It’s faster to write than figuring out the correct template gymnastics and about 100x easier to maintain and extend.
- qsdf38100 2y agoThis reminds me of some coworkers I had that moaned anytime they saw 'template' in some code. They were convinced that templates were just bad, and that they should stay in the STL. Some of these people would then proceed to use void* and enums to perform the same computations (the C++ haters kind), or use virtual functions all over the place (the java-background kind). Not only was the result much more fragile (compile errors are now runtime errors), but it would also prevent inlining, not to mention the dynamic memory allocation fest.
- fsloth 2y agoNo, don’t abuse the language. Wrap everything in types and all is fine. Verbose C++ is the only good kind of C++. Why? C++ code needs to be debuggable and modifiable so when a profiler shows hotspots, you know where they are coming from and react appropriately. How do you fix a hotspot on a one line of code somewhere in the middle of a template thingsmajic? You don’t. You need to unroll the template code to untemplated code and the fix the hotspot. Cases where you don’t need to fix one line hotspots because the resources consumed by the code are irrelevant are fine. But if performance does not matter it likely means you should use a bette language than C++. Using C++ and not caring about performance finetuning is the worst of both worlds - you are using a cumbersome language AND it’s not even for any practical benefit.
- dctwin 2y agoHello! I wrote this short blog post about using pattern-matching-like template metaprogramming to deserialize JSON at build time - please let me know what you think (especially if you see improvements)
- whizzter 2y agoIt's cute and neat to be able to do it 100% constexpr, however as you mention the indexers feels a tad inelegant. I've written 2 iterations of a reflection library where you needed to annotate structs slightly with an ugly macro but once done you could just do: Message msg; if (parse_json(str,msg)) { ..process msg struct.. } The previous iterations were for C++11 and C++17 but it seems that with C++20 features you don't even seem to need the macro uglyness so I personally think libraries need to move in the direction of plain old structs.
- cobbal 2y agosmall note: "JSON in its pure form anarchic" is missing a verb
- dctwin 2y agoThank you!
- worstspotgain 2y agoAs someone who used to have to do this sort of compile-time stuff with previous versions of the standard, I'm jealous of how much more can be done now that I don't have to. If you're looking for an interesting follow-up project, here's something I had to do once that's now become much easier: compute a compile-time hash of the compilation for the current translation unit, e.g. __BASE_FILE__ hashed together with __TIMESTAMP__ or the equivalents for each platform. This allows you to dynamically invalidate on-disk caches and trigger new-build tripwires based on ongoing revisions. Development and release builds are handled identically: if source file X handles a cache and X was recompiled, discard the cache.
- dctwin 2y ago
- nikeee 2y agoCould this be leveraged to emit a parser that is specialized for the provided type that can be used at runtime? Afaik .NET does something like that using code generators. The advantage being that the parser is tailored to the specific type that is deserialized and it writes directly to the struct's fields instead of going through some dictionary.
- leni536 2y agoThis lib does something like that: https://github.com/beached/daw_json_link https://github.com/beached/daw_json_link
- nikeee 2y agoIt seems that you have to maintain hand-coded mappings for each type. Maybe this could be solved by using C++23's compile-time reflections.
- dctwin 2y agoYes, the nonconstexpr version does just that, unless I misunderstood your question. See also boost::spirit for a 'big' version of this
- nikki93 2y agoI use this static reflection hack in C++ -- https://godbolt.org/z/enh8za4ja https://godbolt.org/z/enh8za4ja You do have to tag struct fields with a macro, but you can attach contexpr-visitable attributes. There's also a static limit to how many reflectable fields you can have, all reflectable fields need to be at the front of the struct, and the struct needs to be an aggregate.
- gpderetta 2y agothat forEachProp function... it brings back nightmares of when, before variadics, we used to macro generate up-to N-arity functions (with all the const/non-const permutations(. Now I use the same trick in our code base to generically hash aggregates, but I limit it to 4 fields for sanity.
- asguy 2y agoHoly crap; that's pretty epic. Did you come up with that yourself?
- forrestthewoods 2y agoI think the value of compile-time JSON deserialization is... well I was going to say zero but really it's negative. It's a cute trick, but please don't ever do this in a real project.
- dctwin 2y agoDespite my writing the article I agree
- delfinom 2y agoYou clearly never wrote a Fizz Buzz enterprise grade application ;)
- beached_whale 2y agoSo I have owned a library for 6years or so that does constexpr JSON to data structures, JSON Link. There are a few benefits and in the near future with #embed it gets even better. The big benefit is that we can now get earlier errors and do testing in constexpr that gives more guarantees around the areas of core UB and in most implementations they add constexpr checked preconditions on the std library too. But, just because it is marked constexpr, doesn't mean it will be run at compile time. This also, limits the shenanigans that the library dev can do to get potential perf and work around design limitations. In JSON Link's case, since it was using C++17 at the time, it forced me to think around the problem of allocation and who does it. The library does not allocate, but potentially the data structures being deserialized to will. In C++20 you can get limited constexpr allocations but they are good for things like stacks and eliminating the fixed buffers many devs have used in the past; which is a good thing on it's own but isn't really allowing one to parse to a vector at compile time(as in OP's example) for things that persist. Where this will get really interesting, though, is when #embed is in the major compilers. It's mostly there in clang, with gcc on the way I believe. It will open the door for DSL's and compile time configs in human readable formats or interop with other tools(maybe GUI designers) As for OP's library, I am not a fan of the json_value like library approach that treats JSON as a thing to care about when it is usually just an imp detail to move to ones business objects. TL;DR The big benefit though, is the ability to reason about the quality of the code in the library and have stronger testing.
- stephc_int13 2y agoI am afraid of the compile-time cost. For this kind of things I tend to prefer using a simpler program (written in anything you like) to generate C or C++ instead of having the compile do the same thing much slowly. Meta programming can be good, but it is even better done with an actual meta program, IMO.
- ulrikrasmussen 2y agoI like how code generation is typically done in Kotlin using KSP. Here you write your code generator as a plugin to the compiler, so you have the full expressivity of any JVM language you like. It also operates on the parsed and resolved AST, so you can analyze even derived types. It also allows code generators to run on code which has type errors or even fails to resolve some symbols which is very useful when you generate code from class annotations and then proceed to use the generated code later in the same file. Another advantage of using KSP is that it also handles caching for you and will avoid running code generators again if the output already exists.
- ranger_danger 2y ago> I am afraid of the compile-time cost. Still better than Rust /s
- adolph 2y agoBrings to mind the old story about a JSON DSL https://thedailywtf.com/articles/the-inner-json-effect https://thedailywtf.com/articles/the-inner-json-effect
- threatripper 2y agoIs this real? It can't be real. Nobody can be this stupid. But then again it takes a special kind of person who doesn't understand satire to actually do something like that. Somebody, where they would say "we trained him wrong on purpose as a kind of a joke".
- bruce511 2y ago
- anothername12 2y agoNot a C++ user, but is this the same as #. reader macro in Common Lisp?
- kazinator 2y agoAlso, this: (defmacro macro-time (&rest forms) `(quote ,(eval `(progn ,@forms)))) forms are evaluated at macro-expansion-time, and their result is quoted, and substituted for the (macro-time ...) invocation. For instance, if we have a snarf-file function which reads a text file and returns the contents as a string, we can do: (macro-time (snarf-file "foo.txt")) and we now have the contents of foo.txt as a string literal.
- heisig 2y agoYes, the #. reader macro is one of the ways how you can achieve this in Common Lisp. Using the reader macro is also way more efficient because you don't awkwardly use your compiler as an interpreter for a weird subset of your actual language - you simply call to compiled code. Seeing Greenspun's tenth rule [1] in action again and again is one of the weird things we Common Lisp programmers have to endure. I wish we would have more discussions on how to improve Lisp even further instead of trying to 'fix' C or C++ for the umpteenth time. [1] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
- ykonstant 2y ago>I wish we would have more discussions on how to improve Lisp even further instead of trying to 'fix' C or C++ for the umpteenth time. I agree one million percent; projects like SBCL are great, but my impression is that there are tons of improvements to be had in producing optimized code for modern processors (cache friendliness, SIMD, etc), GPU programming etc. I asked about efforts in those directions here and there, but did not get very clear answers.
- throw2353265 2y agoI don’t know much about Common Lisp, but one of the times I evaluated it I wondered why it fairs so poorly in benchmarks[1], and as a complete noob I went and checked what sort of code it will produce for something completely trivial, like adding 2 fixnums. And oh my god: * (defun fx-add (x y) (declare (optimize (speed 3) (safety 0) (debug 0)) (type fixnum x y)) (+ x y)) FX-ADD * (disassemble #'fx-add) ; disassembly for FX-ADD ; Size: 104 bytes. Origin: #x7005970068 ; FX-ADD ; 68: 40FD4193 ASR NL0, R0, #1 ; 6C: 00048B8B ADD NL0, NL0, R1, ASR #1 ; 70: 0A0000AB ADDS R0, NL0, NL0 ; 74: E7010054 BVC L1 ; 78: BD2A00B9 STR WNULL, [THREAD, #40] ; pseudo-atomic-bits ; 7C: A97A47A9 LDP TMP, LR, [THREAD, #112] ; mixed-tlab.{free-pointer, end-addr} ; 80: 2A410091 ADD R0, TMP, #16 ; 84: 5F011EEB CMP R0, LR ; 88: C8010054 BHI L2 ; 8C: AA3A00F9 STR R0, [THREAD, #112] ; mixed-tlab ; 90: L0: 2A3D0091 ADD R0, TMP, #15 ; 94: 3E2280D2 MOVZ LR, #273 ; 98: 3E0100A9 STP LR, NL0, [TMP] ; 9C: BF3A03D5 DMB ISHST ; A0: BF2A00B9 STR WZR, [THREAD, #40] ; pseudo-atomic-bits ; A4: BE2E40B9 LDR WLR, [THREAD, #44] ; pseudo-atomic-bits ; A8: 5E0000B4 CBZ LR, L1 ; AC: 200120D4 BRK #9 ; Pending interrupt trap ; B0: L1: FB031AAA MOV CSP, CFP ; B4: 5A7B40A9 LDP CFP, LR, [CFP] ; B8: BF0300F1 CMP NULL, #0 ; BC: C0035FD6 RET ; C0: L2: 090280D2 MOVZ TMP, #16 ; C4: 2AFCFF58 LDR R0, #x7005970048 ; SB-VM::ALLOC-TRAMP ; C8: 40013FD6 BLR R0 ; CC: F1FFFF17 B L0 NIL Are you serious? This should be 1, max 2 instructions, with no branches and no memory use. Furthermore, I’ve also decided to evaluate the debuggers available for Common Lisp. However, despite it being touted as a debugger-oriented language, I think the actual debuggers are pretty subpar, compared to debuggers available for C, C++, Java or .NET. No Common Lisp debugger supports watchpoints of any kind. If a given debugger supports breakpoints at all, they’re often done through wrapping code in code that triggers a breakpoint, or making this code run under interpreter instead of being native. Setting breakpoints in arbitrary code won’t work, it needs to be available as source code first. SBCL with SLIME doesn’t have a nice GUI where I could use the standard F[N] keys to step, continue, stop, etc. I don’t see any pane with live disassembly view. No live watch. LispWorks GUI on the other hand looks like a space station, where I struggle to orient myself. The only feature that is somewhat well-done is live code reload, but IMO it’s something far less important than well-implemented breakpoints and watchpoints in other languages, since the main thing I need the debugger for is to figure out what the hell a given piece of code is doing. Editing it is a completely secondary concern. And live code reload is also not unique to Common Lisp. Debugger-wise, Java and .NET seem to be leading in quality, followed by C and C++. [1]: Yes, I have read many comments about the alleged good performance of Common Lisp, but either authors of these comments live in a parallel reality with completely different benchmark results, or they’re comparing to Python. As such I treat those comments as urban legends.
- abbeyj 2y agoCould you use something like `template <StringLiteral str> constexpr inline Key<str> key;`? Then you could write `key<"myKey">` instead of `Key<"myKey">{}`, saving you from needing the `{}` each time.
- dctwin 2y agoOh I think I finally groked what you suggested! Something like template <StringLiteral str> static constexpr Key<str> key in the class namespace - I think this this would work, but if I understand this correctly, You would need to do like User user {...}; user[User::key<"myKey">] Which actually is not so bad...
- dctwin 2y agoHm - so this would instantiate a variable for each key in the class namespace? I admit I haven't seen anything like this but sounds very interesting
- actionfromafar 2y agoWhere are the functional language programmers so I can hold their beer?