6 ms·

Show HN: Ldump – serialize any Lua data

96 points by girvel 2y ago

Some time ago, I was implementing saves for my LOVE2D game. I wanted to do a full dump of the game state -- which included closures (AI), complex graphs, sets with tables as keys and also fundamentally non-serializable data (coroutines and userdata), that require user-defined serialization/deserialization logic. I went through every Lua serialization library -- none covered all data types/cases. So I wrote my own. It is a polished version, thoroughly annotated, tested and documented. It is made to be as functional and customizable as possible (or at least I did everything I could think of). I would be happy to hear suggestions/corrections for both code and documentation -- even nitpicky ones.

bflesch 2y ago
Nice - I wonder why something like this is not built-in to the language itself. Especially debugging tables is painful to say the least :)
- sebstefan 2y ago
  For most things in Lua you have to bring your own wheel
- nicoloren 2y ago
  You can use inspect.lua to debug tables [1] [1] https://github.com/kikito/inspect.lua https://github.com/kikito/inspect.lua
ithkuil 2y ago
"lump" would have been a nice name
- girvel 2y ago
  Damn, it really would
  - darig 2y ago
    [dead]
sebstefan 2y ago
So it also dumps functions and is able to import them back? Does the function still need to be in memory to be loaded again ("does it just dump the pointer") or can I save it to disk, shut off the interpreter, boot it again and it imports it fine (in which case it somehow dumps them as code...?)? Even in the linked test case on the readme you don't show the output/expectation of the serialization
- lifthrasiir 2y ago
  Functions are apparently serialized as a bytecode dump contained in a self-extracting expression. So everything can indeed be serialized as a Lua expression. Seems that the author also tried to preserve as many upvalues as possible, though I feel that is way more dangerous than I would like.
  - girvel 2y ago
    Yep, that is correct. I think ldump is able to preserve all upvalues, even on edge cases such as "_ENV" and joined upvalues (multiple functions referencing one upvalue). A closure is basically an object with a single method and upvalues as fields -- serialization is straightforward. I think I got it covered, but I would be glad to hear ideas about where the serialization can be unstable.
- girvel 2y ago
  The function (even a closure) would be fully recreated on deserialization, it is fully safe to save it to disk. It wouldn't preserve reference equality -- it would be a new function -- but the behaviour and the state (if using closures) would be equivalent. I didn't include asserts in the linked case, because I thought it would be too verbose. You can see asserts in the test, that is linked below the example. Maybe it was the wrong call, I will think about including asserts into the example itself.
  - sebstefan 2y ago
    That's super cool I think you could make it clearer, try reading the readme as someone with the preconceived notion that this is Yet Another Lua Serializer that translates functions, userdata and threads to their tostring() output. There are hundreds of those projects
lifthrasiir 2y ago
Maybe I'm too pedantic but allowing anything to be "deserialized", which equals to "evaluated" here, is not secure. I think it only has to accept a very limited subset of Lua anyway, so you may switch to a non-Lua format which is made easy to parse. That way the library has a total control over what is being evaluated.
- girvel 2y ago
  This is an interesting thought. Currently, it is unsafe and intended to load only the files you trust. I should definitely include a warning into README. Overall, it would be nice to make it safer. I don't think switching to non-Lua format would make it safer, because it is intended to serialize functions too, which can have arbitrary code even if everything else would be stored as data. Maybe it is possible to make a function like `ldump.safe_load` restricting `load`'s environment, so it wouldn't have access to debug/os/io modules.
  - lifthrasiir 2y ago
    Yeah, you would need an allowlist for functions. Using bytecode would make it much harder, I haven't given deep thought yet.
  - gvx 2y ago
    You could take a look at SELÖVE, a (severely out of date) fork of LÖVE that is intended to make it safe to run arbitrary .love games. (It used to be on bitbucket, but it looks like it's gone? I'm not sure if I have the repo locally :/) Running arbitrary code was such a problem that I just completely ruled it out for bitser. Instead of serializing functions, you can register safe functions as resources. This doesn't solve the upvalue problem, though.
  - girvel 2y ago
    I looked into it, and Lua allows limiting the environment when `load`ing -- through `env` argument since 5.2 or through setfenv before. I will add a helper function to produce a minimal needed environment for safe loading and a documentation page about safety.
brunocroh 2y ago
I will try it on my next love2d project, thank you!
JourneyJourney 2y ago
I'm afraid I spent too much time with LUA lately and fell in love with its simplicity. Kinda hard to go back to OOP after that.
- nicoloren 2y ago
  Same for me, I used Lua for a desktop software for a client and I enjoyed it a lot! I'm thinking of starting to dev a game with LOVE2D just to have an excuse to use Lua.
  - girvel 2y ago
    LOVE2D is a great gamedev framework, I can not recommend it enough. It is so pleasant to work with.
- aldanor 2y ago
  It's simple until you dig deep into meta tables lol
- je42 2y ago
  I find that most LUA code sits between functional, imperative and OOP.
gvx 2y ago
Cool to see you were inspired by Ser!
- girvel 2y ago
  Oh wow, didn't expect to meet the author, thank you!
synergy20 2y ago
newbie question,when is this useful in practice
- girvel 2y ago
  It is intended to be used in cases where you need to store data on a disk or transfer it to another machine -- like in a video game save or a network data exchange
jhatemyjob 2y ago
Try running this in a repl and tell me what you get, OP: string.format('%q', 'hi\n')
- myrmidon 2y ago
  If you insinuate that %q obviates the need for ldump then you are wrong. There is not even significant overlap in what they do; all that %q does is sufficiently escape Lua strings so the interpreter can read them back. It does not serialize functions nor even tables in any shape or form. edit: Sorry for being unreasonably harsh after misunderstanding your message.
  - girvel 2y ago
    I actually thought the comment was about ldump implementation: it uses %q to serialize strings, and it may not be a reliable way.
- girvel 2y ago
  On my machine it produces an equivalent string, although differently formatted. It seems that ldump preserves all special characters (`"\a\b\f\n\r\t\v\\\"\'"`), although I will need to test in on all supported versions.
  - jhatemyjob 2y ago
    Ah, you know what, you're right. It's an equivalent string for me too: "hi\ " I didn't know Lua treated \ before newlines like that. That's cool! I made a similar Lua serialization library for myself and was using a chain of `string.match` calls to escape my strings. Now I can make it way simpler. Lol. Thanks
098527222722 2y ago
use computer divices
koeng 2y ago
I've been looking for something similar! Here is what I'd like to do: I have a long-running script. At several steps, the execution of the script has to pause for a long time for operations to be done in-real-life (biological experiments, so think wait time being like 2 days between running), before getting some data and continuing to run. From what I can see in this, I'd add yielding coroutines at data pause points, right? How would you handle that?
appleorchard46 2y ago
Very cool! I was just needing something like this for my Defold game, this looks way better than my hacky solution. Semi-unrelated - you say you're using tables as keys in your project. I didn't know you could do that! What are you using it for?
- GranPC 2y ago
  FWIW anything in Lua can be used as a key - including functions, userdata, etc.
- girvel 2y ago
  For example, you want to keep a set of objects -- then the objects themselves would be keys, and values would be true/nil. Or there is a good example in one of ldump's recent issues: https://github.com/girvel/ldump/issues/44 https://github.com/girvel/ldump/issues/44, where the loaded packages are stored in a table as keys to easily detect external module usage.

96 points by girvel 2y ago

bflesch 2y ago

Nice - I wonder why something like this is not built-in to the language itself. Especially debugging tables is painful to say the least :)

sebstefan 2y ago
For most things in Lua you have to bring your own wheel
nicoloren 2y ago
You can use inspect.lua to debug tables [1] [1] https://github.com/kikito/inspect.lua https://github.com/kikito/inspect.lua

ithkuil 2y ago

"lump" would have been a nice name

girvel 2y ago
Damn, it really would
- darig 2y ago
  [dead]

sebstefan 2y ago

So it also dumps functions and is able to import them back? Does the function still need to be in memory to be loaded again ("does it just dump the pointer") or can I save it to disk, shut off the interpreter, boot it again and it imports it fine (in which case it somehow dumps them as code...?)? Even in the linked test case on the readme you don't show the output/expectation of the serialization

lifthrasiir 2y ago
Functions are apparently serialized as a bytecode dump contained in a self-extracting expression. So everything can indeed be serialized as a Lua expression. Seems that the author also tried to preserve as many upvalues as possible, though I feel that is way more dangerous than I would like.
- girvel 2y ago
  Yep, that is correct. I think ldump is able to preserve all upvalues, even on edge cases such as "_ENV" and joined upvalues (multiple functions referencing one upvalue). A closure is basically an object with a single method and upvalues as fields -- serialization is straightforward. I think I got it covered, but I would be glad to hear ideas about where the serialization can be unstable.
girvel 2y ago
The function (even a closure) would be fully recreated on deserialization, it is fully safe to save it to disk. It wouldn't preserve reference equality -- it would be a new function -- but the behaviour and the state (if using closures) would be equivalent. I didn't include asserts in the linked case, because I thought it would be too verbose. You can see asserts in the test, that is linked below the example. Maybe it was the wrong call, I will think about including asserts into the example itself.
- sebstefan 2y ago
  That's super cool I think you could make it clearer, try reading the readme as someone with the preconceived notion that this is Yet Another Lua Serializer that translates functions, userdata and threads to their tostring() output. There are hundreds of those projects

lifthrasiir 2y ago

Maybe I'm too pedantic but allowing anything to be "deserialized", which equals to "evaluated" here, is not secure. I think it only has to accept a very limited subset of Lua anyway, so you may switch to a non-Lua format which is made easy to parse. That way the library has a total control over what is being evaluated.

girvel 2y ago
This is an interesting thought. Currently, it is unsafe and intended to load only the files you trust. I should definitely include a warning into README. Overall, it would be nice to make it safer. I don't think switching to non-Lua format would make it safer, because it is intended to serialize functions too, which can have arbitrary code even if everything else would be stored as data. Maybe it is possible to make a function like `ldump.safe_load` restricting `load`'s environment, so it wouldn't have access to debug/os/io modules.
- lifthrasiir 2y ago
  Yeah, you would need an allowlist for functions. Using bytecode would make it much harder, I haven't given deep thought yet.
- gvx 2y ago
  You could take a look at SELÖVE, a (severely out of date) fork of LÖVE that is intended to make it safe to run arbitrary .love games. (It used to be on bitbucket, but it looks like it's gone? I'm not sure if I have the repo locally :/) Running arbitrary code was such a problem that I just completely ruled it out for bitser. Instead of serializing functions, you can register safe functions as resources. This doesn't solve the upvalue problem, though.
- girvel 2y ago
  I looked into it, and Lua allows limiting the environment when `load`ing -- through `env` argument since 5.2 or through setfenv before. I will add a helper function to produce a minimal needed environment for safe loading and a documentation page about safety.

brunocroh 2y ago

I will try it on my next love2d project, thank you!

JourneyJourney 2y ago

I'm afraid I spent too much time with LUA lately and fell in love with its simplicity. Kinda hard to go back to OOP after that.

nicoloren 2y ago
Same for me, I used Lua for a desktop software for a client and I enjoyed it a lot! I'm thinking of starting to dev a game with LOVE2D just to have an excuse to use Lua.
- girvel 2y ago
  LOVE2D is a great gamedev framework, I can not recommend it enough. It is so pleasant to work with.
aldanor 2y ago
It's simple until you dig deep into meta tables lol
je42 2y ago
I find that most LUA code sits between functional, imperative and OOP.

gvx 2y ago

Cool to see you were inspired by Ser!

girvel 2y ago
Oh wow, didn't expect to meet the author, thank you!

synergy20 2y ago

newbie question,when is this useful in practice

girvel 2y ago
It is intended to be used in cases where you need to store data on a disk or transfer it to another machine -- like in a video game save or a network data exchange

jhatemyjob 2y ago

Try running this in a repl and tell me what you get, OP: string.format('%q', 'hi\n')

myrmidon 2y ago
If you insinuate that %q obviates the need for ldump then you are wrong. There is not even significant overlap in what they do; all that %q does is sufficiently escape Lua strings so the interpreter can read them back. It does not serialize functions nor even tables in any shape or form. edit: Sorry for being unreasonably harsh after misunderstanding your message.
- girvel 2y ago
  I actually thought the comment was about ldump implementation: it uses %q to serialize strings, and it may not be a reliable way.
girvel 2y ago
On my machine it produces an equivalent string, although differently formatted. It seems that ldump preserves all special characters (`"\a\b\f\n\r\t\v\\\"\'"`), although I will need to test in on all supported versions.
- jhatemyjob 2y ago
  Ah, you know what, you're right. It's an equivalent string for me too: "hi\ " I didn't know Lua treated \ before newlines like that. That's cool! I made a similar Lua serialization library for myself and was using a chain of `string.match` calls to escape my strings. Now I can make it way simpler. Lol. Thanks

098527222722 2y ago

use computer divices

koeng 2y ago

I've been looking for something similar! Here is what I'd like to do: I have a long-running script. At several steps, the execution of the script has to pause for a long time for operations to be done in-real-life (biological experiments, so think wait time being like 2 days between running), before getting some data and continuing to run. From what I can see in this, I'd add yielding coroutines at data pause points, right? How would you handle that?

appleorchard46 2y ago

Very cool! I was just needing something like this for my Defold game, this looks way better than my hacky solution. Semi-unrelated - you say you're using tables as keys in your project. I didn't know you could do that! What are you using it for?

GranPC 2y ago
FWIW anything in Lua can be used as a key - including functions, userdata, etc.
girvel 2y ago
For example, you want to keep a set of objects -- then the objects themselves would be keys, and values would be true/nil. Or there is a good example in one of ldump's recent issues: https://github.com/girvel/ldump/issues/44 https://github.com/girvel/ldump/issues/44, where the loaded packages are stored in a table as keys to easily detect external module usage.