7 ms·

What would you consider good modern design?

by muhehe 1mo ago

supriyo-biswas 1mo ago
SQLar[1], see [2] for some reasoning around why a database is preferred over zipped XML. Though, I'd be fine with a DBM-style database too as we only need the key-value part of it. [1] https://sqlite.org/sqlar/doc/trunk/README.md https://sqlite.org/sqlar/doc/trunk/README.md [2] https://www.sqlite.org/affcase1.html https://www.sqlite.org/affcase1.html
- chungy 1mo ago
  the SQLite archive format (it's probably worth linking to the main documentation[1], instead of the very old experimental repository) may not really be a good fit for something like GIMP's native file format. (To be clear, "SQLite archives" are not special compared to any other database: it's just a well-defined schema for an sqlar table, which the sqlite3 command line tool is able to create, update, and extract using syntax like the tar command.) That being said, SQLite would still be a good choice, especially as it's a format that's really intended to be modified in-place, and has good data integrity features (eg: keep WAL enabled so that mid-save crashes/shutdowns don't corrupt your file), neither of which are provided by Zip. You could even just run zlib on data (be it XML or what have you) if optimizing the on-disk size of the file is desirable. [1] https://sqlite.org/cli.html#sqlite_archive_support https://sqlite.org/cli.html#sqlite_archive_support
  - happymellon 1mo ago
    Isn't the implementation the spec for SQLite? Not really a great option for an image format, where we therefore can't have multiple implementations. Unless I misremembered.
    - SQLite 1mo ago
      SQLite file format spec: <https://sqlite.org/fileformat.html https://sqlite.org/fileformat.html>
      - happymellon 1mo ago
        Thank you for the prompt clarification!
- flohofwoe 1mo ago
  The problem with using database blobs for load/save is that you usually need a full database client in the application. SQlite advertises that use case, but it is complete overkill. You never need to run any sort of complex SQL query on an image file format for instance. Using XML+ZIP in this day and age is also a strange decision, but at least that way the data is inspectable with unzip, a text editor and an image viewer (assuming they use a standard image format to store the raw pixel data).
  - TeMPOraL 1mo ago
    > You never need to run any sort of complex SQL query on an image file format for instance. Sure you will. Plenty of features that don't exist, or are implemented badly, because you can't easily do it. Quick mental translation table: if you think "iterate over every ..." or a `for` loop, that's your SELECT query. If you think about `if` conditions, that's the parts that go after FROM clause.
    - x3ro 1mo ago
      In order to have any advantage from this, you would have the added complexity of splitting your file format into tables that can be queried in a useful manner. However, for an image file format, you most likely need to hold the entire definition in memory at all times anyway. Assuming that’s the case, doesn’t XPath get you there most of the way (assuming XML), with _way_ less complexity?
      - TeMPOraL 1mo ago
        Image data is just binary blobs. You aren't splitting that into channel columns or anything. But an image file for an editor like Gimp isn't one image blob. It's dozens or hundreds of them - one or more per layer - along with tons of associated metadata at every level. All that tends to fit sensible schemas and managing it is what SQLite shines at.
        x3ro 1mo ago
        I don’t see how this addresses my point that zipped XML gives you the same thing, but simpler. I understand that a GIMP file is many images, so that makes a zip feel like a great fit to me. The only advantage I see for using a full-blown DB is ensuring consistency with references, which admittedly is a plus. But beyond that, what do you gain?
speedgoose 1mo ago
Compressed JSON with the binary content encoded in base64 strings, obviously.
- einpoklum 1mo ago
  Why would zipped JSON be fundamentally superior to zipped XML?
  - speedgoose 1mo ago
    To answer seriously, my parent comment is a joke, JSON is a simpler format that maps better to most programming languages internal memory representations. Developers tend to prefer JSON’s simplicity over XML.
    - berkes 1mo ago
      > most programming languages internal memory representations Often heard wrt JSON but incorrect. It maps to the primitive types in JavaScript. But almost all programming languages treat floats and integers different, make distinction between char and strings and many have some form of date/time. JSON has neither. In that direction, XML is much closer since every node is a triple (name, value, attributes) so can have type info, json is a tuple. And Protobuf, while not popular, gets this completely right.
      - speedgoose 1mo ago
        I think it's too risky to treat numbers in JSON as something else than IEEE754 64bits floats. But yes, JSON is small and doesn't do datetimes, char, comments, and a million other things XML does. But you don't need to think much about memory representation when you parse a JSON, and the developer experience is a lot more pleasing than browsing a XML tree. That what used to matter.
    - einpoklum 1mo ago
      If this is intended for reading and writing by humans, then JSON doesn't cut it since you don't get comments (and perhaps also because of the weirdness of 64-bit floating-point values). Plus, XML is more "structurally nuanced". I suppose JSON is simpler to parse, but it's a trade-off of features, it's not like one is bad and the other is good.

What would you consider good modern design?

by muhehe 1mo ago

What would you consider good modern design?

SQLar[1], see [2] for some reasoning around why a database is preferred over zipped XML. Though, I'd be fine with a DBM-style database too as we only need the key-value part of it. [1] https://sqlite.org/sqlar/doc/trunk/README.md https://sqlite.org/sqlar/doc/trunk/README.md [2] https://www.sqlite.org/affcase1.html https://www.sqlite.org/affcase1.html

chungy 1mo ago
the SQLite archive format (it's probably worth linking to the main documentation[1], instead of the very old experimental repository) may not really be a good fit for something like GIMP's native file format. (To be clear, "SQLite archives" are not special compared to any other database: it's just a well-defined schema for an sqlar table, which the sqlite3 command line tool is able to create, update, and extract using syntax like the tar command.) That being said, SQLite would still be a good choice, especially as it's a format that's really intended to be modified in-place, and has good data integrity features (eg: keep WAL enabled so that mid-save crashes/shutdowns don't corrupt your file), neither of which are provided by Zip. You could even just run zlib on data (be it XML or what have you) if optimizing the on-disk size of the file is desirable. [1] https://sqlite.org/cli.html#sqlite_archive_support https://sqlite.org/cli.html#sqlite_archive_support
- happymellon 1mo ago
  Isn't the implementation the spec for SQLite? Not really a great option for an image format, where we therefore can't have multiple implementations. Unless I misremembered.
  - SQLite 1mo ago
    SQLite file format spec: <https://sqlite.org/fileformat.html https://sqlite.org/fileformat.html>
    - happymellon 1mo ago
      Thank you for the prompt clarification!
flohofwoe 1mo ago
The problem with using database blobs for load/save is that you usually need a full database client in the application. SQlite advertises that use case, but it is complete overkill. You never need to run any sort of complex SQL query on an image file format for instance. Using XML+ZIP in this day and age is also a strange decision, but at least that way the data is inspectable with unzip, a text editor and an image viewer (assuming they use a standard image format to store the raw pixel data).
- TeMPOraL 1mo ago
  > You never need to run any sort of complex SQL query on an image file format for instance. Sure you will. Plenty of features that don't exist, or are implemented badly, because you can't easily do it. Quick mental translation table: if you think "iterate over every ..." or a `for` loop, that's your SELECT query. If you think about `if` conditions, that's the parts that go after FROM clause.
  - x3ro 1mo ago
    In order to have any advantage from this, you would have the added complexity of splitting your file format into tables that can be queried in a useful manner. However, for an image file format, you most likely need to hold the entire definition in memory at all times anyway. Assuming that’s the case, doesn’t XPath get you there most of the way (assuming XML), with _way_ less complexity?
    - TeMPOraL 1mo ago
      Image data is just binary blobs. You aren't splitting that into channel columns or anything. But an image file for an editor like Gimp isn't one image blob. It's dozens or hundreds of them - one or more per layer - along with tons of associated metadata at every level. All that tends to fit sensible schemas and managing it is what SQLite shines at.
      - x3ro 1mo ago
        I don’t see how this addresses my point that zipped XML gives you the same thing, but simpler. I understand that a GIMP file is many images, so that makes a zip feel like a great fit to me. The only advantage I see for using a full-blown DB is ensuring consistency with references, which admittedly is a plus. But beyond that, what do you gain?

speedgoose 1mo ago

Compressed JSON with the binary content encoded in base64 strings, obviously.

einpoklum 1mo ago
Why would zipped JSON be fundamentally superior to zipped XML?
- speedgoose 1mo ago
  To answer seriously, my parent comment is a joke, JSON is a simpler format that maps better to most programming languages internal memory representations. Developers tend to prefer JSON’s simplicity over XML.
  - berkes 1mo ago
    > most programming languages internal memory representations Often heard wrt JSON but incorrect. It maps to the primitive types in JavaScript. But almost all programming languages treat floats and integers different, make distinction between char and strings and many have some form of date/time. JSON has neither. In that direction, XML is much closer since every node is a triple (name, value, attributes) so can have type info, json is a tuple. And Protobuf, while not popular, gets this completely right.
    - speedgoose 1mo ago
      I think it's too risky to treat numbers in JSON as something else than IEEE754 64bits floats. But yes, JSON is small and doesn't do datetimes, char, comments, and a million other things XML does. But you don't need to think much about memory representation when you parse a JSON, and the developer experience is a lot more pleasing than browsing a XML tree. That what used to matter.
  - einpoklum 1mo ago
    If this is intended for reading and writing by humans, then JSON doesn't cut it since you don't get comments (and perhaps also because of the weirdness of 64-bit floating-point values). Plus, XML is more "structurally nuanced". I suppose JSON is simpler to parse, but it's a trade-off of features, it's not like one is bad and the other is good.