5 ms·

No, if you are going to change the structure of a structured document that has been saved to disk, your options are: 1) Rewrite the file to disk 2) Append the

No, if you are going to change the structure of a structured document that has been saved to disk, your options are: 1) Rewrite the file to disk 2) Append the new data/metadata to the end of the existing file I suppose you could pre-pad documents with empty blocks and then go modify those in situ by binary editing the file, but that sounds like a nightmare.

cubefox 8mo ago
Aren't there file systems that support data structures which allow editing just part of the data, like linked lists?
- layer8 8mo ago
  Look at the C file API which most software is based on, it simply doesn’t allow it. Writing at a given file position just overwrites existing content. There is no way to insert or remove bytes in the middle. Apart from that, file systems manage storage in larger fixed-size blocks (commonly 4 KB). One block typically links to the next block (if any) of the same file, but that’s about the extent of it.
  - anthk 8mo ago
    DD should.
- formerly_proven 8mo ago
  No. Well yes. On mainframes. This is why “table of contents at the end” is such an exceedingly common design choice.
- PhilipRoman 8mo ago
  Yeah there are, Linux supports parameters FALLOC_FL_INSERT_RANGE and FALLOC_FL_COLLAPSE_RANGE for fallocate(2). Like most fancy filesystem features, they are not used by the vast majority of software because it has to run on any filesystem so you'd always need to maintain two implementations (and extensive test cases).
  - cubefox 8mo ago
    Interesting that after decades of file system history, this is still considered a "fancy feature", considering that editing files is a pretty basic operation for a file system. Though I assume there are reasons why this hasn't become standard long ago.
    - layer8 8mo ago
      File systems aren’t databases; they manage flat files, not structured data. You also can’t just insert/remove random amounts of bytes in RAM. The considerations here are actually quite similar, like fragmentation. If you make a hundred small edits to a file, you might end up with the file taking up ten times as much space due to fragmentation, and then you’d need the file system to do some sort of defragmentation pass to rewrite the file more contiguously again. In addition, it’s generally nontrivial for a program to map changes to an in-memory object structure back to surgical edits of a flat file. It’s much easier to always just serialize the whole thing, or if the file format allows it, appending the serialized changes to the file.
      - PhilipRoman 8mo ago
        Indeed, also userspace-level atomicity is important, so you probably want to save a backup in case power goes out at an unfortunate moment. And since you already need to have a backup, might as well go for a full rewrite + rename combo.
      - ogurechny 8mo ago
        File systems aren't databases, but journaling file systems use journals just like databases. It can theoretically define any granularity for something that might happen to a file to become an irreversible transaction. I suppose that file systems have to remain “general purpose enough” to be useful (otherwise they become part of the specific program or library), and that's why complex features which might become a pitfall for the regular users who expect “just regular files” rarely become the main focus.
      - cubefox 8mo ago

cubefox 8mo ago

Aren't there file systems that support data structures which allow editing just part of the data, like linked lists?

layer8 8mo ago
Look at the C file API which most software is based on, it simply doesn’t allow it. Writing at a given file position just overwrites existing content. There is no way to insert or remove bytes in the middle. Apart from that, file systems manage storage in larger fixed-size blocks (commonly 4 KB). One block typically links to the next block (if any) of the same file, but that’s about the extent of it.
- anthk 8mo ago
  DD should.
formerly_proven 8mo ago
No. Well yes. On mainframes. This is why “table of contents at the end” is such an exceedingly common design choice.
PhilipRoman 8mo ago
Yeah there are, Linux supports parameters FALLOC_FL_INSERT_RANGE and FALLOC_FL_COLLAPSE_RANGE for fallocate(2). Like most fancy filesystem features, they are not used by the vast majority of software because it has to run on any filesystem so you'd always need to maintain two implementations (and extensive test cases).
- cubefox 8mo ago
  Interesting that after decades of file system history, this is still considered a "fancy feature", considering that editing files is a pretty basic operation for a file system. Though I assume there are reasons why this hasn't become standard long ago.
  - layer8 8mo ago
    File systems aren’t databases; they manage flat files, not structured data. You also can’t just insert/remove random amounts of bytes in RAM. The considerations here are actually quite similar, like fragmentation. If you make a hundred small edits to a file, you might end up with the file taking up ten times as much space due to fragmentation, and then you’d need the file system to do some sort of defragmentation pass to rewrite the file more contiguously again. In addition, it’s generally nontrivial for a program to map changes to an in-memory object structure back to surgical edits of a flat file. It’s much easier to always just serialize the whole thing, or if the file format allows it, appending the serialized changes to the file.
    - PhilipRoman 8mo ago
      Indeed, also userspace-level atomicity is important, so you probably want to save a backup in case power goes out at an unfortunate moment. And since you already need to have a backup, might as well go for a full rewrite + rename combo.
    - ogurechny 8mo ago
      File systems aren't databases, but journaling file systems use journals just like databases. It can theoretically define any granularity for something that might happen to a file to become an irreversible transaction. I suppose that file systems have to remain “general purpose enough” to be useful (otherwise they become part of the specific program or library), and that's why complex features which might become a pitfall for the regular users who expect “just regular files” rarely become the main focus.
    - cubefox 8mo ago