10 ms·

I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes we

by pengaru 1mo ago

I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size. Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written. The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.

ValdikSS 1mo ago
journald uses hash tables, I think it update it on every new log line, although I didn't debug it in depth yet. https://github.com/systemd/systemd/blob/199f75205b9c0625bf56e229b12b95a645ed7a6c/src/journal/journal-file.c#L1083 https://github.com/systemd/systemd/blob/199f75205b9c0625bf56...
- pengaru 1mo ago
  The format is documented https://github.com/systemd/systemd/blob/main/docs/JOURNAL_FILE_FORMAT.md https://github.com/systemd/systemd/blob/main/docs/JOURNAL_FI...
cloudie78 1mo ago
Why not just have a SQLite file and call it a day? Also, why mmaped file?
- pengaru 1mo ago
  I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic. There was mailing list discussion at the time journald was conceived though, you can find it if you look. https://0pointer.de/blog/projects/the-journal.html https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.
  - marginalia_nu 1mo ago
    Well there was an ambition, apparently. > Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance > Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
  - otterley 1mo ago
    The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-January/004188.html https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
    - pengaru 1mo ago
      FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.
      - 1mo ago
quotemstr 1mo ago
Thank you for your work. ISTM the workload is naturally LSM-shaped. > If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written. Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple. > I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily.
- amluto 1mo ago
  Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.
  - quotemstr 1mo ago
    Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.
    - hedora 1mo ago
      Or, you could write a plain text file. Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice. If you really care about nulls in logs for ideological reasons, you could write a few lines of code that finds the first stream of nulls in the text file, then truncates there. In practice, no one wants that. It is strictly worse than returning partial entries after the hole, and by the time you are hitting this corner case, you are debugging a kernel crash.
otterley 1mo ago
> I'm probably the main person responsible for making journald usable at all. Thank you for your service!
crabbone 1mo ago
I've met this unwarranted love for mmap() many times in the developers who never professionally worked on storage projects. Especially common with C++ programmers for some reason. There are people who think they found a "trick" to make I/O go faster and never consider why filesystems or databases don't use it... Like, obviously, those losers who wrote eg. Ext4 never bothered to look at the system interface, right? On the other hand, if I was ever to advise anyone on how to do I/O when they are working with an (unknown) filesystem... It's really hard. And I'd probably default to saying "do as few tricks as possible" because filesystems today are very elaborate, with a lot of optimizations that are very difficult to predict from user-space. It's quite possible that someone trying to outsmart a filesystem will end up harming themselves in the process. Doing as few tricks as possible would allow the administrator to configure the filesystem independently of the program writing to it to match the nature of the workload instead of locking the program into a specific pattern of operation that might be impossible to rectify with administrative tools. Not an ideal situation by any means: storage-heavy user-space applications s.a. databases usually do the opposite: they try to optimize for the specific filesystem, its version and quirks... but it takes a lot of effort, obviously.
- ValdikSS 1mo ago
  Libtorrent 2.0 switched exclusively to mmaped read/writes for torrent downloads, which resulted in various performance and especially memory consumption issues on ALL platforms. For some reason Windows handled increased memory consumption the least gracefully. Many people continued to use v1.2 which use regular files. V2.1 ended up using pread/pwrite nowz it's fine now. The issue continued for 3 years more or less. https://github.com/arvidn/libtorrent/issues/6667 https://github.com/arvidn/libtorrent/issues/6667
- pineapplepizza6 1mo ago
  The principled excuse for mmap is when you're reading all over a file at high performance and you want to avoid either excessive syscalls or double caching. Which sounds like what a torrent program does but evidently it doesn't even work well for them.
- fghj117 1mo ago

by pengaru 1mo ago

ValdikSS 1mo ago

journald uses hash tables, I think it update it on every new log line, although I didn't debug it in depth yet. https://github.com/systemd/systemd/blob/199f75205b9c0625bf56e229b12b95a645ed7a6c/src/journal/journal-file.c#L1083 https://github.com/systemd/systemd/blob/199f75205b9c0625bf56...

pengaru 1mo ago
The format is documented https://github.com/systemd/systemd/blob/main/docs/JOURNAL_FILE_FORMAT.md https://github.com/systemd/systemd/blob/main/docs/JOURNAL_FI...

cloudie78 1mo ago

Why not just have a SQLite file and call it a day? Also, why mmaped file?

pengaru 1mo ago
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic. There was mailing list discussion at the time journald was conceived though, you can find it if you look. https://0pointer.de/blog/projects/the-journal.html https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.
- marginalia_nu 1mo ago
  Well there was an ambition, apparently. > Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance > Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
- otterley 1mo ago
  The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-January/004188.html https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
  - pengaru 1mo ago
    FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.
    - 1mo ago

quotemstr 1mo ago

Thank you for your work. ISTM the workload is naturally LSM-shaped. > If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written. Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple. > I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily.

amluto 1mo ago
Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.
- quotemstr 1mo ago
  Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.
  - hedora 1mo ago
    Or, you could write a plain text file. Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice. If you really care about nulls in logs for ideological reasons, you could write a few lines of code that finds the first stream of nulls in the text file, then truncates there. In practice, no one wants that. It is strictly worse than returning partial entries after the hole, and by the time you are hitting this corner case, you are debugging a kernel crash.

otterley 1mo ago

> I'm probably the main person responsible for making journald usable at all. Thank you for your service!

crabbone 1mo ago

I've met this unwarranted love for mmap() many times in the developers who never professionally worked on storage projects. Especially common with C++ programmers for some reason. There are people who think they found a "trick" to make I/O go faster and never consider why filesystems or databases don't use it... Like, obviously, those losers who wrote eg. Ext4 never bothered to look at the system interface, right? On the other hand, if I was ever to advise anyone on how to do I/O when they are working with an (unknown) filesystem... It's really hard. And I'd probably default to saying "do as few tricks as possible" because filesystems today are very elaborate, with a lot of optimizations that are very difficult to predict from user-space. It's quite possible that someone trying to outsmart a filesystem will end up harming themselves in the process. Doing as few tricks as possible would allow the administrator to configure the filesystem independently of the program writing to it to match the nature of the workload instead of locking the program into a specific pattern of operation that might be impossible to rectify with administrative tools. Not an ideal situation by any means: storage-heavy user-space applications s.a. databases usually do the opposite: they try to optimize for the specific filesystem, its version and quirks... but it takes a lot of effort, obviously.

ValdikSS 1mo ago
Libtorrent 2.0 switched exclusively to mmaped read/writes for torrent downloads, which resulted in various performance and especially memory consumption issues on ALL platforms. For some reason Windows handled increased memory consumption the least gracefully. Many people continued to use v1.2 which use regular files. V2.1 ended up using pread/pwrite nowz it's fine now. The issue continued for 3 years more or less. https://github.com/arvidn/libtorrent/issues/6667 https://github.com/arvidn/libtorrent/issues/6667
pineapplepizza6 1mo ago
The principled excuse for mmap is when you're reading all over a file at high performance and you want to avoid either excessive syscalls or double caching. Which sounds like what a torrent program does but evidently it doesn't even work well for them.
fghj117 1mo ago