5 ms·

RavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to-use-mmap-in-your-database-management-system https://ayende.com/bl

RavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to-use-mmap-in-your-database-management-system https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to...

tyingq 5y ago
The really key part seems to be this: "If you aren’t using mmap, on the other hand, you still need to handle of all those issues" Which seems like a reasonable statement. Is it less work to make your own top-to-bottom buffer pool, and would that necessarily avoid similar issues? Or is it less work to use mmap(), but address the issues?
- dboreham 5y ago
  When I worked on/with BerkeleyDB in the late 90s we came to the conclusion that the various OS mmap() implementations had been tweaked/fixed to the point where they worked for the popular high profile applications (in those days: Oracle). So it can appear like everything is fine, but that probably means your code behaves the same way as <popular database du jour>.
  - ayende 5y ago
    Yes, isn't that wonderful? You get to take advantage of literally decades of experience What is more, if you can match the profile of the optimization, you can benefit even more
  - tytso 5y ago
    Um... Oracle (and other enterprise databases like DB2) don't use mmap. They use Direct I/O. Oracle does have anonymous (non-file-backed) memory which is mmap'ed and shared across various Oracle processes, called the Shared Global Area (SGA), but it's not used for I/O.
    - hyc_symas 5y ago
      Fwiw, I wrote a Direct I/O patch for BerkeleyDB but withdrew it later because it didn't ever improve I/O perf or memory footprint.
- AdamProut 5y ago
  I suppose. Some problems with mmap() are a bit hard to fix from user land though. You will hit contention on locks inside the kernel (mmap_sem) if the database does concurrent high throughput mmap()/unmap(). I don't follow linux kernel development closely to know if this has been improved recently, but it was easy to reproduce it 4-5 years ago.
  - tyingq 5y ago
    That makes sense. I wasn't going right to the conclusion that working around mmap() issues was easier, but it didn't seem to be explored much. Is the contention around having one file mmap()ed, or is it reduced if you use more files?
  - ayende 5y ago
    Almost no one is going to have a lot of map calls Uou map the file once, then fault it in
- bluestreak 5y ago
  Questdb's author here. I do share Ayende's sentiment. There are things that the OP paper doesn't mention, which can help mitigate some of the disadvantages: - single-threaded calls to 'fallocate' will help avoiding sparse files and SIGBUS during memory write - over-allocating, caching memory addresses and minimizing OS calls - transactional safety can be implemented via shared memory model - hugetlb can minimize TLB shootdowns I personally do not have any regrets using mmap because of all the benefits they provide
- jandrewrogers 5y ago
  Some issues with mmap() can be avoided entirely if you have your own buffer pool. Others are easier to handle because they are made explicit and more buffer state is exposed to the program logic. That's the positive side. The downside is that writing an excellent buffer pool is not trivial, especially if you haven't done it before. There are many cross-cutting design concerns that have to be accounted for. In my experience, an excellent C++ implementation tends to be on the order of 2,000 lines of code -- someone has to write that. It also isn't simple code, the logic is relatively dense and subtle.
ikawe 5y ago
> Off the top of my head, most embedded databases implement a single writer model. LMDB, Voron (RavenDB’s storage engine), LevelDB, Lucene And let's not forget sqlite! > There can only be a single writer at a time to an SQLite database. (from https://www.sqlite.org/isolation.html https://www.sqlite.org/isolation.html)
tptacek 5y ago
From that article: the whole fsyncgate thing seems like a pretty strong counterargument to "mmap adds more complexity than it removes": https://danluu.com/fsyncgate/ https://danluu.com/fsyncgate/
- ayende 5y ago
  That actually doesn't matter This is orthogonal to mmap
  - aidenn0 5y ago
    As you pointed out in your article, it invalidates much (all?) of the the "Problem #3: Error handling" section of TFA.
10000truths 5y ago
You don't want the OS to take care of reading from disk and page caching/eviction. You want the DB itself to have explicit control over that, because the DB has information on access patterns and table format that the OS is not aware of. It is better equipped than the OS to anticipate what portions of tables/indices need to be cached in memory. It is better equipped to calculate when/where/what/how much to prefetch from disk. It is better equipped to determine when to buffer writes and when to flush to disk. Sure, it might be more work than using mmap. But it's also more correct, forces you to handle edge cases, and much more amenable to platform-specific improvements a la kqueue/io_uring.
- ayende 5y ago
  I'm the author (well, one of) RavenDB You are correct to an extent, but there are a few things yo noted. * you can design your system so the access pattern that the OS is optimized for matches your needs * you can use madvise() to give some useful hints * the amount of complexity you don't have to deal with is staggering
  - tytso 5y ago
    OTOH, if you care about that last 5 percent or so of performance there is the complexity that what the OS has optimized for might be different between different OS's (e.g., MacOS, Linux, FreeBSd, etc.) and indeed, might change between different versions of Linux, or even, in the case of buffered writeback, between different filesystems on the same version of Linux. This is probably historically one of the most important reasons why enterprise databases like Oracle DB, DB2, etc., have used direct I/O, and not buffered I/O or mmap. Speaking as an OS developer, we're not going to try to optimize buffered I/O for a particular database. We'll be using becnhmarks like compilebench and postmark to optimize our I/O, and if your write patterns, or readahead patterns, or caching requirements, don't match those workloads, well.... sucks to be you. I'll also point out that those big companies that actually pay the salarise of us file system developers (e.g., Oracle, Google, etc.) for the most part use Direct I/O for our performance critical workloads. If database companies that want to use mmap want to hire file system developers and contribute benchmarks and performance patches for ext4, xfs, etc., speaking as the ext4 maintainer, I'll welcome that, and we do have a weekly video conference where I'd love to have your engineers join to discuss your contributions. :-)
dwenzek 5y ago
Thank you for these counter-arguments. It's good to have them to make up your own mind, especially when recognized experts use a mocking tone "you will no dare think the contrary".

tyingq 5y ago

The really key part seems to be this: "If you aren’t using mmap, on the other hand, you still need to handle of all those issues" Which seems like a reasonable statement. Is it less work to make your own top-to-bottom buffer pool, and would that necessarily avoid similar issues? Or is it less work to use mmap(), but address the issues?

dboreham 5y ago
When I worked on/with BerkeleyDB in the late 90s we came to the conclusion that the various OS mmap() implementations had been tweaked/fixed to the point where they worked for the popular high profile applications (in those days: Oracle). So it can appear like everything is fine, but that probably means your code behaves the same way as <popular database du jour>.
- ayende 5y ago
  Yes, isn't that wonderful? You get to take advantage of literally decades of experience What is more, if you can match the profile of the optimization, you can benefit even more
- tytso 5y ago
  Um... Oracle (and other enterprise databases like DB2) don't use mmap. They use Direct I/O. Oracle does have anonymous (non-file-backed) memory which is mmap'ed and shared across various Oracle processes, called the Shared Global Area (SGA), but it's not used for I/O.
  - hyc_symas 5y ago
    Fwiw, I wrote a Direct I/O patch for BerkeleyDB but withdrew it later because it didn't ever improve I/O perf or memory footprint.
AdamProut 5y ago
I suppose. Some problems with mmap() are a bit hard to fix from user land though. You will hit contention on locks inside the kernel (mmap_sem) if the database does concurrent high throughput mmap()/unmap(). I don't follow linux kernel development closely to know if this has been improved recently, but it was easy to reproduce it 4-5 years ago.
- tyingq 5y ago
  That makes sense. I wasn't going right to the conclusion that working around mmap() issues was easier, but it didn't seem to be explored much. Is the contention around having one file mmap()ed, or is it reduced if you use more files?
- ayende 5y ago
  Almost no one is going to have a lot of map calls Uou map the file once, then fault it in
bluestreak 5y ago
Questdb's author here. I do share Ayende's sentiment. There are things that the OP paper doesn't mention, which can help mitigate some of the disadvantages: - single-threaded calls to 'fallocate' will help avoiding sparse files and SIGBUS during memory write - over-allocating, caching memory addresses and minimizing OS calls - transactional safety can be implemented via shared memory model - hugetlb can minimize TLB shootdowns I personally do not have any regrets using mmap because of all the benefits they provide
jandrewrogers 5y ago
Some issues with mmap() can be avoided entirely if you have your own buffer pool. Others are easier to handle because they are made explicit and more buffer state is exposed to the program logic. That's the positive side. The downside is that writing an excellent buffer pool is not trivial, especially if you haven't done it before. There are many cross-cutting design concerns that have to be accounted for. In my experience, an excellent C++ implementation tends to be on the order of 2,000 lines of code -- someone has to write that. It also isn't simple code, the logic is relatively dense and subtle.

ikawe 5y ago

> Off the top of my head, most embedded databases implement a single writer model. LMDB, Voron (RavenDB’s storage engine), LevelDB, Lucene And let's not forget sqlite! > There can only be a single writer at a time to an SQLite database. (from https://www.sqlite.org/isolation.html https://www.sqlite.org/isolation.html)

tptacek 5y ago

From that article: the whole fsyncgate thing seems like a pretty strong counterargument to "mmap adds more complexity than it removes": https://danluu.com/fsyncgate/ https://danluu.com/fsyncgate/

ayende 5y ago
That actually doesn't matter This is orthogonal to mmap
- aidenn0 5y ago
  As you pointed out in your article, it invalidates much (all?) of the the "Problem #3: Error handling" section of TFA.

10000truths 5y ago

You don't want the OS to take care of reading from disk and page caching/eviction. You want the DB itself to have explicit control over that, because the DB has information on access patterns and table format that the OS is not aware of. It is better equipped than the OS to anticipate what portions of tables/indices need to be cached in memory. It is better equipped to calculate when/where/what/how much to prefetch from disk. It is better equipped to determine when to buffer writes and when to flush to disk. Sure, it might be more work than using mmap. But it's also more correct, forces you to handle edge cases, and much more amenable to platform-specific improvements a la kqueue/io_uring.

ayende 5y ago
I'm the author (well, one of) RavenDB You are correct to an extent, but there are a few things yo noted. * you can design your system so the access pattern that the OS is optimized for matches your needs * you can use madvise() to give some useful hints * the amount of complexity you don't have to deal with is staggering
- tytso 5y ago
  OTOH, if you care about that last 5 percent or so of performance there is the complexity that what the OS has optimized for might be different between different OS's (e.g., MacOS, Linux, FreeBSd, etc.) and indeed, might change between different versions of Linux, or even, in the case of buffered writeback, between different filesystems on the same version of Linux. This is probably historically one of the most important reasons why enterprise databases like Oracle DB, DB2, etc., have used direct I/O, and not buffered I/O or mmap. Speaking as an OS developer, we're not going to try to optimize buffered I/O for a particular database. We'll be using becnhmarks like compilebench and postmark to optimize our I/O, and if your write patterns, or readahead patterns, or caching requirements, don't match those workloads, well.... sucks to be you. I'll also point out that those big companies that actually pay the salarise of us file system developers (e.g., Oracle, Google, etc.) for the most part use Direct I/O for our performance critical workloads. If database companies that want to use mmap want to hire file system developers and contribute benchmarks and performance patches for ext4, xfs, etc., speaking as the ext4 maintainer, I'll welcome that, and we do have a weekly video conference where I'd love to have your engineers join to discuss your contributions. :-)

dwenzek 5y ago

Thank you for these counter-arguments. It's good to have them to make up your own mind, especially when recognized experts use a mocking tone "you will no dare think the contrary".