9 ms·
How do you implement lockless atomic updates for multiple writers across multiple threads & processes without mmap? With mmap it is straight forward for proces
by randbox 5y ago
How do you implement lockless atomic updates for multiple writers across multiple threads & processes without mmap?
With mmap it is straight forward for processes to open persistent arrays of atomics as a file, and use compare and exchange operations to prevent data races when multiple threads or processes update the same page without any file locks, advisory locks, or mutexes.
With manual read() and write() calls, the data may be overwritten by another writer before the update is committed.
- jstimpfle 5y agoWhy do you need lockless atomic updates to a file-backed memory area? Genuinely curious.
- CyberDildonics 5y agoBecause it allows you to do lock free memory based interprocess communication, which can be extremely fast.
- jstimpfle 5y agoThere is no need for file-backed memory to do that.
- CyberDildonics 5y agoSounds good, what is your solution and why didn't you explain it in your first reply?
- jstimpfle 5y agoWhat is my solution to what? To database I/O? I guess that's what the article is about...
- CyberDildonics 5y agoI said "lock free memory based interprocess communication" You said "There is no need for file-backed memory to do that." If that is true, I want to know what you are talking about, so I'm just asking you to back up the claim you made.
- jstimpfle 5y agoNo. _I_ want to know what we're talking about, as my original question clearly indicates. You can do "lock-free memory based interprocess communication" with memory (obviously). There is no need to back this memory with files, certainly not files on a hard drive that you would otherwise access using read() and write(). Hence my original question.
- CyberDildonics 5y ago_I_ want to know what we're talking about lock free memory based interprocess communication (copied from my first reply) There is no need to back this memory with files Again, I'm interested in how you have two processes read and write lock free directly to the same memory if you don't use a memory mapped file. You have said it isn't necessary, I'm just asking you to back up this claim and explain exactly what you mean.
- jstimpfle 5y agoFirst, I didn't assume it a requirement to have two processes read() and write() _directly_ to the same memory (I suppose you meant "file region" here). And idk, it might not be a good idea to require that. Also, you can use normal (non-file-backed) memory to do the necessary synchronization (lock-free or not). I'm still not seeing why the memory should be backed by a file, that's why I was genuinely asking. One reason why it could be practical that I can now see could be for an embedded database like sqlite, but again I'm not sure it would be a good idea. While it would allow for pretty much setup-less synchronization of otherwise uncoordinated processes, it's a fringe application that might be better implemented with one big flock(). And one reason why it could be not a good idea is that it might couple the file format to a particular CPU architecture. Another big issue I guess is that the atomics actually do have an effect to the underlying file whenever the pages are flushed. What if the computer shuts down unexpectedly? The synchronization affairs aren't cleaned up, yet the original processes are gone.
- pclmulqdq 5y agoNormally, your IPC structures where you put lock-free data structures are mmaped in tmpfs, which is backed by RAM only, not files. A lot of the problems with mmap-ed files only show up when the file is larger than RAM (which is the case with databases). Files for IPC in tmpfs are usually small and don't have that problem.