6 ms·

Automatic memory management is indeed the first thing one needs to look for writing performance critical software, and that's a first in my check-list. But > i

Automatic memory management is indeed the first thing one needs to look for writing performance critical software, and that's a first in my check-list. But > in-memory storage of databases Doesn't that sound a bit expensive to have large capacity memory? Although the expense of R/W IO is far cheaper for in-memory analysis. Is such trade-off worth it?

mbuda 4y ago
Excellent observation/question :D It depends; sometimes, it's worth it, and sometimes it is not (as always with tradeoffs). Graphs are a bit specific because most of the traversals or expensive graph analytics like PageRank touch the whole graph (even multiple times) -> the entire graph will end up in memory -> why not keep it in memory for faster performance? But for a vast dataset, the hardware cost might be too much. I think we are aware of the tradeoff. We'll probably provide disk first storage option at some point because that's definitely a valid setup (sometimes the only possible setup). Ofc, we'll invest time in making it as performant as possible. Do you have some specific workload in mind? :D
- worthless443 4y ago
  If a large graph is needed to be read multiple times, sure memory bandwidth will result in the most performance possible under the context of this workload like interacting with PageRank (and going further with optimization techniques on memory allocation and management, will boost the performance even further). So to my understanding (and a novice one at that), the graph should be stored on disk first, upon initializing the objects will have to be an one-time copy to volatile memory but I question, memory regions are more likely to yield faults and get corrupt and thus graph stored in-memory is also completely flushed? (unless the results are being saved to disk in-between specific intervals of time?) Does that make any sense?
  - mbuda 4y ago
    I'm not sure I understand the part about corruption. How would data in memory become corrupted? How Memgraph currently works, it stores data in memory, and async starts writing data to disk in small data chunks called deltas, later these chunks are deleted and replaced with the whole graph snapshot (there is also a sync option, but that's slower in terms of committing a transaction, letting the user know data is written, e.g., RocksDB works similarly). All disk-related stuff is purely for durability (recovery after the Memgraph process restarts and all interactions with the disk are made automatically in the background during standard system runtime and startup time).
    - worthless443 4y ago
      > it stores data in memory, and async starts writing data to disk in small data chunks called deltas, later these chunks are deleted and replaced with the whole graph snapshot Thanks, that fairly answers my question of recoverability of in-memory graphs.
      - mbuda 4y ago
        Perfect!

mbuda 4y ago

Excellent observation/question :D It depends; sometimes, it's worth it, and sometimes it is not (as always with tradeoffs). Graphs are a bit specific because most of the traversals or expensive graph analytics like PageRank touch the whole graph (even multiple times) -> the entire graph will end up in memory -> why not keep it in memory for faster performance? But for a vast dataset, the hardware cost might be too much. I think we are aware of the tradeoff. We'll probably provide disk first storage option at some point because that's definitely a valid setup (sometimes the only possible setup). Ofc, we'll invest time in making it as performant as possible. Do you have some specific workload in mind? :D

worthless443 4y ago
If a large graph is needed to be read multiple times, sure memory bandwidth will result in the most performance possible under the context of this workload like interacting with PageRank (and going further with optimization techniques on memory allocation and management, will boost the performance even further). So to my understanding (and a novice one at that), the graph should be stored on disk first, upon initializing the objects will have to be an one-time copy to volatile memory but I question, memory regions are more likely to yield faults and get corrupt and thus graph stored in-memory is also completely flushed? (unless the results are being saved to disk in-between specific intervals of time?) Does that make any sense?
- mbuda 4y ago
  I'm not sure I understand the part about corruption. How would data in memory become corrupted? How Memgraph currently works, it stores data in memory, and async starts writing data to disk in small data chunks called deltas, later these chunks are deleted and replaced with the whole graph snapshot (there is also a sync option, but that's slower in terms of committing a transaction, letting the user know data is written, e.g., RocksDB works similarly). All disk-related stuff is purely for durability (recovery after the Memgraph process restarts and all interactions with the disk are made automatically in the background during standard system runtime and startup time).
  - worthless443 4y ago
    > it stores data in memory, and async starts writing data to disk in small data chunks called deltas, later these chunks are deleted and replaced with the whole graph snapshot Thanks, that fairly answers my question of recoverability of in-memory graphs.
    - mbuda 4y ago
      Perfect!