5 ms·

This is byte-addressable persistent memory. They look like DRAM DIMMs and they plug into DIMM slots. You access them using your memory controller and not your s

This is byte-addressable persistent memory. They look like DRAM DIMMs and they plug into DIMM slots. You access them using your memory controller and not your storage controller. People sometimes refer to them as non-volatile memory (NVM). Intel used to call it Apache Pass. They're a nightmare to program because OSes do not have a good abstraction for them (at least not yet). Accessing them through the file-system seems sub-optimal (this is byte-addressable memory and not a block device). Accessing them through virtual memory is also pretty bad because they're much slower than DRAM.

devit 8y ago
Is there are a reason to not just use DRAM along with a battery to achieve the same persistence but as fast as DRAM?
- manigandham 8y ago
  The reason is the battery. These devices can be powered off and save state, like an SSD.
  - piotrkaminski 8y ago
    But in the context of the OP, presumably devices in a data center would never be powered down on purpose to save energy? In which case it seems that battery-backup DRAM would work just as well for this use case, and be both cheaper and faster.
    - jordanthoms 8y ago
      Optane is in between DRAM and flash in terms of cost (and performance) - it's also denser, so you can fit much more storage on each DIMM slot.
- juancampa 8y ago
  Probably because systems that passively do their job tend to be preferred. Also a battery would only last so long. IIRC DRAM needs to be constantly refreshed, so, it would be a trade-off between capacity and duration. Optane seems[1] to be 20~30X slower than DRAM but 4~10X faster than server SSDs [1]: https://superuser.com/a/1195674/187732 https://superuser.com/a/1195674/187732
- zokier 8y ago
  DRAM can not be simply naively battery-backed; it needs active refreshing. And as memory controllers reside in CPUs these days, that would mean keeping the CPU powered up.
- wtallis 8y ago
  There are NVDIMMs that have DRAM and a matching quantity of NAND flash memory to save the contents to in the event of a power failure. They require an external capacitor module and are limited in data capacity by how much DRAM you can fit on the module. You can fit far more 3D XPoint memory on a module than DRAM, and it doesn't require the external capacitors to achieve persistence, and it should be significantly cheaper on a per-GB basis.
  - cdoxsey 8y ago
    Just to reiterate the point... this is an instance with terabytes of near-memory-speed storage. If persistent memory pans out as a technology it will completely upend the way we think about building software and the cost tradeoffs of hardware. (as much as or more so than the transition from spinning disks to ssds)
    - Dylan16807 8y ago
      How do you define "pans out"? What performance and price differences between it, flash, and DRAM do you have in mind? Because I'll keep reminding people that putting a DRAM cache in front of some flash can very closely approximate a large persistent memory. If people wanted to build software for that kind of system, they could do it today. The hardware is not the blocker.
      - wmf 8y ago
        Diablo Memory1 used flash DIMMs with DRAM DIMMs as cache. When we tried it, it worked OK for some workloads and poorly for others, but it was also buggy and then the company went out of business. So a large part of "pans out" is simply a production-quality implementation that you can buy. Note that Optane DIMMs have been delayed by around two years at this point and we still don't know what they will cost.
      - SergeAx 8y ago
        Cache will always stay just cache, unless it is the same size that an underlying persistent storage. You cannot read or write larger-than-cache chunks without performance degradation. Also you have a start-up cache-warming problem.
the8472 8y ago
> They're a nightmare to program because OSes do not have a good abstraction for them (at least not yet). With DAX[0] linux already has the ability to put a filesystem (currently ext4 and xfs) on NVDIMMS and then let userspace address them through mmap while skipping the page cache indirection. I.e. you're directly byte-addressing them through the memory controller via standard memory-mapped file abstractions. Direct block device mapping of nvdimms without filesystem is also possible. [0] https://www.kernel.org/doc/Documentation/filesystems/dax.txt https://www.kernel.org/doc/Documentation/filesystems/dax.txt
- markdoubleyou 8y ago
  The Persistent Memory Development Kit (PMDK) offers high-level abstractions over DAX. Looks like most mortals would use libpmemobj (or its C++ bindings). http://pmem.io/pmdk/ http://pmem.io/pmdk/
pbalcer 8y ago
Disclaimer: I work at Intel on PMDK (pmem.io) Both Windows and Linux implement DAX, which, as @the8472 explained, allows bypassing page cache in memory mapped I/O. Additionally, DAX optionally allows you to flush your data directly from user-space instead of calling msync. And that's the gist of NVM programming model [0], its entire point is to allow applications to avoid the now hugely excessive abstraction layer of traditional storage. And I will freely admit that programming to raw memory mapped files can be difficult, but there is ongoing work on making it easier. An example of that is, excuse the shameless plug, Persistent Memory Development Kit [1], which makes writing new software for this new type of memory much simpler. Performance of an NVDIMM is obviously hardware dependent, but the now widely accepted programming model works with the assumption that persistent memory is fast enough so that it is reasonable to stall a CPU while an instruction is accessing it. I'm not sure on what hardware evaluations you are basing your claims on, but let me assure you that the HW solution being described in the blog post does not violate that assumption. [0] - https://www.snia.org/tech_activities/standards/curr_standards/npm https://www.snia.org/tech_activities/standards/curr_standard... [1] - http://pmem.io/ http://pmem.io/
- simcop2387 8y ago
  Do you know how well tools like Cap'n Proto and Protocol Buffers help for dealing with this kind of scenario? I'd imagine that some kind of low latency/cost serialization system would help significantly in using the device. Cap'n Proto I'd imagine would work nicely for reading data off since it should be able to read and use the structure with no extra copying or decoding, but I have no idea how the situation with writing would win out.
  - wtallis 8y ago
    The simplest way of using this is to not do any serialization at all, just store any information you want persisted in memory allocated from the region you mmaped to the Optane DIMMs instead of the DRAM DIMMs.
    - simcop2387 8y ago
Symmetry 8y ago
It seems like Single-level store[1] would be a really good fit for this. I was going to make a crack about bringing back Multics but apparently IBM has an OS using this. [1]https://en.wikipedia.org/wiki/Single-level_store https://en.wikipedia.org/wiki/Single-level_store

devit 8y ago

Is there are a reason to not just use DRAM along with a battery to achieve the same persistence but as fast as DRAM?

manigandham 8y ago
The reason is the battery. These devices can be powered off and save state, like an SSD.
- piotrkaminski 8y ago
  But in the context of the OP, presumably devices in a data center would never be powered down on purpose to save energy? In which case it seems that battery-backup DRAM would work just as well for this use case, and be both cheaper and faster.
  - jordanthoms 8y ago
    Optane is in between DRAM and flash in terms of cost (and performance) - it's also denser, so you can fit much more storage on each DIMM slot.
juancampa 8y ago
Probably because systems that passively do their job tend to be preferred. Also a battery would only last so long. IIRC DRAM needs to be constantly refreshed, so, it would be a trade-off between capacity and duration. Optane seems[1] to be 20~30X slower than DRAM but 4~10X faster than server SSDs [1]: https://superuser.com/a/1195674/187732 https://superuser.com/a/1195674/187732
zokier 8y ago
DRAM can not be simply naively battery-backed; it needs active refreshing. And as memory controllers reside in CPUs these days, that would mean keeping the CPU powered up.
wtallis 8y ago
There are NVDIMMs that have DRAM and a matching quantity of NAND flash memory to save the contents to in the event of a power failure. They require an external capacitor module and are limited in data capacity by how much DRAM you can fit on the module. You can fit far more 3D XPoint memory on a module than DRAM, and it doesn't require the external capacitors to achieve persistence, and it should be significantly cheaper on a per-GB basis.
- cdoxsey 8y ago
  Just to reiterate the point... this is an instance with terabytes of near-memory-speed storage. If persistent memory pans out as a technology it will completely upend the way we think about building software and the cost tradeoffs of hardware. (as much as or more so than the transition from spinning disks to ssds)
  - Dylan16807 8y ago
    How do you define "pans out"? What performance and price differences between it, flash, and DRAM do you have in mind? Because I'll keep reminding people that putting a DRAM cache in front of some flash can very closely approximate a large persistent memory. If people wanted to build software for that kind of system, they could do it today. The hardware is not the blocker.
    - wmf 8y ago
      Diablo Memory1 used flash DIMMs with DRAM DIMMs as cache. When we tried it, it worked OK for some workloads and poorly for others, but it was also buggy and then the company went out of business. So a large part of "pans out" is simply a production-quality implementation that you can buy. Note that Optane DIMMs have been delayed by around two years at this point and we still don't know what they will cost.
    - SergeAx 8y ago
      Cache will always stay just cache, unless it is the same size that an underlying persistent storage. You cannot read or write larger-than-cache chunks without performance degradation. Also you have a start-up cache-warming problem.

the8472 8y ago

> They're a nightmare to program because OSes do not have a good abstraction for them (at least not yet). With DAX[0] linux already has the ability to put a filesystem (currently ext4 and xfs) on NVDIMMS and then let userspace address them through mmap while skipping the page cache indirection. I.e. you're directly byte-addressing them through the memory controller via standard memory-mapped file abstractions. Direct block device mapping of nvdimms without filesystem is also possible. [0] https://www.kernel.org/doc/Documentation/filesystems/dax.txt https://www.kernel.org/doc/Documentation/filesystems/dax.txt

markdoubleyou 8y ago
The Persistent Memory Development Kit (PMDK) offers high-level abstractions over DAX. Looks like most mortals would use libpmemobj (or its C++ bindings). http://pmem.io/pmdk/ http://pmem.io/pmdk/

pbalcer 8y ago

Disclaimer: I work at Intel on PMDK (pmem.io) Both Windows and Linux implement DAX, which, as @the8472 explained, allows bypassing page cache in memory mapped I/O. Additionally, DAX optionally allows you to flush your data directly from user-space instead of calling msync. And that's the gist of NVM programming model [0], its entire point is to allow applications to avoid the now hugely excessive abstraction layer of traditional storage. And I will freely admit that programming to raw memory mapped files can be difficult, but there is ongoing work on making it easier. An example of that is, excuse the shameless plug, Persistent Memory Development Kit [1], which makes writing new software for this new type of memory much simpler. Performance of an NVDIMM is obviously hardware dependent, but the now widely accepted programming model works with the assumption that persistent memory is fast enough so that it is reasonable to stall a CPU while an instruction is accessing it. I'm not sure on what hardware evaluations you are basing your claims on, but let me assure you that the HW solution being described in the blog post does not violate that assumption. [0] - https://www.snia.org/tech_activities/standards/curr_standards/npm https://www.snia.org/tech_activities/standards/curr_standard... [1] - http://pmem.io/ http://pmem.io/

simcop2387 8y ago
Do you know how well tools like Cap'n Proto and Protocol Buffers help for dealing with this kind of scenario? I'd imagine that some kind of low latency/cost serialization system would help significantly in using the device. Cap'n Proto I'd imagine would work nicely for reading data off since it should be able to read and use the structure with no extra copying or decoding, but I have no idea how the situation with writing would win out.
- wtallis 8y ago
  The simplest way of using this is to not do any serialization at all, just store any information you want persisted in memory allocated from the region you mmaped to the Optane DIMMs instead of the DRAM DIMMs.
  - simcop2387 8y ago

Symmetry 8y ago

It seems like Single-level store[1] would be a really good fit for this. I was going to make a crack about bringing back Multics but apparently IBM has an OS using this. [1]https://en.wikipedia.org/wiki/Single-level_store https://en.wikipedia.org/wiki/Single-level_store