6 ms·

The Storage Tipping Point

70 points by jcreedon 11y ago

mark_l_watson 11y ago
Good to see Robin Harris' article on the front page of HN (we live in the same small town and a mutual friend introduced us). A little off topic from the article: I think the biggest tipping point is the decreasing cost of cloud storage because large companies like Google do so much of their time critical processing with data in RAM (spread over many servers) and these servers can re-purpose a lot of their disk I/O capability to service low cost cloud storage.
- dikaiosune 11y ago
  Can't believe I've seen two people from there on the same day on HN! Flabbergasted. Grew up there, currently living just up the rim. Re: your comment, I'm curious to see how the I/O capacity of those servers could be distributed for cloud storage. I would imagine that even with 10Gbps if those machines are really cranking, they could have a difficult time provisioning network I/O for a storage service.
  - mark_l_watson 11y ago
    Email me when you will be in Sedona and we can have coffee. I can contact Robin to see if he is available. re: I/O capacity: I was commenting on how Google, etc. can provide such inexpensive cloud storage, not access speed.
nickpsecurity 11y ago
Funny he mentioned System/38, which largely eliminated users' storage worries, then says that company's rep never heard of it. Funny cuz they still sell it: IBM i. I posted a link [1] to original system on his site. It got so many things right in a business machine it was hard to believe. That old design still has better security and reliability than Windows or Linux boxes. Most AS/400 deployments I run into just don't... have... downtime... They also pretty much manage themselves. I'd like to see FOSS do designs like that [minus the interface & API]. The labor advantage might lead to something pretty awesome. Meanwhile, they keep building complexity on top of UNIX architecture with a few stragglers on better OS architectures. Not much hope there. Fortunately, CHERI (capability) and SAFE (tags) are each building on mechanisms System/38 used for software reliability and security. An academic team or company might build something great on top of it. Still holding out some hope! [1] http://homes.cs.washington.edu/~levy/capabook/Chapter8.pdf http://homes.cs.washington.edu/~levy/capabook/Chapter8.pdf
- vezzy-fnord 11y ago
  Also worth noting is that IBM i had one of (if not probably the) first implementation of containers, in the form of LPARs.
  - nickpsecurity 11y ago
    True. Add that many "inventions" in virtualization field were in VM/370 in... the 70's. Including running itself on itself which has been celebrated in several news articles for different prototypes over the past few years. Except that was a production system. ;)
    - cbd1984 11y ago
      Add that IBM tried to kill VM multiple times, and wanted everyone using System/360 to standardize around OS/360. "The Mythical Man-Month" is the story of OS/360. Hence VM still exists.
      - nickpsecurity 11y ago
        Lol @ last line. Well-put.
StillBored 11y ago
I was under the impression that the storage model in the system/38 was part of what was carried forward into the as400. Not the capability model, as much as the lack of "files" because everything is just persistent objects/segments. Is that not the case? But I'm not really sure how this by itself solves the problems of modern SSD's. Problems caused by two things, the differing segment/page sizes between the upper layers (OS/filesystem/database/etc) and the need to rewrite static portions of the SSD in order to wear level the entire address space. (and to a lesser extent the need to update small portions in the middle of existing pieces of data). Its the second part that seems to be forgotten by lots of people suggesting alternatives to the current storage stack. Whats really necessary is a higher level communications protocol between the flash and the filesystem. One that provides the metadata to the ssd about the sizes of individual "objects" so that they may be keep together as atomic units. That way the flash layer knows if a write is to a larger object, or a new object itself that can be managed separately.
dang 11y ago
The article says that log-structured apps and filesystems conflict with underlying SSDs, because those are doing similar things and the two don't play well together. But then it says that "products that already incorporate log structured I/O will have a definite advantage" in the future. How can they have an advantage if they're conflicting with the storage layer? Is this a contradiction, or am I missing something?
- tlb 11y ago
  The paper [0] has some suggestions: an API (already existing and somewhat supported, called TRIM) to tell the disk that certain sectors are no longer in use, and atomic writes. I guess I buy the claim that you're better off starting with a log-structured file system that will use new APIs to allocate and free chunks on the SSD, than to use a conventional file system that the SSD tries to optimize [0] https://www.usenix.org/system/files/conference/inflow14/inflow14-yang.pdf https://www.usenix.org/system/files/conference/inflow14/infl...
williesleg 11y ago
I think the storage company Nimble may have figured this out.
- stephengillie 11y ago
  What makes you think they have? Do you have a blog post or statistical data that supports this notion? Please, share!
  - droopyEyelids 11y ago
    Not yet but the rep is taking us to the big game— a fully catered private box. I'm sure I'll hear about it there.
    - robotresearcher 11y ago
      So their prices are for storage and the big game. Big sales seems so wasteful.
    - JupiterMoon 11y ago
      If they had truly solved this problem they wouldn't need to take you to "the big game" to do sales - you'd be taking them to "the big game" to be customers...
    - stephengillie 11y ago
      What is this "big game"? Are they taking you to a sporting event? Sporting events seem like an inefficient place to transfer technical details. They're rather distracting.
- pkaye 11y ago
  I thought they just run a rack full of managed SSDs?
- KaiserPro 11y ago
  Nope. Nimble use a log storage format similar to that of Netapp and ZFS. Their performance special source is using the minimum amount of flash to provide a performance boost over standard HDDs. They also have very nice analytics. They are good for VMware/virtualisation. For general file storage, unless they have changed recently, is pretty mediocre.
narrator 11y ago
Wait till memresistors get here. That will basically eliminate the distinction between ram and disk. https://en.wikipedia.org/wiki/Memristor#1808 https://en.wikipedia.org/wiki/Memristor#1808 "They can potentially be fashioned into non-volatile solid-state memory, which would allow greater data density than hard drives with access times similar to DRAM, replacing both components.[14] HP prototyped a crossbar latch memory that can fit 100 gigabits in a square centimeter,[8] and proposed a scalable 3D design (consisting of up to 1000 layers or 1 petabit per cm3).[66] "
- wmf 11y ago
  HP indefinitely postponed (canceled) memristors recently.
- rmurri 11y ago
  https://news.ycombinator.com/item?id=9810324 https://news.ycombinator.com/item?id=9810324 They seem to be here.
maccam94 11y ago
Yup, storage is going to change a lot. RAID is nearly dead. That means a whole lot of failure detection and correction needs to be reimplemented elsewhere in the stack. We'll also need to build storage layers that can deal with the crazy performance profile of SMR hard drives (at least, until SSD's are cheaper).

70 points by jcreedon 11y ago

Good to see Robin Harris' article on the front page of HN (we live in the same small town and a mutual friend introduced us). A little off topic from the article: I think the biggest tipping point is the decreasing cost of cloud storage because large companies like Google do so much of their time critical processing with data in RAM (spread over many servers) and these servers can re-purpose a lot of their disk I/O capability to service low cost cloud storage.

dikaiosune 11y ago
Can't believe I've seen two people from there on the same day on HN! Flabbergasted. Grew up there, currently living just up the rim. Re: your comment, I'm curious to see how the I/O capacity of those servers could be distributed for cloud storage. I would imagine that even with 10Gbps if those machines are really cranking, they could have a difficult time provisioning network I/O for a storage service.
- mark_l_watson 11y ago
  Email me when you will be in Sedona and we can have coffee. I can contact Robin to see if he is available. re: I/O capacity: I was commenting on how Google, etc. can provide such inexpensive cloud storage, not access speed.

nickpsecurity 11y ago

Funny he mentioned System/38, which largely eliminated users' storage worries, then says that company's rep never heard of it. Funny cuz they still sell it: IBM i. I posted a link [1] to original system on his site. It got so many things right in a business machine it was hard to believe. That old design still has better security and reliability than Windows or Linux boxes. Most AS/400 deployments I run into just don't... have... downtime... They also pretty much manage themselves. I'd like to see FOSS do designs like that [minus the interface & API]. The labor advantage might lead to something pretty awesome. Meanwhile, they keep building complexity on top of UNIX architecture with a few stragglers on better OS architectures. Not much hope there. Fortunately, CHERI (capability) and SAFE (tags) are each building on mechanisms System/38 used for software reliability and security. An academic team or company might build something great on top of it. Still holding out some hope! [1] http://homes.cs.washington.edu/~levy/capabook/Chapter8.pdf http://homes.cs.washington.edu/~levy/capabook/Chapter8.pdf

vezzy-fnord 11y ago
Also worth noting is that IBM i had one of (if not probably the) first implementation of containers, in the form of LPARs.
- nickpsecurity 11y ago
  True. Add that many "inventions" in virtualization field were in VM/370 in... the 70's. Including running itself on itself which has been celebrated in several news articles for different prototypes over the past few years. Except that was a production system. ;)
  - cbd1984 11y ago
    Add that IBM tried to kill VM multiple times, and wanted everyone using System/360 to standardize around OS/360. "The Mythical Man-Month" is the story of OS/360. Hence VM still exists.
    - nickpsecurity 11y ago
      Lol @ last line. Well-put.

StillBored 11y ago

I was under the impression that the storage model in the system/38 was part of what was carried forward into the as400. Not the capability model, as much as the lack of "files" because everything is just persistent objects/segments. Is that not the case? But I'm not really sure how this by itself solves the problems of modern SSD's. Problems caused by two things, the differing segment/page sizes between the upper layers (OS/filesystem/database/etc) and the need to rewrite static portions of the SSD in order to wear level the entire address space. (and to a lesser extent the need to update small portions in the middle of existing pieces of data). Its the second part that seems to be forgotten by lots of people suggesting alternatives to the current storage stack. Whats really necessary is a higher level communications protocol between the flash and the filesystem. One that provides the metadata to the ssd about the sizes of individual "objects" so that they may be keep together as atomic units. That way the flash layer knows if a write is to a larger object, or a new object itself that can be managed separately.

dang 11y ago

The article says that log-structured apps and filesystems conflict with underlying SSDs, because those are doing similar things and the two don't play well together. But then it says that "products that already incorporate log structured I/O will have a definite advantage" in the future. How can they have an advantage if they're conflicting with the storage layer? Is this a contradiction, or am I missing something?

tlb 11y ago
The paper [0] has some suggestions: an API (already existing and somewhat supported, called TRIM) to tell the disk that certain sectors are no longer in use, and atomic writes. I guess I buy the claim that you're better off starting with a log-structured file system that will use new APIs to allocate and free chunks on the SSD, than to use a conventional file system that the SSD tries to optimize [0] https://www.usenix.org/system/files/conference/inflow14/inflow14-yang.pdf https://www.usenix.org/system/files/conference/inflow14/infl...

williesleg 11y ago

I think the storage company Nimble may have figured this out.

stephengillie 11y ago
What makes you think they have? Do you have a blog post or statistical data that supports this notion? Please, share!
- droopyEyelids 11y ago
  Not yet but the rep is taking us to the big game— a fully catered private box. I'm sure I'll hear about it there.
  - robotresearcher 11y ago
    So their prices are for storage and the big game. Big sales seems so wasteful.
  - JupiterMoon 11y ago
    If they had truly solved this problem they wouldn't need to take you to "the big game" to do sales - you'd be taking them to "the big game" to be customers...
  - stephengillie 11y ago
    What is this "big game"? Are they taking you to a sporting event? Sporting events seem like an inefficient place to transfer technical details. They're rather distracting.
pkaye 11y ago
I thought they just run a rack full of managed SSDs?
KaiserPro 11y ago
Nope. Nimble use a log storage format similar to that of Netapp and ZFS. Their performance special source is using the minimum amount of flash to provide a performance boost over standard HDDs. They also have very nice analytics. They are good for VMware/virtualisation. For general file storage, unless they have changed recently, is pretty mediocre.

narrator 11y ago

Wait till memresistors get here. That will basically eliminate the distinction between ram and disk. https://en.wikipedia.org/wiki/Memristor#1808 https://en.wikipedia.org/wiki/Memristor#1808 "They can potentially be fashioned into non-volatile solid-state memory, which would allow greater data density than hard drives with access times similar to DRAM, replacing both components.[14] HP prototyped a crossbar latch memory that can fit 100 gigabits in a square centimeter,[8] and proposed a scalable 3D design (consisting of up to 1000 layers or 1 petabit per cm3).[66] "

wmf 11y ago
HP indefinitely postponed (canceled) memristors recently.
rmurri 11y ago
https://news.ycombinator.com/item?id=9810324 https://news.ycombinator.com/item?id=9810324 They seem to be here.

maccam94 11y ago

Yup, storage is going to change a lot. RAID is nearly dead. That means a whole lot of failure detection and correction needs to be reimplemented elsewhere in the stack. We'll also need to build storage layers that can deal with the crazy performance profile of SMR hard drives (at least, until SSD's are cheaper).