5 ms·

Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location t

by nrzuk 12y ago

Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location they are to be installed? Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the random reads. I would have assumed that having even 100 people streaming from a single box would be an interesting challenge.

weaksauce 12y ago
well, if it means that they need to install n boxes in an isp to avoid peering at a high cost then it might be worth it. > Sometimes it'll even have multiple copies of the same show, if something is in crazy high demand. that statement from the article lends one to believe that the raid arrays are in parallel. These boxes are meant to reduce the number of packets going all the way across the internet not necessarily remove the need. Also, in addition to lots of storage, these boxes probably have loads of ram that they can buffer popular streams into and then serve it.
twoodfin 12y ago
These things can pump out 12Gbit during peak viewing, which is enough for about 4,000 avg. 3 Mbit streams. If they can cram in 256GB of DRAM, that's enough room for a 64MB buffer per stream, or about 170 seconds worth of streaming. Now you only need to be able to fill those buffers at a rate of about 24/second. I'm assuming whatever file system is on the disks uses a massive block size, so the number of seeks you'd have to perform to pull 64MB off is probably pretty low. Eight? Sixteen? Even if it's the latter, that's only 384 seeks/second, which you could very plausibly do striped over only a half dozen disks, and the device presumably has many more.
- runlevel1 12y ago
  > I'm assuming whatever file system is on the disks... Netflix uses UFS+J on FreeBSD 10. Here are some notes from their talk at NYCBSDCon 2014: - 400,000 stream files per appliance. - 5,000 - 25,000 client streams per appliance. - 300 - 500 streams coming off each disk all the time. - Attempt to buffer 1MB ahead, but caching is futile. - Result is completely random disk workload. - System becomes limited by disk latency and CPU load. Video here: https://www.youtube.com/watch?v=FL5U4wr86L4 https://www.youtube.com/watch?v=FL5U4wr86L4
- nrzuk 12y ago
  Makes a little more sense now, thanks :)
wmf 12y ago
Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the random reads. I think this can be fixed by prefetching harder (like 1-2 MB). 100 Netflix streams would be less than 100 MB/s which can be satisfied with a single SATA drive. And some of the Netflix boxes have dozens of SSDs that each do 500 MB/s.
pfg 12y ago
I'm no expert either, but if my napkin math is right, fully utilized in a best-case scenario (no network overhead, etc.), those disks would peak at around 30-40 MByte/s (36 disks, 10 Gbit NIC). I'm not sure if the I/O is really all that random. Theoretically, it's all sequential, but might effectively hit the disk like random I/O because of concurrent streaming sessions, varying network speed and so on. My guess is you can get pretty close to sequential I/O with some simple means like using large block sizes in your RAID, well-tuned readahead and Native Command Queuing.
- justincormack 12y ago
  The new versions are all SSD.
  - deleted 12y ago
    [deleted]
fragmede 12y ago
There's a bit of magic to it, turning the OS to work well in conjunction with the hardware you have, but it's possible. * Use 15k RPM SAS 12Gb/s drives instead of cheap consumer drives, * Use 50 in a chassis of them instead of 12. * Use RAID 0 instead of RAID 5 or 6 - I'll be the quoted storage space is raw, not post-raid. * Have multiple copies of a show on disk, as the article states happens. * Optimize for a read-heavy workflow during peak hours, (eg, mount noatime). The devil is in the details, but what I listed above is probably a good starting point. The article states a data rate of 3GB/hour, which is only .83 megabytes/second, * 100 streams is only 83 megabytes/second, which is easy for the above configuration. Hell, I'll bet the above configuration could do 10,000 streams if the data rate is 3GB/hour with no issues given Netflix's peak read-only workload.
- thrownaway2424 12y ago
  A 15000RPM SAS drive would be an _extremely_ poor choice for this type of workload. Power hungry, hot, extremely sensitive to vibration, very low data density, and, of course, ridiculously expensive.
  - fragmede 12y ago
    What? While I can't argue with them being ridiculously expensive (because they very much are) why would them being power hungry, hot, and extremely sensitive to vibration even matter if they are going to be rack-mounted into an ISP's data center, with redundant power-supplies and a data-center grade cooling system to match? The only possible issue of the ones you raised is low data density, but all engineering is a trade-off, 15k rpm drives get you better seek times, which would generally lead to the ability to support more viewers in a single box - not working for Netflix, I don't know how many users they want to support per-box. It's totally possible that between the number of streams they want to support, and the total storage in the box that they've gone with SATA drives, but to pretend that 15k RPM SAS drives are "an _extremely_ poor choice" for an enterprise-grade storage system would be to ignore the fact that Netflix is making an enterprise-grade storage appliance - and that on the top-end, those appliances commonly use SAS drives.

by nrzuk 12y ago

weaksauce 12y ago

well, if it means that they need to install n boxes in an isp to avoid peering at a high cost then it might be worth it. > Sometimes it'll even have multiple copies of the same show, if something is in crazy high demand. that statement from the article lends one to believe that the raid arrays are in parallel. These boxes are meant to reduce the number of packets going all the way across the internet not necessarily remove the need. Also, in addition to lots of storage, these boxes probably have loads of ram that they can buffer popular streams into and then serve it.

twoodfin 12y ago

These things can pump out 12Gbit during peak viewing, which is enough for about 4,000 avg. 3 Mbit streams. If they can cram in 256GB of DRAM, that's enough room for a 64MB buffer per stream, or about 170 seconds worth of streaming. Now you only need to be able to fill those buffers at a rate of about 24/second. I'm assuming whatever file system is on the disks uses a massive block size, so the number of seeks you'd have to perform to pull 64MB off is probably pretty low. Eight? Sixteen? Even if it's the latter, that's only 384 seeks/second, which you could very plausibly do striped over only a half dozen disks, and the device presumably has many more.

runlevel1 12y ago
> I'm assuming whatever file system is on the disks... Netflix uses UFS+J on FreeBSD 10. Here are some notes from their talk at NYCBSDCon 2014: - 400,000 stream files per appliance. - 5,000 - 25,000 client streams per appliance. - 300 - 500 streams coming off each disk all the time. - Attempt to buffer 1MB ahead, but caching is futile. - Result is completely random disk workload. - System becomes limited by disk latency and CPU load. Video here: https://www.youtube.com/watch?v=FL5U4wr86L4 https://www.youtube.com/watch?v=FL5U4wr86L4
nrzuk 12y ago
Makes a little more sense now, thanks :)

wmf 12y ago

Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the random reads. I think this can be fixed by prefetching harder (like 1-2 MB). 100 Netflix streams would be less than 100 MB/s which can be satisfied with a single SATA drive. And some of the Netflix boxes have dozens of SSDs that each do 500 MB/s.

pfg 12y ago

I'm no expert either, but if my napkin math is right, fully utilized in a best-case scenario (no network overhead, etc.), those disks would peak at around 30-40 MByte/s (36 disks, 10 Gbit NIC). I'm not sure if the I/O is really all that random. Theoretically, it's all sequential, but might effectively hit the disk like random I/O because of concurrent streaming sessions, varying network speed and so on. My guess is you can get pretty close to sequential I/O with some simple means like using large block sizes in your RAID, well-tuned readahead and Native Command Queuing.

justincormack 12y ago
The new versions are all SSD.
- deleted 12y ago
  [deleted]

fragmede 12y ago

There's a bit of magic to it, turning the OS to work well in conjunction with the hardware you have, but it's possible. * Use 15k RPM SAS 12Gb/s drives instead of cheap consumer drives, * Use 50 in a chassis of them instead of 12. * Use RAID 0 instead of RAID 5 or 6 - I'll be the quoted storage space is raw, not post-raid. * Have multiple copies of a show on disk, as the article states happens. * Optimize for a read-heavy workflow during peak hours, (eg, mount noatime). The devil is in the details, but what I listed above is probably a good starting point. The article states a data rate of 3GB/hour, which is only .83 megabytes/second, * 100 streams is only 83 megabytes/second, which is easy for the above configuration. Hell, I'll bet the above configuration could do 10,000 streams if the data rate is 3GB/hour with no issues given Netflix's peak read-only workload.

thrownaway2424 12y ago
A 15000RPM SAS drive would be an _extremely_ poor choice for this type of workload. Power hungry, hot, extremely sensitive to vibration, very low data density, and, of course, ridiculously expensive.
- fragmede 12y ago
  What? While I can't argue with them being ridiculously expensive (because they very much are) why would them being power hungry, hot, and extremely sensitive to vibration even matter if they are going to be rack-mounted into an ISP's data center, with redundant power-supplies and a data-center grade cooling system to match? The only possible issue of the ones you raised is low data density, but all engineering is a trade-off, 15k rpm drives get you better seek times, which would generally lead to the ability to support more viewers in a single box - not working for Netflix, I don't know how many users they want to support per-box. It's totally possible that between the number of streams they want to support, and the total storage in the box that they've gone with SATA drives, but to pretend that 15k RPM SAS drives are "an _extremely_ poor choice" for an enterprise-grade storage system would be to ignore the fact that Netflix is making an enterprise-grade storage appliance - and that on the top-end, those appliances commonly use SAS drives.