8 ms·
IO Devices and Latency
- cmurf 2y agoPlenty of text but also many cool animations. I'm a sucker for visual aids. It's a good balance.
- bddicken 2y agoAuthor of the blog here. I had a great time writing this. By far the most complex article I've ever put together, with literally thousands of lines of js to build out these interactive visuals. I hope everyone enjoys.
- inetknght 2y agoI don't see a single visual. I don't use the web with javascript. Why not embed static images instead or in addition?
- bddicken 2y agoThe visuals add a lot to this article. A big theme throughout is latency, and the visual help the reader see why tape is slower than an hdd, which is slower than an ssd, etc. Also, its just plain fun! I'm curious, what do you do on the internet without js these days?
- inetknght 2y ago> I'm curious, what do you do on the internet without js these days? Browse the web, send/receive email, read stories, play games, the usual. I primarily use native apps and selectively choose what sites are permitted to use javascript, instead of letting websites visited on a whim run javascript willy nilly.
- bddicken 2y agoI respect it. In my very biased opinion, it's worth enabling for this article.
- vel0city 2y agoThey're not just static images or animations, they're interactive widgets.
- zalebz 2y agoThe level of your effort really shows through. If you had to ballpark guess, how much time do you think you put in? and I realize keyboard time vs kicking around in your head time are quite different
- bddicken 2y agoThank you! I started this back in October, but of course have worked on plenty of other things in the meantime. But this was easily 200+ hours of work spread out over that time. If this helps as context, the git diff for merging this into our website was: +5,820 −1
- dormando 2y agoHalf on topic: what libs/etc did you use for the animations? Not immediately obvious from the source page. (it's a topic I'm deeply familiar with so I don't have a comment on the content, it looks great on a skim!) - but I've been sketching animations for my own blog and not liked the last few libs I tried. Thanks!
- bddicken 2y agoI heavily, heavily abused d3.js to build these.
- petedoyle 2y agoSmall FYI that I couldn't see them in Chrome 133.0.6943.142 on MacOS. Firefox works.
- homebrewer 2y agoIt's the complete opposite for me — there are no animations in Firefox even with uBlock Origin disabled, but Brave shows them fine. The browser console spams this link: https://react.dev/errors/418?invariant=418 https://react.dev/errors/418?invariant=418 edit: looks like it's caused by a userstyles extension injecting a dark theme into the page; React doesn't like it and the page silently breaks.
- bddicken 2y agoOhhh interesting! Obviously not ideal, but I guess just an extension issue?
- deleted 2y ago[deleted]
- bddicken 2y agoInteresting. Running any chrome extensions that might be messing with things? Alternatively, if you can share any errors you're getting in the console lmk.
- jasonthorsness 2y agoThe visuals are awesome; the bouncing-box is probably the best illustration of relative latency I've seen. Your "1 in a million" comment on durability is certainly too pessimistic once you consider the briefness of the downtime before a new server comes in and re-replicates everything, right? I would think if your recovery is 10 minutes for example, even if each of three servers is guaranteed to fail once in the month, I think it's already like 1 in two million? and if it's a 1% chance of failure in the month failure of all three overlapping becomes extremely unlikely. Thought I would note this because one-in-a-million is not great if you have a million customers ;)
- bddicken 2y ago> Your "1 in a million" comment on durability is certainly too pessimistic once you consider the briefness of the downtime before a new server comes in and re-replicates everything, right? Absolutely. Our actual durability is far, far, far higher than this. We believe that nobody should ever worry about losing their data, and thats the peace of mind we provide.
- alfons_foobar 2y ago> Instead of relying on a single server to store all data, we can replicate it onto several computers. One common way of doing this is to have one server act as the primary, which will receive all write requests. Then 2 or more additional servers get all the data replicated to them. With the data in three places, the likelihood of losing data becomes very small. Is my understanding correct, that this means you propagate writes asynchronously from the primary to the secondary servers (without waiting for an "ACK" from them for writes)?
- bddicken 2y agoFor PlanetScale Metal, we use semi-sync replication. The primary need to get an ack from at least one replica before committing.
- 2y ago
- alexellisuk 2y agoHi, what actually are _metal_ instances that are being used when you're on EC2 that have local NVME attached? Last time I looked, apart from the smallest/slowest Graviton, you have to spend circa 2.3k USD/mo to get a bare-metal instance from AWS - https://blog.alexellis.io/how-to-run-firecracker-without-kvm-on-regular-cloud-vms/ https://blog.alexellis.io/how-to-run-firecracker-without-kvm...
- lizztheblizz 2y agoHi there, PS employee here. In AWS, the instance types backing our Metal class are currently in the following families: r6id, i4i, i3en and i7ie. We're deploying across multiple clouds, and our "Metal" product designation has no direct link to Amazon's bare-metal offerings.
- logsr 2y agoAmazing presentation. It really helps to understand the concepts. The only add is that it understates the impact of SSD parallelism. 8 Channel controllers are typical for high end devices and 4K random IOPS continue to scale with queue depth, but for an introduction the example is probably complex enough. It is great to see PlanetScale moving in this direction and sharing the knowledge.
- bddicken 2y agoThank you for the info! Do you have any good references on this for those who want to learn more?
- logsr 2y agoJust going off specs sheets from manufacturers and reviews (mostly consumer products, so enterprise should be the same or better). There are only a few major NAND manufacturers: Samsung, Micron, Kioxia / Western Digital, SK Hynix, and their branded products are usually the best. There are also several 3rd party controller developers: Phison, Marvell, Silicon Motion, which I think are the largest, and then a bunch of others. I hadn't looked at this in a couple years, so 16 channel controllers are more common now, but only on high end enterprise devices. 4KB random read/write specs are definitely not trustable without testing. They are usually at max queue depth and, at least for consumer devices, based on writing to a buffer in SLC mode, so they will be a lot lower once the buffer is exhausted. Enterprise specs might be more realistic but there isnt as much public testing data available.
- AlphaWeaver 2y agoWere you at all inspired by the work of Bartosz Ciechanowski? My first thought was that you all might have hired him to do the visuals for this post :)
- bddicken 2y agoBartosz Ciechanowski is incredible at this type of stuff. Sam Rose has some great interactive blogs too. Both have had big hits here on HN.
- tombert 2y agoThe visualizations are excellent, very fun to look at and play with, and they go along with the article extremely well. You should be proud of this, I really enjoyed it.
- bddicken 2y agoThank you!
- b0rbb 2y agoThe animations are fantastic and awesome job with the interactivity. I find myself having to explain latency to folks often in my work and being able to see the extreme difference in latencies for something like a HDD vs SSD makes it much easier to understand for some people. Edit: And for real, fantastic work, this is awesome.
- bddicken 2y agoThank you! The visuals definitely add something special to this post specifically since time is a big element in explaining latencies.
- walterbell 2y agoThanks for leadership on investing marketing budget into quality technical edutainment (and brand building!) for visually oriented humans.
- hakaneskici 2y agoGreat work! Thank you for making this. This is beautiful and brilliant, and also is a great visual tool to explain how some of the fundamental algorithms and data structures originate from the physical characteristics of storage mediums. I wonder if anyone remembers the old days where you programmed your own custom defrag util to place your boot libs and frequently used apps to the outer tracks of the hard drive, so they are loaded faster due to the higher linear velocity of the outermost track :)
- layer8 2y agoI don’t see any animations on Safari. Also, I’d much prefer a variable-width font, monospace prose is hard to read. While I can use Reader Mode, that removes the text coloring, and would likely also hide the visuals (if they were visible in the first place).
- anymouse123456 2y agoI love this kind of datavis. We are generally bad at internalizing comparisons at these scales. The visualizations make a huge difference in building more detailed intuitions. Really nice work, thank you!
- bddicken 2y agoYeah I think the visuals really add to this one, especially given the time element of explaining latencies.
- RKFADU_UOFCCLEL 2y ago[dead]
- hodgesrm 2y agoI was delighted to see your models of tape operations as I used it a lot in the COBOL days. For reasons discussed in your article we would arrange tape processing as much as possible in sequential scans, something at which COBOL was quite excellent. One of the common performance problems was when there was a mismatch between a slower COBOL processing speed that could not keep up with the flow of blocks coming off the drive head. In this case you would see the drive start to overshoot as it read more blocks than the COBOL program could handle. The drive would begin a painful jump forward/spool backward motion which made the performance issue quite visible. You would then eyeball the code to understand way the program was not keeping up, correct, and resubmit until the motion disappeared.
- tealpod 1y agoYour style of explaination and animation are exceptional.
- tonyhb 2y agoThis is really cool, and PlanetScale Metal looks really solid, too. Always a huge sucker for seeing latency huge latency drops on releases: https://planetscale.com/blog/upgrading-query-insights-to-metal https://planetscale.com/blog/upgrading-query-insights-to-met....
- vessenes 2y agoGreat nerdbaiting ad. I read all the way to the bottom of it, and bookmarked it to send to my kids if I feel they are not understanding storage architectures properly. :)
- bddicken 2y agoThe nerdbaiting will now provide generational benefit!
- bob1029 2y agoI've been advocating for SQLite+NVMe for a while now. For me it is a new kind of pattern you can apply to get much further into trouble than usual. In some cases, you might actually make it out to the other side without needing to scale horizontally. Latency is king in all performance matters. Especially in those where items must be processed serially. Running SQLite on NVMe provides a latency advantage that no other provider can offer. I don't think running in memory is even a substantial uplift over NVMe persistence for most real world use cases.
- jstimpfle 2y agoI still measure 1-2ms of latency with an NVMe disk on my Desktop computer, doing fsync() on a file on a ext4 filesystem. Update: about 800us on a more modern system.
- the8472 2y agoI assume fsyncing a whole file does more work than just ensuring that specific blocks made it to the WAL which it can achieve with direct IO or maybe sync_file_range.
- rbranson 2y agoNot so sure that's true. This is single-threaded direct I/O doing a fio randwrite workload on a WD 850X Gen4 SSD: write: IOPS=18.8k, BW=73.5MiB/s (77.1MB/s)(4412MiB/60001msec); 0 zone resets slat (usec): min=2, max=335, avg= 3.42, stdev= 1.65 clat (nsec): min=932, max=24868k, avg=49188.32, stdev=65291.21 lat (usec): min=29, max=24880, avg=52.67, stdev=65.73 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 35], | 30.00th=[ 37], 40.00th=[ 38], 50.00th=[ 40], 60.00th=[ 43], | 70.00th=[ 53], 80.00th=[ 60], 90.00th=[ 70], 95.00th=[ 84], | 99.00th=[ 137], 99.50th=[ 174], 99.90th=[ 404], 99.95th=[ 652], | 99.99th=[ 2311]
- jstimpfle 2y agoI checked again with O_DIRECT and now I stand corrected. I didn't know that O_DIRECT could make such a huge difference. Thanks!
- jhgg 2y agoMetal looks super cool, however at my last job when we tried using instance local SSD's on GCP, there were serious reliability issues (e.g. blocks on the device losing data). Has this situation changed? What machine types are you using? Our workaround was this: https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme-low-latency https://discord.com/blog/how-discord-supercharges-network-di...
- rcrowley 2y agoNeat workaround! We only started working with GCP Local SSDs in 2024 and can report we haven't experienced read or write failures due to bad sectors in any of our testing. That said, we're running a redundant system in which MySQL semi-sync replication ensures every write is durable to two machines, each in a different availability zone, before that write's acknowledged to the client. And our Kubernetes operator plus Vitess' vtorc process are working together to aggressively detect and replace failed or even suspicious replicas. In GCP we find the best results on n2d-highmem machines. In AWS, though, we run on pretty much all the latest-generation types with instance storage.
- bloopernova 2y agoFantastic article, well explained and beautiful diagrams. Thank you bddicken for writing this!
- bddicken 2y agoYou are welcome!
- TechDebtDevin 2y agoProbably the best diagrams I've ever seen in a blog post.
- magicmicah85 2y agoCan I just say that I love how informative this was that I completely forgot it was to promote a product? Excellent visuals and interactivity.
- gozzoo 2y agoCan someeone share their expirience in creating such diagrams. What libraries and tools can be useful for such interactive diagrams?
- Joel_Mckay 2y agoDo you mean something for data visualization, or tricks condensing large data sets with cursors? https://d3js.org/ https://d3js.org/ Best of luck =3
- bddicken 2y agoFor this particular one I used d3.js, but honestly this isn't really the type of thing it's designed for. I've also used GSAP for this type of thing on this article I wrote about database sharding. https://planetscale.com/blog/database-sharding https://planetscale.com/blog/database-sharding
- Nezteb 2y agoYour diagrams are fantastic; I'd enjoy a short blog post or snippet just on how you used D3 for this! As someone who has also use GSAP a decent amount, these days I usually have a better experience with SVG.js [1]. [1] https://github.com/svgdotjs/svg.js https://github.com/svgdotjs/svg.js
- aftbit 2y agoHrm "unlimited IOPS"? I suppose contrasted against the abysmal IOPS available to Cloud block devs. A good modern NVMe enterprise drive is specced for (order of magnitude) 10^6 to 10^7 IOPS. If you can saturate that from database code, then you've got some interesting problems, but it's definitely not unlimited.
- bddicken 2y agoTechnically any drive has a finite IOPS capacity. We have found that no matter how hard we tried, we could not get MySQL to exhaust the max IOPS of the underlying hardware. You hit CPU limits long before hitting IOPS limits. Thus "infinite IOPS."
- ucarion 2y agoReally, really great article. The visualization of random writes is very nicely done. On: > Another issue with network-attached storage in the cloud comes in the form of limiting IOPS. Many cloud providers that use this model, including AWS and Google Cloud, limit the amount of IO operations you can send over the wire. [...] > If instead you have your storage attached directly to your compute instance, there are no artificial limits placed on IO operations. You can read and write as fast as the hardware will allow for. I feel like this might be a dumb series of questions, but: 1. The ratelimit on "IOPS" is precisely a ratelimit on a particular kind of network traffic, right? Namely traffic to/from an EBS volume? "IOPS" really means "EBS volume network traffic"? 2. Does this save me money? And if yes, is from some weird AWS arbitrage? Or is it more because of an efficiency win from doing less EBS networking? I see pretty clearly putting storage and compute on the same machine strictly a latency win, because you structurally have one less hop every time. But is it also a throughput-per-dollar win too?
- the8472 2y agoFor network-attached storage IOPS limits packets per second, not bandwidth, since IO operations can happen at different sizes (e.g. 4K vs. 16K blocks).
- rbranson 2y agoMore specific details for EC2 instances can be seen in the docs here: https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html#gp_storage-ebs https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html...
- rbranson 2y ago> 1. The ratelimit on "IOPS" is precisely a ratelimit on a particular kind of network traffic, right? Namely traffic to/from an EBS volume? "IOPS" really means "EBS volume network traffic"? The EBS volume itself has a provisioned capacity of IOPS and throughput, and the EC2 instance it's attached to will have its own limits as well across all the EBS volumes attached to it. I would characterize it more like a different model. An EBS volume isn't just just a slice of a physical PCB attached to a PCIe bus, it's a share in a large distributed system a large number of physical drives with its own dedicated network capacity to/from compute, like a SAN. > 2. Does this save me money? And if yes, is from some weird AWS arbitrage? Or is it more because of an efficiency win from doing less EBS networking? It might. It's a set of trade-offs.
- gz09 2y agoNice blog. There is also a problem that generally cloud storage is "just unusually slow" (this has been noted by others before, but here is a nice summary of the problem http://databasearchitects.blogspot.com/2024/02/ssds-have-become-ridiculously-fast.html http://databasearchitects.blogspot.com/2024/02/ssds-have-bec...) Having recently added support for storing our incremental indexes in https://github.com/feldera/feldera https://github.com/feldera/feldera on S3/object storage (we had NVMe for longer due to obvious performance advantages mentioned in the previous article), we'd be happy for someone to disrupt this space with a better offering ;).
- bddicken 2y agoThat database architects blog is a great read.
- __turbobrew__ 2y agoI think something about distributed storage which is not appreciated in this article: 1. Some systems do not support replication out of the box. Sure your cassandra cluster and mysql can do master slave replication, but lots of systems cannot. 2. Your life becomes much harder with NVME storage in cloud as you need to respect maintenance intervals and cloud initiated drains. If you do not hook into those system and drain your data to a different node, the data goes poof. Separating storage from compute allows the cloud operator to drain and move around compute as needed and since the data is independent from the compute — and the cloud operator manages that data system and draining for that system as well — the operator can manage workload placements without the customer needing to be involved.
- maayank 2y agowhat do you mean by drains?
- rcrowley 2y agoAWS, for one example, provide a feed of upcoming "events" in EC2 in which certain instances will need to be rebooted or terminated entirely due to whatever maintenance they're doing on the physical infrastructure. If you miss a termination event you miss your chance to copy that data elsewhere. Of course, if you're _always_ copying the data elsewhere, you can rest easy.
- rcrowley 2y agoGood points. PlanetScale's durability and reliability are built on replication - MySQL replication - and all the operational software we've written to maintain replication in the face of servers coming and going, network partitions, and all the rest of the weather one faces in the cloud. Replicated network-attached storage that presents a "local" filesystem API is a powerful way to create durability in a system that doesn't build it in like we have.
- __turbobrew__ 2y agoAgreed, if you are a mature enough and well funded organization you probably should be using NVME and then run distributed systems on top of the NVMEs to manage replication yourself.
- r3tr0 2y agoWe are working on a platform that lets you measure this stuff with pretty high precision in real time. You can check out our sandbox here: https://yeet.cx/play https://yeet.cx/play
- robotguy 2y agoSeeing the disk IO animation reminded me of Melvin Kaye[0]: Mel never wrote time-delay loops, either, even when the balky Flexowriter required a delay between output characters to work right. He just located instructions on the drum so each successive one was just past the read head when it was needed; the drum had to execute another complete revolution to find the next instruction. [0] https://pages.cs.wisc.edu/~markhill/cs354/Fall2008/notes/The_Story_of_Mel.html https://pages.cs.wisc.edu/~markhill/cs354/Fall2008/notes/The...
- Thoreandan 2y agoI was reminded of Mel as well! If you haven't seen it, Usagi Electric on YouTube has gotten a drum-memory system from the 1950s nearly fully-functional again.
- samwho 2y agoGosh, this is beautiful. Fantastic work, Ben. <3
- rsanheim 2y agoThat great infographic at the top illustrates one big reason why 'dev instances in the cloud' is a bad idea.
- cynicalsecurity 2y agoThat was a cool advertisement, I must give them that.
- jgalt212 2y agoDisk latency, and one's aversion to it, is IMHO the only way Hetzner costs can run up on you. You want to keep the database on local disk, and not their very slow attached Volumes (Hetzner EBS). In short, you can have relatively light work-loads that will be on sort of expensive VMs because you need 500GB, or more, of local disk. 1TB local disk is the biggest VM they offer in the US. 300 EUR a month.
- pjdesno 2y agoI love the visuals, and if it's ok with you will probably link them to my class material on block devices in a week or so. One small nit: > A typical random read can be performed in 1-3 milliseconds. Um, no. A 7200 RPM platter completes a rotation in 8.33 milliseconds, so rotational delay for a random read is uniformly distributed between 0 and 8.33ms, i.e. mean 4.16ms. >a single disk will often have well over 100,000 tracks By my calculations a Seagate IronWolf 18TB has about 615K tracks per surface given that it has 9 platters and 18 surfaces, and an outer diameter read speed of about 260MB/s. (or 557K tracks/inch given typical inner and outer track diameters) For more than you ever wanted to know about hard drive performance and the mechanical/geometrical considerations that go into it, see https://www.msstconference.org/MSST-history/2024/Papers/msst24-1.1.pdf https://www.msstconference.org/MSST-history/2024/Papers/msst...
- bddicken 2y agoWhoah, thanks for sharing the paper.
- pjdesno 2y agoI reviewed it three times for different conferences :-) I’m still annoyed they didn’t include the drain time equation I used for calculating track width, which falls out of one of their equations. Oh, and I’m very glad you showed differing track sizes across the platter. (BTW, did you know track sizes differ between platters? Google “disks are like snowflakes”)
- dangoodmanUT 2y agowhat local nvme is getting 20us? Nitro?
- anonymousDan 2y agoNice article, but the replicated approach isn't exactly comparing like with like. To achieve the same semantics you'd need to block for a response from the remote backup servers which would end up with the same latency as the other cloud providers...
- CSDude 2y agoFor years, I just didn't get why replicated databases always stick with EBS and deal with its latency. Like, replication is already there, why not be brave and just go with local disks? At my previous orgs, where we ran Elasticsearch for temporary logs/metrics storage, I proposed we do exactly that since we didn't even have major reliability requirements. But I couldn't convince them back then, we ended up with even worse AWS Elasticsearch. I get that local disks are finite, yeah, but I think the core/memory/disk ratio would be good enough for most use cases, no? There are plenty of local disk instances with different ratios as well, so I think a good balance could be found. You could even use local hard disk ones with 20TB+ disks for implementing hot/cold storage. Big kudos to the PlanetScale team, they're like, finally doing what makes sense. I mean, even AWS themselves don't run Elasticsearch on local disks! Imagine running ClickHouse, Cassandra, all of that on local disks.
- jiggawatts 2y agoI looked into this with an idea of running SQL Server Availability Groups on the Azure Las_v3 series VMs, which have terabytes of local SSD. The main issue was that after a stop-start event, the disks are wiped. SQL Server can’t automatically handle this, even if the rest of the cluster is fine and there are available replicas. It won’t auto repair the node that got reset. The scripting and testing required to work around this would be unsupportable in production for all but the bravest and most competent orgs.
- hodgesrm 2y agoThere are a number of axes of performance that aren't covered in this [wonderful] article on storage performance. One of these is that EBS allows you to scale the VM up / down to change the amount of CPU & RAM available to process data on disk. We run several hundred ClickHouse clusters on this model. Rescaling to address performance issues is far more common than failures. Example; you get a tenant performance issue on Sunday morning US time. The simplest fix is often rescale to a larger VM for the weekend, then get the A team working on the root cause first thing Monday. The incremental cost is minimal and avoids far more costly staff burnout.
- _1tem 2y agoIf this is true, then how do "serverless" database providers like Neon advertise "low latency" access? They use object storage like S3, which I imagine is an order of magnitude worse than networked storage for latency. edit: apparently they build a kafkaesque layer of caching. No thank you, I'll just keep my data on locally attached NVMe.
- hodgesrm 2y ago> edit: apparently they build a kafkaesque layer of caching. No thank you, I'll just keep my data on locally attached NVMe. I can't speak to Neon specifically but I've worked a lot with analytic databases, which often use NVMe SSD caches to operate efficiently on S3 data. For time-ordered datasets like observability (e.g., metrics) most queries go to recent data which in the steady state is not just in NVMe SSD storage but generally RAM as well if you are properly tuned. For example, indexes and other metadata are permanently cached. In realistic tests of the above scenario the effect of nVME SSD can be surprisingly muted. That's especially true if you can use clusters that spread processing across multiple compute nodes, which gives you more RAM to play with and also multiplies storage bandwith. There are downsides to S3 of course like restarts, which require management to avoid performance issues.
- liweixin 2y agoAmazing! The visualizations are so great!
- TheAnkurTyagi 2y agoVery nice animations.
- carderne 2y agoI'm always curious about latency for all these newdb offerings like PlanetScale/Neon/Supabase. It seems like they don't emphasise strongly enough _make sure you colocate your server in the same cloud/az/region/dc as our db. I suspect a large fraction of their users don't realise this, and have loads of server-db traffic happening very slowly over the public internet. It won't take many slow db reads (get session, get a thing, get one more) to trash your server's response latency.
- SAI_Peregrinus 2y ago> The next major breakthrough in storage technology was the hard disk drive. There were a few storage methods in between tape & HDDs, notably core memory & magnetic drum memory.