12 ms·

At my job at a telco, I had a 13 billion record file to scan and index for duplicates and bad addresses. Consultants brought in to move our apps (some of which

by wistlo 3y ago

At my job at a telco, I had a 13 billion record file to scan and index for duplicates and bad addresses. Consultants brought in to move our apps (some of which were Excel macros, others SAS scripts running on old desktop) to Azure. The Azure architects identified Postgres as the best tool. Consultants attempted to create a Postgres index in a small Azure instance but their tests would fail without completion (they were string concatenation rather than the native indexing function). Consultants' conclusion: file too big for Postgres. I disputed this. Plenty of literature out there on Pg handling bigger files. The Postgres (for Windows!) instance on my Core I7 laptop with an nVME drive could index the file about an hour. As an experiment I spun up a bare metal nVME instance on a Ryzen 7600 (lowest power, 6 core) Zen 4 CPU pc with a 1TB Samsung PCIe 4 nVME drive. Got my index in 10 minutes. I then tried to replicate this in Azure, upping the CPUs, memory, and to the nVME Azure CPU family (Ebsv5). Even at a $2000/mo level, I could not get the Azure instance any faster than one fifth (about an hour) of the speed of my bare metal experiment. I probably could have matched it eventually with more cores, but did not want to get called on the carpet for a ten grand Azure bill. All this happened while I was working from home (one can't spin up an experimental bare metal system at a drop-in spot in the communal workroom). What happened next I don't know, because I left in the midst of RTO fever. I was given the option of moving 1000 miles to commute to a hub office, or retire "voluntarily with severance." I chose the latter.

bcaxis 3y ago
This matches my experience. Something about cloud systems makes them incredibly slow compared to real hardware. Not just the disk, the CPU is more limited too. I fire up vCPU or dedicated or bare metal in the cloud, doesn't matter, I simply cannot match the equivalent compute of real hardware and it's not even close.
- silent_cal 3y ago
  Isn't that expected? I would assume cloud stuff is slower because it's essentially an emulation of the real thing.
  - pdimitar 3y ago
    Why should it be expected? I'm being sold compute with quoted GHz CPU speeds, RAM and types of SSDs. I would vaguely expect it to not match my workstation, sure, but all throughout this thread (and others) people have cited outrageous disparities i.e. 5x less performance that you'd expect even if you managed your expectations to e.g. 2x less due to the cloud compute not being a bare metal machine. In other words, and to illustrate this with a bad example: I'd be fine paying for an i7 CPU and ending up at i5 speeds... but I'm absolutely not fine with ending up at Celeron speeds.
    - silent_cal 3y ago
      Makes sense.
  - bcaxis 3y ago
    When I spin up a vm on my hardware and run an application. The performance is generally about 70% of what I can get in a container running on an OS on that very same bare metal. But that isn't the delta I'm seeing, it's 5-10x performance delta not a 30-50% delta.
rcarmo 3y ago
As someone who works with Azure daily, I am amazed not just at the initial consultant's conclusion (that is, alas, typical of folk who do not understand database engines), but also to your struggle with NVMe storage (I have some pretty large SQLite databases on my personal projects). You should not have needed an Ebsv5 (memory-optimised) instance. For that kind of thing, you should only have needed a D-series VM with a premium storage data disk (or, if you wanted a hypervisor-adjacent, very low latency volume, a temp volume in another SKU). Anyway, many people fail to understand that Azure Storage works more like a SAN than a directly attached disk--when you attach a disk volume to the VM, you are actually attaching a _replica set_ of that storage that is at least three-way replicated and distributed across the datacenter to avoid data loss. You get RAID for free, if you will. That is inherently slower than a hypervisor-adjacent (i.e., on-board) volume.
- silverquiet 3y ago
  > Anyway, many people fail to understand that Azure Storage works more like a SAN than a directly attached disk--when you attach a disk volume to the VM, you are actually attaching a _replica set_ of that storage that is at least three-way replicated and distributed across the datacenter to avoid data loss. You get RAID for free, if you will. I've said this a bit more sarcastically elsewhere in this thread, but basically, why would you expect people to understand this? Cloud is sold as abstracting away hardware details and giving performance SLAs billed by the hour (or minute, second, whatever). If you need to know significant details of their implementation, then you're getting to the point where you might as well buy your own hardware and save a bunch of money (which seems to be gaining some steam in a minor but noticeable cloud repatriation movement).
  - rcarmo 3y ago
    Well, in short, people need to understand that cloud is not their computer. It is resource allocation with underlying assumptions around availability, redundancy and performance at a scale well beyond what they would experience in their own datacenter. And they absolutely must understand this to avoid mis-designing things. Failure to do so is just bad engineering, and a LOT of time is spent educating customers on these differences. A case in point that aligns with is that I used to work with Hadoop clusters, where you would use data replication for both redundancy and distributed processing. Moving Hadoop to Azure and maintaining conventional design rules (i.e., tripling the amount of disks) is the wrong way do do things, because it isn't required neither for redundancy nor for performance (they are both catered for by the storage resources). (Of course there are better solutions than Hadoop these days - Spark being one that is very nice from a cloud resource perspective - but many people have nine times the storage they need allocated in their cloud Hadoop clusters because of lack of understanding...)
Too 3y ago
Ebsv5 does not have a local nvme. To get local drive you need to pick the version with a “d” in the name.
- motles 3y ago
  Lsv3 series
motles 3y ago
Was this on the L-series or managed disk?
- wistlo 3y ago
  I tried a variety of configurations. The E-series was one of them, as it's advertised as "Great for relational database servers, medium to large caches, and in-memory analytics." Premium and Premium V2, tried those at larger capacities I didn't need just to get higher IOPS. None came within an order of magnitude of a Ryzen 7600/nVME mobo sitting in my son's old gaming case. An option I did not try was Ultra disk, which I recall being significantly more expensive and was not part of the standard corporate offering. I wasn't itching to get dragged in front of the architecture review board again, anyway.

by wistlo 3y ago

bcaxis 3y ago

This matches my experience. Something about cloud systems makes them incredibly slow compared to real hardware. Not just the disk, the CPU is more limited too. I fire up vCPU or dedicated or bare metal in the cloud, doesn't matter, I simply cannot match the equivalent compute of real hardware and it's not even close.

silent_cal 3y ago
Isn't that expected? I would assume cloud stuff is slower because it's essentially an emulation of the real thing.
- pdimitar 3y ago
  Why should it be expected? I'm being sold compute with quoted GHz CPU speeds, RAM and types of SSDs. I would vaguely expect it to not match my workstation, sure, but all throughout this thread (and others) people have cited outrageous disparities i.e. 5x less performance that you'd expect even if you managed your expectations to e.g. 2x less due to the cloud compute not being a bare metal machine. In other words, and to illustrate this with a bad example: I'd be fine paying for an i7 CPU and ending up at i5 speeds... but I'm absolutely not fine with ending up at Celeron speeds.
  - silent_cal 3y ago
    Makes sense.
- bcaxis 3y ago
  When I spin up a vm on my hardware and run an application. The performance is generally about 70% of what I can get in a container running on an OS on that very same bare metal. But that isn't the delta I'm seeing, it's 5-10x performance delta not a 30-50% delta.

rcarmo 3y ago

As someone who works with Azure daily, I am amazed not just at the initial consultant's conclusion (that is, alas, typical of folk who do not understand database engines), but also to your struggle with NVMe storage (I have some pretty large SQLite databases on my personal projects). You should not have needed an Ebsv5 (memory-optimised) instance. For that kind of thing, you should only have needed a D-series VM with a premium storage data disk (or, if you wanted a hypervisor-adjacent, very low latency volume, a temp volume in another SKU). Anyway, many people fail to understand that Azure Storage works more like a SAN than a directly attached disk--when you attach a disk volume to the VM, you are actually attaching a _replica set_ of that storage that is at least three-way replicated and distributed across the datacenter to avoid data loss. You get RAID for free, if you will. That is inherently slower than a hypervisor-adjacent (i.e., on-board) volume.

silverquiet 3y ago
> Anyway, many people fail to understand that Azure Storage works more like a SAN than a directly attached disk--when you attach a disk volume to the VM, you are actually attaching a _replica set_ of that storage that is at least three-way replicated and distributed across the datacenter to avoid data loss. You get RAID for free, if you will. I've said this a bit more sarcastically elsewhere in this thread, but basically, why would you expect people to understand this? Cloud is sold as abstracting away hardware details and giving performance SLAs billed by the hour (or minute, second, whatever). If you need to know significant details of their implementation, then you're getting to the point where you might as well buy your own hardware and save a bunch of money (which seems to be gaining some steam in a minor but noticeable cloud repatriation movement).
- rcarmo 3y ago
  Well, in short, people need to understand that cloud is not their computer. It is resource allocation with underlying assumptions around availability, redundancy and performance at a scale well beyond what they would experience in their own datacenter. And they absolutely must understand this to avoid mis-designing things. Failure to do so is just bad engineering, and a LOT of time is spent educating customers on these differences. A case in point that aligns with is that I used to work with Hadoop clusters, where you would use data replication for both redundancy and distributed processing. Moving Hadoop to Azure and maintaining conventional design rules (i.e., tripling the amount of disks) is the wrong way do do things, because it isn't required neither for redundancy nor for performance (they are both catered for by the storage resources). (Of course there are better solutions than Hadoop these days - Spark being one that is very nice from a cloud resource perspective - but many people have nine times the storage they need allocated in their cloud Hadoop clusters because of lack of understanding...)

Too 3y ago

Ebsv5 does not have a local nvme. To get local drive you need to pick the version with a “d” in the name.

motles 3y ago
Lsv3 series

motles 3y ago

Was this on the L-series or managed disk?

wistlo 3y ago
I tried a variety of configurations. The E-series was one of them, as it's advertised as "Great for relational database servers, medium to large caches, and in-memory analytics." Premium and Premium V2, tried those at larger capacities I didn't need just to get higher IOPS. None came within an order of magnitude of a Ryzen 7600/nVME mobo sitting in my son's old gaming case. An option I did not try was Ultra disk, which I recall being significantly more expensive and was not part of the standard corporate offering. I wasn't itching to get dragged in front of the architecture review board again, anyway.