10 ms·

AWS in 2025: Stuff you think you know that's now wrong

387 points by keithly 1y ago

romperstomper 1y ago
> DynamoDB You can now have empty fields When this was changed? I think this is still an issue, I've had some such errors quite recently.
- bluecassette 1y ago
  18th May 2020 > DynamoDB now supports empty values for non-key String and Binary attributes in DynamoDB tables. Empty value support gives you greater flexibility to use attributes for a broader set of use cases without having to transform such attributes before sending them to DynamoDB. List, Map, and Set data types also support empty String and Binary values. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DocumentHistory.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
cldcntrl 1y ago
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. Not strictly true.
- rthnbgrredf 1y ago
  Elaborate.
  - cldcntrl 1y ago
    The whole auto-balancing thing isn't instant. If you have a burst of writes with the same key prefix, you'll get throttled.
  - hnlmorg 1y ago
    Not the OP but I’ve had AWS-staff recommend different prefixes even as recently as last year. If key prefixes don’t matter much any more, then it’s a very recent change that I’ve missed.
    - cldcntrl 1y ago
      That's right, same for me as of only a few months ago.
    - williamdclt 1y ago
      Might just be that the AWS staff wasn't up to date on this
      - hnlmorg 1y ago
        That’s possible but they did consult with the storage team prior to our consultation. But I don’t know what conversations did or did not happen behind the scenes.
      - rthnbgrredf 1y ago
        By the way, that happens quite frequently. I regularly ask them about new AWS technologies or recent changes, and most of the time they are not aware. They usually say they will call back later after doing some research.
      - time0ut 1y ago
        I have had the same experience within the last 18 months. The storage team came back to me and asked me to spread my ultra high throughput write workload across 52 (A-Za-z) prefixes and then they pre-partitioned the bucket for me. S3 will automatically do this over time now, but I think there are/were edge cases still. I definitely hit one and experienced throttling at peak load until we made the change.
        hnlmorg 1y ago
        That’s sounds like the problem we were having. Lots of writes to a prefix over a short period of time and then low activity to it after about 2 weeks.
- QuinnyPig 1y ago
  I should have been more clear. You still need to partition, but randomizing the prefixes hasn't been needed since 2018: https://web.archive.org/web/20240227073321/https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ https://web.archive.org/web/20240227073321/https://aws.amazo...
- ed_g 1y ago
  Generally speaking this isn't something Amazon S3 customers need to worry about - as others have said, S3 will automatically scale index performance over time based on load. The challenge primarily comes when customers need large bursts of requests within a namespace that hasn't had a chance to scale - that's when balancing your workload over randomized prefixes is helpful. Please see the documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi... This 2024 re:Invent session "Optimizing storage performance with Amazon S3 (STG328)" which goes very deep on the subject: https://www.youtube.com/watch?v=2DSVjJTRsz8 https://www.youtube.com/watch?v=2DSVjJTRsz8 And this blog that discusses Iceberg's new base-2 hash file layout which helps optimize request scaling performance of large-scale Iceberg workloads running on S3: https://aws.amazon.com/blogs/storage/how-amazon-ads-uses-iceberg-optimizations-to-accelerate-their-spark-workload-on-amazon-s3/ https://aws.amazon.com/blogs/storage/how-amazon-ads-uses-ice...
- vvoyer 1y ago
  This 2024 re:Invent session says exactly the opposite: "If you want to partition your data even better, you can introduce some randomness in your key names": https://youtu.be/2DSVjJTRsz8?t=2206 https://youtu.be/2DSVjJTRsz8?t=2206 FWIW The optimal way we were told was to partition our data was to do this: 010111/some/file.jpg. Where `010111/` is a random binary string which will please both the automatic partitioning (503s => partition) and manual partitioning you could ask AWS. Please as in the cardinality of partitions grows slower at each characters vs prefixes like `az9trm/`. We were told that the later version makes manual partitioning a challenge because as soon as you reach two characters you've already created 36x36 partitions (1,296). The issue with that: your keys are no more meaningful if you're relying on S3 to have "folders" by tenants for example (customer1/..).
SOLAR_FIELDS 1y ago
You know what's still stupid? That if you have an S3 bucket in the same region as your VPC that you will get billed on your NAT Gateway to send data out to the public internet and right back in to the same datacenter. There is simply no reason to not default that behavior to opt out vs opt in (via a VPC endpoint) beyond AWS profiting off of people's lack of knowledge in this realm. The amount of people who would want the current opt-in behavior is... if not zero, infinitesimally small.
- deleted 1y ago
  [deleted]
- afandian 1y ago
  Having experienced the joy of setting up VPC, subnets and PrivateLink endpoints the whole thing just seems absurd. They spent the effort of branding private VPC endpoints "PrivateLink". Maybe it took some engineering effort on their part, but it should be the default out of the box, and an entirely unremarkable feature. In fact, I think if you have private subnets, the only way to use S3 etc is Private Link (correct me if I'm wrong). It's just baffling.
  - time0ut 1y ago
    You can provision gateway endpoints for S3 and DynamoDB. They are free and considered best practice. They are opt-in though, but easy to enable.
    - mdaniel 1y ago
      And ECR, which I would guess impacts more folks than DynamoDB https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html#ecr-setting-up-vpc-create https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-e... And, as as added benefit, they distinguish between "just pull" and "pull and push" which is nice
    - afandian 1y ago
      True, I forgot that. But depending on services you still have to have some Gateway and some Interface endpoints.
simonw 1y ago
S3: "Block Public Access is now enabled by default on new buckets." On the one hand, this is obviously the right decision. The number of giant data breeches caused by incorrectly configured S3 buckets is enormous. But... every year or so I find myself wanting to create an S3 bucket with public read access to I can serve files out of it. And every time I need to do that I find something has changed and my old recipe doesn't work any more and I have to figure it out again from scratch!
- reactordev 1y ago
  They'll teach you how for $250 and a certification test...
- SOLAR_FIELDS 1y ago
  I honestly don't mind that you have to jump through hurdles to make your bucket publically available and that it's annoying. That to me seems like a feature, not a bug
  - simonw 1y ago
    Sure... but last time I needed to jump through those hurdles I lost nearly an hour to them! I'm still not sure I know how to do it if I need to again.
  - dghlsakjg 1y ago
    I think the OPs objection is not that hurdles exist but that they move them every time you try and run the track.
- crinkly 1y ago
  I just stick CloudFront in front of those buckets. You don't need to expose the bucket at all then and can point it at a canonical hostname in your DNS.
  - hnlmorg 1y ago
    That’s definitely the “correct” way of doing things if you’re writing infra professionally. But I do also get that more casual users might prefer not to incur the additional costs nor complexity of having CloudFront in front. Though at that point, one could reasonably ask if S3 is the right choice for causal users.
aaronblohowiak 1y ago
>VPC peering used to be annoying; now there are better options like Transit Gateway, VPC sharing between accounts, resource sharing between accounts, and Cloud WAN. TGW is... twice as expensive as vpc peering?
- alFReD-NSH 1y ago
  And vpc sharing is free. Cost and architecture are tied.
- Hikikomori 1y ago
  More than twice as same AZ is free with peering. But if you're big enough you can get better deals on cost. But unlike peering TGW traffic flows through an additional compute layer so it has additional cost.
- klysm 1y ago
  VPC sharing is the sleeper here. You can do cross account networking all in the same VPC and skip all the expensive stuff.
  - aaronblohowiak 1y ago
    as long as your VPCs aren't too big, yea.
    - Hikikomori 1y ago
      Shared vpcs can get pretty big. Even if you approach the NAU limit you can use privatelink or TGW to have more large shared vpcs.
    - klysm 1y ago
      If you are exhausting an entire VPC I’d be pretty impressed!
      - aaronblohowiak 1y ago
        Used to work at nflx, biiiiig headaches
Ayesh 1y ago
CloudFront also has 1TB of free data transfer a month under the forever-free perks.
csours 1y ago
Strictly off topic: Everything you know is wrong. Weird Al. https://www.youtube.com/watch?v=W8tRDv9fZ_c https://www.youtube.com/watch?v=W8tRDv9fZ_c Firesign Theatre. https://www.youtube.com/watch?v=dAcHfymgh4Y https://www.youtube.com/watch?v=dAcHfymgh4Y
gurjeet 1y ago
It would've been nice if each of those claims in the article also linked to either the relevant announcement or to the documentation. If I'm interested in any of these headline items, I'd like to learn more.
- mdaniel 1y ago
  I don't believe AWS offers permalinks, so it would only help until they rolled over the next documentation release :-( They actually used to have the upstream docs in GitHub, and that was super nice for giving permalinks but also building the docs locally in a non-pdf-single-file setup. Pour one out, I guess
bob1029 1y ago
> Glacier restores are also no longer painfully slow. Wouldn't this always depend on the length of the queue to access the robotic tape library? Once your tape is loaded it should move really quickly: https://www.ibm.com/docs/en/ts4500-tape-library?topic=performance-lto-specifications https://www.ibm.com/docs/en/ts4500-tape-library?topic=perfor...
- hinkley 1y ago
  > Once upon a time Glacier was its own service that had nothing to do with S3. If you look closely (hi, billing data!) you can see vestiges of how this used to be, before the S3 team absorbed it as a series of storage classes. Your assumption holds if they still use tape. But this paragraph hints at it not being tape anymore. The eternal battle between tape versus drive backup takes another turn.
  - bob1029 1y ago
    I am also assuming that Amazon intends for the Deep Archive tier to be a profitable offering. At $0.00099/gb-month, I don't see how it could be anything other than tape.
    - simonw 1y ago
      I wonder if it's where old S3 hard drives go to die? Presumably AWS have the world's single largest collection of used storage devices - if you RAID them up you can probably get reliable performance out of them for Glacier?
      - bob1029 1y ago
        I still don't know if it's possible to make it profitable with old drives in this kind of arrangement, especially if we intend to hit their crazy durability figures. The cost of keeping drives spinning is low, but is double-digit margin % in this context. You can't leave drives unpowered in a warehouse for years on end and say you have 11+ nines of durability.
chisleu 1y ago
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. As of when? According to internal support, this is still required as of 1.5 years ago.
- laurent_du 1y ago
  He's not talking about the prefix, just the beginning of the object key.
  - viccis 1y ago
    The prefix is not separate from the object key. It's part of it. There's no randomization that needs to be done on either anymore.
    - chisleu 1y ago
      and indeed the bucket is not separate from the object key. the API separates it logically "for humans" but it's all one big string
- arpinum 1y ago
  I think there is some nuance needed here. If you ask support to partition your bucket then they will be a bit annoying if you ask for specific partition points and the first part of the prefix is not randomised. They tried to push me to refactor the bucket first to randomise the beginning of the prefix, but eventually they did it. The auto partitioning is different. It can isolate hot prefixes on its own and can intelligently pick the partition points. Problem is the process is slow and you can be throttled for more than a day before it kicks in.
  - chisleu 1y ago
    > but eventually they did it They can do this with manual partitioning indeed. I've done it before, but it's not ideal because the auto partitioner will scale beyond almost anything AWS will give you with manual partitioning unless you have 24/7 workloads. > you can be throttled for more than a day before it kicks in I expect that this would depends on your use case. If you are dropping content you need to scale out to tons of readers, that is absolutely the case. If you are dropping tons of content with well distributed reads, then the auto partitioner is The Way.
scubbo 1y ago
I've had two people tell me in the last week that SQS doesn't support FIFO queues.
TheP1000 1y ago
API gateway timeout increase has been nice.
- coredog64 1y ago
  It was always there but it required much more activity to get it done (document your use case & traffic levels and then work with your TAM to get the limit changed).
- quesera 1y ago
  I don't see that in this post. I just started working with a vendor who has a service behind API Gateway. It is a bit slow(!) and times out at 30 seconds. I've since modified my requests to chunk subsets of the whole dataset, to keep things under the timeout. Has this changed? Is 30 secs the new or the old timeout?
  - TheP1000 1y ago
    https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-api-gateway-integration-timeout-limit-29-seconds/ https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-ap...
Annatar 1y ago
[dead]
stevejb 1y ago
I just saw Weird Al in concert, and one of my favorite songs of his is "Everything You Know is Wrong." This is the AWS version of that song! Nice work Corey!
- havefunbesafe 1y ago
  Weird AL or Weird A.I.?
- jeffbarr 1y ago
  I also saw that concert. Oh yeah, we were in the same row!
nodesocket 1y ago
I'll add: When doing instance to instance communication (in the same AZ) always use private ips. If you use public ip routing (even the same AZ) this is charged as regional data transfer. Even worse, if you run self hosted NAT instance(s) don't use a EIP attached to them. Just use a auto-assigned public IP (no EIP). NAT instance with EIP - AWS routes it through the public AWS network infrastructure (hairpinning). - You get charged $0.01/GB regional data transfer, even if in the same AZ. NAT instance with auto-assigned public IP (no EIP) - Traffic routes through the NAT instance’s private IP, not its public IP. - No regional data transfer fee — because all traffic stays within the private VPC network. - auto-assigned public IP may change if the instance is shutdown or re-created so have automations to handle that. Though you should be using the network interface ID reference in your VPC routing tables.
- themafia 1y ago
  > You get charged $0.01/GB regional data transfer, even if in the same AZ. My understanding is that transfer gets charged on both sides as well. So if you own both sides you'll pay $0.02/GB.
  - pcthrowaway 1y ago
    Is transfer into the instance not always free anymore?
digianarchist 1y ago
Would love an AWS equivalent to Cloud Run but the lambda changes are welcome nonetheless.
- aranelsurion 1y ago
  Isn't Fargate the AWS equivalent of Cloud Run?
  - mdaniel 1y ago
    It seems Cloud Run comes "batteries included" for a lot more things that one must provision separate AWS resources to achieve https://cloud.google.com/architecture/migrate-aws-lambda-to-cloudrun#:~:text=to%20help%20you%20understand%20which%20cloud%20run%20invocation%20or%20execution%20mechanisms%20are%20comparable%20to%20aws%20lambda%20invocation%20mechanisms https://cloud.google.com/architecture/migrate-aws-lambda-to-...
topher200 1y ago
I have a preempt-able workload for which I could use Spot instances or Savings Plans. Does anyone have experience running Spot in 2025? If you were to start over, would you keep using Spot? - I observe with pricing that Spot is cheaper - I am running on three different architectures, which should limit Spot unavailability - I've been running about 50 Spot EC2 instances for a month without issue. I'm debating turning it on for many more instances
- erulabs 1y ago
  In terms of cost, from cheapest to most expensive: 1. Spot with autoscaling to adjust to demand and a savings plan that covers the ~75th percentile scale 2. On-demand with RIs (RIs will definitely die some day) 3. On-demand with savings-plans (More flexible but more expensive than RIs) 3. Spot 4. On-demand I definitely recommend spot instances. If you're greenfielding a new service and you're not tied to AWS, some other providers have hilariously cheap spot markets - see http://spot.rackspace.com/ http://spot.rackspace.com/. If you're using AWS, definitely auto-scaling spot with savings plans are the way to go. If you're using Kubernetes, the AWS Karpenter project (https://karpenter.sh/ https://karpenter.sh/) has mechanisms for determining the cheapest spot price among a set of requirements. Overall tho, in my experience, ec2 is always pretty far down the list of AWS costs. S3, RDS, Redshift, etc wind up being a bigger bill in almost all past-early-stage startups.
  - mdaniel 1y ago
    To "me, too" this, it's not like that AWS spot instance just go "poof," they do actually warn you (my recollection is 60s in advance of the TerminateInstance call), and so a resiliency plane on top of the workloads (such as the cited Kubernetes) can make that a decided "non-event". Shout out to the reverse uptime crew, a subset of Chaos Engineering
    - cptnntsoobv 1y ago
      The spot termination warnings arrive 120 seconds before termination https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-ins...
general1726 1y ago
I think there is more of us who kind of degenerated from doing it the AWS way - API Gateway, serverless lambdas mess around with IAM roles until it works, ... - to - Give me EC2 / LightSail VPS instance maybe an S3 bucket let's set domain through Route53 and go away with the rest of your orchestrion AWS.
- calmbonsai 1y ago
  There are entire industries that have largely de-volved their clouds primarily for footprint flexibility (not all AWS services are in all regions) and billing consistency.
- regularfry 1y ago
  Honestly just having to manage IAM is such a time-suck that the way I've explained it to people is that we've traded the time we used to spend administering systems for time spent just managing permissions, and IAM is so obtuse that it comes out as a net loss. There's a sweet spot somewhere in between raw VPSes and insanely detailed least-privilege serverless setups that I'm trying to revert to. Fargate isn't unmanageable as a candidate, not sure it's The One yet but I'm going to try moving more workloads to it to find out.
  - sklarsa 1y ago
    Usually I write some IaC to automate this tedium so I only have to go through the IAM setup pain once. Now if requirements change, that's an entirely different story...
    - regularfry 1y ago
      So the problem when you combine IAC with CI/CD is that the role assumed by the CI agent needs privileges to deploy things, so you need a bootstrap config to set up what it needs. If you have a mandate to go least-privilege, then that needs to include only the permissions strictly needed by the current deployable. So, no "s3:*", you need each one listed. So far so good, you can do this with a bootstrap script that you only need to run at project setup. If you also have a mandate (effectively) to go fully serverless, then as your project evolves and you add functionality, what you find is that most interesting changes use something new in the platform. So you're not getting away with running the bootstrap script once. You're updating it and running it for almost every change. And you can't tell in advance what permissions you're going to need, because (especially if you're on terraform) there's apparently no documentation connecting the resources you want to manage and the permissions needed to do so. So you try to deploy your change, IAM pops an error or two, you try to figure out what permissions you need to add to the bootstrap script, you run it (fixing it when it breaks at this point), you try deploying again, IAM pops another couple of errors, and then you're in a grind cycle which you can't predict the length of - and you need to get to the end of it before you can even test your feature, because fully serverless means you can't run your application locally (and getting management to pay for the pro localstack licence is a dead end). At some point it won't be clear why IAM is complaining, because the error you get makes no sense whatsoever, so at that point it's off to support to find out a day later that ah, yes, you can't use an assumed role just there, it's got to be an actual role, and no, that's not written down anywhere, you've just got to know it, so you need to redesign how you're using the roles completely, and right about this point is when I usually want to buy a farm, raise goats, and get way too into oil painting, instead of whatever this insane waste of life is.
JCM9 1y ago
Some good stuff here. I wish AWS would just focus on these boring, but ultimately important, things that they’re good at instead of all the current distractions trying to play catch up on “AI.” AWS leadership missed the boat there big time, but that’s OK. Ultimately AWS doesn’t have the right leadership or talent to be good at GenAI, but they do (or at least used to) have decent core engineers. I’d like to see them get back to basics and focus there. Right now leadership seems panicked about GenAI and is just throwing random stuff at the wall desperately trying to get something to stick. Thats really annoying to customers.
- PartiallyTyped 1y ago
  Leadership is looking to provide infrastructure for anyone to just pick a model and get on doing things without the hassle of setting things up.
- anon7000 1y ago
  They continue to have large teams working on core stuff. It’s just that they’re working at such a low level (like high perf virtualized networking on their custom network cards) that most people don’t hear about it or care that much.
causal 1y ago
This is super helpful. I would read a yearly summary like this.
beaviskhan 1y ago
Also S3 related: the bucket owner can now be configured as the object owner no matter where the object originated. In the past this was exceedingly painful if you wanted to allow one account contribute objects to a bucket in another account. You could do the initial contribution, but the contributor always owned the object, and you couldn't delegate access to a third account.
viccis 1y ago
>In EC2, you can now change security groups and IAM roles without shutting the instance down to do it. Hasn't it been this way for many years? >Spot instances used to be much more of a bidding war / marketplace. Yeah because there's no bidding any more at all, which is great because you don't get those super high spikes as availability drops and only the ones who bid super high to ensure they wouldn't be priced out are able to get them. >You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. This one was a nightmare and it took ages to convince some of my more pig headed coworkers in the past that they didn't need to do it any more. The funniest part is that they were storing their data as millions and millions of 10-100kb files, so the S3 backend scaling wasn't the thing bottlenecking performance anyway! >Originally Lambda had a 5 minute timeout and didn’t support container images. Now you can run them for up to 15 minutes, use Docker images, use shared storage with EFS, give them up to 10GB of RAM (for which CPU scales accordingly and invisibly), and give /tmp up to 10GB of storage instead of just half a gig. This was/is killer. It used to be such a pain to have to manage pyarrow's package size if I wanted a Python Lambda function that used it. One thing I'll add that took me an embarrassingly long time to realize is that your Python global scope is actually persisted, not just the /tmp directory.
- indigodaddy 1y ago
  Re: SG, yeah I wasnt doing any cloud stuff when that was the case. Never had to restart anything for an SG change and this must be at least 5-6 years..
  - buzzdenver 1y ago
    IAM Role change is more recent though
- Inufu 1y ago
  > You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. Sorry, this is absolutely still the case if you want to scale throughput beyond the few thousand IOPS a single shard can serve. S3 will automatically reshard your key space, but if your keys are sequential (eg leading timestamp) all your writes will still hit the same shard. Source: direct conversations with AWS teams.
PaulDavisThe1st 1y ago
Can no longer login to my AWS account, because I never set up MFA. Want to set up MFA ... login required to request device. Yes, I know, they warned us far ahead of time. But not being able to request one of their MFA devices without a login is ... sucky.
- berlesi 1y ago
  Looks like something that you could solve easily through their support, no?
  - raffraffraff 1y ago
    Support don't talk to you unless you pay for support
    - danudey 1y ago
      Easy, just log into your account and pay them for support.
    - austinshea 1y ago
      Yes, they do. https://support.aws.amazon.com/#/contacts/one-support?formId=contactUs https://support.aws.amazon.com/#/contacts/one-support?formId...
- austinshea 1y ago
  Talk to support: https://support.aws.amazon.com/#/contacts/one-support?formId=contactUs https://support.aws.amazon.com/#/contacts/one-support?formId...
  - PaulDavisThe1st 1y ago
    I did. That's just an AI, which says this: > I understand your situation is a bit unique, where you are unable to log in to your AWS account without an MFA device, but you also can't order an MFA device without being able to log in. This is a scenario that is not directly covered in our standard operating procedures. The best course of action would be for you to contact AWS Support directly. They will be able to review your specific case and provide guidance on how to obtain an MFA device to regain access to your account. The support team may have alternative options or processes they can walk you through to resolve this issue. Please submit a support request, and one of our agents will be happy to assist you further. You can access the support request form here: https://console.aws.amazon.com/support/home https://console.aws.amazon.com/support/home That last URL? You need to login to use it ...
jp57 1y ago
> Glacier restores are also no longer painfully slow. I had a theory (based on no evidence I'm aware of except knowing how Amazon operates) that the original Glacier service operated out of an Amazon fulfillment center somewhere. When you put it a request for your data, a picker would go to a shelf, pick up some removable media, take it back, and slot it into a drive in a rack. This, BTW, is how tape backups on timesharing machines used to work once upon a time. You'd put in a request for a tape and the operator in the machine room would have to go get it from a shelf and mount it on the tape drive.
- browningstreet 1y ago
  Yeah, but they've been robotic for decades since.
- danudey 1y ago
  The most likely explanation is that they used a tape robot, such as the one seen here: https://www.reddit.com/r/DataHoarder/comments/12um0ga/the_robot_managing_my_universitys_backup_each_of/ https://www.reddit.com/r/DataHoarder/comments/12um0ga/the_ro... Which is basically exactly what you described but the picker is a robot. Data requests go into a queue; when your request comes up, the robot looks up the data you requested, finds the tape and the offset, fetches the tape and inserts it into the drive, fast-forwards it to the offset, reads the file to temporary storage, rewinds the tape, ejects it, and puts it back. The latency of offline storage is in fetching/replacing the casette and in forwarding/rewinding the tape, plus waiting for an available drive. Realistically, the systems probably fetch the next request from the queue, look up the tape it's on, and then process every request from that tape so they're not swapping the same tape in and out twenty times for twenty requests.
  - philistine 1y ago
    I've read very definitive discussions on here that Glacier never used tape. It has always been powered off hard disks.
    - UltraSane 1y ago
ElijahLynn 1y ago
That. Was a decent investment of my time as a devops engineer. Right to the point. I learned things.
kassner 1y ago
I haven’t used AWS in the last 5 years. Is IPv6 still somewhat of an issue? I remember some services not supporting it at all and making it impossible to manage as a IPv6-only network.
- skywhopper 1y ago
  Yeah, it’s still limited, and a few things still require at least a dual stack setup.
- 1oooqooq 1y ago
  gotta milk those ipv4 investment
- mdaniel 1y ago
  You'll be glad(sic) to know they have specialized hostnames for IPv6 control plane endpoints, because AAAA records scare the olds https://github.com/aws/aws-cli/blob/2.28.14/awscli/botocore/data/iam/2010-05-08/endpoint-rule-set-1.json#L159 https://github.com/aws/aws-cli/blob/2.28.14/awscli/botocore/...
  - conradludgate 1y ago
    We enabled ipv6 for our APIs at work. Nothing broke immediately, but we've had a steady stream of unreachable host errors related to ipv6 since then. Turns out there're many incorrect implementations of Happy Eyeballs that cancel the ipv4 connection attempts after the timeout, and then switch to trying the AAAA records and subsequently throwing a "Cannot reach host" error. For context, in Happy Eyeballs you're supposed to continue trying both network families in parallel. This only impacts our customers who live far away from the region they're accessing, however, and there's usually a workaround - in Node you can force the network family to be v4 for instance
  - donavanm 1y ago
    > because AAAA records scare the olds No. They break existing customer expectations. There are heaps of dualstack API endpoints https://docs.aws.amazon.com/general/latest/gr/rande.html#dual-stack-endpoints https://docs.aws.amazon.com/general/latest/gr/rande.html#dua... if that's what the client wants. The amazonaws.com domain endpoints did not introduce ipv6/AAAA directly is (mostly) due to access control. For better or worse there are a lot of "v4 centric" IAM statements, like aws:SourceIp, in identity/resource/bucket policies. Introducing a new v6 value is going to break all of those existing policies with either unexpected DENYs or, worse, ALLOWs. Thats a pretty poor customer experience to unexpectedly break your existing infrastructure or compromise your access control intentions. AWS _could_ have audited every potential IAM policy and run a MASSIVE outreach campaign, but something as simple as increasing (opaque!) instance ID length was a multi year effort. And introducing backwards compatibility on a _per policy_ basis is its own infinite security & UX yak shaving exercise as well. So thats why you have opt-in usage of v6/dualstack in the client/SDK/endpoint name.
lysace 1y ago
Paid AWS support got a lot less capable on average during these two decades . :/ My recent interactions with them would probably have been better if they were an LLM.
- oblio 1y ago
  They probably are an LLM and if they aren't, their higher management is pushing for them to be LLMs by 2027 at the latest.
  - lysace 1y ago
    I still get strong Hyderabad vibes from the copy/pasting plus the occasional original sentence. Perhaps they trained the LLM using that data though. (Small customer though: yearly AWS spend around 80k. Support is 10% of that.)
    - count 1y ago
      TAMs are super hit and miss. We’ve had great ones (hi Nick!) and not so great ones. ($7-10M/mo customer AWS spend, support is a complicated sliding scale % of that, gogo ES!). Non-ES at smaller customers has been universally useless, except at quota increases.
      - mdaniel 1y ago
        If you're pushing 10 mil per month through AWS, I pray you have some bitcoin miners hidden in that spend. The mind reels
        count 1y ago
        Multiple large scale SAP S4/HANA workloads wasting electrons…
        lysace 1y ago
        At this scale it may make sense to run your own mini data centers?
raffraffraff 1y ago
> Availability Zones used to be randomized between accounts (my us-east-1a was your us-east-1c) WTH?
- slashdev 1y ago
  Yeah this one drove me crazy
- skywhopper 1y ago
  They did this to stop people from overloading us-east-1a. It was fine, until there started to be ways of wiring up networks between accounts (eg PrivateLink endpoint services) and you had to figure out which AZ was which so you could be sure you were mapping to the the same AZs in each account. I built a whole methodology for mapping this out across dozens of AWS accounts, and built lookup tables for our internal infrastructure… and then AWS added the zone ID to AZ metadata so that we could just look it up directly instead.
- mpyne 1y ago
  Presumably it would help ensure that everyone selecting us-east-1a in their base configs didn't actually all land in the same AZ.
- zbentley 1y ago
  It was for spreading load out. If someone was managing resources in a bunch of accounts and always defaulted to, say, 1b, AWS randomized what AZs corresponded to what datacenter segments to avoid hot spots. The canonical AZ naming was provided because, I bet, they realized that the users who needed canonical AZ identifiers were rarely the same users that were causing hot spots via always picking the same AZ.
  - Twirrim 1y ago
    Almost everyone went with 1a, every time. It causes significant issues for all sorts of reasons, especially considering the latency target for network connections between data centres in an AD
the8472 1y ago
> As of very recently, you can also force EC2 instances to stop or terminate without waiting for a clean shutdown or a ridiculous timeout Not true for GPU instances, they're stuck 5 minutes in a stopping state because they run some GPU health checks.
- latchkey 1y ago
  For our VM solution, we get around this by hot staging VM's. As soon as one customer stops theirs, we reset everything and start it up again. To the end user, our compute seems to be instantly available. Unless of course, we run out.
biimugan 1y ago
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. From my understanding, I don't think this is completely accurate. But, to be fair, AWS doesn't really document this very well. From my (informal) conversations with AWS engineers a few months ago, it works approximately like this (modulo some details I'm sure the engineers didn't really want to share): S3 requests scale based on something called a 'partition'. Partitions form automatically based on the smallest common prefixes among objects in your bucket, and how many requests objects with that prefix receive. And the bucket starts out with a single partition. So as an example, if you have a bucket with objects "2025-08-20/foo.txt" and "2025-08-19/foo.txt", the smallest common prefix is "2" (or maybe it considers the root as the generator partition, I don't actually know). (As a reminder, a / in an object key has no special significance in S3 -- it's just another character. There are no "sub-directories"). Therefore a partition forms based on that prefix. You start with a single partition. Now if the object "2025-08-20/foo.txt" suddenly receives a ton of requests, what you'll see happen is S3 throttle those requests for approximately 30-60 minutes. That's the amount of time it takes for a new partition to form. In this case, the smallest common prefix for "2025-08-20/foo.txt" is "2025-08-2". So a 2nd partition forms for that prefix. (Again, the details here may not be fully accurate, but this is the example conveyed to me). Once the partition forms, you're good to go. But the key issue here with the above situation is you have to wait for that warm up time. So if you have some workload generating or reading a ton of small objects, that workload may get throttled for a non-trivial amount of time until partitions can form. If the workload is sensitive to multi-minute latency, then that's basically an outage condition. The way around this is that you can submit an AWS support ticket and have them pre-generate partitions for you before your workload actually goes live. Or you could simulate load to generate the partitions. But obviously, neither of these is ideal. Ideally, you should just really not try and store billions of tiny objects and expect unlimited scalability and no latency. For example, you could use some kind of caching layer in front of S3.
- cmcarthur 1y ago
  This is my understanding too, and this is particularly problematic for workloads that are read/write heavy on very recent data. When partitioning by a date or by an auto-incrementing id, you still run into the same issue. Ex: your prefix is /id=12345. S3, under the hood, generates partitions named `/id=` and `/id=1`. Now, your id rolls over to `/id=20000`. All read/write activity on `/id=2xxxx` falls back to the original partition. Now, on rollover, you end up with read contention. For any high-throughput workloads with unevenly distributed reads, you are best off using some element of randomness, or some evenly distributed partition key, at the root of your path.
danpalmer 1y ago
Lots of this seems to boil down to: AWS shipped something that was barely usable, but then iterated. That's a reasonable approach, but the fact this post exists shows that this practice is a reputational risk. By all means do this if you think it's the right thing to do, but be aware that first impressions matter and will stick for a long time.
abullinan 1y ago
This article was a relief. I’m always a tiny bit worried Amazon will change some thing drastically and I’ll have to migrate. I’ve had an ec2 instance running since 2013. It requires effectively zero maintenance. So I am glad there were no surprises in this article. Thanks OP.
bambax 1y ago
Self-hosting is the future. If you have small-to-modetate traffic it's extremely easy to do and very cheap. Then you can self-cloud. Several startips are in this space. It gets you the best of both worlds: scaling, freedom, cost-control. And no marketing jargon that you need to learn, and then unlearn!
albert_e 1y ago
A "Catch me up" on AWS (and for that matter other large platforms) would be very useful for many folks. Ideally it should be a stream of important updates that can be interactively filtered by time-range. For example, if I have not been actively consuming AWS updates firehose for last 18 months, I should be able to "summarize" that length of updates. Why this is not already a feature of "What's New" section of AWS and other platforms -- I dont know. Waiting to be built -- either by OEM or by the Community.
- ash_091 1y ago
  I played a lot of DOTA2 in the past and I've often thought that big tech could learn something from Valve's patch notes. Especially in the context of process changes, stuff you should know, etc. Expecting folk to read a series of lengthy emails/blog posts to stay up to date is unrealistic.
selinkocalar 1y ago
Every AWS update can potentially affect your SOC 2 or HIPAA compliance posture. I've seen companies fail audits because they assumed their security configurations were still current. The cloud moves fast. Compliance processes need to keep up. Manual annual reviews aren't enough when your infrastructure is changing constantly. This is also why we built automated compliance monitoring - because what worked last quarter might not work today.

387 points by keithly 1y ago

> DynamoDB You can now have empty fields When this was changed? I think this is still an issue, I've had some such errors quite recently.

bluecassette 1y ago
18th May 2020 > DynamoDB now supports empty values for non-key String and Binary attributes in DynamoDB tables. Empty value support gives you greater flexibility to use attributes for a broader set of use cases without having to transform such attributes before sending them to DynamoDB. List, Map, and Set data types also support empty String and Binary values. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DocumentHistory.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

cldcntrl 1y ago

> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. Not strictly true.

rthnbgrredf 1y ago
Elaborate.
- cldcntrl 1y ago
  The whole auto-balancing thing isn't instant. If you have a burst of writes with the same key prefix, you'll get throttled.
- hnlmorg 1y ago
  Not the OP but I’ve had AWS-staff recommend different prefixes even as recently as last year. If key prefixes don’t matter much any more, then it’s a very recent change that I’ve missed.
  - cldcntrl 1y ago
    That's right, same for me as of only a few months ago.
  - williamdclt 1y ago
    Might just be that the AWS staff wasn't up to date on this
    - hnlmorg 1y ago
      That’s possible but they did consult with the storage team prior to our consultation. But I don’t know what conversations did or did not happen behind the scenes.
    - rthnbgrredf 1y ago
      By the way, that happens quite frequently. I regularly ask them about new AWS technologies or recent changes, and most of the time they are not aware. They usually say they will call back later after doing some research.
    - time0ut 1y ago
      I have had the same experience within the last 18 months. The storage team came back to me and asked me to spread my ultra high throughput write workload across 52 (A-Za-z) prefixes and then they pre-partitioned the bucket for me. S3 will automatically do this over time now, but I think there are/were edge cases still. I definitely hit one and experienced throttling at peak load until we made the change.
      - hnlmorg 1y ago
        That’s sounds like the problem we were having. Lots of writes to a prefix over a short period of time and then low activity to it after about 2 weeks.
QuinnyPig 1y ago
I should have been more clear. You still need to partition, but randomizing the prefixes hasn't been needed since 2018: https://web.archive.org/web/20240227073321/https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ https://web.archive.org/web/20240227073321/https://aws.amazo...
ed_g 1y ago
Generally speaking this isn't something Amazon S3 customers need to worry about - as others have said, S3 will automatically scale index performance over time based on load. The challenge primarily comes when customers need large bursts of requests within a namespace that hasn't had a chance to scale - that's when balancing your workload over randomized prefixes is helpful. Please see the documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi... This 2024 re:Invent session "Optimizing storage performance with Amazon S3 (STG328)" which goes very deep on the subject: https://www.youtube.com/watch?v=2DSVjJTRsz8 https://www.youtube.com/watch?v=2DSVjJTRsz8 And this blog that discusses Iceberg's new base-2 hash file layout which helps optimize request scaling performance of large-scale Iceberg workloads running on S3: https://aws.amazon.com/blogs/storage/how-amazon-ads-uses-iceberg-optimizations-to-accelerate-their-spark-workload-on-amazon-s3/ https://aws.amazon.com/blogs/storage/how-amazon-ads-uses-ice...
vvoyer 1y ago
This 2024 re:Invent session says exactly the opposite: "If you want to partition your data even better, you can introduce some randomness in your key names": https://youtu.be/2DSVjJTRsz8?t=2206 https://youtu.be/2DSVjJTRsz8?t=2206 FWIW The optimal way we were told was to partition our data was to do this: 010111/some/file.jpg. Where `010111/` is a random binary string which will please both the automatic partitioning (503s => partition) and manual partitioning you could ask AWS. Please as in the cardinality of partitions grows slower at each characters vs prefixes like `az9trm/`. We were told that the later version makes manual partitioning a challenge because as soon as you reach two characters you've already created 36x36 partitions (1,296). The issue with that: your keys are no more meaningful if you're relying on S3 to have "folders" by tenants for example (customer1/..).

SOLAR_FIELDS 1y ago

You know what's still stupid? That if you have an S3 bucket in the same region as your VPC that you will get billed on your NAT Gateway to send data out to the public internet and right back in to the same datacenter. There is simply no reason to not default that behavior to opt out vs opt in (via a VPC endpoint) beyond AWS profiting off of people's lack of knowledge in this realm. The amount of people who would want the current opt-in behavior is... if not zero, infinitesimally small.

deleted 1y ago
[deleted]
afandian 1y ago
Having experienced the joy of setting up VPC, subnets and PrivateLink endpoints the whole thing just seems absurd. They spent the effort of branding private VPC endpoints "PrivateLink". Maybe it took some engineering effort on their part, but it should be the default out of the box, and an entirely unremarkable feature. In fact, I think if you have private subnets, the only way to use S3 etc is Private Link (correct me if I'm wrong). It's just baffling.
- time0ut 1y ago
  You can provision gateway endpoints for S3 and DynamoDB. They are free and considered best practice. They are opt-in though, but easy to enable.
  - mdaniel 1y ago
    And ECR, which I would guess impacts more folks than DynamoDB https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html#ecr-setting-up-vpc-create https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-e... And, as as added benefit, they distinguish between "just pull" and "pull and push" which is nice
  - afandian 1y ago
    True, I forgot that. But depending on services you still have to have some Gateway and some Interface endpoints.

simonw 1y ago

S3: "Block Public Access is now enabled by default on new buckets." On the one hand, this is obviously the right decision. The number of giant data breeches caused by incorrectly configured S3 buckets is enormous. But... every year or so I find myself wanting to create an S3 bucket with public read access to I can serve files out of it. And every time I need to do that I find something has changed and my old recipe doesn't work any more and I have to figure it out again from scratch!

reactordev 1y ago
They'll teach you how for $250 and a certification test...
SOLAR_FIELDS 1y ago
I honestly don't mind that you have to jump through hurdles to make your bucket publically available and that it's annoying. That to me seems like a feature, not a bug
- simonw 1y ago
  Sure... but last time I needed to jump through those hurdles I lost nearly an hour to them! I'm still not sure I know how to do it if I need to again.
- dghlsakjg 1y ago
  I think the OPs objection is not that hurdles exist but that they move them every time you try and run the track.
crinkly 1y ago
I just stick CloudFront in front of those buckets. You don't need to expose the bucket at all then and can point it at a canonical hostname in your DNS.
- hnlmorg 1y ago
  That’s definitely the “correct” way of doing things if you’re writing infra professionally. But I do also get that more casual users might prefer not to incur the additional costs nor complexity of having CloudFront in front. Though at that point, one could reasonably ask if S3 is the right choice for causal users.

aaronblohowiak 1y ago

>VPC peering used to be annoying; now there are better options like Transit Gateway, VPC sharing between accounts, resource sharing between accounts, and Cloud WAN. TGW is... twice as expensive as vpc peering?

alFReD-NSH 1y ago
And vpc sharing is free. Cost and architecture are tied.
Hikikomori 1y ago
More than twice as same AZ is free with peering. But if you're big enough you can get better deals on cost. But unlike peering TGW traffic flows through an additional compute layer so it has additional cost.
klysm 1y ago
VPC sharing is the sleeper here. You can do cross account networking all in the same VPC and skip all the expensive stuff.
- aaronblohowiak 1y ago
  as long as your VPCs aren't too big, yea.
  - Hikikomori 1y ago
    Shared vpcs can get pretty big. Even if you approach the NAU limit you can use privatelink or TGW to have more large shared vpcs.
  - klysm 1y ago
    If you are exhausting an entire VPC I’d be pretty impressed!
    - aaronblohowiak 1y ago
      Used to work at nflx, biiiiig headaches

Ayesh 1y ago

CloudFront also has 1TB of free data transfer a month under the forever-free perks.

csours 1y ago

Strictly off topic: Everything you know is wrong. Weird Al. https://www.youtube.com/watch?v=W8tRDv9fZ_c https://www.youtube.com/watch?v=W8tRDv9fZ_c Firesign Theatre. https://www.youtube.com/watch?v=dAcHfymgh4Y https://www.youtube.com/watch?v=dAcHfymgh4Y

gurjeet 1y ago

It would've been nice if each of those claims in the article also linked to either the relevant announcement or to the documentation. If I'm interested in any of these headline items, I'd like to learn more.

mdaniel 1y ago
I don't believe AWS offers permalinks, so it would only help until they rolled over the next documentation release :-( They actually used to have the upstream docs in GitHub, and that was super nice for giving permalinks but also building the docs locally in a non-pdf-single-file setup. Pour one out, I guess

bob1029 1y ago

> Glacier restores are also no longer painfully slow. Wouldn't this always depend on the length of the queue to access the robotic tape library? Once your tape is loaded it should move really quickly: https://www.ibm.com/docs/en/ts4500-tape-library?topic=performance-lto-specifications https://www.ibm.com/docs/en/ts4500-tape-library?topic=perfor...

hinkley 1y ago
> Once upon a time Glacier was its own service that had nothing to do with S3. If you look closely (hi, billing data!) you can see vestiges of how this used to be, before the S3 team absorbed it as a series of storage classes. Your assumption holds if they still use tape. But this paragraph hints at it not being tape anymore. The eternal battle between tape versus drive backup takes another turn.
- bob1029 1y ago
  I am also assuming that Amazon intends for the Deep Archive tier to be a profitable offering. At $0.00099/gb-month, I don't see how it could be anything other than tape.
  - simonw 1y ago
    I wonder if it's where old S3 hard drives go to die? Presumably AWS have the world's single largest collection of used storage devices - if you RAID them up you can probably get reliable performance out of them for Glacier?
    - bob1029 1y ago
      I still don't know if it's possible to make it profitable with old drives in this kind of arrangement, especially if we intend to hit their crazy durability figures. The cost of keeping drives spinning is low, but is double-digit margin % in this context. You can't leave drives unpowered in a warehouse for years on end and say you have 11+ nines of durability.

chisleu 1y ago

> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. As of when? According to internal support, this is still required as of 1.5 years ago.

laurent_du 1y ago
He's not talking about the prefix, just the beginning of the object key.
- viccis 1y ago
  The prefix is not separate from the object key. It's part of it. There's no randomization that needs to be done on either anymore.
  - chisleu 1y ago
    and indeed the bucket is not separate from the object key. the API separates it logically "for humans" but it's all one big string
arpinum 1y ago
I think there is some nuance needed here. If you ask support to partition your bucket then they will be a bit annoying if you ask for specific partition points and the first part of the prefix is not randomised. They tried to push me to refactor the bucket first to randomise the beginning of the prefix, but eventually they did it. The auto partitioning is different. It can isolate hot prefixes on its own and can intelligently pick the partition points. Problem is the process is slow and you can be throttled for more than a day before it kicks in.
- chisleu 1y ago
  > but eventually they did it They can do this with manual partitioning indeed. I've done it before, but it's not ideal because the auto partitioner will scale beyond almost anything AWS will give you with manual partitioning unless you have 24/7 workloads. > you can be throttled for more than a day before it kicks in I expect that this would depends on your use case. If you are dropping content you need to scale out to tons of readers, that is absolutely the case. If you are dropping tons of content with well distributed reads, then the auto partitioner is The Way.

scubbo 1y ago

I've had two people tell me in the last week that SQS doesn't support FIFO queues.

TheP1000 1y ago

API gateway timeout increase has been nice.

coredog64 1y ago
It was always there but it required much more activity to get it done (document your use case & traffic levels and then work with your TAM to get the limit changed).
quesera 1y ago
I don't see that in this post. I just started working with a vendor who has a service behind API Gateway. It is a bit slow(!) and times out at 30 seconds. I've since modified my requests to chunk subsets of the whole dataset, to keep things under the timeout. Has this changed? Is 30 secs the new or the old timeout?
- TheP1000 1y ago
  https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-api-gateway-integration-timeout-limit-29-seconds/ https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-ap...

Annatar 1y ago

[dead]

stevejb 1y ago

I just saw Weird Al in concert, and one of my favorite songs of his is "Everything You Know is Wrong." This is the AWS version of that song! Nice work Corey!

havefunbesafe 1y ago
Weird AL or Weird A.I.?
jeffbarr 1y ago
I also saw that concert. Oh yeah, we were in the same row!

nodesocket 1y ago

I'll add: When doing instance to instance communication (in the same AZ) always use private ips. If you use public ip routing (even the same AZ) this is charged as regional data transfer. Even worse, if you run self hosted NAT instance(s) don't use a EIP attached to them. Just use a auto-assigned public IP (no EIP). NAT instance with EIP - AWS routes it through the public AWS network infrastructure (hairpinning). - You get charged $0.01/GB regional data transfer, even if in the same AZ. NAT instance with auto-assigned public IP (no EIP) - Traffic routes through the NAT instance’s private IP, not its public IP. - No regional data transfer fee — because all traffic stays within the private VPC network. - auto-assigned public IP may change if the instance is shutdown or re-created so have automations to handle that. Though you should be using the network interface ID reference in your VPC routing tables.

themafia 1y ago
> You get charged $0.01/GB regional data transfer, even if in the same AZ. My understanding is that transfer gets charged on both sides as well. So if you own both sides you'll pay $0.02/GB.
- pcthrowaway 1y ago
  Is transfer into the instance not always free anymore?

digianarchist 1y ago

Would love an AWS equivalent to Cloud Run but the lambda changes are welcome nonetheless.

aranelsurion 1y ago
Isn't Fargate the AWS equivalent of Cloud Run?
- mdaniel 1y ago
  It seems Cloud Run comes "batteries included" for a lot more things that one must provision separate AWS resources to achieve https://cloud.google.com/architecture/migrate-aws-lambda-to-cloudrun#:~:text=to%20help%20you%20understand%20which%20cloud%20run%20invocation%20or%20execution%20mechanisms%20are%20comparable%20to%20aws%20lambda%20invocation%20mechanisms https://cloud.google.com/architecture/migrate-aws-lambda-to-...

topher200 1y ago

I have a preempt-able workload for which I could use Spot instances or Savings Plans. Does anyone have experience running Spot in 2025? If you were to start over, would you keep using Spot? - I observe with pricing that Spot is cheaper - I am running on three different architectures, which should limit Spot unavailability - I've been running about 50 Spot EC2 instances for a month without issue. I'm debating turning it on for many more instances

erulabs 1y ago
In terms of cost, from cheapest to most expensive: 1. Spot with autoscaling to adjust to demand and a savings plan that covers the ~75th percentile scale 2. On-demand with RIs (RIs will definitely die some day) 3. On-demand with savings-plans (More flexible but more expensive than RIs) 3. Spot 4. On-demand I definitely recommend spot instances. If you're greenfielding a new service and you're not tied to AWS, some other providers have hilariously cheap spot markets - see http://spot.rackspace.com/ http://spot.rackspace.com/. If you're using AWS, definitely auto-scaling spot with savings plans are the way to go. If you're using Kubernetes, the AWS Karpenter project (https://karpenter.sh/ https://karpenter.sh/) has mechanisms for determining the cheapest spot price among a set of requirements. Overall tho, in my experience, ec2 is always pretty far down the list of AWS costs. S3, RDS, Redshift, etc wind up being a bigger bill in almost all past-early-stage startups.
- mdaniel 1y ago
  To "me, too" this, it's not like that AWS spot instance just go "poof," they do actually warn you (my recollection is 60s in advance of the TerminateInstance call), and so a resiliency plane on top of the workloads (such as the cited Kubernetes) can make that a decided "non-event". Shout out to the reverse uptime crew, a subset of Chaos Engineering
  - cptnntsoobv 1y ago
    The spot termination warnings arrive 120 seconds before termination https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-ins...

general1726 1y ago

I think there is more of us who kind of degenerated from doing it the AWS way - API Gateway, serverless lambdas mess around with IAM roles until it works, ... - to - Give me EC2 / LightSail VPS instance maybe an S3 bucket let's set domain through Route53 and go away with the rest of your orchestrion AWS.

calmbonsai 1y ago
There are entire industries that have largely de-volved their clouds primarily for footprint flexibility (not all AWS services are in all regions) and billing consistency.
regularfry 1y ago
Honestly just having to manage IAM is such a time-suck that the way I've explained it to people is that we've traded the time we used to spend administering systems for time spent just managing permissions, and IAM is so obtuse that it comes out as a net loss. There's a sweet spot somewhere in between raw VPSes and insanely detailed least-privilege serverless setups that I'm trying to revert to. Fargate isn't unmanageable as a candidate, not sure it's The One yet but I'm going to try moving more workloads to it to find out.
- sklarsa 1y ago
  Usually I write some IaC to automate this tedium so I only have to go through the IAM setup pain once. Now if requirements change, that's an entirely different story...
  - regularfry 1y ago
    So the problem when you combine IAC with CI/CD is that the role assumed by the CI agent needs privileges to deploy things, so you need a bootstrap config to set up what it needs. If you have a mandate to go least-privilege, then that needs to include only the permissions strictly needed by the current deployable. So, no "s3:*", you need each one listed. So far so good, you can do this with a bootstrap script that you only need to run at project setup. If you also have a mandate (effectively) to go fully serverless, then as your project evolves and you add functionality, what you find is that most interesting changes use something new in the platform. So you're not getting away with running the bootstrap script once. You're updating it and running it for almost every change. And you can't tell in advance what permissions you're going to need, because (especially if you're on terraform) there's apparently no documentation connecting the resources you want to manage and the permissions needed to do so. So you try to deploy your change, IAM pops an error or two, you try to figure out what permissions you need to add to the bootstrap script, you run it (fixing it when it breaks at this point), you try deploying again, IAM pops another couple of errors, and then you're in a grind cycle which you can't predict the length of - and you need to get to the end of it before you can even test your feature, because fully serverless means you can't run your application locally (and getting management to pay for the pro localstack licence is a dead end). At some point it won't be clear why IAM is complaining, because the error you get makes no sense whatsoever, so at that point it's off to support to find out a day later that ah, yes, you can't use an assumed role just there, it's got to be an actual role, and no, that's not written down anywhere, you've just got to know it, so you need to redesign how you're using the roles completely, and right about this point is when I usually want to buy a farm, raise goats, and get way too into oil painting, instead of whatever this insane waste of life is.

JCM9 1y ago

Some good stuff here. I wish AWS would just focus on these boring, but ultimately important, things that they’re good at instead of all the current distractions trying to play catch up on “AI.” AWS leadership missed the boat there big time, but that’s OK. Ultimately AWS doesn’t have the right leadership or talent to be good at GenAI, but they do (or at least used to) have decent core engineers. I’d like to see them get back to basics and focus there. Right now leadership seems panicked about GenAI and is just throwing random stuff at the wall desperately trying to get something to stick. Thats really annoying to customers.

PartiallyTyped 1y ago
Leadership is looking to provide infrastructure for anyone to just pick a model and get on doing things without the hassle of setting things up.
anon7000 1y ago
They continue to have large teams working on core stuff. It’s just that they’re working at such a low level (like high perf virtualized networking on their custom network cards) that most people don’t hear about it or care that much.

causal 1y ago

This is super helpful. I would read a yearly summary like this.

beaviskhan 1y ago

Also S3 related: the bucket owner can now be configured as the object owner no matter where the object originated. In the past this was exceedingly painful if you wanted to allow one account contribute objects to a bucket in another account. You could do the initial contribution, but the contributor always owned the object, and you couldn't delegate access to a third account.

viccis 1y ago

>In EC2, you can now change security groups and IAM roles without shutting the instance down to do it. Hasn't it been this way for many years? >Spot instances used to be much more of a bidding war / marketplace. Yeah because there's no bidding any more at all, which is great because you don't get those super high spikes as availability drops and only the ones who bid super high to ensure they wouldn't be priced out are able to get them. >You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. This one was a nightmare and it took ages to convince some of my more pig headed coworkers in the past that they didn't need to do it any more. The funniest part is that they were storing their data as millions and millions of 10-100kb files, so the S3 backend scaling wasn't the thing bottlenecking performance anyway! >Originally Lambda had a 5 minute timeout and didn’t support container images. Now you can run them for up to 15 minutes, use Docker images, use shared storage with EFS, give them up to 10GB of RAM (for which CPU scales accordingly and invisibly), and give /tmp up to 10GB of storage instead of just half a gig. This was/is killer. It used to be such a pain to have to manage pyarrow's package size if I wanted a Python Lambda function that used it. One thing I'll add that took me an embarrassingly long time to realize is that your Python global scope is actually persisted, not just the /tmp directory.

indigodaddy 1y ago
Re: SG, yeah I wasnt doing any cloud stuff when that was the case. Never had to restart anything for an SG change and this must be at least 5-6 years..
- buzzdenver 1y ago
  IAM Role change is more recent though
Inufu 1y ago
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. Sorry, this is absolutely still the case if you want to scale throughput beyond the few thousand IOPS a single shard can serve. S3 will automatically reshard your key space, but if your keys are sequential (eg leading timestamp) all your writes will still hit the same shard. Source: direct conversations with AWS teams.

PaulDavisThe1st 1y ago

Can no longer login to my AWS account, because I never set up MFA. Want to set up MFA ... login required to request device. Yes, I know, they warned us far ahead of time. But not being able to request one of their MFA devices without a login is ... sucky.

berlesi 1y ago
Looks like something that you could solve easily through their support, no?
- raffraffraff 1y ago
  Support don't talk to you unless you pay for support
  - danudey 1y ago
    Easy, just log into your account and pay them for support.
  - austinshea 1y ago
    Yes, they do. https://support.aws.amazon.com/#/contacts/one-support?formId=contactUs https://support.aws.amazon.com/#/contacts/one-support?formId...
austinshea 1y ago
Talk to support: https://support.aws.amazon.com/#/contacts/one-support?formId=contactUs https://support.aws.amazon.com/#/contacts/one-support?formId...
- PaulDavisThe1st 1y ago
  I did. That's just an AI, which says this: > I understand your situation is a bit unique, where you are unable to log in to your AWS account without an MFA device, but you also can't order an MFA device without being able to log in. This is a scenario that is not directly covered in our standard operating procedures. The best course of action would be for you to contact AWS Support directly. They will be able to review your specific case and provide guidance on how to obtain an MFA device to regain access to your account. The support team may have alternative options or processes they can walk you through to resolve this issue. Please submit a support request, and one of our agents will be happy to assist you further. You can access the support request form here: https://console.aws.amazon.com/support/home https://console.aws.amazon.com/support/home That last URL? You need to login to use it ...

jp57 1y ago

> Glacier restores are also no longer painfully slow. I had a theory (based on no evidence I'm aware of except knowing how Amazon operates) that the original Glacier service operated out of an Amazon fulfillment center somewhere. When you put it a request for your data, a picker would go to a shelf, pick up some removable media, take it back, and slot it into a drive in a rack. This, BTW, is how tape backups on timesharing machines used to work once upon a time. You'd put in a request for a tape and the operator in the machine room would have to go get it from a shelf and mount it on the tape drive.

browningstreet 1y ago
Yeah, but they've been robotic for decades since.
danudey 1y ago
The most likely explanation is that they used a tape robot, such as the one seen here: https://www.reddit.com/r/DataHoarder/comments/12um0ga/the_robot_managing_my_universitys_backup_each_of/ https://www.reddit.com/r/DataHoarder/comments/12um0ga/the_ro... Which is basically exactly what you described but the picker is a robot. Data requests go into a queue; when your request comes up, the robot looks up the data you requested, finds the tape and the offset, fetches the tape and inserts it into the drive, fast-forwards it to the offset, reads the file to temporary storage, rewinds the tape, ejects it, and puts it back. The latency of offline storage is in fetching/replacing the casette and in forwarding/rewinding the tape, plus waiting for an available drive. Realistically, the systems probably fetch the next request from the queue, look up the tape it's on, and then process every request from that tape so they're not swapping the same tape in and out twenty times for twenty requests.
- philistine 1y ago
  I've read very definitive discussions on here that Glacier never used tape. It has always been powered off hard disks.
  - UltraSane 1y ago

ElijahLynn 1y ago

That. Was a decent investment of my time as a devops engineer. Right to the point. I learned things.

kassner 1y ago

I haven’t used AWS in the last 5 years. Is IPv6 still somewhat of an issue? I remember some services not supporting it at all and making it impossible to manage as a IPv6-only network.

skywhopper 1y ago
Yeah, it’s still limited, and a few things still require at least a dual stack setup.
1oooqooq 1y ago
gotta milk those ipv4 investment
mdaniel 1y ago
You'll be glad(sic) to know they have specialized hostnames for IPv6 control plane endpoints, because AAAA records scare the olds https://github.com/aws/aws-cli/blob/2.28.14/awscli/botocore/data/iam/2010-05-08/endpoint-rule-set-1.json#L159 https://github.com/aws/aws-cli/blob/2.28.14/awscli/botocore/...
- conradludgate 1y ago
  We enabled ipv6 for our APIs at work. Nothing broke immediately, but we've had a steady stream of unreachable host errors related to ipv6 since then. Turns out there're many incorrect implementations of Happy Eyeballs that cancel the ipv4 connection attempts after the timeout, and then switch to trying the AAAA records and subsequently throwing a "Cannot reach host" error. For context, in Happy Eyeballs you're supposed to continue trying both network families in parallel. This only impacts our customers who live far away from the region they're accessing, however, and there's usually a workaround - in Node you can force the network family to be v4 for instance
- donavanm 1y ago
  > because AAAA records scare the olds No. They break existing customer expectations. There are heaps of dualstack API endpoints https://docs.aws.amazon.com/general/latest/gr/rande.html#dual-stack-endpoints https://docs.aws.amazon.com/general/latest/gr/rande.html#dua... if that's what the client wants. The amazonaws.com domain endpoints did not introduce ipv6/AAAA directly is (mostly) due to access control. For better or worse there are a lot of "v4 centric" IAM statements, like aws:SourceIp, in identity/resource/bucket policies. Introducing a new v6 value is going to break all of those existing policies with either unexpected DENYs or, worse, ALLOWs. Thats a pretty poor customer experience to unexpectedly break your existing infrastructure or compromise your access control intentions. AWS _could_ have audited every potential IAM policy and run a MASSIVE outreach campaign, but something as simple as increasing (opaque!) instance ID length was a multi year effort. And introducing backwards compatibility on a _per policy_ basis is its own infinite security & UX yak shaving exercise as well. So thats why you have opt-in usage of v6/dualstack in the client/SDK/endpoint name.

lysace 1y ago

Paid AWS support got a lot less capable on average during these two decades . :/ My recent interactions with them would probably have been better if they were an LLM.

oblio 1y ago
They probably are an LLM and if they aren't, their higher management is pushing for them to be LLMs by 2027 at the latest.
- lysace 1y ago
  I still get strong Hyderabad vibes from the copy/pasting plus the occasional original sentence. Perhaps they trained the LLM using that data though. (Small customer though: yearly AWS spend around 80k. Support is 10% of that.)
  - count 1y ago
    TAMs are super hit and miss. We’ve had great ones (hi Nick!) and not so great ones. ($7-10M/mo customer AWS spend, support is a complicated sliding scale % of that, gogo ES!). Non-ES at smaller customers has been universally useless, except at quota increases.
    - mdaniel 1y ago
      If you're pushing 10 mil per month through AWS, I pray you have some bitcoin miners hidden in that spend. The mind reels
      - count 1y ago
        Multiple large scale SAP S4/HANA workloads wasting electrons…
        lysace 1y ago
        At this scale it may make sense to run your own mini data centers?

raffraffraff 1y ago

> Availability Zones used to be randomized between accounts (my us-east-1a was your us-east-1c) WTH?

slashdev 1y ago
Yeah this one drove me crazy
skywhopper 1y ago
They did this to stop people from overloading us-east-1a. It was fine, until there started to be ways of wiring up networks between accounts (eg PrivateLink endpoint services) and you had to figure out which AZ was which so you could be sure you were mapping to the the same AZs in each account. I built a whole methodology for mapping this out across dozens of AWS accounts, and built lookup tables for our internal infrastructure… and then AWS added the zone ID to AZ metadata so that we could just look it up directly instead.
mpyne 1y ago
Presumably it would help ensure that everyone selecting us-east-1a in their base configs didn't actually all land in the same AZ.
zbentley 1y ago
It was for spreading load out. If someone was managing resources in a bunch of accounts and always defaulted to, say, 1b, AWS randomized what AZs corresponded to what datacenter segments to avoid hot spots. The canonical AZ naming was provided because, I bet, they realized that the users who needed canonical AZ identifiers were rarely the same users that were causing hot spots via always picking the same AZ.
- Twirrim 1y ago
  Almost everyone went with 1a, every time. It causes significant issues for all sorts of reasons, especially considering the latency target for network connections between data centres in an AD

the8472 1y ago

> As of very recently, you can also force EC2 instances to stop or terminate without waiting for a clean shutdown or a ridiculous timeout Not true for GPU instances, they're stuck 5 minutes in a stopping state because they run some GPU health checks.

latchkey 1y ago
For our VM solution, we get around this by hot staging VM's. As soon as one customer stops theirs, we reset everything and start it up again. To the end user, our compute seems to be instantly available. Unless of course, we run out.

biimugan 1y ago

> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. From my understanding, I don't think this is completely accurate. But, to be fair, AWS doesn't really document this very well. From my (informal) conversations with AWS engineers a few months ago, it works approximately like this (modulo some details I'm sure the engineers didn't really want to share): S3 requests scale based on something called a 'partition'. Partitions form automatically based on the smallest common prefixes among objects in your bucket, and how many requests objects with that prefix receive. And the bucket starts out with a single partition. So as an example, if you have a bucket with objects "2025-08-20/foo.txt" and "2025-08-19/foo.txt", the smallest common prefix is "2" (or maybe it considers the root as the generator partition, I don't actually know). (As a reminder, a / in an object key has no special significance in S3 -- it's just another character. There are no "sub-directories"). Therefore a partition forms based on that prefix. You start with a single partition. Now if the object "2025-08-20/foo.txt" suddenly receives a ton of requests, what you'll see happen is S3 throttle those requests for approximately 30-60 minutes. That's the amount of time it takes for a new partition to form. In this case, the smallest common prefix for "2025-08-20/foo.txt" is "2025-08-2". So a 2nd partition forms for that prefix. (Again, the details here may not be fully accurate, but this is the example conveyed to me). Once the partition forms, you're good to go. But the key issue here with the above situation is you have to wait for that warm up time. So if you have some workload generating or reading a ton of small objects, that workload may get throttled for a non-trivial amount of time until partitions can form. If the workload is sensitive to multi-minute latency, then that's basically an outage condition. The way around this is that you can submit an AWS support ticket and have them pre-generate partitions for you before your workload actually goes live. Or you could simulate load to generate the partitions. But obviously, neither of these is ideal. Ideally, you should just really not try and store billions of tiny objects and expect unlimited scalability and no latency. For example, you could use some kind of caching layer in front of S3.

cmcarthur 1y ago
This is my understanding too, and this is particularly problematic for workloads that are read/write heavy on very recent data. When partitioning by a date or by an auto-incrementing id, you still run into the same issue. Ex: your prefix is /id=12345. S3, under the hood, generates partitions named `/id=` and `/id=1`. Now, your id rolls over to `/id=20000`. All read/write activity on `/id=2xxxx` falls back to the original partition. Now, on rollover, you end up with read contention. For any high-throughput workloads with unevenly distributed reads, you are best off using some element of randomness, or some evenly distributed partition key, at the root of your path.

danpalmer 1y ago

Lots of this seems to boil down to: AWS shipped something that was barely usable, but then iterated. That's a reasonable approach, but the fact this post exists shows that this practice is a reputational risk. By all means do this if you think it's the right thing to do, but be aware that first impressions matter and will stick for a long time.

abullinan 1y ago

This article was a relief. I’m always a tiny bit worried Amazon will change some thing drastically and I’ll have to migrate. I’ve had an ec2 instance running since 2013. It requires effectively zero maintenance. So I am glad there were no surprises in this article. Thanks OP.

bambax 1y ago

Self-hosting is the future. If you have small-to-modetate traffic it's extremely easy to do and very cheap. Then you can self-cloud. Several startips are in this space. It gets you the best of both worlds: scaling, freedom, cost-control. And no marketing jargon that you need to learn, and then unlearn!

albert_e 1y ago

A "Catch me up" on AWS (and for that matter other large platforms) would be very useful for many folks. Ideally it should be a stream of important updates that can be interactively filtered by time-range. For example, if I have not been actively consuming AWS updates firehose for last 18 months, I should be able to "summarize" that length of updates. Why this is not already a feature of "What's New" section of AWS and other platforms -- I dont know. Waiting to be built -- either by OEM or by the Community.

ash_091 1y ago
I played a lot of DOTA2 in the past and I've often thought that big tech could learn something from Valve's patch notes. Especially in the context of process changes, stuff you should know, etc. Expecting folk to read a series of lengthy emails/blog posts to stay up to date is unrealistic.

selinkocalar 1y ago

Every AWS update can potentially affect your SOC 2 or HIPAA compliance posture. I've seen companies fail audits because they assumed their security configurations were still current. The cloud moves fast. Compliance processes need to keep up. Manual annual reviews aren't enough when your infrastructure is changing constantly. This is also why we built automated compliance monitoring - because what worked last quarter might not work today.