TheHydroImpulse

5 ms·

1.
by TheHydroImpulse 4y ago
CDK outputs CloudFormation templates that you can pass to any other tool if you wish. CDK is technically just a glorified shim over CFN (although 1000x better but you do get caught in the awkwardness that is CFN many times).
2.
by TheHydroImpulse 5y ago
still very different but amazon has its influences in many parts of the company now.
3.
by TheHydroImpulse 6y ago
If anyone notices a problem you'll likely need to write a COE, there's no way to get around that. Not updating the status page absolutely doesn't get you out of that task. COE also doesn't lead to negative marks on anyon
4.
by TheHydroImpulse 8y ago
yes, that's a really good way to grow your comp. You have actual leverage when joining a new company that you can use to negotiate. Obviously you can't really do it too often but whenever you do, your earning potential can be incr
5.
by TheHydroImpulse 8y ago
Companies usually like to downgrade when hiring. Amazon does it a lot. So even if you qualified as an L7 they may hire you as an L6 (sometimes with L7 pay) and you need to go through a promotion cycle to get back to L7.
6.
by TheHydroImpulse 8y ago
Github has been a well known Rails monolith. Absolutely they have services that power all sorts of stuff (they have a few blog posts on it) but it's just impractical to even start a discussion on splitting it up for the reasons you ini
7.
by TheHydroImpulse 8y ago
This just isn't possible. No company in their right mind would take their monolith and rewrite a bunch of stuff just to please a couple randoms on the internet. I think you're drastically underestimating the amount of code Github
8.
by TheHydroImpulse 8y ago
when you have to manage a large number of services with a bunch of different devs touching things, Docker is almost required to get a consistent development state. There's literally nothing wrong with reaching for Docker right away. It
9.
by TheHydroImpulse 8y ago
As someone without a degree, getting an interview at the Big 5 was quite easy. Passing those interviews is another thing entirely but having the right experiences in your resume has been more important in my case.
10.
by TheHydroImpulse 8y ago
L3 at other companies is an L4 at Amazon (I believe only contractors or part-time employees can be < 4). L5 also has a much wider surface area and the gap from L5 to L6 is quite big.
11.
by TheHydroImpulse 9y ago
The biggest issue with PubSub and Amazon's alternative is the cost. Being capped at a per-message cost would be a no go. If you can get away with using PubSub or the like it would be far easier than to manage your own Kafka deployment
12.
by TheHydroImpulse 9y ago
We started deploying our Kafka cluster as a set of N EC2 instances but we started running into a bunch of issues (rolling the cluster, rolling an instance without moving partitions around, moving partitions around, etc...) Now we run Kafka
13.
by TheHydroImpulse 9y ago
NSQ has been (IMO) far far easier to operate than Kafka has been. With Kafka you need a Zookeeper cluster in addition to your Kafka brokers. Not to mention developing against NSQ is pretty simple whereas Kafka you need to think about partit
14.
by TheHydroImpulse 9y ago
A bit of an older article but still useful: https://segment.com/blog/scaling-nsq/
15.
by TheHydroImpulse 9y ago
Engineer @ Segment NSQ has served us pretty well but long term persistence has been a massive concern to us. If any of our NSQ nodes go down it's a big problem. Kafka has been far more complicated to operate in production and developin
16.
by TheHydroImpulse 10y ago
A bunch of people at Segment have looked into it. We manage our infrastructure with Terraform and operationally IMO Kubernetes introduces too much complexity compared with ECS and Terraform. ECS is pretty dead simple (just run an agent on t
17.
by TheHydroImpulse 10y ago
One thing we experienced at Segment was the fact that we needed to quickly handle a surge in volume but couldn't overload partners. Essentially we wanted something that scaled up quickly at first but was pretty conservative after that.
18.
by TheHydroImpulse 10y ago
CPU + memory autoscaling has worked pretty well for most services at Segment. We did have to build a custom queue-based autoscaler that feeds into ECS for our integration stack but the limitations of the ECS autoscaling rules makes it a bit
19.
by TheHydroImpulse 10y ago
At a high enough scale, you have a lock-in _somewhere_. Spending time trying to abstract yourself from any lock-in can be wasteful.
20.
by TheHydroImpulse 10y ago
I'm talking about _my_ use case for using AWS. I'm sure other people have similar requirements. We manage hundreds of servers, process over 50 billion events/month and losing data is unacceptable. In the HN echo chamber, you
21.
by TheHydroImpulse 10y ago
a) I don't really have a good enough sample size but I'd imagine a lot don't. The biggest selling point of AWS is everything around it. You don't just get EC2, you get Route53, ELB, VPC, RDS, S3, CloudFront (although it&
22.
by TheHydroImpulse 10y ago
Unfortunately, mixing and matching ends up really complicating things especially with security in mind. Many people run within a VPC and bridging to another private network is, well, I don't really want to think about it at this time.
23.
by TheHydroImpulse 10y ago
Oh, for sure. A lot of startups don't need that ability. I work for a pretty infra heavy startup so AWS is simply required at this point. But we've hit AWS capacity limits during the worst times (one of our clusters processing 20k
24.
by TheHydroImpulse 10y ago
Until you need to add another server, or 10 or 100... Not to mention it's another set of skills you need to have. It's a tradeoff. (I'm not talking about "we may need 100 servers next year because we'll have all thi
25.
by TheHydroImpulse 10y ago
Latency is around 80ms-200ms per invocation from what I've seen. Definitely not suitable for any real-time game.
26.
by TheHydroImpulse 10y ago
You might have already seen this but you might want to check out [1] for inspiration. [1]: https://github.com/segmentio/stack It came out of our infrastructure at Segment and works really well.
27.
by TheHydroImpulse 10y ago
But co-locating them won't actually remove a class of errors because Zk is not HA. The Kafka brokers need to communicate with the leader in the Zk cluster. If we have K1,Z1 -- K2,Z2 -- K3,Z3 -- and one node goes down, you've now t
28.
by TheHydroImpulse 10y ago
FYI, Kafka doesn't need to fetch from disk every time as it caches the logs pretty aggressively, as long as you have enough memory. Running Zk and Kafka on the same nodes is likely not the best thing.
29.
by TheHydroImpulse 10y ago
> dozens of repos ha. I wish I had this many repos. We have 1000+ git repos. To be fair, a ton are open sourced and there are reasons why it's done this way, but still.
30.
by TheHydroImpulse 10y ago
It's not a new implementation so much as compiling Swift for different architectures and environments not supported by the compiler. It still uses the Swift compiler but emits LLVM IR and then uses Clang to target to the different envi

More ›