7 ms·

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a c

• dondraper36 1y ago
  Yes, and this is exactly why LinkedIn-driven development exists in the first place. Listing a million technologies looks much more impressive on paper to recruiters than describing how you managed to only use a modular monolith and a single Postgres instance to make everything work.
• ramraj07 1y ago
  Do you _want_ to work in these places? In my experience, if they expect you to run kube using kube in the interview, thats exactly what they do in their ststems as well.
  • UK-AL 1y ago
    These are the places that actually pay well.
    • dondraper36 1y ago
      There's another reason for that. Deep in my heart, I would love to be part of a team that works on truly data-intensive applications (as Martin Kleppmann would call them) where all the complexity is justified. For example, I am more of the "All you need is Postgres" kind of software engineer. But reading all those fancy blog posts on how some team at Discord works with 1 trillion messages with Cassandra and ScyllaDB makes me envious. Also, it seems that to be hired by such employers you need to prove that you already have such experience, which is a bit of a catch-22 situation.
      • stavros 1y ago
        I feel like the phrase "all you need is Postgres" has the (often unspoken) continuation of "until you actually get to a trillion messages". In other words, the developers you're envious of didn't start with Cassandra and ScyllaDB, they started with the problem of too many messages. That's not an architectural choice, that's product success.
        • dondraper36 1y ago
          Absolutely. To put it differently, unfortunately not everyone has a chance to be part of a product's organic evolution from "all we need is Postgres" to "holy crap, we're a success, what is Cassandra by the way?"
          • SatvikBeri 1y ago
            As a data point, I've been at two data-intensive startups where they eventually needed to pull (some) of their table-like data out of postgres, and for both that was past a $100MM valuation. This varies by domain of course, but non-postgres solutions are generally built for very specific problems – they're worse than postgres at everything except one or two cases.
      • DanielHB 1y ago
        Only places that are making good money can afford to have overengineering. Overengineering is more prevalent the more money a company makes and companies who overengineers will pay good money to keep the overengineering working.
        • no_wizard 1y ago
          Something about my old CTO and VP of Eng I respected is they were still technical enough to call out this kind of thing. For as big as that company was they really held down complexity and overengineering to a real minimum. Unfortunately the rest of the executive has leaned on them so hard about AI boosting productivity they aren’t able to avoid thst becoming a mess
    • paulddraper 1y ago
      Usually out of necessity
• zellyn 1y ago
  I recently had an interview like this. Felt like half the answers I gave were of the form, “You can do scaling/sharding/partitioning thing X here, but once again, for an internal app I’d try really hard to avoid doing any of that”. If you’re interviewing with capable, experienced developers, they’ll appreciate that answer (at least, I got the offer on this one!)
• Swizec 1y ago
  > These are not the answers they're looking for. These ARE the answers we are looking for. As the system design interview (I’ve done hundreds) I want you to start with these answers then we can layer on complexity if you’ve solved the problem and there’s time left to go into navel gazing mode. Seeing the panic slowly build in mid-level engineers’ eyes as it dawns on them that not every problem can be solved by caching is pretty fun too. “Ok cool you’ve cached it there, now how do you fill the cache without running into the same performance issue?”
  • dondraper36 1y ago
    This also happens because plenty of candidates learn the buzzwords and patterns without understanding the trade-offs and nuances. With a competent enough interviewer, the shallowness of knowledge can be revealed immediately.
    • Aurornis 1y ago
      Identifying candidates who repeat buzzwords without understanding tradeoffs is easy. It’s part of the questioning process to understand the tradeoffs. The problem with the comment above is that it’s not discussing tradeoffs at all. It’s just jumping to conclusions and dodging any discussion of tradeoffs. If you answer questions like that, it’s impossible to tell if the candidate is being wise or if they’re simply BSing their way around the topic and pretending to be smart about it, because both types of candidates sound the same. It’s easy to avoid this problem by answering questions as asked and mentioning tradeoffs. Trying to dismiss questions never works in your favor.
      • dondraper36 1y ago
        Yes, I would probably phrase it like this. "Under the current load, I would go super simple and use X, which can work fine long enough until it doesn't. And then we can think about horizontal scaling and use Y and Z". Then proceed with a deeper discussion of Y and Z, probably. After all, interviewing and understanding what your interviewer expects to hear is also a valuable skill (same as with your boss or client).
• Aurornis 1y ago
  > > Interviewer: "Well what about backpressure?" > > "That's not really worth considering for this amount of QPS" There is a good way and a bad way to communicate this in interviews. If an interviewer is asking about back pressure, they’re prompting you to demonstrate your knowledge of back pressure and how and when it would be applied. Treating it as an opening to debate the validity of the question feels like dodging the question or attempting to be contrarian. Explaining when and where you would choose to add back pressure would be good, but then you should go on to answer the question. This question hits close to home for me because I was once working at a small startup that was dealing with a unique problem where back pressure really was the correct way to manage one of our problems, but we had a number of candidates do exactly what you did: Scoff at the idea that such a topic would be relevant at a startup. If we’ve been dealing with a problem for months and a candidate comes in and confidently tells us that problem isn’t something we would experience and dismisses our question, that’s not a positive signal. > > Interviewer: "How would you choose between sql and nosql db?" > > "Doesn't matter much. Whatever the team has most expertise in" This is basically a softball question. Again, if you provide a non-answer or try to dismiss the question it feels like you’re either dodging the topic or trying to be contrarian. It’s also a warning sign to the interviewer that you might gravitate toward what’s easy for you instead of right for the project. This one also resonates with me because I spent years of my life making MongoDB do things that would have been trivial if earlier developers had used something like SQLite instead. The reason they chose MongoDB? Because the team was familiar with it. It was hell to be locked into years of legacy code built around the wrong tool for the job because some early employees thought it didn’t matter “because startup” As an interviewer, let me give some advice: If an interviewer asks a question, you should answer the question. Anything that feels like changing the subject, dodging the question, or arguing the merits of the question feels like the candidate either doesn’t understand the topic or wants to waste time by debating the question. It can be very valuable to explain when and why a topic would become necessary, right before you explain it. Instead of “this application has low QPS and therefore I will not answer your question” (not literally what you said, but how it comes across) you could instead explain how the need for back pressure could be avoided first by scaling servers appropriately and then go on to answer the question that was asked.
  • cryptonector 1y ago
• abound 1y ago
  You don't need to entirely forget this. I've made a habit of regularly seeking out job opportunities and interviewing even when I'm entirely happy with my job, which is to say I've done a ton of these kinds of interviews (on both sides of the table). Unless the initial question requirements are insane (build Twitter at Twitter scale), I start with the smallest, dumbest thing that will work. Usually that's a single machine/VM talking to a database (or even just SQLite!). Compute and storage are so fast these days that you could comfortably run your fledgling service on a Raspberry Pi, even serving three or four-digit QPS depending on the workload. Of course, you still have to "play the game" in the interview, so make sure to be clear about how you'd change this as requirements changed (higher QPS, more features, etc)
• _fat_santa 1y ago
  This goes back to "interviews go both ways". All those answers you gave are very reasonable and if I was your interviewer I'd pass you with flying colors. On the other hand if you're interviewing at a place that doesn't pass you with flying colors for those responses, that really says more about them than it does about you and may not be a great place to work. But to your point, many times one interviews for a job they don't really have the luxury of getting rejections and need to land somewhere fast so they can keep paying the mortgage. So while yes interviewing is a two way street, there's still quite a bit of calibration to make sure you land on the other person's side of the street so to speak.
  • atomicnumber3 1y ago
    If I was your interviewer, I would: respect your answers a lot, not be able to check off anything on my rubric, try to explain this in the debrief, get told we have to stick to the rubric to counter bias, and then watch while they pass on you for someone who decided to play architecture jenga instead. I would potentially even consider emailing you to apologize later, then not do it because I'd probably get in trouble for exposing us to liability or something because apologizing can be construed as admission of guilt.
    • belinder 1y ago
      Exactly, it would only work if you have enough sway with your boss and the willingness to take responsibility for the hire
    • willio58 1y ago
      I’ve interviewed dozens of people and while I rarely do system design questions and our process isn’t nearly as check-all-the-boxes, it’s funny how accurate your comment still is. Near the later stages especially, politics starts coming in.
    • yojo 1y ago
      If a candidate doesn’t ask clarifying questions that lead them to an understanding of QPS, storage requirements, and throughput considerations, that’s a mark against. At that point, if you want to see them design a distributed system with all the bells and whistles, you should stop them, tell them the kind of traffic they need to handle, then let them go again. If they persist in designing a system that cannot handle the specified load, they have probably failed the interview.
• santiagobasulto 1y ago
  I’ve been in software for 20 years and it’s the first time I hear “back pressure”. Am I too old already?
  • Aurornis 1y ago
    Backpressure occurs at many levels, even down to a single machine doing something. If you ever have a producer and a consumer interacting and the consumer can’t consume as fast as the producer can produce, you need some way to have the producer pause or slow down until the consumer catches up. That’s back pressure.
  • marcosdumay 1y ago
    It's a sign that you didn't get into the "let's distribute every problem" rabbit hole. I don't think it correlates with age. But the keep the concept in your mind in case you have to distribute some problem. It's a central one.
  • RaftPeople 1y ago
    > it’s the first time I hear “back pressure”. Am I too old already? It's the opposite, as you get older you will feel this more and more.
  • stavros 1y ago
    You've just never played Factorio.
    • santiagobasulto 1y ago
      I have never played Factorio nor knew about it. It seems to be a very good game, thanks for the recommendation!
      • stavros 1y ago
        Unfortunately, it's too good. At least you'll learn all about backpressure in the days you spend lost to the world!
  • 1y ago
• ozgrakkurt 1y ago
  You don’t want to work at a company like this anyway.
  • 0manrho 1y ago
    True, but people generally don't want to get evicted or have their utilities turned off either. If you need a job you need a job, and the numbers out of Cali's job market puts a lot of tech people in a position where they might not have the luxury of waiting for the "right fit". As always, YMMV and the world is a big place, everyone's different, yadda yadda.
  • __turbobrew__ 1y ago
    If you want to get top dollar at a FAANG you will need to go through these type of system design interviews. You could say you shouldn’t work for a FAANG which is fair, but FAANG pays top dollar.
    • try_the_bass 1y ago
      So if you're after the FAANG money, you have to play the FAANG interview game. If you're unwilling to play the FAANG interview game, then maybe you shouldn't be pursuing FAANG money.
      • __turbobrew__ 1y ago
        Doing FAANG interviews is a necessary, but not sufficient condition to getting a FAANG salary. As others have said, non-FAANGs are trying to do FAANG interviews, and I would tell them to pound sand.
      • UK-AL 1y ago
        It's not just FAANG that do these interviews anymore. Tiny startups do them now as well.
• jstummbillig 1y ago
  If you know that those are not the answers they are looking for, you can reasonably pass by modifying the answer only slightly, while still getting your point across. If you can't, you might be getting interviewed by people you do not what to work with and you should want to know that.
  • flashgordon 1y ago
    Except these are the people in your way of getting that job that could be potentially life/career changing for you financially or otherwise. In this market or depending on your situation that would be hard to ignore.
    • jstummbillig 1y ago
      I think that's a red herring. You are a knowledge worker. You are paid to disagree when necessary. Yes, people will probably take offense when you say "that's just a dumb question" but if they can't at least be approached when you offer your opinion in a palatable(sic) way, that's simply not going to work. Understand what is being asked. Your insight on a topic is being tested. Offer an answer that does not read like a dodge or a coin flip.
• ajmurmann 1y ago
  In some ways it's worse. There are also project review interviews. "We had a Rails/Django/whatever monolith that was backed by Postgres and we didn't need a SPA" makes for a less impressive session with many companies. This creates a lot of incentive to overcomplicate/"future proof" things for resume building.
• corytheboyd 1y ago
  As well as the “two-way street” point made in a sibling comment, I feel like a good interviewer would say “this is great, I would keep it simple too, but I am testing your knowledge of $thing right now.” If the person won’t stop talking about the wrong thing, that’s a bad sign of course.
• mupuff1234 1y ago
  I think you might be missing the point. Your answers are completely valid but you have to communicate to the interviewer that you considered the possibilities and the tradeoffs. If the interviewer needs to "forcefully" extract from you the logic behind your design choices than a lot of times that's enough to fail you.
• uberduper 1y ago
  This is awful advice. Simple and elegant design does not start with dismissing potential problems. Those questions are all prompts to have a discussion in lieu of tech trivia hour. Those responses do not demonstrate wisdom, they reveal a lack of maturity. It's not the interviewers fault you refuse to be interviewed.
  • titanomachy 1y ago
    I agree, the responses give the vibe of "your questions are dumb and I'm too smart to waste the effort to engage with them." If you don't want the job, then don't interview!
• torginus 1y ago
  Who does this? Why make something 10x as complicated as it needs to be, when you could just use the simple thing and get 10x as far? It's not like there's not enough work to do.
  • cryptonector 1y ago
    Who accumulates decades of legacy code?! Real companies do. The moment you deploy one line of code, it's legacy. It goes from there. Soon you have to build systems that interface with other systems you'd rather were better architected and designed, except you have to deal with them as they are. Then your product becomes one of these, and with no need to maintain or expand it for a long time, it rots a bit, and now someone has to pick it up or interface with it, and your product made things more complex, and the complexity can't be magic wand waved away.
• didibus 1y ago
  You're equating simplicity of the design with simplicity of the problem. It's good not to over engineer, over engineering can be a cause of unneeded complexity, but when complexity is warranted the ability to solve for it simply is also needed. More importantly though, you haven't explained or rationalized why? It's not needed for this QPS? Oh ya? Why not? What's your magic threshold? When would it be needed? How do you plan for the team to know that time is approaching? If it's needed later how would you retrofit it? Is that going to be a simple addition? How do you know the max QPS won't be too high and that traffic won't be spiky? What if a surprise incident occurred that caused the system to overload, how would your design, without backpressure, handle that, how would you mitigate and recover? In system design there's no real right answer, as an interviewer you're looking for the candidate to demonstrate their ability to identify the point of concerns, reason through the possibilities, explain their decisions and trade offs, and so on.
• renewiltord 1y ago
  You can always say “Since we’ve got only x QPS, I’m going to do A. If we had say y QPS, I’d do B but that would impact the rest of the design. Let me know if you anticipate growth to y and I can show you how I’d do it” The point of an interview is to lay bare one’s thought process entirely so that the interviewer has full awareness of the person you are. And to likewise extract that from the interviewer. Getting or transmitting less information is just underutilizing the time. Interviewers are also flawed and may not be good enough at extracting the information from you. If you’re an ideal decision maker, you will likely out-skill the majority of interviewers. You’re being hired to make their org succeed. So just do that. I think people who describe system designs frequently fail to demarcate the space they’re operating in, so subsequent engineers cannot determine whether the original designer failed to consider something or whether the original designer considered and dismissed something. The point is to be able to express this concisely. IMHO, doing it well means that not only do you get it right but you send the information down through time so that subsequent observers understand why and also get it right consequently.
• ajuc 1y ago
  Answer what they want and finish with "but in practice it doesn't matter for this much traffic and would be wasted effort". People ask for fizzbuzz in parallel not because it's practical.
• mkozlows 1y ago
  (For context, I've conducted hundreds of system design interviews and trained a dozen other people on how to do them at my company. Other interviewers may do things differently or care about other things, but I think what I'm saying here isn't too far off normal.) I think three things about what you're saying: 1. The answers you're giving don't provide a lot of signal (the queue one being the exception). The question that's implicitly being asked is not just what you would choose, but why you would choose it. What factors would drive you to a particular decision? What are you thinking about when you provide an answer? You're not really verbalizing your considerations here. A good interviewer will pry at you to get the signal they need to make a decision. So if you say that back-pressure isn't worth worrying about here, they'll ask you when it would be, and what you'd do in that situation. But not all interviewers are good interviewers, and sometimes they'll just say "I wasn't able to get much information out of the candidate" and the absence of a yes is a no. As an interviewee, you want to make the interviewer's job easy, not hard. 2. Even if the interviewer is good and does pry the information out of you, they're probably going to write down something like "the candidate was able to explain sensibly why they'd choose a particular technology, but it took a lot of prodding and prying to get the information out of them -- communications are a negative." As an interviewee, you want to communicate all the information your interviewer is looking for proactively, not grudgingly and reluctantly. (This is also true when you're not interviewing.) 3. I pretty much just disagree on that SQL/NoSQL answer. Team expertise is one factor, but those technologies have significant differences; depending on what you need to do, one of them might be way better than the other for a particular scenario. Your answer there is just going to get dinged for indicating that you don't have experience in enough scenarios to recognize this.
  • philjohn 1y ago
    +1 on the signal. A great candidate won't need you to pry further than asking about backpressure, they'll explain WHY it's not necessary for the qps, what qps it would start becoming necessary, and how they would build it into their design down the line if the service takes off. One of the things I tell people preparing for system design interviews is the more senior you are, the more you need to drive the interview yourself, knowing when to go deep, what to go deep on, and how to give the most signal to the interviewer.
• reactordev 1y ago
  Louder for the back. It’s like people crave complexity because it makes them, indispensable? Like if you’re the only one who knows how the billing reconciliation service works, they couldn’t possibly fire you? They will. Being pragmatic is something I look for in engineers. So long as they understand where to draw the line (and use a queue instead of cron). However that’s usually several years away at this point and them being able to say “You don’t need that, all you need is…” is welcome. Then again, that’s probably why I got fired. :shrug:
  • 9dev 1y ago
    I believe the reason is far more mundane: Complex systems are more interesting, with all the shiny knobs and levers and mysterious thingamabobs. Developers have a tendency to get nerd-sniped by interesting problems, and picking overly complex solutions to solve them at an abstract level scratches that itch very succinctly. In my experience, senior engineers learn to control this urge, and staff engineers can accurately decide when to break the rule and the complexity is warranted.
• LoganDark 1y ago
  I don't think they're completely not looking for that entire type of answer, but those examples are pretty dry and don't really go into the reasoning for your opinion, which is probably what they're worried about. Whenever you say something isn't worth considering or doesn't seem necessary, you should be explaining exactly why you think that, and exactly where it would be worth considering or seem necessary, because otherwise you just look like someone who simply doesn't care about whatever kind of scalability they're asking about.
• pragmatic 1y ago
  Yes, and then you get the job there and regret it bc they’ll have either have an over engineered Rube Goldberg contraption or they have system envy bc they’ve read about this architecture in blogs and THINK they need K8s and they still fix all their problems.
• throwaway7783 1y ago
  This. There is also really no easy way of telling how an interviewer is thinking. One interviewer thought not having a warehouse in the design was a mistake, and the other one though having a caching solution made things too complex. It is completely a hit or miss with interviews
• zeroq 1y ago
  Not only theory crafting during interviews but a lot of real life design is driven by what's known as resume driven development. The worst part - some of that is later presented at large conferences as successful and go-to solutions. One time I was working in a body leasing company and our team was hired by bigco for an internal project. Two months earlier an internal employee was tasked to research the project and develop a prototype. When we started all major set pieces were written in stone. Month later said employee left. When we later checked the job listing he likely applied to our tech stack mirrored that to a letter. He got free training, a resume and a new job. We were stuck with these decisions for 3 years. Another time a local branch of another bigco was trying to carve out a major piece of internal cake. Head-of was hired, team was quickly ramped up and they started cooking their foothold. Then a series of major power shifts happened couple levels above our pay grade and another branch came out with competitive strategy. We had a 2 days long internal brainstorm involving 50 people to come up with arguments and strategies how to defend our approach. We bet on blue, they were selling red. Life's were at stake. And many truly believed that blue was the way to go, and red was a recipe for disaster. Two days later we had a rock solid presentation that was trashing red approach. But if course most of these decisions are not made by nerds and middle mgmt do eventually the company placed their bet in red and the whole dept became redundant. No one likes to lose their jobs, so our blue head-of quickly turned his cloak and the team became an outsourcing provider for the winning team. What makes this story particularly funny is the fact that the head-of immediately started campaign of conference presentations where he sweard that all his life he believed that red was the future that will eventually trump blue, and any competition that is still using blue is destined to fail in short future.
• master_crab 1y ago
  Nothing against your content…but Kubernetes does manage Kubernetes. That becomes obvious when you start bootstrapping an HA cluster with multiple control plane nodes. K8s is not for the faint of heart…or rational system designers ;)
• cavisne 1y ago
  The interviewer is just another engineer trying to understand if you are someone who they can have a design discussion with. Dismissive answers that assume they are needlessly over complicating things tells them exactly what they need to know
• paulddraper 1y ago
  Why would someone ask about low QPS? Seems it would evoke the “whatever” answers you gave. > SQL and NoSQL don’t matter much Database is literally the most important architectural decision possible, next to the application programming language. (Prove me wrong)
• AbstractH24 1y ago
  Tools that reduce the barrier to entry to creating things make it easier to solve problems with less scale to pay for the overhead. Generative AI is among these tools, but so are low code platforms, so is React, so is AWS, heck, so is the power grid. But in recent times generative AI is a big leap forward. We’re at the start of another cycle of a lot of niche products followed by the rise of big Acme megacorps who conquer them all economies of scale that compete on margin. It comes just as we’re at the tail end of this cycle with tech as we knew it for the last 50 or so years.
• bccdee 1y ago
  I think the right tack, if you're going to dismiss something as needlessly complex, is to call out the circumstances that would make it necessary and then describe what you'd do under those conditions. "Backpressure? I don't think you'll have enough traffic to make backpressure necessary. The mode of failure here is that you run out of queue space and start dropping messages, and it's not a big deal if some messages get dropped here. But if we do decide that dropped messages are causing problems, and if it starts becoming a regular occurrence (we'll set up observability), here's how the producer can poll the queue size and return an error to the user under heavy load.
• kenny239 1y ago
  lol that's sad and real. modern software engineering has a lot of bloatware, costing security, etc.