24 ms·

In my 10 years at Amazon, I interviewed lots of engineers and I think Amazon's interview process is fatally flawed. Of course Amazon is still able to hire good

In my 10 years at Amazon, I interviewed lots of engineers and I think Amazon's interview process is fatally flawed. Of course Amazon is still able to hire good candidates but I believe it is in spite of this process and not because of it. Here were the most frustrating aspects of interviewing for me: 1. Bar raisers don't ask questions that they themselves don't know the answer to. I believe this is what Roseman is referring to (as a desirable?!? attribute) when he says "One of the things that really pisses me off is people asking questions that they don’t even have a good handle on themselves". What happens in practice is that you miss hiring people who may not be able to answer a question to which every interviewer already knows the answer but whom might be uniquely qualified in some other area. 2. Interviewers still ask candidates to code answers to theoretical questions on the whiteboard. There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform. 3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript. 4. This isn't really part of the Amazon interview process but Amazon's job descriptions are (for the most part) written so generically and absent of personality that I think they miss out on lots of great candidates for whom startups simply seem more fun & interesting.

WalterBright 13y ago
> I think Amazon's interview process is fatally flawed. The topic of interviewing comes up often here, and this sentiment is overwhelmingly the top sentiment for every single company's interview process. Amazon, Google, Apple, Facebook, Microsoft, etc., have all been pilloried here about the interview process. I suspect the real problem is the implication that this is even a solvable problem.
- dylangs1030 13y ago
  I agree. I don't think it's a solvable problem. It's like people who champion the removal of SAT/ACT from college admissions. Okay, so statistics demonstrate the SAT only correlates with the first year of college...great, what do you have to replace it with? We work with what we have, and take the benefits it gives us. Systems can always be gamed, and that's an unfortunate (fortunate for some, I guess) reality.
  - jpalioto 13y ago
    It's a maximization problem. The goal is to have the best possible ROC curve for your hiring process. Maximize true positives while minimizing false positives. My personal threshold attempts to minimize false positives even to the extent that I will pass on people that would probably have done a good job. I tune it this way because I think that bad hires are one of the worst things that can happen to a team. But, each individual has to calibrate this for themselves. I'm not looking to solve the problem, I'm looking to make the best possible decision based on the information I have available.
    - vladimirralev 13y ago
      That sounds accurate, but 1. You need large samples of hires because you must know what features actually correspond with good performance 2. You need to have a near perfect understanding of statistics Once Google and friends collected a large enough sample they concluded their puzzles are not a good indicator. Similarly, people tend to have bias towards certain skills without any statistical evidence that they are correlated with performance. And statistics is all about removing the bias, which has proven time and time again is very hard. Otherwise you would be just gambling on instinct, which at best is just a sanity check.
    - dylangs1030 13y ago
      I agree strongly. But part of the problem with this solution is that you might be caught with a stream of merely adequate people you won't hire because you're afraid of false positives. I also think it's sometimes okay to hire merely adequate people, if you know ahead of time you can quarantine their usefulness to one area they seem stellar at.
agilebyte 13y ago
Re 2.: I have taken some code challenges in the past and I find the Amazon open-ended questions the best. Sure, some people can (and possibly like to?) memorize different data structures and algos, but broad thinking suits me best and companies like Google and Facebook don't seem to be doing it enough. Basing it on the interview questions posted online.
- arecommender 13y ago
  If you have gaps in your algo knowledge check out this book (http://www.algorist.com/ http://www.algorist.com/), it's a dense read but comprehensive. Why be a broad thinker when you can be an everything thinker?
  - agilebyte 13y ago
    Thanks. Even just the website gives a nice outline of some of the different problems and how to solve them. Gaps in my algo knowledge are there on purpose since to me they are a solved problem and easily found online.
pnathan 13y ago
> There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform. When looking for work, I have no room for companies that want to do this: I already have a job and commitments. Also, I might not even be in the same state as the company I am talking to.
- jotux 13y ago
  How is trying out a practical for anything other than unemployed new grads? I see this constantly suggested but I don't know a single person who would even consider quitting or taking vacation time to go work for a prospective company.
  - chris_mahan 13y ago
    Although this would work for people who are unemployed. I took a 3 months contract position, and I was hired full time after 1 year. It's been 2.5 years and I'm still at the company.
  - wpietri 13y ago
    Depends a lot on the person. If somebody has done consulting or contract work, it's a pretty natural thing. I've done it both as an employee and an employer.
  - scrabble 13y ago
    The only way I'd consider doing this is taking vacation if I'm being paid a rate that is significantly higher than my current rate. Otherwise, just ask for a coded solution to some specific problem or user story and I'll work on it after hours and send it in.
- svdad 13y ago
  I barely even have time for the 3-6 technical interviews most companies want to do, let alone multiple days/WEEKS of "try out" period.
  - 13y ago
iyulaev 13y ago
There's one thing this article misses, and one thing I totally disagree on. Hiring for prior experience - the author emphasizes this, and based on my experience this should be de-weighted. The issue is that as soon as you start to define an area of experience, your hiring pool shrinks dramatically. Consequently you are more likely to pick a less powerful engineer. Another, bigger downside here is that companies are terrible at figuring out what they really need. Consequently you've hired someone who can do A and B but one month in you realize that the thing you were missing is really C. A less powerful engineer will have a harder time transitioning over to doing C work, and you'll be worse off. Moreover they might be upset - they came in with the expectation of doing A & B, and they're sitting here doing something unrelated instead. Can't count the number of times I've seen this happen, to both the employee's and company's detriment. The thing that this article misses is how to avoid hiring jerks. Terrible employees can be a net negative to a company. Jerks, particularly those in senior positions, are the thermonuclear version of terrible employees. They can demotivate and destroy the productivity of entire teams and cause your best engineers to rage-quit. The best interviewing tactic I've seen to weed out these people is watching how someone behaves when you "play dumb" in an interview. See if they're happy to explain something to you, or if they get impatient with your lack of knowledge. A related trick is challenging them on something that is obviously correct - see if they carefully prove why their answer is correct, or become antagonistic. Really smart people are good, but much more effective are those that can help bring everyone else up to their level.
- svdad 13y ago
  Great point about not hiring jerks, and those are some excellent ideas about how to avoid it. Having worked with, and currently working with, that kind of jerk I've seen how destructive they can be.
- pekk 13y ago
  Being partly facetious here: I don't know how you actually propose not to hire jerks. The industry is full of jerks. Jerks are doing the interview much of the time. The industry wants people who claim to be the best, and that's hard to do while also selecting for niceness.
marssaxman 13y ago
> There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting Are you seriously proposing that any serious applicant would be willing to take a week off from their current job just to interview for a possible job at Amazon?
Zimahl 13y ago
3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript. Yeah, this is pretty painful. I've phone interviewed with Amazon twice. At the time my resume was pure front-end developer with just a smattering of server-side experience. From the Amazon recruiter everything was smiles and 'perfect'. The interview was for a server-side & database developer position with warehouse software. WTF?! Did you look at my resume for even twenty seconds? I got database design and management questions mostly. That interview was brutal. What a waste of time for both parties.
bane 13y ago
I remember my own interview process with Amazon. I wasn't interviewing for an SDE position but a TPM role. I made it through the gauntlet and it all fell down due to Amazon's grossly unrealistic understanding of what the going rates for experienced PMs are. I mostly had fun with it, but I was also frustrated. The phone screens were the hardest in a purely technical sense, lots of algorithm questions that really didn't have anything to do with the job at all, I had to reach into my way-back machine to dig up some of the runtimes. But I made it through three of phone calls like that and had a bit of fun. After the phone calls and a chat with a senior person in the department, I was pretty sure I had a feel for where I was going to end up and what kind of projects I'd be managing. So I boned up on management theory and algorithms and a few other odds and ends. Flying out to Amazon for the face-to-face, I realized I was wrong. Each face-to-face interview was wildly unfocused, I was interviewing in subjects the interviewer often didn't have any experience in and completely didn't reflect the nature of the phone interviews or the statements I had been given by the senior guy I spoke with. In one interview I'm designing Java APIs for movie tickets, in another I'm specifying contingency plans for overseas warehouse fulfillment issues. In another I was asked by two very junior people who frowned in confusion the entire time as I discussed complex employee and co-worker conflict resolution cases I've experienced over the last 18 years. In fact I was never asked a question about the first 15 years of my work experience, most of which would have been more relevant. But I was asked tons of questions about my side-startup, which was just me and my wife hacking around on the side and getting passive ad revenue and had no relevancy at all to the job. The "lunch chat" was with somebody who couldn't have made small talk if he had a gun to his head. He just sat in silence and ate his soup the entire time while I uncomfortably ate one of the sloppiest turkey pesto sandwiches I've ever encountered. I definitely didn't nail the face-to-face, but that's because I was playing trivial pursuit with the interviewers and the answers were arbitrary, instead of knuckling down on the job, the role, the department, the expectations of performance and the responsibilities of the position. But I did something right because I was called in the next day to go over compensation and was blown away to find out the absolute max pay cap for the entire company was almost a 20% pay cut from what I'd been making in my previous job (a job that I was leaving because I was 10-15% under my peers and had been in a pay freeze for 3 years while the company worked towards profitability -- and yes it was from a metro area with a similar cost of living to Seattle). Despite that, Amazon would have been good to work at, but even my absolute pay floor was 10% over. I even asked to make up the compensation difference with a mix of stocks or other methods, but eventually we had to agree that there was no way I could live the life I had become accustomed to with what Amazon was willing to pay.
serge2k 13y ago
Regarding point 3, I've seen that given out as an option before in talks about interviews. I only see it as a solution that works if the person is actually hired. If not they waste 2 weeks of their time and probably other SDE time. It leaves no room for people who apply while still working at another job. I had to miss 2 days of work for my interview, and that was about the biggest risk I could afford. Even if you say you will pay me for the 2 weeks, I still wouldn't have a job to come back to. It's probably still a poor job, I was just starting to get into my first task towards the end of 2 weeks. I spend most of the time just getting everything setup (granted we ran into issues that had to be resolved with another team, but still).
kb19 13y ago
I was just hired at Amazon as an SDE and found the interview process to be fairly challenging and technical (but what do I know). I appreciated the fact that a lot of the interview focused on core data structures / algorithms and not languages / frameworks of the moment kind of thing. It caused me to go back through my university books and study a lot of the core concepts that I enjoyed learning and have gotten away from by doing more menial programming tasks (not to say that I won't be don't those sorts of things at the job). Also, if they said that they wanted to try me out for a couple weeks I would have refused the offer. I already had a job that I was happy (and well paid) at, so why should I leave or take vacation to move across the country and be tried out at another company?

WalterBright 13y ago

> I think Amazon's interview process is fatally flawed. The topic of interviewing comes up often here, and this sentiment is overwhelmingly the top sentiment for every single company's interview process. Amazon, Google, Apple, Facebook, Microsoft, etc., have all been pilloried here about the interview process. I suspect the real problem is the implication that this is even a solvable problem.

dylangs1030 13y ago
I agree. I don't think it's a solvable problem. It's like people who champion the removal of SAT/ACT from college admissions. Okay, so statistics demonstrate the SAT only correlates with the first year of college...great, what do you have to replace it with? We work with what we have, and take the benefits it gives us. Systems can always be gamed, and that's an unfortunate (fortunate for some, I guess) reality.
- jpalioto 13y ago
  It's a maximization problem. The goal is to have the best possible ROC curve for your hiring process. Maximize true positives while minimizing false positives. My personal threshold attempts to minimize false positives even to the extent that I will pass on people that would probably have done a good job. I tune it this way because I think that bad hires are one of the worst things that can happen to a team. But, each individual has to calibrate this for themselves. I'm not looking to solve the problem, I'm looking to make the best possible decision based on the information I have available.
  - vladimirralev 13y ago
    That sounds accurate, but 1. You need large samples of hires because you must know what features actually correspond with good performance 2. You need to have a near perfect understanding of statistics Once Google and friends collected a large enough sample they concluded their puzzles are not a good indicator. Similarly, people tend to have bias towards certain skills without any statistical evidence that they are correlated with performance. And statistics is all about removing the bias, which has proven time and time again is very hard. Otherwise you would be just gambling on instinct, which at best is just a sanity check.
  - dylangs1030 13y ago
    I agree strongly. But part of the problem with this solution is that you might be caught with a stream of merely adequate people you won't hire because you're afraid of false positives. I also think it's sometimes okay to hire merely adequate people, if you know ahead of time you can quarantine their usefulness to one area they seem stellar at.

agilebyte 13y ago

Re 2.: I have taken some code challenges in the past and I find the Amazon open-ended questions the best. Sure, some people can (and possibly like to?) memorize different data structures and algos, but broad thinking suits me best and companies like Google and Facebook don't seem to be doing it enough. Basing it on the interview questions posted online.

arecommender 13y ago
If you have gaps in your algo knowledge check out this book (http://www.algorist.com/ http://www.algorist.com/), it's a dense read but comprehensive. Why be a broad thinker when you can be an everything thinker?
- agilebyte 13y ago
  Thanks. Even just the website gives a nice outline of some of the different problems and how to solve them. Gaps in my algo knowledge are there on purpose since to me they are a solved problem and easily found online.

pnathan 13y ago

> There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform. When looking for work, I have no room for companies that want to do this: I already have a job and commitments. Also, I might not even be in the same state as the company I am talking to.

jotux 13y ago
How is trying out a practical for anything other than unemployed new grads? I see this constantly suggested but I don't know a single person who would even consider quitting or taking vacation time to go work for a prospective company.
- chris_mahan 13y ago
  Although this would work for people who are unemployed. I took a 3 months contract position, and I was hired full time after 1 year. It's been 2.5 years and I'm still at the company.
- wpietri 13y ago
  Depends a lot on the person. If somebody has done consulting or contract work, it's a pretty natural thing. I've done it both as an employee and an employer.
- scrabble 13y ago
  The only way I'd consider doing this is taking vacation if I'm being paid a rate that is significantly higher than my current rate. Otherwise, just ask for a coded solution to some specific problem or user story and I'll work on it after hours and send it in.
svdad 13y ago
I barely even have time for the 3-6 technical interviews most companies want to do, let alone multiple days/WEEKS of "try out" period.
- 13y ago

iyulaev 13y ago

There's one thing this article misses, and one thing I totally disagree on. Hiring for prior experience - the author emphasizes this, and based on my experience this should be de-weighted. The issue is that as soon as you start to define an area of experience, your hiring pool shrinks dramatically. Consequently you are more likely to pick a less powerful engineer. Another, bigger downside here is that companies are terrible at figuring out what they really need. Consequently you've hired someone who can do A and B but one month in you realize that the thing you were missing is really C. A less powerful engineer will have a harder time transitioning over to doing C work, and you'll be worse off. Moreover they might be upset - they came in with the expectation of doing A & B, and they're sitting here doing something unrelated instead. Can't count the number of times I've seen this happen, to both the employee's and company's detriment. The thing that this article misses is how to avoid hiring jerks. Terrible employees can be a net negative to a company. Jerks, particularly those in senior positions, are the thermonuclear version of terrible employees. They can demotivate and destroy the productivity of entire teams and cause your best engineers to rage-quit. The best interviewing tactic I've seen to weed out these people is watching how someone behaves when you "play dumb" in an interview. See if they're happy to explain something to you, or if they get impatient with your lack of knowledge. A related trick is challenging them on something that is obviously correct - see if they carefully prove why their answer is correct, or become antagonistic. Really smart people are good, but much more effective are those that can help bring everyone else up to their level.

svdad 13y ago
Great point about not hiring jerks, and those are some excellent ideas about how to avoid it. Having worked with, and currently working with, that kind of jerk I've seen how destructive they can be.
pekk 13y ago
Being partly facetious here: I don't know how you actually propose not to hire jerks. The industry is full of jerks. Jerks are doing the interview much of the time. The industry wants people who claim to be the best, and that's hard to do while also selecting for niceness.

marssaxman 13y ago

> There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting Are you seriously proposing that any serious applicant would be willing to take a week off from their current job just to interview for a possible job at Amazon?

Zimahl 13y ago

3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript. Yeah, this is pretty painful. I've phone interviewed with Amazon twice. At the time my resume was pure front-end developer with just a smattering of server-side experience. From the Amazon recruiter everything was smiles and 'perfect'. The interview was for a server-side & database developer position with warehouse software. WTF?! Did you look at my resume for even twenty seconds? I got database design and management questions mostly. That interview was brutal. What a waste of time for both parties.

bane 13y ago

I remember my own interview process with Amazon. I wasn't interviewing for an SDE position but a TPM role. I made it through the gauntlet and it all fell down due to Amazon's grossly unrealistic understanding of what the going rates for experienced PMs are. I mostly had fun with it, but I was also frustrated. The phone screens were the hardest in a purely technical sense, lots of algorithm questions that really didn't have anything to do with the job at all, I had to reach into my way-back machine to dig up some of the runtimes. But I made it through three of phone calls like that and had a bit of fun. After the phone calls and a chat with a senior person in the department, I was pretty sure I had a feel for where I was going to end up and what kind of projects I'd be managing. So I boned up on management theory and algorithms and a few other odds and ends. Flying out to Amazon for the face-to-face, I realized I was wrong. Each face-to-face interview was wildly unfocused, I was interviewing in subjects the interviewer often didn't have any experience in and completely didn't reflect the nature of the phone interviews or the statements I had been given by the senior guy I spoke with. In one interview I'm designing Java APIs for movie tickets, in another I'm specifying contingency plans for overseas warehouse fulfillment issues. In another I was asked by two very junior people who frowned in confusion the entire time as I discussed complex employee and co-worker conflict resolution cases I've experienced over the last 18 years. In fact I was never asked a question about the first 15 years of my work experience, most of which would have been more relevant. But I was asked tons of questions about my side-startup, which was just me and my wife hacking around on the side and getting passive ad revenue and had no relevancy at all to the job. The "lunch chat" was with somebody who couldn't have made small talk if he had a gun to his head. He just sat in silence and ate his soup the entire time while I uncomfortably ate one of the sloppiest turkey pesto sandwiches I've ever encountered. I definitely didn't nail the face-to-face, but that's because I was playing trivial pursuit with the interviewers and the answers were arbitrary, instead of knuckling down on the job, the role, the department, the expectations of performance and the responsibilities of the position. But I did something right because I was called in the next day to go over compensation and was blown away to find out the absolute max pay cap for the entire company was almost a 20% pay cut from what I'd been making in my previous job (a job that I was leaving because I was 10-15% under my peers and had been in a pay freeze for 3 years while the company worked towards profitability -- and yes it was from a metro area with a similar cost of living to Seattle). Despite that, Amazon would have been good to work at, but even my absolute pay floor was 10% over. I even asked to make up the compensation difference with a mix of stocks or other methods, but eventually we had to agree that there was no way I could live the life I had become accustomed to with what Amazon was willing to pay.

serge2k 13y ago

Regarding point 3, I've seen that given out as an option before in talks about interviews. I only see it as a solution that works if the person is actually hired. If not they waste 2 weeks of their time and probably other SDE time. It leaves no room for people who apply while still working at another job. I had to miss 2 days of work for my interview, and that was about the biggest risk I could afford. Even if you say you will pay me for the 2 weeks, I still wouldn't have a job to come back to. It's probably still a poor job, I was just starting to get into my first task towards the end of 2 weeks. I spend most of the time just getting everything setup (granted we ran into issues that had to be resolved with another team, but still).

kb19 13y ago

I was just hired at Amazon as an SDE and found the interview process to be fairly challenging and technical (but what do I know). I appreciated the fact that a lot of the interview focused on core data structures / algorithms and not languages / frameworks of the moment kind of thing. It caused me to go back through my university books and study a lot of the core concepts that I enjoyed learning and have gotten away from by doing more menial programming tasks (not to say that I won't be don't those sorts of things at the job). Also, if they said that they wanted to try me out for a couple weeks I would have refused the offer. I already had a job that I was happy (and well paid) at, so why should I leave or take vacation to move across the country and be tried out at another company?