14 ms·

We Prefer Coding Challenges

76 points by ddispaltro 7y ago

vorpalhex 7y ago
> They have so many offers that writing code for 4-8 hours is just not going to work for them. Yeah, I don't typically work for free either. Are you paying candidates for this time? "Hey, I need you to come in and stock shelves for four hours, you know, just to see if you're a hidden gem for shelf stocking"
- tombert 7y ago
  I've told this story before here, but I actually did have a job interview for a company, where they had me work on site for four hours, doing "real" work, and they actually paid me. Granted, it was only like $35/hour, which is pretty low for an engineer in NYC, but it was still a welcome change of pace.
  - bitL 7y ago
    Usually companies that advertise paid interviews ghost the candidate in case of either unsuccessful interview or candidate opting for somebody else. OTOH if you want to work in some bleeding edge team that would catapult you to elite status, like AI autopilot at Tesla/SpaceX, you would be fine with an unpaid crazy project to pick you out/weed you out in the first phase.
    - tombert 7y ago
      Well, in this particular case they actually made me an offer, which I accepted but then they rescinded because I stupidly lied about having a degree. I can't speak to any other companies, but at least this one did pay me, like they said they would.
- macintux 7y ago
  On the other hand, I’ll spend 4-8 hours on a coding challenge for free. Give me an interesting problem and why not? I was asked to write an AWS integration tool once, and I declined to proceed with the interview process, because I didn’t feel like spending hours learning the Amazon API and re-learning a language with a useful library (I’ve since come to appreciate Python and Boto3, even if I’d rather be Erlanging). It also wasn’t a sufficiently interesting problem to make the time worthwhile.
  - shantly 7y ago
    I think I'd only be willing to do that if I were unemployed. Otherwise I'm already collecting a paycheck for doing tens of hours of software development a week—the last thing I'm interested in doing is more of that, for free, on a project not of my choosing, so someone else will maybe start giving me a paycheck instead of whoever is now.
  - tptacek 7y ago
    Which, for the right job, can be a perfect example of the system working: the firm wants to hire someone super comfortable with AWS, for that person, the tool might just take 30 minutes to bang out, and so it works as a filter.
- shantly 7y ago
  There are so many mid-tier companies in my mediocre city paying (for the area) absolutely stupid money and so desperate for developers that they wouldn't dare ask that much time of a promising candidate for fear of losing them to somewhere else. It is, to an incredible degree, a seller's market for software development. There are two whole different worlds of software interviewing, as far as I can tell, and outside of FAANG and friends the pay difference is just about non-existent so it's not worth putting up with humiliating or time-wasting games.
  - Zimahl 7y ago
    Yup. If a developer wants to have a FAANG company on their resume they're obviously going to have to jump through some hoops and do what's necessary. Non-FAANG companies are going to have to take what they can get and hope it works out because they aren't worth a long drawn-out hiring process.
    - shantly 7y ago
      Yeah, I can get a bunch of phone screens for opportunities with nearly no effort. If comp's high across the board and I have, say, three or four promising calls within 36hrs of flying my "I'm looking" flag, and one says "we'd like you to come in for a whole day for a half-dozen one-hour interviews" or "we'll give you a 4-hour coding challenge" and the others don't, guess which one's not getting a second conversation? And they know that, so most places aren't pulling that crap. Economy turns around, who knows, probably swing back the other way, but right now? No.
  - nickpsecurity 7y ago
    There might be people here interested in what area that is to scoop up some of that stupid money or get contracts. ;)
    - thfuran 7y ago
      I certainly wouldn't say no to stupid money
- Fellshard 7y ago
  That argument falls apart utterly, since software writing requires you to demonstrate a vastly more technical and nuanced suite of skills than manual labor requiring a minor amount of mental overhead.
  - vorpalhex 7y ago
    Hi, I need you to come in and work on proving Fermat's last theorem for four hours, just to see if you're a hidden gem.
- hogFeast 7y ago
  I don't know where you live, maybe it is different there but the majority of jobs that I have worked - from stacking shelves to financial services - usually required a morning or afternoon in the office to see how everything works. If you are a developer or a mechanic, maybe this isn't necessary...but in a lot of service jobs it is very useful. I have chosen not to work places based on spending time there...nothing was wrong with that place, just a bad culture fit and it wouldn't have worked. It is definitely a very cheap way to derisk the decision if you are an employee.
  - vorpalhex 7y ago
    I use the shelf stocking example because that's actually illegal in the US.
deedubaya 7y ago
Coding challenges as your barrier to entry are an optimization for a certain type of employee - low demand workers who don't have the leverage to by-pass who start hiring funnels that start with coding challenges and recruiter calls. You may be gaining access to hidden gems, but you're definitely turning away proven and experienced developers from even entering your hiring funnel. It all depends on what type of employee you're wanting to hire.
- alexk 7y ago
  Author here. Even seasoned developers have to go onsite and spend 1-2 days doing interview. In this scenario, we are asking for similar amount of time, but with no travel, time that works for everyone and has low stress with an interesting task folks can review beforehand. Besides, before even going through this challenge, any engineer can make up their own mind whether it's worth for them because we present compensation brackets upfront on the call.
  - jakebasile 7y ago
    > Even seasoned developers have to go onsite and spend 1-2 days doing interview. No, they don't. Like most people that champion these types of nonsense "code challenge" interviews you present a false dichotomy of "you have to either torture someone in person for multiple days, or torture them with a bullshit made up test for hours!" You can talk to them for a while over the phone (or video chat), split up over a few days, and get to know them just fine.
    - tptacek 7y ago
      Yes, this much is true: in weak hiring cultures, which are very common, people get their friends (or people with dazzling resumes) hired without any real rigor. What's more, in that culture, it's straightforward to generate a dazzling resume: you "fail up", because it takes longer to get fired (or for it to become clear that it's time to get a new job) than it does to get enough time on the job to look good on a resume.
UweSchmidt 7y ago
Another one of those, huh? The only way to find out what kind of salary is possible is to ask for numbers that get rejected a few times. Additionally, there are many companies out there, with a surprising variety of job parameters. Finally, many people are not good or comfortable with the application process, so you need to practice. This means you have to apply to many companies and have to have many job interviews. Long, multi-round interview processes and take-home assignments - reasonable or not - make this very difficult and expensive for the employee. These regular company blogposts seem to respond to the pushback of applicants and try and establish this kind of application process as a norm. Unfortunately this would decrease the market transparency for employees even more - many people go through their working lives underpaid and in subobtimal working conditions without knowing any better. This procedure also feels asymetric - in a time where software people are supposedly in demand, it's the employee who is supposed to jump through hoops to get a job.
- tptacek 7y ago
  If you reached out at Latacora and asked "is there any way you'd be able to pay $X" before you did any of our challenges, we'd get you a quick, straight answer. Why wouldn't we? If you're outside the ballpark for us, working to recruit you isn't a good use of our time.
nscalf 7y ago
I don’t do code challenges anymore. I have a full time job, I’m working on a startup on the side, I do some extra contracting, and I have an ever growing list of potentially profitable opportunities that I want to build. I would love to get paid more for my full time gig, and I should. I’m solidly a senior engineer, but I lack the title or pay. But I don’t have the time to actually spend 8 hours doing some silly app. I get that it makes sense for the hiring manager to give a coding challenge, but I don’t care. And the reality of our world right now is that I don’t need to care. There are plenty of other companies desperate to bring on good people. My skills give companies extreme multiples of ROI, and no other fields have nearly as ridiculous a hiring process. Have someone come onsite for a day and pay them, we can both decide if it’s a decent fit. But I don’t know you, your company, your team, the work process, the supporting teams, your funding, etc... honestly, one of those things are probably really awful. I’m not jumping through hoops for free. You’re not google, you won’t pay me $150,000+. Pass.
- hdhdbdn 7y ago
  $150k is way the fuck under google money. my last fully remote startup of less than 100 people happily paid me more than that to be a sysadmin... Honestly sounds like Gravitational's hiring filter is doing its job...
  - inimino 7y ago
    Compensation and engineering skill aren't as correlated as you seem to think.
- tptacek 7y ago
  Spending a week doing off-and-on phone screens and then a day on-site doing an interview loop is a bigger hoop to jump through than banging out some code on your couch. But, I mean, people have all sorts of reasons not to apply to a job. Some people don't want to touch Java code. Some people don't want to work for a company in Santa Clara. You don't want to work anywhere that asks you to complete a code challenge. That's totally fine.
  - 7y ago
sandGorgon 7y ago
>Split your code submission using pull requests and give the team an opportunity to review the PRs > Not communicating. Candidates who submitted all the code to master branch, which does not give us the ability to provide feedback on the various implementation phases. Because we are a distributed team, structured communication is critical to us. how do you ask your candidate to split your answer using PR ? We would like this during interviewing but it ends up being a single PR. I dont want to impose create separate branches and a PR for each small incremental feature thing. I'm assuming you are using Github
- alexk 7y ago
  Good question! We usually recommend to submit a fully complete logical part, e.g. auth middleware, client, server. We recommend against submitting incomplete functionality in a PR because it is hard for our interview team to understand whether it's incomplete or complete and has defects.
  - sandGorgon 7y ago
    so would each complete logical part be a separate branch with its PR ? and then branch from the previous branch to create the next PR? I'm just wondering if you found there to be any cognitive overhead here.
0xDEFC0DE 7y ago
Unlike the other code-slinging gods who walk amongst us in this thread, I prefer coding challenges because I'm pretty mid-level and average and I do actually need a job. And I hate whiteboarding. No one should do whiteboarding for code.
- shantly 7y ago
  Agreed. Make me whiteboard or live-code if you wanna make me look and feel like I've not actually had over a dozen years of productive experience designing systems and shipping code and fixing other people's broken crap, and you'd love to watch me stress out and forget how to write a damn loop so you can complain on HN about how I was clearly lying about my experience & abilities and that's why you have to make people live-code or whiteboard. Challenges are, at least, preferable to that, for sure.
- lfowles 7y ago
  Same, I've found that I get further in the interview process when dealing with companies that screen with a coding challenge.
- traderjane 7y ago
  Whiteboarding is like coding except you don’t need to get the syntactical details of your loop right. It’s writing out your thoughts not just in pseudo code, but perhaps also in freehand visualizations. This makes it the most ergonomic and beginner friendly programming language ever.
jonfw 7y ago
> “I’ve been hiring people for 10 years, and I still swear by a simple rule: If someone doesn’t send a thank-you email, don’t hire them.” I was contacted by a recruiter a short while ago asking my availability for a phone screen- and I replied simply with my availability. The recruiter still hasn't gotten back to me, and I think that it may be because I didn't go out of my way to thank them for reaching out to me. Are all of the niceties in your email inbox really that important to people? It seems pretty manufactured and boilerplate when people send me emails that they "hope find me well". I may just have to get over it and start letting people know how "excited I am to speak with you and learn more about this opportunity!"
- alexk 7y ago
  Just to make it clear - I posted this as an example of anti-pattern, not the actual advice to follow :)
- olooney 7y ago
  Yes. They're not testing your knowledge of etiquette, people skills, or friendliness though. They're looking for a more basic and far more widely applicable skill: following best practices. If you were configuring Apache for the first time, wouldn't you google for a list of do's and don'ts and spend a few minutes implementing the low-hanging fruit? Not doing that puts you at a huge risk of leaving a huge security hole or massive performance hit, right? It would be amateurish, and if a problem occurred because you didn't take even a few minutes to do the most basic and widely known stuff, you would have no excuse. Likewise, while nobody really cares about thank you emails, not doing that suggests you didn't even bother to Google "Interview Tips" and implement the ones that took less than a few minutes. That's not a good signal to send to people that are going to have to rely on your professionalism and quality of work for the next few years.
  - jgwil2 7y ago
    This would make sense except that the job isn't interviewing for jobs. You shouldn't hold people to the expectation that they're going to approach communicating with recruiters in the same way they would approach actual work they were actually getting paid for. This is a great way to filter good candidates out based on something that has nothing to do with the work you are hiring for.
deeg 7y ago
I do not want to work for a company that does NOT test my technical skills; if they don't verify my skills that means they don't verify others and I'd prefer to work skilled engineers (with exceptions for entry-level or junior engineers). Most people seem to agree that white-boarding is a poor way to judge technical skills. Coding challenges seem a good compromise, assuming they don't require more than a few hours of time.
- kemiller2002 7y ago
  I do get what you're saying, but what I'm really interested in is what are their skills? Any moron can tell me I'm good whether I am or not.
  - recursive 7y ago
    But the chance of getting it right goes down if they don't have direct evidence of their conclusion.
- kevin_nisbet 7y ago
  I agree. I kind of wish https://www.kalzumeus.com/2015/03/09/announcing-starfighter/ https://www.kalzumeus.com/2015/03/09/announcing-starfighter/ had taken off, as instead of doing a coding challenge for every company, you can do one really well designed challenge, and use that challenge with multiple companies. Note: I'm an employee of gravitational.
  - deleted 7y ago
    [deleted]
- noirbot 7y ago
  I had a similar process recently. Went from recruiter through to offer with only a couple design problems and generally talking about projects I'd worked on, no code whatsoever. If anything, it made me want to work for the company less. I didn't feel like they had a great idea if I was actually good at my job, since almost everything I'd shown in interviews was things I could have easily made up to make myself look good. That's especially worrying to me because it was a company that was still building out the team I'd be working for, so it would be hard for me to be really confident in the skills of my coworkers. Maybe the interviewers were just really good judges of character and skill (I think I probably was a good fit for the position), but it was hard to look past the fact that I didn't think I'd proven that I fit the role to them. Not doing any technical/coding problems makes me worry that the company doesn't see technical skill as important at all. Maybe they could preface the process with an explanation of why they don't, and that could be persuasive, but outside that, even a basic FizzBuzz makes me feel like their hiring philosophy is considering those skills.
kemiller2002 7y ago
I get why companies like them, and there is nothing wrong with them. The truth is that I probably don't want the job enough to actually do it. It takes time away from my family, and my friends and most of the time, it's not that stellar of a place to work anyway. If I'm desperate for a job, I could feasibly see doing it, or if I thought it was a really amazing opportunity. The truth is that I'm probably not going to be in either of those positions. What really does irk me is when a company says you have to do it before they'll even talk to me. I'm definitely not going to spend several hours on something without even knowing anything about the company.
- rdelval 7y ago
  The asymmetric time investment required from candidates is what bothers me the most about these coding challenges. My personal experience has been that it's a lopsided process with the candidate putting in hours of unpaid work and the company spending one, maybe two engineering hours if we're being generous, reviewing the submission. If the reviewer decides to reject the challenge, the loop closes, HR sends a nice e-mail, and the candidate has nothing to show for it.
tptacek 7y ago
Unfortunately, the idea behind coding challenges has been cargo culted into a lot of weak hiring cultures: * The challenges often don't actually matter, and are simply part of the recruiting hazing ritual. * They're totally subjective and just a proxy for the same bullshit friends-and-resumes process people were using already. * No consideration is given to the amount of time the entire recruiting process is taking. * The challenges themselves are totally divorced from the day-to-day expectations of the job, so people are doing CS102 algorithms problems for jobs that will mostly be about moving bits from one place to another. I understand why people don't like them. But that doesn't really matter, because code challenges are so much more effective at qualifying people that it would be crazy for any competent shop to stop using them. For a couple years (and, of course, for many years before at Matasano) we've been hiring resume-blind and with almost no interviewing (we do a final on-site, but it's a formality with no tech-out, which is something we tell candidates). We calibrate our challenges so that they'd ideally, for a qualified candidate, take less time than interviews and phone screens would, and have the advantage of being scheduled at the candidate's convenience (all in one night, spread out over a couple weekends, whatever). We have candidates coming out of our ears (volume has been the single biggest problem with our recruiting practice), and the quality of those candidates is just as good as anyone else's comparable funnel. This is the right way to hire technical staff.
- hyperpape 7y ago
  Do you think it matters that you're unusually outspoken and/or prominent? The cargo-culting actually seems like a problem for both employers and applicants, because it poisons the market. You have to know whether the company is treating coding challenges seriously. Because a company that will ghost you or decline you for unclear reasons after giving you a coding challenge has wasted a lot of time. Or put another way, if it's a hoop to jump through for candidates, coding challenges are awful. If it really means a good chance at that $10k+ raise, coding challenges are a great investment.
  - tptacek 7y ago
    I'm sure it matters a lot that I'm loud. But that's not a superpower; in fact, my friends and many of my coworkers would say it's the opposite. Nothing stops other hiring managers from doing the same thing.
rileymat2 7y ago
I do not mind a coding challenge in the hiring process if it is the last step, not the first step.
deleted 7y ago
[deleted]
a0zU 7y ago
Coding challanges are used pretty often to scam programmers to do work for free before the company rejects them.
- twic 7y ago
  I don't believe this actually happens. Do you have any concrete examples?
  - mamon 7y ago
    There's this infamous company from Austin, Texas, which hires programmers for remote work, and they typically give you coding "challenges" that require a week to complete, and are complete (although not very complex) applications.
IshKebab 7y ago
Great article, although 4-8 hours is clearly way too long. 2 hours is much more reasonable.
mooreds 7y ago
I didn't see "pay candidates for their time" as a section even though I saw several mentions of how invested the company was (CEO commenting on PRs, team investment). Saying "We will pay you to do this exercise" is a simple (and cheap!) way to show you value candidate's time.
etxm 7y ago
They’re bull’s excrement, this industry’s interview process is broken and arbitrary. I don’t know if we’ll ever fix it, but code challenges certainly aren’t the answer. I’ve been developing professionally for 20 years. I take about 4 interviews a year when _not_ looking for a job to stay practiced. Held DoE, Staff, and Principle roles. Anecdotal evidence from this year: - given a decent sized MVC app with bugs and missing features. Tasked to green the test suite. PR comment from a pretty established engineer was “this is perfect, I wouldnt change anything.” I used 1 of my 4 hours and thought my work was poor - given a “simple”[1] sort and map/reduce a large log file and didn’t finish after 2 hours because the interviewer kept interrupting to ask me about the details of how internals of the language and GC worked - things I was familiar with, but I can’t focus on code and answering questions at the same time. I was ranked as “junior” and not a fit for the role, cute. This last one rolls up into one of the problems I have (and assume others have) in interviews: “work through the problem as you would, but talk out loud about your process” If I’m talking, I’m not in my zone. Let me shut the fuck up and solve the problem and we can talk about it afterwards. [1] “simply” parse this 500k line CSV file and aggregate some results, no dependencies. CSV file was actually a STDOUT log file that multiple containers were writing to with different formats, JSON, CSV, TSV, SSV, and random ass unstructured logs from a prod system.
etxm 7y ago
I’ve posted on here before about when I’m in a hiring role how I prefer to do code challenges. I enjoy doing BYOC challenges. Let a candidate pick a ticket from an open source project and come in to pair on it (or shoulder surf if uncomfortable with pairing). I don’t see the issue until the day we are pairing. Goal is to let them be the expert and as comfortable as possible. They get some code on their GitHub, open source benefits, and I get a peak into their engineering skills. If you’re thinking, “hey that’s not apples to apples,” well people aren’t fucking apples. I had someone come in once that didn’t have an “issue.” She had been working with a library that had poor docs and wanted to work through the library and document it. THAT! That was an awesome experience. Best engineer? I had no idea. Willing to bang her head against a wall to make the experience better for others. Yes. Fuck yes, get on my team.
pickle-wizard 7y ago
My current job had a coding challenge and I felt they handled it pretty well. It was done one site as part of the interview process. It was related to the work I'd be doing, however it wasn't super complicated. They expected that you'd be able to finish it in about 30 minutes, but you had as much time as you needed. I finished it in about 20 minutes. Since it was over lunch time, they bought me lunch. I figure lunch for 20 minutes of work was a fair reimbursement.
iliaznk 7y ago
I’ve passed through a screening procedure recently that consisted of three steps: 1) They asked for some publicly available samples of my code «I’m proud of». Okay, I prepared three samples from different areas with detailed descriptions and comments, explaining what that code did and why it was good. 2) After that, as a coding challenge, they asked me to write a simple app using the Unsplash API. 3) Finally we had an over an hour long interview during which I was again given three simple coding problems. And then they rejected me. The question I have after all that is do you really need a multi-staged procedure like that to determine a candidate is not a good fit?! When exactly did they decide I wouldn’t fit? If that was after step 1 or 2, why didn’t they stop right there? If that was only after step 3, then what’s the purpose of steps 1 and 2? They don’t mean anything if step 3 can just cancel them out. Then why didn’t they start with step 3 not to waste my and their time?
jgwil2 7y ago
The process outlined in the article shows respect for candidates in many ways, but especially by providing feedback. I have no problem going through an in-depth interview process, because I can always look at it as a learning experience, but it can be very frustrating to go through a process and then not receive feedback, positive or negative, on your performance. I understand that all employers are busy, engineers' time is valuable, and that often communication is managed by HR people who don't have in-depth knowledge of the reasoning behind decision-making, but I think that making constructive feedback an expectation for the hiring process would improve the experience for candidates manifold, and ultimately would result in less arbitrariness in hiring decisions.
rajacombinator 7y ago
I’ll only do these if 1) company pays me or 2) I’m exceptionally interested and think the comp will justify it. (ie. Almost never.) Spending several hours on something that might not even get reviewed - no thanks.
kureikain 7y ago
This seems got some negative review. Let me shared my experience. Alexander is fantastic as an interviewer and co-worker. I was interview by ALEXANDER a few years ago. I ended up not getting an offer. It was my fault. The challenge is the hardest thing I have to research on my own to implement because google found no result for me at the time. I learned a lot when working on his challenge and the idea of being able to solve it is more imporant than passing to interview to me. The way they conduct interview is really great. You are given help by their team. All I can say is Gravitational takes hiring to their heart and setup for success of candidate. TO Alex, I'm Vinh if you remember me :-).
- alexk 7y ago
  Hey Vinh, thank you for the kind words and I'm glad that you got a positive experience with our challenge and the process!

76 points by ddispaltro 7y ago

> They have so many offers that writing code for 4-8 hours is just not going to work for them. Yeah, I don't typically work for free either. Are you paying candidates for this time? "Hey, I need you to come in and stock shelves for four hours, you know, just to see if you're a hidden gem for shelf stocking"

tombert 7y ago
I've told this story before here, but I actually did have a job interview for a company, where they had me work on site for four hours, doing "real" work, and they actually paid me. Granted, it was only like $35/hour, which is pretty low for an engineer in NYC, but it was still a welcome change of pace.
- bitL 7y ago
  Usually companies that advertise paid interviews ghost the candidate in case of either unsuccessful interview or candidate opting for somebody else. OTOH if you want to work in some bleeding edge team that would catapult you to elite status, like AI autopilot at Tesla/SpaceX, you would be fine with an unpaid crazy project to pick you out/weed you out in the first phase.
  - tombert 7y ago
    Well, in this particular case they actually made me an offer, which I accepted but then they rescinded because I stupidly lied about having a degree. I can't speak to any other companies, but at least this one did pay me, like they said they would.
macintux 7y ago
On the other hand, I’ll spend 4-8 hours on a coding challenge for free. Give me an interesting problem and why not? I was asked to write an AWS integration tool once, and I declined to proceed with the interview process, because I didn’t feel like spending hours learning the Amazon API and re-learning a language with a useful library (I’ve since come to appreciate Python and Boto3, even if I’d rather be Erlanging). It also wasn’t a sufficiently interesting problem to make the time worthwhile.
- shantly 7y ago
  I think I'd only be willing to do that if I were unemployed. Otherwise I'm already collecting a paycheck for doing tens of hours of software development a week—the last thing I'm interested in doing is more of that, for free, on a project not of my choosing, so someone else will maybe start giving me a paycheck instead of whoever is now.
- tptacek 7y ago
  Which, for the right job, can be a perfect example of the system working: the firm wants to hire someone super comfortable with AWS, for that person, the tool might just take 30 minutes to bang out, and so it works as a filter.
shantly 7y ago
There are so many mid-tier companies in my mediocre city paying (for the area) absolutely stupid money and so desperate for developers that they wouldn't dare ask that much time of a promising candidate for fear of losing them to somewhere else. It is, to an incredible degree, a seller's market for software development. There are two whole different worlds of software interviewing, as far as I can tell, and outside of FAANG and friends the pay difference is just about non-existent so it's not worth putting up with humiliating or time-wasting games.
- Zimahl 7y ago
  Yup. If a developer wants to have a FAANG company on their resume they're obviously going to have to jump through some hoops and do what's necessary. Non-FAANG companies are going to have to take what they can get and hope it works out because they aren't worth a long drawn-out hiring process.
  - shantly 7y ago
    Yeah, I can get a bunch of phone screens for opportunities with nearly no effort. If comp's high across the board and I have, say, three or four promising calls within 36hrs of flying my "I'm looking" flag, and one says "we'd like you to come in for a whole day for a half-dozen one-hour interviews" or "we'll give you a 4-hour coding challenge" and the others don't, guess which one's not getting a second conversation? And they know that, so most places aren't pulling that crap. Economy turns around, who knows, probably swing back the other way, but right now? No.
- nickpsecurity 7y ago
  There might be people here interested in what area that is to scoop up some of that stupid money or get contracts. ;)
  - thfuran 7y ago
    I certainly wouldn't say no to stupid money
Fellshard 7y ago
That argument falls apart utterly, since software writing requires you to demonstrate a vastly more technical and nuanced suite of skills than manual labor requiring a minor amount of mental overhead.
- vorpalhex 7y ago
  Hi, I need you to come in and work on proving Fermat's last theorem for four hours, just to see if you're a hidden gem.
hogFeast 7y ago
I don't know where you live, maybe it is different there but the majority of jobs that I have worked - from stacking shelves to financial services - usually required a morning or afternoon in the office to see how everything works. If you are a developer or a mechanic, maybe this isn't necessary...but in a lot of service jobs it is very useful. I have chosen not to work places based on spending time there...nothing was wrong with that place, just a bad culture fit and it wouldn't have worked. It is definitely a very cheap way to derisk the decision if you are an employee.
- vorpalhex 7y ago
  I use the shelf stocking example because that's actually illegal in the US.

deedubaya 7y ago

Coding challenges as your barrier to entry are an optimization for a certain type of employee - low demand workers who don't have the leverage to by-pass who start hiring funnels that start with coding challenges and recruiter calls. You may be gaining access to hidden gems, but you're definitely turning away proven and experienced developers from even entering your hiring funnel. It all depends on what type of employee you're wanting to hire.

alexk 7y ago
Author here. Even seasoned developers have to go onsite and spend 1-2 days doing interview. In this scenario, we are asking for similar amount of time, but with no travel, time that works for everyone and has low stress with an interesting task folks can review beforehand. Besides, before even going through this challenge, any engineer can make up their own mind whether it's worth for them because we present compensation brackets upfront on the call.
- jakebasile 7y ago
  > Even seasoned developers have to go onsite and spend 1-2 days doing interview. No, they don't. Like most people that champion these types of nonsense "code challenge" interviews you present a false dichotomy of "you have to either torture someone in person for multiple days, or torture them with a bullshit made up test for hours!" You can talk to them for a while over the phone (or video chat), split up over a few days, and get to know them just fine.
  - tptacek 7y ago
    Yes, this much is true: in weak hiring cultures, which are very common, people get their friends (or people with dazzling resumes) hired without any real rigor. What's more, in that culture, it's straightforward to generate a dazzling resume: you "fail up", because it takes longer to get fired (or for it to become clear that it's time to get a new job) than it does to get enough time on the job to look good on a resume.

UweSchmidt 7y ago

Another one of those, huh? The only way to find out what kind of salary is possible is to ask for numbers that get rejected a few times. Additionally, there are many companies out there, with a surprising variety of job parameters. Finally, many people are not good or comfortable with the application process, so you need to practice. This means you have to apply to many companies and have to have many job interviews. Long, multi-round interview processes and take-home assignments - reasonable or not - make this very difficult and expensive for the employee. These regular company blogposts seem to respond to the pushback of applicants and try and establish this kind of application process as a norm. Unfortunately this would decrease the market transparency for employees even more - many people go through their working lives underpaid and in subobtimal working conditions without knowing any better. This procedure also feels asymetric - in a time where software people are supposedly in demand, it's the employee who is supposed to jump through hoops to get a job.

tptacek 7y ago
If you reached out at Latacora and asked "is there any way you'd be able to pay $X" before you did any of our challenges, we'd get you a quick, straight answer. Why wouldn't we? If you're outside the ballpark for us, working to recruit you isn't a good use of our time.

nscalf 7y ago

I don’t do code challenges anymore. I have a full time job, I’m working on a startup on the side, I do some extra contracting, and I have an ever growing list of potentially profitable opportunities that I want to build. I would love to get paid more for my full time gig, and I should. I’m solidly a senior engineer, but I lack the title or pay. But I don’t have the time to actually spend 8 hours doing some silly app. I get that it makes sense for the hiring manager to give a coding challenge, but I don’t care. And the reality of our world right now is that I don’t need to care. There are plenty of other companies desperate to bring on good people. My skills give companies extreme multiples of ROI, and no other fields have nearly as ridiculous a hiring process. Have someone come onsite for a day and pay them, we can both decide if it’s a decent fit. But I don’t know you, your company, your team, the work process, the supporting teams, your funding, etc... honestly, one of those things are probably really awful. I’m not jumping through hoops for free. You’re not google, you won’t pay me $150,000+. Pass.

hdhdbdn 7y ago
$150k is way the fuck under google money. my last fully remote startup of less than 100 people happily paid me more than that to be a sysadmin... Honestly sounds like Gravitational's hiring filter is doing its job...
- inimino 7y ago
  Compensation and engineering skill aren't as correlated as you seem to think.
tptacek 7y ago
Spending a week doing off-and-on phone screens and then a day on-site doing an interview loop is a bigger hoop to jump through than banging out some code on your couch. But, I mean, people have all sorts of reasons not to apply to a job. Some people don't want to touch Java code. Some people don't want to work for a company in Santa Clara. You don't want to work anywhere that asks you to complete a code challenge. That's totally fine.
- 7y ago

sandGorgon 7y ago

>Split your code submission using pull requests and give the team an opportunity to review the PRs > Not communicating. Candidates who submitted all the code to master branch, which does not give us the ability to provide feedback on the various implementation phases. Because we are a distributed team, structured communication is critical to us. how do you ask your candidate to split your answer using PR ? We would like this during interviewing but it ends up being a single PR. I dont want to impose create separate branches and a PR for each small incremental feature thing. I'm assuming you are using Github

alexk 7y ago
Good question! We usually recommend to submit a fully complete logical part, e.g. auth middleware, client, server. We recommend against submitting incomplete functionality in a PR because it is hard for our interview team to understand whether it's incomplete or complete and has defects.
- sandGorgon 7y ago
  so would each complete logical part be a separate branch with its PR ? and then branch from the previous branch to create the next PR? I'm just wondering if you found there to be any cognitive overhead here.

0xDEFC0DE 7y ago

Unlike the other code-slinging gods who walk amongst us in this thread, I prefer coding challenges because I'm pretty mid-level and average and I do actually need a job. And I hate whiteboarding. No one should do whiteboarding for code.

shantly 7y ago
Agreed. Make me whiteboard or live-code if you wanna make me look and feel like I've not actually had over a dozen years of productive experience designing systems and shipping code and fixing other people's broken crap, and you'd love to watch me stress out and forget how to write a damn loop so you can complain on HN about how I was clearly lying about my experience & abilities and that's why you have to make people live-code or whiteboard. Challenges are, at least, preferable to that, for sure.
lfowles 7y ago
Same, I've found that I get further in the interview process when dealing with companies that screen with a coding challenge.
traderjane 7y ago
Whiteboarding is like coding except you don’t need to get the syntactical details of your loop right. It’s writing out your thoughts not just in pseudo code, but perhaps also in freehand visualizations. This makes it the most ergonomic and beginner friendly programming language ever.

jonfw 7y ago

> “I’ve been hiring people for 10 years, and I still swear by a simple rule: If someone doesn’t send a thank-you email, don’t hire them.” I was contacted by a recruiter a short while ago asking my availability for a phone screen- and I replied simply with my availability. The recruiter still hasn't gotten back to me, and I think that it may be because I didn't go out of my way to thank them for reaching out to me. Are all of the niceties in your email inbox really that important to people? It seems pretty manufactured and boilerplate when people send me emails that they "hope find me well". I may just have to get over it and start letting people know how "excited I am to speak with you and learn more about this opportunity!"

alexk 7y ago
Just to make it clear - I posted this as an example of anti-pattern, not the actual advice to follow :)
olooney 7y ago
Yes. They're not testing your knowledge of etiquette, people skills, or friendliness though. They're looking for a more basic and far more widely applicable skill: following best practices. If you were configuring Apache for the first time, wouldn't you google for a list of do's and don'ts and spend a few minutes implementing the low-hanging fruit? Not doing that puts you at a huge risk of leaving a huge security hole or massive performance hit, right? It would be amateurish, and if a problem occurred because you didn't take even a few minutes to do the most basic and widely known stuff, you would have no excuse. Likewise, while nobody really cares about thank you emails, not doing that suggests you didn't even bother to Google "Interview Tips" and implement the ones that took less than a few minutes. That's not a good signal to send to people that are going to have to rely on your professionalism and quality of work for the next few years.
- jgwil2 7y ago
  This would make sense except that the job isn't interviewing for jobs. You shouldn't hold people to the expectation that they're going to approach communicating with recruiters in the same way they would approach actual work they were actually getting paid for. This is a great way to filter good candidates out based on something that has nothing to do with the work you are hiring for.

deeg 7y ago

I do not want to work for a company that does NOT test my technical skills; if they don't verify my skills that means they don't verify others and I'd prefer to work skilled engineers (with exceptions for entry-level or junior engineers). Most people seem to agree that white-boarding is a poor way to judge technical skills. Coding challenges seem a good compromise, assuming they don't require more than a few hours of time.

kemiller2002 7y ago
I do get what you're saying, but what I'm really interested in is what are their skills? Any moron can tell me I'm good whether I am or not.
- recursive 7y ago
  But the chance of getting it right goes down if they don't have direct evidence of their conclusion.
kevin_nisbet 7y ago
I agree. I kind of wish https://www.kalzumeus.com/2015/03/09/announcing-starfighter/ https://www.kalzumeus.com/2015/03/09/announcing-starfighter/ had taken off, as instead of doing a coding challenge for every company, you can do one really well designed challenge, and use that challenge with multiple companies. Note: I'm an employee of gravitational.
- deleted 7y ago
  [deleted]
noirbot 7y ago
I had a similar process recently. Went from recruiter through to offer with only a couple design problems and generally talking about projects I'd worked on, no code whatsoever. If anything, it made me want to work for the company less. I didn't feel like they had a great idea if I was actually good at my job, since almost everything I'd shown in interviews was things I could have easily made up to make myself look good. That's especially worrying to me because it was a company that was still building out the team I'd be working for, so it would be hard for me to be really confident in the skills of my coworkers. Maybe the interviewers were just really good judges of character and skill (I think I probably was a good fit for the position), but it was hard to look past the fact that I didn't think I'd proven that I fit the role to them. Not doing any technical/coding problems makes me worry that the company doesn't see technical skill as important at all. Maybe they could preface the process with an explanation of why they don't, and that could be persuasive, but outside that, even a basic FizzBuzz makes me feel like their hiring philosophy is considering those skills.

kemiller2002 7y ago

I get why companies like them, and there is nothing wrong with them. The truth is that I probably don't want the job enough to actually do it. It takes time away from my family, and my friends and most of the time, it's not that stellar of a place to work anyway. If I'm desperate for a job, I could feasibly see doing it, or if I thought it was a really amazing opportunity. The truth is that I'm probably not going to be in either of those positions. What really does irk me is when a company says you have to do it before they'll even talk to me. I'm definitely not going to spend several hours on something without even knowing anything about the company.

rdelval 7y ago
The asymmetric time investment required from candidates is what bothers me the most about these coding challenges. My personal experience has been that it's a lopsided process with the candidate putting in hours of unpaid work and the company spending one, maybe two engineering hours if we're being generous, reviewing the submission. If the reviewer decides to reject the challenge, the loop closes, HR sends a nice e-mail, and the candidate has nothing to show for it.

tptacek 7y ago

Unfortunately, the idea behind coding challenges has been cargo culted into a lot of weak hiring cultures: * The challenges often don't actually matter, and are simply part of the recruiting hazing ritual. * They're totally subjective and just a proxy for the same bullshit friends-and-resumes process people were using already. * No consideration is given to the amount of time the entire recruiting process is taking. * The challenges themselves are totally divorced from the day-to-day expectations of the job, so people are doing CS102 algorithms problems for jobs that will mostly be about moving bits from one place to another. I understand why people don't like them. But that doesn't really matter, because code challenges are so much more effective at qualifying people that it would be crazy for any competent shop to stop using them. For a couple years (and, of course, for many years before at Matasano) we've been hiring resume-blind and with almost no interviewing (we do a final on-site, but it's a formality with no tech-out, which is something we tell candidates). We calibrate our challenges so that they'd ideally, for a qualified candidate, take less time than interviews and phone screens would, and have the advantage of being scheduled at the candidate's convenience (all in one night, spread out over a couple weekends, whatever). We have candidates coming out of our ears (volume has been the single biggest problem with our recruiting practice), and the quality of those candidates is just as good as anyone else's comparable funnel. This is the right way to hire technical staff.

hyperpape 7y ago
Do you think it matters that you're unusually outspoken and/or prominent? The cargo-culting actually seems like a problem for both employers and applicants, because it poisons the market. You have to know whether the company is treating coding challenges seriously. Because a company that will ghost you or decline you for unclear reasons after giving you a coding challenge has wasted a lot of time. Or put another way, if it's a hoop to jump through for candidates, coding challenges are awful. If it really means a good chance at that $10k+ raise, coding challenges are a great investment.
- tptacek 7y ago
  I'm sure it matters a lot that I'm loud. But that's not a superpower; in fact, my friends and many of my coworkers would say it's the opposite. Nothing stops other hiring managers from doing the same thing.

rileymat2 7y ago

I do not mind a coding challenge in the hiring process if it is the last step, not the first step.

deleted 7y ago

[deleted]

a0zU 7y ago

Coding challanges are used pretty often to scam programmers to do work for free before the company rejects them.

twic 7y ago
I don't believe this actually happens. Do you have any concrete examples?
- mamon 7y ago
  There's this infamous company from Austin, Texas, which hires programmers for remote work, and they typically give you coding "challenges" that require a week to complete, and are complete (although not very complex) applications.

IshKebab 7y ago

Great article, although 4-8 hours is clearly way too long. 2 hours is much more reasonable.

mooreds 7y ago

I didn't see "pay candidates for their time" as a section even though I saw several mentions of how invested the company was (CEO commenting on PRs, team investment). Saying "We will pay you to do this exercise" is a simple (and cheap!) way to show you value candidate's time.

etxm 7y ago

They’re bull’s excrement, this industry’s interview process is broken and arbitrary. I don’t know if we’ll ever fix it, but code challenges certainly aren’t the answer. I’ve been developing professionally for 20 years. I take about 4 interviews a year when _not_ looking for a job to stay practiced. Held DoE, Staff, and Principle roles. Anecdotal evidence from this year: - given a decent sized MVC app with bugs and missing features. Tasked to green the test suite. PR comment from a pretty established engineer was “this is perfect, I wouldnt change anything.” I used 1 of my 4 hours and thought my work was poor - given a “simple”[1] sort and map/reduce a large log file and didn’t finish after 2 hours because the interviewer kept interrupting to ask me about the details of how internals of the language and GC worked - things I was familiar with, but I can’t focus on code and answering questions at the same time. I was ranked as “junior” and not a fit for the role, cute. This last one rolls up into one of the problems I have (and assume others have) in interviews: “work through the problem as you would, but talk out loud about your process” If I’m talking, I’m not in my zone. Let me shut the fuck up and solve the problem and we can talk about it afterwards. [1] “simply” parse this 500k line CSV file and aggregate some results, no dependencies. CSV file was actually a STDOUT log file that multiple containers were writing to with different formats, JSON, CSV, TSV, SSV, and random ass unstructured logs from a prod system.

etxm 7y ago

I’ve posted on here before about when I’m in a hiring role how I prefer to do code challenges. I enjoy doing BYOC challenges. Let a candidate pick a ticket from an open source project and come in to pair on it (or shoulder surf if uncomfortable with pairing). I don’t see the issue until the day we are pairing. Goal is to let them be the expert and as comfortable as possible. They get some code on their GitHub, open source benefits, and I get a peak into their engineering skills. If you’re thinking, “hey that’s not apples to apples,” well people aren’t fucking apples. I had someone come in once that didn’t have an “issue.” She had been working with a library that had poor docs and wanted to work through the library and document it. THAT! That was an awesome experience. Best engineer? I had no idea. Willing to bang her head against a wall to make the experience better for others. Yes. Fuck yes, get on my team.

pickle-wizard 7y ago

My current job had a coding challenge and I felt they handled it pretty well. It was done one site as part of the interview process. It was related to the work I'd be doing, however it wasn't super complicated. They expected that you'd be able to finish it in about 30 minutes, but you had as much time as you needed. I finished it in about 20 minutes. Since it was over lunch time, they bought me lunch. I figure lunch for 20 minutes of work was a fair reimbursement.

iliaznk 7y ago

I’ve passed through a screening procedure recently that consisted of three steps: 1) They asked for some publicly available samples of my code «I’m proud of». Okay, I prepared three samples from different areas with detailed descriptions and comments, explaining what that code did and why it was good. 2) After that, as a coding challenge, they asked me to write a simple app using the Unsplash API. 3) Finally we had an over an hour long interview during which I was again given three simple coding problems. And then they rejected me. The question I have after all that is do you really need a multi-staged procedure like that to determine a candidate is not a good fit?! When exactly did they decide I wouldn’t fit? If that was after step 1 or 2, why didn’t they stop right there? If that was only after step 3, then what’s the purpose of steps 1 and 2? They don’t mean anything if step 3 can just cancel them out. Then why didn’t they start with step 3 not to waste my and their time?

jgwil2 7y ago

The process outlined in the article shows respect for candidates in many ways, but especially by providing feedback. I have no problem going through an in-depth interview process, because I can always look at it as a learning experience, but it can be very frustrating to go through a process and then not receive feedback, positive or negative, on your performance. I understand that all employers are busy, engineers' time is valuable, and that often communication is managed by HR people who don't have in-depth knowledge of the reasoning behind decision-making, but I think that making constructive feedback an expectation for the hiring process would improve the experience for candidates manifold, and ultimately would result in less arbitrariness in hiring decisions.

rajacombinator 7y ago

I’ll only do these if 1) company pays me or 2) I’m exceptionally interested and think the comp will justify it. (ie. Almost never.) Spending several hours on something that might not even get reviewed - no thanks.

kureikain 7y ago

This seems got some negative review. Let me shared my experience. Alexander is fantastic as an interviewer and co-worker. I was interview by ALEXANDER a few years ago. I ended up not getting an offer. It was my fault. The challenge is the hardest thing I have to research on my own to implement because google found no result for me at the time. I learned a lot when working on his challenge and the idea of being able to solve it is more imporant than passing to interview to me. The way they conduct interview is really great. You are given help by their team. All I can say is Gravitational takes hiring to their heart and setup for success of candidate. TO Alex, I'm Vinh if you remember me :-).

alexk 7y ago
Hey Vinh, thank you for the kind words and I'm glad that you got a positive experience with our challenge and the process!