5 ms·

My startup got acquired last year so I haven't interviewed anyone in a while, but my technical interview has always been: - share your screen - download/open

My startup got acquired last year so I haven't interviewed anyone in a while, but my technical interview has always been: - share your screen - download/open the coding challenge - you can use any website, Stack Overflow, whatever, to answer my questions as long as it's on the screenshare My goal is to determine if the candidate can be technically productive, so I allow any programming language, IDE, autocompleter, etc, that they want. I would have no problem with them using GPT/Copilot in addition to all that, as long as it's clear how they're solving it.

random_walker 2y ago
I love these kind of interviews. This would very closely simulate real world on-job Performance.
- dalmo3 2y ago
  If I had to do real world on-job coding while someone looks over my shoulder at all times (i.e. screensharing), I'd be flipping burgers.
silasdavis 2y ago
I don't care how you're good at it so long as I can watch.
b112 2y ago
I'd be fine with the GPT side of things, as long as I could somehow inject poor answers, and see if the interviewee notices and corrects.
- cpursley 2y ago
  That's actually a horribly awesome idea.
- htrp 2y ago
  the trick is to phrase the problem in a way that GPT4 will always give the incorrect answer (due to vagueness of your problem) and that multiple rounds of guiding/correcting are needed to solve.
  - gtirloni 2y ago
    That's pretty good because it can exhaust the context window quickly and then it starts spiraling out of control, which would require the candidate to act.
    - htrp 2y ago
      If you only use ChatGPT to code, you are only able to copy paste the llm emitted code, then you ask for changes to the code (to reflect for example the evolution of the product)
- hibikir 2y ago
  There's more than one possible AI on the other end, so crafting something that will not annoy a typical candidate, but will lead every AI astray seems pretty difficult.
  - notpushkin 2y ago
    Maybe you could allow using AI, but only through the interviewer-provided interface. That interface would allow using any model the candidate likes, but before sending the response it will inject errors into the code (either randomly or through another AI prompt).
OutOfHere 2y ago
[flagged]
- evilduck 2y ago
  It's pretty obvious when someone's input focus changes to nothing or when their mouse leaves the screen entirely, or you could just ask to see the display settings to begin. Doesn't solve for multiple computers but it's pretty obvious in real time when someone's actual attention drifts or they suddenly have abilities they didn't have before. Either way, screen sharing beats whiteboards. Even if we throw our hands up and give up, we'll be firing frauds before the probationary period ends.
  - OutOfHere 2y ago
    There is nothing fraudulent about using LLMs. If people can use them on the job, it's okay to use them on the interview. They're the calculators of tomorrow if not of today. Interviewing just needs to adapt such as by assessing one's open source projects and contributions. Not much more is needed. And if the candidate completely misrepresents their open source profile, this can be handled by an initial contract-to-hire period.
    - kortilla 2y ago
      > There is nothing fraudulent about using LLMs. There is if you’re asked not to.
      - OutOfHere 2y ago
        Negative. They are not the law.
        kortilla 2y ago
        What are you talking about? If you lie to get a job, you are committing fraud. The company is not making any law here.
shaneoh 2y ago
I recently interviewed for my team and tried this same approach. I thought it made sense because I want to see how people can actually work and problem solve given all the tools at their disposal, just like on the job. It proved to be awkward and clumsy very quickly. Some candidates resisted it since they clearly thought it would make them judged harsher. Some candidates were on the other extreme and basically tried asking ChatGPT the problem straight up, even though I clarified up front "You can even use ChatGPT as long as you're not just directly asking for the solution to the whole problem and just copy/pasting, obviously." After just the initial batch of candidates it became clear it was muddying things too much, so I simply forbade using it for the rest of the candidates, and those interviews went much smoother.
- layer8 2y ago
  Did you tell them that you “want to see how people can actually work and problem solve given all the tools at their disposal, just like on the job”? Just curious.
  - shaneoh 2y ago
    Yup, we told them exactly that.
- Aeolun 2y ago
  What are you supposed to ask chatGPT if you can’t just ask it the answer? That’d confuse me too.
  - datavirtue 2y ago
    Just another interview methodology pulled out of someone's ass. They don't know.
    - edanm 2y ago
      As opposed to all other interviewing methodologies which are rigourously tested? Unfortunately in our industry it's pretty much all personal anecdotes on what works better and what doesn't.
staticautomatic 2y ago
I did this while hiring last year and the number of candidates who got stuff wrong because they were too proud to just look up the answer was shocking.
- emseetech 2y ago
  Is it pride or is it hard to shake the (reasonable, I'd say) fear the reviewer will judge regardless of their claims?
  - ryandrake 2y ago
    Exactly. You never know. Some interviewers will penalize you for not having something memorized and having to look it up, some will penalize you for guessing, some will penalize you for simply not knowing and asking for help. Some interviewers will penalize you for coming up with something quick and dirty and then refining it, some will penalize you for jumping right to the final product. There's no consistency.
  - staticautomatic 2y ago
    I do what I can to allay that fear. The rest is up to them.
bagels 2y ago
If you really don't penalize them for this, you should clearly state it. Some people may still think they'll be penalized as that is the norm.
blazing234 2y ago
im not doing any coding challenges that aren't real world if i see anything remotely challenging i dip out. interviewing is just a numbers game nowadays so i dont waste time on interviews if they seem like they're gonna burn me out for the rest of the day. granted i have 11 years experience
yieldcrv 2y ago
> I would have no problem with them using GPT/Copilot in addition to all that, as long as it's clear how they're solving it. Too many people are the opposite that I would literally never tell you And this works. what can we do to help that? I’ve had interviews where AI use was encouraged as well. but so many casual tirades against it dont make me want to ever try being forthcoming. most organizations are realistically going to be 10 years behind the curve on this
hibikir 2y ago
The difficulty of your questions have to change drastically if they are using good tooling. Many a problem that would take a reasonable candidate half an hour to figure out is 'free' for Claude, so your question might not show any signal. And if you tweak your questions to be sure to not be auto-solved by a strong enough AI, then you better say it's semi-required, because the difficulty level of the question you need to ask goes up quite a bit. Some of the questions in our interview loop have been posted in github... which means every AI has trained on them specifically. They are, therefore, useless if you have AI turned on. And if you interview enough people, someone will post on github, and therefore your question will have a pretty short shelf life before it's in training and instantly solved.
pnathan 2y ago
Screen share or in person are what I think the best ways are. These are not the best options. I do not want AI. The human is the value add. I understand that people won't feel super comfortable with this, and I try not to roast the candidate with leetcode. It should be a conversation where I surface technical reality and understanding.

random_walker 2y ago

I love these kind of interviews. This would very closely simulate real world on-job Performance.

dalmo3 2y ago
If I had to do real world on-job coding while someone looks over my shoulder at all times (i.e. screensharing), I'd be flipping burgers.

silasdavis 2y ago

I don't care how you're good at it so long as I can watch.

b112 2y ago

I'd be fine with the GPT side of things, as long as I could somehow inject poor answers, and see if the interviewee notices and corrects.

cpursley 2y ago
That's actually a horribly awesome idea.
htrp 2y ago
the trick is to phrase the problem in a way that GPT4 will always give the incorrect answer (due to vagueness of your problem) and that multiple rounds of guiding/correcting are needed to solve.
- gtirloni 2y ago
  That's pretty good because it can exhaust the context window quickly and then it starts spiraling out of control, which would require the candidate to act.
  - htrp 2y ago
    If you only use ChatGPT to code, you are only able to copy paste the llm emitted code, then you ask for changes to the code (to reflect for example the evolution of the product)
hibikir 2y ago
There's more than one possible AI on the other end, so crafting something that will not annoy a typical candidate, but will lead every AI astray seems pretty difficult.
- notpushkin 2y ago
  Maybe you could allow using AI, but only through the interviewer-provided interface. That interface would allow using any model the candidate likes, but before sending the response it will inject errors into the code (either randomly or through another AI prompt).

OutOfHere 2y ago

[flagged]

evilduck 2y ago
It's pretty obvious when someone's input focus changes to nothing or when their mouse leaves the screen entirely, or you could just ask to see the display settings to begin. Doesn't solve for multiple computers but it's pretty obvious in real time when someone's actual attention drifts or they suddenly have abilities they didn't have before. Either way, screen sharing beats whiteboards. Even if we throw our hands up and give up, we'll be firing frauds before the probationary period ends.
- OutOfHere 2y ago
  There is nothing fraudulent about using LLMs. If people can use them on the job, it's okay to use them on the interview. They're the calculators of tomorrow if not of today. Interviewing just needs to adapt such as by assessing one's open source projects and contributions. Not much more is needed. And if the candidate completely misrepresents their open source profile, this can be handled by an initial contract-to-hire period.
  - kortilla 2y ago
    > There is nothing fraudulent about using LLMs. There is if you’re asked not to.
    - OutOfHere 2y ago
      Negative. They are not the law.
      - kortilla 2y ago
        What are you talking about? If you lie to get a job, you are committing fraud. The company is not making any law here.

shaneoh 2y ago

I recently interviewed for my team and tried this same approach. I thought it made sense because I want to see how people can actually work and problem solve given all the tools at their disposal, just like on the job. It proved to be awkward and clumsy very quickly. Some candidates resisted it since they clearly thought it would make them judged harsher. Some candidates were on the other extreme and basically tried asking ChatGPT the problem straight up, even though I clarified up front "You can even use ChatGPT as long as you're not just directly asking for the solution to the whole problem and just copy/pasting, obviously." After just the initial batch of candidates it became clear it was muddying things too much, so I simply forbade using it for the rest of the candidates, and those interviews went much smoother.

layer8 2y ago
Did you tell them that you “want to see how people can actually work and problem solve given all the tools at their disposal, just like on the job”? Just curious.
- shaneoh 2y ago
  Yup, we told them exactly that.
Aeolun 2y ago
What are you supposed to ask chatGPT if you can’t just ask it the answer? That’d confuse me too.
- datavirtue 2y ago
  Just another interview methodology pulled out of someone's ass. They don't know.
  - edanm 2y ago
    As opposed to all other interviewing methodologies which are rigourously tested? Unfortunately in our industry it's pretty much all personal anecdotes on what works better and what doesn't.

staticautomatic 2y ago

I did this while hiring last year and the number of candidates who got stuff wrong because they were too proud to just look up the answer was shocking.

emseetech 2y ago
Is it pride or is it hard to shake the (reasonable, I'd say) fear the reviewer will judge regardless of their claims?
- ryandrake 2y ago
  Exactly. You never know. Some interviewers will penalize you for not having something memorized and having to look it up, some will penalize you for guessing, some will penalize you for simply not knowing and asking for help. Some interviewers will penalize you for coming up with something quick and dirty and then refining it, some will penalize you for jumping right to the final product. There's no consistency.
- staticautomatic 2y ago
  I do what I can to allay that fear. The rest is up to them.

bagels 2y ago

If you really don't penalize them for this, you should clearly state it. Some people may still think they'll be penalized as that is the norm.

blazing234 2y ago

im not doing any coding challenges that aren't real world if i see anything remotely challenging i dip out. interviewing is just a numbers game nowadays so i dont waste time on interviews if they seem like they're gonna burn me out for the rest of the day. granted i have 11 years experience

yieldcrv 2y ago

> I would have no problem with them using GPT/Copilot in addition to all that, as long as it's clear how they're solving it. Too many people are the opposite that I would literally never tell you And this works. what can we do to help that? I’ve had interviews where AI use was encouraged as well. but so many casual tirades against it dont make me want to ever try being forthcoming. most organizations are realistically going to be 10 years behind the curve on this

hibikir 2y ago

The difficulty of your questions have to change drastically if they are using good tooling. Many a problem that would take a reasonable candidate half an hour to figure out is 'free' for Claude, so your question might not show any signal. And if you tweak your questions to be sure to not be auto-solved by a strong enough AI, then you better say it's semi-required, because the difficulty level of the question you need to ask goes up quite a bit. Some of the questions in our interview loop have been posted in github... which means every AI has trained on them specifically. They are, therefore, useless if you have AI turned on. And if you interview enough people, someone will post on github, and therefore your question will have a pretty short shelf life before it's in training and instantly solved.

pnathan 2y ago

Screen share or in person are what I think the best ways are. These are not the best options. I do not want AI. The human is the value add. I understand that people won't feel super comfortable with this, and I try not to roast the candidate with leetcode. It should be a conversation where I surface technical reality and understanding.