10 ms·
I've had a displeasure of interviewing someone who used ChatGPT in a live setting. It was pretty obvious: I ask a short question, and I say that I expect a shor
by m1el 3y ago
I've had a displeasure of interviewing someone who used ChatGPT in a live setting.
It was pretty obvious: I ask a short question, and I say that I expect a short answer on which I will expand further. The interviewee sits there in awkward silence for a few seconds, and starts answering in a monotone voice, with sentence structure only seen on Wikipedia. This repeats for each consecutive question.
Of course this will change in the future, with more interactive models, but people who use ChatGPT on the interviews make a disservice to themselves and to the interviewer.
Maybe in the future everybody is going to use LLMs to externalize their thinking. But then why do I interview you? Why would I recommend you as a candidate for a position?
- ptmcc 3y agoYes of course! I'd be happy to answer your short question with a short answer. I look forward to expanding further on the answer, as you previously stated that you expect me to. Jokes aside, something about LLM responses is very uncanny valley and obvious.
- chewxy 3y agoThe peppy, upbeat, ultra-American tone that the LLMs produce can be somewhat toned down with good prompting but ultimately, it does stink of the refinement test set.
- throwaway2037 3y agoTrue. We need an Aussie bogan mode for ChatGPT. Or, Guy Ritchie villian.
- renewiltord 3y agoThat sounds great, doesn't it? You got powerful negative signal.
- lcnPylGDnU4H9OF 3y agoIt sounds like the problem is really that this is the most obvious cheater. Someone better at manipulation and deception might do a better job cheating the interviewer such that they're hired but then be entirely inadequate in their new position.
- deleted 3y ago[deleted]
- m1el 3y agoOh, and to add an insult to the injury, I was using a collaborative editing tool. So I was able to see the person: 1) Select All (most likely followed by the copy) 2) Type the answer 3) Make an obvious mistake when they type else block, before the if
- frabjoused 3y agoThat was me interviewing someone yesterday. The telltale select all is so cringe.
- pests 3y agoSome people compulsively highlight what they are reading.
- jcranmer 3y agoI'm a compulsive highlighter too, but it's generally in the vein as xkcd (https://xkcd.com/1271/ https://xkcd.com/1271/) and not a select all. Frequently, highlighting ends up starting in the middle of a word!
- anonymouskimmer 3y agoOn some modern websites I end up having to select all because the selection mechanism is broken (I'll highlight, remove my finger from the mouse button, highlight another piece of text, and yet the original highlighted text is still totally, or even worse partly, highlighted.). Crtl-A Selecting all and then clicking anywhere is the only way to clear all of the highlighting in these instances. Thanks for the XKCD. I didn't realize how common this is. Now I'm even more annoyed that so many websites and reader apps force context menus or 'gestures' when you highlight, without a way to disable those context menus or gestures.
- baby_souffle 3y agoExactly. Ever go to school with a dyslexic that's using a ruler to expose one line of text at a time? Same thing....
- blharr 3y agoThe idea that spotting cheating is obvious is a case of selection bias. You only notice when it's obvious. Clearly, the person put 0 effort towards cheating (as most cheaters would, to be fair). But slightly adjusting the prompt, or just paraphrasing what ChatGPT is saying, would make the issue much harder to spot.
- al_borland 3y agoMaybe I’m a slow reader, but reading, understanding, and paraphrasing the response seems like it would take enough time to be awkward and obvious as well. I’m not sure why anyone would want a job they clearly aren’t qualified for.
- xboxnolifes 3y ago> I’m not sure why anyone would want a job they clearly aren’t qualified for. $$$,$$$
- dmoy 3y agoWell, five moneys at least. They might figure out and fire you before you get to six moneys (but maybe they won't, who knows).
- WalterBright 3y agoIt will take 3 to 6 months to determine that a new hire is incompetent, especially if you're required to document their incompetence before firing them.
- skeeter2020 3y agoI've never had a job without a probation period where you can let someone go without cause within the first 90 days with nothing more than two weeks pay in lieu of notice. It definitely doesn't take 6 months to identify someone who only got their job because they used AI in the interview.
- duxup 3y agoI’ve wondered how much of the appeal of LLMs is for humans to BS other humans.
- foxyv 3y agoConsidering how much time is spent on manufacturing BS for consumption by bosses, professors, teachers, and advertising? I think this is going to automate at least half of the work office workers and students are doing now...
- outside415 3y agoMe and several friends have used ChatGPT in live interviews to supplement answers to topics we were only learning in order to bridge the gap on checkboxes the interviewer may have been looking for. We’ve all got promotions by changing jobs in the last 6 months using this method. You can be subtle about it if it’s already an area you kind of know.
- jacques_chester 3y agoSo, assuming they didn't know and approve, you cheated.
- lcnPylGDnU4H9OF 3y agoDirty, dirty cheater! Sounds like they would have been able to perform the job duties so I'm not sure why one should care.
- jacques_chester 3y agoThat someone has the skills for a job is distinct from whether they are able to uphold a simple moral principle like "don't cheat".
- lcnPylGDnU4H9OF 3y agoThe interviewer is full of themself if they think someone who can do the job cheated in the interview.
- ThrowawayR2 3y agoThose who lie about one thing are likely to lie about many others.
- InvertedRhodium 3y agoGiven that literally every person alive lies about some things, I'm not sure how much value that observation brings to the discussion.
- foxyv 3y agoTo be honest, I think in the future we will interview people on their ability to work with an LLM. This would be a separate skill from the other ones we are looking for. Maybe even have them do some fact checks on a given prompt and response as well as suggest new prompts that would give better results. There might even be an entire AI based section of an interview. In the end, it's just a new way to "Google" the answer. After all, there isn't much difference between reading off an LLM response and just reading the Wikipedia page after a quick Google search, except for less advertisements.
- jacques_chester 3y agoI agree that this is the likely long term outcome. But for now folks want to think that everyone needs to have memorized every individual screw, nail, nut and bolt in the edifice of computer science.
- SOLAR_FIELDS 3y agoI’ve already been allowed to use it in programming interviews where they’ve said it’s explicitly allowed to use ChatGPT. It’s led to some fun interactions because I use it a lot and as such I’m quite good with it and interviewers are often taken aback by how quickly I’m able to just destroy the question they put out with a good prompt I will say there are still some programming questions you can give that will stump the hell out of ChatGPT. In particular I took one online coding assessment where I used it and there was a question about plotting on a graph with code and calculating areas based on the points plotted that ChatGPT failed miserably at, but someone pretty good with math and geometry would find pretty tractable.
- fragmede 3y agoThere are ChatGPT resistant questions you can ask. ChatGPT recognizes the question but doesn't actually think about it, so if you give it the river crossing problem (farmer, fox, sheep, and grain need to cross a river) but tell it the boat can take all the items, it won't actually read those details and blithely solve the problem the expected way. Give candidates a problem that's trivially solvable if you actually read the question and see if they try and solve it the ChatGPT way.
- lmm 3y ago> But then why do I interview you? Why would I recommend you as a candidate for a position? Presumably you have tasks that you want performed in exchange for money? (Or want to improve your position in the company hierarchy by having more people under you or whatever).
- JohnFen 3y ago> Of course this will change in the future, with more interactive models I think that what will change is that doing interviews remotely will become rarer, in favor of in-person interviews.
- scarlson 3y agoWhy? Interviewing as a process sucks enough as it is. It should just be a culture fit filter that takes you all of 15 minutes to say yes or no to. Technical interviews are lame and filter for people that are good at technical interviews, not people that are good at the job.
- plasticchris 3y agoThat works in a world where everyone is technically competent, but oddly many people applying to software positions are, optimistically, planning to learn on the job. Work with enough folks like that at once and the motivation for the coding interview becomes clear.
- JohnFen 3y ago> Interviewing as a process sucks enough as it is. It truly does, and it sucks just as much for the employer as for the applicants. That's why I suspect that more interviews will be required to be in person: if it's too easy for someone to cheat, that makes everything suck even more for the employer and the employer is likely to adjust the process to minimize that suckage. > Technical interviews are lame and filter for people that are good at technical interviews, not people that are good at the job. Not automatically, but yes, bad technical interviews filter for people who are good at technical interviews. And too many interviews (technical or otherwise) are bad.
- LgWoodenBadger 3y agoLame? It’s a bare minimum demonstration of ability. The number of experienced candidates I’ve interviewed just in the past few months who have trouble writing a for-loop in the language they’re “experienced” in might astound you. Welders sometimes (always?) have to go to a certification center to demonstrate that they can actually perform the types of welds the job they’re applying for requires. https://www.aws.org/Certification-and-Education/Professional-Certification/Certified-Welder-Program/?step=1 https://www.aws.org/Certification-and-Education/Professional...
- appleiigs 3y agoYou're not asking the correct questions as an interviewer. You should be asking specific questions about projects they've worked on, or about them personally to get to know them. ChatGPT should not be able to answer. Pretend you're Harrison Ford in Blade Runner.
- mvdtnz 3y agoChatgpt can easily be instructed to tell a tale about a project it has worked on. It will expand on fake details when pressed.
- Kranar 3y agoAs an experiment I gave ChatGPT my resume and background information and then pretended to interview it, just to see how well it would be able to conduct a mock interview. It did exceptionally well. I'm not sure what specific questions you have in mind, but ChatGPT is almost certainly trained on a vast array of resumes and a diverse range of profiles, possibly even all of LinkedIn itself as well as other job boards. There is little to no reason why it wouldn't be able to make up an entire persona who is capable of passing most job interviews.
- makeitdouble 3y agoYou ask many kind of questions. A candidate can do very well on personal and web project experience questions, and suddenly blank when you ask them how an http request is structured. Or what's CORS. Then you dig further and discover a lot more thing about them that wouldn't have surfaced otherwise because hou assumed they knew all of that. My best advice would be to never skip "dumb" and easy technical questions. You can do it very quick, and warn ahead that it's dumb questions but you ask them to everyone.
- throwaway2037 3y agoI agree with the "drill down" technique. Example: How does a dynamic array class (Vector, List, etc.) work? The very best interview questions have "fractal complexity" that allow you to drill deep.
- 3y ago
- m3kw9 3y agoWould get easier with an API that connects with stuff like whisper and voice cloning and a good prompt
- Espressosaurus 3y agoThis seems readily fixable by doing the interview in-person.
- esafak 3y agoThen you'd filter them by resume first to manage costs. Pick your poison.
- smcin 3y agoHow much can you mitigate this by interviewing them remotely but on video? Then you can see if they're typing and reading the answer (unless they have a friend doing that and feeding them it in an earphone, as I hear happens).
- outworlder 3y ago> The interviewee sits there in awkward silence for a few seconds, and starts answering in a monotone voice, with sentence structure only seen on Wikipedia. This repeats for each consecutive question. That's a bit better than proxy interviews and people lip syncing, but not by much.
- nyc_data_geek1 3y agoUsing LLM's isn't externalizing or outsourcing thinking. LLM's aren't performing that. People doing this are in fact substituting thinking with a process, the output of which masquerades as thoughts after a fashion, but are in fact basically word cloud probability based pattern matching. Sure, the point that superior tool use is a valid job skill makes some sense, but conceding your agency and higher reasoning to a machine which possesses none of these is to my mind not going to be beneficial to a business in the long run.
- neilv 3y ago> but people who use ChatGPT on the interviews make a disservice to themselves and I think most people have been thinking that the interviews are mostly BS with little relationship to the job, which you simply have to get through. Many, many people will cheat to the extent that they think they can get away with it. It's a bit like many people cheat in school. (On classes they consider irrelevant, they might justify it that way. On classes relevant, they might justify it, that passing or their GPA is more relevant to their goals, than learning that material at that time.) I think people generally don't believe a "you're doing a disservice to yourself" argument. They choose the tradeoff or the gamble. Personally, I don't tolerate cheating, and I have a low tolerance for interview BS. Neither is the dominant strategy for the current field.
- WalterBright 3y agoAt Caltech, exams were typically open book, open note. The time limit on the test, however, prevented attempts to learn the material in the time allotted. Calculators were also allowed (though were useless on Caltech exams, as course material didn't care about your arithmetic skillz). I suspect the way to deal with ChatGPT is to allow it. Expect the interviewee to use ChatGPT as a tool. Try out the interview questions beforehand with ChatGPT. Ask questions that ChatGPT won't be good and answering, like how a calculator is useless on a physics exam.
- xarope 3y agoAs a former tertiary educator (for a brief moment, before I decided academia wasn't my thing), that's how open book exams are set; the assumption is you have knowledge of the subject, and the books are there for you to verify and quote examples of/from. NOT to browse through looking for a solution from step 0.
- wakawaka28 3y agoUsing ChatGPT as a tool makes as much sense as allowing a human assistant to take the exam with you. In an open-book test, you have to know what you're looking for and roughly where to find it in the book. That implies some knowledge. With ChatGPT you could type the question verbatim and get a potentially right answer, without even understanding the answer at all. It is therefore unacceptable for use on any exam.
- batch12 3y agoI've had this happen too, with almost the same responses. It was even more obvious because I was able to see the reflection of their lcd backlight glowing across their face as they switched back and forth to answer the questions. I just directly asked if they were using an external resource to answer my questions. They said yes as if it was normal. I thanked them for their time as that was my last question.
- jliptzin 3y agoYou should just openly let them use chatgpt (assuming they can use it on the job too). When I interview people I try to create the same environment as the one they’ll be working in. They can use chatgpt, google, stack overflow, etc. I don’t care how many tools they have to use, as long as the work output is good and done in a reasonable time. I really don’t understand the obsession with coding on whiteboards or other situations that will literally never come up on the job. There will never be a time my employees can’t use google or chatgpt. In any case, you can tell pretty quickly how much someone knows about a topic just based on the questions they’re asking chatgpt.
- recursive 3y agoIf they're just putting the question straight into GPT, then what benefit is the candidate bringing? I can use GPT myself, and for a lot cheaper than the cheapest candidate.
- kmoser 3y agoIf the interview is for a position in which the candidate will be tasked with solving problems that ChatGPT is able to help with significantly, then they have just proven they are capable of doing the job. (If you have time to do this work yourself, why are you interviewing anybody at all?) Assuming the interview is to determine somebody's programming chops, without the benefit of ChatGPT, you'll have to ask questions where ChatGPT is little to no help. This was the conclusion of the article.
- seattle_spring 3y agoBut how will I know if they can implement a function to rotate a binary tree from memory?
- kmoser 3y agoWhy does it matter to you if they can do it from memory, if they can find the answer easily from ChatGPT? It's like asking "how can I tell if somebody knows the exact definition of a word without having to look it up?" If it's really important that they have that ability (e.g. because you will be asking them to perform other tasks which are not so easily solved by ChatGPT, or you simply don't want them using ChatGPT at their job for whatever reason), then you will have to devise an interview scenario where the candidate is incapable of using ChatGPT clandestinely, e.g. by bringing them into your office.
- osigurdson 3y agoPerhaps interviews need to assume the person being interviewed is using an LLM and can be evaluated on how effective they are with it. Presumably this is what employers want. The challenge is interviewers are busy, would prefer to be doing other things and want to stick to their old playbook ("tell me how to invert a binary tree").
- kfk 3y agoAnother take is we don’t like being lied to. Lots of these ChatGPT job candidates don’t disclose they are using an AI during the interview.
- osigurdson 3y agoMy suggestion is ChatGPT should be part of the interview.
- TrackerFF 3y agoYes, agree. If it is out in the open, with the chat/prompts available, you can ask other questions. You're not on your toes trying to catch a cheater. You're not assuming that the interviewee is lying or trying to scam you.
- wakawaka28 3y agoNo, it's not what they want. If they wanted you to use a LLM then they would tell you that up front. It's also too new of a technology to be required anywhere. Hardly anyone I know is even trying LLMs to begin with. Then, what do you do if the interviewee gets garbage code out of the LLM and misses an error? An error that might be forgiven in a normal interview cannot be excused when you didn't even have to write the code. Technically, if the LLM did the coding for you, you might pass without even being able to read code. This is all like the same reason you can't use a laptop on an algebra exam... The tool might do 100% of the work and leave you having shown nothing of your own ability.
- 3y ago
- 8organicbits 3y ago> Maybe in the future everybody is going to use LLMs to externalize their thinking. But then why do I interview you? It will become a skill. In 1900 you'd interview a computer (a person who does math) by asking them to do math on paper. Now you'd let them write some code or use software to do it. If the applicant didn't know how to use a (digital) computer, you'd negatively rate them. I don't love it, but we may reach the point where your skill at coaxing an LLM to do the right thing becomes a desirable skill and you'd negatively rank LLM-illiterate applicants. Looking at LLM quality, we're not at that point for most fields.