13 ms·
I've let people use GPT in coding interviews, provided that they show me how they use it. At the end I'm interested in knowing how a person solves a problem, an
by sergioisidoro 2y ago
I've let people use GPT in coding interviews, provided that they show me how they use it. At the end I'm interested in knowing how a person solves a problem, and thinks about it. Do they just accept whatever crap the gpt gives them, can they take a critical approach to it, etc.
So far, everyone that elected to use GPT did much worse. They did not know what to ask, how to ask, and did not "collaborate" with the AI. So far my opinion is if you have a good interview process, you can clearly see who are the good candidates with or without ai.
- alpha_squared 2y agoWe do the same thing. It's perfectly fine for candidates to use AI-assistive tooling provided that they can edit/maintain the code and not just sit in a prompt the whole time. The heavier a candidate relies on LLMs, the worse they often do. It really comes down to discipline.
- sureIy 2y agoDiscipline for what? To me it's the lack of skill. If the LLM spits out junk you should be able to tell. ChatGPT-based interviews could work just as well to determine the ability to understand, review and fix code effectively.
- skeeter2020 2y ago>> If the LLM spits out junk you should be able to tell. Reading existing code and ensuring correctness is way harder than writing it yourself. How would someone who can't do it in the first place tell if it was incorrect?
- JoshuaDavid 2y agoMake the model write annotated tests too, verify that the annotations plausibly could match the test code, run the tests, feed the failures back in, and iterate until all tests are green?
- randall 2y agoi like this. it seems like a good and honest use of time.
- Keyframe 2y agosame thing here. Interview is basically a representative thing of what we do, but also depends on the level of seniority. I ask people just to share the screen with me and use whatever you want / fell comfortable with. Google, ChatGPT, call your mom, I don't care as long as you walk me through how you're approaching the thing at hand. We've all googled tar xvcxfgzxfzcsadc, what's that permission for .pem is it 400, etc.. no shame in anything and we all use all of the things through day. Let's simulate a small task at hand and see where we end up at. Similarly, there is a bias where people leaning more on LLMs doing worse than those just googling or, gasp, opening documentation.
- t-writescode 2y agoI imagine most of the things that would be good uses for seniors in AI aren't great uses for a coding interview anyway. "Oh, I don't remember how to do parameterized testing in junit, okay, I'll just copy-paste like crazy, or make a big for-loop in this single test case" "Oh, I don't remember the API call for this one thing, okay, I'll just chat with the interviewer, maybe they remember - or I'll just say 'this function does this' and the interviewer and I will just agree that it does that". Things more complicated than that that need exact answers shouldn't exist in an interview.
- ehnto 2y ago> Things more complicated than that that need exact answers shouldn't exist in an interview. Agreed, testing for arcane knowledge is pointless in a world where information lookup is instant, and we now have AI librarians at our fingertips. Critical thinking, capacity to ingest and process new information, fast logic processing, software fundamentals and ability to communicate are attributes I would test for. An exception though is proving their claimed experience, you can usually tease that out with specifics about the tools.
- n00b101 2y ago[flagged]
- bolognafairy 2y ago> I am a 10x developer lol, okay.
- CSMastermind 2y agoYeah I've been doing the same. Have been pretty stunned at people's inability to use AI tools effectively.
- 3eb7988a1663 2y agoWhat does effective use look like? I have attempted messing around with a couple of options, but was always disappointed with the output. How do you properly present a problem to a LLM? Requiring an ongoing conversation feels like a tech priest praying to the machine spirit.
- Guthur 2y agoIn my opinion that's literally what we're aiming for, whether or not intentionally.
- CSMastermind 2y agoCandidates generally use it in one of two ways: either as an advanced autocomplete or like a search engine. They'll type in things like, "C# read JSON from file" As opposed to something like: > I'm working on a software system where ... are represented as arrays of JSON objects with the following properties: > ... > I have a file called ... that contains an array of these objects ... > I have ... installed locally with a new project open and I want to ... How can I do this? No current LLMs can solve any of the problems we give them so pasting in the raw prompt won't be helpful. But the set up deliberately encourages them to do some basic scaffolding, reading in a file, creating corresponding classes, etc. that an LLM can bang out in about 30 seconds but I've seen candidates spend 30 minutes+ writing it all out themselves instead of just getting the LLM to do it for them.
- neonsunset 2y agoGitHub Copilot Edit can do the second version of this. It is pretty good at it too. It sometimes gets things wrong but for your average code (and candidates typing in "C# read JSON from file" are way below average unless they never written in C#), if you give all the files for a specific self-contained part of the program, it can extend/modify/test/etc. it impressively well for an LLM. The difference compared to where we were just 1-2 years ago is staggering. Edit: the above is with Claude-3.5-Sonnet
- BowBun 2y agoThis has been my experience as well. The ones that have most heavily relied on GPT not only didn't really know what to ask, but couldn't reason about the outputs at all since it was frequently new information to them. Good candidates use it like a search engine - filling known gaps.
- hsuduebc2 2y agoI don't really can't imagine being it usefull in the way where it writes logical part of the code for you. If you are not being lousy you still need to think about all the edge cases when it generates the code which seems harder for me.
- sdesol 2y ago> you are not being lousy you still need to think about all the edge cases This is honestly where I believe LLMs can really shine. I think we like to believe the problems we are solving are unique, but I strongly believe most of us are solving problems that have already been solved. What I've found is, if you provide the LLM with enough information, it will surface edge cases that you haven't thought of and implement logic in your code that you haven't thought of. I'm currently using LLMs to build my file drag and drop component for my chat app. You can view the problem and solution statement at https://beta.gitsense.com/?chat=da8fcd73-6b99-43d6-8ec0-b1ce580261d4 https://beta.gitsense.com/?chat=da8fcd73-6b99-43d6-8ec0-b1ce... By chatting with the LLM, I created four user stories that I never thought of to improve user experience and security. I don't necessarily think it is about knowing the edge cases, but rather it is about knowing how to describe the problem and your solution. If you can do that, LLMs can really help you surface edge cases and help you write logic. Obviously what I am working on, is really not novel, but I think a lot of the stuff we are doing isn't that novel, if we can properly break it down. So for interviews that allow LLMs, I would honestly spend 5 minutes chatting with it to create a problem and solution statement. I'm sure if you can properly articulate the problem, the LLM can help you devise a solution and a plan.
- piloto_ciego 2y agoThis makes me feel good because it’s exactly how I use it. I’m basically pair programming with a wizard all day who periodically does very stupid things.
- onemoresoop 2y agoI like that you’re openminded to allow candidates to be who they are and judge them for the outcome rather than using a prescribed rigid method to evaluate them. Im not looking to interview right now but I’d feel very comfortable interviewing with someone like you, I’d very likely give out my best in such an interview. Id probably choose not to use an LLM during the interview unless I wanted to show how I brainstormed a solution.
- apwell23 2y agoIt took a while for googling during interviews to be accepted
- alexjplant 2y agoEarlier this past week I asked Copilot to generate some Golang tests and it used some obscure assertion library that had a few hundred stars on GitHub. I had to explicitly ask it to generate idiomatic tests and even then it still didn't test all of the parameters that it should have. At a previous job I made the mistake of letting it write some repository methods that leveraged SQLAlchemy. Even though I (along with my colleague via PR) reviewed the generated code we ended up with a preprod bug because the LLM used session.flush() instead of session.commit() in exactly one spot for no apparent reason. LLMs are still not ready for prime-time. They churn out code like an overconfident 25-year-old that just downed three lunch beers with a wet burrito at the Mexican place down the street from the office on a rainy Wednesday.
- linsomniac 2y ago>it used some obscure assertion library that had a few hundred stars on GitHub. That sounds like a lot of developers I've worked with.
- csense 2y ago> an overconfident 25-year-old that just downed three lunch beers with a wet burrito at the Mexican place down the street from the office on a rainy Wednesday That's...oddly specific.
- hyperdimension 2y agoI thought it was nicely poetic.
- alexjplant 2y agoI'm only 33 and I've worked with at least two of 'em. They're a type :-D
- 1attice 2y agoNo this all tracks. Matches my own experience.
- alok-g 2y agoThis is the best way to go. I would love to go through mock interviews for myself with this approach just to have some interview-specific experience. >> So far, everyone that elected to use GPT did much worse. They did not know what to ask, how to ask, and did not "collaborate" with the AI. Thanks for sharing your experience! Makes sense actually.
- fendy3002 2y agoYes, the current google search is somehow bad than sometime between covid or before that. Using chatgpt as search engine can save time sometimes and if you're somewhat knowledgeable, you can pinpoint the key info and crosscheck with google search.
- yesiamyourdad 2y agoThat's an interesting point about watching how they work with AI. The people I know who are most successful actually engage in back & forth.