17 ms·

These are two completely different types of mistakes. LLMs are, by now, pretty good at not making "code doesn't work" and "math doesn't work" kind of mistakes

by Levitz 1mo ago

These are two completely different types of mistakes. LLMs are, by now, pretty good at not making "code doesn't work" and "math doesn't work" kind of mistakes when writing code. These are also "easy" things to get good at, you learn what each part does, understand the abstractions, ensure it makes sense, and go on with your day. Unit testing helps here. LLMs are not that good at not making "this works wrong" kind of mistakes. Maybe the code compiles and does what it has to do, but maybe it's 2 lines of code with 6 lines of comments (looking at you Claude), maybe it defines three helper functions it doesn't really need, maybe it does something "here" when it should be doing that something "there" instead, maybe it finds itself in a framework and completely disregards how the framework is supposed to do things, etc etc. These are harder things to get good at and you WILL end up with an unreadable mess if you disregard caution and let the LLM go at it.

ryanSrich 1mo ago
> but maybe it's 2 lines of code with 6 lines of comments (looking at you Claude), maybe it defines three helper functions it doesn't really need, maybe it does something "here" when it should be doing that something "there" instead, maybe it finds itself in a framework and completely disregards how the framework is supposed to do things, etc. I would fully expect a brand new college grad to call these things out. There's nothing in here that requires extensive experience to understand. These are basic principles any SWE should know. Nonsense comments are common sense to pull, if the LLM is pulling in a framework you should look into that framework and understand how it works. Don't know the framework during the interview? Say that. Tell the person interviewing you "looks like it's pulling in XYZ. I'm not entirely familiar with that. I understand at a high level what it's doing, but I'd want to dig deeper and understand if the LLM is doing this part right"

by Levitz 1mo ago

ryanSrich 1mo ago

> but maybe it's 2 lines of code with 6 lines of comments (looking at you Claude), maybe it defines three helper functions it doesn't really need, maybe it does something "here" when it should be doing that something "there" instead, maybe it finds itself in a framework and completely disregards how the framework is supposed to do things, etc. I would fully expect a brand new college grad to call these things out. There's nothing in here that requires extensive experience to understand. These are basic principles any SWE should know. Nonsense comments are common sense to pull, if the LLM is pulling in a framework you should look into that framework and understand how it works. Don't know the framework during the interview? Say that. Tell the person interviewing you "looks like it's pulling in XYZ. I'm not entirely familiar with that. I understand at a high level what it's doing, but I'd want to dig deeper and understand if the LLM is doing this part right"