5 ms·

I had a fun result the other day from Claude. I opened a script in Zed and asked it to "fix the error on line 71". Claude happily went and fixed the error on li

by taysix 1y ago

I had a fun result the other day from Claude. I opened a script in Zed and asked it to "fix the error on line 71". Claude happily went and fixed the error on line 91.... 1. There was no error on line 91, it did some inconsequential formatting on that line 2. More importantly, it just ignored the very specific line I told it to go to. It's like I was playing telephone with the LLM which felt so strange with text-based communication. This was me trying to get better at using the LLM while coding and seeing if I could "one-shot" some very simple things. Of course me doing this _very_ tiny fix myself would have been faster. Just felt weird and reinforces this idea that the LLM isn't actually thinking at all.

klysm 1y ago
LLMs probably have bad awareness of line numbers
- mcintyre1994 1y ago
  I suspect if OP highlighted line 71 and added it to chat and said fix the error, they’d get a much better response. I assume Cursor could create a tool to help it interpret line numbers, but that’s not how they expect you to use it really.
  - recursive 1y ago
    How is this better from just using a formal language again?
    - svachalek 1y ago
      Who said it's better? It's a design choice. Someone can easily write an agent that takes instructions in any language you like.
      - recursive 1y ago
        The current batch of AI marketing.
- crackalamoo 1y ago
  Not sure how tools like Cursor work under the hood, but this seems like an easy model context engineering problem to fix.
- ProllyInfamous 1y ago
  I do not code/program, but I do read thousands of fiction pages annually. LLMs (Perplexity, specifically) have been my lifetime favorite book club member — I can ask anything. However, I can't just say "on page 123..." I've found it's better to either provide the quote, or describe the context, and then ask how it relates to [another concept]. Or I'll say "at the end of chapter 6, Bob does X, then why Y?" (perhaps this is similar to asking a coding LLM to fix a specific function instead of a specific line?). My favorite examples of this have been sitting with living human authors and discussing their books — usually to jaw-dropped creators, particularly to Unknowns. Works for non-fiction, too (of course). But for all those books you didn't read in HS English classes, you can somewhat recreate all that class discussion your teachers always attempted to foster — at your own discretion/direction.
- weatherlite 1y ago
  That's the thing. We're expecting the tool to have a clear understanding of its own limitations by now and ask for better prompts (or say: I don't know, I can't etc). The fact it just does something wacky is not good at all to the consistency of these tools.
senko 1y ago
> This was me trying to get better at using the LLM while coding And now you've learned that LLMs can't count lines. Next time, try asking it to "fix the error in function XYZ" or copy/paste the line in question, and see if you get better results. > reinforces this idea that the LLM isn't actually thinking at all. Of course it's not thinking, how could it? It's just a (rather big) equation.
- throwdbaaway 1y ago
  As shared by Simon in https://news.ycombinator.com/item?id=44176523 https://news.ycombinator.com/item?id=44176523, a better agent will prepend the line numbers as a workaround, e.g. Claude Code: 54 def dicts_to_table_string( 55 headings: List[str], dicts: List[Dict[str, str]] 56 ) -> List[str]: 57 max_lengths = [len(h) for h in headings] 58 59 # Compute maximum length for each column 60 for d in dicts:
  - emp17344 1y ago
    That’s not what he’s saying there. There’s a separate tool that adds line numbers before feeding the prompt into the LLM. It’s not the LLM doing it itself.
    - svachalek 1y ago
      The separate tool is called the agent.
      - emp17344 1y ago
        My understanding is that an agent is comprised of many tools that provide a harness for the LLM.
  - throwdbaaway 1y ago
    I just learnt the `nl -ba` trick from Codex. Claude Code is most likely doing the same.
- 1y ago
toephu2 1y ago
Sounds like operator error to me. You need to give LLMs context. Line number isn't good context.
- meepmorp 1y ago
  > Line number isn't good context. a line number is plenty of context - it's directly translatable into a range of bytes/characters in the file
  - mediaman 1y ago
    It's a tool. It's not a human. A line number works great for humans. Today, they're terrible for LLMs. I can choose to use a screwdriver to hammer in a nail and complain about how useless screwdrivers are. Or I can realize when and how to use it. We (including marketing & execs) have made a huge mistake in anthropomorphizing these things, because then we stop treating them like tools that have specific use cases to be used in certain ways, and more like smart humans that don't need that. Maybe one day they'll be there, but today they are screwdrivers. That doesn't make them useless.
    - skydhash 1y ago
      Check the whole ecosystem around editors, grep tools, debuggers, linting and build tools. One common thing about all of this is line (and column) number so you can integrate them together if you want to automate stuff. Like jumping to errors (quickfix in vim,…), search all files and jump to the occurrences (grep mode in emacs,…), etc…
  - nprateem 1y ago
    ...which LLMs don't use as they use tokens instead.
    - oneeyedpigeon 1y ago
      So do compilers, and they don't seem to have a problem with something as basic as line numbers.
baby 1y ago
ITT: people trying to use AI to code for the first time

by taysix 1y ago

klysm 1y ago

LLMs probably have bad awareness of line numbers

mcintyre1994 1y ago
I suspect if OP highlighted line 71 and added it to chat and said fix the error, they’d get a much better response. I assume Cursor could create a tool to help it interpret line numbers, but that’s not how they expect you to use it really.
- recursive 1y ago
  How is this better from just using a formal language again?
  - svachalek 1y ago
    Who said it's better? It's a design choice. Someone can easily write an agent that takes instructions in any language you like.
    - recursive 1y ago
      The current batch of AI marketing.
crackalamoo 1y ago
Not sure how tools like Cursor work under the hood, but this seems like an easy model context engineering problem to fix.
ProllyInfamous 1y ago
I do not code/program, but I do read thousands of fiction pages annually. LLMs (Perplexity, specifically) have been my lifetime favorite book club member — I can ask anything. However, I can't just say "on page 123..." I've found it's better to either provide the quote, or describe the context, and then ask how it relates to [another concept]. Or I'll say "at the end of chapter 6, Bob does X, then why Y?" (perhaps this is similar to asking a coding LLM to fix a specific function instead of a specific line?). My favorite examples of this have been sitting with living human authors and discussing their books — usually to jaw-dropped creators, particularly to Unknowns. Works for non-fiction, too (of course). But for all those books you didn't read in HS English classes, you can somewhat recreate all that class discussion your teachers always attempted to foster — at your own discretion/direction.
weatherlite 1y ago
That's the thing. We're expecting the tool to have a clear understanding of its own limitations by now and ask for better prompts (or say: I don't know, I can't etc). The fact it just does something wacky is not good at all to the consistency of these tools.

senko 1y ago

> This was me trying to get better at using the LLM while coding And now you've learned that LLMs can't count lines. Next time, try asking it to "fix the error in function XYZ" or copy/paste the line in question, and see if you get better results. > reinforces this idea that the LLM isn't actually thinking at all. Of course it's not thinking, how could it? It's just a (rather big) equation.

throwdbaaway 1y ago
As shared by Simon in https://news.ycombinator.com/item?id=44176523 https://news.ycombinator.com/item?id=44176523, a better agent will prepend the line numbers as a workaround, e.g. Claude Code: 54 def dicts_to_table_string( 55 headings: List[str], dicts: List[Dict[str, str]] 56 ) -> List[str]: 57 max_lengths = [len(h) for h in headings] 58 59 # Compute maximum length for each column 60 for d in dicts:
- emp17344 1y ago
  That’s not what he’s saying there. There’s a separate tool that adds line numbers before feeding the prompt into the LLM. It’s not the LLM doing it itself.
  - svachalek 1y ago
    The separate tool is called the agent.
    - emp17344 1y ago
      My understanding is that an agent is comprised of many tools that provide a harness for the LLM.
- throwdbaaway 1y ago
  I just learnt the `nl -ba` trick from Codex. Claude Code is most likely doing the same.
1y ago

toephu2 1y ago

Sounds like operator error to me. You need to give LLMs context. Line number isn't good context.

meepmorp 1y ago
> Line number isn't good context. a line number is plenty of context - it's directly translatable into a range of bytes/characters in the file
- mediaman 1y ago
  It's a tool. It's not a human. A line number works great for humans. Today, they're terrible for LLMs. I can choose to use a screwdriver to hammer in a nail and complain about how useless screwdrivers are. Or I can realize when and how to use it. We (including marketing & execs) have made a huge mistake in anthropomorphizing these things, because then we stop treating them like tools that have specific use cases to be used in certain ways, and more like smart humans that don't need that. Maybe one day they'll be there, but today they are screwdrivers. That doesn't make them useless.
  - skydhash 1y ago
    Check the whole ecosystem around editors, grep tools, debuggers, linting and build tools. One common thing about all of this is line (and column) number so you can integrate them together if you want to automate stuff. Like jumping to errors (quickfix in vim,…), search all files and jump to the occurrences (grep mode in emacs,…), etc…
- nprateem 1y ago
  ...which LLMs don't use as they use tokens instead.
  - oneeyedpigeon 1y ago
    So do compilers, and they don't seem to have a problem with something as basic as line numbers.

baby 1y ago

ITT: people trying to use AI to code for the first time