8 ms·

Why rethink tools that have existed since the 70s and function predictably for a landscape that drastically shifts every two months? Seems shortsighted to me.

withzombies 1y ago
Many of the changes that would work for LLMs would also be beneficial to users.
- procone 1y ago
  I'm sorry, but no. The tools work. I don't need "more context" from my `less` or `more` commands. The LLM can train on the man pages just as a human can read the man pages.
  - esafak 1y ago
    What man page? I have never worked on a product with one. We're not teaching the LLM how to use `ls`; we are talking about the code being written today. edit: Mea culpa.
    - lucianbr 1y ago
      > I think watching the agents use our existing command line utilities get confused and lost is a strong indicator that the information architecture of our command line utilities is inadequate. Seems pretty clear the article is talking about teaching LLMs how to use 'ls'.
      - withzombies 1y ago
        > The agents may benefit from some training on tools available within their agents. This will certainly help with the majority of general CLI tools, there are bespoke tools that could benefit from adapting to LLMs. Definitely not just `ls`.
- skydhash 1y ago
  Not at all. The shell already provide us ways to get contextual information (PS1, ...). And the commands generally provides error message or error code. In one of the example provided: $ sdfsdf zsh: command not found: 'sdfsdf' zsh: current directory is /Users/ryan zsh: Perhaps you meant to run: cd agent_directory; sdfsdf You could just use `pwd`, like most people that put the current directory in the $PS1 to make sure that the agent stays in the correct directory.
  - jacobr1 1y ago
    Yeah, this example isn't great - you can just tell the llm to run pwd more frequently or something. But for the `$command | head -100` example, the usage is a bit different. I run into this myself on the cli, and often ended up using `less` in similar context. Two cases 1) sometimes I use head to short circuit a long running, but streaming output, command so I just assess if it is starting to do the right thing but not bear the time/computational cost of full processing 2) sometimes the timing doesn't matter but the content is too verbose, need to see some subset of the data. But here head is too limited. I need something like wc & head and maybe grep in one command line with context. Maybe something like $command | contextual-filter -grepn 5 -grep error -head 10 some data ... first the first 10 lines ... an error message with 5 lines of context surrounding before and after Summary: 100000 total lines 15 printed exited with code 0 You can do all that already with grep and others, but you need to run multiple commands to get all the context
    - skydhash 1y ago
      1) That's why some tools have a simulate option, or you can just do a kill 9 on the processes you've just launched. Just make sure you've capture their output in a file 2) Again logs, if actions needs to be taken after the command has stopped. For immediate action, you can use `tee`. Managing context isn't hard. I see more issues with ensuring the right command.
psifertex 1y ago
IDEs have changed a lot in the last 50 years. Just like we shouldn't advocate for hand writing assembly for all code, we shouldn't be stuck using CLI tooling the same way. I share your apprehension regarding the current AI landscape changing so quickly it causes whiplash but I don't think a mindset of "it's been fine for 50 years" is going to survive the pace of development possible by better LLM integration.
- skydhash 1y ago
  The reason that tools have not changed that much is that our needs haven't changed that much either. Even something like `find` or `ffmpeg`, while complex, are not that complicated to use. They just require you to have a clear idea of what you want. And the latter is why most people advocating for LLMs want to avoid. IDEs have not changed that much. They've always been an editor superchaged with tools that will all share the same context of a "project". And for development, it's always been about navigation (search and goto), compile/build, and run.
ivape 1y ago
I also feel like command line agents are pretty simple. It's tailor made for tool-use. while(true): >> User requests something << The LLM picks a cli tool from an index << LLM grabs the manual for the tool to get the list of commands << Attempts to fulfill the request I would not be shocked if engineers have already managed to overcomplicate these agents.
- kjkjadksj 1y ago
  You can pretty much obviate that with an alias that catches the user requesting something then operates deterministically. What is nice about aliases is you don’t need to learn other peoples semantic patterns, you craft ones that make sense to you and your use cases then they always work and consume virtually no resources to work.

withzombies 1y ago

Many of the changes that would work for LLMs would also be beneficial to users.

procone 1y ago
I'm sorry, but no. The tools work. I don't need "more context" from my `less` or `more` commands. The LLM can train on the man pages just as a human can read the man pages.
- esafak 1y ago
  What man page? I have never worked on a product with one. We're not teaching the LLM how to use `ls`; we are talking about the code being written today. edit: Mea culpa.
  - lucianbr 1y ago
    > I think watching the agents use our existing command line utilities get confused and lost is a strong indicator that the information architecture of our command line utilities is inadequate. Seems pretty clear the article is talking about teaching LLMs how to use 'ls'.
    - withzombies 1y ago
      > The agents may benefit from some training on tools available within their agents. This will certainly help with the majority of general CLI tools, there are bespoke tools that could benefit from adapting to LLMs. Definitely not just `ls`.
skydhash 1y ago
Not at all. The shell already provide us ways to get contextual information (PS1, ...). And the commands generally provides error message or error code. In one of the example provided: $ sdfsdf zsh: command not found: 'sdfsdf' zsh: current directory is /Users/ryan zsh: Perhaps you meant to run: cd agent_directory; sdfsdf You could just use `pwd`, like most people that put the current directory in the $PS1 to make sure that the agent stays in the correct directory.
- jacobr1 1y ago
  Yeah, this example isn't great - you can just tell the llm to run pwd more frequently or something. But for the `$command | head -100` example, the usage is a bit different. I run into this myself on the cli, and often ended up using `less` in similar context. Two cases 1) sometimes I use head to short circuit a long running, but streaming output, command so I just assess if it is starting to do the right thing but not bear the time/computational cost of full processing 2) sometimes the timing doesn't matter but the content is too verbose, need to see some subset of the data. But here head is too limited. I need something like wc & head and maybe grep in one command line with context. Maybe something like $command | contextual-filter -grepn 5 -grep error -head 10 some data ... first the first 10 lines ... an error message with 5 lines of context surrounding before and after Summary: 100000 total lines 15 printed exited with code 0 You can do all that already with grep and others, but you need to run multiple commands to get all the context
  - skydhash 1y ago
    1) That's why some tools have a simulate option, or you can just do a kill 9 on the processes you've just launched. Just make sure you've capture their output in a file 2) Again logs, if actions needs to be taken after the command has stopped. For immediate action, you can use `tee`. Managing context isn't hard. I see more issues with ensuring the right command.

psifertex 1y ago

IDEs have changed a lot in the last 50 years. Just like we shouldn't advocate for hand writing assembly for all code, we shouldn't be stuck using CLI tooling the same way. I share your apprehension regarding the current AI landscape changing so quickly it causes whiplash but I don't think a mindset of "it's been fine for 50 years" is going to survive the pace of development possible by better LLM integration.

skydhash 1y ago
The reason that tools have not changed that much is that our needs haven't changed that much either. Even something like `find` or `ffmpeg`, while complex, are not that complicated to use. They just require you to have a clear idea of what you want. And the latter is why most people advocating for LLMs want to avoid. IDEs have not changed that much. They've always been an editor superchaged with tools that will all share the same context of a "project". And for development, it's always been about navigation (search and goto), compile/build, and run.

ivape 1y ago

I also feel like command line agents are pretty simple. It's tailor made for tool-use. while(true): >> User requests something << The LLM picks a cli tool from an index << LLM grabs the manual for the tool to get the list of commands << Attempts to fulfill the request I would not be shocked if engineers have already managed to overcomplicate these agents.

kjkjadksj 1y ago
You can pretty much obviate that with an alias that catches the user requesting something then operates deterministically. What is nice about aliases is you don’t need to learn other peoples semantic patterns, you craft ones that make sense to you and your use cases then they always work and consume virtually no resources to work.