6 ms·

I've had little success with Agentic coding, and what success I have had has been paired with hours of frustration, where I'd have been better off doing it myse

by haar 1y ago

I've had little success with Agentic coding, and what success I have had has been paired with hours of frustration, where I'd have been better off doing it myself for anything but the most basic tasks. Even then, when you start to build up complexity within a codebase - the results have often been worse than "I'll start generating it all from scratch again, and include this as an addition to the initial longtail specification prompt as well", and even then... it's been a crapshoot. I _want_ to like it. The times where it initially "just worked" felt magical and inspired me with the possibilities. That's what prompted me to get more engaged and use it more. The reality of doing so is just frustrating and wishing things _actually worked_ anywhere close to expectations.

aschobel 1y ago
Bingo, it's magical but the learning curve is very very steep. The METR study on open-source productivity alluded to this a bit. I am definitely at a point where I am more productive with it, but it took a bunch of effort.
- devmor 1y ago
  The subjects in the study you are referencing also believed that they were more productive with it. What metrics do you have to convince yourself you aren't under the same illusionary bias they were?
  - simonw 1y ago
    Yesterday I used ffmpeg to extract the frame at the 13 second mark of a video out as a JPEG. If I didn't have an LLM to figure that out for me I wouldn't have done it at all.
    - dfedbeef 1y ago
      Was the answer: ffmpeg -ss 00:00:13:00 -i myvideo.avi -frames:v 1 myimage.jpeg Because this is on stack overflow and it took maybe one second to find. I've found reading the man page for a tool is usually a better way to learn what a tool can do for you now and also in the future.
      - kamranjon 1y ago
        This is the rub for me… people are so quick to forget the original source for a lot of the data these models were trained on, and how easy and useful these platforms were. Now Google will summarize this question for you in an AI overview before you even land on Stack Overflow. It’s killing the network effect of the open web and destroying our crowd sourced platforms in favor of a lossy compression algorithm that will eventually be regurgitating its own entrails.
        dfedbeef 1y ago
        Well, maybe. People will just stop using them and will make fun of people who do. You can only bullshit people for so long.
    - devmor 1y ago
      You wouldn't have just typed "extract frame at timestamp as jpeg ffmpeg" into Google and used the StackExchange result that comes up first that gives you a command to do exactly that?
      - simonw 1y ago
        Before LLMs made ffmpeg no-longer-frustrating-to-use I genuinely didn't know that ffmpeg COULD do things like that.
        devmor 1y ago
        I'm not really sure what you're saying an LLM did in this case. Inspired a lost sense of curiosity?
        Philpax 1y ago
        Translated a vague natural language query ("cli, extract frame 13s into video") into something immediately actionable with specific examples and explanations, surfacing information that I would otherwise not know how to search for. That's what I've done with my ffmpeg LLM queries, anyway - can't speak for simonw!
        wizzwizz4 1y ago
        DuckDuckGo search results for "cli, extract frame 13s into video" (no quotes): • https://stackoverflow.com/questions/10957412/fastest-way-to-extract-frames-using-ffmpeg https://stackoverflow.com/questions/10957412/fastest-way-to-... • https://superuser.com/questions/984850/linux-how-to-extract-frames-from-a-video-lossless https://superuser.com/questions/984850/linux-how-to-extract-... • https://www.aleksandrhovhannisyan.com/notes/video-cli-cheat-sheet/ https://www.aleksandrhovhannisyan.com/notes/video-cli-cheat-... • https://www.baeldung.com/linux/ffmpeg-extract-video-frames https://www.baeldung.com/linux/ffmpeg-extract-video-frames • https://ottverse.com/extract-frames-using-ffmpeg-a-comprehensive-guide/ https://ottverse.com/extract-frames-using-ffmpeg-a-comprehen... Search engines have been able to translate "vague natural language queries" into search results for a decade, now. This pre-existing infrastructure accounts for the vast majority of ChatGPT's apparent ability to find answers.
    - dingnuts 1y ago
      It is nice to use LLMs to generate ffmpeg commands, because those can be pretty tricky, but really, you wouldn't have just used the man page before? That explains a lot about Django that the author is allergic to man pages lol
      - simonw 1y ago
        I just took a look, and the man page DOES explain how to do that! ... on line 3,218: https://gist.github.com/simonw/6fc05ea7392c5fb8a5621d65e0ed01d0#file-ffmpeg-txt-L3218 https://gist.github.com/simonw/6fc05ea7392c5fb8a5621d65e0ed0... (I am very confident I am not the only person who has been deterred by ffmpeg's legendarily complex command-line interface. I feel no shame about this at all.)
        lexh 1y ago
        To be a little more fair... that example is tidily slotted into the EXAMPLES section, under the heading "You can extract images from a video, or create a video from many images". I don't think most people read the man pages top to bottom. And even if they did, then for as much grief as you're giving ffmpeg, llm has an even larger burden... no man page and the docs weigh in at over 8k lines. I get the general point that ffmpeg is a powerful, complex tool... but this is a weird fight to pick.
        simonw 1y ago
        I could not be more confident that "ffmpeg is difficult to figure out" is not a weird fight to pick. It's notorious!
        otabdeveloper4 1y ago
        The correct solution here would have been to feed the man page to an LLM summarizer. Alas instead of correct and easy solutions to problems we are focused on sci-fi robot assitant bullshit.
        quesera 1y ago
        Ffmpeg is genuinely complicated! And the CLI is convoluted (in justifiable, and unfortunate ways). But if you approach ffmpeg from the perspective of "I know this is possible", you are always correct, and can almost always reach the "how" in a handful of minutes. Whether that's worth it or not, will vary. :)
    - throwworhtthrow 1y ago
      LLM's still give subpar results with ffmpeg. For example when I asked Sonnet to trim a long video with ffmpeg, it put the input file parameter before the start time parameter, which triggers an unnecessary decode of the video file. [1] Sure, use the LLM to get over the initial hump. But ffmpeg's no exception to the rule that LLM's produce subpar code. It's worth spending a couple minutes reading the docs to understand what it did so you can do it better, and unassisted, next time. [1] https://ffmpeg.org/ffmpeg.html#:~:text=ss%20position https://ffmpeg.org/ffmpeg.html#:~:text=ss%20position
      - CamperBob2 1y ago
        That says more about suboptimal design on ffmpeg's part than it does about the LLM. Most humans can't deal with ffmpeg command lines, so it's not surprising that the LLM misses a few tricks.
        nottorp 1y ago
        Had a LLM generate 3 lines of working C++ code that was "only" one order of magnitude slower than what i edited the code to in 10 minutes. If you're happy with results like that, sure, LLMs miss "a few tricks"...
        ben_w 1y ago
        You don't have to leave LLM code alone, it's fine to change it — unless, I guess, you're doing some kind of LLM vibe-code-golfing? But this does remind me of a previous co-worker. Wrote something to convert from a custom data store to a database, his version took 20 minutes on some inputs. Swore it couldn't possibly be improved. Obviously ridiculous because it didn't take 20 minutes to load from the old data store, nor to load from the new database. Over the next few hours of looking at very mediocre code, I realised it was doing an unnecessary O(n^2) check, confirmed with the CTO it wasn't business-critical, got rid of it, and the same conversion on the same data ran in something like 200ms. Over a decade before LLMs.
        1y ago
- haar 1y ago
  Apologies if I was unclear. The more I've used it, the more I've disliked how poor the results it's produced, and the more I've realised I would have been better served by doing it myself and following a methodical path for things that I didn't have experience with. It's easier to step through a problem as I'm learning and making small changes than an LLM going "It's done, and production ready!" where it just straight up doesn't work for 101 different tiny reasons.
  - airspresso 1y ago
    My preferred approach to avoid that outcome is to divide & conquer the problem. Ask the LLM to implement each small bit in the order you'd implement it yourself given what you know about the codebase.

by haar 1y ago

aschobel 1y ago

Bingo, it's magical but the learning curve is very very steep. The METR study on open-source productivity alluded to this a bit. I am definitely at a point where I am more productive with it, but it took a bunch of effort.

devmor 1y ago
The subjects in the study you are referencing also believed that they were more productive with it. What metrics do you have to convince yourself you aren't under the same illusionary bias they were?
- simonw 1y ago
  Yesterday I used ffmpeg to extract the frame at the 13 second mark of a video out as a JPEG. If I didn't have an LLM to figure that out for me I wouldn't have done it at all.
  - dfedbeef 1y ago
    Was the answer: ffmpeg -ss 00:00:13:00 -i myvideo.avi -frames:v 1 myimage.jpeg Because this is on stack overflow and it took maybe one second to find. I've found reading the man page for a tool is usually a better way to learn what a tool can do for you now and also in the future.
    - kamranjon 1y ago
      This is the rub for me… people are so quick to forget the original source for a lot of the data these models were trained on, and how easy and useful these platforms were. Now Google will summarize this question for you in an AI overview before you even land on Stack Overflow. It’s killing the network effect of the open web and destroying our crowd sourced platforms in favor of a lossy compression algorithm that will eventually be regurgitating its own entrails.
      - dfedbeef 1y ago
        Well, maybe. People will just stop using them and will make fun of people who do. You can only bullshit people for so long.
  - devmor 1y ago
    You wouldn't have just typed "extract frame at timestamp as jpeg ffmpeg" into Google and used the StackExchange result that comes up first that gives you a command to do exactly that?
    - simonw 1y ago
      Before LLMs made ffmpeg no-longer-frustrating-to-use I genuinely didn't know that ffmpeg COULD do things like that.
      - devmor 1y ago
        I'm not really sure what you're saying an LLM did in this case. Inspired a lost sense of curiosity?
        Philpax 1y ago
        Translated a vague natural language query ("cli, extract frame 13s into video") into something immediately actionable with specific examples and explanations, surfacing information that I would otherwise not know how to search for. That's what I've done with my ffmpeg LLM queries, anyway - can't speak for simonw!
        wizzwizz4 1y ago
        DuckDuckGo search results for "cli, extract frame 13s into video" (no quotes): • https://stackoverflow.com/questions/10957412/fastest-way-to-extract-frames-using-ffmpeg https://stackoverflow.com/questions/10957412/fastest-way-to-... • https://superuser.com/questions/984850/linux-how-to-extract-frames-from-a-video-lossless https://superuser.com/questions/984850/linux-how-to-extract-... • https://www.aleksandrhovhannisyan.com/notes/video-cli-cheat-sheet/ https://www.aleksandrhovhannisyan.com/notes/video-cli-cheat-... • https://www.baeldung.com/linux/ffmpeg-extract-video-frames https://www.baeldung.com/linux/ffmpeg-extract-video-frames • https://ottverse.com/extract-frames-using-ffmpeg-a-comprehensive-guide/ https://ottverse.com/extract-frames-using-ffmpeg-a-comprehen... Search engines have been able to translate "vague natural language queries" into search results for a decade, now. This pre-existing infrastructure accounts for the vast majority of ChatGPT's apparent ability to find answers.
  - dingnuts 1y ago
    It is nice to use LLMs to generate ffmpeg commands, because those can be pretty tricky, but really, you wouldn't have just used the man page before? That explains a lot about Django that the author is allergic to man pages lol
    - simonw 1y ago
      I just took a look, and the man page DOES explain how to do that! ... on line 3,218: https://gist.github.com/simonw/6fc05ea7392c5fb8a5621d65e0ed01d0#file-ffmpeg-txt-L3218 https://gist.github.com/simonw/6fc05ea7392c5fb8a5621d65e0ed0... (I am very confident I am not the only person who has been deterred by ffmpeg's legendarily complex command-line interface. I feel no shame about this at all.)
      - lexh 1y ago
        To be a little more fair... that example is tidily slotted into the EXAMPLES section, under the heading "You can extract images from a video, or create a video from many images". I don't think most people read the man pages top to bottom. And even if they did, then for as much grief as you're giving ffmpeg, llm has an even larger burden... no man page and the docs weigh in at over 8k lines. I get the general point that ffmpeg is a powerful, complex tool... but this is a weird fight to pick.
        simonw 1y ago
        I could not be more confident that "ffmpeg is difficult to figure out" is not a weird fight to pick. It's notorious!
      - otabdeveloper4 1y ago
        The correct solution here would have been to feed the man page to an LLM summarizer. Alas instead of correct and easy solutions to problems we are focused on sci-fi robot assitant bullshit.
      - quesera 1y ago
        Ffmpeg is genuinely complicated! And the CLI is convoluted (in justifiable, and unfortunate ways). But if you approach ffmpeg from the perspective of "I know this is possible", you are always correct, and can almost always reach the "how" in a handful of minutes. Whether that's worth it or not, will vary. :)
  - throwworhtthrow 1y ago
    LLM's still give subpar results with ffmpeg. For example when I asked Sonnet to trim a long video with ffmpeg, it put the input file parameter before the start time parameter, which triggers an unnecessary decode of the video file. [1] Sure, use the LLM to get over the initial hump. But ffmpeg's no exception to the rule that LLM's produce subpar code. It's worth spending a couple minutes reading the docs to understand what it did so you can do it better, and unassisted, next time. [1] https://ffmpeg.org/ffmpeg.html#:~:text=ss%20position https://ffmpeg.org/ffmpeg.html#:~:text=ss%20position
    - CamperBob2 1y ago
      That says more about suboptimal design on ffmpeg's part than it does about the LLM. Most humans can't deal with ffmpeg command lines, so it's not surprising that the LLM misses a few tricks.
      - nottorp 1y ago
        Had a LLM generate 3 lines of working C++ code that was "only" one order of magnitude slower than what i edited the code to in 10 minutes. If you're happy with results like that, sure, LLMs miss "a few tricks"...
        ben_w 1y ago
        You don't have to leave LLM code alone, it's fine to change it — unless, I guess, you're doing some kind of LLM vibe-code-golfing? But this does remind me of a previous co-worker. Wrote something to convert from a custom data store to a database, his version took 20 minutes on some inputs. Swore it couldn't possibly be improved. Obviously ridiculous because it didn't take 20 minutes to load from the old data store, nor to load from the new database. Over the next few hours of looking at very mediocre code, I realised it was doing an unnecessary O(n^2) check, confirmed with the CTO it wasn't business-critical, got rid of it, and the same conversion on the same data ran in something like 200ms. Over a decade before LLMs.
        1y ago
haar 1y ago
Apologies if I was unclear. The more I've used it, the more I've disliked how poor the results it's produced, and the more I've realised I would have been better served by doing it myself and following a methodical path for things that I didn't have experience with. It's easier to step through a problem as I'm learning and making small changes than an LLM going "It's done, and production ready!" where it just straight up doesn't work for 101 different tiny reasons.
- airspresso 1y ago
  My preferred approach to avoid that outcome is to divide & conquer the problem. Ask the LLM to implement each small bit in the order you'd implement it yourself given what you know about the codebase.