6 ms·

Really depends on the repo you’re working in. If it’s very large, especially if the tool needs to refer to documentation for a lot of custom frameworks and API

Really depends on the repo you’re working in. If it’s very large, especially if the tool needs to refer to documentation for a lot of custom frameworks and APIs, you often end up needing very large context windows that burn through tokens faster. If it’s smaller or sticks with common frameworks that the model was trained on, it’s able to do a lot more with smaller context windows and token usage is way lower.

some-guy 5mo ago
I'm currently in repos where the context window required is so large that the output is almost always "wrong" for the problem at hand. Quite a few people at my company burn through tokens this way, and it certainly isn't providing value to the company.
- AlotOfReading 5mo ago
  As always, improving accessibility for humans makes automation more effective. If the humans need to remember a PhD's worth of source code/documentation to contribute effectively, your codebase stinks.
  - bonesss 5mo ago
    I agree, in the general context of how I code. The LLM hype train has me reflecting on what a spoiled existence working in a ‘proper’ language provides though… React devs, JS devs, front-end devs working on large sites and frameworks might be triggering tens of files to be brought into context. What an OCaml dev can bring in through a 5 line union type can look very different in less token-efficient and terse languages.
  - ivirshup 5mo ago
    People at my company have started writing docs specifically for claude. They're quite useful for me too, but kinda disappointing they never wrote these docs for their colleagues.
    - staplers 5mo ago
      As someone who has written many docs, it's because 99% won't read it (rightfully so if it's verbose). You can turn that doc into a skill in a repo and Claude will read it everytime it's needed.
    - dunham 5mo ago
      I recently saw this with the logseq api - the published api was an auto-generated stub. So I tried to grep the source code for the function and found detailed documentation written for claude. So I guess one benefit of all of this is that it's making people actually document things and maybe plan a little bit before implementing.
conartist6 5mo ago
So if the AI could do the same work on huge codebases with far fewer tokens, would it be good or bad for the AI companies do you think?
- lukan 5mo ago
  It would be good for the first AI company offering this.
  - conartist6 5mo ago
    Or an anti-ai company of course too; one whose goal was to level the playing field between humans and AIs again
- anon84873628 5mo ago
  Unquestionably good. They want a product that provides value anywhere it's tried so as to establish the reputation as a magic human replacement. Gaming consumption based pricing at this point would be quitting before the race is over. They can always tweak the pricing knobs later once the industry is fully hooked.
  - conartist6 5mo ago
    Right but what if the thing that made fewer tokens necessary also kneecapped the idea of making humans dependent on AI to write software.
deleted 5mo ago
[deleted]
quaintdev 5mo ago
Begs the question if we should move on to minimal microservices so that whole project lives in context of llm. I hardly have to do anything when I'm working with small project with llm.
- giantg2 5mo ago
  Orchestration between those services and the integration testing for any reasonably complex change can still be quite large.
- mlsu 5mo ago
  Why not take it a step further? Make each function in the codebase its own project. Then the codebase can fit into the context window easily. All you have to do is debug issues between functions calling each other.
  - andai 5mo ago
    Wait, is this a joke about Lambda?
    - dymk 5mo ago
      left-pad
    - ShyCodeGardener 5mo ago
      I don't think it's a joke about left-pad, but the idea that the complexity increases tremendously when you take a cloud of "small" things all communicating with each other. You've just pushed the complexity elsewhere. Claude can easily crunch the small microservice, but you're pushing the complexity to communications issues, race conditions, etc.
      - andai 5mo ago
        Oddly enough I constantly run into the same issue on monolithic codebases too. Things could just be one file but they end up being 12. I had to look through 12 levels of indirection for a single boolean recently. Twice, on two separate projects in the same week. At least in a single codebase, that issue is at least theoretically solvable. At least the indirection wasn't split across 12 repos!
Aurornis 5mo ago
The codebase and the topic you're working on are huge variables. I don't use LLMs to write code (other than simple refactors and throwaway stuff) but I do use them heavily to crawl through big codebases and identify which files and functions I need to understand. Some of the codebases I explore will burn through tokens at a rapid rate because there is so much complex code to get through. If I use the $20 Claude plan and Opus I can go through my entire 5-hour allocation in a single prompt exploring the codebase some times, and it's justified. Other times I'm working on simple topics, even in a large codebase, and it will sip tokens because it only needs to walk a couple files to get to what it needs to answer my questions.
andai 5mo ago
On larger repos it spends a lot of time just finding the one line of code that needs to change. (I have the same problem, as a human!)
ok123456 5mo ago
Will this result in people moving away from large monorepos to per-unit, quasi-micro repositories to save in token use?
th0raway 5mo ago
Yes, in a reasonable microservice land where the places you need to connect to are all documented in very concise places, you have have extremely productive $10 days. In the giant monorepo with everything custom, you can't just rely on built in knowledge of 80% of you libraries, so it's a very different world. A place like Google has to be so much better off just training library concepts in, given how much of the things the LLM will "instinctively" reach for are unlikely to be available. Not unlike the acclimation period what happens when someone comes in or out of a company like that, and suddenly every library and infra tool you were used to are just not available. We need a lot more searching when that happens to us, and the LLM suffers from the same context issue. The human just has all of that trained in after a 6 months, but the LLM doesn't.
- skirmish 5mo ago
  > A place like Google has to be so much better off just training library concepts in They did that, there was a special version of Gemini fine-tuned on internal code. But then the main model moves so fast that it is hard to keep such fine-tunes up to date and on the latest.

some-guy 5mo ago

I'm currently in repos where the context window required is so large that the output is almost always "wrong" for the problem at hand. Quite a few people at my company burn through tokens this way, and it certainly isn't providing value to the company.

AlotOfReading 5mo ago
As always, improving accessibility for humans makes automation more effective. If the humans need to remember a PhD's worth of source code/documentation to contribute effectively, your codebase stinks.
- bonesss 5mo ago
  I agree, in the general context of how I code. The LLM hype train has me reflecting on what a spoiled existence working in a ‘proper’ language provides though… React devs, JS devs, front-end devs working on large sites and frameworks might be triggering tens of files to be brought into context. What an OCaml dev can bring in through a 5 line union type can look very different in less token-efficient and terse languages.
- ivirshup 5mo ago
  People at my company have started writing docs specifically for claude. They're quite useful for me too, but kinda disappointing they never wrote these docs for their colleagues.
  - staplers 5mo ago
    As someone who has written many docs, it's because 99% won't read it (rightfully so if it's verbose). You can turn that doc into a skill in a repo and Claude will read it everytime it's needed.
  - dunham 5mo ago
    I recently saw this with the logseq api - the published api was an auto-generated stub. So I tried to grep the source code for the function and found detailed documentation written for claude. So I guess one benefit of all of this is that it's making people actually document things and maybe plan a little bit before implementing.

conartist6 5mo ago

So if the AI could do the same work on huge codebases with far fewer tokens, would it be good or bad for the AI companies do you think?

lukan 5mo ago
It would be good for the first AI company offering this.
- conartist6 5mo ago
  Or an anti-ai company of course too; one whose goal was to level the playing field between humans and AIs again
anon84873628 5mo ago
Unquestionably good. They want a product that provides value anywhere it's tried so as to establish the reputation as a magic human replacement. Gaming consumption based pricing at this point would be quitting before the race is over. They can always tweak the pricing knobs later once the industry is fully hooked.
- conartist6 5mo ago
  Right but what if the thing that made fewer tokens necessary also kneecapped the idea of making humans dependent on AI to write software.

deleted 5mo ago

[deleted]

quaintdev 5mo ago

Begs the question if we should move on to minimal microservices so that whole project lives in context of llm. I hardly have to do anything when I'm working with small project with llm.

giantg2 5mo ago
Orchestration between those services and the integration testing for any reasonably complex change can still be quite large.
mlsu 5mo ago
Why not take it a step further? Make each function in the codebase its own project. Then the codebase can fit into the context window easily. All you have to do is debug issues between functions calling each other.
- andai 5mo ago
  Wait, is this a joke about Lambda?
  - dymk 5mo ago
    left-pad
  - ShyCodeGardener 5mo ago
    I don't think it's a joke about left-pad, but the idea that the complexity increases tremendously when you take a cloud of "small" things all communicating with each other. You've just pushed the complexity elsewhere. Claude can easily crunch the small microservice, but you're pushing the complexity to communications issues, race conditions, etc.
    - andai 5mo ago
      Oddly enough I constantly run into the same issue on monolithic codebases too. Things could just be one file but they end up being 12. I had to look through 12 levels of indirection for a single boolean recently. Twice, on two separate projects in the same week. At least in a single codebase, that issue is at least theoretically solvable. At least the indirection wasn't split across 12 repos!

Aurornis 5mo ago

The codebase and the topic you're working on are huge variables. I don't use LLMs to write code (other than simple refactors and throwaway stuff) but I do use them heavily to crawl through big codebases and identify which files and functions I need to understand. Some of the codebases I explore will burn through tokens at a rapid rate because there is so much complex code to get through. If I use the $20 Claude plan and Opus I can go through my entire 5-hour allocation in a single prompt exploring the codebase some times, and it's justified. Other times I'm working on simple topics, even in a large codebase, and it will sip tokens because it only needs to walk a couple files to get to what it needs to answer my questions.

andai 5mo ago

On larger repos it spends a lot of time just finding the one line of code that needs to change. (I have the same problem, as a human!)

ok123456 5mo ago

Will this result in people moving away from large monorepos to per-unit, quasi-micro repositories to save in token use?

th0raway 5mo ago

Yes, in a reasonable microservice land where the places you need to connect to are all documented in very concise places, you have have extremely productive $10 days. In the giant monorepo with everything custom, you can't just rely on built in knowledge of 80% of you libraries, so it's a very different world. A place like Google has to be so much better off just training library concepts in, given how much of the things the LLM will "instinctively" reach for are unlikely to be available. Not unlike the acclimation period what happens when someone comes in or out of a company like that, and suddenly every library and infra tool you were used to are just not available. We need a lot more searching when that happens to us, and the LLM suffers from the same context issue. The human just has all of that trained in after a 6 months, but the LLM doesn't.

skirmish 5mo ago
> A place like Google has to be so much better off just training library concepts in They did that, there was a special version of Gemini fine-tuned on internal code. But then the main model moves so fast that it is hard to keep such fine-tunes up to date and on the latest.