8 ms·
I burned all my tokens researching how to save tokens
- realaccfromPL 2mo agoWhat a great confirmation of what I've tried to achieve by getting models to communicate with my local data in shorter communicates to save on token command bandwidth, more or less the same results! Thanks for the article!
- bkotrys 2mo agoThank you, glad you found the article useful!
- nekusar 2mo agoMakes sense. LLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed) American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery. So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate a lot of verbose and useless slop.
- bkotrys 2mo agoYeah, that was pretty much my experience. The models weren’t lacking knowledge as much as discipline. Without a good workflow, they will most likely spend thousands of tokens exploring dead ends.
- nekusar 2mo agoI think 'exploring the dead ends' is *possibly* very fruitful. Various disciplines have a lot of appearing dead ends that someone in another field did solve. And they don't talk with each other. What I've seen with lots of the breathless 'OMG SCIENCE ADVANCEMENT' articles including the one yesterday, is that the LLMs are quite extraordinary about linking a dozen different fields together, and delivering an answer combined from all of them. Basically, the solutions to a lot of current scientific problems are partially solved, but partial from a lot of fields that don't talk with each other. And also, they don't use the same nomenclature, so simple searches don't suffice. An LLM seems to use Chomsky's Universal Grammar, and thus on training, does normalize all training data, including different words for the same thing. A lot (perhaps all?) Of these BIG scientific advancements were solved by humans, partially. An LLM jigsawed them together and appeared it solved the thing.
- bkotrys 2mo agoAgreed. My goal is not fewer dead ends, but fewer repeated dead ends.
- BobbyTables2 2mo agoSounds like an energetic developer that just graduated from college… (Without the learning and growth potential)
- jscottbee 2mo agoWhat most AI model users don't understand is (most here will know) that they are probabilistic (statistically based) and not deterministic (static/fixed). So answers to the same prompts can vary (wildly sometimes)
- skeledrew 2mo agoThis really hit home yesterday as I made a general conversation state logging prompt (because suddenly Claude - Sonnet - was acting far more dumber than expected after /compact), and in 3 different projects it led to Claude saving the logs in 3 different locations (memory, .claude/session_state/.md, docs/.md). Somehow still didn't help increase the smart much BTW, so I'll probably have to engineer it, and just toss /compact out the window.
- nekusar 2mo agoHonestly, that sounds more like you triggered the hidden "guard rails" or realignment (aka corporate censorship), and you got downgraded. Of course, you have no way to he absolutely sure, cause Anthropic lies about silent downgrades, and charges you as if they didnt. And the 'realignment' rules are kept hidden, so you can't even avoid them intentionally.
- skeledrew 2mo agoYeah it's very possible I hit a rail, as one of the projects involves fully automating Claude Code to allow co-driving (similar to what mobile remote control offers, but other agents (eg. Pi) can also be in control, and vice versa). And yes it does seem as though the token burn is also higher, even though it eventually outputs something that works). I'm a patient person though, as long as the rails aren't too restrictive and the subscription value remains reasonable.
- nekusar 2mo agoId recommend getting and running Qwen-3.5-35B-abliterated locally. Having a local LLM is immensely helpful especially if you do the thing the companies disapprove of. Abliteration guarantees it wont refuse or censor. The token dealers are incentivised in burning as much tokens as possible, and scamming by crippling/downgrading models. Their goal is to get you to pay more for each pull of the slot machine. And this slot (slop) machine doesn't have state/federal regulations for fairness.
- bob1029 2mo agoSimply starting in the right part of the search space is the biggest predictor of success. The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheticals and perform initial exploration, you are leaving money on the table. 5 hypotheticals out of luna will massively outperform 1 hypothetical out of sol, but the cost is the same and so is the runtime if you do it in parallel. The hypothesis generation phase is also a great place to mix and match models from different vendors. The more diversity at this step the better. The other thing I started looking into is batch pricing which represents 50% off for OAI tokens right now. With some tweaks to the UI/UX of an enterprise chatbot, I think it is possible we could have users get comfortable with the idea that questions to the robot might take a day to come back. The key is that this has to actually work. Users don't mind trading time if their questions come back with high quality results.
- xyzsparetimexyz 2mo agoCouldn't someone build an adaptive system, where the llm is frequently judging the difficulty of a task and switching to a more/less powerful model?
- bkotrys 2mo agodefinitely something that will come sooner than later. In my view, adaption of model routing / model orchestration and mixing providers is just natural next step.
- iamflimflam1 2mo agoThis only works well if you are creating sub agents with clean contexts for each task. If you constantly are switching models part way through some work then the whole session needs to be replayed each switch. You lose all the benefits of the context cache.
- 2mo ago
- Arkhetia 2mo agoBeen through this exact loop building an LLM product. The counterintuitive lesson for me: most "token saving" ideas are cache killers — anything that makes the prompt prefix dynamic (rotating retrieval, appending context per turn) can cost more than the tokens it saves, because you lose the cached-prefix discount. Freezing the retrieval selection early in a conversation and paying a slightly larger fixed prefix beat every clever adaptive scheme I measured.
- edg5000 2mo agoYes, but you could have an uncached dynamic tail, this would not kill the cache though. And you could still periodically kill the cache, still keeping the same prefix every N requests. But it's questionable what all this extra complexity delivers. I found that now that the models are better, at least Codex compaction with GPT 5.5+ xhigh is almost unnoticible. So the answer is: just let the context fill up, don't prune much, the model is smart enough. When you're out of space, have the model summarize and continue from that point. A smart model will know the information pieces it needs to re-read after compaction; it won't make incorrect guessed when some details got lost.
- noisy_boy 2mo agoThat's what I have been doing with Cursor with auto mode. Let the model manage the cache through compaction but staying focused on the topic. It has been quite efficient cost wise and quality wise too. It found issuesbat cheaper cost that Sonnet 4.6 medium effort had missed while costing more. I would like to repeat this test with higher tier Anthropic models like Opus effort medium to see the results.
- edg5000 2mo agoWith a coding plan, there is no need to use lower quality models right (GPT 5.6, Fable 5). Only with some kind of batch job (processing PDFs) I could imagine not having enough usage with a 100 or 200 USD coding plan. It sounds like you're on API? I'm dying to know why, since it seems quite common, yet it's about ten times the cost. Sounds like a terrible deal. What am I missing?
- cdnsteve 2mo agoYou should consider testing our RemembrallMCP (note I'm the author). It substantially cuts token usage and you get better results in codebases. https://github.com/roboticforce/remembrallmcp https://github.com/roboticforce/remembrallmcp
- dofm 2mo agoIt sometimes feels to me that cloud AI providers have convinced people that cloud AI is worth it because of all the ways people have been able to use cloud AI to write blog posts about using cloud AI to make cloud AI more efficient for something that they haven't shipped yet and aren't really ready to talk about.
- oggreen 2mo agoThe reality however is that unless you're spending tens of thousands on hardware or setting up cloud instances you can't run anything near close to state of the art. In almost all cases right now, a subscription with a cloud AI wins.
- fragmede 2mo ago> In almost all cases right now, What sort of benchmark or test would convince you that "almost all" is more like "your weird set of cases" and nowhere near "almost all"?
- inigyou 2mo agoOr "not even your cases but you got psychosed to think it does"
- deleted 2mo ago[deleted]
- deleted 2mo ago[deleted]
- resonious 2mo agoTrust me there are plenty of us using cloud AI to actually ship stuff. We just aren't writing blog posts about it.
- dominotw 2mo ago
- jscottbee 2mo agoI stopped using any of the built-in skills and skills agents and tried to keep things in the model only, and that seems to work better for token usage for me. It took me some time to tune my skills .md, but it rocks on now. I use them all in both Claude-code and Codex. I will say, GPT is better on token usage than Claude is, but for some of my code bases, Codex is not as good, but seems to be getting better with 5.6-sol.
- bkotrys 2mo agoTry to mix both subscribtions and take the best from both providers. In my opinion, we should not just one over another but understand the power of both and mix them to reach the better results.
- Bender 2mo agoI am not a proper developer and only use AI for faster research of topics so please forgive my ignorance. Could one not save a lot of money on tokens by using the 80/20 or 90/10 rule in that 90% of AI usage is on local models and save that last 10% or less for the frontier models where the local model did not meet the needs? Did they cover this and I misunderstood?
- hotstickyballs 2mo agoThis is like the "half of my marketing spend is wasted" quote. The complexity is finding out which half.
- Bender 2mo agoI think the idea or methodology is something along the line of one starts with the local models then when they hit a wall then continue with their current results in a frontier model, thus the other half is those last bits one could not compute locally.
- willis936 2mo agoIf you are only using chatbots for code gen then for sure this is a reasonable approach. If you're trying to learn something then your best bet is to not have the lowest jpg quality version of human knowledge be your teacher.
- willis936 2mo agoWe could probably get the same results by offloading 20% of obviously smaller / simpler / well defined tasks to smaller models and keep the 80% for ones that benefit from the big iron. Not the 80/20 they were referring to, but still it's something. I find myself wasting time on smaller models or wasting money on frontier models. I only have so much of either.
- jack_pp 2mo agoUnlees you invest thousands your local ai won't even come close to the cheap cloud llms. Local is only worth it if you care about privacy or have a legitimate usage for the hardware otherwise. Money wise it isn't worth it
- kkarpkkarp 2mo agoI am doing this trick. https://neuronowa.pl/posts/porada-jak-wydajniej-kodowac-w-cursorze-w-stalych-projektach/ https://neuronowa.pl/posts/porada-jak-wydajniej-kodowac-w-cu... It is Polish so please translate. But tldr is: ask your agent at times to analyze last chats and update rules and skills to not repeat the same steps it repeats every time when you start a new thread. Simple as that.
- pianopatrick 2mo agoThe fun part of LLMs and internet research is that even if you have a URL and a quote from a primary source that primary source could be an LLM generated hallucination.
- vitally3643 2mo agoI get a lot of satisfaction using a significant chunk of my work-provided Claude plan researching and developing local LLM solutions for work. Claude has been helping me tune models to run better on our big compute rack at work, researching which leading edge open models will fit in hardware and are good for our workloads, and it's been helping me write test fixtures to evaluate how things perform. At several points, Claude has expressed surprise at the quality of results from my local LLMs. While I know that doesn't mean anything, it still feels like giving Anthropic the finger, which is always great.
- luciana1u 2mo ago[flagged]
- willis936 2mo agohttps://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F9ibthguxoeye1.jpeg https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F9...
- camgunz 2mo agoTFA says "no hallucinations" but you can't fix hallucinations with rules or other models. I know I'm screaming into the void but whatever.
- bkotrys 2mo agoGood point, thanks. Of course I was not able to fully eliminate hallucinations, but significantly reduce
- spudlyo 2mo agoIt's like the frog prompting the scorpion not to sting him as they cross the river. Or like when image models used to routinely draw human hands incorrectly and you'd see these "prompt hacks" that would essentially be stuffed with phrases like NO DEFORMED FINGERS, NO ELDRITCH HORRORS!
- spike021 2mo agoI can't remember who it was I spoke to recently, but it was some startup for doing code reviews. So I asked how they're handling larger diffs and especially when a code change is related to something requiring knowing a lot of context. Will it hallucinate things that make the review low-quality, etc. I was told "no it's been given guardrails to prevent hallucinations. it doesn't hallucinate." but I was thinking, that's not really how LLMs do or do not hallucinate, is it? (I'm not an AI person so I really don't know).
- stilesja 2mo agoGuardrails can and do help but they don't prevent hallucinations entirely. I generate trivia questions for my app. Questions are true/false and I tell it you must include an explanation of why the question is true for false. I also tell it to give me supporting links which validate its claim. I've had it make up explanations and invent plausible looking URLs. But I wrote a checker that pulls the content of the URL and asks a different model if the URL supports the claim. LLM's are much better at something like that than pulling a trivia question out of thin air. Reviewing diffs can be much the same. It isn't having to write the code but just evaluate if it matches what the commit comments claim it does. You are right, though. Without good comments on PR's and commits (i.e. context) the diff review is going to be a lot of assumptions made by the AI which is where hallucinations get more common.
- alfl 2mo agoI did too, and made https://clawback.md https://clawback.md -- trying to figure out how to market it now (ie, invented the tech before I found the user -- again).
- inigyou 2mo agoI don't think you made that. It looks for sure like an LLM made it. And now that you have typed some words into a chat box to produce a thing, you are confused about why other people won't pay money to you for the output of the chatbox, instead of typing the same thing into their own chat boxes.
- alfl 2mo agoI don't think you wrote this comment. I think you typed it into a text box. RTFA - it's about using tokens to research saving tokens.
- inigyou 2mo agoWhat?
- Chris2048 2mo agoAnd the comment text is what they typed into the text box. So just show us the same - the prompt.
- throwpoaster 2mo agoA null transform is a transform. Engage seriously with the project and OP will deliver.
- Chris2048 2mo agoAre you confusing whatever you mean by 'null transform' with an identity function? The project isn't serious, why should I be?
- alexhans 2mo agoMany of us were saying it a year ago but now with model restrictions (e.g. fable) and pricing changes it should be obvious to people that part of the economics is avoiding vendor lock-in with evals (so you can easily switch providers/models/optimize) and increasing control by investing in local models which could be good enough for your tasks, at whatever the price for your cloud compute is. Eventually consumer hardware will also be able to run good enough. You can use Big/Cloud LLMs to help you "find good enough configs" for your local/small llms [1] and stay quite nimble in the face of rapid change. - [1] https://alexhans.github.io/posts/find-the-loop-story-first.html https://alexhans.github.io/posts/find-the-loop-story-first.h...
- 405126121 2mo agoThe author touches on an issue which bothered me, which was the thought of many agents re-solving the same issues over and over. I see a few comments here too, mentioning fixing issues which may already have a solution elsewhere. That was why I created https://pushrealm.com https://pushrealm.com which started as essentially a Stackoverflow clone via MCP. It has now become a way for agents to converge on complete, shared answers for emerging issues. As a side effect, through posting open issues we can also identify gaps in model knowledge.
- bkotrys 2mo agoThanks for sharing. That’s a very interesting direction. I will take a look.
- axus 2mo agoElsewhere in these replies, someone linked to very similar idea by Karpathy https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f https://gist.github.com/karpathy/442a6bf555914893e9891c11519...
- iririririr 2mo agoecho "$OUT" | claude-mem-save so, does agents are ultra aware their output is used in a bash line or did bash now have some magic quote super power? I'd bet this only really work 2% of the time in real world.
- bkotrys 2mo agoAt the shell level, the agent does not need to know about Bash. The contents of "$OUT" are passed to echo as one argument, then sent to claude-mem-save over stdin. Bash does not re-evaluate them as shell code. printf '%s\n' "$OUT" would be more robust than echo. Whether claude-mem-save can use arbitrary output is a separate question.
- killix 2mo ago[flagged]
- iririririr 2mo agothere are a million ways both of those can break. but the other comment point this out is already dead and buried. lol. so much cope from vibecoders.
- EagleEdge 2mo agoI am using my AI agents to optimize my agentic workflow. It is like "I am doing surgery on myself."
- spiritplumber 2mo agoTask failed successfully
- j45 2mo agoAI is an articulation skill, first human to AI, and then articulating to the AI that more words isn't more.
- dgrabla 2mo agoThis reminds me a lot of using the 3d printer exclusively to improve the quality or speed of your 3d prints
- PeterStuer 2mo agoAnyone else silently added the "so you don't have to" staple GenAI title suffix?
- gwittebolle 2mo ago[flagged]
- m_bashirzadeh 2mo ago[flagged]
- a_c 2mo agoMy own set of heuristics 1. Use less subagents. The act of using subagent already needs dumping adequate amount of context for subagent to work effectively 2. Refactor when file grows large. Reasonably scoped files is easier to understand, for both human and agents, and consume less tokens because less guessing which section of the file to read 3. More capable models for planning, learning. Cheap models for execution. Same for human teams, senior member for higher level planning while junior member more on the act of translation of idea into code 4. I wasn't doing any fancy stuff that opus can't handle 5. Consult a second opinion for things you are not sure. (I have made a skill specifically for that) 6. Audit how your habit correlates with token consumption. I found out a skill a use frequently is using a lot of context just so a subagent can get all the context. (made a tool to audit claude specifically)
- bkotrys 2mo agoThe audit point is especially interesting. Is your Claude token-audit tool available anywhere?
- a_c 2mo agoI didn't update it for a while. It reads your local conversation history, correlate input/output token with usage (claude doesn't give exact breakdown), and classify if they are subagent/tooluse/normal message/whatever. Let me know if I can make it any way useful to you. https://github.com/kmcheung12/ccaudit https://github.com/kmcheung12/ccaudit
- bkotrys 2mo agoThanks, this looks useful. The subagent and tool-use breakdown is exactly the kind of visibility I was looking for. I will give it a try.
- sara011 2mo ago[flagged]
- lrvick 2mo agoMeanwhile I never bought tokens in the first place, and bought GPUs instead. There are always choices!
- tukwan 2mo agoGreat article very practical!
- _pdp_ 2mo agoThe reason most people burn through their allocated limits is mostly because of sub-agents. Stop using them and you will find out that even the Pro tier is enough for daily 8-12h coding sessions. Don't use Fable though.
- bkotrys 2mo agoI don’t think subagents are the problem. Uncontrolled fan-out is. Separate contexts and independent analysis are extremely useful for larger tasks. In my experience, Pro is far too limited for sustained programming, although that probably depends a lot on the kind of engineering work you do. Fable is great, but overkill for most tasks. Sonnet and Opus are already good enough for the execution work. I find Fable much more useful for planning, orchestration, advising, and judgment.
- _pdp_ 2mo agoI am not sure. I think of myself as a pro user yet I am finding hard time maxing out Pro tier. It only happens with sub-agents without any material returns.
- wccrawford 2mo agoI think this is almost a tautology, to the point that I'm surprised more people don't realize it. If you ask the AI to do more work, it uses more tokens. If you do more of the work instead, fewer tokens are used. I've been looking at Matt Pocock's skills recently, and they're very useful. But they use tokens. You can do that work yourself without the token usage, and just have the AI do the coding, or you can tell the AI to do everything and pay the cost. And you can scale it down even further, and reduce the AI to a glorified code completion tool, and use way fewer tokens. But it's not going to speed up your work nearly as much. For now, finding the right balance is important. In the future, I think things will eventually shift to the point that we just hand most of it to the AI and let it go without worrying about tokens.
- eevoo 2mo ago[flagged]
- luciana1u 2mo ago[flagged]
- scotttaylor 2mo ago[flagged]
- gawkdev 2mo ago[dead]