7 ms·
Not specifically Anthropic but why are we allowing billing to take place in tokens that are nebulous and fully controlled by the operators who have no aligned i
by boredumb 26d ago
Not specifically Anthropic but why are we allowing billing to take place in tokens that are nebulous and fully controlled by the operators who have no aligned incentives?
If I have a user input and then sanitize and inject that into a prompt to do something, I have no idea how much that is going to cost at all and no real way to measure this properly. A parallel example is digital ocean or aws, i can go and measure/limit my compute/fs/memory/startup times/etc and while it can be impossible to get down to the last flop of money allocated - i can run things on a real budget with real constraints, opposed to an LLM where I have to .. prerun a sanitized user prompt through a tokenizer and then ask an LLM to guess what it may do and give token consumption estimates and then act on those in any sane manner for the user?
Perhaps i'm missing something to do realistic and static rails on things but I don't see a serious way at scale to use the token billing model handling things requiring a users free text input short of having to go pander to VC money to throw money at it until someone else figures it out.
*to clarify my rambling...
We should be billed and given controls based on resource usage itself and not an opaque token concept on top of not being able to spin any knobs that control it's resource usage.
- demibabs 26d agoHow else would they bill tho? Their operating cost is per token.
- dijit 26d agoCharge on the input tokens, then you will naturally optimise for fewer output tokens. Theoretically. In reality, one sessions output tokens become the next sessions input tokens (at least if you continue the topic) so, its not as aligned as all that. But the parent is right, when incentives are not aligned, friction will happen. Its inevitable.
- cthaeh 26d ago[dead]
- demibabs 26d ago“Claude, spend the next 10 hours trying to solve the Reimann Hypothesis”. I agree that incentives are misaligned but there’s several competing model providers. If one gets funny with their costs people will jump ship, especially if the gap between the top 2 labs and everyone else keeps shrinking.
- dijit 26d ago“I can take current sources and tell you how solved this is, but I am not willing to work to a timeframe or to solve things that aren’t yet solved by mathematicians or science” These safeguards already exist when they get a whiff that you might be using Claude to fix security issues. Doesn’t seem farfetched given the incentives I outlined that they would apply to this kind of abuse. How loose those controls are becomes a market force.
- dullcrisp 26d agoPerfect, a coding agent that refuses to do things that haven’t already been done before.
- dijit 26d agoHahahaha, I think you’ve misunderstood what LLMs are.
- dullcrisp 25d agoHave you ever tried using one? Or just understanding them?
- dijit 25d agoYes. But just so we're both entirely clear on what an LLM is... it's a token prediction system. it genuinely can't do things except recall things that have already existed. People are having great success composing things together in new ways, but just like the english language has a finite number of sentences, and music has a finite number of chords: LLMs too are just combining things that have existed. I don't want to sound condescending, it is remarkable how useful this technology is, but please don't evangelise them on capabilities that they genuinely can never have. Laptop computers have incredible processing capabilities but nobody expects them to be able to walk your dog, no matter how useful they actually are at doing other things.
- ACCount37 26d agoAnd tokens can be metered reliably. Unlike something like "task completion".
- xmcp123 26d agoNeuralWatt just does it on energy consumption.
- claw-el 25d agoIsn’t their operating costs depreciation of hardware and electricity? Token is just an assumed representation of it?
- bob1029 26d ago> the operators who have no aligned incentives The model providers are quite aligned with concerns like customer retention. These arguments only work if there is no competition. We exist in a marketplace of black boxes. There's not just "the one" you must suffer. You have options. You can build your own too.
- boredumb 26d agothere are roughly three of them and they all use the same pricing model. I am also not in the position to build a frontier model company these days.
- surgical_fire 26d agoThere are more than 3. Hell, I use 3 different providers, and I currently don't give a dime to Anthropic or OpenAI.
- hn_throwaway_99 26d agoA per-token model roughly aligns with the providers' costs, and it is an objective measure, so it seems a reasonable way to charge. I see posts all the time on HN about which models from which providers offer the most bang-for-the-buck, and how to minimize token usage and still get optimal results, so it appears that competition is working.
- olibhel 26d agoYou could try: 1. Self hosting 2. Chinese models 3. Running it locally. Requires upfront cost and compromises on TPS.
- sebastiennight 25d ago> I am also not in the position to build a frontier model company these days. Well if you're not even going to put in the effort... Wait. Let me do this for you. /config model=boredumb effort=xhigh /ultraplan "Build a frontier model that can run @1ktok/sec anywhere DOOM will run"
- shepherdjerred 25d ago
- daishi55 26d agoFor OAI and Anthropic at least you can set a spend limit per response. Also tokens are well-defined.
- boredumb 26d agoI'm not worried about the volatility in the definition, i'm worried that I give it 1 token today and receive 2 token output, tomorrow I receive 40. If i'm doing this a hundred thousand times a day it is difficult to price this in for users downstream or in the extreme cases be able to absorb that at all short of going into a failmode with degraded access until someone goes and buys more tokens or gets the bill. The alternative is just pass the buck and bill your non-technical customers with a "tokens" line iteim every month.
- sroussey 26d agoNo, those doing this 100k times a day have very good data on this, good estimators and modeling. And the API has various knobs to change and evals will give you actionable data.
- Aurornis 26d ago> If i'm doing this a hundred thousand times a day it is difficult to price this in When you’re doing this 100K times per day you get an extremely good idea of what it costs. You also have all the tools to see when something starts changing quickly. This change is for Claude Code the harness. If you’re using the API at scale and paying full price then you get exactly what you put into the request.
- deleted 26d ago[deleted]
- eh_why_not 25d agoOne guess is that their "primary" target audience/market is the large corporations that get their employees unlimited tokens, and not the individual developer who may worry about spending and token accounting.
- taude 25d agoIt's the opposite. The enterprises have all the tooling to monitor token usage of employees, and to limit access. For example, we have a $300 month limit, and then need to file exception tickets when we need more to justify the cost. Pretty similar at other non-silicon valley company process. I don't know any enterprise who'se on unlimitaged token budget for their employees. that's not how enterprises sign contracts. https://code.claude.com/docs/en/admin-setup#set-up-usage-visibility https://code.claude.com/docs/en/admin-setup#set-up-usage-vis...
- m4rtink 25d agoSeems like yet another instance of printing your own money & getting rich by screwing people forced to use them. Goas back to factory towns, gift cards, game money or MtG.
- Glyptodon 25d agoI was just complaining to someone that token billing is like letting a gasoline company control your gas pedal while you nicely ask them to use a specific gear that may or may not actually be in use and you guess what speed it's actually going based on how fast the trees go by because qualitative judgments have to replace the speedometer unless you can just burn money.
- shostack 25d agoDon't forget also letting them dictate the route you drive while you are increasingly blindfolded till you arrive and then find out you need to tell them to drive you to your correct destination again because they drove you across town instead which burned half your tank.