10 ms·
Claude Token Counter, now with model comparisons
- tomglynch 5mo agoInteresting findings. Might need a way to downsample images on upload to keep costs down.
- simonw 5mo agoYeah that should work - it looks like the same pixel dimension image at smaller sizes has about the same token cost for 4.6 and 4.7, so the image cost increase only kicks in if you use larger images that 4.6 would have presumably resized before inspecting.
- mudkipdev 5mo agoWhy do you need an API key to tokenize the text? Isn't it supposed to be a cheap step that everything else in the model relies on?
- simonw 5mo agoI'd love it if that API (which I do not believe Anthropic charge anything for) worked without an API key.
- kouteiheika 5mo agoI'd guess it's because they don't want people to reverse engineer it. Note that they're the only provider which doesn't make their tokenizer available offline as a library (i.e. the only provider whose tokenizer is secret).
- stingraycharles 5mo agoAnthropic is somewhat becoming the Apple of AI in terms of closed ecosystem. Not saying I blame them, I just don't like it as a customer. The fact that it's impossible to get the actual thinking tokens anymore, but we have to do with a rewritten summary, is extremely off-putting. I understand that it's necessary for users, but when writing agentic applications yourself, it's super annoying not to have the actual reasoning of the agent to understand failure modes.
- aftbit 5mo agoIt's _not_ that it's necessary for users. It's that Anthropic got Opus 4.6 ripped off so hard by MiniMax that they no longer want to expose true thinking tokens to random developers. If you're one of the blessed class, you can still get real thinking tokens, but you need to be a major enterprise customer, like the companies that they gave Mythos access.
- weird-eye-issue 5mo agoTo prevent abuse? It's a completely free endpoint so I don't understand your complaint.
- tethys 5mo agoIt may be free, but it cannot be used without credits. Error: {"type":"error","error":{"type":"invalid_request_error","message":"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits."},"request_id":"req_011CaGaBf6uTHfbmdZ39nx1Z"}
- weird-eye-issue 5mo agoAgain it is to help prevent abuse I don't really see how this is a valid concern? Tokenization is actually fairly CPU intensive
- great_psy 5mo agoIs there any provided reason from anthropic why they changed the tokenizer ? Is there a quality increase from this change or is it a money grab ?
- Aurornis 5mo agoThe tokenizer is an important part of overall model training and performance. It’s only one piece of the overall cost per request. If a tokenizer that produces more tokens also leads to a model that gets to the correct answer more quickly and requires fewer re-prompts because it didn’t give the right answer, the overall cost can still be lower. Comparisons are still ongoing but I have already seen some that suggest that Opus 4.7 might on average arrive at the answer with fewer tokens spent, even with the additional tokenizer overhead. So, no, not a money grab.
- ChadNauseam 5mo agoHow would it be a money grab? If the new tokenizer requires more tokens to encode the same information, it costs them more money for inference. The point of charging per token is that the cost is proportional to the number of tokens. That's my understanding anyway
- abrookewood 5mo agoBecause everyone burns through their limits much faster, forcing them to upgrade to higher limits or new tiers.
- simianwords 5mo agoThey could just increase the token cost no? There’s little need for cute conspiracies like these
- sumeno 5mo agoThey would have to tell people if they did that.
- 5mo ago
- kouteiheika 5mo ago> Opus 4.7 tokenizer used 1.46x the number of tokens as Opus 4.6 Interesting. Unfortunately Anthropic doesn't actually share their tokenizer, but my educated guess is that they might have made the tokenizer more semantically aware to make the model perform better. What do I mean by that? Let me give you an example. (This isn't necessarily what they did exactly; just illustrating the idea.) Let's take the gpt-oss-120b tokenizer as an example. Here's how a few pieces of text tokenize (I use "|" here to separate tokens): Kill -> [70074] Killed -> [192794] kill -> [25752] k|illed -> [74, 7905] <space>kill -> [15874] <space>killed -> [17372] You have 3 different tokens which encode the same word (Kill, kill, <space>kill) depending on its capitalization and whether there's a space before it or not, you have separate tokens if it's the past tense, etc. This is not necessarily an ideal way of encoding text, because the model must learn by brute force that these tokens are, indeed, related. Now, imagine if you'd encode these like this: <capitalize>|kill <capitalize>|kill|ed kill| kill|ed <space>|kill <space>|kill|ed Notice that this makes much more sense now - the model now only has to learn what "<capitalize>" is, what "kill" is, what "<space>" is, and what "ed" (the past tense suffix) is, and it can compose those together. The downside is that it increases the token usage. So I wouldn't be surprised if this is what they did. Or, my guess number #2, they removed the tokenizer altogether and replaced them with a small trained model (something like the Byte Latent Transformer) and simply "emulate" the token counts.
- anonymoushn 5mo agotheir old tokenizer performed some space collapsing that allowed them to use the same token id for a word with and without the leading space (in cases where the context usually implies a space and one is not present, a "no space" symbol is used).
- fooker 5mo agoThis is how language models have worked since their inception, and has been steadily improved since about 2018. See embedding models. > they removed the tokenizer altogether This is an active research topic, no real solution in sight yet.
- dannyw 5mo ago
- yogigan 5mo ago[dead]
- aliljet 5mo agoThis is the rugpull that is starting to push me to reconsider my use of Claude subscriptions. The "free ride" part of this being funded as a loss leader is coming to a close. While we break away from Claude, my hope is that I can continue to send simple problems to very smart local llms (qwen 3.6, I see you) and reserve Claude for purely extreme problems appropriate for it's extreme price.
- londons_explore 5mo agoI think an LLM that is a decent chunk smarter/better than other LLM's ought to be able to charge a premium perhaps 10x or 100x it's competitors. See for example the price difference between taking a taxi and taking the bus, or between hiring a real lawyer Vs your friend at the bar who will give his uninformed opinion for a beer.
- DeathArrow 5mo agoQuality of answers from quantized models is noticeable worse than using the full model. You'll be better using Qwen 3.6 Plus through Alibaba coding plan.
- SoMomentary 5mo ago> Quality of answers from quantized models is noticeable worse than using the full model. This is the very reason I've heard I shouldn't use Alibaba!
- KronisLV 5mo ago> This is the rugpull that is starting to push me to reconsider my use of Claude subscriptions. I'm still with them cause the model is good, but yes, I'm noticing my limits burning up somewhat faster on the 100 USD tier, I bet the 20 USD tier is even more useless. I wouldn't call it a rugpull, since it seems like there might be good technical reasons for the change, but at the same time we won't know for sure if they won't COMMUNICATE that to us. I feel like what's missing is a technical blog post that tells uz more about the change and the tokenizer, although I fear that this won't be done due to wanting to keep "trade secrets" or whatever (the unfortunate consequence of which is making the community feel like they're being rugpulled).
- chattermate 5mo ago[dead]
- tpowell 5mo agoI just asked Claude about defaulting to 4.6 and there are several options. I might go back to that as default and use --model claude-opus-4-7 as needed. The token inflation is very real.
- onchainintel 5mo agoMany comparisons between 4.6 & 4.7 at https://tokens.billchambers.me/leaderboard https://tokens.billchambers.me/leaderboard My prompt was 40% more tokens using Opus 4.7.
- lifis 5mo agoI'm really surprised that: 1. Anthropic has not published anything about why they made the change and how exactly they changed it 2. Nobody has reverse engineered it. It seems easy to do so using the free token counting APIs (the Google Vertex AI token count endpoint seems to support 2000 req/min = ~3million req/day, seems enough to reverse engineer it)
- hk__2 5mo ago> It seems easy to do so What are you waiting for? ;)
- hugodan 5mo agoAren't these increases offset by the quality of the responses and reducing the iterations needed to fine-tune the responses?
- anentropic 5mo agoLikely only some of the time
- Majromax 5mo agoOnly for the range of tasks where 4.7 performs well but 4.6 performed suboptimally. If both models can one-shot the task without retries, then the number of iterations is already at the lower bound. This also applies at the sub-task level. If both models need to read three files to figure out which one implements the function they need to modify, then the token tax is paid for all three files even though "not the right file" is presumably an easy conclusion to draw. This is also related to the challenge of optimizing subagents. Presumably the outer, higher-capacity model can perform better with everything in its context (up to limits), but dispatching a less-capable subagent for a problem might be cheaper overall. Anthropic has a 5:1 cost on input tokens between Opus and Haiku, but Google has 8:1 (Gemini Pro : Flash Lite) and OpenAI has 12:1 (GPT 4.2 : 4.2 nano).
- sergiopreira 5mo agoAn interesting question is whether the tokenizer is better at something measurable or just denser. A denser tokenizer with worse alignment to semantic boundaries costs you twice, higher bill and worse reasoning. A denser tokenizer that actually carves at the joints of the model's latent space pays for itself in quality. Nobody outside Anthropic can answer which it is without their eval suite, so the rugpull read is fair but premature. Perhaps the real tell will be whether 4.7 beats 4.6 on the same dollar budget on the benchmarks you care about, not on the per-token ones Anthropic publishes.
- potter098 5mo ago[flagged]
- cubefox 5mo agoOkay, but what about output tokens?
- RITESH1985 5mo ago[flagged]
- vfalbor 5mo agoThis is perfectly legitimate. It's something I've been denouncing day after day. Company X charges you 10dolar per token, while company Y charges you 7dolar, yet company X is cheaper because of the tokenizer they use. The token consumption depends on the tokenizer, and companies create tokenizers using standard algorithms like BPE. But they're charging for hardware access, and the system can be biased to the point that if you speak in English, you consume 17% less than if your prompt is written in Spanish, or even if you write with Chinese characters, you'll significantly reduce your token consumption compared to English speakers. I've written about this several times on HN, but for whatever reason, every time I mention it, they flag my post.
- akdor1154 5mo agoI have often wondered if Chinese is a much 'better' language for LLMs - every character is a token, boom you're done. No weird subword nonsense, no strange semantics being applied to arbitrary chunks of words.. I feel like there must be benefits to being able to have the language tokenized in what must be very close to 1:1.
- vfalbor 5mo agoYes, it is. In fact, I made a small application to reduce the token consumption for translating from one language to another, and I even invented a language called Tokinensis, which is a mix of different languages, and I ran my own tests with savings of 30%. Chinese is amazing because they encapsulate a ton of information in a single symbol, so you can save a ton of tokens.
- touristtam 5mo agoInterested; I came across a post that was mentioning using Kanji for specific use to reduce context.
- vfalbor 5mo agoMaybe in future there will be some "Tokenensis" but in kanjis which could concentrate a lot of info into little space.
- Esophagus4 5mo agoAnyone have good tips or resources on token management best practices? Because I’ve hit the limiter with one single prompt now on Opus 4.7. What I’m reading so far seems to be: -selective use of models based on task complexity -encoding large repos into more digestible and relevant data structures to reduce constant reingesting -ask Claude to limit output to X tokens (as output tokens are more expensive) -reduce flailing by giving plenty of input context -use Headroom and RTK -disable unused MCP, move stuff from CLAUDE.md to skills But I’d love to learn if anyone has any good tips, links, or tools as I’m getting rate limited twice a day now.
- o10449366 5mo agoWhat was your single prompt? That seems highly unlikely.
- Esophagus4 5mo ago“Ok Claude, I bet you can’t hit the usage limit in one shot… let’s see what you got you little token gremlin” ;) I just asked it to do a security analysis in a subagent of an unmaintained browser extension and then go fix vulnerabilities it found so I can use it without worrying.
- o10449366 5mo agoSo it sounds like you were using it on auto mode then if it went ahead and fixed the vulnerabilities without additional turns? If so, that isn't really a single prompt.
- Esophagus4 5mo agoIt got limited while it was reading the repo before starting to fix, but yes you’re right. I think I was in auto mode or at least approving all the exploration.
- deleted 5mo ago[deleted]
- jug 5mo ago[dead]
- alvis 5mo ago[dead]
- hyperpape 5mo agoThis is a great piece of data, but only a piece of the actual question that we need to answer, which is: For a given input, how many tokens will be used for an answer, and how high quality will that answer be? Measuring the tokenizer is just one input into the cost-benefit tradeoff.
- ilioscio 5mo agoAnthropic was pulling ahead of their peers, but if they can't hear their customer's complaints about negatively changing value between releases they're going to undermine their position until no advantage is left.