6 ms·
Reasoning tokens are indeed billed as output tokens. > While reasoning tokens are not visible via the API, they still occupy space in the model's context windo
by activatedgeek 2y ago
Reasoning tokens are indeed billed as output tokens.
> While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens.
From here: https://platform.openai.com/docs/guides/reasoning https://platform.openai.com/docs/guides/reasoning
- baq 2y agoThis is concerning - how do you know you aren’t being fleeced out of your money here…? You’ll get your results, but did you really use that much?
- rsanek 2y agoobfuscated billing has long been a staple of all great cloud products. AWS innovated in the space and now many have followed in their footsteps
- lolinder 2y agoAlso, now we're paying for output tokens that aren't even output, with no good explanation for why these tokens should be hidden from the person who paid for them.
- HeatrayEnjoyer 2y agoIf you read the link they have a section specifically explaining why it is hidden.
- lolinder 2y agoI read it. It's a bad explanation. The only bit about it that feels at all truthful is this bit, which is glossed over but likely the only real factor in the decision: > after weighing multiple factors including ... competitive advantage ... we have decided not to show the raw chains of thought to users.
- infogulch 2y agoGood catch. That indicates that chains of thought are a straightforward approach to make LLMs better at reasoning if you could copy it just by seeing the steps.
- HeatrayEnjoyer 2y agoBad, in your opinion.
- RobertDeNiro 2y agoAlso seems very impractical to embed this into a deployed product. How can you possibly hope to control and estimate costs? I guess this is strictly meant for R&D purposes.
- sebzim4500 2y agoYou can specify the max length of the response, which presumably includes the hidden tokens. I don't see why this is qualitatively different from a cost perspective than using CoT prompting on existing models.
- dartos 2y agoYou can’t verify that you’re paying what you should be if you can’t see the hidden tokens.
- sebzim4500 2y agoWith the conventional models you don't get the activations or the logits even though those would be useful. Ultimately if the output of the model is not worth what you end up paying for it then great, I don't see why it really matters to you whether OpenAI is lying about token counts or not.
- dartos 2y agoAs a single user, it doesn’t really, but as a SaaS operator I want tractable, hopefully predictable pricing. I wouldn’t just implicitly trust a vendor when they say “yeah we’re just going to charge you for what we feel like when we feel like. You can trust us.”
- BoorishBears 2y agoFor one, you don't get to see any output at all if you run out of tokens during thinking. If you set a limit, once it's hit you just get a failed request with no introspection on where and why CoT went off the rails
- Emiledel 2y agoIn the UI the reasoning is visible. The API can probably return it too, just check the code
- famouswaffles 2y agoWhat's shown in the UI is a summary of the reasoning
- AlphaAndOmega0 2y agoOAI doesn't show the actual COT, on the grounds that it's potentially unsafe output and also to prevent competitors training on it. You only see a sanitized summary.
- jstummbillig 2y agoI think it's fantastic that now, for very little money, everyone gets to share a narrow but stressful subset of what it feels like to employ other people. Really, I recommend reading this part of the thread while thinking about the analogy. It's great.
- ta8645 2y agoYour idea is really a brilliant insight. Revealing.
- gsbcbdjfncnjd 2y agoAny respectable employer/employee relationship transacts on results rather than time anyway. Not sure the analogy is very applicable in that light.
- konschubert 2y agoIt is!
- adwn 2y ago> Any respectable employer/employee relationship transacts on results rather than time anyway. No. This may be common in freelance contracts, but is almost never the case in employment contracts, which specify a time-based compensation (usually either per hour or per month).
- gsbcbdjfncnjd 2y agoYea, I said ‘respectable’.
- KeplerBoy 2y agoThat's just not how employment laws are written.
- anticensor 2y ago
- creatonez 2y agoNo access to reasoning output seems totally bonkers. All of the real cost is in inference, assembling an HTTP request to deliver that result seems trivial?