6 ms·

Do you have a source for this? I'm pretty sure responses api is only there to obfuscate reasoning, but that they're still keeping reasoning traces in the backen

Do you have a source for this? I'm pretty sure responses api is only there to obfuscate reasoning, but that they're still keeping reasoning traces in the backend.

wahnfrieden 2mo ago
[flagged]
- CjHuber 2mo ago
  Well, obviously its true as you can see in the silbing comment. But still interesting how experiences differ. I wonder did you ever use codex in more demanding workflows, because that at least something like that is going on is obvious if you use it for a few hours
- satvikpendem 2mo ago
  Looks like you're wrong, if you look at the sibling
- simiones 2mo ago
  You answered at more or less the same time as an OpenAI engineer confirming this is the case, explaining why they did it, and pointing to their docs that also very explicitly explain this "made-up nonsense". You should really consider why you felt so certain about something you were plain wrong about.
  - orliesaurus 2mo ago
    > You should really consider why you felt so certain about something you were plain wrong about. I applaud this reply!
  - satvikpendem 2mo ago
    Exactly, it's ironic that they thought they were right while talking about not having sources for another comment.
tedsanders 2mo ago
I work at OpenAI and can confirm that's correct: reasoning tokens are discarded after each new user turn (though not after each message or tool call). Our docs show a diagram here: https://developers.openai.com/api/docs/guides/reasoning https://developers.openai.com/api/docs/guides/reasoning > Input and output tokens from each step are carried over, while reasoning tokens are discarded. Keeping reasoning tokens around is better for caching and for remembering past insights, so you might reasonably wonder why we designed it this way. The main benefit of dropping reasoning tokens is that you can fit a lot more work inside the model's context window before you're forced into a slow and lossy compaction step. This was a larger consideration with our earlier reasoning models that had shorter context windows (~200k), longer thinking times (up to ~100k per message), and poor compaction. However, now that we've shipped longer context windows, we've trained our models think much more efficiently, and we've made compaction way better than it used to be, the balance of factors is changing. Tune in Thursday!
- CjHuber 2mo ago
  > fit a lot more work inside the model's context window before you're forced into a slow and lossy compaction step This is something I never understood. Why the reasoning is not included until the context is full, then the reasoning stripped optionally to allow the conversation to continue. and only then when its truly full offer a compaction. Was it to optimize caching? Well I guess it doesn't matter now that you hinted that this choice was made because of prior limitations and may change very soon
  - embedding-shape 2mo ago
    > Why the reasoning is not included until the context is full, then the reasoning stripped optionally to allow the conversation to continue. and only then when its truly full offer a compaction. Models are typically trained (at longer conversations/more turns) either with or without the reasoning still in the conversation. If you train a model with those, then using it without them, the model will perform a lot worse, same vice-versa if you train without but then end up using the model with them. That's why you'll see some models have it and others don't, and trying to use them another way, will make them worse, they weren't trained like that. So why aren't the models trained with both? I'm guessing that sort of permutation in the training would lead to double the amount of training time being needed, as you know effectively will have two variants of every session you train on, with and without the reasoning.

Do you have a source for this? I'm pretty sure responses api is only there to obfuscate reasoning, but that they're still keeping reasoning traces in the backend.

wahnfrieden 2mo ago

[flagged]

CjHuber 2mo ago
Well, obviously its true as you can see in the silbing comment. But still interesting how experiences differ. I wonder did you ever use codex in more demanding workflows, because that at least something like that is going on is obvious if you use it for a few hours
satvikpendem 2mo ago
Looks like you're wrong, if you look at the sibling
simiones 2mo ago
You answered at more or less the same time as an OpenAI engineer confirming this is the case, explaining why they did it, and pointing to their docs that also very explicitly explain this "made-up nonsense". You should really consider why you felt so certain about something you were plain wrong about.
- orliesaurus 2mo ago
  > You should really consider why you felt so certain about something you were plain wrong about. I applaud this reply!
- satvikpendem 2mo ago
  Exactly, it's ironic that they thought they were right while talking about not having sources for another comment.

tedsanders 2mo ago

I work at OpenAI and can confirm that's correct: reasoning tokens are discarded after each new user turn (though not after each message or tool call). Our docs show a diagram here: https://developers.openai.com/api/docs/guides/reasoning https://developers.openai.com/api/docs/guides/reasoning > Input and output tokens from each step are carried over, while reasoning tokens are discarded. Keeping reasoning tokens around is better for caching and for remembering past insights, so you might reasonably wonder why we designed it this way. The main benefit of dropping reasoning tokens is that you can fit a lot more work inside the model's context window before you're forced into a slow and lossy compaction step. This was a larger consideration with our earlier reasoning models that had shorter context windows (~200k), longer thinking times (up to ~100k per message), and poor compaction. However, now that we've shipped longer context windows, we've trained our models think much more efficiently, and we've made compaction way better than it used to be, the balance of factors is changing. Tune in Thursday!

CjHuber 2mo ago
> fit a lot more work inside the model's context window before you're forced into a slow and lossy compaction step This is something I never understood. Why the reasoning is not included until the context is full, then the reasoning stripped optionally to allow the conversation to continue. and only then when its truly full offer a compaction. Was it to optimize caching? Well I guess it doesn't matter now that you hinted that this choice was made because of prior limitations and may change very soon
- embedding-shape 2mo ago
  > Why the reasoning is not included until the context is full, then the reasoning stripped optionally to allow the conversation to continue. and only then when its truly full offer a compaction. Models are typically trained (at longer conversations/more turns) either with or without the reasoning still in the conversation. If you train a model with those, then using it without them, the model will perform a lot worse, same vice-versa if you train without but then end up using the model with them. That's why you'll see some models have it and others don't, and trying to use them another way, will make them worse, they weren't trained like that. So why aren't the models trained with both? I'm guessing that sort of permutation in the training would lead to double the amount of training time being needed, as you know effectively will have two variants of every session you train on, with and without the reasoning.