6 ms·

Thanks for sharing this. For session-shaped benchmarks, how would you keep the evaluation fair when cache state and accumulated context differ across Claude Cod

• achalpandey 2mo ago
  Here's my current plan, the "session" will be made up of multiple SWE bench tasks stitched together. Each "task" is the equivalent of a new user query and we also pre-program "cache expiration" (sleep for 5 mins) into the session. This ensures parity across providers (both default to 5 min TTLs). The goal of this exercise is to tease out how Claude Code and Codex differ in managing their context and how that impacts cost and quality for the same simulated session.