What the Codex performance report found
146,358 viewsvia X
Codex degradation report TLDR:
- The investigation found no single root cause; rather a mix of behavior shifts and small bugs, with several fixes already shipped and more on the way.
- Older hardware underperformed in evals and was removed, and improved load balancing is rolling out to reduce latency.
- Compaction was used more often, and repeated compactions hurt accuracy; recursive summaries were blocked and a warning was added to keep chats focused.
- The `apply_patch` flow could delete and then recreate files after failed diffs, which is risky if interrupted; mitigations are in progress and a longer‑term model fix is planned.
- There was no fleet‑wide latency regression, but some sessions escalated retries and timeouts inefficiently; handling of long‑running and interactive tasks is being improved.
- A constrained‑sampling bug produced out‑of‑distribution tokens, including occasional mid‑sentence language switches in under 0.25% of sessions; a fix is rolling out.
- The Responses API added extra newlines around tool definitions; this had no measurable performance impact.
- Evals show CLI 0.45 maintained quality while reducing tokens by roughly 10%; neither `--search` nor recent prompt changes caused regressions.
- Infra analysis found low auth‑cache hit rates adding about 50 ms to tail latency; working‑directory error rates were unchanged.
- The guidance is to prefer minimalist toolsets and shorter, targeted chats, and a dedicated real‑world performance team is being formed and hiring.
Ian's List
Join 15k+ subscribers