Ian Nuttall

Part entrepreneur, part marketer, part software engineer using AI to build stuff on the internet.

What the Codex performance report found

/ 146,358 views / via X

Codex degradation report TLDR:

- The investigation found no single root cause; rather a mix of behavior shifts and small bugs, with several fixes already shipped and more on the way.

- Older hardware underperformed in evals and was removed, and improved load balancing is rolling out to reduce latency.

- Compaction was used more often, and repeated compactions hurt accuracy; recursive summaries were blocked and a warning was added to keep chats focused.

- The `apply_patch` flow could delete and then recreate files after failed diffs, which is risky if interrupted; mitigations are in progress and a longer‑term model fix is planned.

- There was no fleet‑wide latency regression, but some sessions escalated retries and timeouts inefficiently; handling of long‑running and interactive tasks is being improved.

- A constrained‑sampling bug produced out‑of‑distribution tokens, including occasional mid‑sentence language switches in under 0.25% of sessions; a fix is rolling out.

- The Responses API added extra newlines around tool definitions; this had no measurable performance impact.

- Evals show CLI 0.45 maintained quality while reducing tokens by roughly 10%; neither `--search` nor recent prompt changes caused regressions.

- Infra analysis found low auth‑cache hit rates adding about 50 ms to tail latency; working‑directory error rates were unchanged.

- The guidance is to prefer minimalist toolsets and shorter, targeted chats, and a dedicated real‑world performance team is being formed and hiring.

Ian's List

Join 15k+ subscribers