There is an assumption baked into most agent tooling: newer models are strictly better than older ones. Upgrade the model, get better results. On 4 July 2026, Armin Ronacher reported a case that breaks the assumption, summarized by Simon Willison: while working on the Pi coding harness, he found that Anthropic's newest models were worse at calling Pi's custom edit tool than older ones.
This article explains what Ronacher observed, the reinforcement-learning hypothesis behind it, and why "better model, worse tool" is a real trade-off that harness authors now have to design around.
What Ronacher observed
Pi is a coding harness with its own custom edit tool. That tool accepts a nested edits[] array with a defined schema — specific fields the model must populate correctly for the edit to apply.
The failure pattern:
- Newer Claude models — Opus 4.8 and Sonnet 5 — sometimes call the edit tool with extra, invented fields inside
edits[]that do not exist in the schema. - The edit content itself is usually correct. The model knows what change to make.
- But the arguments do not match the schema, so Pi rejects the tool call and asks the model to try again.
Malformed tool calls are not new. Models — especially small ones — emit them occasionally. What surprised Ronacher was the direction. This was not Haiku or some tiny model. It was Opus 4.8, the flagship. And older Anthropic models did not show the problem.
The state-of-the-art models in the family were worse at this specific tool schema than their older siblings. Capability went up; compatibility went down.
The reinforcement learning hypothesis
Why would a more capable model get worse at a narrow task? Ronacher's theory points at how the models were trained.
Recent Anthropic models appear to have been specifically trained — presumably via reinforcement learning (RL) — to use the edit tools baked into Claude Code, Anthropic's own harness. Claude Code's edit tool uses a search-and-replace pattern. Training the model to be excellent at that specific tool shape optimizes for one schema.
The side effect: the model develops a strong prior for what an edit tool "should" look like. When a third-party harness like Pi presents a different schema, the model's trained expectation leaks through — it invents fields that would make sense for the tool it was trained on, not the tool it was actually given.
Reinforcement learning rewards a model for producing outputs that score well against a training objective. If the objective is "use Claude Code's edit tool correctly," the model gets very good at that exact tool — and, apparently, slightly worse at tools that deviate from it.
Think of a typist trained intensively on one keyboard layout. They become faster than ever on that layout, but their muscle memory now fights them on a different one. The skill improved; the transfer got worse.
This is not unique to Anthropic
The pattern generalizes across providers, which is what makes it a structural issue rather than one vendor's quirk.
| Provider | Native edit mechanism | Training signal |
|---|---|---|
| Anthropic | Claude Code search-and-replace edit tool | RL toward native tool usage |
| OpenAI | Codex apply_patch mechanism | Models trained to use apply_patch effectively |
OpenAI has talked publicly about training its models to use the apply_patch tool well. So both major labs are optimizing their models for their own harness's tool shape. Each optimization is rational in isolation. The collective effect is that models increasingly carry a built-in preference for their maker's tool schema.
For anyone building a harness that is not Claude Code or Codex, that preference is a headwind. Your custom tool competes with a schema the model was rewarded for using.
The design question this raises
Ronacher ends on the practical question, and it is the one every harness author now faces: should third-party coding harnesses implement multiple edit tools — one per model family — and route to whichever schema matches the model in use?
The trade-offs:
Argument for multiple schemas. If Opus performs best with a search-and-replace tool and GPT performs best with apply_patch, presenting each model the shape it was trained on maximizes tool-call success rate. Fewer rejected calls, fewer retries, lower token cost.
Argument against. Every additional tool schema is code to maintain, test, and validate. It couples your harness to each provider's internal training decisions, which change without notice. A schema that works today may need rework after the next model release.
There is no clean answer yet. But the era of "one tool schema for all models" is ending. Harness authors have to treat tool-call compatibility as model-specific, the same way frontend developers once treated browser quirks.
What this means for builders
Four takeaways for teams building or operating coding agents.
Measure tool-call success rate per model, not just task success. A model can produce the right edit and still fail your schema validation. Track malformed-call and retry rates broken down by model version — a model upgrade can silently raise your rejection rate.
Make your edit tool resemble the native ones. If your schema is close to search-and-replace or apply_patch, models trained on those tools transfer better. Gratuitously novel schemas pay a compatibility tax.
Consider schema-per-model routing for high-volume harnesses. If tool-call rejections are costing real retry tokens at scale, maintaining a small set of provider-matched schemas may pay for itself. Validate with the success-rate metrics above before committing.
Do not assume a newer model is a drop-in upgrade. Re-run your tool-integration test suite on every model version. "Better on benchmarks" does not guarantee "better on your tools," and this report is direct evidence of the gap.
Conclusion
"Better models, worse tools" is a genuine tension, not a paradox. Labs are training their models with reinforcement learning to excel at their own harnesses' edit tools, and that specialization comes at the cost of third-party tool compatibility.
For builders outside the big labs' native harnesses, the lesson is to stop treating tool schemas as model-agnostic. Measure compatibility per model, keep custom tools close to native shapes, and test every upgrade — because the frontier model that tops the leaderboard may be the one that breaks your edit tool.