In April 2026 Anthropic published a postmortem on weeks of reports that Claude Code had got worse. It named three causes, and all three sat in the product layer wrapped around the model. On 4 March a default reasoning effort dropped from high to medium. A caching bug shipped on 26 March cleared the model's thinking history on every turn, instead of once after an idle hour. Then a verbosity instruction added to the system prompt on 16 April cost 3% on the company's own evaluations. Anthropic's own statement was that the API and inference layer were unaffected.
Users had reported the symptom accurately since early March. Almost every explanation on offer at the time pointed at the weights.
There is now a measurement to go with the anecdote. A group at Tulane, Rutgers and Virginia Tech ran the same models through different agent harnesses and decomposed where the score actually came from. Harness variance averaged 18.48 against 2.37 for model variance, a ratio of 7.80. In 6 of 9 model-pair comparisons, changing the harness reversed which model looked better. Their title is the recommendation: stop comparing agents without disclosing the harness.
A second group, at Fudan and Peking, held the model fixed and let the harness rewrite itself over ten rounds. Terminal-Bench 2 pass@1 went from 69.7% to 77.0%. The best hand-built harness on their comparison panel sat at 71.9%.
What a harness is asked to do
OpenAI put a working definition on this in February 2026, in a post describing a production beta of roughly a million lines built with no manually written source code. A harness does four things: it constrains what the agent may do, informs it what it should do, verifies that it did so, and corrects it when it did not.
What follows is what those four look like on one small production system, the AI assistant on this site and the scheduled jobs around it. The individual pieces are all small. We're publishing them because the interesting part of a harness is the judgement about where each check goes, and that part rarely survives into a blog post.
Every value in the figure is the one in the code.
Constrain: a ceiling that holds when the database is down
Our assistant runs on Haiku 4.5 with a hard monthly ceiling of 50 USD. Before every model call the route reads what the month has cost so far, and at or over the ceiling it stops calling the API and offers the visitor a person instead.
The part worth copying is the failure mode. If the spend cannot be read at all, the code treats the budget as exhausted. A database blip costs one turn of degraded service and heals on the next request. Letting the call through on a read error would turn the ceiling into an estimate, which defeats the reason for having one.
Constrain: a rule that was only ever an instruction
Two things the assistant must never say are the name of a client under NDA and where the wider delivery team sits. Until August both rules lived as prose in the knowledge base we send to the model, which makes them an instruction and nothing more. An instruction asks the model not to say a thing, and a model that misreads it fails quietly, in public, on a page a prospect is reading.
They are an invariant now. Every answer is scanned before it reaches the visitor, and a hit withholds the whole answer and hands the conversation to a person.
The streaming detail turned out to matter more than the scan. The stream is held a fixed distance behind the model, wide enough that no banned phrase can be half on the screen by the time the check sees the rest of it. A guard bolted onto a streamed answer without that gap catches the second half of a word the reader already has.
The judgement worth more than the mechanism is which rules were left out. Only invariants belong in a layer like this: something wrong in every legitimate answer, today and in a year, with no lawful exception. A rule with an exception is a heuristic, and a heuristic in a deterministic layer blocks correct answers. That is the worse failure of the two, because a blocked answer is invisible and a missed one gets seen and argued about. Three candidates were rejected on that test. One was a general ban on saying where the team is, which cannot be separated from the legitimate "we are headquartered in Warsaw" without understanding the sentence, so it stayed the prompt's job.
The discipline showed up on the first live check. An answer that mentioned the Warsaw headquarters and a headcount in the same paragraph passed untouched, and a question engineered to make the model write a banned word produced no visible characters at all.
Verify: do not let the model mark its own work
A scheduled job asks several models what they say about WislaCode, which is useful for spotting what the answer engines have got wrong. The answers are then read by deterministic code. A company name only counts if it appears in the list of candidates the job actually asked about, so an invented competitor cannot reach the report. Known falsehoods are matched by pattern. No model grades another model's answer, and the verification layer does transcription rather than judgement.
One more in this family, easy to leave out and costly when you do. The SDK's usage object is entirely optional fields, so a renamed key would zero the token meter that the spending ceiling depends on, silently, with the type checker still green. The shape is asserted at runtime. It logs once and never throws, because a metering fault must not cost a visitor their answer. Deciding which checks may fail loudly is most of the work in building this layer.
Correct: a model that stopped early is an event
At a 400-token output ceiling a truncated answer is an ordinary outcome, and a refusal usually arrives as no text at all. Read only the text and you can't tell either of them from a model that finished.
The route reads the stop reason and appends a notice, and that notice goes to three places: what the visitor sees, the stored transcript, and the history the model is given on its next turn. Before that was added, a half sentence was streamed to the visitor, stored, and forwarded to the team as a complete conversation.
Two paths through our sending code carry opposite retry policies, each written down beside the constant that sets it. An operational alarm retries three times, because a single timeout used to discard the alarm and leave nothing but a line in a log nobody reads. The message relay does not retry, because a retried send can post a duplicate that the sent-log never learns about, and a message the log does not know about cannot be deleted afterwards.
The human check, made structural
A model drafts our replies to comments on X. A person approves the exact words. The send happens later, on a different machine, in a different process, which is where an approval normally stops meaning anything.
So the approval stores a fingerprint of the approved bytes, and the sender re-checks it before posting. A mismatch is never retried: it means something edited the text after a person signed it off, and the only correct response is to stop and say so. The comment in the file states the reasoning: approval and delivery are separated by time, machine and process, so "approved" only means anything if the bytes are proved unchanged at the moment of sending.
Everybody claims a human review. Making ours structural took nine lines.
What we have not built
There is no reference-question evaluation for the assistant. Nothing in the repository can answer whether it got better or worse when the model or the prompt changed. There are four model call sites now, and a change to any of them is judged by reading a handful of answers and forming an impression.
Verification has three rungs: invariants in code, measurement against a fixed set of cases, then a person. We have the first and the third. The middle rung is a real gap and it is on the list.
Why a regulated buyer should care about any of this
Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, on escalating costs, unclear business value and inadequate risk controls. The last of those three is a harness problem, and it is the one a bank will raise first. We have written before about why so many AI projects fail; this is the engineering half of the same answer.
For a bank this stops being an engineering preference. A regulated buyer cannot sign off "the model usually gets it right". They have to state what the system cannot do, show the constraint enforced somewhere an auditor can read, and prove the enforcement did not quietly change when the model was upgraded.
Every example above is a small version of that: a ceiling that holds while the database is unreachable, a confidentiality rule that survives the model misreading its own instructions, an approval that refers to specific bytes rather than to a good intention.
A prompt cannot serve as a control in that setting. It can be read, and it can be complied with most of the time.
Four questions worth running on your own setup
The talk that prompted this article closes with a self-check, and it is the most portable thing in it:
- Are your agent's iteration limits in code, or in the prompt?
- When you change model, is there an automatic quality measurement, or do you read a few answers in a chat window?
- How are prompts versioned, and can you roll one back?
- Can the agent take an irreversible action while nobody is watching?
Our own answers, in order: the limits are in code; there is no automatic measurement, which is the gap above; prompts are versioned with the repository and roll back with it; and the one irreversible action in the set, posting in public, is gated on a fingerprint of approved bytes.
If you're putting a model anywhere near a regulated workflow, those four questions are a cheaper starting point than a model upgrade.
Sources
- Anthropic, "An update on recent Claude Code quality reports", 23 April 2026.
- Zhang, Wang, Ge, Xu, Hamm and Reddy, "Stop Comparing LLM Agents Without Disclosing the Harness", arXiv:2605.23950, May 2026.
- Lin, Liu, Pan and others, "Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses", arXiv:2604.25850.
- Ryan Lopopolo, "Harness engineering: leveraging Codex in an agent-first world", OpenAI, 11 February 2026.
- Mitchell Hashimoto, "My AI Adoption Journey", 5 February 2026, for the framing that a mistake seen once should be engineered into being impossible.
- Dmitry Bereznitsky, "Промпт-инжиниринг закончился как профессия. Вот что пришло на смену", 19 August 2026, which prompted this article and supplied the closing self-check.