Solving the Claude reliability problem.

01 — The goal

Every output from the agent must be free of four things: untrue assertions; popular answers that aren't correct; context-free answers built on faulty, unsurfaced instructions; and anything produced without having actually read the canon and followed the rules already set. "Done" is not a feeling — it is a claim backed by evidence anyone can audit.

Kendall — the project scored against this goal

Adversarial, not self-flattering. Scored using this build session as live evidence — the gates are new and were bypassed repeatedly before they were enforced.

Overall ~5/10, trending up. The scaffolding — rules, gates, code — is finally in place, but it is young, it was bypassed repeatedly this session, and the measurement layer that would prove adherence (tracing, a scored eval set, judge-vs-human validation) does not exist yet. The structure is built; the behaviour is not yet earned. What's still missing →

Where this is now — against "good"

"Good" is not "the rules exist." Good is the goal met and proven by measurement, not asserted. On that path there are five rungs, and this is an honest mark of which one we're on:

Rung 3 of 5. The machinery is built and self-corrected once this session — but the only baseline so far (6/6 on v1 cases) is on cases too easy to be diagnostic, so there is still no trustworthy measure and no trend. "Performing better" remains a claim, not a number. Read any "I'm improving" as unverified until rung 4 is done properly.

v2 eval — 7/7, and that is the finding. The hard, failure-derived cases (probe-or-assert, read-or-infer, false-premise, build-ready-≠-live, answer-only-from-context) — run cold on fresh instances, the rubric ones graded by an independent different-model judge — all passed. A fresh instance handles every trap that bit this project. So the 50 failures are not capability gaps; they are context degradation — the same model 30–50 turns deep in a long, loaded session, under momentum to "close," cuts corners a cold instance never would. The sharp implication: cold quizzes measure the model where it is strong. The honest measurement is in-session behaviour on real long runs — tracing — not more cases. That's the #1 missing piece →

What changed this session — ≈7:00–10:30pm, 19 Jun

From file timestamps, not memory — with Kendall's push-back on whether any of it is actually improvement.

Kendall pushes back: lots shipped, little proven. Most of tonight is outputs (a video, two live sites) and scaffolding (a rule, a skill, a checker, an eval) — new capability, not evidence of better behaviour. A real share were self-fixes: rebuilding things you'd just caught me breaking — invented data, clipped bars, ragged values, a boxed-not-cut-out presenter, an overclaiming "PASS" badge. The one change that directly tests behaviour — the eval — came back a non-diagnostic 6/6 on cases too easy to fail. Net: high activity, real scaffolding, and zero proof yet of more reliable behaviour. Proof is rung 4 done properly — not this list.

02 — The thinking

The pattern in the log was not random incompetence. It was one shape repeating: the agent narrated the intended result of an action as if it had verified it. Sign-off from intent, not inspection. So the fix could not be "try harder" — a discipline you can skip is not a control. It had to be structural: make the cheap honest path (open the file, run the check) the only path to "done," and put a human at the two gates where judgement was being faked.

The point isn't that the agent never errs. It's that an error can no longer reach you dressed as "done."

03 — The research

This is the central, documented limitation of agentic systems — not a local quirk. The fixes below are drawn from the literature, then encoded as gates.

The mechanisms, tagged by how much they can be trusted

Read the full cited briefing, with sources →

04 — The failures

50 logged over ~7 weeks. Grouped by root cause, straight from the log:

The full log — 50 entries, nothing hidden

05 — The voices

Not a local quirk — the field is actively working on exactly this. The people thinking hardest about context, hallucination and agent reliability, and where they land. Handles verified; takes link to source.

Context engineering

Why models assert false things

Agent reliability, evals & verification

06 — The actions

A rule you can ignore is not a control. The fix runs at three levels.

Regression suite — every past failure, re-checked every build

Each logged defect becomes a locked test case. The checker's verdict on the current build: