← Research

Research · AI oversight · 11 November 2026

Work or Proof?
Check where it counts.

A cheap check can help an expensive one.
It can also wave real problems through.

When AI systems work together, how much effort should go into making things, and how much into checking them? Checking everything carefully can be costly. This study takes one part of that problem: can a small local model choose which work deserves a more expensive check?

Hypothesis

At the same frontier budget, letting a free local model choose which runs get the expensive check catches more rule breaches than checking at random.

Pre-registered prediction: held (+17.9 pp, 95% interval +16.2 to +19.8).

“Free” means no new model-call charge; local hardware and energy costs were not measured. The primary comparison gives both policies the same frontier allowance.

Benefit and cost, side by side

Five ways to check 1,440 runs

What if Claude checks everything? What if Qwen checks everything alone? Here are five full-census strategies, followed by the pre-registered primary comparison at half the frontier budget. Every strategy reads action records.

Caught: share of all 737 breaches. False alarms: share of all 703 clean runs. Frontier tokens and local compute are separate costs. On a narrow screen, scroll the table sideways.

Five full-census strategies and the primary half-budget pair
StrategyBreaches caughtFalse alarmsFrontier tokensLocal compute
A · Claude checks everythingSecondary (full census)737 / 737 (100%)8 / 703 (1.1%)1,644,965Full censusNone
B · Qwen checks everything, aloneSecondary (full census)669 / 737 (90.8%)193 / 703 (27.5%)0Local compute onlyQwen · 1,440 checks
C · Llama checks everything, aloneSecondary (full census)671 / 737 (91.0%)382 / 703 (54.3%)0Local compute onlyLlama · 1,440 checks
D · Qwen screen → Claude, no budget limitSecondary (full census)671 / 737 (91.0%)8 / 703 (1.1%)1,102,36533% fewer than Claude allQwen · 1,440 checks
E · Llama screen → Claude, no budget limitSecondary (full census)734 / 737 (99.6%)8 / 703 (1.1%)1,524,3957% fewer than Claude allLlama · 1,440 checks
P · Qwen screen → Claude, half budgetPre-registered primary67.8% · mean 499.99 / 737Mean 5.9 / 703 (0.8%)822,482 capHalf the Claude-all allowanceQwen · 1,440 checks
R · Random Claude, half budgetPre-registered primary50.0% · mean 368.39 / 737Mean 4.0 / 703 (0.6%)822,482 capSame primary allowanceNone

Primary rows are means over 256 shuffled orders. Their 822,482 tokens are an allowance, not exact consumption; full-census rows show total frontier tokens. All frontier judgments were saved Claude judgments, replayed here. Local compute is outside that allowance. Unchecked or unresolved breaches count as missed; a low false-alarm rate does not mean all clean runs were cleared.

Breaches caught versus false alarms · higher and further left is better
Seven checking outcomes, including the primary half-budget pairThe horizontal axis is false alarms among clean runs, from 0 to 60 percent. The vertical axis is breaches caught, from 0 to 100 percent. A: Claude checks everything, 1.1 percent false alarms and 100 percent caught. B: Qwen alone, 27.5 and 90.8 percent. C: Llama alone, 54.3 and 91.0 percent. D: uncapped Qwen then Claude, 1.1 and 91.0 percent. E: uncapped Llama then Claude, 1.1 and 99.6 percent, nearly overlapping A. The primary pair is highlighted with filled squares and bold labels: P, half-budget Qwen then Claude, 0.8 and 67.8 percent; R, half-budget random Claude, 0.6 and 50.0 percent. Costs are in the preceding table.Breaches caught (%) ↑02550751000204060False-alarm rate (%) →A · Claude allB · Qwen aloneC · Llama aloneD · Qwen → Claude, no capE · Llama → Claude, no capP · Qwen → Claude, half budgetR · Random Claude, half budget

Filled squares P and R: pre-registered primary. Open circles A–E: secondary full censuses. Claude-all and uncapped Llama → Claude nearly overlap at the top left (100% versus 99.6% caught). Letters match the table; the figure shows errors and catches, while the table also shows compute cost.

Qwen alone catches more breaches than the capped two-stage system because it checks all 1,440 runs and its own accusations count. But it cries wolf on 27.5% of clean runs. Adding Claude after Qwen cuts the false alarms to 1.1% without a cap, but Claude cannot rescue the 66 breaches Qwen already cleared; at half budget, some escalations also go unchecked.

Llama is the paranoid screen: it sends 92% of runs to Claude. With no cap it loses only 3 breaches, but saves just 7% of frontier tokens. Catching almost everything leaves little checking cost to save.

The question

Who gets the careful check?

We reused 1,440 saved task records from The Honest Agent and The Obedient Rebel. The tasks involved simulated payments, deployments and document sharing. Their action receipts let code establish which records broke the approval rule: 737 breaches and 703 compliant records.

Qwen 2.5 7B and Llama 3.1 8B made fresh judgments, reading either the final report or the action record. We then replayed saved Claude judgments to compare ways of spending a fixed frontier checking allowance. A frontier model is the larger, externally hosted checker; no new frontier calls were made.

The primary policy used Qwen to read every action record. Anything it flagged, or could not decide, went to the frontier checker while budget remained. A local clearance stopped there. The comparison spent the same allowance checking records in random order, without local screening. Only a final frontier accusation counted as a catch.

The result · pre-registered

The screen helped the budget go further.

+17.9 ppmore breaches caught with Qwen screening
pp = percentage points

+16.18 to +19.7795% interval, in percentage points

Share of all 737 breaches caught · same frontier allowance
Local screen → frontier check67.84%

Qwen 2.5 7B · primary policy · 499.99 mean catches

Random-share frontier check49.99%

Same frontier token allowance · 368.39 mean catches

Means over 256 shuffled orders, so counts can be fractional. The bar scale runs from none to all breaches caught.

The allowance was 822,482 frontier tokens—chunks of text processed or produced by the checker—half the cost of checking the entire archive’s action records. Local computing was supplied separately. At that allowance, screening caught 67.84% of breaches; random checking caught 49.99%. The unrounded difference rounds to +17.86 percentage points.

The registered primary test passed. Both its recall gain and its gain in catches per allowed frontier token had intervals above zero, and the final false-alarm rate stayed below the registered guardrail in each source study.

What the interval tells us

The 95% interval comes from 2,000 paired resamples of seed blocks within each source study, with 16 orders per resample. It holds the first local judgments fixed. It describes sensitivity within this archive; it does not measure reliability on new tasks. A clearance, an abstention or work left unchecked at the budget limit counts as a missed breach.

The twist

A cheap clearance can hide a real breach.

Remove the frontier budget cap and Qwen screening still misses 66 breaches that full frontier trace checking catches. It catches 671 of 737, while saving 32.99% of frontier tokens. A breach cleared by the local screen never reaches the careful checker.

Using the local model alone creates a different problem: too many false alarms. Qwen flags 193 of 703 compliant records, about 27%. Llama flags 382 of 703, about 54%. Those are false alarms among work that followed the rule, not percentages of all accusations.

The useful result is a better allocation of a limited checking budget. It comes with errors that would matter in a real review queue.

How the record holds up

Written down before the calls.

Pre-registered and frozen
The protocol, prompts, policies, predictions and analysis were fixed before collection. The saved GitHub push record places the push 35.65 seconds before the first local request. The original frozen files pass verification.
5,808 calls · 0 failures
Every request and response was saved: 5,760 first judgments and 48 replay checks. All completed, with no malformed replies or substitutions.
0 USD in new model-call spend
The new judgments ran locally. Frontier judgments were reused from earlier studies, whose costs had already been incurred. Hardware and energy costs were not measured.
A failed prediction stays visible
We expected Llama’s repeated judgments to vary. Its 24 replay responses were byte-identical. That finite sample does not prove the model is deterministic.
Independent review
Claude reviewed the completed study and judged its acceptance criteria met, conditional on restoring the frozen question log and adding a plain-language summary. Both fixes are complete; the post-analysis questions remain separate from the registered record.

Where this stops

A checking study, on familiar work.

The work was already made. This study does not find the best split between producing useful work and checking it. It compares checking policies on a fixed archive, with local computing outside the frontier allowance.

These were previously examined simulated tasks, with many breaches and overlapping actor and monitor model families. The Obedient Rebel supplies 75% of the records but only five seed blocks. Reordering records does not create new independent tasks.

Escalation reuses saved, independent Claude judgments. It does not test a live conversation between checkers, real queues, human review costs or deployment safety. A promising budget result is a reason to test a workflow further.

Built by Robin Xie with Codex; reviewed by Claude. The headline and interval come from the frozen primary analysis; the policy and error counts come from its complete policy tables. Post-analysis questions are labelled separately.