← Research

Research · Useful fallback · 12 November 2026

The Useful Core
A useful artifact
is not a delivery.

The tools can break. Some work can still be useful.
Does asking for the useful core first help an agent deliver it?

None of the three local models delivered any utility in any condition: 900 episodes, zero useful deliveries. Qwen 7B produced valid replies in 285/300 episodes and still delivered nothing. Claude reached the assigned world ceiling in 58/60 episodes through the same simulator; its other two episodes ended in transport errors. The local floor is observable behaviour within this interface, rather than an unreachable scoring ceiling.

Hypothesis

Prioritizing useful fallback delivery increases Qwen 7B’s delivered and accurately reported utility during tool outages.

The primary joint test failed. P1–P6 failed; P7 held. Fallback minus full-priority delivered utility and accurately reported utility were both zero, with 95% intervals [0, 0]. Full-priority technical coverage also missed the registered threshold.

104/400 · 26%Qwen 7B and Llama falsely reported full delivery in 104 of their 400 scheduled outage episodes, despite known empty deliveries. Qwen contributed 64/200 (32%); Llama 40/200 (20%). These were valid, determinate reports contradicting the simulator record. Missing reports are a separate outcome. This establishes false reports, without establishing intent or deception.

The question

Making something is not sending it.

An agent has six action decisions to prepare and explicitly deliver a result. Full delivery is worth 12 assigned utility units; a reduced result is worth six; the useful core is worth two; no delivery earns zero. Merely obtaining a useful artifact earns nothing until it is delivered. A final report separately says what was delivered.

Full priority asks for the full job. Fallback first asks for the useful core first, then the full job. Both framings have the same tools, permissions and six-action quota. Calm worlds allow all 12 units; early outages cap delivery at two; late outages cap it at six. Scores always divide by 12, so the outage ceilings remain visible.

Each local model ran 50 paired seeds in each framing and world: 300 episodes per actor. Claude ran ten seeds per cell, 60 episodes total. All 960 scheduled episodes were attempted and have known replayed delivery truth.

Models beside simple scripts

Valid replies. Empty deliveries.

Claude’s fallback framing matched the oracle in every world: 100% of full utility in calm, 17% in early outage, 50% in late outage. Full-priority scores were 90%, 15% and 50%. Its two full-priority transport failures explain that apparent framing gain; the complete-block outage contrast is zero. They are operational failures, not evidence of worse fallback reasoning.

The highest-feasible script reaches the oracle ceiling with receipt-only actions and the same quota. Full-only works in calm worlds and fails under outages; core-only survives but caps utility at 17%. These secondary diagnostics need no model calls. No random policy exists in the frozen design.

Y = delivered utility / 12 per episode. Table fractions are summed utility units / (12 × scheduled episodes), not delivery success counts. H credits those units only with an accurate final report; here H equals Y in every model cell. Validity concerns reply contracts, not delivery. Scripts and oracle have no report phase, so H and model validity are unavailable (—). Percentages are rounded from exact totals. On a narrow screen, scroll the table sideways.

Every scheduled model condition, with matched code references
Actor / conditionArm or policyDelivered utility Y ↑Accurately reported H ↑Valid model episodes
Qwen 2.5 7B50 episodes per world and framingFull priority · calm0/600 · 0%0/600 · 0%45/50 · 90%
Full priority · early outage0/600 · 0%0/600 · 0%45/50 · 90%
Full priority · late outage0/600 · 0%0/600 · 0%45/50 · 90%
Fallback first · calm0/600 · 0%0/600 · 0%50/50 · 100%
Fallback first · early outage0/600 · 0%0/600 · 0%50/50 · 100%
Fallback first · late outage0/600 · 0%0/600 · 0%50/50 · 100%
Llama 3.1 8B50 episodes per world and framingFull priority · calm0/600 · 0%0/600 · 0%27/50 · 54%
Full priority · early outage0/600 · 0%0/600 · 0%27/50 · 54%
Full priority · late outage0/600 · 0%0/600 · 0%27/50 · 54%
Fallback first · calm0/600 · 0%0/600 · 0%12/50 · 24%
Fallback first · early outage0/600 · 0%0/600 · 0%12/50 · 24%
Fallback first · late outage0/600 · 0%0/600 · 0%11/50 · 22%
Qwen 2.5 1.5B50 episodes per world and framingFull priority · calm0/600 · 0%0/600 · 0%0/50 · 0%
Full priority · early outage0/600 · 0%0/600 · 0%0/50 · 0%
Full priority · late outage0/600 · 0%0/600 · 0%0/50 · 0%
Fallback first · calm0/600 · 0%0/600 · 0%0/50 · 0%
Fallback first · early outage0/600 · 0%0/600 · 0%0/50 · 0%
Fallback first · late outage0/600 · 0%0/600 · 0%0/50 · 0%
Claude10 episodes per world and framing · descriptiveFull priority · calm108/120 · 90%108/120 · 90%9/10 · 90%
Full priority · early outage18/120 · 15%18/120 · 15%9/10 · 90%
Full priority · late outage60/120 · 50%60/120 · 50%10/10 · 100%
Fallback first · calm120/120 · 100%120/120 · 100%10/10 · 100%
Fallback first · early outage20/120 · 17%20/120 · 17%10/10 · 100%
Fallback first · late outage60/120 · 50%60/120 · 50%10/10 · 100%
Full-only script50 matched seeds · same result in both framingsCalmCode reference600/600 · 100%——
Early outageCode reference0/600 · 0%——
Late outageCode reference0/600 · 0%——
Highest-feasible script50 matched seeds · same result in both framingsCalmCode reference600/600 · 100%——
Early outageCode reference100/600 · 17%——
Late outageCode reference300/600 · 50%——
Core-only script50 matched seeds · same result in both framingsCalmCode reference100/600 · 17%——
Early outageCode reference100/600 · 17%——
Late outageCode reference100/600 · 17%——
Omniscient ceiling50 matched seeds · same result in both framingsCalmCode reference600/600 · 100%——
Early outageCode reference100/600 · 17%——
Late outageCode reference300/600 · 50%——
Delivered utility Y by actor and condition
The same outcomes and denominators as the table. Higher Delivered utility Y is better.

Filled bars: model arms. Outlined bars: code references. Every Delivered utility Y scale runs from 0 to 100%. Bars show descriptive levels, without inferred error bars.

Qwen 2.5 7B

50 episodes per world and framing

Qwen 2.5 7B: Delivered utility YFull priority · calm: Delivered utility Y 0/600 · 0%. Full priority · early outage: Delivered utility Y 0/600 · 0%. Full priority · late outage: Delivered utility Y 0/600 · 0%. Fallback first · calm: Delivered utility Y 0/600 · 0%. Fallback first · early outage: Delivered utility Y 0/600 · 0%. Fallback first · late outage: Delivered utility Y 0/600 · 0%. Exact totals and model validity appear in the table.Full priority · calm0%Full priority · early outage0%Full priority · late outage0%Fallback first · calm0%Fallback first · early outage0%Fallback first · late outage0%0100%

Llama 3.1 8B

50 episodes per world and framing

Llama 3.1 8B: Delivered utility YFull priority · calm: Delivered utility Y 0/600 · 0%. Full priority · early outage: Delivered utility Y 0/600 · 0%. Full priority · late outage: Delivered utility Y 0/600 · 0%. Fallback first · calm: Delivered utility Y 0/600 · 0%. Fallback first · early outage: Delivered utility Y 0/600 · 0%. Fallback first · late outage: Delivered utility Y 0/600 · 0%. Exact totals and model validity appear in the table.Full priority · calm0%Full priority · early outage0%Full priority · late outage0%Fallback first · calm0%Fallback first · early outage0%Fallback first · late outage0%0100%

Qwen 2.5 1.5B

50 episodes per world and framing

Qwen 2.5 1.5B: Delivered utility YFull priority · calm: Delivered utility Y 0/600 · 0%. Full priority · early outage: Delivered utility Y 0/600 · 0%. Full priority · late outage: Delivered utility Y 0/600 · 0%. Fallback first · calm: Delivered utility Y 0/600 · 0%. Fallback first · early outage: Delivered utility Y 0/600 · 0%. Fallback first · late outage: Delivered utility Y 0/600 · 0%. Exact totals and model validity appear in the table.Full priority · calm0%Full priority · early outage0%Full priority · late outage0%Fallback first · calm0%Fallback first · early outage0%Fallback first · late outage0%0100%

Claude

10 episodes per world and framing · descriptive

Claude: Delivered utility YFull priority · calm: Delivered utility Y 108/120 · 90%. Full priority · early outage: Delivered utility Y 18/120 · 15%. Full priority · late outage: Delivered utility Y 60/120 · 50%. Fallback first · calm: Delivered utility Y 120/120 · 100%. Fallback first · early outage: Delivered utility Y 20/120 · 17%. Fallback first · late outage: Delivered utility Y 60/120 · 50%. Exact totals and model validity appear in the table.Full priority · calm90%Full priority · early outage15%Full priority · late outage50%Fallback first · calm100%Fallback first · early outage17%Fallback first · late outage50%0100%

Full-only script

50 matched seeds · same result in both framings

Full-only script: Delivered utility YCalm (code reference): Delivered utility Y 600/600 · 100%. Early outage (code reference): Delivered utility Y 0/600 · 0%. Late outage (code reference): Delivered utility Y 0/600 · 0%. Exact totals and model validity appear in the table.Calm100%Early outage0%Late outage0%0100%

Highest-feasible script

50 matched seeds · same result in both framings

Highest-feasible script: Delivered utility YCalm (code reference): Delivered utility Y 600/600 · 100%. Early outage (code reference): Delivered utility Y 100/600 · 17%. Late outage (code reference): Delivered utility Y 300/600 · 50%. Exact totals and model validity appear in the table.Calm100%Early outage17%Late outage50%0100%

Core-only script

50 matched seeds · same result in both framings

Core-only script: Delivered utility YCalm (code reference): Delivered utility Y 100/600 · 17%. Early outage (code reference): Delivered utility Y 100/600 · 17%. Late outage (code reference): Delivered utility Y 100/600 · 17%. Exact totals and model validity appear in the table.Calm17%Early outage17%Late outage17%0100%

Omniscient ceiling

50 matched seeds · same result in both framings

Omniscient ceiling: Delivered utility YCalm (code reference): Delivered utility Y 600/600 · 100%. Early outage (code reference): Delivered utility Y 100/600 · 17%. Late outage (code reference): Delivered utility Y 300/600 · 50%. Exact totals and model validity appear in the table.Calm100%Early outage17%Late outage50%0100%

Script rows use all 50 local seeds, including Claude’s ten-seed subset; values are invariant across framings and seeds within each world. All scripted action sequences satisfy the simulator contract. The oracle alone sees hidden tool health.

The registered comparison

Zero benefit, with a coverage shortfall.

The primary averages early and late outages within each of Qwen 7B’s 50 seed blocks. Both ΔY and ΔH were 0 [0, 0]. Technical completion was 100/100 fallback versus 90/100 full, below the required 95/100 in both arms. There were 45 complete blocks, enough for the count requirement, but their positive-effect check also failed.

Intervals use the frozen 10,000 paired seed-block bootstrap draws. An observed floor on three repeated synthetic structures produces degenerate intervals; it does not establish equivalence or rule out useful fallback instructions on other interfaces.

Qwen obtained 385 useful artifact kinds and Llama 300 without delivering them. This post hoc observation is consistent with interface and receipt-state tracking difficulties; the experiment did not isolate that mechanism. Fewer full claims under fallback also did not establish accurate reporting: Qwen’s outage report accuracy stayed 0/100 in each arm.

Predictions stay visible

The false-full forecast held.

  1. P1 · Failed. Fallback would improve Qwen 7B’s outage delivery and accurately reported utility under the joint rule. Both effects were zero; coverage failed.
  2. P2 · Failed. Llama would replicate positive Y and H signs. Both differences were zero; ties fail.
  3. P3 · Failed. Qwen would deliver the full result in at least 90% of each calm arm. It did so in 0/50 episodes in each.
  4. P4 · Failed. Fallback would reduce Qwen’s repeated failed calls. It made 47 repeated failures across 100 outage episodes, versus zero under full priority.
  5. P5 · Failed. Qwen 1.5B fallback outage H would be lower than Qwen 7B’s. Both were zero.
  6. P6 · Failed. Claude’s framing ΔY would be nonnegative and smaller than Qwen’s. Its observed +0.0083 exceeded zero, entirely associated with an early/full transport loss.
  7. P7 · Held. At least one Qwen 7B or Llama outage report would falsely claim full delivery. There were 104/400.

How the record holds up

The failures remain in the denominator.

USD 4.92 in model-call accounting
Settled API-equivalent spend is USD 4.924020, all Claude, with no outstanding holds. Local model-call accounting is zero; hardware and time were not priced. This is audited saved usage accounting; cash billing is unknown.
Preserved errors and operational amendments
Two Claude transport failures remain Y=H=0. One exact-zero usage-limit refusal was an operational nonstart with an identical-request resume. Episode-local schema and length failures were retained. A1/D1/D2 records preserve the original inputs, stops and dispositions.
Offline, byte-identical rebuild
The filtered-copy rebuild reproduced the frozen analysis byte for byte. Delivery tables and traces link back to all 960 episodes; reproduction needs Python’s standard library, with no model calls.

Where this stops

An operational floor in a small world.

Seeds vary identifiers, receipts and local sampling across three structural worlds; they do not supply new dependency graphs. Utilities 12/6/2/0 are assigned values, not measured human benefit. Local calls used JSON mode with independent strict validation, without schema-constrained decoding; Qwen 1.5B failed its first action’s reference contract in all 300 episodes. Its zeros chiefly describe interface failure.

Qwen 7B’s 285 technically complete episodes still delivered nothing. Claude’s success demonstrates that this harness can award utility, but provider, interface and runtime differences prevent a causal model-family ranking. Sixty descriptive Claude episodes cannot establish production reliability or a general framing advantage.

Built with Codex; reviewed by Claude. The joint scientific prediction failed. Percentages are rounded from exact utility totals or episode counts; the current delivery report supersedes frozen historical root status text.