The Useful Core A useful artifact is not a delivery.
The tools can break. Some work can still be useful. Does asking for the useful core first help an agent deliver it?
None of the three local models delivered any utility in any condition: 900 episodes, zero useful deliveries. Qwen 7B produced valid replies in 285/300 episodes and still delivered nothing. Claude reached the assigned world ceiling in 58/60 episodes through the same simulator; its other two episodes ended in transport errors. The local floor is observable behaviour within this interface, rather than an unreachable scoring ceiling.
Hypothesis
Prioritizing useful fallback delivery increases Qwen 7B’s delivered and accurately reported utility during tool outages.
The primary joint test failed. P1–P6 failed; P7 held. Fallback minus full-priority delivered utility and accurately reported utility were both zero, with 95% intervals [0, 0]. Full-priority technical coverage also missed the registered threshold.
104/400 · 26%Qwen 7B and Llama falsely reported full delivery in 104 of their 400 scheduled outage episodes, despite known empty deliveries. Qwen contributed 64/200 (32%); Llama 40/200 (20%). These were valid, determinate reports contradicting the simulator record. Missing reports are a separate outcome. This establishes false reports, without establishing intent or deception.
The question
Making something is not sending it.
An agent has six action decisions to prepare and explicitly deliver a result. Full delivery is worth 12 assigned utility units; a reduced result is worth six; the useful core is worth two; no delivery earns zero. Merely obtaining a useful artifact earns nothing until it is delivered. A final report separately says what was delivered.
Full priority asks for the full job. Fallback first asks for the useful core first, then the full job. Both framings have the same tools, permissions and six-action quota. Calm worlds allow all 12 units; early outages cap delivery at two; late outages cap it at six. Scores always divide by 12, so the outage ceilings remain visible.
Each local model ran 50 paired seeds in each framing and world: 300 episodes per actor. Claude ran ten seeds per cell, 60 episodes total. All 960 scheduled episodes were attempted and have known replayed delivery truth.
Models beside simple scripts
Valid replies. Empty deliveries.
Claude’s fallback framing matched the oracle in every world: 100% of full utility in calm, 17% in early outage, 50% in late outage. Full-priority scores were 90%, 15% and 50%. Its two full-priority transport failures explain that apparent framing gain; the complete-block outage contrast is zero. They are operational failures, not evidence of worse fallback reasoning.
The highest-feasible script reaches the oracle ceiling with receipt-only actions and the same quota. Full-only works in calm worlds and fails under outages; core-only survives but caps utility at 17%. These secondary diagnostics need no model calls. No random policy exists in the frozen design.
Y = delivered utility / 12 per episode. Table fractions are summed utility units / (12 × scheduled episodes), not delivery success counts. H credits those units only with an accurate final report; here H equals Y in every model cell. Validity concerns reply contracts, not delivery. Scripts and oracle have no report phase, so H and model validity are unavailable (—). Percentages are rounded from exact totals. On a narrow screen, scroll the table sideways.
Every scheduled model condition, with matched code references
Actor / condition
Arm or policy
Delivered utility Y ↑
Accurately reported H ↑
Valid model episodes
Qwen 2.5 7B50 episodes per world and framing
Full priority · calm
0/600 · 0%
0/600 · 0%
45/50 · 90%
Full priority · early outage
0/600 · 0%
0/600 · 0%
45/50 · 90%
Full priority · late outage
0/600 · 0%
0/600 · 0%
45/50 · 90%
Fallback first · calm
0/600 · 0%
0/600 · 0%
50/50 · 100%
Fallback first · early outage
0/600 · 0%
0/600 · 0%
50/50 · 100%
Fallback first · late outage
0/600 · 0%
0/600 · 0%
50/50 · 100%
Llama 3.1 8B50 episodes per world and framing
Full priority · calm
0/600 · 0%
0/600 · 0%
27/50 · 54%
Full priority · early outage
0/600 · 0%
0/600 · 0%
27/50 · 54%
Full priority · late outage
0/600 · 0%
0/600 · 0%
27/50 · 54%
Fallback first · calm
0/600 · 0%
0/600 · 0%
12/50 · 24%
Fallback first · early outage
0/600 · 0%
0/600 · 0%
12/50 · 24%
Fallback first · late outage
0/600 · 0%
0/600 · 0%
11/50 · 22%
Qwen 2.5 1.5B50 episodes per world and framing
Full priority · calm
0/600 · 0%
0/600 · 0%
0/50 · 0%
Full priority · early outage
0/600 · 0%
0/600 · 0%
0/50 · 0%
Full priority · late outage
0/600 · 0%
0/600 · 0%
0/50 · 0%
Fallback first · calm
0/600 · 0%
0/600 · 0%
0/50 · 0%
Fallback first · early outage
0/600 · 0%
0/600 · 0%
0/50 · 0%
Fallback first · late outage
0/600 · 0%
0/600 · 0%
0/50 · 0%
Claude10 episodes per world and framing · descriptive
Full priority · calm
108/120 · 90%
108/120 · 90%
9/10 · 90%
Full priority · early outage
18/120 · 15%
18/120 · 15%
9/10 · 90%
Full priority · late outage
60/120 · 50%
60/120 · 50%
10/10 · 100%
Fallback first · calm
120/120 · 100%
120/120 · 100%
10/10 · 100%
Fallback first · early outage
20/120 · 17%
20/120 · 17%
10/10 · 100%
Fallback first · late outage
60/120 · 50%
60/120 · 50%
10/10 · 100%
Full-only script50 matched seeds · same result in both framings
CalmCode reference
600/600 · 100%
—
—
Early outageCode reference
0/600 · 0%
—
—
Late outageCode reference
0/600 · 0%
—
—
Highest-feasible script50 matched seeds · same result in both framings
CalmCode reference
600/600 · 100%
—
—
Early outageCode reference
100/600 · 17%
—
—
Late outageCode reference
300/600 · 50%
—
—
Core-only script50 matched seeds · same result in both framings
CalmCode reference
100/600 · 17%
—
—
Early outageCode reference
100/600 · 17%
—
—
Late outageCode reference
100/600 · 17%
—
—
Omniscient ceiling50 matched seeds · same result in both framings
CalmCode reference
600/600 · 100%
—
—
Early outageCode reference
100/600 · 17%
—
—
Late outageCode reference
300/600 · 50%
—
—
Delivered utility Y by actor and condition The same outcomes and denominators as the table. Higher Delivered utility Y is better.
Filled bars: model arms. Outlined bars: code references. Every Delivered utility Y scale runs from 0 to 100%. Bars show descriptive levels, without inferred error bars.
Qwen 2.5 7B
50 episodes per world and framing
Llama 3.1 8B
50 episodes per world and framing
Qwen 2.5 1.5B
50 episodes per world and framing
Claude
10 episodes per world and framing · descriptive
Full-only script
50 matched seeds · same result in both framings
Highest-feasible script
50 matched seeds · same result in both framings
Core-only script
50 matched seeds · same result in both framings
Omniscient ceiling
50 matched seeds · same result in both framings
Script rows use all 50 local seeds, including Claude’s ten-seed subset; values are invariant across framings and seeds within each world. All scripted action sequences satisfy the simulator contract. The oracle alone sees hidden tool health.
The registered comparison
Zero benefit, with a coverage shortfall.
The primary averages early and late outages within each of Qwen 7B’s 50 seed blocks. Both ΔY and ΔH were 0 [0, 0]. Technical completion was 100/100 fallback versus 90/100 full, below the required 95/100 in both arms. There were 45 complete blocks, enough for the count requirement, but their positive-effect check also failed.
Intervals use the frozen 10,000 paired seed-block bootstrap draws. An observed floor on three repeated synthetic structures produces degenerate intervals; it does not establish equivalence or rule out useful fallback instructions on other interfaces.
Qwen obtained 385 useful artifact kinds and Llama 300 without delivering them. This post hoc observation is consistent with interface and receipt-state tracking difficulties; the experiment did not isolate that mechanism. Fewer full claims under fallback also did not establish accurate reporting: Qwen’s outage report accuracy stayed 0/100 in each arm.
Predictions stay visible
The false-full forecast held.
P1 · Failed. Fallback would improve Qwen 7B’s outage delivery and accurately reported utility under the joint rule. Both effects were zero; coverage failed.
P2 · Failed. Llama would replicate positive Y and H signs. Both differences were zero; ties fail.
P3 · Failed. Qwen would deliver the full result in at least 90% of each calm arm. It did so in 0/50 episodes in each.
P4 · Failed. Fallback would reduce Qwen’s repeated failed calls. It made 47 repeated failures across 100 outage episodes, versus zero under full priority.
P5 · Failed. Qwen 1.5B fallback outage H would be lower than Qwen 7B’s. Both were zero.
P6 · Failed. Claude’s framing ΔY would be nonnegative and smaller than Qwen’s. Its observed +0.0083 exceeded zero, entirely associated with an early/full transport loss.
P7 · Held. At least one Qwen 7B or Llama outage report would falsely claim full delivery. There were 104/400.
How the record holds up
The failures remain in the denominator.
USD 4.92 in model-call accounting
Settled API-equivalent spend is USD 4.924020, all Claude, with no outstanding holds. Local model-call accounting is zero; hardware and time were not priced. This is audited saved usage accounting; cash billing is unknown.
Preserved errors and operational amendments
Two Claude transport failures remain Y=H=0. One exact-zero usage-limit refusal was an operational nonstart with an identical-request resume. Episode-local schema and length failures were retained. A1/D1/D2 records preserve the original inputs, stops and dispositions.
Offline, byte-identical rebuild
The filtered-copy rebuild reproduced the frozen analysis byte for byte. Delivery tables and traces link back to all 960 episodes; reproduction needs Python’s standard library, with no model calls.
Where this stops
An operational floor in a small world.
Seeds vary identifiers, receipts and local sampling across three structural worlds; they do not supply new dependency graphs. Utilities 12/6/2/0 are assigned values, not measured human benefit. Local calls used JSON mode with independent strict validation, without schema-constrained decoding; Qwen 1.5B failed its first action’s reference contract in all 300 episodes. Its zeros chiefly describe interface failure.
Qwen 7B’s 285 technically complete episodes still delivered nothing. Claude’s success demonstrates that this harness can award utility, but provider, interface and runtime differences prevent a causal model-family ranking. Sixty descriptive Claude episodes cannot establish production reliability or a general framing advantage.
Built with Codex; reviewed by Claude. The joint scientific prediction failed. Percentages are rounded from exact utility totals or episode counts; the current delivery report supersedes frozen historical root status text.