The tool changed. The current description was one call away. The local models never asked for it.
An agent looks up a weight, converts it, then submits an answer. Between the lookup and the next step, the tool can change: an argument gets renamed, a unit changes, or a new field becomes compulsory. Does access to the current manual help the agent finish correctly?
Hypothesis
Letting an agent look up the current tool description improves successful completion after an interface changes.
The primary Qwen 7B test failed. Every model failed the joint rule. The local models produced valid output in the corrected study, but none used the description lookup. Claude showed descriptive gains after renaming and unit changes, while failing the technical validity gate.
The joint rule requires completion, truthful acknowledgment, stable-world competence, valid trajectories and reports, and few false completion claims. A gain in one cell does not earn a pass.
What we tested instead, and why
Can an agent use the current manual?
The original Bridge or Bus playable seed asks whether tools should connect through direct adapters or a shared translation hub. This study takes a smaller, cheaper information-access question: does permission to “look up the current tool description” help after an interface changes? It compares stale descriptions with optional registry access; it does not compare adapters with a hub.
Both arms start with the same old descriptions and a warning that tools may change. The stale arm can keep working and learn from error messages. The registry arm may also call describe for a current description. That lookup uses one of the same six decisions; updates are never automatic. Both arms receive the same tool errors.
Stable
The interface stays the same.
Rename
Conversion now expects mass_g instead of grams.
Unit
Conversion returns milligrams instead of grams; submission still expects grams.
Required
Submission now also needs revision=1.
There are 1,280 valid-study assignments: 400 for each of three local models and 80 for Claude. Each local cell has 50 seeds; each Claude cell has 10. The intervention happens after the first successful lookup. Episodes that never reach that step stay in the denominator.
The result · all assigned episodes
Access did not guarantee use.
Qwen 7B completed 28 of 400 assignments. Llama 8B and Qwen 1.5B completed 0 of 400 each: they looped through lookups without ever converting. All three local models had valid trajectories and reports throughout r2, but zero description lookups. Formatting improved; task performance remained poor.
Successful completion (S*) by actor, arm and world · higher is better
Stale (S) Registry (R). Every pair uses a 0–100% scale. Labels show rounded percentages; exact counts follow below.
Qwen 2.5 7B
Primary · n=50/cell · r2
Claude
Descriptive · n=10/cell · r3
Llama 3.1 8B
Secondary · n=50/cell · r2
Qwen 2.5 1.5B
Secondary · n=50/cell · r2
Reactive script
Code baseline · n=50/cell
Oracle script
Code baseline · n=50/cell
Fixed-v0 script
Code baseline · n=50/cell
S* counts a correct, accepted submission, verified by replay of the action record. All assigned episodes count; technical failures count as misses. A model’s claim that it finished cannot create success. All outcomes here are known.
Exact counts behind the figure
Each cell is stale / registry, with counts and rounded rates. On a narrow screen, scroll the table sideways.
S* · stale / registry in every cell
Actor
Stable
Rename
Unit
Required
Qwen 2.5 7BPrimary · n=50/cell · r2
S: 5/50 · 10% R: 2/50 · 4%
S: 0/50 · 0% R: 0/50 · 0%
S: 6/50 · 12% R: 2/50 · 4%
S: 3/50 · 6% R: 10/50 · 20%
ClaudeDescriptive · n=10/cell · r3
S: 10/10 · 100% R: 10/10 · 100%
S: 3/10 · 30% R: 8/10 · 80%
S: 6/10 · 60% R: 9/10 · 90%
S: 9/10 · 90% R: 9/10 · 90%
Llama 3.1 8BSecondary · n=50/cell · r2
S: 0/50 · 0% R: 0/50 · 0%
S: 0/50 · 0% R: 0/50 · 0%
S: 0/50 · 0% R: 0/50 · 0%
S: 0/50 · 0% R: 0/50 · 0%
Qwen 2.5 1.5BSecondary · n=50/cell · r2
S: 0/50 · 0% R: 0/50 · 0%
S: 0/50 · 0% R: 0/50 · 0%
S: 0/50 · 0% R: 0/50 · 0%
S: 0/50 · 0% R: 0/50 · 0%
Reactive scriptCode baseline · n=50/cell
S: 50/50 · 100% R: 50/50 · 100%
S: 50/50 · 100% R: 50/50 · 100%
S: 50/50 · 100% R: 50/50 · 100%
S: 50/50 · 100% R: 50/50 · 100%
Oracle scriptCode baseline · n=50/cell
S: 50/50 · 100% R: 50/50 · 100%
S: 50/50 · 100% R: 50/50 · 100%
S: 50/50 · 100% R: 50/50 · 100%
S: 50/50 · 100% R: 50/50 · 100%
Fixed-v0 scriptCode baseline · n=50/cell
S: 50/50 · 100% R: 50/50 · 100%
S: 0/50 · 0% R: 0/50 · 0%
S: 50/50 · 100% R: 50/50 · 100%
S: 0/50 · 0% R: 0/50 · 0%
The reactive script succeeds in every world, in both arms. It handles the documented errors and units without a registry lookup. The oracle script also succeeds everywhere, using advance knowledge of the world. These are programmed controls, not model runs.
The fixed-v0 script fails after rename and required-field drift. It still succeeds after unit drift because it submits the original looked-up mass in grams. That shortcut is allowed: this unit test does not force an agent to derive its answer from the changed conversion output.
The primary comparison · pre-registered
Two points more. Far short of the rule.
Across the three changing worlds, Qwen 7B succeeds in 12/150 registry assignments (8%) and 9/150 stale assignments (6%): a +2 percentage-point difference, with a 95% interval of −4 to +8 points. The registered rule required at least +10 points with a lower bound above zero, plus the other gates.
Registry success is only 2/50 (4%) even when nothing changes. Qwen also claims completion falsely on 114/150 registry drift assignments (76%). Its only favourable world contrast is the required-field change: 10/50 versus 3/50. Since it never asks for a description, this cannot be explained as successful use of the registry.
How the comparison was calculated
The primary analysis gives equal weight to rename, unit and required worlds within each seed, then averages over 50 seed blocks. Its interval uses 10,000 paired seed-block resamples, keeping worlds and arms together. Successful acknowledgment additionally requires a valid submitted:true report; here its primary counts match S*. The interval describes this synthetic sample, not reliability on new tasks.
Claude · descriptive · n=10/cell
A visible contrast. A failed validity gate.
80% vs 30%Rename · registry 8/10, stale 3/10 +50 percentage points · descriptive
90% vs 60%Unit · registry 9/10, stale 6/10 +30 percentage points · descriptive
Claude succeeds on all stable assignments and on 9/10 in both required-field arms. Across drift worlds it succeeds on 26/30 registry assignments (87%), against 18/30 stale (60%). These are descriptive results from ten seed blocks; they do not rescue the primary local test.
Access and use differ here too. Claude asks for descriptions in 19/30 registry drift episodes: nine unit episodes, nine required-field episodes and only one rename episode. The rename contrast therefore does not establish that reading the new description caused the gain.
Only 64/80 Claude trajectories and reports are valid. Validity ranges from 3/10 to 10/10 per cell, failing the required per-cell threshold of 95%. Technical failures remain misses. Claude makes zero false completion claims in these assignments, but this small sample does not establish a generally reliable workflow.
Predictions stay visible
One of thirteen forecasts held.
These forecasts were re-registered after seeing v0.1, before the valid v0.2 collections. P3 held for Qwen because its rename and required-field contrasts exceeded its negative unit contrast. That narrow ordering did not establish useful registry access.
A truthful report can say that work did not finish. It cannot raise successful completion. Null is a deliberate “can’t tell”; an invalid or missing report is a separate failure. All drift-report denominators include every assigned registry drift episode. Scroll sideways on a narrow screen.
Truthful reports ≥90%; deliberate null reports ≤5% in registry drift
Failed · Truthful 26/30 (87%); null 0/30 (0%).
How the record holds up
The broken runs stay in the record.
v0.1 · interface failures
All 1,200 local episodes produced zero correct accepted submissions; every local cell failed the technical gate. Independent replay found zero legal replies falsely rejected among 953 rejected replies. Claude’s v0.1 cohort was held. This version could not answer the intended information-access question.
v0.2-r1 · void because of our schema bug
Incomplete schema branches let the decoder emit partial action objects. All 1,200 local episodes stopped before an accepted action. These are excluded harness incidents, not evidence of model performance. Their original bytes, costs and failures remain preserved.
R2 closes the broken schema branches and supplies the 1,200 local assignments shown above. R3 adapts the provider schema for Claude’s 80 assignments while retaining strict validation and the r2 local record. Technical failures within a valid collection still count. The versions are kept separate; neither a repaired harness nor completed collection earns a scientific pass.
Informed follow-up forecasts
The v0.2 predictions were registered after inspecting v0.1. Only Qwen’s P3 world-order forecast held; every other model-specific forecast failed, including Claude’s forecast. They were not blind predictions.
Final review · accepted as complete, primary joint test failed
Claude’s final review accepted the study as complete under the final v0.2 stop rule, with r1 excluded. An independently rebuilt filtered copy reproduced the analysis byte-for-byte. Completion leaves the failed scientific test and Claude’s unmet technical validity gate visible.
USD 4.67 in accounted model-call spend
The ledger totals USD 4.667178, with no unresolved money holds. This is model-call accounting, not a verified cash invoice. Local hardware, energy and subscription opportunity costs were not measured. One definite zero-usage rate-limit refusal was a non-start; its saved request was resumed.
Where this stops
A small tool task, with useful shortcuts.
This is a synthetic, three-tool task with six decisions and simple, disclosed error messages. The reactive script shows that registry access is unnecessary to solve these worlds. The study does not establish the best integration architecture, production reliability, or a ranking of model families.
Correct unit answers can come from retaining the original lookup or dividing the milligram result by 1,000. The submitted numbers are identical, so the action record cannot identify which calculation occurred. Exclusive counts for those routes remain unknown.
Claude has only ten seeds per cell; locals have fifty. Output constraints and transports also differ. Do not turn this small descriptive contrast into a general causal or capability claim. Better formatting does not ensure competent action, and providing a manual does not ensure an agent will consult it.
Built with Codex; reviewed by Claude. The figure uses exact successful-submission counts from the frozen v0.2 analysis, with percentages rounded from those counts. Historical v0.1 outcomes and the void r1 incident remain separate.