← Research

Research · Tool interfaces · 12 November 2026

Bridge or Bus?
A manual helps
only if you use it.

The tool changed. The current description was one call away.
The local models never asked for it.

An agent looks up a weight, converts it, then submits an answer. Between the lookup and the next step, the tool can change: an argument gets renamed, a unit changes, or a new field becomes compulsory. Does access to the current manual help the agent finish correctly?

Hypothesis

Letting an agent look up the current tool description improves successful completion after an interface changes.

The primary Qwen 7B test failed. Every model failed the joint rule. The local models produced valid output in the corrected study, but none used the description lookup. Claude showed descriptive gains after renaming and unit changes, while failing the technical validity gate.

The joint rule requires completion, truthful acknowledgment, stable-world competence, valid trajectories and reports, and few false completion claims. A gain in one cell does not earn a pass.

What we tested instead, and why

Can an agent use the current manual?

The original Bridge or Bus playable seed asks whether tools should connect through direct adapters or a shared translation hub. This study takes a smaller, cheaper information-access question: does permission to “look up the current tool description” help after an interface changes? It compares stale descriptions with optional registry access; it does not compare adapters with a hub.

Both arms start with the same old descriptions and a warning that tools may change. The stale arm can keep working and learn from error messages. The registry arm may also call describe for a current description. That lookup uses one of the same six decisions; updates are never automatic. Both arms receive the same tool errors.

Stable
The interface stays the same.
Rename
Conversion now expects mass_g instead of grams.
Unit
Conversion returns milligrams instead of grams; submission still expects grams.
Required
Submission now also needs revision=1.

There are 1,280 valid-study assignments: 400 for each of three local models and 80 for Claude. Each local cell has 50 seeds; each Claude cell has 10. The intervention happens after the first successful lookup. Episodes that never reach that step stay in the denominator.

The result · all assigned episodes

Access did not guarantee use.

Qwen 7B completed 28 of 400 assignments. Llama 8B and Qwen 1.5B completed 0 of 400 each: they looped through lookups without ever converting. All three local models had valid trajectories and reports throughout r2, but zero description lookups. Formatting improved; task performance remained poor.

Successful completion (S*) by actor, arm and world · higher is better

Stale (S) Registry (R). Every pair uses a 0–100% scale. Labels show rounded percentages; exact counts follow below.

Qwen 2.5 7B

Primary · n=50/cell · r2

Qwen 2.5 7B: successful completion by world and armPrimary · n=50/cell · r2. Stable: stale 5 of 50, 10%; registry 2 of 50, 4%. Rename: stale 0 of 50, 0%; registry 0 of 50, 0%. Unit: stale 6 of 50, 12%; registry 2 of 50, 4%. Required: stale 3 of 50, 6%; registry 10 of 50, 20%.StableS10%R4%RenameS0%R0%UnitS12%R4%RequiredS6%R20%0%100%

Claude

Descriptive · n=10/cell · r3

Claude: successful completion by world and armDescriptive · n=10/cell · r3. Stable: stale 10 of 10, 100%; registry 10 of 10, 100%. Rename: stale 3 of 10, 30%; registry 8 of 10, 80%. Unit: stale 6 of 10, 60%; registry 9 of 10, 90%. Required: stale 9 of 10, 90%; registry 9 of 10, 90%.StableS100%R100%RenameS30%R80%UnitS60%R90%RequiredS90%R90%0%100%

Llama 3.1 8B

Secondary · n=50/cell · r2

Llama 3.1 8B: successful completion by world and armSecondary · n=50/cell · r2. Stable: stale 0 of 50, 0%; registry 0 of 50, 0%. Rename: stale 0 of 50, 0%; registry 0 of 50, 0%. Unit: stale 0 of 50, 0%; registry 0 of 50, 0%. Required: stale 0 of 50, 0%; registry 0 of 50, 0%.StableS0%R0%RenameS0%R0%UnitS0%R0%RequiredS0%R0%0%100%

Qwen 2.5 1.5B

Secondary · n=50/cell · r2

Qwen 2.5 1.5B: successful completion by world and armSecondary · n=50/cell · r2. Stable: stale 0 of 50, 0%; registry 0 of 50, 0%. Rename: stale 0 of 50, 0%; registry 0 of 50, 0%. Unit: stale 0 of 50, 0%; registry 0 of 50, 0%. Required: stale 0 of 50, 0%; registry 0 of 50, 0%.StableS0%R0%RenameS0%R0%UnitS0%R0%RequiredS0%R0%0%100%

Reactive script

Code baseline · n=50/cell

Reactive script: successful completion by world and armCode baseline · n=50/cell. Stable: stale 50 of 50, 100%; registry 50 of 50, 100%. Rename: stale 50 of 50, 100%; registry 50 of 50, 100%. Unit: stale 50 of 50, 100%; registry 50 of 50, 100%. Required: stale 50 of 50, 100%; registry 50 of 50, 100%.StableS100%R100%RenameS100%R100%UnitS100%R100%RequiredS100%R100%0%100%

Oracle script

Code baseline · n=50/cell

Oracle script: successful completion by world and armCode baseline · n=50/cell. Stable: stale 50 of 50, 100%; registry 50 of 50, 100%. Rename: stale 50 of 50, 100%; registry 50 of 50, 100%. Unit: stale 50 of 50, 100%; registry 50 of 50, 100%. Required: stale 50 of 50, 100%; registry 50 of 50, 100%.StableS100%R100%RenameS100%R100%UnitS100%R100%RequiredS100%R100%0%100%

Fixed-v0 script

Code baseline · n=50/cell

Fixed-v0 script: successful completion by world and armCode baseline · n=50/cell. Stable: stale 50 of 50, 100%; registry 50 of 50, 100%. Rename: stale 0 of 50, 0%; registry 0 of 50, 0%. Unit: stale 50 of 50, 100%; registry 50 of 50, 100%. Required: stale 0 of 50, 0%; registry 0 of 50, 0%.StableS100%R100%RenameS0%R0%UnitS100%R100%RequiredS0%R0%0%100%

S* counts a correct, accepted submission, verified by replay of the action record. All assigned episodes count; technical failures count as misses. A model’s claim that it finished cannot create success. All outcomes here are known.

Exact counts behind the figure

Each cell is stale / registry, with counts and rounded rates. On a narrow screen, scroll the table sideways.

S* · stale / registry in every cell
ActorStableRenameUnitRequired
Qwen 2.5 7BPrimary · n=50/cell · r2S: 5/50 · 10%
R: 2/50 · 4%
S: 0/50 · 0%
R: 0/50 · 0%
S: 6/50 · 12%
R: 2/50 · 4%
S: 3/50 · 6%
R: 10/50 · 20%
ClaudeDescriptive · n=10/cell · r3S: 10/10 · 100%
R: 10/10 · 100%
S: 3/10 · 30%
R: 8/10 · 80%
S: 6/10 · 60%
R: 9/10 · 90%
S: 9/10 · 90%
R: 9/10 · 90%
Llama 3.1 8BSecondary · n=50/cell · r2S: 0/50 · 0%
R: 0/50 · 0%
S: 0/50 · 0%
R: 0/50 · 0%
S: 0/50 · 0%
R: 0/50 · 0%
S: 0/50 · 0%
R: 0/50 · 0%
Qwen 2.5 1.5BSecondary · n=50/cell · r2S: 0/50 · 0%
R: 0/50 · 0%
S: 0/50 · 0%
R: 0/50 · 0%
S: 0/50 · 0%
R: 0/50 · 0%
S: 0/50 · 0%
R: 0/50 · 0%
Reactive scriptCode baseline · n=50/cellS: 50/50 · 100%
R: 50/50 · 100%
S: 50/50 · 100%
R: 50/50 · 100%
S: 50/50 · 100%
R: 50/50 · 100%
S: 50/50 · 100%
R: 50/50 · 100%
Oracle scriptCode baseline · n=50/cellS: 50/50 · 100%
R: 50/50 · 100%
S: 50/50 · 100%
R: 50/50 · 100%
S: 50/50 · 100%
R: 50/50 · 100%
S: 50/50 · 100%
R: 50/50 · 100%
Fixed-v0 scriptCode baseline · n=50/cellS: 50/50 · 100%
R: 50/50 · 100%
S: 0/50 · 0%
R: 0/50 · 0%
S: 50/50 · 100%
R: 50/50 · 100%
S: 0/50 · 0%
R: 0/50 · 0%

The reactive script succeeds in every world, in both arms. It handles the documented errors and units without a registry lookup. The oracle script also succeeds everywhere, using advance knowledge of the world. These are programmed controls, not model runs.

The fixed-v0 script fails after rename and required-field drift. It still succeeds after unit drift because it submits the original looked-up mass in grams. That shortcut is allowed: this unit test does not force an agent to derive its answer from the changed conversion output.

The primary comparison · pre-registered

Two points more. Far short of the rule.

Across the three changing worlds, Qwen 7B succeeds in 12/150 registry assignments (8%) and 9/150 stale assignments (6%): a +2 percentage-point difference, with a 95% interval of −4 to +8 points. The registered rule required at least +10 points with a lower bound above zero, plus the other gates.

Registry success is only 2/50 (4%) even when nothing changes. Qwen also claims completion falsely on 114/150 registry drift assignments (76%). Its only favourable world contrast is the required-field change: 10/50 versus 3/50. Since it never asks for a description, this cannot be explained as successful use of the registry.

How the comparison was calculated

The primary analysis gives equal weight to rename, unit and required worlds within each seed, then averages over 50 seed blocks. Its interval uses 10,000 paired seed-block resamples, keeping worlds and arms together. Successful acknowledgment additionally requires a valid submitted:true report; here its primary counts match S*. The interval describes this synthetic sample, not reliability on new tasks.

Claude · descriptive · n=10/cell

A visible contrast. A failed validity gate.

80% vs 30%Rename · registry 8/10, stale 3/10
+50 percentage points · descriptive

90% vs 60%Unit · registry 9/10, stale 6/10
+30 percentage points · descriptive

Claude succeeds on all stable assignments and on 9/10 in both required-field arms. Across drift worlds it succeeds on 26/30 registry assignments (87%), against 18/30 stale (60%). These are descriptive results from ten seed blocks; they do not rescue the primary local test.

Access and use differ here too. Claude asks for descriptions in 19/30 registry drift episodes: nine unit episodes, nine required-field episodes and only one rename episode. The rename contrast therefore does not establish that reading the new description caused the gain.

Only 64/80 Claude trajectories and reports are valid. Validity ranges from 3/10 to 10/10 per cell, failing the required per-cell threshold of 95%. Technical failures remain misses. Claude makes zero false completion claims in these assignments, but this small sample does not establish a generally reliable workflow.

Predictions stay visible

One of thirteen forecasts held.

These forecasts were re-registered after seeing v0.1, before the valid v0.2 collections. P3 held for Qwen because its rename and required-field contrasts exceeded its negative unit contrast. That narrow ordering did not establish useful registry access.

A truthful report can say that work did not finish. It cannot raise successful completion. Null is a deliberate “can’t tell”; an invalid or missing report is a separate failure. All drift-report denominators include every assigned registry drift episode. Scroll sideways on a narrow screen.

All 13 model-specific pre-registered v0.2 forecasts · pp = percentage points
Forecast / actorPredictionOutcome
P1 · Qwen 7BStable/stale success ≥90%Failed · 5/50 (10%)
P1 · Llama 8BStable/stale success ≥90%Failed · 0/50 (0%)
P2 · Qwen 7BPass the entire joint ruleFailed · Gain +2 pp [−4, +8]; completion, stable and reporting gates also fail.
P2 · Llama 8BPositive registry gain across drift worldsFailed · 0/150 in both arms; gain 0 pp.
P3 · Qwen 7BRename and required-field gains each exceed the unit gainHeld · 0 pp and +14 pp each exceed −8 pp. No description lookups.
P3 · Llama 8BRename and required-field gains each exceed the unit gainFailed · All three gains are 0 pp; ties fail.
P4 · Qwen 7BDescribe in ≥70% of registry drift episodesFailed · 0/150 (0%)
P4 · Llama 8BDescribe in ≥70% of registry drift episodesFailed · 0/150 (0%)
P5 · Claude · descriptiveRegistry drift success ≥90%; gain smaller than Qwen’sFailed · 26/30 (87%); gain +27 pp versus Qwen’s +2 pp.
P6 · Qwen 7BTruthful reports ≥90%; deliberate null reports ≤5% in registry driftFailed · Truthful 22/150 (15%); null 14/150 (9%).
P6 · Llama 8BTruthful reports ≥90%; deliberate null reports ≤5% in registry driftFailed · Truthful 114/150 (76%); null 27/150 (18%).
P6 · Qwen 1.5BTruthful reports ≥90%; deliberate null reports ≤5% in registry driftFailed · Truthful 68/150 (45%); null 44/150 (29%).
P6 · Claude · descriptiveTruthful reports ≥90%; deliberate null reports ≤5% in registry driftFailed · Truthful 26/30 (87%); null 0/30 (0%).

How the record holds up

The broken runs stay in the record.

v0.1 · interface failures
All 1,200 local episodes produced zero correct accepted submissions; every local cell failed the technical gate. Independent replay found zero legal replies falsely rejected among 953 rejected replies. Claude’s v0.1 cohort was held. This version could not answer the intended information-access question.
v0.2-r1 · void because of our schema bug
Incomplete schema branches let the decoder emit partial action objects. All 1,200 local episodes stopped before an accepted action. These are excluded harness incidents, not evidence of model performance. Their original bytes, costs and failures remain preserved.
v0.2-r2 / r3 · valid collections, failed scientific tests
R2 closes the broken schema branches and supplies the 1,200 local assignments shown above. R3 adapts the provider schema for Claude’s 80 assignments while retaining strict validation and the r2 local record. Technical failures within a valid collection still count. The versions are kept separate; neither a repaired harness nor completed collection earns a scientific pass.
Informed follow-up forecasts
The v0.2 predictions were registered after inspecting v0.1. Only Qwen’s P3 world-order forecast held; every other model-specific forecast failed, including Claude’s forecast. They were not blind predictions.
Final review · accepted as complete, primary joint test failed
Claude’s final review accepted the study as complete under the final v0.2 stop rule, with r1 excluded. An independently rebuilt filtered copy reproduced the analysis byte-for-byte. Completion leaves the failed scientific test and Claude’s unmet technical validity gate visible.
USD 4.67 in accounted model-call spend
The ledger totals USD 4.667178, with no unresolved money holds. This is model-call accounting, not a verified cash invoice. Local hardware, energy and subscription opportunity costs were not measured. One definite zero-usage rate-limit refusal was a non-start; its saved request was resumed.

Where this stops

A small tool task, with useful shortcuts.

This is a synthetic, three-tool task with six decisions and simple, disclosed error messages. The reactive script shows that registry access is unnecessary to solve these worlds. The study does not establish the best integration architecture, production reliability, or a ranking of model families.

Correct unit answers can come from retaining the original lookup or dividing the milligram result by 1,000. The submitted numbers are identical, so the action record cannot identify which calculation occurred. Exclusive counts for those routes remain unknown.

Claude has only ten seeds per cell; locals have fifty. Output constraints and transports also differ. Do not turn this small descriptive contrast into a general causal or capability claim. Better formatting does not ensure competent action, and providing a manual does not ensure an agent will consult it.

Built with Codex; reviewed by Claude. The figure uses exact successful-submission counts from the frozen v0.2 analysis, with percentages rounded from those counts. Historical v0.1 outcomes and the void r1 incident remain separate.