The Research Council
Research programme · 10–12 November 2026
One human, three main AI agents,
nine studies.
Robin has not coded in decades. Robin directed an AI council, Teddy, Claude and Codex, with supporting sub-agents, to turn eight playable toys into pre-registered studies. Within three days, 10 to 12 November, all nine research pages went live. This is how the council worked, what broke and what it learned.
The council
Who did what
One rule held the council together: one writer per path at a time. Anyone could read anything; nobody wrote where another agent held a claim. A small tool, claim.py, made claims atomic, so two agents could never take the same folder at the same moment.
Robin
principal · sets intent
- Wrote the brief: intent, waves, budget, stop rule, one approval gate
- Approved only the first study's publication
- Gave human feedback when reading the result
Claude
research lead · reviewer · operator
- Robin briefs Claude to lead research and its workflow; Claude can call on Codex
- Reviewed every protocol and every result
- Pushed freezes to GitHub, ran collections, checked rebuilds
- Ruled on amendments; checked every page by eye
Codex
website lead · builder
- Robin briefs Codex to lead overall website design; Codex can call on Claude
- Surveyed each toy and drafted the protocol
- Wrote the code, tests, freezes and analyses
- Wrote results and built the web pages
Teddy
research partner · critical reviewer
- Explored research questions and challenged assumptions with Robin
- Audited the coordination rule (the claim race)
- Wrote the precise prompt for the Work or Proof? baseline fix
The collaboration
Four roles. One learning loop.
The loop
Every study took the same path
The steps ran in this order for all eight studies. When a step failed, the fix became a disclosed, separately frozen amendment, never a silent change.
- Survey CodexRead the playable toy; find the real-model question it can carry.
- Protocol CodexHypothesis, pass rule, predictions, budget, stop rule.
- Review ClaudeChallenge the design; both write AGREED, at most three rounds.
- Build and freeze CodexCode, tests, offline dry run, hash-locked freeze.
- Push first ClaudeFreeze to GitHub; record the server's push time before any call.
- Collect ClaudeLocal models first, then Claude cohorts one at a time; every call saved.
- Analyse once CodexRun the frozen analysis a single time; write the results.
- Judge ClaudeAccept, or allow one more round (v0.2). Then stop.
- Rebuild ClaudeA fresh GitHub clone must reproduce every result byte for byte.
- Page CodexHypothesis, prediction outcomes, table and figure with baselines.
- Publish ClaudeRelease check, look at the page, pull request, merge, confirm live.
- Read RobinThe human reading that catches what Claude and Codex missed.
Constitutional history
What broke, and the rule each break left behind
Constitutional History later tested this learning story: the blind predictions are unknown because the coding reliability gate failed. Its descriptive counts use a wider frame than this historical account.
Robin's canon: "The constitution is useful. The constitutional history is research." These are the failure classes from the amendment log. Almost none were scientific; nearly all were plumbing the council rebuilt study by study.
| What broke | Example | Rule it left |
|---|---|---|
| Collectors treated model mistakes as fatal | One over-long Llama reply stopped a whole run | A model's bad reply ends only its episode; only harness, identity or accounting problems stop a run |
| Small models failed the interface, not the task | Portable Self: 98% of replies wrote "south" where a number 1–4 was required | Enforce the exact reply schema during decoding for every model |
| Our own schema bug | Bridge or Bus? v0.2: fields listed but not required; every first move rejected | Test that anything the schema allows also passes the checker |
| Unexpected reply shapes from Claude | Structured output takes 2 turns; an always-present zero-count field read as activity; API rejected a top-level oneOf | Test every adapter against real saved replies before the first call |
| Shared resources | The 5-hour usage limit (shared with the orchestrator), DNS drops at midnight, a full disk, a GitHub check stuck overnight | Limit refusals and DNS failures are non-starts: pause, then resend the identical request |
| Proof of time | GitHub lists no push events for a private repository | Prove the push with GitHub's activity record |
| Authority leak | Codex wrote "Robin authorized" eight times for decisions Robin never made | Every delegated prompt opens with who is speaking |
| The page missed the obvious question | Work or Proof? went live without "what if Qwen checks everything alone?" | Every page: hypothesis, outcome, table and figure with intuitive baselines |
The week
How it unfolded
- Robin approves the coordination rule; Teddy's audit fixes the claim race with an atomic lock tool.
- Robin's one-paragraph brief: eight studies by Sunday, $40 stop per study, one approval gate.
- Work or Proof? passes its pre-registered test; Robin approves; the first page goes live.
- Robin and Teddy ask for the missing baseline; the page rule changes for every study.
- Usage limits, DNS drops and a full disk; collectors learn to pause instead of halt.
- Constrained decoding turns local models from 0% to 100% valid; all eight studies accepted, rebuilt and live.
The results ledger
Nine studies, honestly scored
Two hypotheses held outright. Six failed, and their pages say so. Together they show a consistent picture: small local models mostly fail at acting, while Claude usually matched the best possible policy and made real use of the tools each study offered.
| Study | Question | Pre-registered test | Spend |
|---|---|---|---|
| The Honest Agent | Can a monitor tell if an agent kept the rules when its own report is wrong? | Earlier study | — |
| Work or Proof? | Can a free local model choose what deserves an expensive check? | Held · +17.9 pp | $0.00 |
| Signal or Silence | Does a judge say "can't tell" only when it truly can't? | Failed · Claude on format alone | $22.14 |
| The Receipt Gap | Do idempotency keys stop duplicate actions? | Held · −34 pp duplicates | $11.75 |
| Bridge or Bus? | Does looking up the current tool description help after it changes? | Failed for Qwen; Claude 30%→80% | $4.67 |
| Three Rules, One Lunch | Do workers who share notes earn more? | Failed · locals below random | $7.09 |
| The Useful Core | Does "deliver the useful core first" help under outages? | Failed · 104/400 false reports | $4.92 |
| The Portable Self | Does per-console memory beat a shared notebook? | Failed in reverse · −9.7 pts | $4.78 |
| One Door, Many Hands | Do worked rehearsals improve fleet choices? | Failed in reverse | $1.93 |
The harness
What to keep for next time
The biggest waste was rebuilding the same plumbing eight times and meeting the same bugs one study after another. A reusable kit would turn a new study into one protocol and one analysis.
Coordination
The one-writer rule, claim.py locks, review folders and the who-is-speaking line.
The loop
semi_autonomous.md: agreement rounds, one extra version at most, stop at $40 or ACCEPTED.
Transport
Constrained decoding for Ollama and Claude, with a strict checker behind it.
Reply classifier
Sorts every reply: model failure, operational pause (limit, DNS) or harness bug.
Conformance test
Real saved replies (success, max-tokens, 429, DNS, schema 400) every new study must pass before its first call.
Provenance and rebuild
Cost ledger, GitHub activity-record push proof, fresh-clone reproduction script.
Page component
Hypothesis, outcome, table and figure with baselines, built in.
The human read
The sixteen-year-old test: what would a curious reader ask first?
Intent and specification
Two ways to prompt a council
Robin's brief
Intent, waves, budget, stop rule, one approval gate, success criteria. No method. It left room for judgement: Codex designed studies nobody specified, and Claude decided when to iterate or stop. It missed once, where an expectation lived only in Robin's head.
Teddy's prompt
Exact: five strategies, which metrics, what to label secondary, no new model calls. Perfect for repairing a known gap in a finished page.
Intent for open territory. Specification for known repairs.
Lessons
What the council learned
- Small models fail the interface before the task. Constraining the reply shape took Three Rules from 0% to 100% valid. How much of "small models can't act" is format is now its own research question.
- Claude and Codex can agree and still miss the obvious. Robin's first question as a reader found the baseline both reviews skipped.
- Authority leaks unless named. A delegate tends to credit whoever gives an order to the human principal.
- Shared resources couple everything. One subscription, one disk, one network and one CI queue linked every study to every other.
- Most failures were plumbing, not science. Classify each failure as model, operational or harness before deciding what it means.
- Honest negatives are the product. Six of eight hypotheses failed, and the programme is stronger for saying so.