Empirical study · The council studies itself
Constitutional History
Did the council learn?
Our records let us count the council’s failures, but the coders’ labels did not agree reliably enough, leaving the three blind predictions unknown.
Descriptively, plumbing—bugs in the running software or reply interface—made up about three quarters of classified failures, and its share grew over the programme.
Can a rulebook show that we learned?
When something broke, our research council wrote it down, corrected it and often added a rule. That history should be more useful than a polished constitution alone: it records why the rules exist.
But a new rule is not proof of improvement. Studies overlapped, the work changed, and our records captured some problems better than others. We needed a way to distinguish a learning story from evidence of learning.
Did later studies need fewer corrections, did plumbing become less common, did failures recur less after a rule, and did Robin’s reading catch types the agents missed?
The objective and the hypotheses
Objective: test whether the council’s recorded corrections support its claim to have learned.
- H1(a), known before registration: later-starting studies needed fewer recorded corrections. This is a description we already knew, so it receives no “held” verdict.
- H1(b), blind prediction: plumbing made up a smaller share of later studies’ corrections. It would hold if that share declined and passed the registered adjusted significance test.
- H2, blind prediction: a failure type came back less often after its first adopted rule. It would hold if the rate fell across enough reliably dated failure types and passed the adjusted test.
- H3, blind prediction: Robin’s reading caught failure types that the agents’ earlier checks missed. It would hold if independent studies showed genuinely new types after documented missed checks and passed the adjusted test.
We committed to those blind predictions before coding the records. All required reliable labels first; all are unknown because that requirement failed. The exact pass rules appear below the results.
What one test looks like
A real record · Work or Proof?
“the live page omitted the most intuitive baselines”
The page compared a cheap screen plus an expensive check with random checking. It left out the obvious alternatives: let the cheap model check everything, or let the expensive model check everything. Robin, with Teddy, noticed while reading the delivered page.
What the coder saw: the saved review describing the omission, the earlier agent acceptance and the resulting page rule, with dated supporting context.
- Failure class
- Measurement: a missing comparison made the evidence harder to interpret.
- Stage
- Read: the human principal was reading delivered work after agent checks.
- Who caught it
- Robin and Teddy jointly.
- The rule it left
- Every page must show the hypothesis, prediction outcome, and a table and figure with intuitive alternatives.
These are the Codex coder’s labels, illustrated rather than treated as agreed truth. Every nonmissing label needed a source locator, line span and decisive quotation; an uncited label became unknown. The other coder’s admitted stage and detector labels for this record were unknown.
Source: the saved Work or Proof? page-omission review, quoted in the source note. This motivated the human-reading hypothesis; it is not independent confirmation of it.
How we tested the history
We counted separately recorded corrections, including preventive changes and repeated dispositions. A record is not necessarily a new failure. Coders linked repeated accounts for the recurrence and human-reading tests.
Coverage means the share of records where both coders gave a substantive label. Cohen’s κ (kappa) measures agreement beyond chance. The gate used κ including unknown labels, as well as coverage; a strong score on a small surviving subset could not rescue it.
Our workflow, including the stops
We followed the council loop, with disclosed stops and separately frozen changes. The saved evidence and original failed attempts stayed intact. Every change is listed below.
Results · Descriptions and blind predictions kept separate
We could count the records.
We could not confirm learning.
H1(a): were there fewer later corrections?
Known before registration. Later-starting studies tended to have fewer records, but the studies overlapped and had unequal workloads. “Later” means the first survey claim, not completion order; tied starts share a rank.
| Study | First claim | Records | Both reference means | Constant-rate 95% band |
|---|---|---|---|---|
| Work or Proof? | 11:38 | 6 of 63 | 7.875 per study | 3–14 per study |
| Signal or Silence | 12:12 | 10 of 63 | 7.875 per study | 3–14 per study |
| The Receipt Gap | 16:40 | 12 of 63 | 7.875 per study | 3–14 per study |
| Bridge or Bus? | 16:54 | 9 of 63 | 7.875 per study | 3–14 per study |
| Three Rules, One Lunch | 16:54 | 11 of 63 | 7.875 per study | 3–14 per study |
| The Useful Core | 17:58 | 3 of 63 | 7.875 per study | 3–14 per study |
| The Portable Self | 17:58 | 5 of 63 | 7.875 per study | 3–14 per study |
| One Door, Many Hands | 17:58 | 7 of 63 | 7.875 per study | 3–14 per study |
Across 8 studies: Spearman’s ρ = −0.49; exact one-sided shuffled-order p = 0.108 (4,356 of 40,320 assignments). Constant-rate check p = 0.107 (10,656 extreme draws of 99,999; Monte Carlo corrected). These are descriptive reference checks, not a blind prediction verdict.
Why are the blind predictions unknown?
H1(b), H2 and H3: UNKNOWN. Before resolving disagreements, class missed the κ threshold, stage had no paired substantive coverage, and detector missed both thresholds. The frozen rule stops confirmatory evaluation here.
| Field | Paired coverage | κ including unknown | All-label matches | Paired-only κ / matches (secondary) | Gate |
|---|---|---|---|---|---|
| Failure class | 53 / 63 (84%) | 0.675 | 49 / 63 | 0.850 · 48 / 53 | Failed |
| Detection stage | 0 / 63 (0%) | 0.000 | 1 / 63 | Unavailable · 0 / 0 | Failed |
| Who caught it | 30 / 63 (48%) | 0.593 | 47 / 63 | 1.000 · 30 / 30 | Failed |
Both coders faced the same citation rule. Claude’s many uncited labels became unknown. Rule-presence coverage was also 0 of 63, blocking recurrence analysis independently.
| Coder | Admitted / input citations | Narrowed | Dropped | Fields downgraded |
|---|---|---|---|---|
| Codex | 1,146 / 1,154 | 19 / 1,154 | 8 / 1,154 | 8 |
| Claude Sonnet | 142 / 142 | 0 / 142 | 0 / 142 | 372 |
A downgrade removes a field’s label, not a whole record. Field downgrades are counts, not rates. The separate field-tagged citation counts in the limits below exclude rule citations.
H1(b): did plumbing become a smaller share?
Confirmatory outcome: UNKNOWN. The reconciled labels below are descriptive only. They count a class as known only when the evidence-only reconciliation resolved it; they cannot repair the original gate failure.
| Study | Known of all records | Plumbing / known | Share | Possible share of all records |
|---|---|---|---|---|
| Work or Proof? | 3 known of 6 | 0 / 3 | 0% | 0%–50% |
| Signal or Silence | 4 known of 10 | 2 / 4 | 50% | 20%–80% |
| The Receipt Gap | 11 known of 12 | 9 / 11 | 82% | 75%–83% |
| Bridge or Bus? | 6 known of 9 | 5 / 6 | 83% | 56%–89% |
| Three Rules, One Lunch | 10 known of 11 | 8 / 10 | 80% | 73%–82% |
| The Useful Core | 3 known of 3 | 2 / 3 | 67% | 67%–67% |
| The Portable Self | 4 known of 5 | 4 / 4 | 100% | 80%–100% |
| One Door, Many Hands | 7 known of 7 | 6 / 7 | 86% | 86%–86% |
The possible-share range uses all records as the denominator, assigning every missing class outside or inside plumbing. It is not a confidence interval. The pooled reference is a descriptive comparison; the registered shuffled-order and constant-rate analyses cannot supply a confirmatory plumbing result after the failed gate.
H2: did failures recur less after their rules?
| Required evidence | Effect / p | Eligible denominator | Planned references |
|---|---|---|---|
| Reliable class, incident, rule and time labels | Unavailable / unavailable | Unknown; no eligible set evaluated | Shuffled incident times; constant rates scaled to active-study hours; unavailable after gate failure |
H3: did Robin catch types the agents missed?
| Required evidence | Effect / p | Eligible denominator | Planned references |
|---|---|---|---|
| Reliable class, read stage, detector and prior missed-check evidence | Unavailable / unavailable | Unknown; no eligible set evaluated | Within-study shuffled detector labels; unavailable after gate failure |
What this does not show
- This is one council’s short history: 63 records from 8 overlapping studies over about 2 days, not a census of every failure or evidence about councils generally.
- The coders were fresh sessions, but from the same model families as the participants. Claude and Codex were also participant-analysts; blinding reduces rather than removes that dependence.
- Citation discipline changed what was measurable. Claude supplied 138 field-tagged citations versus Codex’s 1,003. Applying the same uncited-label rule erased many Claude labels, including every usable stage pairing. This gate failure measures admissible coding, not just disagreement about the underlying events.
- Records are uneven. Early logging was less systematic; reviews, preventive copying and repeated dispositions affect counts. Active-study hours measure claimed work opportunity, not CPU time or equal effort.
- Descriptions cannot rescue confirmation. Reconciled plumbing shares and the pre-known count pattern do not establish that rules caused learning. Actual H2/H3 effects, p-values and eligible denominators remain unavailable.
- What could change the conclusion: a new, explicitly exploratory coding version with adequate citations and reliable labels, followed by a longer prospective history with clearer incident times and comparable workloads. The original unknown outcomes would remain.
Integrity and changes
Robin approved protocol v0.3 on 14 November 2026 and reconfirmed it on 15 November, after both agents had agreed. The frozen record was pushed before coding: GitHub’s server activity records . Separately frozen amendments were pushed before their affected steps; the frozen analysis ran once.
Spend: USD 1.89 of the USD 5 cap. The settled call cost was USD 1.88766; Codex coding and page building used the subscription.
- Frame revision, before freeze: review findings joined amendments so an absent early deviation log would not mean zero failures; the widening was chosen knowing Work or Proof? had no deviation entries.
- Build decision, source integrity: exact private evidence bytes were retained; a later public derivative must redact machine details.
- Build decision, blinding: a documented derivative removed hypotheses and thresholds from the coder rulebook without changing labelling definitions.
- Build stop, H3 reference: unknown novelty created by shuffled detector assignments counts as novel in the reference only, a conservative choice that cannot favour H3; observed unknowns still stop the test.
- D1: raised the Claude call reservation from USD 0.50 to USD 2.50 to fit the evidence packet and long reply; the USD 5 total cap stayed fixed.
- D1.1: explicitly set the output-token ceiling to 128,000 so the single structured reply could finish.
- Codex attempt: the first attempt omitted the schema from its prompt and returned no coding; the next attempt coded all 63 records.
- D2: mechanically narrowed over-wide citation spans and dropped mismatched quotes, identically for both coders, instead of rejecting an entire submission.
- D3: the harness mistook quoted “usage limit” text inside a successful reply for a provider error; reclassified the saved envelope without a new call.
- D3.1: corrected the settled-amount guard to match the ledger’s exact recorded amount instead of its rounded display.
- D2.1: also made never-cited labels unknown, enforcing the existing citation requirement for both coders before analysis; thresholds and hypotheses stayed fixed.
Disclosure: while diagnosing D3, the lead glimpsed the first fields of one record in a diagnostic prefix. Before deciding D2.1, the lead counted cited field names and uncited fields per coder, including their unequal citation coverage, without inspecting further label values or any agreement or hypothesis result.
The frozen record is held in a private repository and will be released in redacted form. There is no public repository link yet. We have not published raw evidence containing machine paths or identifiers.
Notes, definitions and sources
- Plumbing
- Harness failures (software that runs, saves or accounts for work) or interface failures (the reply contract). Coordination and scientific errors remain separate classes.
- Shuffled-order reference
- Reorder whole study bundles while keeping each bundle intact. It expects no association with time: ρ = 0 and 7.875 records at each position.
- Constant-rate reference
- A Poisson count model expecting 7.875 records per study, with central 95% prediction limits of 3–14. It is a simple reference, not an equal-workload claim.
- Unknown
- Insufficient admissible evidence to evaluate a registered prediction. Its public effect and p-value are null, not zero. No sensitivities were produced by this analysis run.
Exact pass rules: all blind tests first required κ ≥ 0.70 and ≥ 80% paired substantive coverage on class, stage and detector. H1(b) additionally required ≥ 80% class coverage in every nonempty study, at least 3 nonempty studies and a negative share correlation. H2 required at least 2 evaluable classes, positive exposure before and after adoption, and a positive mean reduction in incident rates. H3 required at least 2 eligible human-reading cases from distinct studies, known class and chronology, positive missed-check evidence, and a novelty fraction of 1. Each also required a one-sided p ≤ 0.05 after Holm adjustment across the 3 blind predictions. H1(a) was excluded from this family.
Worked-example citation: Work or Proof? page-omission review, 11 November 2026, lines 1–8 (saved packet P031): the omission, joint discovery and standing page rule. The example uses the admitted Codex labels; the original review and both coder records remain in the frozen private archive.
Evidence: protocol v0.3 and codebook v0.2; survey and feasibility audit; Robin’s approval records; lead build decisions and amendments; server push activity; citation admission reports; settled ledger; and the single frozen analysis’s results. Numerical displays are transcribed from that analysis and the listed evidence; rounded percentages retain their count denominators.