← Research

Empirical study · The council studies itself

Constitutional History
Did the council learn?

Our records let us count the council’s failures, but the coders’ labels did not agree reliably enough, leaving the three blind predictions unknown.

Descriptively, plumbing—bugs in the running software or reply interface—made up about three quarters of classified failures, and its share grew over the programme.

Can a rulebook show that we learned?

When something broke, our research council wrote it down, corrected it and often added a rule. That history should be more useful than a polished constitution alone: it records why the rules exist.

But a new rule is not proof of improvement. Studies overlapped, the work changed, and our records captured some problems better than others. We needed a way to distinguish a learning story from evidence of learning.

Did later studies need fewer corrections, did plumbing become less common, did failures recur less after a rule, and did Robin’s reading catch types the agents missed?

The objective and the hypotheses

Objective: test whether the council’s recorded corrections support its claim to have learned.

  • H1(a), known before registration: later-starting studies needed fewer recorded corrections. This is a description we already knew, so it receives no “held” verdict.
  • H1(b), blind prediction: plumbing made up a smaller share of later studies’ corrections. It would hold if that share declined and passed the registered adjusted significance test.
  • H2, blind prediction: a failure type came back less often after its first adopted rule. It would hold if the rate fell across enough reliably dated failure types and passed the adjusted test.
  • H3, blind prediction: Robin’s reading caught failure types that the agents’ earlier checks missed. It would hold if independent studies showed genuinely new types after documented missed checks and passed the adjusted test.

We committed to those blind predictions before coding the records. All required reliable labels first; all are unknown because that requirement failed. The exact pass rules appear below the results.

What one test looks like

A real record · Work or Proof?

“the live page omitted the most intuitive baselines”

The page compared a cheap screen plus an expensive check with random checking. It left out the obvious alternatives: let the cheap model check everything, or let the expensive model check everything. Robin, with Teddy, noticed while reading the delivered page.

What the coder saw: the saved review describing the omission, the earlier agent acceptance and the resulting page rule, with dated supporting context.

Failure class
Measurement: a missing comparison made the evidence harder to interpret.
Stage
Read: the human principal was reading delivered work after agent checks.
Who caught it
Robin and Teddy jointly.
The rule it left
Every page must show the hypothesis, prediction outcome, and a table and figure with intuitive alternatives.

These are the Codex coder’s labels, illustrated rather than treated as agreed truth. Every nonmissing label needed a source locator, line span and decisive quotation; an uncited label became unknown. The other coder’s admitted stage and detector labels for this record were unknown.

Source: the saved Work or Proof? page-omission review, quoted in the source note. This motivated the human-reading hypothesis; it is not independent confirmation of it.

How we tested the history

We counted separately recorded corrections, including preventive changes and repeated dispositions. A record is not necessarily a new failure. Coders linked repeated accounts for the recurrence and human-reading tests.

Reliable coding had to come before testing learning.Eight studies supplied 63 records. Two fresh blind coders applied the frozen rulebook. Citation admission came before a reliability gate requiring kappa of at least 0.70 and 80 percent paired substantive coverage for class, stage and detector. Frozen analysis used shuffled-order and constant-rate references.8 overlapping studies63 records: 30 amendments + 33 findingsTwo independent, blind codersFresh Codex + fresh Claude SonnetFrozen rulebook → citation checkEvery nonmissing label needs a source quoteReliability gate before reconciliationκ ≥ 0.70 AND coverage ≥ 80% on all fieldsClass · detection stage · who caught itFrozen analysis, run onceShuffled-order + constant-rate references
Reliable coding had to come before testing learning. Both coders received the same evidence and labelling rules, with hypotheses, thresholds and result commentary removed from their packets.

Coverage means the share of records where both coders gave a substantive label. Cohen’s κ (kappa) measures agreement beyond chance. The gate used κ including unknown labels, as well as coverage; a strong score on a small surviving subset could not rescue it.

Our workflow, including the stops

We followed the council loop, with disclosed stops and separately frozen changes. The saved evidence and original failed attempts stayed intact. Every change is listed below.

The workflow stopped for build decisions and citation repairs before the single analysis.Survey, protocol and review led to Robin’s approval. Build stopped for source and blinding decisions and the H3 reference rule. Freeze and push preceded coding. Codex’s first attempt was empty. Amendments D1 through D3.1 addressed budgets, citations and saved reply classification. Analysis ran once, then judgment, page and Robin’s reading close the loop.Survey → protocol → reviewBoth agents agreed; Robin approvedBuild stops → lead decisionsSource integrity, blinding and H3 reference ruleFreeze → push before codingRobin reconfirmed the approved protocolCollect independent labelsCodex first attempt empty; next attempt completedD1 / D1.1: budget reservation and output ceilingAdmit citations → sealD2 / D2.1: uncited labels become unknownD3 / D3.1: recover the saved successful replyAnalyse once → judge → pageReliability failed; blind predictions unknownRobin reads → next question ↺Any new coding needs a new exploratory version
The workflow stopped for build decisions and citation repairs before the single analysis. Outlined steps mark departures; Robin approved the protocol before the freeze and reconfirmed it before coding.

Results · Descriptions and blind predictions kept separate

We could count the records.
We could not confirm learning.

H1(a): were there fewer later corrections?

Known before registration. Later-starting studies tended to have fewer records, but the studies overlapped and had unequal workloads. “Later” means the first survey claim, not completion order; tied starts share a rank.

Later studies tended to have fewer records, a pattern known before registration.Counts in first-claim order are 6, 10, 12, 9, 11, 3, 5 and 7. All lie inside the constant-rate 95 percent prediction band of 3 to 14. Both constant-rate and shuffled-order references expect 7.875 records per study. The observed rank correlation is minus 0.49 versus zero for shuffled order.Records per study · first-claim orderWork or Proof?6Signal or Silence10The Receipt Gap12Bridge or Bus?9Three Rules, One Lunch11The Useful Core3The Portable Self5One Door, Many Hands70714Band: constant-rate 95% range, 3–14Dashed: both reference means, 7.875Observed ρ −0.49 · shuffled-order ρ 0
Later studies tended to have fewer records, a pattern known before registration. All counts fall inside the constant-rate prediction band. Neither baseline adjusts for unequal work or logging.
All selected records, ordered by first survey claim on 11 November 2026 (HKT)
StudyFirst claimRecordsBoth reference meansConstant-rate 95% band
Work or Proof?11:386 of 637.875 per study3–14 per study
Signal or Silence12:1210 of 637.875 per study3–14 per study
The Receipt Gap16:4012 of 637.875 per study3–14 per study
Bridge or Bus?16:549 of 637.875 per study3–14 per study
Three Rules, One Lunch16:5411 of 637.875 per study3–14 per study
The Useful Core17:583 of 637.875 per study3–14 per study
The Portable Self17:585 of 637.875 per study3–14 per study
One Door, Many Hands17:587 of 637.875 per study3–14 per study

Across 8 studies: Spearman’s ρ = −0.49; exact one-sided shuffled-order p = 0.108 (4,356 of 40,320 assignments). Constant-rate check p = 0.107 (10,656 extreme draws of 99,999; Monte Carlo corrected). These are descriptive reference checks, not a blind prediction verdict.

Why are the blind predictions unknown?

H1(b), H2 and H3: UNKNOWN. Before resolving disagreements, class missed the κ threshold, stage had no paired substantive coverage, and detector missed both thresholds. The frozen rule stops confirmatory evaluation here.

All three coding fields failed the reliability gate.Class coverage is 53 of 63, 84 percent, with kappa 0.675. Stage coverage is 0 of 63, with kappa zero. Detector coverage is 30 of 63, 48 percent, with kappa 0.593. Required coverage is 80 percent and kappa 0.70. Diamonds show secondary kappa on paired substantive records only: 0.850 for class, unavailable for stage and 1.000 for detector.Gate failed · required on every fieldFailure class · UNKNOWNCoverageκ (all labels)Detection stage · UNKNOWNCoverageκ (all labels)Who caught it · UNKNOWNCoverageκ (all labels)01 / 100%Solid marks: κ ≥ 0.70 / coverage ≥ 80%● Gate measure · ◇ paired-only κ (secondary)H1(b) UNKNOWN · H2 UNKNOWN · H3 UNKNOWN
All three coding fields failed the reliability gate. High paired-only agreement excluded missing labels; it could not override the original coverage and κ requirements.
Pre-reconciliation agreement; gate requires κ ≥ 0.70 and coverage ≥ 80% for each field
FieldPaired coverageκ including unknownAll-label matchesPaired-only κ / matches (secondary)Gate
Failure class53 / 63 (84%)0.67549 / 630.850 · 48 / 53Failed
Detection stage0 / 63 (0%)0.0001 / 63Unavailable · 0 / 0Failed
Who caught it30 / 63 (48%)0.59347 / 631.000 · 30 / 30Failed

Both coders faced the same citation rule. Claude’s many uncited labels became unknown. Rule-presence coverage was also 0 of 63, blocking recurrence analysis independently.

Citation admission before sealing; citation totals include field and rule citations
CoderAdmitted / input citationsNarrowedDroppedFields downgraded
Codex1,146 / 1,15419 / 1,1548 / 1,1548
Claude Sonnet142 / 1420 / 1420 / 142372

A downgrade removes a field’s label, not a whole record. Field downgrades are counts, not rates. The separate field-tagged citation counts in the limits below exclude rule citations.

H1(b): did plumbing become a smaller share?

Confirmatory outcome: UNKNOWN. The reconciled labels below are descriptive only. They count a class as known only when the evidence-only reconciliation resolved it; they cannot repair the original gate failure.

Plumbing’s share grew descriptively rather than shrinking.Among reconciled classified records, plumbing shares in first-claim order are 0, 50, 82, 83, 80, 67, 100 and 86 percent. The pooled reference is 75 percent of 48 classified records. Thin ranges show the possible full-record shares if all missing classes were inside or outside plumbing. This is not a confirmatory test.Descriptive only · not a confirmatory testWork or Proof? · 3 known of 60%Signal or Silence · 4 known of 1050%The Receipt Gap · 11 known of 1282%Bridge or Bus? · 6 known of 983%Three Rules, One Lunch · 10 known of 1180%The Useful Core · 3 known of 367%The Portable Self · 4 known of 5100%One Door, Many Hands · 7 known of 786%0%75%100%Dashed: pooled 75% · 48 classified recordsThin line: possible share among all recordsShuffled-order test: unavailable (gate failed)
Plumbing’s share grew descriptively rather than shrinking. It made up 75% of the 48 classified records; missing classes leave wide uncertainty in some studies. This is not a confirmatory test of H1(b).
Reconciled primary class; plumbing = harness or interface; pooled reference 75% of 48 known
StudyKnown of all recordsPlumbing / knownSharePossible share of all records
Work or Proof?3 known of 60 / 30%0%–50%
Signal or Silence4 known of 102 / 450%20%–80%
The Receipt Gap11 known of 129 / 1182%75%–83%
Bridge or Bus?6 known of 95 / 683%56%–89%
Three Rules, One Lunch10 known of 118 / 1080%73%–82%
The Useful Core3 known of 32 / 367%67%–67%
The Portable Self4 known of 54 / 4100%80%–100%
One Door, Many Hands7 known of 76 / 786%86%–86%

The possible-share range uses all records as the denominator, assigning every missing class outside or inside plumbing. It is not a confidence interval. The pooled reference is a descriptive comparison; the registered shuffled-order and constant-rate analyses cannot supply a confirmatory plumbing result after the failed gate.

H2: did failures recur less after their rules?

We cannot tell whether rules reduced recurrence.Unknown: reliability failed before hypothesis evaluation. Observed effect and p-value are unavailable; reference comparisons were not evaluated.H2 · UNKNOWNReliability gate failed → test stoppedObserved effect: unavailableTime shuffle / constant rate: unavailable
We cannot tell whether rules reduced recurrence. The primary coding gate failed; rule-presence coverage was also absent.
H2: unknown is an evidence limit, not a failed prediction
Required evidenceEffect / pEligible denominatorPlanned references
Reliable class, incident, rule and time labelsUnavailable / unavailableUnknown; no eligible set evaluatedShuffled incident times; constant rates scaled to active-study hours; unavailable after gate failure

H3: did Robin catch types the agents missed?

We cannot confirm that human reading found new failure types.Unknown: reliability failed before hypothesis evaluation. Observed effect and p-value are unavailable; reference comparisons were not evaluated.H3 · UNKNOWNReliability gate failed → test stoppedObserved effect: unavailableDetector-label shuffle: unavailable
We cannot confirm that human reading found new failure types. The primary coding gate failed. The motivating page omission cannot stand in for independent cases.
H3: unknown is an evidence limit, not a failed prediction
Required evidenceEffect / pEligible denominatorPlanned references
Reliable class, read stage, detector and prior missed-check evidenceUnavailable / unavailableUnknown; no eligible set evaluatedWithin-study shuffled detector labels; unavailable after gate failure

What this does not show

  • This is one council’s short history: 63 records from 8 overlapping studies over about 2 days, not a census of every failure or evidence about councils generally.
  • The coders were fresh sessions, but from the same model families as the participants. Claude and Codex were also participant-analysts; blinding reduces rather than removes that dependence.
  • Citation discipline changed what was measurable. Claude supplied 138 field-tagged citations versus Codex’s 1,003. Applying the same uncited-label rule erased many Claude labels, including every usable stage pairing. This gate failure measures admissible coding, not just disagreement about the underlying events.
  • Records are uneven. Early logging was less systematic; reviews, preventive copying and repeated dispositions affect counts. Active-study hours measure claimed work opportunity, not CPU time or equal effort.
  • Descriptions cannot rescue confirmation. Reconciled plumbing shares and the pre-known count pattern do not establish that rules caused learning. Actual H2/H3 effects, p-values and eligible denominators remain unavailable.
  • What could change the conclusion: a new, explicitly exploratory coding version with adequate citations and reliable labels, followed by a longer prospective history with clearer incident times and comparable workloads. The original unknown outcomes would remain.

Integrity and changes

Robin approved protocol v0.3 on 14 November 2026 and reconfirmed it on 15 November, after both agents had agreed. The frozen record was pushed before coding: GitHub’s server activity records . Separately frozen amendments were pushed before their affected steps; the frozen analysis ran once.

Spend: USD 1.89 of the USD 5 cap. The settled call cost was USD 1.88766; Codex coding and page building used the subscription.

  • Frame revision, before freeze: review findings joined amendments so an absent early deviation log would not mean zero failures; the widening was chosen knowing Work or Proof? had no deviation entries.
  • Build decision, source integrity: exact private evidence bytes were retained; a later public derivative must redact machine details.
  • Build decision, blinding: a documented derivative removed hypotheses and thresholds from the coder rulebook without changing labelling definitions.
  • Build stop, H3 reference: unknown novelty created by shuffled detector assignments counts as novel in the reference only, a conservative choice that cannot favour H3; observed unknowns still stop the test.
  • D1: raised the Claude call reservation from USD 0.50 to USD 2.50 to fit the evidence packet and long reply; the USD 5 total cap stayed fixed.
  • D1.1: explicitly set the output-token ceiling to 128,000 so the single structured reply could finish.
  • Codex attempt: the first attempt omitted the schema from its prompt and returned no coding; the next attempt coded all 63 records.
  • D2: mechanically narrowed over-wide citation spans and dropped mismatched quotes, identically for both coders, instead of rejecting an entire submission.
  • D3: the harness mistook quoted “usage limit” text inside a successful reply for a provider error; reclassified the saved envelope without a new call.
  • D3.1: corrected the settled-amount guard to match the ledger’s exact recorded amount instead of its rounded display.
  • D2.1: also made never-cited labels unknown, enforcing the existing citation requirement for both coders before analysis; thresholds and hypotheses stayed fixed.

Disclosure: while diagnosing D3, the lead glimpsed the first fields of one record in a diagnostic prefix. Before deciding D2.1, the lead counted cited field names and uncited fields per coder, including their unequal citation coverage, without inspecting further label values or any agreement or hypothesis result.

The frozen record is held in a private repository and will be released in redacted form. There is no public repository link yet. We have not published raw evidence containing machine paths or identifiers.

Notes, definitions and sources

Plumbing
Harness failures (software that runs, saves or accounts for work) or interface failures (the reply contract). Coordination and scientific errors remain separate classes.
Shuffled-order reference
Reorder whole study bundles while keeping each bundle intact. It expects no association with time: ρ = 0 and 7.875 records at each position.
Constant-rate reference
A Poisson count model expecting 7.875 records per study, with central 95% prediction limits of 3–14. It is a simple reference, not an equal-workload claim.
Unknown
Insufficient admissible evidence to evaluate a registered prediction. Its public effect and p-value are null, not zero. No sensitivities were produced by this analysis run.

Exact pass rules: all blind tests first required κ ≥ 0.70 and ≥ 80% paired substantive coverage on class, stage and detector. H1(b) additionally required ≥ 80% class coverage in every nonempty study, at least 3 nonempty studies and a negative share correlation. H2 required at least 2 evaluable classes, positive exposure before and after adoption, and a positive mean reduction in incident rates. H3 required at least 2 eligible human-reading cases from distinct studies, known class and chronology, positive missed-check evidence, and a novelty fraction of 1. Each also required a one-sided p ≤ 0.05 after Holm adjustment across the 3 blind predictions. H1(a) was excluded from this family.

Worked-example citation: Work or Proof? page-omission review, 11 November 2026, lines 1–8 (saved packet P031): the omission, joint discovery and standing page rule. The example uses the admitted Codex labels; the original review and both coder records remain in the frozen private archive.

Evidence: protocol v0.3 and codebook v0.2; survey and feasibility audit; Robin’s approval records; lead build decisions and amendments; server push activity; citation admission reports; settled ledger; and the single frozen analysis’s results. Numerical displays are transcribed from that analysis and the listed evidence; rounded percentages retain their count denominators.