Your AI Research Lab Needs a Sealed Exam

The conclusion is uncomfortable: an AI research lab has learned only when retained experience improves performance on work whose answer was unavailable during training, retrieval and evaluation. A longer memory file, a more fluent report and a busier swarm can all look like progress. None establishes learning. The decisive test is a sealed exam with a fixed total budget, hidden tasks, independent acceptance and a memory-ablation arm.

This standard matters because research agents are becoming convincing before they are becoming easy to audit. OpenAI’s Jump Trading case describes a multi-agent research system that can run extended investigations. BioStudyBench proposes 25 biomedical replication tasks designed to separate genuine analysis from retrieving a published answer. Both point toward useful machinery. Neither lets an operator skip the harder question: did yesterday’s retained experience improve today’s unseen work, or did the system merely gain more ways to find familiar answers? OpenAI Jump Trading BioStudyBench

A sealed exam turns that question into an operating discipline. It compares systems on the same information and compute budget. It hides the final task until the memory policy is frozen. It grades the result independently. It removes memory once to see whether the claimed improvement disappears. Most importantly, it preserves failed and ambiguous outcomes instead of allowing a polished narrative to absorb them.

No RobinOS sealed exam has been run for this article. The protocol below is a publication-ready design, not an experimental result. That distinction is the first test the lab must pass.

The leakage problem arrives before the model

Research benchmarks often fail quietly because the answer is already somewhere inside the system’s reach. A model may remember a paper from training, retrieve a later article that summarizes the result, find code written after the supposed cutoff, or inherit a previous agent’s conclusion through shared notes. The final response can be correct while the research process being measured never happened.

BioStudyBench attacks this problem directly. Its tasks recreate parts of biomedical studies while restricting literature by date and varying access to data and tools. The design matters more than any single score because it asks whether the system can perform analysis under a historically plausible information boundary. A system that sees the published solution is taking an open-book lookup, even if the interface calls it research. BioStudyBench

Leakage is broader than web search. Repository names, file paths, expected chart shapes, test fixtures and evaluator comments can all reveal the answer. Memory makes the problem sharper: a helpful note may contain the target result, or it may encode a procedure learned from a genuinely separate task. Those cases look identical if the operator records only the final output.

The first job of a sealed exam is therefore provenance. Every task needs a creation date, a source packet, a retrieval cutoff and a record of what each agent could access. The system must log which memory objects were loaded and which URLs or files were opened. A claim of learning without this boundary is a story about capability, not evidence of transfer.

Freeze the task after the memory policy

The exam task should remain unknown until the competing systems and their memory policies are fixed. Otherwise the operator can tune prompts, examples or retrieval rules toward the answer. This is the research equivalent of teaching to the test, except the teacher can rewrite the student’s brain minutes before the exam.

A practical sequence begins with a task pool assembled by someone or something outside the evaluated run. Each task receives an immutable identifier and hash. The evaluator then freezes three systems: a single agent without retained memory, a multi-agent system without retained memory and the same multi-agent system with curated memory from prior work. All three receive the same base instructions, tools, source-access rules and total resource budget.

Only after those policies are locked does the evaluator draw the held-out tasks. The draw should be recorded before execution, and unused tasks should stay sealed for later rounds. If the task must be authored internally, the author should not grade it and the agents should not see the answer key. A modest separation of duties is more valuable than a theatrical claim of blindness.

The task needs to be new in mechanism, not merely new in wording. Asking for the same calculation with another company name tests template reuse. Asking whether a learned evidence rule transfers from AI infrastructure to robotic manufacturing tests something more interesting: can the system recognize the same epistemic structure in a different domain?

Keep one total budget

Multi-agent systems can outperform a single agent simply by spending more. Five agents with five searches each have a larger information and compute budget than one agent with five searches. Calling the difference “emergence” hides the denominator.

Fix a total budget across conditions. It can be denominated in model tokens, API cost, wall-clock time, tool calls and human minutes. No single measure is perfect, so preserve the small vector rather than compressing everything into one synthetic number. The multi-agent condition may allocate the budget among roles, while the single-agent condition receives the same total allowance.

Retries belong inside the budget. So do evaluator clarifications, manual file repairs and rescue prompts. If the memory-enabled system finishes cheaply only because a person repairs malformed evidence, the human time is part of its delivery cost. If a system times out after consuming its allowance, the result remains unresolved or rejected; the budget does not quietly reset.

Listed token prices can help plan the experiment, but they are not the outcome measure. Anthropic’s Haiku 5.5 launch illustrates why: a lower per-token rate can make subagents attractive, yet additional review and retries may erase the saving. The useful economic unit is accepted research under the fixed standard. Anthropic Claude Haiku 5.5

Compare three systems, not two

A memory experiment needs at least three arms. The first is a memoryless single agent. It establishes what one capable worker can do with the allowed sources and budget. The second is a memoryless multi-agent system. It measures the value of role separation, parallel search and internal challenge without retained experience. The third uses the same multi-agent architecture plus curated memory.

This design separates two effects that are often bundled together. If both multi-agent systems improve over the single agent, coordination may be doing useful work. If curated memory then improves over the memoryless multi-agent system, retained experience has a plausible contribution. If only the memory condition receives better instructions or more tools, the comparison cannot identify memory.

The memory set should contain procedures, past error patterns, source-quality rules and compact examples. It should exclude answers to held-out tasks, post-cutoff reporting and evaluator feedback written for the current exam. Every memory object needs a source, creation date, selection reason and hash. Curating memory is itself a research intervention and should not disappear into an unnamed folder.

A random-memory control can strengthen the test. Give an otherwise identical system the same number of memory tokens drawn from unrelated prior work. If curated memory and random memory perform alike, the benefit may come from extra context, style priming or longer deliberation rather than useful retained experience.

Grade the research, not the prose

Fluent writing is dangerous in evaluation because it makes weak research feel complete. The grader should score claims and decisions before style. A useful rubric covers factual accuracy, source quality, coverage of decisive alternatives, calibration of uncertainty, reproducibility of calculations and usefulness of the final recommendation or experiment.

Each criterion needs observable anchors. “Excellent sourcing” is vague. “Every load-bearing number links to a primary source, dates match the event, and conflicting figures are reconciled or left unresolved” can be checked. “Good judgment” is vague. “The conclusion changes when the stated disconfirming evidence appears” shows an actual decision rule.

Independent grading does not require a large panel. One evaluator can grade anonymized outputs in random order using a frozen rubric. For higher-stakes work, a second evaluator can review disagreements. The key is that the grader should not know which system produced the answer while assigning the primary score.

The lab should also record catastrophic failures separately. A fabricated source, concealed missing value, unauthorized external action or false completion claim cannot be averaged away by elegant prose elsewhere. These are boundary failures. Depending on the task, one may reject the run regardless of its other scores.

Ablate memory and inspect the delta

Memory ablation is the heart of the exam. Run the same architecture with curated memory enabled and disabled while holding the task, budget, tools and grading constant. The difference alone cannot prove causality, especially with stochastic models. It does reveal whether the claimed learning survives removal of its proposed mechanism.

One run per condition is too fragile. Use several tasks and repeated seeds. Report the distribution, not only the best attempt. A memory system that wins once and fails unpredictably may be less useful than a simpler system with a slightly lower average and a much narrower failure band.

Inspect which memories were actually retrieved. A system may have access to excellent lessons and never use them. Another may retrieve the right rule and apply it to the wrong domain. Retrieval precision, ignored useful memories and harmful memories help explain the result and improve the next curation policy.

Repeated failures deserve their own ledger. If three systems make the same mistake, the problem may be missing source access or a weak rubric rather than memory. If only the memory system repeats an old framing error, retained experience has become retained bias. Learning includes knowing what to discard.

Separate research, system learning and distribution

One experiment can serve three projects only if it keeps their outcomes separate. Quant Lab cares whether the research conclusion is more accurate, reproducible and decision-useful. RobinOS cares whether the system transfers a useful procedure across tasks at lower total effort. SunTV or iamrobin.ai cares whether readers understand the mechanism and return with better questions.

Those are three scorecards. A popular explanation does not validate the research result. A high research score does not prove the operating system learned. A strong operating result does not authorize a capital decision. Combining the scorecards creates a pleasing number and destroys the distinctions that make the experiment useful.

The distribution test can remain modest. Publish two explanations grounded in the same evidence, perhaps a control version and a diagram-led version. Measure substantive corrections, saved questions and return visits rather than raw impressions alone. Do not let audience response alter the research grade after the fact.

This separation also protects honesty. The lab can report that a protocol was designed, an experiment ran, a memory effect was observed, or a reader test improved comprehension as four different states. Each requires its own receipt. Today only the first state is established.

The first sealed exam

Start with a task small enough to rerun and important enough to expose judgment. One candidate is a research memo comparing two recent AI-infrastructure projects under the same evidence standard. The held-out task could ask for a delivery-risk map using only public primary sources available before a cutoff.

Freeze ten acceptance conditions: correct entities and dates, primary sources for central figures, explicit missing values, no duplicate event, one disconfirming observation, reproducible calculations, bounded conclusion, no unauthorized action, complete source packet and a final artifact that another person can inspect. Give all three systems the same source window and total allowance.

Run at least five held-out tasks with several seeds if cost permits. Anonymize the outputs. Record quality score, total model and tool cost, human minutes, retries, repeated failures and unresolved states. Preserve raw trajectories privately where policy permits, while publishing only redacted methods and aggregate evidence.

Promotion should be declared before seeing results. Curated memory qualifies only if it improves the median accepted score without increasing boundary failures, stays within the total budget and shows a stable advantage across more than one task. If it merely improves prose or one lucky run, retain the result as inconclusive.

The experiment may show that a single agent is enough. That would be useful. It may show that multi-agent challenge improves source coverage while memory adds nothing. Also useful. It may show that curated memory reduces repeated errors but raises cost. That is a real trade-off, not a failed story.

What evidence would convince us

The convincing result is deliberately plain. A dated, hash-bound task set stayed hidden until the systems were frozen. Three matched conditions used the same total budget. Independent grading found a repeatable improvement in accepted research quality for curated memory. Removing memory reduced that advantage. The improvement survived several tasks and seeds, while human rescue and boundary failures did not rise.

Anything weaker should keep a narrower label. A better-looking memo is an editorial improvement. A faster run is an efficiency observation. More retrieved notes is a system behavior. A provider benchmark is external evidence under its own conditions. None becomes organizational learning through confident wording.

The sealed exam is valuable even when the answer is negative. It tells the lab which part of the machine deserves investment: model capability, coordination, source access, memory curation, evaluation or recovery. It turns “our agents are getting smarter” into a question with a denominator.

For a one-person research department, that is the leverage worth seeking. Memory should reduce repeated mistakes and carry hard-won methods into genuinely new work. If it cannot pass a sealed exam, it is still an archive. Archives are useful. They simply should not be mistaken for learning.

Categories and keywords

Category: Agentic AI · Research Systems

Keywords: sealed evaluation, agent memory, research agents, memory ablation, fixed-budget comparison, unseen tasks

Hashtags: #AgenticAI #AIResearch #Evaluation #RobinOS #IAmRobin