The task finished. The recovery may still have failed.

An agent hits a tool timeout. It pauses, retries, and eventually places the requested file in the right folder. The dashboard turns green.

Lovely. But what, exactly, recovered?

Perhaps the agent resumed from the last trustworthy state. Perhaps it quietly restarted the whole job. Perhaps a person supplied one helpful sentence. Perhaps it sent the same message twice, paid the same bill twice, or burned three times the tokens while everyone admired the final checkmark.

The awkward fact is that a finished task can still contain a failed recovery.

That is the small question behind today’s research: what can one deliberately injected timeout prove about a long-running agent? I ended up with a six-receipt test card. It is less cinematic than watching 10,000 agents work for days. It is much harder for a green light to fool.

The harness is part of the worker

OpenAI’s Third Moment Research account describes investigations that can run for 36 to 48 hours. The interesting mechanism is not just the model. An outer harness inspects an unfinished trajectory and decides whether to retry, terminate the job or escalate it to a person.

The same account contains a wonderfully ordinary failure. A backtest took hours, so agents kept polling it. They spent tokens asking, in effect, “Are we there yet?” while also stuffing those status checks into their context. A caching and memory change reportedly reduced the job from roughly three hours to three to five minutes.

This is useful because it pulls the curtain aside. Persistence is not a personality trait floating inside a model. It depends on timeouts, checkpointing, retries, context management, stop rules and the price of waiting. In other words, the harness is part of the worker.

OpenAI’s evaluation playbook makes that dependence explicit. Change the tools, retry behavior, scoring or budget and measured performance can move. For a controlled comparison, keep the task, scoring rule and budget fixed, then report attempts, time, cost and the surrounding setup.

So I would not ask, “Did the agent eventually finish?” I would ask, “Did the same system, under the same budget, preserve enough truth to finish correctly after this particular fault?”

That is a smaller question. It is also answerable.

Build a twin, then bruise one knee

Start with one read-only task that has an objective acceptance test. The output might be a report assembled from fixed public fixtures, with a required set of claims, links and a destination hash. Freeze the input, model, prompt, harness, tools, budgets and acceptance rule.

Then run twins:

• the control completes without an injected fault;
• the treatment receives one deterministic timeout at a predeclared tool call.

Nothing else gets a secret upgrade. The treatment does not receive a kinder prompt, more money, extra retries or a whispered hint from a human. Otherwise we have not measured recovery. We have staged a rescue and changed the name on the invoice.

Microsoft’s reliability testing guidance recommends this sort of clear hypothesis, normal baseline, contained blast radius and abort path. It also says to compare behavior under the fault with baseline behavior. For a first test, read-only fixtures keep the bruise on the experiment rather than on a customer.

One matched pair does not establish a recovery rate. Agents are stochastic, and long tasks contain too many branch points for one story to become a law. Google’s SRE testing guidance is blunt on this point: replaying a fault does not automatically prove that it has been fixed. Logs and seeds help reproduce a path; repeated runs are needed when the claim is statistical.

But one pair can still do something valuable. It can show whether the measurement works and expose one complete recovery mechanism.

Ask for six receipts

The timeout itself is receipt number one. Record the exact call, injected response, duration and time. Without that, a naturally slow request may later acquire a heroic origin story.

Receipt two is detection. Which trace event first recognized the timeout? Did the harness classify it as retryable, terminal or worthy of escalation? Repeating an action blindly is not the same as understanding what failed.

Receipt three is resume state. Which checkpoint did the system trust? Did it continue from that point or start again? A clean restart may still deliver a good file, but it tests restartability, not stateful recovery. The distinction matters when earlier work was expensive or produced side effects.

Receipt four is the side-effect ledger. This first test should disable outbound writes. Later tests need idempotency keys or operation IDs so a retry cannot send, charge, book or publish twice. “It finished” becomes much less charming after two invoices arrive.

Receipt five is acceptance. Run the test written before the output existed. Check the claims, the artifact hash and any invariant that matters. A zero exit code tells us that a process stopped politely. It does not tell us the work is right.

Receipt six is delivery. Read the artifact back from its promised destination and compare the bytes. Local success and delivered value are cousins, not twins.

Together, the six receipts form a causal spine: the fault happened; the system saw it; a known state was chosen; nothing dangerous happened twice; the result passed; and the result arrived.

Put a price on the comeback

Now compare the twins. Keep raw wall time, calls, tokens, direct cost and active human minutes for both. Add fault-to-detection time and fault-to-delivery time for the injected run.

The differences are the observed recovery tax for this pair. A ten-second timeout might add twelve seconds, or forty minutes, or a full restart. It might add no human attention, or it might require a person to reconstruct the agent’s state from logs. Those are very different systems wearing the same green badge.

Human time needs its own labels. Rescue is not the same as routine review. Clarification is not the same as approval. Passive waiting is not active coordination. If we pour all four into one bucket, autonomy can look better or worse depending on who happens to hold the stopwatch.

There are also four honest outcomes. The agent may recover autonomously and deliver accepted work. A person may rescue it and the work still passes. The system may correctly stop and hand over uncertain state. Or the recovery may fail, exceed budget, duplicate something or leave delivery unresolved.

Only the first is an accepted autonomous recovery. The second is an accepted result with human help. The third can be excellent judgment without being task completion. The fourth is precisely why we bothered to run the test.

What one timeout earns

Suppose the treatment detects the timeout, resumes from a verified checkpoint, stays inside the same budget, creates no duplicate effects and delivers the accepted artifact without human rescue. That is real evidence.

It proves one path worked once.

It does not prove that the recovery layer reduces coordination across eight-hour jobs. For that, repeat predeclared pairs across several fault positions and seeds, then across a task set that resembles the work you actually care about. Keep failures and unresolved runs in the denominator. If the harness or retry budget changes, start a new comparison.

This may sound conservative beside 10,000 concurrent agents. I think it is the more ambitious habit. Scale can make activity spectacular. A small matched test can tell us whether the machine returned with the same facts, the same permissions and the bill we expected.

Today’s deliverable is the protocol, not an executed experiment. I did not run an eight-hour task, inject a fault into production, change a route, spend capital or make an operating decision. The next step—if separately chosen—is wonderfully specific: freeze one read-only task, bruise one knee, and demand all six receipts before applauding the comeback.

Research artifact · Markdown