When Robots Lose the Reply

Imagine a warehouse robot receives an instruction to place a box on a conveyor. It places the box. Its reply disappears on the way back. The supervisor sees silence and sends the instruction again.

That small gap can turn one successful job into two attempted jobs. The conclusion is simple: dependable autonomy needs a way to discover what happened before deciding what to repeat. Better reasoning helps choose an action. A recovery contract keeps uncertainty about that action from becoming a second physical effect.

This is a hypothetical example, not a reported warehouse incident. Its appeal is that almost nothing dramatic has to fail. The motor works, the plan makes sense and the network merely loses a reply. A system can contain several competent parts and still produce an incompetent outcome.

The interesting line in the release notes

DroneDeploy’s September 24, 2026 notes for unstable version 1.7.52 include retries for boot-time targets and a separation between triggering map deployment and waiting for completion. These are software changes, not published field-reliability results. DroneDeploy release notes

I like the question they raise. What does a caller actually know after it sends something? Sending, accepting, executing and confirming are different events. An interface that compresses them into one cheerful green tick asks the operator to supply the missing reasoning during an outage. The green tick has outsourced its hardest job.

The broader development direction makes that question timely. NVIDIA’s September 22 Isaac ROS 5.0 announcement adds agent-assisted robotics development. That is evidence of better tooling availability; it does not tell us how often a deployed machine needs rescue. NVIDIA announcement

My inference is that easier robot programming increases the value of a precise boundary between a proposed command and a completed job. More people can author workflows. More workflows can reach hardware. The evidence used to call a workflow successful therefore deserves at least as much design attention as the instruction that starts it.

Silence has several meanings

Return to the box. The supervisor’s empty inbox is consistent with several possible worlds. Perhaps the instruction never reached the robot. Perhaps it arrived and was rejected. Perhaps the robot is still moving. Perhaps the box is already on the conveyor and only the acknowledgment was lost.

Those worlds require different responses. A fresh send may help in the first. It adds confusion in the third. It can create duplication in the fourth. Treating all four as a generic failure throws away precisely the distinction needed to recover well.

A timeout therefore answers a narrow question: the caller did not receive the expected evidence within its waiting period. It cannot, on its own, tell us where the box is. The temptation to translate silence into failure comes from the interface, not from the physical world.

There is a familiar human version. You send a colleague a request and receive no reply. Sending the same message again may be reasonable. Sending a second colleague to perform the same irreversible task can be a different decision altogether. Machines need that distinction made explicit, because politeness will not save them from an ambiguous queue.

Give the job an identity that survives the conversation

A proposed design begins by assigning one durable identity to one intended job. A retry carries that same identity. A request for a genuinely new job receives a new identity. The receiver then has a basis for distinguishing a repeated conversation from a repeated obligation.

AWS describes caller-supplied request identifiers as a way to make retries idempotent: repeated requests can preserve one intended effect. This is established distributed-system practice, not a claim that arbitrary physical actions become safe automatically. AWS retry design

For our imaginary conveyor, the command record could contain a job identifier, the selected box, the destination and the authorized operation. The receiver would reject a reuse of the identifier with a different payload. Otherwise the word “same” becomes a loophole: someone can accidentally reuse an old identity while asking for a new action.

The record also needs a lifetime. If a robot forgets the identifier after a restart, a late retry may look new. If an upstream scheduler creates a fresh identifier every time it reconnects, the receiver’s careful duplicate handling never gets a chance to work. Recovery is a property of the whole path.

This proposal has a cost. Durable records consume storage, ownership needs definition and abandoned jobs need a resolution policy. That overhead should be compared with the cost of ambiguity. A job whose consequences are trivial may tolerate a simpler arrangement. A job that moves an expensive object earns a more careful one.

A message result and a physical result are different evidence

ROS 2’s action design separates goals, feedback and results, with cancellation support for long-running work. That vocabulary makes the distinction between starting and finishing visible in the interface. ROS 2 action design

Our proposed acceptance contract still has to decide what counts as a result. “The controller returned success” and “the correct box reached the correct conveyor position” are related observations. They need not be identical. A sensor, an inventory record or a downstream station may provide additional evidence, each with its own failure modes.

The awkward case is a restart after movement and before durable confirmation. A database entry can be missing while the box has already moved. Recording before movement creates the opposite problem: an entry can exist for a motion that never happened. A software transaction cannot simply roll the physical world backward.

The practical response is to design reconciliation around observable state. After uncertainty, establish the robot’s permitted condition, inspect the object and determine which transitions remain valid. A command identity prevents some duplicates. Physical observation helps resolve the effects that a command log cannot prove.

No single sensor should be promoted into magic. An old image may describe an earlier position. An inventory update may lag. Two observations can disagree. The recovery contract needs a defined way to hold the job unresolved and a named party able to settle the discrepancy. Admitting that limit is useful engineering.

Restarting the process does not finish the job

ROS 2’s managed-node design distinguishes configured, inactive, active and other lifecycle conditions. A component’s readiness can therefore be managed separately from the work it performs. ROS 2 managed nodes

In our example, a restarted process is only one step in recovery. It may still need the right map, a valid position estimate, fresh observations and confirmation that another controller has not taken ownership. Restart success is evidence about the process. The operator needs evidence about the job and its surroundings.

That difference changes the question asked of a supplier. “Does it reconnect?” is a useful opening. “What is it allowed to do immediately after reconnecting?” goes further. “Which record and observation justify that permission?” reveals whether the answer is an implemented rule or a hopeful description.

The same reasoning applies to cancellation. A request to cancel cannot reach backward in time and undo a completed movement. If cancellation and completion cross on the network, the system needs to report the resulting state honestly. A cancellation request is an intention; a confirmed safe condition is an outcome.

This is why I would avoid a single uptime number as the entire commercial story. A machine can remain connected while waiting for a person to reconcile its work. It can be briefly disconnected and recover cleanly. Those two conditions have different operational costs even when a dashboard paints both with similar colors.

Draw the recovery boundary before measuring it

The diagram proposes three distinct destinations after a lost reply. Confirm the original job when sufficient evidence establishes completion. Resume only the unfinished, authorized part when the present state supports it. Hold when the observations cannot distinguish safe recovery from duplicate work.

“Hold” needs a useful interface. An operator should receive the original intention, the last trusted observation, the conflicting or missing evidence and the decision still required. A generic error code that forces someone to reconstruct the entire story does not make uncertainty cheaper; it hides the invoice.

For an investor or buyer, that invoice can become a diligence object. How many jobs need human reconciliation? How long does it take? Does the supplier’s support team absorb the labor, or does the customer? Are incidents counted by session, by robot or by requested job? Different denominators can make the same operation look very different.

These are questions, not measured findings about DroneDeploy, NVIDIA or a customer. The public materials cited here do not provide a matched field trial answering them. The purpose of the framework is to specify what evidence would make an autonomy claim economically interpretable.

A small test with a clear stopping point

The next useful experiment is a simulated conveyor with one box, one command issuer and one receiver. No real robot is required. The test should preserve the same intended job while deliberately interrupting the exchange at different points. That isolates recovery behavior from the intelligence of the original plan.

Start with a normal run to define the expected receipt. Then drop the request before delivery, drop the reply after execution, restart the receiver between effect and confirmation, and replay an old request after reconnection. Add a case in which observations conflict. The last case should remain unresolved instead of receiving a convenient success label.

For each case, record attempted effects, observed completed effects, time to an accepted result, manual intervention and unresolved state. The desired property is easy to say: one authorized job should create at most one accepted physical effect, and every completion claim should have the agreed evidence. Availability and timeliness still matter; avoiding duplicates by never doing any work would be a poor service.

A useful comparison would give a simple retry policy and a reconciliation policy the same interruptions and observations. Count both duplicate effects and jobs left unfinished. Report the tradeoff rather than selecting only the metric on which the more elaborate design wins. This is a proposed test specification; I have not run that comparison or tested a physical robot for this essay.

The specification is deliberately small enough to fail clearly. If ownership is ambiguous, if a restart loses the record, or if a late message changes the result, the trace should expose it. Adding more agents before resolving that failure would multiply participants without improving our understanding.

The buyer should ask for the interrupted demonstration

A polished demonstration shows the happy route through a system. A useful procurement conversation also asks the supplier to explain a bounded interruption using its own documented recovery procedure. That request should respect the machine’s approved operating conditions; improvising faults on live equipment is not the experiment proposed here.

The important deliverable is a trace that a second person can follow. What was requested? Which actor accepted responsibility? What actually changed? Which observation supported completion? If the system stopped, what made the next step permissible? A replayable answer is more valuable than an assurance that the model is usually clever.

This framing also gives suppliers a constructive way to compete. One vendor may recover more jobs automatically. Another may escalate less often because its evidence is clearer. A third may simplify installation by reducing the number of systems that must agree. Each improvement can be described in terms of work completed and human effort consumed.

The monitoring card I would keep has four fields: the intended job, evidence of its effect, recovery cost and the condition that requires a hold. Update it when a supplier publishes a reproducible recovery trace or a customer supplies comparable operating evidence. Until then, keep the performance claim open rather than filling the gap with a demo.

A robot that asks for help at the right moment may be doing something valuable. It has located the edge of what its records can prove. The next stage of autonomy is partly about moving that edge outward, one well-observed recovery at a time. The box should reach the conveyor once, and everyone should be able to explain why they believe it did.

Categories and keywords

Categories: Physical AI; Agent Infrastructure; Systems Engineering.

Keywords: robot recovery, idempotency, command identity, ROS 2, reconciliation, cost per completed job.

Hashtags: #PhysicalAI #Robotics #AgenticAI #SystemsEngineering