September 8, 2026

A Warehouse Robot Stops and a Worker Walks Into It. Can Anyone Find Out Why?

Start with the scene, because everything else follows from it. A mobile robot is moving down an aisle. It stops without warning. A worker walking behind it does not, and there is a collision. Nobody is seriously hurt, but somebody now has to write a report explaining what happened.

The investigator has the logs. The robot recorded its sensor streams and its motor commands, and at 14:22:07 the command was stop. Between those two things sits a neural network that turned camera and LiDAR readings into that command through several million parameters. The log tells you what the robot did. It does not tell you what the robot believed, or what it would have done instead, or which reading changed its mind.

That gap is the subject of a paper I published at SoutheastCon 2026, and this post is the version without the equations.

Why this is not only an academic complaint

Service robot sales grew 30 percent in 2023 according to the International Federation of Robotics, and these machines are not in cages any more. They share aisles with warehouse staff, roads with pedestrians, and operating theatres with surgeons.

The auditability gap turns into three concrete problems as soon as deployment is real.

Incident investigation becomes speculative. Without a reconstructable record, the report is a plausible story rather than a finding, and the fix that follows is aimed at a guess.

Safety certification becomes difficult. Standards such as ISO 13482 require documented risk assessment, and a system that cannot say why it acted cannot supply the document.

Regulatory compliance fails outright. The EU AI Act requires high-risk AI systems to keep logs that enable tracing of the system's functioning. Most deployed robotic systems today cannot satisfy that sentence, and it is not a sentence you can satisfy retroactively.

Why explaining the model does not solve it

The obvious answer is to reach for explainable AI, and the obvious answer is aimed at the wrong object.

LIME approximates a complex model with a simple one near a single prediction. SHAP attributes a prediction to input features using game theory. Both are useful and both explain one output. A robot makes hundreds of interdependent decisions per second, and explaining each of them in isolation loses the chain that connects them, which is the only thing the investigator wanted.

Attention weights look like a way out, since the model appears to be telling you what it looked at. Jain and Wallace showed that attention weights often fail to reflect the actual causal relationships. For a safety-critical system, an explanation that is confidently wrong is worse than no explanation, because it ends the investigation early.

Runtime verification is closer. Systems like ModelPlex check that a cyber-physical system stays inside verified bounds, which is genuinely valuable, but a violation flag says that something went out of bounds and not why the controller chose to go there.

So the distinction I would put at the centre of all of this: post-hoc methods explain a model output, and an investigation needs a decision sequence. Those are different objects, and no amount of tuning turns the first into the second.

Building the record instead of recovering it

TRACE stands for Transparent Reasoning Architecture for Credible Execution. The idea is short enough to state in one line: stop trying to extract an explanation from a closed system afterwards, and have the decision pipeline write down its reasons while it runs.

It is a wrapper, not a replacement. The perception module can stay a convolutional network and the predictor can stay a transformer. What the architecture requires is that each stage hand over its intermediate product in a documented form, through four layers. The clearest way to see what that buys you is to follow one decision through all four.

Take the moment before the collision. The robot is moving at 1.2 m/s and something appears at the edge of the aisle.

Layer one records what was seen and on what basis. A conventional pipeline emits a detection: person, 3.2 metres ahead, confidence 0.87. That single line is all an investigator gets, and it cannot be checked. This layer emits the same detection together with its evidence: the LiDAR return that gave the range, the camera region that produced the classification, and how much each contributed. Later, when the question is whether the robot saw a worker or a reflection off a pallet wrapper, that difference is the whole answer.

Layer two records what the robot concluded from it. Detection is not belief. From a person standing near the aisle mouth, the robot infers something it cannot observe: that the person is about to step into the lane. This layer keeps those inferences in a graph, and every inferred state is attached to the observations that produced it. So the belief is not a bare assertion in the log. It is a claim with parents, and an investigator can walk from "the robot expected someone to step out" back to the two frames that suggested it.

Layer three records what else it could have done. This is the layer I would defend hardest, and it is the one nobody builds. The planner considered three options: continue at speed, slow to 0.4 m/s, stop. It chose stop. A normal log keeps the choice and throws away the other two. This layer keeps all three, the reason stop won, and the conditions under which it would not have: if the predicted intent had been below a threshold, the robot would have slowed instead of stopping. That last item is what turns a log into an investigation, because it answers the question the report actually has to answer, which is not "what did it do" but "what would have made it do something else".

Layer four records whether the robot did what it decided. The final layer compares intent with outcome, stamps the whole cycle into one record, and flags the gap when the two disagree. A commanded stop that took 400 ms longer than modelled is a different incident from a stop that should never have been commanded, and only this layer can tell them apart.

Put together, the four produce a record with a shape. The investigator does not sift through raw sensor streams looking for a clue. They open the cycle at 14:22:07 and read what the robot saw, what it concluded, what it weighed, why it chose, and what actually happened.

Three numbers an outside engineer can compute

A framework that grades itself is worth little, so the three metrics are arithmetic over the logged record. Someone who has never met me can compute all three on their own system. Each answers a plain question.

Can every factor be traced back to a reading? Evidence traceability is the share of the factors behind a decision that lead back to a specific sensor observation through a documented chain. There is also a partial version, which matters more than it sounds. When a chain is broken it reports how far down the documentation goes, so a depth of 2 out of 3 tells the investigator that the immediate and secondary causes are on record and the original observation is missing. That is a usable finding. A bare "not traceable" is not.

Does the record have holes? Temporal continuity is one minus the share of the analysis window with no decision recorded. Gaps are what appear first when the system is under load, and a gap at the wrong second makes the rest of the trail worthless.

Could an independent engineer reconstruct the decision? Decision reconstructability is the strict one. A cycle counts only when all three hold at once: the action is recoverable, the alternatives are recoverable, and the reason for preferring the chosen action is recoverable. Two out of three scores zero, because two out of three does not let anybody rebuild what happened.

What it measured

The evaluation is a warehouse simulation, 50 by 30 metres, decisions at 10 Hz, five scenarios of 100 cycles each, running from a clear aisle up to a forklift on a collision course. The simulation does not run clean, which is the point: sensor dropouts, corrupted evidence links, truncated rationales and timeouts are injected, because a record that only holds when nothing goes wrong is not an audit trail.

The overhead is the figure that decides whether the architecture is adoptable, and it is the one I would put first. At 10 Hz the budget for a decision cycle is 100 ms, and the audit trail costs roughly 1 percent of it. Nobody has to weigh auditability against performance at that price.

One note on the comparison table in the paper. The post-hoc baselines are placed there on the basis of what each architecture can do in principle rather than on runs I performed. The paper says so, and I repeat it here, because the claim I am making is architectural: an attribution method and an audit trail are built to answer different questions, and that holds regardless of how either one is tuned.

The part that decides whether a fleet can run it

Work in this area usually stops at the metrics. What actually determines deployment is where all of this goes.

A full audit trail runs 2 to 5 KB per decision cycle. At 10 Hz that is 70 to 175 MB per hour, or 1.7 to 4.2 GB per robot per day. Take a fleet of a hundred thousand robots and the six-month retention that Article 19 of the EU AI Act asks for, and the raw requirement passes 30 petabytes.

A number like that ends the conversation unless the storage is tiered, so the design tiers it. The past 24 hours stay in full detail, which is the window in which incidents are actually investigated. Beyond that, flagged decisions stay complete and routine ones are kept as summaries. Older material is compressed and archived for the regulator.

Compression works unusually well here because causal graphs repeat. The same structures recur cycle after cycle, so dictionary encoding is projected to give a 3 to 5 times reduction, and encoding only the change between consecutive cycles a further 60 to 80 percent during routine operation, where the belief state barely moves. Those two figures are projections from preliminary analysis. Measuring them is on the list.

Limits, and what comes next

The evaluation ran in simulation, with modelled sensor noise and modelled failures. That is the right stage for establishing that the architecture holds its trail under fault conditions, and it is not sufficient for a deployment claim. Validation on physical hardware with real sensors is the next step, and the paper states this plainly rather than leaving it to the reader.

Two other items are open. The storage projections need to be measured rather than estimated. And the architecture should be compared against the cheapest thing that could possibly work, which is logging everything and reconstructing afterwards, because a design that cannot beat the obvious alternative on the same metrics has not earned its complexity. I hold that standard for my own work: earlier this year I measured a wheel-slip detector against a control arm that simply drove slower, and the control arm won.

What I would defend today is narrower than any percentage, and I think more durable. An audit trail is a design requirement, not a reporting feature. Evidence that is not captured while the robot is deciding cannot be recovered afterwards by any tool, however good, and the investigation that needed it ends up producing a story rather than a finding. The engineering question is therefore not which explanation method to add at the end. It is what the system is obliged to write down while it runs.

The paper is arXiv:2609.02861 (free full text), published in SoutheastCon 2026, IEEE, pp. 1-6. The slip-detector story is in the original post and the correction that followed it.

← Back to all posts