Stopping power
In the Toyota Production System, any assembly line worker who detects a problem can pull the andon cord. In Toyota’s own description, the operator “can stop the line by pulling the stop cord themselves,” and the andon board lights up to call the person in charge to the problem.
Call this stopping power.
The silent failure
Automated and administrative governance took the cord out. When a system starts producing clustered errors, misclassifying claims, flagging transactions, withholding services people depend on, the people living with those errors have no fast or collective means to halt it. The errors get filed as noise, edge cases or user error. The machine keeps running, the efficiency numbers keep looking good, and the harm lands in what Madeleine Clare Elish calls the moral crumple zone (2019), on the people nearest the system.
Australia’s Robodebt scheme, which raised welfare debts by averaging annual income across fortnights, ran this way for years. In 2017 a member of the Administrative Appeals Tribunal ruled five times that averaged income could not establish a debt. Those first-tier decisions were private, the department never appealed them, and the scheme kept raising debts by averaging until November 2019, when the government conceded in the Federal Court that the method was unlawful. The errors had clustered where someone could see them. Nobody who saw them could stop the line.
The administrative andon cord
The administrative andon cord gives frontline workers and affected people a legal power to pause an automated or procedural system when they see failures clustering. Management cannot file the halt as a review period and carry on. The system is built to obey it.
Three conditions make the cord real. The person who pulls it has to be protected from professional, economic and social retaliation, because if reporting a defect is treated as insubordination the cord will never be pulled. The system cannot resume until an independent review has found and fixed the cause of the halt. And the right has to rest in statute or contract, beyond the discretion of the people who built the system.
In practice this means statutory rights for caseworkers, nurses and other frontline staff to freeze automated enforcement, such as eviction filings or debt collection, once errors start to cluster.
Who decides that errors are clustering
Deciding when errors cluster is the design problem, and it has no clean answer. Some count has to trip the cord: reversals on appeal, complaints of one kind, overturned decisions within one category of case. Someone has to keep that count, and it cannot be the office running the system, which has every reason to find each error isolated.
The threshold has to be written before the system runs, because a threshold set afterwards gets set wherever the current error rate happens to sit. Any fixed number will be too high for some failures and easy to game for others. Robodebt had the count already, in the tribunal’s own decisions. What it lacked was anyone obliged to add them up and able to act on the total.
After the stop
A halt settles little if the claimant still has to prove the system wrong, without the rules, data or logs that produced the decision. So the halt should move the burden. Call it evidentiary inversion. Once the cord is pulled, the institution loses its presumption of competence for that function. It must open its logs, decision rules and case records to an independent reviewer, show that the system worked as intended, and pay for the inquiry, with nothing charged to the claimant.
The law already has pieces of this. Res ipsa loquitur lets a court infer negligence when an accident is the kind that does not normally happen without it and the cause was under the defendant’s control. The burden of proof stays where it was, yet the doctrine lets an injured person reach a jury without explaining a mechanism they could not see.
Goldberg v. Kelly (1970) held that a state must offer a hearing before it cuts off welfare benefits. Article 22 of the GDPR restricts decisions based solely on automated processing that have legal or similarly significant effects, and where such decisions are allowed it gives people the right to human intervention and to contest them. Article 5(2) requires the organization in charge of the processing to demonstrate compliance with the regulation’s principles, accuracy among them.
None of these goes as far as a halt that holds until the institution proves its system sound. Goldberg and Article 22 secure a human hearing. Res ipsa loquitur and Article 5(2) show that where the evidence sits on one side, the law can already move the load toward the side that holds it.
Watching a system fail without the power to stop it is complicity.
Corrections
- 13 October 2026: a 7 October correction here said Toyota’s cord only called for help. Toyota’s own description says the operator can stop the line by pulling it, so the subtitle says that again. The claim that a stopped line cannot restart until the problem is fixed, which Toyota does not make, is gone.
- 7 October 2026: this essay said the people living with the errors had “no legal means to halt it.” The Federal Court case above was one such means; none was fast or collective enough to stop the scheme.
- 7 October 2026: this essay cited Goldberg v. Kelly and GDPR Article 22 as showing that “the law already leans the burden toward the side that holds” the evidence. Those govern hearings and solely automated decisions; the burden point rests on res ipsa loquitur and GDPR Article 5(2).
Notes
- Toyota Motor Corporation, “Toyota Production System,” https://global.toyota/en/company/vision-and-philosophy/production-system/.
- Madeleine Clare Elish, “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction,” Engaging Science, Technology, and Society 5 (2019).
- What the scheme did to the people it misread while they appealed: What Happens While You’re Right.
- Tribunal decisions: Canberra Times, ‘Like reading your own obituary’: leading robodebt critic reacts to royal commission report (July 2023), https://www.canberratimes.com.au/story/8265616/like-reading-your-own-obituary-leading-robodebt-critic-reacts-to-royal-commission-report. The concession came in Amato v Commonwealth (Federal Court of Australia, November 2019). See also Royal Commission into the Robodebt Scheme, Report (July 2023), https://web.archive.org/web/20260830105238/https://robodebt.royalcommission.gov.au/.
- Goldberg v. Kelly, 397 U.S. 254 (1970).
- Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 5(2) and 22.