Stopping power
In the Toyota Production System, any assembly line worker who detects a problem can pull the andon cord. A team leader comes to help, and if the problem is not cleared within the work cycle, that section of the line stops. It cannot restart until the problem is fixed.
Call this stopping power.
The silent failure
Automated and administrative governance took the cord out. When a system starts producing clustered errors, misclassifying claims, flagging transactions, withholding services people depend on, the people living with those errors have no legal means to halt it. The errors get filed as noise, edge cases or user error. The machine keeps running, the efficiency numbers keep looking good, and the harm lands in the crumple zone.
Australia’s Robodebt scheme, which raised welfare debts by averaging annual income across fortnights, ran this way for years. In 2017 a member of the Administrative Appeals Tribunal ruled five times that averaged income could not establish a debt. Those first-tier decisions were private, the department never appealed them, and the scheme kept raising debts by averaging until November 2019, when the government conceded in the Federal Court that the method was unlawful. The errors had clustered where someone could see them. Nobody who saw them could stop the line.
The administrative andon cord
The administrative andon cord gives frontline workers and affected people a legal power to pause an automated or procedural system when they see failures clustering. It goes further than Toyota’s cord, which asks for help before it stops anything. Management cannot file the halt as a review period and carry on. The system is built to obey it.
Three conditions make the cord real. The person who pulls it has to be protected from professional, economic and social retaliation, because if reporting a defect is treated as insubordination the cord will never be pulled. The system cannot resume until an independent review has found and fixed the cause of the halt. And the right has to rest in statute or contract, beyond the discretion of the people who built the system.
In practice this means statutory rights for caseworkers, nurses and other frontline staff to freeze automated enforcement, such as eviction filings or debt collection, once errors start to cluster.
Who decides that errors are clustering
Deciding when errors cluster is the design problem, and it has no clean answer. Some count has to trip the cord: reversals on appeal, complaints of one kind, overturned decisions within one category of case. Someone has to keep that count, and it cannot be the office running the system, which has every reason to find each error isolated.
The threshold has to be written before the system runs, because a threshold set afterwards gets set wherever the current error rate happens to sit. Any fixed number will be too high for some failures and easy to game for others. Robodebt had the count already, in the tribunal’s own decisions. What it lacked was anyone obliged to add them up and able to act on the total.
After the stop
A halt settles little if the claimant still has to prove the system wrong, without the rules, data or logs that produced the decision. So the halt should move the burden. Call it evidentiary inversion. Once the cord is pulled, the institution loses its presumption of competence for that function. It must open its logs, decision rules and case records to an independent reviewer, show that the system worked as intended, and pay for the inquiry, with nothing charged to the claimant.
The law already has pieces of this. Res ipsa loquitur lets a court infer negligence when an accident is the kind that does not normally happen without it and the cause was under the defendant’s control. The burden of proof stays where it was, yet the doctrine lets an injured person reach a jury without explaining a mechanism they could not see.
Goldberg v. Kelly (1970) held that a state must offer a hearing before it cuts off welfare benefits. Article 22 of the GDPR restricts decisions based solely on automated processing that have legal or similarly significant effects, and where such decisions are allowed it gives people the right to human intervention and to contest them.
None of these goes as far as a halt that holds until the institution proves its system sound. They show that where the evidence sits on one side, the law already leans the burden toward the side that holds it.
Watching a system fail without the power to stop it is complicity.
Notes
- What the scheme did to the people it misread while they appealed: What Happens While You’re Right.
- Tribunal decisions: Canberra Times, ‘Like reading your own obituary’: leading robodebt critic reacts to royal commission report (July 2023), https://canberratimes.com.au/story/8265616/like-reading-your-own-obituary-leading-robodebt-critic-reacts-to-royal-commission-report. The concession came in Amato v Commonwealth (Federal Court of Australia, November 2019). See also Royal Commission into the Robodebt Scheme, Report (July 2023), https://robodebt.royalcommission.gov.au/.
- Goldberg v. Kelly, 397 U.S. 254 (1970).
- Regulation (EU) 2016/679 (General Data Protection Regulation), Article 22.