A technology rollout can arrive with clean performance numbers. An automated claims router reports 94% straight-through processing. A clinical intake tool claims an 80% reduction in review time. An automated triage model boasts sub-second classification speed.
Inspect the workflow where the software meets production, and a second system appears. Pharmacy technicians manually cross-checking mismatched drug codes in shadow spreadsheets. Intake coordinators reformatting automated text before physicians open encounters. Operations staff running parallel queues to catch records the software silently misrouted.
The institution’s metrics treat that second system as zero:
apparent performance = designed performance
The operating reality is different:
apparent performance = designed performance + unaccounted human compensation
The software claims credit for the whole left side of the equation. It achieves that score only by externalizing the uncounted repair work on the right.
Automation dashboard · this week
94%straight-through processing, credited to the tool
Repair done outside the metric
- Codes cross-checked by hand
- in a shadow spreadsheet
- Output reformatted
- before anyone opens the record
- Misrouted records caught
- in a parallel queue
- Straight-through before repair
- not measured
The accounting boundary
Optimization without an audited accounting boundary is cost shifting. When an algorithm is evaluated only on what happens within its own runtime, any work pushed onto human operators looks like free efficiency.
This distortion does not depend on unpaid overtime. Even on regular hours, when skilled pharmacists, underwriters, or coordinators spend a third of their shift reconciling automated errors, the institution misattributes that output to the tool.
Booking human maintenance under routine labor instead of software defects can make the tool appear cheaper than it is. Count the repair work before deciding whether the deployment saves work overall.
What the record misses
When staff repair automated mistakes before those mistakes enter incident records, reported reliability can improve while the underlying defect persists.
The evidence need not disappear. Shadow spreadsheets, messages, and overtime records may show the repair clearly. But if the dashboard counts only errors that reached a customer, patient, or regulator, leadership sees the corrected result without the work that produced it.
That omission can keep a recurring defect off the redesign list. Where workarounds are recorded, reviewed, and used to fund fixes, repair supplies evidence for redesign instead of concealing the need for it.
Anita Tucker and Amy Edmondson found this in hospitals, where workarounds hide failures from the people who could fix them (“Why Hospitals Don’t Learn from Failures,” 2003), and David Woods’s law of fluency holds that well-adapted work hides the difficulty it absorbs. The accounting boundary adds the bill: the hidden repair is a cost, booked to the wrong line.
The misuse of resilience
When organizations notice this friction, they routinely misinterpret it. They praise their teams for resilience, agility, and dedication under pressure.
That praise inverts the problem.
Workarounds are an indictment of system design, not proof of operational health. When an organization must rely on the improvisational capacity of its staff to prevent software from collapsing, it is relying on adaptation it never measures. Resilience engineering treats that adaptation as a capacity to study and fund; the praise treats it as free.
Two claims sit here, and neither proves the other. Morally, an institution has no right to assume this labor as a baseline. Causally, assuming it reduces the institution’s resilience, because the repair stays invisible and unfunded. The first needs a rights argument; the second needs evidence about what the hidden repair costs.
What would change this conclusion
A reporting system that counts workarounds and reliably funds fixes would make the hidden repair visible and removable. If the institution could see the repair and act on it, the claim that reported reliability improves while the defect persists would no longer hold.
Restoring the return path
If an institution wants to evaluate software honestly, it must bring the accounting boundary to the whole workflow:
- Audit the repair burden. Measure the time staff spend checking, reformatting, and fixing automated outputs as a direct operational cost of the software.
- Track workarounds as design findings. Treat informal spreadsheets and parallel queues as telemetry showing where the model failed, not as routine employee behavior.
- Establish an upstream return path. Recurring friction must travel back to engineering to narrow permissions, retrain models, or halt automation, rather than staying trapped on the floor.
Institutions shouldn’t book human beings as spare capacity. They are the ends for which institutions exist. A system that relies on invisible human error correction to look functional has not achieved automation. It has merely hidden who pays for the mistakes.
Revision
11 November 2026: the accounting-boundary diagram is now a working dashboard sample. It shows the same four facts: the reported 94%, the three kinds of repair, and the result counted as the tool’s.
29 October 2026: this essay previously said successful compensation “destroys the diagnostic evidence of failure.” Repair may leave evidence outside incident records. The argument now identifies the reporting omission and distinguishes apparent savings from savings across the whole workflow.