Research
When people keep fixing a system’s mistakes, does their work hide the need to redesign it? I study when institutions can reasonably demand that effort, and how they can reduce it without restricting workers’ judgment or increasing surveillance.
The hypothesis
Staff repair errors, reported outcomes remain green, and managers receive less evidence that redesign is needed. I want to test whether this happens in AI-assisted workflows: does the checking and correction work offset the time a tool is reported to save, and does that work reach the people who can change the design?
Tucker and Edmondson described hospital nurses working around failures that never reached the managers who could remove their causes (“Why Hospitals Don’t Learn from Failures,” California Management Review, 2003). David Woods’s law of fluency describes how skilled adaptation hides the difficulty it absorbs. My inquiry applies these ideas to repair work around AI outputs.
How I would test it
Most of the instruments already exist. NASA-TLX, the NASA Task Load Index (Hart and Staveland, 1988), is a validated questionnaire a worker completes right after a task, rating six dimensions of workload, mental demand among them. Repair latency, the gap between a system’s wrong output and the correction it received, can be read from the audit logs the systems already keep, once someone labels which outputs were wrong. Mental demand is the scarce resource “How to Design for Cognitive Scarcity” designs around — the same low-bandwidth moments where repair happens.
After-hours EHR time is a proxy for repair done outside scheduled hours. It does not measure unpaid work, since most clinicians are salaried, and a timestamp cannot tell repair from routine charting, so it has to be paired with labeled repair events. What is missing is less the measurement than an organization willing to let its repair work be counted.
Three explanations could produce similar experiences for frontline staff, and they do not point to the same fix:
- Missing information. Managers do not see the corrective work or its causes.
- Conflicting incentives. Managers know about the deficiency, but their incentives discourage correction.
- Constraints on correction. Managers recognize the problem but lack the resources or authority to address it.
A first test would follow one AI-assisted documentation workflow and compare the time savings reported for it against the checking, correction, and follow-up work it creates. Stable throughput alongside persistent repair would establish a burden. Where redesign still does not follow, the explanation that holds decides the remedy — instrumentation for the first, changed incentives for the second, authority for the third.
Measurement would also need safeguards against punitive individual scoring and a meaningful role for affected workers in deciding what gets corrected. The argument about unpaid repair work in “Toothless Ethics” motivates the inquiry, which includes paid but misattributed effort.
A second hypothesis: institutional latency
The same evidence reaches a regulator, a guideline panel, and every payer, and each of them acts on its own clock. I call the gap institutional latency: the time between evidence reaching an institution and that institution acting on it, measured at each link of the chain. Time to access is already counted one link at a time, as Europe’s W.A.I.T. indicator does every year.
My hypothesis concerns the rest of the chain: that institutions withdraw a practice more slowly than they adopt one, and that the people covering for the gap supply the compensation this page describes. A 2024 study of cancer guidelines found lopsided updating: NCCN’s recommendations caught up with the FDA in 30 of 36 conversions to regular approval, and in only 7 of 12 withdrawals (Mooghali et al., BMJ Medicine, 2024).
Those are rates, not elapsed times. Timing both directions on comparable evidence is the test, and withdrawal no slower than adoption would refute the hypothesis.
Field observations
| Case | Mechanism | Who absorbs the cost | Which diagnostic questions |
|---|---|---|---|
| Emergency department loudspeaker | Patients are called for triage by name over a loudspeaker. A Deaf patient cannot hear the call, so they cannot tell when their turn has come. | The patient; triage delays compound while the patient has no way to detect or contest the queue. |
|
| Epic: a misrouted clinical alert | A misrouted alert could bury a critical lab result. | Whichever clinician trusted the queue; nobody in the routing system answered for it. |
|
| Andwise: monetization | The monetization paths most likely to fund growth would have made employers or financial institutions the customer. | The physician, the party the product answered to — preserved by shutting the company down. |
|
Field observation: how recommendations shift clinician judgment
In 2012, working on clinical decision support for type 2 diabetes at Georgia Tech and Emory, we ran an early pilot measuring how clinicians revised their recommendations after seeing an algorithm’s output. What stood out was not whether the recommendation was right, but what happened to the clinician’s own judgment once they had seen it.
- Internists n=2 62% 92% change +30 pts
- Familiar with the rules n=4 64% 86% change +22 pts
- Endocrinologists n=4 68% 76% change +8 pts
- Unfamiliar with the rules n=2 71% 71% change +0 pts
Internists’ agreement rose more than endocrinologists’, and reviewers familiar with the rules shifted more than those unfamiliar with them. The two comparisons cannot be separated: both internists knew the rules, and both reviewers who did not were endocrinologists. These are observations from six clinicians.
In this pilot, clinicians’ recommendations shifted toward the algorithm after they saw its output. The figures count agreement, not correctness, so they show movement toward the algorithm’s answer, not deference to a wrong one. The framework requires accountable human review before automated outputs become institutional policy.
Jain, K., Patel, K., Rowland, J., Yong, C. DiaMonD (DIAbetes MONitoring and Dosing) System. Wallace H. Coulter Department of Biomedical Engineering, Georgia Tech / Emory University. Advised by Lawrence Phillips, MD.
Where this work is published and built
I publish this work in three forms: essays, proposed standards, and software.
- Essays: The Crumple Zone, named for Madeleine Clare Elish’s “moral crumple zone” (2019), documents the work people do to compensate for institutional failures and offers ways to identify those demands.
- Proposed standards: Ethotechnics specifies how systems should record evidence, limit authority, and support challenges and corrections. The standards include schemas and audit requirements.
- Software: NextConsensus (commercial), Ambit (experimental), and Refract (live, open source) implement parts of these mechanisms. Current work describes what each project does and its stage of development.
Revised