Research
When people keep fixing a system’s mistakes, does their work hide the need to redesign it? I study when institutions can reasonably demand that effort, and how they can reduce it without restricting workers’ judgment or increasing surveillance.
The corrective value of human experience
Institutions should be evaluated by the outcomes they achieve, the human effort required to achieve them, and their capacity to learn from and eliminate unnecessary demands for compensation. Observed institutional performance decomposes into designed capacity plus human compensation: when uncredited human effort is omitted from the accounting boundary, apparent success is counterfeit.
The aim is to connect reported performance to the effort behind it and the people the institution serves. Recurring workarounds contain knowledge about where design meets operational reality. Correction should return time, discretion, and capacity to workers and improve services, rather than turn that knowledge into higher productivity demands. Eliminating avoidable repair work can be an improvement even when a conventional metric counts less activity.
How successful compensation may conceal failure
My hypothesis is that successful human compensation weakens the error signals an institution needs to recognize its own deficiencies. When frontline workers quietly absorb discrepancies, they act as signal attenuators that blind the organization to systemic failure. Staff repair errors, reported outcomes remain green, and managers receive less evidence that redesign is needed. If that changes decisions about correction, adaptation helps reproduce the conditions that made it necessary.
Empirical mechanism: burden accounting
Resilience engineering and administrative burden. To what extent do conventional organizational metrics treat human compensation and informal improvisation as baseline system performance, and under what conditions does successful adaptation conceal underlying design flaws, externalize operational costs, or deplete long-term adaptive capacity rather than trigger systemic learning?
Normative legitimacy: non-domination
Political philosophy and republican non-domination. Under what conditions, if any, can an institution legitimately require individuals to absorb structural, operational, or resource deficiencies, and what ethical obligations arise for institutional redesign when feasible alternatives exist to alleviate that compensatory burden?
Institutional governance: error-correcting authority
Cybernetics, street-level bureaucracy, and commons governance. Which measurement, accountability, and decision-making architectures allow institutions to surface and address invisible compensatory labor without converting the workers performing it into targets of surveillance, undermining necessary professional judgment, or foreclosing avenues for meaningful contestation?
Application to healthcare AI and operations
When algorithmic tools or hospital operational workflows rely on clinicians and staff to catch errors, bridge system gaps, and perform informal workarounds, does this hidden compensatory labor mask flawed technical design? How can health systems audit and eliminate these recurring administrative burdens without automating away essential clinical discretion or instituting punitive surveillance metrics?
How I would test the hypothesis
The instruments to answer this already exist. NASA-TLX, the NASA Task Load Index, is a validated questionnaire that captures mental demand at the moment of work rather than in a retrospective survey. Repair latency, the gap between a system’s wrong output and the correction it received, can be read from audit logs the systems already keep. After-hours entry timestamps separate paid time from unpaid repair. What is missing is not measurement; it is an organization willing to let its repair work be counted.
Three explanations could produce similar experiences for frontline staff, and they do not point to the same fix:
- Missing information. Managers do not see the corrective work or its causes.
- Conflicting incentives. Managers know about the deficiency, but their incentives discourage correction.
- Constraints on correction. Managers recognize the problem but lack the resources or authority to address it.
A first test would follow one AI-assisted documentation workflow and compare the time savings reported for it against the checking, correction, and follow-up work it creates. Stable throughput alongside persistent repair would establish a burden. Where redesign still does not follow, the explanation that holds decides the remedy — instrumentation for the first, changed incentives for the second, authority for the third.
Measurement would also need safeguards against punitive individual scoring and a meaningful role for affected workers in deciding what gets corrected. The argument about unpaid repair work in “Toothless Ethics” motivates the inquiry, which includes paid but misattributed effort.
Field observations
| Case | Mechanism | Who absorbs the cost | Which diagnostic questions |
|---|---|---|---|
| Emergency department loudspeaker | Patients are called for triage by name over a loudspeaker. For a Deaf patient the mechanism deciding who waits is inaudible. | The patient; triage delays compound while the patient has no way to detect or contest the queue. |
|
| Epic: a misrouted clinical alert | A misrouted alert could bury a critical lab result. | Whichever clinician trusted the queue; nobody in the routing system answered for it. |
|
| Andwise: monetization | The monetization paths most likely to fund growth would have made employers or financial institutions the customer. | The physician, the party the product answered to — preserved by shutting the company down. |
|
Field observation: how recommendations shift clinician judgment
In 2012, working on clinical decision support for type 2 diabetes at Georgia Tech and Emory, we ran an early pilot measuring how clinicians revised their recommendations after seeing an algorithm’s output. What stood out was not whether the recommendation was right, but what happened to the clinician’s own judgment once they had seen it.
- Internists n=2 62% 92% change +30 pts
- Familiar with the rules n=4 64% 86% change +22 pts
- Endocrinologists n=4 68% 76% change +8 pts
- Unfamiliar with the rules n=2 71% 71% change +0 pts
Internists’ agreement rose more than endocrinologists’ agreement. Reviewers familiar with the rules shifted more than those unfamiliar with them. These are observations from six clinicians, with overlapping subgroups.
In this pilot, clinicians’ recommendations shifted toward the algorithm after they saw its output. The framework requires accountable human review before automated outputs become institutional policy.
Jain, K., Patel, K., Rowland, J., Yong, C. DiaMonD (DIAbetes MONitoring and Dosing) System. Wallace H. Coulter Department of Biomedical Engineering, Georgia Tech / Emory University. Advised by Lawrence Phillips, MD.
Where this work is published and built
This work takes three forms: essays, proposed standards, and software.
- Essays: The Crumple Zone documents the work people do to compensate for institutional failures and offers ways to identify those demands.
- Proposed standards: Ethotechnics specifies how systems should record evidence, limit authority, and support challenges and corrections. The standards include schemas and audit requirements.
- Software: NextConsensus (commercial), Ambit (experimental), and Refract (live, open source) implement parts of these mechanisms. Current work describes what each project does and its stage of development.
Revised