Institutional error detection
detecting operational failures standard metrics miss and wiring frontline signals to redesign
Research
An AI tool flags more patients for follow-up, and the hospital expects clinicians to act on every alert without adding time to the schedule. Staff keep up by staying late. Their success hides the shortage.
How do high-performing systems hide operational failures from their own metrics — and what organizational designs let frontline error detection drive redesign?
I am the builder on this work; I am looking for a clinical informatics or health-services Co-PI to be the institutional PI.
Three ways a system locks in
- Possibility When does a diagnosis or risk score start deciding what counts as real? A diagnosis can help a clinician see a problem, and it can stop the next clinician from looking for a different one. I study when a useful category closes the search.
- Authority When a risk score and a patient’s account disagree, who decides what happens next? An appeal can fix one decision without changing the rule behind it. I study which challenges can change the rule, who gets to bring them, and what happens to the people who never appeal.
- Necessity Can a successful product make a better one harder to imagine, or harder to build? A successful product creates the jobs, workflows, and expectations an alternative would have to replace. That accumulated cost is then cited as proof the alternative was never realistic.
Most of this work is in healthcare and AI. The field cases below informed the framework, and the empirical grounding sets out what the literature supports and what it leaves open.
Active research program
- Patch vs. revision
- A patch fixes one error, grants a waiver, or closes a ticket and leaves the workflow untouched. A revision uses the frontline signal to change the rules, thresholds, and review gates that produced the failure.
- Rule change vs. independent judgment
- One test is whether frontline evidence can change the rule. The other is whether users and reviewers can still spot an anomaly and switch paths when the system drifts.
Baseline test: A high throughput or compliance score should never hide the workarounds staff built to keep the system running. A durable design leaves clinicians and reviewers able to spot an anomaly, trigger a rule change, and switch away when performance degrades.
Each study below isolates one testable question with pre-mapped funder fit.
| Study | Core question | Method | Target funders |
|---|---|---|---|
| Study A — AI and discovery | Does fluent AI explanation make clinicians more confident while making them worse at spotting anomalies? | Multi-session study separating explanation fluency from model accuracy to measure whether repeated exposure erodes independent anomaly recognition. | Templeton, NSF HCC, Sloan, ARC, Schmidt AI2050 |
| Study B — participation and authority | When does inviting people to give feedback or sit on advisory boards shift priorities or decision rights? | Comparative study of governance setups that direct feedback into incident queues versus review with power to reopen rules, tracking whether frontline observations trigger formal rule changes. | PCORI (Science of Engagement), RWJF, Siegel, ARC |
| Study C — design and governance | How do you pair exception-surfacing interfaces with real decision rights so objections stick? | Prototyping anomaly-preserving interfaces — tools that surface exceptions instead of smoothing them over — alongside governance charters, audit protocols, and low-cost exit paths. | NSF HCC, Siegel, McGovern, Australian MRFF |
When participation carries authority
For Study B, I would follow a concern from the person raising it to the people who can act on it. I distinguish three questions when examining whether participation changes decision rights.
- Recognition
- Does the institution understand the concern, and does it treat it as credible? These are separate tests: a report can be understood and still disbelieved.
- Standing
- Can the person raising the concern require a formal response, with a named reviewer and a deadline? Being invited to give feedback does not itself grant that right.
- Accountability
- Can affected people challenge the response and the rule behind it, and require the institution to carry out any resulting obligations? A review needs a route to consequences.
Lead study protocol
Lead study: AI explanation and diagnostic independence
How do high-accuracy, fluent AI clinical decision-support systems affect clinicians’ independent anomaly recognition over time?
Design
- N = 120 clinicians across 4 sessions over 8 weeks.
- Factorial design crossing accuracy (92% vs. 78%) × framing (fluent synthesis vs. exception-surfacing, anomaly-preserving presentation), plus a no-AI control.
- Primary outcome: unassisted anomaly recognition on probe vignettes in session 4.
Institutional tracking
- 12-month pre/post audit of frontline overrides and protocol challenges across two governance setups, testing whether a challenge changes the conditions for future objections.
- Checks whether frontline staff can initiate reviews without managerial sponsorship, access previously restricted operational data, and test new practices without incumbent sign-off at every step.
- Measures whether patient representatives influence what counts as a problem, and whether protocol changes reduce hidden workaround labor rather than redistributing it.
Falsifiability: If fluent AI improves unassisted anomaly detection, or if high-performing units show more clinician-initiated protocol review, the lock-in hypothesis is falsified.
Collaborate on Study A
I partner with clinical informatics and health-services researchers as the builder on the study, so a grant team does not have to hire out prototype engineering or translate clinical operations from scratch. The institutional PI brings the academic home, IRB, and site access.
- What I bring
- Over a decade of clinical workflow and EHR implementation work (Epic, Doximity, CancerCompass, Transcarent), full-stack engineering to build experimental software and exception-surfacing prototypes myself (backed by live code in Refract and Ambit), and product framing grounded in hospital deployments.
- What a Co-PI brings
- An eligible academic or health-system home, IRB infrastructure, clinical site access, and PI eligibility for federal or foundation grants such as PCORI Science of Engagement or NSF HCC.
Looking for a clinical informatics or health-services Co-PI for a PCORI Science of Engagement or NSF HCC submission — email me to pressure-test the Study A design. Email to review the Study A protocol →
Mechanism and method
When administrators call a change unrealistic, they usually mean one of five things. The first two are whether the alternative can work and whether it can be organized from where things stand; Erik Olin Wright made the same split in 2010.
Five reasons changes get called unrealistic, and how workarounds raise switching costs
“Unrealistic” can mean
-
It cannot work
-
The capacity does not exist
There are not enough clinicians in the region to answer every alert, whatever anyone pays.
Changes when a constraint moves: fewer alerts, a different mix of staff, or more time to respond.
-
-
It cannot be organized on current terms
-
Nobody would pay
No budget line or contract covers the follow-up time the alerts create.
Changes when someone names the work and funds it: a payer, or the hospital’s own budget.
-
The clinicians who would benefit cannot coordinate
Each clinician absorbs the extra alerts alone, by staying late, and none can change the staffing alone.
Changes when enough of them act together: a department, a union, or a shared count of the hours.
-
The administrators who could approve it have no reason to
Administrators see every alert answered, because staff are staying late to answer them.
Changes when the cost becomes visible to them, or someone with authority signs off on the time.
-
-
Nobody knows yet
-
The evidence is missing
No one has shown that acting on every alert improves outcomes for patients.
Changes when someone collects the evidence.
-
Staying late also makes the fix look expensive. The longer staff absorb the alerts, the more of the hospital’s routine assumes they will, and any alternative has to replace that routine too.
When people absorb the gap
Extra effort can hide a gap and deepen dependence, or it can build the capacity, expertise, and relationships that make change easier. The drawing below shows which conditions produce each path, and the four questions beside the field cases test a record against them.
I have been inside this loop myself. At Epic I wrote custom code to close gaps the base product left; it helped the hospitals I worked with, and it may also have kept those gaps out of the product team’s view.
The life of a claim
The governance of recognition asks what happens when someone brings a problem forward. I follow how the institution interprets the claim, whether the person can require a response, and whether that response can change the rule behind the problem.
-
Expression
A clinician reports an unsafe AI recommendation.
Can stall here The report never reaches a reviewer.
-
Recognition
The hospital records the concern as a safety issue.
Can stall here It is reclassified as a training request, with no way to challenge that classification.
-
Standing
The clinician can require a formal review.
Can stall here The report is acknowledged, but no review is owed.
-
Decision
A reviewer gives reasons for accepting or rejecting the concern.
Can stall here The case stays pending without a review deadline.
-
Consequence
The hospital carries out any required remedy, or explains why no change is warranted.
Can stall here A promised remedy has no owner or follow-through.
A claim can remain acknowledged without reaching a decision someone can challenge. Pending Is a No Nobody Signed develops that argument; the framework’s correction-depth test asks whether review can reach the rules themselves.
Falsifiability criteria and prior literature
What would count against it
- Workarounds that led to a redesign when the people doing them had no authority over it, no collective voice, and nowhere else to go. If that is common, the conditions are not doing the work I claim.
- People who stopped compensating and saw the blame land on the gap rather than on them: a refusal of unpaid overtime, for instance, that managers read as evidence of understaffing. If that is the usual result, the last step of the loop is wrong.
- Costs that were logged and reported, and correction that still waited as long as it did when they were hidden. Then visibility matters less than the drawing says.
- All three conditions present, and the compensation still bought no leverage. Then the conditions are not enough on their own.
- Challenges that rewrote the channel they came through: an appeal that changed the appeals process, or a complaint that changed who reviews complaints. If appeals routinely rewrite the appeals process itself, challengers have more leverage over the system than the framework predicts.
- A target that changed: an institution replaced the outcome it predicted, not the threshold at which it acted, and could show the replacement reduced the harm. If that is common, correction is less rare than the framework implies.
Earlier work
This work builds on health-services and organizational-reliability research: Tucker and Edmondson (2003) on nurses’ quick fixes hiding root causes, Morrison (2015) on workarounds perpetuating shortages, Mark Bovens on operational accountability versus explanation, and Albert Hirschman (1970) on feedback and exit. The empirical question is whether frontline error detection reaches formal system redesign rather than stopping at local workarounds.
Broader theoretical lineage is published in “Where the term comes from” at The Crumple Zone. The Ethotechnics method applies the operational model to automated decisions.
Field observations
Each record below tests the loop against an institutional failure — one of access, one of correction, and one of exit.
None of the three traces what the institution did after it learned of the harm. The next question is what changed: the target, the payment rule, the appeal path, or only the threshold.
I read each case against four questions
- Who bears the cost?
- Could the affected people refuse?
- Does the institution see what the repair work costs?
- Can evidence of that cost alter the cause?
| Case | What the record shows | Source | What I take from it |
|---|---|---|---|
| Deaf patient, missed appointment (Wales) | A Deaf participant in a Welsh focus-group study was the only person left in the waiting room. Staff called names aloud, there was no display downstairs, and the participant missed the appointment. | “They still phone even though they know I’m deaf” Published study | The queue depended on hearing a name, so the first link in the chain, detection, failed; the patient had no way to contest the order, and the cost landed on them.
|
| Epic: the escalation path after go-live | At Epic, a regulatory change or critical bug had no defined route to the person who could fix it. I wrote the escalation path: issue discovery, triage, a named owner, and customer notification. | “Go-live is where the job starts” Firsthand account | The software was installed and still failing because ownership stopped at go-live. Naming the owner and the notification turned a recurring surprise into a route.
|
| Andwise: the wind-down | Andwise reached 1,200+ physicians reviewing employment contracts and wound down in 2024; the May 2024 wind-down email to its medical advisory board cited sustained engagement and funding falling short. | “Reading a physician’s employment contract before they signed it” Firsthand account | Physicians used the tool for a high-stakes contract, but neutral advice for a solo user had no institutional payer that did not compromise confidentiality — showing how funding structure decides which alternatives survive.
|
Tools and writing
These studies run on software, standards, and essays already in progress.
Research infrastructure and standards
- NextConsensus: Working forecasting registry for decisions by named bodies (such as NICE’s appraisal of epcoritamab for follicular lymphoma), scored against frozen resolution rules at the public registry.
- Refract (open source): Logs when underlying sources change so teams and AI systems know which stale claims to recheck.
- Ambit (open source): Maps what tools an AI setup can reach and holds sensitive actions until a person approves them.
- Ethotechnics (proposed standards): Schemas and audit requirements for recording evidence, limiting automated authority, and tying individual remedies back to rule revision — paired with the operational framework.
Stage, source links, and live observation logs for each build are on Current work.
Essays and empirical grounding
- The Crumple Zone: Named for Madeleine Clare Elish’s “moral crumple zone” (2019). Each essay traces an automated decision (fraud filters, benefits queues, clinical alerts) to the clinician, clerk, or patient left carrying the failure.
- The Time AI Saves Has to Go Somewhere: Traces whether faster work gives clinicians control over their time or raises the baseline expected of them.
- Each Mechanism Is Established. The Whole Process Is Untested. Empirical literature review and test design for the framework.
Revised