Usage expanded faster than expected
The system is now used in three additional clinics and in more complex follow-up encounters.
Interactive case
Reliance Lab
Explore how an organization should respond when the facts supporting an AI deployment change after authorization.
The governing question Does the current evidence still justify unchanged reliance?
Healthcare · Case 01
Clinical Summary Assistant
A composite case, not a specific deployment. The figures are constructed to make the decision realistic.
An AI system that produces concise summaries of recent patient visits for clinicians before follow-up appointments.
6 months ago Used by 120 clinicians across 3 ambulatory clinics
Generated summary Follow-up visit
Symptoms improved. Medication list reviewed. Allergy history requires source-note confirmation.
Source verification required
01 · Original authorization
What the organization initially permitted
Authorization is bounded permission to rely on a system under stated conditions—not a permanent declaration that it is safe.
Authorization record Active · bounded use
- The system is assistive and does not replace clinical judgment.
- Use is limited to adult ambulatory follow-up visits.
- Clinicians review the underlying note before acting on the summary.
- Performance is reviewed monthly and material incidents trigger reconsideration.
02 · New developments
What changed after authorization?
Changes in evidence, use, workflow, or policy can invalidate assumptions even when the model itself has not changed.
A recurring omission pattern appeared
A focused review found that approximately 2% of sampled summaries omitted a clinically relevant negation.
Review behavior changed
Median time spent checking source notes fell by 62% compared with the initial rollout period.
Documentation policy changed
A new hospital policy requires explicit documentation of AI use and known limitations.
03 · Evidence and context
The record is mixed, but not ambiguous
No single item determines the disposition. The task is to judge what the combined record means for continued reliance.
Quantitative
Focused error analysis
2.1% of reviewed summaries omitted at least one key negation; errors clustered in medication and symptom histories.
Qualitative
Clinician interviews
Several clinicians described the summary as complete enough to replace a first read of the source note.
Incident
Incident report 47
A missed allergy mention delayed a follow-up prescription until the source note was reviewed.
Quantitative
Review-time telemetry
Median source-note review time decreased from 74 seconds to 28 seconds after the first month.
04 · Make the call
Make the call
New information has emerged since authorization. Should the organization continue relying on the system under its current terms?
Choose a disposition. The lab will expose the consequences and assumptions embedded in your decision.
05 · Impact and feedback
Impact of choosing “Continue”
This choice treats the observed changes as insufficient to alter the original authorization. It preserves speed, but leaves the changed review behavior and omission pattern largely unmanaged.
Near-term effects
- No interruption to clinical workflow
- No new training or documentation burden
- Known omission pattern remains exposed
Longer-term effects
- May normalize reduced source-note review
- Could make future incidents harder to attribute
- Weakens the credibility of the original authorization conditions
What this decision depends on
- The sampled omission rate does not represent broader performance
- Clinicians will independently restore adequate review behavior
- The incident was isolated rather than a leading signal
What the organization must monitor
- Negation omission rate
- Source-note review time
- Medication-related incidents
05 · Impact and feedback
Impact of choosing “Narrow”
This choice recognizes that the system may remain useful while the facts no longer support unchanged reliance. It preserves bounded use, restores review discipline, and creates evidence for a later expansion decision.
Near-term effects
- Reduces exposure to the observed failure pattern
- Requires explicit source-note verification
- Adds targeted documentation and training
Longer-term effects
- Builds evidence about whether controls work
- Makes future expansion conditional rather than assumed
- Preserves institutional correction capacity
What this decision depends on
- Clinicians will follow the revised review requirement
- The highest-risk encounter types can be reliably excluded
- Targeted remediation can materially reduce omission risk
What the organization must monitor
- Negation omission rate
- Review compliance
- Clinician-reported workarounds
05 · Impact and feedback
Impact of choosing “Suspend”
This choice prioritizes uncertainty reduction over continuity. It is most defensible when the organization believes the incident and review drift may indicate an uncontrolled patient-safety risk.
Near-term effects
- Stops exposure while investigation proceeds
- Creates immediate workflow disruption
- Signals a high threshold for tolerated uncertainty
Longer-term effects
- May produce the clearest evidence before reauthorization
- Could reduce clinician confidence in future deployments
- May be disproportionate if the error pattern is readily controlled
What this decision depends on
- The potential harm of continued use exceeds the cost of interruption
- A timely investigation can resolve the central uncertainty
- Manual workflow can absorb the temporary loss of the system
What the organization must monitor
- Investigation findings
- Manual workflow burden
- Readiness for reauthorization
05 · Impact and feedback
Impact of choosing “Expand”
This choice interprets rapid adoption as evidence of value. It does not adequately account for the fact that usage expanded at the same time that source verification declined and a material omission pattern emerged.
Near-term effects
- Extends time savings to more clinicians
- Increases exposure before controls are validated
- Makes the original scope boundary effectively nonbinding
Longer-term effects
- Could compound workflow dependence
- Raises the cost of later correction
- Converts unresolved uncertainty into broader institutional reliance
What this decision depends on
- Adoption is a reliable proxy for safe utility
- Observed errors will not scale with usage
- Broader deployment will not further reduce verification behavior
What the organization must monitor
- Adoption volume
- Incident rate
- Time to detect cross-clinic failures
Decision feedback
The point is not to find a risk-free answer.
A defensible decision identifies the conditions under which reliance remains justified, the uncertainty being accepted, and the facts that would force another review.
A strong reliance decision is:
- bounded to a defined use and population;
- traceable to evidence and explicit assumptions;
- open to challenge by affected people;
- correctable when the operating facts change.