Usage expanded faster than expected
The system is now used in three additional clinics and in more complex follow-up encounters.
The Decide step, with you in it
Reliance Lab
An AI assistant has been in use for six months. New evidence has emerged. Decide whether to continue, narrow, suspend, or expand its use.
Reconsideration loop · step 05, Decide Does the current evidence still justify unchanged reliance?
Healthcare · Ambulatory follow-up
Clinical Summary Assistant
A composite case, not a specific deployment. The figures are constructed to make the decision realistic.
An AI system that produces concise summaries of recent patient visits for clinicians before follow-up appointments.
6 months ago Used by 120 clinicians across 3 ambulatory clinics
Generated summary Follow-up visit
Symptoms improved. Medication list reviewed. Allergy history requires source-note confirmation.
Source verification required
What the organization initially permitted
Authorization is bounded permission to rely on a system under stated conditions — not a permanent declaration that it is safe.
Authorization record Active · bounded use
- The system is assistive and does not replace clinical judgment.
- Use is limited to adult ambulatory follow-up visits.
- Clinicians review the underlying note before acting on the summary.
- Performance is reviewed monthly and material incidents trigger reconsideration.
What changed after authorization?
Changes in evidence, use, workflow, or policy can invalidate assumptions even when the model itself has not changed.
A recurring omission pattern appeared
A focused review found that approximately 2% of sampled summaries omitted a clinically relevant negation.
Review behavior changed
Median time spent checking source notes fell by 62% compared with the initial rollout period.
Documentation policy changed
A new hospital policy requires explicit documentation of AI use and known limitations.
The evidence so far
No single item determines the disposition. The task is to judge what the combined record means for continued reliance.
Quantitative
Focused error analysis
2.1% of reviewed summaries omitted at least one key negation; errors clustered in medication and symptom histories.
Qualitative
Clinician interviews
Several clinicians described the summary as complete enough to replace a first read of the source note.
Incident
Incident report 47
A missed allergy mention delayed a follow-up prescription until the source note was reviewed.
Quantitative
Review-time telemetry
Median source-note review time decreased from 74 seconds to 28 seconds after the first month.
Make the call
New information has emerged since authorization. Should the organization continue relying on the system under its current terms?
Choose an action to see its consequences and the assumptions behind it.
Kept in this browser alongside your disposition, and shown with the consequences below. Nothing is sent anywhere.
Consequences
Choose an action above to see what the organization would need to do next.
Impact of choosing “Continue”
This choice treats the observed changes as insufficient to alter the original authorization. It preserves speed, but leaves the changed review behavior and omission pattern largely unmanaged.
Near-term effects
- No interruption to clinical workflow
- No new training or documentation burden
- Known omission pattern remains exposed
Longer-term effects
- May normalize reduced source-note review
- Could make future incidents harder to attribute
- Weakens the credibility of the original authorization conditions
What this decision depends on
- The sampled omission rate does not represent broader performance
- Clinicians will independently restore adequate review behavior
- The incident was isolated rather than a leading signal
What the organization must monitor
- Negation omission rate
- Source-note review time
- Medication-related incidents
Impact of choosing “Narrow”
This choice recognizes that the system may remain useful while the facts no longer support unchanged reliance. It preserves bounded use, restores review discipline, and creates evidence for a later expansion decision.
Near-term effects
- Reduces exposure to the observed failure pattern
- Requires explicit source-note verification
- Adds targeted documentation and training
Longer-term effects
- Builds evidence about whether controls work
- Makes future expansion conditional rather than assumed
- Preserves institutional correction capacity
What this decision depends on
- Clinicians will follow the revised review requirement
- The highest-risk encounter types can be reliably excluded
- Targeted remediation can materially reduce omission risk
What the organization must monitor
- Negation omission rate
- Review compliance
- Clinician-reported workarounds
Impact of choosing “Suspend”
This choice prioritizes uncertainty reduction over continuity. It is most defensible when the organization believes the incident and review drift may indicate an uncontrolled patient-safety risk.
Near-term effects
- Stops exposure while investigation proceeds
- Creates immediate workflow disruption
- Signals a high threshold for tolerated uncertainty
Longer-term effects
- May produce the clearest evidence before reauthorization
- Could reduce clinician confidence in future deployments
- May be disproportionate if the error pattern is readily controlled
What this decision depends on
- The potential harm of continued use exceeds the cost of interruption
- A timely investigation can resolve the central uncertainty
- Manual workflow can absorb the temporary loss of the system
What the organization must monitor
- Investigation findings
- Manual workflow burden
- Readiness for reauthorization
Impact of choosing “Expand”
This choice interprets rapid adoption as evidence of value. It does not account for usage expanding at the same time that source verification declined and a pattern of material omissions emerged.
Near-term effects
- Extends time savings to more clinicians
- Increases exposure before controls are validated
- Makes the original scope boundary effectively nonbinding
Longer-term effects
- Could compound workflow dependence
- Raises the cost of later correction
- Converts unresolved uncertainty into broader institutional reliance
What this decision depends on
- Adoption is a reliable proxy for safe utility
- Observed errors will not scale with usage
- Broader deployment will not further reduce verification behavior
What the organization must monitor
- Adoption volume
- Incident rate
- Time to detect cross-clinic failures
How your call sits against the other three
The same six fields for every disposition. Your pick is the highlighted column; change it above and the highlight follows.
| Field | Continue Your pick | Narrow Your pick | Suspend Your pick | Expand Your pick |
|---|---|---|---|---|
| What the choice does | This choice treats the observed changes as insufficient to alter the original authorization. It preserves speed, but leaves the changed review behavior and omission pattern largely unmanaged. | This choice recognizes that the system may remain useful while the facts no longer support unchanged reliance. It preserves bounded use, restores review discipline, and creates evidence for a later expansion decision. | This choice prioritizes uncertainty reduction over continuity. It is most defensible when the organization believes the incident and review drift may indicate an uncontrolled patient-safety risk. | This choice interprets rapid adoption as evidence of value. It does not account for usage expanding at the same time that source verification declined and a pattern of material omissions emerged. |
| How well the record supports it | Weakly supported response | Proportionate response | High-control response | Weakly supported response |
| Near-term effects |
|
|
|
|
| Longer-term effects |
|
|
|
|
| What this decision depends on |
|
|
|
|
| What the organization must monitor |
|
|
|
|
Scroll sideways for more
The point is not to find a risk-free answer
A defensible decision identifies the conditions under which reliance remains justified, the uncertainty being accepted, and the facts that would force another review.
A strong reliance decision is:
- bounded to a defined use and population;
- traceable to evidence and explicit assumptions;
- open to challenge by affected people;
- correctable when the operating facts change.
Revised
- Merge pull request #875 from zz-plant/site-ugly-nonsense-fixes
The last changes to this page’s source, as their commits describe them. A change to a file that several pages share appears on each.