Home / Demos / Alert effectiveness
Are the alerts actually helping?
Administrators need to know whether decision-support alerts earn their place in a clinician's day. This walkthrough shows the Effectiveness page. It pairs published benchmark results with live accept, dismiss and review feedback, reports acceptance with a confidence interval and the median time to review, and suppresses any cell too small to share. It then shows the admin-only runtime setting that picks realtime or batch LLM transport for each agent. A safety gate still keeps live PHI on realtime.
What you'll see
- Benchmark precision, recall and cost per case, each with a confidence interval
- Every alert counted in exactly one outcome cell, including ones resolved in the EHR
- Small-cell suppression: counts under 11 are masked and rates are withheld
- Admin-only realtime or batch transport, with the safety gate's reason shown on the page
Chapters
- 0:00 What the Effectiveness page answers
- 0:23 Benchmark results
- 0:41 Live feedback by outcome
- 1:01 Acceptance rate and review time
- 1:19 Small-cell suppression
- 1:38 Realtime or batch LLM transport
- 1:51 Deployment batch flag
- 2:10 Saving and the PHI safety gate
- 2:27 Configuration only, audited
- 2:36 About this recording
Transcript
0:04 The Effectiveness page answers one question for administrators: are the CDS alerts actually helping? It pairs published benchmark results with live clinician feedback, and it shows aggregate figures only. No patient, case, or free text ever appears here.
0:23 Benchmark results come from published evaluation cards: care-gap precision with and without the pre-publish gate and recall, each with its confidence interval, plus cost per case.
0:41 Live feedback is filtered here to one synthetic demo facility. Every alert lands in exactly one cell: accepted, dismissed, reviewed, pending, expired, or resolved in EHR, which counts alerts the EHR closed before any clinician decided.
1:01 When a group is large enough, you get the acceptance rate with a 95% confidence interval, plus the median time to review. Here, 46 of 67 decided alerts were accepted, about 69%, with a median review time of 1.3 hours.
1:19 Small cells are suppressed. Any count from 1 to 10 shows as less than 11, a total that would give a hidden count away is masked, and the rate is withheld so the small count can't be worked out.
1:38 Runtime settings are admin only. Each agent's LLM transport is realtime or batch. Batch costs about half as much and applies to scheduled runs only, and every agent shows the transport actually in effect.
1:51 On this local demo stack the deployment's batch flag is on, so batch can be chosen. Where the flag is off, the option is disabled and the server refuses a batch save.
2:10 Saving stores the choice and pushes it to the agents service. The safety gate still decides: live PHI stays on realtime until the batch retention safeguards are in place and the routing policy approves batch, and the page says exactly why.
2:27 Saving changes configuration only. It doesn't start an agent run, and the stored setting records which admin made the change.
2:36 A note on this recording: the narration uses a synthetic voice. The benchmark results are aggregate figures from a public, de-identified research dataset, and all other data shown is synthetic demo data, not a live customer deployment.
Want to see this on your workflow?
We run walkthroughs against realistic synthetic populations that match your case mix.
Request a walkthrough