How to read a risk score without getting fooled

More and more of the chart is going to show you numbers like this:

Risk of admission, 30 days: 34%

It looks precise. It is a guess with a method behind it. That method can be good or bad, and the screen won't tell you which.

Here are the five questions I teach.

1. Risk of what?

"Admission" can mean any admission, an unplanned one, or one for a specific condition. "Deterioration" can mean ICU transfer, a rapid response, or death. Different outcomes, different scores. If you can't name the outcome, you can't use the number.

2. By when?

Thirty days and one year are different questions. A 34% risk over a year is very different from 34% by next month. The time window changes what you should do this week.

3. Compared to whom?

A model learns from a group of past patients. If that group was mostly older, mostly insured, or mostly from one hospital, the score may not fit the patient in front of you.

Also ask about the base rate. If 30% of similar patients get admitted anyway, a score of 34% tells you very little. If 3% do, it tells you a lot.

4. What couldn't it see?

This is the one people skip. A model can only use the data it has. If the ED visit at the other hospital never reached your chart, the model didn't see it either.

That cuts both ways. A low score might just mean missing data. In the Chart Terminal, the same invented patient scores 15% with what a practice sees today and 65% with everything connected. Nothing about him changed. The data did.

5. Would acting on it change anything?

A good score points to something you can do: a call, a med change, a visit. If there is nothing to do differently, the score is trivia. If there is, the score is only as good as the follow-through.

Why this matters now

Federal rules have required certified EHR vendors to publish "source attributes" for predictive tools, basically a label showing what the model was trained on and how it performs. The HTI-5 proposal would remove that requirement. (Covington) As of this writing it is still a proposal.

If the label goes away, the five questions are still yours to ask.

A risk score is a reason to look. It is not a reason to stop thinking.