Absence of evidence is not evidence of absence — and not evidence of effect either

There is a sentence in the Agency for Healthcare Research and Quality’s 2020 comparative effectiveness review that ought to be better known than it is:

No study evaluated the effectiveness of risk mitigation strategies, such as use of risk assessment instruments, opioid management plans, patient education, urine drug screening, prescription drug monitoring program data review, monitoring instruments, more frequent monitoring intervals, pill counts, abuse-deterrent formulations, or avoidance of co-prescribing of benzodiazepines on risk of overdose, addiction, abuse or misuse.

Chou et al., AHRQ Comparative Effectiveness Review No. 229, 2020

Read what that says

It does not say these things were studied and found not to work. It says the question was not evaluated. For ten separate interventions. Several of which are mandated by state law, required by payers, and built into practice everywhere.

The three positions to keep apart

1. Evaluated and effective. Rare here.
2. Evaluated and not effective. Also rare, and a genuinely useful finding when it exists — the randomized trials of pharmacogenomic-guided opioid therapy are in this category.
3. Not evaluated. Which tells you nothing about whether it works, and everything about the strength of any claim that it does.

Conflating the second and third is the most common error in this area, in both directions. Someone who wants to defend a practice says “there is no evidence against it.” Someone who wants to attack it says “there is no evidence for it.” Both are describing the same void.

What follows practically

• Do not fund something as proven when it is unevaluated. Fund it, if you want, as a reasonable practice pending evidence — and say that in the procurement document, so nobody is surprised later.
• Build the evaluation into the deployment. If a large agency is going to require a practice across a population, that is a natural experiment. Pre-register measures and it becomes evidence. Do not, and it becomes another decade of not knowing.
• Attend to the harm side separately. An unevaluated intervention can still have well-documented harms — drug testing has documented false positives and documented misinterpretation. Absence of demonstrated benefit plus presence of demonstrated harm is a worse position than absence of evidence generally.

More: urine drug testing · what is not established.