Opioid program pilot design
Pilot design
A defensible pilot enrolls everyone who meets a stated rule in a defined geography, compares against a matched population or a stepped-wedge rollout, follows patients for at least 12 months, pre-registers its measures and a stopping rule, and gives an independent evaluator the data.
What a first phase has to look like if it is going to produce an answer rather than a testimonial.
Why do most pilots in this field produce no answer?
The standard pilot in this field enrolls a small, cooperative, easily reached group, runs for a few months, reports process measures, and produces a favorable narrative that nobody can act on. It is not dishonest; it is just uninformative, and it is uninformative in a predictable way.
What makes a pilot informative?
- A population defined by a rule, not by convenience. Everyone in a defined geography meeting defined criteria, including the people who are hard to reach — because they are the population the program claims to help and the one it is most likely to fail.
- A comparison of some kind. A matched population, a stepped-wedge rollout across sites, or at minimum a well-characterized pre-period. Without one, every finding is confounded with time, and time is doing a great deal in this field right now: opioid-involved deaths nationally fell from 82,851 in 2022 to 55,007 in 2024 with no help from anybody’s pilot.
- Long enough to see the outcome you care about. Overdose is a low-frequency event. A three-month pilot in a small population cannot detect a change in it, and reporting one would be noise.
- Pre-registered measures, including the negative ones from measurement and evaluation.
- A stopping rule. Agreed in advance: what result would cause this to be halted early. A program with no stopping rule cannot fail, which means it cannot succeed either.
- An independent evaluator with access to the data.
A worked outline
| Element | A defensible choice |
|---|---|
| Population | All patients in a defined county on long-term opioid therapy for chronic non-cancer pain, identified by a stated rule |
| Comparison | Stepped-wedge across sites, or a matched county |
| Duration | At minimum 12 months of follow-up after enrollment closes |
| Primary outcomes | Fatal and non-fatal overdose; mental health crisis events; loss of prescribing clinician |
| Secondary outcomes | Therapy gaps; emergency department presentations; screening completion and response time |
| Safety outcomes | Patients leaving care; differential outcomes by race, insurance and geography |
| Stopping rule | Pre-agreed thresholds on the safety outcomes |
| Analysis | Agency epidemiology or independent evaluator, plan agreed before data exists |
| Publication | Results published whatever they show |
Will the pilot results be published?
We think pilot results should be published regardless of what they show, and we will agree to that in writing. The literature in this field is distorted by the pilots that were never written up, and a foundation running a program of this kind has no good reason to add to that.
Frequently asked questions
How long should an opioid program pilot run?
At minimum 12 months of follow-up after enrollment closes. Overdose is a low-frequency event, so a three-month pilot in a small population cannot detect a change in it, and any change it reported would be noise. The pilot has to run long enough to see the outcome you actually care about. See evaluating a pilot honestly.
Why does a pilot need a comparison group?
Without one, every finding is tangled up with time. Nationally, opioid-involved deaths fell from 82,851 in 2022 to 55,007 in 2024 with no help from anybody’s pilot. A matched population, a stepped-wedge rollout across sites, or at minimum a well-characterized pre-period separates what the program did from what was already happening. See the 2024 decline.
What is a stopping rule in a pilot?
It is the result, agreed in advance, that would cause the pilot to be halted early. In the outline here it means pre-agreed thresholds on the safety outcomes: patients leaving care and differences in outcomes by race, insurance and geography. A program with no stopping rule cannot fail, which means it cannot succeed either. See questions worth asking any vendor.
Who should be enrolled in the pilot?
Everyone in a defined geography who meets stated criteria, identified by a rule rather than by convenience. That includes the people who are hard to reach, because they are the population the program claims to help and the one it is most likely to fail. A small, cooperative group produces a pleasant story and no answer. See what a program is, and is not.
Sources
Every figure on this page is traceable to the source listed here.
- National Center for Health Statistics. VSRR Provisional Drug Overdose Death Counts (dataset xkb8-kh2a), 12-month-ending counts for the United States. Data accessed 20 August 2026. View source.