Evaluating a pilot honestly

A pilot that cannot produce a negative result is not a pilot. It is a demonstration, and there is a shelf of them in every health department in the country.

What makes a pilot study uninformative?

1. A convenience population. Enrolling whoever is easiest to reach guarantees the program works, because the people it would fail were never in it. A pilot population should be defined by a rule that includes the hard cases.
2. No comparison. This one is acute right now: opioid-involved overdose deaths fell nationally from 82,851 in 2022 to 55,007 in 2024. Any uncontrolled program running in that window will report a large improvement it had nothing to do with.
3. Too short. Overdose is a low-frequency event. A three-month pilot in a small population cannot detect a change in it, and reporting one is reporting noise.
4. Measures chosen after the data exists. Which turns any dataset into a success.

How should an opioid program pilot be evaluated?

• A stepped-wedge design, where sites start at staggered times. Everyone gets the program eventually, which is usually what makes it acceptable, and the staggering supplies the comparison.
• Pre-registered measures, including the safety ones from measurement and evaluation.
• A stopping rule agreed in advance — what result would halt this early. A program with no stopping rule cannot fail, and therefore cannot succeed either.
• An independent evaluator with access to the underlying data, not just to the vendor’s summary.
• Publication regardless of result.

Which measures should an honest pilot report?

A pilot reporting only favorable measures is not reporting. The ones to insist on are the ones that would embarrass the program: patients leaving care, therapy gaps, enrollment skewed toward the easy cases, and differential outcomes by race, insurance status or geography. That last one will not announce itself — it has to be looked for on purpose.

Why we want this

Partly integrity, and partly self-interest. A program that has only ever been evaluated by itself is worth nothing in a legislative hearing or a competitive procurement. The first agency to evaluate this properly, and publish, does the program a larger favor than a decade of favorable case studies.

More: pilot design.

Frequently asked questions

How long should a pilot study run?

Long enough, and large enough, to detect a change in the outcome it claims to move. Overdose is a low-frequency event, so a three-month pilot in a small population cannot detect a change in it, and any change it reports is noise. The length and size of the pilot should be fixed from the outcome before the pilot starts.

What is a stepped-wedge design?

A stepped-wedge design starts the program at different sites at staggered times. Every site gets the program eventually, which is usually what makes the design acceptable to an agency and its communities. The sites still waiting supply the comparison for the sites already running, so the pilot can separate the program’s effect from changes that were happening anyway.

Why does a pilot need a comparison group?

Because outcomes move on their own. Opioid-involved overdose deaths fell nationally from 82,851 in 2022 to 55,007 in 2024. Any uncontrolled program running in that window would report a large improvement it had nothing to do with. A comparison, such as the sites still waiting in a stepped-wedge design, shows how much of the change belongs to the program.

What is a stopping rule in a pilot?

A stopping rule is the result, agreed in advance, that would halt the pilot early. It is written down before any data arrives, alongside the pre-registered measures, including the safety measures. A program with no stopping rule cannot fail, and a program that cannot fail cannot succeed either, because no result would ever count against it.

Who should evaluate a pilot program?

An independent evaluator with access to the underlying data, not just the vendor’s summary. The results should be published regardless of what they show. A program evaluated only by itself carries no weight in a legislative hearing or a competitive procurement, which is why IntellaRx wants the first agency that runs it to evaluate it properly and publish.