Decisions: the price of avoiding false wins
Every rule for saying "yes" trades false wins against missed drugs. What does protection against false wins cost in patients?
A. Two ways to be wrong
A regulator can approve a drug that does nothing or reject a good one. In one trial, the first is a false win. Approval itself needs substantial evidence (module 11). Which mistake is worse?
A false win is ... times worse
Approve only if we are at least 39/(39+1) = 97.5% sure the drug works.
We start with no opinion about the drug. Costs are made up.
Start with no opinion and look once: “approve if 97.5% sure” is the same rule as the one-sided 0.025 test in the simple bell-curve model used here. For yes/no outcomes it is only approximate, which is one more reason to check by simulation. It is also the same as treating a false win as 39 times worse than a missed drug. Interim looks, adaptation and informative priors break this match, which is why regulators ask for simulations.
B. The price of caution: loss of power
A stricter bar for "yes" protects against false alarms. It also makes real drugs harder to find.
Example
Patients
Alpha (one-sided)
Endpoints tested (Bonferroni)
Looks at the data (module 7)
C. What FDA asks for
FDA’s 2026 draft gives three ways to decide a trial succeeded. Read it at fda.gov.
Cap false wins
Default: at most a 2.5% (one-sided) chance of any false win across the primary and key secondary endpoints (module 4).
Judge what we believe after the trial
Start from what is already known, then ask how sure we are now.
Win if we are sure enough (say 97.5%) the drug beats placebo by a set amount.
Use a loss function
Set what each wrong decision costs, as in section A, and pick the decision with the lowest expected loss.
This is separate from FDA's benefit-risk review at approval.
D. Checking a rule with simulated trials
10,000 simulated trials per case. Old drug: 30% improve. Up to 100 patients per arm.
Chance the rule picks the new drug
Average patients per arm
Rules are set up differently, so this is not a ranking. Costs are made up.
E. Predict, then check
Takeaway
- Every rule for saying yes trades false wins against missed drugs and trial size.
- FDA's default bar: at most a 2.5% false win chance, one-sided.
- Simulated trials show what a rule really does: here 0.026 to 0.038 (the source aimed for 0.05). All are above FDA's 2.5% default because the source designed them for 5%. FDA's Bayesian draft allows other routes too.