In Issue 2, I wrote about what a strong SAP includes, and Issue 3 covered sample size. This one is about what a reviewer actually checks.
During my years reviewing statistical analysis plans at the FDA, I checked the same things each time. That applied both to SAPs submitted before a study began and to those reviewed as part of an NDA or BLA. Could I recalculate the sponsor's sample size from what was written? Did the SAP define the estimand and state how intercurrent events would be handled? Were the analysis methods specified in enough detail to be reproduced? Were the missing data approach and multiplicity adjustments pre-specified and justified?
My recalculated sample size almost always matched the sponsor's, but not always. When it didn't, I alerted the sponsor. When I was reviewing a protocol before the study started, that sometimes meant the trial needed more patients than planned, and sometimes fewer. In each case, the difference could be traced to something the SAP stated incorrectly, unclearly, or not at all.
The SAP is where a trial's statistical decisions are written down. When it leaves one of them out, that gap becomes a question the sponsor will need to answer during review.
✅ What a Reviewer Needs to See
In SAPs that moved through review without avoidable questions, I consistently found the following:
A reproducible sample size. Every assumption was stated explicitly, with sources cited. That included the expected treatment difference, variability, event or response rates, and expected discontinuation and withdrawal. I could reconstruct the calculation independently. When my numbers matched, the sample size was confirmed from the SAP itself. When they didn't, I flagged the discrepancy. When the SAP didn't provide enough detail, I had no way to verify the calculation, and that became a question for the sponsor.
A defined estimand with an intercurrent event strategy. The estimand named the population, treatment, patient-level endpoint and population-level summary measure. It also stated how each intercurrent event (e.g., treatment discontinuation, rescue medication use and, depending on the reason, withdrawal from the study) would be handled, and why that choice fit the clinical question.
Statistical methods specified in enough detail to reproduce. A statistician who had never spoken with the sponsor's team could run the analysis from the SAP alone. The model, the covariates and the analysis population were all named.
A pre-specified strategy for handling intercurrent events and missing data. These are related but distinct concerns. Not every intercurrent event results in missing data, and missing data can occur independently of any intercurrent event — for example, when a patient misses a visit but returns for the next one. Withdrawal from the study always leaves data missing; depending on the reason for withdrawal, it may also be an intercurrent event. The SAP should address each separately: how each intercurrent event will be handled in the analysis, and how missing data will be managed regardless of their source. Both decisions should be made before unblinding and justified in the context of the trial. Some sponsors also documented whether missingness patterns differed between treatment arms. That answered questions I would otherwise have had to ask.
Multiplicity addressed explicitly. Where multiple endpoints were tested, the SAP named the testing hierarchy, alpha allocation and adjustment method before the data were seen.
Multiple sensitivity analyses, each targeting the same estimand as the primary. Sensitivity analyses don't change the clinical question. They test whether the primary result holds when the assumptions behind the primary analysis change — most often, the assumptions about missing data. The most complete submissions included several, such as tipping point analyses, pattern-mixture models or multiple imputation under different assumptions.
⚠️ What Leads to Review Questions
These patterns consistently led to information requests and added time to review:
A sample size section that can't be reproduced. Some assumptions are stated without sources. The treatment difference has no reference to supporting data, or the standard deviation has no explanation. If I couldn't reconstruct the calculation, I couldn't confirm the trial was powered as planned. The sponsor would then need to supply what was missing.
A vague or incomplete estimand. An estimand described in a single paragraph, without a strategy for each intercurrent event, leaves the clinical question unresolved. The analysis can't be checked against a question that isn't fully defined.
Statistical methods left open. "Appropriate methods will be applied" is not a pre-specified analysis. Naming a model without its covariates, or defining a population without the rationale, also leaves decisions unstated. Each one becomes a review question.
Intercurrent events and missing data handled without clear pre-specification. Generic imputation language, or a method chosen without justification, doesn't show how missing data will be managed or how they relate to the analysis strategy. Sometimes missingness also differed between treatment arms and the SAP didn't address it. In those cases, I had to ask whether the primary estimate was biased.
Sensitivity analyses that change the question. When an analysis changes the population, the intercurrent event strategy, or the summary measure, it is no longer a sensitivity analysis for the primary estimand — it is answering a different clinical question. A true sensitivity analysis targets the same estimand under different assumptions.
A single sensitivity analysis that confirms the primary result without testing less favorable assumptions.
Imputation methods chosen post hoc — especially when the SAP was amended after database lock without clear justification.
Multiplicity left unaddressed. Without a pre-specified hierarchy, alpha allocation or adjustment method, secondary endpoint results can't be interpreted at their nominal significance level. That holds even when the primary result is clear.
💡 Why All of This Matters Before You Submit
A SAP is not a document you write for the reviewer. It states in writing the clinical question you're asking and the analysis you'll use to answer it, decided before the data are seen. Sponsors who work through those decisions early go into submission with the answers already in the SAP. Otherwise, those answers end up in responses to information requests.
I've seen submissions move through review without delays because each of these elements was specified. I've also seen trials with clear efficacy results held up by rounds of questions because the SAP left key decisions open.
If your team is finalizing a SAP, I'd welcome the conversation — particularly smaller teams without a large in-house biostatistics group. Book a free 30-minute call.
Thank you for reading!
Lisa