We must let the research question drive study methods

Many researchers, like ourselves, will remember being taught to select a study’s analysis method based on the characteristics of your dataset. A study with repeated measures data requires an analysis that accounts for the correlation between measurements, data that has been truncated because of the deaths of participants requires methods for informative censoring, and so on.

This makes sense. These features can lead to biased or incorrect results if they’re not handled appropriately during analysis. But if we’re not careful, this approach can cause more problems than it solves. This is because some analysis methods can change the research question to something that you don’t want to investigate.123 This particularly hit home for one of us a number of years ago, when we realised that the clever statistical method we’d used to handle missing data had in fact estimated how good the treatment would have been if no one in the study had died—a research question that’s of interest to no one.23

This may explain why we find the idea of estimands so appealing.12345678 Their purpose is to prevent exactly the type of mistake we had made. Estimands are a way of outlining the exact research question, including how factors like non-adherence or mortality inform the question. By outlining the research question in this way, estimands give us a target—the question our analysis should be answering.

We’re not alone in appreciating the clarity that estimands provide; their use has been adopted by medicine regulators around the world (including in the USA, UK, Europe, Egypt, Singapore, Saudi Arabia, Switzerland, and China9), and they are appearing both in reporting guidelines10 and study reports.111213 In workshops we’ve delivered, attendees are frequently positive about the value of estimands. But individuals have often expressed uncertainty about how they should be used and estimated. This is perhaps no surprise—the standard guidance on estimands (the ICH E9(R1) addendum8) is a technical document.

In our article, published in The BMJ, we bridge this gap by providing a non-technical explanation of estimands.14 We’ve included an explanation of technical terms, advice on choosing and describing estimands, motivating examples, and more.

But to us, the most important contribution is the description of how we applied the estimands framework to the FLO-ELA trial, which compares two methods of fluid delivery for patients undergoing emergency surgery.15 One of us helped design this trial and, while on the face of it the research question seems straightforward, it was anything but. We describe our thought process in deciding the specific research question using estimands; how this affected the statistical methods; and how the analysis that was originally proposed (intention-to-treat) was, in fact, answering exactly the wrong question for this particular trial.

Estimands are rapidly becoming an essential part of medical research. They serve to remind us of this important truth: the research question should drive our study methods, and not the other way around.