✕
Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Feb 3;7(1):40.
doi: 10.1186/s40814-021-00770-x.

Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back!

M Lewis  1   2 , K Bromley  3   4 , C J Sutton  5 , G McCray  3   4 , H L Myers  4 , G A Lancaster  3   4
Affiliations

Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back!

M Lewis et al. Pilot Feasibility Stud. .

Abstract

Background: The current CONSORT guidelines for reporting pilot trials do not recommend hypothesis testing of clinical outcomes on the basis that a pilot trial is under-powered to detect such differences and this is the aim of the main trial. It states that primary evaluation should focus on descriptive analysis of feasibility/process outcomes (e.g. recruitment, adherence, treatment fidelity). Whilst the argument for not testing clinical outcomes is justifiable, the same does not necessarily apply to feasibility/process outcomes, where differences may be large and detectable with small samples. Moreover, there remains much ambiguity around sample size for pilot trials.

Methods: Many pilot trials adopt a 'traffic light' system for evaluating progression to the main trial determined by a set of criteria set up a priori. We construct a hypothesis testing approach for binary feasibility outcomes focused around this system that tests against being in the RED zone (unacceptable outcome) based on an expectation of being in the GREEN zone (acceptable outcome) and choose the sample size to give high power to reject being in the RED zone if the GREEN zone holds true. Pilot point estimates falling in the RED zone will be statistically non-significant and in the GREEN zone will be significant; the AMBER zone designates potentially acceptable outcome and statistical tests may be significant or non-significant.

Results: For example, in relation to treatment fidelity, if we assume the upper boundary of the RED zone is 50% and the lower boundary of the GREEN zone is 75% (designating unacceptable and acceptable treatment fidelity, respectively), the sample size required for analysis given 90% power and one-sided 5% alpha would be around n = 34 (intervention group alone). Observed treatment fidelity in the range of 0-17 participants (0-50%) will fall into the RED zone and be statistically non-significant, 18-25 (51-74%) fall into AMBER and may or may not be significant and 26-34 (75-100%) fall into GREEN and will be significant indicating acceptable fidelity.

Discussion: In general, several key process outcomes are assessed for progression to a main trial; a composite approach would require appraising the rules of progression across all these outcomes. This methodology provides a formal framework for hypothesis testing and sample size indication around process outcome evaluation for pilot RCTs.

Keywords: Outcome and process assessment; Pilots; Sample size, Statistics.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Fig. 1
Fig. 1
Illustration of power using the 1-tailed hypothesis testing against the traffic light signalling approach to pilot progression. E, observed point estimate; RUL, upper limit of RED zone; GLL, lower limit of GREEN zone; Ac, cut-off for statistical significance (at the 1-sided 5% level); α, type I error; β, type II error
Fig. 2
Fig. 2
Probability of traffic light given true underlying probability of an event using the example from Table 5 (i). Two plots are presented: a relating to normal approximation approach and b relating to binomial exact approach. Based on n = 200, RUL = 40 and GLL = 70
See this image and copyright information in PMC

References

    1. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. 2004;10(2):307–312. doi: 10.1111/j..2002.384.doc.x. - DOI - PubMed
    1. Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. Pilot Feasibility Stud. 2016;2:64. doi: 10.1186/s40814-016-0105-8. - DOI - PMC - PubMed
    1. McDonald AM, Knight RC, Campbell MK, Entwistle VA, Grant AM, Cook JA, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7:9. doi: 10.1186/1745-6215-7-9. - DOI - PMC - PubMed
    1. Sully BG, Julious SA, Nicholl J. A reinvestigation of recruitment to randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies. Trials. 2013;14:166. doi: 10.1186/1745-6215-14-166. - DOI - PMC - PubMed
    1. Walters SJ, Bonacho Dos Anjos Henriques-Cadby I, Bortolami O, Flight L, Hind D, Jacques RM, et al. Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open. 2017;7(3):e015276. doi: 10.1136/bmjopen-2016-015276. - DOI - PMC - PubMed

LinkOut - more resources