The challenge of successfully developing a drug and bringing it to market cannot be overstated. In fact, only 1 in 10,000 compounds make it to markets and costs in excess of $US 1 billion in doing so [1, 2]. This figure represents a huge upfront investment that must be managed well to increase the probability of success. Increasing constraints on resources in pharmaceutical research and development, in parallel, have made it more critical to develop timely and accurate forecasts for clinical trial timelines. These challenges, together with the increasing complexity of trial design, make accurate and efficient forecasting processes a key factor in the management and allocation of limited organizational resources to execute clinical trials.
This paper will involve a discussion of this drug development challenge with a focus on leveraging internal and external data assets to set expectations around clinical trial enrollment – the period in drug development often cited as the most variable and unpredictable. It will include a discussion of methods and processes that can be used to develop trial enrollment forecasts and conclude with sample performance measurements.
Drug Development Challenge
The hurdles to successful drug development are increasing. Trials are getting larger and more complex as regulatory requirements change [3]. Consolidation of companies, departments, and job roles also creates an imperative for allocating resources more efficiently, which often means doing more with less.
In the industry, these challenges frequently manifest themselves in the trial enrollment space. In allocating resources, trial management professionals must therefore balance trial complexity with subject and investigator availability, while considering the regulatory and competitive landscape. An in-depth understanding of the factors that influence trial enrollment is also a prerequisite. Table 1 contains a partial list of factors that directly impact trial enrollment.
Response to Challenges
Traditional approaches to trial enrollment planning had been based on intuition and prior experience, which can be valid, but can also vary considerably from person to person – resulting in poor predictability and inconsistency. Such complex mixture of influencers or predictors of trial performance requires more sophisticated approaches. The industry must therefore consider comprehensive solutions that integrate appropriate human resources, standard processes, and technology to address the challenges.
• Appropriate human resources to work on trials end to end. These resources (FTEs) should be empowered to make clinical trial timeline decisions based on their expertise and data-driven analyses. Cohan [4] endorses this approach, as he believes that decisions should be made by people who have the knowledge and data necessary to make them. He also pointed out the potential risks with having senior management making decisions about detailed processes they are not familiar with. According to Cohan [4], such decisions are best reserved for those lower down in the organization who are closer to the mechanics of the problem.
• Standard processes for assessing protocol feasibility. That is, assessing how likely a given protocol is to enroll by looking at the inclusion and exclusion criteria, treatment guidelines, procedures, epidemiology, and integration of insights from electronic health records where possible.
• Adoption of enrollment modeling and simulation technology to forecast enrollment. There are several enrollment forecasting tools available. These tools can be divided into two broad categories - early assumption-based enrollment modeling and in-life enrollment modeling. The type of enrollment forecasting tool selected should be driven by how much is known about a trial at the time. When little is known (during early study planning, for example), an assumption-based model should be used. When more granular information is known, such as a detailed list of participating countries and a final protocol, then an in-life model is more appropriate.
Prior to using these modeling and simulation technologies, important background research is needed. Once the protocol has been drafted and the study design, sample size, and targeted patient population have been defined, the dedicated FTEs should then scan the current clinical trial landscape for that particular disease area. The intent is to obtain an in-depth understanding of relevant historical trial performance and to derive modeling inputs that correspond to the factors that impact trial performance (Table 1). Some of this historical information can be obtained from internal sources, including clinical trial management systems and purchased proprietary data sources, while others are from public sources, such as clinicaltrials.gov and published study reports. This research informs the enrollment models.
Early Assumption-based Enrollment Modeling
At this early stage of protocol development, very little is known about the countries and sites that will participate in the trial. However, the team must plan early and establish timelines for completing the trial. To address this early planning need, Monte Carlo Simulation (MCS) technology can be applied. MCS is an analytical problem-solving tool that approaches a problem by generating multiple trial runs of the system being explored. It then generates a probabilistic distribution of possible outcomes or solutions based on those trial runs. These trial runs are called simulations.
MCS, as applied in trial enrollment forecasting, is used to help determine the optimum mix of operational parameters needed to enroll a study in a projected time period with a certain probability of success. These parameters include the recruitment rate, patients/site, number of sites, and screen failure ratio. To derive the enrollment period projection, a range is entered into the MCS application for each of the inputs described in Table 1. The application then randomly selects a value from the range provided for each input parameter and runs a virtual simulation of the trial. The standard practice is to run 1,000-5,000 simulations to represent the range of possible outcomes. The outcome of the simulations is then displayed as a distribution chart from which probabilities can be deducted.
Figure 1 is an enrollment period distribution chart, the main output of the MCS. The chart shows that this study has less than a 10% probability of completing enrollment in the 9 months this particular team was requesting. Without the appropriate tool or technology to flag major risks such as this one, a study like this would certainly have gone into rescue later on. That is, the team would need to initiate unplanned addition of new countries and sites or initiate costly advertising or other types of patient recruitment remediation campaigns.
To mitigate this risk, the team can then investigate changes that can be made to the factors that influence trial performance. Areas of exploration could include patient definition, study burden on patient and site, and planned geographic placement of the study.
In-Life Enrollment Modeling
Once a protocol is finalized and more information is known about the geographic placement of the study, one can then switch to an enrollment forecasting technology that is able to accept more granular operational inputs. This type of technology must be capable of using predictive analytics to provide real-time visibility into trial progress, which is critical to appropriate trial management. It should be able to project enrollment timelines based on initial assumptions from the early assumption-based model and geographic footprint of the study. The emphasis of the projection algorithm should then shift from planning assumptions to actual patient accrual rates as study enrollment progresses.
As the data are refreshed on a regular basis (daily or weekly), the tool should update the projections, based on its underlying algorithms and the incoming study data. It should then generate revised estimated trial completion dates, and recalibrate the forecasting model, effectively eliminating the need for manual aggregation and analysis of data in Excel or other systems.
Together, these tools can provide timely information and insights to make important clinical trial execution decisions. The MCS or early assumption-based model enables protocol refinement decisions, adjustments to trial execution plans, and go/no go decisions. The in-life enrollment forecasting technology equips trial managers with information to allocate resources appropriately, diagnose sources of trial delays, simulate recovery options, and implement timely mitigations to keep trials on track.
Performance Measurements
It is important to establish metrics to assess the performance of new processes and other types of organizational changes. Only then will one know if the intended objectives of the decision to change are being achieved [5]. To that end, Merck did an analysis of its internal trial operations performance prior to instituting processes and enrollment forecasting technologies such as the ones described here. The key finding from that analysis was that actual versus predicted clinical trial enrollment cycle times were too unpredictable and deemed unacceptable by senior management, as resources could not be optimally allocated to projects based on such poor predictability.
Results to date indicate that the organization is getting better at predicting clinical trial cycle time. Figure 2 is an illustration of predicted versus actual enrollment period for a subset of the late-stage portfolio that used the processes and technologies being discussed. In the first three studies, the organization was learning and refining the technology. For the remainder of the study, the predicted enrollment period was quite close to what actually occurred at the end of the study. For this particular dataset, 78% of the trials completed enrollment no more than four weeks after the projected enrollment period, a performance that the organization considers acceptable.
Organizations tend to develop metrics that can only be measured upon completion of a project. Such an approach, according to Armenakis, Harris, Cole, Fillmer and Self, does not account for adjustments that occur prior to completion, and suggested that interim metrics are needed to evaluate progress, so that course corrections can be made [6]. Given that some clinical trials can take months to several years to enroll, it is important to establish key performance indicators (KPIs) to assess at various time points. Figure 3 is an example of one of those KPIs. In this example, the randomization rates predicted at protocol approval correlate very well with the randomization rates observed toward the end of the trials for this subset of the portfolio with data at the time of the analysis (correlation coefficient > 0.8).
Conclusion
Improving clinical trial enrollment predictability, while reducing cycle time, requires a coordinated approach from multiple perspectives. Although this is a difficult problem to tackle, the consequences of suboptimal approaches create a compelling reason to invest in processes and technologies such as the ones discussed in this paper. These processes and technologies were intended to improve predictability, while reducing cycle time and improving productivity. They were designed to give the teams confidence in clinical trial cycle time projections, so that they could appropriately manage their trials, including allocation of trial resources. Based on these results, the tools appear to be achieving their objectives.
References
1. Tonkens, R. (2005). An overview of the drug development process. The Physician Executive, May-June.
2. Cook, J., Hunter, G., & Vernon, J. (2009). The future costs, risks, and rewards of drug development: The economics of pharmacogenomics. Pharmacoeconomics, 27 (5), 355-363.
3. Mullins, D., Whicher, D., Reese, E., & Tunis, S. (2010). Generating evidence for comparative effectiveness research using more pragmatic randomized controlled trials. PharmacoEconomics, 28 (10),969-976.
4. Cohan, P. (2007). When the blind lead. Business Strategy Review, 18(3), 65–70.
5. Stryker, P. (2001). How to analyze that problem: part II of a management exercise. Harvard business review on decision making (pp. 113-142). Cambridge, MA: Harvard Business School Press.
6. Armenakis, A., Harris, S., Cole, M., Fillmer, J., & Self, D. (2007). A top management team's reactions to organizational transformation: The diagnostic benefits of five key change sentiments. Journal of Change Management, 7(3/4), 273-290.
Otis Johnson brings over 11 years of experience in clinical R&D, with a focus on protocol development and optimizing clinical operations. He is currently part of the Global Trial Optimization (GTO) organization, a dedicated patient recruitment and protocol feasibility group within clinical research operations at Merck. Within GTO, he developed and now manages an informatics function responsible for the analytics that support study feasibility, enrollment projections, geographic footprint, and other parameters within which each study is run. Prior to taking on this role, he contributed to several process improvement initiatives within Merck, one of which led to the creation of the GTO organization.
He joined the clinical scientist group at Merck in 2000 where he co-authored protocols and clinical study reports. In this position, he also managed clinical supplies, site budgets, provided data management support, participated in the development and implementation of two spirometry systems, and managed a team of five clinical scientists supporting respiratory studies. He started out as a Research Technician in an immunology lab at New York University School of Medicine where he primarily worked on developing a lymphocyte proliferation assay. He has a B.S. in Medical Laboratory Sciences, a Master of Public Administration in Health Policy and Management from NYU, and is currently working on a Ph.D. in Management.