Kenneth Getz Executive Director and Professor, Tufts Center for the Study of Drug Development Tufts University School of Medicine
Protocol deviations and amendments are a common occurrence in clinical trials. Indeed, sponsor companies, contract research organizations (CROs) and investigative sites accept them as a disruptive though necessary problem that must be managed. But it is perhaps more meaningful to view deviations and amendments as solutions to underlying problems associated with the design and feasibility of the clinical trial protocol. And they are a very costly solution.
Protocol amendments in particular are highly disruptive, unplanned and unbudgeted. Research conducted by my team at the Tufts Center for the Study of Drug Development (Tufts CSDD) in 2018 found that a single amendment in a typical phase III protocol adds three months of unplanned time to a clinical trial and more than half-a-million dollars, on average, in direct costs to implement. Other costs that are harder to quantify include the impact that deviations and amendments have on patient retention; investigative site commitment; and the completeness, accuracy, and reliability of the study data.
Substantial amendments are defined as any major change to a protocol on a global level requiring suspending patient enrollment; obtaining internal approval followed by approval from an ethical review board or regulatory authority; and then re-consenting study volunteers. Substantial amendments are implemented for a variety of reasons, most notably to relax eligibility requirements because it is too difficult to recruit volunteers or to accommodate a significant change in the protocol strategy.
Less is known about the magnitude, cost and impact of protocol deviations. They are changes generally executed by the investigative site while the clinical trial is underway in an effort to make it easier for study volunteers to comply with the schedule of visits or adhere to study medication administration requirements. Study staff may also deviate from the protocol when they’re having difficulties finding eligible patients or when they are unable to easily perform procedures dictated by the protocol. Protocol deviations and amendments are now receiving increased attention as part of a broader, industry-wide effort to improve protocol design and execution and to reign in clinical trial costs.
In addition to internal initiatives implemented by sponsor and CRO companies, consortia and collaborative groups (e.g., Drug Information Association’s Good Clinical Practice and Quality Assurance community; TransCelerate BioPharma) have developed consensus definitions, frameworks and strategies to help prevent and manage them.
To inform these efforts, and given the limited data available that characterizes the incidence of protocol deviations and amendments, Tufts CSDD conducted a study in 2021.
Data from this study was provided by twenty major and mid-sized pharmaceutical companies and CROs. Clinical teams from each company gathered data on recently completed protocols or those that had locked the database before December 31st, 2019. CROs participating in the study gathered protocol data specifically from client companies that were participating sponsors in the working group. The teams from each company coded the data on their own protocols.
We collected data on protocol design variables (e.g., number of endpoints, procedures performed, patients enrolled) and performance variables (e.g., study timelines, recruitment and retention rates, deviations and amendments). For study timelines, we gathered planned and actual durations. The data collection process followed an approach that Tufts CSDD has been using to evaluate trends in, and the impact of, protocol design practices since 2008.
The study results show that protocols across all clinical phases have a higher average number of substantial amendments in the 2018- 2020 period compared to those from an earlier 2013-2015 period. Nearly eight-out-of-ten (78%) phase II protocols and 69% of phase III protocols have at least one substantial protocol amendment and they average 2.7 and 3.3 per protocol respectively. Looking at all active FDA-regulated phase II and III clinical trials in the aggregate, Tufts CSDD estimates that pharmaceutical and biotechnology companies are spending $7 - $8 billion ($US) annually to implement protocol amendments.
More complex protocol designs in general are associated with a higher average number of substantial amendments. Oncology, the largest segment of clinical trial activity in the drug development pipeline typically has the most complex study designs. The incidence of protocol amendments is significantly higher. Phase II protocols in oncology have nearly twice the average number of substantial amendments than do non-oncology protocols. And phase III studies average a 40% higher number of amendments per protocol.
In rare diseases, phase II protocols average 4.3 substantial amendments, nearly 65% more than the average for non-rare diseases. In phase III, rare disease protocols average 3.8 substantial amendments, 19% higher than non-rare diseases.
Each phase III protocol averages nearly 120 deviations involving approximately one-third of all patients participating in the clinical trial. Phase I protocols have the lowest average number of deviations (8.7) involving half the proportion of patients than those observed in phase II and III protocols. Across all phases, the variation around the mean number of deviations per protocol is very high underscoring the challenge of anticipating and implementing changes at the investigative site while the clinical trial is underway.
Again, in oncology, the number and proportion of patients with deviations are higher. Phase II and III oncology protocols average 30% more deviations than do non-oncology protocols. And protocol deviations in phase II and III oncology clinical trials involve four-out-of- ten patients, nearly double the proportion observed in non-oncology protocols. Rare disease protocols, on the other hand, average a lower number of deviations among a smaller proportion of patients than do non-rare disease protocols. Protocols associated with a lower average number of substantial amendments tend to have a larger delta between ‘plan’ and ‘actual’ study initiation durations. This suggests that taking more time upfront during the study start-up period may be contributing to fewer downstream changes and disruptions.
Protocols with a high mean number of substantial amendments add an average of three weeks beyond the planned treatment duration (FPFV to LPLV) whereas the actual durations were earlier-than-plan for protocols with a low mean number of substantial amendments. Overall, protocols averaged 14.8 days longer than planned study close-out durations though protocols with the highest average number of substantial amendments had actual durations six times longer than plan, on average, than those with a low mean number of amendments.
Interestingly, in our study, full-service outsourcing of phase II and III protocols was associated with a higher average number of deviations and substantial amendments per protocol. The average number of deviations per protocol was 64% higher and the proportion of patients involved with these deviations was 71% higher than those protocols managed by internal teams.
These observed increases are likely a function of many factors including the relative difficulty of protocols typically outsourced and the higher relative number of change orders that sponsors report in clinical trials managed by CROs.
In the coming months, Tufts CSDD is planning to bring together 20 and 30 biopharmaceutical companies and (CROs) to participate in a new working group study looking at updating and expanding our understanding of protocol amendments and how to optimize their implementation and management. Specifically, this study will (1) update benchmark data gathered more than five years ago to capture trends in the causes and impact of protocol amendments; (2) gather more granular data on the direct and indirect cost and time required to implement protocol amendments; and (3) collect data on not only substantial, but also country-specific, protocol amendments and assess those that are avoidable and unavoidable. The findings from this study will offer invaluable insights into optimizing the implementation and management of protocol amendments. We anticipate that the results will generate valuable time- and cost-saving insights.
Sponsors and CROs are rallying to reduce the effects of protocol deviations and amendments and ultimately improve protocol quality, executional feasibility and efficiency. The anticipated improvements in study performance and cost could not come at a better time, given rapid growth in the scientific and executional complexity of protocol designs and growing interest in supporting more convenient and accessible clinical trials for patients.