- Tufts Center for the Study of Drug Development
Changing or amending a protocol after it has been finalized is a common and widely executed practice. Amendments are implemented for many reasons including the introduction of new standards of care; revisions to eligibility criteria to stimulate patient recruitment; changes to medications permitted before and during the clinical trial; the availability of new safety data; and requests from ethical review boards and regulatory agencies.1-3
Drug developers have long been aware of the burden of protocol amendments on clinical trials. This burden is borne not only operationally, as protocol changes are time and resource intensive regardless of whether they are unplanned and unbudgeted, but also scientifically as they can decrease the validity of clinical trial findings and challenge research integrity.1-2
In 1996, a pioneer study assessing the prevalence of protocol amend-ments found that 45% of clinical trial protocols submitted to the Royal College of General Practitioners in the United Kingdom were amended mainly due to safety concerns.3 The study also found that there were on average 1.5 amendments per protocol.3
The Tufts Center for the Study of Drug Development (Tufts CSDD) has conducted several studies to understand the design elements driving the need to amend a protocol. A 2008 Tufts CSDD study found that protocols that were relatively more complex had on average one more amendment than did those that are relatively less complex (three amendments per protocol as compared to two).4 In 2013, Tufts CSDD conducted a study assessing the types of data collected in clinical trials and found that clinical trials with protocol amendments had a higher proportion of endpoints supporting tertiary and exploratory endpoints as well as a higher proportion of procedure supporting those endpoints.5
Tufts CSDD has also been examining the impact of protocol amendments on clinical study performance and cost. In 2007, Tufts CSDD solicited anecdotal reports from sponsor companies and found that Phase II and Phase III protocols have on average three and seven amendments respectively and can cost anywhere between $250,000 and $450,000 per amendment.6 In 2011, Tufts CSDD, in collaboration with seventeen biopharmaceutical companies conducted a robust assessment.7 Nearly six-out-of-ten (59%) completed studies had at least one amendment, with Phase II and Phase III studies having on average 2.7 and 3.6 amendments per protocol respectively. The 2011 study also found that the mean direct cost of a protocol amendment is approximately $454,000 (median: $72,300 per amendment). Lastly, the study created a framework to classify an amendment by whether it was ‘Avoidable’ (e.g. difficulties recruiting) or not (e.g. results of regulatory requests). The study determined that approximately 33% of all protocol amendments were classified by their respective protocol authors as avoidable.
After the results of the study were published, biopharmaceutical companies implemented a number of initiatives to optimize protocol design. In 2015, Tufts CSDD, in collaboration with fifteen biopharm-aceutical companies, conducted an update to the 2011 study. This recent 2015 study also looked more robustly at the associations between protocol amendments and clinical trial performance.8 The 2015 study also included a qualitative component where sponsor company representatives were interviewed to assess management practices to address protocol amendments specifically.
Later in 2015, Tufts CSDD began performing custom diagnostic studies of individual sponsor company protocol design and amend-ment experience. At that time, Tufts CSDD worked with a top 10 biopharmaceutical company to characterize its protocol design practices and conduct a root cause analysis on amendment implementation factors impacting study duration cycle times (2015 Case Study).
The Prevalence and Cost of Protocol Amendments are Still High
For the 2015 study, Tufts CSDD collected trial performance measures on 836 trials (completed between 2010 and 2013) and cost data on 52 protocols that were ongoing in 2015. Twelve of the fifteen biopharmaceutical companies participated in a survey assessing their amendment management practices.
Tufts CSDD determined that protocol amendment prevalence decreased slightly to 57%, as did the mean number of amendments per Phase II and Phase III trials (down to 2.2 and 2.3 amendments per trial respectively).8 The study also found that the proportion of protocol amendments classified as avoidable increased to 45%, and that the cost of one protocol amendment increased substantially (possibly due to more companies tracking these data). Table 1 summarizes the mean number and cost of a protocol amendment per trial across all studies.
Table 1. Comparison of Mean Number and Cost of Protocol Amendments Per Trial
The Impact of Protocol Amendments on Study Timelines
In the 2015 study, Tufts CSDD also found statistically significant associations between having at least one protocol amendment and the study timelines, even after adjusting for phase and therapeutic area. Trials with no protocol amendments took on average 330 days from protocol approval to last patient first visit, whereas protocols with amendments took on average 510 days, a 180-day (i.e. six-month) difference (p<0.0001).8 The study also found differences in study timelines after study conduct; the mean duration from last patient last visit to database lock was 140 days as compared to 230 days for protocols with at least one amendment, a 90-day (i.e. three-month) difference (P=0.0068).
Company Survey Responses: Initiatives to Reduce Protocol Amendments
In the 2015 study, Tufts CSDD found that seven of twelve companies in the study conduct systematic assessments of the number and causes of protocol amendments in an attempt to create better, less costly, protocols. These initiatives started as early as January 2013 and as late as March 2015. Six of eleven companies also have begun budgeting for roughly two prospective protocol amendments per study. Contingency funds are then set aside if an amendment seems likely.
In addition to conducting systematic assessments on past studies, respondents note that their companies are including a median of 4.5 processes and practices to reduce the number of protocol amendments on future studies. Figure 1 lists the main processes companies have adopted to reduce protocol amendments.
Figure 1.Companies soliciting feedback from investigative sites – one of the most common methods adopted – report that it has been one of the most effective approaches. Better up front protocol planning was the most widely adopted approach and it was ranked second by sponsor companies in terms of impact.
The company with the lowest prevalence of protocol amendments (62% of the company’s protocols do not have protocol amendments) indicated that its primary strategy has been to leveraged a full partnership with one preferred contract research organization (CRO). That CRO now participates in protocol development planning, protocol design, and protocol feasibility given the CRO’s expertise in clinical trial operations. The sponsor company also budgets for amendments only in cases where they anticipate that an amendment will be required once more information about the study drug is known, such as a maximum tolerated dose study.
The Benefits of Systematic Assessments and Root Cause Analyses
Five of the seven companies conducting systematic assessments of the number and causes of protocol amendments indicated that reducing amendments specifically has resulted in fewer disruptions in the study. Companies report conducting these assessments internally, in collaboration with their CRO partners, and externally with consultants and with Tufts CSDD.
Companies have noted that conducting root cause analyses and systematic assessments have resulted in the refinement and design of the development planning and protocol design practices. Examples include creating a formal metric assessing protocol amendment prevalence, removing content from the protocol that was driving the amendment but not essential to the study, and adopting patient feedback into the study protocol.
A 2015 Case Study: Analyzing Protocol Design and Protocol Amendment Prevalence and Causes Yielded Changes in Standard Operating Procedures For One Large Pharmaceutical Company
In its 2015 Case Study, Tufts CSDD performed a number of diagnostic tests comparing a sponsor company’s protocol design practices and experiences with an industry benchmark. The diagnostic tests included an assessment of the protocols’ design and executional characteristics (e.g., the number of unique and total procedures, the number and type of study endpoints, the proportion of procedures supporting tertiary and exploratory endpoints, the number of eligibility criteria) as well protocol performance measures (e.g., screen failure and drop-out rates; the number of protocol amendments; study timelines and milestones).
Results of the diagnostics and root cause analyses
The results of this case study indicated that the sponsor company had more amendments per Phase II and Phase IV protocols than did the benchmark. The analysis demonstrated that clinical trials with amendments were close to two years longer than the sponsor company’s clinical trials without amendments suggesting opportunities to improve the execution and management of the amendments. A root cause analysis used both quantitative and qualitative (i.e., interviews with clinical teams) methods to probe more deeply into protocol design characteristics and the decisions behind them. The root cause assessment found that the sponsor company’s protocol designs were much more complex than the industry benchmark due to a number of clinical team decisions and protocol authoring practices.
Senior management honed in on key process improvements in response to the root cause analysis
Results from the root cause analysis helped senior management differentiate high and low value protocol amendments. Senior management saw examples of protocol amendments that were used strategically (e.g., by adding specific patient populations to ongoing studies, which saved the company time and the cost of conducting a separate clinical trial) versus those that were avoidable and occurred passively.
The root cause analysis also helped determine which processes required modification. The analysis demonstrated that therapeutic areas newer to the company contained protocols with more amendments. Additionally, the company learned that the definition of what required a protocol amendment varied across clinical teams. The company is now in the process of creating training modules for clinical teams to apply the findings from this diagnostic and root cause assessment.
Having learned that protocol amendments yield downstream effects on timelines that are longer than originally anticipated, the company is now incorporating a more vigilant pre-planning process. Their standard operating procedures now incorporate complexity measures so that newly written protocols are compared to industry benchmarks. Additionally, the sponsor company will leverage its feasibility review process by including more stakeholders and the company will set more ambitious targets for clinical teams.
In a few years, the company will conduct a follow-up assessment to determine how these process improvements have impacted protocol design, study timelines, and study costs.
Best Practices in Protocol Design by Reducing Protocol Amendments
Many companies have come forward with strategies in place to create better protocols that are more efficient and less costly.9-10 Companies such as Amgen, Pfizer, GlaxoSmithKlein, Eli Lilly and EMD Serono all have initiatives in place that involve diligent study planning. One such strategy is to vet clinical trial protocols with feasibility review committees that include all stakeholders, such as sites and patients. Another strategy involves keeping a protocol diagnostics database that includes when amendments occur, the cause of the amendment, and the associated costs, changes, and timeline disruptions. As shown in the case study, these data help companies focus on what aspects of study development need improvement.
Companies are also creating robust models and data analytics to facilitate scenario planning and protocol simulation to assess patient populations, look for trends in protocol deviations, and to better anticipate requests from regulatory agencies.
Additionally, consortia such as TransCelerate and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) have created standard protocol templates and checklists to aid clinical teams in creating more streamlined and feasibly executed protocol designs.
Although drug developers have been aware of the burden of protocol amendments on clinical trial operations and study validity for decades, only recently has data become available to indicate how expensive and disruptive poor protocol design can be on clinical development performance and cost. Now that industry groups and companies are tracking these data more systematically, industry and company-specific improvements in study design and processes may ultimately lead to a decrease in protocol amendment incidence. This, ultimately, will lead to the positive downstream effect of bringing medicines to patients more quickly.
References
- Chow S-C, Yang L-Y, Lu Y. Some Controversial Issues in Clinical Trials. Drug Information Journal 2011;45(2):163-74 doi: doi:10.1177/009286151104500211[published Online First: Epub Date]|.
- Lösch C, Neuhäuser M. The statistical analysis of a clinical trial when a protocol amendment changed the inclusion criteria. BMC medical research methodology 2008;8:16-16 doi: 10.1186/1471-2288-8-16[published Online First: Epub Date]|.
- Wise P, Drury M. Pharmaceutical trials in general practice: the first 100 protocols. An audit by the clinical research ethics committee of the Royal College of General Practitioners. Bmj 1996;313(7067):1245-48 doi: 10.1136/bmj.313.7067.1245[published Online First: Epub Date]|.
- Getz KA, Wenger J, Campo RA, Seguine ES, Kaitin KI. Assessing the impact of protocol design changes on clinical trial performance. Am J Ther 2008;15(5):450-7 doi: 10.1097/MJT.0b013e31816b9027[published Online First: Epub Date]|.
- Stergiopoulos S, Surgeon L, Getz KA. Quantifying the Magnitude and Cost of Collecting Nice-to-Have Clinical Trial Data. Chimmica Oggi - Chemistry Today 2016;34(6):50-53
- Analysis of the protocol design process. Clinical Research Roundtable Symposia; April 18, 2007; 2007; Boston, MA.
- Getz KA, Zuckerman R, Cropp AB, Hindle AL, Krauss R, Kaitin KI. Measuring the Incidence, Causes, and Repercussions of Protocol Amendments. Drug Information Journal 2011;45(3):265-75 doi: doi:10.1177/009286151104500307[published Online First: Epub Date]|.
- Getz K, Stergiopoulos S, Short M, et al. The Impact of Protocol Amendments on Clinical Trial Performance and Cost. Therapeutic Innovation & Regulatory Science 2016 doi: 10.1177/2168479016632271[published Online First: Epub Date]|.
- Getz KA. Acknowledging Cycle Time Impact from Protocol Amendments. Applied Clinical Trials. New Jersey: Advanstar, 2016.
- Korieth K. Facing Protocol Amendments Head-On: Cycle Time and Cost IMpact SHighing Light on Avoidable Amendments. The Centerwatch Monthly. Boston, MA: Centerwatch, 2016:6.
Ms. Stergiopoulos manages multi-sponsored and grant funded research projects at Tufts CSDD. She has experience conducting research on pharmaceutical industry practices and trends affecting pharmacovigilance, non-clinical drug development, pharmaceutical outsourcing practices, cycle time metrics, resource management, and protocol design. She has also been a speaker at conferences and has published articles in peer-reviewed and trade journals. Prior to joining Tufts CSDD, Ms. Stergiopoulos was a research associate at The Brattle Group and a researcher at Massachusetts General Hospital. She holds a BA from Brandeis University, and an MS and MPH from Tufts University.
Mr. Getz is an internationally recognized expert on R&D and clinical trial management practices and trends, the global investigative site landscape, site management and patient recruitment and retention practices, and the worldwide market for outsourcing clinical research functions. Mr. Getz’s research studies on protocol design complexity and clinical research efficiency and effectiveness, conducted over the past two decades, are considered by many in the research-based life sciences industry to be pioneering work. His 20+ years of original research benchmarking R&D management practices, global outsourcing and the investigative site landscape have contributed to industry-wide understanding of these critical markets and to improvements in management strategy and execution.