Randomized controlled clinical trials are the gold standard for understanding treatment effects and safety profiles. However, there are situations where under a frequentist framework this gold standard is not possible. For example, in rare diseases it could take years to recruit enough patients to run a well-powered, randomized control trial; whilst time or resource constraints can hinder other trial’s feasibility, and ethical considerations may make control arms inappropriate.
One potential solution to quantifying uncertainty in the face of such issues lies with the use of Bayesian statistics and quantifying prior beliefs through expert opinion. Expert prior elicitation follows an interview process in which experts are asked a series of questions regarding their beliefs about, for example, a treatment effect. Based on the experts’ responses, a probability distribution can be derived which reflects what the experts believe as well as the uncertainty of that belief.
Options for Creating a Prior
Non-informative priors are data starting points with minimal or no prior belief giving effectively equal weighting to all potential options, such as uniform or normal distributions that have been selected to cover an extremely wide variance of possibility. These priors give no information with regards to the treatment effect, safety profile, or other measurements being considered. Non-informative priors can be very useful starting points in Bayesian trials which expect to have multiple interim analyses where the new data from within the trial updates the latest prior as the trial goes on. It can also be useful if there is extremely limited understanding of the measurement in question and there is a desire to avoid any sway from the prior.
Comparatively, informative priors can be created through historic data, expert elicitation or a blended method of the two, taking the belief or previous information available to give an idea of the expected treatment effect, safety profile or other measurements. This can be useful to give a starting position relating to expected outcomes which are updated or confirmed throughout the trial.
Elicitation is the act of creating priors from data or opinions from experts. In the clinical trial setting, data is often previous trials or scientific theory where historic information has been collected and published. This information can be collated and modelled directly whereas clinicians are usually the go-to experts with regards to the information they possess and their ability to interpret the information they are given. Statisticians are used to help the experts turn that information into mathematical models for use as a basis for clinical trial decision making. Often it is more feasible to use expert elicitation, for example, when there is limited or contrasting historic information available, experts can be used to establish a view, whilst the direct mathematical model may lead to a mixed and confused result.
The process of eliciting from experts is performed through iterative questioning and analyzing. Experts are provided with a complete review of available information before being asked questions relating to the measurement of interest, which is collated into their individual elicitation (a distribution of their belief). Behavioral aggregation allows the individual elicitations to be combined with other experts and discussed as a group by the experts. After discussion they then can choose to update their individual elicitations before they are combined and discussed again until experts are happy with the group consensus.
It is, however, important to understand the impact of bias within such calculations and elicitations. This can be introduced in many ways such as how questions are framed and how information is provided to the experts, as well as how interaction and discussion between experts is facilitated.
It therefore falls upon the statisticians and facilitators to be aware of potential biases and reduce them as much as possible during the elicitation session.
A Framework for Successful Elicitation
To ensure the resultant prior from an elicitation is of good quality, good preparation before and during the session needs to occur. Once the decision to conduct elicitation has been made, it is crucial to clearly frame the problem, a clear endpoint that matches the question of interest within the related study must be crafted. Ensuring the patient population of interest and what measurement system is used allows comparability between the elicited prior and the trial. The next step is selecting the experts and providing the evidence dossier. This ensures all experts have the same, unbiased information and everyone is basing their beliefs on the same level of evidence. Training is also critical to ensure experts understand statistical distributions and methods involved in collection of their beliefs. The evidence dossier should encompass all information on the endpoint in the related field, where possible this should focus on the treatment/ comparison of interest as well to be more relevant to the discussion. Trials included within the dossier should be clearly referenced and contextualized with their populations, treatment schedules and results including where this differs from the trial you wish to perform. The dossier should be sent to experts ahead of time to allow them to familiarize themselves with all relevant information as well as give them a chance to flag other trials that should potentially be included to ensure all experts have seen all information.
During elicitation, experts’ opinions are translated using methods such as roulette or quartile, where discussion of means and extreme values can be shown graphically. These graphical representations along with further questioning from the statistician collating their opinions allows experts to fully explore what they are suggesting with their individual priors. These individual priors are then brought forward and discussed as a group alongside their combined beliefs so that each expert can explain their understanding of the data and weigh this up against others’ understanding, which will be influenced by their individual experience and knowledge within the field. The facilitator encourages experts to bring forward arguments for their beliefs and then step back, listen to the comments and arguments from others, and make a rational decision based on what they heard. Once discussions are held, each individual has the option to reconsider their individual priors before another round of discussion is had on these updated views. The aim is to elicit a single aggregate prior which represents the collective belief of experts based on the discussion of their individual priors, with as many rounds of elicitation and group discussion as needed.
Specific elicitation software can be used to ensure that the information gathered from experts can be clearly visualized, allowing for easier interpretation and analysis by the experts both at an individual and a combined level. Frameworks such as the SHeffield ELicitation Framework (SHELF), a software package with associated documentation and templates to carry out elicitation, or other bespoke programs such as the elicitation software with associated templates we have produced at Phastar are vital to ensure clarity and reproducibility when producing priors.
Once elicitation has taken place, it must be documented effectively. This ensures you have a record of what has led to the prior.
Challenges with Elicitation
There are many potential challenges within elicitation. From the start it is important to understand that the question needs to be clear but may not be able to be elicited directly. Consider an endpoint comparing the efficacy of two drugs, it may be necessary to first elicit the efficacy of the comparator drug (where more information is likely known) and then from that starting point elicit the expected efficacy increase of the trial drug. There are also issues of potential bias, asking “how much better drug A is expected to perform versus drug B” becomes a leading question assuming that drug A must be better than drug B, whereas there is always the potential it is not. The evidence dossier must be complete and contextual to ensure a fair comparison, and that data is not missing, otherwise selection bias may lead elicitations to be extreme in a specific direction.
During the meeting itself it is important to avoid group think and dominant voices during discussions. The key is effective facilitation ensuring all participants have time to have their say and not feel intimidated by others within the group’s perceived seniority. Experts may also be over optimistic, especially in early rounds of elicitation, leading them to produce elicitations that are more what they want to see rather than what the evidence suggests. It is crucial to engage the right experts with broad knowledge and to keep bringing discussions back to the evidence – focusing on facts rather than speculation.
Examples of Expert Elicited Priors for Rare Disease/Small Population Trials
There are several examples in the literature of rare disease and small population trials that have used elicitation to help their ability to run on the limited sample sizes that are feasible to recruit.
One trial, TREAT,1 was a randomized non-inferiority trial in patients with severe therapy resistant asthma, which whilst a common disorder is an extremely small subpopulation with an expected recruitment time of over 3 years for a population of only 150 patients. Keeping numbers low would ensure plausible recruitment and timelines for the trial. Eight experts in childhood respiratory disease were brought together to elicit their opinions on expected efficacy of the more familiar therapy of Omalizumab and the expected difference in outcome of a patient had they had Mepolizumab rather than Omalizumab. Information on trials for Omalizumab in the childhood population, comparisons between the two drugs in adult populations, and case studies in Mepolizumab in the childhood population were used in the evidence dossier. By producing a prior and having the trial run within a Bayesian framework, the plausible 150 patients would have enough information to produce an acceptable posterior distribution for realistic conclusions to be drawn.
Another trial, APRICOT,2,3 was a randomized adaptive trial looking at Anakinra as a treatment for palmoplantar pustulosis and its impact on the PPPASI score of patients. This trial was performed under a frequentist framework, but an elicitation was performed to allow for a Bayesian re-analysis. As the trial was approaching final analysis, elicited expectations on the primary endpoint from five primary investigators on the trial were used to re-run the whole trial in the Bayesian framework and conclude how this would change the outcome. An evidence dossier of all related information was given to the investigators who were still blinded to ensure they had enough information for the discussion. The overall result was similar to what was expected in the frequentist setting with the added benefit of allowing interpretation of the posterior distribution to inform any future trials. Since the trial had a very limited sample size, confirmation of the result using a Bayesian reanalysis allowed for checks whether the limited sample size was having a notable impact on the final result.
Another trial in Juvenile Localized Scleroderma (JLS),4 a rare chronic inflammatory disease with a relatively low incidence of 3.4 per 10,000 children, wanted to compare current main line treatment, Methotrexate (MTX), which is not well tolerated with a potential alternative, Mycophenolate Mofetil (MMF) which was thought to be more tolerable but there was very limited evidence around efficacy in this disease and population. A frequentist, non-inferiority trial would require 320 patients which would take 15 years to recruit. Instead, researchers attempted to capture the current understanding of the treatments in a prior to form the basis of a Bayesian clinical trial. They asked three open questions: the best steroid regime to use, the proportion of patients for whom MTX/MMF is successful and the proportion of patients that tolerate MTX/MMF. This information was turned into a prior and used to calculate the necessary sample size to run the trial in a Bayesian framework. This reduced the necessary sample size from 320 to 240, a large reduction but still unfeasibly large and thus they concluded not to run the trial.
One trial Phastar consulted on the design and analysis for utilized the Bayesian Optimal Interval Phase I/II (BOIN12) method within an early phase dose finding oncology trial.5 As part of this trial utility scores, defining utility of health outcomes on a scale of 0-100, needed to be defined. A direct elicitation was gathered from six experts with them defining the scores for different outcomes. These estimated scores were combined and averaged to determine the combined elicitation scores for each outcome. The experts were then shown these averaged scores to decide if it was a fair representation of their views. Experts were also told the standard deviation between the scores to confirm whether they felt the range of scores for any specific outcome had too varied a view and needed further discussion. This highlighted areas which could be opened for discussion where there was lower confidence. By determining the utility scores by the medical experts, the BOIN12 algorithm can appropriately determine when to change dose levels during the trial.
Bayesian Elicitation Tools
These examples highlight the potential uses of elicitation, but also the importance of a well-run elicitation with clear endpoints assisted by the use of tailored software. For example, the elicitation tool produced at Phastar allows experts to visualize their opinion graphically and allows the expert to adjust until they are confident the visualization reflects their opinion.
Once all the experts have completed their individual elicitation, software can also be used to aid the discussion phase. For example, an aggregation tab can overlay all the distributions from individual illustrations, as well as providing a single aggregate distribution. Individual elicitations can also be weighted to allow for greater expertise or knowledge. The univariate mixture distribution can also be fitted to the chosen distribution to provide an easy comparison of priors and a simpler ability to combine into one overarching distribution.
If all the experts are happy and there is a consensus, the fitted distribution can be used as the prior for the trial. However, if the discussions have led to discrepancies, the information which has already been entered can be re-presented to the experts and they can be given the option to update their previous opinion based on the discussion. This process can be repeated as many times as needed until a consensus prior is reached.
Finally, once elicitation is complete, the software can be used to download all the information that has been elicited and produce outputs to show the elicitation cycles and the prior or priors that have been produced from the elicitation, giving a full account of how the elicitation progressed and a clear answer of the final model.
Conclusion
Expert prior elicitation is a valuable addition to clinical trial decision making. The incorporation of expertise can be particularly useful when dealing with rare or emerging conditions where information may be limited.
For elicitation to be successful, there should be a clear framework in place, which includes effective facilitation and training for participants. The approach can be further enhanced with the addition of software and statistical experts who can create end-to-end bespoke solutions for any elicitation challenges which may arise including the production of effective evidence dossiers.
References
- https://bmjopen.bmj.com/content/14/8/e090749
- https://www.bayes-pharma.org/wp-content/uploads/2023/11/06-PARTINGTON-Bayesian-Elicitation-for-rare-diseases-trials.pdf
- https://pubmed.ncbi.nlm.nih.gov/34411292/
- https://pubmed.ncbi.nlm.nih.gov/38708070/
- https://jitc.bmj.com/content/12/Suppl_2/A723
Author Details
Giles Partington, Principal Statistician, Phastar
Publication Details
This article appeared in Pharmaceutical Outsourcing:Vol. 26, No.2 Apr/May/June 2025Pages: 34-36