Editorial 

June 20, 2023

Revisiting the Analogy Between Clinical Trials and Diagnostic Tests by Interpreting a Negative Trial as a Negative Test for Efficacy

Roger J. Lewis, Kert Viele

JAMA. 2023;329(23):2023-2025. doi:10.1001/jama.2023.8972

As noted in JAMA more than 30 years ago,¹ randomized clinical trials are like diagnostic tests in several ways. Clinical trials are designed to detect hypothesized treatment effects in the same way diagnostic tests are designed to detect an abnormality or target illness. A trial’s statistical power, namely the probability that it can detect the treatment effect given that the hypothesized effect exists, is analogous to a diagnostic test’s sensitivity, namely the probability a positive test result is obtained when the target condition is present.¹ Just as the sensitivity of a diagnostic test can be affected by the severity of illness, an effect called “spectrum bias,” the power of a clinical trial, is strongly influenced by the magnitude of the true treatment benefit associated with the experimental treatment.^2,3 The specificity of a diagnostic test, namely the fraction of all persons free of disease who have a true-negative test result, is analogous in clinical trial design to 1 −α, where α is the maximum statistically significant 1-tailed P value (eg, .025) defined for the trial (ie, the type I error risk, or the risk of a false-positive result if there is no treatment effect).

The analogy between the interpretation of negative trials, ie, those with nonsignificant results, and negative diagnostic tests is more nuanced as applied in practice. Although a skilled clinician understands that negative diagnostic test results are best interpreted in the context of the tests’ characteristics (sensitivity and specificity) and the pretest probability of abnormality or illness in the patients to whom it is applied, negative test results are often interpreted as definitively negative. In contrast, clinicians and researchers alike are often cautioned to avoid any interpretation of a negative randomized trial, with the admonition that “absence of evidence is not evidence of absence.”⁴ Also, although some diagnostic tests yield only dichotomous results, and others have reference ranges that dichotomize findings into normal or abnormal, the result of a clinical trial is always a quantitative estimate (eg, the difference in average outcomes between the experimental and control treatments).

A likelihood ratio–based approach may reconcile the apparent divergence in how negative tests and clinical trials are interpreted. In diagnostic testing, the negative likelihood ratio captures the information provided by a negative diagnostic test and is defined as the probability of a negative test result when the disease is present divided by the probability of a negative test result when the disease is not present.⁵ The pretest odds of the disease—the probability of the patient having the disease divided by the probability of not having the disease—can then be multiplied by the negative likelihood ratio result to get a posttest odds of disease after a negative test result.⁵The posttest odds of disease can then be converted into a posttest probability of disease to aid clinical interpretation.

For clinical trials, a likelihood ratio can quantify the support for the null hypothesis of no treatment effect relative to support for the alternative hypothesis of a defined treatment effect, given the observed difference in outcomes in the 2 treatment groups.⁵ The Figure illustrates 2 probability function curves, corresponding to the null and alternative hypotheses, respectively, and how they can be compared for any observed trial result to yield a likelihood ratio.

This figure illustrates calculation of likelihood ratios for a range of hypothetical clinical trial results, for a trial designed to demonstrate an increase of 6 units in a hypothetical quantitative end point with an experimental therapy compared with a control. The gray shaded area shows the probability function associated with the null hypothesis centered on a 0 treatment difference. The orange shaded area shows the upper 2.5% of the area under the null curve representing the “statistically significant difference” region of the distribution. The blue shaded area shows the probability function centered on the hypothesized treatment difference of 6 units. For any possible observed difference in outcomes, the ratio of the height of the blue (alternative) curve to the gray (null) curve at the observed difference in outcomes is the likelihood ratio associated with that observed difference. The bolded letters show 3 hypothetical trial results.

Point A, If an observed difference of 4 units is observed, a result on the boundary for statistical significance, the resulting likelihood ratio is 4.48, representing moderate evidence for the alternative hypothesis.

Point B, If a difference of 3 units is observed, a result halfway between the null and alternative hypothesis where the curves are the same height, the likelihood ratio is 1, providing no evidence in either direction.

Point C, If a difference of 0 units is observed (matching the null hypothesis), the corresponding likelihood ratio is 0.011, providing substantial evidence in favor of the null hypothesis.

The diagonal black line in the bottom graph is the likelihood ratio (the ratio of the heights of the 2 curves in the top panel across the range of possible observed differences in outcomes) graphed on the log scale.

The value of the likelihood ratio is dependent on the trial result, the distance between the 2 hypotheses, the variability in patient outcomes, and the sample size of the trial.

In both diagnostic testing and the interpretation of clinical trials, likelihood ratio values of less than 0.1 or more than 10 provide substantial evidence about the comparison; in diagnostic testing, against or for the presence of disease, and in clinical trials, against or for the hypothesis of a specific treatment effect.

In this issue of JAMA, Perneger and Gayet-Ageron⁶ apply this approach to quantify the likelihood ratios resulting from the results of 130 trials yielding 169 negative primary outcome results published in major medical journals. They calculated likelihood ratios to quantify the degree with which the trial results supported the null hypothesis of no treatment effect vs the originally hypothesized effect size sought by the trial (in their notation, likelihood ratios >1 increase the odds that the null hypothesis is true). The authors found that for more than two-thirds of the results, the likelihood ratio was more than 10 and for more than half it was more than 100, yielding strong or very strong evidence in favor of no treatment effect. Importantly, they found little relationship between the reported P values and the likelihood ratio, confirming the fact that a nonsignificant P value (ie, 1-tailed P > .025) by itself provides little information about the probability that the null hypothesis of no effect is true.⁶

The negative likelihood ratios illustrated in the Figure—and those determined by Perneger and Gayet-Ageron⁶—depend on 2 simplifying assumptions. First, the only hypotheses being considered are the original null and alternative hypothesis, with the latter defining a specific magnitude of treatment benefit. As the authors note, the treatment effect defined by the alternative hypothesis may be based on considerations other than the clinically most plausible, minimal clinically important, or scientifically most evidence-based treatment effect, eg, to justify a feasible sample size for the trial. Second, the magnitude of the treatment effect can only be 1 of those 2 values; no consideration is given to the possibility of a treatment effect somewhere between that specified in the null and alternative hypotheses. The situation is more complex if one considers all possible treatment effects, treating the hypothesized effect as a continuum. A negative trial that provides strong evidence supporting the null against an alternative characterized by a large treatment benefit could, simultaneously, support another alternative hypothesis of a modest but real effect against the null hypothesis of no effect. Even though considering a range of alternatives is more complex, the likelihood ratio may provide compelling evidence that the null is preferred over all clinically meaningful alternatives. And some trial results may be quite inconsistent with both of the prespecified hypothesized treatment effects (eg, a trial that demonstrates unexpected harm).

Many of the challenges associated with rigorous interpretation of negative randomized clinical trials stem from the artificial decision-making structure and thresholds imposed by traditional hypothesis testing. Once the results of the trial are known, the focus should shift from hypothesis testing to estimation of the treatment effect.⁷Hypothesis testing may be of use during planning of a trial, eg, in the determination of sample size; however, it is less useful once the data are known. Likelihood ratios in underpowered trials can demonstrate that negative results are often ambiguous, whereas in sufficiently powered or overpowered trials, negative results are often far more consistent with no effect than any clinically meaningful effect.

The value of the likelihood ratio approach could be maximized by routinely reporting the likelihood function associated with the trial’s final result, across the full range of clinically relevant treatment effects, including the minimal clinically important difference.⁸ The likelihood function is a quantitative representation of the degree of agreement between a fixed set of data and possible values of a parameter (eg, the magnitude of a treatment effect) that might explain the observed data. In the context of a randomized clinical trial, the function is created by taking the formula for the probability of any observed difference in outcomes between treatment groups as a function of a single assumed treatment effect and, instead, assuming the observed data are fixed and allowing the treatment effect to vary.

The likelihood function contains all the information from the trial regarding the degree of support for any treatment effect. Likelihood ratios can be obtained from the likelihood function by comparing the height of the curve at the 2 efficacy points being compared. Thus, providing the likelihood function, rather than just the likelihood ratios tied to the original trial design, would allow readers to consider the evidence supporting treatment effects beyond those initially considered.

Although determining the probability that the hypothesis of no treatment effect is true would require a Bayesian approach,⁹ Perneger and Gayet-Ageron have demonstrated negative trials often provide strong evidence in favor of the hypothesis of no effect. But, equally important, a negative result sometimes provides virtually no evidence one way or the other. Although the common belief that a negative clinical trial does not support a conclusion of no treatment effect is an oversimplification at best, and incorrect at worst, the quantification of the observed treatment effect and its associated likelihood function can appropriately inform clinical decision-making, just like a negative diagnostic test result.⁷