Editor's Note
October 9, 2023
FDA Regulations of AI-Driven Clinical Decision Support Devices Fall Short
Anand R. Habib, Cary P. Gross
JAMA Intern Med. Published online October 9, 2023. doi:10.1001/jamainternmed.2023.5006
We are entering a new era of computerized clinical decision support (CDS) tools. Companies are increasingly using artificial intelligence and/or machine learning (AI/ML) to develop new CDS devices, which are defined by the US Food and Drug Administration (FDA) as software used in disease prevention, diagnosis, or treatment. Recognizing the potential implications for clinical practice, the 21st Century Cures Act enjoined the FDA to regulate these new devices.
In their case series reported in this issue of JAMA Internal Medicine, Lee and colleagues1 analyzed the evidence supporting FDA approval of 10 AI/ML CDS devices intended for use in critical care. Their findings are worrisome. Only 2 device authorizations cited peer-reviewed publications, and only 1 outlined a detailed safety risk assessment. No company provided software code to enable independent validation, evaluated clinical efficacy, or assessed whether the use of algorithms exacerbates health disparities.
The findings from Lee et al1 highlight 4 opportunities to enhance regulation of AI/ML-driven CDS devices: algorithmic transparency, evidentiary standards, 510(k) premarket notification pathway eligibility, and bias evaluation. First, regarding transparency, companies should divulge the algorithms, programming code, and data sets underlying their AI/ML CDS devices. Epitomizing the perils of “black box” algorithms, referring to methods in which users are not privy to information about data inputs and specific details of the analytical strategies, the widely implemented, AI/ML-driven Epic Sepsis Model demonstrated poor test characteristics when independently externally validated.2
Second, the FDA must require rigorous preapproval studies of validity, safety, and efficacy, as well as postmarketing surveillance of clinical utility. A recent review of 37 published studies evaluating AI/ML CDS devices identified a troubling lack of high-quality evidence.3 While the FDA has issued draft guidance outlining how companies can update their devices postmarketing without further FDA scrutiny, how these recommendations will be implemented is unclear.
Third, the FDA must reassess whether AI/ML CDS devices qualify for its 510(k) pathway. Although eligible devices must use the same technological characteristics as their predicates, 6 devices analyzed by Lee et al1 garnered approvals despite their predicates’ use of non-AI/ML methods.
Fourth, the FDA should focus closely on devices’ risks of exacerbating social and racial biases. An AI/ML CDS device trained on data from a demographically homogeneous population or data that arise from existing clinician bias can produce erroneous predictions when applied to diverse patient populations. Without assessing for this risk a priori, we would unknowingly perpetuate health care disparities.
With Medicare already reimbursing for some FDA-approved AI/ML-powered devices, we urgently need updated FDA guidance and enforcement of regulations regarding evidence of safety and clinical benefit for AI/ML-driven CDS devices. Absent those reforms, clinicians and health care institutions should pause using these devices for time-critical decisions.
