Applying large language models to AI safety: identifying and monitoring patient safety events associated with AI-enabled medical devices

Abstract

Artificial intelligence (AI) is being increasingly integrated into healthcare, offering many potential benefits to health professionals and patients. However, it also presents new and often unforeseen risks to patient safety. About 16% of patient safety events reported to the world's largest regulator of medical devices, the US Food and Drug Administration over the last 6-years were associated with harm or death. Identifying safety events involving AI-enabled medical devices is particularly challenging due to their low representation among the millions of reports submitted to the FDA each year as well as the lack of transparency about devices that incorporate AI. Consequently, manual reviews are required to identify AI-enabled medical devices and the safety events associated with these devices, delaying the timely detection of safety issues and making it impossible to keep up with the growing volumes of reports.   One possible solution is to use large language models (LLMs) to automatically identify AI safety events, as they can effectively extract insights from unstructured medical text. This presentation will describe an innovative technical platform and the results of experiments to monitor AI risks using BERT, an open source LLM. The platform can facilitate timely detection of AI safety issues, support regular analysis of safety events, and disseminate reports to healthcare providers, patients, AI developers, researchers and stakeholders.