AI devices without clinical validation linked to more reminders, discovers the study

The artificial intelligence medical devices without clinical validation were more likely to be the subject of recalls, according to a Study published in Jama Health Forum.

The study, published on August 22, examined 950 AI medical devices authorized by the Food and Drug Administration until November 2024. Sixty of the devices were associated with 182 recall events.

The most common causes of reminders were diagnostic or measurement errors, followed by a delay in functionality or loss. About 43% of all reminders also took place in the year following the authorization of the FDA.

Tinglong Dai, the main study of the study and professor at the Johns Hopkins Carey Business School, said that the “vast majority” of the recalled devices had not undergone clinical trials. For the majority of AI compatible devices, which have followed the FDA 510 (K) route of the FDA, clinical studies are not necessary.

“Unfortunately, it is not necessary, and therefore people do not do it,” said Dai in an interview. “This is why we think it is one of the most important engines of reminders.”

In comparison, the study revealed that the devices that had undergone retrospective or prospective validation were subject to fewer reminders.

The study also revealed that listed companies represented more recall events in a disproportionate manner, the status of the public enterprise associated with a chance of almost six times higher of recall. Listed companies represented approximately 53% of compatible AI apparatuses on the market, but they were associated with more than 90% of the study’s recall events and 98.7% of the recalled units.

Public companies also had a lower clinical validation rate compared to private companies. While around 40% of the systems recalled by private companies did not have validation, in comparison, around 78% of devices of larger public companies and 97% of small public enterprises had no validation.

Dai was surprised by this observation, claiming that “this has fundamentally something to do with the clearance route 510 (K)”.

The results raise concerns about the security and post-marketing reliability of the devices. DAI and its co-authors recommended that it requires human tests or clinical trials before a device is authorized, or encourage companies to carry out current studies and collect actual performance data. Pre-commercial and post-marketing data could also help manufacturers identify and reduce dysfunctions and devices.

DAI has also suggested a process where authorizations can be revoked after five years if a device has no public clinical data, validation of the postal market or proof that it is effective in the real world.

In 2023, the FDA Issues three directive projects To improve program 510 (K), including recommendations concerning the choice of appropriate predicate devices and when clinical data may be necessary to demonstrate substantial equivalence. However, orientation documents have still not been finalized.

Researchers from the Johns Hopkins Carey Business School, the Johns Hopkins Bloomberg School of Public Health and the Yale School of Medicine contributed to the study. He was funded by a Prize at Johns Hopkins University.