Approach to Demystify Black Box AI

Research suggests that compared with human clinicians, image heat maps underperform and require further refinement

Artificial intelligence models that interpret medical images hold the promise to enhance clinicians’ ability to make accurate and timely diagnoses, while also lessening workload by allowing busy physicians to focus on critical cases and delegate rote tasks to AI.

But AI models that lack transparency about how and why a diagnosis is made can be problematic. This opaque reasoning — also known “black box” AI — can diminish clinician trust in the reliability of the AI tool and thus discourage its use. This lack of transparency could also mislead clinicians into overtrusting the tool’s interpretation.

Get more HMS news here

In the realm of medical imaging, one way to create more understandable AI models and to demystify AI decision-making have been saliency assessments — an approach that uses heat maps to pinpoint whether the tool is correctly focusing only on the relevant pieces of a given image or homing in on irrelevant parts of it.

Read more