Should Health Care Demand Interpretable Artificial Intelligence or Accept “Black Box” Medicine?

In recent years, health care applications of artificial intelligence (AI), such as detection of atrial fibrillation from electrocardiography, retinopathy from optical coherence tomography, and in-hospital mortality risk from electronic health records, have emerged (1–3). Artificial intelligence is also capable of assisting with more abstract clinical situations, such as predicting the onset of sepsis before clinician recognition (4). Artificial intelligence approaches, such as deep learning, rely on vast amounts of data and complex model structures with millions of parameters. For example, the Inception v3 model (Google), which is more accurate than physicians at identifying diabetic retinopathy from fundus photographs and skin cancer from dermoscopic images, has 23 million parameters (5). This complexity makes it difficult to understand how models make a given decision or prediction. Thus, they are often called “black boxes.” Many clinicians feel uncomfortable with black box AI, even if it does achieve high degrees of diagnostic or prognostic accuracy, and there have been calls for more research on how these models work. However, others have questioned whether explainability is necessary. Geoff Hinton, a leader in AI, noted that clinicians and regulators should not insist on explainability because “People can't explain how they work for most of the things they do” (6). Recent evidence suggests that people may not distrust black box models any more than transparent ones and that transparency may contribute to undue trust, hampering a user's ability to recognize mistakes (7). Given this, should physicians and...