Artificial intelligence systems in health care must be trained on the data of lived experience to prevent bias and disparities
Several years ago, I attended an international health care conference, eagerly awaiting the keynote speaker’s talk about a diabetes intervention that targeted people in lower socioeconomic groups of the U.S. He noted how an AI tool enabled researchers and physicians to use pattern recognition to better plan treatments for people with diabetes.
The speaker described the study, the ideas behind it and the methods and results. He also described the typical person who was part of the project: a 55-year-old Black female with a 7th to 8th grade reading level and a body mass index suggesting obesity. This woman, the speaker said, rarely adhered to her normal diabetes treatment plan. This troubled me: whether or not a person adhered to her treatment was reduced to a binary yes or no. And that did not take into consideration her lived experience—the things in her day-to-day life that led to her health problems and her inability to stick to her treatment.
The algorithm rested on data from medications, laboratory tests and diagnosis codes, among other things, and, based on this study, doctors would be delivering health care and designing treatment plans for middle-aged, lower-income Black women without any notion of how feasible those plans would be. Such practices would undoubtedly add to health disparities and health equity.