Large language models (LLMs), such as the AI behind popular chatbots, show promise in healthcare, but they may make recommendations influenced by patients’ sociodemographic characteristics. Researchers tested nine LLMs on 1,000 emergency department cases, half real and half synthetic. Each case was presented in 32 versions: one for each of 31 sociodemographic groups, plus a control version with no sociodemographic details. The clinical details were always the same. In total, they analysed more than 1.7 million AI-generated recommendations.
Compared with both a physician-defined baseline and the control cases, cases labelled as Black or unhoused or as LGBTQIA+ were more often directed towards urgent care, invasive interventions or mental health evaluations. For example, certain cases from LGBTQIA+ subgroups were recommended mental health assessments around six to seven times more often than clinically indicated. Cases labelled as high-income were more often recommended advanced imaging tests, such as CT or MRI, while low- and middle-income cases were often limited to basic tests or none at all.
These differences were not supported by clinical reasoning or guidelines and may reflect bias in the models, which could lead to health disparities. As they were found in both proprietary and open-source models, the authors stress the need for robust ways to evaluate and reduce bias, so that AI-driven medical advice remains fair and centred on patients.
Learn more about this study here: https://doi.org/10.1038/s41591-025-03626-6
Reference:
Omar, M., Soffer, S., Agbareia, R., Bragazzi, N. L., Apakama, D. U., Horowitz, C. R., Charney, A. W., Freeman, R., Kummer, B., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2025). Sociodemographic biases in medical decision making by large language models. Nature Medicine, 31, 1873–1881.