Tag: AI in Healthcare

  • Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models

    Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models

    Large multi-modal models (LMMs) are a fast-growing type of generative AI that can take in different types of data, such as text, images and video, and produce varied responses. The World Health Organization (WHO) has issued guidance on their use in health, with over 40 recommendations for governments, technology companies and healthcare providers. The guidance covers five broad uses: diagnosis and clinical care, patient-guided use such as checking symptoms, administrative tasks, medical and nursing education, and scientific research and drug development.

    The WHO warns that LMMs can produce false, inaccurate, biased or incomplete answers, which could harm people making health decisions. They may be trained on poor-quality or biased data, including by race, ethnicity, ancestry, sex, gender identity or age. Because most of the data and research used to train them comes from high-income countries, their answers may also be biased towards a high-income perspective. In addition, health professionals may place too much trust in these tools and overlook their errors, a problem known as automation bias.

    Among its recommendations, the WHO calls on governments to pass laws and regulations to ensure LMMs meet ethical and human rights standards, assign regulatory agencies to assess and approve them, and require independent audits and impact assessments after they are released. Developers should involve diverse stakeholders, including patients and health professionals, from the early stages of development, and design LMMs to carry out well-defined tasks accurately and reliably.

    Learn more here: https://www.who.int/publications/i/item/9789240084759


    Reference:

    World Health Organization. (2024). Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models.

  • AI-based diabetes care: risk prediction models and implementation concerns

    AI-based diabetes care: risk prediction models and implementation concerns

    AI is increasingly being used to predict a person’s risk of developing type 2 diabetes, which could help delay or prevent the disease through earlier, more personalised prevention. This editorial discusses a scoping review of 40 studies on AI models for diabetes risk prediction. It found that models combining electronic health records with other types of data, such as genetic or medical imaging data, performed better than models using a single type of data.

    However, the authors warn that these results should be approached with caution. Many of the models showed a pronounced imbalance in gender, ethnicity and age in the data used to train them, only five were tested on external data, and most were not evaluated across different demographic groups. Since Black and Mexican Americans have a higher prevalence of diabetes than White Americans, leaving certain groups out of training data could worsen health inequities for populations already at greater risk.

    To address this, the authors call for mandatory representation criteria for the approval and adoption of these models, tools to assess their risk of bias, and external validation. They argue that AI cannot be adopted on a large scale in healthcare until problems with data quality and bias are addressed, and that this requires collaboration between clinicians, researchers, policymakers and entrepreneurs.

    Learn more about this article here: https://doi.org/10.1038/s41746-024-01034-7


    Reference:

    Wang, S. C. Y., Nickel, G., Venkatesh, K. P., Raza, M. M., & Kvedar, J. C. (2024). AI-based diabetes care: Risk prediction models and implementation concerns. npj Digital Medicine, 7, 36.

  • UnitedHealth lawsuit over alleged AI-driven denials of care moves forward

    UnitedHealth lawsuit over alleged AI-driven denials of care moves forward

    In November 2023, the families of two deceased patients filed a proposed class action lawsuit in the US against UnitedHealth Group, one of the country’s largest health insurers, and its subsidiary naviHealth. They allege the companies used an AI tool called nH Predict to wrongfully deny elderly patients in Medicare Advantage plans coverage for care after a hospital stay, such as physical therapy or skilled nursing care. According to the complaint, Gene Lokken, 91, was covered for only 19 days of care after a fall, despite his doctors recommending more, and his family paid around $150,000 until his death.

    The tool uses information such as age, living situation and physical function to compare patients with millions of past cases and estimate how long they should need care. The lawsuit alleges that the company applied these estimates even when they went against doctors’ recommendations, and that staff were pressured to keep patients’ stays close to the tool’s predictions. Over 90% of the denials that were appealed were reversed, but only about 0.2% of patients ever appealed. UnitedHealth denies the claims, saying the tool is used as a guide and not to make coverage decisions.

    In February 2025, a federal court in Minnesota allowed part of the case to proceed, and in March 2026 a judge ordered UnitedHealth to hand over a wide range of documents on how the tool was implemented and used. The case is still ongoing and raises important questions about transparency and accountability when algorithms influence decisions about patients’ care.

    Learn more here: https://www.statnews.com/2023/11/14/unitedhealth-class-action-lawsuit-algorithm-medicare-advantage/
    Read the latest update here: https://www.yahoo.com/news/us/articles/lawsuit-says-unitedhealth-used-ai-173017952.html


    References:

    Ross, C., & Herman, B. (2023, November 14). UnitedHealth faces class action lawsuit over algorithmic care denials in Medicare Advantage plans. STAT.

    Becker’s Payer Issues. (2026, March 11). Judge orders UnitedHealth to hand over documents in AI coverage denial case.

    Barrientos, A. (2026, September 18). Lawsuit says UnitedHealth used AI with a 90% error rate to deny care. Yahoo News.

  • Predicting cardiovascular events from routine mammograms using machine learning

    Predicting cardiovascular events from routine mammograms using machine learning

    Cardiovascular risk is often underassessed in women, and many women have screening mammograms in midlife, just when their risk of cardiovascular disease starts to rise. Researchers led by The George Institute for Global Health developed an AI model that predicts a woman’s risk of a major cardiovascular event using only routine mammogram images and her age.

    The model was developed using data from 49,196 women in Australia, who were followed for a median of 8.8 years. It analyses a wide range of features in breast tissue, not just calcium deposits in the arteries, and performed as well as standard risk calculators that require medical history and other clinical information. Adding clinical factors improved its performance only slightly.

    Because many women already have regular mammograms for breast cancer screening, this approach could offer a “two-for-one” check, without the need for extra appointments or medical records. The researchers note some limitations, such as differences between mammography machines, and say further work is needed before the tool can be used in practice.

    Learn more about this study here: https://doi.org/10.1136/heartjnl-2025-325705
    Read the news here: https://www.georgeinstitute.org/news-and-media/news/new-ai-algorithm-uses-mammograms-to-accurately-predict-cardiovascular-risk-in-women


    References:

    Barraclough, J. Y., Gandomkar, Z., Fletcher, R. A., Barbieri, S., Kuo, N. I.-H., Rodgers, A., Douglas, K., Poppe, K. K., Woodward, M., Gallego Luxan, B., Neal, B., Jorm, L., Brennan, P., & Arnott, C. (2026). Predicting cardiovascular events from routine mammograms using machine learning. Heart, 112(5), 261–269.

    Wall, T. (2025, September 17). New AI algorithm uses mammograms to accurately predict cardiovascular risk in women. The George Institute for Global Health.

  • Can AI fight sex and gender bias in healthcare?

    Can AI fight sex and gender bias in healthcare?

    AI is transforming medicine, but it is mostly developed by men and trained on data that prioritises men’s bodies and health needs. In this commentary, researchers from the George Institute for Global Health argue that this embeds sex and gender biases that can put women and nonbinary patients at risk. For example, natural language processing models used in psychiatry have shown significant gender biases, and tools that screen for suicide risk may make mistakes if trained mainly on text written by men, since men and women express distress differently.

    The authors also argue that AI could help reduce these inequities if sex and gender are considered deliberately. One example is the GRACE heart attack risk score, which was updated using AI models that analyse male and female data separately to account for sex-specific characteristics of the disease. Another is a chatbot developed by the George Institute to give guideline-based pregnancy advice in rural India, created together with community health workers, clinicians and women.

    They call for sex and gender bias to be actively addressed in the data used to train AI, for tools to be co-designed with the communities they serve, for work on bias to go beyond the English language, and for sex and gender to be considered at every stage of research and clinical use.

    Learn more here: https://www.unsw.edu.au/newsroom/news/2024/10/can-ai-fight-sex-and-gender-bias-in-healthcare-


    Reference:

    Haupt, S., Graham, B., & Hirst, J. (2024, October 21). Can AI fight sex and gender bias in healthcare? 360info.

  • Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations

    Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations

    AI systems now match expert performance in many medical imaging tasks, but there is growing concern that they may reflect and amplify human bias. This is especially worrying in the case of underdiagnosis, when AI wrongly labels a person who has a disease as healthy, which can delay their access to care. This study examined underdiagnosis by AI tools that analyse chest X-rays, using three large US datasets and a combination of all three, with more than 700,000 images from almost 130,000 patients.

    The AI tools consistently and selectively underdiagnosed under-served groups: female patients, patients under 20, Black patients, Hispanic patients and patients with Medicaid, a public health insurance for people with low incomes. Underdiagnosis was even higher for patients who belonged to more than one of these groups, such as Hispanic female patients, and for young patients who were also female, Black or on Medicaid.

    The authors warn that using AI tools with these biases could worsen existing inequalities in care and lead to unequal access to treatment, raising ethical concerns about using these models in clinical practice.

    Learn more about this study here: https://doi.org/10.1038/s41591-021-01595-0


    Reference:

    Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021). Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine, 27(12), 2176–2182.

  • Bias Mitigation in Primary Health Care Artificial Intelligence Models: Scoping Review

    Bias Mitigation in Primary Health Care Artificial Intelligence Models: Scoping Review

    AI models in primary care can help identify quickly and accurately the people who should receive care, but they risk perpetuating or amplifying biases against diverse groups. This scoping review looked at the strategies and methods used to reduce bias in primary care AI models, which groups were considered, and how these strategies affected both bias and model performance.

    Of the 17 studies included, race or ethnicity was the attribute most often examined (12 studies), followed by sex or gender (10 studies), usually defined only as male or female. The approaches to reducing bias fell into four groups: modifying existing models or datasets, sourcing data from electronic health records, developing tools with human input (“human-in-the-loop”), and identifying ethical principles to guide decisions. Adjusting the data before training the model, for example by relabelling or reweighting it, and analysing doctors’ written notes with language-processing tools showed the greatest potential. Other methods sometimes increased errors or made the models less well calibrated.

    The authors conclude that bias appears easier to reduce when data are open source, when multiple stakeholders are involved, and when bias is addressed before the model is trained. They call for further studies that include a wider range of groups, such as Indigenous peoples in Canada.

    Learn more about this review here: https://doi.org/10.2196/60269


    Reference:

    Sasseville, M., Ouellet, S., Rhéaume, C., Sahlia, M., Couture, V., Després, P., Paquette, J.-S., Darmon, D., Bergeron, F., & Gagnon, M.-P. (2025). Bias mitigation in primary health care artificial intelligence models: Scoping review. Journal of Medical Internet Research, 27, e60269.

  • Addressing AI Algorithmic Bias in Health Care

    Addressing AI Algorithmic Bias in Health Care

    Biased AI algorithms can result in some groups of patients not receiving appropriate care, potentially leading to significant harm. In one example, an algorithm designed to identify patients who needed extra care used healthcare costs to estimate their needs. Because Black patients face barriers in accessing care, less was spent on them, and the algorithm may have more than halved the number of Black patients identified for extra care.

    In April 2024, the US Department of Health and Human Services issued a rule that explicitly prohibits discriminatory outcomes from tools that support clinical decisions, including AI. Health care facilities now have to identify and reduce the risk of discrimination when using these tools. However, the authors argue that guidelines, resources and shared responsibilities are still lacking. Algorithms need to be monitored locally over time, which may be possible for well-resourced facilities but is a major challenge for under-resourced ones. These facilities may either avoid AI tools that could benefit their communities, or adopt tools that produce biased results without being aware of it.

    The authors propose a model of shared responsibility between health care facilities, AI developers and regulators. Developers should be transparent, showing that diverse data were used, describing how they tested their algorithms for bias, and disclosing any limitations, with this information stored in a public central repository. Existing federal requirements for AI transparency should also be aligned with the new anti-discrimination rule.

    Learn more about this article here: https://doi.org/10.1001/jama.2024.13486


    Reference:

    Ratwani, R. M., Sutton, K., & Galarraga, J. E. (2024). Addressing AI algorithmic bias in health care. JAMA, 332(13), 1051–1052.

  • Enhancing Fairness and Accuracy in Diagnosing Type 2 Diabetes in Young Adult Population

    Enhancing Fairness and Accuracy in Diagnosing Type 2 Diabetes in Young Adult Population

    Type 2 diabetes is most common in older people, but it is becoming more frequent among young adults, who face a higher risk of early and severe complications. Missing the diagnosis in this group can have serious consequences for their health. This study found that common machine learning models show what the authors call “digital ageism”: they miss diabetes more often in young adults aged 30 to 44 than in other age groups.

    The researchers used data from a large US health survey with more than 400,000 participants, in which young adults are a minority of the people with diabetes. Instead of using a single model for everyone, they trained a separate model for each five-year age band (30–34, 35–39 and 40–44), adjusting the data balance within each group.

    This approach improved the detection of diabetes in young adults by 26% to 40%, and performed better than seven commonly used techniques for balancing data. The authors note that a model trained for one age group did not work equally well for groups defined by gender or ethnicity, but that the same approach could be applied to other underrepresented groups.

    Learn more about this study here: https://doi.org/10.1109/JBHI.2025.3616312


    Reference:

    Pias, T. S., Su, Y., Tang, X., Wang, H., Faghani, S., & Yao, D. (2026). Enhancing fairness and accuracy in diagnosing type 2 diabetes in young adult population. IEEE Journal of Biomedical and Health Informatics, 30(4), 3321–3330.

  • Inteligência Artificial no setor da saúde: desafios jurídicos e regulação

    Inteligência Artificial no setor da saúde: desafios jurídicos e regulação

    Artificial Intelligence in the health sector: legal challenges and regulation

    AI can significantly improve the quality of health services, but its use also raises well-founded concerns. This paper, published by the Office for Strategy and Studies (GEE) of the Portuguese Ministry of Economy, analyses the current landscape of AI in the Portuguese health sector and proposes public policy solutions for Portugal.

    The author focuses on three main challenges: the opacity of AI systems, which can undermine patients’ informed consent; bias and discrimination; and the protection of health data and privacy. On bias, she explains that it comes mostly from the data used to train AI, which can reflect existing social prejudices and be skewed by factors such as age, sex, race and socioeconomic background. The paper also analyses how these systems fit within the EU AI Act, with particular attention to the obligation of human oversight.

    To reduce these risks, the author proposes measures such as ethical requirements in the development and choice of AI systems, impact assessments and independent data audits before approval, periodic evaluation with a mechanism for reporting failures, and training for health professionals so that they can understand AI and recognise its errors.

    Learn more about this paper here: https://EconPapers.repec.org/RePEc:mde:wpaper:187


    Reference:

    Paixão Cansado, M. (2024). Inteligência Artificial no setor da saúde: desafios jurídicos e regulação (GEE Paper No. 187). Gabinete de Estratégia e Estudos, Ministério da Economia.

  • Sociodemographic biases in medical decision making by large language models

    Sociodemographic biases in medical decision making by large language models

    Large language models (LLMs), such as the AI behind popular chatbots, show promise in healthcare, but they may make recommendations influenced by patients’ sociodemographic characteristics. Researchers tested nine LLMs on 1,000 emergency department cases, half real and half synthetic. Each case was presented in 32 versions: one for each of 31 sociodemographic groups, plus a control version with no sociodemographic details. The clinical details were always the same. In total, they analysed more than 1.7 million AI-generated recommendations.

    Compared with both a physician-defined baseline and the control cases, cases labelled as Black or unhoused or as LGBTQIA+ were more often directed towards urgent care, invasive interventions or mental health evaluations. For example, certain cases from LGBTQIA+ subgroups were recommended mental health assessments around six to seven times more often than clinically indicated. Cases labelled as high-income were more often recommended advanced imaging tests, such as CT or MRI, while low- and middle-income cases were often limited to basic tests or none at all.

    These differences were not supported by clinical reasoning or guidelines and may reflect bias in the models, which could lead to health disparities. As they were found in both proprietary and open-source models, the authors stress the need for robust ways to evaluate and reduce bias, so that AI-driven medical advice remains fair and centred on patients.

    Learn more about this study here: https://doi.org/10.1038/s41591-025-03626-6


    Reference:

    Omar, M., Soffer, S., Agbareia, R., Bragazzi, N. L., Apakama, D. U., Horowitz, C. R., Charney, A. W., Freeman, R., Kummer, B., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2025). Sociodemographic biases in medical decision making by large language models. Nature Medicine, 31, 1873–1881.

  • Addressing bias in big data and AI for health care: A call for open science

    Addressing bias in big data and AI for health care: A call for open science

    Most AI tools in healthcare learn from large datasets, but many groups of people have long been missing or misrepresented in medical data. When AI is trained on data that does not reflect the whole population, it can reinforce existing biases and lead to misdiagnoses and even fatal outcomes, particularly for gender and ethnic minorities.

    The authors give several examples. AI tools for classifying skin lesions are often trained on datasets in which only about 5% to 10% of images come from Black patients, and when tested on images of Black patients they reach about half the accuracy their creators claimed. This matters because Black patients have the highest mortality from melanoma. In cardiology, heart attacks are often misdiagnosed in women, yet some models that predict heart attacks were trained mostly on data from men. Many datasets also record gender only as male or female, making it difficult to build AI that includes LGBTQ+ people or to test it for bias.

    To move towards fairer AI, the authors propose open science practices: involving patients and communities in the development of AI, sharing data responsibly while protecting privacy, using inclusive data standards, and sharing code. Sharing code and algorithms allows researchers around the world to test whether a tool developed with data from one population also works for patients elsewhere.

    Learn more about this article here: https://doi.org/10.1016/j.patter.2021.100347


    Reference:

    Norori, N., Hu, Q., Aellen, F. M., Faraci, F. D., & Tzovara, A. (2021). Addressing bias in big data and AI for health care: A call for open science. Patterns, 2(10), 100347.

  • Exploring Gender Bias in AI for Personalized Medicine: Focus Group Study With Trans Community Members

    Exploring Gender Bias in AI for Personalized Medicine: Focus Group Study With Trans Community Members

    Precision medicine aims to tailor healthcare to each person, but trans people are often left out of this research. Little is known about how AI tools in healthcare could be adapted to their specific health needs. This study explored the challenges, obstacles and possible solutions for using AI in personalised medicine for trans people.

    The researchers worked together with trans community representatives to co-design the study and its data collection tools, and held three focus groups with 16 trans adults. Participants raised concerns about data privacy, algorithmic bias and the lack of health data that reflects trans experiences. They also worried that AI built on data models that assume everyone is cisgender could lead to misdiagnosis or inappropriate treatment.

    At the same time, participants saw opportunities for AI to improve health outcomes. They proposed community-led data collection, more transparent algorithms, including trans-specific health markers in datasets, involving community members in AI development, and ethical frameworks that respect gender diversity. The authors conclude that precision medicine needs a trans-inclusive approach, with AI designed around the health needs and lived realities of trans people.

    Learn more about this study here: https://doi.org/10.2196/72325


    Reference:

    Buslón, N., Cirillo, D., Rios, O., & Perera del Rosario, S. (2025). Exploring gender bias in AI for personalized medicine: Focus group study with trans community members. Journal of Medical Internet Research, 27, e72325.

  • Mitigating the risk of artificial intelligence bias in cardiovascular care

    Mitigating the risk of artificial intelligence bias in cardiovascular care

    Digital health technologies produce large amounts of data that can be used to train AI, which is transforming cardiovascular care. However, AI can become biased when the data used to train it comes from a very similar group of people, or from healthcare processes that are themselves unequal. Bias can also be introduced while algorithms are developed, tested and put into practice, and even after they are already in use.

    Biased algorithms can lead to missed diagnoses, misclassified disease, inaccurate risk predictions and inappropriate treatment recommendations, and these harms can fall disproportionately on marginalised groups. This paper, part of a series in The Lancet Digital Health, gives an overview of how AI is used in cardiovascular care, explains where bias can arise at each stage of an algorithm’s development, and presents examples of harm caused by biased algorithms.

    The authors also propose strategies to reduce bias during the training, testing and implementation of algorithms, so that “all those at risk for or living with cardiovascular disease might benefit equally from AI.”

    Learn more about this article here: https://doi.org/10.1016/S2589-7500(24)00155-9


    Reference:

    Mihan, A., Pandey, A., & Van Spall, H. G. C. (2024). Mitigating the risk of artificial intelligence bias in cardiovascular care. The Lancet Digital Health, 6(10), e749–e754.

  • Research Frameworks towards Health Equity

    Research Frameworks towards Health Equity

    AI has great potential in healthcare, but existing algorithms risk perpetuating and worsening existing biases, particularly racial and ethnic disparities. This master’s thesis, from the Faculty of Engineering of the University of Porto and developed with researchers at MIT, looks at these issues through a health equity lens. It reviews challenges such as sharing clinical data, selection bias, biased measurements from medical devices, and common pitfalls in developing AI.

    The thesis then proposes three research frameworks, tested with a large intensive care database. The first found that patients with sepsis from racial and ethnic minority groups were less likely than White patients to receive mechanical ventilation and medication to raise blood pressure. The second found no substantial differences between groups in the effect of these treatments on outcomes such as mortality. The third focused on pulse oximeters, which are less accurate in people with darker skin, and showed that machine learning can be used to correct their readings, although this still needs to be validated in other databases.

    Distinguished in the 2023 IEEE Portugal Section Outstanding MSc Thesis Awards, this work highlights the need to place health equity at the forefront of health research.

    Learn more about this thesis here: https://repositorio-aberto.up.pt/handle/10216/150994


    Reference:

    Matos, J. (2023). Research frameworks towards health equity [Master’s thesis, Faculdade de Engenharia da Universidade do Porto]. Repositório Aberto da Universidade do Porto.

  • Guiding Principles to Address the Impact of Algorithm Bias on Racial and Ethnic Disparities in Health and Health Care

    Guiding Principles to Address the Impact of Algorithm Bias on Racial and Ethnic Disparities in Health and Health Care

    Algorithms are widely used in healthcare to support diagnosis, treatment, risk assessment and decisions about who receives resources. When they are biased, they can worsen outcomes for racial and ethnic minority groups and other marginalised populations. For example, an algorithm that used race to estimate kidney function gave higher results for Black patients, delaying their referral for organ transplants. Another, used to manage chronic diseases, effectively required Black patients to be sicker than White patients to qualify for extra care.

    To address this, two US federal agencies, the Agency for Healthcare Research and Quality and the National Institute on Minority Health and Health Disparities, brought together a diverse expert panel. The panel reviewed the evidence and heard from stakeholders, patients and community members. It developed a framework covering the whole life cycle of an algorithm, from defining the problem and choosing the data to development, deployment and monitoring.

    The framework is built on five guiding principles: promote health equity at every stage of an algorithm’s life cycle; ensure algorithms and their use are transparent and explainable; authentically engage patients and communities and earn their trust; explicitly identify fairness issues and trade-offs; and establish accountability for fair and equitable outcomes.

    Learn more about this article here: https://doi.org/10.1001/jamanetworkopen.2023.45050


    Reference:

    Chin, M. H., Afsar-Manesh, N., Bierman, A. S., Chang, C., Colón-Rodríguez, C. J., Dullabh, P., Duran, D. G., Fair, M., Hernandez-Boussard, T., Hightower, M., Jain, A., Jordan, W. B., Konya, S., Moore, R. H., Moore, T. T., Rodriguez, R., Shaheen, G., Snyder, L. P., Srinivasan, M., Umscheid, C. A., & Ohno-Machado, L. (2023). Guiding principles to address the impact of algorithm bias on racial and ethnic disparities in health and health care. JAMA Network Open, 6(12), e2345050.

  • Doctors use problematic race-based algorithms to guide care every day. Why are they so hard to change?

    Doctors use problematic race-based algorithms to guide care every day. Why are they so hard to change?

    Clinical algorithms are risk calculators and decision tools that doctors use every day, and many of them take a patient’s race into account. In Boston, for example, paediatrician Alexandra Epee-Bounya used a calculator to assess young children’s risk of a urinary tract infection that added two points if the child was a girl and one point if the child was not Black. The more points, the higher the estimated risk, and the more likely the child was to receive follow-up testing.

    This article opens STAT’s “Embedded Bias” series, which spent more than a year investigating why race, a social construct rather than a biological fact, is still so widely used in medicine. Race-adjusted calculators, used for example to assess kidney function, lung capacity and the risk of urinary tract infections, have been shown to underdiagnose and harm patients of colour. Although some hospitals and medical organisations have started to change, many of these algorithms remain in use. Across the series, the reporters identified 48 race-based clinical algorithms still in use.

    Some argue that race can still provide useful predictive information, for example in breast cancer or surgical risk calculators. Critics say this reinforces the false idea that race is a biological category.

    Learn more here: https://www.statnews.com/2024/09/03/embedded-bias-investigation-health-equity-clinical-algorithms/


    Reference:

    Palmer, K., & McFarling, U. L. (2024, September 3). Doctors use problematic race-based algorithms to guide care every day. Why are they so hard to change? STAT.

  • Using artificial intelligence on dermatology conditions in Uganda: a case for diversity in training data sets for machine learning

    Using artificial intelligence on dermatology conditions in Uganda: a case for diversity in training data sets for machine learning

    For AI to work fairly in healthcare, it needs to be trained on data that represents people from different parts of the world. This study, carried out in Uganda, tested how well an AI app called Skin Image Search, which suggests possible skin conditions from a photo, performed on people with very dark skin (Fitzpatrick skin type 6).

    The researchers analysed 123 images of skin conditions from a Ugandan telehealth company, 62% of them from women and 38% from men. The app’s overall diagnostic accuracy was only 17%, far below the 69.9% reported in its training results on white skin. Performance varied between conditions and was highest for dermatitis (80%).

    The authors conclude that the images used to train dermatology algorithms need to be more diverse, so that these tools are accurate across skin types and geographies.

    Learn more about this study here: https://doi.org/10.4314/ahs.v23i2.86


    Reference:

    Kamulegeya, L., Bwanika, J., Okello, M., Rusoke, D., Nassiwa, F., Lubega, W., Musinguzi, D., & Börve, A. (2023). Using artificial intelligence on dermatology conditions in Uganda: A case for diversity in training data sets for machine learning. African Health Sciences, 23(2), 753–763.

  • Gender biases within Artificial Intelligence and ChatGPT: Evidence, Sources of Biases and Solutions

    Gender biases within Artificial Intelligence and ChatGPT: Evidence, Sources of Biases and Solutions

    AI is now used in many areas of our lives, from hiring and education to healthcare, but it often mirrors the prejudices that exist in society. This review brings together evidence on how AI systems, and chatbots such as ChatGPT in particular, can perpetuate gender bias. Examples include a hiring algorithm that learned to favour men, and ChatGPT writing recommendation letters that described women with more social words, such as “family”, while letters for men emphasised their professional skills.

    In healthcare, the review explains that many AI tools are trained on data that comes mostly from men, which can lead to unequal care. Women’s heart disease symptoms are more likely than men’s to be misdiagnosed or attributed to other conditions, even when they report the same symptoms, and AI trained on this kind of data can repeat these patterns. The review also cites studies showing that AI tools for diagnosing chest diseases underdiagnosed women more often, and that GPT-4 was more likely to recommend treatments and further tests for men than for women.

    The authors identify three main sources of bias: the data used to train AI, the way the systems are designed, and the feedback loop created by users’ interactions with them. To address these, they propose improving the quality and diversity of data and of the people who label it, using algorithms designed with fairness in mind, and strengthening policies at company, national and international levels, such as the EU AI Act.

    Learn more about this review here: https://doi.org/10.1016/j.chbah.2025.100145


    Reference:

    Ho, J. Q. H., Hartanto, A., Koh, A., & Majeed, N. M. (2025). Gender biases within artificial intelligence and ChatGPT: Evidence, sources of biases and solutions. Computers in Human Behavior: Artificial Humans, 4, 100145.

  • Equity in medical devices: independent review

    Equity in medical devices: independent review

    Medical devices are often assumed to work equally well for everyone, but an independent review commissioned by the UK government shows this is not always the case. The review began after concerns that pulse oximeters, which measure oxygen levels in the blood, may be less accurate for patients with darker skin. It found that these devices overestimate oxygen levels in people with darker skin tones. In US studies, this has been linked to delayed recognition of disease and to denied or delayed treatment.

    The review found similar problems in other devices. Devices that measure bilirubin through the skin may overestimate its levels in babies with darker skin. Ventilators have default settings designed for men. AI tools that help detect skin cancer may have been trained mostly on images from people with fair skin, and an X-ray dataset that underrepresented women led to lower accuracy when diagnosing them. Genetic risk scores, which estimate a person’s risk of disease, are based on data that is overwhelmingly from people of European ancestry and may not work as well for other groups.

    To address these problems, the review makes 18 recommendations to the government, regulators, the NHS and device developers, from more diverse data and clinical trials to regular checks for bias once devices are in use.

    Learn more about this review here: https://www.gov.uk/government/publications/equity-in-medical-devices-independent-review-final-report/equity-in-medical-devices-independent-review-summary-report


    Reference:

    Whitehead, M., Ali, R., Carrol, E., Holmes, C., & Kee, F. (2024). Equity in medical devices: Independent review. Final report. Department of Health and Social Care.

  • Common heart failure may be underdiagnosed in Black patients, AI-based study finds

    Common heart failure may be underdiagnosed in Black patients, AI-based study finds

    Heart failure with preserved ejection fraction (HFpEF) accounts for around half of heart failure cases in the UK. In this condition, the heart pumps blood normally but does not fill properly, causing symptoms such as breathlessness, fatigue and dizziness. Researchers at King’s College London, funded by the British Heart Foundation, studied how this condition is diagnosed in patients of different ethnicities.

    The team used a natural language processing algorithm, an AI tool that reads and interprets medical text, to analyse electronic health records from a large London hospital. It identified 1,973 patients who met European guidelines for HFpEF: 64% White, 29% Black and 7% Asian. Compared with White patients, Black patients were younger at diagnosis, more often had obesity, diabetes and high blood pressure, and less often had atrial fibrillation. When the researchers compared these results with routine care, they found that Black and Asian patients were more likely to have been missed in routine practice than by the algorithm.

    The researchers believe this is linked to the H2FPEF score, which clinicians partly use to diagnose HFpEF and which gives more weight to atrial fibrillation. Black patients were less likely than White patients to reach a score indicating likely HFpEF (26% vs 44%), so relying on this score may lead to more Black patients being missed. The authors conclude that established markers appear to underdiagnose HFpEF in Black patients and that more research is needed to improve diagnostic tools.

    Learn more here: https://www.bhf.org.uk/what-we-do/news-from-the-bhf/news-archive/2024/july/ai-reduces-underdiagnosis-of-common-heart-failure-in-black-patients-new-research-finds
    Read the study here: https://doi.org/10.1016/j.jacadv.2024.101064


    References:

    British Heart Foundation. (2024, July). AI reduces underdiagnosis of common heart failure in Black patients, new research finds.

    Brown, S., Biswas, D., Wu, J., Ryan, M., Bernstein, B. S., Fairhurst, N., Kaye, G., Baral, R., Cannata, A., Searle, T., Melikian, N., Sado, D., Lüscher, T. F., Teo, J., Dobson, R., Bromage, D. I., McDonagh, T. A., Vazir, A., Shah, A. M., & O’Gallagher, K. (2024). Race- and ethnicity-related differences in heart failure with preserved ejection fraction using natural language processing. JACC: Advances, 3(8), 101064.

  • Artificial intelligence in healthcare: Applications, risks, and ethical and societal impacts

    Artificial intelligence in healthcare: Applications, risks, and ethical and societal impacts

    This report, published by the European Parliamentary Research Service’s Scientific Foresight Unit (STOA), looks at how AI is being used in healthcare and the risks it brings. It identifies seven main risks: patient harm due to AI errors, misuse of medical AI tools, bias and the perpetuation of existing inequities, lack of transparency, privacy and security issues, gaps in accountability, and obstacles to implementation in real-world healthcare.

    On bias, the report explains that the most common cause of unfairness in medical AI is bias in the data used to train it. Some groups are underrepresented in medical datasets; for example, skin cancer datasets mostly contain fair-skinned patients. Data can also carry existing human biases, such as the dismissal of women’s pain or doctors taking Black patients’ pain less seriously. Among the examples given is a chest X-ray algorithm that most often missed disease in young female patients, Black patients and patients on public insurance for people with low incomes.

    To address these risks, the report proposes seven policy options. These include extending AI regulation to cover healthcare-specific risks, involving all stakeholders throughout the development of medical AI, creating an “AI passport” for transparency and traceability, defining accountability, educating health professionals and the public, promoting further research, and reducing the divide in medical AI across Europe.

    Learn more about this report here: https://doi.org/10.2861/568473


    Reference:

    Lekadir, K., Quaglio, G., Tselioudis Garmendia, A., & Gallin, C. (2022). Artificial intelligence in healthcare: Applications, risks, and ethical and societal impacts (PE 729.512). European Parliamentary Research Service, Scientific Foresight Unit (STOA).

  • Bias in medical AI: Implications for clinical decision-making

    Bias in medical AI: Implications for clinical decision-making

    When AI is used to support clinical decisions, bias in the model can lead to poorer care and deepen existing health disparities. This review explains how bias can arise and build up at every stage of an AI tool’s life, from the data used to build it to the way it is used in practice and even how research is published.

    The authors identify several sources of bias. In the data, too few patients from some groups can lead to weaker performance and underestimated risk for them, and missing information, such as on social determinants of health, can also distort results. The labels used to train models may carry clinicians’ unconscious biases or reflect substandard care. Relying on overall performance figures can hide poorer results for specific groups, and a model’s performance may fall, sometimes unevenly, when it is used with patients different from those it was trained on. Finally, where models are developed and published, and by whom, shapes the priorities of future research.

    To address this, the authors recommend larger and more diverse datasets, statistical methods to reduce bias, thorough evaluation, a focus on making models interpretable, and standardised reporting of bias. They also stress that models should be tested in rigorous clinical trials before being used in real-world care.

    Learn more about this review here: https://doi.org/10.1371/journal.pdig.0000651


    Reference:

    Cross, J.L., Choma, M.A., & Onofrey, J.A. (2024). Bias in medical AI: Implications for clinical decision-making. PLOS Digital Health, 3(11), e0000651.

  • Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare

    Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare

    Precision medicine aims to tailor prevention and treatment to each person, based on the genetic and environmental factors that shape their health. AI can help by finding relevant patterns in large amounts of biomedical data. This review examines how sex and gender, which contribute to differences in health and disease, are taken into account, or not, in the AI technologies used in this field.

    Differences between women and men have been reported in how many diseases develop, how common they are, how their symptoms present and how they respond to treatment, including in cardiovascular disease, diabetes and mental health. Yet, according to the authors, most biomedical AI technologies do not include ways of detecting bias, and most algorithms ignore sex and gender altogether. This can lead to poorer results, errors and discriminatory outcomes. In depression, for example, women are diagnosed more often than men, but this may partly reflect the fact that research has historically measured symptoms that are more common in women.

    The authors review the existing sex and gender gaps in several technologies used in precision medicine, and make recommendations on how to use them in ways that improve health outcomes for everyone and reduce inequalities.

    Learn more about this review here: https://doi.org/10.1038/s41746-020-0288-5


    Reference:

    Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M.J., Chadha, A.S., & Mavridis, N. (2020). Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare. npj Digital Medicine, 3, 81.

  • Health for whom? Intersectionality and biases in the use of artificial intelligence in clinical diagnosis

    Health for whom? Intersectionality and biases in the use of artificial intelligence in clinical diagnosis

    AI is growing rapidly in healthcare and can support clinical diagnosis, for example by analysing X-rays, MRI and CT scans to find patterns that may escape the human eye. In this Spanish editorial, the authors warn that enthusiasm for these tools can overlook the risk that they perpetuate or deepen disparities based on gender, race and ethnicity. They argue for an intersectional approach, which looks at how different social identities overlap and create combined forms of disadvantage.

    The authors explain that AI can reflect and amplify the biases in the data it learns from, including the unconscious biases of health professionals. They give several examples: AI systems analysing chest X-rays that more often missed disease in Hispanic patients and patients with public health insurance in the US than in White patients; medical records describing the symptoms of African American patients in pejorative terms; and a fracture risk calculator that underestimated risk in Black women, producing data that may later be used to train AI.

    They conclude that answering the question “health for whom?” requires moving beyond guidelines and good intentions to enforceable rules. They call for more diversity among researchers, better methods to detect intersectional bias, common standards for recording social identities, and legal frameworks that address under-representation and unconscious bias, not only safety and privacy.

    Learn more about this article here: https://doi.org/10.23938/ASSN.1077


    Reference:

    Amaya-Santos, S., Jiménez-Pernett, J., & Bermúdez-Tamayo, C. (2024). ¿Salud para quién? Interseccionalidad y sesgos de la inteligencia artificial para el diagnóstico clínico [Health for whom? Intersectionality and biases in the use of artificial intelligence in clinical diagnosis]. Anales del Sistema Sanitario de Navarra, 47(2), e1077.

  • Addressing bias: artificial intelligence in cardiovascular medicine

    Addressing bias: artificial intelligence in cardiovascular medicine

    AI offers opportunities to transform cardiovascular medicine, but AI algorithms can also have biases of their own. In this commentary, the authors describe several sources of bias in cardiovascular AI. These include differences in how diagnostic tests such as echocardiograms and cardiac MRI are carried out and interpreted, models that learn patterns from their training data that do not hold in new populations, and the biases that clinicians may bring, consciously or not, when labelling the data used to train them.

    A particular concern is that AI could worsen health disparities. Women and minority groups have historically been under-represented in cardiology research, so much of the current evidence may not apply to them. The authors give examples of inequalities that end up in the data used to train AI: men are 2.5 times more likely than women to be referred to a cardiologist for chest pain, and Black patients in emergency departments are 40% less likely to receive pain medication than White patients. In another example, an algorithm identified patients’ postcodes, linked to low-income and predominantly Black neighbourhoods, as the main predictor of a longer hospital stay.

    To prevent this, the authors call for clinicians, ethicists and representatives of minority groups to be involved from the early stages of AI development, for algorithms to be validated in minority and low-income groups, and for regulatory and ethical guidelines to ensure that the benefits of AI are shared equally, along with regular monitoring of AI models once they are in use.

    Learn more about this article here: https://doi.org/10.1016/S2589-7500(20)30249-1


    Reference:

    Tat, E., Bhatt, D.L., & Rabbat, M.G. (2020). Addressing bias: artificial intelligence in cardiovascular medicine. The Lancet Digital Health, 2(12), e635–e636.

  • Fairness in Cardiac MR Image Analysis: An Investigation of Bias Due to Data Imbalance in Deep Learning Based Segmentation

    Fairness in Cardiac MR Image Analysis: An Investigation of Bias Due to Data Imbalance in Deep Learning Based Segmentation

    AI is increasingly used to automatically outline the structures of the heart on cardiac MRI scans, a step that produces measurements used to guide patient care. This study examined whether one of these AI tools performs equally well across racial and gender groups, using cardiac MRI scans from 5,903 participants in the UK Biobank.

    The researchers found no difference in performance between men and women, in a dataset that was balanced by gender. However, they found significant racial bias: the tool was most accurate for White participants and less accurate for all other groups, including Black, Asian, Chinese and mixed-race participants. The authors attribute this to the racial imbalance in the training data, which was mostly made up of White participants.

    The researchers then tested three strategies to reduce this bias, including making sure each racial group was equally represented in every batch of training data and training separate models for each group. All three improved fairness compared with the original model, with separate models for each group giving the best results. The authors conclude that simply leaving race out of the model is not enough to ensure fairness, and that this becomes even more important as AI tools are used with less direct oversight from clinicians.

    Learn more about this study here: https://doi.org/10.1007/978-3-030-87199-4_39


    Reference:

    Puyol-Antón, E., Ruijsink, B., Piechnik, S.K., Neubauer, S., Petersen, S.E., Razavi, R., & King, A.P. (2021). Fairness in Cardiac MR Image Analysis: An Investigation of Bias Due to Data Imbalance in Deep Learning Based Segmentation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 (Lecture Notes in Computer Science, Vol. 12903, pp. 413–423). Springer.

  • An Investigation into the Impact of Deep Learning Model Choice on Sex and Race Bias in Cardiac MR Segmentation

    An Investigation into the Impact of Deep Learning Model Choice on Sex and Race Bias in Cardiac MR Segmentation

    Training data is not the only factor that can make AI tools less accurate for some groups of patients. The choice of model can also play a role. This study compared four AI models used to outline the structures of the heart on cardiac MRI scans, using data from the UK Biobank and varying the proportion of men and women, and of Black and White people, in the training data.

    Three of the four models showed bias by sex, with better results for men when men were the majority or when the training data was balanced. All four models showed bias by race. In general, increasing the share of a group in the training data improved the model’s performance for that group, but the size of this effect varied considerably from one model to another.

    The authors conclude that representation of different groups in training data matters, but that the choice of model also strongly affects bias, so each model needs to be carefully evaluated before use.

    Learn more about this study here: https://doi.org/10.1007/978-3-031-45249-9_21


    Reference:

    Lee, T., Puyol-Antón, E., Ruijsink, B., Aitcheson, K., Shi, M., & King, A.P. (2023). An Investigation into the Impact of Deep Learning Model Choice on Sex and Race Bias in Cardiac MR Segmentation. In S. Wesarg et al. (Eds.), Clinical Image-Based Procedures, Fairness of AI in Medical Imaging, and Ethical and Philosophical Issues in Medical Imaging (Lecture Notes in Computer Science, Vol. 14242). Springer.

  • A Systematic Study of Race and Sex Bias in CNN-Based Cardiac MR Segmentation

    A Systematic Study of Race and Sex Bias in CNN-Based Cardiac MR Segmentation

    AI is increasingly used to automatically outline the structures of the heart on cardiac MRI scans, a step that produces the measurements doctors use to guide patient care. One of the main causes of bias in these tools is imbalance in the data used to train them. This study is described by its authors as the first systematic investigation of how imbalance by race and sex in training data affects bias in this type of AI.

    The researchers trained several AI models on cardiac MRI scans, varying the proportion of people of different races and sexes in the training data each time. They found no significant bias in the experiment on sex, but significant bias in the two experiments on race.

    The authors conclude that the results highlight the need to ensure that different demographic groups are adequately represented in health datasets.

    Learn more about this study here: https://doi.org/10.1007/978-3-031-23443-9_22


    Reference:

    Lee, T., Puyol-Antón, E., Ruijsink, B., Shi, M., & King, A.P. (2022). A Systematic Study of Race and Sex Bias in CNN-Based Cardiac MR Segmentation. In Statistical Atlases and Computational Models of the Heart (STACOM 2022) (Lecture Notes in Computer Science, Vol. 13593, pp. 233–244). Springer.

  • Assessing racial bias in type 2 diabetes risk prediction algorithms

    Assessing racial bias in type 2 diabetes risk prediction algorithms

    Risk prediction tools are widely used to identify people at high risk of type 2 diabetes, so that they can be offered prevention and early treatment. This study examined whether three commonly used tools in the US (the Prediabetes Risk Test, the Framingham Offspring Risk Score and the ARIC model) are equally accurate for non-Hispanic White and non-Hispanic Black adults.

    The researchers used data from 9,987 adults without a diabetes diagnosis who took part in national health surveys between 1999 and 2010, and compared the risk predicted by each tool with the rates of type 2 diabetes actually observed. All three tools were miscalibrated by race, and this pattern was consistent across the years studied. The Framingham score overestimated risk for White adults and underestimated it for Black adults, while the other two tools overestimated risk for both groups, but more so for White adults.

    According to the authors, this means that White adults may be prioritised for prevention, with a greater risk of overdiagnosis and overtreatment, while Black adults may be underprioritised and undertreated. The findings highlight the need to reassess these tools to ensure that diabetes risk is predicted fairly across racial groups.

    Learn more about this study here: https://doi.org/10.1371/journal.pgph.0001556


    Reference:

    Cronjé, H.T., Katsiferis, A., Elsenburg, L.K., Andersen, T.O., Rod, N.H., Nguyen, T.-L., & Varga, T.V. (2023). Assessing racial bias in type 2 diabetes risk prediction algorithms. PLOS Global Public Health, 3(5), e0001556.

  • Implications of Bias in Artificial Intelligence: Considerations for Cardiovascular Imaging

    Implications of Bias in Artificial Intelligence: Considerations for Cardiovascular Imaging

    Cardiovascular disease is the leading cause of death worldwide, but its prevalence, burden and outcomes vary across gender and racial groups. Some biomarkers also vary between populations and ethnic and racial groups. Inequalities in who is included in clinical trials, how the disease presents, and how it is diagnosed and treated are preserved in the health data later used to train AI, which can lead to biased model performance.

    This review is intended as a guide for AI development teams working in cardiovascular imaging. It describes the implicit and explicit biases in cardiovascular care that may be present in existing datasets without being obvious to developers, and explains how assumptions in the data and in the outcomes chosen to train and evaluate a model can affect how it performs in the real world.

    The authors review disparities in cardiovascular outcomes across gender and racial groups, differences in the treatment of historically marginalised groups, and gaps in clinical trial representation. They also summarise research showing bias in cardiovascular AI, and highlight that AI can also help reduce bias, for example through tools that audit models for bias.

    Learn more about this review here: https://doi.org/10.1007/s11883-024-01190-x


    Reference:

    van Assen, M., Beecy, A., Gershon, G., Newsome, J., Trivedi, H., & Gichoya, J. (2024). Implications of Bias in Artificial Intelligence: Considerations for Cardiovascular Imaging. Current Atherosclerosis Reports, 26, 91–102

  • A Narrative Review on Ethical Considerations and Challenges in AI-Driven Cardiology

    A Narrative Review on Ethical Considerations and Challenges in AI-Driven Cardiology

    AI is transforming cardiology, improving the accuracy of diagnosis, the prediction of outcomes and treatment planning. However, its use also raises ethical concerns, including bias, privacy, accountability and the risk of making healthcare less human. This narrative review explores these challenges, focusing on bias in training data, data privacy, the “black-box” nature of many AI systems, and the need for transparency and accountability in clinical decisions.

    The authors explain that if the data used to train an AI model does not adequately represent minority groups or people with lower socioeconomic status, the model may be less accurate for them. This is particularly concerning in cardiology, because conditions such as hypertension, diabetes and heart disease disproportionately affect marginalised populations. They describe examples such as a cardiovascular risk tool that favoured White patients because its training data was not diverse enough, and AI-powered wearable devices that people on lower incomes could not access.

    To address these risks, the authors call for diverse datasets that represent different ethnicities, genders and socioeconomic backgrounds, testing of AI tools across demographic groups, continuous monitoring once they are in use, and close collaboration between clinicians and AI. They conclude that the success of AI in cardiology depends on strong frameworks that promote fairness, transparency and privacy, and note that real-world validation is still lacking.

    Learn more about this review here: https://doi.org/10.1097/MS9.0000000000003349


    Reference:

    Patel, D., Chetarajupalli, C., Khan, S., Khan, S., Patel, T., Joshua, S., & Millis, R.M. (2025). A Narrative Review on Ethical Considerations and Challenges in AI-Driven Cardiology. Annals of Medicine & Surgery, 87, 4152–4164.

  • The Need for Ethnoracial Equity in Artificial Intelligence for Diabetes Management: Review and Recommendations

    The Need for Ethnoracial Equity in Artificial Intelligence for Diabetes Management: Review and Recommendations

    Diabetes outcomes differ across ethnic and racial groups, yet AI tools for diabetes management are often developed without taking this into account. To assess how well these groups are represented, the authors re-analysed 141 articles from an earlier review of AI in diabetes care.

    Only 10 of the 141 articles (7.1%) reported the ethnic or racial background of their participants. In those that reported race, participants were on average 69.5% White, 17.1% Black and 3.7% Asian, and only two articles included Native American participants. According to the authors, AI tools trained on such unrepresentative data risk producing biased results for the groups that are left out.

    To address this, the authors propose a short questionnaire that researchers and clinicians can use to assess ethnoracial equity in AI diabetes research. It covers whether ethnoracial differences in the disease are discussed, whether the training data and its limitations are described, whether the training population reflects the population the tool will serve, and whether strategies to reduce bias are in place. As the authors put it, “AI can either mitigate or exacerbate disparities in health care.”

    Learn more about this article here: https://doi.org/10.2196/22320


    Reference:

    Pham, Q., Gamble, A., Hearn, J., & Cafazzo, J.A. (2021). The Need for Ethnoracial Equity in Artificial Intelligence for Diabetes Management: Review and Recommendations. Journal of Medical Internet Research, 23(2), e22320.

  • Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity

    Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity

    AI models used in clinical care are often trained on data that does not represent the full diversity of patients, which can lead to biased results and worsen health disparities. Experience in drug and medical device development has already shown the dangers of under-representing women and minority populations. This systematic review assessed how often published clinical AI studies report the gender and racial composition of the data used to train their models.

    Of the 390 studies included, 84% did not report the racial composition of their training data and 31% did not report gender. Models developed in the US did slightly better, with 56% reporting race and 77% reporting gender. Only 16% of all models used publicly available, non-proprietary datasets.

    The authors conclude that the low level of demographic reporting and the limited use of open data raise serious concerns about the transparency, generalisability and fairness of clinical AI models. They call for urgent, standardised reporting of gender and racial composition in training data to ensure these technologies are used ethically and equitably.

    Learn more about this review here: https://doi.org/10.1093/oodh/oqaf027


    Reference:

    Otokiti, A., Shih, H., & Williams, K. (2025). Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity. Oxford Open Digital Health, oqaf027.

  • Trustworthy and ethical AI-enabled cardiovascular care: a rapid review

    Trustworthy and ethical AI-enabled cardiovascular care: a rapid review

    AI is increasingly used in the prevention, diagnosis, monitoring and treatment of cardiovascular disease, but it also raises ethical concerns and some mistrust among patients and healthcare professionals. This rapid review analysed 145 papers to identify these concerns and the factors that build or undermine trust in AI, from the perspective of both patients and healthcare providers.

    The review identified six main ethical concerns: privacy and data security (raised in 40.7% of papers), healthcare inequity (24.8%), risk of harm to patients, accountability, informed consent and loss of autonomy, and data ownership. On equity, the main concern was that AI trained on data that does not represent all patients may be less accurate for under-represented groups, such as racial and ethnic minorities, women and people on low incomes, which could widen existing gaps in health outcomes.

    Barriers to trust included concerns about privacy, risk of harm, lack of transparency, fear that AI could replace the human side of care, and limited evidence on its accuracy and limitations. Factors that could increase trust included protecting and validating data, more diverse clinical trials, training and involving patients and professionals, and stronger regulatory oversight. The authors conclude that integrating AI into cardiovascular care will require stronger oversight of how patient data is used and greater transparency about how AI is used in care.

    Learn more about this review here: https://doi.org/10.1186/s12911-024-02653-6


    Reference:

    Mooghali, M., Stroud, A.M., Yoo, D.W., Barry, B.A., Grimshaw, A.A., Ross, J.S., Zhu, X., & Miller, J.E. (2024). Trustworthy and ethical AI-enabled cardiovascular care: a rapid review. BMC Medical Informatics and Decision Making, 24, 247.

  • Artificial intelligence bias in the prediction and detection of cardiovascular disease

    Artificial intelligence bias in the prediction and detection of cardiovascular disease

    AI can help identify people at risk of cardiovascular disease, detect it early and allow timely intervention. However, bias can arise at any step in the development, validation and use of these algorithms, causing them to perform poorly for historically marginalised groups. This article discusses the sources and consequences of AI bias in predicting and detecting cardiovascular disease, and presents strategies to reduce it.

    Bias can come from training data that does not represent the population a tool is meant for, from measurements or devices that work less well for some groups, and from leaving out important factors such as social determinants of health. It can also reflect inequalities in care: because women and ethnic minority patients are more likely to have their cardiovascular disease missed or diagnosed late, algorithms trained on this data may be less accurate at detecting it in these groups. Unequal access to digital technologies can further limit how well some groups are represented in the data. The authors give several examples of AI models that performed worse for women, Black patients or older adults.

    To reduce bias, the authors propose an AI health equity framework covering the whole lifecycle of an algorithm: diverse research teams, early involvement of patients, representative and balanced data, the inclusion of social determinants of health, input from clinicians, validation across different groups, transparent reporting of results and monitoring after implementation. They conclude that, without these measures, AI bias could worsen existing healthcare inequities and miss opportunities to predict and detect cardiovascular disease.

    Learn more about this article here: https://doi.org/10.1038/s44325-024-00031-9


    Reference:

    Mihan, A., Pandey, A., & Van Spall, H.G.C. (2024). Artificial intelligence bias in the prediction and detection of cardiovascular disease. npj Cardiovascular Health, 1, 31.

  • The bias algorithm: how AI in healthcare exacerbates ethnic and racial disparities – a scoping review

    The bias algorithm: how AI in healthcare exacerbates ethnic and racial disparities – a scoping review

    AI health algorithms are increasingly used to predict risk and support treatment decisions, often outperforming humans in diagnostic accuracy. However, they can also underestimate risk for marginalised and underserved patients, delaying the treatment they need. This scoping review analysed 23 sources published between 2020 and 2024 to examine racial and ethnic bias in these algorithms, the role of oversight and stakeholders, and how this bias can be reduced.

    A common explanation for this bias is that algorithms are trained on data in which majority populations are overrepresented, making them less reliable for others. Black, Native American, Asian and Latinx populations are often underrepresented in health datasets. Bias can also come from people: racial stereotypes absorbed during medical training can affect how clinicians assess patients, and these assessments end up in the health records used to train algorithms. The authors also describe a feedback loop: when patients from marginalised groups distrust the health system and use it less, data about them becomes even less representative, and the algorithms less accurate for them.

    To address this, the review highlights the need for continuous monitoring of algorithms once they are in use, in a similar way to how medicines are monitored after approval, and for retraining or withdrawing those that create inequities. It also stresses that simply including or removing race from an algorithm does not solve the problem, and that responsibility should be shared by developers, health systems, clinicians, regulators and patients. The authors conclude that, without this oversight, AI in healthcare risks entrenching health inequities.

    Learn more about this review here: https://doi.org/10.1080/13557858.2024.2422848


    Reference:

    Hussain, S.A., Bresnahan, M., & Zhuang, J. (2025). The bias algorithm: how AI in healthcare exacerbates ethnic and racial disparities – a scoping review. Ethnicity & Health, 30(2), 197–214.

  • Addressing hidden risks: Systematic review of artificial intelligence biases across racial and ethnic groups in cardiovascular diseases

    Addressing hidden risks: Systematic review of artificial intelligence biases across racial and ethnic groups in cardiovascular diseases

    AI models are increasingly used in cardiovascular medicine, but there are concerns that they may not work equally well for people of different racial and ethnic backgrounds. This systematic review examined how accurate and clinically applicable AI models for diagnosing heart disease and predicting outcomes are across racial and ethnic groups.

    Of 1,704 references screened, 11 studies were included, covering diagnosis, prognosis and the analysis of heart images. Nine of them (82%) found racial or ethnic bias, while two found no differences between groups. For example, an AI model predicting heart failure from electrocardiograms was less accurate for young Black patients than for young Asian or White patients, and AI tools analysing cardiac MRI scans were more accurate for White people than for people from minority ethnic groups.

    The authors conclude that AI models can perform well for some groups and less accurately for others, and that bias should be addressed at every stage, from data collection to clinical use. They recommend methods to reduce bias, standardised ways of measuring fairness, considering social determinants of health rather than race alone, and greater transparency through the sharing of code and data.

    Learn more about this review here: https://doi.org/10.1016/j.ejrad.2024.111867


    Reference:

    Cau, R., Pisu, F., Suri, J.S., & Saba, L. (2024). Addressing hidden risks: Systematic review of artificial intelligence biases across racial and ethnic groups in cardiovascular diseases. European Journal of Radiology, 111867.

  • Artificial Intelligence to Promote Racial and Ethnic Cardiovascular Health Equity

    Artificial Intelligence to Promote Racial and Ethnic Cardiovascular Health Equity

    AI has been seen as a tool that could help detect and reduce the racial and ethnic inequities that persist in cardiovascular care. However, AI can also carry and spread biases of its own, which means it must be approached with care. This review looks at the role of AI in cardiology in identifying and addressing these inequities, and in promoting diversity in research, as well as the concerns its use raises and strategies for using it fairly.

    The authors describe how AI has already helped reduce some inequities. In one study, an AI tool identified patients with severe heart valve disease and alerted their doctors, improving follow-up care for patients of all races. Even so, Black patients still faced greater delays in receiving appropriate care than White patients, showing that AI is not a “silver bullet” for eliminating inequities. AI can also perpetuate existing biases, due to limited diversity in training data, biases within the datasets themselves, and inadequate mechanisms for detecting and monitoring bias. For example, an AI model trained mostly on data from White patients to predict heart failure from electrocardiograms performed worse for Black patients, and particularly for Black women.

    To use AI in cardiovascular care in a way that addresses racial and ethnic inequities, the authors stress the need for careful design, starting with meticulous data collection and a thorough review of training data. They also call for rigorous testing and continuous monitoring of algorithms to ensure they perform equally well for all groups, as well as greater diversity in the AI workforce and the involvement of stakeholders.

    Learn more about this review here: https://doi.org/10.1007/s12170-024-00745-6


    Reference:

    Amponsah, D., Thamman, R., Brandt, E., James, C., Spector-Bagdady, K., & Yong, C.M. (2024). Artificial Intelligence to Promote Racial and Ethnic Cardiovascular Health Equity. Current Cardiovascular Risk Reports, 18(11), 153–162.

  • Algorithmic Bias in AI-Based Diabetes Care: Systematic Review of Model Performance, Equity Reporting, and Physiological Label Bias

    Algorithmic Bias in AI-Based Diabetes Care: Systematic Review of Model Performance, Equity Reporting, and Physiological Label Bias

    AI has transformed diabetes management, particularly through tools that predict glucose levels and support clinical decisions. However, there are concerns that these technologies may not work equally well for different racial, ethnic and socioeconomic groups. This systematic review assessed how fair AI-based glucose prediction models are, focusing on differences in performance between groups, how fairness is reported, and biases linked to physiological measures.

    Of 1,243 articles initially identified, only 10 met the criteria for inclusion. Just one of them explicitly evaluated whether AI model performance differed by race. Reporting on fairness was rare: only 7% of AI diabetes studies documented ethnic or racial data, and almost none carried out fairness audits. The review also highlights known physiological differences: for the same average glucose level, African American patients tend to have higher HbA1c values than White patients. If HbA1c is used as a reference to train AI models, this difference can be built into them. Yet none of the AI studies reviewed took this into account.

    The authors also note that traditional diabetes risk scores overestimate risk for White individuals and underestimate it for Black individuals, a bias that AI systems trained on these scores could inherit. They conclude that, without systematic efforts to reduce bias, AI-based diabetes technologies risk worsening existing health disparities, and call for future research to prioritise transparency, representativeness and fairness.

    Learn more about this review here: https://doi.org/10.61186/ist.202502.05.04


    Reference:

    Alipour, M., & Alipour, A. (2025). Algorithmic Bias in AI-Based Diabetes Care: Systematic Review of Model Performance, Equity Reporting, and Physiological Label Bias. InfoScience Trends, 2(5), 33–46.

  • Equity in Healthcare: Analyzing Disparities in Machine Learning Predictions of Diabetic Patient Readmissions

    Equity in Healthcare: Analyzing Disparities in Machine Learning Predictions of Diabetic Patient Readmissions

    Machine learning models can help predict which patients with diabetes are likely to be readmitted to hospital, but they need to work fairly for all patients. This study compared several of these models, including deep learning, generalised linear models, gradient boosting machines and naive Bayes, assessing not only their accuracy but also how fair their predictions were across age, gender and race.

    The gradient boosting model performed best overall, with an accuracy of 82.2% and consistently low error rates for most racial and gender groups. However, across several of the models, African American and Hispanic patients had higher rates of false positives, meaning they were more likely to be wrongly predicted to be readmitted. No single model performed best for every group: some worked better for certain groups of patients and others for other groups.

    The authors conclude that there is no one-size-fits-all solution, and that models need to be chosen and adjusted carefully to ensure they are both accurate and fair for all patients. According to them, fair algorithms can help reduce disparities and improve outcomes for patients with diabetes from all backgrounds.

    Learn more about this study here: https://doi.org/10.1109/ICHI61247.2024.00105


    Reference:

    Al-Zanbouri, Z., Sharma, G., & Raza, S. (2024). Equity in Healthcare: Analyzing Disparities in Machine Learning Predictions of Diabetic Patient Readmissions. In 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI) (pp. 660–669). IEEE.

  • Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study

    Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study

    Large language models such as GPT-4 are seen as promising tools in health care, from automating administrative tasks to supporting clinical decisions. However, they also risk perpetuating biases and giving incorrect diagnoses, with a direct impact on patient care. This study assessed whether GPT-4 shows racial and gender biases in four possible uses: medical education, diagnostic reasoning, treatment planning and assessments of patients.

    The researchers used clinical cases from medical education resources and from published research on implicit bias in health care. When asked to create clinical cases, GPT-4 did not reflect the real demographic diversity of medical conditions, and consistently produced cases that stereotyped how conditions present in different groups. For the same clinical cases, its lists of possible diagnoses were more likely to include diagnoses that stereotype certain races, ethnicities and genders.

    The treatment plans and assessments it produced also varied with patients’ demographic characteristics, including recommendations for more expensive procedures and differences in how patients were perceived. The authors conclude that tools like GPT-4 urgently need thorough and transparent bias assessments for each intended use before they are introduced into clinical care.

    Learn more about this study here: https://doi.org/10.1016/S2589-7500(23)00225-X


    Reference:

    Zack, T., Lehman, E., Suzgun, M., Rodriguez, J.A., Celi, L.A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D.W., Abdulnour, R.-E.E., Butte, A.J., & Alsentzer, E. (2024). Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. The Lancet Digital Health, 6(1), e12–e22.

  • Racial disparities in continuous glucose monitoring-based 60-min glucose predictions among people with type 1 diabetes

    Racial disparities in continuous glucose monitoring-based 60-min glucose predictions among people with type 1 diabetes

    Continuous glucose monitors, small sensors worn on the body that track blood sugar levels throughout the day, are increasingly combined with AI models that predict where glucose levels are heading. Because White people are overrepresented in medical research, these models may perform worse for other groups if they are trained mainly on White patients’ data. This study tested this using data from 205 adults and children with type 1 diabetes in the US, 101 White and 104 Black.

    The researchers trained AI models to predict glucose levels 60 minutes ahead, using datasets with different proportions of White and Black participants, from 0% to 100% White. As the proportion of White participants in the training data increased, prediction errors grew slightly more for Black participants than for White participants. The difference was small but statistically significant.

    When the models were then personalised to each individual, using a technique called transfer learning, this difference became smaller and was no longer statistically significant. The authors note that the differences may not be large enough to matter clinically on their own, but conclude that diverse training data is important for fair prediction models, and that personalisation may help build fairer tools.

    Learn more about this study here: https://doi.org/10.1371/journal.pdig.0000918


    Reference:

    Thomsen, H.B., Li, L.Y., Isaksen, A.A., Lebiecka-Johansen, B., Bour, C., Fagherazzi, G., van Doorn, W.P.T.M., Varga, T.V., & Hulman, A. (2025). Racial disparities in continuous glucose monitoring-based 60-min glucose predictions among people with type 1 diabetes. PLOS Digital Health, 4(6), e0000918.

  • Real-world artificial intelligence-based opportunistic screening for diabetic retinopathy in endocrinology and indigenous healthcare settings in Australia

    Real-world artificial intelligence-based opportunistic screening for diabetic retinopathy in endocrinology and indigenous healthcare settings in Australia

    Diabetic retinopathy, damage to the retina caused by diabetes, can lead to vision loss if it is not detected early. This study tested an AI system for diabetic retinopathy screening in real-world conditions, in two endocrinology clinics in Melbourne and three Aboriginal medical services in Western Australia, and also gathered the views of patients and health professionals.

    Of the 236 people screened, 203 were included in the main analysis. Overall, the system identified diabetic retinopathy requiring referral with a sensitivity of 96.9% and a specificity of 87.7%, and over 93% of participants were satisfied with the service. However, specificity was substantially lower among participants recruited from Aboriginal medical services, meaning that more people without the condition were wrongly flagged for referral. The study used clinic location as an approximation of Indigenous status.

    The authors note that the AI system had been trained exclusively on images from Chinese adults, and that differences in retinal pigmentation may affect its accuracy. They also point to other possible factors, such as poorer image quality, older age and other eye conditions. These explanations were not tested in the study.

    Learn more about this study here: https://doi.org/10.1038/s41598-021-94178-5


    Reference:

    Scheetz, J., Koca, D., McGuinness, M., Holloway, E., Tan, Z., Zhu, Z., O’Day, R., Sandhu, S., MacIsaac, R.J., Gilfillan, C., Turner, A., Keel, S., & He, M. (2021). Real-world artificial intelligence-based opportunistic screening for diabetic retinopathy in endocrinology and indigenous healthcare settings in Australia. Scientific Reports, 11, 15808.

  • Assessing and Mitigating Bias in Medical Artificial Intelligence: The Effects of Race and Ethnicity on a Deep Learning Model for ECG Analysis

    Assessing and Mitigating Bias in Medical Artificial Intelligence: The Effects of Race and Ethnicity on a Deep Learning Model for ECG Analysis

    AI models developed using data from very similar populations may not work as well for other groups, and could reflect or even worsen existing racial and ethnic disparities in health care. This study tested whether this was the case for an AI model that uses electrocardiograms (ECGs) to detect a weakened heart pumping function, known as low left ventricular ejection fraction.

    The model had been developed with data from nearly 45,000 patients, 96.2% of whom were non-Hispanic White. When tested on a separate group of almost 53,000 patients, it performed consistently well across all the racial and ethnic groups analysed: non-Hispanic White, Asian, Black/African American, Hispanic/Latino and American Indian/Native Alaskan. A separate AI model was able to identify patients’ racial group from the ECG to some extent, showing that ECG characteristics do vary by race, but this did not affect the first model’s performance.

    The authors conclude that, in this case, racial differences in the ECG did not affect the model’s ability to detect the condition. Even so, they recommend that the performance of all new AI tools in medicine be reported across different ethnic, racial, age and sex groups, to ensure they are used responsibly.

    Learn more about this study here: https://doi.org/10.1161/CIRCEP.119.007988


    Reference:

    Noseworthy, P.A., Attia, Z.I., Brewer, L.C., Hayes, S.N., Yao, X., Kapa, S., Friedman, P.A., & Lopez-Jimenez, F. (2020). Assessing and Mitigating Bias in Medical Artificial Intelligence: The Effects of Race and Ethnicity on a Deep Learning Model for ECG Analysis. Circulation: Arrhythmia and Electrophysiology, 13(3), e007988.

  • Artificial intelligence-enhanced electrocardiography for accurate diagnosis and management of cardiovascular diseases

    Artificial intelligence-enhanced electrocardiography for accurate diagnosis and management of cardiovascular diseases

    Electrocardiography (ECG) enhanced by artificial intelligence is emerging as a promising tool for diagnosing and managing cardiovascular disease. This review explains how AI can detect subtle patterns in the ECG that human interpreters may miss, helping to identify problems such as arrhythmias and heart failure more quickly and consistently.

    However, the authors also highlight important limitations, including systematic bias. When training datasets do not adequately represent different ages, sexes and racial groups, models can perform worse for those who are underrepresented, since ECG patterns vary with age, physiological differences between the sexes and genetic factors. To make these tools safe and fair in clinical practice, the review calls for more diverse datasets, bias mitigation strategies and continuous validation across populations.

    Learn more about this review here: https://doi.org/10.1016/j.jelectrocard.2024.01.006


    Reference:

    Muzammil, M.A., Javid, S., Afridi, A.K., Siddineni, R., Shahabi, M., Haseeb, M., Fariha, F., Kumar, S., Zaveri, S., & Nashwan, A.J. (2024). Artificial intelligence-enhanced electrocardiography for accurate diagnosis and management of cardiovascular diseases. Journal of Electrocardiology, 83, 30–4

  • Bio-Med AI Summer School debates the role of AI in healthcare

    Bio-Med AI Summer School debates the role of AI in healthcare

    In September 2024, the National Library of Portugal in Lisbon hosted the Bio-Med AI Summer School, a three-day international event on the role of artificial intelligence in healthcare. Organised by whatnext.law, a research centre of NOVA School of Law and Vieira de Almeida, in partnership with the Biomedical Research Foundation of the Academy of Athens, it brought together academics, professionals and students from health, law, technology and ethics. The goal was to bridge the gap between AI and the fields of biomedicine and health.

    Speakers shared examples of how AI is already being used in Portugal and elsewhere. These included summarising medical consultations so doctors can focus on their patients, chatbots that help patients book appointments, detecting signs of prostate cancer in MRI scans, drafting reports from medical images, and a large language model designed to interpret mental health questions.

    Participants also stressed that these tools must remain tools, with the final decision always resting with doctors. They highlighted the importance of knowing where data comes from, ensuring its quality and complying with European regulation, such as the AI Act. In the closing session, former Minister of Health Adalberto Campos Fernandes highlighted the role AI could play in increasing access to healthcare worldwide and accelerating medical research.

    Learn more here: https://eco.sapo.pt/2024/09/18/inteligencia-artificial-uma-revolucao-ao-servico-da-saude/


    Reference:

    ECO. (2024, September 18). Inteligência artificial: Uma revolução ao serviço da saúde. ECO.