Tag: Gender and Racial Biases

  • AI-based diabetes care: risk prediction models and implementation concerns

    AI-based diabetes care: risk prediction models and implementation concerns

    AI is increasingly being used to predict a person’s risk of developing type 2 diabetes, which could help delay or prevent the disease through earlier, more personalised prevention. This editorial discusses a scoping review of 40 studies on AI models for diabetes risk prediction. It found that models combining electronic health records with other types of data, such as genetic or medical imaging data, performed better than models using a single type of data.

    However, the authors warn that these results should be approached with caution. Many of the models showed a pronounced imbalance in gender, ethnicity and age in the data used to train them, only five were tested on external data, and most were not evaluated across different demographic groups. Since Black and Mexican Americans have a higher prevalence of diabetes than White Americans, leaving certain groups out of training data could worsen health inequities for populations already at greater risk.

    To address this, the authors call for mandatory representation criteria for the approval and adoption of these models, tools to assess their risk of bias, and external validation. They argue that AI cannot be adopted on a large scale in healthcare until problems with data quality and bias are addressed, and that this requires collaboration between clinicians, researchers, policymakers and entrepreneurs.

    Learn more about this article here: https://doi.org/10.1038/s41746-024-01034-7


    Reference:

    Wang, S. C. Y., Nickel, G., Venkatesh, K. P., Raza, M. M., & Kvedar, J. C. (2024). AI-based diabetes care: Risk prediction models and implementation concerns. npj Digital Medicine, 7, 36.

  • Can AI fight sex and gender bias in healthcare?

    Can AI fight sex and gender bias in healthcare?

    AI is transforming medicine, but it is mostly developed by men and trained on data that prioritises men’s bodies and health needs. In this commentary, researchers from the George Institute for Global Health argue that this embeds sex and gender biases that can put women and nonbinary patients at risk. For example, natural language processing models used in psychiatry have shown significant gender biases, and tools that screen for suicide risk may make mistakes if trained mainly on text written by men, since men and women express distress differently.

    The authors also argue that AI could help reduce these inequities if sex and gender are considered deliberately. One example is the GRACE heart attack risk score, which was updated using AI models that analyse male and female data separately to account for sex-specific characteristics of the disease. Another is a chatbot developed by the George Institute to give guideline-based pregnancy advice in rural India, created together with community health workers, clinicians and women.

    They call for sex and gender bias to be actively addressed in the data used to train AI, for tools to be co-designed with the communities they serve, for work on bias to go beyond the English language, and for sex and gender to be considered at every stage of research and clinical use.

    Learn more here: https://www.unsw.edu.au/newsroom/news/2024/10/can-ai-fight-sex-and-gender-bias-in-healthcare-


    Reference:

    Haupt, S., Graham, B., & Hirst, J. (2024, October 21). Can AI fight sex and gender bias in healthcare? 360info.

  • Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations

    Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations

    AI systems now match expert performance in many medical imaging tasks, but there is growing concern that they may reflect and amplify human bias. This is especially worrying in the case of underdiagnosis, when AI wrongly labels a person who has a disease as healthy, which can delay their access to care. This study examined underdiagnosis by AI tools that analyse chest X-rays, using three large US datasets and a combination of all three, with more than 700,000 images from almost 130,000 patients.

    The AI tools consistently and selectively underdiagnosed under-served groups: female patients, patients under 20, Black patients, Hispanic patients and patients with Medicaid, a public health insurance for people with low incomes. Underdiagnosis was even higher for patients who belonged to more than one of these groups, such as Hispanic female patients, and for young patients who were also female, Black or on Medicaid.

    The authors warn that using AI tools with these biases could worsen existing inequalities in care and lead to unequal access to treatment, raising ethical concerns about using these models in clinical practice.

    Learn more about this study here: https://doi.org/10.1038/s41591-021-01595-0


    Reference:

    Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021). Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine, 27(12), 2176–2182.

  • Bias Mitigation in Primary Health Care Artificial Intelligence Models: Scoping Review

    Bias Mitigation in Primary Health Care Artificial Intelligence Models: Scoping Review

    AI models in primary care can help identify quickly and accurately the people who should receive care, but they risk perpetuating or amplifying biases against diverse groups. This scoping review looked at the strategies and methods used to reduce bias in primary care AI models, which groups were considered, and how these strategies affected both bias and model performance.

    Of the 17 studies included, race or ethnicity was the attribute most often examined (12 studies), followed by sex or gender (10 studies), usually defined only as male or female. The approaches to reducing bias fell into four groups: modifying existing models or datasets, sourcing data from electronic health records, developing tools with human input (“human-in-the-loop”), and identifying ethical principles to guide decisions. Adjusting the data before training the model, for example by relabelling or reweighting it, and analysing doctors’ written notes with language-processing tools showed the greatest potential. Other methods sometimes increased errors or made the models less well calibrated.

    The authors conclude that bias appears easier to reduce when data are open source, when multiple stakeholders are involved, and when bias is addressed before the model is trained. They call for further studies that include a wider range of groups, such as Indigenous peoples in Canada.

    Learn more about this review here: https://doi.org/10.2196/60269


    Reference:

    Sasseville, M., Ouellet, S., Rhéaume, C., Sahlia, M., Couture, V., Després, P., Paquette, J.-S., Darmon, D., Bergeron, F., & Gagnon, M.-P. (2025). Bias mitigation in primary health care artificial intelligence models: Scoping review. Journal of Medical Internet Research, 27, e60269.

  • Inteligência Artificial no setor da saúde: desafios jurídicos e regulação

    Inteligência Artificial no setor da saúde: desafios jurídicos e regulação

    Artificial Intelligence in the health sector: legal challenges and regulation

    AI can significantly improve the quality of health services, but its use also raises well-founded concerns. This paper, published by the Office for Strategy and Studies (GEE) of the Portuguese Ministry of Economy, analyses the current landscape of AI in the Portuguese health sector and proposes public policy solutions for Portugal.

    The author focuses on three main challenges: the opacity of AI systems, which can undermine patients’ informed consent; bias and discrimination; and the protection of health data and privacy. On bias, she explains that it comes mostly from the data used to train AI, which can reflect existing social prejudices and be skewed by factors such as age, sex, race and socioeconomic background. The paper also analyses how these systems fit within the EU AI Act, with particular attention to the obligation of human oversight.

    To reduce these risks, the author proposes measures such as ethical requirements in the development and choice of AI systems, impact assessments and independent data audits before approval, periodic evaluation with a mechanism for reporting failures, and training for health professionals so that they can understand AI and recognise its errors.

    Learn more about this paper here: https://EconPapers.repec.org/RePEc:mde:wpaper:187


    Reference:

    Paixão Cansado, M. (2024). Inteligência Artificial no setor da saúde: desafios jurídicos e regulação (GEE Paper No. 187). Gabinete de Estratégia e Estudos, Ministério da Economia.

  • Sociodemographic biases in medical decision making by large language models

    Sociodemographic biases in medical decision making by large language models

    Large language models (LLMs), such as the AI behind popular chatbots, show promise in healthcare, but they may make recommendations influenced by patients’ sociodemographic characteristics. Researchers tested nine LLMs on 1,000 emergency department cases, half real and half synthetic. Each case was presented in 32 versions: one for each of 31 sociodemographic groups, plus a control version with no sociodemographic details. The clinical details were always the same. In total, they analysed more than 1.7 million AI-generated recommendations.

    Compared with both a physician-defined baseline and the control cases, cases labelled as Black or unhoused or as LGBTQIA+ were more often directed towards urgent care, invasive interventions or mental health evaluations. For example, certain cases from LGBTQIA+ subgroups were recommended mental health assessments around six to seven times more often than clinically indicated. Cases labelled as high-income were more often recommended advanced imaging tests, such as CT or MRI, while low- and middle-income cases were often limited to basic tests or none at all.

    These differences were not supported by clinical reasoning or guidelines and may reflect bias in the models, which could lead to health disparities. As they were found in both proprietary and open-source models, the authors stress the need for robust ways to evaluate and reduce bias, so that AI-driven medical advice remains fair and centred on patients.

    Learn more about this study here: https://doi.org/10.1038/s41591-025-03626-6


    Reference:

    Omar, M., Soffer, S., Agbareia, R., Bragazzi, N. L., Apakama, D. U., Horowitz, C. R., Charney, A. W., Freeman, R., Kummer, B., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2025). Sociodemographic biases in medical decision making by large language models. Nature Medicine, 31, 1873–1881.

  • Addressing bias in big data and AI for health care: A call for open science

    Addressing bias in big data and AI for health care: A call for open science

    Most AI tools in healthcare learn from large datasets, but many groups of people have long been missing or misrepresented in medical data. When AI is trained on data that does not reflect the whole population, it can reinforce existing biases and lead to misdiagnoses and even fatal outcomes, particularly for gender and ethnic minorities.

    The authors give several examples. AI tools for classifying skin lesions are often trained on datasets in which only about 5% to 10% of images come from Black patients, and when tested on images of Black patients they reach about half the accuracy their creators claimed. This matters because Black patients have the highest mortality from melanoma. In cardiology, heart attacks are often misdiagnosed in women, yet some models that predict heart attacks were trained mostly on data from men. Many datasets also record gender only as male or female, making it difficult to build AI that includes LGBTQ+ people or to test it for bias.

    To move towards fairer AI, the authors propose open science practices: involving patients and communities in the development of AI, sharing data responsibly while protecting privacy, using inclusive data standards, and sharing code. Sharing code and algorithms allows researchers around the world to test whether a tool developed with data from one population also works for patients elsewhere.

    Learn more about this article here: https://doi.org/10.1016/j.patter.2021.100347


    Reference:

    Norori, N., Hu, Q., Aellen, F. M., Faraci, F. D., & Tzovara, A. (2021). Addressing bias in big data and AI for health care: A call for open science. Patterns, 2(10), 100347.

  • Mitigating the risk of artificial intelligence bias in cardiovascular care

    Mitigating the risk of artificial intelligence bias in cardiovascular care

    Digital health technologies produce large amounts of data that can be used to train AI, which is transforming cardiovascular care. However, AI can become biased when the data used to train it comes from a very similar group of people, or from healthcare processes that are themselves unequal. Bias can also be introduced while algorithms are developed, tested and put into practice, and even after they are already in use.

    Biased algorithms can lead to missed diagnoses, misclassified disease, inaccurate risk predictions and inappropriate treatment recommendations, and these harms can fall disproportionately on marginalised groups. This paper, part of a series in The Lancet Digital Health, gives an overview of how AI is used in cardiovascular care, explains where bias can arise at each stage of an algorithm’s development, and presents examples of harm caused by biased algorithms.

    The authors also propose strategies to reduce bias during the training, testing and implementation of algorithms, so that “all those at risk for or living with cardiovascular disease might benefit equally from AI.”

    Learn more about this article here: https://doi.org/10.1016/S2589-7500(24)00155-9


    Reference:

    Mihan, A., Pandey, A., & Van Spall, H. G. C. (2024). Mitigating the risk of artificial intelligence bias in cardiovascular care. The Lancet Digital Health, 6(10), e749–e754.

  • Guiding Principles to Address the Impact of Algorithm Bias on Racial and Ethnic Disparities in Health and Health Care

    Guiding Principles to Address the Impact of Algorithm Bias on Racial and Ethnic Disparities in Health and Health Care

    Algorithms are widely used in healthcare to support diagnosis, treatment, risk assessment and decisions about who receives resources. When they are biased, they can worsen outcomes for racial and ethnic minority groups and other marginalised populations. For example, an algorithm that used race to estimate kidney function gave higher results for Black patients, delaying their referral for organ transplants. Another, used to manage chronic diseases, effectively required Black patients to be sicker than White patients to qualify for extra care.

    To address this, two US federal agencies, the Agency for Healthcare Research and Quality and the National Institute on Minority Health and Health Disparities, brought together a diverse expert panel. The panel reviewed the evidence and heard from stakeholders, patients and community members. It developed a framework covering the whole life cycle of an algorithm, from defining the problem and choosing the data to development, deployment and monitoring.

    The framework is built on five guiding principles: promote health equity at every stage of an algorithm’s life cycle; ensure algorithms and their use are transparent and explainable; authentically engage patients and communities and earn their trust; explicitly identify fairness issues and trade-offs; and establish accountability for fair and equitable outcomes.

    Learn more about this article here: https://doi.org/10.1001/jamanetworkopen.2023.45050


    Reference:

    Chin, M. H., Afsar-Manesh, N., Bierman, A. S., Chang, C., Colón-Rodríguez, C. J., Dullabh, P., Duran, D. G., Fair, M., Hernandez-Boussard, T., Hightower, M., Jain, A., Jordan, W. B., Konya, S., Moore, R. H., Moore, T. T., Rodriguez, R., Shaheen, G., Snyder, L. P., Srinivasan, M., Umscheid, C. A., & Ohno-Machado, L. (2023). Guiding principles to address the impact of algorithm bias on racial and ethnic disparities in health and health care. JAMA Network Open, 6(12), e2345050.

  • Equity in medical devices: independent review

    Equity in medical devices: independent review

    Medical devices are often assumed to work equally well for everyone, but an independent review commissioned by the UK government shows this is not always the case. The review began after concerns that pulse oximeters, which measure oxygen levels in the blood, may be less accurate for patients with darker skin. It found that these devices overestimate oxygen levels in people with darker skin tones. In US studies, this has been linked to delayed recognition of disease and to denied or delayed treatment.

    The review found similar problems in other devices. Devices that measure bilirubin through the skin may overestimate its levels in babies with darker skin. Ventilators have default settings designed for men. AI tools that help detect skin cancer may have been trained mostly on images from people with fair skin, and an X-ray dataset that underrepresented women led to lower accuracy when diagnosing them. Genetic risk scores, which estimate a person’s risk of disease, are based on data that is overwhelmingly from people of European ancestry and may not work as well for other groups.

    To address these problems, the review makes 18 recommendations to the government, regulators, the NHS and device developers, from more diverse data and clinical trials to regular checks for bias once devices are in use.

    Learn more about this review here: https://www.gov.uk/government/publications/equity-in-medical-devices-independent-review-final-report/equity-in-medical-devices-independent-review-summary-report


    Reference:

    Whitehead, M., Ali, R., Carrol, E., Holmes, C., & Kee, F. (2024). Equity in medical devices: Independent review. Final report. Department of Health and Social Care.

  • Artificial intelligence in healthcare: Applications, risks, and ethical and societal impacts

    Artificial intelligence in healthcare: Applications, risks, and ethical and societal impacts

    This report, published by the European Parliamentary Research Service’s Scientific Foresight Unit (STOA), looks at how AI is being used in healthcare and the risks it brings. It identifies seven main risks: patient harm due to AI errors, misuse of medical AI tools, bias and the perpetuation of existing inequities, lack of transparency, privacy and security issues, gaps in accountability, and obstacles to implementation in real-world healthcare.

    On bias, the report explains that the most common cause of unfairness in medical AI is bias in the data used to train it. Some groups are underrepresented in medical datasets; for example, skin cancer datasets mostly contain fair-skinned patients. Data can also carry existing human biases, such as the dismissal of women’s pain or doctors taking Black patients’ pain less seriously. Among the examples given is a chest X-ray algorithm that most often missed disease in young female patients, Black patients and patients on public insurance for people with low incomes.

    To address these risks, the report proposes seven policy options. These include extending AI regulation to cover healthcare-specific risks, involving all stakeholders throughout the development of medical AI, creating an “AI passport” for transparency and traceability, defining accountability, educating health professionals and the public, promoting further research, and reducing the divide in medical AI across Europe.

    Learn more about this report here: https://doi.org/10.2861/568473


    Reference:

    Lekadir, K., Quaglio, G., Tselioudis Garmendia, A., & Gallin, C. (2022). Artificial intelligence in healthcare: Applications, risks, and ethical and societal impacts (PE 729.512). European Parliamentary Research Service, Scientific Foresight Unit (STOA).

  • Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare

    Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare

    Precision medicine aims to tailor prevention and treatment to each person, based on the genetic and environmental factors that shape their health. AI can help by finding relevant patterns in large amounts of biomedical data. This review examines how sex and gender, which contribute to differences in health and disease, are taken into account, or not, in the AI technologies used in this field.

    Differences between women and men have been reported in how many diseases develop, how common they are, how their symptoms present and how they respond to treatment, including in cardiovascular disease, diabetes and mental health. Yet, according to the authors, most biomedical AI technologies do not include ways of detecting bias, and most algorithms ignore sex and gender altogether. This can lead to poorer results, errors and discriminatory outcomes. In depression, for example, women are diagnosed more often than men, but this may partly reflect the fact that research has historically measured symptoms that are more common in women.

    The authors review the existing sex and gender gaps in several technologies used in precision medicine, and make recommendations on how to use them in ways that improve health outcomes for everyone and reduce inequalities.

    Learn more about this review here: https://doi.org/10.1038/s41746-020-0288-5


    Reference:

    Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M.J., Chadha, A.S., & Mavridis, N. (2020). Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare. npj Digital Medicine, 3, 81.

  • Health for whom? Intersectionality and biases in the use of artificial intelligence in clinical diagnosis

    Health for whom? Intersectionality and biases in the use of artificial intelligence in clinical diagnosis

    AI is growing rapidly in healthcare and can support clinical diagnosis, for example by analysing X-rays, MRI and CT scans to find patterns that may escape the human eye. In this Spanish editorial, the authors warn that enthusiasm for these tools can overlook the risk that they perpetuate or deepen disparities based on gender, race and ethnicity. They argue for an intersectional approach, which looks at how different social identities overlap and create combined forms of disadvantage.

    The authors explain that AI can reflect and amplify the biases in the data it learns from, including the unconscious biases of health professionals. They give several examples: AI systems analysing chest X-rays that more often missed disease in Hispanic patients and patients with public health insurance in the US than in White patients; medical records describing the symptoms of African American patients in pejorative terms; and a fracture risk calculator that underestimated risk in Black women, producing data that may later be used to train AI.

    They conclude that answering the question “health for whom?” requires moving beyond guidelines and good intentions to enforceable rules. They call for more diversity among researchers, better methods to detect intersectional bias, common standards for recording social identities, and legal frameworks that address under-representation and unconscious bias, not only safety and privacy.

    Learn more about this article here: https://doi.org/10.23938/ASSN.1077


    Reference:

    Amaya-Santos, S., Jiménez-Pernett, J., & Bermúdez-Tamayo, C. (2024). ¿Salud para quién? Interseccionalidad y sesgos de la inteligencia artificial para el diagnóstico clínico [Health for whom? Intersectionality and biases in the use of artificial intelligence in clinical diagnosis]. Anales del Sistema Sanitario de Navarra, 47(2), e1077.

  • ChatGPT Exhibits Gender and Racial Biases in Acute Coronary Syndrome Management

    ChatGPT Exhibits Gender and Racial Biases in Acute Coronary Syndrome Management

    Please note: this is a preprint study, which means it has not yet been peer reviewed.

    Large language models such as ChatGPT are increasingly being explored as tools to support clinical decisions, but there are concerns that they may reproduce gender and racial biases. This study tested whether ChatGPT 3.5 showed such biases when recommending how to manage acute coronary syndrome, a group of conditions that includes heart attacks and unstable angina.

    When patients were described as female, African American or Hispanic, ChatGPT recommended guideline-based diagnosis, treatment and symptom management less often. The largest gaps were in recommending coronary angiography or stress testing, used to diagnose the condition and plan further treatment, and high-intensity statins. These differences mirror biases already observed in clinical practice, which have been linked to worse outcomes for women and racial minorities with heart disease. The gaps were largest for unstable angina, where clinical guidelines are less explicit.

    The researchers also found that asking ChatGPT to explain its reasoning before giving an answer improved its clinical accuracy and reduced gender and racial bias. According to the authors, this is among the first studies to show that the biases of these models affect clinical management, and that existing strategies to improve their performance can also help reduce bias.

    Learn more about this study here: https://doi.org/10.48550/arXiv.2311.14703


    Reference:

    Zhang, A., Yuksekgonul, M., Guild, J., Zou, J., & Wu, J.C. (2023). ChatGPT Exhibits Gender and Racial Biases in Acute Coronary Syndrome Management. arXiv preprint arXiv:2311.14703.

  • A Systematic Study of Race and Sex Bias in CNN-Based Cardiac MR Segmentation

    A Systematic Study of Race and Sex Bias in CNN-Based Cardiac MR Segmentation

    AI is increasingly used to automatically outline the structures of the heart on cardiac MRI scans, a step that produces the measurements doctors use to guide patient care. One of the main causes of bias in these tools is imbalance in the data used to train them. This study is described by its authors as the first systematic investigation of how imbalance by race and sex in training data affects bias in this type of AI.

    The researchers trained several AI models on cardiac MRI scans, varying the proportion of people of different races and sexes in the training data each time. They found no significant bias in the experiment on sex, but significant bias in the two experiments on race.

    The authors conclude that the results highlight the need to ensure that different demographic groups are adequately represented in health datasets.

    Learn more about this study here: https://doi.org/10.1007/978-3-031-23443-9_22


    Reference:

    Lee, T., Puyol-Antón, E., Ruijsink, B., Shi, M., & King, A.P. (2022). A Systematic Study of Race and Sex Bias in CNN-Based Cardiac MR Segmentation. In Statistical Atlases and Computational Models of the Heart (STACOM 2022) (Lecture Notes in Computer Science, Vol. 13593, pp. 233–244). Springer.

  • Implications of Bias in Artificial Intelligence: Considerations for Cardiovascular Imaging

    Implications of Bias in Artificial Intelligence: Considerations for Cardiovascular Imaging

    Cardiovascular disease is the leading cause of death worldwide, but its prevalence, burden and outcomes vary across gender and racial groups. Some biomarkers also vary between populations and ethnic and racial groups. Inequalities in who is included in clinical trials, how the disease presents, and how it is diagnosed and treated are preserved in the health data later used to train AI, which can lead to biased model performance.

    This review is intended as a guide for AI development teams working in cardiovascular imaging. It describes the implicit and explicit biases in cardiovascular care that may be present in existing datasets without being obvious to developers, and explains how assumptions in the data and in the outcomes chosen to train and evaluate a model can affect how it performs in the real world.

    The authors review disparities in cardiovascular outcomes across gender and racial groups, differences in the treatment of historically marginalised groups, and gaps in clinical trial representation. They also summarise research showing bias in cardiovascular AI, and highlight that AI can also help reduce bias, for example through tools that audit models for bias.

    Learn more about this review here: https://doi.org/10.1007/s11883-024-01190-x


    Reference:

    van Assen, M., Beecy, A., Gershon, G., Newsome, J., Trivedi, H., & Gichoya, J. (2024). Implications of Bias in Artificial Intelligence: Considerations for Cardiovascular Imaging. Current Atherosclerosis Reports, 26, 91–102

  • Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity

    Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity

    AI models used in clinical care are often trained on data that does not represent the full diversity of patients, which can lead to biased results and worsen health disparities. Experience in drug and medical device development has already shown the dangers of under-representing women and minority populations. This systematic review assessed how often published clinical AI studies report the gender and racial composition of the data used to train their models.

    Of the 390 studies included, 84% did not report the racial composition of their training data and 31% did not report gender. Models developed in the US did slightly better, with 56% reporting race and 77% reporting gender. Only 16% of all models used publicly available, non-proprietary datasets.

    The authors conclude that the low level of demographic reporting and the limited use of open data raise serious concerns about the transparency, generalisability and fairness of clinical AI models. They call for urgent, standardised reporting of gender and racial composition in training data to ensure these technologies are used ethically and equitably.

    Learn more about this review here: https://doi.org/10.1093/oodh/oqaf027


    Reference:

    Otokiti, A., Shih, H., & Williams, K. (2025). Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity. Oxford Open Digital Health, oqaf027.

  • Trustworthy and ethical AI-enabled cardiovascular care: a rapid review

    Trustworthy and ethical AI-enabled cardiovascular care: a rapid review

    AI is increasingly used in the prevention, diagnosis, monitoring and treatment of cardiovascular disease, but it also raises ethical concerns and some mistrust among patients and healthcare professionals. This rapid review analysed 145 papers to identify these concerns and the factors that build or undermine trust in AI, from the perspective of both patients and healthcare providers.

    The review identified six main ethical concerns: privacy and data security (raised in 40.7% of papers), healthcare inequity (24.8%), risk of harm to patients, accountability, informed consent and loss of autonomy, and data ownership. On equity, the main concern was that AI trained on data that does not represent all patients may be less accurate for under-represented groups, such as racial and ethnic minorities, women and people on low incomes, which could widen existing gaps in health outcomes.

    Barriers to trust included concerns about privacy, risk of harm, lack of transparency, fear that AI could replace the human side of care, and limited evidence on its accuracy and limitations. Factors that could increase trust included protecting and validating data, more diverse clinical trials, training and involving patients and professionals, and stronger regulatory oversight. The authors conclude that integrating AI into cardiovascular care will require stronger oversight of how patient data is used and greater transparency about how AI is used in care.

    Learn more about this review here: https://doi.org/10.1186/s12911-024-02653-6


    Reference:

    Mooghali, M., Stroud, A.M., Yoo, D.W., Barry, B.A., Grimshaw, A.A., Ross, J.S., Zhu, X., & Miller, J.E. (2024). Trustworthy and ethical AI-enabled cardiovascular care: a rapid review. BMC Medical Informatics and Decision Making, 24, 247.

  • Artificial intelligence bias in the prediction and detection of cardiovascular disease

    Artificial intelligence bias in the prediction and detection of cardiovascular disease

    AI can help identify people at risk of cardiovascular disease, detect it early and allow timely intervention. However, bias can arise at any step in the development, validation and use of these algorithms, causing them to perform poorly for historically marginalised groups. This article discusses the sources and consequences of AI bias in predicting and detecting cardiovascular disease, and presents strategies to reduce it.

    Bias can come from training data that does not represent the population a tool is meant for, from measurements or devices that work less well for some groups, and from leaving out important factors such as social determinants of health. It can also reflect inequalities in care: because women and ethnic minority patients are more likely to have their cardiovascular disease missed or diagnosed late, algorithms trained on this data may be less accurate at detecting it in these groups. Unequal access to digital technologies can further limit how well some groups are represented in the data. The authors give several examples of AI models that performed worse for women, Black patients or older adults.

    To reduce bias, the authors propose an AI health equity framework covering the whole lifecycle of an algorithm: diverse research teams, early involvement of patients, representative and balanced data, the inclusion of social determinants of health, input from clinicians, validation across different groups, transparent reporting of results and monitoring after implementation. They conclude that, without these measures, AI bias could worsen existing healthcare inequities and miss opportunities to predict and detect cardiovascular disease.

    Learn more about this article here: https://doi.org/10.1038/s44325-024-00031-9


    Reference:

    Mihan, A., Pandey, A., & Van Spall, H.G.C. (2024). Artificial intelligence bias in the prediction and detection of cardiovascular disease. npj Cardiovascular Health, 1, 31.

  • Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study

    Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study

    Large language models such as GPT-4 are seen as promising tools in health care, from automating administrative tasks to supporting clinical decisions. However, they also risk perpetuating biases and giving incorrect diagnoses, with a direct impact on patient care. This study assessed whether GPT-4 shows racial and gender biases in four possible uses: medical education, diagnostic reasoning, treatment planning and assessments of patients.

    The researchers used clinical cases from medical education resources and from published research on implicit bias in health care. When asked to create clinical cases, GPT-4 did not reflect the real demographic diversity of medical conditions, and consistently produced cases that stereotyped how conditions present in different groups. For the same clinical cases, its lists of possible diagnoses were more likely to include diagnoses that stereotype certain races, ethnicities and genders.

    The treatment plans and assessments it produced also varied with patients’ demographic characteristics, including recommendations for more expensive procedures and differences in how patients were perceived. The authors conclude that tools like GPT-4 urgently need thorough and transparent bias assessments for each intended use before they are introduced into clinical care.

    Learn more about this study here: https://doi.org/10.1016/S2589-7500(23)00225-X


    Reference:

    Zack, T., Lehman, E., Suzgun, M., Rodriguez, J.A., Celi, L.A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D.W., Abdulnour, R.-E.E., Butte, A.J., & Alsentzer, E. (2024). Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. The Lancet Digital Health, 6(1), e12–e22.

  • Evaluating and mitigating bias in machine learning models for cardiovascular disease prediction

    Evaluating and mitigating bias in machine learning models for cardiovascular disease prediction

    Machine learning models are increasingly used to estimate a person’s risk of cardiovascular disease, but bias can be introduced when health data is collected and prepared, affecting how well these models work for certain groups. This study used electronic health records from 109,490 people treated at Vanderbilt University Medical Center, in the US, to test whether several machine learning models performed equally well across race and gender. It also compared them with a widely used risk equation from the American Heart Association that does not use machine learning.

    Most of the machine learning models were less biased than the American Heart Association equation. Differences between racial groups were small, but differences between women and men were considerably larger. According to the authors, both the risk equation and the machine learning models were biased against women.

    The researchers also tested three ways of reducing this bias. Removing race and gender from the data did not significantly reduce bias in most models, and balancing the number of people from each group in the training data did not consistently help either. Balancing the proportion of people with cardiovascular disease in each group reduced the gender bias, but not the racial bias, and slightly reduced accuracy in many cases. The authors conclude that gender disparities in cardiovascular risk prediction need to be investigated and corrected.

    Learn more about this study here: https://doi.org/10.1016/j.jbi.2023.104294


    Reference:

    Li, F., Wu, P., Ong, H.H., Peterson, J.F., Wei, W.-Q., & Zhao, J. (2023). Evaluating and mitigating bias in machine learning models for cardiovascular disease prediction. Journal of Biomedical Informatics, 138, 104294.

  • Race, Sex, and Age Disparities in the Performance of ECG Deep Learning Models Predicting Heart Failure

    Race, Sex, and Age Disparities in the Performance of ECG Deep Learning Models Predicting Heart Failure

    AI models that analyse electrocardiograms (ECGs) could help identify people at high risk of heart failure early, and potentially reduce racial disparities in heart failure outcomes. But whether these models perform equally well for different groups of patients has been little studied. This study used ECGs from 326,518 patient encounters at Stanford Hospital to evaluate a model trained to predict heart failure within five years, comparing its performance by race, ethnicity, age and sex.

    Overall, there were no significant differences in performance between racial groups, and performance declined with age. However, among patients aged 0 to 40, the model performed significantly worse for Black patients than for any other racial group, with the largest differences among young Black women.

    The researchers tested several ways of reducing these disparities: including race, ethnicity, sex and age in the model, training separate models for each racial group, and training it on data with equal racial representation. None of these improved the disparities. Adjusting the model’s decision thresholds individually by race, age and sex did lead to substantial improvements. The authors warn that machine learning tools for heart failure risk perpetuating existing disparities, and suggest these individualised thresholds as one way to reduce them.

    Learn more about this study here: https://doi.org/10.1161/CIRCHEARTFAILURE.123.010879


    Reference:

    Kaur, D., Hughes, J.W., Rogers, A.J., Kang, G., Narayan, S.M., Ashley, E.A., & Perez, M.V. (2024). Race, Sex, and Age Disparities in the Performance of ECG Deep Learning Models Predicting Heart Failure. Circulation: Heart Failure, 17(1), e010879.

  • Predictive Accuracy of Stroke Risk Prediction Models Across Black and White Race, Sex, and Age Groups

    Predictive Accuracy of Stroke Risk Prediction Models Across Black and White Race, Sex, and Age Groups

    Stroke is a leading cause of death and serious long-term disability in the US, with a particularly high risk among Black individuals. Risk prediction tools help identify who would benefit most from prevention, but they need to work equally well for everyone. This study compared several of these tools, including two stroke-specific models, a general cardiovascular risk model and two new machine learning models, using data from 62,482 Black and White adults aged 45 and over from four large US studies.

    Overall, the models performed similarly, and the new machine learning models did not improve their accuracy. However, all of them were worse at distinguishing between people who would and would not have a stroke when applied to Black individuals. Among women, the models’ accuracy score was 0.76 for White women and 0.69 for Black women. Among men, it ranged from 0.71 to 0.72 for White men and from 0.64 to 0.66 for Black men. The models also performed better in adults under 60 than in older adults.

    The authors conclude that current approaches are not enough to close this gap, and that the range of risk factors considered and the modelling techniques used need to be improved to address these racial disparities.

    Learn more about this study here: https://doi.org/10.1001/jama.2022.24683


    Reference:

    Hong, C., Pencina, M.J., Wojdyla, D.M., Hall, J.L., Judd, S.E., Cary, M., Engelhard, M.M., Berchuck, S., Xian, Y., D’Agostino, R., Howard, G., Kissela, B., & Henao, R. (2023). Predictive Accuracy of Stroke Risk Prediction Models Across Black and White Race, Sex, and Age Groups. JAMA, 329(4), 306–317.

  • Age, sex and race bias in automated arrhythmia detectors

    Age, sex and race bias in automated arrhythmia detectors

    Algorithms that read electrocardiograms (ECGs) automatically are increasingly used to help identify heart conditions, but few studies have examined whether they perform equally well for different groups of patients. This study compared 56 open-source algorithms, developed by participants in the 2021 PhysioNet Challenge, on more than 130,000 ECGs, grouping the results by sex, age and race.

    Several performance measures were similar across groups, but others showed clear differences. Accuracy was significantly lower for Black patients and higher for Asian patients, with a gap of 39% on the test set. Accuracy also fell with age, by around 12% per decade, and results for male and female patients differed depending on the measure used.

    The researchers then retrained one of the best-performing models with an additional constraint designed to reduce performance differences across sex, race and age. This led to a small loss in overall performance but a significant reduction in bias. The authors conclude that bias depends on the model’s design, the population it is trained on and how its performance is measured, and that all of these should be closely examined.

    Learn more about this study here: https://doi.org/10.1016/j.jelectrocard.2022.07.007


    Reference:

    Perez Alday, E.A., Rad, A.B., Reyna, M.A., Sadr, N., Gu, A., Li, Q., Dumitru, M., Xue, J., Albert, D., Sameni, R., & Clifford, G.D. (2022). Age, sex and race bias in automated arrhythmia detectors. Journal of Electrocardiology, 74, 5–9.

  • AEQUITAS

    AEQUITAS

    Proposal: Preparation of the CSOs and public healthcare sector to address gender and racial biases that might arise from the wide usage of AI in order to protect and promote fundamental rights

    Implementation: 2025 to 2027

    Call:  CERV-2024-CHAR-LITI – Promote civil society organizations’ awareness of, capacity building and implementation of the EU Charter of Fundamental Rights

    Proposed Budget:   1 061 761,00€

    Keywords: gender and racial biases, biomedical AI, EU Charter of Fundamental Rights

    Objective: The topic of the AEQUITAS project is the gender and racial biases that have been reported in biomedical AI and which can lead to misdiagnosis and mistreatment and how they pose a threat to the fundamental rights protected by the EU Charter. The aim of the project is 3-fold:

    – to increase the capacity of the CSOs and human rights organizations in educating the public, monitoring the biases and advocating for the protection of the fundamental rights especially regarding the biomedical AI;

    – to increase the knowledge of healthcare staff from public hospitals on the biases that biomedical AI can present due to the biased data from which they were fed and to help them approach and consult these AI systems with a critical mindset;

    – to develop an AI Regulatory Model that will be used by CSOs and public hospitals in their practices;

    – to develop a European network of CSOs and public hospitals that will collaborate and support each other in the effort to raise public’s awareness on the gender and racial biases of biomedical AI and on the applications of the EU Charter;

    – to raise awareness of the EU Charter of fundamental rights and its application in the AI era;

    – to develop and distribute policy recommendations in order to advocate for the need to regulate biomedical AI.

    Partners:

    • Innovation Hive – Kypseli Kainotomias
    • Kentro Ginaikeion Meleton Kai Ereyvnon Astiki Mi K
    • Universitat Zu Koln
    • Center for the Study of Democracy
    • C.I.P. Citizens in Power
    • Moterų Informacijos Centras Asociacija Mic
    • Lobby Europeo de Mujeres en Espana LEM España
    • TIA Formazione Internazionale Associazione APS
    • Health Citizens – European Research Institute
    • Technologiko Panepistimio Kyprou
    • Cyens Centre of Excellence
    • Edex – Educational Excellence Corporation Limited
    • Rite Research Institute for Technological Evolution 
    • Vucable
    • TotalEU Production

    Project Website: https://aequitasproject.eu/