AI models used in clinical care are often trained on data that does not represent the full diversity of patients, which can lead to biased results and worsen health disparities. Experience in drug and medical device development has already shown the dangers of under-representing women and minority populations. This systematic review assessed how often published clinical AI studies report the gender and racial composition of the data used to train their models.
Of the 390 studies included, 84% did not report the racial composition of their training data and 31% did not report gender. Models developed in the US did slightly better, with 56% reporting race and 77% reporting gender. Only 16% of all models used publicly available, non-proprietary datasets.
The authors conclude that the low level of demographic reporting and the limited use of open data raise serious concerns about the transparency, generalisability and fairness of clinical AI models. They call for urgent, standardised reporting of gender and racial composition in training data to ensure these technologies are used ethically and equitably.
Learn more about this review here: https://doi.org/10.1093/oodh/oqaf027
Reference:
Otokiti, A., Shih, H., & Williams, K. (2025). Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity. Oxford Open Digital Health, oqaf027.