Tag: Machine Learning

  • Fairness-Aware Hyperparameter Optimization

    Fairness-Aware Hyperparameter Optimization

    Machine learning is increasingly used to make decisions about people, and there have been several reports of bias in areas such as recidivism assessment, job-applicant screening and mortgage risk. This master’s thesis focused on hyperparameters, the many settings that govern how a model learns. These settings are usually tuned to maximise overall performance, such as accuracy, even though they also affect how fairly the model treats groups of people, for example based on race, gender or age. The thesis developed a framework that adds fairness as a goal when tuning these settings and tested it in a fraud detection case study.

    The work led to an article presenting Fairband, a fairness-aware algorithm for tuning hyperparameters. Tested on four real-world decision-making datasets, Fairband consistently found models with substantially better fairness at a comparatively small cost in accuracy, with no extra training cost. The authors describe it as simple, resource-efficient and easy to integrate into real-world machine learning pipelines.

    Developed at Feedzai Research, the thesis won the Best Thesis Award in the field of Information Technology. According to Feedzai, Fairband increases the impartiality of automated decision-making processes by an average of 93%. Although this work does not focus on healthcare, it addresses a question that is also central to medical AI: how to make sure automated decisions do not disadvantage some groups of people.

    Learn more about the thesis here: https://hdl.handle.net/10216/128959
    Learn more about the article here: https://doi.org/10.48550/arXiv.2010.03665

    Please note: the article is a preprint, which means it has not yet been peer reviewed.


    References:

    Cruz, A. M. F. (2020). Fairness-aware hyperparameter optimization [Master’s thesis, Universidade do Porto]. Repositório Aberto da Universidade do Porto.

    Cruz, A. F., Saleiro, P., Belém, C., Soares, C., & Bizarro, P. (2020). A bandit-based algorithm for fairness-aware hyperparameter optimization (arXiv:2010.03665). arXiv.

  • Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity

    Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity

    AI models used in clinical care are often trained on data that does not represent the full diversity of patients, which can lead to biased results and worsen health disparities. Experience in drug and medical device development has already shown the dangers of under-representing women and minority populations. This systematic review assessed how often published clinical AI studies report the gender and racial composition of the data used to train their models.

    Of the 390 studies included, 84% did not report the racial composition of their training data and 31% did not report gender. Models developed in the US did slightly better, with 56% reporting race and 77% reporting gender. Only 16% of all models used publicly available, non-proprietary datasets.

    The authors conclude that the low level of demographic reporting and the limited use of open data raise serious concerns about the transparency, generalisability and fairness of clinical AI models. They call for urgent, standardised reporting of gender and racial composition in training data to ensure these technologies are used ethically and equitably.

    Learn more about this review here: https://doi.org/10.1093/oodh/oqaf027


    Reference:

    Otokiti, A., Shih, H., & Williams, K. (2025). Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity. Oxford Open Digital Health, oqaf027.

  • The Hidden Ways AI Can Discriminate

    The Hidden Ways AI Can Discriminate

    When we talk about bias in AI, we are referring to “computer systems that systematically and unfairly discriminate against certain individuals or groups in favour of others” (Friedman & Nissenbaum, 1996). This is not about occasional errors, but about consistent, predictable patterns of unfair outcomes. Furthermore, biases in AI systems are complex, as they can enter the system at almost any stage, from the data it’s trained on to the way it’s ultimately used in a hospital setting. Recognising where bias originates, and how different types can reinforce each other, is the first step towards addressing it.

    Three broad roots of bias

    Researchers have identified three broad categories of bias that apply to all computer systems. The first is pre-existing bias, which comes from existing inequalities in society that get absorbed into the system, sometimes without anyone realising it (Friedman & Nissenbaum, 1996). The second is technical bias, which arises from the practical compromises made when translating complex human realities into clean computational form. And the third is emergent bias, which appears only after a system is deployed, as the world around it changes in ways the original design never anticipated.

    Bias in the AI pipeline

    These three categories give us a useful starting point. But for AI and machine learning specifically, researchers have identified seven more precise bias types that can affect these systems throughout their development and use (Suresh & Guttag, 2021):

    Historical bias reflects prejudices and stereotypes already present in training data, even if the data is technically accurate. A 1990 study found that after coronary bypass surgery, male patients received pain medication significantly more often than female patients, who were instead given sedatives more frequently (Calderone, 1990). An AI trained on data like this would learn to replicate that same pattern, perpetuating the very inequality it inherited.

    Representation bias happens when certain groups are underrepresented in training data. An AI trained mostly on data from one demographic will simply perform worse for everyone else. A well-known example is skin cancer detection tools that are significantly less accurate for patients with darker skin tones because the training datasets contained predominantly images from fair-skinned individuals (Guo et al., 2021).

    Measurement bias creeps in when the way something is measured isn’t equally accurate or fair across groups. One striking case involved an algorithm that used healthcare costs as a proxy for how sick someone actually was, and that ended up disadvantaging Black patients, who, facing disproportionate levels of poverty, tended to spend less on healthcare than equally sick white patients (Obermeyer et al., 2019).

    Aggregation bias occurs when a single, one-size-fits-all model is applied to genuinely diverse populations, ignoring the fact that the same data point can mean very different things depending on a person’s background.

    Learning bias emerges from technical decisions made while building the model itself, choices that can amplify disparities already present in the data, sometimes without the developers being aware. Research has shown, for instance, that differential privacy, a technique meant to protect patient confidentiality, can end up reducing a model’s accuracy for underrepresented groups even further (Bagdasaryan & Shmatikov, 2019).

    Evaluation bias happens when the datasets used to test and benchmark the model don’t reflect the real population it will actually serve, meaning a system can look accurate on paper while quietly failing specific groups of patients in practice.

    Finally, deployment bias arises when a tool is used in a context very different from the one it was designed and tested for, undermining its reliability in ways that are easy to miss.

    Why this matters for patients

    Understanding where bias comes from is the first step to catching it, and to making sure the AI tools shaping modern medicine work fairly for everyone, not just the patients who happen to be well represented in the data.

    This is exactly the kind of practical, structured understanding the AEQUITAS training equips healthcare professionals with, helping them recognise these patterns before they translate into real harm for real patients.


    References:

    Bagdasaryan, E., & Shmatikov, V. (2019). Differential Privacy Has Disparate Impact on Model Accuracy (arXiv:1905.12101). arXiv. https://doi.org/10.48550/arXiv.1905.12101

    Calderone, K. L. (1990). The influence of gender on the frequency of pain and sedative medication administered to postoperative patients. Sex Roles, 23(11), 713–725. https://doi.org/10.1007/BF00289259

    Friedman, B., & Nissenbaum, H. (1996). Bias in computer systems. ACM Transactions on Information Systems, 14(3), 330–347. https://doi.org/10.1145/230538.230561

    Guo, L. N., Lee, M. S., Kassamali, B., Mita, C., & Nambudiri, V. E. (2021). Bias in, bias out: Underreporting and underrepresentation of diverse skin types in machine learning research for skin cancer detection-A scoping review. Journal of the American Academy of Dermatology, S0190-9622(21)02086-7. https://doi.org/10.1016/j.jaad.2021.06.884

    Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342

    Suresh, H., & Guttag, J. (2021). A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO ’21, 1–9. https://doi.org/10.1145/3465416.3483305

  • Evaluating and mitigating bias in machine learning models for cardiovascular disease prediction

    Evaluating and mitigating bias in machine learning models for cardiovascular disease prediction

    Machine learning models are increasingly used to estimate a person’s risk of cardiovascular disease, but bias can be introduced when health data is collected and prepared, affecting how well these models work for certain groups. This study used electronic health records from 109,490 people treated at Vanderbilt University Medical Center, in the US, to test whether several machine learning models performed equally well across race and gender. It also compared them with a widely used risk equation from the American Heart Association that does not use machine learning.

    Most of the machine learning models were less biased than the American Heart Association equation. Differences between racial groups were small, but differences between women and men were considerably larger. According to the authors, both the risk equation and the machine learning models were biased against women.

    The researchers also tested three ways of reducing this bias. Removing race and gender from the data did not significantly reduce bias in most models, and balancing the number of people from each group in the training data did not consistently help either. Balancing the proportion of people with cardiovascular disease in each group reduced the gender bias, but not the racial bias, and slightly reduced accuracy in many cases. The authors conclude that gender disparities in cardiovascular risk prediction need to be investigated and corrected.

    Learn more about this study here: https://doi.org/10.1016/j.jbi.2023.104294


    Reference:

    Li, F., Wu, P., Ong, H.H., Peterson, J.F., Wei, W.-Q., & Zhao, J. (2023). Evaluating and mitigating bias in machine learning models for cardiovascular disease prediction. Journal of Biomedical Informatics, 138, 104294.