When we talk about bias in AI, we are referring to “computer systems that systematically and unfairly discriminate against certain individuals or groups in favour of others” (Friedman & Nissenbaum, 1996). This is not about occasional errors, but about consistent, predictable patterns of unfair outcomes. Furthermore, biases in AI systems are complex, as they can enter the system at almost any stage, from the data it’s trained on to the way it’s ultimately used in a hospital setting. Recognising where bias originates, and how different types can reinforce each other, is the first step towards addressing it.
Three broad roots of bias
Researchers have identified three broad categories of bias that apply to all computer systems. The first is pre-existing bias, which comes from existing inequalities in society that get absorbed into the system, sometimes without anyone realising it (Friedman & Nissenbaum, 1996). The second is technical bias, which arises from the practical compromises made when translating complex human realities into clean computational form. And the third is emergent bias, which appears only after a system is deployed, as the world around it changes in ways the original design never anticipated.
Bias in the AI pipeline
These three categories give us a useful starting point. But for AI and machine learning specifically, researchers have identified seven more precise bias types that can affect these systems throughout their development and use (Suresh & Guttag, 2021):
Historical bias reflects prejudices and stereotypes already present in training data, even if the data is technically accurate. A 1990 study found that after coronary bypass surgery, male patients received pain medication significantly more often than female patients, who were instead given sedatives more frequently (Calderone, 1990). An AI trained on data like this would learn to replicate that same pattern, perpetuating the very inequality it inherited.
Representation bias happens when certain groups are underrepresented in training data. An AI trained mostly on data from one demographic will simply perform worse for everyone else. A well-known example is skin cancer detection tools that are significantly less accurate for patients with darker skin tones because the training datasets contained predominantly images from fair-skinned individuals (Guo et al., 2021).
Measurement bias creeps in when the way something is measured isn’t equally accurate or fair across groups. One striking case involved an algorithm that used healthcare costs as a proxy for how sick someone actually was, and that ended up disadvantaging Black patients, who, facing disproportionate levels of poverty, tended to spend less on healthcare than equally sick white patients (Obermeyer et al., 2019).
Aggregation bias occurs when a single, one-size-fits-all model is applied to genuinely diverse populations, ignoring the fact that the same data point can mean very different things depending on a person’s background.
Learning bias emerges from technical decisions made while building the model itself, choices that can amplify disparities already present in the data, sometimes without the developers being aware. Research has shown, for instance, that differential privacy, a technique meant to protect patient confidentiality, can end up reducing a model’s accuracy for underrepresented groups even further (Bagdasaryan & Shmatikov, 2019).
Evaluation bias happens when the datasets used to test and benchmark the model don’t reflect the real population it will actually serve, meaning a system can look accurate on paper while quietly failing specific groups of patients in practice.
Finally, deployment bias arises when a tool is used in a context very different from the one it was designed and tested for, undermining its reliability in ways that are easy to miss.
Why this matters for patients
Understanding where bias comes from is the first step to catching it, and to making sure the AI tools shaping modern medicine work fairly for everyone, not just the patients who happen to be well represented in the data.
This is exactly the kind of practical, structured understanding the AEQUITAS training equips healthcare professionals with, helping them recognise these patterns before they translate into real harm for real patients.
References:
Bagdasaryan, E., & Shmatikov, V. (2019). Differential Privacy Has Disparate Impact on Model Accuracy (arXiv:1905.12101). arXiv. https://doi.org/10.48550/arXiv.1905.12101
Calderone, K. L. (1990). The influence of gender on the frequency of pain and sedative medication administered to postoperative patients. Sex Roles, 23(11), 713–725. https://doi.org/10.1007/BF00289259
Friedman, B., & Nissenbaum, H. (1996). Bias in computer systems. ACM Transactions on Information Systems, 14(3), 330–347. https://doi.org/10.1145/230538.230561
Guo, L. N., Lee, M. S., Kassamali, B., Mita, C., & Nambudiri, V. E. (2021). Bias in, bias out: Underreporting and underrepresentation of diverse skin types in machine learning research for skin cancer detection-A scoping review. Journal of the American Academy of Dermatology, S0190-9622(21)02086-7. https://doi.org/10.1016/j.jaad.2021.06.884
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
Suresh, H., & Guttag, J. (2021). A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO ’21, 1–9. https://doi.org/10.1145/3465416.3483305