Debate

Algorithms, Fairness and Power

It is mathematically proven that no risk score can meet all the usual standards of fairness at once when groups differ in their base rates – how often, say, people reoffend. So when software helps decide who gets a loan, a job or release before trial, which standard should count – and should such decisions be left to software at all?

Automated systems now help to screen job applicants, score credit, flag welfare fraud and predict reoffending. In 2016 ProPublica reported that among defendants who did not go on to reoffend, COMPAS, a risk score used in American courts, had labelled Black defendants as high-risk almost twice as often as white defendants; the developer replied that its scores were equally accurate for both groups. Both could be true at once, and researchers soon proved why: when base rates differ, a score cannot be equally well calibrated for each group and have equal error rates (Kleinberg and colleagues, 2016; Chouldechova, 2017). With large language models the questions have widened: who owns data and infrastructure, how opaque systems can be held to account, and whether the greatest dangers lie in present harms or in future systems. The EU’s AI Act of 2024 is the first comprehensive attempt to regulate the field by law.

Open in the star map

Positions

Fairness can be measured – but not all at once

Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan (2016) and Alexandra Chouldechova (2017) proved that when groups differ in their base rates, a risk score cannot at the same time be equally well calibrated for each group and have equal error rates. Choosing a fairness criterion is therefore a moral decision, not a technical detail. Defenders of algorithmic decision-making add that biased humans are hard to audit, whereas an algorithm’s inputs and outcomes can be tested and corrected, so that well-regulated algorithms may even reduce discrimination (Kleinberg, Ludwig, Mullainathan and Sunstein, 2018). Brian Hedden (2021) shows with a thought experiment that a perfectly fair procedure can violate almost all the usual statistical criteria – so most of them cannot be necessary conditions of fairness.

Old injustice in new code

Sociologists and critical scholars of technology argue that the problem lies less in the formulas than in the world they learn from. Systems trained on past decisions reproduce past discrimination – in hiring, policing or welfare – while appearing neutral; Ruha Benjamin (Race After Technology, 2019) calls this the “New Jim Code”, an allusion to the Jim Crow laws of segregation. Joy Buolamwini and Timnit Gebru (2018) found that commercial face-analysis systems misclassified the gender of darker-skinned women far more often than that of lighter-skinned men, and Virginia Eubanks (Automating Inequality, 2018) shows how automated welfare systems subject poor people to a surveillance others never face. On this view fairness metrics are not enough: what matters is who designs the systems, for what purpose – and whether some should be built at all.

Power over data

Shoshana Zuboff (The Age of Surveillance Capitalism, 2019) argues that the leading platforms claim human experience as free raw material: they collect a “behavioural surplus” beyond what their services need, turn it into predictions, sell these and ultimately seek to shape behaviour itself. Nick Couldry and Ulises Mejias (The Costs of Connection, 2019) speak of “data colonialism”, a new appropriation of human life that, in their view, continues the logic of historical colonialism. The critics differ on remedies – stronger privacy rights, competition law, public or cooperative alternatives – but agree that an ethics of individual algorithms overlooks the concentration of power in a few companies that own the data, the computing capacity and the models.

From principles to practice

Floridi and Josh Cowls (2019) condensed dozens of AI ethics guidelines into five principles: beneficence, non-maleficence, autonomy, justice and explicability. Principles alone, Floridi warns, invite “ethics shopping”, “ethics bluewashing” and “ethics dumping” – picking whichever rules suit, polishing one’s image with ethical claims, or moving questionable research to places with weaker rules. They must be translated into law, design and auditing, as the EU has begun to do with its risk-based AI Act. Vallor adds a virtue-ethical perspective: technology changes the habits through which we form our character, so societies need “technomoral” virtues such as honesty, humility, courage and practical wisdom to live well with it.

Held by

Luciano Floridib. 1964
Shannon Vallorcontemporary

Risks at the frontier

Bostrom (Superintelligence, 2014) and the computer scientist Stuart Russell (Human Compatible, 2019) argue that systems more capable than humans could pursue goals that diverge from ours, and that reliably “aligning” them with human values is an unsolved technical and philosophical problem. In 2023 hundreds of AI researchers and executives signed a one-sentence statement that mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war; an international scientific report chaired by Yoshua Bengio (2025) examined such risks together with present harms. Critics such as Emily Bender and Timnit Gebru reply that speculation about superintelligence distracts from the concrete harms of existing systems – bias, the exploitation of data workers, concentration of power – and serves the companies that profit from the hype.