Algorithmic bias: why a hiring tool penalised the word “women's”

Two identical résumés, one word apart: “women's”. An Amazon tool scored the second lower. Nobody programmed that. So where does the bias come from? In 1988, a British investigation found that a London medical school's application-screening program put women and non-European names at a disadvantage. In 2016, ProPublica analysed the risk scores of a Florida justice software and saw racial bias; its maker disputed those conclusions. In 2024, the European AI Act entered into force; for high-risk systems, it provides for examining the data for bias. Three steps to understand it. Let's go. Step one: bias gets in through the data. The tool learned from ten years of résumés, mostly men's. On its own, it learned a rule nobody wrote: penalise “women's”. Amazon scrapped the tool. Step two: the average hides the gap. In 2018, Joy Buolamwini and Timnit Gebru tested commercial software that classifies faces as male or female. Measured group by group, the error rate was more than forty times higher for some faces than for others. Step three: measure, then fix. Tools like IBM's AI Fairness 360 or Microsoft's Fairlearn compare the results for each group. If the gap is real, three options: fix the data, adjust the threshold, or drop the use case. In Europe, the AI Act requires this check for high-risk systems like hiring. At KOAEE, we apply the same rule to languages. In our demos, at the clinic, the insurer, the town hall: the same questions in all sixteen languages, 1,168 answers, each one reviewed. A language that's served less well can't hide in the average: we see it, and we fix it. To understand AI in 150 seconds, subscribe to the channel. The AI that stays between you. Try it at koaee.ai.