How can AI bias show up in your business?

AI bias means that an AI system treats some groups of people systematically worse than others: it rejects more of their applications, quotes them higher prices or gives them poorer answers. It rarely comes from bad intent, but from the data the system learned from, the way it was designed and the way people use it.

Hypothetical example: an online shop uses an AI tool to set personalised discounts. The tool never sees customers’ age or background, but it sees postcodes and learns that some neighbourhoods respond less to discounts. Customers there end up paying more on average, and nobody notices because nobody compared the results.

Where bias comes from

• Data. If historical data reflects past unequal treatment, or some groups barely appear in it, the system repeats the pattern. A screening tool trained on past hires learns who was hired before, not who would do the job well.
• Design. What the system optimises and which inputs it uses both matter. A seemingly neutral input, such as a postcode, a career gap or a first name, can stand in for a characteristic like age, sex or ethnic origin. This is called a proxy.
• Use. A system built for one purpose or population may work worse for another, and people add bias by trusting the output more for some groups.

The NIST AI Risk Management Framework, a voluntary framework, groups bias in a similar way: systemic, computational and statistical, and human-cognitive. Patterns in the data flow into the output, as explained in what AI can do and where its limits are.

Bias matters most where outputs shape decisions about people: hiring (screening CVs, ranking candidates), lending (who gets credit and on what terms), pricing (personalised prices, discounts and premiums) and customer service (chatbots that understand some customers worse, or routing that gives them slower help).

How to screen for AI bias without data scientists

Recommendation. You do not need access to the model to run an initial screening. Test what comes out of it.

1. Pick one use case and the decision it produces, such as a shortlist score or a price. Agree in advance what size of difference counts as meaningful.
2. Write 20 to 40 realistic, made-up test cases. Do not use real customers’ or applicants’ data.
3. Build pairs that are identical except for one detail: a name typical of a different background, an age, a postcode, a career gap.
4. Run every case with identical settings and record the tool and model version. Where the output varies between runs, run each case several times and compare the averages.
5. Compare the results across groups against your threshold.
6. Where a difference crosses the threshold, the use-case owner decides what to do: fix the inputs, add human oversight of the AI’s output, limit the use or stop it.

Repeat the screening after every model update or change of vendor, and keep the results for your AI risk assessment.

Know its limits. Names and postcodes are weak proxies for group membership, and a small made-up sample cannot show everything. Finding no difference does not prove the system is fair. A difference, such as a price gap, does not on its own prove unlawful discrimination; that depends on the applicable law and any justification. For decisions with serious consequences, such as hiring or credit, escalate to a proper evaluation with specialist support. The NIST Generative AI Profile likewise treats harmful bias as a risk to measure and manage over time, not one cleared by a single test.

When bias becomes a legal issue

Bias is not a separate legal offence, but it becomes a legal issue in two ways. First, anti-discrimination law protects people from unfair treatment based on characteristics such as sex, age, ethnic origin or disability, and it does not stop applying because AI helped make a decision.

Second, the EU AI Act sets data rules for high-risk systems. Legal requirement (EU AI Act, Art. 10). This applies to providers of high-risk systems: from 2 December 2027 for Annex III systems and from 2 August 2028 for Annex I Section A products; Section B products follow the sector route in Art. 2(2). The full timeline is in what the EU AI Act requires. Training, validation and testing data must be subject to data governance, including an examination for possible biases and measures to detect, prevent and reduce them (AI Act, Art. 10). AI used to recruit candidates or assess people’s creditworthiness is listed in Annex III.

Next step: prepare paired test cases for the one use case that affects the most people, and run the screening before the tool’s next update.

Sources and further reading

• Regulation (EU) 2024/1689 (AI Act) — Article 10 and Annex III
• NIST AI Risk Management Framework
• NIST AI 600-1, Generative AI Profile