Building Fair Machine Learning Models: Bias Tests, Fairness Metrics and AI Governance

Ateeb's profile picture
Ateeb TaseerLast Updated on
15-16 Min Read TimeAdd as preferred on Google

Key Takeaways

A model can score well overall and still fail one group, so fairness needs group-level checks and a release gate with thresholds agreed in advance.

  • Dropping race or gender columns does not remove bias, because proxies such as zip code can carry the same information.
  • Bias enters through data disparities, algorithmic amplification, proxy variables, feedback loops and historical labels, and each has a matching check.
  • Subgroup and intersectional error analysis show gaps that overall accuracy hides.
  • Report row counts next to every rate, because a cell with a handful of rows gives an unreliable rate in either direction.
  • Check both demographic parity and equal opportunity, because they catch different failures. The four-fifths selection-rate ratio is a rough screen only.
  • SHAP shows whether a model relies on a suspected proxy.
  • A model card should record the groups tested, per-group rates against agreed thresholds, suspected proxies and the reviewer who signed off.

Start with the model whose decisions affect people most, and record its fairness gaps as the baseline.

A credit, pricing, triage or screening model can post strong accuracy on its test set and still make far more mistakes for one group of people. Overall accuracy hides that gap. Dropping race or gender columns doesn't close it either, because features such as zip code can carry the same information. Building fair machine learning models means checking results group by group and blocking release until those checks pass against thresholds agreed in advance.

 

This guide is for ML engineers, data scientists and the leads approving models for 2027, and it walks through how AI ethics and AI governance divide the work, five ways bias enters a model, four Python checks, what to document and a pre-deployment gate.

 

How AI Ethics and AI Governance Differ for ML Teams

AI ethics defines what a model should do for the people it affects. AI governance is the set of policies, owners, reviews and records that makes a team actually do it. Take a lending model. Ethics says applicants with similar repayment risk should get similar decisions, and governance decides who runs the fairness checks before release and which threshold is strict enough to block it.

 

Four principles usually sit under AI ethics. Fairness means avoiding discrimination and bias, accountability means errors can be challenged and corrected, transparency means decisions can be explained, and privacy means personal data is collected and used with care. Governance turns those principles into internal policies, technical standards, oversight such as audits and impact assessments, and compliance with external regulation.

Concept

What it covers

Question it answers

Typical outputs

Best for

AI ethics

Principles such as fairness, accountability, transparency and privacy

What should this model do, and who could it harm?

An agreed definition of fair for the use case

Deciding what fair means before the model is built

AI governance

Policies, standards, owners, oversight and audits that enforce the principles

Who checks the model, against which thresholds, and when?

Sign-offs, deployment gates, model documentation, audit records

Making fairness checks repeatable and accountable

Each fails without the other, in our view. A principle with no gate rarely stops a biased release, and a gate with no agreed principle has the opposite problem, because it tends to measure whatever happens to be easiest to compute. Our article on the difference between AI ethics and AI governance covers both terms in more depth.

 

Why Bias in Machine Learning Models Causes Real Harm

Biased models move money and access away from specific groups, usually without the people affected knowing a model was involved. Car insurance shows how this plays out. In April 2017, ProPublica and Consumer Reports published an analysis of car insurance premiums and payouts in California, Illinois, Missouri and Texas, and found that some insurers charged premiums that were on average 30 percent higher in minority neighborhoods than in other areas with similar accident costs.

 

Those premiums followed zip code, which is a typical proxy variable and comes back in the checks below. A credit limit or a CV screening score can carry the same pattern, and it stays invisible for as long as the only number anyone reviews is overall accuracy.

 

Five Ways Bias Gets Into Machine Learning Models

Most bias enters through the training data or through the way the model learns from it. Each source below has a check that catches it.

1. Data disparities

Representation shapes what a model learns. It learns most about the groups it sees most often. If most of a diagnostic model's training records come from one demographic, the model has far fewer examples of how a condition presents in everyone else, which raises the chance of misdiagnosing them. Group shares in data exploration catch this early, and subgroup error analysis catches it again after training.

2. Algorithmic amplification

A model that optimises overall accuracy learns the majority group's patterns first, because that is where most of the loss comes from. Small gaps in the data then grow into larger gaps in errors, and a group with a small share of training rows can end up with a much higher error rate while headline accuracy still looks fine. This is what subgroup and intersectional error analysis are for.

3. Proxy variables

Removing race or gender from the feature set leaves that information in the data. Zip code, first name, school attended or job title can each track a sensitive attribute closely enough that the model learns the same pattern anyway, with the column itself nowhere in sight. Data exploration shows whether a feature tracks a sensitive attribute. SHAP shows whether the model relies on it.

4. Feedback loops

A model's decisions shape the data its next version trains on. A lending model that rejects most applicants from one group never learns whether they would have repaid, which leaves the retraining data with little repayment history for that group, and the next model keeps the gap. Fairness metrics on every retrain catch this.

5. Historical labels

Labels often record past decisions. Past decisions carry past inequalities, and a hiring model trained on previous hiring outcomes will learn to repeat whoever was favoured before. You can spot this before training by comparing target rates by group, and the fairness metrics run after training will confirm it if the labels were skewed.

 

How to Test a Machine Learning Model for Bias in Python

Two checks come first. Both run on pandas alone, with one measuring error rates per group and the other profiling the data. They mirror parts of Microsoft's Responsible AI dashboard in Azure Machine Learning, which, as of November 2026, brings data analysis, error analysis, fairness assessment, model interpretability, counterfactual what-if and causal analysis into one interface.

 

The snippets in this guide run in any Python pipeline. They were checked against the open-source releases current in November 2026, SHAP 0.54.1 and Fairlearn 0.15.0.

Subgroup and intersectional error analysis

import pandas as pd

def deep_dive_error_analysis(model, X_test, y_test, sensitive_features):
    # X_test: features only. sensitive_features: DataFrame aligned to X_test's index
    errors = pd.Series(model.predict(X_test) != y_test.to_numpy(), index=X_test.index)
    print(f"Overall error rate: {errors.mean():.3f} (n={len(errors)})")
    for feature in sensitive_features.columns:
        subgroup_errors = errors.groupby(sensitive_features[feature]).agg(['mean', 'size'])
        print(f"\nError rates and counts by {feature}:")
        print(subgroup_errors)

    # Intersectional cells (the size column shows cells too small to trust)
    intersectional_errors = errors.groupby(
        [sensitive_features[f] for f in sensitive_features.columns]
    ).agg(['mean', 'size'])
    print("\nIntersectional error rates and counts:")
    print(intersectional_errors)

Error analysis compares each group's error rate with the overall rate printed at the top of the output, then repeats the comparison for combinations such as gender by age band. Bias often concentrates at intersections. A model can treat women and older applicants acceptably as separate groups and still fail older women.

 

Trouble shows up as a group whose error rate sits well above the overall rate. Watch the size column too. A cell with a handful of rows produces a rate that is unreliable in either direction, and the right response is more data for that group or a closer look at its features before the model ships.

Data exploration for representation, label gaps and proxies

import pandas as pd

def advanced_data_explorer(data, sensitive_features):
    # data includes the 'target' column; sensitive_features is a list of column names
    for feature in sensitive_features:
        print(f"\nAnalyzing {feature}:")
        print("Share of rows by group:")
        print(data[feature].value_counts(normalize=True))
        print("Positive-target rate by group:")
        print(data.groupby(feature)['target'].mean())
        for other_feature in sensitive_features:
            if feature != other_feature:
                # Raw counts, so small intersections stay visible
                print(f"\nIntersection with {other_feature}:")
                print(pd.crosstab(data[feature], data[other_feature]))

The share of rows shows how much of the data each group contributes, while target rates show whether outcomes in the labels already differ by group, which points to historical bias. Crosstabs count rows in every intersection.

 

Two warning signs matter most here: a group with a small share, and target rates that differ sharply across groups. To test a suspected proxy, run the same groupby on a non-sensitive feature, for example data.groupby('zip_code')['ethnicity'].value_counts(normalize=True). If some zip codes are dominated by one group, flag zip code as a likely proxy and check its SHAP importance later.

 

Which Fairness Metrics Should You Check Before Deployment?

Check at least two group fairness metrics, demographic parity and equal opportunity. They catch different failures. Demographic parity compares selection rates, the share of each group that receives the positive prediction, and equal opportunity compares true positive rates, meaning how often people whose real outcome is positive are predicted positive, group by group.

import pandas as pd
from scipy.stats import chi2_contingency

def fairness_assessment(model, X_test, y_test, sensitive_features):
    # Binary 0/1 predictions and labels; sensitive_features aligned to X_test's index
    predictions = pd.Series(model.predict(X_test), index=X_test.index)
    y_true = pd.Series(y_test.to_numpy(), index=X_test.index)
    positives = y_true == 1
    results = {}
    for feature in sensitive_features.columns:
        groups = sensitive_features[feature]
        selection_rate = predictions.groupby(groups).mean()               # demographic parity
        tpr = predictions[positives].groupby(groups[positives]).mean()    # equal opportunity
        _, p_value, _, _ = chi2_contingency(pd.crosstab(groups, predictions))
        results[feature] = {
            "selection_ratio": selection_rate.min() / selection_rate.max(),
            "tpr_gap": tpr.max() - tpr.min(),
            "p_value": p_value,
        }
        print(f"\n{feature}: {results[feature]}")
        print(pd.DataFrame({"selection_rate": selection_rate, "tpr": tpr}))
    return results

Start with the ratio and gap, then look at the per-group rates to see which groups sit at each end. A selection-rate ratio below four-fifths (eighty percent) is a common rough screen. It comes from the four-fifths rule in the US Uniform Guidelines on Employee Selection Procedures, which generally treat a selection rate below four-fifths of the highest group's rate as evidence of adverse impact in hiring.

 

Those Guidelines also say smaller differences can still count where they are significant in both statistical and practical terms. Treat the ratio as a screen only. The threshold for the true positive rate gap should be agreed before anyone sees the results.

 

The chi-square p-value tests whether predictions are independent of group. A small p-value says a gap this size would be unlikely by chance. That cutoff should be set in advance. And read it next to the gap and the group sizes, because large samples make even tiny gaps statistically significant while small samples can hide large ones entirely.

Cross-checking fairness metrics with Fairlearn

Fairlearn gives the hand-written function a second opinion. Version 0.15.0 includes MetricFrame for per-group metrics, along with demographic_parity_ratio and equal_opportunity_difference.

from fairlearn.metrics import (
    MetricFrame, selection_rate, true_positive_rate,
    demographic_parity_ratio, equal_opportunity_difference,
)

y_pred = model.predict(X_test)
group = sensitive_features["gender"]
mf = MetricFrame(
    metrics={"selection_rate": selection_rate, "tpr": true_positive_rate},
    y_true=y_test, y_pred=y_pred, sensitive_features=group,
)
print(mf.by_group)
print(demographic_parity_ratio(y_test, y_pred, sensitive_features=group))
print(equal_opportunity_difference(y_test, y_pred, sensitive_features=group))

The per-group rates should match. A mismatch usually points to misaligned indexes or a label encoded the other way round, and once a gap is confirmed, the next step is to find the features driving it with SHAP before choosing a mitigation.

 

How SHAP Model Interpretability Exposes Proxy Variables

SHAP (SHapley Additive exPlanations) assigns each feature a contribution to each prediction, which shows whether a model leans on a proxy even after the sensitive column was dropped. SHAP's TreeExplainer supports XGBoost, LightGBM, CatBoost, PySpark and most tree-based scikit-learn models.

import numpy as np
import pandas as pd
import shap

def interpret_model_decisions(model, X_sample):
    # X_sample: same feature columns the model was trained on
    explainer = shap.TreeExplainer(model)
    sv = explainer(X_sample)
    if sv.values.ndim == 3:  # scikit-learn classifiers return values per class
        sv = sv[:, :, 1]     # keep the positive class

    print("Global feature importance (mean absolute SHAP value):")
    global_importance = pd.Series(np.abs(sv.values).mean(axis=0), index=X_sample.columns)
    print(global_importance.sort_values(ascending=False))

    print("\nLocal explanation for the first row:")
    local = pd.Series(sv.values[0], index=X_sample.columns)
    print(local.sort_values(key=abs, ascending=False))

The ndim == 3 branch exists because scikit-learn classifiers such as RandomForestClassifier return SHAP values for both classes, while XGBoost and LightGBM with default settings return a single set. Global importance ranks features by their average impact. The local table shows which features pushed one prediction up (positive values) or down (negative values).

 

The warning sign is a likely proxy near the top of global importance, or local explanations for people in a disadvantaged group dominated by that same feature. That's the zip code premium pattern again. Retrain without the feature, compare accuracy and the fairness metrics, and record the result. Our view is that a large gap traced to one proxy is usually better fixed in the features or the data before anyone reaches for a mitigation algorithm.

 

What Fairness Documentation Should a Model Ship With?

Every model that makes decisions about people should ship with a brief record of how it was tested for fairness and who approved it. Model cards were proposed for this. The idea comes from the paper "Model Cards for Model Reporting" by Margaret Mitchell and colleagues. A model card is a short document that accompanies a trained model, reporting evaluation across conditions such as demographic and intersectional groups and stating the context the model is intended for.

 

For the checks in this guide, the fairness section of a model card should record:

 

  • the intended use, and the decisions the model must not be used for
  • the sensitive features and intersections tested, with row counts
  • selection rates, true positive rates and error rates per group, against the agreed thresholds
  • suspected proxies, their SHAP rank and what the team decided about them
  • the reviewer who signed off, the date and the test set version

 

Teams on Azure Machine Learning can export Responsible AI dashboard results as a PDF scorecard and share it with product and compliance stakeholders. A versioned Markdown file stored next to the model artefact does the same job on any stack.

 

A Checklist for Building Fair Machine Learning Models Before Deployment

A fairness check only protects people if a failed result stops the release. We recommend this checklist for any model that makes decisions about people, with thresholds set together with the risk or compliance lead.

 

  1. Write down the decision the model makes, the groups it affects and the fairness metric that matters most for that decision.
  2. Agree thresholds for subgroup error gaps, the selection-rate ratio and the true positive rate gap before anyone sees results.
  3. Run the four checks on the latest test set and record the numbers as the baseline.
  4. Fail the CI step when a threshold is crossed, using a gate like the one below.
  5. Name one owner who signs off on fairness for each model, while the whole team stays responsible for building it fairly.
  6. Rerun the checks on every retrain and on a fixed schedule in production.
import sys

def fairness_gate(results, min_ratio=0.8, max_tpr_gap=0.05):
    # Placeholder thresholds: replace with the values your team agreed
    failures = []
    for feature, r in results.items():
        if r["selection_ratio"] < min_ratio:
            failures.append(f"{feature}: selection-rate ratio {r['selection_ratio']:.2f}")
        if r["tpr_gap"] > max_tpr_gap:
            failures.append(f"{feature}: TPR gap {r['tpr_gap']:.2f}")
    if failures:
        print("Fairness gate failed:\n" + "\n".join(failures))
        sys.exit(1)
    print("Fairness gate passed")

The gate takes the dictionary returned by fairness_assessment. A non-zero exit fails the pipeline step. The same loop shows up in delivery work: Arbisoft's data science services page describes using techniques to identify and minimise bias across data collection, continuously monitoring deployed models in real-world use and implementing automated retraining as new data arrives.

 

Frequently asked questions

Does removing sensitive attributes make a machine learning model fair?

No. Proxies such as zip code or first name can carry the same information as race or gender, and the model learns the same pattern without the sensitive column. Keep the sensitive attributes in the test data even when they're excluded from training, because the fairness checks need them to group results.

Can a model meet demographic parity and equal opportunity at the same time?

Not always. When the share of truly positive cases differs between groups, a perfectly accurate model has a true positive rate of 1.0 for every group and meets equal opportunity, yet its selection rates differ and it fails demographic parity. Pick the metric that matches the harm the decision can cause, and record the reason in the model card.

How small can a subgroup be before its error rate is unreliable?

There is no fixed cutoff. Report counts next to every rate and treat cells with only a few dozen rows as indicative, and if an important group is that small, collect more data for it before setting a threshold.

Do I need the Azure Responsible AI dashboard to run these checks?

The dashboard is optional. Everything here runs with pandas, SciPy, SHAP and Fairlearn. As of November 2026, the dashboard supports classification and regression models trained on tabular data, registered in Azure Machine Learning as scikit-learn MLflow models, and its interface displays up to 5 thousand data points. It suits teams already working in Azure.

How often should fairness checks run after a model is deployed?

Run them on every retrain and on a fixed schedule between retrains, because input data drifts and feedback loops change who gets approved, and a model that passed last quarter can fail today. For most high-stakes models we'd suggest monthly runs, plus an extra one after any product or policy change.

 

Get Help Auditing Fairness in Production ML Models

Building fair machine learning models starts with the model whose decisions carry the most weight for people, whether that is a credit limit, an insurance price, a triage score or a screening result. Run the four checks on it before your 2027 release plans are set and write the gaps into its model card. Those numbers become the baseline for real thresholds. A CI gate with a named owner then makes the thresholds enforceable.

 

Arbisoft's AI development services cover machine learning delivery and, where needed, include human-in-the-loop oversight and full audit trails. We can help audit fairness in models already in production, or build these checks into a training and release pipeline from the start.

...Loading Related Blogs

Explore More

From Introduction to Proposal in Days

Discovery Call
Our sales team reviews your message and asks for a discovery call to gather more information.
Expert Input
Our veterans go through your requirements to provide their take, backed by decades of experience.
Proposal
We provide a proposal specific to what you're building, for you to review at your own pace.

Trusted by top platforms for our transformative solutions and exceptional results:

How Can We Help You Build?

We recommend using your work email.
We'll send a mutual NDA before the discovery call if requested. Zero obligation.