Building Fair Machine Learning Models in 2026: Bias Tests, Fairness Metrics and AI Governance

Ateeb TaseerLast Updated on November 3, 2026

13-14 Min Read TimeAdd as preferred on Google

Key Takeaways

A model can score high on overall accuracy and still fail one group, so fair ML requires group-by-group checks that block deployment when thresholds are crossed.

• Dropping race or gender columns does not remove bias, because proxies such as zip code can carry the same information.
• Bias enters through data disparities, algorithmic amplification, proxy variables, feedback loops and historical labels, and each has a matching check.
• Subgroup error analysis catches gaps that overall accuracy hides, including failures at intersections such as older women. Small cells need more data.
• Check demographic parity and equal opportunity together, since they catch different failures. A four-fifths selection ratio is only a rough screen.
• SHAP shows whether a model relies on a likely proxy. Retrain without that feature and compare accuracy and fairness metrics.
• A failed fairness check should stop the release.

Start with the production model that makes the highest-stakes decisions about people, run the four checks and record the gaps.

A credit, pricing, triage or screening model can post strong accuracy on its test set and still make far more mistakes for one group of people. Overall accuracy hides that gap. Dropping race or gender columns does not close it either, because features such as zip code can carry the same information. Building fair machine learning models means checking results group by group and blocking deployment until those checks pass. This guide is for ML engineers, data scientists and the leads who approve models that make decisions about people. It explains how AI ethics and AI governance divide the work, five ways bias enters a model, four Python checks, what to document and a pre-deployment checklist.

How AI Ethics and AI Governance Differ for ML Teams

AI ethics defines what a model should do for the people it affects. AI governance is the set of policies, owners, reviews and records that makes a team actually do it. For a lending model, ethics says applicants with similar repayment risk should get similar decisions. Governance decides who runs the fairness checks before release and which threshold blocks it.

AI ethics usually rests on four principles. Fairness means avoiding discrimination and bias, and accountability means errors can be challenged and corrected. Transparency means decisions can be explained. Privacy means personal data is collected, stored and used with care. AI governance turns those principles into internal policies, technical standards, oversight such as audits and impact assessments, and compliance with external regulation.

In our view, each fails without the other. A principle with no gate rarely stops a biased release. A gate with no agreed principle tends to measure whatever is easiest to compute. The rest of this article covers the point where the two meet: bias tests produce the evidence, and governance decides what happens to it.

Why Bias in Machine Learning Models Causes Real Harm

Biased models move money and access away from specific groups, usually without the people affected knowing a model was involved. In April 2017, ProPublica and Consumer Reports analysed car insurance premiums in California, Illinois, Missouri and Texas. They found that some insurers charged premiums that were on average 30 percent higher in zip codes where most residents are minorities than in whiter neighborhoods with similar accident costs.

In that analysis, premiums differed by zip code. Zip code is a typical proxy variable, and it comes back in the checks below. The same pattern can appear in a credit limit or a CV screening score, and it stays invisible when the only number anyone reviews is overall accuracy.

Five Ways Bias Gets Into Machine Learning Models

Most bias enters through the training data or through the way the model learns from it. Each source below names the check that catches it.

1. Data disparities

A model learns most about the groups it sees most often. Suppose most of a diagnostic model's training records come from one demographic. The model then has far fewer examples of how a condition presents in everyone else, which raises the chance of misdiagnosing them. Data exploration (group shares) and subgroup error analysis catch this.

2. Algorithmic amplification

A model that optimises overall accuracy learns the majority group's patterns first, because that is where most of the loss comes from. Small gaps in the data can become larger gaps in errors. A group that makes up only a small share of training rows can end up with a much higher error rate while headline accuracy still looks fine. Subgroup and intersectional error analysis catch it.

3. Proxy variables

Removing race or gender from the feature set leaves that information in the data. Zip code, first name or school attended can track a sensitive attribute closely enough for the model to learn the same pattern anyway, as in the insurance premiums above. Data exploration shows whether a feature tracks a sensitive attribute. SHAP shows whether the model relies on it.

4. Feedback loops

A model's decisions shape the data its next version trains on. A lending model that rejects most applicants from one group never learns whether they would have repaid. The retraining data then holds little repayment history for that group, and the next model keeps the gap. Fairness metrics rerun on every retrain catch the drift.

5. Historical labels

Labels often record past decisions, and past decisions carry past inequalities. A hiring model trained on previous hiring outcomes learns to repeat whoever was favoured before. Target rates by group in the data explorer flag this before training, and fairness metrics confirm it after.

How to Test a Machine Learning Model for Bias in Python

Two checks need nothing beyond pandas and should run first: error rates per group and a profile of the data. They mirror components of Microsoft's Responsible AI dashboard in Azure Machine Learning, which, as of October 2026, groups error analysis, data analysis, fairness assessment and model interpretability with counterfactual and causal analysis tools. The snippets in this article use pandas, scipy and the open-source SHAP library, which means they run in any pipeline.

Subgroup and intersectional error analysis

import pandas as pd

def deep_dive_error_analysis(model, X_test, y_test, sensitive_features):
    # X_test: features only. sensitive_features: DataFrame aligned to X_test's index
    errors = pd.Series(model.predict(X_test) != y_test.to_numpy(), index=X_test.index)
    print(f"Overall error rate: {errors.mean():.3f} (n={len(errors)})")
    for feature in sensitive_features.columns:
        subgroup_errors = errors.groupby(sensitive_features[feature]).agg(['mean', 'size'])
        print(f"\nError rates and counts by {feature}:")
        print(subgroup_errors)

    # Intersectional cells (the size column shows cells too small to trust)
    intersectional_errors = errors.groupby(
        [sensitive_features[f] for f in sensitive_features.columns]
    ).agg(['mean', 'size'])
    print("\nIntersectional error rates and counts:")
    print(intersectional_errors)

Error analysis compares each group's error rate with the overall rate printed at the top, then repeats the comparison for combinations such as gender by age band. Bias often concentrates at intersections. A model can treat women and older applicants acceptably as separate groups and still fail older women.

The warning sign is a group whose error rate sits well above the overall rate. Watch the size column too. A cell with a handful of rows produces a rate that is unreliable in either direction, and the right response is more data for that group or a closer look at its features before the model ships.

Data exploration for representation, label gaps and proxies

import pandas as pd

def advanced_data_explorer(data, sensitive_features):
    # data includes the 'target' column; sensitive_features is a list of column names
    for feature in sensitive_features:
        print(f"\nAnalyzing {feature}:")
        print("Share of rows by group:")
        print(data[feature].value_counts(normalize=True))
        print("Positive-target rate by group:")
        print(data.groupby(feature)['target'].mean())
        for other_feature in sensitive_features:
            if feature != other_feature:
                # Raw counts, so small intersections stay visible
                print(f"\nIntersection with {other_feature}:")
                print(pd.crosstab(data[feature], data[other_feature]))

The share of rows shows how much of the data each group contributes. Target rates show whether outcomes in the labels already differ by group, which is a sign of historical bias. The crosstabs give the row count in every intersection.

Warning signs are a group with a very small share, target rates that differ sharply across groups and intersections with almost no rows. To test a suspected proxy, run the same groupby on a non-sensitive feature, for example data.groupby('zip_code')['ethnicity'].value_counts(normalize=True). If some zip codes are dominated by one group, flag zip code as a likely proxy and check its SHAP importance later.

Which Fairness Metrics Should You Check Before Deployment?

Check at least two group fairness metrics, demographic parity and equal opportunity, because they catch different failures. Demographic parity compares selection rates, the share of each group that receives the positive prediction. Equal opportunity compares true positive rates, meaning how often people whose real outcome is positive are predicted positive, group by group.

import pandas as pd
from scipy.stats import chi2_contingency

def fairness_assessment(model, X_test, y_test, sensitive_features):
    # Binary 0/1 predictions and labels; sensitive_features aligned to X_test's index
    predictions = pd.Series(model.predict(X_test), index=X_test.index)
    y_true = pd.Series(y_test.to_numpy(), index=X_test.index)
    positives = y_true == 1
    results = {}
    for feature in sensitive_features.columns:
        groups = sensitive_features[feature]
        selection_rate = predictions.groupby(groups).mean()               # demographic parity
        tpr = predictions[positives].groupby(groups[positives]).mean()    # equal opportunity
        _, p_value, _, _ = chi2_contingency(pd.crosstab(groups, predictions))
        results[feature] = {
            "selection_ratio": selection_rate.min() / selection_rate.max(),
            "tpr_gap": tpr.max() - tpr.min(),
            "p_value": p_value,
        }
        print(f"\n{feature}: {results[feature]}")
        print(pd.DataFrame({"selection_rate": selection_rate, "tpr": tpr}))
    return results

Start with the ratio and gap, then look at the per-group rates to see which groups sit at each end. A selection-rate ratio below four-fifths (80 percent) is a common rough screen. It comes from the four-fifths rule in the US Uniform Guidelines on Employee Selection Procedures, which treat a selection rate below four-fifths of the highest group's rate as evidence of adverse impact in hiring. The same Guidelines say smaller differences can still count where they are significant in both statistical and practical terms. Treat that ratio as a screen only, and agree the threshold for the true positive rate gap before anyone sees the results.

The chi-square p-value tests whether predictions are independent of group. A small p-value says a gap this size would be unlikely by chance, and the cutoff should be agreed in advance. Read it next to the gap and the group sizes, because large samples make tiny gaps significant and small samples can hide large ones. Once a gap is confirmed, find the features driving it with SHAP before choosing a mitigation.

How SHAP Model Interpretability Exposes Proxy Variables

SHAP (SHapley Additive exPlanations) assigns each feature a contribution to each prediction. That shows whether a model leans on a proxy even after the sensitive column was dropped. SHAP's TreeExplainer works with tree ensembles such as XGBoost, LightGBM, CatBoost and most scikit-learn tree models.

import numpy as np
import pandas as pd
import shap

def interpret_model_decisions(model, X_sample):
    # X_sample: same feature columns the model was trained on
    explainer = shap.TreeExplainer(model)
    sv = explainer(X_sample)
    if sv.values.ndim == 3:  # scikit-learn classifiers return values per class
        sv = sv[:, :, 1]     # keep the positive class

    print("Global feature importance (mean absolute SHAP value):")
    global_importance = pd.Series(np.abs(sv.values).mean(axis=0), index=X_sample.columns)
    print(global_importance.sort_values(ascending=False))

    print("\nLocal explanation for the first row:")
    local = pd.Series(sv.values[0], index=X_sample.columns)
    print(local.sort_values(key=abs, ascending=False))

The ndim == 3 branch exists because scikit-learn classifiers return SHAP values for each class, while XGBoost and LightGBM with default settings return one set. Global importance ranks features by their average impact. The local table shows which features pushed one prediction up (positive values) or down (negative values).

The warning sign is a likely proxy near the top of global importance, or local explanations for people in a disadvantaged group dominated by that same feature. That is the zip code premium pattern, found inside your own model. The next test is to retrain without the feature and compare accuracy and the fairness metrics, then record the result so any fix rests on evidence.

What Fairness Documentation Should a Model Ship With?

Every model that makes decisions about people should ship with a short record of how it was tested for fairness and who approved it. The research paper "Model Cards for Model Reporting" by Mitchell and colleagues proposed model cards for this purpose. A model card is a short document that travels with a trained model, reports its evaluation across demographic and intersectional groups and states the context the model is intended for.

For the checks in this article, the fairness section of a model card should record:

• the intended use, and the decisions the model must not be used for
• the sensitive features and intersections tested, with row counts
• selection rates, true positive rates and error rates per group, against the agreed thresholds
• suspected proxies, their SHAP rank and what the team decided about them
• the reviewer who signed off, the date and the test set version

Teams on Azure Machine Learning can export Responsible AI dashboard results as a PDF scorecard for product and compliance stakeholders. A versioned Markdown file stored next to the model artefact does the same job on any stack.

A Fairness Governance Checklist Before Deployment

A fairness check only protects people if a failed result stops the release. The checklist below is the one we recommend for any model that makes decisions about people, with thresholds set together with the risk or compliance lead.

1. Write down the decision the model makes, the groups it affects and the fairness metric that matters most for that decision.
2. Agree thresholds for subgroup error gaps, the selection-rate ratio and the true positive rate gap before anyone sees results.
3. Run the four checks on the latest test set and record the numbers as the baseline.
4. Fail the CI step when a threshold is crossed, using a gate like the one below.
5. Name one owner who signs off on fairness for each model, while the whole team stays responsible for building it fairly.
6. Rerun the checks on every retrain and on a fixed schedule in production.

import sys

def fairness_gate(results, min_ratio=0.8, max_tpr_gap=0.05):
    # Placeholder thresholds: replace with the values your team agreed
    failures = []
    for feature, r in results.items():
        if r["selection_ratio"] < min_ratio:
            failures.append(f"{feature}: selection-rate ratio {r['selection_ratio']:.2f}")
        if r["tpr_gap"] > max_tpr_gap:
            failures.append(f"{feature}: TPR gap {r['tpr_gap']:.2f}")
    if failures:
        print("Fairness gate failed:\n" + "\n".join(failures))
        sys.exit(1)
    print("Fairness gate passed")

The gate takes the dictionary returned by fairness_assessment, and a non-zero exit fails the pipeline step. A practical first step this week is to pick the production model with the highest-stakes decisions about people, run the four checks and write down the gaps. Those numbers become the starting point for real thresholds. Arbisoft's data science services describe the same loop in delivery work: identifying and reducing bias across data collection, model training and predictions, then monitoring a deployed model's inputs and outputs and retraining it automatically on new data.

Frequently asked questions

Does removing sensitive attributes make a machine learning model fair?

Removing sensitive attributes does not make a model fair, because proxies such as zip code, first name or school can carry the same information as race or gender and the model can learn the same pattern without the sensitive column. The data explorer shows whether a feature tracks a sensitive attribute. SHAP then shows whether the model relies on it.

Can a model meet demographic parity and equal opportunity at the same time?

A model cannot always meet both. When the share of truly positive cases differs between groups, a perfectly accurate model has a true positive rate of 1.0 for every group and meets equal opportunity, yet its selection rates differ and it fails demographic parity. Pick the metric that matches the harm the decision can cause, and record why.

How small can a subgroup be before its error rate is unreliable?

There is no fixed cutoff. Report counts next to every rate and treat cells with only a few dozen rows as indicative. If an important group is that small, collecting more data for it usually comes before any threshold decision.

Do I need the Azure Responsible AI dashboard to run these checks?

The dashboard is not required, because the same checks run with pandas, scipy, SHAP and the open-source Fairlearn library, which offers MetricFrame for per-group metrics and functions such as demographic_parity_ratio. The dashboard puts these tools in one interface and supports classification and regression models trained on tabular data, which suits teams already working in Azure Machine Learning.

How often should fairness checks run after a model is deployed?

On every retrain, and on a fixed schedule between retrains. Input data drifts and feedback loops change who gets approved, so a model that passed last quarter can fail the same checks today. In our view, monthly runs suit most high-stakes models, with an extra run after any product or policy change.

Get Help Auditing Fairness in Production ML Models

The best place to start is the model whose decisions carry the most weight for people, whether that is a credit limit, an insurance price, a triage score or a screening result. Running the four checks on it gives a baseline. Documenting the gaps in its model card and turning the agreed thresholds into a CI gate with a named owner makes that baseline enforceable. In our view, large gaps call for fixing the data or the proxy features before reaching for mitigation algorithms. Arbisoft's AI development services cover machine learning delivery and, where needed, include human-in-the-loop oversight and full audit trails. The team can help audit existing models or build these fairness checks into an MLOps pipeline.