Your AI Model Works Well Overall. But Is It Failing One Customer Group?

A company deploys a machine learning model to identify customers who may need additional support.

The model performs well in validation.

Its overall accuracy is 90%.

Management is satisfied. The model appears reliable, and the company begins using its predictions to prioritize customer outreach.

Several weeks later, the operations team notices a problem.

One customer segment is receiving substantially fewer successful identifications than another.

The model still reports 90% overall accuracy.

But that aggregate number conceals a difference in how effectively it identifies customers who actually need support.

This is a common analytical risk: a model can perform well on average while producing materially different error rates across subgroups.

A responsible evaluation must examine not only whether the model performs well overall, but also where it performs poorly, who experiences the errors, and what those errors mean for the business.

The Short Answer: Aggregate Accuracy Can Hide Important Failures

Overall accuracy is the proportion of predictions classified correctly.

Accuracy = Correct Predictions / All Predictions

It is a useful metric in some settings.

But it combines different kinds of errors and averages performance across the entire evaluation population.

If one subgroup is large and another is small, the larger group can dominate the overall result.

Even when groups have similar sizes, their underlying outcome rates may differ.

Therefore, a high overall accuracy score does not establish that a model is equally useful across customer segments.

The appropriate audit depends on the intended use, relevant populations, and consequences of mistakes.

A Worked Example: Two Customer Segments

Consider a hypothetical company using a classifier to identify customers who genuinely require proactive support.

The positive class means that the customer needs support.

The company evaluates the model on 1,000 customers divided into two segments.

Each segment contains 500 customers.

The observed outcomes and model predictions produce the following confusion matrices.

These numbers are synthetic and deliberately constructed.

They are not results from a real company or a deployed AI system.

Calculate Overall Accuracy

For Segment A:

Correct predictions = 90 + 360 = 450.

Accuracy = 450 / 500 = 90%.

For Segment B:

Correct predictions = 50 + 400 = 450.

Accuracy = 450 / 500 = 90%.

Combining both segments:

Total correct predictions = 900.

Total observations = 1,000.

Overall accuracy = 900 / 1,000 = 90%.

At first glance, the model appears equally accurate for both segments.

But accuracy alone does not reveal how many customers who actually need support are identified.

Compare Recall Across Groups

Recall measures the proportion of actual positive cases correctly identified.

Recall = True Positives / (True Positives + False Negatives)

For Segment A:

Recall = 90 / (90 + 10) = 90%.

For Segment B:

Recall = 50 / (50 + 50) = 50%.

The recall gap is:

90% - 50% = 40 percentage points.

Both groups have 90% accuracy.

Yet the model identifies 90% of support-needing customers in Segment A and only 50% in Segment B.

That is a substantial operational difference.

If the purpose of the model is to identify customers who require help, recall may be more relevant than accuracy.

Why the Business Consequences Differ

Suppose failing to identify a customer who needs support creates an estimated $200 in avoidable business cost.

This is a hypothetical planning assumption, not a measured causal impact.

Segment A has 10 false negatives.

Modeled false-negative cost:

10 × $200 = $2,000.

Segment B has 50 false negatives.

Modeled false-negative cost:

50 × $200 = $10,000.

The difference is $8,000.

The company could report identical accuracy for both segments while experiencing substantially different modeled costs from missed cases.

However, this calculation is not a complete cost analysis.

Segment A also has 40 false positives, while Segment B has none.

If unnecessary outreach is expensive or disruptive, those errors matter too.

A decision-ready analysis must evaluate the full consequences of both false negatives and false positives.

Compare False Positive Rates

The false positive rate measures the share of actual negative cases incorrectly classified as positive.

False Positive Rate = False Positives / (False Positives + True Negatives)

For Segment A:

40 / (40 + 360) = 10%.

For Segment B:

0 / (0 + 400) = 0%.

The two segments therefore have different error profiles.

Segment A has higher recall but also more false positives.

Segment B has fewer false positives but misses many more positive cases.

Neither profile can be declared universally preferable without understanding the business objective, error costs, and relevant fairness requirements.

Why Equal Accuracy Does Not Mean Equal Treatment

The example demonstrates an important distinction.

Equal accuracy does not imply equal recall.

Equal recall does not necessarily imply equal false positive rates.

And equal error rates do not automatically establish that a system is fair in every meaningful sense.

Fairness depends on context.

For example, a model used to recommend optional marketing content raises different concerns from one used to determine access to employment, credit, housing, or healthcare.

Relevant legal, ethical, and operational requirements may also differ.

A subgroup audit is therefore evidence for governance, not a universal fairness certificate.

Which Groups Should Be Evaluated?

Subgroups should be selected according to the model's purpose and foreseeable risks.

Possible operational segments include:

• Geographic region.
• Customer tenure.
• Product category.
• Acquisition channel.
• Device type.
• Language preference.
• New versus returning customers.
• Different service locations.

In appropriate and lawful contexts, protected or sensitive demographic groups may also require evaluation.

Such analysis must follow applicable privacy, security, and anti-discrimination requirements.

Organizations should not collect sensitive attributes casually or infer them from proxies without a legitimate, carefully governed purpose.

The goal is to detect meaningful performance differences while protecting the people represented in the data.

Why Subgroup Sample Size Matters

A model may appear to have 100% recall in a subgroup containing only five positive cases.

That result is not strong evidence of consistently perfect performance.

Small samples produce uncertain estimates.

A useful subgroup report should include:

• Total subgroup size.
• Number of actual positive cases.
• Number of actual negative cases.
• Confusion-matrix counts.
• Relevant rates.
• Uncertainty intervals where appropriate.
• Missing-data information.
• Time period and evaluation population.

Some metrics are undefined when their denominator is zero.

For example, recall cannot be calculated when a subgroup contains no actual positive cases.

Such results should be marked as undefined, not reported as zero or 100%.

When subgroup samples are too small, the organization may need more representative data or a carefully designed evaluation strategy.

Check Intersectional Subgroups

Evaluating one dimension at a time can miss problems.

For example, a model may perform similarly across two geographic regions and similarly across two customer-tenure categories.

But performance could still be poor among new customers in one specific region.

An intersectional analysis examines combinations of relevant attributes.

However, each additional split reduces sample size.

The evaluation must balance the value of detailed diagnosis against statistical uncertainty and privacy risks.

A long table of noisy subgroup metrics is not necessarily a better audit.

A Practical Python Subgroup Audit

The following example reproduces the synthetic confusion-matrix results.

groups = {
    "Segment A": {
        "tp": 90,
        "fn": 10,
        "fp": 40,
        "tn": 360
    },
    "Segment B": {
        "tp": 50,
        "fn": 50,
        "fp": 0,
        "tn": 400
    }
}

for name, m in groups.items():

    total = sum(m.values())

    accuracy = (m["tp"] + m["tn"]) / total

    recall = m["tp"] / (m["tp"] + m["fn"])

    fpr = m["fp"] / (m["fp"] + m["tn"])

    print(
        name,
        "accuracy:", round(accuracy, 2),
        "recall:", round(recall, 2),
        "false positive rate:", round(fpr, 2)
    )

Expected output:

Segment A accuracy: 0.9 recall: 0.9 false positive rate: 0.1

Segment B accuracy: 0.9 recall: 0.5 false positive rate: 0.0

The example demonstrates that identical accuracy can coexist with substantially different error rates.

In a real audit, the evaluation data should be independent of model training and threshold selection.

Could Different Thresholds Solve the Problem?

Possibly, but not automatically.

Classification thresholds determine how model scores become positive or negative decisions.

Changing a threshold may alter recall and false positive rates.

A lower threshold may identify more positive cases but also generate more false positives.

A higher threshold may reduce unnecessary interventions while missing additional customers who need support.

Organizations should examine these trade-offs carefully.

Applying different thresholds to different groups can raise additional fairness, governance, and legal concerns.

Such changes should not be made solely to make one metric look equal.

The model's intended purpose, legitimate decision criteria, and consequences for affected individuals must guide policy design.

For a complementary discussion of threshold economics, see:

Why Can a Lower-Accuracy Classification Threshold Save More Money?

Why Calibration Is a Separate Question

A model can also produce inaccurate probability estimates.

For example, predictions near 70% should correspond to an observed event frequency near 70% among appropriately comparable cases if the model is well calibrated.

Calibration and subgroup error rates are related but distinct evaluation dimensions.

A model may be well calibrated overall while having unequal error rates across groups.

Conversely, a model can have similar recall across groups while producing poorly calibrated probabilities.

UAD Consulting's existing article on probability calibration explains this separate issue:

Your Model Has a Good AUC. Can You Trust Its Probabilities?

A comprehensive audit may need both analyses.

Does an Observed Disparity Prove Discrimination?

No.

A difference in model performance is an important diagnostic finding, but it does not automatically establish its cause.

Potential explanations include:

• Different data quality.
• Unequal representation in training data.
• Different outcome prevalence.
• Measurement errors.
• Missing features.
• Changes in customer behavior.
• Differences in how outcomes are observed.
• Inappropriate target definitions.
• Historical processes reflected in the data.

A responsible investigation should examine plausible mechanisms rather than assuming that one observed metric identifies the cause.

At the same time, the absence of proven causation is not a reason to ignore a material disparity.

The organization should assess whether the difference creates unacceptable risks and what corrective action is justified.

Prediction Is Not Causal Impact

Suppose the company improves recall in Segment B.

That means the model identifies a greater share of actual positive cases under the evaluation conditions.

It does not prove that customer outcomes will improve.

The organization must still determine whether its outreach intervention is effective.

A model can correctly identify customers who need support while recommending an intervention that has little impact.

To estimate the causal effect of outreach, the company may need a randomized pilot or another credible counterfactual evaluation.

Prediction quality, intervention effectiveness, and business impact are separate questions.

A Seven-Step Subgroup Model Audit

1. Define the Decision

Document what the model predicts and how predictions affect people or business processes.

2. Identify Relevant Populations

Select subgroups based on foreseeable risk, operational context, and applicable governance requirements.

3. Validate the Evaluation Data

Check label quality, missingness, representation, time periods, and independence from training.

4. Measure Multiple Performance Dimensions

Calculate confusion-matrix counts, recall, false positive rates, precision, and calibration where relevant.

5. Quantify Uncertainty

Report denominators and uncertainty rather than treating small-sample estimates as exact.

6. Investigate Material Differences

Examine data collection, model design, thresholds, and the consequences of errors.

7. Monitor After Deployment

Track performance over time, document decisions, and establish clear escalation and review responsibilities.

This process aligns with the broader risk-management principle that AI systems require ongoing measurement and governance rather than one-time validation.

Frequently Asked Questions

Can a model have high accuracy and still be unfair?

Yes. High accuracy does not rule out unequal error rates or other harmful differences. Whether a disparity constitutes unfairness requires contextual assessment.

What is subgroup performance evaluation?

It is the assessment of model behavior separately across relevant populations instead of relying only on aggregate metrics.

Which metric matters most for fairness?

There is no universal answer. Relevant metrics depend on the decision, potential harms, outcome prevalence, and legal or organizational requirements.

Is equal accuracy enough?

No. Groups can have equal accuracy but very different recall and false positive rates.

Can a model be fair without using sensitive attributes?

Excluding sensitive attributes does not guarantee fair outcomes. Other variables may act as proxies, and evaluation may still reveal disparities.

How often should subgroup performance be audited?

The schedule should reflect model risk, decision frequency, data changes, and operational consequences. High-impact systems may require more frequent monitoring.

The Bottom Line

A machine learning model's average performance is only part of the story.

In the hypothetical example, both customer segments achieve 90% accuracy.

Yet recall is 90% for Segment A and 50% for Segment B.

That difference means the model misses substantially more customers who actually need support in one segment, even though the aggregate accuracy appears strong.

The example does not establish that a real system is discriminatory or that changing the model will automatically improve business outcomes.

It demonstrates why organizations need more detailed evaluation before trusting model predictions in operational decisions.

The better question is not simply:

"How accurate is our model?"

It is:

"Where does our model make mistakes, who experiences those mistakes, and what consequences follow?"

That is how model evaluation becomes responsible decision support.

References and Methodological Notes

1. National Institute of Standards and Technology. AI Risk Management Framework.

1. National Institute of Standards and Technology. NIST AI RMF Playbook.

1. scikit-learn. Metrics and Scoring: Quantifying the Quality of Predictions.

1. scikit-learn. Confusion Matrix Documentation.

Methodological Notes

All customer counts, confusion matrices, accuracy values, recall values, false positive rates, and financial consequences are hypothetical.

The example is not based on actual customer data or a deployed AI model.

The $200 false-negative cost is an illustrative assumption, not a measured causal estimate.

The article does not equate observed performance disparities with legally established discrimination.

It does not claim that any single fairness metric is universally appropriate.

Real evaluations require representative data, uncertainty assessment, contextual governance, privacy protections, and appropriate legal review where applicable.

Evaluate Your Models Beyond Average Accuracy

UAD Consulting helps organizations evaluate predictive models, understand operational risk, and connect machine learning performance to defensible business decisions.

Explore ServicesGet in Touch