Why Does a Machine Learning Model Look Accurate but Fail in Production?

A company builds a machine learning model to predict customer churn.

During development, the results look excellent.

The model achieves an AUC of 0.94 on its test dataset. This is a hypothetical example, not a reported result.

Management approves deployment.

Three months later, the model performs much worse on newly observed customers.

The team investigates.

The algorithm has not necessarily malfunctioned.

The problem may have existed long before deployment.

The model may have been evaluated using information that would never have been available when a real prediction was required.

This problem is known as data leakage.

It is one reason a technically sophisticated machine learning model can appear successful during development while providing disappointing results in practice.

The central question is not simply:

“How accurate is our model?”

It is:

“Was the model evaluated under conditions that realistically represent the decisions it will make?”

That distinction changes how machine learning projects should be designed, validated, and interpreted.

What Is Data Leakage in Machine Learning?

Data leakage occurs when information enters model development or evaluation that would not legitimately be available for the prediction being simulated.

The information may come from:

• the future,
• the outcome being predicted,
• the evaluation dataset,
• related records belonging to the same entity,
• or preprocessing operations performed before separating training and evaluation data.

The result can be an overly optimistic performance estimate. The model appears to recognize useful patterns, but some of those patterns depend on information it will not have in production.

Scikit-learn’s guidance on common pitfalls explains that leakage can arise when information unavailable at prediction time is used during model building, including when evaluation data influence preprocessing or feature selection.

This is not merely a coding convention. It is a requirement for credible evaluation.

A Model Can Be Correctly Trained and Incorrectly Evaluated

Consider a customer churn prediction project. The business wants to identify customers likely to cancel their subscriptions during the next 30 days. Predictions will be generated on the first day of each month.

At that moment, the company knows the customer’s subscription plan, historical usage, previous payments, support interactions, and account tenure.

But it does not yet know whether the customer will cancel later that month, whether a retention offer will be accepted, whether a cancellation request will be submitted next week, or how much usage will occur after the prediction date.

Suppose the training dataset contains a variable called cancellation_request_submitted. If this field is recorded after the prediction date, it may reveal the future outcome.

A model using that variable could achieve impressive accuracy. But the variable would not be available when the company actually needs to predict churn. The model is effectively being asked to predict something after receiving evidence of the answer.

This is a form of target leakage. The solution is not necessarily to remove every variable associated with churn. The solution is to establish exactly when each variable becomes available.

Prediction Time Is the Most Important Boundary

A useful machine learning project should explicitly define its prediction timestamp.

This creates a point-in-time boundary. A feature can be perfectly legitimate in the source database and still be invalid for a particular prediction.

The database may contain the complete customer history. The model must not automatically receive the complete history. It should receive only the information that would have been available at the relevant decision point.

This is why machine learning data preparation often requires more than ordinary database joins. It requires point-in-time correctness.

A Simple Example of Temporal Leakage

Suppose a business wants to predict whether a customer will churn during November. Its dataset contains:

The last three fields are not automatically suitable prediction features because they occur after the prediction timestamp.

Even the previous 30-day usage variable needs verification. If the data pipeline does not finalize October usage until November 3, the finalized value may not have been available on November 1.

Therefore, two timestamps may matter:

Event time: When the activity occurred.

Availability time: When the information became usable by the model.

This distinction is especially important in systems with delayed processing, billing adjustments, medical coding, or asynchronous event ingestion.

The Hidden Problem With Random Train-Test Splits

Random train-test splitting is common in machine learning. For independent observations drawn from a stable population, it can be appropriate. But many business datasets contain repeated observations from the same underlying entity.

Examples include multiple transactions from one customer, repeated hospital visits from one patient, multiple claims from one policyholder, several observations from one machine, and repeated activity records from one website user.

Suppose a dataset contains 1,000 customers. Each customer contributes four records.

The analyst randomly assigns individual records to 80% training and 20% testing. Records belonging to the same customer can then appear in both datasets.

If the intended production task is predicting outcomes for entirely new customers, this evaluation may be misleading. The model may learn stable customer-specific patterns from training records and then be evaluated on other records from the same customer.

A Worked Example: How Often Can Customers Appear in Both Datasets?

Assume each of the four records belonging to a customer is independently assigned to training with probability 0.80 and testing with probability 0.20.

First calculate the probability that all four records go to training:

Now calculate the probability that all four records go to testing:

The customer appears in both datasets unless all records are assigned to one side. Therefore:

Approximately 58.88% of customers would appear in both training and testing under this simplified independent-assignment model.

For 1,000 customers, the expected number appearing in both is:

Approximately 589 customers.

This is a mathematical illustration, not an empirical estimate from a real customer dataset. It assumes independent record-level assignment rather than enforcing an exact total number of training and test records.

The result demonstrates an important point: a random split of records is not necessarily an independent split of customers.

A dataset can contain thousands of test records without providing a genuinely independent test of performance on new entities. For a deeper explanation of repeated records and analytical grain, see understanding the grain of the data.

The Correct Split Depends on the Deployment Question

There is no universally correct splitting method. The correct evaluation design depends on what the model will encounter in production.

A customer-grouped split ensures that customers in the test set are not also represented in training. A forward-time split evaluates whether the model can predict future observations using earlier information. A hospital-level holdout tests whether a model transfers to organizations not represented in training.

These designs answer different questions. Excluding every previously observed customer from testing may be unnecessarily restrictive if the deployed model will routinely predict future behavior for existing customers. Conversely, random row-level testing may be too optimistic when the business wants predictions for entirely new customers.

The evaluation should reflect the intended deployment population and prediction workflow.

Grouped Validation

Scikit-learn provides GroupKFold for situations in which observations belong to groups. It keeps a group from appearing in both the training and test portions of a fold.

For customer-level data, the grouping variable could be customer_id. For healthcare data it could be patient_id. For hospital generalization it could be hospital_id.

The grouping level should match the independence requirement. Grouping by patient does not automatically establish generalization across hospitals. Similarly, grouping by hospital may not reproduce the temporal conditions of future deployment.

A model may require multiple validation strategies to answer multiple deployment questions.

Temporal Validation

Time introduces another challenge. Suppose the model will be trained using historical data and used to predict future outcomes. A random split may allow later observations to influence training while earlier observations are used for evaluation, creating an unrealistic evaluation.

A forward-time design is often more appropriate:

The model is developed using earlier information and evaluated on later information. Scikit-learn’s TimeSeriesSplit supports time-ordered training and test partitions.

A time-series splitter does not automatically solve every leakage problem. The analyst must still consider feature availability, overlapping outcome windows, repeated entities, reporting delays, and whether training labels would have been known by the training cutoff.

For some problems, a temporal gap or more specialized purging strategy is necessary. The required gap depends on the label horizon, information availability, and the way observations overlap. It should not be selected arbitrarily.

Leakage Can Happen During Preprocessing

Data leakage is not limited to obvious future variables. It can also occur during ordinary preprocessing.

Suppose an analyst loads the complete dataset, calculates the mean and standard deviation of every feature, standardizes all observations, splits the standardized dataset into training and testing, and then fits the model.

The test data influenced the standardization parameters. That means information from the evaluation dataset entered model development.

The same problem can occur with missing-value imputation, feature selection, dimensionality reduction, target encoding, and other transformations that learn parameters from data.

The appropriate sequence is:

Scikit-learn recommends pipelines because they help keep learned preprocessing within the appropriate training partitions. This is particularly important during cross-validation.

Cross-Validation Does Not Automatically Prevent Leakage

A team might report, “Our model was validated using five-fold cross-validation.” That sounds reassuring, but cross-validation is not automatically leakage-proof.

Suppose feature selection is performed on the entire dataset before cross-validation begins. The feature-selection process has already used information from observations that will later serve as validation records. The subsequent cross-validation may produce optimistic results.

The same problem can arise if target encoding is calculated using outcomes from all observations. The correct approach is to fit learned preprocessing and feature-selection steps separately within each training fold.

This is why validation architecture matters more than merely choosing a larger number of folds. A sophisticated evaluation procedure can still be invalid if data preparation crosses the evaluation boundary.

A SQL Example: Building Point-in-Time Features

Consider a customer churn model. The business generates predictions at specific snapshot timestamps. We want to calculate how many customer events occurred during the preceding 30 days while ensuring those events were available to the prediction system at the snapshot time.

SELECT
    s.customer_id,
    s.snapshot_at,
    COUNT(e.event_id) AS events_prior_30d
FROM scoring_snapshots AS s
LEFT JOIN customer_events AS e
    ON e.customer_id = s.customer_id
   AND e.event_time >= s.snapshot_at - INTERVAL '30 days'
   AND e.event_time < s.snapshot_at
   AND e.available_at <= s.snapshot_at
GROUP BY
    s.customer_id,
    s.snapshot_at;

The important conditions are e.event_time < s.snapshot_at and e.available_at <= s.snapshot_at. The first condition excludes future events. The second excludes events that occurred earlier but were not yet available to the system.

This is an illustrative feature-building query. A production implementation would also consider source-system corrections, late-arriving data, historical feature definitions, and whether stored timestamps accurately represent availability.

The principle is straightforward: a feature should represent what the system could have known when the prediction was made. Point-in-time correctness also depends on trustworthy pipelines and definitions; see our guide to investigating why dashboard numbers do not match.

Data Leakage, Overfitting, and Data Drift Are Different Problems

These terms are often discussed together, but they should not be treated as synonyms.

Data Leakage

Information unavailable for the intended prediction enters training or evaluation. Example: using a cancellation request submitted after the churn prediction date.

Overfitting

A model learns patterns specific to its training data that do not generalize well to new observations. Example: a highly complex model fits historical noise rather than stable relationships.

Data Drift

The statistical properties of incoming data change over time. Example: customer behavior changes after a major pricing update. Related changes in the relationship between features and outcomes are often discussed as concept drift.

A model can suffer from more than one problem simultaneously. An overfit model may also have been evaluated with leakage, and even a correctly validated model can later deteriorate because its deployment environment changes.

Google’s Machine Learning Crash Course describes overfitting as strong performance on training data that does not generalize to unseen examples. Data leakage creates a different concern: the evaluation itself may have received information that invalidates the performance estimate.

The distinction matters because the remedies differ.

A Practical Diagnostic Table

These symptoms are diagnostic clues, not definitive proof. A high AUC does not automatically mean leakage exists. Likewise, a decline in production performance does not establish that leakage was the cause. The investigation should identify which explanation is supported by evidence.

A Model Validation Audit Before Deployment

Before approving a predictive model, a business should be able to answer seven questions.

1. What exactly is being predicted?

Define the target outcome and prediction horizon. Example: churn within 30 days.

2. When is the prediction made?

Specify the operational prediction timestamp. Without this boundary, feature availability cannot be assessed reliably.

3. Were all features available at that moment?

Check event time and availability time. Audit features that may be consequences of the target.

4. Does the validation split match deployment?

Determine whether predictions concern new customers, existing customers, future time periods, new locations, or new organizations. Choose the evaluation design accordingly.

5. Was preprocessing fitted only on training data?

Review scaling, imputation, feature selection, encoding, resampling, and hyperparameter tuning.

6. Was the final test set protected?

A test set should not become another validation set through repeated experimentation. If model developers repeatedly inspect test results and modify the model in response, the test set gradually becomes part of model selection. A genuinely independent final evaluation may then be required.

7. Do the reported metrics represent the business decision?

AUC, accuracy, precision, recall, calibration, and expected decision cost answer different questions. A technically valid evaluation can still be commercially unhelpful if it measures the wrong objective.

Better Validation Can Produce a Lower Score—and a Better Decision

Imagine a churn model produces the following results. These numbers are hypothetical.

The decline does not prove that the original result was inflated by leakage. Differences in evaluation populations, time periods, and task difficulty can also affect performance.

But the pattern warrants investigation. If the production task involves future customers and future outcomes, the stricter evaluations may provide more relevant evidence.

A model with an honestly estimated AUC of 0.72 can be more useful than one advertised with an unreliable AUC of 0.94. A credible performance estimate helps organizations choose intervention thresholds, estimate operational workload, evaluate costs, allocate staff, and decide whether deployment is justified. An inflated estimate can distort all of those decisions.

Model Accuracy Is Not the Same as Business Value

Suppose a churn model correctly identifies customers at elevated risk. That does not automatically mean a retention campaign will create incremental value.

Some customers would have stayed without an intervention. Others may leave regardless of what the business does. Some retention offers may be expensive.

The model estimates an outcome under the observed data-generating process. The business decision asks whether acting on the prediction changes the outcome enough to justify the cost.

This is the distinction between prediction and causal intervention effectiveness. A model can predict churn accurately without identifying which customers can be persuaded to stay.

Estimating the incremental impact of a retention offer requires an appropriate causal design, such as a randomized experiment or another credible identification strategy. Similarly, feature importance does not prove that changing a feature will change the outcome. A predictor can be highly informative without being a causal driver.

Reliable machine learning therefore requires two separate questions: “Can we predict the outcome?” and “Will acting on this prediction improve the business result?” For a broader perspective, see our article on predictive analytics and business decision-making.

What Should a Model Validation Report Include?

A useful validation report should provide more than one performance number. At minimum, document:

• Prediction task: target, horizon, population, and decision timing.
• Data boundaries: training period, validation period, test period, and grouping rules.
• Feature availability: evidence that each feature was available at the relevant prediction timestamp.
• Preprocessing: which transformations were learned, when, and from which data.
• Performance: relevant metrics with uncertainty where appropriate.
• Subgroup analysis: performance across important customer groups, locations, or operational contexts.
• Calibration: whether predicted probabilities correspond reasonably to observed event frequencies when probability estimates are used for decisions.
• Deployment assumptions: how the production population compares with the evaluation population.
• Monitoring: how future changes in performance, data quality, and input distributions will be detected.

This documentation makes the evaluation auditable. It also helps future analysts understand whether the original validation remains relevant after the business changes.

The Better Question Before Deployment

Machine learning teams often ask: “Which algorithm produces the highest score?”

A better starting question is:

That question should come before model selection.

This sequence does not guarantee a successful model. But it makes the evidence supporting deployment substantially more defensible.

The objective is not to maximize the score displayed in a notebook. It is to build a model whose reported performance remains meaningful when the model encounters the real world.