A model that looked solid last quarter can look lost this quarter. Most leaders blame the algorithm. I start with the data.
The training set is a photograph of a moment. Production data is a live feed. When the inputs no longer match that photograph, the model is not getting worse. The data in front of it moved.
That gap is data drift. You can see it. You can fix it. You cannot manage it by guessing.
What data drift is
Data drift is a change in the distribution of the inputs that reach your model after you train it.
Feature values move. A channel you used to see every day goes quiet. A new segment arrives in volume.
That is a different problem from concept drift. Concept drift is when the mapping from features to the outcome changes. The same features still arrive. They no longer mean what they meant when you trained.
Treat them as different problems. Data drift often points to a pipeline or a market shift. Concept drift often means the process you modeled actually changed.
Do not mix the terms. I use data drift for the input change. I use concept drift when the mapping from features to the outcome changes. If your team uses other names, pick one set and keep it.
How you recognize it
You recognize drift by comparing production data to a frozen baseline. The baseline is the training set, or a later window you already accepted.
If you have no baseline, you are not monitoring drift. You are watching a dashboard and hoping.
1. Watch feature distributions
Start with the features the model uses. For each one, compare the current window to the training set.
For a numeric feature, look at the mean, the median, the tails, and the share of missing values. For a category, look at the share of each value. A new category that did not exist in training is a signal. A category that used to dominate and now almost disappears is a signal.
Plot the two distributions on the same axis. If a stakeholder cannot see the shift in one glance, the chart is not done.
Do this on a schedule. Weekly is a start. Daily is better on a model that scores every hour.
2. Measure population stability
A chart is not a threshold. You need a number you can alert on.
Population Stability Index (PSI) is the common one. You bin a feature the same way in the baseline and in the current window. Then you score how much the mass in each bin moved.
Many credit-risk teams treat PSI below 0.10 as stable. They treat 0.10 to 0.25 as a warning. They treat a value above 0.25 as a material shift. Those cutoffs are a common starting rule in that field. They are not a study result, and they are not a law. Set the alert to the cost of a miss in your own process.
Use PSI on the features that drive the score. Also use it on the score itself. If the score distribution moves while the features look stable, look again. You missed a feature, or the mapping changed.
3. Run a Kolmogorov-Smirnov test
The Kolmogorov-Smirnov test (KS test) compares two samples of the same numeric feature. It finds the largest gap between the two cumulative distributions.
If that gap is large enough, you reject the idea that the two samples come from the same distribution.
The KS test is useful when you want a statistical yes or no. PSI is useful when you want a size you can trend. Use both. Do not pick one and call the job done.
Do not hunt for a p-value that lets you ignore a shift you can already see. The test is a check. The chart is still the evidence.
4. Compare model performance to the training baseline
Drift in features is a warning. Drift in results is the fire.
Hold a performance baseline from the training window, or from a later window you already accepted. Track the same metric in production. Use the metric the business already believes. Accuracy, error, lift, or a calibration plot. Pick one primary metric and keep it.
When performance moves and the features moved, you have a drift story. When performance moves and the features did not, check three causes. You may have concept drift. You may have a label delay. You may have a broken outcome pipeline.
When features move and performance does not, you still have work. The next shift may not be so kind.
5. Watch the boring pipeline
A lot of "drift" is a silent pipeline change.
Watch volume. Watch null rates. Watch schema. If a column type changes, or a join starts dropping rows, the model sees a new world even if the real process did not.
This is not fancy. It is how you stop a code deploy from looking like a market event.
How you solve it
Recognition without an owner is a report that nobody reads. Solve drift with a loop you can run every week.
1. Make monitoring a job, not a project
Put the baseline, the PSI, the KS tests, the performance chart, and the volume checks in one place. Alert a named person. If the alert goes to a shared inbox, it will die there.
Review the alerts on a fixed cadence. A weekly 20-minute review beats a quarterly meeting about why this broke.
Write what you will do at each threshold. Warning means inspect. Material shift means stop and decide. Do not invent the plan in Slack after the alert.
2. Put a data contract on the inputs
A data contract is a written agreement on the shape and meaning of the data that feeds the model.
It names the fields. It names the types. It names the allowed values. It names freshness. It names who owns a break.
When a producer changes a field, the contract fails before the model quietly absorbs the change. That is the point. Drift that comes from an upstream surprise should fail in the pipeline, not in the score.
Keep the contract small. Cover the features the model uses and the keys you join on. A 40-page contract nobody updates is not a contract. It is a wiki.
3. Decide if you retrain
Retraining is the right move when the world actually changed and you have new labeled data that reflects that world.
It is the wrong move when the drift is a broken pipeline, a new null pattern, or a segment you never meant to score.
Before you retrain, answer three questions.
- Did the real process change, or did the pipeline change?
- Do you have labels for the new period that you trust?
- Will the new training set still represent the cases you need to score next month?
If the pipeline broke, fix the pipeline. Then score again on clean data. Retraining on a broken pipeline teaches the model the break.
If the process changed and the labels are good, retrain. Compare the new model to the current one on a holdout from the new period. Promote it only if it wins on the metric you already chose.
4. Do not score what you never trained
If a new segment appears, or a category the model never saw, do not pretend the score is valid.
Route those cases out. Use a rule. Use a human. Use a different model. The honest answer is better than a confident number on data the training set never contained.
5. Close the loop with the business
Tell the owner of the decision what moved. Use the chart, the PSI, and the performance line. Do not start with the method names.
If a campaign, a price change, or a new product caused the shift, that is not a model failure. It is a change you now have to include. If a vendor changed its codes, that is an operations failure. Treat it that way.
A simple operating rhythm
Here is the rhythm I use.
- Freeze a training baseline when you ship.
- Compute feature PSI, a KS test on key numeric features, and the score PSI on a fixed window.
- Compare production performance to the training baseline on the same metric.
- Alert on the thresholds you wrote down.
- Split pipeline breaks from real-world change.
- Fix the pipeline, or retrain, or stop scoring the new cases.
- Write what you did, then reset the baseline if you accepted the new world.
That is the whole system. You do not need a new platform to start. You need a baseline, a few tests, an owner, and the will to stop a bad score from shipping.
What I would do this week
If you have a model in production and no drift watch, do not start with a tool contest.
Pick the 5 features that move the score the most. Add the score. Compare this week to the training set. Compute PSI. Look at the distributions. Look at performance against the baseline.
If nothing moved, you now have a baseline and a habit. If something moved, you have a real problem instead of a vague worry.
Data drift is not a research topic. It is operations. Treat it that way, and the model stays useful after the demo.
Pro tip
Use the above to build an agent that checks for drift.