Model Monitoring: Concept Drift and Data Drift

Learn how to monitor production models. Understand data drift, concept drift, feature performance tracking, and retraining strategies.

Introduction

Imagine you train a model in 2019 to predict flight ticket demand. The model has 98% accuracy. Then, 2020 arrives, and global travel patterns lock down. Suddenly, your model's predictions are completely wrong, causing millions of dollars in losses.

Unlike traditional software code, which is static, machine learning models degrade over time. The real world is dynamic—customer tastes change, inflation shifts prices, and unforeseen events happen.

Model Monitoring is the practice of tracking inputs, outputs, and performance metrics of production models to detect when they start to decay, allowing teams to intervene and retrain models before they cause business failures.

What You Will Learn

  • Why production models fail over time.
  • The difference between Data Drift and Concept Drift.
  • How to measure drift mathematically (PSI, Kolmogorov-Smirnov test).
  • Strategies for model retraining.
  • Logging and alert monitoring workflows.

Why This Topic Matters

Deploying a model is not a one-time event; it is an ongoing cycle. Without automated monitoring, you will only realize your model is failing when customers complain or revenue drops. By tracking drift proactively, you can build self-healing pipelines that automatically flag anomalies, alert engineering teams, and trigger retraining runs.

Prerequisites

Detailed Explanation

When a model is running in production, we categorize its decay into two main types of drift:

Data Drift vs. Concept Drift

graph TD
    A[Model Decay Types]
    A --> B[1. Data Drift <br> Input distribution changes <br> P X changes, but relation P Y|X holds]
    A --> C[2. Concept Drift <br> Target relation changes <br> P Y|X changes, but inputs look same]

1. Data Drift (Covariate Shift)

Data drift occurs when the statistical properties of the input features change over time, even though the relationship between features and target remains constant.

  • Example: An app trains a face recognition model on high-resolution camera photos. Users start uploading low-quality, blurry photos from cheap phones. The input data distribution ($P(X)$) has shifted.

2. Concept Drift

Concept drift occurs when the mathematical relationship between the input features ($X$) and the target labels ($Y$) changes, even if the input features themselves look similar.

  • Example: A real estate model predicts house prices based on size. Suddenly, inflation or a mortgage interest rate hike occurs. A 1,000 sqft house that sold for $100,000 in 2019 now sells for $250,000. The inputs are the same, but the "concept" ($P(Y|X)$) has shifted.

Measuring Drift Mathematically

To detect drift before accuracy drops (which is hard to calculate because labels $Y$ might take months to arrive), we compare the distribution of production data against training baseline data using two statistical metrics:

A. Population Stability Index (PSI)

Measures how much a variable has shifted between two distributions:

  • PSI < 0.1: No significant shift.
  • 0.1 $\le$ PSI < 0.2: Moderate shift (requires monitoring).
  • PSI $\ge$ 0.2: Significant shift (requires retraining).

B. Kolmogorov-Smirnov (KS) Test

A non-parametric statistical test that compares the cumulative distributions of production and baseline features. If the calculated $p$-value is less than $0.05$, we reject the null hypothesis and confirm that data drift has occurred.

Retraining Strategies

Once drift is detected, we can trigger three types of retraining workflows:

  1. Schedule-Based: Retrain the model every week, month, or quarter (e.g., retail models retraining every Sunday night).
  2. Drift-Triggered (Recommended): Automatically trigger the training pipeline (via Airflow or Kubeflow) when PSI or KS tests cross a specific threshold.
  3. Manual Intervention: Alert a data scientist to investigate when drift is detected, allowing them to clean the new data or engineer new features first.

Visual Diagram (Mermaid)

graph TD
    A[Incoming Request Data] --> B[Inference API]
    A --> C[Log Aggregator: Elasticsearch/S3]
    C --> D[Drift Analyzer Engine]
    D -->|Calculate PSI / KS-Test| E{Is PSI > 0.2?}
    E -->|No| F[Keep Serving]
    E -->|Yes| G[Trigger Alert & Airflow Retraining Pipeline]
    
    style E fill:#F59E0B,stroke:#fff,color:#fff
    style G fill:#EF4444,stroke:#fff,color:#fff

Python Code Examples

We will write a python script using NumPy to simulate how a KS-test detects data drift between two distributions.

import numpy as np
from scipy import stats

# 1. Baseline Training Data (normal distribution centered around 0)
baseline_data = np.random.normal(loc=0.0, scale=1.0, size=1000)

# 2. Production Data - Week 1 (no drift, centered around 0)
production_week1 = np.random.normal(loc=0.05, scale=1.0, size=1000)

# 3. Production Data - Week 2 (drifted, centered around 0.5)
production_week2 = np.random.normal(loc=0.5, scale=1.0, size=1000)

# 4. Run KS Test
def check_drift(baseline, production, name):
    # Null hypothesis: Both datasets come from the same distribution
    ks_stat, p_value = stats.ks_2samp(baseline, production)
    
    print(f"KS-Test for {name}:")
    print(f" - Statistic: {round(ks_stat, 4)}")
    print(f" - P-value: {p_value}")
    
    # If p-value is extremely small (< 0.05), drift has occurred
    if p_value < 0.05:
        print(" -> ALERT: Data Drift detected! Retraining recommended.")
    else:
        print(" -> STATUS: Normal. No significant drift.")
    print("-" * 40)

check_drift(baseline_data, production_week1, "Week 1 Data")
check_drift(baseline_data, production_week2, "Week 2 Data")

Industry Use Cases

  • Loan Approval Systems: Monitoring credit bureaus for changes in national average debt-to-income ratios to adjust risk scoring.
  • Ad Click Optimization: Tracking changes in user interests and clicking patterns (e.g., during seasonal holidays like Black Friday).
  • Industrial Sensors: Monitoring vibrations and temperature outputs on manufacturing rigs to detect sensor wear and tear (data drift).

Summary

Models decay because the physical world shifts. While data drift represents changes in input feature distributions (shifting camera resolutions), concept drift alters target mappings (pricing inflation). By logging predictions and tracking population statistics via metrics like PSI or KS tests, MLOps teams can automatically fire alerts and launch retraining pipelines.

Next Topic

Congratulations! You have completed the core technical syllabus. Now, we look at the ultimate culmination: building hands-on projects, preparing for placements, and designing your career roadmap in: Capstone Projects, Interview Prep, and Careers.

← Back to Syllabus