I have personally watched 23 ML models get deployed to production over the past four years. Of those, 14 had to be rolled back, patched, or quietly turned off within the first 90 days. Not because the models were bad. The offline metrics were always great. AUC above 0.92, RMSE beating every baseline, the whole deck of green charts that make stakeholders nod approvingly. The models failed because nobody planned for what happens after model.predict() runs on real data, in real time, with real users whose behavior drifts in ways your training set never anticipated.
This article is the postmortem I keep rewriting. Every production ML failure I have witnessed falls into a handful of patterns, and most of them are completely preventable if you know where to look. I am going to walk you through the seven most common failure modes, share the Python code I actually use to catch them, and give you a checklist that would have saved my teams hundreds of hours of firefighting.
Who this is for: ML engineers and data scientists who have trained models that work in notebooks but struggle in production. If you have ever stared at a dashboard wondering why your model's accuracy dropped 15% overnight with no code changes, keep reading.
The Uncomfortable Truth About ML in Production
There is a widely cited stat from Gartner that 85% of ML projects fail to deliver business value. In my experience, the number is not that dramatic, but the reasons are consistent. The model itself is rarely the problem. The infrastructure around the model is.
Think about it. You spend weeks or months on feature engineering, model selection, hyperparameter tuning, cross-validation. Then deployment is a one-afternoon task where someone wraps the model in a Flask endpoint and calls it done. The ratio of effort is completely inverted from where the actual risk lives.
Production ML failures are insidious because they are often silent. A web application crashes and you get a 500 error. A model degrades and you get slightly worse predictions that nobody notices until revenue dips three weeks later and the business team starts asking uncomfortable questions.
The 7 Most Common ML Production Failure Modes
After years of debugging these situations, I have catalogued them into seven categories. Some are well-known. Others are surprisingly under-discussed. All of them have bitten me personally.
1. Training-Serving Skew
This is the big one. Training-serving skew is when the data your model sees at inference time differs systematically from what it saw during training. It sounds obvious, but it sneaks in through dozens of cracks.
The most common cause I have seen is feature computation differences. During training, you compute features in a batch Spark job or a pandas pipeline. During serving, you compute the same features in a Java microservice or a real-time feature pipeline. Subtle differences in how nulls are handled, how timestamps are rounded, or how categorical variables are encoded create a gap that silently destroys model performance.
I once spent two weeks debugging a recommendation model that worked perfectly in offline evaluation but produced bizarre results in production. The culprit: the training pipeline used pandas.Categorical which sorts categories alphabetically, while the serving code used a dictionary-based encoder that preserved insertion order. The category-to-integer mapping was completely different. The model was seeing scrambled inputs and doing its best with garbage.
"""
Detecting training-serving skew with feature validation.
Compare feature distributions between training data and live serving data.
"""
import numpy as np
from scipy import stats
from dataclasses import dataclass
from typing import Dict, List, Optional
@dataclass
class SkewReport:
feature_name: str
psi: float # Population Stability Index
ks_statistic: float
ks_pvalue: float
alert: bool
def compute_psi(
training_dist: np.ndarray,
serving_dist: np.ndarray,
bins: int = 20
) -> float:
"""
Population Stability Index (PSI) between training and serving distributions.
PSI < 0.1 = no significant shift
PSI 0.1-0.2 = moderate shift, investigate
PSI > 0.2 = significant shift, action required
"""
# Create bins from training distribution
breakpoints = np.linspace(
min(training_dist.min(), serving_dist.min()),
max(training_dist.max(), serving_dist.max()),
bins + 1
)
training_counts = np.histogram(training_dist, bins=breakpoints)[0]
serving_counts = np.histogram(serving_dist, bins=breakpoints)[0]
# Avoid division by zero
training_pct = (training_counts + 1) / (training_counts.sum() + bins)
serving_pct = (serving_counts + 1) / (serving_counts.sum() + bins)
psi = np.sum((serving_pct - training_pct) * np.log(serving_pct / training_pct))
return float(psi)
def detect_skew(
training_features: Dict[str, np.ndarray],
serving_features: Dict[str, np.ndarray],
psi_threshold: float = 0.2,
ks_alpha: float = 0.01
) -> List[SkewReport]:
"""Run skew detection across all features."""
reports = []
for feature_name in training_features:
if feature_name not in serving_features:
reports.append(SkewReport(
feature_name=feature_name,
psi=float('inf'),
ks_statistic=1.0,
ks_pvalue=0.0,
alert=True
))
continue
train_vals = training_features[feature_name]
serve_vals = serving_features[feature_name]
psi = compute_psi(train_vals, serve_vals)
ks_stat, ks_pval = stats.ks_2samp(train_vals, serve_vals)
reports.append(SkewReport(
feature_name=feature_name,
psi=psi,
ks_statistic=float(ks_stat),
ks_pvalue=float(ks_pval),
alert=(psi > psi_threshold or ks_pval < ks_alpha)
))
return reportsThe fix for training-serving skew is conceptually simple and operationally hard: use the same code path for training and serving. Feature stores like Feast or Tecton exist precisely for this reason. If you cannot adopt a feature store, at minimum log your serving-time features and periodically compare them against your training set distributions.
2. Data Drift That Nobody Is Watching
Data drift is when the statistical properties of your input data change over time. It is different from training-serving skew because skew is about your pipeline having bugs, while drift is about the world changing.
I worked on a fraud detection model at a fintech company that had exceptional performance for the first six months. Then COVID happened. Consumer spending patterns completely shifted. People who never shopped online were suddenly making dozens of e-commerce transactions. The model flagged thousands of legitimate purchases as fraudulent because its training data had learned that a burst of online transactions from a traditionally in-store customer was suspicious.
Nobody had monitoring in place to catch the drift. By the time the customer support team escalated the complaint volume, the model had been producing bad decisions for three weeks.
"""
Data drift monitoring using Evidently AI.
Set this up as a scheduled job that runs daily against your production data.
"""
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
from evidently.metrics import (
DataDriftTable,
DatasetDriftMetric,
ColumnDriftMetric,
)
import pandas as pd
from datetime import datetime, timedelta
def run_daily_drift_report(
reference_data: pd.DataFrame,
current_data: pd.DataFrame,
numerical_features: list,
categorical_features: list,
output_path: str = "/var/log/ml-monitoring/drift/"
) -> dict:
"""
Generate drift report comparing reference (training) data
against the latest production data window.
Returns dict with drift summary and saves HTML report.
"""
report = Report(metrics=[
DatasetDriftMetric(),
DataDriftTable(
num_stattest="ks", # Kolmogorov-Smirnov for numerical
cat_stattest="chisquare", # Chi-square for categorical
num_stattest_threshold=0.01,
cat_stattest_threshold=0.01,
),
])
report.run(
reference_data=reference_data,
current_data=current_data,
column_mapping={
"numerical_features": numerical_features,
"categorical_features": categorical_features,
}
)
# Save HTML report for manual inspection
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
report.save_html(f"{output_path}/drift_report_{timestamp}.html")
# Extract machine-readable results
results = report.as_dict()
dataset_drift = results["metrics"][0]["result"]["dataset_drift"]
drift_share = results["metrics"][0]["result"]["drift_share"]
return {
"dataset_drift_detected": dataset_drift,
"share_of_drifted_features": drift_share,
"report_path": f"{output_path}/drift_report_{timestamp}.html",
"timestamp": timestamp,
}
# Example: wire this into your monitoring pipeline
def check_and_alert(reference_df, production_df, features_num, features_cat):
result = run_daily_drift_report(
reference_data=reference_df,
current_data=production_df,
numerical_features=features_num,
categorical_features=features_cat,
)
if result["dataset_drift_detected"]:
# Send PagerDuty/Slack alert
print(f"DRIFT ALERT: {result['share_of_drifted_features']:.1%} "
f"of features drifted. Report: {result['report_path']}")
# In production, trigger your alerting system here
return resultThe key lesson: monitor data drift from day one. Do not wait for someone to notice performance degradation. By the time humans notice, the damage is already done.
3. Feature Store Inconsistencies
Even teams that use a feature store can get burned. The most common issue I see is time-travel violations: using features at training time that would not have been available at prediction time. This is a form of data leakage that creates artificially inflated offline metrics.
A concrete example: your feature store computes a "rolling 7-day average transaction amount" for each user. At training time, this feature is computed correctly using a point-in-time join. But in production, the feature store serves the most recent value, which might include data from after the event you are trying to predict. Or worse, the batch pipeline that updates the feature store runs at 2 AM, so predictions made at 8 PM are using features that are 18 hours stale.
Staleness is the silent killer of feature stores. I always validate feature freshness as part of the serving pipeline:
"""
Feature freshness and consistency validation.
Run these checks before every prediction in production.
"""
from datetime import datetime, timedelta
from typing import Dict, Any, Optional
import logging
logger = logging.getLogger("feature_validation")
class FeatureValidator:
"""Validates features before they reach the model."""
def __init__(self, feature_specs: Dict[str, dict]):
"""
feature_specs: {
"user_avg_txn_7d": {
"dtype": "float",
"min": 0.0,
"max": 100000.0,
"max_staleness_minutes": 60,
"nullable": False,
},
...
}
"""
self.specs = feature_specs
self.violation_counts: Dict[str, int] = {}
def validate(
self,
features: Dict[str, Any],
feature_timestamps: Dict[str, datetime],
) -> tuple[bool, list[str]]:
"""
Returns (is_valid, list_of_violations).
In production, you decide whether to block or fallback on violations.
"""
violations = []
now = datetime.utcnow()
for name, spec in self.specs.items():
# Check presence
if name not in features:
violations.append(f"MISSING: {name}")
continue
value = features[name]
# Null check
if value is None and not spec.get("nullable", True):
violations.append(f"NULL: {name} is null but not nullable")
continue
if value is not None:
# Type check
expected = spec.get("dtype")
if expected == "float" and not isinstance(value, (int, float)):
violations.append(
f"TYPE: {name} expected float, got {type(value).__name__}"
)
# Range check
if isinstance(value, (int, float)):
if "min" in spec and value < spec["min"]:
violations.append(
f"RANGE: {name}={value} below min={spec['min']}"
)
if "max" in spec and value > spec["max"]:
violations.append(
f"RANGE: {name}={value} above max={spec['max']}"
)
# Staleness check
if name in feature_timestamps:
age_minutes = (now - feature_timestamps[name]).total_seconds() / 60
max_stale = spec.get("max_staleness_minutes", 1440)
if age_minutes > max_stale:
violations.append(
f"STALE: {name} is {age_minutes:.0f}min old "
f"(max {max_stale}min)"
)
if violations:
for v in violations:
logger.warning(f"Feature violation: {v}")
key = v.split(":")[0]
self.violation_counts[key] = self.violation_counts.get(key, 0) + 1
return (len(violations) == 0, violations)4. Latency Surprises That Kill User Experience
Your model runs inference in 12ms on your MacBook Pro. In production, behind a load balancer, with feature fetches, pre-processing, post-processing, and logging, your p99 latency is 2.4 seconds. I have seen this exact scenario play out at least five times.
The breakdown usually looks like this:
| Component | Expected | Actual (p99) | Why |
|---|---|---|---|
| Feature fetch | 5ms | 180ms | Redis cache miss, falls back to database |
| Pre-processing | 2ms | 85ms | Tokenizer loads lazily on first request |
| Model inference | 12ms | 450ms | CPU contention, no GPU in staging |
| Post-processing | 1ms | 35ms | Business rules engine queries external service |
| Logging/telemetry | 0ms | 120ms | Synchronous logging to remote collector |
| Total | 20ms | 870ms | And this is an optimistic day |
The biggest offenders are almost never the model itself. It is the feature retrieval, the data serialization overhead, and the synchronous I/O that nobody profiled. I once debugged a model serving endpoint where 60% of the latency came from a single json.dumps() call serializing a large numpy array that should have been converted to a Python list first.
Profile your entire serving path, not just the model.predict() call. Use distributed tracing (Jaeger, Zipkin) so you can see exactly where time is spent per request.
5. Silent Model Degradation
This is the failure mode that terrifies me the most. Your model is live, serving predictions, returning 200 OK on every request. But the quality of those predictions is slowly getting worse and nobody knows because there is no real-time performance monitoring.
The root cause is usually straightforward: ground truth labels arrive with a delay. If you are predicting whether a user will churn in 30 days, you need to wait 30 days to know if you were right. During that window, the model could be completely wrong and you would have no idea.
The solution is proxy metrics. You cannot always measure accuracy in real time, but you can measure things that correlate with accuracy:
- Prediction distribution shifts: If your model suddenly starts predicting "high risk" for 40% of users instead of the usual 12%, something changed.
- Confidence score distributions: A model that is getting uncertain (more predictions near 0.5 for binary classification) is often a model that is struggling with unfamiliar data.
- Feature contribution stability: If SHAP values for a top feature suddenly invert, the model is interpreting that feature differently than expected.
- Downstream business metrics: Click-through rates, conversion rates, approval rates. These react faster than ML metrics.
"""
Real-time model performance monitoring with Prometheus.
Expose these metrics from your serving endpoint.
"""
from prometheus_client import (
Histogram, Counter, Gauge, Summary, start_http_server
)
import numpy as np
from collections import deque
from threading import Lock
import time
# -- Prometheus metrics --
PREDICTION_LATENCY = Histogram(
"ml_prediction_latency_seconds",
"Time spent generating a prediction",
buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5]
)
PREDICTION_VALUE = Histogram(
"ml_prediction_value",
"Distribution of model output scores",
buckets=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
)
PREDICTION_COUNT = Counter(
"ml_predictions_total",
"Total predictions made",
["model_version", "outcome"]
)
FEATURE_MISSING_COUNT = Counter(
"ml_feature_missing_total",
"Count of missing features at serving time",
["feature_name"]
)
MODEL_CONFIDENCE = Summary(
"ml_model_confidence",
"Summary of prediction confidence (distance from 0.5)"
)
DRIFT_SCORE = Gauge(
"ml_drift_score",
"Current data drift score (PSI)",
["feature_name"]
)
class ModelMonitor:
"""Wraps model inference with comprehensive monitoring."""
def __init__(self, model, model_version: str, window_size: int = 1000):
self.model = model
self.model_version = model_version
self.recent_predictions = deque(maxlen=window_size)
self.lock = Lock()
def predict_and_monitor(self, features: dict) -> float:
# Track missing features
for fname, fval in features.items():
if fval is None:
FEATURE_MISSING_COUNT.labels(feature_name=fname).inc()
# Time the prediction
start = time.perf_counter()
prediction = float(self.model.predict(features))
elapsed = time.perf_counter() - start
# Record metrics
PREDICTION_LATENCY.observe(elapsed)
PREDICTION_VALUE.observe(prediction)
confidence = abs(prediction - 0.5)
MODEL_CONFIDENCE.observe(confidence)
outcome = "positive" if prediction >= 0.5 else "negative"
PREDICTION_COUNT.labels(
model_version=self.model_version,
outcome=outcome
).inc()
# Track for windowed statistics
with self.lock:
self.recent_predictions.append(prediction)
return prediction
def get_prediction_stats(self) -> dict:
"""Current window statistics for dashboarding."""
with self.lock:
preds = np.array(self.recent_predictions)
if len(preds) == 0:
return {}
return {
"mean": float(np.mean(preds)),
"std": float(np.std(preds)),
"median": float(np.median(preds)),
"pct_positive": float((preds >= 0.5).mean()),
"pct_low_confidence": float(
((preds > 0.4) & (preds < 0.6)).mean()
),
"window_size": len(preds),
}Set up Grafana dashboards that show these metrics in real time. I create alert rules for three conditions: prediction distribution shift (mean deviates more than 2 standard deviations from the training baseline), confidence collapse (more than 30% of predictions in the 0.4-0.6 range), and latency spikes (p99 exceeds SLA).
6. The "It Works on My Machine" Pipeline Problem
ML pipelines have a reproducibility problem that is worse than traditional software. Your model depends on specific versions of Python, NumPy, scikit-learn, your custom feature library, the data snapshot it was trained on, and the random seed. Change any one of these and you get a different model.
I have seen a production incident caused by a minor scikit-learn version bump (0.24 to 1.0) that changed the default behavior of StandardScaler when encountering constant features. The model was re-trained in CI with the new version, deployed, and immediately started producing NaN for 3% of predictions. Nobody caught it because the CI pipeline did not compare the new model's prediction distribution against the previous version.
Use MLflow (or a similar experiment tracker) to pin every artifact and dependency:
"""
MLflow model logging with full environment capture.
This ensures you can reproduce any model exactly.
"""
import mlflow
import mlflow.sklearn
from mlflow.models.signature import infer_signature
import json
from datetime import datetime
def train_and_log_model(
X_train, y_train, X_val, y_val,
model, model_params: dict,
feature_names: list,
training_data_hash: str,
):
"""
Train a model and log everything needed for reproducibility
and production validation.
"""
with mlflow.start_run() as run:
# Log parameters
mlflow.log_params(model_params)
mlflow.log_param("training_data_hash", training_data_hash)
mlflow.log_param("n_training_samples", len(X_train))
mlflow.log_param("n_features", len(feature_names))
# Train
model.fit(X_train, y_train)
# Evaluate
train_score = model.score(X_train, y_train)
val_score = model.score(X_val, y_val)
y_pred = model.predict(X_val)
mlflow.log_metric("train_accuracy", train_score)
mlflow.log_metric("val_accuracy", val_score)
# Log prediction distribution (critical for monitoring)
if hasattr(model, "predict_proba"):
y_proba = model.predict_proba(X_val)[:, 1]
mlflow.log_metric("pred_mean", float(y_proba.mean()))
mlflow.log_metric("pred_std", float(y_proba.std()))
mlflow.log_metric("pred_median", float(
__import__("numpy").median(y_proba)
))
# Log feature importance baseline
if hasattr(model, "feature_importances_"):
importance = dict(zip(feature_names, model.feature_importances_))
mlflow.log_dict(importance, "feature_importance.json")
# Log model with signature and input example
signature = infer_signature(X_val[:5], y_pred[:5])
mlflow.sklearn.log_model(
model,
"model",
signature=signature,
input_example=X_val[:1],
pip_requirements=[
"scikit-learn==1.3.2",
"numpy==1.26.2",
"pandas==2.1.4",
],
)
# Log training data statistics for drift detection baseline
training_stats = {}
for i, fname in enumerate(feature_names):
col = X_train[:, i] if hasattr(X_train, "__array__") else X_train[fname]
training_stats[fname] = {
"mean": float(col.mean()),
"std": float(col.std()),
"min": float(col.min()),
"max": float(col.max()),
"q25": float(__import__("numpy").percentile(col, 25)),
"q75": float(__import__("numpy").percentile(col, 75)),
}
mlflow.log_dict(training_stats, "training_feature_stats.json")
print(f"Run ID: {run.info.run_id}")
print(f"Train acc: {train_score:.4f}, Val acc: {val_score:.4f}")
return run.info.run_idThe training feature statistics artifact is especially important. It gives your monitoring system a baseline to compare against without needing access to the original training data.
7. Missing Fallback Strategies
What happens when your model is down? What happens when the feature store is unreachable? What happens when the model returns a prediction that is obviously wrong (a price prediction of negative $4 billion)?
Most teams I have worked with have no answer to these questions. The serving endpoint either returns a 500 error or, worse, returns whatever garbage the model produced.
Every production model needs three things:
- A fallback model or heuristic. When the primary model fails, serve predictions from a simpler model (like logistic regression) or a rule-based system. It will not be as good, but it will not be zero.
- Output validation. The model's prediction must pass sanity checks before being returned. If you are predicting prices, they must be positive. If you are predicting probabilities, they must be between 0 and 1. If you are classifying into 5 categories, the output must be one of those 5.
- Circuit breakers. If the model's error rate exceeds a threshold, automatically switch to the fallback. Do not wait for a human to notice and flip a switch.
"""
Production model serving with fallback and circuit breaker.
"""
import time
from typing import Optional, Any
from enum import Enum
class CircuitState(Enum):
CLOSED = "closed" # Normal operation
OPEN = "open" # Fallback mode
HALF_OPEN = "half_open" # Testing if primary recovered
class ResilientPredictor:
"""
Wraps a primary model with fallback, output validation,
and circuit breaker pattern.
"""
def __init__(
self,
primary_model,
fallback_model,
error_threshold: float = 0.1, # 10% error rate triggers circuit
window_seconds: int = 300, # 5-minute rolling window
recovery_timeout: int = 60, # Try primary again after 60s
):
self.primary = primary_model
self.fallback = fallback_model
self.error_threshold = error_threshold
self.window_seconds = window_seconds
self.recovery_timeout = recovery_timeout
self.state = CircuitState.CLOSED
self.errors: list = []
self.successes: list = []
self.last_failure_time: float = 0
def _clean_window(self):
cutoff = time.time() - self.window_seconds
self.errors = [t for t in self.errors if t > cutoff]
self.successes = [t for t in self.successes if t > cutoff]
def _error_rate(self) -> float:
self._clean_window()
total = len(self.errors) + len(self.successes)
if total < 10: # Not enough data to judge
return 0.0
return len(self.errors) / total
def _validate_output(self, prediction: Any, context: dict) -> bool:
"""Override this with your domain-specific validation."""
if prediction is None:
return False
if isinstance(prediction, float):
if prediction != prediction: # NaN check
return False
# Example: probability must be in [0, 1]
if context.get("output_type") == "probability":
return 0.0 <= prediction <= 1.0
return True
def predict(self, features: dict, context: dict = None) -> dict:
context = context or {}
now = time.time()
# Check circuit state
if self.state == CircuitState.OPEN:
if now - self.last_failure_time > self.recovery_timeout:
self.state = CircuitState.HALF_OPEN
else:
return self._fallback_predict(features, "circuit_open")
# Try primary model
try:
prediction = self.primary.predict(features)
if not self._validate_output(prediction, context):
self.errors.append(now)
return self._fallback_predict(
features, f"validation_failed: {prediction}"
)
self.successes.append(now)
# If half-open and success, close circuit
if self.state == CircuitState.HALF_OPEN:
self.state = CircuitState.CLOSED
# Check if error rate warrants opening circuit
if self._error_rate() > self.error_threshold:
self.state = CircuitState.OPEN
self.last_failure_time = now
return {
"prediction": prediction,
"model": "primary",
"confidence": "normal",
}
except Exception as e:
self.errors.append(now)
self.last_failure_time = now
if self._error_rate() > self.error_threshold:
self.state = CircuitState.OPEN
return self._fallback_predict(features, f"exception: {str(e)}")
def _fallback_predict(self, features: dict, reason: str) -> dict:
try:
prediction = self.fallback.predict(features)
return {
"prediction": prediction,
"model": "fallback",
"fallback_reason": reason,
"confidence": "degraded",
}
except Exception:
# Even fallback failed, return safe default
return {
"prediction": None,
"model": "none",
"fallback_reason": f"both_failed: {reason}",
"confidence": "none",
}The confidence field in the response is critical. Downstream consumers need to know when they are getting degraded predictions so they can adjust their behavior. A recommendation system in fallback mode might show trending items instead of personalized picks. A fraud system in fallback mode might route transactions to manual review instead of auto-approving.
The Root Cause: We Treat ML Deployment as a One-Time Event
If I had to boil all seven failure modes down to a single underlying cause, it is this: teams treat model deployment as a one-time event rather than an ongoing operation. You do not deploy a model. You operate a model. The difference in mindset changes everything.
Traditional software deployment is also continuous, but the system either works or it does not. It returns the right response or it throws an error. ML systems have a third state: they return a response that looks correct but is wrong. You get a valid JSON object with a plausible-looking probability, and the only way to know if it is right is to compare it against ground truth that might not arrive for weeks.
This is why MLOps is genuinely harder than DevOps. It is not just CI/CD with a model file. It is CI/CD plus data validation plus model validation plus performance monitoring plus drift detection plus fallback orchestration. Every one of those layers can fail independently.
Data Validation with Great Expectations
One tool I have come to rely on heavily is Great Expectations for data validation. It sits at the boundary between your data pipeline and your model, ensuring that the data flowing in meets the assumptions your model was trained on.
"""
Data validation suite for ML feature pipeline using Great Expectations.
Run this before every training job AND on production data before inference.
"""
import great_expectations as gx
def build_feature_expectations(context):
"""
Define expectations for your ML feature table.
These should mirror your training data's properties.
"""
datasource = context.sources.add_or_update_pandas("feature_store")
# Create expectation suite
suite = context.add_or_update_expectation_suite("ml_features_suite")
validator = context.get_validator(
batch_request=datasource.get_batch_request(),
expectation_suite_name="ml_features_suite",
)
# Schema expectations
validator.expect_table_columns_to_match_ordered_list(
column_list=[
"user_id", "session_count_7d", "avg_txn_amount_30d",
"days_since_last_login", "device_type", "country_code",
"feature_timestamp"
]
)
# Null checks (matching training data assumptions)
validator.expect_column_values_to_not_be_null("user_id")
validator.expect_column_values_to_not_be_null("session_count_7d")
validator.expect_column_values_to_not_be_null("device_type")
# Range checks (based on training distribution)
validator.expect_column_values_to_be_between(
"session_count_7d", min_value=0, max_value=500
)
validator.expect_column_values_to_be_between(
"avg_txn_amount_30d", min_value=0, max_value=50000
)
validator.expect_column_values_to_be_between(
"days_since_last_login", min_value=0, max_value=730
)
# Categorical checks
validator.expect_column_values_to_be_in_set(
"device_type", ["mobile", "desktop", "tablet", "smart_tv"]
)
# Distribution checks (detect drift at the data level)
validator.expect_column_mean_to_be_between(
"session_count_7d", min_value=2.5, max_value=8.0
)
validator.expect_column_stdev_to_be_between(
"avg_txn_amount_30d", min_value=50, max_value=500
)
# Freshness check
validator.expect_column_max_to_be_between(
"feature_timestamp",
min_value="2025-12-18T00:00:00", # Not older than yesterday
parse_strings_as_datetimes=True,
)
validator.save_expectation_suite(discard_failed_expectations=False)
return suiteThe beauty of Great Expectations is that the same validation suite runs against both your training pipeline output and your production data. If production data violates expectations that held during training, you know something has changed and you can halt predictions before bad data reaches the model.
The Production ML Checklist
I keep this checklist in a shared document that every ML project at my company must complete before going live. It is not glamorous, but it has prevented more incidents than any fancy tooling.
Before Deployment
- Feature parity verified: Training and serving feature computation uses the same code path, or differences are documented and tested.
- Latency profiled: End-to-end inference latency measured under realistic load, including feature fetch, pre/post-processing, and logging. P99 meets SLA.
- Output validation implemented: Model outputs are sanity-checked before being returned (range checks, type checks, NaN detection).
- Fallback strategy defined: There is a documented plan for what happens when the model is unavailable or producing bad predictions.
- Model artifact versioned: The exact model file, code version, dependency versions, and training data hash are recorded in MLflow or equivalent.
- Shadow mode tested: The model has run in shadow mode alongside the existing system (or heuristic) for at least one week, and predictions have been compared.
- Rollback plan tested: You have verified that you can revert to the previous model version within 5 minutes.
Monitoring (Day 1)
- Prediction distribution tracked: Histogram of model outputs is recorded and compared against training-time distribution.
- Input data drift monitored: Daily drift detection (PSI or KS test) runs against all features, with alerts for significant shifts.
- Latency dashboards live: p50, p95, p99 latency visible in Grafana, with alerts for SLA breaches.
- Error rate monitored: Prediction failures, validation failures, and fallback invocations are counted and alerted on.
- Feature freshness tracked: Age of each feature at serving time is measured, with alerts for stale features.
Ongoing Operations
- Ground truth feedback loop: There is a pipeline to collect ground truth labels and compute real model accuracy on a regular cadence.
- Retraining trigger defined: Clear criteria for when the model should be retrained (drift threshold, accuracy drop, time-based schedule).
- A/B testing framework ready: New model versions are validated against the incumbent through controlled experiments, not vibes.
- Incident runbook written: On-call engineers have documented procedures for common model incidents (drift alert, latency spike, fallback activated).
- Quarterly model review scheduled: Every 90 days, review model performance, drift trends, and business impact. Decide whether to retrain, rebuild, or retire.
Tools I Actually Use
There is no shortage of MLOps tools. Here are the ones that have earned their place in my stack through actual production use, not conference demos:
| Category | Tool | Why I Use It |
|---|---|---|
| Experiment tracking | MLflow | Self-hosted, no vendor lock-in, handles artifacts well |
| Data validation | Great Expectations | Declarative, integrates with Airflow/Prefect, great docs |
| Drift detection | Evidently | Best visualization, supports both batch and real-time monitoring |
| Metrics/alerting | Prometheus + Grafana | Industry standard, scales well, huge ecosystem of exporters |
| Feature store | Feast | Open source, handles online/offline serving, good community |
| Model serving | BentoML or FastAPI | BentoML for complex models, FastAPI for simple ones |
| CI/CD for ML | GitHub Actions + DVC | DVC for data versioning, GHA for pipeline orchestration |
A note on tool selection: do not adopt all of these at once. Start with monitoring (Prometheus + Evidently) because you cannot fix what you cannot see. Add data validation (Great Expectations) next because bad data is the most common failure mode. Then add experiment tracking (MLflow) to ensure reproducibility. Layer in the rest as your team matures.
Final Thoughts
The gap between a model that works in a notebook and a model that works in production is not a technology gap. It is a mindset gap. The notebook is where science happens. Production is where engineering happens. Both require skill, but they are different skills.
If you take one thing from this article, let it be this: instrument everything from day one. Log your inputs, log your outputs, log your latencies, log your feature distributions. You will not need most of this data most of the time. But on the day your model silently degrades, and that day will come, you will be grateful that the data is there to diagnose the problem in hours instead of weeks.
Every model deployment is a bet that the future will look like the past. Sometimes that bet pays off for years. Sometimes the world changes overnight. The difference between a resilient ML system and a fragile one is not the model architecture. It is whether you built the infrastructure to detect and respond when your assumptions break.
I have made every mistake in this article at least once. Some of them twice. The checklist exists because I got tired of making the same mistakes on different projects. I hope it saves you some of that pain.
Leave a Comment