"We just merge to main and it deploys" is a fine workflow for a REST API that returns JSON. It is dangerous for a system where a one-line change to a feature engineering function can silently degrade predictions for millions of users while every automated check stays green.

Traditional CI/CD was built for deterministic software: write code, tests pass or fail, ship a binary. Machine learning breaks those assumptions. The build artifact is a trained model that depends on terabytes of data you cannot store in git. The tests need GPU hours and statistical reasoning, not simple assertions. And a model scoring 0.94 AUC on your eval set can perform worse in production than the 0.91 model it replaces, because the eval set drifted three weeks ago and nobody noticed. What follows is a working ML CI/CD pipeline - the YAML, the Python evaluation gates, and the rollback strategies - rather than a diagram of neat boxes.

Why traditional CI/CD falls apart for ML

Data dependencies are the real source code

In traditional software, the code is the single source of truth. In ML, the training data matters as much, and it changes independently of the code. A retrain pipeline can run nightly with code untouched for three months, then drop precision 12% one Tuesday because an upstream team changed how a categorical field was encoded - and a CI system that only tracks code changes sees nothing to test. The pipeline has to treat data as a first-class versioned artifact, not an external dependency assumed to be stable.

Long training times kill fast feedback

A typical software CI run takes 5 to 15 minutes; training a production model can take hours or days. You cannot block a pull request on a full training run, but skipping training in CI ships untested changes to the most critical logic. That tension between fast feedback and thorough validation is the central design problem of ML CI/CD.

Non-determinism is the norm

Run the same training code twice on the same data and the weights differ - random initialization, data shuffling, GPU floating-point non-determinism, and distributed training all add variance. A test that asserts accuracy == 0.943 fails randomly. Evaluation needs statistical thresholds, not exact comparisons.

The artifact is not the code

Software CI builds an image or binary from the commit. In ML, the deployed artifact (the model) is produced by running code against data and can be gigabytes in size. Storage, versioning, and promotion of that artifact are first-class operations, not afterthoughts.

A six-stage pipeline

Not every team needs all six on day one, but this is the architecture worth building toward:

1. Data validation - verify the training data before burning GPU hours
2. Training - reproducible, tracked model training
3. Evaluation - automated metric checks against baselines
4. Model validation - behavioral tests, bias checks, latency profiling
5. Registry promotion - versioned model artifact in a registry
6. Canary deployment - gradual rollout with automated rollback

Stage 1: Data validation

This is the stage most teams skip, and it prevents a large share of production incidents. Before training anything, confirm the data is sane.

"""data_validation.py — pre-training data quality gate."""
import pandas as pd, numpy as np, json, sys
from pathlib import Path
from dataclasses import dataclass, asdict
from typing import Dict

@dataclass
class ValidationResult:
    passed: bool
    checks: Dict[str, bool]
    details: Dict[str, str]

def validate_training_data(data_path, schema_path, baseline_stats_path,
                           max_null_fraction=0.05, max_drift_threshold=0.1):
    df = pd.read_parquet(data_path)
    schema = json.loads(Path(schema_path).read_text())
    baseline = json.loads(Path(baseline_stats_path).read_text())
    checks, details = {}, {}

    missing = set(schema["required_columns"]) - set(df.columns)
    checks["schema_valid"] = not missing
    if missing: details["schema_valid"] = f"Missing columns: {missing}"

    high_nulls = df.isnull().mean()[lambda s: s > max_null_fraction]
    checks["null_rates_ok"] = len(high_nulls) == 0

    ratio = len(df) / baseline["row_count"]   # guard against 10x or 0.1x
    checks["row_count_ok"] = 0.5 < ratio < 2.0

    for col in schema.get("numeric_columns", []):
        if col in df.columns and col in baseline.get("distributions", {}):
            psi = _calculate_psi(baseline["distributions"][col]["bins"],
                                 baseline["distributions"][col]["counts"],
                                 df[col].dropna().values)
            checks[f"drift_{col}"] = psi < max_drift_threshold

    return ValidationResult(all(checks.values()), checks, details)

def _calculate_psi(reference_bins, reference_counts, actual_values):
    """Population Stability Index between reference and actual distributions."""
    act_counts, _ = np.histogram(actual_values, bins=reference_bins)
    ref = np.clip(np.array(reference_counts, float) / sum(reference_counts), 1e-6, None)
    act = np.clip(act_counts.astype(float) / act_counts.sum(), 1e-6, None)
    return float(np.sum((act - ref) * np.log(act / ref)))

if __name__ == "__main__":
    r = validate_training_data(sys.argv[1], "configs/data_schema.json",
                               "configs/baseline_stats.json")
    print(json.dumps(asdict(r), indent=2))
    sys.exit(0 if r.passed else 1)

The Population Stability Index check is the part that earns its keep: a PSI above 0.1 on a critical feature means the distribution shifted versus a known baseline, and training on that without investigation is reckless.

Stage 2: Reproducible training with DVC

You cannot version multi-gigabyte datasets in git. DVC (Data Version Control) tracks data and model files alongside code without bloating the repository, and its pipeline definition gives you dependency tracking for free - change only the model config and DVC skips data preparation; change the data and it reruns everything downstream.

# dvc.yaml
stages:
  prepare:
    cmd: python src/prepare.py --config configs/data_config.yaml
    deps: [src/prepare.py, configs/data_config.yaml, data/raw/]
    params: [configs/data_config.yaml:split_ratio, configs/data_config.yaml:random_seed]
    outs: [data/processed/train.parquet, data/processed/val.parquet, data/processed/test.parquet]
  train:
    cmd: python src/train.py --config configs/model_config.yaml
    deps: [src/train.py, configs/model_config.yaml, data/processed/train.parquet]
    params: [configs/model_config.yaml:learning_rate, configs/model_config.yaml:epochs]
    outs: [models/latest/model.pt]
    metrics: [models/latest/metrics.json]
  evaluate:
    cmd: python src/evaluate.py --model models/latest/model.pt --data data/processed/test.parquet
    deps: [src/evaluate.py, models/latest/model.pt, data/processed/test.parquet]
    metrics: [reports/evaluation.json]

The params section ties hyperparameters to the metrics they produced, giving you an experiment log with no extra tooling.

Stage 3: Automated evaluation gates

The heart of the pipeline: an automated gate that compares the candidate model against the current production model and decides whether to promote it.

"""evaluate_gate.py — promotion gate with hard and relative thresholds."""
import json, sys
from pathlib import Path

def evaluate_promotion(candidate_path, baseline_path, config_path="configs/gate_config.json"):
    candidate = json.loads(Path(candidate_path).read_text())
    baseline = json.loads(Path(baseline_path).read_text())
    cfg = json.loads(Path(config_path).read_text())
    report = {"checks": {}}

    # Hard gates: absolute minimums regardless of baseline
    for metric, threshold in cfg.get("hard_gates", {}).items():
        higher = cfg["metric_directions"][metric] == "higher"
        v = candidate[metric]
        report["checks"][f"hard_{metric}"] = {
            "passed": v >= threshold if higher else v <= threshold, "value": v}

    # Relative gates: must not regress beyond tolerance vs baseline
    for metric, tol in cfg.get("relative_gates", {}).items():
        higher = cfg["metric_directions"][metric] == "higher"
        c, b = candidate[metric], baseline[metric]
        regression = (b - c) / b if higher else (c - b) / b
        report["checks"][f"rel_{metric}"] = {
            "passed": regression < tol, "regression_pct": round(regression * 100, 2)}

    # Latency gate against serving SLA
    if "p99_latency_ms" in candidate:
        report["checks"]["latency"] = {
            "passed": candidate["p99_latency_ms"] <= cfg["max_p99_latency_ms"]}

    ok = all(c["passed"] for c in report["checks"].values())
    report["promoted"] = ok
    return ok, report

if __name__ == "__main__":
    promoted, report = evaluate_promotion(sys.argv[1], sys.argv[2])
    print(json.dumps(report, indent=2))
    sys.exit(0 if promoted else 1)

Hard gates and relative gates do different jobs. A hard gate says the model must clear 0.85 AUC no matter what; a relative gate says it cannot be more than 2% worse than production. You need both - a model can pass every relative gate and still be unacceptable if the baseline itself had degraded.

Stage 4: The full GitHub Actions pipeline

Tying the stages together, with each job gating the next. Training runs on a self-hosted GPU runner (GitHub's standard runners cannot train real models), the canary stage uses an environment: production approval gate, and the canary monitor watches live metrics for 30 minutes before promoting to full traffic.

# .github/workflows/ml-pipeline.yml
on:
  push: { branches: [main], paths: ['src/**', 'configs/**', 'dvc.yaml', 'dvc.lock'] }
  schedule: [{ cron: '0 6 * * 1' }]   # weekly retrain, Monday 06:00 UTC
jobs:
  data-validation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: dvc pull data/processed/
      - run: python src/data_validation.py data/processed/train.parquet
  train:
    needs: data-validation
    runs-on: [self-hosted, gpu]
    outputs: { model_version: ${{ steps.version.outputs.version }} }
    steps:
      - uses: actions/checkout@v4
      - run: dvc pull data/processed/ && dvc repro train
      - id: version
        run: echo "version=v$(date +%Y%m%d)-${GITHUB_SHA::8}" >> $GITHUB_OUTPUT
  evaluate:
    needs: train
    runs-on: ubuntu-latest
    outputs: { promoted: ${{ steps.gate.outputs.promoted }} }
    steps:
      - run: aws s3 cp $MODEL_REGISTRY/production/metrics.json models/baseline/metrics.json
      - run: python src/evaluate.py --model models/latest/model.pt --data data/processed/test.parquet
      - id: gate
        run: |
          python src/evaluate_gate.py reports/evaluation.json models/baseline/metrics.json
          echo "promoted=true" >> $GITHUB_OUTPUT
  register-model:
    needs: [train, evaluate]
    if: needs.evaluate.outputs.promoted == 'true'
    runs-on: ubuntu-latest
    steps:
      - run: aws s3 cp models/latest/model.pt $MODEL_REGISTRY/versions/${VERSION}/model.pt
  canary-deploy:
    needs: [register-model, train]
    runs-on: ubuntu-latest
    environment: production            # manual approval gate
    steps:
      - run: python scripts/deploy.py --version $VERSION --traffic-pct 10
      - run: python scripts/monitor_canary.py --duration-minutes 30 --error-rate-threshold 0.01
      - run: python scripts/deploy.py --version $VERSION --traffic-pct 100

Stage 5: Canary monitoring and rollback

Checking that the new model "works" is not enough; the canary monitor verifies it works at least as well as the current model on live traffic, and rolls back automatically when it does not.

"""monitor_canary.py — watch canary metrics, auto-rollback if degraded."""
import time, requests, sys
from datetime import datetime, timedelta

def check_health(endpoint, err_threshold, lat_threshold, drift_threshold):
    m = requests.get(endpoint, timeout=10).json()
    return {
        "error_rate": {"passed": m["canary"]["error_rate"] <= err_threshold},
        "latency":    {"passed": m["canary"]["latency_p99_ms"] <= lat_threshold},
        "drift":      {"passed": m.get("prediction_drift_score", 0) <= drift_threshold},
    }

def monitor(args):
    end = datetime.utcnow() + timedelta(minutes=args.duration_minutes)
    failures, MAX = 0, 3
    while datetime.utcnow() < end:
        checks = check_health(args.metrics_endpoint, args.error_rate_threshold,
                              args.latency_p99_threshold, args.prediction_drift_threshold)
        if all(c["passed"] for c in checks.values()):
            failures = 0
        else:
            failures += 1
            if failures >= MAX:                     # 3 consecutive = genuine problem
                requests.post(args.rollback_endpoint, json={"action": "rollback"}, timeout=30)
                sys.exit(1)
        time.sleep(60)

The three-consecutive-failures threshold is deliberate. A single blip right after deployment is normal - caches are cold, the model has not warmed up. Three failures across three minutes means something is genuinely wrong.

Rollback strategies that work

Rolling back an ML model is harder than rolling back a code deploy. Think in three levels:

Level	When	Mechanism	Recovery
Traffic routing	Canary degrades	Shift 100% back to previous model	Seconds
Model version	Post-deploy issue found	Re-deploy previous version from registry	Minutes
Full retrain	Data corruption	Retrain from last known-good data + code	Hours

The rule that matters most: keep the previous production model deployed and ready to take traffic. Never tear down the old model before the new one is fully validated - the temptation to delete the old endpoint to save compute is exactly what leaves you stranded at midnight.

CML vs custom vs managed

CML (Continuous Machine Learning by Iterative) adds ML capabilities to GitHub Actions and GitLab CI - posting metrics and plots as PR comments, provisioning cloud GPU runners on demand, integrating with DVC. Low barrier to entry and genuinely useful metric diffs in pull requests, but limited orchestration: no built-in registry or serving, and you still write your own training and deployment logic. Strong for simple, fast-training models; you outgrow it on complex multi-step pipelines.

Custom pipelines (the GitHub Actions approach above) give complete control and no vendor lock-in, at the cost of significant engineering investment - budget two to four engineer-weeks to build a solid one, plus ongoing GPU-runner maintenance.

Managed platforms (SageMaker Pipelines, Vertex AI Pipelines) ship a registry, A/B testing, auto-scaling, and drift detection out of the box, which can save months for small teams already committed to a cloud. The trade is deep lock-in - migrating between them is essentially a rewrite - plus opinionated structure and debugging through abstraction layers.

A reasonable path: start with CML plus a custom GitHub Actions pipeline for data validation, automated evaluation, and metric tracking in PRs. Move to a managed platform when you hit the limits - multi-model pipelines, complex A/B testing, or GPU-runner management becoming a full-time job. Reaching for SageMaker Pipelines to deploy a single XGBoost model usually costs more in SDK-learning time than a custom pipeline would have cost to ship.

Model registry

Once more than one model is in production, a registry is the bridge between the CI pipeline and serving. It should track artifacts, evaluation metrics, lineage (which data version and code commit produced the model), stage (staging/canary/production/archived), and metadata. MLflow includes one; a custom registry built on an S3 bucket with a consistent naming convention works well:

s3://model-registry-prod/models/fraud-detector/
  versions/
    v20260215-a3b4c5d6/{model.pt, metrics.json, metadata.json, requirements.txt}
  production/
    metrics.json      # current production baseline
    version.txt       # "v20260215-a3b4c5d6"  <- serving reads this pointer
  candidates/latest   # latest candidate version string

Promotion is updating version.txt; rollback is pointing it at the previous version. The simplicity is intentional - at 3 AM you do not want to navigate a UI to roll back a model.

A/B testing in production

Canary deployment tells you whether the new model is broken; A/B testing tells you whether it is actually better. A model can clear every offline gate and still underperform in production because real user behavior differs from the test set. The simplest setup is a traffic-splitting proxy that assigns variants deterministically by user-ID hash:

import hashlib
AB = {"control": {"endpoint": "http://model-v1:8080/predict", "pct": 80},
      "treatment": {"endpoint": "http://model-v2:8080/predict", "pct": 20}}

def assign_variant(user_id: str) -> str:
    bucket = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 100
    return "control" if bucket < AB["control"]["pct"] else "treatment"

Deterministic assignment is the critical part: a user must see the same variant for the duration of the experiment, or you cannot measure the effect. Random per-request assignment adds noise that makes the results unreliable.

Patterns that consistently matter

• Pin your data, not just your code. Every artifact in the registry should link to the exact data version (DVC hash, S3 version ID, warehouse snapshot) that produced it. When a model degrades, the first question is always "did the data change?" - without lineage you are guessing.
• Keep a frozen golden test set. Run it alongside the standard test set. If metrics drop on the standard set but hold on the golden set, the data drifted; if both drop, the code has a bug.
• Make every stage idempotent. Re-running a stage after a half-finished failure must not produce a corrupted model or double-counted data - easy to get wrong with stateful, checkpointed training loops.
• Separate the promotion decision from the deployment action. The gate decides worthiness; a separate step (with human approval for critical models) does the push. That split gives you an audit trail and a point of intervention.
• Tune the gates quarterly. Drift thresholds too loose let bad models through; too tight and the team starts overriding the gates. Review them like alert thresholds.

The point of ML CI/CD is not to automate humans out of the loop - it is to put humans in the loop at the right moments, armed with the right information, instead of paging them at 3 AM because nobody checked whether the training data was valid before deploy. Invest in evaluation gates before faster training: a model that trains in ten minutes but ships without validation is more dangerous than one that trains overnight behind rigorous checks.