Home > Docs > Best Practices > Model Monitoring & Drift Detection

📡 Model Monitoring & Drift Detection¶

Last Updated: 2026-04-27 | Version: 1.0.0 | Wave 2 Sub-Topic of: MLOps for Fabric Production

📑 Table of Contents¶

• 🎯 Overview
• 🧭 Drift Type Taxonomy
• 📐 Statistical Methods
• 🏗️ Reference Architecture
• ⚙️ Implementation in Fabric
• 🔍 KQL Drift Library
• 📉 Performance Drift Patterns
• 🌀 Concept Drift Detection
• 🚨 Alert Wiring
• 🔁 Retraining Trigger Patterns
• 🛡️ False Positive Mitigation
• 🎰 Casino Implementation
• 🏛️ Federal Implementation
• 🚫 Anti-Patterns
• 📋 Implementation Checklist
• 📚 References

🎯 Overview¶

Models decay. The world they predict on changes faster than the world they were trained on. Without drift detection, you only learn that a model has gone bad after it's hurt the business — missed revenue, compliance failure, customer harm. Drift detection is the "smoke alarm" of production ML: it fires before the fire spreads.

This is the deep-dive companion to the MLOps for Fabric Production anchor. It covers four drift categories, five statistical methods, KQL implementations against Workspace Monitoring + Eventhouse, alert wiring, retraining triggers, and false-positive mitigation.

Why Four Drift Types — Not Just "Drift"¶

Most teams say "drift" and mean "the input data looks different." That's only one of four failure modes:

Each type has a different signal, test, and remediation. A single "drift detected" alert is useless; on-call needs to know which kind fired — retrain, investigate input pipeline, or escalate to product.

Where This Lives in Fabric¶

Three native surfaces: Workspace Monitoring (built-in endpoint metrics; KQL-queryable), Eventhouse (high-cardinality storage for prediction logs, feature snapshots, drift metrics), Real-Time Dashboards (trend visualization). Add a Lakehouse-stored reference distribution (the training-time feature snapshot) and you have full drift coverage.

📝 Scope: This doc covers detection. For remediation, see MLOps for Fabric Production § Retraining Triggers and § Canary, A/B, and Champion-Challenger.

🧭 Drift Type Taxonomy¶

Decision Tree: Which Drift Fired?¶

Performance dropped?
├── Yes → Check input PSI
│         ├── PSI high (> 0.2) → Data drift caused it; investigate upstream
│         ├── PSI normal (< 0.1) → Concept drift; relationship has changed (retrain, possibly rebuild)
│         └── PSI medium (0.1–0.2) → Mixed; investigate both
│
└── No (performance stable) → Is prediction distribution shifted?
          ├── Yes + inputs shifted → Data drift, but model is robust (monitor; lower urgency)
          ├── Yes + inputs stable → Suspicious; likely instrumentation bug, not real drift
          └── No → Healthy

Label Drift — A Special Case¶

Label drift (P(y) shift) is detectable only when ground truth arrives. In casino fraud detection, labels arrive after investigators close the case — sometimes weeks later. In USDA crop yield, labels arrive at harvest — months later. Label drift detection must operate on the delayed label stream and compare to the prior-period label distribution. See § Performance Drift Patterns — Proxy Metrics for what to do while waiting.

📐 Statistical Methods¶

Five methods cover almost every drift detection scenario you'll meet. Each is implementable in KQL or PySpark and tested against the reference distribution stored in the Lakehouse.

PSI — Population Stability Index¶

Use when: Feature is binned (continuous discretized into deciles, or categorical). Most common drift metric.

Formula: PSI = Σ (actual% - expected%) × ln(actual% / expected%)

Thresholds (industry-standard):

KQL:

// PSI between current 7-day window and reference distribution
let reference = ReferenceDistribution
    | where ModelName == "casino-player-churn-lightgbm" and FeatureName == "avg_daily_spend"
    | project Bucket, ExpectedPct;
let current = PredictionLog
    | where ModelName == "casino-player-churn-lightgbm"
    | where TimeGenerated > ago(7d)
    | summarize ActualCount = count() by Bucket = bin(avg_daily_spend, 50.0)
    | extend ActualPct = todouble(ActualCount) / toscalar(PredictionLog
        | where ModelName == "casino-player-churn-lightgbm"
        | where TimeGenerated > ago(7d) | count);
reference
| join kind=fullouter (current) on Bucket
| extend ExpectedPct = coalesce(ExpectedPct, 0.0001), ActualPct = coalesce(ActualPct, 0.0001)
| extend PSI_contrib = (ActualPct - ExpectedPct) * log(ActualPct / ExpectedPct)
| summarize PSI = sum(PSI_contrib)

PySpark (offline reference build):

from pyspark.sql.functions import col, log

def compute_psi(ref_df, cur_df, feature, n_bins=10):
    qs = ref_df.approxQuantile(feature, [i/n_bins for i in range(n_bins+1)], 0.001)
    bucketize = lambda df: df.withColumn("bucket",
        F.expr(f"width_bucket({feature}, array{tuple(qs[1:-1])})"))
    ref = bucketize(ref_df).groupBy("bucket").count().withColumnRenamed("count", "rc")
    cur = bucketize(cur_df).groupBy("bucket").count().withColumnRenamed("count", "cc")
    rt, ct = ref_df.count(), cur_df.count()
    joined = ref.join(cur, "bucket", "fullouter").fillna(1)
    return joined.withColumn("psi", (col("cc")/ct - col("rc")/rt) * log((col("cc")/ct) / (col("rc")/rt))) \
                 .agg({"psi": "sum"}).collect()[0][0]

KS Test — Kolmogorov–Smirnov¶

Use when: Feature is continuous and you don't want to discretize.

Formula: Maximum vertical distance between the two empirical CDFs. KS statistic = max|F_ref(x) − F_cur(x)|.

Threshold: Reject null at p < 0.01 with effect size > 0.1 (D-statistic). Bonferroni-correct when running across many features.

KQL:

// KS test approximation via percentile comparison
let ref_pcts = ReferenceDistribution
    | where ModelName == "casino-player-churn-lightgbm" and FeatureName == "avg_session_minutes"
    | summarize percentiles_ref = make_list(percentile_value) by FeatureName;
let cur_pcts = PredictionLog
    | where ModelName == "casino-player-churn-lightgbm"
    | where TimeGenerated > ago(7d)
    | summarize percentiles_cur = percentiles(avg_session_minutes,
        5, 10, 25, 50, 75, 90, 95);
ref_pcts
| extend ks_stat = todouble(...) // compute in notebook for full KS

KQL approximates KS via percentile binning. For full KS p-value, run a daily PySpark job (scipy.stats.ks_2samp) over a sample.

Chi-Square — Categorical Drift¶

Use when: Feature is categorical (region, channel, device type).

Threshold: p < 0.01 with Cramér's V > 0.1 (effect size).

KQL:

// Chi-square contributions per category
let reference = ReferenceDistribution
    | where ModelName == "casino-fraud-lightgbm" and FeatureName == "merchant_category"
    | project Category, ExpectedCount;
let current = PredictionLog
    | where ModelName == "casino-fraud-lightgbm"
    | where TimeGenerated > ago(1d)
    | summarize ObservedCount = count() by Category = merchant_category;
reference
| join kind=fullouter (current) on Category
| extend ExpectedCount = coalesce(ExpectedCount, 1.0), ObservedCount = coalesce(ObservedCount, 1.0)
| extend chi_contrib = pow(ObservedCount - ExpectedCount, 2) / ExpectedCount
| summarize chi_square = sum(chi_contrib), df = count() - 1

Wasserstein Distance (Earth Mover's Distance)¶

Use when: You want a single, scale-invariant number that respects the ordering of bins. Better than PSI when bin boundaries shift slightly.

Formula: Minimum "work" (mass × distance) to transform one distribution into another.

Threshold: Domain-dependent; calibrate against historic stable periods (e.g., > 2σ of baseline = drift).

PySpark (using scipy):

from scipy.stats import wasserstein_distance
ref_sample = reference_df.select("amount").sample(0.1).toPandas()["amount"]
cur_sample = current_df.select("amount").sample(0.1).toPandas()["amount"]
wd = wasserstein_distance(ref_sample, cur_sample)
mlflow.log_metric("wasserstein_amount", wd)

Jensen–Shannon Divergence¶

Use when: Both distributions are categorical or binned, and you want a symmetric, bounded [0, 1] metric.

Formula: JSD(P‖Q) = 0.5 × KL(P‖M) + 0.5 × KL(Q‖M) where M = 0.5(P+Q).

Threshold: JSD > 0.1 = notable; JSD > 0.2 = severe (similar to PSI bands).

import numpy as np
from scipy.spatial.distance import jensenshannon

def jsd(p, q, epsilon=1e-9):
    p = np.array(p) + epsilon; q = np.array(q) + epsilon
    p /= p.sum(); q /= q.sum()
    return jensenshannon(p, q) ** 2  # scipy returns sqrt(JSD)

Method Selection Matrix¶

🏗️ Reference Architecture¶

flowchart LR
    subgraph Serving["🚢 Production Serving"]
        EP[ML Model<br/>Endpoint]
        BATCH[Batch Inference<br/>Pipeline]
        STREAM[Stream Inference<br/>Eventstream]
    end

    subgraph Logging["📝 Prediction Logging"]
        LOG[Log Hook]
        EH[(🔥 Eventhouse<br/>PredictionLog)]
    end

    subgraph Reference["📚 Reference"]
        REF[(🥇 Lakehouse<br/>ReferenceDistribution)]
        SNAP[Training-time<br/>Snapshot]
    end

    subgraph Drift["🧪 Drift Detection"]
        SCH[Scheduled<br/>Notebook]
        KQL[KQL Drift<br/>Queries]
        DM[(🔥 Eventhouse<br/>DriftMetrics)]
    end

    subgraph Action["🚨 Action"]
        DASH[Real-Time<br/>Dashboard]
        ALERT[Action Group]
        RT[Retrain<br/>Pipeline]
    end

    EP --> LOG
    BATCH --> LOG
    STREAM --> LOG
    LOG --> EH
    SNAP --> REF
    REF --> KQL
    EH --> KQL
    SCH --> KQL
    KQL --> DM
    DM --> DASH
    DM --> ALERT
    ALERT -.->|drift sustained| RT
    RT -.->|new model| EP

Component Responsibilities¶

⚙️ Implementation in Fabric¶

Step 1 — Log Predictions to Eventhouse¶

Wrap every inference call to emit a structured record: model name, model version, request id, inputs (or hashed/quantized for PII), prediction, score, latency, timestamp.

# notebooks/ml/_helpers/prediction_logger.py
import json, uuid
from datetime import datetime
from azure.kusto.data import KustoConnectionStringBuilder
from azure.kusto.ingest import QueuedIngestClient, IngestionProperties, DataFormat

_client = QueuedIngestClient(KustoConnectionStringBuilder.with_aad_managed_service_identity(
    "https://ingest-<cluster>.kusto.fabric.microsoft.com"))

def log_prediction(model_name, model_version, request_id, features, prediction, score, latency_ms):
    record = {
        "TimeGenerated": datetime.utcnow().isoformat(),
        "ModelName": model_name, "ModelVersion": model_version,
        "RequestId": request_id or str(uuid.uuid4()),
        "Features": json.dumps(features), "Prediction": str(prediction),
        "Score": float(score), "LatencyMs": float(latency_ms),
    }
    _client.ingest_from_stream(json.dumps(record).encode("utf-8"),
        IngestionProperties(database="ml_monitoring", table="PredictionLog",
                            data_format=DataFormat.JSON))

📝 Hot path note: For low-latency endpoints, use fire-and-forget logging (queue, don't block the response). For batch scoring, log in micro-batches of 10–50K records using ingest_from_dataframe.

Step 2 — Build Reference Distribution at Training Time¶

# notebooks/ml/02_build_reference_distribution.py
import mlflow
from pyspark.sql import functions as F

NUMERIC = ["avg_daily_spend", "avg_session_minutes", "days_since_last_visit"]
CATEGORICAL = ["preferred_game", "tier", "channel"]
MODEL, VERSION = "casino-player-churn-lightgbm", "v42"
ref_df = spark.table("lh_gold.gold_player_360_training"); total = ref_df.count()
records = []

for feat in NUMERIC:
    qs = ref_df.approxQuantile(feat, [i/10 for i in range(11)], 0.001)
    binned = ref_df.withColumn("bucket",
        F.expr(f"width_bucket({feat}, array{tuple(qs[1:-1])})").cast("string"))
    for r in binned.groupBy("bucket").count().collect():
        records.append((MODEL, VERSION, feat, r["bucket"], r["count"]/total, qs))

for feat in CATEGORICAL:
    for r in ref_df.groupBy(feat).count().orderBy(F.desc("count")).limit(50).collect():
        records.append((MODEL, VERSION, feat, r[feat], r["count"]/total, None))

spark.createDataFrame(records,
    ["ModelName", "ModelVersion", "FeatureName", "Bucket", "ExpectedPct", "Quantiles"]) \
    .write.mode("overwrite").saveAsTable("lh_gold.reference_distribution")

mlflow.log_param("reference_distribution_table", "lh_gold.reference_distribution")
mlflow.log_param("reference_distribution_version",
    spark.sql("DESCRIBE HISTORY lh_gold.reference_distribution LIMIT 1").collect()[0]["version"])

Step 3 — Schedule Drift Computation¶

# Fabric Pipeline (drift_detection_daily)
- name: compute-drift-casino-churn
  type: TridentNotebook
  notebook: 03_drift_detection_daily
  parameters:
    model_name: casino-player-churn-lightgbm
    window_hours: 24
    reference_table: lh_gold.reference_distribution
    output_eventhouse_db: ml_monitoring
    output_table: DriftMetrics
  trigger: schedule(0 4 * * *)  # daily 4am

The notebook reads PredictionLog (last 24h) + ReferenceDistribution, computes PSI/KS/Chi-square per feature, and writes to DriftMetrics.

Step 4 — Real-Time Dashboard¶

Build a Real-Time Dashboard against DriftMetrics with tiles for:

• Top-5 drifted features over the last 24 hours (heatmap)
• PSI trend per feature, last 30 days (line chart)
• Prediction-score histogram, last 24h vs reference (overlay)
• Performance metrics (when ground truth available) vs reference window

See Real-Time Intelligence for dashboard authoring patterns.

🔍 KQL Drift Library¶

Five runnable queries. All assume PredictionLog and DriftMetrics tables in Eventhouse, and ReferenceDistribution materialized into Eventhouse via Shortcut from lh_gold.

Query 1 — PSI per feature, last 24 hours¶

let model = "casino-player-churn-lightgbm";
let win_start = ago(1d);
let total = toscalar(PredictionLog
    | where ModelName == model and TimeGenerated > win_start
    | count);
PredictionLog
| where ModelName == model and TimeGenerated > win_start
| extend feats = parse_json(Features)
| mv-expand feats
| extend FeatureName = tostring(bag_keys(feats)[0]),
         FeatureValue = tostring(feats[tostring(bag_keys(feats)[0])])
| summarize ActualCount = count() by FeatureName, Bucket = FeatureValue
| extend ActualPct = todouble(ActualCount) / total
| join kind=inner (
    ReferenceDistribution
    | where ModelName == model
    | project FeatureName, Bucket, ExpectedPct
) on FeatureName, Bucket
| extend ExpectedPct = iff(ExpectedPct == 0, 0.0001, ExpectedPct),
         ActualPct = iff(ActualPct == 0, 0.0001, ActualPct)
| extend psi_contrib = (ActualPct - ExpectedPct) * log(ActualPct / ExpectedPct)
| summarize PSI = sum(psi_contrib) by FeatureName
| order by PSI desc

Query 2 — KS-like statistic per numeric feature¶

let model = "casino-player-churn-lightgbm";
PredictionLog
| where ModelName == model and TimeGenerated > ago(1d)
| extend feats = parse_json(Features), v = todouble(feats.avg_daily_spend)
| summarize cp = percentiles(v, 5, 25, 50, 75, 95)
| extend FeatureName = "avg_daily_spend",
         KS_stat = max_of(
             abs(cp.percentile_v_5  - toscalar(ReferenceDistribution | where FeatureName=="avg_daily_spend" and Bucket=="p5"  | project ExpectedPct)),
             abs(cp.percentile_v_50 - toscalar(ReferenceDistribution | where FeatureName=="avg_daily_spend" and Bucket=="p50" | project ExpectedPct)),
             abs(cp.percentile_v_95 - toscalar(ReferenceDistribution | where FeatureName=="avg_daily_spend" and Bucket=="p95" | project ExpectedPct))
         )
| project FeatureName, KS_stat

Query 3 — Prediction distribution shift (output drift)¶

let model = "casino-player-churn-lightgbm";
let cur = PredictionLog
    | where ModelName == model and TimeGenerated > ago(1d)
    | summarize cur_count = count() by ScoreBucket = bin(Score, 0.1)
    | extend cur_pct = todouble(cur_count) / toscalar(PredictionLog
        | where ModelName == model and TimeGenerated > ago(1d) | count);
let ref = ReferenceDistribution
    | where ModelName == model and FeatureName == "__prediction__"
    | project ScoreBucket = todouble(Bucket), ref_pct = ExpectedPct;
cur
| join kind=fullouter (ref) on ScoreBucket
| extend cur_pct = coalesce(cur_pct, 0.0001), ref_pct = coalesce(ref_pct, 0.0001)
| extend psi_contrib = (cur_pct - ref_pct) * log(cur_pct / ref_pct)
| summarize PredictionPSI = sum(psi_contrib)

Query 4 — Performance vs reference window¶

let model = "casino-player-churn-lightgbm";
let perf_ref = toscalar(DriftMetrics
    | where ModelName == model and MetricName == "AUC_baseline"
    | top 1 by TimeGenerated desc | project MetricValue);
DriftMetrics
| where ModelName == model and MetricName == "AUC"
| where TimeGenerated > ago(7d)
| summarize CurAUC = avg(MetricValue)
| extend RefAUC = perf_ref,
         AUC_delta = CurAUC - perf_ref,
         AUC_pct_change = (CurAUC - perf_ref) / perf_ref * 100

AUC is computed in the daily reconciliation notebook (sklearn.metrics.roc_auc_score) and written to DriftMetrics. KQL handles the trend comparison.

Query 5 — Top-N drifted features (alert candidate)¶

DriftMetrics
| where ModelName == "casino-player-churn-lightgbm"
| where MetricName == "PSI"
| where TimeGenerated > ago(1d)
| summarize MaxPSI = max(MetricValue) by FeatureName
| where MaxPSI > 0.2
| top 10 by MaxPSI desc
| extend Severity = case(MaxPSI > 0.25, "P1", MaxPSI > 0.2, "P2", "P3")

📉 Performance Drift Patterns¶

Performance drift requires ground truth. Three patterns by label arrival latency:

Pattern A — Realized vs Predicted (immediate labels)¶

For models where ground truth arrives within minutes/hours: click-through, fraud-claim resolution, real-time forecasting.

from sklearn.metrics import roc_auc_score, brier_score_loss

preds = spark.sql("""
    SELECT p.Score, r.ActualLabel
    FROM ml_monitoring.PredictionLog p
    JOIN lh_silver.realized_outcomes r ON p.RequestId = r.RequestId
    WHERE p.TimeGenerated >= current_date() - INTERVAL 1 DAY
      AND p.ModelName = 'casino-fraud-lightgbm'
""").toPandas()

auc = roc_auc_score(preds.ActualLabel, preds.Score)
brier = brier_score_loss(preds.ActualLabel, preds.Score)
mlflow.log_metric("daily_auc", auc); mlflow.log_metric("daily_brier", brier)

spark.createDataFrame([(datetime.utcnow(), "casino-fraud-lightgbm", "AUC", auc),
                       (datetime.utcnow(), "casino-fraud-lightgbm", "Brier", brier)],
    ["TimeGenerated", "ModelName", "MetricName", "MetricValue"]) \
    .write.mode("append").saveAsTable("ml_monitoring.DriftMetrics")

Pattern B — Proxy Metrics (delayed labels)¶

When labels arrive in days/weeks (USDA crop yield, casino player churn over 90 days):

Pattern C — A/B Holdout vs Production¶

For mission-critical models, deliberately route 5% of traffic to a holdout that uses the previous model. Compare performance side-by-side as labels arrive. This isolates "is it the model" from "is it the world."

-- Power BI / DAX measure for holdout-vs-prod
PerformanceDelta =
VAR ProdAUC = CALCULATE(AVERAGE(DriftMetrics[Value]),
                        DriftMetrics[Group] = "production",
                        DriftMetrics[MetricName] = "AUC")
VAR HoldoutAUC = CALCULATE(AVERAGE(DriftMetrics[Value]),
                           DriftMetrics[Group] = "holdout-v41",
                           DriftMetrics[MetricName] = "AUC")
RETURN ProdAUC - HoldoutAUC

🌀 Concept Drift Detection¶

Concept drift is the hardest. Inputs look fine, but the relationship between inputs and target has changed. Three signals:

Signal 1 — Performance drops while inputs stable¶

// Concept drift candidate: AUC fell, but input PSI is normal
let model = "casino-player-churn-lightgbm";
let perf_drop = DriftMetrics
    | where ModelName == model and MetricName == "AUC"
    | where TimeGenerated > ago(7d)
    | summarize CurAUC = avg(MetricValue);
let baseline = DriftMetrics
    | where ModelName == model and MetricName == "AUC_baseline"
    | top 1 by TimeGenerated desc
    | project BaseAUC = MetricValue;
let max_input_psi = DriftMetrics
    | where ModelName == model and MetricName == "PSI" and TimeGenerated > ago(7d)
    | summarize MaxInputPSI = max(MetricValue);
perf_drop | extend BaseAUC = toscalar(baseline), MaxInputPSI = toscalar(max_input_psi)
| extend
    AUC_drop_pct = (BaseAUC - CurAUC) / BaseAUC * 100,
    ConceptDriftSuspected = AUC_drop_pct > 5 and MaxInputPSI < 0.1

Signal 2 — Tree-based feature importance shift¶

Train a shadow model on a recent labeled window and compare feature importances to production. Spearman ρ < 0.7 between importance vectors signals the relationship has changed.

import lightgbm as lgb
from scipy.stats import spearmanr

prod = mlflow.lightgbm.load_model("models:/casino-player-churn-lightgbm/Production")
prod_imp = dict(zip(prod.feature_name(), prod.feature_importance()))

recent = spark.table("ml_monitoring.PredictionLog") \
    .filter("TimeGenerated > current_date() - INTERVAL 14 DAY AND ActualLabel IS NOT NULL").toPandas()
shadow = lgb.LGBMClassifier(n_estimators=200).fit(
    recent.drop(columns=["ActualLabel"]), recent["ActualLabel"])
shadow_imp = dict(zip(shadow.feature_name_, shadow.feature_importances_))

feats = sorted(prod_imp.keys())
rho, _ = spearmanr([prod_imp[f] for f in feats], [shadow_imp[f] for f in feats])
mlflow.log_metric("feature_importance_spearman", rho)
if rho < 0.7: raise Alert("Concept drift suspected — feature importances diverged")

Signal 3 — Sliding-window comparison¶

Slice predictions into rolling windows (e.g., 7d × 4 weeks). Performance trending down monotonically, even with stable inputs, is the textbook concept-drift signature.

DriftMetrics
| where ModelName == "casino-player-churn-lightgbm" and MetricName == "AUC"
| where TimeGenerated > ago(28d)
| summarize WeekAvg = avg(MetricValue) by Week = bin(TimeGenerated, 7d)
| serialize
| extend Trend = row_number() - 1
| extend Slope = todouble(WeekAvg) / Trend

If Slope is consistently negative for 3+ weeks → concept drift; trigger retraining with fresh labels (not augmented historical data).

🚨 Alert Wiring¶

Drift alerts must be actionable, deduplicated, and routed to the right team. Reuse the patterns from SLO/SLI for Fabric and Observability Stack.

Alert Severity Matrix¶

Scheduled Query Alert (Workspace Monitoring)¶

# Azure Monitor scheduled query alert pointing at the Fabric KQL endpoint
name: drift-p1-casino-churn
query: |
    DriftMetrics
    | where ModelName == "casino-player-churn-lightgbm" and MetricName == "PSI"
    | where TimeGenerated > ago(72h)
    | summarize WindowsOver = countif(MetricValue > 0.25) by FeatureName
    | where WindowsOver >= 3
schedule: every 1 hour
threshold: results > 0
severity: 1
action_group: ml-oncall-pagerduty   # P1 → PagerDuty; P2 → Teams; trigger → retraining Logic App

🔁 Retraining Trigger Patterns¶

See MLOps for Fabric Production § Retraining Triggers for the master list. Drift-specific patterns:

Cool-Down Logic¶

Prevent retrain storms. After triggering a retrain, suppress drift-driven retrains for the cool-down window. Alerts still fire (humans should know), but the automated pipeline is gated.

let last_retrain = toscalar(DriftMetrics
    | where ModelName == "casino-player-churn-lightgbm" and MetricName == "RetrainTriggered"
    | top 1 by TimeGenerated desc | project TimeGenerated);
DriftMetrics
| where ModelName == "casino-player-churn-lightgbm" and MetricName == "PSI"
| where TimeGenerated > ago(72h) and TimeGenerated > last_retrain + 7d  // cool-down
| summarize WindowsOver = countif(MetricValue > 0.2) by FeatureName
| where WindowsOver >= 3

🛡️ False Positive Mitigation¶

Most drift alerts are false positives the first time you turn on monitoring. Tune for signal, not noise.

// Sample-size gate — applied to every PSI alert
let n_samples = toscalar(PredictionLog
    | where ModelName == "casino-player-churn-lightgbm" and TimeGenerated > ago(1d) | count);
DriftMetrics
| where ModelName == "casino-player-churn-lightgbm" and MetricName == "PSI"
| extend SampleSize = n_samples
| where SampleSize >= 1000   // suppress small-sample noise

# Seasonal reference build — keyed lookup at drift-compute time
ref_seasonal = training_df.groupBy("day_of_week", "hour", "is_holiday") \
    .agg(F.expr("percentile_approx(amount, 0.5)").alias("p50_amount"))
ref_seasonal.write.mode("overwrite").saveAsTable("lh_gold.reference_distribution_seasonal")

🎰 Casino Implementation¶

Compliance: Fraud-model drift is a BSA/SAR concern — a drifting fraud model that misses structuring is a compliance failure. Drift alerts on casino-fraud-lightgbm are P0; page the AML team, not just on-call ML.

🏛️ Federal Implementation¶

Public-safety models (NOAA storm, USGS earthquake) require human-in-the-loop retraining. Drift fires; retraining is manually approved by the agency's chief scientist. Never auto-promote.

🚫 Anti-Patterns¶

📋 Implementation Checklist¶

Before declaring a model "monitored":

• Prediction logging hook deployed (every inference emits a structured record)
• PredictionLog table in Eventhouse with hot/cold tiering policy
• Reference distribution built at training time and stored in Lakehouse with version pinned to MLflow run
• Reference distribution materialized to Eventhouse via Shortcut for KQL joins
• PSI computed per feature, scheduled at appropriate cadence (hourly for streaming, daily for batch)
• Prediction-output PSI computed (output drift)
• Performance metric pipeline in place (realized vs predicted, A/B holdout, or proxy metrics)
• Concept-drift detection: importance shift + sliding-window performance
• DriftMetrics table populated and dashboarded
• Real-Time Dashboard published with: top-N drifted features, prediction histogram, performance trend
• Alert thresholds calibrated against 30-day stable history (no first-day-on alerts at default thresholds)
• Alerts wired to Action Groups with P1/P2/P3 routing
• Retraining trigger Logic App / Function deployed and tested end-to-end
• Cool-down logic implemented to prevent retrain storms
• Seasonality / holiday calendar joins applied to drift queries
• Suppression mechanism for intentional product changes (ChatOps or table-driven)
• Fairness-drift detection (regulated domains only)
• Runbook published: "What to do when drift fires"
• On-call team trained on the runbook
• Quarterly drift-detection review: are thresholds still right? Any silent failures?

📚 References¶

Microsoft Fabric Documentation¶

• Workspace Monitoring
• Eventhouse and KQL Database
• ML Model Endpoints — Monitoring (Preview)
• MLflow Tracking in Fabric
• Real-Time Dashboards

Industry Standards & Papers¶

• Gama et al. — A Survey on Concept Drift Adaptation (ACM Computing Surveys, 2014)
• Lipton et al. — Detecting and Correcting for Label Shift with Black Box Predictors (ICML 2018)
• Rabanser et al. — Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (NeurIPS 2019)
• Google — The ML Test Score: A Rubric for ML Production Readiness
• Microsoft — Responsible AI Standard (Reliability & Safety Goal)

Related Wave 2 Docs¶

• MLOps for Fabric Production — anchor doc
• Feature Store on OneLake
• Responsible AI Framework
• LLM Cost Tracking
• LLM Evaluation Harness

Related Operational Docs (Wave 1)¶

• SLO/SLI for Fabric
• Observability Stack
• Incident Response Template
• On-Call Rotation Handbook

Related Existing Docs¶

• AutoML & Model Endpoints
• Real-Time Intelligence
• Eventhouse Vector Database
• Monitoring & Observability
• Alerting with Data Activator

⬆️ Back to Top | 📚 Best Practices Index | 🏠 Home