Home > Docs > Runbooks > Data Quality Incident

🧪 Data Quality Incident Runbook¶

Last Updated: 2026-05-05 | Version: 2.0 Audience: Data engineers, on-call SRE, data stewards, compliance officers Purpose: Detect data quality degradation, assess downstream impact, quarantine bad records, communicate with stakeholders, and remediate — without propagating bad data through the medallion architecture.

📑 Table of Contents¶

1. Trigger Conditions
2. Severity Classification
3. Decision Flowchart
4. Step-by-Step Procedure
5. Quarantine Procedures
6. Stakeholder Communication
7. Remediation Patterns
8. Escalation Path
9. Post-Incident Review Checklist
10. Related Documents

Trigger Conditions¶

Use this runbook when any of the following conditions are observed:

Severity Classification¶

Rule of thumb: Classify by the furthest-downstream layer the bad data has reached, not by the layer where it was detected. A null spike caught in Bronze is SEV4. The same null spike found in a Power BI exec report is SEV2.

Compliance Override¶

Any incident touching CTR, SAR, W-2G, HIPAA PHI, FedRAMP audit logs, 42 CFR Part 2 SUD records, or PCI cardholder data is automatically SEV1 or SEV2 regardless of layer.

Decision Flowchart¶

flowchart TD
    A([Data Quality Alert]) --> B{Where was bad<br/>data detected?}
    B -->|Bronze| C{Quarantine<br/>caught it?}
    C -->|Yes| D[SEV4 — Review DLQ,<br/>fix source → Step 8]
    C -->|No| E[SEV3 — Stop Silver pipeline,<br/>quarantine manually → Step 7]

    B -->|Silver| F{Propagated<br/>to Gold?}
    F -->|No| E
    F -->|Yes| G{Consumed by<br/>reports or consumers?}

    B -->|Gold| G
    G -->|No| H[SEV3 — Pause Gold refresh,<br/>reprocess → Step 9]
    G -->|Yes| I{Compliance<br/>data affected?}
    I -->|Yes| J[SEV1 — Page IC + Compliance,<br/>contain immediately → Step 5]
    I -->|No| K[SEV2 — Page on-call,<br/>flag stale reports → Step 6]

Step-by-Step Procedure¶

Phase 1 — Detect and Classify (0–15 min)¶

Step 1. Identify the quality issue type from the trigger condition. Collect: - Affected table(s) and column(s). - Number of bad records. - Time range of affected data. - Detection source (GE checkpoint, consumer report, anomaly alert).

Step 2. Determine the furthest-downstream layer the bad data has reached:

# Check if bad data propagated to Silver
bad_keys = spark.sql("""
    SELECT DISTINCT machine_id FROM bronze_slot_telemetry
    WHERE machine_id IS NULL AND process_date = '2026-05-05'
""")

silver_impact = spark.sql("""
    SELECT COUNT(*) as affected FROM silver_slot_cleansed
    WHERE machine_id IS NULL AND process_date = '2026-05-05'
""").collect()[0]["affected"]

print(f"Silver impact: {silver_impact} rows")

Step 3. Classify severity using the table above. If SEV1 or SEV2, open an incident channel immediately.

Step 4. If compliance data is affected (CTR, SAR, W-2G, PHI), notify the Compliance Officer regardless of severity.

Phase 2 — Contain (15–45 min)¶

Step 5. SEV1 — Compliance data affected: 1. Halt all downstream processing — pause Silver and Gold pipelines that consume the affected table. 2. Mark affected Power BI reports with a data quality banner or pause scheduled refreshes. 3. Notify the Compliance Officer with affected record count and time range. 4. Preserve the bad data for forensic analysis (do not delete).

Step 6. SEV2 — Gold data consumed by reports: 1. Pause the semantic model refresh for affected datasets. 2. Add a note to the Power BI report indicating data under investigation. 3. Notify affected report consumers via Teams / email.

Step 7. SEV3 — Bad data in Silver, not yet in Gold: 1. Pause Gold pipelines that consume the affected Silver table. 2. Quarantine bad records (see Quarantine Procedures). 3. Allow good records to continue flowing.

Step 8. SEV4 — Caught in Bronze quarantine: 1. Review the dead-letter queue (DLQ) table for the affected batch. 2. Determine root cause from source system. 3. No immediate downstream action needed — quarantine worked as designed.

Phase 3 — Diagnose (45–120 min)¶

Step 9. Investigate the root cause by issue type:

Step 10. Determine the scope of impact:

# Count affected records by date
impact = spark.sql("""
    SELECT process_date, COUNT(*) as bad_records
    FROM affected_table
    WHERE quality_flag = 'FAILED'
    GROUP BY process_date
    ORDER BY process_date
""")
impact.show()

Phase 4 — Remediate¶

Step 11. Fix the root cause: - Source-side fix: Coordinate with the source system team to correct the data or schema. - ETL fix: Update the notebook or pipeline transformation logic. - Validation fix: Add or tighten Great Expectations rules to catch this earlier.

Step 12. Reprocess affected data:

# Reprocess affected partitions through Silver
affected_dates = ["2026-05-04", "2026-05-05"]

for date in affected_dates:
    mssparkutils.notebook.run(
        "01_silver_slot_cleansing",
        timeout_seconds=3600,
        arguments={"process_date": date}
    )

# Then reprocess Gold
for date in affected_dates:
    mssparkutils.notebook.run(
        "01_gold_slot_performance",
        timeout_seconds=3600,
        arguments={"process_date": date}
    )

Step 13. Validate the reprocessed data:

great_expectations checkpoint run silver_checkpoint
great_expectations checkpoint run gold_checkpoint

Phase 5 — Verify and Close¶

Step 14. Confirm downstream reports show correct data: - Refresh affected semantic models. - Verify Power BI report values match expected results. - Remove any data quality banners from reports.

Step 15. Notify stakeholders that the issue is resolved (see Stakeholder Communication).

Step 16. Document the incident and complete the Post-Incident Review Checklist.

Quarantine Procedures¶

Bronze Quarantine (DLQ Pattern)¶

Route records that fail validation to a dead-letter queue table instead of blocking the pipeline:

from pyspark.sql.functions import col, lit, current_timestamp

# Validate incoming batch
valid = df.filter(col("machine_id").isNotNull() & col("event_timestamp").isNotNull())
invalid = df.filter(col("machine_id").isNull() | col("event_timestamp").isNull())

# Write valid records to Bronze
valid.write.format("delta").mode("append").save("Tables/bronze_slot_telemetry")

# Write invalid records to DLQ with metadata
invalid.withColumn("_dlq_reason", lit("null_required_field")) \
       .withColumn("_dlq_timestamp", current_timestamp()) \
       .write.format("delta").mode("append") \
       .save("Tables/_dlq_slot_telemetry")

Silver Quarantine¶

Tag records that fail business rules without removing them:

df_silver = df_silver.withColumn(
    "quality_flag",
    when(col("coin_out") > col("coin_in") * 100, "SUSPECT")
    .when(col("revenue") < 0, "SUSPECT")
    .otherwise("PASS")
)

# Only promote PASS records to Gold
df_gold_ready = df_silver.filter(col("quality_flag") == "PASS")

Stakeholder Communication¶

Remediation Patterns¶

Escalation Path¶

Post-Incident Review Checklist¶

• Affected table(s), column(s), and record count documented
• Time range of bad data identified
• Furthest-downstream layer reached by bad data documented
• Root cause category identified (source change, ETL bug, schema drift, validation gap)
• Quarantine procedure executed (DLQ, tagging, pipeline pause)
• Data reprocessed and validated with Great Expectations
• Downstream reports refreshed and verified
• Stakeholders notified of resolution
• New or tightened GE expectations added to prevent recurrence
• Source system team notified (if source-side root cause)
• Monitoring alert created or tuned for this failure pattern
• Compliance impact assessed and documented (if applicable)
• Runbook accuracy reviewed — any steps to add or update?
• Blameless postmortem completed within 48 hours (SEV1/SEV2 only)

Related Documents¶

⬆️ Back to Top | 📋 Runbook Index | 🏠 Home