Skip to content
Operations runbooks — incident response, key rotation, DR drills, cost-alert response, and routine platform maintenance

Runbooks

Operational playbooks for responding to incidents and routine maintenance on the CSA-in-a-Box platform. Each runbook is structured for first-responder use: detection signals, triage steps, mitigation, root-cause analysis, and links to the relevant ADRs and code.

Incident response

Runbook When to use
Break-Glass Access Emergency elevation when normal access paths are unavailable
Security Incident Suspected breach, credential exposure, or anomalous activity
Data Pipeline Failure ADF/Databricks pipeline failures, retry exhaustion, dead-letter overflow
Dead Letter Queue Drop-event triage from streaming and batch pipelines
DR Drill Quarterly disaster-recovery exercises and the live failover playbook
Purview Scan Failure Catalog scans not completing or returning unexpected results
Azure OpenAI Throttling 429s from Azure OpenAI; capacity reallocation
Databricks Cost Runaway Spend anomaly response for Databricks clusters
Cost Alert Response Budget threshold response across the platform

Routine operations

Runbook When to use
Certificate Expiration Pre-emptive cert rotation and last-minute renewal
Key Rotation Scheduled rotation of Key Vault keys and secrets
Azure Deployment Service Principal Managing the limitlessdata_deploy SP used by CI/CD
Copilot Studio Agent Rotation Refreshing Copilot Studio agent registrations
Fabric Capacity Management Scaling Fabric SKUs up/down based on workload
Tenant Onboarding Adding a new tenant to multi-tenant landing zones
Portal Rollout Strategy Staged portal release process
dbt CI dbt pipeline CI workflow, hotfixes, and rerun playbook
Release-Please Status Bypass Working around release-please CI status when it stalls