📊 Monitoring & Alerting¶

Implement comprehensive monitoring and alerting for Azure Data Factory pipelines using Azure Monitor and custom dashboards.

📋 Table of Contents¶

Monitoring Overview
Pipeline Monitoring
Azure Monitor Integration
Custom Alerts
Next Steps

📊 Monitoring Overview¶

Monitoring Capabilities¶

Pipeline run history
Activity-level metrics
Integration runtime metrics
Data movement statistics
Error tracking and diagnostics

🔍 Pipeline Monitoring¶

ADF Studio Monitoring¶

Navigate to Monitor tab in ADF Studio to view:

Pipeline runs
Trigger runs
Integration runtime status
Data flow debug sessions

Key Metrics¶

Metric	Description	Alert Threshold
Failed Runs	Number of pipeline failures	> 0
Duration	Pipeline execution time	> SLA
Data Read/Written	Data volume processed	Unexpected changes
DIU Hours	Data Integration Units consumed	Budget limits

📈 Azure Monitor Integration¶

Enable Diagnostic Settings¶

# Enable diagnostic settings
az monitor diagnostic-settings create \
  --name "ADF-Diagnostics" \
  --resource "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.DataFactory/factories/xxx" \
  --workspace "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.OperationalInsights/workspaces/xxx" \
  --logs '[{"category": "PipelineRuns", "enabled": true}, {"category": "ActivityRuns", "enabled": true}]' \
  --metrics '[{"category": "AllMetrics", "enabled": true}]'

Log Analytics Queries¶

// Failed pipeline runs in last 24 hours
ADFPipelineRun
| where TimeGenerated > ago(24h)
| where Status == "Failed"
| project TimeGenerated, PipelineName, Status, ErrorMessage
| order by TimeGenerated desc

🚨 Custom Alerts¶

Create Alert Rule¶

# Create alert for pipeline failures
az monitor metrics alert create \
  --name "ADF-Pipeline-Failures" \
  --resource-group "rg-adf-tutorial-dev" \
  --scopes "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.DataFactory/factories/xxx" \
  --condition "count PipelineFailedRuns > 0" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action-group "adf-alerts-action-group"

📚 Additional Resources¶

🚀 Next Steps¶

→ 16. Performance Optimization

Module Progress: 15 of 18 complete

Tutorial Version: 1.0 Last Updated: January 2025