Microsoft Purview Guide¶

Purview is the data governance hub for CSA-in-a-Box — providing data cataloging, lineage, classification, and access policies across the entire medallion architecture. See ADR-0006 for the decision rationale.

Why Purview¶

Microsoft Purview was chosen over Apache Atlas, DataHub, and Collibra because it is Gov-GA with FedRAMP High inheritance, integrates natively with Microsoft Information Protection (MIP) sensitivity labels, and provides built-in scanners for every service in the CSA stack (ADLS, Databricks, Synapse, SQL Server, Power BI). Purview's Entra ID RBAC fits the platform's existing persona model, and its metadata maps forward into Fabric Purview when tenants migrate.

Architecture Overview¶

graph TD
    subgraph "Data Sources"
        ADLS[(ADLS Gen2<br/>Bronze / Silver / Gold)]
        DBR[Databricks<br/>Unity Catalog]
        SQL[Azure SQL]
        COSMOS[Cosmos DB]
        PBI[Power BI<br/>Semantic Models]
        S3[(AWS S3<br/>Cross-cloud)]
    end

    subgraph "Microsoft Purview"
        DM[Data Map<br/>Scanners + Ingestion]
        CAT[Catalog<br/>Search + Glossary]
        LIN[Lineage<br/>ADF + dbt + Databricks]
        CLS[Classification<br/>Built-in + Custom]
        POL[Policies<br/>Access + DevOps]
        COL[Collections<br/>RBAC Hierarchy]
    end

    subgraph Consumers
        ANALYST[Data Analysts]
        STEWARD[Data Stewards]
        ENGINEER[Data Engineers]
        AUDITOR[Compliance / Auditors]
    end

    ADLS --> DM
    DBR --> DM
    SQL --> DM
    COSMOS --> DM
    PBI --> DM
    S3 --> DM

    DM --> CAT
    DM --> LIN
    DM --> CLS
    DM --> POL
    DM --> COL

    CAT --> ANALYST
    LIN --> AUDITOR
    CLS --> STEWARD
    POL --> ENGINEER

Setup¶

Account Creation (Bicep)¶

resource purviewAccount 'Microsoft.Purview/accounts@2021-12-01' = {
  name: 'pview-csa-${environment}-${location}'
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    publicNetworkAccess: 'Disabled'
    managedResourceGroupName: 'rg-pview-managed-${environment}'
  }
}

Managed Identity Permissions¶

Purview's system-assigned managed identity needs read access to every source it scans. Grant these roles before configuring scans.

# Grant ADLS read access to Purview managed identity
PURVIEW_MI=$(az purview account show \
  --name "pview-csa-dev-eastus" \
  --resource-group "rg-dmlz-dev" \
  --query "identity.principalId" -o tsv)

az role assignment create \
  --assignee "$PURVIEW_MI" \
  --role "Storage Blob Data Reader" \
  --scope "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Storage/storageAccounts/{sa}"

Data Map¶

Registering Sources¶

Register each data source in Purview's Data Map before scanning.

graph LR
    REG[Register Source] --> SCAN[Configure Scan]
    SCAN --> RULE[Apply Scan Rule Set]
    RULE --> SCHED[Set Schedule]
    SCHED --> RUN[Run Scan]
    RUN --> ASSETS[Assets in Catalog]

Source Registration Examples¶

Scan Rule Sets¶

Scan rule sets determine which file types and classification rules to apply. CSA-in-a-Box uses a custom rule set that includes Delta Lake metadata and government-specific classifiers.

{
    "name": "csa-scan-rules",
    "kind": "AzureStorage",
    "scanRulesetType": "Custom",
    "properties": {
        "fileExtensions": [".parquet", ".delta", ".json", ".csv", ".avro"],
        "classificationRuleNames": [
            "MICROSOFT.GOVERNMENT.US_SOCIAL_SECURITY_NUMBER",
            "MICROSOFT.FINANCIAL.CREDIT_CARD_NUMBER",
            "CSA.GOVERNMENT.CUI_MARKING",
            "CSA.GOVERNMENT.FOUO_MARKING"
        ]
    }
}

Catalog¶

Business Glossary¶

The glossary provides a controlled vocabulary for business terms. CSA-in-a-Box organizes glossary terms by domain.

Collections Hierarchy¶

Collections control both organization and RBAC. CSA-in-a-Box uses three levels.

Root Collection (CSA-in-a-Box)
├── Production
│   ├── Finance
│   ├── Healthcare
│   └── Defense
├── Staging
│   ├── Finance
│   └── Healthcare
└── Development
    └── Sandbox

Permissions flow downward. A user with Data Reader on Production can browse assets in Finance, Healthcare, and Defense unless explicitly denied at the child level.

Search¶

Purview search supports faceted queries across the entire data estate.

# Find all Delta tables in the Gold layer with PII classifications
qualifiedName:*gold* AND classification:*SSN*

Lineage¶

Automatic Lineage Sources¶

dbt Lineage Integration¶

"""Push dbt lineage to Purview via the Atlas API endpoint."""
import json
from azure.purview.catalog import PurviewCatalogClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
client = PurviewCatalogClient(
    endpoint="https://pview-csa-dev-eastus.purview.azure.com",
    credential=credential,
)

# Parse dbt manifest.json for model relationships
with open("target/manifest.json") as f:
    manifest = json.load(f)

for node_id, node in manifest["nodes"].items():
    if node["resource_type"] == "model":
        # Create lineage entity linking upstream sources to this model
        entity = {
            "typeName": "dbt_model",
            "attributes": {
                "qualifiedName": f"dbt://{node['database']}.{node['schema']}.{node['name']}",
                "name": node["name"],
                "description": node.get("description", ""),
            },
        }
        client.entity.create_or_update(entity={"entity": entity})

graph LR
    SRC[Source Tables<br/>Bronze] -->|ADF Copy| STG[Staging<br/>Silver]
    STG -->|dbt model| GOLD[Gold Tables]
    GOLD -->|Power BI scan| PBI[Semantic Model]
    PBI --> RPT[Report / Dashboard]

    style SRC fill:#cd7f32
    style STG fill:#c0c0c0
    style GOLD fill:#ffd700

Classification¶

Built-in Classifiers¶

Purview ships with 200+ system classifiers. The most relevant for government workloads:

Custom Classifiers for Government Data¶

{
    "name": "CSA.GOVERNMENT.CUI_MARKING",
    "kind": "Custom",
    "properties": {
        "classificationName": "CUI Marking",
        "description": "Controlled Unclassified Information marking per NIST SP 800-171",
        "pattern": {
            "kind": "Regex",
            "pattern": "(?i)(CUI|CONTROLLED UNCLASSIFIED|NOFORN|FOUO|LAW ENFORCEMENT SENSITIVE)"
        },
        "minimumPercentageMatch": 60
    }
}

Sensitivity Labels¶

Purview integrates with Microsoft Information Protection to apply sensitivity labels that follow data across the estate.

Label hierarchy:
├── Public
├── Internal
├── Confidential
│   ├── Confidential - PII
│   └── Confidential - Financial
└── Highly Confidential
    ├── Highly Confidential - CUI
    └── Highly Confidential - ITAR

Sensitivity labels applied in Purview propagate to Power BI datasets automatically. A Highly Confidential label on a Gold table will appear on any Power BI semantic model built on that table.

Data Access Policies¶

Policy Types¶

Enable self-service access policies so analysts can request access to datasets through the Purview portal. Data stewards approve or deny requests, creating an auditable access trail for compliance.

Purview + Unity Catalog¶

CSA-in-a-Box uses a dual governance pattern: Unity Catalog governs data inside the Databricks workspace (fine-grained ACLs, row/column filters), while Purview provides the enterprise-wide catalog that non-Databricks consumers (SQL, Power BI, Fabric) use.

graph LR
    UC[Unity Catalog<br/>Databricks-internal governance] -->|Federation| PV[Purview<br/>Enterprise catalog]
    PV --> PBI[Power BI]
    PV --> SQL[Azure SQL]
    PV --> FAB[Fabric]
    UC --> DBR[Databricks Users]

Unity Catalog metadata is federated into Purview, not duplicated. Purview scans the Unity Catalog metastore and creates linked assets — the source of truth for table definitions remains Unity Catalog.

Purview + Fabric¶

When tenants migrate to Fabric (ADR-0010), Purview metadata moves forward:

• Fabric Purview is a superset of standalone Purview
• OneLake scanning replaces ADLS scanning
• Direct Lake semantic models appear automatically in the catalog
• Sensitivity labels propagate through OneLake to Fabric workloads

Monitoring¶

Key Metrics¶

Cost Optimization¶

ADR-0006 warns that if scanner cost exceeds 10% of storage cost, revisit scanner scope and cadence — not the catalog choice. Scope scans at the domain/collection level rather than tenant-wide.

Anti-Patterns¶

Tenant-wide scans discover assets you do not govern, inflating costs and cluttering the catalog. Register specific sources and scope scans to relevant collections.

Without glossary terms, the catalog is a technical inventory — not a governance tool. Invest in glossary terms early; they compound in value as adoption grows.

System classifiers catch common patterns (SSN, credit cards) but miss domain-specific sensitive data. Create custom classifiers for CUI, FOUO, ITAR, and any organization-specific data categories.

Maintain one source of truth per domain. Databricks teams use Unity Catalog; everyone else discovers the same data through Purview.

Labels applied in Purview should flow to Power BI and downstream consumers automatically. This is the primary mechanism for demonstrating data classification compliance.

Checklist¶

• Purview account deployed via DMLZ Bicep templates
• Managed identity granted read access to all governed sources
• Collection hierarchy created (Organization > Environment > Domain)
• Sources registered: ADLS, Databricks, SQL, Cosmos DB, Power BI
• Custom classifiers created for CUI, FOUO, ITAR
• Scan rule sets configured with Delta Lake file types
• Scan schedules configured (daily production, weekly dev)
• Business glossary seeded with initial domain terms
• Sensitivity labels configured and propagation enabled
• dbt lineage integration tested (manifest parser or OpenLineage)
• Unity Catalog federation verified
• Self-service access policies enabled
• Diagnostic settings forwarding to Log Analytics

Related Documentation¶

• ADR-0006 — Purview over Atlas
• ADR-0010 — Fabric Strategic Target
• Governance — Purview Setup
• Governance — Data Lineage
• Governance — Data Cataloging
• Governance — Data Access
• Best Practices — Data Governance
• Guide — OSS Ecosystem (Atlas alternative)