Best Practices — Databricks to Fabric Migration¶

Status: Authored 2026-04-30 Audience: Migration leads, platform engineers, and architects executing a Databricks-to-Fabric migration or building a hybrid Databricks + Fabric architecture. Scope: Hybrid strategy patterns, workspace mapping, capacity planning, notebook conversion checklist, common pitfalls, and operational runbook.

1. Hybrid Databricks + Fabric strategy¶

1.1 Why hybrid is the default recommendation¶

Most enterprises should not attempt a full Databricks-to-Fabric migration. Instead, adopt a hybrid architecture where each platform handles what it does best:

1.2 Hybrid architecture reference¶

flowchart TB
    subgraph Databricks
        ML[ML Training<br/>Photon + GPU]
        ETL[Heavy ETL<br/>Jobs Compute]
        MLF[MLflow + Feature Store]
    end

    subgraph SharedStorage[Shared ADLS Gen2]
        Delta[(Delta Tables<br/>Bronze / Silver / Gold)]
    end

    subgraph Fabric
        LH[Lakehouse<br/>OneLake shortcuts]
        SQL[SQL Endpoint<br/>ad-hoc queries]
        PBI[Power BI<br/>Direct Lake]
        RTI[Real-Time Intelligence<br/>Eventhouse]
        DP[Data Pipelines<br/>orchestration]
    end

    ML --> Delta
    ETL --> Delta
    Delta --> LH
    LH --> SQL
    LH --> PBI
    LH --> RTI
    DP --> LH

1.3 Integration points¶

1.4 What to migrate first¶

Prioritize workloads by ROI:

2. Workspace mapping¶

2.1 Databricks-to-Fabric workspace mapping patterns¶

Pattern A: 1-to-1 (simple)

One Databricks workspace maps to one Fabric workspace:

Databricks: production-workspace  -->  Fabric: Production-Analytics
Databricks: development-workspace -->  Fabric: Development-Analytics

Best for: Small teams, single-project organizations.

Pattern B: Catalog-to-workspace (recommended)

Unity Catalog catalogs map to Fabric workspaces:

UC catalog: production  -->  Fabric workspace: Production
UC catalog: development -->  Fabric workspace: Development
UC catalog: staging     -->  Fabric workspace: Staging

Best for: Organizations using UC for environment isolation.

Pattern C: Domain-to-workspace (enterprise)

Business domains map to separate workspaces:

UC catalog: finance     -->  Fabric workspace: Finance-Analytics
UC catalog: marketing   -->  Fabric workspace: Marketing-Analytics
UC catalog: operations  -->  Fabric workspace: Operations-Analytics

Best for: Large enterprises with domain-driven data ownership.

2.2 Workspace naming conventions¶

Recommendation: Use {Domain}-{Environment} for production workspaces and {Team}-{Purpose} for development.

2.3 Lakehouse vs Warehouse decision¶

Default recommendation: Use Lakehouse for most workloads. Use Warehouse only when you need T-SQL compatibility or fine-grained SQL security.

3. Capacity planning¶

3.1 Sizing methodology¶

1. Measure current Databricks usage: Export 3 months of DBU consumption from system.billing.usage
2. Identify peak and average: Calculate peak hour and 24-hour average
3. Apply smoothing factor: Fabric smoothing averages CU over 24 hours; this usually means you need a smaller SKU than peak suggests
4. Start one tier lower: Fabric capacity can be scaled up in minutes; start conservatively
5. Monitor and adjust: Use the Fabric Capacity Metrics app for the first 2-4 weeks

3.2 Sizing reference¶

3.3 Capacity management best practices¶

4. Notebook conversion checklist¶

Use this checklist for every notebook being migrated:

Pre-migration assessment¶

• Classify notebook: ETL (migrate), ML training (keep on DBR), ad-hoc (migrate), DLT (convert to dbt)
• Count lines of code: <200 lines = simple, 200-500 = moderate, >500 = complex (consider rewriting)
• Identify languages: PySpark (migrate), SQL (migrate or convert to dbt), Scala (rewrite in PySpark), R (migrate)
• List dependencies: External libraries, internal %run references, dbutils calls
• Identify data sources: ADLS mounts, Unity Catalog tables, DBFS paths
• Check Photon dependency: Run without Photon on Databricks first; note performance difference

Migration execution¶

• Create Fabric environment with required libraries
• Replace dbutils with mssparkutils (see notebook-migration.md)
• Update file paths from /mnt/ to OneLake paths
• Update table references from catalog.schema.table to Lakehouse tables
• Remove Databricks-specific configs (spark.databricks.*)
• Convert Scala cells to PySpark
• Replace %sql magic with SQL cell type
• Test interactively in Fabric notebook
• Validate output against Databricks (row counts, schema, sample data)

Post-migration¶

• Schedule via Data Pipeline (replace Databricks Workflow)
• Set up monitoring (pipeline alerts, run history)
• Update downstream consumers (Power BI, APIs, other notebooks)
• Run parallel for 2 weeks
• Decommission Databricks notebook (archive to Git, disable job)

5. Common pitfalls and mitigations¶

5.1 Technical pitfalls¶

5.2 Organizational pitfalls¶

5.3 Cost pitfalls¶

6. Operational runbook¶

6.1 Day-to-day operations comparison¶

6.2 Monitoring setup¶

After migration, establish these monitoring practices:

1. Fabric Capacity Metrics app -- install from AppSource; monitors CU consumption
2. Azure Monitor alerts -- set alerts for capacity utilization > 80%
3. Data Pipeline alerts -- configure failure notifications for each pipeline
4. dbt test dashboard -- build Power BI report on store_failures audit tables
5. OneLake storage monitoring -- track storage growth via Azure portal
6. Power BI usage metrics -- monitor report views and refresh patterns

6.3 Rollback plan¶

If a migrated workload does not perform as expected:

1. Immediate: Re-enable the Databricks job/cluster (should not be deleted during parallel run)
2. Repoint consumers: Switch Power BI / downstream APIs back to DBSQL endpoint
3. Investigate: Compare benchmarks, identify performance gap
4. Decide: Optimize Fabric workload, increase capacity, or keep on Databricks
5. Document: Update the migration plan with lessons learned

7. Migration timeline template¶

8. Quick reference: key commands¶

Fabric notebook commands¶

# List files in OneLake
mssparkutils.fs.ls("Files/")

# Get secret from Key Vault
secret = mssparkutils.credentials.getSecret("keyvault-name", "secret-name")

# Get notebook parameter
param = mssparkutils.notebook.getParam("param_name", "default_value")

# Run another notebook
result = mssparkutils.notebook.run("other_notebook", timeout_seconds=300, parameters={"key": "value"})

# Exit with value
mssparkutils.notebook.exit("SUCCESS")

Fabric REST API (common operations)¶

# List workspaces
curl -H "Authorization: Bearer $TOKEN" \
     "https://api.fabric.microsoft.com/v1/workspaces"

# List Lakehouse items
curl -H "Authorization: Bearer $TOKEN" \
     "https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/items?type=Lakehouse"

# Trigger notebook run
curl -X POST -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     "https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/items/{notebook_id}/jobs/instances?jobType=RunNotebook"

Related¶

• Why Fabric over Databricks -- strategic context
• TCO Analysis -- cost modeling
• Feature Mapping -- capability comparison
• Benchmarks -- performance data for capacity planning
• Notebook Migration -- detailed notebook conversion
• Unity Catalog Migration -- governance mapping
• DLT Migration -- pipeline conversion
• ML Migration -- ML workload guidance
• Streaming Migration -- real-time workload guidance
• Parent guide: 5-phase migration
• Fabric documentation: https://learn.microsoft.com/fabric/

Maintainers: csa-inabox core team Source finding: CSA-0083 (HIGH, XL) -- approved via AQ-0010 ballot B6 Last updated: 2026-04-30