CDP Data Engineering Migration: CDE, CML, CDW to Azure¶

A detailed guide for migrating Cloudera Data Platform (CDP) components -- Data Engineering (CDE), Machine Learning (CML), and Data Warehouse (CDW) -- to Databricks, Azure ML, and Fabric.

Overview¶

CDP represents Cloudera's modern platform, built on Kubernetes and available in Private Cloud and Public Cloud editions. Organizations on CDP are in a different position than CDH shops: the infrastructure is more modern, the APIs are cleaner, and the migration paths are more direct. However, the core economic and strategic arguments for Azure migration still apply -- CDP licensing costs are rising, and Azure-native services provide broader capabilities at lower total cost.

This guide covers three CDP-specific components that require dedicated migration approaches beyond the core HDFS/Hive/Spark/Oozie playbook.

1. CDE Virtual Clusters to Databricks Workspaces¶

Architecture comparison¶

CDE job definition to Databricks job¶

# CDE job definition (CDE CLI format)
name: daily-sales-etl
type: spark
application-file: s3a://cde-resources/jobs/sales_etl.py
driver-cores: 2
driver-memory: 4g
executor-cores: 4
executor-memory: 8g
num-executors: 10
conf:
    spark.sql.shuffle.partitions: 400
schedule:
    enabled: true
    cron-expression: "0 2 * * *"
    start: "2025-01-01T00:00:00Z"

// Databricks Job definition (Jobs API 2.1 format)
{
    "name": "daily-sales-etl",
    "tasks": [
        {
            "task_key": "run_sales_etl",
            "spark_python_task": {
                "python_file": "dbfs:/jobs/sales_etl.py"
            },
            "new_cluster": {
                "spark_version": "15.4.x-scala2.12",
                "node_type_id": "Standard_DS4_v2",
                "driver_node_type_id": "Standard_DS4_v2",
                "autoscale": {
                    "min_workers": 4,
                    "max_workers": 10
                },
                "spark_conf": {
                    "spark.sql.shuffle.partitions": "400"
                }
            }
        }
    ],
    "schedule": {
        "quartz_cron_expression": "0 0 2 * * ?",
        "timezone_id": "UTC"
    }
}

Key differences in job management¶

2. CDE Airflow to Databricks Workflows¶

CDE includes Apache Airflow for orchestration. This is one of the more straightforward migrations because Databricks Workflows provides a native alternative, and ADF provides a broader orchestration layer.

Migration targets¶

Airflow DAG to Databricks Workflow conversion¶

# CDE Airflow DAG (before)
from airflow import DAG
from airflow.providers.cde.operators.cde_job import CDEJobRunOperator
from datetime import datetime

dag = DAG(
    'daily_sales_pipeline',
    schedule_interval='0 2 * * *',
    start_date=datetime(2025, 1, 1),
    catchup=False
)

extract = CDEJobRunOperator(
    task_id='extract_orders',
    job_name='extract_orders_job',
    dag=dag
)

transform = CDEJobRunOperator(
    task_id='transform_sales',
    job_name='transform_sales_job',
    dag=dag
)

load = CDEJobRunOperator(
    task_id='load_warehouse',
    job_name='load_warehouse_job',
    dag=dag
)

extract >> transform >> load

// Databricks Workflow (after)
{
    "name": "daily_sales_pipeline",
    "tasks": [
        {
            "task_key": "extract_orders",
            "spark_python_task": {
                "python_file": "dbfs:/jobs/extract_orders.py"
            },
            "new_cluster": {
                "spark_version": "15.4.x-scala2.12",
                "node_type_id": "Standard_DS4_v2",
                "autoscale": { "min_workers": 2, "max_workers": 8 }
            }
        },
        {
            "task_key": "transform_sales",
            "depends_on": [{ "task_key": "extract_orders" }],
            "spark_python_task": {
                "python_file": "dbfs:/jobs/transform_sales.py"
            },
            "new_cluster": {
                "spark_version": "15.4.x-scala2.12",
                "node_type_id": "Standard_DS4_v2",
                "autoscale": { "min_workers": 2, "max_workers": 10 }
            }
        },
        {
            "task_key": "load_warehouse",
            "depends_on": [{ "task_key": "transform_sales" }],
            "spark_python_task": {
                "python_file": "dbfs:/jobs/load_warehouse.py"
            },
            "new_cluster": {
                "spark_version": "15.4.x-scala2.12",
                "node_type_id": "Standard_DS4_v2",
                "num_workers": 4
            }
        }
    ],
    "schedule": {
        "quartz_cron_expression": "0 0 2 * * ?",
        "timezone_id": "UTC"
    },
    "email_notifications": {
        "on_failure": ["data-eng@example.com"]
    }
}

Airflow operator mapping¶

3. CML to Azure ML + Databricks ML¶

Architecture comparison¶

Migration decision: Azure ML vs Databricks ML¶

Recommendation: If your data engineering runs on Databricks, use Databricks ML for tight integration. If you need Responsible AI dashboards, LLM fine-tuning with Azure AI Foundry, or complex multi-step ML pipelines, use Azure ML. Many organizations use both.

CML model migration script¶

# Step 1: Export model from CML (on CML cluster)
import cmlapi
import mlflow
import os

# Connect to CML
client = cmlapi.default_client()

# Download model artifacts
mlflow.set_tracking_uri("https://cml-workspace.example.com/mlflow")
model_uri = "models:/sales_forecast/Production"
local_path = mlflow.artifacts.download_artifacts(model_uri, dst_path="/tmp/models")

# Package model for transfer
# azcopy copy /tmp/models abfss://ml@storage.dfs.core.windows.net/models/sales_forecast/

# Step 2: Register model on Databricks (on Databricks)
import mlflow

mlflow.set_registry_uri("databricks-uc")

# Load model from ADLS
model_path = "abfss://ml@storage.dfs.core.windows.net/models/sales_forecast/"

# Register in Unity Catalog
mlflow.register_model(
    f"runs:/{run_id}/model",  # Or from local path
    "ml_catalog.models.sales_forecast"
)

# Step 3: Deploy as serving endpoint
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
w.serving_endpoints.create(
    name="sales-forecast-endpoint",
    config={
        "served_models": [{
            "model_name": "ml_catalog.models.sales_forecast",
            "model_version": "1",
            "workload_size": "Small",
            "scale_to_zero_enabled": True
        }]
    }
)

4. CDP Data Warehouse (CDW) to Databricks SQL + Fabric¶

CDW architecture to Azure mapping¶

CDW to Fabric SQL endpoint (alternative)¶

For organizations adopting Microsoft Fabric, CDW workloads can also target Fabric SQL endpoint:

When to choose Fabric vs Databricks SQL:

5. Migration order for CDP components¶

Recommended sequence¶

flowchart TD
    A[1. Migrate CDW Hive VW<br/>to Databricks SQL] --> B[2. Migrate CDW Impala VW<br/>to Databricks SQL]
    B --> C[3. Migrate CDE Spark Jobs<br/>to Databricks Jobs]
    C --> D[4. Migrate CDE Airflow<br/>to Databricks Workflows / ADF]
    D --> E[5. Migrate CML Experiments<br/>to Databricks ML / Azure ML]
    E --> F[6. Migrate CML Model Serving<br/>to Databricks Serving / Azure ML Endpoints]
    F --> G[7. Decommission CDP]

Rationale for this order¶

1. CDW first: SQL workloads are the easiest to validate (row counts, checksums) and have the highest business visibility (dashboards break immediately if wrong).
2. CDE Spark next: Spark code is highly portable; the main changes are path updates and YARN config removal.
3. CDE Airflow after Spark: Orchestration migration depends on the compute tasks being available on the target platform.
4. CML last: ML workloads are often the most self-contained and can continue running on CML while other components migrate.

6. CDP vs CDH migration differences¶

If you are migrating from CDP rather than CDH, several things are easier:

CDP migration checklist¶

• Inventory CDP components in use (CDW, CDE, CML, Data Hub)
• Export CDE job definitions (CDE CLI: cde job list --output json)
• Export CDE Airflow DAGs (from Git repo or CDE filesystem)
• Export CML experiment metadata (MLflow tracking export)
• Export CML model artifacts (MLflow model download)
• Document CDW virtual warehouse configurations (size, auto-scaling, user groups)
• Map CDE virtual cluster configs to Databricks cluster policies
• Convert CDE Spark jobs to Databricks Job definitions
• Convert Airflow DAGs to Databricks Workflows or ADF pipelines
• Register CML models in Databricks Model Registry or Azure ML
• Deploy model serving endpoints on Databricks or Azure ML
• Validate SQL workload results (CDW vs Databricks SQL output comparison)
• Performance benchmark (CDW query latency vs Databricks SQL)
• Update BI tool connections (JDBC/ODBC from CDW to Databricks SQL Warehouse)
• Train users on Databricks SQL Editor (replacing Hue)
• Parallel run for 2+ weeks (both CDP and Azure processing same workloads)
• Decommission CDP virtual clusters after validation

Next steps¶

1. Review the Migration Playbook for the full HDFS/Hive/Spark/Oozie migration
2. See the Impala Migration Guide for CDW Impala-specific conversion
3. Review the Benchmarks for CDP vs Azure performance data
4. Read the Best Practices for cluster-by-cluster migration strategy

Last updated: 2026-04-30 Maintainers: CSA-in-a-Box core team