NiFi Migration: NiFi to Azure Data Factory + Logic Apps¶

A detailed guide for migrating Apache NiFi data flows to Azure Data Factory, Logic Apps, and supporting Azure services, including processor-by-processor mapping, flow conversion patterns, and worked examples.

Overview¶

Apache NiFi is one of Cloudera's strongest components -- a mature, visual data flow platform with 300+ processors, built-in backpressure, data provenance, and real-time routing. Azure Data Factory is not a 1:1 replacement for NiFi. It is a different tool with a different design philosophy.

NiFi's paradigm: FlowFiles move through processor chains. Each processor transforms, routes, or delivers a single FlowFile (or small batch). Back-pressure, prioritization, and provenance are built into the framework.

ADF's paradigm: Pipelines orchestrate activities. Each activity operates on datasets (tables, files, blobs). Activities run sequentially or in parallel within a pipeline. Data movement is batch-oriented with support for streaming through integration with Event Hubs and Databricks.

Understanding this paradigm difference is essential. Do not attempt to replicate NiFi processor chains as ADF activities 1:1. Instead, redesign the flow to fit ADF's strengths.

Processor-by-processor mapping¶

Data ingestion processors¶

Data transformation processors¶

Routing processors¶

Delivery processors¶

NiFi Registry to ADF Git integration¶

NiFi clustering to ADF Integration Runtime scaling¶

NiFi Record processing to ADF Mapping Data Flows¶

NiFi's Record-oriented processors (ConvertRecord, QueryRecord, LookupRecord, UpdateRecord, PartitionRecord) provide in-flow data transformation. The ADF equivalent is Mapping Data Flows.

Mapping Data Flow equivalents¶

Example: NiFi Record flow to ADF Data Flow¶

NiFi flow:
  ConsumeKafka → ConvertRecord (JSON→Avro) → UpdateRecord (add timestamp)
  → LookupRecord (enrich from DB) → RouteOnAttribute (valid/invalid)
  → PutAzureDataLakeStorage (valid) / PutKafka (invalid → DLQ)

ADF equivalent:
  Event Hubs trigger → Databricks Structured Streaming job:
    - Read from Event Hubs (JSON)
    - Add timestamp column (withColumn)
    - Join with reference table (broadcast join)
    - Filter valid/invalid
    - Write valid to ADLS (Delta)
    - Write invalid to Event Hubs DLQ

NiFi Site-to-Site to ADF Self-Hosted IR¶

NiFi Site-to-Site (S2S) transfers data between NiFi instances across networks. The ADF equivalent is the Self-Hosted Integration Runtime (SHIR).

Worked example: convert NiFi flow to ADF pipeline¶

Original NiFi flow¶

A common NiFi pattern: ingest data from an SFTP server, validate and transform the files, and load into HDFS (now ADLS Gen2).

NiFi Flow: "Daily Vendor File Ingestion"

ListSFTP (vendor-sftp.example.com:/outbound/)
  → FetchSFTP (download file)
    → ValidateRecord (CSV schema validation)
      → [valid] → ConvertRecord (CSV → Parquet)
        → UpdateAttribute (add ingestion_timestamp, source_system)
          → PutHDFS (/data/raw/vendor_files/)
      → [invalid] → PutEmail (alert data-eng@example.com)
        → PutHDFS (/data/quarantine/vendor_files/)

Converted ADF pipeline¶

{
    "name": "daily_vendor_file_ingestion",
    "properties": {
        "activities": [
            {
                "name": "list_vendor_files",
                "type": "GetMetadata",
                "typeProperties": {
                    "dataset": { "referenceName": "sftp_vendor_outbound" },
                    "fieldList": ["childItems"]
                }
            },
            {
                "name": "for_each_file",
                "type": "ForEach",
                "dependsOn": [
                    {
                        "activity": "list_vendor_files",
                        "dependencyConditions": ["Succeeded"]
                    }
                ],
                "typeProperties": {
                    "items": "@activity('list_vendor_files').output.childItems",
                    "isSequential": false,
                    "batchCount": 10,
                    "activities": [
                        {
                            "name": "copy_and_convert",
                            "type": "Copy",
                            "typeProperties": {
                                "source": {
                                    "type": "DelimitedTextSource",
                                    "storeSettings": {
                                        "type": "SftpReadSettings"
                                    }
                                },
                                "sink": {
                                    "type": "ParquetSink",
                                    "storeSettings": {
                                        "type": "AzureBlobFSWriteSettings"
                                    }
                                },
                                "enableStaging": false
                            }
                        }
                    ]
                }
            },
            {
                "name": "validate_data",
                "type": "DatabricksNotebook",
                "dependsOn": [
                    {
                        "activity": "for_each_file",
                        "dependencyConditions": ["Succeeded"]
                    }
                ],
                "typeProperties": {
                    "notebookPath": "/pipelines/vendor_files/validate_and_quarantine",
                    "baseParameters": {
                        "source_path": "abfss://bronze@storage.dfs.core.windows.net/vendor_files/",
                        "quarantine_path": "abfss://quarantine@storage.dfs.core.windows.net/vendor_files/"
                    }
                }
            }
        ],
        "annotations": ["vendor-ingestion", "daily"]
    }
}

Key design decisions in the conversion¶

When to use Logic Apps instead of ADF¶

Some NiFi patterns map better to Logic Apps than ADF:

Migration strategy for complex NiFi environments¶

Phase 1: Inventory (1-2 weeks)¶

1. Export all NiFi flow definitions (XML or JSON via NiFi REST API)
2. Categorize flows by type: batch ingestion, real-time streaming, API integration, file processing
3. Identify processor usage frequency (which processors are used most)
4. Map data lineage from source to sink for each flow

Phase 2: Classify migration approach (1 week)¶

Phase 3: Convert and test (4-8 weeks)¶

• Convert flows by priority (highest business value first)
• Run NiFi and ADF in parallel for validation
• Compare throughput, latency, and data quality
• Decommission NiFi flows only after ADF equivalent is validated

Phase 4: Decommission NiFi (2 weeks)¶

• Redirect all data sources to ADF endpoints
• Archive NiFi flow definitions for reference
• Shut down NiFi cluster
• Reassign NiFi administrators to ADF pipeline management

Next steps¶

1. Walk through the NiFi to ADF Tutorial for a hands-on conversion exercise
2. Review the Feature Mapping for the full component comparison
3. See the Benchmarks for NiFi vs ADF throughput comparisons

Last updated: 2026-04-30 Maintainers: CSA-in-a-Box core team