Skip to content
Architecture Decision Records — the durable rationale behind every platform choice from ADF over Airflow to APIM as integration fabric

Home > Docs > Architecture Decision Records

Architecture Decision Records (ADRs)

Comparative positioning note

This document is written from the perspective of Microsoft Azure, Cloud Scale Analytics, and CSA Loom. Any description of third-party or competing products, services, pricing, or capabilities is derived from publicly available documentation and sources believed accurate at the time of writing, and is provided for general comparison only. We do not claim expertise in, or authority over, any non-Microsoft product or service; the respective vendor's official documentation is the authoritative source for their offerings, which may change over time. Nothing here is intended to disparage any vendor — where a competing product has genuine advantages, we aim to note them honestly. Verify all third-party details against the vendor's current official documentation before making decisions.

Architects, platform engineers, auditors, federal customers preparing ATO packages

Note

Quick Summary: This directory captures the why behind the core technology choices in CSA-in-a-Box. Each record follows the MADR format. Records are immutable once accepted — revisions happen by adding a superseding ADR with an incremented number.

Index

# Title Status Date One-line summary
0001 ADF (+ dbt) over Airflow as primary orchestration accepted 2026-04-19 Managed Gov PaaS orchestrator with SQL-native transforms and Purview lineage.
0002 Azure Databricks over open-source Spark-on-AKS for heavy compute accepted 2026-04-19 Managed Spark + Unity Catalog + Photon in Gov; OSS Spark on AKS is too operationally expensive.
0003 Delta Lake over Iceberg and Parquet as canonical table format accepted 2026-04-19 Delta is Databricks- and Fabric-OneLake-native with ACID MERGE and Purview lineage.
0004 Bicep over Terraform as primary IaC (for now; Terraform path planned) accepted 2026-04-19 Day-one Azure API coverage, no state-file custody, aligned to Enterprise-Scale Landing Zone.
0005 Event Hubs over open-source Kafka for streaming ingestion accepted 2026-04-19 Managed PaaS broker with Kafka-protocol endpoint, Capture to Bronze, Gov-GA.
0006 Microsoft Purview over Apache Atlas for data catalog and lineage accepted 2026-04-19 Gov-GA catalog with MIP label propagation and native Azure scanners.
0007 Azure OpenAI over self-hosted LLM for AI integration accepted 2026-04-19 FedRAMP High inference endpoint with Private Endpoints; self-hosted fallback remains open.
0008 dbt Core over dbt Cloud for transformations accepted 2026-04-19 Open-source CLI keeps metadata inside the tenant boundary; no SaaS FedRAMP surface to clear.
0009 SQLite (portal dev) → Postgres (portal prod) phased database strategy accepted 2026-04-19 Zero-install dev loop; managed Postgres PaaS in Gov for production durability.
0010 Microsoft Fabric as strategic target; current build as Fabric-parity on Azure PaaS accepted 2026-04-19 Every primitive (Delta, Purview, dbt, Spark) maps forward into Fabric when Gov GA lands.
0011 Multi-cloud scope: OneLake shortcuts + Purview scans only; defer federated compute accepted 2026-04-20 Honest scope — ships governance story for S3/GCS/Snowflake/BigQuery/Redshift; defers cross-cloud compute.
0012 Data-mesh federation model: contract-driven, Purview-governed, portal-surfaced accepted 2026-04-20 Contract-first in-monorepo mesh — contract.yaml → CI validates → Purview registers → marketplace surfaces; per-domain CODEOWNERS.
0013 dbt Core as the canonical transformation layer accepted 2026-04-20 Deduplicates Bronze → Silver → Gold logic — dbt owns medallion transforms; Spark notebooks are deprecated for that path and reserved for exploration / provisioning / ML.
0014 MSAL Backend-for-Frontend (BFF) auth pattern accepted 2026-04-20 Phased CSA-0020 remediation — Phase 1 strict CSP + Trusted Types on the SPA; Phase 2 server-side Auth Code + PKCE flow with an httpOnly csa_sid session cookie. Tokens never reach the browser.
0015 Portal persistence: StoreBackend protocol + SQLite (dev) + Postgres Flexible Server (prod) accepted 2026-04-20 CSA-0046 implementation — dual-backend Protocol, managed-identity AAD tokens for Azure Database for PostgreSQL Flexible Server, Alembic migrations, SQLite kept as the zero-install dev loop.
0016 Async StoreBackend canonical; sync layer transitional accepted 2026-04-20 CSA-0046 follow-on — AsyncStoreBackend Protocol + AsyncSqliteStore (aiosqlite) + AsyncPostgresStore (asyncpg + SQLAlchemy AsyncEngine); FastAPI routers go async def via Depends; migration CLI ships; sync layer kept one release for backward compat.
0017 RAG pipeline service-layer extraction (CSA-0133) accepted 2026-04-20 Split the 1,285-line pipeline.py god-class into six submodules behind a RAGService facade; legacy pipeline module is preserved as a compat shim for one release.
0018 Fabric Real-Time Intelligence adapter (pre-GA, env-gated) accepted 2026-04-20 CSA-0137 follow-on — ship FabricRTISource today behind FABRIC_RTI_ENABLED; raise-with-pointer when the flag is unset so Gov tenants fail loudly until RTI Gov-GA lands.
0019 BFF reverse-proxy + HMAC-sealed MSAL token cache accepted 2026-04-20 CSA-0020 Phase 3 — mount /api/* proxy behind BFF_PROXY_ENABLED; persist MSAL SerializableTokenCache to Redis with HMAC sealing so a Redis compromise is tamper-evident. Completes AQ-0012's long-term column — tokens never reach the browser.
0020 Portal observability (OTel + Prometheus) and per-principal rate limiting accepted 2026-04-20 CSA-0042 / CSA-0061 / CSA-0030 — OpenTelemetry with OTLP exporter + W3C trace-context, Prometheus /metrics on a private registry, per-principal sliding-window rate limiter on every write endpoint. All three feature-flagged and lazy-imported so the portal still boots without the optional extras.
0021 Two rate limiters are intentional, not duplicates accepted 2026-04-23 The portal write-path limiter (per-principal, sliding-window, observability) and the AI router limiter (per-IP, fixed-window, abuse-defense) protect orthogonal failure modes; do not collapse them.
0022 Copilot surfaces vs. docs-site widget are intentional, not duplicates accepted 2026-04-23 The Azure Function (func-csa-inabox-copilot-fg) is the production chat backend; the docs-site widget (docs/javascripts/copilot-chat.js) is a thin client that talks to it. Two artifacts, one service.
0023 release-please PRs auto-pass required status checks accepted 2026-04-27 GITHUB_TOKEN-created PRs don't trigger downstream workflows, leaving release PRs permanently BLOCKED. The release-please workflow itself posts success commit statuses for each required check on the PR head SHA, gated on a strict allow-list of three version-metadata files.

Format

All records follow MADR 3.x. Each ADR has frontmatter (status, date, deciders, consulted, informed) and these sections:

  • Context and Problem Statement
  • Decision Drivers
  • Considered Options
  • Decision Outcome
  • Consequences (positive and negative)
  • Pros and Cons of the Options
  • Validation (how we'll know the decision was right)
  • References (decision trees, concrete code, compliance-control mappings)

Status lifecycle

proposed  →  accepted  →  deprecated
                   │
                   └──────→  superseded by NNNN
  • proposed — open for comment on a PR.
  • accepted — merged; the decision is in effect.
  • deprecated — no longer in effect; no replacement chosen yet.
  • superseded by NNNN — replaced by a newer ADR. The superseding ADR states what changed and why; the superseded ADR stays on disk as history.

ADRs are immutable once accepted. Corrections are made by authoring a new ADR that supersedes the old one. Typos and broken links may be fixed without superseding.

Authoring a new ADR

  1. Copy 0001-adf-dbt-over-airflow.md as a template.
  2. Increment the number (4-digit, zero-padded).
  3. Slugify the title (NNNN-short-slug.md).
  4. Keep the frontmatter + section order identical to 0001.
  5. Cite at least one concrete artifact in the repo and at least one compliance-control mapping from governance/compliance/*.yaml when the decision maps to a control family.
  6. Link forward to any relevant decision tree under docs/decisions/.
  7. Open a PR. Reviewers from security, governance, and dev-loop are expected. Status stays proposed until the PR merges.

Cross-references

Upstream references