Skip to content
CSA Loom — the Microsoft Fabric experience for Azure tenants where Fabric isn't yet available: lakehouses, warehouses, notebooks, semantic models, Activator rules, Data Agents, across Commercial, GCC, GCC-High, and DoD IL5

Runbook — Deploy failure

This is the short triage card. The full failure taxonomy — eight classes, the ARM codes that map to each, what retries, what the platform can fix itself, and which of this repo's own error messages are currently untrue — is Failure recovery. Start here for a known symptom; go there for anything not in the table below.

On Azure Government, none of this is automated (#3017). The classifier and the bounded-retry harness are wired into the two Commercial deploy workflows only; deploy-fiab-gcch.yml does not invoke scripts/ci/deploy-retry.mjs, and deploy-fiab-il5.yml has never run. The class column below is still the right way to triage a Gov failure — you just do it by hand, and no deploy-failure.json is produced.

Symptom

azd up or Loom Setup Wizard deploy fails with a non-Succeeded ARM deployment state. Console URL doesn't return 200.

A deploy run that shows green but deployed nothing: "Stood down: estate paused"

If the run summary says Stood down: estate paused, the deploy was skipped on purpose: Estate Power has the estate PAUSING or PAUSED, and applying would start ADX and rewrite the paused Container Apps (#5111). The run is green, but it is not a deployed SHA and the staleness check does not count it. To deploy: resume the estate in Admin > Scale by SKU > Estate power, then re-run; or dispatch the lane with override_estate_pause: true if you accept waking paused resources. A scheduled run cannot override. If the summary says Stood down: estate state could not be established on a scheduled run (fail-closed), the gate could not read the state (or its job failed or timed out in azure/login) on a scheduled run, so the lane skipped rather than risk waking a paused estate; the next scheduled run retries. Check the Estate pause gate job's log and the CSA_LOOM_CONSOLE_URL / _AUDIENCE repo variables. A manual dispatch with an unreadable state prints ::warning::estate-pause-state is UNKNOWN and proceeds, so a paused estate can still be woken by a dispatch whose gate was unreadable. Details: estate-idle-auto-pause.

Diagnosis

Start here. The top-level ARM error on a failed subscription deploy is content-free — it is always DeploymentFailed: At least one resource deployment operation failed — and az deployment sub create writes ~200 lines of bicep linter warnings to stderr, so the real cause is in neither place. It sits two or more levels down, inside the failed deployment operations. Drill to it:

node scripts/ci/deploy-arm-errors.mjs --name <deployment-name> --scope sub [--subscription <sub>]

It walks the failed operations recursively (following nested Microsoft.Resources/deployments targets) and prints every leaf code: message with the resource it belongs to. It has three outcomes and only one is a pass: found, none (ARM answered and nothing failed), and unreadable (ARM did not answer — nothing is asserted). No workflow invokes this script directly; six deploy workflows reach the same walk through scripts/ci/deploy-retry.mjs --arm-deployment, which runs it on failure and feeds the leaves to the classifier, so a live run's annotation names the cause instead of reporting "could not classify" (#3039).

--json output is UNREDACTED — do not paste it into an issue (#3829)

The default (human-readable) render above is redacted: subscription and tenant ids collapse to <redacted> and any GUID — including the object id in a flexibleServers/administrators leaf name, <server>/<objectId> — becomes <guid>. Paste it freely.

--json is deliberately raw. It exists so the operator keeps the full ARM ids that some remediations in the table below actually need (the RoleAssignmentExists row's az role assignment delete --ids <id> is the one that matters). That means its output carries subscription ids, tenant ids, full resource ids and Entra object ids. This repo is public — treat --json output as local-only. If you need to attach evidence to a public issue, attach the default render, or the deploy-failure.json artifact, which is redacted at its own boundary.

What the lane redacts for you, and the ONE thing it does not (#3829)

Everything the deploy lane composes is redacted at the boundary it leaves through, and there is exactly one boundary per surface:

surface boundary
the auto-filed notice issue's title and body deploy-notify-failure.mjs — notifyFailure()
that script's stdout / stderr formatStdout() / formatStderr()
::error:: / ::warning:: / ::notice:: annotations deploy-retry.mjs — formatAnnotation()
deploy-retry.mjs's own run-log lines formatStderr()
the deploy-failure.json artifact one redact() over the whole serialization
deploy-arm-errors.mjs's default render, and its usage errors renderLeaves(), formatStdout() / formatStderr()

So a workflow or resource name can read back as deploy-fiab-<guid> or uami-loom-directlake<guid>. That is the redactor substituting in place, not a corrupted name.

The exception, stated plainly: the deployed command's OWN output is NOT redacted. deploy-retry.mjs streams the child's stdout live and echoes its stderr back verbatim — that is deliberate, because rewriting a command's own output would make the wrapper's log disagree with the command's and send an investigation somewhere the evidence does not support (R7). If az itself prints a subscription, tenant or object id, that id is in the public run log — with or without this harness, exactly as it would be under a bare az deployment sub create. Treat a failed deploy's raw run log the same way you treat --json: read it, do not paste it.

Two narrower residuals in the redactor itself, disclosed rather than implied away (scripts/ci/_azure-redact.mjs carries the measurements): an undashed 32-hex run is left alone on purpose, because ARM prints the blocking role-assignment id that way and the #3439 auto-converger reads it back; and a GUID immediately followed by a hex character (<guid>abc) is not matched.

By hand, the same walk is:

# List recent sub-scoped deployments
az deployment sub list \
  --query "[?starts_with(name, 'csa-loom')] | [?properties.provisioningState != 'Succeeded'] | [].{name:name,state:properties.provisioningState,error:properties.error.message}" \
  -o table

# The top-level error — expect it to say nothing useful
az deployment sub show --name <deployment-name> --query "properties.error"

# The operations. Repeat at group scope for every nested deployment that failed:
az deployment operation sub   list --name <deployment-name> -o json
az deployment operation group list -g <rg> --name <nested-deployment-name> -o json

Common failure modes:

Symptom Class Likely cause Fix
Quota exceeded for Databricks Premium quota Region quota Request quota via Azure portal → Subscriptions → Usage + quotas; or pick a different region. Do not retry — it is deterministic
QuotaExceeded: standardDDSv5Family Cores during an image build quota The region's VM-family quota is fully used, so the ACR-task agent pool cannot get cores Read the quota inventory first to see who holds the cores, then free them or raise the quota. Do not assume a leaked pool
RoleAssignmentExists config The grant is already in place under a different assignment NAME. ARM enforces uniqueness on the (scope, principalId, roleDefinitionId) triple, not on the name, so a name-seed change (e.g. the Website Contributor role id correction …706ee → …84772) or a grant created out-of-band by az role assignment create (random GUID) blocks the template's create forever Converge the name: az role assignment delete --ids <the id printed in the error message>, then re-run — the template recreates it under its deterministic name. skip_role_grants=true suppresses the symptom and leaves the estate un-reconciled; it is a workaround, not a fix
BadRequest: A virtual network cannot be linked to multiple zones with overlapping namespaces config The hub VNet is already linked to a different Private DNS zone of the same namespace — usually a leftover zone from a superseded design in another resource group or subscription. Azure permits one link per namespace per VNet, so the admin plane's own link can never be created Do not delete the existing link first — the A records live in that zone and unlinking takes the service dark. Run node scripts/csa-loom/migrate-private-dns-zone-owner.mjs … (dry-run by default; --apply to execute). It dual-registers under a new zone-group config name, moves the links, removes only the config that still points at the stale zone, refuses any removal that would empty the group, and re-verifies the group and the record before deleting the stale zone — see Private DNS zone-owner migration. node scripts/csa-loom/preflight-private-dns-links.mjs detects it before the deploy, and deploy-fiab-commercial.yml runs that preflight automatically
InvalidTemplateDeployment on Container Apps in IL4 config Container Apps not at IL4 Set containerPlatform = 'aks' in .bicepparam
Forbidden on Key Vault Premium HSM permission Lacks Microsoft.KeyVault/managedHsms/write Request elevated role
VnetAddressRangeInUse config CIDR conflict Pick a different CIDR; update hubVnetCidr. The DLZ spoke CIDR (10.100.0.0/16) is not settable from the root template today
PrivateDnsZoneAlreadyExists config Re-deploy after a previous failure az network private-dns zone delete the conflicts, or deploy into a clean resource group. There is no existingPrivateDnsZones parameter — an earlier version of this table said there was; it has never existed in main.bicep
EnterpriseTenantAlreadyExists config A Purview account already exists in the tenant — only one is allowed Adopt it: EXISTING_PURVIEW=<name> (+ _RG, _SUB). No enable-flag override is needed — provisionPurview is already false for an adopt decision. See Brownfield
ManagedIdentityRoleAssignmentDelay / PrincipalNotFound eventual-consistency RBAC replication lag Wait 5 min; re-run
MANIFEST_UNKNOWN / image pull failure on a Container App config Phase 1 ran with deployAppsEnabled=true against an empty ACR Re-run phase 1 with false, then the image phase. See Greenfield
ResourceGroupNotFound early in full-app-deploy-commercial config The workflow's region input does not match the estate's region Pass -f region=<your-region>
ResourceGroupNotFound on a multi-sub DLZ config A sub-scoped deploy cannot create an RG in a remote subscription bash scripts/csa-loom/bootstrap-dlz-rgs.sh first
ContainerAppOperationInProgress transient A previous roll is still settling Wait and retry; serialize rolls, never cancel an in-flight roll of the same SHA

ACR agent-pool quota failures and the reaper

Signature (Gov console roll, 2026-10-23, run 40021972356): az acr build fails with The status for Agent Pool loombuild2 is Error ... exceeding approved standardDDSv5Family Cores quota ... Current Limit: 100, Current Usage: 100, Additional Required: 8 (an S3 pool instance is 8 vCPUs; S1 is 2, S2 is 4).

What keeps pools from leaking. Two layers, both fail closed:

  • Each build lane ends with an if: always() step that waits up to 5 minutes for the pool queue to drain, scales to 0 and reads the count back (scripts/ci/acr-build-pool.mjs release --wait-seconds 300). It leaves a pool alone while another run is queued or active on it, and says so.
  • acr-agentpool-reaper.yml runs hourly in Commercial and GCC-High and scales any idle pool (queue 0, no active run, idle 30+ minutes, Succeeded) to 0, reading the count back. Dispatch it with dry_run=true (default) to report only.

Diagnose before you blame the pool. The reaper also runs scripts/ci/acr-quota-inventory.mjs per registry: usage vs limit per family, the visible consumers (VMs, VMSS, AKS node pools) with cores, the pools with tier cores, and the unattributed remainder. A read that fails prints UNKNOWN and fails the job; it is never shown as zero. Measured on 2026-10-23, the Gov reaper had read loombuild2 at count 0 about 90 minutes before the failure, so the 100 cores were held by something else; a leaked pool is only the answer when the inventory says so.

Fallback: request a quota increase. If the inventory shows the cores are legitimately in use, raise the limit: Azure portal, Subscriptions, Usage + quotas, filter the region and family (for the incident: usgovvirginia, standardDDSv5Family), request a limit above used + 8. In Azure Government use the Government portal. Then re-run the failed roll.

When ACR Tasks itself is out of capacity (#5102)

Backend-capacity signature. The inventory reads our subscription's standardDDSv5Family as 0/100 (or far from the limit), yet az acr build fails with QuotaExceeded ... standardDDSv5Family ... 100/100 while deploying the ACR service's OWN backing VMSS, and a fresh pool name (loombuild3) fails the same way. The exhausted quota is in the ACR service's subscription, so raising ours does nothing; only Microsoft can restore ACR Tasks agent-pool capacity in the region (operator decision 2026-10-24: no support case, build on the GitHub runner instead). bypass_agent_pool is not an escape: the default 8 GB agent OOMs on the Next.js build (#2706).

The GitHub runner is the Gov default (2026-10-24). gov-console-roll.yml and gov-build-images.yml build with docker buildx on the GitHub-hosted runner and push to the Gov ACR. Same Dockerfile, context, build args and tags, inside the same ACR firewall lease and after the data-plane gate; the verified-digest check, Trivy CRITICAL gate and cosign sign/verify are unchanged. No pool is acquired or released. Operator decision: no Microsoft support case; gh-runner stays the default.

  • Precedence: dispatch input builder, then the repo variable (GOV_CONSOLE_BUILDER for the console roll, GOV_IMAGES_BUILDER for gov-build-images), then gh-runner. Any other value fails the build step.
  • Pool path on demand: gh workflow run gov-console-roll.yml -f builder=acr-pool.

Layer cache. gh-runner builds use a registry cache at <acr>.azurecr.us/buildcache/<image>:cache (mode=max,image-manifest=true,oci-mediatypes=true, docker-container buildx driver), via scripts/ci/gh-runner-buildx-push.sh. A cache EXPORT failure prints a ::warning:: and the build is re-run once without --cache-to; a push failure still fails the step. A missing cache is a cold build, not an error. The script fails early if az acr login left no docker credentials for the registry.

Test whether a small pool tier fits. gh workflow run gov-console-roll.yml -f builder=acr-pool -f agent_pool_tier=S1 -f agent_pool_name=loombuilds1. The tier is used only when the run creates the pool. Expect the Next.js build to OOM below S3; read the result as a capacity measurement.

Clean up dead pools. gh workflow run acr-agentpool-reaper.yml -f dry_run=false -f prune_failed=true deletes pools that are Failed AND count 0 AND queue 0, then reads the pool list back to confirm. A pool whose queue cannot be read (the wedged InternalServerError shape) is reported and left alone. With dry_run=true it only reports. Dispatch only, never scheduled; applies to both boundaries in the matrix.

Remediation

  1. Triage — note the failed module + Azure error code
  2. Classify — match the code to a class in Failure recovery. Retrying a quota or config failure cannot help
  3. Apply fix — per table above
  4. Resume — re-run the same az deployment sub create with the fix added. The deployment is incremental and idempotent, so it resumes rather than restarts. Two exceptions: a partially-created Private DNS zone or a taken global name must be removed first, and reconciling an existing hub through the workflow needs allow_existing_hub=true and a region that matches the estate (a mismatched region is refused — #3029). keep_resources defaults to true; a teardown also needs confirm_teardown_rg=rg-csa-loom-admin-<region> (#3028)
  5. Verify — curl <console-url>/api/health returns 200, then open the Console and confirm sign-in and /admin/readiness

Private DNS zone-owner migration (#3039 / #3046)

scripts/csa-loom/migrate-private-dns-zone-owner.mjs adopts a privatelink.* namespace onto the zone that should own it, without taking the service dark. It is dry-run by default; --apply executes.

node scripts/csa-loom/migrate-private-dns-zone-owner.mjs \
  --namespace privatelink.azuredatabricks.net \
  --keep-zone-rg  rg-csa-loom-admin-<region>       --keep-zone-subscription  <sub> \
  --stale-zone-rg rg-csa-loom-dlz-default-<region> --stale-zone-subscription <sub> \
  [--pe-resource-group <rg> …]      # extra scopes to search for private endpoints

The plan is 10 steps; the two terminal verifications and the removal guard are the part to understand, because their absence caused a live outage.

# step what it does
1 ensure-keep-zone creates the surviving zone if it is missing
2 dual-register adds a config pointing at the keep zone, under a name nothing else in the group owns (<ns>-loom-keep). The record is then in both zones
3 verify-record-in-keep-zone hard gate — refuses to unlink anything until the record is provably in the keep zone
4–5 unlink-stale → link-keep moves each VNet link. A resolution gap of seconds: Azure permits one link per namespace per VNet
6 verify-links-on-keep-zone hard gate — every VNet the stale zone served resolves on the keep zone
7 single-register removes only the config that still points at the stale zone, selected by the name read from the estate
8 verify-zone-group-bound-to-keep terminal gate — the group still carries a config on the keep zone
9 verify-record-after-single-register terminal gate — the A records are still in the keep zone
10 delete-stale-zone the stale zone goes last

Why steps 2, 7, 8 and 9 look the way they do (#3046)

az network private-endpoint dns-zone-group add|remove --zone-name selects the config by its name, never by the zone id (measured from the az CLI source: _add.py / _remove.py, SubresourceSelector). add replaces a config whose name matches and only appends when it does not.

The first cut of this script passed <namespace with dots→dashes> for both. That string is byte-identical to the config name the bicep creates (landing-zone/databricks.bicep:135), so step 2 replaced the stale config instead of appending — dual-registration never happened — and step 7 then removed the only remaining config, leaving the group empty, which deregisters the endpoint's A record from every zone. The service went dark and the script exited 0, because nothing ran after step 7.

Now: step 2 uses a name nothing owns and refuses if that name is taken; step 7 refuses to remove a config that no longer points at the stale zone, that it cannot re-read, or whose removal would leave the group empty or with no config on the keep zone; steps 8 and 9 re-check the outcome before anything is deleted. Every destructive step re-reads its subject immediately before acting, so a re-run converges rather than re-applying.

What the verifications do NOT prove. They read ARM — the record set exists, the links exist, the group is bound. They do not prove the name resolves from inside the hub VNet; a hosted runner is not in the VNet. For that, run a lookup from the in-VNet runner (loom-aca-runner-smoke.yml) or over the admin P2S VPN.

Empty-zone shadowing. Step 1 creates the keep zone but does not link it, and step 5 links it only after step 3 has proved the record is there — so the migration never leaves an empty zone linked to a VNet. That matters: a linked but empty privatelink.* zone shadows public resolution for the whole namespace and returns NXDOMAIN rather than falling through (the Gov Purview incident). If you stop the migration part-way, an unlinked empty keep zone is harmless; a linked empty one is not.

Prevention

  • Run bicep what-if (az deployment sub create … --what-if) before every deploy
  • Pre-check quotas: az vm list-usage -l <region>
  • Pre-check the deploy identity's roles and the six resource-provider registrations — see Failure recovery → preflight
  • On a brownfield estate, validate adoption candidates (HNS, SKU, deployments, metastore assignment) before the deploy — see Brownfield → what is validated
  • Keep .bicepparam files under Git review

Escalation

If the error doesn't match the table above: - Open GitHub issue with label csa-loom + csa-bug. Attach the redacted evidence: the default (non---json) deploy-arm-errors.mjs render, or the deploy-failure.json artifact. Do not paste raw az deployment operation … -o json or --json output, and do not paste the run log's "full captured stderr" block — this repo is public, that output is the command's own and is therefore unredacted, and it carries subscription, tenant and Entra object ids (#3829) - An unclassifiable failure is also a gap in the taxonomy — say so in the issue so the class gets added - Internal Microsoft: #csa-loom-build Teams channel