AI Readiness Assessment Framework¶

Organizational maturity model for AI adoption covering data readiness, infrastructure, skills, governance, and use case prioritization. Provides a structured assessment across eight dimensions, each scored 1--5, with an aggregate maturity level that maps directly to Azure services and CSA-in-a-Box components.

Purpose¶

Enterprises routinely over-invest in model experimentation and under-invest in the surrounding platform: data quality, MLOps, governance, and culture. The result is a graveyard of Jupyter notebooks that never reach production. This framework gives leadership and architecture teams a repeatable, score-based tool to answer three questions before committing budget:

1. Where are we today? -- current maturity across eight dimensions.
2. What must we fix first? -- the weakest dimensions that will block production AI.
3. What Azure services and CSA-in-a-Box components close the gap? -- concrete platform mapping.

The framework is intentionally opinionated toward Azure PaaS and the CSA-in-a-Box reference architecture. Organizations using other clouds or on-premises tooling can adapt the dimension definitions while retaining the scoring model.

Maturity Model¶

Five levels describe an organization's overall AI posture. The aggregate level is derived from the eight dimension scores described in the next section.

Level 1 -- Ad Hoc¶

No formal AI strategy exists. Individual teams run isolated experiments in personal notebooks or shadow-IT compute. Data is accessed on an ad-hoc basis, often through manual exports. There is no model registry, no deployment pipeline, and no governance review. Success depends entirely on individual heroics, and learnings are not shared across teams.

Typical signals: Jupyter notebooks on local machines, CSV files emailed between teams, no budget line item for ML infrastructure, models are "deployed" by handing a pickle file to an engineer.

Level 2 -- Aware¶

The organization has recognized AI as strategically important. An AI strategy or charter exists, even if it is aspirational. Pilot projects have been identified and scoped. A dedicated data science team -- or at least a cross-functional working group -- is forming. Initial tooling decisions are being made, but end-to-end pipelines are not yet in place. Data access is still largely manual but cataloging has started.

Typical signals: executive sponsor identified, first pilot in a sandbox environment, initial Databricks or Azure ML workspace provisioned, data catalog effort underway.

Level 3 -- Operational¶

Production ML models exist and serve business workloads. An MLOps pipeline covers at least one project end-to-end: feature engineering, training, validation, deployment, and basic monitoring. Data quality checks run before training. Infrastructure is provisioned through IaC rather than portal clicks. The organization can answer "what models are in production and who owns them."

Typical signals: at least one model behind an API endpoint, CI/CD pipeline for model training, model registry in use (MLflow or Azure ML), scheduled retraining, basic drift alerting.

Level 4 -- Managed¶

AI governance is formalized. A responsible-AI review board or process gates model deployments. Model risk management covers bias testing, explainability, and adversarial robustness. Multiple business units consume AI capabilities through shared platform services. Cost attribution for GPU compute is tracked. Data contracts ensure upstream changes do not silently break downstream models.

Typical signals: responsible-AI checklist enforced before production promotion, model cards published for every production model, GPU spend attributed to business units via tags, data contracts validated in CI, model monitoring dashboards reviewed weekly.

Level 5 -- Optimizing¶

AI is embedded in the organization's operating model. Continuous improvement loops feed production model performance back into data quality and feature engineering. Autonomous systems (agents, closed-loop control) operate within well-defined guardrails. The platform team publishes internal SLAs for model serving latency, availability, and freshness. New AI use cases are evaluated against a portfolio framework with clear ROI thresholds.

Typical signals: AI-first product decisions, automated retraining triggered by drift detection, agent architectures in production, internal AI platform has an SLA, FinOps for AI is mature.

graph LR
    L1["Level 1<br/>Ad Hoc"]
    L2["Level 2<br/>Aware"]
    L3["Level 3<br/>Operational"]
    L4["Level 4<br/>Managed"]
    L5["Level 5<br/>Optimizing"]

    L1 --> L2 --> L3 --> L4 --> L5

    style L1 fill:#f44336,color:#fff
    style L2 fill:#ff9800,color:#fff
    style L3 fill:#ffc107,color:#000
    style L4 fill:#4caf50,color:#fff
    style L5 fill:#2196f3,color:#fff

Assessment Dimensions¶

Eight dimensions capture the breadth of capabilities required for sustainable AI adoption. Each dimension is scored 1--5 using the rubrics below. The aggregate maturity level is the floor of the average score, ensuring that severe weakness in any single dimension pulls the overall rating down.

Aggregate formula: floor(mean(D1, D2, ..., D8))

Dimension 1 -- Data Readiness¶

Data is the foundation. Without high-quality, discoverable, fresh data, every model is built on sand.

Dimension 2 -- Infrastructure¶

AI workloads demand compute, storage, and networking patterns that general-purpose cloud landing zones do not provide out of the box.

Dimension 3 -- Skills and Talent¶

Technology without skilled people is shelfware. This dimension assesses the depth and breadth of AI-related skills.

Dimension 4 -- Governance and Ethics¶

Responsible AI is not optional. Governance ensures models are fair, explainable, and auditable.

Dimension 5 -- Use Case Portfolio¶

Not every problem should be solved with AI. This dimension assesses how the organization identifies, prioritizes, and tracks AI use cases.

Dimension 6 -- MLOps Maturity¶

MLOps is the bridge between notebook experiments and production value.

Dimension 7 -- Security and Compliance¶

AI introduces novel security and compliance challenges: model poisoning, prompt injection, data leakage through model outputs, and regulatory requirements specific to automated decision-making.

Dimension 8 -- Organizational Culture¶

Technology and process are necessary but not sufficient. Culture determines whether AI investments deliver lasting value.

Assessment Questionnaire¶

Use the following questions to guide scoring conversations with stakeholders. For each dimension, average the question scores to produce the dimension score.

Data Readiness¶

1. Can a new team member discover and access the data they need for a model within one business day, without filing a ticket? (1 = no catalog; 5 = self-service marketplace with instant access)
2. What percentage of your analytical data has automated quality checks running before it reaches the Gold layer? (1 = 0%; 5 = >90%)
3. Do data producers publish freshness SLAs, and are those SLAs monitored with alerts? (1 = no SLAs; 5 = SLAs monitored with automated escalation)

Infrastructure¶

1. How long does it take to provision a GPU-enabled training environment for a new project? (1 = weeks/manual; 5 = minutes/self-service IaC)
2. Is model serving infrastructure autoscaled and cost-optimized (spot instances, scale-to-zero)? (1 = no serving infra; 5 = fully automated with cost attribution)
3. Do you have separate environments (dev/staging/prod) for ML workloads with identical IaC definitions? (1 = single environment; 5 = parity across all environments)

Skills and Talent¶

1. Do you have dedicated ML engineering roles (distinct from data science) responsible for productionizing models? (1 = no; 5 = mature ML platform team)
2. What percentage of business stakeholders have completed AI literacy training? (1 = 0%; 5 = >80%)
3. Is there a structured career path for AI/ML roles in your organization? (1 = no; 5 = well-defined with mentorship)

Governance and Ethics¶

1. Is there a mandatory review gate before any model reaches production? (1 = no review; 5 = formal board with documented criteria)
2. Do you test for bias and fairness as part of your model validation pipeline? (1 = never; 5 = automated in CI for every model)
3. Can you explain to a regulator how any production model makes its decisions? (1 = no; 5 = explainability reports auto-generated and archived)
4. Do you have a process for ongoing monitoring of fairness metrics in production? (1 = no; 5 = automated alerts on bias drift)

Use Case Portfolio¶

1. Is there a central registry of all AI/ML use cases (active, planned, retired)? (1 = no; 5 = portfolio dashboard with ROI tracking)
2. Do you have a scoring framework for prioritizing new AI use cases? (1 = ad-hoc; 5 = formalized criteria reviewed quarterly)
3. Are underperforming models actively retired, or do they linger? (1 = linger indefinitely; 5 = kill criteria enforced automatically)

MLOps Maturity¶

1. Can you retrain and redeploy a model without any manual steps? (1 = fully manual; 5 = fully automated with rollback)
2. Do you monitor model performance (accuracy, latency, drift) in production? (1 = no monitoring; 5 = dashboards with automated alerts)
3. Is your feature engineering code tested and versioned alongside model code? (1 = no; 5 = shared feature store with CI/CD)

Security and Compliance¶

1. Do models access data through service principals with least-privilege RBAC, or through personal credentials? (1 = personal creds; 5 = managed identity with JIT)
2. Have you tested your AI systems against adversarial attacks (prompt injection, data poisoning)? (1 = never; 5 = regular red-team exercises)
3. Can you produce an audit trail showing who deployed which model version, when, and with what data? (1 = no; 5 = immutable audit log)

Organizational Culture¶

1. Does your executive team include AI/ML outcomes in their OKRs or performance goals? (1 = no; 5 = AI outcomes are key results)
2. When an ML experiment fails, is it treated as a learning opportunity or a career risk? (1 = career risk; 5 = expected and celebrated)
3. Do business units have embedded AI champions who translate between technical and business language? (1 = no; 5 = in every major BU)

Scoring Summary¶

Use the following table to record dimension scores and compute the aggregate maturity level.

radar
    title "AI Readiness Radar"
    "Data Readiness" : 0
    "Infrastructure" : 0
    "Skills & Talent" : 0
    "Governance" : 0
    "Use Cases" : 0
    "MLOps" : 0
    "Security" : 0
    "Culture" : 0

Action Plan Template¶

Based on the aggregate maturity level, the following actions are recommended. Focus first on dimensions scoring below the aggregate -- these are the bottlenecks.

Moving from Level 1 to Level 2¶

Goal: Establish foundations and executive alignment.

Moving from Level 2 to Level 3¶

Goal: Achieve first production ML workload with repeatable processes.

Moving from Level 3 to Level 4¶

Goal: Scale AI across the organization with governance guardrails.

Moving from Level 4 to Level 5¶

Goal: AI-first culture with autonomous systems and continuous optimization.

Azure Platform Mapping¶

Each dimension maps to specific Azure services. This table helps teams translate assessment gaps into procurement and provisioning actions.

CSA-in-a-Box Component Mapping¶

CSA-in-a-Box provides reference implementations that accelerate readiness improvement across every dimension. The table below maps each dimension to the specific repo components that address it.

Assessment Process¶

The assessment is most effective when conducted as a facilitated workshop with representation from data engineering, data science, ML engineering, security, compliance, and business leadership. A typical engagement follows this flow:

flowchart LR
    A["Kick-off<br/>1 day"] --> B["Interviews<br/>1--2 weeks"]
    B --> C["Score &<br/>Calibrate<br/>1 day"]
    C --> D["Gap Analysis<br/>& Roadmap<br/>1 week"]
    D --> E["Present to<br/>Leadership<br/>1 day"]
    E --> F["Reassess<br/>Quarterly"]
    F --> C

1. Kick-off -- align on scope, identify stakeholders, distribute pre-read materials.
2. Interviews -- use the questionnaire above with each stakeholder group. Collect evidence (dashboards, pipeline configs, governance docs) to calibrate scores.
3. Score and calibrate -- convene the full group to discuss and agree on dimension scores. Resolve disagreements with evidence, not hierarchy.
4. Gap analysis and roadmap -- identify the lowest-scoring dimensions, map them to the action plan template, and assign owners and timelines.
5. Present to leadership -- use the radar chart and action plan to secure budget and sponsorship.
6. Reassess quarterly -- track progress and adjust the roadmap. Celebrate dimension-level improvements.

Interpreting Results¶

Balanced profiles (all dimensions within 1 point of each other) indicate healthy, well-rounded AI capability. Spiked profiles (one or two dimensions far ahead of others) indicate investment imbalance -- the advanced dimensions will be throttled by the lagging ones.

Common patterns and their implications:

Related¶

• AI/ML Platform Architecture -- production reference architecture for the Azure AI stack
• Azure AI Foundry Guide -- deployment and operational guide for AI Foundry within CSA-in-a-Box
• ML Lifecycle Example -- end-to-end training-to-serving walkthrough
• Data Governance Best Practices -- Purview-centric governance patterns
• Security and Compliance Best Practices -- defense-in-depth controls
• Platform Research Report -- strategic platform analysis and competitive landscape
• ADR-0007 Azure OpenAI -- decision rationale for Azure OpenAI over self-hosted LLMs
• Compliance Frameworks -- NIST 800-53, FedRAMP, CMMC, HIPAA, SOC 2 mappings