---
type: "Article"
title: "A Paved Road for AI Workloads on Azure"
description: "The Azure platform defaults we set before a team ships its first AI feature: managed identity, private model endpoints, an AI gateway, tracing, cost attribution and eval gates."
resource: "https://www.theknowledgeelephant.com/blog/a-paved-road-for-ai-workloads-on-azure/"
tags:
  - "Azure"
  - "Platform engineering"
  - "Infrastructure as code"
  - "OpenTofu"
  - "FinOps"
status: "stable"
stale_after: "2027-05-07"
tke:
  published_at: "2026-11-07"
  author: "Dane Hollenbach"
  category: "Platform Engineering"
  service: "azure-platform-engineering"
---
# A Paved Road for AI Workloads on Azure

> **Answer:** Give product teams a template where the secure path is the fast path: managed identity instead of API keys, model endpoints behind private networking, an API gateway that meters and limits tokens per consumer, OpenTelemetry traces across retrieval and model calls, and a pipeline where evaluation suites gate deployment like unit tests.

Platform engineering exists to make the right thing the easy thing. AI workloads raise the stakes, because every weakness in the platform shows up as a security finding, a latency spike or an invoice nobody can explain.

This is the baseline we put in place on Azure before a team ships its first AI feature. None of it is exotic. The value is in having it as a paved road, a set of templates and modules a team can adopt in an afternoon, rather than a set of guidelines each team reinterprets.

## 1. Identity: no keys in configuration

Model endpoints, search services, storage and databases should all be reached with **managed identities** and Entra ID role assignments, not connection strings or API keys copied into app settings.

- Each workload gets its own user-assigned managed identity.
- Role assignments are granted in infrastructure as code, scoped to the specific resource.
- Where a secret is unavoidable (a third-party API), it lives in Key Vault and is referenced, never copied.

This removes the most common cause of AI-related incidents we see: a long-lived key with broad access, pasted into a config file during a prototype and never rotated.

## 2. Network: model traffic stays private

Put Azure OpenAI, Azure AI Search, storage and SQL behind **Private Endpoints** and disable public network access. Application traffic reaches them over the virtual network; public traffic enters only through **Azure Front Door with WAF** in front of your own applications.

For compute, **Azure Container Apps** is our default for most AI-backed APIs: it scales to zero, integrates with managed identity and VNets, and avoids running a Kubernetes control plane you do not need. We reach for **AKS** when a team needs GPU node pools, custom operators or very fine-grained scheduling.

## 3. A gateway in front of every model

Do not let each service call model endpoints directly. Route model traffic through **Azure API Management** configured as an AI gateway:

- Per-consumer token rate limits and quotas, so one feature or customer cannot exhaust shared capacity.
- Token usage emitted as metrics broken down by consumer, feature and customer.
- Load balancing and failover across deployments and regions.
- One place to enforce content safety and logging policy.

The gateway is what turns "the AI bill doubled" into "customer 412's bulk import job tripled its summarisation calls on Tuesday".

## 4. Observability: one trace per answer

Instrument with **OpenTelemetry** and export to Application Insights. A single trace should show the full path of an AI-backed request:

1. Inbound request and the customer it belongs to.
2. Retrieval: query, index, number of results, latency.
3. Each tool call with arguments (redacted) and outcome.
4. Each model call with deployment, prompt revision, token counts and latency.

With that in place you can answer "why was this answer wrong?" and "why was this request slow?" from the same trace, and you can build cost dashboards per customer and per feature from the same data.

## 5. Infrastructure as code with preview environments

Everything above is defined in **Bicep, Terraform or OpenTofu** modules, versioned and consumed by product teams. Every pull request gets an ephemeral preview environment, so a retrieval or prompt change can be exercised end to end before merge.

Keep modules small and composable: a landing-zone module owned by the platform team, and workload modules (API, retrieval, model access) that product teams compose. Policy as code catches the drift: public endpoints, missing diagnostics and unapproved regions fail the pipeline, not a quarterly audit.

## 6. Evaluation as a pipeline gate

Prompts, model choices and retrieval settings are code, so they go through the pipeline like code. A typical stage in Azure Pipelines or GitHub Actions runs a golden question set against the preview environment and fails the build when quality drops:

```yaml
- stage: Evaluate
  dependsOn: DeployPreview
  jobs:
    - job: GoldenSet
      steps:
        - script: dotnet run --project eval/Eval.csproj -- --target $(PreviewUrl) --suite golden --min-grounded 0.95
          displayName: Run grounded-answer evaluation
```

The thresholds are yours to set; the point is that a regression blocks a merge instead of reaching customers.

## The result

Teams on a paved road ship their first AI feature faster, not slower, because identity, networking, telemetry and evaluation are already solved. FAQ Hub runs on this pattern: ASP.NET Core on Container Apps behind Front Door and WAF, Azure SQL and Blob Storage, Azure AI services, and Azure Pipelines from commit to release.

Our [Azure platform engineering](https://www.theknowledgeelephant.com/services/azure-platform-engineering/) work builds this road for your teams, in your subscriptions, and hands it over with the modules and runbooks to own it.

## Related

- [What AI readiness actually means](/blog/what-ai-readiness-means.md) - testing and release, and cost and performance, are areas four and five
- [Agentic workflows in regulated fintech](/blog/agentic-workflows-in-regulated-fintech.md) - residency and private networking for regulated data