Production AI Lab: Build the Northstar Governed RAG Assistant

Northstar Assistant is a secure web API that answers questions only from approved documents. The production-shaped design separates ingestion from serving, uses managed identity, trims search by tenant/access, validates citations, traces each stage, evaluates releases, and keeps every expensive component removable through infrastructure as code.

Time: 3–6 hours · Cost: multiple billable services; estimate first, use smallest approved development tiers, set a budget, and delete the complete environment · Do not use: confidential documents in the learning deployment.

Method note: This production lab is one integrated delivery sequence, not a set of interchangeable single-tool procedures. Use the Portal, CLI, SDK, and IaC alternatives taught in the linked service chapters when implementing each resource, while keeping this end-to-end security and verification order intact.

What will you build?

What are the security boundaries?

• Entra authenticates users; the API authorizes tenant and group membership.
• The frontend never receives model/search credentials.
• API and ingestion use different managed identities.
• Search documents contain access metadata; server-built filters enforce it.
• Storage and search use private endpoints in production.
• Tools are read-only in this capstone.
• Logs contain IDs and metrics, not full documents or tokens.

Step 1: Create a repository

Protect the main branch. Add secret scanning, dependency checks, tests, and IaC validation.

Step 2: Provision the platform

Use one environment resource group and explicit parameters. Deploy:

1. VNet and subnets.
2. Storage with secure defaults and a documents container.
3. Azure AI Search.
4. Microsoft Foundry resource/project and approved model deployment.
5. App Service or Container Apps.
6. Ingestion Function or Container Apps job plus queue.
7. Key Vault only for values that cannot use identity.
8. Managed identities and role assignments.
9. Private endpoints/DNS for production; a documented restricted-public learning mode if necessary.
10. Application Insights/Log Analytics, alerts, and budget.

Run Bicep what-if or Terraform plan. Save non-secret outputs such as hostnames and resource IDs. Never output account keys.

Step 3: Implement ingestion

1. Upload only synthetic sample documents.
2. Emit a queue event with document/version ID.
3. Validate type, size, malware policy, tenant, and checksum.
4. Extract text and page/section provenance.
5. Chunk semantically and attach authorization metadata.
6. Generate embeddings with bounded concurrency.
7. Upsert a new index version.
8. Run retrieval quality checks.
9. Switch document visibility and remove stale chunks.

Persist status so a retry does not duplicate chunks. Propagate deletion and access changes.

Step 4: Implement the query API

Pseudocode:

Reject a request if the claimed tenant differs from the authenticated context. Verify every citation maps to a retrieved, authorized chunk.

Step 5: Add evaluation and release gates

Create at least 30 synthetic questions covering answerable, unanswerable, ambiguous, multilingual, injection, cross-tenant, and citation cases. Measure retrieval recall, access correctness, groundedness, citation validity, schema validity, latency, tokens, and estimated cost.

Block deployment on any cross-tenant retrieval or critical injection/tool failure. Canary prompt/model changes and record the exact deployment/model version.

Step 6: Observe the live path

Use one correlation ID through edge, API, search, model, and ingestion. Dashboard:

• Request rate, errors, and latency percentiles.
• Search latency/no-hit rate and security-filter failures.
• Model latency, tokens, throttles, content-safety outcomes.
• Queue backlog/age and ingestion failures.
• App availability and dependency failures.
• Daily cost and forecast.

Create runbooks for quota exhaustion, search outage, model deployment failure, poisoned document, access leak, and bad release.

Step 7: Verify security and reliability

• An unauthorized synthetic user retrieves zero protected chunks.
• Prompt injection in a document is treated as data, not instruction.
• The API refuses a citation it did not retrieve.
• Disabling the app identity removes data access.
• Duplicate ingestion events do not duplicate visible chunks.
• A model throttle produces bounded retry and a safe user message.
• Rollback restores the last approved app/prompt while preserving compatible data.

Step 8: Clean up completely

Delete the application resource group through IaC or Azure. Then verify separately created role assignments, Entra applications, DNS records, model deployments, search service, storage, log retention, container registry artifacts, and budgets. If private DNS or shared networking is central, remove only the lab records/endpoints—not shared infrastructure.

Production extension

Add active/passive regional deployment, a global edge, regional model/search capacity, a content governance workflow, private connectivity, data-loss prevention, online feedback review, and quarterly recovery/red-team exercises. Complete the Well-Architected review before real data.

Official references

• RAG solution design guide
• Baseline OpenAI end-to-end chat architecture
• Azure Well-Architected AI workloads
• Microsoft Foundry documentation