Microsoft Azure AI Foundry

Not A Medium Member, Read this Article, here!

TL;DR: Azure AI Foundry (now branded in many places as Microsoft Foundry) is Microsoft's unified "AI app and agent factory" for building, optimizing, and governing AI applications and agentic systems — all from a single portal and SDK surface, with first‑class support for models, agents, tools, knowledge/RAG, fine‑tuning, evaluations, and guardrails.

Table of Contens:

· 1) The Foundry Model: 8 primitives + one lifecycle · 2) Models in Foundry: catalog → deployment → cost/perf control · 3) Agents: what they are and how Foundry operationalizes them · 4) Tools: giving agents capabilities (and governance) · 5) Knowledge: Foundry IQ, RAG, and grounded retrieval that scales · 6) Data: datasets and artifacts (the fuel for tuning and evals) · 7) Fine-tuning: specialize models for reliability, cost, and tool-use consistency · 8) Evaluations: the missing discipline that makes agents shippable · 9) Guardrails: safety + security controls that apply everywhere · 10) The "one-stop" build path: from zero → production-grade · 11) Expert patterns (what separates "users" from "experts") · 12) Quick glossary (so you can read Foundry docs like a native)

1) The Foundry Model: 8 primitives + one lifecycle

To become fluent quickly, you need the platform grammar. Foundry is best understood as 8 primitives:

1. Models : choose and deploy foundation models (and embeddings)
2. Agents : wrap a model with instructions, identity, and runtime behavior
3. Tools : give agents capabilities (web, code, files, APIs, enterprise systems)
4. Knowledge : connect agents to grounded retrieval (Foundry IQ / Azure AI Search KBs)
5. Data : manage datasets & artifacts used by tuning/evals and (often) RAG
6. Fine-tune : specialize models (SFT / DPO / RFT) for reliability and cost
7. Evaluations : measure quality/safety/behavior systematically before shipping
8. Guardrails : enforce safety/security policies at multiple intervention points

And the lifecycle is:

Design → Build → Ground → Customize → Evaluate → Guard → Deploy → Operate → Iterate

This article will go wide (what exists) and deep (how it works, how to use it, and the expert patterns).

2) Models in Foundry: catalog → deployment → cost/perf control

2.1 The model catalog: "pick the brain"

Foundry exposes a model catalog where you can discover models by capability (text, multimodal, embeddings), benchmarks, licensing, and deployment options. The portal experience is tightly coupled to deployment — you don't just "select GPT‑4o"; you deploy it into your Foundry project so it has endpoints, quotas, and governance attached.

"Direct from Azure" models: Some models are "sold directly by Azure" (Microsoft-hosted and managed), designed for u_nified billing_, governance, and easier switching across models within Foundry. This matters operationally: the model becomes a governed asset you can monitor, evaluate, and constrain centrally

2.2 Deployments: models become callable endpoints

In Foundry, a model is "real" only once deployed. Deployments are what:

• enforce quotas and throughput,
• attach guardrails/content filters,
• show up in monitoring dashboards, and
• become selectable for agents, fine‑tune jobs, and evaluations.

2.3 Example: GPT‑4o mini (what the model card teaches you)

The GPT‑4o mini model highlights what you should look for in any Foundry model selection:

• Function/tool calling strength (important for agents)
• Long context (e.g., 128K context window for GPT‑4o mini)
• Cost & latency profile (mini models are often ideal for tool-heavy, multi-call agent workflows)

Expert tip: In agentic apps, small/fast models frequently outperform bigger models at the system level because tool calls + retrieval + orchestration dominate latency and cost.

3) Agents: what they are and how Foundry operationalizes them

3.1 What is an "agent" in Foundry?

At its simplest:

Agent = Model + Instructions + Tool access + (optional) Knowledge + Policy (guardrails) + Runtime

Foundry Agent Service treats tools as first-class extensions: "an agent on its own can generate text, but tools let it take action — searching the web, running code, querying your data, or calling your APIs."

3.2 Agent definition anatomy (conceptual)

An Agent typically includes:

• Model (deployment name)
• System instructions (role, boundaries, style, tool usage rules)
• Tools list (web search, file search, OpenAPI, MCP…)
• Guardrails assignment (safety/security controls)
• Versioning (update behavior safely; compare via evals)

3.3 Why agents are different from "chat with a model"

A plain chat completion is one model call. An agentic system is a multi-step loop:

1. interpret request
2. decide whether to call tools
3. call tool(s) with structured inputs
4. ingest tool results
5. produce final answer and/or next action

Foundry's built-in support for tool calling, knowledge retrieval, evaluations, and guardrails at multiple intervention points is exactly what turns this from a demo into a production pattern

3.4 Compare agent types

4) Tools: giving agents capabilities (and governance)

Tools are the "arms and legs" of your agent.

4.1 Foundry's tool taxonomy (built-in vs custom)

Foundry Agent Service defines two broad categories:

Built-in tools

These are "preconfigured capabilities" where Foundry handles execution , no extra hosting required. The most common are:

• Web search (real-time public web + citations)
• Code Interpreter (sandboxed Python execution for analysis/charts)
• File Search (vector search over uploaded files / docs)
• Function calling (your app executes the function; agent decides when to call)

By default, tool configurations such as file IDs, vector store IDs, and MCP server endpoints are fixed when you create the agent. Structured inputs in tool properties allow you to override these values at runtime without creating a new agent version.

Structured inputs are useful when:

• Different users need different vector stores or files based on their context.
• You want to reuse the same agent definition across environments (development, staging, production).
• MCP server endpoints or authentication tokens vary per request.

The following tool properties support customization via structured inputs:

For example, an agent definition with a templated vector store:

{
  "tools": [
    {
      "type": "file_search",
      "vector_store_ids": ["vs_base_kb", "{{customer_kb}}"]
    }
  ],
  "structured_inputs": {
    "customer_kb": {
      "description": "Vector store ID for the customer's knowledge base",
      "required": true,
      "schema": { "type": "string" }
    }
  }
}

{
  "tools": [
    {
      "type": "file_search",
      "vector_store_ids": ["vs_base_kb", "{{customer_kb}}"]
    }
  ],
  "structured_inputs": {
    "customer_kb": {
      "description": "Vector store ID for the customer's knowledge base",
      "required": true,
      "schema": { "type": "string" }
    }
  }
}

At runtime, provide the actual value:

{
  "agent": { "type": "agent_reference", "name": "support-agent", "version": "1" },
  "input": [{ "type": "text", "text": "How do I upgrade my account?" }],
  "structured_inputs": {
    "customer_kb": "vs_premium_kb_2024"
  }
}

{
  "agent": { "type": "agent_reference", "name": "support-agent", "version": "1" },
  "input": [{ "type": "text", "text": "How do I upgrade my account?" }],
  "structured_inputs": {
    "customer_kb": "vs_premium_kb_2024"
  }
}

Custom tools

When built-ins aren't enough, you bring your own:

• OpenAPI tools : connect any HTTP API described by OpenAPI 3.0/3.1
• MCP (Model Context Protocol) : connect to tools hosted behind MCP server endpoints (great for shared tool backends)
• Agent-to-Agent (A2A) (preview): connect agents to other agents for delegation/collaboratio

The following example connects to an MCP server by using key-based authentication. Store the credential in a project connection, and then reference the connection name when you create the tool:

from azure.ai.projects.models import MCPTool

tool = MCPTool(
    server_label="github",
    server_url="https://api.githubcopilot.com/mcp",
    require_approval="always",
    project_connection_id="my-github-connection",
)

from azure.ai.projects.models import MCPTool

tool = MCPTool(
    server_label="github",
    server_url="https://api.githubcopilot.com/mcp",
    require_approval="always",
    project_connection_id="my-github-connection",
)

4.2 Tool calling flow (the operational truth)

The important detail: tools are not "plugins the model executes." Instead, the model requests a tool call; the platform/app executes it; results come back into the conversation; then the model continues. That's why governance matters, data may leave your boundary depending on the tool. Foundry explicitly warns that when you connect non‑Microsoft services, some prompt content may be sent to those services and you are responsible for the resulting data handling

4.3 OpenAPI tools in depth (authentication is the make-or-break)

Foundry supports connecting agents to OpenAPI tools using:

• Anonymous
• API key
• Managed identity

And it enforces practical constraints like operationId requirements and supported request content types for bodies.

Expert pattern: Use project connections for secrets, not code or specs. Foundry recommends storing keys/tokens in project connections and using least‑privilege access, especially with managed identities.

4.4 Structured inputs: reuse one agent across many contexts (advanced)

Foundry supports "structured inputs" that let you override tool parameters at runtime (e.g., which vector store to search) without creating a new agent version. This is a key scaling pattern for SaaS / multi-tenant agents.

5) Knowledge: Foundry IQ, RAG, and grounded retrieval that scales

"Knowledge" is about grounding, ensuring answers are based on trusted sources and returned with traceability.

5.1 Foundry IQ knowledge bases (what they do)

Foundry IQ knowledge bases can be connected to agents. When invoked, the knowledge base orchestrates a retrieval pipeline that:

• plans and decomposes the user query into subqueries,
• processes subqueries in parallel using keyword, vector, or hybrid retrieval,
• applies semantic reranking, and
• synthesizes results into a unified response with source references.

That is agentic retrieval: your agent isn't just stuffing top‑k chunks into a prompt; the retrieval layer reasons about how to fetch the right evidence.

5.2 The bridge: MCP (Model Context Protocol)

Foundry's current integration approach uses MCP tools to connect the agent to the knowledge base endpoint. The docs describe creating a project connection (often using managed identity) to securely call the knowledge base MCP endpoint.

5.3 Instruction design for knowledge grounding (practical)

Foundry explicitly recommends instructions like:

• "Use the knowledge base tool to answer user questions."
• "If the knowledge base doesn't contain the answer, respond with 'I don't know.'"
• "Include citations when using knowledge base information."

Expert insight: This is not optional. Most "hallucination" in RAG systems comes from instruction ambiguity and retrieval mismatch, not from "bad models."

6) Data: datasets and artifacts (the fuel for tuning and evals)

The "Data" section in Foundry is where teams typically manage:

• datasets for fine‑tuning (training/validation JSONL),
• datasets for evaluations (test queries, ground truth, scenario tags),
• files used by file search / vector stores,
• and often connections to enterprise stores (varies by project type and org setup).

Even when the portal UI evolves, the underlying engineering requirement stays constant: your tuning and eval workflows depend on versioned, repeatable datasets.

6.1 Data formats you must get right (non-negotiable)

Fine-tuning datasets

Fine-tuning data must be in JSONL, in chat-completions conversational format, and files have size/encoding constraints (UTF‑8 with BOM; <512 MB per file)

Evaluation datasets

Agent evaluations commonly use JSONL datasets containing test cases (e.g., a query per line), uploaded as datasets into the project.

Expert pattern: Treat datasets like code:

• version them,
• gate changes through review,
• and measure impact via eval runs.

7) Fine-tuning: specialize models for reliability, cost, and tool-use consistency

Fine-tuning is not "make the model smarter." It's "make the model behave more predictably for your domain and workflows."

7.1 What Foundry fine-tuning enables (and why it beats endless prompt hacks)

Foundry's fine-tuning docs emphasize:

• higher quality than prompt engineering alone,
• training on more examples than fit in context,
• token savings (shorter prompts),
• lower latency (especially with smaller models).

7.2 The core technique: LoRA

Foundry uses Low-Rank Adaptation (LoRA) to fine-tune efficiently by updating a smaller set of parameters, making training faster and more affordable than heavier methods.

7.3 Supported methods: SFT vs DPO vs RFT (how to choose)

Foundry names three customization methods:

• SFT (Supervised Fine-Tuning): train on labeled input/output pairs; best default.
• DPO (Direct Preference Optimization): align outputs to preferred responses; useful when "style/quality preference" is the gap.
• RFT (Reinforcement Fine-Tuning): optimize complex behaviors using reward signals from graders.

Rule of thumb (expert):

• If you can write correct target outputs → SFT
• If you have "better vs worse" preferences → DPO
• If success requires multi-step policy and tool discipline → consider RFT

7.4 Training tiers: Standard vs Global vs Developer (preview)

Foundry exposes tiers:

• Standard: regional training, data residency guarantees.
• Global: more affordable by using capacity outside your region; data/weights copied.
• Developer (preview): cheapest, uses idle capacity; may be preempted; no residency guarantees.

This is a major architectural lever for enterprise compliance.

7.5 Hyperparameters, checkpoints, and operational tuning

Foundry exposes key hyperparameters (batch size, learning rate multiplier, epochs) and provides training metrics like loss and token accuracy, plus checkpoints created per epoch that can be deployed and even used for further fine-tuning.

Expert pattern: deploy checkpoints to compare behaviors early, before overfitting becomes entrenched.

8) Evaluations: the missing discipline that makes agents shippable

If you want to become "expert," this is where expertise actually shows.

8.1 What evaluations are (in Foundry terms)

Foundry frames evaluation as essential for meeting "quality and safety standards before deployment," recommending baselines and acceptance thresholds (example: 85% task adherence).

An evaluation run:

• sends each test query to your agent,
• captures the response (including tool calls),
• applies evaluators to produce scores,
• and generates an analysis report

8.2 Evaluators: quality, safety, and agent behavior

Foundry built-in evaluators include:

• Task Adherence (did it follow system instructions?)
• Coherence (logical, well-structured response)
• Safety evaluators like Violence detection, and more.

Some evaluators are "AI-as-judge" and require a model deployment as the judge

8.3 The most powerful detail: evaluating tool behavior

Foundry evaluation data mapping can target:

• {{sample.output_items}} → full response including tool calls
• {{sample.output_text}} → just the text response

That means you can test what actually matters in agents:

• Did it call the right tool?
• Did it pass correct parameters?
• Did it ground answers when required?

8.4 Expert evaluation strategy For Production AI Agents

Build three layers of eval datasets:

1. Golden path scenarios (core tasks)
2. Edge cases (ambiguous requests, missing info, conflicting docs)
3. Adversarial (prompt injection, data exfiltration attempts, unsafe asks)

Then gate deployments on:

• task adherence,
• groundedness & citation correctness (for RAG),
• safety pass rate.

Foundry's evaluation framework is explicitly intended to integrate into workflows and CI/CD gates

9) Guardrails: safety + security controls that apply everywhere

Guardrails are your enforceable policy layer.

9.1 Guardrails are collections of controls

Foundry defines:

• Guardrail = named collection of controls
• Control = (risk to detect) + (intervention points) + (response action)

This is the most important conceptual shift: safety isn't only "model training," it's "runtime policy."

9.2 Intervention points

Foundry supports four intervention points:

• User input (prompt to model/agent)
• Tool call (agent-proposed action/data to tool) (preview for agents)
• Tool response (tool output back to agent) (preview for agents)
• Output (final completion returned to user)

This is a big deal: it means you can protect not only what users say and what the model answers, but also what the agent sends to tools and what tools return.

9.3 Actions: annotate vs block

Guardrail actions include:

• Annotate
• Annotate and block

This enables progressive hardening:

• start in annotate mode to observe,
• then move to block once false positives are understood.

9.4 Risks covered (and what's preview)

The guardrails overview lists risks applicable to models and agents, including:

• harm categories (hate, sexual, self-harm, violence),
• user prompt attacks,
• indirect attacks,
• protected material (text/code),
• groundedness (preview),
• task adherence, and more

9.5 Prompt Shields: defending against prompt injection (direct + indirect)

Foundry's Prompt Shields detect and prevent adversarial attempts to manipulate the model:

• User prompt attacks (jailbreak/prompt injection attempts in user input)
• Document attacks (hidden instructions embedded in third‑party content)

They are scanned at specific intervention points:

• user prompt attacks → user input
• document attacks → user input and tool response

Spotlighting (preview)

A further defense called Spotlighting can transform document content (base‑64 encoding) to signal lower trust to the model, improving resistance to indirect prompt injection — at the cost of more tokens and potential context overflow

9.6 Content Safety capabilities that power guardrails

Azure AI Content Safety is positioned as a safety system to monitor prompts and outputs, including:

• harmful content detection (violence/hate/sexual/self-harm),
• Prompt Shields,
• groundedness detection,
• protected material detection.

Expert pattern: Treat guardrails as an engineering system:

• define policy,
• instrument with annotate-only,
• evaluate,
• then enforce with block.

10) The "one-stop" build path: from zero → production-grade

Here's a practical blueprint that maps to the Foundry sidebar.

Step 0 — Create a Foundry project

Projects are the organizing unit for models, agents, tools, and datasets in the portal and SDK.

Step 1 — Deploy a model

Start with a cost-effective model suitable for tool-heavy workflows (often a "mini" class model) and deploy it into the project so you can monitor and govern it.

Step 2 — Create an agent

Define system instructions and attach a baseline toolset (usually at least web search or file search depending on the use case).

Step 3 — Add tools (capabilities)

Attach built-in tools first; graduate to OpenAPI/MCP for enterprise integrations. Keep secrets in project connections; use managed identity when possible.

Step 4 — Add knowledge (grounding)

If the agent must be factual with enterprise context, connect a Foundry IQ knowledge base and enforce citations + "I don't know" behavior.

Step 5 — Evaluate early (before you "feel ready")

Create a JSONL test set; run task adherence + coherence + safety evaluators; inspect tool-call behavior in outputs.

Step 6 — Add guardrails

Apply guardrails at user input/output at minimum; for agents, also protect tool call/response pathways where available. Enable prompt shields for injection resistance.

Step 7 — Fine-tune (only when it's the right lever)

If prompting + retrieval still produces inconsistent behavior (especially tool usage discipline), fine‑tune using SFT or DPO; consider RFT for complex policy/tool behaviors.

Step 8 — Operate: monitor tokens/cost/quality drift

Use monitoring dashboards (like the one in your screenshot) and continuous evaluation patterns to detect regressions when prompts/tools/models change.

11) Expert patterns (what separates "users" from "experts")

Pattern A — Tool-first architecture

Design the agent so it must use tools/knowledge for factual claims; never let it "wing it." This is directly aligned with Foundry's recommended instruction patterns for knowledge retrieval.

Pattern B — Runtime configurability with structured inputs

Use structured inputs to swap:

• vector store IDs,
• MCP endpoints,
• headers/tokens, without creating new agent versions — this makes multi-tenant + multi-environment deployments tractable.

Pattern C — Safety at every boundary

Apply guardrails to:

• user input,
• output,
• tool call,
• tool response, and enable prompt shields. This is the "agent-ready" safety model the guardrails system was built for.

Pattern D — Evaluation is the product

Build evals that measure what you truly care about:

• task adherence,
• correct tool selection,
• correct parameters,
• grounded answers with citations,
• safety compliance.

Foundry's evaluation framework explicitly supports scoring based on full response artifacts including tool calls.

Pattern E — Fine-tune for discipline, not "knowledge"

Use fine-tuning to reduce:

• prompt length/token cost,
• behavioral variance,
• tool misuse,
• formatting drift, and to encode consistent "house style" and policy adherence. Foundry's fine‑tuning documentation explicitly calls out token savings and lower latency benefits.

12) Quick glossary (so you can read Foundry docs like a native)

• Model: a foundation model you deploy for inference.
• Deployment: the operational endpoint/config wrapper around a model in your project.
• Agent: a defined runtime that uses a deployed model, instructions, and tools.
• Tool: a callable capability (built-in or custom) the agent can invoke during conversation.
• Knowledge base (Foundry IQ): orchestrated retrieval pipeline providing grounded evidence with
• Dataset: uploaded versioned file(s) used by evals or fine-tuning (often JSONL).
• Evaluation: dataset + evaluators + runs that score agent/model behavior.
• Guardrail: named set of controls applied to model or agent.
• Prompt Shields: guardrail control that detects direct/indirect prompt injection attacks.