AI Gateway Labs
Hands-on labs for building an enterprise-grade AI gateway over models, MCP tools, and agents using Azure API Management and Microsoft Foundry.
- Source repo
- Azure-Samples/AI-Gateway
- Stars
- ★ 1k
- Last updated
- 12d ago
- License
- MIT
- Primary language
- Jupyter Notebook
- FA score
- 47/100 · Major gaps
At a glance
- How it runs
- Works with
- Platform-specificOpenAI API (Partial support)
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A platform engineering team evaluating APIM as the gateway for production AI workloads can run the backend pool load balancing and model routing labs to validate multi-model high availability.
- Not a fit if
- Teams that need to run outside Azure, on AWS, GCP, or on-premises
- Teams without a paid Azure subscription and Contributor/RBAC roles
- Users wanting an installable product rather than building infrastructure
- Website
- aka.ms/ai-gateway/labs ↗ set on GitHub
- Maintenance
- Commits in the last 30 dayslast commit 2026-11-05
- Source review
- 47/100 · Major gaps
What does this agent do, and when should you use it?
AI Gateway Labs is an MIT-licensed Azure-Samples repository containing 30+ hands-on labs delivered as Jupyter Notebooks, each paired with Bicep infrastructure templates and Azure API Management (APIM) policy XML. The labs span three themes: model governance (backend pool load balancing, token rate limiting, semantic caching, model routing, FinOps quota management), tool integration (Model Context Protocol, MCP client authorization, function calling, realtime audio with MCP), and agent orchestration (Foundry Agent Service, OpenAI Agents SDK, Gemini MCP agents, A2A-enabled agents). The deployment boundary is your own Azure subscription: after cloning the repo and syncing the Python environment with uv, each notebook drives Azure CLI and Bicep deployments of APIM instances and model backends, then issues live inference requests against them. The repository also ships Copilot Agent Skills (lab-creator, apim-bicep, apim-policies, etc.) for AI-assisted creation of new labs, plus developer tools including a mock OpenAI server and OAuth test client. It is best understood as an evaluation and enablement toolkit for teams deciding whether to adopt APIM as their AI gateway — not a ready-to-install product.
The repository is a collection of executable learning and evaluation assets. Each lab (e.g., labs/backend-pool-load-balancing/backend-pool-load-balancing.ipynb) uses your configured Azure subscription credentials to deploy an APIM instance and Azure OpenAI / Foundry model backends via Bicep, then walks through notebook steps that configure and exercise APIM policies — token rate limiting, semantic caching, model routing, content safety, OAuth — and finally emits real inference calls whose token metrics, logs, and traces you inspect. The tools/ folder adds tracing, streaming, rate-limit testing, an OpenAI API mock server, and an OAuth client notebook. Copilot skills (lab-creator, apim-bicep, apim-terraform, apim-policies, apim-kql, mcp-builder) generate complete new lab skeletons — notebook, Bicep, policy XML, README, and cleanup script — from prompts in VS Code.
- A platform engineering team evaluating APIM as the gateway for production AI workloads can run the backend pool load balancing and model routing labs to validate multi-model high availability.
- A FinOps owner building AI budget controls can use the FinOps Framework lab to test automated token quotas and rate limiting.
- A developer adding MCP tools to an agent can work through the MCP and MCP client authorization labs to learn OAuth credential management at the gateway layer.
- An architect deciding whether semantic caching pays off can run the semantic-caching lab and compare cost and latency with vector-similarity caching.
- An enterprise aligning to the Azure Well-Architected Framework can adopt labs mapped to the Security, Reliability, Performance, Operations, and Cost pillars.
- A team that wants to author its own gateway labs can use the bundled Copilot skills (e.g., lab-creator) in VS Code to scaffold notebooks, Bicep, and policies from a prompt.
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Install / run
irm https://astral.sh/uv/install.ps1 | iex
- Config files
pyproject.toml- Runtime
- Python 3.12+
- Install notes cover
- Windows · macOS · Linux
How do you install or deploy this agent?
Prerequisites: Python 3.12+, the uv package manager (Linux/macOS: curl -LsSf https://astral.sh/uv/install.sh | sh; Windows: powershell -c "irm https://astral.sh/uv/install.ps1 | iex"), VS Code with the Jupyter extension, an Azure subscription with Contributor and RBAC Administrator roles, and an authenticated Azure CLI. Install: git clone https://github.com/Azure-Samples/AI-Gateway.git && cd AI-Gateway && uv sync && uv pip install -r pyproject.toml && code . Alternatively, open directly in GitHub Codespaces.
How do you use this agent?
Open any lab notebook (e.g., labs/token-rate-limiting/token-rate-limiting.ipynb) in VS Code, select the .venv interpreter created by uv sync as the Jupyter kernel, and execute cells step by step: deploy the Bicep template to create APIM and model backends, run inference tests against the APIM endpoint, then review token metrics and logs in Azure. Run each lab's cleanup notebook afterwards to release resources.
What are this agent's strengths and limitations?
- Maintained by the official Azure-Samples organization, with labs designed around all five Azure Well-Architected Framework pillars.
- Unusually broad coverage: 30+ labs spanning model governance (limiting, caching, routing) through MCP/A2A agents across OpenAI Agents SDK, Autogen, and Gemini.
- Each lab is a runnable end-to-end asset — notebook plus Bicep template plus APIM policy XML — deployable to your own subscription for real verification.
- Bundled Copilot Agent Skills, an enterprise e-Book, a workshop, and conference videos support team-wide enablement.
- Tightly coupled to the Azure ecosystem — APIM, Azure OpenAI, Foundry, and Bicep are all required; the pattern cannot be reproduced on AWS, GCP, or fully on-premises.
- Requires a paid Azure subscription with Contributor and RBAC Administrator permissions; labs incur deployment and inference costs that the source does not quantify.
- It is a samples/lab repository, not an installable product — no production packaging (Docker image, Helm chart) exists, so adoption requires your own engineering effort.
- The full lab path involves several toolchain moving parts (Python, uv, Azure CLI, VS Code), creating initial setup friction.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| AI Gateway Labs This agent | 47 · Major gaps | CLIFree + model costs | ★ 1k | 12d ago | Jupyter Notebook | — |
| OptimNow Cloud FinOps Library | 67 · Some gaps | Agent plugin / skillFree | ★ 74 | today | Python | Claude Code |
| x-use | 88 · Good | MCP serverFree | ★ 183 | today | Python | Codex · Claude Code · Claude.ai |
| Runx Skill Workflow Runtime | 82 · Good | CLIFree | ★ 103 | 2d ago | Rust | — |
How does FollowAgents rate this agent?
Why each dimension lost points
The README says labs deploy to an Azure subscription and require Contributor plus RBAC Administrator roles; the packet does not show per-resource least privilege or a deployment confirmation flow, so least_privilege, user_confirmation, and external_effects are reduced. The README describes OAuth, managed identity, and MCP credential management, while the Graph MCP example documents its OBO flow and the personal fields it retrieves, supporting some data-flow transparency. However, sample code prints received messages or request content to logs, so sensitive-data handling is thin. The root dependency list includes many packages, but this packet shows no security policy or complete lock information. Cleanup notebooks are described in AGENTS.md, but recovery safeguards cannot be verified here. The README links to the repository and related resources, but does not clearly state product maintenance ownership or update commitments.
The README and AGENTS.md consistently describe labs built from notebooks, Python, Bicep, and APIM policies. However, the README installation steps use both uv sync and uv pip install -r pyproject.toml, and the sample does not verify the described per-lab structure, limiting self-consistency. The root pyproject.toml specifies Python 3.12+ and lists dependencies, and the README gives basic setup steps; this packet does not establish dependency availability or compatibility. Example error handling returns error JSON or generic messages, but the mock server uses a broad bare except and the SLM example reduces exceptions to a generic response, limiting diagnosis.
The README groups concrete labs around models, tools, and agents and explains the use of notebooks, Bicep, and APIM policies, covering several exploration scenarios. Skill descriptions say when they apply, but some examples provide broad deployment and policy patterns without clearly stating capability limits or unsuitable cases. Skill triggers are focused on APIM Bicep, Terraform, KQL, and policies. Python, uv, VS Code/Jupyter, Azure CLI, and Azure subscription prerequisites are explicit and useful for the target environment; the packet cannot establish environment differences across all labs.
The README organizes the repository into a Quick Start, lab categories, tools, resources, and license sections; AGENTS.md adds directory conventions, providing useful navigation for a large repository. The README gives Python, uv, VS Code/Jupyter, Azure CLI, and Azure RBAC prerequisites and setup commands, though the need for the extra uv pip install command is unclear. Several skills have clear names and descriptions, but this sample does not show a stable naming or versioning policy. The README includes lab tables and a detailed lab prompt, and the skills contain code examples; FAQ and limitation coverage is sparse. LICENSE.md provides the complete MIT license text, justifying full marks. The packet has no clear changelog or maintenance owner; AGENTS.md is workspace guidance, not a maintenance commitment.
The README connects labs to models, MCP tools, and agents, and the skills provide directly usable Bicep, Terraform, KQL, and XML examples for hands-on exploration and sample building. Multiple lab categories and reusable skills add value beyond a single example, but the packet cannot establish the completeness or output quality of every lab. The product requires an Azure subscription, elevated roles, and multiple cloud dependencies, so setup and use can be costly; the README does not give per-lab cost estimates or cost-control steps, limiting the cost-benefit score.
The README links many product claims to specific labs, skills, and Azure documentation, making them traceable. Some enterprise capability claims appear only as README summaries and lack matching implementation evidence in this packet. The README and AGENTS.md corroborate the technology and lab structure, but both are documents and the sampled code covers only a few utilities, so cross-source corroboration is limited. Architectural and security benefits are often presented as descriptive claims without consistently distinguishing demonstrated implementation from intended use, reducing fact-inference separation.
- These labs deploy cloud resources, and the README requires Contributor and RBAC Administrator roles. Review the Bicep, APIM policies, target subscription, and expected costs before deployment.
- Sample servers log some received prompts or messages. Avoid sending real personal data, credentials, or confidential content during exploration.
FAQ
Is AI Gateway Labs free?
Does AI Gateway Labs work on Windows?
irm https://astral.sh/uv/install.ps1 | iex.Is AI Gateway Labs open source?
Is AI Gateway Labs safe to use?
Is AI Gateway Labs still maintained?
Who makes AI Gateway Labs?
What is AI Gateway Labs's official website?
Thanks — we use this to improve the page.