AI Gateway Labs

Hands-on labs for building an enterprise-grade AI gateway over models, MCP tools, and agents using Azure API Management and Microsoft Foundry.

Stars
★ 1k
Last updated
12d ago
License
MIT
Primary language
Jupyter Notebook

At a glance

How it runs
CLIIDE extension
Works with
Platform-specificOpenAI API (Partial support)
Cost
Free software; you pay for model usage
Setup effort
High · needs real infrastructure
You'll need
Python 3.12+uvVS Code with Jupyter extensionAzure subscription with Contributor + RBAC Administrator rolesAzure CLIShell / CLINetwork accessLocal filesystem
Typical use
A platform engineering team evaluating APIM as the gateway for production AI workloads can run the backend pool load balancing and model routing labs to validate multi-model high availability.
Not a fit if
  • Teams that need to run outside Azure, on AWS, GCP, or on-premises
  • Teams without a paid Azure subscription and Contributor/RBAC roles
  • Users wanting an installable product rather than building infrastructure
Website
aka.ms/ai-gateway/labs ↗ set on GitHub
Maintenance
Commits in the last 30 dayslast commit 2026-11-05

What does this agent do, and when should you use it?

AI Gateway Labs is an MIT-licensed Azure-Samples repository containing 30+ hands-on labs delivered as Jupyter Notebooks, each paired with Bicep infrastructure templates and Azure API Management (APIM) policy XML. The labs span three themes: model governance (backend pool load balancing, token rate limiting, semantic caching, model routing, FinOps quota management), tool integration (Model Context Protocol, MCP client authorization, function calling, realtime audio with MCP), and agent orchestration (Foundry Agent Service, OpenAI Agents SDK, Gemini MCP agents, A2A-enabled agents). The deployment boundary is your own Azure subscription: after cloning the repo and syncing the Python environment with uv, each notebook drives Azure CLI and Bicep deployments of APIM instances and model backends, then issues live inference requests against them. The repository also ships Copilot Agent Skills (lab-creator, apim-bicep, apim-policies, etc.) for AI-assisted creation of new labs, plus developer tools including a mock OpenAI server and OAuth test client. It is best understood as an evaluation and enablement toolkit for teams deciding whether to adopt APIM as their AI gateway — not a ready-to-install product.

The repository is a collection of executable learning and evaluation assets. Each lab (e.g., labs/backend-pool-load-balancing/backend-pool-load-balancing.ipynb) uses your configured Azure subscription credentials to deploy an APIM instance and Azure OpenAI / Foundry model backends via Bicep, then walks through notebook steps that configure and exercise APIM policies — token rate limiting, semantic caching, model routing, content safety, OAuth — and finally emits real inference calls whose token metrics, logs, and traces you inspect. The tools/ folder adds tracing, streaming, rate-limit testing, an OpenAI API mock server, and an OAuth client notebook. Copilot skills (lab-creator, apim-bicep, apim-terraform, apim-policies, apim-kql, mcp-builder) generate complete new lab skeletons — notebook, Bicep, policy XML, README, and cleanup script — from prompts in VS Code.

  1. A platform engineering team evaluating APIM as the gateway for production AI workloads can run the backend pool load balancing and model routing labs to validate multi-model high availability.
  2. A FinOps owner building AI budget controls can use the FinOps Framework lab to test automated token quotas and rate limiting.
  3. A developer adding MCP tools to an agent can work through the MCP and MCP client authorization labs to learn OAuth credential management at the gateway layer.
  4. An architect deciding whether semantic caching pays off can run the semantic-caching lab and compare cost and latency with vector-similarity caching.
  5. An enterprise aligning to the Azure Well-Architected Framework can adopt labs mapped to the Security, Reliability, Performance, Operations, and Cost pillars.
  6. A team that wants to author its own gateway labs can use the bundled Copilot skills (e.g., lab-creator) in VS Code to scaffold notebooks, Bicep, and policies from a prompt.

Key setup facts

Pulled from the install and usage notes below. The project README has the current values.

Install / run
  • irm https://astral.sh/uv/install.ps1 | iex
Config files
pyproject.toml
Runtime
Python 3.12+
Install notes cover
Windows · macOS · Linux

How do you install or deploy this agent?

Prerequisites: Python 3.12+, the uv package manager (Linux/macOS: curl -LsSf https://astral.sh/uv/install.sh | sh; Windows: powershell -c "irm https://astral.sh/uv/install.ps1 | iex"), VS Code with the Jupyter extension, an Azure subscription with Contributor and RBAC Administrator roles, and an authenticated Azure CLI. Install: git clone https://github.com/Azure-Samples/AI-Gateway.git && cd AI-Gateway && uv sync && uv pip install -r pyproject.toml && code . Alternatively, open directly in GitHub Codespaces.

How do you use this agent?

Open any lab notebook (e.g., labs/token-rate-limiting/token-rate-limiting.ipynb) in VS Code, select the .venv interpreter created by uv sync as the Jupyter kernel, and execute cells step by step: deploy the Bicep template to create APIM and model backends, run inference tests against the APIM endpoint, then review token metrics and logs in Azure. Run each lab's cleanup notebook afterwards to release resources.

What are this agent's strengths and limitations?

Pros
  • Maintained by the official Azure-Samples organization, with labs designed around all five Azure Well-Architected Framework pillars.
  • Unusually broad coverage: 30+ labs spanning model governance (limiting, caching, routing) through MCP/A2A agents across OpenAI Agents SDK, Autogen, and Gemini.
  • Each lab is a runnable end-to-end asset — notebook plus Bicep template plus APIM policy XML — deployable to your own subscription for real verification.
  • Bundled Copilot Agent Skills, an enterprise e-Book, a workshop, and conference videos support team-wide enablement.
Limitations
  • Tightly coupled to the Azure ecosystem — APIM, Azure OpenAI, Foundry, and Bicep are all required; the pattern cannot be reproduced on AWS, GCP, or fully on-premises.
  • Requires a paid Azure subscription with Contributor and RBAC Administrator permissions; labs incur deployment and inference costs that the source does not quantify.
  • It is a samples/lab repository, not an installable product — no production packaging (Docker image, Helm chart) exists, so adoption requires your own engineering effort.
  • The full lab path involves several toolchain moving parts (Python, uv, Azure CLI, VS Code), creating initial setup friction.

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

Agent Source review Form / cost Stars Updated Language Full support on
AI Gateway Labs This agent 47 · Major gaps CLIFree + model costs ★ 1k 12d ago Jupyter Notebook —
OptimNow Cloud FinOps Library 67 · Some gaps Agent plugin / skillFree ★ 74 today Python Claude Code
x-use 88 · Good MCP serverFree ★ 183 today Python Codex · Claude Code · Claude.ai
Runx Skill Workflow Runtime 82 · Good CLIFree ★ 103 2d ago Rust —

How does FollowAgents rate this agent?

FollowAgents source review · FARS-2.1
Major gaps
47/ 100 5-point scale 2.4 / 5
Trust 11/29
Reliability 5/14
Adaptability 10/18
Convention 10/18
Effectiveness 7/13
Verifiability 4/8
Why each dimension lost points
Trust11 / 29 · 1.9/5

The README says labs deploy to an Azure subscription and require Contributor plus RBAC Administrator roles; the packet does not show per-resource least privilege or a deployment confirmation flow, so least_privilege, user_confirmation, and external_effects are reduced. The README describes OAuth, managed identity, and MCP credential management, while the Graph MCP example documents its OBO flow and the personal fields it retrieves, supporting some data-flow transparency. However, sample code prints received messages or request content to logs, so sensitive-data handling is thin. The root dependency list includes many packages, but this packet shows no security policy or complete lock information. Cleanup notebooks are described in AGENTS.md, but recovery safeguards cannot be verified here. The README links to the repository and related resources, but does not clearly state product maintenance ownership or update commitments.

Reliability5 / 14 · 1.8/5

The README and AGENTS.md consistently describe labs built from notebooks, Python, Bicep, and APIM policies. However, the README installation steps use both uv sync and uv pip install -r pyproject.toml, and the sample does not verify the described per-lab structure, limiting self-consistency. The root pyproject.toml specifies Python 3.12+ and lists dependencies, and the README gives basic setup steps; this packet does not establish dependency availability or compatibility. Example error handling returns error JSON or generic messages, but the mock server uses a broad bare except and the SLM example reduces exceptions to a generic response, limiting diagnosis.

Adaptability10 / 18 · 2.8/5

The README groups concrete labs around models, tools, and agents and explains the use of notebooks, Bicep, and APIM policies, covering several exploration scenarios. Skill descriptions say when they apply, but some examples provide broad deployment and policy patterns without clearly stating capability limits or unsuitable cases. Skill triggers are focused on APIM Bicep, Terraform, KQL, and policies. Python, uv, VS Code/Jupyter, Azure CLI, and Azure subscription prerequisites are explicit and useful for the target environment; the packet cannot establish environment differences across all labs.

Convention10 / 18 · 2.8/5

The README organizes the repository into a Quick Start, lab categories, tools, resources, and license sections; AGENTS.md adds directory conventions, providing useful navigation for a large repository. The README gives Python, uv, VS Code/Jupyter, Azure CLI, and Azure RBAC prerequisites and setup commands, though the need for the extra uv pip install command is unclear. Several skills have clear names and descriptions, but this sample does not show a stable naming or versioning policy. The README includes lab tables and a detailed lab prompt, and the skills contain code examples; FAQ and limitation coverage is sparse. LICENSE.md provides the complete MIT license text, justifying full marks. The packet has no clear changelog or maintenance owner; AGENTS.md is workspace guidance, not a maintenance commitment.

Effectiveness7 / 13 · 2.7/5

The README connects labs to models, MCP tools, and agents, and the skills provide directly usable Bicep, Terraform, KQL, and XML examples for hands-on exploration and sample building. Multiple lab categories and reusable skills add value beyond a single example, but the packet cannot establish the completeness or output quality of every lab. The product requires an Azure subscription, elevated roles, and multiple cloud dependencies, so setup and use can be costly; the README does not give per-lab cost estimates or cost-control steps, limiting the cost-benefit score.

Verifiability4 / 8 · 2.5/5

The README links many product claims to specific labs, skills, and Azure documentation, making them traceable. Some enterprise capability claims appear only as README summaries and lack matching implementation evidence in this packet. The README and AGENTS.md corroborate the technology and lab structure, but both are documents and the sampled code covers only a few utilities, so cross-source corroboration is limited. Architectural and security benefits are often presented as descriptive claims without consistently distinguishing demonstrated implementation from intended use, reducing fact-inference separation.

Risks and how to mitigate them
  • These labs deploy cloud resources, and the README requires Contributor and RBAC Administrator roles. Review the Bicep, APIM policies, target subscription, and expected costs before deployment.
  • Sample servers log some received prompts or messages. Avoid sending real personal data, credentials, or confidential content during exploration.
Evidence confidence: Low Reviewed Nov 08, 2026 Reviewed revision 98c242510ba5
See the full review method →
Maintain this project? Add the FARS badge to your README

The badge shows the current review score and verdict and links back to this page. It refreshes about once an hour after a re-review.

FARS

[![FARS review](https://followagents.com/badge/ai-gateway.svg)](https://followagents.com/en/agents/ai-gateway)
<a href="https://followagents.com/en/agents/ai-gateway"><img src="https://followagents.com/badge/ai-gateway.svg" alt="FARS review"></a>

FAQ

Is AI Gateway Labs free?
The software is free, but AI Gateway Labs calls an AI model, and you pay that model provider for usage (or run a local model).
Does AI Gateway Labs work on Windows?
Yes. The install notes on this page include a Windows command: irm https://astral.sh/uv/install.ps1 | iex.
Is AI Gateway Labs open source?
Yes. The source code is on GitHub (Azure-Samples/AI-Gateway) under the MIT licence.
Is AI Gateway Labs safe to use?
FollowAgents' FARS review (Nov 08, 2026) scored AI Gateway Labs 11/29 on trust and safety. None of the eight safety controls the review checks for was missing outright.
Is AI Gateway Labs still maintained?
Yes. The latest commit on GitHub is from 2026-11-05, within the last 30 days.
Who makes AI Gateway Labs?
AI Gateway Labs is published on GitHub by the account Azure-Samples. FollowAgents has not verified which person or company is behind that account.
What is AI Gateway Labs's official website?
The project's GitHub repository lists https://aka.ms/ai-gateway/labs as its website. That link is set by the repository owner; the source code itself is on GitHub (Azure-Samples/AI-Gateway).
Was this page helpful?
View on GitHub ↗ Install ↓

Compare agents like this one

The same FARS review applied across the shortlist this agent qualifies for.

Related concepts