Forge: Reliability Layer for Self-Hosted LLM Tool Calling
A Python framework that adds guardrails to self-hosted LLM tool calling, boosting multi-step agentic workflow success rates.
Evidence: Proxy does not authenticate callers but provides --backend-api-key for backend auth; tool calls require user definition but no user confirmation mechanism; data flow transparency partially shown (e.g., proxy mode does not compact history); sensitive data handling not mentioned; dependencies only pydantic and httpx, but no vulnerability scan evidence; external effects include proxy can spawn backend processes, but docs say stopping unloads Ollama model; no rollback mechanism; source attribution clear (author and license). Deductions: user confirmation missing, sensitive data handling not mentioned, rollback missing.
Evidence: Project structure clear, modules well-organized; dependencies pydantic and httpx are common, but no version pinning; error handling has ForgeError hierarchy, but failure message quality not detailed. Deductions: dependency availability not fully verified, failure messages not detailed.
Evidence: Clear target audience (developers), scenarios include proxy server, WorkflowRunner, middleware; capability boundaries clear (not orchestrator, not coding tool); trigger conditions clear (guardrails applied on tool calls); environment fit supports multiple backends and Python versions. Deductions: no major deductions, but environment fit details not fully verified.
Evidence: Information architecture clear (project structure, docs links); install notes detailed (pip install); naming stable (version 0.9.0); examples abundant (quick start, proxy mode); known limitations clear (proxy mode does not compact history); license MIT; changelog exists but not detailed; maintenance responsibility clear (author). Deductions: naming stability deducted due to breaking change in 0.9.0; changelog not detailed.
Evidence: Output usability high (proxy mode compatible with OpenAI and Anthropic); marginal value clear (improves model reliability); cost-benefit reasonable (local models reduce API costs). Deductions: no major deductions, but cost-benefit not quantified.
Evidence: README claims eval results with links (Hugging Face dataset), but raw data not provided; cross-source verification limited; fact vs inference separation unclear. Deductions: claimed eval results not fully traceable, cross-source verification insufficient.
- Proxy server does not authenticate callers; add your own authentication layer when deploying on untrusted networks.
- Sensitive data handling (e.g., API keys, user data) is not explicitly documented; assess before use.
- Version 0.9.0 includes breaking changes; read the migration guide before upgrading.
What does this agent do, and when should you use it?
Forge is a Python framework focused on making tool calls from self-hosted LLMs reliable. It offers three usage modes: as a proxy server (speaking both OpenAI and Anthropic API formats), as a WorkflowRunner for structured agent loops, and as guardrails middleware for integration into existing loops. Forge supports multiple backends including OpenAI-compatible endpoints, Ollama, llama-server, Llamafile, vLLM, and Anthropic. Core capabilities include response validation, rescue parsing (handling malformed tool calls), retry loops, and optional step enforcement. Forge is not an agent orchestrator nor a coding harness, but a reliability layer inside a single agent loop. It claims to lift an 8B local model from single digits to 84% and Sonnet 4.6 from 85% to 98% on its eval suite.
Forge provides several operational modes: python -m forge.proxy starts a proxy server that listens on OpenAI chat-completions and Anthropic Messages APIs, forwards requests to a local model backend, and transparently applies guardrails, including response validation (checking tool calls against the tools array), rescue parsing (extracting tool calls from code fences or XML tags), retry loops (up to 3 attempts), and optional respond tool injection. The WorkflowRunner class takes a Workflow object with tool definitions and optional step constraints (required_steps, prerequisites, terminal_tool) and manages the full lifecycle: system prompts, tool execution, context compaction (via ContextManager and TieredCompact strategy), and guardrails. The Guardrails facade allows applying guardrails in foreign loops. Forge connects to backends via LlamafileClient, OllamaClient, OpenAICompatClient, VLLMClient, and AnthropicClient.
- Developers using existing harnesses like opencode, Continue, aider, or Cline who want guardrails via the
python -m forge.proxydrop-in proxy without rewriting code. - Teams building multi-step agentic workflows with
WorkflowRunnerandSlotWorkerfor shared GPU slots with priority queuing and preemption. - Users needing reliable tool calling on local models, avoiding malformed outputs like Mistral's
[TOOL_CALLS]or Qwen's XML. - Claude Code users who want to use a local model while preserving the Anthropic Messages API via the proxy with
ANTHROPIC_BASE_URLset. - Researchers evaluating model+backend reliability using the eval harness (
tests.eval.eval_runnerandbatch_eval) on a standardized scenario suite. - Engineers who want to integrate guardrails into their own orchestration loop using the
Guardrailsmiddleware without adopting the fullWorkflowRunner.
What are this agent's strengths and limitations?
- Multiple guardrails: response validation, rescue parsing, and retry loops significantly improve tool-calling success on self-hosted models.
- Three integration modes: proxy server,
WorkflowRunner, andGuardrailsmiddleware suit different needs. - Backend flexibility: supports OpenAI-compatible endpoints, Ollama, llama-server, Llamafile, vLLM, and Anthropic.
- Includes an eval harness (
eval_runner,batch_eval) and a Hugging Face dataset for measuring performance. - Supports both Anthropic and OpenAI API formats, including Claude Code integration.
- Requires Python 3.12+, which adds runtime requirements for older environments.
- Proxy mode is single-request and does not support multi-turn workflow features like context compaction or step enforcement—those are only in
WorkflowRunner. - Managed mode: stopping Forge does not stop or own the Ollama daemon, and some Ollama options are rejected.
- Benchmark data may be dated: Anthropic numbers measured in v0.6.0, not re-run in v0.7.0.
- Proxy server does not authenticate callers; assumes a trusted network.
How do you install or deploy this agent?
Install Forge: pip install forge-guardrails or pip install "forge-guardrails[anthropic]". Requires Python 3.12+. For development: git clone https://github.com/antoinezambelli/forge.git && cd forge && pip install -e ".[dev]". Choose a backend: llama-server (recommended), Ollama, or Anthropic. For llama-server, download from llama.cpp releases and run: llama-server -m path/to/Ministral-3-8B-Instruct-2512-Q8_0.gguf --jinja -ngl 999 --port 8080. For Ollama: ollama pull ministral-3:8b-instruct-2512-q4_K_M. For Anthropic: set ANTHROPIC_API_KEY.
How do you use this agent?
Start a backend, then run Forge in proxy mode: python -m forge.proxy --backend-url http://localhost:8080 --backend llamaserver --port 8081. Configure your client to use http://localhost:8081/v1 as the API base URL. Alternatively, use WorkflowRunner: import Workflow, ToolDef, ToolSpec, WorkflowRunner, LlamafileClient, ContextManager, and TieredCompact. Define a Workflow object with tools and optional step constraints, and run await runner.run(workflow, "your prompt"). For evaluations, use python -m tests.eval.eval_runner or batch_eval to generate JSONL reports.
How does this agent compare with similar options?
Forge is often compared to managed offerings from OpenAI and Anthropic, but it is not an orchestrator like LangChain or AutoGen. It positions itself as a specialized layer for improving tool-calling reliability, not a full agent framework.
FAQ
Is Forge an agent framework?
Can Forge be used with Claude Code?
/v1/messages), so you can set ANTHROPIC_BASE_URL=http://localhost:8081 and ANTHROPIC_AUTH_TOKEN=anything to point Claude Code at a local model.Does Forge manage conversation history?
WorkflowRunner.