Forge: Reliability Layer for Self-Hosted LLM Tool Calling
A Python framework that adds guardrails to self-hosted LLM tool calling, boosting multi-step agentic workflow success rates.
- Source repo
- antoinezambelli/forge
- Stars
- ★ 2.3k
- Last updated
- 4d ago
- License
- MIT
- Primary language
- Python
- FA score
- 49/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- Developers using existing harnesses like opencode, Continue, aider, or Cline who want guardrails via the
python -m forge.proxydrop-in proxy without rewriting code. - Not a fit if
- Teams needing multi-agent orchestration or DAG planners
- Users unwilling to run a local model backend or GPU
- Teams needing caller authentication on the proxy
- Source review
- 49/100 · Major gaps 3 safety controls not found
What does this agent do, and when should you use it?
Forge is a Python framework focused on making tool calls from self-hosted LLMs reliable. It offers three usage modes: as a proxy server (speaking both OpenAI and Anthropic API formats), as a WorkflowRunner for structured agent loops, and as guardrails middleware for integration into existing loops. Forge supports multiple backends including OpenAI-compatible endpoints, Ollama, llama-server, Llamafile, vLLM, and Anthropic. Core capabilities include response validation, rescue parsing (handling malformed tool calls), retry loops, and optional step enforcement. Forge is not an agent orchestrator nor a coding harness, but a reliability layer inside a single agent loop. It claims to lift an 8B local model from single digits to 84% and Sonnet 4.6 from 85% to 98% on its eval suite.
Forge provides several operational modes: python -m forge.proxy starts a proxy server that listens on OpenAI chat-completions and Anthropic Messages APIs, forwards requests to a local model backend, and transparently applies guardrails, including response validation (checking tool calls against the tools array), rescue parsing (extracting tool calls from code fences or XML tags), retry loops (up to 3 attempts), and optional respond tool injection. The WorkflowRunner class takes a Workflow object with tool definitions and optional step constraints (required_steps, prerequisites, terminal_tool) and manages the full lifecycle: system prompts, tool execution, context compaction (via ContextManager and TieredCompact strategy), and guardrails. The Guardrails facade allows applying guardrails in foreign loops. Forge connects to backends via LlamafileClient, OllamaClient, OpenAICompatClient, VLLMClient, and AnthropicClient.
- Developers using existing harnesses like opencode, Continue, aider, or Cline who want guardrails via the
python -m forge.proxydrop-in proxy without rewriting code. - Teams building multi-step agentic workflows with
WorkflowRunnerandSlotWorkerfor shared GPU slots with priority queuing and preemption. - Users needing reliable tool calling on local models, avoiding malformed outputs like Mistral's
[TOOL_CALLS]or Qwen's XML. - Claude Code users who want to use a local model while preserving the Anthropic Messages API via the proxy with
ANTHROPIC_BASE_URLset. - Researchers evaluating model+backend reliability using the eval harness (
tests.eval.eval_runnerandbatch_eval) on a standardized scenario suite. - Engineers who want to integrate guardrails into their own orchestration loop using the
Guardrailsmiddleware without adopting the fullWorkflowRunner.
How do you install or deploy this agent?
Install Forge: pip install forge-guardrails or pip install "forge-guardrails[anthropic]". Requires Python 3.12+. For development: git clone https://github.com/antoinezambelli/forge.git && cd forge && pip install -e ".[dev]". Choose a backend: llama-server (recommended), Ollama, or Anthropic. For llama-server, download from llama.cpp releases and run: llama-server -m path/to/Ministral-3-8B-Instruct-2512-Q8_0.gguf --jinja -ngl 999 --port 8080. For Ollama: ollama pull ministral-3:8b-instruct-2512-q4_K_M. For Anthropic: set ANTHROPIC_API_KEY.
How do you use this agent?
Start a backend, then run Forge in proxy mode: python -m forge.proxy --backend-url http://localhost:8080 --backend llamaserver --port 8081. Configure your client to use http://localhost:8081/v1 as the API base URL. Alternatively, use WorkflowRunner: import Workflow, ToolDef, ToolSpec, WorkflowRunner, LlamafileClient, ContextManager, and TieredCompact. Define a Workflow object with tools and optional step constraints, and run await runner.run(workflow, "your prompt"). For evaluations, use python -m tests.eval.eval_runner or batch_eval to generate JSONL reports.
What are this agent's strengths and limitations?
- Multiple guardrails: response validation, rescue parsing, and retry loops significantly improve tool-calling success on self-hosted models.
- Three integration modes: proxy server,
WorkflowRunner, andGuardrailsmiddleware suit different needs. - Backend flexibility: supports OpenAI-compatible endpoints, Ollama, llama-server, Llamafile, vLLM, and Anthropic.
- Includes an eval harness (
eval_runner,batch_eval) and a Hugging Face dataset for measuring performance. - Supports both Anthropic and OpenAI API formats, including Claude Code integration.
- Requires Python 3.12+, which adds runtime requirements for older environments.
- Proxy mode is single-request and does not support multi-turn workflow features like context compaction or step enforcement—those are only in
WorkflowRunner. - Managed mode: stopping Forge does not stop or own the Ollama daemon, and some Ollama options are rejected.
- Benchmark data may be dated: Anthropic numbers measured in v0.6.0, not re-run in v0.7.0.
- Proxy server does not authenticate callers; assumes a trusted network.
How does this agent compare with similar options?
Forge is often compared to managed offerings from OpenAI and Anthropic, but it is not an orchestrator like LangChain or AutoGen. It positions itself as a specialized layer for improving tool-calling reliability, not a full agent framework.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Forge: Reliability Layer for Self-Hosted LLM Tool Calling This agent | 49 · Major gaps | CLIFree + model costs | ★ 2.3k | 4d ago | Python | OpenAI API · Claude API |
| little-coder: Lightweight Local Coding Agent | 58 · Major gaps | CLIFree + model costs | ★ 2.7k | 1mo ago | TypeScript | — |
| Functionary: Conversational LLM for Controlled Tool Use | 30 · Major gaps | Self-hosted serviceFree | ★ 1.6k | 3mo ago | Python | OpenAI API |
| Synapse AI — Multi-Agent Orchestration Platform | 70 · Some gaps | CLIFree + model costs | ★ 329 | 2mo ago | Python | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence: Proxy does not authenticate callers but provides --backend-api-key for backend auth; tool calls require user definition but no user confirmation mechanism; data flow transparency partially shown (e.g., proxy mode does not compact history); sensitive data handling not mentioned; dependencies only pydantic and httpx, but no vulnerability scan evidence; external effects include proxy can spawn backend processes, but docs say stopping unloads Ollama model; no rollback mechanism; source attribution clear (author and license). Deductions: user confirmation missing, sensitive data handling not mentioned, rollback missing.
Evidence: Project structure clear, modules well-organized; dependencies pydantic and httpx are common, but no version pinning; error handling has ForgeError hierarchy, but failure message quality not detailed. Deductions: dependency availability not fully verified, failure messages not detailed.
Evidence: Clear target audience (developers), scenarios include proxy server, WorkflowRunner, middleware; capability boundaries clear (not orchestrator, not coding tool); trigger conditions clear (guardrails applied on tool calls); environment fit supports multiple backends and Python versions. Deductions: no major deductions, but environment fit details not fully verified.
Evidence: Information architecture clear (project structure, docs links); install notes detailed (pip install); naming stable (version 0.9.0); examples abundant (quick start, proxy mode); known limitations clear (proxy mode does not compact history); license MIT; changelog exists but not detailed; maintenance responsibility clear (author). Deductions: naming stability deducted due to breaking change in 0.9.0; changelog not detailed.
Evidence: Output usability high (proxy mode compatible with OpenAI and Anthropic); marginal value clear (improves model reliability); cost-benefit reasonable (local models reduce API costs). Deductions: no major deductions, but cost-benefit not quantified.
Evidence: README claims eval results with links (Hugging Face dataset), but raw data not provided; cross-source verification limited; fact vs inference separation unclear. Deductions: claimed eval results not fully traceable, cross-source verification insufficient.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Proxy server does not authenticate callers; add your own authentication layer when deploying on untrusted networks.
- Sensitive data handling (e.g., API keys, user data) is not explicitly documented; assess before use.
- Version 0.9.0 includes breaking changes; read the migration guide before upgrading.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v0.9.5 v0.9.5: Equivalent dual-auth compatibility
Release notes excerpt
[0.9.5] — 2026-08-29 A credential-interoperability release for Forge's direct clients and standalone Proxy. Equivalent representations of one credential can now pass through without weakening Forge's rejection of genuinely distinct credentials. Changed - Equivalent dual-auth headers represent one credential identity. Direct clients and Forge Proxy now accept Authorization plus X-Api-Key when both carry the same normalized token. Same-protocol traffic preserves both headers; cross-protocol traffic emits one canonical target header. Unequal …View on GitHub ↗ -
v0.9.4 v0.9.4: Anthropic reasoning capture
Release notes excerpt
[0.9.4] — 2026-08-26 An Anthropic interoperability release for Forge Proxy and direct Anthropic client usage. Native plaintext thinking returned alongside tool calls now participates in Forge's existing reasoning-replay policy. Added - Anthropic tool-call reasoning is captured consistently. Streamed and non-streamed native thinking content is mapped to ToolCall.reasoning, preferred over the existing visible-text fallback, and exposed through Forge Proxy's OpenAI-compatible responses according to the configured …View on GitHub ↗ -
v0.9.3 v0.9.3: Proxy command ownership hotfix
Release notes excerpt
[0.9.3] — 2026-08-21 A command-ownership hotfix for the standalone Forge Proxy distribution. Proxy request handling, routing, backend behavior, and guardrail policy are unchanged. Fixed - The standalone installer exclusively owns forge-proxy. The forge-guardrails Python package no longer creates a competing global command; Python-managed Proxy execution remains available through python -m forge.proxy, including in the Docker image. - Foreign commands are preserved. Installation and update refuse when an …View on GitHub ↗
FAQ
Is Forge an agent framework?
Can Forge be used with Claude Code?
/v1/messages), so you can set ANTHROPIC_BASE_URL=http://localhost:8081 and ANTHROPIC_AUTH_TOKEN=anything to point Claude Code at a local model.Does Forge manage conversation history?
WorkflowRunner.