Monocle GenAI Tracing Framework
An OpenTelemetry-based tracing and testing framework for GenAI apps and agents: turn black-box agent runs into structured, assertable OTLP traces with a few lines of code — or none at all.
- Source repo
- monocle2ai/monocle
- Stars
- ★ 343
- Last updated
- 4d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 58/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free software; you pay for model usage
- Setup effort
- Low · running in minutes
- You'll need
- Typical use
- An app developer who wants to trace every model call and tool invocation in a LangChain or LlamaIndex agent across environments without hand-writing OpenTelemetry code.
- Not a fit if
- Teams whose observability stack does not consume OTLP, which requires extra adaptation
- Teams needing non-Python implementations, as this repo is Python-only
- Source review
- 58/100 · Major gaps
What does this agent do, and when should you use it?
Monocle is an open-source project under the Linux Foundation AI & Data umbrella; this repository hosts its Python implementation (monocle_apptrace), a GenAI-specific observability layer built on OpenTelemetry. It comprises a metamodel that maps entities like agents, prompts, tools, and vector operations to standardized span attributes, instrumentation adapters that auto-create spans for major GenAI frameworks, and OTLP-compatible trace export. A single setup_monocle_telemetry() call auto-instruments frameworks such as LangChain, LlamaIndex, Google ADK, OpenAI, and Anthropic, exporting traces by default to local JSON files or any OTEL-compatible backend. The companion monocle_test_tools package, built on pytest, lets you assert on traces themselves — agent invocations, tool behavior, token costs — not just input/output pairs. It targets app developers, platform engineers, and enterprise SRE teams who need debuggability, reliability, and compliance for AI agents.
After pip-installing monocle_apptrace, one call to setup_monocle_telemetry(workflow_name=...) wires up OpenTelemetry, configures the Monocle metamodel, and auto-instruments supported frameworks; adapters create standardized spans for agent runs, tool calls, LLM invocations, and retrieval queries. Traces are exported by default as JSON files under ./monocle (monocle_trace_{workflow}_{trace_id}_{timestamp}.) or to stdout, Azure Blob, AWS S3, GCS, PostgreSQL, ClickHouse, Okahu cloud, and more. monocle_trace_scope / monocle_trace_scope_method attach business context like user_id or tenant_id to spans. The monocle_test_tools package provides fluent assertions via expected() (called_agent, called_tool, under_token_limit, under_duration) and multi-turn session evaluation via MonocleValidator; a zero-code wrapper mode (monocle-apptrace claude-setup / codex-setup / copilot-setup) registers hooks for Claude CLI, Codex CLI, and GitHub Copilot.
- An app developer who wants to trace every model call and tool invocation in a LangChain or LlamaIndex agent across environments without hand-writing OpenTelemetry code.
- A platform team that needs to inject tracing into a multi-tenant AI platform via wrapper mode without touching product code, separating tenants with tenant_id scopes.
- A QA engineer using monocle_test_tools in CI/CD to assert that agents called the right tools, outputs match expectations, and token usage stays under budget.
- An SRE team sending GenAI traces into an existing OTEL collector, APM, or Okahu cloud for production monitoring and debugging.
- A developer using Claude Code, Codex CLI, or GitHub Copilot who wants visibility into how these coding assistants interact with a codebase.
- A team evaluating multi-turn session quality (role adherence, knowledge retention) by aggregating traces under a session_id and running agentic_sessions evaluations.
How do you install or deploy this agent?
pip install monocle_apptrace# Testing framework (optional)
pip install monocle_test_tools# Hook tracing into AI coding CLIs (optional)
uv tool install monocle_apptrace
monocle-apptrace claude-setup # or codex-setup / copilot-setupRequires a Python runtime; no mandatory cloud credentials — local file export works out of the box. Okahu cloud export needs a GitHub sign-in or a pasted API key.
How do you use this agent?
from monocle_apptrace import setup_monocle_telemetry
setup_monocle_telemetry(workflow_name="simple_math_app")# Run your app, then inspect JSON trace files under ./monocle/# or visualize them with the Okahu VS Code extension.
# Business context:
from monocle_apptrace.instrumentation.common.instrumentor import monocle_trace_scopewith monocle_trace_scope("user_id", "user-123"):
result = my_agent.run("What's the weather in London?")# Testing:
from monocle_test_tools import expecteddef test_weather_agent():
result = expected(input="What is the weather in London?",
expected_output="weather report for London")
result.called_agent("weather_agent")
result.called_tool("get_weather", agent_name="weather_agent")
result.under_token_limit(5000)What are this agent's strengths and limitations?
- A GenAI-specific metamodel maps agents, tools, and vector operations to standardized span attributes, producing consistent traces consumable by humans, dashboards, and automation.
- Broad first-class framework coverage: LangGraph, LlamaIndex, Google ADK, OpenAI Agent SDK, AWS Strands, CrewAI, Microsoft Agent Framework, FastMCP/A2A are auto-instrumented.
- Speaks OpenTelemetry natively — OTLP spans drop into existing collectors, backends, and dashboards with no tracing-model migration.
- monocle_test_tools asserts on the traces themselves (agent invocations, tool parameters, token cost, session quality) rather than only I/O pairs.
- Zero-code wrapper mode and AI coding-assistant hooks let platform teams inject tracing without modifying product code.
- Non-Python languages require a separate repository (monocle-typescript); this repo is the Python implementation only.
- Several integrations are still pending: Azure ML, Google Function, OpenSearch, and Milvus are marked 🔜 (coming soon).
- Deep reliance on the OpenTelemetry ecosystem requires additional adaptation if your observability stack does not consume OTLP.
- Full evaluation features (the okahu evaluator and cloud export) are tied to the Okahu cloud service, creating ecosystem coupling.
- Vector store support is currently limited to FAISS, constraining choices.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Monocle GenAI Tracing Framework This agent | 58 · Major gaps | Library / SDKFree + model costs | ★ 343 | 4d ago | Python | — |
| LangWatch | 78 · Good | CLIFreemium | ★ 5k | today | TypeScript | OpenAI API · Claude API |
| OpenTelemetry Agent Skills | 54 · Major gaps | Agent plugin / skillFree | ★ 116 | 2d ago | Go | Codex · Claude Code |
| Judgeval | 40 · Major gaps | Library / SDKFree + model costs | ★ 1.1k | 3d ago | Python | OpenAI API |
How does FollowAgents rate this agent?
Why each dimension lost points
Tracing SDK with default local JSON export — a modest privilege surface; deducted because CLI hooks write into ~/.claude/, ~/.codex/ etc. and the hook payloads are not visible in evidence. Interactive *-setup prompts (global vs project, auth choice) give adequate user confirmation (2). Data flow (spans, exporters, metamodel) is well documented (2), but traces capture prompts/responses/user_id by default and no redaction or sensitive-data guidance appears — hence 1. Dependency security is supported by CodeQL, OpenSSF badge, and SECURITY.md (2); however SECURITY.md claims dependency version pinning while the provided requirements.txt pins nothing, a contradiction. External effects (Okahu cloud login, global hook writes) appear opt-in and interactive; 1 because hook persistence behavior cannot be statically verified. No uninstall/rollback instructions (1). Attribution clear: Apache-2.0, LF AI & Data, maintainer links (2).
README, SECURITY.md, and metamodel paths are broadly self-consistent (2); deducted for the pinning claim vs unpinned requirements.txt. Dependency availability is weak: requirements.txt uses bare unpinned names (pytest, mistralai, llama-index) (1). Failure handling in test helper code swallows exceptions with logger.info and returns generic 'Failure' messages (1).
Audiences (app developers, platform engineers, enterprise SREs) and scenarios are well articulated (3). Capability boundaries are explicit via the support matrix with 🟢/🔜 markers (2). Trigger precision is reasonable: auto-instrumentation scoped per framework, scope API semantics documented (2). Environment fit is good with in-code, wrapper, and CLI-hook modes (2); deducted for missing framework version-compatibility ranges.
Good information architecture: docs/, examples/, test_tools/ layers (2). Install notes are direct (2); deducted for naming inconsistency: PyPI package monocle_apptrace vs publishing link monocle-apptrace. Naming is stable overall (2). Rich examples but no FAQ/troubleshooting (2). Known limitations are only implied by roadmap/🔜, no explicit limitations section (1). Full Apache-2.0 license text present (3). No CHANGELOG or version history in evidence (1). Maintenance responsibility: MAINTAINER.md reference and 3-working-day response commitment in SECURITY.md (2).
Output is structured OTel span JSON consumable by any backend and a VS Code extension — good usability (2). Marginal value: asserting on trace facts (agent calls, tokens, duration) is a differentiated capability (2). Cost/benefit looks favorable statically: lightweight init, CI-friendly (2); deducted because the evaluation path depends on the external Okahu service with opaque costs.
Key claims (metamodel path, test-tools path, CLI commands) are traceable to repo files (2). Cross-source corroboration is weak: static file sample only, no execution, and PyPI/source consistency unverified (1). Fact/inference separation is maintained: 🟢 vs 🔜 distinguished, and this review only concludes from visible files (2).
- CLI *-setup installs hooks into global directories (~/.claude/, ~/.codex/, ~/.copilot/) by default; inspect hook contents and prefer --project mode.
- Traces capture prompts/responses and attributes like user_id/tenant_id by default; assess sensitive-data exposure before exporting to cloud or shared storage.
- SECURITY.md claims dependency pinning, but the sample requirements.txt pins nothing; lock versions yourself for production.
- Evaluation and cloud export depend on the Okahu commercial service — note vendor lock-in and data egress.
- No hook uninstall/rollback instructions; verify manual removal steps before installing.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v0.8.15 Version 0.8.15
Release notes excerpt
This is a patch release on the previous 0.8.14 release, fixing the issue(s) below. - fix(instrumentation): a streaming inference whose consumer stops reading early now emits a span — the tokens it spent were previously unreported. close()/aclose() on the stream is the only deterministic end signal, since Python leaves an abandoned generator to the garbage collector, so both the in-place patch path and the fallback wrapper classes emit there, guarded so a close() after full consumption cannot produce a second span ([#818](https://github.com/monocle2ai/monocle/pull/818)) …View on GitHub ↗ -
v0.8.14 Version 0.8.14
Release notes excerpt
This is a patch release on the previous 0.8.13 release, fixing the issue(s) below. - fix(exporters): span obfuscation no longer swallows the sentence punctuation after an unquoted credential value — password=hunter2. Next sentence. now redacts to password=<REDACTED>. Next sentence. instead of password=<REDACTED> Next sentence.. Dots inside the value are still treated as part of the secret …View on GitHub ↗ -
v0.8.13 Version 0.8.13
Release notes excerpt
This is a patch release on the previous 0.8.12 release, fixing the issue(s) below. - feat(test_tools): check_eval's eval_name accepts either a built-in eval template name or the path of a custom eval template JSON file (a pathlib.Path or a path-like string), instead of the caller having to switch to the template_path parameter for custom templates. Which kind it is is detected from the value, using the evaluator's existing built-in vs. custom rule (BaseEval.classify_eval_input) ([#791](https://github.com/monocle2ai/monocle/pull/791)) …View on GitHub ↗