PandaProbe
An engineering platform for tracing, evaluating, monitoring, and debugging AI agents.
- Source repo
- chirpz-ai/pandaprobe
- Stars
- ★ 795
- Last updated
- 4d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 59/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesClaude API (Partial support)
- Cost
- Free tier plus a paid hosted plan
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A LangGraph engineering team that needs centralized trace and span records to investigate agent execution issues.
- Not a fit if
- Teams that need fully offline or on-prem operation with no data leaving their network
- Solo developers unwilling to run a Docker stack of Next.js, FastAPI, Celery, PostgreSQL, and Redis
- Teams wanting evaluation out of the box without configuring an LLM-as-a-judge model
- Source review
- 59/100 · Major gaps
What does this agent do, and when should you use it?
PandaProbe is an open-source agent engineering platform from Chirpz AI for collaborative tracing, evaluation, monitoring, and debugging. Its documented stack includes a Next.js dashboard, FastAPI application server, Celery workers and scheduler, PostgreSQL, and Redis. Data-plane clients submit trace or evaluation work with X-API-Key and X-Project-Name, while Redis queues that work for background processing. Workers persist traces and spans to PostgreSQL; evaluation workers use a LiteLLM-backed LLM-as-a-judge flow to store a verdict and score. Teams can use PandaProbe Cloud or self-host the service with Docker.
A client sends POST /traces or POST /evaluations to the FastAPI API. The service resolves organization and project context through the Identity Service, enqueues ingestion or evaluation work in Redis, and returns 202 Accepted. A Celery Worker consumes the job, persists traces and spans in PostgreSQL, or calls the LiteLLM LLM Engine for an LLM-as-a-judge verdict and score before persisting the evaluation result. GET /traces and GET /sessions return filtered, paginated data. Management-plane requests use a Bearer token authenticated through the Auth Service with Supabase or Firebase.
- A LangGraph engineering team that needs centralized trace and span records to investigate agent execution issues.
- A CrewAI team that wants to enqueue evaluations and retain LLM-as-a-judge scores for its multi-agent workflows.
- A team using the Claude Agent SDK that needs one service for observing and evaluating agent runs.
- An organization with internal deployment requirements that wants to run the dashboard, API, workers, PostgreSQL, and Redis under Docker.
- A platform team that needs project-scoped agent data and can identify data-plane requests with X-API-Key and X-Project-Name.
How do you install or deploy this agent?
Docker must be installed and running. Run:
git clone https://github.com/chirpz-ai/pandaprobe.git
cd pandaprobe
./start.shAfter startup, open the dashboard at http://localhost:3000 and the API reference at http://localhost:8000/scalar.
How do you use this agent?
Start the service, then open http://localhost:3000 for the dashboard. For the data plane, provide X-API-Key and X-Project-Name and submit traces to POST /traces or evaluations to POST /evaluations; both are processed asynchronously and return 202 Accepted. Query filtered, paginated records through GET /traces and GET /sessions. Management-plane requests require a Bearer token. The supplied material does not document API-key provisioning, project creation, or copyable request bodies.
What are this agent's strengths and limitations?
- Trace ingestion and evaluations are queued through Redis and handled by Celery, allowing callers to receive 202 Accepted after submission.
- The documented architecture covers trace/span persistence, trace and session querying, and stored LLM-as-a-judge verdicts and scores.
- It offers both a managed cloud option and a documented Docker self-hosting path with named service components.
- The repository explicitly names integrations with LangGraph, CrewAI, and the Claude Agent SDK.
- Self-hosting requires Docker and a multi-service stack comprising Next.js, FastAPI, Celery, PostgreSQL, and Redis.
- Evaluation depends on a LiteLLM-backed LLM-as-a-judge flow, but the supplied material does not specify supported model providers, model configuration, or cost controls.
- Data-plane writes require X-API-Key and X-Project-Name, while management requests require a Bearer token validated through Supabase or Firebase; credential issuance is not documented here.
- The supplied material does not include client SDK initialization, complete API payloads, or integration configuration examples.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| PandaProbe This agent | 59 · Major gaps | Self-hosted serviceFreemium | ★ 795 | 4d ago | Python | — |
| PenguinHarness | 67 · Some gaps | Desktop appFree + model costs | ★ 3.2k | today | TypeScript | OpenAI API · Claude API |
| Open Swarm | Insufficient evidence | — | ★ 826 | 1mo ago | JavaScript | Claude API |
| Agent Systems Handbook | 62 · Some gaps | Web appFree | ★ 313 | 23d ago | MDX | — |
How does FollowAgents rate this agent?
Why each dimension lost points
The docs describe IdP bearer JWTs for the management plane and project API keys for the data plane, trace/evaluation flows through Redis, workers, the database, and an LLM, and local persistence for Reflect rules and evidence. They also disclose added API/model calls. Deductions: the supplied excerpts do not establish tenant-key scope, trace redaction or retention, or adequate user confirmation and rollback controls; secret-handling details are limited despite external service and model calls.
The README and AGENTS.md broadly agree on the FastAPI, Redis, worker, database, and evaluation architecture. The application entry point fails fast when required Stripe configuration is absent and returns structured domain and validation errors. Deductions: the system depends on external identity providers, Redis, PostgreSQL, Stripe, and LiteLLM, while the packet does not establish availability guarantees or complete fallback behavior; static evidence cannot establish runtime reliability.
The README targets teams tracing, evaluating, and debugging agents and offers cloud and self-hosted deployment. The Reflect quickstart provides Python and TypeScript LangChain examples, while its docs describe custom loops, workspace options, and runtime constraints. Deduction: the supplied integration example is concentrated on LangChain, with no concrete setup for other integrations in this packet.
Repository guidance organizes services, architecture, and development commands; the README gives Docker self-hosting steps; Reflect settings document defaults, limits, supported environments, and limitations; and a full Apache 2.0 license is included. Deductions: the packet contains no release history or changelog, and installation and troubleshooting coverage is not exhaustive.
The platform offers tracing, evaluation, monitoring, and debugging, and the Reflect quickstart shows a consumable report output. Configuration docs explain sampling, concurrency, attempt limits, and added costs. Deductions: evaluation and reflection add latency and cost, the daily evaluation cap is unset by default, and the packet does not quantify benefits.
The README architecture diagram, AGENTS.md repository description, application entry point, and test environment configuration corroborate the main service composition and error handling. Reflect docs specify concrete defaults and boundaries. Deductions: many product capabilities and limits are described in documentation alone, and the packet cannot independently establish runtime outcomes; no execution or independent testing is claimed.
- Reflect persists rules and evidence in a local workspace; confirm directory access, backup, and retention practices before deployment.
- Evaluation and reflection add API/model calls, latency, and cost; the daily evaluation cap is unset by default, so configure a limit if needed.
- Self-hosting involves JWTs, API keys, a database, Redis, and external identity/model services; the supplied material does not fully explain trace redaction or retention.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/pandaprobe)<a href="https://followagents.com/en/agents/pandaprobe"><img src="https://followagents.com/badge/pandaprobe.svg" alt="FARS review"></a>Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
What's Changed * Feat/python reflect judge guidance by @sinatayebati in https://github.com/chirpz-ai/pandaprobe/pull/136 Full Changelog: https://github.com/chirpz-ai/pandaprobe/compare/v0.7.0...v0.7.1View on GitHub ↗ -
Release notes excerpt
What's Changed * Fix/redis timeout by @sinatayebati in https://github.com/chirpz-ai/pandaprobe/pull/131 * Docs by @sinatayebati in https://github.com/chirpz-ai/pandaprobe/pull/133 * Fix/stripe invoice webhooks by @sinatayebati in https://github.com/chirpz-ai/pandaprobe/pull/135 Full Changelog: https://github.com/chirpz-ai/pandaprobe/compare/v0.6.1...v0.7.0View on GitHub ↗ -
Release notes excerpt
What's Changed * refining eval by @sinatayebati in https://github.com/chirpz-ai/pandaprobe/pull/127 * Fix/stripe invoice webhooks by @sinatayebati in https://github.com/chirpz-ai/pandaprobe/pull/130 Full Changelog: https://github.com/chirpz-ai/pandaprobe/compare/v0.6.0...v0.6.1View on GitHub ↗
FAQ
Can PandaProbe be self-hosted?
Are trace and evaluation requests completed synchronously?
What credentials do data requests require?
What does an evaluation produce?
Thanks — we use this to improve the page.