Tracely

What does this agent do, and when should you use it?

Tracely is a self-hostable observability and regression-testing platform built specifically for AI agents. Traces arrive over OTLP/HTTP from any language, and agent semantics such as agent.id, conversation.id, turn and step are promoted to first-class indexed columns in ClickHouse, so runs group into conversation threads instead of flat span noise. Online evaluators grade every run as it lands — structural checks that need no model, plus LLM-as-judge at conversation, run or span level — and failures cluster into issues with counts rather than hundreds of rows to read. One click freezes a failing trace into a hermetic case with a fail-to-pass contract, and CI replays it against recorded tool and LLM fixtures: offline, no API keys, no model spend. tracely gate then exits non-zero, posts a commit status and upserts a PR comment. The stack itself is a FastAPI backend, Celery/Redis workers, a Next.js frontend, Postgres, ClickHouse and MinIO, deployable via Docker Compose, Railway, Coolify or EKS.

Data enters through the SDK or any OTLP/HTTP exporter at POST /v1/traces, lands in S3/MinIO first for durability, moves through Redis/Celery, and a worker performs OTel mapping, registry upserts and ClickHouse event writes before running evaluate_run_task to produce scores and structural clustering and to fire alert flows. Evaluators are columns on the trace table: no-model structural checks plus LLM judges at conversation, run and span level, with scores streaming in live over SSE. Failures cluster structurally and semantically into issues, and each cluster can suggest the evaluator that would have caught it. Promoting a failing trace produces a hermetic case bundling recorded input, tool and LLM outputs as fixtures with a fail-to-pass contract; multi-turn behaviour uses scenarios, either scripted conversations Tracely drives at your agent's endpoint or an adversarial goal a red-team model pursues. On the CI side, tracely simulate, tracely replay and tracely gate plus a GitHub Action deliver per-case verdicts, a commit status and a PR comment. Alerts are a trigger plus a visual flow of steps: condition, Slack, email, templated webhook, LLM prompt and Python expression.

1. A production support agent occasionally answers order questions wrong: ingest its OTLP traces, let a conversation-level judge grade each run, cluster the failures into one issue, then promote it to a regression case so it can never silently return.
2. Debugging a multi-agent system: read the agent → thinking → skill → generation → hand-off waterfall with the failing span highlighted, and use the Fleet pixel-art view to explain hand-offs to non-engineers.
3. Wiring a merge gate into GitHub Actions so a PR that reintroduces a known failure mode is blocked with a non-zero exit, a commit status and a results comment.
4. A team whose CI cannot make live model calls: replay promoted cases hermetically against recorded fixtures, getting deterministic verdicts at zero API cost.
5. Calibrating judges before trusting them: label judge verdicts against human review and read per-evaluator agreement to catch an over-flagging judge ahead of release.
6. A compliance-bound organisation that needs traces and eval data to stay in its own network: self-host the whole stack via Docker Compose, Railway, Coolify or EKS.
7. Letting a coding agent drive the workspace: point Claude Code or another MCP client at the /mcp endpoint to read traces and create evaluators, cases and alerts with an ordinary ingest key.

How do you install or deploy this agent?

The fastest path to the whole stack is Docker Compose, which brings up ClickHouse, Postgres, Redis, MinIO, the API, the worker and the frontend and seeds demo data:

git clone https://github.com/jwuthri-cloud/Tracely-ai && cd Tracely
docker compose --profile demo up -d --build --wait
open http://localhost:3001

For local development you need uv and Node 20+/pnpm:

cp .env.example .env
make infra-up      # clickhouse, postgres, redis, minio
make install       # uv sync + pnpm install
make migrate       # ClickHouse DDL + Alembic (Postgres)
make seed          # default project + ingest key → tracely_dev_key

The instrumentation SDK:

pip install "tracelyai-sdk[openai]"     # or [anthropic], [langchain], [all]

For hosted deployment, a Railway one-click template provisions the full stack; Coolify users run python3 deploy/coolify/setup.py --coolify-url … --app-domain … --api-domain …; Kubernetes manifests live under deploy/eks/README.md. Set SESSION_SECRET and SECRETS_ENCRYPTION_KEY (openssl rand -hex 32 each) when prompted; AUTH_MODE defaults to dev, which is login-free.

How do you use this agent?

Initialize the SDK once at startup and wrap a run — provider calls are captured automatically:

import tracely_sdk as tracely

tracely.init(
    endpoint="http://localhost:8000",
    api_key="tracely_dev_key",
    service_name="support-agent",
    env="prod",
    instrument="auto",
)

with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Where is order ORD-4471?"}],
    )

Any OTLP/HTTP exporter works too — point it at POST {endpoint}/v1/traces with Authorization: Bearer tracely_dev_key. To gate PRs with Tracely driving your registered agent endpoint:

      - uses: jwuthri-cloud/Tracely-ai/.github/actions/tracely-gate@master
        with:
          api:   https://tracely.your-co.dev
          key:   ${{ secrets.TRACELY_KEY }}

Or replay recorded cases hermetically:

pip install tracelyai-sdk
tracely replay planner --entrypoint my_pkg.agent:run

And connect a coding agent over MCP:

claude mcp add --transport http tracely http://localhost:8000/mcp \
  --header "Authorization: Bearer tracely_dev_key"

What are this agent's strengths and limitations?

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

How does FollowAgents rate this agent?

FAQ