Automation & Ops observability tracing evaluation monitoring prompt-management

Arize Phoenix

Open-source AI observability platform for tracing, evaluating, and troubleshooting LLM applications.

Source repo: Arize-ai/phoenix
Stars: ★ 12k
Last updated: 2d ago
License: NOASSERTION
Primary language: Python
FA score: 61/100 · Some gaps

At a glance

How it runs: CLISelf-hosted serviceWeb app
Works with: Universal · cross-platformOpenAI API · Claude API
Cost: Free tier plus a paid hosted plan
Setup effort: Low · running in minutes
You'll need: Python 3.8+DockerHelmShell / CLINetwork accessLocal filesystem
Typical use: AI engineers debugging response quality issues in production LLM applications.
Not a fit if: Teams that must run fully offline with no external LLM APIs
Teams wanting a managed service without maintaining Docker/Kubernetes
Production teams needing a stable TypeScript evals library (still alpha)
Source review: 61/100 · Some gaps

What does this agent do, and when should you use it?

Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. It provides OpenTelemetry-based tracing to monitor LLM application runtime, LLM-driven evaluations for response and retrieval quality, versioned datasets, experiments, a prompt playground, and an integrated AI engineering agent (PXI). It is vendor- and language-agnostic, with out-of-the-box support for popular frameworks and LLM providers like OpenAI, Anthropic, and Google GenAI. Phoenix can run locally, in containers, or in the cloud, with Docker images and Helm charts available.

Phoenix offers comprehensive observability: it captures traces of LLM applications via OpenTelemetry, supporting frameworks like LangChain, LlamaIndex, and OpenAI Agents SDK. It allows users to evaluate responses and retrieval quality using LLMs as evaluators. Users can create versioned datasets, run experiments to test prompt and model changes, and use a playground for prompt optimization. PXI (Phoenix Intelligence) is a built-in AI engineering agent for debugging traces, iterating on prompts, and navigating the product. It also includes a remote MCP server, enabling clients like Claude Code and Cursor to query traces, datasets, and experiments.

AI engineers debugging response quality issues in production LLM applications.
ML engineers comparing different prompts or models on specific datasets.
Development teams integrating LLM tracing into existing OpenTelemetry monitoring stacks.
Product managers quantifying retrieval and answer relevance for LLM applications.
Coding agents like Claude Code needing access to traces and datasets for debugging.

How do you install or deploy this agent?

To install Phoenix, you need Python 3.8+. Install via pip: pip install arize-phoenix. Then start the server: phoenix serve. Alternatively, you can run without install using uvx arize-phoenix serve.

How do you use this agent?

After installation, start Phoenix locally: phoenix serve. To trace your application, use the CLI to auto-detect and set up: run npx @arizeai/phoenix-cli setup (requires Node.js) or px setup (if Phoenix installed) from your project directory. This installs OpenInference instrumentation and configures trace export in your app code. Then, your app can send traces to Phoenix, which you can view in the UI. For deployment, you can use Docker: docker run -p 6006:6006 arizephoenix/phoenix.

What are this agent's strengths and limitations?

Pros

Framework and language agnostic with broad integrations (Python, JavaScript, Java, Go).
Built on OpenTelemetry, easy to integrate into existing observability setups.
Offers comprehensive LLM-driven evaluation for response and retrieval quality.
Supports versioned datasets and experiments for systematic prompt and model improvement.

Limitations

Self-hosting requires maintaining infrastructure (Docker, Kubernetes) and may incur costs if cloud-hosted.
Evaluation features rely on LLM APIs, incurring potential additional costs.
Full setup may have a learning curve involving OpenTelemetry and API integrations.

How does this agent compare with similar options?

Phoenix positions itself as an AI observability platform; it does not directly name competitors, but based on the description, it competes with Langfuse, LangSmith, and Arize's cloud offering.

Key facts side by side with the most closely related agents.

Agent	Source review	Form / cost	Stars	Updated	Language	Full support on
Arize Phoenix This agent	61 · Some gaps	CLIFreemium	★ 12k	2d ago	Python	OpenAI API · Claude API
RagaAI Catalyst	46 · Major gaps	Library / SDKFree + model costs	★ 16k	1y ago	Python	—
Hermes HUD Web UI	55 · Major gaps	CLIFree + model costs	★ 1.8k	7d ago	Python	—
MLflow	64 · Some gaps	CLIFree + model costs	★ 28k	today	Python	Claude Code · OpenAI API · Claude API

How does FollowAgents rate this agent?

FollowAgents source review · FARS-2.1

Some gaps

61/ 100 5-point scale 3.1 / 5

This review covers the source as of Aug 09, 2026. The project has shipped 20 stable release(s) since, which have not been re-reviewed. The score and verdict describe the reviewed version only. Recent releases ↓

Trust 14/29

Reliability 9/14

Adaptability 12/18

Convention 12/18

Effectiveness 9/13

Verifiability 5/8

Why each dimension lost points

Trust14 / 29 · 2.4/5

Evidence shows role-based permissions (admin/member/viewer) and API key management, but least privilege principle not explicit; user confirmation exists (e.g., confirm when creating keys) but not comprehensive; data flow transparency good with tracing and telemetry docs; sensitive data handling has security policy and key management but not deep; dependency security has version pins and CVE comments but not full audit; external effects have sandbox and deployment options but side effects not explicit; rollback has version control and migrations but rollback mechanism not explicit; source attribution has authors and docs but publisher unverified.

Reliability9 / 14 · 3.2/5

Self-consistency good, docs and code align; dependency availability has version pins and compatibility notes; failure messages have error handling docs and tests but not comprehensive.

Adaptability12 / 18 · 3.3/5

Audience clear (developers, data scientists), scenarios diverse (tracing, evaluation, experiments); capability boundaries documented; trigger precision has integration and config docs; environment fit has local, container, cloud deployment options.

Convention12 / 18 · 3.3/5

Information architecture clear with TOC and docs; install notes detailed; naming stable with versions and package names; examples and FAQ documented; known limitations stated; license clear (Elastic-2.0); versioning changelog present; maintenance responsibility has community and support channels.

Effectiveness9 / 13 · 3.5/5

Output usability high with API and CLI; marginal value high with unique features; cost-benefit reasonable as open-source free.

Verifiability5 / 8 · 3.1/5

Claims traceable with docs and code; cross-source corroboration with multiple integrations and community; fact-inference separation, docs distinguish feature descriptions from speculation.

Risks and how to mitigate them

Publisher identity unverified, caution with supply chain risks.
Dependency security has version pins but no full security audit report.
Rollback mechanism not explicit, relies on external version control.

Evidence confidence: Low Reviewed Aug 09, 2026 Reviewed revision b4d9b19e6c68 New commits since this review; the score may not cover them

Review evidence README.md LICENSE SECURITY.md pyproject.toml workflows/auto-label-external-issues.yaml workflows/check-card-links.yml tests/admin-access.ts tests/auth-refresh.spec.ts tests/auth.setup.ts

See the full review method →

Recent releases

Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.

arize-phoenix-v20.23.0 arize-phoenix: v20.23.0 2026-10-16

Release notes excerpt

[20.23.0](https://github.com/Arize-ai/phoenix/compare/arize-phoenix-v20.22.0...arize-phoenix-v20.23.0) (2026-10-16) Features * server: export server logs over OTLP ([#17010](https://github.com/Arize-ai/phoenix/issues/17010)) ([add77bf](https://github.com/Arize-ai/phoenix/commit/add77bf75b6acd36ea93c7364e09a31fe276649b)) Bug Fixes * playground: forward Bedrock tool results directly as text ([#16951](https://github.com/Arize-ai/phoenix/issues/16951)) ([15355fd](https://github.com/Arize-ai/phoenix/commit/15355fd1a8a62c7218b1c5e434541b8976b3f62c)) Documentation …
View on GitHub ↗
arize-phoenix-v20.22.0 arize-phoenix: v20.22.0 2026-10-15

Release notes excerpt

[20.22.0](https://github.com/Arize-ai/phoenix/compare/arize-phoenix-v20.21.4...arize-phoenix-v20.22.0) (2026-10-15) Features * playground: add Meta muse-wonder-1.0 and gemini-4-argon ([#16976](https://github.com/Arize-ai/phoenix/issues/16976)) ([d5168d1](https://github.com/Arize-ai/phoenix/commit/d5168d1645198ad7b7a47f1d597170e4b291ecef)) Bug Fixes …
View on GitHub ↗
arize-phoenix-v20.21.4 arize-phoenix: v20.21.4 2026-10-14

Release notes excerpt

[20.21.4](https://github.com/Arize-ai/phoenix/compare/arize-phoenix-v20.21.3...arize-phoenix-v20.21.4) (2026-10-14) Bug Fixes * ui: keep labels on Escape and fix table sort, a11y and accessor warnings ([#16980](https://github.com/Arize-ai/phoenix/issues/16980)) ([53a1735](https://github.com/Arize-ai/phoenix/commit/53a173532c18ac0fd04370cdfcd1ea4cfefdd706))
View on GitHub ↗

FAQ

Does Phoenix collect my data for telemetry?

Yes, but by default it only collects basic web analytics (e.g., page views), not trace data or evaluation results. You can opt out by setting the environment variable PHOENIX_TELEMETRY_ENABLED=false.

Do I need to pay for LLM evaluations?

Yes, the evaluation functionality uses LLMs to analyze your application's responses, which incurs API call costs through your chosen LLM provider.

Is Phoenix only for production monitoring or also for development?

Both. It is designed for experimentation (playground, datasets, experiments) and troubleshooting (traces, PXI), so it covers development and production workflows.

Can I integrate Phoenix with my existing monitoring system?

Yes, because it is based on OpenTelemetry, you can export traces to other backends supporting OTLP, or use Phoenix as a standalone system.

Arize Phoenix

At a glance

What does this agent do, and when should you use it?

How do you install or deploy this agent?

How do you use this agent?

What are this agent's strengths and limitations?

How does this agent compare with similar options?

How does FollowAgents rate this agent?

Recent releases

FAQ

Related agents

RagaAI Catalyst

Hermes HUD Web UI

MLflow

Coze Loop