MLflow
Trace, evaluate, monitor, and optimize production agents, LLM applications, and machine-learning models.
- Source repo
- mlflow/mlflow
- Stars
- ★ 28k
- Last updated
- today
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 64/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformClaude Code · OpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- An agent engineering team needs complete execution traces to diagnose model responses, tool calls, and multi-step behavior without a major application rewrite.
- Not a fit if
- Teams wanting a single-language tool without operating a separate MLflow server
- Users needing documented production auth, storage, and high-availability setup
- Source review
- 64/100 · Some gaps
What does this agent do, and when should you use it?
MLflow is an open-source AI engineering platform spanning agents, LLM applications, and conventional machine-learning models. Its generative AI stack includes OpenTelemetry-based tracing, systematic evaluation, prompt registration and optimization, and an AI Gateway with an OpenAI-compatible interface. Teams can inspect traces, quality metrics, costs, and safety signals in the MLflow UI, compare experiments, and detect production regressions. For model development, it also supplies experiment tracking, model evaluation, a model registry, and batch or real-time deployment paths. It can run locally, on-premises, in cloud environments, or as a managed service, with documented support for Python, TypeScript/JavaScript, Java, MCP, more than 60 tracing integrations, and numerous model providers.
A basic workflow starts uvx mlflow server, points an application at it with mlflow.set_tracking_uri("http://localhost:5000"), and enables OpenAI instrumentation through mlflow.openai.autolog(). The application then makes a normal OpenAI().responses.create(...) request, while MLflow sends traces and metrics to the UI at http://localhost:5000. The platform captures complete agent and LLM execution traces, monitors quality, cost, and safety, and runs evaluations using built-in metrics, LLM judges, or custom definitions. Prompt Registry versions, tests, deploys, and records lineage for prompts, while prompt optimization can improve them automatically. AI Gateway exposes a unified OpenAI-compatible API that routes provider requests and handles rate limits, fallbacks, credentials, guardrails, cost controls, and A/B traffic splitting. Its ML features record parameters, metrics, models, and evaluation results; manage registered-model lifecycles; and support deployment to targets including Docker, Kubernetes, Azure ML, and AWS SageMaker.
- An agent engineering team needs complete execution traces to diagnose model responses, tool calls, and multi-step behavior without a major application rewrite.
- A quality team wants repeatable LLM evaluations with built-in metrics, LLM judges, or custom measures and needs to catch regressions before release.
- A platform team must govern several model providers through one gateway, including credentials, rate limits, fallbacks, cost controls, and access policies.
- A prompt engineering group needs versioning, testing, deployment, lineage, and automated optimization for production prompts.
- An ML team wants one system for experiment tracking, model evaluation, registry workflows, and batch or real-time deployment.
- An organization with data-control requirements wants to self-host a vendor-neutral observability and governance platform across multiple frameworks and providers.
How do you install or deploy this agent?
For the fastest agent-tracing setup, run uvx mlflow@latest agent setup. The documented command installs MLflow skills and launches the coding agent of your choice to add tracing. For manual setup, run uvx mlflow server to start a local server. The supplied material does not explain how to install uvx or name the environment variables used for model-provider credentials; credentials must be configured according to the provider before making external model calls.
How do you use this agent?
Run uvx mlflow server. In a Python application, add import mlflow, mlflow.set_tracking_uri("http://localhost:5000"), and mlflow.openai.autolog(). Create an OpenAI client and invoke client.responses.create(model="gpt-5.4-mini", input="Hello!"). After the application runs, open http://localhost:5000 to inspect traces and metrics. For other stacks, use the corresponding MLflow integration; the source explicitly documents OpenTelemetry, MCP, and integrations across Python, TypeScript, Java, model providers, gateways, and agent frameworks.
What are this agent's strengths and limitations?
- It covers the full operational loop for generative AI: tracing, evaluation, production monitoring, prompt management, prompt optimization, and gateway governance.
- OpenTelemetry foundations plus documented support for many providers, languages, frameworks, and more than 60 automatic tracing integrations reduce provider lock-in.
- AI Gateway goes beyond request proxying with rate limits, fallbacks, credential management, guardrails, cost controls, and A/B traffic splitting.
- The same platform supports generative AI and conventional ML through experiment tracking, evaluation, model registration, and deployment.
- Local and self-hosted operation is documented alongside cloud and managed options, which suits teams with data or access-control requirements.
- This is a broad platform rather than a small single-purpose library; adopters must operate an MLflow service and decide how tracing, evaluation, gateway, and registry components fit their architecture.
- Even the minimal example requires a running server, application initialization changes, and credentials for the external model provider.
- Frameworks and languages use separate integrations; broad compatibility does not prove identical instrumentation depth or feature parity for every combination.
- The supplied material does not document
uvxinstallation, production authentication, storage backends, server hardening, or high-availability setup, limiting production-readiness assessment. - Managed and deployment options span Databricks, SageMaker, Azure ML, Nebius, Kubernetes, and self-hosting, so migrations can still require environment-specific configuration.
How does this agent compare with similar options?
Compared with a framework-specific tracer, MLflow is explicitly positioned across agent frameworks and model providers and uses OpenTelemetry natively. It also has a wider boundary than LLM-only observability products because it includes conventional ML experiment tracking, evaluation, model registration, and deployment. Hosting can be self-managed or placed in environments such as Databricks, Amazon SageMaker, Azure ML, or Nebius, although configuration and management characteristics will differ by target.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| MLflow This agent | 64 · Some gaps | CLIFree + model costs | ★ 28k | today | Python | Claude Code · OpenAI API · Claude API |
| Langtrace | 42 · Major gaps | Library / SDKFreemium | ★ 1.2k | 11mo ago | TypeScript | OpenAI API · Claude API |
| Arize Phoenix | 61 · Some gaps | CLIFreemium | ★ 12k | 2d ago | Python | OpenAI API · Claude API |
| ClawMetry | 81 · Good | CLIFreemium | ★ 432 | 1d ago | Python | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
The evidence shows a CI workflow starting with empty default permissions, granting only pull-requests: write, and disabling credential persistence. The platform also claims credential management, access control, guardrails, and tracing, while SECURITY.md defines private reporting and coordinated disclosure. Deductions apply because the product spans model calls, deployment, gateways, data, and credentials without showing a unified least-privilege model, sensitive-field redaction/retention rules, or confirmations for high-impact operations. The setup command installs skills and launches a coding agent without documented stepwise consent. Prompt versioning and lineage offer limited recovery support, but no general rollback procedure is shown. License, copyright, maintainer, and project locations are clear, justifying full source-attribution credit; unknown registry identity is not treated as suspicious.
README, package metadata, and configuration are broadly consistent about MLflow's platform role, entry points, and support surface. Dependency declarations cover core and optional features, platform conditions, compatibility constraints, and pinned development tools. Validator tests demonstrate structured validation, nonzero failure exits, and specific, consistently ordered error messages. Deductions reflect broad runtime version ranges and the explicit policy that applications—not MLflow constraints—must exclude vulnerable dependency versions. Error-handling evidence is also concentrated on a review-payload utility and does not establish equivalent behavior throughout the platform.
The material clearly addresses developers, end users, researchers, IT teams, agent/LLM workflows, and conventional ML, while listing Python, TypeScript, Java, OpenTelemetry, MCP, frameworks, and model providers. Installation entry points, autologging, and feature extras provide reasonable trigger and environment guidance. Deductions apply because absolute statements such as supporting all frameworks, providers, tools, and languages are not demonstrated by the supplied files. Autologging scope, exclusions, and side effects are not precisely documented here, and the core package itself requires Python 3.10+, so cross-language fit is represented mainly through integration claims.
The README is organized into onboarding, agent capabilities, model training, and integrations, with website, documentation, demo, and issue routes. Pyproject metadata supplies package identity, Python requirements, scripts, extras, maintainer details, and version 3.15.2.dev0. The complete Apache-2.0 text agrees with copyright and package metadata, earning full license credit; the maintainer email, issue tracker, and security-reporting path make responsibility clear. Deductions apply because the fastest install uses latest/uvx, source and production setup details are limited, no revision-specific changelog is supplied, and FAQ/known-limitations coverage is sparse. Broad compatibility promises also limit confidence in stable support naming.
Tracing, evaluation, prompt management, gateway, registry, and deployment functions form usable AI-engineering workflows, and the example gives a short path from starting a server to inspecting traces. A unified lifecycle and broad integration surface plausibly add value beyond a single-purpose tracing library. Deductions apply because static material cannot establish actual result quality, and claims such as 60 million monthly downloads, largest, production-grade, and cost control lack independent support in the supplied files. The large dependency surface, service operation, and external model calls may carry substantial installation, operational, and usage costs, with no quantified cost-benefit analysis.
Major capabilities link to focused documentation, quickstarts, and demos, while package metadata cross-checks identity, version, entry points, dependencies, license, and maintenance channels. The security policy and workflow configuration provide additional corroboration for limited governance claims. Deductions apply because core marketing and scale assertions mostly originate in one README, and the supplied tests cover a Claude review-payload utility rather than MLflow's central agent functionality. Facts, aspirations, and marketing inferences are not explicitly separated, and universal compatibility statements lack item-by-item static evidence.
- Autotracing may capture prompts, responses, and application context; independently verify redaction, access controls, storage locations, and retention before using real data.
- Do not use uvx mlflow@latest agent setup directly in controlled or production environments. Pin a reviewed version and inspect the changes caused by skill installation and agent launch first.
- The declared dependency ranges are explicitly not a security-blocking mechanism. Deployers should create a lockfile or SBOM, scan resolved versions, and maintain a critical-vulnerability upgrade process.
- AI Gateway, cloud deployment, and model-provider integrations may create outbound requests, use credentials, and incur charges. Validate permissions, destinations, limits, and failure fallback in isolation.
- README claims about scale, universal compatibility, and production readiness are not independently verified by the supplied material and should not alone support procurement or compliance decisions.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v3.17.1 MLflow 3.17.1
Release notes excerpt
MLflow 3.17.1 is a patch release that contains some minor feature enhancements, bug fixes, and documentation updates. Features: - [Docs / Evaluation] Support Decision API in make_judge() (#26578, @B-Step62) - [Evaluation] Add mlflow.genai.databricks.review_queues for managing review queues on Databricks (#26414, @harupy) Bug fixes: - [Tracing] Fix Claude Code OTel trace rendering in the MLflow UI (#26547, @joshuawong-db) - [Evaluation] Load latest Databricks scorers from the scheduled-scorers REST config so feedback records the scorer version (#26542, @joshuawong-db) …View on GitHub ↗ -
v3.17.0 MLflow 3.17.0
Release notes excerpt
MLflow 3.17.0 includes several major features and improvements. Major New Features - [Jev Decisions Across Evaluation, Gateway, and Tracing](https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/custom-judges/typesafe/): Use TypeSafe models in built-in scorers and custom judges with typed Boolean or categorical feedback. (#26100, @B-Step62) …View on GitHub ↗ -
v2.11.5 MLflow 2.11.5
Release notes excerpt
2.11.5 (2026-09-23) MLflow 2.11.5 is a patch release. Features: - [Model Registry / Models] Add opt-in support for routing Unity Catalog model registry artifact uploads and downloads through the Databricks SDK Files API (#25986, @tonycai96)View on GitHub ↗