Oumi
An open, end-to-end platform for fine-tuning, evaluating, and deploying foundation models from 10M to 400B+ parameters.
- Source repo
- oumi-ai/oumi
- Stars
- ★ 9.4k
- Last updated
- 1d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 63/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformClaude Code · OpenAI APIChatGPT · Claude API (Partial support)
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A team that wants to fine-tune Qwen3 or a Gemma model for its own domain and would rather start from an existing YAML recipe than write a training script.
- Not a fit if
- Users who only want a chat UI and no config files or CLI
- Teams with no GPU access doing only light experimentation
- Production buyers who need a commercial SLA and vendor support
- Website
- oumi.ai ↗ set on GitHub
- Maintenance
- Commits in the last 30 dayslast commit 2026-11-20latest release v0.9 (2026-09-16)+1.1 ★/day over 7 days
- Source review
- 63/100 · Some gaps
What does this agent do, and when should you use it?
Oumi is a fully open-source platform that covers the full lifecycle of foundation models: data synthesis and curation, pre-training and fine-tuning, evaluation, inference, and deployment. It is driven by a single oumi CLI plus YAML recipes, so you rarely have to write your own training loop or data pipeline. Training supports SFT, LoRA, QLoRA, DPO, and GRPO, scaling through FSDP, DeepSpeed, and DDP; inference integrates vLLM and SGLang, and models can be pushed to dedicated endpoints with oumi deploy. Beyond text models it handles vision-language models and agentic, tool-using models: you can define executable tool environments (database, HTTP endpoint, lookup, simulated) and run verl GRPO reinforcement learning over them. Evaluation spans standard benchmarks plus multi-criteria RubricJudge and LLM-as-a-Judge. It runs from a laptop up to clusters and clouds via oumi launch (AWS, Azure, GCP, Lambda, Modal, Slurm), and oumi-mcp exposes an MCP server for Claude and Cursor.
Oumi centers on the oumi command-line interface. You author or reuse a YAML config from configs/recipes and run oumi train -c <config> to launch training (SFT, LoRA, QLoRA, DPO, GRPO on PyTorch, optionally with FSDP/DeepSpeed/DDP), oumi evaluate -c <config> to score a model on standard benchmarks or custom judges, and oumi infer -c <config> --interactive for interactive inference through engines such as vLLM, SGLang, Fireworks, or OpenRouter. On the data side it reads raw corpora and uses LLM judges, including the multi-criteria RubricJudge, to synthesize and filter training data, including multi-turn tool-use conversations. For agentic training it manages executable tool environments (database, HTTP endpoint, lookup, simulated) and applies verl GRPO over them. For delivery, oumi deploy publishes models to dedicated inference endpoints on Fireworks and Parasail, oumi launch up submits jobs to cloud platforms, and oumi-mcp runs an MCP server consumable by Claude and Cursor. oumi analyze covers analysis, and the configs/recipes tree ships ready-made configurations for Qwen, Gemma, Llama, DeepSeek, GLM, Phi, OLMo, gpt-oss, Falcon, and more.
- A team that wants to fine-tune Qwen3 or a Gemma model for its own domain and would rather start from an existing YAML recipe than write a training script.
- A research group running reproducible GRPO reinforcement-learning experiments across multiple GPUs or a cloud cluster, needing FSDP or DeepSpeed.
- A developer training a tool-calling agent model who needs to construct database, HTTP, or lookup tool environments and do reinforcement learning over them.
- A data team that must synthesize, filter, and curate training data at scale with LLM-as-a-Judge or RubricJudge, including vision-language datasets.
- An engineering team deploying a fine-tuned model locally through vLLM/SGLang, or publishing it to a managed endpoint with oumi deploy.
- Developers already working inside an AI-assisted IDE such as Claude or Cursor who want to drive Oumi workflows through the oumi-mcp MCP server.
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Install / run
uv pip install oumiuv pip install 'oumi[gpu]'uv pip install git+https://github.com/oumi-ai/oumi.gitdocker pull ghcr.io/oumi-ai/oumi:latest
- Config files
configs/recipes/smollm/sft/135m/quickstart_train.yamlconfigs/recipes/smollm/evaluation/135m/quickstart_eval.yamlconfigs/recipes/smollm/inference/135m_infer.yamlconfigs/recipes/smollm/sft/135m/quickstart_gcp_job.yaml- Install notes cover
- Docker
How do you install or deploy this agent?
The README recommends pip (uv):
# Basic installation
uv pip install oumi
# With GPU support
uv pip install 'oumi[gpu]'
# Latest development version
uv pip install git+https://github.com/oumi-ai/oumi.gitDocker and an experimental one-line script are also documented:
docker pull ghcr.io/oumi-ai/oumi:latest
docker run --gpus all -it ghcr.io/oumi-ai/oumi:latest oumi --helpcurl -LsSf https://oumi.ai/install.sh | bashOumi is in beta and under active development: core features are stable, but some advanced features may change. More installation options are in the official installation guide.
How do you use this agent?
Pick a YAML config under configs/recipes and drive it with the oumi CLI:
# Training
oumi train -c configs/recipes/smollm/sft/135m/quickstart_train.yaml
# Evaluation
oumi evaluate -c configs/recipes/smollm/evaluation/135m/quickstart_eval.yaml
# Inference
oumi infer -c configs/recipes/smollm/inference/135m_infer.yaml --interactiveTo run remotely, submit with oumi launch; to train inside the container, mount your working directory and pass --config:
oumi launch up -c configs/recipes/smollm/sft/135m/quickstart_gcp_job.yaml --resources.cloud awsdocker run --gpus all -v $(pwd):/workspace -it ghcr.io/oumi-ai/oumi:latest \
oumi train --config /workspace/my_config.yamlWhat are this agent's strengths and limitations?
- One CLI plus YAML recipes covers the whole pipeline — training, evaluation, inference, data synthesis, and deployment — with no custom training loop.
- Broad training method coverage: SFT, LoRA, QLoRA, DPO, and GRPO, with native FSDP, DeepSpeed, and DDP support for scaling.
- Agentic focus backed by concrete components: executable tool environments (database, HTTP endpoint, lookup, simulated) and verl GRPO reinforcement learning over them.
- Evaluation is first-class, combining standard benchmarks with multi-criteria RubricJudge and LLM-as-a-Judge.
- Multiple delivery paths: vLLM/SGLang inference, oumi deploy to dedicated endpoints, oumi-mcp for Claude and Cursor, and oumi launch to AWS, Azure, GCP, Lambda, Modal, or Slurm.
- Apache-2.0 licensed and fully open source, with no vendor lock-in.
- Oumi is explicitly in beta, and the README warns that some advanced features may change as the platform evolves.
- Full capability needs GPU hardware and a heavy Python/PyTorch environment, so the barrier to entry is higher than with lightweight tools.
- Cloud job submission and dedicated endpoint deployment depend on external accounts and quotas (AWS, Azure, GCP, Lambda, Modal, Slurm, Fireworks, Parasail).
- Many supported model weights ship under custom licenses (Llama, Gemma, Qwen and others), so commercial use requires checking each license.
- The README's news list contains dates that run into 2026, so release history should be verified against the actual release pages.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Oumi This agent | 63 · Some gaps | CLIFree + model costs | ★ 9.4k | 1d ago | Python | Claude Code · OpenAI API |
| DeepEval | 69 · Some gaps | CLIFree + model costs | ★ 20k | 1d ago | Python | OpenAI API · Claude API |
| AI Agents Projects & Tutorials | Insufficient evidence | Library / SDKFree + model costs | ★ 3k | 14d ago | Jupyter Notebook | OpenAI API · Claude API |
| LLM-in-Sandbox | 47 · Major gaps | CLIFree + model costs | ★ 249 | 3mo ago | Python | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
The code exposes CLI, cloud job launch, deployment, and MCP interfaces, indicating local, network, and cloud effects. The quick installer pipes curl output to a shell, while the packet does not explain the script contents, granular permissions, pre-execution confirmation, or a general recovery path; least privilege, confirmation, data-flow transparency, sensitive-data handling, external effects, and rollback therefore score 1. Dependencies have many version bounds and comments documenting compatibility constraints, supporting a 2 for dependency security. The Apache license and source copyright headers provide attribution, but publisher identity remains unknown as stipulated, so source attribution scores 2.
The README claims production-grade reliability while also describing the project as beta. The installation guide requires Python 3.9+, whereas pyproject.toml requires 3.10 through 3.14; this concrete inconsistency limits self-consistency to 1. Dependencies have numerous constraints and optional groups, but the packet provides no lockfile or installation-success evidence, so availability scores 1. Troubleshooting gives concrete symptoms and remedies for Windows, Intel Macs, OOM, cloud mounts, and inference, earning 2 for failure messages.
Documentation covers researchers and developers across local machines, clusters, and multiple cloud platforms, with training, evaluation, inference, and agent tool environments; audience and scenarios score 3. The README says configurations are examples rather than an exhaustive support list and notes that advanced features may change, but the packet does not systematically define boundaries, so capability boundaries score 2. CLI entry points, command help, and configuration examples help users identify applicable tasks, earning 2 for trigger precision. The docs explicitly exclude Intel Macs, recommend WSL for Windows, and describe GPU and cloud-specific optional dependencies, earning 2 for environment fit.
The README, installation guide, quickstart, troubleshooting page, and project metadata form a clear information structure, scoring 3. Installation methods, Python requirements, optional dependencies, and a verification command are documented, but the quick installer script's contents are not shown, so install notes score 2. CLI and package names consistently center on Oumi/oumi, with some variation between OSS wording, project naming, and command presentation; naming stability scores 2. Tutorials, recipes, FAQ material, and concrete troubleshooting cases are extensive, scoring 3. The packet states beta status, Intel Mac limitations, Windows guidance, and some dependency compatibility issues, but coverage is not comprehensive, so known limitations score 2. A full Apache-2.0 license is included, scoring 3. The README has dated release and feature updates, but the sample mixes very recent future-dated entries with older updates and shows no revision policy, so versioning and changelog score 2. Security reporting and contribution paths are described, but maintenance responsibility is attributed mainly to a team/community without a verifiable named contact, scoring 2.
A unified CLI, configuration overrides, extensive recipes, and examples for training, evaluation, inference, and cloud jobs make outputs directly usable, scoring 3. The packet supports substantial added value through integrated workflows for open-model fine-tuning, tool environments, RL, evaluation, and deployment, scoring 3. Cost-benefit scores 2: the docs warn that cloud services can be expensive and advise tearing down clusters, and provide a small-model quickstart, but no systematic resource or cost estimates.
README feature claims are supported in part by CLI registration, configuration paths, package metadata, and documentation examples, but broad model-compatibility and production-quality claims lack itemized evidence in this sample; claim traceability scores 2. The README, quickstart, installation guide, pyproject, and CLI entry points corroborate one another in several areas, while also revealing a Python-version mismatch, earning 2 for cross-source corroboration. The material distinguishes beta status, an experimental installer, examples, and exhaustive support lists, but does not consistently separate promotional capability claims from demonstrated facts, so fact-inference separation scores 2.
- The quick-install command pipes a remote script directly to a shell; the reviewed packet does not include that script, so inspect its source and actions before use.
- The installation guide's Python 3.9+ requirement conflicts with project metadata requiring 3.10 through 3.14; choosing an environment from the metadata is safer.
- Cloud launch and deployment may incur charges and involve credentials or file mounts; the quickstart explicitly warns about cloud costs and that cluster teardown deletes remote data.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
Oumi v0.9 Release Notes This release makes Oumi an end-to-end stack for agentic, tool-using models: you can now train on tool-calling data, execute tool calls against real environments (databases, HTTP endpoints, deterministic lookups, and model-simulated tools), synthesize multi-turn tool conversations, and run GRPO/RL over any of those environments with verl. It also adds a multi-criteria RubricJudge, partial-failure-tolerant inference/synthesis/judging, a Modal GPU launcher and SkyPilot-routed Slurm, and new Gemma 4 recipes. Highlights 🛠️ Tool-use training …View on GitHub ↗ -
Release notes excerpt
Oumi v0.8 Release Notes This release lands the new oumi deploy CLI for shipping models to dedicated inference endpoints, an oumi-mcp server for MCP-capable assistants, batch-API parity across hosted providers, and a major dependency push to Transformersv5 / TRL 0.30+ / vLLM` 0.20+. Highlights oumi deploy — new CLI for dedicated inference endpoints …View on GitHub ↗ -
Release notes excerpt
Oumi 0.7 Release ✨ Highlights This release brings major platform upgrades (Python 3.14, PyTorch 2.9), new inference engines, rule-based evaluation judges, and significant CLI/documentation improvements. --- 🚀 New Features Inference - Fireworks inference engine - New backend for Fireworks AI (#2158) ```yaml Fireworks example (set FIREWORKS_API_KEY env var) model: model_name: "accounts/fireworks/models/llama4-maverick-instruct-basic" engine: FIREWORKS ``` ```bash oumi infer -i -c configs/recipes/llama4/inference/maverick_instruct_fireworks_infer.yaml ``` …View on GitHub ↗
FAQ
Do I need a GPU to use Oumi?
Does Oumi itself cost money?
Can it train tool-calling agents?
How do I serve a model in production?
Which models are supported?
Is Oumi open source?
Is Oumi safe to use?
Is Oumi still maintained?
Who makes Oumi?
What is Oumi's official website?
Thanks — we use this to improve the page.