RAI — Embodied AI Agent Framework for Robotics
A vendor-agnostic agent framework that lets developers build multimodal AI capabilities for physical robots on top of ROS 2.
- Source repo
- RobotecAI/rai
- Stars
- ★ 607
- Last updated
- 10d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 55/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- Robotics teams that want to add natural-language human-robot interaction and voice dialogue to ROS 2 hardware without building it from scratch.
- Not a fit if
- Teams not on Ubuntu 22.04/24.04 with ROS 2 humble or jazzy
- Users wanting a turnkey app rather than building their own robot agent stack
- Source review
- 55/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
RAI is an open-source (Apache-2.0) flexible AI agent framework by RobotecAI for developing and deploying Embodied AI features on robots. Built on ROS 2 tools, it performs complex actions, defined scenarios, free-form instruction execution, log summaries, and voice interaction. The framework is split into purpose-built packages: rai core for multi-agent systems, human-robot interaction and multimodality; rai_whoami, which synthesizes robot embodiment information from structured documentation, images and URDFs; rai_asr and rai_tts for speech-to-text and text-to-speech; rai_sim for connecting to simulation environments; rai_bench for benchmarking agents, models, tools and simulators; rai_perception for open-set object detection; and rai_nomad, integrating NoMaD for navigation. It supports Ubuntu 22.04/24.04, Python 3.10/3.12 and ROS 2 humble/jazzy, with simulation demos spanning an autonomous orchard tractor, a Franka Panda arm, a Husarion ROSbot XL, and an RB-KAIROS mobile manipulator. RAI is a flagship project of the ROS Embodied AI Community Group and is backed by an arXiv paper (2505.07532).
RAI reads structured robot-specific material — documentation, images, and URDFs via rai_whoami — to synthesize an embodiment description that agents use as context. At runtime, rai core orchestrates multi-agent systems that invoke robot capabilities through ROS 2 tools, executing complex actions, predefined scenarios, and free-form natural-language instructions. Multimodal I/O spans text, voice (rai_asr transcribes speech; rai_tts synthesizes spoken replies) and vision (rai_perception performs open-set object detection). rai_sim connects agents to simulators, rai_bench benchmarks agents, models, tools and simulators, and rai_nomad integrates NoMaD for navigation. The end-to-end flow is: configure your robot → extract embodiment info → agents execute tasks and interact with humans in real or simulated environments.
- Robotics teams that want to add natural-language human-robot interaction and voice dialogue to ROS 2 hardware without building it from scratch.
- Agriculture or outdoor automation scenarios (like the virtual orchard tractor demo) where an agent must reason about the best course of action for unexpected obstacles, not just detect them.
- Labs using a Franka Panda arm that need flexible manipulation tasks driven by natural language, combining RAI with Grounded SAM 2.
- Mobile robot developers (e.g., Husarion ROSbot XL) seeking agent-based navigation and control demos with a validated setup.
- Researchers who need systematic benchmarking of different agents, models, tools, or simulators via rai_bench.
- Engineering teams building embodied AI experiments in simulators such as O3DE and reproducing paper scenarios.
How do you install or deploy this agent?
The README does not inline full installation commands; it points to the setup guide. Confirmed requirements: Ubuntu 22.04/24.04, Python 3.10/3.12, ROS 2 humble/jazzy. Follow the official quick setup guide at https://robotecai.github.io/rai/setup/install . Missing information: the source does not include the exact pip/colcon build commands, API key configuration steps, or a complete first runnable invocation.
How do you use this agent?
- Install the environment per the official docs (https://robotecai.github.io/rai/setup/install) — Ubuntu 22.04/24.04, Python 3.10/3.12, ROS 2 humble/jazzy. 2. Create a configuration for your robot; the documentation includes dedicated instructions (https://robotecai.github.io/rai/). 3. Run official demos: the agriculture scenario, Franka Panda manipulation (with Grounded SAM 2), the ROSbot XL mobile robot, and the RB-KAIROS mobile manipulator (in the separate RobotecAI/agentic-mobile-manipulator repo). 4. Evaluate your agents with rai_bench. For questions, use the Q&A category in GitHub Discussions.
What are this agent's strengths and limitations?
- Vendor-agnostic design with multi-agent systems and multimodal (speech, vision, text) support — no lock-in to a single model or vendor.
- Clear modular package structure (rai core, whoami, asr, tts, sim, bench, perception, nomad) so teams can adopt only what they need.
- Ships with an official benchmarking suite (rai_bench), enabling objective comparison of agents and models.
- Active community: a focus project of the ROS Embodied AI Community Group, with a ROSCon 2024 talk, an arXiv paper, and Discord support.
- Documented runtime requirements are specific and demanding: Ubuntu 22.04/24.04, Python 3.10/3.12, ROS 2 humble/jazzy — teams on other stacks face migration cost.
- The planned rai_finetune package (fine-tuning LLMs on embodied data) is not yet available.
- The README lacks inline install commands and API-key setup steps; onboarding depends on external documentation.
- Some demos (e.g., the RB-KAIROS mobile manipulator) live in a separate repository, adding extra pull-in and evaluation effort for adopters.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| RAI — Embodied AI Agent Framework for Robotics This agent | 55 · Major gaps | FrameworkFree + model costs | ★ 607 | 10d ago | Python | — |
| PhyAgentOS — Session-Centered Runtime for Embodied Intelligence | 47 · Major gaps | CLIFree + model costs | ★ 3k | 2d ago | Python | — |
| Octochains | 69 · Some gaps | Library / SDKFree + model costs | ★ 378 | 2d ago | Python | OpenAI API |
| WorldSeed | 37 · Major gaps | CLIFree + model costs | ★ 821 | 5mo ago | Python | Codex · OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows agents that execute robot actions and call external LLM/telemetry services (langfuse/langsmith), but no least-privilege documentation, user-confirmation mechanism, data-flow documentation, or rollback mechanism is visible; a git+https main-branch dependency (visualnav_transformer@main) weakens reproducibility. Attribution is clear (author emails, Apache-2.0, license headers), so source_attribution gets 2; the rest are deducted for missing static evidence.
pyproject pins dependency ranges, CI builds on both jazzy and humble with billable/ci_only/manual test markers, and message-reduction behaviors are unit-tested — decent consistency. However CI uses continue-on-error: true and self-hosted runners, weakening failure signals, so not full marks.
README clearly targets robotics scenarios (agriculture, manipulation, navigation demos) and supports Python 3.10/3.12 with ROS 2 humble/jazzy — audience and environment fit are well documented. Capability boundaries and trigger precision have no in-file evidence, hence deductions.
Clear information architecture (package-split monorepo), README with quick start, demos, community, citation and Q&A; full license plus CI license-header check; named maintainers. But no CHANGELOG, known limitations barely stated, install details delegated to external docs, and a minor version inconsistency (SECURITY.md lists 1.1.x vs pyproject 1.0.0).
Output usability is supported by concrete mechanisms (HRICallbackHandler, message-reduction behaviors); marginal value (multimodal, voice, benchmarking suite) is plausible but mostly asserted in the README without implementation detail in the provided files, so conservatively 2.
Claims are traceable via arXiv paper, ROSCon talk, codecov badge and citation entry, and README largely separates fact from aspiration; however key claims (benchmarks, coverage) point to external resources unverifiable in this static review, hence the deduction.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- This is a low-confidence static review; no code was executed — do not infer runtime safety from it.
- Agents can drive real robots and call external cloud services (LLMs, langfuse/langsmith telemetry); audit permissions, data egress, and confirmation flows before deployment.
- A git dependency pinned to a GitHub main branch (visualnav_transformer) is non-reproducible and carries supply-chain risk.
- SECURITY.md only says to report vulnerabilities as ordinary bugs — no encrypted channel or response SLA.
- CI uses continue-on-error: true and self-hosted runners, which can mask test failures.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
What's Changed * fix(s2s): dispatch on the transcription model names the config uses by @maciejmajek in https://github.com/RobotecAI/rai/pull/873 * feat(nav2): add workspace bounds and coordinate safety validation by @ved197338 in https://github.com/RobotecAI/rai/pull/871 New Contributors * @ved197338 made their first contribution in https://github.com/RobotecAI/rai/pull/871 Full Changelog: https://github.com/RobotecAI/rai/compare/2.12.3...2.12.4View on GitHub ↗ -
Release notes excerpt
What's Changed * fix: @Bartok9 contributions (batch 2) by @maciejmajek in https://github.com/RobotecAI/rai/pull/872 Full Changelog: https://github.com/RobotecAI/rai/compare/2.12.2...2.12.3View on GitHub ↗ -
Release notes excerpt
What's Changed * feat(rai_perception): setup time script for downloading weights by @maciejmajek in https://github.com/RobotecAI/rai/pull/791 * chore: remove unused sumbodule by @maciejmajek in https://github.com/RobotecAI/rai/pull/794 * fix: @Bartok9 contributions by @maciejmajek in https://github.com/RobotecAI/rai/pull/824 Full Changelog: https://github.com/RobotecAI/rai/compare/2.12.1...2.12.2View on GitHub ↗