RAI — Embodied AI Agent Framework for Robotics
A vendor-agnostic agent framework that lets developers build multimodal AI capabilities for physical robots on top of ROS 2.
Evidence shows agents that execute robot actions and call external LLM/telemetry services (langfuse/langsmith), but no least-privilege documentation, user-confirmation mechanism, data-flow documentation, or rollback mechanism is visible; a git+https main-branch dependency (visualnav_transformer@main) weakens reproducibility. Attribution is clear (author emails, Apache-2.0, license headers), so source_attribution gets 2; the rest are deducted for missing static evidence.
pyproject pins dependency ranges, CI builds on both jazzy and humble with billable/ci_only/manual test markers, and message-reduction behaviors are unit-tested — decent consistency. However CI uses continue-on-error: true and self-hosted runners, weakening failure signals, so not full marks.
README clearly targets robotics scenarios (agriculture, manipulation, navigation demos) and supports Python 3.10/3.12 with ROS 2 humble/jazzy — audience and environment fit are well documented. Capability boundaries and trigger precision have no in-file evidence, hence deductions.
Clear information architecture (package-split monorepo), README with quick start, demos, community, citation and Q&A; full license plus CI license-header check; named maintainers. But no CHANGELOG, known limitations barely stated, install details delegated to external docs, and a minor version inconsistency (SECURITY.md lists 1.1.x vs pyproject 1.0.0).
Output usability is supported by concrete mechanisms (HRICallbackHandler, message-reduction behaviors); marginal value (multimodal, voice, benchmarking suite) is plausible but mostly asserted in the README without implementation detail in the provided files, so conservatively 2.
Claims are traceable via arXiv paper, ROSCon talk, codecov badge and citation entry, and README largely separates fact from aspiration; however key claims (benchmarks, coverage) point to external resources unverifiable in this static review, hence the deduction.
- This is a low-confidence static review; no code was executed — do not infer runtime safety from it.
- Agents can drive real robots and call external cloud services (LLMs, langfuse/langsmith telemetry); audit permissions, data egress, and confirmation flows before deployment.
- A git dependency pinned to a GitHub main branch (visualnav_transformer) is non-reproducible and carries supply-chain risk.
- SECURITY.md only says to report vulnerabilities as ordinary bugs — no encrypted channel or response SLA.
- CI uses continue-on-error: true and self-hosted runners, which can mask test failures.
What does this agent do, and when should you use it?
RAI is an open-source (Apache-2.0) flexible AI agent framework by RobotecAI for developing and deploying Embodied AI features on robots. Built on ROS 2 tools, it performs complex actions, defined scenarios, free-form instruction execution, log summaries, and voice interaction. The framework is split into purpose-built packages: rai core for multi-agent systems, human-robot interaction and multimodality; rai_whoami, which synthesizes robot embodiment information from structured documentation, images and URDFs; rai_asr and rai_tts for speech-to-text and text-to-speech; rai_sim for connecting to simulation environments; rai_bench for benchmarking agents, models, tools and simulators; rai_perception for open-set object detection; and rai_nomad, integrating NoMaD for navigation. It supports Ubuntu 22.04/24.04, Python 3.10/3.12 and ROS 2 humble/jazzy, with simulation demos spanning an autonomous orchard tractor, a Franka Panda arm, a Husarion ROSbot XL, and an RB-KAIROS mobile manipulator. RAI is a flagship project of the ROS Embodied AI Community Group and is backed by an arXiv paper (2505.07532).
RAI reads structured robot-specific material — documentation, images, and URDFs via rai_whoami — to synthesize an embodiment description that agents use as context. At runtime, rai core orchestrates multi-agent systems that invoke robot capabilities through ROS 2 tools, executing complex actions, predefined scenarios, and free-form natural-language instructions. Multimodal I/O spans text, voice (rai_asr transcribes speech; rai_tts synthesizes spoken replies) and vision (rai_perception performs open-set object detection). rai_sim connects agents to simulators, rai_bench benchmarks agents, models, tools and simulators, and rai_nomad integrates NoMaD for navigation. The end-to-end flow is: configure your robot → extract embodiment info → agents execute tasks and interact with humans in real or simulated environments.
- Robotics teams that want to add natural-language human-robot interaction and voice dialogue to ROS 2 hardware without building it from scratch.
- Agriculture or outdoor automation scenarios (like the virtual orchard tractor demo) where an agent must reason about the best course of action for unexpected obstacles, not just detect them.
- Labs using a Franka Panda arm that need flexible manipulation tasks driven by natural language, combining RAI with Grounded SAM 2.
- Mobile robot developers (e.g., Husarion ROSbot XL) seeking agent-based navigation and control demos with a validated setup.
- Researchers who need systematic benchmarking of different agents, models, tools, or simulators via rai_bench.
- Engineering teams building embodied AI experiments in simulators such as O3DE and reproducing paper scenarios.
What are this agent's strengths and limitations?
- Vendor-agnostic design with multi-agent systems and multimodal (speech, vision, text) support — no lock-in to a single model or vendor.
- Clear modular package structure (rai core, whoami, asr, tts, sim, bench, perception, nomad) so teams can adopt only what they need.
- Ships with an official benchmarking suite (rai_bench), enabling objective comparison of agents and models.
- Active community: a focus project of the ROS Embodied AI Community Group, with a ROSCon 2024 talk, an arXiv paper, and Discord support.
- Documented runtime requirements are specific and demanding: Ubuntu 22.04/24.04, Python 3.10/3.12, ROS 2 humble/jazzy — teams on other stacks face migration cost.
- The planned rai_finetune package (fine-tuning LLMs on embodied data) is not yet available.
- The README lacks inline install commands and API-key setup steps; onboarding depends on external documentation.
- Some demos (e.g., the RB-KAIROS mobile manipulator) live in a separate repository, adding extra pull-in and evaluation effort for adopters.
How do you install or deploy this agent?
The README does not inline full installation commands; it points to the setup guide. Confirmed requirements: Ubuntu 22.04/24.04, Python 3.10/3.12, ROS 2 humble/jazzy. Follow the official quick setup guide at https://robotecai.github.io/rai/setup/install . Missing information: the source does not include the exact pip/colcon build commands, API key configuration steps, or a complete first runnable invocation.
How do you use this agent?
- Install the environment per the official docs (https://robotecai.github.io/rai/setup/install) — Ubuntu 22.04/24.04, Python 3.10/3.12, ROS 2 humble/jazzy. 2. Create a configuration for your robot; the documentation includes dedicated instructions (https://robotecai.github.io/rai/). 3. Run official demos: the agriculture scenario, Franka Panda manipulation (with Grounded SAM 2), the ROSbot XL mobile robot, and the RB-KAIROS mobile manipulator (in the separate RobotecAI/agentic-mobile-manipulator repo). 4. Evaluate your agents with rai_bench. For questions, use the Q&A category in GitHub Discussions.