SimWorld Embodied Agent Simulator

Develop and evaluate multimodal autonomous agents in realistic physical and social worlds.

Stars
★ 819
Last updated
7mo ago
License
Apache-2.0
Primary language
Python

At a glance

How it runs
Library / SDKSelf-hosted service
Works with
Portable with changesOpenAI API (Partial support)
Cost
Free software; you pay for model usage
Setup effort
High · needs real infrastructure
You'll need
Python 3.10 or laterUnreal Engine server packageWindows or LinuxDedicated GPU with at least 6GB VRAM32GB RAM50-200GB disk spaceShell / CLINetwork accessLocal filesystem
Typical use
Embodied-AI researchers evaluating VLM navigation, planning, or object interaction in physically simulated urban and indoor scenes.
Not a fit if
  • Developers without a dedicated GPU or 50-200GB of disk space
  • Users wanting a single-command run without launching a separate UE server
Website
simworld.org ↗ set on GitHub
Maintenance
No commits for 6+ monthslast commit 2026-04-16latest release v0.1.0 (2026-05-30)+0.6 ★/day over 7 days

What does this agent do, and when should you use it?

SimWorld is an Unreal Engine 5 simulation platform for developing and evaluating LLM/VLM agents in complex physical and social environments. Its three-layer architecture combines an Unreal Engine backend, an environment layer for city and traffic generation, and an agent layer that reasons over multimodal observations and executes planned actions. The Python package includes Communicator, Humanoid, Map, BaseLLM, a local action planner, city generation, traffic, weather, and asset-retrieval components. Agents can consume position, orientation, RGB, depth, or segmentation observations and interact through movement, rotation, pickup, sitting, and other documented action types. Deployment spans a local Python client and a separately launched UE server; the required Base package supplies two lightweight city maps and one empty map, with more than 100 optional prebuilt maps.

A typical run connects to an active Unreal Engine server with Communicator(UnrealCV()), constructs a Humanoid, and places it in the world through communicator.spawn_agent. The environment reads position and orientation through UnrealCV and obtains visual input with communicator.get_camera_observation, producing observations such as position, direction, and an egocentric image. In the navigation example, BaseLLM("gpt-4o") turns the observation and target into a textual action; the environment parses that action and invokes communicator.humanoid_step_forward or communicator.humanoid_rotate. Other APIs can place objects with communicator.spawn_object, procedurally build a city with communicator.generate_world, and remove environment objects with communicator.clear_env. YAML configuration controls the seed, time step, UE blueprint paths, city generation, traffic, asset retrieval, and agent/model options; a task loop can return updated multimodal observations, a distance-based reward, and success state.

  1. Embodied-AI researchers evaluating VLM navigation, planning, or object interaction in physically simulated urban and indoor scenes.
  2. Robotics teams testing pedestrian, vehicle, or robot behavior in UE environments before deciding whether to run physical trials.
  3. Multi-agent researchers constructing traffic and social scenarios for cooperation, strategic reasoning, and behavior evaluation.
  4. Simulation and data engineers collecting structured RGB, depth, and segmentation observations for training or benchmark experiments.
  5. Unreal Engine developers bringing existing maps, assets, or character models into a Python-controlled simulator through custom .pak files.
  6. Teaching and prototyping teams demonstrating autonomous navigation with the Base maps and a Gym-like reset/step loop.

Key setup facts

Pulled from the install and usage notes below. The project README has the current values.

Install / run
  • pip install -e
Environment variables
MAP_PATH
Config files
config/example.yaml config/your_config.yaml
Runtime
Python 3.10
Install notes cover
Windows · Linux

How do you install or deploy this agent?

The documented requirements are Windows or Linux, Python 3.10 or later, a dedicated GPU with at least 6GB VRAM, 32GB RAM, and 50–200GB of disk space depending on the package.

Install the Python client:

git clone https://github.com/SimWorld-AI/SimWorld.git
cd SimWorld
conda create -n simworld python=3.10
conda activate simworld
pip install -e .

Download and extract the Base UE server package for the operating system. Base is required and includes two city maps plus one empty map. Optional environments are distributed as separate .pak files and should be copied into SimWorld/Content/Paks/ under the Base server directory. For custom settings, run cp config/example.yaml config/your_config.yaml, fill in the local paths, and load it with Config('path/to/your_config') using an absolute path. The example constructs BaseLLM("gpt-4o"), but the supplied material does not document the required API credential variable, configuration key, or credential-acquisition procedure.

How do you use this agent?

Start the UE server before running Python code. On Windows, double-click SimWorld.exe or run ./SimWorld.exe <MAP_PATH>; on Linux, run ./SimWorld.sh <MAP_PATH>. <MAP_PATH> is an Unreal Engine internal map path such as /Game/hospital/map/demo.umap. If it is omitted, the server opens /Game/Maps/demo_1.

With the server running, execute an example under examples/ or open examples/gym_interface_demo.ipynb. A minimal loop creates Communicator(UnrealCV()), calls communicator.spawn_agent from reset() to place a Humanoid, and reads position, orientation, and camera output. It then repeatedly asks the agent for an action and passes it to step(action), which invokes movement or rotation APIs and returns the next observation and reward. Separate examples cover city layout generation, traffic simulation, asset retrieval, cameras, and direct UE commands. Custom maps, objects, and agent models must first be packaged as .pak files and installed in the server's Paks directory.

What are this agent's strengths and limitations?

Pros
  • The UE5 backend combines realistic rendering and physics with procedural cities, traffic simulation, and agent control in one architecture.
  • Observation support includes position, orientation, RGB, depth, and segmentation, while the action space covers pedestrians, vehicles, and robots.
  • The small Base package supports initial testing, while more than 100 separately packaged .pak maps can be installed selectively.
  • Custom UE environments, assets, and agent models can be imported, extending the simulator beyond its bundled content.
  • Gym-like interaction and concrete Communicator APIs support navigation tasks, reward functions, and automated evaluation loops.
Limitations
  • Adoption has substantial hardware and storage costs: at least 6GB of dedicated VRAM, 32GB RAM, and 50–200GB of disk space.
  • The Python package is not a complete standalone runtime; it depends on a separately downloaded and launched UE server.
  • The example selects gpt-4o, but credential setup, model costs, rate limits, and model-call failure handling are not documented in the supplied material.
  • A unified training and evaluation framework, RL pipeline, 1K-plus city-scale multi-agent simulation, and MuJoCo integration remain roadmap items.
  • Arbitrary natural-language grounding into UE actions is not yet complete; the shown workflow parses a constrained action string and calls explicit APIs.

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

Agent Source review Form / cost Stars Updated Language Full support on
SimWorld Embodied Agent Simulator This agent 54 · Major gaps Library / SDKFree + model costs ★ 819 7mo ago Python —
RLinf RL Infrastructure 54 · Major gaps Library / SDKFree ★ 5.7k today Python —
MobileGym 67 · Some gaps CLIFree + model costs ★ 872 1mo ago Python —
Windows Agent Arena 62 · Some gaps CLIFree + model costs ★ 918 25d ago Python OpenAI API

How does FollowAgents rate this agent?

FollowAgents source review · FARS-2.1
Major gaps
54/ 100 5-point scale 2.7 / 5
Trust 10/29
Reliability 6/14
Adaptability 12/18
Convention 13/18
Effectiveness 9/13
Verifiability 4/8
Why each dimension lost points
Trust10 / 29 · 1.7/5

The README exposes APIs for clearing environments, generating worlds, spawning objects and agents, and explains that users download and start a UE server. It does not describe permission boundaries, sensitive-data handling, external data flows, operation confirmation, or recovery, so these areas receive limited credit. The example shows LLM inputs and camera observations but does not say whether data is sent to an external service. The code and license provide some attribution evidence, but the sample does not establish maintenance ownership or a clear update path.

Reliability6 / 14 · 2.1/5

The three usage tiers, setup steps, default-map guidance, and configuration instructions are broadly consistent. However, server packages depend on external downloads, and the documentation sample gives little assurance about dependency availability or troubleshooting. The quick-tour code is explicitly simplified and contains placeholders and omitted logic, so it does not establish a directly runnable implementation.

Adaptability12 / 18 · 3.3/5

The project clearly targets LLM/VLM agent development and evaluation, with navigation, planning, cooperation, sensors, maps, and custom environments among its use cases. It describes core capabilities and marks several features as planned, but gives limited detail on triggers, boundaries, or differences among operating environments. Windows/Linux support, hardware requirements, and UE configuration provide some environment guidance.

Convention13 / 18 · 3.6/5

The README has a clear table of contents and sections for project structure, configuration, setup, examples, and customization. Installation covers the Python client, UE packages, and map extensions, though external downloads and local paths still require user action. Names and navigation are mostly stable, and there are several example and documentation entry points. The README identifies roadmap items that are not yet complete. The full Apache-2.0 license is included. News and roadmap entries have dates, but the sample does not identify a maintenance owner or support commitment.

Effectiveness9 / 13 · 3.5/5

The simulator combines physical and social environments, action interfaces, sensors, procedural cities, and extensible maps, offering clear value for agent research and evaluation. Its observation, action, and reward interfaces provide some output usability. The resource cost is substantial: the README calls for a dedicated GPU, 32 GB RAM, 50–200 GB disk, and a separate UE server. The sample provides no performance or cost measurements to support a higher score.

Verifiability4 / 8 · 2.5/5

The README links major claims to configuration, API, sensor, and example references, and labels roadmap items that remain planned, making some claims traceable. The supplied material is mostly README and license text, without implementation or test material for cross-checking; the quick-tour also says it is simplified. The README distinguishes current capabilities from future plans, but claims about performance and simulation fidelity remain project assertions not substantiated by this sample.

Risks and how to mitigate them
  • The quick-tour code is explicitly simplified and contains placeholders and omitted logic; do not treat it as evidence that the complete program runs.
  • Operation depends on large UE server packages and substantial hardware; check the target machine, download source, and disk requirements.
  • The material does not explain external transmission of camera observations or LLM prompts, sensitive-data handling, or recovery after clearing an environment.
Evidence confidence: Low Reviewed Nov 08, 2026 Reviewed revision 1f588ec94699
Review evidence README.mdLICENSE
See the full review method →
Maintain this project? Add the FARS badge to your README

The badge shows the current review score and verdict and links back to this page. It refreshes about once an hour after a re-review.

FARS

[![FARS review](https://followagents.com/badge/simworld.svg)](https://followagents.com/en/agents/simworld)
<a href="https://followagents.com/en/agents/simworld"><img src="https://followagents.com/badge/simworld.svg" alt="FARS review"></a>

Recent releases

Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.

  1. v0.1.0 v0.1.0: Prototype Implementation
    Release notes excerpt
    SimWorld v0.1.0: Prototype Implementation This release provides the prototype implementation of SimWorld, the system introduced in our NeurIPS 2025 Spotlight Poster, [SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds.](https://neurips.cc/virtual/2025/loc/san-diego/poster/119002) …
    View on GitHub ↗

FAQ

Can I run SimWorld after installing only the Python package?
No. The Python client connects to a SimWorld UE server that must be downloaded and started first; the Base server package is required.
Is GPT-4o mandatory?
The minimal example uses BaseLLM("gpt-4o"), but the architecture targets LLM/VLM agents and does not document ready-made adapters for other providers. Switching models may require configuration or integration work.
What environments are included in the basic installation?
The Base package includes two lightweight city scenes and one empty map. More than 100 additional environments are optional downloads installed as separate .pak files.
Can I use my own Unreal Engine content?
Yes. Custom environments, assets, and agent models can be packaged as .pak files and imported into SimWorld.
Does it already provide a complete RL and large-scale multi-agent training pipeline?
No. The unified agent training and evaluation framework, RL pipeline, and city-scale simulation with more than 1,000 concurrent agents are listed as future work.
Is SimWorld Embodied Agent Simulator free?
The software is free, but SimWorld Embodied Agent Simulator calls an AI model, and you pay that model provider for usage (or run a local model).
Is SimWorld Embodied Agent Simulator open source?
Yes. The source code is on GitHub (SimWorld-AI/SimWorld) under the Apache-2.0 licence.
Is SimWorld Embodied Agent Simulator safe to use?
FollowAgents' FARS review (Nov 08, 2026) scored SimWorld Embodied Agent Simulator 10/29 on trust and safety. None of the eight safety controls the review checks for was missing outright.
Is SimWorld Embodied Agent Simulator still maintained?
The GitHub repository has had no commits since 2026-04-16, more than six months ago. That can mean the project is finished or abandoned; check its issues before relying on it. The latest release is v0.1.0 (2026-05-30).
Who makes SimWorld Embodied Agent Simulator?
SimWorld Embodied Agent Simulator is published on GitHub by the account SimWorld-AI. FollowAgents has not verified which person or company is behind that account.
What is SimWorld Embodied Agent Simulator's official website?
The project's GitHub repository lists https://simworld.org/ as its website. That link is set by the repository owner; the source code itself is on GitHub (SimWorld-AI/SimWorld).
Was this page helpful?
View on GitHub ↗ Install ↓

Related concepts