SimWorld Embodied Agent Simulator
Develop and evaluate multimodal autonomous agents in realistic physical and social worlds.
- Source repo
- SimWorld-AI/SimWorld
- Stars
- ★ 819
- Last updated
- 7mo ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 54/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API (Partial support)
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- Embodied-AI researchers evaluating VLM navigation, planning, or object interaction in physically simulated urban and indoor scenes.
- Not a fit if
- Developers without a dedicated GPU or 50-200GB of disk space
- Users wanting a single-command run without launching a separate UE server
- Website
- simworld.org ↗ set on GitHub
- Maintenance
- No commits for 6+ monthslast commit 2026-04-16latest release v0.1.0 (2026-05-30)+0.6 ★/day over 7 days
- Source review
- 54/100 · Major gaps
What does this agent do, and when should you use it?
SimWorld is an Unreal Engine 5 simulation platform for developing and evaluating LLM/VLM agents in complex physical and social environments. Its three-layer architecture combines an Unreal Engine backend, an environment layer for city and traffic generation, and an agent layer that reasons over multimodal observations and executes planned actions. The Python package includes Communicator, Humanoid, Map, BaseLLM, a local action planner, city generation, traffic, weather, and asset-retrieval components. Agents can consume position, orientation, RGB, depth, or segmentation observations and interact through movement, rotation, pickup, sitting, and other documented action types. Deployment spans a local Python client and a separately launched UE server; the required Base package supplies two lightweight city maps and one empty map, with more than 100 optional prebuilt maps.
A typical run connects to an active Unreal Engine server with Communicator(UnrealCV()), constructs a Humanoid, and places it in the world through communicator.spawn_agent. The environment reads position and orientation through UnrealCV and obtains visual input with communicator.get_camera_observation, producing observations such as position, direction, and an egocentric image. In the navigation example, BaseLLM("gpt-4o") turns the observation and target into a textual action; the environment parses that action and invokes communicator.humanoid_step_forward or communicator.humanoid_rotate. Other APIs can place objects with communicator.spawn_object, procedurally build a city with communicator.generate_world, and remove environment objects with communicator.clear_env. YAML configuration controls the seed, time step, UE blueprint paths, city generation, traffic, asset retrieval, and agent/model options; a task loop can return updated multimodal observations, a distance-based reward, and success state.
- Embodied-AI researchers evaluating VLM navigation, planning, or object interaction in physically simulated urban and indoor scenes.
- Robotics teams testing pedestrian, vehicle, or robot behavior in UE environments before deciding whether to run physical trials.
- Multi-agent researchers constructing traffic and social scenarios for cooperation, strategic reasoning, and behavior evaluation.
- Simulation and data engineers collecting structured RGB, depth, and segmentation observations for training or benchmark experiments.
- Unreal Engine developers bringing existing maps, assets, or character models into a Python-controlled simulator through custom
.pakfiles. - Teaching and prototyping teams demonstrating autonomous navigation with the Base maps and a Gym-like
reset/steploop.
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Install / run
pip install -e
- Environment variables
MAP_PATH- Config files
config/example.yamlconfig/your_config.yaml- Runtime
- Python 3.10
- Install notes cover
- Windows · Linux
How do you install or deploy this agent?
The documented requirements are Windows or Linux, Python 3.10 or later, a dedicated GPU with at least 6GB VRAM, 32GB RAM, and 50–200GB of disk space depending on the package.
Install the Python client:
git clone https://github.com/SimWorld-AI/SimWorld.git
cd SimWorld
conda create -n simworld python=3.10
conda activate simworld
pip install -e .Download and extract the Base UE server package for the operating system. Base is required and includes two city maps plus one empty map. Optional environments are distributed as separate .pak files and should be copied into SimWorld/Content/Paks/ under the Base server directory. For custom settings, run cp config/example.yaml config/your_config.yaml, fill in the local paths, and load it with Config('path/to/your_config') using an absolute path. The example constructs BaseLLM("gpt-4o"), but the supplied material does not document the required API credential variable, configuration key, or credential-acquisition procedure.
How do you use this agent?
Start the UE server before running Python code. On Windows, double-click SimWorld.exe or run ./SimWorld.exe <MAP_PATH>; on Linux, run ./SimWorld.sh <MAP_PATH>. <MAP_PATH> is an Unreal Engine internal map path such as /Game/hospital/map/demo.umap. If it is omitted, the server opens /Game/Maps/demo_1.
With the server running, execute an example under examples/ or open examples/gym_interface_demo.ipynb. A minimal loop creates Communicator(UnrealCV()), calls communicator.spawn_agent from reset() to place a Humanoid, and reads position, orientation, and camera output. It then repeatedly asks the agent for an action and passes it to step(action), which invokes movement or rotation APIs and returns the next observation and reward. Separate examples cover city layout generation, traffic simulation, asset retrieval, cameras, and direct UE commands. Custom maps, objects, and agent models must first be packaged as .pak files and installed in the server's Paks directory.
What are this agent's strengths and limitations?
- The UE5 backend combines realistic rendering and physics with procedural cities, traffic simulation, and agent control in one architecture.
- Observation support includes position, orientation, RGB, depth, and segmentation, while the action space covers pedestrians, vehicles, and robots.
- The small Base package supports initial testing, while more than 100 separately packaged
.pakmaps can be installed selectively. - Custom UE environments, assets, and agent models can be imported, extending the simulator beyond its bundled content.
- Gym-like interaction and concrete Communicator APIs support navigation tasks, reward functions, and automated evaluation loops.
- Adoption has substantial hardware and storage costs: at least 6GB of dedicated VRAM, 32GB RAM, and 50–200GB of disk space.
- The Python package is not a complete standalone runtime; it depends on a separately downloaded and launched UE server.
- The example selects
gpt-4o, but credential setup, model costs, rate limits, and model-call failure handling are not documented in the supplied material. - A unified training and evaluation framework, RL pipeline, 1K-plus city-scale multi-agent simulation, and MuJoCo integration remain roadmap items.
- Arbitrary natural-language grounding into UE actions is not yet complete; the shown workflow parses a constrained action string and calls explicit APIs.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| SimWorld Embodied Agent Simulator This agent | 54 · Major gaps | Library / SDKFree + model costs | ★ 819 | 7mo ago | Python | — |
| RLinf RL Infrastructure | 54 · Major gaps | Library / SDKFree | ★ 5.7k | today | Python | — |
| MobileGym | 67 · Some gaps | CLIFree + model costs | ★ 872 | 1mo ago | Python | — |
| Windows Agent Arena | 62 · Some gaps | CLIFree + model costs | ★ 918 | 25d ago | Python | OpenAI API |
How does FollowAgents rate this agent?
Why each dimension lost points
The README exposes APIs for clearing environments, generating worlds, spawning objects and agents, and explains that users download and start a UE server. It does not describe permission boundaries, sensitive-data handling, external data flows, operation confirmation, or recovery, so these areas receive limited credit. The example shows LLM inputs and camera observations but does not say whether data is sent to an external service. The code and license provide some attribution evidence, but the sample does not establish maintenance ownership or a clear update path.
The three usage tiers, setup steps, default-map guidance, and configuration instructions are broadly consistent. However, server packages depend on external downloads, and the documentation sample gives little assurance about dependency availability or troubleshooting. The quick-tour code is explicitly simplified and contains placeholders and omitted logic, so it does not establish a directly runnable implementation.
The project clearly targets LLM/VLM agent development and evaluation, with navigation, planning, cooperation, sensors, maps, and custom environments among its use cases. It describes core capabilities and marks several features as planned, but gives limited detail on triggers, boundaries, or differences among operating environments. Windows/Linux support, hardware requirements, and UE configuration provide some environment guidance.
The README has a clear table of contents and sections for project structure, configuration, setup, examples, and customization. Installation covers the Python client, UE packages, and map extensions, though external downloads and local paths still require user action. Names and navigation are mostly stable, and there are several example and documentation entry points. The README identifies roadmap items that are not yet complete. The full Apache-2.0 license is included. News and roadmap entries have dates, but the sample does not identify a maintenance owner or support commitment.
The simulator combines physical and social environments, action interfaces, sensors, procedural cities, and extensible maps, offering clear value for agent research and evaluation. Its observation, action, and reward interfaces provide some output usability. The resource cost is substantial: the README calls for a dedicated GPU, 32 GB RAM, 50–200 GB disk, and a separate UE server. The sample provides no performance or cost measurements to support a higher score.
The README links major claims to configuration, API, sensor, and example references, and labels roadmap items that remain planned, making some claims traceable. The supplied material is mostly README and license text, without implementation or test material for cross-checking; the quick-tour also says it is simplified. The README distinguishes current capabilities from future plans, but claims about performance and simulation fidelity remain project assertions not substantiated by this sample.
- The quick-tour code is explicitly simplified and contains placeholders and omitted logic; do not treat it as evidence that the complete program runs.
- Operation depends on large UE server packages and substantial hardware; check the target machine, download source, and disk requirements.
- The material does not explain external transmission of camera observations or LLM prompts, sensitive-data handling, or recovery after clearing an environment.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v0.1.0 v0.1.0: Prototype Implementation
Release notes excerpt
SimWorld v0.1.0: Prototype Implementation This release provides the prototype implementation of SimWorld, the system introduced in our NeurIPS 2025 Spotlight Poster, [SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds.](https://neurips.cc/virtual/2025/loc/san-diego/poster/119002) …View on GitHub ↗
FAQ
Can I run SimWorld after installing only the Python package?
Is GPT-4o mandatory?
BaseLLM("gpt-4o"), but the architecture targets LLM/VLM agents and does not document ready-made adapters for other providers. Switching models may require configuration or integration work.What environments are included in the basic installation?
.pak files.Can I use my own Unreal Engine content?
.pak files and imported into SimWorld.Does it already provide a complete RL and large-scale multi-agent training pipeline?
Is SimWorld Embodied Agent Simulator free?
Is SimWorld Embodied Agent Simulator open source?
Is SimWorld Embodied Agent Simulator safe to use?
Is SimWorld Embodied Agent Simulator still maintained?
Who makes SimWorld Embodied Agent Simulator?
What is SimWorld Embodied Agent Simulator's official website?
Thanks — we use this to improve the page.