SimWorld Embodied Agent Simulator

What does this agent do, and when should you use it?

SimWorld is an Unreal Engine 5 simulation platform for developing and evaluating LLM/VLM agents in complex physical and social environments. Its three-layer architecture combines an Unreal Engine backend, an environment layer for city and traffic generation, and an agent layer that reasons over multimodal observations and executes planned actions. The Python package includes Communicator, Humanoid, Map, BaseLLM, a local action planner, city generation, traffic, weather, and asset-retrieval components. Agents can consume position, orientation, RGB, depth, or segmentation observations and interact through movement, rotation, pickup, sitting, and other documented action types. Deployment spans a local Python client and a separately launched UE server; the required Base package supplies two lightweight city maps and one empty map, with more than 100 optional prebuilt maps.

A typical run connects to an active Unreal Engine server with Communicator(UnrealCV()), constructs a Humanoid, and places it in the world through communicator.spawn_agent. The environment reads position and orientation through UnrealCV and obtains visual input with communicator.get_camera_observation, producing observations such as position, direction, and an egocentric image. In the navigation example, BaseLLM("gpt-4o") turns the observation and target into a textual action; the environment parses that action and invokes communicator.humanoid_step_forward or communicator.humanoid_rotate. Other APIs can place objects with communicator.spawn_object, procedurally build a city with communicator.generate_world, and remove environment objects with communicator.clear_env. YAML configuration controls the seed, time step, UE blueprint paths, city generation, traffic, asset retrieval, and agent/model options; a task loop can return updated multimodal observations, a distance-based reward, and success state.

1. Embodied-AI researchers evaluating VLM navigation, planning, or object interaction in physically simulated urban and indoor scenes.
2. Robotics teams testing pedestrian, vehicle, or robot behavior in UE environments before deciding whether to run physical trials.
3. Multi-agent researchers constructing traffic and social scenarios for cooperation, strategic reasoning, and behavior evaluation.
4. Simulation and data engineers collecting structured RGB, depth, and segmentation observations for training or benchmark experiments.
5. Unreal Engine developers bringing existing maps, assets, or character models into a Python-controlled simulator through custom .pak files.
6. Teaching and prototyping teams demonstrating autonomous navigation with the Base maps and a Gym-like reset/step loop.

How do you install or deploy this agent?

The documented requirements are Windows or Linux, Python 3.10 or later, a dedicated GPU with at least 6GB VRAM, 32GB RAM, and 50–200GB of disk space depending on the package.

Install the Python client:

git clone https://github.com/SimWorld-AI/SimWorld.git
cd SimWorld
conda create -n simworld python=3.10
conda activate simworld
pip install -e .

Download and extract the Base UE server package for the operating system. Base is required and includes two city maps plus one empty map. Optional environments are distributed as separate .pak files and should be copied into SimWorld/Content/Paks/ under the Base server directory. For custom settings, run cp config/example.yaml config/your_config.yaml, fill in the local paths, and load it with Config('path/to/your_config') using an absolute path. The example constructs BaseLLM("gpt-4o"), but the supplied material does not document the required API credential variable, configuration key, or credential-acquisition procedure.

How do you use this agent?

Start the UE server before running Python code. On Windows, double-click SimWorld.exe or run ./SimWorld.exe <MAP_PATH>; on Linux, run ./SimWorld.sh <MAP_PATH>. <MAP_PATH> is an Unreal Engine internal map path such as /Game/hospital/map/demo.umap. If it is omitted, the server opens /Game/Maps/demo_1.

With the server running, execute an example under examples/ or open examples/gym_interface_demo.ipynb. A minimal loop creates Communicator(UnrealCV()), calls communicator.spawn_agent from reset() to place a Humanoid, and reads position, orientation, and camera output. It then repeatedly asks the agent for an action and passes it to step(action), which invokes movement or rotation APIs and returns the next observation and reward. Separate examples cover city layout generation, traffic simulation, asset retrieval, cameras, and direct UE commands. Custom maps, objects, and agent models must first be packaged as .pak files and installed in the server's Paks directory.

What are this agent's strengths and limitations?

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

How does FollowAgents rate this agent?

FAQ