VirtualHome
A Python-API-driven multi-agent household simulator that renders activity videos, evolving environment graphs, and trains embodied AI agents.
- Source repo
- xavierpuigf/virtualhome
- Stars
- ★ 643
- Last updated
- 5mo ago
- License
- MIT
- Primary language
- Python
- FA score
- 40/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free, no paid service needed
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- Embodied AI researchers needing a reproducible household environment to train and evaluate agent policies
- Not a fit if
- Teams wanting a hosted, zero-setup service with no local Unity executable
- Users seeking a chat-style assistant with no graphics/GPU environment
- Homeowners looking for smart-home automation — this is a research simulator
- Source review
- 40/100 · Major gaps 1 safety controls not found
What does this agent do, and when should you use it?
VirtualHome is an interactive platform for simulating complex household activities via programs, supporting rich interactions such as picking up objects, switching appliances on/off, and opening appliances, including multi-agent activities. It ships two simulators: the Unity-based Unity Simulator, which generates videos of activities, and Evolving Graph, a pure-Python simulator that outputs graphs of how the environment evolves step by step. You write activities as simple instruction sequences via the Python API, and the simulator renders them as videos or graph sequences, streaming ground truth such as time-stamped actions, instance/semantic segmentation, optical flow, and depth. Version 2.3 adds procedural generation of unique environments, enhanced physics, synchronized day/night time management, and real-time indoor lighting. The project also provides an OpenAI Gym-like RL environment (unity_environment.py) that can be combined with Ray for parallel training of embodied AI agents.
It reads activity programs (instruction sequences) and environment graphs, then runs two simulators: Unity Simulator renders videos and snapshots of the activity via a downloaded executable, while Evolving Graph, running fully in Python, produces sequences of graphs as the program executes. You connect to the executable through simulation.unity_simulator.comm_unity.UnityCommunication, optionally specifying a port and X display and opening multiple executables in parallel. The UnityEnvironment base class in simulation/environment/unity_environment.py serves as an RL environment; test_unity_environment_mp.py demonstrates parallel use with Ray. Dataset programs can be fetched via helper_scripts/download_dataset.sh, and dataset_utils scripts augment_dataset_locations.py and augment_dataset_exceptions.py augment scripts with the KB-RealEnv and KB-ExceptionHandler knowledge bases.
- Embodied AI researchers needing a reproducible household environment to train and evaluate agent policies
- Computer vision researchers needing synthetic data with instance/semantic segmentation, depth, and optical flow ground truth
- RL engineers wanting a Gym-style environment that scales with parallel Ray workers
- Multi-agent collaboration researchers simulating human-AI collaboration scenarios such as Watch-and-Help
- Teams generating datasets of household activity videos or snapshot grids
How do you install or deploy this agent?
Install the Python package:
pip install virtualhomeDownload the Unity Simulator executable for your platform and move it under simulation/unity_simulator (Linux x86-64, macOS, and Windows builds are hosted at virtual-home.org), or fetch the dataset with:
./helper_scripts/download_dataset.shFor machines without a monitor, use Docker (docker folder in the repo) or an X server.
How do you use this agent?
- Start the simulator (windowed mode recommended):
./path_to_exec -screen-fullscreen 0 -screen-quality 4- Run the Jupyter demo
demo/unity_demo.ipynb, or generate videos/snapshots directly:
cd demo/
python generate_video.pycd demo/
python generate_snapshots.py- On headless Linux, start an X server, then launch the executable with a chosen display:
sudo python helper_scripts/startx.py $display_numDISPLAY=:display_num ./{path_sim}/{exec_file}.x86_64 -batchmodeOr specify the executable from Python:
from simulation.unity_simulator import comm_unity
comm = comm_unity.UnityCommunication(file_name=file_name, port={your_port}, x_display={your_display})- RL environment:
cd demo
python test_unity_environment.pycd demo
python test_unity_environment_mp.pyWhat are this agent's strengths and limitations?
- Two complementary simulators: Unity renders videos while Evolving Graph runs fully in Python to verify programs without a display
- Streams multiple ground truths — time-stamped actions, instance/semantic segmentation, optical flow, depth — for vision and embodied research
- Built-in Gym-style RL environment with Ray-based parallel scaling and support for multiple simultaneous executable instances
- Version 2.3 supports procedural generation of infinite unique environments and on-the-fly environment modification
- Requires separately downloading and running a local Unity executable; headless machines need X server or Docker setup
- Some objects and actions in Evolving Graph are not yet supported in Unity Simulator
- Procedural generation and photorealism capabilities are still in development
- Modifying the simulator itself requires fetching a separate Unity source repository and building your own executable with the Unity Editor
How does this agent compare with similar options?
The README names no direct competing simulators; it only points to the companion virtualhome_unity source repository for modifying the simulator.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| VirtualHome This agent | 40 · Major gaps | Library / SDKFree | ★ 643 | 5mo ago | Python | — |
| IR-SIM: Lightweight Robot Simulator | 43 · Major gaps | Library / SDKFree | ★ 1.2k | 15d ago | Python | — |
| AgentGuide - All-in-One AI Agent Development & Interview Guide | 23 · Major gaps | Library / SDKFree + model costs | ★ 11k | 1mo ago | MDX | OpenAI API · Claude API |
| ChatArena | 32 · Major gaps | Library / SDKFree + model costs | ★ 1.6k | 1y ago | Python | OpenAI API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows locally-run code communicating with a local Unity executable via a Python API; no evidence of network credentials or upload logic; license attribution is clear (MIT, MIT copyright) with a named contributor list. Deductions: security/permission guidance is missing for port communication, X server startup (startx.py requires sudo), and downloadable executables (HTTP, not HTTPS links); no rollback/recovery mechanisms documented; no user-confirmation or dangerous-operation prompts; no dependency lockfile or security audit, so dependency security cannot be verified from an essentially empty pyproject.
README provides install, test, and demo script paths and is internally consistent; pyproject uses standard setuptools build. Deductions: with only README and pyproject, tests, error messages, and failure handling cannot be confirmed; the stated version (2.3) has no matching version field in pyproject; availability of the external Unity executable depends entirely on a third-party download site with no checksums.
Targeted at embodied-AI/RL researchers with clear scenario coverage (Linux/macOS/Windows, Docker, headless X server), good audience fit. Deductions: capability boundaries mentioned only in one sentence ('some objects and actions not yet supported in Unity Simulator'), incomplete; trigger/API behavior details deferred to external docs; environment setup (X server, GPUs) steps are rough.
README is well structured (contents, install, examples, dataset, citation, contributors), install notes match the pip package, examples are plentiful (notebook, generate_video, etc.), MIT LICENSE is complete. Deductions: changelog is narrative 'What is New' only, no formal version file (no version in pyproject); known-limitations notes are scattered; maintenance responsibility and update path are not explicitly stated; external assets (executable, datasets) depend on virtual-home.org with no stated persistence plan.
As a multi-agent household simulation API it holds a distinct position with clear marginal value, paper backing, and PyPI distribution. Deductions: actual output usability (video/graph quality, API stability) is unverified without execution and static evidence is limited; runtime cost (large Unity executable download, GPU/monitor) is high for casual users, though documented.
Claims are traceable to cited papers and external documentation links. Deductions: performance and 'stability improvements' claims lack metrics; no locked dependencies or test evidence available for cross-checking; claim/inference separation relies mostly on external docs, with limited in-repo evidence.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Unity executables are downloaded over HTTP links with no checksums; verify integrity independently before running.
- startx.py requires sudo to start an X server, granting elevated privileges; prefer isolated environments such as Docker.
- Dependency metadata is incomplete (pyproject is nearly empty); static review cannot verify dependency security — do not assume supply-chain safety.
- This is a static review; no simulation was executed, so output quality and API stability remain unverified.