CRAB Cross-Environment Benchmark
A Python-centric framework for building and evaluating multimodal language-model agent environments.
What does this agent do, and when should you use it?
CRAB is a Python-centric framework for building benchmark environments for LLM agents. It brings in-memory environments, Docker-hosted environments, virtual machines, and distributed physical machines behind a unified interface when they are accessible through Python functions. Developers add actions by applying the @action decorator to Python functions and define environments by combining actions. Tasks and their evaluators are also defined in Python, while a graph evaluator produces fine-grained metrics. The repository includes the CRAB-Benchmark-v0 datasets and experiment code, plus template examples that run with an OpenAI agent.
CRAB uses Python functions as the environment-access boundary: applying @action to a function adds an action, and combining actions defines an environment. An agent accesses multiple environments at the same time through a unified interface; the documented deployment options include in-memory, Docker-hosted, virtual-machine, and distributed physical-machine environments. Users define tasks and corresponding evaluators in Python, and the graph evaluator returns fine-grained metrics. The supplied example workflow sets OPENAI_API_KEY and runs python examples/single_env.py or python examples/multi_env.py.
- A research team creating repeatable tasks and evaluators for multimodal language-model agents.
- An engineering group evaluating an agent that must coordinate several environments accessible through Python functions.
- A benchmark author bringing Docker-hosted services, virtual machines, and in-memory environments into one evaluation setup.
- An agent evaluator who needs fine-grained task metrics from a graph evaluator.
- A developer validating an OpenAI-agent integration with the provided single-environment or multi-environment templates.
What are this agent's strengths and limitations?
- Its @action decorator makes Python functions the direct mechanism for defining environment actions.
- A unified interface spans in-memory, Docker-hosted, virtual-machine, and distributed physical-machine deployment options.
- The graph evaluator is explicitly intended to provide fine-grained metrics.
- The repository includes runnable OpenAI-agent templates for both single- and multi-environment setups.
- The documented runnable templates require an OPENAI_API_KEY, so that path depends on OpenAI credentials.
- Target environments must be accessible through Python functions; systems outside that boundary need integration work.
- Docker, virtual machines, and physical machines are named as deployment options, but the supplied material does not give configuration details for each.
- License information is unknown in the supplied repository metadata.
How do you install or deploy this agent?
Python 3.10 or newer is required. Install with: pip install crab-framework[client]. To run the provided OpenAI template examples, set credentials with export OPENAI_API_KEY=<your api key>, then run python examples/single_env.py or python examples/multi_env.py.
How do you use this agent?
Add @action to Python functions to create actions, then combine those actions to define an environment. Define tasks and their evaluators in Python, and expose multiple accessible environments through the unified interface. For a first run, set OPENAI_API_KEY and execute python examples/single_env.py; use python examples/multi_env.py for the multi-environment template. CRAB-Benchmark-v0 datasets and experiment code are in crab-benchmark-v0/.