AssetOpsBench for Industrial Operations
An MCP-based benchmark and framework for building and evaluating industrial asset-maintenance agent workflows.
- Source repo
- IBM/AssetOpsBench
- Stars
- ★ 2.3k
- Last updated
- 6d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 50/100 · Major gaps
At a glance
- Works with
- Universal · cross-platformOpenAI API · Claude API
- You'll need
- Typical use
- A maintenance engineer needs to list assets in the MAIN site or identify the sensors installed on a particular chiller.
- Main limitation
- The only supplied local scenario command is marked “to be enabled,” leaving the first verifiable CLI execution path incomplete.
- Source review
- 50/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
AssetOpsBench is an open framework and benchmark for developing, orchestrating, and evaluating domain-specific agents for industrial asset operations and maintenance. It exposes MCP services for IoT, FMSR, TSFM, work orders, and vibration analysis, alongside reproducible scenarios and evaluation pipelines for multi-step workflows. The README reports 9 asset classes, 141+ scenarios, 5 domain agents, and two orchestration frameworks. Its agent implementations include Plan Execute for use with any LLM, Deep Agent with planning, sub-agents, and a virtual filesystem, plus ReAct orchestrators for Claude and OpenAI models. The delivery boundary is a local Python package and MCP-server environment, with linked Hugging Face dataset, Playground, and Colab resources.
AssetOpsBench invokes industrial-data and analysis capabilities through MCP. The IoT server exposes sites, asset_ids, asset_detail, assets, find_assets_by_sensors, installed_sensors, and measured_sensors; FMSR exposes get_failure_modes, generate_failure_modes, and add_failure_modes; TSFM covers task and series analysis, model and feature catalogs, recipe execution, evaluation, and run/result lookup. The stated TSFM surface has 41 tools for model cards, feature cards, recipe execution, evaluation, and result/run retrieval. Plan Execute runs a sequential plan-and-execute workflow, while Deep Agent adds planning, sub-agents, and a virtual filesystem; Claude Agent and OpenAI Agent use ReAct orchestration with agent-as-tool delegation. Documented example tasks include listing assets at a site, retrieving pump failure modes, finding and loading a forecasting model for Chiller 6, and generating a work order for anomaly detection.
- A maintenance engineer needs to list assets in the MAIN site or identify the sensors installed on a particular chiller.
- A reliability specialist needs to retrieve known pump failure modes or generate and add new failure-mode records.
- A data scientist needs to profile Chiller 6 time series, find a forecasting model, select features, execute a recipe, and inspect evaluation results.
- A facilities operations team needs to turn an anomaly-detection task for Chiller 6 into a work-order workflow.
- A researcher needs reproducible industrial scenarios and trajectory evaluation when comparing LLM-based multi-step workflows.
How do you install or deploy this agent?
The README-supported installation is:
git clone https://github.com/IBM/AssetOpsBench.git
cd AssetOpsBench
pip install -e .This requires a Python environment with python and pip. The README identifies INSTRUCTIONS.md as the source for MCP-server and plan-execute-runner setup, but the supplied material does not provide model API credentials, environment variables, MCP startup commands, or versioned dependency requirements.
How do you use this agent?
The README provides this scenario command:
python -m assetopsbench.run --scenario "List all sensors of Chiller 6 in MAIN site"However, it is explicitly marked “to be enabled,” so it is not confirmed as a working local first-run command. The README also links a Colab LLM Agent illustration and a Hugging Face Playground. The concrete local MCP configuration and plan-execute execution steps are said to be in INSTRUCTIONS.md, whose contents are not supplied here.
What are this agent's strengths and limitations?
- It unifies IoT, failure-mode, time-series model/feature, work-order, and vibration capabilities behind MCP for cross-domain industrial workflows.
- The TSFM server has an explicitly described 41-tool surface spanning series characterization, model discovery, feature work, recipe execution, evaluation, and result tracing.
- It offers distinct execution approaches: sequential Plan Execute, long-horizon Deep Agent, and provider-specific ReAct orchestrators with agent-as-tool delegation.
- It includes reproducible scenarios and an evaluation pipeline described as measuring reasoning, execution, and data handling across six dimensions.
- The only supplied local scenario command is marked “to be enabled,” leaving the first verifiable CLI execution path incomplete.
- MCP startup details, model API credentials, environment variables, and exact dependency versions require
INSTRUCTIONS.md, which is not included in the supplied evidence. - The documented scope is industrial asset operations and simulated industrial environments; there is no evidence for general-purpose business or consumer workflows.
- The Claude Agent and OpenAI Agent paths explicitly depend on their respective model providers.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| AssetOpsBench for Industrial Operations This agent | 50 · Major gaps | ★ 2.3k | 6d ago | Python | OpenAI API · Claude API |
| Safari MCP | 78 · Good | ★ 205 | today | JavaScript | Claude Code · Claude.ai |
| Nuphus Desktop Automation MCP | 72 · Some gaps | ★ 308 | 7d ago | Rust | Claude.ai · OpenAI API · Claude API |
| Keinsaas Navigator | 69 · Some gaps | ★ 1.2k | 1mo ago | TypeScript | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: secret-scan workflow (gitleaks/trufflehog) and couchdb data file guard indicate some attention to sensitive data and dependency security; but no user confirmation mechanism (default bypassPermissions), data flow transparency documentation, rollback mechanism, or explicit least-privilege design found. Deductions: user confirmation missing, rollback missing, least-privilege evidence insufficient.
Evidence shows: unit tests (test_runner.py) cover ClaudeAgentRunner, and README and pyproject.toml are consistent; but dependency versions are partially pinned (litellm==1.94.0) and partially loose (>=), and no explicit failure message design provided. Deductions: dependency availability not fully verified, failure message documentation insufficient.
Evidence shows: README describes multiple scenarios (IoT, FMSR, TSFM, WO), multiple agent frameworks, and MCP servers with clear audience; but capability boundaries and trigger precision only partially specified (e.g., TSFM tool list), environment fit (e.g., MCP config) relies on INSTRUCTIONS.md not provided. Deductions: capability boundaries and trigger precision evidence insufficient.
Evidence shows: README, LICENSE (Apache-2.0), pyproject.toml, install instructions (pip install -e .), example scenarios, and contribution guidelines present; but no explicit version changelog, known limitations section, or naming stability statement. Deductions: version changelog and known limitations missing.
Evidence shows: outputs are AgentResult and Trajectory, evaluation pipeline (LLM Judge) exists, marginal value shown via multiple scenarios and agent frameworks; but cost-benefit analysis (e.g., API costs, runtime) not provided. Deductions: cost-benefit evidence insufficient.
Evidence shows: README references papers, datasets, competitions, and multiple external links, providing good traceability; but separation of facts and inferences (e.g., benchmark results) not explicitly stated. Deductions: fact-inference separation insufficient.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Default permission mode is bypassPermissions, which may execute arbitrary tool calls; user confirmation or restrictions needed.
- Dependency versions are partially unpinned, potentially introducing incompatibility or security risks.
- Lack of explicit rollback mechanism and data flow transparency documentation.
- Benchmark results and competition claims lack independent verification; cite with caution.
FAQ
What industrial functions can it connect to?
Is it locked to one model provider?
Can I run the README CLI example immediately?
python -m assetopsbench.run example is marked “to be enabled.”