AssetOpsBench for Industrial Operations
An MCP-based benchmark and framework for building and evaluating industrial asset-maintenance agent workflows.
- Source repo
- IBM/AssetOpsBench
- Stars
- ★ 2.5k
- Last updated
- 6d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 54/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A maintenance engineer needs to list assets in the MAIN site or identify the sensors installed on a particular chiller.
- Not a fit if
- Users wanting a turnkey tool without configuring MCP servers and multiple services
- Developers building general business or consumer apps; scope is industrial asset operations only
- Maintenance
- Commits in the last 30 dayslast commit 2026-11-15+5.4 ★/day over 7 days
- Source review
- 54/100 · Major gaps
What does this agent do, and when should you use it?
AssetOpsBench is an open framework and benchmark for developing, orchestrating, and evaluating domain-specific agents for industrial asset operations and maintenance. It exposes MCP services for IoT, FMSR, TSFM, work orders, and vibration analysis, alongside reproducible scenarios and evaluation pipelines for multi-step workflows. The README reports 9 asset classes, 141+ scenarios, 5 domain agents, and two orchestration frameworks. Its agent implementations include Plan Execute for use with any LLM, Deep Agent with planning, sub-agents, and a virtual filesystem, plus ReAct orchestrators for Claude and OpenAI models. The delivery boundary is a local Python package and MCP-server environment, with linked Hugging Face dataset, Playground, and Colab resources.
AssetOpsBench invokes industrial-data and analysis capabilities through MCP. The IoT server exposes sites, asset_ids, asset_detail, assets, find_assets_by_sensors, installed_sensors, and measured_sensors; FMSR exposes get_failure_modes, generate_failure_modes, and add_failure_modes; TSFM covers task and series analysis, model and feature catalogs, recipe execution, evaluation, and run/result lookup. The stated TSFM surface has 41 tools for model cards, feature cards, recipe execution, evaluation, and result/run retrieval. Plan Execute runs a sequential plan-and-execute workflow, while Deep Agent adds planning, sub-agents, and a virtual filesystem; Claude Agent and OpenAI Agent use ReAct orchestration with agent-as-tool delegation. Documented example tasks include listing assets at a site, retrieving pump failure modes, finding and loading a forecasting model for Chiller 6, and generating a work order for anomaly detection.
- A maintenance engineer needs to list assets in the MAIN site or identify the sensors installed on a particular chiller.
- A reliability specialist needs to retrieve known pump failure modes or generate and add new failure-mode records.
- A data scientist needs to profile Chiller 6 time series, find a forecasting model, select features, execute a recipe, and inspect evaluation results.
- A facilities operations team needs to turn an anomaly-detection task for Chiller 6 into a work-order workflow.
- A researcher needs reproducible industrial scenarios and trajectory evaluation when comparing LLM-based multi-step workflows.
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Install / run
pip install -e
How do you install or deploy this agent?
The README-supported installation is:
git clone https://github.com/IBM/AssetOpsBench.git
cd AssetOpsBench
pip install -e .This requires a Python environment with python and pip. The README identifies INSTRUCTIONS.md as the source for MCP-server and plan-execute-runner setup, but the supplied material does not provide model API credentials, environment variables, MCP startup commands, or versioned dependency requirements.
How do you use this agent?
The README provides this scenario command:
python -m assetopsbench.run --scenario "List all sensors of Chiller 6 in MAIN site"However, it is explicitly marked “to be enabled,” so it is not confirmed as a working local first-run command. The README also links a Colab LLM Agent illustration and a Hugging Face Playground. The concrete local MCP configuration and plan-execute execution steps are said to be in INSTRUCTIONS.md, whose contents are not supplied here.
What are this agent's strengths and limitations?
- It unifies IoT, failure-mode, time-series model/feature, work-order, and vibration capabilities behind MCP for cross-domain industrial workflows.
- The TSFM server has an explicitly described 41-tool surface spanning series characterization, model discovery, feature work, recipe execution, evaluation, and result tracing.
- It offers distinct execution approaches: sequential Plan Execute, long-horizon Deep Agent, and provider-specific ReAct orchestrators with agent-as-tool delegation.
- It includes reproducible scenarios and an evaluation pipeline described as measuring reasoning, execution, and data handling across six dimensions.
- The only supplied local scenario command is marked “to be enabled,” leaving the first verifiable CLI execution path incomplete.
- MCP startup details, model API credentials, environment variables, and exact dependency versions require
INSTRUCTIONS.md, which is not included in the supplied evidence. - The documented scope is industrial asset operations and simulated industrial environments; there is no evidence for general-purpose business or consumer workflows.
- The Claude Agent and OpenAI Agent paths explicitly depend on their respective model providers.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| AssetOpsBench for Industrial Operations This agent | 54 · Major gaps | CLIFree + model costs | ★ 2.5k | 6d ago | Python | OpenAI API · Claude API |
| Keinsaas Navigator | 72 · Some gaps | Self-hosted serviceFree + model costs | ★ 1.2k | 14d ago | TypeScript | OpenAI API · Claude API |
| Nuphus Desktop Automation MCP | 64 · Some gaps | MCP serverFree + model costs | ★ 335 | 9d ago | Rust | Claude.ai · OpenAI API · Claude API |
| Safari MCP | 63 · Some gaps | MCP serverFree | ★ 246 | 5d ago | JavaScript | Claude Code · Claude.ai |
How does FollowAgents rate this agent?
Why each dimension lost points
The code shows domain-separated MCP tools, an IoT read-only description, CouchDB credentials loaded from environment variables, and sanitized database errors. However, there is no unified least-privilege policy; WO exposes write operations, and the Claude runner test shows a bypassPermissions default. The packet does not show confirmation for writes, recovery mechanisms, a sensitive-data policy, or a complete data-flow account, so these areas lose points. FMSR result types distinguish curated and LLM sources, and secret-scan workflows provide limited attribution evidence, but they do not establish traceability for all data and generated content.
The code and README broadly agree on the servers, tools, and write interfaces. The README quick-start scenario is marked “to be enabled,” which conflicts with the broader framework presentation, and the excerpts do not verify every interface. Pyproject lists dependencies and version floors, pins LiteLLM, and constrains two dependencies, but many ranges remain broad; availability or security cannot be confirmed from this alone. IoT, FMSR, and TSFM expose concrete errors or graceful degradation in some paths, but error handling is not shown to be consistent everywhere.
The README identifies maintenance engineers, reliability specialists, facility planners, and researchers, and covers multiple asset classes, single-domain and multistep tasks, and several Agent frameworks. Tool descriptions distinguish read-only evidence, model execution, and work-order writes, with some clear input and task constraints. Cross-agent capability boundaries, authorization triggers, and production-environment requirements are not fully documented, so full marks are not justified.
The README, linked MCP documentation, setup notes, task examples, limitation notes, and Apache-2.0 LICENSE provide useful entry points; server and script names are reasonably stable. Deductions reflect the quick-start command still being marked “to be enabled,” no changelog or clear maintenance responsibility/update commitment in the supplied material, and only limited cross-checking between documentation and source excerpts.
Domain-specific MCP tools, benchmark scenarios, structured results, plan-execute workflows, and multi-agent orchestration offer clear added value for industrial maintenance tasks. Outputs include typed results and errors that are generally usable downstream. The packet does not adequately explain full result presentation, operating costs, model/data service expenses, or measured benefit, so output and cost-benefit scores are reduced.
The README names tools and links to detailed documentation; source excerpts show some matching tools, input descriptions, and result models. The test sample supports limited Claude runner behavior. Deductions reflect that the packet is partial, offers no independent corroboration for many scale, performance, paper, and competition claims, and cannot cross-check most claims against code. The code shows some separation between factual evidence and model generation, but the broader claims do not consistently mark inference versus measured results.
- The work-order server exposes create, update, approve, assign, close, and cancel writes; the supplied material does not show pre-call confirmation or undo/recovery safeguards.
- The README quick-start scenario command is marked as not yet enabled; do not assume that entry point is available.
- This is a static review of a file sample; no code was executed and paper, competition, performance, and scale claims were not independently verified.
FAQ
What industrial functions can it connect to?
Is it locked to one model provider?
Can I run the README CLI example immediately?
python -m assetopsbench.run example is marked “to be enabled.”What does the evaluation cover?
Is AssetOpsBench for Industrial Operations free?
Is AssetOpsBench for Industrial Operations open source?
Is AssetOpsBench for Industrial Operations safe to use?
Is AssetOpsBench for Industrial Operations still maintained?
Who makes AssetOpsBench for Industrial Operations?
Thanks — we use this to improve the page.