OpenRCA Software Failure Analyst
Benchmarks and diagnoses software failures across metrics, traces, and logs.
The README identifies model-provider configuration, telemetry placement, result directories, and evaluation-file flow, while the official organization, paper citation, and MIT copyright establish unusually clear attribution, justifying full source-attribution credit. Dependencies are pinned and SECURITY.md supplies a formal vulnerability-reporting channel, but no dependency audit, vulnerability response, or upgrade process is shown. There is no documented least-privilege design, pre-action confirmation, sandbox, network boundary, or rollback mechanism. The example stores an API key in plaintext YAML, and although private telemetry is mentioned, sanitization, retention, transmission, and log-protection practices are absent. External API use and local writes can be inferred, but the full data flow is not documented.
Installation, data layout, prediction format, evaluation, reproduction, and reconstruction instructions are broadly consistent, with explicit Python, storage, memory, timezone, and sampling constraints. Exact dependency pins and Python requirements support availability, but the dataset relies on Google Drive without checksums, mirrors, or offline guarantees. The FAQ addresses several common result mismatches, yet the supplied material does not show runtime error handling, retries, timeouts, partial-failure behavior, or actionable agent diagnostics.
Audience and scenarios are thoroughly identified: software-operations RCA research across Telecom, Bank, Market, custom agents, and reconstruction from private telemetry. The benchmark, RCA-agent baseline, I/O contract, and prohibition on reading ground truth are reasonably bounded, but model-safety boundaries and unsuitable use cases are incomplete. CLI entry points and parameters provide precise activation. Environment guidance covers Python versions, directory layout, minimum resources, and timezone, while omitting OS support, provider compatibility, network assumptions, and finer deployment requirements.
The README has a usable structure spanning installation, evaluation, reproduction, reconstruction, FAQ, citation, and disclaimers; examples are abundant and the FAQ is concrete. Naming is mostly stable, but dataset/datasets, record.csv/records.csv, and the TESTS placeholder create minor ambiguity. Resource demands, timezone, sampling, and network-fault analysis limits are disclosed, though there is no systematic known-issues inventory. The complete MIT license earns full credit. No release versioning or changelog evidence is supplied. Microsoft ownership and a security escalation route are clear, but named maintainers, maintenance cadence, and a general support path are not.
The prediction JSON schema, minimum CSV field, failure-ordering rule, report location, and complete command examples make outputs directly usable for evaluation, supporting full output-usability credit. Using Python retrieval and analysis to reduce long-context processing presents plausible incremental value over direct prompting, but the supplied evidence contains no comparative results. The README candidly states an approximately 80GB storage and 32GB memory recommendation and implies paid model-API use, but does not quantify tokens, runtime, monetary cost, or resource-to-benefit tradeoffs.
The paper citation, fixed model identifier, archived prediction paths, explicit commands, data schemas, and license make major claims traceable. README, LICENSE, SECURITY.md, and pinned requirements corroborate provenance, licensing, setup, and governance across sources, but no paper body, tests, or result tables are supplied to independently corroborate performance or scalability. The documentation generally distinguishes the benchmark, baseline, example models, and disclaimers; however, claims about avoiding excessive context and scalability are not directly substantiated in the provided files, so fact/inference separation is not complete.
- The API-key example uses plaintext YAML; do not commit it, and establish secret management, log redaction, and telemetry-transfer controls before using real data.
- Runs invoke external model APIs and create local test and monitor files, but per-action confirmation, network boundaries, retention, and rollback are undocumented.
- The dataset is downloaded from external Google Drive without supplied checksums or mirrors; independently verify integrity and provenance after download.
- Pinned versions do not constitute a dependency-security audit; no scan results, lockfile hashes, or update policy are provided.
- This is a research benchmark and baseline, not a production incident-response agent with demonstrated operational safety; neither the paper venue nor Microsoft provenance guarantees runtime safety or correct diagnoses.
What does this agent do, and when should you use it?
OpenRCA is a benchmark for root-cause analysis in software operating environments, accompanied by RCA-agent as its baseline agent. Given a natural-language query, it examines KPI time series, dependency trace graphs, and semi-structured logs to identify when a failure occurred, which component caused it, and why. RCA-agent uses Python to retrieve and analyze telemetry instead of placing the entire dataset in the model context. The repository includes command-line workflows for evaluation, paper-result reproduction, and generation of new tasks from OpenRCA or schema-compatible private telemetry. It runs in a local Python environment, reads and writes filesystem data, and requires model API configuration when executing model-backed baselines.
The workflow starts with query.csv, record.csv, and date-partitioned telemetry/log, telemetry/metric, and telemetry/trace data under dataset/{SYSTEM}, where the system can be Telecom, Bank, or Market. RCA-agent retrieves and analyzes that telemetry with Python and uses a configured model to reason about the root cause. python -m rca.run_agent_standard --dataset ... runs the standard agent, while run_baseline_balanced and run_baseline_oracle invoke other paper baselines. Predictions are written as CSV data with at least a prediction field containing a JSON-like, chronologically ordered list of failures. Each failure specifies root cause occurrence datetime, root cause component, and root cause reason. python -m main.evaluate compares predictions with ground-truth query files and writes a report CSV, while python -m main.generate creates new queries from main/task_specification.json and records matching the OpenRCA schema.
- An LLM research team can benchmark models on root-cause localization that requires joint reasoning over metrics, traces, and logs.
- An agent developer can run RCA-agent on the Telecom, Bank, or Market datasets and compare it with the balanced and oracle baseline scripts.
- An observability researcher can study failure diagnosis that depends on several telemetry types rather than KPI-only detection.
- A team with private telemetry can build internal benchmark tasks after mapping its records to the OpenRCA
record.csvschema. - An ICLR 2025 researcher can evaluate the archived predictions or rerun the documented experimental entry points.
What are this agent's strengths and limitations?
- It combines KPI time series, dependency traces, and semi-structured logs in one root-cause-analysis benchmark.
- RCA-agent retrieves and analyzes telemetry through Python, reducing the need to send an excessively long dataset directly to the model.
- A defined prediction schema, evaluation command, and report output make custom-model and custom-agent comparisons practical.
- The task generator supports both OpenRCA telemetry and private records that follow the required schema.
- The documented resource footprint is substantial: at least 80GB of storage and 32GB of memory are recommended.
- The telemetry is not included in the empty
dataset/directory and must be downloaded separately from Google Drive. - Model-backed experiments require external API credentials and may incur provider charges; no cost estimate is documented.
- All fault records use UTC+8, so incorrect timezone handling can produce misleading timestamp or component matches.
- Network faults commonly require parent-child span latency analysis and often cannot be identified from KPIs alone.
How do you install or deploy this agent?
Python 3.10 or 3.11 is required. The repository recommends at least 80GB of storage and 32GB of memory.
# Optional Conda environment
conda create -n openrca python=3.10
conda activate openrca
git clone https://github.com/microsoft/OpenRCA.git
cd OpenRCA
pip install -r requirements.txtDownload the telemetry dataset from the Google Drive location supplied by the repository and place it under the initially empty dataset/ directory. Model-backed runs also require rca/api_config.yaml. The documented OpenAI example sets SOURCE: "OpenAI", MODEL: "gpt-4o-2024-05-13", and API_KEY: "sk-xxxxxxxxxxxxxx".
How do you use this agent?
A first RCA-agent run can target the Bank dataset:
python -m rca.run_agent_standard --dataset BankThe two Market cloudbeds are processed separately:
python -m rca.run_agent_standard --dataset Market/cloudbed-1
python -m rca.run_agent_standard --dataset Market/cloudbed-2Test runs create a test directory containing results and monitor files. To evaluate a custom prediction file, run:
python -m main.evaluate \
-p path/to/prediction.csv \
-q dataset/Bank/query.csv \
-r test/report.csvThe prediction CSV must contain at least a prediction field. To generate a task from compatible records, use:
python -m main.generate \
-s main/task_specification.json \
-r path/to/record.csv \
-q path/to/query.csv \
-t UTC+8python -m main.generate -d True regenerates random OpenRCA queries.
How does this agent compare with similar options?
The repository provides RCA-agent alongside the run_baseline_balanced and run_baseline_oracle paper baselines. RCA-agent's stated distinction is its use of Python for telemetry retrieval and analysis, which avoids requiring the model to process an excessively long context. The supplied material does not report comparative scores, so it does not establish which approach performs best.
FAQ
Does running it require a paid model API?
Can I evaluate my own agent?
prediction field. For submitted results, you must disclose whether the agent is open source; a closed agent also requires descriptions of its tools, MCPs, or skills so reviewers can check that it does not read ground truth from records.csv.Why might the observed fault time differ from the recorded time?
Can it ingest any private observability dataset unchanged?
record.csv, and main/task_specification.json must be configured. No direct, conversion-free connector to arbitrary observability platforms is documented.Are deployment relationships modeled identically across all datasets?
metric_container.csv, and Telecom deployment relationships are provided through a linked spreadsheet.