Semia — Security Audit for AI Agent Skills
Statically audits AI agent skill files without executing them, revealing every shell command, network call, and secret read before you decide to trust a skill.
Per-dimension scores and reasoning
Evidence shows: README states skills are 'read as data — never executed'; SECURITY.md gives a full threat model (no execution, no exfiltration, no artifact forging); pyproject declares zero runtime dependencies; CI uses read-only permissions and SHA-pinned actions. Deductions: analyzer core source is not in the provided evidence, so 'never executes' is a documented assertion; repair rewrites SKILL.md with no visible confirmation gate; skill text is sent to third-party LLM APIs without explicit disclosure in the README; API keys are covered only via env vars, no sensitive-data guidance; no rollback/uninstall path; attribution stops at the RiemaLabs copyright line.
Evidence shows: README, SECURITY.md, pyproject and CI are mutually consistent (0.1.3, Alpha, pre-1.0 support policy); CI spans platforms with wheel-install smoke tests and zipapp runnability checks; synthesis loop has retries and metadata. Deductions: no user-facing failure-message documentation — LlmSynthesisError is only inferable from test code; availability/packaging of external components such as Soufflé is mentioned but not shown.
Evidence shows: clear audience (developers and skill reviewers), three plugin hosts plus CLI, CI across Linux/macOS/Windows and Python 3.11/3.12, explicit scope and threat boundary in SECURITY.md. Deductions: plugin SKILL.md trigger text is absent, so trigger precision cannot be verified; runtime requirement around Soufflé is unstated; ADVANCED_USAGE.md is referenced but not provided.
Evidence shows: very complete install notes (pip plus shell and interactive paths for three hosts), full Apache-2.0 text with SPDX and NOTICE reference, artifact tables, DCO contribution flow, and security response targets (3/10/90 days). Deductions: no CHANGELOG — only a pyproject version; known limitations appear only scattered (permissive mypy baseline, pre-1.0); no FAQ; maintenance responsibility rests on documented promises rather than visible release history.
Evidence shows: strong output design — severity-ranked report.md with every finding anchored to a source line, SARIF 2.1.0 for GitHub Code Scanning, JSON for programmatic use; only the synthesize stage needs an LLM while three stages are deterministic, keeping cost boundaries clear. Deductions: no claims were executed in this static review; audit quality depends on LLM synthesis accuracy and that risk is not quantified.
Evidence shows: 'every finding traces back to a source line' is the core design, carried through README and SARIF usage; the 'deterministic acceptance boundary' explicitly separates LLM-generated facts from validated facts; tests demonstrate evidence-match-rate style validation metrics. Deductions: the cited arXiv paper and CI badges cannot be independently corroborated in this static review; core implementation files were not provided, so fact/inference separation is verifiable only at the documentation level.
- Scanning sends the audited skill's content to your configured third-party LLM API (OpenAI/Anthropic etc.); confirm this data flow before auditing sensitive skills.
- 'Never executes skill code' is a documented claim, not verified by execution in this static review; trial in an isolated environment first.
- semia repair rewrites SKILL.md directly; back up or version-control targets yourself — no rollback mechanism is documented.
- semia scan requires LLM credentials (or a host agent); the synthesize stage is unavailable in fully offline/keyless settings.
- The project is pre-1.0 (0.1.3, Alpha): rules and APIs may change and there is no CHANGELOG to track this.
- Publisher identity is unverified; the cited paper and detection-effectiveness claims are not independently corroborated — do not base security decisions solely on this tool's output.
What does this agent do, and when should you use it?
Semia is a security audit tool for AI agent skill files such as SKILL.md. It treats skills purely as data — never executing them — and runs a four-stage pipeline (prepare → synthesize → detect → report) that produces an evidence-backed report where every finding traces to a specific source line. Only the synthesize stage uses an LLM (supporting OpenAI, Anthropic, DeepSeek/OpenRouter/vLLM-compatible endpoints, or local Claude Code / Codex CLI logins); the remaining stages are fully deterministic and require no API key. When run as a plugin inside Codex, Claude Code, or OpenClaw, the host agent itself performs the synthesize step, so no separate API key is needed. Reports default to report.md and can be exported as SARIF 2.1.0 for GitHub Code Scanning or as structured JSON. A semia repair command traces each violation back through Datalog rules to its root cause and has an LLM generate a SKILL.md patch. The underlying technique is described in an arXiv paper (2605.00314), and the code ships under the Apache-2.0 license.
A single command, semia scan ./some-skill, runs the full pipeline: prepare (normalizes skill text with stable line anchors) → synthesize (calls your configured LLM provider to extract behavior facts) → detect (evaluates Datalog rules to derive findings) → report (writes severity-ranked findings to .semia/runs/<skill-slug>/report.md). semia report converts results to SARIF or JSON. semia repair reads an existing scan, traces each violation through Datalog rules to its root cause, then has an LLM generate a patched/SKILL.md fix. When installed as a plugin in Codex / Claude Code / OpenClaw, you simply ask the host agent to 'Run Semia audit on ./some-skill'; the host handles synthesize while the bundled semia.pyz performs deterministic prepare/detect/report.
- Security teams vetting third-party Claude Code skills before adoption, auditing hidden shell commands, network calls, and secret reads as part of a review gate.
- CI platform engineers wiring SARIF output into a skill PR so GitHub Code Scanning annotates audit findings directly on the PR.
- Individual developers quickly checking whether a community skill reads credentials or makes unexpected network calls before installing it.
- Plugin users on Codex / Claude Code / OpenClaw running an audit conversationally via the host agent, without configuring any API key.
- Skill authors using semia repair before publishing to auto-generate SKILL.md security patches or add explicit constraints to problematic content.
What are this agent's strengths and limitations?
- Every finding is anchored to a specific source line, making reports verifiable and traceable rather than vague trust statements.
- Only the synthesize stage needs an LLM; prepare/detect/report are fully deterministic and can run offline or keylessly in plugin mode.
- Works across three host agents (Codex, Claude Code, OpenClaw) plus CLI, with SARIF output that plugs directly into GitHub Code Scanning.
- semia repair closes the loop from detection to auto-generated fixes, not just reporting.
- The scan mode requires configuring at least one LLM provider (OpenAI/Anthropic/local CLI login), incurring API cost; without it synthesize cannot run.
- Coverage is limited to Markdown skill files (e.g. SKILL.md); it does not audit other agent tools or source code.
- While detection is deterministic, fact extraction still depends on LLM synthesis, with project-mitigated uncertainty (bounded by Datalog acceptance rules).
- As a newer project, the maturity of its community ecosystem and detector rule library is unclear; rule coverage should be evaluated independently.
How do you install or deploy this agent?
CLI: pip install semia-audit (requires Python 3.11+). As a host plugin: Codex — run codex plugin marketplace add berabuddies/Semia and enable [plugins."semia@semia"] enabled = true in ~/.codex/config.toml; Claude Code — claude plugin marketplace add berabuddies/Semia && claude plugin install semia@semia; OpenClaw — openclaw plugins install clawhub:semia. For CLI scans (not plugins), configure an LLM provider first, e.g. export OPENAI_API_KEY=sk-... or export SEMIA_LLM_PROVIDER=anthropic && export ANTHROPIC_API_KEY=sk-ant-....
How do you use this agent?
CLI: semia scan ./some-skill runs a full audit into .semia/runs/<skill-slug>/; semia report .semia/runs/some-skill --format sarif or --format exports machine-readable results; semia repair .semia/runs/some-skill --from-scan or semia repair ./some-skill (scan + repair in one shot). Override the model with --model <name> or SEMIA_LLM_MODEL. Plugin mode: just ask your host agent to 'Run Semia audit on ./some-skill' — no API key required.