AuditPilot
A traceable enterprise audit workflow for evidence retrieval, governed execution, review, remediation, and delivery.
Per-dimension scores and reasoning
Evidence shows RBAC/ABAC, tenant isolation, upload governance, HMAC-signed checkpoints, but user confirmation is only implicit in human review, not mandatory; data flow transparency has tool call logs and evidence graph, but no full data flow diagram; sensitive data handling has upload scanning and secret management guidance, but no runtime redaction; dependency security has pip-audit and bandit, but no vulnerability remediation records; external effects have tool permission checks and circuit breakers, but no detailed audit of external interactions; rollback has delete operations and versioning, but no data rollback mechanism; source attribution has audit logs and evidence lineage, but no third-party content sources.
Self-consistency is good, module interfaces are clear, test coverage is extensive; dependency availability has requirements.txt and CI matrix, but no availability guarantees; failure messages have error types and status codes, but no user-friendly error prompts.
Target audience is clear for enterprise audit scenarios, multiple use cases provided; capability boundaries have design boundary notes, but no detailed permission matrix; trigger precision has intent routing and tool schemas, but not all trigger conditions; environment fit has local and production configuration notes, but no detailed container deployment guide.
Information architecture is clear, project layout and module descriptions provided; install notes are detailed with Windows and macOS/Linux steps; naming is stable and consistent; examples and FAQ have basic operations and screenshots, but no FAQ; known limitations have Reality notes, but not all limitations; license is missing; version and changelog have version number but no changelog; maintenance responsibility is unclear, but CI and tests indicate maintenance activity.
Output usability has reports and delivery packages, but no diversity of output formats; marginal value has clear differentiation, but no comparison with alternatives; cost-benefit has optional configurations and default fallback mode, but no performance benchmarks.
Claim traceability has tests and CI, but not all claims verified; cross-source corroboration has tests and documentation, but no independent verification; fact-inference separation has Reality notes, but not all facts and inferences clearly separated.
- License is missing, need to specify an open-source license.
- User confirmation is not mandatory, need to clarify human review triggers.
- Changelog is missing, need to add version history.
- Dependency security scans lack vulnerability remediation records.
What does this agent do, and when should you use it?
AuditPilot is a FastAPI workbench for enterprise audit delivery, covering audit projects, control matrices, procedures, sampling, findings, remediation, and delivery packages. Its runtime coordinates Planner, Evidence, Control, Risk, Compliance, Remediation, Verification, and Delivery roles through a bounded, dependency-aware Plan / Execute / Reflect loop. Evidence work is supported by multi-path retrieval, metadata filtering, source locations, and an Evidence Graph linking tasks, steps, tool runs, and artifacts. The repository also includes a layered Evaluation Harness, release gating, human-review exits, RBAC plus project ABAC, and local SQLite WAL persistence. It fits self-hosted prototypes or internal tools that need governed audit traces, but it does not establish business accuracy, horizontal scale, or a complete enterprise identity stack.
A user creates audit scope in the web workbench and enters an audit subject, standard, or risk scenario in Agent Collaboration. The Hybrid Intent Router uses Working, Episodic, and Profile Memory to route work; the role chain plans, retrieves evidence, performs control and risk checks, and produces remediation and delivery results in a bounded dependency-aware loop. Agentic RAG uses TF-IDF, keyword retrieval, fusion reranking, and metadata filtering, returns page or section sources, and flags conflicting or missing evidence; semantic embeddings are optional. Skills / MCP-style Tools apply RBAC, tenant isolation, caching, and circuit-breaking before execution, while recording call logs and metrics. The Evaluation Harness assesses task results, trajectories, tool calls, evidence grounding, security permissions, context, and robustness; failed critical assertions block release, while audit, evaluation, task, and memory records persist in tenant-scoped SQLite WAL storage.
- An internal audit team needs one project workspace to manage scope, control testing, evidence requests, findings, and remediation status.
- An audit lead needs to inspect the page or section sources behind a conclusion and identify evidence gaps or contradictions.
- A platform team needs pre-release regression evaluation across task outcomes, trajectories, tools, security, and robustness, with critical assertions acting as release gates.
- A multi-tenant audit environment needs RBAC, project-membership checks, and traceable tool use at HTTP and Skill boundaries.
- A delivery team needs high-risk or low-confidence outputs to collect more evidence and receive human review before remediation and delivery packaging.
What are this agent's strengths and limitations?
- Combines the audit workbench, dependency-aware runtime, evidence retrieval, human review, remediation, and delivery packaging in one traceable workflow.
- Its Evaluation Harness assesses more than final outputs: it covers trajectories, tools, evidence, security, context, and robustness, with critical assertions able to block releases.
- Hybrid RAG offers source locations, contradictory-evidence detection, missing-evidence prompts, and fail-closed filtering without requiring a local embedding model.
- It documents concrete authorization and storage controls: RBAC at HTTP and Skill boundaries, project ABAC, tenant isolation, signed checkpoints, and SQLite WAL optimistic concurrency.
- The default persistence layer is single-node SQLite WAL, and RAG chunks remain a single-node snapshot/index, so it should not be assumed to be a multi-node production architecture.
- The built-in evaluation suite is a regression and smoke baseline; independent held-out data and domain-expert review are still needed for business-accuracy or release-grade claims.
- Its bearer-token RBAC is only a deployable baseline; enterprise OIDC/SSO, external KMS, object storage, and multi-node databases require additional integration.
- Local semantic embeddings are not installed by default; enabling them requires requirements-embeddings.txt and RAG_ENABLE_EMBEDDINGS=1.
How do you install or deploy this agent?
Python 3.10 or later is required. On macOS/Linux:
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp config.env.example config.env
python start.pyOn Windows PowerShell:
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
Copy-Item config.env.example config.env
python start.pyOpen the address printed by the terminal. Without a model key, the application runs in deterministic fallback mode. For LLM enhancement, set DEEPSEEK_API_KEY or a compatible-provider variable in the local config.env. Shared or network deployments require SECURITY_MODE=enforced, AUDITPILOT_API_TOKENS_JSON, and AUDIT_LOG_SIGNING_KEY.
How do you use this agent?
After starting the server, open the address printed in the terminal. In Agent Collaboration, enter an audit subject, standard, or risk scenario to observe planning, evidence, control, risk, and remediation collaboration. In Audit Projects, define scope and run control tests, Deep Research, evidence requests, and the remediation loop. Maintain searchable audit knowledge in Knowledge Base, run layered assessments and inspect release gates in Agent Evaluation, and review tool schemas, permission declarations, call records, and runtime status in Skills / MCP.