iFixAi Agent Auditor
Audit AI-agent governance failures and operational risks in about 120 seconds.
Per-dimension scores and reasoning
Evidence shows: project explicitly states keys are read from environment variables and not persisted; provides redaction mechanism (scrub_secrets) and telemetry disclosure; user confirmation mechanisms (e.g., --dry-run, cost display) exist. Deductions: no explicit least-privilege principle, external effects (e.g., network calls) not fully documented, rollback mechanism not mentioned, source attribution relies on unverified publisher.
Evidence shows: documentation and code structure consistent, failure message examples provided (e.g., mock run intentionally fails), CI configuration exists. Deductions: dependency availability not fully verified (e.g., external LLM providers), comprehensiveness of failure messages not confirmed.
Evidence shows: clear target audience (developers, AI agents), multiple execution modes (CLI, plugin, Skill), support for multiple environments (Claude Code, Codex, etc.), clear trigger methods (commands, plugins). Deductions: capability boundaries not detailed enough, environment fit details limited.
Evidence shows: clear documentation structure (docs/), detailed installation notes, clear license (Apache-2.0), examples and FAQ exist, known limitations mentioned (e.g., not a certification tool). Deductions: naming stability not explicit, version changelog not provided, maintenance responsibility not clear (publisher unverified).
Evidence shows: clear output formats (JSON+Markdown), scorecards and reports provided, high marginal value (diagnosing AI misalignment), cost-benefit estimates (cost tables). Deductions: actual effectiveness not verified (static review), cost estimates based on assumptions.
Evidence shows: claims supported by documentation (e.g., docs/scoring.md), case studies provided, facts and inferences separated (e.g., case studies reconstructed from public info). Deductions: cross-source corroboration insufficient, some claims lack independent evidence.
- Publisher identity is unverified; treat source with caution.
- Static review cannot verify actual runtime behavior; recommend hands-on testing.
- Relies on external LLM providers; be aware of their availability and security.
- Telemetry is disclosed, but confirm it meets your privacy requirements.
What does this agent do, and when should you use it?
iFixAi is a black-box auditing tool for AI agents, delivered through a CLI, Claude Code/Codex plugins, and scaffolded agent skills. It can test bare model providers, OpenAI-compatible HTTP endpoints, or other agents connected through ChatProvider.send_message. A run executes smoke, strategic, core, extended, or all suites and produces a terminal scorecard plus JSON and Markdown reports. Its A–F grade is based on five core pillars—Fabrication, Manipulation, Deception, Unpredictability, and Opacity—while additional risk categories are reported separately. It fits teams that need repeatable checks for issues such as prompt injection, privilege escalation, unsupported claims, instruction drift, and weak human escalation controls.
ifixai setup writes an ifixai.yaml configuration, while ifixai run can also be driven entirely with explicit flags. A run calls the selected SUT (system under test) provider, an OpenAI-compatible HTTP endpoint, or a custom ChatProvider.send_message implementation; a judge evaluates the responses, normally from a provider different from the SUT. The tool executes the selected --suite, scores observable governance behavior, and labels unobservable capabilities as insufficient_evidence. It writes JSON and Markdown reports to ./ifixai-results/ and renders a rich terminal scorecard; ifixai run --print-telemetry shows the telemetry fields it would send.
- A security team needs to black-box audit a deployed customer-support agent with an OpenAI-compatible endpoint, including the guardrails already active in production.
- A platform engineering team wants explicit CLI flags for recurring strategic or core-suite checks in CI or scripted audit batches.
- A developer integrating a provider for the first time wants to validate the local pipeline with ifixai run --provider mock --api-key not-used --eval-mode self.
- A team using Claude Code or Codex wants the plugin to discover configuration, build a fixture, estimate cost, run the diagnostic, and explain the scorecard.
- A developer with a nonstandard agent runtime can implement ChatProvider.send_message and optionally expose list_tools, get_audit_trail, or retrieve_sources for broader inspection coverage.
What are this agent's strengths and limitations?
- One diagnostic engine is available through guided CLI, fully scriptable CLI, Codex/Claude Code plugins, and scaffolded skills for several agent environments.
- It can target an OpenAI-compatible HTTP endpoint directly and offers an integration seam through ChatProvider.send_message for other runtimes.
- Core grading has documented pillar weights, A–F thresholds, and mandatory-minimum rules for B01, B08, and P01.
- Results are produced as JSON, Markdown, and a terminal scorecard, supporting both automated handling and human review.
- A citable independent grade requires keys from at least two different vendors; a single-provider setup must use self-judging.
- The default fixture intentionally contains seeded defects, so a run without your own --fixture is not a score for your actual agent.
- Non-HTTP agents require a ChatProvider.send_message implementation, and missing adapter capabilities result in insufficient_evidence.
- Full mode requires a hand-built fixture, and the README estimates roughly 2,000 judge calls for a full run, so cost depends on the selected models.
How do you install or deploy this agent?
Python 3.10+ is required. For example, to test Anthropic, run: pip install "ifixai[anthropic]". For guided configuration, run ifixai setup and select the provider, model, judge, and suite; it writes ifixai.yaml. To install the Codex plugin, run codex plugin marketplace add ifixai-ai/iFixAi, then codex plugin add ifixai@ifixai-ai. For the skill route, run uvx ifixai install --agents codex; that route also requires uv.
How do you use this agent?
Run ifixai setup once, then run ifixai run; reports are written to ./ifixai-results/. For a no-key pipeline check, run: ifixai run --provider mock --api-key not-used --eval-mode self. To audit a deployed compatible endpoint, run: ifixai run --provider http --endpoint <agent-url> --grounding sut. A citable grade requires an SUT key and a judge key from a different vendor in the environment; use --eval-mode self when only self-judging is available.