iFixAi Agent Auditor
Audit AI-agent governance failures and operational risks in about 120 seconds.
What does this agent do, and when should you use it?
iFixAi is a black-box auditing tool for AI agents, delivered through a CLI, Claude Code/Codex plugins, and scaffolded agent skills. It can test bare model providers, OpenAI-compatible HTTP endpoints, or other agents connected through ChatProvider.send_message. A run executes smoke, strategic, core, extended, or all suites and produces a terminal scorecard plus JSON and Markdown reports. Its A–F grade is based on five core pillars—Fabrication, Manipulation, Deception, Unpredictability, and Opacity—while additional risk categories are reported separately. It fits teams that need repeatable checks for issues such as prompt injection, privilege escalation, unsupported claims, instruction drift, and weak human escalation controls.
ifixai setup writes an ifixai.yaml configuration, while ifixai run can also be driven entirely with explicit flags. A run calls the selected SUT (system under test) provider, an OpenAI-compatible HTTP endpoint, or a custom ChatProvider.send_message implementation; a judge evaluates the responses, normally from a provider different from the SUT. The tool executes the selected --suite, scores observable governance behavior, and labels unobservable capabilities as insufficient_evidence. It writes JSON and Markdown reports to ./ifixai-results/ and renders a rich terminal scorecard; ifixai run --print-telemetry shows the telemetry fields it would send.
- A security team needs to black-box audit a deployed customer-support agent with an OpenAI-compatible endpoint, including the guardrails already active in production.
- A platform engineering team wants explicit CLI flags for recurring strategic or core-suite checks in CI or scripted audit batches.
- A developer integrating a provider for the first time wants to validate the local pipeline with ifixai run --provider mock --api-key not-used --eval-mode self.
- A team using Claude Code or Codex wants the plugin to discover configuration, build a fixture, estimate cost, run the diagnostic, and explain the scorecard.
- A developer with a nonstandard agent runtime can implement ChatProvider.send_message and optionally expose list_tools, get_audit_trail, or retrieve_sources for broader inspection coverage.
What are this agent's strengths and limitations?
- One diagnostic engine is available through guided CLI, fully scriptable CLI, Codex/Claude Code plugins, and scaffolded skills for several agent environments.
- It can target an OpenAI-compatible HTTP endpoint directly and offers an integration seam through ChatProvider.send_message for other runtimes.
- Core grading has documented pillar weights, A–F thresholds, and mandatory-minimum rules for B01, B08, and P01.
- Results are produced as JSON, Markdown, and a terminal scorecard, supporting both automated handling and human review.
- A citable independent grade requires keys from at least two different vendors; a single-provider setup must use self-judging.
- The default fixture intentionally contains seeded defects, so a run without your own --fixture is not a score for your actual agent.
- Non-HTTP agents require a ChatProvider.send_message implementation, and missing adapter capabilities result in insufficient_evidence.
- Full mode requires a hand-built fixture, and the README estimates roughly 2,000 judge calls for a full run, so cost depends on the selected models.
How do you install or deploy this agent?
Python 3.10+ is required. For example, to test Anthropic, run: pip install "ifixai[anthropic]". For guided configuration, run ifixai setup and select the provider, model, judge, and suite; it writes ifixai.yaml. To install the Codex plugin, run codex plugin marketplace add ifixai-ai/iFixAi, then codex plugin add ifixai@ifixai-ai. For the skill route, run uvx ifixai install --agents codex; that route also requires uv.
How do you use this agent?
Run ifixai setup once, then run ifixai run; reports are written to ./ifixai-results/. For a no-key pipeline check, run: ifixai run --provider mock --api-key not-used --eval-mode self. To audit a deployed compatible endpoint, run: ifixai run --provider http --endpoint <agent-url> --grounding sut. A citable grade requires an SUT key and a judge key from a different vendor in the environment; use --eval-mode self when only self-judging is available.