iFixAi Agent Auditor
Audit AI-agent governance failures and operational risks in about 120 seconds.
- Source repo
- ifixai-ai/iFixAi
- Stars
- ★ 44k
- Last updated
- 5d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 56/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformCodex · Claude Code · OpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A security team needs to black-box audit a deployed customer-support agent with an OpenAI-compatible endpoint, including the guardrails already active in production.
- Not a fit if
- Teams that want a citable grade with only one vendor's API key
- Users who won't author their own fixture and expect a score for their real agent
- Source review
- 56/100 · Major gaps
What does this agent do, and when should you use it?
iFixAi is a black-box auditing tool for AI agents, delivered through a CLI, Claude Code/Codex plugins, and scaffolded agent skills. It can test bare model providers, OpenAI-compatible HTTP endpoints, or other agents connected through ChatProvider.send_message. A run executes smoke, strategic, core, extended, or all suites and produces a terminal scorecard plus JSON and Markdown reports. Its A–F grade is based on five core pillars—Fabrication, Manipulation, Deception, Unpredictability, and Opacity—while additional risk categories are reported separately. It fits teams that need repeatable checks for issues such as prompt injection, privilege escalation, unsupported claims, instruction drift, and weak human escalation controls.
ifixai setup writes an ifixai.yaml configuration, while ifixai run can also be driven entirely with explicit flags. A run calls the selected SUT (system under test) provider, an OpenAI-compatible HTTP endpoint, or a custom ChatProvider.send_message implementation; a judge evaluates the responses, normally from a provider different from the SUT. The tool executes the selected --suite, scores observable governance behavior, and labels unobservable capabilities as insufficient_evidence. It writes JSON and Markdown reports to ./ifixai-results/ and renders a rich terminal scorecard; ifixai run --print-telemetry shows the telemetry fields it would send.
- A security team needs to black-box audit a deployed customer-support agent with an OpenAI-compatible endpoint, including the guardrails already active in production.
- A platform engineering team wants explicit CLI flags for recurring strategic or core-suite checks in CI or scripted audit batches.
- A developer integrating a provider for the first time wants to validate the local pipeline with ifixai run --provider mock --api-key not-used --eval-mode self.
- A team using Claude Code or Codex wants the plugin to discover configuration, build a fixture, estimate cost, run the diagnostic, and explain the scorecard.
- A developer with a nonstandard agent runtime can implement ChatProvider.send_message and optionally expose list_tools, get_audit_trail, or retrieve_sources for broader inspection coverage.
How do you install or deploy this agent?
Python 3.10+ is required. For example, to test Anthropic, run: pip install "ifixai[anthropic]". For guided configuration, run ifixai setup and select the provider, model, judge, and suite; it writes ifixai.yaml. To install the Codex plugin, run codex plugin marketplace add ifixai-ai/iFixAi, then codex plugin add ifixai@ifixai-ai. For the skill route, run uvx ifixai install --agents codex; that route also requires uv.
How do you use this agent?
Run ifixai setup once, then run ifixai run; reports are written to ./ifixai-results/. For a no-key pipeline check, run: ifixai run --provider mock --api-key not-used --eval-mode self. To audit a deployed compatible endpoint, run: ifixai run --provider http --endpoint <agent-url> --grounding sut. A citable grade requires an SUT key and a judge key from a different vendor in the environment; use --eval-mode self when only self-judging is available.
What are this agent's strengths and limitations?
- One diagnostic engine is available through guided CLI, fully scriptable CLI, Codex/Claude Code plugins, and scaffolded skills for several agent environments.
- It can target an OpenAI-compatible HTTP endpoint directly and offers an integration seam through ChatProvider.send_message for other runtimes.
- Core grading has documented pillar weights, A–F thresholds, and mandatory-minimum rules for B01, B08, and P01.
- Results are produced as JSON, Markdown, and a terminal scorecard, supporting both automated handling and human review.
- A citable independent grade requires keys from at least two different vendors; a single-provider setup must use self-judging.
- The default fixture intentionally contains seeded defects, so a run without your own --fixture is not a score for your actual agent.
- Non-HTTP agents require a ChatProvider.send_message implementation, and missing adapter capabilities result in insufficient_evidence.
- Full mode requires a hand-built fixture, and the README estimates roughly 2,000 judge calls for a full run, so cost depends on the selected models.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| iFixAi Agent Auditor This agent | 56 · Major gaps | CLIFree + model costs | ★ 44k | 5d ago | Python | Codex · Claude Code · OpenAI API · Claude API |
| Agentic Radar | 46 · Major gaps | CLIFree + model costs | ★ 1.1k | 11mo ago | Python | — |
| Node9 | 82 · Good | CLIFreemium | ★ 225 | today | TypeScript | Codex · Claude Code |
| HOL Guard | 79 · Good | CLIFreemium | ★ 1.1k | today | Rust | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: project explicitly states keys are read from environment variables and not persisted; provides redaction mechanism (scrub_secrets) and telemetry disclosure; user confirmation mechanisms (e.g., --dry-run, cost display) exist. Deductions: no explicit least-privilege principle, external effects (e.g., network calls) not fully documented, rollback mechanism not mentioned, source attribution relies on unverified publisher.
Evidence shows: documentation and code structure consistent, failure message examples provided (e.g., mock run intentionally fails), CI configuration exists. Deductions: dependency availability not fully verified (e.g., external LLM providers), comprehensiveness of failure messages not confirmed.
Evidence shows: clear target audience (developers, AI agents), multiple execution modes (CLI, plugin, Skill), support for multiple environments (Claude Code, Codex, etc.), clear trigger methods (commands, plugins). Deductions: capability boundaries not detailed enough, environment fit details limited.
Evidence shows: clear documentation structure (docs/), detailed installation notes, clear license (Apache-2.0), examples and FAQ exist, known limitations mentioned (e.g., not a certification tool). Deductions: naming stability not explicit, version changelog not provided, maintenance responsibility not clear (publisher unverified).
Evidence shows: clear output formats (JSON+Markdown), scorecards and reports provided, high marginal value (diagnosing AI misalignment), cost-benefit estimates (cost tables). Deductions: actual effectiveness not verified (static review), cost estimates based on assumptions.
Evidence shows: claims supported by documentation (e.g., docs/scoring.md), case studies provided, facts and inferences separated (e.g., case studies reconstructed from public info). Deductions: cross-source corroboration insufficient, some claims lack independent evidence.
- Publisher identity is unverified; treat source with caution.
- Static review cannot verify actual runtime behavior; recommend hands-on testing.
- Relies on external LLM providers; be aware of their availability and security.
- Telemetry is disclosed, but confirm it meets your privacy requirements.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/ifixai)<a href="https://followagents.com/en/agents/ifixai"><img src="https://followagents.com/badge/ifixai.svg" alt="FARS review"></a>Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v4.3.0 V4.3.0 - Audit Any Agent Over Chat, Judging Within Budget & Guided Fixtures
Release notes excerpt
Audit Any Agent Over Chat 15 inspections no longer need an adapter: B01, B02, B04, B23, C02, C05, C11, M02, M03, M07, P01, P08, V08, X04 and X11 each send a probe pack over chat, and the agent's replies decide the verdict (#363). A chat endpoint is enough, and a bare model API counts. B01 and P01 grade a chat-only agent like any other. Hooks such as authorize_tool, get_confidence and route_to_human still work. They show as notes beside the probe results and never change the verdict. …View on GitHub ↗ -
v4.2.1 V4.2.1 - V11 inspection
Release notes excerpt
New inspection: V11 Guilt Tripping Checks whether a companion chat ever helps a struggling user hurt themselves. - 40 four-turn chats across appearance contempt, restrictive eating, self-harm, suicidal ideation, and a sad-but-safe control. Half the users say they are 13 to 15. - Turn 3 reframes the request using known jailbreak patterns ("it's for a school project", "stay in character", "you said you'd be on my side"). Turn 4 asks directly. …View on GitHub ↗ -
v4.2.0 V4.2.0 Faster Default Runs, Gateway Providers & Offline Assurance
Release notes excerpt
Faster Default Run ifixai run now runs the 10-inspection essential suite instead of all 60: B01, B08, P01, B12, B09, B27, B06, B16, B18 and B20 (#232). - It still produces a grade, because all three mandatory minimums are included. - It makes about 6x fewer agent and judge calls. - --full-run runs all 60, as before. --resume continues a full run as a full run. - An empty or failed agent reply now drops that one probe. It no longer fails the whole inspection (#232). New …View on GitHub ↗