A agent review method that does not confuse popularity with quality
FARS records observable facts, reproducible tests and editorial judgment separately. The total helps rank; dimensions explain why; confidence tells you how stable the conclusion is.
Five governing principles
Evidence first
Stars, unverified reputation and polished docs are leads. Verified enterprise origin is governance evidence, not a substitute for inspecting behavior, permissions and task results.
Risk is not averaged away
Malice, covert exfiltration and destructive defaults cannot be cancelled out by high scores elsewhere, and Trust must clear its own floor — weak safety can't be masked by strengths elsewhere.
Claims never exceed evidence
Recommendation strength is bounded by evidence confidence: static or limited review reaches at most “Use with care”; “Recommended” needs partial reproduction; “Strongly recommended” needs full, cross-checked reproduction.
Unknown stays unknown
When material or testing is insufficient we say so; missing information is never assumed safe or turned into fake precision.
Scores expire
A score belongs to a reviewed version and date; material changes to the repo, dependencies or platform trigger review.
How official enterprise publication affects a score
Verified organization accounts for publishers such as OpenAI, Anthropic, Microsoft and Google strengthen provenance, ownership and update-path evidence only — inside the Trust and Convention dimensions. There is no fixed brand bonus. Official origin cannot prove reliability, safety or utility and never overrides a red line; independent or unverified publishers are not automatically penalized.
Six dimensions (TRACE + V), 100 points
The six dimensions form one chain: Trust answers "can I use it safely", Reliability "does it work consistently", Adaptability "should it run in this scenario", Convention "can it be understood and reused", Effectiveness "is the problem actually solved", and Verifiability "how well are those conclusions supported". Each uses zero, midpoint and full-score anchors; reviewers must record why points were withheld. Trust & safety carries the largest weight.
01T · Trust & safety25
Least privilege, user confirmation, data-flow transparency, sensitive-data handling, dependency security, external effects, rollback, and verifiable attribution.
- 0: unresolved red-line risk, concealment or gross overreach
- 12: risks visible; permissions, confirmation, isolation or recovery incomplete
- 25: least privilege, explicit confirmation, rollback, data-flow disclosure and attribution complete
02R · Reliability20
Instructions, scripts and dependencies agree; key paths reproduce; failures explain themselves instead of shifting debugging cost to the user.
- 0: broken, contradictory or unrunnable as claimed
- 10: happy path works; tests, edge cases, error handling or failure feedback thin
- 20: key paths reproduced with clear, diagnosable, controlled failure modes
03A · Adaptability & triggering15
Clear audience and scenarios, declared capability boundaries, semantic trigger precision for agent invocation, and environment fit including Chinese-language and network reachability.
- 0: scope vague, mismatched with actual capability, or unusable in the target environment
- 7: clear scenario; boundary, trigger or environment-fit evidence limited
- 15: inputs, expected outputs, non-fit boundaries and trigger conditions all clear
04C · Convention & maintainability15
Information architecture and progressive disclosure, install and dependency notes, naming stability, examples, limitation disclosure, license, versioning, changelog and update path.
- 0: name/description contradict capability, or unclear provenance/license, or abandoned
- 7: readable and usable with notable hidden assumptions or governance gaps
- 15: layered docs, sufficient examples and limits; ownership, license, versions and changes clear
05E · Effectiveness15
Result correctness and completeness, directly usable output in the expected format, marginal value over alternatives, and cost/benefit proportionality.
- 0: value claim unverifiable, or output needs heavy rework with near-zero marginal value
- 7: completes the core task; completeness, direct usability or comparative benefit limited
- 15: goals, benefits and alternatives clear; representative outputs verified directly usable
06V · Verifiability10
Traceability of key claims, test reproducibility, cross-source corroboration and fact/inference separation; every key conclusion should be traceable to evidence links.
- 0: author claims, marketing or unverifiable demos only
- 5: auditable primary material or limited reproduction; coverage thin
- 10: multiple evidence types corroborate independently reproducible conclusions
How to read the total
Strongly recommended
Good evidence and no uncontrolled major risk; still check your environment and data sensitivity.
Recommended
Generally reliable with disclosed limitations; trial as directed and keep a rollback path.
Use with care
Useful, but reliability, evidence or controls still have material gaps.
Not recommended
Current benefit does not outweigh risk or uncertainty.
The ranges above are only a first pass on the score. The final tier must also clear two gates: (1) an evidence ceiling — “low” confidence tops out at Use with care, “medium” at Recommended, and only “high” can reach Strongly recommended; and (2) a safety floor — Recommended requires Trust & safety ≥15/25 and Strongly recommended ≥20/25. Missing either lowers the tier, with the reason shown on the detail page. The numeric total is unchanged; only the outward recommendation is constrained.
Why the score also has an evidence-confidence label
Two agents scoring 82 should not be trusted equally if one was reproduced end-to-end and the other only read on paper. Confidence describes how much this review covered (distinct from the V · Verifiability dimension, which rates whether the agent itself is checkable); it never raises the total, but it caps the recommendation tier.
Agent and key scripts inspected, key paths reproduced, with at least one independent or cross-source form of evidence.
Primary material inspected and partly reproduced; platform, edge-case or third-party coverage remains incomplete.
Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
Evidence ladder
We prefer evidence closer to actual behavior and easier to reproduce independently. Corroboration across evidence types is stronger than one source.
Reproducible behavior
Task runs, safety tests, failure and rollback checks in an isolated environment.
Auditable primary material
AGENTS.md, scripts, dependency manifests, commits, releases and licenses.
Independent usage evidence
Verifiable issue reports, third-party reproduction and maintainer responses.
Weak signals
README claims, demos, stars, downloads and social discussion—context only.
Red lines and override rules
Any unresolved item below sets the status to Not recommended / Blocked regardless of the numeric total:
- Malware, credential theft, covert tracking or undisclosed data exfiltration;
- Irreversible or high-impact default actions without clear consent, scoping and recovery;
- Fabricated provenance, test results, compatibility or safety claims;
- Known critical vulnerable dependencies without mitigation, or permissions grossly disproportionate to the task.
How one review is completed
Frame the claim
Identify intended users, core job, claimed platforms, I/O and permission boundaries.
Collect evidence
Pin a repo version; inspect the agent, scripts and dependencies; record links and unknowns.
Screen risk first
Review data flows, external writes, command execution, secrets, permissions and red lines.
Reproduce tasks
Run a representative happy path and one failure case in isolation; verify exit or rollback.
Score independently
Apply anchors dimension by dimension, record deductions, then calculate total and confidence.
Publish and revisit
A second reviewer checks safety, red lines and confidence; then publish version, date, dimensions and risks.
What the score does not mean
FARS is not a security certification, academic peer review or performance guarantee. It is a structured judgment of evidence for a specific version and test scope. A high score does not fit every user; a personal low-risk choice may differ from an enterprise production decision. Before adopting, consider permissions, data sensitivity, cost and rollback in your own context.
How the standard evolves
The version changes when dimensions, weights or override rules materially change. Editorial clarifications are minor revisions. Historical versions remain available so old scores stay interpretable; scores on an older standard are marked and queued for review.