ReLE Chinese LLM Benchmark & Defect Library
Continuously updated Chinese LLM evaluation across 7 domains, 300+ dimensions, with leaderboards and a defect library of over 2 million cases.
- Source repo
- jeinlee1991/chinese-llm-benchmark
- Stars
- ★ 6.5k
- Last updated
- 4d ago
- FA score
- 14/100 · Major gaps
At a glance
- How it runs
- Works with
- Platform-specific
- Cost
- Free tier plus a paid hosted plan
- Setup effort
- Low · running in minutes
- You'll need
- Typical use
- Model selection: Compare models on comprehensive, education, or medical leaderboards to choose the right one for a specific domain.
- Not a fit if
- Teams that need fully offline or self-hosted evaluation
- Teams that require open-source eval code and fully transparent methodology
- Users unwilling to depend on nonelinear.com for results and gateway
- Source review
- 14/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
ReLE (Really Reliable Live Evaluation for LLM) is a Chinese LLM evaluation project maintained by NoneLinear, formerly known as CLiB. It has evaluated 395 models including GPT, Gemini, Claude, ERNIE, Qwen, DeepSeek and other commercial/open-source models, providing leaderboards across 7 major domains (comprehensive, reasoning, education, medical, finance, legal, agent tool-calling). The project also offers a defect library of over 2 million failure cases for analysis. Additionally, it provides a unified gateway for multi-model access with load balancing and failover, and a model selection tool claiming to reduce costs by up to 90%. Free evaluation for private models is available upon contact.
ReLE runs an end-to-end evaluation pipeline: 1) aggregates accuracy, latency, token usage across 300+ dimensions for each model from leaderboard files; 2) exposes badcase links for each dimension to analyze failures; 3) provides online evaluation services allowing custom leaderboard filtering and data upload; 4) integrates a unified API gateway (api.nonelinear.com/v1) with OpenAI-compatible SDK, enabling load balancing across providers; 5) offers a model selection tool that compares models on user-specific test data to optimize cost-performance. It also merges LMArena and Artificial Analysis scores for cross-language comparison.
- Model selection: Compare models on comprehensive, education, or medical leaderboards to choose the right one for a specific domain.
- Model improvement: Use the 2M+ defect library to identify weaknesses and guide fine-tuning.
- Research: Reference the ReLE technical report for studying capability anisotropy in Chinese LLMs.
- Enterprise integration: Utilize the unified gateway for multi-model access with high availability and auto-failover.
- Cost reduction: Upload proprietary test data to the selection tool to find the most cost-effective model, potentially saving 90%.
How do you install or deploy this agent?
No installation needed for the benchmark data. To use the API gateway, sign up at nonelinear.com, get an API key, and use the OpenAI SDK with the custom base URL.
How do you use this agent?
1) Browse leaderboards in the repo, e.g., leaderboard/overall.md; 2) View badcases via links; 3) Use the online tool at nonelinear.com/static/benchmarking.html to filter or upload data; 4) Call the API: set base_url to https://api.nonelinear.com/v1, authenticate with your key, and invoke models; 5) Use the model selection tool to get recommendations based on your data.
What are this agent's strengths and limitations?
- Large model coverage (395+) with weekly updates
- Fine-grained dimensions: 300+ sub-domains including professional exams
- Defect library of 2M+ cases for deep analysis
- Online evaluation and model selection tools with potential 90% cost reduction
- Integration with international leaderboards (LMArena, AA) for comparison
- Commercial dependencies: results and gateway tied to nonelinear.com
- Evaluation methodology not fully transparent; no open-source evaluation code
- Some domains are TODO (e.g., 中考, 初中奥数) indicating incomplete coverage
- API and selection services likely require payment; pricing unclear
- Large data but may be restricted for community use without permission
How does this agent compare with similar options?
Compared to LM Arena, ReLE focuses on Chinese capability and professional domains (e.g., medical exams) with finer granularity. Unlike OpenCompass, ReLE emphasizes commercial services and defect library. Similar to SuperCLUE, ReLE is a Chinese benchmark, but ReLE covers more models and dimensions and includes a unified gateway.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| ReLE Chinese LLM Benchmark & Defect Library This agent | 14 · Major gaps | Web appFreemium | ★ 6.5k | 4d ago | — | — |
| Agent Memory Benchmark (AMB) | 39 · Major gaps | CLIFree + model costs | ★ 153 | 5d ago | Python | — |
| MobileGym | 50 · Major gaps | CLIFree + model costs | ★ 827 | 9d ago | Python | — |
| Potato Annotation Platform | 59 · Major gaps | CLIFree | ★ 436 | 3d ago | Python | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows the repository only provides evaluation data and leaderboards, with no code or executable components. Therefore, there are no mechanisms for least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All trust criteria score 0 because no evidence of such mechanisms exists.
Self-consistency: The repository's internal structure is consistent; leaderboard files listed in README match the directory structure, but no tests or validation data are provided, so score 1. Dependency availability: No dependencies mentioned, score 0. Failure messages: No error handling or failure messages provided, score 0.
Audience and scenarios: Clearly targets Chinese AI model evaluation across multiple educational subjects, score 2. Capability boundaries: Only describes evaluation scope, not model capability boundaries, score 1. Trigger precision: No trigger mechanisms provided, score 0. Environment fit: No runtime environment or platform specified, score 0.
Information architecture: Clear directory structure with categories and links, score 2. Install notes: None, score 0. Naming stability: Leaderboard file naming is consistent, but no version stability mentioned, score 1. Examples and FAQ: Provides example images and links, but no FAQ, score 1. Known limitations: Not mentioned, score 0. License: Not provided, score 0. Versioning and changelog: None, score 0. Maintenance responsibility: Not stated, score 0.
Output usability: Provides leaderboards and defect library, but format not specified, score 1. Marginal value: Provides large-scale defect library and leaderboards, valuable for community, score 2. Cost-benefit: No cost or benefit analysis provided, score 0.
Claim traceability: Leaderboard data has links, but no raw data or methodology, score 1. Cross-source corroboration: No other sources for verification, score 0. Fact-inference separation: Does not distinguish facts from inferences, score 0.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- The repository only provides evaluation data and leaderboards, with no code or executable components, making it impossible to verify the accuracy and fairness of the evaluation methodology.
- No license information is provided, posing legal risks for use and distribution.
- No data sources or evaluation methodology are provided, limiting the credibility of the leaderboards.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.