ReLE Chinese LLM Benchmark & Defect Library
Continuously updated Chinese LLM evaluation across 7 domains, 300+ dimensions, with leaderboards and a defect library of over 2 million cases.
Evidence shows the repository only provides evaluation data and leaderboards, with no code or executable components. Therefore, there are no mechanisms for least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All trust criteria score 0 because no evidence of such mechanisms exists.
Self-consistency: The repository's internal structure is consistent; leaderboard files listed in README match the directory structure, but no tests or validation data are provided, so score 1. Dependency availability: No dependencies mentioned, score 0. Failure messages: No error handling or failure messages provided, score 0.
Audience and scenarios: Clearly targets Chinese AI model evaluation across multiple educational subjects, score 2. Capability boundaries: Only describes evaluation scope, not model capability boundaries, score 1. Trigger precision: No trigger mechanisms provided, score 0. Environment fit: No runtime environment or platform specified, score 0.
Information architecture: Clear directory structure with categories and links, score 2. Install notes: None, score 0. Naming stability: Leaderboard file naming is consistent, but no version stability mentioned, score 1. Examples and FAQ: Provides example images and links, but no FAQ, score 1. Known limitations: Not mentioned, score 0. License: Not provided, score 0. Versioning and changelog: None, score 0. Maintenance responsibility: Not stated, score 0.
Output usability: Provides leaderboards and defect library, but format not specified, score 1. Marginal value: Provides large-scale defect library and leaderboards, valuable for community, score 2. Cost-benefit: No cost or benefit analysis provided, score 0.
Claim traceability: Leaderboard data has links, but no raw data or methodology, score 1. Cross-source corroboration: No other sources for verification, score 0. Fact-inference separation: Does not distinguish facts from inferences, score 0.
- The repository only provides evaluation data and leaderboards, with no code or executable components, making it impossible to verify the accuracy and fairness of the evaluation methodology.
- No license information is provided, posing legal risks for use and distribution.
- No data sources or evaluation methodology are provided, limiting the credibility of the leaderboards.
What does this agent do, and when should you use it?
ReLE (Really Reliable Live Evaluation for LLM) is a Chinese LLM evaluation project maintained by NoneLinear, formerly known as CLiB. It has evaluated 395 models including GPT, Gemini, Claude, ERNIE, Qwen, DeepSeek and other commercial/open-source models, providing leaderboards across 7 major domains (comprehensive, reasoning, education, medical, finance, legal, agent tool-calling). The project also offers a defect library of over 2 million failure cases for analysis. Additionally, it provides a unified gateway for multi-model access with load balancing and failover, and a model selection tool claiming to reduce costs by up to 90%. Free evaluation for private models is available upon contact.
ReLE runs an end-to-end evaluation pipeline: 1) aggregates accuracy, latency, token usage across 300+ dimensions for each model from leaderboard files; 2) exposes badcase links for each dimension to analyze failures; 3) provides online evaluation services allowing custom leaderboard filtering and data upload; 4) integrates a unified API gateway (api.nonelinear.com/v1) with OpenAI-compatible SDK, enabling load balancing across providers; 5) offers a model selection tool that compares models on user-specific test data to optimize cost-performance. It also merges LMArena and Artificial Analysis scores for cross-language comparison.
- Model selection: Compare models on comprehensive, education, or medical leaderboards to choose the right one for a specific domain.
- Model improvement: Use the 2M+ defect library to identify weaknesses and guide fine-tuning.
- Research: Reference the ReLE technical report for studying capability anisotropy in Chinese LLMs.
- Enterprise integration: Utilize the unified gateway for multi-model access with high availability and auto-failover.
- Cost reduction: Upload proprietary test data to the selection tool to find the most cost-effective model, potentially saving 90%.
What are this agent's strengths and limitations?
- Large model coverage (395+) with weekly updates
- Fine-grained dimensions: 300+ sub-domains including professional exams
- Defect library of 2M+ cases for deep analysis
- Online evaluation and model selection tools with potential 90% cost reduction
- Integration with international leaderboards (LMArena, AA) for comparison
- Commercial dependencies: results and gateway tied to nonelinear.com
- Evaluation methodology not fully transparent; no open-source evaluation code
- Some domains are TODO (e.g., 中考, 初中奥数) indicating incomplete coverage
- API and selection services likely require payment; pricing unclear
- Large data but may be restricted for community use without permission
How do you install or deploy this agent?
No installation needed for the benchmark data. To use the API gateway, sign up at nonelinear.com, get an API key, and use the OpenAI SDK with the custom base URL.
How do you use this agent?
1) Browse leaderboards in the repo, e.g., leaderboard/overall.md; 2) View badcases via links; 3) Use the online tool at nonelinear.com/static/benchmarking.html to filter or upload data; 4) Call the API: set base_url to https://api.nonelinear.com/v1, authenticate with your key, and invoke models; 5) Use the model selection tool to get recommendations based on your data.
How does this agent compare with similar options?
Compared to LM Arena, ReLE focuses on Chinese capability and professional domains (e.g., medical exams) with finer granularity. Unlike OpenCompass, ReLE emphasizes commercial services and defect library. Similar to SuperCLUE, ReLE is a Chinese benchmark, but ReLE covers more models and dimensions and includes a unified gateway.