GenoTEX Gene Expression Benchmark
An expert-curated benchmark for evaluating automated gene-expression and gene–trait association analysis workflows.
- Source repo
- Liu-Hy/GenoTEX
- Stars
- ★ 65
- Last updated
- 4mo ago
- License
- NOASSERTION
- Primary language
- Jupyter Notebook
- FA score
- 36/100 · Major gaps
At a glance
- Works with
- Universal · cross-platform
- You'll need
- Typical use
- An agent-development team that needs a common GTA benchmark to measure dataset filtering, preprocessing, and significant-gene identification.
- Main limitation
- The documented download total is 82.0 GB, so full use requires substantial local disk space; the README also states 41.5 GB of input data, leaving a size-definition difference to resolve.
- Source review
- 36/100 · Major gaps 7 safety controls not found
What does this agent do, and when should you use it?
GenoTEX is a benchmark dataset for automated gene-expression analysis, with particular emphasis on evaluating LLM-based agents. It contains 1,384 gene–trait association (GTA) problems, 911 datasets, and bioinformatician-curated analysis code and results. Dataset-specific Jupyter notebooks perform cohort preprocessing, while code/regress.py centralizes association analysis and model selection. Large input, output, and metadata directories are obtained separately from cloud storage; bundled code-and-data versions are also available through Kaggle and Hugging Face. Evaluation is local through eval.py, which compares dataset selection, preprocessing, and statistical-analysis outputs rather than exposing a hosted service or chat interface.
The preprocessing notebooks read GEO and TCGA inputs organized by trait, clean and standardize them, and link clinical and gene-expression data. They produce clinical_data, gene_data, linked CSV files, and cohort_info.json under output/preprocess/. code/regress.py selects suitable preprocessed cohorts for each trait-condition problem, applies statistical models and hyperparameter tuning, and writes significant_genes_condition_{condition name}.json files under output/regress/. These result files include importance-ranked genes, coefficients, absolute coefficients, and cv_performance. A candidate method must emit the same output structure, then can be evaluated with python eval.py -p {prediction_directory} -r {reference_directory} -t selection preprocessing analysis -s gene clinical linked.
- An agent-development team that needs a common GTA benchmark to measure dataset filtering, preprocessing, and significant-gene identification.
- A bioinformatician who wants to inspect cohort-specific preprocessing notebooks and reproduce the documented association-analysis workflow.
- A team building automated GEO and TCGA processing that needs to produce CSV and JSON outputs comparable with benchmark references.
- An evaluator who needs separate scores for selection, preprocessing, and analysis, or for gene, clinical, and linked preprocessing subtasks.
- A computational-genomics researcher studying gene associations with a trait while accounting for age, gender, or a co-existing trait.
How do you install or deploy this agent?
Clone the repository and enter it: git clone https://github.com/Liu-Hy/GenoTEX.git && cd GenoTEX. Download the metadata, input, and output directories and place them at the repository root; the README lists Google Drive and Baidu Cloud Disk, while complete code-and-data bundles are available on Kaggle and Hugging Face. Create the documented environment and install dependencies: conda create -n genotex python=3.10, conda activate genotex, pip install -r requirements.txt. For the Kaggle version, run python recompress_files.py first to recompress required files.
How do you use this agent?
Run the applicable cohort notebooks in ./code/{trait_name}/ to create preprocessed data, then run python code/regress.py from the repository root for GTA association analysis. To evaluate a custom method, make its outputs match the documented output CSV/JSON structure and run: python eval.py -p {prediction_directory} -r {reference_directory} -t selection preprocessing analysis -s gene clinical linked. -p is required; -r defaults to ./output. Omit -t or -s to evaluate all tasks or all preprocessing subtasks.
What are this agent's strengths and limitations?
- It separates evaluation into 1,384 unconditional and conditional GTA problems across dataset selection, preprocessing, and statistical analysis.
- It supplies 911 datasets, 237,907 lines of expert-curated analysis code, and cohort-level Jupyter notebooks with step outputs.
- Its reference outputs specify concrete clinical, gene-expression, linked-data, and significant-gene file formats for offline comparison.
- It covers both GEO and TCGA data and includes gene_synonym.json for gene-symbol normalization.
- The documented download total is 82.0 GB, so full use requires substantial local disk space; the README also states 41.5 GB of input data, leaving a size-definition difference to resolve.
- The complete workflow depends on externally hosted data or Kaggle/Hugging Face bundles, plus local conda, Python 3.10, and filesystem access.
- Preprocessing is primarily distributed across many cohort-specific notebooks; the documentation does not describe one command that batch-runs all preprocessing.
- The supplied repository metadata says NOASSERTION while the README says the dataset is CC BY 4.0, so the applicable license should be verified before adoption.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| GenoTEX Gene Expression Benchmark This agent | 36 · Major gaps | ★ 65 | 4mo ago | Jupyter Notebook | — |
| GenoMAS | 51 · Major gaps | ★ 134 | 5mo ago | Python | OpenAI API · Claude API |
| Open Science Workbench | 52 · Major gaps | ★ 4.9k | today | TypeScript | Codex · Claude Code |
| Scientific Agent Skills | 51 · Major gaps | ★ 46k | 3d ago | Python | Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: repository provides data sources (GEO, TCGA, Open Targets, NCBI) and license (CC BY 4.0), but lacks permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, or rollback mechanisms. Deductions: these aspects are absent in static review.
Evidence shows: README and datasheet provide detailed data structure descriptions, internal consistency is good; dependency list is explicit but lacks version pinning or integrity checks; error handling only mentions eval.py reports format errors, but no detailed failure message mechanism. Deductions: dependency availability unverified, failure messages insufficient.
Evidence shows: targets bioinformaticians and LLM agents, provides multiple usage scenarios (exploring benchmark, evaluating methods); but capability boundaries are not clearly stated, trigger conditions (e.g., eval.py parameters) are documented but not precise; environment requirements (Python 3.10, conda) are specified. Deductions: capability boundaries and trigger precision insufficient.
Evidence shows: README is well-structured, includes installation instructions, usage examples, license, and citation; naming conventions (e.g., trait name normalization) are documented; but lacks version history, changelog, and clear maintenance responsibility. Deductions: versioning and maintenance responsibility missing.
Evidence shows: output formats are detailed, facilitating use; benchmark provides expert-annotated data and code, adding marginal value; but cost-benefit is not discussed, and large data download (82GB) may increase usage cost. Deductions: cost-benefit not assessed.
Evidence shows: README cites arXiv paper and external data sources, but lacks specific data processing steps or validation results; cross-validation insufficient, facts and inferences not clearly separated. Deductions: cross-source corroboration and fact-inference separation insufficient.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Static review cannot verify availability of data download links and data integrity.
- Dependencies are not version-pinned, which may affect reproducibility.
- No explicit permission management or security audit; handle sensitive data with caution.