AIDE ML
Iteratively writes, evaluates, and improves machine-learning code through tree search.
- Source repo
- WecoAI/aideml
- Stars
- ★ 1.5k
- Last updated
- 20d ago
- License
- MIT
- Primary language
- Python
- FA score
- 48/100 · Major gaps
At a glance
- Works with
- Universal · cross-platformOpenAI API · Claude API
- You'll need
- Typical use
- An ML practitioner with a tabular dataset who wants to explore modeling code from a request such as churn prediction measured by AUROC.
- Main limitation
- Initial use needs access to an LLM service and its applicable credentials; the documented default coding model is
gpt-4-turbo. - Source review
- 48/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
AIDE ML is the open-source reference implementation of the AIDE algorithm for machine-learning engineering tasks defined by a dataset, goal, and evaluation metric. It uses LLM-guided tree search over code: Python scripts become solution-tree nodes, generated patches create children, and metric feedback guides or prunes exploration. The repository includes the `aide` CLI, the `aide.Experiment` Python API, an HTML solution-tree visualizer, and a Streamlit web UI. A run writes the best code to `logs/<id>/best_solution.py` and an inspectable tree to `logs/<id>/tree_plot.html`. It is positioned for research and prototyping; the repository positions Weco as the related platform for broader code optimization, experiment tracking, and enhanced control.
A user supplies a data directory, natural-language goal, and metric through aide data_dir=... goal=... eval=... or aide.Experiment(data_dir=..., goal=..., eval=...). AIDE ML treats each Python script as a solution-tree node, calls the configured coding model to generate patches and child nodes, evaluates the code, and uses the resulting metric to steer search. Experiment.run(steps=...) returns the best solution, including valid_metric and code. CLI runs write logs/<id>/best_solution.py and logs/<id>/tree_plot.html; the web UI accepts an API key and uploaded data, lets users set Goal and Metric, and displays live logs, the solution tree, and the best code.
- An ML practitioner with a tabular dataset who wants to explore modeling code from a request such as churn prediction measured by AUROC.
- An agent-architecture researcher who wants to swap search heuristics, evaluators, or LLM backends and inspect the resulting code-search tree.
- A researcher reproducing the AIDE paper or testing a new code-generation and evaluation strategy.
- A team testing
qwen2.5through a local Ollama endpoint as the coding model for iterative ML optimization. - A user who prefers a browser workflow for uploading data, setting a goal and metric, starting AIDE, and watching results in Streamlit.
How do you install or deploy this agent?
Python 3.10+ is required. Install with pip install -U aideml. For the documented OpenAI quick start, set export OPENAI_API_KEY=<your-key>. Run: aide data_dir="example_tasks/house_prices" goal="Predict the sales price for each house" eval="RMSE between log-prices". For development installation: git clone https://github.com/WecoAI/aideml.git, then cd aideml && pip install -e ..
How do you use this agent?
Use agent.code.model to choose a coding model and agent.steps to control iterations, for example: aide agent.code.model="claude-4-sonnet" agent.steps=50 data_dir=... goal=... eval=.... In Python, create aide.Experiment(...) and call exp.run(steps=2). For the UI, change to aide/webui and run streamlit run app.py. For Ollama through an OpenAI-compatible endpoint, set OPENAI_BASE_URL="http://localhost:11434/v1" and use agent.code.model="qwen2.5"; for a fully local code-and-evaluator setup, also set agent.feedback.model="qwen2.5".
What are this agent's strengths and limitations?
- Its code-tree search makes scripts, LLM-generated patches, and metric feedback explicit, rather than presenting exploration as a single linear edit sequence.
- It provides a CLI, the
aide.ExperimentAPI, an HTML tree visualizer, and a Streamlit UI for scripted, embedded, and interactive workflows. - The documentation names OpenAI, Anthropic, Gemini, and OpenAI API-compatible local-model paths, with a concrete Ollama configuration.
- Runs preserve both the best Python code and the full solution tree for reviewing the final result and the search path.
- Initial use needs access to an LLM service and its applicable credentials; the documented default coding model is
gpt-4-turbo. - Local-model mode can still default to
gpt-4ofor evaluation; a fully local setup requires an extra feedback-model setting, and the documentation warns of possible performance loss. - Evaluation depends on the user-supplied data directory, goal, and metric; the supplied material does not establish universal support for arbitrary data formats or task types.
- The repository describes itself as a lean, research-friendly implementation; broader code optimization, experiment tracking, and enhanced controls are positioned in a separate platform product.
How does this agent compare with similar options?
The README cites an MLE-Bench result across 75 Kaggle competitions in which AIDE tree search earned four times as many medals as the best linear agent, OpenHands. That comparison concerns medals in that benchmark and does not establish a general performance result for every ML task.
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| AIDE ML This agent | 48 · Major gaps | ★ 1.5k | 20d ago | Python | OpenAI API · Claude API |
| BambooAI Data Analyst | 85 · Good | ★ 790 | 7d ago | Python | OpenAI API · Claude API |
| K-Dense BYOK | 49 · Major gaps | ★ 1.2k | 12d ago | TypeScript | ChatGPT · Codex · Claude.ai |
| MindSearch Deep Search | 42 · Major gaps | ★ 6.9k | 1y ago | JavaScript | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: README requires users to provide API keys (e.g., OPENAI_API_KEY) and supports local LLM options, indicating external API calls. Code execution is handled by an interpreter, but no explicit user confirmation is required. Data flow transparency is limited; no detailed explanation of how data is processed or transmitted. Sensitive data handling is not mentioned; no encryption or anonymization. Dependency security: some versions are pinned in requirements.txt, but no vulnerability scanning or security audit is provided. External effects: the agent generates and executes code, which may have external side effects, but no explicit restrictions. Rollback mechanism is not mentioned. Source attribution: README cites papers and external research, but code provenance is not clearly stated. Deductions: lack of user confirmation, insufficient data flow transparency, unclear sensitive data handling, inadequate dependency security, unclear external effects, missing rollback.
Evidence shows: README and test files provide clear examples and expected behavior, showing good self-consistency. Dependency availability: requirements.txt lists many dependencies but no version compatibility notes. Failure messages: tests cover timeout and exception cases, but no user-friendly error messages. Deductions: dependency availability not fully explained, failure messages may not be user-friendly.
Evidence shows: README clearly identifies target audiences (researchers and ML practitioners) and provides multiple usage scenarios (CLI, Web UI, Python API). Capability boundaries: describes the agent's purpose and limitations (e.g., requires LLM key). Trigger precision: provides detailed CLI parameters and examples. Environment fit: supports multiple LLM backends and local deployment. Deductions: capability boundaries not detailed enough, environment fit not covering all possible scenarios.
Evidence shows: README is well-structured, providing installation instructions, quick start, advanced options, and examples. Naming stability: no version history or naming conventions provided. Examples and FAQ: multiple examples provided, but no FAQ. Known limitations: only mentions local model performance drop. License is MIT, clear. Versioning and changelog: no CHANGELOG provided. Maintenance responsibility: no clear maintainers or contribution guidelines. Deductions: missing FAQ, incomplete known limitations, no changelog, unclear maintenance responsibility.
Evidence shows: Output usability: provides best solution code and visualization tree. Marginal value: agent automates ML pipeline development, providing clear value. Cost-benefit: requires LLM API key, may incur costs, but no cost estimation provided. Deductions: cost-benefit not fully explained.
Evidence shows: README cites papers and external research, providing traceable claims. Cross-source corroboration: cites multiple external projects. Fact-inference separation: README distinguishes algorithm description from product features. Deductions: some claims (e.g., performance improvements) lack specific data support.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- The agent executes generated code, which may have external side effects; run in an isolated environment.
- Requires LLM API keys; ensure key security to avoid leakage.
- Large dependency list may have version conflicts or security vulnerabilities; use virtual environments and update regularly.
FAQ
What credentials does it need?
OPENAI_API_KEY. The repository also documents configuration for Anthropic, Gemini, and OpenAI API-compatible local models; its Ollama example points OPENAI_BASE_URL at a local endpoint.Can it run fully locally?
OPENAI_BASE_URL to Ollama and set both agent.code.model and agent.feedback.model to a local model. The documentation notes that performance may decline.What artifacts does a run produce?
logs/<id>/best_solution.py and logs/<id>/tree_plot.html. The Python API also exposes the returned best solution's validation metric and code.