A-Evolve
Evolves an agent’s prompts, skills, and memory from benchmark feedback.
- Source repo
- A-EVO-Lab/a-evolve
- Stars
- ★ 803
- Last updated
- 1mo ago
- Primary language
- Python
- FA score
- 38/100 · Major gaps
At a glance
- Works with
- Universal · cross-platformOpenAI API · Claude API
- You'll need
- Typical use
- A research team improving a code-repair agent on SWE-bench Verified can begin with the built-in
swe-verifiedseed workspace and adapter. - Main limitation
- Outcomes depend on the benchmark, base model, evolution algorithm, and task setup; the reported results do not establish comparable gains for a custom agent.
- Source review
- 38/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
A-Evolve is Python infrastructure for evolving an existing agent, not a standalone end-user agent. Developers provide an agent workspace and benchmark adapter to `ae.Evolver`, which runs a Solve, Observe, Evolve, Gate, Reload loop. Its filesystem workspace contract covers `manifest.yaml`, prompts, `SKILL.md` skills, tool configuration, and JSONL memory. Evolution engines modify those files from trajectories and evaluation feedback, then validate candidates on holdout tasks and can roll back regressions through Git. The repository includes adapters and seed workspaces for SWE-bench Verified, MCP-Atlas, Terminal-Bench, SkillsBench, and CL-Bench, while exposing interfaces for custom agents, benchmarks, and algorithms.
When Evolver.run(cycles=10) is called, A-Evolve has the agent run solve() on task batches, collects Trajectory objects and benchmark feedback, and records structured observations. An EvolutionEngine.step() implementation can inspect the workspace, observations, history, and trial context before changing prompt, skill, or memory files. The Gate phase validates a candidate on holdout tasks; regressed mutations are reverted through Git, while accepted versions are tagged as evo-1, evo-2, and so on. A custom agent implements BaseAgent.solve(self, task: Task) -> Trajectory; a custom benchmark implements BenchmarkAdapter.get_tasks() and .evaluate().
- A research team improving a code-repair agent on SWE-bench Verified can begin with the built-in
swe-verifiedseed workspace and adapter. - A developer building an MCP tool-calling agent can use
mcp-atlasand let evaluation feedback drive changes to skill files. - An algorithm researcher testing a new self-improvement strategy can implement
EvolutionEngine.step()while reusing trial, history, and Git rollback primitives. - A team with an existing custom agent that wants a benchmark-driven loop for prompts, skills, and memory can wrap it as a
BaseAgent. - An evaluator comparing evolution approaches for terminal-operation or skill-discovery tasks can use the Terminal-Bench or SkillsBench examples.
How do you install or deploy this agent?
Python 3.11+ is required. Install the core package with pip install a-evolve; Claude support with pip install a-evolve[anthropic]; MCP-Atlas support with pip install a-evolve[mcp]; SWE-bench support with pip install a-evolve[swe]; or all extras with pip install a-evolve[all]. For source development, run git clone https://github.com/A-EVO-Lab/a-evolve.git && cd a-evolve, then pip install -e ".[all,dev]". The README names Anthropic, OpenAI, and AWS Bedrock as LLM Provider examples, but does not document the exact credential configuration; configure authentication for the selected provider before use.
How do you use this agent?
For a minimal run, import with import agent_evolve as ae, create evolver = ae.Evolver(agent="swe-verified", benchmark="swe-verified"), then run results = evolver.run(cycles=10) and inspect results.final_score and results.converged. To evolve your own agent, subclass BaseAgent, implement solve() to return a Trajectory, and pass it as ae.Evolver(agent=MyAgent("./my_workspace"), benchmark="mcp-atlas"). The agent workspace should follow the documented filesystem contract and be reloadable by the agent after each cycle; accepted changes are Git-tagged.
What are this agent's strengths and limitations?
- Uses a standard directory as the evolvable state, so prompts, skills, and memory can be processed through one workflow without requiring knowledge of the agent’s internals.
- Separately pluggable
BaseAgent.solve(),BenchmarkAdapter, andEvolutionEngine.step()interfaces let adopters replace the agent, evaluation, or evolution strategy. - Candidate changes are checked on holdout tasks, with Git rollback for regressions and
evo-*tags for accepted versions. - Includes benchmark adapters and seed workspaces spanning code repair, MCP tool calling, terminal operations, skill discovery, and continual-learning evaluation.
- Outcomes depend on the benchmark, base model, evolution algorithm, and task setup; the reported results do not establish comparable gains for a custom agent.
- It automatically changes workspace files and relies on Git-style version control, so teams must define which prompts, skills, and memory files are safe to evolve.
- Although Anthropic, OpenAI, and AWS Bedrock are listed as provider examples, the supplied material does not specify credential setup, model selection, or cost controls.
- The Terminal-Bench domain explicitly involves Docker, so adopting that adapter may require a container runtime.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| A-Evolve This agent | 38 · Major gaps | ★ 803 | 1mo ago | Python | OpenAI API · Claude API |
| Webwright Browser Coding Agent | 66 · Some gaps | ★ 6k | 1mo ago | Python | Codex · Claude Code · OpenAI API · Claude API |
| hax | 62 · Some gaps | ★ 813 | today | C | OpenAI API · Claude API |
| ReadmeAI - AI-Powered README Generator | 51 · Major gaps | ★ 3k | 1d ago | Python | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows the repository lacks explicit implementation or documentation for permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. README mentions git rollback and audit trail but provides no concrete mechanism or code evidence. Hence all trust criteria score 0.
Self-consistency: README, pyproject.toml, and test files are largely consistent in interface and functionality descriptions, but full code verification is missing. Dependency availability: dependency list is clear, but version availability is unverified. Failure messages: no detailed error handling or failure message documentation in tests or docs.
Audience and scenarios: README clearly targets research community and developers, providing multiple usage scenarios. Capability boundaries: documentation explains pluggable design but lacks explicit limitations. Trigger precision: API design is clear but detailed trigger conditions are absent. Environment fit: supports multiple LLM providers and benchmarks, but environment configuration details are missing.
Information architecture: README is well-structured with quick start, architecture, algorithms sections. Install notes: provides pip and source installation guides. Naming stability: API naming is consistent but no version history. Examples and FAQ: provides code examples and benchmark demos, but lacks FAQ. Known limitations: not explicitly listed. License: MIT license is clear. Versioning and changelog: version number exists but no changelog. Maintenance responsibility: no explicit maintainers or contribution guidelines.
Output usability: clear API and examples provided, but actual output unverified. Marginal value: claims benchmark improvements but lacks independent verification. Cost-benefit: no cost analysis or resource requirements provided.
Claim traceability: benchmark results in README lack detailed experimental setup or reproduction steps. Cross-source corroboration: relies solely on README claims, no external verification. Fact-inference separation: does not clearly distinguish facts from inferences.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- Benchmark results and performance claims lack independent verification and may be overstated.
- Dependency security is unassessed, posing supply chain risks.
- Permission and data handling mechanisms are opaque, potentially involving sensitive information.
FAQ
Does it directly train or fine-tune my language model?
What happens when an evolved change performs worse?
evo-* tags.Can I use my own agent and benchmark?
BaseAgent.solve(), and a custom benchmark implements BenchmarkAdapter.get_tasks() and .evaluate().