Weco Autoresearch
Iteratively optimize code, prompts, and ML workflows against metrics you define.
Per-dimension scores and reasoning
The default does not automatically apply the best change, and review, diff, stop, resume, and source-snapshot mechanisms provide meaningful confirmation and recovery controls. File mutation, evaluation execution, dashboard interaction, and credit-related operations are also described. Deductions apply because the tool can execute arbitrary user-supplied evaluation commands and repeatedly rewrite source without documented sandboxing or privilege reduction; the exact code and telemetry sent to Weco, model providers, LangSmith, or other services is unclear; and API-key storage, redaction, rotation, and log-leakage controls are not documented. Most dependencies are not pinned and no vulnerability-scanning or update policy is evidenced, although releases use OIDC and Sigstore signing. Project, algorithm, and external benchmark attribution are present, but publisher identity remains unverified and responsibility for the unspecified upstream mirror is unclear.
The README explains observe exit codes, retries, and strict mode, while tests cover argument validation, non-interactive failure, and backend dispatch, supporting a good but incomplete failure-message score. Deductions chiefly reflect static inconsistencies: the Python 3.8+ badge conflicts with pyproject's >=3.10 requirement, and the claim that an unspecified model is selected automatically conflicts with a documented fixed default model. There is no lockfile, several dependencies have only minimum or no version constraints, and fallback behavior for unavailable external services and models is incompletely documented.
The material thoroughly identifies agent-harness, prompt, ML, CUDA/Triton, and general performance-optimization audiences, with single-file, multi-file, custom-metric, model-selection, external-observation, and multiple coding-assistant workflows. Boundaries include the backtick limitation, resumable statuses, timeouts, device differences, and experimental setup commands, but there is no systematic account of unsuitable projects, isolation of dangerous evaluation commands, or behavior during service outages. CLI and natural-language triggers are fairly precise. macOS, Linux, Windows, pip, and source installs are covered, but the Python-version contradiction and lack of a fuller platform/dependency matrix prevent full environment-fit credit.
The README has strong information architecture spanning installation, quickstart, argument tables, command reference, run management, log layout, and numerous scenario examples. The complete Apache-2.0 file agrees with package metadata, and installation notes are particularly comprehensive. Deductions reflect the absence of a dedicated FAQ and systematic limitations section, plus conflicting defaults or requirements. Version 0.4.0 and an automated release path exist, but no changelog or migration notes are supplied. The Weco AI Team identity, contact email, and release workflow provide a maintenance route, yet the repository is described as a mirror of an unnamed upstream source of truth, leaving responsibility and contribution/update paths partially unclear.
Programmatic JSON status/results, unified diffs, best-node inspection, indexed logs, snapshots, resume, and human review make outputs highly usable. Automated metric-driven code search plausibly adds value over manual iteration, with examples spanning several domains. However, production-grade positioning and performance gains are primarily documentary claims, while the supplied tests cover only LangSmith argument and command construction rather than the product's overall effectiveness. The README acknowledges that difficult tasks may require tens of hours and offers controls for steps, models, and balances, but gives no pricing, typical token/compute consumption, or decision threshold for worthwhile use, limiting the cost-benefit score.
Many operational claims trace to concrete commands, options, log structures, tests, and release workflows; README, pyproject, tests, and CI also corroborate the package name, CLI entry point, license, and parts of the integration surface. Deductions apply because core optimization quality, production readiness, and broad model availability are not substantially supported by the supplied implementation or tests, and the external benchmark is only cited in the README. Python requirements and model-default descriptions also conflict. Experimental labels, limitations, and the external-report attribution generally distinguish facts from guidance, but marketing conclusions are not consistently marked as inferences unverified by this static review.
- The tool repeatedly modifies selected source files and executes a user-provided evaluation command. Run it in an isolated environment, review the command first, and keep an independent version-control backup.
- Before submitting sensitive code or data through login, BYOK, observe, LangSmith, or dashboard workflows, verify the actual upload scope, credential storage, retention period, and deletion controls for every service.
- Do not rely on the README's Python 3.8+ badge: pyproject requires Python >=3.10. Also verify the conflicting automatic-versus-fixed model-default behavior before deployment.
- Dependencies are not comprehensively locked. Generate a lockfile, perform supply-chain and vulnerability scans, and inspect remote installer scripts instead of piping them directly into a shell.
- Costs may grow substantially with model choice, step count, and long evaluations. Set budgets, timeouts, and resource limits before enabling automatic top-ups or large runs.
What does this agent do, and when should you use it?
Weco is a Python CLI that uses LLM-guided tree search to modify and evaluate code against a measurable objective. Its primary interface, `weco run`, accepts one or more source files, an evaluation command, a metric, and a maximize or minimize goal, then retains candidate implementations and results across iterations. It targets agent harnesses, prompts, machine-learning pipelines, and performance-sensitive CUDA or Triton code, but requires the user to supply an evaluator that emits a numeric metric. For loops controlled elsewhere, `weco observe` records external agent, scripted, or manual experiments and displays their tree, code diffs, and metrics in the Weco dashboard. The deployment boundary is a local CLI or Python workflow connected to Weco services or supported model-provider credentials, while source files and evaluation commands execute in the user's environment.
weco run reads a file supplied through --source or multiple files supplied through --sources, asks the selected LLM to propose changes, and executes --eval-command. The evaluator must print the value associated with --metric to stdout or stderr; Weco parses that value, follows --goal maximize or --goal minimize, and continues its tree search. Candidate code, metrics, and optional execution logs are stored under .runs/. Operators can inspect a run through weco run status, results, show, and diff, update guidance with instruct, stop while retaining the tree, or use review, revise, and submit for approval workflows. weco resume <run-id> continues interrupted work, while --apply-change writes the best result back to the source. Separately, weco observe init and weco observe log send descriptions, metrics, and source snapshots from externally controlled experiment loops to the dashboard.
- An LLM-agent engineer with an accuracy, win-rate, or cost benchmark who wants to tune prompts, tools, and agentic scaffolding together.
- A prompt engineer who needs to refine or compress prompts while measuring win rate, relevance, or format adherence.
- An ML researcher with a repeatable validation script who wants to search feature transformations, architectures, or an entire training pipeline.
- A CUDA or Triton developer who can benchmark latency, throughput, or memory bandwidth and wants automated kernel exploration.
- A team already running a custom agent, script, or manual experiment loop that wants dashboard tracking through
weco observewithout transferring control of the loop.
What are this agent's strengths and limitations?
- Optimization is driven by the user's executable evaluator and real numeric objective, covering accuracy, cost, latency, throughput, and other measurable goals rather than static code suggestions alone.
- Single-file and multi-file optimization are both supported, with commands for ranked results, node details, unified diffs, review, revision, submission, stopping, and resuming.
- Model selection spans OpenAI, Anthropic, Google Gemini, hosted open models, and Concentrate AI routes, with a BYOK option through
--api-key. weco observecan provide experiment tracking and visualization without requiring an existing optimization loop to surrender control to Weco.- Setup commands install Weco skills for Claude Code, Cursor, Codex, and OpenClaw so a coding assistant can help create evaluations, configure runs, monitor iterations, and interpret results.
- Adopters must design a reliable, repeatable evaluator that prints a numeric metric; weak or noisy evaluation directly limits the credibility of the search.
- Complex research tasks may require many steps and tens of hours, increasing model usage, compute consumption, and Weco credit costs.
- The tool modifies source candidates and executes user-supplied commands, so it needs a controlled, recoverable execution environment;
--apply-changealso writes the winner directly to source files. - Backticks in the code being optimized are currently unsupported and may require preprocessing existing sources.
- Evaluation output beyond 51,000 characters is truncated, so the metric should appear near the end to reduce parsing risk.
How do you install or deploy this agent?
Python 3.8.0 or newer is required. On macOS or Linux, run:
curl -fsSL https://weco.ai/install.sh | shOn Windows PowerShell, run:
irm https://weco.ai/install.ps1 | iexThe Python package can also be installed with:
pip install wecoFor a source installation:
git clone https://github.com/wecoai/weco-cli.git
cd weco-clipip install -e .
Run weco login before first use of the hosted service. Alternatively, pass one or more provider credentials as --api-key provider=key. Coding-assistant skills can be installed with weco setup claude-code, weco setup cursor, weco setup codex, weco setup openclaw, or weco setup all.
How do you use this agent?
Prepare the source to modify and an executable evaluation script. The evaluator must print the named metric and a numeric value, such as speedup: 1.5. A minimal run is:
weco run --source module.py --eval-command "python evaluate.py --path module.py" --metric speedup --goal maximize --steps 10For a multi-file task, use:
weco run --sources model.py utils.py config.py --eval-command "python evaluate.py" --metric accuracy --goal maximize --steps 10After launch, use weco run status <run-id> to inspect progress, weco run results <run-id> --top 5 for ranked outcomes, and weco run diff <run-id> --step best for the best diff. Add --apply-change to write the winner automatically, or run weco resume <run-id> after an interruption. To track an external loop, initialize it with weco observe init --name "my-experiment" --metric val_bpb --goal min --source train.py, then log a step with weco observe log --run-id <run-id> --step 0 --description "baseline" --metrics '{"val_bpb": 2.36}' --source train.py.
FAQ
Does Weco cost money to run?
weco credits balance, weco credits topup, and weco credits autotopup, so hosted runs use credits. Users may also supply model-provider keys through --api-key provider=key; the supplied material does not document exact pricing.Which files can it read or overwrite?
--source or --sources. Weco creates candidates, evaluates them, and may store snapshots and logs under .runs/; automatic replacement of the original source is enabled only with --apply-change.What happens after an evaluation or network failure?
weco run can be continued with weco resume <run-id>. weco observe log retries Weco-side transient failures and normally reports them without failing the caller, while bad flags, unreadable files, missing authentication, or rejected requests return nonzero; --strict makes service-side failures fatal too.Is it locked to one model provider?
--model selects a specific model. Some Claude offerings are delivered through Vertex AI while one listed model uses Anthropic directly, so provider paths differ by model.Will it work for a task with no objective metric?
weco run requires an evaluation command that emits a parseable numeric metric and a direction to maximize or minimize it.