optim-agent
LLM coding agents as your hyperparameter optimizer: they read your code, reason about parameter semantics, propose trials, and record measured objective results.
Least privilege: SECURITY.md states agent CLIs launch as local subprocesses inheriting user permissions, but no concrete code evidence of permission bounding or sandboxing — 1. User confirmation: the loop is user-driven locally, but no source evidence of any pre-execution confirmation mechanism — 0. Data flow transparency: agents only propose values; the harness validates against the declared space and falls back to safe sampling, with full JSON/SQLite records — 2. Sensitive data: SECURITY.md instructs keeping secrets out of prompts and warns about inherited ANTHROPIC_API_KEY in nested sessions — 2, though it relies on user discipline. Dependency security: clean optional-dependency grouping and least-privilege CI (contents: read), but no lockfile or audit evidence — 2. External effects: drives tuning of arbitrary user systems via coding-agent CLIs; blast radius depends on user objective code and is not explicitly warned — 1. Rollback: study./SQLite persistence and resume are documented, but no failure-recovery policy detail — 1. Source attribution: acknowledges Optuna/OpenCode, complete MIT notice and author metadata — 2. No red-line risks found.
Self-consistency: install paths, quickstart, and sampler parameters align with tests that enforce README/docs/API coherence; however the README's language links contain garbled text (한국어, Deutsch rows render as mojibake) — 2. Dependency availability: requires-python>=3.9 with CI on 3.9/3.11/3.13 and optional extras; no pinned lockfile verified — 2. Failure messages: documented failure paths (401 remediation, timeout fallback to random sampling, ValueError naming the fix, OpenCode distributed limitation) — 2.
Audience/scenarios: the 'Where It Applies' table spans training, inference, quant, RL, scientific, and black-box cases, and explicitly states RL tuning does not replace policy learning — 2. Capability boundaries: 'Bounded execution', the credit-benchmark disclaimer, GIL/concurrency limits are all stated; this is the project's strongest area — 3. Trigger precision: sampler/pruner/mock controls are well specified, but validation beyond malformed agent replies is not shown — 2. Environment fit: mock backend for offline testing, SQLite WAL for distributed workers, clear Python compatibility; depends on a pre-authenticated paid/free CLI on PATH — 2.
Information architecture: clear sectioning (Why/Install/Quickstart/Where/Usage/Troubleshooting), docs site, tutorials, paper; deducted for garbled language-link labels in the README. Install notes: three install paths (skill, plugins, pip) with commands and prerequisites — 3. Naming stability: PyPI, module, and org names align; tests enforce public metadata — 2. Examples/FAQ: multiple reproducible examples, reproduction commands, and test-enforced example contracts — 3. Known limitations: GIL, in-flight invisibility under concurrency, credit-benchmark methodology caveat, OpenCode distributed gap — 3. License: MIT consistent across LICENSE and pyproject — 3. Versioning/changelog: v0.2.0; CHANGELOG.md is test-enforced to have Unreleased/0.1.0 sections but its content is not in the evidence — 2. Maintenance responsibility: CONTRIBUTING/SECURITY/CODE_OF_CONDUCT/ROADMAP present, CI coverage gate at 85%; publisher identity is an unverified pseudonym — 2.
Output usability: dual verbose table/line modes, four-section summaries, JSON/SQLite persistence — 2. Marginal value: semantic context in HPO proposals is a clear differentiator and benchmarks show context beats no-context; deducted because several reported gains are tiny (e.g., +0.2 on Acrobot) and pinned model version names (GPT-5.5, Opus-4.8) cannot be corroborated — 2. Cost benefit: small-budget leverage, free OpenCode models, and a zero-cost mock backend are highlighted, but real per-proposal token/time costs are unquantified — 2.
Claim traceability: benchmarks are fixed by benchmarks/manifest. and a Provenance/Publication-gate contract with full reproduction commands — 2. Cross-source corroboration: tests enforce README/docs/pyproject/benchmark coherence and an 85% coverage gate; but benchmark results exist only as in-repo artifacts with no independent replication — 2. Fact/inference separation: hedging like 'trajectory illustration' and 'methodological benchmark' is used appropriately, and the credit benchmark self-reports test leakage — 2. All judgments are static; no code was executed.
- This is a static source review (low confidence); no code was executed. Benchmark numbers come from in-repo artifacts and were not independently reproduced.
- The tool launches authenticated coding-agent CLIs as subprocesses inheriting your user permissions; when tuning arbitrary local systems, assess blast radius yourself and avoid running in directories containing sensitive data.
- Beware nested-session inheritance of ANTHROPIC_API_KEY (docs suggest env -u ANTHROPIC_API_KEY), and keep secrets and private instructions out of parameter/study context strings.
- Several README language-link labels are garbled, suggesting possible quality issues in the docs pipeline; verify translated documentation before relying on it.
- Reported gains in RL and credit benchmarks are small (e.g., +0.2 on Acrobot) and the credit benchmark self-reports validation/test selection leakage — do not extrapolate production benefits from these numbers.
- Publisher identity is not verified by any curated registry; establish the supply-chain trust chain yourself.
What does this agent do, and when should you use it?
optim-agent is a Python library and coding-agent skill that lets Claude Code, Codex, or OpenCode act as a hyperparameter optimizer. It exposes an Optuna-style create_study / Study / Trial API centered on AgentSampler: the agent proposes the next configuration using each parameter's semantic context plus the trial history, while the library validates proposals against the declared search space and falls back to safe sampling when replies are invalid. Objective evaluations stay in your code, and results persist to JSON or SQLite storage. Beyond the pip package, it installs as a Claude Code plugin, Codex plugin, or skill-installer skill so the agent can read your project code and drive the loop itself via study.ask / study.tell. It ships with AgentPruner, an agent-written study summary, max_concurrency plus distributed SQLite storage, and benchmarks covering Branin/Ackley functions, MNIST/CIFAR-10, Gymnasium RL control, and credit-default gradient boosting.
You declare parameters with context via trial.suggest_float / suggest_int in an objective function; AgentSampler (backends: claude / codex / opencode / mock) reads the study- and parameter-level context and the last history trials, and the agent proposes the next configuration. optim-agent validates the proposal, runs your objective, and records value and state to study. or study.db. Optionally AgentPruner (loose/medium/tight) compares learning curves to answer prune/keep, and summarize=True makes the agent produce a four-section summary after the last trial, persisted in storage. max_concurrency enables in-process threaded evaluation, while a shared SQLite (WAL) storage coordinates cross-process and cross-machine workers. In skill mode the agent reads project code and drives trials via study.ask(params) / study.tell(trial, value). Examples include examples/mnist.py, cifar10.py, hard_functions.py, credit_card.py, rl_control.py, plus reproducibility scripts.
- ML engineers with expensive model-training evaluations who need small-budget tuning (10-20 trials) and can supply semantic parameter descriptions
- Inference/serving teams tuning quantization, batching, decoding, caching, or routing parameters to trade off quality, latency, throughput, and cost
- Quant researchers optimizing signal windows, thresholds, rebalance rules, and risk controls against walk-forward return, drawdown, and turnover
- RL researchers tuning discretization, exploration schedules, and reward weights around a Q-learning control loop on Acrobot-v1 or LunarLander-v3
- Scientists tuning simulation inputs, solver settings, and experimental controls where faster convergence than random or TPE matters
- Owners of any black-box system with bounded categorical/integer/continuous parameters and a scalar objective
What are this agent's strengths and limitations?
- Semantic proposals: the agent combines parameter meaning with trial history; benchmarks show GPT-5.5 with context cuts MNIST cumulative error by 20.9% vs TPE and CIFAR-10 error by 20.8% vs Random
- Bounded execution: agents only propose values; the library validates against the declared space and falls back to safe sampling on invalid output, keeping objective evaluations authoritative
- Auditable: JSON/SQLite studies retain configurations, outcomes, states, context, and optional agent rationale, with resume and distributed-shared history
- Pluggable backends: claude / codex / opencode / mock — proposal quality improves as underlying models improve, with no optimization-code changes
- Requires an authenticated agent CLI (claude/codex/opencode); each proposal is a model call with associated cost
- Without context it can underperform baselines: context-free GPT-5.5 is 24.3% worse than TPE on MNIST, so users must invest in writing good context strings
- OpenCode currently does not support distributed studies; single-process or switch backends for distributed runs
- Threads share the GIL, so pure-Python CPU-bound objectives need separate processes with shared SQLite; concurrent workers cannot see each other's in-flight points
- Nested agent sessions can hit 401 from claude due to inherited ANTHROPIC_API_KEY, requiring env -u ANTHROPIC_API_KEY
How do you install or deploy this agent?
Requires Python and one authenticated agent CLI on PATH (claude, codex, or opencode). Install any of: pip package — python -m pip install optim-agent; from source — python -m pip install "optim-agent @ git+https://github.com/Optim-Agent/optim-agent.git"; Codex skill — $skill-installer install https://github.com/Optim-Agent/optim-agent; Claude Code plugin — claude plugin marketplace add Optim-Agent/optim-agent && claude plugin install optim-agent@optim-agent.
How do you use this agent?
Define an objective that declares parameters with trial.suggest_float("threshold", 0.05, 0.95, context="...") and returns the measured value; create study = oa.create_study(direction="maximize", sampler=oa.AgentSampler(backend="claude", effort="high", context="...", history=5), storage="study.", summarize=True); run study.optimize(objective, n_trials=20), then read study.best_value, study.best_params, and study.summary. For pruning: pruner=oa.AgentPruner(backend="codex", level="medium", effort="medium") with trial.report / trial.should_prune in the loop. For concurrency: storage="study.db" and max_concurrency=8. Use AgentSampler(backend="mock") for offline testing.
How does this agent compare with similar options?
The README benchmarks against Optuna's Random, TPE, and GP-BO: with semantic context at small budgets optim-agent's agent sampling wins, but without context it can lose to TPE. optim-agent deliberately adopts Optuna's Study/Trial interface.