AutoHarness: Self-Learning Skills for Claude Code
Automatically distill, merge, update, and prune skills from real Claude Code sessions—keeping a clean, self-maintaining skill library without a daemon or benchmark.
Evidence shows: the plugin only operates on skills it generated, leaving user or third-party skills untouched; writes use atomic rename to avoid half-states; snapshots support rollback; capture windows are redacted at egress, with tests verifying keys and emails are replaced; zero dependencies reduce supply-chain risk. Deductions: user confirmation is weak, only /learn command, background reflection triggers automatically without explicit user approval; publisher identity unverified, source attribution relies only on repository metadata.
Evidence shows: configuration defaults are consistent; tests cover atomic writes, capture windows, redaction, etc.; dependencies are only Python standard library, high availability; failure messages are mentioned in README (e.g., rejection reasons recorded in runs/), but specific error handling details are limited. Deductions: no detailed runtime failure recovery documentation.
Evidence shows: clearly targets Claude Code users; scenarios include learning skills, merging, updating, pruning; capability boundaries clear (only operates on own skills); trigger conditions explicit (tool call count); environment requirements clear (Python 3.11+, Linux/macOS). Deductions: no Windows support mentioned, limited environment fit.
Evidence shows: README structure clear, includes install, configuration, how it works, comparison; install steps detailed; naming stable (autoharness); examples and FAQ section has walkthrough; license MIT; version info via plugin.json, but no CHANGELOG; maintenance responsibility by Tigerless Labs, but publisher unverified. Deductions: known limitations not explicitly listed, versioning changelog missing.
Evidence shows: output is native skill files, directly usable by host; marginal value high, provides self-learning skill layer reducing repetitive work; cost-benefit reasonable, zero dependencies, no daemon, saves tokens. Deductions: no benchmark, actual effectiveness depends on usage scenario.
Evidence shows: README claims (e.g., 42%→78% improvement) cite external papers, but no internal validation data; tests cover core functionality but not end-to-end learning effectiveness; facts and inferences clearly separated, e.g., defaults explicitly marked as placeholders. Deductions: cross-source corroboration insufficient, relies only on single repository.
- Publisher identity unverified; review code before installation.
- Background reflection may modify skills without explicit user confirmation; consider disabling or adjusting trigger thresholds in sensitive environments.
- Redaction rules may not be comprehensive; tests only cover AWS keys and emails, other sensitive data might leak.
What does this agent do, and when should you use it?
AutoHarness is a plugin for Claude Code that learns skills from your actual working sessions, distilling them into native SKILL.md files under .claude/skills/. It operates through a pipeline of components: Capture (CAP) hooks session events, Reflect (REF) proposes skill intents, a promoter validates and atomically stores them, IDX injects a session-start index, MNG handles skill lifecycle based on usage rates, and a curator consolidates similar skills. Each generated skill includes a ledger (.ledger.jsonl) and sidecar counters, with evidence files for auditability. The plugin only manages skills it created, leaving user-authored ones untouched. It requires Python 3.11+ and integrates exclusively with Claude Code, running zero-config with tunable environment variables.
AutoHarness registers hooks in Claude Code to capture each turn's input, output, and tool I/O. After a configurable number of tool calls (default 50), it spawns a background reflection that analyzes the recent episode, compares it with existing skills, and proposes an intent: add, merge, patch, delete, or drop a support file. The promoter validates the intent (safety, structure, ledger completeness) and writes the result atomically into the live skill directory. At session start, IDX builds a grouped index of AutoHarness-written skills and prepends it to the context. MNG runs once per session to update usage counters (loads, views, patches) and archive symbols that have never been used or viewed after maturity. The curator periodically (default every 250 tool calls) consolidates near-duplicate skills, taking snapshots for rollback. All operations are logged in per-skill ledgers. Users can manually trigger learning with the /learn command.
- A developer using Claude Code on a long-term project wants to automatically capture recurring workflows (e.g., build commands, debugging patterns) without writing files manually.
- A team standardizes on Claude Code and wants a shared project skill library that stays current, merging redundant entries as members contribute similar knowledge.
- A power user wants their Claude Code skills to adapt over time, retiring techniques that are no longer relevant as their workflow evolves.
- An auditor or team lead needs to review how and why skills were learned or changed, using the ledger and evidence files for traceability.
- A developer who has hand-curated skills wants an isolated layer that manages only its own creations, so their custom skills remain untouched.
- A researcher wants to build a benchmark from real usage data, using the accumulated ledgers and evidence as a foundation.
What are this agent's strengths and limitations?
- Self-learning without a daemon or benchmark, using real usage data, avoiding extra token costs for evaluation.
- Consolidation merges overlapping skills and resolves contradictions, keeping the library coherent.
- Isolation: only affects skills it created, leaving user-written or installed skills untouched.
- Full audit trail with per-skill ledgers and evidence files, enabling traceability and future benchmark construction.
- Archive instead of delete, with snapshots to safeguard against bad merges and easy recovery.
- Tightly coupled to Claude Code; cannot be used with other AI assistants.
- Requires Python 3.11+ on PATH; plugin functions fail without it.
- Defaults may be suboptimal for short sessions; requires manual tuning for faster learning.
- Injects a session-start index and runs background reflections, consuming context and tokens even if not learning.
- Graduation review only removes skills never viewed or used; may not catch skills that are viewed but never adhered to.
- Project is in early stages; default parameters are placeholders pending empirical calibration.
How do you install or deploy this agent?
- Ensure python3 is on your PATH (the plugin is pure Python, zero third-party dependencies). 2. In Claude Code's input box, run: /plugin marketplace add tigerless-labs/autoharness and /plugin install autoharness@autoharness. 3. Run /reload-plugins or restart Claude Code.
How do you use this agent?
After installation, AutoHarness runs automatically in the background. It triggers a reflection after every 50 tool calls (configurable) and consolidation every 250. To learn the current session on demand, type /learn. Skills are stored in .claude/skills/ (project) or ~/.claude/skills/ (global). Inspect lifecycle via cat .claude/skills/<name>/.ledger.jsonl and .sidecar.json. To uninstall, run /plugin uninstall autoharness@autoharness and /plugin marketplace remove autoharness.
How does this agent compare with similar options?
Compared with hermes-agent (NousResearch), AutoHarness requires no resident daemon, triggers learning based on session work rather than timers, and validates skills through adherence in use rather than a benchmark. Unlike Self-Harness (paper), AutoHarness operates online without offline gating and needs no oracle. It also controls library growth via capacity caps and archiving, unlike unbounded growth approaches.