earshot: Ears for Coding Agents

What does this agent do, and when should you use it?

earshot is an open-source toolkit that gives AI coding agents hearing: it turns audio files into word descriptions, description-match verdicts, best-moment trims, real-vs-synthetic detection, second-by-second listening timelines, annotated listening sheets and scorecards against professional launch films. The same capabilities ship as a CLI, a Python API and an MCP server, all running locally on CPU with no GPU required. Its forge component lets an agent write sounds as physical-synthesis code recipes and improve them by listening through a critic and evolutionary search, while fit translates a recording back into a code recipe with knobs and morphs. The sfx/kit layer ships a bank of 252 code-rendered sound effects and per-film sound kits with measured sync points. Two open models (CLAP and an AudioSet tagger) download on first use and then run fully offline; the repository itself contains no audio.

earshot reads audio or video files and runs two open models — CLAP (laion/larger_clap_general) for text-audio embeddings and an AST AudioSet tagger (MIT/ast-finetuned-audioset-10-10-0.4593) — plus a set of model-free measurements. Core commands: describe (calibrated z tags, AudioSet labels, measurements), match/rank (how well a description fits, candidate ordering with held-out evidence), find (locating best moments as sample trims), realness (real recording vs code-synthesized), timeline and sheet (event transcripts and annotated PNGs), and critique (a scorecard against a profile of 7 professional launch films with timestamped fixes). forge renders candidates from physical-synthesis recipes (modal resonators, filtered noise, a computed room and microphone) and iterates them with a critic, model-free penalties and evolutionary search; fit fits a small synthesis model to a recording by analysis by synthesis and emits a recipe module with render(), variations and morphs. sfx render and kit render produce sounds from the 252-effect bank (no models, no network) with an audit for clicks, clipping and other artifacts, and the launch preset places each role against a music bed and picture cuts. Everything is exposed to agents like Claude Code via earshot mcp as an MCP server.

1. A developer producing a product launch film with Claude Code: have the agent listen to the soundtrack and run critique for a scored list of fixes against professional reference films
2. A team adding sound to code-made films: use kit render with the launch preset to place bank effects by measured music-bed level, cut offsets and minimum run gaps
3. Agent workflows trialing many audio assets: rank candidates by one sentence of description and find key moments inside long recordings
4. A sound designer collaborating with an agent: write forge recipes, iterate from the critic's worded feedback and side-by-side spectrograms
5. Anyone needing a recording turned into parametric code: fit converts a click or key press into a recipe module that re-renders, varies and morphs it

How do you install or deploy this agent?

Prerequisites: Python 3.10+, ffmpeg/ffprobe on PATH; CPU only, no GPU.

git clone https://github.com/Leonxlnx/earshot && cd earshot
python -m venv .venv
.venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cpu   # Windows: .venv/Scripts/python
.venv/bin/python -m pip install -e ".[mcp,llm,dev]"
.venv/bin/python -m earshot doctor --load

First model use downloads two open models (~1.1 GB total) into .cache/hf inside the repo (or EARSHOT_HOME): CLAP laion/larger_clap_general and AST MIT/ast-finetuned-audioset-10-10-0.4593, both at pinned revisions; afterwards it never contacts the Hub. earshot doctor --fetch downloads them up front.

Register as an MCP server for Claude Code:

claude mcp add earshot --scope user -e EARSHOT_THREADS=3 -- /path/to/earshot/.venv/bin/python -m earshot mcp

How do you use this agent?

CLI usage (each command prints a plain-language summary first; --json gives the full result):

earshot describe enter.ogg
earshot match key.ogg "a deep thocky mechanical keyboard press" --alts "a mouse click"
earshot find keys.ogg "a single mechanical key press" --top 4
earshot critique film.mp4
earshot forge whoosh --refset whoosh_fast --budget 96 --dir made/whoosh
earshot fit click.wav -o made/click_fit.py --report
earshot sfx render whoosh_fast/crisp -o w.wav --dur 0.8 --apex 0.32
earshot kit init --preset launch -o audio/kit.json && earshot kit render audio/kit.json --out audio/kit

Python API:

import earshot
earshot.describe('whoosh.wav')
earshot.find('typing.ogg', 'a single isolated key press')
from earshot import api

Note: each CLI call is a new process and loads the models (~20-40 s); for multiple calls use the MCP server (models stay loaded) or a Python script with from earshot import api. The optional llm_critique tool needs your own GEMINI_API_KEY and explicit consent to upload.

What are this agent's strengths and limitations?

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

How does FollowAgents rate this agent?

FAQ