Voice Lab
Evaluate voice-agent language models and prompts with configurable scenarios and metrics.
- Source repo
- saharmor/voice-lab
- Stars
- ★ 176
- Last updated
- 1y ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 27/100 · Major gaps
At a glance
- How it runs
- Works with
- Platform-specificOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- Low · running in minutes
- You'll need
- Typical use
- A team building a pharmacy call assistant tests alternative system prompts against customer personas and required confirmations before release.
- Not a fit if
- Teams needing audio-level voice analysis; only the text layer is covered
- Teams needing multi-provider models out of the box; only an OpenAI key is documented
- Teams needing parallel runs, detailed reports, or conversation replay; still planned
- Source review
- 27/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
Voice Lab is a testing and evaluation framework for voice agents and other LLM-powered agents. Its current scope is the text layer of a voice agent—the underlying language model and prompt—not audio processing. A predefined test is run through llm_testing/example_test.py, while llm_testing/test_details.json defines scenarios, system prompts, tool calls, success criteria, and personas. The project supports JSON-defined custom metrics scored with LLM-as-a-Judge, tracks results across configurations, and describes model comparison tables. Configurations can be edited directly or generated with the Voice Lab Configuration Editor.
After OPENAI_API_KEY is configured in the project-root .env file, run python llm_testing/example_test.py to execute the predefined test. Test scenarios live in llm_testing/test_details.json and can include system_prompt, initial_message, tool_calls, success_criteria, and persona. A persona can specify its name, role, traits, mood, and response style; tool_calls can define functions such as end_conversation and their parameter schema. Custom metrics can be defined in JSON, with metrics.json named as an example, then scored through LLM-as-a-Judge to assess model, prompt, or persona configurations.
- A team building a pharmacy call assistant tests alternative system prompts against customer personas and required confirmations before release.
- An engineer maintaining an LLM customer-service agent compares configurations while considering a move from a larger model to a smaller one.
- A developer validates structured conversation termination by defining an end_conversation function with required evidence and fields.
- An evaluation owner replaces manual review of many call logs with consistently scored JSON-defined metrics across test configurations.
- A team creating commerce or airline agent scenarios configures personas, contextual details, and scenario-specific success conditions.
How do you install or deploy this agent?
Clone and enter the repository: git clone https://github.com/saharmor/voice-lab.git && cd voice-lab. Create and activate a virtual environment: python3 -m venv venv && source venv/bin/activate. Install dependencies: pip install -r requirements.txt. Create .env in the project root and set OPENAI_API_KEY=your_openai_api_key.
How do you use this agent?
With the environment configured, run: python llm_testing/example_test.py. To add a scenario, edit llm_testing/test_details.json or use the Voice Lab Configuration Editor to generate JSON configuration. A scenario may configure system_prompt, initial_message, tool_calls, success_criteria, and persona.
What are this agent's strengths and limitations?
- Keeps scenarios, personas, tool calls, and success criteria in JSON, making test setups reproducible.
- Combines custom metrics with LLM-as-a-Judge scoring for a consistent way to evaluate prompt or model changes.
- Is designed to compare model configurations for performance, cost, and latency tradeoffs.
- Persona definitions include traits, mood, and response style for testing varied interaction styles.
- It currently covers only the text portion of voice agents; interruption and pause analysis remain planned work.
- The documented setup requires an OPENAI_API_KEY and provides no ready multi-provider execution path; LiteLLM support is still planned.
- The standalone eval agent is marked “Coming soon,” without a documented implementation or invocation.
- Parallel execution, detailed reporting, conversation replay, and batch processing are listed as future improvements.
How does this agent compare with similar options?
The project names Claude Sonnet, GPT-4, and GPT-4 Mini as examples for model migration and comparison, aiming to evaluate performance, cost, and latency tradeoffs. It does not provide benchmark results for them.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Voice Lab This agent | 27 · Major gaps | CLIFree + model costs | ★ 176 | 1y ago | Python | OpenAI API |
| AgentEvals Trace Evaluation | 68 · Some gaps | CLIFree | ★ 168 | 1d ago | Python | — |
| Giskard | 47 · Major gaps | Library / SDKFree + model costs | ★ 5.9k | 4d ago | Python | OpenAI API · Claude API |
| AI Eval Skills | 52 · Major gaps | Agent plugin / skill | ★ 1.7k | 1mo ago | — | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: The repository provides no mechanisms or documentation for permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All trust criteria lack evidence, hence all scores are 0.
Evidence shows: The README's described functionality is broadly consistent with the code structure, but no error handling or failure message documentation is provided. Dependencies are listed but their availability is unverified. Thus self-consistency scores 1, dependency availability scores 1, and failure messages scores 0.
Evidence shows: The README clearly identifies the target audience (voice agent developers) and multiple use cases (model migration, cost optimization, prompt testing), but capability boundaries are unclear (e.g., only text part supported), trigger precision is partially described, and environment fit is limited (Python virtual environment only). Hence audience and scenarios score 2, capability boundaries score 1, trigger precision score 1, and environment fit score 1.
Evidence shows: The README provides clear installation and usage instructions with example configurations, but lacks version history, changelog, and explicit maintenance responsibility. License is Apache-2.0 but copyright holder is not specified. Thus information architecture scores 2, install notes score 2, naming stability scores 1, examples and FAQ score 2, known limitations score 1, license score 2, versioning and changelog score 0, and maintenance responsibility score 1.
Evidence shows: Output usability (test reports) is not detailed, marginal value (cost optimization, model comparison) is clearly described, and cost-benefit (cost savings) is mentioned but not quantified. Hence output usability scores 1, marginal value scores 2, and cost-benefit scores 1.
Evidence shows: Claims in the README (e.g., cost optimization) are not supported by specific data or test results, and facts are not distinguished from inferences. Thus claim traceability scores 1, cross-source corroboration scores 0, and fact-inference separation scores 1.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- The repository provides no security mechanisms or permission management; assess risks before use.
- Dependencies are not version-pinned, posing potential supply chain risks.
- The README claims cost optimization but provides no benchmark data or evidence.
- No version history or changelog is provided; maintenance status is unclear.