Voice Lab
Evaluate voice-agent language models and prompts with configurable scenarios and metrics.
Per-dimension scores and reasoning
Evidence shows: The repository provides no mechanisms or documentation for permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All trust criteria lack evidence, hence all scores are 0.
Evidence shows: The README's described functionality is broadly consistent with the code structure, but no error handling or failure message documentation is provided. Dependencies are listed but their availability is unverified. Thus self-consistency scores 1, dependency availability scores 1, and failure messages scores 0.
Evidence shows: The README clearly identifies the target audience (voice agent developers) and multiple use cases (model migration, cost optimization, prompt testing), but capability boundaries are unclear (e.g., only text part supported), trigger precision is partially described, and environment fit is limited (Python virtual environment only). Hence audience and scenarios score 2, capability boundaries score 1, trigger precision score 1, and environment fit score 1.
Evidence shows: The README provides clear installation and usage instructions with example configurations, but lacks version history, changelog, and explicit maintenance responsibility. License is Apache-2.0 but copyright holder is not specified. Thus information architecture scores 2, install notes score 2, naming stability scores 1, examples and FAQ score 2, known limitations score 1, license score 2, versioning and changelog score 0, and maintenance responsibility score 1.
Evidence shows: Output usability (test reports) is not detailed, marginal value (cost optimization, model comparison) is clearly described, and cost-benefit (cost savings) is mentioned but not quantified. Hence output usability scores 1, marginal value scores 2, and cost-benefit scores 1.
Evidence shows: Claims in the README (e.g., cost optimization) are not supported by specific data or test results, and facts are not distinguished from inferences. Thus claim traceability scores 1, cross-source corroboration scores 0, and fact-inference separation scores 1.
- The repository provides no security mechanisms or permission management; assess risks before use.
- Dependencies are not version-pinned, posing potential supply chain risks.
- The README claims cost optimization but provides no benchmark data or evidence.
- No version history or changelog is provided; maintenance status is unclear.
What does this agent do, and when should you use it?
Voice Lab is a testing and evaluation framework for voice agents and other LLM-powered agents. Its current scope is the text layer of a voice agent—the underlying language model and prompt—not audio processing. A predefined test is run through llm_testing/example_test.py, while llm_testing/test_details.json defines scenarios, system prompts, tool calls, success criteria, and personas. The project supports JSON-defined custom metrics scored with LLM-as-a-Judge, tracks results across configurations, and describes model comparison tables. Configurations can be edited directly or generated with the Voice Lab Configuration Editor.
After OPENAI_API_KEY is configured in the project-root .env file, run python llm_testing/example_test.py to execute the predefined test. Test scenarios live in llm_testing/test_details.json and can include system_prompt, initial_message, tool_calls, success_criteria, and persona. A persona can specify its name, role, traits, mood, and response style; tool_calls can define functions such as end_conversation and their parameter schema. Custom metrics can be defined in JSON, with metrics.json named as an example, then scored through LLM-as-a-Judge to assess model, prompt, or persona configurations.
- A team building a pharmacy call assistant tests alternative system prompts against customer personas and required confirmations before release.
- An engineer maintaining an LLM customer-service agent compares configurations while considering a move from a larger model to a smaller one.
- A developer validates structured conversation termination by defining an end_conversation function with required evidence and fields.
- An evaluation owner replaces manual review of many call logs with consistently scored JSON-defined metrics across test configurations.
- A team creating commerce or airline agent scenarios configures personas, contextual details, and scenario-specific success conditions.
What are this agent's strengths and limitations?
- Keeps scenarios, personas, tool calls, and success criteria in JSON, making test setups reproducible.
- Combines custom metrics with LLM-as-a-Judge scoring for a consistent way to evaluate prompt or model changes.
- Is designed to compare model configurations for performance, cost, and latency tradeoffs.
- Persona definitions include traits, mood, and response style for testing varied interaction styles.
- It currently covers only the text portion of voice agents; interruption and pause analysis remain planned work.
- The documented setup requires an OPENAI_API_KEY and provides no ready multi-provider execution path; LiteLLM support is still planned.
- The standalone eval agent is marked “Coming soon,” without a documented implementation or invocation.
- Parallel execution, detailed reporting, conversation replay, and batch processing are listed as future improvements.
How do you install or deploy this agent?
Clone and enter the repository: git clone https://github.com/saharmor/voice-lab.git && cd voice-lab. Create and activate a virtual environment: python3 -m venv venv && source venv/bin/activate. Install dependencies: pip install -r requirements.txt. Create .env in the project root and set OPENAI_API_KEY=your_openai_api_key.
How do you use this agent?
With the environment configured, run: python llm_testing/example_test.py. To add a scenario, edit llm_testing/test_details.json or use the Voice Lab Configuration Editor to generate JSON configuration. A scenario may configure system_prompt, initial_message, tool_calls, success_criteria, and persona.
How does this agent compare with similar options?
The project names Claude Sonnet, GPT-4, and GPT-4 Mini as examples for model migration and comparison, aiming to evaluate performance, cost, and latency tradeoffs. It does not provide benchmark results for them.