ChatTTS Conversational Speech
Generate expressive, multispeaker Chinese and English speech for everyday dialogue.
The product is presented primarily as a locally invoked speech-generation library and examples, with no shown request for credentials or broad system permissions; the README also discloses command-line audio-file creation. Deductions apply because model and dependency download flows, telemetry, and network behavior are not comprehensively documented, no policy covers sensitive input text or generated audio, and there is no operation-level confirmation or rollback facility. Most requirements are loosely bounded, while the workflow downloads and installs a third-party deb without a checksum in the shown steps and pins Actions to release tags rather than commits. Licensing, model terms, upstream acknowledgements, contributors, and contact routes provide thorough source attribution. The unverified publisher is treated only as unknown.
The README, dependency list, and workflow describe a mostly coherent installation and use path, with PyPI, GitHub, Hugging Face, and several installation choices supporting ordinary availability. Deductions apply because English is both marked supported and called experimental, important artifacts depend on external services, and no offline availability guarantee is given. Failure guidance is limited to installation warnings, known quality problems, and a compatibility fallback for saving audio; the example uses a bare except and offers no systematic diagnostics.
The documentation clearly identifies conversational TTS, LLM assistants, research use, Chinese and English, and WebUI, CLI, and Python audiences. Boundaries are unusually explicit: research and educational model use, noncommercial model licensing, experimental English, limited control tokens, unstable output quality, and unfinished emotion control. Deductions apply because trigger and special-token semantics are largely example-driven and omit validation or conflict behavior. Environment guidance covers Python 3.11, Linux-only dependencies, GPU options, and a VRAM estimate, but lacks a complete OS, CPU, driver, and compatibility matrix.
The README has strong organization and includes installation routes, quick starts, basic and advanced examples, an FAQ, roadmap, licensing, and support contacts. It clearly separates AGPLv3+ code from the CC BY-NC 4.0 model and supplies the full AGPL text, justifying full license credit. Deductions apply because package-name casing and multiple installation sources may create some ambiguity, no API stability commitment is stated, and a PyPI badge plus roadmap is not a version policy or changelog. Issues, pull requests, email, and communities provide maintenance paths, but no accountable maintainers are named and publisher identity remains unknown.
The outputs are directly usable audio arrays and MP3 or WAV files, and examples demonstrate speaker sampling, prosody controls, and multiple interfaces, supporting substantial marginal value for dialogue speech generation. Cost-benefit is deducted because only a minimum 4GB VRAM estimate, one RTX 4090 speed figure, and an RTF estimate are supplied; model size, CPU performance, memory, energy, and production deployment costs are not systematically documented. The model's noncommercial restriction also narrows practical adoption.
Installation commands, API calls, dependency constraints, hardware estimates, and license statements are traceable to specific supplied files. Major claims about prosody superiority, training-hour scale, naturalness, and the effectiveness of safety mitigations lack accompanying benchmarks, dataset documentation, or auditable results, so claim traceability remains thin. README and LICENSE provide limited corroboration for the code license, while requirements and workflows support some environment facts, but nearly all evidence is first-party. Released features, roadmap items, experimental behavior, and disclaimers are generally separated, earning partial credit for fact-inference separation; comparative marketing and safety-effect claims are not clearly labeled as unverified.
- The model uses CC BY-NC 4.0 and is expressly limited to educational and research use; do not infer commercial model rights from the code's AGPL license.
- Input text and generated audio may be sensitive, but the supplied material gives no retention, telemetry, caching, remote-interaction, or deletion policy.
- Most requirements are not exactly locked; create a lockfile, audit transitive dependencies, and verify model and binary download integrity before deployment.
- The workflow directly downloads and installs a third-party deb without first checking a fixed digest in the shown steps; automation should use immutable pins and provenance checks.
- The README acknowledges multispeaker and poor-quality failures requiring repeated sampling; single outputs should not be trusted in high-risk or identity-sensitive settings.
- Training-data provenance, comparative quality claims, and the claimed safety effect of added high-frequency noise are not independently substantiated by the supplied files.
What does this agent do, and when should you use it?
ChatTTS is a generative text-to-speech model built for dialogue, including spoken output for LLM assistants, rather than a complete autonomous agent. The repository supplies algorithm infrastructure, pretrained-model access, a Python package, a command-line example, and a local WebUI that turn text into WAV or MP3 audio. It supports Chinese and English, sampled speaker identities, and limited control over conversational features such as laughter, pauses, and oral delivery. The public release is a 40,000-hour pretrained model without SFT, while the main model is described as having been trained on more than 100,000 hours of Chinese and English audio. The code uses AGPLv3+, but the model uses CC BY-NC 4.0 and is restricted to educational and research use, making it unsuitable for direct commercial adoption.
A program creates ChatTTS.Chat(), loads the model with chat.load(compile=False), and passes one or more strings to chat.infer(texts), which returns audio arrays. Those arrays can be written as 24 kHz WAV files with torchaudio.save; the command-line example instead saves results as ./output_audio_n.mp3. chat.sample_random_speaker() produces a speaker embedding, while ChatTTS.Chat.InferCodeParams accepts spk_emb, temperature, top_P, and top_K. ChatTTS.Chat.RefineTextParams applies sentence-level prompts such as [oral_2][laugh_0][break_6]; with text refinement skipped, [laugh], [uv_break], and [lbreak] provide word-level controls. The repository also launches a local interface through python examples/web/webui.py and lists streaming audio generation and zero-shot inference code as completed features.
- Speech researchers evaluating Chinese and English conversational TTS or investigating autoregressive generation, audio tokenization, and prosody.
- LLM-assistant prototype teams that need to render responses with conversational pauses, laughter, and an oral delivery style.
- Interactive-demo developers using sampled speaker embeddings to prototype dialogue involving multiple character voices.
- Python developers processing batches of text and saving the returned arrays as 24 kHz WAV files.
- Research teams comparing repeated samples through a local WebUI, command line, or Python API to select an acceptable result.
What are this agent's strengths and limitations?
- It is optimized for dialogue and supports Chinese, English, and multiple speakers instead of targeting only single-voice narration.
- Sentence- and word-level controls can explicitly introduce laughter, pauses, and oral-style prompting.
- Python, command-line, and local WebUI interfaces cover programmatic batches, terminal workflows, and interactive experiments.
- Random speaker embeddings, streaming audio generation, and zero-shot inference code provide useful foundations for research and prototyping.
- The released model is under CC BY-NC 4.0 and explicitly limited to educational and research use, blocking straightforward commercial deployment.
- Autoregressive generation can produce unintended multiple speakers or poor audio quality; the documented workaround is to sample repeatedly.
- Current word-level controls are limited to
[laugh],[uv_break], and[lbreak]; multi-emotion control remains unfinished. - English is described as experimental, and the source provides no production-stability guarantee.
- A 30-second clip requires at least 4 GB of GPU memory; even the documented RTX 4090 result has an RTF of about 0.3.
- The 40,000-hour model was trained with some high-frequency noise and its audio quality was deliberately compressed with MP3 to deter misuse, limiting high-fidelity applications.
How do you install or deploy this agent?
Clone the repository and install its requirements:
git clone https://github.com/2noise/ChatTTS
cd ChatTTS
conda create -n chattts python=3.11
conda activate chattts
pip install -r requirements.txtAlternatively, install the stable package:
pip install ChatTTSInstall the latest GitHub version with:
pip install git+https://github.com/2noise/ChatTTSFor editable local development:
pip install -e .The first run must load the model; no API credential is documented. The FAQ states that generating a 30-second clip requires at least 4 GB of GPU memory. Linux users may optionally install safetensors vllm==0.2.7 torchaudio. The documented TransformerEngine integration cannot currently run properly, and FlashAttention-2 is also discouraged because it slows generation.
How do you use this agent?
From the project root, launch the WebUI:
python examples/web/webui.pyGenerate MP3 files from the command line:
python examples/cmd/run.py "Your text 1." "Your text 2."Minimal Python invocation:
import ChatTTS
import torch
import torchaudio
chat = ChatTTS.Chat()
chat.load(compile=False)
wavs = chat.infer(["Hello from ChatTTS.", "你好，这是 ChatTTS。"] )
torchaudio.save("output.wav", torch.from_numpy(wavs[0]).unsqueeze(0), 24000)Some torchaudio versions require torch.from_numpy(wavs[0]) without adding a dimension. For prosody control, create ChatTTS.Chat.RefineTextParams(prompt='[oral_2][laugh_0][break_6]') and pass it to chat.infer.
How does this agent compare with similar options?
The repository cites Bark, XTTSv2, and VALL-E as autoregressive TTS references and notes that systems in this class commonly face hard-to-avoid speaker-mixing and quality instability. Fish Speech is credited as a reference for GVQ audio tokenization in LLM modeling, while ChatTTS uses Vocos as a pretrained vocoder. No consistent named benchmark is provided, so the source does not support a broad claim that ChatTTS is faster, more stable, or higher quality than each of those systems.
FAQ
Can I use the released model in a commercial product?
How much GPU memory does inference require?
Does it require ChatGPT or another hosted model API?
Why can output contain mixed speakers or poor audio?
Which emotions and delivery features can I control today?
[laugh], [uv_break], and [lbreak]; broader emotion control is still on the roadmap.