CyberVerse Digital Human Assistant

What does this agent do, and when should you use it?

CyberVerse is a self-hosted realtime digital-human agent framework built around WebRTC and voice interaction. Its foreground PersonaAgent maintains conversation flow, handles interruptions and context switches, while background SubAgents take on longer work such as search, research, material organization, summarization, and HTML report generation. Character conversation history is persisted on local disk, and imported knowledge bases, documents, and biographical material are indexed for retrieval-augmented generation. Its plugin-oriented stack makes brain, voice, hearing, tools, memory, and face replaceable, with provider keys and endpoints configurable at `/settings`. It can run as a voice-only assistant or optionally deliver a realtime video persona through local FlashHead or LiveAct models and listed cloud digital-human services.

Runtime behavior is controlled through config/cyberverse.yaml; omni, LLM, embedding, TTS, and ASR definitions are discovered from infra/config/*_models/ and can be overridden under config/*_models/. The system handles realtime voice interaction, calls configured ASR, LLM/omni, and TTS services, saves and reloads character history, and retrieves indexed character material for responses; background SubAgents can perform search, research, organization, summarization, and HTML report generation asynchronously. With inference.avatar.enabled: true, the Python inference service can drive FlashHead or LiveAct according to config/avatar_models/; Baidu Xiling is embedded as an H5 iframe in the frontend and uses sendAudioData / AUDIO_STREAM_RENDER for audio delivery. Deployment starts the Python inference service, Go API server, and frontend with make inference, make server, and make frontend, and exposes health checking at GET /api/v1/health.

1. A developer building a self-hosted voice assistant can disable avatar inference with `inference.avatar.enabled: false` and operate CyberVerse as a realtime voice-only system.
2. A character-driven application team can import biographies, documents, or knowledge bases for each persona and use persisted history plus RAG to maintain continuity across sessions.
3. A product team seeking video-call-style interaction can use FlashHead or LiveAct when it has the documented CUDA-capable GPU environment, weights, and model configuration.
4. A research-oriented assistant can delegate search, material organization, summarization, or HTML report generation to a SubAgent while the user continues the foreground voice conversation.
5. A deployer without a local avatar GPU can use the documented cloud digital-human paths, including Baidu Xiling, Xunfei Digital Human, or Vidu S1.

What are this agent's strengths and limitations?

How do you install or deploy this agent?

Prerequisites are Node 18+, Go 1.25, Conda, Python 3.10+, FFmpeg, libopus-dev, libopusfile-dev, libsoxr-dev, and pkg-config.

git clone https://github.com/dsd2077/CyberVerse.git
cd CyberVerse
conda create -n cyberverse python=3.10
conda activate cyberverse
cp -r infra/config config

Populate config/env with at least one documented model credential, such as DASHSCOPE_API_KEY, or both DOUBAO_ACCESS_TOKEN and DOUBAO_APP_ID. Edit config/cyberverse.yaml for voice-only operation:

inference:
  avatar:
    enabled: false

Then install project and voice-agent dependencies:

make setup
pip install -e ".[all]"

How do you use this agent?

Start the services in three terminals:

conda activate cyberverse
make inference

make server

make frontend

Verify the API:

curl -s http://localhost:8080/api/v1/health

Open http://localhost:5173 in a browser. For local digital-human video, install the documented CUDA 12.8+ PyTorch environment and model weights, set inference.avatar.enabled to true, and configure model paths in config/avatar_models/flash_head.yaml or config/avatar_models/live_act.yaml.

FAQ