Architectural Overview
NVIDIA's NemotronLabs VoiceChat 11B model employs a unified network architecture, streamlining speech understanding and generation. This approach eliminates the need for multi-model orchestration and API handoffs, reducing end-to-end latency. The model's ability to listen while speaking enables smooth turn-taking and user interruption handling.
Key Specifications
- Hybrid Mamba/Transformer architecture
- 11B parameter model
- Trained on approximately 550k hours of audio data
- Supports tool calling with a separate output channel for
<TOOLCALL>scripts - Constraints include a maximum of five tools per session and ASCII-only system prompts and tool responses
Performance and Limitations
The model achieves a smooth turn-taking latency of ~450 ms on Full-Duplex-Bench 1.0, with a take-over rate of 1.00 at 480 ms. However, it is not yet suitable for production use due to documented failure modes, including a two-minute audio context ceiling and degradation into non-recoverable gibberish after several turns. The model is available under a permissive license, with weights and container publicly accessible for research purposes.