Frontier-Bench v0.1: Performance vs. cost-effectiveness by effort level setting.
PolyAI has unveiled Dialog-RSN-1, an innovative audio-native dialog model designed to revolutionize conversational AI by directly processing a caller's raw audio. Unlike traditional systems that rely on text transcripts, Dialog-RSN-1 integrates turn-taking, speech recognition, function calling, and response generation into a single, cohesive model. This allows it to perceive crucial vocal nuances like tone and hesitation, which are often lost in text-based approaches. Already deployed in live production calls, this technology is currently exclusive to PolyAI customers, with no public API or open weights available yet.
This new architecture addresses significant limitations found in existing conversational AI models. Cascaded systems, which convert speech to text early, discard valuable acoustic cues and often require tedious manual tuning. Conversely, speech-to-speech models, while retaining audio, can bake the voice into the model, limiting pronunciation flexibility and demanding constant GPU resources. Dialog-RSN-1 circumvents these issues by being audio-aware only on input, reasoning over raw audio, and then handing off generation to a separate, promptable text-to-speech system. It operates on demand, intelligently deciding when to respond based on acoustic cues and full context, rather than continuously streaming.
For developers and researchers, Dialog-RSN-1 represents a leap forward in building more natural and efficient voice AI. PolyAI achieved impressive performance by post-training open-weight multimodal models with supervised and reinforcement learning on proprietary data, ensuring the pipeline is base-model agnostic. The model targets sub-300ms latency on A100 GPUs through various optimizations, including prefilling attention caches and speculative drafting. Benchmarking on PolyAI's internal Dialog-Eval (slated for open-sourcing) shows Dialog-RSN-1 outperforming cascaded models and even GPT Realtime in real-time scenarios, demonstrating superior accuracy and transcription quality. This advancement promises more robust and human-like interactions for next-generation voice assistants and customer service solutions.