NVIDIA Releases NemotronLabs VoiceChat 11B: An Open Full-Duplex Speech-to-Speech Model with ~450 ms Turn-Taking and Live Tool Calling

NVIDIA has released NemotronLabs VoiceChat 11B, an open 11B end-to-end speech-to-speech model for real-time, full-duplex conversation. Instead of chaining ASR, an LLM, and TTS, it performs streaming speech understanding and speech generation in one unified network. That removes the multi-model orchestration and API handoffs a cascaded stack requires, and cuts end-to-end latency: measured smooth turn-taking latency is 448 ms on Full-Duplex-Bench 1.0. The model listens while it speaks, so a user can barge in mid-turn and the agent yields, with a take-over rate of 1.00 at 480 ms. It is also first open full-duplex model to support tool calling while conversation keeps flowing, using a separate output channel for scripts along with operator-defined “on-hold” lines that fill the gap while an API runs.

Is it deployable?

PARTIAL — deployable today for pilots, not for production. Weights and container are both public, and the license is permissive. But NVIDIA team states the checkpoint is ‘ready for research purposes only,’ and the repo documents real failure modes: a two-minute audio context ceiling, degradation into non-recoverable gibberish after several turns, runaway self-talk after a turn ends, and dropped words in user transcription.

Which companies: any team that can allocate one GPU with at least 80 GB of VRAM — A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux. That covers AI-native startups, funded scaleups, enterprise R&D and innovation labs, GPU cloud providers, and university speech groups. There is no hosted API and no inference provider currently serves the model, so teams without GPU access may not evaluate it.
Industries: contact centers and CX platforms, automotive in-cabin assistants, retail and drive-thru ordering, telecom IVR modernization, games and NPC dialogue, and accessibility tooling.
Applications: barge-in-capable voice agents, voice front-ends over internal APIs, live-lookup assistants (weather, pricing, order status), and duplex latency benchmarking harnesses.

Architecture

The model is a hybrid Mamba/Transformer, assembled from three existing NVIDIA components along with one new output path:

A Fast Conformer speech encoder from Nemotron-Speech-Streaming-En-0.6b, which encodes the incoming 16 kHz stream continuously.
The NVIDIA Nemotron Nano v2 LLM backbone, which consumes audio tokens and predicts text tokens.
An NVIDIA TTS decoder and codec that predicts audio codes, rendered as 22.05 kHz agent speech.
A separate output channel dedicated to tool-calling scripts.

Outputs include agent audio, agent text, and a running user transcription. Training used roughly 550k hours of audio across real and synthetic corpora, building on SALM-Duplex and Audio Flamingo 3.

Tool calls are emitted on the side channel as a block; your code returns results in a block. The notable piece is the on-hold message: per tool, an operator defines a line the agent speaks the moment the model generates the text triggering the call, so the conversation does not fall silent while an API runs.

Constraints are explicit. NVIDIA recommends a maximum of five tools per session, the model cannot reliably call multiple tools simultaneously, and the user cannot interrupt the agent during tool execution. System prompts and tool responses must be ASCII-only and TTS-friendly.

Performance

On Full-Duplex-Bench 1.0: smooth turn-taking TOR 0.82 at 448 ms, user-interruption TOR 1.00 at 480 ms, and pause-handling TOR of 0.153 (synthetic) and 0.255 (Candor), where lower is better.

On AU Harness BFCL-v3 spoken tool calling: 58.5% simple, 62.5% multiple, 42.5% parallel, 27.5% parallel-multiple, 89.6% irrelevance, 56.1% average. On Full-Duplex-Bench v3: 82.5% tool selection, 44.2% argument accuracy, 33% pass@1.

NVIDIA reports the model ranks #2 among open full-duplex models on VoiceBench and #2 among open models on Full-Duplex-Bench 1.0.

Interactive explainer

<![CDATA[<![CDATA[<![CDATA[<![CDATA[ steps‘;},s[1]);}); at(function(){$(‘#c2’).textContent=”agent speaking”;},1250); at(function(){$(‘#c1’).textContent=”still in the pipeline”;},1400); at(function(){$(‘#c1’).textContent=”agent speaking”;},2900); }; /* pane 1 */ var D=[ ‘Fast Conformer speech encoder. Taken from Nemotron-Speech-Streaming-En-0.6b. It encodes the incoming 16 kHz stream continuously, so the model is listening while it talks rather than after the user stops.’, ‘Nemotron Nano v2 LLM backbone. A hybrid Mamba/Transformer network. It consumes audio tokens and predicts text tokens for the agent reply, plus a running transcription of the user.’, ‘NVIDIA TTS decoder and codec. Turns predicted text tokens into audio codes, which are rendered as 22.05 kHz agent speech. This replaces a separate TTS service in the stack.’, ‘Separate tool-calling channel. Tool-call scripts are predicted on their own output channel, so a function call does not have to be squeezed into the spoken reply.’]; var order=[‘0′,’1′,’2′,’3’]; $$(‘.vnode’).forEach(function(b){b.onclick=function(){ $$(‘.vnode’).forEach(function(x){x.classList.remove(‘on’)});b.classList.add(‘on’); $(‘#p1d’).innerHTML=D[+b.dataset.n];ping();}}); $(‘#p1run’).onclick=function(){clr();$$(‘.vnode’).forEach(function(x){x.classList.remove(‘lit’)}); order.forEach(function(n,i){at(function(){$(‘.vnode[data-n="’+n+’"]’).classList.add(‘lit’);},i*420);});}; $(‘#p1reset’).onclick=function(){clr();$$(‘.vnode’).forEach(function(x){x.classList.remove(‘lit’);x.classList.remove(‘on’)}); $(‘#p1d’).innerHTML=’Tap a block. The whole path is one 11B model — no ASR, LLM and TTS services to orchestrate, and no API handoffs between them.’;}; /* pane 2 */ var W=$(‘#p2w’);for(var i=0;i<44;i++){var d=document.createElement('div');d.className="vbar";W.appendChild(d)} var bars=$$('.vbar',W),an=null,t0=0,cut=false; function stopA(){if(an){clearInterval(an);an=null}} function idleBars(){bars.forEach(function(b){b.style.height="10%";b.className="vbar"})} $('#p2go').onclick=function(){clr();stopA();cut=false;t0=Date.now(); $('#p2st').textContent="agent speaking";$('#p2clk').innerHTML='— yield time‘; $(‘#p2cut’).disabled=false; an=setInterval(function(){bars.forEach(function(b){b.className=”vbar act”;b.style.height=(18+Math.random()*78)+’%’})},95); at(function(){if(!cut){stopA();idleBars();$(‘#p2st’).textContent=”turn completed”;$(‘#p2cut’).disabled=true;}},7000); }; $(‘#p2cut’).onclick=function(){if(!an)return;cut=true;$(‘#p2cut’).disabled=true; $(‘#p2st’).textContent=”user cut in — yielding”; bars.forEach(function(b){b.className=”vbar usr”}); at(function(){stopA();idleBars();$(‘#p2st’).textContent=”floor released to user”; $(‘#p2clk’).innerHTML=’480 ms yield‘;},480); }; idleBars(); /* pane 3 */ var st=$$(‘.vstep’,$(‘#p3s’)),msg=[‘User speaking’,’Tool call emitted’,’On-hold line spoken’,’Awaiting tool response’,’Agent resumes’]; function p3reset(){clr();st.forEach(function(s){s.classList.remove(‘on’)});$(‘#p3st’).textContent=”Ready”;} $(‘#p3reset’).onclick=p3reset; $(‘#p3go’).onclick=function(){p3reset(); st.forEach(function(s,i){at(function(){s.classList.add(‘on’);$(‘#p3st’).textContent=msg[i];ping();},i*1100);}); at(function(){$(‘#p3st’).textContent=”Turn complete”;},st.length*1100+700); }; /* pane 4 */ var did=false; function bench(){$$(‘.vbfill’).forEach(function(f){f.style.width=”0″}); setTimeout(function(){$$(‘.vbfill’).forEach(function(f,i){setTimeout(function(){f.style.width=f.dataset.v+’%’},i*70)})},90);did=true} ping(); })(); </script> </div> </body></html>“>]]]]]]]]><![CDATA[><![CDATA[><![CDATA[>]]]]]]><![CDATA[><![CDATA[>]]]]><![CDATA[>]]>

Key Takeaways

One 11B model replaces the ASR → LLM → TTS chain, at 448 ms measured turn-taking latency.
First open full-duplex model with tool calling, using a side channel plus operator-defined on-hold messages.
Weights are OpenMDW-1.1 permissive, but NVIDIA labels the checkpoint research-only.
Requires one 80 GB GPU; no hosted API exists today.

Check out the Hugging Face model card, GitHub (NeMo Speech) and NGC container. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

How To Use Claude’s Voice Mode

Top LLM Observability and Evaluation Platforms in 2026: Langfuse, LangSmith, Braintrust, Arize, and More Compared

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.

Credit: Source link