NVIDIA Releases Nemotron 3 Diarization: A 100M-Parameter Open-Weight Model That Tracks 8 Speakers in Real Time

NVIDIA has released Nemotron 3 Diarization, an open-weight speaker diarization model on Hugging Face. It answers one question about any conversation: who spoke when. The 100M-parameter model tracks up to 8 speakers, including when voices overlap. One checkpoint handles both offline recordings and real-time streaming.

Is it deployable? Yes. The weights are released under the OpenMDW License 1.1, which permits commercial use. It runs on Linux through NVIDIA NeMo, using Ampere, Ada Lovelace, Hopper, or Blackwell GPUs.

Why Speaker Diarization?

Automatic speech recognition (ASR) gives you the words. It does not tell you who said them. Without attribution, a summarizer cannot tell who made a commitment or who raised an objection.

Diarization outputs the time intervals where each speaker is active. Those timestamps combine with ASR output to produce a speaker-attributed transcript. Meeting tools, call analytics, podcast pipelines, and voice-agent memory all depend on this step.

What Changed From Streaming Sortformer

NVIDIA’s earlier Streaming Sortformer checkpoint, diar_streaming_sortformer_4spk-v2.1, supported 4 speakers. Nemotron 3 Diarization doubles that limit to 8. According to NVIDIA’s announcement, the target is messy multi-party audio where people talk at once.

How the Architecture Works

The model accepts 16 kHz, single-channel audio in .wav, .flac, .opus, or .mp3 format. It converts the audio into Mel-spectrogram features with a 10 ms step. The features are stacked by a factor of 8, which produces 80 ms encoder frames.

A 31-layer Transformer encoder with rotary positional embeddings (RoPE) processes those frames. A Conv1D layer then upsamples the predictions back to 10 ms resolution. The output is a [T, 8] tensor of per-speaker activity probabilities.

This design handles overlap directly. If 2 people talk at the same time, 2 channels activate in the same frame.

The model follows the Sortformer approach of ordering speakers by arrival time. The first new voice takes channel 1, the next takes channel 2, and so on. This keeps labels stable across streaming chunks, so the model does not have to re-match speakers to channels for every chunk.

Streaming uses 2 memory mechanisms. The Arrival-Order Speaker Cache (AOSC) keeps speaker information from earlier chunks. A FIFO queue supplies recent frame context. The labels are anonymous, and mapping them to real identities is left to downstream applications.

4 Latency Operating Points

Input-buffer latency equals (chunk + right context) × 80 ms. The table uses DIHARD III full-set DER and batch-32 compiled throughput from the model card.

This latency excludes compute, networking, and ASR time. The model can technically run with an 80 ms buffer, but 0.32 s is the lowest recommended setting.

Benchmark Results

In Voice Arena’s initial Diarization-Bench results, the model ranked first among 12 systems and 17 configurations. The test covered 139 English conversations totaling about 22 hours. It scored 14.72% DER against 19.3% for the next-ranked system, roughly a 24% relative reduction. NVIDIA notes these results may change once Voice Arena completes its Version 1 evaluation.

Against the 4-speaker baseline at 1.04 s latency, DER dropped on all 8 evaluation conditions. Relative reductions ranged from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM. The unweighted mean across the 8 conditions was 41.0%.

There is one regression. On 2-speaker CALLHOME at 30.4 s, DER rose from 5.68% to 5.98%. Full-set CALLHOME-Part2 still improved from 10.32% to 9.10%.

Throughput also jumped. At 30.4 s, the model reached 15,113× RTFx versus 2,619× for the baseline. The tests used BF16 on an NVIDIA RTX PRO 5000 with torch.compile(). These are batched numbers, not single-stream application latency.

Training Data

Training combined about 10,000 hours of real conversations with 82,611 hours of simulated multi-talker mixtures. The mix included real-world multi-speaker audio licensed from David AI. Adding the David AI data cut compound DER from 11.19% to 10.42%. The licensed source audio for the simulated mixtures spans 21 languages.

Getting Started

Install NVIDIA NeMo Speech with Python 3.12 or later:

uv pip install 'nemo-toolkit[asr]'

from nemo.collections.asr.models import SortformerEncLabelModel
diar_model = SortformerEncLabelModel.from_pretrained("nvidia/Nemotron-3-Diarization")
diar_model.eval()
segments = diar_model.diarize(audio=["conversation.wav"], batch_size=1)

Output segments take the form start end speaker_id. To get the words as well, pair the model with Parakeet TDT 0.6B v3 using the ASR integration guide.

The live demo Space offers synthetic conversations, a live mic, a multilingual live mic, and audio upload. For production, NVIDIA lists Baseten and DigitalOcean. On-device support is available through Argmax Pro SDK 3. The model is not yet available through Hugging Face Inference Providers.

The model has limits. Recordings with more than 8 speakers can produce missed or misassigned speech. Heavy noise, reverberation, and far-field capture can also raise error rates.

Interactive Explainer

FIFO

chunk

right

‘;
function lat(i){var c=L[i],d=buf.children,vals=[c.sc,c.fi,c.ch,c.rc];
for(var k=0;k1?’66px’:’0′)}
d[0].textContent=”cache “+c.sc;d[1].textContent=”FIFO “+c.fi;d[2].textContent=”chunk “+c.ch;d[3].textContent=”right “+c.rc;
document.getElementById(‘n3dL’).textContent=c.lat;document.getElementById(‘n3dD’).textContent=c.der;document.getElementById(‘n3dR’).innerHTML=c.rt+’×’;
document.getElementById(‘n3dCfg’).textContent=c.n+’: (‘+c.ch+’ + ‘+c.rc+’) x 80 ms=”+c.lat+”. Cache update period ‘+c.up+’ frames. Block widths are frame counts; small blocks are enlarged for visibility.’}
var ls=document.getElementById(‘n3dLat’);ls.oninput=function(){lat(+ls.value)};lat(1);

/* slide 4 */
var B=[[“DIHARD III”,19.60,13.18],[“CALLHOME-Part2”,11.31,10.29],[“AliMeeting Near”,12.47,6.59],[“AliMeeting Far”,15.58,10.80],[“AMI MHM”,16.36,9.48],[“AMI SDM”,21.73,12.80],[“NOTSOFAR1 MHM”,22.12,7.70],[“NOTSOFAR1 SC”,31.81,12.77]];
var bars=document.getElementById(‘n3dBars’),bi=document.getElementById(‘n3dBarInfo’),rows=[];
B.forEach(function(r){var red=(r[1]-r[2])/r[1]*100;var d=document.createElement(‘div’);d.className=”br”;
d.innerHTML=’‘+r[0]+’-‘+red.toFixed(1)+’%‘;
d.onclick=function(){rows.forEach(function(x){x.classList.remove(‘sel’)});d.classList.add(‘sel’);bi.textContent=r[0]+’ at 1.04 s: Streaming Sortformer 4spk v2.1 ‘+r[1].toFixed(2)+’% DER vs Nemotron 3 Diarization ‘+r[2].toFixed(2)+’% DER, a ‘+red.toFixed(1)+’% relative reduction.’};
bars.appendChild(d);rows.push(d)});
function growBars(){rows.forEach(function(d,i){var a=d.querySelector(‘.a’),b=d.querySelector(‘.b’);a.style.width=”0″;b.style.width=”0″;
setTimeout(function(){a.style.width=(B[i][1]/32*100)+’%’;b.style.width=(B[i][2]/32*100)+’%’},60+i*70)})}

/* slide 5 */
var M=document.getElementById(‘n3dM’),F=document.getElementById(‘n3dF’),Cn=document.getElementById(‘n3dC’);
function der(){var m=+M.value,f=+F.value,c=+Cn.value;document.getElementById(‘n3dMv’).textContent=m+’ s’;document.getElementById(‘n3dFv’).textContent=f+’ s’;document.getElementById(‘n3dCv’).textContent=c+’ s’;
document.getElementById(‘n3dBm’).style.width=m+’%’;document.getElementById(‘n3dBf’).style.width=f+’%’;document.getElementById(‘n3dBc’).style.width=c+’%’;
document.getElementById(‘n3dDer’).textContent=(m+f+c).toFixed(1)+’%’}
[M,F,Cn].forEach(function(x){x.oninput=der});der();

window.addEventListener(‘load’,post);setTimeout(post,300);setTimeout(post,1200);
if(window.ResizeObserver)new ResizeObserver(post).observe(R);
})();