Streaming semantic endpoint detection · Full-duplex speech

FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech

Can a voice agent tell the difference between a user who is pausing and a user who is actually done — directly from streaming audio?

Puneet Mathur · Dinesh Manocha

University of Maryland, College Park

Paper Code Model

0.853

TurnBench EOT recall

zero-shot · FP ≤ 0.10

0.6%

chunk false-stop rate

held-out Smart Turn

45.8 ms

per 2.56 s window

single NVIDIA A100

10.3M

trainable parameters

1.7% of total

Illustration contrasting a mid-turn pause with a true endpoint, showing acoustic VAD and semantic VAD decisions. — **Semantic endpointing, not just silence detection.** A pause may occur inside a turn. FD-VAD reasons from the speech observed so far to decide whether the user should continue or yield the floor.

01 · Abstract

Endpoint detection as causal audio-language reasoning

Natural turn-taking in full-duplex voice interaction requires determining from partial speech whether a pause reflects hesitation or a completed conversational intent. Acoustic VAD lacks this semantic information, while cascaded ASR-based endpointing introduces transcription dependence and additional processing stages. FD-VAD formulates semantic endpoint detection as a causal audio-language reasoning task and maps bounded causal audio windows directly to CONTINUE/STOP decisions.

The model combines a frozen speech encoder, a lightweight modality adapter, and a parameter-efficiently adapted language model with a last-chunk training objective for streaming inference. Confidence-gated endpoint commitment controls the interruption–delay trade-off, while boundary-focused hard-negative sampling emphasizes ambiguous regions near true turn boundaries.

02 · Why semantic VAD?

Silence is an acoustic event. Turn completion is a semantic decision.

Acoustic VAD

Detects whether speech energy is present. It can find a silence boundary, but a silence can be a hesitation, a planning pause, or the end of a turn.

streamingno semantic completion

ASR → semantic classifier

Provides lexical semantics, but inserts transcription into the latency-critical turn-taking loop and inherits ASR errors and extra processing stages.

semanticASR dependency

FD-VAD

Consumes causal audio directly and predicts whether the user still holds the conversational floor. No transcript generation is required.

streamingsemanticASR-free

03 · Method

A compact audio-to-LLM semantic endpointer

Every 320 ms, FD-VAD evaluates at most 2.56 s of causal audio, converts WavLM features into a compact sequence of LLM audio tokens, and predicts a single semantic state.

Frozen WavLM encoder

WavLM-base-plus maps raw waveform to frame-level speech features at 50 Hz.

Modality adapter

Two linear layers with ReLU jointly downsample by k = 4 and project speech features into the LLM embedding space.

LoRA-adapted Qwen2.5-0.5B

Adapted audio embeddings are combined with a short instruction prompt. The LLM predicts CONTINUE or STOP.

Last-chunk supervision

Training applies the loss only to the state of the terminal chunk, matching the streaming decision made at the newest observation.

2.56 smaximum causal window

320 msinference stride

50 HzWavLM features

12.5 HzLLM audio tokens

Confidence gating

Control interruption vs. delay

A single erroneous STOP can interrupt the user. FD-VAD can commit STOP only when P(STOP) ≥ τ for K consecutive decisions, exposing an explicit operating-point control.

0.310.520.93→STOP

Boundary-focused sampling

Train harder near ambiguous boundaries

Hard negatives are concentrated near the true endpoint. Training oversamples windows ending within ±2 chunks of the boundary while leaving inference unchanged.

04 · Results

Strong semantic endpointing in-domain and zero-shot in dialogue

FD-VAD combines high endpoint recall with a low premature-stop rate, while remaining causal and streaming.

0.832

Chunk F1

on the held-out smart-turn-v3.1 test set, with only 0.6% false-stop on incomplete-speech chunks.

0.966

Utterance accuracy

balanced across complete and incomplete turns and comparable to the strongest offline semantic classifier.

0.853

TurnBench EOT recall

zero-shot at FP = 0.097, the highest recall among systems satisfying the benchmark FP ≤ 0.10 constraint.

TurnBench dev: zero-shot EOT transfer

Recall under the benchmark false-positive constraint. † exceeds FP ≤ 0.10.

continuous dialogue

System	Category	Recall ↑	FP ↓	Median latency (ms) ↓
VAP	Turn-taking	0.841	0.045	463
ESPnet turn-taking	Turn-taking	0.836	0.074	895
Kyutai semantic VAD	Semantic	0.803	0.100	1024
WavLM-large anchor	Acoustic	0.825	0.100	1017
FD-VAD	Ours	0.853	0.097	1019
OpenAI server VAD	Semantic	0.933	0.563†	281

Selected systems are shown here for readability; see the paper for the complete benchmark table.

Utterance-level semantic endpoint classification

Held-out smart-turn-v3.1. Streaming systems are marked ✓.

isolated utterances

Model	Streaming	Complete	Incomplete	Overall
Energy VAD	✓	0.998	0.680	0.839
VAP	✓	0.728	0.489	0.608
X2-Turn	✓	0.652	0.812	0.733
Whisper + LLM	×	0.860	0.768	0.814
UltraVAD	×	0.920	0.840	0.880
Smart Turn	×	0.966	0.966	0.965
FD-VAD	✓	0.972	0.969	0.976

Selected baselines are shown here for readability; the paper reports the full comparison with 95% bootstrap confidence intervals.

Semantic cues matter

Acoustic termination alone performs poorly on incomplete speech, while FD-VAD maintains balanced endpoint accuracy across complete and incomplete turns.

Streaming need not mean weak semantics

FD-VAD reaches 0.966 utterance-level accuracy while making incremental causal decisions rather than waiting for a completed utterance.

Reliability and speed remain a trade-off

On TurnBench, FD-VAD reaches high zero-shot recall under the FP budget, but its 1019 ms median commit latency is higher than VAP's 463 ms.

05 · Streaming latency

Reason while the user is still speaking

Offline cascades wait for the complete utterance before ASR and semantic inference. FD-VAD updates its endpoint state continuously, so post-speech latency is governed by the streaming decision schedule rather than utterance duration.

Per-window processing 45.8 ms 46.8 ms p90 · 2.56 s causal window · single A100

Operating point 190 ms median endpoint delay · τ = 0.9 · K = 1

False interruption 2.0% at the same confidence-gated operating point

06 · Quickstart

Run FD-VAD on an audio file

The code repository provides a simple inference entry point. Model weights are hosted separately on Hugging Face.

# clone the code
git clone https://github.com/fd-vad/fd-vad.git
cd fd-vad

# install
pip install -r requirements.txt

# download the trained adapter / LoRA checkpoint
huggingface-cli download puneetUMD/fd-vad fd_vad.pt --local-dir .

# run end-of-turn inference
python infer.py path/to/audio.wav

# inspect per-320 ms P(STOP)
python infer.py path/to/audio.wav --show-stream

07 · Citation

BibTeX

@inproceedings{fdvad,
  title     = {FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech},
  author    = {Mathur, Puneet and Manocha, Dinesh},
  booktitle = {Submitted},
  year      = {2026}
}