Vox-Infinity

Abstract

Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient information. This issue is even more pronounced in the speech domain, where audio, as a low-compression modality, requires substantially more embeddings than text to preserve both semantic content and acoustic cues. To address this challenge, we introduce Vox-Infinity, the first benchmark specifically designed to evaluate long-context understanding in spoken language models. Vox-Infinity systematically extends audio history along two dimensions: turn count and turn duration. It covers a diverse range of representative scenarios with varying interaction structures and semantic complexity. Crucially, Vox-Infinity provides explicit answer-provenance annotations and organizes samples according to the amount of historical context required to resolve each query, enabling precise and length-aware evaluation. Extensive evaluations of seven representative spoken language models reveal a clear overall recency effect: models generally achieve higher accuracy when answer-supporting evidence is closer to the query, but struggle to retrieve and use evidence located farther back in the dialogue history.

Three Long-Context Spoken Scenarios

Vox-Infinity scales dialogue history along turn count and turn duration across three complementary scenarios, each targeting a distinct long-context capability.

Fine-grained retrieval

(a) Ultra Multi-Turn Dialogue

Conversations with 20+ turns (from SpokenWOZ). Frequent back-and-forth exchanges that require retrieving exact slot values, e.g. phone numbers and train IDs, from dense histories.

Multi-hop reasoning

(b) Personal Monologue

Single-speaker speeches longer than two minutes (from GigaSpeech). Questions require multi-hop reasoning over evidence distributed throughout long-form speech.

Paralinguistic memory

(c) Beyond-Semantic Dialogue

Extended conversations (from IEMOCAP + emotion-controlled TTS) that require retaining and retrieving emotions conveyed through vocal expression, not text alone.

Key Statistics

1,474

Sessions

13,516

QA pairs

127.96 h

Total audio

312.5 s

Avg. session

36.38

Avg. turns

7

Models tested

Benchmark	Sessions	QAs	Dur. (h)	Avg. Dur. (s)	Avg. Trns.	Provenance
VoxDialogue	4,526	4,526	42.56	33.9	6.70	✗
ContextDialog	653	2,612	12.50	68.9	10.60	✗
LA-Bench	2,429	2,429	79.12	117.3	1.00	✗
BLAB	1,176	1,600	833.0	2,550.0	1.00	✗
Vox-Infinity (ours)	1,474	13,516	127.96	312.5	36.38	✓
— Ultra Multi-Turn	991	7,086	47.10	171.1	35.89	✓
— Personal Monologues	159	1,050	52.83	1,196.1	1.00	✓
— Beyond-Semantic	324	5,380	28.03	311.4	55.23	✓

Table 1. Comparison with existing long-context audio benchmarks. Vox-Infinity is the first to provide explicit answer-provenance annotations and multiple required-context-duration bins (0–2, 2–4, 4–8, 8+ min).

Dataset scale comparison — Dataset-scale comparison by average turns and average session duration (marker size = total hours).

Required history duration distribution — Distribution of required history durations across the three subsets (50 s bins, log count).

Word cloud of common topics — many queries target concrete, verifiable slots (phone numbers, train IDs, restaurants).

Main Findings

We benchmark seven spoken language models — Qwen3-Omni, GLM-4-Voice, AudioFlamingo3, AudioFlamingo Next, MiMo-Audio, Gemini-3.1-Pro and Qwen3.5-Plus — with dialogue history given as either text or audio.

Recency curves — Fine-grained accuracy under audio history across 30 s required-context-duration intervals. Thick gray curves = model average. Despite local fluctuations, both subsets show an overall recency effect.

Model	Ultra Multi-Turn		Personal Monologues		Beyond-Semantic
Model	Text	Audio	Text	Audio	Text	Audio
Qwen3-Omni	84.8	55.0	85.5	70.5	46.2	53.2
GLM-4-Voice	59.4	34.8	27.6	13.2	41.0	22.3
AudioFlamingo3	43.4	34.4	49.0	44.0	18.5	25.5
AudioFlamingo Next	74.9	32.2	62.2	45.4	41.2	46.7
MiMo-Audio	76.6	22.5	67.7	12.5	30.0	40.3
Gemini-3.1-Pro^†	79.0	62.8	82.6	60.1	53.2	57.5
Qwen3.5-Plus^†	81.1	56.8	84.4	64.6	52.9	58.1

Table 2. Average accuracy (%) per subset under text vs. audio history. Higher of the two is highlighted; ^† marks closed-source models.

Overall recency effect

Accuracy declines as supporting evidence moves farther from the query. On Ultra Multi-Turn (audio), the model average falls 43.4%→38.2% from 0–2 to 4–8 min; on Beyond-Semantic it drops 46.3%→39.7%→23.7% beyond eight minutes.

Text history: semantics

Compact transcripts better retain precise lexical evidence. On Ultra Multi-Turn, text history beats audio for every model (mean 71.3% vs. 42.6%), where questions need exact slot values.

Audio history: prosody

Transcripts discard intonation and intensity. On Beyond-Semantic, audio history outperforms text for six of seven models, preserving paralinguistic emotion cues unavailable in text.

Citation

@inproceedings{voxinfinity2027,
  title     = {Vox-Infinity: Benchmarking the Limits of Long-Context
               Spoken Language Models},
  author    = {Anonymous},
  booktitle = {Under review},
  year      = {2027},
  url       = {https://vox-infinity.github.io}
}

Cases and datasets are available at https://vox-infinity.github.io.