Hao Shi 史 昊

02

Evidence Attribution

Multi-Talker & Speaker-Grounded Speech

Multi-Talker ASR · Speaker Grounding · Separation / TSE

How can speech models distinguish, attribute, and preserve acoustic evidence from overlapping speakers?

In meetings and multi-party dialogue, a system can produce fluent text and still mix content across speakers, miss a weak target speaker, or hallucinate who said what. This work uses serialized output training (SOT) and asks how each part of the output can stay tied to the right talker's evidence.

Research Questions

  1. How should talker-specific evidence be extracted from a mixture and organized so that it matches a serialized output?
  2. Is richer acoustic content in the prompt enough, or does the decoder need access to evidence throughout generation?
  3. Can the semantic priors of LLMs be retained without the cost and fragility of autoregressive decoding?

Research Progression

  1. 2024

    Extracting speaker-specific acoustic evidence

    SOT is not easy to train with attention loss alone. EncSep inserted a separator after the encoder, trained with CTC losses, and GEncSep concatenated the separated streams to guide attention during decoding, reaching more than 12% and 9% relative improvement on the noisy Libri2Mix and Libri3Mix evaluation sets.

    IEEE SLT 2024

  2. 2025

    Representing evidence for language models

    With an LLM decoder, SOT worked well for two talkers but did not fully use the LLM for three. Serialized output prompts used a separator with serialized CTC to extract each talker's content in speaking order, then gave the decoded text to the LLM as a structured prompt. This improved both conditions, although the evidence is still supplied before generation begins.

    IEEE ASRU 2025

  3. 2026

    Maintaining access to acoustic evidence

    This study first tested whether a static prefix could be enriched with CTC tokens, hybrid prompts, or continuous talker representations. The results suggested that acoustic content alone does not fully address the conditioning bottleneck. Talker-specific representations were therefore kept as serialized acoustic memory in onset order, which LLM layers query through gated residual cross-attention, with consistent improvements on LibriMix.

    IEEE TASLP 2026

  4. 2026
    Parallel approach

    Exploring alternatives to autoregressive decoding

    This work does not extend acoustic memory. It addresses the cost and fragility of an autoregressive LLM decoder instead. An adapted LLM acts as a teacher signal during training and distills semantic guidance into the encoder, while inference keeps fast CTC-style decoding. A Talker-Count Head selects the decoding branch for variable numbers of talkers. On LibriMix, the model was comparable to LLM-based systems with two talkers and significantly better with three.

    INTERSPEECH 2026

The first three studies address successive questions about how evidence reaches the decoder. The fourth is a parallel approach. All four are separate studies that share a benchmark and a separator design, not stages of one pipeline.

Key Insights

  • The talker count is where LLM decoding strains. Across these studies, systems that do well with two talkers need more support with three. IEEE ASRU 2025; IEEE TASLP 2026; INTERSPEECH 2026
  • How evidence is exposed matters as well as what it contains: effective LLM-based multi-talker ASR appears to depend on persistent access to evidence structured according to the serialized output. IEEE ASRU 2025; IEEE TASLP 2026
  • Explicit separation after the encoder has remained part of these systems as the decoder changed, from attention-based SOT to LLM prompting and encoder-only models. IEEE SLT 2024; IEEE ASRU 2025; INTERSPEECH 2026

Open Questions

  • How do these approaches behave on real conversations rather than on mixtures such as LibriMix?
  • How should the binding between speakers and content be evaluated beyond WER and DER?
  • Can a speech LLM represent who said what explicitly, so that a wrong attribution can be detected?

I am co-organizing the ICASSP 2027 special session Speaker Grounding in Speech LLMs, which covers recognition, attribution, and understanding in multi-talker scenes, and how speaker-content binding should be evaluated beyond WER and DER.

Selected Publications

All 4 papers on this topic