Hao Shi 史 昊

03

Speech–Language Alignment & Grounding

Speech-Language Modeling

Pronunciation-Aware ASR · Speech Language Models · Speech Generation

How can speech and language models exploit linguistic knowledge while preserving pronunciation, spoken-language properties, and acoustic fidelity?

Speech systems built on text LLMs or supervised with written text inherit assumptions from written language. The shared question here is where those assumptions conflict with speech, mostly in Japanese, where one written form can have several readings. The work has two strands. Pronunciation and acoustic fidelity concerns the signal directly; spoken-language adaptation addresses the mismatch between written and spoken language more broadly.

Research Questions

  1. How should supervision preserve what was actually said when orthographic text does not encode it?
  2. How can pronunciation correctness be measured directly, independently of orthographic variation?
  3. How can speech models be aligned to spoken style and conversational context?
  4. How does the robustness of a text LLM relate to that of the speech language model built on it?

Research Strands

Pronunciation & Acoustic Fidelity

Whether a model's output stays tied to what was actually said.

  1. Q1

    Reading in recognition

    One written form can correspond to different readings. Ruby-ASR uses a span-bound target from which orthography and reading can both be recovered deterministically, since the distinction cannot be recovered reliably by text-only conversion afterwards. This improved reading recovery across five benchmarks.

    arXiv 2026

  2. Q2

    Reading in generation

    Sarashina2.2-TTS faces the same ambiguity in generation. It introduces the Joyo Kanji Yomi Benchmark (2,136 kanji, 4,378 readings) and Kana-CER, and the paper reports state-of-the-art kanji-level reading accuracy.

    arXiv 2026 (co-author)

  3. Related

    LLM priors in multi-talker recognition

    The LLM-based multi-talker papers also belong here, because they test the tension between linguistic priors and acoustic evidence most directly. They are described under Evidence Attribution.

    IEEE ASRU 2025; IEEE TASLP 2026; INTERSPEECH 2026

Spoken-Language Adaptation

How models built on written language adapt to the way language is spoken. This work does not address acoustic grounding directly.

  1. Q3

    Spoken style and context

    Two collaborative studies use preference optimization: one for speech-worthy SpeechLLM output, evaluated on the new SpokenElyza benchmark, and one over discrete units for contextually appropriate emotional expression in conversational TTS, where emotion recognition accuracy on STUDIES rose from 48.45% to 67.14%.

    INTERSPEECH 2026 (co-author); APSIPA ASC 2025 (co-author)

  2. Q4

    Dialect robustness

    This evaluation compares LLMs and speech language models on Japanese dialects, defining robustness as the ratio of performance on dialectal versus standard inputs. Robustness correlated between speech models and their text-based counterparts, and training with dialectal data and fine-tuning the speech encoder each improved it.

    IEEE ASRU 2025 (co-author)

Q numbers refer to the research questions above. The strands developed in parallel rather than as a sequence.

Key Insights

  • What a written target leaves out may not be recoverable from text afterwards, so it has to be represented in the supervision or measured directly, as in the kana-space evaluation of reading. arXiv 2026; arXiv 2026 (co-author)
  • Properties of spoken language can respond to preference optimization where standard training falls short, in both text output and speech output. INTERSPEECH 2026 (co-author); APSIPA ASC 2025 (co-author)
  • Speech models carry over behavior from the text models they are built on, such as written-style output and dialect robustness. INTERSPEECH 2026 (co-author); IEEE ASRU 2025 (co-author)

These are readings across separate studies, not established general rules.

Open Questions

  • Could span-bound supervision extend to other properties of speech that orthography omits, and to languages other than Japanese?
  • How much of a speech language model's dialect robustness comes from the base LLM, and how much from the speech components?
  • How can output that is plausible but unsupported by the acoustics be detected in speech LLMs?

Selected Publications

All 8 papers on this topic