Hao Shi 史 昊

01

Reliable Perception

Robust Speech Processing

Speech Enhancement · Noise-Robust ASR · Dereverberation

How can reliable acoustic evidence be recovered from noisy, reverberant, and degraded speech?

Enhancement can make a recording sound cleaner and still harm recognition, through the artifacts it adds and the information it removes, and models trained on simulated noise face a mismatch with real recordings. This direction studies how to recover usable speech for both listeners and ASR.

Research Questions

  1. Where are mapping and masking, waveform and spectrogram, and predictive and generative models complementary, and how should they be combined?
  2. Why does better-sounding enhancement not always improve recognition, and how should a front-end be designed for ASR?
  3. How can a pretrained model adapt to real noise without losing what it learned in pretraining?
  4. What can a diffusion-based enhancement model use as a reliable condition when the noisy input is itself unreliable?

Research Evolution

  1. 2020–2022

    Fusing complementary estimates

    Spectrogram fusion combined complementary systems, first for dereverberation with minimum difference masks, then as a jointly trained front-end for end-to-end ASR. Subband fusion located the complementarity: mapping suited low-frequency bands and masking suited high-frequency bands.

    IEEE ICASSP 2020; APSIPA ASC 2021; APSIPA ASC 2022

  2. 2022–2024

    Spectral information in enhancement models

    Multiple-bandwidth spectrograms supplied complementary information, and multi-resolution spectrograms were added to time-domain encoders. WaveSpecEnc refined each waveform encoder layer with spectrogram encoding, which helped perceived quality but gave only a minor ASR improvement. WaveSpecEnc+ therefore also passed this information to the ASR back-end, and the gains on real CHiME-4 data held even with a strong WavLM baseline.

    APSIPA ASC 2022; IEEE ICASSP 2023; IEEE/ACM TASLP 2024

  3. 2024

    Adapting pretrained models to real noise

    An adapter study on CHiME-4 found that shallow layers were the most effective place to adapt, and that real data was more effective than the same amount of simulated data. Dual-path adaptation addressed the feature extraction module, which is usually frozen, by keeping a frozen pretrained path beside a finetuned one, with significant improvements even for noise-robust WavLM models.

    arXiv 2024; INTERSPEECH 2024

  4. 2024–2025

    Diffusion-based enhancement

    Generative enhancement brought its own problems: slow reverse diffusion and unreliable conditioning on noisy features. Joint generative and predictive decoders gave faster decoding and higher PESQ than StoRM and SGMSE+ on Voice-Bank. Ensemble inference exploited the sampling behavior of diffusion, with SplitTree reducing its complexity by about 40% at a similar level of performance. DERDM-SE then used deterministic features as conditions, despite the distortion those models introduce, and gave more stable performance than other diffusion-based models on CHiME-4.

    IEEE ICASSP 2024; IEEE ICASSPW 2024; IEEE TASLP 2025

  5. 2025–2026

    Observation addition between enhancement and ASR

    Observation addition balances noisy and enhanced speech without changing the enhancement or ASR models. Collaborative work trained the module that sets this balance on real noisy speech, and later derived the fusion weights from intelligibility estimates of the ASR back-end, which removes the need for a trained predictor.

    IEEE ICASSP 2025 (co-author); INTERSPEECH 2026 (co-author)

Key Insights

Open Questions

  • When perceptual metrics and recognition accuracy disagree, how should an enhancement system be evaluated for a given downstream use?
  • Can generative enhancement be prevented from producing speech detail that the recording does not support, and how would this be measured?
  • How well do results on benchmarks such as CHiME-4 and Voice-Bank carry over to other real recording conditions?

Selected Publications

All 19 papers on this topic