Projects

Open-source code and models, grouped by topic. More on GitHub.

Multi-Talker ASR

Research code

LLM-based multi-talker ASR with serialized output training on mixture speech.

Background: Implements the systems of the ASRU 2025 and SLT 2024 papers: an audio mixture of overlapping speakers is encoded by a WavLM front-end, with an optional separator splitting encoder features into per-speaker streams supervised by serialized CTC.

Japanese ASR with reading-aware output

One model, two complementary outputs: actual-pronunciation kana morae and transcripts with inline bracket furigana.

Background: Built on Qwen3-ASR-1.7B. A written form can correspond to different lexical readings in speech, and that distinction is lost under purely orthographic supervision.
Release: The main model stays byte-compatible with stock Qwen3-ASR, so ruby transcription runs on unmodified vLLM and the official qwen-asr toolkit. Mora reading uses the auxiliary CTC branch through the small ruby_asr package.

Co-author · SB Intuitions

A fine-tuned Whisper large-v3-turbo that transcribes Japanese speech directly into katakana sequences.

Background: The ASR component of the Joyo Kanji Yomi Benchmark, developed as part of the Sarashina2.2-TTS project. It is the backbone of the Kana CER metric, which evaluates the kanji-level reading accuracy of Japanese TTS systems.

Iroha Studio · Open model

Expressive Japanese TTS for character dialogue: clone a voice, choose an emotion, adjust its intensity, and explore different takes on the same line.

Background: Built on Fish Audio S2-Pro with supervised fine-tuning and reinforcement learning. Weights are released for research and non-commercial use under the Fish Audio Research License.
Features: Emotion tags with adjustable intensity, mood changes within one generation, voice cloning from a short reference recording, non-verbal sounds such as laughter and sighs, whispering, and kana annotations that specify the reading of names and heteronyms.
Results: As reported on the model card: 85.6% emotion accuracy (emo2vec large) against 17.9% for the original Fish S2 Pro, and a subjective Japanese MOS of 3.85 against 3.47.

Co-author · SB Intuitions

Japanese-centric LLM-based TTS tackling kanji polyphony, released with the Joyo Kanji Yomi Benchmark.

Background: Trained on approximately 361k hours of Japanese and English speech, with targeted data covering all 2,136 Joyo kanji. The benchmark covers their 4,378 readings and is scored with Kana-CER, which compares synthesized speech to reference readings in kana space.