Apple researchers propose speech-based code-switching detection with Whisper-small encoder
Overview
Apple researchers Jie Chi, Zakaria Aldeneh, Masha Fedzechkina, Maureen de Seyssel and Natalie Schluter propose a lightweight method that detects code-switching, the mixing of languages within speech, directly from audio without transcription.
The method uses language-selective hidden-state dimensions in pretrained speech encoders and a small code-switching set to tune its decision rule.
In the paper's experiments on CS-FLEURS, the Whisper-small encoder reaches a mean F1 score of 0.798 across 14 language pairs. That compares with 0.776 for a CNN and 0.751 for a HuBERT linear baseline trained for monolingual language identification.
Written by AI from the articles below · updated Oct 14, 12:31 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Apple Machine Learning ResearchOfficialApple researchers detect code-switching directly from speech using Whisper-small encoder
AIApple researchers Jie Chi, Zakaria Aldeneh, Masha Fedzechkina, Maureen de Seyssel and Natalie Schluter propose detecting code-switching directly from speech using language-selective dimensions in pretrained speech encoders. With a Whisper-small encoder, the method reaches a mean F1 of 0.798 across 14 language pairs on CS-FLEURS, versus 0.776 for a CNN and 0.751 for a HuBERT linear baseline. The approach requires no transcription and uses a small code-switching set to tune its decision rule.
Heat trend
Not enough continuous observations to show a trend yet.