Apple researchers detect code-switching directly from speech using Whisper-small encoder

Apple researchers Jie Chi, Zakaria Aldeneh, Masha Fedzechkina, Maureen de Seyssel and Natalie Schluter propose detecting code-switching directly from speech using language-selective dimensions in pretrained speech encoders.

With a Whisper-small encoder, the method reaches a mean F1 of 0.798 across 14 language pairs on CS-FLEURS, versus 0.776 for a CNN and 0.751 for a HuBERT linear baseline.

The approach requires no transcription and uses a small code-switching set to tune its decision rule.