Researchers tested open-source Whisper speech tools on real-world YouTube videos in seven languages and found that automatic transcripts are far from perfect out of the box. On average the raw transcriptions had an error rate of about 30 percent. With a modest amount of language-specific fine-tuning, the average error rate fell to about 20 percent, making the text much more useful for later analysis.

The team focused on seven languages chosen to support cross-cultural research: Mandarin, Spanish, Korean, Japanese, Russian, Turkish, and Hebrew. They collected hundreds of publicly available YouTube videos using common data tools (yt-dlp and the Jtubespeech toolkit) and filtered for Creative Commons and subtitle availability when possible. The collected sets ranged from a few dozen to several hundred videos per language, and from under 10 hours up to a couple hundred hours of audio depending on the language.

For recognition they used Whisper (a widely used open-source speech model trained on roughly 680,000 hours of audio) and a faster derivative called WhisperX. WhisperX adds speedups and speaker diarization, which means it can separate who is speaking when — a useful feature for analyzing conversations. The authors used the “large-v2” model because it balances accuracy and robustness on noisy, in-the-wild data; they noted that a newer “large-v3” variant can sometimes hallucinate on noisy inputs.

This work matters because many tools for cultural analysis rely on converting spoken language into text before using large language models. Benchmarks that report very low error rates often use clean, scripted speech. In-the-wild video and audio are harder, and cultural signals tend to appear sparsely across many hours, so realistic transcription accuracy and processing speed are critical practical concerns. The study shows that a modest fine-tuning step can make transcripts substantially more usable for downstream tasks.