DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation
Yu Zhou, Sohyun An, Haikang Deng, Da Yin, Clark Peng, Cho-Jui Hsieh, Kai-Wei Chang, Nanyun Peng
2025-10-17
Summary
This research investigates how well AI image and video generators understand and respond to different accents and ways of speaking English, specifically dialects. It finds that these AI models struggle with dialects, producing lower quality results when prompts use dialectal words.
What's the problem?
AI models are trained mostly on 'standard' English, like what you'd hear on national news. Because of this, they don't perform as well when given instructions using regional dialects – different ways people speak in different parts of the country or world. The study showed a significant drop in quality, between 32% and 48%, when even just a single word from a dialect was used in the prompt. Simply retraining the AI or rewriting the prompt didn't really fix the problem and sometimes even made the AI worse at understanding standard English.
What's the solution?
The researchers developed a new technique that helps the AI recognize and understand dialectal features *without* forgetting how to understand standard English. They did this by modifying the way the AI processes the input text, essentially adding a component that specifically learns to identify dialect characteristics. This method improved performance on five different dialects, bringing their results up to the same level as standard English, and importantly, didn't negatively impact the AI's ability to understand standard English.
Why it matters?
This work is important because it addresses a fairness issue in AI. If AI models don't understand everyone equally, they could perpetuate biases and not serve diverse populations well. By making AI more inclusive of different dialects, we can ensure that everyone can benefit from these powerful tools, and that the AI doesn't unfairly prioritize or misunderstand certain groups of people.
Abstract
Contact languages like English exhibit rich regional variations in the form of dialects, which are often used by dialect speakers interacting with generative models. However, can multimodal generative models effectively produce content given dialectal textual input? In this work, we study this question by constructing a new large-scale benchmark spanning six common English dialects. We work with dialect speakers to collect and verify over 4200 unique prompts and evaluate on 17 image and video generative models. Our automatic and human evaluation results show that current state-of-the-art multimodal generative models exhibit 32.26% to 48.17% performance degradation when a single dialect word is used in the prompt. Common mitigation methods such as fine-tuning and prompt rewriting can only improve dialect performance by small margins (< 7%), while potentially incurring significant performance degradation in Standard American English (SAE). To this end, we design a general encoder-based mitigation strategy for multimodal generative models. Our method teaches the model to recognize new dialect features while preserving SAE performance. Experiments on models such as Stable Diffusion 1.5 show that our method is able to simultaneously raise performance on five dialects to be on par with SAE (+34.4%), while incurring near zero cost to SAE performance.