Geographic and perceptual bias in multimodal LLMs: evidence from a global dataset spanning more than 200 cities
Abstract
Multimodal large language models (MLLMs) are increasingly deployed for urban informatics applications, from objective built environment attribute extraction to subjective assessments. However, do MLLMs apply the same standards across regions while executing these urban-related tasks? Are MLLMs reliable and neutral judges? Do they adapt without criticizing, raising concerns about equitable deployment and further hidden bias? Here, we present a geographic red-teaming framework for diagnosing geographic biases in MLLMs applied to image object detection and urban visual perception. Using annotated imagery from more than 200 cities as a standardized benchmark, we demonstrate that MLLMs exhibit significant geographic disparities in both object detection accuracy and perceptual assessments across different regions. Quantitatively, perceptual scores for attributes such as Wealthy and Beautiful dropped by an average of 80% for African cities after geo-referencing, whereas regions such as Asia and North America showed score increases of 26%. A spatial-scale sensitivity test further showed that coordinates, city, country, and continent information produced broadly similar directional shifts, while combined geographic cues generated the strongest perceptual changes. Among the multiple state-of-the-art models, GPT-4o showed the lowest perceptual bias. Our geographic red-teaming stress-tests MLLM performance across underrepresented urban environments, revealing consistent patterns of bias that mirror training-data imbalances. We also provide quantitative metrics for measuring geographic equity in model outputs and establish recommendations for responsible MLLM deployment in urban research. These findings highlight critical limitations in current multimodal AI systems and demonstrate the urgent need for geographically inclusive model development to prevent the perpetuation of urban inequalities through automated analysis systems.
Matias Quintana
Research Fellow
