Urban Socio-Semantic Segmentation with Vision-Language Reasoning
Yu Wang, Yi Wang, Rui Dai, Yujie Wang, Kaikui Liu, Xiangxiang Chu, Yansheng Li
2026-01-16
Summary
This paper focuses on teaching computers to understand what different areas in cities *mean* based on satellite images, going beyond just identifying physical things like buildings and roads.
What's the problem?
Current computer models are really good at spotting things like buildings or water in satellite images, but they struggle with understanding places defined by what people *do* there, like identifying schools, parks, or shopping centers. These 'socially defined' categories are harder for computers because they aren't just about what something looks like, but what it's used for.
What's the solution?
The researchers created a new dataset called SocioSeg, which includes satellite images paired with maps and detailed labels showing where these socially important places are. They also built a new system called SocioReasoner that works like a human thinking through the problem. It uses both images and descriptions (vision-language model reasoning) and learns through trial and error (reinforcement learning) to figure out what each area represents, reasoning through multiple steps to arrive at the correct answer.
Why it matters?
Being able to automatically identify these socially important areas is crucial for a lot of things, like urban planning, disaster response, and understanding how cities function. This research improves our ability to analyze cities using satellite imagery, providing valuable information for making better decisions about how we live and build our communities.
Abstract
As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic segmentation by vision-language model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel vision-language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Experiments demonstrate our approach's gains over state-of-the-art models and strong zero-shot generalization. Our dataset and code are available in https://github.com/AMAP-ML/SocioReasoner.