Summary

SpatialLM is a 3D large language model designed to process 3D point cloud data and transform unstructured geometric information into structured, semantically rich 3D scene representations. It can identify and categorize architectural elements such as walls, doors and windows, providing oriented bounding boxes with precise semantic labels. Unlike earlier methods that require specialized hardware for data acquisition, SpatialLM supports diverse input sources including monocular video sequences, RGBD images and LiDAR sensors.

Key Features

Processes 3D point clouds from monocular video, RGBD images and LiDAR sensors
Generates structured 3D scene understanding, including walls, doors and windows
Produces oriented object bounding boxes with semantic categories
Reconstructs 3D point clouds from RGB video using MASt3R-SLAM
Outputs convertible to 3D layouts, 2D floorplans and industry-standard IFC files
Trained on a large-scale photorealistic dataset of 12,328 indoor scenes
Standard multimodal LLM architecture fine-tuned from open-source LLMs
Open-source models, code and test set for research use

About SpatialLM


The model uses a point cloud encoder that compacts dense spatial data into feature vectors, which a large language model then processes to generate structured scene codes. These codes can be converted into 3D oriented bounding boxes, 2D floorplans and industry-standard IFC files, facilitating integration with architectural and engineering workflows. It is trained on a large-scale photorealistic dataset of 12,328 indoor scenes with ground-truth 3D annotations, and it incorporates SLAM techniques to reconstruct 3D point clouds from RGB videos.


SpatialLM advances spatial reasoning for applications in embodied robotics, autonomous navigation and complex 3D scene analysis, where detailed semantic outputs help autonomous systems understand and interact with their environments. Accepted at NeurIPS 2025, it is released open-source with pretrained models, code and a test set, encouraging collaborative research and deployment across robotics, architecture and spatial computing.

Get more likes & reach the top of search results by adding this button on your site!

Embed button preview - Light theme
Embed button preview - Dark theme
TurboType Banner

Subscribe to the AI Search Newsletter

Get top updates in AI to your inbox every weekend. It's free!