SpatialLM

SpatialLM: Large Language Model for Spatial Understanding

Introduction

SpatialLM is a cutting-edge 3D large language model developed for processing 3D point cloud data and generating structured outputs that enhance understanding of spatial environments. The model is integral for applications that involve interpreting and reconstructing three-dimensional layouts based on point cloud inputs, which can arise from numerous sources such as monocular video sequences, RGBD images, and LiDAR sensors. The primary focus of SpatialLM is to bridge the gap between unstructured 3D geometric data and structured 3D representations, contributing significantly to spatial reasoning tasks essential for fields like embodied robotics and autonomous navigation.

Features and Capabilities

SpatialLM distinguishes itself through its ability to comprehensively interpret 3D data and produce detailed outputs, including:

• Identification of Architectural Elements: The model can recognize and classify various components of a space such as walls, doors, windows, and furniture items, along with providing their associated bounding boxes.
• Multimodal Input Handling: Unlike traditional systems that require specific hardware for data collection, SpatialLM accommodates various data formats and types, thus offering flexibility in usage.
• Robust Spatial Reasoning: Its extensive training enables enhanced reasoning abilities when navigating or interpreting complex 3D scenes.

Applications

SpatialLM's architecture allows it to be applied in diverse scenarios, including:

• Embodied Robotics: Facilitating robots that operate in physical spaces to understand their surroundings and navigate efficiently.
• Autonomous Navigation: Supporting vehicles and drones in interpreting environments to make real-time navigation decisions.
• 3D Scene Analysis: Offering insights and interpretations for applications in virtual reality, augmented reality, and even urban planning and architectural design.

SpatialLM Models

The repository hosts different models that cater to varying computational needs:

1. SpatialLM-Llama-1B: A powerful variant designed for comprehensive scene interpretation.
2. SpatialLM-Qwen-0.5B: A lightweight version suitable for less demanding applications.

These models are available on platforms like Hugging Face, allowing for easy testing and deployment.

Getting Started

Setting up SpatialLM involves a few straightforward steps for installation, followed by inference and visualization:

Installation

To begin using SpatialLM, ensure the following environment setup:

• Python 3.11
• Pytorch 2.4.1
• CUDA 12.4

The basic installation procedure includes cloning the repository, setting up a conda environment, and installing dependencies using poetry.

# Clone the repository
git clone https://github.com/manycore-research/SpatialLM.git
cd SpatialLM

# Create a conda environment with CUDA 12.4
conda create -n spatiallm python=3.11
conda activate spatiallm
conda install -y nvidia/label/cuda-12.4.0::cuda-toolkit conda-forge::sparsehash

# Install dependencies
pip install poetry && poetry config virtualenvs.create false --local
poetry install

Running Inference

SpatialLM requires that the input point clouds are aligned on an axis where the z-axis is consistently used as the vertical axis. Example codes for running inference can be found in the repository:

# Run inference
python inference.py --point_cloud <path_to_point_cloud> --output <output_file> --model_path <model_path>

Visualization

To visualize the outputs, users can use specific scripts provided to convert the predicted layouts into Rerun format for effective displays.

SpatialLM Testset and Evaluation

SpatialLM comes with a Testset featuring 107 preprocessed point clouds. This Testset is more complex than previous datasets, containing real-world challenges such as noise and occlusions. Users can benchmark their model's performance against the provided test set using an evaluation script that gives detailed insights into various metrics.

Example Evaluation Steps

1. Download the Testset:

   huggingface-cli download manycore-research/SpatialLM-Testset --repo-type dataset --local-dir SpatialLM-Testset
   

2. Run evaluation:

   python eval.py --metadata SpatialLM-Testset/test.csv --gt_dir SpatialLM-Testset/layout --pred_dir SpatialLM-Testset/pred --label_mapping SpatialLM-Testset/benchmark_categories.tsv
   

Benchmark Results

The performance of the models is benchmark-tested on the challenging SpatialLM-Testset, revealing a range of results across different tasks (such as floorplan and object recognition) and revealing the robustness of SpatialLM in understanding diverse spatial conditions.

Licensing and Citation

Models within SpatialLM are derived from other notable frameworks and are governed under various licenses, including the Llama3.2 license and the Apache 2.0 License for the Qwen series. The community is encouraged to cite the work if utilized in projects or research.

@misc{spatiallm,
  title        = {SpatialLM: Large Language Model for Spatial Understanding},
  author       = {ManyCore Research Team},
  year         = {2025},
  howpublished = {\url{https://github.com/manycore-research/SpatialLM}}
}

Conclusion

SpatialLM represents a significant advancement in spatial understanding through the processing of 3D point data. It is particularly beneficial for developers, researchers, and engineers working in fields that require objective and structured 3D scene analysis. Given its unique capabilities and flexible model availability, SpatialLM is positioned to be an invaluable tool in the ever-evolving landscape of machine learning and spatial intelligence.