GST-Bench: Can VLMs Develop Global Spatial Awareness from Video?

Qifeng Zhang1, 2, *, Kaixiang Huang2, Heng Dong1, Huang Fang1, Junting Chen1, 3, *,
Junjie Zhu2, Yonghang Chen2, Zhiyu Zhang2, Wei Li1, †
1ByteDance Seed       2Zhejiang University       3National University of Singapore      
*Work done during internship at ByteDance Seed
†Corresponding authors
Correspondence: liwei.85@bytedance.com
Paper Internal Approval Pending Internal Approval Pending

Examples of GST-Bench

A Top-down option A
B Top-down option B
C Top-down option C
D Top-down option D
Top-down Selection (Easy)

Among the four images below, which one is closest to a top-down view of the entire scene? Please directly answer with the letter of the correct option.

A Top-down option A
B Top-down option B
C Top-down option C
D Top-down option D
Top-down Selection (Medium)

Among the four images below, which one is closest to a top-down view of the entire scene? Please directly answer with the letter of the correct option.

A Trajectory option A
B Trajectory option B
C Trajectory option C
D Trajectory option D
Trajectory Selection

Which of the four red trajectories on the top-down maps best matches the path shown in the video? Note that the blue dot marks the starting point of each trajectory. Please directly answer with the letter of the correct option.

Current View Current view for egocentric direction
Egocentric Direction (Visual)

In which direction relative to you is the object marked with a red box in the video? Taking your current viewing direction as 0°, with left turns being positive, please only give the angle as an integer between –180° and 180°, without the "°" symbol.

Current View Current view for egocentric distance
Egocentric Distance (Visual)

How far in meters is the object marked with a red box in the video from you? Answer only with a number rounded to one decimal place, without the "m" suffix.

Top-down Image Top-down image for self localization
Current View Current view for self localization
Global Position (Self Localization)

Where on the top-down image does your current location correspond to? Please provide coordinates normalized to the range [0, 999], with the x-axis pointing to the right of the top-down image and the y-axis pointing to the bottom of the top-down image. Please provide the answer directly in the format <point>x y</point>.

Overview of GST-Bench

Overview of GST-Bench


Figure 1. Overview of GST-Bench. We evaluate global spatial awareness along three core competencies — self localization (Where am I?), object localization (Where is the target?), and scene structure understanding (What does the scene look like?). Given an egocentric exploration video and a novel query view, models must localize themselves, reason about targets unseen in the current view and map observations onto a global top-down representation. As shown on the right, even the strongest VLM (Gemini-3-Pro, 42.68) falls far below the human baseline (79.08).

Abstract

Spatial intelligence is fundamental to embodied agents, yet existing benchmarks focus on local spatial perception from single or few viewpoints, overlooking global spatial awareness over continuous, long-horizon visual streams. To address this limitation, we introduce the Global-Spatial-Temporal Benchmark (GST-Bench), a VQA benchmark for global spatial intelligence in video understanding, comprising human-verified questions derived from 6,790 minutes of synthetically generated video. It requires models to perform accurate spatial inference from novel viewpoints unseen in the input video and to map egocentric observations onto global top-down images. A comprehensive evaluation of 22 state-of-the-art VLMs exposes a striking gap between models and humans: the strongest zero-shot model attains only 42.68, far below the human score of 79.08. To probe the cause of this gap, we construct GST-Bench-Local and find that models, despite strong local spatial understanding under the same task formulation, still fail to consolidate long-horizon observations into a globally consistent scene representation. We further provide GST-Train, a dataset for global spatial reasoning, as a complementary resource to facilitate future research on this challenge.

Tasks of GST-Bench

Tasks of GST-Bench


Figure 2. Representative GST-Bench samples from each of the twelve task types, organized by the three core competencies: self localization, object localization, and scene structure understanding.

Data Construction

GST-Bench is generated from diverse simulation scenes through a controlled pipeline that produces exploration videos, object-annotated videos, current-view images, top-down maps, ground-truth spatial annotations, and template-based QA pairs. Automated filtering and human verification are used to remove low-quality cases and keep the questions grounded in global spatial reasoning.

Data construction pipeline


Figure 3. Overview of the automatic data generation pipeline. Starting from diverse simulation scenes, the pipeline produces exploration videos, object-annotated videos, novel-viewpoint images, top-down images at three abstraction levels, and template-based QA pairs, followed by automated filtering and human verification.

Benchmark Results

We evaluate 22 state-of-the-art VLMs across proprietary, open-source, and embodied-understanding model families. GST-Bench exposes a large human-model gap: the strongest zero-shot model, Gemini-3-Pro, reaches 42.68, while the human baseline reaches 79.08.

Performance of evaluated models on GST-Bench


Table 1. Performance of various models on GST-Bench across three core competencies: Object Localization, Self Localization, and Scene Structure Understanding. The Avg. is the arithmetic mean of all 12 subtask scores. The highest, second-highest, and third-highest scores in each column are marked in light red, light orange, and light yellow, respectively, excluding Qwen3-VL-8B (fine-tuned), which is fine-tuned on our proposed training set GST-Train. ED = Ego. Direction, EDist = Ego. Distance, GP = Global Position, Pos = Top-Down Position, Ori = Top-Down Orientation, TDS = Top-Down Selection, Traj = Trajectory Selection. v / s = visual / semantic modality.

Key Findings

  • 🌟Global spatial awareness remains difficult. Even the best zero-shot model trails humans by 36.4 points on average.
  • 🌟Open-source models are substantially behind. The strongest open-source zero-shot model remains around 30 points, with many models close to random guessing.
  • 🌟Targeted supervision helps. Fine-tuning Qwen3-VL-8B on GST-Train improves the average score from 25.89 to 53.52, surpassing all zero-shot models evaluated here.

Global vs. Local Spatial Reasoning

GST-Bench-Local disentangles local spatial perception from long-horizon global integration. The controlled settings compare the original global task against local video and local image variants where the target appears in the current view or the problem reduces to single-image spatial reasoning.

Controlled local and global settings


Figure 4. Illustration of the three controlled settings used to disentangle global reasoning from local perception: Global, Local-Video, and Local-Image.


Performance comparison under local settings


Table 2. Performance comparison under disentangled local settings for Egocentric Direction and Egocentric Distance. L-Video and L-Image denote Local-Video and Local-Image settings, respectively. Values in parentheses denote absolute gains over the Global setting.

BibTeX

BibTeX will be available after internal approval and public release.