Natural Language Camera Movement Understanding

Yuwen Tan¹ · Joey Huang¹ · Jin Huang² · Haoxiang Li² · Boqing Gong¹

¹Boston University ²Pixocial Technology

ECCV 2026

Paper (arXiv) Code Dataset Checkpoints BibTeX

Examples of camera movements in cinematographic practice: rotations (pan, tilt, roll) and translations (boom, dolly, truck).

We introduce ACaM, an extensive benchmark and instruction-tuning framework for evaluating whether vision-language models (VLMs) can recognize fine-grained, atomic camera movements in real-world and synthetic videos. Existing VLMs fail this task in surprising ways — confusing translation with rotation, left with right, optical zoom with physical dolly, and object motion with camera motion. Our fine-tuned VLM-8B outperforms Gemini 3.1 Pro by 10% on real and 11% on synthetic videos, while a substantial gap to human performance remains.

Abstract

Understanding camera movement in natural language is critical for training and evaluating video generation models, among other applications. However, we demonstrate that existing vision-language models (VLMs) fail this task in surprising ways, frequently confusing translation with rotation, left with right, and object movement with camera movement. To address these limitations, we establish natural language camera movement understanding as a standalone research task. We introduce a two-level cinematographic taxonomy and an extensive, atomic benchmark featuring both real and synthetic videos. Furthermore, we curate a large-scale, multi-source training set enhanced by targeted camera movement augmentation. Our fine-tuned VLM-8B outperforms Gemini 3.1 Pro by 10% and 11% on our benchmark's real and synthetic videos, respectively. Despite these gains, a significant gap remains relative to human performance, underscoring the need to promote and facilitate future research on natural language camera movement understanding.

A Two-Level Cinematographic Taxonomy

Drawing on established cinematography literature, we organize camera movement into a two-level taxonomy of 17 fine-grained classes, spanning physical translations, angular rotations, focal-length changes, static shots, and object-centric movements.

Group	Camera Movement Class
Translation	Dolly InMove in, Push in, Move forward	Dolly OutMove out, Pull out, Move backward
	Truck LeftMove to the left	Truck RightMove to the right
	Boom UpPedestal up, Crane up, Move upwards	Boom DownPedestal down, Crane down, Move downwards
Rotation	Pan Left	Pan Right
	Tilt Up	Tilt Down
	Roll Counterclockwise	Roll Clockwise
Focal Length	Zoom In	Zoom Out
Static	StaticStationary, Fixed, No movement
Object-centric Movement	TrackingFollow shot	ArcOrbit

Taxonomy of camera movements with cinematographic terminology and common aliases.

Why Is Camera Movement Understanding Hard for VLMs?

Through preliminary evaluations, we find that VLMs' deficiencies fall under five recurring failure modes. They are insensitive to subtle inter-frame change, confuse physical translation with angular rotation, frequently misinterpret movement direction, struggle to distinguish optical zoom from physical dolly, and conflate object movement with global camera movement.

1. Small movement insensitivity

VLMs misclassify dynamic shots as static, especially under small, low-intensity motion that humans easily detect.

2. Translation ↔ Rotation confusion

Models rely on superficial 2D shifts and confuse a lateral translation (truck) with an angular rotation (pan), and vice versa.

3. Left ↔ Right confusion

Even strong models predict the opposite direction within the same movement category, visible as off-diagonal mass in confusion matrices.

4. Optical Zoom vs. Physical Dolly

Models fail to use parallax cues and misclassify focal-length changes as physical translation, or vice versa.

5. Object motion vs. Camera motion

VLMs follow salient foreground subjects rather than global background cues, mistaking object motion for camera ego-motion.

ACaM: An Atomic Camera Movement Benchmark

ACaM compiles an extensive, atomic benchmark covering both real-world cinematography and synthetic videos. We purposely choose atomic camera movements — one and only one dominant movement per video clip — leaving compound movement combinations for future work.

Real-world videos

We unify several existing motion understanding benchmarks (CameraBench, ShotBench, CineTechBench, MotionBench, FavorBench) and apply tailored filtering to retain single-movement clips. We further enrich underrepresented classes with curated YouTube videos. The assembled benchmark is manually verified by graduate students with cinematography training.

Synthetic videos

ACaM also includes AI-generated videos that mimic the automatic rating of generative models. We feed real-world videos and target movement labels to Gemini 3 Pro to produce structured generation prompts, then pass them to Veo 3.1 for synthesis. Each clip is manually quality-controlled and re-prompted in iterative passes.

Pipeline for generating synthetic videos for each camera movement class, using Veo 3.1.

Dataset statistics

Dataset statistics of ACaM. Top: source composition and duration distribution of real-world videos. Bottom: class-wise distribution of real-world and synthetic videos.

Comparison with existing benchmarks

Benchmark	Real	Syn	#Train	#Test	Yes/No	MCQ	Human	#Labels
Cinematic2K	✓	✗	✗	1,047	✗	✗	✗	11
CameraBench	✓	✗	1,402	1,071	✓	✗	✗	50
ShotBench-Subset	✓	✗	1,058	464	✗	✓	✗	17
CinetechBench-Subset	✓	✗	✗	120	✗	✓	✗	15
MotionBench-Subset	✓	✗	5,000	385	✗	✓	✗	–
FavorBench-Subset	✓	✗	17,000	546	✗	✓	✗	–
ACaM (Ours)	✓	✓	24,321	2,602	✓	✓	✓	17

Comparison with prior camera-movement understanding benchmarks. ACaM is the first to include synthetic videos and human performance.

Training: Multi-Source Data with Targeted Augmentation

We construct an instruction-tuning set of 27K samples, drawn from 45K raw clips spanning five sources: CameraBench, ShotBench, SpatialVID, MultiCamVideo, and GenDoP. We re-balance the long-tailed distribution with class-aware re-sampling and targeted augmentation.

Sources

Dataset	#Extracted Clips	Annotation	Label Source	#Labels
CameraBench	723	Motion caption	Human annotation	16
ShotBench	375	MCQ format	Human annotation	14
SpatialVID	29,733	Movement labels	Estimated poses	12
MultiCamVideo	12,048	Camera pose	Synthetic trajectory	13
GenDoP	2,438	Movement caption	Estimated poses	6
Overall	45,317	MCQ format	Mixed	17

Targeted augmentation for distribution balancing

Targeted data augmentation for distribution balancing.

Top: shift from an unbalanced 45K raw set to a refined 27K instruction-tuning set. Bottom: per-class augmentation operators — progressive cropping for zoom, temporal reversal for dolly, horizontal flipping for truck, and continuous affine rotation for roll.

Results

We supervised fine-tune Qwen3-VL-4B and 8B on our training set and compare to geometry-based models, spatial VLMs, general VLMs (open-source and proprietary), and prior camera-movement specialized VLMs. Numbers below are accuracy (%).

Real-world videos

Model	Static	Rot.	Trans.	Zoom	Arc	Track	Overall	Avg
Random Guess	24.50	24.28	24.00	24.54	24.78	25.00	24.27	24.37
Human	99.00	93.07	90.39	90.83	97.10	96.97	93.44	93.14
Visual Geometry Models
Mega-SaM	89.55	74.94	54.16	–	–	–	–	–
ViPE	61.81	49.71	73.17	–	–	–	–	–
Spatial VLMs
G²VLM-2B	9.50	4.52	4.52	7.44	5.71	7.58	5.66	5.24
Spatial-MLLM	59.00	16.98	31.72	41.67	24.29	77.27	34.22	29.68
VLM-3R-7B	94.50	20.69	28.55	35.83	51.43	89.39	40.22	35.02
General VLMs
Qwen3-VL-4B	93.50	53.47	34.21	61.67	56.52	80.30	53.01	46.96
Qwen2.5-VL-7B	81.00	33.91	34.71	58.33	66.67	92.42	46.86	45.62
Qwen3-VL-8B	85.00	65.84	39.50	61.67	79.71	74.24	58.27	55.25
LLaVa-OV-7B	92.50	28.71	25.79	24.17	62.32	75.76	39.55	34.85
InternVideo2.5-8B	82.50	31.44	28.43	64.17	50.72	92.42	43.51	41.07
InternVL3.5-8B	89.00	48.51	30.58	58.33	40.58	86.36	48.77	46.15
InternVL3.5-14B	81.00	43.56	39.83	63.33	55.07	89.39	51.37	47.21
Qwen3-VL-32B	92.00	64.11	47.93	64.17	57.97	86.36	61.95	58.66
GPT-5	61.50	66.58	54.55	61.67	68.12	80.30	61.20	61.54
Gemini-3-Pro	83.50	64.36	60.84	65.00	82.61	96.97	68.24	66.83
Gemini-3.1-Pro	83.92	63.34	64.00	62.39	76.81	95.38	68.44	67.73
Camera Movement Specialized VLMs
CameraModel-7B	55.50	68.81	47.93	45.83	91.30	98.48	58.88	60.56
ShotVL-7B	91.00	60.40	51.07	32.50	68.12	98.48	60.52	57.09
CamReasoner-7B	76.50	41.58	32.89	59.17	39.13	80.30	45.83	44.99
SFT Qwen3-VL-4B (Ours)	76.00	71.53	64.13	76.67	78.26	93.94	70.83	72.27
SFT Qwen3-VL-8B (Ours)	85.50	70.05	64.13	84.17	84.06	96.97	72.75	74.28

Synthetic videos

Model	Static	Rot.	Trans.	Zoom	Arc	Track	Overall	Avg
Random Guess	25.00	25.00	25.00	25.00	25.00	25.00	25.00	25.00
Human	98.88	97.72	98.04	96.15	100.00	97.62	98.05	97.61
Visual Geometry Models
Mega-SaM	95.53	76.35	61.66	–	–	–	–	–
ViPE	68.16	52.42	78.65	–	–	–	–	–
Spatial VLMs
G²VLM-2B	4.44	2.91	6.99	1.92	5.45	13.10	5.54	4.51
Spatial-MLLM	68.33	29.65	27.07	50.00	43.64	94.05	40.75	36.04
VLM-3R-7B	99.44	18.90	43.45	51.92	47.27	89.29	48.68	38.94
General VLMs
Qwen3-VL-4B	97.77	54.42	42.92	73.08	40.74	72.62	58.02	53.68
Qwen2.5-VL-7B	91.06	52.99	50.54	82.69	61.11	94.05	62.43	58.50
Qwen3-VL-8B	90.50	62.39	45.10	80.77	55.56	83.33	61.92	60.91
LLaVa-OV-7B	96.09	39.60	37.25	36.54	51.85	86.90	51.06	41.59
InternVideo2.5-8B	94.97	44.73	31.15	75.00	27.78	91.67	50.98	46.71
InternVL3.5-8B	96.09	47.29	33.99	67.31	35.19	73.81	51.74	47.73
InternVL3.5-14B	88.27	41.60	47.93	76.92	31.48	86.90	55.47	51.15
Qwen3-VL-32B	91.62	68.95	55.77	82.69	51.85	67.86	67.01	64.17
GPT-5	64.80	69.80	57.95	82.69	53.70	63.10	63.78	65.09
Gemini-3-Pro	89.94	61.54	68.63	73.08	53.70	88.10	70.65	66.27
Gemini-3.1-Pro	92.07	62.00	74.07	69.23	55.56	85.37	73.01	69.58
Camera Movement Specialized VLMs
CameraModel-7B	58.10	62.68	51.63	71.15	88.89	98.81	61.83	68.45
ShotVL-7B	96.09	69.80	63.62	63.46	29.63	98.81	71.33	64.49
CamReasoner-7B	84.92	51.28	42.70	67.31	55.56	77.38	55.81	53.74
SFT Qwen3-VL-4B (Ours)	82.68	74.36	66.67	78.85	68.52	84.52	73.28	74.40
SFT Qwen3-VL-8B (Ours)	94.41	72.36	74.51	82.69	68.52	91.67	78.20	77.44

Ablation: data sampling, augmentation, and LoRA rank

Sampling	Augmentation	LoRA	Model	Real (Avg)	Syn (Avg)
✗	✗	r=64	4B	48.42	58.56
✗	✓	r=64	4B	63.78	72.07
✓	✓	r=64	4B	70.88	75.03
✓	✓	r=128	4B	71.88	75.20
✓	✓	r=256	4B	72.27	74.40
✓	✓	r=256	8B	74.29	77.44

Combining targeted sampling and augmentation lifts synthetic-video accuracy from 58.56% to 75.03%. The 8B model with r=256 yields the best overall result.

Key takeaways

SFT > Specialized 7B

Our SFT Qwen3-VL-4B beats prior camera-movement specialized 7B models by 9–19% on both real and synthetic videos.

SFT 8B > Gemini-3.1-Pro

Our 8B model outperforms Gemini-3.1-Pro by 10% (real) and 11% (synthetic) overall accuracy.

Gap to humans persists

Even the strongest model trails human performance by ~20%, leaving headroom for future research.