Temporally Grounded Compositional Camera Motion
Understanding via Geometric Knowledge Distillation
Temporally grounded, compositional camera motion. A single shot is annotated as a sequence of motion-consistent intervals, each carrying all simultaneous movements (top). CamChoreo covers 12 movement types (20 direction-aware labels) grouped into five families: rotation, translation, optical, subject-referenced, and stability (bottom).
Abstract
Understanding camera motion is fundamental to video perception. Multimodal large language models (MLLMs) provide a natural interface, but existing work typically assigns one or more labels to an entire clip. Such clip-level recognition overlooks two defining properties of real camera motion: it can change within a shot, and multiple movements can occur simultaneously. We therefore formulate camera-motion understanding as temporally grounded, compositional recognition, which requires localizing motion-consistent intervals and identifying every movement active within each interval. We introduce CamChoreo, a benchmark of 4,229 real single-shot clips with expert-annotated temporal segments over a compact vocabulary of 20 direction-aware labels; nearly half of the segments contain compound camera motion. Recognizing such fine-grained, compositional motion is hard for current MLLMs, whose visual encoders emphasize semantic content rather than the geometric evidence on which camera motion depends. Directly injecting features from a frozen 3D foundation model addresses this gap but requires running the expensive geometry model on every input (a baseline we call CamInject). We instead propose CamDistill, which distills the same geometric knowledge into lightweight camera tokens during training and removes the 3D model at inference. CamDistill matches the accuracy of direct feature injection without running the 3D teacher at inference.
At a Glance
clips
segments
segments
micro F1
baseline (Gemini)
The Task: What, When, and Which
Most camera-motion benchmarks reduce a clip to a single set of labels. But even within an uninterrupted shot, the camera may transition between movements or execute several at once, so clip-level labels lose both temporal structure and physical composition. We instead ask a model to partition each shot into motion-consistent intervals and predict the complete set of direction-aware movements active in each interval.
Current MLLMs perform poorly in this setting, revealing a representational gap: their vision encoders are optimized for semantic alignment, while camera motion depends on cross-frame geometry — parallax, perspective change, and horizon rotation.
CamChoreo Benchmark
Figure: CamChoreo statistics — (A) video duration, (B) segments per clip, (C) simultaneous movements per segment, (D) the long-tailed distribution of 20 direction-aware labels, and (E) nine content domains.
CamChoreo contains 4,229 real single-shot YouTube clips (6.88 hours), 8,591 expert-annotated segments, and 14,258 movement instances, with boundaries placed at 0.1-second resolution. Temporal variation and compound motion are the rule rather than the exception: 2,411 clips contain multiple segments, and 3,797 segments (44.2%) contain compound camera motion.
Figure: Construction pipeline. Automatic curation reduces 36,313 source videos to 13,358 candidate clips; expert annotators then replace all pseudo labels with temporal camera-motion annotations, and review produces 4,229 released clips.
| Benchmark | Source | #Clips | Real | #Cls. | Multi-label | Cam-specific | Temporal |
|---|---|---|---|---|---|---|---|
| Cinematic2K | Web | 2,000 | ✔ | 11 | ✘ | ✘ | ✘ |
| VidComposition | Movies | 982 | ✔ | 7 | ✔ | ✘ | ✘ |
| CineTechBench | Movies | 120 | ✔ | 15 | ✔ | ✘ | ✘ |
| ShotBench | Movies | 464 | ✔ | 16 | ✘ | ✘ | ✘ |
| CameraBench | Web | ~3,000 | ✔ | 23 | ✔ | ✔ | ✘ |
| CameraMotionVQA | Synthetic | 12,274 | ✘ | 15 | ✔ | ✔ | ✘ |
| CamChoreo (Ours) | Web | 4,229 | ✔ | 20 | ✔ | ✔ | ✔ |
CamChoreo is the only real-video benchmark combining camera-specific, multi-label annotation with variable-length temporal grounding.
Method: CamDistill
A 3D foundation model such as VGGT-Ω attaches a dedicated camera token to each frame, from which its pose can be decoded — a compact, geometry-aligned representation. CamInject injects the teacher's camera token beside the visual tokens, but must run the expensive 3D model on every test video. CamDistill instead moves the teacher entirely to training: a lightweight extractor predicts per-frame camera tokens from the frozen MLLM features, and a distillation loss aligns them with the teacher. At inference, the 3D model is removed.
Figure: (a) CamInject runs the frozen 3D model beside the vision encoder and inserts each frame's teacher camera token, so the 3D model is required at inference. (b) CamDistill trains a lightweight GCTE to predict student camera tokens, aligned with the teacher by a distillation loss that is removed at inference.
🎥 Geometry-aware Camera Token Extractor (GCTE)
A lightweight branch attached to the frozen vision encoder. It reads intermediate visual features (which retain parallax and perspective cues) and produces one camera token per frame through alternating frame-wise cross-attention and global camera self-attention blocks — without modifying the pretrained visual stream.
🧭 Camera-Token Distillation
A cosine-distance loss aligns each student camera token with the teacher's pose-associated token. Through this objective the model learns a geometry-informed camera representation during training, and the 3D teacher is discarded at inference.
⚡ Near-Zero Inference Overhead
Removing the teacher leaves a compact geometry-aware stream. On the 4B backbone, CamDistill adds only 0.1 s latency and 1.8 GB memory over the base model, versus 5.9 s and 4.8 GB for CamInject.
🔁 Injection-Level Accuracy
Despite dropping the teacher at test time, CamDistill matches CamInject in micro F1 (67.5 on 4B) and stays within 0.4 SegDet — preserving almost all the benefit of direct feature injection.
Main Results
Frame-level micro F1 is evaluated every 0.1 s over 20 direction-aware labels. SegLoc measures temporal overlap; SegDet additionally requires an exact label-set match (IoU 0.5). CamDistill and CamInject lead by more than 20 micro-F1 points.
| Model | Frame-Level Micro | Segment F1 @IoU 0.5 | ||
|---|---|---|---|---|
| F1 | Macro F1 | SegLoc | SegDet | |
| Geometry-only baseline | ||||
| VGGT-Ω + rules | 25.3 | 17.2 | 63.9 | 2.9 |
| Closed-source MLLMs | ||||
| GPT-5.4 | 38.0 | 21.3 | 76.1 | 22.4 |
| Gemini-3.1-Pro | 43.3 | 27.4 | 76.1 | 23.7 |
| Open-source MLLMs | ||||
| Qwen2.5-VL-7B | 21.8 | 4.0 | 39.7 | 6.4 |
| InternVL3.5-8B | 23.6 | 7.1 | 21.0 | 4.0 |
| Qwen3-VL-4B | 24.2 | 7.8 | 55.6 | 11.6 |
| Qwen3-VL-8B | 28.3 | 9.1 | 67.7 | 15.0 |
| Qwen3-VL-235B | 33.4 | 15.4 | 66.1 | 15.6 |
| Qwen3.6-35B | 35.9 | 14.3 | 68.8 | 17.9 |
| Ours (Qwen3-VL backbones) | ||||
| CamDistill-4B | 67.5 | 57.7 | 80.4 | 38.2 |
| CamInject-4B | 67.5 | 57.6 | 80.8 | 38.6 |
| CamDistill-8B | 67.8 | 57.9 | 80.4 | 38.8 |
| CamInject-8B | 68.3 | 59.2 | 80.8 | 39.0 |
Highlighted rows are our methods. CamDistill matches CamInject without running the 3D teacher at inference.
Efficiency & Generalization
Inference cost (Qwen3-VL-4B)
| Method | Micro F1 | Latency (s) | Mem (GB) |
|---|---|---|---|
| Base | 24.2 | 10.1 | 18.3 |
| + SFT | 62.2 | 10.1 | 18.3 |
| + CamDistill | 67.5 | 10.2 | 20.1 |
| + CamInject | 67.5 | 16.0 | 23.1 |
Transfer to external benchmarks
| Model | CameraBench (mAP) | CMVQA (Acc) |
|---|---|---|
| Qwen2.5-VL-7B | 31.0 | 24.8 |
| Qwen3-VL-8B | 39.6 | 23.5 |
| Cam-Motion-7B | 49.7 | 29.7 |
| CamDistill-8B | 60.3 | 40.1 |
| CamInject-8B | 62.9 | 42.4 |
CamDistill achieves the accuracy of direct injection at essentially the base model's inference cost, and the distilled representation transfers to external camera-motion benchmarks with different task formats.
Where the Difficulty Lies
Per-class F1 broadly follows frequency: common movements (Static, Dolly, Pan, Truck) are reliable, while rare optical/rotational movements (Zoom, Roll, Focus Shift) remain hard.
Difficulty grows with composition and temporal structure: compound frames are harder than single-motion frames, and per-video F1 falls as a clip splits into more segments.
Qualitative Results
Predictions of CamDistill-8B on CamChoreo, from single-motion shots to densely compositional ones. In each panel, sampled frames appear above the ground-truth (GT) and predicted (Pred) segment timelines; segments matched by temporal IoU share a color, and each segment lists its active movements.
A single-motion shot: one interval, cleanly recovered.
Three consecutive motion phases with correct boundaries and labels.
Four motion phases, each carrying co-occurring movements.
A long 16-second shot split into five phases, recovered end to end.
Four phases with compound camera motion within each interval.
Contributions
BibTeX
@article{du2026camchoreo,
title={Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation},
author={Du, Dazhao and Du, Shiyan and Liu, Jian and Yu, Yongjian and Gu, Bohai and Han, Tao and Liu, Hualuo and Liu, Eric and Zhang, Yujia and Chen, Xi and Guo, Song},
journal={arXiv preprint arXiv:2608.10932},
year={2026},
url={https://arxiv.org/abs/2608.10932}
}