Video Object Coordinate Map Estimation

1 · What this task is

给模型一段 24-frame RGB 视频，要求输出同一时刻、同一像素位置的 object coordinate map 视频。每个前景像素的 RGB 不是外观颜色，而是该表面点在物体自身 normalized 3D coordinate system 中的 XYZ。

Measured snapshot

2 · Source roles and publisher-native evidence

Source A · MOVi-E TFDS 1.0.0, 256×256 FULL VERIFIED

Contribution. 统一 24-frame 的 synthetic moving-camera scenes，source-native fields 同时包含 RGB、16-bit object coordinates、instance segmentations、object/category/dynamic metadata。完整 release 已下载、逐记录解析、Raw S3 对账并全量 streamed SHA-256 readback。

Publisher: Google Research / Kubric · GCS release: gs://kubric-public/tfds/movi_e/256x256/1.0.0/ · Structural fitness: strong · Release eligibility: aggregate license review required

Publisher-native evidence · 2 / 2

A1 · test:218

较稀疏的 foreground case。 可以清楚看到 raw coordinate pass 的背景并不是黑色；这正是后续必须用 native segmentation 做 foreground masking 的原因。

Native key: (test, 218, frame 0…23) · shard movi_e-test.tfrecord-00230-of-00256 · record 0 · source record 27,313,770 bytes

Native RGB

Raw object coordinates

Native segmentation labels

Direct TFRecord decode; 24/24 frames. MP4s are presentation-only H.264 proxies; coordinates are uint16→uint8 display quantization without foreground masking. Contact sheet 168,545 bytes · SHA-256 bab9e07d…793070 · Provenance JSON.

A2 · test:823

不同 object composition / motion case。 与 A1 构成覆盖对比，用于检查 coordinate colors、instance labels 和 RGB 在完整时间轴上的同步关系。

Native key: (test, 823, frame 0…23) · shard movi_e-test.tfrecord-00230-of-00256 · record 1 · source record 29,224,258 bytes

Native RGB

Raw object coordinates

Native segmentation labels

Direct TFRecord decode; 24/24 frames. Same source-faithful display-only transforms as A1. Contact sheet 156,275 bytes · SHA-256 fe24c43a…848ae0 · Provenance JSON.

Related-only portfolio

3 · Data type and format

movi_e/256x256/1.0.0/
├── movi_e-train.tfrecord-00000-of-01024 … 01023
├── movi_e-validation.tfrecord-00000-of-00064 … 00063
├── movi_e-test.tfrecord-00000-of-00256 … 00255
├── dataset_info.json
├── features.json
└── label files

Complete release: 1,024 train shards + 64 validation shards + 256 test shards = 1,344 TFRecords. Every example has exactly 24 frames at 12 FPS. Common 8-bit image readers can silently lose precision; audit uses OpenCV IMREAD_UNCHANGED and checks the PNG IHDR bit depth.

4 · How Raw becomes task samples

1. Read the aligned native video, object_coordinates, and segmentations for one complete source example.
2. Verify 24 frames, 256×256 geometry, native ordering, segmentation range, and foreground presence.
3. Compute target = round(uint16_coordinates / 257); set pixels with segmentation == 0 to RGB (0,0,0).
4. Package one Input video, one lossless H.264 RGB Target video, prompt.txt, and provenance-rich metadata.json in the separate pipeline layer.

Full-production Final sample census

• Split census: 9,749 train + 250 validation + 999 test.
• Formula: one complete eligible source example → one Final sample. No window multiplier.
• Inclusion tests: all three task fields present; successful PNG decode; exact 24-frame 256×256 alignment; valid segmentation IDs; foreground present.
• Exclusions: 0 decode failures, 0 missing fields/keys, 0 shape/alignment failures, 0 empty-foreground examples.
• Split/leakage: publisher split is frozen before transformation; every derived artifact stays with its complete source example. Pairing key includes split because 999 numeric IDs are reused across splits.
• Tail policy: not applicable. No padding, repeat, backward shift, or temporal crop.

5 · Data volume and integrity

Official-versus-measured discrepancy: high-level MOVi documentation describes approximately 10k train / 1k test scenes. The distributed TFDS metadata and complete parse measure exactly 9,749 train, 250 validation, and 999 test examples: 10,998 total.

6 · License, limitations and benchmark

• License: Kubric code is Apache-2.0; Google Scanned Objects is described as CC BY 4.0 and Poly Haven as CC0. MOVi-E dataset_info.json has no aggregate license field, so Raw/Final public or commercial redistribution is not cleared by this report.
• Synthetic-domain gap: rendered scenes underrepresent real sensor noise, hands, nonrigid objects, transparency, severe occlusion, and uncontrolled lighting.
• Coordinate convention: coordinates are normalized per object bounding box; this is useful object-local geometry but not always semantic cross-category correspondence.
• Symmetry: symmetric objects can admit multiple visually plausible coordinate solutions.
• Encoding: ordinary YUV H.264 changes coordinate values. The measured exact option is libx264rgb -crf 0 / gbrp; standard browser proxies are viewing aids only.
• Evaluation: foreground XYZ MAE/RMSE and threshold accuracy are primary. Temporal consistency and background accuracy are separate. NOCS-style pose/3D IoU metrics require a downstream pose recovery stage.

7 · Source, Raw S3 and reproducibility

• Official MOVi documentation · MOVi-E builder · Kubric paper
• NOCS paper · official NOCS code/data instructions
• Raw S3: s3://tv2v-data/raw/taxonomy/Task_160_video_object_coordinate_map_estimation/movi_e/256x256/1.0.0/
• Durable local audit: manifests/source_objects.jsonl, audit_records.jsonl, audit_summary.json, s3_archive_summary.json, s3_readback_results.jsonl
• Reproducible scripts: download_movi_e_release.py, audit_movi_e_release.py, archive_movi_e_to_s3.py, extract_native_evidence.py, probe_target_encoding.py