← 返回研究面板

Task_15 背景替换 · B 样品
绿幕 + 文字描述 → 真实背景：输入=绿幕+主体+背景描述，答案=原始真视频

Winnie 选定 B：合成的绿幕放输入、真实素材放答案。prompt 详细描述目标背景(英文)→ target(原视频)成为唯一正确答案。全程只用 DAVIS，无外部背景库。3 条真实样品，每条含完整 4 文件盒(下方展开 prompt/metadata),未入 final_v1。

Task_15 · 背景替换B · 绿幕+文字→真背景 · 答案是真货样品 · 待确认

1B「绿幕 + 文字描述 → 真实背景」

按 Winnie 定的方向:输入 = 绿幕前的主体(把原视频背景抠成一块绿幕) + 一段文字描述目标背景; 答案 = 原始真视频。模型学的是「给绿幕抠像 + 背景描述 → 生成带该背景的真实视频」。

为什么答案质量更好:A/A2 的答案是合成的;B 的答案 = 100% 真实 DAVIS 素材、零合成—— 把"造出来的"那一侧(绿幕)放进输入、真货放答案。而且全程只用 DAVIS,不需要 Places365。
为什么 prompt 要详细描述背景(Winnie 指出):若 prompt 只说"换个合理背景",模型没线索、答案不唯一,和我们特定的 target 对不上。 让 prompt 描述出原视频的背景,target(原视频)就成了唯一正确答案——任务从"随便生成"变成「按文字精准重建」,干净可训练。 每个样品的 prompt.txt 全文见下方(英文)。
sample_00/
├── input.mp4          ← 输入①:绿幕 + 主体(用 DAVIS 真值掩码合成)
├── prompt.txt         ← 输入②:英文文字,详细描述要生成的背景场景(见下方样品)
├── target.mp4         ← 答案:原始 DAVIS 真视频(主体+真实背景) —— 真实素材、无合成
└── metadata.json      ← 记两输入 + 目标 + 背景描述 + 绿幕 RGB/合成式 + 主体来源(DAVIS 掩码 GT)

批量时背景描述怎么来:preview 这 3 条我按真实画面手写;批量对每个 clip 用图像描述模型(如 BLIP-2)对 target 自动生成 caption,可扩展到全量。

2三条真实 B 样品 · 绿幕输入 → 真视频答案

sample_00 · 主体 blackswan 48 帧 · 640x360

输入① — 绿幕 + 主体（合成）input.mp4
答案 — 原视频（真实素材·真 GT）target.mp4
📄 输入② prompt.txt (英文·描述要生成的背景)
Input: a black swan gliding on water, filmed against a solid green screen (chroma-key green).
Task: replace the green background with the real-world scene described below, while keeping the subject's silhouette, position, scale and motion identical in every frame — regenerate only the background, do not alter the subject.
Background to generate: a calm natural pond in a park. The swan floats on dark, gently rippling water near the far bank; dense green foliage and leafy shrubs line the shore behind it, a pale concrete-edged waterline runs across the middle, and soft reflections of the greenery shimmer on the water surface.
🧾 metadata.json (点开看完整字段)
{
  "schema": "v2v-task/1.0",
  "task": "Task_15_background_replacement",
  "method": "B_greenscreen_to_real (text-conditioned)",
  "inputs": [
    {
      "role": "greenscreen_video",
      "file": "input.mp4",
      "desc": "subject on a solid green screen (composited with the DAVIS ground-truth mask)"
    },
    {
      "role": "text_prompt",
      "file": "prompt.txt",
      "desc": "describes the target background scene to generate behind the subject"
    }
  ],
  "target": {
    "file": "target.mp4",
    "desc": "the original DAVIS video (subject on its real background) — real footage, no compositing; because the prompt describes THIS background, the target is the unique correct answer"
  },
  "prompt_file": "prompt.txt",
  "metadata": {
    "schema": "v2v-metadata/2.0",
    "subject_source": {
      "dataset": "DAVIS-2017",
      "sequence": "blackswan",
      "mask": "ground-truth per-frame segmentation"
    },
    "green_screen": {
      "rgb": [
        0,
        177,
        64
      ],
      "composite": "input = subject*mask + green*(1-mask)",
      "feather_px": 0.8,
      "note": "green only marks where the model must paint background; the subject region is untouched"
    },
    "note": "B direction: synthesized side (green screen) is the INPUT, real footage is the TARGET. The prompt names the actual background, so this is text-conditioned reconstruction — the real target video is the unique ground truth, not one of many plausible outputs.",
    "frames": 48,
    "size": "640x360",
    "fps": 12,
    "pipeline_version": "task15-b-preview-0.2",
    "background_prompt": {
      "subject": "a black swan gliding on water",
      "scene": "a calm natural pond in a park. The swan floats on dark, gently rippling water near the far bank; dense green foliage and leafy shrubs line the shore behind it, a pale concrete-edged waterline runs across the middle, and soft reflections of the greenery shimmer on the water surface.",
      "source": "described from the real DAVIS footage (the target); batch stage will auto-caption per clip (e.g. BLIP-2)"
    }
  }
}

sample_01 · 主体 dog 48 帧 · 640x360

输入① — 绿幕 + 主体（合成）input.mp4
答案 — 原视频（真实素材·真 GT）target.mp4
📄 输入② prompt.txt (英文·描述要生成的背景)
Input: a golden-retriever puppy running, filmed against a solid green screen (chroma-key green).
Task: replace the green background with the real-world scene described below, while keeping the subject's silhouette, position, scale and motion identical in every frame — regenerate only the background, do not alter the subject.
Background to generate: an open outdoor field of dry, sun-bleached tan grass with a few green blades at the edges. The ground streaks past with natural motion blur as the puppy bounds forward, bright daylight, shallow depth of field.
🧾 metadata.json (点开看完整字段)
{
  "schema": "v2v-task/1.0",
  "task": "Task_15_background_replacement",
  "method": "B_greenscreen_to_real (text-conditioned)",
  "inputs": [
    {
      "role": "greenscreen_video",
      "file": "input.mp4",
      "desc": "subject on a solid green screen (composited with the DAVIS ground-truth mask)"
    },
    {
      "role": "text_prompt",
      "file": "prompt.txt",
      "desc": "describes the target background scene to generate behind the subject"
    }
  ],
  "target": {
    "file": "target.mp4",
    "desc": "the original DAVIS video (subject on its real background) — real footage, no compositing; because the prompt describes THIS background, the target is the unique correct answer"
  },
  "prompt_file": "prompt.txt",
  "metadata": {
    "schema": "v2v-metadata/2.0",
    "subject_source": {
      "dataset": "DAVIS-2017",
      "sequence": "dog",
      "mask": "ground-truth per-frame segmentation"
    },
    "green_screen": {
      "rgb": [
        0,
        177,
        64
      ],
      "composite": "input = subject*mask + green*(1-mask)",
      "feather_px": 0.8,
      "note": "green only marks where the model must paint background; the subject region is untouched"
    },
    "note": "B direction: synthesized side (green screen) is the INPUT, real footage is the TARGET. The prompt names the actual background, so this is text-conditioned reconstruction — the real target video is the unique ground truth, not one of many plausible outputs.",
    "frames": 48,
    "size": "640x360",
    "fps": 12,
    "pipeline_version": "task15-b-preview-0.2",
    "background_prompt": {
      "subject": "a golden-retriever puppy running",
      "scene": "an open outdoor field of dry, sun-bleached tan grass with a few green blades at the edges. The ground streaks past with natural motion blur as the puppy bounds forward, bright daylight, shallow depth of field.",
      "source": "described from the real DAVIS footage (the target); batch stage will auto-caption per clip (e.g. BLIP-2)"
    }
  }
}

sample_02 · 主体 camel 48 帧 · 640x360

输入① — 绿幕 + 主体（合成）input.mp4
答案 — 原视频（真实素材·真 GT）target.mp4
📄 输入② prompt.txt (英文·描述要生成的背景)
Input: a Bactrian (two-hump) camel walking, filmed against a solid green screen (chroma-key green).
Task: replace the green background with the real-world scene described below, while keeping the subject's silhouette, position, scale and motion identical in every frame — regenerate only the background, do not alter the subject.
Background to generate: a sandy outdoor zoo enclosure. A rustic wooden post-and-rail fence runs horizontally behind the camel, with green trees and bushes beyond it and a thatched straw roof at the upper right; the ground is pale packed dirt with a few scattered stones.
🧾 metadata.json (点开看完整字段)
{
  "schema": "v2v-task/1.0",
  "task": "Task_15_background_replacement",
  "method": "B_greenscreen_to_real (text-conditioned)",
  "inputs": [
    {
      "role": "greenscreen_video",
      "file": "input.mp4",
      "desc": "subject on a solid green screen (composited with the DAVIS ground-truth mask)"
    },
    {
      "role": "text_prompt",
      "file": "prompt.txt",
      "desc": "describes the target background scene to generate behind the subject"
    }
  ],
  "target": {
    "file": "target.mp4",
    "desc": "the original DAVIS video (subject on its real background) — real footage, no compositing; because the prompt describes THIS background, the target is the unique correct answer"
  },
  "prompt_file": "prompt.txt",
  "metadata": {
    "schema": "v2v-metadata/2.0",
    "subject_source": {
      "dataset": "DAVIS-2017",
      "sequence": "camel",
      "mask": "ground-truth per-frame segmentation"
    },
    "green_screen": {
      "rgb": [
        0,
        177,
        64
      ],
      "composite": "input = subject*mask + green*(1-mask)",
      "feather_px": 0.8,
      "note": "green only marks where the model must paint background; the subject region is untouched"
    },
    "note": "B direction: synthesized side (green screen) is the INPUT, real footage is the TARGET. The prompt names the actual background, so this is text-conditioned reconstruction — the real target video is the unique ground truth, not one of many plausible outputs.",
    "frames": 48,
    "size": "640x360",
    "fps": 12,
    "pipeline_version": "task15-b-preview-0.2",
    "background_prompt": {
      "subject": "a Bactrian (two-hump) camel walking",
      "scene": "a sandy outdoor zoo enclosure. A rustic wooden post-and-rail fence runs horizontally behind the camel, with green trees and bushes beyond it and a thatched straw roof at the upper right; the ground is pale packed dirt with a few scattered stones.",
      "source": "described from the real DAVIS footage (the target); batch stage will auto-caption per clip (e.g. BLIP-2)"
    }
  }
}

3三种方向对照 + 待定

方向输入答案(TARGET)答案是真是合成
A(文字式)原视频 + 一句话描述新背景主体+新背景(合成)合成
A2(背景给定)原视频 + 一张新背景图主体+该背景(合成)合成
B(绿幕→真背景) ✅ 选定绿幕+主体 + 文字描述背景原始真视频✅ 真实素材

Winnie 选定 B。下面是批量前的剩余小决定:

数据来源:全部来自 DAVIS-2017(真值掩码,已入湖),绿幕纯色生成,无外部依赖。样品在 winnie-task15-b(us-east-2,部分抽取、用后已 terminate)合成。未入 final_v1。相关:A2 样品 · Task_15 报告。