Video relighting requires balancing long-form temporal consistency with physically grounded understanding of light transport, which depends on accurate estimation of intrinsic scene properties such as materials, geometry, and illumination. Existing methods follow two paradigms: (1) Given an input video, explicitly reconstruct its photometric properties via inverse rendering, and then relight the reconstructions to a target illumination via forward rendering, either via physically-based rendering (PBR) or a neural rendering engine. Such methods suffer from noisy reconstructions and struggle to capture hard-to-model illumination effects such as global illumination. (2) Alternatively, frame the task as a generative video-to-video translation task that conditions on relighting targets (specified as target environment map or text). However, such a framing limits relighting control and temporal stability since generative diffusion models struggle to translate long-form videos. Moreover, such data-driven methods are limited by availability of training pairs of input videos and their relit targets.
We propose LightCrafter, a hybrid pipeline that reformulates video relighting as video translation of a proxy video: rather than directly translating the input video to the target, we translate a PBR rendering (of the input video under the target illumination conditions) to the final target. This allows us to "bake" illumination targets into the PBR-proxy rendering, removing the need to explicitly teach the diffusion model about illumination concepts like environment maps. We find PBR proxy renderings allow for more intricate lighting control while naturally providing long-form temporal consistency. In fact, we show that PBR renders already outperform some prior art for relighting, but struggle to model intricate effects like global illumination. To capture such effects, we leverage photometric priors implicit in video generation models. Specifically, we post-train CogVideoX on synthetic video pairs and real-world unpaired videos. We outperform prior state-of-the-art on existing real-world relighting benchmarks and also contribute our own synthetic benchmark for further analysis. We will release our dataset, benchmark, metrics, and code.
Overview of the proposed method. Our method recovers a relightable scene state from an input video, including photometric properties, geometry, camera motion, and illumination, and uses it to render a frame-aligned PBR render. The rendering provides an explicit interface for environment relighting, indoor light insertion, and G-buffer or scene-state editing, while a video diffusion refiner removes artifacts from imperfect PBR renders and produces a photorealistic, temporally coherent relit video.
Real-world videos relit and edited by our pipeline. For each scene we show the source clip, the PBR proxy our inverse-rendering produces, and the diffusion-refined relit output. Light insertion edits start from a clean scene render and add a new light source.
We compare against PCRP-Video, UniRelight, LightX, and DiffusionRenderer on long video relighting tasks. Choose which method appears on each side of the slider and drag the handle to A/B the results — on a real-world scene below, and on our synthetic benchmark (where a held-out Target is also available).
Baselines often produce plausible relighting but entangle illumination with generation, causing baked-in source lighting, color-tone drift, or inconsistent shadows; our explicit PBR proxy anchors the target-light interaction before diffusion refinement, enabling more faithful and temporally consistent responses to the target illumination.
Multi-illumination relighting on a variety of scenes. Each row shows the input video alongside a single slider; drag the vertical handle to compare PBR (left) vs Refined (right), and use the Illum 1–3 buttons to switch the target illumination. The chrome-ball lighting probe in the corner previews the selected target environment.
autumn_field_puresky
autumn_field_puresky
autumn_field_puresky
autumn_field_puresky
autumn_field_puresky
autumn_field_puresky
@article{guo2026lightcrafter,
title = {LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting},
author = {Guo, Zixin and Litman, Yehonathan and He, Yifeng and Miller, John and Chen, Chuhan and Ramanan, Deva},
journal = {arXiv preprint arXiv:2607.08016},
year = {2026}
}