Back to articles
Proceedings Paper
Volume: 38 | Article ID: GENAI-181
Image
Motion-adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusion
  DOI :  10.2352/EI.2026.38.12.GENAI-181  Published OnlineMarch 2026
Abstract
Abstract

We present a motion-adaptive temporal attention mechanism for parameter-efficient video generation built upon frozen Stable Diffusion models. Rather than treating all video content uniformly, our method dynamically adjusts temporal attention receptive fields based on estimated motion content: high-motion sequences attend locally across frames to preserve rapidly changing details, while low-motion sequences attend globally to enforce scene consistency. We inject lightweight temporal attention modules into all UNet transformer blocks via a cascaded strategy—global attention in down-sampling and middle blocks for semantic stabilization, motion-adaptive attention in up-sampling blocks for fine-grained refinement. Combined with temporally correlated noise initialization and motion-aware gating, the system adds only 25.8M trainable parameters (2.9% of the base UNet) while achieving competitive results on WebVid validation when trained on 100K videos. We demonstrate that the standard denoising objective alone provides sufficient implicit temporal regularization, outperforming approaches that add explicit temporal consistency losses. Our ablation studies reveal a clear trade-off between noise correlation and motion amplitude, providing a practical inference-time control for diverse generation behaviors.

Subject Areas :
Views 1
Downloads 0
 articleview.views 1
 articleview.downloads 0
  Cite this article 

Rui Hong, Shuxue Quan, "Motion-adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusionin Electronic Imaging,  2026,  pp 181-1 - 181-7,  https://doi.org/10.2352/EI.2026.38.12.GENAI-181

 Copy citation
  Copyright statement 
Copyright ©2026 Society for Imaging Science and Technology 2026
ei
Electronic Imaging
2470-1173
2470-1173
Society for Imaging Science and Technology
IS&T 7003 Kilworth Lane, Springfield, VA 22151 USA