Systems and methods for a text-to-video generation framework

Embodiments described herein provide a generation model comprising a video-specific variational auto-encoder (VAE) for effective compression of video pixel information with reduced spatial and temporal dimensions and a video diffusion transformer (vDiT) to generate latent representations of frames. Specifically, the VAE may, instead of encoding each frame independently, incorporate both temporal and spatial compression. This significantly decreases the token length, improves the computational cost of training and inference, and facilitates the generation of long videos. The encoded training video, in the form of latent representations from a VAE encoder may then be passed to the vDiT to reconstruct the latent representations during training. The trained vDiT may then generate latent representations of a video in response to a text input, and the latent representations may be converted to a video output by a VAE decoder.