Text-to-image models have gained attention in recent times. Models like GPT and DALL-E have become popular due to Generative Artificial Intelligence. These models have made it possible to generate content like humans. Now, text-to-video generation is also possible. However, current methods struggle with creating videos that meet cinematic standards.
To address these limitations, a team of researchers has proposed a unique video generation approach called Animate-A-Story. This approach uses existing video content from external databases as a guide signal for text-to-video creation. By using retrieved videos as a reference, users have more control over the layout and composition of the generated videos.
The approach consists of two modules: Motion Structure Retrieval and Structure-Guided Text-to-Video Synthesis. The Motion Structure Retrieval module obtains video candidates based on text prompts and extracts motion structures from them using a commercial video retrieval system. The Structure-Guided Text-to-Video Synthesis module uses these motion structures and text prompts to generate videos that follow a storyline. This framework allows for customizable video production with control over the plot and characters.
The team focuses on preserving visual coherence between footage and has developed a concept personalization strategy. Through text prompts, viewers can select preferred character identities, maintaining consistency throughout the video. The approach has been evaluated and compared to existing methods, showing significant advantages in generating high-quality, coherent, and visually engaging storytelling videos.
In summary, this research introduces a retrieval-augmented paradigm for narrative video synthesis, allowing the use of existing videos for storytelling. It proposes a flexible structure-guided text-to-video approach that reconciles character production and structure guiding. The team also introduces a new concept, TimeInv, in the personalization approach.
To learn more about this research, you can check out the paper, Github, and project page. Credit goes to the researchers involved. Don’t forget to join our ML SubReddit, Discord Channel, and Email Newsletter for the latest AI research news and projects.
Tanya Malhotra is an undergraduate student specializing in Artificial Intelligence and Machine Learning. She has a passion for Data Science and is skilled in analytical thinking, acquiring new skills, and managing work effectively.