On Friday, Meta unveiled Movie Gen, the latest in its series of multimodal video artificial intelligence tools. This new technology aims to craft customized videos and audio, edit existing video content, and convert personal images into distinctive video pieces, all while demonstrating superior performance compared to models like Runway’s Gen-3, Kuaishou Technology’s Kling 1.5, and OpenAI’s Sora.
Building upon its previous advancements, Movie Gen integrates insights from Meta’s earlier models, including the innovative Make-A-Scene models and Llama’s image foundation models. As a holistic suite, Movie Gen encompasses capabilities for video creation, personalized video content, detailed editing, and audio generation, thereby enhancing creators’ control over their projects. Meta envisions that these models will pave the way for novel products that could significantly boost creativity, as mentioned in their announcement.
For video generation, Movie Gen relies on a 30 billion parameter model that can produce clips up to 16 seconds long, albeit at a moderate frame rate of 16 frames per second. According to Meta, “These models can analyze object motion, interactions between subjects and objects, and camera dynamics, while learning realistic movements for numerous concepts,” positioning them at the forefront of their category. Using this same framework, Movie Gen can generate personalized videos tailored for creators using still images.
Meta has adapted this video-generation model to utilize both video and text inputs, allowing for meticulous editing of content. This includes localized modifications like the addition or removal of elements, as well as global changes such as applying new cinematic styles. For audio creation, Movie Gen employs a distinct 13 billion parameter model capable of generating 45 seconds of audio, which can include ambient sounds, effects, or musical scores, all matched to the video content automatically.
Meta’s research indicates that Movie Gen consistently outperformed other leading video AIs in various tests, including those against Gen3, Sora, and Kling 1.5 for video generation, in addition to excelling in personalized video and audio generation over competitors like ID-animator and Pika Labs Sound Gen. The evidence suggests that Movie Gen surpasses existing free video generator options in quality as well.
The organization aims to closely collaborate with filmmakers and creators to incorporate their feedback during ongoing development while stressing that its goal isn’t to replace human creators with AI. “We share this research because we believe this technology can empower individuals to express themselves in innovative ways and offer new opportunities to those who might not have otherwise had them,” Meta stated. “Our aspiration is that one day, everyone will have the ability to realize their artistic visions, creating high-definition videos and audio using Movie Gen.”
