Sunday 23 February 2025
Generative video models have made tremendous progress in recent years, allowing us to create realistic and dynamic scenes that can simulate a wide range of scenarios. However, these models often struggle when tasked with generating videos that adhere to complex compositional prompts – that is, prompts that describe specific objects, actions, and relationships within the scene.
A new paper from researchers at the University of Hong Kong seeks to address this limitation by proposing a novel multi-agent framework for generative video modeling. The approach, dubbed GENMAC, uses multiple neural networks, each with its own specialized role, to collaboratively generate videos that meet complex compositional prompts.
At the core of GENMAC is a hierarchical decomposition of the prompt into simpler sub-tasks. Each agent in the system is responsible for addressing one or more of these sub-tasks, and they work together to ensure that the generated video meets the overall requirements of the prompt.
The first stage of the process, known as the DESIGN stage, involves each agent generating a proposal for how it plans to contribute to the final video. This might involve determining the position and movement of an object within the scene, or deciding on the background texture and lighting.
In the REDESIGN stage, the agents review each other’s proposals and provide feedback in the form of suggestions and corrections. For example, if one agent has proposed a location for an object that conflicts with another agent’s plan, it will suggest alternative locations or adjustments to the object’s movement.
The output from this iterative process is a refined set of proposals that are then used to generate the final video. The researchers demonstrate the effectiveness of their approach using a range of complex compositional prompts, including scenes involving multiple objects, dynamic motion, and interactions between characters.
One particularly impressive example involves generating a video of a rabbit police officer directing traffic on a busy street. In this scenario, the GENMAC system is able to create a realistic scene with a consistent background, accurately positioned toy cars, and a rabbit that appears to be actively directing traffic.
The implications of this research are significant, as it opens up new possibilities for applications such as video game development, film production, and even virtual reality experiences. By providing a more flexible and adaptable framework for generative video modeling, GENMAC has the potential to enable the creation of more complex and engaging visual content than ever before.
Cite this article: “GENMAC: A Multi-Agent Framework for Generative Video Modeling”, The Science Archive, 2025.
Generative Video Models, Multi-Agent Framework, Genmac, Neural Networks, Compositional Prompts, Hierarchical Decomposition, Scene Generation, Object Positioning, Video Synthesis, Computer Vision, Artificial Intelligence.
