Saturday 01 March 2025
A team of researchers has made a significant breakthrough in the field of multimodal large language models, developing a novel method for pruning visual tokens that can greatly reduce computation while retaining high performance.
Multimodal large language models (MLLMs) are designed to process and understand both text and images. However, these models often require enormous computational resources, making them impractical for real-world applications. To address this issue, the researchers proposed a graph-based method called G-Prune, which selectively retains only the most critical visual tokens.
G-Prune works by constructing a graph where visual tokens are nodes, and their connections represent semantic similarities between objects in an image. The algorithm then iteratively propagates information through the graph to identify and retain the most representative tokens for each object.
Experimental results show that G-Prune can significantly reduce computational overhead while maintaining high performance on both coarse-grained and fine-grained tasks. For instance, G-Prune reduced the number of floating-point operations (FLOPs) by 63.57% on a popular benchmark dataset while only incurring a 2% decrease in accuracy.
The researchers also demonstrated that G-Prune can effectively preserve detailed information from both foreground and background objects, even when pruning up to 90% of the tokens. This is particularly important for tasks that require understanding complex visual scenes or relationships between multiple objects.
G-Prune’s ability to selectively retain only the most critical visual tokens has significant implications for the development of more efficient and scalable MLLMs. The method can be applied to a wide range of applications, including image recognition, object detection, and visual question answering.
In the future, the researchers plan to explore further optimizations and extensions of G-Prune, with the goal of making it even more effective and widely applicable. With its potential to significantly reduce computational costs while maintaining high performance, G-Prune is an exciting development that could have a major impact on the field of multimodal large language models.
Cite this article: “Efficient Pruning of Visual Tokens in Multimodal Large Language Models”, The Science Archive, 2025.
Multimodal, Large Language Models, Pruning, Visual Tokens, Graph-Based Method, Computational Overhead, Accuracy, Floating-Point Operations, Image Recognition, Object Detection
