Friday 21 March 2025
A new approach to reducing the computational complexity of large language models, known as prompt-based depth pruning, has been proposed by researchers. The method involves dynamically determining which layers in a model are most important for processing specific prompts or tasks.
Large language models have revolutionized the field of natural language processing, enabling applications such as language translation, text summarization, and chatbots. However, these models require significant computational resources to train and deploy, making them impractical for many devices and applications.
One approach to reducing the complexity of large language models is to prune unnecessary layers or weights, a technique known as depth pruning. This involves identifying which layers in the model are most important for processing specific tasks or prompts, and removing or compressing the less important ones.
The proposed prompt-based depth pruning method takes this idea a step further by dynamically determining which layers are most important based on the specific prompt or task being processed. This is achieved through the use of a router network that learns to select the optimal layer set for each prompt, taking into account the model’s architecture and the complexity of the task.
The researchers evaluated their method on several large language models, including LLaMA-3.1 8B, OPT 6.7B, and Vicuna 1.5 7B. They found that the proposed approach significantly reduced the computational complexity of the models while maintaining or even improving their performance on a range of tasks.
For example, when pruning the LLaMA-3.1 8B model to 20% sparsity (removing seven layers), the prompt-based depth pruning method achieved an average accuracy of 61.93%, compared to 59.32% for a static pruning approach that used a fixed layer set across all tasks.
The proposed method also showed promising results on more complex tasks, such as question answering and natural language inference. For example, when evaluated on the Winograd Schema Challenge, a task that requires understanding subtle nuances in language, the prompt-based depth pruning method achieved an accuracy of 73.94%, compared to 72.96% for the original model.
The potential applications of this technology are vast, from improving the performance of language translation systems and chatbots, to enabling more efficient processing of large datasets and reducing the energy consumption of data centers.
Cite this article: “Dynamic Pruning of Large Language Models Improves Efficiency and Performance”, The Science Archive, 2025.
Language Models, Prompt-Based Depth Pruning, Computational Complexity, Natural Language Processing, Large Language Models, Depth Pruning, Router Network, Neural Networks, Sparse Model, Accuracy.
Reference: Juyun Wee, Minjae Park, Jaeho Lee, “Prompt-based Depth Pruning of Large Language Models” (2025).
