Friday 28 February 2025
A new attention engine has been developed, designed to boost the performance of large language models (LLMs) and make them more efficient for real-world applications. The engine, called FlashInfer, is a custom-built software that optimizes the way LLMs process and generate text.
One of the key challenges facing LLMs is their reliance on attention mechanisms, which allow them to focus on specific parts of the input data when generating output. However, these mechanisms can be slow and inefficient, particularly for longer sequences or larger models. FlashInfer addresses this issue by introducing a new attention engine that leverages advanced GPU architecture and specialized instructions to accelerate attention computation.
The engine uses a novel combination of techniques, including sparse loading, head-group fusion, and overlap with other operations. Sparse loading allows the model to quickly retrieve relevant information from the KV-cache, while head-group fusion enables multiple query heads to share the same key-value heads, reducing memory access and improving performance. Overlap with other operations, meanwhile, enables the engine to execute attention computation in parallel with other tasks, such as GEMM (generalized matrix-matrix multiplication) and inter-device communication.
FlashInfer also supports mixed-precision attention, a technique that stores KV-cache values in 8-bit floating-point numbers (fp8) while maintaining the query and output in 16-bit floating-point numbers (fp16). This allows for reduced memory footprints and higher bandwidth utilization without sacrificing accuracy.
The engine has been tested on various LLMs, including those with up to 32 heads and head dimensions of 128. Results show significant improvements in performance, with throughput increases of up to 30% compared to previous engines. Additionally, FlashInfer’s support for mixed-precision attention has allowed models to achieve similar accuracy levels as their full-precision counterparts while reducing memory consumption by up to 50%.
The implications of this technology are far-reaching, enabling LLMs to be deployed in a wider range of applications, from natural language processing and machine translation to text summarization and chatbots. As the demand for AI-powered language models continues to grow, FlashInfer’s innovative approach is poised to play a key role in driving their development forward.
Cite this article: “FlashInfer: A Novel Attention Engine for Large Language Models”, The Science Archive, 2025.
Large Language Models, Attention Engine, Flashinfer, Gpu Architecture, Mixed-Precision Attention, Kv-Cache, Head-Group Fusion, Sparse Loading, Overlap With Other Operations, Gemm
