Wednesday 26 March 2025
A team of researchers has made a significant breakthrough in accelerating transformer inference, a crucial component of modern artificial intelligence. Transformers are complex neural networks that enable machines to understand and generate human-like language, making them essential for applications such as language translation, text summarization, and chatbots.
Traditionally, transformer inference is performed on central processing units (CPUs) or graphics processing units (GPUs), which can be slow and energy-hungry. To address this limitation, the researchers designed a novel hardware-software co-design approach that leverages compute-in-memory (CIM) technology to accelerate transformer inference.
CIM technology integrates computing capabilities into memory devices, reducing the need for data movement between different components of a system. This approach enables faster and more efficient processing of large amounts of data, making it particularly well-suited for applications like transformer inference.
The researchers developed a hardware accelerator called HASTILY, which is designed to accelerate softmax computation, an integral operation in transformer attention mechanisms. Softmax is a mathematical function that normalizes the output of neural networks to ensure they produce probabilities that add up to 1. In traditional CPU- or GPU-based implementations, softmax computation can be a bottleneck, limiting the overall performance of transformer inference.
HASTILY’s architecture features novel CIM units called unified compute and lookup modules (UCLMs), which integrate both lookup and multiply-accumulate functionality within the same SRAM array. This design minimizes area overhead over standard CIM arrays while enabling concurrent exponential and matrix-vector multiplication operations.
The researchers also developed a fine-grained pipelining strategy for scheduling attention and feed-forward layers, reducing the quadratic dependence on sequence length to linear dependence. Additionally, they parallelized fast softmax computation across multiple cores using reduce and gather strategies.
Evaluation results show that HASTILY achieves significant improvements in end-to-end throughput and energy efficiency compared to existing solutions. For BERT models with INT-8 precision, HASTILY outperforms Nvidia A40 GPU by 4.4x to 9.8x in terms of throughput and matches the energy efficiency of baseline CIM hardware.
The researchers’ approach has far-reaching implications for the development of efficient AI systems. By accelerating transformer inference, they can enable faster and more accurate language processing, which has numerous applications in areas such as healthcare, finance, and education.
Cite this article: “Accelerating Transformer Inference with Compute-in-Memory Technology”, The Science Archive, 2025.
Artificial Intelligence, Transformer Inference, Compute-In-Memory, Neural Networks, Language Translation, Text Summarization, Chatbots, Softmax Computation, Hastily, Cim Technology
