New Transformer architecture could enable powerful LLMs without GPUs
What are the main components of the novel MatMul-free architecture introduced by the researchers?
The novel MatMul-free architecture introduced by the researchers consists of the following main components:
-
Ternary Weights: The researchers suggest replacing the traditional 16-bit floating point weights used in Transformers with 3-bit ternary weights that can take one of three states: -1, 0 and +1.
-
Additive Operations: They replace MatMul with additive operations that provide equally good results at much less computational costs. The models are composed of “BitLinear layers” that use ternary weights.
-
MatMul-free Linear Gated Recurrent Unit (MLGRU): The token mixer is implemented using a MatMul-free Linear Gated Recurrent Unit (MLGRU). The MLGRU processes the sequence of tokens by updating hidden states through simple ternary operations without the need for expensive matrix multiplications.
-
Gated Linear Unit (GLU): The channel mixer is implemented using a Gated Linear Unit (GLU), which is also used in Llama-2 and Mistral. The GLU is modified to also work with ternary weights instead of MatMul operations, enabling the reduction of computational complexity and memory usage while maintaining the effectiveness of feature integration.
By combining the MLGRU token mixer and the GLU channel mixer with ternary weights, the proposed architecture relies solely on addition and element-wise products.
How have researchers from the University of California, Santa Cruz, Soochow University, and University of California, Davis addressed the issue of matrix multiplications in language models?
Researchers from the University of California, Santa Cruz, Soochow University, and University of California, Davis have developed a novel architecture that completely eliminates matrix multiplications from language models while maintaining strong performance at large scales. In their paper, they introduce MatMul-free language models that achieve performance on par with state-of-the-art Transformers while requiring far less memory during inference.
To achieve this, the researchers replaced the traditional 16-bit floating point weights used in Transformers with 3-bit ternary weights that can take one of three states: -1, 0, and +1. They also replaced MatMul with additive operations that provide equally good results at much less computational cost. The models are composed of "BitLinear layers" that use ternary weights.
Furthermore, the researchers made more profound changes to the language model architecture. They implemented the token mixer using a MatMul-free Linear Gated Recurrent Unit (MLGRU), which processes the sequence of tokens by updating hidden states through simple ternary operations without the need for expensive matrix multiplications. The channel mixer was implemented using a Gated Linear Unit (GLU), which was also modified to work with ternary weights instead of MatMul operations.
By combining the MLGRU token mixer and the GLU channel mixer with ternary weights, the proposed architecture relies solely on addition and element-wise products, significantly reducing computational complexity and memory usage while maintaining the effectiveness of feature integration.
What is the primary computational challenge associated with large language models (LLMs) using the Transformer architecture?
The primary computational challenge associated with large language models (LLMs) using the Transformer architecture is the matrix multiplications (MatMul). As LLMs scale to larger sizes, the cost of MatMul grows significantly, increasing memory usage and latency during both training and inference phases. This becomes a bottleneck, requiring very large GPU clusters for the computation.