Improvements in areas like text creation, few-shot learning, reasoning, and protein sequence modelling have been made possible by large language models (LLMs). These models have hundreds of billions of parameters, so deploying them is complex. To address this, Cornell University conducted research and found that quantizing LLM parameters after training can enhance their performance in real-world scenarios.
The researchers discovered that when the weight and proxy Hessian matrices are incoherent, it’s easier to round the weights to a finite set of compressed values. This is because the weights and the directions that require accurate rounding are not too large in any one coordinate. Based on this insight, the researchers developed a technique called quantization with incoherence processing (QuIP) for LLM-sized models.
QuIP consists of two phases: efficient pre- and post-processing to ensure the Hessian matrices are incoherent and an adaptive rounding procedure that minimizes the error between the original weights and the quantized weights using an estimate of the Hessian.
The researchers not only implemented QuIP practically but also conducted a theoretical study to analyze its impact and compare it to other rounding techniques. They found that incoherence processing significantly improves large-model quantization, especially at higher compression rates. With QuIP, they were able to achieve usable results with only two bits per weight, which is a significant accomplishment.
However, the proxy objective used in QuIP doesn’t consider interactions between transformer blocks or within layers of a block. It’s uncertain whether including these interactions at this scale would be beneficial or if the computational effort required would be worth it.
In conclusion, Cornell University’s research on quantizing LLM parameters using QuIP demonstrates the potential for enhancing the performance of large language models. With the insights provided by this research, it may be possible to achieve accurate 2-bit inference in LLMs, even with billions of parameters.
For more details, you can read the full paper on arXiv and find the code on GitHub. Please credit the researchers for their work.
If you’re interested in staying updated on the latest AI research news and projects, consider joining our ML SubReddit, Facebook Community, Discord Channel, and subscribing to our Email Newsletter. Follow us on Twitter for more updates.
About the author: Dhanshree Shenwai is a Computer Science Engineer with experience in fintech companies. She is passionate about AI applications and exploring new technologies to make life easier for everyone.