Large Language Models (LLMs) are typically used for tasks like translating languages and generating code. However, researchers at Meta AI wanted to see if LLMs could handle the complexity of code optimizations that regular compilers, like LLVM, perform. They didn’t think LLMs were capable of this because compiler optimizations involve complex thinking and mathematical techniques. But to their surprise, the results showed that a well-trained LLM can not only predict the best optimizations for code, but it can also directly perform those optimizations without relying on the compiler.
The researchers used a 7-billion-parameter LLM architecture sourced from LLaMa 2 and trained it on a massive dataset of LLVM assembly examples. Each example was paired with the best compiler options and the resulting optimized code. Through this training, the LLM gained the ability to optimize code with remarkable precision.
The researchers’ work is significant because they are the first to apply LLMs to the task of code optimization. Their LLMs achieved a 3.0% improvement in code size reduction on a single compilation, compared to a search-based approach that only achieved a 5.0% improvement with 2.5 billion compilations. Machine learning approaches typically result in regressions and require thousands of compilations. The researchers also provided supplementary experiments and code examples to further demonstrate the potential and limitations of LLMs in code reasoning.
If you’re interested in learning more about this research, you can check out the paper. And don’t forget to join our ML subreddit, Facebook community, Discord channel, and subscribe to our email newsletter for the latest AI research news and projects.
About the author: Janhavi Lande is an upcoming data scientist who has been working in the world of ML/AI research for the past two years. She is fascinated by the ever-changing world of technology and enjoys traveling, reading, and writing poems.
🚀 Sponsored: Check out Height, the end of project management by humans. – Advertisement –