Automating Code Review Comments with Machine Learning
Code-change reviews are an essential part of software development at scale. They take up a significant amount of time for both the code authors and reviewers. At Google, millions of reviewer comments are received each year, with authors spending an average of around 60 minutes addressing the comments before submitting the changes. But with the help of machine learning (ML), we have an opportunity to automate and streamline the code review process.
Prediction of Code Edits
We started by training a model to predict the code edits needed to address reviewer comments. The model was pre-trained on various coding tasks and then fine-tuned for this specific task using reviewed code changes, reviewer comments, and the edits made by the author. We improved the model quality by iterating on the training dataset and experimenting with classifiers.
Serving Infrastructure and User Experience
To ensure a better user experience, we designed and implemented the feature on top of the trained model. We explored different user experience alternatives through user studies and refined the feature based on internal beta testing and user feedback. The final model was calibrated for a target precision of 50% to strike a balance between suggested edits and accuracy.
Architecture and Results
The ML-suggested edits infrastructure processed code and infrastructure from multiple services and provided suggestions to the user through the code review tool and integrated development environment (IDE). Offline evaluations showed that the model addressed 52% of comments with a target precision of 50%. During beta testing, 40% to 50% of previewed suggested edits were applied by code authors. We also made improvements based on user feedback, increasing the overall fraction of comments addressed with an ML-suggested edit by a factor of 2.
Future Work
In conclusion, we have introduced an ML-assistance feature to reduce the time spent on code review related changes. This feature has had a positive impact on productivity and allows developers to focus on more creative tasks. In the future, we plan to further refine and improve the model to provide even more accurate and helpful suggestions.
By implementing machine learning in the code review process, we expect to save hundreds of thousands of hours annually at Google scale. This automation will significantly improve efficiency and productivity for developers.