Machine learning techniques are being used to improve the alignment of large language models (LLMs) with human values, which is becoming crucial in creating next-generation text-based assistants. Aligning LLMs aims to enhance the accuracy, coherence, and safety of the content generated in response to user queries. This study explores the impact of different feedback protocols on the alignment process, focusing specifically on ratings and rankings feedback.
### Understanding Feedback Protocols: Ratings vs. Rankings
In the comparison of feedback protocols, ratings involve assigning a score to a response using a predefined scale, while rankings require annotators to select their preferred response from a pair. This study identifies inconsistency issues in both human and AI feedback data, highlighting the significant impact of feedback acquisition protocols on the alignment pipeline.
### Exploring Feedback Inconsistency
The study uncovers a consistency problem in feedback data as it varies based on the protocols used for obtaining feedback. A range of 40%−42% accuracy was observed for both humans and AI, indicating that the choice of feedback protocol can significantly impact the alignment of LLMs.
### Feedback Data Acquisition
The researchers used diverse instructions to collect feedback data and found that AI feedback was more balanced compared to human annotators. An analysis revealed that GPT-3.5-Turbo was close to human gold label ratings and rankings for the responses.
The study also assessed the impact of feedback protocols on alignment and model evaluation. The findings indicate that the choice of feedback protocol significantly influences the alignment pipeline and the performance of LLMs during evaluations.
In conclusion, the study emphasizes the importance of meticulous data curation within sparse feedback protocols and highlights the need to explore richer forms of feedback for a more comprehensive understanding and improved alignment of LLMs. The study acknowledges limitations and points to the need for further research to develop more robust and universally applicable alignment methodologies in artificial intelligence.