AI Models May Not Replicate Human Decisions, MIT Researchers Find
Machine-learning models designed to mimic human decision-making may not accurately replicate human judgments when it comes to rule violations, according to researchers from MIT. The problem lies in the training data. If models are not trained with the right data, they are likely to make different and often harsher judgments than humans would. In particular, models trained with descriptive data, which focuses on factual features, tend to over-predict rule violations. This disparity in accuracy could have serious implications in real-world applications, such as determining bail amounts or criminal sentences.
The researchers conducted a study to explore the differences between descriptive and normative labels. Descriptive labels focus on factual features, while normative labels determine whether data violate a certain rule. They found that participants were more likely to label an object as a violation when using descriptive labels compared to normative labels. However, machine-learning models are typically trained using descriptive data and then later used to make normative judgments.
To test the impact of using descriptive data, the researchers trained two models: one with descriptive data and the other with normative data. They found that the model trained with descriptive data underperformed compared to the model trained with normative data. Specifically, the model trained with descriptive data was more likely to misclassify inputs and had lower accuracy when classifying objects that human labelers disagreed about.
Improving transparency in dataset collection could help mitigate this issue. By knowing how data were gathered, researchers can better understand how they should be used. Additionally, the researchers suggest fine-tuning a descriptively trained model with a small amount of normative data to improve performance. They also plan to conduct a similar study with expert labelers to see if the label disparity persists.
Overall, it is crucial to acknowledge the impact of training data on AI models. Only by using data collected in the specific judgment setting can models accurately reproduce human judgments. Without this transparency and proper training, AI systems may exhibit overly harsh moderations and lack the nuanced decision-making abilities of humans.
This research was supported by various organizations, including the Schwartz Reisman Institute for Technology and Society, Microsoft Research, and the Vector Institute.