Natural Language Processing has made significant progress in recent years. Advanced language models like GPT 3.5, GPT 4, BERT, and PaLM have improved performance in various natural language tasks such as translation and reasoning. To evaluate these models, benchmarks are used to test their abilities in tasks. However, existing benchmarks like GLUE and SuperGLUE are no longer effective in assessing the capabilities of these models.
To address this issue, a team of researchers has developed a new benchmark called ARB (Advanced Reasoning Benchmark). Unlike previous benchmarks, ARB focuses on complex reasoning problems in areas like mathematics, physics, biology, chemistry, and law. The team has also included a subset of math and physics questions that require sophisticated thinking and subject knowledge.
The team evaluated models like GPT-4 and Claude on the ARB benchmark and found that these models struggled with the complexity of the tasks, scoring below 50%. To improve the evaluation process, a rubric-based approach was introduced that allows GPT-4 to evaluate its own intermediate reasoning processes. Human annotators also solved the problems and their evaluations aligned reasonably well with the model’s self-assessment.
The ARB benchmark includes a mix of short-answer and open-response questions, making it more challenging for language models to be evaluated. This combination of expert-level reasoning tasks and realistic question formats provides a more accurate assessment of the models’ ability to handle real-world problems.
You can find more information about the ARB benchmark in the paper, Github, and project links provided.