An artificial intelligence language model called GPT-3 is impressively good at solving reasoning problems, according to research conducted by UCLA psychologists. The study, published in Nature Human Behaviour, found that GPT-3 performs at a similar level to college undergraduates on tasks typically found on intelligence and standardized tests. However, the researchers are unsure if GPT-3 is simply imitating human reasoning or using a new cognitive process. They highlight that while GPT-3 excels at certain reasoning tasks, it fails spectacularly at others.
Reasoning Abilities of GPT-3
GPT-3’s reasoning abilities were tested using problems inspired by Raven’s Progressive Matrices, a complex test involving predicting the next image in a series of shapes. In this study, the shapes were converted into a text format so that GPT-3 could process them. Surprisingly, GPT-3 performed as well as humans on these tests and made similar mistakes. It solved 80% of the problems correctly, which is higher than the average score of the human subjects.
The researchers also tested GPT-3 on a set of SAT analogy questions that have never been published online. GPT-3 outperformed the average human score on these questions. However, when asked to solve analogies based on short stories, GPT-3 did not perform as well as human students. The researchers noted that GPT-4, the latest iteration of OpenAI’s technology, performed better than GPT-3.
GPT-3’s Limitations
The study found that GPT-3 struggles with problems that require understanding physical space. For example, when given a set of tools and asked to describe how to transfer gumballs from one bowl to another, GPT-3 provided nonsensical solutions. The researchers developed their own computer model inspired by human cognition, and it outperformed commercial AI models on analogy problems until GPT-3 was introduced. The researchers are curious to explore whether GPT-3 is truly thinking like a human or if it’s using a completely different method.
Understanding AI’s Cognitive Processes
The researchers acknowledge that in order to determine how AI models like GPT-3 reason, they would need access to the software and the training data. This would enable them to administer tests that the software hasn’t been exposed to yet. They believe this is the next step in understanding the potential of AI and deciding its future.