Research conducted by Google DeepMind and Stanford University reveals a significant weakness in Large Language Models (LLMs) when faced with reordered premises, leading to a drop in performance by over 30% in some cases. A new benchmark named R-GSM was created to study this phenomenon, showing that even slight changes in premise order can drastically impact LLMs’ ability to reach accurate conclusions. This highlights the need for more robust models capable of handling varying data inputs. Various advanced models, including GPT-4-turbo and GPT-3.5-turbo, experienced a decrease in accuracy on reordered problems, emphasizing the importance of addressing this issue for the future of LLM development. The study calls for reevaluation of LLM training and modeling techniques to improve reasoning capabilities and adaptability. By understanding and addressing the premise order effect, AI researchers can make significant strides in developing more intelligent and reliable reasoning models for a wide range of applications. This research sheds light on an important aspect of LLM behavior, paving the way for advancements in AI capabilities and the development of more versatile models. Stay updated on this research and other AI news by following us on Twitter and Google News. Join our ML SubReddit, Facebook Community, Discord Channel, and LinkedIn Group for more updates. Don’t forget to subscribe to our newsletter for the latest updates in the AI world.