In 2016, AlphaGo became the first AI program to defeat humans in the game of Go. Two years later, AlphaZero surpassed its predecessor by mastering Go, chess, and shogi. Now, a new breakthrough has been achieved with MuZero, a general-purpose AI algorithm. The remarkable thing about MuZero is that it can master these games without being given the rules beforehand.
For a long time, researchers have been trying to develop methods that can learn a model of their environment and use it to plan actions. However, most approaches have struggled in complex domains where the rules are unknown, like the Atari games. MuZero solves this problem by focusing on the most important aspects of the environment for planning. By combining this model with AlphaZero’s lookahead tree search, MuZero achieved the best results on the Atari benchmark and matched AlphaZero’s performance in Go, chess, and shogi.
The ability to plan is crucial to human intelligence, and we want our AI algorithms to have the same capability. There have been two main approaches to tackle this challenge: lookahead search and model-based planning. Lookahead search has been successful in classic games but relies on knowing the environment’s dynamics. Model-based planning aims to learn an accurate model of the environment but struggles in visually rich domains like Atari.
MuZero takes a different approach by only modeling the important aspects for decision-making, such as the value, policy, and reward. These are learned using a neural network and enable MuZero to understand the consequences of its actions and plan accordingly. The experience it collects from interacting with the environment is used to train its network. Additionally, MuZero can repeatedly use its learned model to improve planning, rather than collecting new data.
MuZero’s performance was tested in Go, chess, shogi, and Atari games. It outperformed all prior algorithms in the Atari benchmark and matched AlphaZero’s superhuman performance in the other games. Furthermore, experiments showed that increasing the amount of planning time per move improved MuZero’s playing strength and learning speed.
Overall, MuZero’s ability to learn a model and use it for planning represents a significant advancement in reinforcement learning and general-purpose algorithms. It opens up new possibilities for applications in various domains.