The Power of Curious Exploration with AI

AI-powered robots are now capable of exploring their environments and learning new skills through curiosity. The JACO arm and OP3 humanoid robot provide fascinating examples of how curiosity learning can lead to the emergence of complex and meaningful behaviors.

Exploring the Environment with Curiosity

Intrinsic motivation, a concept that drives exploration without specific tasks, is a key factor in curiosity learning. By training a predictive model or a “world model,” the AI agent can make predictions about the outcomes of its actions and compare them with the real observations. This helps the agent receive rewards for taking unpredictable actions and updates the world model to improve future predictions.

The Emergence of Diverse Behaviors

Curiosity learning has proven successful in various settings, such as beating computer games and training adaptable policies for different tasks. But the real value lies in the diverse behaviors that emerge during the exploration process. These behaviors could be valuable for future tasks if they are retained and not overwritten.

Introducing SelMo and Retaining Emerging Behaviors

In a recent paper, researchers introduce SelMo, an off-policy method for self-motivated exploration. They demonstrate that meaningful and diverse behaviors can emerge solely through optimizing the curiosity objective in simulated manipulation and locomotion domains. The researchers also propose focusing on identifying and retaining these emerging intermediate behaviors, as they could enhance the learning of new tasks in hierarchical reinforcement learning systems.

Curiosity in Action: JACO Arm and OP3 Robot

SelMo was tested on a JACO arm and an OP3 humanoid robot. The JACO arm learned to pick up and move cubes without any supervision, while the OP3 robot learned to balance on one foot and sit down safely. These impressive behaviors emerged as a result of curiosity-based exploration.

Retaining Emergent Behaviors for Skill Reuse

The drawback of curious exploration is that emergent behaviors are not persistent and keep changing with the curiosity reward function. However, retaining these behaviors can equip the AI agent with valuable skills for faster learning of new tasks. To investigate this, an experiment was conducted, using self-discovered behaviors as auxiliary skills. The results showed that using these auxiliary skills significantly accelerated the learning of a new target skill.

In conclusion, curiosity-driven exploration has immense potential in AI research and development. By harnessing the emergent behaviors and retaining them, AI agents can become more versatile and efficient learners, paving the way for advancements in unsupervised reinforcement learning.