Perceiver and Perceiver IO: Multi-purpose Tools for AI
The majority of AI systems in use today are designed for specific tasks and purposes. For example, a 2D residual network is great for processing images, but it’s not well-suited for other types of data, such as Lidar signals or robot torques. Standard architectures are often limited to one task and require engineers to modify inputs and outputs to fit the architecture’s requirements. Processing multiple types of data, like sound and images in videos, is even more complex and typically requires intricate systems with many components.
DeepMind, in its mission to advance science and humanity through intelligence, sought to develop a more versatile architecture that can handle various types of data. This led to the creation of the Perceiver, a general-purpose architecture capable of processing images, point clouds, audio, video, and combinations of these data types. However, the Perceiver was limited to tasks with simple outputs, like classification.
In a recent paper presented at ICML 2021 and published on arXiv, DeepMind introduced Perceiver IO. This new version of the Perceiver architecture is more generalized and can produce a wide range of outputs from different inputs. It is applicable to real-world domains, including language, vision, multimodal understanding, and even complex games like StarCraft II. DeepMind has generously open-sourced the code to support researchers and the wider machine learning community.
The Perceiver architecture builds on the Transformer, a well-known architecture that uses attention to map inputs to outputs. However, Transformers can become computationally expensive as the number of inputs grows, making them less effective for large datasets like images, videos, and books. The original Perceiver addressed this scalability issue by using attention to encode inputs into a small latent array. This array can then be further processed independent of the input’s size, allowing the Perceiver to handle larger inputs without compromising performance.
Perceiver IO takes this scalability one step further by using attention not only for encoding inputs but also for decoding from the latent array. This enhances the architecture’s flexibility, enabling it to accommodate both large inputs and outputs. With Perceiver IO, multiple tasks and different data types can be handled simultaneously, simplifying the overall architecture and making it applicable to a wide range of applications.
In various experiments, Perceiver IO has demonstrated its effectiveness across multiple domains, including language, vision, multimodal data, and games. It serves as a versatile tool for handling diverse data types without the need for building custom solutions with specialized systems. DeepMind hopes that by sharing their latest research and providing access to the code, researchers and practitioners can tackle problems more efficiently and effectively.
As DeepMind continues to explore and learn from different types of data, they are committed to further enhancing the general-purpose Perceiver architecture. By making it faster and easier to solve problems in science and machine learning, DeepMind aims to drive advancements that benefit society as a whole.