Transformers: The Future of Artificial Intelligence
Transformers have emerged as one of the most groundbreaking innovations in the field of artificial intelligence (AI). Introduced in 2017, these neural network architectures have completely transformed how machines understand and generate human language. Unlike their predecessors, transformers utilize self-attention mechanisms to process input data simultaneously, allowing them to capture hidden relationships and dependencies within sequences of information. This parallel processing capability not only speeds up training times but also enables the development of highly sophisticated and high-performing models like ChatGPT.
In recent years, we have witnessed the impressive capabilities of artificial neural networks in various tasks, such as language and vision tasks. However, the real potential lies in crossmodal tasks, where these networks integrate different sensory modalities, like vision and text. By incorporating additional sensory inputs, these models have achieved remarkable results in tasks that involve understanding and processing information from multiple sources.
The Molyneux Problem, a thought experiment proposed by philosopher William Molyneux in 1688, has captivated scholars for centuries. It raises the question of whether a person blind from birth can recognize objects visually after gaining their sight. Vision neuroscientists embarked on a mission to solve this age-old puzzle in 2011. While immediate visual recognition of touch-only objects was found to be impossible, the study revealed that our brains are incredibly adaptable. Individuals who underwent sight-restoring surgery could quickly learn to recognize objects visually, bridging the gap between different sensory modalities. But what about multimodal neurons? Let’s find out.
We are currently in the midst of a technological revolution, where artificial neural networks trained on language tasks have demonstrated great success in crossmodal tasks involving vision and text integration. One popular approach is to use an image-conditioned version of prefix-tuning, where a separate image encoder is aligned with a text decoder through a learned adapter layer. While previous methods relied on image encoders like CLIP trained alongside language models, a recent study called LiMBeR introduced a unique scenario resembling the Molyneux Problem. They connected a self-supervised image network called BEIT, which had never encountered linguistic data, to a language model called GPT-J using a linear projection layer trained on an image-to-text task. This setup raises fundamental questions regarding the translation of semantics between modalities.
The research conducted by MIT aims to shed light on this centuries-old mystery and understand how multimodal models work. First, the researchers found that the transformer’s embedding space does not encode interpretable semantics for image prompts. Instead, the translation between modalities occurs within the transformer itself. Second, they discovered the existence of multimodal neurons within the text-only transformer MLPs. These neurons have the ability to process both image and text information with similar semantics, playing a crucial role in translating visual representations into language. Most importantly, modulating these multimodal neurons can remove specific concepts from image captions, demonstrating their significance in multimodal content understanding.
This investigation into the inner workings of individual units in deep networks has uncovered valuable insights. Just as convolutional units in image classifiers detect colors and patterns, and later units recognize object categories, transformers have been found to harbor multimodal neurons. These neurons selectively respond to images and text with similar semantics, even when vision and language are learned separately. They effectively convert visual representations into coherent text. This alignment across modalities has far-reaching implications, making language models powerful tools for sequential modeling tasks such as game strategy prediction and protein design.
To learn more about this exciting research, check out the paper and project. All credit goes to the researchers involved in this project. Don’t forget to join our ML SubReddit, Facebook community, Discord channel, and email newsletter for the latest AI research news and cool projects. If you enjoy our work, you’ll love our newsletter.
About the Author:
Ekrem Çetinkaya received his B.Sc. in 2018 and M.Sc. in 2019 from Ozyegin University. He completed his Ph.D. in 2023 at the University of Klagenfurt, focusing on video coding enhancements using machine learning for HTTP adaptive streaming. His research interests include deep learning, computer vision, video encoding, and multimedia networking.
(Sponsored) The End of Project Management by Humans