Communicating with others is important, but humans can only use a limited number of words to explain what they see in the world. This shows that the words we use are connected to what we see, especially when it comes to vision. Scientists have found that the brain uses the same kind of information when processing language and visuals. For example, when you hear the word “cat,” your brain creates an image of a cat.
In the past, researchers have been able to recreate visual content using brain scans. However, the images were often blurry and didn’t make much sense. On the other hand, studies have shown that the visual cortex of the brain can understand language. This has led to the development of new technology that can translate what you see into words. This technology could be used to help people with brain injuries or disabilities communicate.
Researchers at Zhejiang University have created a non-invasive neural language decoder called MindGPT. It takes the patterns created in the brain when you see something and turns them into words. The researchers used a language model called GPT-2 and a method called CLIP-guided fMRI encoding. This allows the decoder to understand the meaning behind what the brain is seeing and translate it into natural English.
The MindGPT decoder has been trained to recognize visual cues and understand language. It can accurately describe what it sees and understand the meaning behind it. The researchers found that even with minimal training data, the decoder was able to pick up on visual cues and understand them. They also found that the decoder’s brain representations were consistent with findings in neuroscience.
Overall, the MindGPT decoder is a breakthrough in understanding how the brain processes visual and linguistic information. It shows that we can understand the relationship between what we see and what we say without relying on temporal resolution. This technology has the potential to revolutionize communication for people with brain injuries or disabilities.
To learn more about the research and see the decoder in action, check out the paper and Github. And don’t forget to join our ML SubReddit, Facebook Community, Discord Channel, and Email Newsletter for the latest AI news and updates. If you like our work, you’ll love our newsletter. We’re also on WhatsApp, so join our AI Channel there too.
Meet Aneesh Tickoo, a consulting intern at MarktechPost. He’s studying Data Science and Artificial Intelligence at the Indian Institute of Technology. Aneesh is passionate about image processing and loves collaborating on interesting projects. Check out our YouTube channel for AI research updates.
[Watch Now]