Language and its role in demonstrating comprehension are vital aspects of human intelligence. It allows us to communicate thoughts, express ideas, create memories, and foster mutual understanding. DeepMind recognizes the importance of language and its processing in both artificial agents and humans. As part of their AI research, DeepMind focuses on developing more powerful language models that can predict and generate text. These models have the potential to be used in various applications such as summarizing information, providing expert advice, and following instructions through natural language.
DeepMind acknowledges the need for research into the impacts and risks associated with these language models. This requires collaboration between experts from diverse backgrounds to anticipate and tackle the challenges that arise from training algorithms on existing datasets.
In line with their interdisciplinary approach, DeepMind has released three papers on language models. These papers explore a 280 billion parameter transformer language model called Gopher, analyze the ethical and social risks associated with large language models, and propose a new architecture for better training efficiency.
Gopher, the largest language model developed by DeepMind, demonstrates significant advancements in various tasks compared to existing models. It outperforms prior work in the Massive Multitask Language Understanding (MMLU) benchmark, approaching human expert performance. Additionally, Gopher shows surprising coherence in dialogue interactions.
However, DeepMind’s research also identifies certain failures across different model sizes, including repetition, propagation of biases, and the dissemination of incorrect information. Understanding and documenting these failure modes is critical in addressing potential harms that could arise from large language models.
DeepMind’s second paper focuses on the ethical and social risks associated with language models. They present a taxonomy of these risks and failure modes, highlighting the need for a broad perspective on potential issues. Narrowly focusing on one risk can worsen other problems. This taxonomy serves as a foundation for experts and public discourse to make responsible decisions and develop strategies to mitigate risks.
Furthermore, DeepMind highlights the need for improved benchmarking tools to assess risks related to misinformation and insufficient analysis of human-computer interaction. They emphasize the importance of ongoing research efforts to address the reproduction of harmful social stereotypes.
DeepMind’s final paper introduces the Retrieval-Enhanced Transformer (RETRO), an improved language model architecture. RETRO reduces the energy cost of training and allows for better traceability of model outputs to their sources. It achieves comparable performance to regular Transformers with fewer parameters and excels in various language modeling benchmarks.
These papers lay the groundwork for DeepMind’s future language research, particularly in areas that impact the evaluation and deployment of language models. DeepMind aims to ensure safe interactions with AI agents and actively explores natural language explanations, communication to reduce uncertainty, and language-based approaches to complex decision-making.
DeepMind approaches their research on language models with caution and thoughtfulness. They strive for transparency by acknowledging the limitations of their models and actively working to mitigate identified risks. The multidisciplinary expertise of their teams in language, deep learning, ethics, and safety is instrumental in creating large language models that benefit society and advance scientific understanding, aligning with DeepMind’s mission.