Scaling neural networks has become a popular trend in recent years. Deep networks are being designed with increased depth to enhance their expressiveness. To effectively expand the hidden dimension of the neural network, sparse MoE models and model parallelism techniques are employed. However, the sequence length, which is the last atomic dimension of the neural network, plays a significant role in its performance.
Removing the restriction on sequence length offers several benefits. Firstly, it allows models to have a larger memory and receptive field, enabling them to interact with people and the outside environment. Secondly, longer sequences include more complex causal chains and thought processes, which contribute to better training data for the models. On the other hand, shorter sequences tend to have more erroneous correlations, which can hinder generalization. Additionally, longer sequences can address the issue of catastrophic forgetting in models during many-shot education.
The main challenge in scaling up the sequence length is finding the right balance between computational complexity and model expressivity. RNN-style models aim to extend the sequence length, but training parallelization is limited due to their sequential nature. State space models, on the other hand, combine the strengths of CNN and RNN models and perform well at long-range benchmarks. However, they fall short compared to Transformers in terms of expressivity.
To overcome this limitation, researchers have explored techniques to reduce the complexity of self-attention in Transformers. This includes implementing sliding windows or convolution modules over the attention, which can make the complexity almost linear. Another approach is using sparse attention, which sparsifies the attention matrix while retaining the ability to recall distant information. These techniques have shown promise in scaling the sequence length.
Researchers from Microsoft Research have developed LONGNET, a model that swaps the focus of conventional Transformers with dilated attention. This enables a logarithmic dependence between tokens and linear processing complexity, addressing the conflict between accessibility of all tokens and limited attention resources. LONGNET can be converted into a dense Transformer implementation, allowing for standard optimizations without issues. This linear complexity allows for efficient scaling of the sequence length to 1 billion tokens.
In conclusion, scaling the sequence length in neural networks is crucial for improving their performance. Various techniques and models, such as sparse attention and LONGNET, have shown promising results in extending the sequence length. These advancements contribute to the overall progress in AI research and development.
Check out the Paper and Github link for more information on this topic. Don’t forget to join our ML SubReddit, Discord Channel, and Email Newsletter for the latest AI research news and updates. If you have any questions or feedback, feel free to reach out to us.