Introduction to Generative Modeling in AI
Generative Modeling is a type of machine learning where the model learns to find patterns in input data and use that knowledge to generate new, similar data. This is a form of unsupervised learning. There have been many advancements in generative AI, with different types of networks being used, such as autoregressive models, deep VAEs, and diffusion models. However, these models have limitations when it comes to continuous or discrete data.
Introducing Bayesian Flow Networks (BFNs)
Researchers have introduced a new type of generative model called Bayesian Flow Networks (BFNs). BFNs can be thought of in the context of Alice and Bob. Bob starts with an initial distribution and uses it to generate new parameters for an “output distribution.” Alice adds noise to the data in a planned way to create a “sender distribution.” Bob combines the output distribution with the same noise to create a “receiver distribution.” He considers all possible data values and their probabilities according to the output distribution.
Alice sends a sample from her sender distribution to Bob, who updates his initial distribution using Bayesian rules based on this sample. Bob repeats this process multiple times until his predictions become accurate enough for Alice to send the data without noise.
This process creates a loss function for a certain number of steps, which can also be extended to continuous time. In continuous time, the Bayesian updates become a flow of information from the data to the network, known as the Bayesian flow. BFNs trained with continuous-time loss can be run for any number of discrete steps during inference and sampling, with better performance as the number of steps increases.
BFNs for Continuous Data
For continuous data, BFNs are most closely related to variational diffusion models, with a similar continuous-time loss function. The main difference is that the network inputs in BFNs are less noisy compared to variational diffusion and other continuous diffusion models. This is because the generative process of BFNs starts with a fixed prior, while diffusion models start with pure noise.
Experimental Results and Conclusion
The framework of BFNs has been applied to continuous, discrete, and discretized data. Experimental results were carried out over CIFAR-10 (color images), dynamically binarized MNIST (handwritten digits), and text8 (character sequences), and BFN outperformed other models in all benchmarks. This study provides a fresh perspective on BFNs in generative modeling and opens up new possibilities in this field.
Check out the paper here.
If you like this research, make sure to follow us on Twitter!