Generating Images with Contrastive Models

Contrastive models like CLIP have proven their ability to learn robust image representations that encompass both semantics and style. To harness these representations for image generation, a two-stage model is proposed. The first stage generates a CLIP image embedding based on a text caption, while the second stage utilizes a decoder to generate an image conditioned on the image embedding. This approach not only enhances image diversity but also maintains photorealism and caption similarity.

Improved Image Diversity and Preservation of Semantics and Style

The explicit generation of image representations vastly improves image diversity without compromising on the realism of the generated images or their similarity to the provided captions. By conditioning the decoders on image representations, it becomes possible to generate variations of an image while preserving both its semantics and style. Additionally, the non-essential details missing from the image representation can be experimented with through this approach.

Language-Guided Image Manipulation and Zero-Shot Learning

The joint embedding space of CLIP allows for language-guided image manipulations without the need for prior training. With this capability, images can be modified according to specific textual instructions, even if these instructions were not encountered during training. This zero-shot learning improves the flexibility and adaptability of the model.

Efficiency and Sample Quality

The decoder leverages diffusion models, and both autoregressive and diffusion models are considered for the prior stage. However, it is observed that diffusion models are more computationally efficient while delivering higher-quality generated samples.