Stability AI Releases Text-to-Image Model DeepFloyd IF

Stability AI and its multimodal AI research lab, DeepFloyd, have announced the research release of DeepFloyd IF, a cutting-edge text-to-image cascaded pixel diffusion model. The model is initially released under a non-commercial, research-permissible license, but an open-source release is planned for the future.

DeepFloyd IF boasts several remarkable features, including:

1. Deep text prompt understanding: The model uses T5-XXL-1.1 as a text encoder, with numerous text-image cross-attention layers, ensuring better alignment between prompts and images.
2. Coherent and clear text alongside generated images: DeepFloyd IF can generate images containing objects with varying properties and spatial relations.
3. High degree of photorealism: The model has achieved an impressive zero-shot FID score of 6.66 on the COCO dataset.
4. Aspect ratio shift: The model can generate images with non-standard aspect ratios, including vertical, horizontal, and the standard square aspect.
5. Zero-shot image-to-image translations: The model can modify an image’s style, patterns, and details while preserving its basic form.

Below are some of the example concepts created by DeepFloyd IF:

DeepFloyd IF’s modular, cascaded, pixel diffusion design consists of several neural modules interacting synergistically. The model works in pixel space, processing high-resolution data in a cascading manner using individually trained models at different resolutions. This involves a base model that generates low-resolution samples and successive super-resolution models that produce high-resolution images.

The model was trained on a custom high-quality LAION-A dataset containing 1 billion (image, text) pairs, a subset of the English part of the LAION-5B dataset. DeepFloyd’s custom filters were used to remove watermarked, NSFW, and other inappropriate content.

DeepFloyd IF’s process

Initially, DeepFloyd IF is released under a research license. The researchers aim to encourage the development of novel applications across domains such as art, design, storytelling, virtual reality, and accessibility. To inspire potential research, they have proposed several technical, academic, and ethical research questions.

Technical research questions include:

• Optimizing the IF model to enhance performance, scalability, and efficiency.
• Improving output quality by refining sampling, guiding, or fine-tuning the model.
• Applying techniques used to modify Stable Diffusion output to DeepFloyd IF.

Academic research questions include:

• Exploring the role of pre-training for transfer learning.
• Enhancing the model’s control over image generation.
• Expanding the model’s capabilities beyond text-to-image synthesis by integrating multiple modalities.
• Assessing the model’s interpretability to improve understanding of generated images’ visual features.

Ethical research questions include:

• Identifying and mitigating biases in DeepFloyd IF.
• Assessing the model’s impact on social media and content generation.
• Developing an effective fake image detector that utilizes the model.

To access the model’s weights, users must accept the license on DeepFloyd’s Hugging Face space. For more information, you can visit the model’s website, GitHub repository, Gradio demo, or join public discussions through DeepFloyd’s Linktree.