CM3leon by Meta
CM3leon stands as a cutting-edge generative model, unlocking the capabilities of both text-to-image and image-to-text generation. This model boasts multimodal functionality, fusing the strengths of autoregressive models with cost-effective training and efficient inference.
The training process of CM3leon follows a tailored recipe, borrowing elements from text-focused language models. It encompasses retrieval-augmented pre-training and multitask supervised fine-tuning phases. Even when utilizing only a fraction of the computational resources compared to prior transformer-based methods, CM3leon attains a state-of-the-art performance in text-to-image generation.
CM3leon is adept at generating text and images that are conditioned upon arbitrary sequences of other text and image inputs. This elevates its capabilities beyond previous models, which were confined to either text-to-image or image-to-text generation. The model has undergone multitask instruction fine-tuning for both image and text generation, resulting in substantial enhancements across various tasks, such as image captioning, visual question answering, text-based editing, and conditional image generation.
Notably, CM3leon surpasses Google’s text-to-image model, boasting an impressive Fréchet Inception Distance (FID) score of 4.88 on the widely accepted image generation benchmark, establishing a new pinnacle in this field. Its strengths are particularly evident in the realms of intricate object generation and text-guided image manipulation tasks.
CM3leon excels in producing coherent visuals that align with input prompts, even when faced with constraints and complex compositional structures. Moreover, the model demonstrates remarkable proficiency in tasks like text-guided image editing, text-to-image generation based on compositional prompts, and answering questions about images. Remarkably, despite being trained on a relatively modest dataset, CM3leon’s zero-shot performance rivals that of larger models trained on more extensive datasets.
The accomplishments of CM3leon underscore the potential of retrieval augmentation and the impact of scaling strategies on the performance of autoregressive models. With its versatility and exceptional performance, CM3leon emerges as an invaluable tool for a wide spectrum of vision-language tasks.