Perfusion by Nvidia

Perfusion by Nvidia is a text-to-image personalization method designed for users who need to creatively portray personalized objects while maintaining their identity. With a model size of only 100KB, Perfusion allows significant changes in the appearance of objects using a novel mechanism called key-locking. This method enables controlling the trade-off between visual and textual alignment at inference time, covering the entire Pareto front with just a single trained model. Perfusion produces appealing images effortlessly, and a batch size of 8 is typically sufficient to ensure several good samples. The key-locking mechanism avoids overfitting by introducing a new mechanism that locks new concepts' cross-attention keys to their superordinate category. Perfusion also develops a gated rank-1 approach that enables controlling the influence of a learned concept during inference time and combining multiple concepts. This allows runtime-efficient balancing of visual-fidelity and textual-alignment with a single 100KB trained model. Perfusion can enable more animate results, with better prompt-matching and less susceptibility to background traits from the original image. For each concept, Perfusion shows exemplars from the training set, along with generated images, their conditioning texts, and comparisons to Custom-Diffusion and Dreambooth baselines. Perfusion can generate images with both high visual-fidelity and textual-alignment when training with a single image. The method can also generalize to fine-tuned variants. Perfusion is particularly useful for users who need to personalize text-to-image models for specific use cases, such as generating images of objects with unique appearances or combining multiple concepts into a single image. These users can benefit from Perfusion's ability to control the trade-off between visual and textual alignment, allowing for more flexibility and creativity in the image generation process. Additionally, Perfusion's small model size and efficient inference process make it a valuable tool for users who need to generate high-quality images quickly and efficiently.