Instruction-tuning Stable Diffusion with InstructPix2Pix

AI Impact Summary

Stable Diffusion can be instruction-tuned using a method inspired by FLAN and InstructPix2Pix, allowing it to follow specific image editing instructions like "cartoonize the image." This approach leverages paired datasets created by prompting ChatGPT and utilizing pre-trained models like Whitebox CartoonGAN, effectively transferring biases and improving performance on tasks beyond the initial capabilities of the pre-trained InstructPix2Pix model. This technique offers a flexible way to customize Stable Diffusion for specific image transformations.