Accelerated LoRA inference for Flux.1-Dev with Diffusers and PEFT (FA3, FP8, hotswap)

AI Impact Summary

This guidance describes a practical optimization recipe for LoRA inference with the Flux.1-Dev diffusion model, combining DiffusionPipeline, FA3 (Flash Attention 3), FP8 quantization via TorchAO, and torch.compile to improve latency while keeping LoRA adapters hot-swappable. Reported results show ~2.23x speedups on optimized paths and memory reductions suitable for consumer GPUs with CPU offload, expanding feasible deployments beyond high-end GPUs. Key caveats include requiring max_rank across all adapters, limiting LoRA targets to the same layers (text encoder not supported yet), and potential recompilation stalls if hot-swapping is not properly managed.