Fast LoRA inference for Flux.1-Dev with Diffusers and PEFT — ~2.3x speedups via FA3, FP8, and hot-swapping

AI Impact Summary

The post describes a performance optimization recipe for LoRA-enabled diffusion pipelines on Flux.1-Dev that combines FA3 (Flash Attention 3), FP8 quantization, and hot-swapping with torch.compile to avoid recompilation when swapping LoRA adapters. This enables significant inference speedups (around 2.3x) and more flexible model customization, even on consumer GPUs with CPU offload considerations. Practical constraints include max_rank predefinition across adapters, restricted target layers (no text encoder support yet), FP8 lossy trade-offs, and the first invocation still being slow due to JIT compilation; these factors shape deployment and runtime tuning for production workloads.