LM Studio 0.3.10: Speculative Decoding for Faster Inference

AI Impact Summary

LM Studio version 0.3.10 introduces Speculative Decoding, a technique that speeds up token generation by up to 3x using a combination of a larger and smaller model. This feature leverages a draft model to predict tokens, accepting or rejecting them based on the main model's confirmation, optimizing inference speed without sacrificing quality. This update is primarily focused on improving performance for models like Llama 8B and Llama 1B, particularly on Apple M3 Macs, and offers a new sidebar for managing this feature.