Hugging Face: PEFT v0.17.0: MiSS replaces Bone, SHiRA adds sparse adapters; LoRA targets nn.Parameter | SignalBreak | SignalBreak

HighDeprecation

PEFT v0.17.0: MiSS replaces Bone, SHiRA adds sparse adapters; LoRA targets nn.Parameter

Action Required

Bone-based PEFT workflows must migrate to MiSS before Bone removal in PEFT v0.19.0, or training using Bone checkpoints will fail.

AI Impact Summary

SHiRA introduces Sparse High Rank Adapters that train only 1-2% of weights, potentially reducing adapter-switching costs and improving concept loss versus LoRAs. MiSS is introduced as a successor to Bone, promising better performance and memory efficiency, with Bone slated for removal in PEFT v0.19.0 and a script to convert Bone checkpoints to MiSS. LoRA now supports targeting nn.Parameter directly (target_parameters), which is useful for MoE models like Llama4, though it remains experimental and may increase memory usage; additional injection options via inject_adapter_in_model and state_dict-based adapter injection are provided to handle unknown configs. Changes also include fixes for prompt-learning module saving and MoE weight handling, along with compatibility considerations for large MoE parameter layouts.

PEFT v0.17.0: MiSS replaces Bone, SHiRA adds sparse adapters; LoRA targets nn.Parameter

Affected Systems

Date: Date not specified
Change type: deprecation
Severity: high

Checking your AI register…

More from Hugging Face

Get alerts for Hugging Face

SignalBreak monitors Hugging Face and 27 other AI providers across 150+ endpoints. Sign up free to get notified when things change.

Sign up free — no credit card required

Source text

Highlights New Methods SHiRA @kkb-code contributed Sparse High Rank Adapters (SHiRA, paper ) which promise to offer a potential gain in performance over LoRAs - especially the concept loss when using multiple adapters is improved. Since the adapters only train on 1-2% of the weights and are inherently sparse, switching between adapters may be cheaper than with LoRAs. ( #2584 ) MiSS @JL-er added a new PEFT method, MiSS ( Matrix Shard Sharing ) in #2604 . This method is an evolution of Bone , which, according to our PEFT method comparison benchmark , gives excellent results when it comes to performance and memory efficiency. If you haven't tried it, you should do so now. At the same time, Bone will be deprecated in favor of MiSS and will be removed in PEFT v0.19.0. If you already have a Bone checkpoint, you can use scripts/convert-bone-to-miss.py to convert it into a MiSS checkpoint and proceed with training using MiSS. Enhancements LoRA for nn.Parameter LoRA is now able to target nn.Parameter directly ( #2638 , #2665 )! Ever had this complicated nn.Module with promising parameters inside but it was too custom to be supported by your favorite fine-tuning library? No worries, now you can target nn.Parameters directly using the target_parameters config attribute which works similarly to target_modules . This option can be especially useful for models with Mixture of Expert (MoE) layers, as those often use nn.Parameter s directly and cannot be targeted with target_modules . For example, for the Llama4 family of models , use the following config to target the MoE weights: config = LoraConfig ( ..., target_modules = [], # <= prevent targeting any modules target_parameters = [ "feed_forward.experts.down_proj" , "feed_forward.experts.gate_up_proj" ], ) Note that this feature is still experimental as it comes with a few caveats and therefore might change in the future. Also, MoE weights with many experts can be quite huge, so expect a higher memory usage than compared to targ

View original source