Table of contents
In this article
Technical page · Models
Parameter-efficient fine-tuning
LoRA, QLoRA and DoRA in practice
- Published
- 8 November 2026
- Last reviewed
- 8 November 2026
- Re-checked every
- 90 days
- Reviewed by
- Ales Analytics engineering
In one paragraph
Parameter-efficient fine-tuning adapts a model by training a small number of new parameters while the original weights stay frozen. LoRA, the most widely used method, adds a low-rank update to selected weight matrices; QLoRA does the same with the frozen model held in 4-bit precision, so large models can be adapted on modest hardware. The result is an adapter of a few megabytes to a few hundred megabytes that can be merged into the model or swapped at serving time.
How it works
- LoRA. A frozen matrix W (d × k) is used as W + (α / r) · B · A, with B of size d × r and A of size r × k. Only A and B are trained. For a 4096 × 4096 matrix at r = 16, trainable parameters fall from about 16.8 million to about 131 thousand.
- rsLoRA. Scales by α / √r so higher ranks train stably.
- QLoRA. Stores the frozen model in 4-bit NF4 with double quantisation, and uses paged optimisers to absorb memory spikes.
- DoRA. Splits each weight into magnitude and direction and applies the low-rank update to the direction.
- Targets. Adapters are usually applied to the attention projections and often the MLP layers; which ones is a tuning choice.
Design choices and trade-offs
| Choice | Trade-off |
|---|---|
| Rank r | Higher rank can learn more; more parameters and more risk of overfitting small datasets |
| Merge vs hot-swap | Merging gives the simplest serving; keeping adapters separate lets one base model serve many tasks |
| QLoRA vs LoRA in 16-bit | QLoRA fits larger models on smaller GPUs; 16-bit training is faster where memory allows |
Data hygiene matters more than any of these: deduplicate, remove personal data not needed for the task, and keep a held-out split that shares no near-duplicates with training.
When not to use it
When the gap is missing knowledge rather than behaviour, use retrieval. When thousands of high-quality examples exist and the budget allows, full fine-tuning may perform better.
How we evaluate it
Compare base model, prompted base model and adapted model on the same frozen test set, plus a general-ability check to catch forgetting. Training and evaluation data are checked for overlap before results are trusted.
Sources
- Hu et al. (2021), LoRA (opens in a new tab); Dettmers et al. (2023), QLoRA (opens in a new tab); Liu et al. (2024), DoRA (opens in a new tab)
- Hugging Face PEFT LoRA guide (opens in a new tab), TRL v1 (opens in a new tab), Axolotl (opens in a new tab)
Should you have any questions regarding this page, please contact us at solutions@alesanalytics.com.