# Parameter-efficient fine-tuning

> How LoRA, QLoRA and DoRA adapt a frozen model with few trainable parameters, and the data hygiene that decides whether it works.

- Web page: https://www.alesanalytics.com/docs/parameter-efficient-fine-tuning
- Markdown: https://www.alesanalytics.com/docs/parameter-efficient-fine-tuning.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. LoRA, QLoRA and DoRA in practice. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

Parameter-efficient fine-tuning adapts a model by training a small number of new parameters while the original weights stay frozen. LoRA, the most widely used method, adds a low-rank update to selected weight matrices; QLoRA does the same with the frozen model held in 4-bit precision, so large models can be adapted on modest hardware. The result is an adapter of a few megabytes to a few hundred megabytes that can be merged into the model or swapped at serving time.

## How it works

- **LoRA.** A frozen matrix W (d × k) is used as W + (α / r) · B · A, with B of size d × r and A of size r × k. Only A and B are trained. For a 4096 × 4096 matrix at r = 16, trainable parameters fall from about 16.8 million to about 131 thousand.
- **rsLoRA.** Scales by α / √r so higher ranks train stably.
- **QLoRA.** Stores the frozen model in 4-bit NF4 with double quantisation, and uses paged optimisers to absorb memory spikes.
- **DoRA.** Splits each weight into magnitude and direction and applies the low-rank update to the direction.
- **Targets.** Adapters are usually applied to the attention projections and often the MLP layers; which ones is a tuning choice.

![Figure 1. LoRA: a frozen weight plus a trainable low-rank path.](https://www.alesanalytics.com/diagrams/ma-lora.svg)

## Design choices and trade-offs

| Choice                  | Trade-off                                                                                          |
| ----------------------- | -------------------------------------------------------------------------------------------------- |
| Rank r                  | Higher rank can learn more; more parameters and more risk of overfitting small datasets            |
| Merge vs hot-swap       | Merging gives the simplest serving; keeping adapters separate lets one base model serve many tasks |
| QLoRA vs LoRA in 16-bit | QLoRA fits larger models on smaller GPUs; 16-bit training is faster where memory allows            |

Data hygiene matters more than any of these: deduplicate, remove personal data not needed for the task, and keep a held-out split that shares no near-duplicates with training.

## When not to use it

When the gap is missing knowledge rather than behaviour, use retrieval. When thousands of high-quality examples exist and the budget allows, full fine-tuning may perform better.

## How we evaluate it

Compare base model, prompted base model and adapted model on the same frozen test set, plus a general-ability check to catch forgetting. Training and evaluation data are checked for overlap before results are trusted.

## Sources

- Hu et al. (2021), [LoRA](https://arxiv.org/abs/2106.09685); Dettmers et al. (2023), [QLoRA](https://arxiv.org/abs/2305.14314); Liu et al. (2024), [DoRA](https://arxiv.org/abs/2402.09353)
- [Hugging Face PEFT LoRA guide](https://huggingface.co/docs/peft/en/developer_guides/lora), [TRL v1](https://huggingface.co/blog/trl-v1), [Axolotl](https://docs.axolotl.ai/)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt