Low-Rank Adaptation

LoRA, short for low-rank adaptation, is a fine-tuning method that adapts a large model by training a small set of added parameters while leaving the original weights frozen. Instead of updating a full weight matrix, LoRA learns two much smaller matrices whose product approximates the change, exploiting the observation that task-specific adjustments tend to be low rank. Only those small matrices are trained and stored, which cuts memory use and training time sharply and produces adapter files measured in megabytes rather than gigabytes. That makes fine-tuning a mid-sized open model feasible on a single consumer GPU. Because the base model is untouched, many adapters can sit side by side and be swapped or merged at load time, which is how image tools such as Stable Diffusion distribute community styles and how serving stacks host many customer-specific variants on shared hardware. QLoRA extends the idea by quantizing the frozen base model to four bits, lowering the memory floor further. LoRA belongs to the broader family of parameter-efficient fine-tuning methods. It is not a substitute for retrieval: fine-tuning shapes style, format, and task behavior, while facts that change over time are better supplied through RAG.