Llms Can Be Fine-tuned Using Which Technique

Large Language Models (LLMs) arrive pre-trained on vast corpora of text, granting them a broad understanding of language, reasoning, and world knowledge. Still, their general-purpose nature often falls short when applied to specific domains, proprietary data, or specialized tasks like legal contract analysis, medical coding, or brand-specific customer support. To bridge this gap, practitioners employ fine-tuning—the process of further training a pre-trained model on a narrower, curated dataset to adapt its weights for a specific objective Simple, but easy to overlook. But it adds up..

While the concept is straightforward, the techniques for fine-tuning have evolved rapidly. So naturally, the answer to "which technique" is no longer singular; it is a spectrum ranging from updating every parameter to modifying almost none. Choosing the right approach depends entirely on computational budget, dataset size, catastrophic forgetting risks, and deployment constraints.

Full Fine-Tuning: The Gold Standard for Performance

Historically, full fine-tuning (often called supervised fine-tuning or SFT) was the default method. This technique unfreezes all model parameters—often billions of them—and updates them via backpropagation on the target dataset And that's really what it comes down to..

How it works: The model continues training from the pre-trained checkpoint using a task-specific dataset (e.g., instruction-response pairs). The optimizer (typically AdamW) adjusts every weight matrix in the transformer layers, embedding layers, and output heads.

Pros:

• Maximum Adaptability: The model can fundamentally reshape its internal representations to perfectly fit the new data distribution.
• Highest Ceiling: Given sufficient data and compute, this usually yields the absolute best performance on the target task.

Cons:

• Prohibitive Hardware Requirements: Fine-tuning a 7B parameter model in FP16/bfloat16 requires roughly 56GB+ of VRAM just for model states, gradients, and optimizer states (often 2-4x model size). This demands multi-GPU clusters (A100s or H100s).
• Catastrophic Forgetting: The model risks overwriting the vast general knowledge acquired during pre-training, degrading its ability to reason or handle out-of-distribution prompts.
• Storage Overhead: You must store a separate copy of the entire model weights for every distinct task or domain.

Because of these constraints, full fine-tuning is now largely reserved for well-funded research labs or enterprises training foundational models from strong base checkpoints (like Llama 3 base or Mistral base) rather than adapting existing instruct models Practical, not theoretical..

Parameter-Efficient Fine-Tuning (PEFT): The Modern Standard

The explosion of model sizes (7B to 400B+ parameters) made full fine-tuning inaccessible to most. This catalyzed the development of Parameter-Efficient Fine-Tuning (PEFT). PEFT methods freeze the vast majority of pre-trained weights and inject a small number of trainable parameters (often < 1% of total params). This drastically reduces memory footprint, prevents catastrophic forgetting, and enables portable adapters.

Easier said than done, but still worth knowing.

Low-Rank Adaptation (LoRA): The Industry Workhorse

LoRA (Low-Rank Adaptation) is currently the most dominant PEFT technique. It operates on a powerful mathematical hypothesis: the change in weights during fine-tuning (ΔW) has a low "intrinsic rank."

Instead of updating the full weight matrix $W \in \mathbb{R}^{d \times k}$, LoRA freezes $W$ and learns two smaller matrices, $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times k}$, where rank $r \ll \min(d, k)$. The forward pass becomes: $h = Wx + \frac{\alpha}{r}BAx$

No fluff here — just what actually works.

Why it wins:

• Zero Latency at Inference: Because $\Delta W = BA$, you can mathematically merge the trained $A$ and $B$ matrices back into the base $W$ ($W_{new} = W + \Delta W$) after training. The model runs at exact base model speed with zero added latency.
• Extreme Efficiency: For a 7B model, LoRA often trains only ~0.1% to 1% of parameters. VRAM drops from >50GB to <16GB (often fitting on a single consumer GPU like RTX 3090/4090).
• Modularity: You can swap LoRA adapters (e.g., "Spanish Adapter," "SQL Adapter," "Tone Adapter") on the fly without reloading the base model.

Key Hyperparameters:

• Rank ($r$): Typically 8, 16, 32, 64, or 128. Higher rank = more capacity but more VRAM.
• Alpha ($\alpha$): Scaling factor. Often set to $2r$ or $r$.
• Target Modules: Usually applied to Query ($W_q$) and Value ($W_v$) projection matrices in attention layers. Applying to all linear layers (including MLPs) increases performance but doubles trainable params.

QLoRA: Quantization Meets LoRA

QLoRA (Quantized LoRA) pushed accessibility further by combining 4-bit quantization (NF4 - NormalFloat 4) with LoRA. It loads the frozen base model in 4-bit precision (reducing 7B model VRAM from ~14GB to ~4GB) but performs the LoRA gradient computation in bfloat16 (via a "dequantize -> compute -> quantize" flow for the adapter weights only) That's the part that actually makes a difference..

QLoRA Innovations:

1. 4-bit NormalFloat (NF4): An information-theoretically optimal data type for normally distributed weights.
2. Double Quantization: Quantizing the quantization constants (saving ~0.37 bits/param).
3. Paged Optimizers: Offloading optimizer states to CPU RAM via NVIDIA Unified Memory to prevent OOM (Out of Memory) errors during gradient spikes.

Impact: QLoRA democratized fine-tuning. A 33B parameter model can be fine-tuned on a single 24GB GPU (RTX 3090/4090 or A10G), and a 7B model on a free Google Colab T4 (16GB). It matches 16-bit full fine-tuning performance on benchmarks like MMLU and MT-Bench.

Other Notable PEFT Variants

While LoRA/QLoRA dominate, other techniques serve niche needs:

• AdaLoRA (Adaptive LoRA): Dynamically allocates rank budget across layers during training using Singular Value Decomposition (SVD), pruning unimportant ranks. Better parameter efficiency for fixed budgets.
• LoRA+ / LoRA-GA: Improved learning rate scheduling and gradient scaling strategies specifically for the $A$ and $B$ matrices to stabilize convergence.
• Prefix Tuning / P-Tuning / Prompt Tuning: Prepend trainable continuous vectors (virtual tokens) to the input or every transformer layer. Only these "soft prompts" are trained.
  • Pros: Extremely few parameters (0.01%).
  • Cons: Reduces effective context window; generally underperforms LoRA on complex reasoning tasks; adds inference latency (cannot be merged).
• IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations): Rescales inner activations with learned vectors. Very lightweight, merges easily, but often slightly lower ceiling than LoRA.
• VeRA (Vector-based Random Matrix Adaptation): Uses a single shared frozen low-rank matrix pair across all layers, learning only scaling vectors. Drastically reduces trainable params further (10x less than LoRA) with competitive results.

Alignment Techniques: Beyond Supervised Fine-Tuning

Fine-tuning isn't just about teaching new facts (SFT); it's critically about alignment—making the model helpful, harmless, and honest. This requires distinct techniques applied after SFT.

Reinforcement Learning from

Human Feedback (RLHF) has become the industry standard for aligning large language models with human preferences. Unlike SFT, which teaches the model to imitate expert demonstrations, RLHF optimizes the model to generate outputs that humans actually prefer.

The RLHF pipeline typically involves three stages:

1. Supervised Fine-Tuning (SFT): The base model is first fine-tuned on high-quality demonstration data to learn instruction-following behavior.
2. Reward Model Training: A separate reward model—often the same architecture as the LLM but with a scalar regression head—is trained on pairwise preference data. Human annotators rank model outputs, and the reward model learns to assign higher scores to preferred responses using a Bradley-Terry loss.
3. Reinforcement Learning Optimization: The SFT model is further optimized using an RL algorithm—most commonly Proximal Policy Optimization (PPO)—to maximize the reward model’s scores while staying close to the reference model via a KL-divergence penalty.

While effective, RLHF is notoriously complex and resource-intensive. It requires storing multiple models (policy, reference, reward, and critic), careful hyperparameter tuning, and is susceptible to reward hacking, where the model exploits loopholes in the reward model rather than learning genuine alignment.

To address these challenges, Direct Preference Optimization (DPO)