Why Fine-Tune When Prompting Works?

Prompting is great for one-off instructions. But it has limits. Every request costs context tokens. The model can’t learn from corrections. And complex style rules in system prompts are unreliable — the model forgets them on long conversations.

Fine-tuning bakes knowledge directly into the model weights. Your fine-tuned model automatically writes in your style, follows your conventions, and knows your domain — without spending a single context token on instructions.

The question isn’t “should you fine-tune?” It’s “is your use case specific enough to justify 2-4 hours of training?” If you keep copy-pasting the same system prompt, the answer is yes.

Full Fine-Tuning vs. LoRA vs. QLoRA

Method	Trainable Params	VRAM (7B model)	Quality	Speed
Full fine-tuning	100% (7B)	~56GB	Best	Slowest
LoRA	~0.1-1% (~7M)	16-24GB	95-99% of full	Fast
QLoRA	~0.1-1% (~7M)	8-12GB	93-98% of full	Fastest

LoRA (Low-Rank Adaptation) freezes the entire base model and trains small “adapter” matrices that modify the model’s behavior. Instead of updating 7 billion parameters, you update ~7 million. The quality difference is almost undetectable for domain-specific tasks.

QLoRA goes further: it quantizes the frozen base model to 4-bit precision and trains LoRA adapters in higher precision (BF16). This cuts VRAM by another 40-50%. An RTX 4070 Ti (12GB) can fine-tune a 7B model. That was unthinkable in 2023.

The 2026 Fine-Tuning Stack

# The modern stack
pip install torch torchvision     # PyTorch 2.5+
pip install transformers datasets  # HuggingFace ecosystem
pip install peft trl              # LoRA + training utilities
pip install unsloth                # 2x faster training, 50% less memory
pip install bitsandbytes           # 4-bit quantization for QLoRA

Step 1: Prepare Your Data

Fine-tuning data is a collection of input-output pairs in conversational format. Quality matters more than quantity. 500 carefully crafted examples beat 10,000 sloppy ones.

[
  {
    "conversations": [
      {"role": "system", "content": "You are a senior Python developer at Acme Corp. Follow PEP 8, use type hints, prefer dataclasses over dicts."},
      {"role": "user", "content": "Write a function to validate email addresses"},
      {"role": "assistant", "content": "```python\nfrom dataclasses import dataclass\nimport re\n\n@dataclass\nclass EmailValidation:\n    is_valid: bool\n    error: str | None = None\n\ndef validate_email(email: str) -> EmailValidation:\n    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$'\n    if not email:\n        return EmailValidation(is_valid=False, error='Email cannot be empty')\n    if not re.match(pattern, email):\n        return EmailValidation(is_valid=False, error='Invalid email format')\n    return EmailValidation(is_valid=True)\n```"}
    ]
  }
]

Step 2: Configure and Train with QLoRA

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments

# Load base model with 4-bit quantization
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.1-8b-instruct-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,                # Rank — higher = more capacity, more VRAM
    lora_alpha=32,       # Scaling factor (usually 2x rank)
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
)

# Train
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,          # Your prepared dataset
    args=TrainingArguments(
        output_dir="./output",
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=10,
        num_train_epochs=3,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
    ),
)

trainer.train()

On an RTX 4090, this trains a 7B model on 1,000 examples in about 30-45 minutes. On an RTX 4070 Ti, roughly 60-90 minutes.

Step 3: Merge and Export

After training, merge the LoRA adapters back into the base model and export to GGUF format for use with Ollama:

# Merge LoRA adapters into base model
model.save_pretrained_merged(
    "./merged-model",
    tokenizer,
    save_method="merged_16bit",
)

# Export to GGUF for Ollama / llama.cpp
model.save_pretrained_gguf(
    "./merged-model-gguf",
    tokenizer,
    quantization_method="q4_k_m",  # 4-bit for deployment
)

Step 4: Load in Ollama

# Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./merged-model-gguf/unsloth.Q4_K_M.gguf
TEMPLATE "{{ .System }}\n{{ .Prompt }}"
SYSTEM "You are a senior Python developer at Acme Corp."
PARAMETER temperature 0.7
PARAMETER top_p 0.9
EOF

# Create the Ollama model
ollama create acme-coder -f Modelfile

# Run it
ollama run acme-coder
>>> Write a data validation module for our user service

When NOT to Fine-Tune

Your task changes frequently — prompting is more flexible
You have fewer than 100 examples — the model won’t learn enough to generalize
You need factual knowledge updates — fine-tuning teaches style and patterns, not facts (use RAG for facts)
Your base model already does it well with prompting — don’t fix what isn’t broken

Fine-tuning 101 — Customize a Model for Your Use Case

Related Resources