Problem
Day 3: Adapt a small model within a compute budget
Case: FieldNote's technician assistant
All organizations, budgets, and dataset sizes are fictional. Allow 10–15 minutes.
FieldNote's assistant receives an equipment report and an approved manual excerpt. It must suggest one permitted next check, cite the section, and ask for a missing model number. A prompt-only small model still skips that last behavior.
The team has a compatible, licensed 3-billion-parameter model, 1,200 reviewed demonstrations, and separate validation and test sets. It can use one graphics processing unit (GPU) with 16 GB of memory for a short pilot. A colleague proposes: “Use QLoRA instead of SFT. The model will be 4-bit, so the whole training run needs only 1.5 GB.”
Working vocabulary:
- SFT is the learning objective: make reviewed target responses more likely.
- Full fine-tuning updates the model's ordinary weights.
- LoRA, low-rank adaptation, freezes the original weights and trains small added parameter matrices called adapters.
- QLoRA trains LoRA adapters while the frozen base weights use a memory-saving 4-bit representation. Four bits describe storage precision, not a four-choice answer space.
- An epoch is one pass through the training examples; a checkpoint is a saved model state.
Your experiment memo — at most 180 words:
- Correct both parts of your colleague's claim. Calculate raw base-weight storage at 16 bits and 4 bits, using 8 bits per byte and decimal GB. Name two other things that occupy training memory.
- Choose an initial adaptation approach and describe what gets updated and what stays fixed.
- Propose a fair comparison against the existing model, including one small feasibility check, one behavior metric, and one regression check. Explain when you would stop or reject the adaptation.