Problem
Day 1: The support bot with yesterday's facts
Case: ParcelFlow's support assistant
All organizations, observations, and numbers in this case are fictional. Allow 10–15 minutes.
ParcelFlow sells shipping software. Its integration instructions change weekly. The assistant should answer from the current approved documentation, cite the relevant section, and ask for missing details instead of inventing an answer.
On 100 diagnostic tickets used for development, the current model gives 56 fully supported answers and follows the required response format 82 times. A clearer prompt with three good examples raises format compliance to 96 but fully supported answers only to 58. These checks can overlap. Reviewers find obsolete instructions and missing documentation in many remaining factual failures. You also have 8,000 historical support threads, some containing outdated advice and personal details.
Working vocabulary: Post-training changes a pretrained model through further training. Supervised fine-tuning (SFT) teaches it using examples of desired responses. Retrieval-augmented generation (RAG) supplies retrieved documents when the model answers; a basic RAG application can do this without changing model weights. An evaluation set is a collection of cases with a defined scoring rubric.
Your decision memo — at most 180 words:
- Choose the next experiment: more prompting, retrieval, SFT, or a combination. Explain which observed failure each part addresses.
- Describe the input and desired answer for one training example you would collect if a behavior gap remains. Decide what to do with the 8,000 old threads.
- Define two quality checks and a fair comparison on fresh cases. State what result would justify trying SFT after the first experiment.