Problem
Day 2: Turn messy tickets into SFT demonstrations
Case: Alder's returns-triage assistant
All policies, data, and numbers below are fictional. Allow 10–15 minutes.
Alder wants a model to turn a customer message and a supplied policy excerpt into a structured recommendation. A person still approves the action. A good prompt produces valid structure, but reviewers see inconsistent decisions and guessed missing facts. You are planning an SFT pilot.
The supplied policy R7 says:
- Proof of purchase is required. If its presence is unknown, ask for it before deciding.
- Unopened items can be returned within 30 days.
- Faulty items qualify for warranty review within 365 days.
- If no rule applies or the policy is unavailable, send the case to a person.
The output must have exactly these fields: decision, policy_id, missing_fields, and reason. Allowed decisions are return, warranty, needs_info, and manual_review. Use an array for missing fields and a short explanation for the reason. Use purchase_proof for missing proof.
You have 2,400 historical ticket threads with repeat customers and copied messages. Some answers follow old policy R6. Another model generated 100 paraphrases from 20 of these tickets. A later month's untouched tickets are available for evaluation.
Working vocabulary: An SFT demonstration pairs the model's available input with a reviewed target response. Training data changes model weights; validation data helps choose a setup; a final test estimates performance after choices are fixed. Leakage occurs when evaluation information improperly influences training or model selection.
Your data-design memo — at most 200 words:
- Write the target object for each case: A: faulty item, 200 days old, proof present; B: unopened item, 12 days old, proof unknown. Both inputs include R7.
- Give three preparation rules for turning the historical threads into trustworthy demonstrations.
- Describe the training/validation/test split. Where do the 100 paraphrases belong, and what must be excluded from the model's input?