When Fine-Tuning Actually Makes Sense

Part 9 covered the real, technical mechanism of fine-tuning. This part covers the honest, practical question that actually matters first: should GreenDesk fine-tune a model for its email-formatting problem at all, or is that premature?

The real, correct first step: exhaust prompt engineering first

# a real, genuine attempt at solving this with prompting alone,
# before reaching for fine-tuning at all
SYSTEM_PROMPT = """Write GreenDesk follow-up emails in this EXACT
format, required by legal:

Subject: [under 60 characters]
Greeting: "Hi [Name],"
Body: exactly 3 sentences, no more
Closing: "Best,\\nThe GreenDesk Team"

Example of correct format:
Subject: Following up on your demo
Hi Priya,
Thanks for exploring GreenDesk's automation features...
Best,
The GreenDesk Team"""

This directly applies the AI Fundamentals series' own few-shot and explicit-format techniques — a genuinely well-constructed prompt with real, concrete examples often achieves consistency good enough to make fine-tuning entirely unnecessary. The honest, correct order of operations is: prompt engineering first, RAG if the problem is knowledge-related, fine-tuning only once both have been genuinely tried and demonstrably fall short.

The real, concrete signal that prompting has actually failed

A real, honest test: run 50 genuinely varied, real email requests
  through the best prompt-engineered version, and measure the ACTUAL
  compliance rate against the required real format

Result: 94% compliant → prompting is probably good enough
Result: 61% compliant → a real, genuine case for fine-tuning

This is a real, measurable, honest threshold rather than a vague impression — part 19's evaluation techniques cover exactly how to build and run this kind of real, systematic test, which is the actual, correct basis for this decision, not intuition alone.

A real, honest cost calculation before committing

Real, upfront costs: gathering 100+ real, correctly-formatted
  training examples (genuine, real human time), the actual real
  fine-tuning run's compute cost
Real, ongoing costs: every future FORMAT change requires a new real
  fine-tuning run — genuinely slower to iterate than editing a prompt

For GreenDesk specifically: if legal's real formatting requirement is likely to change again within the next few months, the real, ongoing cost of re-fine-tuning after every change may outweigh fine-tuning's real benefit over a well-maintained, editable prompt.

A real, genuine, good fine-tuning candidate, concretely

GreenDesk's real formatting requirement, confirmed stable for the
  foreseeable future, with a real 61% prompt-only compliance rate
  measured against 50 real test cases, and 150 real, existing,
  correctly-formatted historical emails already available as
  real training data

This is what a real, evidence-based "yes, fine-tune this" decision actually looks like — not a vague sense that fine-tuning sounds more sophisticated, but a specific, measured gap that prompting genuinely couldn't close, with real training data already available to close it.

A real, genuine case where fine-tuning is the wrong call

Bright Leaf Coffee wants its support assistant to "sound more like
  our brand" — vague, not a specific, measurable format failure, and
  the AI Projects series' own style-guide technique (few-shot
  examples in the prompt, per part 15) already addresses this
  directly and far more cheaply

Next: building a real PDF question-answering AI — applying this series' RAG foundation to a genuinely new, real document format.