Skip to content
7 min

The hidden cost of fine-tuning

A team fine-tunes a 7B-parameter model on 50,000 customer support tickets. The training run costs $200. The team is delighted. They ship. Six months later they are running a fine-tuning team of two engineers, a $3,000-a-month GPU bill for serving, an annual retrain cycle, a separate eval pipeline, a custom monitoring dashboard, and a growing list of requests for 'small adjustments' that each require a new fine-tune. Total operational cost: $40,000 a month. Training was the cheapest line item.

This is the standard pattern of fine-tuning projects. The training run gets quoted in proposals and slide decks. The downstream cost is discovered in production, after the architectural commitment is hard to reverse.

Fine-tuning is a model-shaped solution that creates infrastructure-shaped obligations. The model is the small part. The infrastructure around the model is the large part. Most of the time, the infrastructure has to be re-built specifically for the fine-tuned model, because the alternative (using a prompted hosted API) externalises that infrastructure to the API provider.

Five hidden costs

Five hidden costs that compound over the life of a fine-tuned model.

The eval set has to be yours. With a hosted API, the provider's evaluation work is some of the leverage you get for free. Your fine-tuned model has none of that; if you don't build the eval set, nobody has built it. The eval set is the first invisible cost: usually a month of SME time to build, then ongoing maintenance.

The retraining cadence is yours. Hosted models are continuously improved by the provider. Your fine-tuned model is frozen at training time. As the world drifts (new product names, new customer language, new edge cases), the fine-tuned model degrades silently. The retraining cadence is the second invisible cost: usually quarterly retrains, each requiring fresh labels, eval, and deployment.

The serving infrastructure is yours. A hosted API has known latency, known availability, known cost-per-call. Your fine-tuned model needs GPU servers, autoscaling, failover, monitoring, and someone on-call. The serving cost is rarely the GPU bill itself; it's the engineering hours that go into making the GPU bill not surprise you.

The lock-in is yours. The fine-tuned model is tied to a specific base model, a specific training stack, a specific framework version. When the base model is deprecated (it will be), you fine-tune again. When the framework changes (it will), the training pipeline breaks. The lock-in is the fourth invisible cost: usually a partial rebuild every 12-18 months.

The improvement plateau is yours. Fine-tuning improvements have diminishing returns. The first 1,000 examples buy you a lot. The next 10,000 buy you less. The 100,000-example milestone buys you almost nothing additional. Hosted models, in contrast, get better as the provider invests. Two years out, the fine-tuned model is still where you put it. The hosted model has moved.

When fine-tuning is the right choice

When fine-tuning is the right choice. There are real cases. When the data can't leave your perimeter (regulated content, customer-specific data with strict residency requirements). When the latency requirement is sub-50ms and the network round-trip to a hosted API is too long. When the prompt-based approach has measurably failed against the eval set after honest effort. When the use case is narrow and stable enough that the eval set doesn't churn. When the volume is high enough that the per-token economics tip in favour of self-hosting.

When fine-tuning is the wrong choice. Most of the time. Specifically, when prompting a strong hosted model gets within 10% of the fine-tuned model's accuracy on your eval set, and the prompt-based approach's total cost is lower over a 24-month horizon. Run the math, including all five hidden costs. The math usually says: don't fine-tune, prompt better.

Fine-tuning is a serious operational commitment, not a model-quality decision. The teams that adopt it deliberately, with the infrastructure budget and the eval discipline already in place, succeed. The teams that adopt it because the training run looked cheap on a single line of a slide deck, find the rest of the costs the hard way.

// The artefact
# planning/fine_tune_tco.py: the training run is line one of about a dozen
@dataclass
class FineTuneTCO:
    training_run_usd: float            # one-shot, looks tiny
    monthly_serving_usd: float         # GPU + autoscaling + on-call
    monthly_eng_hours: float           # eval, retrains, debugging
    retrains_per_year: int             # full training cost × N
    base_model_lifecycle_months: int   # rebuild when base model deprecates

    def two_year_total(self, hourly_rate: float = 150) -> float:
        eng_cost = self.monthly_eng_hours * hourly_rate * 24
        serve    = self.monthly_serving_usd * 24
        retrain  = self.training_run_usd * self.retrains_per_year * 2
        rebuild  = self.training_run_usd if self.base_model_lifecycle_months <= 24 else 0
        return self.training_run_usd + serve + eng_cost + retrain + rebuild

Add the five line items the slide deck didn't show. The training run is rarely the dominant cost over a real horizon.

← Back to Insights