The Cost of Machine Learning
By Edward Grundy, Bay Information Systems. Published , updated . In Product & Strategy.
Most ML projects that fail do so on budget. The money runs out before the model has good enough data to work with, or nobody accounted for the cost of labelling, or the infrastructure bill arrives three months in. In each case, nobody broke down what each component would cost before the project started.
We can make that breakdown with one expression, F(X) = Y, and then follow the costs that sit around it: what the model costs to run, what it costs to keep reproducible and monitored, and what it costs to improve.
The three cost components
Machine learning reduces to a simple expression: F(X) = Y. A model F takes input data X and produces output Y. Every cost in an ML project maps to one of these three components, and every role on the team owns one of them.
X is the data. Acquiring it, cleaning it, storing it, and maintaining its quality over time. Data costs are the most underestimated in early project estimates, partly because raw data often already exists inside the organisation and looks free until someone has to clean it. Two roles make it usable: the domain expert, who understands what the data means and whether it is trustworthy, and the data scientist, who understands what methods it supports and what it cannot do.
F is the model. Building it, training it, deploying it, and keeping it running. This is where most budget conversations start, usually because it is the most visible part of the work. The ML engineer implements and deploys the model to infrastructure. The developer integrates it into the application. Compute costs live here too (training runs, inference endpoints, monitoring), and the sections after the worked example take them in turn.
Y is the labels. The desired outputs that tell the model what correct looks like. Labelling is frequently treated as a minor line item and frequently turns out to be the largest single cost in a production ML project. The labeller does the tagging. The domain expert defines what correct means in the first place; without that definition, the labeller has nothing to work from and the model has nothing to learn.
A project that is missing any of these roles is missing a component of the system. The domain expert who understands the data and the output owns two thirds of the equation. Without their time budgeted from the start, a project can produce a model nobody is able to evaluate.
A worked example: CakeyBakey
To make the cost structure concrete, consider a simple pipeline. CakeyBakey is a hypothetical service that takes a tweet describing a cake, generates an image of it, rates it as delicious or not, and posts the result back. No human involvement. Three model calls per event: language processing, image generation, classification.
Each event generates £0.50 in revenue. The first build decision is whether to use hosted model APIs or run self-hosted models on rented compute.
Hosted APIs (OpenAI equivalents for language and image generation) cost roughly £0.015 per event. Profit per event: £0.485.
Self-hosted models (open-weight equivalents on AWS GPU instances) cost roughly £0.12 per event. Profit per event: £0.38.
The hosted option is nearly eight times cheaper per event at this scale. Barring regulatory constraints or data sensitivity requirements, it is the obvious starting point for a new system. The self-hosted option becomes competitive at volume, or where data cannot leave the organisation. (The private inference articles cover that case.)
Prices move, and the method outlasts them. Each component of F(X) = Y has an identifiable cost per event, and once you have that, you can model the project.
What the model costs to run
CakeyBakey’s result depends on volume, and the reason lies in how each option is billed. Inference, the cost of running a trained model on each request, is sold in three shapes.
A hosted API charges per token or per image. The cost tracks use exactly: nothing at zero traffic, and a predictable amount per event at any volume. The provider carries the idle capacity and prices it into the rate. (Per-token prices have fallen steadily; falling token prices tracks the trend, and AWS Bedrock costs works through one provider’s bill.)
A rented GPU charges per hour, busy or idle. The cost per request is the hourly rate divided by the requests served in that hour, so utilisation sets the price: a GPU busy one hour in five costs five times as much per request as one kept busy throughout. CakeyBakey’s self-hosted figure is high because an hourly bill is spread over few events; at higher volume the same instance serves more events for the same rate.
Owned hardware moves the cost up front. Once a machine is bought, the marginal cost of a request is close to the electricity it draws: a measured run of 14 open-weight models through 1,204 requests on one RTX 3060 drew 226 watt-hours, or 5.6p. The remaining costs are the purchase, the engineer’s time to run the machine, and its capacity (a model has to fit in the card’s memory; the VRAM calculator does that arithmetic, and the GPU market covers what to buy).
Utilisation decides between the three. Low or unpredictable traffic favours per-token billing, steady high traffic favours hourly or owned capacity, and data that cannot leave the organisation removes the first option whatever the traffic.
The costs around the model
Training runs and inference endpoints are the visible compute. Around them sit costs that rarely appear in a first estimate, each driven by a different quantity.
Deployment is mostly developer time. Serving a model behind an authenticated, logged API in staging and production is a small software project, and it costs about the same whatever the model.
Monitoring costs scale with how fast the world the model describes changes. A model watched for drift needs its inputs and outputs logged and compared against a reference period. The logging is cheap to run; most of the cost is the person who reads the comparison and decides what to do.
Reproducibility and lineage cost storage and discipline. Rebuilding last month’s model needs last month’s data, code, labels and settings, so each version of each is kept and linked to the others. Storage is the smaller part; the larger part is keeping the links, a habit someone has to own.
Experiment tracking costs little to set up. Recording each run’s data version, parameters, score and compute cost (GPU-hours times the hourly rate) is what lets a team say what an improvement cost, which the next section depends on.
Evaluation has become a running cost in its own right. Systems built on language models are often scored by another model acting as a judge, and every evaluation run is a batch of paid calls: a test set of 500 examples scored on four criteria is 2,000 judge calls per release.
The cost of getting better
A model’s performance degrades as the world it was trained on recedes. New patterns emerge in the data. Errors accumulate in the output. At some point, retraining becomes necessary.
Retraining costs reduce to two questions: how much will new labels cost, and how much improvement will they buy?
The label cost formula is straightforward:
label_cost = (n_labels / labels_per_day) x labeller_daily_cost
The improvement can be estimated from a learning curve. Retrain the current model on 50%, 75% and 100% of the existing labels, score each version on the same held-out test set, and extend the trend. Gains usually shrink as labels are added (each doubling of the data buys less than the one before), so the curve also shows when more labels stop paying.
Returning to CakeyBakey: suppose the classifier sits at 70% accuracy on 500 labels, and its learning curve suggests 100 more would lift it to about 82%. At £500 per labeller day and 20 labels per day, those 100 labels cost £2,500. The improvement reduces the break-even point from around 7,800 events to around 6,700. At 500 events per day, that is two days recovered.
Whether that investment makes sense depends on the business. It is a calculable decision, and the same framework applies at any scale: what does a label cost, what does the improvement buy, and can the business afford the delay while the labelling happens.
What “good enough” means
Not every ML system needs to be maximally accurate. A classification model that is right 85% of the time may be entirely adequate if the fallback for the remaining 15% is a lightweight human review. A model that is right 95% of the time may be inadequate if the 5% failure rate occurs on high-value or high-risk cases.
Defining “good enough” before the project starts is the most valuable thing a business stakeholder can do. It sets the label budget, determines the evaluation criteria, and gives the technical team a fixed, specific target. Without it, projects tend to optimise indefinitely toward a standard nobody defined.
F(X) = Y is a framework for that conversation as much as a cost model. X tells you what data investment is required. Y tells you what correct looks like and what it costs to specify. F tells you what engineering work sits in between, and what it costs to run once it exists. The project is estimable once all three are defined.
(If you are assessing whether your organisation has the data foundation to start an ML project, the data maturity article covers the infrastructure side of that question, and the AI readiness assessment covers the rest of the decision.)