The Scaling Cliff

TL;DR

• The pilot is not a small version of production; it is a different system with different economics. A pilot runs on clean data, low concurrency, short contexts, free-tier credits and forgiving users. Production runs on messy data, bursty concurrency, long contexts, list-price billing and users who retry. The cost per transaction is not the same number scaled up, it is a different number, and it is almost always higher.

• Three multipliers, not one, drive the bill. Output tokens cost three to six times input tokens on nearly every provider. Retrieval-augmented and agentic patterns multiply token consumption per task by one to two orders of magnitude through context loading, retries, and sub-agent branching. And roughly seventy per cent of total production AI cost sits outside the model invoice altogether, in data engineering, evaluation, governance, integration and the people who keep it running. A budget built from the price-per-million-tokens line alone will be wrong by a large factor, in a predictable direction.

• API prices are falling roughly eighty per cent a year, and this does not save you. Per-token prices have collapsed while per-task consumption has risen faster. The correct planning assumption is that unit prices keep falling and total bills keep rising, because falling prices are what make previously uneconomic agentic patterns viable, and viable patterns get adopted.

• Self-hosting wins less often than its advocates claim, and the honest breakeven is computable. Against a frontier-tier API at roughly $5 per million blended tokens, a single well-utilised H100 crosses over at somewhere around 8 to 12 million tokens per day. Against a budget hosted open-weight endpoint at $0.14 per million, the crossover sits in the billions of tokens per month, a volume most enterprises will never reach. Anyone who tells you self-hosting is categorically cheaper has not done the arithmetic.

• Where you deploy changes the arithmetic more than most planning admits. Import duties on accelerators, industrial electricity tariffs, and national subsidised-compute programmes each move the ownership crossover by a wide margin, and they vary by a factor of two or more between jurisdictions. Several governments now offer sovereign compute at rates well below international on-demand markets, which for a large class of workloads points toward renting sovereign capacity rather than buying hardware. None of these are constants, so none of them should be assumed.

• Cost control is an architecture decision made once, not a procurement negotiation repeated annually. Routing most traffic to small tuned models, distilling task-specific models, caching aggressively, and using deterministic code wherever judgement is not actually required will move cost per transaction by five to thirty times. None of it is possible without an evaluation harness, which is why cost discipline and quality discipline are the same project.

Key Findings

1. The cost question is asked too late to be answerable. In most enterprises the sequence is: pick a use case, build a pilot, demonstrate it, then ask finance to fund production. By that point the architecture has been chosen, the model has been chosen, and the cost curve has been fixed. The economics gate belongs before the build, not after the demo, and it is the single cheapest intervention available, because the projects it kills cost nothing further.

2. Output-token pricing is the most commonly missed line in every budget we have seen. Generation is more compute-intensive than reading, so providers charge three to six times more for output than input. A team that estimates its bill from input volume, or from a blended "price per million tokens" headline, will understate a generation-heavy workload by a large multiple. Summarisation, drafting, report generation and code assistance are all output-heavy.

3. Retrieval and agentic patterns are token multipliers, and they compound. Each conversational turn reloads context. Each tool call adds a round trip. Each retry on a malformed output pays for the failed attempt as well as the successful one. Each sub-agent branch runs its own loop. Industry estimates place the per-task multiplication for agentic workflows in the range of fifty to five hundred times against a single-shot completion. The pilot, which runs three-turn conversations against a small document set, never surfaces this.

4. The invisible seventy per cent is where projects actually die. Inference compute is a minority of total cost of ownership. The majority is data engineering, model maintenance, evaluation, governance, integration and talent. These costs are invisible in a pilot because the pilot borrows a clean data extract someone prepared by hand, has no governance requirement, and is maintained by the person who built it in their spare time. Production has none of those luxuries.

5. Utilisation, not hardware price, determines whether self-hosting works. A GPU costs the same whether it is at ninety per cent load or five per cent. At ten per cent utilisation, every per-token number is ten times worse. The decisive question is not "what does an H100 cost" but "can we keep it busy," and for most single-workload enterprise deployments the honest answer is no, which argues for consolidating multiple workloads onto shared owned capacity, or renting.

6. Hosted open-weight APIs are the forgotten middle option, and they usually win on cost. The public debate is framed as frontier API versus self-hosting. The third option, the same open-weight model served by a specialist provider who bought GPUs at volume, solved batching, and amortises operations across thousands of tenants, is cheaper than both for most volumes. It fails only on the sovereignty requirement, which is precisely why sovereignty, not cost, should be the stated reason for on-premise deployment when on-premise is chosen.

7. The strongest case for owned inference is not cost. It is control. Data residency obligations, sector mandates, customer contract clauses, and the erasure and purpose-limitation provisions of statutes such as the GDPR are architectural requirements that a per-token API cannot satisfy by contract alone. Where those requirements bind, the comparison of running costs is secondary. It should still be done honestly, so the organisation knows what compliance is costing it.

Part A, Why Pilot Economics Systematically Mislead

A.1 The structural asymmetry

Every enterprise AI pilot we have examined shares a set of conditions that do not survive contact with production. They are not oversights. They are what makes a pilot a pilot.

Clean, curated data. The pilot runs against a data extract that a human being selected, cleaned and structured. It contains the documents that answer the questions the demo will ask. Production runs against the actual data estate: duplicated, contradictory, stale, badly scanned, partially structured, and full of documents nobody has read since they were filed.

Low, predictable concurrency. Ten users generate a smooth request pattern. Ten thousand users generate bursts, Monday morning, month-end close, the day after a product launch. Capacity must be provisioned for the peak, not the mean, which means paying for headroom that sits idle most of the time.

Short contexts. Demo questions are answered from two or three retrieved chunks. Real questions require more context, longer histories, and larger retrieved sets, and context length drives cost superlinearly, because attention memory grows faster than the sequence itself.

Forgiving users. The people in the demo room want it to work. Real users retry when the answer is wrong, rephrase when it is unclear, and abandon when it is slow. Every retry is a second billable request for the same task.

Subsidised or invisible pricing. Pilots frequently run on free tiers, trial credits, or a corporate account whose bill nobody scrutinises at that volume.

The result is a cost per transaction measured under conditions that will never recur, then multiplied by a user count as though the relationship were linear. It is not linear, and the error runs in one direction.

A.2 The three-question test

Before any pilot begins, three questions should have written answers. In our experience fewer than one enterprise programme in ten can answer all three.

1. 1At our target user count and usage frequency, what is the projected monthly inference bill, using list prices and a realistic token estimate including retries and context reloading?
2. 2What is the value per transaction, expressed in the same units: per resolved ticket, per document processed, per query answered?
3. 3At what user count does cost per transaction exceed value per transaction?

If question three has no answer because the curves never cross, the use case is economically sound and should proceed. If the curves cross below the target scale, the use case is not viable in its current architecture, and building it will produce a successful pilot followed by a cancelled programme. That outcome is expensive, demoralising, and entirely predictable eighteen months in advance.

A.3 What the failure data shows

The broader failure statistics are consistent with cost being a primary killer rather than a secondary one. S&P Global Market Intelligence found the share of companies abandoning most of their AI initiatives before production rose from seventeen per cent to forty-two per cent year over year, with cost named among the top cited obstacles alongside data privacy and security. Gartner has forecast that more than forty per cent of agentic AI projects will be cancelled by the end of 2027, citing escalating costs and unclear business value first among the reasons.

These are not model-quality failures. A model that works in the demo still works at scale. What stops working is the business case.

Part B, Token Economics: The Primer Nobody Reads Until It Is Too Late

B.1 What a token is, and why the unit matters

A token is a sub-word unit, not a word or a character. As a working approximation, one token is roughly four English characters, or about 0.75 of an English word. For non-English content and for mixed-script documents, token counts per unit of meaning run higher, which is a factor rarely modelled and one we return to in Part E.

Every commercial API bills on two counters: tokens sent in, and tokens generated out. Everything you send counts as input, the system prompt, the conversation history, the retrieved context, and the user's actual question. On retrieval-augmented applications, the retrieved context usually dominates, and the user's question is a rounding error.

B.2 The output multiplier

This is the single most consequential fact in AI budgeting, and it is buried on most pricing pages.

Generating tokens is more computationally expensive than reading them. Accordingly, nearly every provider charges three to six times more for output than input. A model priced at $2.50 per million input tokens and $15 per million output is applying a six-times multiplier.

The practical consequence is that trimming response length saves substantially more than trimming prompts. A team that spends a sprint compressing its system prompt while allowing the model to produce verbose, unstructured answers has optimised the cheap side of the ledger. Structured outputs, forcing the model to return a defined schema rather than prose, are simultaneously a quality control and a cost control.

B.3 Computing a blended rate

Headline comparisons between providers are close to meaningless because they compare input prices while the bill is driven by the input/output mix. The correct method:

Blended cost per million tokens = (input share × input price) + (output share × output price)

Two workloads on the same model can differ in blended cost by a factor of two or more. A document-classification workload might run at a ninety/ten input-to-output split. A report-generation workload might run at sixty/forty. On a model priced at $2 in and $10 out, those blend to $2.80 and $5.20 respectively, the same model, nearly twice the cost per token.

Every organisation should compute its own blended rate before comparing vendors. Anyone comparing headline input prices across providers is comparing the wrong number.

B.4 The two discounts everyone forgets

Batch processing. Every major provider, OpenAI, Anthropic, Google, Alibaba, offers a batch API at approximately a fifty per cent discount on both input and output, in exchange for asynchronous processing typically within twenty-four hours. An enormous share of enterprise AI workload is not interactive: overnight document classification, bulk summarisation, back-catalogue extraction, periodic report generation. Any workload where a human is not waiting should be on the batch tier. In our assessment work, moving eligible workload to batch is routinely the largest single cost reduction available and requires no architectural change beyond a queue.

Prompt caching. Cached input tokens bill at roughly ten per cent of the standard rate across the major providers, a ninety per cent discount. Where a long, stable system prompt or a fixed document corpus is sent repeatedly, caching converts the dominant cost line into a rounding error. This requires structuring prompts so the stable portion comes first and the variable portion last, which is a five-minute change with a very large payoff.

Together these two mechanisms can reduce a bill by well over half before any change of model, and both are routinely unexploited.

B.5 The API market as of August 2026

Prices move constantly, and any specific figure in this report should be re-verified before it enters a budget. The shape of the market is more durable than the numbers, and the shape is a three-tier structure:

Indicative list prices, August 2026. Premium reasoning tiers sit above the frontier band. Re-verify before budgeting; several providers repriced more than once in the preceding twelve months, and at least one major model carried promotional pricing scheduled to expire.

Two observations matter more than the specific numbers.

The spread is enormous. Between the cheapest broadly available endpoint and the most expensive reasoning tier there is a spread exceeding a hundred times. No single-model architecture can be cost-optimal, because no single workload is uniformly difficult.

Prices fell roughly eighty per cent between early 2025 and early 2026. This is the fact most often cited as a reason not to worry about cost, and it is the wrong conclusion, for reasons developed in the next section.

Part C, The Multiplication Problem

C.1 Why falling prices have not produced falling bills

If unit prices fall eighty per cent and bills nonetheless rise, consumption must have risen by more than five times. It has, and the mechanism is not mysterious.

Falling prices make previously uneconomic patterns viable. An agentic workflow that costs fifty times a single completion is unthinkable at 2024 prices and routine at 2026 prices. So it gets built. Then it gets adopted. Then it gets extended with more tool calls, longer context, and more sub-agents, each of which was individually justified by the cheap unit price.

This is Jevons' paradox operating on tokens, and the planning implication is direct: budget on the assumption that unit prices keep falling and total spend keeps rising. An organisation that treats declining prices as a reason to defer cost architecture will find that it deferred the work until the bill was large enough to force it, which is the most expensive possible moment.

C.2 Where the multiplication comes from

Four mechanisms, which compound rather than add.

Context reloading. In a stateless API, each turn of a conversation resends the entire history. A ten-turn conversation does not cost ten times a single turn; it costs the sum of a growing series. Without caching, a long conversation's input cost is roughly quadratic in turn count.

Retrieval payload. A retrieval-augmented query sends retrieved documents as input. Retrieving twenty chunks of a thousand tokens each means twenty thousand input tokens before the user's question. Teams tune retrieval for recall, retrieve more, to be safe, which is a direct, linear cost increase, and one that also degrades answer quality through context pollution.

Retries and validation. Non-deterministic outputs fail schema validation, contain hallucinated fields, or miss required content. Each retry pays again. A ten per cent failure rate with one retry adds ten per cent to cost; a workflow with five sequential model calls each at ten per cent failure has a compounded end-to-end failure rate above forty per cent, and the retry cost rises accordingly.

Sub-agent branching. An orchestrating agent that spawns specialist sub-agents multiplies calls per task. Each sub-agent carries its own system prompt, its own context, and its own retry behaviour.

Estimates of the combined per-task multiplication for agentic workflows commonly fall in the fifty to five hundred times range against a single completion. Whatever the precise figure in any given system, the point stands: a per-task cost measured on a single-shot pilot tells you almost nothing about the cost of the agentic version of the same task.

C.3 The invisible seventy per cent

Analyses of total AI cost of ownership consistently find that inference compute is a minority of total cost, commonly cited at fifteen to twenty per cent, with recent work putting roughly seventy-two per cent of production AI cost outside the model invoice entirely.

The remainder comprises:

• Data engineering. Connectors, pipelines, normalisation, entity resolution, quality monitoring. This is the largest single line and the subject of our previous report.
• Evaluation. Building and maintaining a test harness against real workflows, and re-running it on every model or prompt change.
• Governance and compliance. Access control, logging, audit trails, DPIAs, model documentation, retention policy.
• Integration. Connecting to ERP, CRM, ticketing and identity systems, and maintaining those connections through upgrades.
• Observability. Token metering, latency monitoring, drift detection, cost attribution.
• Talent. The people who do all of the above.

None of these appear in a pilot, because a pilot borrows a data extract, has no governance requirement, is evaluated by looking at it, and is maintained by an enthusiast. The pilot is not merely cheaper than production. It is cheaper in a way that conceals the entire structure of the real cost.

C.4 A worked illustration

Consider a mid-sized enterprise deploying an internal document assistant. Assume ten thousand employees, twenty per cent of whom use it on a given working day, averaging six queries each.

• Daily queries: 10,000 × 0.20 × 6 = 12,000
• Monthly queries (22 working days): 264,000

Now the token estimate. A retrieval-augmented query with a system prompt, eight retrieved chunks and a conversation history might send 12,000 input tokens and generate 600 output tokens.

• Monthly input: 264,000 × 12,000 = 3.17 billion tokens
• Monthly output: 264,000 × 600 = 158 million tokens

At a mid-tier list price of $2 input and $10 output:

• Input: 3,170 × $2 = $6,340
• Output: 158 × $10 = $1,580
• Subtotal: approximately $7,920 per month

Now apply the corrections a pilot omits. Assume a fifteen per cent retry rate, and assume half the queries are conversational continuations that reload context (adding, conservatively, forty per cent to input volume):

• Adjusted: approximately $12,500 per month

Now apply prompt caching to the stable system prompt and the most frequently retrieved documents, say fifty per cent of input volume becomes cacheable at ten per cent of rate:

• Adjusted: approximately $8,600 per month

And route the sixty per cent of queries that are routine lookups to a high-volume-tier model at $0.20/$1.20:

• Adjusted: approximately $4,100 per month

The same workload, the same quality target, at roughly a third of the naive cost, achieved entirely through caching and routing, with no change to the user experience. This is the entire argument for treating cost as an architecture problem. It is also why the un-architected version, which is what most organisations build first, produces the sticker shock that kills programmes.

These figures are illustrative and depend heavily on token estimates specific to a given corpus and prompt design. The method matters more than the numbers; every organisation should run this calculation with its own inputs.

Part D, Self-Hosting: The Honest Breakeven

D.1 The question, correctly framed

The question is never "is self-hosting cheaper than the API." It is: at what volume, at what utilisation, with whose engineering time, does running your own inference beat paying per token?

Self-hosting converts a variable per-token cost into a fixed per-hour cost. Below the crossover, the idle hours you still pay for keep your effective per-token cost above the API rate. Above it, marginal volume is free until you hit the throughput ceiling of the hardware, at which point cost steps up in whole-GPU increments.

D.2 The formula

Cost per million tokens = (GPU cost per hour) ÷ (tokens per second × 3,600) × 1,000,000

Breakeven volume (millions of tokens/day) = (self-hosted daily cost) ÷ (blended API rate per million)

The difficult input is an honest throughput number for your specific workload. Vendor throughput figures are measured under ideal batching with short sequences; real workloads with long contexts and variable concurrency achieve materially less.

D.3 Worked breakeven, mid-2026 market rates

As a reference point, using approximate mid-2026 international market figures:

• An H100 rents for roughly $2 to $3 per GPU-hour on-demand from specialist providers, and costs roughly $25,000 to $30,000 to purchase.
• An eight-GPU H100 node runs approximately $11,500 to $17,500 per month rented continuously, or over $200,000 as a capital purchase before power and facilities.
• A single H100 running a 70-billion-parameter model in FP8 with continuous batching delivers on the order of 400 tokens per second at typical concurrency, roughly 1.44 million tokens per hour, giving a ceiling near 34.5 million tokens per day before a second GPU is required.

At approximately $2.54 per hour, one H100 costs about $61 per day, flat, regardless of volume.

Against a frontier-tier API at a blended $5 per million: breakeven ≈ $61 ÷ $5 = 12.2 million tokens per day. On an output-heavy sixty/forty split blending to $7.50, it falls to about 8.1 million tokens per day. Both sit inside the single-GPU ceiling, making this a clean crossover.

Against a budget hosted open-weight endpoint at $0.14 per million input: one published analysis puts the single-H100 breakeven near 5.7 billion tokens per month, a volume the overwhelming majority of enterprises will never approach.

D.4 The three findings that follow

First, the crossover against frontier APIs is real and reachable. An organisation genuinely processing eight to twelve million tokens per day, which the illustration in C.4 approaches, has a legitimate economic case for owned inference, provided utilisation holds.

Second, the crossover against hosted open-weight APIs is not reachable for most enterprises. Specialist providers serving the same open weights bought GPUs at volume, solved high-utilisation batching, and amortise operations across thousands of tenants. They carry the utilisation risk. This is why an organisation whose only reason for self-hosting is cost should usually not self-host. If you want open weights and you do not have a sovereignty requirement, a hosted open-weight endpoint is very likely your cheapest option.

Third, utilisation dominates everything. At ten per cent utilisation, every self-hosted per-token figure is ten times worse, which pushes the crossover ten times further away. Realistic all-in cost including engineering, redundancy and monitoring commonly runs three to five times the raw GPU rental line. A single-workload deployment on dedicated hardware is the worst case; consolidating several workloads onto shared capacity is what makes ownership work.

D.5 The costs that do not appear in the GPU quote

• Power. An H100 has a 700W thermal design power, drawing perhaps 400–500W at partial utilisation and 700W sustained. Two H100s at an average 500W consume roughly 720 kWh per month.
• Cooling and facilities. High-density AI racks (20–50 kW) require different thermal design from standard IT racks, frequently including liquid cooling.
• Redundancy. N+1 at minimum if the service is business-critical. This is a second set of hardware that is idle by design.
• Engineering. Serving-framework operation, model updates, quantisation, batching configuration, monitoring, incident response.
• Depreciation and obsolescence. Accelerator generations turn over quickly. Three-year amortisation is common; whether the hardware remains competitive for three years is a genuine risk.
• Secondary-market risk. Used GPUs carry no manufacturer warranty; budget for burn-in and early failure.

D.6 A decision table

Part E - Regional Variables That Change the Arithmetic

The analysis above assumes international list prices. Three regional variables move it, and in some jurisdictions one of them is large enough to change the conclusion outright. None of them is a constant, so none of them should be inherited from somebody else's model.

E.1 Import duties raise the cost of owning

Accelerators cross borders, and in many jurisdictions they arrive with a duty load attached. Combined customs duty and import taxation on high-end accelerators runs from effectively zero in some free-trade regimes to roughly 30 per cent in others, which is enough on its own to push the ownership crossover out by several months.

Export controls matter separately from duty. Access to current-generation accelerators is straightforward in some jurisdictions, licensed in others, and unavailable in a few. Both questions belong in a sizing exercise before any capital is committed, because both change the landed cost of the same card by a wide margin.

E.2 Electricity raises the cost of running

Industrial electricity tariffs vary by more than a factor of three across the markets where enterprises actually deploy, and energy represents thirty to forty per cent of total data-centre operating cost. That single input is enough to move a breakeven calculation meaningfully.

The mitigation is significant and under-used. Open-access power purchase agreements, group captive structures and renewable procurement routinely secure materially better rates than default utility tariffs, and the difference compounds over the life of a facility. Many jurisdictions also offer preferential tariffs or duty waivers under data-centre incentive policies. Site selection should be evaluated jointly with power strategy rather than after it.

For an enterprise deploying a small number of accelerators inside an existing facility rather than building a data centre, the practical implication is simpler: check your own marginal industrial tariff before modelling anything, because the range across regions is wide enough to change the answer.

E.3 Subsidised sovereign compute changes the conclusion

This is the largest single distortion available, and most enterprise planning has not absorbed it.

Several governments now operate national compute programmes that make accelerator capacity available at subsidised rates, in some cases at a third or less of international on-demand pricing, on the condition that the workload stays inside the jurisdiction. Where a programme like that is accessible, it compresses the breakeven substantially. A rate at one third of the on-demand market moves the crossover against a frontier-tier API from roughly 12 million tokens a day to something closer to 4 million.

The practical conclusion is that where subsidised sovereign compute is available and the workload qualifies, renting it is likely to beat both buying hardware and calling frontier APIs across a wide band of volumes, while also satisfying data-residency requirements. It deserves to be the first option evaluated rather than the last.

Verify before relying on it. Eligibility criteria, workload restrictions, availability, allocation processes and the continuation of any subsidy are matters of live policy in every jurisdiction that offers one. Anyone building a business case on a subsidised rate should confirm current terms directly with the programme, and should model a sensitivity case at unsubsidised rates.

E.4 The tokenisation penalty for non-English content

A factor almost universally omitted from deployment planning outside the English-speaking world: tokenisers optimised for English produce more tokens per unit of meaning for other languages, and substantially more for non-Latin scripts. Mixed-script enterprise documents, such as a purchase order with English field labels and free text in another language, or a customer complaint written in transliterated form, consume more per unit of content than their English equivalents, sometimes by a large factor.

For any organisation processing significant non-English content, this warrants direct measurement rather than estimation. Take a representative sample of real documents, count tokens under the candidate model's tokeniser, and derive a corpus-specific tokens-per-page figure. In several corpora we would expect the penalty to be large enough to influence model selection, since tokeniser efficiency for a given script varies considerably between model families.

E.5 What this means in combination

Putting the adjustments together produces an ordering of options that depends on where you are, and which frequently differs from the international default:

1. 1Subsidised sovereign compute. Likely cheapest where a programme exists, is accessible, and the workload qualifies. Satisfies residency requirements by construction.
2. 2Hosted open-weight endpoints. The cheapest unsubsidised option for most volumes, but it fails sovereignty requirements wherever data must remain under organisational control in a named jurisdiction.
3. 3Owned on-premise hardware. Justified by binding sovereignty requirements, very high sustained volume, or multi-workload consolidation. Duty load and electricity tariff decide how high that hurdle is for you.
4. 4Frontier APIs. Correct for the genuinely difficult minority of queries. Wrong as a default for everything.

Part F, The Architecture That Controls Cost

Everything in Parts A through E describes the problem. This part describes the response, and the response is architectural. Each mechanism below is a design decision made once, in the platform, rather than a negotiation repeated with each vendor.

F.1 Routing

The single highest-leverage intervention. The observation is that enterprise query traffic is not uniformly difficult: a large majority, commonly estimated around ninety per cent, is routine, and does not require frontier capability. Classification, extraction, lookup, simple summarisation and template completion are all comfortably handled by small tuned models.

A router classifies each incoming request and dispatches it to the cheapest model capable of handling it, escalating only genuine edge cases. Documented cost reductions from routing fall in the five to thirty times range.

The router itself must be cheap and fast, a small classifier, not a frontier model, or it becomes the cost it was meant to eliminate.

The prerequisite is an evaluation harness. Without measurement, you cannot establish which queries the small model handles adequately, so you cannot set the routing threshold, so you either route too aggressively and degrade quality or too conservatively and save nothing. This is why cost work and quality work are the same project.

F.2 Distillation and small tuned models

Where a high-volume task is well-defined, a small model fine-tuned on that specific task will frequently match or exceed a large general model at a fraction of the cost. The canonical result remains instructive: DistilBERT reduced model size by forty per cent while retaining approximately ninety-seven per cent of language understanding and running sixty per cent faster (Sanh et al. 2019, arXiv:1910.01108).

The pattern generalises. For invoice field extraction, ticket classification, entity tagging, or compliance flagging, bounded tasks with abundant training examples in the organisation's own history, a tuned small model is usually the correct answer, and it can run on modest hardware.

There is also a growing body of argument, including a widely-cited NVIDIA-authored position paper, that small language models are the appropriate default for agentic systems specifically, because agentic loops multiply call volume and therefore multiply the penalty for over-provisioning capability.

F.3 Caching

Two layers, both under-exploited.

Prompt caching at the provider, discussed in B.4, delivering roughly a ninety per cent discount on the cached portion. Requires prompts structured stable-first.

Semantic caching in your own layer: if a semantically equivalent question has been answered recently and the underlying data has not changed, return the stored answer. In internal helpdesk and policy-lookup workloads, repeat-question rates are high, and semantic caching can eliminate a substantial fraction of calls entirely. The critical constraint is invalidation, a cache that serves stale answers after a policy change is worse than no cache, which ties directly to the freshness monitoring described in our data foundation report.

F.4 Hybrid deterministic and probabilistic design

The most under-used cost control is not using a model at all.

A large amount of what gets built as AI does not require judgement. Validation rules, threshold checks, lookups, format conversions, routing by explicit criteria, and calculations are all cheaper, faster, more reliable and more auditable as deterministic code. Using a language model for them is expensive and introduces non-determinism into a process that had none.

The design discipline: decompose the workflow, identify the steps that genuinely require judgement, implement everything else as code, and invoke the model only at the judgement points. This reduces cost, reduces latency, reduces failure modes, and improves auditability simultaneously. It is also the step most often skipped, because "add AI to the process" is easier to authorise than "re-engineer the process and add AI where it is needed."

F.5 Output discipline

Because output costs three to six times input, controlling generation length is disproportionately valuable. Structured outputs with defined schemas, explicit length constraints, and prohibition of conversational preamble are each straightforward and each measurably reduce cost. A model that begins every answer with a restatement of the question is billing you for it.

F.6 Observability and FinOps

None of the above can be managed without measurement. From day one, production systems should instrument:

• Token consumption per request, per user, per use case, per model
• Cost per transaction, tracked against the value-per-transaction assumption from the economics gate
• Cache hit rate and routing distribution
• Retry rate and failure classification
• Latency at percentiles, not means

The specific question that observability must be able to answer at all times: at what user count do we begin losing money per transaction? An organisation that cannot answer that question is not managing its AI cost; it is discovering it.

Part G, A Decision Framework

A sequence any enterprise can run before committing capital.

Step 1, Establish value per transaction. What is a resolved ticket, a processed invoice, an answered query actually worth? Time saved × loaded hourly cost is a defensible starting point. If this number cannot be established, stop: the use case cannot be evaluated, and building it is a bet, not a decision.

Step 2, Estimate tokens per transaction honestly. Measure against a real sample of your documents and your prompts, with your tokeniser. Include system prompt, retrieved context, conversation history, and expected output. Then apply a realistic retry rate.

Step 3, Project volume at target scale. Users × adoption rate × frequency × working days. Be honest about adoption; assumed adoption rates are the most commonly inflated input in these models.

Step 4, Compute the naive bill. Blended rate × projected volume. This is your unoptimised ceiling.

Step 5, Apply the architecture discount. Model the same workload with batch processing on eligible traffic, prompt caching on stable context, and routing to the cheapest capable tier. In our experience this typically lands between a third and a half of the naive figure.

Step 6, Add the invisible seventy per cent. Data engineering, evaluation, governance, integration, observability, people. If inference is fifteen to twenty per cent of total cost of ownership, multiply accordingly. This is the number to take to finance.

Step 7, Compare against value. Plot cost per transaction against value per transaction across the volume range. Identify the crossover, if any.

Step 8. Choose the deployment model. Use the table in D.6, adjusted for the regional factors in Part E. Where a subsidised sovereign compute programme is available to you, evaluate it first.

Step 9, Decide, and record the decision. If the economics do not work, do not build it. Record why, and what would have to change for the answer to be different, a price fall, a volume increase, a value increase. Revisit when one of those changes.

The organisations that succeed with AI are not the ones that build the most. They are the ones that decline to build the things that were never going to work, and concentrate their capacity on the things that will.

Recommendations

Immediate (weeks 0–4). Instrument what you already run. Most organisations with AI in production cannot state their cost per transaction. Add token metering and cost attribution before anything else, because every subsequent decision depends on it. In parallel, audit for the two free wins: move all non-interactive workload to batch pricing, and restructure prompts for caching. These require no architectural change and routinely halve a bill.

Benchmark that changes the plan: if you cannot attribute cost to use case within four weeks, your observability is the bottleneck, not your model choice.

Near-term (months 1–3). Build the evaluation harness before building the router. Establish, against your real workflows, which query classes a small model handles at acceptable quality. Only then implement routing. Simultaneously, run the Part G framework retrospectively on every AI use case currently in flight, and be prepared to cancel those whose curves cross below target scale.

Benchmark: a use case that cannot demonstrate positive unit economics at projected scale should not proceed, regardless of how well the pilot demonstrated.

Medium-term (months 3–9). Decompose your highest-volume workflow and replace with deterministic code every step that does not require judgement. Fine-tune a small model for your highest-volume bounded task. Evaluate subsidised sovereign compute against your current serving arrangement, with a sensitivity case at unsubsidised rates.

Benchmark: cost per transaction should fall by a factor, not a percentage. If routing, caching and distillation together produce less than a two-times improvement, the implementation is wrong.

Ongoing. Re-run the economics as prices move. Falling unit prices do not reduce the need for this discipline; they change where the crossovers sit. Treat the cost architecture as a living system with an owner, not a project with an end date.

What would change these recommendations. If inference costs collapse faster than agentic consumption rises, genuinely possible, and the trend to watch, the routing and distillation investment becomes less urgent, and the priority shifts entirely to data quality and workflow redesign. If a subsidised sovereign compute programme you were counting on is withdrawn or restricted, the sovereign-compute recommendation weakens and the ownership calculation reverts toward the international model, with the local duty load as a permanent penalty. If your organisation acquires a binding data-residency obligation it does not currently have, cost ceases to be the deciding variable and the analysis in Part E.5 reorders.

Caveats and Limitations

Prices are a snapshot, not a constant. Every API price in this report reflects publicly listed rates as of approximately August 2026, drawn from aggregator sources whose figures occasionally disagree with one another and with provider documentation. At least one major model carried promotional pricing due to expire. Any figure entering a budget should be verified against the provider's own pricing page on the day.

Subsidised compute rates require verification. Published rates for national compute programmes derive from government statements and are described as post-subsidy. Eligibility, workload restrictions, allocation mechanisms, availability and the continuation of any subsidy are live policy matters outside our control. The conclusions in E.3 are contingent on such a rate being accessible for a given workload. Model an unsubsidised sensitivity case.

Throughput figures are workload-dependent. The ~400 tokens/second and ~34.5 million tokens/day figures for a single H100 serving a 70B model reflect specific assumptions about quantisation, serving framework, batching and sequence length. Your workload will differ, possibly by a large margin. Measure rather than assume.

Self-hosting breakeven estimates vary widely across published analyses, because they make different assumptions about utilisation, engineering cost, and which API they compare against. We have presented ranges and named the comparison basis in each case. Treat all crossover points as directional.

Cost-of-ownership ratios are estimates. The "inference is fifteen to twenty per cent of TCO" and "seventy-two per cent of cost sits outside the model invoice" figures come from industry analyses with varying methodologies. They agree on direction and rough magnitude, which is the load-bearing claim; the precise ratio for any given organisation will differ.

The worked example in C.4 is illustrative. Its token estimates are plausible but not measured against a real corpus. It demonstrates a method, not a result.

Vendor-published analyses carry commercial incentives. Several sources for the self-hosting economics are GPU providers or inference vendors with an interest in the conclusion. We have preferred figures where multiple independent sources converge, and have deliberately included the finding least favourable to our own service offering, that self-hosting frequently loses to hosted open-weight endpoints on cost alone, because it is well-supported.

Forward-looking statements are predictions. Gartner's 2027 agentic-cancellation forecast, projections of data-centre power consumption, and expectations about future price movements are forecasts, not measurements, and are identified as such where they appear.

This report is published as research and is not a commercial proposal. Its figures should be re-verified before use in financial planning. We are happy to run the Part G framework against your specific workload; that engagement is a fixed-fee, two-week diagnostic with a written output, and it frequently concludes that a use case should not be built.