What decides the cost and speed of inference

Why is the token price the wrong number to compare?

Token pricing across major inference providers converged for comparable models some time ago. That makes it the line every comparison starts with and the one that tells you least, because the rest of the bill is not on the pricing page.

Forrester's write-up of the Linux Foundation analyst summit gave the rest of the bill a name: total cost of AI. Its claim is that tokens are only 10% to 25% of total AI spend once infrastructure, energy, capital, software and licensing, and the labour to produce and consume AI are counted. The State of Tokenomics 2026 study behind it covers 472 respondents in 11 industries: 96% use frontier models, 87% cloud providers and 64% AI embedded in existing products, and half work across at least four of six channels.

Context is the token multiplier a team chooses for itself. IBM Research's study of agent memory on AppWorld is the cleanest illustration we have reported. On gpt-oss-120b, a curated set of retrieved guidelines added 5% to the tokens per run, from 110K to 116K, while the full guideline set added 51%, to 166K. On DeepSeek-V3.2 the full set pushed 148K to 263K, a 78% increase. The same memory configuration bought 16.1 points of task completion on one model and nothing at all on GLM-5.

What actually decides latency?

For interactive products the deciding number is time to first token, and it varies by more than a factor of three for the same model on the same prompt. That difference is the purchasing decision for a chat product and invisible on any price list. In trading the same logic is sharper: when desks run similar models, the edge goes to whoever answers first, and whoever can slow your inference or poison the retrieval feeding it can move your decisions without touching the model.

Underneath latency sits placement. SemiAnalysis's analysis of mixture-of-experts serving makes the point that MoE changed which tensors are active per token and therefore which parts of a cluster have to sit physically close together. Serving runs inside a cluster coordinated by an orchestration layer such as NVIDIA Dynamo or Mooncake, alongside inference servers like vLLM or SGLang, and the unit of work is the turn. The server keeps a KV cache for the session, and an agent on a long task can generate thousands of turns an hour and pause for hours or days before resuming. That load profile puts weight on cache retention and scheduling rather than on raw arithmetic.

What do the new inference chips change, and when?

SemiAnalysis's numbers on OpenAI's Jalapeno, an inference chip designed with Broadcom and built by TSMC on N3P, are 13.4 PFLOPs of MXFP4 and 15.4TB/s of HBM4 per package at 700W per compute die. On single-token prediction it is reported above 700 tokens per second per user on DeepSeek R1 at low concurrency, and performance per dollar comes out comparable to Nvidia's Vera Rubin. Production ramps through 2027, with volume concentrated at the end of the year.

Cerebras took the other route with the CS-4: the same wafer, rebuilt power and cooling, roughly double the clock. The claimed result is about 4,000 tokens per second per user on frontier models against 2,000 for the CS-3. The cost is power: a CS-3 wafer draws about 23kW, while a CS-4 rack holding three wafers is quoted at 125 to 135kW. Neither pricing nor general availability was stated.

A chip that wins on throughput per megawatt but arrives in quantity at the end of 2027 does not change a 2026 or 2027 buildout. For capacity planning the delivery date and the rack power line matter more than the peak figure.

Is decentralised GPU capacity cheaper for inference?

For interruptible batch work — rendering, fine-tuning, evaluation sweeps — the discount against hyperscaler on-demand pricing is substantial and holds up. For latency-sensitive inference it largely disappears once cold starts and routing overhead are counted. The networks whose utilisation rose were the ones that solved verification and positioned themselves as batch markets. MoE placement adds a structural reason: useful serving depends on what sits close to what, which a geographically scattered fleet cannot advertise.

How do you make a model cheaper to serve?

Distillation moves cost from inference to training: a mid-size model trained largely on a larger system's output reaches accuracy that once needed far more parameters. That suits anyone serving high volume and suits nobody serving a long tail of low-traffic tasks. Quantisation attacks the same bill from the other side. Liquid AI's quantisation-aware distillation checkpoints in Q4_0 recover 96.5% to 97.4% of full-precision accuracy and match Q5_K_M or Q4_K_M quantisations while decoding 3% to 33% faster.

Who controls the inference path?

Hosted inference puts a policy you do not own in the middle of your workload. When Hugging Face investigated the intrusion of July 2026, the hosted commercial models it tried first refused to reason about the attack code, so it ran an open-weight model, GLM-5.2, on its own hardware. The lesson is not that guardrails are wrong; it is that a team depending solely on hosted inference should know in advance which workloads might be refused.

Verifiable inference is the opposite trade: proof that a specific model produced a specific output, so a buyer can pay for the result automatically. Proving cost scales badly with model size, which keeps it economic for small specialised models and not for frontier ones.