Our Times — Technology

Inference Is Now the Line Item That Decides Which AI Products Survive

Maya Iyer — Wed, 12 Aug 2026 00:00:00 GMT

For three years the number that mattered in machine learning was the size of the training run. It was legible, it was expensive, and it made for a clean narrative about scale. That number now tells you almost nothing about whether a product works as a business. The number that does is the cost of answering a single request, and six engineering teams who shared their figures with Our Times describe the same uncomfortable arithmetic.

Training is a capital expense you amortise. Inference is a variable cost you pay on every interaction, forever, and it scales with exactly the thing you are trying to grow.

The gap between demo economics and production economics

A retrieval-heavy assistant that costs a fraction of a cent per query in a demo will often cost twenty to forty times that in production. The teams we spoke to attributed the gap to four things, and they listed them in roughly the same order.

Context growth. Demos use short prompts. Real users paste documents. Attention cost grows faster than linearly in sequence length, so the tail of long requests dominates the average.
Retries and cascades. A single user action fans out into several model calls once you add reranking, tool use, and a validation pass.
Idle capacity. Reserved accelerators are billed whether or not they are saturated. One team reported 31% average utilisation against a peak they had provisioned for.
Guardrails. Safety classification, moderation, and output checking are additional forward passes that nobody puts in the pitch deck.

We shipped a feature with a gross margin that was negative for eight months and nobody noticed, because the cost sat in a shared infrastructure budget rather than against the product line.

That quote came from a platform engineering lead at a company with more than 400 employees, on condition that we not name the product. It was the most common structural failure described to us: serving cost is real, but it is not attributed, so it cannot be managed.

What the teams that fixed it actually did

None of the successful interventions involved a better model. They involved routing, caching, and a willingness to say no to requests.

The most effective single change, reported by four of six teams, was tiered routing: classify the request, send the easy majority to a small model, and reserve the expensive path for cases that need it. The teams that measured it found that between 60% and 80% of production traffic did not need their flagship model at all. Users did not notice the difference, because the requests being downgraded were the ones with unambiguous answers.

Caching came second, and the surprise was how much of it was viable. Semantic caching on normalised queries produced hit rates of 18% to 35% in support and documentation workloads. That is a direct multiplier on cost, and it also cuts latency, which improves the metric the product team actually cares about.

The third lever was the least popular and the most effective: limiting context. Two teams capped retrieved context aggressively and measured no degradation in answer quality, because most of the retrieved material had been noise. One of them had been paying to process an average of 14,000 tokens of context to answer questions that needed 1,200.

Why the accounting matters more than the optimisation

The teams that had cost under control shared one non-technical trait. They had a per-request cost figure that a product manager could see, in the same dashboard as engagement and retention.

Where serving cost lived in a central infrastructure line, it behaved like weather: everyone complained, nobody owned it. Where it was attributed per feature, the optimisation happened without anyone mandating it, because the person who had to justify the feature also had to justify its margin.

This is not a new lesson. It is the same lesson cloud migration taught a decade ago, arriving again with a different bill attached. The difference is magnitude. A wasteful web service costs you a percentage. A wasteful inference path costs you the product.

The part that does not optimise away

There is a floor, and several teams have hit it. If the task genuinely requires a large model over long context with verification, the cost is the cost, and the only remaining moves are pricing and scope. Two of the six teams had raised prices. One had removed a feature entirely after concluding that no plausible efficiency gain would make it viable at the price point customers would accept.

That is a healthy outcome, and it is happening more often. The supply picture matters here too: as we reported in our analysis of the advanced packaging bottleneck, accelerator availability is not improving on the timeline most 2025 capacity plans assumed. Teams that budgeted for cost declines driven by hardware abundance are revising those assumptions.

The teams likeliest to survive the next two years are not the ones with the best benchmark scores. They are the ones who can tell you, to the cent, what a request costs and what it earns. For more on how open-weight alternatives change that calculation, see our reporting on where open models still lose on deployment.

Open-Weight Models Closed the Benchmark Gap. Deployment Is Where They Lose.

Maya Iyer — Mon, 10 Aug 2026 00:00:00 GMT

On the public leaderboards the argument is over. The best open-weight models now sit within a few points of the best proprietary ones on most published evaluations, and on several tasks they are ahead. Teams that have actually deployed both describe a gap that remains substantial, and it has almost nothing to do with capability.

The difference shows up in the parts of a system that no benchmark scores: throughput under concurrency, behaviour at the edges of the input distribution, and the operational cost of being the party responsible when it breaks.

What benchmarks measure and what they miss

A published evaluation measures single-request quality on a curated input distribution, usually with generous latency budgets and no cost ceiling. Production measures something else.

Four teams running both classes of model in production identified the same divergences.

Throughput at target latency. A model that matches on quality can require substantially more accelerator time to hit the same p95 latency under real concurrency. That is a cost difference, not a quality difference, and it does not appear on a leaderboard.
Tail behaviour. Malformed input, adversarial prompts, mixed languages, and very long context are underrepresented in evaluation sets and overrepresented in real traffic.
Structured output reliability. Teams consistently reported more schema violations from open-weight models when asked for strict JSON, which matters enormously when the output feeds a downstream system rather than a human.
Change management. A hosted endpoint changes underneath you, which is a real risk. A self-hosted model does not change unless you change it, which sounds better until you own the upgrade, the regression testing, and the rollback.

We did not switch back because the open model was worse. We switched back because we were spending two engineers on serving infrastructure and the vendor bill was cheaper than those two engineers.

The cases where open weights clearly win

The picture is not one-sided, and the teams that stayed on open weights had specific, legible reasons.

Data residency was the most common. If the requirement is that inputs never leave a jurisdiction or a private network, the decision is made before quality enters the conversation.

The second was high-volume narrow tasks. Classification, extraction, and routing at large scale is exactly where a smaller fine-tuned open model is not just adequate but preferable, because the per-request cost difference compounds and the task distribution is narrow enough that tail behaviour is controllable. This is the same insight driving the tiered-routing pattern we documented in our reporting on serving-cost economics.

The third was pricing leverage. Two teams described maintaining a functional open-weight deployment specifically as a negotiating position, and both reported it worked.

What the honest comparison requires

The comparison most teams run is not the comparison they should run. Quality on a held-out set is the easy part. The complete accounting includes accelerator hours at target latency, engineering time for serving and upgrades, the cost of the evaluation harness you now have to maintain yourself, and the residual risk you have absorbed by becoming the responsible party.

Run that comparison and the answer stops being ideological. It becomes a straightforward function of volume, task breadth, and how much engineering capacity you have to spend. High volume and narrow tasks favour open weights. Low volume and broad tasks favour a hosted endpoint. Most organisations have both, which is why most end up running both.

The framing that will age worst is the one that treats this as a single decision with a single answer. Hardware supply shapes it too: as we reported on the packaging bottleneck, the cost of self-hosting depends on an accelerator market that is not loosening as quickly as 2025 plans assumed.

Advanced Packaging, Not Lithography, Is the Real Chip Bottleneck

Maya Iyer — Thu, 06 Aug 2026 00:00:00 GMT

Ask a policy analyst where accelerator supply is constrained and you will usually hear about lithography. Ask someone who schedules a fab and you will hear about packaging. The second answer has been the correct one for roughly two years, and the distinction matters because the two constraints relax on very different timelines.

Front-end wafer capacity for leading-edge logic has expanded substantially. The step that has not kept pace is the back end: the process of stacking high-bandwidth memory beside a logic die on an interposer and getting acceptable yield out of the result.

Why the back end became the constraint

A modern accelerator is not one chip. It is a logic die, several stacks of high-bandwidth memory, and a substrate that connects them at a pitch fine enough that the memory bandwidth is usable. That assembly step involves thermal cycling, precise alignment, and a yield penalty applied to components that are already expensive.

Three properties make it hard to scale quickly.

The equipment is specialised and the vendor list is short. Bonders and inspection tools for fine-pitch interposers come from a handful of suppliers with their own multi-quarter lead times.
Yield loss compounds. Scrapping an assembly late in the process discards good logic and good memory together, so effective capacity is lower than nameplate capacity by a margin that varies with process maturity.
Memory supply is coupled. High-bandwidth memory qualification is slow, and the number of stacks per accelerator has been rising, so each unit consumes more of a separately constrained input.

Adding a lithography tool is a capital decision. Adding qualified packaging capacity is a capital decision plus eighteen months of process learning that you cannot buy.

What the numbers look like from the outside

Public disclosure here is thin, which is part of the problem. Foundries report capacity in wafer-equivalent terms that obscure the back end, and packaging subcontractors report revenue rather than qualified units.

The signals that are visible point in one direction. Lead times quoted to large buyers have stayed extended even through periods when wafer starts increased. Substrate suppliers have been running expansion programmes continuously since 2024 without lead times normalising. And the pricing structure has shifted: buyers now commonly contract for packaged units rather than wafers, which is what happens when the scarce step moves downstream.

The forecasting error this creates

Capacity plans written in 2025 generally assumed that accelerator availability would improve as wafer capacity came online, and that per-unit costs would fall accordingly. Teams built serving-cost forecasts on that assumption. As we found reporting on the serving-cost squeeze, several of those forecasts are now being revised upward, because the hardware abundance they priced in has not arrived.

The correction is not dramatic. It is a matter of quarters, not years. But it lands on organisations that committed to product margins based on the earlier curve, and the ones with the least slack are the ones who reserved capacity at fixed prices without a corresponding pricing mechanism on the revenue side.

What would actually change the picture

Three developments would loosen the constraint, in descending order of near-term plausibility.

The first is process maturity at existing packaging lines, which raises effective capacity without new equipment. This is happening steadily and is the main reason supply has improved at all.

The second is architectural: designs that need fewer memory stacks per unit of useful throughput. There is real work here, and it does more for effective supply than a new facility would, because it reduces demand on the coupled input.

The third is new qualified capacity at scale, which is under construction and will matter in 2027 and beyond. Anyone promising relief sooner than that from new facilities is describing an announcement, not a shipment.

For the demand side of this equation, see our reporting on where open-weight models still lose, which shapes how much of the market needs frontier-class hardware at all.