What AI hosting companies actually sell

An open model gives a team access to the weights. Turning those weights into a dependable service means deciding how requests share expensive hardware, how much previous computation can be reused, and how much capacity to keep available.

The answer starts when several customers arrive at once, bringing different prompt lengths, output lengths, and expectations about how long they should wait. The provider has to turn that traffic into useful work on expensive hardware. It must do this reliably enough that the customer can build a product around it.

I think this is the most useful way to understand the market for hosting open models: providers compete on how well they organize demand, preserve previous computation, and deliver capacity under a particular service requirement. I would assess their token prices alongside that system's performance, rather than treating a price as a direct measurement of production cost.

Two recent developments make the distinction concrete. DeepSeek has released DeepInfer, exposing more of its serving engine. Providers are also offering commitments to inference capacity alongside familiar pay-per-token APIs. Both developments force a closer look at what the customer is buying.

An inference engine leaves a business to operate#

DeepInfer's README describes an offline inference engine supporting DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It explicitly lists pieces still required for a complete service: a KV-cache system, vision/prefill/decode disaggregation, and load balancing across data-parallel replicas. The engine includes important building blocks, but integrating the full service remains work for its operator. DeepInfer

Those missing pieces affect economics. A KV cache stores intermediate attention state so the system can reuse work from an earlier prompt. Load balancing decides which machines receive requests. Disaggregation lets prompt processing and answer generation use separate resource pools.

DeepSeek's deployment Q&A describes a dedicated storage cluster for persistent cache, with storage and compute able to scale independently. It reports different retention policies for agent and chatbot traffic. This is a substantial storage and scheduling system wrapped around the model. DeepSeek's deployment Q&A

Opening more of the engine should make efficient deployments easier to build. My inference is that it also raises the standard for commercial hosts: customers can reasonably ask which operational capabilities and workload-specific improvements justify the service.

Traffic changes the cost of producing a token#

Prompt processing, usually called prefill, and answer generation, called decode, put different pressures on hardware. Long prompts can make prefill compute-intensive. Decode is often constrained by memory movement, with the exact bottleneck depending on model, batch size, and context length.

Serving several sequences together lets an engine amortize work across requests. Continuous batching allows requests to enter and leave that working set as others finish; the Orca research describes the underlying idea of scheduling work at each generation iteration. But adding concurrency also consumes memory and can worsen latency. A high aggregate token rate can coexist with an experience that feels slow to each user. Orca

The 2024 DistServe research frames the objective as goodput: serving requests while meeting specified time-to-first-token and time-per-output-token requirements. That is a more useful purchasing concept than maximum tokens per second. DistServe

Consider a hypothetical replica: one running copy of a model and its allocated compute. Assume it costs $8 per provisioned hour and can generate an aggregate 4,000 output tokens per second across concurrent requests while processing the associated inputs and meeting the required latency. These are invented assumptions for a sensitivity calculation, not a DeepSeek hardware recommendation or benchmark.

The arithmetic is hourly compute cost divided by delivered output tokens. This normalizes the cost of serving the whole workload against output volume; it is not an API output-token rate. Here, utilization means delivered workload relative to the assumed capacity, rather than a GPU activity counter. Different prompt lengths, cache reuse, and latency targets require their own capacity measurements.

The tenfold cost difference comes from the same provisioned machine doing less useful work. Redundancy, storage, networking, and operations would add costs. A provider with sufficient compatible demand can spread those costs across more requests. A small deployment has to find another advantage, such as a specialized workload or a requirement for direct control.

The cache discount changes the comparison#

Current DeepSeek Flash pricing makes the importance of reuse unusually visible. These are published USD rates per million tokens, checked on October 21, 2026:

DeepSeek identifies its Flash endpoint as V4.1-Flash-1017. Its peak schedule is 01:00–04:00 and 06:00–10:00 UTC on weekdays, excluding Chinese public holidays. These list rates do not establish equal performance, cache availability, contractual terms, or negotiated prices. DeepSeek pricing, Together pricing

For a hypothetical off-peak workload with 100 million input tokens and 10 million output tokens, the model-token bill is $21 with no input cache hits. If 90% of input tokens hit cache, it becomes $7.77: $1.50 for new input, $0.27 for reused input, and $6 for output. A fiftyfold cached-input discount produces about a 2.7-fold reduction in this token bill. These figures cover model tokens only; tool, storage, and other service charges are outside the example.

A buyer comparing self-hosting against the uncached API bill could reach a different decision from one using the actual cache-heavy bill. The alternative to operating your own GPUs may already include considerable savings from reuse.

The hit rate must be measured. DeepSeek documents prefix-persistence and matching rules, provides separate usage counters for cache hits and misses, and describes caching as best effort. Repeated text does not establish that the provider has the corresponding reusable state when a request arrives. Context caching

This also makes routing interesting. Moving a request to another provider may change its access to warm cache. My expectation is that a useful routing policy will consider the bill for completing a session, including cache misses and retries, rather than choosing each call solely by its advertised token rate.

Be careful with the margin stories#

The bycloud video that prompted this investigation makes the provider business tangible with a rental-and-throughput calculation. It is a useful way to expose assumptions, but its example is not a measurement of current industry margins. The economics video

The widely repeated DeepSeek figure of 545% needs particular care. In its February 2025 V3/R1 disclosure, DeepSeek assumed GPU rental at $2 per GPU-hour and calculated theoretical revenue as if all traffic were billed at R1 rates. It compared $562,027 in hypothetical daily revenue with $87,072 in assumed GPU rental. The 545% is the excess divided by rental cost, rather than a revenue-denominated profit margin. The company explicitly said actual revenue was substantially lower because of cheaper V3 traffic, free app and web use, and off-peak discounts. Its input throughput also included cache hits. DeepSeek's original serving disclosure

That calculation illustrates a serving system's potential. It does not establish realized company profitability. An API list price, a throughput result, and a rental quote cannot answer that question without knowing how much capacity was occupied, what was billed, and which costs were included.

The market is selling different kinds of commitment#

The choice already extends beyond a shared API and a server you operate yourself.

Together's provisioned-throughput offering reserves normalized token capacity. Its FAQ defines an availability target of 99% monthly success for eligible requests within purchased capacity; excess traffic can overflow to serverless at list price on a best-effort basis. At this check, the FAQ lists MiniMax M3 and GLM 5.2 as supported models, so this commitment should not be assumed to cover the Flash endpoint in the earlier price table. Together provisioned throughput

Baseten offers model APIs and dedicated deployments, with enterprise options for deployment in a customer's VPC or a hybrid arrangement. This separates who manages the serving stack from where the compute runs. Baseten deployment options

I would compare those options by the responsibilities and risks being purchased:

The right choice depends on the workload. Predictable volume can support a commitment. Irregular demand makes idle capacity harder to justify. Data-location or customization requirements can make direct control valuable before it wins a token-cost comparison.

Start with a workload you can replay#

For a buying decision, I would capture representative sessions and their arrival pattern, including long prompts, cache reuse, bursts, tool calls, and output lengths. Then I would replay them against the candidate deployments under the same acceptance criteria.

The record should show completed tasks, failures, retries, first-token latency, output speed under load, and the resulting bill. For dedicated capacity, it should include provisioned hours and spare capacity. Quality checks should cover the actual API behavior, not just the name of the weights: Moonshot's vendor verifier provides one concrete example of testing served model behavior. Kimi Vendor Verifier

I would call self-hosting a cost win when compute, storage, networking, redundancy, and the allocated engineering and operations cost together fall below the API bill for the same accepted workload. Both sides need their observed cache behavior and failure costs. Monthly token volume alone cannot establish that break-even point.

I would review the total cost per accepted task alongside latency and reliability, keeping those dimensions visible. A cheap failed request still leaves work to finish. A fast endpoint that regularly exhausts its quota may require a more expensive fallback.

There is good evidence that comparable benchmark capability is becoming cheaper. Across five primary benchmarks, Epoch AI's September analysis estimates that the cost of achieving a given performance level has fallen roughly 47% per quarter since 2023. The authors explicitly warn that benchmark progress is not equivalent to useful work and that customers do not continuously switch to the cheapest qualifying model. Epoch AI

Those falling prices are a reason to revisit deployment choices. Before signing a capacity commitment or assembling a private fleet, I would want a replay of the work it must carry, at the service level it must maintain, with the costs that someone will actually pay.

Research and prices checked on October 21, 2026. The utilization table and token-bill figure are illustrative calculations; the historical traffic figure redraws vendor-reported counts. External performance and business claims are attributed to their authors; no serving benchmarks were run for this article.

Cover photo: Taylor Vick on Unsplash.