Sizing GPUs for a 70B model you have to host yourself

Someone asks for a 70B model. Someone else asks how many GPUs. The honest answer is that nobody can tell you from those two facts, and the gap between the cheapest correct answer and the most expensive one is roughly a factor of eight.

Here is the arithmetic we actually use, and the places it goes wrong.

Start with weights, but do not stop there#

At bf16, a parameter costs two bytes. Seventy billion parameters is therefore about 140 GB before you have served a single request. On 80 GB cards that is two cards’ worth of memory, which people immediately translate into “two GPUs”. It is not two GPUs. It is two GPUs’ worth of weights, with nothing left for the thing that actually varies.

Eight-bit quantisation takes you to roughly 70 GB. Four-bit takes you to roughly 35 GB. Both are real options and both cost you something on task quality, how much is entirely task-dependent, which is why the only defensible way to choose is to run your own evaluation set at each precision. Published benchmarks will tell you a 4-bit model is “within 1%”, on their corpus. Whether that holds on a domain corpus with unusual vocabulary is not something a benchmark can tell you, and it is cheap to check against your own task set before you commit to a hardware order. Measure.

The KV cache is the term that gets forgotten#

Every token in every active sequence needs cached keys and values. Per token, per sequence, the cost is roughly:

2 (K and V) × layers × kv_heads × head_dim × bytes_per_element

Multiply by the context length you actually serve, then by the number of sequences in flight. For a 70B-class model at 8k context, that is on the order of a gigabyte per sequence. Thirty concurrent users is thirty gigabytes, a substantial fraction of a card, spent on nothing but bookkeeping.

This is why grouped-query attention matters so much for self-hosting economics, and why the same parameter count from two different model families can have wildly different serving costs. Check the kv_heads, not just the parameter count.

Then add the parts nobody budgets for#

Activation memory and allocator fragmentation: budget 10–15%. Paged attention reduces fragmentation considerably but does not remove the need for slack.

Evaluation capacity: if your regression suite competes with production for GPUs, your team will stop running the regression suite. This is not a hypothesis; it is what happens. Reserve for it explicitly.

Failure headroom: sizing so that the cluster is at 95% utilisation when everything is healthy means the loss of one node is an outage.

Size to the tail, not the median#

The single most common sizing error we see is arithmetic done against average context length and average concurrency. Real traffic is not average. Someone pastes a forty-page document. Two teams start a bulk review in the same hour.

We size to p95 context and p95 concurrency, and we treat the result as a floor. If your workload has a genuine hard peak (month-end, market open, a scheduled batch), size to that instead and accept the idle capacity, or schedule the batch somewhere it cannot collide with interactive traffic.

The question that saves the most money#

Before any of the above: does it have to be a 70B model?

The comparison worth running is a well-chosen smaller model, properly quantised, with good retrieval in front of it, against the 70B you were going to buy for. On document workloads where the answer is mostly in the retrieved context rather than in the weights, the smaller model is often close enough, and it answers faster on a fraction of the hardware. The 70B earns its cost on tasks with genuine reasoning depth, long multi-step synthesis, and unusual domains where the smaller model’s knowledge simply runs out.

We are not going to tell you which way it lands for you. That is the point: it is a question with a cheap empirical answer and an expensive assumed one.

The way to find out is to build the evaluation set first and run both. That takes about a week and routinely changes the hardware order.

Practical starting points#

• One node, model fits: use a single-node serving engine. Do not introduce a distributed serving layer you will then have to operate at 3 a.m.
• Model does not fit on one node, or one cluster must host several models: now a distributed serving layer earns its complexity.
• Long contexts, moderate concurrency: memory-bound. Buy memory.
• Short contexts, high concurrency, long generations: compute-bound. Buy throughput.

None of this is a substitute for a load test against your own traffic shape. It is a way to arrive at the load test with a configuration that has a chance of passing it.