HBM Explained: Why AI GPUs Run Out of Memory Before They Run Out of Compute

When people compare AI chips, they usually start with compute: how many trillion operations per second a GPU can perform. But for running large language models, the number that often decides real-world speed and cost is memory. How much a GPU can hold, and how fast it can move data in and out, sets how big a model fits on one chip and how quickly it can produce each word.

That memory is high-bandwidth memory, or HBM. It sits beside the processor on every flagship AI accelerator, from NVIDIA's H100 to AMD's Instinct MI355X, and it is a big reason these chips are scarce and expensive. This guide explains what HBM is, how to read the memory lines on a spec sheet, and how to estimate whether a model will fit.

Key facts at a glance

• HBM stacks DRAM chips vertically and connects them to the processor through a very wide interface, trading distance and pins for bandwidth.
• The current standard is HBM4, published by the industry body JEDEC on April 16, 2025: a 2048-bit interface at up to 8 Gb/s per pin, for up to 2 TB/s per stack.
• Capacity has more than tripled in three years: NVIDIA's H100 SXM ships with 80 GB, the H200 with 141 GB, and AMD's MI355X with 288 GB.
• Model weights alone need about 2 bytes per parameter at 16-bit precision, so a 7-billion-parameter model takes roughly 14 GB before any user data.
• Generating text is usually memory-bound, meaning faster memory, not more compute, is often what speeds it up.

Who this guide is for

This explainer is written for engineers, IT buyers and technically minded readers in the US, UK, Canada and Australia who are choosing GPU instances, comparing cloud offers or trying to understand why AI hardware headlines keep mentioning memory. You do not need a hardware background. If you can read a spec table and do simple multiplication, you can use the sizing method below.

What HBM is and why AI chips use it

Ordinary graphics cards use GDDR memory chips placed around the processor on the circuit board. Each chip connects through a relatively narrow bus running at very high speed. That works well for games, but large AI models need to stream tens or hundreds of gigabytes of weights through the processor for every batch of output.

HBM takes a different approach. It stacks several DRAM dies on top of each other and places the stack right next to the GPU on the same package. Because the connection is so short, it can be extremely wide. JEDEC's HBM4 standard defines a 2048-bit interface per stack, with transfer speeds up to 8 Gb/s, which works out to up to 2 TB/s of bandwidth from a single stack.

HBM4 also doubles the number of independent channels per stack, from 16 in HBM3 to 32, each split into two pseudo-channels. More channels let the processor run more memory operations in parallel. The standard supports stacks 4, 8, 12 or 16 dies high, with die densities of 24 or 32 gigabits, giving up to 64 GB in one 16-high stack.

Power is part of the story too. HBM4 lets vendors choose lower supply voltages (as low as 0.7 V for one rail), and it adds Directed Refresh Management to better resist "row-hammer" style memory disturbance. HBM4 controllers are also designed to remain compatible with HBM3 memory, so chip designers can support both.

"Memory bandwidth is one of the key pillars of performance for AI computing systems."

— Nikhil Jayaram, Vice President, Google Cloud Silicon, in JEDEC's HBM4 announcement

Jayaram added that HBM4 represents the big step in bandwidth Google needs for next-generation training and inference systems. Barry Wagner of NVIDIA, who chairs JEDEC's HBM subcommittee, said the standard is designed to drive a leap forward in efficient, high-performance computing for AI, and memory maker Micron said it played a pivotal role in developing it.

How to read the memory lines on a GPU spec sheet

Every AI accelerator datasheet lists two memory figures: capacity (in GB) and bandwidth (in TB/s). Here is how the major data centre parts compare, using the makers' own published specifications.

A few points help when reading this table.

Same compute, different memory. NVIDIA lists identical compute figures for the H100 SXM and H200 SXM, for example 1,979 teraflops at BF16 with sparsity. The H200's advantage is almost entirely memory: NVIDIA says it offers nearly double the H100's capacity with 1.4 times the bandwidth, and quotes 1.9 times faster inference on Llama 2 70B.

System figures are totals. NVIDIA lists the DGX B200 system, with eight Blackwell GPUs, at 1,440 GB of GPU memory and 64 TB/s of HBM3e bandwidth. Dividing by eight gives the per-GPU figures in the table. Always check whether a cloud listing quotes a single GPU or a whole server.

Vendors compare each other differently. AMD's own comparison lists a B200 SXM5 part at 180 GB and 7.7 TB/s against the MI355X's 288 GB and 8 TB/s. Small differences like this often reflect different product variants or clock settings, so compare like for like.

How to estimate whether a model will fit

NVIDIA's developer guidance on inference identifies two main consumers of GPU memory when serving a language model: the model weights and the key-value (KV) cache.

Step 1: Size the weights

Multiply the parameter count by the bytes per parameter. At 16-bit precision (FP16 or BF16) each parameter takes 2 bytes. NVIDIA's own worked example: a 7-billion-parameter model such as Llama 2 7B needs about 14 GB just for weights.

The same arithmetic scales up. A 70-billion-parameter model at 16-bit precision needs about 140 GB, which is more than an H100's 80 GB but fits within an H200's 141 GB only with almost nothing left over. Using 8-bit weights halves the figure; 4-bit formats roughly quarter it.

Step 2: Size the KV cache

As a model generates text, it stores intermediate attention data for every token so it does not recompute them. NVIDIA gives the formula as batch size × sequence length × 2 × number of layers × hidden size × bytes per value. For Llama 2 7B at 16-bit precision, a single request with a 4,096-token context needs about 2 GB of KV cache.

That cache grows linearly with both batch size and context length. Serving 32 users at once with the same context would need roughly 64 GB, more than four times the weights themselves. This is why long-context and high-traffic services hit memory limits before compute limits.

Step 3: Leave headroom

Add the two figures together and leave a margin for the software framework and activations. If the total is close to the card's capacity, plan on a bigger GPU, more GPUs, or a smaller numeric format.

Why memory bandwidth sets your token speed

NVIDIA describes LLM inference as having two phases: a prefill phase that processes the prompt in parallel, and a decode phase that generates output tokens one at a time and is memory-bound. In decode, the GPU must read the model's weights and the KV cache from memory for each new token, so bandwidth often limits speed more than raw compute.

This explains why vendors highlight memory gains even when compute barely changes, as with the H100 to H200 step. It also explains techniques such as multi-query attention, which NVIDIA notes reduce the amount of key and value data read from memory and shrink the KV cache, freeing room for larger batches.

For buyers, the practical rule is simple. If your workload is chat-style generation with long contexts or many concurrent users, weight memory capacity and bandwidth heavily. If it is mostly prompt processing or training with large batches, compute figures matter more.

What can go wrong

• Out-of-memory errors at launch: the weights plus cache do not fit. Reduce batch size or context length, use a smaller precision, or split the model across GPUs.
• Slow responses under load: the KV cache crowds out batching. Cap the maximum context per request or add GPUs.
• Mismatched comparisons: a "per server" figure compared with a "per GPU" figure. Normalise before deciding.

Conclusion: practical takeaways

1. Read capacity and bandwidth before compute when choosing GPUs for inference; they decide what fits and how fast it runs.
2. Use the 2-bytes-per-parameter rule at 16-bit precision for a quick weights estimate, then add the KV cache for your real batch size and context length.
3. Check whether figures are per GPU or per system, especially on cloud listings and server datasheets.
4. Expect HBM4 to raise the ceiling: the standard allows up to 2 TB/s and 64 GB per stack, so the next generation of accelerators should ease today's memory limits.