How Much GPU Memory Do You Need for Local AI?
Model weights are only one part of the memory budget. A useful estimate separates weights, working memory and request-dependent state, then checks the result on the intended runtime.
Calculate a lower bound for the weights
For a simple estimate, multiply the parameter count by the bits per stored parameter and divide by eight. Using decimal units, an 8-billion-parameter model is approximately 16 GB at 16 bits, 8 GB at 8 bits or 4 GB at 4 bits. A 32-billion-parameter model is approximately 64, 32 or 16 GB respectively.
These are arithmetic examples for an idealized uniformly stored model. They exclude scales, metadata, higher-precision tensors and runtime allocations. Real quantization formats mix storage choices, so use the actual artifact size and the runtime's documentation when available. GB and GiB are different units; keep the same unit throughout your comparison.
| Illustrative model | 16-bit weights | 8-bit weights | 4-bit weights |
|---|---|---|---|
| 8B parameters | 16 GB | 8 GB | 4 GB |
| 32B parameters | 64 GB | 32 GB | 16 GB |
Budget for inference, not just loading
The serving process also needs execution buffers and model-dependent state. Transformer inference commonly retains a key/value cache. Longer sequences and more simultaneous requests can increase the required memory. Cache behavior varies by model architecture, precision and runtime; do not apply a conventional attention formula blindly to hybrid or state-space models.
A model that starts successfully with a short prompt may still fail under the intended production workload. The important test includes realistic prompts, generated output and concurrent requests.
Do not add memory without an execution plan
Two devices with 40 GB each do not automatically provide an application with the same behavior as one 80 GB device. Decide how weights and execution will be distributed and inspect the communication path. Offloading to system memory can make a model runnable while changing latency substantially.
For unified-memory computers, the operating system and other applications share the available pool. The installed capacity is therefore not a promise that every byte can be allocated to a model.
Validate the configuration
- Fix the model release, artifact, quantization and runtime version.
- Set the maximum input/output lengths and simultaneous requests.
- Measure peak memory during loading and during the target workload.
- Record time to first token, generation rate and failure behavior.
- Keep enough measured headroom for the workload you intend to support, rather than choosing an arbitrary universal percentage.
What to send for a hardware recommendation
Share the model name, whether you need inference or training, the intended number of users and the available installation space. If you already have hardware, include its exact configuration. We can then compare a compact computer, a standalone accelerator and a complete server against the same requirements.
Sources and documentation
- Model memory estimator ↗Hugging Face Accelerate · checked 17 Nov 2026
Caveats: model loading estimate does not include full inference requirements
- KV cache strategies ↗Hugging Face Transformers · checked 17 Nov 2026
Cache strategies, memory/performance trade-offs and model-dependent applicability