Local LLM on a Company Server: What Hardware You Need (GPU, VRAM)
How much hardware a local language model really needs in a factory. How model size maps to VRAM and GPU class, why real utilization matters more than peak, and when a single card is enough.
In this note
Local LLM on a Company Server: What Hardware You Need (GPU, VRAM)
Reading time: approx. 7 minutes
Short answer up front: the hardware requirements of a local language model are driven above all by the size of the model, not by the card's "power" in the everyday sense. The key number is VRAM, the memory on the graphics card. A model with 8 billion parameters fits comfortably on a single 24 GB card and is enough for most assistant use cases. A 70B model usually needs several data-center-class cards. Below I break this down, without marketing.
Why VRAM decides, not card "speed"
For a model to answer, all of its parameters (the weights) must fit into the card's memory. If they don't fit, you either add more cards or the model simply won't run with sensible performance. Only once the weights fit does throughput and card architecture matter, because those decide how fast the answers come out.
Roughly, the weights take up this much:
- Full precision (FP16): about 2 bytes per parameter. So an 8B model is about 16 GB, a 70B model about 140 GB.
- 4-bit quantization: about 0.5 byte per parameter. An 8B model drops to about 5 to 6 GB, a 70B model to about 40 GB.
On top of that comes overhead for context (the so-called KV cache), which grows with conversation length and the number of simultaneous users. In practice, leave a buffer of around 20 to 40 percent above the weights alone.
Quantization: cheaper hardware at a small quality cost
Quantization means storing the weights at lower precision. It lets you fit a larger model on a smaller card and speeds up responses. The cost is real but usually small: at 4-bit, most assistant and retrieval (RAG) tasks perform practically indistinguishably from full precision. For tasks that require precise reasoning, the difference can be noticeable. Practical rule: start with quantization, and step away from it only when tests show a real quality drop on your own cases.
How much card for which model
The table below is orientation, not gospel. The real choice depends on context length and the number of simultaneous queries.
| Model size | Typical factory use | VRAM (4-bit) | VRAM (FP16) | Indicative card |
|---|---|---|---|---|
| 7 to 8B | Service assistant, RAG over docs, simple summaries | approx. 6 to 10 GB | approx. 16 to 20 GB | one 24 GB card (RTX 4090 / RTX 6000 class) |
| 13 to 14B | Harder RAG, better answer quality | approx. 10 to 16 GB | approx. 28 to 32 GB | one 24 to 48 GB card (L40S class) |
| 32 to 34B | Advanced reasoning, longer context | approx. 20 to 28 GB | approx. 64 to 70 GB | one 48 GB card or two smaller ones |
| 70B | Highest quality, complex tasks | approx. 40 to 48 GB | approx. 140 GB | two 48 to 80 GB cards (H100 class) |
For most mid-sized companies the starting point is a model in the 8 to 14B range on a single good card. That is enough for a service assistant and documentation search, and the entry cost is bearable.
The number of users changes the math
Model size is one thing. The other is how many people ask at once. Every parallel query consumes memory for context and competes for card time. Ten technicians asking occasionally is a completely different load than fifty typing at once during a shift.
This is where the choice of inference engine, the software that serves the model, comes in. Tools designed for load can handle many times more queries on the same card than simple developer tools, because they pack parallel queries better. In other words: before you add hardware, check whether you are simply missing the right engine.
The trap: planning for peak utilization
A common planning mistake is assuming the card runs at 90 percent of its capacity. In real production deployments, utilization is more like 40 to 65 percent, because traffic is uneven and you must keep headroom for peaks. If you size your needs at peak utilization, you will end up short of resources during the busiest hours. It is better to plan with headroom and realistic utilization.
What else the server needs besides the card
The card is not everything, though it costs the most. On top of it you have:
- RAM and disk for documentation, the search index and logs. With RAG this is usually not a problem, but it has to be planned.
- Power and cooling. Data-center cards draw a lot of electricity and run hot. This is a real line item in running costs, often underestimated.
- Network and access. Who connects to the assistant and from where, and how access is secured.
I cover the full cost picture, where power and people can exceed the price of the card itself, separately in the note on on-premise versus cloud.
What this post doesn't cover
I don't get into choosing a specific model here, nor into whether it makes more sense to put the hardware on your own site, in colocation, or to rent it. I also don't settle when a local model beats the cloud, since that is a separate cost and regulatory decision. The focus here was one practical question: what hardware you need for a local LLM to run at all and serve your team.
Related notes
Related notes
On-Premise vs. Cloud AI for Manufacturing: What to Choose and When
When to keep your AI on-premise and when to use the cloud. An honest checklist: data sensitivity, volume, hidden costs (people, power), latency, NIS2, and the hybrid option.