Server sizing for large language models has followed one crude rule: count the parameters, multiply by the precision, buy that much GPU memory. DeepSeek's V4.1 Flash breaks the rule. Its weights total 763 billion, which at FP8 implies a 763GB floor of accelerator memory, yet the model can be served from roughly 567GB because 196 billion of those parameters are designed to run from ordinary system RAM instead.
Key takeaways
- Roughly 196GB of one model's weights can be moved off the accelerator entirely, changing which server configuration a deployment requires.
- The weights that can move are lookup tables read a few dozen entries at a time, so they never compete for the memory bandwidth that limits token generation.
- The saving only holds if the tier beneath the GPU is fast, which shifts engineering effort from GPU budgets toward system memory and storage I/O planning.
Why bandwidth, not capacity, sets the ceiling
Generating a token requires reading a model's active weights out of memory. That read, repeated for every token, is why memory bandwidth rather than raw arithmetic usually caps throughput, and why accelerator memory commands the price it does. Capacity matters, but the reason weights sit on the GPU at all is the speed at which they must be streamed.
That logic only applies to weights that get streamed. Some do not. DeepSeek's conditional memory module holds its 196 billion parameters as N-gram lookup tables β learned associations over short runs of tokens β and consulting them costs a few dozen table reads per token rather than a sweep through the whole block. Weights accessed that sparsely have no claim on the fastest memory in the system.
The practical consequence is a split inventory. About 8 billion parameters stay active per token and must be fast. The lookup pool can be parked in system RAM, or on a storage array quick enough to answer without stalling. The Register's reading of the technical report describes the goal as separating what a model knows from what it must compute.
What this does to a procurement conversation
Roughly 200GB is not a rounding error. It is frequently the gap between one class of node and the next, and therefore between one capital plan and another. A buyer who sizes on parameter count alone over-provisions; a buyer who sizes on the offloaded figure without checking the memory tier under it under-provisions and pays in latency.
Neither error is easy to avoid today, because model cards publish one memory number rather than two. The useful disclosure is a split: how much must be resident on the accelerator, and how much may live a tier down. Expect that question to enter vendor evaluations before it enters documentation.
There is a second figure worth demanding. Weights are only part of what fills a GPU β key-value caches grow with context length and with the number of simultaneous users, and in high-throughput chat or agent workloads they often dominate. The same DeepSeek release attacked that separately, using reworked attention and a new causal encoder-decoder to bring cache use down to between 13 and 25 percent of the previous Flash model. In capacity terms that is four to eight times the concurrent sessions per unit of memory, which for most operators lands closer to home than the parameter arithmetic does.
How portable is the technique
The approach is not proprietary. Google's Gemma team built a related mechanism, Per-Layer Embedding, to fit useful models onto phones, where every tier of the hierarchy is constrained. DeepSeek published its N-gram variant in January, and Alibaba has since built a 51-billion-parameter lookup pool into an experimental Qwen model. The design is spreading because the economics are legible to anyone who has priced an accelerator against a stick of DRAM.
What remains unsettled is how far down the hierarchy it survives. System RAM is a proven destination. Storage is a claim, and one that depends heavily on the array. Teams evaluating this should benchmark their own inference stack rather than trust a headline figure, because the saving is real but conditional, and the condition is infrastructure they already own or do not.
It also adds a variable to change management. A provider that alters what runs behind an endpoint can alter the memory profile with it, a risk that surfaced this week when a pinned model name began serving different weights with no opt-out.
FAQ
Can any model's weights be offloaded to system RAM?
No. Only weights accessed sparsely enough to avoid bulk streaming can move without a latency penalty, which in practice means lookup-style parameters rather than the active weights used for every token. Offloading conventional dense weights reintroduces the bandwidth bottleneck the GPU exists to solve.
How much GPU memory does DeepSeek V4.1 Flash actually need?
Its 763 billion parameters at FP8 imply a 763GB minimum if everything is resident on the accelerator. Offloading the 196-billion-parameter lookup pool lowers that floor to about 567GB. Production serving requires additional headroom for key-value caches, which scale with context length and concurrent users.
Does offloading weights slow the model down?
It depends entirely on the tier receiving them. Lookups served from system RAM are fast enough that the architecture holds, while a slow storage array can stall a lookup and return the latency the design was meant to save. Benchmarking against your own hardware is the only reliable answer.
