Inside DeepSeek V4.1 Flash: Deriving the 890-Byte KV Cache

DeepSeek V4.1 Flash is best understood as a redesign of where an autoregressive model computes, which layers own memory, and which state deserves to survive a paused session. The headline is 890 bytes of global KV cache per token. The interesting engineering is the set of dependencies that makes that number possible without making every layer identical.

This technical reading follows DeepSeek's 51-page report, Pushing the Limits of KV Cache Compression and cross-checks its architecture against the released code. It assumes familiarity with Transformers, KV caching, and mixture-of-experts routing. For availability, prices, and the Pro migration timeline, see our release analysis.

Evidence status: dimensions and training details below are reported by DeepSeek. Equations marked as derivations are our reconstruction from those dimensions. We inspected the paper and reference implementation; we did not run this checkpoint, reproduce its benchmarks, or measure production GPU memory. Official figures retain their original labels and are reproduced under the repository's MIT license.

The workload: an agent repeatedly returns with more input

A conventional cached decoder stores history so that generation does not recompute every previous token. But an agent repeatedly appends tool output, resumes an old prefix, and moves requests between workers. Three costs become separate bottlenecks: compute for uncached input, HBM for active attention state, and storage plus transfer for reusable prefixes.

The report addresses them with different mechanisms. Causal Encoder–Decoder, or CED, reduces the depth traversed by most prompt tokens. Compressed Sparse Attention 2, or CSA2, reduces duplicated global state and repeated selection. FP4 storage shrinks each retained entry. SWA Bounded Replay changes the trade-off between preserving local state and reconstructing it.

Treating all four as “compression” hides their different failure modes. Quantization changes numerical precision; sparse selection changes what can be retrieved; cross-layer sharing constrains the representation; replay approximates missing history. A good deployment evaluation must exercise each boundary separately.

Forty layers, but only four owners of global KV

DeepSeek report, Figure 3, page 7: the full 20-layer encoder and 20-layer decoder, with Engram, vision embeddings, shared global memory, and DSpark. Arrows and repetition counts are retained from the original.

Open the figure at full resolution

The pinned inference configuration makes the design concrete:

The paper counts 552B backbone parameters plus 196B Engram parameters, with approximately 8B active during prefill and 16B during decode. Adding the first two figures gives 748B for those named components; it is not a byte-size estimate or a complete accounting of every auxiliary checkpoint tensor. The distinction between stored capacity and per-token work is essential.

CED changes the source of decoder memory

In an ordinary decoder, an upper layer builds its historical keys and values from its own hidden states. Constructing that cache requires passing the prompt through the preceding layers. CED breaks this dependency for the global branch: decoder global KV is projected from the final encoder output instead.

Writing H_E for final encoder states, the paper's equation 1 can be read as:

C_l = H_E W_KV,l
Z_l = H_E W_Z,l

C_l contains candidate KV representations and Z_l the compression weights. In the combined released design, the decoder's Full layer owns this global cache and later decoder layers share it. The decoder still forms its own queries and maintains layer-local sliding-window KV from its own hidden states.

This is a causal encoder, not a bidirectional encoder that can see future tokens. During decode, the new token still traverses both halves. The saving concerns prefill: most prompt positions need only the encoder and cheap projections for decoder global memory. A short decoder replay supplies the missing local state.

For prompt length N, total depth L, and local window W, the paper gives the approximate layer-work comparison:

ordinary prefill: N L
CED prefill:      N L/2 + W L/2
ratio:           1/2 + W/(2N)

At N = 1,000,000 and W = 128, that ratio is about 0.500064. This is our arithmetic illustration of the paper's simplified work model, not a predicted wall-clock speedup. Attention kernels, projections, vision encoding, communication, and cache conditions also matter. For a short suffix, the replay term is proportionally more important.

CSA2 separates memory ownership from retrieval decisions

DeepSeek report, Figure 4, page 10: Full computes global memory and indices; Reindex shares memory but selects again; Reuse shares both. Every mode still computes its own main query and local sliding-window KV.

Open the figure at full resolution

The three CSA2 modes are statically assigned to layers. They are not modes chosen by a user prompt.

Sharing KV saves storage. Reusing indices saves selection work. These are independent: keeping the same memory does not require asking the same retrieval question at every depth. Reindex allows deeper layers to choose different entries without allocating another global cache.

Reuse does not copy the earlier layer's attention output. Its new main query can assign different weights to the shared selected entries, and its local branch remains layer-specific. That retained computation is why the design is more expressive than copying one result throughout the network.

CSA2 also simplifies sequence compression. It removes CSA's overlapping source groups and separate absolute-position embeddings in the compressor. Indexer K is projected from main KV rather than following another compression path from hidden states. Compression ratio one is a supported case: sparse retrieval and cross-layer reuse remain useful even without merging adjacent token positions.

Reading the exact layer schedule

Using zero-based layer indices, layers 0–1 are local-only. Encoder layers 2–19 form three groups of six: one Full layer followed by five Reuse layers. Their compression ratio is two. Decoder layers 20–39 form five groups of four with ratio one. Layer 20 is Full; layers 24, 28, 32, and 36 are Reindex; the remaining decoder layers are Reuse.

encoder: SWA, SWA, [Full, Reuse ×5] ×3
decoder: [Full, Reuse ×3], [Reindex, Reuse ×3] ×4

The result is four global-memory owners and eight layers that produce index selections. All 40 backbone layers still have local attention and expert computation. Do not multiply a single global cache by 40, or conclude that only four layers actually run.

The configuration is a useful implementation contract. A port should preserve the association between a KV owner and the selections made against its positions. Reusing an index tensor from a different compression ratio or sequence offset can produce plausible-looking output while attending to the wrong history.

Hierarchical indexing bounds the later searches

DeepSeek report, Figure 5, page 11: the first decoder indexer builds a block-based candidate pool; later Reindex layers choose their own Top-512 within it.

Open the figure at full resolution

The first decoder Full layer still scores the causally visible global positions. In addition to selecting its own 512 entries, it ranks blocks by their maximum position score. It retains up to 2,048 blocks of eight positions, creating a pool of at most 16,384 candidates. Later Reindex layers choose their own Top-512 from that pool.

The pool is deliberately larger than the attention selection. At one million positions, a later indexer can search about 61 times fewer candidates than a full scan. That is a candidate-count ratio, not an end-to-end acceleration factor. The first global scan remains, encoder indexing still has its own work, and attention must read the selected values.

An early selection error also constrains later layers: they cannot recover a remote position excluded from the shared candidate pool merely by changing their query. The paper introduces this restriction during post-training so that the model learns under the same search domain. It is not a harmless inference-only mask.

Deriving the 890 bytes per token

The paper gives the cache formats; the layer schedule explains their multiplicity. Here is a storage derivation for the packed global cache, excluding alignment and allocator overhead.

Main KV has 512 channels. E2M1 values use four bits each, with one one-byte E4M3 scale for every 16 channels:

main entry = 512 × 4/8 + 512/16 × 1 = 288 bytes

Indexer K has 128 channels. Its MXFP4-style storage uses four-bit values and one one-byte scale per 32 channels:

indexer entry = 128 × 4/8 + 128/32 × 1 = 68 bytes
one global entry pair = 288 + 68 = 356 bytes

Each of the three encoder owners stores one entry per two original tokens. The decoder owner stores one per token:

global bytes per original token
  = (3/2 + 1) × 356
  = 890

At one million tokens, that is 890 MB in decimal units, approximately 849 MiB, for global KV alone. At the configuration's 1,048,576 positions, it is 890 MiB. Local KV, weights, Engram, temporary buffers, communication, replication, and allocation overhead are outside this accounting. It does not mean the model fits in a one-gigabyte GPU.

DeepSeek's original global-cache comparison. The reported 890-byte point measures global KV per token, not model weights or total process memory.

Open the figure at full resolution

The distinction is visible in the reference model implementation: it simulates low-precision values in readable tensor operations and allocates cache buffers with its runtime default dtype. Measuring those buffers is not a reproduction of the production packed-cache claim.

FP4 is a storage choice with a training requirement

Main KV quantization is applied after rotary position encoding. DeepSeek uses E2M1 with E4M3 block scales, following the local scaling idea of NVFP4 while omitting a second global scale. It dequantizes values for attention, so native matrix multiplication in that exact storage format is not required just to read the cache.

The paper introduces main-cache quantization-aware training during post-training. Local SWA KV remains FP8 because it is more sensitive. “FP4 model” is therefore too imprecise: expert weights, indexer tensors, global KV, and local KV have different roles and formats.

A backend port must preserve block size, scale dtype, placement relative to RoPE, and rounding behavior. A four-bit container with a different scaling convention is not equivalent. Compare logits and task outcomes against a higher-precision reference, including unusually large activations and contexts near the supported limit.

Bounded replay trades exact state for cheaper resumption

Layer-local attention creates a subtle dependency. With a window W and depth L, exact reconstruction of deep local state can require replaying roughly L × W earlier tokens. For the 20-layer decoder and W = 128, that is 2,560 positions rather than 128.

DeepSeek replays only the last 128 positions and truncates local attention at the replay boundary. For replay starting at s, a query at i sees local positions from max(s, i − W + 1) through i. These states are approximate, not mathematically identical to an uninterrupted full forward pass.

There are two distinct paths. Encoder replay handles a global-prefix cache hit whose local SWA state has expired: replayed tokens rebuild only local state while preserving cached global KV; the uncached suffix generates both. Decoder replay runs the final prompt segment through the upper half to prepare local state for generation. Decoder SWA is not persisted as reusable prefix state.

This changes the storage policy. Long-lived global memory stays in persistent storage; short-lived encoder SWA uses a host-memory pool and can be reconstructed after eviction. The report's roughly eightfold persistent-cache reduction combines smaller global entries with removing SWA from the long-retention store. It does not follow from four-bit quantization alone.

The paper acknowledges that resumed states can depend on the cache-hit boundary. Evaluate repeated resumptions, short follow-up turns, and facts crossing that boundary. Bitwise equality is the wrong success criterion for this approximate path; task-quality changes and worst-case failures are the relevant measurements.

Single-Pass mHC removes a reduction dependency

Manifold-constrained Hyper-Connections carry several residual streams. Let X_l be the streams, A the input mixing, B the residual mixing, and C the output expansion. The conventional form is:

X_(l+1) = B_l X_l + C_l F_l(A_l X_l)
(A_l, B_l, C_l) = H(X_l)

Input mixing must wait for A_l, which depends on a reduction over the current hidden state. V4.1 shifts that particular coefficient by one block:

X_(l+1) = B_l X_l + C_l F_l(A_(l-1) X_l)

Now a tile can contribute to input mixing and next-coefficient prediction in one pass. The production Mega-mHC kernel fuses residual update, mixing, coefficient prediction, normalization, and conversion. For n residual streams of width d, the paper compares (4n + 4)d activation traffic in the original implementation with (2n + 2)d in the fused design. With four streams, that is 20d versus 10d values moved.

This is a halving of traffic for that residual operation, not a halving of all model bandwidth. The architectural shift enables the kernel optimization; it cannot be assumed to be a numerically identical rewrite of the original mHC network.

Engram stores conditional memory outside expert computation

Engram contributes two approximately 98B-parameter modules at zero-based layers 1 and 14. Each uses token n-gram orders 2, 3, and 4, with eight hash heads per order and 2,048 embedding dimensions per order. Distinct prime-sized tables reduce systematic collisions. Context-aware gating determines how retrieved memory enters the residual computation.

The address depends on token identities, so a serving system can begin fetching relevant rows before reaching the consuming layer. The report describes FP8 embeddings and background host-memory transfers overlapped with useful work. This gives large stored capacity without performing a dense multiplication over every memory row for each token.

It is neither a user-editable knowledge database nor conversation KV. Engram stores learned parameters; KV stores request state. Deleting a prefix cache does not delete learned Engram memory. Compared with the earlier Engram design, V4.1 removes the short causal convolution and uses momentum updates with Sinkhorn balancing for the tables.

The operational question is whether lookup latency is hidden at your batch size and interconnect speed. A small active-parameter count says nothing about whether a host-memory fetch misses its deadline. Include table placement and bandwidth in capacity planning.

Native vision enters before the language backbone

DeepSeek-ViT has 32 layers, width 1,024, 16 heads, and a patch size of 14. A 3 × 3 pixel-unshuffle folds spatial neighborhoods into channels before a two-layer projector maps them to language-model width. The resulting visual embeddings join text embeddings in the same causal sequence.

For a 1,344 × 1,344 image, the arithmetic is explicit: 1344/14 = 96 patches per side, then 96/3 = 32 visual positions per side, or 1,024 visual tokens. This is an architectural example at that resolution, not a claim that every image is billed at that count.

Routing also distinguishes modalities. Separate correction biases balance image-token and text-token expert loads; unmodified routing scores still determine the selected experts' output weights. Aggregate load alone could hide a modality concentrating on a few experts. This mechanism addresses training utilization, not a promise of equal accuracy across languages or document types.

DSpark predicts drafts, then decides how much to verify

DSpark uses three Transformer blocks with a 128-token window. It computes base logits for five draft positions in parallel, adds a lightweight Markov head for token dependencies, and predicts conditional acceptance probabilities. A scheduler combines estimated prefix survival with measured engine throughput curves to choose a verification length under current load.

Five draft positions do not guarantee five accepted tokens or a fivefold speedup. The target model must verify the proposal; rejected work, batch occupancy, and verification cost determine the gain. A scheduler can prefer a shorter prefix when verifying a longer one would reduce system throughput.

The released inference guide explicitly says generation is ordinary autoregressive sampling despite including the DSpark forward path. It also describes the small self-test as a check of shapes and kernel plumbing with uninitialized weights. Neither running that test nor seeing a forward_spec method establishes production speculative-decoding performance.

Training connects the architecture to usable behavior

The report describes 45T pretraining tokens, starting sparse attention at 64K context and extending to 1M after 34T tokens. Its post-training recipe remains supervised fine-tuning, reinforcement learning, and on-policy distillation; it attributes the main post-training changes to data and environment construction rather than a new RL algorithm.

Reasoning effort is trained as a scalar from 1 to 100. Within each prompt-and-effort subgroup, the reward includes a capped length penalty with an exponentially decreasing coefficient:

length_penalty = -min(C_max, k(b) × reasoning_length / L_norm)
k(b) = k_0 × exp(-(b - b_min) / tau)

Higher effort weakens the pressure to stop early. The report maps public API low, high, and max to 50, 75, and 100. The prompt-encoding reference documents numeric effort in the underlying format; that does not imply every hosted SDK accepts an arbitrary integer in its public effort field.

Architectural efficiency and reasoning length interact. Halving input work does not guarantee a cheaper completed task if a chosen effort produces far more output. Compare cost and completion at fixed effort first, then evaluate the quality–cost frontier rather than attributing every change to the cache design.

How to audit a port or a serving claim

Begin with the source arrays in the configuration. Trace ownership through Compressor, Indexer, Attention, and the shared-attention state in the reference model. Confirm that index selections refer to the correct cache owner and that position arithmetic remains correct across compressed chunks.

Then separate reference semantics from optimized execution. The readable indexer forms scores before masking the candidate domain; that is not evidence of a production kernel which avoids scoring excluded positions. The reference forward traverses the backbone, rather than demonstrating the complete prefill-skipping, bounded-replay, disaggregated production pipeline. Packed storage and scheduling need their own evidence.

A practical stress matrix should include cold prefill, full cache hits, global-only hits, repeated resumptions, long retrieval with distractors, image-heavy sequences, and concurrent sessions. Record first-token time, inter-token time, total completion time, peak memory, bytes transferred, and accepted-task rate. The open-model operations guide provides the surrounding revision and rollout discipline.

The paper itself identifies sparse-selection errors and approximate replay boundaries as incompletely characterized. Its engineering contribution is a coherent set of trade-offs across computation, memory, and storage. Reproducing that contribution requires preserving both the representation and the execution policy—and testing the cases where their approximations interact.

Source notes — reviewed 2026

Reviewed September 11, 2026. Report page numbers refer to printed pages. Figures 3–5 are direct renderings; the cache chart is the repository's original asset. All original diagrams remain DeepSeek's work. Calculations, explanatory comparisons, and proposed tests are ZharfAI analysis.

• DeepSeek V4.1 Flash technical report: architecture pp. 7–15; serving pp. 18–20; configuration pp. 21–23; effort pp. 29–30; limitations p. 37.
• Pinned inference configuration: dimensions, layer ownership, candidate sizes, and vision settings.
• Pinned model implementation: executable reference of attention, sharing, precision simulation, residuals, and draft forward path.
• Pinned inference guide: reference-runtime scope and self-test limitations.
• Pinned encoding guide: prompt format and reasoning controls.