FP4 quantization stores the numbers an LLM works with in 4 bits instead of 16 or 8. Applied to the KV cache, the memory that grows with every token of context, NVIDIA reports that its NVFP4 format uses about 50% less memory than an FP8 KV cache, so roughly twice as much context fits on the same GPU. The catch is quality: how much accuracy you keep depends on which 4-bit format you use and how outliers are handled. This guide walks through the mechanics, the cost effects, the accuracy risks and a practical checklist, using papers and official documentation available as of November 2026.

Two arXiv papers posted on November 6, 2026 put the topic back in focus. BitDecoding-FP4 proposes a way to decode directly from an FP4 KV cache, and CodeQuant2 targets low-precision quantization of large reasoning models (LRMs). Each paper's numbers come from its own models, GPUs and benchmarks, so treat them as data points rather than general guarantees.

What is the KV cache and why is it so big?

LLMs generate text one token at a time, and each new token attends to the key and value vectors of every earlier token. Recomputing those vectors at every step would be wasteful, so inference engines store them once and read them back. That store is the KV cache. As NVIDIA's technical blog puts it, the cache saves compute but costs memory capacity and bandwidth.

The cache grows linearly with context length and with the number of concurrent requests. When the memory pool fills up, older context gets evicted, and if a later request needs it, the engine has to recompute it. A smaller cache therefore buys longer contexts, larger batches and higher cache-hit rates.

A worked sizing example

Per-token KV size is roughly 2 (K and V) × layers × KV heads × head dimension × bytes per value. Take a hypothetical model with 32 layers, 8 KV heads and a head dimension of 128: that is 65,536 values per token. The table shows what one 131,072-token (128K) context would occupy. Real models differ, so plug in your own architecture.

Only the memory left after loading the weights is available for the cache, so dropping one long conversation from 16 GiB to 4.5 GiB changes how many of them a GPU can serve at once.

FP4 quantization formats: FP8 vs NVFP4 vs MXFP4 vs INT4

According to NVIDIA's NVFP4 introduction, the base FP4 layout is E2M1: 1 sign bit, 2 exponent bits and 1 mantissa bit, covering roughly -6 to 6. Four bits alone cannot capture the spread of real tensors, so each small block of values shares a scale factor. How that scale is stored is what separates the formats.

NVIDIA says NVFP4 lands at about 4.5 bits per value, cutting memory by roughly 3.5x versus FP16 and 1.8x versus FP8. The TensorRT LLM tech blog puts it as 0.5625 bytes per value, against 2 bytes for FP16/BF16 and 1 byte for FP8. The MXFP4 figure of about 4.25 bits follows from one 8-bit power-of-two scale shared across 32 four-bit values.

How a 4-bit KV cache changes LLM inference cost

The savings show up in three places. Capacity: NVIDIA says an NVFP4 KV cache is about 50% smaller than FP8 and effectively doubles the context budget. Bandwidth: decoding reads the cache over and over, so a smaller cache means less data moved per token. Cache hits: with more context resident on the GPU, fewer prefixes must be recomputed, which shortens time to first token (TTFT).

• NVIDIA, Qwen3-Coder-480B-A35B: up to 3x better TTFT and up to 20% higher cache-hit rate than FP8, with the gap narrowing once cache memory is large enough.
• BitDecoding-FP4, NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B: 2.4 to 3.0x higher peak serving throughput than a BF16 KV cache.
• TensorRT LLM NVFP4 cold-page compression: only pages offloaded to host memory or disk are stored in NVFP4, and on an agentic replay it completed up to 1.64x more requests with up to 1.9x lower TTFT.

These are best-case figures from different setups. A chatbot that mostly handles short prompts at low concurrency may not be bound by the KV cache at all. Long-document analysis and coding agents that reuse the same prefix turn after turn benefit the most. Also note that NVIDIA's current implementation dequantizes NVFP4 to FP8 before the attention math, so storage shrinks but the attention arithmetic is not cut to a quarter.

To see how GPU supply and AI pricing are being discussed among traders and builders, the Theddari AI Telegram trends page tracks the conversation.

Page Reference

Trendstheddari.com›TrendsTelegram · AILive posts from AI channels

What happens to quality: outliers and where the error comes from

On Qwen3-480B-A35B, NVIDIA measured under 1% loss versus BF16 and FP8. MMLU-PRO came in at about 78.2% (FP16), 78.1% (FP8) and 77.4% (NVFP4); Ruler 64K, a long-context test, at about 95.6%, 95.5% and 94.6%. Format choice matters a lot, though: on Llama 3.3 70B, MMLU scored about 81.9% with an NVFP4 KV cache but about 77.8% with MXFP4.

Those results assume a carefully designed format and calibration. The BitDecoding-FP4 authors argue that existing FP4 KV systems use vanilla quantization that causes severe accuracy drops, and they trace the error to three sources.

1. Query error: splitting each query into two FP4 tensors cut query logit error from 60% to 1% in their tests.
2. Probability underflow: in long contexts, attention probabilities get tiny and E4M3 scales can round them to zero.
3. Key outliers: a few channels with very large values drag the block scale up and wipe out precision for everything else. The paper combines static smoothing, Hadamard rotation and scale search to suppress them.

With those fixes, the paper reports staying within 0.5 points of BF16 KV accuracy with NVFP4 and within 1 point with MXFP4. CodeQuant2 is about weight and activation quantization rather than the KV cache, but its warning is related: reasoning models show larger activation swings in the first decoding steps, and calibration sets that skip those states can miss the outliers. Its clustering-based lookup-table approach reports up to 2.85x speedup over BF16.

LLM quantization checklist before you switch

1. 1

   Confirm hardware support

   NVFP4 is hardware-accelerated on NVIDIA Blackwell. On earlier GPUs, an FP8 KV cache is usually the practical choice.

2. 2

   Check engine flags and calibration

   The vLLM docs note that enabling an FP8 KV cache without calibration sets every scale to 1.0, and recommend dataset calibration with llm-compressor. Calibration matters even more at 4 bits.

3. 3

   Skip sensitive layers

   vLLM documents that some layer types, such as sliding-window attention, are more sensitive to KV quantization and lets you keep them at native precision.

4. 4

   Evaluate on long, multi-step tasks

   Compare BF16, FP8 and FP4 side by side on prompts that match your real context lengths, reasoning depth and code workloads.

5. 5

   Measure throughput and latency together

   Record concurrency, average context length, TTFT and tokens per second under identical conditions before claiming a cost saving.

New quantization papers arrive weekly, but production support usually lags. The Theddari blog trends page is a quick way to see which inference topics developers are actually writing about before you commit engineering time.

Page Reference

Trendstheddari.com›TrendsBlog PostsSummarized Naver blog posts

Bottom line: when a 4-bit KV cache makes sense

• Long contexts, high concurrency and reused prefixes get the biggest win from moving FP8 to NVFP4.
• Quality depends on the format (NVFP4, MXFP4, INT4), block size, calibration data and outlier handling, not on the FP4 label alone.
• Published numbers are best cases on specific models and GPUs. Your own side-by-side evals against BF16 and FP8 are the evidence that counts.
• Middle-ground options exist, such as compressing only cold KV pages that leave the GPU, which trims cost while limiting accuracy risk.