Every time you ask an AI a question, something remarkable happens beneath the surface. The model doesn't just "think"; it remembers. It tracks every word, every relationship, every nuance in your conversation. That memory is an investment. It's measured in gigabytes, burned in milliseconds, and it is quietly becoming the single most valuable asset in AI infrastructure today. It's called the KV Cache, and it's not just growing. It's compounding.
What Is a Token, and Why Should You Care About Its “Dividend”?
A token is the smallest unit of data that a system uses to represent, process, or authenticate information, depending on the application. Think of tokens as the currency of AI. Every word, syllable, or punctuation mark you type gets converted into tokens, the fundamental unit of meaning for a large language model (LLM). When a model generates a response, it doesn't just look at your latest message. It looks at everything, the entire conversation history, and computes relationships between every token using what's called the Attention Mechanism: the model's working memory.
“The KV cache is the memory dividend on every token your AI generates, the investment that compounds into context, coherence, and continuity.”
To avoid recomputing these relationships from scratch with every new word generated, models store intermediate mathematical representations, Key (K) and Value (V) vectors, in what is known as the KV Cache. The formula that governs its size is unforgiving:
layers × heads × head_dim × sequence_length × bytes per element
Here's the tokenomics reality check: that cache doesn't scale gently. A single 100K-token request on a 70B-class model like Llama-3.3-70B already requires 10s of GB of KV cache. Scale to a frontier-class model like Llama 405B and a single 1M-token context already requires more than 2 TB of combined HBM and DRAM, on one system, for one user. Now multiply by tens of concurrent users. You're looking at tens of terabytes of KV cache demand from a single mid-size deployment. That’s the per-server reality. The fleet-wide picture is orders of magnitude larger, and it’s growing fast.
And the problem compounds further: reasoning models, the kind powering today's most capable AI assistants, generate 3–10× more tokens internally than they return to the user. Every one of those intermediate tokens must be cached, dramatically inflating the memory footprint of each request.
HBM on current GPU platforms, with up to 288 GB per node, still cannot hold all the KV cache for concurrent long-context sessions. Something must give.
A Macro Shift Nobody Can Ignore
Before we look at the solution, it's worth understanding the sheer scale of the wave approaching. AI compute today is roughly split between AI training and AI inference. In the next couple of years, inference becomes the dominant workload, accounting for two-thirds of all AI compute.[1] By 2030, inference is projected to represent 70–90% of all AI compute demand.
Inference is the AI business. And memory and storage are inference economics.
Zoom out from any single server to the industry, and the trajectory becomes stark. Leading model context windows have grown approximately 30× annually between 2023 and 2025, from 128K tokens to 1M tokens, and they continue to grow.[2] Today, only a small percentage of deployments use long context windows. As that percentage grows, driven by multi-document reasoning, large software repositories, agentic workflows, and multimodal applications, total KV cache demand across the global inference fleet accelerates in lockstep. We are not talking about incremental pressure on memory capacity. We are talking about a step-change in the memory and storage requirements of the entire AI industry.
Algorithmic improvements continue to reduce KV cache per token. But here is the critical insight: every efficiency gain gets immediately reinvested into longer contexts, more concurrent users, and more complex agentic workloads. Agentic AI alone demands 10–40× more memory capacity than standard inference. The demand for memory and storage is not diminishing. It is accelerating.
Click image to expand
KV cache memory and storage tiers for AI inference, from near memory through networked data lakes.
The Pyramid That Compounds the Returns
The solution is not a single memory technology. It is a five-tier hierarchy, a pyramid where KV cache data flows downward as it ages, with each layer purpose-built for a specific moment in the inference lifecycle. Each section lists the applicable memory or storage products, KV cache storage size, data bandwidth, and the time KV cache is typically resident for recall.
Memory and Storage Tiers
| KV Cache Storage per GPU | Bandwidth per GPU | Purpose (Data Residency) | Memory Category | Memory Solutions |
|---|---|---|---|---|
| 10 – 100 GB | 1 – 20 TB/s | Active token generation (milliseconds to seconds) | Near Memory | Micron HBM4 and HBM3E, Micron GDDR7 — Memory closest to GPUs and other accelerators needs extremely high bandwidth to enable LLM training and inferencing |
| 200 – 400 GB | 100 – 500 GB/s | Long context queries (seconds to minutes) | Main Memory | Micron high-capacity DDR5 RDIMM and Micron SOCAMM2 — Main memory needs a balance between capacity and bandwidth to serve data to CPUs and GPUs |
| 100s of GBs to TBs | 100 – 200 GB/s | Query scheduling, context switching (minutes to hours) | Expansion Memory | Disaggregated memory array of Micron high-capacity DDR5 RDIMM — Network attached memory delivers elastic, pooled capacity to expand the capacity of main memory with comparable performance |
| 10s to 100s of TB | 20 – 200 GB/s | Fast-paced multi-turn workflows (hours to days) | Context Memory Storage | Micron 9650 SSD, Micron 7600 SSD — Inferencing systems can avoid re-computing on long context multi-turn workflows |
| PBs | <10 GB/s | RAG & storage (days to years) | Networked Data Lakes | Micron 6600 ION SSD — Data lakes on network file or object storage store massive amounts of data with the performance needed to serve multiple AI servers |
| NOTE: KV cache is evicted down the layers as it ages | ||||
Table: KV Cache Memory and Storage Tiering in Inference
Tier 1: Near Memory (HBM), The Trading Floor
Micron HBM4, HBM3E, GDDR7 | 10–100 GB of KV data per GPU | 1–20 TB/s | Milliseconds to seconds
At the apex sits High Bandwidth Memory, the fastest, most expensive real estate in AI. This is where active token generation happens: live attention computation and real-time inference. With bandwidths reaching 20 TB/s, HBM is the trading floor of the token economy. Every microsecond counts, and every byte must be immediately accessible.
But it is finite and costly. Model weights alone occupy a significant share of HBM, leaving KV cache to compete for what remains. As models grow larger and context windows expand, HBM pressure intensifies. When demand exceeds capacity, KV cache must flow down, or inference grinds to a halt.
Tier 2: Main Memory (DDR5 / LPDDR), The Working Capital
Micron High-Capacity DDR5, SOCAMM2 LPDDR | 200–400 GB of KV | 100–500 GB/s | Seconds to minutes
When long-context queries extend beyond the GPU's immediate reach, KV cache flows into main memory. Think of this as working capital: highly liquid and substantially larger than HBM but still governed by bandwidth constraints. Micron's high-capacity DDR5 RDIMMs and LPDDR SOCAMM2 modules strike the balance between capacity and bandwidth needed to serve data fluidly to both CPUs and GPUs.
Long-context queries, the ones powering sophisticated document analysis, legal review, or multi-round research sessions, live here during active processing. As deployments push toward 1M+ token contexts, this tier is under growing and relentless pressure.
Tier 3: Expansion Memory (Disaggregated DDR5), The Credit Line
Disaggregated Micron High-Capacity DDR5 RDIMM | 100s of GBs to TBs of KV | 100–200 GB/s | Minutes to hours
Expansion memory is the elastic credit line of the token economy, comprising network-attached disaggregated memory pools that absorb overflow when query scheduling and context switching demand more than a single server’s DRAM can provide. This tier is critical in multi-tenant environments where many users share infrastructure and workloads shift dynamically. As context length and concurrency climb together, this layer becomes the bridge between what a single server can hold and what an AI service actually needs to deliver.
Tier 4: Context Memory Storage (NVMe SSD), The Long-Term Account
Micron 9650 SSD, Micron 7600 SSD | 10s–100s of TBs of KV | 20–200 GB/s | Hours to days
As AI evolves from single-turn chatbots to multi-step agentic workflows, the KV cache transforms from a transient structure into a persistent storage problem. An AI agent managing a complex enterprise task cannot afford to recompute its entire reasoning history with every new action step. That recomputation is slow, expensive, and economically unsustainable at scale.
Micron's Data Center Workload Engineering (DCWE) team validated NVMe SSD-based KV cache offload across hundreds of test configurations with zero throughput penalty. Agentic workloads demand 10–40× more capacity than standard inference, and multi-turn workflows spanning hours to days make this tier's role mission-critical. As agentic AI goes mainstream, context memory storage is set for some of the fastest capacity growth in the entire hierarchy.
Tier 5: Networked Data Lakes, The Vault
Micron 6600 ION SSD | Petabytes of KV | <10 GB/s | Days to years
At the base of the pyramid, the widest, deepest layer, sits the networked data lake. This is the vault of the token economy, where KV caches for Retrieval-Augmented Generation (RAG), long-horizon agentic memory, and enterprise knowledge bases reside for days, weeks, or longer. Micron's 6600 ION SSD is engineered to serve multiple AI servers simultaneously from network file or object storage, delivering the density and endurance that petabyte-scale AI memory demands. As RAG architectures proliferate and enterprises build persistent AI knowledge stores, this tier's capacity requirements will be the most staggering of all.
The Bottom Line
The KV cache is the memory dividend on every token your AI generates, the investment that compounds into context, coherence, and continuity. And that dividend is growing in every direction simultaneously: more tokens per request, longer contexts, more concurrent users, and more persistent agentic sessions.
No single memory technology can deliver this dividend alone. The full hierarchy, from HBM at the apex to the networked data lake at the base, must work in concert, with each tier playing its role as KV cache compounds value flowing through the pyramid over time.
As inference becomes the dominant form of AI compute, and as KV cache becomes inference's dominant memory consumer, the companies that optimize the entire memory and storage hierarchy will define who can serve AI at scale, at speed, and sustainably.
The pyramid isn't just an architecture. It's the economic foundation of AI at scale, and every performance gain, every efficiency win, and every TCO advantage flows from getting this hierarchy right. Every tier of that pyramid is a Micron product.
References
[1] Deloitte Insights, Nov. 2025. “Why AI’s next phase will likely demand more computational power, not less.”
[2] Epoch AI, June 2025. “LLMs now accept longer inputs, and the best models can use them more effectively.”
