FMS 2026 | Part 2 of 4
Same brain, two memory systems
I used Part 1 to explain how agentic AI runs on two compute stacks, not one. The GPU reasons, generates and plans. The CPU orchestrates, acts and carries the plan out. Neither works without memory designed for its role.
There I highlighted the scale of the problem: context can grow into tens of terabytes under concurrency, while agentic workflows can drive infrastructure traffic to 100 times the traditional baseline. Now let’s look at how the reasoning stack manages KV cache, then at what the execution stack demands.
Figure 1: Redefining the KV cache hierarchy. Source: IEEE Computer Society (journal article); IBM (technical paper).
Click image to expand
An optimized hierarchy for KV cache
The KV cache is not a location. It is a critical inference object that must be available to feed the next iteration of the decode pipeline. If it is retained, the system has to retrieve it quickly enough to keep inference moving. If it is discarded, the system has to recompute it, spending time, energy and compute to reproduce work it has already done. That makes KV cache placement a system-level design choice: what to keep, where to store it and when retrieval is more efficient than recomputation. Some systems may use as few as two storage tiers. The five-tier architecture shown here represents an optimized future state, with different capacity and retrieval-time tradeoffs at each level. Near memory, or HBM, holds the most active KV closest to the GPU. Main memory adds capacity for active context. Disaggregated memory pools larger working sets beyond the node. Context memory storage provides persistence with faster recall than recomputation. Networked KV lakes extend the hierarchy to exabytes of long-term context. Data center architects will configure their KV Cache tiers to optimize for their unique workload and application base and efficiency will come from placing KV cache in the right tier, at the right time, and orchestrating retrieval, eviction and recomputation optimally.
Figure 2: Executing agentic AI. Source: IBM; Red Hat; Databricks.
Click image to expand
Now the side that acts
Reasoning is only half the job. Agentic AI requires action, and that’s where the motor cortex takes over. Main memory supports operational execution: reasoning, generating and deciding in real time. Disaggregated memory can be used to extend the memory footprint to ensure CPU memory requirements don't outgrow the box, because we’d rather expand the system than underutilize it. Content memory storage lets applications pause and resume, and store local data persistantly. And networked data lakes provide the large scale storage required to source and store the data across the data center. An agent draws on all four tiers with every action while executing on the CPU. Similar memory and storage technologies appear across both stacks, but the pressure comes from different directions. The reasoning side must keep the decode pipeline fed. The execution side must preserve state and turn decisions into action. Both stacks drive the need for greater amounts of faster memory and storage.
Both stacks, every tier
That’s the architecture breaking down the memory wall: an optimized reasoning hierarchy that can scale to five tiers, alongside a four-tier execution stack. Not every system will implement every tier, and the optimal design will depend on workload scale, latency targets and the cost of retrieval versus recomputation. Design the hierarchy intentionally across both stacks, and the wall comes down. In Part 3, I’ll look at how Micron products are addressing the needs of the entire tier hierarchy of both stacks.
