FMS 2026 | Part 1 of 4
The next AI constraint is not compute alone
The next constraint in AI will not be compute alone. It will be memory: how much context systems can hold, how quickly they can move it and how efficiently they can reuse it while agents keep working.
For years, the industry has described this as the memory wall: the widening gap between how quickly processors can calculate and how efficiently systems can feed them with data. That wall was already shaping AI performance. Agentic AI raises the stakes because it turns memory from a component-level bottleneck into a system-level design constraint.
Traditional inference was built around an interaction: a user asked a question, the model generated a response, and the session largely ended. Agents behave differently. They keep track of goals, retain more context, call tools, coordinate with other agents, query data sources and continue working across many steps. That sustained activity pulls memory and storage into the center of AI infrastructure.
Agents change what infrastructure has to remember
That shift is already visible in how leading AI systems are being described. James Bradbury, Head of Compute at Anthropic, has explained that Claude is moving beyond answering questions into carrying work through: reading across a codebase, planning the next step, calling tools and correcting course. Each of those steps increases the amount of context the system has to preserve and reuse.
That is the architectural shift. As agents become more capable, infrastructure has to support persistent, reusable and intelligently tiered context—not just faster math. Memory and storage have to move closer to compute, and the hierarchy has to behave less like a set of separate layers and more like one coordinated system.
Figure 1: The AI revolution. Source: IETF, Carnegie Mellon Institute
From one-time, to real-time, to all-time AI
One way to see the change is through time. Training was one-time: build the model, complete the run, and move on. Inference became real-time: a user entered a prompt and waited for an answer. Agentic AI is all-time. Agents can reason, call APIs, use tools, query databases, coordinate with other agents and keep progressing across many steps.
That shift is moving quickly. Gartner has predicted that up to 40% of enterprise applications will include integrated task-specific AI agents by the end of 2026, up from less than 5% in 2025. The important point is not just how many applications add agents. It is what those agents do to infrastructure: they turn isolated prompts into persistent work.
Figure 2: GPUs reason. CPUs execute. Source: Micron
Two stacks, one memory problem
It is tempting to frame this as a GPU scaling problem. The more useful frame is a full-system one: agentic AI depends on two memory-bound stacks operating together.
The GPU stack handles reasoning and generation. It needs fast access to model weights, active tokens and context. The CPU stack handles orchestration: tool calls, database queries, agent coordination, state management and the workflow around the model. Agentic AI puts sustained pressure on both sides at once.
If the reasoning side stalls, the model waits. If the execution side stalls, the agent waits. Either way, the user experience suffers and utilization drops. That is why the memory wall shows up twice in agentic AI: once around reasoning, and again around execution.
Figure 3: KV scaling greater than LLMs. Sources: KV cache optimization strategies for scalable and efficient LLM inference, vLLM engineering blog
Why KV cache becomes the forcing function
The KV cache makes the reasoning-side pressure tangible. As you can see in Figure 3, a 256,000-token context window can require roughly 22 gigabytes of KV cache for a single session. That may sound manageable in isolation, but AI services are designed for concurrency. They serve many users at once to keep infrastructure utilized.
At 64 concurrent users, that same workload can expand to about 1.4 terabytes. At 1,000 users, it can reach about 22 terabytes. A one-million-token context can require roughly 88 gigabytes for one session, and close to 90 terabytes across 1,000 concurrent sessions.
That is why GPU memory alone cannot carry the full load. HBM is essential for the hottest, most active tokens. DRAM can provide capacity for larger working sets. NAND provides persistence and scale. The design challenge is to make those tiers work together, with data placed where it creates the best combination of performance, capacity and efficiency.
Figure 4: Agentic AI generates 100x the traffic of traditional compute. Source: Google (blog), DigitalOcean (community)
The execution side has its own memory wall
Execution creates the second pressure point. Agents do not just reason; they act. They call APIs, search documents, retrieve data, update plans and coordinate work across tools. Each action creates memory movement and storage access outside the GPU stack.
That changes the traffic pattern around the CPU stack. Traditional applications may generate bursts of activity around a user request. Agentic workloads can sustain that activity across many steps, creating far more tool calls, database queries and document-store accesses than conventional flows.
So the solution cannot be only more GPU capacity. The CPU side also needs the bandwidth, capacity and storage performance to keep agents moving while the reasoning stack stays fed. Reasoning and execution have to scale together.
Figure 5: Redefining the KV cache hierarchy Source: Micron
Breaking the wall requires a coordinated hierarchy
The memory wall was built by physics. Agentic AI is now pressing against it from two directions at once: reasoning and execution. Breaking through will not come from simply adding more of everything. It will come from putting the right memory in the right place, at the right time, for the right job.
That calls for a coordinated hierarchy across HBM, DRAM, NAND and the data infrastructure agents depend on. Memory and storage can no longer be treated as background plumbing. They are strategic design points for AI systems.
In Part 2, I’ll look at what that architecture needs to become—and why the path forward is a coordinated memory hierarchy where HBM, DRAM and NAND work together to keep agents reasoning, acting and scaling efficiently.
