Invalid input. Special characters are not supported.
Embracing AI: My journey from compute to memory at Micron
A shift in perspective
When I first encountered artificial intelligence, I was captivated by its potential but overwhelmed by how little I understood. My early exploration focused on compute accelerators — CPUs, GPUs and NPUs — and their role in enabling AI on edge devices like smartphones. However, joining Micron marked a pivotal shift in my perspective. I began to see memory and storage not just as supporting players, but as critical enablers of AI innovation. This realization sparked a deeper journey: embracing AI not just as a technology, but as a mindset for product development and transformation.
How Micron is powering AI adoption
As AI models grow in complexity and size, often reaching billions or even trillions of parameters, the demand for faster, more efficient data access has surged. This has brought renewed attention to a long-standing challenge in computing: the memory wall problem. This refers to the growing disparity between processor speeds and memory bandwidth/latency, which can severely bottleneck AI performance, especially during training and inference of large models.
Micron recognizes that overcoming this bottleneck is essential for scaling AI. That’s why we view memory and storage not just as support components, but as critical enablers of AI innovation. Our product portfolio is designed to address these challenges head-on:
- Near memory: Solutions like HBM and GDDR are tightly integrated with CPUs and GPUs, offering ultra-fast access to data and model parameters, minimizing latency and maximizing throughput.
- Main memory: High-capacity, low-latency options such as standard DIMMs, MRDIMMs and low-power DRAM deliver performance with power efficiency, helping to keep up with the demands of modern AI workloads.
- Expansion memory: Technologies like Compute Express Link (CXL) enable scalable memory capacity for data-intensive workloads, reducing the total cost of ownership while addressing memory bottlenecks.
- Storage solutions: From high-performance NVMe SSDs to cost-effective options for data lakes, our storage products are optimized for AI’s demanding I/O needs, ensuring that data is always available when and where it’s needed.
By aligning our innovations with the evolving needs of AI, Micron is helping to break through the memory wall and unlock new levels of performance and efficiency across the AI landscape.
For a deeper dive into Micron’s product portfolio and future innovations, I highly recommend reading this insightful interview with Praveen Vaidyanathan on The Register.
Our innovations are shaping the future of AI
Micron’s leadership in AI-enabling technologies is grounded in cutting-edge innovation:
- 1γ (1-gamma) DRAM: Offers over 30% higher bit density per wafer, up to 20% power savings, and 15% speed improvements over the previous generation. (Learn more here… Micron’s 1-gamma node technology)
- G9 NAND: Delivers the industry’s fastest NAND I/O at 3.6 GB/s, enabling up to 99% better read speeds and 88% better write performance — ideal for data-centric AI workloads. (Learn more here… G9 NAND)
These advancements are not just technical milestones — they’re foundational to building AI systems that are faster, more efficient and more scalable.
Metrics that matter in AI — and how memory and storage influence them
One of the most fascinating aspects of AI is the shared set of performance metrics that span diverse platforms — whether it's data centers, client devices or mobile systems. Despite the differences in application use cases, architectures and software frameworks, the core enablers — compute, memory and storage — are all evaluated against similar system-level KPIs.
When it comes to AI inferencing, especially for large language models (LLMs), the key metrics include:
- TTFT time to first token — How quickly the system can begin generating output.
- Tokens per second — A measure of throughput.
- Tokens per second per watt — A critical metric for evaluating power efficiency.
These metrics are central to both performance and energy efficiency, and they are influenced heavily by the capabilities of memory and storage subsystems.
Storage’s role in AI metrics
From a systems perspective, storage performance is often measured in terms of:
- IOPS (input/output operations per second) — Especially relevant in data center and client SSD environments.
- Read/write speeds — Crucial for mobile platforms using UFS (universal flash storage).
Additional factors like access patterns, chunk sizes and logical block addressing (LBA) sizes introduce further complexity. However, the overarching impact of these parameters on LLM inference performance remains consistent across platforms.
Micron’s innovations in storage technology are designed to meet these demands across all segments, ensuring that storage is not a bottleneck but a performance enabler.
Memory’s expanding role
On the memory front, we’re witnessing a significant shift. Technologies like LPDDR, traditionally used in mobile and client devices, are now making their way into data center environments. This transition is driven by the need for power-efficient, high-performance memory that can sustain demanding AI workloads. (Learn more here... Every watt matters: How low-power memory is transforming data centers)
Key memory configuration parameters, such as rank, channel count, I/O width and density, vary across segments. Yet, the goal remains the same: to deliver sustained performance at optimal speed grades while maintaining energy efficiency.
Micron’s memory solutions, including DDR, LPDDR, GDDR and HBM, are engineered to support the full AI inference pipeline — from embedding to prefill, decoding and post-processing (Learn more here… 1 million token context: The good, the bad and the ugly). This ensures that memory works seamlessly with compute and storage to avoid bottlenecks and maximize throughput.
In essence, whether you're deploying AI in a hyperscale data center or on a mobile device, the same fundamental metrics apply. And Micron’s memory and storage technologies are purpose-built to optimize these metrics, enabling faster, more efficient, and more scalable AI systems.
Food for thought: Dive deeper into AI performance metrics
For those interested in exploring how AI performance is measured across platforms — and how memory and storage play a pivotal role, I highly recommend the following articles:
NVIDIA NIM LLMs benchmarking — A comprehensive look at inference metrics in data center environments.
Large Language Model performance benchmarking on mobile platforms — A detailed evaluation of LLM performance on mobile devices.
Inference = IOPS: Why AI’s next frontier runs on storage — A Micron perspective on how storage performance, especially IOPS, is becoming central to AI workloads.
These resources offer valuable context and technical depth for anyone looking to understand the evolving landscape of AI system performance.
AI at Micron: Enabling AI to build AI
Micron doesn’t just enable AI — we use it to build better products. From smart manufacturing to yield analytics, AI is embedded across our operations. We apply it in:
- Image and sound analysis
- Process automation
- Engineering workflows
- Business operations
This internal adoption of AI enhances quality, efficiency and innovation across Micron.
A story from Micron India: AI in action
At Micron India’s Center of Technical Excellence (CoTE), part of the Core Data Center Business Unit (CDBU), we see the power of AI integration. With nearly every major function co-located and vertically integrated, this center plays a vital role in translating new technologies into commercial products. From defining requirements to enabling customers, AI helps us streamline processes, improve collaboration and accelerate innovation — without compromising on quality or confidentiality.
Final thoughts: Memory as the heart of AI
My journey from compute to memory has been transformative. At Micron, I’ve come to appreciate that memory and storage are not just enablers of AI— they are the foundation upon which AI is built. As AI continues to evolve, Micron’s commitment to innovation ensures we’re not just keeping pace — we’re helping shape the future.
