I’m usually pretty easy-going. I don’t get excited too easily (well—maybe a huge sale on new SSDs, a double discount on DRAM, or free rail kits with a new server purchase would do it—but aside from that, I stay pretty calm and collected).
So what’s got me wound up enough to write this blog? It’s the number of SSD companies—who, frankly, should know better—trying to distinguish themselves in the market by advertising their enterprise SSD performance as “up to” some number. And worse yet, some are succeeding! It got me wondering: What does “up to” performance really mean in the enterprise? Personal storage, sure. But enterprise?
I know what “steady state” performance means— SNIA has even defined it in their Performance Test Specification:
2.1.18 Steady State: A device is said to be in Steady State when, for the dependent variable (y) being tracked:
a) Range(y) is less than 20% of Ave(y): Max(y)−Min(y) within the Measurement Window is no more than 20% of the Ave(y) within the Measurement Window; and
b) Slope(y) is less than 10%: Max(y)-Min(y), where Max(y) and Min(y) are the maximum and minimum values on the best linear curve fit of the y-values within the Measurement Window, is within 10% of Ave(y) value within the Measurement Window.
So in steady state, the performance of the SSD isn’t changing much (and “much” is also defined in detail—thanks to SNIA). But what about a new SSD? SNIA calls that “FOB” and defines it too:
2.1.5 Fresh Out of the Box (FOB): State of SSS prior to being put into service.
Okay, so a FOB SSD is one that has yet to be written to since it left the factory. If we look at the plot below of SSD performance over time—which shows several SSDs being written from FOB to steady state—both of these performance states pretty much jump out at you:
The Easy "Up To" Performance Definition
So now that we understand FOB and steady states, let’s dig in to “up to” performance. The easy “up to” performance definition is: The performance of an SSD at the FOB state. But what if I want to measure write “up to” performance? Once I start to measure write performance, I must have written something to the drive, and then write performance has already started to fall off as the SSD moves to steady state (they all do!), so the SSD is no longer at FOB.
“Up To” Write Performance
Since my write performance drops as soon as I start writing, I can’t possibly know what “up to” write performance is unless I write only a single bit to the SSD. I could time how long that single bit takes to travel from my application down the physical data path to the SSD, and how long it takes for the SSD to acknowledge the one bit. That may be “up to” performance, but when on earth would anyone actually do that?
Since writing data causes the SSD write performance to change, how about I measure write performance by not writing anything? I’ll just look at the interface speed and use that. That’s “up to” performance because my SSD can’t go any faster than the interface, right? Well yes, I suppose. But what about “up to” read performance?
“Up To” Read Performance
If I’m going to read something, I must have written something to the SSD first (some SSDs will report read performance even without any data to read—but it’s a very artificial number). How much do I need to read? I guess I could do whatever makes my SSD look good because, hey, these are contrived tests anyway, right? Or maybe I could just use the maximum interface speed again? No, neither of these is a very good idea for enterprise SSDs.
When I hear sustained performance, I immediately want to ask, “Sustained for how long?” Looking at the SNIA graph (above), if I sustain my performance for an arbitrary length of time, I can claim anything I want—FOB, FOB + 1 bit, or FOB + just a little bit more. And because I didn’t tell you how long my SSD sustained the performance, I’m safe, right? That may be technically correct, but it really isn’t right.
Steady State Numbers Tell the Right Story
This begs the question: Enterprise SSD vendors—what do you want to do? Want to hit a home run with your customers? Or is a foul tip good enough? In the enterprise market, SSDs get hammered, hammered, and hammered again. Customers expect that their storage devices will “take a licking and keep on ticking” (thanks Timex). They don’t care one whit what an SSD will do for a few seconds, minutes, or hours. They care what it will do next week, next month, and next year. They care how well it performs over the long haul—in steady state. So instead of advertising vague “up to” numbers—go with something more tangible and relevant—broadcast steady state performance, please! And for you customers out there, don’t settle for anything less.