How Fast Is Fast? Understanding Competitive Benchmarks for Flash Storage

By Andrew Mierau - 2015-12-03

Quick, tell me how much faster your laptop will be if I replace your drive with one that has 1000 more IOPS? What if I increased the transfer speed by 50 MB/s? Would that make it faster?

Most people don’t know. Most people don’t want to know what IOPS and MB/s are. They want to know that the money they just spent on an SSD is going to change the way they work (and play). To put it in simpler terms, how much faster will it make my system? Will I notice? Can I brag about it to my buddies?

Benchmarking tools answer a lot of questions, but do they answer ones that are relevant to you?

Benchmarks are like a freshly sharpened gourmet kitchen knife: in the hands of a skilled chef it will help you make wonderful meals. But you might cut your finger if you don’t use it correctly.

When you read reviews for an SSD you will likely be presented with results from a list of common synthetic benchmarks (ATTO, Crystal Disk Mark, IOMeter, and the like). These are all good: they enable a standardized way of measuring performance. But they are also bad: they all access SSDs in a way that most users never will. Consumers will almost never do what these test do – we don’t usually write a continuous, random 4KB data stream across the entire span of our SSD. We just don’t use SSDs like that (but benchmarks do!).

We do actual work with our SSDs and when we talk about this work (you’ll also hear the term “workload”), we are talking about the typical consumer. Not the gamer. Not the enthusiast. Rarely the business/professional user. Those users have much more robust storage demands.

Typical consumer workloads are both light and sporadic. We don’t really do all that much (from a storage perspective) and we don’t do it all that often. With SSDs, writing data is by far the most strenuous activity but most consumers write very little. I used my first SSD for over two years and barely wrote 2TB of data…total…over 2 years! Sure, I read many times that. And when I did rarely write data to the SSD, the majority of what I write was through installing applications and copying photos, music and videos from a much slower source (DVD, SD Card, and the Internet).

Consumer workloads are also sporadic in that they spend most of their time idling. Even when you think you are stressing your drive, you probably aren’t. When looking deeper at what is really happening (I’m talking at the electrical level…way, deep down), there is a huge portion of time where the SSD is doing nothing at all.

Some application benchmarks (such as SYSmark) attempt to capture this type of usage by running applications that you will actually use and show an average “score” (performance rating). When looking at these results, a couple of things stand out. First: there is a large speed difference between HDDs and SSDs. Even for consumer workloads, which are read centric, reading data from an SSD is orders of magnitude faster than an HDD.

Second: that there isn’t always a big difference in performance between SSDs because most consumer SSDs have very similar read performance, which makes up most of a consumer workload.

So if synthetic benchmarks don’t test a common use case and application benchmarks aren’t able to differentiate SSDs, how do we know which SSD to choose?

If the choice is between an SSD and an HDD, the answer is simple: choose a Micron or Crucial SSD (although, as a Micron employee I may be biased). Even the most casual user will notice a huge increase in productivity when they transition from an HDD to an SSD.

When choosing between different SSDs, read the media and customer reviews but understand that certain benchmark results won’t always translate into real-world performance. Even a 20-30% difference may not be observable in everyday tasks.

Be sure to look beyond the benchmarks too: every key feature/benefit may not show up in benchmark results. What is the warranty? Does the manufacturer say anything about reliability? If so, is it as good as other SSDs that are otherwise similar?

Benchmarks are great for giving you a lot of data that you can use to make a final choice. But they don’t show everything. Remember – that shiny knife can also cut your finger.

Andrew Mierau