logo-micron

Add Bookmark(s)


To:

Email


Bookmark(s) shared successfully!

Please provide at least one email address.

Microchip background

Improve Apache Kafka performance with flash storage.

Improve Apache Kafka performance with flash storage.

Flash storage enables higher throughput which means a higher number of sustained messages per second can be processed.

Big Data

My official role at Micron is Principal Storage Solutions Engineer. What that title really means is that I work to determine how flash storage can improve the performance and results of workload applications in the real world. To that end, I decided to evaluate one of the most used distributed messaging systems in the Big Data ecosystem, Apache Kafka. I wanted to see how Micron SSDs could be best applied to Apache Kafka and what sort of gains might be had by doing so. If you aren’t familiar with Apache Kafka, you might want to check out this, http://kafka.apache.org/, for a quick primer.

Monitoring the various resources involved, i.e. CPU, memory, disk activity and network, I saw that the primary bottleneck factors were disk and networking, in that order.

It’s A Throughput Game

What I found is that, with Apache Kafka, it was a throughput game. The Kafka developers have done an excellent job of passing writes directly to the kernel page cache to minimize any amount of I/O related issues. However, regardless of this development feature, the I/O still eventually translates into sequential writes being made to the Kafka partition (topic log file). So obviously, the higher the throughput of the disk being used, the more significant increases in performance should be achieved.

Once it was figured out how best to test Apache Kafka as well as what parameters to best use, a simple test was done using their built-in producer test scripts. A total of 600 million 100 byte messages were sent from 3 producers to a single Kafka broker.

The test consisted of the following: (No tuning was done, only default configurations were used.)

  • A total of 600 topics were generated.
  • Each producer was assigned 200 topics, specific to that producer.
  • Each producer created 1,000,000 messages per topic.
  • The message size used was 100 bytes per message.

Apache Kafka Test

Hardware used:

  • Each server, the 1 broker and 3 producers were identical in configuration.
  • Two Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz processors.
  • 384 GB of memory
  • Two 10Gb NICS bonded in ALB mode.

A comparison was done using a 6TB 7.2k HDD, the Micron 5100 ECO 1920GB SSD and the Micron 9100 Pro 3.2 TB NVMe drive.

For each test, the Apache Kafka broker partitions were located on drive being tested.

The results were as follows:

Drive Drive Throughput MB/s I/O per Second Messages (MBytes)
6TB 7.2k HDD 160 300 13.16
Micron 5100 ECO 1920GB SSD 500 2,000 37.94
Micron 9100 Pro 3.2TB NVMe 960 7,500 62.21

As you can see from the above table, the higher the throughput translates into higher I/O per second which for Apache Kafka, means a higher number of sustained messages per second can be processed, shown in MB/s.

Conclusion

Using a higher throughput disk device in your Apache Kafka configuration, such as the Micron 5100 series SSD or a Micron NVMe SSD, will significantly increase performance in your Apache Kafka installation.

Login or Sign Up Now for an account to leave a comment.