In the last two weeks, benchmarks sponsored by Couchbase and MongoDB were released that compared their flagship products versus each other and Cassandra. This is a good thing for the NoSQL industry, so long as the benchmarks are open, reproducible, and are not over-engineered to favor one solution over another. Competitive benchmarks can provide valuable information to developers and ops engineers who are evaluating various solutions.
Benchmarks Increasingly Important in Phase 2 of NoSQL Adoption: Mission Critical Deployments
The release of an increasing number of benchmarks isn’t surprising. During the first phase of NoSQL adoption, benchmarks were somewhat less important because most users were experimenting with NoSQL or using it on fairly lightweight applications that operated at small scale. In 2013 we entered phase two of NoSQL adoption, in which enterprises are deploying NoSQL for mission-critical applications operating at significant scale. The use of benchmarks is increasing during this phase, because performance at scale is critical for most of these applications. Developers and ops engineers need to know which products perform best for their specific use cases and workloads.
Different Benchmarks for Different Use Cases
It’s entirely legitimate for benchmarks to focus on use cases and workloads that align with the target market and “sweet spots” of the sponsoring vendor’s products. That doesn’t make them invalid. It just points out the importance of highlighting what those use cases and workloads are so developers and ops engineers can assess whether the benchmark is applicable to their specific situations.
For example, Couchbase is very focused on supporting enterprise-class mission-critical applications that operate at significant scale with mixed read/write workloads. As a result, our benchmarks run on clusters with many servers and reflect those workloads. By contrast, MongoDB is presumably more focused on supporting applications that operate at much smaller scale and therefore sponsored a benchmark with a small amount of data running on a single server. Both are valid, but for completely different situations and users.
Keeping It Fair
To be useful, however, benchmarks need to be fair. Otherwise, they’re of little value to developers and ops engineers. Vendors may complain that a benchmark isn’t fair because it’s focused on a use case and workload that’s not a sweet spot for them. Those aren’t valid complaints. On the other hand, benchmarks need to make every effort to achieve an apples-to-apples comparison and, for example, use the most recent versions of software. Apples-to-apples comparisons can be difficult, because the architectures and operational setups of each product are so different, but significant effort should be made to achieve this. Using the right version of software should be very easy to achieve and should promptly be fixed when it isn’t.
Keeping It Transparent
Transparency implies at least two things: (1) Clearly communicating the use cases and workloads that are being measured, and (2) making the benchmarks open so others can reproduce them, verify the results, and modify them to align more closely with the specific use cases they care about.
A Sign of NoSQL Growth and Adoption
Vendors will continue to sponsor and publish benchmarks, and they’ll continue to gear them toward the use cases the vendor supports best. All of this is just another indicator of the rising importance of NoSQL, which is growing fast. According to a new report from Allied Market Research, the global NoSQL market is expected to reach $4.2 billion by 2020 – an annual growth rate of 35.1% from 2014 through 2020. When done fairly and transparently, competitive benchmarks can help enterprises choose the right product for their particular set of requirements.
English