TPC Benchmarking Results

Feed: SingleStore Blog.
Author: .

We used benchmarks derived from the industry-standard TPC-H and TPC-C benchmarks to evaluate SingleStore compared to other leading cloud databases and data warehouses. The results below show SingleStore achieves leading-edge performance on both TPC-H, an OLAP benchmark, and TPC-C, an OLTP benchmark.

We compared SingleStore with three other popular, state-of-the-art products: two cloud data warehouses we’ll refer to as CDW1 and CDW2, and a cloud operational database we’ll refer to as CDB. Let’s start with a summary of the results across both TPC-H and TPC-C:

Summary of TPC-H and TPC-C results. TPC-H: 10TB scale factor, cold runtimes, lower is better. TPC-C: higher is better, up to the limit of 12.86 tpmC/warehouse

As the results show, SingleStore had excellent performance and cost-performance on the analytic benchmark TPC-H compared to the cloud data warehouses, and on the transactional benchmark TPC-C compared to CDB. The main point of this comparison is to demonstrate that SingleStore delivers state-of-the-art performance on both analytical and transactional workloads, while CDW1, CDW2 and CDB can only run well on either TPC-C or TPC-H, not both. CDW1 and CDW2 only support data warehousing and cannot run TPC-C. CDB can run both benchmarks, but it performs orders of magnitude worse than the cloud data warehouses on TPC-H. Additionally, CDB offers limited scalability on OLTP workloads — previous results show its performance does not scale well to much larger numbers of warehouses.

This chart summarizes the performance on both TPC-C and TPC-H. On this chart, we show the throughput for TPC-C (tpmC, as defined by TPC-C) and the queries per dollar for TPC-H (calculated as 100/geomean query cost in cents). The higher the number, the better. 

Our benchmark runs used the schema, data and queries as defined by the TPC. However, they do not meet all the criteria for official TPC benchmark runs, and are not official TPC results.

We ran the benchmarks in January 2022 on the latest software versions and hardware instance types available from each database vendor — for SingleStore, this was version 7.6.7.

TPC-H

We measured the performance of SingleStore compared to two popular, state-of-the-art data warehouses, CDW1 and CDW2, on TPC-H, an industry-standard OLAP benchmark at the 10TB scale factor.

We ran comparisons on public cloud instances with cluster sizes that were chosen to be as similar in price as possible. The costs of the cluster configurations we used are shown in the appendix.

We measured the runtime of one cold run of each query, and then measured the average runtime of three warm runs of each query, with results caching disabled. The warm run allows for query compilation and data caching.

We computed the cost of each query by multiplying the runtime by the price per second of the product configuration. We then computed the geomean of the runtime and cost results across all the queries.

When we tested the operational database CDB on the largest available size, it performed orders of magnitude worse: most queries failed to complete within 1 hour, and a single run of all the benchmark queries failed to complete within 24 hours (compared to about 15 minutes for SingleStore, CDW1 and CDW2).

The results, shown in the chart below, demonstrate that SingleStore achieves competitive performance on TPC-H compared to both leading-edge, specialized cloud data warehouses. A breakdown of results by query can be found in the appendix.

TPC-C

We compared SingleStore against CDB, a popular, state-of-the-art cloud operational database, on TPC-C, an industry-standard OLTP benchmark. All SingleStore results were on our columnar-based universal storage, and were competitive with CDB which is a rowstore-based operational database. We ran comparisons on public cloud instances with cluster sizes that were chosen to match in vCPU count.

Note that the cloud data warehouses CDW1 and CDW2 do not support running TPC-C. As an example, unique constraints are one feature that are not supported in the data warehouses but are supported in SingleStore (they are supported in all table types, including in our universal storage). CDW1 and CDW2 only support informational, unenforced unique constraints — their systems cannot enforce the integrity of the constraint, they only use it as an informational hint to the query planner.

We measured the throughput (tpmC), as defined by the TPC-C benchmark. We compared against results previously published by CDB. Note that TPC-C specifies a maximum possible tpmC of 12.86 per warehouse, and both SingleStore and CDB are essentially reaching this maximum at a data size of 1,000 warehouses.

We also tested SingleStore on TPC-C at a data size of 10,000 warehouses, and SingleStore delivers excellent scale-out performance — performance on larger TPC-C sizes scales linearly as we scale out the SingleStore cluster size. On the other hand, CDB offers limited scalability on OLTP workloads — previous results show its performance does not scale well to much larger numbers of warehouses.

TPC-C results (higher is better, up to the limit of 12.86 tpmC/warehouse)

Conclusion

Here’s the summary of both the TPC-C and TPC-H results again:

These results demonstrate that SingleStore achieves state-of-the-art performance competitive with leading operational databases, as well as analytical databases on benchmarks specific to each workload. SingleStore can meet workload requirements that previously required using multiple specialized database systems.

See the TPC-H section above for the details of what results we measured.

Details of runtimes by query (runtimes are in seconds):