Database Benchmarks — Transaction Processing Performance
It is interesting to discover that the TPC in TPC.ORG — the industry benchmark for database (and whole lot more) benchmark stands for Transaction Processing Performance Council. So instead of TPPC it is just TPC.
Here is the list of benchmarks as of Jan 22 along with what they stand for on one page. Please note that text is heavily borrowed from TPC.ORG, and minor modification (ahem.. mistakes) are mine :)
TPC-C is an On-Line Transaction Processing Benchmark
TPC Benchmark C is an on-line transaction processing (OLTP) benchmark. TPC-C involves a mix of five concurrent transactions of different types and complexity either executed on-line or queued for deferred execution. The database is comprised of nine types of tables with a wide range of record and population sizes. TPC-C is measured in transactions per minute (tpmC). While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any particular business segment, but, rather represents any industry that must manage, sell, or distribute a product or service.
TPC-E is an On-Line Transaction Processing Benchmark
TPC Benchmark E is an on-line transaction processing (OLTP) benchmark. TPC-E is more complex than previous OLTP benchmarks such as TPC-C because of its diverse transaction types, more complex database and overall execution structure. TPC-E involves a mix of twelve concurrent transactions of different types and complexity, either executed on-line or triggered by price or time criteria. The database is comprised of thirty-three tables with a wide range of columns, cardinality, and scaling properties. TPC-E is measured in transactions per second (tpsE). While the benchmark portrays the activity of a stock brokerage firm, TPC-E is not limited to the activity of any particular business segment, but rather represents any industry that must report upon and execute transactions of a financial nature.
Note — TPC-E is more complex and at per second.
TPC-H is a Decision Support Benchmark
The TPC-H is a decision support benchmark. This benchmark illustrates decision support systems that examine large volumes of data, execute queries with a high degree of complexity, and give answers to critical business questions. The performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@Size). The TPC-H Price/Performance metric is expressed as Price/QphH@Size for Version 2 and Price/kQphH@Size for Version 3.
TPC-DS is a Decision Support Benchmark
TPC-DS is a decision support benchmark that models several generally applicable aspects of a decision support system, including queries and data maintenance. A benchmark result measures query response time in single user mode, query throughput in multi user mode and data maintenance performance for a given hardware, operating system, and data processing system configuration under a controlled, complex, multi-user decision support workload. The TPC-DS Price/Performance metric is expressed as Price/QphDS@Size for Version 2 and Price/kQphDS@Size for Version 3.
TPC-DI is a benchmark for Data Integration
The TPC-DI benchmark combines and transforms data extracted from an On-Line Transaction Processing (OTLP) system along with other sources of data, and loads it into a data warehouse. The source and destination data models, data transformations and implementation rules have been designed to be broadly representative of modern data integration requirements.
TPCx-V is a Virtualization Benchmark for Database Workloads
The TPC Express Benchmark V (TPCx-V) benchmark measures the performance of a virtualized server platform under a demanding database workload. It loads the databases, runs the benchmark, validates the results, and even performs many of the routine audit steps.
TPCx-HCI is a Benchmark for Hyper-Converged Infrastructure
The TPC Express Benchmark HCI (TPCx-HCI) benchmark measures the performance of Hyper-Converged Infrastructure clusters under a demanding database workload.
TPC-H is a Decision Support Benchmark
The TPC-H is a decision support benchmark. It consists of a suite of business oriented ad-hoc queries and concurrent data modifications. The queries and the data populating the database have been chosen to have broad industry-wide relevance. This benchmark illustrates decision support systems that examine large volumes of data, execute queries with a high degree of complexity, and give answers to critical business questions. The performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@Size), and reflects multiple aspects of the capability of the system to process queries. These aspects include the selected database size against which the queries are executed, the query processing power when queries are submitted by a single stream, and the query throughput when queries are submitted by multiple concurrent users. The TPC-H Price/Performance metric is expressed as Price/QphH@Size for Version 2 and Price/kQphH@Size for Version 3.
TPCx-BB is a Big Data Benchmark
TPCx-BB Express Benchmark BB (TPCx-BB) measures the performance of Hadoop-based Big Data systems systems. It measures the performance of both hardware and software components by executing 30 frequently performed analytical queries in the context of retailers with physical and online store presence. The queries are expressed in SQL for structured data and in machine learning algorithms for semi-structured and unstructured data. The SQL queries can use Hive or Spark, while the machine learning algorithms use machine learning libraries, user defined functions, and procedural programs.
TPCx-AI is an end to end AI benchmark standard developed by the TPC.
The benchmark measures the performance of an end-to-end machine learning or data science platform. The benchmark development has focused on emulating the behavior of representative industry AI solutions that are relevant in current production datacenters and cloud environments.
TPCx-IoT is a Benchmark for IoT Gateway Systems
TPCx-IoT, the industry’s first benchmark which enables direct comparison of different software and hardware solutions for IoT gateways. TPCx-IoT was specifically designed to provide verifiable performance, price-performance and availability metrics for commercially available systems that typically ingest massive amounts of data from large numbers of devices, while running real-time analytic queries. The workload is representative of activities typical in IoT gateway systems, running on commercially available hardware and software platforms. The TPCx-IoT can be used to assess a broad range of system topologies and implementation methodologies in a technically rigorous and directly comparable, in a vendor-neutral manner. The TPCx-IoT Price/Performance metric is expressed as Price/IoTps for Version 1 and Price/kIoTps for Version 2.
I am sorry if some of the details that you need are omitted or some benchmarks are skipped. You can always visit: http://tpc.org/default5.asp to get the finer details. However I feel captured points above are important.
Reference/Credit: http://tpc.org/default5.asp Much indebted to TPC for providing the details of these benchmarks.
