TPC BenchmarksTPC Benchmarks

Charles LevineMicrosoft

clevine@microsoft.comModified by Jim GrayGray @ Microsoft.com

March 1997

OutlineOutline

Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures

Benchmarks: What and WhyBenchmarks: What and Why

What is a benchmark? Domain specific

No single metric possible The more general the benchmark, the less useful it is for

anything in particular. A benchmark is a distillation of the essential attributes of a

workload

Benchmarks: What and WhyBenchmarks: What and Why

Desirable attributes Relevant meaningful within the target domain

Understandable Good metric(s) linear, orthogonal, monotonic

Scaleable applicable to a broad spectrum of hardware/architecture

Coverage does not oversimplify the typical environment

Acceptance Vendors and Users embrace it

Portable Not limited to one hardware/software vendor/technology

Benefits and LiabilitiesBenefits and Liabilities

Good benchmarks Define the playing field Accelerate progress

Engineers do a great job once objective is measurable and repeatable

Set the performance agenda Measure release-to-release progress Set goals (e.g., 10,000 tpmC, < 100 $/tpmC) Something managers can understand (!)

Benchmark abuse Benchmarketing Benchmark wars

more $ on ads than development

Benchmarks have a LifetimeBenchmarks have a Lifetime

Good benchmarks drive industry and technology forward. At some point, all reasonable advances have been made. Benchmarks can become counter productive by encouraging

artificial optimizations. So, even good benchmarks become obsolete over time.

OutlineOutline

Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures

What is the TPC?What is the TPC?

TPC = Transaction Processing Performance Council Founded in Aug/88 by Omri Serlin and 8 vendors. Membership of 40-45 for last several years

Everybody who’s anybody in software & hardware

De facto industry standards body for OLTP performance

Administered by:Shanley Public Relations ph: (408) 295-8894777 N. First St., Suite 600 fax: (408) 295-9768San Jose, CA 95112-6311 email: td@tpc.org

Most TPC specs, info, results on web page: www.tpc.org

TPC database (unofficial): www.microsoft.com/sql/tpc/

News: Omri Serlin’s FT Systems News (monthly magazine)

Two Seminal Events Leading to TPCTwo Seminal Events Leading to TPC

Anon, et al, “A Measure of Transaction Processing Power”, Datamation, April fools day, 1985. Anon, Et Al = Jim Gray (Dr. E. A. Anon) and 24 of his closest

friends Sort: 1M 100 byte records Mini-batch: copy 1000 records DebitCredit: simple ATM style transaction

Tandem TopGun Benchmark DebitCredit 212 tps on NonStop SQL in 1987 (!) Audited by Tom Sawyer of Codd and Date (A first) Full Disclosure of all aspects of tests (A first) Started the ET1/TP1 Benchmark wars of ’87-’89

1987: 256 tps Benchmark

14 M$ computer (Tandem) A dozen people False floor, 2 rooms of machines

Simulate 25,600 clients

A 32 node processor array

A 40 GB disk array (80 drives)

OS expert

Network expert

DB expert

Performance expert

Hardware experts

Admin expert

Auditor

Manager

1988: DB2 + CICS Mainframe 65 tps

IBM 4391 Simulated network of 800 clients 2m$ computer Staff of 6 to do benchmark

2 x 3725 network controllers

16 GB disk farm4 x 8 x .5GB

Refrigerator-sizedCPU

1997: 10 years later1 Person and 1 box = 1250 tps

1 Breadbox ~ 5x 1987 machine room 23 GB is hand-held One person does all the work Cost/tps is 1,000x less

25 micro dollars per transaction

4x200 Mhz cpu1/2 GB DRAM12 x 4GB disk

Hardware expertOS expertNet expertDB expertApp expert

3 x7 x 4GB disk arrays

What Happened?

Moore’s law: Things get 4x better every 3 years (applies to computers, storage, and networks)

New Economics: Commodityclass price/mips software $/mips k$/yearmainframe 10,000 100 minicomputer 100 10microcomputer 10 1

GUI: Human - computer tradeoffoptimize for people, not computers

mainframeminimicro

time

pric

e

TPC MilestonesTPC Milestones

1989: TPC-A ~ industry standard for Debit Credit 1990: TPC-B ~ database only version of TPC-A 1992: TPC-C ~ more representative, balanced OLTP 1994: TPC requires all results must be audited 1995: TPC-D ~ complex decision support (query) 1995: TPC-A/B declared obsolete by TPC Non-starters:

TPC-E ~ “Enterprise” for the mainframers TPC-S ~ “Server” component of TPC-C Both failed during final approval in 1996

TPC vs. SPECTPC vs. SPEC

SPEC (System Performance Evaluation Cooperative) SPECMarks

SPEC ships code Unix centric CPU centric

TPC ships specifications Ecumenical Database/System/TP centric Price/Performance

The TPC and SPEC happily coexist There is plenty of room for both

OutlineOutline

Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures

TPC-A OverviewTPC-A Overview

Transaction is simple bank account debit/credit Database scales with throughput Transaction submitted from terminal

Read 100 bytes including Aid, Tid, Bid, Delta from terminal (see Clause 1.3)BEGIN TRANSACTION

Update Account where Account_ID = Aid:Read Account_Balance from AccountSet Account_Balance = Account_Balance + Delta

Write Account_Balance to AccountWrite to History:

Aid, Tid, Bid, Delta, Time_stampUpdate Teller where Teller_ID = Tid:

Set Teller_Balance = Teller_Balance + DeltaWrite Teller_Balance to Teller

Update Branch where Branch_ID = Bid:Set Branch_Balance = Branch_Balance + DeltaWrite Branch_Balance to Branch

COMMIT TRANSACTIONWrite 200 bytes including Aid, Tid, Bid, Delta, Account_Balance to terminal

TPC-A TransactionTPC-A Transaction

TPC-A Database SchemaTPC-A Database Schema

LegendLegend

Table NameTable Name<cardinality><cardinality>

one-to-manyone-to-manyrelationshiprelationship

BranchBranchBB

AccountAccountB*100KB*100K

100K100K

HistoryHistoryB*2.6MB*2.6M

TellerTellerB*10B*101010

10 Terminals per Branch row10 Terminals per Branch row10 second cycle time per terminal10 second cycle time per terminal1 transaction/second per Branch row1 transaction/second per Branch row

TPC-A TransactionTPC-A Transaction

Workload is vertically aligned with Branch Makes scaling easy But not very realistic

15% of accounts non-local Produces cross database activity

What’s good about TPC-A? Easy to understand Easy to measured Stresses high transaction rate, lots of physical IO

What’s bad about TPC-A? Too simplistic! Lends itself to unrealistic optimizations

TPC-A Design RationaleTPC-A Design Rationale

Branch & Teller in cache, hotspot on branch

Account too big to cache requires disk access

History sequential insert hotspot at end 90-day capacity ensures reasonable ratio of disk to cpu

RTE SUTRTE SUT

RTE - Remote Terminal Emulator Emulates real user behavior

Submits txns to SUT, measures RT Transaction rate includes think time Many, many users (10 x tpsA)

SUT - System Under Test All components except for terminal

Model of system:

T

T

T - C Network*

CL

IE

NT

C - S Network*

SUTRTE

Response Time Measured Here

Host System(s)

S - S Network*

SER

VER

TPC-A MetricTPC-A Metric

tpsA = transactions per second, average rate over 15+ minute interval, at which 90% of txns get <= 2 second RT

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Response time (seconds)

Nu

mb

er

of

Tra

ns

acti

on

s

Average Response Time

90th Percentile Response Time

TPC-A PriceTPC-A Price

Price 5 year Cost of Ownership:

hardware, software, maintenance

Does not include development, comm lines, operators, power, cooling, etc.

Strict pricing model one of TPC’s big contributions List prices System must be orderable & commercially available Committed ship date

Differences between TPC-A and TPC-BDifferences between TPC-A and TPC-B

TPC-B is database only portion of TPC-A No terminals No think times

TPC-B reduces history capacity to 30 days Less disk in priced configuration

TPC-B was easier to configure and run, BUT Even though TPC-B was more popular with vendors,

it did not have much credibility with customers.

TPC LoopholesTPC Loopholes

Pricing Package pricing Price does not include cost of five star wizards needed to get optimal

performance, so performance is not what a customer could get.

Client/Server Offload presentation services to cheap clients, but report performance

of server

Benchmark specials Discrete transactions Custom transaction monitors Hand coded presentation services

TPC-A/B LegacyTPC-A/B Legacy

First results in 1990: 38.2 tpsA, 29.2K$/tpsA (HP) Last results in 1994: 3700 tpsA, 4.8 K$/tpsA (DEC) WOW! 100x on performance & 6x on price in 5 years !! TPC cut its teeth on TPC-A/B; became functioning, representative

body Learned a lot of lessons:

If benchmark is not meaningful, it doesn’t matter how many numbers or how easy to run (TPC-B).

How to resolve ambiguities in spec How to police compliance Rules of engagement

TPC-A Established OLTP Playing FieldTPC-A Established OLTP Playing Field

TPC-A criticized for being irrelevant, unrepresentative, misleading

But, truth is that TPC-A drove performance, drove price/performance, and forced everyone to clean up their products to be competitive.

Trend forced industry toward one price/performance, regardless of size.

Became means to achieve legitimacy in OLTP for some.

OutlineOutline

Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures

TPC-C OverviewTPC-C Overview

Moderately complex OLTP The result of 2+ years of development by the TPC Application models a wholesale supplier managing orders. Order-entry provides a conceptual model for the benchmark;

underlying components are typical of any OLTP system. Workload consists of five transaction types. Users and database scale linearly with throughput. Spec defines full-screen end-user interface. Metrics are new-order txn rate (tpmC) and price/performance

($/tpmC) Specification was approved July 23, 1992.

TPC-C’s Five TransactionsTPC-C’s Five Transactions

OLTP transactions: New-order: enter a new order from a customer Payment: update customer balance to reflect a payment Delivery: deliver orders (done as a batch transaction) Order-status: retrieve status of customer’s most recent order Stock-level: monitor warehouse inventory

Transactions operate against a database of nine tables. Transactions do update, insert, delete, and abort;

primary and secondary key access. Response time requirement:

90% of each type of transaction must have a response time 5 seconds, except (queued mini-batch) stock-level which is 20 seconds.

TPC-C Database SchemaTPC-C Database Schema

WarehouseWarehouseWW

LegendLegend

Table NameTable Name<cardinality><cardinality>

one-to-manyone-to-manyrelationshiprelationship

secondary indexsecondary index

DistrictDistrictW*10W*10

1010

CustomerCustomerW*30KW*30K

3K3K

HistoryHistoryW*30K+W*30K+

1+1+

ItemItem100K (fixed)100K (fixed)

StockStockW*100KW*100K100K100K WW

OrderOrderW*30K+W*30K+1+1+

Order-LineOrder-LineW*300K+W*300K+

10-1510-15

New-OrderNew-OrderW*5KW*5K0-10-1

22

TPC-C WorkflowTPC-C Workflow

11

Select txn from menu:Select txn from menu:1. New-Order 1. New-Order 45%45%2. Payment 2. Payment 43%43%3. Order-Status3. Order-Status 4%4%4. Delivery 4. Delivery 4%4%5. Stock-Level 5. Stock-Level 4%4%

Input screenInput screen

Output screenOutput screen

Measure menu Response TimeMeasure menu Response Time

Measure txn Response TimeMeasure txn Response Time

Keying time

Think time

33

Go back to 1Go back to 1

Cycle Time DecompositionCycle Time Decomposition(typical values, in seconds,(typical values, in seconds, for weighted average txn)for weighted average txn)

Menu = 0.3Menu = 0.3

Keying = 9.6Keying = 9.6

Txn RT = 2.1Txn RT = 2.1

Think = 11.4Think = 11.4

Average cycle time = 23.4Average cycle time = 23.4

Data SkewData Skew

NURand - Non Uniform Random NURand(A,x,y) =

(((random(0,A) | random(x,y)) + C) % (y-x+1)) + x Customer Last Name: NURand(255, 0, 999) Customer ID: NURand(1023, 1, 3000) Item ID: NURand(8191, 1, 100000)

bitwise OR of two random values skews distribution toward values with more bits on

75% chance that a given bit is one (1 - ½ * ½) data skew repeats with period “A” (first param of NURand())

NURand DistributionNURand Distribution

TPC-C NURand function: frequency vs 0...255

Record Identitiy [0..255]

Rel

ativ

e F

requ

ency

of

Acc

ess

to T

his

Rec

ord

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

0

10

20

30

40

50

60

70

80

90

10

0

110

12

0

13

0

14

0

15

0

16

0

17

0

18

0

19

0

20

0

21

0

22

0

23

0

24

0

25

0

cumulativedistribution

ACID TestsACID Tests

TPC-C requires transactions be ACID. Tests included to demonstrate ACID properties met. Atomicity

Verify that all changes within a transaction commit or abort.

Consistency Isolation

ANSI Repeatable reads for all but Stock-Level transactions. Committed reads for Stock-Level.

Durability Must demonstrate recovery from

Loss of power Loss of memory Loss of media (e.g., disk crash)

TransparencyTransparency

TPC-C requires that all data partitioning be fully transparent to the application code. (See TPC-C Clause 1.6) Both horizontal and vertical partitioning is allowed All partitioning must be hidden from the application

Most DB do single-node horizontal partitioning. Much harder: multiple-node transparency.

For example, in a two-node cluster:

Any DML operation must beAny DML operation must beable to operate against the able to operate against the entire database, regardless of entire database, regardless of physical location.physical location.

Warehouses:Warehouses:1-1001-100 101-200101-200

Node ANode Aselect * select * from warehousefrom warehousewhere W_ID = 150where W_ID = 150

Node BNode Bselect * select * from warehousefrom warehousewhere W_ID = 77where W_ID = 77

Transparency (cont.)Transparency (cont.)

How does transparency affect TPC-C? Payment txn: 15% of Customer table records are non-local

to the home warehouse. New-order txn: 1% of Stock table records are non-local to

the home warehouse.

In a cluster, cross warehouse traffic cross node traffic 2 phase commit, distributed lock management, or both.

For example, with distributed txns:Number of nodesNumber of nodes % Network Txns% Network Txns

11 0022 5.55.533 7.37.3nn10.910.9

TPC-C Rules of ThumbTPC-C Rules of Thumb

1.2 tpmC per User/terminal (maximum) 10 terminals per warehouse (fixed) 65-70 MB/tpmC priced disk capacity (minimum) ~ 0.5 physical IOs/sec/tpmC (typical) 300-700 KB main memory/tpmC So use rules of thumb to size 10,000 tpmC system:

How many terminals? How many warehouses? How much memory? How much disk capacity? How many spindles?

Response TimeResponse Timemeasured heremeasured here

Typical TPC-C Configuration (Conceptual)Typical TPC-C Configuration (Conceptual)

DatabaseDatabaseServerServer

......

ClientClient

C/SLAN

Term.LAN

Presentation ServicesPresentation Services Database FunctionsDatabase FunctionsEmulated User LoadEmulated User Load

Har

dwar

eH

ardw

are

RTERTE, e.g.:, e.g.:EmpowerEmpowerpreVuepreVueLoadRunnerLoadRunnerS

oftw

are

Sof

twar

e TPC-C application +TPC-C application +Txn Monitor and/orTxn Monitor and/ordatabase RPC librarydatabase RPC librarye.g., Tuxedo, ODBCe.g., Tuxedo, ODBC

TPC-C application TPC-C application (stored procedures) + (stored procedures) + Database engine +Database engine +Txn MonitorTxn Monitore.g., SQL Server, Tuxedoe.g., SQL Server, Tuxedo

Driver SystemDriver System

Competitive TPC-C Configuration TodayCompetitive TPC-C Configuration Today

7,128 tpmC; $89/tpmC; 5-yr COO= 569 K$ 2 GB memory, 85x9-GB disks (733 GB total) 6500 users

Demo of SQL Server + Web interfaceDemo of SQL Server + Web interface

User interface implemented w/ Web browser via HTML

Client to Server via ODBC SQL Server database engine All in one nifty little box!

TPC-C Current ResultsTPC-C Current Results

Best Performance is 30,390 tpmC @ $305/tpmC (Oracle/DEC)

Best Price/Perf. is 6,712 tpmC @ $65/tpmC (MS SQL/DEC/Intel) graphs show

high price of UNIX diseconomy of UNIX scaleup

tpmC vs $/tpmC(ignores Oracle 30ktpmC on 4x12 Alpha)

0

50

100

150

200

250

300

350

400

0 5000 10000 15000 20000

tpmC

$/tp

mC

DB2

Informix

MS SQL Server

Oracle

Sybase

tpmC & Price Performance(only "best" data shown for each vendor)

0

50

100

150

200

250

300

350

400

0 5000 10000 15000 20000

tpmC

$/t

pm

C

DB2

Informix

MS SQL Server

Oracle

Sybase

Compare SMP PerformanceCompare SMP Performance

tpmC vs CPS

0

5,000

10,000

15,000

20,000

0 5 10 15 20

CPUs

tpm

C

SUN Scaleability

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

20,000

0 5 10 15 20

cpus

tpm

C

SUN Scaleability

SQL Server

TPC-C SummaryTPC-C Summary

Balanced, representative OLTP mix Five transaction types Database intensive; substantial IO and cache load Scaleable workload Complex data: data attributes, size, skew

Requires Transparency and ACID Full screen presentation services De facto standard for OLTP performance

OutlineOutline

Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures

TPC-D OverviewTPC-D Overview

Complex Decision Support workload The result of 5 years of development by the TPC Benchmark models ad hoc queries

extract database with concurrent updates multi-user environment

Workload consists of 17 queries and 2 update streams SQL as written in spec

Database load time must be reported Database is quantized into fixed sizes Metrics are Power (QppD), Throughput (QthD), and Price/Performance

($/QphD) Specification was approved April 5, 1995.

TPC-D SchemaTPC-D Schema

CustomerCustomerSF*150KSF*150K

LineItemLineItemSF*6000KSF*6000K

OrderOrderSF*1500KSF*1500K

SupplierSupplierSF*10KSF*10K

NationNation2525

RegionRegion55

PartSuppPartSuppSF*800KSF*800K

PartPartSF*200KSF*200K

Time2557

Legend:Legend:• • Arrows point in the direction of one-to-many relationships.Arrows point in the direction of one-to-many relationships.• • The value below each table name is its cardinality. SF is the Scale The value below each table name is its cardinality. SF is the Scale Factor.Factor.• • The Time table is optional. So far, not used by anyone.The Time table is optional. So far, not used by anyone.

TPC-D Database Scaling and LoadTPC-D Database Scaling and Load

Database size is determined from fixed Scale Factors (SF): 1, 10, 30, 100, 300, 1000, 3000 (note that 3 is missing, not a typo) These correspond to the nominal database size in GB.

(I.e., SF 10 is approx. 10 GB, not including indexes and temp tables.) Indices and temporary tables can significantly increase the total disk capacity.

(3-5x is typical)

Database is generated by DBGEN DBGEN is a C program which is part of the TPC-D spec. Use of DBGEN is strongly recommended. TPC-D database contents must be exact.

Database Load time must be reported Includes time to create indexes and update statistics. Not included in primary metrics.

TPC-D Query SetTPC-D Query Set

17 queries written in SQL92 to implement business questions. Queries are pseudo ad hoc:

Substitution parameters are replaced with constants by QGEN QGEN replaces substitution parameters with random values No host variables No static SQL

Queries cannot be modified -- “SQL as written” There are some minor exceptions. All variants must be approved in advance by the TPC

TPC-D Update StreamsTPC-D Update Streams

Update 0.1% of data per query stream About as long as a medium sized TPC-D query

Implementation of updates is left to sponsor, except: ACID properties must be maintained Update Function 1 (UF1)

Insert new rows into ORDER and LINEITEM tables equal to 0.1% of table size

Update Function 2 (UF2) Delete rows from ORDER and LINEITEM tables

equal to 0.1% of table size

TPC-D ExecutionTPC-D Execution

Power Test Queries submitted in a single stream (i.e., no concurrency) Sequence:

Throughput Test Multiple concurrent query streams Single update stream Sequence:

CacheCache FlushFlush

QueryQuerySet 0Set 0(optional)(optional)

UF1UF1QueryQuerySet 0Set 0 UF2UF2

Timed SequenceTimed SequenceWarm-up, untimedWarm-up, untimed

Query Set 1Query Set 1Query Set 2Query Set 2

Query Set NQuery Set N

UF1 UF2 UF1 UF2 UF1 UF2UF1 UF2 UF1 UF2 UF1 UF2Updates:Updates:

.... ..

TPC-D MetricsTPC-D Metrics

Power Metric (QppD) Geometric queries per hour times SF

Throughput (QthD) Linear queries per hour times SF

QppD Size SF

QI i UI jj

j

i

i@

( , ) ( , )

3600

0 0191

2

1

17

where

QI(i,0) Timing Interval for Query i, stream 0

UI(j,0) Timing Interval for Update j, stream 0

SF Scale Factor

QthD SizeS

SFTS@

17

3600

where:

S number of query streams

T elapsed time of test (in seconds)S

TPC-D Metrics (cont.)TPC-D Metrics (cont.)

Composite Query-Per-Hour Rating (QphD) The Power and Throughput metrics are combined to get

the composite queries per hour.

Reported metrics are: Power: QppD@Size Throughput: QthD@Size Price/Performance: $/QphD@Size

Comparability: Results within a size category (SF) are comparable. Comparisons among different size databases are strongly discouraged.

QphD Size QppD Size QthD Size@ @ @

TPC-D Current ResultsTPC-D Current Results

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

0 200 400 600 800 1000 1200 1400 1600 1800 2000

Performance

Pri

ce/P

erfo

rman

ce

IBM/DB2

NCR/Teradata

Pyramid/Oracle

Sun/Oracle

Sun/Oracle

TPC-D 300 GB Results

TPC-D 100 GB Results

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

0 200 400 600 800 1000 1200 1400 1600

HP/Oracle

NCR/Teradata

Sun/Oracle

Tandem/NonStop SQL

Digital/Oracle (9/96)

Digital/Oracle (11/96)

IBM/DB2

Example TPC-D ResultsExample TPC-D Results

Want to learn more about TPC-D?Want to learn more about TPC-D?

TPC-D Training Video Six hour video by the folks who wrote the spec. Explains, in detail, all major aspects of the benchmark.

Available from the TPC:Shanley Public Relations ph: (408) 295-8894777 N. First St., Suite 600 fax: (408) 295-9768San Jose, CA 95112-6311 email: td@tpc.org

OutlineOutline

Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures

TPC Future DirectionTPC Future Direction

TPC-Web The TPC is just starting a Web benchmark effort. TPC’s focus will be on database and transaction

characteristics. The interesting components are:

Web Server

WebServer

Appl.

FileSystem

DBMS Server

SQLEngine Stored

Procs

Database

TCP/IP

Browser Browser

...

Rules of ThumbRules of Thumb

Answer Set for TPC-C rules of Thumb (slide 38)a 10 ktpmC system

» » 8340 terminals ( = 5000 / 1.2)8340 terminals ( = 5000 / 1.2)

» 834 warehouses ( = 8340 / 10)» 834 warehouses ( = 8340 / 10)

» 3GB-7GB DRAM of memory (10,000 * [3KB..7KB])» 3GB-7GB DRAM of memory (10,000 * [3KB..7KB])

» 650 GB disk space = 10,000 * 65» 650 GB disk space = 10,000 * 65

» # Spindles depends on MB capacity vs. physical IO. » # Spindles depends on MB capacity vs. physical IO. Capacity: 650 / 4GB = 162 spindlesCapacity: 650 / 4GB = 162 spindlesIO: 10,000*.5 / 140 = 31 IO/sec IO: 10,000*.5 / 140 = 31 IO/sec (OK!)(OK!)

but 9GB or 23GB disks would be but 9GB or 23GB disks would be TOO HOT!TOO HOT!

Reference MaterialReference Material

Jim Gray, The Benchmark Handbook for Database and Transaction Processing Systems, Morgan Kaufmann, San Mateo, CA, 1991.

Raj Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, John Wiley & Sons, New York, 1991.

William Highleyman, Performance Analysis of Transaction Processing Systems, Prentice Hall, Englewood Cliffs, NJ, 1988.

TPC Web site: www.tpc.org Microsoft db site: www.microsoft.com/sql/tpc/ IDEAS web site: www.ideasinternational.com