introduction to database benchmarking with benchmark factory

1 Global Marketing

Bert Scalzo PhD, MBA & Oracle ACE Database Expert & Product Architect

Avoiding the Risks of Database Change using Benchmark Factory

https://access-new.dell.com/int/marketing/globalcreative/Brand Logo Avatars/logo-blue.png

https://access-new.dell.com/int/marketing/globalcreative/Brand Logo Avatars/logo-blue.png

2 Quest Software is now a part of Dell

Books by Bert Scalzo


• Database Benchmarking Basics

• Benchmark Factory Overview

• Benchmarking Options

• Customizing Benchmarking Projects

• Running a Benchmark

• Examining Runtime Reports

• Error Messages and Log Files

• Benchmark Factory Demonstration

• Questions & Answers

Purpose / Agenda


Database Benchmarking is complicated Even with a Benchmarking Tool

There is no substitute for

your knowledge and

expertise


Successful Database Benchmarking

Expert – must know database, O/S, hardware, software, tuning & have read the benchmark specs Stress test the database

?


Database Benchmarking Most Common Mistakes


Wrong Person Doing the Heavy Lifting


Trying to make do without the right tools


Wrong number of virtual users Don’t cut corners


Industry Standard Database Benchmarks

• Industry Standard Database Benchmarks have been around for over 20 years

• TPC.ORG – TPC-C: Older OLTP benchmark. Basic “order entry” type application.

– TPC-H: Data warehousing queries. 22 queries – not star schema design.

– TPC-E: Newer OLTP benchmark. Simulates workload of brokerage firms.

– TPC-DS: New, evolving complete DW. Spec still evolving – not yet in BMF.

– TPC-VMS: New, evolving virtual DB’s. Spec still evolving – not yet in BMF.

• Be Sure to Read the Specs – Understand basic database design

– Implementation options up to you

– Sizing options for data load and concurrency

– How to interpret results and/or gauge success

• Majority of failures due to insufficient preparation, unrealistic expectations, and uncertain metrics for success


Examples of Info in the Spec: TPC-C

BMF Prompts for DB size – you must know how to answer

100 concurrent users = scale factor of 10


Examples of Info in the Spec: TPC-H BMF does not know what Oracle options you are licensed for (partitioning), nor your hardware setup such as spindles per LUN, RAID level, CPU count, RAM size, and son on.

BMF creates basic & correct tables and indexes – you must manually optimize them within spec.


Database Design – Reference Full Disclosure Reports for Similar Setups


Database Design Reference Full Disclosure Reports for Similar Setups


Database Design – Appendix B Will Show DDL for Optimal Design

Very common technique to load very large data sets into ETL or staging tables, and then to do create table as select (CTAS) to populate the benchmark tables


Common Database Benchmark Metrics TPS and Avg. Response Time

• Transactions per Second

– Gets far too much attention

– Meaningless to most users

– Sort of like automobile RPM’s (how fast internal engine is working – not how fast car is moving or how soon we’ll arrive)

– Misconception that TPS equates to IOPS (IO Operations per Second) – ignores database memory caching and logging

– BMF does not report IOPS (on roadmap for possibly 7.0)

• Average Response Time

– Gets far too less attention

– Meaningful to most users

– Sort of like MPH (or KPH) (how fast car actually is moving – so infers how soon we’ll arrive or amount of fuel we’ll use)

– When examined in conjunction with TPS, then a generally observable and clear pattern often emerges (next slide) …

• Be sure to read my blog: The Myth of Transactions per Second (TPS)

http://www.toadworld.com/products/benchmark-factory/b/weblog/archive/2009/05/06/the-myth-of-transactions-per-second-tps.aspx


Common Database Benchmark Pattern – True Point of Saturation

0

1000

2000

3000

4000

5000

6000

7000

100 200 300 400 500 600 700 800 900 1000

TPS Avg Resp Time

Looking for inflection point where TPS is still increasing or just decreasing and close to max where average response time is below customer defined SLA

Notice the line characteristics between roughly 750 and 850 concurrent users – for current configuration and optimization benchmark results interesting

Common mistake to simply attempt maximize TPS – remember TPS is not IOPS


Benchmark Factory Architecture – Console (Control Center) and Agents

You must manually install BMF agent software on servers – otherwise BMF will simply spawn agents on your console machine (major bottleneck)

Each 32-bit BMF agent can handle 500 concurrent user sessions

Each 64-bit BMF agent can handle 1,000 concurrent user sessions

Each server running one or more agents requires its own separate BMF license!

Agent servers can be virtual machines


Hardware Sizing is Very Difficult Requires Lots of Thought

Let’s assume doing TPC-C benchmark for sale factor 100, so 1,000 concurrent users, and roughly 8-10 BMF agents

Note we already have a huge built-in bottleneck – there’s only one NIC for 1,000 concurrent database sessions!

Let’s assume using virtual machines for entire BMF architecture on a seemingly large VMware ESX / vSphere server


Benchmark Factory Architecture BMF Offers Two Repository Options

DEFAULT


BMF Offers Many Types of Tests Industry Standard Benchmarks


BMF Offers Many Types of Tests Scalability & Workload Capture/Replay


BMF Offers Many Types of Tests Custom Load Scenario


BMF Offers Many Types of Tests Goal Testing to Meet Customer SLA

BMF will test from 100 to 1,000 concurrent users in increments of 100 additional users, and will stop when the average response time is more than 2 seconds (e.g. common customer defined SLA).


Example of Benchmark Factory Project – Default Creation


Example of Benchmark Factory Project Custom (Highly Recommended)


Example of Benchmark Factory Running


Benchmark Factory Runtime Reports – Results from Runs


Benchmark Factory Error Logging – Where to Look


On-Demand Webcast - Intro to Database Benchmarking and product demo

• Download the webcast now

http://www.quest.com/webcast-ondemand/introduction-to-database-benchmarking-using-benchmark-factory825458.aspx


• www.toadworld.com – Expert Blogs

– Articles

– How-To Videos

– User Forum

– Wiki

– Direct access to product experts and much more

Additional Resources

http://www.toadworld.com/


Additional Resources

• Benchmark Factory Home Page

• Download a free 30 day trial edition of Benchmark Factory

• White Paper – Top 10 Benchmarking Misconceptions

• On-Demand Webcast - Introduction to Database Benchmarking

http://software.dell.com/products/benchmark-factory/



https://software.dell.com/register/54678/

https://software.dell.com/whitepaper/top-10-benchmarking-misconceptions820272/











introduction to database benchmarking with benchmark factory

Technology