1

Self-tuning DB Technology & Info Services:from Wishful Thinking to Viable Engineering

Gerhard Weikumweikum@cs.uni-sb.de

Acknowledgements to collaborators:Surajit Chaudhuri, Christoff Hasse, Arnd Christian König, Achim

Kraiss, Axel Mönkeberg, Peter Muth, Guido Nerjes, Elizabeth O‘Neil,

Patrick O‘Neil, Peter Scheuermann, Markus Sinnwell, Peter Zabback

Teamwork is essential.It allows you to blame someone else.

2

OutlineAuto-Tuning: What and Why?

Where Do We Go From Here?- Dreams and Directions -

The Feedback-Control Approach

Example 1: Load Control

Where Do We Stand Today? - Myths and Facts -

The COMFORT Experience

Example 2: Workflow System Configuration

3

Auto-Tuning: What and Why?DBA manual (10 years ago):

• tuning experts are expensive• system cost dominated and growth limited by human care & feed automate sys admin and tuning!

Härder 1981: mission impossible

4

Intriguing and Treacherous ApproachesInstant tuning: rules of thumb

KIWI principle: kill it with iron

DBA joystick method: feedback control loop

Columbus / Sisyphus approach: trial and error

+ ok for page size, striping unit, min cache size– insufficient for max cache size, MPL limit, etc.

+ ok if applied with care– waste of money otherwise

+ ok with simulation tools– risky with production system

An engineer is someone who can do for a dimewhat any fool can do for a dollar.

+ ok when it converges under stationary workload– susceptible to instability

5

OutlineAuto-tuning: What and Why?

Where Do We Go From Here?- Dreams and Directions -

The Feedback-Control Approach

Example 1: Load Control

Where Do We Stand Today? - Myths and Facts -

The COMFORT Experience

Example 2: Workflow System Configuration

6

Feedback Control Loop for Automatic Tuning

• Observe

• Predict

• React

Need aquantitativemodel !

7

Performance Predictability is Key”Our ability to analyze and predict the performance of the enormously complex software systems ...are painfully inadequate”

(Report of the US President’s Technology Advisory Committee 1998)

ability to predict workload knobs performance !!! !!! ???

is prerequisite for finding the right knob settings workload knobs performance goal !!! ??? !!!

8

Level, Scope, and Time Horizonof Tuning Issues

(workflow) system configuration

time

scopelevel

index selection

query opt.& db stats mgt.

caching

load control data placement

9

Level, Scope, and Time Horizonof Tuning Issues

(workflow) system configuration

time

scopelevel

index selection

query opt.& db stats mgt.

caching

load control data placement

uncontrolled memory or lock contentioncan lead to performance catastrophe

Load Control for Locking (MPL Tuning)

Locking for Concurrent Transactions

Locking and Lock Contention

t

lock conflict

lockwait

lockthrashing

How Difficult Can This Be?

typical Sisyphus problem

arrivingtransactions

DBS

activetrans,

trans.queue

10 20 30 40 50

0.2

0.4

0.6

0.8

1.0

MPL

response time [s]

Adaptive Load Control

transaction admission

transaction cancellation

transaction execution

abortedtrans.

committed trans.

arriving trans.

conflict ratio

conflict ratio =

.transrunningbyheldlocks#

.transallbyheldlocks#

criticalconflict ratio 1.3

restartedtrans.

backed upby math (Tay,Thomasian)

15

WFMS Architecture for E-Services

Ms3.lnk

...

WF server type 1

App server type 1

Clients

Comm server

WF server type 2

App server type n

...

16

Workflow System Configuration ToolWorkflowRepository Operational Workflow System Config.

Admin

Modeling Calibration

Evaluation

Recommendation

MonitoringMapping

Hypotheticalconfig

Max. ThroughputAvg. waiting timeExpected downtime

17

Workflow System Configuration ToolWorkflowRepository Operational Workflow System Config.

Admin

Modeling Calibration

Evaluation

Recommendation

MonitoringMapping

Min-costre-config.

Goals:min(throughput)max(waiting time)max(downtime)+ constraints

Long-term feedback control• aims at global, user- perceived metrics and• uses more advanced math for prediction

18

OutlineThe Problem – 10 Years Ago and Now

Where Do We Go From Here?- Dreams and Directions -

The Feedback-Control Approach

Example 1: Load Control

Where Do We Stand Today? - Myths and Facts -

The COMFORT Experience

Example 2: Workflow System Configuration

19

Where Do We Stand Today? - Good News

Advances in Engineering:• Eliminate second-order knobs• Robust rules of thumb for some knobs• KIWI method where applicable

Scientific Progress:+ Feedback control approach+ Storage systems have become self-managing+ Index selection wizards hard to beat+ Materialized view wizards + Synopses selection and space allocation for DB statistics well understood

Index Selection: Live or Let DieOrdersOrdersONo ODate ...

PartsPNo PName ...

LineItemsONo LNo Qty Price ODate ShipDate ...

CustomersCNo CName City State Discount ...

Data Workload

Select ... From ... Where City = ... And State = ...

Select ... From ... Where ShipDate – ODate > ...

Select ... From ... Where L.Ono = O.ONo And Price < ...

Update ... Set ShipDate = ...

Insert LineItems ...

100 x / min

20 x / min

15 x / min

300 x / min

300 x / min

NP-hard combinatorial problemwith complex cost formulas

Create Index ...ONo

ONo Price

ShipDate

CityState

21

Where Do We Stand Today? - Good News

Advances in Engineering:• Eliminate second-order knobs• Robust rules of thumb for some knobs• KIWI method where applicable

Scientific Progress:+ Feedback control approach+ Storage systems have become self-managing+ Index selection wizards hard to beat+ Materialized view wizards + Synopses selection and space allocation for DB statistics well understood

22

Where Do We Stand Today? - Bad News

, wrong Complex problems have simple, easy-to-understand answers

– Automatic system tuning based on few principles:

– Interactions across components and interference among different workload classes can make entire system unpredictable

23

OutlineThe Problem – 10 Years Ago and Now

Where Do We Go From Here?- Dreams and Directions -

The Feedback-Control Approach

Example 1: Load Control

Where Do We Stand Today? - Myths and Facts -

The COMFORT Experience

Example 2: Workflow System Configuration

24

Autonomic Computing: Path to Nirvana ?

My interpretation: need component design for predictability: self-inspection, self-analysis, self-tuning

Vision: all computer systems must be self-managed, self-organizing, and self-healing

Motivation:• ambient intelligence (sensors in every room, your body etc.)• reducing complexity and improving manageability of very large systems

aka. observation, prediction, reaction

Role model: biological, self-regulating systems (really ???)

25

Summary & Concluding RemarksMajor advances towards automatic tuningduring last decade:

Major challenges remain:path towards „autonomic“ systems requiresrethinking & simplifying component architectures with design-for-predictability paradigm

• workload-aware feedback control approach fruitful• math models and online stats are vital assets• „low-hanging fruit“ engineering successful• important contributions from research community (AutoRAID, AutoAdmin, LEO, Härder/Rahm book, etc.)

Problem is long-standing but very difficult and requires good research stamina Success is a lousy teacher. (Bill Gates)