testing database applications

Testing Database Applications

Donald Kossmann

http://www.dbis.ethz.ch

Joint work with: Carsten Binnig, Eric Lo

Thanks: i-TV-T AG, Porsche, Microsoft, Baden-Würtemberg

Quotes• „50% of our cost is on testing (QA)“

(Bill Gates @ Opening of Gates Building)

• „Testing alone makes up for six months of the 18 month product release cycle“(Anonymous SAP Executive)

• Estimated damage of USD 60 bln per year in USA caused by software bugs (US Department of Commerce, 2004)

• Mercury: 30% of testing can be automated

• HP: Buys Mercury for $4.5 bln

Observations• Everybody loves writing code

Everybody hates testing it– more work on new models etc. than on testing– solution: automate the testing!– (Computers are cheap and do not complain)

• Test Automation is a DB Problem– several optimizations in different flavors– it is all about logical data independence– it is a far cry from being solved!

• Research on Testing (Automation) is fun!

Test Automation• Idea: Testing ~ Programming

– Testprg = Actions + (desired) Responses– Examples: JUnit, Caputre & Replay

• But ...– Programming is expensive; Tests aren‘t any better– Maintenance of (Test-) Programs is expensive– Test-programs often have more bugs than the

systems they test– Test-programs are not enough: test databases – Test-programs must be optimized

• Idea of Test Automation is good!Risk: Going from bad to worse.

Project Goals

• Higher level of Abstraction of Tests (Prgs+DBs)– Avoid „over-specification“ of test components– Automate testing: execution, generation, evol., ...

• Automate Generation of Test Databases– Generate relevant Test Databases– Generate scalable Test Databases

• Automate Execution of Regression Tests– Optimization and Parallelization

• (Automate Evolution of Test-DBs + Programs)

Project Goals

• Higher level of Abstraction of Tests (Prgs+DBs)– Avoid „over-specification“ of test components– Automate testing: execution, generation, evol., ...

• Automate Generation of Test Databases– Generate relevant Test Databases– Generate scalable Test Databases

• Automate Execution of Regression Tests– Optimization and Parallelization

There are tools for some of these goals available. But they do not fit together, and have limitations.

Approach• Bottom-up: Various ideas and tools

– model-based testing (not invented by ETH)– Reverse Query Processing (ETH)– HTPar (ETH)

• Exploit „Standards“– Databases everywhere: „Fluch & Segen“– Web-based apps (simplifies tooling)

• Prototypes and Industry Collaboration– Canoo, i-TV-T, Microsoft, Porsche– Everything is implemented and „tested“

Agenda

• Motivation and Overview

• Database Regression Testing: Overview

• RQP: Generating Test Databases

• Related Work

• Conclusion

Structure of a Test Program

• Phase 1: Setup– Initialize Variables / State (= DB)– (long) sequence of SQL „insert“ statements

• Phase 2: Execute– Execute test step by step– Check responses of the system; compute ‘s

• Phase 3: Cleanup– Release resources– e.g., SQL „drop table“ statements

• Phase 4: Report– Report ‘s from execute

What is our contribution?• Get rid of setup und cleanup

– Only specify the id of a test-DB used for initialization– implicit setup / cleanup by testing infrastructure– reduces size of test code by up to 80% (IBM)– optimize the setup and cleanup

improve testing performance by a factor of 500 (Unilever)

• Execute– Model-based testing (HTTrace: Web-based C&R Toolkit)

• Only specify behavior you want to test; ignore randomness

– flexible, app-dependent function– XML representation of test runs for better evolution / queries

• Report– Understand the HTML page (tables, keys, same errors, etc.)

How to optimize regression tests

• Test 1: Insert a new order(Load Test-DB 1)insertOrder(Schmitz, 1000, Staples)showAllOrders()

• Test 2: Show all pending orders(Load Test-DB 1)showAllOrders()

How to optimize regression tests

• Test 1: Insert a new order(Load Test-DB 1)insertOrder(Schmitz, 1000, Staples)showAllOrders()

• Test 2: Show all pending orders(Load Test-DB 1)showAllOrders()

• How do you execute these 2 tests efficiently?– How do you do that with 1 million tests?– IBM manually defines test buckets! (bad!)

Solution Overview [Haftmann et al. 2007]

• Optimistic Execution of Test Programs– execute a test (setup = cleanup = NOP)– if it fails, reset the database and try again– if it fails again, then it really fails– (watch out for „false negatives“)

• Slice Algorithm– remember conflicts between test runs– create a conflict graph between test runs– order execution of test runs according to graph– (more smarts in the fineprint of the algo)

• Parallelization– shared nothing vs. shared DB– clever scheduling and reset strategies

Model-based Testing<project name="SimpleTest“ basedir=".“ default="main">

<property name="webtest.home“ location="C:/java/webtest"/>

<import file="${webtest.home}/lib/taskdef.xml"/>

<target name="main"> <webtest name="myTest">

<config host=www.myserver.com port="8080“ protocol="http“ basepath="myApp"/>

<steps><invoke description="getLoginPage“ url="login"/>

<verifyTitle description=“blabla“ text="Login Page"/>

</steps>

</webtest> </target></project>

Understanding HTML

Agenda

• Motivation and Overview

• Database Regression Testing: Overview

• RQP: Generating Test Databases

• Related Work

• Conclusion

State-of-the Art: DB Generation• Commercial Products and Open Source Tools

– Input: •DB Schema (Tables + Constraints)•Scaling Factor•(Constants)

– Output: SQL „insert“ Statements

• Result of the following query on generated DB?SELECT c.name, o.priceFROM Customer c, Order o, Region r, Product pWHERE c.region = r.id AND r.name = Asia ...

State-of-the Art: DB Generation• Commercial Product and Open Source Tools

– Input: •DB Schema (Tables + Constraints)•Scaling Factor•(Constants)

– Output: SQL „insert“ Statements

• What is the result of the following query?SELECT c.name, o.priceFROM Customer c, Order o, Region r, Product pWHERE c.region = r.id AND r.name = Asia ...

c.name o.price

The Solution

• Reverse Query Processing– Input:

•DB Schema (Tables + Constraints) •Scaling Factor•(Constants)•Application Program (SQL Queries)•Meaningful Query Results (Tables)

– Output: SQL „insert“ Statements

• Generate „Relevant“ Test-DBs– a Test-DB must be specific to the application– Evolution: Create new or extend Test-DB when the

schema evolves and application has new queries

Finding the right query• In practice, application has many queries

– workflow might involve sequence of queries– independent workflows have multiple queries

• Idea: generate one generic query that „encodes“ needs of a whole workflow– logically a „join/union“ of all queries

• Status: Create this query manually– still better than state-of-the-art: higher abstraction– but, indeed sometimes difficult to find

• Future Work: semi-automatic process– symbolic computation helps (SIGMOD 07)

RQP: Problem Statement• Given:

– Query Q, Table R– Schema S (including integrity constraints)

• Generate a database instance D such that:

R = Q(D)R = Q(D)

such that D matches S and its constraints

• Yes, the problem is undecidable (Q with “-”)– even undecidable whether such a D exists

• Who cares? You can always check D?– semi-automatic approach, if check fails

RQP Example

select c.name, sum(amount) as revenuefrom order o, customer cwhere o.cid = c.cid and c.age > 18group by c.name;

Database Constraint: Order.amount <= 70

c.name revenue

Paul 130 R

Q

S

RQP Example

select c.name, sum(amount) as revenuefrom Order o, Customer cwhere o.cid = c.cid and c.age > 18group by c.name;Database Constraint: Order.amount <= 70

c.name revenue

Paul 130 R

Q

S

cid amount

1 70

1 60

Order

cid name age salary

1 Paul 23 5000

Customer

D

Trichotomy (thanks to MJF)

Answers

Database

Queries

Trichotomy (thanks to MJF)

Answers

Database

QueriesQuery Processing

Trichotomy (thanks to MJF)

Answers

Database

Queries

Reverse Query Processing

Trichotomy (thanks to MJF)

Answers

Database

QueriesProgramming By Example

Architecture

Reverse Query Processor

run-time

compile-time

Bottom-up query

annotation

Query parser and

translator

Query optimizer

Modelchecker

Top-downdata

instantiation

Formula L Instantiation I

RTable R Parameter values

Database D

Query Q

Database Schema S

Reverse query tree TQ

Annotated T+Q

Optimized T’Q

Example• SQL Query Q:

SELECT SUM(price)

FROM Lineitem, Orders

WHERE l_oid=oid

GROUP BY orderdate

HAVING AVG(price)<=100;

• Schema S:CREATE TABLE Lineitem (

lid INTEGER PRIMARY KEY,

name VARCHAR(20),

price FLOAT,

discount FLOAT

CHECK (1>= discount >=0),

l_oid INTEGER);

CREATE TABLE Orders(

oid INTEGER PRIMARY KEY,

orderdate DATE);

Parser - RRA Tree

orderdateχ-1SUM(price), AVG(price)

σ-1AVG(price)<=100

П-1SUM(price)

-1l_oid=oid

Lineitem Orders

Traditional SQL Parsing; 1:1 relationship from RA to RRA

Parser - RRA Tree

orderdateχ-1SUM(price), AVG(price)

σ-1AVG(price)<=100

П-1SUM(price)

-1l_oid=oid

Lineitem Orders

Dat

a F

low

Traditional SQL Parsing; 1:1 relationship from RA to RRA

Reverse Projection

SUM(price)

100

120

orderdate SUM(price) AVG(price)

1990-01-02 100 100

2006-07-31 120 60

П-1SUM(price)

orderdateχ-1SUM(price), AVG(price)

σ-1AVG(price)<=100

П-1SUM(price)

-1l_oid=oid

Lineitem Orders

Reverse Projection

1st attempt (count=1):orderdate!=19900102 & sum_price=120 & avg_price<=100 &sum_price=price1 & avg_price=sum_price/1 Not Satisfiable!

2nd trial (count=2):orderdate!=19900102 & sum_price=120 & avg_price<=100 &sum_price=price1+price2 & avg_price=sum_price/2 Satisfiable!

Instantiationsum_price=120, avg_price=60,price1=80, price2=40, orderdate=20060731

Reverse Selection

σ-1AVG(price)<=100

orderdate SUM(price) AVG(price)

1990-01-02 100 100

2006-07-31 120 60

orderdate SUM(price) AVG(price)

1990-01-02 100 100

2006-07-31 120 60

Reverse Aggregation

orderdateχ-1SUM(price), AVG(price)

lid name price discount L_oid oid orderdate

1 A 100 0.0 1 1 1990-01-02

2 B 70 0.0 2 2 2006-07-31

3 C 50 0.0 2 2 2006-07-31

orderdate SUM(price) AVG(price)

1990-01-02 100 100

2006-07-31 120 60

Reverse Equi Join

-1l_oid=oid

oid orderdate

1 1990-01-02

2 2006-07-31

lid name price discount L_oid

1 A 100 0.0 1

2 B 70 0.0 2

3 C 50 0.0 2

lid name price discount L_oid oid orderdate

1 A 100 0.0 1 1 1990-01-02

2 B 70 0.0 2 2 2006-07-31

3 C 50 0.0 2 2 2006-07-31

Architecture

Reverse Query Processor

run-time

compile-time

Bottom-up query

annotation

Query parser and

translator

Query optimizer

Modelchecker

Top-downdata

instantiation

Formula L Instantiation I

RTable R Parameter values

Database D

Query Q

Database Schema S

Reverse query tree TQ

Annotated T+Q

Optimized T’Q

Reverse Query Optimization• Some observations

– projections and group by‘s are expensive– (equi-) joins and selections are cheap– nested queries are expensive– calls to the model checker are expensive

• depend on number of free variables, types of vars

• Conclusions– apply „smarts“ to avoid model checker calls– apply smarts to simplify model checker calls– do aggressive query rewriting– but do not worry about join ordering, push-down

Correctness Criterion

• Rewrite of Plan P1 into Plan P2 allowed iff

Q(P1(R) = Q(P2(R)) = R

• That is, P1 and P2 may produce different databases!!! That is okay.

• Goal (here): generate large databases fast. (Alternative goal: „good“ DBs)

R = Q(D)R = Q(D)

Query Unnestingselect name from lineitemwhere l_oid not in (select max(cid)

from ordersgroup by odate)

becomes

select name from lineitem

• (An empty „orders“ table is generated!)

Query Unnestingselect name, price from lineitemwhere price = (select min(price)

from lineitem)

becomes

select name, price from lineitem

• (Precise definition of rules in the Tech.Rep.)

• (Of course, all traditional rules are applicable.)

Some Tricks (see TechRep for complete list)

• (Constrictive) Independent Attributes– can take random values (avoid model checker)– or can take fixed values (in „distinct“ queries)

• Infer cardinalities from AVG and SUM– avoid trial-error algorithm

• Bound cardinalities from MAX, MIN, SUM– limit trial-error algorithm

• Simplify constraint formulae– use SUM(a) / n for aggregations

• Memoization: cache model checker calls

Performance Experiments

• Use dbgen in order to generate TPC-H DBs– use three scaling factors: 100 MB, 1 GB, 10 GB

• Run 22 TPC-H Queries on DBs (PostGres)– get 22 x 3 RTables

• Run RQP on 22x3 RTables– get 22x3 different DBs

• Compare original DB with generated DBs

• Measure Running Time of RQP

Results (DB Size)

100 MB 1 GB 10 GB

Query RTable Generated RTable Generated RTable Generated

1 4 600.572 4 6.001.215 4 59.986.052

2 44 220 460 2.300 4.667 23.335

3 1216 3.648 11.620 34,86 114.003 342.009

4 5 10.186 5 105.046 5 1.052.080

5 5 30 5 30 5 30

6 1 1 1 1 1 1

7 4 24 4 24 4 24

8 2 32 2 32 2 32

… … … … … … …

Results (hh:mm:ss)

Query 100 MB 1 GB 10 GB

1 26:51:00 207:11:00 2054:19:00

2 00:24 00:47 04:02

3 19:20 183:49:00 1819:48:00

4 00:20 02:26 24:15:00

5 00:12 00:12 00:12

6 00:02 00:01 00:01

7 00:10 00:10 00:09

8 00:15 00:17 00:14

… … … …

Other RQP Applications

• Updating (non-updateable) Views– find all possible update scenarios– define a policy that selects update scenario

• Privacy / Security– what can be inferred from the published data

• SQL Debugger– determine operator that screws up result

• Program Verification (weakest pre-condition)

• Database Compression / Sampling– real DB -> queries -> results -> queries -> small DB

Symbolic SQL Computation• Goal: control of intermediate results

– selectivity of query operators, cardinality, distr., ...– test database system (not app); e.g., optimizer

• Idea: Process tuples with variables as values

– Put constraints on variables: e.g., $z > 1000– (Reverse/Forward) process variables with query – instantiate variables at the end -> test database

Customer Product Volume

Paul Smith $x1 5000

$y $x2 $z

Agenda

• Motivation and Overview

• Database Regression Testing: Overview

• RQP: Generating Test Databases

• Related Work

• Conclusion

Related Work• Testing owned by Software Eng. Community

– JUnit: Mother of regression testing for Java– focus on processes and methodology– database often simulated using mock objects :-)

• noticeable exception: AGENDA Project (Chays et al.)

– no mention of „optimization“, „data independence“

• Generating Test Databases– Gray et al. (SIGMOD 94), ...– Bruno, Chaudhuri, Thomas (TKDE 06)

• Generating SQL Test Queries– Slutz (VLDB 98), Poess, Stephens (VLDB 04)

Conclusion• Automated testing has many hidden costs

– Definition of test modules / buckets – Definition of the order of test execution (manual parallel.)– Generierating Test-DBs (adjusting Test-DBs)– Evolution of tests and Test-DBs with new releases– Writing code for setup and cleanup– Definition of delta function– ...

• Vendors solve one problem at cost of another• We don‘t have a good solution, but ...

– we have some fun ideas– and we are honest

Research Challenges (CIDR 05)

• Test Run Generation (in progress)– automatic (robot), teach-in, monitoring,

decl. specification

• Test Database Generation (in progress)• Test Run, DB Management and Evolution (uns.)• Execution Strategies (solved), Incremental (uns.)• Computation and visualization of (solved)• Quality parameters (in progress)

– functionality (solved)– performance (in progress)– availability, concurrency, scalability, security (unsolved)

• Cost Model, Test Economy (unsolved)