A Multi User Unit Test Framework for Testing Database Applications

Project Period: Dat6, February 1st, 2004 � June 1st, 2004

Supervisor: Kristian Torp

A Multi User Unit Test Framework

for Testing Database Applications

Claus Abildgaard Christensen Steen GundersborgKristian de Linde Jacob Richard Thornber

Department of Computer Science, Aalborg University, Denmark{cac,eraser,kdl,jrt}@cs.auc.dk

31st May 2004

Abstract

Unit testing is the foundation for building correct androbust applications. Unit testing of database appli-cations di�ers from traditional unit testing becauseof persistency and foreign key constraints. Before ex-ecuting a test case, data associated via foreign keysmust exist. Furthermore, after each test case it isnecessary to clean up any modi�cations made to thedatabase. Failure to do so will cause the second unittest to fail.

Existing test frameworks assume independency be-tween test cases. Retaining this assumption whentesting database applications, results in a high test�xture. Therefore, a new approach is preferable. Wepropose that test cases and unit tests should be al-lowed to depend on each other. This enables us toreuse test �xture and avoid redundant code. Further-more, as test �xture is only set up and torn down onceper unit test, as opposed to once per test case, ourframework o�ers a signi�cant performance improve-ment compared to existing frameworks.

We have designed and implemented the proposedtest framework, minimizing test �xtures and provid-ing automatic cleanup of the database, also in thecase of a system crash. We conclude that the testframework is simpler to use than existing frameworks.It allows for faster execution of unit tests, requiresless work from developers, and it can be integratedwith an existing framework.

1 Introduction

The eXtreme Programming (XP) methodology [2]is becoming increasingly popular among developers[17]. To ensure the correctness of the produced soft-ware, an important aspect of XP is the use of auto-mated unit testing. Moreover, to guarantee the cor-

rectness of software, even after the addition of newfunctionality or test cases, regression testing is rec-ommended.

Furthermore, development moves more and moretowards using databases, e.g., in ERP systems andin e-commerce. As competition in the software mar-ket increases, it is important to ensure correctness ofapplications. To ensure this, it is preferable to intro-duce unit testing in this domain.

Integrity constraints bind database tables together.Therefore, it is, in some cases, necessary to insertdata in one table before being able to test meth-ods implemented on another table. These integrityconstraints make some unit tests dependent on eachother.

Current unit test frameworks as JUnit [12] and ut-PLSQL [9] all share the common assumption, thattest cases are independent, as described in the XPmethodology. This article settles with this assump-tion. We argue that, when testing database appli-cations or programs using associations and aggrega-tions, it is bene�cial to allow both unit tests and testcases to be dependent.

Testing of database applications di�ers from test-ing traditionally software in that most stored pro-cedures have persistent side e�ects where as testingtraditional applications has data stored in main mem-ory and is cleaned up by the test framework at pro-gram termination. For example an insert methodcannot typically insert the same row twice in a ta-ble, without violating the integrity constraints in thedatabase. Therefore, it is typically not possible to ex-ecute the same test methods twice, without cleanupbetween the executions of the tests. This makes re-gression testing of database applications more di�-cult as clean up of the database also have to be con-sidered.

The contributions of this article are in the �eld oftesting database applications. First, it presents a new

1

approach for testing database applications that canbe combined with existing frameworks. Secondly, itexplains the ideas behind a partially automatic test�xture setup and teardown methods. Furthermore, ourframework allows for reuse of the text �xture and au-tomatic clean up of the database in case of malfunc-tioning test cases or a system crash. Thus, the timespent before and after testing is minimized. Hereby,we avoid the repetitive test code that is necessary inexisting unit test frameworks. The article gives de-sign and implementation details of the proposed testframework. Finally, we present an extension to thetest framework that allows for multi-user support.

The paper is structured as follows. Section 2 looksat the related work done in the �eld of database test-ing. In Section 3, we present the test setup environ-ment, i.e., a sample database schema used through-out the article as basis for examples. Section 4presents the ideas behind the test framework. Section5 describes the multiple user extension to the frame-work. It presents how the single user test frameworkhas to be extended in order to support for multipleusers testing simultaneously. Section 6 describes howthe test �xture is reduced by ordering the unit test,and allowing for concepts to remain set up duringseveral unit tests. In Section 7, the utPLSQL frame-work is examined. Section 8 explains how the testframework could exploit Oracle speci�c functional-ity. In Section 9, implementation details of the singleuser framework are given. Furthermore, an approachfor combining our framework with utPLSQL is given.Section 10 gives a performance analysis of three testapproaches. Finally, in 11 the conclusion and futurework are given.

2 Related Work

The JUnit testing framework [7, 12] has become thede facto standard for implementing Java unit tests.JUnit supports for the selective retesting, also knowas regression testing [8], of a software system that hasbeen modi�ed as a result of a bug. Moreover, no pre-viously working functions is failing as a result of thereparations made. Also, to ensure that newly addedfunctions does not cause previously de�ned functionsto fail. Bottom line, JUnit is a testing frameworkthat de�nes classes for testing. The main class isknown as the TestCase class. A developer uses JU-nit by extending this class and de�nes assertions andtest methods denoted by the pre�x test. These testmethods are responsible for the calls to the applica-tion that the developer wants to test. If any of theassertions fails, it is displayed. JUnit's test �xtureis a common set of test data and objects, which isshared by all tests. It is instantiated by the means

of two special methods, the setUp and the tearDownmethods. Once the test �xture has been setup, JU-nit allows for easily testing the application. The testcase, or a collection of test cases also known as a testsuite, can easily be run.

The DbUnit testing framework [1] is a JUnit [12]extension that is aimed speci�cally at database-driven applications. Before a test case is executed,the framework puts the database in a known state,which per default is the empty state. This is done bytruncating all tables. Therefore, test cases that failcannot corrupt the database. Several best practicesare proposed, e.g., all testing should be conductedon a dedicated test server. As with all frameworksbuild on the JUnit framework, DbUnit also make theassumption that test cases are independent.

In [4], Chays et al. present a design of a frame-work for testing database applications. They discussthe role of the database state which makes testingdatabase application di�erent from testing applica-tions that do not store data persistently. They pro-pose a tool-set architecture consisting of four tools,including a prototype implementation of the �rsttool. This tool populates a given database with datasatisfying integrity constraints.

Books on traditional software testing [2, 11, 16]provide no speci�c handling of testing of databaseapplications. In [13], Lewis does consider testing ofdatabase applications. However, Lewis only considershow to test if integrity constraints are ful�lled, e.g.,how to see if all primary keys are valid, somethingmost RDBMSs do automatically.

In [2], Beck advocate for automatic testing, how-ever, as with JUnit [12] an important assumption isthat all test cases are independent. In this article weclaim that when testing database applications it isbene�cial to let test cases depend on each other.

The utPLSQL framework described in [9] is anOracle PL/SQL unit-testing framework. The frame-work is modeled on the JUnit [12] framework. TheutPLSQL framework enables testing of each packageautomatically. However, utPLSQL is developed fortesting PL/SQL applications, where our frameworkfocuses on testing the interaction between the appli-cation and the database.

In [10], Daou et. al categorize the problems thatSQL in�icts on regression testing into three groups:control dependencies, data �ow dependencies, andcomponent dependencies. They suggest a methodfor performing regression testing on database applica-tions. According to Daou et. al, programming withSQL and database triggers often involves exceptionhandling, that is many exceptions have to be caughtand handled. As traditional control �ow modelingdoes not contain a way of modeling exception han-dling, Daou et. al propose a way of doing this. Each

2

statement is represented as a node, each node has twoexits one for successful endings the other for unhan-dled exceptions. If no exception handling is available,all exception links will be linked to an unhandled ex-ception node. A speci�c exception is modeled by apredicate node, which has two exits, the �rst links tothe start node of the exception block, the other to thenext handled exception. The regression test methodis split into two parts, part one consists of modi�ca-tion detection, and impact analysis, i.e., estimatingwhat will be a�ected if a change in the software ismade. As a change in one component often a�ectsother components, because the state of the databaseis altered, it is necessary to �nd all dependencies be-tween components of the application in order to de-termine which components should be tested. A com-ponent is marked as modi�ed and placed in a list ofmodi�ed components, called the component �rewall ,if either itself has been modi�ed or deleted, or if it isdependent on a modi�ed or deleted component. Alltest cases that involves components in the �rewallare selected for regression testing. This article di�ersfrom [10] by Daou et. al in that we consider unit test-ing, whereas Daou et. al strictly considers regressiontesting, i.e., it is not discussed how a single test caseshould be set up.

3 The Test Setup

This section introduces a small database schema thatwill be used as an example throughout the article.Hereafter, the used terminology is explained. Theschema depicted in Figure 1 is used as an exampleschema when describing the test framework.

Figure 1: The University Example Schema. At-tributes in italics denotes the primary key of a table

The example schema represents a small university.The university has a number of students, informationon these is stored in the student table. The course ta-ble stores the courseidenti�er (cid ), the course name

and the foreign key teacher identi�er (tid). The par-ticipant table is a relationship between the studentand the course tables. The primary key of the par-ticipant table are the sid and cid attributes, whichare also foreign keys referencing the student and thecourse tables, respectively. In addition, the partici-pant has three additional attributes. First, enrolledthat describes when the student is enrolled to thecourse. Secondly, a type attribute that describes ifthe student is following the course in the traditionalmanner or if he is distant learning. Finally, the statusattribute describes if the student is active or inac-tive in the course. As depicted in the ER-diagram inFigure 1, a student can follow zero or many coursesand a course can have zero or many students. Re-call the foreign key tid in the course table, this keyreferences the teacher table. Each course has exactlyone teacher associated. A teacher can teach zero ormany courses. A teacher can act as a supervisor forother teachers. This is modeled by the self-referenceand the bossid attribute on the teacher table. Fi-nally, each teacher is situated in exactly one o�ce.This is modelled through the composite foreign keyconsisting of building and room, which references theo�ce table. The semester table is used to connectstudents and courses to speci�c semesters, e.g., anintroductory semester. The primary key of the ta-ble is the semid attribute, which is used as a foreignkey in the student and course tables. Moreover, thesemester table has two date attributes indicating thestart and end date of a semester.

The terminology used in [5] is also used in thisarticle. Meaning that a concept is equivalent to aclass in an object-oriented language or a table in thedatabase. A record corresponds to an instance of aconcept or a row of a database table. Finally, a recordid is a unique identi�er of a record, corresponding tothe primary key of a database table.

4 Test Framework

In the following section a framework for unit testingdatabase applications is designed. It aims at correct-ness testing and not performance testing. Further-more, the framework aims at minimizing test �xture[3], also known as the amount of work needed beforeafter doing actual testing. The framework consists ofa set of methods associated with each concept and apersistent stack containing the concepts active dur-ing each step of the testing. These will be describedindividually. Hereafter, a set of conventions are de-scribed. These conventions are necessary for the de-veloper to follow for the framework to work. Finally,examples of the test framework are presented.

3

4.1 The idea of the frameworkTraditionally, when testing code, test cases are run,errors found are corrected and the test cases are re-run. Moreover, they might be executed every night toensure any modi�cations made to the code during theday do not violate any test cases. Several tools likeJUnit [12] and utPLSQL [9] exist that are well suitedfor this task � when persistency is not considered.Generally, test cases are grouped into unit tests,

each testing a speci�c part of the given application.The entire collection of unit tests is again groupedtogether, forming a test suite for testing the entireapplication. Using a part of the schema depicted inFigure 1, Figure 2(a) shows an example of a test suitecomprised of three unit tests. The example shows anexecution of the test suite, running the three unittests. It is important to note that the unit tests canbe run independently, and furthermore, that each testcase can be run independent from other test cases.This is due to the fact that each test case only usesmain memory data structures. When the test is com-pleted the data is simply discarded because the pro-gram exits. At a higher level of abstraction this canbe viewed as testing functions, i.e., on a given input,compare the output with the expected output.

Figure 2: Di�erence from standard unit testing tounit testing of database applications. Part A is thestandard way of testing. Part B test of database ap-plications, where test cases are dependent of eachother, i.e., to test course relies on tests on teacherwhich requires o�ce to be tested �rst.

When testing code that uses data stored persis-tently in a database things are more di�cult becausethe outcome of a test case depends on the state of thedatabase. Furthermore, each test case may have per-sistent side e�ects, altering the state of the database.This can be viewed as testing procedures. Since thereis no output it is testing for side e�ects, in contrastto existing test frameworks.Testing database applications poses two main

problems: Persistency and foreign keys. The persis-tent side e�ect of a unit test is already an issue when

testing an independent concept. Since connected con-cepts require several concepts to be set up in orderto test a single concept, foreign keys pose a prob-lem when testing these. In the following, solutions toboth problems are proposed.

4.1.1 Persistency

When testing database applications data used in thetest cases are stored persistently in the database.Hence, without proper cleanup after running a testcase, a second execution might not yield the sameresult or fail. Thus, to preserve the property of testcases being independent, proper cleanup must be per-formed after each test case. However, since this im-poses a high performance overhead we suggest an-other approach, where we allow test cases within aunit test to depend on each other, making room forreuse of code. This implies that test cases no longerare independent since any modi�cations made by onetest case a�ects all succeeding test cases. Hence, ingeneral, single test cases cannot be run individuallybut only as a part of executing a unit test. This im-plies that cleanup is only necessary after running theunit test.To be able to do the cleanup of the database after

running a unit test, it is necessary to setup some rulesfor what is allowed when modifying data and keeptrack of what needs to be undone. These rules arethe following.

• Primary keys or any other candidate keys usedin the unit test must be public.

• It is only allowed to change the data that thetest itself has inserted. This includes primarykeys and candidate keys.

If these rules are followed then the framework ensuresthat it can clean up, even if the test crashes.

4.1.2 Foreign keys

In database schemas there can be dependenciesamong data. Concepts may rely on data from otherconcepts, posing a problem when testing. Propersetup is required such that all needed concepts validdata.Traditional testing methods dictate that unit tests

must be self containing. Using this idea, imagine aunit test for the teacher concept. Figure 3 shows thatconcepts are associated to each other via foreign keysbetween tables, and unit test for these concepts aretherefore also associated. Since the concept associ-ated with the unit test for teacher depends on theo�ce concept. Thus, the unit test must supply validdata for the table associated with the o�ce conceptto remain independent. Now imagine a unit test for

4

the course concept. The course concept depends onthe teacher concept and on the o�ce concept viatransitivity. Therefore, the unit test for the courseconcept must supply valid data, not only for the ta-ble associated with the teacher concept, but also forthe table connected with the o�ce concept. Hence,repetitive code for creating valid data for table con-nected to the o�ce concept exists in the unit testsfor both the teacher � and the course concept.

Figure 3: The coherence between tables, conceptsand unit tests.

This is only a simple example schema. Realschemas are much more complex, thus the amountof redundant code becomes very high. Similarly, theproblem with repetitive occurs when cleaning up.Figure 2(b) shows a test suite comprising of threeunit tests. This �gure illustrates how the unit tests inFigure 3 depends on each other. Using our approach,reconsider the unit test for the teacher concept. In-stead of supplying valid data for the o�ce concept,we reuse the code already in a unit test for the o�ceconcept. When testing the course concept, the codefrom a unit test for teacher is reused, which againreuses code from a unit test for o�ce. This way, aunit test must only supply valid data for the conceptit is testing, reusing code from unit tests belonging tothe concepts, the current concept is directly relatedto. Again, the same approach is applied when clean-ing up. Compared to the traditional approach thisgives a signi�cantly lower test �xture.

4.2 Extending the ideaThe idea of the test framework can be utilized inother contexts than database applications. Testingprograms with associations or aggregations, e.g., inthe Java programming language, pose some of thesame problems. Thinking of references between con-cepts as foreign-key dependencies, there is a majordi�erence. Associations and aggregations result inmain-memory structures only. Hence, cleanup is notnecessary between test runs since data placed in main

memory is removed at unit test termination. In otherwords, persistency is not an issue. However, populat-ing main memory with references to valid conceptsprior to running a given unit test is conceptually thesame as supplying valid data for tables related to con-cepts needed for a given unit test. Thus, the exam-ple given in Section 4.1.2 is just as relevant to testingassociations and aggregations as testing database ap-plications. However, since they only pose a subset ofthe problems associated with testing database appli-cations, we choose to focus only on the latter.

4.3 The FrameworkThe framework consists of a set of methods, publicconstants and a persistent stack: A setup method, ateardown method, a run method, a group of test meth-ods, public constants and �nally a table containingthe active concepts, i.e., the concepts needed for thecurrent test. This is depicted in Figure 4. The con-structs will now be described in further detail.

Figure 4: An UML class diagram of the framework.

4.4 Framework ConstructsIn the following, each of the methods supplied by theframework will be described.

Setup: The setup method is used to ensure thatthere is a valid foundation for testing, i.e., thedatabase is put in a known state. Recall Figure3, in order to test the course concept it isnecessary to have data in the teacher concept.The purpose of the setup method is to ensurethat for the given concept there is valid data inthe table related to it. Since tables can haveforeign keys to other tables these also need tocontain valid data. This is achieved by havingthe setup method call the setup method on allconcepts the current concept is related to viaforeign key constraints.

If the setup method is on the concept currentlybeing tested then it only calls the setup on otherconnected concepts. Thus, it does not insert anydata. However, if the setup is not on the conceptbeing tested then other needed setup methods are

5

called and data is inserted. This is done becausethe inserts is a part of the tests themselves. Thesetup method is schema dependent.

Teardown: The teardown method is the reverse ofthe setup method. It ensures that modi�cationsmade to the table related to a given concept dur-ing setup and testing is undone, leaving the ta-ble in the state it was in before testing begun.As the setup method, the teardown method reusesteardown code from other unit tests. When sev-eral concepts have been setup for a given test,teardown must be called in the reverse order, e.g.,the last concept that is setup is the �rst to betorn down. This is also illustrated in Figure 2b.Here, course was the last to be setup, thus it isthe �rst to be torn down. The teardown methodis schema dependent.

Run: The run method is the method called for exe-cuting the unit test itself. The run method en-sures that there are valid data in all required ta-bles by calling the setup method. The test itselfis done by calling all the test methods, in the cor-rect order. Finally, the cleaning of the databaseis done by calling the teardown method. The runmethod is schema independent.

4.4.1 Active concepts

A persistent stack of the concepts currently active ismaintained. This information ensures the robustnessof the framework, since in case of a system break-down any modi�cations made to the database can beundone using the cleanup method, when the systemagain is up and running. Whenever the setup methodhave been called on a concept, this concept is pushedonto the stack using the push method. As soon as theteardown method is called on a concept, this conceptis popped o� the stack, via the pop method. However,the concept is only popped if it is on top of the stack,and this is checked by the top method. To determineif a concept already is active we allow looking at allthe elements of the stack, not being able to modifythem, however. The isActive method o�ers this func-tionality. The methods using the persistent stack areexecuted in an autonomous transaction. The stack isschema independent.

4.5 Conventions

This section describes the conventions that the de-veloper has to follow in order for the test frameworkto work.

4.5.1 Public constants

The public constants are used by the test frame-work to refer to primary keys. These are not onlyon the concept which is currently being tested, butalso on the concepts that are connected via foreignkeys. There are three types of these constants.

1. The �rst type is mandatory and used to insertdata. Thereby, they are available as foreign keyswhen inserting in other tables. These primarykeys are valid and used in the database.

2. The second type is optional and used to enablemodi�cations of primary keys and insertion ofadditional data in the actual test methods be-tween the setup and the teardown phases of thetest.

3. The third type is optional and consists of pri-mary keys that are valid and does not exist inthe database. These are not to be used for in-sertion, but for testing for non-existing rows.

The public constants are schema dependent andare created by the developer.

4.5.2 Ordering of Tests

It is necessary to enforce a bottom-up approach whentesting concepts. The developer has to start by test-ing concepts that other concept does not reference.In the university example, this means that the de-veloper should start by testing either the semesteror the o�ce concepts. When the semester concepthas been tested, the developer can continue on to thestudent concept and so forth.As test cases are dependent, it is also important

that the developer consider this, when designing aunit test. For example, it would be natural to in-sert data in a concept, before testing, e.g., an existmethod.

4.5.3 Naming conventions

Packages and methods that are to be tested, are bydefault pre�xed with test_. Then, the test frameworkwill automatically execute these tests. Naturally, thedefault pre�x can be rede�ned by the user. Withouta pre�x, the framework would execute every method,which often would not be desirable.

4.6 Reuse of codeThe framework reuses code for setup and teardown tominimize the test �xture. The following examplesequence diagram in Figure 5 shows the process ofsetting up the database for a test and the following

6

sequence diagram in Figure 6 shows the process ofcleaning up the database after the test. The exam-ple diagrams use a subsection of the schema shown inFigure 1. The subsection of the schema only containsthe tables student, participants, course and semester.

4.6.1 Setting up the database

Figure 5: Sequence diagram showing the process ofsetting up the database for a test run. The abbrevi-ations P: participants unit test, S: student unit test,Sem: semester unit test and C: course unit test.

The example sequence diagram in Figure 5 showsthe process of setting up the database for a unit teston the participants concept. The test is initiatedby calling the run method on the participants unittest. The run method ensures that all the neededmethods are called in the correct order. Hereafter,the run method calls the setup method on the par-ticipants unit test. The setup method is called withthe boolean value true that tells the setup methodthat it is this unit test being executed now. Thesetup knows from the data dictionary [14] that datais needed in both the student table and the coursetable. Therefore, setup calls the setup method on thestudent unit test. However, setup on student needsdata from the semester table. Therefore, it calls thesetup method on the semester unit test and the �rstdata is inserted. The setup method from the semesterunit test returns after inserting the data and setup onstudent is now able to insert data and return. Asthe participants table dependens on the course ta-ble, setup is still not ready to insert data, so setup onthe course unit test is called. The course setup needsdata from the semester table and, therefore, calls thesetup method on semester. The semester setup returnswithout doing anything since data is already inserted.The course setup now inserts data and returns. Theparticipants setup method is now �nished and returnsto the run method which initiates the unit test.

Figure 6: Sequence diagram showing the process ofcleaning up the database after a test run. The ab-breviations P: participants unit test, S: student unittest, Sem: semester unit test and C: course unit test.

4.6.2 Cleaning up the database

The example sequence diagram in Figure 6 shows theprocess of cleaning up the database after running theunit test on the participants concept. The cleanupis initiated by the run method calling the teardownmethod on the participants unit test. The teardownmethod removes all test data from the participantstable and calls the teardown method on the courseunit test. The course teardown removes the test datafrom the course table and calls the teardown methodon the semester unit test. The teardown on semesterreturns without removing any data, because its datais still in use and the teardown on course returns. Theparticipants teardown now calls the teardown methodon the student unit test. The student teardown re-moves the test data for the student table and callsthe teardown method on the semester unit test. Thesemester teardown removes test data and returns tothe student teardown which returns to the participantsteardown method which returns. This concludes thecleanup of the database.

4.7 ExamplesIn this section we give two examples of how the testframework functions. Initially, we consider a test in-volving only one concept. Hereafter, we consider acase involving two concepts.

4.7.1 Testing an independent concept

In this section, we consider a scenario where we wishto test a single concept, in this case the o�ce conceptfrom the small university scheme depicted in Figure1. The test is initiated by calling the run method.The run method starts by calling the setup method.The run method is shown in Listing 1.

1 procedure run is2 begin

7

3 setup(true);4 test ();5 teardown();6 end;

Listing 1: The run method when testing on a singleconcept.

As the o�ce concept has no dependencies to otherconcepts, no setup of any other concepts are nec-essary. When only one concept is tested, the setupmethod does not insert data, this is left to the devel-oper. Instead, the setup method initializes the publicconstants that are available to the developer. Theseconstants include those that are to be used as primarykeys and primary keys that the developer are allowedto update, as described in Section 4.5.1. Hereafter,the setup method returns to the run method, whichcalls the test methods. The setup method is shown inListing 2.

1 −− package header2 building constant := 'E4';3 room constant := '110';4 building_update constant := 'E3';5 room_update constant := '210';67 −− package body8 procedure setup(self_test boolean) is9 begin

10 if (not stack. is_active(' ' o�ce ' ' )) then11 stack. push('' o�ce ' ' );1213 if (not self_test ) then14 insert (building, room);15 end if ;16 end if ;17 end;

Listing 2: The setup method when testing on a singleconcept.

In lines 2-5 of Listing 2 the public constants are setup. Note that in PL/SQL this is done the header ofa package. In line 10, we check if the o�ce conceptis already on the persistent stack. If this is not thecase, the concept is set up. In line 11, we push theo�ce concept onto the persistent stack. This enablescleanup of the database, should an error occur. Asthis is a test of the o�ce concept, the setup methoddoes not insert any records into the o�ce table. Thedeveloper will have to do this manually in the testmethods. Here it is important to note that the de-veloper should use the de�ned constants.An example of a test method written in pseudo

code is shown in Listing 3.

1 procedure test2 begin3 insert (building, room);4 assert . eq(' Building not found!' ,5 exist (building, room), TRUE);67 update(building, room8 building_update, room_update);9 assert . eq(' Building_update not found!',

10 exist (building_update,

11 room_update), TRUE);12 end;

Listing 3: The test method for testing the o�ce con-cept.

In line 3 of Listing 3 a record is inserted into theo�ce table, and the public constants de�ned in thesetup method is used. Immediately after the insertion,in lines 4 and 5, an exist method is used to check if thenew record in fact is inserted. Should this not be thecase, the assert method will return the error string' Bulding not found'. In line 7 we update the primarykey of the record inserted in line 3. Hereafter, theexist method and the assert . eq method is used to checkif the record has been updated. Again an error stringis returned if the record does not exist.As shown in Listing 2, the setup method calls the

teardown method immediately after the test methodcompletes its execution. The teardown method isshown in Listing 4.

1 procedure teardown2 begin3 if ( stack. top(' o�ce ' )) then4 delete (building_update, room_update);5 delete (building, room);6 commit();7 stack. pop();8 end if ;9 end;

Listing 4: The teardown method for testing the o�ceconcept.

First, the teardown method checks that the o�ceconcept is on top of the stack, in line 3. If not, otherconcepts still depend on it, and the teardown methoddoes nothing. If the o�ce concept is on top of thestack, all records inserted into the o�ce concept areremoved, in lines 4 and 5. Please note that the trans-action is commited in line 6. This is done to ensurethat any concepts that have not been cleaned up areremoved from the stack. Finally, line 7 pops the of-�ce concept of the persistent stack as it is no longerin use, and the o�ce concept has been properly re-stored to it's default state.In the next section we consider unit test, where we

are to test the teacher concept. The interesting thinghere is that in order to test the teacher concept weneed data from the o�ce concept.

4.7.2 Testing connected concepts

In this section we show an example of a unit test onthe teacher concept. As mentioned previously, theinteresting thing here is that the teacher concept isconnected through foreign keys to the o�ce concept.The unit test initiates in the same way as described inthe previous section, by calling the run method. Thedi�erence lies in the setup method and the teardownmethod. The setup method is depicted in Listing 5.

8

1 −− package header2 tid constant := 1;3 tid_update constant := 10;45 −− package body6 procedure setup(self_test boolean) is7 begin8 if (not stack. is_active(' teacher' )) then9 o�ce . setup(false );

10 stack. push('teacher' );1112 if (not self_test ) then13 insert (tid , o�ce . building, o�ce . room);14 end if ;15 end if ;16 end;

Listing 5: The setup method used to test the teacherconcept.

As in the previous section the public constants arede�ned in the package header. The �rst things thathappen in the setup method is that we check if theteacher concept is active. If this is not the case, thesetup method on the o�ce concept is called with theboolean value false. The false value indicates thatthe setup method on o�ce should load data into theo�ce table, as another concept is using it for testingpurposes. As described in the previous section, theo�ce concept is also pushed onto the persistent stack,indicating that the concept is now active and shouldbe restored to its default state upon completion of theunit test. Hereafter, in line 10, the teacher interfaceis pushed onto the persistent stack. All concepts arenow properly set up and, we are ready to run the testmethods, that is shown in Listing 6.

1 procedure test2 begin3 insert (tid , o�ce . building, o�ce . room);4 assert . eq(' Teacher not found!',5 exist (tid , o�ce . building,6 o�ce . room), TRUE);7 end;

Listing 6: An example test method on the teacherconcept.

In line 3, a record is inserted into the teacher con-cept. Note here that some attributes have been omit-ted for brevity � naturally they should be included ina real test case. The values used in the insertion arethe publically de�ned tid, and the public constantbuilding and room from the o�ce concept. In lines 4and 5, the exist and the assert . eq methods are used tocheck if the record is in fact inserted. Should this notbe the case an error string is printed. After comple-tion of the test, control is returned to the run methodthat calls the teardown method on the teacher concept.The teardown method is shown in Listing 7.

1 procedure teardown2 begin3 if ( stack. top(' teacher' )) then4 delete (tid_update);5 delete (tid );

6 commit();7 stack. pop();8 o�ce . teardown(false);9 end if ;

10 end;

Listing 7: The teardown method of the teacher con-cept.

First, the teardown method checks that the teacherconcept is on top of the stack, in line 3. If not, theteardown method does nothing. If the teacher conceptis on top of the stack, all records inserted into theteacher concept are removed, in lines 4 and 5. Notethat the transaction is committed in line 6. This isdone to ensure that any concepts that have not beencleaned up are removed from the stack. In line 7, theteacher concept is popped of the persistent stack asit is no longer in use, and has been properly restoredto the default state. In line 8, the teardown methodon the o�ce concept is called. When that methodhas �nished executing, all a�ected concepts have beenrestored and all elements have been popped o� thestack.

4.7.3 Summary

The above examples show how to use the test frame-work. There are some advantages of using the testframework. The framework provides a systematicway of setting up the database for a unit test andan automatic cleanup after testing. Furthermore, theframework is able to cleanup test data when exposedto system crashes. Additionally, the framework has alittle test �xture. However, these advantages come ata cost that comes in the form of some requirementsto the user of the test framework. The user mustpublish primary keys and call methods in the cor-rect order. Furthermore, the user may only changedata that the test itself had inserted and only modifyspeci�c primary keys. Furthermore, the frameworkrequires that the schema must exist.

5 Multi User Framework

The framework is extended to support multiple users.In the following section, the challenges on the multi-ple user extension are discussed. First, an overviewof the proposed solution is given. Then, limitationsof that solution are discussed. Hereafter, details ofthe extension are presented. Finally, an example oftwo users testing simultaneously is given.

5.1 ChallengesWhen extending the framework to support multipleusers the following challenges arise.

9

Concurrency Several users could try to test thesame concept at the same time. This is a prob-lem because both users would try to insert datausing the same primary keys, which would leadto primary key violations.

Dependent concepts Users could try to test con-cepts that depend on each other. For exam-ple, the student and the course concepts canbe tested simultaneously. However, the studentand the semester cannot be tested simultane-ously, because the test on the semester conceptmight change the data of the semester table andthereby corrupting the student test.

Clean up Cleaning up after a unit test is more dif-�cult, since several concepts being tested canshare data inserted into other concepts. Hence,when one unit test has �nished, data insertedinto other concepts might still be used by otherunit tests. Therefore, it cannot be removed yet.As an example, consider unit testing both thestudent and the course concepts. If the unit testof student is completed, data inserted into thesemester concept cannot be removed until afterthe unit test of the course concept also is com-pleted.

As a solution to the challenges above, we proposethree extensions. The solution to the �rst challengeis not to allow several users to test the same conceptat the same time. To solve the second challenge, wepropose that users cannot test concepts that dependon each other, at the same time. As a solution to thethird challenge, we propose that users must supplythe maximum running time of a unit test.

Figure 7: Illustration of the worst case idle time sce-nario with two testers.

Figure 7 shows a possible scenario with two testers,A and B, testing two dependent concepts. At time1, A initiates a unit test. A expects his unit test toterminate no later than at time 3. At time 2, A's unittest crashes. Since A's unit test has crashed and Adoes not clean up the database afterwards, B cannotstart testing before time 3 even though A's unit testis no longer executing. B has to wait unnecessarilyas the maximum execution time deviates from the ac-tual test time. The larger this deviation is the longerB has to wait unnecessarily. Thus, it is important

not to over-estimate the maximum execution time.At time 4, B's unit test is �nished.

The following section describes the proposed ex-tensions needed to the test framework and their con-sequences in detail.

5.2 The ExtensionsThe underlying table for the stack is extended withfour columns. The �rst is the user name of the userthat is testing a concept. This enables us to cleanup after a unit test from one user, while leaving unittest from other users una�ected. This implies thatif the data inserted by a unit test is shared betweenseveral unit tests, the data inserted is removed whenthe last unit test is �nished. The second column is atime stamp describing when data inserted by the setupmethod for the given concept is invalidated. This en-sures that if a test crashes and the user does not ini-tiate a cleanup afterwards, other tests must not waitinde�nitely for the crashed test to �nish. This im-plies that users must provide reasonable time stamps.This is the only additional convention needed whengoing from the single-user framework to the multi-user framework. The third column describes if theconcept itself is being tested, in the following this isdenoted as a self test. This is necessary because if itis the concept itself that is being tested, it cannot beused by any other tests, since data in the underlyingtable may be modi�ed. The last column describesif the concept is in the process of setting up, or ithas been setup already. This ensures that a test set-ting up will not be aborted on account of one of theconcepts which are being setup. For example, whentesting the participant concept the �rst row insertedon the stack will indicate that the participant conceptis being setup for a self test by this user. The timestamp will indicate the maximum time boundary ofthe test. When this test reaches the setup of the stu-dent concept another row is put on the stack. Thisrow indicates that the student concept is being setupfor a test and will prevent other users from startingtesting the student concept itself, and thereby abort-ing the participant test.

The cleanup method is extended to remove all datainserted by the current user, by calling the teardownmethod on all concepts the user has on the stack.Furthermore, any data invalidated by the time stamp,for any user, is removed as well.

The top method is extended to consider only thecurrent user.

The isActive method is modi�ed to return the statusof a concept, instead of a boolean value. This makesit possible to di�erentiate between several states ofa concept. The �rst state is that the concept is noton the stack. The second state is that the concept

10

currently is being setup. The third state is that theconcept is setup already. The last state is that theconcept is undergoing test.The push method is extended such that it takes

the user name of the given user, a time stamp, ifthe concept is being used or tested itself, and if theconcept is being setup or has been already. Theseparameters correspond to the new attributes of thestack table.The only extension needed to the pop method is

the addition of the user name, such that the stacknot necessarily pops the top element, but the topelement for that user.The setup method needs to be modi�ed, to avoid

problems with two tests wanting to setup the sameconcept for a self test, and one test trying to setupa concept another wants to test, concurrently. Whenaccessing the stack to check the current state of thegiven concept, a lock on the stack table must be es-tablished beforehand. This is because, depending onthe state of the concept, a new row is pushed on thestack, or another row is altered. This must happenwithout any other users accessing the stack. Other-wise, it could be possible for a concept to be testeditself and be used by other unit tests at the sametime. If the concept depends on other concepts, itis marked as being setup on the stack, and the ac-tual insertion of data is deferred until the dependantconcepts have been setup. If the concept does notdepend on other concepts, data is inserted immedi-ately. It should be noted that the lock on the stacktable is held while data is inserted in the concept.The teardown method needs modi�cation as well.

When tearing down, inserted data should not be re-moved if other unit tests are using the concept. Aswith the setup method, the stack is locked before ac-cessing it and until after the inserted data has beenremoved.The run method is only extended by calling the

cleanup method before starting the test.

5.3 Testing Concepts SimultaneouslyThe example sequence diagram in Figure 8 shows theprocess of two users testing the student concept andthe course concept simultaneously. The example di-agram uses a subsection of the schema shown in Fig-ure 1. The subsection of the schema only containsthe concepts student, course, and semester.The two tests are initiated by two di�erent users

that independently execute the run methods of thestudent unit test and the course unit test. First,the student unit test is initiated by when the runmethod is executed. Then, the run method of thestudent unit test cleans up the database by removingany data owned by the user and any data containing

Figure 8: Sequence diagram showing the process oftesting the student concept and the course conceptsimultaneously. The abbreviations S: student unittest, Sem: semester unit test and C: course unit test.

an invalid time stamp. After the clean up, the setupmethod of the student unit test is called. The setupmethod checks that the concept is not in use. Then,a row is pushed onto the stack, declaring that theconcept is being setup for a self test. Hereafter, thesetup of the semester unit test is called. The semestersetup checks that it is not in use and inserts the testdata into the concept. Then, a row is pushed ontothe stack. This row indicates that the �rst user hassetup the semester concept, and that it is not for aself test. Afterwards, the semester setup returns. Si-multaneously, the second unit test is initiated, andthe run method of the course unit test cleans up thedatabase and calls the setup of the course unit test.The setup method of the course unit test puts a rowonto the stack, indicating that the concept is beingsetup for a self test. Then, the setup method of thesemester unit test is called. The setup method ofthe semester unit test checks the stack. Because thesemester concept is already setup, the setup methodjust puts a row onto the stack indicating that theconcept is also used in the second unit test, and thesetup method returns. At the same time, the �rst unittest is at the setup of the student unit test. Here, therow on the stack is updated so it indicates that it issetup for a self test. Then, the row is moved to thetop of the stack and the setup method returns to therun method. The run method executes the test casesfrom the student unit test. Meanwhile, the secondtest is in the setup method of the course unit test.Here, the row on the stack is updated such that it ismoved to the top of the stack, and the setup methodreturns. The run method of the course unit test ex-ecutes the test cases of the course unit test. Afterthey have �nished, the teardown method of the courseunit test is called. The teardown methods checks thestack and removes the test data from the concept.

11

Then, the teardown method of the semester unit testis called. The teardown method of the semester unittest checks the stack. Since another unit test is usingthe concept, only the row on the stack used in thisunit test is removed. Hereafter, the teardown methodreturns. The teardown of the course unit test returns,and the second unit test is over. Simultaneously, the�rst unit test has �nished its test cases and calls theteardown method of the student unit test. The teardownmethod removes all the test data from the conceptand the row on the stack. Then, the teardown methodof the semester unit test is called. Now, no other unittests are using the semester concept, so both the rowon the stack and the test data is removed, and theteardown method returns. The teardown method of thestudent unit test returns, and the �rst unit test is�nished.

5.4 GeneralizationThe stack is locked during all access to it and all inser-tion of test data. This ensures that only one changecan be made to the system at the time. Therefore, itis impossible to set up more than one concept at thetime. Thus, two users cannot test the same conceptat the same time, since one user will always be the�rst to lock the stack and hereby be able to set up aconcept. The same argument applies when multipleusers tries to test the same concept. Furthermore,this applies for dependent concepts as well. Hence,the idea of the framework holds when scaling fromtwo to several users.

5.5 LimitationsThis section describes the limitations of the multi-user framework. First, users cannot test the sameconcept at the same time. Furthermore, dependentconcepts cannot be tested by more than one user ata time. This is because that the two unit tests wouldtry to insert the same data twice, which will cause aprimary key violation. Secondly, it is important toprovide reasonable estimates of the maximum execu-tion time of the unit tests. If the developer makes toooptimistic an estimate, other unit tests will removehis unit test data while the unit test is still execut-ing. On the other hand, a too pessimistic estimatewill cause other unit tests to wait unnecessarily incase of a unit test crash.

6 Performance Optimization of

the Test Framework

Time spent waiting for tests to complete reduces theamount of time the developer can devote to other

aspects of the development.

This section describes how the performance of thetest framework can be improved when testing largetest suites. This is done by minimizing the numberof calls to the setup and teardown methods. For ex-ample, consider a scenario where a developer wantto test the student concept and hereafter the partic-ipant concept. Following the approach described inSection 4.7, all concepts would be restored to theirinitial state, by calling the teardown method on thestudent concept, after completion of the unit test onthe student concept. Hereafter, the setup method onthe participant concept would start by loading data,�rst into the semester concept and hereafter the stu-dent concept. In this scenario it would be bene�cialif only the student concept is torn down and setup,while the semester concept remains unchanged. Inthis way, one call to a setup and one call to a teardownmethod are removed, thus the time spend setting upthe test �xture is minimized. To optimize the test-ing process it is necessary to �nd an ordering of theunit test. The following section describes how this isdone.

6.1 Ordering Unit Tests

To �nd the order in which the test suites should beexecuted, we create a directed graph representing theconcepts and their dependencies from the underlyingschema. The graph is created by querying the Oracledata dictionary. Then we check if the graph is acyclic.If this is the case, the unit test can be ordered. Other-wise, no optimizations can be performed. A directedacyclic graph (DAG) representation of the databaseexample schema from Figure 1 is shown in Figure 9.The concepts are vertices and the foreign keys be-tween them are edges.

Figure 9: Directed acyclic graph representing the uni-versity schema of Figure 1.

Hereafter, a topological sort [6] is performed onthis graph. Topological sort only works on DAGs,therefore, we ignore self-references as the referencefrom teacher to teacher. The result of a topologicalsort on Figure 9 could look as depicted in Figure 10.

12

Figure 10: Topological sort of Figure 9.

The topological sort de�nes the order in which theunit tests in the test suite should be executed. Forinstance, we see from Figure 10 that the semesterconcept is the �rst1 concept to be tested.

Furthermore, the teardown methods are simpli�edsuch that they only clean up the concept on whichthey are implemented. For instance, the teardownmethod on the student concept cleans up the studentinterface and terminates. In the former implemen-tation the teardown method of the student concept,would call the teardown method on the semester con-cept.

As mentioned, the semester concept is the �rst tobe tested. On completion of the test, the teardownmethod cleans up the concept. Hereafter, the o�ceconcept is to be tested. Upon completion of the test,the o�ce concept is cleaned up. Then, the studentconcept is to be tested. The setup method automati-cally sets up the semester concept. The test on stu-dent concept is then performed. Hereafter, the mod-i�ed teardown method cleans the student concept, butleaves the semester concept active. The teacher con-cept requires the o�ce concept, thus this is setup.Upon completion of the teacher unit test, the o�ceconcept remains setup, and only the teacher conceptis torn down. Then, the course concept is tested. Thecourse concept sets up the teacher semester. The of-�ce semester is already active, and is therefore notsetup again. Finally, the test continues by settingup the participant concept. The setup method on theparticipant concept sets up the student concept. Notethat the semester concept already was setup, fromwhen the student concept began testing.

The task of ordering the unit test is handled by thetest framework without user involvement.

Please note, that when a concept is tested, the unittest data is setup, test cases are executed, and thedata is torn down again. When the test data forthis concept is used by other unit tests, the test datais setup but teardown is deferred until completion ofthe test suite. Therefore, subsequent calls to the setupmethod simply returns because the concept alreadyhas been setup.

1Please note that a topological sort on the universityschema could also yield a result where the o�ce concept isthe �rst to be tested.

7 The utPLSQL Framework

In this section the popular unit test framework ut-PLSQL is examined. First, it is described how asingle concept is tested using the utPLSQL frame-work. Hereafter, an example of testing a dependentconcept is presented. Then, the test approach takenby utPLSQL is compared to our approach.

7.1 utPLSQLThe utPLSQL framework is a test framework for theOracle PL/SQL programming language. It is mod-elled on the JUnit framework following the extremeprogramming approach. This means that all testcases and unit tests should be independent.For the utPLSQL framework to execute a test case

it has to conform to certain requirements. For eachpackage the developer wish to test, a separate testpackage has to be developed. The test package shouldcontain a setup and a teardown method. These are,similarly to the approach proposed in this article,used to create and remove data structures used inthe tests. Furthermore, the test package should con-tain all test cases for the methods to be tested. Eachtest case consists of a call to one of the functions orprocedures to be tested and a call to an assert pack-age. The assert package is used to ensure that certainconditions hold. If all assertions for a test packagehold the test cases are successful.

7.1.1 Testing an Independent Concept

This section describes how to perform the test fromSection 4.7.1 using the utPLSQL framework. Thetest is conducted on a package wrapping the o�cetable. The method to be tested is the insert method.The PL/SQL code for the insert method is shown inListing 8.

1 procedure insert(rec o�ce_rec ) is2 begin3 insert into o�ce4 values(rec. building, rec. room, rec.size )5 end;

Listing 8: The insert method.

The insert method takes a record as input. The of-�ce_rec parameter contains all attributes in the o�cetable. Lines 3 to 4 insert a new row into the o�cetable.As the insert method does not return any value,

it is harder to check the correctness of the method.Therefore, we use an auxiliary method to check if arow has been inserted. The exist method is listed inListing 9.

1 function exist (rec o�ce_rec )2 return boolean is3 b_exist boolean;

13

4 count_number integer;5 begin6 select count(∗)7 into count_number8 from o�ce9 where building = rec.building

10 and room = rec.room;11 if ( count_number > 0) then12 b_exist := true;13 else14 b_exist := false ;15 end if ;16 return b_exist;17 end;

Listing 9: The exist method.

Lines 6 to 10 count the number of records thatmatches the input given to the method. If this num-ber is greater than 0, the method returns true, oth-erwise false is returned.

To test the insert method we create a test packageas shown in Listing 10.

1 create or replace package body ut_insert is2 procedure ut_setup is3 begin null ; end;45 procedure ut_teardown is6 begin7 execute immediate8 ' delete from o�ce9 where room = 'B2'

10 and building = '110a''11 end;1213 procedure ut_insert is14 begin15 insert (' B2',' 110a',15);1617 ut.Assert.eq(18 exist (' B2',' 110',15),19 TRUE20 );21 end;22 end;

Listing 10: A utPLSQL unit test package for testingthe insert method.

All utPLSQL test packages have a setup and ateardown method. However, in this example no setupis needed. Therefore, the body of the setup methodis empty (null). The teardown method is automati-cally called after completion of the test. The teardownmethod removes the row inserted by the ut_insertmethod. In line 15 a row is inserted into the o�cetable. In line 17 the eq method, supplied by the ut-PLSQL framework, is used to validate if the correctrow is inserted into the o�ce table. In line 18 the existmethod, that was described in Listing 9, is called tocheck if the row inserted in line 15 exist. In line 19the expected return value from the exist method isgiven. If the exist method returns true the test caseis considered a success, otherwise it is a failure. Thetest completes by calling the teardown method thatcleans up the o�ce table.

7.1.2 Testing connected concepts

This section will show how to test a concept that isconnected to another concept through foreign keys.As in Section 4.7.2 the subject of the test is theteacher table. As in the previous section, the methodto be tested is the insert method2. Recall from Figure1 that the teacher table is connected to the o�ce ta-ble. In other words, we cannot insert a row into theteacher table without the o�ce table is populated.The test package is shown in Listing 11.

1 create or replace package body ut_insert is2 procedure ut_setup is3 begin4 execute immediate5 ' insert into o�ce6 value(' B2',' 110a',15) ' ;7 end;89 procedure ut_teardown is

10 begin11 execute immediate12 ' delete from teacher13 where tid = 10';1415 execute immediate16 ' delete from o�ce17 where room = 'B2'18 and building = '110a''19 end;2021 procedure ut_insert is22 begin23 insert (10, ' Hans','Jensen',24 null , ' B2',' 110a');2526 ut.Assert.eq(27 exist (10, ' Hans','Jensen',28 null , ' B2',' 110a'),29 TRUE30 );31 end;32 end;

Listing 11: A utPLSQL unit test package for testingthe teacher concept.

As opposed to the previous section, the setupmethod is not empty. In lines 2-7, a row is insertedinto the o�ce table. The primary key of this rowwill be used when inserting into the teacher table. Inline 9, the teardown method is declared. In line 12,the row inserted into the teacher concept is removed.Lines 15-18 cleans up the o�ce table, by deletingthe row inserted by the setup method. Please notethat it is important to remove the inserted rows in aspeci�c order, to avoid breaking referential integrityconstraints. The ut_insert method in lines 21-30 in-serts a row into the teacher table using the primarykey from the row inserted into the o�ce table. Asin the previous section, the Assert package is used tovalidate the result from the exist method.

2The code for the insert and the exist methods are thesame as in Section 7.1.1, except that the table is no longero�ce but teacher. Thus, the attributes have changed.

14

7.1.3 Comparison

This section provides a series of guidelines describ-ing when it is bene�cial to use the utPLSQL frame-work, and when the approach described in this articleshould be preferred.As shown in Listings 10 and 11 the setup and

teardown methods quickly become larger. Imagine,a unit test conducted on the participant table fromFigure 1. The test �xture quickly increases when atable is dependent on more than one other table. Us-ing utPLSQL, the developer has to create the test�xture in the setup method by copying and pastingcode from existing unit tests. Similarly, the teardownmethod have to be constructed manually. This isnot the case in the approach proposed in this article.Section 4.6 described how the the presented frame-work could reuse setup and teardown methods. Thus,the workload for the developer is minimized. There-fore, it is bene�cial to use the approach proposed inthis article, when testing connected concepts. Theframework proposed in this article gives a partiallyautomatically creation of setup and teardown meth-ods. However, the utPLSQL framework is well-suitedwhen a limited test �xture is needed. Especially, theassert package allows the developer to easily test thecorrectness of programs.

8 Oracle Speci�c Functionality

This section describes how to use Oracle speci�c func-tionality that could aid to achieve the goals of thedatabase unit test framework. This functionality in-cludes the �ashback query and workspace manage-ment. Although the design criteria specify a vendorindependent unit test framework, this section looksat what could have been exploited in Oracle DBMSin order to make the implementation easier.

8.1 The Flashback QueryThe Oracle DBMS o�ers the possibility to query dataat a point in time. By default, database operationsquery the most recently committed data. However,the �ashback query can be used to query data froma previous database state. There are two ways toachieve this, either by using a special system changenumber or by specifying a point in time. To queryhistorical data could prove bene�cial in the unit testframework as it allows for the database to return toits state before the unit tests were executed. Us-ing the �ashback functionality does not eliminate theneed for creating the test �xture. However, it wouldmake the teardown method unnessesary. This �ash-back functionality is well-suited when the unit testframework is only considered to be in a single-user

environment. If the testing environment is a multi-user environment the framework cannot �ashback intime when one of the testers is �nished testing, astest �xture may be used by another test.

8.2 Workspace ManagementWorkspace Management allows for multiple versionsof rows in user-tables. The versioning infrastruc-ture is not visible to the users of the database. SQLselect, insert, update, and delete statements continueto operate as normal. New row versions are logicallygroup in each workspace until these versions are ex-plicitly merged into production data or simply dis-carded. Users in a workspace always see a transac-tionally consistent view of the entire database, i.e.,they see changes made in their current workspaceplus the rest of the data in the database. This dataeither existed when the workspace was created orwhen the workspace was most recently refreshed withchanges from the parent workspace. The workspacemanagement could be exploited in a multi-user unittest framework environment. By allowing each userto test in their own workspace, i.e., they have theirown version of the tables or rows they want to test.Their testing activity does not a�ect any other con-current tester. By performing the unit tests in a vir-tual environment without committing the changes tothe parent workspace o�ers maximum concurrency asit isolates the multiple testers completely.

9 Implementation

This section describes core implementation issues ofthe test framework. It should be noted that this sec-tion only covers the implementation of the single userversion of the framework, because the changes to thesetup and teardown methods are limited, and have al-ready been described in Section 5. The frameworkis implemented on the Oracle 9i platform. Initially,the generation of setup and teardown methods are de-scribed. Hereafter, the implementation of the runmethod is described. By auto generating the setupand teardown methods, the framework becomes moreresistant to changes in the database schema. For ex-ample, should a new concept be added to the schema,the developer simply generates a new set of setup andteardown methods. Thus, the workload required to ac-commodate changes is reduced. Finally, an approachfor coupling our framework and the utPLSQL frame-work is proposed.

9.1 The setup and teardown methodsThe generator for the setup and teardown methodsis written in C#. Furthermore, the object-oriented

15

framework to represent PL/SQL constructs presentedin [5] is used to output PL/SQL source code. As de-scribed in Section 4.7 the setup and teardown methodsautomatically call other setup and teardown methods asneeded. The dependencies between concepts are de-termined using the RDBMS meta data. It is thereforenecessary to have access to information about foreignkeys. As an example, to �nd the dependencies of theparticipant concept from Figure 1 on the Oracle 9iplatform, the query in Listing 12 is executed.

1 select distinct2 ucc.table_name as table_name,3 uc2.table_name as references_table4 from5 user_cons_columns ucc6 inner join7 user_constraints uc8 on9 ucc.constraint_name = uc.constraint_name

10 inner join11 user_constraints uc212 on13 uc.r_constraint_name = uc2.constraint_name14 where15 uc.constraint_type = 'R' −− foreign key16 and ucc.table_name <> uc2.table_name17 and ucc.table_name = 'PARTICIPANT';

Listing 12: Finding dependencies for the participantconcept.

The distinct clause in line 1 ensures multiple ref-erences to the same concept are ignored. With-out the distinct clause the teacher concept would re-turn two references, one for the building attributeand one for the room attributes. Lines 2 and3 project the attributes needed. Lines 4 to 14join the tables needed to �nd the referenced con-cepts. The user_cons_columns view contains in-formation of the constraints on a given attribute.The user_constraints view is used to �nd the at-tributes a given constraint references. Line 15 statesthat only constraints of type 'R' are interesting. Inthe Oracle data dictionary 'R' indicates that a con-straint is a foreign key. Recall from Figure 1 thatthe teacher concept has a self-reference. Line 16 re-moves self-references in the result. If self-referenceswere not removed, the setup method on the teacherconcept would contain a call to the setup method onthe teacher concept. This would generate a recursivemethod that never terminates. Line 17 speci�es forwhich concept the setup and teardown are being gen-erated.Note that the query �nds dependencies between

tables and not between unit tests. It could also bedone between unit tests. The result of the query inListing 12 is the concepts student and teacher. Thus,the setup method for the participant concept looks asshown in Listing 13.

1 procedure setup(self_test) is2 begin

3 if (not stack. is_active(' participant ' )) then4 student.setup(false );5 teacher. setup(false );6 stack. push('participant ' );78 if (not self_test ) then9 −− insert data here

10 end if ;11 end if ;12 end;

Listing 13: The auto generated setup method of theparticipant concept.

In line 3, it is checked whether the participant con-cept is already on the persistent stack. If the partic-ipant concept is not active, the student and teacherconcept are setup. In line 6, the participant conceptis pushed onto the stack. In line 9, the developershould make a call to the methods that inserts datausing the public constants.Similarly, the teardown method is auto generated

such that proper cleanup is performed.

9.2 The run methodAs described in Section 4.4 the tester calls the runmethod to initiate a test. Thus, the task of the runmethod is to setup up the test �xture, execute thetest methods, and perform the necessary clean up.For example, to test the participant concept, the runmethod calls the setup method on the participant con-cept. Hereafter, all test methods on the participantconcept are executed. Finally, the teardown methodon the participant concept cleans the database. Thissection describes how the run method locates all testmethods on a concept.As with the setup and teardown methods, access to

meta data is also required for the run method to �ndtest methods. On the Oracle 9i platform the viewsuser_procedures and user_source are used. Theuser_procedures view contains all the active meth-ods. Unfortunately, the methods are sorted alpha-betically according to the method names. As testmethods are dependent it is necessary to join theuser_procedure with the user_source in order to pre-serve the same ordering of methods as in the sourcecode. The join between the to views is shown in List-ing 14.

1 select procedure_name from (2 select distinct3 up.procedure_name,4 us. line5 from6 user_procedures up7 inner join8 user_source us9 on

10 (ltrim(lower(us. text)) like ' function %'11 or ltrim(lower(us. text)) like ' procedure %')12 and (lower(us.text) like13 ' %' || lower(up.procedure_name) || ' %'14 or lower(us. text) like

16

15 ' %' || lower(up.procedure_name) || '(%'16 or lower(us. text) like17 ' %' || lower(up.procedure_name) || ';%')18 where19 up.object_name = sought_package_name20 and up.object_name = us.name21 and us.type = PACKAGE22 order by us.line );

Listing 14: The SQL statement used to �nd all activeprocedures and functions in a package.

Lines 9-17 contain the join clause. Two things needto be satis�ed to ful�ll the join clause. First, a row inthe user_source view has to contain either the wordfunction or the word procedure. This is checked inlines 10 and 11. Second, the method name from theuser_procedures view has to match the name foundin the user_source view. The procedure name shouldbe followed by a white space, an opening parenthe-sis or a semicolon. Lines 19 and 20, speci�es whichPL/SQL packages the methods should be found in.Line 21 ensures that only the package header is con-sidered.

9.3 Integration with utPLSQL

As described in the previous section, the utPLSQLframework provides an assert library that helps thedeveloper in testing his programs. However, as de-scribed the utPLSQL requires the developer to copyand paste source code between setup and teardownmethods when testing dependent concepts. There-fore, it is interesting to combine the two frameworksto achieve the best from both worlds, namely theassert library and the large user community from theutPLSQL world, and the extensive reuse of code fromour world. This section describes how the two frame-works can, with very little e�ort, be combined.

As described in Section 7.1.3 the primary di�er-ence between the two frameworks is the setup andteardown methods. The overall strategy for combin-ing the frameworks is to modify our framework suchthat utPLSQL follows our test strategy, i.e., reuse thesetup and teardown methods instead of copying them.

First, the generator from Section 9 is modi�ed suchthat the source code produced follows the conven-tions of the utPLSQL framework. This means thatprocedure names should have the ut_ pre�x. The de-veloper will now use the generated setup and teardownmethods as described in 4.4. This implies that thedeveloper should follow all conventions described inSection 4.5 except for the naming conventions, thatnow should conform to the utPLSQL naming con-ventions. Thus, the public constants must be pub-lished. Furthermore, the developer should rememberthat test cases are dependent. Moreover, the stackand the methods associated with it must be included.

Hereafter, the developer follows the standard ut-PLSQL way of writing test cases using the assert li-brary. Again the developer should recall the con-ventions from Section 4.5, as the order in which testsuites and test cases are executed is now important.With this slight modi�cation of the generator, the

two frameworks have been combined to achieve thebest of both.

10 Performance Analysis

This section describes the performance improvementgained by using the test methods proposed in this ar-ticle. Initially, the di�erent approaches to testing aredescribed. Hereafter, the cost of testing the univer-sity schema is evaluated.

10.1 The simple approachThe simple approach uses the same bottom-up ap-proach as the approach proposed in this article. Thedi�erence is that not only the setup and teardown meth-ods are executed, but also the actual test cases. Forexample, to test the semester concept from Figure1, the test is setup, the test cases are executed, and�nally the test �xture is torn down. If the studentconcept is to be tested, the entire semester test, in-cluding the setup and teardown methods, are executed.Hereafter, the student concept is setup, tested andtorn down. For each concept to be tested, all con-cepts that the current concept depends upon are alsotested. Thus, to test the entire university schemafrom Figure 1 the tests in Table 1 are performed.

Test Dependencies

SemesterO�ceStudent SemesterTeacher O�ceCourse Semester, O�ce, TeacherParticipant Semester, O�ce, Teacher, Course,

Student

Table 1: The tests of the university schema and theirdependencies.

The �rst column in Table 1 shows the concept to betested. The second column shows the dependenciesof that test. For example, to test the teacher concept,the o�ce concept is also tested.

10.2 Our approachThe approach proposed in this article resembles theapproach in the previous section. The di�erence isthat the test cases are not executed for depending

17

concepts when they are setup. Only the setup andteardown methods are reused. For example, to testthe student concept the setup method on the semesteris used. The test cases for the student concept isexecuted. Then the student concept is torn down.Finally, the semester is torn down as well. Thus,compared to the simple approach, the test cases fordependant concepts are not executed. However, thesame number of setup and teardown methods is used.

10.3 Using topological sortAs described in Section 6 our approach is further op-timized by specifying the order of the unit tests andby modifying the teardown methods. Table 2 showsthe steps performed in testing the university schema.

Test Steps performed

Semester sem.setup, sem.test, sem.teardownO�ce o�.setup, o�.test, o�.teardownStudent stu.setup, sem.setup,

stu.test, stu.teardownTeacher tea.setup, o�.setup, tea.test,

tea.teardownCourse cou.setup, tea.setup, o�.setup,

sem.setup, cou.test, cou.teardownParticipant par.setup, cou.setup, tea.setup,

o�.setup, stud.setup, sem.setup,par.test, par.teardown

Clean up par.teardown, cou.teardown,tea.teardown, stu.teardown,o�.teardown, sem.teardown

Table 2: The steps taken by the topological approach.

10.4 ComparisonThis section compares the performance of the threeapproaches presented. The performance is measuredaccording to the number of calls to the setup andteardown methods, and to the number of unit testsperformed when the entire university schema is to betested. The number of unit tests refers to the numberof times all the test cases for a concept is executed.Table 3 show a comparison of the three approaches.

# setups # teardowns # testsSimple 16 16 16Our appr. 16 16 6Topologic 15 12 6

Table 3: Number of methods call when testing theuniversity example schema

From Table 3 we see that the number of times eachconcept is tested drops dramatically from the simpleapproach to the two alternative approaches. As the

test cases typically carry more lines of code than thesetup and teardown methods, this reduction is mostlikely causing a bigger performance gain than the re-duction of setup and teardown methods. Note that ex-ecuting six tests are minimal. Using the topologi-cal sort approach, it is possible to cut o� one setupmethod and four teardown methods and thus, furtherimproving the performance of the test framework.

The reduction of one setup method is caused whenthe setup method from the participant concept is ex-ecuted. The setup method of the teacher concept re-turns immediately and skips the setup of the o�ceconcept, as the teacher concept already has been setup. Hence, the o�ce concept must have been setup already as well. The di�erence between the num-ber of teardown methods executed is caused by thedeferral of teardown in the topological approach. Ingeneral, the more relations between the concepts inthe database schema the greater the bene�t is fromusing topological sorting.

11 Conclusion

In this paper, we presented and discussed the issuesof designing and implementing a unit test framework.Alternative approaches for testing database applica-tions require that all test cases should be indepen-dent. In this article we argue that it is an advantagefor test cases to depend on each other.

The contribution of this paper is a design and animplementation of a unit test framework. The frame-work supports multiple users testing simultaneouslywhen the conventions of the framework are followed.Compared to existing frameworks, the approach pro-posed in this article minimizes the test �xture. Theminimal test �xture is achieved by only having setupand teardown code in a single place, whereas exist-ing frameworks requires the developer to copy sourcecode between test packages. Furthermore, becauseexisting frameworks require that test cases must beindependent, they must perform setup and teardownbetween every test case. In our framework, this isonly done once per unit test. Moreover, the test �x-ture is further minimized by allowing concepts to re-main set up through several unit tests.

The framework gives the possibility of automaticcleanup of the database, in case of a system crash ora malfunctioning test case.

The ideas presented in this article, are not onlyapplicable in a database setting. It was shown howreferences between objects using collections resemblethe foreign keys in databases. Thus, the ideas of theframework are also applicable to programs using as-sociation or aggregation.

With slight modi�cations of the generator pre-

18

sented, we are able to apply the ideas from this articleinto the utPLSQL framework and hereby achieve thebest features from both frameworks.In the following, future work is discussed. A spe-

ci�c problem is that some tables have surrogate pri-mary keys that automatically increment. This posesa problem because the next key value is unknown atcompile-time. This makes it more di�cult for thetest programmer to satisfy the constraints given bythe test framework, namely that usable primary keysmust be available at compile-time.At this moment, the framework does not allow test-

ing of real-time constraints. Imagine a conveyor beltin an airport transporting luggage from passengersto their planes. Here, it might be desirable to specifyand test a constraint, stating that a piece of luggageshould reach the airplane no later than one minuteafter the passenger checked it in.

Acknowledgements

For providing us with references to related work, wethank Brian Nielsen of CISS, Aalborg University.

References

[1] Dbunit. http://dbunit.sourceforge.net/. As of31.05.2004.

[2] Kent Beck. Extreme Programming Explained -Embrace Change. Addison-Wesley, 2000.

[3] Kent Beck and Erich Gamma. Junit cookbook.http://junit.sourceforge.net/doc/cookbook/cookbook.htm. As of 31.05.2004.

[4] David Chays, Saikat Dan, Phyllis G. Frankl,Filippos I. Vokolos, and Elaine J. Weber. Aframework for testing database applications. InProceedings of the International Symposium onSoftware Testing and Analysis , pages 147�157,2000.

[5] Claus Abildgaard Christensen, SteenGundersborg, Kristian de Linde, andJacob Richard Thornber. A generic andportable database api.http://www.cs.aau.dk/∼kdl/master/, 2004.Aalborg University.

[6] Thomas H. Cormen, Charles E. Leiserson,Ronald L. Rivest, and Cli�ord Stein.Introduction to Algorithms . MIT press, 2001.

[7] Haifa University Software Engineering Course.http://cs.haifa.ac.il/courses/softe/JUnit.ppt.As of 31.05.2004.

[8] Webopedia Regression Testing De�nition.http://www.webopedia.com/TERM/R/regression_testing.html.As of 31.05.2004.

[9] Steven Feuerstein. utPLSQL project.http://utplsql.sourceforge.net/. As of31.05.2004.

[10] Ramzi A. Haraty, Nash'at Mansour, and BasselDaou. Regression testing of databaseapplications. In Proceedings of the 2001 ACMSAC, pages 285�289, 2001.

[11] Paul C. Jorgensen. Software Testing - ACraftsman's Approach . CRC Press LLC, secondedition, 2002.

[12] JUnit. http://www.junit.org. As of 31.05.2004.

[13] Willian E. Lewis. Software Testing andContinuous Quality Improvement . Auerbach,2000.

[14] Oracle. Catalog Views / Data DictionaryViews. http://otn.oracle.com/pls/db901/db901.catalog_views?remark=homepage. As of31.05.2004.

[15] Oracle. Managing Viws, Sequences, andSynonyms.http://download-west.oracle.com/docs/cd/B10501_01/server.920/a96521/views.htm#581. As of 31.05.2004.

[16] William Perry. E�ective Methods for SoftwareTesting. John Wiley & Sons, Inc., 1995.

[17] Don Wells. Extreme programming: A gentleintroduction.http://www.extremeprogramming.org/people.html.As of 31.05.2004.

19