1

An Exploration of Errors in Web-based Applications in the Contextof Web-based Application Testing

PhD Proposal

Kinga Dobolyi

May 2009

2

The shopping cart

3

The shopping cart

4

The shopping cart

5

What is going on

• Problem: faults in web-based applications cause losses of revenue, and they are hard to test

• Approach: study errors in web-based applications in the context of web testing

• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

6

Outline

• Introduction and motivation

• Thesis statement

• Background

• Goals and approaches

• Preliminary work

• Expected contributions

7

Motivation

• Testing of web-based applications in particular deserves further examination due to economic considerations:– Monetary throughput: Backbone of e-

commerce and communications businesses– Customers: low customer loyalty– Development: Companies are choosing not to

test due to resource constraints

8

Motivation: e-commerce

• Internet usage: 73% of people in the US in 2008– Browsers are dominant application– $204 billion in Internet retail sales annually

• Global online B2B transactions total several $trillions annually

• One hour of downtime at Amazon.com cost $1.5 million dollars

• 70% of major online sites exhibit user-visible failures

9

Motivation: customers

• Customer loyalty is notoriously low– Determined by the usability of the application

[Offutt 2002]– Freedom and options

10

Motivation: customers

• Lesson learned: web-based applications need to be well-designed and well tested

• Are they?

11

Motivation: development

• Technology

challenges:– Heterogeneous, opaque

components – Dynamic page content generation– Persistent state operated upon by

concurrent, global users around the clock

12

Motivation: development

• Web-based applications are often not tested– Enormous pressure to change

• Short delivery times, high developer turnover rates, and quickly evolving user needs

– No formal process model

13

Motivation: summary

• Problem: faults in web-based applications cause losses of revenue, and they are hard to test

• Approach: study errors in web-based applications in the context of web application testing

• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

14

Thesis statement

• Hypothesis: web-based applications have special properties that can be exploited to build tools and models that improve the current state of web application testing and development:– Tend to fail in predictable and similar ways– Human centric definition of acceptability

15

Thesis statement

• Problem: faults in web-based applications cause losses of revenue, and they are hard to test

• Approach: study errors in web-based applications in the context of web testing

• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

16

Background: testing techniques

• Non-functional (static) validation– Server load testing– Link testing– HTML/spelling validation

• Modeling approaches

• Capture-replay– User session-based testing

17

Background: oracles

• Oracles (oracle-comparator)

1 < <P>The same table could be indented.

2 < <TABLE border="1">

3 ---

4 > <p>The same table could be indented.</p>

5 > <table border="1" summary="">

– False positives from

diff-like tools– Want precise comparators

18

Background: automation

• Automation– Test case generation: VeriWeb, PHP– Test case replay

• URL + post-data

– Failure detection

19

Background: metrics

• How do we measure success?– Code coverage– Fault seeding

• Human• Automatic

– Cost

• How do we know these are indicative of the real world?

20

Background: fault definition• Defining an error:

– “the inability to obtain and deliver information, such as documents or computational results, requested by webusers.” [Ma & Tian 2003]

• Fault taxonomies– Figure from

[Marchetto et al2007]

21

Background: challenges

• Functional validation remains a challenge– Regression testing should be more precise

and automatic

• We do not know if test suite efficacy metrics are indicative of the real world– We should examine the severity of uncovered

faults

22

Goals and approaches

• Problem: faults in web-based applications cause losses of revenue, and they are hard to test

• Approach: study errors in web-based applications in the context of web testing

• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

23

Goals and approaches

• I propose to:– Model errors in web-based applications

• Identify them more accurately • Automate the oracle-comparator process

– Make web testing more cost-effective• Devise a model of fault severity that will guide test

case design, selection, and prioritization• Validate or refute the current underlying

assumption that all faults are equally severe in fault-based testing

24

Goals and approaches: Goals

• Reduce the cost of regression testing web-based applications – Use special structure of web-based application output

to precisely identify errors

• Automate web-based application regression testing– Unrelated web-based applications tend to fail in

similar ways

• Understand customer-perceived severities of web application errors.

25

Goals and approaches: Goals

• Formally ground the current state of industrial practice– Validate or refute fault injection as a standard for

measuring web application test suite quality

• Understand how to avoid high-severity faults during web application design and development

• Reduce the cost of regression testing web applications by exposing high-severity faults– Test case design, selection, and prioritization (test

suite reduction)

26

Goals and approaches: Outline

27

Goals and approaches: Step 1 – oracle-comparator

• Construct a precise oracle-comparator that uses the tree-structured nature of XML/HTML output and other features– Model errors on a per-project basis– Semantic distance metric to reduce false

positives

28

Goals and approaches: Step 2 – automated oracle-comparator

• Exploit the similar way in which web applications fail to avoid the need for human annotations in training a precise oracle-comparator– Train a precise oracle-comparator on data

from other, unrelated web applications– Use fault injection to improve the results when

necessary

29

Goals and approaches: Step 3 – human study

• Conduct a human study of real-world fault severity to identify a model of fault severity– Severities different than self-reported in bug

repositories– Screenshots of current-next idiom– Also survey developers

30

Goals and approaches: Step 4 – fault seeding validation

• Compare the severities of real-world faults to seeded faults using human data (validate fault seeding)– The severities of seeded errors have uniform

distributions?– The severity distribution of seeded errors

matches the distribution of real-world errors, according to the results of the survey from Step 3?

31

Goals and approaches: Step 5 – software engineering guidelines

• Identify underlying technologies and methodologies that correlate with high-severity faults– As an alternative to testing– Tie high severity errors to underlying code,

components, programming languages, and software engineering practices

32

Goals and approaches: Step 6 – testing techniques

• Identify testing techniques to maximize return on investment by targeting high-severity faults– Introduce a new metric for the (web

application) test suite reduction research community

33

Preliminary Work: Step 1

• Step 1: Construct a precise oracle-comparator using tree structured XML/HTML output and other features– Multiple versions of

10 open-source

benchmarks– 7154 pairs of oracle-

testcase output, 919

of which labeled as

“possibly an error”

34

Preliminary Work: Step 1

• Evaluation: F-measure (precision and recall) using our model

35

Preliminary Work: Step 1• Longitudinal study to measure effort saved

– Calculate cost of looking : cost of missing– Low ratio means we are saving effort

36

Preliminary Work: Step 2

• Step 2: Exploit similarities in web application failures to avoid human annotations when training a precise oracle-comparator– Same setup as Step 1– Use existing, annotated pairs of test-oracle

output from unrelated applications to train a comparator for the application at test

37

Preliminary Work: Step 2

• Evaluation: measure precision and recall

38

Preliminary Work: Step 2

• Use fault seeding to introduce project-specific faults into the training data set

39

Preliminary Work: Step 3

• Step 3: Model real-world fault severity based on a human study.– Collect 400 real-world faults

• Evaluation:

have subjects

use the model

to classify

faults

40

Preliminary Work: Step 4

• Step 4: Compare the severities of real-world faults to seeded faults using human data.– Test same human subjects with 200 + 200

manually-injected and automatically-injected faults

– Conduct a survey

of developers for

fault severity

41

Preliminary Work: Step 5

• Step 5: Identify technologies and methodologies that correlate with high-severity faults

• Do high severities correlate with:– Programming Language– Level in three-tier architecture– COTS component– User error (usability issue)– Type of error (business logic, resource allocation, syntax error,

etc)– Fault taxonomies (existing)– Surface features of visible output: white screens, stack traces,

misspellings, formatting errors

• Evaluation: developer survey (time permitting)

42

Preliminary Work: Step 6

• Step 6: Identify testing techniques to target high-severity faults– Targets testing– Assign a testcase a severity rating a priori

• Evaluation: compare the severity of faults exposed with our technique versus other test suite reduction approaches

43

Expected Contributions

• Problem: faults in web-based applications cause losses of revenue, and they are hard to test

• Approach: study errors in web-based applications in the context of web testing

• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

44

Expected Contributions

• Reduce the cost of regression testing by constructing a precise oracle-comparator

• Develop a model of customer-perceived severities of web application faults

• Validate or refute fault injection as a standard for measuring web application test suite quality

• Propose new software engineering guidelines for web application development and testing

45

Questions?

46

Original Contributions

• Fault Severity Model– Severity has not been studied in this domain, and

customers are an integral part of these applications– Providing a new metric to the research community– Validate/refute fault seeding

• Precise-oracle comparators– First to use different versions of benchmarks– Can be completely automated– XML and HTML

47

Expected Impact

• Fault Severity Model– Can be applied to testing techniques in this field to

make them financially feasible for developers– Change the way in which test suite efficacy is

measured– Potentially impact web site design as usability issues

may become more evident• Precise oracle-comparator

– Automation makes it much more feasible for adoption than existing techniques

– Potentially allow companies to conduct regression testing if they were not testing beforehand

48

Timeline

• Steps 1 and 2: precise comparators– Completed, 2 papers under submission

• Steps 3 and 4: human study– Data collection completed, analysis under way for submission of

1 paper– Expected completion by September

• Step 5: software engineering guidelines– Expected completion by October– Expected 0.5 paper

• Step 6: testing according to fault model– Expected completion by February– Expected 1 paper