Quality Assessment I
Peter Dologdolog [at] cs [dot] aau [dot] dk2.2.05Intelligent Web and Information SystemsOctober 28, 2010
This Lecture
Performance and Reliability TestingData Flow Testing (White Box)Terminology in the quality assessment
2Peter Dolog, Web Engineering 2010, Quality Assesment I
Performance and Reliability Testing
Functional Steps
4Peter Dolog, Web Engineering 2010, Quality Assesment I
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Thoughput with technologies
5Peter Dolog, Web Engineering 2010, Quality Assesment I
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Typical tasks
Representation of workload model definitionTraffic generationData Collection and analysis
6Peter Dolog, Web Engineering 2010, Quality Assesment I
Representation of Workload Model
File lists with access frequences and analytical distributiongood enough for static resources, less flexible
File tracesDepends on representativeness of log
Finate State AutomataFlexible and yet suitable for complex patternsNeeds to be defined by hands which can be error prone and labourous
7Peter Dolog, Web Engineering 2010, Quality Assesment I
Traffic Generation
Emulates user interactionNeeds to generate and take into account cookies,
session management, and security (e.g. HTTPS)Needs to take into account network delaysGiven workflow model is taken as an input for traffic
generation
8Peter Dolog, Web Engineering 2010, Quality Assesment I
Data Collection and Analysis
You have to choose a metric and granularity samples
Mean, minimum and maximum may not be sufficient for modelling performance of advanced applications
Advanced statistics is needed thenSubject of white box or black box strategies:
resource vs. system based metrics
9Peter Dolog, Web Engineering 2010, Quality Assesment I
Black Box Testing
The whole web system as seen from outsideServed pages, hits, bytes per second ->
administrator to check the architectureResponse time: elapsed time between user click
until arrival of last object from the request -> interesting for users (not so much throughput)
Response time higher than 30seconds is unacceptable for users according to IBM
Less then 20 seconds is considered adequate
10Peter Dolog, Web Engineering 2010, Quality Assesment I
Testing Approach
Define several workload models as patterns for youVerify whether the performance stays stableIf not, analyse what could have caused problemsWeb usage logs could give you a hint of a
bottelnecksWeb application needs to operate with safe limits
11Peter Dolog, Web Engineering 2010, Quality Assesment I
Response time as a function of traffic volume
12Peter Dolog, Web Engineering 2010, Quality Assesment I
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
White Box testing
Evaluates internal web applications components for different client request rates (usually maximum)
You need to run it after black box in case of performane problems
Reveals what causes the problemMetrics look at resources such as CPU utilization,
network, central memory, file and socket descriptors, process table entries, …
Metrics look also at the functions such as function call frequency, percentage of time spent by program in each main function …
13Peter Dolog, Web Engineering 2010, Quality Assesment I
Performance improvement
See last lecture
14Peter Dolog, Web Engineering 2010, Quality Assesment I
Example workload models (400 concurrent users)
15Peter Dolog, Web Engineering 2010, Quality Assesment I
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
System Design
HTML, PhP (4), MySQL (4), Apache (2.0)Linux, 2.4GHz, 1GB main memory, 80GB disks. Fast
ethernet
16Peter Dolog, Web Engineering 2010, Quality Assesment I
Testing Loop
17Peter Dolog, Web Engineering 2010, Quality Assesment I
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Results
18Peter Dolog, Web Engineering 2010, Quality Assesment I
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Additional response time testing and stressing
19Peter Dolog, Web Engineering 2010, Quality Assesment I
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
White box testing (2)
20Peter Dolog, Web Engineering 2010, Quality Assesment I
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
White Box Testing
21Peter Dolog, Web Engineering 2010, Quality Assesment I
© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.
Data Flow White Box Testing
Control Flow Graph
ORD = (V, E)V … Set of nodes representing objectsE … VxV relationships betwen objects
Objects: client pages, server pages, componentsObjects have attributes: variables or document elementsObjects have operations: functions in scripting or prog.
Language
Graph can be defined recursively
Relationships: inheritance, aggregation, association, request, response, navigation, and redirect
23Peter Dolog, Web Engineering 2010, Quality Assesment I
Example: bidding web application
24Peter Dolog, Web Engineering 2010, Quality Assesment I
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.
Principles
25Peter Dolog, Web Engineering 2010, Quality Assesment I
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.
Composite CFG
26Peter Dolog, Web Engineering 2010, Quality Assesment I
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.
27Peter Dolog, Web Engineering 2010, Quality Assesment I
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.
OCFG for the example
28Peter Dolog, Web Engineering 2010, Quality Assesment I
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.
5-level Data flow testing
Function level (CFG)Function cluster level (ICFG)Object level (OCFG + criterion)Object cluster level (ICFG cross object, CCFG for
http)Application level (CCFG)
29Peter Dolog, Web Engineering 2010, Quality Assesment I
Def use chains from graphs
30Peter Dolog, Web Engineering 2010, Quality Assesment I
© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.
Summary
Can uncover error at the lovest levelsProvides a systemHowever, does not scale for complex applicationsYou can use it in your project for the most important
functions
31Peter Dolog, Web Engineering 2010, Quality Assesment I
Terminology
Quality
According to ISO 8402-86 Quality is the totality of features and characteristics of software product that relate to its ability to satisfy stated or implied needs.
33Peter Dolog, Web Engineering 2010, Quality Assesment I
Quality Attributes (ISO/IEC 9126-1)
Functionality – correct and adequate functionsReliability – capability of ensuring given level of
performanceUsability – related to ease of use by end usersEfficiency – related to the ratio of the performance
to the number of used resourcesMaintanability – ease of modification even after
deploymentPortability – being able to run on different platforms
34Peter Dolog, Web Engineering 2010, Quality Assesment I
Quality Model
Quality model is a selection of criteria to assess properties related to quality characteristics
Quality assessment => measuring the properties related to the above according to specified quality metrics for those
35Peter Dolog, Web Engineering 2010, Quality Assesment I
Quality Characteristics (ISO/IEC 9126)
36Peter Dolog, Web Engineering 2010, Quality Assesment I
© Springer, Web Applications Engineering 2009
Quality Perspectives
37Peter Dolog, Web Engineering 2010, Quality Assesment I
© Springer, Web Applications Engineering 2009
Quality in Use
Effectiveness – capability of reaching the users goals with accuratness and completeness
Productivity – capability of spending appropriate amount of resources in relation to effectiveness to be achieved
Safety – capability of achieving acceptable level of risks for the users, the usage environment, and the users’ activities
Satisfaction – a capability to satisfy the user in a given context of use
38Peter Dolog, Web Engineering 2010, Quality Assesment I
Evaluation of Quality
Internal attributes – e. g. static measures on the code
External attributes – e. g. dynamic measures of behavior of the code
Quality in use attributes – e. g. reduction of time to produce an invoice
39Peter Dolog, Web Engineering 2010, Quality Assesment I
Web Application Functionality
ContentNavigationCorrectness and Adequateness
=> Functional Testing
40Peter Dolog, Web Engineering 2010, Quality Assesment I
Performance
Throughput:number of requests per unit of time
Response time:the time needed to serve a request
Evaluation in average and peak conditions
41Peter Dolog, Web Engineering 2010, Quality Assesment I
Usability
Easy-to-use navigation mechanismsQuickly reaching relevant content and operationNot getting lostAccessibility
42Peter Dolog, Web Engineering 2010, Quality Assesment I
Testing Web Applications
White Box TestingBlack Box TestingGray Box TestingUnit TestingIntegration TestingSystem Testing
43Peter Dolog, Web Engineering 2010, Quality Assesment I
Types of Web Applications Testing
Functional testing: Ghost pages, Unreachable pagesModel-based testing: using design models to find
out whether quality criteria are satifiedData-flow testing: to reveal invalid use of data
values due to coding errors based on control flow
44Peter Dolog, Web Engineering 2010, Quality Assesment I
Structural Model supporting Testing
45Peter Dolog, Web Engineering 2010, Quality Assesment I
© Springer, Web Applications Engineering 2009
Data-Flow testing
Path coverage Branch coverageNode coverage
46Peter Dolog, Web Engineering 2010, Quality Assesment I
Control Flow Model
47Peter Dolog, Web Engineering 2010, Quality Assesment I
© Springer, Web Applications Engineering 2009