non-functional testing last update: 21-05-2014 philippe charman [email protected] charman
TRANSCRIPT
Non-Functional Testing
Last update: 21-05-2014
Philippe [email protected]
http://users.polytech.unice.fr/~charman/
Non exhaustive list of non-functional testing
• Performance Testing– Load testing– Stress Testing– Endurance Testing
• Usability Testing• Accessibility Testing• Compatibility Testing • Compliance Testing• Internationalization and localization Testing• Doc & Training Testing• Recovery Testing• Etc.
Load Testing
• Load testing is the process of putting demand on a system or device and measuring its response
• Some examples:– simulating multiple users accessing a web server– giving a huge document to a word processor– subjecting a mail server to a large amount of e-mail
traffic– writing and reading data to and from a hard disk
continuously• It helps to identify the maximum operating capacity of an
application as well as any bottlenecks and determine which element is causing degradation
• Synonyms: volume testing, scalability testing, etc.
Load Testing
Stress Testing
• Testing used to determine the stability of a system
• Testing beyond normal operational capacity, often to a breaking point, in order to observe the results
• The goals may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space),
Endurance Testing
• Kind of load testing over a longer period of time
• During endurance tests, memory utilization is monitored to detect potential leaks
• Synonyms: reliability testing, soak testing
Performance Tests
Main types What we are mainly concerned at
Load Tests
Scalability Tests
System response time, big memory peak, bottlenecks
Stress Tests Will it crash ?
Endurance Tests Reliability TestsSoak Tests
System response time, memory/resource consumption over a long period of time
Usability Evaluation Methods
• Usability : how well users can learn and use a product to achieve their goals and how satisfied they are with that process
• How to evaluate usability:– Ease of learning– Efficiency of use– Memorability– Error frequency and severity– Subjective satisfaction
• Some evaluation methods:– Inspection– Testing– Inquiry– Rapid prototyping
Usability Inspection
• Usability inspection is aimed at finding usability problems in the design,
• An evaluator inspects a user interface
• Some inspection methods:– Cognitive walkthrough– Heuristic evaluation
Usability Testing
• Technique used to evaluate a product by testing it on real users
• How many users to test ?• Eye Tracking
Eye Tracking Demo
Accessibility Testing
• Software accessibility is a general term used to describe the degree to which an application is available to as many people as possible
• Access issues:– Visual impairments – Motor skills – Hearing impairments– Photosensitive epilepsy
Accessibility Testing
• Typical test cases:– Make sure that all functions are available via keyboard only (do
not use mouse)– Make sure that information is visible when display setting is
changed to High Contrast modes– Make sure that screen reading tools can read all the text
available and every picture/Image have corresponding alternate text associated with it
– Make sure that product defined keyboard actions do not affect accessibility keyboard shortcuts
– Make sure that Web pages do not contain anything that flashes more than two times in any one second period
– etc ...
Accessibility Testing
• Do a simple test:– Start menu > Control Panel > Accessibility Options >
Display > – Check the checkbox "Use High Contrast"– Are your main applications (web browser, email reader,
etc.) still usable ?
• Another test:– Install the tool Color Oracle from http://colororacle.org/– Is it still easy to distinguish colors differences with your
main applications?
Color impaired vision
Compatibility Testing
• Testing conducted on the application to evaluate the application's compatibility with the computing environment that can be composed of:– Hardware platform : x86, x64, UltraSPARC-I,
POWER7 etc. – OS: Windows XP 32-bit, Debian 4.0, Solaris 11, etc.– Database: Oracle, Sybase, DB2, etc.– Browser: IE, Firefox, Chrome,– Peripheral: printer, scanner, – Etc.
• Compatibility Matrix• Pairwise Testing methods
Compatibility Matrix
Backward & Forward Compatibility Testing
• Backward compatibility
• Forward compatibility
Compliance Testing
• Some exemples:– Blue Wash d’IBM– HTLM5 Conformance Tests – Coding rules in C/C++ from MISRA, JSF, etc.– Etc.
C/C++ Standards
• Code should be:– Portable– Reliable– Maintenable– Testable– Reutilisable– Extensible– Readable
C/C++ Standards
• Some examples
• Avoid long and complex functions. • Rationale: long functions are difficult to understand and
to test
• Do not use #define to use a constant• Use "const« instead
• Use #include to include .h only and not .cpp files
Compliance Tests
• More:
• MISRA web site (Motor Industry Software Reliability Association)
http://www.misra.org.uk/
• JSF web site (Joint Strike Fighter)http://www.jsf.mil/downloads/down_documentation.htm
• Web site of the tool Understandhttp://www.scitools.com/
Internationalization& Localization Testing
• Globalisation = i18n + l10n• I18N & l10n
– OS– Keyboard– Display text– Date format– Number format– Etc.
Internationalization& Localization Testing
• To be performed once functional testing has been completed and all the functionality related defects are identified and removed
• 2 phases:First, to make sure that application’s GUI or
functionality will not be broken with the translated text
Second to make sure that translation of all the strings have happened properly
Recovery Testing
• Testing how well an application is able to recover from crashes, hardware failures and other similar problems.
• Recovery testing != reliability testing
References
• Srinivasan Desikan & Gopalaswamy Ramesh, Software Testing: Principles and Practice, Pearson Education
• J. Nielsen, "Heuristic Evaluation". In Jakob Nielsen and Robert L. Mack, editors, "Usability Inspection Methods". John Wiley and Sons, Inc. 1994.
• Mauro, C.L.: Professional usability testing and return on investment as it applies to user interface design for web-based products and services
More information …
• http://www.usability.gov/• http://www.usabilityhome.com/• http://www.testinggeek.com/
• http://www.misra.org.uk/