introduction to automated tests -...
TRANSCRIPT
![Page 1: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/1.jpg)
Introduction to Automated Tests
Paulo Cheque
Summer 2009
License: Creative Commons: Attribution-Share Alike 3.0 Unportedhttp://creativecommons.org/licenses/by-sa/3.0/
![Page 2: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/2.jpg)
2
About● Language● Types of tests● Introduction to test frameworks
![Page 3: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/3.jpg)
3
Many communities
● A lot of terminology and dialects:● Agile communities● Formal communities● QA communities● Tools● Clients
![Page 4: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/4.jpg)
4
Standards Specifications● BS 79252:1998. Software Component Testing.● DO178B:1992. Software Considerations in Airborne Systems
and Equipment● Certification, Requirements and Technical Concepts for
Aviation (RTCA SC167).● IEEE 610.12:1990. Standard Glossary of Software Engineering
Terminology.● IEEE 829:1998. Standard for Software Test Documentation.● IEEE 1008:1993. Standard for Software Unit Testing.● IEEE 1012:1986. Standard for Verification and Validation
Plans● ...
![Page 5: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/5.jpg)
5
Jargons
• Defect/Mistake/Error/Failure• Verification• Validation• Inspection• Debugging• ...
![Page 6: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/6.jpg)
6
Terms
• Blackbox/White/Gray/Glass• Functional Tests• Structural Tests
• Test• Test case• Test suite
![Page 7: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/7.jpg)
7
Symbols
• SUT/AUT: System/Application Under Test• TFD/POUT: TestFirst Development / Plain Old
Unit Test• TAD: TestAfter Development• TDD: TestDriven Development / Design• BDD/EDD/STDD: Behaviour/Example/Story
Test Driven Development• TDDD: TestDriven Design Databases
![Page 8: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/8.jpg)
8
Double● Dummy: A simple object to make a test executable● Fake: Lightweight object that contains a false
implementation● Stub: Object with data to be used in tests● Mock: Object with an appropriate interface that
contains data and behavior that will be used in tests● Spy: Object to capture indirect calls
![Page 9: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/9.jpg)
9
Type of Automated Tests
● Unit● Integration● Acceptance● Interface● Smoke● Sanity● Mutation● ...
● Performance● Load● Stress● Longevity● Security● Installation● ...
![Page 10: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/10.jpg)
10
Unit test
● Unit: Class/Module or Method/Function● Unit or “miniintegration”
● Test a feature, not an implementation● Basic but very important test● Solid test● Avoid debugging
![Page 11: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/11.jpg)
11
Integration tests
● In general: Test that involve more than one unit
● Verify errors in the relationship between modules, components or layers that may work fine individually
![Page 12: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/12.jpg)
12
Acceptance
● Test a feature from the user's point of view● Assurance for a client that software works
perfectly and correctly
● Clients, users and developers may (must) work together to write tests● Implementation details hidden● Client language – Domain Specific Language
![Page 13: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/13.jpg)
13
Interface Tests● User Interface
● Console● GUI● WUI
● => Acceptance Tests● => Usability● => Layout Tests
![Page 14: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/14.jpg)
14
Smoke Tests
● Rapid, extensive and superficial tests
● After software installation
● Example:● Visit a Web page and verify that 404 text is not
visible
![Page 15: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/15.jpg)
15
Sanity Tests
● A smoke test
● Appropriate to find huge errors in algorithms
● Use particular technique of an algorithm to identify possible errors● Theorems● Mathematical properties
![Page 16: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/16.jpg)
16
Mutation tests
● Test a test● “Tests may show some errors, but not its
absence” Dijkstra● Infinite possibility of errors?
● Competent developer● Mutant: SUT with few changes● Run a suite of tests under mutants● Analysis of results: Mutants and Tests
![Page 17: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/17.jpg)
17
Performance tests● Evaluate response time of a specific module● Profilers help finding bottlenecks● Unit test suites without mocks may be used
like profilers (this is not the main goal of those suites)
● PS: Don't evaluate computational complexity
![Page 18: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/18.jpg)
18
Load tests● Simulate a lot of users and requests
● Simultaneous or inside a time interval
● Test● InfraStructure● Hardware● Network bandwidth● Database and servers
![Page 19: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/19.jpg)
19
Stress tests● Maximum load test● Identify maximum amount of:
● users● requests● data
● Limitations of an environment
![Page 20: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/20.jpg)
20
Longevity tests● Test to verify performance and correction
within a large period of time● Find problems with memory leakage● Find cache errors
![Page 21: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/21.jpg)
21
Other types...● Tests of:
● Installation: Verify if all components are interconnected properly without hardware incompatibility
● Recovery: Appropriated reaction after errors● Configuration: Different configurations and
distinct environments (portability)● ...
● Alfa / Beta versions
![Page 22: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/22.jpg)
22
Security tests● Useful for applications exposed to bad
intentioned users● Web, banks ...
● Test:● Permissions errors● Errors in model layer (SQLInjection ...)● Bugs in servers● Fragility to attacks like DoS (Stress tests)
![Page 23: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/23.jpg)
23
Pertinent information● Machine: CPU, Memory, HD, Swap ...● OS: Name/Distribution, Version, Kernel, Locale ...● Servers: NameVersion, Pertinent files,
● Tomcat 5.5, Jboss 4.0.4, (/etc/init.d/jboss)● Database: NameVersion, Machine, Schema● Network: IP, Firewall, Port, DNS ...● Dependencies: NameVersion
● JDK 1.5.0_06, Ruby 1.8● Product: Name, Version, installation data, logs
![Page 24: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/24.jpg)
24
Unit tests
• Main• XUnit frameworks
– Test case– Verification– Exceptions– Results– Report
![Page 25: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/25.jpg)
25
Main
public class MainExample {
public static void main(String[] args) {if(!new Double(Math.sqrt(1)).isNaN())
throw new RuntimeException("Ops!");// Orassert new Double(Math.sqrt(1)).isNaN() == true;
}
}
![Page 26: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/26.jpg)
26
xUnit● Article:
http://junit.sourceforge.net/doc/cookstour/cookstour.htm
![Page 27: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/27.jpg)
27
Test case
● Conventions● public void testSomeMethodName() { ... }
● Annotations@Test public void someMethodName() { ... }
● Configurations● xml...
![Page 28: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/28.jpg)
28
Verification● AssertTrue / AssertFalse● AssertEquals● AssertSame● Fail● Expected Exceptions
● Friendly messages
● Hamcrest: Matchers + AssertThat
![Page 29: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/29.jpg)
29
Demonstration
● C● Java● Scala● Groovy
![Page 30: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/30.jpg)
30
Setup e Teardown
● Test cases must be independent:● From each other● From number of times it has been executed● From external factors
● Setup: Prepare environment to test ● Teardown: Clean environment for other
tests
![Page 31: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/31.jpg)
31
Results
● Skip● Fail● Error● Success
![Page 32: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/32.jpg)
32
Reports
● Language● APIDocs
● Tools● StackTrace● Logs● Screenshots
● Metrics
![Page 33: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/33.jpg)
33
Some tools● Unit tests:
● CxxTest (C++): http://cxxtest.sourceforge.net● CUnit: http://cunit.sourceforge.net● JUnit (Java): http://www.junit.org● DUnit (Delphi): http://dunit.sourceforge.net● VBUnit (Visual Basic): http://www.vbunit.com● TestNG (Java): http://testng.org● RSpec (Ruby): http://rspec.info/
● http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks
![Page 34: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/34.jpg)
34
+some tools● Mock Objects:
● SevenMock (Java): http://sevenmock.sourceforge.net● EasyMock (Java): http://www.easymock.org/● JMock (Java): http://www.jmock.org● Rhino.Mocks (.NET):
http://www.ayende.com/projects/rhinomocks.aspx● SMock (Smalltalk):
http://www.macta.f2s.com/Thoughts/smock.html● Mockpp (C++): http://mockpp.sourceforge.net● GoogleMock (C++): http://code.google.com/p/googlemock
![Page 35: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/35.jpg)
35
+some tools● User Interface tests for Desktop:
● Fest: http://fest.easytesting.org/swing● Jemmy: http://jemmy.netbeans.org● Marathon: http://www.marathontesting.com
● Web interface tests:● Selenium: http://www.openqa.org/selenium● Watir: http://wtr.rubyforge.org
● Acceptance tests: ● Fit: http://fit.c2.com
![Page 36: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/36.jpg)
36
+some tools● Mutation tests:
● Jabuti (USP São Carlos): http://jabuti.incubadora.fapesp.br
● Jester: http://jester.sourceforge.net● Heckle: https://rubyforge.org/projects/seattlerb
● Performance / Load tests:● JMeter: http://jakarta.apache.org/jmeter
● Others: ● JPDFUnit: http://jpdfunit.sourceforge.net
![Page 37: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/37.jpg)
37
Some links http://www.testing.com http://www.opensourcetesting.org http://xunitpatterns.com http://www.mockobjects.com http://javasource.net/opensource/testingtools http://www.junit.org http://www.agilcoop.org.br
![Page 38: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/38.jpg)
38
Some Books Gerard Meszaros, “xUnit Test Patterns, Refactoring Test Code”,
AddisonWesley, 2007 Robert V. Binder, “Testing ObjectOriented Systems”, Addison
Wesley Professional, 1999 L. Crispin, T. House, “Testing Extreme Programming”, Addison
Wesley, 2005 R. Mugridge, W. Cunningham, “Fit for Developing Software”,
Prentice Hall, 2006 M. Delamaro, J. Maldonado, M. Jino, “Introdução ao Teste de
Software”, Campus, 2007
![Page 39: Introduction to Automated Tests - CCSLccsl.ime.usp.br/files/AutomatedTests-1-2-Introduction-AgilCoop.pdf · Introduction to Automated Tests Paulo Cheque ... Web interface tests: Selenium:](https://reader031.vdocument.in/reader031/viewer/2022013113/5b5c6e8c7f8b9a65028bd2c2/html5/thumbnails/39.jpg)
39
Contact
http://www.agilcoop.org.brhttp://ccsl.ime.usp.br
http://qualipso.org
[email protected]@agilcoop.org.br
License: Creative Commons: Attribution-Share Alike 3.0 Unportedhttp://creativecommons.org/licenses/by-sa/3.0/