unit tests are not enough€¦ · agenda • why unit tests are not enough • characteristics of...
TRANSCRIPT
![Page 1: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/1.jpg)
Unit Tests Are Not EnoughPractical Functional Testing in Java
Martin Kloseklose brothers
[email protected]@martinklose
1
![Page 2: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/2.jpg)
Agenda• Why unit tests are not enough
• Characteristics of good tests scenarios
• The vision: an Executable Specification
• The tools we can use
• Tool demo
• Tips for functional testing
• What can go wrong
2
![Page 3: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/3.jpg)
A Fairy Tale
• Little John and the Air Balloons
3
![Page 4: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/4.jpg)
Customer not satisfied
4
![Page 5: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/5.jpg)
What’s covered
test scenarios
automationlayer
(glue code)
application
test tool
unit/integration
tests
functional testing,acceptance testing,specification by example
5
![Page 6: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/6.jpg)
Test scenarios should...
• speak the customer’s language• cover only the important aspects• be complete and realistic examples• be self explaining
• NOT be scripts!
6
![Page 7: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/7.jpg)
Test scenarios should...
•speak the customer’s language‣ Uses business terms
‣ About business functionality not SW design
‣ Ideally customer should be able to extend and change
‣ Used for communication and clarification
- Leading to a shared understanding
- Spot inconsistencies
7
![Page 8: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/8.jpg)
Test scenarios should...
•cover only the important aspects‣ Hide technical and implementation details
‣ Only business related aspects
‣ Do not cover all edge & corner cases
- We still have testers
- We have lots of unit tests, too
8
![Page 9: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/9.jpg)
Test scenarios should...
•be complete and realistic example‣ Combinations of parameters
and expected outputs
‣ Using real data helps
9
![Page 10: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/10.jpg)
Test scenarios should...
•be self explaining‣ Understandable for users
with domain knowledge
‣ All & only important information
‣ Explaining underlying logic & business rules
‣ Make default values explicit when relevant for business logic
10
![Page 11: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/11.jpg)
Test scenarios should...•NOT be scripts‣ Describe what instead of how
‣ Don’t make the user work backward from single steps to understand what is illustrated
‣ Scripts will cost a lot of time in the long term
- Difficult to understand
- Reason for test failure is difficult to find
- Higher maintenance, UI & Workflows change more often than business rules
11
![Page 12: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/12.jpg)
The vision: an Executable Specification
• Frequently validated• Reliable information about system functionality• Easy to read and understand• Well organized and easy accessible• Updated when system functionality changes
12
![Page 13: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/13.jpg)
Tools for functional testing should allow...
• to formulate test scenarios in the appropriate abstraction/language
• to connect to the system under test at any layer or API
• to use the VCS of your choice• to be easily integrated into your CI/
Build
13
![Page 14: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/14.jpg)
Tools to consider
• Keyword- / table-centric frameworks‣ FitNesse, Robot Framework, Twist
• Behaviour-Driven-Development tools‣ Cucumber, JBehave, EasyB
• Free text tools‣ Concordion
14
![Page 15: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/15.jpg)
Demo
•Source code is on github:‣ https://github.com/mklose/goto2011aar
15
![Page 16: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/16.jpg)
Hide technical details
• Use the testing tool's set up / tear down / abstraction mechanisms sparingly
• Write dedicated fixture code which does the complicated set up
16
![Page 17: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/17.jpg)
Choose the appropriate "point of attac"
Business Facade
UI
AppModel
View
Persistence
Credit Rules
customer type monthly turnover
overdraft?
Private 2500 7500Private 1200 5000Private 99 0
Business 25000 25000
Business 50000 25000 expected50000 actual
backdoorDomain
Customer
Account
CreditRules
17
![Page 18: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/18.jpg)
Rules for Automation through the User Interface
• Rule 1: Never automate through the UI‣ UI changes more often => Brittle tests
‣ UI automation is technically complex
‣ UI automation is slow
18
![Page 19: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/19.jpg)
Rules for Automation through the User Interface
• Rule 2: Only automate through the UI in case of emergency‣ Sometimes necessary to build customer
trust in test automation
‣ Abstract the UI away
‣ Be aware of what you're missing out on
19
![Page 20: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/20.jpg)
Decouple from systems not under your control
• You cannot reliably include external (test) systems in your regression testing
• Use an additional set of tests for checking real collaboration with external systems‣ Usually started by hand
20
![Page 21: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/21.jpg)
Organization of Tests• Goals‣ I want to see the state of current features at a
single glance
‣ I want to organize all test scenarios in a consistent manner
• How to reach both goals ‣ Organize test scenarios for current work by
stories/features
‣ Reorganize regression tests by functional area
21
![Page 22: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/22.jpg)
What can go wrong ...
• No close collaboration between developers, testers & domain experts
• Abuse of acceptance tests as replacement for thorough unit testing
• Neglecting clean up, refactoring and rework of acceptance tests
➡ It's all about communication!
22
![Page 23: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/23.jpg)
Given a birthday balloon with a face on itWhen you hang it to the ceilingThen the face is smiling friendly
23
![Page 24: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/24.jpg)
• Kudos goes to @johanneslink‣ shirt http://605644.spreadshirt.de/
• global-day-of-coderetreat Sat, Dec 3, 2011https://github.com/coreyhaines/coderetreat/wiki/Cities
‣ Bielefeld*, Germany too*and yes it exists!
Nodes
24
![Page 25: Unit Tests Are Not Enough€¦ · Agenda • Why unit tests are not enough • Characteristics of good tests scenarios • The vision: an Executable Specification • The tools we](https://reader031.vdocument.in/reader031/viewer/2022011913/5fa7aba6cb9c52497878c66a/html5/thumbnails/25.jpg)
References
• Gojko Adzic: Specification by Example,How Successful Teams Deliver the Right Software
• Cunningham & Mugridge: FIT for Developing Software: Framework for Integrated Tests
25