how to make your ui tests stable, scalable, and maintainable
TRANSCRIPT
![Page 1: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/1.jpg)
How to Make Your UI Tests
Stable, Scalable, and
Maintainable
![Page 2: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/2.jpg)
2
SmartBear’s technology stack
![Page 3: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/3.jpg)
3
Quick Intro
Nikhil Kaul Jeffrey Martin
![Page 4: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/4.jpg)
4
On our plate today
1
Why you shouldn't tolerate flakiness?
2
Common reasons for flaky UI tests
4
What to do with flaky UI tests?
5
How to separate signal from noise?
3
How to overcome these challenges?
![Page 5: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/5.jpg)
5
Flaky tests shouldn't be tolerated
• 300 tests
• Each test has .5% failure rate
• Pass rate for each tests: 99.5%
• Pass rate for the suite: (99.5%)^300 = 22.23%
![Page 6: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/6.jpg)
6
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
![Page 7: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/7.jpg)
7
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
![Page 8: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/8.jpg)
8
Don’t interact with UI elements directly
Test services seperately from the UI
Use page objects
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use UI encapsulation Build modular test design
‒ Identify user journeys
‒ Unit
‒ Service
‒ UI
Automate at three levels Use UI encapsulation
‒ Page Objects
UI Service Unit
![Page 9: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/9.jpg)
9
How to implement in real world
Add Item
Click on an item with name XYZ Clickitem()
Set element with name ABC to quantity Z Set element()
Click on button labeled place in cart clickbutton()
What? How?
![Page 10: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/10.jpg)
10
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
![Page 11: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/11.jpg)
11
Separate the test data from test design
Test services seperately from the UI
Use page objects
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use page objects Build modular test design
– Test specific data
– Reference data
Limit data
dependency
between tests Tests should run
independently
Using databases
‒ Conduct tests
inside a transaction
![Page 12: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/12.jpg)
12
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
![Page 13: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/13.jpg)
13
Limit your dependency on environments
Test services seperately from the UI
Use page objects
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use page objects Build modular test design
‒ Application API
‒ HTML API
‒ Identify user journeys
Limit calls to external systems
Parameterize connections
Hermetic Servers
![Page 14: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/14.jpg)
14
Understand the nature of flakiness
Tech Complexities
Test Enviroment
Test Data
Test Design
![Page 15: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/15.jpg)
15
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use page objects
Build modular test design ‒ Application API
‒ HTML API
Limit calls to external systems
Parameterize connections
Hermetic Servers
Technical complexities can be overcome
AJAX Calls
‒ Callback
‒ Polling Interval
‒ WaitUntil
No Response Calls
‒ Fake objects
‒ Mock
‒ Virtual Service
![Page 16: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/16.jpg)
16
Acceptance test builds should never be red
Separate flaky
from healthy tests
Limit the number of tests as well as the time
![Page 17: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/17.jpg)
17
Use statistics for measuring test stability
Sample Size (N)
• 50
Desired confidence (α)
• 95% confidence
P
• 29/50 = .58
1-p
• 21/50
𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙 𝐶𝐼 = 𝑝 ± z1−α 2
𝑝(1 − p)
n
𝐶𝐼 = .58 ± 1.96.58(1 − .58)
50
𝐶𝐼 = .58 ± .14
𝐶𝐼 = between 44% and 72%.
±z1−α 2 = . ±z.95 = 1.96
![Page 18: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/18.jpg)
18
Takeaways
1
Flaky tests shouldn't be tolerated
2
Understand the nature of flakiness
4
Use statistics for measuring test stability
3
Acceptance test builds should never be red
![Page 19: How to Make Your UI Tests Stable, Scalable, and Maintainable](https://reader031.vdocument.in/reader031/viewer/2022021922/58773daa1a28ab342e8b5f6d/html5/thumbnails/19.jpg)
Questions?