applying generative testing in the browser with selenium
TRANSCRIPT
![Page 1: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/1.jpg)
Generating & Running 1 million tests per hour
#Seconf16
![Page 3: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/3.jpg)
Caveats• Thought experiment • Greenfield projects • Functional stack (language, database, etc.) • Assume basics covered: Fully-parallelized test setup
• Browsers • Servers • Runners/reporters
![Page 4: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/4.jpg)
What’s testing for?Exploring state-transition space for invalid states
![Page 5: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/5.jpg)
Step Transition State Valid?
1 Add item {:item-ids [1]} TRUE
2 Remove item {:item-ids []} TRUE
3Add
nonexistent item
{:item-ids [nil]} FALSE
Cart state transitions
![Page 6: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/6.jpg)
Testing Today• Largely example based
• Hand-Written unit tests (including Page object, etc.) • Recorded examples
• We tend to think we can manually enumerate all important state transitions. We’re very optimistic in that way (combinatorial explosion!)
![Page 7: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/7.jpg)
Example app: Osmium
Book summarizing site
![Page 8: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/8.jpg)
OSMIUM ACTIONS
• Signup • Update account • Login • Logout • View account
ACCOUNTS:
• List • Submit book • View book • Rate book • Update book
Books:
![Page 9: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/9.jpg)
![Page 10: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/10.jpg)
![Page 11: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/11.jpg)
![Page 12: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/12.jpg)
Site flow
![Page 13: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/13.jpg)
View book
List books
Site flow
![Page 14: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/14.jpg)
View book
List books
If not logged inAlways availableSite flow
![Page 15: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/15.jpg)
View book
List books
Sign up
Login
If not logged inAlways availableSite flow
![Page 16: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/16.jpg)
View book
List books
Sign up
Login
If not logged inAlways available
If logged inSite flow
![Page 17: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/17.jpg)
View book
List books
Sign up
Login View account
Update account
Logout
If not logged inAlways available
If logged inSite flow
![Page 18: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/18.jpg)
View book
List books
Sign up
LoginSubmit book
View account
Update account
Logout
If not logged inAlways available
If logged inSite flow
![Page 19: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/19.jpg)
View book
List books
Sign up
LoginSubmit book
Rate book
View account
Update account
Logout
If not logged inAlways available
If logged inSite flow
![Page 20: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/20.jpg)
View book
List books
Sign up
LoginSubmit book
Rate book
Update book
View account
Update account
Logout
If not logged inAlways available
If logged inSite flow
![Page 21: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/21.jpg)
Example user flows
Action
Action
- When logged in
- When logged out
![Page 22: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/22.jpg)
List books
Example user flows
Action
Action
- When logged in
- When logged out
![Page 23: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/23.jpg)
List books
Example user flows
View book
Action
Action
- When logged in
- When logged out
![Page 24: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/24.jpg)
List books
Example user flows
View book
Sign up
Action
Action
- When logged in
- When logged out
![Page 25: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/25.jpg)
List books
Example user flows
Login
View book
Sign up
Action
Action
- When logged in
- When logged out
![Page 26: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/26.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Action
Action
- When logged in
- When logged out
![Page 27: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/27.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
Action
Action
- When logged in
- When logged out
![Page 28: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/28.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Action
Action
- When logged in
- When logged out
![Page 29: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/29.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List booksLogout
Action
Action
- When logged in
- When logged out
![Page 30: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/30.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Submit book
Logout
Action
Action
- When logged in
- When logged out
![Page 31: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/31.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
Action
Action
- When logged in
- When logged out
![Page 32: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/32.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
Logout
Action
Action
- When logged in
- When logged out
![Page 33: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/33.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View account
Logout
Action
Action
- When logged in
- When logged out
![Page 34: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/34.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
Action
Action
- When logged in
- When logged out
![Page 35: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/35.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
Logout
Action
Action
- When logged in
- When logged out
![Page 36: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/36.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
Logout
Action
Action
- When logged in
- When logged out
![Page 37: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/37.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Logout
Action
Action
- When logged in
- When logged out
![Page 38: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/38.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Logout
Action
Action
- When logged in
- When logged out
![Page 39: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/39.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 40: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/40.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 41: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/41.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 42: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/42.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 43: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/43.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 44: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/44.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 45: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/45.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?…?
…?…?
…?…?List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 46: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/46.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?…?
…?…?
…?…?
…?…?
…?
…?
…?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 47: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/47.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 48: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/48.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…?…?…?
…?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 49: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/49.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…?…?…?
…?…?
…?…
?…
?…
?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 50: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/50.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 51: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/51.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 52: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/52.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?…
…………
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 53: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/53.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
…?
…?…
?…
?…
?
…………
…
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 54: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/54.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
…?…?
…?…?
…?
…?
…?…
?…
?…
?
…………
…
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 55: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/55.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
…?…?
…?…?
…?…?…?
…?…?
…?
…?
…?…
?…
?…
?
…………
…
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 56: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/56.jpg)
List books
Example user flows
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
…?…?
…?
…?…?
…?…?
…?…?…?
…?…?…?
…?…?
…?…?
…?
…?
…?
…?…
?…
?…
?…
?
…?…?…?…?
…?…?
…?…
?…
?…
? …?…
?…
?…
?…
?
…?…?
…?…?
…?…?…?
…?…?
…?…?
…?…?
…?…?
…?
…?…
?…
?…
?
…………
…
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
Action
Action
- When logged in
- When logged out
![Page 57: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/57.jpg)
List books
Test cases trace specific paths
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
![Page 58: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/58.jpg)
List books
Test cases trace specific paths
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
![Page 59: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/59.jpg)
List books
Test cases trace specific paths
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
![Page 60: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/60.jpg)
List books
Test cases trace specific paths
Login
View book
Sign up
Sign up Login
List books
Login
List books
View book
Login
List books
Login
View
Login
List books
Submit book
Logout
View book
View account
Logout
List books
View account
Update description
Rate book
Logout
![Page 61: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/61.jpg)
Even worse when considering concurrent accessing to
shared resource
![Page 62: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/62.jpg)
Even worse when considering concurrent accessing to
shared resourceCan introduce state transitions not possible in linear flow
![Page 63: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/63.jpg)
Even worse when considering concurrent accessing to
shared resourceCan introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
![Page 64: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/64.jpg)
Even worse when considering concurrent accessing to
shared resource
TAB A TAB B
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
![Page 65: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/65.jpg)
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
![Page 66: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/66.jpg)
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
![Page 67: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/67.jpg)
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”Fill out form
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
![Page 68: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/68.jpg)
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”Fill out form
Logout
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
![Page 69: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/69.jpg)
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”Fill out form
LogoutSubmit form
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
![Page 70: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/70.jpg)
Even worse when considering concurrent accessing to
shared resource
TAB A TAB BLogin
Go to “Add book page”Fill out form
LogoutSubmit form
Can introduce state transitions not possible in linear flow
Scenario, two tabs open in a browser:
What’s the resultant state? Book created? Did we write a test for it?
![Page 71: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/71.jpg)
Alternative:
![Page 72: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/72.jpg)
Don’t write (or record) tests
Alternative:
![Page 73: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/73.jpg)
Don’t write (or record) tests
Alternative:
tests, generate them!
![Page 74: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/74.jpg)
Don’t write (or record) tests
- John Hughes
Alternative:
tests, generate them!
![Page 75: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/75.jpg)
or….
![Page 76: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/76.jpg)
“Automate all the things!”
- Priti Biyani, 1 hour ago
or….
![Page 77: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/77.jpg)
“Automate all the things!”
- Priti Biyani, 1 hour ago
Even writing tests…
or….
![Page 78: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/78.jpg)
Core prop: Use infrastructure instead of human time.
![Page 79: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/79.jpg)
Nearly same effort (with practice) as traditional example-based tests, but scales
to millions of examples
Constant effort vs linear effort
![Page 80: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/80.jpg)
What properties can we assert over the whole
graph?• Page
• Should never have a JavaScript error • Should never have a 200, 30*, 401, 426
response • If a user is logged in, they should always
see a “logout” link (and vice-versa)
Login
List books
List books Login
View book
Sign up
Sign up Login
List books
View book
Login
List books
Submit book
Logout
View book
View account
List books
View account
Update Rate
Logout
![Page 81: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/81.jpg)
What properties can we assert over specific
transitions?• State
• User must have been logged in • Number of ratings for the book should be
increased by 1 • updated_at timestamp should have changed
• Page • Logout link must be visible
Update book
• State • User must have been logged in • updated_at timestamp should have changed
• Page • Logout link must be visible • Must be at the view_book url for the reviewed book
Rate book Logout
• State • User must no longer be logged in
• Page • Login link must be visible
Submit book
• State • Book must have a submitter_id • User therefore must be logged in • Book must have certain properties
• Page • Must be at the view_book url
![Page 82: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/82.jpg)
How can we teach a computer to navigate our
app?
![Page 83: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/83.jpg)
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
![Page 84: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/84.jpg)
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
![Page 85: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/85.jpg)
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.
![Page 86: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/86.jpg)
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?
![Page 87: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/87.jpg)
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.
![Page 88: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/88.jpg)
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action
![Page 89: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/89.jpg)
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action• Automatically tag page structure with a .osmium-action class
![Page 90: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/90.jpg)
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action• Automatically tag page structure with a .osmium-action class• Walk down an edge and recur to generate possible paths
![Page 91: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/91.jpg)
How can we teach a computer to navigate our
app?• Each page has allowed inbound edges and outbound edges
• Page model can explicitly list allowed transitions as a list of predicate -> generation-vectors (i.e. “from the list view page, if not logged in, the page can possibly go these three other pages, and here is a function to generate the link).
• Bit clunky and prone to rot.• Can we simply capture the transitions at render time, and use
that as a model for generating valid transitions?• Render page to intermediate data structure.• Use functions to link to next action• Automatically tag page structure with a .osmium-action class• Walk down an edge and recur to generate possible paths• At each step along the walk, run the relevant assertions
![Page 92: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/92.jpg)
Example path
![Page 93: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/93.jpg)
Example path[
![Page 94: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/94.jpg)
Example path[[:to “demo.localbox.info:3005"]
![Page 95: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/95.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"]
![Page 96: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/96.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"]
![Page 97: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/97.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"]
![Page 98: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/98.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"]
![Page 99: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/99.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"]
![Page 100: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/100.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"]
![Page 101: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/101.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"]
![Page 102: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/102.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click “#osmium-1"]
![Page 103: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/103.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click “#osmium-1"][:osmium.user/email "#osmium-14"]
![Page 104: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/104.jpg)
Example path[[:to “demo.localbox.info:3005"][:click "#osmium-1"][:osmium.user/password "#osmium-15"][:click "#osmium-16"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click "#osmium-1"][:click “#osmium-1"][:osmium.user/email "#osmium-14"]]
![Page 105: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/105.jpg)
Example test (we should never show errors)
(deftest no-errors-test (testing "We should never show an error page" (let [driver (sl/new-sauce-labs-browser) db (:db @o/system) sessions (-> @o/system :session-store .session-map)] (eval! driver [:to "demo.localbox.info:3005"])
(let [actions (walk-n-steps! driver 5 (fn check-step [a path] (let [title (taxi/title driver) error (re-find #"Internal Error" title)] (record-failing-path-when! (not (nil? error)) path))))] (reset! replay {:actions actions :step 0})) (taxi/quit driver))))
![Page 106: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/106.jpg)
Demo:
• Assumptions • Fully immutable database (Datomic) • Functional, referentially-transparent application (explicit
state)
![Page 107: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/107.jpg)
![Page 108: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/108.jpg)
![Page 109: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/109.jpg)
![Page 110: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/110.jpg)
![Page 111: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/111.jpg)
Challenges in Generative Testing
![Page 112: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/112.jpg)
Challenges in Generative Testing
• Testing the right properties
![Page 113: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/113.jpg)
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalize
![Page 114: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/114.jpg)
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)
![Page 115: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/115.jpg)
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
![Page 116: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/116.jpg)
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long tests
![Page 117: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/117.jpg)
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long testsFailing tests may contain thousands and thousands of retro steps, many of which are irrelevant.
![Page 118: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/118.jpg)
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long testsFailing tests may contain thousands and thousands of retro steps, many of which are irrelevant.
Answer: Shrinking!
![Page 119: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/119.jpg)
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long testsFailing tests may contain thousands and thousands of retro steps, many of which are irrelevant.
Answer: Shrinking!• Extreme scale
![Page 120: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/120.jpg)
Challenges in Generative Testing
• Testing the right propertiesBig shift in approach takes time and practice to internalizeReproducibility is critical! (functional programming approach can help)Flakiness is a killer here
• Long testsFailing tests may contain thousands and thousands of retro steps, many of which are irrelevant.
Answer: Shrinking!• Extreme scale
• Depth of tests + desired coverage + shrinking process = massive demands on infrastructure (both server and browser side)
![Page 121: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/121.jpg)
Shrinking
![Page 122: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/122.jpg)
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
![Page 123: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/123.jpg)
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!
![Page 124: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/124.jpg)
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
![Page 125: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/125.jpg)
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
(imaging a user reporting this case, then trying to debug it…)
![Page 126: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/126.jpg)
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
(imaging a user reporting this case, then trying to debug it…)
• Take out some steps, rerun path, see if it still fails
![Page 127: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/127.jpg)
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
(imaging a user reporting this case, then trying to debug it…)
• Take out some steps, rerun path, see if it still fails
• Repeat until you have a minimum repro case
![Page 128: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/128.jpg)
ShrinkingFailing case: 2000+ Steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click "#osmium-16"] [:click "#osmium-1"] [:click "#osmium-1"] [:click “#osmium-1"] [:click "#osmium-1"] [:osmium.user/password "#osmium-15"]
• Good news: Found a failing path!• Bad news, it’s 2000+ steps long
(imaging a user reporting this case, then trying to debug it…)
• Take out some steps, rerun path, see if it still fails
• Repeat until you have a minimum repro case
Minimum case: 7 steps [ [:to “demo.localbox.info:3005"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:click "#osmium-1"] [:osmium.user/email "#osmium-14"] [:submit "#osmium-13"] [:click "#osmium-16"] ]
![Page 129: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/129.jpg)
Recording has its place, but it’s not in the browser.
![Page 130: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/130.jpg)
Recording has its place, but it’s not in the browser.
It’s in the database.
![Page 131: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/131.jpg)
Record state at each step of the test, then run very fast tests over the state
![Page 132: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/132.jpg)
So why 1,000,000 assertions per hour?
![Page 133: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/133.jpg)
So why 1,000,000 assertions per hour?
• Very cool, but not the point:
![Page 134: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/134.jpg)
So why 1,000,000 assertions per hour?
• Very cool, but not the point:• Depth (longer sessions, more changes in state)
![Page 135: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/135.jpg)
So why 1,000,000 assertions per hour?
• Very cool, but not the point:• Depth (longer sessions, more changes in state)• Breadth (desired coverage over branches)
![Page 136: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/136.jpg)
So why 1,000,000 assertions per hour?
• Very cool, but not the point:• Depth (longer sessions, more changes in state)• Breadth (desired coverage over branches)• Shrinking process
![Page 137: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/137.jpg)
Infrastructure is a challenge, but tractable
![Page 138: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/138.jpg)
Infrastructure is a challenge, but tractable
• Generative testing has seen uses in many areas, but almost nothing in Selenium/Appium world - infrastructure is the biggest challenge
![Page 139: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/139.jpg)
Infrastructure is a challenge, but tractable
• Generative testing has seen uses in many areas, but almost nothing in Selenium/Appium world - infrastructure is the biggest challenge
• CircleCI - scale out your build process automatically
![Page 140: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/140.jpg)
Infrastructure is a challenge, but tractable
• Generative testing has seen uses in many areas, but almost nothing in Selenium/Appium world - infrastructure is the biggest challenge
• CircleCI - scale out your build process automatically• Sauce Labs - Scale out browser infrastructure as on
demand
![Page 141: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/141.jpg)
Future Areas
![Page 142: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/142.jpg)
Concolic Testing
![Page 143: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/143.jpg)
Concolic TestingConcrete Tests
![Page 144: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/144.jpg)
Concolic TestingConcrete Tests Symbolic Execution
![Page 145: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/145.jpg)
Concolic TestingConcrete Tests Symbolic Execution
Terrible name
![Page 146: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/146.jpg)
Concolic TestingConcrete Tests Symbolic Execution
Terrible nameMind-blowingly cool idea
![Page 147: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/147.jpg)
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 148: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/148.jpg)
• We have to generate a very specific value (42) to trigger the exception. We can run infinite tests in either directions and still miss it.
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 149: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/149.jpg)
• We have to generate a very specific value (42) to trigger the exception. We can run infinite tests in either directions and still miss it.
• Bring in code analysis!
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 150: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/150.jpg)
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 151: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/151.jpg)
Three code paths(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 152: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/152.jpg)
Three code pathsGuarded by three conditions
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 153: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/153.jpg)
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 154: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/154.jpg)
• (< internet-points 42) <- Constrain generator for input to super-user?
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 155: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/155.jpg)
• (< internet-points 42) <- Constrain generator for input to super-user?
• Repeat for (> internet-points 42)
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 156: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/156.jpg)
• (< internet-points 42) <- Constrain generator for input to super-user?
• Repeat for (> internet-points 42)• Finally repeat for (= internet-points 42)
(defn super-user? [internet-points] (cond (< internet-points 42) false (> internet-points 42) true (= internet-points 42) (throw (Exception. "No user can have 42 points!"))))
![Page 157: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/157.jpg)
Predictive Testing
![Page 158: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/158.jpg)
Predictive Testing• Record paths your users take through your site
![Page 159: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/159.jpg)
Predictive Testing• Record paths your users take through your site
• If an error is triggered, email devs with repro steps automatically
![Page 160: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/160.jpg)
Predictive Testing• Record paths your users take through your site
• If an error is triggered, email devs with repro steps automatically
• Before pushing new code, re-run previous e.g. 10,000 sessions on your site, see if you’ve broken anything your users actually do in your app
![Page 161: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/161.jpg)
![Page 162: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/162.jpg)
![Page 163: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/163.jpg)
Summary
![Page 164: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/164.jpg)
Summary
• Humans are terrible at manually enumerating all possible state-space
![Page 165: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/165.jpg)
Summary
• Humans are terrible at manually enumerating all possible state-space
• Computers are very good at finding surprising edge cases (just like users!) given some help
![Page 166: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/166.jpg)
Summary
• Humans are terrible at manually enumerating all possible state-space
• Computers are very good at finding surprising edge cases (just like users!) given some help
• Don’t write tests - generate them!
![Page 167: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/167.jpg)
CreditAlmost all the ideas/experimentation/actual
work done by Sebastian Bensusan
![Page 168: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/168.jpg)
CreditAlmost all the ideas/experimentation/actual
work done by Sebastian Bensusan
![Page 169: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/169.jpg)
CreditAlmost all the ideas/experimentation/actual
work done by Sebastian Bensusan
@sebasbensu http://github.com/bensu
![Page 170: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/170.jpg)
Credit
https://www.paygarden.com/
![Page 172: Applying generative testing in the browser with Selenium](https://reader031.vdocument.in/reader031/viewer/2022021814/58ecb6721a28ab33718b4635/html5/thumbnails/172.jpg)
Resources
• John Hughes - Testing the Hard Stuff and Staying Sane [Clojure/West 2014]
• Kostis Sagonas - A CutEr Tool [The Erlang Factory 2016] • Reid Draper - Powerful Testing with test.check [Clojure/
West 2014]