browser-level testing
DESCRIPTION
Overview of Selenium, WebDriver, Watir and related open source cross-browser testing technologies. Presentation given by Martin Kleppmann, founder of browser testing service Go Test It, at Ruby Manor 2009.TRANSCRIPT
![Page 1: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/1.jpg)
Browser-level testing
Martin KleppmannGo Test It
Email: [email protected]: yes-no-cancel.co.uk
Twitter: @martinkl
![Page 2: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/2.jpg)
“But I have unit tests!”
![Page 3: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/3.jpg)
![Page 4: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/4.jpg)
?
![Page 5: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/5.jpg)
Array literals?
var x = [1,];
x.length == 1 // Firefox 3.5x.length == 2 // IE 7
![Page 6: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/6.jpg)
Uninitialised vars?
<div id="hello"></div>
typeof(hello) == 'undefined' // Firefox 3.5typeof(hello) == 'object' // IE 7
![Page 7: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/7.jpg)
![Page 8: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/8.jpg)
Attempt #1:Simulate events in JavaScript.
(Selenium, Windmill, Sahi)
![Page 9: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/9.jpg)
Selenium example.com
Attempt #1: Simulate events in JavaScript.
![Page 10: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/10.jpg)
Seleniumclick
example.com
Attempt #1: Simulate events in JavaScript.
![Page 11: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/11.jpg)
Selenium
http://test.example.comhttp://localhost:8080
clickexample.com
Attempt #1: Simulate events in JavaScript.
![Page 12: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/12.jpg)
Selenium
http://test.example.comhttp://localhost:8080
clickexample.com
Cross-domain security policy
Attempt #1: Simulate events in JavaScript.
![Page 13: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/13.jpg)
Selenium example.com
Attempt #1: Simulate events in JavaScript.
![Page 14: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/14.jpg)
Selenium example.com
test.example.com proxy
Attempt #1: Simulate events in JavaScript.
![Page 15: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/15.jpg)
Selenium example.com
test.example.com proxy/selenium-server/ /*
Attempt #1: Simulate events in JavaScript.
![Page 16: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/16.jpg)
clickSelenium example.com
test.example.com proxy/selenium-server/ /*
Attempt #1: Simulate events in JavaScript.
![Page 17: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/17.jpg)
Limitations
• Redirect to other domain or https
• Outside of DOM: pop-ups, Flash, ...
• Slow
![Page 18: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/18.jpg)
Attempt #2:Browser-specific automation
APIs.
(WebDriver, Watir)
![Page 19: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/19.jpg)
![Page 20: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/20.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
![Page 21: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/21.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
![Page 22: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/22.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
![Page 23: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/23.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
Watir
![Page 24: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/24.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
Watir
![Page 25: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/25.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
Watir
Selenium
![Page 26: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/26.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
Watir
Selenium
![Page 27: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/27.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
Selenium
WebDriver
Watir
![Page 28: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/28.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
Selenium WebDriver Watir
![Page 29: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/29.jpg)
Selenium WebDriver Watir
HTA/Proxy ✔ COM ✔ COM
Ext/Proxy ✔ Extension ✔ Extension
Proxy ✘ ✔ AppleScript
Proxy ✔ Extension ✔ V8 Debug
Proxy Dragonfly? ✘
✘ ✔ ✔ Celerity
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
![Page 30: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/30.jpg)
Watir
require 'watir'Watir::Browser.default = 'firefox'
describe 'Google' do before(:each) { @browser = Watir::Browser.new } after(:each) { @browser.close }
it 'should return search results for "hello world"' do @browser.goto "http://www.google.co.uk" @browser.text_field(:name, "q").set("hello world") @browser.button(:name, "btnG").click @browser.contains_text( "Hello world program - Wikipedia").should be_true endend
![Page 31: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/31.jpg)
WebDriver
require 'selenium-webdriver'
describe 'Google' do before(:each) { @browser = Selenium::WebDriver.for :firefox } after(:each) { @browser.quit }
it 'should return search results for "hello world"' do @browser.navigate.to "http://www.google.co.uk" @browser.find_element(:name, "q").send_keys("hello world") @browser.find_element(:name, "btnG").submit @browser.find_element(:partial_link_text, "Hello world program - Wikipedia") endend
![Page 32: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/32.jpg)
Selenium
gem 'selenium-client'; require 'selenium/client'
describe 'Google' do before(:each) { @browser = Selenium::Client::Driver.new :browser => "*firefox", :url => "http://www.google.co.uk" @browser.start } after(:each) { @browser.stop }
it 'should return search results for "hello world"' do @browser.open "/" @browser.type "q", "hello world" @browser.click "btnG", :wait_for => :text, :text => 'Results' @browser.is_text_present("Hello world program - Wikipedia"). should be_true endend
![Page 33: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/33.jpg)
Go Test It (≈Selenium)
require 'gotest'
gotest "http://www.google.co.uk" do open "/" type "q", "hello world" click "btnG" verify_text_present "Hello world program - Wikipedia"end
![Page 34: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/34.jpg)
Cucumber
Feature: Google search In order to learn something new As an enthusiast for obscure programming languages I want to find "Hello world" programs on Google
Scenario: Search for "hello world" Given that I am on google.co.uk When I enter "hello world" into the search box And I submit the form Then I should be shown a list of results And the page should contain "Hello world program - Wikipedia"
![Page 35: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/35.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
Watir
Selenium
![Page 36: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/36.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
Watir
Selenium
![Page 37: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/37.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
Watir
Selenium
![Page 38: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/38.jpg)
IE
Firefox
Safari
Chrome
Opera
HTMLUnit
WebDriver
Watir
Selenium
The future?
Watir API
Selenium API
![Page 39: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/39.jpg)
Ok.
![Page 40: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/40.jpg)
Fundamental goal:
Robust tests
![Page 41: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/41.jpg)
Choose your locators carefully
![Page 42: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/42.jpg)
Choose your locators carefully
id = searchInputname = search
xpath = //form[@id='searchform']/input[2]
![Page 43: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/43.jpg)
![Page 44: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/44.jpg)
![Page 45: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/45.jpg)
link = EU makes 7bn euro climate pledge
![Page 46: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/46.jpg)
link = EU makes 7bn euro climate pledge
xpath = //td[@class='text']/div[1]/a
![Page 47: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/47.jpg)
link = EU makes 7bn euro climate pledge
css = a.tsh
xpath = //td[@class='text']/div[1]/a
![Page 48: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/48.jpg)
People are complicated
⇒Frontend tests are
messy
![Page 49: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/49.jpg)
2 approaches:
Disposable tests(record/playback, quick&dirty)
Domain-specific abstractions(carefully crafted)
![Page 50: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/50.jpg)
Which approach will
you take?
![Page 51: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/51.jpg)
Thank you!
Martin KleppmannGo Test It
Email: [email protected]: yes-no-cancel.co.uk
Twitter: @martinkl
![Page 52: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/52.jpg)
Fundamental goal:
Parallel tests
![Page 53: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/53.jpg)
Don’t assume a clean database
![Page 54: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/54.jpg)
Don’t assume a clean database
Unit tests:
Load fixtures
Given...
When...
Then...
Reset database
![Page 55: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/55.jpg)
Don’t assume a clean database
Unit tests:
Load fixtures
Given...
When...
Then...
Reset database
Functional tests:
Set up example data
Do stuff
Checkoutcome
Do stuff
Checkoutcome
![Page 56: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/56.jpg)
Tolerate partially run tests
![Page 57: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/57.jpg)
Tolerate partially run tests
(Assume there is no frob)
Create frob
Check frob was created
Delete frob
![Page 58: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/58.jpg)
Tolerate partially run tests
(Assume there is no frob)
Create frob
Check frob was created
Delete frob
![Page 59: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/59.jpg)
Tolerate partially run tests
(Assume there is no frob)
Create frob
Check frob was created
Delete frob
Boom!
Bad:
![Page 60: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/60.jpg)
Tolerate partially run tests
(Assume there is no frob)
Create frob
Check frob was created
Delete frob
Boom!
Bad:
Delete frob if it exists
Create frob
Check frob was created
Delete frob (optional)
Better:
![Page 61: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/61.jpg)
Avoid race conditions: isolation
![Page 62: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/62.jpg)
Avoid race conditions: isolation
Delete frob if it exists
Create frob
Check frob was created
Delete frob if it exists
Create frob
Check frob was created
![Page 63: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/63.jpg)
Avoid race conditions: isolation
Delete frob if it exists
Create frob
Check frob was created
Delete frob if it exists
Create frob
Check frob was created
Log in as User A Log in as User B
![Page 64: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/64.jpg)
Avoid race conditions: isolation
Delete frob if it exists
Create frob
Check frob was created
Delete frob if it exists
Create frob
Check frob was created
Log in as User A Log in as User B
User A and User B: independent & not used concurrently
![Page 65: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/65.jpg)
Make test runs unique
![Page 66: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/66.jpg)
Make test runs unique
![Page 67: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/67.jpg)
Make test runs unique
![Page 68: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/68.jpg)
Take care with external services
![Page 69: Browser-level testing](https://reader034.vdocument.in/reader034/viewer/2022052321/554a12fbb4c905825d8b4c17/html5/thumbnails/69.jpg)
Thank you!
Martin KleppmannGo Test It
Email: [email protected]: yes-no-cancel.co.uk
Twitter: @martinkl