Quality Assurance Continuous
…using Selenium WebDriver
San Francisco Selenium Meetup Group - May 16, 2013
Fabrizio Branca Lead System Developer at
San Francisco, CA
Janine
Fiona
that’s me
It’s a boy!
E-Commerce: Magento
CMS: TYPO3
Portals: ZF, FLOW,…
Mobile Searchperience: SOLR
120 people in 7 offices world-wide
High Performance
/Scale
Global Enterprise Projects
Continuous Integration
Deployment
Development
Delivery
Improvement
Quality Assurance Continuous
Everything sounds better with “continuous”
“Continuous” == “Doing stuff over and over again”
Builds Deployment Tests Infrastructure
Setup
Devbox Setup
== “Automation”
tim
e s
pe
nt
task size
does it manually
does it manually
gets annoyed
writes script to automate
runs script
wins
loses
Geeks vs. Non-Geeks
Makes fun of geek’s complicated method
— Ovidiu Platon
We are not shipping your machine! if it works on your machine! I don’t care
Picture The BIG
Vagrant Boxes
Awesome stuff
Every single line potentially
can break production!
Bratwurst (Germans)
Donut (Americans)
Pizza (Italians)
Version Control
GIT, SVN,…
PHP lint
Version Control Code Reviews
“Gerrit”
Version Control Code Reviews Build
Files
DB
Installer
Settings
Triggered by commit, by time
or manually
Jenkins
minify js/css
.tar.gz
“virtual package” (revision)
Multiple Release Branches?*
Multiple Build Pipelines!
…
…
… *http://nvie.com/posts/a-successful-git-branching-model/
Version Control Code Reviews Build
Files
DB
Installer
Settings
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
CodeSniffer
php-pmd
php-cpd
php-depend
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration”
First system where code from all
developers/teams meets each other
unstable system. Might be broken. Will be rebuild
without warning
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests
PHPUnit
Code coverage
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests
AOE PMD (for Magento)
broken links, HTTP headers, W3C validity,…
Test features that rely on other parts (e.g. framework, external
services,…) External components (SSL, Redis, Varnish,…)
Basically this is also PHPUnit, but
addressing other stuff
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Here we go:
Selenium!
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Install on
“Staging”
Behat (Cucumber)
Same setup as production, but different machines
Client QA / approval
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Install on
“Staging”
*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning
Stress
Tests
Jmeter
Spin up EC2 instances in the cloud to produce
traffic*
Again: multiple servers or autoscaling setup
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Install on
“Staging”
Stress
Tests
Install on
“Production”
A/B Deployments on different
scales
Plan B: rollback
Zero Downtime
Varnish Array EC2
Frontend Array Backend Array EC2 EC2
EC2
Worker Array EC2 EC2
Deployment x
ELB Load Balancer
Route 53 DNS-Service
shop.angrybirds.com
Deployment
Varnish Array EC2
Frontend Array Backend Array EC2 EC2
EC2
Worker Array EC2 EC2
Deployment x
ELB Load Balancer
Varnish Array EC2
Frontend Array Backend Array EC2 EC2
EC2
Worker Array EC2 EC2
Deployment x+1 Release Y
ELB Load Balancer
Route 53 DNS-Service
shop.angrybirds.com
Deployment
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Install on
“Staging”
Stress
Tests
Install on
“Production”
Deployment != Feature Release
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Install on
“Staging”
Stress
Tests
Install on
“Production” Enable Features
Version
Control Code
Reviews Build Static Code
Analysis
Install on
“Integration”
Unit Tests Integration
Tests
Acceptance
Tests
Install on
“Staging”
Install on
“Production” Enable
Features
Pipeline Visualization
#284
#283
#282
#281
#280
Build Static Code
Analysis Unit Tests
Install on “Integration”
Integration Tests
Acceptance Tests
#285
for team dashboards
Acceptance Tests
test automation is a developer’s task
we need to provide them some good tools…
us
Selenium 1 vs. Selenium 2
What’s out there? Selenium for PHP
PHPUnit_Selenium (Sebastian Bergmann)
WebDriver-PHP (Chibimagic)
Php-webdriver-bindings (Lukasz Kolczynski)
php-webdriver
PHPUnit_Selenium (Giorgio Sironi)
php-webdriver Selenium 2
08/2011 Facebook/php-webdriver (Justin Bishop)
Element-34/php-webdriver (Adam Goucher)
instaclick/php-webdriver (Anthon Pang)
php-webdriver
“A very thin wrapper of WebDriver”
So what?
A thin layer is good! But not enough…
no test logic
no higher level components
dealing with Selenium specific tasks
not compatible to legacy test cases
No abstraction / structure
Architecture
HTTP AUT
No direct access (DB, classes,…)
REST* (over HTTP)
*http://code.google.com/p/selenium/wiki/JsonWireProtocol
Project test cases
PHPUnit
Project Components
Components Library
Menta
php-webdriver
reusable
project specific
Menta Selenium 2 Testing Framework for PHP
Who is “Menta”?
What is “Menta”?
Why “Menta”?
Features Menta The framework.
Not the cat…
Components Session Management
? Bonus Features :) Event / Observer Configuration Management
PHPUnit integration
Why?
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
Data Providers
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
Dependent Tests
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
Test Infrastructure
Menta for PHPUnit Abstract test class
PHPUnit_Framework_TestCase Menta_PHPUnit_Testcase_Selenium2 ProjectY_Abstract_TestCase
Very thin layer on top of PHPUnit
ProjectY_TestCase_Registration ProjectY_TestCase_Checkout the “magic” happens in the components
Legacy Selenium 1 tests PHPUnit_Framework_TestCase PHPUnit_Extensions_SeleniumTestCase
ProjectY_TestCase_Registration ProjectY_TestCase_Checkout
Legacy Selenium 1 tests PHPUnit_Framework_TestCase Menta_PHPUnit_Testcase_Selenium2
ProjectY_TestCase_Registration ProjectY_TestCase_Checkout
Menta_PHPUnit_Testcase_Selenium1
Selenium1Facade
Compatible with PHPUnit_Extensions_SeleniumTestCase
HTML Report Screenshots (on demand
and on failure)
test description from phpdoc
additional messages
error details
HTML Report “Duration Heatmap”
shows long running tests
Menta for PHPUnit Text Result Printer
PHPUnit 3.7.20 by Sebastian Bergmann. Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml > SUITE: > SUITE: > SUITE: Acceptance_Tests_Account_Register > TEST: Acceptance_Tests_Account_Register::register SUCCESS. (Duration: 8.18 sec) > TEST: Acceptance_Tests_Account_Register::login SUCCESS. (Duration: 6.5 sec) > TEST: Acceptance_Tests_Account_Register::testRegistrationMail FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22 sec) < Duration: 36 sec < Duration: 36 sec > SUITE: > SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest > TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Failed asserting < Duration: 9.87 sec > SUITE: Acceptance_Tests_Checkout_CheckoutTest > TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView SUCCESS. (Duration: 1.79 sec) > TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::checkout" to pass. (Duration: 5.34 sec)
Components?
Components!
Manager Component
Menta_ComponentManager::get(‘PageObject_Cart’) PageObject_Cart
Rewrites Component
PageObject_Cart ProjectZ_PageObject_Cart
extends
change locators
add functionality
change behavior
overwrite labels
Manager Component
Menta_ComponentManager::get(‘PageObject_Cart’) ProjectZ_PageObject_Cart
Menta_ComponentManager::addRewrite(
‘PageObject_Cart’,
‘ProjectZ_PageObject_Cart’
);
Components
Common Assert Wait Screenshot Selenium1Facade
Imap WebService W3C Validator Curl …
Some might not even require a Selenium session
Gallery Screenshot
Components! PageObjects?
PageObjects Components
{ } }⊇{
PageObjects are
components
PageObjects?
represents services offered by a page or feature
is the only thing with a deep knowledge of
the HTML
clickCheckoutButton() isInCart($product) getTotals()
✔ ✔ ✔
Component Libraries
Menta
Magento
(your library)
Project-
specific
Magento
(community-provided)
General
(your library)
Session Management
Let Menta manage your sessions
auto-close
auto-start
incl. on exceptions and ctrl+c
Reusing sessions
Cleaning sessions
Configuration Management
Fallback mechanism for global settings
Create multiple configurations for different browsers,
application instances or Selenium Servers
Compatible to PHPUnit (xml)
Event / Observer Menta_Events::dispatchEvent('after_session_create');
Menta_Events::addObserver('after_session_create', function($session) {
$session->window('main')->postSize(array('width' => 1024, 'height' => 768));
}
Bonus Features
Sauce Labs Support
*configuration: http://username:[email protected]:80/wd/hub
works fine*
update test result (REST)
Sauce Labs Support
Speaking name Test results
Perceptual diffs Build X
Build X-1
Perceptual Diff
github.com / AOEmedia / Menta github.com / AOEmedia / Menta_SampleProject
It’s composer based and you’ can get started in a few minutes!
Thank you! Any questions?
http://www.aoemedia.com
http://www.fabrizio-branca.de
@fbrnc Follow me on twitter!
My blog