autoscaled distributed automation expedia know how
TRANSCRIPT
![Page 1: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/1.jpg)
AUTOSCALED DISTRIBUTED AUTOMATION
SELENIUM GRID / AWS
Ragavan Ambighananthan@ragsambi
Expedia
Expedia Know How 2016!
AKA ‘RUNNING TESTS WITHIN THE TIME TAKEN BY THE SLOWEST TEST CASE’
![Page 2: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/2.jpg)
WHAT DO I GET?• Autoscaled Distributed Automation(Selenium Grid /
AWS) • DA will phenomenally shorten the UI automation
run time• Faster feedback cycle• Fewer Jenkins jobs to run automation, instead of
few hundreds• Cost effective and reliable• Enables Continuous Integration / Continuous
Deployment
![Page 3: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/3.jpg)
AGENDA
• Setting up
• Making the Grid stable
• Grid topologies
• Cost saving
• Reducing UI Tests
• Reporting / Dashboard
![Page 4: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/4.jpg)
PROBLEM DESCRIPTION
TOO MANY UI TESTS
![Page 5: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/5.jpg)
PROBLEM DESCRIPTION
SLOW TEST / EXECUTION
![Page 6: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/6.jpg)
PROBLEM DESCRIPTION
• Hundreds of Jenkins jobs to run all the tests• Not having a system to run vast amount of UI
automation reliably, fast and scalable in a cost effective way is a blocker for CI / CD
• No intelligent automation report to narrow down failures quickly!
![Page 7: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/7.jpg)
SOLUTION
• To be able to run all UI automation scenarios within the time taken by the slowest test case
• Cost effective, auto scalable and reliable
• Teams focussing on automation• Note: This is not about cross browser test coverage rather using
grid for parallel test execution
![Page 8: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/8.jpg)
SETTING UPTECHNOLOGIES / TOOLS USED
SELENIUMGRIDSCALER
![Page 9: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/9.jpg)
SETTING UPBIG PICTURE
![Page 10: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/10.jpg)
SETTING UP
• Cucumber allows to run a scenario with the following syntax
• sample_featurefile.feature:12• For Scenario Outline, the line number would
be that of the line from the example table
line no 12 Scenario: eat 5 out of 12 13 Given there are 12
cucumbers 14 When I eat 5 cucumbers 15 Then I should have 7
cucumbers
CUCUMBER SCENARIO GENERATION
![Page 11: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/11.jpg)
SETTING UP
checkout/lx: features/lx_fraud.feature:21:en_US features/lx_fraud.feature:47:en_US features/lx_responsive_design.feature:25:en_US features/lx_responsive_design.feature:26:en_US features/lx_responsive_design.feature:27:en_US features/lx_responsive_design.feature:90:en_US features/lx_responsive_design.feature:240:en_USsearch_landing_pages/flights_tg: features/tg_flights_revamp_hero_image.feature:120:en_US features/tg_flights_revamp_social_sharing.feature:156:en_US features/tg_flights_revamp_search_wizard.feature:202:en_US features/tg_flights_revamp_search_wizard.feature:203:nl_NL features/tg_flights_revamp_top_destinations.feature:159:en_US features/tg_flights_revamp_top_destinations.feature:160:en_US features/tg_flights_revamp_top_destinations.feature:161:en_US features/tg_flights_revamp_top_destinations.feature:207:en_US
• Only scenarios that matches @stubbed | @live and @acceptance | @regression will be included in the list to run
• All these tests will be executed concurrently
SAMPLE GENERATED SCENARIOS
![Page 12: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/12.jpg)
SETTING UP
./gradlew -PnumBrowsers=150 :loyalty.ui:scalaAcceptance -i -Denvironment=JENKINS_STUBBED -Dbrowser=Grid
SAMPLE GENERATED SCENARIOS
![Page 13: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/13.jpg)
SETTING UP
• c3.4xlarge (16 cpu / 30 GB RAM / High BW)• Node should have high network bandwidth
but low CPU / Memory is fine• Running SeleniumGridScaler jar, which will
act as the hub that can autoscale• https://github.com/mhardin/SeleniumGridScaler
SELENIUM GRID HUB SETUP
![Page 14: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/14.jpg)
SETTING UP
• Open Source• Acts as an intelligent hub• Auto scales grid nodes depending on the
number of tests• Terminates nodes when not in use• Adhoc launch of new nodes is also possible• Talks to AWS using EC2
SELENIUMGRIDSCALER - HUB
![Page 15: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/15.jpg)
• c3.xlarge• Capable of running maximum 24 Firefox• Number of Chrome that can be run is lesser• Node created out of AMI has bootstrap code
to help attach to the hub
SETTING UPSELENIUM GRID NODE SETUP
![Page 16: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/16.jpg)
SETTING UP
• Node has bootstrap code to start and attach to hub
• Either you have to get the node AMI or create an AWS instance, bootstrap it,create an AMI out of it and refer it in the Hub config.
• Hub creates the node based on a config: AMI ID, subnet, security group, node
type,etc.
SELENIUMGRIDSCALER - NODE
![Page 17: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/17.jpg)
SELENIUM NODE BOOTSTRAP CODE
[root@ip-10-2-12-167 ~]# more /home/grid/grid/grid_start_node.sh#!/bin/shPATH=/sbin:/usr/sbin:/bin:/usr/bincd /home/grid/gridexport EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id || die \"wget instance-id has failed: $?\"`"# Pull down the user data, which will be a zip file containing necessary informationexport NODE_TEMPLATE="/home/grid/grid/nodeConfigTemplate.json"curl http://169.254.169.254/latest/user-data -o /home/grid/grid/data.zip# Now, unzip the data downloaded from the userdataunzip -o /home/grid/grid/data.zip -d /home/grid/ubuntu/grid# Replace the instance ID in the node config filesed "s/<INSTANCE_ID>/$EC2_INSTANCE_ID/g" $NODE_TEMPLATE > /home/grid/grid/nodeConfig.json# Finally, run the java process in a window so browsers can runxvfb-run --auto-servernum --server-args='-screen 0, 1600x1200x24' java -jar /home/grid/grid/selenium-server-node.jar -role node -nodeConfig /home/grid/grid/nodeConfig.json -Dwebdriver.chrome.driver="/home/grid/grid/chromedriver" -log /home/grid/grid/grid.log &
![Page 18: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/18.jpg)
MAKING THE GRID STABLE
• Timeouts in json config• “timeout”:240000 (ms)• “browserTimeout”:390000 (ms)• browserTimeout has to be bigger than
‘timeout’ and ‘webDriver’ timeout• browserTimeout is specified in secs in
command line
TIMEOUTS
![Page 19: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/19.jpg)
• If browser instance hangs (for any reason what so ever), it will take 3hrs (http client socket timeout) for the particular slot to become free.
• This timeouts the Jenkins job• Solution:
• Fix the particular test scenario causing this issue• Add a cronjob to kill any browser instances that is running
for more than 10mins. • Make this as part of your Chef knife plugin• Ref: selenium repo, PR: 227 / 285
MAKING THE GRID STABLETIMEOUTS
![Page 20: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/20.jpg)
• Grid setup should be in the same AWS subnet• Using multiple subnets will result in lots of
FORWARDING_TO_NODE_FAILED errors
MAKING THE GRID STABLEAWS - SUBNET
![Page 21: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/21.jpg)
• Subnet you are using should have enough free IP addresses
• It will be a blocker for autoscaling the grid nodes
MAKING THE GRID STABLEAWS - IP ADDRESS
![Page 22: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/22.jpg)
• The webDriver object creation consumes bandwidth in the range of 6Gbits/5min in the Hub for 250+ tests in parallel
MAKING THE GRID STABLEAWS - HUB BANDWIDTH
c3.4xlarge bandwidth is “High”
![Page 23: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/23.jpg)
• Fine tune your • -Xms • -Xmx • -DPOOL_MAX
MAKING THE GRID STABLEAWS - HUB / NODE MEMORY
![Page 24: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/24.jpg)
• HUB becomes unstable after running thousands of tests
• Automate restarting of Hub after every 2000+ tests
MAKING THE GRID STABLEAWS - RESTARTING HUB
![Page 25: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/25.jpg)
• Jenkins executor which would be running hundreds of tests in parallel, needs to have enough CPU power.
MAKING THE GRID STABLEAWS - JENKINS EXECUTOR CPU
c3.8xlarge when running 250+ tests in parallel
![Page 26: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/26.jpg)
• Don’t rely too much on Selenium Grid’s queuing policy
• If your average test execution time is greater than webDriver timeout, tests will timeout at webDriver creation itself
MAKING THE GRID STABLEHUB QUEUING POLICY
![Page 27: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/27.jpg)
• Update browsers in the node and create a new node AMI
• Necessary browser settings:
MAKING THE GRID STABLEUPDATE BROWSERS
profile =Selenium::WebDriver::Firefox::Profile.new profile['app.update.auto'] = false profile['app.update.enabled'] = false profile['app.update.service.enabled'] = false profile['dom.max_script_run_time'] = 60 profile['dom.max_chrome_script_run_time'] = 60 profile['focusmanager.testmode']=true profile['accept_untrusted_certs']=true profile['assume_untrusted_certificate_issuer'] = false
![Page 28: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/28.jpg)
MAKING THE GRID STABLESCALE THE TEST INFRASTRUCTURE
![Page 29: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/29.jpg)
GRID TOPOLOGIES• Decide what you want before selecting the topology to be cost efficient!• I want to release code to production ..
1. Every CL (change list)2. Once a day3. Once a week4. When ever I want (on demand!)
• Based on the above answers, Do I want to run all UI automation for 5. Every CL ?6. Every 2 hours7. Four times a day8. Once a week
![Page 30: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/30.jpg)
GRID TOPOLOGY - 1
HUB
Jenk ins J ob
• parallel execution for small projects• 1 executor - 1 hub - 14 nodes• eg: c3.8xlarge can execute 250*+ tests in parallel• Test run would finish in ~5mins
c3.8xlarge
c3.4xlarge
c3.xlarge
….
![Page 31: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/31.jpg)
GRID TOPOLOGY - 2
HUB
Job Execu tor
Job Execu tor
• Suitable for medium size projects (500+ tests)
• Adding one more executor (2 executors 1 hub and 28 node),this could double your parallel execution cases, still taking only ~5mins
c3.8xlarge
c3.8xlarge
c3.xlarge
….
….
![Page 32: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/32.jpg)
GRID TOPOLOGY - 3
HUB
• Takes 2x times as previous topology, but half the cost! (1 executor - 1 hub - 14 nodes)
• Suitable for medium size projects• Test run would finish in ~10mins
Job Execu tor
Job Execu tor
c3.8xlarge
c3.xlargejob runs sequentially….
c3.4xlarge
![Page 33: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/33.jpg)
GRID TOPOLOGY
HUB
Job Execu tor
Job Execu tor
• One more job? Probably NOT as HUB network traffic would make it unstable especially during webDriver creation
• c3.8xlarge network bandwidth limit is 10Gbit
c3.8xlarge
c3.8xlarge
c3.xlarge
….
….
![Page 34: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/34.jpg)
GRID TOPOLOGY - 4
HUB
HUB
• Use two hubs to
double the tests
(1000+)• But speed is same
as topology 2
(~5mins)• Double the cost
c3.8xlarge
c3.xlarge
c3.8xlarge
c3.8xlarge
![Page 35: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/35.jpg)
COST SAVING
![Page 36: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/36.jpg)
OPTIMAL USE OF GRID NODES
• Running 250+ tests on a grid setup with 250 slots will take around 5mins
• Nodes are idling for the remaining 55mins of time which is already billed by AWS
• Even during the 5mins of run, only very minority of the tests takes around 4mins and majority of the test complete in less than 1 min
COST SAVING
![Page 37: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/37.jpg)
OPTIMAL USE OF GRID NODESCOST SAVING
![Page 38: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/38.jpg)
• On a c3.8xlarge 250 tests can be run at one go before all 32 CPU reach 100%
• Start 250 cases• Then between every ~50 seconds, start 100
tests in batch, repeat this until all tests are executed
• Fine tune the delay according to your observation
BATCH PROCESSINGCOST SAVING
![Page 39: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/39.jpg)
GRID TOPOLOGY - BATCH PROCESSING
HUB
• Cost saving topology 1 executor - 1 hub - 16 nodes• Can run any number of tests• Can run 5500 UI tests within ~1hr 40min
job runs sequentially
c3.8xlarge c3.xlarge
COST SAVING
c3.4xlarge
![Page 40: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/40.jpg)
PIPELINE
HUB
CI build
Deploy Job
CI Stubbed
acceptance stub regression stub
rest
art
hub
auto
scal
e no
des
1hr 40min
![Page 41: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/41.jpg)
COMPARING AWS COST VS DATA CENTRE• 1 Medium box (~$8000 / per month)• 1 Large box (~$10000 / per month)• 1 VM (~$2000 / per month)• Total AWS cost for 2 Batch Processing
Topologies• ~$1300 / month (fully autoscaled and runs
9000+ UI test)• Frequency: 9-11 times a day
COST SAVING
![Page 42: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/42.jpg)
AUTOSCALING OF GRID NODES
• SeleniumGridScaler autoscales the grid nodes• Nodes are created on demand (autoscaled)• It creates only required number of nodes for
running the available tests• Nodes are terminated in an optimal way• Autoscaling is cheaper than stop/starting
the nodes (outside of SeleniumGridScaler plugin)
COST SAVING
![Page 43: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/43.jpg)
• http://x.x.x.x:4444/grid/admin/AutomationTestRunServlet?uuid=testRun1&threadCount=275&browser=firefox”
• For 275 test cases, it will create 275/24 == 12 nodes
• It returns status codes
• 202 - request can be fulfilled by current capacity
• 201 - request can be fulfilled but AMI must be started to meet capacity (wait for ~5mins)
AUTOSCALING OF GRID NODESCOST SAVING
![Page 44: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/44.jpg)
REDUCING UI TESTSMONITOR UI TEST WITH STRICT REVIEW PROCESS
![Page 45: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/45.jpg)
REDUCING UI TESTSBREAK DOWN BIGGER SCENARIOS
![Page 46: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/46.jpg)
REDUCING UI TESTS
• Create more unit / integration tests• Categorize test cases appropriately• Each test should focus only on one use case
![Page 47: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/47.jpg)
REPORTING / DASHBOARD• All automaton results are stored in MongoDB• html/json report / failure screenshots, splunk
query, failure status,etc• Nodejs / Express / Hightchart based dashboard
for viewing• RSS feed for every projects so teams can
subscribe to them. Feed has html report / screenshot / war_file version / splunk query
• HipChat notification
![Page 48: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/48.jpg)
REPORTING / DASHBOARDTREND CHARTS
![Page 49: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/49.jpg)
REPORTING / DASHBOARDPOINT OF SALE GRID
![Page 50: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/50.jpg)
REPORTING / DASHBOARDUNIQUE ERROR REPORT
![Page 51: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/51.jpg)
REPORTING / DASHBOARDFAILURE HISTORY / ONE PAGE
![Page 52: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/52.jpg)
REPORTING / DASHBOARDHIPCHAT NOTIFICATION
![Page 53: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/53.jpg)
FEW WORDS
• Few differences in Expedia specific SeleniumGrid Scaler
• https://github.com/ambirag/SeleniumGridScaler, branch: SeleniumGridScalerExp
• Dockerised!
![Page 54: Autoscaled Distributed Automation Expedia Know How](https://reader038.vdocument.in/reader038/viewer/2022102615/586ff8791a28ab92198b767f/html5/thumbnails/54.jpg)
QUESTIONS
?