"experiences of test automation at spotify" with kristian karl
DESCRIPTION
View webinar: http://www.eurostarconferences.com/community/member/webinar-archive/webinar-87-experiences-of-test-automation-at-spotify- At Spotify, we want the manual testing effort to be focused as much as possible at feature testing, less on regression tests. But we still have to do regression. So, we tried to automate a big chunk of that. Regression tests are run on our Desktop, Android, iOS and WebPlayer clients, and also some backend services. I will share with you how far we have come. What techniques, tools and methodologies we have tried. What experiences has been good, and what has been not that good.TRANSCRIPT
![Page 1: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/1.jpg)
December 2, 2013
Experiences Of Test Automation
@ Spotify Kristian Karl
![Page 2: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/2.jpg)
![Page 3: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/3.jpg)
Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds, by Henrik Kniberg & Anders Ivarsson
![Page 4: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/4.jpg)
![Page 5: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/5.jpg)
Why automation?
• We want to move faster
• Quick dev feed-back
• Continous Delivery
• Continously monitor the
state of the product
• Bug hunting From: ”Intelligent Test Automation” by Harry Robinson
![Page 6: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/6.jpg)
What to automate?
Graphical user interface testing Usability testing Software performance testing System testing Functional testing Load testing Volume testing Stress testing Security testing Scalability testing Sanity testing
Unit testing Smoke testing Component testing API testing Regression testing Installation testing Maintenance testing Recovery and failover testing. Accessibility testing Monkey testing Integration testing
![Page 7: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/7.jpg)
What to automate?
Graphical user interface testing
![Page 8: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/8.jpg)
Our goals
• Create automated end-user regression tests on 4 major
platforms 1) Desktop – Windows and OSX
2) iOS – iPhone and iPad
3) Android
4) Webplayer
• Help testers, not replacing them
• Deliver automated regression tests for a feature as a part of
definition of done
• Deliver short feedback loops to teams using Dashboards
![Page 9: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/9.jpg)
Our challenges
• Hard-to-test SUT (Experiences of test automation: Case Study 1, An Agile Team’s Test
Automation Journey: The First Year), Dorothy Graham & Mark Fewster)
• Maintenance of automation
• Expectations
• Flaky tests... or is it flaky SUT?
• Testability
• Test data, test environments
• Supporting services
![Page 10: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/10.jpg)
![Page 11: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/11.jpg)
Model-based testing
• Models are the abstraction layer
• Testers design the automation using
models
• Developers implements the automation
using the models as drivers
![Page 12: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/12.jpg)
Login view displayed
Main view displayed
Valid credentials
App not running
Start app
Log out
Close
Exit app
Invalid credentials Toggle
’Remember Me’
Start app
![Page 13: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/13.jpg)
![Page 14: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/14.jpg)
public interface SimpleLogin {
public void e_Close();
public void e_Exit();
public void e_Init();
public void e_InvalidCredentials();
public void e_Logout();
public void e_StartClient();
public void e_ToggleRememberMe();
public void e_ValidPremiumCredentials();
public void v_ClientNotRunning();
public void v_LoginPrompted();
public void v_WhatsNew();
}
![Page 16: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/16.jpg)
Test automator
• Professional Java developer.
• Test experience is not mandatory.
• Embedded in the squad (team).
• Test automators form their own Guild
![Page 17: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/17.jpg)
Developers and developers
• Why not use developers for TA?
• Why use developers for TA?
• Test API’s
- Defined by TA
- Implemented by developers
![Page 18: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/18.jpg)
Section name 18
Test interface + wrappers
Client
Before
Test model Test model Test model
QA/TA
Devs
Strings
Java test interface
Client
Test model Test model Test model
ObjC test interface
JSON
After
![Page 19: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/19.jpg)
Before
String cmd = "android.view.View seekBarView = solo.getView(com.spotify.mobile.android.ui.view.CancellableSeekBar.class, 0);"; cmd += "int[] xy = new int[2];"; cmd += "seekBarView.getLocationOnScreen(xy);"; cmd += "solo.clickOnScreen(xy[0] + 9 + (seekBarView.getWidth() - 18) * " + position + "f, xy[1] + seekBarView.getHeight() / 2.0f)"; BeanRemoteClient.sendToServer(cmd);
After
PlayerAuto player = Pages.remote(PlayerAuto.class);
player.seekTrack(position);
![Page 20: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/20.jpg)
Supporting services
• Continuous testing
• QA Lab
• TDS – Test Data Service
• TRS – Test Result Service
• Virtualization
![Page 21: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/21.jpg)
![Page 22: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/22.jpg)
Open source (and freeware) tools that we use
• yEd – editing FSM models [www.yworks.com]
• GraphWalker – generate test sequences from models [graphwalker.org github.com/spotify/python-graphwalker]
• Sikuli – image recognition tool [www.sikuli.org]
• NuRemote - communicating with iOS client [github.com/nevyn/NuRemoting]
• TestNG – group tests into suites and and run them [testng.org]
• Java – code implementation
• Jenkins – [test] job scheduler 24/7 [jenkins-ci.org]
![Page 23: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/23.jpg)
De
mo
![Page 24: "Experiences Of Test Automation At Spotify" with Kristian Karl](https://reader034.vdocument.in/reader034/viewer/2022042512/554f3869b4c905cd048b4ec9/html5/thumbnails/24.jpg)
Q & A