experiences of test automation at spotify€¦ · •hard-to-test sut (experiences of test...
TRANSCRIPT
Kristian Karl, Spotify
Experiences Of Test Automation at Spotify
www.eurostarconferences.com
@esconfs#esconfs
Insert speaker picture here, no more than 150x150 pixels
October 16, 2013
Experiences Of Test Automation
@ SpotifyKristian Karl
Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds, by Henrik Kniberg & Anders Ivarsson
Why automate?
• We want to be faster
• Shorter release cycles
• Facilitate the testing in
agile dev cycle
From: ”Intelligent Test Automation” by Harry Robinson
What to automate?
Graphical user interface testingUsability testingSoftware performance testingSystem testingFunctional testingLoad testingVolume testingStress testingSecurity testingScalability testingSanity testing
Unit testingSmoke testingComponent testingAPI testingRegression testingInstallation testingMaintenance testingRecovery and failover testing.Accessibility testingMonkey testingIntegration testing
Graphical user interface testing
Our challenges• Hard-to-test SUT
(Experiences of test automation: case study 1, An Agile Team’s Test
Automation Journey: The First Year), Dorothy Graham & Mark
Fewster)
• Maintenance of automation
• Peoples expectations
• Flaky SUTs
• Flaky tests
• Re-prioritizations of non-critical but test hindering bugs
• Test data, test environments
• Supporting services
Our goals• Create automated end-user regression tests on 4 major
platforms1) Desktop – Windows and OSX
2) iOS – iPhone and iPad
3) Android
4) Webplayer
• To facilitate the workload for testers
• To deliver automated regression tests for a feature as a part of
definition of done
• To deliver short feedback loops to teams using Dashboards
Model-based testing
• The models are the abstraction layer
• The testers designs the automation
using models
• The developers implements the code
of the automation
Login view displayed
Main view displayed
Valid credentials
App not running
Start app
Log out
Close
Exit app
Invalid credentialsToggle
’Remember Me’
Start app
public interface SimpleLogin {
public void e_Close();
public void e_Exit();
public void e_Init();
public void e_InvalidCredentials();
public void e_Logout();
public void e_StartClient();
public void e_ToggleRememberMe();
public void e_ValidPremiumCredentials();
public void v_ClientNotRunning();
public void v_LoginPrompted();
public void v_WhatsNew();
}
Dem
o
Developers and developers
• Why not use developers for TA?
• Why use developers for TA?
• Using Test API’s
• Defined by TA
• Implemented by developers
Before
android.view.View seekBarView =
solo.getView(com.spotify.mobile.android.ui.view.CancellableSeekBar.class, 0);
int[] xy = new int[2];
seekBarView.getLocationOnScreen(xy);
solo.clickOnScreen(xy[0] + 9 + (seekBarView.getWidth() - 18) * position, xy[1] +
seekBarView.getHeight() / 2.0f);
After
page().seekTrack(position)
Supporting services
• Continuous testing
• QA Lab
• TDS – Test Data Service
• TRS – Test Result Service
Open source tools that we use
• yEd – editing FSM models [www.yworks.com]
• GraphWalker – generate test sequences from models [graphwalker.org github.com/spotify/python-graphwalker]
• Sikuli – image recognition tool [www.sikuli.org]
• NuRemote - communicating with iOS client [github.com/nevyn/NuRemoting]
• TestNG – group tests into suites and and run them [testng.org]
• Java – code implementation
• Jenkins – [test] job scheduler 24/7 [jenkins-ci.org]
Demo
Q & A