seamless automated test management at scale with bol.com

Seamless Automated Test Management at Scale Organizing Massive Test Automation @ bol.com

Jeroen Ruijgers and

Viktor Clerc

Agenda

▪  Housekeeping

▪  About bol.com & XebiaLabs

▪  bol.com’s Challenge

▪  Test Automation @ bol.com

▪  Questions

Your Presenters

▪  Responsible for development, marketing, and support of test management and analytics solutions as part of its delivery automation product suite.

Jeroen Ruijgers Sr. Software Engineer, bol.com

▪  Worked as a (Java) software engineer on various Scrum teams since 2010.

▪  Currently member of a support team that focuses on test automation.

▪  Has given several trainings related to Java and test automation

Viktor Clerc Product Manager, XebiaLabs

Introducing

Started by Bertelsman Online

2015

bol.com becomes separate (and only)

entity

1999

6 employees

#1 online retailer in the Netherlands and Belgium

> 900 employees

2008

Started with Scrum

•  5 million customers •  9 million products

> 35 scrum teams

2001

5 Copyright 2015. Confiden4al – Distribu4on prohibited without permission

Introducing XebiaLabs

We help DevOps achieve Con4nuous Delivery at scale.

Sales in US, EMEA and APAC

Headquartered in Boston, MassachuseLs

200+ premiere customers


XebiaLabs Products

Visualize, orchestrate and manage your Con6nuous Delivery

pipelines from end to end.

Automate your applica6on deployments so releases occur in a repeatable, standard and efficient way.

Manage, analyze and op6mize all your test tools in one complete

view.

XebiaLabs products are designed to help Enterprise DevOps professionals achieve true Con'nuous Delivery at scale.


Key Features of XL TestView

Visualize and analyze test results, trends, and correlation in real-‐time

Collect results from – and control – all major test

tools

Automate go/no-‐go decisions

Optimize testing for value and speed

Install and configure in minutes

Positioning Test Automation as part of Continuous Delivery • Practice at bol.com

History of testing and test automation at bol.com

Organizing it all: making sense of test results • Practice at bol.com

What’s Up?

First Principle Agile Manifesto

“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”

Continuous Delivery The Three Basic Principles

Basic Principle 1: Rigorous Automation

Basic Principle 2: Extreme Feedback

Basic Principle 3: Continuous Change

Transformation to Test Automation SPECIFY DESIGN BUILD TEST INTEGRATE REGRESSION USER

ACCEPTANCE RELEASE

Acceptance Driven Testing

Development = TestTest = Development

Automate ALL

User Acceptance

Test effort

INTEGRATE REGRESSION USER ACCEPTANCE

•  Focus on Test Automation •  Special team with focus on improvements for all Scrum teams •  Teams do test automation themselves

•  Running Tests as Often as Possible •  Not only once a day, but also after deployments (to test) •  Focus on fast test à Fast feedback à Earlier insight in regression

•  Development of MayFly (Soon to be Open Sourced!) •  User story-based •  Creates separate dev and test environments per application •  More information @ Goto Amsterdam 2015

Continuous Delivery & Test Automation at bol.com

•  All testing done manually until 2010

•  2010 Started automating with QFTest

•  2012 Evaluated other tooling and started working with Around that time also developed in-house tool for testing databases

Where did we come from?

History of Testing at bol.com

•  FitNesse is mostly used •  It is Java based and we are a Java company •  The wiki format makes easy reading for business

•  New tools are introduced

Where are we now?

History of Testing at bol.com

Spock

•  Started with 1 project containing all tests •  Sharing knowledge •  Structured the same as our use cases, i.e.

WebshopSuite.BusinessAccountSuite.UseCase1500

•  Nightly runs from the beginning •  Indication by labels (nightly) •  First sequential per application

WebshopSuite •  Later parallel by split running of suites on the lowest level (a.k.a. suites with only tests)

WebshopSuite.BusinessAccountSuite.UseCase1500

FitNesse Implementation

•  Maven is used to bootstrap FitNesse •  In combination with a JUnit Runner

•  Specialized Jenkins job for parallel execution •  Knows the structure of FitNesse and fires corresponding jobs

•  The tests are executed by Jenkins Slaves •  Specialized Linux and Windows hosts with browsers •  Chrome, Firefox, IE8 and IE11

•  Suites are run and results are sent to the reporting tool •  Data stored in MySQL database •  Reports generated by BIRT

Nightly structure Running & Reporting

•  1 FitNesse project is too big •  More smaller projects need more maintenance

•  BIRT was getting very slow •  To much data to process •  Reports did not look very fancy

•  Re-running (failed) tests was not intuitive •  No links in reporting •  Need to know where to find it in Jenkins

Issues with the setup

Running & Reporting

•  One endpoint for reports

•  XL TestView takes over orchestration from Jenkins •  Custom scripting options for parallel execution of suites •  Custom graphs for bol.com •  We need a dashboard with all tests for a day, per team & application

•  Added analytics for test stability •  Reports on duration and flakiness

•  Developed in close collaboration between bol.com and XebiaLabs •  Tailored to the needs of bol.com •  Ideal kick-start of the development of (now) XL TestView

Introducing XL TestView at bol.com

Running & Reporting

Introducing XL TestView

The first test results management and analysis tool that allows you to define and execute tests across your full spectrum of test tools, then aggregate, and visualize results to provide insight into your application’s quality.

Test Results in Perspective

More Test Results! ;-)

Aligning Results for Go-Live Recommendations

Never Change a (Winning) Jenkins Setup

XL TestView at bol.com: Running & Reporting

•  Continuous Delivery needs Test Automation

•  Test automation done right needs structure and oversight!

•  XL TestView helped bol.com and can help you make sense of your test results

Key Take-Aways


Key Benefits of XL TestView

Visualize and analyze your entire software testing process.

Become more informed, organized and in control of your software testing process.

Intelligently optimize your testing to make it faster, more automated and more effective.

Quickly make confident, data-driven, go/no-go deployment decisions for smooth and successful software releases.


See for Yourself…

Learn more: https://xebialabs.com/products/xl-‐testview/ Free trial: hLps://xebialabs.com/download/xl-‐testview/

After inserting a picture, it should be placed behind de blue background. If not, click with your right mouse button in the picture and choose Send to Back

Thanks!

’til next bol.com!

Questions?

Jeroen Ruijgers Senior Software Engineer, bol.com

[email protected]

Viktor Clerc Product Manager, XebiaLabs

[email protected]

seamless automated test management at scale with bol.com

Documents

test test

massive test automation

positioning test automation

test environments

test automation special

coms challenge test

support of test management

design build test