Continuous Delivery and Test Automation in Agile SW projects with Robot Framework 7.6.2013 Antti Pohjonen
© Copyright Knowit Oy 2013 | Public | Version 1.0
110
145 150
180
220
2007 2008 2009 2010 2011
Trusted Partner in Nordics
Experts in 1800 Sweden, Finland, Norway,
Denmark, Russia and Estonia
Knowit AB is listed on the Nordic Exchange in Stockholm
Gro
up
re
ve
nu
e (
ME
UR
)
More productivity in
information-intensive work
with efficient digital tools
Improved sales, collaboration
and communication with
advanced web solutions.
Better decisions based on
refined business information.
Shorter lead times and lower
costs in product development
through quality management
and flexible resourcing
We develop our customers competitiveness
1. Idea
The Traditional Way (Simplified!)
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 4
6. Fetch &
Compile 9. Fetch SW,
Testing
11. Exploratory
Testing
3. Code
Commit
How this could be done in a better way?
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 5
All human figures ©Pictofigo / www.pictofigo.com, Creative Commons Attribution-
NonCommercial-ShareAlike 3.0 Unported License
Overview
• Continuous Delivery
• Test Automation
• Robot Framework
• Case I: A large project from scratch
• Case II: Continuous Improvement
• How to start & build your own Test Automation?
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 6
Continuous Delivery
Based on Continuous Integration (CI)
"Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. " - Martin Fowler
Continuous Delivery just adds Test Automation to it
A Pipeline concept, from bottom to upper levels and layers
Focus on speed and automation!!!
Small difference, but significant, to Continuous Deployment
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 7
Continuous Delivery
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 8
Test Automation
More than just Automated Tests
Fully automated flow (pipeline)
DevOps (Development and Operations) tasks
• Own server and test environment administration
• Means involvement in to company politics
Requires CI system and some kind of test automation
framework / tool
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 9
Test Automation
Test automation contains steps like:
Continuous Integration, Commissioning, Integration tests,
Functional / System Tests, (User) Acceptance Tests
• And is triggered by external event: scheduled / commit etc.
Test automation developers should have a wide and good
understanding over used SW development process
Focus on Building Right It vs. Building It Right. Verification what is done vs. finding bugs
Test Automation development is Continuous activity
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 10
Robot Framework
Generic Test Automation framework • Many internal & external “ready made” test libraries • Uses keyword-driven testing approach • Can be used in vast various ways
• ATDD, BDD, Data-Driven Tests, Specification by Example etc.
• Implemented with Python • Can be extended natively using Python or Java
• Easy integration to other systems • Open source
• Sponsored by Nokia Siemens Networks
• Active development and growing community
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 11 Robot Framework © Copyright Nokia Siemens Networks
Creative Commons Attribution 3.0 License
Continuous Delivery & Test Automation
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 12
Continuous
Polling
Compilation &
Unit Tests
Context
Management
Automatic
Testing
1.
2.
3. 4.
5.
6.
7.
3.-7.
8.
Jenkins ©jenkins-ci.org Creative Commons Attribution 3.0 Unported License
Case I: A Large Project from Scratch
© Copyright Knowit Oy 2013 | Public | Version 1.0 17.6.2013 13
Case I: A Large Project from Scratch
Started as automated tests project with Robot Framework as
replacement for old testing tool.
• Goal was to make automated tests
Turned to Test Automation project by interest to execute tests
automatically (scheduled) in unbiased environment.
• Opportunity and curiosity, free resources
• Gained interest of management and other stakeholders
• Increased visibility and velocity
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 14
Case I: A Large Project from Scratch
Grew larger and more significant with time, more features
were injected and requested by management and users.
• Spread around development teams & sites.
• Became de facto, added to Way of Working
Next focus on Continuous Delivery, added Continuous
Integration as beginning of pipeline
After implementing CD, we had huge amount of testing data
• Focus moved on faster feedback and reporting
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 15
Case I: A Large Project from Scratch
Old product / project had:
• 2000-4000 semi automatic test cases
• Took around 2 weeks to execute
• High dependency to test environments
New product / project:
• over 3500 fully automated test case
• Took around 12 hours to execute
• Test environment independent by design
• Focus moved to reporting
• Continuous Test Automation development
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 16
Case I: A Large Project from Scratch
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 17
Case II: Continuous Improvement
© Copyright Knowit Oy 2013 | Public | Version 1.0 17.6.2013 18
Case II: Continuous Improvement
I participated to the project, when the second rewrite and
enhancement round of Robot Framework test libraries was
starting.
Test Automation was handled by development teams as side
activity, taking more time than they dare to spent on it.
Need for a new test automation core team was apparent
New team’s role was act as supportive team for SW
development teams
© Copyright Knowit Oy 2013 | Confidential | Version 1.0
Case II: Continuous Improvement
New team’s main goals were:
• Take over DevOps tasks
• Maintenance and enhancement of test libraries
• Enhance the Test Automation pipeline concept.
• Make Test Automation solution scalable and productize it.
• Act as "Community of Practise" for different projects
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 20
Case II: Continuous Improvement
Main Achievements:
• Developers can focus on their main tasks
• Better visibility and reporting
• Test Automation is spread to multiple new software projects
• Tool and technology changes are made easier
• Trust and "dependency" to Test Automation is increased
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 21
How to start & build your own Test Automation?
Little by little...
Aim high, but start from basics and easy tasks
Have fun and make little experiments first!!
Fail often and Fail fast!
Select best and the most suitable tools for your project, evaluate those before making final decision!
For Example: Use Jenkins as CI system tool and Robot Framework as test automation framework
Learn from others', don't repeat same mistakes again...
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 22
Thanks!!
Questions?
@AnttiPohjonen
LinkedIn: fi.linkedin.com/in/anttipohjonen/
17.6.2013 © Copyright Knowit Oy 2013 | Public | Version 1.0 23