continuous delivery testing @hiq
DESCRIPTION
Talk about Continuous Delivery and Testing. Held at HiQ april 2014.TRANSCRIPT
![Page 1: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/1.jpg)
Continuous Delivery
Continuous Testing
![Page 2: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/2.jpg)
Tomas Riha
Architect @ VGT/WirelessCar
Passionate about creativity, change and improvementHorrible at following instructions and performing repetitive tasks
MAJOR Project Liability
mail: [email protected]: @TomasRihaSE
blog: continuous-delivery-and-more.blogspot.com
![Page 3: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/3.jpg)
Agenda
Why do we want to do Continuous DeliveryIntro to Continuous Delivery
Principles of Continuous DeliveryLook at a Pipe
Test Automation for Continuous DeliveryTest Driven Development and Continuous Delivery
Exploratory Testing and Continuous DeliverySome words on Tooling
Areas not covered
![Page 4: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/4.jpg)
Why do we want to do Continuous Delivery
Our highest priority is to satisfy the customerthrough early and continuous delivery
of valuable software.
![Page 5: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/5.jpg)
Why do we want to do Continuous Delivery
Decreased lead time not only improves our time to market but also our ability to learn and improve through a faster
feedback loop
![Page 6: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/6.jpg)
Why do we want to do Continuous Delivery
As soon as we have added value to the application it needs to be in the hands of the customer or else we have just
added cost.
![Page 7: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/7.jpg)
Why do we want to do Continuous Delivery
Reduced risk through delivery of small changes.
![Page 8: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/8.jpg)
Are we really delivering as soon as possible?
Pre Planning
Dev Sys Test Reg Test
Pre Planning
Dev Sys Test Reg Test
Scrummerfall happens because its hard for developers to have something deployable for the testers to test.
System testing on something that has not been regression tested is fundamentally flawed.
Cost of regression test tends to bloat sprint content.
Sprint 2-4 weeks
![Page 9: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/9.jpg)
Continuous Delivery
Build Release Deploy Test Prod
Your application always builds, passes its unit tests, deploys and passes system tests.
![Page 10: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/10.jpg)
Continuous Integration/Delivery/Deploy
Continuous Integration Your application always builds and passes its unit tests.
Continuous DeliveryYour application always builds, passes its unit tests, deploys and passes its
system tests making it always ready to deploy to production through an automated process
Continuous DeploymentYour application automatically deploys to production through a fully
automated process
![Page 11: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/11.jpg)
Continuously as in all the time?
Yes!
End of sprint isn't continuously. Weekly isn't continuously. Nightly isn't continuously.
Every code commit is continuously!
![Page 12: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/12.jpg)
What Tests? Unit? Component? System?
Yes!
UnitComponent
SystemIntegrationRollback
LoadFailover
![Page 13: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/13.jpg)
Principles of Continuous Delivery
1. The process MUST be repeatable, reliable and independent of key personnel
![Page 14: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/14.jpg)
2. Automate everything
Principles of Continuous Delivery
![Page 15: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/15.jpg)
3. If something is difficult or painful, do it more often
Principles of Continuous Delivery
![Page 16: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/16.jpg)
4. Keep everything in source control AND release it
Principles of Continuous Delivery
![Page 17: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/17.jpg)
5. Done means released
Principles of Continuous Delivery
![Page 18: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/18.jpg)
6. Build Quality In
Principles of Continuous Delivery
![Page 19: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/19.jpg)
7. Fail Fast
Principles of Continuous Delivery
![Page 20: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/20.jpg)
8. Everybody has responsibility for the release process
Principles of Continuous Delivery
![Page 21: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/21.jpg)
9. Improve continuously
Principles of Continuous Delivery
![Page 22: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/22.jpg)
The pipe - Build once!
Build Release
Build and Unit test then release it!
![Page 23: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/23.jpg)
The pipe - Release everything!
Build Release
Build Release
Build Release
Build Release
Code
DB Scripts
Server Config
Deploy script
Feedback Feedback
![Page 24: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/24.jpg)
The pipe -Bundle released artifacts
Build Release
Build Release
Build Release
Build Release
Assembly
Feedback
![Page 25: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/25.jpg)
The pipe - Use Same Deploy Mechanism
Function TestAssembly Deploy
DB
Server
Function Test
Function Test
Pipe Status
Feedback
Feedback Feedback
![Page 26: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/26.jpg)
The pipe - Use Same Deploy Mechanism
Deploy
DB
Server
Pipe Status
Server
DB
Server Server
DB
Server Server
Deploy
Pre Prod Prod
UAT
FeedbackFeedback Feedback
![Page 27: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/27.jpg)
Continuous Delivery
Build Release Assemble Deploy Deploy PreProd/ProdTest
Summary
We build onceRelease everything
Automate everythingFast feedback
Automate all tests except UAT
![Page 28: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/28.jpg)
Obvious benefits
Continuous Regression Testing gives instant feedback.
Continuously deploying to test servers tests deploy mechanism several hundred times per release.
Always ready to push new release into UAT
We can get customer feedback as soon as value has been added
![Page 29: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/29.jpg)
Automate all tests!
Wait a minute lets look at what we have been doing in the past.
![Page 30: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/30.jpg)
Testing in the pastHow its been done
SYSTEM
GUI
![Page 31: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/31.jpg)
Testing in the pastWhat we have done
100s of manual tests that..... verify our requirements
... ensures bad input doesn't break system... data is stored in the right place
... find bugs
![Page 32: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/32.jpg)
Testing in the pastWe have automated it!
We have tried to automate the 100s of manual tests that..... verify our requirements
... ensures bad input doesn't break system... data is stored in the right place
... find bugs
We do it through instrumenting the GUI and verifying Database, GUI and System resources.
With very bad results....
![Page 33: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/33.jpg)
Robust and Fast Test Automation
Tests that execute 100s of times per day have to be...
... executable without any human interaction... robust and give the same response every single time
... robust and have to survive refactoring... fast and parallelizable as adding 10 tests should not slow
down the feedback
![Page 34: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/34.jpg)
We need to change the way we look at Test
On System Level we do verification of Use Case Acceptance Criterias. Requirement Verification.
On Component Level we do testing of Component Functionality. Testing.
![Page 35: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/35.jpg)
We need to change the way we look at Architecture
Remember “Build quality in”.
We need to improve testability of our applications through...
...isolation of functionality into independent services with life cycles of their own.
...building interfaces for defined test points.
![Page 36: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/36.jpg)
Building testable systems
Internal Service
GUI
Internal Service
Internal Service
Service API
![Page 37: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/37.jpg)
Component Tests
Internal Service
Internal Service
Internal Service
Each Service Tested as Black Box in isolationand with high detail level
![Page 38: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/38.jpg)
GUI is a component
GUI
Service API Mock Impl
GUI Test are much faster and robust if they test the GUI in isolation. High detail level.
![Page 39: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/39.jpg)
Requirement Verification
Internal Service
GUI
Internal Service
Internal Service
Service API
Verifying Use Case acceptance criterias.
Low level of detail
![Page 40: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/40.jpg)
Continuous Delivery
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Assemble Release Deploy Test
Component Pipe
100s of tests run in parallel testing
System Pipe
10s Use Case Requirements Verified
![Page 41: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/41.jpg)
Test Driven Development
Pre Planning
Dev Sys Test Reg Test
Pre Planning
Dev Sys Test Reg Test
Scrummerfall happens because its hard for developers to have something deployable for the testers to test.
System testing on something that has not been regression tested is fundamentally flawed.
Cost of regression test tends to bloat sprint content.
Sprint 2-4 weeks
![Page 42: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/42.jpg)
Test Driven Development
Pre Planning
Dev
Test Automation
Reg Test
Design Requirement Verification and Tests on multiple levels with the feature.
Develop test automation in parallel with feature development.
Cross functional pairing to improve quality of Tests, Test Architecture and Feature Testability
Pre Planning
Reg Test Reg Test
Verification
Verification
![Page 43: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/43.jpg)
Test Driven Development
Who should automate the tests?
Test Design - Done by TestersTest Automation - Done by ????Coding - Done by Developers
![Page 44: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/44.jpg)
Test Driven Development
Tests as Code
Test Automation IS Code and needs to be treated as first class application.
Test Architecture is super important and needs to be worked with in parallel with Application Testability
Architecture.
![Page 45: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/45.jpg)
Test Driven Development
Tests as Code
Internal Service
DSL
Glue Code
SpecificationText or Code based Specification
Implementation of Specification
Domain Specific Language packaging service testability
![Page 46: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/46.jpg)
Test Driven Development
Internal Service
DSL
Glue Code
SpecificationGive a valid user name and passwordwhen login is calledthen user is authorized
boolean authorized = loginService.authorizeUser(“triha”,”cdrockz”);assert(authorized, true);
public boolean authorizeUser(String userName, String password){ //Implementation of REST call to internal service return true; }
![Page 47: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/47.jpg)
Test Driven Development
Tests as Code
Specification & Glue Code IS the Test
DSL is testability
Test Automation is Code
![Page 48: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/48.jpg)
Test Driven Development
Pre Planning
Dev
Test Automation
Reg Test
Specifications of Requirement Verification written during Pre-Planning.Specification of Component Tests written during Pre-Planning.
Implementation of DSL and Glue Code in parallel with feature development.Implementation of Component Tests in parallel with feature development.
Pre Planning
Reg Test Reg Test
Verification
Verification
![Page 49: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/49.jpg)
Test Driven Development
Tomorrow Agile Team Member
Developers with Testing skills Testers with Development skills
WILL BE SUPER HOT AND EARN SUPER NICE $$$$
Will be the only ones who get assignments
![Page 50: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/50.jpg)
Exploratory Testing
Automate Everything!
Yes!
Everything that needs to be done in order to release a system into production.
Yet we still need Exploratory testing.
![Page 51: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/51.jpg)
Exploratory Testing
Has been used to cover up manual regression testing
or worse
A way to get away with non reproducible manual regression testing
![Page 52: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/52.jpg)
Exploratory Testing
Test Automation ensures we build the AGREED application
Exploratory Testing ensures we IMPROVE the application
![Page 53: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/53.jpg)
Exploratory Testing
Exploratory Testing should be...
...done without knowledge of requirements....in time based sessions with specific focus area....outside the release lifecycle of the application.
![Page 54: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/54.jpg)
Benefits
Continuous Regression testing gives instant feedback.
Continuously deploying to test servers tests deploy mechanism several hundred times per release.
Always ready to push new release into UAT
We can get customer feedback as soon as value has been added
Test Driven Development
Everyone is responsible for Quality
Sustainable Quality
![Page 55: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/55.jpg)
Tools Used
We love to talk about what tools are best and spend ages arguing about it.
We need to understand what we must do before we pick the tools.
Still which ever tool we pick it will be insufficient and crap.
![Page 56: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/56.jpg)
Tools Used
Don't use one tool to rule them all.
Cucumber and Fitnesse are ok for system level testing
RESTAssured, JUnit or pure Groovy, Ruby or Python scripts are often better for component level.
For Load testing one should use load testing tools such as JMeter
Continuous Delivery Process should execute tests through interface such as maven or gradle so that it can be Test Tool agnostic.
![Page 57: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/57.jpg)
Areas Not Covered
To do Continuous Delivery well we also have to...
Reproducible Test EnvironmentsProduction like Test Environments
“Non functional requirements” as functional requirementsA/B testing
Power of Metricsand lots more...
![Page 58: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/58.jpg)
Good Reading
Continuous Delivery by Jez Humble & Dave Farley
http://www.amazon.com/dp/0321601912?tag=contindelive-20
![Page 59: Continuous Delivery Testing @HiQ](https://reader033.vdocument.in/reader033/viewer/2022052823/55526da1b4c9052e1f8b5137/html5/thumbnails/59.jpg)
Thats it!
Feedback & Any questions you forgot to ask?
http://continuous-delivery-and-more.blogspot.se