1©2015SOASTAInc.-Allrightsreserved.
InThisWebinar
TODAY’SPRESENTERS
TomChavez:SOASTA,Sr.ProductMarkeEngManagerAndyPemberton:CloudBees,SeniorDirector,SoluEonsArchitecture&ConsulEngTWOCOMPANIES,ASINGLESOLUTION
SOASTA:TheleaderinCloudandMobileTesEng
CloudBees:TheEnterpriseJenkinsCompany
Together,weofferacompletecloud-basedservice:
ConEnuousLoadTesEng&ConEnuousDeliverywithJenkins
QuesEons?AskinQuesEonwindow
3©2015SOASTAInc.-Allrightsreserved.
PollQuesEon
WhatbestdescribesyourstatuswithConEnuousIntegraEon/ConEnuousDelivery(CI/CD)?(singleanswer) a) DeterminingifCI/CDisagoodopEon
a) ImplemenEngCI/CD
a) PracEcingCI
b) PracEcingCD
c) WhatisCI/CD?
4©2015SOASTAInc.-Allrightsreserved.
Agenda• LoadtesEng:why?• Whydon’tweloadtestearlier?• ConEnuousLoadandPerformanceTesEng• CloudBeesandtheEnterpriseJenkinsPlaYorm• ConnecEngloadteststoyourCDPipeline• Q&AQuesEons:SubmitinthequesEonboxduringevent
5©2015SOASTAInc.-Allrightsreserved.
WhyShouldWeLoadTest?
o Ensurethatsitewillperformwellatload• TesthardwareconfiguraEons
• TestapplicaEon,database,components
• Testthirdpartypartners
o Tobereadyforpeakseason• BlackFriday,CyberMonday,Christmas
• Taxfiling,Insurancefiling,SuperBowlcommercial
6©2015SOASTAInc.-Allrightsreserved.
WhyShouldWeLoadTest?
7©2015SOASTA,Inc.-Allrightsreserved.
WhyDon’tWeLoadTestEarlier?
Copyright © 2014 SOASTA, Inc. - All rights reserved.
8©2015SOASTAInc.-Allrightsreserved.
WhyDON’TWeLoadTestEarlier?
“Because it’s hard.” o Tools require coding skills we don’t have (C, VB?) o We don’t have a scalable test environment o We don’t have hardware for load generation o It’s a different tool o There are too many other tests to do
9©2015SOASTAInc.-Allrightsreserved.
WhyDON’TWeLoadTestEarlier?
“Because it’s hard.” o Tools require coding skills we don’t have (C, VB?) o We don’t have a scalable test environment o We don’t have hardware for load generation o It’s a different tool o There are too many other tests to do
“Because it takes too long.” o Scripting takes forever o Results are slow o Fast setup of environments is impossible o We’re dependent on IT or other teams o We’re too agile
10©2015SOASTAInc.-Allrightsreserved.
WhyDON’TWeLoadTestEarlier?
“Because it’s hard.” o Tools require coding skills we don’t have (C, VB?) o We don’t have a scalable test environment o We don’t have hardware for load generation o It’s a different tool o There are too many other tests to do
“Because it takes too long.” o Scripting takes forever o Results are slow o Fast setup of environments is impossible o We’re dependent on IT or other teams o We’re too agile
“Because no one ever told us to.” o There are no “performance coverage” requirements o There isn’t a performance baseline for reference o There isn’t a defined performance target o No one is managing to a performance metric
11©2015SOASTAInc.-Allrightsreserved.
WhyDON’TWeLoadTestEarlier?
“Because it’s hard.” o Tools require coding skills we don’t have (C, VB?) o We don’t have a scalable test environment o We don’t have hardware for load generation o It’s a different tool o There are too many other tests to do
“Because it takes too long.” o Scripting takes forever o Results are slow o Fast setup of environments is impossible o We’re dependent on IT or other teams o We’re too agile
“Because no one ever told us to.” o There are no “performance coverage” requirements o There isn’t a performance baseline for reference o There isn’t a defined performance target o No one is managing to a performance metric
Until Now.
12©2015SOASTAInc.-Allrightsreserved.
PollQuesEon
WhatdoyouseeholdingbackamoreiteraEveloadtesEngapproach?(checkallthatapply)
a) Skillsgaps
b) Lackoftestenvironment(s)
c) Slowtestdevelopment
d) TesttoolslimitaEons
e) 3rdPartyDependencies
f) UndefinedPerformanceRequirements
g) LackofManagementPush
h) Other
13©2015SOASTAInc.-Allrightsreserved.
LoadTesEngHasBeenanEliteEvent
STheIssue:
• HighexpectaEonsfromtestsrunwithdiminishingEmewindows
• Code-levelperformancebugsfoundlatesetenEreprojectback
• Separateteamandscheduleconflictwithdeliverygoals
• MajorissuesescapetoproducEon
o pecialtyskillsandtasksarealegacybarrier
Iteration 1
Automated unit tests
Iteration 2
Automated unit & functional tests
Iteration 3
Regression test suite
Iteration n
Regression test suite
Load & Performance Testing
Deploy
Issues?
14©2015SOASTAInc.-Allrightsreserved.
ConEnuousLoadTesEngValidateperformancepiecebypiece
Iteration 1 Automated unit
tests
Deploy
Production perf tests
Iteration 2 Automated unit & functional tests
Iteration 3 Regression test
suite Incl:
Iteration n Regression test
suite Incl:
End to End Production Scale
Test Small perf test Small perf tests
Perf smoke tests Perf smoke tests
TheSoluEon:• RunincrementalperformancetestsautomaEcallywitheachiteraEon
• Establishatrendlineforperformanceearly• Developersfindcodeandarchitectureissuesearly
• Performanceteamsfocusonend-to-endinfrastructureanduserrelatedtesEng
• MinimizeproducEonescapesandimplementproducEontesEng
Development/Test team Performance/Ops team DevOps
15©2015SOASTAInc.-Allrightsreserved.
ConEnuousPerformance(forWeb&Mobile)ThePathtoaPerformanceCulture I JUST MADE A
CODE CHANGE
CloudTest runs load tests via Jenkins Daily (or more frequent) updates keep
teams “performance focused”
Identify bottlenecks & inform teams,
just like unit and functional
tests in Jenkins
Jenkins launches suitable cloud-based test environment
Manage with a new metric!
16©2015SOASTAInc.-Allrightsreserved.
PollQuesEon
WhattypesofLoadTesEngareyoudoingtoday?(SingleAnswer)
a) NotdoinganyloadtesEng
b) Non-WebProtocols(notmobile)
c) Primarilywebprotocols
d) Primarilymobilewebtraffic
e) Bothwebandmobile
f) Allweb,mobile&non-webprotocols
17©2015SOASTA,Inc.Allrightsreserved.
EnablingConEnuousPerformanceTesEngwithJenkinsPipeline
Andy Pemberton Twitter: @apemberton
Email: [email protected]
18©2015SOASTAInc.-Allrightsreserved.
HaveyoumetJenkins?
#1 CI/CD Automation Engine
• Open Source
• Highly Extensible
About 8 years old
Easy to install/use
1000+ plugins
Widely adopted
• 100K+ installations
19©2015SOASTAInc.-Allrightsreserved.
Jenkins is the #1 Automation Platform Used for CI
Continuous Integration
Developer
Source Code
Control System
Compile/Build Code
Scan Tests Code Commit
Results Report
Build Commit Test
20©2015SOASTAInc.-Allrightsreserved.
Jenkins is Also Your Key to Continuous Delivery
Developer
Source Code
Control System
Compile/Build Code
Scan Tests Code Commit
Results Report
Build Commit Test Stage Deploy
Deploy Prod
Deploy Stage
Complex Delivery Pipelines
Delivery of App and Config
Continuous Delivery
Prod Dev
21©2015SOASTAInc.-Allrightsreserved.
Over 1000 Jenkins Plugins for Tool Integration
Integrated Development Environments (IDEs), editors/compilers/debuggers, code review and collaboration tools.
Source code control systems and related asset management tools.
Automated API-based testing frameworks.
Release management and release automation software
Automated API-based testing frameworks, static code analysis tools.
Load Testing Tools and Security Scanning technology
Automated and manual UAT tools
Release management, release automation, and change control software.
Environment and Configuration Management Software Develop
Commit
Build & Integrate
Unit Tests
Release (to Test)
Automated Regression Tests
Release (to Prod)
User Acceptance Tests
Load & Security Tests
22©2015SOASTAInc.-Allrightsreserved.
Jenkins is the Hub of the CD Ecosystem
On-premise or in the cloud
Plug-ins for all your tools
Run Deploy Stage Test Build Commit
23©2015SOASTAInc.-Allrightsreserved.
AreYouUsingToday?
23
Chances are someone in your
organization is.
24©2015SOASTAInc.-Allrightsreserved.24
25©2015SOASTAInc.-Allrightsreserved.25
How Do You Manage Jenkins and Testing at Enterprise Scale?
26©2015SOASTAInc.-Allrightsreserved.
About CloudBees
Jenkins founder, Kohsuke Kawaguchi, CloudBees CTO We optimize Jenkins for the enterprise
• Jenkins Operations Center
• Jenkins OSS Support
• Enterprise-class plugins
• Commit code and support the Jenkins Community
• Jenkins training and consulting
• Jenkins User Conferences (JUC) and newsletters
…we are the Jenkins Enterprise experts!
27©2015SOASTAInc.-Allrightsreserved.
CloudBees Jenkins Enterprise
27
Professional support
High availability
Large installations
Security
Optimized utilization
Continuous Delivery
28©2015SOASTAInc.-Allrightsreserved.
Additional Jenkins Services from CloudBees
28
Jenkins Certified Training • Delivered via CloudBees partners
Jenkins Consulting • Delivered via CloudBees Professional Services Team
Jenkins Newsletter • Community news and expert tips
Jenkins User Conferences • Learn from other community members
29©2015SOASTAInc.-Allrightsreserved.
HowCanCloudBeesJenkinsEnterpriseHelpwithYourCDStrategy?
29
PipelineStage View Provides deep insight into the execution of your workflow stages so you can detect problems and performance issues Manage complex “level-of-detail” testing pipelines
Jenkins Pipeline Created by CloudBees to provide Jenkins with a way to define flexible delivery pipelines that allow for a rich set of new delivery scenarios Manage and share “testing-as-code”
Dynamic Environment Creation Spin up new test/deployment environments automatically as part of your delivery pipeline
Deployment Tracking and Verfiication Track the history of your deployments – including who kicked off a deployment and what code changes were included in it Create pipelines that leverage verification tests to determine the quality of a new deployment
Jenkins Analytics Monitor and analyze performance across all your Jenkins instances Monitor, measure and manage adoption of testing, CI and CD best practices Restart builds, tests, and deployments where they
break Leverage pipeline checkpoints to restart delivery pipelines without a full (expensive) rebuild
...central source of orchestration to bridge dev, test, and ops alike
And more…
30©2015SOASTAInc.-Allrightsreserved.
#Jenkins2GreatlyEnhancedUI/UXforeasierstartupandbeOersecurity
31©2015SOASTAInc.-Allrightsreserved.
#Jenkins2JenkinsPipeline-as-Codefrontandcenter
32©2015SOASTAInc.-Allrightsreserved.
Jenkins Pipeline and Pipeline Stage View Enable Full Continuous Delivery Pipelines
Continuous Delivery
Prod Dev
Developer
Source Code
Control System
Code Commit
33©2015SOASTAInc.-Allrightsreserved.33
CloudBees Jenkins Pipeline Stage View
Restartable Checkpoints
Artifact Downloads
34©2015SOASTAInc.-Allrightsreserved.
Monitoring, Visualization and Analytics
Jenkins Pipeline Stage View Visibility into Pipeline status
Jenkins Analytics Cross-Team Build Analytics
… among many other features, tailored for enterprise organizations using Jenkins @ scale
35©2015SOASTAInc.-Allrightsreserved.
36©2015SOASTAInc.-Allrightsreserved.
ConEnuousLoadTesEng
o SOASTACloudTestComponents/Architecture
o Demo:
• CloudTestUIOverview
• JenkinsSetup• DownloadingandinstallingCloudTestplug-in
• CreaEngaPerformanceJobinJenkins
• CreaEngaPipelineinJenkins
• ReviewingResults
Whatarewegoingtoshow
37©2015SOASTA,Inc.Allrightsreserved.
Architecture:
Jenkins Jill
Sri Raj
DevOps Team 1
CloudTest Manager
Always up at: Cloudtestmanager.soasta.com
CloudTest server
any IP Address/ Domainname Up ‘as needed’ at
“US Grid1”
2 Azure California
2 AWS Virginia
2x Google Iowa
“WW Grid1”
2 Azure California
2 AWS Virginia
2x Google Iowa
CloudTest “Main” server
any IP Address/ Domainname Up ‘as needed’ at
dev.acme.com
CPU,DISK, Memory,I/O Etc.
38©2015SOASTA,Inc.Allrightsreserved.
Jenkins Enterprise on CloudBees
Jill
Sri Raj
DevOps Team 1
CloudTest Manager
Always up at: Cloudtestmanager.soasta.com
dev.acme.com
“US Grid1”
2 Azure California
2 AWS Virginia
2x Google Iowa
CloudTest “Main” server
any IP Address/ Domainname Up ‘as needed’ at
DemoPreview:
39©2015SOASTA,Inc.Allrightsreserved.
Demo
QuesEons?AskinQuesEonwindow
41©2015SOASTAInc.-Allrightsreserved.
Resources
ContactSOASTA:www.soasta.com/cloudtest/[email protected]:
twiOer.com/SoastaIncfacebook.com/cloudtest
KnowledgeCenter• WhitePapers• WebinarRecordings• CaseStudiesCloudLinkCommunity• Support• Tutorials• Video
ContactCloudBeeswww.cloudbees.cominfo@cloudbees.com1.781.404.5100Followus:
twiOer.com/cloudbeesfacebook.com/CloudBees
www.SOASTA.com
[email protected]/get-started
Documenta]on• CloudBees Jenkins Platform • Workflow in Jenkins WhitePapers:• Making the Leap from Continuous Integration to
Continuous Delivery • The Business Value of Continuous Delivery
HowtoIntroduceCloudBasedLoadTesEngtoyourJenkinsConEnuousDeliveryPipelineApril20,2016TomChavez,SOASTA@TomChavezAndyPemberton,CloudBees@apemberton