agenda - stpcon · what about this docker thing? •containerization software •uses resource...

3/25/2019

1

1© 2018 Rogue Wave Software, Inc. All Rights Reserved.

Warp Speed Testing at an Enterprise LevelScott MillsAndrei Juc


Agenda

1

2

3/25/2019

2


What are we covering in this session?

1. About us

2. About Rogue Wave

3. The problem

4. The challenge

5. The solution

6. AWS Tips

7. Jenkins plugins

8. What’s next

9. Closing thoughts


What will you take away from this session?

1. Configuring Jenkins to work with AWS

2. Setting up Jenkins jobs to use AWS resources

3. Using Docker with AWS

4. Useful Jenkins Plugins

5. Using on-demand resources

6. AWS Tips

7. Potential pitfalls & limitations

3

4

3/25/2019

3


About us


Ottawa, ON

San Francisco

CA

5

6

3/25/2019

4


Scott Mills

Technical background

• Almost 20 years designing, developing, maintain and supporting

automation systems

• Application performance testing

• Application security testing (OWASP Top 10)

• Automated web testing

• Build & test automation framework development and evolution

utilizing Python, Jenkins, Docker and Amazon Web Services


Scott Mills

While not automating…

7

8

https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project

3/25/2019

5


Andrei Juc

Technical background

• Automation framework developer

• VM to Docker migration

• Amazon Web Services migration

• Troubleshooting enthusiast

• Innovator


Andrei Juc

While not problem solving…

9

10

3/25/2019

6


About Rogue Wave Software


“We help software eat the world faster”

• Our Mission: We simplify complex software development, improve

application quality and security, and shorten cycle times.

• Founded in 1989

• Global company, headquartered in Louisville, CO.

– 400 employees, 13 offices worldwide, 53 patents.

• Our customers:

– Automotive Software Development

– Military and Aerospace

– Medical Devices and Software

– Embedded Software

– Mobile and Communications

– Game Development

11

12

3/25/2019

7


Rogue Wave Customers

We ensure the most critical and complex aspects of the entire software

lifecycle to support strategic business goals while minimizing risk.


• Continuous Static Code Analysis• Find defects early!• Web-based portal for reporting and analyzing results• Ability to be built into your Continuous Integration process• Very popular with mission critical industries• Allows customers to meet industry standards

13

14

http://www.baesystems.com/en/home

https://www.bankofamerica.com/

https://www.libertymutual.com/

https://www.att.com/

https://www.nasa.gov/

http://www.boeing.com/

https://www.oracle.com/index.html

https://www.ca.com/us.html

https://www.morganstanley.com/

http://www.goldmansachs.com/

http://www.hp.com/country/us/en/welcome.html

http://www.dreamworksanimation.com/

http://www.saudiaramco.com/en/home.html

https://www.chevron.com/

https://www-935.ibm.com/services/us/en/it-services/technical-support-services/index.html

https://www.sap.com/index.html

https://www.deere.com/

http://www.schneider-electric.com/ww/en/

https://www.home.barclays/

http://www.polycom.com/

http://www.northropgrumman.com/Pages/default.aspx

https://www.qualcomm.com/

http://www.disney.com/

http://www.lg.com/

https://www.siemens.com/global/en/home.html

https://www.prudential.com/

https://www.llnl.gov/

http://www.lockheedmartin.com/us.html

http://www.sandia.gov/

http://www.fujitsu.com/

http://www.navy.mil/

http://www.raytheon.com/

https://www.bloomberg.com/

http://www.aig.com/

https://www.sprint.com/

http://www.kohls.com/

https://www.firstdata.com/en_us/home.html

https://www.mentor.com/

http://www.netapp.com/us/index.aspx

http://www.cvs.com/

https://www.ge.com/

https://www.gov.sg/

https://www.gm.com/

https://www.audi.com/

https://www.bmw.com/

3/25/2019

8


The Problem


Testing at scale

Requirements

• Reliable, repeatable and timely test results

• Easy to reproduce/verify test results with high level of confidence

• Scalable

• Highly dynamic

• Easy to maintain

15

16

3/25/2019

9


The Challenge


The early beginnings

• Test Lab filled with physical machines

• Maintenance nightmare

• Every machine is slightly different

–Hard to replicate and pin down test failure causes

• “Dirty” environment

• Long execution time (18 hours per machine per run)

Let’s get physical!

17

18

3/25/2019

10


Evolution started…

• VMWare and Jenkins

• Easier to maintain

• Pooled resources

• Template functionality helps with machine continuity

Virtually first steps!


Stunted growth…

• Limited scalability with VMs

– "We're giving her all she's got!"

• "Dirty" machines

• Test all the things!!

The terrible twos

19

20

3/25/2019

11


What about this Docker thing?

• Containerization software

• Uses resource isolation features of Linux in order to allow

isolated OS's to run on a single machine at once.

• Unlike VMs, Docker containers share the host OS kernel.

(Lightweight)

• Capture, store and replicate OS state

• Images and containers and Dockerfiles! Oh, my!

Docker 101



• Images to share with testers and developers

• Clean machine image for every test execution

• Reliable, repeatable results

• More scalability

The good!☺

21

22

3/25/2019

12



• Scalable to a point, limited # of hosts

–CPU thermal throttling

–Hardware degradation

• Environment overhead (Mesos)

The not so good☹



• How can we reap the benefits of the system and make it

scalable?

• And remember, TEST ALL THE THINGS!!!

But this Docker shows a LOT of promise!

23

24

3/25/2019

13


The Solution Future!


Introducing AWS

• Essentially unlimited resources

• Very scalable

• No more hardware woes

• On-demand resources

• Tools

– So many tools…

Resources as far as the eye can see…

25

26

3/25/2019

14


Amazon Elastic Compute

• Amazon Machine Images

• Preconfigured services, service optimized images or a blank canvas

• Elastic Container Service optimized AMIs

Turning dreams into reality!


Amazon Elastic Container Registry (ECR)

• Docker container registry for storage and deployment

• Used everywhere in our build and test automation

• Images updated nightly with latest build

• Allows developers and testers to use the same images

used in building and testing

Essential part of our infrastructure!

27

28

3/25/2019

15





29

30

3/25/2019

16



echo "Logging in to docker registry..."

export AWS_ACCESS_KEY_ID=<key>

export AWS_SECRET_ACCESS_KEY=<key>

export AWS_DEFAULT_REGION=<region>

/usr/local/bin/aws ecr get-login --no-include-email --region <region>

environment="-e SUITE_NAME=$SUITE_NAME -e RELEASE=$RELEASE -e USE_COMPILER=$CC \

–e release=$RELEASE -v /sharedbuild:/sharedbuild -p 8080:8080 \

--entrypoint /sharedbuild/utils/run_scripts.sh "

docker pull <ecr-url>.amazonaws.com/$image:$build_type #For example: ubuntu18:nightly

docker run -i $environment <ecr-url>.amazonaws.com/$image:$build_type

Accessing ECR images from our Jenkins test jobs


AWS Tips

31

32

3/25/2019

17


Mind the firehose!

• Start small and work your way up.

• Sometimes it's better to stay lightweight

• AWS offers a lot of tools, all great under the right

application, but some more cumbersome than needed

AWS Offers many services


Keep a good balance of AMIs

• Avoid homebrew AMIs as much as possible

• Use out-of-the-box solutions for parts of the framework

you don't want to maintain

• Instance snapshots are an acceptable substitute for

docker

Don't spend more time than necessary

33

34

3/25/2019

18


Instance type selection is important

• Use appropriate application types

• Conduct experiments to find the best size for you

• Process intensive applications benefit from higher core

count

Cheaper is not always cheaper


Cattle not pets

• Use portable initialize scripts as much as possible to

keep agile

• Always think in terms of scale

What's good for one is good for all

35

36

3/25/2019

19


Shepherd your flock

• What goes up must come down

• Minimize idle uptime but maximize usage of AWS hourly

cycle

• Mistakes can be costly

Management of your resources


Be prepared to maintain two systems

• Some legacy hardware may remain

• AWS only offers Linux and Windows Server AMIs

• AWS OVA to AMI Service

• AWS File Syncs

Some tasks are just not possible...

37

38

3/25/2019

20


Jenkins Plugins


Jenkins Matrix Project Plugin

• The Configuration Matrix allows users create a multiple-

axis graph of the configurations to run

• In other words, "Run each of these test suites on each of

these platforms with each of these modes"

• Filters can be set to prevent some configs from running

Makes our testing possible

39

40

3/25/2019

21



All tests passed

Some tests failed

Failure within the configuration

Not run



41

42

3/25/2019

22


Matrix Combinations Parameter Plugin


Matrix Combinations Parameter Plugin

43

44

3/25/2019

23


EC2 Plugin


EC2 Plugin

45

46

3/25/2019

24


What’s Next


AWS is much more

• AWS offers enough services to allow for full system

migration

• Can be used for more than just development and R&D

• Marketing, customer services, support and interactions

This is just the tip of the iceberg

47

48

3/25/2019

25


More OS Support?

• OSX Support

• AIX and Solaris

• “Delayed” Windows releases

The selection is great, but could be better


Closing Thoughts

49

50

3/25/2019

26


Closing Thoughts

• Invest in your automation infrastructure

• Look for opportunities to evolve

• "There's a plugin for that"

• AWS does cost $$$

Always be improving


Questions?

51

52

3/25/2019

27


Contact Us!

• Scott Mills

– [email protected]

– Twitter: @ScoringHen

– LinkedIn: Look for Ottawa Scott, not the DJ in the UK.

• Andrei Juc

– [email protected]

– Twitter: @Andrei_Juc

– LinkedIn


53

54

mailto:[email protected]

mailto:[email protected]

agenda - stpcon · what about this docker thing? •containerization software •uses resource...

Documents