wa2451 devops fundamentals - … devops fundamentals web age solutions inc. ... • the application...

WA2451 DevOpsFundamentals

Web Age Solutions Inc.USA: 1-877-517-6540Canada: 1-866-206-4644Web: http://www.webagesolutions.com

The following terms are trademarks of other companies:

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

IBM, WebSphere, DB2 and Tivoli are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

Other company, product, and service names may be trademarks or service marks of others.

For customizations of this book or other sales inquiries, please contact us at:

USA: 1-877-517-6540, email: [email protected]: 1-866-206-4644 toll free, email: [email protected]

Copyright © 2016 Web Age Solutions Inc.

This publication is protected by the copyright laws of Canada, United States and any other country where this book is sold. Unauthorized use of this material, including but not limited to, reproduction of the whole or part of the content, re-sale or transmission through fax, photocopy or e-mail is prohibited. To obtain authorization for any such activities, please write to:

Web Age Solutions Inc.439 University AveSuite 820TorontoOntario, M5G 1Y8

Table of ContentsChapter 1 - DevOps Introduction.........................................................................................7

1.1 Dev and Ops Views...................................................................................................71.2 Leading By Example ................................................................................................71.3 What is DevOps?.......................................................................................................71.4 More DevOps Definitions.........................................................................................81.5 DevOps and Software Delivery Life Cycle..............................................................81.6 Main DevOps' Objectives.........................................................................................81.7 The Term "DevOps" is Evolving!.............................................................................91.8 Infrastructure as Code...............................................................................................91.9 Agile IT in the Cloud ............................................................................................101.10 DevOps on the Cloud............................................................................................101.11 Prerequisites for DevOps Success ........................................................................111.12 Alignment with the Business Needs.....................................................................111.13 Collaborative Development .................................................................................121.14 Continuous Testing and Integration......................................................................121.15 Continuous Release and Deployment ..................................................................121.16 Continuous Application Monitoring.....................................................................131.17 Summary...............................................................................................................13

Chapter 2 - Standing Up DevOps .....................................................................................152.1 Standing Up DevOps...............................................................................................152.2 Things to Look For and Avoid................................................................................152.3 IT Assets Ownership...............................................................................................162.4 Viewing Applications As Products, not Projects....................................................162.5 DevOps in the Enterprise........................................................................................172.6 IT Governance ........................................................................................................172.7 Governance and Risk Mitigation............................................................................182.8 DevOps Adoption Steps..........................................................................................182.9 DevOps Adoption Steps..........................................................................................192.10 DevOps Adoption Steps........................................................................................192.11 DevOps Adoption Steps........................................................................................192.12 Select DevOps Techniques and Practices.............................................................202.13 Select DevOps Techniques and Practices.............................................................202.14 Select DevOps Techniques and Practices.............................................................212.15 Select DevOps Techniques and Practices.............................................................212.16 Select DevOps Techniques and Practices.............................................................222.17 Service Quality Metrics.........................................................................................232.18 Summary...............................................................................................................24

Chapter 3 - DevOps Tools.................................................................................................253.1 The Choice of Cloud Platform ...............................................................................253.2 IaaS for DevOps......................................................................................................253.3 PaaS for DevOps.....................................................................................................263.4 Containerization Tools............................................................................................263.5 System Configuration Automation and Management.............................................273.6 System Configuration Automation and Management.............................................27

3.7 Continuous Integration (CI) Systems......................................................................283.8 Build and Dependency Management Systems........................................................293.9 Build and Dependency Management Systems........................................................293.10 Select DevOps Tools.............................................................................................303.11 Collaborative Lifecycle Management Solutions from IBM..................................303.12 The Collaborative Lifecycle Management Diagram.............................................313.13 The IBM Collaborative Lifecycle Management Platform ..................................313.14 Rational Team Concert (RTC)..............................................................................313.15 Rational Quality Manager (RQM)........................................................................323.16 Rational DOORS Next Generation (DNG)...........................................................333.17 Summary...............................................................................................................33

Chapter 4 - Introduction to Puppet.....................................................................................354.1 What is Puppet........................................................................................................354.2 Puppet's Domain Specific Language.......................................................................354.3 "Infrastructure-as-code" in Puppet .........................................................................364.4 Example of the Puppet DSL....................................................................................364.5 Main Puppet Artifacts.............................................................................................374.6 Puppet Design ........................................................................................................384.7 Puppet Workflow Orchestration.............................................................................384.8 Facter.......................................................................................................................384.9 Facter Fact Discovery ............................................................................................394.10 Facter Example......................................................................................................404.11 Extending Facter...................................................................................................404.12 Geppetto................................................................................................................414.13 Puppet Lab Services..............................................................................................414.14 Puppet Enterprise Licensing.................................................................................414.15 Puppet Enterprise Support ....................................................................................424.16 Puppet Enterprise Feature Set (1/2).....................................................................424.17 Puppet Enterprise Feature Set (2/2).....................................................................424.18 Summary...............................................................................................................43

Chapter 5 - Docker Introduction........................................................................................455.1 What is Docker........................................................................................................455.2 Where Can I Ran Docker? .....................................................................................455.3 Docker and Containerization on Linux .................................................................465.4 Linux Kernel Features: cgroups and namespaces .................................................465.5 The Docker-Linux Kernel Interfaces......................................................................475.6 Docker Containers vs Traditional Virtualization....................................................475.7 Docker Containers vs Traditional Virtualization....................................................475.8 Docker as Platform-as-a-Service.............................................................................485.9 Docker Integration..................................................................................................485.10 Docker Services....................................................................................................495.11 Docker Application Container Public Repository.................................................495.12 Competing Systems...............................................................................................505.13 Docker Command-line..........................................................................................505.14 Starting, Inspecting, and Stopping Docker Containers.........................................505.15 Summary...............................................................................................................51

Chapter 6 - Introduction to Continuous Integration and Jenkins-CI..................................536.1 Agile Development.................................................................................................536.2 Agile Development (cont'd)....................................................................................536.3 Agile Development (cont'd)....................................................................................546.4 What is Continuous Integration..............................................................................546.5 What is Continuous Integration (cont'd).................................................................546.6 What is Continuous Integration (cont'd).................................................................556.7 What is Continous Integration (cont'd)...................................................................556.8 Typical Setup for Continuous Integration...............................................................566.9 Typical Setup for Continuous Integration...............................................................566.10 Jenkins Continuous Integration.............................................................................576.11 Jenkins Features....................................................................................................576.12 Running Jenkins....................................................................................................576.13 Summary...............................................................................................................58

Chapter 1 - DevOps Introduction

ObjectivesKey objectives of this chapter

DevOps introduction

Business value of DevOps

Standing up DevOps capability

1.1 Dev and Ops Views

The Dev View The Ops View • We have aggressive deadlines --

Business is all over us• Ops are much too sluggish supporting

us (provisioning integration environment, etc.)

• They lost the application zip file we emailed them yesterday night -- it was eventually found in their "junk mail" folder

• Overall, we don't have trust and confidence in Operations (Ops) -- they are more like Oops, then Ops

• Dev is all over us• The application set-up guide sent by Dev

was not complete -- they missed some critical steps, which resulted in our wasted time

• With so many new applications being released in the environment, we can no longer guarantee uninterrupted services

• Overall, we don't trust Dev

1.2 Leading By Example ...

1.3 What is DevOps?

DevOps is short for Development and Operations

It is an approach to delivering software solutions in a continuous manner


based on lean (minimizing waste of resources, reducing number of defects, etc.) and agile practices

DevOps help manage complexities of Enterprise applications by creating a collaborative environment with participants coming not only from Development and Operations, but also from Business, QA, and other stakeholder groups

◊ In other words, DevOps is not only about Development and Operations!

The DevOps practice has been popularized by organizations adopting the Cloud-as-a-Service computing model

1.4 More DevOps Definitions

You can view DevOps as

◊ a culture, or

◊ a cross-team software delivery discipline (paradigm)

that tries to reconcile competing perspectives (e.g. those of Dev vs Ops) and promote collaboration by stepping over the silos of isolated, group-centric interests

1.5 DevOps and Software Delivery Life Cycle

To efficiently increase the velocity of application delivery, DevOps activities span the whole software delivery life cycle (not only its deployment!):

◊ Development

◊ Testing

◊ Deployment

◊ Operation

1.6 Main DevOps' Objectives

Continuous software delivery planning and control

8


Software delivery processes optimization

Software delivery process consistency and predictability

Minimization of the number of software defects and unnecessary re-work

Software delivery cycle time reduction

Notes:

Planning is viewed by some developers as an unnecessary overhead hampering their "real work"; those developers rely on tribal knowledge of some opaque team processes they establish themselves. While it may work in a short-term perspective, the lack of transparency and overall planning across various stakeholder groups would normally result in problems with project delivery sustainability at faster rates.

1.7 The Term "DevOps" is Evolving!

Originally, DevOps was used to refer to the practice adopted by Operations to borrow some of the tools and processes used in software development

◊ For example:

Admin scripts were placed in a version control system

Now DevOps is used in the context of the shared responsibility between Development and Operations for delivering high quality software products

◊ For that, where appropriate, the reporting lines are merged and simplified

◊ Development goals (introduce code, configuration, etc. changes to the system) are aligned with those of Operations (maintain the target system stability)

DevOps nowadays is becoming more embracing being not only about Tools, but also about People and Processes

1.8 Infrastructure as Code

"Infrastructure as Code" is a practice of provisioning infrastructure by executing system management and configuration scripts

9


Under DevOps, Dev is granted system administration privileges to run the infrastructure set-up scripts to automatically provision the necessary development and testing environments

◊ Provisioning of other environments (staging, production) may still be the exclusive prerogative of personnel in the DevOps' Operations role

Infrastructure as Code is effectively supported by such tools as Chef, Puppet, and IBM UrbanCode Deploy

1.9 Agile IT in the Cloud

Cloud facilitates agile IT practices that allow businesses to quickly respond to market forces

IT responds to rapidly changing business requirements by creating a cloud-based execution environment that allows effective and optimized reuse of existing services through

◊ re-configuration,

◊ re-composition, and

◊ introducing new services that also lend themselves to composition and reuse

1.10 DevOps on the Cloud

Some of the more important activities performed by Operations are related to provisioning computing resources

DevOps agility can been dramatically enhanced with adopting Cloud Computing

◊ Developers can easily self-provision the needed resources (virtual servers, extra disk storage, back-end systems, etc.)

◊ Many cloud platforms allow for application code snapshotting which can be used for environment replication / cloning (Dev → QA → UAT → Prod)

This feature also facilitates defect reproduction

10


◊ In the cloud, it is much easier to quickly set up and tear down "Production-like" systems at a minimal cost

1.11 Prerequisites for DevOps Success

A good relationship between Dev and Ops is necessary but not sufficient for the overall success of DevOps operations

DevOps success depends on the proper execution of all the technical aspects of the Software Development Life Cycle (SDLC) phases and steps established in the organization

◊ Sometimes, the existing SDLC documentation needs to be adjusted to make it aligned with the agile practices used by DevOps

The following elements and capabilities need to be put in place for DevOps to meet its objectives:

◊ Alignment with the business needs

◊ Collaborative development

◊ Continuous testing and integration

◊ Continuous release and deployment

◊ Continuous application monitoring

1.12 Alignment with the Business Needs

In essences, DevOps is a business-driven software solutions delivery process

The DevOps practice is instrumental in reliably materializing a business idea in a software product, ultimately delivering value to customers

DevOps processes improve product time-to-market metric (faster time to value) and enable organizations to react to new market demands more quickly

With DevOps, customer feedback on product is captured and quickly incorporated in the next iteration of the product delivered in a continues manner

11


◊ The ultimate result of a fast accommodation of the customer feedback is an enhanced customer experience, customer loyalty, and larger market share

1.13 Collaborative Development

High quality software development is predicated on the collaborating development practices, including:

◊ Development teams work is done in accordance with established code standards, styles, and living centralized developer documentation (wiki pages, etc.)

Development teams may be geographically dispersed or formed at the last moment, making the above practice indispensable

◊ The stewards of the target system's architecture and design are cooperating with development to quickly accommodate any design flaws / gaps identified during development

1.14 Continuous Testing and Integration

Development of discrete software components must go hand-in-hand with the development and application of appropriate unit tests as prescribed by the Test-Driven Development (TDD) process

◊ TDD is an agile software development practice

Integration of software components must start as early as possible (even though the work on components may not yet been fully completed) and conducted frequently (sometimes, several times a day)

◊ This process is known as the Continuous Integration (CI) agile practice which helps with catching integration problems early

1.15 Continuous Release and Deployment

Deployment has always been one of the primary activities of Operations

◊ With the advent of the Cloud Computing, Development can take over some workload of application deployment and perform self-provisioning

12


of cloud computing resources they require

To support reliable continuous releases, the deployment process must be automated; any failed deployments must be rolled back in its entirety in an atomic operation without affecting applications currently running in production

Deployment parameters, such as average deployment time, size of the deployment bundle, etc., must be recorded and kept for reference

1.16 Continuous Application Monitoring

Application run-time behavior monitoring should begin in production-like environments where the application would have setup, configuration, and other parameters close to those used in production

Things to look for:

◊ Run-time application behavior (CPU, RAM, I/O, average duration of garbage collection pauses, if applicable, and other metrics)

◊ Response time per application interface

◊ Excessive or insufficient logging

◊ Logging of sensitive information

◊ etc.

1.17 Summary

DevOps can be viewed as a cross-team software delivery discipline (paradigm)

The DevOps practice has been popularized by organizations adopting the Cloud-as-a-Service computing model

The following elements and capabilities need to be put in place for DevOps to meet its objectives:

◊ Alignment with the business needs

◊ Collaborative development

13


◊ Continuous testing and integration

◊ Continuous release and deployment

◊ Continuous application monitoring

14

Chapter 2 - Standing Up DevOps


DevOps Adoptions in the Enterprise

IT Governance

Select Techniques and Practices

Service Quality Metrics

2.1 Standing Up DevOps

There is no cast-in-stone rules on how to set up the DevOps capability

Every organization finds its own organizational form for DevOps

◊ Some companies create a joint DevOps group with a single reporting line

◊ Others establish a small contact group with representatives from all stakeholder groups

◊ Still others completely delegate Ops functions to Dev (mostly the case with Cloud-based shops)

2.2 Things to Look For and Avoid

Wrong people and inefficient processes may turn DevOps into a liability rather than an asset

Despite the "communal" nature of the DevOps operations, there should be a clear separation of operational scopes demarcated along discrete roles within DevOps; also, strict operational rules must be established and enforced, including:

◊ Production passwords must not be shared across DevOps

Failure to do so may result in developers sneaking in production environment with Ops' credentials to perform unauthorized actions

◊ All operational activities in production environment must be traceable with tamper-proof logging of such information as: user id, source IP


address, timestamp, and performed activity

◊ All admin scripts along with their supporting documentation must be checked in a version control system

◊ System and application production passwords must be kept in a safe way accessible only to the authorized personnel

2.3 IT Assets Ownership

What happens when something isn't “owned” by someone?

◊ “Tragedy of the commons” – Garrett Hardin, Science 1968

◊ “That which is common to the greatest number has the least care bestowed upon it.“ – Aristotle, Politics 1261b34

IT assets ownership is critical

◊ Accountability to the enterprise

◊ Evolution alongside changing business needs

◊ Motivation to maintain and support

◊ Quality service and customer satisfaction

Notes:

The Basic Idea of Garrett Hardin's article, “Tragedy of the commons”:

"If a resource is held in common for use by all, then ultimately that resource will be destroyed. "

"Held in common" means the resource is owned by no one, or owned by a group, all of whom have access to the resource.

2.4 Viewing Applications As Products, not Projects

In some cases, taking ownership of an IT asset can be promoted by adopting the software product model

The traditional model is project-centric where delivery of the project in production means the end of life of the project for the core development team who hands-off (and forgets) the deployed application to the maintenance team to support it

16


In some organizations, ownership of the application does not end there

◊ E.g. AWS uses this rule: "you build it, you run it", where a development team retains ownership of and responsibility for the application released in production. In this context, the application is treated as a product

Generally, the product development model (as opposed to the project model) leads to the feeling of "ownership" of the application (e.g. your team carries the application production pager 24/7 on a rotating basis), which contributes to the improved quality of the software product and yields other intangible benefits

2.5 DevOps in the Enterprise

While originally DevOps was popularized by Web (Cloud) -based companies, such as Flickr and Netflix, in one form or another, large enterprises have long been using DevOps practices

For deeper penetration of DevOps in the Enterprise space and establishing it as a true enterprise capability, it needs to be placed under control within the existing enterprise governance processes

Embracing DevOps practices help organizations to more efficiently and effectively manage aggressive software delivery schedules by minimizing the number of software defects and deployment failures

Also, in addition to the known benefits (higher delivery velocity and more predictable application development and production deployment cycles, etc.), DevOps help promote collaborative environment within organization

In some cases, a certain (sometimes very painful) shift in mentality and organizational culture is required to fully exercise the benefits of DevOps practice

2.6 IT Governance

IT governance is about managing IT resources / applications / systems and ensuring that IT decisions are aligned with strategic and operational business requirements

DevOps activities are no exception

17


IT governance processes are created with a view to

◊ Mitigate IT risks

◊ Make the outcome of IT activities predictable

◊ Measure IT performance

◊ Promote standards and best practices

◊ Establish proven policies and procedures to ensure projects' success

2.7 Governance and Risk Mitigation

In essence, governance is about mitigating risk

This risk mitigation is manifested in several ways

◊ Defined procedures, templates, and checklists

◊ Design-, Change-, Deployment- and Run-time management

◊ Enterprise-wide resources (technical and human)

◊ Documentation and promotion of reference architectures, design patterns, standards, and best practices

2.8 DevOps Adoption Steps

1. Identify your business drivers✔ Product time-to-market metric (faster time to value), etc.

2. Get educated✔ Learn / educate / evangelize about various DevOps techniques:

Continuous Integration, cross-team interests, transparency, etc.

3. Articulate DevOps' value proposition✔ Minimization of resource wasting (e.g. cutting down on avoidable

overtime), reduction of the number of defects, creation of the reliable software delivery pipe-line, etc.

18



4. Define one or more scenarios of software delivery with DevOps techniques✔ If possible, set up and run a [small] proof of concept (PoC) project

✔ Show how DevOps can address some of the business drivers

5. Produce a road map✔ Provide time-specific and quantifiable plans for installing DevOps-

related tools/systems, staff training, etc.

6. Gain stakeholder buy-in✔ Get all parties concerned on board with regard the DevOps initiative

✔ Keep them informed on the progress of the DevOps initiative

✔ Gently educate them, if needed


7. Establish governance for risk mitigation✔ This is particularly important for Enterprise environments

✔ Poorly managed DevOps activities can backfire on this initiative

✗ "Oops, we promoted the wrong code (there was no Change Request for that)"

8. Establish a core team✔ It may be a contact group with representatives from various

disciplines/departments

✔ Team members must be psychologically compatible -- some team-building events might help with identifying those candidates


9. Invest in infrastructure (not applicable if you operate in the Cloud)✔ Provision infrastructure for DevOps operations (servers, version control

19


system, etc.)

10. Pilot✔ You can build on any PoC projects you managed to sneak in Step 4

above

✔ Make sure you can demonstrate the repeatable and reliable nature of DevOps' software delivery process

11. Enterprise roll-out

✔ Don't forget that it was a joint effort

✔ Time to celebrate!

2.12 Select DevOps Techniques and Practices

The DevOps capability is supported by a number of common techniques and practices, including:

◊ Collaborative steering

◊ Continuous testing of all aspects of the application delivery pipe-line

◊ A Version Control System

◊ A Bug Tracking System

◊ Iterative and frequent integration and deployment

◊ Automation

◊ Change Management

◊ Monitoring and auditing


Collaborative steering ◊ Collaborative and transparent working environment promotes visibility

and agility of software development processes

◊ In order to receive timely notifications or feedback, you may want to set up a Web UI dashboard publishing application lifecycle status in real

20


time for all parties concerned

Continuous testing of all aspects of the application delivery pipe-line ◊ Where applicable, DevOps assures quality of software code;

deployment scripts for all environments (QA, UAT, Staging, etc.); scripts for setting up the infrastructure components (VMs, databases, application servers, etc.),


A Version Control System ◊ For change tracking and consistency of your application code, admin

and deployment scripts, keep them in a version control system (VCS). Changes in your VCS should be accompanied with a check-in message referencing the defect # or change request # addressed by the code check-in, e.g. cvs commit -m "Bug #12YYZ fix" stringUtils.c

A Bug Tracking System ◊ Defects and issues must be tracked, accounted for, and acted upon

◊ The use of a bug tracking system is regarded as a hallmark of a mature software engineering practice

Iterative and frequent integration and deployment ◊ With this practice in place, you can guarantee consistent and

predictable release times; applications can be installed reliably and repeatably


Automation◊ Processes that need manual intervention may occasional fail due to the

human factor (to err is human!)

◊ Establish the continuous automation practice. Keep automation scripts in your VCS

21


Change Management (CM) ◊ This is a critical aspect of DevOps that is often overlooked

◊ CM is a core part of the overall IT governance process

◊ CM helps track changes introduced to the target system by recording the reason for change, scope of change, references to the applicable VCS revisions of assets involved in the change, etc.

◊ CM as a system of record addresses the audit requirements and promotes transparency


Monitoring and auditing◊ Runtime parameters (response time, computing throughput, etc.) and

other metrics of the application (service availability, time to recover from an outage, etc.) deployed in production must meet the client's Service Level Agreement (SLA); the only practical way to capture those parameters is to run your application in a production-like environment

◊ Have a dedicated production-like Performance (a.k.a. Production) Testing Environment (PTE) to collect critical runtime parameters of the application before its release into production

Notes:

Cloud vendors offer run-time monitoring systems that provide sufficient data for clients to have a clear picture of their run-time environments. For example, AWS offers the CloudWatch monitoring system that lets developers view and monitor operational and performance metrics for AWS resources such as Amazon EC2 instances, EBS volumes, Elastic Load Balancers, and RDS DB instances. In addition, clients can add application metrics and business information to CloudWatch for monitoring alongside AWS resource metrics. Clients can use current and historical metrics to troubleshoot issues and discover trends, create and edit alarms to be notified of problems, etc.

22


AWS's CloudWatch Service Dashboard

2.17 Service Quality Metrics

An SLA lists contractual terms and mutual obligations of the client and the service provider and run-time metrics (Quality of Service parameters) to help physically measure SLA parameters, including:

◊ Availability – total up-time / total time per reporting period (month/year)

Up-time is affected by outages and interrupted service duration

Is also impacted by mean-time to enlist new resources and release of unneeded ones

◊ Reliability – guaranteed rate of successful responses

Expressed as mean-time between failures (MTBF)

◊ Performance – service delivery time guarantees

Network throughput, serviced requests per second, response time, etc.

◊ Resiliency – capability to efficiently absorb load spikes

23


2.18 Summary

There are no prescribed solutions for standing up the DevOps capability

In order to make it an integral part of the Enterprise, DevOps needs to be placed under control within the existing enterprise governance processes

Some of the techniques and practices that help with establishing DevOps capability in the Enterprise include:

◊ Collaborative steering

◊ Continuous testing of all aspects of the application delivery pipe-line

◊ A Version Control System

◊ Iterative and frequent integration and deployment

◊ Automation

◊ Change Management

◊ Monitoring and Auditing

24

Chapter 3 - DevOps Tools


Compare IaaS with PaaS

Overview of select DevOps tools

Overview of the IBM Collaborative Lifecycle Management unified platform

3.1 The Choice of Cloud Platform

The DevOps practice has been popularized by organizations adopting the Cloud-as-a-Service computing model (Netflix, Flickr, et al)

Many enterprises are adopting the Cloud-as-a-Service computing model either by setting up a Virtual Private Cloud (VPC) within a public cloud environment or setting up on-premise cloud-like environments

You have a choice between the IaaS and the PaaS platform options which have different implications for DevOps

3.2 IaaS for DevOps

An IaaS platform gives you the lowest level of access to cloud infrastructure: VMs, virtual networks and load balances, a choice of storage solutions (NoSQL, Relational Databases, etc.), etc.

◊ Popular IaaS platforms are: Amazon Web Services(AWS), Google Compute Engine, Microsoft Azure, Cloud Foundry, OpenStack, Rackspace

To effectively and efficiently manage large cloud environments, heavy DevOps involvement is needed

◊ DevOps will be responsible for patching OS / software, formatting raw block storage units, setting up security (open/close virtual firewall ports, managing ACL), etc.


3.3 PaaS for DevOps

A PaaS platform gives users a selection of sandboxed run-time environments, APIs for accessing managed storage and messaging systems, etc.

Popular PaaS platforms are: Microsoft Azure (built on top of the Azure IaaS platform), Google App Engine, AWS Elastic Beanstalk (built on top of the AWS IaaS platform), Heroku (acquired by Salesforce.com in 2010), OpenShift, CloudFoundry, IBM Bluemix (built on top of CloudFoundry)

Due to the managed nature of PaaS, most of the run-time and provisioning tasks are handled by the platform vendor; DevOps involvement is limited to code promotion and some allowed high-level environment tuning

In essence, DevOps perform the push, scale, update types of activities on PaaS

◊ The Ops side of DevOps can be safely handled by Dev alone

Notes:

The first release of the Bluemix Enterprise Cloud Platform (originally printed as BlueMix) was announced by IBM on 24 Feb 2014; the platform was positioned as

"… a unique new development environment and capabilities-as-a-service to help clients and developers speed the adoption of "hybrid" clouds, which have the potential to usher in a new era of innovation across the enterprise. As part of its initiative, IBM has invested more than $1 billion for software cloud development and is launching new capabilities running on SoftLayer." [http://www-03.ibm.com/press/us/en/pressrelease/43257.wss]

3.4 Containerization Tools

A popular approach to gain a better utilization of a single physical / virtual machine's resources is to use containerization tools that allow for creating and running multiple VMs in containers on a single control host

Popular containerization tools are:

◊ LXC

Works on Linux hosts by leveraging modern Linux kernel's cgroups

26


capability for resource containerization such that applications' view of the underlying OS is completely isolated

◊ Solaris Containers (Zones) Cause a very low CPU and RAM overhead

First bundled with Oracle Solaris 11 release

◊ Docker An open-source project that automates the deployment of

applications inside software containers (e.g. LXC)

3.5 System Configuration Automation and Management

For large environments, configuration automation and management becomes a dire necessity

Most of configuration automation and management systems are built around the "infrastructure-as-code" paradigm

Popular Configuration Management tools are:

◊ Puppet Open source tool written in Ruby; runs on Linux and Windows

Allows to manage system configuration declaratively via its domain-specific language (DSL)

You can get Enterprise level support from Puppet Labs, a privately held company behind Puppet

◊ Chef Used to manage both Linux and Microsoft Windows OSes

Written in Ruby and Erlang

Some claim that Chef is more flexible than Puppet, albeit at the expense of more complex system administration

3.6 System Configuration Automation and Management

SaltStack (or, simply, Salt)

27


◊ A Python-based open source configuration management system that competes primarily with Puppet, Chef, and Ansible

Ansible◊ A Python-based open source configuration management system

◊ Commercial support is provided by Ansible, Inc.

◊ Supports Red Hat, Debian, CentOS, OS X, and BSD Linux and Unix flavors; Windows OSes support started as of version 1.7

Ubuntu Juju◊ An open source service orchestration management tool sponsored by

Canonical, the company behind Ubuntu

◊ Juju provides services via charms ("smart" software bundles)

◊ Juju is used to install software, start/stop a service, manage relationships with other charms, upgrade charms, etc.

3.7 Continuous Integration (CI) Systems

Jenkins

◊ A Java-based open source continuous integration (CI) tool

◊ Forked from “Hudson” in 2010

Hudson is now part of Eclipse Foundation with much weaker traction in the CI IT community

◊ At its core, Jenkins is a Java Web server (e.g. Tomcat)

◊ It supports integration with a number of version control systems (VCS), including AccuRev, CVS, Git, Perforce, Subversion, Clearcase, et al

◊ Integrated with Apache Ant and Apache Maven build systems

TeamCity

◊ An automated build management system and CI server written in Java

◊ Sponsored by JetBrains (https://www.jetbrains.com/)

◊ TeamCity is a commercial product licensed from JetBrains

28


There is a free edition that supports up to 20 build configurations and 3 build agents

3.8 Build and Dependency Management Systems

Apache Ant◊ A Java-based command-line tool for automating software build

processes by way of scripted targets and tasks

◊ Does not impose any coding conventions nor prescribed directory layouts for build projects

◊ Uses Apache Ivy for dependency management

Maven◊ Distributed under Apache License 2.0

◊ Contrary to Apache Ant, Maven uses naming conventions and prescribed folder structure for the build processes

◊ Comes with many pre-defined targets for common project tasks (code compilation, packaging, etc.)

◊ Maven dynamically downloads the required dependencies (Java libraries and Maven plug-ins) from one or more Central repositories and stores them locally

3.9 Build and Dependency Management Systems

Gradle◊ A project automation tool that is built around concepts of Apache Ant

and Apache Maven

◊ Written in Java and Groovy

◊ Uses Groovy-based domain-specific language (DSL)

◊ Designed for managing large projects backed up by complex build graphs, optimizing the build time by skipping parts of the project which have already been built

29


◊ Gradle is tightly integrated with Ant importing Ant build files as external scripts to be executed

Apache Ivy

◊ A sub-project of the Apache Ant project for resolving project dependencies

◊ To some extend, Ivy competes with Apache Maven, which also manages dependencies

However, Maven does more: it is a complete build system

3.10 Select DevOps Tools

Issues & Project Management Software◊ JIRA, Bugzilla, Trello

Monitoring, Alerting, and Trending:◊ Cacti, Ganglia, Graphite, Icinga, Nagios, New Relic, PagerDuty

Logging◊ Loggly, Logstash, PaperTrail, Splunk, SumoLogic

Process Supervisors◊ Bluepill, Monit , Upstart, systemd

3.11 Collaborative Lifecycle Management Solutions from IBM

IBM identifies Collaborative Lifecycle Management (CLM) as a holistic discipline focusing on improving software quality and increasing the velocity of software delivery

The CLM capability is built around a combination of Enterprise practices and disciplines, including:

◊ requirements management

◊ quality management

◊ change and configuration management

30


◊ project planning and tracking

3.12 The Collaborative Lifecycle Management Diagram

Collaborative Lifecycle Management connects business analysts, developers, and testers

Source: IBM Knowledge Center

3.13 The IBM Collaborative Lifecycle Management Platform

IBM integrates a number of software delivery tools and systems into a unified Collaborative Lifecycle Management (CLM) platform:

◊ Rational Team Concert

◊ Rational Quality Manager

◊ Rational DOORS Next Generation

The CLM platform brings together the complete set of application lifecycle management (ALM) capabilities that are mandated in the Enterprise space

3.14 Rational Team Concert (RTC)

Rational Team Concert (RTC) creates one single agile project

31


collaborative environment for:

◊ Task and issue tracking

◊ Source control

◊ Agile project management and planning

◊ Continuous Builds and Integration

Some DevOps staff working with RTC mention its high price and not consistent quality across the functional areas in its first versions

RTC is free for teams up to 10 developers

Notes:

RTC is shipped with adapters for HP ALM, Atlassian JIRA and open source Git should you wish to integrate with your existing Application Lifecyle Management systems.

Source: IBM RTC

3.15 Rational Quality Manager (RQM)

Rational Quality Manager (RQM) is positioned as a "collaborative hub" for the Quality Assurance aspect of Application Lifecyle Management (ALM)

RQM has the following capabilities:

◊ Test planning and management

◊ Test design, creation, and execution

32


◊ Traceability with requirements

◊ Change management

RQM provides public REST API for CRUD (create, read, update and delete) operations against its repositories

3.16 Rational DOORS Next Generation (DNG)

Rational DOORS Next Generation (DNG) is a requirements management solution

It helps with activities related to requirements capturing, storing, tracing and management

◊ DNG helps assess the impact of any planned application change by linking submitted change requests to original requirements

DNG provides a secure repository for storing requirements as a Word document or an Excel spreadsheet (including CSV files)

DNG help maintain compliance with industry standards and applicable regulations

It integrates with IBM Rational Team Concert, IBM Rational Quality Manager and IBM Rational Rhapsody Design Manager on IBM Jazz collaborative lifecycle management platform

3.17 Summary

To effectively and efficiently manage large environments on an IaaS Cloud platform, heavy DevOps involvement is needed

DevOps involvement in a PaaS environment is limited to simple code promotion activities and some high-level environment tuning

DevOps practice has a wide variety of tools for the job at its disposal from both open source and closed source projects

IBM is very active in the area of Collaborative Lifecycle Management (CLM) offering clients the unified CLM platform, which is built around the following systems:

33


◊ Rational Team Concert

◊ Rational Quality Manager

◊ Rational DOORS Next Generation

34

Chapter 4 - Introduction to Puppet


Puppet introduction

Design of Puppet

Facter utility

Geppetto

4.1 What is Puppet

Puppet is an open source configuration management system that supports the "Infrastructure-as-code" configuration model

It runs on most Unix-like and Windows OSes

Puppet is developed and sponsored by Puppet Labs

Until version 2.7.0 released under GNU General Public License (GPL), later versions are released under the Apache License 2.0

Notes:

One of the differences between GPL and Apache Licensing is in their handling of releases / code forks. Apache License allows releases and forks under a different license, while GPL prevents this "mutation". GPL forces developers to release their software under a GPL-compatible license if their software links to any GPL library. Many developers find this GPL arrangement rather restrictive, calling it "viral" and "cancerous".

4.2 Puppet's Domain Specific Language

Puppet uses its own Ruby-based Domain Specific Language (DSL) to describe the target system configuration, resources and their state

Resources that can be described in the Puppet DSL include users, files, services and software packages

Portability across supported operating systems is achieved by way of abstracting away system details in a set of primitives implemented in the Puppet's Resource Abstraction Layer (RAL)


◊ RAL shields system administrators from direct interactions with system commands of the target OS, e.g. rpm, yum, apt for software package management , etc.

◊ Puppet uses the target system-specific providers

4.3 "Infrastructure-as-code" in Puppet

Basically, you tell Puppet how you want your machines be configured in instructions you specify in the Puppet DSL; instructions are translated into underlying OS commands executed against resources specified in Puppet's "manifest" files

The actions prescribed in the manifests are distributed across all nodes of your manged system

For a given system resource, Puppet applies changes, if any required, via a three-step process:

◊ Retrieve the current state of that resource

◊ Compare the current state of the resource to the configured (desired) state

◊ Apply any changes necessary, if states differ

4.4 Example of the Puppet DSLfile {'/my/folder/some_file.dat': ensure => present, mode => 640, content => "The new file content",}

The above code is written in the Puppet DSL which is, basically, a set of attributes that specify the state desired for a given configuration resource (a file in our case)

The above code will be translated in the following actions against the file resource:

◊ The /my/folder/some_file.dat will be created, if it doesn't exist

36


◊ Content of the file will be set to the value of the content attribute

If the file already exists, its content will be updated if it is different from the one specified by the content attribute

◊ File permissions will be set to 640 ( -rw-r-----) This file is a Puppet manifest that you save on the disk, and instructions

contained in it can be applied by running the following command against the manifest file:

puppet apply <the name of your manifest file>

4.5 Main Puppet Artifacts

Fact ◊ System data collected on each node and used to compile

configurations

◊ Facts are discovered by a stand-alone utility called Facter (more on it a bit later)

Manifest ◊ Files containing Puppet DSL code

Modules◊ Collections of manifests

Catalog

◊ A catalog contains a compilation of all the resources to be applied to a given node as well as the relationships between those resources.

Report ◊ The collection of all events generated during application of a given

Catalog

◊ Provided by agents (more on them a bit later)

Certificates

◊ Used for authentication

37


4.6 Puppet Design

At its core, Puppet is designed as a client/server system with elements of the "serverless" mode

◊ The "serverless" mode, clients pull all the configuration files from the server and perform prescribed actions on their own

The server is called the Puppet Master; clients are called agents

Agents run on all managed nodes with one agent process per host

The Puppet Master has access to all of the installed Puppet modules, and any configured back-end databases and services

Every agent is matched up with a utility process called Facter running on the same machine as the agent; Facter is used to collect the host's system configuration information

4.7 Puppet Workflow Orchestration

The process begins with Agents starting to gather system information about their hosts using Facter

Information about hosts is packaged as facts (system inventory data) which are sent to the Master Puppet server

The Puppet Master compiles the facts into a Catalog which specifies how the nodes should be configured based on their OS type and version, resources, etc.

The compiled configuration is pushed out to the respective agents

Agents apply the configuration on their local hosts

◊ Agents use system-specific providers to enforce the resource state specified in the manifests

Agents report the results back to the Master Puppet server

4.8 Facter

System information discovery and gathering in the form of facts is performed via a stand-alone utility written in Ruby called Facter

38


Facts about nodes / systems are provided in the key=>value pairs and include the following information:

◊ Target OS type and version, hardware details, network settings, IP addresses, MAC addresses, SSH keys, and more

You can extend Facter by including your own custom facts or site-specific facts

Facter can be used independently of Puppet

4.9 Facter Fact Discovery

First, Facter figures out target OS's type and version

Based on the target OS type and version, Facter begins gathering system information by introspecting underlying OS's system configuration files and interfacing with the system-specific utilities

When used in conjunction with Puppet, facts gathered through the system introspection by Facter, are presented as variables in the manifest files using the $::key_name notation, e.g.

case $::operatingsystem { 'CentOS': { include centos } 'MacOS': { include mac } }

The Puppet Master applies actual values when evaluating the conditional expressions during manifest compilation

Notes:

The list below lists all of the built-in facts that ship with Facter 2.4.x. Not all of them are supported on every system, and you may be required to use some of the custom facts delivered via Puppet modules. For mode details, visit http://docs.puppetlabs.com/facter/latest/core_facts.html#summary

architecture kernelmajversion ps augeasversion kernelrelease puppetversionblockdevice_<devicename>_size kernelversion rsc_<RACK INST DATA>blockdevice_<devicename>_vendor ldom rubyplatformblockdevice_<devicename>_model lsbdistcodename rubysitedirblockdevices lsbdistdescription rubyversioncfkey lsbdistid selinux

39


dhcp_servers lsbdistrelease selinux_config_modedomain lsbmajdistrelease selinux_config_policyec2_<EC2 INSTANCE DATA> lsbminordistrelease selinux_enforcedfacterversion lsbrelease selinux_policyversionfilesystems macaddress sshfqdn macosx system32gid manufacturer system_uptimehardwareisa memory timezonehardwaremodel netmask uptimehostname network uptime_daysid operatingsystem uptime_hoursinterfaces operatingsystemmajrelease uptime_secondsipaddress operatingsystemrelease virtualipaddress6 os is_virtualiphostnumber osfamily vlanskernel partitions xendomains path zones_<ZONE> physicalprocessorcount processor processors

4.10 Facter Examplearchitecture => i386bios_release_date => 05/20/2014bios_vendor => Phoenix Technologies LTDbios_version => 6.00blockdevices => sda,sr0os => {"name"=>"CentOS", "family"=>"RedHat", "release"=>{"major"=>"6", "minor"=>"5", "full"=>"6.5"}}gid => rootinterfaces => eth0,loipaddress => 192.168.89.128processorcount => 2processors => {"models"=>["Intel(R) Core(TM) i5-3350P CPU @ 3.10GHz", "Intel(R) Core(TM) i5-3350P CPU @ 3.10GHz"], "count"=>2, "physicalcount"=>2}system_uptime => {"seconds"=>3272, "hours"=>0, "days"=>0, "uptime"=>"0:54 hours"}

Snippets of system information captured by Facter on a x86 machine running CentOS 6.5

4.11 Extending Facter

If the default set of fact finding methods is not sufficient, you can create a new fact finder and add it to Facter

40


You have two options for Facter extension:

◊ Using environment variables

◊ Using Ruby and packaging fact finders as *.rb files

For more details, visit https://puppetlabs.com/blog/facter-part-1-facter-101

4.12 Geppetto

Geppetto is an Eclipse-based IDE for Puppet and Puppet Enterprise

It provides a toolset for developing Puppet modules and manifests

Geppetto supports syntax highlighting, tooltips, code completion, code formatting, and error tracing/debugging of problems with your Puppet code that are causing configuration failures

Geppetto supports integration with active PuppetDB (which stores most recent facts from every node and most recent catalog for every node) which gives visibility in the Puppet Resource Events through the Geppetto's Events View

◊ The Events View lists failed / successful changes undertaken by Puppet along with affected resources

4.13 Puppet Lab Services

Puppet Labs, the company behind Puppet, provides the following services:

◊ Enterprise Puppet support

◊ Puppet training and professional services

◊ The Puppet certification program

4.14 Puppet Enterprise Licensing

Puppet is open source software and free and you can get community-based support

Puppet Labs provide the paid version of Puppet called Puppet Enterprise

Puppet Enterprise includes personalized support from onboarding to

41


ongoing maintenance of clients' Puppet infrastructure

Clients of Puppet Enterprise have two licensing models to chose from:

◊ The annual subscription fee model paid yearly

◊ The perpetual license where clients make a one-time license fee payment. Support can be chosen as required and is renewed yearly

4.15 Puppet Enterprise Support

Support comes in two flavors:

◊ Standard (sufficient in many practical implementations)

6am – 6pm PST, Monday – Friday; 5 cases / month

◊ Premium (recommended for mission critical applications or organizations working in different time zones)

24 x 7 x 365 for Priority 1; unlimited number of cases

4.16 Puppet Enterprise Feature Set (1/2)

Puppet Enterprise includes the following functionality that has been tested in complex, heterogeneous enterprise environments:

◊ Pre-configured Puppet

◊ An Admin Web UI

◊ Automatic discovery of resources and configuration across the supported environment

◊ Orchestration features

Help you push out critical updates and patches across the supported environment

4.17 Puppet Enterprise Feature Set (2/2)

Cloud provisioning

◊ Supported by a suite of command-line tools for creating and managing

42


the lifecycles of virtual machine instances in cloud computing infrastructures based on VMware vSphere, Amazon EC2 and Google Compute Engine

Role-based access control against external authentication systems, including:

◊ Active Directory

◊ LDAP

◊ Google Apps Directory,

◊ etc.

4.18 Summary

Puppet is an open source configuration management system that supports the "Infrastructure-as-code" configuration model

It runs on most Unix-like and Windows OSes

The desired state of your managed nodes is coded in the Puppet DSL which is applied across all nodes in a three-step process:

◊ Retrieve the current state of that resource

◊ Compare the current state of the resource to the configured (desired) state

◊ Apply any changes necessary, if states differ

While Puppet is free and you can get community-based support, Puppet Labs provide the paid version of Puppet called Puppet Enterprise

43

Chapter 5 - Docker Introduction


Docker introduction

Linux cgroups and namespaces

Docker vs traditional virtualization

Docker command-line

5.1 What is Docker

Docker is an open-source (and 100% free) project for IT automation

You can view Docker as a system for creating virtual environments which are extremely lightweight virtual machines

Docker allows the deployment of applications and their dependencies inside Linux containers supporting the multi-tenancy deployment model on a single host

◊ A container is a group of controlled processes associated with a separate tenant executed in isolation from other tenants

Written in the Go programming language

Notes:

The Go programming language (also referred to as golang) was developed at Google in 2007 and release in 2009. It is a compiled language – it does not require a VM to run it (like in C# or Java) – with automated garbage collection. Go offers a balance between type safety and dynamic type capabilities; it supports imperative and concurrent programming paradigms.

5.2 Where Can I Ran Docker?

Docker runs on any modern-kernel Linux distributions


◊ You can install Docker inside a VirtualBox and run it on OS X or Windows

◊ Docker can be booted from the small footprint Linux distribution boot2docker

5.3 Docker and Containerization on Linux

Docker leverages resource isolation features of the modern Linux kernel offered by cgroups and kernel namespaces

◊ The cgroups and kernel namespaces features allow creation of strongly isolated containers acting as very lightweight virtual machines running on a single Linux host

Docker helps abstract operating-system-level virtualization on Linux using abstracted virtualization interfaces based on libvirt, LXC (LinuX Containers) and systemd-nspawn

◊ As of version 0.9, Docker has the capability to directly use virtualization facilities provided by the Linux kernel via its own libcontainer library

5.4 Linux Kernel Features: cgroups and namespaces

The control group kernel feature (cgroup) is used by the Linux kernel to allocate system resources such as CPU, I/O, memory, and network subject to limits, quotas, prioritization, and other control arrangements

The kernel provides access to multiple subsystems through the cgroup interface

◊ Examples of subsystems (controllers):

The memory controller for limiting memory use

The cpuacct controller for keeping track of CPU usage

The cgroups facility was merged into the Linux kernel version 2.6.24

Systems that use cgroups: Docker, Linux Containers (LXC), Hadoop, etc.

The namespaces feature is a related to cgroups facility that enables different applications to act as separate tenants with completely isolated

46


views of the operating environment, including users, process trees, network, and mounted file systems

5.5 The Docker-Linux Kernel Interfaces

Source: Adapted from http://en.wikipedia.org/wiki/Docker_(software)

5.6 Docker Containers vs Traditional Virtualization

System virtualization tools or emulators like KVM, Xen, HyperV, VMware, etc. boot virtual machines from a complete guest OS image (of your choice) and basically emulate a complete machine, which results in a high operational overhead

Virtual environments created by Docker run on the existing kernel’s image of the host's OS without a need for a hypervisor

◊ This leads to very low overhead and significantly faster container start-up time

Docker-provisioned containers do not include or require a separate operating system (it runs in the host's OS)

◊ This circumnstance puts a significant limitation on your OS choices

5.7 Docker Containers vs Traditional Virtualization

Overall, traditional virtualization has advantages over Docker in that you

47


have a choice of guest OSes (as long as the machine architecture is supported)

◊ You can get only some (limited) choice of Linux distros

You still have some choice: e.g. you can deploy a Fedora container on a Debian host

◊ You can, however, run a Windows VM inside a Linux machine using virtual machine emulators like VirtualBox (with less engineering efficiency)

With Linux containers, you can achieve a higher level of deployed application density compared with traditional VMs (10x more units!)

Docker runs everything through a central daemon which is not a particularly reliable and secure processing model

5.8 Docker as Platform-as-a-Service

Docker defines an API for creating, deploying and managing containers that make up highly distributed systems spanning multiple physical machines

◊ Docker-based systems can also efficiently run multiple isolated applications on a single physical machine

On-demand provisioning of applications by Docket supports the Platform-as-a-Service (PaaS)–style deployment and scaling

5.9 Docker Integration

Docker can be integrated with a number of IT automation tools that extend its capabilities, including

◊ Ansible

◊ Chef

◊ Jenkins

◊ Puppet

◊ Salt

48


Docker is also deployed on a number of Cloud platforms

◊ Amazon Web Services

◊ Google Cloud Platform

◊ Microsoft Azure

◊ OpenStack

◊ Rackspace

5.10 Docker Services

Docker deployment model is application-centric and in this context provides the following services and tools:

◊ A uniform format for bundling an application along with its dependencies which is portable across different machines

◊ Tools for automatic assembling a container from source code: make, maven, Debian packages, RPMs, etc.

◊ Container versioning with deltas between versions

5.11 Docker Application Container Public Repository

Docker community maintains the repository for official and public domain Docker application images: https://hub.docker.com/account/signup

49


5.12 Competing Systems

Rocket container runtime from CoreOS (an open source lightweight Linux kernel-based operating system)

LXD for Ubuntu from Canonical (the company behind Ubuntu)

The LXC (Linux Containers), used by Docker internally

Many more are on the way …

Other systems exist for non-Linux OSes

5.13 Docker Command-line

The following commands are shown as executed by the root (privileged) user:

docker run ubuntu echo 'Yo Docker!'◊ This command (not very useful) will create a docker container on the fly

and execute the echo command on it and then shuts down

docker ps -a◊ This command will list all the containers along with their IDs created by

Docker

5.14 Starting, Inspecting, and Stopping Docker Containersdocker start -i <container_id>

◊ This command will start an existing stopped container in interactive (-i) mode (you will get the target system shell on start-up)

docker inspect <container_id>◊ This command will provide JSON-encoded information about the

running container identified by container_id

docker stop <container_id>◊ This command will stop the running container identified by container_id

50


5.15 Summary

Docker is a system for creating virtual environments which are, for all intents and purposes, lightweight virtual machines

Docker containers can only run the type of OS that matches the host's OS

Docker containers are extremely lightweight (although not so robust and secure), allowing you to achieve a higher level of deployed application density compared with traditional VMs (10x more units!)

On-demand provisioning of applications by Docket supports the Platform-as-a-Service (PaaS)–style deployment and scaling

51

Chapter 6 - Introduction to Continuous Integration and Jenkins-CI


Agile Development

Continuous Integration

History of Jenkins

6.1 Agile Development

Agile Development begins with Extreme Programming (XP)

◊ Invented/Promoted by Kent Beck and associates

Beck describes XP as:

◊ A philosophy of software development based on the values of communication, feedback, simplicity, courage, and respect

◊ A body of practices proven useful in improving software development.

◊ A set of complementary principles, intellectual techniques for translating the values into practice, useful when there isn't a practice handy for your particular problem.

◊ A community that shares these values and many of the same practices.

6.2 Agile Development (cont'd)

As time went on, people took inspiration from XP and developed other approaches with much the same goals

◊ Speed

◊ Include the Customer Early

◊ Don't wait til you understand the entire problem - you'll understand it while you fix it

◊ Deliver usable value at every iteration


◊ Make reasonable use of modelling and development tools

6.3 Agile Development (cont'd)

Two themes emerge from XP and Agile Development

◊ Trust

Vendor, customer, managers, developers all need to trust each other

◊ Automation

If we're going to go fast, we need tools that make it easy to produce software

Trust is a social problem – solution is all about different approaches to process, requirements gathering, project management, etc.

Automation is about tools – and Continuous Integration is a primary tool of Agile Development

6.4 What is Continuous Integration

“Integrate and test changes after no more than a couple of hours.”

Beck & Andres “Extreme Programming Explained”

◊ Integrate and build the complete product

◊ If a website, build the website

◊ If a GUI install, build the installer

◊ Etc.

6.5 What is Continuous Integration (cont'd)

Purposes

◊ You find out quickly about integration problems

◊ Immediately evident if a code change “breaks the build”

◊ Prevents a long drawn-out integration step at the end of code changes

54


◊ Should be complete enough that eventual first deployment of the system is “no big deal”.

6.6 What is Continuous Integration (cont'd)

Usually, CI goes along with Test-First design and automated QA

◊ Run the unit-test suite and smoke testing to ensure the build isn’t broken

Clearly, automatic build is a pre-requisite

◊ So we need a build system – Maven for instance

Can be synchronous or asynchronous

◊ Asynchronous – Integration happens automatically on code committal

◊ Synchronous – Trigger the build manually after a development session

6.7 What is Continous Integration (cont'd)

Side Effects

◊ Generate development reports

◊ Install to QA, User Test, etc

◊ Always have an installable artifact

It’s a great time to generate development metrics

◊ E.g. Code Coverage, Standards Compliance, Static Analysis

55


6.8 Typical Setup for Continuous Integration

6.9 Typical Setup for Continuous Integration

Notes:

◊ CI system gets the code directly from version control

◊ Build is independent of any local artifacts that are on the developer’s machine

Controlled links to corporate repository and Maven Central

Goal is to ensure that the package can be built from the corporate repository

◊ Jenkins can have connections to a deployment environment

Production Staging

User Acceptance Testing

QA

Load Test

Etc…

◊ It turns out that if we use Maven, we ABSOLUTELY NEED a local repository manager

56


More info to follow…

6.10 Jenkins Continuous Integration

Originally developed at Sun by Kohsuke Kawaguchi?

◊ Originally “Hudson” on java.net circa 2005

◊ Jenkins forked in November 2010

◊ Hudson is still live, part of Eclipse Foundation

◊ But Jenkins seems to be far more active

6.11 Jenkins Features

Executes jobs based on a number of triggers

◊ Change in a version control system

◊ Time

◊ Manual Trigger

A Job consists of some instructions

◊ Run a script

◊ Execute a Maven project or Ant File

◊ Run an operating system command

User Interface can gather reports

◊ Each job has a dashboard showing recent executions

6.12 Running Jenkins

You can run Jenkins Standalone or inside a web container

You can setup distributed instances that cooperate on building software

Can setup jobs in place of what might have been script commands.

57


6.13 Summary

Continuous Integration is a powerful tool for agile software development

Jenkins is currently the dominant Continuous Integration tool

58

wa2451 devops fundamentals - … devops fundamentals web age solutions inc. ... • the application...

Documents