devops - introduction to data science

Introduction to Data Science

Frank Kienle

DevOps – Professional Development

DevOps is about Culture, Automation, Measurement and Sharing (CAMS) and has one major goal

DevOps exists to help the business win •  The scope is broad, but centered on IT •  The foundations are found in Agile and Lean. •  Culture is very important •  Feedback is fuel for innovation •  Automation helps ----- IBM definition DevOps – designing processes for coordinating software development teams with IT operations teams.

DevOps (a portmanteau of development and operations)

30.09.17 Frank Kienle p. 2

DevOps For Dummies, IBM Limited Edition (free book) or https://sdarchitect.blog/understanding-devops/

Collaboration extends to everyone with a stake in the delivery of software -not just between Dev and Ops, but all teams, including test, product management, and executives Continuous Integration to provide rapid feedback so that if a defect is introduced into the code base, it can be identified and corrected as soon as possible Continuous integration (CI) is a software engineering practice in which isolated changes are immediately tested and reported on when they are added to a larger code base Automation: DevOps relies on toolchains to automate large parts of the end-to-end software development and deployment process

DevOps Essentials, see e.g. DevOps For Dummies, IBM Limited Edition (free book) or https://sdarchitect.blog/understanding-devops/


Continuous Delivery Continuous delivery means that the team ensures every change can be deployed to production but may choose not to do it, usually due to business reasons. In order to do continuous deployment one must be doing continuous delivery. Continuous testing is not just a QA function, in fact, it starts in the development environment. The days are over when developers could simply throw the code over the wall to QA and say, “Have at it.” In a DevOps environment, everyone is involved in testing. Devel- opers make sure that, along with delivering error-free code, they provide test data sets. They also help test engineers con gure the testing environment to be as close to the production environment as possible. Continuous monitoring has the goal to quickly determine when a service is unavailable, understand the underlying causes, and most importantly, apply these learnings to anticipate problems before they occur

DevOps Essentials


Monitoring to ensure Visibility – All aspects of the delivery system including building, deploying, testing, and releasing are visible to every member of the team to promote collaboration. Testing to ensure Feedback – Team members learn of problems as soon as possible when they occur so that they are able to fix them as quickly as possible. Continually deploy – Through a fully automated process, you can deploy and release any version of the software to any environment

Deployment Pipeline


Different teams are involved, don’t underestimate the typical conflicting goal of these teams


Dev Team — would like to release as much as possible Ops Team — seeks stability Testers — seek risk reduction

See discussion: https://landing.google.com/sre/interview/ben-treynor.html Site reliability engineering (SRE) is a discipline that incorporates aspects of software engineering and applies that to operations whose goals are to create ultra-scalable and highly reliable software systems. Both the development and SRE teams share a single staffing pool, so for every SRE that is hired, one less developer headcount is available (and vice versa). This ends the never-ending headcount battle between Dev and Ops, and creates a self-policing system where developers get rewarded with more teammates for writing better performing code

Running Productive Systems: Site reliability engineering one specific implementation of DevOps


DevOps: Example Tool Landscape (https://www.slideshare.net/KeithPleas/devopsasaservice-towards-automating-the-automation)


…‘One main reason for using services as components (rather than libraries) is that services are independently deployable’

In the hart of of managing complex software is the usage of Micro Services, see e.g. https://martinfowler.com/articles/microservices.html


•  Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. •  Welcome changing requirements, even late in development. Agile processes harness change for

the customer's competitive advantage. •  Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the

shorter timescale. •  Business people and developers must work together daily throughout the project. •  Build projects around motivated individuals. Give them the environment and support they need,

and trust them to get the job done. •  The most efficient and effective method of conveying information to and within a development

team is face-to-face conversation. •  Working software is the primary measure of progress. •  Agile processes promote sustainable development. The sponsors, developers, and users should be able to

maintain a constant pace indefinitely. •  Continuous attention to technical excellence and good design enhances agility. •  Simplicity--the art of maximizing the amount of work not done--is essential. •  The best architectures, requirements, and designs emerge from self-organizing teams. •  At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior

accordingly.

Manifesto for Agile Software Development, http://agilemanifesto.org


Vagrant is a tool to create and configure lightweight, reproducible, and portable development environments

One possible ’developer environment’ setup


VirtualBox is a free general purpose virtualizer it is free and available on many platforms

Ansible is a provisioning tool used to deploy software and configure systems. With Ansible Galaxy there exist a huge community for sharing

Goal should be to develop in sandboxed environments which can be automated, tested and deployed in other environments as well

…‘Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team’…

Vagrant


Ansible is an open source automation platform. ... Ansible can help you with configuration management, application deployment, task automation. It can also doIT orchestration, where you have to run tasks in sequence and create a chain of events which must happen on several different servers or devices.

Ansible


RUN Ansible: Ad-Hoc: anisible <inventory> -m Playbooks: ansible-playbook file_name.yml Automation Framework: Ansible Tower

•  Ansible playbooks are a way to send commands to remote computers in a

scripted way. •  Instead of using Ansible commands individually to remotely configure

computers from the command line, you can configure entire complex environments by passing a script to one or more systems.

•  Plain-text YAML files that describe the desired state of something. Human and machine readable

Playbooks


Example setup to install python on a bare metal host

30.09.17 Frank Kienle

1 playbook playbooks are a way to send commands to remote hosts in a scripted way. You can configure entire complex environments by passing a script to one or more systems. §  Each playbook is composed of one or more ‘plays’

in a list. (in this example, one play) §  The goal of a play is to map a group of hosts to

some well defined roles, represented by things ansible calls tasks.

Roles build on the idea of include files and combine them to form clean, reusable abstractions –only dive down into the details when needed.

At a basic level, a task is nothing more than a call to ansible modules, here apt and pip package installation, http://docs.ansible.com/ansible/latest/pip_module.html#examples, for dependencies

2

3

p. 15

https://galaxy.ansible.com

Automation projects with content from the community


devops - introduction to data science

Data & Analytics