how to triple your speed of development using automation

How to Triple Your Speed of Development Using Automation | 26/03/2017

How to Triple Your Speed of Development Using AutomationJohanan Liebermann26/03/2017

AWS and GCP Division Overview | Nov. 2016

© 2016 AllCloud

“AWS Managed Service Partners are skilled at cloud infrastructure and application migration, and deliver value to customers by offering proactive monitoring, automation, and management of their customer’s environment.”https://aws.amazon.com/partners/msp/http://www.allcloud.io/msp

AWS Next-Gen (v3) Managed Service Partner (MSP)

https://aws.amazon.com/partners/msp/

http://www.allcloud.io/msp


Johanan Liebermann

Software Developer at [email protected]

About Me


●Intro●Using automation to speed up your development●Two real use cases (instance-based and container-

based)

Topics


Intro


●Today’s software world moves fast!●Tiny startups can challenge giants●Delivering a product to the masses is easy●Release cycle takes days, not months●Time to market can make or break your product

The Need for Speed


How?


Automate Everything!


“A set of practices which make software lifecycle more efficient by bridging the gap between development and operations and aiming for automation in every aspect.”

DevOps


●Continuous Integration & Continuous Delivery / Deployment

●A major part of DevOps●Continually building, testing and deploying your

code●Catch problems as soon as possible●Work on multiple features in parallel●Ship code often in small chunks

CI/CD


●Version Control●Build Process●Testing●Configuration Management●Infrastructure Provisioning●Deployment & Release

What Areas Can You Automate?


Version Control Managing changes to

source code


●Git hooks○Trigger build, test and deploy processes

○Enforce commit message style

○Enforce coding standards (can’t commit bad code)

○Notify developers about code-related events

●Automated handling of branches and pull requests○GitHub API

○Bitbucket API

○GitLab API

Version Control Automation


Build Process Generating artifacts from

source code


●Build automation tools○Make, Maven, Gradle, Grunt, Gulp, Rake...

●Dependency / package managers○pip, npm, RubyGems, Conda, Bower...

Build Process Automation


●Your artifacts should be properly-versioned●Your artifacts should be consistently-available●Many types of artifacts

○Language-specific - JAR, Gem

○OS-specific - RPM, Deb, APK, IPA

○Images - Docker, Vagrant, AMI

Handling Artifacts


●Dedicated solutions: Artifactory, Nexus, ECR, Docker Hub

●Simple file storage: S3

Artifact Storage Options


Verifying the quality of your code

Testing


Unit Testing

Types of Testing

Integration Testing

Functional Testing


●Making sure a small, isolated piece of code works properly

●Done as part of the build process●Unit tests should be an integral part of the code

●A failed unit test should always stop the build process●Typically done using built-in language features or

dedicated frameworks (JUnit, pytest, rspec…)●Provides a fast feedback for new code●Easy to notify the person who broke the code

Unit Testing


●Making sure a couple of components work together properly

●Should be done as part of the deployment / release process

●Usually requires deployment to an environment●Typically done using dedicated tools or custom scripts●Typically involves “external” entities such as databases,

3rd party APIs, managed services etc.

Integration Testing


●End-to-end testing of a product●Should be done as part of the deployment / release

process●Typically simulates user interaction for front-end

applications using tools such as Selenium and Appium●Typically simulates API calls for back-end services

using tools such as Ready! API or vREST

Functional Testing


Ensuring your servers are properly configured to serve your application

Configuration

Management


●Typically includes:○Installing packages

○Ensuring configuration files are in the desired state

○Setting OS-level configuration parameters (file permissions, environment variables...)

○Configuring services

●Declarative syntax & idempotent in most cases

Configuration Management


●Dedicated tools: Ansible, Chef, Puppet, SaltStack●Built-in functionality: Dockerfiles, ebextensions●Custom scripts (not recommended)

Configuration Management Tools


●Two approaches to managing your infrastructure○Mutable infrastructure: modifying existing nodes

○ Immutable infrastructure: replacing existing nodes

●Both options are achievable using CM tools○Use CM tools to manage your nodes directly

○Use CM tools to generate images

Mutable vs. Immutable Infrastructure


Infrastructure

ProvisioningCreating, updating and destroying cloud infrastructure


●Relevant mainly for IaaS●Typical resources which need provisioning:

○Compute resources (EC2, ECS, Lambda)

○Network & security (VPC, ELB, CloudFront)

○Databases, cache clusters and queues (RDS, DynamoDB, ElastiCache, SQS)

○Storage (EBS, EFS, S3)

○Miscellaneous services (IAM, SNS, SES, CloudWatch...)

Infrastructure Provisioning


●An approach in which infrastructure == code○Version-controlled

○Tested

●Provides reproducibility●Usually declarative & idempotent

Infrastructure as Code


●Dedicated tools: CloudFormation, Terraform●Built-in functionality: Elastic Beanstalk, Heroku

Infrastructure Provisioning Tools


Making your code available to the users

Deployment & Release


●Back-end services○Update or replace instances / containers / Lambda functions

○Run DB migrations

●Front-end services○Run minification & CSS preprocessing

○Sync S3 buckets

○Invalidate CDN

●Mobile apps○Sign apps

○Upload to App Store / Google Play

Deployment Process


●In-place - simply updating the live nodes●Canary - deploying to a small amount of nodes first,

making sure everything is fine, then updating all the nodes

●Blue/Green - maintaining two identical production environments and switching the activity between them

Back-End Deployment Strategies


Sample Pipeline:

CodeDeploy


●A managed deployment service by AWS●Handles the deployment process for you●Minimizes downtime

○In-place rolling updates

○Built-in support for Blue/Green deployment

●Automatic or manual rollbacks●Platform-agnostic: use with any OS, any runtime and

any CM●Native integration with Auto Scaling and ELB●If you use EC2 - it’s free!

AWS CodeDeploy


●A REST API written in Python on Amazon Linux●Jenkins as an automation server●Artifactory for artifact storage●Ansible for CM●CodeDeploy for deployment

Pipeline Setup


The Pipeline2. Run unit tests3. Build pip package4. Build Ansible RPM

1. Checkout

5. Store pip & RPM

6. Push new revision7. Trigger CodeDeploy

8. Deregister from ELB9. Install RPM using YUM10. Run Ansible11. Verify service12. Register with ELB

Pull RPM & pip - Run pip install- Modify config files- Start service


Sample Pipeline:

ECS


●EC2 Container Service●The native AWS container management service●Runs on EC2●Contains a private Docker registry (ECR)●Native integration with Auto Scaling, ELB and IAM●ECS is free (but ECR has charges)

AWS ECS


●A REST API written in NodeJS

●Jenkins as an automation server●ECR for artifact storage●No need for CM (Docker handles it)●ECS for deployment

Pipeline Setup


The Pipeline2. Build Docker image3. Run unit tests

4. Push image

1. Checkout5. Update Task Definition6. Update Service

7. Replace containers

Pull image


Conclusion


●Implement fully-automated pipelines to leverage the full potential of the cloud

●Automation will boost your speed of development and allow you to focus on your differentiating factors

●Relying on managed services wherever possible will reduce the manpower needed for operating your service and may also reduce your costs

Conclusion


Thank [email protected]

how to triple your speed of development using automation

Technology