Cloud & DevOps ProgramJune Executive Committee

June 18, 2015 Thursday 3:00-­4:00 p.m. 50 Church St., CR-­385

• Meeting Purpose and Intended Outcomes (5 min)

• Progress Against the Plan: Accomplishments (10 min)

• HPAC: www.harvard.edu Relaunch (20 min)

– Project Objectives– Incidents and Design Improvements– Plan Overview

• Cloud & DevOps Executive Status Dashboard (20 min)

Agenda

2

PurposeTo provide the Cloud & DevOps Executive Committee with an update on progress made against the Program Plan

Intended Outcomes• Distribute and review the Cloud & DevOps Executive Status Dashboard

• Discuss ideas for improving DevOps engineering skills• Go over methods for improving engagement and expectation-­setting within application teams

• Provide an overview of HPAC cloud strategy

Meeting Purpose and Intended Outcomes

3

To date, the Cloud & DevOps team has achieved the following key objectives and deliverables:

Progress Against the Plan: Accomplishments

4

Project Description Date Impact UpdateQlikView Provide AA&D

project with application infrastructure in the cloud

July2015

• Cloud deployment will allow team to release codebase 4x faster

• Cost ultimately reduced by x

• Development and stage environments are complete

• Production deployment on target for June 30

Remote DR (SunGard Replacement)

Alternative, improved DR support for applications currently using SunGard

June 2015

(Delayed to July)

• Reduced costs for applications using SunGard

• Improved RTO for applications

• Security approval to move forward with CloudEndure

• CloudEndure code update done• Aleph/PeopleSoft migration plan approved and ready to proceed

Cloud Architecture

Define patterns for configuring environments

Ongoing • Patterns ensure consistent usage and reduced cost

• Decreased development time

• VPC and account configuration completed

• RDS and WebAppStack creation pattern completed

• CloudEndure Microsoft consulting scheduled for July

Simplify and improve code updates and deployment process• Use new AWS technologies to streamline updates and reduce the steps required to release new code

• Improve reliability of code updates

Improve scalability, performance, and maintainability of the production environment• Modernize and maintain the environment without significant impact to the application

Improve security to protect against DDoS attacks• Implement a cloud web app firewall to limit impact to application availability

Additional objectives may be identified for remaining HPAC sites with an intent to reuse the architecture and best practices implemented for www.harvard.edu.

5

Project Objectives: www.harvard.edu Relaunch

6

Category Issue Improvements Benefits

Deployment Complex, error-­prone deployment process Examples: INC01187086: Admin instance out of syncINC01227463: Prod site out of syncINC01268214: Partial code updateINC01112954: Campaign site out of syncINC01133346: Campaign dev not availableINC01246985: GitRepo permission errors

• Implement CodeDeployto manage deployments and updates to code

• Streamline S3 organization

• Create HPAC-­owned Git repository

• Error-­free, self-­validating code deployment

Architecture VPC improvements and improvements for root cause analysis

• Implement Direct Connect for admin access

• Implement self-­service validation

• Support VPN access to ensure encrypted access to instances

• Secure access for developers/admins

• Improved visibility into instance performance

Initial Scope: Improve www.harvard.edu and ensure that the resulting design addresses the underlying causes of known incidents.

Incidents and Design Improvements

7

Category Issue Improvements Benefits

Security Address current DDoS vulnerability and provide facility to perform security scansExamples:INC01055632: Implement security scansINC01184616: PHP security vulnerabilityINC01257035: Denial of service, max DB connections exhausted

• Implement a WAF (web application firewall) to scrub unwanted traffic

• Implement security scanning capabilities

• Reduce the scope and likelihood of a denial of service attack

Maintenance Web server lifecycle managementExamples:INC01026043: PHP update issueINC01184616: PHP update schedulingINC01270406: Online learning blank pagesINC01288325: PHP active support ending

• Improve AMI baking and rollout process

• Quicker launch of newly updated web servers

• Ensures latest PHP updates

Monitoring Provide enhanced server monitoringExamples:INC00990720: Install New RelicINC01287931: Monitor staging environment

• Provide application level monitoring using New Relic

• Implement non-­production monitoring

• Deep analytical performance and availability metrics

Incidents and Design Improvements

8

Item Description Dates

Deliver proposed design

Cloud team delivers proposed conceptual design for www.harvard.edu;; HPAC reviews and provides comments

June 9-­16

Deliver stage environment

Cloud team smoke-­tests and delivers stage environment, providing URL & IAM credentials. HPAC provides users SSH public keys for OS access.

June 30

Package code and perform acceptance testing

HPAC packages code and places artifact in designated S3 bucket, as well as functionally validates environment and produces consolidated list of infrastructure defects;; Cloud team updates/tunes based on defects

July 1-­10

Deliver production environment

Cloud team smoke-­tests and delivers production environment (URL, IAM credentials)

July 28

Package code and perform acceptance testing

HPAC packages code and places artifact in designated S3 bucket, as well as functionally validates environment and produces consolidated list of infrastructure defects;; Cloud team updates/tunes based on defects

July 30-­Aug. 10

Go-­live Relaunch of www.harvard.edu Aug. 14

Key dates for re-­launch are July 14 (stage) and Aug. 14 (production). Following are intermediate deliverables and dates:

Plan Overview

Please see the handout for the most recent Cloud & DevOpsExecutive Status Dashboard.

Cloud & DevOps Executive Dashboard

9

Thank you!

Supporting Materials

Objectives Guiding Principles Key Performance Indicators

The Vision for the Cloud & DevOps Program

1. We are committed to staff growth and development as we pursue program goals

2. We will ensure close collaboration between the program and other HUIT teams to maintain high levels of existing services

3. Improving deployment methods and processes are as important as the technologies we use

4. Consistent architectural and design patterns are critical to achieving enterprise-­level results

5. Communicating with all employees, partners, and customers is crucial to program awareness and understanding

To improve HUIT’s delivery of information technology solutions to the Harvard Community, we will employ new methodologies, tools, and processes that will enable us to simplify and deliver higher-­quality

solutionswith improved robustness and resiliency in a more timely manner.

1. Develop training to transition staff from administrator roles to cloud and DevOps engineering roles

2. Lead staff transition process and create an empowered, service-­focused culture

3. Implement application design and deployment patterns to maximize consistency, quality, and reliability

4. Migrate existing app workloads with a goal of 75% of existing compute from on-­premise data centers to the public cloud

5. Establish operational toolsets and processes to ensure operational effectiveness, awareness, and partnership with service teams

1. Percentage of HUIT employees who have successfully completed Cloud & DevOps training

2. Percentage of total apps migrated to cloud providers

3. Improved app availability from monitoring (uptime percentage)

4. Successful DR testing processes in place — average time to recovery for migrated applications

5. Percent deployment rollbacks6. Cost of deployment solutions compared with onsite measurement

Harvard’s Cloud & DevOps Vision

12