© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Adrian Cockcroft - AWS VP Cloud

Architecture Strategy

December 1, 2016

Open Source at AWSContributions, Support and Engagement

ARC213

What to Expect from the Session

• Examine how leading AWS customers use open source

• See the value of an open source program

• Understand the trends in open source adoption

• Baseline existing contributions from AWS

• Begin the discussion of next steps in engagement

What do customers want?

Capital One use of open source

Capital Onehttps://developer.capitalone.com/open-source/

“We’re an open source first organization —

actively using, contributing to and managing

open source software projects.”

Capital One Open Source Projectshttps://capitalone.github.io

Cloud Custodian – AWS Lambda based policy engine – ARC312

Hygieia – DevOps delivery pipeline dashboard

OAS-nodegen – Open API Specification support

Ops-Pipeline – scripts for creating environments

Hydrograph – User interface for ETL management

Giraffez – Client library for Teradata

React Native PathJS Charts

SWHttpTrafficRecorder – Swift mobile library

CQRS Manager for Distributed Reactive Services

Netflix open source program

From 2012 NetflixOSS Presentation by @adrianco

Current NetflixOSS Programhttp://www.slideshare.net/aspyker/netflix-open-source-building-a-distributed-and-automated-open-source-program

NetflixOSS in 2016http://netflix.github.io

Around 200 projects currently on Github

Deep integration into SpringCloud Java framework

Hystrix circuit-breaker ported and adopted widely

Curator migrated from NetflixOSS to Apache Curator

New: Titus team working with AWS ECS – See CON313

Startups inspired by NetflixOSS:• DynomiteDB supporting and extending Dynomite Redis clustering

• Armory providing Spinnaker deployment pipeline as a service

• Gremlin implementing Chaos Monkey testing as a product

Many Benefits of Open Source Programs

Give back to the communities that you depend on

Attract and curate better developers

Improved code quality, docs, modularity and re-use

Pooled development and testing resources

Validate architectural decisions in public

Enable fast-followers and remain centered in an ecosystem

Build trust and basis for collaboration

Open Source is the Default

Leading customers have their own open source programs

Many AWS products have an open source component

Blackduck Annual Future Of Open Source survey“The survey reports that 78 percent of its respondents are now running their businesses with open

source software, and two-thirds are building software for their customers that’s based on open source

software. More significant, the percentage of respondents actually participating in open source

projects has increased from 50 percent to 64 percent, and 88 percent say they expect to contribute

to projects within the next three years.”

Baseline current status

Memberships

Code contributions

Projects hosted on GitHub

Open source based services

Open Source Memberships

Linux Foundation Silver Member

• Founding advisory member of Xen @ Linux Foundation

• Founding member of the Core Infrastructure Initiative

• Founding member of the Open Container Initiative

ToDo Group - Companies committed to open source

Alliance for Open Media – Video format

Open-MPI - consortium for high performance computing

Amazon & AWS contributions

Bug Fix and Feature Contributions

Linux & Drivers

Xen

Apache Tomcat

PostgreSQL

Docker

Boto

Apache Hadoop

Apache Hive

Apache Bigtop

Apache Oozie

Apache Drill

Apache Zeppelin

Apache Pig

Cloudera HUE

Apache Lucene

Apache Solr

Kuromoji

ElasticSearch

CBMC

MXNet

Moses

Apache Joshua

Highlights for Bug/Feature Contributions

Linux Contributions

• ENA – network driver

• Annapurna – ARM64 chipset support

• AWS recently hired Linux committer David Woodhouse

Boto – Python SDK – AWS hired creator Mitch Garnaat

Hadoop ecosystem projects – over 100 contributions

MXNet – Deep Learning Frameworkhttp://www.allthingsdistributed.com/2016/11/mxnet-default-framework-deep-learning-aws.html

Created at CMU

Major investments by AWS

Scalable, high performance

Supported alongside many other deep learning frameworks

Contributions to developer experience, documentation

Tools for visualization, development, migration

Supporting academic contributions with AWS credits

See MAC401 - Scalable Deep Learning Using MXNet

Repositories Owned by AWS & Amazonhttps://github.com/aws https://github.com/awslabs https://github.com/amznlabs https://blox.github.io/

Blox – Container orchestration for ECS – CON316

s2n – Secure replacement for openssl – NET405

Chalice – Python serverless microframework for AWS – DEV308

AWS-CLI – Built by Mitch Garnaat – DEV402

AWS-shell – Autocomplete based user interface for AWS-CLI

AWS SDKs

AWS Mobile SDKs

Cloud9 Ace – Cloud based interactive development editor

Cfncluster – Build and manage HPC clusters

ION – Amazon data serialization libraries

Blox – Container Orchestration Frameworkhttps://blox.github.io/

New governance model for Blox – built in the open

All development and roadmap in public

Provides a venue for collaboration with AWS customers

Intent to build an ecosystem

Contribute/port schedulers

See more details in CON316

Contributing to Bloxhttps://blox.github.io/

• Blox is licensed under Apache 2.0

• Open an issue or pull request

• Watch our roadmap on GitHub

• Check out our Gitter channel

Open Source Made Easy by AWS Services

Amazon EMR – Elastic Hadoop

Amazon ElastiCache - Memcached

Amazon RDS – MySQL

Amazon RDS – PostgreSQL

Amazon RDS – Scalable Aurora back-end for MySQL & PostgreSQL

AWS OpsWorks – Chef Automate Server

Amazon ECS – Docker Orchestration

Amazon CloudSearch

Amazon Elasticsearch

Let AWS Do The Undifferentiated Heavy Lifting

Amazon EMR updates & patches are frequent and automatic

Engagement and next steps

Program Team

Represents all of Amazon

Manage consumption of projects

Manage engineering contributions

Represent AWS (foundations etc.)

Work with legal on licensing

Assist with standalone projects

Operate GitHub accounts

Did everything to this point

AWS Teams Managing Open Source

New – Outbound Programs Team

Engage with AWS customers

Support academic credit program

Create content: blogs, talks etc.

Represent AWS (foundations etc.)

Present at industry events

Manage example code projects

Measure sentiment/engagement

Starting now!

AWS Cloud Credits

for Research

AWS has been supporting global research projects through the

Cloud Credits for Research program since 2009.

AWS believes that researchers are key drivers of technological innovation

AWS Cloud Credits for Research is aimed at helping researchers move their

research endeavors to the cloud so they can innovate more quickly and at

lower cost

AWS has been supporting global research projects through the

Cloud Credits for Research program since 2009.

We support researchers who seek to:

Build reusable tools to facilitate their future research and the research of

their community

Perform pilot, proof of concept, or benchmark tests evaluating the efficacy

of moving research workloads or open data sets to the cloud

Train a broader community on the usage of cloud for research workloads via

workshops or tutorials

AWS Cloud Credits for Research Recipientshttps://aws.amazon.com/research-credits/2016-recipients/

Categories

Proof of Concept

Development of Open Source Software

Development of a Cloud Service

Open Dataset Development

Benchmark for Comparison

Workshop or Tutorial

AWS Credits for Open Source Projects

Awarded on a case-by-case basis:

Berkeley AMP Lab – Founding Sponsor

Apache Mesos, Spark and other open source tools

Open-MPI Consortium support

PostgreSQL project was given credits to support testing

How can we help?

Adrian Cockcroft

@adrianco

Experience:

Startup Open Source Ecosystems

NetflixOSS

New - Outbound TeamContact us via AWS account management, or in person at #reinvent

Zaheda Bhorat

@zahedab

Experience:

UK Gov Open Document Format

Salesforce Open Source Program

Google Summer of Code

Sun OpenOffice & Netbeans

First steps, we’re listening…

Thank you!

Remember to complete

your evaluations!

Related Sessions

DCS201 - Open-Source Resources and Certifications

BDM401 - Deep Dive: Amazon EMR Best Practices & Design Patterns

BAP305 - Zero to Google Chrome in 60 Minutes: Lightweight and Inexpensive Client Devices for Amazon WorkSpaces

NET405 - Amazon s2n: Cryptography and Open Source at AWS

MAC401 - Scalable Deep Learning Using MXNet

CON316 - State of the Union: Containers

CON313 - Netflix: Container Scheduling, Execution, and Integration with AWS

DAT305 - Deep Dive on Amazon Relational Database Service

DEV308 - Chalice: a Serverless Microframework for Python

DEV402 - The Effective AWS-CLI User