aws re:invent 2016: deploying and managing .net pipelines and microsoft workloads (dev303)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

November 30, 2016

Deploying and Managing .NET

Pipelines and Microsoft Workloads

DEV303

Santiago Cardenas, Partner Solutions Architect – AWS Quick Start

Theo Carpenter, Systems Manager – Woot!

Tom Fuller, Sr. Solutions Architect – Strategic Accounts

What to expect from the session

We will highlight best practices and explain the reusable artifacts

produced by the AWS Quick Start team for Windows workloads.

We will describe and demonstrate a Windows-specific

CI/CD pipeline for .NET in AWS.

We will hear from Woot! and learn how they built an

efficient DevOps process for their Windows workloads.

Disclaimer

• Best practices can sometimes be

subjective or sensitive to variables

• “Easy” can be relative … what we

are assuming for this session is

working knowledge of AWS

automation techniques

• Innovations are not frozen in time!

State of Windows development on AWS

• AWS SDK for .NET

• AWS Toolkit for Visual Studio

• AWS Tools for Windows PowerShell

• AWS higher-level services

• AWS CodeDeploy

• AWS Elastic Beanstalk

• RDS for SQL Server

• Amazon EC2 Simple Systems

Manager

AWS Quick Starts

AWS Quick Starts

• Automated reference architecture deployments on AWS

• Community Quick Starts

• Distributed team across 3 sites covering the globe

• Part of the AWS Partner Program

• Visit us: https://aws.amazon.com/quickstart

Windows-based AWS Quick Starts

• Active Directory Domain Services

• Web Application Proxy + AD FS

• SQL Server

• SharePoint Server

• Exchange Server

• Lync/Skype For Business Server

• Remote Desktop Gateway

• CI/CD for Windows

Recurring themes

ScriptingOrchestration Source control

Gives developers and systems administrators

an easy way to create and manage a collection

of related AWS resources

Understands dependencies and supports

rollbacks and versioning

Allows for reusable component design

strategies

Supports a portable JSON or YAML format

Authoring tools continue to improve

AWS CloudFormation

Sourcing files and relative paths

"C:\\cfn\\scripts\\Install-ADFS.ps1": {

"source": {"Fn::Join": [

"/",[

{"Fn::FindInMap": [

"AWSInfoRegionMap",

{"Ref": "AWS::Region"},

"QuickStartS3URL"]},

{"Ref": "QSS3BucketName"},

{"Ref": "QSS3KeyPrefix"},

"scripts/Install-ADFS.ps1"

]]}}

Nested CloudFormation stacks

"Resources": {

"VPCStack": {

"Type": "AWS::CloudFormation::Stack",

"Properties": { "TemplateURL": {...},

"Parameters": {...}}},

"ADStack": {

"Type": "AWS::CloudFormation::Stack",

"Properties": { "TemplateURL": {...},

"Parameters": {...}}},

...

}

The last mile of your automation comes down

to this

In the Windows world, this is almost always

PowerShell

Some Windows challenges are always there

(reboots, service restarts, security models)

The techniques you standardize on here can

make or break your approach

Bootstrapping

PowerShell module

Initialize

New-AWSQuickStartWaitHandle

New-AWSQuickStartResourceSignal

Finalize

Write-AWSQuickStartException

Write-AWSQuickStartStatus

Reusable PowerShell module and scripts

PowerShell scripts

Create-

ADServiceAccount

Create-Share

Disable-AutoLogon

DisableCredSsp

Download-File

Enable-AutoLogon

EnableCredSsp

Install-Module

Install-WMF5

Invoke-

ADReplication

Join-Domain

Mount-DiskImage

New-

LocalAdministrator

Rename-Computer

Set-Dns

Unzip-Archive

https://github.com/aws-quickstart/quickstart-microsoft-utilities

PowerShell scripts

param(

[Parameter(Mandatory=$true)]

[string]$Parameter1

)

try {

$ErrorActionPreference = "Stop"

#DO STUFF

}

catch {

$_ | Write-AWSQuickStartException

}

Configuration management

• Making your scripts idempotent ensures that they can be

rerun in place as needed

• PowerShell DSC or AWS Partner solutions can help you

PowerShell DSC Ansible Chef Puppet

Invoking PowerShell scripts

"commands": {

...

"e-install-windows-failover-clustering": {

"command": "powershell.exe -Command \"C:\\cfn\\scripts\\Install-WindowsFailoverClustering.ps1\"",

"waitAfterCompletion": "0"

},

...

}

Handling Windows reboots

"b-rename-computer": {

"command": {"Fn::Join": ["",[

"powershell.exe ",

"–Command

"\"C:\\cfn\\scripts\\Rename-Computer.ps1 ",

"-Restart -NewName '",

{"Ref": "WSFCNode1NetBIOSName"},

"'\""

]]},

"waitAfterCompletion": "forever"

}

CloudFormation signaling

"commands": {

"a-reboot": {

"command": "powershell.exe -Command \"Restart-Computer -Force\"",

"waitAfterCompletion": "forever"

},

"b-signal-success": {

"command": "powershell.exe -Command \"Write-AWSQuickStartStatus\""

}

}

Code reuse

Continuous Integration/Continuous Delivery

• Monitors GitHub branches

• Tests the Quick Starts in all supported regions

• Updates AMIs referenced in templates

• Merges code to branches

• Copies release artifacts to S3

Quick Start: CI/CD for

Windows

Release processes levels

Source Build Test Production

Continuous integration

Continuous delivery

Continuous deployment

Automated actions and transitions, from check-in to production

Development benefits:

• Faster

• Safer

• Simplification & standardization

• Visualization of the process

Pipelines

Continuous delivery service for fast and

reliable application updates

Model and visualize your software release

process

Builds, tests, and deploys your code every time

there is a code change

Integrates with third-party tools and AWS

AWS CodePipeline

Extend AWS CodePipeline using custom actions

Update tickets Provision resources

Update dashboards

Mobile testing

Send notifications Security scan

Popular open source tool for delivering CI/CD

Huge ecosystem of plugins, including MSBuild

and AWS CodePipeline

Capable of robust configurations like

master/slave and external data store and user

repository (LDAP)

Automation-friendly

(MSBuild) No longer bundled with Visual

Studio and has an open source license

Jenkins + MSBuild

Automates code deployments to any instance

Handles the complexity of updating your

applications

Avoid downtime during application deployment

Deploy to Amazon EC2 or on-premises

servers, in any language and on any operating

system

Integrates with third-party tools and AWS

AWS CodeDeploy

.NET CI pipeline in AWS

AWS CodeDeploy

SourceBucket

Push/Pull

Poll

Push

BuildServer

TestServer(s)

HTTPS ELB

endpoint

HTTPS ELB

endpoint

DEMO

What is Woot!?

What does DevOps mean to Woot?

“DevOps is the practice of operations and development

engineers participating together in the entire service

lifecycle, from design through the development

process to production support.” – Ernest Mueller

theagileadmin.com

How does this apply to Woot?

• Full cycle management (beyond launch)

• Configuration drift and infrastructure as code

Woot’s DevOps timeline2010

Amazon acquisition,

Physical servers in a data center

(~30 servers)

2011

Site rebuild

on AWS,

First site built in AWS

2012

New site launches,

Automated deployment

(~220 EC2 instances)

2013

Automation services,

Scaling and reliability,

90 actions

(~800 Instances)

2014

AWS Managed Services,

GP2 EBS,

HA SQL DB,

(~1000 instances)

2015

ELK stack,

Directory Service AD Connector

KMS

(~900 instances)

2016

AWS CodeDeploy,

Directory Service,

SSM,

(~800 instances)

At acquisition - 2010

• Physical servers

• Manual install

• Single project

• Manual deployment

• Single database server

• Performance and monitoring issues

Post-acquisition - 2011

• First project built in AWS

• Migrate existing infrastructure to AWS

• Site redesign requirements

• Automated deployment

• Database redundancy

• Scalable

• Secure

Infrastructure deployment workflow

New feature

Code

review

BuildDeploy

Execution

New site and deployment - 2012

• PowerShell 2.0/3.0

• Custom AMI

• Base actions:

• Install roles (SQL,IIS,Daemon)

• Rename and join domain

• Install patches

• Notify when complete

• 45 minutes from launch to ready

Custom script actions example

#turn off Ec2SetComputerName

write-log "turning off Ec2SetComputerName"

C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy unrestricted -NoLogo.\DisableEc2SetComputerName.ps1

write-log "Ec2SetComputerName Completed. Check associated log for any errors.“

#Download and install hot fix kb2634328 if it isn't installed.

$iiselbinstaller =“.\resources\ELB_IIS_hotfixWindows6.1-KB2634328-x64.msu"

$iiselbhotfix = Check-IsPatchInstalled -id "KB2634328"

Scaling, security, automation- 2013 to 2014

• PowerShell 4.0

• Custom deployment scripts reach 90 actions

• Kerberos-based security service

• Metadata service for configuration

• 35 minutes from launch to ready

• GP2 SSD volumes

• SQL 2014 AlwaysOn

Example event log code

#Get script name for Source

$scriptname = ($MyInvocation.MyCommand.Name).Replace(".ps1","")

$sourcename = “Woot.” + $scriptname

$logname = “Application"

#check if source exists if log already found

Write-Host "Checking if existing event source is registered"

$logSourceExists = [System.Diagnostics.EventLog]::SourceExists($sourcename)

if (! $logSourceExists)

{

Write-Output "Creating missing event source"

New-EventLog -LogName $logname -Source $sourcename

}

else

{

write-output "Already registered, woot on"

}

}

New technologies, efficiency - 2015 to 2016

• AWS AD Connector and full AD

• Custom deployment scripts reach 109 actions

• Custom ELK stack; Amazon Elasticsearch Service

• AWS KMS – Secure all the things!

• SSM

• AWS CodeDeploy for infrastructure code

• 25 minutes from launch to ready

Custom script action highlights

1. Domain Join

2. Servername

3. IIS

4. NXLog

5. Custom event

log

6. Gzip

7. Windows

updates

8. 8 hotfixes

9. .NET 4.5

10. PowerShell 5.0

11. Custom app

deployment

“Troop”

12. SSL

13. Certificates

14. Anti-virus

15. IIS advanced

logging

16. Join AD group

17. AD service

account

18. Deploy and

configure apps

19. URL rewrite

20. OS security

SSM domain join and name change

<powershell>

$region = $(New-ObjectNet.WebClient).DownloadString("http://169.254.169.254/latest/meta-data/placement/availability-zone");

Set-DefaultAWSRegion -region $region.trimend('[a-z]');

$instanceId = $(New-ObjectNet.WebClient).DownloadString("http://169.254.169.254/latest/meta-data/instance-id");

$nameTag = Get-EC2Tag | ` Where-Object {$_.ResourceId -eq $instanceId -and $_.Key -eq 'Name'};

$computername = $nameTag.Value.Split(".");

Rename-Computer -NewName $computername[0];

New-SSMAssociation -InstanceId $instanceId -Name DomainJoinWithOUConfig;

Start-Sleep -s 5;

</powershell>

What’s next?

• AWS base image

• PowerShell DSC

• AWS CodeDeploy for application code

• AWS CodePipeline

• Serverless services (when it makes sense)

• Immutable infrastructure

• 10 minutes or less goal for launch time

Thank you!

Remember to complete

your evaluations!

Related Sessions