Download - AWS re:Invent 2016: Deploying and Managing .NET Pipelines and Microsoft Workloads (DEV303)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
November 30, 2016
Deploying and Managing .NET
Pipelines and Microsoft Workloads
DEV303
Santiago Cardenas, Partner Solutions Architect – AWS Quick Start
Theo Carpenter, Systems Manager – Woot!
Tom Fuller, Sr. Solutions Architect – Strategic Accounts
What to expect from the session
We will highlight best practices and explain the reusable artifacts
produced by the AWS Quick Start team for Windows workloads.
We will describe and demonstrate a Windows-specific
CI/CD pipeline for .NET in AWS.
We will hear from Woot! and learn how they built an
efficient DevOps process for their Windows workloads.
Disclaimer
• Best practices can sometimes be
subjective or sensitive to variables
• “Easy” can be relative … what we
are assuming for this session is
working knowledge of AWS
automation techniques
• Innovations are not frozen in time!
State of Windows development on AWS
• AWS SDK for .NET
• AWS Toolkit for Visual Studio
• AWS Tools for Windows PowerShell
• AWS higher-level services
• AWS CodeDeploy
• AWS Elastic Beanstalk
• RDS for SQL Server
• Amazon EC2 Simple Systems
Manager
AWS Quick Starts
AWS Quick Starts
• Automated reference architecture deployments on AWS
• Community Quick Starts
• Distributed team across 3 sites covering the globe
• Part of the AWS Partner Program
• Visit us: https://aws.amazon.com/quickstart
Windows-based AWS Quick Starts
• Active Directory Domain Services
• Web Application Proxy + AD FS
• SQL Server
• SharePoint Server
• Exchange Server
• Lync/Skype For Business Server
• Remote Desktop Gateway
• CI/CD for Windows
Recurring themes
ScriptingOrchestration Source control
Gives developers and systems administrators
an easy way to create and manage a collection
of related AWS resources
Understands dependencies and supports
rollbacks and versioning
Allows for reusable component design
strategies
Supports a portable JSON or YAML format
Authoring tools continue to improve
AWS CloudFormation
Sourcing files and relative paths
"C:\\cfn\\scripts\\Install-ADFS.ps1": {
"source": {"Fn::Join": [
"/",[
{"Fn::FindInMap": [
"AWSInfoRegionMap",
{"Ref": "AWS::Region"},
"QuickStartS3URL"]},
{"Ref": "QSS3BucketName"},
{"Ref": "QSS3KeyPrefix"},
"scripts/Install-ADFS.ps1"
]]}}
Nested CloudFormation stacks
"Resources": {
"VPCStack": {
"Type": "AWS::CloudFormation::Stack",
"Properties": { "TemplateURL": {...},
"Parameters": {...}}},
"ADStack": {
"Type": "AWS::CloudFormation::Stack",
"Properties": { "TemplateURL": {...},
"Parameters": {...}}},
...
}
The last mile of your automation comes down
to this
In the Windows world, this is almost always
PowerShell
Some Windows challenges are always there
(reboots, service restarts, security models)
The techniques you standardize on here can
make or break your approach
Bootstrapping
PowerShell module
Initialize
New-AWSQuickStartWaitHandle
New-AWSQuickStartResourceSignal
Finalize
Write-AWSQuickStartException
Write-AWSQuickStartStatus
Reusable PowerShell module and scripts
PowerShell scripts
Create-
ADServiceAccount
Create-Share
Disable-AutoLogon
DisableCredSsp
Download-File
Enable-AutoLogon
EnableCredSsp
Install-Module
Install-WMF5
Invoke-
ADReplication
Join-Domain
Mount-DiskImage
New-
LocalAdministrator
Rename-Computer
Set-Dns
Unzip-Archive
https://github.com/aws-quickstart/quickstart-microsoft-utilities
PowerShell scripts
param(
[Parameter(Mandatory=$true)]
[string]$Parameter1
)
try {
$ErrorActionPreference = "Stop"
#DO STUFF
}
catch {
$_ | Write-AWSQuickStartException
}
Configuration management
• Making your scripts idempotent ensures that they can be
rerun in place as needed
• PowerShell DSC or AWS Partner solutions can help you
PowerShell DSC Ansible Chef Puppet
Invoking PowerShell scripts
"commands": {
...
"e-install-windows-failover-clustering": {
"command": "powershell.exe -Command \"C:\\cfn\\scripts\\Install-WindowsFailoverClustering.ps1\"",
"waitAfterCompletion": "0"
},
...
}
Handling Windows reboots
"b-rename-computer": {
"command": {"Fn::Join": ["",[
"powershell.exe ",
"–Command
"\"C:\\cfn\\scripts\\Rename-Computer.ps1 ",
"-Restart -NewName '",
{"Ref": "WSFCNode1NetBIOSName"},
"'\""
]]},
"waitAfterCompletion": "forever"
}
CloudFormation signaling
"commands": {
"a-reboot": {
"command": "powershell.exe -Command \"Restart-Computer -Force\"",
"waitAfterCompletion": "forever"
},
"b-signal-success": {
"command": "powershell.exe -Command \"Write-AWSQuickStartStatus\""
}
}
Code reuse
Continuous Integration/Continuous Delivery
• Monitors GitHub branches
• Tests the Quick Starts in all supported regions
• Updates AMIs referenced in templates
• Merges code to branches
• Copies release artifacts to S3
Quick Start: CI/CD for
Windows
Release processes levels
Source Build Test Production
Continuous integration
Continuous delivery
Continuous deployment
Automated actions and transitions, from check-in to production
Development benefits:
• Faster
• Safer
• Simplification & standardization
• Visualization of the process
Pipelines
Continuous delivery service for fast and
reliable application updates
Model and visualize your software release
process
Builds, tests, and deploys your code every time
there is a code change
Integrates with third-party tools and AWS
AWS CodePipeline
Extend AWS CodePipeline using custom actions
Update tickets Provision resources
Update dashboards
Mobile testing
Send notifications Security scan
Popular open source tool for delivering CI/CD
Huge ecosystem of plugins, including MSBuild
and AWS CodePipeline
Capable of robust configurations like
master/slave and external data store and user
repository (LDAP)
Automation-friendly
(MSBuild) No longer bundled with Visual
Studio and has an open source license
Jenkins + MSBuild
Automates code deployments to any instance
Handles the complexity of updating your
applications
Avoid downtime during application deployment
Deploy to Amazon EC2 or on-premises
servers, in any language and on any operating
system
Integrates with third-party tools and AWS
AWS CodeDeploy
.NET CI pipeline in AWS
AWS CodeDeploy
SourceBucket
Push/Pull
Poll
Push
BuildServer
TestServer(s)
HTTPS ELB
endpoint
HTTPS ELB
endpoint
DEMO
What is Woot!?
What does DevOps mean to Woot?
“DevOps is the practice of operations and development
engineers participating together in the entire service
lifecycle, from design through the development
process to production support.” – Ernest Mueller
theagileadmin.com
How does this apply to Woot?
• Full cycle management (beyond launch)
• Configuration drift and infrastructure as code
Woot’s DevOps timeline2010
Amazon acquisition,
Physical servers in a data center
(~30 servers)
2011
Site rebuild
on AWS,
First site built in AWS
2012
New site launches,
Automated deployment
(~220 EC2 instances)
2013
Automation services,
Scaling and reliability,
90 actions
(~800 Instances)
2014
AWS Managed Services,
GP2 EBS,
HA SQL DB,
(~1000 instances)
2015
ELK stack,
Directory Service AD Connector
KMS
(~900 instances)
2016
AWS CodeDeploy,
Directory Service,
SSM,
(~800 instances)
At acquisition - 2010
• Physical servers
• Manual install
• Single project
• Manual deployment
• Single database server
• Performance and monitoring issues
Post-acquisition - 2011
• First project built in AWS
• Migrate existing infrastructure to AWS
• Site redesign requirements
• Automated deployment
• Database redundancy
• Scalable
• Secure
Infrastructure deployment workflow
New feature
Code
review
BuildDeploy
Execution
New site and deployment - 2012
• PowerShell 2.0/3.0
• Custom AMI
• Base actions:
• Install roles (SQL,IIS,Daemon)
• Rename and join domain
• Install patches
• Notify when complete
• 45 minutes from launch to ready
Custom script actions example
#turn off Ec2SetComputerName
write-log "turning off Ec2SetComputerName"
C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy unrestricted -NoLogo.\DisableEc2SetComputerName.ps1
write-log "Ec2SetComputerName Completed. Check associated log for any errors.“
#Download and install hot fix kb2634328 if it isn't installed.
$iiselbinstaller =“.\resources\ELB_IIS_hotfixWindows6.1-KB2634328-x64.msu"
$iiselbhotfix = Check-IsPatchInstalled -id "KB2634328"
Scaling, security, automation- 2013 to 2014
• PowerShell 4.0
• Custom deployment scripts reach 90 actions
• Kerberos-based security service
• Metadata service for configuration
• 35 minutes from launch to ready
• GP2 SSD volumes
• SQL 2014 AlwaysOn
Example event log code
#Get script name for Source
$scriptname = ($MyInvocation.MyCommand.Name).Replace(".ps1","")
$sourcename = “Woot.” + $scriptname
$logname = “Application"
#check if source exists if log already found
Write-Host "Checking if existing event source is registered"
$logSourceExists = [System.Diagnostics.EventLog]::SourceExists($sourcename)
if (! $logSourceExists)
{
Write-Output "Creating missing event source"
New-EventLog -LogName $logname -Source $sourcename
}
else
{
write-output "Already registered, woot on"
}
}
New technologies, efficiency - 2015 to 2016
• AWS AD Connector and full AD
• Custom deployment scripts reach 109 actions
• Custom ELK stack; Amazon Elasticsearch Service
• AWS KMS – Secure all the things!
• SSM
• AWS CodeDeploy for infrastructure code
• 25 minutes from launch to ready
Custom script action highlights
1. Domain Join
2. Servername
3. IIS
4. NXLog
5. Custom event
log
6. Gzip
7. Windows
updates
8. 8 hotfixes
9. .NET 4.5
10. PowerShell 5.0
11. Custom app
deployment
“Troop”
12. SSL
13. Certificates
14. Anti-virus
15. IIS advanced
logging
16. Join AD group
17. AD service
account
18. Deploy and
configure apps
19. URL rewrite
20. OS security
SSM domain join and name change
<powershell>
$region = $(New-ObjectNet.WebClient).DownloadString("http://169.254.169.254/latest/meta-data/placement/availability-zone");
Set-DefaultAWSRegion -region $region.trimend('[a-z]');
$instanceId = $(New-ObjectNet.WebClient).DownloadString("http://169.254.169.254/latest/meta-data/instance-id");
$nameTag = Get-EC2Tag | ` Where-Object {$_.ResourceId -eq $instanceId -and $_.Key -eq 'Name'};
$computername = $nameTag.Value.Split(".");
Rename-Computer -NewName $computername[0];
New-SSMAssociation -InstanceId $instanceId -Name DomainJoinWithOUConfig;
Start-Sleep -s 5;
</powershell>
What’s next?
• AWS base image
• PowerShell DSC
• AWS CodeDeploy for application code
• AWS CodePipeline
• Serverless services (when it makes sense)
• Immutable infrastructure
• 10 minutes or less goal for launch time
Thank you!
Remember to complete
your evaluations!
Related Sessions