master chef class: learn how to quickly cook delightful cq/aem infrastructures

Master Chef class

https://github.com/francoisledroff/connectcon-chef-repo

http://www.slideshare.net/francoisledroff/master-chef-class-learn-how-to-quickly-cook-delightful-cqaem-infrastructures

Francois Le Droff – Nicolas Peltier

Let’s cook delightful AEM infrastructures

/master-chef-class-learn-how-to-quickly-cook-delightful-cqaem-infrastructures

@francoisledroff

@npeltier

Agenda

•!What •!How •!Why

What ?

“ An automation platform that transforms

infrastructure into code ”

Infrastructure ? noun \ˈin-frə-ˌstrək-chər

[1] A Collection of –!Resources:

•! Network nodes •! File systems, files, folders, symbolic links, disk mounts •! Users, groups •! Packages, software •! Configurations

–!Acting in concert –!To offer a service

–! [1] Introduction to Chef

Infrastructure as Code?

•! Does not only replace your shell scripts •! Does allow:

–! to build your infra from a set of bare metal servers (called Nodes) Even on heterogeneous OS and Architecture

–! to abstract the convergence of your infrastructure (your Nodes) through code

–! to make this (Nodes) convergence idempotent

State A --> State B (ex: install)

Chef Client finished, 147/153 resources updated

State B --> State B (ex: check)

State C --> State B (ex: back to normal)

•! Created in 2009, Edited by Opscode/Chef

•! Apache License

•! On-top-of/in Ruby

•! Very active community

•! http://community.opscode.com

•! #learnChef

•! https://learnchef.opscode.com/

Chef ?

p://community.opscode.com

What does Chef code look like ?

•!Recipes •!Attributes •!Resources

•!Cookbooks

Recipes

Fundamental configuration element •! Authored in Ruby •! Queries , defines attributes •! Manipulates Resources •! Manipulates LW Resources •! Leverages Libraries & Templates

•! Stored in Cookbooks –! Default Attributes –! Recipes –! LW Resource Providers –! Libraries –! Templates

recipes

cookbooks

•! aem-cookbook/recipes/default.rb case node['platform_family']

when 'rhel' log 'this platform family is supported' do level :info

end else log 'this platform family is not supported' do

level :warn

end end

•! aem-cookbook/attributes/node.rb default['aem']['mode'] = 'publish'

default['aem']['port'] = '4503'

default['aem']['url'] = 'http://localhost:'+ node['aem']['port']

default['aem']['mode'] = 'publish'

default['aem']['port'] = '4503'

default['aem']['url'] = 'http://localhost:'+ node['aemaem']['port']

aem-cookbook/recipes/default.rbcase node['platform_family']

when 'rhel' log 'this platform family is supported' do level :info

end else log 'this platform family is not supported' do

level :warn

end end

Attributes attributes

Recipes : Resources & LW Resources

•! aem-cookbook/recipes/default.rb

remote_file node['aem']['quickstart_jar'] do source node['aem']['repo_url']

owner node['aem']['user']

group node['aem']['group']

checksum '3043859473'

action :create_if_missing

notifies :restart, 'service[aem]', :delayed

end or

artifact_file node['aem']['quickstart_jar'] do location 'com.day.cq:cq5:jar:5.5.0.20120220'

nexus_configuration nexus_configuration_object

artifact_file node['aem']['quickstart_jar'] do location 'com.day.cq:cq5:jar:5.5.0.20120220'

nexus_configuration nexus_configuration_object

remote_file node['aem']['quickstart_jar'] do source node['aem']['repo_url']

checksum '3043859473'

action :create_if_missing

recipes

Recipes : AEM LW Resources •! tacit-aem-cookbook/recipes/author.rb

node[:aem][:author][:deploy_pkgs].each do |pkg| aem_package pkg[:name] do version pkg[:version]

aem_instance ‘author’

package_url pkg[:url]

update pkg[:update]

user node[:aem][:author][:admin_user]

password node[:aem][:author][:admin_password]

port node[:aem][:author][:port]

group_id pkg[:group_id]

recursive pkg[:recursive]

properties_file pkg[:properties_file]

version_pattern pkg[:version_pattern]

action pkg[:action]

end end

node[:aem][:author][:deploy_pkgs].each do |pkg| aem_package pkg[:name] do version pkg[:version]

aem_instance ‘author’

package_url pkg[:url]

update pkg[:update]

user node[:aem][:author][:admin_user]

password node[:aem][:author][:admin_password]

port node[:aem][:author][:port]

group_id pkg[:group_id]

recursive pkg[:recursive]

properties_file pkg[:properties_file]

version_pattern pkg[:version_pattern]

action pkg[:action]

end end

recipes

–! aem-cookbook/recipes/start.rb

ruby_block 'block_until_cq_operational' do block do Chef::Log.info 'Waiting until CQ is listening on port '+node['aem']['port']

until CQHelper.service_listening?(node['aem']['port']) sleep 1

Chef::Log.info('.')

Chef::Log.info 'Waiting until the CQ default page is responding'

test_url = URI.parse(node['aem']['url'])

until CQHelper.endpoint_responding?(test_url) sleep 1

Chef::Log.info('.')

end action :nothing

Recipes: Libraries recipes

ruby_block 'block_until_cq_operational' do block do Chef::Log.info 'Waiting until CQ is listening on port '+node['aem']['port']

until CQHelper.service_listening?(node['aem']['port']) sleep 1

Chef::Log.info('.')

Chef::Log.info 'Waiting until the CQ default page isresponding'

test_url = URI.parse(node['aem']['url'])

until CQHelper.endpoint_responding?(test_url) sleep 1

Chef::Log.info('.')

end action :nothing

Cookbook patterns [1]

•! Library cookbooks: –! https://github.com/RiotGames/artifact-cookbook –! https://github.com/francoisledroff/chef-vault-util

•! Application cookbooks –! https://github.com/tacitknowledge/aem-cookbook –! https://github.com/francoisledroff/aem-cookbook –! https://github.com/onehealth-cookbooks/apache2 –! https://github.com/socrata-cookbooks/java –! https://github.com/hw-cookbooks/haproxy –! https://github.com/stevendanna/logrotate –! https://github.com/opscode-cookbooks/chef-splunk

•! Organization specific

–! Wrapper cookbooks –! Base cookbooks –! Environment cookbooks

[1] http://blog.vialstudios.com/the-environment-cookbook-pattern/

Library cookbooks: ps://github.com/RiotGames/artifact-cookbookps://github.com/francoisledro

cookbooks

I got it

Wait a quick overview

Chef-server nodes

workstation

Chef-server nodes

roles attributes

recipes

cookbooks

workstation

Chef-server nodes

RSA key Auth

roles attributes

recipes

Chef-DK

data bags

cookbooks RSA Keys

workstation

Chef-server nodes

RSA key Auth

roles attributes

recipes

Chef-DK

data bags

cookbooks RSA Keys

Search API

cookbooks org

data bags

attributes

recipes

node object run-list

roles Web UI

versions

workstation

Chef-server nodes

RSA key Auth

roles attributes

recipes

Chef-DK

data bags

cookbooks RSA Keys

Search API

cookbooks org

data bags

attributes

recipes

roles Web UI

versions

chef-clients

RSA Key

Search API knife

workstation

roles attributes

cookbooks

recipes

Chef-DK

github

opscode rubygem

data bags

attributes

data bags

cookbooks recipes

Chef-server nodes

RSA key Auth

yymaven redhat

RSA Keys

roles Web UI

versions

RSA key Auth

chef-clients

RSA Key

Use case 0 : one AEM Author

author

Use case 0 : Chef Automation

–! Install the jdk –! Download the jar –! Install it as a service

–! A few aem cookbooks on github

•! https://github.com/francoisledroff/aem-cookbook •! https://github.com/tacitknowledge/aem-cookbook •! https://github.com/manosriglis/chef-aem •! https://github.com/QVCItalia/chef-aem

OSS from opscode and elsewhere Chef, Ruby, rvm, bundler

workstation

Chef-DK

opscode

$ curl –L https://www.opscode.com/chef/install.sh | sudo bash $ curl -sSL https://get.rvm.io | bash -s stable –-ruby=1.9.3 $ gem install bundler http://www.getchef.com/downloads/chef-dk

workstation

Git ssh

A Chef repo in Git Every Chef automated infra needs a Chef Repository

github

$ git clone https://github.com/opscode/chef-repo.git

Made available for you at https://github.com/francoisledroff/connectcon-chef-repo

Get Few Machines/Nodes to deploy your code/infra to

workstation

~/workspace/github/connectcon-chef-repo on ! master! $ vagrant plugin install vagrant-berkshelf --plugin-version 2.0.1 $ cat Vagrantfile Vagrant.configure("2") do |config| config.vm.box = "CentOS-6.4-x86_64" config.vm.hostname = "connectcon-francois" config.vm.box_url = "http://developer.nrel.gov/downloads/vagrant-boxes/CentOS-6.4-x86_64-v20130427.box" config.berkshelf.enabled = true End $ vagrant up $ vagrant ssh Welcome to your Vagrant-built virtual machine. [vagrant@connectcon-francois ~]$

A Chef Server comes in 3 flavors

•! On premise OS Chef Server

•! On premise Enterprise Chef

•! Hosted Enterprise Chef server

•! Local Alternatives: •! Chef-zero •! Chef-solo

Chef-server

A Chef Org top-level entity for role-based access

Chef-server

https (ldap) Auth

Web UI

ps (ldap) Auth ps (ldap) Auth

https://chef.corp.adobe.com/organizations

workstation

A few private keys to associate your new chef repo with your chef server user and org

~/workspace/github/connectcon-chef-repo on ! master! $ ll .chef total 24 connectcon-validator.pem knife.rb ledroff.pem

org Chef-server Web UI

RSA Keys

Bootstrap your nodes

workstation Chef-server

RSA Keys

chef-clients

RSA Key

~/workspace/github/connectcon-chef-repo on ! master! $ knife bootstrap <your-node-fqdn> --sudo -x <your-sudoer>

Chef-server nodes

RSA key Auth

Search API attributes node object run-list chef-clients

Node Objects

Start Coding: Manage your Ruby dependencies

•! Use Bundler for RubyGem

~/workspace/github/connectcon-chef-repo on ! master! $ cat Gemfile source 'https://rubygems.org’ gem 'chef', '~> 11.10.0’ gem 'berkshelf', '~> 3.1.3‘ $ bundle install

Start Coding: Manage your Chef dependencies

•! A few aem cookbooks on github •! https://github.com/francoisledroff/aem-cookbook •! https://github.com/tacitknowledge/aem-cookbook

•! Use Berkshelf

$ cat Berksfile source "https://api.berkshelf.com” cookbook 'ntp', '~> 1.5.4' cookbook 'chef-client', '~> 3.2.0’ cookbook 'artifact', git: 'https://github.com/francoisledroff/artifact-cookbook.git', tag: '1.11.4’ cookbook 'aem', git: 'https://github.com/francoisledroff/aem-cookbook

Upload your cookbooks chef-client and ntp declared in your chef server org

workstation org

Chef-DK

Chef-server

RSA Keys cookbooks

~/workspace/github/connectcon-chef-repo on ! master! $ berks install $ berks upload

cookbooks

Run-list

run_list "recipe[chef-client::default]", "recipe[chef-client::delete_validation]", "recipe[ntp]", "recipe[openssl::upgrade]"

name ”base” Description ”connecton base server role” run_list "recipe[chef-client::default]",

"recipe[chef-client::delete_validation]", "recipe[ntp]", "recipe[openssl::upgrade]"

Typical Roles connectcon-chef-repo/roles/

name ”publish” Description ”connecton aem publish server role” run_list ”role[base]”,”recipe[aem::publish]”

Mongo Author

Publish

Dispatcher

name "lb_dispatcher" description "connectcon roles for lb" run_list ”role[base]”,”recipe[haproxy::app_lb]” override_attributes('haproxy' => {'app_server_role' => ’dispatcher’})

name ”base” Description ”connecton base server role” run_list "recipe[chef-client::default]",

"recipe[chef-client::delete_validation]", "recipe[ntp]", "recipe[openssl::upgrade]"

Apply Run-List to Nodes attributes

https://chef.corp.adobe.com/organizations

~/workspace/github/connectcon-chef-repo on ! master! $ knife node list $ knife node edit one.node.fqdn

Chef-client run [root@ot1slu010 ~]# sudo chef-client Starting Chef Client, version 11.12.8 resolving cookbooks for run list: ["chef-client::delete_validation", "ntp", “aem::author", “aem::start”]

* remote_file[/apps/publish/aem-author-4502.jar] action create_if_missing (up to date)

* file[/apps/publish/license.properties] action create (up to date) * cookbook_file[/apps/author/serverctl] action create (up to date) * execute[java] action run (skipped due to not_if) * service[aem] action enable (up to date)

Recipe: aaem::start * ruby_block[block_until_cq_operational] action nothing (skipped due to action :nothing)

* log[ensure_cq_is_running] action write Recipe: aaem::default * service[aem] action start - start service service[aem]

Recipe: aaem::start * ruby_block[block_until_cq_operational] action create - execute the ruby block block_until_cq_operational

Running handlers: Running handlers complete Chef Client finished, 2/27 resources updated in 9.770664 seconds

So we have an Author : Use Case 0

Author Author

{ "name": ”<author-connectcon-fqdn>", "chef_environment": "dev”, "run_list": [ "role[author]" ] }

$ knife node edit author-connectcon-fqdn

Let’s add a publish : Use Case 1

{ "name": ”<publish-connectcon-fqdn>", "chef_environment": "dev”, "run_list": [ "role[publish]" ] }

$ knife node edit publish-connectcon-fqdn

Publish

Author

And there is magic: Chef search API

•! tacit-aem-cookbook/providers/replicator.rb

https://github.com/tacitknowledge/aem-cookbook

hosts = [] search(:node, %Q(role:"#{role}"

AND aem_cluster_name:"#{cluster_name}")) do |n| log "Found host: #{n[:fqdn]}" hosts << { :ipaddress => n[:ipaddress],

… :name => n[:fqdn] } end … hosts.each do |h| … end

Replication configuration happens

#{role}:”publish” #{cluster_name}:”dev”

Author

Publish

Aem-Replicator API

aem_replicator "replicate_to_publish_servers" do local_user node[:aem][:author][:admin_user]

local_password node[:aem][:author][:admin_password]

local_port node[:aem][:author][:port]

remote_hosts node[:aem][:author][:replication_hosts]

dynamic_cluster node[:aem][:author][:find_replication_hosts_dynamically]

cluster_name node[:aem][:cluster_name]

cluster_role node[:aem][:publish][:cluster_role]

type :publish

action :add

•! tacit-aem-cookbook/recipes/author.rb

What about Secret Management ?

aem_replicator "replicate_to_publish_servers" do local_user node[:aem][:author][:admin_user]

local_password node[:aem][:author][:admin_password] local_port node[:aem][:author][:port]

remote_hosts node[:aem][:author][:replication_hosts]

dynamic_cluster node[:aem][:author][:find_replication_hosts_dynamically]

cluster_name node[:aem][:cluster_name]

cluster_role node[:aem][:publish][:cluster_role]

type :publish

action :add

•! tacit-aem-cookbook/recipes/author.rb

UX/Dev/QA/Ops

dev dev-stable prod ps

Chef-server

https RSA private key Auth

chef-client chef-client

https RSA private key Auth •! Chef encrypted data bags

•! Encrypted for •! admin users •! whitelisted nodes

•! Managed by chef-vault ruby gem

•! Git Back up •! Encrypted obviously

Encrypted Data Bags with Chef-vault

Chef-Vault in Action:

include_recipe 'chef-vault-util::default'

item = chef_vault_item(node['aem']['vault_accounts'], node['aem']['vault_accounts_cq_admin_item']) cq_admin_username = item[node['aem']['vault_accounts_username_property']]

cq_admin_password = item[node['aem']['vault_accounts_password_property']]

log 'the secret accounts credential was fetched from a chef-vault enabled encrypted data_bag for username '+ cq_admin_username do level :info

•! francois-aem-cookbook/recipes/secure.rb

https://github.com/francoisledroff/aem-cookbook https://github.com/francoisledroff/chef-vault-util

$ knife vault update accounts prod_cq_admin -J accounts/prod_cq_admin.json -A "ledroff," --mode client -S "role:aem_server environment:prod"

Let’s add a Dispatcher : Use Case 2

Dispatcher

Author

Publish

Dispatcher Dispatcher

{ "name": ”<dispatcher-connectcon-fqdn>", "chef_environment": "dev”, "run_list": [ "role[dispatcher]" ] }

$ knife node edit dispatcher-connectcon-fqdn

Another Search Publish

Another Search

Let’s Add a Load Balancer : Use Case 3

Dispatcher

{ "name": ”<lb-connectcon-fqdn>", "chef_environment": "dev”, "run_list": [ "role[lb_dispatcher]" ] }

$ knife node edit lb-connectcon-fqdn

Author

Publish

Author

Publish Publish Publish

Dispatcher Dispatcher Dispatcher

Another Search

Dispatcher Dispatcher Dispatcher Dispatcher

Another Search

•! myapp-cookbook/recipes/log.rb logrotate_app 'aem' do cookbook 'logrotate'

path node['myapp']['log_dir']

frequency node['myapp']['logrotate']['frequency']

rotate node['myapp']['logrotate']['rotate']

end include_recipe 'it-splunkforwarder::default’

•! myapp-cookbook/attributes/log.rb

default.splunkforwarder.inputs = [ {"input_path" =>

"#{node['aem']['log_dir']}/error.log",

default.splunkforwarder.inputs = [ {"input_path" =>

"#{node['aem']['log_dir']}/error.log",

Let’s add log monitoring: Use Case 4

myapp /recipes/log.rblogrotate_app 'aem' do

cookbook 'logrotate'

path node['myapp']['log_dir']

frequency node['myapp']['logrotate']['frequency']

rotate node['myapp']['logrotate']['rotate']

end include_recipe 'it-splunkforwarder::default’

Dispatcher

Author

Publish

Author

Publish Publish Publish

Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher

Publish

Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher

Publish Publish

Publish

Add the above in the base role Add the above in the base role

Let’s cluster things!

AEM Production Infrastructure

Dispatcher

Publish

Author

MongoDB servers

Dispatcher

!" 3rd parties (your legacy, your cloud)

you got it ?

Search API knife

workstation

roles attributes

cookbooks

recipes

Chef-DK

github

opscode rubygem

data bags

attributes

data bags

cookbooks recipes

Chef-server nodes

RSA key Auth

yymaven redhat

RSA Keys

roles Web UI

versions

RSA key Auth

chef-clients

RSA Key

#LearnChef

Learn Stuff Automate IT infrastructure and application delivery

Learn Chef

Slide inspired by Spice up your recipe By Seth Vargo

What Stuff ? •! Git Stuff •! Ruby stuff •! VM / Container stuff •! Cloud stuff •! Network stuff •! *nix tools and stuff •! OS Stuff

–! package management –! services

•! Chef Community Stuff •! Build and packaging stuff •! Continuous Delivery stuff •! Monitoring stuff •! Analytics stuff •! Messaging stuff •! Security Stuff

Devops?

“ You built it You run it! ”

Chef, Devops ? No silver bullet.

https://twitter.com/DEVOPS_BORAT/status/52857016670105600

Chef, Devops ? No silver bullet.

https://twitter.com/mindweather/status/458653460234502144

Why ? “Accelerate, Simplify, Scale”

•! Breaking down the wall of confusion

•! As infra is code, it becomes: –! Testable –! Versionable –! Disposable –! Reproducible

•! “If it’s not in the source control system, it does not exist” @bdelacretaz

https://ww

w.flickr.com

/photos/francoisledroff/6107220850/in/set-72157626126325552

https://www.flickr.com/photos/blmoregon/7883684692

AEM Production Infrastructure

Dispatcher

Publish

Author

MongoDB servers

Dispatcher

!" 3rd parties (your legacy, your cloud)

One more publish?

Dispatcher

Publish

Author

!" MongoDB servers

Dispatcher

!" 3rd parties (cloud stuff)

This needs to be configured

Dispatcher

Publish

Author

MongoDB servers

Dispatcher

!" 3rd parties (cloud stuff)

A more complex production

Next scale of complexity…

Let’s Share the love

master chef class: learn how to quickly cook delightful cq/aem infrastructures

workstation git ssh

cq default page

workstation git chef

end end action

recipes node objectrun

chef client finished

rb case node

add end

Software

cq presentation cq

dynamic components using single-page-application concepts in...

aem/cq hack expo @ adobe sydney

delightful careers

installing the agility aem connector...1. open the aem...

delightful bitefuls

newsletter 15, 2014 ocean indien on - africa...

aem & apache slingŒauš... · sling models apache sling ....

applications explorer missions (aem) - nasa explorer...

aem (cq) dispatcher caching webinar 2013

delightful chocolate

compact cylinder series cq2 - rs online · 1-553 1-537 10...

integrating apache wookie with aem || aem-wookie connector...

aem manual

aem (cq) maintenance webinar 2013

adobe® cq 6® - tekno...

spx-cq presentation april, 2006. about spx-cq about spx-cq...

delightful partnerships

lingotek aem adobe experience formerly cq€¦ · adobe...

adobe cq development environment guide - jeffrey w. … ·...