the kitchen cloud how to: automating joyent smartmachines with chef

Proprietary and Confidential

Automating Joyent SmartMachines with Chef

Chef on SmartOS

Eric Saxby@sax @ecdysone @sax

Who am I?


■ Application developeroperational experience with many technologies, project by project

■ BSD/AIX/UbuntuSolaris in 2002, but I was very much out of my element

■ Switched to DevOps-y team 18 months agoMultiple back end services for a large e-commerce site, transitioning to SmartOS

■ Now I’m at Wanelo

From a certain point of view...


What is Wanelo?


■ Wanelo (“Wah-nee-lo” from Want, Need Love) is a global platform for shopping.


Marketing-free shopping across 100s of thousands of unique stores


Personal feed of products from any store on the internet

Technology overview


■ MRI Ruby 1.9.3 & Rails 3.2

■ PostgreSQL 9.2.4, Solr 3.6

■ Joyent Cloud, SmartOSZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace

■ Circonus, Chef + OpscodeMonitoring, graphing, alerting, automation

■ Amazon S3 + Fastly CDN

■ NewRelic, statsd, Graphite, nagios

What’s SmartOS?


■ Illumos branch optimized for cloud computing

■ Developed by Joyent for their public cloud

What’s Illumos?


■ It’s what OpenSolaris became after Oracle killed the project

■ Umbrella for various distributions, each committed to pushing their improvements upstream

■ http://wiki.illumos.org/display/illumos/About+illumos

What does SmartOS look


■ Compute Node — physical server

■ Global Zone — host OS (SmartOS)

■ Non-Global Zone — like a virtual machine, with native system calls (no fake hardware layer)

■ Very secure

■ Can run KVM for guest OS (Ubuntu, Centos)

How is it deployed?


■ Can manage from global zone (imgadm, zoneadm)

■ Tools provide APIs

■ Smart Data Center (Joyent’s tools, can be licensed)

■ Project FIFO (SDC API in free package)

■ Joyent Public Cloud

■ Many compute nodes working in a cluster, PXE booted from a head node

■ Service Management Facility (SMF)If init.d and monit and god were one thing, and actually awesome

Why should I care?


■ Visibility tools dtrace, kstat, snoop, truss

■ ZFSFile system built for speed and data integrity

■ Application LatencyZones are OS virtualization, so fasterProcesses are scheduled in global zone kernel,not in a hardware virtualization layer

■ # cores, RAM required =~ # processes

Lower latency == less cost


■ # processes required =~ requests/second of site

■ Requests/second of single process =~ request latency

$$$

On to Chef!


Terminology


■ Image / Dataset — OS at a particular version, snapshotted at base state

■ Flavor / Package— RAM, CPU shares

■ API URL — Each data center has its own URL

■ Server ID / Zonename — Each zone gets a UUID

knife-joyent


Installation/Configuration


■ Update knife.rb

■ Add to Gemfile

knife[:joyent_username] = 'sax'knife[:joyent_keyname] = 'EricSaxby'knife[:joyent_keyfile] = "#{ENV['HOME']}/.ssh/id_rsa"knife[:joyent_api_url] = 'https://us-sw-1.api.joyentcloud.com/'

■ Add first public key in cloud APIhttps://my.joyentcloud.com

gem 'knife-joyent'

Managing keys


■ No role based access, but at least you can make each user upload their own keyknife joyent key add -f ~/.ssh/id_rsa -k KeyNameknife joyent key delete KeyName

■ Passphrase protected keys are annoyingEach API request includes data signed with the private key. Ruby does not have a good way of signing private keys with ssh-agent.

Creating servers!


■ See what images are availableknife joyent image list

cf7e2f40-9276-11e2-af9a-0bad2233fb0b base64 1.9.1 smartos

f4bc70ca-5e2c-11e1-8380-fb28785857cb smartosplus64 3.1.0 smartos

da144ada-a558-11e2-8762-538b60994628 ubuntu-12.04 2.4.1 linux

■ base / base64 — minimal install, you add what you need

■ smartosplus — many more things pre-installed, but can get in the way

13328c9a-9173-11e2-a9a5-2ff43d306c21 ws2008ent-r2-sp1 2.0.2 windows

Creating servers!


■ See what flavors are availableknife joyent flavor listName RAM Disk SwapExtra Small 512 MB 0 GB 15 GB 1 GBSmall 1GB 1 GB 30 GB 2 GBMedium 2GB 2 GB 60 GB 4 GBMedium 4GB 4 GB 120 GB 8 GBLarge 8GB 8 GB 240 GB 16 GBLarge 16GB 16 GB 480 GB 32 GB

■ Custom networking can be done in a custom flavor (ie public or private VLAN, routes)

Creating servers already!


knife joyent server create --image cf7e2f40-9276-11e2-af9a-0bad2233fb0b --flavor 'Medium 2GB' -N server.domain.com -E environment -d distro -r run_list

■ No Omnibus, so you have to provide your own distro bootstrap template

https://gist.github.com/sax/5457464

knife joyent server list

See what's there...


a597a3a7-3fdf-481f-af08-e7c1e0ae7dca admin.prod running smartmachine sdc:sdc:base64:1.8.1 8.19.1.1 10.100.1.1 8 GB 240 GB5c066e6e-8af2-4d4f-a81e-c8e2691ae8a0 demo.dev running smartmachine sdc:sdc:base64:1.8.1 10.12.1.1 165.225.1.1 8 GB 240 GBb3370d52-3bed-462e-857a-e17eba15ab06 app010.c1.prod running smartmachine sdc:sdc:base64:1.8.1 10.100.1.2 165.225.1.2 8 GB 240 GB

■ ID / zonename■ Name■ Run state■ Type

■ Image■ IP addresses■ RAM■ Disk

Other management


knife joyent server delete <server_id>knife joyent server start <server_id>knife joyent server stop <server_id>knife joyent server reboot <server_id>

knife joyent server resize <server_id> -f <flavor>knife joyent snapshot create <server_id> <snapshot_name>

■ Snapshots are full ZFS snapshotsCopy-on-write snapshot of local file system.Each snapshot is locally mounted in zone at/checkpoints

So now you have a smartmachine...


What's different?


■ Things you expect in /usr/local are in /opt/local■ For historical reasons■ If you're used to Linux, this can be annoying■ Joyent is working on a more Linux friendly image■ For now, add /opt/local/bin to PATH

■ Many configs are in /opt/local/etc instead of /etc■ Some utilities are different■ This is not the grep you're looking for....

■ Symlink your "correct" version into /opt/local/bin

■ Add /opt/local/lib to CFLAGS and LDFLAGS

Caveats?


■ Zones inside of zones inside of...

■ Vagrant does not currently work with SmartOS

■ VirtualBox only works in Bridged network mode

■ Local integration tests do not work

Where are all the things?


■ Services■ svcs -a■ svcadm < enable | disable | clear > service

■ Packages■ pkgin search packagename■ pkgin -y install packagename

Public vs. Private IP


■ ipaddr_extensions gem

■ Adds 'privateaddress' attribute to ohai

■ Useful to add this to bootstrap

■ Smartmachines may have a public IP and a private IP

■ Recipes can be configured to use ipaddress or privateaddress

System preparation


■ smartos cookbook■ https://github.com/modcloth-cookbooks/smartos■ fixes chef providers

■ smartmachine_functions

■ links nicer utils into /opt/local/bin

■ https://github.com/higanworks-cookbooks/smartmachine_functions

■ fixes chef providers

■ provides access to Joyent metadata API

or

Useful LWRPs


SMF


■ https://github.com/modcloth-cookbooks/smf■ Chef knows how to use SMF, not how to configure it■ Uses nokogiri, which requires libxslt

smf 'postgres' do user 'postgres' group 'postgres' project 'postgres' start_command 'postgres-service.sh start' stop_command 'postgres-service.sh stop' working_directory '/var/pgsql/data' environment 'PATH' => '/opt/postgres/bin'end

SMF (cnt'd)


smf 'postgres' do user 'postgres' group 'postgres' project 'postgres' start_command 'postgres-service.sh start' stop_command 'postgres-service.sh stop' stop_timeout 120 restart_command 'postgres-service.sh restart' refresh_command 'postgres-service.sh reload' working_directory '/var/pgsql/data' environment 'PATH' => '/opt/postgres/bin'end

service 'postgres' do supports :status => true, :restart => true, :reload => trueend

Resource Control /


■ https://github.com/wanelo-chef/resource-control■ configure max file descriptors, shared memory, etc■ Bunch up master/worker processes to view in

prstat -J

resource_control_project "postgres" do comment "PostgreSQL 9.2" users "postgres" project_limits "max-shm-memory" => 12000000, "max-lwps" => 6 process_limits "max-file-descriptor" => { "value" => 32768, "deny" => true } action :createend

Role Based Access Control


■ https://github.com/modcloth-cookbooks/rbac■ Allows delegation of authority without sudo■ Implementation currently too simple, only useful for SMF

delegation

rbac 'solr' do user 'wanelo' action :add_management_permissionsend

Contributing to cookbooks


■ ~95% just require SMF, correct package names

■ ~5% of those need a special init script

■ The rest usually require custom compile

`postgres -D /path/to/data` not granular enough`pg_ctl -D /path/to/data < start | stop | reload | refresh >`

--with-libraries=/opt/local/lib --with-includes=/opt/local/includeLDFLAGS='-R/opt/local/lib -L/opt/local/lib'

Comments? Questions? Find me.

https://github.com/wanelo

https://github.com/wanelo-chef

https://github.com/wanelo-chef/smartos-chef-repo


@sax @ecdysone @sax

the kitchen cloud how to: automating joyent smartmachines with chef

Technology

costproprietary andcondential

joyent cloud

global zone imgadm

global zone kernel

comgem knifejoyent

whats smartos

os virtualization

global platform