rock-solid magento development and deployment workflows
TRANSCRIPT
2 October 26, 2015
San Francisco, CA
Janine
Fiona
that’s me
Leo
Continuous Development
Continuous Delivery
Continuous Integration
Continuous Deployment
Continuous Monitoring
Continuous
Jenkins
Everything sounds better with “continuous”
plan
code build
test
deploy operate
http://kirbiecravings.com/2014/08/blueberry-mug-cake-2.html
code
CMS
Magento
Solr
ERP
Project
Applications
Magento Magento
117
Magento
118
Magento
119
Builds
Release
Staging
Magento
72
Magento
103
Magento
119
Magento
72
Magento
103
Magento
119
Magento
72
Magento
103
Magento
119
Environment
Servers
Instances
Staging Integration Devbox Production Deploy
transient, potentially broken
Staging Integration Devbox Production Deploy
VirtualBox + Vagrant
Jenkins Server
identical setup
One size doesn’t fit all!
What is it? A simple shell script. That’s it! No Ant, no Phing, no fancy PHP tool/framework
Where is it used?
In the build job of a Jenkins server
What does it need to know? Where to get all the source code from
What does it do? add metadata, composer, minify, packaging, upload to S3
What is it? A couple of chef cookbooks. Run by AWS OpsWorks or Vagrant,…
Where is it used?
On the target servers
What does it need to know? The role of the server (web server, devbox, Redis,…)
What does it do? Installing and configuring web server, PHP, MySQL, Redis,…
What is it? A simple shell script. Or the “deploy” chef cookbook Or EasyDeployWorkflows
Where is it used?
On the target servers
What does it need to know? How to get the build and where to deploy it to
What does it do? (Resetting the environment), getting the build, extracting it, managing release folders and symlinks, triggering installer, cleanup old releases
What is it? A simple shell script. Sometimes triggered inside “deploy” before_migrate.rb
Where is it used?
On the target servers, called from deployment step
What does it need to know? What environment this is. Injected settings
What does it do? Run modman, apply settings, trigger update scripts, customize cache prefix, indexing,…
doesn’t know anything about the app, except where to get it
from and where to put it
knows how to get the app up and
running
no creating CMS blocks
no creating product attributes
no updating of any configuration settings!
no clearing the cache
no purging CDN or Reverse Proxy
content!
media directory (sometimes minified)
Database dump
(Vanilla) Magento core
Modules
Scripts
Settings
Tools
Structure Project
project web root root
This is the complete project
This is where your webserver points to.
project web root root
htdocs
<Name>
<theme>
layout
template
code
design
etc
locale
<area>
core
community
local
<area>
<language>
modules
<package>
<Namespace>
<package>
*.xml
*.csv
<theme>
app
skin
lib
htdocs
<Name>
<theme>
layout
template
code
design
etc
locale
<area>
core
community
local
<area>
<language>
modules
<package>
<Namespace>
<package>
*.xml
*.csv
<theme>
app
skin
lib
Module2
default
layout
template
code
design
etc
locale
frontend
local
frontend
de_DE
modules
base
My
base
My_Module2.xml
My_Module2.csv
default
app
skin
etc, Block, …
js, css, img, …
my_module2
my_module2
Magento My_Module My_Module2
Module
default
layout
template
code
design
etc
locale
frontend
local
frontend
de_DE
modules
base
My
base
My_Module.xml
My_Module.csv
default
app
skin
etc, Block, …
js, css, img, …
my_module
my_module
htdocs
Module
default
layout
template
code
design
etc
locale
local
frontend
de_DE
modules
My
base
*.xml
app
My_Module2.xml
My_Module2.csv
my_module2
My_Module.xml
My_Module.csv
etc, Block, …
my_module
Module2
etc, Block, …
my_module2
my_module
by @colinmollenhour
htdocs
code
core
community
local
app
app/code/…/My/Module
.modman
My_Module
app/design/…
skin/frontend/…
app/etc/modules/My_Module.xml
modman
app/code/…/My/Module
My_Module2
app/design/…
skin/frontend/…
app/etc/modules/My_Module.xml
modman
<theme>
layout
template
design
etc
locale
<area>
<area>
<language>
modules
<package>
<package>
*.xml
*.csv
<theme>
skin
modman doesn’t even need to be in the web root!
Vanilla Magento
absolutely no core hacks!
…and core bugs you’ve fixed (and submitted to
Magento, of course…)
(except official patches)
modman “connects” existing modules to the Magento core, but how do we manage them?
htdocs
code
core
community
local
app
app/code/…/My/Module
.modman
My_Module
app/design/…
skin/frontend/…
app/etc/modules/My_Module.xml
modman
app/code/…/My/Module
My_Module2
app/design/…
skin/frontend/…
app/etc/modules/My_Module.xml
modman
<theme>
layout
template
design
etc
locale
<area>
<area>
<language>
modules
<package>
<package>
*.xml
*.csv
<theme>
skin
tools
n98-magerun.phar
phpunit.phar
modman
EnvSettingsTool
composer.phar
htdocs
.modman
Configuration
Installer
(Metadata, Doc,…)
My_Module
My_Module2
app,…
and usually there’s even more stuff…
These should come from different origins
“inspired” by composer/installers
handles package types
<?php namespace Aoepeople\ComposerInstallers; class MagentoInstaller extends BaseInstaller { protected $locations = array( 'module' => '.modman/{$name}/', 'source' => 'htdocs/' ); }
• Fully compatible with magento-hackathon/magento-composer-installer
• Adds support for type “magento-source”
• Only puts packages into place
• Native modman script is used to deploy modules
• Wrapped in a simple installer script
{ "minimum-stability":"dev", "require":{ "aoepeople/composer-installers": "*", "fbrnc/aoe_scheduler": "*", "fbrnc/aoe_cachecleaner": "*", "aoepeople/magento_enterprise": "1.13.1.0", "aoepeople/envsettingstool": "*" }, "config": { "bin-dir": "bin" } }
replaces Hackathon Installer
some modules
Vanilla Magento Source
Tools,…
Webroot: /var/www/<projectName>/<environmentName>/current/htdocs
var
var
<projectName>
releases
<environmentName>
shared
www
media
build_117
build_118
build_119
.modman
htdocs
Configuration
tools
install.sh
app,…
var
media
previous
latest
current
next
My_Module
My_Module2
https://github.com/AOEpeople/Magento_Boilerplate
discover use
code
review
add
modman
add
composer git integrate test
deploy
download
good luck with that!
pay $xx
to author provide FTP
access
seriously?!
one-click
install
upload
“Step 1”
upload
“Step 2”
clear
caches
Code Quality Modules
Not enterprise ready
Not production ready
Not cloud ready Bad performance
Security
Urls
Database Settings
Payment Provider Configuration Feature
Flags…
Everything that’s different between two
environments
Zettr
Every file is a potential security risk
Don’t deploy anything you don’t need to
operate the website
.git*
./htdocs/includes
./htdocs/downloader
./htdocs/pkginfo
./htdocs/LICENSE*
./htdocs/RELEASE_NOTES.txt
./htdocs/phpunit.xml*
./htdocs/*.sample
./htdocs/var
./htdocs/media
./.modman/Aoe_TemplateHints
./.modman/Aoe_Profiler
./.modman/EcomDev_PHPUnit
…
production, staging,…
devbox and integration environment
Vagrant Boxes
Awesome stuff
Every single line potentially
can break production!
Bratwurst (Germans)
Donut (Americans)
Pizza (Italians)
master
Feature B
Feature A
Hotfix (if master is not modified)
Hotfix (if master is modified)
Feature C
build
install on integration
tests
build
install on integration
tests
install on deploy
build
install on integration
tests
deploy to stage
deploy to prod
build
install on integration
tests
deploy to stage
deploy to prod
build
install on integration
tests
deploy to stage
deploy to prod
single mainline
no commits, only --no-ff
merges
Version Control
GIT, SVN,…
PHP lint
Version Control Code Reviews
Crucible,…
Version Control
Code Reviews
Build
Files
DB
Installer
Settings
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
CodeSniffer
php-pmd
php-cpd
php-depend
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
Install on “Integration”
First system where code from all
developers/teams meets each other
unstable system, might be broken,
will be rebuilt without warning
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
Install on “Integration”
Unit Tests
PHPUnit
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
Install on “Integration”
Unit Tests Integration Tests
AOE PMD (for Magento)
broken links, HTTP headers, W3C validity,…
Test features that rely on other parts (e.g. framework, external services,…)
External components (SSL, Redis, Varnish, ERP, Data Import,…)
Basically this is also PHPUnit, but addressing
other things
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
Install on “Integration”
Unit Tests Integration Tests Acceptance Tests
Selenium
Firefox
Chrome
browser
size
640
980
1280
1900
US languages
UK DE FR IT
language
size
browser
touchstone job
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
Install on “Integration”
Unit Tests Integration Tests Acceptance Tests
Install on “Staging”
Behat, PDIFF,…
Same setup as production, but different machines
Client QA / approval
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
Install on “Integration”
Unit Tests Integration Tests Acceptance Tests
Install on “Staging”
Stress
Tests
JMeter
Spin up EC2 instances to
produce traffic
Again: multiple servers or autoscaling setup
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
Install on “Integration”
Unit Tests Integration Tests Acceptance Tests
Install on “Staging”
Stress
Tests
Install on “Production”
A/B Deployments on different
scales
Plan B: rollback
Zero Downtime
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
Install on “Integration”
Unit Tests Integration Tests Acceptance Tests
Install on “Staging”
Stress
Tests
Install on “Production”
Deployment != Feature Release
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code Analysis
Install on “Integration”
Unit Tests Integration Tests Acceptance Tests
Install on “Staging”
Stress
Tests
Install on “Production”
Enable Features
Version Control
Code Reviews Build Static Code Analysis
Install on “Integration”
Unit Tests Integration Tests
Acceptance Tests
Install on “Staging”
Install on “Production”
Enable Features
Pipeline Visualization
#284
#283
#282
#281
#280
Build Static Code
Analysis Unit Tests
Install on “Integration”
Integration Tests
Acceptance Tests
#285
for team dashboards
type “YESIKNOWWHATIMDOING” + click
automatically
http://joind.in/talk/view/15588
Follow me on twitter!
My blog