best practices for mission-critical jenkins
DESCRIPTION
From the Jenkins User Conference 2012 in NYC.TRANSCRIPT
![Page 1: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/1.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Best Practices for a Mission-Critical Jenkins
Mike RooneyConsultant/Jenkins Connoisseur
http://linkedin.com/in/mcrooney
![Page 2: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/2.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Jenkins Uses
Genius.com– staging deployment, code reviews,
automated branching and merging, monitors
Canv.as– continuous deployment, scoring, monitoring,
newsletter mailing
Conductor– environment creation, staging / prod
deployment, selenium monitoring
![Page 3: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/3.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Hand-check: How critical is your Jenkins?
![Page 4: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/4.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
What problems have you faced?
![Page 5: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/5.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Problems
disk failure / data losshardware failure / downtimeload / latency
![Page 6: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/6.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Solution
make Jenkins instance trivial to respin– ideally a one-liner that even handles DNS– “create.sh jenkins”
![Page 7: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/7.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Persistence
$JENKINS_HOME– plugins, users, jobs, builds, configuration
![Page 8: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/8.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Persistence
git / svn– make $JENKINS_HOME a checkout– have a Jenkins job that commits daily– examples:
http://jenkins-ci.org/content/keeping-your-configuration-and-data-subversion
![Page 9: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/9.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Persistence
EBS on AWS– put $JENKINS_HOME on an EBS volume– snapshot nightly via a Jenkins job– trivial to attach to a new host, restore
snapshot
a NAS + RAID / backups works similarly
![Page 10: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/10.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Environment
Jenkins is more than $JENKINS_HOME– specific Jenkins .war / .deb / .rpm version– startup options– dependent packages: git, ruby gems, pip– ssh keys, m2 settings– swap, tmpfs, system configuration
![Page 11: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/11.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Environment
configuration management:Puppet/Chef*
* https://wiki.jenkins-ci.org/display/JENKINS/Puppet
![Page 12: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/12.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Environment
standalone– puppet apply path/to/your/manifest.pp
puppetmaster– set up /etc/puppet.conf, run puppet agent
![Page 13: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/13.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Putting it Together
have manifest handle $JENKINS_HOME– clone git repo, mount EBS volume, etc
![Page 14: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/14.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Putting it Together…on AWS
upload manifests to S3 on check-in– a Jenkins SCM job using S3 plugin
use cloud-init to install puppet, download manifests, and run puppet– a custom AMI with an rc.local script also
works
when it dies: “create.sh jenkins”– ec2-launch-instance config user-data
![Page 15: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/15.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Monitoring
… but how do you know when it’s down?check out services like Pingdom– notifies you when a URL does give HTTP 200
OK
![Page 16: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/16.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Going further: Elastic Beanstalk
handles provisioning simply from a .warpros– just give it a war– automatically replaces unhealthy instances– behind a load-balancer (consistent URL)– normally hard AWS changes like AMI, Security
Groups, or Key Pairs are now trivial to make
cons– behind a load-balancer (cost overhead)– no UI option (yet) for controlling AZ– no great way to pass data to instances for puppet– locked in to Amazon Linux AMI (CentOS)
![Page 17: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/17.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Going further: Elastic Beanstalk
set min/max instances to 1– ignore scaling triggers, irrelevant in this case
use beanstalk CLI to set desired AZ (if EBS)– https://forums.aws.amazon.com/thread.jspa?th
readID=61409
puppet– use a custom AMI that specifically runs Jenkins
manifests– but this requires a specific AMI for each
Beanstalk application.– let’s get creative…
![Page 18: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/18.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Going further: Elastic Beanstalk
passing data to instances
PARAM1..5 meant as args to .warend up in /etc/sysconfig/tomcat7 JAVA_OPTSparse out and:– puppet apply –certname=$PARSED_ROLE
![Page 19: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/19.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Questions?
![Page 20: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/20.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
High Availability Artifacts
protect: artifacts, reports, userContentfrom:– planned downtime:
Jenkins restarts/upgrades, server upgrades– unplanned downtime:
software/hardware failure– unresponsive Jenkins:
very high load
![Page 21: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/21.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
High(er) Availability Artifacts
easy mode:– put Jenkins behind nginx/apache, shadow
userContent and relevant directories– still available during Jenkins restarts, or very
high Jenkins load/latency– not safe from server downtime
![Page 22: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/22.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
High Availability Artifacts
advanced mode: S3– 99.99% availability, 99.999999999%
durability*• if you store 10K objects, expect to lose one every
10 million years
– use Jenkins S3 plugin to upload artifacts to S3
* http://aws.amazon.com/s3/faqs
![Page 23: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/23.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Fault-tolerant Jobs
design with possible downtime in mind– SCM triggering is great, but keep polling too
![Page 24: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/24.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Fault-tolerant Jobs
*/15 * * * *– BAD:
update users where join_time < 15m ago– GOOD:
update users where id > last_id_updated
![Page 25: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/25.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Error handling
for non-critical jobs, use email / IM post-build notifiers– but be careful of creating too much noise,
people will ignore or filter it out
for critical jobs, integrate Jenkins with a service like PagerDuty– Jenkins emails [email protected]– PagerDuty texts / calls the people on-call
until resolved– a failing build will wake you up at 4AM
![Page 26: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/26.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Questions?
![Page 27: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/27.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Security: Authentication
read-onlymatrix-basedHTTP basic auth
![Page 28: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/28.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Security: Authentication
but what about traffic sniffing?
![Page 29: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/29.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Security: HTTPS
throw nginx/apache in front of Jenkins– proxy mode– ssl (self-signed or just buy one)
![Page 30: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/30.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Security: Authorization
use project-based matrix authenticationgive anonymous/authenticated readonlyuse it if you’ve got it:LDAP, Active Directory, UNIXJenkin’s own database also works fineensure each user has their own account– each build will have an audit trail
![Page 31: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/31.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Security: Authorization (AWS)
when interfacing with AWS API/CLI, use IAM so Jenkins can only access what it needs
![Page 32: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/32.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Questions?
![Page 33: Best Practices for Mission-Critical Jenkins](https://reader036.vdocument.in/reader036/viewer/2022081602/554f41a3b4c905423f8b45aa/html5/thumbnails/33.jpg)
Jenkins User ConferenceNew York, May 17 2012 #jenkinsconf
Thank You To Our SponsorsPlatinumSponsor
GoldSponsors
SilverSponsors
BronzeSponsors