![Page 1: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/1.jpg)
PuppetCamp Feb 7, 2013
Case Study
Scaling Puppet Usage to a Global Organization
![Page 2: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/2.jpg)
PuppetCamp Feb 7, 2013
The Presenter
Tor-Åke– Hands-on System Architect– Linux Pro since 2000– DevOps / Operation Automation since 2005– Puppet since 0.25
RedBridge– Open Source Infrastructure Builders– Consulting, Operations and Open Source
Products Since 2003– Partners with PuppetLabs, Redhat, Amazon,
Zimbra, Others
![Page 3: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/3.jpg)
PuppetCamp Feb 7, 2013
TOC
Part 1: Case description– The Customer and the Challenge
Part 2 : Way of Working– How to Develop Code
Part 3 : Technical Platform– Puppet Masters etc
![Page 4: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/4.jpg)
PuppetCamp Feb 7, 2013
Part I : Case Description
A Swedish telecommunications companyAbout 10 sites around the worldThousands of usersThousands of systems
![Page 5: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/5.jpg)
PuppetCamp Feb 7, 2013
System Types (in scope)
Virtual and metal serversSuSE, RedHat, CentOS, Ubuntu and SolarisMostly OSS and Third Party SoftwareDivisions choose from predef:d system types
– Some unique applications
Supporting infrastructure– Network flesystems for applications and user
data– OS-native deployment systems (Satellite, Ops
Center etc)
![Page 6: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/6.jpg)
PuppetCamp Feb 7, 2013
The Challenge
Bring Home R&D IT From Outsourcing
... and in the process:
Shorten Lead Time– Automate Deployment and (Change)
Management
Increase Cost Efficiency– Solve each problem once– Share the solution globally– A scalable technical platform
![Page 7: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/7.jpg)
PuppetCamp Feb 7, 2013
Additional Requirements
Leverage existing expertiseMaintain site-local freedom to solve unique problemsKeep site freedom to plan and execute code updates
![Page 8: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/8.jpg)
PuppetCamp Feb 7, 2013
Puppet?
Puppet is naturally only a part of the solution...
...but an important focal point as it touches all services!
![Page 9: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/9.jpg)
PuppetCamp Feb 7, 2013
Part II : Way of Working
Why and when is code developed?How and by whom?How is code shared?
![Page 10: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/10.jpg)
PuppetCamp Feb 7, 2013
The Facilitator
A Global Team, coordinating Puppet DevelopmentKnowledge identifcation and sharingKeeper of the code standardDevelop and support a Puppet architecture
![Page 11: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/11.jpg)
PuppetCamp Feb 7, 2013
Example Why and When
Site users need a service e.g. ”Hosted Jenkins”Site team calls for Puppet codeGlobal team fnds a Jenkins expertGlobal team helps Jenkins expert write modulesModules are delivered to requesting site team and
users
![Page 12: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/12.jpg)
PuppetCamp Feb 7, 2013
Another example
Site users need a service e.g. Hosted TomcatSite has a Tomcat expert who can write Puppet
moduleGlobal Team is notifed that Tomcat module existsIf another site requests same service, existing code
is ”globalized” with assistance from Global Team
![Page 13: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/13.jpg)
PuppetCamp Feb 7, 2013
Development Sync FlowchartNeedcodeNeedcode
Globalavailable?
Globalavailable?
Askaround
Askaround
Anyavailable?
Anyavailable?
Modifyand testModify
and test
Fetchand testFetch
and test
Post globalsuggestionPost globalsuggestion
CodeOK?
CodeOK?
Done.Deploy!Done.
Deploy!
Developand testDevelopand test
No
Yes
Yes
Yes
No
No
![Page 14: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/14.jpg)
PuppetCamp Feb 7, 2013
Code Sharing
Global Git repository (actually per module)Each site pulls code to site-local repositoryTest locally, and deploy (ITIL Change)Global team is notifed of any local changesIf changes are to be globalized, GT pulls code from
site-local repo
![Page 15: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/15.jpg)
PuppetCamp Feb 7, 2013
Code Standard
Code structure optimized for sharing some parts, while keeping others site-private
Readability and documentation built inUnit test
![Page 16: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/16.jpg)
PuppetCamp Feb 7, 2013
Code Structure
Parameters module local to the siteparams::jenkins in
moduleroot/params/manifests/jenkins.pp
All parameters can be overridden per nodePriority:
1)Node defnition (class params)2)Params module3)Module default (in init.pp!)
![Page 17: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/17.jpg)
PuppetCamp Feb 7, 2013
README
What is the scope of this moduleWhat site and what OS:es has it been tested onExample params fle for params moduleNo description of params!
– Those go in init.pp
![Page 18: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/18.jpg)
PuppetCamp Feb 7, 2013
Predictable Results
Else-clause with a fail()– e.g.
If $::operatingsystem == Solaris { …} else { fail ( ”we have not tested this OS yet” )
}
![Page 19: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/19.jpg)
PuppetCamp Feb 7, 2013
Code Review
Members of the Global Team send code for review”Please look at this code and test it on your site”
– Code deemed unreadable = FAIL– Code breaks other modules unit test = FAIL
![Page 20: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/20.jpg)
PuppetCamp Feb 7, 2013
Adherance to Standard
Lots of code contributors– Varying experience with Puppet
Not always developers– Unfamiliar with peer review, Scrum, XP, Unit
tests etc.
Global Team must fll the gaps– But we are not subject experts!– Educate eachother
![Page 21: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/21.jpg)
PuppetCamp Feb 7, 2013
Boilerplate
A module with all elementsWell commentedCopy and fll out the blanks
![Page 22: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/22.jpg)
PuppetCamp Feb 7, 2013
Manifest Patch Strategy
Many small increments?ORTake a big hit when needed?
+ 10 sites with slight differences, ever-evolving+ 7 different OS:es+ System experts distributed on the sites= Regression testing must also be distributed!
![Page 23: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/23.jpg)
PuppetCamp Feb 7, 2013
Part III : Puppet Platform
Serve thousands of clientsDeployable by Puppet (apply)Support Way of Working
![Page 24: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/24.jpg)
PuppetCamp Feb 7, 2013
Part III : Puppet Platform
Serve thousands of clientsDeployable by Puppet (apply)Support Way of Working
![Page 25: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/25.jpg)
PuppetCamp Feb 7, 2013
Deploying Puppet
A global network flesystem (rsync+nfs)Git repos with puppet code and packagesClone itChange parametersBootstrap a frst Puppet Master
![Page 26: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/26.jpg)
PuppetCamp Feb 7, 2013
Adding More Masters
DNS alternate names in the RR certifcateAdd serverMount shared storageBootstrap server from another serverpuppet agent –server=... --ca_server=...
![Page 27: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/27.jpg)
PuppetCamp Feb 7, 2013
Puppet Masters
3-10 Masters1 CA ServerShared (NFS) storage
– Manifests– Certifcates
Apache plus PassengerRound Robin DNS Records
![Page 28: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/28.jpg)
PuppetCamp Feb 7, 2013
Foreman
Just reports (for now...)Masters store Yaml report on diskSpool to foreman db periodically
– Foreman server can be ofine indefnitly w/o losing reports
![Page 29: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/29.jpg)
PuppetCamp Feb 7, 2013
Agents
Deployed with Kick/JumpstartVersion 2.7Installed under /optRun by cron, hourly with splay
![Page 30: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/30.jpg)
PuppetCamp Feb 7, 2013
puppet:/// fles
We try to avoid them. Why?– NAS is faster than Passenger
Packages are installed from OS native channels– Available in Global NFS
What to do with Solaris?– Packages directly from Global NFS
![Page 31: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/31.jpg)
PuppetCamp Feb 7, 2013
Example: installing Solaris pkg
With puppet fle transferif $::custom_fact == install { file
{ “local.pkg”: source => puppet:///remote.pkg
} file { “local.adm”: source => puppet:///remote.adm
} File[“local.pkg”] -> Package[‘VNDRpkg’]}package { ‘VNDRPkg’: source => “local.pkg”, adminfile => “local.adm”,
}
# + the custom fact ruby code!
With NFS Mountpackage { ‘VNDRpkg’ : source =>
‘/net/nfsserver/remote.pkg’, adminfile => ‘/net/nfsserver/remote.adm’,
}
# We’re done!
![Page 32: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/32.jpg)
PuppetCamp Feb 7, 2013
Orchestration
We don't have it (yet!)Generous ITIL Change WindowsSprawling networks
– Firewall red tape
![Page 33: Scaling Puppet Usage to a Global Organization](https://reader034.vdocument.in/reader034/viewer/2022051514/54868e4ab4af9f730d8b5240/html5/thumbnails/33.jpg)
PuppetCamp Feb 7, 2013
Lessons Learned
Modules should not depend on modules– 10 sites with prod, dev and test environments– Slightly different module version mix
Puppet is not for Everything!– Template shellscripts are powerful– So is Rpm/Deb/Pkg
The biggest issues are with people– Aligning expectations– Consensus about everything from way-of-
working to variable naming