technical architecture - university of edinburgh€¦ · web viewa git hook (a method to trigger...
TRANSCRIPT
Puppet Configuration Management
Technical Architecture
Version 1.0
26/02/2014
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
Contents1 Project Context............................................................................................................51.1 APP004-20 Automation Tools - Options Appraisal.......................................................51.1.1 Key Stakeholders..........................................................................................................51.1.2 Key Deliverables...........................................................................................................51.1.3 Summary of technical changes.....................................................................................5
2 Service Description.....................................................................................................62.1 Key Technologies..........................................................................................................7
3 Users, Roles and Groups...........................................................................................8
4 Application / Database Connectors...........................................................................94.1 Live Environment..........................................................................................................9
5 Application Server Requirements.............................................................................9
6 Database Server Requirements.................................................................................9
7 Portal Requirements.................................................................................................10
8 Financial Requirements............................................................................................108.1 Estimated Capital Costs..............................................................................................108.2 Annual Recurrent Costs..............................................................................................10
9 Application URLs......................................................................................................109.1 Live..............................................................................................................................10
10 File System Requirements.......................................................................................1110.1 File System Shares.....................................................................................................11
11 Scheduled Tasks.......................................................................................................1111.1 Live..............................................................................................................................11
12 Technical Diagrams..................................................................................................12
13 Backup Policy............................................................................................................13
14 Disaster Recovery Requirements............................................................................13
15 Resilience Measures.................................................................................................13
16 External Access........................................................................................................13
17 Security Considerations...........................................................................................13
18 Authentication and Authorisation...........................................................................14
19 Load Testing..............................................................................................................14
20 Interfaces and Dependencies..................................................................................14
Page 2 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
21 Firewall Configuration..............................................................................................14
22 Software Licence Requirements.............................................................................14
23 Patching and Machine Restarts...............................................................................15
24 Exceptions and Other Issues...................................................................................15
25 Document Sign-off....................................................................................................15
Page 3 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
Version ControlDate Version Author Sections Amendments26/02/2014 1.0 Riky Harris All Initial draft
Page 4 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
1 Project Context
1.1 APP004-20 Automation Tools - Options Appraisal
1.1.1 Key Stakeholders
Role Unit NameTechnical Architect (Owner) IS Apps Development
TechnologyRiky HarrisEwan Scott
Peer Reviewer IS Apps Development Technology
Iain Fiddes
Project Manager IS Apps Project Services Adam Wadee
Production Management Representative
IS Apps Technology Management
Heather Larnach
IS ITI Representative IS ITI Unix Martin Campbell
1.1.2 Key Deliverables
Deliverable Business BenefitInstallation of a Puppet Master A central server to manage Puppet manifests and Puppet
certificates for client servers to connect to, allowing for the consistency, repeatability, and accuracy of the configuration of servers setup and managed by DevTech and TechMan
Installation of Foreman A web application to allow an easy way for reports and Facts sent from Puppet agents to be viewed, and is able to deploy configurations to new ‘bare metal’ servers
Installation of R10K Puppet DSL, Hiera configuration and third party modules can be imported from Git and Puppet Forge in a more streamlined and manageable workflow allowing for growth of the infrastructure
Installation of MCollective Gives an easy method for commands to be run on multiple servers and can be used to trigger Puppet agent runs
Use of Veeam VM replication The Puppet Master is replicated to a secondary site allowing near-zero data loss and very little downtime in a DR event
Written documentation on Puppet usage and workflow
Allows users to follow a set of standards when writing code and configuration and when deploying these to servers
1.1.3 Summary of technical changes
As part of APP004-20, a Technical Design for a Puppet Master has been produced. This contains all of the proposed services required for the intended future workflow: Puppet, Hiera, R10K, MCollective and Foreman. All of these are installed on a single RHEL VM with Veeam replication of the server to a secondary site.
Page 5 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
Page 6 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
2 Service DescriptionPuppet is a configuration management system and can be used to apply and enforce the setup of servers from a central location. It works in a master-slave configuration, with a Puppet Agent installed on each client server (known as a ‘node’), requesting its correct configuration (a ‘manifest’) from the Puppet Master. Facter is part of Puppet, and is used to gather information (‘Facts’) about a node, which can be used to control how manifests are applied and is useful to see information about servers.
By using Puppet and the extra tools, a new workflow is being used by IS Applications where server configuration can be represented as ‘code’. This allows the same configurations to be reliably deployed to servers and keeps environments consistent which has become vital as the infrastructure configured and managed by the department has grown so vast, the inefficiencies of handcrafting artisan servers has become too great.
This configuration is represented as a combination of Puppet Domain Specific Language for Puppet modules, YAML for Hiera settings and Ruby for Puppet templates and is all stored centrally within repositories on the IS Applications Git service. It is vital that a Version Control System is used so that changes to configuration can be tracked and easily reverted if issues are found.
Hiera is an accompanying tool to substitute different variables in Puppet DSL dependant on various qualities of the server, in a prioritised, hierarchical manner. This allows servers of different environments or sites to have different settings while being able to use the same Puppet modules.
R10K is used to download requested versions of local and third party modules from Git and Puppet Forge (a central repository of Puppet modules for public use) onto the Puppet Master. A Git hook (a method to trigger tasks when updates are made) is used to trigger R10K to run when a commit has been made to the various Puppet repositories on the IS Applications Git service.
As changes made to the Puppet and Hiera configuration can make massive changes to many servers, peer review takes place before commits to Git are merged into the “Trunk” of the central repositories. There are also Git hooks in place to perform ‘lint testing’ on Puppet manifests to ensure that syntax errors are caught before updates can be committed to Git. This testing can be further developed in the future by using a Continuous Integration style of workflow for the development, automated testing and potentially even deployment of Puppet manifests.
Page 7 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
Foreman is used to give a visual representation of the Puppetised infrastructure and is also used for the initial configuration of new servers. It can display reports on the Puppet agents and show discovered Facter facts. It also can deploy configurations onto new ‘bare metal’ servers, including cloud infrastructures and virtual hosting services like IS ITI’s VMware ESX.
MCollective (“The Marionette Collective”) is another tool produced by Puppet Labs which is not required by Puppet nor requires Puppet itself. It is very useful though and is commonly found as part of Puppet deployments, as it gives the ability run commands with elevated privileges on multiple servers. It is being used to trigger Puppet agent refreshes, and it is possible to use this to perform tasks on multiple servers for project Implementation tasks.
The terminology is oddly reversed though, MCollective requires a single Client (which runs on the Puppet Master) to send requests to the many Servers (which each of the Puppet nodes runs one of) via a single Middleware service which handles the communications (also runs on the Puppet Master and uses ActiveMQ to communicate with the STOMP protocol with added SSL).
2.1 Key Technologies
Technology Version New or existingPuppet Labs Puppet 3.5.1 NewPuppet Labs Hiera 1.3.2 NewPuppet Labs MCollective 2.5.0 NewR10K 1.3.0 NewForeman 1.5.1 NewOracle MySQL 5.1.73 Community Edition ExistingApache HTTPD 2.2.15 ExistingRed Hat Enterprise Linux 6.5 x64 ExistingRuby 1.9.3 Existing (as 2.1.0 for GitLab)Apache ActiveMQ 5.8.0 New
Page 8 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
3 Users, Roles and Groups
Application Username
Home Directory Description
puppet /var/lib/puppet User for Puppet master and Foreman applicationforeman /usr/share/foreman User for Foreman applicationforeman-proxy /usr/share/foreman-
proxyUser for Foreman proxy daemon
Group Name Members Descriptionpuppet puppet, foreman,
foreman-proxyGroup for the puppet users
foreman foreman Group for the foreman userforeman-proxy foreman-proxy Group for the foreman-proxy user
Database Username Roles Descriptionpuppetlab@[email protected]
“All” User for Foreman application connection
OPS$ Username Roles DescriptionN/A
Database Role Description“All” SELECT, LOCK TABLES, INSERT, UPDATE, DELETE, CREATE, DROP,
INDEX, ALTER - full access to all MySQL privileges
Database Schema Tablespace Descriptionpuppetlabhq_production N/A Database for Foreman application
Page 9 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
4 Application / Database ConnectorsNB: There is no Development or Test environment. Ruby applications frequently refer to the Live environment as “Production”.
4.1 Live Environment
Application Name
Puppet Hiera MCollective R10K Foreman
Technology N/A N/A N/A N/A Ruby mysql2 gem
Username puppetlabInstance puppetlabhq_p
roductionAdditional settings
5 Application Server Requirements
Development Test LiveServers puppet.is.ed.ac.uk
Usernames puppet
Application home directories
/var/lib/puppet
Software and versions
Puppet 3.5.1
Additional dependencies
Hiera 1.3.2MCollective 2.5.0
6 Database Server Requirements
Development Test LiveServers puppet.is.ed.ac.ukInstance names puppetlabhq_productionDatabase versions MySQL 5.1.73Project specific requirements
Page 10 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
7 Portal Requirements
Development Test LivePortal channel N/A N/A N/A
8 Financial RequirementsThere is a 5 year cost for running puppet.is which is being met by the IS Apps central budget.
8.1 Estimated Capital Costs
Item Development Test Livee.g. hardware £ £ £e.g. disk on SAN £ £ £e.g. backup £ £ £e.g. licences £ £ £
Subtotals £ £ £Estimated Total £
8.2 Annual Recurrent Costs
Item Development Test Livemaintenance £ £ £support £ £ £backups £ £ £
Subtotals £ £ £Total £
9 Application URLs
9.1 Live
Application URLForeman https://puppet.is.ed.ac.uk
Page 11 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
10 File System Requirements
Server Volume Size Purposepuppet.is / 20Gb OSpuppet.is /u01 180Gb Data
10.1 File System Shares
Server Path Shared as Users / Groups PermissionsN/A
11 Scheduled Tasks
11.1 Live
Task Schedule Username Descriptioncron: MySQL DB Backup
Nightly oracle mysqldump of puppetlabhq_production database
Something else?
Page 12 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
12 Technical Diagrams
Server Estate
Puppet 25/06/2014
APP004-20
CentralFirewall
User connecting to Foremanvia web interface
HTTPS
HTTPS
SSH
SSH
Appleton TowerKings Buildings
CentralFirewall
User running commands on agentsvia MCollective
Veeam VM Replication
HTTPS
Puppet Master VM Offline Replicated Puppet Master VM
Managed Server Estate
STOMP+SSL
Page 13 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
13 Backup Policy
Component Detail any variance from the currently agreed backup policyOperating System backup
Veeam replica is the backup
Database backup Veeam replica is the backupFilesystem backup Veeam replica is the backupOther N/A
14 Disaster Recovery Requirements
Application Category Recovery URLPuppet 2/1? N/AMCollective 3 (2/1 if required for deployment) N/AR10K 2/1? N/AForeman 3 (2/1 if required for deployment) N/A
15 Resilience MeasuresThe Puppet Master server is constantly having the changes made to it replicated to a standby VM at the Appleton Tower site. In Disaster Event, the replica VM can be brought online very quickly and with near-zero data loss.
16 External Access
External Details (telephone number, email etc.)
Access Type Authentication and authorisation methods
Description of need
N/A
17 Security ConsiderationsThe Puppet tool is very powerful, and therefore a rigorous process needs to be followed. In the same way, MCollective’s ability to run commands on multiple servers with elevated privileges also makes it very useful, but also needs to be used with care. Thorough peer-review and testing of all Puppet and Hiera code must be used to ensure that mistakes are not made.
Page 14 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
Day to day updates of configurations are made via changes to the repositories being pushed through by Git hooks calling R10K. Manual use of MCollective should be used for special tasks and are audited by the Kerberised logins to the Puppet Master server.
18 Authentication and AuthorisationThe Puppet deployment process is mainly done by changes made to the Git repositories. Therefore the authentication and authorisation section of the Git service TAD should be read for further details (K:\ISAPPS\dsg\DevelopmentTechnology\ServiceDocs\Infrastructure\ Git\Technical Architecture\Git Technical Architecture.docx).
User accounts on Foreman will be initially created via the Foreman interface by administrative users (IS Applications team leaders). Users will receive their initial passwords and will be able to log into the Foreman interface.
19 Load TestingNo load testing has been performed on the service, however, as the infrastructure grows and more clients connect to the service, horizontal scaling of the Puppet Master VM may be required. It is possible to run with multiple Puppet Masters, but this greatly increases the complexity of the setup and this vertical scaling should be avoided until it becomes the only sensible method to increase the capacity of the service.
20 Interfaces and DependenciesThe Puppet service depends on the Git hosting service for providing updates to the configurations. If the Git service is unavailable, it won’t be possible to push updates to the configurations to the Git “Origin” and so it won’t be possible to make lasting changes to server setup via R10K.
21 Firewall Configuration
Source Destination Port ProtocolEduninets puppet.is.ed.ac.uk 443 HTTPSpuppet.is.ed.ac.uk Eduninets 8139 HTTPSEduninets puppet.is.ed.ac.uk 8140 HTTPSpuppet.is.ed.ac.uk Eduninets 61614 STOMP+SSL
Page 15 of 16
Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014
22 Software Licence Requirements
Software and Supplier Details
Licence Requirements
Already Licensed? (Y/N)
New or Additional Licence Costs
Red Hat Enterprise Linux
1x x64 server Y New (included by IS ITI Unix as part of VM)
23 Patching and Machine Restarts
Has restart been tested to confirm that policy has been achieved? Y / N
24 Exceptions and Other IssuesNone.
25 Document Sign-off
Architect Name Name (date)IS ITI Representative Name (date)Project Manager Acceptance Document URL of email or PPDRBusiness Partner Acceptance Document URL of email or PPDR
Page 16 of 16