system management with spacewalk - froscon · system management with spacewalk tips for managing...
TRANSCRIPT
MotivationInstallation & administration
Tips & tricks
System management with SpacewalkTips for managing Linux and Solaris
Christian Stankowic
http://www.stankowic-development.net, @stankowic_devel
Free and Open Source Software Conference, 2014
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
whoami
$ whoami
Christian StankowicVMware, UNIX, Linux administratorMesser Information Services GmbH
$ apropos
Spacewalk / RHN Satellite / SUSE ManagerIcinga / OMDEnterprise Linux, SUSE, VMware vSphere
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Requirements and necessityor: IT administrators tortures
Normally less administrators manage many systemsOften rapid projects and requests
"We need 10 servers ASAP.""We need this till the end of the week - tomorrow.""Can you make those adjustments quickly? I’m having ademo with the management soon."
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Requirements and necessityor: IT administrators tortures
Resultat: Standards and documentation are often neglected
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Requirements and necessityProduct variety
Central system management is essential - but which tool?The variety is very big, some examples1
PuppetChefAnsible. . .
Alternative suites: Spacewalk, Red Hat Satellite, SUSEManagerCombines amongst others software, configuration andcontent management
1These tools are only offering some of the Spacewalk featuresChristian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Spacewalk variety
2002: First version of Red Hat Network Satellite Server2008: Satellite source code releases as SpacewalkSpacewalk is the upstream project for Red Hat SatelliteServer and SUSE ManagerService contract for SUSE Manager and Red Hat Satelliteneeded, Spacewalk is freeFeatures tested in Spacewalk, often adopted in theEnterprise products
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Common features
Multi-client-capabilityConfiguration managementSoftware/update managementContent provisioning/caching, no dedicated downloads perclient necessary
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Common features
System provisioningSecurity and license auditingCrash reportingMonitoring2
2Nagios / Icinga is definitely more powerful!Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Differences
Spacewalk Satellite SUSE Mgr.Release 2-5 months 9-12 months ?Arch i386, x86_64 + s390x + s390x, ia64,
ppc/ppc64Distro EL, Debian3,
openSUSE,Fedora
+ RHEL + RHEL4,SLES
Database PostgreSQL, Oracle 10gR2/11gExclusive jQuery UI,
Solaris, Powermanagement5
Solaris, RHNconnection
jQuery UI,Power man-agement
3limited support4omits Red Hat support, SUSE Expanded Support5enables cobbler to kickstart hosts more efficient
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Spacewalk architecture
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Spacewalk architectureOpen Source Architecture Daemon
(missing in figure)osad - Open Source Architecture DaemonReal-time system managementAction are started using the Jabber protocolNetwork port 5222/tcp needs to be opened on the client
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Spacewalk architectureAdditional components
tftp-server - required for client network bootcobblerd - automatic TFTP, DHCP and DNS configurationSpacewalk Proxy - software packages are cached locally,reducing load/traffic
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Requirements and necessitySpacewalk varietyNews
New in Spacewalk 2.2
Version 2.2 was released on 07/16/2014Enterprise Linux 7 clients are now supported6
Read-only API user for auditing purposesAction-Chains, grouping interdependent actions7
Updated Perl, Python and Ruby API (new calls)Solaris support now deprecated
6Host: Enterprise Linux 5/67Demo: http://turing.suse.de/%7Esmoioli/Action%
20Chaining%20screencast.webm
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
System requirements
2 GB+ memoryEnterprise Linux 5/6 host 8
Spacewalk repositoryRHEL Server Optional channel9
JPackage and EPEL repository10
Fedora 20: install rpm-{build,python}, downgradeRPM libraries11
8Versions newer than Spacewalk 2.2 are only supporting EL69Red Hat Enterprise Linux only
10CentOS/Scientific Linux/OEL only11Version 4.11.1-7.fc20
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Network
FQDN and short name needs to be available12
Firewall configurationtcp 80,443 - web interfacetcp 5222 - task scheduling (client systems)tcp 5269 - task scheduling (Proxy)udp 69 - TFTP (Kickstart)
12Check using hostname -s|-f!Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Storage capacity
Storage calculation:At least 12 GB for PostgreSQL database13
At least 6 GB for RPM packagesDepending on your system landscape
250 KiB per registered system500 KiB per software channel230 KiB per package in software channel
13Oracle: differing depending on version and editionChristian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Storage capacityExample
Example calculation:10 CentOS 6 systems, 2 repositories (base + updates)10 systems: 3 MiB (10* 250 KiB)CentOS 6 Base: 500 KiB
6367 packages: 1,4 GiB (6367 * 230 KiB)CentOS 6 Updates: 500 KiB
1103 packages: 248 MiB (1103 * 230 KiB)
Summary: 1,7 GiB
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Installation
PostgreSQL:yum install spacewalk-{,setup-}postgresql
Oracle:yum install spacewalk-{,setup,oracle}XE: use Oracle Instant client14
Details: https://fedorahosted.org/spacewalk/wiki/FullOracleSetup
14The XE client isn’t working at all !Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Initial configuration
Listing 1: Initial configuration
1 # spacewalk-setup --disconnected2 Admin Email Address? admin@localhost3 CA certificate password?4 Organization? MyCompany5 ...6 ** SSL: Generation CA certificate.7 ...8 # chkconfig spacewalk-service on9 # service spacewalk-service start
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Initial configuration
Disable Admin -> Spacewalk Configuration ->Disconnected Spacewalk
Customize to match your company’s structureEnable Solaris support?Create additional user accountsCreate additional organizations and trusts. . .
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Channels, child channels and repositories
Every distribution are mapped to one or more channelsEach channel can consist of multiple child channelsEvery channel is synchronized using a repositoryChannel access can be limited per system
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Channels, child channels and repositoriesExample
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Channels, child channels and repositories
Repository content synchronization:spacwalk-repo-sync
Cronjob or taskomatic plan
Listing 2: Synchronize repository
1 # /usr/bin/spacewalk-repo-sync --channel ⤦Ç centos6-base-x86_64 --url http://mirror.⤦Ç centos.org/centos/6/os/x86_64/ --type ⤦Ç yum
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Channels, child channels and repositoriesAccess limitation per system
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Maintenance tasks
Some possible system maintenance tasks:Installing, updating and removing software packagesApplying errataExecuting shell commandsRestarting systemsUpdating configuration filesand much more. . .
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Maintenance tasksSystem Set Manager
Similar systems can be grouped (web/databaseservers,. . . )All systems of a group can be managed like a single hostFacilitates maintaining big system landscapesTip: groups per application and priority (test, development,production)
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Configuration management
Configuration files15 are stored in one or moreconfiguration channelsChannels can be ordered hierarchically (depending onnetwork/application, . . . )If a configuration file is part of multiple channels the firstoccurrence is selectedUploading/customizing central configuration files using theWebUI
15Symbolic links and binary files are also supported!Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Configuration management
WebUI offers an integrated ASCII editorMacros can insert system profile values (hostname, IPaddress,...)16
Updates stored as revisions, MD5 checksum verificationNo automatic update rollouts
16See Red Hat Satellite documentationChristian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Configuration managementExample
Specify:File name/pathOwner and file modeSELinux contextcustom macro delimiter (if necessary)Configuration file content
Configuration files can also be uploaded
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Configuration managementExample - Configuration channel priorities
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Configuration managementBeispiel - Deploying a new revision
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
CEFS - CentOS Errata for Spacewalk
RHEL customers are receiving errata by RHNCentOS fixes are marked as regular updatesCEFS service17 creates errata automatically (mailing lists)CEFS imports errata locallyErrata information can be combined with Red HatSecurity Announcements (RHSA), more details
17Thank you very much, Steve!Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
CEFS - CentOS Errata for SpacewalkExample
Listing 3: Import recent errata
1 $ wget -N http://cefs.steve-meier.de/errata.⤦Ç latest.xml
2 $ ./errata-import.pl --server localhost --⤦Ç errata errata.latest.xml --include-⤦Ç channels=... --publish
1 Downloading recent definitions2 Importing errata
–errata - XML file–include-channels - import for these channels–publish - publish errata
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
CEFS - CentOS Errata for Spacewalk
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Solaris integration
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Solaris integration
Spacewalk / Red Hat Satellite are offering “UNIXsupport“18
Solaris systems can be registered / managed like LinuxhostsSUN/Oracle Solaris 8 to 10 (x86 + SPARC) officiallysupportedUnofficially also working19:
Oracle Solaris 11OpenIndiana / OpenSolarisIllumos derivates (napp-it, SmartOS,. . . ) should also work
18deprecated since Spacewalk 2.219successfully tested
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Limitations
Software cannot be imported using repositories.pkg files need to be converted (solaris2mpm) anduploadedReal-time maintenance (osad) not possible, rhnsdchecks periodicallyRemote commands unreliable on somearchitectures/releasesHardware / package information partially errorneous
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Preparation - Spacewalk
Enable Solaris supportRestart Spacewalk / Red Hat SatelliteCreate Solaris base channel and sub-channelsCreate activation key and link with base channel
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Installation - Solaris
Download appropriate Solaris Bootstrap package20:http://spacewalkproject.org/solaris21
Install OpenSSL and ZIP libraries and GCC runtime22
Install Bootstrap package and adjust LD Library pathsRegister system using rhnreg_ks, enable remoteconfiguration (rhn-actions-control, optional)
20Use i386-sol10 for newer versions21on Satellite local: http://fqdn/pub/bootstrap/22SUNWgccruntime, SUNWopensslr, SUNWzlib packages
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Installation - Solaris
Configure rhnsd (set interval)Up to Solaris 9: create / start init script
/etc/init.d/rhnsd start
Solaris 10+: create SMF manifest23 (or use mine):https://github.com/stdevel/rhnsd-solmansvcadm validate|import rhnsd.xmlsvcadm enable rhnsd
# ps -ef|grep -i rhnsd
23Details: http://st-devel.net/blssvChristian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Basic setup and system maintenanceErrata for CentOSSolaris
Upload Solaris packages
Download .pkg packageConvert package in .mpm using solaris2mpm24
Upload file to Satellite / Spacewalk server using rhnpush
24Use --select-arch in case of errors!Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Kickstart automation
Enterprise Linux needs Kickstart distribution and profileKS distribution consists of a minimal boot environmentRequired files are stored on DVD or network mirrorsKS profile starts distribution + installationDisadvantage: manual work needed
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Kickstart automation
mkelfs can help you!Python tool for downloading needed files from networkmirrorsCan also create Kickstart distributionsSupports CentOS, Scientific Linux, FedoraDownload: https://github.com/stdevel/mkelfs
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Kickstart automationExamples
./mkelfs.py --release 6.5 --arch x86_64 -c
Downloads CentOS 6.5, x86_64, creates KS distributionFiles are stored in /var/satellite/kickstart_tree
./mkelfs.py -r 6.2 -a i386 -o scientific -fq
Downloads ScientificLinux 6.2, i386overwrites pre-existing files, no output
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Clean-up
All executed tasks are documented as actionsalso includes automated tasks (checking deployedconfigurations)!Additional researching often not required, deleting actionsmostly forgottenResult: database is full unneeded information!
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Clean-up
arsa can help you!Python tool for archiving / deleting actionsGood idea to run as weekly cronjobDownload: https://github.com/stdevel/arsa
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Clean-upExamples
./arsa.py -l
Lists completed actions (dry-run)
./arsa.py -rf
Archives completed and failed actionsDeletes archived actions afterwards
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Agenda
1 MotivationRequirements and necessitySpacewalk varietyNews
2 Installation & administrationBasic setup and system maintenanceErrata for CentOSSolaris
3 Tips & tricksKickstart automationClean-upPatch reporting
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Patch reporting
Management often requests detailed patch reportsMight be essential depending on the companiescertification (e.g. ISO/IEC 27001:2005)Very time-consuming task depending on system landscapeThere must be a way to automate this. . .
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Patch reporting
satprep can help you!Python toolkit for creating detailed patch reportsReports are created as PDF using TEXLists patch-relevant and also general system informationDownload: https://github.com/stdevel/satprep
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Patch reportingFunctionality
1 Creating a snapshot of relevant errata / patch information:./satprep_snapshot.py
2 Patching and rebooting systems3 Creating another snapshot: ./satprep_snapshot.py4 Calculating the delta and creating PDF reports:./satprep_diff.py 20140707*.csv
5 (Sign document and be happy about having saved time)
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Patch reportingFunctionality
Custom info keys defining meta information:SYSTEM_OWNER - System ownerSYSTEM_CLUSTER - Cluster node / standalone systemSYSTEM_MONITORING - monitoring stateSYSTEM_MONITORING_NOTES - notes about systemmonitoringSYSTEM_BACKUP - Backup state. . .
Christian Stankowic System management with Spacewalk
MotivationInstallation & administration
Tips & tricks
Kickstart automationClean-upPatch reporting
Patch reportingCustomization
Reports customization:Potrait / landscapeCompany logoSelecting particular system, patch and errata informationConventional TEXdocument is used as template
Christian Stankowic System management with Spacewalk
Appendix Further information
Further information I
http://fedorahosted.org/spacewalkSpacewalk wiki.
http://cefs.steve-meier.deCentOS Errata for Spacewalk.Steve Meier
http://red.ht/1mJA1q1Manage Solaris with Spacewalk and Red Hat SatelliteChristian Stankowic, Guest post in official Red Hat blog
http://www.freiesmagazin.deSpacewalk articlesChristian Stankowic, 08/2014 - xx/2014
Christian Stankowic System management with Spacewalk
Appendix Further information
Thank your for your attention!Questions / feedback?
Stay in touch:
Twitter: @stankowic_devel
Also check-out my blog for Spacewalk stuff:http://www.stankowic-development.net
Christian Stankowic System management with Spacewalk