zenpackers documentation - read the docs · 2019-04-02 · zenpackers documentation zenoss, inc....
TRANSCRIPT
ZenPackers Documentation
Zenoss, Inc.
Dec 10, 2018
Contents
1 Welcome to Zenoss 11.1 Setup Your GitHub Credentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Setup SSH Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 ZenPack Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Installing Zenoss 4.2.X . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.5 Upgrading Zenoss 4.2.X . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.6 Installing Zenoss 5.X . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.7 Europa 5.X Design: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.8 Monitoring Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.9 Other Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.10 Recommended Linux apps (Anonymous) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.11 Recommended MAC apps (Anonymous) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Zenpackers Documentation 72.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.3 How to Contribute to Zenpackers: Sphinx-Doc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.4 Important Security Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 Zenpack Troubleshooting and Triage 113.1 ZenPack Troubleshooting and Triage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4 Topics for Zenpack Administration 234.1 Commandline Monitoring For Slackers (ZP Developers) . . . . . . . . . . . . . . . . . . . . . . . . 234.2 Databases and Rabbits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.3 Attribute Errors and Other Oddities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.4 Dealing with Bad Relations in ZODB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304.5 RRD Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.6 Third Party Licensing Agreements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344.7 Jenkins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344.8 Zenpack Versioning Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.9 Zenpack Release Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.10 The MediaWiki Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.11 HotFix Release Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.12 L3 Support Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.13 Daemons & Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484.14 Zenwipe.sh: Erase Everything but Core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
i
4.15 Zendmd Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554.16 Parsing an Objects.xml file using JQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.17 Converting to the new Catalog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.18 RST Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5 General Zenpack Development 635.1 File Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.2 General Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.3 Event Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.4 Special Events and Their Consequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 885.5 Event Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 915.6 GUI Properties and JavaScript Related Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1005.7 Impact Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1075.8 DynamicVeiw Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1235.9 Modeling for Zenpacks: Selected Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1265.10 Collection: General Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1385.11 Datasources in Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1435.12 Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1455.13 ZenPython (PythonCollector) Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1455.14 ZenPack Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1475.15 Reserved Words in Zenoss Modeler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1485.16 Solr Catalogs (ModelIndex) Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
6 Special Zenpack Development Topics 1616.1 Analytics Bundle Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1616.2 Catalog Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1786.3 Migration Guide for Slackers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1816.4 Monkey Patching in ZenPacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1886.5 Templating Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1896.6 Unit Testing Rough Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2006.7 ZEP Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2086.8 Bread and Butter: A Quick Primer on Pickles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
7 Topics for ZenpackLib 2117.1 Welcome to ZenPack Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
8 Topics for ControlCenter (ControlPlane) 2298.1 Welcome to ControlCenter! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2298.2 Setup and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2328.3 Zenpack Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2408.4 Special Topics: ControlCenter and Others . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
9 Topics for OpenStack Group 2619.1 Welcome to the OpenStack Group! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2619.2 Setup and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
10 Pythonic Topics: 26310.1 Regular Expression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26310.2 Understanding Twisted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26410.3 Tab Completion in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26510.4 PDB Tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
11 Cisco General: 26711.1 Cisco Background Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
ii
12 Other Topics: 28512.1 Ansible Rough Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28512.2 Git for Gits ;) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28812.3 Git and Code Reviews: Keeping History Clean . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29512.4 Git-Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29612.5 Installing a VIM IDE setup with Synstastic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30312.6 DB2 Information for Scavengers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30412.7 Convenience Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31112.8 Simulate devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
iii
iv
CHAPTER 1
Welcome to Zenoss
Hail Might ZenPacker,
Welcome to Zenoss. I will be your website spirit-guide.
Here are the basics:
• Wunderground Austin Weather: http://www.wunderground.com/US/TX/Austin.html
• Austin Traffic: http://goo.gl/hlZvsE
• Google:
– Mail: http://mail.google.com (You can use pop/imap clients too)
– Calendar: https://www.google.com/calendar
• Slack Channels (We use it heavily):
– Solutions
– ZenPackers Lounge (ZenPack Technical Topics)
– Zia (General Technical Topics)
– Engineering (Platform Topics)
– Zenoss Austin
– Zenossians (Social)
– Zenpack Code Review
– Developer Code Review Requests
• IRC (Server, Channel)
– (freenode, #zenoss)
– (freenode, #python)
• Rally https://rally1.rallydev.com
• Jira: https://jira.zenoss.com
1
ZenPackers Documentation
• Github Repo: https://github.com/zenoss/
• Jenkins for Solutions: http://jenkins.zenosslabs.com
• Evernote: https://www.evernote.com/
1.1 Setup Your GitHub Credentials
Important: Make sure the one of the first things you do when you setup your development system is to setup yourgithub credentials and identifying username and email. This is to assist in tracking and documenting developmenthistory:
git config --global user.name "Jane Smith"git config --global user.email "[email protected]"
1.2 Setup SSH Keys
You probably want to setup SSH keys to authenticate to Github. If so:
1.2.1 Setup Your SSH Keys on Linux
You first need a key. Generate one as follows:
ssh-keygen -t rsa
If you have a passphrase you can setup Keychain as per
• https://wiki.gentoo.org/wiki/Keychain
1.2.2 Setup Your SSH Keys on OSX
First create an SSH key:
ssh-keygen -t rsa
To add your key to keychain (because you used a passphrase):
ssh-add -K ~/.ssh/id_rsa
1.3 ZenPack Development
• Start Here: http://zenpacklib.zenoss.com
• Next: http://docs.zenosslabs.com
• IDE’s (Integrated Development Environments):
– SublimeText (very popular): http://www.sublimetext.com/2
2 Chapter 1. Welcome to Zenoss
ZenPackers Documentation
– Intellij PyCharm (popular in Platform group): http://www.jetbrains.com/pycharm/download/
– Vim: https://github.com/scrooloose/syntastic
– Komodo: http://komodoide.com/
• Useful glossary of Zenoss-related terms. http://www.zenoss.com/documents/zenoss-dosss.pdf
• ZenPack Standards Guide: http://docs.zenosslabs.com/en/latest/zenpack_standards_guide.html
• ZenPack Development Guide: http://docs.zenosslabs.com/en/latest/zenpack_development/index.htmlStep-by-step guide to building a fairly complex ZenPack for modeling and monitoring a custom deviceusing SNMP.
1.4 Installing Zenoss 4.2.X
Several methods exist for installing Zenoss 4.2.X
• General: : http://wiki.zenoss.org/Install_Zenoss
• Debian: http://www.pietervanos.net/knowledge/debian-7-install-zennos-monitoring/
• Various Platforms: http://hydruid-blog.com/?p=710
1.5 Upgrading Zenoss 4.2.X
Zenup is a 4.X updating system for Zenoss.
• ZenUp: http://wiki.zenoss.org/ZenUp
1.6 Installing Zenoss 5.X
• http://wiki.zenoss.org/Zenoss_Core_5_Alpha/Download
• CP-Alpha: https://github.com/zenoss/serviced/wiki/Starting-CP-Alpha
• CP-Alpha: https://github.com/zenoss/serviced/wiki/_pages
• ZenDev: http://zenoss.github.io/zendev/
1.7 Europa 5.X Design:
• http://goo.gl/X0Mqx9
1.4. Installing Zenoss 4.2.X 3
ZenPackers Documentation
1.8 Monitoring Targets
This is a potentially useful page when looking for monitoring targets we may already have setup.
https://sites.google.com/a/zenoss.com/qa/home/qa-knowledge-base/qa-monitoring-targets
1.9 Other Documentation
• New Developers - START HERE: http://zenpacklib.zenoss.com
• Python Coding - Best Practices/Guidelines https://media.readthedocs.org/pdf/zenosslabs/latest/zenosslabs.pdf
• Older Docs: http://wiki.zenoss.org/ZenPack_Development_Guide
• Official Docs: https://docs.zenoss.com
• ZenPackers on ReadTheDocs: This document on http://zenpackers.readthedocs.org
• http://wiki.zenoss.org/ - End-user documentation for all the Solutions ZenPacks.
• Zenoss Corp Docs: https://intranet.zenoss.com
• Rally project management: https://rally1.rallydev.com/
• http://jira.zenoss.com/ - Jira defect tracking, make sure you can login
• http://support.zenoss.com/ - ZenDesk issue tracking for customers
1.10 Recommended Linux apps (Anonymous)
• Vim
• Sublime Text
• PyCharm: Python IDE
• Intellij: Java and Python
• Xen
• KVM
• VirtualBox
• HexChat: IRC
• WireShark
• ngrep
1.11 Recommended MAC apps (Anonymous)
• Skype
• Alfred (Quick Launcher)
• Sublime Text
• iTerm 2 (Terminal Emulator)
4 Chapter 1. Welcome to Zenoss
ZenPackers Documentation
• Homebrew (Software Installer)
• VirtualBox
• Colloquy (IRC Client)
• CoRD (RDP Client)
• Cyberduck (File Transfers)
• Dropbox (File Sharing and Synchronization)
• WireShark (Protocol Analyzer)
• Synergy (IP based KVM)
• Tunnelblick (OpenVPN)
• SizeUp (Screen manager)
1.11. Recommended MAC apps (Anonymous) 5
ZenPackers Documentation
6 Chapter 1. Welcome to Zenoss
CHAPTER 2
Zenpackers Documentation
2.1 Description
Zenpackers is a documentation source for advanced ZP developers that goes beyond what is documented in the existingonline documents. Its not meant to be a permanent storage but rather a staging ground for ZP development ideas thatcan be later moved to permanent documentation. The Sphinx technology we use is very easy to write in and has aneasy build target.
2.2 Motivation
This documentation is intended to augment the existing documentation to assist ZenPack developers. It is the hope ofthis project to reduce the amount of repeated support and questions related to ZP dev, facilitate development progress.
This project is located at https://github.com/zenoss/ZenPackers
Consider writing and transcribe your juicy morsels of knowledge into new documentation. We’d like to venture thateach of us can easily put in 30 minutes of documentation per iteration, to this end.
For veterans of ZP development, this can help reduce the amount of repeating of answers that could easily have awell documented solution. For newbies, it could help you to cement your ideas together and provide you with a gooddocumentation that can remind you 3 weeks later when you may have forgotten what you have done.
2.3 How to Contribute to Zenpackers: Sphinx-Doc
We use Sphinx to write these documents. Its easy to install and use in Linux and OSX. Sphinx uses Restructured Textformat, which can be found here:
• http://sphinx-doc.org/rest.html
We cover several OS setup scenarios below.
7
ZenPackers Documentation
2.3.1 Installing Sphinx-Doc
General documentation is at http://sphinx-doc.org/latest/install.html
In Debian/Ubuntu, you need (at minimum) these packages:
apt-get install python-sphinx make
In Centos or Redhat:
yum install python-sphinx.noarch make
In MacOSX:
brew install sphinx
For Windows, read the installation docs:
• http://sphinx-doc.org/latest/install.html
2.3.2 Downloading the ZenPackers Source from Git
Download the source here:
git clone [email protected]:zenoss/ZenPackers.gitcd ZenPackers
2.3.3 Installing and using Git-Flow to create a feature
• Install git-flow as per Git-Flow and https://github.com/petervanderdoes/gitflow-avh/wiki/Installation
In that same document we outline how to:
• Initialize Git flow on your repo
• Create a feature branch bases on develop
For example, from in the ZenPackers repo:
git flow initgit flow feature start JoesGarage
Now you can start making changes to the repo.
2.3.4 Building HTML or Other Targets for testing
Once you have made changes to the RST files, you can test it by generating HTML. In order to build the HTML yousimply do this:
cd Zenpackersmake html ; cp -a build/html /tmp/
You can view the HTML on the local system at /tmp/build/html. If you work remotely, you’ll have to copy those filesto your local system using the file tranfer tool of your choice.
8 Chapter 2. Zenpackers Documentation
ZenPackers Documentation
2.3.5 Submitting a Pull Request
Once you finish your changes, you need to commit and push up your feature changes as such:
git commit -am "Adding/Modifying a file for JoesGarage Zenpack"git push
Now go to the github repo page, and make a pull request:
• Navigate to https://github.com/zenoss/ZenPackers
• Select your feature in the pulldown
• Click on New pull request
2.3.6 Review a Pull Request
Once your pull request is in, someone can review it and merge it into the develop branch, using the GitHub webinterface.
This allows others to review and make change recommendations. You may have to make changes, commit, and pushchanges as above. You don’t have to make a new pull request.
2.4 Important Security Information
Make sure that you sanitize any document that may have proprietary or security related information. At minimum,this includes:
• IP addresses
• network Numbers
• usernames
• passwords
• personal information of any type
2.4. Important Security Information 9
ZenPackers Documentation
10 Chapter 2. Zenpackers Documentation
CHAPTER 3
Zenpack Troubleshooting and Triage
3.1 ZenPack Troubleshooting and Triage
3.1.1 General Instructions
Before asking for L3 help you should ensure that
1. You have read the official documenation for the ZP
2. This checklist is fully satisfied.
Target Checks
1. Ensure: Target is running
2. Ensure: No blocking firewall between Target and Modelers/Collectors
3. Ensure: No firewall running on Target
4. Ensure: Manual connect (telnet, SSH, SNMP, etc) to Target from Zenoss/Collectors.
Zenoss ZP Checks
1. Ensure that you have followed the Wiki documentation for setup faithfully.
2. Ensure that all zProperties are correct as per docs
3. Ensure that zPythonClass is correct for the device-class AND device
4. Ensure that your device is in the correct class.
5. Ensure you can ping all Targets or set zPingMonitorIgnore to True
6. Ensure access to SNMP Targets or set zSnmpMonitorIgnore to True
11
ZenPackers Documentation
Zenoss Logs Checklist
Make sure you at least have these logs on hand for the L3 Engineer (Some may not apply, but if they do, get them):
* ALL: zenoss xstatus &> zstatus.log
* ALL: zenpack --list &> /tmp/zenpacks.log
* Modeler: zenmodeler run -v10 -d $DEVICE &> /tmp/zenmo.log
* Collector: zencommand run -v10 -d $DEVICE &> /tmp/zenco.log
* ZenPython: zenpython run -v10 -d $DEVICE &> /tmp/zenpy.log
* ZenHub: zenhub run -v10 --workers 0 -d $DEVICE &> /tmp/zenhub.log
* SNMP: zenperfsnmp run -v10 --workers 0 -d $DEVICE &> /tmp/zenperfsnmp.log
Modeler Data: Getting Valid Data
Make sure the modeler data has actual model data in it.
• Make sure there are NO errors
• Make sure there are NO missing devices
• Make sure there are NO authentication problems
If you see anything like the following you have a missing device:
2017-06-27 14:50:20,706 INFO zen.ZenModeler: Device 10.1.1.11 not returned is it down?2017-06-27 14:50:20,706 INFO zen.ZenModeler: Device fluffydog not returned is it down?
Other problems appear if the target is missing a service:
2017-07-20 11:52:30,829 ERROR zen.PythonClient: nx5000: An error occurred during data→˓collection for cisco.netconf.NXPortChannels, Failed to open SSH subsystem: xmlagent2017-07-20 11:52:30,834 ERROR zen.PythonClient: nx5000: An error occurred during data→˓collection for cisco.netconf.NXIPAddresses, Failed to open SSH subsystem: xmlagent
Authentication Problems:
2017-07-21 11:30:16,138 ERROR zen.PythonClient: nx5000: A connection failure occurred→˓for cisco.netconf.Status, Authentication error for user: zenoss3234234. Check→˓zCommandUsername and zCommandPassword
SNMP Device Walk Data
Solutions needs all SNMP data in a particular format that allows us to diagnose the device with our tools:
snmpwalk -v2c -c public -ObentU $DEVICE_IP > /tmp/device_name.snmp
Depending on your device you may have to adjust the following items:
• -c public: This is the community string and may differer on your device*
• -v2c: This is the SNMP version. It can be (-v1, -v2c, -v3)
If Support is Still Needed. . .
You can save everyone time by requesting (in advance) data and access as follows:
• On behalf of the assigned L3 Engineer:
12 Chapter 3. Zenpack Troubleshooting and Triage
ZenPackers Documentation
– Gather or Create a full description of the network, environment, Targets.
– Provide a description of what has been configured, changed, and tried
– Arrange for GUI access to the system in question.
– For ZAAS: Open a ticket requesting access (For the L3 Engineer)
– For other customers: Ask the Client for SSH credentials in advance
– If a VPN is required, you should arrange for that ASAP! It can take alota time.
3.1.2 CiscoMonitor
CiscoMonitor covers a huge array of switches and routers. While some models are simple to configure with SNMP,others require SSH and NXAPI type of connection information.
There are three basic rules when configuring CiscoMonitor
• Read the official documentation
• Make sure the modeler (with -v10 verbosity) on your device has NO errors.
• Read the official documentation
Those docs can be found here:
• https://www.zenoss.com/product/zenpacks/cisco-devices
CiscoMonitor Checks
Make sure to check all the following
• SNMP1 and SNMPv2c:
– zSnmpCommunity
– zSnmpCommunities
– zSnmpPrivType
– zSnmpPrivPassword
• SNMP3 is configured correctly:
– zSnmpSecurityName
– zSnmpAuthType
– zSnmpPrivType
– zSnmpPrivPassword
• Netconf/SSH/NXAPI access:
– zCommandProtocol
– zCommandPort
– zCommandUsername
– zCommandPassword
• NXAPI access:
– zCommandUsername
3.1. ZenPack Troubleshooting and Triage 13
ZenPackers Documentation
– zCommandPassword
– zCiscoNXAPIUseSSL
– zCiscoNXAPIPort
3.1.3 DB2
Please make sure the following settings are in order prior to filing a ticket and requesting L3 support.
Database Checks
1. Ensure: Database is running
2. Ensure: User/Permissions are set according to documentation
3. Ensure: All tables/views exist according to documentation
4. Ensure: telnet to 1521 works.
Zenoss DB2 ZP Checks
1. Ensure: zDB2ConnectionStrings Matches the Device IP.
2. Ensure: zCommandCommandTimeout is set to 120 or more.
3. Ensure: Manual connect and query from zenoss collectors:
• (a) Set the following (triple-check that settings are correct):
export constr="jdbc:db2://mp3.zenoss.loc:50000/SAMPLE:user=zenoss;→˓password=zenoss;"export ZP_BASE=$ZENHOME/ZenPacks/ZenPacks.zenoss.DB2-X.X.X-py2.7-linux-→˓x86_64.egg/ZenPacks/zenoss/DB2export JARFILE=$ZP_BASE/lib/db2-jdbc-client-1.0-SNAPSHOT.jar
function db2(){
/usr/bin/java -Duser.timezone=UTC -jar "$JARFILE" -c "$constr" -q "$1"}
• (c) Test setup with the following query:
db2 'select * from sysibmadm.snapdb'
If this fails, you have a configuration/connection problem that must be fixed.
• (d) Test the queries manually:
– Modeler db2_instances query:
db2 'select inst_name,product_name from sysibmadm.env_inst_info,→˓sysibmadm.snapdbm'
– Modeler db2_database query:
14 Chapter 3. Zenpack Troubleshooting and Triage
ZenPackers Documentation
db2 "select * from (select dbpartitionnum,server_platform from sysibmadm.→˓snapdb),
(select npages bp_npages from syscat.bufferpools where bpname=→˓'IBMDEFAULTBP'),
(select value logbufsz from sysibmadm.dbcfg where name=→˓'logbufsz')"
– Tablespace Modeler query:
db2 "select a.tbsp_name,tbsp_type,tbsp_state,tbsp_used_size_kb,tbsp_free_size_kb,tbsp_usable_size_kb,tbsp_utilization_percent,tbsp_auto_resize_enabled,container_name,container_typefrom sysibmadm.snapcontainer ajoin sysibmadm.tbsp_utilization bon a.tbsp_name=b.tbsp_name"
• (d) Test the queries semi-manually:
– cd $ZP_BASE/lib
– vi txdb2jdbc.py
* Go to bottom of file near “if __name__ == ‘__main__’:”
* Modify the connectionString in Database() to yours
* fix the query in a_query to one of the above
– run this:
python txdb2jdbc.py
3.1.4 DatabaseMonitor
Please make sure the following settings are in order prior to filing a ticket and requesting L3 support.
Oracle Database Checks
1. Ensure: Oracle Database is running
2. Ensure: User/Permissions are set according to documentation on Oracle
3. Ensure: All tables/views exist according to documentation on Oracle
4. Ensure: telnet to 1521 works.
Zenoss DatabaseMonitor ZP Checks
1. For Zenoss 5.X and above, attach to a Zope container
2. Ensure: zOracleConnectionString Matches the Device IP.
3.1. ZenPack Troubleshooting and Triage 15
ZenPackers Documentation
3. Ensure: zCommandCommandTimeout is set to 120 or more.
4. Ensure: Manual connect and query from zenoss collectors:
• (a) Set the following in bash(double-check $constr and $ZP_BASE):
export constr="jdbc:oracle:thin:USERNAME/[email protected].→˓loc:1521:XE"export ZP_BASE=$ZENHOME/ZenPacks/ZenPacks.zenoss.DatabaseMonitor*/→˓ZenPacks/zenoss/DatabaseMonitorexport JARFILE=$ZP_BASE/lib/zenpack-ojdbc-client-1.0-SNAPSHOT.jarfunction ora(){
/usr/bin/java -Djava.security.egd=file:///dev/urandom \-Duser.timezone=UTC -jar "$JARFILE" -c "$constr" -q "$1"
}
# Test function to use CONNECTION_STRING via environmentexport CONNECTION_STRING="jdbc:oracle:thin:zenoss/[email protected].→˓loc:1521:XE"
function eora(){
/usr/bin/java -Djava.security.egd=file:///dev/urandom \-Duser.timezone=UTC \-jar "$JARFILE" \-q "$1"
}
• (b) Test the following query:
ora 'select * from v$instance'
If this fails, you have a connectivity problem that must be fixed.
• (c) Test the existence for special tablespace views:
ora 'select * from sys.vw_x$kccfn'ora 'select * from sys.vw_x$ktfthc'
5. If any of these tests fail, your Zenoss system is not properly configured. Please consult the documentation.
3.1.5 Windows ZenPack
Overview
• Methods of Monitoring
– WMI enumeration over winrm
– PowerShell using a remote shell over winrm
• Differences
– WMI queries through DCOM
* Mostly modeling. Service, IIS, Process, WinRMPing datasources
– Powershell for everything else
16 Chapter 3. Zenpack Troubleshooting and Triage
ZenPackers Documentation
Authentication
• Local Authentication
– Simple user/password sent over clear text
– Can be useful for tcpdump
• Domain Authentication (Kerberos)
– Encrypted
– Single user over multiple trusted domains
Kerberos Authentication
Process:
• Kinit asks for a krbtgt (kerberos ticket granting ticket) from the kdc
1. a.Sends user/pass to kdc for authentication
2. b.Any error messages you see that begin with kinit: only deal with authenticating to the kdc
• After successful login, we then create a context for the device to send our winrm messages
1. c.We request from the kdc a ticket to use the HTTP/HTTPS/WSMAN(2.7.0) service on the device
(a) Server not found in kerberos database
(b) Usually reverse DNS mismatch
(c) HTTPS is the only spn that would need to be added
(d) zWinRMKrb5DisableRDNS
2. b.Successfully get HTTP/HTTPS/WSMAN ticket
(a) i.Happily send/receive encrypted payloads to/from device
• Trusted realms
1. d.If an org has a trust between other domains, a single user can be used from the main domain
2. e.Set up zWinKDC, zWinRMServerName, zWinRMPassword normally
3. f.Use zWinTrustedRealm/zWinTrustedKDC to specify the domain and kdc for that domain
3.1. ZenPack Troubleshooting and Triage 17
ZenPackers Documentation
Example:
• example.com is the main domain
• austin.example.com is a trusted subdomain
– zWinKDC: dc01.example.com (recommend using ip for fewer dns lookups)
– zWinRMUserName: [email protected]
– zWinRMPassword: SomeCrazyPassword
– zWinTrustedRealm: austin.example.com
– zWinTrustedKDC: dc01.austin.example.com (or ip)
Miscellany
• Location is /opt/zenoss/var/krb5/krb5.conf
• Cache files are located in /opt/zenoss/var/krb5cc/user\@domain.com
• To verify user/pass, at a zenoss prompt:
– [zenoss]$ kinit [email protected]
* Note the domain in all caps
• To view cache for a user and verify we have a good ticket for the HTTP/HTTPS/WSMAN service:
– [zenoss]$ klist -c /opt/zenoss/var/krb5cc/user\@domain.com
• Extra include directory
– Default is /opt/zenoss/var/krb5/config
– zWinRMKrb5inlcludedir
– Can ONLY contain valid kerberos configuration files
– Any name you want for the file(s)
• Enter multiple kdcs for zWinKDC
– 10.10.10.10, 10.10.10.20, kdc.example.com
• What do various errors mean? And how to debug or solve?
– https://technet.microsoft.com/en-us/library/bb463167.aspx
– Check the wiki + comments section
– Server not found in kerberos database
* Usually reverse DNS mismatch
* HTTPS is the only spn that would need to be added
* zWinRMKrb5DisableRDNS
18 Chapter 3. Zenpack Troubleshooting and Triage
ZenPackers Documentation
Performance Counters
• Windows Performance Counters
– Uses get-counter PowerShell cmdlet
• MS SQL Performance Counters
– Pulls counters directly from SQL Server, i.e. sql SELECT statement
• Debugging:
– Counter corruption and how to deal with those?
* lodctr: https://technet.microsoft.com/en-us/library/bb490926.aspx
• Missing/corrupted counters are occasionally an issue on the monitoring target
– Can occur after installing software that includes its own counters, such as MS SQL Server
– Another possible cause - disabled in Registry
– May also need to investigate possible KB issues
• Example Resources:
– https://support.microsoft.com/en-us/help/2554336/how-to-manually-rebuild-performance-counters-for-windows-server-2008-64bit-or-windows-server-2008-r2-systems
– https://social.technet.microsoft.com/Forums/windowsserver/en-US/46a21b31-f5fc-4a44-bd4c-c9884a923943/performance-monitor-in-windows-server-2012?forum=winserver8gen
– https://support.solarwinds.com/Success_Center/Server_Application_Monitor_(SAM)/Performance_Counters_are_not_working_or_missing_in_PerfMon
– http://naftalyshprai.blogspot.com/2009/08/rebuild-procedure-for-corrupt-windows.html
• Different solutions exist depending on Windows version and other factors
• Debugging:
– Missing counters
* typeperf -qx
* PowerShell - no need to sift through list
· Get-Counter -listset process*
· Returns counter sets for Processor Information, Processor, Process
· (Get-Counter -listset objectname).pathswithinstances
· Shows all currently installed counters for specific object
– wbemtest
* Available from run command
* Connects to local windows host
· WQL queries
· CIM data
* Can also check existence and security settings using WMI Control in MMC
3.1. ZenPack Troubleshooting and Triage 19
ZenPackers Documentation
Services Monitoring
• Options for Services Monitoring
– ~130-200 service components per device
– Templates
* Making a change to DS can be expensive
– Windows Services Page
* Any change IS expensive
• How does the RegEx work?
• WinService template, Windows Service datasource
• Service State
– Based on Windows services status
– Running, Stopped, Paused
– state_state datapoint
• Service Status
– Based on zenoss events
Order of precedence for enabling/disabling monitoring
1. User manually sets monitoring
2. ‘DefaultService’ datasource from the template associated with the service
3. Datasource other than the DefaultService in the template associated with the service
4. Monitoring is enabled via the Infrastructure -> Windows Services page
Note: At least one Windows Service datasource must be enabled
20 Chapter 3. Zenpack Troubleshooting and Triage
ZenPackers Documentation
Event Log
• PowerShell
– Get-WinEvent for .NET >= 3.5
* Can use xml generated by MS in Event Viewer
* Returns EventLogRecord objects
* $$_.Id
– Get-EventLog for .NET < 3.5
* Slower, no xml query
* Returns EventLogEntry objects
* $$_.EventID
Performance / Scale
• What to check for performance/scale issues?
– Twisted uses file handles for open connections
– ulimit
• What to avoid?
– Very low cycle time for datasources
* Perfmon ok to have lower interval as the get-counter cmdlet runs over the entire zenpython cycle
Tracing & Logging
What logs are relevant for supporting Windows ZenPack?
• Modeling:
– zenmodeler and/or zenhub logs
• Monitoring:
– zenhub, zenpython logs
• GUI
– event.log
• Event Viewer:
Event Viewer-> Applications and Services Logs
-> Microsoft-> Windows
-> Windows Remote Management-> Analytic
– wevtutil: https://github.com/zenoss/microsoft.tools/blob/develop/scripts/zenoss-winrm-analytic-logs.ps1
– Shows requests received by Windows and what the responses are
3.1. ZenPack Troubleshooting and Triage 21
ZenPackers Documentation
Pitfalls / Limitations
• Encourage users to test windows service datasources in a subclass
• See wiki
• Installing 2.7.x can take a long time
References
https://www.zenoss.com/product/zenpacks/microsoft-windows
22 Chapter 3. Zenpack Troubleshooting and Triage
CHAPTER 4
Topics for Zenpack Administration
4.1 Commandline Monitoring For Slackers (ZP Developers)
4.1.1 Description
You can model, collect, and analyze everything from the commandline.
4.1.2 Prerequisites
• Zenoss ZenPack Developement
• Python 2.7
We assume that you are familiar with ZenPack developement and Python coding. We further assume that you have aconsole/terminal/xterm open on the system running Zenoss. The CLI prompt is [zenoss:~]:
4.1.3 Monitoring Data Collection
You can use these examples for zencommand:
• Just run it and see:
– zencommand run -d test-oracle-1.zenoss.loc
– zencommand run -d test-oracle-1.zenoss.loc -v 10
• Show the full command in addition:
– zencommand run -v10 –device=<dev> –show-full-command –show-raw-results
• Send it to tee to save output file:
– zencommand run -d test-oracle-1.zenoss.loc -v 10 |& tee /tmp/zenco.log
23
ZenPackers Documentation
4.1.4 Device Modeling
You can use these examples for zenmodeler (notice the –now flag):
• Just run it and see:
– zenmodeler run –now -d test-oracle-1.zenoss.loc
– zenmodeler run –now -d test-oracle-1.zenoss.loc -v 10
• Send it to tee to save output file:
– zenmodeler run –now -d test-oracle-1.zenoss.loc -v 10 |& tee /tmp/zenco.log
4.1.5 ZenDMD
ZenDMD can set and display real-time zenoss hierarchical properties. You really need to use it!
• Here is an example
[zenoss:~]: zendmdWelcome to the Zenoss dmd command shell!'dmd' is bound to the DataRoot. 'zhelp()' to get a list of commands.Use TAB-TAB to see a list of zendmd related commands.Tab completion also works for objects -- hit tab after an object name and '.'(eg dmd. + tab-key).
>>> device = find("test-oracle-1.zenoss.loc")>>> device.oracle_instances()[0]<Instance at /zport/dmd/Devices/Server/Linux/devices/test-oracle-1.zenoss.loc/oracle_→˓instances/orainst-XE>
>>> device.oracle_instances()[0].instanceRole'PRIMARY_INSTANCE'
>>> device.Devices.setZenProperty("zWinPerfCycleSeconds",300)>>> a=device.getProperty("zWinPerfCycleSeconds")>>> a300>>> device.zOracleUser'zenoss'>>> exit()
[zenoss@cdev:~/bin]:
4.2 Databases and Rabbits
This chapter covers various Zenoss database problems and how to cure them.
4.2.1 Corrupted Mysql Partition File
If you see this in your /opt/zenoss/log/zeneventserver.log file:
SQL state [HY000]; error code [1696]; Failed to read from the .par file;nested exception is java.sql.SQLException:Failed to read from the .par file at ..........
24 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
then there is a chance you have a corrupted Mysql database.
You may have to do the following to heal it:
• Stop Mysql
• Drop the zenoss_zep database (you may have to remove /var/lib/zenoss_zep)
• Start Mysql
• Recreate the zenoss_zep database:
[zenoss@monitor:~]: zeneventserver-create-db --dbtype=mysql
• Restart Zenoss
• Test
4.2.2 Corrupted RabbitMQ DB (Changed Hostnames?)
You see:
http://xyz.zenoss.loc:8080/zport/acl_users/cookieAuthHelper/login
If you find that you have broken RabbitMQ you may have some errors like this in your Event.log:
2013-10-03T13:16:24 ERROR Zope.SiteErrorLog 1380824184.320.0639042181339 \http://192.168.12.24:8080/zport/dmd/Devices/Server/Linux/devices/10.18.10.7/device_→˓routerTraceback (innermost last):Module ZPublisher.Publish, line 134, in publishModule Zope2.App.startup, line 301, in commitModule transaction._manager, line 89, in commitModule transaction._transaction, line 327, in commitModule transaction._transaction, line 397, in _callBeforeCommitHooksModule Products.ZenMessaging.queuemessaging.publisher, line 269, in→˓beforeCompletionHookModule Products.ZenMessaging.queuemessaging.publisher, line 429, in channelModule zenoss.protocols.amqp, line 90, in getChannelModule zenoss.protocols.amqp, line 47, in __init__Module amqplib.client_0_8.connection, line 144, in __init__Module amqplib.client_0_8.abstract_channel, line 95, in waitModule amqplib.client_0_8.connection, line 202, in _wait_methodModule amqplib.client_0_8.method_framing, line 221, in read_methoderror: [Errno 104] Connection reset by peer
This indicates that RabbitMQ’s internal setup has been corrupted. RabbitMQ does not have a simple configuration fileyou can tweak. It must be fixed by setting environment variables and then restarting RabbitMQ. Its been rumored thateven changing the hostname can make the Rabbit go insane. This is becuase RabbitMQ names the database foldersusing the hostname in part.
You may be able to fix this by applying the following (Thanks PC and JC):
• Stop Zenoss
• Execute the following:
sudo su - rootexport VHOST="/zenoss"export USER="zenoss"
(continues on next page)
4.2. Databases and Rabbits 25
ZenPackers Documentation
(continued from previous page)
export PASS="zenoss"rabbitmqctl stop_apprabbitmqctl resetrabbitmqctl start_apprabbitmqctl add_vhost "$VHOST"rabbitmqctl add_user "$USER" "$PASS"rabbitmqctl set_permissions -p "$VHOST" "$USER" '.*' '.*' '.*'exit
• Start Zenoss
There is a script that may help too:
#!/bin/bash
# 1. Stops Zenoss# 2. Removes all Zenoss RabbitMQ information and re-initializes the known vhosts# with the appropriate permissions for the Zenoss user.# 3. Starts Zenoss# Run as zenoss: The zenoss user must have full sudo access.# If you run as root, you (may) need to source ~zenoss/.profile
set +e
sudo service zenoss stop
ZENOSS_VHOST=`zenglobalconf -p amqpvhost 2>/dev/null`ZENOSS_USER=`zenglobalconf -p amqpuser 2>/dev/null`ZENOSS_PASS=`zenglobalconf -p amqppass 2>/dev/null`
set -e
ZENOSS_VHOST=${ZENOSS_VHOST:-"/zenoss"}# Add the zenoss-test VHOST (required by ZEP integration tests)VHOSTS="$ZENOSS_VHOST /zenoss-test"USER=${ZENOSS_USER:-"zenoss"}PASS=${ZENOSS_PASS:-"zenoss"}RABBITMQCTL="rabbitmqctl"if [ `uname -s` = "Linux" ]; then
RABBITMQCTL="/usr/sbin/$RABBITMQCTL"if [ `id -u` -ne 0 ]; then
RABBITMQCTL="sudo $RABBITMQCTL"fi
fi
$RABBITMQCTL stop_app$RABBITMQCTL reset$RABBITMQCTL start_app$RABBITMQCTL add_user "$USER" "$PASS"# Allow Zenoss user to use the management plug-in# This is only available on newer RabbitMQ versions$RABBITMQCTL set_user_tags "$USER" "administrator" > /dev/null 2>&1 || :for vhost in $VHOSTS; do
$RABBITMQCTL add_vhost "$vhost"$RABBITMQCTL set_permissions -p "$vhost" "$USER" '.*' '.*' '.*'
done
(continues on next page)
26 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
(continued from previous page)
sudo service zenoss start
You don’t need to stop zenoss services, as it does this.
4.2.3 Corrupted Lucene Index
This only affects 4.X upgrades via ZenUP and is rare. Newer Lucene index is required but older one is installed prior.
Diagnosis:
On new install you get this in zeneventserver.log:
2016-03-02T14:35:15.691 [main] ERRORorg.springframework.web.context.ContextLoader - Context initialization failedorg.springframework.beans.factory.BeanCreationException: Error creating beanwith name 'luceneSummaryIndexWriter' defined in class path resource[zep-config-daos.xml]: Instantiation of bean failed; nested exception isorg.springframework.beans.BeanInstantiationException: Could not instantiatebean class [org.apache.lucene.index.IndexWriter]: Constructor threw exception;nested exception is java.io.FileNotFoundException:/opt/zenoss/var/zeneventserver/index/summary/_f.si (No such file or directory)
In events.log, you might see this or similar:
Module Products.PageTemplates.Expressions, line 218, in evaluateStructureModule zope.tales.tales, line 696, in evaluate
- URL: /opt/zenoss/Products/ZenUI3/browser/templates/base-new.pt- Line 47, Column 8- Expression: <StringExpr u'jssnippets'>- Names:
{'args': (),'container': <DataRoot at /zport/dmd>,'context': <DataRoot at /zport/dmd>,'default': <object object at 0x7f2f249f3ae0>,'here': <DataRoot at /zport/dmd>,'loop': {},'nothing': None,'options': {},'repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0x6455100>,'request': <HTTPRequest, URL=http://mp4.zenoss.loc:8080/zport/dmd/
→˓itinfrastructure>,'root': <Application at >,'template': <Products.Five.browser.pagetemplatefile.ViewPageTemplateFile object
→˓at 0x5128b90>,'traverse_subpath': [],'user': <PropertiedUser 'zenoss'>,'view': <Products.Five.metaclass.ITInfrastructure object at 0x687a890>,'views': <Products.Five.browser.pagetemplatefile.ViewMapper object at 0x687afd0>
→˓}Module zope.contentprovider.tales, line 80, in __call__Module Products.ZenUI3.utils.javascript, line 28, in renderModule Products.ZenUI3.utils.javascript, line 28, in <genexpr>Module Products.ZenUI3.utils.javascript, line 40, in render
(continues on next page)
4.2. Databases and Rabbits 27
ZenPackers Documentation
(continued from previous page)
Module Products.ZenUI3.browser.eventconsole.grid, line 168, in snippetModule Products.ZenUI3.browser.eventconsole.grid, line 124, in column_configModule Products.ZenUI3.browser.eventconsole.grid, line 80, in _find_column_definitionsModule Products.Zuul.facades.zepfacade, line 761, in getUnmappedDetailsModule Products.Zuul.facades.zepfacade, line 931, in getUnmappedDetailsModule Products.Zuul.facades.zepfacade, line 889, in _initDetailsModule zenoss.protocols.services.zep, line 283, in getDetailsModule zenoss.protocols.services, line 181, in getModule zenoss.protocols.services, line 160, in _requestModule zenoss.protocols.services, line 154, in _executeRequestServiceResponseError: Service Unavailable
To fix this we have to remove the old index file and restart Zep:
• Stop zeneventserver
• Then do:
[zenoss@home]: cd /opt/zenoss/var/zeneventserver/index/[zenoss@home]: ls archive summary[zenoss@home]: rm -rf archive/ summary/[zenoss@home]: zeneventserver start
4.2.4 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZen-Events:
This indicates that your RabbitMQ server doesn’t have the correct permissions for your Zenoss setup. You can runthis scirpt as root to fix it:
#!/usr/bin/env bash
set -e
VHOSTS="/zenoss"USER="zenoss"PASS="grep amqppassword \$ZENHOME/etc/global.conf | awk '{print \$2}'"
if [ $(id -u) -eq 0 ]then
RABBITMQCTL=$(which rabbitmqctl)$RABBITMQCTL stop_app$RABBITMQCTL reset$RABBITMQCTL start_app$RABBITMQCTL add_user "$USER" "$(su -l zenoss -c "$PASS")"
for vhost in $VHOSTS; do$RABBITMQCTL add_vhost "$vhost"$RABBITMQCTL set_permissions -p "$vhost" "$USER" '.*' '.*' '.*'
doneexit 0
elseecho "Error: Run this script as the root user." >&2exit 1
fi
You may notice some similarities between this script and the others above.
28 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
4.3 Attribute Errors and Other Oddities
We discuss various problems with attribute errors. Most of these errors come from a mismatch between the Zenpacksoftware and the ZODB database that should already have the data structure in place, but for some reason does not.Make sure you re-start Zenoss when you install your zenpacks, and before adding any devices to the Zenpack classes.
4.3.1 Attribute Error After Installing a Device
If you get an “Attribute Error” for a Zenpack zProperty after installing a device to that Zenpack, you mayhave forgotten to restart Zenoss after installing the Zenpack. The class structure and attributes are not yetsetup in ZODB.
4.3.2 Attribute errors of the type “__of__”
If you get an error that specifies the “__of__” attribute, especially when starting, stopping, or installing aZP, it could be related to misfit objects in the ZODB.
AttributeErrors on the __of__ attribute almost always are an indication that there’s an object in ZODB (ofclass DatabaseMonitorDataSource in this case) for which there’s no longer a Python class file for in theexpected place in the file system.
The ZenPack appears to have that module/class. Something is out-of-sorts with the ZenPack. When thishappens you can install the ZenPack twice, then remove it. The reason to install twice is that oftentimesthe first time will fail, but it at least copies the code to the right place that allows the second installation towork.
Warning: It has also been reported that if an Egg file does not end in –py2.7.egg then it will throw an error of thistype.
4.3.3 You still have problem removing the device: Possible stale devices
If you still have errors removing the device with an error that refers to your ZP objects, similar to:
... lots of stuff before this ...ImportError: No module named xyz_implementationERROR:zen.ZenPackCmd:zenpack command failed
It may indicate that you have a stale object that failed to be removed. You might be able to
• Remove the bad device by going into zendmd and deleting it twice:
zenoss@mp2:/opt/zenoss/ZenPacks]: zendmd>>> device = find('mp8.osi')>>> device
<Endpoint at /zport/dmd/Devices/ABC/XYZ/devices/mp8.osi>>>> device.deleteDevice()
File "/zenpacks/ZenPacks.zenoss.XYZ/ZenPacks/zenoss/XYZ/catalogs.py",line 84, in get_xyz_core_catalog
from ZenPacks.zenoss.OpenStackInfrastructure.xyz_implementationimport all_core_componentsImportError: No module named xyz_implementation
(continues on next page)
4.3. Attribute Errors and Other Oddities 29
ZenPackers Documentation
(continued from previous page)
>>> device.deleteDevice()... more tracebacks ...
>>> commit()>>> exit()
• Now try to remote the ZP again:
zenpack --remove ZenPacks.zenoss.XYZ
• Now restart all Zenoss services
4.3.4 If all else Fails
If these trials fail you may have to:
• Somehow manually edit the ZoDB and remove the problem object
– See Dealing with Bad Relations in ZODB
• Use heavier methods like zenwipe.sh (4.X only)
• Restore your system to a know good snapshot
4.4 Dealing with Bad Relations in ZODB
If you have a relationship that is damaged and prevents you from deleting or otherwise managing your Zenoss systemyou have few recourses.
Most of these errors come from a mismatch between the Zenpack software and the ZODB database.
4.4.1 Scenario I: Re-create the relationship in your ZenPack, then delete it.
If your forgot to remove the related component from your device, you may be able to do the following:
• Re-add the relationship to your Zenpack (zenpack.yaml or old-style)
• Restart base Zenoss Services (zenhub, zope, etc)
• Start: zendmd
• for each component in badcomponents: - component.buildRelations() - de-vice.components._delObject(component.id) - commit()
• Re-remove the relationship from your Zenpack
• Re-start services and remodel
4.4.2 Scenario II: You found an incorrect relationship that you can’t explain
Removing a bad relationship from a component is not a simple thing to fix.
1. Find bad components and bad relationship:
2. Remove/filter the objects from the _objects
3. Remove the component.attribute
30 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
4. Delete the device (if you need to)
5. Commit your changes with commit()
6. Exit from zendmd
Exit zendmd and clean Zodb, ZenRelations and Catalogs
Now you need to
• zenrelationscan -f # Check and cleanup relations
• zenossdbpack # (Cleans up deleted objects)
• zencatalogscan -f # Check and cleanup catalogs
Example: You have a bad relationship called ‘hostedPlace’ on Instance:
device = find('osi.911')
for h in device.getDeviceComponents(type='OpenStackInfrastructureInstance'):if hasattr(h, 'hostedPlace'):
h._objects = tuple([x for x in h._objects if x['id'] !='hostedPlace'])delattr(h, 'hostedPlace')
>>> commit()>>> device.deleteDevice()>>> exit()
# zenrelationscan -f# zenossdbpack# zencatalogscan -f
4.5 RRD Topics
We use RRD to store monitoring data. The data is stored in the system root:
RRD_HOME=/opt/zenoss/perf/Devices
4.5.1 Using RRDtool to Query the Data
If you want to check or verify data stored in RRD, simply navigage to $RRD_HOME and look for your data file:
find . -name "*.rrd"find . -name "*my_special_name*"
RRDtool Info
Once you find the correct file you can query it with rrdtool . First use the info method:
rrdtool info path/to/my/file.rrd
You should get something that looks like the following:
4.5. RRD Topics 31
ZenPackers Documentation
filename = "./oracle_instances/orainst-XE/oracle_tablespaces/XE_USERS/TableSpaceDU_→˓bytes_used.rrd"rrd_version = "0003"step = 300last_update = 1386705130header_size = 1832ds[ds0].index = 0ds[ds0].type = "GAUGE"ds[ds0].minimal_heartbeat = 900ds[ds0].min = NaNds[ds0].max = NaNds[ds0].last_ds = "104857600.0"ds[ds0].value = 1.3631488000e+10ds[ds0].unknown_sec = 0rra[0].cf = "AVERAGE"rra[0].rows = 600rra[0].cur_row = 347rra[0].pdp_per_row = 1rra[0].xff = 5.0000000000e-01rra[0].cdp_prep[0].value = NaNrra[0].cdp_prep[0].unknown_datapoints = 0rra[1].cf = "AVERAGE"rra[1].rows = 600rra[1].cur_row = 394rra[1].pdp_per_row = 6rra[1].xff = 5.0000000000e-01rra[1].cdp_prep[0].value = 4.1943040000e+08rra[1].cdp_prep[0].unknown_datapoints = 0rra[2].cf = "AVERAGE"rra[2].rows = 600rra[2].cur_row = 150rra[2].pdp_per_row = 24rra[2].xff = 5.0000000000e-01rra[2].cdp_prep[0].value = 1.1534336000e+09rra[2].cdp_prep[0].unknown_datapoints = 11rra[3].cf = "AVERAGE"rra[3].rows = 600rra[3].cur_row = 537rra[3].pdp_per_row = 288rra[3].xff = 5.0000000000e-01rra[3].cdp_prep[0].value = 1.4470348800e+10rra[3].cdp_prep[0].unknown_datapoints = 100rra[4].cf = "MAX"rra[4].rows = 600rra[4].cur_row = 515rra[4].pdp_per_row = 6rra[4].xff = 5.0000000000e-01rra[4].cdp_prep[0].value = 1.0485760000e+08rra[4].cdp_prep[0].unknown_datapoints = 0rra[5].cf = "MAX"rra[5].rows = 600rra[5].cur_row = 517rra[5].pdp_per_row = 24rra[5].xff = 5.0000000000e-01rra[5].cdp_prep[0].value = 1.0485760000e+08rra[5].cdp_prep[0].unknown_datapoints = 11rra[6].cf = "MAX"
(continues on next page)
32 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
(continued from previous page)
rra[6].rows = 600rra[6].cur_row = 206rra[6].pdp_per_row = 288rra[6].xff = 5.0000000000e-01rra[6].cdp_prep[0].value = 1.0485760000e+08rra[6].cdp_prep[0].unknown_datapoints = 100
RRDtool Dump
Now you can dump your data in XML format to see the entire dataset:
rrdtool dump path/to/my/file.rrd
You should get something that looks like the following:
<?xml version="1.0" encoding="utf-8"?><!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd"><!-- Round Robin Database Dump --><rrd>
<version>0003</version><step>300</step> <!-- Seconds --><lastupdate>1386704280</lastupdate> <!-- 2013-12-10 13:38:00 CST -->
<ds><name> ds0 </name><type> GAUGE </type><minimal_heartbeat>900</minimal_heartbeat><min>NaN</min><max>NaN</max>
<!-- PDP Status --><last_ds>104857600.0</last_ds><value>1.8874368000e+10</value><unknown_sec> 0 </unknown_sec>
</ds>
<!-- Round Robin Archives --><rra><cf>AVERAGE</cf><pdp_per_row>1</pdp_per_row> <!-- 300 seconds -->
<params><xff>5.0000000000e-01</xff>
</params><cdp_prep>
<ds><primary_value>1.0485760000e+08</primary_value><secondary_value>1.0485760000e+08</secondary_value><value>NaN</value><unknown_datapoints>0</unknown_datapoints>
</ds></cdp_prep><database>
<!-- 2013-12-08 11:40:00 CST / 1386524400 --> <row><v>1.0485760000e+08</v></row><!-- 2013-12-08 11:45:00 CST / 1386524700 --> <row><v>1.0485760000e+08</v></row>
(continues on next page)
4.5. RRD Topics 33
ZenPackers Documentation
(continued from previous page)
<!-- 2013-12-08 11:50:00 CST / 1386525000 --> <row><v>1.0485760000e+08</v></row><!-- 2013-12-08 11:55:00 CST / 1386525300 --> <row><v>1.0485760000e+08</v></row><!-- 2013-12-08 12:00:00 CST / 1386525600 --> <row><v>1.0485760000e+08</v></row><!-- 2013-12-08 12:05:00 CST / 1386525900 --> <row><v>1.0485760000e+08</v></row><!-- 2013-12-08 12:10:00 CST / 1386526200 --> <row><v>1.0485760000e+08</v></row><!-- 2013-12-08 12:15:00 CST / 1386526500 --> <row><v>1.0485760000e+08</v></row>.......
</database></rra>
</rrd>
4.6 Third Party Licensing Agreements
This chapter covers the process for a Third Party License Agreement that is required when you include non-free codein your Zenpack.
This process requires that you have a decent understanding of the licensing terms for the proprietary software you areincluding in the Zenpack.
4.6.1 Procedure for Adding Third Party Code to Zenoss Core/Enterprise
• First get the Third Party Code Checklist from GoogleDocs:
https://docs.google.com/document/d/1j_n2bbFyNNMWjGSc8hJZFzAA4z50Dn4VSusHVfHGv4Qhttps://sites.google.com/a/zenoss.com/engineering/home/getting-started/third-party-code
• Fill out only the “Engineering” portions of that form.
• Place the semi-completed form in a new GoogleDocs document.
• Share that document with the appropriate contacts.
• Email our Legal staff and include that link from the prior step.
• Our Legal department is located at:
Zenoss Legal [email protected]
4.7 Jenkins
Jenkins is a automated build system.
4.7.1 General Points and Goals
Jenkins does a number of automatic ZenPack building tasks. We currently have 2 Jenkins servers:
• http://solutions-jenkins.zenoss.eng (Solutions)
• jenkins.zenoss.com
34 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
4.7.2 Task List
Jenkins task list is the entry point. It allows you to search for specific builds. If you search these
• Master: Builds latest release
• Develop: Builds latest develop branch
• Request: Pull Requests: Feature -> Develop merge build tests
4.7.3 Zenpack Workflow
• Make your zenpack in Github with a Develop branch
• Install and Enable Git Flow
• Push your development branch up to Github
• Now go up to Github and create a pull request
• Someone else will review and merge
• See the documentation on workflow at Git-Flow and Git for Gits ;)
4.7.4 Go to Jenkins and Probe Build
• Jenkins has a 5 minute delay
• You can force build (Build Now)
• Jenkins will automatically increment the revision number and commit hash
4.7.5 Releasing
• Once devel is stable you can release: (See Zenpack Release Notes)
• If working on a hotfix, see HotFix Release Notes
4.7.6 New Builds
• ssh to the server jenkins.zenosslabs.com
• Ask someone for credentials
• add-zenpack-jobs -o zenoss -t <very_long_hash_key>:
– Find the <very_long_hash_key> in the crontab entry
– Wait that script start prompting for unselected ZenPacks. You will see:
What does [Syaunq?] mean?s) Skip to next job. Don't do anything.y) Add or update Jenkins job.a) Add or update and always do the same in future.u) Update Jenkins job only if it already exists.n) Skip and never ask about this job again.q) Quit.?) This help."
4.7. Jenkins 35
ZenPackers Documentation
– Answer ‘s’ until you see yours, then ‘a’ and ‘q’
4.7.7 Tests on Builds
• Unit tests are not yet available
• Jenkins will spin-up a 4.2.4 Zenoss and compile the egg
• All Java artifacts are imported each time
• Includes: See the Manifest.in file
4.8 Zenpack Versioning Guidelines
This guide covers ZenPack version and revision numbering.
4.8.1 Version Suffix
The version suffixes that have changed to better to conform to setuptools version parsing and PEP 440. Here’s howthe auto-suffixing now works:
1. Base version is read from setup.py’s VERSION. 1.2.3dev becomes 1.2.3 (just the release numbers are extracted)
2. Suffix is automatically determined. See below.
3. Suffix is added to base version.
How is suffix determined? It’s based on the branch type:
• develop: .dev<commits>+g<ref>
• release/*: rc<commits>+g<ref>
• hotfix/*: rc<commits>+g<ref>
• master: no suffix
• support/*: .post<commits>
• feature/*: .dev<commits>+g<ref>.<normalized-feature-name>
The <commits> is determined by running and extracting the second part of:
git describe --tags --always --long $BRANCH
The <ref> is determined by extracting the third part of the same command.
The <normalized-feature-name> is extracted from everything after feature/ in the branch name, and with all nonalphanumeric characters converted to periods.
This should be completely compatible with the way we’re currently doing things, even if means that there’s somereordering to be aware of.
Another implication of this is that it’s no longer necessary to even put the “dev” suffix into setup.py’s VERSION.
The reason for these changes is to support the platform teams efforts in product-assembly which will be part of thenew build system that assembles products from artifacts rather than attempting to build the whole monolith at once.There will be a manifest describing which version of each ZenPack to include in each product assembly. We want thatmanifest to be able to say things like:
36 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
ZenPacks.zenoss.PythonCollector>=1.8,<1.9
so we don’t have to remember to update multiple product manifests every time we make a patch to a ZenPack.
4.8.2 Version Numbering Guidelines
To get this kind of manifest expressiveness mentioned above, we need to make sure our versions order correctly. Wealso need to have discipline about when we’re fixing bugs vs. when we’re changing behavior.
The version format is:
<major>.<minor>.<patch>
The rules are:
• Changing behavior in a non-backwards compatible way? Increment major.
• Changing behavior in a backwards-compatible way? Increment minor.
• Fixing a bug that doesn’t change behavior? Increment patch.
Note: The key points here are:
• When working on ANY branch, you should set VERSION in setup.py to the next version you expect to bereleased from that branch without any suffix. Jenkins will handle the suffix.
• So the same conventions we’ve been following all still apply. Bump the patch version when starting a hotfixbranch. Bump the minor or major version after finishing a release branch.
References:
• See the git_version_suffix() definition in https://github.com/zenoss/salt-data/blob/master/salt/zenosslabs/jenkins/fabfile.py
4.9 Zenpack Release Notes
We have a formal process for releasing Zenpacks. The steps are:
• 3rd Party Software approval
• Ensure an appropriate copyright is included in each source file
• Process the Repo through Git-Flow
• Build the Master Egg on Jenkins
• Inform the Release Team
• Publish the Documentation on Wiki
Warning: Do not cut-and-paste from the HTML of this document. The HTML contains control charactersthat will corrupt your git commands! Instead you can use the source from github at https://github.com/zenoss/zenpackers
4.9. Zenpack Release Notes 37
ZenPackers Documentation
Warning: Hotfix releases need to be tested and approved by QA
4.9.1 3rd Party Software Approval
This is covered in Third Party Licensing Agreements
4.9.2 Appropriate copyright for each Source File
Something similar to this should be in each sourcecode file:
############################################################################## Copyright (C) Zenoss, Inc. 2014 all rights reserved.## This content is made available according to terms specified in# License.zenoss under the directory where your Zenoss product is installed.#############################################################################
Following command prints list of non-empty (size > 1 byte to skip files with just one line-break symbol) Js, Java,Shell, and Python files without string Copyright (C) Zenoss, Inc in the current directory:
# find . -size +1 -regex ".*\.\(js\|java\|sh\|py\)" -exec grep -i -L 'Copyright (C)→˓Zenoss, Inc' {} \;
4.9.3 Process Repo through Git-Flow
We use Git Flow to release with some custom conventions. Refer to Installing Git-Flow for details.
Set the Version
First pick a version: We recommend using a format: Major.Minor.Revision, where Major, Minor, Revision are digits.We typically starting at 1.0.0. The first digit is the major version, the second is the minor version, and the third is therevision.
• We set this version in the $ZP_HOME/setup.py file.
For a revision, maintenance, or a hotfix, we increment the Revision number for master and Minor number for develop:
export OLD=3.0.7 # The previous Master releaseexport CURRENT=3.0.8 # The current Develop -> new Master releaseexport NEW=3.1.0 # The new develop after release, if not already!
For a Minor release, we bump the minor number for master and develop. For example:
export OLD=3.0.7 # The previous Master releaseexport CURRENT=3.1.0 # The current Develop -> new Master releaseexport NEW=3.2.0 # The new Develop branch after release; bump Minor
For a Major release, we bump the major number master, and minor for develp. For example:
38 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
export OLD=3.0.7 # The previous Master releaseexport CURRENT=4.0.0 # The current Develop -> new Master releaseexport NEW=4.1.0 # The new Develop branch after release
Start the Release Process
Starting from a clean develop branch:
git checkout developgit status(ensure develop is clean)
• Start the Release:
git flow release start $CURRENT
• Update the documentation (catalog, wiki, etc. . . ) and associated docs:
– Make sure to update essential functionality that has changed
– Make sure to update the Changes section
– Include any relevant fixes
• Edit setup.py (set the correct version numbers):
– Typically you just remove the “dev” in the version number if any
* Note: we no longer use the “dev” extension which is deprecated
* This means you probably have to do NOTHING.
– Ensure that the version is the $CURRENT value
• Commit:
– For new release:
git commit -a -m "release: version $CURRENT"
– For update release:
git commit -a -m "release: version ${OLD} -> ${CURRENT}"
• Finish the release:
git flow release finish $CURRENT
– You will be prompted for the *Commit String*, enter:
– “tag $CURRENT”
– You will automatically pushed back into develop
• Update develop’s setup.py: Bump number:
– Example: $CURRENT -> ${NEW}
• Commit again:
git commit -a -m "post release: $CURRENT -> ${NEW}"
4.9. Zenpack Release Notes 39
ZenPackers Documentation
• Push and tag the revision with a crpyto-secure key for reference:
git pushgit push --tags
• Finally, push up the master changes:
git checkout mastergit pushgit checkout develop
4.9.4 Publish the ZenPack Egg
OpenSource Zenpacks
For OpenSource zenpacks, our MediaWiki should take care of grabbing the source and building the Egg. This onlyhappens with the right tag:
• Go into the MediaWiki Page for your ZP
• Log in (if not already)
• Add the Git Tag you created above in the release phase. (cf: $CURRENT)
• MediaWiki should grab the source, build it, and post the link on the ZP page.
– If the link does not appear, you may have to upload it directly to the wiki.
– Look at wiki notes in Slack if this is the case.
– Make sure to omit the -py2.7 extension in the name
• Ensure that the correct “Source URI” is listed, usually the Git-Clone URL.
Commercial Zenpacks
• For commercial ZenPacks, we send the Egg to our release team.
• Go to the master branch on Jenkins and build/download it. The link will look like:
http://jenkins.zenosslabs.com/job/master-ZenPacks.zenoss.XYZ/
• Ensure that the Egg is valid and contains all correct components. This is espeicially true of the ZP has extrasoftware artifacts that are built by the GNUMakefile. This may require that installing and smoke-testing the ZP.
Email the ZP release team as follows:
You need to let us know that you have a released a ZenPack. This is done via email as follows:
• Send to: [email protected]
• If a commercial ZP, Include the egg as an attachement
• Include summary, list of dependencies, and changes in the email body. For example:
40 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
Summary:
The OpenXYZ (ZenPacks.zenoss.OpenXYZ) ZenPack has beenupdated to 1.7.0 and is attached. This is a feature andfix release that accumulates changes made to support ABC v8.
Dependencies
* Zenoss >= 4.2 (no change)
* ZenPacks.zenoss.PythonCollector >= 1.7 (no change)
Changes
* Show servers by type chart on overview. (ZPS-78730)
* Remove unused "Software" from Central view. (ZPS-78731)
* Reduce number of XYZ Central API sessions used. (ZPS-78732)
* Add API support for launch-in-context from XYZ Central UI.
- James [email protected]
4.9.5 Publish the Documentation
Update the appropriate documentation. At the very least:
• Update your docs as per your corrections in section Start the Release Process :
– Make sure to update essential documentation that has changed
– Make sure to update the Release date and data!
– Make sure to update the Changes section
Tip: When you’re creating a new ZP, at the bottom of the page on the left had side, there’s a little menu. The lastmenu option says “Publishing options”. If you click on that and then deselect the check box, you can put all of theZP info in and click save. That will save your work but will not make the ZP publish. After you hit save, you will beshown what the completed page will look like. Once you’re satisfied, you can go back in and reselect the “Published”checkbox.
4.9.6 Clean Git Repo Branches
Having abandon or inactive Git repos can lead to unintended merges or history rewrites. Once you finish with a featureor hotfix branch, remove them.
• Remove the feature branch as soon as you merge it.
• Remove any old and unused Git feature/hotfix branches.
• Make sure that feature/hotfix branches are merged prior to removal.
• Make sure that unfamiliar feature/hotfix branchs are not active prior to removal.
4.9. Zenpack Release Notes 41
ZenPackers Documentation
4.9.7 Stage Develop to the Next Major/Minor Release
Since we use Hotfixes to push out maintenance releases, make sure you bump develop to either the next Major orMinor release as per required
• Note again that we use the 3 digit versioning: Major.Minor.Revision
• Assuming you release 3.0.8:
export OLD=3.0.7 # The previous Master releaseexport CURRENT=3.0.8 # The current Develop -> new Master releaseexport NEW=3.1.0 # The new develop after release, if not already!
4.9.8 Quick Fixes to Documentation
• Tue Apr 4 2017
If you have already pushed out a release and you need to make a subsequent fix to the docs, you’ll need to make thatchange to both master and develop. The general idea is to make change in master and then merge that change intodevelop:
• Checkout master:
– Make your changes to docs.
– Commit changes to master
• Checkout develop:
– git merge –squash master
– git commit -a
• Once you are convinced that your changes are correct, push up both branches to origin.
Note: Make sure Jenkins is happy once the new master is built. There is a chance that this change will cause Jenkinsto fail. Ensure that this URL load properly:
• http://zenpacks.zenosslabs.com/requirement/ZenPacks.zenoss.YourZenPack
If it fails, you must log into jenkins.zenosslabs.com and restart gunicorn:
systemctl restart gunicorn
4.10 The MediaWiki Site
Warning: Tue Apr 1, 2017:
The Mediawiki site has be deprecated in favor of another documentation source.
DO NOT USE WIKI.ZENOSS.ORG ANY LONGER!
4.10.1 Creating an Account
• First go to http://wiki.zenoss.org and click on Create Account
42 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
– Go through all the steps, and validation.
• Afterwards, request that a wiki administrator add you to the editors list
4.10.2 Adding a New Wiki Page (Deprecated)
• Go to http://wiki.zenoss.org/ZenPack_Catalog
• Search for “Add a new” in the Publishing ZenPacks section
• Fill in the various information
4.10.3 Adding a New Release Form (Deprecated)
On the wiki page for your ZP, go to the Create/Edit Releases area. Click on the ‘+’ button to add a new release section.(See also the notes on Publish the ZenPack Egg)
To summarize,
• Set the version and Git Tag/SHA1: as the release tag.
• Enter the release date
• Enter any magjor changes in bullet format
• Select the Compatible entries
• Skip the Incompatibles
• Enter any Requires items (Use known links if possible)
• Download URL: For commercial ZP’s only
– For an OpenSource ZP, your SHA1/Tag (above) must match that of your git release so that MediaWikican auto-identify and build the target.
– For proprietery ZP, you have to link it with our https://zenoss.leapfile.net/
4.10.4 Editing an existing ZenPack Page
Always use the [Edit with Form] link at the top, and the associated text box at the bottom. Otherwise you’ll endup changing meta-data that is set by the form part.
Note:
• In general, your ZenPack should have a README.mediawiki file in the root folder that is the exact content ofthat text box. This way we have a backup and a way that developers and QA can review your docs.
• Typically, the README.mediawiki goes through the same type of review process as all other code.
4.11 HotFix Release Notes
Generated: Dec 10, 2018
Our hotfix process is used now on all ZenPacks. It requires that all maintenance revisions be made into a hotfix branchfirst, and then merged into both master and develop branches. This process requires that:
4.11. HotFix Release Notes 43
ZenPackers Documentation
• We use a 3 digit versioning system: Major.Minor.Revision, EX: 2.4.5
• Develop branch already differs from master by a Major or Minor number
Note: You should be familiar with the details of Zenpack Release Notes if you have any questions concerning releasedetails. In particular see the publishing notes at Publish the ZenPack Egg for post-egg creation steps.
Warning: Do not release a hotfix until QA has tested it from the Jenkins branch. Make sure you have included alink to the hotfix egg in the Jira fix commentary.
4.11.1 Preliminaries
First lets get all the version numbers in order. We’ll make bash variables for these.
• Assume that develop is already incremented by a Major or Minor revision.
• We use the following bash variables to reference our version numbers:
export OLD_MASTER=2.2.0 export NEW_MASTER=2.2.1 export DEVELOP=2.3.0
4.11.2 Create the hotfix for the next patch release if it does not exit
You should, in general, always have a hotfix branch created that is incremented one Revision number. Once you finishthis process, go ahead and make the new one as well with the appropriate number.
If you don’t already have one:
1. git checkout master
2. git pull
3. git flow hotfix start ${NEW_MASTER}
4. Edit setup.py: change VERSION ${OLD_MASTER} -> ${NEW_MASTER}
5. git commit -am “Start ${NEW_MASTER} Hotfix: Version ${OLD_MASTER} -> ${NEW_MASTER}”
6. git flow hotfix publish
4.11.3 Commit Changes to Hotfix:
Here follow a normal defect-fix process but adapted to be based on the hotfix. Once defects are made and hotfix isfinished, hotfix will get merged into both master and develop branches. Details ensue:
1. Make and commit changes needed in ${NEW_MASTER} hotfix
(a) git flow feature start ZPS-XXX hotfix/${NEW_MASTER}
(b) git flow feature publish
(c) work, work, commit & push
(d) make pull request from feature/ZPS-XXX to hotfix/${NEW_MASTER}
(e) Ensure small commits are squashed: :ref:‘squashing_multiple_commits‘
(f) review and merge
44 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
(g) git checkout hotfix/${NEW_MASTER}
(h) git pull
(i) repeat for all other existing issues
2. Make sure to clean up all features branches that are merged: Clean Git Repo Branches
4.11.4 Finish hotfix when all fixes are made:
Now all hotfix changes are made and you are ready to merge into master (and implicitly, develop).
Warning: QA must test and approve the hotfix branch before you perform the folloing steps. Make sure you haveapproval before releasing.
1. UNIT TESTS MUST PASS
2. QA has tested and approved all defects, or you have higher approval
3. Update Release section
4. Update Changes section
5. git commit -am “Finish ${NEW_MASTER} Hotfix: Version ${OLD_MASTER} -> ${NEW_MASTER}”
6. git push
7. git flow hotfix finish ${NEW_MASTER}
• When asked for the tag, use:
tag ${NEW_MASTER}
8. On the entry page specify: tag ${NEW_MASTER}
9. You will be automatically merged into develop
10. More than likely you will see a merge conflict with setup.py on the develop branch
(a) fix the version in setup.py to match ${DEVELOP}
(b) git add setup.py
(c) git commit -a # Remove any conflict log entry, assuming you resolved it
(d) git push
(e) fix any other conflicts, commit, and push
(f) git flow hotfix finish ${NEW_MASTER}
11. git push origin master develop
12. git push –tags
4.11.5 Sanity Check
You should now be in the develop branch. Make sure all branch logs look like what you expect them to be. At the veryleast make sure that:
• git status on all branches is clean
• master logs have the hotfix changes
4.11. HotFix Release Notes 45
ZenPackers Documentation
• develop logs have the hotfix changes
• The master and develop diff has no hotfix changes:
git diff master..develop
Here are some other (undocumented) commands that may be useful:
• git status
• git log –all -p –graph
• git diff master –stat
• git checkout master
• git log –graph
• git checkout - # (returns to develop branch)
4.11.6 Build the Master on Jenkins
Now run a build of the master-ZenPacks.zenoss.X Jenkins job if it hasn’t already run it, and verify that the artifact hasthe new ${NEW_MASTER} version. See Publish the ZenPack Egg for further steps.
4.11.7 Workflow: Easy CLI Pull Requests on a Hotfix Branch
If you have hub from https://github.com/github/hub and hub is aliased to git, you can do the following in the CLI tomake pull requests:
git flow hotfix start 1.2.3git flow hotfix publish 1.2.3git flow feature start ZPS-54321 hotfix/1.2.3git flow feature publish ZPS-54321# work work work commit & pushgit pull-request -o -b hotfix/1.2.3
4.11.8 See this Document Without Variables
To see this document without the variables execute the following:
OLD_MASTER=2.2.0NEW_MASTER=2.2.1DEVELOP=2.3.0
wget -O /tmp/hotfix_release.txt \https://raw.githubusercontent.com/zenoss/ZenPackers/develop/hotfix_release.rst
sed "s/\${DEVELOP}/${DEVELOP}/g; \s/\${OLD_MASTER}/${OLD_MASTER}/g; \s/\${NEW_MASTER}/${NEW_MASTER}/g" \/tmp/hotfix_release.txt
46 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
4.12 L3 Support Process
Solutions provides Level-3 (L3) support to our customers via the Zendesk system.
• https://zenoss.zendesk.com
L3 issues require us to provide either a develop release for trial, a hotfix for master, or some other form of relief forour customers. We outline here the basic process here.
4.12.1 General
• Try to get as much information as possible about the problem.
– Ask our services and/or support guys first if they have notes or details.
– Ask the customer in the ticket and submit as Pending
– Often we’ll need full logs. Refer to ./triage_checklist
• Try to find a developer with recent context to the ZenPack. git log is your friend.
4.12.2 Hotfixes
• Requirements:
– Issue is pervasive: The problem is in both master and develop branches, or may be affecting manycustomers adversely.
The problem may be in master and a fix was already provided in develop, in which case a cherry-pick maybe in order.
– Issue is serious: The problem is affecting production both internally and externally. Examples: blocksQA, customers escalate.
• Provide a dev version to vet through the fix in the customer environment before proceeding with the release ofhotfix. If customer allows you to triage on their system, good. Otherwise provide a development egg.
• High Priority and/or Severity issues that are blocking a customer and can go out without a QA cycle. Make sureyou test internally, and get feedback from the customer after deployment.
• Run through committee of experts/ownership group familiar with the ZenPack. This is always good practice:Someone may know the best solution to the problem that causes least disruption.
4.12.3 Wiki
• Make sure you update the docs outlining the key fix information.
• Every hotfix has to be released and published to the Wiki.
• Wiki Updates: Keep minor releases and remove older hotfix releases. Remember: the numbering scheme forZenPacks is:
Major.Minor.Revision
4.12. L3 Support Process 47
ZenPackers Documentation
4.12.4 Backlog
• Only keep ZenDesk tickets that are active.
• Send back to originator if inactive.
4.13 Daemons & Debugging
Zenoss Core has lot of daemons that can be used for various purpose.
While you might not use all of the daemons in your Zenoss environment, it is still essential for you to understand allavailable zenoss daemons, which you can use to solve a specific enterprise monitoring requirement.
In Zenoss v4 all daemons interact with Mysql and zenhub. By defaultall the zenoss daemons run in info mode.
Note: Only some of this information will be relevant to 5.X
4.13.1 1. zeneventserver
In Zenoss 3.X the event processing was managed by zenhub daemon because of that there was bottleneck in eventprocessing which has caused some performance related issues, so to avoid that in Zenoss 4.x they have introduced newdaemon zeneventserver.
zeneventserver daemon which is written in java, interact with Rabbitmq and fetch the events from zeneventsqueue[zenoss.queues.zep.zenevents]and insert into the events tables that are in zenoss_zep database where the eventsare getting stored.
Another responsibility of this daemon is to fetch the event related information like [resource, summary, last seen, firstseen, severity etc.], in Java Script Object Notation [JSON] format so that it get displayed in zenoss UI also movesthe event to zenoss zenoss.queues.zep.signal queue. Zenpack installation is taken care by this daemon. [Event Classcreation while installing zenpack]
cat /opt/zenoss/etc/zeneventserver.conf [config file of zeneventserver]zep.jdbc.dbname=zenoss_zepzep.jdbc.hostname=localhostzep.jdbc.port=13306zep.jdbc.password=xxxxxxzep.jdbc.username=xxxxxx
Note: if we stop zeneventserver daemon and check rabbitmq queues by the command [rabbitmqctl -p /zenosslist_queues], we could see value of queue zenoss.queues.zep.zenevents gets increasing.
# zeneventserver stop [Need to run as zenoss user]
# rabbitmqctl -p /zenoss list_queuesListing queues ...celery 0zenoss.queues.zep.migrated.summary 0zenoss.queues.zep.migrated.archive 0
(continues on next page)
48 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
(continued from previous page)
zenoss.queues.zep.rawevents 0zenoss.queues.zep.heartbeats 0zenoss.queues.zep.zenevents 12zenoss.queues.zep.modelchange 0zenoss.queues.zep.signal 0...done.
Once we start zeneventserver we can see the queue count get back to ‘0’ [Zero]
4.13.2 2. zenhub
zenhub daemon is written in python. Event processing gets triggered when zenhub daemon gets data from otherdaemons. In paid version zenoss has included invalidation worker process for better performance. In distributedzenoss architecture we normally deploy a Linux VM which runs a dedicated zenhub daemon only. It’s the single pointof failure of events getting displayed in zenoss.
4.13.3 3. zenping
zenping daemon does normal ping on interface and the polling interval is one-minute. If the interface is down zenossgenerate an alert with critical severity. If the interface comes up zenoss automatically clears the error event with aclear message. zenping has ability to get data point for ping monitoring and can be used to generate availability graphsfor the monitored device [Feature in Zenoss4]. The following runs this daemon in debug mode against a device, andused mainly for debugging.
zenping -d 3.14.1.59 -v10
Warning: If your device doesn’t allow ping connections, your modeler and collector may not work. To circumventthis, set zPingMonitorIgnore to False.
4.13.4 4. zeneventd
The zeneventd daemon’s major role is to apply event transforms (Python code that manipulates events). This daemonfetches incoming events from zenoss.queues.zep.rawevents queue and applies transform and necessary device contextsuch as Device group/Device State [Production/Development] and some event related information such as [last seen,first seen] time of event. If the daemon stops running, we could see size of rawevents queue getting increased. We candebug execution of transform code by putting log.info(‘For Debug’) statement inside the code which will be printedin ‘/opt/zenoss/log/zeneventd.log’
# zeneventd stop
# rabbitmqctl -p /zenoss list_queuesListing queues ...celery 0zenoss.queues.zep.migrated.summary 0zenoss.queues.zep.migrated.archive 0zenoss.queues.zep.rawevents 10zenoss.queues.zep.heartbeats 0zenoss.queues.zep.zenevents 0zenoss.queues.zep.modelchange 0
(continues on next page)
4.13. Daemons & Debugging 49
ZenPackers Documentation
(continued from previous page)
zenoss.queues.zep.signal 0...done.
# zeneventd run -d 3.14.1.59 -v10
If you have code that is run as a result of an event transform, place your pdb.set_trace in that code, and run zeneventdin the foreground to catch it.
4.13.5 5. zensyslog
zensyslog daemon process syslog messages [/var/log/messages] that are received from monitored device on to zenosson port UDP/514.
4.13.6 6. zenprocess
zenprocess daemon, process monitoring capability is integrated tozenoss by using HOST-RESOURCES MIB which get loaded into zenoss as partof default installation. zenprocess uses SNMP table and get processinformation like PID, path to the binary that is being executed andnumber of running instances.
Etc. zenprocess daemon default polling interval is 3 min [180seconds]. Not possible to customize the polling interval per devicelevel. The following runs this daemon in debug mode against a single device.
zenprocess run -d 3.14.1.59 -v10
4.13.7 7. zenstatus
zenstatus daemon monitors TCP/UDP services that are available on the device such as [http/https/net-bios/].
4.13.8 8. zentrap
zentrap daemon process the incoming traps that are send from hardware on port UDP/162. The daemon decodes theincoming trap to a format that is understandable by zenoss [Python dictionary format] and handover to zeneventd forfurther processing and to generate events.
4.13.9 9. zenactiond
zenactiond daemon the daemon interact with signal queue in Rabbitmq and trigger notification via Email/Paging/etc.Signal queue get piled up if this daemon stops running.
50 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
# rabbitmqctl -p /zenoss list_queuesListing queues ...celery 0zenoss.queues.zep.migrated.summary 0zenoss.queues.zep.migrated.archive 0zenoss.queues.zep.rawevents 0zenoss.queues.zep.heartbeats 0zenoss.queues.zep.zenevents 0zenoss.queues.zep.modelchange 0zenoss.queues.zep.signal 5...done.
# zenactiond start
4.13.10 10. zenperfsnmp
zenperfsnmp daemon collects SNMP performance metrics such as CPU, Memory, File system Usage via snmpwalkand store the information in RRD [Round Robin Database] files, the data collection interval is 300 sec by default. Thepoll time interval is not customizable per device level, if we change it, it get reflected globally. The following runs thisdaemon in debug mode against a single device.
zenperfsnmp run -d 3.14.1.59 -v10
4.13.11 11. zencommand
zencommand is responsible for running routine collection for devices not using snmp style collection. It collects everycycletime which is set in the UI in “Cycle Time” field for the datasource.
zencommand daemon is capable of running custom scripts against the device over ssh, SQL, and other protocols toachieve this ssh username/password need to be configured in zenoss for each monitored device, which is hard if weare monitoring a huge DC. So performance monitoring is done by configuring net-snmp on client device.
zencommand run -d 3.14.1.59 -v10
Debugging
Put your pdb.set_trace() in the collect() or onSucess() methods before running zencommand as above.
4.13.12 12. zenmodeler
zenmodeler daemon gets initial device information such as interfaces/filesystem/ipservices etc. It collects structuraland topological data as well. The daemon polling interval is 12hrs by default. It mainly detects configuration changesthat happen on device eg: Additional interface gets added, new partition etc.
zenmodeller run -d 3.14.1.59 -v10
4.13.13 13. zenrrdcached
zenrrdcached daemon is a performance enhancer, helps to cache RRD metrics in the memory, which is used to generategraphs in zenoss. If the daemon fails to fetch the metrics from the memory it will get the metrics from rrd files that is
4.13. Daemons & Debugging 51
ZenPackers Documentation
stored in the file system.
4.13.14 14. zopectl
zopectl daemon is call zopeclient, used while developing zenpack. To reflect the code change that we make duringzenpack development this daemon need to be restarted.
zopectl restart
To run it in the foreground for debugging:
zopectl fg
4.13.15 15. zenjobs
zenjobs daemon run background tasks like discovering network or adding device these tasks gets added to queueand zenjobs process them. Once a device gets added successfully [Discovered] modeling happens with the help ofzenmodeler daemon and returns a job ID. If we want to add devices in a bulk we use zenbatchload command utility.
4.13.16 16. zenrdis
zenrdis daemon is used to collect distributed ping-tree data from collector to build a complete map.
In zenoss each daemon has a config file that is located in /opt/zenoss/etc/ directory. By default all daemons are in infomode. There are two ways to enable debug mode for daemons.
1. Edit config file for daemon change logseverity 10
2. daemon name debug Eg: zeneventd debug [toggle daemon between Info and Debug mode]
By listing Rabbitmq queue one can easily determine whether zenoss is working without any problem or not. If we findany events that get struck in any of the queue, restart the corresponding daemon that is responsible for fetching events.
4.13.17 17. zenpython
zenpython is part of the ZenPacks.zenoss.PythonCollector zenpack that allows data collection of standard COM-MAND data source type’s functionality without requiring a new shell and shell subprocess to be spawned each timethe data source is collected. This allows a pure python method to collect data (and to pass python structures) anddebug. You can run it in the background or in the foreground
zenpython run -v10 --device=target.example.com
Debugging
Place your pdb.set_trace() inside your datasource’s collect() or onSuccess() methods and run the above forgroundcommand.
52 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
4.13.18 18. Others (5X and above)
• zenminion: Performs distributed bash exectution
• zenvsphere: vSphere collection
• zenucsevents: Cisco UCS metrics and events
4.14 Zenwipe.sh: Erase Everything but Core
Zenwipe will remove everything except the Core product. This includes a full purge of:
• ZODB relstorage
• ZODB session
• ZEP
• Mysql
• Zenpacks
You can get it at https://dev.zenoss.com/tracint/browser/branches/core/zenoss-4.x/inst/zenwipe.sh
We list it here also:
#!/bin/bash################################################################################ Copyright (C) Zenoss, Inc. 2007, all rights reserved.## This content is made available according to terms specified in# License.zenoss under the directory where your Zenoss product is installed.###############################################################################
# THIS SCRIPT WILL BLOW AWAY YOUR DATABASE# Use --xml option to this script to rebuild using DmdBuilder and the XML files# Default is simply to reload from SQL dump
if [ -z "${ZENHOME}" ]; thenif [ -d /opt/zenoss ] ; then
ZENHOME=/opt/zenosselse
echo "Please define the ZENHOME environment variable"exit 1
fifi
zengc=$ZENHOME/bin/zenglobalconf
# read configuration from global.confdbtype=$(${zengc} -p zodb-db-type)host=$(${zengc} -p zodb-host)port=$(${zengc} -p zodb-port)user=$(${zengc} -p zodb-user)userpass=$(${zengc} -p zodb-password)admin=$(${zengc} -p zodb-admin-user)adminpass=$(${zengc} -p zodb-admin-password)
(continues on next page)
4.14. Zenwipe.sh: Erase Everything but Core 53
ZenPackers Documentation
(continued from previous page)
dbname=$(${zengc} -p zodb-db)
# Shut down zenoss if it's running.# Faster to check pid files than run the zenoss script.for pidfile in $(find $ZENHOME/var -name \*.pid); do
pid=$(cat $pidfile)if ps -p $pid >/dev/null; then
echo "Stopping Zenoss"$ZENHOME/bin/zenoss stopbreak
fidone
# Drop and recreate the ZODB relstorage databasezeneventserver-create-db --dbtype $dbtype --dbhost $host --dbport $port --dbadminuser→˓$admin --dbadminpass "${adminpass}" --dbuser $user --dbpass "${userpass}" --force --→˓dbname $dbname --schemadir $ZENHOME/Products/ZenUtils/relstorage
# Drop and recreate the ZODB session databasezeneventserver-create-db --dbtype $dbtype --dbhost $host --dbport $port --dbadminuser→˓$admin --dbadminpass "${adminpass}" --dbuser $user --dbpass "${userpass}" --force --→˓dbname ${dbname}_session --schemadir $ZENHOME/Products/ZenUtils/relstorage
# Drop and recreate the ZEP event databasezeneventserver-create-db --dbtype $dbtype --dbhost $host --dbport $port --dbadminuser→˓$admin --dbadminpass "${adminpass}" --dbuser $user --dbpass "${userpass}" --force
echo "Deleting Zenpacks"rm -rf $ZENHOME/ZenPacks/*
if [ -d $ZENHOME/var/catalogservice ]; thenrm -rf $ZENHOME/var/catalogservice
fi
# Creates the initial user file for zenbuildpython -m Zope2.utilities.zpasswd -u admin -p zenoss $ZENHOME/inituserzenbuild -v 10 -u$admin -p "$adminpass" "$@"zendmd <<EOFdmd._rq = Truecommit()EOFsed -i -e 's/# debug-mode on/debug-mode on/' $ZENHOME/etc/zope.conf# truncate daemons.txt filecp /dev/null $ZENHOME/etc/daemons.txt
You can also selectively purge single items by editing the script. For example, you can remove the ZEP database onlywith this derivative script:
#!/bin/bash############################################################################### THIS SCRIPT WILL BLOW AWAY YOUR ZEP EVENT DATABASE##############################################################################
if [ -z "${ZENHOME}" ]; thenif [ -d /opt/zenoss ] ; then
ZENHOME=/opt/zenoss
(continues on next page)
54 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
(continued from previous page)
elseecho "Please define the ZENHOME environment variable"exit 1
fifi
zengc=$ZENHOME/bin/zenglobalconf
# read configuration from global.confdbtype=$(${zengc} -p zodb-db-type)host=$(${zengc} -p zodb-host)port=$(${zengc} -p zodb-port)user=$(${zengc} -p zodb-user)userpass=$(${zengc} -p zodb-password)admin=$(${zengc} -p zodb-admin-user)adminpass=$(${zengc} -p zodb-admin-password)dbname=$(${zengc} -p zodb-db)
zenoss stop
# Drop and recreate the ZEP event databasezeneventserver-create-db --dbtype $dbtype --dbhost $host --dbport $port --dbadminuser→˓$admin --dbadminpass "${adminpass}" --dbuser $user --dbpass "${userpass}" --force
zenoss start
4.14.1 Removing Broken Zenpacks
Daniel Robbins has written a script that detects damaged zenpacks and attempts to remove them: http://wiki.zenoss.org/Removing_Broken_ZenPacks
To use it you must use it via zendmd:
zendmd --script=zenpack-remove.py
Or you can try to manually remove it interactively in zendmd:
dmd.ZenPackManager.packs._delObject('ZenPacks.zenoss.ZenPackName')commit()
4.15 Zendmd Examples
4.15.1 Preliminaries
Setup your variables first in the CLI:
export device='max.zenoss.com'
You can import that to dmd or python like this:
4.15. Zendmd Examples 55
ZenPackers Documentation
import osdevstring = os.environ['device']device = find(devstring)
4.15.2 Examples
Zendmd examples, plain and simple. You’ll want to see these tips also:
http://wiki.zenoss.org/Category:ZenDMD
Required: You must be in the zendmd interpreter for all these examples.
• Find a device:
device = find(devstring)
• Build Relations for a Device:
d=find(devstring)d.buildRelations()commit()
• Delete a Device:
device = find(devstring)device.deleteDevice()commit()
• List/Delete Components:
# Naive way: Does not remove (global, component) catalog entries. and networksdevice = find(devstring)for component in device.getDeviceComponents():
component.getPrimaryParent()._delObject(component.id)commit()
# Better: Delete all components of a certain type using Facadedef DeleteAllComponents(type):
df = getFacade('device')for d in dmd.Devices.getSubDevices():
components = df.getComponents(d.getPrimaryId(), meta_type=type)if components.total > 0:
del_list = []for brain in components:
del_list.append(brain.uid)try:
df.deleteComponents(del_list)except:
print "Error on %s" % d.getPrimaryId()print del_listcommit()
DeleteAllComponents("IpService")
• Print templates for a device:
56 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
devices = dmd.Devices.getSubDevicesGen()for d in devices:
datasource_ids = []templates = d.getRRDTemplates()for t in templates:
datasources = t.getRRDDataSources()for ds in datasources:
if ds.id not in datasource_ids:datasource_ids.append(ds.id)
else:print("Device has duplicate datasource:")print("Device: %s" % d)print("Template: %s" % t)print("Datasource: %s" % ds)
• Delete Local Device Property:
for d in dmd.Devices.getSubDevices():if d.isLocal('zDeviceTemplates'):
d.deleteZenProperty('zDeviceTemplates')
• Get a Component by Name:
def getComponentByName(device, component_name):cs = device.componentSearch()for brain in cs:
component = brain.getObject()if component.name() == component_name:
return component
device = find(devstring)comp = getComponentByName(device, 'vm_1')
• Get a Component by ID:
def getComponentById(device, component_id):cs = device.componentSearch()for brain in cs:
component = brain.getObject()if component.id == component_id:
return component
device = find(devstring)comp = getComponentById(device, 'network-10a893c1-01a6-438a-b231-3d5102cbc639')
• Catalog: Re-Index a device:
from zope.event import notifyfrom Products.Zuul.catalog.events import IndexingEvent
device = find(devstring)device.index_object()notify(IndexingEvent(device))commit()
• Events: Access Notifications and Triggers
First method through Facade:
4.15. Zendmd Examples 57
ZenPackers Documentation
facade = getFacade('triggers')for trig in facade.getTriggerList():
trigData = facade.getTrigger(trig['uuid'])
print 'Notificatioons: '
for note in facade.getNotifications():print note.name
Second way through dmd.NotificationSubscriptions:
dmd.NotificationSubscriptions.getChildNodes()for item in dmd.NotificationSubscriptions.getChildNodes():
print item
4.16 Parsing an Objects.xml file using JQ
You first need to convert your objects.xml file to json, objects.json
• This one selects “datasources” from objects:
jq '[ .objects.object[][][] |select ( .["@id"] == "datasources")[] ]' objects.json
• Selects datasource “SnapTBSP” from objects::
jq ‘[ .objects.object[][] | select ( .[“@id”] == “datasources”)[] | select (.[“@id”] == “SnapTBSP”) ]’objects.json
• Select datapoints[][] from SnapTBSP ds:
jq '[ .objects.object[][][] |select ( .["@id"] == "datasources")[] |select (.["@id"] == "SnapTBSP")[] |select (.["@id"] == "datapoints")[][] ]' objects.json
• List just the @id labels from the datapoints of SnapTBSP ds:
jq '[ .objects.object[][][] |select ( .["@id"] == "datasources")[] |select (.["@id"] == "SnapTBSP")[] |select (.["@id"] == "datapoints")[][] |{"@id"} ]' objects.json
• Select datapoints[][] from DB2DataSource class:
jq '[ .objects.object[][][] |select ( .["@id"] == "datasources")[] |select (.["@class"] == "DB2DataSource")[] |select (.["@id"] == "datapoints")[][] |{"@id"} ]' objects.json
58 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
4.17 Converting to the new Catalog
We’re going to a HTML format for our ZP documentation, which means we need to either convert directly or havesome intermediate format. In this doc we talk about how to use RST as the master source for documentation.
The overall conversion strategy: MediaWiki => RST => HTML Get a copy of pandoc installed on your system. Wewill first convert our MediaWiki page to RST.
• First on Linux or OSX, get pandoc and convert README.wikimedia to RST:
pandoc -o README.rst README.wikimedia
You may have to change the extension of the source file to “wiki”.
• You should add the Title area as mentioned.
The title should be underlined by equals signs. RST looks at the type of underlines to deduce the Ti-tle/Section/Subsection hierarchy.
• Add a table of contents using the .. contents:: tag. There are examples in Ceph and Zenpackers. Thiswill generate the TOC once we render.
• Avoid block quotes like the plague because they tend to be ugly. See RST Sample for samples and examples.:
• Ensure all Notes are rendered as per Ceph: ie.. .. Note:: Many notes will be rendered as some strange combi-nation of text and marks. If you see .. raw:: mediawiki those need to be removed and replace. Justcreate a new note like this one:
• RST also supports Warnings, Info, Tip etc:
Warning: Don’t eat the yellow snow!
Tip: Eat at Saint Elfonso’s Pancake House
• Look for strange or wierd bullets or indentations that can be simplified.
• For gallery images: Look at the Ceph sample for images which should look like:
.. image:: apple.jpg:height: 200px:width: 200px:alt: alternate text:target: apple.jpg
.. image:: banana.jpg:height: 200px:width: 200px:alt: alternate text:target: banana.jpg
.. image:: cherry.jpg:height: 200px:width: 200px:alt: alternate text:target: cherry.jpg
(continues on next page)
4.17. Converting to the new Catalog 59
ZenPackers Documentation
(continued from previous page)
.. image:: duas.jpg:height: 200px:width: 200px:alt: alternate text:target: duas.jpg:align: right
• For fancier “figures” do something like this:
.. figure:: eagle.jpg:width: 200px:alt: map to buried treasure:target: eagle.jpg:align: right
This is the caption of the figure (just a paragraph).
• Fancy code-blocks can often be replaces by a double-colon mark. see Ceph
• You can convert your document to HTML with pandoc or use the tool provided in the Ceph ZP base dir calleddoc.py which requires Python Docutils.
• There are plent of examples of things in Zenpackers sources as well.
Table of Contents
• RST Sample
– Section
– Avoid block quotes.
– Gallery
4.18 RST Sample
Note: This is a note, and is noteworthy. Note how subsequent lines are started under the “N” in note.
Warning: This is a note, and is noteworthy
Important: You can find references for RST at:
• http://docutils.sourceforge.net/docs/user/rst/quickref.html
• http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
• http://docutils.sourceforge.net/rst.html
60 Chapter 4. Topics for Zenpack Administration
ZenPackers Documentation
4.18.1 Section
Here is a bullet list:
• Item one starts 2 chars after the star. You must have whitespace above.
• Item two can be right after last one or have space..
– Subitem one must have space at the start
– Subitem two can be right after one. If you want extra stuff for an item, it must start in same column as theoriginal text. You can wrap if you want or let it run off into the far reaches of oblivion.
• Item three
• Last item must have whitespace afterwards.
4.18.2 Avoid block quotes.
Block Quotes should be avoided because it injects <blockquote> into the html. Block quotes start with indented textlike following:
Crazy Heading This is a block quote and should be avoided. This is a block quote and should be avoided. This is ablock quote and should be avoided.
Symbol Meaning
tent.png
Campground
waves.png
Lake
peak.png
Mountain
4.18. RST Sample 61
ZenPackers Documentation
4.18.3 Gallery
apple.jpg banana.jpg cherry.jpg duas.jpg
eagle.jpg
Make sure to add this CSS snippet to your stylesheet:
img.gallery { padding:5px; border:2px; margin:2px; }
The docs.py tool in the Ceph ZP automatically does this for you.
Note: The current site may insert a nasty blue rectangle for the :target: link style. This can be avoided by simplyomitting any :target:.
62 Chapter 4. Topics for Zenpack Administration
CHAPTER 5
General Zenpack Development
This is the documentation for Inter-Galactic ZenPack Developers
5.1 File Locations
The location of specific files within a ZenPack’s directory structure is technically mandated in some circumstances,and open to the developer’s desires in others. To make it easier for other developers to more easily get up to speedwith the ZenPack in the future, the following recommendations for file locations should be used.
Note that with ZenPacklib (ZPL), some locations are deprecated. Please see examples to clarify this. When in doublewe will endeavor to label the sections with (ZPL) and (No-ZPL).
• ZenPacks.<namespace.PackName>/
• ZenPacks.<namespace.PackName>/ZenPacks/namespace/PackName/
– analytics/
The analytics bundle in .zip format: analytics-bundle.zip
– browser/ (No-ZPL. See /resources below)
* configure.zcml
All ZCML definitions related to defining browser views or wiring browser-only concerns. Thisconfigure.zcml should be included by the default configure.zcml in its parent directory.
* resources/
· css/ - All stylesheets loaded by users’ browsers.
· img/ - All images loaded by users’ browsers.
· js/ - All javascript loaded by users’ browser.
63
ZenPackers Documentation
– resources/ (Note: ZPL, See /browser above)
Any javascript code that modifies views go here. Especially note these JS file correlations:
* device.js - Modifies the default device page.
* ComponentClass.js - Modifies the component ComponentClass page.
Folders inside resources have the following properties:
* icon/ (Note: ZPL)
All images and icons loaded by the browser. Within this folder note the following name correspon-dence:
· DeviceClass.png - Icon used in top left corner.
· ComponentClass.png - Icon used in Impact diagrams for component.
– datasources/
All datasources plugin files. Ensure your datasource has a descriptive name that closely correlates to theplugin name.
– lib/
Any third-party modules included with the ZenPack should be located in this directory. In the case ofpure-Python modules they can be located directly here. In the case of binary modules the build processmust install them here. See the section of License Compliance below for more information on how toproperly handle third-party content.
– libexec/
Any scripts intended to be run by the zencommand daemon must be located in this directory.
– migrate/
All migration code.
– modeler/
All modeling plugins.
– objects/
There should only ever be a single file called objects.xml in this directory. While the ZenPack installationcode will load objects from any file in this directory with a .xml extension, the ZenPack export codewill dump all objects back to objects.xml so creating other files only creates future confusion betweeninstallation and export.
– parsers/
All custom parsers here.
– patches/
All monkeypatches. Note: your patches/__init__.py must specify patch loading.
– protocols/
AMQP schema: Javascript code is read into the AMQP protocol to modify queues and exchanges.
– services/
Custom collector services plugins.
– service-definition/ (Note: 5.X+)
Service definitions for 5.X services containers.
64 Chapter 5. General Zenpack Development
ZenPackers Documentation
– skins/
All TAL template skins in .pt format. These change the UI look.
– tests/
All unit tests go here.
– facades.py
All facades (classes implementing Products.Zuul.interfaces.IFacade) should be defined in this file. InZenPacks where this single file becomes hard to maintain, a facades/ directory should be created contain-ing individual files named for the group of facades they contain.
– info.py
All info adapters (classes implementing Products.Zuul.interfaces.IInfo) should be defined in this file. InZenPacks where this single file becomes hard to maintain, an info/ directory should be created containingindividual files named for the group of info adapters they contain.
– interfaces.py
All interfaces (classes extending zope.interface.Interface) should be defined in this file. In ZenPackswhere this single file becomes hard to maintain, an interfaces/ directory should be created containingindividual files named for the group of interfaces they contain.
– routers.py
All routers (classes extending Products.ZenUtils.Ext.DirectRouter) should be defined in this file. InZenPacks where this single file becomes hard to maintain, a routers/ directory should be created contain-ing individual files named for the group of routers they contain.
5.2 General Considerations
Get yourself a good IDE and copy of both the core code and a few zenpacks to look at.
5.2.1 Coding Standards
• Python
– PEP 8 – Style Guide or Python Code – PEP 257 – Docstring Conventions •
• ZCML
– Zope’s ZCML Style Guide
5.2.2 Zenoss Process Overview
Here is a picture of the overall Zenoss processes. You will need to have a rudimentary understanding of this.
5.3 Event Management
5.3.1 About Events
Events, and the graphs generated from performance monitoring, are the primary operational tools for understandingthe state of your environment. This chapter defines events and describes the Zenoss event management system.
5.2. General Considerations 65
ZenPackers Documentation
Fig. 1: Zenoss Flow Chart
66 Chapter 5. General Zenpack Development
ZenPackers Documentation
For a full list of EventClasses see Event Class List
Basic Event Fields
Warning: To enter the Zenoss event management system, an event must contain values for the Device, Severity,and Summary fields. If an event is missing any of these fields, then Zenoss rejects it.
Basic event fields are:
• device
• severity
• summary
• ipAddress
• eventState
• message
• evid
device and ipAddress Fields
The device field is a free-form text field that allows up to 128 characters. Zenoss accepts any value for this field,including devices that are not in the database. If the device field contains an IP address, then Zenoss queries thedatabase for devices with a matching address. If it finds a match, it changes the device field to the found device name.
The ipAddress field is a free-form text field that allows up to 15 characters. This field is not required. If Zenoss cannotsuccessfully locate a device based on the event’s device field content, it attempts to find the device based the eventipAddress field content, if present.
Zenoss automatically adds information to incoming events that match a device in its database. Fields added are:
• prodState - Specifies the device’s current production state.
• Location - Specifies the location (if any) to which the device is assigned.
• DeviceClass - Classifies the device.
• DeviceGroups - Specifies the groups (if any) to which the device is assigned.
• Systems - Systems (if any) to which the device is assigned.
• DevicePriority - Priority assigned to the device.
For more information about these fields, refer to the chapters titled “Production States and Maintenance Windows”and “Organizers and Path Navigation.”
eventState Field
The eventState field defines the current state of the event. This field is often updated after an event has been created.Values for this numeric field are 0-2, defined as follows:
5.3. Event Management 67
ZenPackers Documentation
Number Name0 New1 Acknowledged2 Suppressed
severity Field
The severity field defines the severity of the event. Values for this numeric field are 0-5, defined as follows:
Number Name Color0 Clear Green1 Debug Grey2 Info Blue3 Warning Yellow4 Error Orange5 Critical Red
summary and message Fields
The summary and message fields are free-form text fields. The summary field allows up to 128 characters. Themessage field allows up to 65535 characters. These fields usually contain similar data.
Zenoss handles these fields differently, depending on whether one or both are present on an incoming event:
• If only summary is present, then Zenoss copies its contents into message and truncates summary contents to 128characters.
• If only message is present, then Zenoss copies its contents into summary and truncates summary contents to 128characters.
• If summary and message are both present, then Zenoss truncates summary contents to 128 characters.
As a result, data loss is possible only if the message or summary content exceeds 65535 characters, or if both fieldsare present and the summary content exceeds 128 characters.
To ensure that enough detail can be contained within the 128-character summary field limit, avoid reproducing infor-mation in the summary that exists on other fields (such as device, component, or severity).
evid
The evid field is the event identifier, or event ID. It is a 36-character, unique identifier for every event that comes intothe system. An incoming event should never have an evid assigned to it, because Zenoss creates it immediately beforethe event is inserted into the database. If an incoming event does have an assigned evid, then Zenoss ignores it andreplaces it with a generated evid.
Other Fields
Zenoss events include numerous other standard fields. Some control how an event is mapped and correlated; othersprovide information about the event.
The following table lists additional event fields.
68 Chapter 5. General Zenpack Development
ZenPackers Documentation
Field Descriptiondepuid Dynamically generated fingerprint that allows Zenoss to perform de-duplication on repeating events
that share similar characteristics.compo-nent
Free-form text field (maximum 255 characters) that allows additional context to be given to events(for example, the interface name for an interface threshold event).
event-Class
Name of the event class into which this event has been created or mapped.
eventKey Free-form text field (maximum 128 characters) that allows another specificity key to be used to drivethe de-duplication and auto-clearing correlation process.
event-ClassKey
Free-form text field (maximum 128 characters) that is used as the first step in mapping an unknownevent into an event class.
event-Group
Free-form text field (maximum 64 characters) that can be used to group similar types of events. Thisis primarily an extension point for customization. Currently not used in a standard Zenoss system.
state-Change
Last time that any information about the event changed.
firstTime First time that the event occurred.lastTime Most recent time that the event occurred.count Number of occurrences of the event between the firstTime and lastTime.prodState Production state of the device when the event occurred. If an event is still active when a device’s
production state is changed, the event’s prodState will be updated accordingly.suppid If this event has been suppressed by another event, then suppid contains the other event’s evid.manager Deprecated. The monitor field replaces this field.agent Typically the name of the daemon that generated the event. For example, an SNMP threshold event
will have zenperfsnmp as its agent.Device-Class
Device class of the device that the event is related to.
Location Location of the device that the event is related to.Systems Pipe-delimited list of systems that the device is contained within.Device-Groups
Pipe-delimited list of systems that the device is contained within.
facility Only present on events coming from syslog. The syslog facility.priority Only present on events coming from syslog. The syslog priority.ntevid Only present on events coming from Windows event log. The NT Event ID.ownerid Name of the user who acknowledged this event.clearid Only present on events in history that were auto-cleared. The evid of the event that cleared this one.Devi-cePriority
Priority of the device that the event is related to.
event-ClassMap-ping
If this event was matched by one of the configured event class mappings, contains the name of thatmapping rule.
monitor In a distributed setup, contains the name of the collector from which the event originated.
Details
In addition to the standard fields, Zenoss also allows events to add an arbitrary number of additional name/value pairsto events to give them more context. The name and value of these details are limited to 255 characters in length.
De-Duplication
Zenoss uses an event “de-duplication” feature, based on the concept of an event’s fingerprint. Within Zenoss, thisfingerprint is the “depuid.” All of the standard events that Zenoss creates as a result of its polling activities are de-
5.3. Event Management 69
ZenPackers Documentation
duplicated, with no setup required. However, you can apply de-duplicating to events that arrive from other sources,such as syslog, SNMP traps, or a Windows event log.
The most important de-duplication concept is the fingerprint. In all cases, an event’s fingerprint (or dedupid) iscomposed of a pipe-delimited string that contains these event fields:
• device
• component (can be blank)
• eventClass
• eventKey (can be blank)
• severity
• summary (omitted from the dedupid if eventKey is non-blank)
When the component and eventKey fields are blank, a dedupid appears similar to:
www.example.com||/Status/Web||4|WebTx check failed
When the component and eventKey fields are present, a dedupid appears similar to:
router1.example.com|FastEthernet0/1|/Perf/Interface|threshName
When a new event comes into the system, the dedupid is constructed. If it matches the dedupid for any active event,the existing event’s count field is incremented by one, and its lastTime field is updated to be the current time. If it doesnot match the dedupid of any active events, then it is inserted into the active event table with a count of 1, and thefirstTime and lastTime fields are set to the current time.
The following illustration depicts a de-duplication scenario in which an identical event occurs three times, followedby one that is different in a single aspect of the dedupid fingerprint.
Fig. 2: Event De-Duplication
70 Chapter 5. General Zenpack Development
ZenPackers Documentation
If you want to change the way de-duplication behaves in Zenoss, you can use an event transform to alter one of thefields used to build the dedupid. You also can use a transform to directly modify the dedupid field, for more powerfulcross-device event de-duplication.
Auto-Clear Correlation
Zenoss’ auto-clearing feature is similar to its de-duplication feature. It also is based on the event’s fingerprint. Thedifference is which event fields make up the fingerprint, and what happens when a new event matches an existingevent’s fingerprint.
All of the standard events that Zenoss creates as a result of its polling activities do auto-clearing by themselves. Aswith de-duplication, you would invoke auto-clearing manually only to handle events that come from other sources,such as syslog, a Windows event log, or SNMP traps.
The auto-clear fingerprint for an event is built by using the combination of these fields:
• device
• component (can be blank)
• eventKey (can be blank)
• eventClass (including zEventClearClasses from event class zProperties)
When a new event comes into Zenoss with a special 0 (Clear) severity, Zenoss checks all active events to see if theymatch the auto-clear fingerprint of the new event. All active events that match the auto-clear fingerprint are movedfrom the active events table to history, and their clearid field is set to the evid of the event that cleared them.
If an event is cleared by the clear event, it is also inserted into the event history; otherwise, it is dropped. This is doneto prevent extraneous clear messages from filling your events database.
The following illustration depicts a standard ping down event and its associated clear event.
If you need to manually invoke the auto-clearing correlation system, you can use an event transform to make sure thatthe clear event has the 0 (Clear) severity set. You also need to ensure that the device, component, and eventClass fieldsmatch the events you intend to clear.
Note
Avoid making clear events too generic; otherwise, you may inadvertently clear a wider variety of events that youintend.
Event Consoles
Zenoss features multiple event consoles that allow you to view and manage events. Each console shows differentevents subsets, depending on your current context.
Zenoss event consoles are:
• Master - To access this console, click Event Console in the Navigation menu. You can view all events from thisconsole.
• Custom - Users can create custom event consoles from the Event Views tab within their user preferences. Eachcustom event console has access to the same events as the global console, but can be filtered more specifically(from the Edit tab).
• Contextual - Contextual event consoles are found throughout Zenoss. Each time you see an Events tab on adevice, device organizer, component, or event class, you can view event information that has been automaticallyfiltered to show events specific to the current context.
5.3. Event Management 71
ZenPackers Documentation
Fig. 3: Event Auto Clearing
72 Chapter 5. General Zenpack Development
ZenPackers Documentation
The master event console is Zenoss’ central nervous system, enabling you to view and manage events. It displays therepository of all events that have been collected by the system.
Fig. 4: Event Console
Sorting and Filtering Events
You can sort and filter events that appear in the event console to customize your view.
You can sort events by any column that appears in the event console. To sort events, click a column header. Clickingthe header toggles between ascending and descending sort order.
Filter options appear below each column header.
You can filter the events that appear in the list in several ways, depending on the field type. Date fields (such as FirstSeen and Last Seen) allow you to enter a value or use a date selection tool to limit the list. For other fields, such asDevice, Component, and Event Class, enter a match value to limit the list.
The Count field allows you to filter the list when compared to a value:
• n - Displays events with counts greater than or equal to that value.
• <n - Displays events with counts less than that value.
• <=n - Displays events with counts less than or equal to that value.
• =n - Displays events with counts equal to that value.
To clear filters, select Configure > Clear filters.
saving a custom view
you can save your custom event console view by bookmarking it for quick access later. to do this:
1. select configure > save this configuration.
a dialog containing a link to the current view appears.
5.3. Event Management 73
ZenPackers Documentation
Fig. 5: Event Console Filter Options
2. click and drag the link to the bookmarks link on your browser’s menu bar.
zenoss adds a link titled “event console” to your bookmarks list.
_static/event_console_save_configuration.png
Fig. 6: saving a custom view (bookmark)**
tip: you may want to re-title the bookmark, particularly if you choose to save more than one event console view.
Refreshing the View
You can refresh the list of events manually or specify that they refresh automatically. To manually refresh the view,click Refresh. You can manually refresh at any time, even if you have an automatic refresh increment specified.
To set up automatic refresh, select one of the time increments from the Refresh list.
Viewing Event Details
You can view details for any event in the system. To view details, double-click an event row.
Tip: Be sure not to click other links in the row. These go to other pages.
The Event Details area appears.
74 Chapter 5. General Zenpack Development
ZenPackers Documentation
Fig. 7: Automatic Refresh Selections
Fig. 8: Event Details
5.3. Event Management 75
ZenPackers Documentation
To see more information about the event, click Show more details. To display the event information in a new window,click the icon located at the top right.
You can use the Log area to add specific information about the event. Enter details, and then click Add.
Selecting Events
To select one or more events in the list, you can:
• Click a row to select a single event
• Ctrl-Click rows to select multiple events, or Shift-Click to select a range of events
• Click Select to select all, none, new, acknowledged, or suppressed events
Acknowledging Events
You may want to mark an event as “acknowledged” to indicate, for example, that you have taken action to remedy aproblem. To mark events as acknowledged:
1. Select one or more events in the event console view.
#. Click .. image:: _static/acknowledge_button.png
A check mark appears for each acknowledged event.
Returning Events to New Status
You may want to return a previously acknowledged event to “new” status (revoke its “acknowledged” status). To dothis:
1. Select one or more events in the event console view.
2. Click
A check mark no longer appears in the event row, and the event is returned to “new” status.
Classifying Events
Classifying events lets you associate one or more events with a specific event class. To classify events:
1. Select one or more events in the event console view.
#. Click .. image:: _static/classify_button.png
The Classify Events dialog appears.
1. Select an event class from the list of options, and then click Submit.
Note
You can also classify events from event history.
76 Chapter 5. General Zenpack Development
ZenPackers Documentation
Exporting Event Data
You can export data from the event console to a comma-separated value (.csv) or XML file. To do this, select Export> CSV or Export > XML. By default, the exported file is named events.Extension.
Moving Events to History (Close)
When you no longer want to actively monitor event (such as after you acknowledge it, for example), you can move itto history. To do this:
1. Select one or more events in the event console view.
2. Click
The selected events are moved to history.
To view events in history, click the Event History link (located at the bottom left of the Event Console page).
Returning Events to Active Status
You can return events that have been moved to history to active status. When you do this, the events reappear in theevent console.
To return events in history to active status:
1. Click Event History to go to the event history page.
2. Select one or more events.
3. Click
The selected events are returned to active status and appear in the event console.
Creating Events
To create events from the event console, click
For more information about manual event creation, see the section titled “Creating Events Manually.”
Event Sources
Events come into Zenoss in two ways. Generated events are created as a result of active polling. Captured events aretransmitted by external actions into Zenoss.
5.3. Event Management 77
ZenPackers Documentation
Generated Events
These standard daemons are responsible for generating events in Zenoss. They automatically perform appropriatede-duplication and auto-clearing.
• zenping - Ping up/down events
• zenstatus - TCP port up/down events
• zenperfsnmp - SNMP agent up/down events, threshold events
• zencommand - Generic status events, threshold events
• zenprocess - Process up/down events, threshold events
• zenwin - Windows service up/down events
Captured Events
Captured events are those events that Zenoss does not specifically know will occur in advance. De-duplication isperformed on these events, but in some cases may need to be tuned. By default, no auto-clearing is done on capturedevents. Event transforms must be used to create the auto-clear correlations.
These standard daemons are responsible for collecting captured events:
• zensyslog - Events created from syslog messages.
• zentrap - Events created from SNMP traps and informs.
• zeneventlog - Events created from the Windows event log.
There are a number of APIs available for submitting events into Zenoss. For more information, see the Zenoss Devel-oper’s Guide.
Any ZenPacks you install may optionally include their own daemons. For more information, see Zenoss ExtendedMonitoring.
Creating Events Manually
You can manually create events. While this is not something you would do as part of normal Zenoss operation, it canbe helpful when you are attempting to test mappings and transforms you have created.
Creating Events through the User Interface
To create events manually through the user interface:
1. Navigate to Events, and then select Add Event from the table menu.
Note
You also can create events from the Event Console.
The Add an Event dialog appears.
1. Complete the basic event fields. If you want any event class mappings to be applied to the event you are creating,you must select the blank Event Class (rather than the default /). Event class mappings are applied only for eventsthat do not already have an event class.
78 Chapter 5. General Zenpack Development
ZenPackers Documentation
Fig. 9: Add Event Dialog
Creating Events from the Command Line
To send events from the command line, use the zensendevent script, in this format:
Common options include:
• -d DEVICE, –device=DEVICE
• -i IPADDRESS, –ipAddress=IPADDRESS
• -y EVENTKEY, –eventkey=EVENTKEY
• -p COMPONENT, –component=COMPONENT
• -k EVENTCLASSKEY, –eventclasskey=EVENTCLASSKEY
• -s SEVERITY, –severity=SEVERITY
• -c EVENTCLASS, –eventclass=EVENTCLASS
Example
The following example shows how to use the zensendevent script to simulate a ping down event:
Event Classes
Zenoss event classes are a simple organizational structure for the different types of events that Zenoss generates andreceives. This organization is useful for driving alerting and reporting. You can, for example, create an alertingrule that sends you an email or pages you when the availability of a Web site or page is affected by filtering on the/Status/Web event class.
Following is a subset of the Zenoss default event classes. You can create additional event classes as needed.
5.3. Event Management 79
ZenPackers Documentation
• /Status - Used for events affecting availability.
– /Status/Ping - Ping up/down events
– /Status/Snmp - SNMP up/down events
– /Status/Web - Web site or page up/down events
• /Perf - Used for performance threshold events.
– /Perf/CPU - CPU utilization events
– /Perf/Memory - Memory utilization or paging events
– /Perf/Interface - Network interface utilization events
– /Perf/Filesystem - File system usage events
• /App - Application-related events.
• /Change - Events created when Zenoss finds changes in your environment.
Event Class zProperties
Just as device classes and devices have zProperties, so do event classes and event class mappings. zProperties areapplied hierarchically, with the most specific zProperty being applied.
The following zProperties are available on event classes and even class mappings.
• zEventAction - How and where affected events are stored when they occur.
– status - Active events table
– history - Historical event table
– drop - Events are not stored
• zEventClearClasses - Optional list of event class names whose active events will be cleared by clear eventsoccurring in this class.
• zEventSeverity - The severity of affected events is changed to this value unless the Original value is used.
A good example of how Zenoss uses the event class zProperties is found in the /Change event class. Within the/Change event class’ zProperties, zEventAction is set to drop and zEventSeverity is set to Info. This configurationcauses all of the changes in your environment to be stored as info severity events in the history table.
For more information about event manipulation techniques, see the section titled “Mapping and Transformation.”
Mapping and Transformation
Zenoss’ event mapping and transformation system allows you to perform a wide range of operations, from altering theseverity of certain events to altering nearly every field on an event, based on complex rules.
You cannot alter the following fields through event transformation. (This is because they are set after transformationhas been performed.)
• evid
• firstTime
• lastTime
• count
80 Chapter 5. General Zenpack Development
ZenPackers Documentation
Fig. 10: Event Processing
5.3. Event Management 81
ZenPackers Documentation
The following illustration shows the path followed by an incoming event in the event mapping system.
The mapping and transformation process begins with the “eventClass field exists” decision. This also is one of themore important differentiators in how you must handle a particular type of event.
Event Class Mappings
There are two primary ways to view the event class mappings that exist in the system. The first is to go to Eventsin the navigation menu and click on the Mappings tab. This allows you to see all event class mappings in a singlelocation. The EventClass column shows which event class the mapping is in. The other way to view the existing eventclass mappings is to go to the Classes tab of any event class. This shows you only event class mappings related to thecurrent event class.
You can create event class mappings directly from the event classes, but this requires that you know the eventClassKey.A simpler way to create event class mappings is through the event console. Find an event that you want to match, selectit, and then click Classify. Choose the event class that you want the event to be mapped to, and then click OK. Thiswill automatically create the event class mapping with the correct eventClassKey, and example text against which youpotentially can developer your regular expression.
From the Edit tab of an event class mapping, you can control which events it will match, as well as other properties:
• Name - An identifier for this event class mapping. Not important for matching events.
• Event Class Key - Must match the incoming event’s eventClassKey field for this mapping to be considered as amatch for events.
• Sequence - Sequence number of this mapping, among mappings with an identical event class key property. Goto the Sequence tab to alter its position.
• Rule - Provides a programmatic secondary match requirement. It takes a Python expression. If the expressionevaluates to True for an event, this mapping is applied.
• Regex - The regular expression match is used only in cases where the rule property is blank. It takes a PerlCompatible Regular Expression (PCRE). If the regex matches an event’s message field, then this mapping isapplied.
• Transform - Takes Python code that will be executed on the event only if it matches this mapping. For moredetails on transforms, see the section titled “Event Class Transform.”
• Explanation - Free-form text field that can be used to add an explanation field to any event that matches thismapping.
• Resolution - Free-form text field that can be used to add a resolution field to any event that matches this mapping.
The sequence tab of an event class mapping allows you to handle situations where you need to provide more than onepossible mapping for the same eventClassKey. In this case, the sequence is evaluated in ascending order until a full(rule or regex) match is found.
Mappings have the same zProperties as event classes. Any zProperty set locally on a mapping will override the sameproperty set on the event class. This works in the same hierarchical, most specific match, concept that device class anddevice zProperties work.
When a captured event (see the section titled “Event Sources”) occurs, it will not have an event class pre-defined. Forthis type of event, you must create an event class mapping if you want to affect the event. If a captured event occursand none of the event class mappings in the system match it, its event class will be set to /Unknown, and it will retainall of the default properties that it began with.
The next step of evaluation for events without an event class is to check on the eventClassKey field. This is the firstand most important field that controls which event class mapping the event will match. If the event has a blank event-ClassKey, or its eventClassKey does not match any event class mappings in the system, the special “defaultmapping”
82 Chapter 5. General Zenpack Development
ZenPackers Documentation
eventClassKey is searched for instead. This provides for a way to map events even if they have a blank or unpredictableeventClassKey.
Event Class Transform
When a generated event occurs, it has an event class assigned to it. This causes the event class mapping step to beskipped. The only way to affect the fields of one of these events is through the event class’ zProperties and transform.
To access the transform for an event class:
1. Navigate to the event class.
2. From the page menu, select More > Transform.
3. Enter information into the dialog (as Python code).
The objects available in this Python context are evt (the event); and, if the event matches a device that exists in theZenoss database, a device object.
#Example
The following example shows how you can validate that a device object exists before using it to drop events from aparticular location.
Event Life Cycle
In addition to some of the manual methods for getting events into the status or history tables, there are some automatedprocesses that move events from status into history. The event life cycle is defined as all of the ways that events canget into the database, be moved within the database, and be deleted from the database.
The following illustration depicts the event life cycle.
Automatic Event Aging
From the event manager, you can set up automatic aging of certain events from the status table to the history table.This allows you to have lower severity events that do not reoccur for a specified length of time to be automaticallyarchived to the history table.
Event manager properties that control this behavior are:
• Event Aging Threshold (hours) - By default, set to 4 hours.
• Don’t Age This Severity and Above - By default, set to Warning.
With the default settings, Debug, Info, and Warning events that do not occur for four hours are automatically movedinto the history table.
Automatic Historical Event Cleanup
You can set up automatic purging of events from the history table from the event manager. When events are purgedfrom the history table, they can be recovered only from Zenoss backups. For this reason, the default setting is 0, whichspecifies that events are never automatically purged from history.
The event manager property that controls this behavior is Delete Historical Events Older Than (days).
5.3. Event Management 83
ZenPackers Documentation
Fig. 11: Event Life Cycle
84 Chapter 5. General Zenpack Development
ZenPackers Documentation
Event Commands
Event commands allow Zenoss to run arbitrary shell commands when events occur that match pre-configured criteria.This allows almost any action to be taken in response to events that occur.
Common uses of event commands include:
• Auto-remediation of events. You can use SSH to remotely restart services on a UNIX system when they fail, orwinexe to do the same for Windows services.
• Integration with external systems. This includes sending SNMP traps to other management systems, or openingtickets in your incident management system.
• Extending alerting mechanisms. Currently, Zenoss supports only email and pagers as alerting mechanisms “outof the box” through normal alerting rules. You could use event commands to alert through instant messagingsystems, or by playing sounds.
The event commands that you configure are evaluated and executed by the zenactions daemon once each minute(just as in alerting rules).
Creating Event Commands
To create or edit event commands, go to the Commands tab of the event manager. From here, you can adjust theseproperties:
• Enabled - If set to True, then the command is evaluated and executed.
• Default Command Timeout (secs) - Length of time in seconds that Zenoss will wait for the commands to runthat you specify in the Command and Clear Command fields. If the command takes longer than this, Zenosskills it.
• Delay (secs) - Specifies the minimum age (in seconds) of an event before the command will be executed on it.This prevents commands from being run for flapping events.
• Repeat Time (secs) - If the command runs, then it will run again in the specified seconds if the triggering eventis still active. Setting this value to 0 causes the command to be run only one time.
• Command - Specifies the command that will be executed in the shell when the criteria specified in the Wherefield match an event. This command is executed as the zenoss user, and uses TALES syntax for variable substi-tution. (For more information about TALES, see the appendix titled “TALES Expressions.”) Available variablesare evt, device, and component.
• Clear Command - Similar to the Command property. This is executed only when an event that originallymatched the criteria is cleared.
• Where - Defines the criteria that an event must match to trigger this event command.
Capturing Email Messages as Zenoss Events
ZenMail and ZenPop allow you to capture email messages as events in Zenoss. This capability can be useful forsituations in which embedded systems (such as WAPs, NAS devices, or RAID controllers) rely on email notificationfor events.
ZenMail
ZenMail serves as an SMTP server that you can bind to a specific TCP port. You can then configure your embeddedsystem to send mail to the Zenoss server explicitly by using the Zenoss server’s IP address as the relay.
5.3. Event Management 85
ZenPackers Documentation
ZenMail supports these configuration directives:
• ${ZENHOME}/bin/zenmail (no arguments) - Default operation. Binds to port 25 on all ports and listensfor email messages to arrive. Ignores the TO field in the email and uses the FROM address as the device IPaddress.
• ${ZENHOME}/bin/zenmail --listenPort - Bind to the port provided. Useful in situations in whichan SMTP server is already running on the Zenoss server and you do not want to interfere with the existing maildelivery system. Semantics are the same as the no argument version (FROM address is used as the device IP).
ZenPop
ZenPop allows you to retrieve event email from a POP server. ZenPop supports these configuration directives:
• -usessl - Issue the STARTTLS command to the POP server and attempt to transfer email messages usingSSL encryption. This is required if retrieving mail from Google.
• --nodelete - Do not issue the DELE command after retrieving all messages. Typically this is used duringinitial testing so that you do not have to resend test messages to the POP account. Some email systems (such asGoogle) do not actually delete messages when the DELE command is issued.
• --pophost - The hostname or IP address of the POP server from which to retrieve messages.
• --popport - The TCP port the POP server listens on. Defaults to 110. Used in situations where the POPprovider listens on another port (for example, Google on port 995).
• --popuser - The user name that contains email messages to retrieve.
• --poppass - The password to use for the user name provided.
• --cycletime - The time to sleep between polls. After all email is retrieved, ZenPop sleeps for this amountof time before waking up and attempting to pull new email.
Translating Message Elements to the Event
Zenoss translates various message elements to the event, as follows:
• FROM Field - If the FROM field is an IP address, then Zenoss associates the event with the device with thesame IP address. If the FROM field is a fully qualified domain name, then Zenoss resolves it to an IP address,and then performs the device association using the resolved IP address. The resolution of hostname uses “A”records rather than “MX” records.
• TO Field - Zenoss ignores the TO field in the email message. ZenMail accepts email to any user and domainname combination. ZenPop also drops the TO field, and uses only the FROM field.
• SUBJECT Field - ZenMail and ZenPop use the SUBJECT as the event summary.
• Message Body - ZenMail and ZenPop use the first mime attachment as the event details. Zenoss ignoressecondary message bodies (typically HTML-encoded versions of the message). It also ignores attachments(such as files).
5.3.2 Guidelines for Event Handling in Zenpack Development
• Ensure that every event you create has a matching clearing event for auto-clearing to work. Its common to send aclear event each time a successful run occurs. The event handler is designed to handle such repetitive messages.
• Setting the event’s eventClass does prevent the user from easily remapping it. You should never set both event-Class and eventClassKey. You should set eventClassKey and provide a mapping for it.
86 Chapter 5. General Zenpack Development
ZenPackers Documentation
• Your event generation code should allow for an EventClass set by the user on the datasource to propagate intoappropriate events so that the user can use the datasource field to override the standard mapping process.
• Modeler Events should be used sparingly (if ever) in modelers since they only run every 12 hours. If there’s acondition worth sending an event about, use a datasource to monitor it.
• To remove all events, you can re-create the db:
zeneventserver-create-db --dbtype=mysql --force --dbadminpass=zenoss
5.3.3 Guidelines for Creating EvenClassKay Mappings
For more fine-grain control over events, it is often better to create EventMappings and eventClassKeys associated tothose mappings.
In order for this to work you need to remove any hard-coded eventClass in your datasources. The high-level picturelooks like this:
• Remove all eventClass settings from your affected DataSources.
• Add in an eventClassKey for each event you have.
• For each eventClassKey, create an EventMapping either from the UI or in your templates (XML or ZPL). In theZenoss GUI:
– Go to the Events tab
– Click on Event Classes
– Select the Event class on the left list, or create the needed event class.
– Next, click the add (+) button on the right panel
* Fill in your Instance name and Event Class Key value from above
* Repeat the add for all other events sharing this Event Class Key
* Make sure you sequence the created events by hitting Sequence button
• For fine grain control over your sequence numbers you may have to go into zendmd or alternatively, manuallyedit the event templates (XML or ZPL).
• Manually add XML Event Mappings manually requires some care. For example to add the key MySqlMonitor-Plugin to /Status, you would have XML that looks like:
<!-- ('', 'zport', 'dmd', 'Events', 'Status', 'instances', 'MySqlMonitorPlugin') -→˓-><object id="/zport/dmd/Events/Status/instances/MySqlMonitorPlugin" module=→˓"Products.ZenEvents.EventClassInst" class="EventClassInst"><property type="string" id="eventClassKey" mode="w">MySqlMonitorPlugin</property><property type="int" id="sequence" mode="w">1010</property></object>
• In ZPL the same addition would look like:
5.3. Event Management 87
ZenPackers Documentation
event_classes:/Status:
description: MySqlMonitor Eventmappings:
MySqlMonitorPlugin:eventClassKey: MySqlMonitorPluginsequence: 1010remove: true
5.3.4 Get Status of a Device via Curl
You can test a device status with curl:
curl -k -u "<zenoss_username>:<zenoss_password>" -X POST -H "Content-Type:application/→˓json" -d \"{\"action\":\"DeviceRouter\",\"method\":\"getInfo\",\"data\":[{\"uid\":\"/zport/dmd/→˓Devices/Server/SSH/Linux/devices/<my_device_id>\",\"keys\":[\"status\"]}],\"type\":\→˓"rpc\",\"tid\":14}" \"https://<Zenoss_URL>:443/zport/dmd/device_router"
Where you need to replace:
• <zenoss_username>
• <zenoss_password>
• <my_device_id>
• <Zenoss_URL>
5.4 Special Events and Their Consequences
5.4.1 General Stuff
5.4.2 Putting Events in Things
If you’d like to create an event in a modeler you can add these lines:
import zope.componentfrom Products.ZenEvents import ZenEventClassesself._eventService = zope.component.queryUtility(IEventService)........self._eventService.sendEvent(evt)
Here is an example
def _deviceUp(self, result):msg = 'WMI connection to %s up.' % self._devIdself._eventService.sendEvent(dict(
summary=msg,eventClass=Status_Wmi,device=self._devId,severity=ZenEventClasses.Clear,component='zenwin'))
(continues on next page)
88 Chapter 5. General Zenpack Development
ZenPackers Documentation
Fig. 12: Zenoss Events
5.4. Special Events and Their Consequences 89
ZenPackers Documentation
(continued from previous page)
return result
def _failure(self, result):""" Errback for an unsuccessful asynchronous connection. """
err = result.getErrorMessage()log.error("Unable to scan device %s: %s", self._devId, err)
self._reset()
summary = """ Could not read Wins services (%s). Check your 6 """ % err
self._eventService.sendEvent(dict(summary=summary,component='zenwin',eventClass=Status_Wmi,device=self._devId,severity=ZenEventClasses.Error,))
return result
5.4.3 Clearing Events
In order for an event to auto-clear or automatically clear, the event fingerprint must match for both events (the eventthat set the error and the one that clears it). The fingerprint is composed of the following items:
• device
• component (can be blank)
• eventKey (can be blank)
• eventClass (including zEventClearClasses from event class configuration properties)
Reference: https://community.zenoss.org/docs/DOC-9437
5.4.4 INFO zen.ZenModeler Mesages
If you have modeling messages that you get from the zen.ZenModeler that you want to zoom in on, try these settings:
• set CollectorLogChanges to True
• remodel the device
• Look for closed info severity events in teh /Change event class
• You can also look in zenhub for events
5.4.5 How to Set Status on a Device without Ping (Please Read!)
If your device doesn’t have a pingable address or lives in a cluster, then you need a good way to determin status. TheStatus indicator only has three states (Up, Down, Unknown).
The current way is to use ZenPackLib’s overriddent getStatus() method. It mimics the way the Impact determinesstatus of a device by looking at events.
90 Chapter 5. General Zenpack Development
ZenPackers Documentation
To make this work correctly:
• Copy over ZenPackLib with version newer than commit::16b7676d945bbf3067e59937b2118fcb41fc7c7c
• Create an event that will determine your status. Typically this is some critical API or connectivity that governsthe system in question. You must ensure that severity = ZenEventClasses.Critical so that the getStatus() methodwill return a positive value, which shows the device as down. This status MUST be in eventClass /Status .
• Create another event that clears the critical event and gets sent for every successful API (or whatever) access.
• A good way (but not the only way) to set status is through Thresholds:
templates:CephCluster:
thresholds:storage_capacity_critical:dsnames: [cluster_totalUsedBytes]type: MinMaxThresholdeventClass: /Statusseverity: 5maxval: "here.total_bytes() * 0.95"
5.5 Event Classes
You can get an updated list of Event Classes by executing the following script:
#! /usr/bin/env python
import Globalsfrom Products.ZenUtils.ZenScriptBase import ZenScriptBase
dmd = ZenScriptBase(connect=True).dmdfor eventClass in sorted([dmd.Events] + dmd.Events.getSubOrganizers(),
key=lambda c: c.getOrganizerName().lower()):print eventClass.getOrganizerName()
if hasattr(eventClass, 'instances') and len(eventClass.instances()) > 0:for eventClassKey in sorted(eventClass.instances(), key=lambda k: k.id.
→˓lower()):print '\t' + eventClassKey.id
5.5.1 Event Class List
The following lists some of the Event Classes:
//App/App/Citrix
IMAService_3615IMAService_3622MetaFrameEvents_1103MetaFrameEvents_1106
/App/ConnFileMaker Server 7_30FileMaker Server 7_518FileMaker Server 7_520
(continues on next page)
5.5. Event Classes 91
ZenPackers Documentation
(continued from previous page)
MSFTPSVC_10/App/Conn/Max
TermService_1013termservice_1013
/App/EmailEXCDO_8263MSExchangeIS Mailbox Store_1029MSExchangeIS Mailbox Store_1032MSExchangeSA_9320MSExchangeSA_9323smtpsvc_5000
/App/Email/Loopmsexchangetransport_3005
/App/FailedActive Server Pages_9Application Error_1000Application Hang_1002Application Popup_26BROWSER_8032Cloudmark Server Edition_2011EventLog_6000MisysReportsMultiView License Manager_114NETLOGON_5719Norton AntiVirus_6Remote Storage_4002Remote Storage_6021Server Agents_1562Service Control Manager_7031Srv_2022USER32_1073W3SVC_101Winlogon_1218Winlogon_1219
/App/Info.NET Runtime Optimization Service_1102
/App/InstallWSH_0
/App/JobMSSQLSERVER_17055MSSQLSERVER_17177SQLSERVERAGENT_208WUSyncService_104
/App/Job/FailNtfs_50Print_6161Schedule_7901SQLSERVERAGENT_322
/App/LogCRONDpmtapostfix_smtpdprocmail
/App/Oracle/App/Print
Print_10Print_13
(continues on next page)
92 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
Print_15Print_42Print_47Print_9
/App/Reloadpostfix
/App/StartFileMaker Server 7_408FileMaker Server 7_418FileMaker Server 7_486ntsyslogSNMP Informant_0
/App/StopFileMaker Server 7_140FileMaker Server 7_168FileMaker Server 7_412FileMaker Server 7_490ntsyslogService Control Manager_7034
/App/VNCWinVNC4_1
/App/Zenosszenpython-datasources-disabled
/Archivedefaultmappingdefaultmapping_FileMaker
/Capacity/Change/Change/Add/Change/Add/Blocked/Change/Remove/Change/Remove/Blocked/Change/Set/Change/Set/Blocked/Cisco
LINK-2-BADVCALL/Cisco/FW
defaultmapping/Cluster
ClusSvc_1135/Cluster/Heartbeat
ClusSvc_1125ClusSvc_1126ClusSvc_1128ClusSvc_1130
/Cmd/Cmd/Fail/Cmd/Ok/Config
sendmailSYS-5-CONFIG_I
/Conn/Conn/Fail/DB
MSSQLSERVER_17052/DB/Filemaker
FileMaker Server 7_190(continues on next page)
5.5. Event Classes 93
ZenPackers Documentation
(continued from previous page)
FileMaker Server 7_22FileMaker Server 7_601FileMaker Server 7_98
/Environdiagnostic-alarm-trap-nodeenvironment-alarm-trap-majorenvironment-alarm-trap-normal
/HeartbeatdefaultmappingevtsysMARK
/HWServer Administrator_1306Server Agents_1090
/HW/Backupavrdavrd_dropbptm_dropltid_droptldcd_dropvmdvmd_drop
/HW/Backup/Robottldcdtldd
/HW/Backup/SCSIbptm
/HW/Backup/Tape Driveltid
/HW/Backup/Tape Mediatldcd
/HW/BladealertDrscAuthErroralertDrscSEL80percentFullalertDrscSELCriticalalertDrscSELNormalalertDrscSELWarning
/HW/ManagementDell - System Update_6049hplopts_5
/HW/MemoryServer Administrator_1404
/HW/NetworkiANSMiniport_22iANSMiniport_42kernelsystem-config-network
/HW/Network/AdapterDownCPQTeamMP_434CPQTeamMP_436iANSMiniport_11iANSMiniport_13iANSMiniport_14iANSMiniport_15
/HW/Network/TeamDowniANSMiniport_16iANSMiniport_17
(continues on next page)
94 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
/HW/PowerC6KPWR-SP-2-PSFAILC6KPWR-SP-4-PSREDUNDANTBOTHSUPPLYconfigChangeSNMPcpqasm2_4154cpqasm2_4157cpqasm2_4173cpqasm_4157Server Administrator_1304Server Administrator_1352Server Administrator_1354Server Agents_1118Server Agents_1125
/HW/Power/PowerLossC6KPWR-SP-4-PSFAILC6KPWR-SP-4-PSOKC6KPWR-SP-4-PSREDUNDANTONESUPPLY
/HW/Power/UPSaccessViolationHTTPnoBatteriespasswordChangepowerRestoredupsBatteryNeedsReplacementupsBatteryReplacedupsDiagnosticsFailedupsDiagnosticsPassedupsOnBatteryxupstdOnBattery
/HW/Power/UPS/SmartBoostsmartBoostOffsmartBoostOn
/HW/StoreCPQCISSE_24598CPQCISSE_24625CPQCISSE_24644cpqcissm_11QDLT32_7Server Administrator_2048Server Administrator_2049Server Administrator_2057Server Administrator_2083Server Administrator_2094Server Administrator_2123Server Administrator_2124Storage Agents_1145Storage Agents_1146
/HW/Store/FiberChannelStorageAgents_1215StorageAgents_1215_up
/HW/Temperaturecpqasm2_4111Server Administrator_1052Server Administrator_1053Server Agents_1091
/HW/Temperature/FanC6KENV-SP-4-PSFANFAILEDC6KENV-SP-4-PSFANOK
(continues on next page)
5.5. Event Classes 95
ZenPackers Documentation
(continued from previous page)
cpqasm2_4118Server Agents_1130
/IgnoreanacronCRONDdefaultmappingdefaultmapping_local7kernelkernel_capture_cardRadOncNS5XPsendmailServer Agents_1126syslog
/Ignore/WinAutoenrollment_13ClusSvc_1123DirXML Remote Loader_1dirxml remote loader_1excdo_8206excdo_8230excdo_8263FileMaker Server 7_94general_1173Kerberos_3Kerberos_4MetaframeEvents_1106msexchangeactivesyncnotify_10306MSExchangeDSAccess_2110MSExchangeDSAccess_2115MSExchangeIS Mailbox Store_1173msexchangeis_9554msexchangeis_9646msexchangesa_5008msexchangesa_9040msexchangesa_9234MSExchangeTransport_3005msexchangetransport_327NETLOGON_5722NETLOGON_5723NETLOGON_5805NETLOGON_5807norton antivirus_6Print_2Print_20Print_3Print_4Print_8Remote Storage_3037Removable Storage Service_102replication_1232service_102smtp_2smtpsvc_5000store_1025store_1173store_1211Symantec AntiVirus_6
(continues on next page)
96 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
Symantec Mail Security for SMTP_2TermServDevices_1106TermServDevices_1111
/Licenseexchange_292
/NetC4K_EBM-4-HOSTFLAPPINGCDP-4-DUPLEX_MISMATCHdefaultmappingMicrosoft ISA Server Control_15104Microsoft ISA Server Control_15105SNMP-3-CPUHOGSYS-3-CPUHOG
/Net/ConflictDhcp_1005NetBT_4319
/Net/DhcpDhcpServer_1020DhcpServer_1041
/Net/Dhcp/Ackudhcpd
/Net/Dhcp/Inform/Net/Dhcp/Obtained
udhcpc/Net/Dhcp/Request
udhcpd/Net/Link
CPQTeamMP_449LINK-3-UPDOWNLINK-3-UPDOWN_ADMIN_DOWNLINK-3-UPDOWN_UPNIC Agents_1285q57w2k_4snmp_linkDownsnmp_linkUp
/Net/NSDNS_3000DNS_6525DNS_6527DNS_9999NETLOGON_5789Wins_4102
/Net/SpanTreeSPANTREE_VLAN_SW-2-MAX_INSTANCE
/Net/TimeW32Time_22
/OpenStackOpenStack Events Default
/OpenStack/compute/OpenStack/compute/instance
compute.instance default mappingcompute.instance.create.errorcompute.instance.existscompute.instance.exists.verified.old
/OpenStack/dhcp_agentdhcp_agent defaultmapping
/OpenStack/firewall(continues on next page)
5.5. Event Classes 97
ZenPackers Documentation
(continued from previous page)
firewall defaultmapping/OpenStack/firewall_policy
firewall_policy defaultmapping/OpenStack/firewall_rule
firewall_rule defaultmapping/OpenStack/floatingip
floatingip defaultmapping/OpenStack/network
network defaultmapping/OpenStack/port
port defaultmapping/OpenStack/router
router defaultmapping/OpenStack/security_group
security_group defaultmapping/OpenStack/security_group_rule
security_group_rule defaultmapping/OpenStack/subnet
subnet defaultmapping/OS
MisysReports/OS/Pagefile
Srv_2019Srv_2020
/OS/RebootEventLog_6008
/Perf/Perf/CPU/Perf/Filesystem/Perf/Interface/Perf/Memory/Perf/Snmp/Perf/XmlRpc/Security/Security/Auth
CiscoAAA_5SNMP-3-AUTHFAILsnmp_authenticationFailure
/Security/Conn/Security/Conn/Close
dropbeardropbear_resetsshd_pam_unixultravnc
/Security/Conn/Opendropbearsshd_pam_unixultravnc
/Security/LogindefaultmappingMSExchangeIS Mailbox Store_1009MSExchangeIS Mailbox Store_1011
/Security/Login/BadPassdropbearsshd
/Security/Login/Faildropbear
(continues on next page)
98 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
MSFTPSVC_100remote(pam_unix)W3SVC_100
/Security/Sudosudo
/Security/VirusSymantec AntiVirus_4Symantec AntiVirus_46Symantec AntiVirus_5Symantec AntiVirus_51
/StatusopenStackCeilometerHeartbeatopenStackFailureopenStackIniFileAccessopenStackIniFileOptionParsingopenStackNeutronAgentStatusopenStackNovaServiceStatusopenStackRestored
/Status/Flapping/Status/Heartbeat/Status/Image
openstackImageStatus/Status/IpService/Status/Nagios/Status/OSProcess/Status/Perf/Status/Ping/Status/Snmp/Status/Update/Status/VM
openstackServerStatus/Status/Web/Status/WinService
Service Control Manager_7000Service Control Manager_7038
/Status/Wmi/Status/Wmi/Conn/Status/XmlRpc/Status/zenwinmodeler/Storage
CPQCISSE_24597Storage Agents_1202VxSvc_PercPro_509VxSvc_PercPro_580
/Storage/FullRemote Storage_6027Srv_2013
/Storage/SANEmcpBase_100EmcpBase_106
/Unknown/Users/Win/Win/AD
AD WebManager_0DirXML Remote Loader_2MRxSmb_8003
(continues on next page)
5.5. Event Classes 99
ZenPackers Documentation
(continued from previous page)
MSExchangeAL_8270MSExchangeIS_9548Userenv_1006Userenv_1030
/Win/DCOMDCOM_10016
/Win/evtsysevtsys
/Win/ExchangeCloudmark Server Edition_2007MSExchangeIS_9646MSExchangeIS_9665MSExchangeMGMT_3
/Win/NetBiosAutoEnrollment_13BROWSER_8021Userenv_1000
/Win/PWMgrPassword Manager_100Password Manager_200
/Win/SharePointWindows SharePoint Services 2.0_1000
/Win/ShellWinlogon_1002
/Win/UserenvUserenv_1517Userenv_1524
/Win/WindowsUpdate/Win/WindowsUpdate/Failure
Windows Update Agent_16Windows Update Agent_20WindowsUpdateAgent_16WindowsUpdateAgent_20WUSyncService_101
/Win/WindowsUpdate/PendingWindowsUpdateAgent_18
/Win/WindowsUpdate/SuccessExchange Server 2003_4377MDAC_4377NtServicePack_4377OutlookExpress6_4377Windows Update Agent_19WindowsUpdateAgent_19
5.6 GUI Properties and JavaScript Related Methods
5.6.1 Description
Zenpack GUIs need to cute. This is done by getting more properties in the GUI. We explain how to do this here abit. . .
100 Chapter 5. General Zenpack Development
ZenPackers Documentation
5.6.2 Prerequisites
• Zenoss ZenPack Developement
• Python 2.7
We assume that you are familiar with ZenPack developement and Python coding. We further assume that you have aconsole/terminal/xterm open on the system running Zenoss. The CLI default prompt is [zenoss:~]:
# Polling (TBD) # Device Modeling (TBD) # RRD (TBD) # Event Traps (TBD)
5.6.3 Variables: Who, What, Where, and How
Variables in the GUI are primarily set by our ExtJS Javascript library configuration. These are the steps:
• The variables must first be defined in ZP defined classes that inherit IComponentInfo and ComponentInfo
• Then these variables must be set in either the modeler or the datasource collectors.
• Finally they can be used in the ExtJS javascript
We use the DatabaseMonitor ZP (Oracle) as an example. We first define the Instance class (heavily abbreviated):
class Instance(DeviceComponent, ManagedEntity):meta_type = portal_type = 'OracleInstance'.......
hostname = Noneinstance_name = None.......
_properties = _properties + ({'id': 'hostname', 'type': 'string', 'mode': 'w'},{'id': 'instance_name', 'type': 'string', 'mode': 'w'},...
).......
class IInstanceInfo(IComponentInfo):
hostname = schema.TextLine(title=_t(u'Hostname'), readonly=True)instance_name = schema.TextLine(title=_t(u'Instance Name'), readonly=True)
.......
class InstanceInfo(ComponentInfo):implements(IInstanceInfo)
hostname = ProxyProperty('hostname')instance_name = ProxyProperty('instance_name')
.......
Now you can use your variables in an anonmous function inside your ExtJS:
ZC.OracleInstancePanel = Ext.extend(ZC.DatabaseMonitorComponentGridPanel, {constructor: function(config) {
config = Ext.applyIf(config||{}, {componentType: 'OracleInstance',autoExpandColumn: 'name',sortInfo: {
(continues on next page)
5.6. GUI Properties and JavaScript Related Methods 101
ZenPackers Documentation
(continued from previous page)
field: 'name',direction: 'asc',
},fields: [
{name: 'uid'},{name: 'name'},{name: 'hostname'},{name: 'instance_name'},...
],columns: [{
id: 'severity',dataIndex: 'severity',............
},{id: 'name',dataIndex: 'name',...........
},{id: 'hostname',dataIndex: 'hostname',header: _t('Hostname'),width: 160
},{dataIndex: 'instance_name',header: _t('DB Name'),id: 'instance_name',width: 75
}, .... etc ....
5.6.4 Renderer: Changing Column Appearances
Change the “First Seen” and “Last Seen” columns in the event console to show how long ago the event occurred in amore human-friendly way. This is done through the following JS function exampe (time_ago_columns.js):
(function() {
var time_ago_column = {renderer: function(value, metaData, record, rowIndex, colIndex, store) {
var seconds = Math.floor((new Date() - value) / 1000);var interval = Math.floor(seconds / 31536000);
if (interval > 1)return interval + " years ago";
interval = Math.floor(seconds / 2592000);if (interval > 1)
return interval + " months ago";
interval = Math.floor(seconds / 86400);if (interval > 1)
return interval + " days ago";
interval = Math.floor(seconds / 3600);if (interval > 1)
(continues on next page)
102 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
return interval + " hours ago";
interval = Math.floor(seconds / 60);if (interval > 1)
return interval + " minutes ago";
return Math.floor(seconds) + " seconds";}
};
Zenoss.events.registerCustomColumn('firstTime', time_ago_column);Zenoss.events.registerCustomColumn('lastTime', time_ago_column);
}());
5.6.5 Renderer: Linking Grid Elements to other Component Views
Again we are looking at the JS files in $ZPDIR/browser/resources/js/ . Inside your anonymous function, we need todefine a custom renderer (extending Zenoss.render) that creates URL links:
Ext.apply(Zenoss.render, {ZenPacks_zenoss_DatabaseMonitor_entityLinkFromGrid: function(obj, col, record) {
if (!obj)return;
if (typeof(obj) == 'string')obj = record.data;
if (!obj.title && obj.name)obj.title = obj.name;
var isLink = false;
if (this.refName == 'componentgrid') {// Zenoss >= 4.2 / ExtJS4if (this.subComponentGridPanel || this.componentType != obj.meta_type)
isLink = true;} else {
// Zenoss < 4.2 / ExtJS3if (!this.panel || this.panel.subComponentGridPanel)
isLink = true;}
if (isLink) {return '<a href="javascript:Ext.getCmp(\'component_card\').componentgrid.
→˓jumpToEnti>} else {
return obj.title;}
},});
Once that is defined, its really a piece of cake to get your item to link from your grid objects:
5.6. GUI Properties and JavaScript Related Methods 103
ZenPackers Documentation
ZC.OracleTableSpacePanel = Ext.extend(ZC.DatabaseMonitorComponentGridPanel, {constructor: function(config) {
config = Ext.applyIf(config||{}, {autoExpandColumn: 'name',componentType: 'OracleTableSpace',fields: [
{name: 'uid'},............
],columns: [{
id: 'severity',dataIndex: 'severity',header: _t('Events'),renderer: Zenoss.render.severity,sortable: true,width: 40
},{id: 'instance',dataIndex: 'instance',header: _t('Instance'),
>> renderer: Zenoss.render.ZenPacks_zenoss_DatabaseMonitor_→˓entityLinkFromGrid,
sortable: true,width: 70
},... etc ...
Now this ZC.OracleTableSpacePanel grid will have a link to the ZC.OracleInstancePanel grid.
5.6.6 GUI: Adding an Extra Panel to the Navigator
So you have your new component, say TableSpaces and your associated OracleTableSpacePanel grid as above. Butyou may want to have a Nav info view that associates the containing component (Instance in our case). Do do sowe again modify the $ZPDIR/browser/resources/js/DatabaseMonitor.js source as follows; Inside the main anonymousfunction, add the following (See also the JS for PostgreSQL.js):
Zenoss.nav.appendTo('Component', [{id: 'component_tablespaces',text: _t('TableSpaces'),xtype: 'OracleTableSpacePanel',subComponentGridPanel: true,filterNav: function(navpanel) {
switch (navpanel.refOwner.componentType) {case 'OracleInstance': return true;default: return false;
}},setContext: function(uid) {
ZC.OracleTableSpacePanel.superclass.setContext.apply(this, [uid]);}
}]);
Notice that the switch that returns True for the super-component OracleInstance.
104 Chapter 5. General Zenpack Development
ZenPackers Documentation
5.6.7 GUI: Changing Detail Values in the Navigator
The Navigator (Nav Panel) contains the Detail View (and others) below the component frame. We will show how tochange the values presented in the Details window of the Nav.
In your component source you have two classes: TablespaceInfo and ITablespaceInfo (replace “TableSpace” with youractual component name). In order to change the units on our tablespace_allocbytes and tablespace_maxbytes, whichare in bytes, we create decorated class methods:
class ITableSpaceInfo(IComponentInfo):tablespace_name = schema.TextLine(title=_t(u'TableSpace'), readonly=True)allocSize = schema.Float(title=_t(u'Allocated Size'), readonly=True)maxSize = schema.Float(title=_t(u'Max Size'), readonly=True)...
class TableSpaceInfo(ComponentInfo):implements(ITableSpaceInfo)
tablespace_name = ProxyProperty('tablespace_name')tablespace_allocbytes = ProxyProperty('tablespace_allocbytes')tablespace_maxbytes = ProxyProperty('tablespace_maxbytes')....
@propertydef allocSize(self):
return convToUnits(self._object.tablespace_allocbytes)
@propertydef maxSize(self):
return convToUnits(self._object.tablespace_maxbytes)
The Nav will auto-magically pick up the values in the ITableSpaceInfo and present that in the Details page.
5.6.8 GUI: Auto-Expanding Columns and minWidth for Component Grids
Component grids traditionally use the Name field to take up all the extra slack in the spacing. To do that you first set itup as an autoExpandColumn . I also like to set it with a minimum with using the minWidth parameter so it remainsvisible.
So in your $ZPDIR/browser/resources/js/DatabaseMonitor.js you should have something like this:
ZC.OracleInstancePanel = Ext.extend(ZC.DatabaseMonitorComponentGridPanel, {constructor: function(config) {
config = Ext.applyIf(config||{}, {componentType: 'OracleInstance',autoExpandColumn: 'name',sortInfo: {
field: 'name',direction: 'asc',
},fields: [
{name: 'uid'},{name: 'name'},{name: 'meta_type'},{name: 'status'},{name: 'severity'},..................
(continues on next page)
5.6. GUI Properties and JavaScript Related Methods 105
ZenPackers Documentation
(continued from previous page)
],columns: [{
id: 'severity',dataIndex: 'severity',header: _t('Events'),renderer: Zenoss.render.severity,sortale: true,width: 50
},{id: 'name',dataIndex: 'name',header: _t('Name'),sortable: true,minWidth: 70
},........
5.6.9 Missing DocString in Component Class results in Site-Error
If you have a component class that has no docstring, or inherits from a class that is missing a docstring, a direct linkto it will result in a Site-Error in the UI.
Zope will only publish resources with a docstring. When you navigate directly to an object’s primary path you’reasking Zope to publish it. It may look like:
Site errorAn error was encountered while publishing this resource.The requested resource does not exist.Please click here to return to the Zenoss dashboard
It’s hard to troubleshoot because the missing docstring results in a NotFound exception. Zope suppresses logging ofNotFound errors by default. If you go to http://yourzenoss:8080/error_log/manage_workspace and remove NotFoundfrom the list of ignored exception types, you’ll see a very helpful error in event.log when that “Site Error” occurs:
Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publishModule ZPublisher.BaseRequest, line 521, in traverseModule ZPublisher.HTTPResponse, line 727, in debugError
NotFound: Site Error: An error was encountered while publishing this resource.Debugging Notice: Zope has encountered a problem publishing your object.The object athttp://mpX.zenoss.loc:8080/zport/dmd/Devices/OpenStack/Infrastructure/devices/mp8.osi/→˓components/tenant-uuidhas an empty or missing docstring. Objects must have a docstring to be published.
• Example: Zenpacks.zenoss.OpenStackInfrastructure
– Jira: ZPS-17701:
– commit: 73fd0fb8599f07e5f0b4174f21ad96dd50952536
Since the NeutronIntegrationComponent class didn’t have a docstring, and it was the first class extended byTenant/etc, Tenant/etc didn’t have a docstring.
– Log Entry Sample:
106 Chapter 5. General Zenpack Development
ZenPackers Documentation
10.1.1.50 - Anonymous [07/May/2015:10:06:24 -0500]"GET /zport/dmd/Devices/OpenStack/Infrastructure/devices/mp8.osi/components/→˓tenant-uuid HTTP/1.1"404 1136"http://mp2.zenoss.loc:8080/zport/dmd/Devices/OpenStack/Infrastructure/→˓devices/mp8.osi/devicedetail""Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)→˓Chrome/41.0.2272.118 Safari/537.36
5.7 Impact Guide
In general, Impact is a way to create a dependency diagram for your devices and components. It leverages the existingrelationships you already have. It requires a two way set of relations:
• impacts
• impacted_by
Those relations specify your exising device/component relationships.
5.7.1 Debugging Impact
Before we start down the Impact road, we need to know how to debug. Its often difficult to get inside Impact becauseit is not a Python service that can be started in the foreground.
Instead we can use zendmd to get inside Impacts head. This dmd snippet will all you to debug your Impact specificcode:
device = find("mydevice")
from Products.Zuul.catalog.events import IndexingEventfrom zope.event import notify
notify(IndexingEvent(device))
for component in device.getDeviceComponents():notify(IndexingEvent(component))
commit()
One must realize that this only invokes the Impact indexing code and does not apply this to Zenoss in general. Zenossonly invokes Impact when modeling changes are applied. This means that something must change in the model.
In general please note:
• Insert your pdb’s in the appropriate place, usually a class file (see below)
• You must use the commit() to get your debugger code to execute
• Despite the commit(), your code will not be applied to the running ZP
• To get your debugged code to apply to your ZP, you must:
– Remove a component or relationship
– Remodel your device
5.7. Impact Guide 107
ZenPackers Documentation
Rebuilding the Impact Model/Graph
Impact only rebuilds its object index when the model changes. This means you need to modify your model (remove acomponent for example) and re-model.
• You can rebuild your Impact graph manually in the appropriate container or system:
zenimpactgraph run --update
• Another way is to issue a notify event to impact using dmd:
def force_impact(device=device):from Products.Zuul.catalog.events import IndexingEventfrom zope.event import notify
notify(IndexingEvent(device))for component in device.getDeviceComponents():
notify(IndexingEvent(component))commit()
Put this function into your zendmd startup script for convenience.
5.7.2 Impact ZPL Style
Description
Zenoss uses Impact to define the dependency tree (graph) of all devices. ZenPackLib does a lot of setup for Impact,hiding much of the old-style code.
Prerequisites
• Zenoss ZenPack Developement
• Python 2.7
• A ZPL zenpack
• All normal relationships are setup between devices and components
Impact Outline
The Basic idea behind Impact is as follows:
• Create a diagram of your impacts, based on device/component relations
• Inside each of your devices/components define the following sections which contain a list of relationships thatdefine your impact diagram:
– impacts
– impacted_by
• Note: The defining list for (impacts, impacted_by) can also be a function that returns a list of objects thatbelong to your relationship. In this case, its common to want a smaller subset of objects because the barerelationship has objects that don’t affect the Impact relationship. For example, your device contains manyhosts, but only one of those hosts has the API service that your device depends on, so you filter out onlythat one.
108 Chapter 5. General Zenpack Development
ZenPackers Documentation
A ZPL Example
This example uses ControlCenter which has the following devices and components:
• ControlCenter Device (think API)
• Host
• Pool
• Service (definitions of services)
• Running (the actual running service based on a definition)
The ControlCenter devices contains all other components. We focus on the Host relations for this example. The classrelationships look like this:
class_relationships:- ControlCenter 1:MC (controlcenter)Host- Pool(poolHosts) 1:M (hostPool)Host- Host(hostRuns) 1:M (runHost)Running
Then the Impact relations look like this:
Pool:label: CC-Poolimpacts: [parentPool]impacted_by: [poolHosts, childPools]
Host:label: CC-Hostimpacts: [hostPool, hostRuns]impacted_by: [containing_linux_device]
Running:label: CC-RunningServiceimpacts: [serviceDef]impacted_by: [runHost]
Note that containing_linux_device is method that gets impacts from Linux devices that may be running the (virtual)Host. See the following code for hints on this functionality:
* ZenPacks.zenoss.ControlCenter/ZenPacks/zenoss/ControlCenter/patches/__init__.py
* ZenPacks.zenoss.ControlCenter/ZenPacks/zenoss/ControlCenter/patches/platform.py
* ZenPacks.zenoss.ControlCenter/ZenPacks/zenoss/ControlCenter/Host.py
* ZenPacks.zenoss.ControlCenter/ZenPacks/zenoss/ControlCenter/configure.zcml
A Second ZPL Example
It is also common to have a function that returns just a subset of your full relationship components. Using the sameZP as an example, we have relationships for Service:
class_relationships:- Pool(assignedServices) 1:M (assignedPool)Service- Service(serviceRuns) 1:M (serviceDef)Running- Service(childServices) 1:M (parentService)Service
and components Impact relations:
5.7. Impact Guide 109
ZenPackers Documentation
Pool:label: CC-Poolimpacts: [parentPool]impacted_by: [poolHosts, childPools]
Service:label: CC-Serviceimpacts: [parentService]impacted_by: [childServices, serviceRuns, getImports]
Running:label: CC-RunningServiceimpacts: [serviceDef]impacted_by: [runHost]
where getImports() has a signature:
def getImports(self):'''Defines iterable of services that it imports for Impact....
* Input: <Service>self. We use endpoints and services()
* Output: <list>[service-id] for Impact
* Don't change any of the modeling data.. import it all..During modeling, model endpoints on each service...
* Take what is in endpoints and model it here.. Don't do it inmodeling because of the auto-diffing mechanisms...for service in self.device().services():
'''
... do some work ...
... do some more work ...
... do alot more work than you want to see ...# see ZenPacks.zenoss.ControlCenter/ZenPacks/zenoss/ControlCenter/Service.py
# Return a list of filtered services unique to this Impact.return [service(i) for i in _imports]
5.7.3 Impact Rough Guide: Pre-ZPL Style
Description
Zenoss uses Impact to define the dependency tree (graph) of all devices. It does this so that it can determine the causalrelationships of device failure. This is useful when you need to know how devices depend on one another.
Prerequisites
• Zenoss ZenPack Developement
• Python 2.7
We assume that you are familiar with ZenPack developement and Python coding. We further assume that we workfrom the base of ZP_DIR. For NetBotz for example:
export ZP_DIR_TOP=$ZENHOME/ZenPacks/ZenPacks.training.NetBotz export ZP_DIR=$ZP_DIR_TOP/ZenPacks/training/NetBotz
110 Chapter 5. General Zenpack Development
ZenPackers Documentation
Impact Outline
The Basic idea behind Impact is as follows:
• Identify what devices are dependent on one another.
– Create a visual diagram that shows dependency
– Make sure you understand how a component or device failure will affect other systems.
– In your base classes you have defined your _relations which can be (ToOne, ToMany, ToManyCont, etc).Example minus Boilerplate:
In Instance.py:
_relations = _relations + (('Instance_host', ToOne(
ToManyCont,'Products.ZenModel.Device.Device','oracle_instances')),
('oracle_tablespaces', ToManyCont(ToOne,'ZenPacks.zenoss.DatabaseMonitor.TableSpace.
→˓TableSpace','instance')),
)
In TableSpace.py:
_relations = _relations + (('instance', ToOne(ToManyCont,
'Products.ZenModel.Instance.Instance','oracle_tablespaces'),
),)
• Define the depenency classes for your ZP
– You need to define a class object that summarizes the depency list for each device or component.
– This is done with inheritance from BaseRelationsProvider
– There is a tall bit of boilerplate code in this example_
– For example: class InstanceRelationsProvider(BaseRelationsProvider)
– Here is an example (minus boilerplate):
# Give Impact (one-direction => ) dependencies for Devicesclass DeviceRelationsProvider(BaseRelationsProvider):
impact_relationships = ( 'oracle_instances',)
# Give Impact the (bi-directional<=> ) dependencies for Instancesclass InstanceRelationsProvider(BaseRelationsProvider):
impacted_by_relations = ( 'Instance_host',)impact_relationships = ( 'oracle_tablespaces',)
# Tell Impact the (one-directional <= ) dependencies of TableSpacesclass TableSpaceRelationsProvider(BaseRelationsProvider):
impacted_by_relationships = ( 'instance',)
• Now that the dependencies are made, you can register this code with Impact:
5.7. Impact Guide 111
ZenPackers Documentation
– Create an impact.zcml file: Yes, it is XML.
– Populuate for .Device.Device, .Instance.Instance or .MyModule.MyClass entries:
– Here is an example for DatabaseMonitor:
<?xml version="1.0" encoding="utf-8"?><configure
xmlns="http://namespaces.zope.org/zope"xmlns:browser="http://namespaces.zope.org/browser"xmlns:zcml="http://namespaces.zope.org/zcml">
<!-- API: Info Adapters -->... boilderplate ...
<!-- Impact --><include package="ZenPacks.zenoss.Impact" file="meta.zcml"/>
<subscriberprovides="ZenPacks.zenoss.Impact.impactd.interfaces.
→˓IRelationshipDataProvider"for="Products.ZenModel.Device.Device"factory=".impact.DeviceRelationsProvider"/>
<subscriberprovides="ZenPacks.zenoss.Impact.impactd.interfaces.
→˓IRelationshipDataProvider"for=".Instance.Instance"factory=".impact.InstanceRelationsProvider"/>
<subscriberprovides="ZenPacks.zenoss.Impact.impactd.interfaces.
→˓IRelationshipDataProvider"for=".TableSpace.TableSpace"factory=".impact.TableSpaceRelationsProvider"/>
</configure>
Boiler Plate Code Example
############################################################################### Boiler Plate Code for Impact! file: impact.py##############################################################################
from ZenPacks.zenoss.XenServer import ZENPACK_NAMEfrom ZenPacks.zenoss.XenServer.utils import guid
# Lazy imports to make this module not require Impact.ImpactEdge = NoneTrigger = None
# Constants to avoid typos.AVAILABILITY = 'AVAILABILITY'
(continues on next page)
112 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
PERCENT = 'policyPercentageTrigger'THRESHOLD = 'policyThresholdTrigger'DOWN = 'DOWN'DEGRADED = 'DEGRADED'ATRISK = 'ATRISK'
def edge(source, target):'''Create an edge indicating that source impacts target.
source and target are expected to be GUIDs.'''# Lazy import without incurring import overhead.# http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Import_Statement_
→˓Overheadglobal ImpactEdgeif not ImpactEdge:
from ZenPacks.zenoss.Impact.impactd.relations import ImpactEdge
return ImpactEdge(source, target, ZENPACK_NAME)
class BaseImpactAdapterFactory(object):'''Abstract base for Impact adapter factories.'''
def __init__(self, adapted):self.adapted = adapted
def guid(self):if not hasattr(self, '_guid'):
self._guid = guid(self.adapted)
return self._guid
class BaseRelationsProvider(BaseImpactAdapterFactory):'''Abstract base for IRelationshipDataProvider adapter factories.'''
relationship_provider = ZENPACK_NAME
impact_relationships = Noneimpacted_by_relationships = None
def belongsInImpactGraph(self):return True
def impact(self, relname):relationship = getattr(self.adapted, relname, None)if relationship and callable(relationship):
related = relationship()if not related:
return(continues on next page)
5.7. Impact Guide 113
ZenPackers Documentation
(continued from previous page)
try:for obj in related:
yield edge(self.guid(), guid(obj))
except TypeError:yield edge(self.guid(), guid(related))
def impacted_by(self, relname):relationship = getattr(self.adapted, relname, None)if relationship and callable(relationship):
related = relationship()if not related:
return
try:for obj in related:
yield edge(guid(obj), self.guid())
except TypeError:yield edge(guid(related), self.guid())
def getEdges(self):if self.impact_relationships is not None:
for impact_relationship in self.impact_relationships:for impact in self.impact(impact_relationship):
yield impact
if self.impacted_by_relationships is not None:for impacted_by_relationship in self.impacted_by_relationships:
for impacted_by in self.impacted_by(impacted_by_relationship):yield impacted_by
class BaseTriggers(BaseImpactAdapterFactory):'''Abstract base for INodeTriggers adapter factories.'''triggers = []
def get_triggers(self):'''Return list of triggers defined by subclass' triggers property.'''# Lazy import without incurring import overhead.# http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Import_Statement_
→˓Overheadglobal Triggerif not Trigger:
from ZenPacks.zenoss.Impact.impactd import Trigger
for trigger_args in self.triggers:yield Trigger(self.guid(), *trigger_args)
# ------------------------------------------------------------------------#""" The critical part of Impact: We define the impact relations """
(continues on next page)
114 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
# ------------------------------------------------------------------------#
# This tells Impact what (bi-directional) dependencies of Devices for this ZPclass DeviceRelationsProvider(BaseRelationsProvider):
impact_relationships = ( 'oracle_instances',)
# Tell Impact of the (bi-directional) dependencies Instances for this ZPclass InstanceRelationsProvider(BaseRelationsProvider):
impacted_by_relationships = ( 'Instance_host',)impact_relationships = ( 'oracle_tablespaces',)
# Tell Impact of the (bi-directional) dependencies of TableSpaces for this ZPclass TableSpaceRelationsProvider(BaseRelationsProvider):
impacted_by_relationships = ( 'instance',)
Show Impacts for Thing
This is some sample code that shows impacts on an object:
from zope.component import subscribersfrom Products.ZenUtils.guid.interfaces import IGUIDManagerfrom ZenPacks.zenoss.Impact.impactd.interfaces import IRelationshipDataProvider
def show_impacts_for(thing):guid_manager = IGUIDManager(thing.getDmd())
for subscriber in subscribers([thing], IRelationshipDataProvider):print "%s:" % subscriber.relationship_providerfor edge in subscriber.getEdges():
source = guid_manager.getObject(edge.source)impacted = guid_manager.getObject(edge.impacted)print " %s (%s) -> %s (%s)" % (
source.id, source.meta_type, impacted.id, impacted.meta_type)print
show_impacts_for(find("VACC").os.interfaces._getOb('VLAN0200'))
Installing Impact on 5.X Zendev
https://github.com/zenoss/impact-server/wiki/Running-Impact-Server-in-development-mode-(5.x)
5.7.4 External Device Impacts
Sometimes its important to connect your device components to an external device. The typical example is connectinga VM from a cluster to its related /Server/SSH/Linux device. We’ll use ZPL for the relations.
The Impact relations are:
• NutanixVM ->(impacts) LinuxMonitor device.
• LinuxMonitor <-(impacted_by) NutanixVM component.
5.7. Impact Guide 115
ZenPackers Documentation
Recall that we need relations to be created on both sides for Impact to be happy. First we need to concentrate onLinuxMonitor side.
Setup the RelationsProvider LinuxDevice <-(impacted_by) NutanixVM
Note: Both relationships require device and component to be in Zenoss global catalog. In this case we match onMAC addresses. Notice the TAG_IMPACTED_BY in self.constructRelationTo()
In your ZP setup dynamicview.py in this way:
# dynamicview.pyimport logginglog = logging.getLogger("zen.Nutanix.dynamicview")
from ZenPacks.zenoss.DynamicView import TAG_ALL, TAG_IMPACTED_BYfrom ZenPacks.zenoss.DynamicView.model.adapters import BaseRelationsProviderfrom ZenPacks.zenoss.ZenPackLib.lib.functions import catalog_search
# This class creates the relationship on the LinuxMonitor sideclass ExternalDeviceRelationsProvider(BaseRelationsProvider):
def relations(self, type=TAG_ALL):if type in (TAG_ALL, TAG_IMPACTED_BY):
vm = nutanixVM(self._adapted)if vm:
yield self.constructRelationTo(vm, TAG_IMPACTED_BY)
# This method on LinuxMonitor returns the NutanixVM corresponding to its MACdef nutanixVM(device):
"""Return the first VM found that corresponds to device's MACs."""
# Get rid of bogus MACs like 00:00:00:00:00:00macs = [m for m in device.getMacAddresses() if m != '00:00:00:00:00:00']if macs:
for catalog_vm in catalog_search(device.dmd.Devices,'ZenPacks_zenoss_Nutanix_NutanixVM',mac_addresses=macs):
try:return catalog_vm.getObject()
except Exception:# Ignore stale catalog entries and continuecontinue
Now you need to register this in your configure.zcml:
<!-- DynamicView --><configure zcml:condition="installed ZenPacks.zenoss.DynamicView">
<include package="ZenPacks.zenoss.DynamicView" file="meta.zcml"/>
<subscriberprovides="ZenPacks.zenoss.DynamicView.interfaces.IRelationsProvider"for="Products.ZenModel.Device.Device"factory=".dynamicview.ExternalDeviceRelationsProvider"/>
(continues on next page)
116 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
</configure>
Setup the RelationsProvider NutanixVM -> LinuxDevice
In your zenpack.yaml file we need several things to happen:
1. The property for NutanixVM must be cataloged.
2. Setup the relationship: NutanixVM -> LinuxMonitor(guestDevice)
Here is an example:
NutanixVM:label: VMproperties:mac_addresses:
label: MACstype: linesgrid_display: falseindex_scope: global # This is requiredindex_type: keyword # This is required too
... etc ...
dynamicview_relations:impacts: [guestDevice] # Yes, this is the one we need.impacted_by: [nutanixCluster, nutanix_vdisks]
Since guestDevice is a method on NutanixVM, we need to specify that method in the class file as follows:
# Helper function to find interfaces belonging to LinuxMonitor devicedef findIpInterfacesByMAC(dmd, mac_addresses):"""Yield IpInterface objects that match the MAC addresses."""if not mac_addresses:
return
layer2_catalog = dmd.ZenLinkManager._getCatalog(layer=2)if layer2_catalog is not None:
for result in layer2_catalog(macaddress=mac_addresses):try:
iface = result.getObject()except Exception:
# ignore a stale catalog entrycontinue
else:yield iface
class NutanixVM(schema.NutanixVM):"""Methods for NutanxVM components"""def guestDevice(self):
"""Return the first device found that corresponds to a MAC"""for interface in findIpInterfacesByMAC(self.dmd, self.mac_addresses):
return interface.device()
5.7. Impact Guide 117
ZenPackers Documentation
Indexing the Objects
Once you have the pieces in place you need to do the following:
1. Re-Install the ZenPack
2. Remove the components associated with the Impact chain, because Impact only re-indexes objects that are newor modified.
3. Remodel the device associated with the ZenPack
4. Go to the Services tab and create a new Dynamic Service
• Add the Component from the ZenPack
• Add the Device assocated to the component
• Go to the Impact View and see if they are now connected
5. If you have problems consult Debugging Impact to manually check your Impact relations.
5.7.5 A Framework for Testing DynamicView and Impact (DVI)
DynamicView and Impact can share the same relationships in ZPL if you desire. They can also be different. How doesone handle Unit Tests for either feature?
In this section we discuss how to test both Impact and DyanamicView using the same overall framework. We do thisusing a module called dvi.py that we will share later.
Using the DVI Framework
Lets first start by showing how to use the DVI framework in the simplest case. We’ll use CiscoUCS as an example.The actual code will differ if we use DynamicView version < 1.5. You can see the actual code for the differences.
The first step is the define the device and components you want to test. This is done in this way:
from Products.DataCollector.plugins.DataMaps import RelationshipMap# Devices against which the above relationships will be tested.DEVICES = {
"ucs1": {"deviceClass": "/CiscoUCS/UCS-Manager","zPythonClass": "ZenPacks.zenoss.CiscoUCS.Device","dataMaps": [
RelationshipMap(modname="ZenPacks.zenoss.CiscoUCS.ServiceProfile",relname="serviceprofiles",objmaps=[{
"id": "sp1","wwns": [
"20:00:00:25:B5:00:02:FF","20:00:00:25:B5:00:02:DF",]
}]),
RelationshipMap(modname="ZenPacks.zenoss.CiscoUCS.VnicFc",compname="serviceprofiles/sp1",relname="vnicfcs",
(continues on next page)
118 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
objmaps=[{"id": "vnicfc-1","wwnn": "20:00:00:25:B5:00:02:FF","wwpn": "20:00:00:25:B5:00:02:DF",}]),
]},}
Clearly there is going to be some sort of DVI relationship between VnicFC and ServiceProfile. We’ve already coveredhow those are made before, so we omit those details. Rest assures that both ZPL and old-style DVI relations will workin general.
Next we define the relationships to be defined using YUML notation. Notice that we can define DVI relations for bothDV and Impact, or separate them out:
# Relationships expected in DynamicView and Impact.EXPECTED_BOTH = """// UCS Manager Device[ucs1/vnicfc-1]->[ucs1/sp1]"""
# Relationships only expected in DynamicView.EXPECTED_DYNAMICVIEW_ONLY = """"""
# Relationships only expected in Impact.EXPECTED_IMPACT_ONLY = """"""
Next we define the class that will test the above code. Note that we add imports near the code for illustration purposes:
from ZenPacks.zenoss.CiscoUCS.tests import dviclass TestManager_DVI(dvi.DVITest):
device_data = DEVICESexpected_both = EXPECTED_BOTHexpected_dynamicview_only = EXPECTED_DYNAMICVIEW_ONLYexpected_impact_only = EXPECTED_IMPACT_ONLY
def afterSetUp(self):
# Load superclass last so that EMC components exist prior to testing.super(TestManager_DVI, self).afterSetUp()
The entire (simplified) code looks like this:
############################################################################### Copyright (C) Zenoss, Inc. 2016, all rights reserved.##############################################################################"""Test suite for DynamicView and Impact.Test the UCS Manager DynamicView/Impact relationships using dvi module. """
from Products.DataCollector.plugins.DataMaps import RelationshipMap
from ZenPacks.zenoss.CiscoUCS.tests import dvi
# Relationships expected in DynamicView and Impact.(continues on next page)
5.7. Impact Guide 119
ZenPackers Documentation
(continued from previous page)
EXPECTED_BOTH = """// UCS Manager Device[ucs1/vnicfc-1]->[ucs1/sp1]"""
# Relationships only expected in DynamicView.EXPECTED_DYNAMICVIEW_ONLY = """"""
# Relationships only expected in Impact.EXPECTED_IMPACT_ONLY = """"""
# Devices against which the above relationships will be tested.DEVICES = {
"ucs1": {"deviceClass": "/CiscoUCS/UCS-Manager","zPythonClass": "ZenPacks.zenoss.CiscoUCS.Device","dataMaps": [
RelationshipMap(modname="ZenPacks.zenoss.CiscoUCS.ServiceProfile",relname="serviceprofiles",objmaps=[{
"id": "sp1","wwns": [
"20:00:00:25:B5:00:02:FF","20:00:00:25:B5:00:02:DF",]
}]),
RelationshipMap(modname="ZenPacks.zenoss.CiscoUCS.VnicFc",compname="serviceprofiles/sp1",relname="vnicfcs",objmaps=[{
"id": "vnicfc-1","wwnn": "20:00:00:25:B5:00:02:FF","wwpn": "20:00:00:25:B5:00:02:DF",}]),
]},}
class TestManager_DVI(dvi.DVITest):
device_data = DEVICESexpected_both = EXPECTED_BOTHexpected_dynamicview_only = EXPECTED_DYNAMICVIEW_ONLYexpected_impact_only = EXPECTED_IMPACT_ONLY
def afterSetUp(self):# Load superclass last so that EMC components exist prior to testing.super(TestManager_DVI, self).afterSetUp()
This code when run will show errors in YUML format that look like:
120 Chapter 5. General Zenpack Development
ZenPackers Documentation
[ucs1/]!!!-.-!!!>[ucs1/sp1] # This is a 2-way missing relation[ucs1/]-.-!!!>[ucs1/sp1] # This is a 1-way missing relation[ucs1/]-.-XXX>[ucs1/vnicfc-1] # This is a 1-way extra relation[ucs1/]XXX-.-XXX>[ucs1/vnicfc-1] # This is a 2-way extra relation[ucs1/]->[ucs1/vnicfc-1] # This relation is correct...
The YUML will show full 1-way or 2-way relation deficiencies. It will also show extra relations that don’t belong. Ineach case you must determine why your relations are broken and find a way to fix them.
When the code and relations are correct, you will simply see a passing test.
Sometimes its easiest to start with all empty relations, run the code and use the YUML output to scrape the relationshipsyou need, already formatted for you except for all the !!!’s. Just change those !!!-.-!!!> into -> and you’ll have a baseset of relations to start with.
Adding External Devices to Test
You can add external devices as well and test those. A simple example might look like:
############################################################################### Copyright (C) Zenoss, Inc. 2016, all rights reserved.##############################################################################"""Test suite for DynamicView and Impact.Test the UCS Manager DynamicView/Impact relationships using dvi module. """
from Products.DataCollector.plugins.DataMaps import RelationshipMap
from ZenPacks.zenoss.CiscoUCS.tests import dvi
# Relationships expected in DynamicView and Impact.EXPECTED_BOTH = """// UCS Manager Device[ucs1/vnicfc-1]->[ucs1/sp1]"""
# Relationships only expected in DynamicView.EXPECTED_DYNAMICVIEW_ONLY = """"""
# Relationships only expected in Impact.EXPECTED_IMPACT_ONLY = """"""
# Devices against which the above relationships will be tested.DEVICES = {
"ucs1": {"deviceClass": "/CiscoUCS/UCS-Manager","zPythonClass": "ZenPacks.zenoss.CiscoUCS.Device","dataMaps": [
RelationshipMap(modname="ZenPacks.zenoss.CiscoUCS.ServiceProfile",relname="serviceprofiles",objmaps=[{
"id": "sp1","wwns": [
(continues on next page)
5.7. Impact Guide 121
ZenPackers Documentation
(continued from previous page)
"20:00:00:25:B5:00:02:FF","20:00:00:25:B5:00:02:DF",]
}]),
RelationshipMap(modname="ZenPacks.zenoss.CiscoUCS.VnicFc",compname="serviceprofiles/sp1",relname="vnicfcs",objmaps=[{
"id": "vnicfc-1","wwnn": "20:00:00:25:B5:00:02:FF","wwpn": "20:00:00:25:B5:00:02:DF",}]),
]},}
# Load ZenPacks.zenoss.EMC.base devices and data if the ZP is installed.try:
from ZenPacks.zenoss.EMC.base import SMISProviderexcept ImportError:
print("ZenPacks.zenoss.EMC.base is not installed. Skipping EMC.base tests.")else:
DEVICES["emc"] = {"deviceClass": "/Storage/EMC/VNX","zPythonClass": "ZenPacks.zenoss.EMC.base.SMISProvider","dataMaps": [
RelationshipMap(modname="ZenPacks.zenoss.EMC.base.LUN",relname="luns",compname="os",objmaps=[{
"id": "emc-lun1","wwn": "600601607FC038005890F60865EBE311","initiator_ids": [
"20000025B500021F20000025B50003BF","20000025B500021F20000025B50003DF","20000025B50002FF20000025B50002DF","20000025B50002FF20000025B50002EF","20000025B500033F20000025B500030F","20000025B500033F20000025B500031F","20000025B500034F20000025B500032F","20000025B500034F20000025B500035F"],
}]),]
}
EXPECTED_BOTH += """[emc/emc-lun1]->[ucs1/sp1]"""
class TestManager_DVI(dvi.DVITest):
device_data = DEVICESexpected_both = EXPECTED_BOTH
(continues on next page)
122 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
expected_dynamicview_only = EXPECTED_DYNAMICVIEW_ONLYexpected_impact_only = EXPECTED_IMPACT_ONLY
def afterSetUp(self):# Load EMC.base DV adapters manually since it lacks ZPL capability.try:
import ZenPacks.zenoss.EMC.basezcml.load_config('configure.zcml', ZenPacks.zenoss.EMC.base)
except:pass
# Load superclass last so that EMC components exist prior to testing.super(TestManager_DVI, self).afterSetUp()
Note that we added to DEVICES and the EXPECTED_BOTH in a simple way.
All that is left is to show you the code for dvi.py. You can find that in ZenPacks.zenoss.CiscoUCS.tests.dvi . There isa more complex example there too in testCIMC_DVI.py
5.8 DynamicVeiw Topics
DynamicView uses similar concepts from Impact. As you have seen the DynamicView and impact adatpers can bedefined in a unified way.
In this section we only use ZPL: https://zenpack-sdk.zenoss.com/en/latest/ implementation.
5.8.1 Adjusting Order of Component Diagrams
Sometimes your DyanamicView diagrams will not be ordered properly from right to left. In order to get the properorder you can set dynamicview_weight on each component class:
dynamicview_weight: 220dynamicview_relations:
impacted_by: [k8sNamespace]
Make sure you order your weights from low to high corresponding to least impacted to most impacted components.For example, your device would probably be the most impacted object, and thus have the highest weight.
5.8.2 Creating a Device-Level DynamicView Diagram
Normally when you select the DynamicView link on the Navigation panel, you would only see the device object inisolation. This is because it isn’t usually impacted by any of its components.
In order to see all objects that impact the device, you have to do two things:
1. Add JavaScript at the device-level that defines links and views
2. Adjust your zenpack.yaml file
3. Restart Zope to test
5.8. DynamicVeiw Topics 123
ZenPackers Documentation
Add JavaScript at the device-level that defines links and views
Inside of your device-level anonymous function add the following adjustProjectDynamicView() method to the globalcontext, and then activate it in the Ext.onReady(function() method:
(function () {/*** Adjust device-level DynamicView.
** We want to use k8s_view/k8s_link instead of service_view/impacted_by.
* Otherwise the project-level Dynamic View would be just the boring
* project node because it isn't impacted by anything.
*/function adjustProjectDynamicView() {
var COMPONENT_VIEW_ID = "component_view";
Zenoss.nav.onAvailable("Device", function(collection) {if (collection.containsKey(COMPONENT_VIEW_ID)) {
// Handle when component_view is added before we run.collection.replace(COMPONENT_VIEW_ID, {
id: "component_view",text: _t('Dynamic View'),xtype: 'dynamicview',relationshipFilter: 'k8s_link',viewName: 'k8s_view'
});} else {
// Handle when component_view is added after we run.collection.addListener("add", function(index, object, key){
if (key == COMPONENT_VIEW_ID) {object.viewName = "k8s_view";object.relationshipFilter = "k8s_link";
}});
}});
}
// Other definitions etc...
Ext.onReady(function(){// Use k8s_view/k8s_link for device-level Dynamic View.adjustProjectDynamicView();
// Other component level Ext.* methods.
});}());
Adjust your zenpack.yaml
Now comes the easy part where you add some entries to your YAML. Note that we’ll be adding entries for the linkcalled k8s_link and a view for the k8s_view. Each visible component and device in DV must be part of the k8s_view:
classes:
(continues on next page)
124 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
K8sCluster:label: Kubernetes Cluster
properties:cluster_uid:
label: UID
dynamicview_views: [service_view, k8s_view]dynamicview_weight: 240dynamicview_relations:
impacts: [k8sNodes, k8sNamespaces, gcp_cluster]k8s_link: [k8sNodes, k8sNamespaces]
K8sNamespace:label: NameSpace
properties:namespace_uid:
label: UID
dynamicview_views: [service_view, k8s_view]dynamicview_weight: 225dynamicview_relations:
impacted_by: [k8sCluster]impacts: [k8sPods, k8sServices, k8sPersistentVolumes]k8s_link: [k8sServices]
K8sService:label: Service
properties:service_uid:
label: UID
dynamicview_views: [service_view, k8s_view]dynamicview_weight: 215dynamicview_relations:
impacts: [k8sPods]impacted_by: [k8sNamespace]
Note: Note that K8sService, while not defining any k8s_link, is referenced by K8sNamespace and thus needs thedynamicview_views line to allow K8sService to be visible in the DV display.
• Now you can restart Zope to test your setup.
5.8.3 Troubleshooting: You see a DV component that looks out of sorts!
If you see a component that looks like its out of place or wrong, it could be that its Plural Label matches that ofanother device or component. If so, you can set a unique plural label in ZPL like this:
K8sContainer:label: Containerplural_short_label: K8s Containers
5.8. DynamicVeiw Topics 125
ZenPackers Documentation
In this example we know that there are several ZenPacks with a component whose label is called Container, whichcould cause DV to render that object instead of the intended K8s entity. Adding the plural_short_label fixes thisproblem.
5.9 Modeling for Zenpacks: Selected Topics
Modeling is an integral part of Zenoss. This article explores specific tasks related to modeling.
5.9.1 Prerequisites
• Zenoss ZenPack Developement Guide
We assume that you are familiar with ZenPack developement and Python coding. We further assume that we workfrom the base of ZP_DIR. For NetBotz for example:
ZP_DIR_TOP=$ZENHOME/ZenPacks/ZenPacks.training.NetBotzZP_DIR=$ZP_DIR_TOP/ZenPacks/training/NetBotz
As you should know, modelers typically live in the folder:
$ZP_DIR/modeler/plugins/zenoss/MyModeler.py
5.9.2 Models in General
Making or changing a model in Zenoss means that you are modifying the ZoDB representation of your model. ZoDBcreates a hierarchical model of
• Containing Relations:
Choose this type when the sub-object can’t live without its parent. Ex: A DB table MUST live inside a database.Its stuck inside of the DB.
Objects and sub-objects are contained within each other. When the parent object is removed, its children aretoo. Its hierarchical. Its dangerous! (not really)
• Non-Containing Relations
You choose this type of relation when an object can be moved or exist without its partner relation. Example: Ahost is associated with a service. But the service can be moved, so its not “stuck” to that host.
Objects are loosely associated with each other. Deletion of one object does not force removal of a related object.
5.9.3 Developing Tips
Given the information above you need to understand a few critical points of developing. The ZoDB model represen-tation is sensitive to change and must be handled carefully. If you damage the ZoDB it may cause many hours ofrepair.
We discuss two major types of model development: Adding and Modifying.
• Adding to a Model
When you add to a model, new relations are safely inserted into ZoDB. This means that you can update theZoDB model for your device by doing:
126 Chapter 5. General Zenpack Development
ZenPackers Documentation
– Make your model changes in the code
– Insert those changes into Zenoss; Re-install the ZP or similar
– Restart services: zenjobs, zenhub, zope (all python daemons that talk to zodb)
– In zendmd, execute:
device = find('mydev.xyz')device.buildRelations()commit()
– Model your device if required
– Test
– Repeat
• Changing a Model
Changing a model is more serious, because if done improperly, you end up with relations on a device that don’texist in the ZP model. This makes ZoDB very angry. To make model changes safely, you must do something ofthis sort:
– Remove your ZP and associated devices
– Make your model changes
– Install the ZP or otherwise install your model to ZoDB
– Model your device
– Test
– Repeat
Warning: Changing a model for a ZP that has been deployed to the field should be avoided at all costs. Askingthousands of customers to wipe out their zenpacks and re-install is a major inconvenience to them.
5.9.4 Debugging Tips in General
Note: If you set zCollectorLogChanges to True on the device, remodel, then look for closed info severity events inthe /Change event class, you’ll see specific model changes.
• Run the modeler manually like this:
zenmodeler run -workers=0 -v10 -d mp3.zenoss.loc |& tee mod.txt
• If you don’t get anything modeled at all you can try this:
– Restart zenhub: it may have given up loading the modeler
– Rerun the zenmodeler command above and also monitor the zenhub log in /opt/zenoss/log/zenhub.log forgood measure.
• Restarting Services: If you make any changes to your modeler code, you SHOULD restart ZenHub whichcaches that code at start-time.
5.9. Modeling for Zenpacks: Selected Topics 127
ZenPackers Documentation
Note: You really need to restart ZenHub because if you don’t and your code fails, it will raise an error but referencethe new code. Thus, it will not show or reference the broken (cached) code in the logs. This can be confusing.
Deleting Components from a Device
Warning: When you re-model a device or component, if the returned configuration of those devices/componentsdo not change, ZenHub will *NOT* trigger an event to update any device settings. I other words, if you want to re-model and test a device/component, you *MUST* remove that device/component from Zenoss before attemptingany remodel it; otherwise you won’t see any changes.
Often you want to remodel a device and don’t have access to the GUI. In order to remove components via Zendmd youcan use the following:
device = find("mp6.zenoss.loc")for component in device.getDeviceComponents():
component.getPrimaryParent()._delObject(component.id)
commit()
5.9.5 General Introduction
Modeler classes generally have two methods that are used by the zenmodeler service. They are:
• collect(): This method collects the data in an asychronous way. It returns a dict called results Its signature istypically:
@inlineCallbacksdef collect(self, device, log)
....returnValue(results)
• process(self, device, results, log): This method (asynchronously) takes that results dict uses it to populate thedevice model. It has a signature resembling:
def process(self, device, results, log):'''results comes back from collect via twisted.'''
for label, data in results.items():... set your class instance values .............
..... create relationmaps between object instances .....return relationmaps
5.9.6 Major Concepts of Modeling
There are two major parts of Modeling that you must always keep in mind:
• Model Definition
– Base classs must define relationships
128 Chapter 5. General Zenpack Development
ZenPackers Documentation
– Both Containing and Non-Containing are possible
– The relationship *MUST* be created in both directions
• Model Mapping/Creation/Population
– Modeler must correctly associate the devices and component
– The maps are created/defined in one direction only!
– ApplyDataMap() takes care of the bi-directional associations
Model Definition
In our example we’ll use two classes: Instance and TableSpace from the Oracle ZP. Instance is a component off ofDevice.Device, and Tablespace will hang off of Instance. We need two defining relation:
• In Instance() we need two relations. The first binds Instance to Device.Device and the second give us multipleTableSpace to Instance:
# Generic relations (from ZP Generator)_relations = ()for Klass in Klasses:
_relations = _relations + getattr(Klass, '_relations', ())
# These are the ones we need to define:_relations = _relations + (
('Instance_host',ToOne(ToManyCont,
'Products.ZenModel.Device.Device','oracle_instances',)),
('oracle_tablespaces',ToManyCont(
ToOne,'ZenPacks.zenoss.DatabaseMonitor.TablesSpace.TableSpace','instance',)),
)
• In TableSpace() we need just one to define Instance -> TablesSpaces:
# Generic relations (from ZP Generator)_relations = ()for Klass in Klasses:
_relations = _relations + getattr(Klass, '_relations', ())
# This is the one we define._relations = _relations + (
('instance', ToOne(ToManyCont,'Products.ZenModel.Instance.Instance','oracle_tablespaces',),
),)
Model Mapping/Creation/Population
We now discuss what is in your modeler (in our example, Instance) class.
5.9. Modeling for Zenpacks: Selected Topics 129
ZenPackers Documentation
In the collect() method in your modeler, we assume you have collected all the required data and stored in in the resultsdictionary. How you do that is fairly general.
In our Instance modeler’s process method, we will first create a temporary storage dictionary called datamap, whichhas Instance as a key and a list of TableSpace objects as the values. Once this datamap is created and populated, wethen iterate through it to setup the RelationshipMap() structures.
To set this up we first loop through the results data to create the temporary datamap:
for connectionString, data in results.items():
instance1 = data['instance'][0]instance_name = instance1.get('INSTANCE_NAME')ts_list = data['tablespaces']
om = self.objectMap()om.id = self.prepId('orainst-%s' % instance_nameom.title = instance_name
tablespaces = []for ts in ts_list:
tablespaces.append(ObjectMap(data=dict(id='{0}_{1}'.format(instance_name, ts['TABLESPACE_NAME'])tablespace_name = ts['TABLESPACE_NAME'],tablespace_instance = instance1.get('INSTANCE_ROLE'),tablespace_maxbytes = ts['BYTES_MAX'],
)))
# Add to map: Map the om object to the tsdatamap[om] = tablespaces
So now you have your datamap setup. Its only used to feed our RelationshipMap. Notice that in this example we must:
1. Get the list of Instances outside the loop using the dict.keys() for the Instance -> Device.Device relation.
2. We need to then loop over the Instances to attache the assiciated TableSpace list objects
#------------------------------------------------------------------# Now loop over objects to create relation maps.#------------------------------------------------------------------
relmaps = []
relmaps.append(RelationshipMap(relname='oracle_instances',modname='ZenPacks.zenoss.DatabaseMonitor.Instance',objmaps=datamap.keys()))
for inst, ts in datamap:print type(inst), type(ts)
relmaps.append(RelationshipMap(compname='oracle_instances/{0}'.format(inst.id),relname='oracle_tablespaces',modname='ZenPacks.zenoss.DatabaseMonitor.TableSpace',objmaps=ts))
(continues on next page)
130 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
log.info('%s: %s instances found', device.id, len(relmaps))return relmaps
This is a simple example. To see this how this was implemented see the ZenPacks.zenoss.DatabaseMonitor’s modelerplugin.
To see other examples:
• ZenPacks.zenoss.PostgreSQL (simpler)
• ZenPacks.zenoss.XenServer (more complex)
5.9.7 Miscellaneous Tasks
Deleting a Device
Open zendmd and remove the device:
[zenoss@mp4]: zendmddevice = find('xyz.zenoss.loc')device.deleteDevice()commit()
Deleting Components from a Device
This might be needded if you want to remodel a device and don’t have access to the GUI:
[zenoss@mp4]: zendmddevice = find("mp6.zenoss.loc")for component in device.getDeviceComponents():
component.getPrimaryParent()._delObject(component.id)commit()
Finding Device Components with IInfo
You an find a device’s components using the IInfo interface:
[zenoss@mp4]: zendmddevice = find("mp6.zenoss.loc")from Products.Zuul.interfaces import IInfodeviceinfo = IInfo(device)deviceinfo<ControlCenter Info "mp6.zenoss.loc">dir(deviceinfo)
If you have a component object:
for component in device.getDeviceComponents(): break # Just to get onefrom Products.Zuul.interfaces import IInfo
cinfo = IInfo(component)dir(cinfo)
5.9. Modeling for Zenpacks: Selected Topics 131
ZenPackers Documentation
Get Templates and Thresholds
You can to the templates with a Facade:
tfc=getFacade('template')tfc.getTemplates('/zport/dmd/Devices/DB2/devices/xyz.zenoss.loc/hosts/host-5/CP-Host')<generator object _getTemplateLeaves at 0x7ddfcd0>
list = tfc.getTemplates('/zport/dmd/Devices/DB2/devices/xyz.zenoss.loc/hosts/host-5/→˓CP-Host')for i in list:
print i
<RRDTemplate Info "CP-Host..ControlCenter.devices.mp6.zenoss.loc.mp6.zenoss.loc">
list = tfc.getThresholds('/zport/dmd/Devices/DB2/devices/xyz.zenoss.loc/hosts/host-5/→˓CP-Host')for i in list:
print i
5.9.8 Miscellaneous Errors and Debugging
Error: No Classifier Found, KeyError
If you get an error this this nature:
2014-02-06 13:59:01,678 DEBUG zen.Classifier: No classifier defined2014-02-06 13:59:01,814 ERROR zen.ZenModeler: : Traceback (most recent call last):
File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 85, in innerreturn callable(*args, **kw)
File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 132, in remote_→˓applyDataMaps
result = inner(map)File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 128, in innerreturn self._do_with_retries(action)
File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 154, in _do_→˓with_retries
return action()File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 127, in actionreturn bool(adm._applyDataMap(device, map))
File "/opt/zenoss/lib/python/ZODB/transact.py", line 44, in gr = f(*args, **kwargs)
File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 202, in _→˓applyDataMap
tobj = device.getObjByPath(datamap.compname)File "/opt/zenoss/Products/ZenModel/ZenModelBase.py", line 624, in getObjByPathreturn getObjByPath(self, path)
File "/opt/zenoss/Products/ZenUtils/Utils.py", line 299, in getObjByPathnext=obj[name]
File "/opt/zenoss/lib/python/OFS/ObjectManager.py", line 777, in __getitem__raise KeyError, key
KeyError: 'db2_databases': <no traceback>Traceback (most recent call last):
File "/opt/zenoss/Products/DataCollector/zenmodeler.py", line 693, in processClientif driver.next():
(continues on next page)
132 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
File "/opt/zenoss/Products/ZenUtils/Driver.py", line 63, in resultraise ex
you probably have a problem where ZODB does not have a relationship map built to handle your data structure. Thiscan happen if:
• You forgot to restart Zenoss services after installing the ZP, then added a device. Such a device won’t have yourZP’s deviceClass available to it.
• The device structure was changed after the ZP was installed, while the old relationship map still persists.
• Your ZenPack failed to execute buildRelations() on your device.
• You somehow damaged the relations structure in ZODB.
You may be able to fix this in zendmd by issuing these commands:
[zenoss:~]: zendmd>>> d=find('mp3.zenoss.loc')>>> d.buildRelations()>>> commit()
Also, sometimes the deviceClass will be wrong/missing for same reasons above. You can fix this also in zendmd:
d = find('blah')d.buildRelations()if d.__class__ != d.deviceClass().getPythonDeviceClass():
d.changeDeviceClass(d.getDeviceClassPath())commit()
• Now try to remodel and see if those problems persist
DEBUG zen.Classifier: No classifier defined
This is a remnant of another ZP that never got implemented. Please ignore this one.
INFO zen.ZenModeler: No change in configuration detected (or similar)
So you’ve made changes to your ZP’s class structure and have pushed those changes out. You may have even re-installed the ZP for good measure. Your modeler seems to be working correctly and gathering data. But your modelerisn’t applying any changes!
If your modeler get this message after modeling, you could be a victim of ZenDMD Class Mismatch Syndrome (TM).This means that the old structure is still in place and so none of your changes are being compared to the new classstructure. There are 2 easy ways to fix this:
1. Completely remove and reinstall your ZP, now remodel.
2. Go into ZenDMD and simply load the new class, then remodel:
[zenoss@mp4:/home/zenoss]: zendmd
>>> from ZenPacks.zenoss.ExampleZP import ExampleZP>>> ^D[zenoss@mp4:/home/zenoss]: zenmodeler run -v10 -d mydev.zenoss.loc
5.9. Modeling for Zenpacks: Selected Topics 133
ZenPackers Documentation
TypeError: unhashable type: ‘dict’
You may see and error of the following type:
2014-07-28 17:02:51,109 ERROR zen.ZenModeler: : Traceback (most recent call last):File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 85, in inner
return callable(*args, **kw)File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 132, in remote_→˓applyDataMapsresult = inner(map)
File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 128, in innerreturn self._do_with_retries(action)
File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 154, in _do_with_→˓retriesreturn action()
File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 127, in actionreturn bool(adm._applyDataMap(device, map))
File "/opt/zenoss/lib/python/ZODB/transact.py", line 44, in gr = f(*args, **kwargs)
File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 213, in _applyDataMapchanged = self._updateRelationship(tobj, datamap)
File "/zenpacks/ZenPacks.zenoss.PythonCollector/ZenPacks/zenoss/PythonCollector/→˓patches/platform.py", line 36, in _updateRelationshipreturn original(self, device, relmap)
File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 265, in _→˓updateRelationshipobjchange = self._updateObject(obj, objmap)
File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 378, in _updateObjectchange = not isSameData(value, getter())
File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 53, in isSameDatax = set( tuple(sorted(d.items())) for d in x )
TypeError: unhashable type: 'dict': <no traceback>
The modeler is being passed data that is not a plain dict, string, int, or float. In this case it sees a <dict> key of itemand it doesn’t know how to handle it. ie: You’re trying to use a dict as a key to another dict or in a set.
The data you are passing to the modeler should be a dictionary type or a simple base type.
• One way to get around this is to ensure you are passing a dictionary object to the modeler.
• Another way is to serialize and pass in your data (perhaps with JSON). Of course you’ll have to de-serialize itwhen you need to use it.
TypeError: ‘str’ object is not callable
You added a component or object somewhere and have defined a “name” or “device” (or some other) object. You getan error as per example:
File "/opt/zenoss/Products/Zuul/catalog/global_catalog.py", line 290, in searchExcerpto.name(), o.device().titleOrId())
TypeError: 'str' object is not callable
What you have done was to create property on a reserved method, which is causing the modeler to think that its ownmethod is of wrong type.
Reserved words include:
134 Chapter 5. General Zenpack Development
ZenPackers Documentation
• api_backendtype
• api_only
• class_spec,
• content_width
• datapoint_cached
• datapoint_default
• datapoint
• default
• details_display
• device
• display
• editable
• enum
• grid_display
• index_scope
• index_type
• label
• label_width
• name
• order
• renderer
• short_label
• _source_location
• type
• type_
A full list of reserved words are found here: Reserved Words in Zenoss Modeler
TypeError(‘name() takes exactly 1 argument (2 given)’,)
You get this when modeling:
2015-10-13 16:00:05,044 ERROR zen.ZenModeler: Client cc.cupid finished with message:→˓[Failure instance: Traceback (failure with no frames): <class 'Products.ZenHub.→˓PBDaemon.RemoteException'>: : Traceback (most recent call last):File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 85, in inner
return callable(*args, **kw)File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 144, in remote_→˓applyDataMaps
result = inner(map)
(continues on next page)
5.9. Modeling for Zenpacks: Selected Topics 135
ZenPackers Documentation
(continued from previous page)
File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 140, in innerreturn self._do_with_retries(action)
File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 166, in _do_with_→˓retries
return action()File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 131, in action
completed= bool(adm._applyDataMap(device, map))File "/opt/zenoss/lib/python/ZODB/transact.py", line 44, in g
r = f(*args, **kwargs)File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 217, in _applyDataMap
changed = self._updateRelationship(tobj, datamap)File "/z/ZenPacks.zenoss.PythonCollector/ZenPacks/zenoss/PythonCollector/patches/→˓platform.py", line 38, in _updateRelationship
return original(self, device, relmap)File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 271, in _→˓updateRelationship
objchange = self._updateObject(obj, objmap)File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 402, in _updateObject
setter(*args)TypeError: name() takes exactly 1 argument (2 given)
There is a good chance you’ve tried to set an attribute using “name”. This is a reserved word. You probably meant touse “title” for your attribute.
AttributeError: networks
You see this when modeling:
2016-10-19 10:31:02,765 DEBUG zen.ZenModeler: Running 1 clients2016-10-19 10:31:02,770 ERROR zen.ZenModeler: : Traceback (most recent call last):File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 97, in inner
return callable(*args, **kw)File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 157, in remote_→˓applyDataMaps
result = inner(map)File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 153, in inner
return self._do_with_retries(action)File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 179, in _do_with_→˓retries
return action()File "/opt/zenoss/Products/ZenHub/services/ModelerService.py", line 144, in action
completed= bool(adm._applyDataMap(device, map))File "/opt/zenoss/lib/python/ZODB/transact.py", line 44, in g
r = f(*args, **kwargs)File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 217, in _applyDataMap
changed = self._updateRelationship(tobj, datamap)File "/z/ZenPacks.zenoss.PythonCollector/ZenPacks/zenoss/PythonCollector/patches/→˓platform.py", line 38, in _updateRelationship
return original(self, device, relmap)File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 280, in _→˓updateRelationship
objchange, obj = self._createRelObject(device, objmap, rname)File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 477, in _→˓createRelObject
(continues on next page)
136 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
up_changed = self._updateObject(remoteObj, objmap)File "/opt/zenoss/Products/DataCollector/ApplyDataMap.py", line 400, in _updateObject
setter(*args)File "/opt/zenoss/Products/ZenModel/IpInterface.py", line 335, in setIpAddresses
self.addIpAddress(ip)File "/opt/zenoss/Products/ZenModel/IpInterface.py", line 282, in addIpAddress
ipobj = networks.createIp(ip, netmask)File "/opt/zenoss/Products/ZenModel/IpNetwork.py", line 329, in createIp
netobj = self.createNet(ip, netmask)File "/opt/zenoss/Products/ZenModel/IpNetwork.py", line 253, in createNet
self.dmd.ZenLinkManager.networks.add_net(net)AttributeError: networks: <no traceback>
Somehow the ZenLinkManager (class Products.ZenModel.LinkManager) has not been instantiated correctly and it’snetworks attribute is missing. Fix this in zendmd:
[zenoss@mp4:/home/zenoss]: zendmd>>> zlm = dmd.ZenLinkManager>>> from Products.ZenModel.LinkManager import DrawNetworks>>> zlm.networks = DrawNetwork()>>> commit()
You can resume modeling now.
You see consecutive “Changes in configuration applied” messages
Once you model a device, any subsequent model should not see any changes. You should see a “No change inconfiguration detected” message on the 2nd model, assuming nothing changed. This is important because the if themodeler thinks there are changes to be recorded, it will go through and run ApplyDatamaps() on all the model data.This can be an expensive operation.
If you continue to see changes, try these steps:
• Ensure no changes show up after setting zCollectorLogChanges to true as above.
– Inspect your Events for the /Change event class,
– Do a verbose model, extract the datamaps and compare them using Unix diff command for example. Ifno difference is visible, go to the next sections.
• Short-circuit your modeler maps until you finally get the “no-changes” message after the 2nd model. Then addback in each map sequentially, modeling twice each try until you identify the guilty map.
• Ensure your property type match what you are setting in the modeler:
– If your property is the default type (string) and feed modeler int, it will show the changes message.
• Look for attribute setters of None or empty strings. Those should only be set if there are actual values.
5.9.9 Simulating Modeling
Gathering Modeler Results
There are a number of ways to do this:
5.9. Modeling for Zenpacks: Selected Topics 137
ZenPackers Documentation
• run the modeler with –save_raw_results or –save_processed_results
• Use pickler to grab the maps directly
Playing Back the Modeler
If you use –save_raw_results or –save_processed_results you can use the playback tool at https://github.com/zenoss/solutions-private/blob/develop/apply-zenmodeler-results
If you used a pickler, you have to insert that data in your modeler.
5.10 Collection: General Concepts
5.10.1 General Considerations
Collecting data is the fundamental goal of Zenoss. This section explores specific tasks related to data collection andparsing that data.
Fig. 13: Zenoss Collector Subsystem
Prerequisites
• Zenoss ZenPack Developement Guide
We assume that you are familiar with ZenPack developement and Python coding. We further assume that we workfrom the base of ZP_DIR. For NetBotz for example:
138 Chapter 5. General Zenpack Development
ZenPackers Documentation
ZP_DIR_TOP=$ZENHOME/ZenPacks/ZenPacks.training.NetBotzZP_DIR=$ZP_DIR_TOP/ZenPacks/training/NetBotz
As you should know, collectors and parsers typically live in the folder:
$ZP_DIR/datasources$ZP_DIR/parsers
Debugging Tips in General
• Run the collector manually like this:
zencommand run -v10 -d mp3.zenoss.loc |& tee collect.log
• If you don’t get any output, you can try these basic steps:
– Restart zenhub: it may have given up loading the modeler
– Rerun zencommand and also monitor /opt/zenoss/log/zenhub.log for good measure. You may want to runzenhub in the foreground.
• To test your parser command at a low level use:
cd /opt/zenoss/Products/ZenHub/servicespython CommandPerformanceConfig.py -d mp1.zenoss.loc
General Background Information
Collection process has the following steps:
• A collection process (a collector) is started, often with ZenCommand against a collector name (db2.zenoss.loc).
• The collector contacts Zenhub and load the commands to be run against devices for that collector (target)
• Zencommand runs the collection command on the remote target.
• If and when results are returned, a parser is created.
• Zencommand passes results to the processResults() method of the parser.
• The processResults() method is passed the command configuration fetched from ZenHub, and also an ob-ject into which parsed results will be placed.
• Zencommand takes the returned Python dictionary from the parser and updates the ZODB
𝐻𝑂𝑆𝑇𝑎 𝐶𝑜𝑙𝑙𝑒𝑐𝑡(𝑡𝑎𝑟𝑔𝑒𝑡𝑠)𝑎−→ (𝑅𝑒𝑠𝑢𝑙𝑡𝑠)
↓ ↑𝑎 ↓ZC/ZP 𝑎
𝑏 Zenhub ←−𝑏
(𝑃𝑎𝑟𝑠𝑒𝑟,𝑅𝑒𝑠𝑢𝑙𝑡)
↓𝑏ZODB
Ref: http://docs.huihoo.com/zenoss/dev-guide/2.4.2/ch12s05.html
5.10. Collection: General Concepts 139
ZenPackers Documentation
5.10.2 Custom Collectors and Custom Parsers
Collectors/Pollers
Collection can happen with a native plugin like [ssh, snmp, ping, https, etc.] or a custom plugin that you create. Theoutput can come back in several formats like [Nagios, Cacti, JSON] or a custom format that you specify. Customformats must be handled by you using a custom parser.
Sometimes we refer to collectors as pollers or collection plugins. Since the collector/plugin information is passed toZenhub, it must be an exectuable program or script.
An example of a custom poller that outputs JSON is shown below. The example is pilfered from ZenPacks.zenoss.DB2.Note the following:
• The poller is self-contained and self-calling. There are no magic functions that Zenoss calls automatically.
• The poller can import from global and local modules
• You normally setup your plugins in your __init__.py so that they have proper scope and permissions at installa-tion.
• In your Monitoring Templates setup, you must use the “COMMAND” type for this datasource.
• When you specify the datasource command, you will have to specifiy the full path (using TALES) to the poller.For example: ${here/ZenPackManager/packs/ZenPacks.zenoss.DB2/path}/poll_db2.py
• See the PostgreSQL, DB2, and DatabaseMonitor zenpacks for more examples.
1 #!/usr/bin/env python2 import sys3 from lib import txdb2jdbc4
5 class db2Poller(object):6 _connectionString = None # DB2 JDBC ConnectionStrings only7 _query = None # A Valid DB2 SQL query8 _data = None # This is JSON data from Java connector9
10 def __init__(self, conString, myQuery):11 self._connectionString = conString12 self._query = myQuery13
14 def getData(self):15 db2 = txdb2jdbc.Database(self._connectionString)16 self._data = db2.query_json(self._query)17 return self._data18
19 def printJSON(self):20 data = None21 try:22 data = self.getData()23
24 except Exception, ex:25 print "Exception", ex26 print data27
28 if __name__ == '__main__':29 usage = "Usage: {0} <connectionString> <query>"30 connectionString = None31
32 try:(continues on next page)
140 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
33 connectionString = sys.argv[1]34 query = sys.argv[2]35
36 except IndexError:37 print "poll_db2 : insufficient parameters"38 print >> sys.stderr, usage.format(sys.argv[0])39 sys.exit(1)40
41 except ValueError:42 print >> sys.stderr, usage.format(sys.argv[0])43 sys.exit(1)44
45 poller = db2Poller(connectionString, query)46 poller.printJSON()
Custom Parsers
The parser is invoked after a successful collection has occured. If you are not using one of the standard parsers like[Nagios, Cacti, JSON], then you must create your own custom parser.
Custom parsers usually are located in the $ZP_DIR/parsers folder. Whatever parser you create can only be used whenconfigured for the datasource (for your device) in the Monitoring Templates area. This information is typically storedin the $ZP_DIR/objects/objects.xml file.
Our example is from the DatabaseMonitor zenpack (OracleDB):
• Starting on line 10 we see the processResults() method definition.
• On line 13, we try to determine if the returned data is valid data.
• On line 38, we start to process the validated data
• At 55, we return results determined by status returned from probed targets
• Finally on line 69, we update our datapoints.
1 # --------------------------------------------------------------------------2 # File: $ZP_DIR/parsers/tablespaces.py -------------------------------------3 # --------------------------------------------------------------------------4 import json5
6 from Products.ZenRRD.CommandParser import CommandParser7 from ZenPacks.zenoss.DatabaseMonitor.lib import locallibs8
9 class tablespace(CommandParser):10 def processResults(self, cmd, result):11
12 data = None13 try:14 data = json.loads(cmd.result.output)15 # Auto-clear if possible16 result.events.append({17 'severity': 0,18 'summary': 'Command parser status',19 'eventKey': 'tablespace.parser.key',20 'eventClassKey': 'tablespace.parse.class',21 'component': cmd.component,
(continues on next page)
5.10. Collection: General Concepts 141
ZenPackers Documentation
(continued from previous page)
22 })23 except Exception, ex:24 result.events.append({25 'severity': cmd.severity,26 'summary': 'Command parser status',27 'eventKey': 'tablespace.parser.key',28 'eventClassKey': 'tablespace.parse.class',29 'command_output': cmd.result.output,30 'component': cmd.component,31 'exception': str(ex),32 })33
34 return result35
36 # Data is a list of dict: Iterate over them to find the right row37 tbsp = None38 for row in data:39 if tbsp is not None:40 break41
42 inst_name = row['INSTANCE_NAME']43 tbsp_name = row['TABLESPACE_NAME']44 component_id = '{0}_{1}'.format(inst_name, tbsp_name)45
46 # Select the correct row here. Break when found. Set Status47 if component_id == cmd.component:48 tbsp = row49
50 # If the TS reports an error, mark it as Critical.51 ts_message = 'Tablespace Status is: %s' % tbsp['ONLINE_STATUS']52
53 severity = locallibs.tbsp_status_map(tbsp['ONLINE_STATUS'])54
55 result.events.append({56 'severity': severity,57 'summary': ts_message,58 'eventKey': 'tablespace.status.Key',59 'eventClassKey': 'oracle.tablespace.ClassKey',60 'eventClass': "/Status",61 'component': component_id,62 })63
64 break # Break the "for row" , component found.65
66 #----------------------------------------------------------------------67 # Update/Filter on all datapoints. No need to check for non-numericals.68 #----------------------------------------------------------------------69 for point in cmd.points:70 if tbsp and point.id in tbsp:71 result.values.append((point, tbsp[point.id]))72
73 return result
142 Chapter 5. General Zenpack Development
ZenPackers Documentation
5.11 Datasources in Detail
5.11.1 How to Determine what Daemon Runs a DataSource
The ZenPack documentation should list of daemons and datasource types. If that is not quite enough, then try this:
• First look at the collector daemon’s class, it will specify a configurationService.
– For a custom daemon, look in the daemons folder.
• Then look at that service, to see what kinds of datasources it includes.
– For custom services, look in the services folder.
• If your datasource inherits from PythonCollector, it probably runs zenpython
5.11.2 How to Determine How Templates Set the DataSource
• Look for the sourcetypes variable in objects.xml or datasource.type in zenpack.yaml.
• Now compare that to existing datasources’ sourcetype variable.
5.11.3 General Questions and Answers
• Question: What’s the difference between a datasource and a datasource plugin?
A datasource is an instance of RRDDataSource. It’s the object that’s part of a monitoring template and stored inZODB. A datasource plugin is not a persistent ZODB object. It’s a class that is used to perform collection of aspecific subclass of RRDDataSource instance: PythonDataSource.
A PythonDataSource has a plugin_classname property that must be set to the fully qualified classname of aPythonDataSourcePlugin class such as “ZenPacks.zenoss.AWS.dsplugins.S3BucketPlugin”.(edited)
Also datasource plugins are a zenpython concept. So all PythonDataSources have PythonDataSourcePlugins,but other kinds of RRDDataSources do not.
So for instance, vsphere has a datasource, but not a plugin.
5.11.4 PythonCollector
When using PythonCollector there are two ways to configure collection as described in PythonCollector’s documen-tation.
1. Using the existing PythonDataSource. http://wiki.zenoss.org/ZenPack:PythonCollector#Using_the_Python_Data_Source_Type_Directly
2. Subclassing PythonDataSource. http://wiki.zenoss.org/ZenPack:PythonCollector#Extending_the_Python_Data_Source_Type
• How do you decide which method to use?
It’s pretty easy to figure out. Do you expect that a Zenoss admin would ever be adding one ofyour datasources to a monitoring template? If so, the answer is to subclass PythonDataSource tocreate your own type. If the answer is no, the answer is to not create a PythonDataSource subclass,but to just deliver your own monitoring templates that use the Python data source type with theplugin_classname populated.
5.11. Datasources in Detail 143
ZenPackers Documentation
Zenoss adds each PythonDataSource subclass as an option in the datasource type drop-down whencreating a new datasource in a monitoring template. So don’t pollute that drop-down with thingsthe user will never use, but don’t make the user create “Python” type datasources where they haveto provide the correct value for plugin_classname.
Using the Ceph ZenPack as an example, the suggestion would be that none of the PythonDataSour-cePlugins implemented therein are generic enough to deserve a PythonDataSource subclass.
Good examples of plugins that deserve a PythonDataSource subclass are “Windows Perfmon”,“Windows Event Log”, “Windows Process”.
5.11.5 Invalid config_keys() in Datasources: Clearing Redis
Sometimes if you work on datasource, you can accidentlly insert a bad entry into your config_key() method. This canget cached in redis and result your datasource using the cached value, even if you change the config_key() methodlater.
You may see a messages like this in your collector logs:
2017-08-17 13:12:08,839 DEBUG zen.collector.config: Fetching daemon configuration→˓propertiesTraceback (most recent call last):File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
msg = self.format(record)File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
return fmt.format(record)File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format
record.message = record.getMessage()File "/usr/lib64/python2.7/logging/__init__.py", line 328, in getMessage
msg = msg % self.argsTypeError: %d format: a number is required, not str
This probably means that the cached config_keys are not in the proper order. Ensure your config_key returns items inthe following order:
* context.device().id,
* datasource.getCycleTime(context),
* <whatever>
* <whatever>
* <whatever>
You must ensure that the first and second fields are (str, int).
If you get a broken config key n collectorredis or redis, you may have to flush redis’ cache :
• connect to the redis container(s)
• list the contents of table 2:
(zenoss) [zenoss@6f913fd44dbb ~]$ redis-cli -n 2 --scan --pattern '*'collector_cache_zenpython-0_devices@@nutanix64collector_cache_zenpython-0_threshold_classescollector_cache_zenpython-0_property_itemscollector_cache_zenpython-0_thresholds
• Open up redis table 2 in redis-cli:
144 Chapter 5. General Zenpack Development
ZenPackers Documentation
(zenoss) [zenoss@6f913fd44dbb ~]$ redis-cli -n 2127.0.0.1:6379[2]>127.0.0.1:6379[2]> del collector_cache_zenpython-0_devices@@nutanix64
(integer) 1127.0.0.1:6379[2]> del collector_cache_zenpython-0_property_item
(integer) 1127.0.0.1:6379[2]> exit(zenoss) [zenoss@6f913fd44dbb ~]$(zenoss) [zenoss@6f913fd44dbb ~]$ redis-cli -n 2 --scan --pattern '*'
• Now retry your datasource
5.12 Graphs
5.12.1 Special Examples
5.12.2 Putting Events in Things
• Jos: does anyone know if it’s possible to include a variable (like component id) in a Graph title?
• Bri: Yes
• Bri: I’ll give you an example
• Bri: https://github.com/zenoss/ZenPacks.zenoss.UCSCapacity/blob/develop/ZenPacks/zenoss/UCSCapacity/__init__.py#L212-L225
• Bri: Note that this one is monkeypatched, but you can just include it in your regular class
• Bri: All you need to do is update the graph object before you release the list
• Che: That seems dangerous. Wouldn’t Zope automatically commit that after processing the request?
• Che: According to Joseph you have to implement getDefaultGraphDefs for Zenoss 4 and getGraphObjects forZenoss 5. Also, you should do it like this. Note setting the volatile _v_title property.
https://github.com/zenoss/ZenPacks.zenoss.CiscoUCS/blob/develop/ZenPacks/zenoss/CiscoUCS/ServiceProfile.py#L193
5.13 ZenPython (PythonCollector) Topics
This article by Jane Curry is excellent:
http://www.skills-1st.co.uk/papers/jane/PythonZenPacks.pdf
Source code and docs:
http://wiki.zenoss.org/ZenPack:PythonCollector
5.13.1 Method Notes
• def config_key(cls, datasource, context):
Return list that is used to filter/split configurations at the collector. Check zenhub.log for errors.Note: restart zenhub after modification
5.12. Graphs 145
ZenPackers Documentation
• def params(cls, datasource, context):
Return params dictionary needed for plugin. Executed in zenhub. Note: restart zenhub after modi-fication
• def __init__(self, config=None):
Initialize the plugin with a configuration.
• def collect(self, config):
No default collect() behaviour. You must implement this method. This method *MUST* return aTwisted deferred. Results will be sent to the onResult then either onSuccess() or onError() callbacksbelow. Check zenpython.log for any log messages.
Note: collect()’s deferred return = result for onSuccess(), onError(), and onComplete()
• def onSuccess(self, result, config):
Called only on success of collect(). After onResult, before onComplete. Return a data structurewith zero or more events, values and maps. Note: values is a dictionary and events and maps arelists.
Note: The result input is the output of collect()’s deferred return.
It is easier to send events from onError()/onSuccess(), since you can just return them, along withmaps and data values. But you can do them from collect() if you use the service directly.
The thing to be careful about with collect() is that you need to return (or yield, if using inlinecall-backs) a deferred as soon as possible- don’t do anything cpu-intensive before you do this, or yourplugin will get blocked.
Even though this is really only critical on that first deferred you return, stylistically it’s best to keepthe IO in collect() and the processing in onSuccess()/onError(), just so that you’re not tempted todo things in collect() that you probably shouldn’t.
• def onError(self, result, config):
Called only on error. After onResult, before onComplete. You can omit this method if you want theerror result of the collect() to be used without further processing. Recommended to implement thismethod to capture errors.
• def onComplete(self, result, config):
Called last for success and error. You can omit this method if you want the result of either theonSuccess() or onError() method to be used without further processing.
• def cleanup(self, config):
Called when collector exits, or task is deleted or recreated. May be omitted.
5.13.2 Errors and Debugging
• Configuration for XYZ unavailable – is that the correct name? If you see an error like this in the zenpython logs:
146 Chapter 5. General Zenpack Development
ZenPackers Documentation
DEBUG zen.collector.config: Fetching configurationsERROR zen.collector.config: Configuration for XYZ unavailable -- is that the→˓correct name?DEBUG zen.collector.scheduler: Task configLoader finished, result: 'Configuration→˓loaded'... etc ...
Yes, you checked that your device name is correct, so what gives? This usually indicates a massive errorsomewhere in config_key() or params(). Check zenhub’s log for obvious clues and errors.
Don’t forget to restart zenhub after your fixes.
5.14 ZenPack Security
We take the security of our systems and customers seriously. This means that we provide the most secure access toour customers devices as possible. The following outlines some of the general concepts.
5.14.1 General Stuff
• Always use password type of zProperty for passwords
• Use TALES expressions with password zProps in composite credentials like ConnectionStrings
• Document the creation of special restricted accounts on targets for more secure access.
5.14.2 SSH and Sudo Access
Often we use SSH in combination with sudo to access system data for Unix type systems. Here are some guidelinesto mitigate customers having objections or other access problems.
1. Clearly document that we assume sudo NOPASSWD access for full functionality.
2. Document the exact minimal sudoers configuration we require.
3. Verify that we fail gracefully during modeling and monitoring in the following cases.
(a) sudo not installed.
(b) user not in sudoers.
(c) sudo requiring password.
Below is an example of an /etc/sudoers file that has minimal access for the commands:
• /sbin/service –status-all
• /sbin/initctl list
This is the /etc/sudoers file:
# This file MUST be edited with the 'visudo' command as root.## Please consider adding local content in /etc/sudoers.d/ instead of# directly modifying this file.Defaults mail_badpassDefaults secure_path="/usr/sbin:/usr/bin:/sbin:/bin"
(continues on next page)
5.14. ZenPack Security 147
ZenPackers Documentation
(continued from previous page)
Cmnd_Alias STATUS = /sbin/service --status-all, /sbin/initctl list
# User privilege specificationroot ALL=(ALL:ALL) ALL
# Allow members of group sudo to execute any command%sudo ALL=(ALL:ALL) ALL
# Allow zenoss user access to STATUS commandszenoss ALL = NOPASSWD: STATUS
On newer systems, it is often better to take the advice listed in /etc/sudoers and put local commands into/etc/sudoers.d/zenoss because there is less chance that this setup is lost on upgrades:
# file: /etc/sudoers.d/zenoss for zenoss access# This file won't get overwritted by upgrades
# Commands for getting system statusCmnd_Alias STATUS = /sbin/service --status-all, /sbin/initctl list# Allow zenoss user access to STATUS commandszenoss ALL = NOPASSWD: STATUS
5.15 Reserved Words in Zenoss Modeler
Here is a list of all reserved words except those that start with double-underscore:
absolute_urlabsolute_url_pathaccess_debug_infoac_inherited_permissionsacquiredRolesAreUsedByaddDeviceGroupaddLocalTemplateaddLocationaddManufactureraddRelation_addRoleaddSystemaddToZenPackall_meta_typesapplyDataMap_applyProdContextapplyProductContextaqBaseHasAttravailabilitybindTemplatesbobobase_modification_timebreadCrumbsbuildMenusbuildRelationscacheComponentscacheRRDValuecallZenScreen_canCopy
(continues on next page)
148 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
cb_dataItemscb_dataValidcb_isCopyablecb_isMoveablecb_userHasCopyOrMovePermissionchangeDeviceClasschangeOwnershipchangePythonClass_checkIdcheckRelationscheckRemotePermcheckValidIdcollectDevicecompileCOPY_create_componentSearch_createComponentSearchPathIndexcreateHierarchyObjcreatorcustPropertyIdscustPropertyMapdav__initdav__simpleifhandlerdav__validateDELETEdeleteDevice_deleteOwnershipAfterAdddeleteZenProperty_delOb_delObject_delProperty_delPropValue_delRolesdevicedoCommandForTargeteditableDeviceListeligiblePacksexportXmlexportXmlHookexportXmlPropertiesexportXmlRelationshipsexportZPropertiesfetchRRDValuefetchRRDValuesfilterAutomaticCreationfiltered_manage_optionsfiltered_meta_typesfindChildfindObject_findParentWithPropertyfollowNextHopIpsfullRRDPathgetgetAdministrativeRolesgetAllPathsgetAqChainForUserCommandsgetAqProperty
(continues on next page)
5.15. Reserved Words in Zenoss Modeler 149
ZenPackers Documentation
(continued from previous page)
getAttributegetAttributeNodegetAttributesgetAvailableTemplatesgetBreadCrumbNamegetBreadCrumbUrlPathgetChildNodesgetClassObjectgetCollectorNamegetCollectorsgetComments_getCopygetCreatedTimeStringget_csrf_token_getCurrentUserNamegetDataForJSONgetDataSourceOptionsgetDefaultGraphDefsgetDescriptiongetDevicegetDeviceClassNamegetDeviceClassPathgetDeviceComponentsgetDeviceComponentsNoIndexGengetDeviceGroupNamesgetDeviceIpgetDeviceIpAddressgetDeviceLinkgetDeviceMacaddressgetDeviceNamegetDeviceUrlgetDmdgetDmdKeygetDmdObjgetDmdRootgetElementsByTagNamegetEventManagergetEventSeveritiesgetEventSeveritiesCountgetEventSummarygetExpandedLinksgetFirstChildgetGraphDefgetGraphDefUrlgetHierarchyObjgetHTMLEventSummarygetHWManufacturerNamegetHWProductClassgetHWProductKeygetHWProductNamegetHWSerialNumbergetHWTaggetIconPath_get_idgetIdgetIdLink_getImportPaths
(continues on next page)
150 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
getInstDescriptiongetIpAddressgetLastChangegetLastChangeStringgetLastChildgetLastPollSnmpUpTimegetLinksget_local_rolesget_local_roles_for_useridgetLocationLinkgetLocationNamegetMacAddressCachegetMacAddressesgetMaintenanceWindowsgetManageInterfacegetManageIpgetManageIpObjgetMenusgetModificationTimeStringgetMonitoredComponentsgetNagiosTemplategetNamegetNetworkRootgetNextLockableParentgetNextSiblinggetNodeNamegetNodeTypegetNodeValuegetNonLoopbackIpAddressesgetNowString_getObgetObjByPathgetOSManufacturerNamegetOSProductKeygetOSProductName_getOtherExpandedLinksgetOverriddenObjectsgetOwnergetOwnerDocumentgetOwnerTuplegetParentDeviceNamegetParentDeviceTitlegetParentDeviceUrlgetParentNodegetPeerDeviceClassNamesgetPerformanceServergetPerformanceServerNamegetPhysicalPathgetPingStatusgetPingStatusNumbergetPingStatusStringgetPreMWProductionStategetPrettyLinkgetPreviousSiblinggetPrimaryDmdIdgetPrimaryIdgetPrimaryParent
(continues on next page)
5.15. Reserved Words in Zenoss Modeler 151
ZenPackers Documentation
(continued from previous page)
getPrimaryPathgetPrimaryUrlPathgetPrioritygetPriorityStringgetProdStategetProdStateManagergetProductionStategetProductionStateStringgetPropertygetPropertyTypegetRackSlotgetRelationshipManagerIdgetRelationshipNamesgetRelationships_getRelNamegetReportableComponents_get_request_var_or_attrgetRRDContextDatagetRRDDataPointgetRRDDataPoints_getRRDDataPointsGengetRRDFileNamegetRRDNamesgetRRDPathsgetRRDSumgetRRDTemplategetRRDTemplateByNamegetRRDTemplateNamegetRRDTemplatesgetRRDValuegetRRDValuesgetSerialNumbergetSiteManagergetSnmpAgentgetSnmpConnInfogetSnmpContactgetSnmpLastCollectiongetSnmpLastCollectionStringgetSnmpLocationgetSnmpStatusgetSnmpStatusNumbergetSnmpStatusStringgetSnmpSysNamegetSnmpV3EngineId_getSourceObjectsgetStatusgetStatusCssClassgetStatusImgSrcgetStatusStringgetSubComponentsNoIndexGengetSubObjectsgetSystemNamesgetSystemNamesStringgetTagNamegetTagNumbergetThresholdInstancesgetThresholds
(continues on next page)
152 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
getTreeItemsgetUnusedIdgetUrlForUserCommandsgetUserCommandgetUserCommandEnvironmentgetUserCommandIdsgetUserCommandsgetUserCommandTargetsgetUUIDget_valid_useridsgetWorstEventSeveritygetWrappedOwnergetXMLEdgesgetZgetZenRootNodegetZopeObjhasChildNodeshas_local_roleshasObjecthasProperty_has_user_defined_roleHEADhelpLinkhostnamehttp__etaghttp__parseMatchListhttp__processMatchHeadershttp__refreshEtag_importObjectFromFileidindex_objectipAddressAsInt_isBeingUsedAsAMethod_isDuplicateIpisLocalisLocalNameisLockedFromDeletionisLockedFromUpdatesisResultLockedFromDeletionisResultLockedFromUpdatesisTempDeviceisUnlockeditemskeyslistDAVObjectslist_importsLOCKlockFromDeletionlockFromUpdateslockStatuslockWarninglookupSchemamakeLocalRRDTemplatemanagemanage_accessmanage_ackEventsmanage_acquiredForm
(continues on next page)
5.15. Reserved Words in Zenoss Modeler 153
ZenPackers Documentation
(continued from previous page)
manage_acquiredPermissionsmanage_addAdministrativeRolemanage_addItemsToZenMenumanage_addLocalRolesmanage_addMaintenanceWindowmanage_addPropertymanage_addRelationmanage_addUserCommandmanage_addZenMenumanage_addZenMenuItemmanage_afterAddmanage_afterClonemanage_afterHistoryCopymanage_beforeDeletemanage_beforeHistoryCopymanage_change_historymanage_change_history_pagemanage_changeOwnershipTypemanage_changePermissionsmanage_changePropertiesmanage_changePropertyTypesmanage_clonemanage_CopyContainerAllItemsmanage_CopyContainerFirstItemmanage_copyObjectsmanage_copyrightmanage_cutObjectsmanage_DAVgetmanage_defined_rolesmanage_deleteAdministrativeRolemanage_deleteComponentmanage_deleteEventsmanage_deleteMaintenanceWindowmanage_deleteObjectsmanage_deleteUserCommandmanage_deleteZenMenumanage_deleteZenMenuItemmanage_delLocalRolesmanage_delObjectsmanage_delPropertiesmanage_doUserCommandmanage_editAdministrativeRolesmanage_editDevice_manage_editedDialogmanage_editedDialogmanage_editLocalRolesmanage_editPropertiesmanage_editRolesmanage_editUserCommandmanage_exportObjectmanage_fixupOwnershipAfterAddmanage_form_titlemanage_FTPlistmanage_FTPstatmanage_getPermissionMappingmanage_getUserRolesAndPermissionsmanage_hasId
(continues on next page)
154 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
manage_historicalComparisonmanage_historyCompare_manage_historyComparePagemanage_historyCopymanage_importExportFormmanage_importObjectmanage_index_mainmanageIpVersionmanage_linkObjectsmanage_listAdministrativeRolesmanage_listLocalRolesmanage_mainmanage_menumanage_ownermanage_page_footermanage_page_headermanage_page_style.cssmanage_pasteObjectsmanage_permissionmanage_permissionFormmanage_propertiesFormmanage_propertyTypeFormmanage_removeRelationmanage_renameFormmanage_renameObjectmanage_renameObjectsmanage_reportUserPermissionsmanage_rolemanage_roleFormmanage_saveMenuItemOrderingmanage_setLocalRolesmanage_setPermissionMappingmanage_snmpCommunitymanage_tabsmanage_takeOwnershipmanage_top_framemanage_undeleteEventsmanage_UndoFormmanage_undo_transactionsmanage_unlinkObjectsmanage_workspacemanage_zmi_logoutmanage_zmi_prefsMKCOLmonitorDevicemonitoredMOVEmoveMeBetweenRelsmoveObjectname_normal_manage_access_notifyOfCopyToobjectIdsobjectIds_dobjectItemsobjectItems_dobjectMap
(continues on next page)
5.15. Reserved Words in Zenoss Modeler 155
ZenPackers Documentation
(continued from previous page)
objectMap_dobjectValuesobjectValues_d_onlystarsOPTIONS_organizerInfoosProcessClassMatchDataowner_info_p_activatepastSnmpMaxFailurespath_p_deactivate_p_delattrpermission_settingspermissionsOfRole_p_getattrpingCommand_p_invalidatepossible_permissions_postCopyprepIdprimaryAqprimarySortKeypropdictpropertyDescriptionpropertyIdspropertyItemspropertyLabelpropertyMap_propertyMappropertyValuesPROPFINDPROPPATCH_p_setattrpushConfigPUT_redirectToEventConsoleredirectToUserCommandsreindex_allremoveLocalRRDTemplateremoveRelationremoveZDeviceTemplatesrenamerenameDevicerenameDeviceInPerformanceresetProductionStaterestoreCurrentProdStatesrestrictedTraverserolesOfPermissionrrdPath_sanitizeIPaddresssaveCurrentProdStatessaveCustPropertiessaveZenProperties_selectedTabNamesendEventWhenBlockedsendEventWhenResultBlocked
(continues on next page)
156 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
setAdminLocalRolessetAqPropertysetCollectorsetCommentssetDescriptionsetEventSeveritiessetGroupssetHWProductsetHWProductKeysetHWSerialNumbersetHWTag_setIdsetIpAddresssetLastChangesetLastPollSnmpUpTimesetLocationsetManageIpsetName_setOb_setObjectsetOSProductsetOSProductKeysetPerformanceMonitorsetPreMWProductionStatesetPrioritysetProdStatesetProductInfosetProductionState_setProductionState_setProperty_setPropValuesetRackSlot_setRelations_setRolessetSendEventWhenBlockedFlagsetSerialNumbersetSiteManagersetSnmpAgentsetSnmpContactsetSnmpLastCollectionsetSnmpLocationsetSnmpSysNamesetSnmpV3EngineIdsetSystemssetTagNumbersetTerminalServersetTitlesetWorstEventSeveritysetZenPropertysnmpAgeChecksnmpIgnoresnmpMonitorDevicesnmpwalkPrefix_subobject_permissionssuperValuessysUpTimetabs_path_default
(continues on next page)
5.15. Reserved Words in Zenoss Modeler 157
ZenPackers Documentation
(continued from previous page)
tabs_path_infothistitle_and_idtitle_or_idtitleOrIdtodayDatetpURLtpValuesTRACEtraceRoutetracerouteCommandundoable_transactionsunindex_ipsunindex_objectunlockUNLOCKunrestrictedTraverseunsetSendEventWhenBlockedFlagupdateDevice_updatePropertyuptimeStrupToOrganizerBreadCrumbsurlLinkuserCanTakeOwnershipuserdefined_rolesusers_with_local_roleuuidvalidate_rolesvalidClipDatavalid_property_idvalid_rolesvalidRolesvalues_verifyObjectLink_verifyObjectPasteviewviewNamevirtual_url_pathvisibleCustPropertyMapwl_clearLockswl_delLockwl_getLockwl_hasLockwl_isLockedwl_lockItemswl_lockmappingwl_lockTokenswl_lockValueswl_setLock_wrapperCheckwriteyesterdayDatezenpathjoinzenpathsplitzenPropertyIdszenPropertyItemszenPropertyMap
(continues on next page)
158 Chapter 5. General Zenpack Development
ZenPackers Documentation
(continued from previous page)
zenPropertyOptionszenPropertyPathzenPropertyStringzenPropIsPasswordzenScreenUrlzentinelTabszmanage_addPropertyzmanage_delObjectszmanage_delPropertieszmanage_editPropertieszmanage_exportObjectzmanage_importObjectzmanage_importObjectszope_quick_start
5.16 Solr Catalogs (ModelIndex) Review
5.16.1 What is Solr?
http://lucene.apache.org/solr/
Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene™.
Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced query-ing, automated failover and recovery, centralized configuration and more. Solr powers the search and navigationfeatures of many of the world’s largest internet sites.
5.16.2 What do we call it?
• Solr:
– Zenoss.resmgr/Infrastructure/Solr service in Control Center
– Show Solr UI
• modelindex:
– Python library: https://github.com/zenoss/modelindex
• model_catalog:
– The Solr “collection” in which all of this is stored
– Currently the only Solr collection we use
• ModelCatalog:
– Products.Zuul.catalog.model_catalog
– Provides an API to index, unindex, and search Zenoss objects using modelindex
– Adds ZODB transactional layer via two-phase commit
• ModelCatalogTool:
– Products.Zuul.catalog.model_catalog_tool
– Provides an interface similar to ICatalogTool (IModelCatalogTool)
5.16. Solr Catalogs (ModelIndex) Review 159
ZenPackers Documentation
5.16.3 What catalogs are moved to Solr?
• global_catalog (ICatalogTool still works)
• deviceSearch (via LegacyCatalogAdapter)
• layer2_catalog (via LegacyCatalogAdapter)
• layer3_catalog (via LegacyCatalogAdapter)
5.16.4 What is LegacyCatalogAdapter?
• A ZODB “SimpleItem” that behaves like ZCatalog, but uses ModelCatalog.
5.16.5 What is ZenPackLib going to do?
• Move all “index_scope: global” catalogs into Solr using LegacyCatalogAdapter.
• Migrate existing global catalogs to Solr? Maybe, but when?
• Backwards compatibility can be maintained.
• Going into ZenPackLib 2.1.0.
5.16.6 Other
• Some things that indexed via obj.index_object() now required notify(IndexingEvent(object))
5.16.7 Performance
• Queries in generally have a much more stable response time.
• “Hot” queries are intrinsically slower due to networking overhead to Solr.
• There are effectively no “cold” queries.
5.16.8 How is this different than CatalogSerice?
• Same in respect to performance trade-offs: Reliable response times, but often slower.
• CatalogService uses Lucene. ModelCatalog uses Solr.
• Think of Solr as a clustered (distributed) Lucene.
• Removes one of the remaining single points of failure.
• Enables moving to a future of active/active HA.
• ModelCatalog is in the Core platform. CatalogService is only in RM/UCS-PM.
5.16.9 Show LegacyCatalogAdapter usage in zendmd.
160 Chapter 5. General Zenpack Development
CHAPTER 6
Special Zenpack Development Topics
6.1 Analytics Bundle Topics
6.1.1 Introduction and General Notes
The following documents outline how Analytics support is built into our Zenpacks. If you see any errors, pleasecontribute to these docs as needed.
Warning: The Analytics Server must have a valid DNS resolution of the Zenoss system that it collects from.This means that either 1) the DNS servers know about your Zenoss system or 2) the /etc/hosts on Analytics has anentry for your Zenoss system. Without this host resolution, collection will not take place.
ZenETL
Issues related to troubleshooting ETL and expeditiing model load and aggregation should be directed to:
• https://github.com/zenoss/ZenPacks.zenoss.psSelfMonitoring
Specifically the “Additional Analytics related scripts provided by the zenpack” section
To install ZenETL, you must find the correct version for your version of Zenoss:
• Always use the latest version of the Analytics Server
• ZenETL version must match your version of Analytics Server!
• You will see the server RPM and Zenoss ZenPack egg in the folder:
– http://artifacts.zenoss.loc/releases/X.Y.Z/ga/analytics/
* Ex: http://artifacts.zenoss.loc/releases/5.0.0/analytics/ca1/
– grab the egg and install that on your Zenoss system ZenPacks.zenoss.ZenETL-X.Y.Z.a.b.cccc-py2.7.egg
– Replace the numerical values: X.Y.Z and a.b.cccc
161
ZenPackers Documentation
Analytics Server
We briefly mention that TCP network connectivity must be allow TO the Analytics server.
The following table lists Analytics pieces that typically must communicate, and the default network ports that shouldbe open for communication:
From To Default Port NumbersAnalytics Analytics database server 3306Analytics Resource Manager 8080Resource Manager Analytics 7070Analytics Web Users zenperfetl Analytics 7070(on remote collectors) Analytics 7070
6.1.2 Analytics Aliases
Analytics relies on aliases for datapoints. Alias must conform to the following rules:
• Aliases must be a single word
• Aliases must be less than 30 characters
• Aliases should be as descriptive as possible
• Aliases should end with double-underscore and unit (see below)
These aliases can be added easily in ZPL:
sysUpTime:aliases:sys_uptime__secs: "100,/"
cpuUtilization:aliases:cpu__pct: null
datapoints:numOfCpus:aliases:
cpu_socket__count: ""
numOfCores:aliases:
cpu_core__count: ""
Non-ZPL aliases are added by modifying the objects/objects.xml files.
Note: Aliases can have an optional RPN operation to normalize data. You can see this in the first example forsysUpTime.
Alias Units
Make sure to use the following unit postfix conventions for aliases:
162 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
Unit Descriptionpct percent (0-100)bytes bytes (base 1024)secs secondspersec (something) per secondbytessec bytes per second(none) where the unit is inherent (eg: load)count discrete object count (eg: packets)hz hertzseverity Zenoss severity number (0-5)watts poweramps electrical currentvolts electrical voltagecelsius Temperature in celsiusfahrenheit Temperature in fahrenheitpktssec packets per second
Instance Modifiers
Instance modifiers are used to modify the context or meaning of an alias name. See examples below as well.
Instance Modifier Name Descriptionadmin Administrativeavg Average valuecurrent Current valuemin Minimum valuemax Maximum valueoper Operational statein Input trafficout Output traffic
Reserved Aliases
It is important to make aliases as uniform across as possible, so that report builders can compare similar quantitiesacross multiple ZenPacks.
Here is an incomplete set of reserved aliases for common items. Use these aliases when possible:
Alias Name Descriptioncpu__pct CPU percentage usecpu_idle__pct CPU idle percentcpu_nice__pct CPU nice usecpu_system__pct CPU system usecpu_user__pct CPU user usecpu_wait__pct CPU wait time percentdisk_read__bytessec Disk read ratedisk_write__bytessec Disk write ratefan_oper_severity Fan operational statefanmodule_oper_severity Fan module operational state
Continued on next page
6.1. Analytics Bundle Topics 163
ZenPackers Documentation
Table 1 – continued from previous pageAlias Name Descriptionfs__pct filesystem used percentfs_used__pct filesystem used percent (repeat)mem__pct Memory percentage usemem_used__pct Memory percentage use (repeat)mem_buffers__pct Memory percentage usemem_cached__pct Memory cache usemem_free__pct Memory free percentmem_swap_free__pct Memory swap freemem_swap_used__pct Memory swap used percentif_in__pct Interface input traffic percentif_out__pct Interface output traffic percentin__pct input RX % for interfacesout__pct output TX % for interfacesinputOctets__bytes interface RX traffic in bytesoutputOcteds__bytes interface TX traffic in bytespower_oper_severity Power operational statebytes_free disk_free__bytesbytes_used disk_used__bytescache_hit_ratio cache_hit_ratio__pctphysicalreads physicalreads__bytesphysicalwrites physicalwrites__bytesreads reads__counttotal_size total_size__byteswrites writes__countwritetime writetime__sec
6.1.3 Setting up Zenoss Analytics
Setting up Zenoss Analytics on your Zenoss system has a few steps. Note that we don’t discuss how to setup the actualAnalytics Server here.
How to point your Zenoss 5.X system to an existing analytics 5.0.X server
Setup the ZenETL ZenPack and Services
• wget ZenPacks.zenoss.ZenETL-5.0.2-py2.7.egg from your local repo
• serviced service run zope zenpack-manager install ZenPacks.zenoss.ZenETL-5.0.2-py2.7.egg
• serviced service stop Zenoss.resmgr
• Wait until all services is fully stopped
• serviced service start Zenoss.resmgr
• Ensure that all AnalyticsETL services are started
Setup Analytics Server Information in RM
• Log into a non-admin account.
164 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
• Go to Resource Manager interface navigation bar, select Reports > Configuration.
• For Internal and External url, fill in url for analytics and resource manager
Note: The Zenoss entry is usually auto-populated. If not, use the address of the RM endpoint, typicallyhttps://zenoss5.yoursite.com
• Example:
Analytics: http://analytics.example.com:7070Zenoss: https://zenoss5.zenoss.example.com
• Fill in the User and Pass for the query service.
Note: For Query service, input zenoss username/password with Manager access or higher (don’t use the adminaccount)
How to point your Zenoss 4.2.X system to an existing analytics 5.0.X server
• Become the Zenoss user
• wget ZenPacks.zenoss.ZenETL-5.0.2-py2.7.egg from your local repo
• zenpack –install ZenPacks.zenoss.ZenETL-5.0.0.0.0-py2.7.egg
• zenoss stop; zenoss start
• Go to Resource Manager interface navigation bar, select Reports > Configuration.
• For Internal and External url, fill in url for analytics and resource manager
• Example:
Analytics: http://analytics.example.com:7070Zenoss: http://zenoss4.example.com:8080
Note: The Zenoss entry is usually auto-populated
References: – https://docs.zenoss.com https://www.zenoss.com/sites/default/files/documentation/Zenoss_Analytics_Release_Notes_r5.0.2_d1003.16.088.pdf
6.1.4 Loading the Bundle into Analytics
Once you have created the analytics bundle, you can load and test it as follows:
• Log into Analytics UI as superuser
• Go to Manage => Server Setting
• Select Import
• Deselect all options
• Browse: Select your zip file
• Make sure you deselect all options
6.1. Analytics Bundle Topics 165
ZenPackers Documentation
• Hit OK
6.1.5 Steps to create bundle with create-analytics-bundle scirpt:
http://zenpackers.readthedocs.io/en/latest/analytics/create-analytics-bundle.html
1. Setup your aliases in the ZP
• ensure all aliases are unique per device and component
2. Setup your labels in each component:
_properties = BaseComponent._properties + ({'id': 'path', 'label': 'Full Path', 'type': 'string', 'mode': ''},{'id': 'replication', 'label': 'Replication', 'type': 'string', 'mode': ''},{'id': 'did_status', 'label': 'DID status', 'type': 'string', 'mode': ''},)
• See: https://github.com/zenoss/ZenPacks.zenoss.SolarisMonitor/blob/develop/ZenPacks/zenoss/SolarisMonitor/ClusterDID.py#L25
• In a ZenPackLib ZP, you simply set valid labels on your properties.
1. Model a device that has all possible components available.
Note: Note you must have a device fully modelled and with all the aliases in place to get the measures to come out!
1. You can also simplify creation of the bundle in the Makefile: https://github.com/zenoss/ZenPacks.zenoss.CiscoAPIC/blob/develop/GNUmakefile#L29 https://github.com/zenoss/ZenPacks.zenoss.Microsoft.HyperV/blob/develop/GNUmakefile https://github.com/zenoss/ZenPacks.zenoss.Microsoft.HyperV/blob/develop/get_aliases_dsv.xsl
2. In order to control what properties and aliases get created in the bundle, see the reportables information inReportables: Modify what is Reported .
3. Run the create-analytics-bundle
• Note: the resulting bundle only needs a fully modeled device to work.
4. Install ZenETL, and allow it to run for 24 to 48 hours
• Ensure that all your tables (including raw_v2_*, hourly_*, daily_* tables) are in the reporing database on ana-lytics server.
1. Check that Analytics server has your tables created
2. After editing, .dsv file could be added to dmd using chkaliases script, and
from there exported to objects.xml, so you don’t have to edit it from UI or by hand in XML.
1. Check results
6.1.6 Debug a Domain Created with a Tool or Script
This test will show if a schema file is correctly formatted and can be used later for Ad-Hoc view creation.
166 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
Assumptions
• You have a domain bundle (.zip) or a schema.xml.data from a domain
• Your Zenoss system is configured to talk to Analytics
• You have data tables in the Analytics server for your domain
Debugging
• First extract the domain file and copy schema.xml.data to /tmp/schema.xml Find it in your analyticsbundle files (approximate):
./resources/public/XYZ_ZenPack/XYZ_Domain_files/schema.xml.data
• Log into the analytics UI as superuser
– Click on Domain -> Create
– Fill in:
* Name (give it a dummy name)
* Resource ID (Should auto populate with Name)
* Data Source:
/organizations'-> zenoss
'-> Data_Sources'-> zenoss_reporting
– Click Upload
– Browse: select your schema.xml file
• You should see a window with any errors
• Look for missing:
– tables
– fields
– other errors
6.1.7 Creating Analytics Bundles Manually
Create your Aliases
First create the aliases for each datapoint in your zenpack. They should conform to the naming convention in: http://goo.gl/WKOUKI * Alias length should be <= 32
For each subcomponent, join it to its parent component This will possibly do the trick:
• dim_oracle_instance.oracle_instance_key -> dim_oracle_table_space.instance_key
6.1. Analytics Bundle Topics 167
ZenPackers Documentation
Configure Analytics in your Zenpack
This step requires you to setup the analytics data in the zenpack so that the metric data can be collected.
• Install the ZenPacks.zenoss.ZenETL zenpack if not already
• Restart services if needed
• Go to Reports and add your Analytics server in the Configuration:
Internal URLS:Analytics: http://your-analytics.zenoss.lab:7070Zenoss: Should be auto-populated with your Zenoss URL
External URLS:Analytics: http://your-analytics.zenoss.lab:7070Zenoss: Should be auto-populated with your Zenoss URL
Create the Joins
Create the required joins between your dataources, metrics, and dimention tables. See Creating Joins in UI for details.
Create the Reports
• From Domain-Designer
• Click on create -> Ad-Hoc Report
• Change pulldown “Crosstab” to “Table”. You should see Groups now.
– Add major headings to groups like device_name or component_name
– Add metrics to rows
Save the Bundle
Save the data bundle
6.1.8 Creating Joins in UI
Before we start, lets talk about the overall strategy:
• Always use left outer joins
• Do not join dim device directly to component level metrics
• Joins are made in a hierarchical fashion to your metrics. This means that if your metric is device-level, you joinit to the device. With components-metrics, you join your component to the device, then join the metric to thatcomponent.
Device-Component Joins
Join dim_device to dim_component dimension tables on device key:
device.device_key -> component.device_key
168 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
Component-SubComponent Joins
You’ll want to join your related components in some way. For each subcomponent, join it to its parent componentThis will possibly do the trick:
dim_oracle_instance.oracle_instance_key -> dim_oracle_table_space.instance_key
Metric-Date Joins
For each metric in your domain, join its date_key to dim_date.date_key:
metric.data_key -> dim_data.date_key
Device Metric Joins
If you have device-level metrics, join them to device_key:
device metric.device_key -> dim_device.device_key
Metric-Component Joins
For each of your metrics you’ll want to join them to the right device. You’ll need the dim_device table:
metric.device_key -> dim_device.device_keymetric.data_key -> dim_data.data_key
Metric-Component Joins
For each metric in your domain, join it’s component key to your component’s key:
instance_metric.component_key -> dim_oracle_instance.oracle_instance_keytablespace_metric.component_key -> dim_oracle_tables_space.oracle_table_space_key
You may have to check exactly what keys match up: Example:
SELECT component_key FROM reporting.daily_cache_hit_ratio__pct A inner joinreporting.dim_oracle_instance B where A.component_key = B.oracle_instance_key;
6.1.9 Kicking off Quick Aggregation Jobs
Assumptions:
• There are raw_v2_* tables containing yesterday’s data.
• The are no records in meta_batch where batch_status is not COMPLETED or FAILED with a timestamp lessthan today. The aggregation won’t run if it finds incomplete batches.
6.1. Analytics Bundle Topics 169
ZenPackers Documentation
Database Adjustment to Start Aggregation
This involves 3 database-modifying steps:
1. Make sure that Analytics is running.
2. reset yesterday’s aggregation status (‘reporting’ database):
update meta_agg_dailyset status = 'UNRUN'where date_key = to_days(now()) - 1;
• If you don’t have a record, you can insert one:
insert into meta_agg_daily (date_key)values (to_days(now()) - 1);
3. Kick off the aggregation job (‘reporting’ database):
update QRTZ_TRIGGERSset next_fire_time = now()where trigger_name = "AggregateDataTrigger";
Using the Tools from ZenPacks.zenoss.psSelfMonitoring
• Clone the repo from https://github.com/zenoss/ZenPacks.zenoss.psSelfMonitoring
• On the Analytics server, copy the files as zenoss user:
git clone [email protected]:zenoss/ZenPacks.zenoss.psSelfMonitoring.gitcd ZenPacks.zenoss.psSelfMonitoring/ZenPacks/zenoss/psSelfMonitoringcp scripts/* /opt/zenoss_analytics/bin
• Ensure you have /opt/zenoss_analytics/bin in your $PATH env variable. Once the files are in place you can usethem. Of note are:
• set_analytics_env.sh
This one sets up all the DB settings for other scripts. Its worth looking at. In fact you can use a similar script toquickly enter the MySql db:
#!/bin/sh# Name: enter_mysql.sh# Function: Enter the analytics database# Sets environment variables based on analytics configuration
if [[ -x $(which mysql) ]]; thenMYSQL_PATH=$(which mysql)
elseecho "No apparent MySQL client exists! exiting..."exit 1
fi
CONFIG_FILE='/etc/zenoss_analytics/zenoss_analytics.conf'ETLHOSTNAME=`grep "^etl.jdbc.hostname" $CONFIG_FILE | cut -d"=" -f2`ETLPORT=`grep "^etl.jdbc.port" $CONFIG_FILE | cut -d"=" -f2`DBUSER=`grep "^etl.jdbc.username" $CONFIG_FILE | cut -d"=" -f2`DBPASS=`grep "^etl.jdbc.password" $CONFIG_FILE | cut -d"=" -f2`
(continues on next page)
170 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
(continued from previous page)
ETLDB=`grep "^etl.jdbc.dbname" $CONFIG_FILE | cut -d"=" -f2`$MYSQL_PATH -u $DBUSER --password=$DBPASS -h $ETLHOSTNAME -P$ETLPORT $ETLDB -→˓A
• start_model_load_now.sh If you haven’t yet loaded your model onto the server, this will expedite. Sets triggeringto 10 minutes in the future.
• start_aggregation_now.sh If you are ready to aggregate and have setup batch jobs for it, this will speed up thatprocess by setting the trigger time to 10 minutes.
• remove_extractor.sh This one can remove some of your unwanted batches
The others we list without comment:
• show_aggregation_detail.sh
• show_aggregation_config.sh
• show_aggregation_highlevel.sh
• show_analytics_task_status.sh
• show_batch_status.sh
• show_processlist.sh
• show_queries.sh
• show_settings.sh
• monitor_aggregation_counts.sh
• monitor_analytics_counts.sh
• monitor_db_connection_counts.sh
• monitor_etl_batch_states.sh
• monitor_etl_performance.sh
• remove_metric.sh
• remove_zenoss_instance.sh
• retry_batch.sh
Dealing with Time: Forcing Aggregation
Make sure your current time in correct. Analytics is very sensitive to time in terms of aggregation triggering.
Things to remember:
• If you started your Analytics server from a snapshot, you should IMMEDIATELY reboot that snapshot to ensureall services are running normally.
• Check that NTP is running
• Check the current data and insure it is correct.
• Analytics will only create daily and hourly tables from yesterday’s data!
– One dirty trick you can do to force faster aggregation is to set the system time to tomorrow’s date, makingAnalytics think the data you currently have is from yesterday.
– Once aggregation is finished to your satisfaction, start/restart NTP.
6.1. Analytics Bundle Topics 171
ZenPackers Documentation
6.1.10 Reportables: Modify what is Reported
Reportables allow you to control what data Analytis will extract from your zenpack. To invoke reportables you needto create a reportable.py in your zenpack toplevel directory. You can an example of this in:
• ZenPacks.zenoss.OpenStackInfrastructure
Things to consider about Reportables:
1. Reportables are reporting on properties that come from metric data, which bogs down zenmodeletl: Sometimesthis is unavoidable, but sometimes the property isn’t really that important, or worth the extra load on zenmod-eletl, so it can be removed to save load.
2. Datasource configs are using information from metric data, which bogs down zenperfetl: This is much morelikely to be something that is a bug in the zenpack in question, as such values probably are not being propagatedto the collector properly anyway, since there would be no invalidations associated with them changing- it ismost, at best, a waste of time, and at worst, a bug that will bite us in subtle ways later.
Setup
Reportables are configured by creating a reportable.py which inherits from either:
• ZenPacks.zenoss.ZenETL.reportable.Reportable
• ZenPacks.zenoss.ZenETL.reportable.BaseReportable
In addition you will probably need to import a Factory from:
• ZenPacks.zenoss.ZenETL.reportable.BaseReportableFactory
Once you have defined a reportable class for your device or component, you will need to register it in configure.zcmlor your equivalent.
Note: Many ZenPacks have a basereportable.py that defines a normalized set of properties. This basereportable is thenimported into reportable.py. The purpose of this is to create a simpler reportable.py and provide a more appropriatedefault set of reportables.
Examples
Below is a simple self-contained example which modifies a PortChannel: Note the following:
1. entity_class_name must be defined for each property you modify and it must be less than 28 characters long.Exceeding this created dimension (dim_*) table names that cause problems in Analytics.
2. reportProperties() is the main class method that provides the change.
import logginglog = logging.getLogger("zen.CiscoMonitor.etl")
from zope.interface import implementsfrom zope.component import adaptsfrom ZenPacks.zenoss.CiscoMonitor.PortChannel import PortChannel
from Products.Zuul.interfaces import (
(continues on next page)
172 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
(continued from previous page)
IReportable)
from ZenPacks.zenoss.ZenETL.reportable import (MARKER_LENGTH,BaseReportableFactory,IpInterfaceReportable,Reportable
)
class CiscoMonitorBaseRelReportable(Reportable):implements(IReportable)
def __init__(self, fromObject, toObject):super(CiscoMonitorBaseRelReportable, self).__init__()self.fromObject = fromObjectself.toObject = toObject
@propertydef id(self):
return "%s__%s" % (self.fromObject.id, self.toObject.id)
@propertydef uid(self):
return "%s__%s" % (self.fromObject.uid, self.toObject.uid)
@propertydef sid(self):
return "%s__%s" % (IReportable(self.fromObject).sid, IReportable(self.toObject).→˓sid)
class PortChannelReportableFactory(BaseReportableFactory):adapts(PortChannel)
def exports(self):# The interface as Interfaceyield IpInterfaceReportable(self.context)
for child in self.context.member_interfaces():yield PortChannelToMemberInterfaceReportable(self.context, child)
class PortChannelToMemberInterfaceReportable(CiscoMonitorBaseRelReportable):@propertydef entity_class_name(self):
return 'cisco_portchannel_interfaces'
def reportProperties(self):return [
('portchannel_ip_interface_key', 'reference', IReportable(self.→˓fromObject).sid, MARKER_LENGTH),
('ip_interface_key', 'reference', IReportable(self.toObject).sid, MARKER_→˓LENGTH),
]
6.1. Analytics Bundle Topics 173
ZenPackers Documentation
This is the associated ZCML entry:
<configure zcml:condition="installed ZenPacks.zenoss.ZenETL">
<adapter provides="Products.Zuul.interfaces.IReportableFactory"for=".PortChannel.PortChannel"factory=".reportable.PortChannelReportableFactory"/>
</configure>
6.1.11 Analytics trobleshooting guide
Based on docs by William Gerber.
What problem do you have?
• I get no reports.
I get no reports
Are your batches working?
• Yes: Can you log into Zenoss Analytics at all?
• No: My batches are not working
My batches are not working
See at what state they are stuck in at the web UI:
• Unstarted: Zenetl or zenperfetl daemon fails
• Extracting: Zenetl or zenperfetl daemon fails
• Staging: Is there model data in staging?
• Failed: ????
Note: Bi-Directional DNS Resolution
Make sure your DNS on both the RM and Analytics side can fully resolve the each other. If required, add lines in/etc/hosts on the Analytics server and RM so that they can resolve whatever configuration URLs you have set.
Once you set /etc/hosts on Zenoss, make sure to install, configure, and restart dnsmasq so that containers can get theright DNS data.
Can you log into Zenoss Analytics at all?
Zenetl or zenperfetl daemon fails
Have you configured the 4 URLS for Analytics through the WEB UI?
If no, then go configure them.
174 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
Update each Hub and Collector so it gets the proper config properties
Check to see if the collector can hit the Analytics server. From the command prompt do the following: wget http://analytics_server_url/reporting
Did you get a text file without error?
• Yes: Is there enough space on the hard drives to write files to?
• No: Check analytics availability from collector
Is there enough space on the hard drives to write files to?
If no then get more space or edit the config file for daemon to point to a drive with more space.
Is the log file complaining that it can’t hit the Hub? - Yes: Check daemon connection with the HUB - No: Doesdeamon try to upload a file at all?
Check analytics availability from collector
Double check that the Analytics server is running by open a browser to the URL http://analytics_server_url/reporting.
If you got a text file than your analytics server is not reachable from collector and you should check firewalss proxiesand ports.
If you got an error, you should figure out why your Analytics server is not running.
Is there model data in staging?
• If debugging just tail ALL the logs:
cd /opt/zenoss_analytics/logstail -f *.log *.out--or--
tail -F -n100 *.out *.log |grep -iE '(failed|error)'
• No: Check triggers
– Also check the log /opt/zenoss_analytics/catalina.out
– Ensure Analytics can resolve (via DNS or /etc/hosts) the Zenoss server!
• Yes: Has it been in staging over 4 hours?
– No: We only check for Model batches ready to load into DB once every 4 hours.
– Yes: Check triggers
Check triggers
Log into the reporting database and run this query a few times over a couple of minutes:
select trigger_name, from_unixtime(next_fire_time/1000) as next_time,from_unixtime(prev_fire_time/1000) as last_time,trigger_state from reporting.QRTZ_TRIGGERS;
6.1. Analytics Bundle Topics 175
ZenPackers Documentation
Are some of the triggers state stuck in blocked? If yes, then something caused one or more of the triggers to failrepeatedly and the system finally gave up. Fix the root cause and then update the QRTZ_TRIGGERS table to a stateof WAITING and bounce the zenoss_analytics service.
Go to the Analytics application server and look through the logs:
/opt/zenoss_analytics/logs/zenoss_analytics.log: file for glaring errors.
**or**
/opt/zenoss_analytics/webapps/reports/WEB-INF/logs/jasperserver.log
After Importing Bundle, I Get this Error when I edit my bundle
• Cannot get metadata for /organizations/zenoss/Data_Sources/zenoss_reporting: com.mysql.jdbc.Driver Thismeans you imported your bundle without turning off the checkboxs on the import. The Zenoss_reportingdatasource is now corrputed.
To fix this, you must restore the zenoss_reporting datasource from a working install e.g. export it and reimportit into your broken install, or revert your Analytics server to a clean state.
This can be done in a number of ways:
1. Revert your Analytics instance to the default state: You’ll lose your dim tables though.
2. This way restores zenoss_reporting without destroying Reporting db.
– Stop Analytics
– drop zenoss_analytics db from mysql
– As zenoss, run update_db.py
– Start Analytics
Check daemon connection with the HUB
–tbd–
Does deamon try to upload a file at all?
–tbd–
6.1.12 Accessing Mysql in Analytics
Quick Access to the Mysql DB
Use this script to get quick access to the Mysql DB:
#!/bin/sh## Quicly enter Mysql# Sets environment variables based on config
if [[ -x $(which mysql) ]]; then(continues on next page)
176 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
(continued from previous page)
MYSQL_PATH=$(which mysql)else
echo "No apparent MySQL client exists! exiting..."exit 1
fi
CONFIG_FILE='/etc/zenoss_analytics/zenoss_analytics.conf'ETLHOSTNAME=`grep "^etl.jdbc.hostname" $CONFIG_FILE | cut -d"=" -f2`ETLPORT=`grep "^etl.jdbc.port" $CONFIG_FILE | cut -d"=" -f2`DBUSER=`grep "^etl.jdbc.username" $CONFIG_FILE | cut -d"=" -f2`DBPASS=`grep "^etl.jdbc.password" $CONFIG_FILE | cut -d"=" -f2`ETLDB=`grep "^etl.jdbc.dbname" $CONFIG_FILE | cut -d"=" -f2`
# Start mysql in this way to get to the right database:$MYSQL_PATH -u $DBUSER --password=$DBPASS \
-h $ETLHOSTNAME -P$ETLPORT $ETLDB --skip-column-names -A
Dealing with Job Triggers
• To see the Cronjobs for triggers from inside mysql:
select * from QRTZ_CRON_TRIGGERS;
• Look at the Triggers:
select * from QRTZ_TRIGGERS;select trigger_name, trigger_state from QRTZ_TRIGGERS;
• Remove all your batches from DB directly:
select extractor_key from meta_extractor where extractor_fqdn = 'mine.zenoss.loc';
# find the numbers it outputs : Say (20,22,31)delete from meta_batch where extractor_key in (20,22,31);
6.1.13 Tips and Tricks
Disable Session Timeout
By default timeout value is set to 20 minutes. To make sessions endless you can decrease session-timeout property in/opt/zenoss_analytics/webapps/reports/WEB-INF/web.xml file.
<session-config><!--Default to 20 minute session timeouts--><session-timeout>0</session-timeout>
</session-config>
List Registered RM Systems in Analytics
curl http://analytics_server.zenoss.loc:7070/etl/ | jq
6.1. Analytics Bundle Topics 177
ZenPackers Documentation
This will show you the UUID’s for various RM systems communicating with the Analytics server.
If you know the UUID, you can specifiy it on the commandline:
curl http://solutions-analytics.zenoss.loc:7070/etl/<uuid> | jqeg:curl http://solutions-analytics.zenoss.loc:7070/etl/8054d966-97ec-11e7-a03a-→˓0242ac110017 | jq
Remove All Batches
From the reporting database:
delete from meta_batch; delete from meta_batch_file;
Then use the script:
show_batch_status.sh
6.1.14 References
Docs
https://docs.google.com/document/d/1I6QeEjzKTyg1d-SAFY5iF_YHlE582_UfGutROvMjpkM/edit#
Videos
We have some local MP4 vids at:
https://zenpackers.zenoss.loc/videos/
6.2 Catalog Tool
For fast searching and discovery, Zenoss uses Zope’s Catalog Service. Most catalogs must be created by indexingzProperties, although see the note later about ComponentBase searches.
6.2.1 Debugging Catalog Indexing
This is done in exactly the same was as Debugging Impact.
6.2.2 Setting Index Properties
In order to put a ZenPack property into the catalog you must set its index_type to field. In ZenPackLib it would bedone as follows (eg: macAddress):
178 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
classes: !ZenPackSpecDEFAULTS:base: [zenpacklib.Component]
ControlCenter:base: [zenpacklib.Device]meta_type: ZenossControlCenter
Host:meta_type: ZenossControlCenterHostproperties:
ipaddr:label: IP Address
macAddress:label: Mac Addressindex_type: fieldindex_scope: device
Notice that index_scope is set to “device” by default, and is not needed. It can also be set to global, which will allowit to be searched globally.
When a catalog’s scope is set to global, the actual catalog object will be created on dmd.Devices. The name of thecatalog will be the fully-qualified module name of the class under which the property with index_type and index_scopeare set with a “Search suffix”. So let’s say you had a ZenPack named ZenPacks.example.Test that defined a class namedWidget. The catalog would be dmd.Devices.ZenPacks_example_Test_WidgetSearch.
Note: There is a default device-scoped catalog created by any ZenPack that uses zenpacklib to create a “Component”class. That’s the ComponentBase catalog. This catalog does not require any index_type property to be set.
It’s created because the standard componentSearch catalog doesn’t have an id index, and some of the relationshiphelper methods require an id index to more efficiently find components.
It’s normal to see a ComponentBaseSearch on any device associated with a zenpacklib ZenPack.
Note: Removing a ZenPack would remove any associated catalog items.
6.2.3 Examples of Catalog Use
Here is an example of how you use the Catalog:
############################################################################# Copyright (C) Zenoss, Inc. 2014, all rights reserved.############################################################################
# Zenoss Importsfrom Products.Zuul.interfaces import ICatalogTool
# ZenPack Importsfrom ..APIC import APIC
def cisco_apic_devs(self):"""Return associated Cisco APIC instances."""
catalog = ICatalogTool(self.getDmdRoot('Devices'))apic_devices = []
(continues on next page)
6.2. Catalog Tool 179
ZenPackers Documentation
(continued from previous page)
for apic_result in catalog.search(types=[APIC]):try:
apic = apic_result.getObject()apic_devices.append(apic)
except Exception:continue
return apic_devices
Here is a self contained python search:
#!/usr/bin/env zendmd## Print details of all WebTx datasources.
from Products.Zuul.interfaces import ICatalogToolfrom ZenPacks.zenoss.ZenWebTx.datasources.WebTxDataSource import WebTxDataSource
catalog = ICatalogTool(dmd.Devices)
for result in catalog.search(WebTxDataSource):datasource = result.getObject()template = datasource.rrdTemplate()label = "{} - {}".format(template.getUIPath(), datasource.id)print "--[ {} ]{}".format(label, "-" * (73 - len(label)))print "Initial URL: {}".format(datasource.initialURL)print "Initial User: {}".format(datasource.initialUser)print "Timeout: {}".format(datasource.webTxTimeout)print "Interval: {}".format(datasource.cycletime)
if datasource.commandTemplate:print "Twill Script:"printprint datasource.commandTemplate
else:print "Twill Script: n/a"
Here is class method:
from zope.event import notifyfrom Products.Zuul.interfaces import ICatalogTool
@classmethoddef reindex_implementation_components(cls, dmd):
device_class = dmd.Devices.getOrganizer('/Network/OpenvSwitch')results = ICatalogTool(device_class).search(
('ZenPacks.zenoss.OpenvSwitch.Port.Port','ZenPacks.zenoss.OpenvSwitch.Interface.Interface',)
)
for brain in results:obj = brain.getObject()obj.index_object()notify(IndexingEvent(obj))
180 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
Here is a (truncated) class method example from ZenPackLib:
def remove(self, app, leaveObjects=False):from Products.Zuul.interfaces import ICatalogToolif not leaveObjects:
dc = app.Devicesfor catalog in self.GLOBAL_CATALOGS:
catObj = getattr(dc, catalog, None)if catObj:
LOG.info('Removing Catalog %s' % catalog)dc._delObject(catalog)
if self.NEW_COMPONENT_TYPES:LOG.info('Removing %s components' % self.id)cat = ICatalogTool(app.zport.dmd)for brain in cat.search(types=self.NEW_COMPONENT_TYPES):
component = brain.getObject()component.getPrimaryParent()._delObject(component.id)
# Remove our Device relations additions.from Products.ZenUtils.Utils import importClassfor device_module_id in self.NEW_RELATIONS:
Device = importClass(device_module_id)Device._relations = tuple([x for x in Device._relations
if x[0] not in self.NEW_RELATIONS[device_→˓module_id]])
LOG.info('Removing %s relationships from existing devices.' % self.id)self._buildDeviceRelations()
6.2.4 References
• http://docs.zope.org/zope2/zope2book/SearchingZCatalog.html
• https://pypi.python.org/pypi/Products.AdvancedQuery
• http://community.zenoss.org/docs/DOC-2535 (Removes items from the catalog)
• See AdvancedQuery.html in the doc subfolder.
• http://wiki.zenoss.org/ZenDMD_Tip_-_Refresh_DeviceSearch_Catalog
6.3 Migration Guide for Slackers
6.3.1 Introduction
Zenpacks that get upgraded often need some sort of migration of configuration data from the older version to the new.This is achieved via migration scripts that are described in this doc.
Note:
• zSetZenProperty(): is safest way to set a zproperty. . .
• getTitleOrId(): Best way to get name of device.
• getOrganizerName(): is best way to get Device Class name.
6.3. Migration Guide for Slackers 181
ZenPackers Documentation
Warning: Never ever set a zProperty directly on a device or device class. Always use setZenProperty() whichtakes care of ZoDB bookkeeping items.
General Information on Template Migration
Any template migration might do the following things for XYZ ZenPack:
• Identify the old zenpack objects, bound to a device.
• Find all device classess and devices where “XYZ” template is bound.
• Look for zDeviceTemplates that are overridden, you don’t want to migrate templates that a user has customized,they probably want to keep their changes.
• Extract the old information from those templates
– Leave them bound. . . for continuity
– Enable our modeler plugin for device class or device (zCollectorPlugins gets invoked, gets added to list).
– Populate zProperties
– Create new Instance components from the old templates and populuate the new.
• Leave the old ZP objects alone: Provide documentation on how to un-bind the old templates.
• XYZ Template needs to be bound to device
General Information on Relationship Migration
In general, we try to avoid changing existing relationships on a device class because that requires users to completelyremove those devices, upgrade, and then re-model their devices. We do this becuase existing devices have theirrelations embedded in ZODB and can’t easily be changed without significant database surgery.
On the other hand, adding new relations is considered acceptable behavior. Once you add those relationships on adevice you must ensure that you execute:
device.buildRelations()
6.3.2 Examples
We assume this basic set of knowledge for this article:
• Zenoss ZenPack Development
• Python 2.7
• Familiarity with ZenPack development and Python coding.
• We work from the base of ZP_DIR. For NetBotz for example:
export ZP_DIR_TOP=$ZENHOME/ZenPacks/ZenPacks.training.NetBotzexport ZP_DIR=$ZP_DIR_TOP/ZenPacks/training/NetBotz
Relative to this folder all migration scripts will reside in:
182 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
$ZP_DIR/migrate/
In the first example. we have a simple migration of the a zProperty (zOraclePassword) to the new ZP version. In thesecont second-example we will see a much more significant migration() implementation.
6.3.3 Migrating Properties
In this example DatabaseMonitor (OracleDB) is a device that inherits its /Device base from the parent server, be itLinux, AIX, Solaris, or some “other” operating system. This means that it needs to be able to patch itself underneaththe device tree of that server target type and not have a stand-alone device root.
The basic code strategy is to create a class that has a “migrate” method. The migrate() method gets called automaticallyby the ZenPack Installer. The first example is a very old version that does nothing but migrate the password from oneversion to the next.
We only need to set a single zProperty value. In order to do so, we use: MigratePassword.py:
############################################################################### Copyright (C) Zenoss, Inc. 2009, all rights reserved.##############################################################################
import logginglog = logging.getLogger("zen.migrate")
import Globalsfrom Products.ZenModel.migrate.Migrate import Versionfrom Products.ZenModel.ZenPack import ZenPackMigrationfrom Products.ZenModel.migrate.MigrateUtils import migratePropertyType
class MigratePassword(ZenPackMigration):version = Version(2, 2, 0)
def migrate(self, dmd):log.info("Migrating zOraclePassword")migratePropertyType("zOraclePassword", dmd, "string")
MigratePassword()
Notice that there is a “version” line just after the class definition. This version must identify the new version numberof the ZP being migrated to.
The migration() method is very simple; in fact just one line that uses the migratePropertyType() method to migrate thezOraclePassword.
6.3.4 Migration of Relations
Here we added a ThinPool relationship for device in ControlCenter. Notice the cc_device.buildRelations():
import logginglog = logging.getLogger("zen.migrate")
from Acquisition import aq_basefrom Products.ZenModel.migrate.Migrate import Versionfrom Products.ZenModel.ZenPack import ZenPackMigrationfrom Products.Zuul.utils import safe_hasattr
(continues on next page)
6.3. Migration Guide for Slackers 183
ZenPackers Documentation
(continued from previous page)
class AddThinPoolsRel(ZenPackMigration):version = Version(1, 2, 2)
def migrate(self, pack):dmd = pack.dmd
if dmd.Devices._getOb("ControlCenter", None) is not None:for cc_device in dmd.Devices.ControlCenter.getSubDevicesGen():
if not safe_hasattr(aq_base(cc_device), 'thinPools'):log.info("Updating missing relationship for ControlCenter device
→˓{0}".format(cc_device.id))
cc_device.buildRelations()
6.3.5 Migration of Modeling Templates
This section is specific to a modeling templates.
The Basic idea behind this migration scenario is as follows:
• Identify the old zenpack objects, bound to a device.
– Find all device classes and devices where “Oracle” modeling template is bound.
– Look for all zDeviceTemplates that are overridden
• Extract the old information from those templates
– Leave them bound, for continuity sake.
– Enable our modeler plugin for device class or device (uses zCollectorPlugins)
• Populate the New ZP Data Structures
– Create new Instance components from the old templates
– Populate the new instances or components with data
– As always, test your migration script by installing the new ZP over the old.
4. Give Users Instructions on Removing Old Object Templates
• Since you may have left the old ZP objects in tact, provide documentation on how to un-bind the oldtemplates.
Migration Example for DatabaseMonitor
This migration updates the older 2.X version to the 3.X version of the ZP, and transitions from a dedicated Device toa pure component model (Instance). You don’t need to worry about handling component binding because that is takencare of by the actual modeler. Here is the ~migration/AddInstances.py code:
1 ############################################################################2 # Copyright (C) Zenoss, Inc. 2013, all rights reserved.3 # File: ~migration/AddInstances.py4 ############################################################################5
6 import logging(continues on next page)
184 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
(continued from previous page)
7 log = logging.getLogger("zen.migrate")8
9 from Products.ZenModel.DeviceClass import DeviceClass10 from Products.ZenModel.migrate.Migrate import Version11 from Products.ZenModel.ZenPack import ZenPackMigration12
13 # You must have Oracle's modeling template bound for migration to work14 TEMPLATE_NAME = 'Oracle'15 MODELER_PLUGIN_NAME = 'zenoss.ojdbc.Instances'16
17 def name_for_thing(thing):18 ''' Helper function to provide the name of the Device or DeviceClass '''19
20 if isinstance(thing, DeviceClass):21 return thing.getOrganizerName()22
23 return thing.titleOrId()24
25 class AddInstances(ZenPackMigration):26 '''27 Main class that contains the migrate() method. Note version setting.28 '''29 version = Version(3, 0, 0)30
31 def migrate(self, dmd):32 '''33 This is the main method. Its searches for overridden objects (templates)34 and then migrates the data to the new format or properties.35 In this case bound objects get assigned the new modeler pluging.36 '''37 overridden_on = dmd.Devices.getOverriddenObjects(38 'zDeviceTemplates', showDevices=True)39
40 for thing in overridden_on:41 if TEMPLATE_NAME in thing.zDeviceTemplates:42 self.enable_plugin(thing)43 self.populate_connection_strings(thing)44
45 def enable_plugin(self, thing):46 ''' Associate a collector plugin with the thing we have found.47 zCollectorPlugins is used by ModelerService.createDeviceProxy()48 to add associated (modeler) plugins to the list for self-discovery.49 ModelerService.remote_getDeviceConfig() actually calls the modelers.50 '''51 current_plugins = thing.zCollectorPlugins52 if MODELER_PLUGIN_NAME in current_plugins:53 return54
55 log.info(56 "Adding %s modeler plugin to %s",57 MODELER_PLUGIN_NAME, name_for_thing(thing))58
59 current_plugins.append(MODELER_PLUGIN_NAME)60 thing.setZenProperty('zCollectorPlugins', current_plugins)61
62 def populate_connection_strings(self, thing):63 ''' Just a helper method to collect data for this ZP '''
(continues on next page)
6.3. Migration Guide for Slackers 185
ZenPackers Documentation
(continued from previous page)
64 if thing.zOracleConnectionStrings:65 return66
67 connection_string = (68 'jdbc:oracle:thin:'69 '${here/zOracleUser}'70 '/${here/zOraclePassword}'71 '@${here/manageIp}'72 ':${here/zOraclePort}'73 ':${here/zOracleInstance}'74 )75
76 log.info(77 "Setting zOracleConnectionStrings for %s",78 name_for_thing(thing))79
80 thing.setZenProperty('zOracleConnectionStrings', [connection_string])81
82 AddInstances()
6.3.6 Pre-Install and Post-Install Migration Events
There are certain actions that need to happen before or after the installation process, due to different dependencyrequirements. We discuss both cases.
In the file __init__.py we have an install method that looks like this:
def install(self, app):
self.pre_install(app)super(ZenPack, self).install(app)self.post_install(app)
The super method is responsible for calling all the migration scripts.
Pre-Installation and Monitoring Templates
This section is specific to monitoring (collection) templates.
Monitoring templates get generated at install time, so it is important to have all the correct templates you need in theright place. Its also important to not-have any wrong templates.
Attention: This works because it’s ultimately the super() method that uses objects.xml to create the monitoringtemplates you’ve added to the ZenPack. So if you delete what exists before calling super, you’ll be sure to end upwith exactly what the objects.xml contains.
Note that the following:
• The migration scripts [ in ~/migration/* ] are invoked only after the initial installation has taken place.
• We must remove/replace older monitoring templates in __init__.py before the migration scripts are invoked.
• We use ZenPacks.zenoss.CiscoMonitor as the prime example
186 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
• We must implement the pre_install() method
For our purposes the pre_install() method performs the following inside __init__.py:
1 REPLACED_MONITORING_TEMPLATES = (2 'Network/Cisco/ACE/rrdTemplates/Device',3 'Network/Cisco/ASA/rrdTemplates/Device',4 'Network/Cisco/Nexus/rrdTemplates/Device',5 'Network/Cisco/rrdTemplates/Device',6 'Network/Cisco/rrdTemplates/ethernetCsmacd',7 )8
9 class ZenPack(ZenPackBase):10 """CiscoMonitor loader."""11
12 ...some code...13
14 def install(self, app):15 self.pre_install(app)16 super(ZenPack, self).install(app)17 self.post_install(app)18
19 def pre_install(self, app):20 """Perform work that must be done b efore normal ZenPack install."""21 # objects.xml assumes /Reports/Enterprise Reports exists.22 # Validating the organizer exists here is cleaner than defining a23 # hard requirement on the Enter priseReports ZenPack.24 app.zport.dmd.Reports.createOrganizer('Enterprise Reports')25
26 # Allow objects.xml to replace all the following monitoring templates.27 LOG.info('Preparing monitoring templates for updates')28 for subpath in REPLACED_MONITORING_TEMPLATES:29 try:30 template = app.zport.dmd.Devices.getObjByPath(subpath)31 template.getPrimaryParent()._delObject(template.id)32 except (KeyError, NotFound):33 pass
Post-Installation Migration
The post-installation has its own dependencies and is implemented as:
def post_install(self, app):"""Perform work that can be done after normal ZenPack install."""
self.symlinkPlugins()
# Remove event class mappings that this ZenPack supersedes.try:
net_link_ec = app.zport.dmd.Events.Net.Linkfor mapping_id in ('snmp_linkDown', 'snmp_linkUp'):
if mapping_id in net_link_ec.instances.objectIds():LOG.info("Removing standard %s mapping", mapping_id)net_link_ec.removeInstances((mapping_id,))
(continues on next page)
6.3. Migration Guide for Slackers 187
ZenPackers Documentation
(continued from previous page)
except (AttributeError, KeyError):pass
Dec 10, 2018
6.4 Monkey Patching in ZenPacks
Sometimes you need to monkey-patch platform code, or code in other ZenPacks when developing ZenPacks. Typicallyyou should look for ways to accomplish your objective without monkey-patching first.
Once you’ve decided that your ZenPack should do monkey-patching, you should consider the following convention.It has a couple of advantages such as making it clear what patches your ZenPack is performing, separating them intothe areas of code they patch, and handling cases where the code to be patched may not even be installed on the user’ssystem.
6.4.1 Where do the patches go?
Historically ZenPacks have tended to do all of their monkey-patching in their main init__.py. This was donebecause __init__.py is the only code in a ZenPack that is run automatically when any Zenoss Python processstarts. This makes it a great place to monkey-patch because you make sure your patch gets installed before it wouldbe called.
My advice to you is to not put your patches into __init__.py, but to build out the following structure instead.
1. Create a patches/ directory in the same directory as the ZenPack’s main __init__.py:
[bash]: mkdir -p patches/
2. Add an __init__.py into the patches directory with the following contents.
Note that the calls to optional_import() at the bottom should only include modules that you want topatch. This mechanism makes sure that the module in the first argument exists before it imports the modulenamed by the second argument from the patches/ directory:
from importlib import import_module
def optional_import(module_name, patch_module_name):"""Import patch_module_name only if module_name is importable."""try:
import_module(module_name)except ImportError:
passelse:
import_module('.{0}'.format(patch_module_name),'ZenPacks.zenoss.CiscoAPIC.patches')
optional_import('Products.ZenModel', 'platform')optional_import('ZenPacks.zenoss.vSphere', 'vSphere')
3. Add a module (file) into the patches/ directory for each module you want to patch.
188 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
For the above example this would mean adding platform.py and vSphere.py into the patches/ direc-tory:
[bash]: touch patches/platform.py[bash]: touch patches/vSphere.py
4. Add your platform patches into patches/platform.py.
The following example adds a new method to the standard Device class:
from Products.ZenUtils.Utils import monkeypatch
@monkeypatch('Products.ZenModel.Device.Device')def mycustom_usefulMethod(self):
"""Return something useful."""return "This is a useful return value."
5. Lastly import the patches module from the ZenPack’s main __init__.py.
The best practice is to import the patches module as the last thing you do in the ZenPack’s __init__.py.This can solve some import recursion problems. If your __init__.py is going to trigger code that needs toexecute the monkey-patched functionality, that part would have to go after the patches import:
# Patch last to avoid import recursion problems.from Products.ZenUtils.Utils import unusedfrom . import patchesunused(patches)
The first and last line are just there to keep pyflakes happy.
In the end you should have a directory and file structure something like the following. Files irrelevant to this monkey-patching exercise have been omitted:
ZenPacksexample/
MonkeyPatching/__init__.pypatches/
__init__.pyplatform.pyvSphere.py
6.5 Templating Topics
6.5.1 Override Standard Monitoring Templates
This question about how a user would override standard monitoring templates has come up a few times lately.
Here’s the best way of doing this:
1. Create a ZenPack such as ZenPacks.telus.vSphereExtensions
2. Specifically configure said ZenPack to depend on ZenPacks.zenoss.vSphere
3. Create the threshold in one of the monitoring templates delivered by ZenPacks.zenoss.vSphere
6.5. Templating Topics 189
ZenPackers Documentation
4. Add said monitoring template to the extension ZenPack
5. Export the extension ZenPack
6. Put the extension ZenPack in version control
Your must be aware and considerate that there’s now an ordering dependency on installing the original and extensionZenPacks. Anytime the original ZenPack is installed, you must make sure that the extension ZenPack is subsequentlyinstalled. Circumstances where this will happen are:
• Zenoss upgrade (only if the original ZenPack is installed by default with Zenoss)
• Zenoss RPS application (only if the original ZenPack is contained in the RPS)
• Manually installation of original ZenPack
All of this assumes that you don’t think the extension would make for a good out of the box default for the ZenPack.Let someone know if you think the change would be widely beneficial so we can incorporate it and save you thetrouble.
6.5.2 Load-Templates Script
This script is designed to take a yaml file as an input on the commandline and update a live instance of Zenoss.
Seee this link first: http://docs.zenosslabs.com/en/latest/zenpack_development/monitoring_http_api/using_yaml_templates.html
Note: load-templates requires PyYAML to be installed.
The commandline accepts a single argument, the filename:
python load-templates.py template_definition_file.yaml
The YAML formatting can be inferred from the zenoss doc for performance monitoring Doc-9385
yaml formatting: headers
The formatting section headers are the paths to the ZenPack. For example, Cisco lives under /Devices/Network, butCiscoUCS exists under /Devices.
The CiscoUCS heading might be:
/CiscoUCS:
The Cisco heading might be:
/Network/Cisco:
For a new hypothetical SpecialExample device that lives under /Devices/Network, the Yaml heading might be:
/Network/SpecialExample:
In addition, each of the components and subcomponents may also have a heading. SpecialExample has three com-ponents: Server, Client and Services. The headings for each of these components would look something like thefollowing:
/NetworkSpecialExample/SpecialExampleServer:/NetworkSpecialExample/SpecialExampleClient:/NetworkSpecialExample/SpecialExampleServices:
190 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
yaml formatting: sections
Each section may contain three sub-sections: datasources, thresholds and graphs.
Datasources:
The datasources sub-section contains configuration necessary to capture datapoints.
The datasources has a named source with several attributes associated with that source:
.../*header/path*:
...datasources:
*name_of_source*:type: datasource_type_
*attributes_associated_with_type01*: *attribute_value**attributes_associated_with_type02*: *attribute_value*...datapoints:
*name_of_source*: source_type_
*another_name*:type: source_type_rrdmin: *value*rrdmax: *value*aliases:
*alias*: *value*...
The hypothetical /Network/SpecialExample has an SNMP datasource, a command datasource and a JMX datasource:
/Network/SpecialExample:datasources:
hrMemoryUsed:type: SNMPoid: "1.3.6.1.2.1.25.2.3.1.6"datapoints:hrMemoryUsed: GAUGE_MIN_0
hrProcessorLoad:type: SNMPoid: "1.3.6.1.2.1.25.3.3.1.2.1"datapoints:hrProcessorLoad: GAUGE_MIN_0
specialExampleCommand:datasources:echo:type: COMMANDcommandTemplate: 'echo "OK|val1=123 val2=987.6'parser: Nagiosseverity: infocycletime: 10datapoints:val1:rrdmin: 0aliases:
(continues on next page)
6.5. Templating Topics 191
ZenPackers Documentation
(continued from previous page)
value1: "100,/"val2: DERIVE_MIN_0
heapMemoryUsage:type: JMXjmxPort: "12345"authenticate: TrueobjectName: "java.lang:type=Memory"attributeName: "HeapMemoryUsage"datapoints:committed: GAUGE_MIN_0used: GAUGE_MIN_0
nonHeapMemoryUsage:type: JMXjmxPort: "12345"authenticate: TrueobjectName: "java.lang:type=Memory"attributeName: "NonHeapMemoryUsage"datapoints:committed: GAUGE_MIN_0used: GAUGE_MIN_0
• SNMP:
• COMMAND:
• JMX:
RRD types:
• COUNTER - Saves the rate of change of the value over a step period. This assumes that the value is alwaysincreasing (the difference between the current and the previous value is greater than 0). Traffic counters on arouter are an ideal candidate for using COUNTER.
• GAUGE - Does not save the rate of change, but saves the actual value. There are no divisions or calculations.To see memory consumption in a server, for example, you might want to select this value.
**NOTE**Rather than COUNTER, you may want to define a data point usingDERIVED and with a minimum of zero. This creates the same conditionsas COUNTER, with one exception. Because COUNTER is a "smart" datatype, it can wrap the data when a maximum number of values isreached in the system. An issue can occur when there is a loss ofreporting and the system (when looking at COUNTER values) thinksit should wrap the data. This creates an artificial spike in thesystem and creates statistical anomalies.
• DERIVE - Same as COUNTER, but additionally allows negative values. If you want to see the rate of change infree disk space on your server, for example, then you might want to select this value.
• ABSOLUTE - Saves the rate of change, but assumes that the previous value is set to 0. The difference betweenthe current and the previous value is always equal to the current value. Thus, ABSOLUTE stores the currentvalue, divided by the step interval.
192 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
RRD suffixes:
In addition, suffixes can be added to reduce YAML:
• _MIN_*value* - sets rrdmin to value
• _MAX_*value* - sets rrdmax to value
Examples:
• GAUGE_MIN_0_MAX_100 - sets the rrd type to gauge; rrd minimum to 0 and the rrd maximum to 100
• DERIVE_MAX_10 - sets the rrd type to derive; rrd maximum to 10
Thresholds:
• The thresholds sub-section contains configuration necessary to capture thresholds with relation to the data-sources datapoints.
Note:
*Make sure that the threshold is really needed. Too manyextra events may be overwhelming to a user.*
The general format for the thresholds is as follows:
.../*header/path*:
...thresholds:
*human friendly name*:type: threshold_type_dsnames: ["*datasource_name*_*datapoint_name*"]
*attributes_associated_with_type01*: *attribute_value**attributes_associated_with_type02*: *attribute_value*...
The hypothetical /Network/SpecialExample has an SNMP threshold, and a Command threshold. The SNMP thresholdlooks for a processor load of greater than 95%. The Command threshold looks for a value greater than 99.
/Network/SpecialExample:datasources:
...
thresholds:high load:
type: MinMaxThresholddsnames: ["hrProcessorLoad_hrProcessorLoad"]maxval: "95"
high values:type: MinMaxThresholddsnames: ["ds1_val1", "ds1_val2"]maxval: "99"
Standard Types:
6.5. Templating Topics 193
ZenPackers Documentation
• MinMaxThreshold -
• ValueChangeThreshold -
• CiscoStatus -
• HoltWintersFailure -
Graphs:
• The graphs sub-section contains the configuration necessary to capture graphs with relation to the thresholdsand datasources datapoints:
.../*header/path*:
...graphs:
*human friendly graph title*:units: "human friendly units"miny: *y-axis minimum value*maxy: *y-axis maximum value*graphpoints:
*human friendly datapoint name*:dpName: "*datasource*_*datapoint*"format: rrd_graph_type_format_rpn: *reverse_polish_notation*
The /Network/SpecialExample device has several graphs that need to be displayed. More specifically, the Servercomponents utilize the SNMP, the clients utilize JMX and the Services require a Command.
/Network/SpecialExample/SpecialExampleServer:graphs:
CPU Utilization:units: "percent"miny: 0maxy: 100graphpoints:Used:dpName: "hrProcessorLoad_hrProcessorLoad"format: "%4.0lf%%"
Memory Utilization:units: "percent"miny: 0maxy: 100graphpoints:Used:dpName: "hrMemoryUsed_hrMemoryUsed"format: "%7.2lf%%"rpn: "1024,*,${here/hw/totalMemory},/,100,*"
/Network/SpecialExample/SpecialExampleClient:graphs:
Values:units: numberminy: 0graphpoints:Value 1:dpName: "ds1_val1"
(continues on next page)
194 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
(continued from previous page)
format: "%7.2lf%s"Value 2:
dpName: "ds1_val2"format: "%7.2lf%s"
/Network/SpecialExample/SpecialExampleService:JVM Memory Usage:
units: bytesbase: trueminy: 0graphpoints:Heap Committed:dpName: heapMemoryUsage_committed
Heap Used:dpName: heapMemoryUsage_used
NonHeap Committed:dpName: nonHeapMemoryUsage_committed
NonHeap Used:dpName: nonHeapMemoryUsage_used
Stolen from: http://oss.oetiker.ch/rrdtool/doc/rrdgraph_graph.en.html
%% Just prints a literal ‘%’ character
%#.#le Prints numbers like 1.2346e+04. The optional integers # denote field width and decimal precision.
%#.#lf Prints numbers like 12345.6789, with optional field width and precision.
%s Place this after %le, %lf or %lg. This will be replaced by the appropriate SI magnitude unit and thevalue will be scaled accordingly (123456 -> 123.456 k).
%S is similar to %s. It does, however, use a previously defined magnitude unit. If there is no such unityet, it tries to define one (just like %s) unless the value is zero, in which case the magnitude unit staysundefined. Thus, formatter strings using %S and no %s will all use the same magnitude unit except forzero values.
If you PRINT a VDEF value, you can also print the time associated with it by appending the string:strftime to the format. Note that RRDtool uses the strftime function of your OSs C library. This meansthat the conversion specifier may vary. Check the manual page if you are uncertain. The following is alist of conversion specifiers usually supported across the board.
%a The abbreviated weekday name according to the current locale.
%A The full weekday name according to the current locale.
%b The abbreviated month name according to the current locale.
%B The full month name according to the current locale.
%c The preferred date and time representation for the current locale.
%d The day of the month as a decimal number (range 01 to 31).
%H The hour as a decimal number using a 24-hour clock (range 00 to 23).
%I The hour as a decimal number using a 12-hour clock (range 01 to 12).
%j The day of the year as a decimal number (range 001 to 366).
%m The month as a decimal number (range 01 to 12).
6.5. Templating Topics 195
ZenPackers Documentation
%M The minute as a decimal number (range 00 to 59).
%p Either ‘AM’ or ‘PM’ according to the given time value, or the corresponding strings for the currentlocale. Noon is treated as ‘pm’ and midnight as ‘am’. Note that in many locales and ‘pm’ notation isunsupported and in such cases %p will return an empty string.
%s The second as a decimal number (range 00 to 61).
%S The seconds since the epoch (1.1.1970) (libc dependent non standard!)
%U The week number of the current year as a decimal number, range 00 to 53, starting with the firstSunday as the first day of week 01. See also %V and
%V The ISO 8601:1988 week number of the current year as a decimal number, range 01 to 53, whereweek 1 is the first week that has at least 4 days in the current year, and with Monday as the first day of theweek. See also %U and
%w The day of the week as a decimal, range 0 to 6, Sunday being 0. See also %u.
%W The week number of the current year as a decimal number, range 00 to 53, starting with the firstMonday as the first day of week 01.
%x The preferred date representation for the current locale without the time.
%X The preferred time representation for the current locale without the date.
%y The year as a decimal number without a century (range 00 to 99).
%Y The year as a decimal number including the century.
%Z The time zone or name or abbreviation.
%% A literal ‘%’ character.
Events:
• Not Yet Implemented
6.5.3 Automatic Templates for Zenpacks
Templating is an important aspect of data modeling and colleciton. This article explores automatic template construc-tion for modeling.
Prerequisites
• The usual
Automatically Applied Setters and Getters on Class Objects
In Products/DataCollector/ApplyDataMap.py there is code that automatically executes all methods in your class thatstart with In order for this to work, follow these general steps:
• Create Methods getXYZ(self, data), setXYZ(self, data)
• Create data attribute ‘setXYZ’ (in your datamap) for the setter in modeling
• Apply that datamap to your model. ApplyDataMap will process and apply it. Some details:
• getXYZ(self, data)
196 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
• setXYZ(self, data)
The data that is referenced must be provided by your modeling template in the form of a map attribute with the samename as the method:
{'setMonitoringTemplateData': attributes['MonitoringProfile'],'someOtherAttribute': somedata,... etc ...
}
Example of Auto-Generated and Applied Local Templates
The context of this is using ZenPackLib and the ZenPacks.zenoss.ControlPlane . The goal is to have an automaticallygenerated template to be locally applied to every component object that gets created.
Based on an API call to the ControlPlane, you need to:
* Create the model data, based on API defs.
* Insert Data into datamap.
* Append the map to the object map. ApplyDataMap processes all
* Object Instantiation: Create the Local Template based on ObjectMaps
* Ensure template attribute's name matches a getter in object class.
Create the Model Data
In our example we create a “MonitoringTemplateData” based on the API’s return value of MonitoringProfile data:
def map_pool(attributes):"""Return ObjectMap data given classname and attributes. """
return {'id': get_pool_id(attributes['ID']),'title': attributes['ID'],'set_parentPool': get_pool_id(attributes['ParentID']),'priority': attributes['Priority'],'coreLimit': attributes['CoreLimit'],'memoryLimit': attributes['MemoryLimit'],'setMonitoringTemplateData': attributes['MonitoringProfile'],}
Insert Data into Datamap
Later on this data is processed in DataMapProducer class (abbreviated):
class DataMapProducer(object):"""Produce the DataMap objects required to model """
def __init__(self, client):self.client = client
@inlineCallbacksdef getmaps(self):
(continues on next page)
6.5. Templating Topics 197
ZenPackers Documentation
(continued from previous page)
""" Return datamaps to modeler. """
maps = []
pools = yield self.client.pools()pool_maps = []for attributes in pools:
pool_map = map_pool(attributes)if pool_map:
pool_maps.append(pool_map)
Understand that these modules are called by the Modeler and get fed directly to the other core services. The nextsection is also part of the same modeler:
Append the Map to the Object Map
This is the usual and traditional step for appending object maps. Once you have the dictionary data (pool_maps in ourcase) for your object or component, process it in the usual way:
maps.append(RelationshipMap(
relname='pools',modname='ZenPacks.zenoss.ControlPlane.Pool',objmaps=pool_maps))
returnValue(maps)
Ensure Template Attribute’s Name Matches a Getter in Object Class
After the modeler hands these maps off to the other ancillary parts of the core, the data in the object classes getsinitialized with the map data.
Consider the Pool.py class code:
# ZenPack Importsfrom . import schemafrom .utils import replaceLocalTemplate
class Pool(schema.Pool):
"""Custom model code for Pool class."""
_monitoringTemplateData = None
def getMonitoringTemplateData(self):"""Return last set monitoring template data."""return self._monitoringTemplateData
def setMonitoringTemplateData(self, data):"""Create local monitoring template using data."""replaceLocalTemplate(self, data, 'ZenPacks.zenoss.ControlPlane.Pool')
Notice that the setMonitoringTemplateData is exactly the same name as that of the mapped data above. This is criticalsince the ApplyDataMap class is looking for this match, and without it the process fails.
198 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
Object Instantiation: Create the Local Template based on ObjectMap Data
Once ApplyDataMap gets the data above, it calls setMonitoringTemplateData() with the associated data and configuresall the templates.
If we look in the util.py import you’ll find the replaceLocalTemplate(obj, data, targetPythonClass) helper utility whichconfigures that actual Template: ( See reference: https://github.com/zenoss/ZenPacks.zenoss.Microsoft.Windows/blob/develop/load-templates)
def replaceLocalTemplate(obj, data, targetPythonClass):"""Replace local monitoring template using data."""obj._monitoringTemplateData = data
# Delete the local template if it already exists.template_name = obj.getRRDTemplateName()if not template_name:
return
template = getattr(aq_base(obj), template_name, None)if template:
obj._delObject(template_name)
# Create the template.template = RRDTemplate(template_name)obj._setObject(template.id, template)template = obj._getOb(template_name)
# Configure the template.template.description = data.get('Description', '')template.targetPythonClass = targetPythonClass
# Add datasources to the template.for metricConfig in data.get('MetricConfigs', []):
datasource = template.manage_addRRDDataSource(metricConfig['ID'],'ControlPlaneDataSource.Control Plane')
# Configure the datasource.datasource.component = '${here/id}'datasource.eventClass = '/Ignore'datasource.severity = 0datasource.cycletime = 300datasource.perfURL = metricConfig['PerfURL']
# Add datapoints to the datasource.for metric in metricConfig['Metrics']:
datapoint = datasource.manage_addRRDDataPoint(metric['ID'])
# Add a default alias for each datapoint.datapoint.addAlias(metric['ID'])
6.5. Templating Topics 199
ZenPackers Documentation
6.6 Unit Testing Rough Guide
6.6.1 Description
Zenpacks need to be tested for internal consistency at build time by Jenkins.
6.6.2 Introduction
Unit Tests are becoming more important as continuous deployment technology grows. These tests provide a sanitycheck for any software that gets moved into production.
Unit Tests in Zenoss are intended to test major components of ZenPacks whenever possible.
6.6.3 Jenkins
Jenkins builds a Zenoss environment in order to test the Zenpacks at build time. This means that it does have a liveversion of Zenoss to test against. At this time however, Jenkins is not configured to probe Unit Tests.. You will haveto test them manually (see below)
6.6.4 Manual Testing
You can test Unit Tests manually with runtests as follows:
• Your test class TestCheckOracle is part of ZenPacks.zenoss.DatabaseMonitor (see below for example).
• You wish to test this entire ZenPack’s unit tests.
• You are on the zenoss system as user zenoss:
[zenoss@cdev:$zpdir/tests]: runtests --type unit ZenPacks.zenoss.DatabaseMonitor==============================Packages to be tested:
ZenPacks.zenoss.DatabaseMonitor==============================Parsing /opt/zenoss/etc/zope.confRunning tests at level 1Running Products.ZenTestCase.BaseTestCase.ZenossTestCaseLayer tests:
Set up Testing.ZopeTestCase.layer.ZopeLite in 0.422 seconds.Set up Products.ZenTestCase.BaseTestCase.ZenossTestCaseLayer in 0.002 seconds.Running: ..Ran 2 tests with 0 failures and 0 errors in 4.450 seconds.
Tearing down left over layers:Tear down Products.ZenTestCase.BaseTestCase.ZenossTestCaseLayer in 0.000
→˓seconds.Tear down Testing.ZopeTestCase.layer.ZopeLite in 0.000 seconds.
In this case we ran 2 tests and had 0 failures. We got lucky.
You can also specify the module to run:
runtests --type unit ZenPacks.zenoss.DatabaseMonitor -m test_modeler
200 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
6.6.5 Unit Test Framework
The unit test framework test functionality in isolation of any monitored devices. It inherits from the class BaseTest-Case and uses it’s assert methods for determining pass or failure.
• Unit tests inherit from BaseTestCase
• All tests go in the $ZPDIR/tests/ folder
• Inside that folder you need to create your test classes which will be called from the following method:
from Products.ZenTestCase.BaseTestCase import BaseTestCasefrom ZenPacks.zenoss.GoodieMonitor.Candy import Candy
class TestGoodies(BaseTestCase):
def testCheckGoodies(self):item = Candy("chocolate")state = item.getSweetOrSourself.assertTrue("sweet" in state, "Item %s not sweet" % item)
def test_suite():from unittest import TestSuite, makeSuitesuite = TestSuite()suite.addTest( makeSuite(TestGoodies) )return suite
6.6.6 DataSource Example
This example shows the test code we created above. We assume:
• You are in your $ZP_DIR/tests/ folder
• You are in your dev environment.
• You have created an empty (or otherwise) __init__.py file
• Your test has a fake or simulated set of data, ie.
• You cant’t rely on real device to gather data.
• The filename we create is test_datasources_plugin.py:
########################################################################### Copyright (C) Zenoss, Inc. 2013, all rights reserved.# test_datasources_plugin.py##########################################################################
from Products.ZenTestCase.BaseTestCase import BaseTestCase
class TestCheckOracle(BaseTestCase):'''This calls check_oracle.py -c "connectionString" -q "query" -tRewrote check_oracle with -t (test) flag, adjusted txojdbc.py'''
def testCheckOracle(self):import subprocessimport os
(continues on next page)
6.6. Unit Testing Rough Guide 201
ZenPackers Documentation
(continued from previous page)
connectionString = "zenoss/[email protected]:1521:XE"query = 'select * from v$sysstat'path=os.path.join(os.path.dirname(__file__), "..")checkOracle=os.path.join(path, "check_oracle.py")
output=subprocess.check_output(["python", checkOracle, "-c",connectionString, "-q", query, "-t"])
outputRequired="logonscumulative"
# BaseTestCase.assertTrue is the method that determines pass/failself.assertTrue(outputRequired in output,
"Output does not contain valid data %s" % outputRequired)
def test_suite():from unittest import TestSuite, makeSuitesuite = TestSuite()suite.addTest(makeSuite(TestCheckOracle))return suite
In this example, the testCheckOracle method of TestCheckOracle will be tested. The check_oracle.py will call aroutine (txojdbc.py) that has some pre-made flat files of JSON data, so there is no dependency on an actual device tomonitor for data. This is critical because eventually Jenkins will have to run the unit tests is a vacuum environment.
Note: The BaseTestCase.assertTrue is the key method that you need to determine pass/fail of your test. If this testcas the “logonscumulative” string, it passes, otherwise it fails. In general you need one of the assert methods in theBaseTestCase class
6.6.7 Impact Example
This example shows how to test Impact. We assume
• You are in your $ZP_DIR/tests/ folder
• You are in your dev environment.
• You have created an empty (or otherwise) __init__.py file
• You have your environment setup with Impact installed (for testing).
This example uses a lot of boilerplate code. It is much simpler than the XenServer unit tests though. It can beconsidered one of the simplest impact tests you will find, because the Instance class is only dependent on the containingserver. Nothing depends on Instance.
Most of the code is simply building a node-link tree diagram. The two methods that are non-boilerplate are:
• create_endpoint()
• The (decorated) test_Instance():
Notice also that the tests will always pass if Impact ZP is not installed so you won’t be able to test it properly.
################################################################################ Copyright (C) Zenoss, Inc. 2013, all rights reserved.## This content is made available according to terms specified in
(continues on next page)
202 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
(continued from previous page)
# License.zenoss under the directory where your Zenoss product is installed.###############################################################################
'''Unit test for all-things-Impact.'''
import transactionfrom zope.component import subscribersfrom Products.Five import zcmlfrom Products.ZenTestCase.BaseTestCase import BaseTestCasefrom Products.ZenUtils.guid.interfaces import IGUIDManagerfrom Products.ZenUtils.Utils import monkeypatch
from ZenPacks.zenoss.DatabaseMonitor.utils import guid, require_zenpackfrom ZenPacks.zenoss.DatabaseMonitor.tests.utils import (
add_contained, add_noncontained,)
@monkeypatch('Products.Zuul')def get_dmd():
'''Retrieve the DMD object. Handle unit test connection oddities.
This has to be monkeypatched on Products.Zuul instead ofProducts.Zuul.utils because it's already imported into Products.Zuulby the time this monkeypatch happens.'''try:
# original is injected by the monkeypatch decorator.return original()
except AttributeError:connections = transaction.get()._synchronizers.data.values()[:]for cxn in connections:
app = cxn.root()['Application']if hasattr(app, 'zport'):
return app.zport.dmd
def impacts_for(thing):'''Return a two element tuple.
First element is a list of object ids impacted by thing. Second element isa list of object ids impacting thing.'''from ZenPacks.zenoss.Impact.impactd.interfaces \
import IRelationshipDataProvider
impacted_by = []impacting = []
guid_manager = IGUIDManager(thing.getDmd())for subscriber in subscribers([thing], IRelationshipDataProvider):
(continues on next page)
6.6. Unit Testing Rough Guide 203
ZenPackers Documentation
(continued from previous page)
for edge in subscriber.getEdges():if edge.source == guid(thing):
impacted_by.append(guid_manager.getObject(edge.impacted).id)elif edge.impacted == guid(thing):
impacting.append(guid_manager.getObject(edge.source).id)
return (impacted_by, impacting)
def triggers_for(thing):'''Return a dictionary of triggers for thing.
Returned dictionary keys will be triggerId of a Trigger instance andvalues will be the corresponding Trigger instance.'''from ZenPacks.zenoss.Impact.impactd.interfaces import INodeTriggers
triggers = {}
for sub in subscribers((thing,), INodeTriggers):for trigger in sub.get_triggers():
triggers[trigger.triggerId] = trigger
return triggers
def create_endpoint(dmd):'''Return an Endpoint suitable for Impact functional testing.This is non-boilerplate code...'''# DeviceClassdc = dmd.Devices.createOrganizer('/Server/Linux')dc.setZenProperty('zPythonClass', '')linux = dc.createInstance('linux')
# Instancefrom ZenPacks.zenoss.DatabaseMonitor.Instance import Instanceadd_contained(linux, 'instances', Instance('instance1'))
return linux
class TestImpact(BaseTestCase):def afterSetUp(self):
super(TestImpact, self).afterSetUp()
import Products.ZenEventszcml.load_config('meta.zcml', Products.ZenEvents)
try:import ZenPacks.zenoss.DynamicViewzcml.load_config('configure.zcml', ZenPacks.zenoss.DynamicView)
except ImportError:return
(continues on next page)
204 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
(continued from previous page)
try:import ZenPacks.zenoss.Impactzcml.load_config('meta.zcml', ZenPacks.zenoss.Impact)zcml.load_config('configure.zcml', ZenPacks.zenoss.Impact)
except ImportError:return
import ZenPacks.zenoss.DatabaseMonitorzcml.load_config('configure.zcml', ZenPacks.zenoss.DatabaseMonitor)
def endpoint(self):'''Return a DatabaseMonitor endpoint device populated in a suitable wayfor Impact testing.'''if not hasattr(self, '_endpoint'):
self._endpoint = create_endpoint(self.dmd)
return self._endpoint
def assertTriggersExist(self, triggers, expected_trigger_ids):'''Assert that each expected_trigger_id exists in triggers.'''for trigger_id in expected_trigger_ids:
self.assertTrue(trigger_id in triggers, 'missing trigger: %s' % trigger_id)
@require_zenpack('ZenPacks.zenoss.Impact')def test_Instance(self):
'''Decorator will disable tests if required ZenPacks are not installed!ZenPacks.zenoss.Impact and ZenPacks.zenoss.DynamicView must be installed!Jenkins will eventually be setup to do unit tests at build time.....'''instance1 = self.endpoint().getObjByPath('instances/instance1')impacts, impacted_by = impacts_for(instance1)
# Host -> Instanceself.assertTrue(
'linux' in impacted_by,'missing impact: {} -> {}'.format('linux', instance1))
def test_suite():from unittest import TestSuite, makeSuitesuite = TestSuite()suite.addTest(makeSuite(TestImpact))return suite
6.6.8 Another Simple Example
Here is another simple example that may help:
6.6. Unit Testing Rough Guide 205
ZenPackers Documentation
from Products.Five import zcml
from Products.ZenTestCase.BaseTestCase import BaseTestCasefrom Products.Zuul.interfaces import IReportable
from ZenPacks.zenoss.OpenVZ.Container import Container
class TestAnalytics(BaseTestCase):def afterSetUp(self):
super(TestAnalytics, self).afterSetUp()
# Required to prevent erroring out when trying to define viewlets in# ../browser/configure.zcml.import Products.ZenUI3.navigationzcml.load_config('testing.zcml', Products.ZenUI3.navigation)
import ZenPacks.zenoss.OpenVZzcml.load_config('configure.zcml', ZenPacks.zenoss.OpenVZ)
def testContainerReportable(self):device = self.dmd.Devices.createInstance('openvz_host')
container = Container('101')device.openvz_containers._setObject(container.id, container)container = device.openvz_containers._getOb(container.id)
reportable = IReportable(container)report_properties = reportable.reportProperties()
self.assertEqual(reportable.entity_class_name, 'container')
self.assertEqual(len(report_properties), 3)self.assertEqual(report_properties[0][0], 'id')# .. and so on..
def test_suite():from unittest import TestSuite, makeSuitesuite = TestSuite()suite.addTest(makeSuite(TestAnalytics))return suite
6.6.9 DMD fixtures
You could use dmd in descendants of BaseTestCase after it afterSetUp method was executed.
But, you need to remember that this dmd will be clean, so, if you want to use something like dmd.Devices.Network.Router it will give you AttributeError: Network.
To create device class you need to call dmd.Devices.createOrganizer('/Network').
6.6.10 Testing IEventService with Crochet and Twisted
References: * http://docs.zope.org/zope.component/narr.html * Special Events and Their Consequences
206 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
If you need to test a Modeling or Collection that has an IEvent service, you have code that looks like this:
import zope.componentfrom Products.ZenCollector.interfaces import IEventService
...
...class Temper(object):
def __init__(self, calamari, graphite, device):self._eventService = zope.component.queryUtility(IEventService)
evt = dict(device=device.id,component='',summary=summary,severity=severity,eventClass='/Status',eventClassKey='CephTemperConfigurationClass',
)self._eventService.sendEvent(evt)........def someMethod():
evt = dict(device=device.id,component='',summary=summary,severity=severity,eventClass='/Status',eventClassKey='CephTemperConfigurationClass',)
self._eventService.sendEvent(evt)
If you try to execute a test on this code from UnitTests you may encounter this type of traceBack:
...File "/opt/zenoss/lib/python2.7/site-packages/crochet-1.5.0-py2.7.egg/crochet/_→˓eventloop.py", line 461, in wrapperreturn eventual_result.wait(timeout)
File "/opt/zenoss/lib/python2.7/site-packages/crochet-1.5.0-py2.7.egg/crochet/_→˓eventloop.py", line 231, in wait
result.raiseException()File "/opt/zenoss/lib/python/twisted/python/failure.py", line 338, in raiseException
raise self.type, self.value, self.tbAttributeError: 'NoneType' object has no attribute 'sendEvent'
The problem is that there is no sendEvent() method inside the Unit Test environment. That is to say you need toprovide the IEventService as a singleton utility. You also need to define the sendEvent() method.
The corrected Unit Test code should look like this now:
import zope.componentfrom Products.ZenCollector.interfaces import IEventService
class TestOSI(zenpacklib.TestCase):
def setUp(self):super(TestModeling, self).setUp()
(continues on next page)
6.6. Unit Testing Rough Guide 207
ZenPackers Documentation
(continued from previous page)
zope.component.provideUtility(self, IEventService)
def sendEvent(self, event, **kw):pass
...
...-- code that tests your original (modeler, collector) ---
6.7 ZEP Topics
ZEP is the Zenoss Event Processor. We’ll just include a few snippets that are floating around.
6.7.1 Close Old Acknowledged Events
import time
from zenoss.protocols.protobufs.zep_pb2 import (SEVERITY_CRITICAL,SEVERITY_ERROR,STATUS_ACKNOWLEDGED,)
from Products.Zuul import getFacade
now = int(time.time()) * 1000one_hour_ago = now - (3600 * 1000)
# last_seen can be a range: beginning of range, end of rangelast_seen = (one_hour_ago, now)
zep = getFacade('zep')event_filter = zep.createEventFilter(
status=[STATUS_ACKNOWLEDGED],severity=[SEVERITY_ERROR, SEVERITY_CRITICAL],last_seen=last_seen,)
zep.closeEventSummaries(eventFilter=event_filter)
6.8 Bread and Butter: A Quick Primer on Pickles
Sometimes you need a way to preserve objects in their exact state to be reproduced later. This is especially useful fortesting, or recreating customer issues. Fortunately, Python offers a facility to do just this: Pickles. For a bit more aboutpickles, check the Python docs: https://docs.python.org/2/library/pickle.html
6.8.1 Mix Up Some Brine: How to Make a Pickle
Any object in Python can be pickled simply by calling:
208 Chapter 6. Special Zenpack Development Topics
ZenPackers Documentation
import picklewith open(pickle_filename, 'wb') as pickle_file:
pickle.dump(results, pickle_file, 2)
The following patch, if applied to zenmodeler.py, will create pickle files for all the datamaps generated during model-ing:
diff --git a/Products/DataCollector/zenmodeler.py b/Products/DataCollector/zenmodeler.→˓pyindex 386847f..ef4eb54 100644--- a/Products/DataCollector/zenmodeler.py+++ b/Products/DataCollector/zenmodeler.py@@ -634,6 +634,26 @@ class ZenModeler(PBDaemon):
try:results = plugin.preprocess(results, self.log)if results:
+###############################################################################+ import os+ if not os.path.isdir(device.id):+ os.mkdir(device.id)++ plugin_prefix_length = None+ if plugin.__module__.startswith('ZenPacks'):+ plugin_prefix_length = 5+ else:+ plugin_prefix_length = 0++ pickle_filename = '%s/%s.pickle' % (+ device.id, '.'.join(+ plugin.__module__.split(+ '.')[plugin_prefix_length:]))++ self.log.info("Dumping results to %s", pickle_filename)+ with open(pickle_filename, 'wb') as pickle_file:+ pickle.dump(results, pickle_file, 2)+###############################################################################
datamaps = plugin.process(device, results, self.log)if datamaps:
pluginStats.setdefault(plugin.name(), plugin.weight)
6.8.2 Opening the Pickle Jar
After you have save an object to a Pickle file, you can load it back again as a Python object as follows:
import pickledatamaps = pickle.load("filename")
Once loaded, you can interact with the object just as you would any other object.
6.8.3 I Wanted Sweet, Not Dill: Modifying Pickles
Pickles are saved as binary files, which can be easily copied and transferred. However, they are not so easy to read,or edit. In some cases you may want to take an existing pickle and inspect its contents, or even update the object’sattributes. The following script is a sort editor for pickle files. It loads the pickle file from the specified path to an
6.8. Bread and Butter: A Quick Primer on Pickles 209
ZenPackers Documentation
object, and then drops you to an interactive Python shell, where you can interact with the object however you need to.Once finished, you can write the object back to the original pickle file.
This was something I created for a specific purpose, but should be generally useful. However, it has not been testedextensively. Please update this doc if you find any bugs.
#!/usr/bin/env python
# from IPython import embedfrom pprint import pprintimport pickleimport os.pathimport osimport sysfrom datetime import datetime
# Greetingprint "Welcome to edPick: a simple editor for pickle files\n"
# Get file pathpath = sys.argv[1] if 1 < len(sys.argv) else ''valid_path = os.path.isfile(path)while not valid_path:
path = raw_input('Enter file path [./modeling_data.pickle]: ')path = path or './modeling_data.pickle'if os.path.isfile(path) :
valid_path = Trueelse:
print "Weird. I can't find that file. Try again.\n"
# Load pickled datadata = pickle.load( open( path, "rb" ) )pprint(data)
# Modify dataprint "Modify the <data> object as needed.\n"print "Then hit Ctrl-D to exit interpreter and write out your new pickle file.\n"print "Or hit Ctrl-Z to abort.\n"# embed()import readline # optional, will allow Up/Down/History in the consoleimport codevars = globals().copy()vars.update(locals())shell = code.InteractiveConsole(vars)shell.interact()
# Backup old filesuffix = '.'+datetime.fromtimestamp(os.path.getmtime(path)).strftime('%Y%m%d-%H%M')+'.→˓old'os.rename(path,path+suffix)
# Write out new pickle datapickle.dump( data, open( path, "wb" ) )print "<data> has been written in place of the file you specified.\n"print "Your original pickle file has been moved to '%s%s'. You can remove it if you→˓do not need it.\n" % (path, suffix)
Just save it to your path, and you can run it whenever you need it.
210 Chapter 6. Special Zenpack Development Topics
CHAPTER 7
Topics for ZenpackLib
Documentation for ZenPackLib
7.1 Welcome to ZenPack Library
The ZenPackLib allows you to automate much of the ZenPack creation process. It does this by:
• Automating boilerplate code into dynamically created classes and modules
• Automating certain Zenpack components like the GUI and ZCML
• Defining standard classes and modules needed for Zenpack modification
7.1.1 ZenPackLib Overview
The ZenPackLib allows you to automate much of the ZenPack creation process.
Main Reference: http://zenpacklib.zenoss.com
Description
ZenPackLib’s use of automatic class creation requires a bit of explanation. We will attempt to cover some of this here.The official documentation will be listed at . . . .
Prerequisites
• Zenoss ZenPack Developement
• Python 2.7
• ZenPackLib familiarity
211
ZenPackers Documentation
We assume that you are familiar with ZenPack developement and Python coding. We work from the base of $ZP_DIR.A few things to ensure:
• You have created a DeviceClass for your ZP
• You have set zPythonClass for your DeviceClass to Zenpacks.zenoss.ZP.Class. For example: Zen-packs.zenoss.ControlCenter.ControlCenter
Class Definition and Overview (Simplified Form)
In your class definition (possibly __init__.py) you will have definitions of your Pool classes like:
1 RELATIONSHIPS_YUML = """2 // --------------------------------------------3 // Containing Relations4 // --------------------------------------------5 [ControlCenter]++pools -controlcenter[Pool]6 // --------------------------------------------7 // Non-containing Relations8 // --------------------------------------------9 [Pool]*parentPool -.-childPools 0..1[Pool]
10 """11
12 CFG = zenpacklib.ZenPackSpec(13 name=__name__,14
15 zProperties={16 'DEFAULTS': {'category': 'Control Plane'},17
18 'zControlCenterHost': {},19 'zControlCenterPort': {'default': '8787'},20 },21
22 classes={23 'DEFAULTS': {24 'base': zenpacklib.Component,25 },26
27 'ControlCenter': {28 'base': zenpacklib.Device,29 'meta_type': 'ZenossControlCenter',30 'label': 'Zenoss Control Plane',31 },32
33 'Pool': {34 'meta_type': 'ZenossControlCenterPool',35 'label': 'CP-Pool', # <-- *** Note this label is ZPL magic.36 # ZenpackLib: Properties that Auto-magically appear in GUI37 'properties': {38 'priority': {'label': 'Priority'},39 'coreLimit': {'label': 'CPU Core Limit'},40 'memoryLimit': {'label': 'Memory Limit'},41 },42 'relationships': {43 # This is required since the pool <=> pool relationship44 # can cause recursive ambiguity problems. This breaks that symmetry.45 'parentPool': {'label': 'Parent', 'order': 1.0},
(continues on next page)
212 Chapter 7. Topics for ZenpackLib
ZenPackers Documentation
(continued from previous page)
46 'childPools': {'label': 'Children', 'order': 1.1},47 }48 },49 },50 class_relationships=zenpacklib.relationships_from_yuml(RELATIONSHIPS_YUML),51 )52 CFG.create()
Special things to note and observe:
•Note: When you add and new zProperties to your Zenpack you MUST re-install your Zenpack. This is becausethose properties get created in the ZODB only when you install. New classes create relationship maps thatsubsequently are stored in ZODB. If your new class doesn’t create a new zProperty then you can get away witha zenos restart and a remodel.
• name is set to the name of the class which will resolve to ZenPacks.zenoss.ControlCenter . This should typicallynot be changed.
• DEFAULTS is a special value that will cause it’s properties to be added as the default for all of the other listedzProperties. Otherwise you have to add them manually to each zProp.
• The zProp entry for zControlCenterHost is a shorthand for the more verbose:
'zControlCenterHost': {'type': 'string', 'default': ''}
• The class DEFAULTS specifies that all classes will be sub-classes of the standard zenpacklib.Component bydefault. We could choose from:
– zenpacklib.Device
– zenpacklib.Component
– A user defined class
• The Pools class is automatically created based on YAML or YUML definition
• Go to Advanced->Monitoring Templates: Hit the + at bottom left
• The ‘relationships’ labels are added in order to disambiguate parent-child names. The parent and child nameprefix are ZPL automagically determined from the YUML spec defined in RELATIONSHIPS_YUML.
• The label attached to Pool is “CP-Pool”. It exists to disambiguate the relationship between Pool and contained-Pool objects.
• In particular, you will need to.
– Create: A template for each label with the EXACT same name as label.
– Ensure: Template is in the appropriate Template Path (/ControlCenter)
– Ensure: All relationship names are unique in the YUML spec
About YAML Relationships Map
See the official documentation for this at
• http://zenpacklib.zenoss.com/en/latest/
In general the relationships look like this:
7.1. Welcome to ZenPack Library 213
ZenPackers Documentation
class_relationships:- ComponentA Cardinality:Cardinality ComponentB
Cardinality can be one of the following:
• 1:MC => One-to-Many containing
• 1:M => One-to-Many
• M:M => Many-to-Many
• 1:1 => One-to-One
For example:
class_relationships:- CephEndpoint 1:MC CephComponent- CephHost 1:M CephService- CephPool M:M CephOSD- CephPool 1:M CephRBD- CephPool 1:M CephSnap- CephRGW 1:M CephPool
About YUML Relationships Map (Deprecated)
The YUML relationship maps have a very specific format. In the following generic form:
[LeftClass](l_cardinality)leftToRightName (seperator) rightToLeftName(r_→˓cardinality)[RightClass]
For example:
[Service]0..1serviceRuns -.-serviceDef *[Running]
• [LeftClass] and [RightClass] are classes
• The cardinalities can be: (*, 0..1, 1..7, 1, +)
• LeftToRightName and RightToLeftName are the labels that identify the relationships created. For example:
– The pools relationship on ControlCenter defines the contained pools.
– The controlcenter relationship on Pool defines the containing controlcenter.
• Relationships do not need a name unless there is ambiguity in relations. I recommend naming all your relationsthough just incase you later add a relationship that ambiguates your schema.
•Note: Make Sure All Relationships Have Unique Names! If relationships don’t have unique names ZPL willnot be able to process the relationships in a predictable way. Make sure all relation names are unique and youshould be ok.
Class Definition: Advanced Topics
In the begginning there is ZenPacks.zenoss.XYZ class. Its created by Zenoss when you create the class and install the__init__.py. ZPL creates these two objects by defaults:
214 Chapter 7. Topics for ZenpackLib
ZenPackers Documentation
.schema: A module that allows customization (overrides) of the ZPLcreated Zenpack class
.ZenPack: The class that contains all the properties, install(), remove(),and cleanup methods for the Zenpack.
When ZPL creates any components (for example, Pool), it creates several objects relative to ZenPacks.zenoss.XYZ:
.Pool : The Pool component class itself
.schema.Pool : The Pool schema space for class modification
If you don’t create your own Pool.py class file (analagous to .ZenPack), ZPL will do this for you. Again, this is forproperty managment and initializations.
Attribute Definition
In order to modify attributes you must change those attributes in your __init__.py. The various properties you canchange are:
• base: Base Class Type
• meta_type: Component-level identifier
• label: The display label in the GUI
• index_type: index types for component Catalog search efficiency: (field, keyword)
• impacts: What this component impacts: can be list or list-output of a function
• impacted_by: What is component is impacted by: can be list or function
• order: Order of display in the grid
ZPL Modeling
ZPL Automatic set_ and get_ for Non-Containing Relations
You will automatically get a set_var() and get_var() method when you invoke them in the modeler. The set_ methodwill create or update the relationship. The set__* method takes a string or list of strings for its values.
You use it by first creating a non-containing relationship in the YUML like this:
Tenant 1:M FloatingipFloatingip 1:M Networks
Now you set the relationship up in the modeler by using set_tenant:
1 floatingips = []2 for floatingip in results['floatingips']:3
4 network_list = []5 for net in results['networks']:6 if net in some_net_list:7 network_list.append(net)8
9 floatingips.append(ObjectMap(10 modname = 'ZenPacks.zenoss.OpenStackInfrastructure.FloatingIp',11 data = dict(
(continues on next page)
7.1. Welcome to ZenPack Library 215
ZenPackers Documentation
(continued from previous page)
12 id = 'floatingip-{0}'.format(floatingip['id']),13 floatingipId = floatingip['id'],14 set_tenant = tenant_name[0],15 set_networks = network_list,16 )))17
18 tenants = []19 ... similar to floatingips above ...20 ... etc ...21
22 objmaps = {23 'tenants': tenants,24 'floatingips': floatingips,25 }26
27 # Apply the objmaps in the right order.28 componentsMap = RelationshipMap(relname = 'components')29 for i in ('tenants', 'floatingips'):30 for objmap in objmaps[i]:31 componentsMap.append(objmap)32
33 return (componentsMap)
Note: ApplyDataMap does not accept string or list objects by default. ZPL does this for us.
If you overide the set_ functions in your Component class you need to either make a call to ZPL’s su-per(ComponentClass, self).set_networks() or write your function as per ApplyDataMap()’s requirements.
ZPL Modeling Templates
Our modeling example is a very simplified version of the ControlCenter ZenPack. The modeler itself grabs a pre-madeObjectMap from the helper class in $ZP_DIR/modeling:
• $ZPDIR/modeler/plugins/zenoss/ControlCenter.py (wrapper for modeling.py)
• $ZPDIR/modeling (Does the heavy lifting)
In the modeler wrapper, ControlCenter.py we have:
1 import logging2 LOG = logging.getLogger('zen.ControlCenter')3
4 from twisted.internet.defer import inlineCallbacks, returnValue5 from Products.DataCollector.plugins.CollectorPlugin import PythonPlugin6 from ZenPacks.zenoss.ControlCenter import modeling, txcpz7
8 class ControlCenter(PythonPlugin):9
10 """ControlCenter modeler plugin."""11
12 required_properties = (13 'zControlCenterHost',14 'zControlCenterPort',15 'zControlCenterUser',16 'zControlCenterPassword',
(continues on next page)
216 Chapter 7. Topics for ZenpackLib
ZenPackers Documentation
(continued from previous page)
17 )18
19 deviceProperties = PythonPlugin.deviceProperties + required_properties20
21 @inlineCallbacks22 def collect(self, device, unused):23 """Asynchronously collect data from device. Return a deferred."""24 LOG.info("%s: Collecting data", device.id)25
26 # Loop through the required_properties and balk if missing.27 for required_property in self.required_properties:28 if not getattr(device, required_property, None):29 LOG.warn(30 "%s: %s not set. Modeling aborted",31 device.id,32 required_property)33
34 returnValue(None)35
36 client = txcpz.Client(37 device.zControlCenterHost,38 device.zControlCenterPort,39 device.zControlCenterUser,40 device.zControlCenterPassword)41
42 producer = modeling.DataMapProducer(client)43
44 try:45 results = yield producer.getmaps()46 except Exception as e:47 LOG.exception(48 "%s %s ControlCenter error: %s",49 device.id, self.name(), e)50
51 returnValue(None)52
53 returnValue(results)54
55 def process(self, device, results, unused):56 """Process results. Return iterable of datamaps or None."""57 if results is None:58 return None59
60 LOG.info("%s: Processing data", device.id)61 results = tuple(results)62 return results
In the helper class, $ZPDIR/modeling we have (abbreviated to Pools). Notice in line 26, the set_parentPool attributeis processed by ZPL as a ManyToOne relationship between Pools and sub-Pools.
1 #------------------------------------------------------------------------------2 # Zenpacks.zenoss.ControlCenter.modeling3 # ControlCenter Modeling: Modeling code for ControlCenter.4 #------------------------------------------------------------------------------5 from twisted.internet.defer import inlineCallbacks, returnValue6 from Products.DataCollector.plugins.DataMaps import RelationshipMap
(continues on next page)
7.1. Welcome to ZenPack Library 217
ZenPackers Documentation
(continued from previous page)
7 from .util import get_pool_id, get_host_id, get_service_id, get_running_id8
9 def map_pool(attributes):10 """Return ObjectMap data given attributes.11
12 Example attributes:13
14 {15 "Id": "Alternate",16 "ParentId": "default",17 "Priority": 0,18 "CoreLimit": 1,19 "MemoryLimit": 1,20 }21 """22 return {23 'id': get_pool_id(attributes['Id']),24 'title': attributes['Id'],25 'set_parentPool': get_pool_id(attributes['ParentId']),26 'priority': attributes['Priority'],27 'coreLimit': attributes['CoreLimit'],28 'memoryLimit': attributes['MemoryLimit'],29 }30
31
32 class DataMapProducer(object):33 """Produce the DataMap objects required to model """34
35 def __init__(self, client):36 self.client = client37
38 @inlineCallbacks39 def getmaps(self):40 """Return a datamaps map. """41 maps = []42
43 pools = yield self.client.pools()44 pool_maps = []45 for pool in pools:46 pool_map = map_pool(pool)47 if pool_map:48 pool_maps.append(pool_map)49
50 maps.append(51 RelationshipMap(52 relname='pools',53 modname='ZenPacks.zenoss.ControlCenter.Pool',54 objmaps=pool_maps))55
56 returnValue(maps)
ZPL Monitoring Templaees
The datapoints for this model are essentially the dictionary keys of the JSON data sources. That means the datapointsmust match the keys exactly.
218 Chapter 7. Topics for ZenpackLib
ZenPackers Documentation
• Create a Template: the name must matche the label in __init__.py: CP-Pool
• Add a DataSource: The name is arbitrary
• Add a DataPoint to that DataSource: The name must match an attribute (ZPL)
• Some example points:
– Priority
– CoreLimit
– MemoryLimit
ZPL Details Auto-Rendering
Thu May 29 16:01:30 CDT 2014
You can now use the same rendering in the details that are used elsewhere. In your __init__.py you set the rendererproperty in the class properties section:
1 classes={ ....2
3 'Flavor': {4 'base': 'LogicalComponent',5 'meta_type': 'OpenStackFlavor',6 'label': 'Flavor',7 'order': 1,8 'properties': {9 'flavorId': { 'grid_display': False }, # 1
10 'flavorRAM': { 'type_': 'int',11 'renderer': 'Zenoss.render.bytesString',12 'label': 'RAM' }, # bytes13 'flavorDisk': { 'type_': 'int',14 'renderer': 'Zenoss.render.bytesString',15 'label': 'Disk' } # bytes16 }17 },18 ... etc ...19 }
The ZPL will take care of setting this renderer wherever those variables are used.
Ref: https://github.com/zenoss/ZenPacks.zenoss.OpenStack
Dynamic Classes
There are several classes that are created on the fly when ZPL is instantiated. This includes:
• All the classes created from your YUML description
• schema: Classes schema created from the YUML spec. You’ll see this in your class files outside of __init__.py
from . import schema
7.1. Welcome to ZenPack Library 219
ZenPackers Documentation
Impact in ZPL
Impact adapters are provided for in the ZPL. In order to get them to work you must provide the impacts and im-pacted_by attributes in the class specification in __init.py__. The values of these attributes can be one of the follow-ing:
• A valid relationship name as defined in the YUML
• A valid function that returns a list of component ID’s.
Creating a ZenPack (and git repo) using ZPL
In order to do this all from the commandline, you need the prerequisites:
• You use github SSH authentication, and it authenticates without prompting
• You use the hub tool https://github.com/github/hub
We first start by creating the ZP data using ZPL’s tools:
zenpacklib --create ZenPacks.zenoss.SomeUniqueZPcd ZenPacks.zenoss.SomeUniqueZP
Next we create the repo using hub:
hub create zenoss/ZenPacks.zenoss.SomeUniqueZP
Now initialize the repo data and push up to origin:
git initgit commit -m "first commit"git remote add origin [email protected]:zenoss/ZenPacks.zenoss.SomeUniqueZP.gitgit push -u origin master
7.1.2 Relationship Management in the ZPL
Within the context of Zenoss, a relationship establishes a connection between Zope objects. A relationship MUSTbe bi-directional: a left side object and a right side object. However, if one side is containing, then the other side iscontained.
Currently ZenPacks supports three types of relationships:
• ToMany
• ToOne
• ToManyCont
Read zenosslabs section Background Information for an explanation of Relationship, and what is meant to be ‘con-taining’. Here is the webpage link: http://docs.zenosslabs.com/en/latest/zenpack_development/background.html#relationship-types
The Relationship classes are defined in /opt/zenoss/Products/ZenRelations/RelSchema.py. A relationship is expressedvia a yUML statement in ZenPack’s __init__.py.
• For a left side to be ToManyCont, there must be ‘++’ on the left.
• For a right side to be ToManyCont, there must be ‘++’ on the right.
220 Chapter 7. Topics for ZenpackLib
ZenPackers Documentation
• For a left side to be ToMany, there must be ‘*’ on the right.
• For a right side to be ToMany, there must be ‘*’ on the left.
• If there is no ‘++’ on the left AND no ‘*’ on the right, the left is ToOne.
• If there is no ‘++’ on the right AND no ‘*’ on the left, the right is ToOne.
• You can specify a number for a fixed quantify for a non-containing
• You can also specify a numerical range (0..2) on ends of non-containing relationships.
Example 1:
[Image]1-.-*[Instance]
Here Image and Instance are class names. This yUML statement says an Image object can have many Instance objects;whereas an Instance object can only have one Image object. An Image object has a ToMany relationship w.r.t Instanceobjects; whereas an Instance object a ToOne relationship w.r.t. Image object. The relationship between Image andInstance is non-containing.
‘-‘ and/or ‘.’ separates left side from right side.
Example 2:
[Endpoint]++components-endpoint1[OpenstackComponent]
• Endpoint and OpenstackComponent are again class names. components and endpoint here are relationshipnames. As pointed out in zenosslabs, relationships are themselves objects.
• This yUML statement says Endpoint can contain multiple OpenstackComponent(s); whereas an Openstack-Component can only belong to one Endpoint.
• Endpoint has a containing ToMany relationship w.r.t. OpenstackComponent; whereas OpenstackComponenthas a contained ToOne relationship w.r.t. Endpoint.
• The relationship between Endpoint and OpenstackComponent is containing.
zendmd can be used to find the relationship object components:
>>> dev=find('stack')>>> dev.components<ToManyContRelationship at /zport/dmd/Devices/OpenStack/devices/stack/components>
Example 3:
[Hypervisor]1-.-1[Host]
• Hypervisor and Host has a mutual ToOne non-containing relationship.
• Hypervisor and Host are class names.
• The relationship between Hypervisor and Host is non-containing.
For those who went through the SNMP ZenPack Development guide, http://docs.zenosslabs.com/en/latest/zenpack_development/index.html, just as a comparison, the relation expressed in NetBotzDevice.py:
7.1. Welcome to ZenPack Library 221
ZenPackers Documentation
_relations = Device._relations + (('temperature_sensors', ToManyCont(ToOne,
'ZenPacks.training.NetBotz.TemperatureSensor','sensor_device',)),
)
and in TemperatureSensor.py:
_relations = ManagedEntity._relations + (('sensor_device', ToOne(ToManyCont,
'ZenPacks.training.NetBotz.NetBotzDevice','temperature_sensors',)),
)
is equivalent to:
[NetBotzDevice]++temperature_sensors-.-sensor_device1[TemperatureSensor]
Example 4:
[EtherCard]0..1 switches -.-ecards 0..1 [Switch]
• The endpoint relationships are now named
• Named relationships can have their properties changes (GUI, order, etc)
• Each side can handle 0 or 1 connections
TODO: Examples of illegal yUML statements.
7.1.3 ZPL Errors
Certain errors are unique to ZPL.
Warning:
Never Never Never use mutable values as a default arguments! It causes a horrible side-effect of mutatingsubsequent call parameters! Use None instead.
• => mostly this means default values of {} and [] types
• String, booleans, int, float are all safe because they’re immutable
• The reason this happens is because default argument values are actually in the class scope, not in the methodscope. So doing it as above gets the empty list into method scope and unable to be shared by other instancesof the class.
In other words, once you mutate it in the function body, that value will be used in all subsequent calls to thatfunction. Thats probably now what you want.
• In terms of ZPL, this means that you NEVER set a default value of [] or {} in the YAML. {} is normallynot an option in the ZPL YAML spec, but still . . .
222 Chapter 7. Topics for ZenpackLib
ZenPackers Documentation
• Best practice for this case (default empty list) you make the default value in the method signature None, thenset the default value inside:
def genericMethod(a, x = None)if x is None: x = []... do something with x ...
Reference: https://twitter.com/raymondh/status/576047090905677824
ClassRelationshipSpec Error
If you get a ZPL schema error like this:
zenoss@zenoss> zenpack --listERROR:zen.ZenMessaging:Error encountered while processing ZenPacks.zenoss.→˓OpenStackInfrastructureTraceback (most recent call last):File "/opt/zenoss/Products/ZenMessaging/queuemessaging/schema.py", line 58, in _→˓getZenPackSchemas
pkg_path = zpkg.load().__path__[0]File "/opt/zenoss/lib/python/pkg_resources.py", line 1954, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])File "/zenpacks/ZenPacks.zenoss.OpenStackInfrastructure/ZenPacks/zenoss/→˓OpenStackInfrastructure/__init__.py", line 624, in <module>
class_relationships = zenpacklib.relationships_from_yuml(RELATIONSHIPS_YUML),File "/zenpacks/ZenPacks.zenoss.OpenStackInfrastructure/ZenPacks/zenoss/→˓OpenStackInfrastructure/zenpacklib.py", line 937, in __init__
if relationship.schema.remoteClass in self.imported_classes.keys():AttributeError: 'ClassRelationshipSpec' object has no attribute 'schema'ERROR:zen.ZenossStartup:Error encountered while processing ZenPacks.zenoss.→˓OpenStackInfrastructureTraceback (most recent call last):File "/opt/zenoss/Products/ZenossStartup/__init__.py", line 27, in <module>
pkg_path = zpkg.load().__path__[0]File "/opt/zenoss/lib/python/pkg_resources.py", line 1954, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])File "/zenpacks/ZenPacks.zenoss.OpenStackInfrastructure/ZenPacks/zenoss/→˓OpenStackInfrastructure/__init__.py", line 21, in <module>
from . import zenpacklibImportError: cannot import name zenpacklibERROR:zen.ZenPackCmd:zenpack command failedTraceback... etc ...
This error may indicate you changed your internal class relationship (YUML or JSON)without first removing existing devices from your system. You therefore havean inconsistency between ZODB and your current schema.
One way to fix this is to revert the changes:
• Revert your changes to the schema,
• Restart your Z services (make sure this works)
• Remove the device or zenpack
• Make your changes again to your __init__.py
• Install/re-install the zenpack
7.1. Welcome to ZenPack Library 223
ZenPackers Documentation
7.1.4 Converting from Old-Style ZenPackLib to the New-Style.
If you started out using the the original ZPL, you have your YUML specified in your $ZP/__init__.py, and a dictionarystyle of class specification. The new style moves almost all of this to a YAML file called $ZP/zenpack.yaml. Onlyspecial overridden classes get left behind in $ZP/__init__.py.
You might eventually want to convert to the new format. Fortunately there are tools and notes here to help you do so.
• Assumptions:
– You zenpack is named ZenPacks.zenoss.XYZ
– You zenpack uses an older form of ZPL that has YUML class structure and properties specified inside of$ZP/__init__.py
– You want to convert __init__.py’s YUML+dict to YAML format inside of $ZP/zenpack.yaml
• Create zenpack.yaml (based on monitoring_templates.yaml.. or an existing ZP)
– First create a dummy zenpack (or use the existing one), and get a new copy of zenpacklib.py installed.
– Execute:
python zenpacklib py_to_yaml ZenPacks.zenoss.XYZ
– basically nest the existing content under a device class)
• For Local Inherited Classes Only:
Ensure: In zenpack.yaml: locally inherited $ZP/modules.classes in separate files must (for now) use the fullyqualified path specification:
ZenPacks.zenoss.XYZ.module.class
The current py_to_yaml can interpret your local base class modules incorrectly as simply:
class
For example:
'Router': {'base': [SomeComponent],....}
can get incorrectly translated into YAML as:
Router:base:
[SomeComponent]
whereas you really need:
Router:base:
[ZenPacks.zenoss.OpenStackfrastructure.NeutronIntegrationComponent.→˓SomeComponent]
• In $ZP/__init__.py:
– Remove all unnecessary imports to local modules
224 Chapter 7. Topics for ZenpackLib
ZenPackers Documentation
– Add line to init.py to make it load it. In place of:
RELATIONSHIPS_YUML = """... some YUML class spec stuff ...
"""
CFG = zenpacklib.ZenPackSpec(... lots of old stuff ...... lots more old stuff ...... etc ...class_relationships=zenpacklib.relationships_from_yuml(RELATIONSHIPS_
→˓YUML),)CFG.create()
You will have something of this sort:
CFG = zenpacklib.load_yaml()
– restart zenoss,
Warning: Some of the next steps will eventually be automatic. You should be able to simply export yourobjects.xml after setting up your zenpack.yaml and the correct components will be included.
• Pruning Objects.xml
You need to remove all the superfluous junk that now resides in objects.xml. This is because zenpack.yaml takescare of most of it. Follow these guidelines for pruning:
– Navigate to Advanced -> ZenPacks -> YourZenPack
– Go to the the Zenpack’s “Provides” section to reduce objects.xml
– Remove all device classes listed in zenpack.yaml:device_classes
– Remove all the rrdTemplates items
– Ensure: Leave all Event related items in your Provides section for now.
– Hint: To get all contiguous Provides, click top item, then Shift-click bottom item.
– Export zenpack to regenerate objects.xml,
– Check that objects.xml is indeed smaller and has no rrdTemplate items
– Finally: Re-Export the ZP if other changes were made.
• Save the new Objects.xml
This only needs to be done if you development environment is not mounted/linked to your git repo. Otherwiseskip this step.
– Copy the new $ZP/objects/objects.xml from above to your ZP source tree
7.1. Welcome to ZenPack Library 225
ZenPackers Documentation
• Remove Old Files that Are No Longer Needed:
– remove no-longer-needed monitoring_templates.yaml
– remove load-templates files.
• You need to be careful that nothing breaks, but it should be really obvious breakage,
– Try zendmd between each change, watching for errors
– For example: all monitoring templates vanish or something.
– It should not be subtle if it’s not working
• You may wish to manually prune out at least one montioring template to convince yourself that ZPL is re-creating them at install time.
• Though it would also be clear when you tried to export.. Either they’d get pruned out or they would not. - (ifnot, ZPL doesn’t think it’s managing the monitoring templates!)
• Double Checking the Results
– Remove the existing ZP completely
– Restart all services
– Install your ZP
– Restart all services (again)
– Install a device on your ZP class
– Check that all is correct
References
https://github.com/zenoss/ZenPacks.zenoss.OpenStackInfrastructure https://github.com/zenoss/ZenPacks.zenoss.OpenvSwitch https://github.com/zenoss/ZenPacks.zenoss.ControlCenter
7.1.5 Component-Only Modeling: Add to Existing Device
If your zenpacklib created zenpack does not need a device class and will just be adding components to an existingdevice, you will need to add a relationship between the base device class and your new component. In your zen-pack.yaml:
classes:
ExchangeServer:base: [zenpacklib.Component]
class_relationships:
(continues on next page)
226 Chapter 7. Topics for ZenpackLib
ZenPackers Documentation
(continued from previous page)
- Products.ZenModel.Device.Device 1:MC ExchangeServer
... etc ...
As always, be sure to either install or reinstall the zenpack and restart your services when making this change. If not,you’ll see this in the info zenhub log:
If you see a warning in zenhub.log saying you have no relationship when applying the data map for your new compo-nent objects, you may just need to reinstall your zenpack.
If you’ve done things correctly, you’ll see this in the zenhub log next time you model:
2015-08-31 11:28:06,967 INFO zen.zenpacklib: AddingZenPacks.zenoss.Microsoft.Exchange relationships to existing devices
Now, in your modeler, you can set class level vars that apply to the current (the one that is being processed in theplugin) component:
• relname
• modname
This applies to self.ObjectMap() only just as in http://zenpacklib.zenoss.com/en/latest/tutorial-snmp-device/component-modeling.html .
Alternatively, you can set relname and modname on a per-component basis using local ObjectMap() as in the follow-ing:
class WinExchange(WinRMPlugin):
relname = 'exchangeServers'modname = 'ZenPacks.zenoss.Microsoft.Exchange.ExchangeServer'
@defer.inlineCallbacksdef collect(self, device, log):
.... etc ....
maps = {}
device_om = ObjectMap()maps['device'] = device_om
results = yield get_exchange_server
... etc ...
exchange_om = ObjectMap()try:
... etc ...exchange_om.title = exchange_om.id =self.prepId(results.stdout[0])exchange_om.role = results.stdout[1]exchange_om.version = results.stdout[2]exchange_om.relanme = 'exchangeServers' exchange_om.modname = 'ZenPacks.
→˓zenoss.Microsoft.Exchange.ExchangeServer'
(continues on next page)
7.1. Welcome to ZenPack Library 227
ZenPackers Documentation
(continued from previous page)
... etc ...except IndexError:
log.info("Invalid data returned from Exchange Server: }")
maps['device'] = device_ommaps['exchange_server'] = exchange_om
.... etc etc ....
defer.returnValue(maps)
References: https://github.com/zenoss/ZenPacks.zenoss.Microsoft.Windows
228 Chapter 7. Topics for ZenpackLib
CHAPTER 8
Topics for ControlCenter (ControlPlane)
Documentation for ZenPackLib
8.1 Welcome to ControlCenter!
Many of the documents for ControlPlane are in flux, so good luck if you find something useful that stays relevant formore than a few weeks.
8.1.1 General Concepts for CP
Links for Serviced
• https://github.com/zenoss/serviced/wiki
• https://github.com/zenoss/serviced/wiki/Starting-CP-Beta
General Links for Zendev
• http://zenoss.github.io/zendev
• http://zenoss.github.io/zendev/devimg.html
Development Workflow Cycle
Normally, once you deploy zendev, your workflow is very much the same as it always is:
• Create a feature against develop
• git flow feature publish (once only)
• Fix Fix Fix, Commit, Push
229
ZenPackers Documentation
• Pull Requests etc. . .
• Someone merges
• All rejoice and continue!
However, during release we have a special workflow in Zendev:
• zendev restore europa-release
• [ cd to right repo ]
• CURRENTBRANCH=$(git rev-parse –abbrev-ref HEAD)
• git flow feature start CC-1234 $CURRENTBRANCH
• git flow feature publish (once only)
• [. . . CODE FIX CODE FIX, COMMIT, push. . . ]
• Someone merges
• All rejoice and continue!
General ZenDev Tasks
ZOPE=$(serviced service list | grep Zope | awk '{print $2}')
# Open zendmdserviced service run -i $ZOPE zendmd
# Run zenupserviced service run $ZOPE zenup [args...]
# Install a zenpack in Zendev: Log into container, then do install.zendev devshellzenpack --link --install /mnt/src/zenpacks/ZenPacks.zenoss.ControlCenter
# Install a zenpackserviced service run $ZOPE zenpack --link --install ZenPacks.zenoss.ControlCenter
# Report daemon statsserviced service action $DAEMON stats
# Set daemon to debugserviced service action $DAEMON debug
# Attachserviced service attach $SERVICEID bash
# Shellserviced service shell -i $SERVICEID bash
Container Creation Specifics
# zendev devshell: defaults to a Zope importszendev devshell
(continues on next page)
230 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
(continued from previous page)
# Create a Zenhub imports environment this wayzendev devshell zenhub
8.1.2 Data Storage and Retreival: OpenTSDB and Friends
Data Storage is done through two primary services:
• CentralQuery
• OpenTSDB
Logical Data Flow for Europa GUI Process
When you view the GUI, the system processes in the general order.
[𝐺𝑈𝐼] ⇒ JS/Perf⇓
Central Query⇓
OpenTSDB:Reader ⇒ HBase
Logical Data Flow for Europa Modeler
This model is extremely simplified. Data flow for the Modeler looks similar to before.
[𝑍𝑒𝑛𝑃𝑎𝑐𝑘] ⇒ Modeler⇕
Central Query⇕
ZenHub ⇔ ZODB
Logical Data Flow for Europa Collection
Data flow for Collection looks like this:
[𝑍𝑒𝑛𝑃𝑎𝑐𝑘] ⇒ Collectors:Various⇓
CC- Central Query⇓
MetricShipper ⇐= Redis⇓
MetricConsumer ⇒ OpenTSDB:Writer⇓
Hbase
8.1. Welcome to ControlCenter! 231
ZenPackers Documentation
8.2 Setup and Configuration
8.2.1 Startup: Serviced A-la-Carte
This describes how to run and update serviced in a non-development environment. Later we show how to update thisenvironment in a similar way.
Requirements
• You must have setup zendev on an Ubuntu 14.04 system or better.
• Make sure your zenoss user is in the docker group, otherwise you need sudo for all docker commands..
Note: See https://github.com/zenoss/serviced/wiki/Starting-CP-Beta
Host Prep
INTERFACE=eth0IPADDR=$(ifconfig $INTERFACE | grep -o "inet addr:[\.0-9]*" | cut -f2 -d":")BOXNAME=$(uname -n)sudo su -c "echo $IPADDR $BOXNAME zenoss5x.$BOXNAME hbase.$BOXNAME >> /etc/hosts"
Docker Prep
Only do this once. Docker will store credentials in your account:
docker login -u zenossinc+alphaeval -e "[email protected]" \-p GETTHEMAGICKEYFROMTHESOURCELUKE https://quay.io/v1/
Set these local Variables as needed:
BUILD=521IMAGE=resmgrIMAGE=core
You need to grab the magic key from your docker manager. Only do once per revision or as needed:
# Pull the images... (Make sure to tag the zenoss image).docker pull quay.io/zenossinc/daily-zenoss5-${IMAGE}:5.0.0_${BUILD}docker tag quay.io/zenossinc/daily-zenoss5-${IMAGE}:5.0.0_${BUILD} zenoss/zenoss5xdocker pull quay.io/zenossinc/opentsdb:latestdocker pull quay.io/zenossinc/hbasedocker pull quay.io/zenossinc/isvcs:v9
Setup Zendev Services:
Execute the following:
232 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
zendev use europacdz serviced
Starting Serviced
You can do it one of 2 ways, I prefer the first, which requires you install serviced.init in your ~/bin. It also logsto /tmp/serviced.log
• serviced.init start
• cdz serviced ; serviced -master -agent (Don’t use this please)
Add a Host:
export IP_ADDRESS=$(ifconfig eth0 | grep 'inet addr:'| cut -d: -f2 | awk '{ print $1}→˓')serviced host add $IP_ADDRESS:4979 default
Compile Template:
serviced template compile $(zendev root)/src/service/services/Zenoss.${IMAGE} > /tmp/→˓Zenoss.xxx.tpl
( or pipe it into the following command )
Add the Template:
TEMPLATE_ID=$(serviced template add /tmp/Zenoss.xxx.tpl)
Note: Make sure you keep track of the TEMPLATE_ID number
Deploy the Template:
Method 1: CLI
Examples:
serviced template deploy $TEMPLATE_ID default zebraserviced template deploy $TEMPLATE_ID default zenoss
Method 2: GUI
• Go to the UI at: https://you.own.ip/ and Log in as zenoss/zenoss.
• Deploy template.
• Done.
8.2. Setup and Configuration 233
ZenPackers Documentation
8.2.2 Cleaning out all Docker Images
• cdz build; ./services/repos/docker_mrclean.sh
• Now go through manual install of all docker images
• Manually rebuild serviced
8.2.3 Upgrading/Cleaning the Docker Folder /var/lib/docker
If you need to upgrade or purfiy Docker you can do:
sudo stop dockerumount $(grep 'aufs' /proc/mounts | awk '{print$2}' | sort -r)rm -rf /var/lib/docker# => [Upgrade Docker if needed]# => Re-install your images or rebuild zendev
8.2.4 Cleaning out Serviced Templates
For now these live in /tmp/serviced-root . Here are the steps:
• stop serviced
• rm -rf /tmp/serviced-root
• Re-deploy your templates
8.2.5 Compiling Serviced in Go
Environmental Setting
These environmental settings are required:
zendev use europa
export GOPATH=$(zendev root)/src/golangexport PATH=$(zendev root)/src/golang/bin:${PATH}export ZENHOME=$(zendev root)/zenhome
Now clear out old data:
serviced.init stopsudo rm -rf /tmp/serviced-root
czd servicedgit pullmake
Update Compatiblity
Update Europa Build Environment (Before updating template )
Contains service templates. . . .
234 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
• cdz
• cd build
• git pull
• Now rebuild the template and re-deploy
8.2.6 Updating Serviced in Go
Execute these commands:
zendev use europa# I'd probably put these into .bashrcexport IPADDR=$(ifconfig eth0 | grep 'inet addr:'| cut -d: -f2 | awk '{ print $1}')export GOPATH=$(zendev root)/src/golangexport PATH=$(zendev root)/src/golang/bin:${PATH}export ZENHOME=$(zendev root)/zenhome
Clear out old Serviced data:
# stop servicedserviced.init stopsudo rm -rf /tmp/serviced-root
Clear out old Docker data (you may have to do this more than once):
$(zendev root)/build/services/repos/docker_mrclean.sh
Normal Update Serviced:
# Make sure docker is runningcdz servicedgit pullmake
Enhanced Update Serviced:
# go into the europa environmentcdz servicedgit status# ( This next line may be required if you can't pull properly )git checkout HEAD isvcs/resources/logstash/logstash.confgit pullmake
Update Europa Build Environment (Before updating template )
First you need the OpenTSDB and Hbase images to start. Remember, you just blew those away:
docker pull quay.io/zenossinc/opentsdb:v1docker pull quay.io/zenossinc/hbase:v1
You must update Europa too:
8.2. Setup and Configuration 235
ZenPackers Documentation
cdzcd buildgit pull# Now rebuild/update the templates and re-deploy
Update Templates Method I (Preferred): Tag Docker image to match template
First you need the template for the generic service (Do this only once):
serviced template compile $(zendev root)/build/services/Zenoss.${IMAGE} > /tmp/Zenoss.→˓xxx.tplserviced.init startTEMPLATE_ID=$(serviced template add /tmp/Zenoss.xxx.tpl)serviced host add $IPADDR:4979 default
This method pulls the docker image and tags it:
# Alternative to mapping the template: Tag the image:docker pull quay.io/zenossinc/zenoss-${IMAGE}-testing:5.0.0b1_${BUILD}docker tag quay.io/zenossinc/zenoss-${IMAGE}-testing:5.0.0b1_${BUILD} zenoss/zenoss5x
You don’t need to deploy the template since it already matches your docker image. If this is your first time to deploythough:
serviced template deploy $TEMPLATE_ID default zenoss
Update Templates Method II (Hard Way): Map the template to match Docker
Set these local Variables as needed:
BUILD=521IMAGE=resmgrIMAGE=core
Build the Template:
ZVER=daily-zenoss5-${IMAGE}:5.0.0_${BUILD}cdz servicedserviced template compile -map zenoss/zenoss5x,quay.io/zenossinc/$ZVER \$(zendev root)/build/services/Zenoss.${IMAGE} > /tmp/x.tpl
# serviced template compile $(zendev root)/build/services/Zenoss.${IMAGE} > /tmp/x.tpl
( you probably need to start serviced now )TEMPLATE_ID=$(serviced template add /tmp/x.tpl)serviced host add $IPADDR:4979 default
Deploy Templates:
serviced template deploy $TEMPLATE_ID default zenoss
236 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
Updating Platform-Build with Json-only Changes
Surgically, you could remove the template from the UI, then compile/add template and start all services. You couldalso serviced service edit each of those services and restart those services after editting, which is probably easiest sincethere are only a few minor changes The incantation to compile/add:
serviced template compile \-map zenoss/zenoss5x,zendev/devimg $(zendev root)/build/services/Zenoss.core \| serviced template add
8.2.7 CP Beta and Release Images
Installing Zenpacks in Beta Images
Typical commands in the Beta and Release containers are different from Zendev For example, there is no link-installin the following:
serviced service run -i zope zenpack [args]or
serviced service run -i zope zenpack install ZenPacks
For example:
serviced service run -i zope zenpack install ZenPacks.zenoss.DB2-XYZ.egg
Note: Don’t install from /root where perms are not readable by non-root. The Zenpack egg must be in a readablefolder like /tmp
Also, note there are NO double-dashes on the zenpack options.
8.2.8 Managing Containers
This section covers various Serviced container topics.
How to Customize and Commit a Container
To customize a container and make those changes permanent:
[zenoss@cc]: serviced service shell -i -s mychange zope bash
[root@ZOPE]: su - zenoss[zenoss@ZOPE]: [make your change change changes in the shell][zenoss@ZOPE]: exit[root@ZOPE]: exit
[zenoss@cc]: serviced snapshot commit mychange[zenoss@cc]: serviced service restart ...
For example: To add PyYAML to the zope containers:
8.2. Setup and Configuration 237
ZenPackers Documentation
[zenoss@cc]: serviced service shell -i -s mychange zope bash
[root@ZOPE]: su - zenoss[zenoss@ZOPE]: easy_install pyyaml[zenoss@ZOPE]: zendmd
In [1]: import yaml=> (if no errors, its installed correctly)
[zenoss@ZOPE]: exit[root@ZOPE]: exit
[zenoss@cc]: serviced snapshot commit mychange[zenoss@cc]: serviced service restart ...
To customize a remote container you need to provice the full path to the container instance:
serviced service shell -i -s solutions_xxx \golfballs/Zenoss.resmgr/Zenoss/Collection/localhost/GOLF1/zenpython \bash
To reconnect to your container after you exit, you need to start it:
docker start solutions_xxxdocker attach solutions_xxx
Installing Linked Zenpack that needs Maven, Java, etc. . .
ZenPacks that need maven offer a slight problem, because you need to install Maven and Java (OpenSDK) to get theminstalled in link-mode. There is also a minor package dependency problem in zenoss-centos-deps.
They can be installed in the following way in a container:
[zenoss@cc]:serviced service attach zenhub
[root@Zenhub]: yum remove zenoss-centos-deps[root@Zenhub]: yum install maven[root@Zenhub]: su - zenoss
[zenoss@Zenhub]: zenpack --link --install /z/ZenPacks.zenoss.XYZ... exit out of the containers ...[zenoss@cc]: serviced service restart zenoss.core
Installing Impact into RM with other Link-Mounted ZP’s
Overview:
1. Ensure you have the Zenoss Impact docker image installed (assume true)
2. Ensure ImpactServer and Impact zenpacks are in /z on host; (assume true)
3. Create container that has ImpactServer zenpack installed; commit container
4. Start ImpactServer
5. Create a container that has the Impact zenpack installed; commit container
6. Restart services
238 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
Start
• You should first ensure you have the following 2 zp’s in /z:
– ZenPacks.zenoss.ImpactServer-5.x.y.z-py2.7.egg
– ZenPacks.zenoss.Impact-5.x.y.z-py2.7.egg
• Create container that has ImpactServer
One issue here is that your /z folder may not be properly mounted. We work around this by manual mounting itin a container:
[you@host]: serviced service shell -i -s impacter --mount /z,/z zope bash[root@65b1f2340c77 /]# su - zenossLast login: Thu Jul 7 18:27:21 UTC 2016[zenoss@65b1f2340c77 ~]$ cd /z[zenoss@65b1f2340c77 /z]$ zenpack --install ZenPacks.zenoss.ImpactServer-5.x.y.z-→˓py2.7.egg[zenoss@65b1f2340c77 /z]$ exit[root@65b1f2340c77 /]# exit[you@host]: serviced snapshot commit impacter
• Start ImpactServer: serviced service start impact
• Create container that has Impact: (Similar as for ImpactServer); commit
• Restart core zenoss services
You should see Impact under Infrastructure and Impactstate under Events in UI
Installing ZenETL as a Linked ZP in a ZSD Installation
This requires some care because you have to ensure the zen*etl tools get installed into the Zope image. This normallydoes not happen because in a link install, the normal tools don’t get copied to /opt/zenoss/bin.
• First compile the ZP with mvn support externally. Copy the ZP in place.
• Edit GNUMakefile and comment out the mvn commands so the container doesn barf.
• Execute:
serviced service shell -i -s mychange --mount /z,/z zope su - zenoss
[zenoss@612b45db88f8]: cd /z/ZenPacks.zenoss.ZenETL/ZenPacks/zenoss/ZenETL[zenoss@612b45db88f8]: cp daemons/\*.py /opt/zenoss/bin/[zenoss@612b45db88f8]: cp cp zen\*etl.py /opt/zenoss/[zenoss@612b45db88f8]: exit
• Grab the container number from the shell’s prompt, say: 612b45db88f8
• cleanup:
serviced service commit 612b45db88f8docker rm mychange
8.2. Setup and Configuration 239
ZenPackers Documentation
8.2.9 Backups and Snapshots
Backups
Backups for ControlCenter are the most durable way to preserve your Z5 installation. It is larger and more timeconsuming than a Snapshot but it is more reliable in preserving all (Docker) images and configurations.
Backups are done as follows:
• First, decide what folder backups should be stored. Often: /opt/serviced/var/backups/
• Second, note that backups can be large. Make sure you have room.
• To create a backup from the 5.X system host:
serviced backup /opt/serviced/var/backups/
• Once finished, its advisable to copy that file to another safe system:
scp /opt/serviced/var/backups/file.tgz [email protected]:/tmp/
• To restore:
serviced restore /path/to/file.tgz
SnapShots
Snapshots normally temporary and have a limited lifetime in the system.
• To perform a snapshot:
serviced snapshot commit Zenoss.core
8.3 Zenpack Development
8.3.1 ZenPack Development with Zendev: Setup in ControlCenter
Welcome to the Control Center cetnral for Zenpackers! The following sections are hoped to bring clarity and peace toyou in your quest for ZenPack developement.
First let us establish the base directory for Zendev:
export ZENDEV_ROOT=$(zendev root)$ZENDEV_ROOT/src/service/services
This folder is typically in $HOME/src/europa, but it could be elsewhere.
Setting up the Development Environment
Setup for ZenPack development in ControlCenter requires a fully functional ZenDev environment as specified items1-10, but stop before starting serviced on the last line if item 10:
• https://github.com/zenoss/zendev/blob/zendev2/docs/installation.rst
240 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
Warning: Only Zendev2 is supported now. Everything here relates to Zendev2. When in doubt, refer to theZendev2 documentation above.
Once you have zendev fulling installed but before starting serviced, you need to do the following items:
• cd into your zendev home folder’s Zenpack folder:
cd $ZENDEV_ROOT/src/zenpacks/
• Now git clone all the ZenPacks you need for your project
• Edit the following file and add (or subtract) the Zenpack names from the ZENPACKS environment var ZEN-PACKS in $ZENDEV_ROOT/build/devimg/install_core.sh:
ZENPACKS="PythonCollector ControlCenter"
•Note: In order to have a valid Resmgr image, you’ll need a LOT more zenpacks built into into your image.This means that the above line MUST look more like:
ZENPACKS= “PythonCollector ControlCenter ZenJMX DynamicView AdvancedSearch EnterpriseCollectorEnterpriseSkin DistributedCollector ImpactServer Impact “
• Now rebuild the devimg:
zendev build devimg
• [Optional but recommended]: Pull docker images to avoid timeouts at deployment :
docker pull zenoss/serviced-isvcs:v16 docker pull zendev/devimg:latest docker pullzenoss/hbase:v2 docker pull zenoss/opentsdb:v3
• Now start serviced:
zendev serviced --reset --deploy
• Then start the application (yes, case-insensitive):
zendev serviced service start zenoss.core
• Watch the services with:
watch serviced service status
•Warning: Don’t use the GUI to start the application as that currently uses a lot of resources. This could eatup a lot of CPU and memory just to render a few graphs.
• Now Zenoss will run on the same IP but with a virtual name: ‘zenoss5x.*’: So if your host is xyz.zenoss.loc,your Zenoss will run on:
zenoss5x.xyz.zenoss.loc
You will need to either do one of two things to connect to Zenoss:
1. Add an entry into your /etc/hosts:
8.3. Zenpack Development 241
ZenPackers Documentation
192.168.1.45 xyz.zenoss.loc zenoss5x.xyz.zenoss.loc hbase.xyz.zenoss.loc
2. Add a CNAME entry in your DNS that points zenoss5x to xyz.zenoss.loc
• You should be able to connect now to: https://zenoss5.xyz.zenoss.loc If not, go back and check your networkingsetup.
• Once you can connect, you must connect to the container for the daemon you are debugging. For example, todebug modeling you connect to the modeler container:
zendev attach zenmodeler-or-
serviced service attach zenmodeler
Note: Since you can have multiple containers running a service you may want to reduce that to a single service.This is done in the ControlCenter GUI by changing the Instance value, saving, and restarting.
Warning: The normal user to attach to a container with is root! This will cause you many sleepless nightsand untold problems because zenoss commands as root will change file owernship. Instead use the “zenoss”user. To make this easy, you can use this bash function described in section: Attaching to Containers
• References
– http://zenoss.github.io/zendev/devimg.html
– http://zenoss.github.io/zendev/installation.html#ubuntu
Updating Zendev, Bare Bones Style
Updating Zendev is getting simpler. Eventually there will be a single button to push. Until that time try these directions:
• Stop docker:
sudo stop dockersudo umount $(grep 'aufs' /proc/mounts | awk '{print$2}' | sort -r)sudo rm -fr /var/lib/docker
• Log back in to host system:
sudo reboot (Host System)sudo start docker (if not started)zendev selfupdate; zendev sync
• Create devimg and pull in isvcs:
cdz serviced && make clean && makezendev build devimg --clean
• start serviced and pull other images:
zendev serviced -dx
To cut-n-paste:
242 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
sudo stop dockersudo umount $(grep 'aufs' /proc/mounts | awk '{print$2}' | sort -r)sudo rm -fr /var/lib/dockersudo reboot# Log in to hostsudo start dockerzendev selfupdate; zendev sync# Now time to build serviced and zendevcdz serviced && make clean && makezendev build devimg --cleanzendev serviced -dx
Installing Zenpacks for Development
In development we usually need to install the zenpacks in link-mode. To do this note that zenpacks in your zendev:$ZENDEV_ROOT/src/zenpackas/* will be located in the container at /mnt/src/zenpacks/* . So here is the process:
1. Attach to the Zope Container. If you have more than one, use the UUID:
serviced service attach Zope
2. cd /mnt/src/zenpacks
3. Make sure your zenpack is present
4. Execute the zenpack command:
zenpack --link --install ZenPacks.zenoss.XYZ
Sometimes you have no choice but to install using Egg. In that case you must be in the host system (zendev orotherwise):
serviced service run zope zenpack install ZenPacks.zenoss.OpenStack-XXX.egg
Serviced Essentials
Here are some Serviced topics are relevant.
Getting Listings
You’ll want to remove all non-ascii characters from a serviced command output. This is because serviced service listwill output some non-ascii “tree” characters that can make the awk error prone. Do it like this:
serviced service list | tr -cd '\11\12\40-\176'
Now use that output to capture any SERVICE_ID like this:
ID=$(serviced service list | grep zenmodeler | tr -cd '\11\12\40-\176' | awk '{print→˓$2}')
8.3. Zenpack Development 243
ZenPackers Documentation
Attaching to Containers
Serviced has a utility to attach to containers. By default the user you attach with is root, which is BAD if you intendto issue zenoss commands.
You can attach to a container as root by simply doing:
serviced service attach <NAME>
where <NAME> is one of the services (zendev, zeneventserver, Zope, etc..). But as mentioned above, doing anythingthat involves Zenoss will change the ownership of files in /opt/zenoss and potentially BREAK your install.
Instead, place this bash function in your .bashrc:
attach(){
local target=$1serviced service attach $target su - zenoss
}
then you can just do a:
attach zenhub
You can also just do it manually:
serviced service attach zenhub su - zenoss
Editing Serviced Service Definitions From CLI
If you are unwilling or unable to use the GUI to edit services, this will be an invaluable tool for 5X. The method issimple, find the ID, and use serviced to edit the serviced template.
• Find the ID for a service. In our example Zope:
ZOPE_SERVICE_ID=$(serviced service list | grep Zope | awk '{print $2}')
• To edit the Zope service definition:
serviced service edit zope# or the old fashioned way:serviced service edit $ZOPE_SERVICE_ID
• Once you have finished editing the service you can verify it by either looking at the GUI or re-editing the GUI.
• Restart the Service. There are two ways, the first way in the link serviced.init is preferred:
– Using the script:
serviced.init restart
– Manually:
* Kill serviced manually
* zendev serviced
244 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
Note: You must restart Zope to activate your changes.
Monitoring Logs in Zendev
Monitoring logs in Zendev is easier than one might think. That is because the entire Zenoss core folder is bind-mounted from the Zendev environment across ALL Zope containers. You don’t need to access ANY container to seethem.
The logs are located in: $ZENDEV_ROOT/zenhome/log/ . If Serviced and Zenoss are active you should see thesefiles being updated often.
Testing Modelers, Collectors, and Services
In the 4.X world we usually turn off the services and run them manually. This still can work in 5.X. First you muststop the container that has the service you want to test, then you run it manually from another container like Zope.Here are the steps:
• Identify the service you want to test, and grab the ID. We use zenmodeler for example:
• Turn off the zenmodeler container in the GUI or manually:
[zenoss@mp6:~]: serviced service stop zenmodeler
• Attach to another service like Zope and run zenmodeler manually:
[zenoss@mp6:~]: zendev attach ZopeYo, you can probably just use serviced attach
[root@zope /]# zenmodeler run -d xyz.zenoss.loc -v10
2014-07-05 00:56:58 DEBUG zen.ZenModeler: Run in foreground, starting→˓immediately.2014-07-05 00:56:58 DEBUG zen.ZenModeler: Starting PBDaemon initialization...etc......etc...
• When you are finished with your debug session just exit the container and restart your zenmodeler service (ifyou want it to run):
(zenoss)[root@zope /]# exit[zenoss@mp6:~]: serviced service stop 24x2cfz4b16ww8gakhgcgnv87
Cross Mounted Directories!
Experimentation shows that there are several shared directories in the containers. Your core and zenpacks will beshared from your Zendev development directories.
If you edit core code in one container it is changed in other containers that share this. This includes:
8.3. Zenpack Development 245
ZenPackers Documentation
Share Source Target Mount Point Mount Type$DEV:$ZENDEV_ROOT/src/core /mnt/src/core NFS (From Dev)$DEV:$ZENDEV_ROOT/zenhome /opt/zenoss NFS (From Dev)/mnt/src/core/Products /opt/zenoss/Products Local/opt/zenoss/otherwise /opt/zenoss/otherwise Local
Questions and Possible Answers
• I don’t have a DNS server. How do I deal with DNS issues?
The easiest way is to set your host values in /etc/hosts and run dnsmasq. This will ensure all containerswill see your hosts as thought a real DNS was setup and running.
If you also point your /etc/resolv.conf to an upstream DNS, you get the benefits of both hosts and DNSresolution.
Note: You can skip /etc/hosts modification. Assume, that hostname of your development server isfqdn.domain.tld, IP address is static value 192.168.0.1 and you want to resolve any subdomains likezenoss5.fqdn.domain.tld into development server’s IP address. So just add the following into /etc/NetworkManager/dnsmasq.d/zenoss.conf and restart NetworkManager service:
address=/fqdn.domain.tld/192.168.0.1
• What is the best way to debug the container processes? Candidates include:
– dgbp: http://docs.activestate.com/komodo/4.4/debugpython.html
– winpdb: http://winpdb.org/docs/embedded-debugging/
– pdb: put your pdb in the right place and run a service in forground
You may need to try several differnent methods.
• Can I Run Zenhub in the foreground?
According to the experts, Maybe. In fact, you can run zenhub in the foreground using a different shell. Howeverif you actually want other daemons to connect to your new zenhub, that won’t work because of TCP portmismatch.
One solution is to attach to the Zenhub container, kill and start Zenhub in the foreground in one step:
zendev attach zenhubpid=$(ps ax | grep -E "[[:digit:]]{2} su - zenoss -c" | awk '{print $1;}')kill $pid; zenhub run -v10 --workers 0
Zenhub must be in in full contact with all the other containers via TCP port connections. The fallback plan is ususe a remote debugger like winpdb or dbgp.
• How do run Zope in the foreground?
– Ensure that there is only one Zope instance running
– serviced service attach an existing Zope container,
– Kill zopectl in the background and immediately restart in the foreground similarly to Zenhub:
kill 30440 ; su - zenoss -c /opt/zenoss/bin/runzope
246 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
• You upgraded Go, but you can’t build anymore. You get errors like this:
../domain/metric.go:10: import$ZENDEV_ROOT/src/golang/pkg/linux_amd64/github.com/zenoss/glog.a:object is [linux amd64 go1.2.1 X:none] expected [linux amd64 go1.3 X:precisestack]
The problem is that you have older libraries from prior version of go. You need to clean out the older librariesand rebuild:
rm $GOPATH/pkg/* -Rfcdz servicedmake cleanmake
• Your entire Zendev environment seems broken, and builds fail. What to do?
You may have broken your zendev environment by upgrading or getting some environment vars wrong. Checkthose env vars and try this:
zendev restore develop:wq
• Unit Tests:
zendev devshell run tests
8.3.2 Debugging Control Center: Kung-Pow Style
Debugging ControlCenter ain’t easy. Ask anyone.
Instead of general theory, lets talk about examples.
Catching a PDB in Zope
First you need to have a single instance of Zope so that your PDB will guarantee to fire in the instance you monitor.You also need to run zope in the foreground and you need to know which one.
• Attach to your 5x host
• Edit the service definition for zope and set the InstanceLimits:Min parameter to 1:
[[email protected]]: serviced service edit zope(search for "Min", change that value from 2 to 1; if already 1, continue)
• In the 5x GUI, go into your Zope defintion, click “Edit Services”. Set Instances to 1. You should see the instancecount become a singleton.
• Now we attach to that unique instance of Zope:
[[email protected]]: serviced service attach zope[root@42b39f16c058 /]#: su - zenoss[zenoss@42b39f16c058 /]#:
• Now insert your pdb in the right place:
8.3. Zenpack Development 247
ZenPackers Documentation
[zenoss@42b39f16c058 /]#: ZP=$ZENHOME/ZenPacks/ZenPacks.zenoss.ZenJMX-3.11.0-py2.→˓7.egg/[zenoss@42b39f16c058 /]#: cd $ZP/ZenPacks/zenoss/ZenJMX/[zenoss@42b39f16c058 /]#: vi __init__.py[zenoss@42b39f16c058 /]#: (... insert your pdb.set_trace() ... )
• Now restart zope in place. First find out the PID. Its the commmand that looks like “su - zenoss -c/opt/zenoss/bin/runzope” You can use this sed expression to get it:
pid=$(ps aux | sed -n 's|^root *\(\w\+\)\? .*su - zenoss .*/runzope$|\1|p')echo $pid
• Restarting Zope will just revert your pdb. You need to kill that process and restart it just afterwards. Here is howyou do it:
kill $pid && su - zenoss -c /opt/zenoss/bin/runzope
• Easy way to restart Zope in foreground with pkill:
pkill -f zope.conf ; zopectl fg
• Now you need to trigger your bug by using the GUI. Go to the GUI and do this now.
• If you are lucky, your terminal will have caught your pdb at the correct place.
Running Various Daemons in Foreground
For each of these next commands, you must be attached to the <container> and su’d into the zenoss account there likethis:
serviced service attach <container> su - zenoss
• Zope:
pkill -f zope.conf ; zopectl fg# --or for debug mode--pkill -f zope.conf ; zopectl -X debug-mode=on fg
• Zenhub:
pkill -f zenhub.conf ; zenhub run --workers=0 -v10
• Zeneventd:
pkill -f zeneventd.conf ; zeneventd run --workers=0 -v10
• Zenpython, Zenmodeler, Zencommand
You would normally stop the container and run the daemon manually from a zope container.
8.3.3 Working with Templates
Templates form the core of the Serviced service definitions. They define nearly everything that goes into Zenoss. Inorder to use them you will have to start with some base set to make life easier.
In zendev these templates correspond to entire folders of templates. They live in:
248 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
export ZENDEV_ROOT=$(zendev root)$ZENDEV_ROOT/src/service/services
You may see these folders which hold many other subfolders:
drwxrwxr-x 19 zenoss zenoss 4096 Aug 7 20:40 Zenoss.coredrwxrwxr-x 17 zenoss zenoss 4096 Aug 7 20:40 Zenoss.coredrwxrwxr-x 21 zenoss zenoss 4096 Aug 7 20:40 Zenoss.resmgrdrwxrwxr-x 19 zenoss zenoss 4096 Aug 7 20:40 Zenoss.resmgr.lite
Basic Commands
• List a Template:
serviced template list
• Compile a Template:
serviced template compile Zenoss.coreserviced template compile Zenoss.core > /tmp/Zenoss.core.tpl
• Add a Template:
serviced template add /tmp/Zenoss.core.tpl
– However, its best to grab the Template ID in this way:
TEMPLATE_ID=$(serviced template add /tmp/Zenoss.core.tpl)
• Deploy a Template:
After you deploy a template, it becomes a real Application:
# ----------------------------------------------------------# serviced template deploy TEMPLATE_ID POOLID DEPLOY_ID# ----------------------------------------------------------serviced template deploy $TEMPLATE_ID default zenmaster
•Note: Note on Resmgr
In order to compile and deploy Resmgr your Docker image must be created correctly. This means that allthe required Zenpacks must be compiled into that Docker image. See the note in Setting up the DevelopmentEnvironment.
Compile the Generic Template
This method is most approprate when you work with a pre-made image like Beta or a Release image. This is becauseits a bit easier to tag an image with the same tag (as the template) than to change the numbers on the templates.
Compile the template:
8.3. Zenpack Development 249
ZenPackers Documentation
serviced template compile $ZENDEV_ROOT/src/services/services/Zenoss.core > /tmp/→˓Zenoss.core.tplserviced.init startTEMPLATE_ID=$(serviced template add /tmp/Zenoss.xxx.tpl)
serviced template deploy $TEMPLATE_ID default zenoss
Map the template to match your Docker Image
This may be more appropriate if you use Zendev, since the zendev images are already created with tags, and you don’twant to mess with those.
Build the Template:
cdz servicedserviced template compile -map zenoss/zenoss5x,zendev/devimg \$ZENDEV_ROOT/src/service/services/Zenoss.core > /tmp/xxx.tplTEMPLATE_ID=$(serviced template add /tmp/xxx.tpl)
Deploy Templates:
serviced template deploy $TEMPLATE_ID default zen_monkey
Updating Templates with Json-only Changes
Surgically, you could remove the template from the UI, then compile/add template and start all services. You couldalso serviced service edit each of those services and restart those services after editting, which is probably easiest ifthere are only a few minor changes. The incantation to compile/add (with template mapping):
serviced template compile -map zenoss/zenoss5x,zendev/devimg \$ZENDEV_ROOT/src/service/services/Zenoss.core \| serviced template add
UtilityScripts: But Wait Folks! Thats not all!
Yes, thats right folks, we’ve worked hard to make life easier for you. How easy you may ask? So easy, you can do itwith one hand tied behind your back and both eyes closed!
Here is a bash function that will fix up your template and insert it all in one command:
liten_up_dude(){
IMAGE=core
# Compile the Template and *MAP* it to the right zendev image:serviced template compile -map zenoss/zenoss5x,zendev/devimg \
$ZENDEV_ROOT/src/service/services/Zenoss.${IMAGE} > \/tmp/Zenoss.xxx.tpl
# Add the Template to serviced definitionsTEMPLATE_ID=$(serviced template add /tmp/Zenoss.xxx.tpl)
# Deploy the template
(continues on next page)
250 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
(continued from previous page)
# serviced template deploy TEMPLATE_ID POOL_ID DEPLOYMENT_ID# ----------------------------------------------------------
serviced template deploy $TEMPLATE_ID default zenmaster
# Get rid of the old Zenoss.core applicationCORE_ID=$(serviced service list | grep -E 'Zenoss.core\s' \
| tr -cd '\11\12\40-\176' | awk '{print $2}')
serviced service remove $CORE_IDunset CORE_ID
# Now you should use the GUI to start the Zenoss.core application# Warning! Untested: You can also add that to this function if you like::# LITE_ID=$(serviced service list | grep -E 'Zenoss.core' \# | tr -cd '\11\12\40-\176' | awk '{print $2}')# serviced service start $LITE_ID
}
Warning: Make sure you Don’t start or use the standard Zenoss.core application before starting the Zenoss.coreapplication. Experiments have shown that there is some docker image mismatches that happen as a result of startingZenoss.core, stopping it, and starting Zenoss.core.
So here is the workflow scenario for this tool:
• zendev build devimg
• zendev serviced -dx
• liten_up_dude
• Go into GUI, select Zenoss.core, Start it
Modifying Service Templates in Europa
Modifying templates may be required to add functionality to your zenpacks or to core. Often, the containers have avery restricted access in order for good security and simplicity.
Modfying the Default Templates
If the service you are modifying is in the default template, you will have to ensure that ALL the template definitionshave that change, otherwise someone will deploy a service that is missing a requirement. Make sure that the templateis modified in folder. The Current Zendev has these template folders:
8.3. Zenpack Development 251
ZenPackers Documentation
Template Location
Zenoss.core $ZENDEV_ROOT/src/service/services/Zenoss.core/Zenoss.core.full $ZENDEV_ROOT/src/service/services/Zenoss.core.full/Zenoss.resmgr $ZENDEV_ROOT/src/service/services/Zenoss.resmgr/Zenoss.resmgr.lite $ZENDEV_ROOT/src/service/services/Zenoss.resmgr.lite/Zenoss.ucspm $ZENDEV_ROOT/src/service/services/Zenoss.ucspm/Zenoss.ucspm.lite $ZENDEV_ROOT/src/service/services/Zenoss.ucspm.lite/
Each of these will be modified by adding the following to the Endpoint list:
{"Name": "rabbitmq","Application": "rabbitmq","PortNumber": 5672,"Protocol": "tcp","Purpose": "import"
}
Once the change is made you can compile any of these templates and deploy just as we have outline above.
Adding RabbitMQ Ports to the Zenpython (PythonCollector) Zenpack
If you are adding to a service that is bundle with a Zenpack, you must provide extra templating instructions to Zenossso that any installation of that Zenpack will guarantee to have the required services you need.
In our example, ZenPacks.zenoss.OpenStackInfrastructure requires that zenpython be able to poll the RabbitMQ con-tainer. This was not available when we started. We start by reviewing https://github.com/zenoss/ZenPacks.zenoss.ExampleService which outlines the way to add template support to a zenpack. We’ll try to only outline the criticalcomponents that supplement that document.
• First create folders in $ZP_DIR of ZenPacks.zenoss.PythonCollector:
su - zenosscd ZenPacks.zenoss.PythonCollector/ZenPacks/zenoss/PythonCollectormkdir service_definitionmkdir -p service_definition/-CONFIGS-/opt/zenoss/etc
• Copy the service configuration file into the right place:
cp /tmp/zenpython.conf service_definition/-CONFIGS-/opt/zenoss/etc/
• Now copy the template into place:
cp Zenoss.core/localhost/localhost/zenpython/service.json service_definition/
• You now must change a few key items in this template (be sure to add quotes_):
– servicePath: /hub/collector
– serviceDefinition: (Encapsulate the entire contents of original services.json)
– serviceDefinition (Make it somewhat different from standard)
– ConfigFiles:
252 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
"/opt/zenoss/etc/zenpython.conf": {"FileName": "/opt/zenoss/etc/zenpython.conf","Owner": "zenoss:zenoss","Permissions": "0664"}
– Endpoints (Add):
{"Name": "rabbitmq","Application": "rabbitmq","PortNumber": 5672,"Protocol": "tcp","Purpose": "import"
}
– ImageID: (set to empty string to be overridden later)
• Once all this is in place, you need to test it by removing and reinstalling the ZP. Then you can check the servicedefinition in Zendev:
serviced service edit zenpython
If you see your changes, and a nice ImageID, then all is hopefully well. See the actual Zen-Packs.zenoss.PythonCollector repo for more detail.
8.4 Special Topics: ControlCenter and Others
Many of the documents for ControlPlane are in flux, so good luck if you find something useful that stays relevant formore than a few weeks.
8.4.1 Installing Impact in Zendev
Note: This document is outdated. See this link for something more modern: https://github.com/zenoss/impact-server/wiki/Running-Impact-Server-in-development-mode-(5.x)
• Pull and tag latest impact image (Currently 121):
# Note: Need to use an user that has docker privilegesexport IMPACT_IMAGE_TAG=4.2.6.70.0_130 # Warning: this tag changes!docker pull zenoss/impact-unstable:$IMPACT_IMAGE_TAGdocker tag zenoss/impact-unstable:$IMPACT_IMAGE_TAG zenoss/impact-unstable:latest
• Deploy zenoss.core or resmgr: # Use whatever workflow you wish to add and deploy core/resmgr service
• Start services: It is best to start all services. If you prefer, this subset is the minimal that necessary for zenpackinstall:
for svc in mysql rabbitmq /redis zencatalogservice zeneventserver ; \do serviced service start $svc ; done
• Install Impact ZenPacks:
8.4. Special Topics: ControlCenter and Others 253
ZenPackers Documentation
Note:
– You must install them in link mode
– You should not use the devshell environment (its broken): Instead use zope container: zendevattach zope Optionally (from zendev): zendev attach zope su - zenoss -c “zenpack –list”
– *CRITICAL*: Make sure that ImpactServer and Impact zenpacks are on the develop branch.Zendev may put them on master by default.
– Enterprise zenpacks are in /mnt/src/enterprise-zenpacks/, aka: $EZ:
export EZ=/mnt/src/enterprise_zenpacks
– Normal zenpacks are in /mnt/src/zenpacks/, aka: $ZP
export ZP=/mnt/src/zenpacks
The install now:
zenpack --link --install $ZP/ZenPacks.zenoss.ZenJMX
zenpack --link --install $EZ/ZenPacks.zenoss.DynamicViewzenpack --link --install $EZ/ZenPacks.zenoss.AdvancedSearchzenpack --link --install $EZ/ZenPacks.zenoss.EnterpriseCollectorzenpack --link --install $EZ/ZenPacks.zenoss.EnterpriseSkinzenpack --link --install $EZ/ZenPacks.zenoss.DistributedCollectorzenpack --link --install $EZ/ZenPacks.zenoss.ImpactServer
• Before you install the Impact zenpack, you MUST have ImpactServer running:
serviced service start Impact
• Now you can finally install the Impact zenpack:
zenpack --link --install $EZ/ZenPacks.zenoss.Impact
• Optionally install any zenpacks you are testing under Impact
• Restart zenoss:
– serviced service stop zenoss.core
– serviced service start zenoss.core
8.4.2 Docker Images
We use a lot of docker images in Europa.
Finding Docker Images
To find Images for Impact on docker:
• https://registry.hub.docker.com/u/zenoss/impact-unstable/tags/manage/
• Use curl:
254 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
curl -s -u <<YOUR DOCKER HUB USERNAME>>https://registry.hub.docker.com//v1/repositories/zenoss/impact-unstable/tags \| python -m json.tool \| sed -n 's/.*"name": "\([^"]*\)"/\1/p' \| tail -1
Downgrading Docker in ControlCenter
If you find yourself needing to down grade docker:
1. Stop all services:
serviced service stop zenoss.core
2. Stop/Kill/Maim serviced:
service serviced stop (if running Release Images)killall serviced && killall -9 serviced (Otherwise)
3. Stop Docker:
service docker stop
4. Remove docker:
sudo apt-get purge lxc-docker\*sudo apt-get purge docker
5. Install the new (or older) docker:
sudo apt-get install lxc-docker-1.3.3
6. Restart Docker and Service:
service docker startservice serviced start (If running Release Images)zendev serviced >& /tmp/serviced.log & (Otherwise)
7. Start your Zenoss services if required
8.4.3 Getting Data Out of OpenTSDB
• Attach to one of your OpenTSDB instances:
serviced service attach reader
• Idenitfy one of your metrics:
/opt/opentsdb/build/tsdb uid --config /opt/zenoss/etc/opentsdb/opentsdb.conf grep→˓metrics ''
or from API:
lynx 'http://mp7:4242/api/suggest?type=metrics&max=1000000'
8.4. Special Topics: ControlCenter and Others 255
ZenPackers Documentation
• Metrics will be of the form:
device_name/metric_name
ex:
solutions-xenserver/rrd_memoryFree
• Query one of your metrics:
/opt/opentsdb/build/tsdb query --config /opt/zenoss/etc/opentsdb/opentsdb.conf 2h-→˓ago sum "solutions-xenserver/rrd_memoryFree"
8.4.4 Convenience Tools for ControlCenter
Mass Editing Serviced Templates in Place
If you had access to the source templates, you could just edit those in place or in some automated way. Often though,you find a stand-alone system without source templates and you may not have that choice.
You can still use the EDITOR env variable to effect an automated edit using sed. Here is an example script:
#!/bin/bash# See: https://jira.zenoss.com/browse/CC-888 for automation# This one will set every template in $collectors to manually start
collectors="zenprocesszenmailtxzenstatuszenucseventszenvspherezenpythonzenmodelerzenpop3zentrapzenwebtxzencommandzensyslogzenjmxzenpingzenmailzenperfsnmpzenpropertymonitor"
cat << EOF > /tmp/rename#!/bin/bashsed -i -e 's/^ "Launch": "auto",/ "Launch": "manual",/' "\$1"EOF
chmod 700 /tmp/rename
for c in $collectors; doEDITOR=/tmp/rename serviced service edit $c
done
256 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
8.4.5 Control Center service integration
Preface
Please note that this is not a step by step tutorial on how to create your Control Center(CC) service from scratch.
This topic will fill the missing gaps in CC documentation and shed some light on important things such as graphsdefinition.
Image
Each custom service in CC should have it’s own container.
In order to avoid problems, such as important env variables missing, you should use zenoss/centos-base as a baseimage in your Dockerfile.
When you’re done building your container you have to tag & push it into Docker Hub or any other Docker Registry,Makefile should take care of that.
Service Definition fields
To instruct Serviced that your ZenPack has a custom service defined you should have service_definition folder withservice.json file inside.
Read more at => Defining services.
• ImageID
This field is important, here you specify repo/image:tag that you’ve built before, it will be used by Serviced tocreate a container for your service.
There are couple of things to be aware of:
– if you’re using your private Docker Hub repository, make sure you’re logged in with your hub accountand then manually pull your image before ZenPack installation to avoid errors
– beware that you’ll probably face ControlCenterError: registry image collisionwhen updating your image at development stage and sadly there is no fix for this yet
• Command
This is what Serviced will treat as a service itself and run when you start your CC service. You can think of itas an ENTRYPOINT for your container.
Good to know:
– make sure you use absolute path for your bin/command as WORKDIR instruction from your Dockerfilewill be overridden by Serviced
• Endpoints
If you ever need to make your service available to other services or vice versa - you do that in ‘Endpoints’section.
Beware:
8.4. Special Topics: ControlCenter and Others 257
ZenPackers Documentation
– if your application resides in collector level you must have a unique ‘Application’ field for each servicecopy.
"Endpoints": [{
"Name": "zendotnet","Application": "{{(parent .).Name}}_zendotnet","PortNumber": 5001,"Protocol": "tcp","Purpose": "export"
}]
Metrics & Graphs
This is very important part of your service. Everyone wants to know some specific to service data and nobody wantsto look through tons of JSON so you need graphs as well.
• Technical background
Control Center has it’s own instance of OTSDB, the graphs you see at CC UI are retrieved from that instance.Note that this is not the OTSDB under the Infrastructure part.
The graphs that are defined in service.json address individual metrics stored in CC’s OTSDB, more specificallydatapoints reference the metrics.
To get your data into CC’s OTSDB you should send a POST request to CC’s metricconsumer whose URL isstored in CONTROLPLANE_CONSUMER_URL env variable.
• MetricConfigs
In here you will define your metric sources which you have to reference later on in GraphConfigs.
Important things are ID field and Metrics list. Each metric in Metrics list will have it’s own ID as well andthese must start with ID of a MetricConfig it’s sitting in followed by a dot.
Here’s an example:
"MetricConfigs": [{
"ID": "dotnet","Name": "dotnet internal metrics","Description": "dotnet internal metrics","Metrics": [
{"ID": "dotnet.total_requests","Name": "Total Requests","Description": "Number of total requests made by dotnet service"
},{
"ID": "dotnet.succeeded_requests","Name": "Suceeded Requests","Description": "Number of succeded requests by dotnet service"
}
(continues on next page)
258 Chapter 8. Topics for ControlCenter (ControlPlane)
ZenPackers Documentation
(continued from previous page)
]}
]
• GraphConfigs
In simple terms - here you define what kind and how many graphs you want and map data from metric sourceto them.
Each GraphConfig have one very important field called datapoints, that’s where you do the mapping.
In metricSource field you have to put ID of MetricConfig.
Inner IDs that are sitting in Metrics list will go in metric field.
Make sure to always set the format of data explicitly to avoid unknown values on graphs. CC uses **GO fmtpackage** for formatting.
Here’s an example:
"GraphConfigs": [{
"id": "requestsCount","name": "Requests Count","footer": false,"returnset": "EXACT","type": "line","format": "%%d","yAxisLabel": "requests","description": "Number of requests","range": {
"start": "1h-ago","end": "0s-ago"
},"datapoints": [
{"aggregator": "avg","fill": true,"id": "total_requests","legend": "Total","metric": "dotnet.total_requests","metricSource": "dotnet","name": "Total","rate": false,"rateOptions": {
"counter": true,"counterMax": null,"resetThreshold": 1048576
},"type": "area"
},{
"aggregator": "avg","fill": true,"id": "succeeded_requests","legend": "Succeeded",
(continues on next page)
8.4. Special Topics: ControlCenter and Others 259
ZenPackers Documentation
(continued from previous page)
"metric": "dotnet.succeeded_requests","metricSource": "dotnet","name": "Suceeded","rate": false,"rateOptions": {
"counter": true,"counterMax": null,"resetThreshold": 1048576
},"type": "area"
}]
}]
• POST request
As was previously mentioned, for you to get the data into OTSDB you have to submit a POST request.
Typically you should write a service that will POST your data periodically.
Here’s an example of that requests’ body:
"metrics": [{
"metric": "dotnet.total_requests","value": "10","timestamp": "0000000000","tags": {
"service": "dotnet"}
},{
"metric": "dotnet.succeeded_requests","value": "8","timestamp": "0000000000","tags": {
"service": "dotnet"}
}]
Afterword
An easy way to achieve success with this thing is to do by analogy first.
Great repos to take a look at would be: * zenoss-service * ZenPacks.zenoss.ExampleService * Zen-Packs.zenoss.Microsoft.Core
260 Chapter 8. Topics for ControlCenter (ControlPlane)
CHAPTER 9
Topics for OpenStack Group
Documentation for OpenStack Group of Zenpacks
9.1 Welcome to the OpenStack Group!
We have many ZenPacks that relate or integrate with OpenStack. Some of them are:
• ZenPacks.zenoss.OpenStack
• ZenPacks.zenoss.OpenStackInfrastructure
• ZenPacks.zenoss.OpenvSwitch
• ZenPacks.zenoss.Ceph
• ZenPacks.zenoss.CiscoAPIC
• ZenPacks.zenoss.NSX
We also have projects that enable OpenStack:
• ceilometer_zenoss
9.2 Setup and Configuration
9.2.1 Ceilometer Troubleshooting for OpenStack
Ceilometer heartbeats take the following path through the systems:
OpenStack-Ceilometer|V
Zenoss5.zenpython -> Zenoss5.rabbitmq (exchange E1)
(continues on next page)
261
ZenPackers Documentation
(continued from previous page)
|V
Zenoss5.rabbitmq (queue Q1)|V
OSI.service.OpenStackService.OpenStackService.remote_expected_→˓ceilometer_heartbeats
|V
Zenoss5.MetricConsumer
1. Make sure the time for both OpenStack hosts and Zenoss host is correct. A running ntpd helps.
2. Follow Link text , in the Ceilometer Enablement section.
3. Restart ceilometer services:
• On a centos 7 OpenStack host:
$ ceil=$(systemctl -t service | grep ceil | awk '{print $1}' | awk -F. '→˓{print $1}' )$ echo $ceil$ for i in $ceil; do service $i restart; done
• Check ceilometer collector log on the OpenStack host: /var/log/ceilometer/collector.log - make sure ev-erything is OK. - You should see something like:
INFO ceilometer_zenoss.dispatcher.zenoss [-] Sending heartbeat to zenoss.→˓openstack.heartbeat.<zenoss device name>.<openstack hostname>.ceilometer-→˓collector
4. Make sure the rabbitmq port 5672 is open on both (?) openstack host and zenoss host:
$ telnet <ip> 5672
5. ngrep is a useful tool for debugging. Download ngrep. Use it as follows:
$ export zenoss=<zenoss host ip address>$ export port=<rabbitmq port>$ ngrep -d $device -l -W none -qi "heartbeat" "dst host $zenoss and dst port $port→˓"
For example to find heartbeats:
export device=$(ip route get 8.8.8.8 | head -1 | cut -d' ' -f5)export zenoss="v8.veggie.com"export zenoss="10.11.12.123"ngrep -d $device -l -W none -qi "heartbeat" "dst host $zenoss and dst port 5672"ngrep -d $device -l -W none -qi "heartbeat"
To see events and not heartbeats:
ngrep -l -W none -qiv "heartbeat" "dst host $zenoss and dst port 5672"
262 Chapter 9. Topics for OpenStack Group
CHAPTER 10
Pythonic Topics:
Some Useful Python Topics
10.1 Regular Expression
10.1.1 Breaking Regular Expressions into Readable Parts
You can write your regular expression so that you can easily read them:
cs_RX = re.compile("^" # BOL"(jdbc:db2://)?" # cs_prefix (jdbc:db2://)"([\w.-]+)\:(\d+)" # hostnme:port"/([\w_\$\d]+)\:" # /DB_name:"user=([\w.-_@]+);" # username=joe;"password=([\w.-_@]+)" # password=rabbits;";" # Final semicolon"$" # EOL)
10.1.2 Matching Regular Expressions by Key
To match a Regular Expression by named groups, you use the following syntax:
Regex pattern... (?P<varname><regex>)
An example of this is:
import re
CS_REGEX = re.compile('^' # BOL
(continues on next page)
263
ZenPackers Documentation
(continued from previous page)
'(?P<username>\w+)' # Username'\/' # slash'(?P<password>\w+)' # Password'@' # at-symbol'(?P<host>[\w.-]+)' # Host':' # colon'(?P<port>\d+)' # Port'[:/]' # color or slash'(?P<instance>[\.\w_\$\d]+)' # Instance'$' # EOL)
Another Example:
def get_engine(eng=None, extra=None):if eng == none:
pattern = "(?p<year>[0-9]{4})(?p<month>[0-9]{2})(?p<day>[0-9]{2})_"pattern += "(?p<hour>[0-9]{2})(?p<minute>[0-9]{2})(?p<second>[0-9]{2})_"pattern += "(?p<tablename>[a-za-z_0-9.]{1,})[.](?p<kind>lock|partial|tmp)"if extra:
pattern += extraeng = re.compile(pattern)
return eng
def get_match(self, filepath):eng = get_engine()match = {}filename = os.path.basename(filepath)if eng.match(filename):
data = [m.groupdict() for m in eng.finditer(filename)][0]for name, value in data.iteritems():
try:match[name] = int(value)
except:match[name] = value
return match
Basically, it looks for a filepath with the following pattern:
<year><month><day>_<hour><minute><second>_<name>.<kind>
Examples might be: 20140122_103244_something.tmp, ‘data’ inside of ‘get_match’ holds the key:value dictionary
10.2 Understanding Twisted
• http://krondo.com/?page_id=1327
264 Chapter 10. Pythonic Topics:
ZenPackers Documentation
10.3 Tab Completion in Python
Python supports tab-completion in several areas. This allows you to hit the tab key and get possible attributes andfunctions listed on the screen.
10.3.1 PDB Completion
Two simple lines in a ~/.pdbrc file are enough to give you tab completion in your pdb session:
import rlcompleterpdb.Pdb.complete = rlcompleter.Completer(locals()).complete
10.3.2 Python Interpreter Completion
Add a ~/.pythonstartup.py file which contains:
# --------------------------------------------------------------------# ~/.pythonstartup.py# --------------------------------------------------------------------try:
import readlineimport rlcompleterimport atexitimport os
except ImportError:print "Python shell enhancement modules not available."
else:histfile = os.path.join(os.environ["HOME"], ".pythonhistory")import rlcompleterreadline.parse_and_bind("tab: complete")if os.path.isfile(histfile):
readline.read_history_file(histfile)atexit.register(readline.write_history_file, histfile)del os, histfile, readline, rlcompleter, atexitprint "Python shell history and tab completion are enabled."
10.4 PDB Tricks
10.4.1 Running loops in a PDB session
• You could do this while in pdb to launch a temporary interactive Python session with all the local variablesavailable:
(pdb) !import code; code.interact(local=vars())>>> for k in ctxt: print k
10.3. Tab Completion in Python 265
ZenPackers Documentation
266 Chapter 10. Pythonic Topics:
CHAPTER 11
Cisco General:
11.1 Cisco Background Information
11.1.1 CiscoStatus Threshold
The CiscoStatus threshold is a special threshold that uses preconfigured maps of numeric values returned by SNMPdatasources to Zenoss event severities. The following OIDs and values are supported.
CISCO-ENTITY-FRU-CONTROL-MIB::cefcModuleOperStatus
1. unknown - Critical
2. OK - Clear
3. disabled - Clear
4. OK (diag failed) - Warning
5. boot - Warning
6. self-test - Warning
7. failed - Critical
8. missing - Critical
9. mismatch w/parent - Critical
10. mismatch w/config - Critical
11. diag-failed - Critical
12. dormant - Critical
13. out of service (admin) - Info
14. out of service (environ) - Critical
15. powered down - Critical
267
ZenPackers Documentation
16. powered up - Critical
17. power denied - Critical
18. power cycle - Warning
19. OK (power warning) - Warning
20. OK (power critical) - Error
21. sync in progress - Clear
22. upgrading - Critical
23. OK (auth failed) - Critical
ENTITY-STATE-MIB::entStateOperDisabled ENTITY-STATE-MIB::entStateOperEnabled ENTITY-STATE-MIB::entStateOper
1. unknown - Error
2. disabled - Clear
3. enabled - Clear
4. testing - Critical
CISCO-ENTITY-FRU-CONTROL-MIB::cefcFRUPowerOperStatus
1. off (other) - Critical
2. on - Clear
3. off (admin) - Info
4. off (denied) - Critical
5. off (environmental) - Critical
6. off (temperature) - Critical
7. off (fan) - Critical
8. failed - Critical
9. on (fan failed) - Error
10. off (cooling) - Critical
11. off (connector rating) - Critical
12. on (no inline power) - Error
CISCO-ENTITY-FRU-CONTROL-MIB::cefcModuleAdminStatus
1. enabled - Clear
2. disabled - Clear
3. reset - Info
4. out of service (admin) - Clear
IF-MIB::ifOperStatus IPV6-MIB::ipv6IfOperStatus
1. up - Clear
2. down - Critical
3. testing - Critical
268 Chapter 11. Cisco General:
ZenPackers Documentation
4. unknown - Critical
5. dormant - Critical
6. not present - Critical
7. lower layer down - Critical
IF-MIB::ifAdminStatus IPV6-MIB::ipv6IfAdminStatus
1. up - Clear
2. down - Critical
3. testing - Critical
CISCO-ENTITY-FRU-CONTROL-MIB::cefcFanTrayOperStatus
1. unknown - Error
2. up - Clear
3. down - Critical
4. warning - Warning
MPLS-L3VPN-STD-MIB::mplsL3VpnVrfOperStatus
1. up - Clear
2. down - Critical
CISCO-ENTITY-FRU-CONTROL-MIB::cefcPowerRedundancyOperMode
1. not supported - Clear
2. redundant - Clear
3. combined - Clear
4. non-redundant - Clear
5. output redundant - Clear
6. input redundant - Clear
7. single input redundant - Clear
CISCO-ENTITY-FRU-CONTROL-MIB::cefcPowerNonRedundantReason
1. not applicable - Clear
2. unknown - Critical
3. single supply - Critical
4. mismatched supplies - Critical
5. supply error - Critical
CISCO-ENTITY-SENSOR-MIB::entSensorStatus
1. OK - Clear
2. unavailable - Error
3. non-operational - Critical
CISCO-VIRTUAL-NIC-MIB::cvnVethIfAdditionalState
1. none - Critical
11.1. Cisco Background Information 269
ZenPackers Documentation
2. participating - Clear
3. suspended - Clear
4. error-disabled - Critical
5. non-participating - Clear
CISCO-VTP-MIB::vtpVlanState
1. operational - Clear
2. suspended - Critical
3. mtuTooBigForDevice - Critical
4. mtuTooBigForTrunk - Critical
CISCO-VRF-MIB::cvVrfOperStatus
1. up - Clear
2. down - Critical
CISCO-VSAN-MIB::vsanOperState
1. up - Clear
2. down - Critical
11.1.2 Cisco UCS Glossary
Your guide to Cisco UCS terminology.
This document is an attempt to be a comprehensive references to any terms and abbreviations that may come up whendiscussing Cisco UCS.
Pictures
A picture is worth 1000 words? In case you wanted to know what this Cisco UCS hardware actually looks like. All ofthese pictures have been taken from the UCS Manager GUI .
6248UP Fabric Interconnect
Front view. The highlighted left-most 32 ports (banks of 16) are built-in. The shaded right-most 16 portswould be added by a fabric expansion module.
A production UCS domain has two of these for redundancy. Like a Microsoft Cluster server, there areIP addresses for each Fabric Interconnect (aka FI) and one for the current virtual master. We talk to themaster.
Fabric interconnects carry both LAN and SAN traffic.
Any physical port on a FI can be defined by software as a LAN uplink (out of the UCS Domain toa corporate data network), a SAN uplink (out of the UCS domain to a storage device or network), anappliance port (to a NetApp FAS), a server port (which can connect either to a fabric extender, a chassisfabric extender, or an actual server), or unused. And port channels exist, of course.
270 Chapter 11. Cisco General:
ZenPackers Documentation
6248UP Fabric Interconnect 16-Port Expansion Module
Front view. The highlighted right-most 16 ports are added to the fabric interconnect by this expansionmodule.
Not managed separately from the Fabric Interconnect.
6248UP Fabric Interconnect Fan Module
Rear view. The highlighted section is one of two possible fabric interconnect fan modules.
6248UP Fabric Interconnect PSU
Rear view. The highlighted section is one of two possible fabric interconnect PSUs.
2232PP Fabric Extender
Front view. Left-most 32 ports are server ports to be connected to C-Series rack servers. Right-most 8ports are fabric ports to be connected to fabric interconnects.
Fabric Extenders can be used within a UCS Domain or can be connected to a Nexus 5000 switch. Ifyou think of an old-style router with a bunch of blades that slot into a backplane to provide ports, thena Fabric extender is one of those blades, except that its backplane is some Ethernet cables and it can besome distance away.
A Fabric Extender is managed from the Fabric Interconnect or Nexus 5000 it is plugged into - it’s not astandalone device.
One key thing to know is that both SAN and LAN traffic can be carried on the same set of wires betweenthe Fabric Extender and the Fabric Interconnnect.
11.1. Cisco Background Information 271
ZenPackers Documentation
2232PP Fabric Extender Fan Module
Rear view. Highlighted section is the single possible fabric extender fan module.
2232PP Fabric Extender PSU
Rear view. Highlighted section is the single possible fabric extender PSU.
C200 M2 Rack Server
Front view. 1RU rack server.
C210 M2 Rack Server
Front view. 2RU rack server.
5108 Chassis
Front view. Fully-populated with 4 half-width blades, 2 full-width blades and 4 PSUs.
272 Chapter 11. Cisco General:
ZenPackers Documentation
5108 Chassis PSU
Front view. Highlighted section is one of four possible PSUs.
5108 Chassis Fan Module
Rear view. Highlighted section is one of 8 possible fan modules.
11.1. Cisco Background Information 273
ZenPackers Documentation
5108 Chassis Fabric Extender aka IO Module
Rear view. Highlighted section is one of two fabric extenders.
Functionally, these are extremely similar to the 2232 Fabric Extender. But they’re a different shape andthe ports to the servers are internal to the chassis. Which eliminates a whole bunch of Ethernet cables inthe back of the rack, lowering cost, eliminating cabling mistakes, and improving airflow.
One key thing to know is that both SAN and LAN traffic is carried on the same set of wires between theFabric Extender and the Fabric Interconnnect.
274 Chapter 11. Cisco General:
ZenPackers Documentation
B200 M1 Blade Server
Front view. Highlighted section is B200 M1 half-width blade server.
B200 M4 Blade Server
Schematic front view
1. Asset pull tag. Each server has a plastic tag that pulls out of the front panel. The tag contains the server serialnumber as well as the product ID (PID) and version ID (VID). The tag also allows you to add your own assettracking label withou t interfering with the intended air flow.
2. Blade ejector handle
3. Ejector captive screw
4. Drive bay 1
5. Drive bay 2
6. Power button and LED
7. Network link status LED
8. Blade health LED
11.1. Cisco Background Information 275
ZenPackers Documentation
9. Console connector
10. Reset button access
11. Beaconing LED and button
.
B250 M1 Blade Server
Front view. Highlighted section is B250 M1 full-width blade server.
HyperFlex HX220c M4 All Flash Node
Front and back views.
• Node size: 1 rack unit
• Node capacity:
276 Chapter 11. Cisco General:
ZenPackers Documentation
– 1 x 400 GB SAS SSD
– 6 x 3.8 TB / 960 GB SDDs
• CPU: 2 x Intel Xeon processor E5 2600 v3 or v4 CPUs
• Memory and cache:
– 128 GB to 1.5 TB of memory
– 400 GB 2.5-Inch Enterprise Performance 12 Gbps SAS SSD
• SSD
– 6 x 960 GB 6 Gbps SATA SSD or
– 6 x 3.8 TB 6 Gbps SATA SSD
• Software
– VMware 5.5 or 6.0 u1
– Cisco HyperFlex HX Data Platform Release 2.0.1
• Cluster
– Minimum of 3 All Flash Nodes
– Scale compute-only nodes with additional Cisco UCS B200, C220, and C240 server nodes
• Management : Cisco UCS Manager and VMware vCenter plug-in
Background Information
Useful information to understanding Cisco UCS that doesn’t necessarily fit into a glossary.
Cisco Product Versioning
Cisco has a specific way of versioning their hardware product lines that can make it easy to understand a lot about theproduct just by knowing its name and version. For example, a product named “Cisco UCS 6248UP 48-Port FabricInterconnect” is usually just referred to as a “6248” because in-the-know people know that..
• Models beginning with 6 are Nexus switches, which means they run the NX-OS operating system.
• Models beginning with 61 or 62 (so far) are specialized Nexus switches that are always used as UCS fabricinterconnects.
• The third and fourth digits (48) commonly refer to the number of ports in a switch model.
• The UP stands for “unified ports”. Meaning that the ports can carry LAN and SAN traffic.
Given this, you could probably guess what a 6296 is.
Typically Cisco uses the hundreds place in the model number to refer to the product line. This leads to peoplecommonly referring to a product line instead of a specific product in conversation because architecturally the specificproduct doesn’t really matter. So if someone says, “It’s a 6200,” they mean that it’s a current generation fabricinterconnect.
More examples can be found in the hardware components section below.
11.1. Cisco Background Information 277
ZenPackers Documentation
UCS Manager XML API
This is the web-services API through which UCS Manager can be managed. It runs from the fabric interconnect.
Cisco Documentation Link:
http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/api/pref.html
It doesn’t matter that UCS also publishes data through syslog and SNMP, we only need XML for all of our needs. It’sactually a bit confusing if you try to process the other data sources.
B-Series vs. C-Series
There are two types of servers supported by UCS. B-Series and C-Series. In B-Series the B stands for blade. InC-Series the C stands for chassis. The fact is that within a single UCS domain you can have B- and C-Series serversintermixed, or exclusively one or the other.
The C-Series rack-mount servers are just servers and also have the option of being deployed without a connection toUCS Manager . This makes them as difficult to manage as any other standalone server. Specifically it means they arenot manageable through the UCS Manager XML API, and therefore are not supported by Zenoss without the UCS CSeries ZenPack currently scheduled for 1H 2014. Standalone C-Series servers won’t be supported by UCS Explorerin 2014.
Architecture
Architectural concepts that are distinct from hardware and logical components.
Domain
A Cisco UCS domain refers to the entirety of hardware and software that is managed by a single UCS Manager soft-ware instance. It’s two (almost always) or one Fabric Interconnect, and some number of chassis with B-series serversand/or C-Series servers and a rack to put them in. While I suppose you could call a standalone Fabric Interconnect aUCS Domain that really wouldn’t be very interesting.
For most customers, the maximum number of servers maintained in a domain is between 100 and 150. The maximumis currently 160 with 20 chassis and 8 blades each.
Multi-Tier Network Architecture
Most datacenter networks have a tiered architecture. The tiers are usually referred to as layers. How many layersvaries, the following three layers are common.
Wikipedia Link: http://en.wikipedia.org/wiki/Hierarchical_internetworking_model
Core
The core layer aggregates the distribution layer. There is typically only one core per datacenter. The core is normallyexpected to be fully redundant and have enough bandwidth to interconnect all distribution.
278 Chapter 11. Cisco General:
ZenPackers Documentation
Distribution Layer
The distribution layer aggregates the access layer. There are typically multiple distribution points per datacenter thatfeed into a single core. The distribution layer is normally expected to be fully redundant and have enough bandwidthto provide transit between connected access switches and transit between the connected access switches and the core.
For many customers, a UCS system is a distribution node.
Access
The access layer connects downstream directly to servers and provides transit to the distribution layer. Historically theaccess layer wasn’t redundant because that requires servers have multiple NICs connected to different switches andperform multipathing. In today’s network even the access layer is commonly redundant.
In a UCS Domain, access is redundant. And software defined.
Software
Software that runs within or in association with Cisco UCS.
UCS Manager
UCS Manager is server software responsible all management of a single domain . The server software can be run ona single fabric interconnect, but is typically run on a redundant pair of fabric interconnects .
Zenoss considers each UCS Manager to be a device. All of its managed hardware and software components arecomponents of that device.
Abbreviations: UCSM, UCS-M
Cisco Product Link: http://www.cisco.com/en/US/products/ps10281/
UCS Manager GUI
The UCS Manager GUI is a Java application that is often referred to as UCS Manager. This should be avoidedas it creates ambiguity about the true meaning of UCS Manager. The UCS Manager GUI provides an easy to useadministration interface by which UCS Manager can be configured.
You can’t download or install the UCS Manager GUI indepedently. You must go browse the UCS Manager’s webinterface and click to launch the Java application. This forces you to use the right UCS Manager GUI version for theUCS Manager.
UCS Central
Cisco UCS Central integrates with many UCS Manager instances to provide a single point of control for many UCSdomains . UCS Central does not provide monitoring facility, instead focusing on administrative tasks.
Abbreviations: UCS-C
Cisco Product Link: http://www.cisco.com/en/US/products/ps12502/
11.1. Cisco Background Information 279
ZenPackers Documentation
UCS Director
Cisco UCS Director is a software suite previously known as Cloupia Unified Infrastructure Controller which wasacquired by Cisco on December 10, 2012. Its purpose is primarily to act as an orchestration system for UCS, but it isnot a required component of the system. It has limited monitoring capabilities but is primarily focused on provisioningorchestration.
Cisco Product Link: http://www.cisco.com/en/US/products/ps13050/
Hardware Components
Hardware components as differentiated from logical or software components. Components that have physical weightand dimensions.
Fabric Interconnect
A fabric interconnect is a specialized Nexus 6100 Series or 6200 Series switch that acts both as the network distributionlayer, and as the UCS Manager server.
Abbreviations: FI
6200 Series
The 6200 series are the current generation UCS fabric interconnects. The product line includes the 6148UP and6196UP which are 48 and 96 ports respectively.
Cisco Product Link: http://www.cisco.com/en/US/products/ps11544/
6100 Series
The 6100 series were the first generation UCS fabric interconnects, and are now officially “end-of-sale.” The productline includes only the 6120XP and 6140XP which are 20 and 40 ports respectively.
Cisco Product Link: http://www.cisco.com/en/US/products/ps10276/
Fabric Extender
There are two major types of fabric extenders to consider in UCS. The first are used to connect blade (B-Series) serversto fabric interconnects. These are often referred to as IO Modules, and they are installed as a pair in each chassis. Thesecond are used to connect rack-mount servers to the fabric interconnects. These are rack-mount switches, and visuallyindistinguishable from any other rack-mount switch.
Abbreviations: FEX
2200 Series
All current generation UCS fabric extenders are in the 2200 series. It can be hard to figure out from Cisco’s productliterature which models are installed into a UCS chassis and which are rack-mount switches. The rack-mount switchesare also referred to as top-of-rack (TOR) fabric extenders.
In-Chassis: 2204XP, 2208XP. 4 and 8 10GB ports respectively.
280 Chapter 11. Cisco General:
ZenPackers Documentation
Rack-Mount: 2232PP, 2248PQ, and many more.
In-Chassis Product Link:
http://www.cisco.com/en/US/prod/collateral/ps10265/ps10276/data_sheet_c78-675243.html
Rack-Mount Product Link:
http://www.cisco.com/en/US/products/ps10110/
2100 Series
These are all end-of-life.
Cisco Product Link: http://www.cisco.com/en/US/products/ps10278/
Chassis
A server blade chassis. Not to be confused with a C-Series server. For the most part the chassis is a hunk of metal withslots for more useful things to plug into. There has only ever been one chassis model released by Cisco.
5100 Series
For now, the 5108 is Cisco’s only server chassis. It can be populated with up to 4 full-width, or 8 half-width serverblades. It can contain up to 2 fabric extenders, up to 4 PSUs and up to 8 fans.
Product Link: http://www.cisco.com/en/US/products/ps10279/
Servers
There are two types of servers supported by UCS. B-Series and C-Series. In B-Series the B stands for blade. InC-Series the C stands for chassis. The fact is that within a single UCS domain you can have B- and C-Series serversintermixed, or exclusively one or the other.
B-Series Blade Servers
The B-Series blades are designed only to be installed into a UCS Chassis . They come in half-width and full-widthconfigurations. Some are optimized for CPU, some for storage.
Product Link: http://www.cisco.com/en/US/products/ps10280/
C-Series Rack Servers
The C-Series rack-mount servers are just servers and also have the option of being deployed without a connection toUCS Manager . This makes them as difficult to manage as any other standalone server. Specifically it means they arenot manageable through the UCS Manager XML API, and therefore are not supported by Zenoss.
There is more variance with the configuration of rack servers than blade servers because they can potentially be larger.There are 1RU, 2RU and 4RU servers available.
Product Link: http://www.cisco.com/en/US/products/ps10493/
11.1. Cisco Background Information 281
ZenPackers Documentation
Converged Network Adapter
The Cisco Unified Computing System supports converged network adapters (CNAs) from a broad set of technologyleaders such as Emulex, Qlogic, Broadcom, Intel, and Cisco. CNAs obviate the need for multiple network interfacecards (NICs) and host bus adapters (HBAs) by converging LAN and SAN traffic in a single interface.
CNAs seem to be also interchangeably referred to as ‘host adapters’ or simply “adapters.” CNAs are available forB-Series blade servers and C-Series rack servers.
Example models: VIC1280, M72KR-E, M71KR-E, M72KR-Q, M71KR-Q, 82598KR, etc.
Abbreviations: CNA
Product Link:
http://www.cisco.com/en/US/products/ps10277/prod_models_comparison.html
Power Supply
There are many power supply units in a UCS system. Nearly all of them are FRUs .
The following hardware components have PSUs.
• Fabric Interconnect
• Chassis
• Fabric Extender (rack-mount only)
• C-Series Rack Server
Abbreviations: PSU
Fans
There are many fans in a UCS system. Nearly all of them are FRUs .
The following hardware components have fans.
• Fabric Interconnect
• Chassis
• Fabric Extender (rack-mount only)
• C-Series Rack Server
Software Components
UCS Software components. These are mostly logical conceptualizations provided by the underlying hardware com-ponents.
Port Channel
A logical bundling of multiple physical network interfaces. Also commonly known as aggregate or aggregation inter-faces, or bundles.
282 Chapter 11. Cisco General:
ZenPackers Documentation
Unified Port
A port that handles LAN and SAN traffic. All fabric interconnect and fabric extender ports meet this criteria.
VM-FEX
Short for “Virtual Machine Fabric Extender.” You’ll always hear this called VM-FEX because it’s just too long to sayotherwise. These extend the physical switch port through a hypervisor to a specific virtual interface on a VM. Thisskips the overhead of the hypervisor’s software-based networking stack to provide excellent network performance toVMs.
All current-generation CNAs provide this capability.
Cisco Solution Link: http://www.cisco.com/en/US/netsol/ns1124/
Abbreviations and Other Terms
Space for other terms that don’t fit elsewhere. There are more in the document under their non-abbrievated concept.
CNA
Converged network adapter. A network adapter that provides LAN (NIC) and SAN (HBA) devices to thehost operating system while being connected to the network by only Ethernet.
FEX
Fabric extender. A dumb switch that is connected via Ethernet to a parent switch primarily to add portsto it. If you’re familiar with a line card, a fabric extender is the same thing except that instead of beingplugged into a switch-chassis’ backplane, it’s connected via Ethernet.
Fibre Channel
Fibre Channel is a high-speed network technology (commonly running at 2-, 4-, 8- and 16-gigabit persecond rates) primarily used to connect computer data storage .
Fibre Channel over Ethernet (FCoE)
FCoE is the technology Cisco uses to transmit Fibre Channel traffic over the same wire as regular ethernettraffic transmitted to the Fiber Interconnect.
FRU
Field replaceable unit. You can replace these items individually. Many times without powering off theassociated equipment.
11.1. Cisco Background Information 283
ZenPackers Documentation
UCS
Unified Computing System.
UCSPE
UCS Platform Emulator. A VMware VM that emulates UCS Manager.
VM
Virtual machine.
284 Chapter 11. Cisco General:
CHAPTER 12
Other Topics:
12.1 Ansible Rough Guide
12.1.1 Description
Ansible is a DevOps tool. Its actions are called Plays and groups of those actions are called Playbooks. Ansible isdesigned to be easy to install, implement and use.
12.1.2 Prerequisites
• Linux
• Python 2.7
• Ansible
285
ZenPackers Documentation
12.1.3 Setup
Once you setup your server with the ansible software (its CLI driven), you simply make sure you can SSH into yourtarget hosts. You can Setup SSH passphrases if you like, but it works regardless. You can also run on non-standard(non-22) tcp port numbers.
12.1.4 Inventory
You define your managed hosts with Inventory Files.
# ------------------------------------------------------------------# file: production
[monitors]mp1.zenoss.locmp2.zenoss.loc ansible_ssh_port=2201 ansible_ssh_host=90.14.17.12
[locals]localhost ansible_connection=local
12.1.5 Group Vars and Host Vars
Group and Host vars are simple YAML files. From your ansible root, they are located in:
$ANSIBLE/group_vars$ANSIBLE/host_vars
An example of group and host vars:
[joe@zen:~/ansible]: cat group_vars/all
# ================= Ansible Group Vars: ALL ========================---# Variables listed here are applicable to all host groups
http_port: 80https_port: 443ntpserver: us.pool.ntp.orggit_repo: http://github.com/xxx/mywebapp.gitsudo_user: joeopt_bin: /opt/binopt_log: /opt/logmaven_url: http://www.eng.lsu.edu/3.0/binaries/apache-maven-3.0.5-bin.tar.gz
and some host vars. . . .
[joe@zen:~/ansible]: cat host_vars/mp1.zenoss.loc
backup_tools:- backup_files- backup_home- disk_backup_dirvish- find_duplicates- report_df- reset_cronlog
(continues on next page)
286 Chapter 12. Other Topics:
ZenPackers Documentation
(continued from previous page)
- restart_apcmon- sudo_dirvish- rsync-new
dirvish_master: False
hourly_jobs:- check_raid- check_space
daily_jobs: []
12.1.6 Playbooks
Playbooks contain the commands that configure the targets. A playbook is YAML file that has some commands in it.It can be simply a bunch of commands in a single file or group of files.
Each section is YAML and is indented by 2 at each level. A simple example is:
# A very simple example for a CentOS box---- name: Ensure needed packages are the latest version
yum: pkg={{item}} state=latestwith_items:- lxc- make- icewm- openbox
12.1.7 Roles: Getting Organized
Roles allow you to organize your tasks. A Roles folder lives in the Ansible root and has a series of folders thatcorrespond to task groups:
ansible||-- group_vars| \-- all|-- host_vars| |-- 192.168.12.7| \-- mp2.zenoss.loc|-- laptops.yml|-- production|-- roles| |-- common| |-- network| |-- security| \-- zenoss\-- zenoss.yml
Inside of each role are the following directories which ansible will automatically search for needed files:
[joe@zenpad:~/ansible]: tree -L 1 roles/security
12.1. Ansible Rough Guide 287
ZenPackers Documentation
ansible/roles/security/|-- files|-- handlers|-- tasks|-- templates\-- vars
Here is the lowdown on what goes in these folders:
• files: Just plain old files for copy
• handlers: plays that get triggered by a notify event
• tasks: the big enchilada play
• templates: files that get templated
• vars: any vars local to the role
So full blown security folder looks like this:
[joe@zenpad:~/ansible]: tree -L 5 roles/network
roles/network/|-- files|-- handlers| |-- main.yml|-- tasks| |-- centos.yml| |-- debian.yml| |-- main.yml| |-- ubuntu.yml|-- templates| |-- ifcfg-eth0| |-- ifcfg-eth2| |-- ifcfg-static| |-- network| \-- sysconfig| \-- network-scripts\- vars
Thats enough theory. Lets do some demonstrations!
Links:
• Ansible Intro: http://docs.ansible.com/intro.html
• Ansible Modules: http://www.ansibleworks.com/docs/modules.html
• Best Practices: http://www.ansibleworks.com/docs/playbooks_best_practices.html
• http://jpmens.net/2012/08/30/ansible-variables-variables-and-more-variables/
• Download the associated video ansible.avi
12.2 Git for Gits ;)
Git is a development version control tool.
288 Chapter 12. Other Topics:
ZenPackers Documentation
12.2.1 References
• https://git-scm.com/book/en/v2
• https://danielkummer.github.io/git-flow-cheatsheet/
12.2.2 Git Access For Zenossians
If you yourself need a change in user permissions, or are asking on someone else’s behalf, here’s how we handle itgoing forward:
1. Send an e-mail to [email protected] with a summary of your request in the subject. Examples:
(a) Need to grant pull and push access to joe_user to xyz_repo
(b) Need to remove all access from former_employee
(c) Need to enable write access for joe_user to all Zenoss repos
2. The email body needs to include the following:
(a) The user’s full name
(b) The user’s email address
(c) The user’s github name
(d) The user’s department or role (why are they being added to the org in the first place?)
(e) The user’s Slack name
(f) The user’s Docker Hub username, if available
(g) Whether the user is an employee, or a contractor
12.2.3 Special note on the Owners group in Github
Owners have access to the billing information for the organization, and can create and delete teams. Owners can alsodelete any repo, which is one of the main risks we want to contain by better managing GitHub permissions.
12.2.4 Setup Your GitHub Credentials
Important: Make sure the one of the first things you do when you setup your development system is to setup yourgithub credentials and identifying username and email. This is to assist in tracking and documenting developmenthistory:
git config --global user.name "Jane Smith"git config --global user.email "[email protected]"
12.2.5 Clone a Repo
To just pull (download) a repository from the web:
• Find the repo online and get the clone string.
• Copy that string.
12.2. Git for Gits ;) 289
ZenPackers Documentation
• pull it down with “git clone”
[bash]: git clone [email protected]:zenoss/ZenPackers.git
- or for https -
[bash]: git clone https://github.com/zenoss/ZenPackers.git
12.2.6 Recommended Commit Message Format (Important!)
We regard the git log as an important tool to track down and summarize problems and solutions in our code. Thereforeits critical that the commit messages be as descriptive as possible. Remember that the summary line is the one thatcarries a lot of weight and is only 50 characters or less.
Note: Here are the rules for commit message
1. First line is Capitalized description, under 50 chars, no period
2. Second line blank
3. Third line: Fixes ZPS-13123, ZPS-34345, and ZPS-99099
4. Fourth Line blank
5. Next lines all less than 72 charactors, elaborated description. It can be a bit more open.
Here is an ideal commit message:
Correct the instance-table relations in the modeler
Fixes ZPS-123455.
Here is a more description of the fix (but only if needed). Its wrappedto 72 lines so that it doesn't get ugly.
If your commit is bigger and you need more bullets or description you can still do it. For example:
Correct the hypervisor-ip relations in the zenpack.yaml
Fixes ZPS-123455, ZPS-92548.
We had to fix the relationships so that the original model was notmodified. We still limit to 72 lines to avoid ugliness.I also added another tire to the system so that it would roll better.These are other things I have fixed:
* Fixed model to keep old model intact
* Added migration script to ensure model was correct at upgrade
* Added a big fat tire for better rolling
* Updated docs
Reasons for this Format
Why do we prefer this format? Partly because it allows a quick history view with a command like:
290 Chapter 12. Other Topics:
ZenPackers Documentation
git log --decorate --graph --oneline --date-order
If the first line is just ‘Fixes ZPS-XYZ123’ you get things that are not helpful at all like:
|/ /
* | c568cfe Merge pull request #93 from zenoss/feature/ZPS-22568|\ \| * | a367040 (origin/feature/ZPS-22568) fix ZPS-22568|/ /
* | 160f150 Merge pull request #92 from zenoss/feature/ZPS-22599|\ \| * | f2d37aa (origin/feature/ZPS-22599) fix ZPS-22599
. . . instead of somthing useful like this:
|/|
* | 8f38040 Merge pull request #106 from zenoss/feature/ZPS-22673|\ \| * | 58f9857 (origin/feature/ZPS-22673) add default values to properties; fix ZPS-→˓22673|/ /
* | f18a2ac (feature/ZPS-22598) Merge pull request #105 from zenoss/feature/ZPS-→˓22639|\ \| * | 01d4868 (origin/feature/ZPS-22639, feature/ZPS-22639) Ensure modeling on→˓Centos5, Suse 12
. . . there are other reasons too. See references.
References:
• http://chris.beams.io/posts/git-commit/
• http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
12.2.7 Cool Ways to Show Logs and Diffs
Often you will need to see your logs and compare different versions:
git loggit log --oneline --graph --decorate --allgit diffgit diff fe492a1 #(between current and some other node)git diff be158f6 aee7163 #(between two nodes)
12.2.8 Typical Workflow Scenario
Now that you have a repo, go into the repo folder.
• Add any files you want
• Make any changes you want to files
• Commit your changes
• Push your changes
To add your new files:
12.2. Git for Gits ;) 291
ZenPackers Documentation
[bash]: git add -a abc.py def.py[bash]: git add -A (Danger: adds and removes files from working tree)
To commit all changes execute:
[bash]: git commit -a
To finally push up your changes to your Repo hub (or github.com):
[bash]: git push
12.2.9 New Repo Workflow Scenario
• First go to github.com and create your account
• Then create an empty repository in the GUI
• Now on your workstation, pull down (clone) the empty repo:
[bash]: git clone https://github.com/zenoss/bogus.git[bash]: cd bogus/
• Now start writing your code, make files. . . . . .
• Now add files to your repo and push:
[bash]: git add -A[bash]: git commit -a[bash]: git push[bash]: git status.. Already up-to-date ..
12.2.10 Setting Repo Parameters
• Configure the Username and Email for the Repo:
[bash]: git config --global user.name "Joe Frazer"[bash]: git config --global user.email [email protected]
• Reset the Author for the Repo:
[bash]: git commit --amend --reset-author
12.2.11 Changing Branches
• Change branch from master to develop with checkout:
[bash]: git checkout develop[bash]: git status
292 Chapter 12. Other Topics:
ZenPackers Documentation
12.2.12 Merging Branches
You like the work you’ve done in develop and think it should be merged into master. You can do this by using themerge option.
• First change branches from develop to master:
[bash]: git checkout master
Now things are as before with master in its original state.
• Now you want to merge from develop:
[bash]: git merge developUpdating 1530600..2873dc4Fast-forward.gitignore | 2 +Makefile | 11 +-....
• Now you must push these changes up to your Hub:
[bash]: git pushTotal 0 (delta 0), reused 0 (delta 0)To [email protected]:zenoss/ZenPackers.git1530600..2873dc4 master -> master
12.2.13 Delete Unwanted Branches
If you want to eject unwanted branches from your repo, make sure to read the git-branch docs and the warnings aboutbeing fully merged (–delete option).
To remove a local branch:
git branch -D <branchName>
To remove a remote branch:
git push origin --delete <branchName>
12.2.14 Synchronizing Local Branches and References: Pruning
Sometimes you’ll have a lot of old remote branch references that have been long deleted on the hub. You can synchro-nize them with fetch:
git fetch -p
12.2.15 Revert a Branch to a Prior Commit
git revert will create a new commit that will undo what the prior commit(s) have done and put that into your history.It gives you a log of your undo.
12.2. Git for Gits ;) 293
ZenPackers Documentation
12.2.16 Resetting a Branch to a Prior Commit
• git checkout feature/area51
• Identify the number of your last “good” commit:
git log(grab the good commit number: e3f1e37)
• Reset your feature/area51 to that commit level:
git reset --hard e3f1e37
• Push it up to github:
git push --force origin feature/area51
• Test the diff between local and remote: Should show nothing:
git diff feature/area51..origin/feature/area51
12.2.17 Comparison of Git Branches
• Show only relevant commits between two git refs:
git log --no-merges master..develop
12.2.18 Avoiding Many Small Commits
You can make as many small changes as you like and still have a clean single commit by using git’s amend flag onyour commit:
git commit --amend(make your commit message)(write/quit)
Every time you make a new commit in this way, you get the benefit of small incremental changes and a clean commitlog. If you have already made a mess of things you can try the next technique to Squash your commits.
12.2.19 Squashing Multiple Commits
This allows you to take a lot of many small commits (and their messages) and convert them to a single coherentcommit. It keeps the history clean and clear.
In order to do this safely, we recommend only doing this in a feature branch (based on develop) that is not beingshared.
• From your feature branch, do a rebase with the -i flag:
git rebase -i develop
• When it shows you the multiple commits, change command in commits after the first “pick” to “squash”. Thussomething like this:
294 Chapter 12. Other Topics:
ZenPackers Documentation
pick 01d1124 Adding licensepick 6340aaa Moving license into its own filepick ebfd367 Jekyll has become self-aware.pick 30e0ccb Changed the tagline in the binary, too.
now becomes:
pick 01d1124 Adding licensesquash 6340aaa Moving license into its own filesquash ebfd367 Jekyll has become self-aware.squash 30e0ccb Changed the tagline in the binary, too.
• Now you write that out and it will ask you to fix-up the commit logs. Do this by changing to a unified commitmessage:
# This is a combination of 4 commits.# The first commit's message is:Dr Jekyll's final revisions to persona.
Add that license thingMoving license into its own fileJekyll has become self-aware.Changed the tagline in the binary, too.
• Once you write that out, you need to push it up with force flag to rewrite history:
git push -f
• If you have already pushed it up prior to this, or even created a Pull, your upstream commits and pulls will getreplaced with the unified commit.
12.3 Git and Code Reviews: Keeping History Clean
Keep the code history clean!
Note: Its a common practice to review the git log history by using git-log and git-blame. This history is important andvital to understanding the evolution of the code and how to effectively modify it without repeating historical mistakes.
It’s not as academic a point as it might seem. Almost every day we dig through code while troubleshooting, and oneof the most useful things is finding the JIRA related to a line of code.
We use git blame for this, and if the last time a line of interest was modified (or lines around it) wasn’t functional,we have to find the parent of that commit and re-run blame. We have to repeat this process for each non-functionalchange. It’s very annoying.
It also has the affect of making a section of code appear to be modified by many authors over a long period of time,which might indicate a troublesome area of code, even though the code is exactly the same functionally as the first andonly time it was really written.
The following recommendations are made in order to maximize the clarity of the history and blame records:
12.3. Git and Code Reviews: Keeping History Clean 295
ZenPackers Documentation
12.3.1 For new code: Always Check for PEP Errors
Getting those correct initially avoids any cleanup or PEP annoyances later. You don’t want to fix formatting errorslater because they muddy the log and blame history.
12.3.2 Reviewing Code: Don’t Make PEP Style Corrections
If style corrections are to be made they should be done during code development. Don’t make style/PEP fixes fordefects.
12.3.3 Reviewing Code: Question Unrelated Cosmetic Changes
If you are reviewing code for a defect, and you see style changes that are unrelated to the bug or feature at hand, askthe author to revert those changes so that the git-blame log will not be altered.
12.3.4 Avoid Many Small Commits for Defects
This doesn’t mean that you should have super-huge 10000 line commits. It does mean that small commits that are re-lated should be put into the same commit. See above sections on how to Avoiding Many Small Commits and SquashingMultiple Commits .
12.4 Git-Flow
Git flow simplifies development flow cycle. See http://danielkummer.github.io/git-flow-cheatsheet/
12.4.1 Installing Git-Flow
Note:
There are two versions of Git-Flow: NVIE and AVH (newer). Although both versions have the same basic com-mands, some of the commands in AHV are easier to use than older NVIE. The NVIE version has not beenactive since 2012.
For this reason we recommend using the active AVH version.
Git flow installation is well documented and simple. Refer to the following links for installation instructions:
• AVH: https://github.com/petervanderdoes/gitflow-avh
• NVIE: https://github.com/nvie/gitflow (last commit Sept 2012)
– It appears that no development will continue on this branch.
– Discussion: https://github.com/nvie/gitflow/issues/285
296 Chapter 12. Other Topics:
ZenPackers Documentation
12.4.2 Setup Git-Flow in the Existing Repo
First go into the repo base folder. Make sure you get a clean git status. Then you initialize the git repo for git flow asfollows:
[bash]: git flow init
Which branch should be used for bringing forth production releases?- develop
Branch name for production releases: [] master
Which branch should be used for integration of the "next release"?- develop
Branch name for "next release" development: [develop]
How to name your supporting branch prefixes?Feature branches? [feature/]Release branches? [release/]Hotfix branches? [hotfix/]Support branches? [support/]Version tag prefix? []Hooks and filters directory? [/data/zp/ZenPacks.zenoss.DB2/.git/hooks]
12.4.3 Create New Features and Work Flow
In features, you don’t want to use version numbers because it can cause chaos when multiple authors work the sameproject. Instead give the version a name, and only after the resulting develop is reviewed, you give it a version. (SourceUnknown: Rob B).
To start a new feature:
[bash]: git flow feature start area51[bash]: git flow feature publish
- (This creates the feature branch on Github, and allows "push")[bash]: git status
On branch feature/area51nothing to commit (working directory clean)
.... do some work ....
.... do some more work ....
.... you are finished ....
.... now commit ....
[bash]: git commit -am "Comment: This fixes bug ZPS-3234823"[bash]: git push (nothing happens)
. Counting objects: 4, done.
. Writing objects: 100% (4/4), 647 bytes | 0 bytes/s, done.
. Total 4 (delta 3), reused 0 (delta 0)
. To [email protected]:zenoss/DB2.git6f1c83e..faf56f5 feature/area51 -> feature/area51
- (Later you ask for a Pull Request or continue modifications)- (Someone may merge your Pull req into develop)- (Now you are finished with this feature...)- (You can either delete this branch or git-flow finish it)
(continues on next page)
12.4. Git-Flow 297
ZenPackers Documentation
(continued from previous page)
[bash]: git flow feature finish area51[bash]: git statusOn branch developnothing to commit (working directory clean)
Now you are back on develop. You still need to push your changes up:
[bash]: git pushTotal 0 (delta 0), reused 0 (delta 0)To [email protected]:zenoss/ZenPackers.git1530600..2873dc4 develop -> develop
12.4.4 Push the Develop onto the old Feature that is Stale
Warning: This flow can be dangerous. Use with caution!
You have created a branch (forgotten) that has been left behind and wish upgrade it with all the new changes thathave been made with other feature enhancements. You don’t have anything to save in it. Use these commands (withcaution) to merge develop back onto feature/forgotten:
[bash]: git checkout feature/forgotten[bash]: git push . develop:feature/forgotten[bash]: get checkout feature/forgotten[bash]: git commit -a[bash]: git push
12.4.5 Push a new Feature up to Origin for storage:
Sometimes you want a feature to be stored on your Hub. Git-Flow does not automatically push your features. You canpush it up to the hub like this:
[bash]: git push -u origin feature/new
12.4.6 Heavy Feature Workflow IMPORTANT: PLEASE READ
During heavy workflows on a project, we expect multiple teams to concurrently work on multiple features that getmerged into develop. This is common. This heavy workflow can be managed by the following workflow:
• Create your feature
• Work/commit/push feature branch
• Another team does works on feature/Atlantis (don’t be jealous!)
• Another teams merges feature/Atlantis into develop
• You need to rebase those changes into feature/area51 as follows:
git checkout develop && git pull # First update the local develop branchgit checkout feature/area51git rebase develop
298 Chapter 12. Other Topics:
ZenPackers Documentation
• If you have conflicts see merge_conflicts below, else continue
• Now continue work on feature/area51
• Repeat the rebases as needed when team Atlantis updates develop
• Once finished, create your final Pull Request
• Once merged, delete the feature branch.
12.4.7 Rebasing a Feature on Develop
Warning: If your team is still working on a feature and you notice that develop has been updated, you should tryto rebase those changes into your feature. This will avoid conflicts later when you merge back into develop.
You may get these messages
Note: Branches ‘develop’ and ‘origin/develop’ have diverged. Fatal: And branch ‘develop’ may be fast-forwarded.
Someone has added to develop during your work on feature/area51. This is common in a multi-user environment. Youwill have to merge the two together. To solve this, you need to:
• Sync local develop with origin: checkout develop, pull from origin to develop:
git checkout develop && git pull origin
• Rebase your feature on develop. You may have conflicts here if you’re unlucky:
git checkout feature/area51; git rebase develop# Same as: git flow feature rebase area51
• Check that nothing is broken:
git statusgit push # Push feature/area51 up to origin
• If there are conflicts you have to fix here. See merge_conflicts
12.4.8 Merge Conflicts: Fixing a Rebase
If you do have conflicts with your merge you can take a simple approach
to fixing them:
• Rebase against develop:
[zenoss@austin]: git rebase developFirst, rewinding head to replay your work on top of it...Applying: Make Tenant rels concreteUsing index info to reconstruct a base tree...M ZenPacks/zenoss/DB2/__init__.pyFalling back to patching base and 3-way merge...Auto-merging ZenPacks/zenoss/DB2/__init__.pyFalling back to patching base and 3-way merge...
→˓
→˓ [588/1329](continues on next page)
12.4. Git-Flow 299
ZenPackers Documentation
(continued from previous page)
Auto-merging ZenPacks/zenoss/DB2/__init__.pyCONFLICT (content): Merge conflict in ZenPacks/zenoss/DB2/__init__.pyFailed to merge in the changes.Patch failed at 0001 Make Tenant rels concreteThe copy of the patch that failed is found in:
/data/zp/ZenPacks.zenoss.DB2/.git/rebase-apply/patch
When you have resolved this problem, run "git rebase --continue".If you prefer to skip this patch, run "git rebase --skip" instead.To check out the original branch and stop rebasing, run "git rebase --abort".
• Edit the problem file and fix:
[zenoss@austin]: vi __init__.py( fix fix fix )
[zenoss@austin]: git status
rebase in progress; onto 34ae002You are currently rebasing branch 'feature/ZPS-17143_installWarnings' on
→˓'34ae002'.(fix conflicts and then run "git rebase --continue")(use "git rebase --skip" to skip this patch)(use "git rebase --abort" to check out the original branch)
Unmerged paths:(use "git reset HEAD <file>..." to unstage)(use "git add <file>..." to mark resolution)
both modified: __init__.py
• Add this file back into to index:
[zenoss@austin]: git add __init__.py
• Continue:
[zenoss@austin]: git rebase --continue
Applying: Make Tenant rels concreteApplying: fix impact relationsUsing index info to reconstruct a base tree...M ZenPacks/zenoss/DB2/Tenant.pyM ZenPacks/zenoss/DB2/__init__.pyFalling back to patching base and 3-way merge...Auto-merging ZenPacks/zenoss/DB2/__init__.pyCONFLICT (content): Merge conflict in ZenPacks/zenoss/DB2/__init__.pyCONFLICT (modify/delete): ZenPacks/zenoss/DB2/Tenant.py deleted in fix impact
→˓relations and modified in HEAD. Version HEAD of ZenPacks/zenoss/DB2/Tenant.py left in tree.Failed to merge in the changes.Patch failed at 0002 fix impact relationsThe copy of the patch that failed is found in:
/data/zp/ZenPacks.zenoss.DB2/.git/rebase-apply/patch
When you have resolved this problem, run "git rebase --continue".
(continues on next page)
300 Chapter 12. Other Topics:
ZenPackers Documentation
(continued from previous page)
If you prefer to skip this patch, run "git rebase --skip" instead.To check out the original branch and stop rebasing, run "git rebase --abort"
• Repeat: You may have to edit/re-edit a file, re-add, and continue as before:
[zenoss@austin]: vi __init__,py[zenoss@austin]: git add __init__.py[zenoss@austin]: git rebase --continue
ZenPacks/zenoss/DB2/Tenant.py: needs mergeYou must edit all merge conflicts and thenmark them as resolved using git add
• Delete what is required. You deleted a file but it is confused by this:
[zenoss@austin]: git rm Tenant.pyZenPacks/zenoss/DB2/Tenant.py: needs mergerm 'ZenPacks/zenoss/DB2/Tenant.py'
[zenoss@austin]: git rebase --continueApplying: fix impact relations
• If at this point the merge is good, but it asks you to pull, don’t pull! You really want to push your changes:
[zenoss@austin]: git statusOn branch feature/ZPS-17143_installWarningsYour branch and 'origin/feature/ZPS-17143_installWarnings' have diverged,and have 14 and 2 different commits each, respectively.(use "git pull" to merge the remote branch into yours)nothing to commit, working directory clean
[zenoss@austin]: git pushTo [email protected]:zenoss/ZenPacks.zenoss.DB2.git! [rejected] feature/ZPS-17143_installWarnings ->feature/ZPS-17143_installWarnings (non-fast-forward)error: failed to push some refs to'[email protected]:zenoss/ZenPacks.zenoss.DB2.git'hint: Updates were rejected because the tip of your current branch isbehindhint: its remote counterpart. Integrate the remote changes (e.g.hint: 'git pull ...') before pushing again.hint: See the 'Note about fast-forwards' in 'git push --help' for details.
[zenoss@austin]: git push --forceCounting objects: 12, done.Delta compression using up to 8 threads.Compressing objects: 100% (12/12), done.Writing objects: 100% (12/12), 1.22 KiB | 0 bytes/s, done.Total 12 (delta 6), reused 0 (delta 0)To [email protected]:zenoss/ZenPacks.zenoss.DB2.git+ 2bfc0a6...f7ddee9 feature/ZPS-17143_installWarnings ->
feature/ZPS-17143_installWarnings (forced update)
• If you see a clean status, its probably good. Make sure to test:
[zenoss@austin]: git statusOn branch feature/ZPS-17143_installWarnings
(continues on next page)
12.4. Git-Flow 301
ZenPackers Documentation
(continued from previous page)
Your branch is up-to-date with 'origin/feature/ZPS-17143_installWarnings'.nothing to commit, working directory clean
12.4.9 Git Stash: Stashing Modified Files
Git’s stash option allows you to put modified files into a temporary holding area. The usual scenario is to stash yourmods away then pull from the origin, and then re-place your stash’ed files into the tree. Then you can push the resultsback up to origin. Here is a possible workflow:
.... you made changes to develop, but you'd rather it be in a feature....
[bash]: git stash> Saved working directory and index state WIP on develop: e38b798 postrelease: 1.0.1 -> 1.0.2dev.....
[bash]: git flow feature start cleanup_on_aisle_7> Switched to a new branch 'feature/cleanup_on_aisle_7'
[bash]: git stash pop.... now you have your new mods overlaid ........ make whatever other modifications ........ now you can commit all your mods ....
[bash]: git commit -a
[bash]: git flow feature finish cleanup_on_aisle_7
[bash]: git push
12.4.10 Pull Requests: The Easy Way
The easiest way we have to get your code reviewed and merged into a major branch is to use Git-Flow to create afeature, push that feature up to Github, and have someone review it.
Here is the workflow in a nutshell:
• Create your feature with git flow
• Make your mods
• Commit your mods
• Push (or publish) your feature up to Gitflow
• Go into the Github GUI, select your feature
• Make your pull request
• Ask for a review
• That reviewer then merges your changes into develop
• Finish your feature locally:
– Using git push.default=simple: Everything on Github is cleaned for you. (See the Push Defaults section)
– Otherwise: After finishing, remove the feature repo in Github
302 Chapter 12. Other Topics:
ZenPackers Documentation
• Finally, from your local repo, do a “git pull” to sync up
12.4.11 Push Defaults
To set your push defaults you can edit your .gitconfig and put this option:
[user]name = Pat Mibakemail = [email protected]
[push]default = simple
• Note: See git-config man page: Search /push.default for more details
12.4.12 Git 1.X and 2.X Warnings and Errors
• You may you get this warning when trying to push a new branch to origin:
[bash]: git pushfatal: The current branch develop has no upstream branch.To push the current branch and set the remote as upstream, use
git push --set-upstream origin develop
Its usually safe to follow this suggestion
12.5 Installing a VIM IDE setup with Synstastic
See http://sontek.net/blog/detail/turning-vim-into-a-modern-python-ide#file-browser as a base reference. We’vechanged up some of the packages that follow for a more modern environment.
You want to do the following:
1. Copy your old .vimrc (and .vim if you have on) to a backup
2. Make some folders and init git:
[zenoss@~]: mkdir -p ~/.vim/{autoload,bundle}[zenoss@~]: cd ~/.vim/[[email protected]]: git init
3. Install Pathogen:
curl -Sso ~/.vim/autoload/pathogen.vim https://raw.github.com/tpope/vim-pathogen/master/autoload/pathogen.vim
4. Paste the following into your .vimrc:: execute pathogen#infect() syntax on filetype plugin indent on
5. Now execute the following inside the .vim folder
12.5. Installing a VIM IDE setup with Synstastic 303
ZenPackers Documentation
git submodule add http://github.com/tpope/vim-fugitive.git bundle/fugitivegit submodule add https://github.com/msanders/snipmate.vim.git bundle/snipmategit submodule add https://github.com/tpope/vim-surround.git bundle/surroundgit submodule add https://github.com/tpope/vim-git.git bundle/gitgit submodule add https://github.com/ervandew/supertab.git bundle/supertabgit submodule add https://github.com/sontek/minibufexpl.vim.git bundle/minibufexplgit submodule add https://github.com/wincent/Command-T.git bundle/command-tgit submodule add https://github.com/mitechie/pyflakes-pathogen.gitgit submodule add https://github.com/mileszs/ack.vim.git bundle/ackgit submodule add https://github.com/sjl/gundo.vim.git bundle/gundogit submodule add https://github.com/fs111/pydoc.vim.git bundle/pydocgit submodule add https://github.com/scrooloose/syntastic.git bundle/systasticgit submodule add https://github.com/alfredodeza/pytest.vim.git bundle/py.testgit submodule add https://github.com/reinh/vim-makegreen bundle/makegreengit submodule add https://github.com/vim-scripts/TaskList.vim.git bundle/tasklistgit submodule add https://github.com/vim-scripts/The-NERD-tree.git bundle/nerdtreegit submodule add https://github.com/sontek/rope-vim.git bundle/ropevimgit submodule initgit submodule updategit submodule foreach git submodule initgit submodule foreach git submodule update
12.6 DB2 Information for Scavengers
12.6.1 Some Base links:
• http://www.tldp.org/HOWTO/DB2-HOWTO/planning.html
• http://www.centos.org/docs/2/rh-cm-en-1.0/s1-service-db2.html
• http://www.dbforums.com/db2/1655091-db2-cpu-time-wait-time.html
• http://stackoverflow.com/questions/15382561/adding-db2-jars-to-java-webapp-using-maven
12.6.2 Supported drivers for JDBC and SQLJ
The DB2 product includes support for two types of JDBC driver architecture. According to the JDBC specification,there are four types of JDBC driver architectures:
• Type 1 (Wont work for Zenpack. Not supported in DB2)
Drivers that implement the JDBC API as a mapping to another data access API, such as OpenDatabase Connectivity (ODBC). Drivers of this type are generally dependent on a native library,which limits their portability. The DB2 database system does not provide a type 1 driver.
• Type 2 (Might work, but not portable)
Drivers that are written partly in the Java programming language and partly in native code. Thedrivers use a native client library specific to the data source to which they connect. Because of thenative code, their portability is limited.
• Type 3 (Standard Choice without the fancy Type 4 options. Use if possible)
Drivers that use a pure Java client and communicate with a data server using a data-server-independent protocol. The data server then communicates the client’s requests to the data source.
• Type 4 (Fancy Options: Not needed)
304 Chapter 12. Other Topics:
ZenPackers Documentation
Drivers that are pure Java and implement the network protocol for a specific data source. The clientconnects directly to the data source.
12.6.3 DB2 vs Oracle
DB2-Oracle Terminology Mapping
Because Oracle applications can be enabled to work with DB2® data serverswhen the DB2 environment is set up appropriately, it is important tounderstand how certain Oracle concepts map to DB2 concepts.
Table 1 provides a concise summary of commonly used Oracle terms andtheir DB2 equivalents.
Table 1. Mapping of common Oracle concepts to DB2 concepts+------------------------------------------------------------------------+| Oracle concept | DB2 concept | Notes ||------------------------+-------------------+---------------------------|| active log | active log | This is the same concept. ||------------------------+-------------------+---------------------------|| actual parameter | argument | This is the same concept. ||------------------------+-------------------+---------------------------|| alert log | db2diag log files | The db2diag log files are || | and | primarily intended for || | administration | use by IBM Software || | notification log | Support for || | | troubleshooting purposes. || | | The administration || | | notification log is || | | primarily intended for || | | troubleshooting use by || | | database and system || | | administrators. || | | Administration || | | notification log messages || | | are also logged to the || | | db2diag log files using a || | | standardized message || | | format. ||------------------------+-------------------+---------------------------|| archive log | offline-archive | This is the same concept. || | log | ||------------------------+-------------------+---------------------------|| archive log mode | log archiving | This is the same concept. ||------------------------+-------------------+---------------------------|| background_dump_dest | diagpath | This is the same concept. ||------------------------+-------------------+---------------------------|| created global | created global | This is the same concept. || temporary table | temporary table | ||------------------------+-------------------+---------------------------|| cursor sharing | statement | This is the same concept. || | concentrator | ||------------------------+-------------------+---------------------------|| data block | data page | This is the same concept. ||------------------------+-------------------+---------------------------|| data buffer cache | buffer pool | This is the same concept. |
(continues on next page)
12.6. DB2 Information for Scavengers 305
ZenPackers Documentation
(continued from previous page)
| | | However, in DB2 you can || | | have as many buffer pools || | | of any page size you || | | like. ||------------------------+-------------------+---------------------------|| data dictionary | system catalog | The DB2 system catalog || | | contains metadata in the || | | form of tables and views. || | | The database manager || | | creates and maintains two || | | sets of system catalog || | | views that are defined on || | | the base system catalog || | | tables: || | | || | | * SYSCAT views, which || | | are read-only views || | | * SYSSTAT views, which || | | are updatable views || | | that contain || | | statistical || | | information that is || | | used by the optimizer ||------------------------+-------------------+---------------------------|| data dictionary cache | catalog cache | This is the same concept. ||------------------------+-------------------+---------------------------|| data file | container | DB2 data is physically || | | stored in containers, || | | which contain objects. ||------------------------+-------------------+---------------------------|| database link | nickname | A nickname is an || | | identifier that refers to || | | an object at a remote || | | data source (a federated || | | database object). ||------------------------+-------------------+---------------------------|| dual table | dual table | This is the same concept. ||------------------------+-------------------+---------------------------|| dynamic performance | snapshot monitor | Snapshot monitor SQL || views | SQL | administrative views, || | administrative | which use schema || | views | SYSIBMADM, return monitor || | | data about a specific || | | area of the database || | | system. For example, the || | | SYSIBMADM.SNAPBP SQL || | | administrative view || | | provides a snapshot of || | | buffer pool information. ||------------------------+-------------------+---------------------------|| extent | extent | A DB2 extent is made up || | | of a set of contiguous || | | data pages. ||------------------------+-------------------+---------------------------|| formal parameter | parameter | This is the same concept. ||------------------------+-------------------+---------------------------|| global index | nonpartitioned | This is the same concept. |
(continues on next page)
306 Chapter 12. Other Topics:
ZenPackers Documentation
(continued from previous page)
| | index | ||------------------------+-------------------+---------------------------|| inactive log | online-archive | This is the same concept. || | log | ||------------------------+-------------------+---------------------------|| init.ora and Server | database manager | A DB2 instance can || Parameter File | configuration | contain multiple || (SPFILE) | file and database | databases. Therefore, || | configuration | configuration parameters || | file | and their values are || | | stored at both the || | | instance level, in the || | | database manager || | | configuration file, and || | | at the database level, in || | | the database || | | configuration file. These || | | files are managed through || | | the GET or UPDATE DBM CFG || | | command and the GET or || | | UPDATE DB CFG command, || | | respectively. ||------------------------+-------------------+---------------------------|| instance | instance or | An instance is a || | database manager | combination of background || | | processes and shared || | | memory. A DB2 instance is || | | also known as a database || | | manager. Because a DB2 || | | instance can contain || | | multiple databases, there || | | are DB2 configuration || | | files at both the || | | instance level (the || | | database manager || | | configuration file) and || | | at the database level || | | (the database || | | configuration file). ||------------------------+-------------------+---------------------------|| large pool | utility heap | The utility heap is used || | | by the backup, restore, || | | and load utilities. ||------------------------+-------------------+---------------------------|| library cache | package cache | The package cache, which || | | is allocated from || | | database shared memory, || | | is used to cache sections || | | for static and dynamic || | | SQL and XQuery statements || | | on a database. ||------------------------+-------------------+---------------------------|| local index | partitioned index | This is the same concept. ||------------------------+-------------------+---------------------------|| materialized view | materialized | An MQT is a table whose || | query table (MQT) | definition is based on || | | the results of a query |
(continues on next page)
12.6. DB2 Information for Scavengers 307
ZenPackers Documentation
(continued from previous page)
| | | and is meant to be used || | | to improve performance. || | | The DB2 SQL compiler || | | determines whether a || | | query would run more || | | efficiently against an || | | MQT than it would against || | | the base table on which || | | the MQT is based. ||------------------------+-------------------+---------------------------|| noarchive log mode | circular logging | This is the same concept. ||------------------------+-------------------+---------------------------|| Oracle Call Interface | DB2CI Interface | DB2CI is a 'C' and 'C++' || (OCI) | | application programming || | | interface that uses || | | function calls to connect || | | to DB2 Version 9.7 || | | databases, manage || | | cursors, and perform SQL || | | statements. See [4]IBM || | | Data Server Driver for || | | DB2CI for a list of OCI || | | APIs supported by the || | | DB2CI driver. ||------------------------+-------------------+---------------------------|| Oracle Call Interface | Call Level | CLI is a C and C++ || (OCI) | Interface (CLI) | application programming || | | interface that uses || | | function calls to pass || | | dynamic SQL statements as || | | function arguments. In || | | most cases, you can || | | replace an OCI function || | | with a CLI function and || | | relevant changes to the || | | supporting program code. ||------------------------+-------------------+---------------------------|| ORACLE_SID environment | DB2INSTANCE | This is the same concept. || variable | environment | || | variable | ||------------------------+-------------------+---------------------------|| partitioned tables | partitioned | This is the same concept. || | tables | ||------------------------+-------------------+---------------------------|| Procedural | SQL Procedural | SQL PL is an extension of || Language/Structured | Language (SQL PL) | SQL that consists of || Query Language | | statements and language || (PL/SQL) | | elements. SQL PL provides || | | statements for declaring || | | variables and condition || | | handlers, assigning || | | values to variables, and || | | implementing procedural || | | logic. SQL PL is a subset || | | of the SQL Persistent || | | Stored Modules (SQL/PSM) || | | language standard. Oracle |
(continues on next page)
308 Chapter 12. Other Topics:
ZenPackers Documentation
(continued from previous page)
| | | PL/SQL statements can be || | | compiled and executed || | | using DB2 interfaces. ||------------------------+-------------------+---------------------------|| program global area | application | Application shared memory || (PGA) | shared memory and | stores information that || | agent private | is shared between a || | memory | database and a particular || | | application: primarily, || | | rows of data being passed || | | to or from the database. || | | Agent private memory || | | stores information used || | | to service a particular || | | application, such as sort || | | heaps, cursor || | | information, and session || | | contexts. ||------------------------+-------------------+---------------------------|| redo log | transaction log | The transaction log || | | records database || | | transactions and can be || | | used for recovery. ||------------------------+-------------------+---------------------------|| role | role | This is the same concept. ||------------------------+-------------------+---------------------------|| segment | storage object | This is the same concept. ||------------------------+-------------------+---------------------------|| session | session; database | This is the same concept. || | connection | ||------------------------+-------------------+---------------------------|| startup nomount | db2start | The command that starts || | | the instance. ||------------------------+-------------------+---------------------------|| synonym | alias | An alias is an || | | alternative name for a || | | table, view, nickname, or || | | another alias. The term || | | "synonym" is tolerated || | | and can be specified in || | | place of "alias". Aliases || | | are not used to control || | | what version of a DB2 || | | procedure or user-defined || | | function is being used by || | | an application; to do || | | this, use the SET PATH || | | statement to add the || | | required schema to the || | | value of the CURRENT PATH || | | special register. ||------------------------+-------------------+---------------------------|| system global area | instance shared | The instance shared || (SGA) | memory and | memory stores all of the || | database shared | information for a || | memory | particular instance, such || | | as lists of all active |
(continues on next page)
12.6. DB2 Information for Scavengers 309
ZenPackers Documentation
(continued from previous page)
| | | connections and security || | | information. The database || | | shared memory stores || | | information for a || | | particular database, such || | | as package caches, log || | | buffers, and buffer || | | pools. ||------------------------+-------------------+---------------------------|| SYSTEM table space | SYSCATSPACE table | The SYSCATSPACE table || | space | space contains the system || | | catalog. This table space || | | is created by default || | | when you create a || | | database. ||------------------------+-------------------+---------------------------|| table space | table space | This is the same concept. ||------------------------+-------------------+---------------------------|| user global area (UGA) | application | Application global memory || | global memory | comprises application || | | shared memory and || | | application-specific || | | memory. |+------------------------------------------------------------------------+
12.6.4 Installing DB2 Express on Linux
• First you must install Linux
• Next install the prerequisites for DB2
• Make sure the client can do X11 Forwarding to your workstation
• Download DB2 Express, extract it in /opt
• SSH into your instance as root, X11 forwarding on
• Run the installer at /opt/expc/db2setup
• Select defaults and make sure to save set in the Response File
• If you can’t type into Java, just cut-n-paste passwords
• Make sure the installer finishes without error
• Save the Response File
• Next time use the Response File to install all
• Once installed, issue “db2sampl” to create a sample db. - db2 connect to sample - db2 ‘select * from dept’
Links for Installation:
• http://www.ibiblio.org/pub/linux/docs/HOWTO/DB2-HOWTO
• http://www.tldp.org/HOWTO/DB2-HOWTO/prerequisites.html
• http://www.sqlpanda.com/2013/08/install-db2-105-on-centos-64.html
310 Chapter 12. Other Topics:
ZenPackers Documentation
12.6.5 Removing DB2 on Linux
• Remove the Database Administration Server:
sudo su - dasusr1 db2admin stop/opt/ibm/db2/V10.5/instance/dasdrop dasusr1
• Remove the DB2 instance(s):
sudo su - db2inst1 -c db2stop/opt/ibm/db2/V10.5/instance/db2ilist/opt/ibm/db2/V10.5/instance/db2idrop db2inst1/opt/ibm/db2/V10.5/instance/db2ilist
• Remove the software installation:
/opt/ibm/db2/V10.5/install/db2_deinstall -a
• Remote the user accounts too:
userdel -r db2inst1userdel -r dasusr1userdel -r db2fenc1
12.7 Convenience Tools
Tools can make work easier and efficient.
12.7.1 Bash Navigation Tool
This tool allows you to quickly get to your ZP source tree. It has filename completion:
export zpdefault=Ceph
if [ -z $zpbase ]; thenif [ -d /z ]; then
zpbase="/z"elif [ -d /mnt/src/zenpacks ]; then
zpbase="/mnt/src/zenpacks"elif [ -d /home/zenoss/src/europa/src/zenpacks ]; then
zpbase="/home/zenoss/src/europa/src/zenpacks"elif [ -d /zenpacks ]; then
zpbase="/zenpacks"elif [ -d /data/zp ]; then
zpbase="/data/zp"else
zpbase="/tmp"fi
export zpbasefi
zp(){
(continues on next page)
12.7. Convenience Tools 311
ZenPackers Documentation
(continued from previous page)
# --------------------------------------------------------------------------# Setup up variables..# --------------------------------------------------------------------------if [[ -z "$zpdefault" ]]; then
export zpdefault=PythonCollectorfi
if [[ -z "$zplast" && -z "$zpcurrent" ]]; thenexport zplastexport zpcurrent
fi
# --------------------------------------------------------------------------# Find and go to right folders# --------------------------------------------------------------------------if [ -n "$1" ]; then
if [ -d "$zpbase"/"$1" ]; thenecho "zenpack exists";cd "$zpbase"/"$1";return;
fi;if [[ "$1" == "-" ]]; then
if [[ -n "$zplast" && -n "$zpcurrent" ]]; thenecho "returning to prior zenpack";zptemp=$zplastzplast=$zpcurrentzpcurrent=$zptempcd "$zpcurrent"
fireturn;
fi;second="ZenPacks.zenoss.$1";third=${second//./\/};final="$zpbase"/"$second"/"$third"/;
if [ -d $final ]; thenecho "zenpack exists";zplast="$PWD";zpcurrent="$final";cd "$final";
elseecho "no such Zenpack $final";return;
fi;else
echo "Using default zp";zp $zpdefault;
fi}
function _zp(){
local cmd="${1##*/}"local word=${COMP_WORDS[COMP_CWORD]}# Pointer to current completion word.
(continues on next page)
312 Chapter 12. Other Topics:
ZenPackers Documentation
(continued from previous page)
# By convention, it's named "cur" but this isn't strictly necessary.
words=""case "$word" in
*)for i in $zpbase/ZenPacks.zenoss.${word}*; do
if [[ -d $i ]]; thenwords+="${i##*.}"words+=" "
fidone
esac
COMPREPLY=() # Array variable storing the possible completions.COMPREPLY=( $( compgen -W "$words" -- $cur ) )
}
complete -o filenames -F _zp zp
To use it, put the contents into a zp.sh file and source it from your .bashrc and then use it as follows:
zp Py<tab>
It should complete the name to PythonCollector, assuming you have the PythonCollector ZenPack installed, and takeyou to the folder:
$zpbase/ZenPacks.zenoss.PythonCollector/ZenPacks/zenoss/PythonCollector/
Make sure your ZenPacks are stored in one of the following directories:
• /z
• /mnt/src/zenpacks
• /home/zenoss/src/europa/src/zenpacks
• /zenpacks
• /data/zp
• /tmp
12.8 Simulate devices
12.8.1 Simulate devices for NetAppMonitor ZenPack
In this example we will use data from zenmodeler’s debug log for 7-Mode or C-Mode devices to simulate the samedevice with all components in our test environment. To do this, we need to perform some additional steps:
1. Create a folder on Zenoss system and name it ‘data’:
/home/zenoss/data
2. Open debug log and copy a list with data which goes after “<plugin name> results = ”:
NetApp.ZAPI.CMode.SystemNodes results =
12.8. Simulate devices 313
ZenPackers Documentation
3. Save the file with this list into the folder you have created earlier. The file should be named according to thistemplate <plugin name>.dump:
NetApp.ZAPI.CMode.SystemNodes.dump
4. Optionally, copy existing PythonClient.py file, it is located in /opt/zenoss/Products/ DataCollector/
cp /opt/zenoss/Products/DataCollector/PythonClient.py /opt/zenoss/Products/→˓DataCollector/ original_PythonClient.py
5. Copy PythonClient.patch file into zenoss home directory:
/home/zenoss/
6. Apply patch to PythonClient.py, if you are in /home/zenoss folder you can perform:
patch /opt/zenoss/Products/DataCollector/PythonClient.py < PythonClient.patch
7. Create NetApp device through Zenoss UI with any IP Address but do not model it
8. Run zenmodeler for already existing NetApp device with appropriate device class through terminal:
zenmodeler run -d <device ID> or zenmodeler run -d <device ID> -collect=<plugin→˓name>
9. In case you have done copy in step 4 you can move original file back:
cp /opt/zenoss/Products/DataCollector/original_PythonClient.py /opt/zenoss/→˓Products/ DataCollector/PythonClient.py
12.8.2 Patch for PythonClient.py
PythonClient.patch:
@@ -50,8 +50,21 @@"""deferreds = []for plugin in self.plugins:
- log.debug("Running collection for plugin %s", plugin.name())- r = plugin.collect(self.device, log)+ import os+ dump_fname = '/home/zenoss/data/%s.dump' % (plugin.name(),)+ if os.path.exists(dump_fname):+ log.info('Use %s dump file for %s plugin',+ dump_fname,+ plugin.name())+ with open(dump_fname, 'r') as f:+ data = f.read()++ results = eval(data)+ r = Deferred()+ r.callback(results)+ else:+ log.debug("Running collection for plugin %s", plugin.name())+ r = plugin.collect(self.device, log)
if isinstance(r, Deferred):
(continues on next page)
314 Chapter 12. Other Topics:
ZenPackers Documentation
(continued from previous page)
deferreds.append(r)r.addBoth(self.collectComplete, plugin)
12.8. Simulate devices 315