Download - 20 cool things python
20 Cool Things I
have done in Python
Maurice ManeschiRedWaratah.com
Who am I
I am a former mainframe developer
I work for Financial Advisory company
All Microsoft and Excel
However, Python gives me a tool kit that easily solves dvierse, apparently intractable data and integration problems
I am a hero for the problems I can solve
Format
I will fly through 20 examples of what I have done
Slow me down for the interesting slides, speed me up for the boring ones
For details, grab me afterwards over coffee
See the key libraries on the bottom rightSearch the web for samples
Send me email [email protected]
Configuration
Python header
Command line
INI file
Windows Registry
Configuration in the header
Configuration command line
Using argparse library
Configuration ini file
Using configparser library
Configuration Windows registry
Using winreg library
Using DLLs
I had to debug a Windows program in production without a debugger
Using Python ctypes, I called each subroutine until I discovered the defect
Using DLLs
Using ctypes library
Driving a website
Single window website for MSIE only
Running reports is fiddly, same report routinely is painful
Python proxy server to log the responses
I built an application that mimicked the browser and download the reports in PDF and Excel
Driving a web site part 1
Using urllib, html, http and re libraries
Driving a web site part 2
Regular Expression Log Parsing
MS IIS logs are nightly copied to a folder
Python uses RE to capture log ons and report runs
Converts IP address to client name via firewall config
Saves the data to an SQLite database
This data is provided regularly to the wiki article on active clients
Regular Expression Log Parsing
Using re, sqlite3 libraries
Regular Expression Magic
We have the Mediawiki (Wikipedia) for documentation and TestTrack for defect tracking
Python does a nightly read of the wiki articles and builds a huge regular expression (3 actually) of all titles
Python does a nightly extract of the contents of TestTrack to HTML and hyper links any reference to the articles back to the wiki
Regular Expression Magic - pt1
Using re, subprocess libraries
Regular Expression Magic pt 2
Scheduled Task Status
We have about 30 scheduled tasks that run each day (or more frequently)
Python scans the status return codes every 5 minutes and puts the results into a CSV files
MSDN defines the format of the job files and Python can read these formats
Nagios warns if any status codes are not 0
Scheduled Task Status
Using struct, csv libraries
Scheduled Task Status
Static Data Checker
Our key software tool does not audit every Static Data table
Changes can happen inadvertently
Python reads every nominated table, pickles each row and pops the results into an SQLite table
Daily report of new, changed or deleted rows
Static Data Checker
Static data
Row
Key
ValuePickle
SQlite
Using adodb, pickle, sqlite3 libraries
Static Data Checker
Static Data Checker
Conan: Flagging required email
Purchased Software is meant to send email confirming financial transactions that we track
It fails to correctly ascertain settlements
Python creates a web site that shows which settlements have to be sent
Annotations
Validates against the transmission logs
Conan
Using html, cgi, sqlite3, adodb libraries
Conan
Nagios
I spoke three years ago about Python for writing Nagios plugins
Our Nagios server got virtualised two years ago
We have a Ubuntu desktop PC showing the Nagios and RateFeed status
Sounds an alarm when new errors emerge
Nagios
Using numerous libraries
Tkinter
I love it
Not sexy or super responsive, but fast to code and reliable
Six different Tkinter applications
Using tkinter, tkmessagebox, tkdialog libraries
Tkinter example 1
Tkinter example 2
Tkinter example 3
Tkinter example 4
Tkinter example 5
RateFeed
Three Tkinter based applications written in Python
Download rates from Reuters via FTP
Calculate curves from a detailed INI file
Keep results and logs in two SQLite files
Write to two servers via DCOM
Respond to queries via RPC
RateFeed
Using numerous libraries
RateFeed Export
Managers cannot understand our RateFeed
Python generates two exports a day in ExcelImport openpyxl
Tabs for different curve types
Columns for incoming data, formulae and outgoing data
Used to get sign off on curve changes
RateFeed Export
Using openpyxl, sqlite3 libraries
Firewall settings
We open our firewall to registered customers only
Their IP addresses change as they change ISP, proxy settings, location etc.
Checking IP addresses quite error prone.
Now after each change, we export our firewall configuration to a set location
On change, Python updates our wiki with current client data
Firewall settings
Using re, smtplib libraries
Crystal Report Management
We have 300 Crystal Reports
Our test server databases have different names to production, but Crystal Reports requires each table to be reset to use a different database
Python shifts changes in the production version and the test version by using the Crystal API to fix every table
All via accessing the DCOM interface into Crystal Reports
Crystal Reports Management
Using win32com.client library (pywin32)
Change Request Management
Change requests go into our defect management system
When assessed and approved, the go into the work list.
Python picks up newly assessed work and adds it to the end of Planner XML files for project management
Email reports show projections on updates to these plans
Change Request Management
Using sax, adodb libraries
Crystal Report Testing
Crystal Reports used for confirmation of trading
These reports have loads of business logic
If we need to make a change, what is the impact
Python re runs every such confirmation for last 12 months under the new and old template, exports to Excel and flags any differences
Tkinter GUI shows progress
Crystal Report Testing
Using adodb, html, win32com.client, tkinter libraries
Grabs parameters from past confos
Applies them to the new and old template
Exports to HTML and compare
Note differences and store
Risk scenario building
Have special Risk reports that model the impact of different current shifts
28 currencies x 11 scenarios x 15 points per currency
GUI is slow, errors high
GUI exports to XML and imports
Python generates XML with all required scenarios built from an INI file
Tkinter interface
Risk scenario building
Given this...
Risk scenario building
...builds this:
Using sax, win32com.client, tkinter libraries
Backup tool
Test servers are not backed up
Python scheduled task puts contents of key folder and anything linked to it into a ZIP file and puts this on the main file server
Follows file links
Backup tool
Using os, win32com.client, zip libraries
Disaster Recovery
Much of our DRP is in soft copy
The detail is in a Media Wiki database and TestTrack database
Python exports all MediaWiki articles and TestTrack items into HTML
These are burned to DVDs quarterly along with ZIPs from the backup tool
Disaster Recovery
Using subprocess, urllib libraries
RESTful Interface
I wrote a dashboard using the Underscore backbone (in JavaScript). It wanted to interact via Restful interface
I created such an interface as a Python CGI with the Apache rewriting
RESTful Interface
Using cgi, sqlite3, xmlrpc libraries
RESTful Interface
ImageMagick Scripting
I have a Python script running unsupervised on the server
It responds to RPC requests
Python drives ImageMagick to build a visual representation
Also, did a target for a competition
ImageMagick Scripting
Using subprocess, os libraries
ImageMagick Scripting
Test Server Symbol Management
Each server has different resource locationsServer name is in the UNC
These are in an INI file
Python library to access this INI file
Caches the results in a pickle file to reduce network traffic and improve resiliency
Test Server Symbol Management
Rate Checker
Many rates go in each day at different times
Required to confirm the success of rates
Python compares today's rates with yesterday, a week ago and last month end
Reports via email
Emails are logged as proof of our diligence
Rate Checker
Using smtplib, adodb, configparser libraries
Rate Checker
Special Thanks to...
Guido Van Rossum keeping it simple and powerful
Mark Hammond Making it work on Windows
LibariesTkinter for easy UI
Openpyxl for Excel import and export
email and smtplib for sending email
Urllib and httplib for capture and parsing
Re for regular expressions
argparse, configparser and winreg for parameterisation
Subprocess for process management
And the Python community for sample code all over the internet
Go forth and code!
RedWaratah.comOpen Source Solutions for Business