20 Cool Things I
have done in Python

Maurice ManeschiRedWaratah.com

Who am I

I am a former mainframe developer

I work for Financial Advisory company

All Microsoft and Excel

However, Python gives me a tool kit that easily solves dvierse, apparently intractable data and integration problems

I am a hero for the problems I can solve

Format

I will fly through 20 examples of what I have done

Slow me down for the interesting slides, speed me up for the boring ones

For details, grab me afterwards over coffee

See the key libraries on the bottom rightSearch the web for samples

Send me email anything@redwaratah.com

Configuration

Python header

Command line

INI file

Windows Registry

Configuration in the header

Configuration command line

Using argparse library

Configuration ini file

Using configparser library

Configuration Windows registry

Using winreg library

Using DLLs

I had to debug a Windows program in production without a debugger

Using Python ctypes, I called each subroutine until I discovered the defect

Using DLLs

Using ctypes library

Driving a website

Single window website for MSIE only

Running reports is fiddly, same report routinely is painful

Python proxy server to log the responses

I built an application that mimicked the browser and download the reports in PDF and Excel

Driving a web site part 1

Using urllib, html, http and re libraries

Driving a web site part 2

Regular Expression Log Parsing

MS IIS logs are nightly copied to a folder

Python uses RE to capture log ons and report runs

Converts IP address to client name via firewall config

Saves the data to an SQLite database

This data is provided regularly to the wiki article on active clients

Regular Expression Log Parsing

Using re, sqlite3 libraries

Regular Expression Magic

We have the Mediawiki (Wikipedia) for documentation and TestTrack for defect tracking

Python does a nightly read of the wiki articles and builds a huge regular expression (3 actually) of all titles

Python does a nightly extract of the contents of TestTrack to HTML and hyper links any reference to the articles back to the wiki

Regular Expression Magic - pt1

Using re, subprocess libraries

Regular Expression Magic pt 2

Scheduled Task Status

We have about 30 scheduled tasks that run each day (or more frequently)

Python scans the status return codes every 5 minutes and puts the results into a CSV files

MSDN defines the format of the job files and Python can read these formats

Nagios warns if any status codes are not 0

Scheduled Task Status

Using struct, csv libraries

Scheduled Task Status

Static Data Checker

Our key software tool does not audit every Static Data table

Changes can happen inadvertently

Python reads every nominated table, pickles each row and pops the results into an SQLite table

Daily report of new, changed or deleted rows

Static Data Checker

Static data

Row

Key

ValuePickle

SQlite

Using adodb, pickle, sqlite3 libraries

Static Data Checker

Static Data Checker

Conan: Flagging required email

Purchased Software is meant to send email confirming financial transactions that we track

It fails to correctly ascertain settlements

Python creates a web site that shows which settlements have to be sent

Annotations

Validates against the transmission logs

Conan

Using html, cgi, sqlite3, adodb libraries

Conan

Nagios

I spoke three years ago about Python for writing Nagios plugins

Our Nagios server got virtualised two years ago

We have a Ubuntu desktop PC showing the Nagios and RateFeed status

Sounds an alarm when new errors emerge

Nagios

Using numerous libraries

Tkinter

I love it

Not sexy or super responsive, but fast to code and reliable

Six different Tkinter applications

Using tkinter, tkmessagebox, tkdialog libraries

Tkinter example 1

Tkinter example 2

Tkinter example 3

Tkinter example 4

Tkinter example 5

RateFeed

Three Tkinter based applications written in Python

Download rates from Reuters via FTP

Calculate curves from a detailed INI file

Keep results and logs in two SQLite files

Write to two servers via DCOM

Respond to queries via RPC

RateFeed

Using numerous libraries

RateFeed Export

Managers cannot understand our RateFeed

Python generates two exports a day in ExcelImport openpyxl

Tabs for different curve types

Columns for incoming data, formulae and outgoing data

Used to get sign off on curve changes

RateFeed Export

Using openpyxl, sqlite3 libraries

Firewall settings

We open our firewall to registered customers only

Their IP addresses change as they change ISP, proxy settings, location etc.

Checking IP addresses quite error prone.

Now after each change, we export our firewall configuration to a set location

On change, Python updates our wiki with current client data

Firewall settings

Using re, smtplib libraries

Crystal Report Management

We have 300 Crystal Reports

Our test server databases have different names to production, but Crystal Reports requires each table to be reset to use a different database

Python shifts changes in the production version and the test version by using the Crystal API to fix every table

All via accessing the DCOM interface into Crystal Reports

Crystal Reports Management

Using win32com.client library (pywin32)

Change Request Management

Change requests go into our defect management system

When assessed and approved, the go into the work list.

Python picks up newly assessed work and adds it to the end of Planner XML files for project management

Email reports show projections on updates to these plans

Change Request Management

Using sax, adodb libraries

Crystal Report Testing

Crystal Reports used for confirmation of trading

These reports have loads of business logic

If we need to make a change, what is the impact

Python re runs every such confirmation for last 12 months under the new and old template, exports to Excel and flags any differences

Tkinter GUI shows progress

Crystal Report Testing

Using adodb, html, win32com.client, tkinter libraries

Grabs parameters from past confos

Applies them to the new and old template

Exports to HTML and compare

Note differences and store

Risk scenario building

Have special Risk reports that model the impact of different current shifts

28 currencies x 11 scenarios x 15 points per currency

GUI is slow, errors high

GUI exports to XML and imports

Python generates XML with all required scenarios built from an INI file

Tkinter interface

Risk scenario building

Given this...

Risk scenario building

...builds this:

Using sax, win32com.client, tkinter libraries

Backup tool

Test servers are not backed up

Python scheduled task puts contents of key folder and anything linked to it into a ZIP file and puts this on the main file server

Follows file links

Backup tool

Using os, win32com.client, zip libraries

Disaster Recovery

Much of our DRP is in soft copy

The detail is in a Media Wiki database and TestTrack database

Python exports all MediaWiki articles and TestTrack items into HTML

These are burned to DVDs quarterly along with ZIPs from the backup tool

Disaster Recovery

Using subprocess, urllib libraries

RESTful Interface

I wrote a dashboard using the Underscore backbone (in JavaScript). It wanted to interact via Restful interface

I created such an interface as a Python CGI with the Apache rewriting

RESTful Interface

Using cgi, sqlite3, xmlrpc libraries

RESTful Interface

ImageMagick Scripting

I have a Python script running unsupervised on the server

It responds to RPC requests

Python drives ImageMagick to build a visual representation

Also, did a target for a competition

ImageMagick Scripting

Using subprocess, os libraries

ImageMagick Scripting

Test Server Symbol Management

Each server has different resource locationsServer name is in the UNC

These are in an INI file

Python library to access this INI file

Caches the results in a pickle file to reduce network traffic and improve resiliency

Test Server Symbol Management

Rate Checker

Many rates go in each day at different times

Required to confirm the success of rates

Python compares today's rates with yesterday, a week ago and last month end

Reports via email

Emails are logged as proof of our diligence

Rate Checker

Using smtplib, adodb, configparser libraries

Rate Checker

Special Thanks to...

Guido Van Rossum keeping it simple and powerful

Mark Hammond Making it work on Windows

LibariesTkinter for easy UI

Openpyxl for Excel import and export

email and smtplib for sending email

Urllib and httplib for capture and parsing

Re for regular expressions

argparse, configparser and winreg for parameterisation

Subprocess for process management

And the Python community for sample code all over the internet

Go forth and code!

RedWaratah.comOpen Source Solutions for Business