Python for database access and Event driven programming in Python

But first, this week …• This is Computer Science

Education Week • Date chosen to honor Grace

Murray Hopper, whose birthday is December 9

• Rear Admiral, U.S. Navy• Computer Science Pioneer• Credited with the term

“debugging”• Developed the first compiler• Conceptualized the idea of a

machine independent programming language

• Explains nano seconds!

Database access in Python• A set of tutorials, including access to

MySQL and PostgreSQLhttp://python.about.com/od/pythonanddatabases/Connecting_to_Databases_With_Python.htm

• A brief introduction to SQLite3– Lightweight disk-based database– See http://docs.python.org/library/sqlite3.html

Example, using sqlite3• First, import sqlite3• Create an object that represents the database• Use standard sql• This example from Nakul Rathod

import sqlite3def main(): db = sqlite3.connect('test.db') # connect to the database or create the database if it doesn't exist db.row_factory = sqlite3.Row # the powerful Row Factory interface allows one to access information in the database like a dictionary db.execute('drop table if exists test') db.execute('create table test(t1 text, i1 int)’) # the database object accepts standard sql db.execute('insert into test (t1, i1) values(?,?)', ('one', 1)) # you can directly substitute the values in ?,? but variable names allow for code re-usability db.execute('insert into test (t1, i1) values(?,?)', ('two', 2)) db.execute('insert into test (t1, i1) values(?,?)', ('three', 3)) db.commit() # without a commit no changes to the database will be recorded cursor = db.execute('select * from test order by i1') for row in cursor: print row['i1'], row['t1'] # Each record is fully indexed by the field heading due to Row Factorymain()

Spot check• Create a database using sqlite3• Each row in the database contains the

following information (Make about 5 entries)– ID (alphanumeric)– Last name (alpha)– First name (alpha)– Hire year (int)

• Enter data in random order, then display all in order of hire year

Event driven programming• This is another way to think about flow

control in a program.• An important consideration is parallelism (or

pseudo-parallelism)– If there is only one processor, and we write the

program to support more than one event at once, then the one processor gives slices of time to each event.

– All modern computers have multi core architecture – more than one processor available. It is possible to have more than one event process happening at the same time.

So far• Most of our programs are basically

sequential, with excursions into functions and classes for special purposes

• There is a clearly defined sequence of actions that the program will perform.

Event Driven Programs• In event-driven programs, the sequence of

actions is not known in advance• The program has the facility to respond to

external events and take appropriate action.

• Now, programming means – determining what events might happen– recognizing each event– taking appropriate action for the event– waiting for the next

• sometimes accommodating simultaneous events

Error events• An event may be a special case of normal

programs – such as an error event• We have seen how to handle errors

– This is a form of interrupt handling

number = 0while not 1 <= number <= 10: try: number= int(raw_input('Enter number from 1 to 10: '))

if not 1 <= number <= 10:print 'Your number must be from

1 to 10:'except ValueError: print 'That is not a valid integer.'

Servers, GUIs• Some programs do nothing except

respond to events– Servers listen for a request and respond

• GUI interfaces present a pallet to the user– The operating system responds to the

choices made by the user

Wait loops and threads

• One way to implement event handlers is with an endless loop that checks for an indication of the event, then responds

• If there is only one process running, the event takes control and no other event can be serviced until that one is complete

• Threads allow multiple flows of execution simultaneously– Same processor– Multiple processors

Network programming

• Book introduces event driven programming in the context of drawing– Fine, but let’s look at networks

• A brief introduction to networks to start– Peer – to – Peer – Client-Server

Client-Server Computing• Much of our computing environment is

client-server based• Server:

– provides a service– waits for a request and does something to

satisfy the request• Client:

– The controlling side of the interaction– Connects to a server– Issues a request

• Server must be connected to a network• Server software waits for requests,

does some processing, sends responses

Server waits for requests

Locating a server• Web as our primary communication method to

servers• URL (or URI) names the server where a resource is

provided• Class web page is located at http://www.csc.villanova.edu/~cassel/8000/Syllabus.html

• http: the protocol• www.csc.villanova.edu: the domain• rest: the path to the resource• Domain Name Server (DNS) translates from the

domain name to the IP address of the machine.– Multi-level – edu, villanova, csc

Network connections• Each machine on the network has a location,

called an IP address– think of it like a phone number

• Many possible connection requests– Within the machine, a port identifies the

specific communication line• think of it like an extension in a phone system

– A client requests a connection on a particular port to reach the service of interest

• Now, messages flow between the client process and the desired server process

Protocols• How does the client request the desired

service? • A protocol is a standard set of messages

to exchange between cooperating entities to achieve a desired result– What protocols do you know?

Spot check• You know protocols• How do you introduce two people?

– Suppose person A is senior to person B• Can you think of any other standard

message exchanges?– Anyone remember Lawrence Welk –

• And a 1 and a 2 and a 3 …

Network protocols• There are many that we will not consider• For Client-server computing

– Server listens on the port (end point of the socket)

– Client directs a message to the known port– Message content tells the server what service

is wanted– Server responds with a standard message, in

an agreed format• Each side understands only the pre-defined

messages!

Python Socket classSyntax Semantics

s.connect(host, port) connects socket s to remote host at specified port.

s.send(data) sends data to connected remote host.

s.recv(maxNumChars)

receives a maxNumChars from the remote host.

s.close( ) Disconnects socket s from remote host.

Simple client• Get time from the time server at NIST• No need to send anything because the

server only does one thing. – As soon as the connection is requested, the

desired service is identified

>>> from socket import socket>>> connection = socket()>>> connection.connect(('time.nist.gov',13))>>> print connection.recv(1024)

55902 11-12-07 16:41:46 00 0 0 380.0 UTC(NIST) *

Process the returned data• We know the exact format of the data,

because that is part of the protocol• We can now process that, format it, do

anything we like with it.>>> from socket import socket>>> connection = socket()>>> server = 'time.nist.gov'>>> connection.connect((server,13))>>> fields= connection.recv(1024).split()>>> date = fields[1]>>> >>> time = fields[2]>>> print 'Date (YY-MM-DD) is %s, time is %s (UTC)'%(date,time)Date (YY-MM-DD) is 11-12-07, time is 16:46:50 (UTC)>>>

Spot check• Connect to the NIST time server• Print out the current date and time in

this format:It is now <time> on <date>

• How would you make the date be in the form– December <day>, 2011– (or, if you prefer) <day> December 2011

Terry Scott University of Northern Colorado 2007 Prentice Hall

24

recv Method• There are four possibilities for the

outcome of a recv command1. Receive string of length equal to maximum

length specified. May have further data that can be retrieved by another call to recv.

2. Receive less data than maximum. That is the end of that batch of data.

3. No data was sent so recv waits indefinitely.4. Empty string is received. Connection was

closed by sender or network was disrupted.

Client side• Nothing particularly event driven on the

client side• This is where the event originates• The event driven software is on the

server side

Writing a server – event driven programming

• A server is a basic event-driven program• Server

– listen on a particular port for incoming connections

– create a socket to manage each connection– follow the chosen protocol for

communication with the client• From the server side

– send means from server to client– recv means from client to server

A bit about network protocols• References to TCP

– Transmission control protocol– Connection-oriented protocol for

establishing and maintaining a connection between communicating entities on the Internet

• handles things like corrupted transmissions, dropped calls, etc.

TCPServer class• Hides all the details of establishing a

reliable connection and making sure all the data is delivered correctly, and in order.

• Customize by defining a class to handle each connection

A simple server • Echo server

– Receives a message from a client and sends the same message back.

• Actually useful for testing that a connection is working, and the round trip time for a message

# Example from Goldwasser bookfrom SocketServer import TCPServer, BaseRequestHandlerclass EchoHandler(BaseRequestHandler): def handle(self): message = self.request.recv(1024) self.request.send(message)

# may need to customize localhost and port for your machineechoServer = TCPServer( ('localhost', 9128), EchoHandler)echoServer.serve_forever()

The client side of echo• Connect to the server• Send the message• Get the message back

>>> from socket import socket>>> echo = socket()>>> echo.connect(('localhost',9128))>>> echo.send('This is a test')14>>> print echo.recv(1024)This is a test>>>

Spot check

• Get the echo server running on your machine.

• Make the echo client connect to the server and echo a message.

Basic web server• Similar to the echo server, except return

a requested file instead of echoing the message

• Overall outline– Accept the connection– Receive the file request– Search for the file in the path provided– If not there, return error message (404!)– If there, return the file

Web serverfrom SocketServer import TCPServer, BaseRequestHandler

class WebHandler(BaseRequestHandler): def handle(self): command = self.request.recv(1024) if command[:3] == 'GET': pagename = command.split()[1][1:] # remove leading '/' for filename try: requestedFile = file(pagename, 'r') content = requestedFile.read() requestedFile.close() header = 'HTTP/1.0 200 OK\r\n' header += 'Content-Length: %d\r\n\r\n' % len(content) self.request.send(header) self.request.send(content) except IOError: # could not open the file self.request.send('HTTP/1.0 404 Not Found\r\n\r\n')

webServer = TCPServer( ('localhost', 8080), WebHandler)webServer.serve_forever()

Very simple. Assume file in local directory

Note creation of file header information sent back to requestor with the file

Note: runs on local host, not accessible over the Internet

Spot check• If not now, then later – • Write the corresponding client to

request a file and receive it back.

Persistence• So far, connection made, something

happens, connection ends.• Server is then free to handle another

connection.• Client is disconnected entirely• Some applications require persistence –

continuing connection between client and server

Terry Scott University of Northern Colorado 2007 Prentice Hall

36

Case Study:Network Chat-Room

• Develop our own protocol for chat room.• Connection has permanence unlike previous

examples – persistent connection.• Client must monitor keyboard and also listen to

the socket. • Use multithreading to accomplish this. Without

multithreading can get hung-up on listening to socket or the keyboard.

• Following two slides show different messages sent from client to server and server to client.

Terry Scott University of Northern Colorado 2007 Prentice Hall

37

Client to Server MessagesMessage Type FormatJoin the room using given identity.

'ADD %s\n' % screenName

Broadcast the message to everyone.

'MESSAGE %s\n'%content

Send a private message.

'PRIVATE %s\n%s\n' %(recipient, content)

Quit the chat room.

'QUIT\n'

Terry Scott University of Northern Colorado 2007 Prentice Hall

38

Server to Client MessagesMessage Type Format

New user has joined. 'NEW %s\n' % screenname

Message was broadcast to everyone.

'MESSAGE %s\n%s\n' %(sender, content)

Private message was sent to user.

'PRIVATE %s\n%s\n' % (sender, content)

Someone has left the room.

'LEFT %s\n' % screenName

Acknowleges request to quit.

'GOODBYE\n'

Terry Scott University of Northern Colorado 2007 Prentice Hall

39

Chat Server• Uses ThreadingTCPServer rather than

TCPServer. Needed since can be multiple people in the chat room.

• _broadcast function used to send to clients in the chatroom.

• Each person that joins the chat-room is given a new socket that is accessed via a dictionary with the person screen name as the key.

• Code for the server is on the next three slides.

Terry Scott University of Northern Colorado 2007 Prentice Hall

40

Chat Server Codefrom SocketServer import ThreadingTCPServer,

BaseRequestHandler_socketLookup = dict()

def _broadcast(announcement):for connection in _socketLookup.values():

connection.send(announcement)

class ChatHandler(BaseRequestHandler): def handle(self):

username = 'Unknown'active = True

Terry Scott University of Northern Colorado 2007 Prentice Hall

41

Chat Server Code (continued)while active: transmission = self.request.recv(1024) if transmission: command = transmission.split()[0] data = transmission[1+len(command):] if command == 'ADD': username = data.strip()

_socketLookup[username] = self.request

_broadcast('NEW %s\n' % username) elif command == 'MESSAGE':

_broadcast('MESSAGE %s\n%s\n' % (username,data))

Terry Scott University of Northern Colorado 2007 Prentice Hall

42

Chat Server Code (continued) elif command == 'PRIVATE':

rcpt = data.split('\n')[0] if rcpt in _socketLookup:

content = data.split('\n')[1]_socketLookup[rcpt].send(

'PRIVATE %s\n%s\n' % (username,content)) elif command == 'QUIT':

active = FALSE self.request.send('GOODBYE\n')

else: active = False

self.request.close()_socketLookup.pop(username)_broadcast('LEFT %s\n' %username)

myServer=ThreadingTCPServer('localhost',9000), ChatHandler)myServer.serve_forever()

Sufficient unto the day• or the semester• Hopefully, we have accomplished a good

bit in the last four months• Python, object-oriented programming,• Some natural language processing

– as examples of language modules and algorithms

• A bit of networking even.