kamaelia europython tutorial
DESCRIPTION
This was the Kamaelia Tutorial at Europython. It goes from basics - ie building a mini-kamaelia from scratch, through to a file multicaster, through a video recording application all the walk through to a multiuser bulletin board system.TRANSCRIPT
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Kamaelia: Pragmatic Concurrency
A Tutorial
Michael SparksEuropython '09, Birmingham UK
http://www.kamaelia.org/PragmaticConcurrency [email protected]
About me• - Been using python for several years• - Always been interested in concurrency
• - Kamaelia aims to embody safe practices.
• - Work at BBC R&D's Northern Lab, based in Manchester, moving to Media City:UK in 2011
• - Kamaelia is born out of a variety of R&D projects, and shaped by needs, not aesthetics or purity.
• Kamaelia is not an active R&D project. It is used in R&D projects, and hence actively maintained.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
About me•
Disclaimer: Like you, I'm doing this on my time, not the BBC's. This doesn't represent BBC opinion on anything.
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
About Kamaelia• - Also born from a desire to make concurrency
in programs easier to work with• - Because it's nice in the shell – I want software
concurrency that easy :-)• - Expressiveness is favoured over performance,
but not to preclude optimisation• - Means we're cautious about adding syntactic
sugar.•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
About Kamaelia• - Adapts Unix Philosophy to make a program
concurrent internally, but with the purpose of simplifying maintenance.
• Unix Philosophy:• Write programs that do one thing and do it well.
Write programs to work together.Write programs to handle text streams, because that is a universal interface.
• --Doug McIlroy•
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
About Kamaelia• - Adapts Unix Philosophy to make a program
concurrent internally, but with the purpose of simplifying maintenance.
• Kamaelia Philosophy:• Write components that do one thing and do it well.
Write components to work together. Write components to handle python object streams, because that is a universal interface.
• --With apologies to Doug McIlroy•
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
What we're covering• - An overview of Kamaelia
• & it's view on concurrency• - How to build a mini Kamaelia
• – to get under the hood.• - Building components & systems• - Examining larger systems, and debugging.• - Building a large(ish) system
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Time Estimates• First part:• - An overview of Kamaelia (lightning talkstyle : 5-10 mins)• - How to build a mini Kamaelia (exercise 40-60 mins)• - Starting building components & systems (remainder)
After break:
• - More advanced stuff (demo/etc 20 mins)• - Larger systems and debugging. (20 mins)• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Format• I generally welcome questions at any point• That said...
• 'Except in the overview• - the rest of this morning should answer them!!
• Copious notes provided - covers more than today
• Materials available from URL below• Is a mixture of “explain then do”.
Perhaps hold questions for during “do” :-) ?
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Caveat•
• * First time I've given this tutorial !• * If we run short of time on any section, we'll
skip ahead.•
• * BUT, have copious notes that cover everything.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Part 1•
• First part:• - An overview of Kamaelia (lightning talk style : 5-10
mins)• - How to build a mini Kamaelia (exercise 40-60 mins)• - Starting building components & systems (remainder)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Kamaelia, a 20:20 overview
20:20 PresentationA presentation style based on the “Pecha Kucha”style 20 slides, 20 seconds each(Similar to a lightning talk)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Hardware finally going massively concurrent ...
.... PS3, high end servers, trickling down to desktops, laptops)
Why?
“And one language to inthe darkness bind them”... can just we REALLY abandon 50 years of code for Erlang, Haskelland occam?
Opportunity!
Problems
“many hands make lightwork” but Viewed as Hard... do we just have crap tools?
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Missing Something?Fundamental Control Structures... in imperative languages number greater than 3! (despite what you get taught...!)
Control Structure Traditional Abstraction Biggest Pain Points
Sequence Function Global VarSelection Function Global VarIteration Function Global VarParallel Thread Shared Data
Usually Skipped Lost or duplicate update are most common bugs
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Regarding using concurrency, what sort of applications are we
talking about here?
Desktop
APPS
Network
Novice
3rd Party
Media
gsoc
trainee
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Desktop
APPS
Network
Novice
3rd Party
P2P Whiteboard
ER DB Modeller Kids
Programming(logo) Simple
Games
Speak 'n Write
UGCBackend
Transcoder
Think backendneeded foryoutube/flickrtype systems
Media
Compose
Shot ChangeDetection
MobileReframing
DVB
Macro“record
everything”
Podcasts
Email &SpamSMTP
GreylistingPop3Proxy ClientSide
Spam Tools
IRCWeb
Serving
gsoc
trainee
AIM
AWS(Amazon)
SednaXMLDBXMPP
pubsubQt
Gtkmicroblogging
MiniAxonScriptReaderMediaPreviewon MobileReliableMulticast
P2P RadioTorrent3D SystemsRealtime MusicPaint AppP2P Web ServerSecure “phone”Social Net Vis...
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Core Approach: Concurrent things with comms points Generally send messages Keep data private, don't share
inbox
control
...
outbox
signal
...
inbox
control
...
outbox
signal
...
http://www.kamaelia.org/PragmaticConcurrency [email protected]
But I must share data?Use Software Transactional Memoryie version control for variables.
1. Check out the collection of values you wish to work on2. Change them3. Check them back in 4. If conflict/clash, go back to 1
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Perspectives in APIs! (1/2)
inbox
control
...
outbox
signal
...
1st Person - I change my state
2nd Person – YOUwant to me to dosomething(you sendme a message)
3rd Person –Bla shoulddo something(I send a message)
If you have concurrency it becomes natural to think in terms of 1st 2nd and 3rd person. This affects an API's structure, and can be vital for understanding it!
This is one we've found that makes sense
http://www.kamaelia.org/PragmaticConcurrency [email protected]
inbox
control
...
outbox
signal
...
private real methods
Messagesfrom publicinboxes
Messages sentto public outboxes
Perspectives in APIs! (2/2)
Also, thinkabout stdin
Also, thinkabout stdout
If you have concurrency it becomes natural to think in terms of 1st 2nd and 3rd person. This affects an API's structure, and can be vital for understanding it!
This is one we've found that makes sense
http://www.kamaelia.org/PragmaticConcurrency [email protected]
inbox
control
...
private real methods
Messagesfrom publicinboxes
Actor SystemsDistinction can be unclear,potential source of ambiguity*
No outbox conceptPossible issues withrate limiting*Hardcodes recipientin the sender
*system dependent issue
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Advantages of outboxes
inbox
control
...
outbox
signal
...
No hardcoding of recipientallows: - Late Binding - Dynamic rewiring
Concurrency Patterns as Reusable Code ... a concurrency DSL
http://www.kamaelia.org/PragmaticConcurrency [email protected]
A Core Concurrency DSLPipeline(A,B,C)Graphline(A=A,B=B, C=C, linkages = {})Tpipe(cond, C)Seq(A,B,C), PAR(), ALT()Backplane(“name”), PublishTo(“name”), SubscribeTo(“name”)Carousel(...)PureTransformer(...)StatefulTransformer(...)PureServer(...)MessageDemuxer(...)Source(*messages)NullSink
Some of these are work in progress – they've been identified as useful, but not implemented as chassis, yet
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Pipeline ExamplePipeline( MyGamesEventsComponent(up="p", down="l", left="a", right="s"), BasicSprite("cat.png", name = "cat", border=40),).activate() MyGames
EventsComponent
BasicSprite
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Graphline ExampleGraphline( NEXT = Button(...), PREVIOUS = Button(...), FIRST = Button(...), LAST = Button(...), CHOOSER = Chooser(...), IMAGE = Image(...), ...).run()
FIRST(button)
Chooser
LAST(button)
PREVIOUS(button)
NEXT(button)
Image
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Server Example
MainServer Core
Remote User
Protocol Handler Factory
Socket handler
Protocol handler
datato
user
datafromuser
Created at runtime to handle the connection
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Server Example
MainServer Core
Protocol Handler Factory
You therefore need to provide this bit.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Server Examplefrom Kamaelia.Chassis.ConnectedServer import ServerCorefrom Kamaelia.Util.PureTransformer import PureTransformer
def greeter(*argv, **argd): return PureTransformer(lambda x: "hello" +x)
class GreeterServer(ServerCore): protocol=greeter port=1601
GreeterServer().run()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Backplane Example# Streaming Server for raw DVB of Radio 1 Backplane(“Radio”).activate()
Pipeline( DVB_Multiplex(850.16, [6210], feparams), # RADIO ONE PublishTo("RADIO"),).activate()
def radio(*argv,**argd): return SubscribeTo(“RADIO”)
ServerCore(protocol=radio, port=1600).run()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
So that's the 5 minute version
Short Q&A before we move on?
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Part 1•
• First part:• - An overview of Kamaelia (lightning talk style : 5-10
mins)• - How to build a mini Kamaelia (exercise 40-60 mins)• - Starting building components & systems (remainder)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon• Kamaelia is divided into two halves• * One part handles all the concurrency
stuff, providing you a component model• * The other is a large collection of
components and some apps using them.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon
• Axon is the part that handles concurrency and provides the component model, and is the key to understanding why & how Kamaelia works.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon
• ... is a collection of exercises where you build just such a beast.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon•
• Generators* Python's smallest unit of concurrency
• Microprocesses* Generators with context
• Scheduler* Something to run lots of microprocesses
• Components* A microprocess with boxes (treated as in & outboxes)
• Postman* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon•
• Generators* Python's smallest unit of concurrency
• Microprocesses* Generators with context
• Scheduler* Something to run lots of microprocesses
• Components* A microprocess with boxes (treated as in & outboxes)
• Postman* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Generators• * Python's smallest unit of concurrency • * Single function you call, get a
generator back• * Can then call it's .next() method to
* Get a new value from it * Give it CPU time
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Fibonacci Generator• def fib(a,b):
while 1: yield a a, b = b, b + a
•
• Demo
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Fibonacci Generator• >>> def fib(a,b):
... while 1:
... yield a
... a, b = b, b + a
...>>> g = fib(1,1)>>> g<generator object at 0xb7bf9c6c>>>> [ g.next() for _ in range(10) ][1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Lots of Generators• Using the same fib generator, make 10 of them:
>>> GS = [ fib(x,x) for x in range(10) ]
• And “run” them:>>> [ G.next() for G in GS ][0, 1, 2, 3, 4, 5, 6, 7, 8, 9]>>> [ G.next() for G in GS ][0, 1, 2, 3, 4, 5, 6, 7, 8, 9]>>> [ G.next() for G in GS ][0, 2, 4, 6, 8, 10, 12, 14, 16, 18]>>> [ G.next() for G in GS ][0, 3, 6, 9, 12, 15, 18, 21, 24, 27]>>> [ G.next() for G in GS ][0, 5, 10, 15, 20, 25, 30, 35, 40, 45]
•
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Generators as co-routines• def fib(a,b):
while 1: yield 1 # Just to say “keep running me” print a a, b = b, b + a
•
• Demo
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Generators as co-routines• def printer(tag):
while 1: yield 1 # Makes it a generator print tag
•
• Demo
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon•
• Generators* Python's smallest unit of concurrency
• Microprocesses* Generators with context
• Scheduler* Something to run lots of microprocesses
• Components* A microprocess with boxes (treated as in & outboxes)
• Postman* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Microprocesses• * Generators with context•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
microprocess exercise• Write a class microprocess with methods:
• __init__(self)* Takes no arguments
* Uses super to call superclass __init__ method• main(self)
* No arguments * Should yield 1
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
microprocess answer•
• class microprocess(object): def __init__(self): super(microprocess, self).__init__() def main(self): yield 1
•
Generally, we'll skip answers in this presentation, they're all in the web version of this tutorial here:
• http://www.kamaelia.org/MiniAxon/
• (and in the notes :-)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
microprocess usage• class printer(microprocess):
def __init__(self, tag): super(printer, self).__init__() self.tag = tag def main(self): while 1: yield 1 print self.tag
•
• “Printer” isn't particularly interesting, but allows things like components, but let's see how this can be used.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
microprocess usage 2• >>> X = printer("Something")
>>> G = X.main()>>> X,G(<__main__.printer object at 0xb7bfd2ac>, <generator object at 0xb7bfd4cc>)>>> G.next()1>>> G.next()Something1>>> X.tag = "Something else">>> G.next()Something else1
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon•
• Generators* Python's smallest unit of concurrency
• Microprocesses* Generators with context
• Scheduler* Something to run lots of microprocesses
• Components* A microprocess with boxes (treated as in & outboxes)
• Postman* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Scheduler•
* Something to run lots of microprocesses•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
scheduler exercise• Write a class scheduler with 3 methods:
• __init__(self)* Uses super to call superclass __init__ method
* subclasses microprocess * Creates 2 queues – active & newqueue
• main(self)* Is a generator
* Runs the microprocesses activated• activateMicroprocess(self, someprocess): * Calls somprocess.main() * Adds generator to newqueue
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
scheduler exercise• Scheduler logic
• main(self)* loops 100 times, yields 1 at start of loop
* loop through generators in self.active * call their .next() method * If result is not -1 and no StopIteration, append to newqueue * at end of loop, newqueue becomes active * newqueue reset to empty list
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
scheduler usage• Using same printer class..
•
>>> X = printer("Hello World")>>> Y = printer("Game Over") # :-)
• >>> myscheduler = scheduler()>>> myscheduler.activateMicroprocess(X)>>> myscheduler.activateMicroprocess(Y)
• >>> for _ in myscheduler.main():... pass
• <prints Hello world/Game over repeatedly>
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon•
• Generators* Python's smallest unit of concurrency
• Microprocesses* Generators with context
• Scheduler* Something to run lots of microprocesses
• Components* A microprocess with boxes (treated as in & outboxes)
• Postman* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Component•
• * microprocess with a standard interface.• boxes as inboxes/outboxes
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
component exercise• Write a class component subclass of microprocess with 4 methods:
• __init__(self)
send(self, value, outboxname)
recv(self, inboxname)
dataReady(self, inboxname)
• Behaviour coming up!
http://www.kamaelia.org/PragmaticConcurrency [email protected]
component exercise 1/4• __init__(self) logic:
* Add an attribute self.boxes , default value:
{ “inbox”: [], “outbox”: [] }• * Ensure you call the superclass
__init__ method appropriately
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
component exercise 2/4• send(self,value, outboxname) logic:
Finds the list named outboxname in self.boxes, and appends value to it.
• Before: X.send(“hello”, “outbox”) { “inbox”: [], “outbox”: [] }
• After:
{ “inbox”: [], “outbox”: [“hello”] }•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
component exercise 3/4• recv(self, inboxname) logic:
Finds the list named inboxname in self.boxes, and pops the first value
• Given:
{ “inbox”: [“hello”, “world”], “outbox”: [] }• self.recv(“inbox”) returns, “hello” leaving...
{ “inbox”: [“world”], “outbox”: [] }•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
component exercise 4/4•
dataReady(self, inboxname) logic: Finds the list named inboxname in self.boxes: returns the length of the list.
• Given:
{ “inbox”: [“hello”, “world”], “outbox”: [] }• dataReady(“inbox”) --> 2
(allows if self.dataReady(“inbox”) )
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
component usage•
Until we add a means for data to get from outboxes to inboxes, using components is no more interesting than microprocesses
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon•
• Generators* Python's smallest unit of concurrency
• Microprocesses* Generators with context
• Scheduler* Something to run lots of microprocesses
• Components* A microprocess with boxes (treated as in & outboxes)
• Postman* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Postman• * Something to do deliveries•
• Note: This is more conceptual in real Axon,
but was real in v. old Axon. ie real Axon is more efficient!
http://www.kamaelia.org/PragmaticConcurrency [email protected]
postman exercise• Write a class postman subclass of microprocess
with 2 methods:• __init__(self, source, sourcebox, sink, sinkbox)
main(self)
• Behaviour coming up!
http://www.kamaelia.org/PragmaticConcurrency [email protected]
postman exercise 1/4•
__init__(self, source, sourcebox,
sink, sinkbox) logic: * Copy all the arguments as local attributes
• * Ensure you call the superclass __init__ method
appropriately!
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
postman exercise 2/4•
main(self) logic:
• In a loop:• yield a non -1 value (eg 1)
Check if source's sourcebox has dataReady
• If it has, use recv to collect it from there, and sink's send method to deliver it.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Producer/Consumer 1/3•
• class Producer(component): def __init__(self, message): super(Producer, self).__init__() self.message = message
• def main(self): while 1: yield 1 self.send(self.message, "outbox")
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Producer/Consumer 2/3•
• class Consumer(component): def main(self): count = 0 while 1: yield 1 count += 1 if self.dataReady("inbox"): data = self.recv("inbox") print data, count
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Producer/Consumer 3/3•
• p = Producer("Hello World")c = Consumer()postie = postman(p, "outbox", c, "inbox")
• myscheduler = scheduler()myscheduler.activateMicroprocess(p)myscheduler.activateMicroprocess(c)myscheduler.activateMicroprocess(postie)
• for _ in myscheduler.main(): pass
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Producer/Consumer Output•
• Hello World 2Hello World 3Hello World 4
• ...
• Hello World 96Hello World 97Hello World 98
•
• Not 100, because of yields before start of scheduler loop. (scheduler terminates early)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon•
• Using it for more useful stuff
http://www.kamaelia.org/PragmaticConcurrency [email protected]
File Reader•
• class FileReader(component):
• def __init__(self, filename): super(FileReader, self).__init__() self.file = open(filename, "rb",0)
• def main(self): yield 1 for line in self.file.xreadlines(): self.send(line, "outbox") yield 1
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Multicast sender 1/2• class Multicast_sender(component):
def __init__(self, laddr, lport, daddr, dport): super(Multicast_sender, self).__init__() self.laddr = laddr self.lport = lport self.daddr = daddr self.dport = dport
•
... continued
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Multicast sender 2/2• class Multicast_sender(component):
... continued
def main(self): sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP) sock.bind((self.laddr,self.lport)) sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 10) while 1: if self.dataReady("inbox"): data = self.recv("inbox") l = sock.sendto(data, (self.daddr,self.dport) ) yield 1
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
FileReader -> MulticastSender•
• reader = FileReader("Ulysses")sender = Multicast_sender("0.0.0.0", 0, "224.168.2.9", 1600)postie = Postman(reader, "outbox", sender, "inbox")
•
• myscheduler = scheduler()myscheduler.activateMicroprocess(reader)myscheduler.activateMicroprocess(sender)myscheduler.activateMicroprocess(postie)
• for _ in myscheduler.main(): pass
•
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Mini Axon --> Axon•
• The rest is:Syntactic Sugar
Ways of using it Optimisations (eg direct delivery – no postman) + a couple of other ideas we'll come to (STM, threadedcomponents, services, processcomponents)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Questions?•
• ... before we look how to build some real components, and how to use them?
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Part 1•
• First part:• - An overview of Kamaelia (lightning talk style : 5-10
mins)• - How to build a mini Kamaelia (exercise 40-60 mins)• - Starting building components & systems (remainder)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Building a Video Recorder•
• Before the break, we'll build a simple video recorder.• Approach:• 1 Build a webcam• 2 Clean up the code for “normal” reuse• 3 Componentise in least effort manner• 4 Separate input/transform/display parts• 5 Hook webcam up to a dirac encoder and filewriter•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 1: Build a simple Webcam• Pygame 1.9.1 alpha has basic Linux webcam support
which works nicely, so we're using that.• First some initialisation
• import pygameimport pygame.camera
• pygame.init()pygame.camera.init()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 1: Build a simple Webcam• The some definitions:
• displaysize = (800, 600)capturesize = (640, 480)imagesize = (352, 288)imageorigin = (0,0)device = “/dev/video0”
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 1: Build a simple Webcam• Initialise the display, allocate a camera, and activate it
• display = pygame.display.set_mode(displaysize)camera = pygame.camera.Camera(device, capturesize)camera.start()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 1: Build a simple Webcam• Loop round capturing images and display them
• while 1: snapshot = camera.get_image() snapshot = pygame.transform.scale(snapshot, imagesize) display.blit(snapshot, imageorigin) pygame.display.flip()
•
• Demo!
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 2: Clean up for reuse• Given a collection of config options, wrapping this in a
class makes sense.• As before these parts are unchanged:
• import pygameimport pygame.camera
• pygame.init()pygame.camera.init()
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 2: Clean up for reuse• We then define a class, and put the config options as
class attributes:• class VideoCapturePlayer(object):• displaysize = (800, 600) capturesize = (640, 480) imagesize = (352, 288) imageorigin = (0,0) device = “/dev/video0”
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 2: Clean up for reuse• Put our initialisation in the __init__. Allow the user to
override our defaults
def __init__(self, **argd): self.__dict__.update(**argd) super(VideoCapturePlayer,self).__init__() self.display = pygame.display.set_mode(self.displaysize) self.camera = pygame.camera.Camera(self.device, self.capturesize) self.camera.start()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 2: Clean up for reuse• Wrap up body of loop in a method
def get_and_flip(self): snapshot = self.camera.get_image() snapshot = pygame.transform.scale(snapshot, self.imagesize) self.display.blit(snapshot, self.imageorigin) pygame.display.flip()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 2: Clean up for reuse• Provide a hook to start it going, wrapping the main loop.
def main(self): while 1: self.get_and_flip()
• Then run it!
VideoCapturePlayer().main()
•
• Demo!
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 3: Componentise•
• In this case, least effort approach is to use a threaded component. Would could make it a generator component later if needed.
• Changes:• * imports
* class's baseclass / extraction of display_flip into a method. (to simplify later integration with existing components)* how we run it.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 3: Componentise•
• Imports change to
import pygameimport pygame.camerafrom Axon.ThreadedComponent import threadedcomponent
• pygame.init()pygame.camera.init()
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 3: Componentise•
• Baseclass changes & initialiser changes:
class VideoCapturePlayer(threadedcomponent):... def __init__(self, **argd): # no longer update __dict__ here super(VideoCapturePlayer,self).__init__(**argd)...
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 3: Componentise•
• Extract display flipping out to a separate method:
def pygame_display_flip(self): pygame.display.flip()
def get_and_flip(self): snapshot = self.camera.get_image() snapshot = pygame.transform.scale(snapshot, self.imagesize) self.display.blit(snapshot, self.imageorigin) self.pygame_display_flip()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 3: Componentise•
• Change to how we run:
• We had:• VideoCapturePlayer().main()
• We now:• VideoCapturePlayer().run()
•
• Demo!
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 4: Split into input & output•
• Now we can split the component in two:• * VideoCaptureSource• * Surface Displayer
• The Video capture source now needs to be self regulating,so it needs to sleep during the loop, relative to a target frame rate.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 4: Split into input & output•
• VideoCaptureSource class preamble:• import timeclass VideoCaptureSource(threadedcomponent): capturesize = (352, 288) delay = 0 fps = -1 device = “/dev/video0”
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 4: Split into input & output•
• VideoCaptureSource initialiser• def __init__(self, **argd): super(VideoCaptureSource, self).__init__(**argd) self.camera = pygame.camera.Camera(self.device, self.capturesize) self.camera.start() if self.fps != -1: self.delay = 1.0/self.fps self.snapshot = None
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 4: Split into input & output•
• Main loop body now just captures images, capturing a reference
• def capture_one(self):• self.snapshot = None self.snapshot = self.camera.get_image()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 4: Split into input & output•
• Main loop still runs it, and sends the image out an outbox, and then sleeps.
• def main(self):• while 1: self.capture_one() self.send(self.snapshot, “outbox”) time.sleep(self.delay)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 4: Split into input & output•
• The surface displayer takes the other code chunks• from Axon.Component import component• class SurfaceDisplayer(component): displaysize = (800,600) imagesize = (352, 288) imageorigin = (0,0)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 4: Split into input & output•
• Has it's own initialiser...• def __init__(self, **argd):
super(SurfaceDisplayer, self).__init__(**argd) self.display = pygame.display.set_mode(self.displaysize)
•
Retains the following method unchanged:• def pygame_display_flip(self):
pygame.display.slip()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 4: Split into input & output• Then mainbody waits for surfaces to display, sleeping
when there isn't any:• def main(self):
• while 1: while self.dataReady(“inbox”): snapshot = self.recv(“inbox”) snapshot = pygame.transform.scale(snapshot, self.imagesize) self.display.blit(snapshot, self.imageorigin)
• while not self.anyReady(): self.pause() yield 1
• yield 1
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 4: Split into input & output•
• We then join these back together in a pipeline:• from Kamaelia.Chassis.Pipeline import Pipeline
• Pipeline(
• VideoCaptureSource(),
• SurfaceDisplayer(),
• ).run()
• Demo!
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 5: Recording• Working back...•
• We want to write to a file... (SimpleWriter)• We want to write dirac encoded video DiracEncoder• That expects YUV Frames (ToYUV420_planar)• That expects RGB data + metadata
PureTransformation of somr RGB data• Which needs a picture source
• Which is where we started.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 5: Recording• Our imports therefore need to add:•
• from Kamaelia.File.Writing import SimpleFileWriter• from Kamaelia.Codec.Dirac import DiracEncoder• from Kamaelia.Video.PixFormatConversion import ToYUV420_planar• from Kamaelia.Util.PureTransformer import PureTransformer•
• And we delete everything related to display
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 5: Recording• We then need to join it all together:• Pipeline(
• VideoCaptureSource(), PureTransformer(lambda F : \ {"rgb" : pygame.image.tostring(F, "RGB"), "size" : (352, 288), "pixformat" : "RGB_interleaved", }), ToYUV420_planar(), DiracEncoder(preset="CIF", encParams={"num_L1":0}), SimpleFileWriter("X.drc"),
• ).run()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Step 5: Recording•
• This then records dirac encoded video, which we can now playback with a simple dirac player!
•
• Demo !•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Questions?• ... before we break?•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
End of Part 1•
• Before the second half, download and install Kamaelia from the link below, if you haven't already.
•
• http://tinyurl.com/kot49x •
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Kamaelia: Pragmatic Concurrency
A Tutorial
Michael SparksEuropython '09, Birmingham UK
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Part 2•
• Second part:• - More advanced stuff (demo/etc 20 mins)• - Larger systems, embedding and debugging. (20 mins)• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Questions?• ... before we carry on?•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Part 2•
• Second part:• - More advanced stuff (demo/etc 20 mins)• - Larger systems and debugging. (20 mins)• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
More advanced Stuff• Have seen how to build a basic component and a basic
system - covers a wide set of problems. Now we broaden the scope.
•
• If we have time we'll come back to Kamaelia's STM model & concepts of services, and (experimental) multiple process support.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Well behaved Shutdown• Components should expect to receive these messages on their
“control” inbox, and behave as follows:
• Axon.Ipc.shutdownMicroprocess – if you receive this, you are expected to shutdown immediately, and to pass this message on.
• Axon.Ipc.producerFinished – if you receive this, someone sending you data has shutdown. You may want to shutdown depending on your application's logic. You may process all outstanding messages in this case. You may wish to pass this message on, or change it to shutdownMicroprocess if it was unexpected.
• Due to different component needs no syntactic sugar exists for this, but common cases are being discussed at present.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Components• Inheritable default values – we've already seen this!• class VideoCapturePlayer(threadedcomponent):
displaysize = (800, 600) capturesize = (640, 480) imagesize = (352, 288) imageorigin = (0,0) device = "/dev/video0"
• Also usable:• VideoCapturePlayer(device = "/dev/video0")
VideoCapturePlayer(device = "/dev/video1")
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Components• Inheritable default values – other places:
class ServerCore(component): port=1601 protocol=None socketOptions=None TCPS=TCPServer ....
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Components• Inheritable default values:
class MailServer(ServerCore): port=25 protocol=SMTPProtocol
• Usage:• MailServer().run()
MailServer(port=8025).run()MailServer(socketOptions=...).run()
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Components• Inheritable default values - nests
class MyReconfiguredMailServer(MailServer): port=25 protocol=SMTPProtocol socketOptions = socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) class TCPS(TCPServer): CSA=MyDebugWrapper(ConnectedSocketAdapter)
• Allows for deep reconfiguration of components, without rewriting.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Components• Inheritable default values – real usage – Kamaelia Grey
class GreylistServer(MoreComplexServer): logfile = config["greylist_log"] debuglogfile = config["greylist_debuglog"] socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) port = config["port"] class TCPS(TCPServer): CSA = NoActivityTimeout(ConnectedSocketAdapter, timeout=config["inactivity_timeout"], debug=False) class protocol(GreyListingPolicy): servername = config["servername"] serverid = config["serverid"] smtp_ip = config["smtp_ip"] smtp_port = config["smtp_port"] allowed_senders = config["allowed_senders"] allowed_sender_nets = config["allowed_sender_nets"] ...
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Pipelines• Seen how Pipelines can join components together.• However also can do:
• Pipeline( Lsystem(), ConsoleEchoer(tag="\n", forwarder=True), Damage(),
circular = True,).run()
• To enable a feedback loop. (Demo)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Pipelines• Transcode Engine• Pipeline(
DirectoryWatcher(watch = conf["main_incoming_queue"]), ImageMover(destdir = conf["image_queue"]),).activate()
• Pipeline( DirectoryWatcher(watch = conf["image_queue"]), ImageTranscoder(destdir = conf["image_moderation_queue"]),).activate()
• Pipeline( DirectoryWatcher(watch = conf["main_incoming_queue"]), VideoMover(destdir = conf["video_queue"]),).activate()
• Pipeline( DirectoryWatcher(watch = conf["video_queue"]), VideoTranscoder(destdir = conf["video_moderation_queue"]),).run()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Pipelines• Reusable Dirac Video Source:• def DiracWebCam():
return Pipeline( VideoCaptureSource(), PureTransformer(lambda F : \ {"rgb" : pygame.image.tostring(F, "RGB"), "size" : (352, 288), "pixformat" : "RGB_interleaved", }), ToYUV420_planar(), DiracEncoder(preset="CIF", encParams={"num_L1":0}), )
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Graphlines• Graphlines are like pipelines, but any shape. For
example, a simple presentation tool:• Graphline(
CHOOSER = Chooser(items = files), IMAGE = Image(size=(800,600), position=(8,48)), NEXT = Button(caption="Next", msg="NEXT", position=(72,8)), PREVIOUS = Button(caption="Previous", msg="PREV", position=(8,8)), FIRST = Button(caption="First", msg="FIRST",position=(256,8)), LAST = Button(caption="Last", msg="LAST",position=(320,8)), linkages = { ("NEXT","outbox") : ("CHOOSER","inbox"), ("PREVIOUS","outbox") : ("CHOOSER","inbox"), ("FIRST","outbox") : ("CHOOSER","inbox"), ("LAST","outbox") : ("CHOOSER","inbox"), ("CHOOSER","outbox") : ("IMAGE","inbox"), }).run()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Graphlines•
• * Swiss Army Knife of Kamaelia components• * Allows almost any topology• - BUT an outbox may only link to one inbox• - many outbox may link to one inbox• * Can contain any component – including pipelines and
graphlines•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Graphlines• SMS Outbound processor• Graphline(
SMS_SOURCE = Pipeline( DirectoryWatcher(watch = "outgoingsms"), FilesToProcessSource(), FileSlurper(tuplemode=True), ), CLEANER = Pipeline( PureTransformer(lambda (x,y): y ), Mover(), ), SENDER = Pipeline( PureTransformer(lambda (x,y): cjson.decode(x) ), SMSSender(), ), SPLIT = TwoWaySplitter(),
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Graphlines• SMS Outbound processor (cont)• Linkages = {
("SMS_SOURCE","outbox") : ("SPLIT","inbox"), ("SPLIT", "outbox") : ("CLEANER", "inbox"), ("SPLIT", "outbox2") : ("SENDER", "inbox"), }).run()
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
• Linkages to or from “self” (or “”) link to the Graphline's boxes.• Graphline(
SPLIT = SomeSplitter() MERGE = SomeMerger() P1 = SomeTransformer() P2 = SomeTransformer() P3 = SomeTransformer() linkages { (“self”, “inbox”) : (“SPLIT”, “inbox”), (“MERGE”, “outbox”) : (“self”, “outbox”), ... }
• Allows graphlines to nest, and higher level abstractions
Graphlines
http://www.kamaelia.org/PragmaticConcurrency [email protected]
• Whiteboard using nesting graphlines:• Pipeline(
chunks_to_lines(), lines_to_tokenlists(), Graphline( ROUTER = Router( ((lambda T : T[0]=="SOUND"), "audio"), ((lambda T : T[0]!="SOUND"), "whiteboard"), ), WHITEBOARD = FilteringPubsubBackplane(whiteboardBackplane), AUDIO = Pipeline( SimpleDetupler(1), # remove 'SOUND' tag SpeexDecode(3), FilteringPubsubBackplane(audioBackplane, dontRemoveTag=True), RawAudioMixer(), SpeexEncode(3), Entuple(prefix=["SOUND"],postfix=[]), ),
•
Graphlines
http://www.kamaelia.org/PragmaticConcurrency [email protected]
• Whiteboard using nesting graphlines:• Linkages = {
# incoming messages go to a router ("", "inbox") : ("ROUTER", "inbox"),
• # distribute messages to appropriate destinations ("ROUTER", "audio") : ("AUDIO", "inbox"), ("ROUTER", "whiteboard") : ("WHITEBOARD", "inbox"),
• # aggregate all output ("AUDIO", "outbox") : ("", "outbox"), ("WHITEBOARD", "outbox") : ("", "outbox"),
• # shutdown routing ("", "control") : ("ROUTER", "control"), ("ROUTER", "signal") : ("AUDIO", "control"), ("AUDIO", "signal") : ("WHITEBOARD", "control"), ("WHITEBOARD", "signal") : ("", "signal") }, ), tokenlists_to_lines(),)
Graphlines
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Backplanes•
• * For where you want 1 to many or many to many• * Declare a backplane• * Components can publish data to it• * Components may subscribe to it.• * Subscribers get a copy of all data sent•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Backplanes• Can be useful for updating a central state.•
• For example players locations can be posted here:• Backplane("PLAYERS").activate()
•
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Backplanes• Players Post their updates like this:• Pipeline(
• MyGamesEventsComponent(up="p", down="l", left="a", right="s"),
• BasicSprite("cat.png", name = "cat", border=40),
• PureTransformer(lambda x: ("Cat ", x)),
• PublishTo("PLAYERS"),
• ).activate()
•
•
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Backplanes• The system can analyse their updates like this:• Pipeline(
• SubscribeTo("PLAYERS"),
• PlayerAnalyser(),
• Distancer(),
• ConsoleEchoer(),
• ).activate()
•
•
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Backplanes & Servers• When combined with a server, you instantly get a pub/
sub capable server, or splitter, or merger.• Backplane(“SPLIT”).activate()
Pipeline( DiracWebCam(), PublishTo(“SPLIT”),).activate()
def VideoProtocol(**argd): return SubscribeTo(“SPLIT”)
• ServerCore(protocol=VideoProtocol, port=1700).run()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Backplanes & Servers• When combined with a server, you instantly get a pub/
sub capable server, or splitter, or merger.• Backplane(“CHAT”).activate()
def ChatProtocol(**argd): return Pipelines( SubscribeTo(“CHAT”) NullComponent(), PublishTo(“CHAT”) )
• ServerCore(protocol=ChatProtocol, port=1700).run()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
PAR & Seq• PAR & Seq are wrapper components with concepts
nabbed from Occam. (hence their names)• * Seq runs each component it's given, one after
another, wiring up inboxes/outboxes such that each component handles it. Useful in staged protocols (later)
• * PAR runs all the components concurrently. Their output is merged. Shutdown messages sent to PAR are forwarded to all subcomponents – making it useful for managing system shutdown.
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Using PAR• PAR can be used to simplify some systems:• Pipeline(
PAR( Button(caption="Next", msg="NEXT", position=(72,8)), Button(caption="Previous", msg="PREV", position=(8,8)), Button(caption="First", msg="FIRST" ,position=(256,8)), Button(caption="Last", msg="LAST", position=(320,8)), ),
• Chooser(items = files), Image(size=(800,600), position=(8,48)),).run()
• This is the same slideshow as the previous Graphline...
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Using PAR• ... to run & shutdown multiple subsystems• Pipeline(
timedShutdown(TTL=15), PAR( Pipeline( ReadFileAdaptor(file, readmode="bitrate", bitrate = 300000*8/5), DiracDecoder(), MessageRateLimit(framerate), VideoOverlay(position=(260,48), size=(200,300)), ), Pipeline( ReadFileAdaptor(file, readmode="bitrate", bitrate = 2280960*8), DiracDecoder(), ToRGB_interleaved(), VideoSurface(size=(200, 300), position=(600,48)), ),
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Using PAR• ... to run & shutdown multiple subsystems• Pipeline(
PAR(Button(caption="Next", msg="NEXT", position=(72,8)), Button(caption="Previous", msg="PREV", position=(8,8)), Button(caption="First", msg="FIRST" ,position=(256,8)), Button(caption="Last", msg="LAST", position=(320,8)), ), Chooser(items = files), Image(size=(200,300), position=(8,48), maxpect=(200,300)), ), Pipeline( Textbox(size=(200,300), position=(8,360)), TextDisplayer(size=(200,300), position=(228,360)), ), Ticker(size=(200,300), position=(450,360)), ),).run()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Using Seq• An example from the mobile reframer:• Seq( "Decoding & separating frames...",
Graphline( MAXF = DetermineMaxFrameNumber(edlfile), DO = Carousel( ... ), STOP = TriggeredOneShot(""),... ), "Processing edits...", Graphline( REFRAMING = ReframeVideo(edlfile... SOUND = PassThroughAudio(edlfile... ENCODING = ReEncode(outFileName...... ), "Cleaning up...", StopSelector(), ).run()
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Using Seq• An example of passing state in a protocol handler:• def CompositeBulletinBoardProtocol(**argd):
ConnectionInfo = {} ConnectionInfo.update(argd) return Seq( Authenticator(State = ConnectionInfo), UserRetriever(State = ConnectionInfo), MessageBoardUI(State = ConnectionInfo), StateSaverLogout(State = ConnectionInfo), )
ServerCore(protocol=CompositeBulletinBoardProtocol, ...)
• Worth noting the similarity between this and wsgi.(Except Seq can contain graphlines, pipelines, etc too)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Part 2•
• Second part:• - More advanced stuff (demo/etc 20 mins)• - Larger systems and debugging. (20 mins)• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Larger Systems & Debugging•
Large systems which are long running can develop bugs which are awkward to debug.
• Kamaelia has a collection of tools you can use which we'll walk through / demonstrate next.
•
• Practical example.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Larger Systems & Debugging•
Key Tools:• * Instrumentation using backplanes & logging• * Axon Visualiser• * Embeddable python console•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Larger Systems & Debugging•
• Practical examples.
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Part 2•
• Second part:• - More advanced stuff (demo/etc 20 mins)• - Larger systems and debugging. (20 mins)• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Building a Bulletin Board•
We're going to build an “old school” bulletin board system.
• * Someone telnets in & authenticates• * Can get help, exit or read messages• * Reading messages, they can read, reply,
exit reading, or get help• * State is restored/saved at session start/end
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Building a Bulletin Board•
For practicality, we won't implement message posting, nor message deletion, nor persistent user state. (Though these would be useful exercises)
• All on-disk objects encoded as json objects: * users * Messages – stored in “folders” (directories) with filenames == messageid
• * message fields: from, to, __body__, date, message, reply-to, subject
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Building a Bulletin Board• Getting started:• * Start with basic server for request/response
Just echo initially - BB1.py
• * make restarting quicker for debugging - BB2.py* abstract out “getting a line of data” - BB3.py* Then use that abstraction - BB4.py* Simplify “control” box handling - BB5.py* Abstract out reusable bits from domain specific – BB6.py
• We'll go through the code for these in the actual code files rather than on these slides
• (rather unwieldy as slides)
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Building a Bulletin Board• Intermediate Plan• This is the logic of the user protocol
• Seq(
• Authenticator( <some shared state> ),
• UserRetriever( <some shared state> ),
• MessageBoardUI( <some shared state> ),
• StateSaverLogout( <some shared state> ),
• )
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Building a Bulletin Board• Intermediate Plan• * Tidy up control handling a touch more & add netPrint method
to reusable bit - BB7.py
• * Change Authenticator component to support the “passed on state”, change protocol handler factory to create that Seq pipeline. - BB8.py
• * Change Authenticator to authenticate against a password file and set “remoteuser” in the shared state - BB9.py
• * Write stubs for UserRetriever & StateSaverLogout - BB10.py
•
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Building a Bulletin Board• Final UI Plan• * Change usage of waitMsg to yield self.waitMsg() -- BB11.py
• * Copy Authenticator's patten to create initial menu for MessageBoardUI, including stub for messages menu - BB12.py
• * Use same pattern for messages menu, use waitMsg pattern for logic. - BB13.py
• * Implement Folders to hold messages. Numbers as filenames (msg 1, 2, 3, 4) Messages as json encoded objects - BB14.py
• * Clean up to add a line oriented buffer to handle partial lines. (necessary for cross platform/real world) - BB15.py
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Building a Bulletin Board• Final UI Plan• * Change usage of waitMsg to yield self.waitMsg() -- BB11.py
• * Copy Authenticator's patten to create initial menu for MessageBoardUI, including stub for messages menu - BB12.py
• * Use same pattern for messages menu, use waitMsg pattern for logic. - BB13.py
• * Implement Folders to hold messages. Numbers as filenames (msg 1, 2, 3, 4) Messages as json encoded objects - BB14.py
• * Clean up to add a line oriented buffer to handle partial lines. (necessary for cross platform/real world) - BB15.py
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Summary• We've covered• * the 30,000 view of kamaelia
• * Building your own core
• * Building components, including a video recording application
• * Tools for building systems
• * Tools for debugging systems
• * Built a large(ish) system (an authenticated staged protocol)
• * Illustrated the majority of this using real world systems
http://www.kamaelia.org/PragmaticConcurrency [email protected]
Thank you!¬