python twisted

Download Python twisted

Post on 19-Jun-2015

1.776 views

Category:

Technology

3 download

Embed Size (px)

DESCRIPTION

Introduction to twisted programming (Pycon 2010)

TRANSCRIPT

  • 1. PythonTwistedMahendra M@mahendra http://creativecommons.org/licenses/by-sa/3.0/

2. Methodsofconcurrency Workers Threadsandprocesses EventdrivenLetusexaminethiswiththecaseofawebserver 3. Workermodelrequest dispatch() worker_1()read(fp) db_rd() db_wr()sock_wr() worker_n() 4. Workermodelrequest dispatch() worker_1()read(fp) BLOCKING!!db_rd() db_wr()sock_wr() worker_n() 5. Howdoesitscale? AworkergetsCPUwhenitisnotblocking Whenitmakesablockingcall,itsleepstillthesys callisrequested AtthistimeanotherworkergetsCPU Workermightblockbeforeitcompletesits allocatedtimeslice. Thismodelhasworkedgreatandstillworksgreat Eg:Apache,Squid 6. Overheads Workermanagement Threadcreation Processcreationandmanagement Synchronization Scheduling(thoughthisislefttoOS) Moresystemcallsforblockingoperations 7. EventDrivenCode Nonblockingcodeblocks Codeexecutiononevents Dataonsockets,timer,newconnection Executiontriggeredfromaneventloop FulluseofCPUtimeslice 8. Visually...Non blocking functionsevent_1 hdler_1()event_2 block_on_events( .. ) hdler_2()Events are postedevent_n hdler_n() 9. Visually... Non blocking functionsevent_1hdler_1()ev()event_2block_on_events() hdler_2()Events are postedevent_nhdler_n() 10. WebServer Non blocking functions request open(fp) reg() opened parse()event_loop()read_sql()reg()sql_read wri_sql()reg()sql_writ sock_wr()reg()responded close() 11. EventDrivenDesigns Nginx,Tornado Varnish Memcached OSsupport epollLinux kqueueBSDs 12. EventDrivenLibraries Libevent Pythontwisted JavaNIO ApacheMINA,Tomcat(notdefault) Jetty QT 13. Drawbacks Toughertocode,designandmaintain WorkersrequiredtomakeuseofmultipleCPUs Allcallbacksmustbenonblocking Toughtogetnonblockinglibrariesforallmodules Noisolation Ablockinanyeventloopcanfreezeeverything 14. PythonTwisted Eventdrivenprogrammingframework MITlicensed 8yearsoldandstable Supportforlargenumberofprotocols Clientandserversupport HTTPSOAP,REST,CouchDB,XMLRPC,.... Sockets,TCP/IP,Multicast,TLS,SSH,IMAP SMTP,NNTP,FTP,Memcached,AMQP,XMPP,... 15. Deferred Thecentralconceptoftwisted Acallbackreturnsadeferredtoindicatethatthejob isnotdoneyet. Thecallercanaddcallbackstoadeferred. ThecallbacksareinvokedthenthejobisdoneEh? 16. Deferredexamplefrom twisted.internet import reactor# Define a success callbackdef cb( arg1, arg2 ):print Timeout after %d %s % ( arg1, arg2 )# Define an error callbackdef eb( error ):Print error %s % error# Invoke a non blocking taskdeferred = someTimeout( 4 )# Register the callbacks on the returned deferreddeferred.addCallback( cb, 4, twisted is great )deferred.addErrback( eb )# Run the event loopreactor.run() 17. TwistedServerfrom twisted.internet.protocol import Protocol, Factoryfrom twisted.internet import reactorclass QOTD(Protocol):def connectionMade(self):self.transport.write("Welcomern")self.transport.loseConnection()# Next lines are magic:factory = Factory()factory.protocol = QOTD# 8007 is the port you want to run under.reactor.listenTCP(8007, factory)reactor.run() 18. Deferredchaining Acallbackcanregisterandreturnanotherdeferred Thiscallbackcanreturnanotherdeferred Inshortwehaveadeferredchain Webserverexample: 19. DeferredChaining 20. Advancedtwisted Twistedapplicationsupport Mixingandmatchingtwistedapplications Commandlinetwistdrunner Predefinedtwistedapps Web,telnet,xmpp,dns,conch(ssh),mail, Pluginarchitecture Livedebugging ADBAPIforRDBMS 21. Advancedtwisted PerspectiveBroker RPCandobjectsharing Spreadingoutservers CredAuthenticationframework Deferringtothreads Externalloops(GTK,QT) Streamingsupport MVCframework(Complex.Verycomplex) 22. Drawbacks Singlethreadedbydesign Makesuseofonlyonecore/CPU NeedexternalmodulesforusingmultipleCPU Runmultipleinstancesoftwistedonabox num_instances=num_cpus Usenginx(HTTP),HAProxyforloadbalancing 23. Demos 24. Links http://twistedmatrix.com/trac/