the onward journey: porting twisted to python 3
TRANSCRIPT
Twisted
What is Twisted?
Python library, written by Glyph Lefkowitz and many others
provides basic building blocks for writing networking clients and servers:protocol implementations: SSH, IRC, SMTP, IMAPevent “reactors”: select, kqueue, epoll, IOCP,
asyncio…...
More resources
Web site with docs, examples, tutorials: http://www.twistedmatrix.com
Main code repository: https://github.com/twisted/twisted
O’Reilly book:
Twisted sub-projects
Klein (similar to Flask, Bottle) https://github.com/twisted/klein
Treq (similar to Requests) https://github.com/twisted/treq
Which projects Twisted?
buildbot
scrapy
Lots of other projects: https://twistedmatrix.com/trac/wiki/ProjectsUsingTwisted
Who uses Twisted?
Hipchat
Apple Calendar Server
Lots of companies: https://twistedmatrix.com/trac/wiki/SuccessStories
Twisted and asynchronous programming
Twisted has promoted the technique of using callbacks and asynchronous programming for writing network servers for a long time
Twisted’s Deferred class is central to this, and has been used extensively
Guido van Rossum consulted with Glyph in the design of Python’s new asyncio framework ( Python 3.4 and higher )
Interesting things
First commit to Twisted was in 2001
Twisted includes a tool trial which runs unittest-style tests. (similar to pytest or nose)
Unittests and code coverage are very important to Twisted development process
Why bother?
Motivation
I like Twisted and the community behind it
I had some time in between jobs and wanted to improve my Python skills and learn more about Python 3
I wanted to help out projects which depended on Twisted, but couldn’t move to Python 3, such as buildbot
Core Python devs are dropping Python 2 support in 2020: https://pythonclock.org/
Major motivation: Twisted moved to GitHub in 2016!!
Following process to submit patches via Subversion was cumbersome
Moving to GitHub and pull requests made things easier for submitters and reviewers
Integration with Continuous Integration (CI) was improved: codecov, travis, appveyor, buildbot
Submitting patches is easier!!
Moving Twisted to GitHub
Moving to Python 3
Twisted started moving to Python 3
Original Python 3 porting plan developed in 2012
Worked on by various developers: Jean-Paul Calderone, Itamar Turner-Trauring, Amber Brown, Glyph Lefkowitz, Ralph Meijer, and others
Canonical funded some Python 3 porting work
Some parts ported, many parts still unported
Moving Twisted to Python 3 is a tough job!
Old codebase (since 2001)
Advanced framework which uses many, many features of Python
Twisted development process requires unit tests and coverage
Submitting hundreds of patches in Subversion workflow was slow moving
Porting to Python 3
What has changed in Python 3?
Lots of little changes to make the language cleaner
Deprecated code has been removed
Some changes are backwards incompatible. Previously working code is now broken on Python 3
http://python3porting.com has an extensive list of changes in Python 3
print is now a function
print “hello world”
now must be:
print(“hello world”)
dict.has_key() is gone
some_dict = { “one” : 1, “two”: 2} some_dict.has_key(“one”)
Now should be:
“one” in some_dict
obj.__cmp__() and cmp() is gone
Developers are supposed to implement __lt__(), __gt__(), __ge__(), __le__(), __ne__(), __eq__() functions on an object instead of __cmp__()
Developers need to use <, >, >=, <=, !=, == operators instead of cmp() which is gone
Less things allocate lists
These functions no longer allocate lists in Python 3:range(), dict.items(), dict.values(), map(), filter()
Users should iterate over these functions, which only allocate items as they are needed:
for n in range(99): ...
C API for Python C extensions changed
C API changed in a backwards incompatible way
Very challenging when porting the Twisted IOCP reactor (Windows only) which has parts written in C
Python str type has changed
Python 2:u”Some unicode string 銩” is of type unicode
“Some string” is of type str and also of type bytes
b”Some string” is of type str and also of type bytes
type(unicode) != type(str), type(str) == type(bytes)
Python 3:u”Some string 銩” is of type str
“Some string 銩” is of type str
b”Some string” is of type bytes
type(str) != type(bytes), unicode is gone, don’t need u prefix
Python str type has changed
Twisted protocols must send out bytes over the wire on sockets
Lots of code written assuming type(str) == type(bytes), did not account for unicode
This type of porting needs extensive analysis and testing, cannot be automated
Python str type has changed
Ned Batchelder unicode presentation very good: https://nedbatchelder.com/text/unipain/unipain.html
“Unicode sandwich” technique (write bytes to sockets and files, keep data as unicode internally in application) cannot be used 100% when dealing with network protocols
Technique for porting to Python 3
Porting technique: use virtualenvs
Checkout the code from git
Create a python2 virtualenv in one window:virtualenv myenv_2
source myenv_2/bin/activate
python setup.py develop
Create a python3 virtualenv in another window:python3 -m venv myenv_3
source myenv_3/bin/activate
python setup.py develop
Porting technique: run unit tests
After modifying code, run unittests using tox and trial
See what breaks, make sure it works on Python 2.7 and Python 3
Write new unit tests if necessary
Always try to improve code coverage:https://codecov.io/gh/twisted/twisted/
Python 3 status for Twisted
June 3, 2016
Python 2.7: 8425 tests PASS
Python 3.5: 4834 tests PASS ( approx. 57% of Python 2.7 tests)
March 21, 2017
Python 2.7: 9692 tests PASS
Python 3.5: 9025 tests PASS ( approx. 93% of Python 2.7 tests)
My contributions: 325 pull requests!
What’s left?
A few modules need to be ported such as:twisted.mail
twisted.news
twisted.web
Lessons learned
What I learned
Extensive unittests and code coverage are very important for this kind of effort
Porting an old and large codebase to Python 3 can be a lot of work
Benefits of porting: code cleanliness and keeping up with Python direction...the benefits vs. the effort required sometimes doesn’t feel worth it
Hope for the future and performance
CPython 3.6 and 3.7 performance seems to be improving and is comparable to Python 2.7:http://speed.python.org
Pypy 3.5 just came out….hopefully better performance with that
Now that Python has asyncio built in, the “Twisted way” of doing things has some validation, and is pervading more libraries and projects in Python
Thanks
All who started before me on the Python 3 effort: Jean-Paul, Itamar, Amber, Glyph, Ralph, many others
All who helped code review my patches: Adi Roiban, Alex Gaynor, Glyph, many others
Special thanks to Abhishek Choudhary for help on code reviews