the onward journey: porting twisted to python 3

The Onward Journey:Porting Twisted to Python 3

Craig Rodrigues <rodrigc@crodrigues.org>

Twisted

What is Twisted?

Python library, written by Glyph Lefkowitz and many others

provides basic building blocks for writing networking clients and servers:protocol implementations: SSH, IRC, SMTP, IMAPevent “reactors”: select, kqueue, epoll, IOCP,

asyncio…...

More resources

Web site with docs, examples, tutorials: http://www.twistedmatrix.com

Main code repository: https://github.com/twisted/twisted

O’Reilly book:

Twisted sub-projects

Klein (similar to Flask, Bottle) https://github.com/twisted/klein

Treq (similar to Requests) https://github.com/twisted/treq

Which projects Twisted?

buildbot

scrapy

Lots of other projects: https://twistedmatrix.com/trac/wiki/ProjectsUsingTwisted

Who uses Twisted?

Hipchat

Apple Calendar Server

Lots of companies: https://twistedmatrix.com/trac/wiki/SuccessStories

Twisted and asynchronous programming

Twisted has promoted the technique of using callbacks and asynchronous programming for writing network servers for a long time

Twisted’s Deferred class is central to this, and has been used extensively

Guido van Rossum consulted with Glyph in the design of Python’s new asyncio framework ( Python 3.4 and higher )

Interesting things

First commit to Twisted was in 2001

Twisted includes a tool trial which runs unittest-style tests. (similar to pytest or nose)

Unittests and code coverage are very important to Twisted development process

Why bother?

Motivation

I like Twisted and the community behind it

I had some time in between jobs and wanted to improve my Python skills and learn more about Python 3

I wanted to help out projects which depended on Twisted, but couldn’t move to Python 3, such as buildbot

Core Python devs are dropping Python 2 support in 2020: https://pythonclock.org/

Major motivation: Twisted moved to GitHub in 2016!!

Following process to submit patches via Subversion was cumbersome

Moving to GitHub and pull requests made things easier for submitters and reviewers

Integration with Continuous Integration (CI) was improved: codecov, travis, appveyor, buildbot

Submitting patches is easier!!

Moving Twisted to GitHub

Moving to Python 3

Twisted started moving to Python 3

Original Python 3 porting plan developed in 2012

Worked on by various developers: Jean-Paul Calderone, Itamar Turner-Trauring, Amber Brown, Glyph Lefkowitz, Ralph Meijer, and others

Canonical funded some Python 3 porting work

Some parts ported, many parts still unported

Moving Twisted to Python 3 is a tough job!

Old codebase (since 2001)

Advanced framework which uses many, many features of Python

Twisted development process requires unit tests and coverage

Submitting hundreds of patches in Subversion workflow was slow moving

Porting to Python 3

What has changed in Python 3?

Lots of little changes to make the language cleaner

Deprecated code has been removed

Some changes are backwards incompatible. Previously working code is now broken on Python 3

http://python3porting.com has an extensive list of changes in Python 3

print is now a function

print “hello world”

now must be:

print(“hello world”)

dict.has_key() is gone

some_dict = { “one” : 1, “two”: 2} some_dict.has_key(“one”)

Now should be:

“one” in some_dict

obj.__cmp__() and cmp() is gone

Developers are supposed to implement __lt__(), __gt__(), __ge__(), __le__(), __ne__(), __eq__() functions on an object instead of __cmp__()

Developers need to use <, >, >=, <=, !=, == operators instead of cmp() which is gone

Less things allocate lists

These functions no longer allocate lists in Python 3:range(), dict.items(), dict.values(), map(), filter()

Users should iterate over these functions, which only allocate items as they are needed:

for n in range(99): ...

C API for Python C extensions changed

C API changed in a backwards incompatible way

Very challenging when porting the Twisted IOCP reactor (Windows only) which has parts written in C

Python str type has changed

Python 2:u”Some unicode string 銩” is of type unicode

“Some string” is of type str and also of type bytes

b”Some string” is of type str and also of type bytes

type(unicode) != type(str), type(str) == type(bytes)

Python 3:u”Some string 銩” is of type str

“Some string 銩” is of type str

b”Some string” is of type bytes

type(str) != type(bytes), unicode is gone, don’t need u prefix

Python str type has changed

Twisted protocols must send out bytes over the wire on sockets

Lots of code written assuming type(str) == type(bytes), did not account for unicode

This type of porting needs extensive analysis and testing, cannot be automated

Python str type has changed

Ned Batchelder unicode presentation very good: https://nedbatchelder.com/text/unipain/unipain.html

“Unicode sandwich” technique (write bytes to sockets and files, keep data as unicode internally in application) cannot be used 100% when dealing with network protocols

Technique for porting to Python 3

Porting technique: use virtualenvs

Checkout the code from git

Create a python2 virtualenv in one window:virtualenv myenv_2

source myenv_2/bin/activate

python setup.py develop

Create a python3 virtualenv in another window:python3 -m venv myenv_3

source myenv_3/bin/activate

python setup.py develop

Porting technique: run unit tests

After modifying code, run unittests using tox and trial

See what breaks, make sure it works on Python 2.7 and Python 3

Write new unit tests if necessary

Always try to improve code coverage:https://codecov.io/gh/twisted/twisted/

Python 3 status for Twisted

June 3, 2016

Python 2.7: 8425 tests PASS

Python 3.5: 4834 tests PASS ( approx. 57% of Python 2.7 tests)

March 21, 2017

Python 2.7: 9692 tests PASS

Python 3.5: 9025 tests PASS ( approx. 93% of Python 2.7 tests)

My contributions: 325 pull requests!

What’s left?

A few modules need to be ported such as:twisted.mail

twisted.news

twisted.web

Lessons learned

What I learned

Extensive unittests and code coverage are very important for this kind of effort

Porting an old and large codebase to Python 3 can be a lot of work

Benefits of porting: code cleanliness and keeping up with Python direction...the benefits vs. the effort required sometimes doesn’t feel worth it

Hope for the future and performance

CPython 3.6 and 3.7 performance seems to be improving and is comparable to Python 2.7:http://speed.python.org

Pypy 3.5 just came out….hopefully better performance with that

Now that Python has asyncio built in, the “Twisted way” of doing things has some validation, and is pervading more libraries and projects in Python

Thanks

All who started before me on the Python 3 effort: Jean-Paul, Itamar, Amber, Glyph, Ralph, many others

All who helped code review my patches: Adi Roiban, Alex Gaynor, Glyph, many others

Special thanks to Abhishek Choudhary for help on code reviews