Rolling with the TimesUsing Wheels, pbr, and Twine

for Distributing and Installing Software

Doug Hellmann PyATL Feb 2015

Dist Formats• sdist

• source only

• requires compiler to install C extensions

• egg

• binary importable format

• defacto standard

• wheel

• binary installable format

• follows PyPA team standards

Wheel File Names{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl

• distribution – software name

• version-buildtag – version string for distribution

• python tag – version of python

• abitag – C extension application binary interface

• platform tag – CPU architecture

Building sdists$ python setup.py sdist…$ tar ztf dist/entry_point_inspector-0.1.tar.gzentry_point_inspector-0.1/entry_point_inspector-0.1/entry_point_inspector/entry_point_inspector-0.1/entry_point_inspector/__init__.pyentry_point_inspector-0.1/entry_point_inspector/app.pyentry_point_inspector-0.1/entry_point_inspector/ep.pyentry_point_inspector-0.1/entry_point_inspector/group.pyentry_point_inspector-0.1/entry_point_inspector.egg-info/entry_point_inspector-0.1/entry_point_inspector.egg-info/dependency_links.txtentry_point_inspector-0.1/entry_point_inspector.egg-info/entry_points.txtentry_point_inspector-0.1/entry_point_inspector.egg-info/namespace_packages.txtentry_point_inspector-0.1/entry_point_inspector.egg-info/not-zip-safeentry_point_inspector-0.1/entry_point_inspector.egg-info/PKG-INFOentry_point_inspector-0.1/entry_point_inspector.egg-info/requires.txtentry_point_inspector-0.1/entry_point_inspector.egg-info/SOURCES.txtentry_point_inspector-0.1/entry_point_inspector.egg-info/top_level.txtentry_point_inspector-0.1/PKG-INFOentry_point_inspector-0.1/README.rstentry_point_inspector-0.1/setup.cfgentry_point_inspector-0.1/setup.py

Building eggs$ python setup.py bdist_egg…$ unzip -l dist/entry_point_inspector-0.1-py2.7.eggArchive: dist/entry_point_inspector-0.1-py2.7.egg Length Date Time Name -------- ---- ---- ---- 1 01-02-15 11:50 EGG-INFO/dependency_links.txt 224 01-02-15 11:50 EGG-INFO/entry_points.txt 1 01-02-15 11:50 EGG-INFO/namespace_packages.txt 1 07-26-13 15:47 EGG-INFO/not-zip-safe 5132 01-02-15 11:50 EGG-INFO/PKG-INFO 6 01-02-15 11:50 EGG-INFO/requires.txt 511 01-02-15 11:50 EGG-INFO/SOURCES.txt 22 01-02-15 11:50 EGG-INFO/top_level.txt 0 07-26-13 15:49 entry_point_inspector/__init__.py 154 01-02-15 11:50 entry_point_inspector/__init__.pyc 659 07-26-13 17:12 entry_point_inspector/app.py 1346 01-02-15 11:50 entry_point_inspector/app.pyc 2148 07-26-13 17:13 entry_point_inspector/ep.py 2264 01-02-15 11:50 entry_point_inspector/ep.pyc 1527 07-26-13 16:59 entry_point_inspector/group.py 2503 01-02-15 11:50 entry_point_inspector/group.pyc -------- ------- 16499 16 files

Building Wheels$ python setup.py bdist_wheelusage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help

error: invalid command ‘bdist_wheel'

Building Wheels$ python setup.py bdist_wheelusage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help

error: invalid command ‘bdist_wheel'

$ pip install wheelCollecting wheel Using cached wheel-0.24.0-py2.py3-none-any.whlInstalling collected packages: wheel

Successfully installed wheel-0.24.0

Building Wheels$ python setup.py bdist_wheel…$ unzip -l dist/entry_point_inspector-0.1-py2-none-any.whlArchive: dist/entry_point_inspector-0.1-py2-none-any.whl Length Date Time Name -------- ---- ---- ---- 0 07-26-13 15:49 entry_point_inspector/__init__.py 659 07-26-13 17:12 entry_point_inspector/app.py 2148 07-26-13 17:13 entry_point_inspector/ep.py 1527 07-26-13 16:59 entry_point_inspector/group.py 3761 01-02-15 11:59 entry_point_inspector-0.1.dist-info/DESCRIPTION.rst 224 01-02-15 11:59 entry_point_inspector-0.1.dist-info/entry_points.txt 1265 01-02-15 11:59 entry_point_inspector-0.1.dist-info/metadata.json 1 01-02-15 11:59 entry_point_inspector-0.1.dist-info/namespace_packages.txt 22 01-02-15 11:59 entry_point_inspector-0.1.dist-info/top_level.txt 92 01-02-15 11:59 entry_point_inspector-0.1.dist-info/WHEEL 4518 01-02-15 11:59 entry_point_inspector-0.1.dist-info/METADATA 1134 01-02-15 11:59 entry_point_inspector-0.1.dist-info/RECORD -------- ------- 15351 12 files

Wheel File Contents

• .py source files

• .so/.dll library files

• dist-info standard metadata

• no setup.py

sdist Installation$ mktmpenv$ pip install --no-cache-dir entry_point_inspectorCollecting entry-point-inspector Downloading entry_point_inspector-0.1.tar.gzCollecting cliff (from entry-point-inspector) Downloading cliff-1.9.0-py2-none-any.whlCollecting cmd2>=0.6.7 (from cliff->entry-point-inspector) Downloading cmd2-0.6.8.tar.gzCollecting stevedore>=1.1.0 (from cliff->entry-point-inspector) Downloading stevedore-1.1.0-py2.py3-none-any.whlCollecting pyparsing>=2.0.1 (from cliff->entry-point-inspector) Downloading pyparsing-2.0.3-py2.py3-none-any.whlCollecting pbr!=0.7,<1.0,>=0.6 (from cliff->entry-point-inspector) Downloading pbr-0.10.7-py2.py3-none-any.whl (70kB) 100% |################################| 73kB 1.0MB/sCollecting PrettyTable<0.8,>=0.7 (from cliff->entry-point-inspector) Downloading prettytable-0.7.2.tar.bz2Requirement already satisfied (use --upgrade to upgrade): pip in ./lib/python2.7/site-packages (from pbr!=0.7,<1.0,>=0.6->cliff->entry-point-inspector)Installing collected packages: PrettyTable, pbr, pyparsing, stevedore, argparse, six, cmd2, cliff, entry-point-inspector Running setup.py install for PrettyTable

Running setup.py install for cmd2

Running setup.py install for entry-point-inspector Installing epi script to /Users/dhellmann/Envs/tmp-14f1bec315f89c0c/binSuccessfully installed PrettyTable-0.7.2 argparse-1.3.0 cliff-1.9.0 cmd2-0.6.8 entry-point-inspector-0.1 pbr-0.10.7 pyparsing-2.0.3 six-1.9.0 stevedore-1.1.0

Speeding Up Installation

$ cat ~/.pip/pip.conf[global]wheel-dir = /Users/dhellmann/.pip/wheelhousefind-links = /Users/dhellmann/.pip/wheelhouse

Building Wheels of Other Packages

$ pip wheel cmd2Collecting cmd2 Using cached cmd2-0.6.8.tar.gzCollecting pyparsing>=2.0.1 (from cmd2) Using cached pyparsing-2.0.3-py2.py3-none-any.whl Saved /Users/dhellmann/.pip/wheelhouse/pyparsing-2.0.3-py2.py3-none-any.whlSkipping pyparsing, due to already being wheel.Building wheels for collected packages: cmd2 Running setup.py bdist_wheel for cmd2 Destination directory: /Users/dhellmann/.pip/wheelhouseSuccessfully built cmd2

Wheel Installation$ mktmpenv$ pip install entry_point_inspectorCollecting entry-point-inspectorCollecting cliff (from entry-point-inspector)Collecting cmd2>=0.6.7 (from cliff->entry-point-inspector)Collecting six>=1.7.0 (from cliff->entry-point-inspector)Collecting argparse (from cliff->entry-point-inspector)Collecting stevedore>=1.1.0 (from cliff->entry-point-inspector)Collecting pyparsing>=2.0.1 (from cliff->entry-point-inspector)Collecting pbr!=0.7,<1.0,>=0.6 (from cliff->entry-point-inspector)Collecting PrettyTable<0.8,>=0.7 (from cliff->entry-point-inspector)Requirement already satisfied (use --upgrade to upgrade): pip in ./lib/python2.7/site-packages (from pbr!=0.7,<1.0,>=0.6->cliff->entry-point-inspector)Installing collected packages: PrettyTable, pbr, pyparsing, stevedore, argparse, six, cmd2, cliff, entry-point-inspector

Successfully installed PrettyTable-0.7.2 argparse-1.3.0 cliff-1.9.0 cmd2-0.6.8 entry-point-inspector-0.1 pbr-0.10.7 pyparsing-2.0.3 six-1.9.0 stevedore-1.1.0

pbr – Python Build Reasonableness

• Automatically gather meta-data

• Static meta-data as data, not code

• Automatically generate release files

pbr – Gathered Meta-data

• git ls → SOURCES.txt

• requirements.txt → install_requires

• README.rst → long_description

• git tag → version

pbr – Static Meta-data

• Name

• Trove Classifiers

• Home Page

• Python Packages

• Entry Points

pbr – Automatically Generate Release Files

• AUTHORS

• ChangeLog

pbr – setup.py#!/usr/bin/env python

import setuptools

setuptools.setup( setup_requires=['pbr'], pbr=True)

pbr – setup.cfg[metadata]name = oslo.configauthor = OpenStackauthor-email = openstack-dev@lists.openstack.orgsummary = Oslo Configuration APIdescription-file = README.rsthome-page = https://launchpad.net/osloclassifier = Development Status :: 4 - Beta Environment :: OpenStack Intended Audience :: Developers

Intended Audience :: Information Technology License :: OSI Approved :: Apache Software License Operating System :: OS Independent Programming Language :: Python Programming Language :: Python :: 2 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 Programming Language :: Python :: 3.3[files]packages = oslo_config

twine – Upload dists to PyPI

• Upload Existing Packages

• Supports sdist, egg, & wheel

• GPG Signature Files

twine$ twine -husage: twine [-h] [--version] {upload}

positional arguments: {upload}

optional arguments: -h, --help show this help message and exit --version show program's version number and exit

twine – Uploading Signed dists

python setup.py sdistpython setup.py bdist_wheel

gpg --detach-sign -a $sdistfilegpg --detach-sign -a $wheelfile

twine upload $sdistfile $sdistsignfile $wheelfile $wheelsignfile

References

• https://www.python.org/dev/peps/pep-0427/

• http://lucumr.pocoo.org/2014/1/27/python-on-wheels/

• http://pythonwheels.com

• https://www.berrange.com/posts/2014/11/14/faster-rebuilds-for-python-virtualenv-trees/