10 11-hart installing pythonsoftware
Post on 13-May-2015
839 Views
Preview:
DESCRIPTION
TRANSCRIPT
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration
under contract DE-AC04-94AL85000.
Installing Python Software Packages:The Good, The Bad and the Ugly
William E. HartSandia National Laboratories
wehart@sandia.gov
Overview
This talks focuses on installing on Python packages– Installing Python itself is usually a no-brainer
The Good1. Installing on Windows with an installer executable2. Installing with Linux application utility3. Installing a Python package from the PyPI repository4. Installing a Python package from source
The Bad5.Using a virtual environment to isolate package installations6.Using an installer executable on Windows with a virtual
environmentThe Ugly
7. Installing a Python extension package from source8.PyCoinInstall – Managing builds for Python extension packages
Slide 2
About Python
Recent releases:– 2.5 Very stable– 2.6 Very stable– 2.7 Recently released. Supports transition to Python 3.x– 3.0 Includes changes that are not backwards compatible.– 3.1 Stable 3.x release. Resolves 3.0 performance issues.
Package management for 3.x is still a bit snarky. We’ll focus on examples for 2.x
Slide 3
Python Packages
Package distributions– Include one or more *.py files– May include compiled source files (e.g. C++)– Include a top-level setup.py file to coordinate installation
Three categories:– Pure-Python packages
• Only include Python *.py files
– Independent extension packages• Include compiled source files only on standard system
libraries
– Dependent extension packages• Include compiled source files that rely on third-party
libraries
Slide 4
The Good
Assumption: you have administrative privileges– Windows administrator group– Linux super-user
Impact: you can install Python in system directories!
Idea:– Install packages in the system Python directories
• Specifically, the site-packages directory is used
– Can easily install pure-Python packages or pre-compiled extension packages
Slide 5
Example 1
Installing on Windows with an installer executable
Steps:1.Download installer executable from package website2.Execute installer
Notes:– This uses the registry to find the Python installation– The installer can contain pre-compiled extension packages
• These may differ for 32- and 64-bit installations– The installer will likely be specific to the Python version– This may work OK for dependent extension packages if the
dependent libraries are stored in a standard location
Slide 6
Example 2
Installing with Linux application utility
Idea: use a standard Linux package manager– Ubuntu (and other Debian-based Linux variants)
• apt-get install packagename– Fedora
• yum install packagename
Notes:– The package manager can install pre-compiled extension packages– The package manager handles package dependencies
automatically– Not all Python packages are available within these managers– It’s unclear whether these packages are updated frequently
Slide 7
Example 3
Installing a Python package from the PyPI repository
PyPI:– A web-based service for hosting Python packages– Includes some pre-compiled extension packages– Several Python packages are used to install from PyPI
• setuptools, distribute, pip
Installing with pip:– pip install packagename
Notes:– The package manager handles package dependencies automatically– Most Python packages are available from PyPI, including most releases– PyPI does not include many pre-compiled extension packages– The PyPI service has become a bit overloaded in the past year– You need to specify the HTTP_PROXY environment to talk to proxy servers
Slide 8
Example 3(cont)Problem: pip is not a standard Python package!
Installing pip:– wget http://peak.telecommunity.com/dist/ez_setup.py– python ez_setup.py– easy_install pip
Notes:– pip is now generally recommended over setuptools and
distribute– The wget command is not a standard MS Windows utility
• Download from http://users.ugent.be/~bpuype/wget/
Slide 9
Example 4
Installing a Python package from source
Steps:– Download package source from PyPI or project website– Extract package from tarball or zipfile– python setup.py install
Notes:– This process avoids the need to install pip– This is useful for packages that are not available on PyPI– This installation process can build and install some extension packages
• Configuration of the compiler may be a problem• Configuration of extension package dependencies is a problem
– This installation does not handle package dependencies• Package dependencies may cause an install to fail
Slide 10
The Bad
Problem: users do not have administrative privileges– This is not uncommon in industry
Problem: different users on a computer need different package versions
Idea:– A user can create a virtual Python environment– The user installs packages in this virtual Python directories
Impact:– Users have complete control over their Python environment
• Can even have multiple environments (e.g. for developers)– Can install both pure-Python packages or pre-compiled
extension packages
Slide 11
Virtual Python
Idea: create an isolated Python environment
virtualenv:– Creates a directory structure that mimics the system Python
installation– Works on MS Windows, Linux and Mac OS
Installing virtualenv (with administrative privileges)– Debian Linux
• apt-get install python-virtualenv
– Other• wget http://peak.telecommunity.com/dist/ez_setup.py• python ez_setup.py• easy_install virtualenv
Slide 12
Virtual Python (cont)Creating a virtual Python environment:
– virtualenv directory
Virtual Python executable:– MS Windows: directory/Scripts/python.exe– Linux: directory/bin/python
Notes:– On MS Windows the executables are installed in the Scripts
directory• For simplicity, all subsequent examples will use the bin directory
– The setuptools package is installed in a virtual environment• Can use the easy_install command to install packages
Example:– directory/bin/easy_install pip
Slide 13
Virtual Python(cont)Problem: setup a virtual environment without administrative privileges
vpy_install:– Bootstrap a virtualenv installation from a script
Steps:– wget http://doiop.com/pyutilib/vpy_install– python vpy_install directory
Notes:– Standard tools like setuptools and pip are installed by vpy_install– The specified directory is a virtualenv Python environment– On Linux, you can use a simpler command-line:
• vpy_install directory– The –-site-packages option can be used to expose packages that are
installed in the system Python installation• This is particularly useful when using precompiled extension packages,
which are installed in system Python directories
Slide 14
Example 5
Using a virtual environment to isolate package installations
Idea:– Create a virtual python installation with vpy_install– Install packages with pip
Example (Linux):– wget http://doiop.com/pyutilib/vpy_install– vpy_install mypy– mypy/bin/pip install packagename
Notes:– Within the virtual environment, pip is installed as an executable
• You do not need to execute it with the Python executable
Slide 15
Example 5(cont)Using a virtual environment to isolate package installations
Note: source package installations using a virtual Python executable are installed in the virtual environment
Example (Linux):– wget http://doiop.com/pyutilib/vpy_install– vpy_install mypy– cd packagename– ../mypy/bin/python setup.py install
Slide 16
Example 6
Using an installer executable on Windows with a virtual environment
Problem:– MS Windows installers use the system Python installation– If a user does not have administrative privileges, then an installer
cannot be used directly!
Idea:– “Install” the executable with the easy_install command
Example:– easy_install.exe matplotlib-1.0.0.win32-py2.6.exe
Note: this only works if the executable was created by the Python distutils package!
Slide 17
Example 6(cont)Using an installer executable on Windows with a virtual
environment
Idea:– Modify the MS Windows registry to register the virtual python
executable
Steps:
1.Download the the_python.py script that is described at
http://doiop.com/nedbatchelder/register_python.html
2.Register the virtual python executable (as administrator)
• mypy\Scripts\python.exe the_python.py
3.Execute the MS Windows installer
4.Reregister the original python executable (as administrator)
Slide 18
The Ugly
Problem:– you need to use an extension package that has dependencies– prebuilt packages are not available
Observation:– There is very little consistency between installation setup for
different packages– Few packages use configuration management tools like
autotools or cmake– Build instructions can be very different for different platforms
There are few options beyond diving deep into the package build process!
Slide 19
Example 7
Installing a Python extension package from source
Note: there is no standard way for doing this
Example: SciPy documentation– General build instructions– Separate instructions for Linux, Mac OS, Windows, and Solaris
Recommendations:– Make sure that this package is going to be worth your time– Don’t hesitate to ask the developer team for help– Have patience ...
Slide 20
Example 8
PyCoinInstall – Managing builds for Python extension packages
Note: this script is a relatively new project within CoinBazaar– https://projects.coin-or.org/CoinBazaar/wiki/Projects/PyCoinInstall
Idea:– Create a virtual environment with virtualenv– Define package installation with a simple plugin system
• Packages define download, build and/or install operations– A user creates a simple configuration file to specify the packages that
will be installed– The pci command is used to manage the installation
Notes:– Current development is focused on COIN-OR related packages– Current plugins are tailored for building under Linux
Slide 21
Final Thoughts
Installing and distributing pure-Python packages is quite easy.
Installing extension packages is much more difficult.– Try to leverage pre-compiled extension packages!
Virtual Python environments offer a flexible paradigm for managing different Python configurations
– These are worth using even if you have administrative privileges!
Slide 22
top related