The goal of shrinkwrap is to create Python packages for non-Python software and install them using Python package management. To this end, shrinkwrap “packages” are simply setuptools setup scripts which know how to download and install other software.
A shrinkwrap package may either use a built-in installer function (for common installation methods) or define a custom one.
An installer for software using autoconf is available to simplify things. The autoconf shrinkwrap installer assumes the tarfile unpacks to a directory with the same name as the tarfile with .tar.{gz,bz2} removed, contains a configure script, and understands the –prefix option to control where the package is installed.
For example, to package curl, one may write the following (as, e.g. curl-7.27.0.py):
try:
from shrinkwrap.install import ShrinkwrapInstall
except ImportError:
import subprocess; subprocess.check_call('pip install -b $VIRTUAL_ENV/build/build-shrinkwrap shrinkwrap', shell=True)
from shrinkwrap.install import ShrinkwrapInstall
from setuptools import setup
version = '7.27.0'
setup(
name='curl',
version=version,
author='Stan Seibert',
author_email='stan@mtrr.org',
shrinkwrap_installer='autoconf',
shrinkwrap_source_url='http://curl.haxx.se/download/curl-%s.tar.bz2' % version,
cmdclass={'install': ShrinkwrapInstall},
)
To install curl, one would simply run:
python curl-7.27.0.py install
The boilerplate at the top of the script is required to ensure that shrinkwrap is installed before setuptools is imported. The cmdclass option must be set as shown above in order to use the ShrinkwrapInstall command rather than the default install command provided by setuptools.
The filename of the python script is not used by shrinkwrap.
The following package for bzip2 illustrates a custom installer function:
try:
from shrinkwrap.install import ShrinkwrapInstall
except ImportError:
import subprocess; subprocess.check_call('pip install shrinkwrap', shell=True)
from shrinkwrap.install import ShrinkwrapInstall
import os
from setuptools import setup
version = '1.0.6'
source_url = 'http://www.bzip.org/%(version)s/bzip2-%(version)s.tar.gz' % {'version': version}
def installer(inst):
self.download_and_unpack_tarball(source_url)
bzip2_dir = 'bzip2-' + version
os.chdir(bzip2_dir)
self.make(extra_opts=['install', 'PREFIX=%s' % self.virtualenv])
setup(
name='bzip2',
version=version,
author='Andy Mastbaum',
author_email='mastbaum@hep.upenn.edu',
shrinkwrap_installer=installer,
cmdclass={'install': ShrinkwrapInstall},
)
As before, installing bzip2 simply requires:
python bzip2-1.0.6.py install
Here, the shrinkwrap_installer argument to setup() is set to our own installer function. The installer function is called with an instance of ShrinkwrapInstall as the argument, which provides several convenience functions. This installer uses two of these functions, download_and_unpack_tarball and make to download, untar, and compile the bzip2 library. See shrinkwrap.install for a complete list of available functions. By passing extra options to make, the software is installed into the root of the active virtualenv.
Note
For several examples of custom installers, see https://bitbucket.org/seibert/shrinkwrap_pkgs.
Examples include getting code from version control, installing with cmake, and customizing install paths.
In some situations, you only want to install a package if the operating system does not already provide it. The shrinkwrap_skip argument to setup() allows you to specify a function to call to check if installation is performed. If the skip function returns True, then the installer() function is skipped, but the package is marked as installed.
Here is an example of a package that installs curl only if version 7.26.0 or newer is not present:
try:
from shrinkwrap.install import ShrinkwrapInstall
except ImportError:
import subprocess; subprocess.check_call('pip install -b $VIRTUAL_ENV/build/build-shrinkwrap shrinkwrap', shell=True)
from shrinkwrap.install import ShrinkwrapInstall
from setuptools import setup
from distutils.version import LooseVersion
version = '7.27.0'
def skip(inst):
try:
output = inst.cmd_output('curl-config --version')
name, version_str = output.split()
system_version = LooseVersion(version_str)
min_version = LooseVersion('7.26.0')
if system_version > min_version:
return True # Don't install
else:
return False # Version too old
except:
return False # install package if anything went wrong
setup(
name='curl-check',
version=version,
author='Stan Seibert',
author_email='stan@mtrr.org',
shrinkwrap_installer='autoconf',
shrinkwrap_skip=skip,
shrinkwrap_source_url='http://curl.haxx.se/download/curl-%s.tar.bz2' % version,
cmdclass={'install': ShrinkwrapInstall},
)
The handling of package dependencies in pip (as of version 1.1, anyway) does not, unfortunately, meet the requirements for installing compiled software. Packages listed in the install_requires keyword argument to setup() will be discovered by pip and installed, but in an arbitrary order. Packages listed in the setup_requires keyword argument are not actually installed to the virtualenv, but rather made available in .egg files in the build directory, which also is not a solution for compiled code.
As a result, shrinkwrap adds a new keyword argument, shrinkwrap_requires, to setup(). All dependencies listed here are guaranteed to be fully installed to the virtualenv before the installer() function is run. A separate pip process is spawned to install each one, so the dependency string can include package versions, just as setup_requires and install_requires allow. The shrinkwrap dependencies can be any kind of package, so if your library uses SCons as the build system, you can list it in the shrinkwrap_requires field:
setup(
name='foo',
version=version,
author='Example Author'
author_email='author@example.com'
shrinkwrap_installer=installer,
shrinkwrap_requires=['scons'],
cmdclass={'install': ShrinkwrapInstall},
)
If your package also needs to set some shell environment variables for operation, they can be placed in the $VIRTUAL_ENV/env.d directory, and they will be sourced automatically when the $VIRTUAL_ENV/bin/activate script is sourced. See the package file for CERN’s ROOT library for an example of using the install_env() convenience function.
To share shrinkwrap packages via a package index like PyPI (remember: don’t actually upload shrinkwrap packages to PyPI), you must create distribution tarballs. Shrinkwrap includes a tool to create these automatically from wrapper setup.py files:
shrinkwrap createpkg bzip2-1.0.6.py
This will create a tarball in the current directory suitable for uploading. Wildcards are valid for generating many packages at once.
Note
You can name the python file anything you want (not just setup.py). It will be copied to setup.py in a temp directory and the name of the resulting tarball will be derived from the package metadata you specified.
As mentioned in the Limitations section, shrinkwrap packages are small and easy to deploy. It is better to have different package repositories when you want to build a source package with different options in different situations, rather than have One Repository To Rule Them All [1] .
If you wish to serve your own package index, use the -p option to place output tarballs into one properly-formatted directory:
shrinkwrap createpkg -p packages my_shrinkwrap_pkgs/*
Simply serve the packages directory on the web, and pip clients can interact with it just like PyPI:
client$ export PIP_EXTRA_INDEX_URL=http://your-server.com/packages/
client$ pip install bzip2
Extra package indices can be specified in a requirements.txt file with an option at the top:
--extra-index-url http://mtrr.org/packages/
| [1] | In order to avoid creating One Repository To Rule Them All, we do not plan to ever offer a repository of “official” shrinkwrap packages. Feel free to copy other people’s package files to your repository, however. |