diff options
Diffstat (limited to 'eggs/zc.buildout-1.5.2-py2.6.egg/EGG-INFO/PKG-INFO')
| -rw-r--r-- | eggs/zc.buildout-1.5.2-py2.6.egg/EGG-INFO/PKG-INFO | 8235 |
1 files changed, 8235 insertions, 0 deletions
diff --git a/eggs/zc.buildout-1.5.2-py2.6.egg/EGG-INFO/PKG-INFO b/eggs/zc.buildout-1.5.2-py2.6.egg/EGG-INFO/PKG-INFO new file mode 100644 index 0000000..ba54ddc --- /dev/null +++ b/eggs/zc.buildout-1.5.2-py2.6.egg/EGG-INFO/PKG-INFO @@ -0,0 +1,8235 @@ +Metadata-Version: 1.0 +Name: zc.buildout +Version: 1.5.2 +Summary: System for managing development buildouts +Home-page: http://pypi.python.org/pypi/zc.buildout +Author: Jim Fulton +Author-email: jim@zope.com +License: ZPL 2.1 +Description: ******** + Buildout + ******** + + .. contents:: + + The Buildout project provides support for creating applications, + especially Python applications. It provides tools for assembling + applications from multiple parts, Python or otherwise. An application + may actually contain multiple programs, processes, and configuration + settings. + + The word "buildout" refers to a description of a set of parts and the + software to create and assemble them. It is often used informally to + refer to an installed system based on a buildout definition. For + example, if we are creating an application named "Foo", then "the Foo + buildout" is the collection of configuration and application-specific + software that allows an instance of the application to be created. We + may refer to such an instance of the application informally as "a Foo + buildout". + + To get a feel for some of the things you might use buildouts for, see + the `Buildout examples`_. + + To lean more about using buildouts, see `Detailed Documentation`_. + + To see screencasts, talks, useful links and more documentation, visit + the `Buildout website <http://www.buildout.org>`_. + + Recipes + ******* + + Existing recipes include: + + `zc.recipe.egg <http://pypi.python.org/pypi/zc.recipe.egg>`_ + The egg recipe installes one or more eggs, with their + dependencies. It installs their console-script entry points with + the needed eggs included in their paths. It is suitable for use with + a "clean" Python: one without packages installed in site-packages. + + `z3c.recipe.scripts <http://pypi.python.org/pypi/z3c.recipe.scripts>`_ + Like zc.recipe.egg, this recipe builds interpreter scripts and entry + point scripts based on eggs. It can be used with a Python that has + packages installed in site-packages, such as a system Python. The + interpreter also has more features than the one offered by + zc.recipe.egg. + + `zc.recipe.testrunner <http://pypi.python.org/pypi/zc.recipe.testrunner>`_ + The testrunner egg creates a test runner script for one or + more eggs. + + `zc.recipe.zope3checkout <http://pypi.python.org/pypi/zc.recipe.zope3checkout>`_ + The zope3checkout recipe installs a Zope 3 checkout into a + buildout. + + `zc.recipe.zope3instance <http://pypi.python.org/pypi/zc.recipe.zope3instance>`_ + The zope3instance recipe sets up a Zope 3 instance. + + `zc.recipe.filestorage <http://pypi.python.org/pypi/zc.recipe.filestorage>`_ + The filestorage recipe sets up a ZODB file storage for use in a + Zope 3 instance created by the zope3instance recipe. + + Buildout examples + ***************** + + Here are a few examples of what you can do with buildouts. We'll + present these as a set of use cases. + + Try out an egg + ============== + + Sometimes you want to try an egg (or eggs) that someone has released. + You'd like to get a Python interpreter that lets you try things + interactively or run sample scripts without having to do path + manipulations. If you can and don't mind modifying your Python + installation, you could use easy_install, otherwise, you could create + a directory somewhere and create a buildout.cfg file in that directory + containing:: + + [buildout] + parts = mypython + + [mypython] + recipe = zc.recipe.egg + interpreter = mypython + eggs = theegg + + where theegg is the name of the egg you want to try out. + + Run buildout in this directory. It will create a bin subdirectory + that includes a mypython script. If you run mypython without any + arguments you'll get an interactive interpreter with the egg in the + path. If you run it with a script and script arguments, the script + will run with the egg in its path. Of course, you can specify as many + eggs as you want in the eggs option. + + If the egg provides any scripts (console_scripts entry points), those + will be installed in your bin directory too. + + Work on a package + ================= + + I often work on packages that are managed separately. They don't have + scripts to be installed, but I want to be able to run their tests + using the `zope.testing test runner + <http://www.python.org/pypi/zope.testing>`_. In this kind of + application, the program to be installed is the test runner. A good + example of this is `zc.ngi <http://svn.zope.org/zc.ngi/trunk/>`_. + + Here I have a subversion project for the zc.ngi package. The software + is in the src directory. The configuration file is very simple:: + + [buildout] + develop = . + parts = test + + [test] + recipe = zc.recipe.testrunner + eggs = zc.ngi + + I use the develop option to create a develop egg based on the current + directory. I request a test script named "test" using the + zc.recipe.testrunner recipe. In the section for the test script, I + specify that I want to run the tests in the zc.ngi package. + + When I check out this project into a new sandbox, I run bootstrap.py + to get setuptools and zc.buildout and to create bin/buildout. I run + bin/buildout, which installs the test script, bin/test, which I can + then use to run the tests. + + This is probably the most common type of buildout. + + If I need to run a previous version of zc.buildout, I use the + `--version` option of the bootstrap.py script:: + + $ python bootstrap.py --version 1.1.3 + + The `zc.buildout project <http://svn.zope.org/zc.buildout/trunk>`_ + is a slightly more complex example of this type of buildout. + + Install egg-based scripts + ========================= + + A variation of the `Try out an egg`_ use case is to install scripts + into your ~/bin directory (on Unix, of course). My ~/bin directory is + a buildout with a configuration file that looks like:: + + + [buildout] + parts = foo bar + bin-directory = . + + [foo] + ... + + where foo and bar are packages with scripts that I want available. As + I need new scripts, I can add additional sections. The bin-directory + option specified that scripts should be installed into the current + directory. + + Multi-program multi-machine systems + =================================== + + Using an older prototype version of the buildout, we've build a number + of systems involving multiple programs, databases, and machines. One + typical example consists of: + + - Multiple Zope instances + + - Multiple ZEO servers + + - An LDAP server + + - Cache-invalidation and Mail delivery servers + + - Dozens of add-on packages + + - Multiple test runners + + - Multiple deployment modes, including dev, stage, and prod, + with prod deployment over multiple servers + + Parts installed include: + + - Application software installs, including Zope, ZEO and LDAP + software + + - Add-on packages + + - Bundles of configuration that define Zope, ZEO and LDAP instances + + - Utility scripts such as test runners, server-control + scripts, cron jobs. + + Questions and Bug Reporting + *************************** + + Please send questions and comments to the + `distutils SIG mailing list <mailto://distutils-sig@python.org>`_. + + Report bugs using the `zc.buildout Launchpad Bug Tracker + <https://launchpad.net/zc.buildout/+bugs>`_. + + System Python and zc.buildout 1.5 + ********************************* + + The 1.5 line of zc.buildout introduced a number of changes. + + Problems + ======== + + As usual, please send questions and comments to the `distutils SIG + mailing list <mailto://distutils-sig@python.org>`_. Report bugs using + the `zc.buildout Launchpad Bug Tracker + <https://launchpad.net/zc.buildout/+bugs>`_. + + If problems are keeping you from your work, here's an easy way to + revert to the old code temporarily: switch to a custom "emergency" + bootstrap script, available from + http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap.py . + + This customized script will select zc.buildout 1.4.4 by default. + zc.buildout 1.4.4 will not upgrade itself unless you explicitly specify + a new version. It will also prefer older versions of zc.recipe.egg and + some other common recipes. If you have trouble with other recipes, + consider using a standard buildout "versions" section to specify older + versions of these, as described in the Buildout documentation + (http://pypi.python.org/pypi/zc.buildout#repeatable-buildouts-controlling-eggs-used). + + Working with a System Python + ============================ + + While there are a number of new features available in zc.buildout 1.5, + the biggest is that Buildout itself supports usage with a system Python. + This can work if you follow a couple of simple rules. + + 1. Use the new bootstrap.py (available from + svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/bootstrap.py). + + 2. Use buildout recipes that have been upgraded to work with zc.buildout 1.5 + and higher. Specifically, they should use + ``zc.buildout.easy_install.sitepackage_safe_scripts`` to generate + their scripts, if any, rather than ``zc.buildout.easy_install.scripts``. + See the `Recipes That Support a System Python`_ section below for more + details on recipes that are available as of this writing, and + `Updating Recipes to Support a System Python`_ for instructions on + how to update a recipe. Note that you should generally only need to + update recipes that generate scripts. + + You can then use ``include-site-packages = false`` and + ``exec-sitecustomize = false`` buildout options to eliminate access to + your Python's site packages and not execute its sitecustomize file, if + it exists, respectively. + + Alternately, you can use the ``allowed-eggs-from-site-packages`` buildout + option as a glob-aware whitelist of eggs that may come from site-packages. + This value defaults to "*", accepting all eggs. + + It's important to note that recipes not upgraded for zc.buildout 1.5.0 + should continue to work--just without internal support for a system Python. + + Using a system Python is inherently fragile. Using a clean, + freshly-installed Python without customization in site-packages is more + robust and repeatable. See some of the regression tests added to the + 1.5.0 line for the kinds of issues that you can encounter with a system + Python, and see + http://pypi.python.org/pypi/z3c.recipe.scripts#including-site-packages-and-sitecustomize + for more discussion. + + However, using a system Python can be very convenient, and the + zc.buildout code for this feature has been tested by many users already. + Moreover, it has automated tests to exercise the problems that have been + encountered and fixed. Many people rely on it. + + Recipes That Support a System Python + ==================================== + + zc.recipe.egg continues to generate old-style scripts that are not safe + for use with a system Python. This was done for backwards + compatibility, because it is integral to so many buildouts and used as a + dependency of so many other recipes. + + If you want to generate new-style scripts that do support system Python + usage, use z3c.recipe.scripts instead + (http://pypi.python.org/pypi/z3c.recipe.scripts). z3c.recipe.scripts has + the same script and interpreter generation options as zc.recipe.egg, + plus a few more for the new features mentioned above. In the simplest + case, you should be able to simply change ``recipe = zc.recipe.egg`` to + ``recipe = z3c.recipe.scripts`` in the pertinent sections of your + buildout configuration and your generated scripts will work with a system + Python. + + Other updated recipes include zc.recipe.testrunner 1.4.0 and + z3c.recipe.tag 0.4.0. Others should be updated soon: see their change + documents for details, or see `Updating Recipes to Support a System + Python`_ for instructions on how to update recipes yourself. + + Templates for creating Python scripts with the z3c.recipe.filetemplate + recipe can be easily changed to support a system Python. + + - If you don't care about supporting relative paths, simply using a + generated interpreter with the eggs you want should be sufficient, as + it was before. For instance, if the interpreter is named "py", use + ``#!${buildout:bin-directory/py}`` or ``#!/usr/bin/env + ${buildout:bin-directory/py}``). + + - If you do care about relative paths, (``relative-paths = true`` in + your buildout configuration), then z3c.recipe.scripts does require a + bit more changes, as is usual for the relative path support in that + package. First, use z3c.recipe.scripts to generate a script or + interpreter with the dependencies you want. This will create a + directory in ``parts`` that has a site.py and sitecustomize.py. Then, + begin your script as in the snippet below. The example assumes that + the z3c.recipe.scripts generated were from a Buildout configuration + section labeled "scripts": adjust accordingly. + + :: + + #!${buildout:executable} -S + ${python-relative-path-setup} + import sys + sys.path.insert(0, ${scripts:parts-directory|path-repr}) + import site + + Updating Recipes to Support a System Python + =========================================== + + You should generally only need to update recipes that generate scripts. + These recipes need to change from using ``zc.buildout.easy_install.scripts`` + to be using ``zc.buildout.easy_install.sitepackage_safe_scripts``. + The signatures of the two functions are different. Please compare:: + + def scripts( + reqs, working_set, executable, dest, + scripts=None, + extra_paths=(), + arguments='', + interpreter=None, + initialization='', + relative_paths=False, + ): + + def sitepackage_safe_scripts( + dest, working_set, executable, site_py_dest, + reqs=(), + scripts=None, + interpreter=None, + extra_paths=(), + initialization='', + include_site_packages=False, + exec_sitecustomize=False, + relative_paths=False, + script_arguments='', + script_initialization='', + ): + + In most cases, the arguments are merely reordered. The ``reqs`` + argument is no longer required in order to make it easier to generate an + interpreter alone. The ``arguments`` argument was renamed to + ``script_arguments`` to clarify that it did not affect interpreter + generation. + + The only new required argument is ``site_py_dest``. It must be the path + to a directory in which the customized site.py and sitecustomize.py + files will be written. A typical generation in a recipe will look like + this. + + (In the recipe's __init__ method...) + + :: + + self.options = options + b_options = buildout['buildout'] + options['parts-directory'] = os.path.join( + b_options['parts-directory'], self.name) + + (In the recipe's install method...) + + :: + + options = self.options + generated = [] + if not os.path.exists(options['parts-directory']): + os.mkdir(options['parts-directory']) + generated.append(options['parts-directory']) + + Then ``options['parts-directory']`` can be used for the ``site_py_dest`` + value. + + If you want to support the other arguments (``include_site_packages``, + ``exec_sitecustomize``, ``script_initialization``, as well as the + ``allowed-eggs-from-site-packages`` option), you might want to look at + some of the code in + svn://svn.zope.org/repos/main/zc.buildout/trunk/z3c.recipe.scripts\_/src/z3c/recipe/scripts/scripts.py . + You might even be able to adopt some of it by subclassing or delegating. + The Scripts class in that file is the closest to what you might be used + to from zc.recipe.egg. + + Important note for recipe authors: As of buildout 1.5.2, the code in + recipes is *always run with the access to the site-packages as + configured in the buildout section*. + + virtualenv + ========== + + Using virtualenv (http://pypi.python.org/pypi/virtualenv) with the + --no-site-packages option already provided a simple way of using a + system Python. This is intended to continue to work, and some automated + tests exist to demonstrate this. + + However, it is only supported to the degree that people have found it to + work in the past. The existing Buildout tests for virtualenv are only + for problems encountered previously. They are very far from + comprehensive. + + Using Buildout with a system python has at least three advantages over + using Buildout in conjunction with virtualenv. They may or may not be + pertinent to your desired usage. + + - Unlike ``virtualenv --no-site-packages``, Buildout's support allows you + to choose to let packages from your system Python be available to your + software (see ``include-site-packages`` in + http://pypi.python.org/pypi/z3c.recipe.scripts). + + You can even specify which eggs installed in your system Python can be + allowed to fulfill some of your packages' dependencies (see + ``allowed-eggs-from-site-packages`` in + http://pypi.python.org/pypi/z3c.recipe.scripts). + + At the expense of some repeatability and platform dependency, this + flexibility means that, for instance, you can rely on + difficult-to-build eggs like lxml coming from your system Python. + + - Buildout's implementation has a full set of automated tests. + + - An integral Buildout implementation means fewer steps and fewer dependencies + to work with a system Python. + + Detailed Documentation + ********************** + + Buildouts + ========= + + The word "buildout" refers to a description of a set of parts and the + software to create and assemble them. It is often used informally to + refer to an installed system based on a buildout definition. For + example, if we are creating an application named "Foo", then "the Foo + buildout" is the collection of configuration and application-specific + software that allows an instance of the application to be created. We + may refer to such an instance of the application informally as "a Foo + buildout". + + This document describes how to define buildouts using buildout + configuration files and recipes. There are three ways to set up the + buildout software and create a buildout instance: + + 1. Install the zc.buildout egg with easy_install and use the buildout + script installed in a Python scripts area. + + 2. Use the buildout bootstrap script to create a buildout that + includes both the setuptools and zc.buildout eggs. This allows you + to use the buildout software without modifying a Python install. + The buildout script is installed into your buildout local scripts + area. + + 3. Use a buildout command from an already installed buildout to + bootstrap a new buildout. (See the section on bootstraping later + in this document.) + + Often, a software project will be managed in a software repository, + such as a subversion repository, that includes some software source + directories, buildout configuration files, and a copy of the buildout + bootstrap script. To work on the project, one would check out the + project from the repository and run the bootstrap script which + installs setuptools and zc.buildout into the checkout as well as any + parts defined. + + We have a sample buildout that we created using the bootstrap command + of an existing buildout (method 3 above). It has the absolute minimum + information. We have bin, develop-eggs, eggs and parts directories, + and a configuration file: + + >>> ls(sample_buildout) + d bin + - buildout.cfg + d develop-eggs + d eggs + d parts + + The bin directory contains scripts. + + >>> ls(sample_buildout, 'bin') + - buildout + + >>> ls(sample_buildout, 'eggs') + - setuptools-0.6-py2.4.egg + - zc.buildout-1.0-py2.4.egg + + The develop-eggs directory is initially empty: + + >>> ls(sample_buildout, 'develop-eggs') + + The develop-eggs directory holds egg links for software being + developed in the buildout. We separate develop-eggs and other eggs to + allow eggs directories to be shared across multiple buildouts. For + example, a common developer technique is to define a common eggs + directory in their home that all non-develop eggs are stored in. This + allows larger buildouts to be set up much more quickly and saves disk + space. + + The parts directory just contains some helpers for the buildout script + itself. + + >>> ls(sample_buildout, 'parts') + d buildout + + The parts directory provides an area where recipes can install + part data. For example, if we built a custom Python, we would + install it in the part directory. Part data is stored in a + sub-directory of the parts directory with the same name as the part. + + Buildouts are defined using configuration files. These are in the + format defined by the Python ConfigParser module, with extensions + that we'll describe later. By default, when a buildout is run, it + looks for the file buildout.cfg in the directory where the buildout is + run. + + The minimal configuration file has a buildout section that defines no + parts: + + >>> cat(sample_buildout, 'buildout.cfg') + [buildout] + parts = + + A part is simply something to be created by a buildout. It can be + almost anything, such as a Python package, a program, a directory, or + even a configuration file. + + Recipes + ------- + + A part is created by a recipe. Recipes are always installed as Python + eggs. They can be downloaded from a package server, such as the + Python Package Index, or they can be developed as part of a project + using a "develop" egg. + + A develop egg is a special kind of egg that gets installed as an "egg + link" that contains the name of a source directory. Develop eggs + don't have to be packaged for distribution to be used and can be + modified in place, which is especially useful while they are being + developed. + + Let's create a recipe as part of the sample project. We'll create a + recipe for creating directories. First, we'll create a recipes source + directory for our local recipes: + + >>> mkdir(sample_buildout, 'recipes') + + and then we'll create a source file for our mkdir recipe: + + >>> write(sample_buildout, 'recipes', 'mkdir.py', + ... """ + ... import logging, os, zc.buildout + ... + ... class Mkdir: + ... + ... def __init__(self, buildout, name, options): + ... self.name, self.options = name, options + ... options['path'] = os.path.join( + ... buildout['buildout']['directory'], + ... options['path'], + ... ) + ... if not os.path.isdir(os.path.dirname(options['path'])): + ... logging.getLogger(self.name).error( + ... 'Cannot create %s. %s is not a directory.', + ... options['path'], os.path.dirname(options['path'])) + ... raise zc.buildout.UserError('Invalid Path') + ... + ... + ... def install(self): + ... path = self.options['path'] + ... logging.getLogger(self.name).info( + ... 'Creating directory %s', os.path.basename(path)) + ... os.mkdir(path) + ... return path + ... + ... def update(self): + ... pass + ... """) + + Currently, recipes must define 3 methods [#future_recipe_methods]_: + + - a constructor, + + - an install method, and + + - an update method. + + The constructor is responsible for updating a parts options to reflect + data read from other sections. The buildout system keeps track of + whether a part specification has changed. A part specification has + changed if it's options, after adjusting for data read from other + sections, has changed, or if the recipe has changed. Only the options + for the part are considered. If data are read from other sections, + then that information has to be reflected in the parts options. In + the Mkdir example, the given path is interpreted relative to the + buildout directory, and data from the buildout directory is read. The + path option is updated to reflect this. If the directory option was + changed in the buildout sections, we would know to update parts + created using the mkdir recipe using relative path names. + + When buildout is run, it saves configuration data for installed parts + in a file named ".installed.cfg". In subsequent runs, it compares + part-configuration data stored in the .installed.cfg file and the + part-configuration data loaded from the configuration files as + modified by recipe constructors to decide if the configuration of a + part has changed. If the configuration has changed, or if the recipe + has changed, then the part is uninstalled and reinstalled. The + buildout only looks at the part's options, so any data used to + configure the part needs to be reflected in the part's options. It is + the job of a recipe constructor to make sure that the options include + all relevant data. + + Of course, parts are also uninstalled if they are no-longer used. + + The recipe defines a constructor that takes a buildout object, a part + name, and an options dictionary. It saves them in instance attributes. + If the path is relative, we'll interpret it as relative to the + buildout directory. The buildout object passed in is a mapping from + section name to a mapping of options for that section. The buildout + directory is available as the directory option of the buildout + section. We normalize the path and save it back into the options + directory. + + The install method is responsible for creating the part. In this + case, we need the path of the directory to create. We'll use a path + option from our options dictionary. The install method logs what it's + doing using the Python logging call. We return the path that we + installed. If the part is uninstalled or reinstalled, then the path + returned will be removed by the buildout machinery. A recipe install + method is expected to return a string, or an iterable of strings + containing paths to be removed if a part is uninstalled. For most + recipes, this is all of the uninstall support needed. For more complex + uninstallation scenarios use `Uninstall recipes`_. + + The update method is responsible for updating an already installed + part. An empty method is often provided, as in this example, if parts + can't be updated. An update method can return None, a string, or an + iterable of strings. If a string or iterable of strings is returned, + then the saved list of paths to be uninstalled is updated with the new + information by adding any new files returned by the update method. + + We need to provide packaging information so that our recipe can be + installed as a develop egg. The minimum information we need to specify + [#packaging_info]_ is a name. For recipes, we also need to define the + names of the recipe classes as entry points. Packaging information is + provided via a setup.py script: + + >>> write(sample_buildout, 'recipes', 'setup.py', + ... """ + ... from setuptools import setup + ... + ... setup( + ... name = "recipes", + ... entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']}, + ... ) + ... """) + + Our setup script defines an entry point. Entry points provide + a way for an egg to define the services it provides. Here we've said + that we define a zc.buildout entry point named mkdir. Recipe + classes must be exposed as entry points in the zc.buildout group. we + give entry points names within the group. + + We also need a README.txt for our recipes to avoid an annoying warning + from distutils, on which setuptools and zc.buildout are based: + + >>> write(sample_buildout, 'recipes', 'README.txt', " ") + + Now let's update our buildout.cfg: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = mystuff + ... """) + + Let's go through the changes one by one:: + + develop = recipes + + This tells the buildout to install a development egg for our recipes. + Any number of paths can be listed. The paths can be relative or + absolute. If relative, they are treated as relative to the buildout + directory. They can be directory or file paths. If a file path is + given, it should point to a Python setup script. If a directory path + is given, it should point to a directory containing a setup.py file. + Development eggs are installed before building any parts, as they may + provide locally-defined recipes needed by the parts. + + :: + + parts = data-dir + + Here we've named a part to be "built". We can use any name we want + except that different part names must be unique and recipes will often + use the part name to decide what to do. + + :: + + [data-dir] + recipe = recipes:mkdir + path = mystuff + + + When we name a part, we also create a section of the same + name that contains part data. In this section, we'll define + the recipe to be used to install the part. In this case, we also + specify the path to be created. + + Let's run the buildout. We do so by running the build script in the + buildout: + + >>> import os + >>> os.chdir(sample_buildout) + >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Installing data-dir. + data-dir: Creating directory mystuff + + We see that the recipe created the directory, as expected: + + >>> ls(sample_buildout) + - .installed.cfg + d bin + - buildout.cfg + d develop-eggs + d eggs + d mystuff + d parts + d recipes + + In addition, .installed.cfg has been created containing information + about the part we installed: + + >>> cat(sample_buildout, '.installed.cfg') + [buildout] + installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link + parts = data-dir + <BLANKLINE> + [data-dir] + __buildout_installed__ = /sample-buildout/mystuff + __buildout_signature__ = recipes-c7vHV6ekIDUPy/7fjAaYjg== + path = /sample-buildout/mystuff + recipe = recipes:mkdir + + Note that the directory we installed is included in .installed.cfg. + In addition, the path option includes the actual destination + directory. + + If we change the name of the directory in the configuration file, + we'll see that the directory gets removed and recreated: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = mydata + ... """) + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling data-dir. + Installing data-dir. + data-dir: Creating directory mydata + + >>> ls(sample_buildout) + - .installed.cfg + d bin + - buildout.cfg + d develop-eggs + d eggs + d mydata + d parts + d recipes + + If any of the files or directories created by a recipe are removed, + the part will be reinstalled: + + >>> rmdir(sample_buildout, 'mydata') + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling data-dir. + Installing data-dir. + data-dir: Creating directory mydata + + Error reporting + --------------- + + If a user makes an error, an error needs to be printed and work needs + to stop. This is accomplished by logging a detailed error message and + then raising a (or an instance of a subclass of a) + zc.buildout.UserError exception. Raising an error other than a + UserError still displays the error, but labels it as a bug in the + buildout software or recipe. In the sample above, of someone gives a + non-existent directory to create the directory in: + + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = /xxx/mydata + ... """) + + We'll get a user error, not a traceback. + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + data-dir: Cannot create /xxx/mydata. /xxx is not a directory. + While: + Installing. + Getting section data-dir. + Initializing part data-dir. + Error: Invalid Path + + + Recipe Error Handling + --------------------- + + If an error occurs during installation, it is up to the recipe to + clean up any system side effects, such as files created. Let's update + the mkdir recipe to support multiple paths: + + >>> write(sample_buildout, 'recipes', 'mkdir.py', + ... """ + ... import logging, os, zc.buildout + ... + ... class Mkdir: + ... + ... def __init__(self, buildout, name, options): + ... self.name, self.options = name, options + ... + ... # Normalize paths and check that their parent + ... # directories exist: + ... paths = [] + ... for path in options['path'].split(): + ... path = os.path.join(buildout['buildout']['directory'], path) + ... if not os.path.isdir(os.path.dirname(path)): + ... logging.getLogger(self.name).error( + ... 'Cannot create %s. %s is not a directory.', + ... options['path'], os.path.dirname(options['path'])) + ... raise zc.buildout.UserError('Invalid Path') + ... paths.append(path) + ... options['path'] = ' '.join(paths) + ... + ... def install(self): + ... paths = self.options['path'].split() + ... for path in paths: + ... logging.getLogger(self.name).info( + ... 'Creating directory %s', os.path.basename(path)) + ... os.mkdir(path) + ... return paths + ... + ... def update(self): + ... pass + ... """) + + If there is an error creating a path, the install method will exit and + leave previously created paths in place: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = foo bin + ... """) + + >>> print system(buildout), # doctest: +ELLIPSIS + Develop: '/sample-buildout/recipes' + Uninstalling data-dir. + Installing data-dir. + data-dir: Creating directory foo + data-dir: Creating directory bin + While: + Installing data-dir. + <BLANKLINE> + An internal error occurred due to a bug in either zc.buildout or in a + recipe being used: + Traceback (most recent call last): + ... + OSError: [Errno 17] File exists: '/sample-buildout/bin' + + We meant to create a directory bins, but typed bin. Now foo was + left behind. + + >>> os.path.exists('foo') + True + + If we fix the typo: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = foo bins + ... """) + + >>> print system(buildout), # doctest: +ELLIPSIS + Develop: '/sample-buildout/recipes' + Installing data-dir. + data-dir: Creating directory foo + While: + Installing data-dir. + <BLANKLINE> + An internal error occurred due to a bug in either zc.buildout or in a + recipe being used: + Traceback (most recent call last): + ... + OSError: [Errno 17] File exists: '/sample-buildout/foo' + + Now they fail because foo exists, because it was left behind. + + >>> remove('foo') + + Let's fix the recipe: + + >>> write(sample_buildout, 'recipes', 'mkdir.py', + ... """ + ... import logging, os, zc.buildout + ... + ... class Mkdir: + ... + ... def __init__(self, buildout, name, options): + ... self.name, self.options = name, options + ... + ... # Normalize paths and check that their parent + ... # directories exist: + ... paths = [] + ... for path in options['path'].split(): + ... path = os.path.join(buildout['buildout']['directory'], path) + ... if not os.path.isdir(os.path.dirname(path)): + ... logging.getLogger(self.name).error( + ... 'Cannot create %s. %s is not a directory.', + ... options['path'], os.path.dirname(options['path'])) + ... raise zc.buildout.UserError('Invalid Path') + ... paths.append(path) + ... options['path'] = ' '.join(paths) + ... + ... def install(self): + ... paths = self.options['path'].split() + ... created = [] + ... try: + ... for path in paths: + ... logging.getLogger(self.name).info( + ... 'Creating directory %s', os.path.basename(path)) + ... os.mkdir(path) + ... created.append(path) + ... except: + ... for d in created: + ... os.rmdir(d) + ... raise + ... + ... return paths + ... + ... def update(self): + ... pass + ... """) + + And put back the typo: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = foo bin + ... """) + + When we rerun the buildout: + + >>> print system(buildout), # doctest: +ELLIPSIS + Develop: '/sample-buildout/recipes' + Installing data-dir. + data-dir: Creating directory foo + data-dir: Creating directory bin + While: + Installing data-dir. + <BLANKLINE> + An internal error occurred due to a bug in either zc.buildout or in a + recipe being used: + Traceback (most recent call last): + ... + OSError: [Errno 17] File exists: '/sample-buildout/bin' + + .. Wait for the file to really disappear. My linux is weird. + + >>> wait_until("foo goes away", lambda : not os.path.exists('foo'), + ... timeout=200) + + we get the same error, but we don't get the directory left behind: + + >>> os.path.exists('foo') + False + + It's critical that recipes clean up partial effects when errors + occur. Because recipes most commonly create files and directories, + buildout provides a helper API for removing created files when an + error occurs. Option objects have a created method that can be called + to record files as they are created. If the install or update method + returns with an error, then any registered paths are removed + automatically. The method returns the files registered and can be + used to return the files created. Let's use this API to simplify the + recipe: + + >>> write(sample_buildout, 'recipes', 'mkdir.py', + ... """ + ... import logging, os, zc.buildout + ... + ... class Mkdir: + ... + ... def __init__(self, buildout, name, options): + ... self.name, self.options = name, options + ... + ... # Normalize paths and check that their parent + ... # directories exist: + ... paths = [] + ... for path in options['path'].split(): + ... path = os.path.join(buildout['buildout']['directory'], path) + ... if not os.path.isdir(os.path.dirname(path)): + ... logging.getLogger(self.name).error( + ... 'Cannot create %s. %s is not a directory.', + ... options['path'], os.path.dirname(options['path'])) + ... raise zc.buildout.UserError('Invalid Path') + ... paths.append(path) + ... options['path'] = ' '.join(paths) + ... + ... def install(self): + ... paths = self.options['path'].split() + ... for path in paths: + ... logging.getLogger(self.name).info( + ... 'Creating directory %s', os.path.basename(path)) + ... os.mkdir(path) + ... self.options.created(path) + ... + ... return self.options.created() + ... + ... def update(self): + ... pass + ... """) + + .. + + >>> remove(sample_buildout, 'recipes', 'mkdir.pyc') + + We returned by calling created, taking advantage of the fact that it + returns the registered paths. We did this for illustrative purposes. + It would be simpler just to return the paths as before. + + If we rerun the buildout, again, we'll get the error and no + directories will be created: + + >>> print system(buildout), # doctest: +ELLIPSIS + Develop: '/sample-buildout/recipes' + Installing data-dir. + data-dir: Creating directory foo + data-dir: Creating directory bin + While: + Installing data-dir. + <BLANKLINE> + An internal error occurred due to a bug in either zc.buildout or in a + recipe being used: + Traceback (most recent call last): + ... + OSError: [Errno 17] File exists: '/sample-buildout/bin' + + >>> os.path.exists('foo') + False + + Now, we'll fix the typo again and we'll get the directories we expect: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = foo bins + ... """) + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Installing data-dir. + data-dir: Creating directory foo + data-dir: Creating directory bins + + >>> os.path.exists('foo') + True + >>> os.path.exists('bins') + True + + Configuration file syntax + ------------------------- + + As mentioned earlier, buildout configuration files use the format + defined by the Python ConfigParser module with extensions. The + extensions are: + + - option names are case sensitive + + - option values can use a substitution syntax, described below, to + refer to option values in specific sections. + + - option values can be appended or removed using the - and + + operators. + + The ConfigParser syntax is very flexible. Section names can contain + any characters other than newlines and right square braces ("]"). + Option names can contain any characters other than newlines, colons, + and equal signs, can not start with a space, and don't include + trailing spaces. + + It is likely that, in the future, some characters will be given + special buildout-defined meanings. This is already true of the + characters ":", "$", "%", "(", and ")". For now, it is a good idea to + keep section and option names simple, sticking to alphanumeric + characters, hyphens, and periods. + + Annotated sections + ------------------ + + When used with the `annotate` command, buildout displays annotated sections. + All sections are displayed, sorted alphabetically. For each section, + all key-value pairs are displayed, sorted alphabetically, along with + the origin of the value (file name or COMPUTED_VALUE, DEFAULT_VALUE, + COMMAND_LINE_VALUE). + + >>> print system(buildout+ ' annotate'), + ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + <BLANKLINE> + Annotated sections + ================== + <BLANKLINE> + [buildout] + accept-buildout-test-releases= false + DEFAULT_VALUE + allow-hosts= * + DEFAULT_VALUE + allow-picked-versions= true + DEFAULT_VALUE + allowed-eggs-from-site-packages= * + DEFAULT_VALUE + bin-directory= bin + DEFAULT_VALUE + develop= recipes + /sample-buildout/buildout.cfg + develop-eggs-directory= develop-eggs + DEFAULT_VALUE + directory= /sample-buildout + COMPUTED_VALUE + eggs-directory= eggs + DEFAULT_VALUE + exec-sitecustomize= true + DEFAULT_VALUE + executable= ... + DEFAULT_VALUE + find-links= + DEFAULT_VALUE + include-site-packages= true + DEFAULT_VALUE + install-from-cache= false + DEFAULT_VALUE + installed= .installed.cfg + DEFAULT_VALUE + log-format= + DEFAULT_VALUE + log-level= INFO + DEFAULT_VALUE + newest= true + DEFAULT_VALUE + offline= false + DEFAULT_VALUE + parts= data-dir + /sample-buildout/buildout.cfg + parts-directory= parts + DEFAULT_VALUE + prefer-final= false + DEFAULT_VALUE + python= buildout + DEFAULT_VALUE + relative-paths= false + DEFAULT_VALUE + socket-timeout= + DEFAULT_VALUE + unzip= false + DEFAULT_VALUE + use-dependency-links= true + DEFAULT_VALUE + <BLANKLINE> + [data-dir] + path= foo bins + /sample-buildout/buildout.cfg + recipe= recipes:mkdir + /sample-buildout/buildout.cfg + <BLANKLINE> + + Variable substitutions + ---------------------- + + Buildout configuration files support variable substitution. + To illustrate this, we'll create an debug recipe to + allow us to see interactions with the buildout: + + >>> write(sample_buildout, 'recipes', 'debug.py', + ... """ + ... class Debug: + ... + ... def __init__(self, buildout, name, options): + ... self.buildout = buildout + ... self.name = name + ... self.options = options + ... + ... def install(self): + ... items = self.options.items() + ... items.sort() + ... for option, value in items: + ... print option, value + ... return () + ... + ... update = install + ... """) + + This recipe doesn't actually create anything. The install method + doesn't return anything, because it didn't create any files or + directories. + + We also have to update our setup script: + + >>> write(sample_buildout, 'recipes', 'setup.py', + ... """ + ... from setuptools import setup + ... entry_points = ( + ... ''' + ... [zc.buildout] + ... mkdir = mkdir:Mkdir + ... debug = debug:Debug + ... ''') + ... setup(name="recipes", entry_points=entry_points) + ... """) + + We've rearranged the script a bit to make the entry points easier to + edit. In particular, entry points are now defined as a configuration + string, rather than a dictionary. + + Let's update our configuration to provide variable substitution + examples: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir debug + ... log-level = INFO + ... + ... [debug] + ... recipe = recipes:debug + ... File 1 = ${data-dir:path}/file + ... File 2 = ${debug:File 1}/log + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = mydata + ... """) + + We used a string-template substitution for File 1 and File 2. This + type of substitution uses the string.Template syntax. Names + substituted are qualified option names, consisting of a section name + and option name joined by a colon. + + Now, if we run the buildout, we'll see the options with the values + substituted. + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling data-dir. + Installing data-dir. + data-dir: Creating directory mydata + Installing debug. + File 1 /sample-buildout/mydata/file + File 2 /sample-buildout/mydata/file/log + recipe recipes:debug + + Note that the substitution of the data-dir path option reflects the + update to the option performed by the mkdir recipe. + + It might seem surprising that mydata was created again. This is + because we changed our recipes package by adding the debug module. + The buildout system didn't know if this module could effect the mkdir + recipe, so it assumed it could and reinstalled mydata. If we rerun + the buildout: + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Updating data-dir. + Updating debug. + File 1 /sample-buildout/mydata/file + File 2 /sample-buildout/mydata/file/log + recipe recipes:debug + + We can see that mydata was not recreated. + + Note that, in this case, we didn't specify a log level, so + we didn't get output about what the buildout was doing. + + Section and option names in variable substitutions are only allowed to + contain alphanumeric characters, hyphens, periods and spaces. This + restriction might be relaxed in future releases. + + We can ommit the section name in a variable substitution to refer to + the current section. We can also use the special option, + _buildout_section_name_ to get the current section name. + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir debug + ... log-level = INFO + ... + ... [debug] + ... recipe = recipes:debug + ... File 1 = ${data-dir:path}/file + ... File 2 = ${:File 1}/log + ... my_name = ${:_buildout_section_name_} + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = mydata + ... """) + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Updating data-dir. + Installing debug. + File 1 /sample-buildout/mydata/file + File 2 /sample-buildout/mydata/file/log + my_name debug + recipe recipes:debug + + Automatic part selection and ordering + ------------------------------------- + + When a section with a recipe is referred to, either through variable + substitution or by an initializing recipe, the section is treated as a + part and added to the part list before the referencing part. For + example, we can leave data-dir out of the parts list: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug + ... log-level = INFO + ... + ... [debug] + ... recipe = recipes:debug + ... File 1 = ${data-dir:path}/file + ... File 2 = ${debug:File 1}/log + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = mydata + ... """) + + + It will still be treated as a part: + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Updating data-dir. + Installing debug. + File 1 /sample-buildout/mydata/file + File 2 /sample-buildout/mydata/file/log + recipe recipes:debug + + >>> cat('.installed.cfg') # doctest: +ELLIPSIS + [buildout] + installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link + parts = data-dir debug + ... + + Note that the data-dir part is included *before* the debug part, + because the debug part refers to the data-dir part. Even if we list + the data-dir part after the debug part, it will be included before: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug data-dir + ... log-level = INFO + ... + ... [debug] + ... recipe = recipes:debug + ... File 1 = ${data-dir:path}/file + ... File 2 = ${debug:File 1}/log + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = mydata + ... """) + + + It will still be treated as a part: + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Updating data-dir. + Updating debug. + File 1 /sample-buildout/mydata/file + File 2 /sample-buildout/mydata/file/log + recipe recipes:debug + + >>> cat('.installed.cfg') # doctest: +ELLIPSIS + [buildout] + installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link + parts = data-dir debug + ... + + Extending sections (macros) + --------------------------- + + A section (other than the buildout section) can extend one or more + other sections using the ``<=`` option. Options from the referenced + sections are copied to the refering section *before* variable + substitution. This, together with the ability to refer to variables + of the current section allows sections to be used as macros. + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = myfiles + ... log-level = INFO + ... + ... [debug] + ... recipe = recipes:debug + ... + ... [with_file1] + ... <= debug + ... file1 = ${:path}/file1 + ... color = red + ... + ... [with_file2] + ... <= debug + ... file2 = ${:path}/file2 + ... color = blue + ... + ... [myfiles] + ... <= with_file1 + ... with_file2 + ... path = mydata + ... """) + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Uninstalling data-dir. + Installing myfiles. + color blue + file1 mydata/file1 + file2 mydata/file2 + path mydata + recipe recipes:debug + + In this example, the debug, with_file1 and with_file2 sections act as + macros. In particular, the variable substitutions are performed + relative to the myfiles section. + + Adding and removing options + --------------------------- + + We can append and remove values to an option by using the + and - + operators. + + This is illustrated below; first we define a base configuration. + + >>> write(sample_buildout, 'base.cfg', + ... """ + ... [buildout] + ... parts = part1 part2 part3 + ... + ... [part1] + ... recipe = + ... option = a1 a2 + ... + ... [part2] + ... recipe = + ... option = b1 b2 b3 b4 + ... + ... [part3] + ... recipe = + ... option = c1 c2 + ... + ... """) + + Extending this configuration, we can "adjust" the values set in the + base configuration file. + + >>> write(sample_buildout, 'extension1.cfg', + ... """ + ... [buildout] + ... extends = base.cfg + ... + ... # appending values + ... [part1] + ... option += a3 a4 + ... + ... # removing values + ... [part2] + ... option -= b1 b2 + ... + ... # alt. spelling + ... [part3] + ... option+=c3 c4 c5 + ... + ... # normal assignment + ... [part4] + ... option = h1 h2 + ... + ... """) + + An additional extension. + + >>> write(sample_buildout, 'extension2.cfg', + ... """ + ... [buildout] + ... extends = extension1.cfg + ... + ... # appending values + ... [part1] + ... option += a5 + ... + ... # removing values + ... [part2] + ... option -= b1 b2 b3 + ... + ... """) + + To verify that the options are adjusted correctly, we'll set up an + extension that prints out the options. + + >>> mkdir(sample_buildout, 'demo') + >>> write(sample_buildout, 'demo', 'demo.py', + ... """ + ... def ext(buildout): + ... print [part['option'] for name, part in buildout.items() \ + ... if name.startswith('part')] + ... """) + + >>> write(sample_buildout, 'demo', 'setup.py', + ... """ + ... from setuptools import setup + ... + ... setup( + ... name="demo", + ... entry_points={'zc.buildout.extension': ['ext = demo:ext']}, + ... ) + ... """) + + Set up a buildout configuration for this extension. + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = demo + ... parts = + ... """) + + >>> os.chdir(sample_buildout) + >>> print system(os.path.join(sample_buildout, 'bin', 'buildout')), + Develop: '/sample-buildout/demo' + Uninstalling myfiles. + Getting distribution for 'recipes'. + zip_safe flag not set; analyzing archive contents... + Got recipes 0.0.0. + warning: install_lib: 'build/lib' does not exist -- no Python modules to install + + Verify option values. + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = demo + ... extensions = demo + ... extends = extension2.cfg + ... """) + + >>> print system(os.path.join('bin', 'buildout')), + ['a1 a2/na3 a4/na5', 'b1 b2 b3 b4', 'c1 c2/nc3 c4 c5', 'h1 h2'] + Develop: '/sample-buildout/demo' + + Annotated sections output shows which files are responsible for which + operations. + + >>> print system(os.path.join('bin', 'buildout') + ' annotate'), + ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + <BLANKLINE> + Annotated sections + ================== + ... + <BLANKLINE> + [part1] + option= a1 a2 + a3 a4 + a5 + /sample-buildout/base.cfg + += /sample-buildout/extension1.cfg + += /sample-buildout/extension2.cfg + recipe= + /sample-buildout/base.cfg + <BLANKLINE> + [part2] + option= b1 b2 b3 b4 + /sample-buildout/base.cfg + -= /sample-buildout/extension1.cfg + -= /sample-buildout/extension2.cfg + recipe= + /sample-buildout/base.cfg + <BLANKLINE> + [part3] + option= c1 c2 + c3 c4 c5 + /sample-buildout/base.cfg + += /sample-buildout/extension1.cfg + recipe= + /sample-buildout/base.cfg + <BLANKLINE> + [part4] + option= h1 h2 + /sample-buildout/extension1.cfg + + Cleanup. + + >>> os.remove(os.path.join(sample_buildout, 'base.cfg')) + >>> os.remove(os.path.join(sample_buildout, 'extension1.cfg')) + >>> os.remove(os.path.join(sample_buildout, 'extension2.cfg')) + + Multiple configuration files + ---------------------------- + + A configuration file can "extend" another configuration file. + Options are read from the other configuration file if they aren't + already defined by your configuration file. + + The configuration files your file extends can extend + other configuration files. The same file may be + used more than once although, of course, cycles aren't allowed. + + To see how this works, we use an example: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... extends = base.cfg + ... + ... [debug] + ... op = buildout + ... """) + + >>> write(sample_buildout, 'base.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug + ... + ... [debug] + ... recipe = recipes:debug + ... op = base + ... """) + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Installing debug. + op buildout + recipe recipes:debug + + The example is pretty trivial, but the pattern it illustrates is + pretty common. In a more practical example, the base buildout might + represent a product and the extending buildout might be a + customization. + + Here is a more elaborate example. + + >>> other = tmpdir('other') + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... extends = b1.cfg b2.cfg %(b3)s + ... + ... [debug] + ... op = buildout + ... """ % dict(b3=os.path.join(other, 'b3.cfg'))) + + >>> write(sample_buildout, 'b1.cfg', + ... """ + ... [buildout] + ... extends = base.cfg + ... + ... [debug] + ... op1 = b1 1 + ... op2 = b1 2 + ... """) + + >>> write(sample_buildout, 'b2.cfg', + ... """ + ... [buildout] + ... extends = base.cfg + ... + ... [debug] + ... op2 = b2 2 + ... op3 = b2 3 + ... """) + + >>> write(other, 'b3.cfg', + ... """ + ... [buildout] + ... extends = b3base.cfg + ... + ... [debug] + ... op4 = b3 4 + ... """) + + >>> write(other, 'b3base.cfg', + ... """ + ... [debug] + ... op5 = b3base 5 + ... """) + + >>> write(sample_buildout, 'base.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug + ... + ... [debug] + ... recipe = recipes:debug + ... name = base + ... """) + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Installing debug. + name base + op buildout + op1 b1 1 + op2 b2 2 + op3 b2 3 + op4 b3 4 + op5 b3base 5 + recipe recipes:debug + + There are several things to note about this example: + + - We can name multiple files in an extends option. + + - We can reference files recursively. + + - Relative file names in extended options are interpreted relative to + the directory containing the referencing configuration file. + + Loading Configuration from URLs + ------------------------------- + + Configuration files can be loaded from URLs. To see how this works, + we'll set up a web server with some configuration files. + + >>> server_data = tmpdir('server_data') + + >>> write(server_data, "r1.cfg", + ... """ + ... [debug] + ... op1 = r1 1 + ... op2 = r1 2 + ... """) + + >>> write(server_data, "r2.cfg", + ... """ + ... [buildout] + ... extends = r1.cfg + ... + ... [debug] + ... op2 = r2 2 + ... op3 = r2 3 + ... """) + + >>> server_url = start_server(server_data) + + >>> write('client.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug + ... extends = %(url)s/r2.cfg + ... + ... [debug] + ... recipe = recipes:debug + ... name = base + ... """ % dict(url=server_url)) + + + >>> print system(buildout+ ' -c client.cfg'), + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Installing debug. + name base + op1 r1 1 + op2 r2 2 + op3 r2 3 + recipe recipes:debug + + Here we specified a URL for the file we extended. The file we + downloaded, itself referred to a file on the server using a relative + URL reference. Relative references are interpreted relative to the + base URL when they appear in configuration files loaded via URL. + + We can also specify a URL as the configuration file to be used by a + buildout. + + >>> os.remove('client.cfg') + >>> write(server_data, 'remote.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug + ... extends = r2.cfg + ... + ... [debug] + ... recipe = recipes:debug + ... name = remote + ... """) + + >>> print system(buildout + ' -c ' + server_url + '/remote.cfg'), + While: + Initializing. + Error: Missing option: buildout:directory + + Normally, the buildout directory defaults to directory + containing a configuration file. This won't work for configuration + files loaded from URLs. In this case, the buildout directory would + normally be defined on the command line: + + >>> print system(buildout + ... + ' -c ' + server_url + '/remote.cfg' + ... + ' buildout:directory=' + sample_buildout + ... ), + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Installing debug. + name remote + op1 r1 1 + op2 r2 2 + op3 r2 3 + recipe recipes:debug + + User defaults + ------------- + + If the file $HOME/.buildout/default.cfg, exists, it is read before + reading the configuration file. ($HOME is the value of the HOME + environment variable. The '/' is replaced by the operating system file + delimiter.) + + >>> old_home = os.environ['HOME'] + >>> home = tmpdir('home') + >>> mkdir(home, '.buildout') + >>> write(home, '.buildout', 'default.cfg', + ... """ + ... [debug] + ... op1 = 1 + ... op7 = 7 + ... """) + + >>> os.environ['HOME'] = home + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Installing debug. + name base + op buildout + op1 b1 1 + op2 b2 2 + op3 b2 3 + op4 b3 4 + op5 b3base 5 + op7 7 + recipe recipes:debug + + A buildout command-line argument, -U, can be used to suppress reading + user defaults: + + >>> print system(buildout + ' -U'), + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Installing debug. + name base + op buildout + op1 b1 1 + op2 b2 2 + op3 b2 3 + op4 b3 4 + op5 b3base 5 + recipe recipes:debug + + >>> os.environ['HOME'] = old_home + + Log level + --------- + + We can control the level of logging by specifying a log level in out + configuration file. For example, so suppress info messages, we can + set the logging level to WARNING + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... log-level = WARNING + ... extends = b1.cfg b2.cfg + ... """) + + >>> print system(buildout), + name base + op1 b1 1 + op2 b2 2 + op3 b2 3 + recipe recipes:debug + + Uninstall recipes + ----------------- + + As we've seen, when parts are installed, buildout keeps track of files + and directories that they create. When the parts are uninstalled these + files and directories are deleted. + + Sometimes more clean up is needed. For example, a recipe might add a + system service by calling chkconfig --add during installation. Later + during uninstallation, chkconfig --del will need to be called to + remove the system service. + + In order to deal with these uninstallation issues, you can register + uninstall recipes. Uninstall recipes are registered using the + 'zc.buildout.uninstall' entry point. Parts specify uninstall recipes + using the 'uninstall' option. + + In comparison to regular recipes, uninstall recipes are much + simpler. They are simply callable objects that accept the name of the + part to be uninstalled and the part's options dictionary. Uninstall + recipes don't have access to the part itself since it maybe not be + able to be instantiated at uninstallation time. + + Here's a recipe that simulates installation of a system service, along + with an uninstall recipe that simulates removing the service. + + >>> write(sample_buildout, 'recipes', 'service.py', + ... """ + ... class Service: + ... + ... def __init__(self, buildout, name, options): + ... self.buildout = buildout + ... self.name = name + ... self.options = options + ... + ... def install(self): + ... print "chkconfig --add %s" % self.options['script'] + ... return () + ... + ... def update(self): + ... pass + ... + ... + ... def uninstall_service(name, options): + ... print "chkconfig --del %s" % options['script'] + ... """) + + To use these recipes we must register them using entry points. Make + sure to use the same name for the recipe and uninstall recipe. This is + required to let buildout know which uninstall recipe goes with which + recipe. + + >>> write(sample_buildout, 'recipes', 'setup.py', + ... """ + ... from setuptools import setup + ... entry_points = ( + ... ''' + ... [zc.buildout] + ... mkdir = mkdir:Mkdir + ... debug = debug:Debug + ... service = service:Service + ... + ... [zc.buildout.uninstall] + ... service = service:uninstall_service + ... ''') + ... setup(name="recipes", entry_points=entry_points) + ... """) + + Here's how these recipes could be used in a buildout: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = service + ... + ... [service] + ... recipe = recipes:service + ... script = /path/to/script + ... """) + + When the buildout is run the service will be installed + + >>> print system(buildout) + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Installing service. + chkconfig --add /path/to/script + <BLANKLINE> + + The service has been installed. If the buildout is run again with no + changes, the service shouldn't be changed. + + >>> print system(buildout) + Develop: '/sample-buildout/recipes' + Updating service. + <BLANKLINE> + + Now we change the service part to trigger uninstallation and + re-installation. + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = service + ... + ... [service] + ... recipe = recipes:service + ... script = /path/to/a/different/script + ... """) + + >>> print system(buildout) + Develop: '/sample-buildout/recipes' + Uninstalling service. + Running uninstall recipe. + chkconfig --del /path/to/script + Installing service. + chkconfig --add /path/to/a/different/script + <BLANKLINE> + + Now we remove the service part, and add another part. + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug + ... + ... [debug] + ... recipe = recipes:debug + ... """) + + >>> print system(buildout) + Develop: '/sample-buildout/recipes' + Uninstalling service. + Running uninstall recipe. + chkconfig --del /path/to/a/different/script + Installing debug. + recipe recipes:debug + <BLANKLINE> + + Uninstall recipes don't have to take care of removing all the files + and directories created by the part. This is still done automatically, + following the execution of the uninstall recipe. An upshot is that an + uninstallation recipe can access files and directories created by a + recipe before they are deleted. + + For example, here's an uninstallation recipe that simulates backing up + a directory before it is deleted. It is designed to work with the + mkdir recipe introduced earlier. + + >>> write(sample_buildout, 'recipes', 'backup.py', + ... """ + ... import os + ... def backup_directory(name, options): + ... path = options['path'] + ... size = len(os.listdir(path)) + ... print "backing up directory %s of size %s" % (path, size) + ... """) + + It must be registered with the zc.buildout.uninstall entry + point. Notice how it is given the name 'mkdir' to associate it with + the mkdir recipe. + + >>> write(sample_buildout, 'recipes', 'setup.py', + ... """ + ... from setuptools import setup + ... entry_points = ( + ... ''' + ... [zc.buildout] + ... mkdir = mkdir:Mkdir + ... debug = debug:Debug + ... service = service:Service + ... + ... [zc.buildout.uninstall] + ... uninstall_service = service:uninstall_service + ... mkdir = backup:backup_directory + ... ''') + ... setup(name="recipes", entry_points=entry_points) + ... """) + + Now we can use it with a mkdir part. + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = dir debug + ... + ... [dir] + ... recipe = recipes:mkdir + ... path = my_directory + ... + ... [debug] + ... recipe = recipes:debug + ... """) + + Run the buildout to install the part. + + >>> print system(buildout) + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Installing dir. + dir: Creating directory my_directory + Installing debug. + recipe recipes:debug + <BLANKLINE> + + Now we remove the part from the configuration file. + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug + ... + ... [debug] + ... recipe = recipes:debug + ... """) + + When the buildout is run the part is removed, and the uninstall recipe + is run before the directory is deleted. + + >>> print system(buildout) + Develop: '/sample-buildout/recipes' + Uninstalling dir. + Running uninstall recipe. + backing up directory /sample-buildout/my_directory of size 0 + Updating debug. + recipe recipes:debug + <BLANKLINE> + + Now we will return the registration to normal for the benefit of the + rest of the examples. + + >>> write(sample_buildout, 'recipes', 'setup.py', + ... """ + ... from setuptools import setup + ... entry_points = ( + ... ''' + ... [zc.buildout] + ... mkdir = mkdir:Mkdir + ... debug = debug:Debug + ... ''') + ... setup(name="recipes", entry_points=entry_points) + ... """) + + + Command-line usage + ------------------ + + A number of arguments can be given on the buildout command line. The + command usage is:: + + buildout [options and assignments] [command [command arguments]] + + The following options are supported: + + -h (or --help) + Print basic usage information. If this option is used, then all + other options are ignored. + + -c filename + The -c option can be used to specify a configuration file, rather than + buildout.cfg in the current directory. + + + -t socket_timeout + + Specify the socket timeout in seconds. + + -v + Increment the verbosity by 10. The verbosity is used to adjust + the logging level. The verbosity is subtracted from the numeric + value of the log-level option specified in the configuration file. + + -q + Decrement the verbosity by 10. + + -U + Don't read user-default configuration. + + -o + Run in off-line mode. This is equivalent to the assignment + buildout:offline=true. + + -O + Run in non-off-line mode. This is equivalent to the assignment + buildout:offline=false. This is the default buildout mode. The + -O option would normally be used to override a true offline + setting in a configuration file. + + -n + Run in newest mode. This is equivalent to the assignment + buildout:newest=true. With this setting, which is the default, + buildout will try to find the newest versions of distributions + available that satisfy its requirements. + + -N + Run in non-newest mode. This is equivalent to the assignment + buildout:newest=false. With this setting, buildout will not seek + new distributions if installed distributions satisfy it's + requirements. + + Assignments are of the form:: + + section_name:option_name=value + + Options and assignments can be given in any order. + + Here's an example: + + >>> write(sample_buildout, 'other.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug + ... installed = .other.cfg + ... log-level = WARNING + ... + ... [debug] + ... name = other + ... recipe = recipes:debug + ... """) + + Note that we used the installed buildout option to specify an + alternate file to store information about installed parts. + + >>> print system(buildout+' -c other.cfg debug:op1=foo -v'), + Develop: '/sample-buildout/recipes' + Installing debug. + name other + op1 foo + recipe recipes:debug + + Here we used the -c option to specify an alternate configuration file, + and the -v option to increase the level of logging from the default, + WARNING. + + Options can also be combined in the usual Unix way, as in: + + >>> print system(buildout+' -vcother.cfg debug:op1=foo'), + Develop: '/sample-buildout/recipes' + Updating debug. + name other + op1 foo + recipe recipes:debug + + Here we combined the -v and -c options with the configuration file + name. Note that the -c option has to be last, because it takes an + argument. + + >>> os.remove(os.path.join(sample_buildout, 'other.cfg')) + >>> os.remove(os.path.join(sample_buildout, '.other.cfg')) + + The most commonly used command is 'install' and it takes a list of + parts to install. if any parts are specified, only those parts are + installed. To illustrate this, we'll update our configuration and run + the buildout in the usual way: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug d1 d2 d3 + ... + ... [d1] + ... recipe = recipes:mkdir + ... path = d1 + ... + ... [d2] + ... recipe = recipes:mkdir + ... path = d2 + ... + ... [d3] + ... recipe = recipes:mkdir + ... path = d3 + ... + ... [debug] + ... recipe = recipes:debug + ... """) + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling debug. + Installing debug. + recipe recipes:debug + Installing d1. + d1: Creating directory d1 + Installing d2. + d2: Creating directory d2 + Installing d3. + d3: Creating directory d3 + + >>> ls(sample_buildout) + - .installed.cfg + - b1.cfg + - b2.cfg + - base.cfg + d bin + - buildout.cfg + d d1 + d d2 + d d3 + d demo + d develop-eggs + d eggs + d parts + d recipes + + >>> cat(sample_buildout, '.installed.cfg') + ... # doctest: +NORMALIZE_WHITESPACE + [buildout] + installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link + parts = debug d1 d2 d3 + <BLANKLINE> + [debug] + __buildout_installed__ = + __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== + recipe = recipes:debug + <BLANKLINE> + [d1] + __buildout_installed__ = /sample-buildout/d1 + __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== + path = /sample-buildout/d1 + recipe = recipes:mkdir + <BLANKLINE> + [d2] + __buildout_installed__ = /sample-buildout/d2 + __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== + path = /sample-buildout/d2 + recipe = recipes:mkdir + <BLANKLINE> + [d3] + __buildout_installed__ = /sample-buildout/d3 + __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== + path = /sample-buildout/d3 + recipe = recipes:mkdir + + Now we'll update our configuration file: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug d2 d3 d4 + ... + ... [d2] + ... recipe = recipes:mkdir + ... path = data2 + ... + ... [d3] + ... recipe = recipes:mkdir + ... path = data3 + ... + ... [d4] + ... recipe = recipes:mkdir + ... path = ${d2:path}-extra + ... + ... [debug] + ... recipe = recipes:debug + ... x = 1 + ... """) + + and run the buildout specifying just d3 and d4: + + >>> print system(buildout+' install d3 d4'), + Develop: '/sample-buildout/recipes' + Uninstalling d3. + Installing d3. + d3: Creating directory data3 + Installing d4. + d4: Creating directory data2-extra + + >>> ls(sample_buildout) + - .installed.cfg + - b1.cfg + - b2.cfg + - base.cfg + d bin + - buildout.cfg + d d1 + d d2 + d data2-extra + d data3 + d demo + d develop-eggs + d eggs + d parts + d recipes + + Only the d3 and d4 recipes ran. d3 was removed and data3 and data2-extra + were created. + + The .installed.cfg is only updated for the recipes that ran: + + >>> cat(sample_buildout, '.installed.cfg') + ... # doctest: +NORMALIZE_WHITESPACE + [buildout] + installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link + parts = debug d1 d2 d3 d4 + <BLANKLINE> + [debug] + __buildout_installed__ = + __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== + recipe = recipes:debug + <BLANKLINE> + [d1] + __buildout_installed__ = /sample-buildout/d1 + __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== + path = /sample-buildout/d1 + recipe = recipes:mkdir + <BLANKLINE> + [d2] + __buildout_installed__ = /sample-buildout/d2 + __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== + path = /sample-buildout/d2 + recipe = recipes:mkdir + <BLANKLINE> + [d3] + __buildout_installed__ = /sample-buildout/data3 + __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== + path = /sample-buildout/data3 + recipe = recipes:mkdir + <BLANKLINE> + [d4] + __buildout_installed__ = /sample-buildout/data2-extra + __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== + path = /sample-buildout/data2-extra + recipe = recipes:mkdir + + Note that the installed data for debug, d1, and d2 haven't changed, + because we didn't install those parts and that the d1 and d2 + directories are still there. + + Now, if we run the buildout without the install command: + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Uninstalling d2. + Uninstalling d1. + Uninstalling debug. + Installing debug. + recipe recipes:debug + x 1 + Installing d2. + d2: Creating directory data2 + Updating d3. + Updating d4. + + We see the output of the debug recipe and that data2 was created. We + also see that d1 and d2 have gone away: + + >>> ls(sample_buildout) + - .installed.cfg + - b1.cfg + - b2.cfg + - base.cfg + d bin + - buildout.cfg + d data2 + d data2-extra + d data3 + d demo + d develop-eggs + d eggs + d parts + d recipes + + Alternate directory and file locations + -------------------------------------- + + The buildout normally puts the bin, eggs, and parts directories in the + directory in the directory containing the configuration file. You can + provide alternate locations, and even names for these directories. + + >>> alt = tmpdir('sample-alt') + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = + ... develop-eggs-directory = %(developbasket)s + ... eggs-directory = %(basket)s + ... bin-directory = %(scripts)s + ... parts-directory = %(work)s + ... """ % dict( + ... developbasket = os.path.join(alt, 'developbasket'), + ... basket = os.path.join(alt, 'basket'), + ... scripts = os.path.join(alt, 'scripts'), + ... work = os.path.join(alt, 'work'), + ... )) + + >>> print system(buildout), + Creating directory '/sample-alt/scripts'. + Creating directory '/sample-alt/work'. + Creating directory '/sample-alt/basket'. + Creating directory '/sample-alt/developbasket'. + Develop: '/sample-buildout/recipes' + Uninstalling d4. + Uninstalling d3. + Uninstalling d2. + Uninstalling debug. + + >>> ls(alt) + d basket + d developbasket + d scripts + d work + + >>> ls(alt, 'developbasket') + - recipes.egg-link + + You can also specify an alternate buildout directory: + + >>> rmdir(alt) + >>> alt = tmpdir('sample-alt') + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... directory = %(alt)s + ... develop = %(recipes)s + ... parts = + ... """ % dict( + ... alt=alt, + ... recipes=os.path.join(sample_buildout, 'recipes'), + ... )) + + >>> print system(buildout), + Creating directory '/sample-alt/bin'. + Creating directory '/sample-alt/parts'. + Creating directory '/sample-alt/eggs'. + Creating directory '/sample-alt/develop-eggs'. + Develop: '/sample-buildout/recipes' + + >>> ls(alt) + - .installed.cfg + d bin + d develop-eggs + d eggs + d parts + + >>> ls(alt, 'develop-eggs') + - recipes.egg-link + + Logging control + --------------- + + Three buildout options are used to control logging: + + log-level + specifies the log level + + verbosity + adjusts the log level + + log-format + allows an alternate logging for mat to be specified + + We've already seen the log level and verbosity. Let's look at an example + of changing the format: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = + ... log-level = 25 + ... verbosity = 5 + ... log-format = %(levelname)s %(message)s + ... """) + + Here, we've changed the format to include the log-level name, rather + than the logger name. + + We've also illustrated, with a contrived example, that the log level + can be a numeric value and that the verbosity can be specified in the + configuration file. Because the verbosity is subtracted from the log + level, we get a final log level of 20, which is the INFO level. + + >>> print system(buildout), + INFO Develop: '/sample-buildout/recipes' + + Predefined buildout options + --------------------------- + + Buildouts have a number of predefined options that recipes can use + and that users can override in their configuration files. To see + these, we'll run a minimal buildout configuration with a debug logging + level. One of the features of debug logging is that the configuration + database is shown. + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... parts = + ... """) + + >>> print system(buildout+' -vv'), # doctest: +NORMALIZE_WHITESPACE + Installing 'zc.buildout', 'setuptools'. + We have a develop egg: zc.buildout X.X. + We have the best distribution that satisfies 'setuptools'. + Picked: setuptools = V.V + <BLANKLINE> + Configuration data: + [buildout] + accept-buildout-test-releases = false + allow-hosts = * + allow-picked-versions = true + allowed-eggs-from-site-packages = * + bin-directory = /sample-buildout/bin + develop-eggs-directory = /sample-buildout/develop-eggs + directory = /sample-buildout + eggs-directory = /sample-buildout/eggs + exec-sitecustomize = true + executable = python + find-links = + include-site-packages = true + install-from-cache = false + installed = /sample-buildout/.installed.cfg + log-format = + log-level = INFO + newest = true + offline = false + parts = + parts-directory = /sample-buildout/parts + prefer-final = false + python = buildout + relative-paths = false + socket-timeout = + unzip = false + use-dependency-links = true + verbosity = 20 + <BLANKLINE> + + All of these options can be overridden by configuration files or by + command-line assignments. We've discussed most of these options + already, but let's review them and touch on some we haven't discussed: + + allowed-eggs-from-site-packages + Sometimes you need or want to control what eggs from site-packages are + used. The allowed-eggs-from-site-packages option allows you to specify a + whitelist of project names that may be included from site-packages. You + can use globs to specify the value. It defaults to a single value of '*', + indicating that any package may come from site-packages. + + Here's a usage example:: + + [buildout] + ... + + allowed-eggs-from-site-packages = + demo + bigdemo + zope.* + + This option interacts with the ``include-site-packages`` option in the + following ways. + + If ``include-site-packages`` is true, then + ``allowed-eggs-from-site-packages`` filters what eggs from site-packages + may be chosen. Therefore, if ``allowed-eggs-from-site-packages`` is an + empty list, then no eggs from site-packages are chosen, but site-packages + will still be included at the end of path lists. + + If ``include-site-packages`` is false, the value of + ``allowed-eggs-from-site-packages`` is irrelevant. + + See the ``include-site-packages`` description for more information. + + bin-directory + The directory path where scripts are written. This can be a + relative path, which is interpreted relative to the directory + option. + + develop-eggs-directory + The directory path where development egg links are created for software + being created in the local project. This can be a relative path, + which is interpreted relative to the directory option. + + directory + The buildout directory. This is the base for other buildout file + and directory locations, when relative locations are used. + + eggs-directory + The directory path where downloaded eggs are put. It is common to share + this directory across buildouts. Eggs in this directory should + *never* be modified. This can be a relative path, which is + interpreted relative to the directory option. + + exec-sitecustomize + Normally the Python's real sitecustomize module is processed. + If you do not want it to be processed in order to increase the + repeatability of your buildout, set this value to 'false'. This will + be honored irrespective of the setting for include-site-packages. + This option will be honored by some recipes and not others. + z3c.recipe.scripts honors this and zc.recipe.egg does not, for + instance. + + executable + The Python executable used to run the buildout. See the python + option below. + + include-site-packages + You can choose not to have the site-packages of the underlying Python + available to your script or interpreter, in addition to the packages + from your eggs. This can increase repeatability for your buildout. + This option will be better used by some recipes than others. + z3c.recipe.scripts honors this fully and zc.recipe.egg only + partially, for instance. + + installed + The file path where information about the results of the previous + buildout run is written. This can be a relative path, which is + interpreted relative to the directory option. This file provides + an inventory of installed parts with information needed to decide + which if any parts need to be uninstalled. + + log-format + The format used for logging messages. + + log-level + The log level before verbosity adjustment + + parts + A white space separated list of parts to be installed. + + parts-directory + A working directory that parts can used to store data. + + python + The name of a section containing information about the default + Python interpreter. Recipes that need a installation + typically have options to tell them which Python installation to + use. By convention, if a section-specific option isn't used, the + option is looked for in the buildout section. The option must + point to a section with an executable option giving the path to a + Python executable. By default, the buildout section defines the + default Python as the Python used to run the buildout. + + relative-paths + The paths generated by zc.buildout are absolute by default, and this + option is ``false``. However, if you set this value to be ``true``, + bin/buildout will be generated with code that makes the paths relative. + Some recipes, such as zc.recipe.egg and z3c.recipe.scripts, honor this + value as well. + + unzip + By default, zc.buildout doesn't unzip zip-safe eggs ("unzip = false"). + This follows the policy followed by setuptools itself. Experience shows + this policy to to be inconvenient. Zipped eggs make debugging more + difficult and often import more slowly. You can include an unzip option in + the buildout section to change the default unzipping policy ("unzip = + true"). + + use-dependency-links + By default buildout will obey the setuptools dependency_links metadata + when it looks for dependencies. This behavior can be controlled with + the use-dependency-links buildout option:: + + [buildout] + ... + use-dependency-links = false + + The option defaults to true. If you set it to false, then dependency + links are only looked for in the locations specified by find-links. + + verbosity + A log-level adjustment. Typically, this is set via the -q and -v + command-line options. + + + Creating new buildouts and bootstrapping + ---------------------------------------- + + If zc.buildout is installed, you can use it to create a new buildout + with it's own local copies of zc.buildout and setuptools and with + local buildout scripts. + + >>> sample_bootstrapped = tmpdir('sample-bootstrapped') + + >>> print system(buildout + ... +' -c'+os.path.join(sample_bootstrapped, 'setup.cfg') + ... +' init'), + Creating '/sample-bootstrapped/setup.cfg'. + Creating directory '/sample-bootstrapped/bin'. + Creating directory '/sample-bootstrapped/parts'. + Creating directory '/sample-bootstrapped/eggs'. + Creating directory '/sample-bootstrapped/develop-eggs'. + Generated script '/sample-bootstrapped/bin/buildout'. + + Note that a basic setup.cfg was created for us. + + >>> ls(sample_bootstrapped) + d bin + d develop-eggs + d eggs + d parts + - setup.cfg + + >>> ls(sample_bootstrapped, 'bin') + - buildout + + >>> _ = (ls(sample_bootstrapped, 'eggs'), + ... ls(sample_bootstrapped, 'develop-eggs')) + - setuptools-0.6-py2.3.egg + - zc.buildout-1.0-py2.3.egg + + (We list both the eggs and develop-eggs directories because the + buildout or setuptools egg could be installed in the develop-eggs + directory if the original buildout had develop eggs for either + buildout or setuptools.) + + If relative-paths is ``true``, the buildout script uses relative paths. + + >>> write(sample_bootstrapped, 'setup.cfg', + ... ''' + ... [buildout] + ... relative-paths = true + ... parts = + ... ''') + + >>> print system(buildout + ... +' -c'+os.path.join(sample_bootstrapped, 'setup.cfg') + ... +' bootstrap'), + Generated script '/sample-bootstrapped/bin/buildout'. + + >>> buildout_script = join(sample_bootstrapped, 'bin', 'buildout') + >>> import sys + >>> if sys.platform.startswith('win'): + ... buildout_script += '-script.py' + >>> print open(buildout_script).read() # doctest: +ELLIPSIS + #!... -S + <BLANKLINE> + import os + <BLANKLINE> + join = os.path.join + base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) + base = os.path.dirname(base) + <BLANKLINE> + import sys + sys.path[0:0] = [ + join(base, 'parts/buildout'), + ] + <BLANKLINE> + <BLANKLINE> + import os + path = sys.path[0] + if os.environ.get('PYTHONPATH'): + path = os.pathsep.join([path, os.environ['PYTHONPATH']]) + os.environ['BUILDOUT_ORIGINAL_PYTHONPATH'] = os.environ.get('PYTHONPATH', '') + os.environ['PYTHONPATH'] = path + import site # imports custom buildout-generated site.py + <BLANKLINE> + import zc.buildout.buildout + <BLANKLINE> + if __name__ == '__main__': + zc.buildout.buildout.main() + <BLANKLINE> + + + Note that, in the above two examples, the buildout script was installed + but not run. To run the buildout, we'd have to run the installed + buildout script. + + If we have an existing buildout that already has a buildout.cfg, we'll + normally use the bootstrap command instead of init. It will complain + if there isn't a configuration file: + + >>> sample_bootstrapped2 = tmpdir('sample-bootstrapped2') + + >>> print system(buildout + ... +' -c'+os.path.join(sample_bootstrapped2, 'setup.cfg') + ... +' bootstrap'), + While: + Initializing. + Error: Couldn't open /sample-bootstrapped2/setup.cfg + + >>> write(sample_bootstrapped2, 'setup.cfg', + ... """ + ... [buildout] + ... parts = + ... """) + + >>> print system(buildout + ... +' -c'+os.path.join(sample_bootstrapped2, 'setup.cfg') + ... +' bootstrap'), + Creating directory '/sample-bootstrapped2/bin'. + Creating directory '/sample-bootstrapped2/parts'. + Creating directory '/sample-bootstrapped2/eggs'. + Creating directory '/sample-bootstrapped2/develop-eggs'. + Generated script '/sample-bootstrapped2/bin/buildout'. + + + Newest and Offline Modes + ------------------------ + + By default buildout and recipes will try to find the newest versions + of distributions needed to satisfy requirements. This can be very + time consuming, especially when incrementally working on setting up a + buildout or working on a recipe. The buildout newest option can be + used to to suppress this. If the newest option is set to false, then + new distributions won't be sought if an installed distribution meets + requirements. The newest option can be set to false using the -N + command-line option. + + The offline option goes a bit further. If the buildout offline option + is given a value of "true", the buildout and recipes that are aware of + the option will avoid doing network access. This is handy when + running the buildout when not connected to the internet. It also + makes buildouts run much faster. This option is typically set using + the buildout -o option. + + Preferring Final Releases + ------------------------- + + Currently, when searching for new releases of your project's + dependencies, the newest available release is used. This isn't usually + ideal, as you may get a development release or alpha releases not ready + to be widely used. You can request that final releases be preferred + using the ``prefer-final`` option in the buildout section:: + + [buildout] + ... + prefer-final = true + + When the ``prefer-final`` option is set to true, then when searching for + new releases, final releases are preferred. If there are final + releases that satisfy distribution requirements, then those releases + are used even if newer non-final releases are available. + + In buildout version 2, all final releases will be preferred by + default--that is ``prefer-final`` will also default to 'true'. You will + then need to use a 'false' value for ``prefer-final`` to get the newest + releases. + + A separate option controls the behavior of the build system itself. + When buildout looks for recipes, extensions, and for updates to itself, + it does prefer final releases by default, as of the 1.5.0 release. The + ``accept-buildout-test-releases`` option will let you override this behavior. + However, it is typically changed by the --accept-buildout-test-releases + option to the bootstrap script, since bootstrapping is the first step to + selecting a buildout. + + Finding distributions + --------------------- + + By default, buildout searches the Python Package Index when looking + for distributions. You can, instead, specify your own index to search + using the `index` option:: + + [buildout] + ... + index = http://index.example.com/ + + This index, or the default of http://pypi.python.org/simple/ if no + index is specified, will always be searched for distributions unless + running buildout with options that prevent searching for + distributions. The latest version of the distribution that meets the + requirements of the buildout will always be used. + + You can also specify more locations to search for distributions using + the `find-links` option. All locations specified will be searched for + distributions along with the package index as described before. + + Locations can be urls:: + + [buildout] + ... + find-links = http://download.zope.org/distribution/ + + They can also be directories on disk:: + + [buildout] + ... + find-links = /some/path + + Finally, they can also be direct paths to distributions:: + + [buildout] + ... + find-links = /some/path/someegg-1.0.0-py2.3.egg + + Any number of locations can be specified in the `find-links` option:: + + [buildout] + ... + find-links = + http://download.zope.org/distribution/ + /some/otherpath + /some/path/someegg-1.0.0-py2.3.egg + + Dependency links + ---------------- + + By default buildout will obey the setuptools dependency_links metadata + when it looks for dependencies. This behavior can be controlled with + the use-dependency-links buildout option:: + + [buildout] + ... + use-dependency-links = false + + The option defaults to true. If you set it to false, then dependency + links are only looked for in the locations specified by find-links. + + Controlling the installation database + ------------------------------------- + + The buildout installed option is used to specify the file used to save + information on installed parts. This option is initialized to + ".installed.cfg", but it can be overridden in the configuration file + or on the command line: + + >>> write('buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = debug + ... + ... [debug] + ... recipe = recipes:debug + ... """) + + >>> print system(buildout+' buildout:installed=inst.cfg'), + Develop: '/sample-buildout/recipes' + Installing debug. + recipe recipes:debug + + >>> ls(sample_buildout) + - b1.cfg + - b2.cfg + - base.cfg + d bin + - buildout.cfg + d demo + d develop-eggs + d eggs + - inst.cfg + d parts + d recipes + + The installation database can be disabled by supplying an empty + buildout installed option: + + >>> os.remove('inst.cfg') + >>> print system(buildout+' buildout:installed='), + Develop: '/sample-buildout/recipes' + Installing debug. + recipe recipes:debug + + >>> ls(sample_buildout) + - b1.cfg + - b2.cfg + - base.cfg + d bin + - buildout.cfg + d demo + d develop-eggs + d eggs + d parts + d recipes + + + Note that there will be no installation database if there are no parts: + + >>> write('buildout.cfg', + ... """ + ... [buildout] + ... parts = + ... """) + + >>> print system(buildout+' buildout:installed=inst.cfg'), + + >>> ls(sample_buildout) + - b1.cfg + - b2.cfg + - base.cfg + d bin + - buildout.cfg + d demo + d develop-eggs + d eggs + d parts + d recipes + + Extensions + ---------- + + A feature allows code to be loaded and run after + configuration files have been read but before the buildout has begun + any processing. The intent is to allow special plugins such as + urllib2 request handlers to be loaded. + + To load an extension, we use the extensions option and list one or + more distribution requirements, on separate lines. The distributions + named will be loaded and any ``zc.buildout.extension`` entry points found + will be called with the buildout as an argument. When buildout + finishes processing, any ``zc.buildout.unloadextension`` entry points + found will be called with the buildout as an argument. + + Let's create a sample extension in our sample buildout created in the + previous section: + + >>> mkdir(sample_bootstrapped, 'demo') + + >>> write(sample_bootstrapped, 'demo', 'demo.py', + ... """ + ... def ext(buildout): + ... print 'ext', list(buildout) + ... def unload(buildout): + ... print 'unload', list(buildout) + ... """) + + >>> write(sample_bootstrapped, 'demo', 'setup.py', + ... """ + ... from setuptools import setup + ... + ... setup( + ... name = "demo", + ... entry_points = { + ... 'zc.buildout.extension': ['ext = demo:ext'], + ... 'zc.buildout.unloadextension': ['ext = demo:unload'], + ... }, + ... ) + ... """) + + Our extension just prints out the word 'demo', and lists the sections + found in the buildout passed to it. + + We'll update our buildout.cfg to list the demo directory as a develop + egg to be built: + + >>> write(sample_bootstrapped, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = demo + ... parts = + ... """) + + >>> os.chdir(sample_bootstrapped) + >>> print system(os.path.join(sample_bootstrapped, 'bin', 'buildout')), + Develop: '/sample-bootstrapped/demo' + + Now we can add the extensions option. We were a bit tricky and ran + the buildout once with the demo develop egg defined but without the + extension option. This is because extensions are loaded before the + buildout creates develop eggs. We needed to use a separate buildout + run to create the develop egg. Normally, when eggs are loaded from + the network, we wouldn't need to do anything special. + + >>> write(sample_bootstrapped, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = demo + ... extensions = demo + ... parts = + ... """) + + We see that our extension is loaded and executed: + + >>> print system(os.path.join(sample_bootstrapped, 'bin', 'buildout')), + ext ['buildout'] + Develop: '/sample-bootstrapped/demo' + unload ['buildout'] + + Allow hosts + ----------- + + On some environments the links visited by `zc.buildout` can be forbidden + by paranoiac firewalls. These URL might be on the chain of links + visited by `zc.buildout` wheter they are defined in the `find-links` option, + wheter they are defined by various eggs in their `url`, `download_url`, + `dependency_links` metadata. + + It is even harder to track that package_index works like a spider and + might visit links and go to other location. + + The `allow-hosts` option provides a way to prevent this, and + works exactly like the one provided in `easy_install`. + + You can provide a list of allowed host, together with wildcards:: + + [buildout] + ... + + allow-hosts = + *.python.org + example.com + + All urls that does not match these hosts will not be visited. + + .. [#future_recipe_methods] In the future, additional methods may be + added. Older recipes with fewer methods will still be + supported. + + .. [#packaging_info] If we wanted to create a distribution from this + package, we would need specify much more information. See the + `setuptools documentation + <http://peak.telecommunity.com/DevCenter/setuptools>`_. + + Always unzipping eggs + ===================== + + By default, zc.buildout doesn't unzip zip-safe eggs. + + >>> write('buildout.cfg', + ... ''' + ... [buildout] + ... parts = eggs + ... find-links = %(link_server)s + ... + ... [eggs] + ... recipe = zc.recipe.egg + ... eggs = demo + ... ''' % globals()) + + >>> _ = system(buildout) + >>> ls('eggs') + - demo-0.4c1-py2.4.egg + - demoneeded-1.2c1-py2.4.egg + d setuptools-0.6c8-py2.4.egg + - zc.buildout.egg-link + + This follows the policy followed by setuptools itself. Experience shows + this policy to to be inconvenient. Zipped eggs make debugging more + difficult and often import more slowly. + + You can include an unzip option in the buildout section to change the + default unzipping policy. + + >>> write('buildout.cfg', + ... ''' + ... [buildout] + ... parts = eggs + ... find-links = %(link_server)s + ... unzip = true + ... + ... [eggs] + ... recipe = zc.recipe.egg + ... eggs = demo + ... ''' % globals()) + + + >>> import os + >>> for name in os.listdir('eggs'): + ... if name.startswith('demo'): + ... remove('eggs', name) + + >>> _ = system(buildout) + >>> ls('eggs') + d demo-0.4c1-py2.4.egg + d demoneeded-1.2c1-py2.4.egg + d setuptools-0.6c8-py2.4.egg + - zc.buildout.egg-link + + Repeatable buildouts: controlling eggs used + =========================================== + + One of the goals of zc.buildout is to provide enough control to make + buildouts repeatable. It should be possible to check the buildout + configuration files for a project into a version control system and + later use the checked in files to get the same buildout, subject to + changes in the environment outside the buildout. + + An advantage of using Python eggs is that depenencies of eggs used are + automatically determined and used. The automatic inclusion of + depenent distributions is at odds with the goal of repeatable + buildouts. + + To support repeatable buildouts, a versions section can be created + with options for each distribution name whos version is to be fixed. + The section can then be specified via the buildout versions option. + + To see how this works, we'll create two versions of a recipe egg: + + >>> mkdir('recipe') + >>> write('recipe', 'recipe.py', + ... ''' + ... class Recipe: + ... def __init__(*a): pass + ... def install(self): + ... print 'recipe v1' + ... return () + ... update = install + ... ''') + + >>> write('recipe', 'setup.py', + ... ''' + ... from setuptools import setup + ... setup(name='spam', version='1', py_modules=['recipe'], + ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, + ... ) + ... ''') + + >>> write('recipe', 'README', '') + + >>> print system(buildout+' setup recipe bdist_egg'), # doctest: +ELLIPSIS + Running setup script 'recipe/setup.py'. + ... + + >>> rmdir('recipe', 'build') + + >>> write('recipe', 'recipe.py', + ... ''' + ... class Recipe: + ... def __init__(*a): pass + ... def install(self): + ... print 'recipe v2' + ... return () + ... update = install + ... ''') + + >>> write('recipe', 'setup.py', + ... ''' + ... from setuptools import setup + ... setup(name='spam', version='2', py_modules=['recipe'], + ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, + ... ) + ... ''') + + + >>> print system(buildout+' setup recipe bdist_egg'), # doctest: +ELLIPSIS + Running setup script 'recipe/setup.py'. + ... + + and we'll configure a buildout to use it: + + >>> write('buildout.cfg', + ... ''' + ... [buildout] + ... parts = foo + ... find-links = %s + ... + ... [foo] + ... recipe = spam + ... ''' % join('recipe', 'dist')) + + If we run the buildout, it will use version 2: + + >>> print system(buildout), + Getting distribution for 'spam'. + Got spam 2. + Installing foo. + recipe v2 + + We can specify a versions section that lists our recipe and name it in + the buildout section: + + >>> write('buildout.cfg', + ... ''' + ... [buildout] + ... parts = foo + ... find-links = %s + ... versions = release-1 + ... + ... [release-1] + ... spam = 1 + ... eggs = 2.2 + ... + ... [foo] + ... recipe = spam + ... ''' % join('recipe', 'dist')) + + Here we created a release-1 section listing the version 1 for the spam + distribution. We told the buildout to use it by specifying release-1 + as in the versions option. + + Now, if we run the buildout, we'll use version 1 of the spam recipe: + + >>> print system(buildout), + Getting distribution for 'spam==1'. + Got spam 1. + Uninstalling foo. + Installing foo. + recipe v1 + + Running the buildout in verbose mode will help us get information + about versions used. If we run the buildout in verbose mode without + specifying a versions section: + + >>> print system(buildout+' buildout:versions= -v'), # doctest: +ELLIPSIS + Installing 'zc.buildout', 'setuptools'. + We have a develop egg: zc.buildout 1.0.0. + We have the best distribution that satisfies 'setuptools'. + Picked: setuptools = 0.6 + Installing 'spam'. + We have the best distribution that satisfies 'spam'. + Picked: spam = 2. + Uninstalling foo. + Installing foo. + recipe v2 + + We'll get output that includes lines that tell us what versions + buildout chose a for us, like:: + + zc.buildout.easy_install.picked: spam = 2 + + This allows us to discover versions that are picked dynamically, so + that we can fix them in a versions section. + + If we run the buildout with the versions section: + + >>> print system(buildout+' -v'), # doctest: +ELLIPSIS + Installing 'zc.buildout', 'setuptools'. + We have a develop egg: zc.buildout 1.0.0. + We have the best distribution that satisfies 'setuptools'. + Picked: setuptools = 0.6 + Installing 'spam'. + We have the distribution that satisfies 'spam==1'. + Uninstalling foo. + Installing foo. + recipe v1 + + We won't get output for the spam distribution, which we didn't pick, + but we will get output for setuptools, which we didn't specify + versions for. + + You can request buildout to generate an error if it picks any + versions: + + >>> write('buildout.cfg', + ... ''' + ... [buildout] + ... parts = foo + ... find-links = %s + ... versions = release-1 + ... allow-picked-versions = false + ... + ... [release-1] + ... spam = 1 + ... eggs = 2.2 + ... + ... [foo] + ... recipe = spam + ... ''' % join('recipe', 'dist')) + + Using the download utility + ========================== + + The ``zc.buildout.download`` module provides a download utility that handles + the details of downloading files needed for a buildout run from the internet. + It downloads files to the local file system, using the download cache if + desired and optionally checking the downloaded files' MD5 checksum. + + We setup an HTTP server that provides a file we want to download: + + >>> server_data = tmpdir('sample_files') + >>> write(server_data, 'foo.txt', 'This is a foo text.') + >>> server_url = start_server(server_data) + + We also use a fresh directory for temporary files in order to make sure that + all temporary files have been cleaned up in the end: + + >>> import tempfile + >>> old_tempdir = tempfile.tempdir + >>> tempfile.tempdir = tmpdir('tmp') + + + Downloading without using the cache + ----------------------------------- + + If no download cache should be used, the download utility is instantiated + without any arguments: + + >>> from zc.buildout.download import Download + >>> download = Download() + >>> print download.cache_dir + None + + Downloading a file is achieved by calling the utility with the URL as an + argument. A tuple is returned that consists of the path to the downloaded copy + of the file and a boolean value indicating whether this is a temporary file + meant to be cleaned up during the same buildout run: + + >>> path, is_temp = download(server_url+'foo.txt') + >>> print path + /.../buildout-... + >>> cat(path) + This is a foo text. + + As we aren't using the download cache and haven't specified a target path + either, the download has ended up in a temporary file: + + >>> is_temp + True + + >>> import tempfile + >>> path.startswith(tempfile.gettempdir()) + True + + We are responsible for cleaning up temporary files behind us: + + >>> remove(path) + + When trying to access a file that doesn't exist, we'll get an exception: + + >>> try: download(server_url+'not-there') # doctest: +ELLIPSIS + ... except: print 'download error' + ... else: print 'woops' + download error + + Downloading a local file doesn't produce a temporary file but simply returns + the local file itself: + + >>> download(join(server_data, 'foo.txt')) + ('/sample_files/foo.txt', False) + + We can also have the downloaded file's MD5 sum checked: + + >>> try: from hashlib import md5 + ... except ImportError: from md5 import new as md5 + + >>> path, is_temp = download(server_url+'foo.txt', + ... md5('This is a foo text.').hexdigest()) + >>> is_temp + True + >>> remove(path) + + >>> download(server_url+'foo.txt', + ... md5('The wrong text.').hexdigest()) + Traceback (most recent call last): + ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt' + + The error message in the event of an MD5 checksum mismatch for a local file + reads somewhat differently: + + >>> download(join(server_data, 'foo.txt'), + ... md5('This is a foo text.').hexdigest()) + ('/sample_files/foo.txt', False) + + >>> download(join(server_data, 'foo.txt'), + ... md5('The wrong text.').hexdigest()) + Traceback (most recent call last): + ChecksumError: MD5 checksum mismatch for local resource at '/sample_files/foo.txt'. + + Finally, we can download the file to a specified place in the file system: + + >>> target_dir = tmpdir('download-target') + >>> path, is_temp = download(server_url+'foo.txt', + ... path=join(target_dir, 'downloaded.txt')) + >>> print path + /download-target/downloaded.txt + >>> cat(path) + This is a foo text. + >>> is_temp + False + + Trying to download a file in offline mode will result in an error: + + >>> download = Download(cache=None, offline=True) + >>> download(server_url+'foo.txt') + Traceback (most recent call last): + UserError: Couldn't download 'http://localhost/foo.txt' in offline mode. + + As an exception to this rule, file system paths and URLs in the ``file`` + scheme will still work: + + >>> cat(download(join(server_data, 'foo.txt'))[0]) + This is a foo text. + >>> cat(download('file:' + join(server_data, 'foo.txt'))[0]) + This is a foo text. + + >>> remove(path) + + + Downloading using the download cache + ------------------------------------ + + In order to make use of the download cache, we need to configure the download + utility differently. To do this, we pass a directory path as the ``cache`` + attribute upon instantiation: + + >>> cache = tmpdir('download-cache') + >>> download = Download(cache=cache) + >>> print download.cache_dir + /download-cache/ + + Simple usage + ~~~~~~~~~~~~ + + When using the cache, a file will be stored in the cache directory when it is + first downloaded. The file system path returned by the download utility points + to the cached copy: + + >>> ls(cache) + >>> path, is_temp = download(server_url+'foo.txt') + >>> print path + /download-cache/foo.txt + >>> cat(path) + This is a foo text. + >>> is_temp + False + + Whenever the file is downloaded again, the cached copy is used. Let's change + the file on the server to see this: + + >>> write(server_data, 'foo.txt', 'The wrong text.') + >>> path, is_temp = download(server_url+'foo.txt') + >>> print path + /download-cache/foo.txt + >>> cat(path) + This is a foo text. + + If we specify an MD5 checksum for a file that is already in the cache, the + cached copy's checksum will be verified: + + >>> download(server_url+'foo.txt', md5('The wrong text.').hexdigest()) + Traceback (most recent call last): + ChecksumError: MD5 checksum mismatch for cached download + from 'http://localhost/foo.txt' at '/download-cache/foo.txt' + + Trying to access another file at a different URL which has the same base name + will result in the cached copy being used: + + >>> mkdir(server_data, 'other') + >>> write(server_data, 'other', 'foo.txt', 'The wrong text.') + >>> path, is_temp = download(server_url+'other/foo.txt') + >>> print path + /download-cache/foo.txt + >>> cat(path) + This is a foo text. + + Given a target path for the download, the utility will provide a copy of the + file at that location both when first downloading the file and when using a + cached copy: + + >>> remove(cache, 'foo.txt') + >>> ls(cache) + >>> write(server_data, 'foo.txt', 'This is a foo text.') + + >>> path, is_temp = download(server_url+'foo.txt', + ... path=join(target_dir, 'downloaded.txt')) + >>> print path + /download-target/downloaded.txt + >>> cat(path) + This is a foo text. + >>> is_temp + False + >>> ls(cache) + - foo.txt + + >>> remove(path) + >>> write(server_data, 'foo.txt', 'The wrong text.') + + >>> path, is_temp = download(server_url+'foo.txt', + ... path=join(target_dir, 'downloaded.txt')) + >>> print path + /download-target/downloaded.txt + >>> cat(path) + This is a foo text. + >>> is_temp + False + + In offline mode, downloads from any URL will be successful if the file is + found in the cache: + + >>> download = Download(cache=cache, offline=True) + >>> cat(download(server_url+'foo.txt')[0]) + This is a foo text. + + Local resources will be cached just like any others since download caches are + sometimes used to create source distributions: + + >>> remove(cache, 'foo.txt') + >>> ls(cache) + + >>> write(server_data, 'foo.txt', 'This is a foo text.') + >>> download = Download(cache=cache) + + >>> cat(download('file:' + join(server_data, 'foo.txt'), path=path)[0]) + This is a foo text. + >>> ls(cache) + - foo.txt + + >>> remove(cache, 'foo.txt') + + >>> cat(download(join(server_data, 'foo.txt'), path=path)[0]) + This is a foo text. + >>> ls(cache) + - foo.txt + + >>> remove(cache, 'foo.txt') + + However, resources with checksum mismatches will not be copied to the cache: + + >>> download(server_url+'foo.txt', md5('The wrong text.').hexdigest()) + Traceback (most recent call last): + ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt' + >>> ls(cache) + + >>> remove(path) + + Finally, let's see what happens if the download cache to be used doesn't exist + as a directory in the file system yet: + + >>> Download(cache=join(cache, 'non-existent'))(server_url+'foo.txt') + Traceback (most recent call last): + UserError: The directory: + '/download-cache/non-existent' + to be used as a download cache doesn't exist. + + Using namespace sub-directories of the download cache + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + It is common to store cached copies of downloaded files within sub-directories + of the download cache to keep some degree of order. For example, zc.buildout + stores downloaded distributions in a sub-directory named "dist". Those + sub-directories are also known as namespaces. So far, we haven't specified any + namespaces to use, so the download utility stored files directly inside the + download cache. Let's use a namespace "test" instead: + + >>> download = Download(cache=cache, namespace='test') + >>> print download.cache_dir + /download-cache/test + + The namespace sub-directory hasn't been created yet: + + >>> ls(cache) + + Downloading a file now creates the namespace sub-directory and places a copy + of the file inside it: + + >>> path, is_temp = download(server_url+'foo.txt') + >>> print path + /download-cache/test/foo.txt + >>> ls(cache) + d test + >>> ls(cache, 'test') + - foo.txt + >>> cat(path) + This is a foo text. + >>> is_temp + False + + The next time we want to download that file, the copy from inside the cache + namespace is used. To see this clearly, we put a file with the same name but + different content both on the server and in the cache's root directory: + + >>> write(server_data, 'foo.txt', 'The wrong text.') + >>> write(cache, 'foo.txt', 'The wrong text.') + + >>> path, is_temp = download(server_url+'foo.txt') + >>> print path + /download-cache/test/foo.txt + >>> cat(path) + This is a foo text. + + >>> rmdir(cache, 'test') + >>> remove(cache, 'foo.txt') + >>> write(server_data, 'foo.txt', 'This is a foo text.') + + Using a hash of the URL as the filename in the cache + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + So far, the base name of the downloaded file read from the URL has been used + for the name of the cached copy of the file. This may not be desirable in some + cases, for example when downloading files from different locations that have + the same base name due to some naming convention, or if the file content + depends on URL parameters. In such cases, an MD5 hash of the complete URL may + be used as the filename in the cache: + + >>> download = Download(cache=cache, hash_name=True) + >>> path, is_temp = download(server_url+'foo.txt') + >>> print path + /download-cache/09f5793fcdc1716727f72d49519c688d + >>> cat(path) + This is a foo text. + >>> ls(cache) + - 09f5793fcdc1716727f72d49519c688d + + The path was printed just to illustrate matters; we cannot know the real + checksum since we don't know which port the server happens to listen at when + the test is run, so we don't actually know the full URL of the file. Let's + check that the checksum actually belongs to the particular URL used: + + >>> path.lower() == join(cache, md5(server_url+'foo.txt').hexdigest()).lower() + True + + The cached copy is used when downloading the file again: + + >>> write(server_data, 'foo.txt', 'The wrong text.') + >>> (path, is_temp) == download(server_url+'foo.txt') + True + >>> cat(path) + This is a foo text. + >>> ls(cache) + - 09f5793fcdc1716727f72d49519c688d + + If we change the URL, even in such a way that it keeps the base name of the + file the same, the file will be downloaded again this time and put in the + cache under a different name: + + >>> path2, is_temp = download(server_url+'other/foo.txt') + >>> print path2 + /download-cache/537b6d73267f8f4447586989af8c470e + >>> path == path2 + False + >>> path2.lower() == join(cache, md5(server_url+'other/foo.txt').hexdigest()).lower() + True + >>> cat(path) + This is a foo text. + >>> cat(path2) + The wrong text. + >>> ls(cache) + - 09f5793fcdc1716727f72d49519c688d + - 537b6d73267f8f4447586989af8c470e + + >>> remove(path) + >>> remove(path2) + >>> write(server_data, 'foo.txt', 'This is a foo text.') + + + Using the cache purely as a fall-back + ------------------------------------- + + Sometimes it is desirable to try downloading a file from the net if at all + possible, and use the cache purely as a fall-back option when a server is + down or if we are in offline mode. This mode is only in effect if a download + cache is configured in the first place: + + >>> download = Download(cache=cache, fallback=True) + >>> print download.cache_dir + /download-cache/ + + A downloaded file will be cached: + + >>> ls(cache) + >>> path, is_temp = download(server_url+'foo.txt') + >>> ls(cache) + - foo.txt + >>> cat(cache, 'foo.txt') + This is a foo text. + >>> is_temp + False + + If the file cannot be served, the cached copy will be used: + + >>> remove(server_data, 'foo.txt') + >>> try: Download()(server_url+'foo.txt') # doctest: +ELLIPSIS + ... except: print 'download error' + ... else: print 'woops' + download error + >>> path, is_temp = download(server_url+'foo.txt') + >>> cat(path) + This is a foo text. + >>> is_temp + False + + Similarly, if the file is served but we're in offline mode, we'll fall back to + using the cache: + + >>> write(server_data, 'foo.txt', 'The wrong text.') + >>> get(server_url+'foo.txt') + 'The wrong text.' + + >>> offline_download = Download(cache=cache, offline=True, fallback=True) + >>> path, is_temp = offline_download(server_url+'foo.txt') + >>> print path + /download-cache/foo.txt + >>> cat(path) + This is a foo text. + >>> is_temp + False + + However, when downloading the file normally with the cache being used in + fall-back mode, the file will be downloaded from the net and the cached copy + will be replaced with the new content: + + >>> cat(download(server_url+'foo.txt')[0]) + The wrong text. + >>> cat(cache, 'foo.txt') + The wrong text. + + When trying to download a resource whose checksum does not match, the cached + copy will neither be used nor overwritten: + + >>> write(server_data, 'foo.txt', 'This is a foo text.') + >>> download(server_url+'foo.txt', md5('The wrong text.').hexdigest()) + Traceback (most recent call last): + ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt' + >>> cat(cache, 'foo.txt') + The wrong text. + + + Configuring the download utility from buildout options + ------------------------------------------------------ + + The configuration options explained so far derive from the build logic + implemented by the calling code. Other options configure the download utility + for use in a particular project or buildout run; they are read from the + ``buildout`` configuration section. The latter can be passed directly as the + first argument to the download utility's constructor. + + The location of the download cache is specified by the ``download-cache`` + option: + + >>> download = Download({'download-cache': cache}, namespace='cmmi') + >>> print download.cache_dir + /download-cache/cmmi + + If the ``download-cache`` option specifies a relative path, it is understood + relative to the current working directory, or to the buildout directory if + that is given: + + >>> download = Download({'download-cache': 'relative-cache'}) + >>> print download.cache_dir + /sample-buildout/relative-cache/ + + >>> download = Download({'directory': join(sample_buildout, 'root'), + ... 'download-cache': 'relative-cache'}) + >>> print download.cache_dir + /sample-buildout/root/relative-cache/ + + Keyword parameters take precedence over the corresponding options: + + >>> download = Download({'download-cache': cache}, cache=None) + >>> print download.cache_dir + None + + Whether to assume offline mode can be inferred from either the ``offline`` or + the ``install-from-cache`` option. As usual with zc.buildout, these options + must assume one of the values 'true' and 'false': + + >>> download = Download({'offline': 'true'}) + >>> download.offline + True + + >>> download = Download({'offline': 'false'}) + >>> download.offline + False + + >>> download = Download({'install-from-cache': 'true'}) + >>> download.offline + True + + >>> download = Download({'install-from-cache': 'false'}) + >>> download.offline + False + + These two options are combined using logical 'or': + + >>> download = Download({'offline': 'true', 'install-from-cache': 'false'}) + >>> download.offline + True + + >>> download = Download({'offline': 'false', 'install-from-cache': 'true'}) + >>> download.offline + True + + The ``offline`` keyword parameter takes precedence over both the ``offline`` + and ``install-from-cache`` options: + + >>> download = Download({'offline': 'true'}, offline=False) + >>> download.offline + False + + >>> download = Download({'install-from-cache': 'false'}, offline=True) + >>> download.offline + True + + + Regressions + ----------- + + MD5 checksum calculation needs to be reliable on all supported systems, which + requires text files to be treated as binary to avoid implicit line-ending + conversions: + + >>> text = 'First line of text.\r\nSecond line.\r\n' + >>> f = open(join(server_data, 'foo.txt'), 'wb') + >>> f.write(text) + >>> f.close() + >>> path, is_temp = Download()(server_url+'foo.txt', md5(text).hexdigest()) + >>> remove(path) + + + Clean up + -------- + + We should have cleaned up all temporary files created by downloading things: + + >>> ls(tempfile.tempdir) + + Reset the global temporary directory: + + >>> tempfile.tempdir = old_tempdir + + Using a download cache + ====================== + + Normally, when distributions are installed, if any processing is + needed, they are downloaded from the internet to a temporary directory + and then installed from there. A download cache can be used to avoid + the download step. This can be useful to reduce network access and to + create source distributions of an entire buildout. + + The buildout download-cache option can be used to specify a directory + to be used as a download cache. + + In this example, we'll create a directory to hold the cache: + + >>> cache = tmpdir('cache') + + And set up a buildout that downloads some eggs: + + >>> write('buildout.cfg', + ... ''' + ... [buildout] + ... parts = eggs + ... download-cache = %(cache)s + ... find-links = %(link_server)s + ... + ... [eggs] + ... recipe = zc.recipe.egg + ... eggs = demo ==0.2 + ... ''' % globals()) + + We specified a link server that has some distributions available for + download: + + >>> print get(link_server), + <html><body> + <a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br> + <a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br> + <a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br> + <a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br> + <a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br> + <a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br> + <a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br> + <a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br> + <a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br> + <a href="index/">index/</a><br> + <a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br> + </body></html> + + + We'll enable logging on the link server so we can see what's going on: + + >>> get(link_server+'enable_server_logging') + GET 200 /enable_server_logging + '' + + We also specified a download cache. + + If we run the buildout, we'll see the eggs installed from the link + server as usual: + + >>> print system(buildout), + GET 200 / + GET 200 /demo-0.2-py2.4.egg + GET 200 /demoneeded-1.2c1.zip + Installing eggs. + Getting distribution for 'demo==0.2'. + Got demo 0.2. + Getting distribution for 'demoneeded'. + Got demoneeded 1.2c1. + Generated script '/sample-buildout/bin/demo'. + + We'll also get the download cache populated. The buildout doesn't put + files in the cache directly. It creates an intermediate directory, + dist: + + + >>> ls(cache) + d dist + + >>> ls(cache, 'dist') + - demo-0.2-py2.4.egg + - demoneeded-1.2c1.zip + + If we remove the installed eggs from eggs directory and re-run the buildout: + + >>> import os + >>> for f in os.listdir('eggs'): + ... if f.startswith('demo'): + ... remove('eggs', f) + + >>> print system(buildout), + GET 200 / + Updating eggs. + Getting distribution for 'demo==0.2'. + Got demo 0.2. + Getting distribution for 'demoneeded'. + Got demoneeded 1.2c1. + + We see that the distributions aren't downloaded, because they're + downloaded from the cache. + + Installing solely from a download cache + --------------------------------------- + + A download cache can be used as the basis of application source + releases. In an application source release, we want to distribute an + application that can be built without making any network accesses. In + this case, we distribute a buildout with download cache and tell the + buildout to install from the download cache only, without making + network accesses. The buildout install-from-cache option can be used + to signal that packages should be installed only from the download + cache. + + Let's remove our installed eggs and run the buildout with the + install-from-cache option set to true: + + >>> for f in os.listdir('eggs'): + ... if f.startswith('demo'): + ... remove('eggs', f) + + >>> write('buildout.cfg', + ... ''' + ... [buildout] + ... parts = eggs + ... download-cache = %(cache)s + ... install-from-cache = true + ... find-links = %(link_server)s + ... + ... [eggs] + ... recipe = zc.recipe.egg + ... eggs = demo + ... ''' % globals()) + + >>> print system(buildout), + Uninstalling eggs. + Installing eggs. + Getting distribution for 'demo'. + Got demo 0.2. + Getting distribution for 'demoneeded'. + Got demoneeded 1.2c1. + Generated script '/sample-buildout/bin/demo'. + + Caching extended configuration + ============================== + + As mentioned in the general buildout documentation, configuration files can + extend each other, including the ability to download configuration being + extended from a URL. If desired, zc.buildout caches downloaded configuration + in order to be able to use it when run offline. + + As we're going to talk about downloading things, let's start an HTTP server. + Also, all of the following will take place inside the sample buildout. + + >>> server_data = tmpdir('server_data') + >>> server_url = start_server(server_data) + >>> cd(sample_buildout) + + We also use a fresh directory for temporary files in order to make sure that + all temporary files have been cleaned up in the end: + + >>> import tempfile + >>> old_tempdir = tempfile.tempdir + >>> tempfile.tempdir = tmpdir('tmp') + + + Basic use of the extends cache + ------------------------------ + + We put some base configuration on a server and reference it from a sample + buildout: + + >>> write(server_data, 'base.cfg', """\ + ... [buildout] + ... parts = + ... foo = bar + ... """) + + >>> write('buildout.cfg', """\ + ... [buildout] + ... extends = %sbase.cfg + ... """ % server_url) + + When trying to run this buildout offline, we'll find that we cannot read all + of the required configuration: + + >>> print system(buildout + ' -o') + While: + Initializing. + Error: Couldn't download 'http://localhost/base.cfg' in offline mode. + + Trying the same online, we can: + + >>> print system(buildout) + Unused options for buildout: 'foo'. + + As long as we haven't said anything about caching downloaded configuration, + nothing gets cached. Offline mode will still cause the buildout to fail: + + >>> print system(buildout + ' -o') + While: + Initializing. + Error: Couldn't download 'http://localhost/base.cfg' in offline mode. + + Let's now specify a cache for base configuration files. This cache is + different from the download cache used by recipes for caching distributions + and other files; one might, however, use a namespace subdirectory of the + download cache for it. The configuration cache we specify will be created when + running buildout and the base.cfg file will be put in it (with the file name + being a hash of the complete URL): + + >>> mkdir('cache') + >>> write('buildout.cfg', """\ + ... [buildout] + ... extends = %sbase.cfg + ... extends-cache = cache + ... """ % server_url) + + >>> print system(buildout) + Unused options for buildout: 'foo'. + + >>> cache = join(sample_buildout, 'cache') + >>> ls(cache) + - 5aedc98d7e769290a29d654a591a3a45 + + >>> import os + >>> cat(cache, os.listdir(cache)[0]) + [buildout] + parts = + foo = bar + + We can now run buildout offline as it will read base.cfg from the cache: + + >>> print system(buildout + ' -o') + Unused options for buildout: 'foo'. + + The cache is being used purely as a fall-back in case we are offline or don't + have access to a configuration file to be downloaded. As long as we are + online, buildout attempts to download a fresh copy of each file even if a + cached copy of the file exists. To see this, we put different configuration in + the same place on the server and run buildout in offline mode so it takes + base.cfg from the cache: + + >>> write(server_data, 'base.cfg', """\ + ... [buildout] + ... parts = + ... bar = baz + ... """) + + >>> print system(buildout + ' -o') + Unused options for buildout: 'foo'. + + In online mode, buildout will download and use the modified version: + + >>> print system(buildout) + Unused options for buildout: 'bar'. + + Trying offline mode again, the new version will be used as it has been put in + the cache now: + + >>> print system(buildout + ' -o') + Unused options for buildout: 'bar'. + + Clean up: + + >>> rmdir(cache) + + + Specifying extends cache and offline mode + ----------------------------------------- + + Normally, the values of buildout options such as the location of a download + cache or whether to use offline mode are determined by first reading the + user's default configuration, updating it with the project's configuration and + finally applying command-line options. User and project configuration are + assembled by reading a file such as ``~/.buildout/default.cfg``, + ``buildout.cfg`` or a URL given on the command line, recursively (depth-first) + downloading any base configuration specified by the ``buildout:extends`` + option read from each of those config files, and finally evaluating each + config file to provide default values for options not yet read. + + This works fine for all options that do not influence how configuration is + downloaded in the first place. The ``extends-cache`` and ``offline`` options, + however, are treated differently from the procedure described in order to make + it simple and obvious to see where a particular configuration file came from + under any particular circumstances. + + - Offline and extends-cache settings are read from the two root config files + exclusively. Otherwise one could construct configuration files that, when + read, imply that they should have been read from a different source than + they have. Also, specifying the extends cache within a file that might have + to be taken from the cache before being read wouldn't make a lot of sense. + + - Offline and extends-cache settings given by the user's defaults apply to the + process of assembling the project's configuration. If no extends cache has + been specified by the user's default configuration, the project's root + config file must be available, be it from disk or from the net. + + - Offline mode turned on by the ``-o`` command line option is honoured from + the beginning even though command line options are applied to the + configuration last. If offline mode is not requested by the command line, it + may be switched on by either the user's or the project's config root. + + Extends cache + ~~~~~~~~~~~~~ + + Let's see the above rules in action. We create a new home directory for our + user and write user and project configuration that recursively extends online + bases, using different caches: + + >>> mkdir('home') + >>> mkdir('home', '.buildout') + >>> mkdir('cache') + >>> mkdir('user-cache') + >>> os.environ['HOME'] = join(sample_buildout, 'home') + >>> write('home', '.buildout', 'default.cfg', """\ + ... [buildout] + ... extends = fancy_default.cfg + ... extends-cache = user-cache + ... """) + >>> write('home', '.buildout', 'fancy_default.cfg', """\ + ... [buildout] + ... extends = %sbase_default.cfg + ... """ % server_url) + >>> write(server_data, 'base_default.cfg', """\ + ... [buildout] + ... foo = bar + ... offline = false + ... """) + + >>> write('buildout.cfg', """\ + ... [buildout] + ... extends = fancy.cfg + ... extends-cache = cache + ... """) + >>> write('fancy.cfg', """\ + ... [buildout] + ... extends = %sbase.cfg + ... """ % server_url) + >>> write(server_data, 'base.cfg', """\ + ... [buildout] + ... parts = + ... offline = false + ... """) + + Buildout will now assemble its configuration from all of these 6 files, + defaults first. The online resources end up in the respective extends caches: + + >>> print system(buildout) + Unused options for buildout: 'foo'. + + >>> ls('user-cache') + - 10e772cf422123ef6c64ae770f555740 + >>> cat('user-cache', os.listdir('user-cache')[0]) + [buildout] + foo = bar + offline = false + + >>> ls('cache') + - c72213127e6eb2208a3e1fc1dba771a7 + >>> cat('cache', os.listdir('cache')[0]) + [buildout] + parts = + offline = false + + If, on the other hand, the extends caches are specified in files that get + extended themselves, they won't be used for assembling the configuration they + belong to (user's or project's, resp.). The extends cache specified by the + user's defaults does, however, apply to downloading project configuration. + Let's rewrite the config files, clean out the caches and re-run buildout: + + >>> write('home', '.buildout', 'default.cfg', """\ + ... [buildout] + ... extends = fancy_default.cfg + ... """) + >>> write('home', '.buildout', 'fancy_default.cfg', """\ + ... [buildout] + ... extends = %sbase_default.cfg + ... extends-cache = user-cache + ... """ % server_url) + + >>> write('buildout.cfg', """\ + ... [buildout] + ... extends = fancy.cfg + ... """) + >>> write('fancy.cfg', """\ + ... [buildout] + ... extends = %sbase.cfg + ... extends-cache = cache + ... """ % server_url) + + >>> remove('user-cache', os.listdir('user-cache')[0]) + >>> remove('cache', os.listdir('cache')[0]) + + >>> print system(buildout) + Unused options for buildout: 'foo'. + + >>> ls('user-cache') + - 0548bad6002359532de37385bb532e26 + >>> cat('user-cache', os.listdir('user-cache')[0]) + [buildout] + parts = + offline = false + + >>> ls('cache') + + Clean up: + + >>> rmdir('user-cache') + >>> rmdir('cache') + + Offline mode and installation from cache + ----------------------------~~~~~~~~~~~~ + + If we run buildout in offline mode now, it will fail because it cannot get at + the remote configuration file needed by the user's defaults: + + >>> print system(buildout + ' -o') + While: + Initializing. + Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. + + Let's now successively turn on offline mode by different parts of the + configuration and see when buildout applies this setting in each case: + + >>> write('home', '.buildout', 'default.cfg', """\ + ... [buildout] + ... extends = fancy_default.cfg + ... offline = true + ... """) + >>> print system(buildout) + While: + Initializing. + Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. + + >>> write('home', '.buildout', 'default.cfg', """\ + ... [buildout] + ... extends = fancy_default.cfg + ... """) + >>> write('home', '.buildout', 'fancy_default.cfg', """\ + ... [buildout] + ... extends = %sbase_default.cfg + ... offline = true + ... """ % server_url) + >>> print system(buildout) + While: + Initializing. + Error: Couldn't download 'http://localhost/base.cfg' in offline mode. + + >>> write('home', '.buildout', 'fancy_default.cfg', """\ + ... [buildout] + ... extends = %sbase_default.cfg + ... """ % server_url) + >>> write('buildout.cfg', """\ + ... [buildout] + ... extends = fancy.cfg + ... offline = true + ... """) + >>> print system(buildout) + While: + Initializing. + Error: Couldn't download 'http://localhost/base.cfg' in offline mode. + + >>> write('buildout.cfg', """\ + ... [buildout] + ... extends = fancy.cfg + ... """) + >>> write('fancy.cfg', """\ + ... [buildout] + ... extends = %sbase.cfg + ... offline = true + ... """ % server_url) + >>> print system(buildout) + Unused options for buildout: 'foo'. + + The ``install-from-cache`` option is treated accordingly: + + >>> write('home', '.buildout', 'default.cfg', """\ + ... [buildout] + ... extends = fancy_default.cfg + ... install-from-cache = true + ... """) + >>> print system(buildout) + While: + Initializing. + Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. + + >>> write('home', '.buildout', 'default.cfg', """\ + ... [buildout] + ... extends = fancy_default.cfg + ... """) + >>> write('home', '.buildout', 'fancy_default.cfg', """\ + ... [buildout] + ... extends = %sbase_default.cfg + ... install-from-cache = true + ... """ % server_url) + >>> print system(buildout) + While: + Initializing. + Error: Couldn't download 'http://localhost/base.cfg' in offline mode. + + >>> write('home', '.buildout', 'fancy_default.cfg', """\ + ... [buildout] + ... extends = %sbase_default.cfg + ... """ % server_url) + >>> write('buildout.cfg', """\ + ... [buildout] + ... extends = fancy.cfg + ... install-from-cache = true + ... """) + >>> print system(buildout) + While: + Initializing. + Error: Couldn't download 'http://localhost/base.cfg' in offline mode. + + >>> write('buildout.cfg', """\ + ... [buildout] + ... extends = fancy.cfg + ... """) + >>> write('fancy.cfg', """\ + ... [buildout] + ... extends = %sbase.cfg + ... install-from-cache = true + ... """ % server_url) + >>> print system(buildout) + While: + Installing. + Checking for upgrades. + An internal error occurred ... + ValueError: install_from_cache set to true with no download cache + + + Clean up + -------- + + We should have cleaned up all temporary files created by downloading things: + + >>> ls(tempfile.tempdir) + + Reset the global temporary directory: + + >>> tempfile.tempdir = old_tempdir + + Using zc.buildout to run setup scripts + ====================================== + + zc buildout has a convenience command for running setup scripts. Why? + There are two reasons. If a setup script doesn't import setuptools, + you can't use any setuptools-provided commands, like bdist_egg. When + buildout runs a setup script, it arranges to import setuptools before + running the script so setuptools-provided commands are available. + + If you use a squeaky-clean Python to do your development, the setup + script that would import setuptools because setuptools isn't in the + path. Because buildout requires setuptools and knows where it has + installed a setuptools egg, it adds the setuptools egg to the Python + path before running the script. To run a setup script, use the + buildout setup command, passing the name of a script or a directory + containing a setup script and arguments to the script. Let's look at + an example: + + >>> mkdir('test') + >>> cd('test') + >>> write('setup.py', + ... ''' + ... from distutils.core import setup + ... setup(name='sample') + ... ''') + + We've created a super simple (stupid) setup script. Note that it + doesn't import setuptools. Let's try running it to create an egg. + We'll use the buildout script from our sample buildout: + + >>> print system(buildout+' setup'), + ... # doctest: +NORMALIZE_WHITESPACE + Error: The setup command requires the path to a setup script or + directory containing a setup script, and its arguments. + + Oops, we forgot to give the name of the setup script: + + >>> print system(buildout+' setup setup.py bdist_egg'), + ... # doctest: +ELLIPSIS + Running setup script 'setup.py'. + ... + + >>> ls('dist') + - sample-0.0.0-py2.5.egg + + Note that we can specify a directory name. This is often shorter and + preferred by the lazy :) + + >>> print system(buildout+' setup . bdist_egg'), # doctest: +ELLIPSIS + Running setup script './setup.py'. + ... + + Automatic Buildout Updates + ========================== + + When a buildout is run, one of the first steps performed is to check + for updates to either zc.buildout or setuptools. To demonstrate this, + we've created some "new releases" of buildout and setuptools in a + new_releases folder: + + >>> ls(new_releases) + d setuptools + - setuptools-99.99-py2.4.egg + d zc.buildout + - zc.buildout-100.0b1-pyN.N.egg + - zc.buildout-99.99-py2.4.egg + + Let's update the sample buildout.cfg to look in this area: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... find-links = %(new_releases)s + ... index = %(new_releases)s + ... parts = show-versions + ... develop = showversions + ... + ... [show-versions] + ... recipe = showversions + ... """ % dict(new_releases=new_releases)) + + We'll also include a recipe that echos the versions of setuptools and + zc.buildout used: + + >>> mkdir(sample_buildout, 'showversions') + + >>> write(sample_buildout, 'showversions', 'showversions.py', + ... """ + ... import pkg_resources + ... + ... class Recipe: + ... + ... def __init__(self, buildout, name, options): + ... pass + ... + ... def install(self): + ... for project in 'zc.buildout', 'setuptools': + ... req = pkg_resources.Requirement.parse(project) + ... print project, pkg_resources.working_set.find(req).version + ... return () + ... update = install + ... """) + + + >>> write(sample_buildout, 'showversions', 'setup.py', + ... """ + ... from setuptools import setup + ... + ... setup( + ... name = "showversions", + ... entry_points = {'zc.buildout': ['default = showversions:Recipe']}, + ... ) + ... """) + + + Now if we run the buildout, the buildout will upgrade itself to the + new versions found in new releases: + + >>> print system(buildout), + Getting distribution for 'zc.buildout'. + Got zc.buildout 99.99. + Getting distribution for 'setuptools'. + Got setuptools 99.99. + Upgraded: + zc.buildout version 99.99, + setuptools version 99.99; + restarting. + Generated script '/sample-buildout/bin/buildout'. + Develop: '/sample-buildout/showversions' + Installing show-versions. + zc.buildout 99.99 + setuptools 99.99 + + Notice that, even though we have a newer beta version of zc.buildout + available, the final "99.99" was selected. If you want to get non-final + versions, specify a specific version in your buildout's versions + section, you typically want to use the --accept-buildout-test-releases + option to the bootstrap script, which internally uses the + ``accept-buildout-test-releases = true`` discussed below. + + Our buildout script's site.py has been updated to use the new eggs: + + >>> cat(sample_buildout, 'parts', 'buildout', 'site.py') + ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS + "... + def addsitepackages(known_paths): + """Add site packages, as determined by zc.buildout. + <BLANKLINE> + See original_addsitepackages, below, for the original version.""" + setuptools_path = '/sample-buildout/eggs/setuptools-99.99-pyN.N.egg' + sys.path.append(setuptools_path) + known_paths.add(os.path.normcase(setuptools_path)) + import pkg_resources + buildout_paths = [ + '/sample-buildout/eggs/zc.buildout-99.99-pyN.N.egg', + '/sample-buildout/eggs/setuptools-99.99-pyN.N.egg' + ] + for path in buildout_paths: + sitedir, sitedircase = makepath(path) + if not sitedircase in known_paths and os.path.exists(sitedir): + sys.path.append(sitedir) + known_paths.add(sitedircase) + pkg_resources.working_set.add_entry(sitedir) + sys.__egginsert = len(buildout_paths) # Support setuptools. + original_paths = [ + ... + ] + for path in original_paths: + if path == setuptools_path or path not in known_paths: + addsitedir(path, known_paths) + return known_paths + ... + + Now, let's recreate the sample buildout. If we specify constraints on + the versions of zc.buildout and setuptools (or distribute) to use, + running the buildout will install earlier versions of these packages: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... find-links = %(new_releases)s + ... index = %(new_releases)s + ... parts = show-versions + ... develop = showversions + ... zc.buildout-version = < 99 + ... setuptools-version = < 99 + ... distribute-version = < 99 + ... + ... [show-versions] + ... recipe = showversions + ... """ % dict(new_releases=new_releases)) + + Now we can see that we actually "upgrade" to an earlier version. + + >>> print system(buildout), + Upgraded: + zc.buildout version 1.0.0, + setuptools version 0.6; + restarting. + Develop: '/sample-buildout/showversions' + Updating show-versions. + zc.buildout 1.0.0 + setuptools 0.6 + + There are a number of cases, described below, in which the updates + don't happen. + + We won't upgrade in offline mode: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... find-links = %(new_releases)s + ... index = %(new_releases)s + ... parts = show-versions + ... develop = showversions + ... + ... [show-versions] + ... recipe = showversions + ... """ % dict(new_releases=new_releases)) + + >>> print system(buildout+' -o'), + Develop: '/sample-buildout/showversions' + Updating show-versions. + zc.buildout 1.0.0 + setuptools 0.6 + + Or in non-newest mode: + + >>> print system(buildout+' -N'), + Develop: '/sample-buildout/showversions' + Updating show-versions. + zc.buildout 1.0.0 + setuptools 0.6 + + We also won't upgrade if the buildout script being run isn't in the + buildout's bin directory. To see this we'll create a new buildout + directory: + + >>> sample_buildout2 = tmpdir('sample_buildout2') + >>> write(sample_buildout2, 'buildout.cfg', + ... """ + ... [buildout] + ... find-links = %(new_releases)s + ... index = %(new_releases)s + ... parts = + ... """ % dict(new_releases=new_releases)) + + >>> cd(sample_buildout2) + >>> print system(buildout), + Creating directory '/sample_buildout2/bin'. + Creating directory '/sample_buildout2/parts'. + Creating directory '/sample_buildout2/eggs'. + Creating directory '/sample_buildout2/develop-eggs'. + Getting distribution for 'zc.buildout'. + Got zc.buildout 99.99. + Getting distribution for 'setuptools'. + Got setuptools 99.99. + Not upgrading because not running a local buildout command. + + >>> ls('bin') + + As mentioned above, the ``accept-buildout-test-releases = true`` means that + newer non-final versions of these dependencies are preferred. Typically + users are not expected to actually manipulate this value. Instead, the + bootstrap script creates a buildout buildout script that passes in the + value as a command line override. This then results in the buildout + script being rewritten to remember the decision. + + We'll mimic this by passing the argument actually in the command line. + + >>> cd(sample_buildout) + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... find-links = %(new_releases)s + ... index = %(new_releases)s + ... parts = show-versions + ... develop = showversions + ... + ... [show-versions] + ... recipe = showversions + ... """ % dict(new_releases=new_releases)) + + >>> print system(buildout + + ... ' buildout:accept-buildout-test-releases=true'), + ... # doctest: +NORMALIZE_WHITESPACE + Getting distribution for 'zc.buildout'. + Got zc.buildout 100.0b1. + Upgraded: + zc.buildout version 100.0b1, + setuptools version 99.99; + restarting. + Generated script '/sample-buildout/bin/buildout'. + NOTE: Accepting early releases of build system packages. Rerun bootstrap + without --accept-buildout-test-releases (-t) to return to default + behavior. + Develop: '/sample-buildout/showversions' + Updating show-versions. + zc.buildout 100.0b1 + setuptools 99.99 + + The buildout script shows the change. + + >>> buildout_script = join(sample_buildout, 'bin', 'buildout') + >>> import sys + >>> if sys.platform.startswith('win'): + ... buildout_script += '-script.py' + >>> print open(buildout_script).read() # doctest: +ELLIPSIS + #... + sys.argv.insert(1, 'buildout:accept-buildout-test-releases=true') + print ('NOTE: Accepting early releases of build system packages. Rerun ' + 'bootstrap without --accept-buildout-test-releases (-t) to return to ' + 'default behavior.') + ... + + Debugging buildouts + =================== + + Buildouts can be pretty complex. When things go wrong, it isn't + always obvious why. Errors can occur due to problems in user input or + due to bugs in zc.buildout or recipes. When an error occurs, Python's + post-mortem debugger can be used to inspect the state of the buildout + or recipe code where the error occurred. To enable this, use the -D + option to the buildout. Let's create a recipe that has a bug: + + >>> mkdir(sample_buildout, 'recipes') + + >>> write(sample_buildout, 'recipes', 'mkdir.py', + ... """ + ... import os, zc.buildout + ... + ... class Mkdir: + ... + ... def __init__(self, buildout, name, options): + ... self.name, self.options = name, options + ... options['path'] = os.path.join( + ... buildout['buildout']['directory'], + ... options['path'], + ... ) + ... + ... def install(self): + ... directory = self.options['directory'] + ... os.mkdir(directory) + ... return directory + ... + ... def update(self): + ... pass + ... """) + + >>> write(sample_buildout, 'recipes', 'setup.py', + ... """ + ... from setuptools import setup + ... + ... setup(name = "recipes", + ... entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']}, + ... ) + ... """) + + And create a buildout that uses it: + + >>> write(sample_buildout, 'buildout.cfg', + ... """ + ... [buildout] + ... develop = recipes + ... parts = data-dir + ... + ... [data-dir] + ... recipe = recipes:mkdir + ... path = mystuff + ... """) + + If we run the buildout, we'll get an error: + + >>> print system(buildout), + Develop: '/sample-buildout/recipes' + Installing data-dir. + While: + Installing data-dir. + Error: Missing option: data-dir:directory + + + If we want to debug the error, we can add the -D option. Here's we'll + supply some input: + + >>> print system(buildout+" -D", """\ + ... up + ... p self.options.keys() + ... q + ... """), + Develop: '/sample-buildout/recipes' + Installing data-dir. + > /zc/buildout/buildout.py(925)__getitem__() + -> raise MissingOption("Missing option: %s:%s" % (self.name, key)) + (Pdb) > /sample-buildout/recipes/mkdir.py(14)install() + -> directory = self.options['directory'] + (Pdb) ['path', 'recipe'] + (Pdb) While: + Installing data-dir. + Traceback (most recent call last): + File "/zc/buildout/buildout.py", line 1352, in main + getattr(buildout, command)(args) + File "/zc/buildout/buildout.py", line 383, in install + installed_files = self[part]._call(recipe.install) + File "/zc/buildout/buildout.py", line 961, in _call + return f() + File "/sample-buildout/recipes/mkdir.py", line 14, in install + directory = self.options['directory'] + File "/zc/buildout/buildout.py", line 925, in __getitem__ + raise MissingOption("Missing option: %s:%s" % (self.name, key)) + MissingOption: Missing option: data-dir:directory + <BLANKLINE> + Starting pdb: + + Testing Support + =============== + + The zc.buildout.testing module provides an API that can be used when + writing recipe tests. This API is documented below. Many examples of + using this API can be found in the zc.buildout, zc.recipe.egg, and + zc.recipe.testrunner tests. + + zc.buildout.testing.buildoutSetUp(test) + --------------------------------------- + + The buildoutSetup function can be used as a doctest setup function. + It creates a sample buildout that can be used by tests, changing the + current working directory to the sample_buildout. It also adds a + number of names to the test namespace: + + ``sample_buildout`` + This is the name of a buildout with a basic configuration. + + ``buildout`` + This is the path of the buildout script in the sample buildout. + + ``ls(*path)`` + List the contents of a directory. The directory path is provided as one or + more strings, to be joined with os.path.join. + + ``cat(*path)`` + Display the contents of a file. The file path is provided as one or + more strings, to be joined with os.path.join. + + On Windows, if the file doesn't exist, the function will try + adding a '-script.py' suffix. This helps to work around a + difference in script generation on windows. + + ``mkdir(*path)`` + Create a directory. The directory path is provided as one or + more strings, to be joined with os.path.join. + + ``rmdir(*path)`` + Remove a directory. The directory path is provided as one or + more strings, to be joined with os.path.join. + + ``remove(*path)`` + Remove a directory or file. The path is provided as one or + more strings, to be joined with os.path.join. + + ``tmpdir(name)`` + Create a temporary directory with the given name. The directory + will be automatically removed at the end of the test. The path of + the created directory is returned. + + Further, if the the normalize_path normlaizing substitution (see + below) is used, then any paths starting with this path will be + normalized to:: + + /name/restofpath + + No two temporary directories can be created with the same name. A + directory created with tmpdir can be removed with rmdir and recreated. + + Note that the sample_buildout directory is created by calling this + function. + + ``write(*path_and_contents)`` + Create a file. The file path is provided as one or more strings, + to be joined with os.path.join. The last argument is the file contents. + + ``system(command, input='')`` + Execute a system command with the given input passed to the + command's standard input. The output (error and regular output) + from the command is returned. + + ``get(url)`` + Get a web page. + + ``cd(*path)`` + Change to the given directory. The directory path is provided as one or + more strings, to be joined with os.path.join. + + The directory will be reset at the end of the test. + + ``join(*path)`` + A convenient reference to os.path.join. + + ``register_teardown(func)`` + Register a tear-down function. The function will be called with + no arguments at the end of the test. + + ``start_server(path)`` + Start a web server on the given path. The server will be shut + down at the end of the test. The server URL is returned. + + You can cause the server to start and stop logging it's output + using: + + >>> get(server_url+'enable_server_logging') + + and: + + >>> get(server_url+'disable_server_logging') + + This can be useful to see how buildout is interacting with a + server. + + + ``sdist(setup, dest)`` + Create a source distribution by running the given setup file and + placing the result in the given destination directory. If the + setup argument is a directory, the thge setup.py file in that + directory is used. + + ``bdist_egg(setup, executable, dest)`` + Create an egg by running the given setup file with the given + Python executable and placing the result in the given destination + directory. If the setup argument is a directory, then the + setup.py file in that directory is used. + + ``find_python(version)`` + Find a Python executable for the given version, where version is a + string like "2.4". + + This function uses the following strategy to find a Python of the + given version: + + - Look for an environment variable of the form PYTHON%(version)s. + + - On windows, look for \Pythonm%(version)s\python + + - on Unix, try running python%(version)s or just python to get the + executable + + ``zc.buildout.testing.buildoutTearDown(test)`` + ---------------------------------------------- + + Tear down everything set up by zc.buildout.testing.buildoutSetUp. Any + functions passed to register_teardown are called as well. + + ``install(project, destination)`` + --------------------------------- + + Install eggs for a given project into a destination. If the + destination is a test object, then the eggs directory of the + sample buildout (sample_buildout) defined by the test will be used. + Tests will use this to install the distributions for the packages + being tested (and their dependencies) into a sample buildout. The egg + to be used should already be loaded, by importing one of the modules + provided, before calling this function. + + ``install_develop(project, destination)`` + ----------------------------------------- + + Like install, but a develop egg is installed even if the current egg + if not a develop egg. + + ``Output normalization`` + ------------------------ + + Recipe tests often generate output that is dependent on temporary file + locations, operating system conventions or Python versions. To deal + with these dependencies, we often use + zope.testing.renormalizing.RENormalizing to normalize test output. + zope.testing.renormalizing.RENormalizing takes pairs of regular + expressions and substitutions. The zc.buildout.testing module provides + a few helpful variables that define regular-expression/substitution + pairs that you can pass to zope.testing.renormalizing.RENormalizing. + + + ``normalize_path`` + Converts tests paths, based on directories created with tmpdir(), + to simple paths. + + ``normalize_script`` + On Unix-like systems, scripts are implemented in single files + without suffixes. On windows, scripts are implemented with 2 + files, a -script.py file and a .exe file. This normalization + converts directory listings of Windows scripts to the form + generated on UNix-like systems. + + ``normalize_egg_py`` + Normalize Python version and platform indicators, if specified, in + egg names. + + Python API for egg and script installation + ========================================== + + The easy_install module provides some functions to provide support for + egg and script installation. It provides functionality at the python + level that is similar to easy_install, with a few exceptions: + + - By default, we look for new packages *and* the packages that + they depend on. This is somewhat like (and uses) the --upgrade + option of easy_install, except that we also upgrade required + packages. + + - If the highest-revision package satisfying a specification is + already present, then we don't try to get another one. This saves a + lot of search time in the common case that packages are pegged to + specific versions. + + - If there is a develop egg that satisfies a requirement, we don't + look for additional distributions. We always give preference to + develop eggs. + + - Distutils options for building extensions can be passed. + + Distribution installation + ------------------------- + + The easy_install module provides a function, install, for installing one + or more packages and their dependencies. The install function takes 2 + positional arguments: + + - An iterable of setuptools requirement strings for the distributions + to be installed, and + + - A destination directory to install to and to satisfy requirements + from. The destination directory can be None, in which case, no new + distributions are downloaded and there will be an error if the + needed distributions can't be found among those already installed. + + It supports a number of optional keyword arguments: + + links + A sequence of URLs, file names, or directories to look for + links to distributions. + + index + The URL of an index server, or almost any other valid URL. :) + + If not specified, the Python Package Index, + http://pypi.python.org/simple/, is used. You can specify an + alternate index with this option. If you use the links option and + if the links point to the needed distributions, then the index can + be anything and will be largely ignored. In the examples, here, + we'll just point to an empty directory on our link server. This + will make our examples run a little bit faster. + + executable + A path to a Python executable. Distributions will be installed + using this executable and will be for the matching Python version. + + path + A list of additional directories to search for locally-installed + distributions. + + always_unzip + A flag indicating that newly-downloaded distributions should be + directories even if they could be installed as zip files. + + working_set + An existing working set to be augmented with additional + distributions, if necessary to satisfy requirements. This allows + you to call install multiple times, if necessary, to gather + multiple sets of requirements. + + newest + A boolean value indicating whether to search for new distributions + when already-installed distributions meet the requirement. When + this is true, the default, and when the destination directory is + not None, then the install function will search for the newest + distributions that satisfy the requirements. + + versions + A dictionary mapping project names to version numbers to be used + when selecting distributions. This can be used to specify a set of + distribution versions independent of other requirements. + + use_dependency_links + A flag indicating whether to search for dependencies using the + setup dependency_links metadata or not. If true, links are searched + for using dependency_links in preference to other + locations. Defaults to true. + + include_site_packages + A flag indicating whether Python's non-standard-library packages should + be available for finding dependencies. Defaults to true. + + Paths outside of Python's standard library--or more precisely, those that + are not included when Python is started with the -S argument--are loosely + referred to as "site-packages" here. + + relative_paths + Adjust egg paths so they are relative to the script path. This + allows scripts to work when scripts and eggs are moved, as long as + they are both moved in the same way. + + The install method returns a working set containing the distributions + needed to meet the given requirements. + + We have a link server that has a number of eggs: + + >>> print get(link_server), + <html><body> + <a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br> + <a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br> + <a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br> + <a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br> + <a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br> + <a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br> + <a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br> + <a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br> + <a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br> + <a href="index/">index/</a><br> + <a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br> + </body></html> + + Let's make a directory and install the demo egg to it, using the demo: + + >>> dest = tmpdir('sample-install') + >>> import zc.buildout.easy_install + >>> ws = zc.buildout.easy_install.install( + ... ['demo==0.2'], dest, + ... links=[link_server], index=link_server+'index/') + + We requested version 0.2 of the demo distribution to be installed into + the destination server. We specified that we should search for links + on the link server and that we should use the (empty) link server + index directory as a package index. + + The working set contains the distributions we retrieved. + + >>> for dist in ws: + ... print dist + demo 0.2 + demoneeded 1.1 + + We got demoneeded because it was a dependency of demo. + + And the actual eggs were added to the eggs directory. + + >>> ls(dest) + - demo-0.2-py2.4.egg + - demoneeded-1.1-py2.4.egg + + If we remove the version restriction on demo, but specify a false + value for newest, no new distributions will be installed: + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/', + ... newest=False) + >>> ls(dest) + - demo-0.2-py2.4.egg + - demoneeded-1.1-py2.4.egg + + If we leave off the newest option, we'll get an update for demo: + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/') + >>> ls(dest) + - demo-0.2-py2.4.egg + - demo-0.3-py2.4.egg + - demoneeded-1.1-py2.4.egg + + Note that we didn't get the newest versions available. There were + release candidates for newer versions of both packages. By default, + final releases are preferred. We can change this behavior using the + prefer_final function: + + >>> zc.buildout.easy_install.prefer_final(False) + True + + The old setting is returned. + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/') + >>> for dist in ws: + ... print dist + demo 0.4c1 + demoneeded 1.2c1 + + >>> ls(dest) + - demo-0.2-py2.4.egg + - demo-0.3-py2.4.egg + - demo-0.4c1-py2.4.egg + - demoneeded-1.1-py2.4.egg + - demoneeded-1.2c1-py2.4.egg + + Let's put the setting back to the default. + + >>> zc.buildout.easy_install.prefer_final(True) + False + + We can supply additional distributions. We can also supply + specifications for distributions that would normally be found via + dependencies. We might do this to specify a specific version. + + >>> ws = zc.buildout.easy_install.install( + ... ['demo', 'other', 'demoneeded==1.0'], dest, + ... links=[link_server], index=link_server+'index/') + + >>> for dist in ws: + ... print dist + demo 0.3 + other 1.0 + demoneeded 1.0 + + >>> ls(dest) + - demo-0.2-py2.4.egg + - demo-0.3-py2.4.egg + - demo-0.4c1-py2.4.egg + - demoneeded-1.0-py2.4.egg + - demoneeded-1.1-py2.4.egg + - demoneeded-1.2c1-py2.4.egg + d other-1.0-py2.4.egg + + We can request that eggs be unzipped even if they are zip safe. This + can be useful when debugging. (Note that Distribute will unzip eggs by + default, so if you are using Distribute, most or all eggs will already be + unzipped without this flag.) + + >>> rmdir(dest) + >>> dest = tmpdir('sample-install') + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/', + ... always_unzip=True) + + >>> ls(dest) + d demo-0.3-py2.4.egg + d demoneeded-1.1-py2.4.egg + + >>> rmdir(dest) + >>> dest = tmpdir('sample-install') + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/', + ... always_unzip=False) + + >>> ls(dest) + - demo-0.3-py2.4.egg + - demoneeded-1.1-py2.4.egg + + We can also set a default by calling the always_unzip function: + + >>> zc.buildout.easy_install.always_unzip(True) + False + + The old default is returned: + + >>> rmdir(dest) + >>> dest = tmpdir('sample-install') + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/') + + >>> ls(dest) + d demo-0.3-py2.4.egg + d demoneeded-1.1-py2.4.egg + + + >>> zc.buildout.easy_install.always_unzip(False) + True + + >>> rmdir(dest) + >>> dest = tmpdir('sample-install') + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/') + + >>> ls(dest) + - demo-0.3-py2.4.egg + - demoneeded-1.1-py2.4.egg + + >>> rmdir(dest) + >>> dest = tmpdir('sample-install') + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/', + ... always_unzip=True) + + >>> ls(dest) + d demo-0.3-py2.4.egg + d demoneeded-1.1-py2.4.egg + + Specifying version information independent of requirements + ---------------------------------------------------------- + + Sometimes it's useful to specify version information independent of + normal requirements specifications. For example, a buildout may need + to lock down a set of versions, without having to put put version + numbers in setup files or part definitions. If a dictionary is passed + to the install function, mapping project names to version numbers, + then the versions numbers will be used. + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/', + ... versions = dict(demo='0.2', demoneeded='1.0')) + >>> [d.version for d in ws] + ['0.2', '1.0'] + + In this example, we specified a version for demoneeded, even though we + didn't define a requirement for it. The versions specified apply to + dependencies as well as the specified requirements. + + If we specify a version that's incompatible with a requirement, then + we'll get an error: + + >>> from zope.testing.loggingsupport import InstalledHandler + >>> handler = InstalledHandler('zc.buildout.easy_install') + >>> import logging + >>> logging.getLogger('zc.buildout.easy_install').propagate = False + + >>> ws = zc.buildout.easy_install.install( + ... ['demo >0.2'], dest, links=[link_server], + ... index=link_server+'index/', + ... versions = dict(demo='0.2', demoneeded='1.0')) + Traceback (most recent call last): + ... + IncompatibleVersionError: Bad version 0.2 + + >>> print handler + zc.buildout.easy_install DEBUG + Installing 'demo >0.2'. + zc.buildout.easy_install ERROR + The version, 0.2, is not consistent with the requirement, 'demo>0.2'. + + >>> handler.clear() + + If no versions are specified, a debugging message will be output + reporting that a version was picked automatically: + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/', + ... ) + + >>> print handler + zc.buildout.easy_install DEBUG + Installing 'demo'. + zc.buildout.easy_install DEBUG + We have the best distribution that satisfies 'demo'. + zc.buildout.easy_install DEBUG + Picked: demo = 0.3 + zc.buildout.easy_install DEBUG + Getting required 'demoneeded' + zc.buildout.easy_install DEBUG + required by demo 0.3. + zc.buildout.easy_install DEBUG + We have the best distribution that satisfies 'demoneeded'. + zc.buildout.easy_install DEBUG + Picked: demoneeded = 1.1 + + >>> handler.uninstall() + >>> logging.getLogger('zc.buildout.easy_install').propagate = True + + We can request that we get an error if versions are picked: + + >>> zc.buildout.easy_install.allow_picked_versions(False) + True + + (The old setting is returned.) + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/', + ... ) + Traceback (most recent call last): + ... + UserError: Picked: demo = 0.3 + + >>> zc.buildout.easy_install.allow_picked_versions(True) + False + + The function default_versions can be used to get and set default + version information to be used when no version information is passes. + If called with an argument, it sets the default versions: + + >>> zc.buildout.easy_install.default_versions(dict(demoneeded='1')) + {} + + It always returns the previous default versions. If called without an + argument, it simply returns the default versions without changing + them: + + >>> zc.buildout.easy_install.default_versions() + {'demoneeded': '1'} + + So with the default versions set, we'll get the requested version even + if the versions option isn't used: + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/', + ... ) + + >>> [d.version for d in ws] + ['0.3', '1.0'] + + Of course, we can unset the default versions by passing an empty + dictionary: + + >>> zc.buildout.easy_install.default_versions({}) + {'demoneeded': '1'} + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, links=[link_server], index=link_server+'index/', + ... ) + + >>> [d.version for d in ws] + ['0.3', '1.1'] + + Dependencies in Site Packages + ----------------------------- + + Paths outside of Python's standard library--or more precisely, those that are + not included when Python is started with the -S argument--are loosely referred + to as "site-packages" here. These site-packages are searched by default for + distributions. This can be disabled, so that, for instance, a system Python + can be used with buildout, cleaned of any packages installed by a user or + system package manager. + + The default behavior can be controlled and introspected using + zc.buildout.easy_install.include_site_packages. + + >>> zc.buildout.easy_install.include_site_packages() + True + + Here's an example of using a Python executable that includes our dependencies. + + Our "py_path" will have the "demoneeded," and "demo" packages available. + We'll simply be asking for "demoneeded" here, but without any external + index or links. + + >>> from zc.buildout.tests import create_sample_sys_install + >>> py_path, site_packages_path = make_py() + >>> create_sample_sys_install(site_packages_path) + + >>> example_dest = tmpdir('site-packages-example-install') + >>> workingset = zc.buildout.easy_install.install( + ... ['demoneeded'], example_dest, links=[], executable=py_path, + ... index=None) + >>> [dist.project_name for dist in workingset] + ['demoneeded'] + + That worked fine. Let's try again with site packages not allowed. We'll + change the policy by changing the default. Notice that the function for + changing the default value returns the previous value. + + >>> zc.buildout.easy_install.include_site_packages(False) + True + + >>> zc.buildout.easy_install.include_site_packages() + False + + >>> zc.buildout.easy_install.clear_index_cache() + >>> rmdir(example_dest) + >>> example_dest = tmpdir('site-packages-example-install') + >>> workingset = zc.buildout.easy_install.install( + ... ['demoneeded'], example_dest, links=[], executable=py_path, + ... index=None) + Traceback (most recent call last): + ... + MissingDistribution: Couldn't find a distribution for 'demoneeded'. + >>> zc.buildout.easy_install.clear_index_cache() + + Now we'll reset the default. + + >>> zc.buildout.easy_install.include_site_packages(True) + False + + >>> zc.buildout.easy_install.include_site_packages() + True + + Dependency links + ---------------- + + Setuptools allows metadata that describes where to search for package + dependencies. This option is called dependency_links. Buildout has its + own notion of where to look for dependencies, but it also uses the + setup tools dependency_links information if it's available. + + Let's demo this by creating an egg that specifies dependency_links. + + To begin, let's create a new egg repository. This repository hold a + newer version of the 'demoneeded' egg than the sample repository does. + + >>> repoloc = tmpdir('repo') + >>> from zc.buildout.tests import create_egg + >>> create_egg('demoneeded', '1.2', repoloc) + >>> link_server2 = start_server(repoloc) + + Turn on logging on this server so that we can see when eggs are pulled + from it. + + >>> get(link_server2 + 'enable_server_logging') + GET 200 /enable_server_logging + '' + + Now we can create an egg that specifies that its dependencies are + found on this server. + + >>> repoloc = tmpdir('repo2') + >>> create_egg('hasdeps', '1.0', repoloc, + ... install_requires = "'demoneeded'", + ... dependency_links = [link_server2]) + + Let's add the egg to another repository. + + >>> link_server3 = start_server(repoloc) + + Now let's install the egg. + + >>> example_dest = tmpdir('example-install') + >>> workingset = zc.buildout.easy_install.install( + ... ['hasdeps'], example_dest, + ... links=[link_server3], index=link_server3+'index/') + GET 200 / + GET 200 /demoneeded-1.2-pyN.N.egg + + The server logs show that the dependency was retrieved from the server + specified in the dependency_links. + + Now let's see what happens if we provide two different ways to retrieve + the dependencies. + + >>> rmdir(example_dest) + >>> example_dest = tmpdir('example-install') + >>> workingset = zc.buildout.easy_install.install( + ... ['hasdeps'], example_dest, index=link_server+'index/', + ... links=[link_server, link_server3]) + GET 200 / + GET 200 /demoneeded-1.2-pyN.N.egg + + Once again the dependency is fetched from the logging server even + though it is also available from the non-logging server. This is + because the version on the logging server is newer and buildout + normally chooses the newest egg available. + + If you wish to control where dependencies come from regardless of + dependency_links setup metadata use the 'use_dependency_links' option + to zc.buildout.easy_install.install(). + + >>> rmdir(example_dest) + >>> example_dest = tmpdir('example-install') + >>> workingset = zc.buildout.easy_install.install( + ... ['hasdeps'], example_dest, index=link_server+'index/', + ... links=[link_server, link_server3], + ... use_dependency_links=False) + + Notice that this time the dependency egg is not fetched from the + logging server. When you specify not to use dependency_links, eggs + will only be searched for using the links you explicitly provide. + + Another way to control this option is with the + zc.buildout.easy_install.use_dependency_links() function. This + function sets the default behavior for the zc.buildout.easy_install() + function. + + >>> zc.buildout.easy_install.use_dependency_links(False) + True + + The function returns its previous setting. + + >>> rmdir(example_dest) + >>> example_dest = tmpdir('example-install') + >>> workingset = zc.buildout.easy_install.install( + ... ['hasdeps'], example_dest, index=link_server+'index/', + ... links=[link_server, link_server3]) + + It can be overridden by passing a keyword argument to the install + function. + + >>> rmdir(example_dest) + >>> example_dest = tmpdir('example-install') + >>> workingset = zc.buildout.easy_install.install( + ... ['hasdeps'], example_dest, index=link_server+'index/', + ... links=[link_server, link_server3], + ... use_dependency_links=True) + GET 200 /demoneeded-1.2-pyN.N.egg + + To return the dependency_links behavior to normal call the function again. + + >>> zc.buildout.easy_install.use_dependency_links(True) + False + >>> rmdir(example_dest) + >>> example_dest = tmpdir('example-install') + >>> workingset = zc.buildout.easy_install.install( + ... ['hasdeps'], example_dest, index=link_server+'index/', + ... links=[link_server, link_server3]) + GET 200 /demoneeded-1.2-pyN.N.egg + + + Script generation + ----------------- + + The easy_install module provides support for creating scripts from eggs. + It provides two competing functions. One, ``scripts``, is a + well-established approach to generating reliable scripts with a "clean" + Python--e.g., one that does not have any packages in its site-packages. + The other, ``sitepackage_safe_scripts``, is newer, a bit trickier, and is + designed to work with a Python that has code in its site-packages, such + as a system Python. + + Both are similar to setuptools except that they provides facilities for + baking a script's path into the script. This has two advantages: + + - The eggs to be used by a script are not chosen at run time, making + startup faster and, more importantly, deterministic. + + - The script doesn't have to import pkg_resources because the logic that + pkg_resources would execute at run time is executed at script-creation + time. (There is an exception in ``sitepackage_safe_scripts`` if you + want to have your Python's site packages available, as discussed + below, but even in that case pkg_resources is only partially + activated, which can be a significant time savings.) + + + The ``scripts`` function + ~~~~~~~~~~~~~~~~~~~~~~~~ + + The ``scripts`` function is the first way to generate scripts that we'll + examine. It is the earlier approach that the package offered. Let's + create a destination directory for it to place them in: + + >>> bin = tmpdir('bin') + + Now, we'll use the scripts function to generate scripts in this directory + from the demo egg: + + >>> import sys + >>> scripts = zc.buildout.easy_install.scripts( + ... ['demo'], ws, sys.executable, bin) + + the four arguments we passed were: + + 1. A sequence of distribution requirements. These are of the same + form as setuptools requirements. Here we passed a single + requirement, for the version 0.1 demo distribution. + + 2. A working set, + + 3. The Python executable to use, and + + 3. The destination directory. + + The bin directory now contains a generated script: + + >>> ls(bin) + - demo + + The return value is a list of the scripts generated: + + >>> import os, sys + >>> if sys.platform == 'win32': + ... scripts == [os.path.join(bin, 'demo.exe'), + ... os.path.join(bin, 'demo-script.py')] + ... else: + ... scripts == [os.path.join(bin, 'demo')] + True + + Note that in Windows, 2 files are generated for each script. A script + file, ending in '-script.py', and an exe file that allows the script + to be invoked directly without having to specify the Python + interpreter and without having to provide a '.py' suffix. + + The demo script run the entry point defined in the demo egg: + + >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 + <BLANKLINE> + import sys + sys.path[0:0] = [ + '/sample-install/demo-0.3-py2.4.egg', + '/sample-install/demoneeded-1.1-py2.4.egg', + ] + <BLANKLINE> + import eggrecipedemo + <BLANKLINE> + if __name__ == '__main__': + eggrecipedemo.main() + + Some things to note: + + - The demo and demoneeded eggs are added to the beginning of sys.path. + + - The module for the script entry point is imported and the entry + point, in this case, 'main', is run. + + Rather than requirement strings, you can pass tuples containing 3 + strings: + + - A script name, + + - A module, + + - An attribute expression for an entry point within the module. + + For example, we could have passed entry point information directly + rather than passing a requirement: + + >>> scripts = zc.buildout.easy_install.scripts( + ... [('demo', 'eggrecipedemo', 'main')], + ... ws, sys.executable, bin) + + >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 + <BLANKLINE> + import sys + sys.path[0:0] = [ + '/sample-install/demo-0.3-py2.4.egg', + '/sample-install/demoneeded-1.1-py2.4.egg', + ] + <BLANKLINE> + import eggrecipedemo + <BLANKLINE> + if __name__ == '__main__': + eggrecipedemo.main() + + Passing entry-point information directly is handy when using eggs (or + distributions) that don't declare their entry points, such as + distributions that aren't based on setuptools. + + The interpreter keyword argument can be used to generate a script that can + be used to invoke the Python interactive interpreter with the path set + based on the working set. This generated script can also be used to + run other scripts with the path set on the working set: + + >>> scripts = zc.buildout.easy_install.scripts( + ... ['demo'], ws, sys.executable, bin, interpreter='py') + + + >>> ls(bin) + - demo + - py + + >>> if sys.platform == 'win32': + ... scripts == [os.path.join(bin, 'demo.exe'), + ... os.path.join(bin, 'demo-script.py'), + ... os.path.join(bin, 'py.exe'), + ... os.path.join(bin, 'py-script.py')] + ... else: + ... scripts == [os.path.join(bin, 'demo'), + ... os.path.join(bin, 'py')] + True + + The py script simply runs the Python interactive interpreter with + the path set: + + >>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 + <BLANKLINE> + import sys + <BLANKLINE> + sys.path[0:0] = [ + '/sample-install/demo-0.3-pyN.N.egg', + '/sample-install/demoneeded-1.1-pyN.N.egg', + ] + <BLANKLINE> + _interactive = True + if len(sys.argv) > 1: + _options, _args = __import__("getopt").getopt(sys.argv[1:], 'ic:m:') + _interactive = False + for (_opt, _val) in _options: + if _opt == '-i': + _interactive = True + elif _opt == '-c': + exec _val + elif _opt == '-m': + sys.argv[1:] = _args + _args = [] + __import__("runpy").run_module( + _val, {}, "__main__", alter_sys=True) + <BLANKLINE> + if _args: + sys.argv[:] = _args + __file__ = _args[0] + del _options, _args + execfile(__file__) + <BLANKLINE> + if _interactive: + del _interactive + __import__("code").interact(banner="", local=globals()) + + If invoked with a script name and arguments, it will run that script, instead. + + >>> write('ascript', ''' + ... "demo doc" + ... print sys.argv + ... print (__name__, __file__, __doc__) + ... ''') + >>> print system(join(bin, 'py')+' ascript a b c'), + ['ascript', 'a', 'b', 'c'] + ('__main__', 'ascript', 'demo doc') + + For Python 2.5 and higher, you can also use the -m option to run a + module: + + >>> print system(join(bin, 'py')+' -m pdb'), + usage: pdb.py scriptfile [arg] ... + + >>> print system(join(bin, 'py')+' -m pdb what'), + Error: what does not exist + + An additional argument can be passed to define which scripts to install + and to provide script names. The argument is a dictionary mapping + original script names to new script names. + + >>> bin = tmpdir('bin2') + >>> scripts = zc.buildout.easy_install.scripts( + ... ['demo'], ws, sys.executable, bin, dict(demo='run')) + + >>> if sys.platform == 'win32': + ... scripts == [os.path.join(bin, 'run.exe'), + ... os.path.join(bin, 'run-script.py')] + ... else: + ... scripts == [os.path.join(bin, 'run')] + True + >>> ls(bin) + - run + + >>> print system(os.path.join(bin, 'run')), + 3 1 + + The ``scripts`` function: Including extra paths in scripts + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + We can pass a keyword argument, extra paths, to cause additional paths + to be included in the a generated script: + + >>> foo = tmpdir('foo') + >>> scripts = zc.buildout.easy_install.scripts( + ... ['demo'], ws, sys.executable, bin, dict(demo='run'), + ... extra_paths=[foo]) + + >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 + <BLANKLINE> + import sys + sys.path[0:0] = [ + '/sample-install/demo-0.3-py2.4.egg', + '/sample-install/demoneeded-1.1-py2.4.egg', + '/foo', + ] + <BLANKLINE> + import eggrecipedemo + <BLANKLINE> + if __name__ == '__main__': + eggrecipedemo.main() + + The ``scripts`` function: Providing script arguments + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + An "argument" keyword argument can be used to pass arguments to an + entry point. The value passed is a source string to be placed between the + parentheses in the call: + + >>> scripts = zc.buildout.easy_install.scripts( + ... ['demo'], ws, sys.executable, bin, dict(demo='run'), + ... arguments='1, 2') + + >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 + import sys + sys.path[0:0] = [ + '/sample-install/demo-0.3-py2.4.egg', + '/sample-install/demoneeded-1.1-py2.4.egg', + ] + <BLANKLINE> + import eggrecipedemo + <BLANKLINE> + if __name__ == '__main__': + eggrecipedemo.main(1, 2) + + The ``scripts`` function: Passing initialization code + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + You can also pass script initialization code: + + >>> scripts = zc.buildout.easy_install.scripts( + ... ['demo'], ws, sys.executable, bin, dict(demo='run'), + ... arguments='1, 2', + ... initialization='import os\nos.chdir("foo")') + + >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 + import sys + sys.path[0:0] = [ + '/sample-install/demo-0.3-py2.4.egg', + '/sample-install/demoneeded-1.1-py2.4.egg', + ] + <BLANKLINE> + import os + os.chdir("foo") + <BLANKLINE> + import eggrecipedemo + <BLANKLINE> + if __name__ == '__main__': + eggrecipedemo.main(1, 2) + + The ``scripts`` function: Relative paths + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Sometimes, you want to be able to move a buildout directory around and + have scripts still work without having to rebuild them. We can + control this using the relative_paths option to install. You need + to pass a common base directory of the scripts and eggs: + + >>> bo = tmpdir('bo') + >>> ba = tmpdir('ba') + >>> mkdir(bo, 'eggs') + >>> mkdir(bo, 'bin') + >>> mkdir(bo, 'other') + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], join(bo, 'eggs'), links=[link_server], + ... index=link_server+'index/') + + >>> scripts = zc.buildout.easy_install.scripts( + ... ['demo'], ws, sys.executable, join(bo, 'bin'), dict(demo='run'), + ... extra_paths=[ba, join(bo, 'bar')], + ... interpreter='py', + ... relative_paths=bo) + + >>> cat(bo, 'bin', 'run') # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 + <BLANKLINE> + import os + <BLANKLINE> + join = os.path.join + base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) + base = os.path.dirname(base) + <BLANKLINE> + import sys + sys.path[0:0] = [ + join(base, 'eggs/demo-0.3-pyN.N.egg'), + join(base, 'eggs/demoneeded-1.1-pyN.N.egg'), + '/ba', + join(base, 'bar'), + ] + <BLANKLINE> + import eggrecipedemo + <BLANKLINE> + if __name__ == '__main__': + eggrecipedemo.main() + + Note that the extra path we specified that was outside the directory + passed as relative_paths wasn't converted to a relative path. + + Of course, running the script works: + + >>> print system(join(bo, 'bin', 'run')), + 3 1 + + We specified an interpreter and its paths are adjusted too: + + >>> cat(bo, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 + <BLANKLINE> + import os + <BLANKLINE> + join = os.path.join + base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) + base = os.path.dirname(base) + <BLANKLINE> + import sys + <BLANKLINE> + sys.path[0:0] = [ + join(base, 'eggs/demo-0.3-pyN.N.egg'), + join(base, 'eggs/demoneeded-1.1-pyN.N.egg'), + '/ba', + join(base, 'bar'), + ] + <BLANKLINE> + _interactive = True + if len(sys.argv) > 1: + _options, _args = __import__("getopt").getopt(sys.argv[1:], 'ic:m:') + _interactive = False + for (_opt, _val) in _options: + if _opt == '-i': + _interactive = True + elif _opt == '-c': + exec _val + elif _opt == '-m': + sys.argv[1:] = _args + _args = [] + __import__("runpy").run_module( + _val, {}, "__main__", alter_sys=True) + <BLANKLINE> + if _args: + sys.argv[:] = _args + __file__ = _args[0] + del _options, _args + execfile(__file__) + <BLANKLINE> + if _interactive: + del _interactive + __import__("code").interact(banner="", local=globals()) + + The ``sitepackage_safe_scripts`` function + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + The newer function for creating scripts is ``sitepackage_safe_scripts``. + It has the same basic functionality as the ``scripts`` function: it can + create scripts to run arbitrary entry points, and to run a Python + interpreter. The following are the differences from a user's + perspective. + + - It can be used safely with a Python that has packages installed itself, + such as a system-installed Python. + + - In contrast to the interpreter generated by the ``scripts`` method, which + supports only a small subset of the usual Python executable's options, + the interpreter generated by ``sitepackage_safe_scripts`` supports all + of them. This makes it possible to use as full Python replacement for + scripts that need the distributions specified in your buildout. + + - Both the interpreter and the entry point scripts allow you to include the + site packages, and/or the sitecustomize, of the Python executable, if + desired. + + It works by creating site.py and sitecustomize.py files that set up the + desired paths and initialization. These must be placed within an otherwise + empty directory. Typically this is in a recipe's parts directory. + + Here's the simplest example, building an interpreter script. + + >>> interpreter_dir = tmpdir('interpreter') + >>> interpreter_parts_dir = os.path.join( + ... interpreter_dir, 'parts', 'interpreter') + >>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin') + >>> mkdir(interpreter_bin_dir) + >>> mkdir(interpreter_dir, 'eggs') + >>> mkdir(interpreter_dir, 'parts') + >>> mkdir(interpreter_parts_dir) + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server], + ... index=link_server+'index/') + >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts( + ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir, + ... interpreter='py') + + Depending on whether the machine being used is running Windows or not, this + produces either three or four files. In both cases, we have site.py and + sitecustomize.py generated in the parts/interpreter directory. For Windows, + we have py.exe and py-script.py; for other operating systems, we have py. + + >>> sitecustomize_path = os.path.join( + ... interpreter_parts_dir, 'sitecustomize.py') + >>> site_path = os.path.join(interpreter_parts_dir, 'site.py') + >>> interpreter_path = os.path.join(interpreter_bin_dir, 'py') + >>> if sys.platform == 'win32': + ... py_path = os.path.join(interpreter_bin_dir, 'py-script.py') + ... expected = [sitecustomize_path, + ... site_path, + ... os.path.join(interpreter_bin_dir, 'py.exe'), + ... py_path] + ... else: + ... py_path = interpreter_path + ... expected = [sitecustomize_path, site_path, py_path] + ... + >>> assert generated == expected, repr((generated, expected)) + + We didn't ask for any initialization, and we didn't ask to use the underlying + sitecustomization, so sitecustomize.py is empty. + + >>> cat(sitecustomize_path) + + The interpreter script is simple. It puts the directory with the + site.py and sitecustomize.py on the PYTHONPATH and (re)starts Python. + + >>> cat(py_path) + #!/usr/bin/python -S + import os + import sys + <BLANKLINE> + argv = [sys.executable] + sys.argv[1:] + environ = os.environ.copy() + path = '/interpreter/parts/interpreter' + if environ.get('PYTHONPATH'): + path = os.pathsep.join([path, environ['PYTHONPATH']]) + environ['PYTHONPATH'] = path + os.execve(sys.executable, argv, environ) + + The site.py file is a modified version of the underlying Python's site.py. + The most important modification is that it has a different version of the + addsitepackages function. It sets up the Python path, similarly to the + behavior of the function it replaces. The following shows the part that + buildout inserts, in the simplest case. + + >>> sys.stdout.write('#\n'); cat(site_path) + ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + #... + def addsitepackages(known_paths): + """Add site packages, as determined by zc.buildout. + <BLANKLINE> + See original_addsitepackages, below, for the original version.""" + buildout_paths = [ + '/interpreter/eggs/demo-0.3-pyN.N.egg', + '/interpreter/eggs/demoneeded-1.1-pyN.N.egg' + ] + for path in buildout_paths: + sitedir, sitedircase = makepath(path) + if not sitedircase in known_paths and os.path.exists(sitedir): + sys.path.append(sitedir) + known_paths.add(sitedircase) + return known_paths + <BLANKLINE> + def original_addsitepackages(known_paths):... + + Here are some examples of the interpreter in use. + + >>> print call_py(interpreter_path, "print 16+26") + 42 + <BLANKLINE> + >>> res = call_py(interpreter_path, "import sys; print sys.path") + >>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + ['', + '/interpreter/parts/interpreter', + ..., + '/interpreter/eggs/demo-0.3-pyN.N.egg', + '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'] + <BLANKLINE> + >>> clean_paths = eval(res.strip()) # This is used later for comparison. + + If you provide initialization, it goes in sitecustomize.py. + + >>> def reset_interpreter(): + ... # This is necessary because, in our tests, the timestamps of the + ... # .pyc files are not outdated when we want them to be. + ... rmdir(interpreter_bin_dir) + ... mkdir(interpreter_bin_dir) + ... rmdir(interpreter_parts_dir) + ... mkdir(interpreter_parts_dir) + ... + >>> reset_interpreter() + + >>> initialization_string = """\ + ... import os + ... os.environ['FOO'] = 'bar baz bing shazam'""" + >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts( + ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir, + ... interpreter='py', initialization=initialization_string) + >>> cat(sitecustomize_path) + import os + os.environ['FOO'] = 'bar baz bing shazam' + >>> print call_py(interpreter_path, "import os; print os.environ['FOO']") + bar baz bing shazam + <BLANKLINE> + + If you use relative paths, this affects the interpreter and site.py. (This is + again the UNIX version; the Windows version uses subprocess instead of + os.execve.) + + >>> reset_interpreter() + >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts( + ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir, + ... interpreter='py', relative_paths=interpreter_dir) + >>> cat(py_path) + #!/usr/bin/python -S + import os + import sys + <BLANKLINE> + join = os.path.join + base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) + base = os.path.dirname(base) + <BLANKLINE> + argv = [sys.executable] + sys.argv[1:] + environ = os.environ.copy() + path = join(base, 'parts/interpreter') + if environ.get('PYTHONPATH'): + path = os.pathsep.join([path, environ['PYTHONPATH']]) + environ['PYTHONPATH'] = path + os.execve(sys.executable, argv, environ) + + For site.py, we again show only the pertinent parts. Notice that the egg + paths join a base to a path, as with the use of this argument in the + ``scripts`` function. + + >>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS + #... + def addsitepackages(known_paths): + """Add site packages, as determined by zc.buildout. + <BLANKLINE> + See original_addsitepackages, below, for the original version.""" + join = os.path.join + base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) + base = os.path.dirname(base) + base = os.path.dirname(base) + buildout_paths = [ + join(base, 'eggs/demo-0.3-pyN.N.egg'), + join(base, 'eggs/demoneeded-1.1-pyN.N.egg') + ]... + + The paths resolve in practice as you would expect. + + >>> print call_py(interpreter_path, + ... "import sys, pprint; pprint.pprint(sys.path)") + ... # doctest: +ELLIPSIS + ['', + '/interpreter/parts/interpreter', + ..., + '/interpreter/eggs/demo-0.3-pyN.N.egg', + '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'] + <BLANKLINE> + + The ``extra_paths`` argument affects the path in site.py. Notice that + /interpreter/other is added after the eggs. + + >>> reset_interpreter() + >>> mkdir(interpreter_dir, 'other') + >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts( + ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir, + ... interpreter='py', extra_paths=[join(interpreter_dir, 'other')]) + >>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS + #... + def addsitepackages(known_paths): + """Add site packages, as determined by zc.buildout. + <BLANKLINE> + See original_addsitepackages, below, for the original version.""" + buildout_paths = [ + '/interpreter/eggs/demo-0.3-pyN.N.egg', + '/interpreter/eggs/demoneeded-1.1-pyN.N.egg', + '/interpreter/other' + ]... + + >>> print call_py(interpreter_path, + ... "import sys, pprint; pprint.pprint(sys.path)") + ... # doctest: +ELLIPSIS + ['', + '/interpreter/parts/interpreter', + ..., + '/interpreter/eggs/demo-0.3-pyN.N.egg', + '/interpreter/eggs/demoneeded-1.1-pyN.N.egg', + '/interpreter/other'] + <BLANKLINE> + + The ``sitepackage_safe_scripts`` function: using site-packages + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + The ``sitepackage_safe_scripts`` function supports including site + packages. This has some advantages and some serious dangers. + + A typical reason to include site-packages is that it is easier to + install one or more dependencies in your Python than it is with + buildout. Some packages, such as lxml or Python PostgreSQL integration, + have dependencies that can be much easier to build and/or install using + other mechanisms, such as your operating system's package manager. By + installing some core packages into your Python's site-packages, this can + significantly simplify some application installations. + + However, doing this has a significant danger. One of the primary goals + of buildout is to provide repeatability. Some packages (one of the + better known Python openid packages, for instance) change their behavior + depending on what packages are available. If Python curl bindings are + available, these may be preferred by the library. If a certain XML + package is installed, it may be preferred by the library. These hidden + choices may cause small or large behavior differences. The fact that + they can be rarely encountered can actually make it worse: you forget + that this might be a problem, and debugging the differences can be + difficult. If you allow site-packages to be included in your buildout, + and the Python you use is not managed precisely by your application (for + instance, it is a system Python), you open yourself up to these + possibilities. Don't be unaware of the dangers. + + That explained, let's see how it works. If you don't use namespace packages, + this is very straightforward. + + >>> reset_interpreter() + >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts( + ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir, + ... interpreter='py', include_site_packages=True) + >>> sys.stdout.write('#\n'); cat(site_path) + ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + #... + def addsitepackages(known_paths): + """Add site packages, as determined by zc.buildout. + <BLANKLINE> + See original_addsitepackages, below, for the original version.""" + setuptools_path = None + buildout_paths = [ + '/interpreter/eggs/demo-0.3-pyN.N.egg', + '/interpreter/eggs/demoneeded-1.1-pyN.N.egg' + ] + for path in buildout_paths: + sitedir, sitedircase = makepath(path) + if not sitedircase in known_paths and os.path.exists(sitedir): + sys.path.append(sitedir) + known_paths.add(sitedircase) + sys.__egginsert = len(buildout_paths) # Support distribute. + original_paths = [ + ... + ] + for path in original_paths: + if path == setuptools_path or path not in known_paths: + addsitedir(path, known_paths) + return known_paths + <BLANKLINE> + def original_addsitepackages(known_paths):... + + It simply adds the original paths using addsitedir after the code to add the + buildout paths. + + Here's an example of the new script in use. Other documents and tests in + this package give the feature a more thorough workout, but this should + give you an idea of the feature. + + >>> res = call_py(interpreter_path, "import sys; print sys.path") + >>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + ['', + '/interpreter/parts/interpreter', + ..., + '/interpreter/eggs/demo-0.3-py2.4.egg', + '/interpreter/eggs/demoneeded-1.1-py2.4.egg', + ...] + <BLANKLINE> + + The clean_paths gathered earlier is a subset of this full list of paths. + + >>> full_paths = eval(res.strip()) + >>> len(clean_paths) < len(full_paths) + True + >>> set(os.path.normpath(p) for p in clean_paths).issubset( + ... os.path.normpath(p) for p in full_paths) + True + + Unfortunately, because of how setuptools namespace packages are implemented + differently for operating system packages (debs or rpms) as opposed to + standard setuptools installation, there's a slightly trickier dance if you + use them. To show this we'll needs some extra eggs that use namespaces. + We'll use the ``tellmy.fortune`` package, which we'll need to make an initial + call to another text fixture to create. + + >>> from zc.buildout.tests import create_sample_namespace_eggs + >>> namespace_eggs = tmpdir('namespace_eggs') + >>> create_sample_namespace_eggs(namespace_eggs) + + >>> reset_interpreter() + >>> ws = zc.buildout.easy_install.install( + ... ['demo', 'tellmy.fortune'], join(interpreter_dir, 'eggs'), + ... links=[link_server, namespace_eggs], index=link_server+'index/') + >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts( + ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir, + ... interpreter='py', include_site_packages=True) + >>> sys.stdout.write('#\n'); cat(site_path) + ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + #... + def addsitepackages(known_paths): + """Add site packages, as determined by zc.buildout. + <BLANKLINE> + See original_addsitepackages, below, for the original version.""" + setuptools_path = '...setuptools...' + sys.path.append(setuptools_path) + known_paths.add(os.path.normcase(setuptools_path)) + import pkg_resources + buildout_paths = [ + '/interpreter/eggs/demo-0.3-pyN.N.egg', + '/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg', + '...setuptools...', + '/interpreter/eggs/demoneeded-1.1-pyN.N.egg' + ] + for path in buildout_paths: + sitedir, sitedircase = makepath(path) + if not sitedircase in known_paths and os.path.exists(sitedir): + sys.path.append(sitedir) + known_paths.add(sitedircase) + pkg_resources.working_set.add_entry(sitedir) + sys.__egginsert = len(buildout_paths) # Support distribute. + original_paths = [ + ... + ] + for path in original_paths: + if path == setuptools_path or path not in known_paths: + addsitedir(path, known_paths) + return known_paths + <BLANKLINE> + def original_addsitepackages(known_paths):... + + >>> print call_py(interpreter_path, "import sys; print sys.path") + ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + ['', + '/interpreter/parts/interpreter', + ..., + '...setuptools...', + '/interpreter/eggs/demo-0.3-pyN.N.egg', + '/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg', + '/interpreter/eggs/demoneeded-1.1-pyN.N.egg', + ...] + + As you can see, the script now first imports pkg_resources. Then we + need to process egg files specially to look for namespace packages there + *before* we process process lines in .pth files that use the "import" + feature--lines that might be part of the setuptools namespace package + implementation for system packages, as mentioned above, and that must + come after processing egg namespaces. + + The most complex that this function gets is if you use namespace packages, + include site-packages, and use relative paths. For completeness, we'll look + at that result. + + >>> reset_interpreter() + >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts( + ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir, + ... interpreter='py', include_site_packages=True, + ... relative_paths=interpreter_dir) + >>> sys.stdout.write('#\n'); cat(site_path) + ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + #... + def addsitepackages(known_paths): + """Add site packages, as determined by zc.buildout. + <BLANKLINE> + See original_addsitepackages, below, for the original version.""" + join = os.path.join + base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) + base = os.path.dirname(base) + base = os.path.dirname(base) + setuptools_path = '...setuptools...' + sys.path.append(setuptools_path) + known_paths.add(os.path.normcase(setuptools_path)) + import pkg_resources + buildout_paths = [ + join(base, 'eggs/demo-0.3-pyN.N.egg'), + join(base, 'eggs/tellmy.fortune-1.0-pyN.N.egg'), + '...setuptools...', + join(base, 'eggs/demoneeded-1.1-pyN.N.egg') + ] + for path in buildout_paths: + sitedir, sitedircase = makepath(path) + if not sitedircase in known_paths and os.path.exists(sitedir): + sys.path.append(sitedir) + known_paths.add(sitedircase) + pkg_resources.working_set.add_entry(sitedir) + sys.__egginsert = len(buildout_paths) # Support distribute. + original_paths = [ + ... + ] + for path in original_paths: + if path == setuptools_path or path not in known_paths: + addsitedir(path, known_paths) + return known_paths + <BLANKLINE> + def original_addsitepackages(known_paths):... + + >>> print call_py(interpreter_path, "import sys; print sys.path") + ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE + ['', + '/interpreter/parts/interpreter', + ..., + '...setuptools...', + '/interpreter/eggs/demo-0.3-pyN.N.egg', + '/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg', + '/interpreter/eggs/demoneeded-1.1-pyN.N.egg', + ...] + + The ``exec_sitecustomize`` argument does the same thing for the + sitecustomize module--it allows you to include the code from the + sitecustomize module in the underlying Python if you set the argument to + True. The z3c.recipe.scripts package sets up the full environment necessary + to demonstrate this piece. + + The ``sitepackage_safe_scripts`` function: writing scripts for entry points + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + All of the examples so far for this function have been creating + interpreters. The function can also write scripts for entry + points. They are almost identical to the scripts that we saw for the + ``scripts`` function except that they ``import site`` after setting the + sys.path to include our custom site.py and sitecustomize.py files. These + files then initialize the Python environment as we have already seen. Let's + see a simple example. + + >>> reset_interpreter() + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server], + ... index=link_server+'index/') + >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts( + ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir, + ... reqs=['demo']) + + As before, in Windows, 2 files are generated for each script. A script + file, ending in '-script.py', and an exe file that allows the script + to be invoked directly without having to specify the Python + interpreter and without having to provide a '.py' suffix. This is in addition + to the site.py and sitecustomize.py files that are generated as with our + interpreter examples above. + + >>> if sys.platform == 'win32': + ... demo_path = os.path.join(interpreter_bin_dir, 'demo-script.py') + ... expected = [sitecustomize_path, + ... site_path, + ... os.path.join(interpreter_bin_dir, 'demo.exe'), + ... demo_path] + ... else: + ... demo_path = os.path.join(interpreter_bin_dir, 'demo') + ... expected = [sitecustomize_path, site_path, demo_path] + ... + >>> assert generated == expected, repr((generated, expected)) + + The demo script runs the entry point defined in the demo egg: + + >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 -S + <BLANKLINE> + import sys + sys.path[0:0] = [ + '/interpreter/parts/interpreter', + ] + <BLANKLINE> + <BLANKLINE> + import os + path = sys.path[0] + if os.environ.get('PYTHONPATH'): + path = os.pathsep.join([path, os.environ['PYTHONPATH']]) + os.environ['BUILDOUT_ORIGINAL_PYTHONPATH'] = os.environ.get('PYTHONPATH', '') + os.environ['PYTHONPATH'] = path + import site # imports custom buildout-generated site.py + <BLANKLINE> + import eggrecipedemo + <BLANKLINE> + if __name__ == '__main__': + eggrecipedemo.main() + + >>> demo_call = join(interpreter_bin_dir, 'demo') + >>> if sys.platform == 'win32': + ... demo_call = '"%s"' % demo_call + >>> print system(demo_call) + 3 1 + <BLANKLINE> + + There are a few differences from the ``scripts`` function. First, the + ``reqs`` argument (an iterable of string requirements or entry point + tuples) is a keyword argument here. We see that in the example above. + Second, the ``arguments`` argument is now named ``script_arguments`` to + try and clarify that it does not affect interpreters. While the + ``initialization`` argument continues to affect both the interpreters + and the entry point scripts, if you have initialization that is only + pertinent to the entry point scripts, you can use the + ``script_initialization`` argument. + + Let's see ``script_arguments`` and ``script_initialization`` in action. + + >>> reset_interpreter() + >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts( + ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir, + ... reqs=['demo'], script_arguments='1, 2', + ... script_initialization='import os\nos.chdir("foo")') + + >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE + #!/usr/local/bin/python2.4 -S + import sys + sys.path[0:0] = [ + '/interpreter/parts/interpreter', + ] + <BLANKLINE> + import os + path = sys.path[0] + if os.environ.get('PYTHONPATH'): + path = os.pathsep.join([path, os.environ['PYTHONPATH']]) + os.environ['BUILDOUT_ORIGINAL_PYTHONPATH'] = os.environ.get('PYTHONPATH', '') + os.environ['PYTHONPATH'] = path + import site # imports custom buildout-generated site.py + import os + os.chdir("foo") + <BLANKLINE> + import eggrecipedemo + <BLANKLINE> + if __name__ == '__main__': + eggrecipedemo.main(1, 2) + + Handling custom build options for extensions provided in source distributions + ----------------------------------------------------------------------------- + + Sometimes, we need to control how extension modules are built. The + build function provides this level of control. It takes a single + package specification, downloads a source distribution, and builds it + with specified custom build options. + + The build function takes 3 positional arguments: + + spec + A package specification for a source distribution + + dest + A destination directory + + build_ext + A dictionary of options to be passed to the distutils build_ext + command when building extensions. + + It supports a number of optional keyword arguments: + + links + a sequence of URLs, file names, or directories to look for + links to distributions, + + index + The URL of an index server, or almost any other valid URL. :) + + If not specified, the Python Package Index, + http://pypi.python.org/simple/, is used. You can specify an + alternate index with this option. If you use the links option and + if the links point to the needed distributions, then the index can + be anything and will be largely ignored. In the examples, here, + we'll just point to an empty directory on our link server. This + will make our examples run a little bit faster. + + executable + A path to a Python executable. Distributions will be installed + using this executable and will be for the matching Python version. + + path + A list of additional directories to search for locally-installed + distributions. + + newest + A boolean value indicating whether to search for new distributions + when already-installed distributions meet the requirement. When + this is true, the default, and when the destination directory is + not None, then the install function will search for the newest + distributions that satisfy the requirements. + + versions + A dictionary mapping project names to version numbers to be used + when selecting distributions. This can be used to specify a set of + distribution versions independent of other requirements. + + + Our link server included a source distribution that includes a simple + extension, extdemo.c:: + + #include <Python.h> + #include <extdemo.h> + + static PyMethodDef methods[] = {}; + + PyMODINIT_FUNC + initextdemo(void) + { + PyObject *m; + m = Py_InitModule3("extdemo", methods, ""); + #ifdef TWO + PyModule_AddObject(m, "val", PyInt_FromLong(2)); + #else + PyModule_AddObject(m, "val", PyInt_FromLong(EXTDEMO)); + #endif + } + + The extension depends on a system-dependent include file, extdemo.h, + that defines a constant, EXTDEMO, that is exposed by the extension. + + We'll add an include directory to our sample buildout and add the + needed include file to it: + + >>> mkdir('include') + >>> write('include', 'extdemo.h', + ... """ + ... #define EXTDEMO 42 + ... """) + + Now, we can use the build function to create an egg from the source + distribution: + + >>> zc.buildout.easy_install.build( + ... 'extdemo', dest, + ... {'include-dirs': os.path.join(sample_buildout, 'include')}, + ... links=[link_server], index=link_server+'index/') + ['/sample-install/extdemo-1.4-py2.4-unix-i686.egg'] + + The function returns the list of eggs + + Now if we look in our destination directory, we see we have an extdemo egg: + + >>> ls(dest) + - demo-0.2-py2.4.egg + d demo-0.3-py2.4.egg + - demoneeded-1.0-py2.4.egg + d demoneeded-1.1-py2.4.egg + d extdemo-1.4-py2.4-unix-i686.egg + + Let's update our link server with a new version of extdemo: + + >>> update_extdemo() + >>> print get(link_server), + <html><body> + <a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br> + <a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br> + <a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br> + <a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br> + <a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br> + <a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br> + <a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br> + <a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br> + <a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br> + <a href="extdemo-1.5.zip">extdemo-1.5.zip</a><br> + <a href="index/">index/</a><br> + <a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br> + </body></html> + + The easy_install caches information about servers to reduce network + access. To see the update, we have to call the clear_index_cache + function to clear the index cache: + + >>> zc.buildout.easy_install.clear_index_cache() + + If we run build with newest set to False, we won't get an update: + + >>> zc.buildout.easy_install.build( + ... 'extdemo', dest, + ... {'include-dirs': os.path.join(sample_buildout, 'include')}, + ... links=[link_server], index=link_server+'index/', + ... newest=False) + ['/sample-install/extdemo-1.4-py2.4-linux-i686.egg'] + + >>> ls(dest) + - demo-0.2-py2.4.egg + d demo-0.3-py2.4.egg + - demoneeded-1.0-py2.4.egg + d demoneeded-1.1-py2.4.egg + d extdemo-1.4-py2.4-unix-i686.egg + + But if we run it with the default True setting for newest, then we'll + get an updated egg: + + >>> zc.buildout.easy_install.build( + ... 'extdemo', dest, + ... {'include-dirs': os.path.join(sample_buildout, 'include')}, + ... links=[link_server], index=link_server+'index/') + ['/sample-install/extdemo-1.5-py2.4-unix-i686.egg'] + + >>> ls(dest) + - demo-0.2-py2.4.egg + d demo-0.3-py2.4.egg + - demoneeded-1.0-py2.4.egg + d demoneeded-1.1-py2.4.egg + d extdemo-1.4-py2.4-unix-i686.egg + d extdemo-1.5-py2.4-unix-i686.egg + + The versions option also influences the versions used. For example, + if we specify a version for extdemo, then that will be used, even + though it isn't the newest. Let's clean out the destination directory + first: + + >>> import os + >>> for name in os.listdir(dest): + ... remove(dest, name) + + >>> zc.buildout.easy_install.build( + ... 'extdemo', dest, + ... {'include-dirs': os.path.join(sample_buildout, 'include')}, + ... links=[link_server], index=link_server+'index/', + ... versions=dict(extdemo='1.4')) + ['/sample-install/extdemo-1.4-py2.4-unix-i686.egg'] + + >>> ls(dest) + d extdemo-1.4-py2.4-unix-i686.egg + + Handling custom build options for extensions in develop eggs + ------------------------------------------------------------ + + The develop function is similar to the build function, except that, + rather than building an egg from a source directory containing a + setup.py script. + + The develop function takes 2 positional arguments: + + setup + The path to a setup script, typically named "setup.py", or a + directory containing a setup.py script. + + dest + The directory to install the egg link to + + It supports some optional keyword argument: + + build_ext + A dictionary of options to be passed to the distutils build_ext + command when building extensions. + + executable + A path to a Python executable. Distributions will be installed + using this executable and will be for the matching Python version. + + We have a local directory containing the extdemo source: + + >>> ls(extdemo) + - MANIFEST + - MANIFEST.in + - README + - extdemo.c + - setup.py + + Now, we can use the develop function to create a develop egg from the source + distribution: + + >>> zc.buildout.easy_install.develop( + ... extdemo, dest, + ... {'include-dirs': os.path.join(sample_buildout, 'include')}) + '/sample-install/extdemo.egg-link' + + The name of the egg link created is returned. + + Now if we look in our destination directory, we see we have an extdemo + egg link: + + >>> ls(dest) + d extdemo-1.4-py2.4-unix-i686.egg + - extdemo.egg-link + + And that the source directory contains the compiled extension: + + >>> ls(extdemo) + - MANIFEST + - MANIFEST.in + - README + d build + - extdemo.c + d extdemo.egg-info + - extdemo.so + - setup.py + + Download cache + -------------- + + Normally, when distributions are installed, if any processing is + needed, they are downloaded from the internet to a temporary directory + and then installed from there. A download cache can be used to avoid + the download step. This can be useful to reduce network access and to + create source distributions of an entire buildout. + + A download cache is specified by calling the download_cache + function. The function always returns the previous setting. If no + argument is passed, then the setting is unchanged. If an argument is + passed, the download cache is set to the given path, which must point + to an existing directory. Passing None clears the cache setting. + + To see this work, we'll create a directory and set it as the cache + directory: + + >>> cache = tmpdir('cache') + >>> zc.buildout.easy_install.download_cache(cache) + + We'll recreate our destination directory: + + >>> remove(dest) + >>> dest = tmpdir('sample-install') + + We'd like to see what is being fetched from the server, so we'll + enable server logging: + + >>> get(link_server+'enable_server_logging') + GET 200 /enable_server_logging + '' + + Now, if we install demo, and extdemo: + + >>> ws = zc.buildout.easy_install.install( + ... ['demo==0.2'], dest, + ... links=[link_server], index=link_server+'index/', + ... always_unzip=True) + GET 200 / + GET 404 /index/demo/ + GET 200 /index/ + GET 200 /demo-0.2-py2.4.egg + GET 404 /index/demoneeded/ + GET 200 /demoneeded-1.1.zip + + >>> zc.buildout.easy_install.build( + ... 'extdemo', dest, + ... {'include-dirs': os.path.join(sample_buildout, 'include')}, + ... links=[link_server], index=link_server+'index/') + GET 404 /index/extdemo/ + GET 200 /extdemo-1.5.zip + ['/sample-install/extdemo-1.5-py2.4-linux-i686.egg'] + + Not only will we get eggs in our destination directory: + + >>> ls(dest) + d demo-0.2-py2.4.egg + d demoneeded-1.1-py2.4.egg + d extdemo-1.5-py2.4-linux-i686.egg + + But we'll get distributions in the cache directory: + + >>> ls(cache) + - demo-0.2-py2.4.egg + - demoneeded-1.1.zip + - extdemo-1.5.zip + + The cache directory contains uninstalled distributions, such as zipped + eggs or source distributions. + + Let's recreate our destination directory and clear the index cache: + + >>> remove(dest) + >>> dest = tmpdir('sample-install') + >>> zc.buildout.easy_install.clear_index_cache() + + Now when we install the distributions: + + >>> ws = zc.buildout.easy_install.install( + ... ['demo==0.2'], dest, + ... links=[link_server], index=link_server+'index/', + ... always_unzip=True) + GET 200 / + GET 404 /index/demo/ + GET 200 /index/ + GET 404 /index/demoneeded/ + + >>> zc.buildout.easy_install.build( + ... 'extdemo', dest, + ... {'include-dirs': os.path.join(sample_buildout, 'include')}, + ... links=[link_server], index=link_server+'index/') + GET 404 /index/extdemo/ + ['/sample-install/extdemo-1.5-py2.4-linux-i686.egg'] + + >>> ls(dest) + d demo-0.2-py2.4.egg + d demoneeded-1.1-py2.4.egg + d extdemo-1.5-py2.4-linux-i686.egg + + Note that we didn't download the distributions from the link server. + + If we remove the restriction on demo, we'll download a newer version + from the link server: + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, + ... links=[link_server], index=link_server+'index/', + ... always_unzip=True) + GET 200 /demo-0.3-py2.4.egg + + Normally, the download cache is the preferred source of downloads, but + not the only one. + + Installing solely from a download cache + --------------------------------------- + + A download cache can be used as the basis of application source + releases. In an application source release, we want to distribute an + application that can be built without making any network accesses. In + this case, we distribute a download cache and tell the easy_install + module to install from the download cache only, without making network + accesses. The install_from_cache function can be used to signal that + packages should be installed only from the download cache. The + function always returns the previous setting. Calling it with no + arguments returns the current setting without changing it: + + >>> zc.buildout.easy_install.install_from_cache() + False + + Calling it with a boolean value changes the setting and returns the + previous setting: + + >>> zc.buildout.easy_install.install_from_cache(True) + False + + Let's remove demo-0.3-py2.4.egg from the cache, clear the index cache, + recreate the destination directory, and reinstall demo: + + >>> for f in os.listdir(cache): + ... if f.startswith('demo-0.3-'): + ... remove(cache, f) + + >>> zc.buildout.easy_install.clear_index_cache() + >>> remove(dest) + >>> dest = tmpdir('sample-install') + + >>> ws = zc.buildout.easy_install.install( + ... ['demo'], dest, + ... links=[link_server], index=link_server+'index/', + ... always_unzip=True) + + >>> ls(dest) + d demo-0.2-py2.4.egg + d demoneeded-1.1-py2.4.egg + + This time, we didn't download from or even query the link server. + + .. Disable the download cache: + + >>> zc.buildout.easy_install.download_cache(None) + '/cache' + + >>> zc.buildout.easy_install.install_from_cache(False) + True + + Distribute Support + ================== + + Distribute is a drop-in replacement for Setuptools. + + zc.buildout is now compatible with Distribute 0.6. To use Distribute in your + buildout, you need use the ``--distribute`` option of the ``bootstrap.py`` + script:: + + $ python bootstrap.py --distribute + + This will download and install the latest Distribute 0.6 release in the + ``eggs`` directory, and use this version for the scripts that are created + in ``bin``. + + Notice that if you have a shared eggs directory, a buildout that uses + Distribute will not interfer with other buildouts that are based on Setuptools + and that are sharing the same eggs directory. + + Form more information about the Distribute project, see: + http://python-distribute.org + + + + Change History + ************** + + 1.5.2 (2010-10-11) + ================== + + - changed metadata 'url' to pypi.python.org in order to solve + a temporary outage of buildout.org + + - IMPORTANT: For better backwards compatibility with the pre-1.5 line, + this release has two big changes from 1.5.0 and 1.5.1. + + - Buildout defaults to including site packages. + + - Buildout loads recipes and extensions with the same constraints to + site-packages that it builds eggs, instead of never allowing access + to site-packages. + + This means that the default configuration should better support + pre-existing use of system Python in recipes or builds. + + - To make it easier to detect the fact that buildout has set the PYTHONPATH, + BUILDOUT_ORIGINAL_PYTHONPATH is always set in the environment, even if + PYTHONPATH was not originally set. BUILDOUT_ORIGINAL_PYTHONPATH will + be an empty string if PYTHONPATH was not set. + + 1.5.1 (2010-08-29) + ================== + + New features: + + - Scripts store the old PYTHONPATH in BUILDOUT_ORIGINAL_PYTHONPATH if it + existed, and store nothing in the value if it did not exist. This allows + code that does not want subprocesses to have the system-Python-protected + site.py to set the environment of the subprocess as it was originally. + + Bugs fixed: + + - https://bugs.launchpad.net/bugs/623590 : If include-site-packages were + true and versions were not set explicitly, system eggs were preferred + over newer released eggs. Fixed. + + 1.5.0 (2010-08-23) + ================== + + New features: + + - zc.buildout supports Python 2.7. + + - By default, Buildout and the bootstrap script now prefer final versions of + Buildout, recipes, and extensions. This can be changed by using the + --accept-buildout-test-releases flag (or -t for short) when calling + bootstrap. This will hopefully allow beta releases of these items to + be more easily and safely made in the future. + + NOTE: dependencies of your own software are not affected by this new + behavior. Buildout continues to choose the newest available versions + of your dependencies regardless of whether they are final releases. To + prevent this, use the pre-existing switch ``prefer-final = true`` in + the [buildout] section of your configuration file (see + http://pypi.python.org/pypi/zc.buildout#preferring-final-releases) or + pin your versions using a versions section (see + http://pypi.python.org/pypi/zc.buildout#repeatable-buildouts-controlling-eggs-used). + + Bugs fixed: + + - You can now again use virtualenv with Buildout. The new features to let + buildout be used with a system Python are disabled in this configuration, + and the previous script generation behavior (1.4.3) is used, even if + the new function ``zc.buildout.easy_install.sitepackage_safe_scripts`` + is used. + + 1.5.0b2 (2010-04-29) + ==================== + + This was a re-release of 1.4.3 in order to keep 1.5.0b1 release from hurting + workflows that combined virtualenv with zc.buildout. + + 1.5.0b1 (2010-04-29) + ==================== + + New Features: + + - Added buildout:socket-timout option so that socket timeout can be configured + both from command line and from config files. (gotcha) + + - Buildout can be safely used with a system Python (or any Python with code + in site-packages), as long as you use (1) A fresh checkout, (2) the + new bootstrap.py, and (3) recipes that use the new + ``zc.buildout.easy_install.sitepackage_safe_scripts`` function to generate + scripts and interpreters. Many recipes will need to be updated to use + this new function. The scripts and interpreters generated by + ``zc.recipe.egg`` will continue to use the older function, not safe + with system Pythons. Use the ``z3c.recipe.scripts`` as a replacement. + + zc.recipe.egg is still a fully supported, and simpler, way of + generating scripts and interpreters if you are using a "clean" Python, + without code installed in site-packages. It keeps its previous behavior in + order to provide backwards compatibility. + + The z3c.recipe.scripts recipe allows you to control how you use the + code in site-packages. You can exclude it entirely (preferred); allow + eggs in it to fulfill package dependencies declared in setup.py and + buildout configuration; allow it to be available but not used to + fulfill dependencies declared in setup.py or buildout configuration; + or only allow certain eggs in site-packages to fulfill dependencies. + + - Added new function, ``zc.buildout.easy_install.sitepackage_safe_scripts``, + to generate scripts and interpreter. It produces a full-featured + interpreter (all command-line options supported) and the ability to + safely let scripts include site packages, such as with a system + Python. The ``z3c.recipe.scripts`` recipe uses this new function. + + - Improve bootstrap. + + * New options let you specify where to find ez_setup.py and where to find + a download cache. These options can keep bootstrap from going over the + network. + + * Another new option lets you specify where to put generated eggs. + + * The buildout script generated by bootstrap honors more of the settings + in the designated configuration file (e.g., buildout.cfg). + + * Correctly handle systems where pkg_resources is present but the rest of + setuptools is missing (like Ubuntu installs). + https://bugs.launchpad.net/zc.buildout/+bug/410528 + + - You can develop zc.buildout using Distribute instead of Setuptools. Use + the --distribute option on the dev.py script. (Releases should be tested + with both Distribute and Setuptools.) The tests for zc.buildout pass + with Setuptools and Python 2.4, 2.5, 2.6, and 2.7; and with Distribute and + Python 2.5, 2.6, and 2.7. Using zc.buildout with Distribute and Python 2.4 + is not recommended. + + - The ``distribute-version`` now works in the [buildout] section, mirroring + the ``setuptools-version`` option (this is for consistency; using the + general-purpose ``versions`` option is preferred). + + Bugs fixed: + + - Using Distribute with the ``allow-picked-versions = false`` buildout + option no longer causes an error. + + - The handling and documenting of default buildout options was normalized. + This means, among other things, that ``bin/buildout -vv`` and + ``bin/buildout annotate`` correctly list more of the options. + + - Installing a namespace package using a Python that already has a package + in the same namespace (e.g., in the Python's site-packages) failed in + some cases. It is now handled correctly. + + - Another variation of this error showed itself when at least two + dependencies were in a shared location like site-packages, and the + first one met the "versions" setting. The first dependency would be + added, but subsequent dependencies from the same location (e.g., + site-packages) would use the version of the package found in the + shared location, ignoring the version setting. This is also now + handled correctly. + + 1.4.3 (2009-12-10) + ================== + + Bugs fixed: + + - Using pre-detected setuptools version for easy_installing tgz files. This + prevents a recursion error when easy_installing an upgraded "distribute" + tgz. Note that setuptools did not have this recursion problem solely + because it was packaged as an ``.egg``, which does not have to go through + the easy_install step. + + + 1.4.2 (2009-11-01) + ================== + + New Feature: + + - Added a --distribute option to the bootstrap script, in order + to use Distribute rather than Setuptools. By default, Setuptools + is used. + + Bugs fixed: + + - While checking for new versions of setuptools and buildout itself, + compare requirement locations instead of requirement objects. + + - Incrementing didn't work properly when extending multiple files. + https://bugs.launchpad.net/zc.buildout/+bug/421022 + + - The download API computed MD5 checksums of text files wrong on Windows. + + 1.4.1 (2009-08-27) + ================== + + New Feature: + + - Added a debug built-in recipe to make writing some tests easier. + + Bugs fixed: + + - (introduced in 1.4.0) option incrementing (-=) and decrementing (-=) + didn't work in the buildout section. + https://bugs.launchpad.net/zc.buildout/+bug/420463 + + - Option incrementing and decrementing didn't work for options + specified on the command line. + + - Scripts generated with relative-paths enabled couldn't be + symbolically linked to other locations and still work. + + - Scripts run using generated interpreters didn't have __file__ set correctly. + + - The standard Python -m option didn't work for custom interpreters. + + 1.4.0 (2009-08-26) + ================== + + - When doing variable substitutions, you can omit the section name to + refer to a variable in the same section (e.g. ${:foo}). + + - When doing variable substitution, you can use the special option, + ``_buildout_section_name_`` to get the section name. This is most handy + for getting the current section name (e.g. ${:_buildout_section_name_}). + + - A new special option, ``<`` allows sections to be used as macros. + + - Added annotate command for annotated sections. Displays sections + key-value pairs along with the value origin. + + - Added a download API that handles the download cache, offline mode etc and + is meant to be reused by recipes. + + - Used the download API to allow caching of base configurations (specified by + the buildout section's 'extends' option). + + 1.3.1 (2009-08-12) + ================== + + - Bug fixed: extras were ignored in some cases when versions were specified. + + 1.3.0 (2009-06-22) + ================== + + - Better Windows compatibility in test infrastructure. + + - Now the bootstrap.py has an optional --version argument, + that can be used to force zc.buildout version to use. + + - ``zc.buildout.testing.buildoutSetUp`` installs a new handler in the + python root logging facility. This handler is now removed during + tear down as it might disturb other packages reusing buildout's + testing infrastructure. + + - fixed usage of 'relative_paths' keyword parameter on Windows + + - Added an unload entry point for extensions. + + - Fixed bug: when the relative paths option was used, relative paths + could be inserted into sys.path if a relative path was used to run + the generated script. + + 1.2.1 (2009-03-18) + ================== + + - Refactored generation of relative egg paths to generate simpler code. + + 1.2.0 (2009-03-17) + ================== + + - Added a relative_paths option to zc.buildout.easy_install.script to + generate egg paths relative to the script they're used in. + + 1.1.2 (2009-03-16) + ================== + + - Added Python 2.6 support. Removed Python 2.3 support. + + - Fixed remaining deprecation warnings under Python 2.6, both when running + our tests and when using the package. + + - Switched from using os.popen* to subprocess.Popen, to avoid a deprecation + warning in Python 2.6. See: + + http://docs.python.org/library/subprocess.html#replacing-os-popen-os-popen2-os-popen3 + + - Made sure the 'redo_pyc' function and the doctest checkers work with Python + executable paths containing spaces. + + - Expand shell patterns when processing the list of paths in `develop`, e.g:: + + [buildout] + develop = ./local-checkouts/* + + - Conditionally import and use hashlib.md5 when it's available instead + of md5 module, which is deprecated in Python 2.6. + + - Added Jython support for bootstrap, development bootstrap + and zc.buildout support on Jython + + - Fixed a bug that would cause buildout to break while computing a + directory hash if it found a broken symlink (Launchpad #250573) + + 1.1.1 (2008-07-28) + ================== + + - Fixed a bug that caused buildouts to fail when variable + substitutions are used to name standard directories, as in:: + + [buildout] + eggs-directory = ${buildout:directory}/develop-eggs + + 1.1.0 (2008-07-19) + ================== + + - Added a buildout-level unzip option tp change the default policy for + unzipping zip-safe eggs. + + - Tracebacks are now printed for internal errors (as opposed to user + errors) even without the -D option. + + - pyc and pyo files are regenerated for installed eggs so that the + stored path in code objects matches the the install location. + + 1.0.6 (2008-06-13) + ================== + + - Manually reverted the changeset for the fix for + https://bugs.launchpad.net/zc.buildout/+bug/239212 to verify thet the test + actually fails with the changeset: + http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/buildout.py?rev=87309&r1=87277&r2=87309 + Thanks tarek for pointing this out. (seletz) + + - fixed the test for the += -= syntax in buildout.txt as the test + was actually wronng. The original implementation did a split/join + on whitespace, and later on that was corrected to respect the original + EOL setting, the test was not updated, though. (seletz) + + - added a test to verify against https://bugs.launchpad.net/zc.buildout/+bug/239212 + in allowhosts.txt (seletz) + + - further fixes for """AttributeError: Buildout instance has no + attribute '_logger'""" by providing reasonable defaults + within the Buildout constructor (related to the new 'allow-hosts' option) + (patch by Gottfried Ganssauge) (ajung) + + + 1.0.5 (2008-06-10) + ================== + + - Fixed wrong split when using the += and -= syntax (mustapha) + + 1.0.4 (2008-06-10) + ================== + + - Added the `allow-hosts` option (tarek) + + - Quote the 'executable' argument when trying to detect the python + version using popen4. (sidnei) + + - Quote the 'spec' argument, as in the case of installing an egg from + the buildout-cache, if the filename contains spaces it would fail (sidnei) + + - Extended configuration syntax to allow -= and += operators (malthe, mustapha). + + 1.0.3 (2008-06-01) + ================== + + - fix for """AttributeError: Buildout instance has no attribute '_logger'""" + by providing reasonable defaults within the Buildout constructor. + (patch by Gottfried Ganssauge) (ajung) + + 1.0.2 (2008-05-13) + ================== + + - More fixes for Windows. A quoted sh-bang is now used on Windows to make the + .exe files work with a Python executable in 'program files'. + + - Added "-t <timeout_in_seconds>" option for specifying the socket timeout. + (ajung) + + 1.0.1 (2008-04-02) + ================== + + - Made easy_install.py's _get_version accept non-final releases of Python, + like 2.4.4c0. (hannosch) + + - Applied various patches for Windows (patch by Gottfried Ganssauge). (ajung) + + - Applied patch fixing rmtree issues on Windows (patch by + Gottfried Ganssauge). (ajung) + + 1.0.0 (2008-01-13) + ================== + + - Added a French translation of the buildout tutorial. + + 1.0.0b31 (2007-11-01) + ===================== + + Feature Changes + --------------- + + - Added a configuration option that allows buildouts to ignore + dependency_links metadata specified in setup. By default + dependency_links in setup are used in addition to buildout specified + find-links. This can make it hard to control where eggs come + from. Here's how to tell buildout to ignore URLs in + dependency_links:: + + [buildout] + use-dependency-links = false + + By default use-dependency-links is true, which matches the behavior + of previous versions of buildout. + + - Added a configuration option that causes buildout to error if a + version is picked. This is a nice safety belt when fixing all + versions is intended, especially when creating releases. + + Bugs Fixed + ---------- + + - 151820: Develop failed if the setup.py script imported modules in + the distribution directory. + + - Verbose logging of the develop command was omitting detailed + output. + + - The setup command wasn't documented. + + - The setup command failed if run in a directory without specifying a + configuration file. + + - The setup command raised a stupid exception if run without arguments. + + - When using a local find links or index, distributions weren't copied + to the download cache. + + - When installing from source releases, a version specification (via a + buildout versions section) for setuptools was ignored when deciding + which setuptools to use to build an egg from the source release. + + 1.0.0b30 (2007-08-20) + ===================== + + Feature Changes + --------------- + + - Changed the default policy back to what it was to avoid breakage in + existing buildouts. Use:: + + [buildout] + prefer-final = true + + to get the new policy. The new policy will go into effect in + buildout 2. + + 1.0.0b29 (2007-08-20) + ===================== + + Feature Changes + --------------- + + - Now, final distributions are prefered over non-final versions. If + both final and non-final versions satisfy a requirement, then the + final version will be used even if it is older. The normal way to + override this for specific packages is to specifically require a + non-final version, either specifically or via a lower bound. + + - There is a buildout prefer-final version that can be used with a + value of "false":: + + prefer-final = false + + To prefer newer versions, regardless of whether or not they are + final, buildout-wide. + + - The new simple Python index, http://cheeseshop.python.org/simple, is + used as the default index. This will provide better performance + than the human package index interface, + http://pypi.python.org/pypi. More importantly, it lists hidden + distributions, so buildouts with fixed distribution versions will be + able to find old distributions even if the distributions have been + hidden in the human PyPI interface. + + Bugs Fixed + ---------- + + - 126441: Look for default.cfg in the right place on Windows. + + 1.0.0b28 (2007-07-05) + ===================== + + Bugs Fixed + ---------- + + - When requiring a specific version, buildout looked for new versions + even if that single version was already installed. + + 1.0.0b27 (2007-06-20) + ===================== + + Bugs Fixed + ---------- + + - Scripts were generated incorrectly on Windows. This included the + buildout script itself, making buildout completely unusable. + + 1.0.0b26 (2007-06-19) + ===================== + + Feature Changes + --------------- + + - Thanks to recent fixes in setuptools, I was able to change buildout + to use find-link and index information when searching extensions. + + Sadly, this work, especially the timing, was motivated my the need + to use alternate indexes due to performance problems in the cheese + shop (http://www.python.org/pypi/). I really home we can address + these performance problems soon. + + 1.0.0b25 (2007-05-31) + ===================== + + Feature Changes + --------------- + + - buildout now changes to the buildout directory before running recipe + install and update methods. + + - Added a new init command for creating a new buildout. This creates + an empty configuration file and then bootstraps. + + - Except when using the new init command, it is now an error to run + buildout without a configuration file. + + - In verbose mode, when adding distributions to fulful requirements of + already-added distributions, we now show why the new distributions + are being added. + + - Changed the logging format to exclude the logger name for the + zc.buildout logger. This reduces noise in the output. + + - Clean up lots of messages, adding missing periods and adding quotes around + requirement strings and file paths. + + Bugs Fixed + ---------- + + - 114614: Buildouts could take a very long time if there were + dependency problems in large sets of pathologically interdependent + packages. + + - 59270: Buggy recipes can cause failures in later recipes via chdir + + - 61890: file:// urls don't seem to work in find-links + + setuptools requires that file urls that point to directories must + end in a "/". Added a workaround. + + - 75607: buildout should not run if it creates an empty buildout.cfg + + 1.0.0b24 (2007-05-09) + ===================== + + Feature Changes + --------------- + + - Improved error reporting by showing which packages require other + packages that can't be found or that cause version conflicts. + + - Added an API for use by recipe writers to clean up created files + when recipe errors occur. + + - Log installed scripts. + + Bugs Fixed + ---------- + + - 92891: bootstrap crashes with recipe option in buildout section. + + - 113085: Buildout exited with a zero exist status when internal errors + occurred. + + + 1.0.0b23 (2007-03-19) + ===================== + + Feature Changes + --------------- + + - Added support for download caches. A buildout can specify a cache + for distribution downloads. The cache can be shared among buildouts + to reduce network access and to support creating source + distributions for applications allowing install without network + access. + + - Log scripts created, as suggested in: + https://bugs.launchpad.net/zc.buildout/+bug/71353 + + Bugs Fixed + ---------- + + - It wasn't possible to give options on the command line for sections + not defined in a configuration file. + + 1.0.0b22 (2007-03-15) + ===================== + + Feature Changes + --------------- + + - Improved error reporting and debugging support: + + - Added "logical tracebacks" that show functionally what the buildout + was doing when an error occurs. Don't show a Python traceback + unless the -D option is used. + + - Added a -D option that causes the buildout to print a traceback and + start the pdb post-mortem debugger when an error occurs. + + - Warnings are printed for unused options in the buildout section and + installed-part sections. This should make it easier to catch option + misspellings. + + - Changed the way the installed database (.installed.cfg) is handled + to avoid database corruption when a user breaks out of a buildout + with control-c. + + - Don't save an installed database if there are no installed parts or + develop egg links. + + 1.0.0b21 (2007-03-06) + ===================== + + Feature Changes + --------------- + + - Added support for repeatable buildouts by allowing egg versions to + be specified in a versions section. + + - The easy_install module install and build functions now accept a + versions argument that supplied to mapping from project name to + version numbers. This can be used to fix version numbers for + required distributions and their depenencies. + + When a version isn't fixed, using either a versions option or using + a fixed version number in a requirement, then a debug log message is + emitted indicating the version picked. This is useful for setting + versions options. + + A default_versions function can be used to set a default value for + this option. + + - Adjusted the output for verbosity levels. Using a single -v option + no longer causes voluminous setuptools output. Uisng -vv and -vvv + now triggers extra setuptools output. + + - Added a remove testing helper function that removes files or directories. + + 1.0.0b20 (2007-02-08) + ===================== + + Feature Changes + --------------- + + - Added a buildout newest option, to control whether the newest + distributions should be sought to meet requirements. This might + also provide a hint to recipes that don't deal with + distributions. For example, a recipe that manages subversion + checkouts might not update a checkout if newest is set to "false". + + - Added a *newest* keyword parameter to the + zc.buildout.easy_install.install and zc.buildout.easy_install.build + functions to control whether the newest distributions that meed + given requirements should be sought. If a false value is provided + for this parameter and already installed eggs meet the given + requirements, then no attempt will be made to search for newer + distributions. + + - The recipe-testing support setUp function now adds the name + *buildout* to the test namespace with a value that is the path to + the buildout script in the sample buildout. This allows tests to + use + + >>> print system(buildout), + + rather than: + + >>> print system(join('bin', 'buildout')), + + + Bugs Fixed + ---------- + + - Paths returned from update methods replaced lists of installed files + rather than augmenting them. + + 1.0.0b19 (2007-01-24) + ===================== + + Bugs Fixed + ---------- + + - Explicitly specifying a Python executable failed if the output of + running Python with the -V option included a 2-digit (rather than a + 3-digit) version number. + + 1.0.0b18 (2007-01-22) + ===================== + + Feature Changes + --------------- + + - Added documentation for some previously undocumented features of the + easy_install APIs. + + - By popular demand, added a -o command-line option that is a short + hand for the assignment buildout:offline=true. + + Bugs Fixed + ---------- + + - When deciding whether recipe develop eggs had changed, buildout + incorrectly considered files in .svn and CVS directories. + + 1.0.0b17 (2006-12-07) + ===================== + + Feature Changes + --------------- + + - Configuration files can now be loaded from URLs. + + Bugs Fixed + ---------- + + - https://bugs.launchpad.net/products/zc.buildout/+bug/71246 + + Buildout extensions installed as eggs couldn't be loaded in offline + mode. + + + 1.0.0b16 (2006-12-07) + ===================== + + Feature Changes + --------------- + + - A new command-line argument, -U, suppresses reading user defaults. + + - You can now suppress use of an installed-part database + (e.g. .installed.cfg) by sprifying an empty value for the buildout + installed option. + + Bugs Fixed + ---------- + + - When the install command is used with a list of parts, only + those parts are supposed to be installed, but the buildout was also + building parts that those parts depended on. + + 1.0.0b15 (2006-12-06) + ===================== + + Bugs Fixed + ---------- + + - Uninstall recipes weren't loaded correctly in cases where + no parts in the (new) configuration used the recipe egg. + + 1.0.0b14 (2006-12-05) + ===================== + + Feature Changes + --------------- + + - Added uninstall recipes for dealing with complex uninstallation + scenarios. + + Bugs Fixed + ---------- + + - Automatic upgrades weren't performed on Windows due to a bug that + caused buildout to incorrectly determine that it wasn't running + locally in a buildout. + + - Fixed some spurious test failures on Windows. + + 1.0.0b13 (2006-12-04) + ===================== + + Feature Changes + --------------- + + - Variable substitutions now reflect option data written by recipes. + + - A part referenced by a part in a parts list is now added to the parts + list before the referencing part. This means that you can omit + parts from the parts list if they are referenced by other parts. + + - Added a develop function to the easy_install module to aid in + creating develop eggs with custom build_ext options. + + - The build and develop functions in the easy_install module now + return the path of the egg or egg link created. + + - Removed the limitation that parts named in the install command can + only name configured parts. + + - Removed support ConfigParser-style variable substitutions + (e.g. %(foo)s). Only the string-template style of variable + (e.g. ${section:option}) substitutions will be supported. + Supporting both violates "there's only one way to do it". + + - Deprecated the buildout-section extendedBy option. + + Bugs Fixed + ---------- + + - We treat setuptools as a dependency of any distribution that + (declares that it) uses namespace packages, whether it declares + setuptools as a dependency or not. This wasn't working for eggs + intalled by virtue of being dependencies. + + + 1.0.0b12 (2006-10-24) + ===================== + + Feature Changes + --------------- + + - Added an initialization argument to the + zc.buildout.easy_install.scripts function to include initialization + code in generated scripts. + + 1.0.0b11 (2006-10-24) + ===================== + + Bugs Fixed + ---------- + + `67737 <https://launchpad.net/products/zc.buildout/+bug/67737>`_ + Verbose and quite output options caused errors when the + develop buildout option was used to create develop eggs. + + `67871 <https://launchpad.net/products/zc.buildout/+bug/67871>`_ + Installation failed if the source was a (local) unzipped + egg. + + `67873 <https://launchpad.net/products/zc.buildout/+bug/67873>`_ + There was an error in producing an error message when part names + passed to the install command weren't included in the + configuration. + + 1.0.0b10 (2006-10-16) + ===================== + + Feature Changes + --------------- + + - Renamed the runsetup command to setup. (The old name still works.) + + - Added a recipe update method. Now install is only called when a part + is installed for the first time, or after an uninstall. Otherwise, + update is called. For backward compatibility, recipes that don't + define update methiods are still supported. + + - If a distribution defines namespace packages but fails to declare + setuptools as one of its dependencies, we now treat setuptools as an + implicit dependency. We generate a warning if the distribution + is a develop egg. + + - You can now create develop eggs for setup scripts that don't use setuptools. + + Bugs Fixed + ---------- + + - Egg links weren't removed when corresponding entries were removed + from develop sections. + + - Running a non-local buildout command (one not installed in the + buildout) ket to a hang if there were new versions of zc.buildout or + setuptools were available. Now we issue a warning and don't + upgrade. + + - When installing zip-safe eggs from local directories, the eggs were + moved, rather than copied, removing them from the source directory. + + 1.0.0b9 (2006-10-02) + ==================== + + Bugs Fixed + ---------- + + Non-zip-safe eggs were not unzipped when they were installed. + + 1.0.0b8 (2006-10-01) + ==================== + + Bugs Fixed + ---------- + + - Installing source distributions failed when using alternate Python + versions (depending on the versions of Python used.) + + - Installing eggs wasn't handled as efficiently as possible due to a + bug in egg URL parsing. + + - Fixed a bug in runsetup that caused setup scripts that introspected + __file__ to fail. + + 1.0.0b7 + ======= + + Added a documented testing framework for use by recipes. Refactored + the buildout tests to use it. + + Added a runsetup command run a setup script. This is handy if, like + me, you don't install setuptools in your system Python. + + 1.0.0b6 + ======= + + Fixed https://launchpad.net/products/zc.buildout/+bug/60582 + Use of extension options caused bootstrapping to fail if the eggs + directory didn't already exist. We no longer use extensions for + bootstrapping. There really isn't any reason to anyway. + + + 1.0.0b5 + ======= + + Refactored to do more work in buildout and less work in easy_install. + This makes things go a little faster, makes errors a little easier to + handle, and allows extensions (like the sftp extension) to influence + more of the process. This was done to fix a problem in using the sftp + support. + + 1.0.0b4 + ======= + + - Added an **experimental** extensions mechanism, mainly to support + adding sftp support to buildouts that need it. + + - Fixed buildout self-updating on Windows. + + 1.0.0b3 + ======= + + - Added a help option (-h, --help) + + - Increased the default level of verbosity. + + - Buildouts now automatically update themselves to new versions of + zc.buildout and setuptools. + + - Added Windows support. + + - Added a recipe API for generating user errors. + + - No-longer generate a py_zc.buildout script. + + - Fixed some bugs in variable substitutions. + + The characters "-", "." and " ", weren't allowed in section or + option names. + + Substitutions with invalid names were ignored, which caused + missleading failures downstream. + + - Improved error handling. No longer show tracebacks for user errors. + + - Now require a recipe option (and therefore a section) for every part. + + - Expanded the easy_install module API to: + + - Allow extra paths to be provided + + - Specify explicit entry points + + - Specify entry-point arguments + + 1.0.0b2 + ======= + + Added support for specifying some build_ext options when installing eggs + from source distributions. + + 1.0.0b1 + ======= + + - Changed the bootstrapping code to only install setuptools and + zc.buildout. The bootstrap code no-longer runs the buildout itself. + This was to fix a bug that caused parts to be recreated + unnecessarily because the recipe signature in the initial buildout + reflected temporary locations for setuptools and zc.buildout. + + - Now create a minimal setup.py if it doesn't exist and issue a + warning that it is being created. + + - Fixed bug in saving installed configuration data. %'s and extra + spaces weren't quoted. + + 1.0.0a1 + ======= + + Initial public version + + Download + ********************** + +Keywords: development build +Platform: UNKNOWN +Classifier: Intended Audience :: Developers +Classifier: License :: OSI Approved :: Zope Public License +Classifier: Topic :: Software Development :: Build Tools +Classifier: Topic :: Software Development :: Libraries :: Python Modules |