././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2147336 zc_buildout-6.0.1/0000755000076500000240000000000015264141651013455 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.392994 zc_buildout-6.0.1/.coveragerc0000644000076500000240000000027215264141650015576 0ustar00mauritsstaff[run] source = zc.buildout parallel = true omit = */zc/buildout/_vendor/* [report] exclude_lines = pragma: no cover if __name__ == '__main__': raise NotImplementedError ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3939688 zc_buildout-6.0.1/CHANGES.rst0000644000076500000240000002761315264141650015267 0ustar00mauritsstaffChange History ************** .. You should *NOT* be adding new change log entries to this file. You should create a file in the news directory instead. For helpful instructions, please see: https://github.com/buildout/buildout/blob/master/doc/ADD-A-NEWS-ITEM.rst .. towncrier release notes start 6.0.1 (2026-10-15) ------------------ Bug fixes: - Also hide the deprecation warning when importing the vendored ``pkg_resources``. [icemac] - Fix ``DeprecationWarning`` about ``setDaemon()`` in ``zc.buildout.testing``. [icemac] 6.0.0 (2026-10-06) ------------------ New features: - Add support for python316-319 in conditional section expressions. [maurits] - Test on Python 3.15 and setuptools 84.0.0. [maurits] Bug fixes: - Don't show warnings from importing ``pkg_resources``. We know it is deprecated, but we have vendorized it. [maurits] Tests: - Add simple but real test projects. [maurits] 6.0.0a1 (2026-08-14) -------------------- Breaking changes: - Require ``setuptools >= 75.8.2``, up from ``61.0.0``. That release fixed ``pkg_resources.WorkingSet.find`` to consider several spellings of a requirement name, which is what most of buildout's remaining ``setuptools`` compatibility code worked around. Consequently the following patches are gone from ``zc.buildout.patches``: ``patch_pkg_resources_working_set_find`` (back-ported the 75.8.2 fix), ``patch_interpret_distro_name`` (only needed below setuptools 70, and already inoperative because it imported a module that had been renamed) and ``patch_Distribution`` (disabled for years). Require ``pip >= 25.0``, and test only against the latest pip release of each year. Require Python 3.10 as a minimum; 3.9 is out of security support. Use Python 3.13 instead of 3.10 for the CI jobs that vary something other than the Python version. See `issue 755 `_. (#755) New features: - Vendor ``pkg_resources`` from setuptools 81.0.0 (the last release that contained it) as ``zc.buildout._vendor.pkg_resources`` and install it as ``sys.modules['pkg_resources']`` when ``zc.buildout`` is imported. Buildout, recipes and extensions keep using the ``pkg_resources`` API inside a buildout run without needing setuptools to provide it. See `issue 685 `_. (#685) - Lifted the ``setuptools < 82`` restriction: both the ``setuptools`` requirement of ``zc.buildout`` and the automatic constraint applied to an unpinned ``setuptools`` during installs are gone, thanks to the vendored ``pkg_resources`` copy. Environments can now use setuptools 82 and later, including 83.0.0 which fixes the published vulnerability PYSEC-2026-3447 (GHSA-h35f-9h28-mq5c). Exception: distributions using pkg_resources-style namespace packages preferably get ``setuptools < 82`` added to their working set (unless setuptools is pinned), because their generated scripts import ``pkg_resources`` at script run time. When no such setuptools version is available, buildout falls back to the installed one. See `issue 750 `_. (#750) Bug fixes: - Fix support for PEP 660 develop packages (pyproject.toml only) by preserving .dist-info and processing .pth files. (pep660) Tests: - Fix detecting Windows in our ``prepare.sh`` script. [maurits] - Fix script tests. [maurits] - Test with ``pip`` 26.1.2. [maurits] - Update GitHub workflow action versions. [maurits] 5.2.0 (2026-04-29) ------------------ New features: - Restrict ``setuptools`` to less than 82 if it is not pinned. When ``zc.buildout`` checks if it should upgrade itself or ``setuptools``, under some circumstances this could lead to a too new ``setuptools`` version getting installed. [maurits] Tests: - Test with ``pip`` 26.1. [maurits] 5.1.3 (2026-03-06) ------------------ Bug fixes: - Replace 4 bare except clauses with except Exception. [haosenwang1018] 5.1.2 (2026-02-10) ------------------ Bug fixes: - Accept ``setuptools`` 81. Require ``setuptools<82``. [maurits] 5.1.1 (2025-11-25) ------------------ Bug fixes: - Store buildout index url in the installer, so pip can use it. [maurits] (#731) - When installing packages in development, treat the name as a file uri. Otherwise ``pip install -e package_name`` will look on PyPI, instead of a local ``package_name`` directory. [maurits] (#734) 5.1.0 (2025-11-20) ------------------ New features: - Support Python 3.14. No changes were needed. [maurits] (#314) - Warn when old-style namespaces are used in packages under development. [maurits] (#729) Bug fixes: - Require ``setuptools<81``. We need the ``pkg_resources`` module which is scheduled for removal in 81. [maurits] (#81) 5.0.0 (2025-11-12) ------------------ Tests: - Test with ``pip`` 25.3. (#727) 5.0.0a3 (2025-09-24) -------------------- Bug fixes: - Fix logic in detecting namespace init files for deletion. [maurits] (#720) 5.0.0a2 (2025-09-11) -------------------- Bug fixes: - Fix reading metadata files from dist-info in Windows. [maurits] (#722) 5.0.0a1 (2025-09-11) -------------------- Breaking changes: - Install development eggs (editable installs) using pip. Theoretically this means that you could develop a package that uses for example ``hatchling`` as build system. In practice this does not work yet, but the foundation is there. [maurits] (#676) - Install all namespace packages as native namespaces. [maurits] (#676) - Store eggs in a sub directory: ``eggs/v5``. Or with abi tags for example: ``eggs/v5/cp313``. [maurits] (#676) - Install most packages with pip, with only rare exceptions. [maurits] (#676) - The ``zc.buildout`` package itself uses native namespaces now. [maurits] (#676) - Require at least ``setuptools`` version 61.0.0. This is needed due to the changes in the test setup. [maurits] Tests: - The tests have mostly been changed to use wheels instead of eggs, so they more closely resemble real life. [maurits] (#676) - Removed inactive tests for no longer existing ``bootstrap.py``. [maurits] - Split the ``buildout.txt`` test file into multiple files. [maurits] 4.1.12 (2025-06-11) ------------------- Bug fixes: - Fix error ``get_win_launcher`` not found on Windows on setuptools 80.3+. [maurits] (#713) 4.1.11 (2025-06-11) ------------------- Bug fixes: - Fix development installs to still work when using setuptools 80.0.0. From then on, setuptools internally calls ``pip install --editable``. Note that "distutils scripts" can no longer be detected with setuptools 80. This seems an ancient technology, and probably hardly used. [maurits] (#708) - Use a copy of ``package_index.py`` from ``setuptools`` 80.2.0. This fixes compatibility with ``setuptools`` 80.3.0 where this module was removed. Merged some of our patches into this copy. [maurits] (#710) 4.1.10 (2025-05-21) ------------------- Bug fixes: - Override `pkg_resources.Environment.can_add` to have better results on Mac. Without this, a freshly created Mac-specific egg may not be considered compatible. This can happen when the Python you use was built on a different Mac OSX version. [maurits] (#609) 4.1.9 (2025-04-09) ------------------ Bug fixes: - Fix accidental changes to ``PYTHONPATH`` in ``os.environ`` when calling ``pip install``. [xavth] (#639) Tests - Use ``wheel`` 0.45.1 when testing with ``setuptools`` older than 70.1.0. Otherwise, when combining an older ``setuptools`` with a newer ``wheel`` version, the ``bdist_wheel`` command exists in neither of these packages. [maurits] (#705) 4.1.8 (2025-04-09) ------------------ Bug fixes: - Use the canonical name of a package when checking for a version constraint. [maurits] (#689) - Get actual project name from dist. Use this for naming the egg that gets created after installing a wheel or after doing a pip install of a source dist. [maurits] (#695) - Log all http errors when processing package url. [maurits] (#1013) 4.1.7 (2025-04-08) ------------------ Bug fixes: - Prevent getting package pages twice. Since version 4.1.5 we first request normalized package url on PyPI servers, but a subsequent check needed a fix. [maurits] (#634) - No longer recompile py files if we moved the dist. This code was never updated for Python 3, where the `.pyc` files are in a `__pycache__` directory, so it had no effect. [maurits] (#699) - Require at least `packaging` version 23.2. Needed because we use the `utils.is_normalized_name` function. [maurits] (#700) 4.1.6 (2025-04-03) ------------------ Tests - While creating sample packages for testing, mostly create wheels instead of eggs. For the sample source distributions, create ``tar.gz`` instead of ``zip`` files. Then our package index for testing is more like the actual PyPI. [maurits] (#675) 4.1.5 (2025-03-31) ------------------ Bug fixes: - Implement PEP 503: request normalized package url on PyPI servers. [andreclimaco] (#634) - Install ``wheel`` before ``setuptools`` when checking if an upgrade and restart are needed. [maurits] (#691) 4.1.4 (2025-03-07) ------------------ Bug fixes: - If needed, copy and rename wheels before making an egg out of them. This helps for wheels of namespace packages created with ``setuptools`` 75.8.1 or higher. For namespace package we need a dot instead of an underscore in the resulting egg name. [maurits] (#686) 4.1.3 (2025-03-05) ------------------ Bug fixes: - Patch the ``find`` method from ``pkg_resources.WorkingSet``. Let this use the code from ``setuptools`` 75.8.2, if the currently used version is older. This is better at finding installed distributions. But don't patch ``setuptools`` versions older than 61: the new version of the method would give an error there. [maurits] (#682) 4.1.2 (2025-03-05) ------------------ Bug fixes: - Fix error finding the ``zc.buildout`` distribution when checking if we need to upgrade/restart. This depends on your ``setuptools`` version. [maurits] (#681) 4.1.1 (2025-03-04) ------------------ Bug fixes: - Fix error adding minimum ``zc.buildout`` version as requirement. [maurits] (#679) 4.1 (2025-03-04) ---------------- New features: - In the ``ls`` testing method, add keyword argument ``lowercase_and_sort_output``. The default is False, so no change. When true, as the name says, it sorts the output by lowercase, and prints it lowercase. We need this in one test because with ``setuptools`` 75.8.1 we no longer have a filename ``MIXEDCASE-0.5-pyN.N.egg``, but ``mixedcase-0.5-pyN.N.egg``. [maurits] (#7581) Bug fixes: - When trying to find a distribution for ``package.name``, first try the normalized name (``package_name``). This fixes an error finding entry points for namespace packages. The error is: ``TypeError: ('Expected str, Requirement, or Distribution', None)``. [maurits] (#7581) Development: - Test with latest ``setuptools`` 75.8.2 and with ``pip`` 25.0.1. Note that ``setuptools`` 75.8.1 can be troublesome and should be avoided. [maurits] (#7581) 4.0 (2025-01-30) ---------------- Breaking changes: - Drop Python 3.8 support. Require 3.9 as minimum. (#38) Development: - Test against `setuptools == 75.6.0`. (#671) 4.0.0a1 (2024-10-22) -------------------- Breaking changes: - Add dependency on ``packaging``. This gets rid of ugly compatibility code. [maurits] (#38) - Require ``setuptools >= 49.0.0``. This is the first version that supports PEP 496 environment markers, for example ``demo ==0.1; python_version < '3.9'``. An earlier change had ``setuptools >= 42.0.2``, otherwise we got ImportErrors. Also, since this is higher than 38.2.3, we are sure to have support for wheels. Remove support for ``distribute``, which was probably already broken. [maurits] (#38) - Drop support for Python 2. Require Python 3.8 as minimum. [maurits] (#38) New features: - Support Python 3.12 and 3.13. This only needed a few test fixes. [maurits] (#38) ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.394024 zc_buildout-6.0.1/CONTRIBUTING.rst0000644000076500000240000000760415264141650016124 0ustar00mauritsstaffHow To Contribute ***************** Thank you for considering contributing to ``buildout``! Workflow ======== - No contribution is too small! - Please make sure to create one pull request for one change. - Please try to add tests for your code. - Make sure your changes pass **continuous integration**. When CI fails, please try to fix it or ask for help. Developing buildout itself and running the test suite ===================================================== When you're developing buildout itself, you need to know a few things: - The presence of ``~/.buildout/default.cfg`` may interfere with the tests. If you suspect this, you may want to temporarily rename it so that it does not get in the way. - Similarly, if you manually run ``bin/buildout`` you can run it with ``-U``. This ignores user (default) settings which can interfere with using the development version. For your convenience we provide a Makefile to create a Python virtual environment in the ``venvs`` subdirectory of the buildout checkout. You can have several venvs next to each other, but easiest is to use only one: the venvs are quick to recreate. To start, simply run ``make``. This uses ``python3``, so you get a venv with whatever your default ``python3`` version is. * It creates a venv named after this Python, for example ``venvs/python3.10``. * This has our dependencies: ``pip``, ``setuptools``, ``wheel``, ``packaging``. Plus ``build``, which may become a dependency, and that we need for the next step. * Using this venv, it creates a ``bin/buildout`` script in the main repository directory. * It then calls ``bin/buildout`` and this creates a ``bin/test`` script. * Then it calls ``bin/test`` to run the tests. You can call ``make clean`` to remove everything that was created. You can use environment variables to influence used versions: * Python, for example: ``PYTHON_VERSION=3.12`` * pip, for example: ``PIP_VERSION=25.3`` * setuptools, for example: ``SETUPTOOLS_VERSION=75.8.2`` * Use ``PIP_ARGS`` for extra arguments. The default is ``PIP_ARGS=-U``, so upgrade packages to the latest available version. So you can use this on the command line:: PYTHON_VERSION=3.12 PIP_VERSION=25.3 SETUPTOOLS_VERSION=75.8.2 make Releases: zc.buildout, zc.recipe.egg ==================================== Buildout consists of two Python packages that are released separately: zc.buildout and zc.recipe.egg. zc.recipe.egg is changed much less often than zc.buildout. zc.buildout's setup.py and changelog is in the same directory as this ``CONTRIBUTING.txt`` and the code is in ``src/zc/buildout``. zc.recipe.egg, including setup.py and a separate changelog, is in the ``zc.recipe.egg_`` subdirectory. When releasing, make sure you also build a (universal) wheel in addition to the regular .tar.gz:: $ ./venvs/python3.10/bin/python -m build . You can also use zest.releaser to release it. If you've installed it as ``zest.releaser[recommended]`` it builds the wheel for you and uploads using ``twine``. Roadmap ======= Currently, there are two active branches: - master (development branch for the upcoming version 6) - 5.x (maintenance branch for the current version 5) Active feature development and bug fixes only happen on the **master** branch. Supported Python Versions ========================= We align the support of Python versions with `Zope `_ and `Plone `_ development. This means, the upcoming version 6 will support Python 3.10-3.14. If you need support for Python 3.9, use version 5. If you need support for Python 2.7 or older Python 3 versions, use version 3. Version 6 also requires ``setuptools >= 75.8.2`` and ``pip >= 25.0``. Licensing ========= This project is licensed under the Zope Public License. Unlike contributing to the Zope and Plone projects, you do not need to sign a contributor agreement to contribute to **buildout**. ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.394093 zc_buildout-6.0.1/COPYRIGHT.txt0000644000076500000240000000004015264141650015557 0ustar00mauritsstaffZope Foundation and Contributors././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3941987 zc_buildout-6.0.1/HISTORY.rst0000644000076500000240000015171415264141650015360 0ustar00mauritsstaffChange History -- Old changes ***************************** 3.3 (2024-10-17) ---------------- New features: - Allow the ``-I`` option in the Python interpreter wrapper installed by buildout when using the ``zc.recipe.egg`` recipe's `interpreter =` directive. This solves the issue when VSCode calls the designated Python interpreter for a workspace with this option to determine the Python version etc. (`#627 `_) 3.2.0 (2024-09-26) ------------------ New features: - Add config option: ``optional-extends``. This is the same as the ``extends`` option, but then for optional files. The names must be file paths, not URLs. If the path does not exist, it is silently ignored. This is useful for optionally loading a ``local.cfg`` or ``custom.cfg`` with options specific for the developer or the server. [maurits] (`#665 `_) 3.1.1 (2024-09-20) ------------------ Bug fixes: - Fix: a variable defined with initial ``+=`` was undefined and would lead to a corrupted ``.installed.cfg``. Fixes `issue 641 `_. [distributist] - Fix: extends with increments could result in missing values. Buildout processes them in the correct order now and combines them correctly. Fixes `issue 176 `_ and `issue 629 `_. [distributist] (#644) - Fix: Multiple ``+=`` or ``/-=`` in one file would lose assignment in a previous file. Fixes `issue 656 `_. [distributist] 3.1.0 (2024-08-29) ------------------ Breaking changes: - Drop support for Python 3.5. It is unsupported, and testing it is too hard. [maurits] (#35) Bug fixes: - Normalize package names when gathering packages. This should help find all distributions for package ``name.space``, whether they are called ``name.space-1.0.tar.gz`` with a dot or ``name_space-1.0.tar.gz`` with an underscore (created with ``setuptools`` 69.3 or higher). [maurits] (#647) - Fix ImportError: cannot import name ``packaging`` from ``pkg_resources`` with setuptools 70. Done by adding a compatibility module that tries to import `packaging` from several places. Fixes `issue 648 `_. [maurits] (#648) 3.0.1 (2022-11-08) ------------------ Bug fixes: - Fixed import of packaging.markers. [maurits] (#621) 3.0.0 (2022-11-07) ------------------ New features: - Add support for PEP 508 markers in section condition expressions. For example: ``[versions:python_version <= "3.9"]``. [maurits] (#621) Bug fixes: - Command-line 'extends' now works with dirs in file names [gotcha] (cli-extends) - Add support for python311-315 in conditional section expressions. (#311) - Make compatible with pip 22.2+, restoring Requires-Python functionality there. Fixes `issue 613 `_. [maurits] (#613) 3.0.0rc3 (2022-04-07) --------------------- Bug fixes: - Fix `TypeError: dist must be a Distribution instance` due to issue between `setuptools` and `pip`. (#600) 3.0.0rc2 (2022-03-04) --------------------- New features: - add support for PEP496 environment markers (pep496) Bug fixes: - Fix TypeError for missing required `use_deprecated_html5lib` with pip 22. Keep compatible with earlier pip versions. (#598) 3.0.0rc1 (2021-12-16) --------------------- Bug fixes: - Call pip via `python -m pip`. (#569) 3.0.0b5 (2021-11-29) -------------------- Bug fixes: - Fix when c extension implements namespace packages without the corresponding directories. (#589) - Honor command-line buildout:extends (#592) 3.0.0b4 (2021-11-25) -------------------- New features: - Allow to run buildout in FIPS enabled environments. (#570) - Proper error message if extends-cache tries to expand ${section:variable} (#585) Bug fixes: - Forward verbose option to pip (#576) - Check that file top_level.txt exists before opening. Add check for other files as well. (#582) - Return code of pip install subprocess is now properly returned to buildout. (#586) 3.0.0b3 (2021-10-08) -------------------- New features: - Improve warning message when a section contains unused options. (#483) Bug fixes: - Fix support of ``pip>=21.1`` (#567) - Fix confusion when using multiple Python versions and installing packages with C extensions without proper binary wheel available. (#574) Development: - Avoid broken jobs on Travis because of security on PRs (travis-pr) 3.0.0b2 (2021-03-09) -------------------- New features: - Improve error message when a package version is not pinned and `allow-picked-versions = false`. (#481) Bug fixes: - Fix FileNotFoundError when installing eggs with top-level directory without code (like doc). (#556) Development: - Login to docker hub to avoid pull limits (travis) - Initialize towncrier (#519) 3.0.0b1 (2021-03-07) ==================== - Fix issue with combination of `>` specs and `extras` and recent `setuptools`. - Fix issue with incrementing options from `.buildout/default.cfg`. - Support python37, python38 and python39 in conditional section expressions. - Fix bootstrapping for python27 and python35. 3.0.0a2 (2020-05-25) ==================== - Ignore `.git` when computing signature of a recipe develop egg - Warn when the name passed to `zc.recipe.egg:scripts` is not defined in egg entry points. - Show pip warning about Python version only once. - Better patch for ``pkg_resources.Distribution.hashcmp`` performance. 3.0.0a1 (2020-05-17) ==================== - Scripts: ensure eggs are inserted before ``site-packages`` in ``sys.path``. - Fix forever loop when changing ``zc.buildout`` version via ``buildout``. - Add support for ``Requires-Python`` metadata. Fragile monkeypatch that relies on ``pip._internal``. Emits a warning when support is disabled due to changes in ``pip``. - Use ``pip install`` instead of deprecated ``setuptools.easy_install``. - Patch ``pkg_resources.Distribution`` to make install of unpinned versions quicker. Most obvious with ``setuptools``. 2.13.3 (2020-02-11) =================== - Fix DeprecationWarning about MutableMapping. (`#484 `_) 2.13.2 (2019-07-03) =================== - Fixed DeprecationWarning on python 3.7: "'U' mode is deprecated". 2.13.1 (2019-01-29) =================== - Documentation update for the new ``buildout query`` command. 2.13.0 (2019-01-17) =================== - Get information about the configuration with new command ``buildout query``. 2.12.2 (2018-09-04) =================== - Upon an error, buildout exits with a non-zero exit code. This now also works when running with ``-D``. - Fixed most 'Deprecation' and 'Resource' warnings. 2.12.1 (2018-07-02) =================== - zc.buildout now explicitly requests zc.recipe.egg >=2.0.6 now. 2.12.0 (2018-07-02) =================== - Add a new buildout option ``allow-unknown-extras`` to enable installing requirements that specify extras that do not exist. This needs a corresponding update to zc.recipe.egg. See `issue 457 `_. zc.recipe.egg has been updated to 2.0.6 for this change. 2.11.5 (2018-06-19) =================== - Fix for `issue 295 `_. On windows, deletion of temporary egg files is more robust now. 2.11.4 (2018-05-14) =================== - Fix for `issue 451 `_: distributions with a version number that normalizes to a shorter version number (3.3.0 to 3.3, for instance) can be installed now. 2.11.3 (2018-04-13) =================== - Update to use the new PyPI at https://pypi.org/. 2.11.2 (2018-03-19) =================== - Fix for the #442 issue: AttributeError on ``pkg_resources.SetuptoolsVersion``. 2.11.1 (2018-03-01) =================== - Made upgrade check more robust. When using extensions, the improvement introduced in 2.11 could prevent buildout from restarting itself when it upgraded setuptools. 2.11.0 (2018-01-21) =================== - Installed packages are added to the working set immediately. This helps in some corner cases that occur when system packages have versions that conflict with our specified versions. 2.10.0 (2017-12-04) =================== - Setuptools 38.2.0 started supporting wheels. Through setuptools, buildout now also supports wheels! You need at least version 38.2.3 to get proper namespace support. This setuptools change interfered with buildout's recent support for `buildout.wheel `_, resulting in a sudden "Wheels are not supported" error message (see `issue 435 `_). Fixed by making setuptools the default, though you can still use the buildout.wheel if you want. 2.9.6 (2017-12-01) ================== - Fixed: could not install eggs when sdist file name and package name had different case. 2.9.5 (2017-09-22) ================== - Use HTTPS for PyPI's index. PyPI redirects HTTP to HTTPS by default now so using HTTPS directly avoids the potential for that redirect being modified in flight. 2.9.4 (2017-06-20) ================== - Sort the distributions used to compute ``__buildout_signature__`` to ensure reproducibility under Python 3 or under Python 2 when ``-R`` is used on ``PYTHONHASHSEED`` is set to ``random``. Fixes `issue 392 `_. **NOTE**: This may cause existing ``.installed.cfg`` to be considered outdated and lead to parts being reinstalled spuriously under Python 2. - Add support code for doctests to be able to easily measure code coverage. See `issue 397 `_. 2.9.3 (2017-03-30) ================== - Add more verbosity to ``annotate`` results with ``-v`` - Select one or more sections with arguments after ``buildout annotate``. 2.9.2 (2017-03-06) ================== - Fixed: We unnecessarily used a function from newer versions of setuptools that caused problems when older setuptools or pkg_resources installs were present (as in travis.ci). 2.9.1 (2017-03-06) ================== - Fixed a minor packaging bug that broke the PyPI page. 2.9.0 (2017-03-06) ================== - Added new syntax to explicitly declare that a part depends on other part. See http://docs.buildout.org/en/latest/topics/implicit-parts.html - Internal refactoring to work with `buildout.wheel `_. - Fixed a bugs in ``zc.buildout.testing.Buildout``. It was loading user-default configuration. It didn't support calling the ``created`` method on its sections. - Fixed a bug (windows, py 3.4) When processing metadata on "old-style" distutils scripts, .exe stubs appeared in ``metadata_listdir``, in turn reading those burped with ``UnicodeDecodeError``. Skipping .exe stubs now. 2.8.0 (2017-02-13) ================== - Added a hook to enable a soon-to-be-released buildout extension to provide wheel support. 2.7.1 (2017-01-31) ================== - Fixed a bug introduced in 2.6.0: zc.buildout and its dependeoncies were reported as picked even when their versions were fixed in a ``versions`` section. Worse, when the ``update-versions-file`` option was used, the ``versions`` section was updated needlessly on every run. 2.7.0 (2017-01-30) ================== - Added a buildout option, ``abi-tag-eggs`` that, when true, causes the `ABI tag `_ for the buildout environment to be added to the eggs directory name. This is useful when switching Python implementations (e.g. CPython vs PyPI or debug builds vs regular builds), especially when environment differences aren't reflected in egg names. It also has the side benefit of making eggs directories smaller, because eggs for different Python versions are in different directories. 2.6.0 (2017-01-29) ================== - Updated to work with the latest setuptools. - Added (verified) Python 3.6 support. 2.5.3 (2016-09-05) ================== - After a dist is fetched and put into its final place, compile its python files. No longer wait with compiling until all dists are in place. This is related to the change below about not removing an existing egg. [maurits] - Do not remove an existing egg. When installing an egg to a location that already exists, keep the current location (directory or file). This can only happen when the location at first did not exist and this changed during the buildout run. We used to remove the previous location, but this could cause problems when running two buildouts at the same time, when they try to install the same new egg. Fixes #307. [maurits] - In ``zc.buildout.testing.system``, set ``TERM=dumb`` in the environment. This avoids invisible control characters popping up in some terminals, like ``xterm``. Note that this may affect tests by buildout recipes. [maurits] - Removed Python 2.6 and 3.2 support. [do3cc] 2.5.2 (2016-06-07) ================== - Fixed ``-=`` and ``+=`` when extending sections. See #161. [puittenbroek] 2.5.1 (2016-04-06) ================== - Fix python 2 for downloading external config files with basic auth in the URL. Fixes #257. 2.5.0 (2015-11-16) ================== - Added more elaborate version and requirement information when there's a version conflict. Previously, you could get a report of a version conflict without information about which dependency requested the conflicing requirement. Now all this information is logged and displayed in case of an error. [reinout] - Dropped 3.2 support (at least in the automatic tests) as setuptools will soon stop supporting it. Added python 3.5 to the automatic tests. [reinout] 2.4.7 (2015-10-29) ================== - Fix for #279. Distutils script detection previously broke on windows with python 3 because it errored on ``.exe`` files. [reinout] 2.4.6 (2015-10-28) ================== - Relative paths are now also correctly generated for the current directory ("develop = ."). [youngking] 2.4.5 (2015-10-14) ================== - More complete fix for #24. Distutils scripts are now also generated for develop eggs. [reinout] 2.4.4 (2015-10-02) ================== - zc.buildout is now also released as a wheel. (Note: buildout itself doesn't support installing wheels yet.) [graingert] 2.4.3 (2015-09-03) ================== - Added nested directory creation support [guyzmo] 2.4.2 (2015-08-26) ================== - If a downloaded config file in the "extends-cache" gets corrupted, buildout now tells you the filename in the cache. Handy for troubleshooting. [reinout] 2.4.1 (2015-08-08) ================== - Check the ``use-dependency-links`` option earlier. This can give a small speed increase. [maurits] - When using python 2, urllib2 is used to work around Python issue 24599, which affects downloading from behind a proxy. [stefano-m] 2.4.0 (2015-07-01) ================== - Buildout no longer breaks on packages that contain a file with a non-ascii filename. Fixes #89 and #148. [reinout] - Undo breakage on Windows machines where ``sys.prefix`` can also be a ``site-packages`` directory: don't remove it from ``sys.path``. See https://github.com/buildout/buildout/issues/217 . - Remove assumption that ``pkg_resources`` is a module (untrue since release of `setuptools 8.3``). See https://github.com/buildout/buildout/issues/227 . - Fix for #212. For certain kinds of conflict errors you'd get an UnpackError when rendering the error message. Instead of a nicely formatted version conflict message. [reinout] - Making sure we use the correct easy_install when setuptools is installed globally. See https://github.com/buildout/buildout/pull/232 and https://github.com/buildout/buildout/pull/222 . [lrowe] - Updated buildout's `travis-ci `_ configuration so that tests run much quicker so that buildout is easier and quicker to develop. [reinout] - Note: zc.recipe.egg has also been updated to 2.0.2 together with this zc.buildout release. Fixed: In ``zc.recipe.egg#custom`` recipe's ``rpath`` support, don't assume path elements are buildout-relative if they start with one of the "special" tokens (e.g., ``$ORIGIN``). See: https://github.com/buildout/buildout/issues/225. [tseaver] - ``download-cache``, ``eggs-directory`` and ``extends-cache`` are now automatically created if their parent directory exists. Also they can be relative directories (relative to the location of the buildout config file that defines them). Also they can now be in the form ``~/subdir``, with the usual convention that the ``~`` char means the home directory of the user running buildout. [lelit] - A new bootstrap.py file is released (version 2015-07-01). - When bootstrapping, the ``develop-eggs/`` directory is first removed. This prevents old left-over ``.egg-link`` files from breaking buildout's careful package collection mechanism. [reinout] - The bootstrap script now accepts ``--to-dir``. Setuptools is installed there. If already available there, it is reused. This can be used to bootstrap buildout without internet access. Similarly, a local ``ez_setup.py`` is used when available instead of it being downloaded. You need setuptools 14.0 or higher for this functionality. [lrowe] - The bootstrap script now uses ``--buildout-version`` instead of ``--version`` to pick a specific buildout version. [reinout] - The bootstrap script now accepts ``--version`` which prints the bootstrap version. This version is the date the bootstrap.py was last changed. A date is handier or less confusing than either tracking zc.buildout's version or having a separate bootstrap version number. [reinout] 2.3.1 (2014-12-16) ================== - Fixed: Buildout merged single-version requirements with version-range requirements in a way that caused it to think there wasn't a single-version requirement. IOW, buildout through that versions were being picked when they weren't. - Suppress spurious (and possibly non-spurious) version-parsing warnings. 2.3.0 (2014-12-14) ================== - Buildout is now compatible with (and requires) setuptools 8. 2.2.5 (2014-11-04) ================== - Improved fix for #198: when bootstrapping with an extension, buildout was too strict on itself, resulting in an inability to upgrade or downgrade its own version. [reinout] - Setuptools must be at 3.3 or higher now. If you use the latest bootstrap from http://downloads.buildout.org/2/bootstrap.py you're all set. [reinout] - Installing *recipes* that themselves have dependencies used to fail with a VersionConflict if such a dependency was installed globally with a lower version. Buildout now ignores the version conflict in those cases and simply installs the correct version. [reinout] 2.2.4 (2014-11-01) ================== - Fix for #198: buildout 2.2.3 caused a version conflict when bootstrapping a buildout with a version pinned to an earlier one. Same version conflict could occur with system-wide installed packages that were newer than the pinned version. [reinout] 2.2.3 (2014-10-30) ================== - Fix #197, Python 3 regression [aclark4life] 2.2.2 (2014-10-30) ================== - Open files for ``exec()`` in universal newlines mode. See https://github.com/buildout/buildout/issues/130 - Add ``BUILDOUT_HOME`` as an alternate way to control how the user default configuration is found. - Close various files when finished writing to them. This avoids ResourceWarnings on Python 3, and better supports doctests under PyPy. - Introduce improved easy_install Install.install function. This is present in 1.5.X and 1.7X but was never merged into 2.X somehow. 2.2.1 (2013-09-05) ================== - ``distutils`` scripts: correct order of operations on ``from ... import`` lines (see https://github.com/buildout/buildout/issues/134). - Add an ``--allow-site-packges`` option to ``bootstrap.py``, defaulting to False. If the value is false, strip any "site packages" (as defined by the ``site`` module) from ``sys.path`` before attempting to import ``setuptools`` / ``pkg_resources``. - Updated the URL used to fetch ``ez_setup.py`` to the official, non-version- pinned version. 2.2.0 (2013-07-05) ================== - Handle both addition and subtraction of elements (+= and -=) on the same key in the same section. Forward-ported from buildout 1.6. - Suppress the useless ``Link to ***BLOCKED*** by --allow-hosts`` error message being emitted by distribute / setuptools. - Extend distutils script generation to support module docstrings and __future__ imports. - Refactored picked versions logic to make it easier to use for plugins. - Use ``get_win_launcher`` API to find Windows launcher (falling back to ``resource_string`` for ``cli.exe``). - Remove ``data_files`` from ``setup.py``: it was installing ``README.txt`` in current directory during installation (merged from 1.x branch). - Switch dependency from ``distribute 0.6.x`` to ``setuptools 0.7.x``. 2.1.0 (2013-03-23) ================== - Meta-recipe support - Conditional sections - Buildout now accepts a ``--version`` command-line option to print its version. Fixed: Builout didn't exit with a non-zero exit status if there was a failure in combination with an upgrade. Fixed: We now fail with an informative error when an old bootstrap script causes buildout 2 to be used with setuptools. Fixed: An error incorrectly suggested that buildout 2 implemented all of the functionality of dumppickedversions. Fixed: Buildout generated bad scripts when no eggs needed to be added to ``sys.path``. Fixed: Buildout didn't honour Unix umask when generating scripts. https://bugs.launchpad.net/zc.buildout/+bug/180705 Fixed: ``update-versions-file`` didn't work unless ``show-picked-versions`` was also set. https://github.com/buildout/buildout/issues/71 2.0.1 (2013-02-16) ================== - Fixed: buildout didn't honor umask settings when creating scripts. - Fix for distutils scripts installation on Python 3, related to ``__pycache__`` directories. - Fixed: encoding data in non-entry-point-based scripts was lost. 2.0.0 (2013-02-10) ================== This is a backward incompatible release of buildout that attempts to correct mistakes made in buildout 1. - Buildout no-longer tries to provide full or partial isolation from system Python installations. If you want isolation, use buildout with virtualenv, or use a clean build of Python to begin with. Providing isolation was a noble goal, but it's implementation complicated buildout's implementation too much. - Buildout no-longer supports using multiple versions of Python in a single buildout. This too was a noble goal, but added too much complexity to the implementation. - Changed the configuration file format: - Relative indentation in option values is retained if the first line is blank. (IOW, if the non-blank text is on the continuation lines.) As in:: [mysection] tree = /root branch In such cases, internal blank lines are also retained. - The configuration syntax is more tightly defined, allowing fewer syntax definitions. Buildout 1 configuration files were parsed with the Python ConfigParser module. The ConfigParser module's format is poorly documented and wildly flexible. For example: - Any characters other than left square brackets were allowed in section names. - Arbitrary text was allowed and ignored after the closing bracket on section header lines. - Any characters other than equal signs or colons were allowed in an option name. - Configuration options could be spelled as RFC 822 mail headers (using a colon, rather than an equal sign). - Comments could begin with "rem". - Semicolons could be used to start inline comments, but only if preceded by a whitespace character. See `Configuration file syntax`_. - Buildout now prefers final releases by default (buildout:prefer-final now defaults to true, rather than false.) However, if buildout is bootstrapped with a non-final release, it won't downgrade itself to a final release. - Buildout no-longer installs zipped eggs. (Distribute may still install a zipped egg of itself during the bootstrapping process.) The ``buildout:unzip`` option has been removed. - Buildout no-longer supports setuptools. It now uses distribute exclusively. - Integrated the `buildout-versions `_ extension into buildout itself. For this, a few options were added to buildout: - If ``show-picked-versions`` is set to true, all picked versions are printed at the end of the buildout run. This saves you from running buildout in verbose mode and extracting the picked versions from the output. - If ``update-versions-file`` is set to a filename (relative to the buildout directory), the ``show-picked-versions`` output is appended to that file. - Buildout options can be given on the command line using the form:: option_name=value as a short-hand for:: buildout:option_name=value - The ``versions`` option now defaults to ``versions``, so you no longer need to include:: versions = versions in a ``buildout`` section when pinning versions. A ``versions`` section is added, if necessary, if a ``versions`` option isn't used. - Buildout-defined default versions are included in the versions section, if there is one. - The ``buildout:zc.buildout-version`` and ``buildout:distribute-version`` options have been removed in favor of providing version constraints in a versions section. - Error if install-from-cache and offline are used together, because offline largely means "don't install". - Provide better error messages when distributions can't be installed because buildout is run in offline mode. - Versions in versions sections can now be simple constraints, like >=2.0dev in addition to being simple versions. Buildout 2 leverages this to make sure it uses zc.recipe.egg>=2.0.0a3, which mainly matters for Python 3. - The buildout init command now accepts distribution requirements and paths to set up a custom interpreter part that has the distributions or parts in the path. For example:: python bootstrap.py init BeautifulSoup - Added buildout:socket-timeout option so that socket timeout can be configured both from command line and from config files. (gotcha) - Distutils-style scripts are also installed now (for instance pyflakes' and docutils' scripts). https://bugs.launchpad.net/zc.buildout/+bug/422724 - Avoid sorting the working set and requirements when it won't be logged. When profiling a simple buildout with 10 parts with identical and large working sets, this resulted in a decrease of run time from 93.411 to 15.068 seconds, about a 6 fold improvement. To see the benefit be sure to run without any increase in verbosity ("-v" option). (rossp) - Introduce a cache for the expensive `buildout._dir_hash` function. - Remove duplicate path from script's sys.path setup. - Make sure to download extended configuration files only once per buildout run even if they are referenced multiple times (patch by Rafael Monnerat). - Removed any traces of the implementation of ``extended-by``. Raise a UserError if the option is encountered instead of ignoring it, though. Fixed: relative-paths weren't honored when bootstrapping or upgrading (which is how the buildout script gets generated). Fixed: initialization code wasn't included in interpreter scripts. Fixed: macro inheritance bug, https://github.com/buildout/buildout/pull/37 Fixed: In the download module, fixed the handling of directories that are pointed to by file-system paths and ``file:`` URLs. Fixed if you have a configuration with an extends entry in the [buildout] section which points to a non-existing URL the result is not very user friendly. https://bugs.launchpad.net/zc.buildout/+bug/566167 Fixed: https://bugs.launchpad.net/bugs/697913 : Buildout doesn't honor exit code from scripts. Fixed. 1.4.4 (2010-08-20) ================== The 1.4.4 release is a release for people who encounter trouble with the 1.5 line. By switching to `the associated bootstrap script `_ you can stay on 1.4.4 until you are ready to migrate. 1.4.3 (2009-12-10) ================== Bugs fixed: - Using pre-detected setuptools version for easy_installing tgz files. This prevents a recursion error when easy_installing an upgraded "distribute" tgz. Note that setuptools did not have this recursion problem solely because it was packaged as an ``.egg``, which does not have to go through the easy_install step. 1.4.2 (2009-11-01) ================== New Feature: - Added a --distribute option to the bootstrap script, in order to use Distribute rather than Setuptools. By default, Setuptools is used. Bugs fixed: - While checking for new versions of setuptools and buildout itself, compare requirement locations instead of requirement objects. - Incrementing didn't work properly when extending multiple files. https://bugs.launchpad.net/zc.buildout/+bug/421022 - The download API computed MD5 checksums of text files wrong on Windows. 1.4.1 (2009-08-27) ================== New Feature: - Added a debug built-in recipe to make writing some tests easier. Bugs fixed: - (introduced in 1.4.0) option incrementing (-=) and decrementing (-=) didn't work in the buildout section. https://bugs.launchpad.net/zc.buildout/+bug/420463 - Option incrementing and decrementing didn't work for options specified on the command line. - Scripts generated with relative-paths enabled couldn't be symbolically linked to other locations and still work. - Scripts run using generated interpreters didn't have __file__ set correctly. - The standard Python -m option didn't work for custom interpreters. 1.4.0 (2009-08-26) ================== - When doing variable substitutions, you can omit the section name to refer to a variable in the same section (e.g. ${:foo}). - When doing variable substitution, you can use the special option, ``_buildout_section_name_`` to get the section name. This is most handy for getting the current section name (e.g. ${:_buildout_section_name_}). - A new special option, ``<`` allows sections to be used as macros. - Added annotate command for annotated sections. Displays sections key-value pairs along with the value origin. - Added a download API that handles the download cache, offline mode etc and is meant to be reused by recipes. - Used the download API to allow caching of base configurations (specified by the buildout section's 'extends' option). 1.3.1 (2009-08-12) ================== - Bug fixed: extras were ignored in some cases when versions were specified. 1.3.0 (2009-06-22) ================== - Better Windows compatibility in test infrastructure. - Now the bootstrap.py has an optional --version argument, that can be used to force buildout version to use. - ``zc.buildout.testing.buildoutSetUp`` installs a new handler in the python root logging facility. This handler is now removed during tear down as it might disturb other packages reusing buildout's testing infrastructure. - fixed usage of 'relative_paths' keyword parameter on Windows - Added an unload entry point for extensions. - Fixed bug: when the relative paths option was used, relative paths could be inserted into sys.path if a relative path was used to run the generated script. 1.2.1 (2009-03-18) ================== - Refactored generation of relative egg paths to generate simpler code. 1.2.0 (2009-03-17) ================== - Added a relative_paths option to zc.buildout.easy_install.script to generate egg paths relative to the script they're used in. 1.1.2 (2009-03-16) ================== - Added Python 2.6 support. Removed Python 2.3 support. - Fixed remaining deprecation warnings under Python 2.6, both when running our tests and when using the package. - Switched from using os.popen* to subprocess.Popen, to avoid a deprecation warning in Python 2.6. See: http://docs.python.org/library/subprocess.html#replacing-os-popen-os-popen2-os-popen3 - Made sure the 'redo_pyc' function and the doctest checkers work with Python executable paths containing spaces. - Expand shell patterns when processing the list of paths in `develop`, e.g:: [buildout] develop = ./local-checkouts/* - Conditionally import and use hashlib.md5 when it's available instead of md5 module, which is deprecated in Python 2.6. - Added Jython support for bootstrap, development bootstrap and buildout support on Jython - Fixed a bug that would cause buildout to break while computing a directory hash if it found a broken symlink (Launchpad #250573) 1.1.1 (2008-07-28) ================== - Fixed a bug that caused buildouts to fail when variable substitutions are used to name standard directories, as in:: [buildout] eggs-directory = ${buildout:directory}/develop-eggs 1.1.0 (2008-07-19) ================== - Added a buildout-level unzip option to change the default policy for unzipping zip-safe eggs. - Tracebacks are now printed for internal errors (as opposed to user errors) even without the -D option. - pyc and pyo files are regenerated for installed eggs so that the stored path in code objects matches the install location. 1.0.6 (2008-06-13) ================== - Manually reverted the changeset for the fix for https://bugs.launchpad.net/zc.buildout/+bug/239212 to verify thet the test actually fails with the changeset: http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/buildout.py?rev=87309&r1=87277&r2=87309 Thanks tarek for pointing this out. (seletz) - fixed the test for the += -= syntax in buildout.txt as the test was actually wrong. The original implementation did a split/join on whitespace, and later on that was corrected to respect the original EOL setting, the test was not updated, though. (seletz) - added a test to verify against https://bugs.launchpad.net/zc.buildout/+bug/239212 in allowhosts.txt (seletz) - further fixes for """AttributeError: Buildout instance has no attribute '_logger'""" by providing reasonable defaults within the Buildout constructor (related to the new 'allow-hosts' option) (patch by Gottfried Ganssauge) (ajung) 1.0.5 (2008-06-10) ================== - Fixed wrong split when using the += and -= syntax (mustapha) 1.0.4 (2008-06-10) ================== - Added the `allow-hosts` option (tarek) - Quote the 'executable' argument when trying to detect the python version using popen4. (sidnei) - Quote the 'spec' argument, as in the case of installing an egg from the buildout-cache, if the filename contains spaces it would fail (sidnei) - Extended configuration syntax to allow -= and += operators (malthe, mustapha). 1.0.3 (2008-06-01) ================== - fix for """AttributeError: Buildout instance has no attribute '_logger'""" by providing reasonable defaults within the Buildout constructor. (patch by Gottfried Ganssauge) (ajung) 1.0.2 (2008-05-13) ================== - More fixes for Windows. A quoted sh-bang is now used on Windows to make the .exe files work with a Python executable in 'program files'. - Added "-t " option for specifying the socket timeout. (ajung) 1.0.1 (2008-04-02) ================== - Made easy_install.py's _get_version accept non-final releases of Python, like 2.4.4c0. (hannosch) - Applied various patches for Windows (patch by Gottfried Ganssauge). (ajung) - Applied patch fixing rmtree issues on Windows (patch by Gottfried Ganssauge). (ajung) 1.0.0 (2008-01-13) ================== - Added a French translation of the buildout tutorial. 1.0.0b31 (2007-11-01) ===================== Feature Changes --------------- - Added a configuration option that allows buildouts to ignore dependency_links metadata specified in setup. By default dependency_links in setup are used in addition to buildout specified find-links. This can make it hard to control where eggs come from. Here's how to tell buildout to ignore URLs in dependency_links:: [buildout] use-dependency-links = false By default use-dependency-links is true, which matches the behavior of previous versions of buildout. - Added a configuration option that causes buildout to error if a version is picked. This is a nice safety belt when fixing all versions is intended, especially when creating releases. Bugs Fixed ---------- - 151820: Develop failed if the setup.py script imported modules in the distribution directory. - Verbose logging of the develop command was omitting detailed output. - The setup command wasn't documented. - The setup command failed if run in a directory without specifying a configuration file. - The setup command raised a stupid exception if run without arguments. - When using a local find links or index, distributions weren't copied to the download cache. - When installing from source releases, a version specification (via a buildout versions section) for setuptools was ignored when deciding which setuptools to use to build an egg from the source release. 1.0.0b30 (2007-08-20) ===================== Feature Changes --------------- - Changed the default policy back to what it was to avoid breakage in existing buildouts. Use:: [buildout] prefer-final = true to get the new policy. The new policy will go into effect in buildout 2. 1.0.0b29 (2007-08-20) ===================== Feature Changes --------------- - Now, final distributions are preferred over non-final versions. If both final and non-final versions satisfy a requirement, then the final version will be used even if it is older. The normal way to override this for specific packages is to specifically require a non-final version, either specifically or via a lower bound. - There is a buildout prefer-final version that can be used with a value of "false":: prefer-final = false To prefer newer versions, regardless of whether or not they are final, buildout-wide. - The new simple Python index, http://cheeseshop.python.org/simple, is used as the default index. This will provide better performance than the human package index interface, http://pypi.python.org/pypi. More importantly, it lists hidden distributions, so buildouts with fixed distribution versions will be able to find old distributions even if the distributions have been hidden in the human PyPI interface. Bugs Fixed ---------- - 126441: Look for default.cfg in the right place on Windows. 1.0.0b28 (2007-07-05) ===================== Bugs Fixed ---------- - When requiring a specific version, buildout looked for new versions even if that single version was already installed. 1.0.0b27 (2007-06-20) ===================== Bugs Fixed ---------- - Scripts were generated incorrectly on Windows. This included the buildout script itself, making buildout completely unusable. 1.0.0b26 (2007-06-19) ===================== Feature Changes --------------- - Thanks to recent fixes in setuptools, I was able to change buildout to use find-link and index information when searching extensions. Sadly, this work, especially the timing, was motivated my the need to use alternate indexes due to performance problems in the cheese shop (http://www.python.org/pypi/). I really home we can address these performance problems soon. 1.0.0b25 (2007-05-31) ===================== Feature Changes --------------- - buildout now changes to the buildout directory before running recipe install and update methods. - Added a new init command for creating a new buildout. This creates an empty configuration file and then bootstraps. - Except when using the new init command, it is now an error to run buildout without a configuration file. - In verbose mode, when adding distributions to fulfil requirements of already-added distributions, we now show why the new distributions are being added. - Changed the logging format to exclude the logger name for the buildout logger. This reduces noise in the output. - Clean up lots of messages, adding missing periods and adding quotes around requirement strings and file paths. Bugs Fixed ---------- - 114614: Buildouts could take a very long time if there were dependency problems in large sets of pathologically interdependent packages. - 59270: Buggy recipes can cause failures in later recipes via chdir - 61890: file:// urls don't seem to work in find-links setuptools requires that file urls that point to directories must end in a "/". Added a workaround. - 75607: buildout should not run if it creates an empty buildout.cfg 1.0.0b24 (2007-05-09) ===================== Feature Changes --------------- - Improved error reporting by showing which packages require other packages that can't be found or that cause version conflicts. - Added an API for use by recipe writers to clean up created files when recipe errors occur. - Log installed scripts. Bugs Fixed ---------- - 92891: bootstrap crashes with recipe option in buildout section. - 113085: Buildout exited with a zero exist status when internal errors occurred. 1.0.0b23 (2007-03-19) ===================== Feature Changes --------------- - Added support for download caches. A buildout can specify a cache for distribution downloads. The cache can be shared among buildouts to reduce network access and to support creating source distributions for applications allowing install without network access. - Log scripts created, as suggested in: https://bugs.launchpad.net/zc.buildout/+bug/71353 Bugs Fixed ---------- - It wasn't possible to give options on the command line for sections not defined in a configuration file. 1.0.0b22 (2007-03-15) ===================== Feature Changes --------------- - Improved error reporting and debugging support: - Added "logical tracebacks" that show functionally what the buildout was doing when an error occurs. Don't show a Python traceback unless the -D option is used. - Added a -D option that causes the buildout to print a traceback and start the pdb post-mortem debugger when an error occurs. - Warnings are printed for unused options in the buildout section and installed-part sections. This should make it easier to catch option misspellings. - Changed the way the installed database (.installed.cfg) is handled to avoid database corruption when a user breaks out of a buildout with control-c. - Don't save an installed database if there are no installed parts or develop egg links. 1.0.0b21 (2007-03-06) ===================== Feature Changes --------------- - Added support for repeatable buildouts by allowing egg versions to be specified in a versions section. - The easy_install module install and build functions now accept a versions argument that supplied to mapping from project name to version numbers. This can be used to fix version numbers for required distributions and their dependencies. When a version isn't fixed, using either a versions option or using a fixed version number in a requirement, then a debug log message is emitted indicating the version picked. This is useful for setting versions options. A default_versions function can be used to set a default value for this option. - Adjusted the output for verbosity levels. Using a single -v option no longer causes voluminous setuptools output. Using -vv and -vvv now triggers extra setuptools output. - Added a remove testing helper function that removes files or directories. 1.0.0b20 (2007-02-08) ===================== Feature Changes --------------- - Added a buildout newest option, to control whether the newest distributions should be sought to meet requirements. This might also provide a hint to recipes that don't deal with distributions. For example, a recipe that manages subversion checkouts might not update a checkout if newest is set to "false". - Added a *newest* keyword parameter to the zc.buildout.easy_install.install and zc.buildout.easy_install.build functions to control whether the newest distributions that need given requirements should be sought. If a false value is provided for this parameter and already installed eggs meet the given requirements, then no attempt will be made to search for newer distributions. - The recipe-testing support setUp function now adds the name *buildout* to the test namespace with a value that is the path to the buildout script in the sample buildout. This allows tests to use >>> print system(buildout), rather than: >>> print system(join('bin', 'buildout')), Bugs Fixed ---------- - Paths returned from update methods replaced lists of installed files rather than augmenting them. 1.0.0b19 (2007-01-24) ===================== Bugs Fixed ---------- - Explicitly specifying a Python executable failed if the output of running Python with the -V option included a 2-digit (rather than a 3-digit) version number. 1.0.0b18 (2007-01-22) ===================== Feature Changes --------------- - Added documentation for some previously undocumented features of the easy_install APIs. - By popular demand, added a -o command-line option that is a short hand for the assignment buildout:offline=true. Bugs Fixed ---------- - When deciding whether recipe develop eggs had changed, buildout incorrectly considered files in .svn and CVS directories. 1.0.0b17 (2006-12-07) ===================== Feature Changes --------------- - Configuration files can now be loaded from URLs. Bugs Fixed ---------- - https://bugs.launchpad.net/products/zc.buildout/+bug/71246 Buildout extensions installed as eggs couldn't be loaded in offline mode. 1.0.0b16 (2006-12-07) ===================== Feature Changes --------------- - A new command-line argument, -U, suppresses reading user defaults. - You can now suppress use of an installed-part database (e.g. .installed.cfg) by specifying an empty value for the buildout installed option. Bugs Fixed ---------- - When the install command is used with a list of parts, only those parts are supposed to be installed, but the buildout was also building parts that those parts depended on. 1.0.0b15 (2006-12-06) ===================== Bugs Fixed ---------- - Uninstall recipes weren't loaded correctly in cases where no parts in the (new) configuration used the recipe egg. 1.0.0b14 (2006-12-05) ===================== Feature Changes --------------- - Added uninstall recipes for dealing with complex uninstallation scenarios. Bugs Fixed ---------- - Automatic upgrades weren't performed on Windows due to a bug that caused buildout to incorrectly determine that it wasn't running locally in a buildout. - Fixed some spurious test failures on Windows. 1.0.0b13 (2006-12-04) ===================== Feature Changes --------------- - Variable substitutions now reflect option data written by recipes. - A part referenced by a part in a parts list is now added to the parts list before the referencing part. This means that you can omit parts from the parts list if they are referenced by other parts. - Added a develop function to the easy_install module to aid in creating develop eggs with custom build_ext options. - The build and develop functions in the easy_install module now return the path of the egg or egg link created. - Removed the limitation that parts named in the install command can only name configured parts. - Removed support ConfigParser-style variable substitutions (e.g. %(foo)s). Only the string-template style of variable (e.g. ${section:option}) substitutions will be supported. Supporting both violates "there's only one way to do it". - Deprecated the buildout-section extendedBy option. Bugs Fixed ---------- - We treat setuptools as a dependency of any distribution that (declares that it) uses namespace packages, whether it declares setuptools as a dependency or not. This wasn't working for eggs installed by virtue of being dependencies. 1.0.0b12 (2006-10-24) ===================== Feature Changes --------------- - Added an initialization argument to the zc.buildout.easy_install.scripts function to include initialization code in generated scripts. 1.0.0b11 (2006-10-24) ===================== Bugs Fixed ---------- `67737 `_ Verbose and quite output options caused errors when the develop buildout option was used to create develop eggs. `67871 `_ Installation failed if the source was a (local) unzipped egg. `67873 `_ There was an error in producing an error message when part names passed to the install command weren't included in the configuration. 1.0.0b10 (2006-10-16) ===================== Feature Changes --------------- - Renamed the runsetup command to setup. (The old name still works.) - Added a recipe update method. Now install is only called when a part is installed for the first time, or after an uninstall. Otherwise, update is called. For backward compatibility, recipes that don't define update methods are still supported. - If a distribution defines namespace packages but fails to declare setuptools as one of its dependencies, we now treat setuptools as an implicit dependency. We generate a warning if the distribution is a develop egg. - You can now create develop eggs for setup scripts that don't use setuptools. Bugs Fixed ---------- - Egg links weren't removed when corresponding entries were removed from develop sections. - Running a non-local buildout command (one not installed in the buildout) led to a hang if there were new versions of buildout or setuptools were available. Now we issue a warning and don't upgrade. - When installing zip-safe eggs from local directories, the eggs were moved, rather than copied, removing them from the source directory. 1.0.0b9 (2006-10-02) ==================== Bugs Fixed ---------- Non-zip-safe eggs were not unzipped when they were installed. 1.0.0b8 (2006-10-01) ==================== Bugs Fixed ---------- - Installing source distributions failed when using alternate Python versions (depending on the versions of Python used.) - Installing eggs wasn't handled as efficiently as possible due to a bug in egg URL parsing. - Fixed a bug in runsetup that caused setup scripts that introspected __file__ to fail. 1.0.0b7 ======= Added a documented testing framework for use by recipes. Refactored the buildout tests to use it. Added a runsetup command run a setup script. This is handy if, like me, you don't install setuptools in your system Python. 1.0.0b6 ======= Fixed https://launchpad.net/products/zc.buildout/+bug/60582 Use of extension options caused bootstrapping to fail if the eggs directory didn't already exist. We no longer use extensions for bootstrapping. There really isn't any reason to anyway. 1.0.0b5 ======= Refactored to do more work in buildout and less work in easy_install. This makes things go a little faster, makes errors a little easier to handle, and allows extensions (like the sftp extension) to influence more of the process. This was done to fix a problem in using the sftp support. 1.0.0b4 ======= - Added an **experimental** extensions mechanism, mainly to support adding sftp support to buildouts that need it. - Fixed buildout self-updating on Windows. 1.0.0b3 ======= - Added a help option (-h, --help) - Increased the default level of verbosity. - Buildouts now automatically update themselves to new versions of buildout and setuptools. - Added Windows support. - Added a recipe API for generating user errors. - No-longer generate a py_zc.buildout script. - Fixed some bugs in variable substitutions. The characters "-", "." and " ", weren't allowed in section or option names. Substitutions with invalid names were ignored, which caused misleading failures downstream. - Improved error handling. No longer show tracebacks for user errors. - Now require a recipe option (and therefore a section) for every part. - Expanded the easy_install module API to: - Allow extra paths to be provided - Specify explicit entry points - Specify entry-point arguments 1.0.0b2 ======= Added support for specifying some build_ext options when installing eggs from source distributions. 1.0.0b1 ======= - Changed the bootstrapping code to only install setuptools and buildout. The bootstrap code no-longer runs the buildout itself. This was to fix a bug that caused parts to be recreated unnecessarily because the recipe signature in the initial buildout reflected temporary locations for setuptools and buildout. - Now create a minimal setup.py if it doesn't exist and issue a warning that it is being created. - Fixed bug in saving installed configuration data. %'s and extra spaces weren't quoted. 1.0.0a1 ======= Initial public version ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3942728 zc_buildout-6.0.1/LICENSE.txt0000644000076500000240000000646015264141650015305 0ustar00mauritsstaffZope Public License (ZPL) Version 2.1 A copyright notice accompanies this license document that identifies the copyright holders. This license has been certified as open source. It has also been designated as GPL compatible by the Free Software Foundation (FSF). Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions in source code must retain the accompanying copyright notice, this list of conditions, and the following disclaimer. 2. Redistributions in binary form must reproduce the accompanying copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Names of the copyright holders must not be used to endorse or promote products derived from this software without prior written permission from the copyright holders. 4. The right to distribute this software or to use it for any purpose does not give you the right to use Servicemarks (sm) or Trademarks (tm) of the copyright holders. Use of them is covered by separate agreement with the copyright holders. 5. If any files are modified, you must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. Disclaimer THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. =============================================================================== The module, zc.buildout.pep425tags, is copied from the wheel package, which uses the MIT license: "wheel" copyright (c) 2012-2014 Daniel Holth and contributors. The MIT License Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.394316 zc_buildout-6.0.1/MANIFEST.in0000644000076500000240000000053415264141650015214 0ustar00mauritsstaffinclude *.py include *.rst include *.txt include .coveragerc include pyproject.toml recursive-include specifications *.txt recursive-include src/zc *.test recursive-include src/zc *.txt recursive-include src/zc *.py recursive-include src/zc *.rst prune doc prune old-tutorial prune zc.recipe.egg_ exclude .deepsource.toml prune news prune projects ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2146652 zc_buildout-6.0.1/PKG-INFO0000644000076500000240000005570315264141651014564 0ustar00mauritsstaffMetadata-Version: 2.4 Name: zc.buildout Version: 6.0.1 Summary: System for managing development buildouts Home-page: http://buildout.org Author: Jim Fulton Author-email: jim@zope.com License: ZPL 2.1 Keywords: development build Classifier: Development Status :: 6 - Mature Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Zope Public License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: 3.13 Classifier: Programming Language :: Python :: 3.14 Classifier: Programming Language :: Python :: 3.15 Classifier: Topic :: Software Development :: Build Tools Classifier: Topic :: Software Development :: Libraries :: Python Modules Requires-Python: >=3.10 License-File: LICENSE.txt Requires-Dist: setuptools>=75.8.2 Requires-Dist: packaging>=23.2 Requires-Dist: pip>=25.0 Requires-Dist: platformdirs Requires-Dist: wheel Provides-Extra: test Requires-Dist: zope.testing; extra == "test" Requires-Dist: manuel; extra == "test" Requires-Dist: bobo==2.3.0; extra == "test" Requires-Dist: zdaemon; extra == "test" Requires-Dist: zc.zdaemonrecipe; extra == "test" Requires-Dist: zc.recipe.deployment; extra == "test" Requires-Dist: hatchling; extra == "test" Requires-Dist: build; extra == "test" Dynamic: author Dynamic: author-email Dynamic: classifier Dynamic: description Dynamic: home-page Dynamic: keywords Dynamic: license Dynamic: license-file Dynamic: provides-extra Dynamic: requires-dist Dynamic: requires-python Dynamic: summary ******** Buildout ******** .. image:: https://github.com/buildout/buildout/actions/workflows/run-tests.yml/badge.svg :alt: GHA tests report :target: https://github.com/buildout/buildout/actions/workflows/run-tests.yml Buildout is a project designed to solve 2 problems: 1. Application-centric assembly and deployment *Assembly* runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.). Buildout might be confused with build tools like make or ant, but it is a little higher level and might invoke systems like make or ant to get its work done. Buildout might be confused with systems like puppet or chef, but it is more application focused. Systems like puppet or chef might use buildout to get their work done. Buildout is also somewhat Python-centric, even though it can be used to assemble and deploy non-python applications. It has some special features for assembling Python programs. It's scripted with Python, unlike, say puppet or chef, which are scripted with Ruby. 2. Repeatable assembly of programs from Python software distributions Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren't locked down, or in a deployment environment where dependency versions are fully specified. You should be able to check buildout into a VCS and later check it out. Two checkouts built at the same time in the same environment should always give the same result, regardless of their history. Among other things, after a buildout, all dependencies should be at the most recent version consistent with any version specifications expressed in the buildout. Buildout supports applications consisting of multiple programs, with different programs in an application free to use different versions of Python distributions. This is in contrast with a Python installation (real or virtual), where, for any given distribution, there can only be one installed. To learn more about buildout, including how to use it, see https://www.buildout.org/. Major changes in 6.x ******************** * Require ``setuptools >= 75.8.2``. * Require ``pip >= 25.0``. * Require Python 3.10 as a minimum. * Vendor ``pkg_resources`` from setuptools 81.0.0 (the last release that contained it). * Lifted the ``setuptools < 82`` restriction. This was there because we need ``pkg_resources``, but we now have our own copy. * Fix support for PEP 660 develop packages (``pyproject.toml`` only). See the change history below for more details. What does this mean in practice? * You can now develop packages that use other build systems than ``setuptools``. This means for example: `hatchling `_, `flit `_, `pdm `_. ``setuptools`` as build system still works of course And since ``zc.buildout`` 5 you already don't need a ``setup.py``, but can use ``pyproject.toml`` for all your package configuration. * This comes at a cost: packages that still use the deprecated ``pkg_resources`` style namespaces can cause problems: * When installed as standard package, all should still be well. * When used in development as package that simply gets pulled in by a recipe, it should still work. * But if it shares a namespace (for example ``collective.*``) with another package, and the other package uses the modern native namespaces, and one or both of the packages is a buildout extension or buildout recipe, then Buildout will most likely quit with a ``ModuleNotFound`` error. In that case, you should migrate the package to use native namespaces, or go back to ``zc.buildout`` 5, and maybe install ``horse-with-no-namespace``. See the next section. Since ``zc.buildout`` is used a lot in Plone, as one of the ways to install it: * Plone 6.2 will keep using ``zc.buildout`` 5 by default. * Plone 6.3 will most likely use ``zc.buildout`` 6. * In your own projects you are free to use ``zc.buildout`` 6 on Plone 6.2 already. * But beware: if you use ``collective.recipe.omelette`` and you develop a ``collective`` package that still uses ``pkg_resources`` namespaces, you will run into the problems described above. Native namespaces and breaking changes in 5.x ********************************************* Summary: ``zc.buildout`` 5.x installs most distributions with pip, even editable installs (with some remarks). It automatically uses native namespaces for all packages, except editable installs. Eggs go to directory ``eggs/v5`` to avoid compatibility problems. If a package name has a dot in it, it is a namespace package. ``zc.buildout`` has ``zc`` as namespace. Namespaces can be implemented in three ways: native namespaces (PEP 420, since Python 3.3), pkg_resources style (with ``__init__.py`` files that call ``pkg_resources.declare_namespace``) and pkgutil style (with ``__init__.py`` files that call ``pkgutil.extend_path``). Native namespaces are the modern way to do it. They work better with pip and other modern tools. The pkg_resources style depends on setuptools and is considered deprecated: setuptools 82 dropped support for it, removing its foundational ``pkg_resources`` module. ``zc.buildout`` ships its own copy of ``pkg_resources`` (taken from setuptools 81, see ``src/zc/buildout/_vendor/README.rst``), so buildout itself, recipes and extensions can keep using it inside a buildout run with any setuptools version. Scripts generated for distributions that use pkg_resources-style namespaces still need a real ``pkg_resources`` at script run time, so buildout prefers adding ``setuptools < 82`` to their working set (unless you pin setuptools yourself). When no such setuptools version can be found or installed, buildout falls back to the available setuptools; if that is 82 or later, those scripts will not find ``pkg_resources`` at run time. Problems start when you have multiple packages in the same namespace, that use different implementations. But it depends on what you use to install the packages. In the following examples, we have two packages in the same namespace, say ``ns.native`` (using native namespaces) and ``ns.deprecated`` (using pkg_resources style). * Make editable installs of both packages (``pip install -e`` or in buildout, ``develop =``): - This works neither in pip nor in buildout. - You can install the `horse-with-no-namespace `_ package to get this working. * Make a normal install of both packages: - This works fine in pip. - This fails in buildout 4.x. - This works fine in buildout 5.x. * Make a normal install of one package and an editable install of the other: - This works fine in pip. - This fails in buildout 4.x. - This fails in buildout 5.x as well. But again, you can use ``horse-with-no-namespace`` to get this working. So the big difference between buildout 4.x and 5.x is that with 5.x you can mix namespace styles when you do normal installs. This is possible due to the following major changes in 5.x: * The ``zc.buildout`` package itself uses native namespaces now. * ``zc.buildout`` 5.x installs most packages with pip, with only rare exceptions. * The tests have mostly been changed to use wheels instead of eggs, so they more closely resemble real life. * Buildout now automatically treats all namespace packages as having native namespaces. Previously, after we installed a ``pkg_resources-style`` namespace package with pip, the ``__init__.py`` files would be missing, so we would explicitly add them again. Now we no longer do this. In fact, in some cases these files *are* there after installation, and we explicitly *remove* them. * This means that "eggs" created by this Buildout version are not compatible with eggs from a previous Buildout version, at least for namespace packages. So Buildout now stores the eggs in a sub directory: ``eggs/v5``. Or with abi tags for example: ``eggs/v5/cp313``. So it should be fine to keep using the same shared eggs cache, even if you are using different zc.buildout versions. * Development eggs (editable installs) are now also installed by pip. Previously, zc.buildout would call ``python setup.py develop`` to install them, failing if there was no ``setup.py`` file. Now this file is no longer needed. Theoretically this means that you could develop a package that uses for example ``hatchling`` as build system. In practice this does not work yet, but the foundation is there. The Plone project is a major user of buildout, although alternative installation methods are becoming more popular. So what does this mean for Plone? * Plone 6.1 and earlier will keep using ``zc.buildout`` 4.x and ``pkg_resources``-style namespaces * Plone 6.2 will switch to ``zc.buildout`` 5.x. * Plone 6.2 will temporarily use ``horse-with-no-namespace`` both when installed with ``pip`` and with ``zc.buildout``. * This combination means that each package in Plone 6.2 can switch to native namespaces at its own pace. Previously, all packages in a namespace had to switch at the same time, and the tests would be broken until that happened. * In your own projects on any Plone 6 version, you are free to use any buildout and setuptools version you want. If Buildout 5 works on your Plone 6.0 project, or Buildout 4 works on your Plone 6.2 project, that is fine. Change History ************** .. You should *NOT* be adding new change log entries to this file. You should create a file in the news directory instead. For helpful instructions, please see: https://github.com/buildout/buildout/blob/master/doc/ADD-A-NEWS-ITEM.rst .. towncrier release notes start 6.0.1 (2026-10-15) ------------------ Bug fixes: - Also hide the deprecation warning when importing the vendored ``pkg_resources``. [icemac] - Fix ``DeprecationWarning`` about ``setDaemon()`` in ``zc.buildout.testing``. [icemac] 6.0.0 (2026-10-06) ------------------ New features: - Add support for python316-319 in conditional section expressions. [maurits] - Test on Python 3.15 and setuptools 84.0.0. [maurits] Bug fixes: - Don't show warnings from importing ``pkg_resources``. We know it is deprecated, but we have vendorized it. [maurits] Tests: - Add simple but real test projects. [maurits] 6.0.0a1 (2026-08-14) -------------------- Breaking changes: - Require ``setuptools >= 75.8.2``, up from ``61.0.0``. That release fixed ``pkg_resources.WorkingSet.find`` to consider several spellings of a requirement name, which is what most of buildout's remaining ``setuptools`` compatibility code worked around. Consequently the following patches are gone from ``zc.buildout.patches``: ``patch_pkg_resources_working_set_find`` (back-ported the 75.8.2 fix), ``patch_interpret_distro_name`` (only needed below setuptools 70, and already inoperative because it imported a module that had been renamed) and ``patch_Distribution`` (disabled for years). Require ``pip >= 25.0``, and test only against the latest pip release of each year. Require Python 3.10 as a minimum; 3.9 is out of security support. Use Python 3.13 instead of 3.10 for the CI jobs that vary something other than the Python version. See `issue 755 `_. (#755) New features: - Vendor ``pkg_resources`` from setuptools 81.0.0 (the last release that contained it) as ``zc.buildout._vendor.pkg_resources`` and install it as ``sys.modules['pkg_resources']`` when ``zc.buildout`` is imported. Buildout, recipes and extensions keep using the ``pkg_resources`` API inside a buildout run without needing setuptools to provide it. See `issue 685 `_. (#685) - Lifted the ``setuptools < 82`` restriction: both the ``setuptools`` requirement of ``zc.buildout`` and the automatic constraint applied to an unpinned ``setuptools`` during installs are gone, thanks to the vendored ``pkg_resources`` copy. Environments can now use setuptools 82 and later, including 83.0.0 which fixes the published vulnerability PYSEC-2026-3447 (GHSA-h35f-9h28-mq5c). Exception: distributions using pkg_resources-style namespace packages preferably get ``setuptools < 82`` added to their working set (unless setuptools is pinned), because their generated scripts import ``pkg_resources`` at script run time. When no such setuptools version is available, buildout falls back to the installed one. See `issue 750 `_. (#750) Bug fixes: - Fix support for PEP 660 develop packages (pyproject.toml only) by preserving .dist-info and processing .pth files. (pep660) Tests: - Fix detecting Windows in our ``prepare.sh`` script. [maurits] - Fix script tests. [maurits] - Test with ``pip`` 26.1.2. [maurits] - Update GitHub workflow action versions. [maurits] 5.2.0 (2026-04-29) ------------------ New features: - Restrict ``setuptools`` to less than 82 if it is not pinned. When ``zc.buildout`` checks if it should upgrade itself or ``setuptools``, under some circumstances this could lead to a too new ``setuptools`` version getting installed. [maurits] Tests: - Test with ``pip`` 26.1. [maurits] 5.1.3 (2026-03-06) ------------------ Bug fixes: - Replace 4 bare except clauses with except Exception. [haosenwang1018] 5.1.2 (2026-02-10) ------------------ Bug fixes: - Accept ``setuptools`` 81. Require ``setuptools<82``. [maurits] 5.1.1 (2025-11-25) ------------------ Bug fixes: - Store buildout index url in the installer, so pip can use it. [maurits] (#731) - When installing packages in development, treat the name as a file uri. Otherwise ``pip install -e package_name`` will look on PyPI, instead of a local ``package_name`` directory. [maurits] (#734) 5.1.0 (2025-11-20) ------------------ New features: - Support Python 3.14. No changes were needed. [maurits] (#314) - Warn when old-style namespaces are used in packages under development. [maurits] (#729) Bug fixes: - Require ``setuptools<81``. We need the ``pkg_resources`` module which is scheduled for removal in 81. [maurits] (#81) 5.0.0 (2025-11-12) ------------------ Tests: - Test with ``pip`` 25.3. (#727) 5.0.0a3 (2025-09-24) -------------------- Bug fixes: - Fix logic in detecting namespace init files for deletion. [maurits] (#720) 5.0.0a2 (2025-09-11) -------------------- Bug fixes: - Fix reading metadata files from dist-info in Windows. [maurits] (#722) 5.0.0a1 (2025-09-11) -------------------- Breaking changes: - Install development eggs (editable installs) using pip. Theoretically this means that you could develop a package that uses for example ``hatchling`` as build system. In practice this does not work yet, but the foundation is there. [maurits] (#676) - Install all namespace packages as native namespaces. [maurits] (#676) - Store eggs in a sub directory: ``eggs/v5``. Or with abi tags for example: ``eggs/v5/cp313``. [maurits] (#676) - Install most packages with pip, with only rare exceptions. [maurits] (#676) - The ``zc.buildout`` package itself uses native namespaces now. [maurits] (#676) - Require at least ``setuptools`` version 61.0.0. This is needed due to the changes in the test setup. [maurits] Tests: - The tests have mostly been changed to use wheels instead of eggs, so they more closely resemble real life. [maurits] (#676) - Removed inactive tests for no longer existing ``bootstrap.py``. [maurits] - Split the ``buildout.txt`` test file into multiple files. [maurits] 4.1.12 (2025-06-11) ------------------- Bug fixes: - Fix error ``get_win_launcher`` not found on Windows on setuptools 80.3+. [maurits] (#713) 4.1.11 (2025-06-11) ------------------- Bug fixes: - Fix development installs to still work when using setuptools 80.0.0. From then on, setuptools internally calls ``pip install --editable``. Note that "distutils scripts" can no longer be detected with setuptools 80. This seems an ancient technology, and probably hardly used. [maurits] (#708) - Use a copy of ``package_index.py`` from ``setuptools`` 80.2.0. This fixes compatibility with ``setuptools`` 80.3.0 where this module was removed. Merged some of our patches into this copy. [maurits] (#710) 4.1.10 (2025-05-21) ------------------- Bug fixes: - Override `pkg_resources.Environment.can_add` to have better results on Mac. Without this, a freshly created Mac-specific egg may not be considered compatible. This can happen when the Python you use was built on a different Mac OSX version. [maurits] (#609) 4.1.9 (2025-04-09) ------------------ Bug fixes: - Fix accidental changes to ``PYTHONPATH`` in ``os.environ`` when calling ``pip install``. [xavth] (#639) Tests - Use ``wheel`` 0.45.1 when testing with ``setuptools`` older than 70.1.0. Otherwise, when combining an older ``setuptools`` with a newer ``wheel`` version, the ``bdist_wheel`` command exists in neither of these packages. [maurits] (#705) 4.1.8 (2025-04-09) ------------------ Bug fixes: - Use the canonical name of a package when checking for a version constraint. [maurits] (#689) - Get actual project name from dist. Use this for naming the egg that gets created after installing a wheel or after doing a pip install of a source dist. [maurits] (#695) - Log all http errors when processing package url. [maurits] (#1013) 4.1.7 (2025-04-08) ------------------ Bug fixes: - Prevent getting package pages twice. Since version 4.1.5 we first request normalized package url on PyPI servers, but a subsequent check needed a fix. [maurits] (#634) - No longer recompile py files if we moved the dist. This code was never updated for Python 3, where the `.pyc` files are in a `__pycache__` directory, so it had no effect. [maurits] (#699) - Require at least `packaging` version 23.2. Needed because we use the `utils.is_normalized_name` function. [maurits] (#700) 4.1.6 (2025-04-03) ------------------ Tests - While creating sample packages for testing, mostly create wheels instead of eggs. For the sample source distributions, create ``tar.gz`` instead of ``zip`` files. Then our package index for testing is more like the actual PyPI. [maurits] (#675) 4.1.5 (2025-03-31) ------------------ Bug fixes: - Implement PEP 503: request normalized package url on PyPI servers. [andreclimaco] (#634) - Install ``wheel`` before ``setuptools`` when checking if an upgrade and restart are needed. [maurits] (#691) 4.1.4 (2025-03-07) ------------------ Bug fixes: - If needed, copy and rename wheels before making an egg out of them. This helps for wheels of namespace packages created with ``setuptools`` 75.8.1 or higher. For namespace package we need a dot instead of an underscore in the resulting egg name. [maurits] (#686) 4.1.3 (2025-03-05) ------------------ Bug fixes: - Patch the ``find`` method from ``pkg_resources.WorkingSet``. Let this use the code from ``setuptools`` 75.8.2, if the currently used version is older. This is better at finding installed distributions. But don't patch ``setuptools`` versions older than 61: the new version of the method would give an error there. [maurits] (#682) 4.1.2 (2025-03-05) ------------------ Bug fixes: - Fix error finding the ``zc.buildout`` distribution when checking if we need to upgrade/restart. This depends on your ``setuptools`` version. [maurits] (#681) 4.1.1 (2025-03-04) ------------------ Bug fixes: - Fix error adding minimum ``zc.buildout`` version as requirement. [maurits] (#679) 4.1 (2025-03-04) ---------------- New features: - In the ``ls`` testing method, add keyword argument ``lowercase_and_sort_output``. The default is False, so no change. When true, as the name says, it sorts the output by lowercase, and prints it lowercase. We need this in one test because with ``setuptools`` 75.8.1 we no longer have a filename ``MIXEDCASE-0.5-pyN.N.egg``, but ``mixedcase-0.5-pyN.N.egg``. [maurits] (#7581) Bug fixes: - When trying to find a distribution for ``package.name``, first try the normalized name (``package_name``). This fixes an error finding entry points for namespace packages. The error is: ``TypeError: ('Expected str, Requirement, or Distribution', None)``. [maurits] (#7581) Development: - Test with latest ``setuptools`` 75.8.2 and with ``pip`` 25.0.1. Note that ``setuptools`` 75.8.1 can be troublesome and should be avoided. [maurits] (#7581) 4.0 (2025-01-30) ---------------- Breaking changes: - Drop Python 3.8 support. Require 3.9 as minimum. (#38) Development: - Test against `setuptools == 75.6.0`. (#671) 4.0.0a1 (2024-10-22) -------------------- Breaking changes: - Add dependency on ``packaging``. This gets rid of ugly compatibility code. [maurits] (#38) - Require ``setuptools >= 49.0.0``. This is the first version that supports PEP 496 environment markers, for example ``demo ==0.1; python_version < '3.9'``. An earlier change had ``setuptools >= 42.0.2``, otherwise we got ImportErrors. Also, since this is higher than 38.2.3, we are sure to have support for wheels. Remove support for ``distribute``, which was probably already broken. [maurits] (#38) - Drop support for Python 2. Require Python 3.8 as minimum. [maurits] (#38) New features: - Support Python 3.12 and 3.13. This only needed a few test fixes. [maurits] (#38) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3944502 zc_buildout-6.0.1/README.rst0000644000076500000240000002267315264141650015155 0ustar00mauritsstaff******** Buildout ******** .. image:: https://github.com/buildout/buildout/actions/workflows/run-tests.yml/badge.svg :alt: GHA tests report :target: https://github.com/buildout/buildout/actions/workflows/run-tests.yml Buildout is a project designed to solve 2 problems: 1. Application-centric assembly and deployment *Assembly* runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.). Buildout might be confused with build tools like make or ant, but it is a little higher level and might invoke systems like make or ant to get its work done. Buildout might be confused with systems like puppet or chef, but it is more application focused. Systems like puppet or chef might use buildout to get their work done. Buildout is also somewhat Python-centric, even though it can be used to assemble and deploy non-python applications. It has some special features for assembling Python programs. It's scripted with Python, unlike, say puppet or chef, which are scripted with Ruby. 2. Repeatable assembly of programs from Python software distributions Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren't locked down, or in a deployment environment where dependency versions are fully specified. You should be able to check buildout into a VCS and later check it out. Two checkouts built at the same time in the same environment should always give the same result, regardless of their history. Among other things, after a buildout, all dependencies should be at the most recent version consistent with any version specifications expressed in the buildout. Buildout supports applications consisting of multiple programs, with different programs in an application free to use different versions of Python distributions. This is in contrast with a Python installation (real or virtual), where, for any given distribution, there can only be one installed. To learn more about buildout, including how to use it, see https://www.buildout.org/. Major changes in 6.x ******************** * Require ``setuptools >= 75.8.2``. * Require ``pip >= 25.0``. * Require Python 3.10 as a minimum. * Vendor ``pkg_resources`` from setuptools 81.0.0 (the last release that contained it). * Lifted the ``setuptools < 82`` restriction. This was there because we need ``pkg_resources``, but we now have our own copy. * Fix support for PEP 660 develop packages (``pyproject.toml`` only). See the change history below for more details. What does this mean in practice? * You can now develop packages that use other build systems than ``setuptools``. This means for example: `hatchling `_, `flit `_, `pdm `_. ``setuptools`` as build system still works of course And since ``zc.buildout`` 5 you already don't need a ``setup.py``, but can use ``pyproject.toml`` for all your package configuration. * This comes at a cost: packages that still use the deprecated ``pkg_resources`` style namespaces can cause problems: * When installed as standard package, all should still be well. * When used in development as package that simply gets pulled in by a recipe, it should still work. * But if it shares a namespace (for example ``collective.*``) with another package, and the other package uses the modern native namespaces, and one or both of the packages is a buildout extension or buildout recipe, then Buildout will most likely quit with a ``ModuleNotFound`` error. In that case, you should migrate the package to use native namespaces, or go back to ``zc.buildout`` 5, and maybe install ``horse-with-no-namespace``. See the next section. Since ``zc.buildout`` is used a lot in Plone, as one of the ways to install it: * Plone 6.2 will keep using ``zc.buildout`` 5 by default. * Plone 6.3 will most likely use ``zc.buildout`` 6. * In your own projects you are free to use ``zc.buildout`` 6 on Plone 6.2 already. * But beware: if you use ``collective.recipe.omelette`` and you develop a ``collective`` package that still uses ``pkg_resources`` namespaces, you will run into the problems described above. Native namespaces and breaking changes in 5.x ********************************************* Summary: ``zc.buildout`` 5.x installs most distributions with pip, even editable installs (with some remarks). It automatically uses native namespaces for all packages, except editable installs. Eggs go to directory ``eggs/v5`` to avoid compatibility problems. If a package name has a dot in it, it is a namespace package. ``zc.buildout`` has ``zc`` as namespace. Namespaces can be implemented in three ways: native namespaces (PEP 420, since Python 3.3), pkg_resources style (with ``__init__.py`` files that call ``pkg_resources.declare_namespace``) and pkgutil style (with ``__init__.py`` files that call ``pkgutil.extend_path``). Native namespaces are the modern way to do it. They work better with pip and other modern tools. The pkg_resources style depends on setuptools and is considered deprecated: setuptools 82 dropped support for it, removing its foundational ``pkg_resources`` module. ``zc.buildout`` ships its own copy of ``pkg_resources`` (taken from setuptools 81, see ``src/zc/buildout/_vendor/README.rst``), so buildout itself, recipes and extensions can keep using it inside a buildout run with any setuptools version. Scripts generated for distributions that use pkg_resources-style namespaces still need a real ``pkg_resources`` at script run time, so buildout prefers adding ``setuptools < 82`` to their working set (unless you pin setuptools yourself). When no such setuptools version can be found or installed, buildout falls back to the available setuptools; if that is 82 or later, those scripts will not find ``pkg_resources`` at run time. Problems start when you have multiple packages in the same namespace, that use different implementations. But it depends on what you use to install the packages. In the following examples, we have two packages in the same namespace, say ``ns.native`` (using native namespaces) and ``ns.deprecated`` (using pkg_resources style). * Make editable installs of both packages (``pip install -e`` or in buildout, ``develop =``): - This works neither in pip nor in buildout. - You can install the `horse-with-no-namespace `_ package to get this working. * Make a normal install of both packages: - This works fine in pip. - This fails in buildout 4.x. - This works fine in buildout 5.x. * Make a normal install of one package and an editable install of the other: - This works fine in pip. - This fails in buildout 4.x. - This fails in buildout 5.x as well. But again, you can use ``horse-with-no-namespace`` to get this working. So the big difference between buildout 4.x and 5.x is that with 5.x you can mix namespace styles when you do normal installs. This is possible due to the following major changes in 5.x: * The ``zc.buildout`` package itself uses native namespaces now. * ``zc.buildout`` 5.x installs most packages with pip, with only rare exceptions. * The tests have mostly been changed to use wheels instead of eggs, so they more closely resemble real life. * Buildout now automatically treats all namespace packages as having native namespaces. Previously, after we installed a ``pkg_resources-style`` namespace package with pip, the ``__init__.py`` files would be missing, so we would explicitly add them again. Now we no longer do this. In fact, in some cases these files *are* there after installation, and we explicitly *remove* them. * This means that "eggs" created by this Buildout version are not compatible with eggs from a previous Buildout version, at least for namespace packages. So Buildout now stores the eggs in a sub directory: ``eggs/v5``. Or with abi tags for example: ``eggs/v5/cp313``. So it should be fine to keep using the same shared eggs cache, even if you are using different zc.buildout versions. * Development eggs (editable installs) are now also installed by pip. Previously, zc.buildout would call ``python setup.py develop`` to install them, failing if there was no ``setup.py`` file. Now this file is no longer needed. Theoretically this means that you could develop a package that uses for example ``hatchling`` as build system. In practice this does not work yet, but the foundation is there. The Plone project is a major user of buildout, although alternative installation methods are becoming more popular. So what does this mean for Plone? * Plone 6.1 and earlier will keep using ``zc.buildout`` 4.x and ``pkg_resources``-style namespaces * Plone 6.2 will switch to ``zc.buildout`` 5.x. * Plone 6.2 will temporarily use ``horse-with-no-namespace`` both when installed with ``pip`` and with ``zc.buildout``. * This combination means that each package in Plone 6.2 can switch to native namespaces at its own pace. Previously, all packages in a namespace had to switch at the same time, and the tests would be broken until that happened. * In your own projects on any Plone 6 version, you are free to use any buildout and setuptools version you want. If Buildout 5 works on your Plone 6.0 project, or Buildout 4 works on your Plone 6.2 project, that is fine. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3945484 zc_buildout-6.0.1/dev.py0000644000076500000240000000322715264141650014610 0ustar00mauritsstaff# Create bin/buildout script. # This should be called with python from a virtualenv that has all our # dependencies already installed. from pathlib import Path import build # NOQA: assert the build dependency is installed import os import platform import sys EGG_INFO_PATH = "src/zc.buildout.egg-info" if not os.path.exists(EGG_INFO_PATH): print(f"ERROR: {EGG_INFO_PATH} does not exist.") print("You should run 'python -m build --sdist' (or 'python setup.py egg_info').") sys.exit(1) # The 'bin' directory must exist. os.makedirs('bin', exist_ok=True) # zc.buildout must be importable in the current session. Put 'src' on the # path *before* importing it: importing zc.buildout also makes its vendored # pkg_resources available as plain `pkg_resources` (setuptools >= 82 no # longer provides it), and the pkg_resources working set is built from # sys.path at import time, so this way it includes src/zc.buildout.egg-info. sys.path.insert(0, "src") # Important note: isort must NOT move these lines. import zc.buildout.easy_install import pkg_resources # And then Buildout can install its own script. zc.buildout.easy_install.scripts( ['zc.buildout'], pkg_resources.working_set, sys.executable, 'bin' ) if platform.system() == "Windows": buildout_script = Path("bin/buildout.exe") else: buildout_script = Path("bin/buildout") if buildout_script.exists(): print(f"SUCCESS: Generated {buildout_script} script.") if platform.system() != "Windows": # On Windows you get a UnicodeDecodeError that I don't want to debug. print(buildout_script.read_text()) else: print(f"ERROR: Generating {buildout_script} failed.") sys.exit(1) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3988051 zc_buildout-6.0.1/pyproject.toml0000644000076500000240000000106515264141650016372 0ustar00mauritsstaff[tool.towncrier] filename = "CHANGES.rst" directory = "news/" title_format = "{version} ({project_date})" underlines = ["-", ""] [[tool.towncrier.type]] directory = "breaking" name = "Breaking changes:" showcontent = true [[tool.towncrier.type]] directory = "feature" name = "New features:" showcontent = true [[tool.towncrier.type]] directory = "bugfix" name = "Bug fixes:" showcontent = true [[tool.towncrier.type]] directory = "develop" name = "Development:" showcontent = true [[tool.towncrier.type]] directory = "tests" name = "Tests:" showcontent = true ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2149198 zc_buildout-6.0.1/setup.cfg0000644000076500000240000000037715264141651015305 0ustar00mauritsstaff[bdist_wheel] universal = 0 [check-manifest] ignore = *.cfg .dockerignore Makefile prepare.sh zc.recipe.egg_/* zc.recipe.egg_ .github/* .github [zest.releaser] create-wheel = yes extra-message = [ci skip] [egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3989348 zc_buildout-6.0.1/setup.py0000644000076500000240000000525415264141650015174 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2006-2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## name = "zc.buildout" version = "6.0.1" import os from setuptools import setup def read(*rnames): with open(os.path.join(os.path.dirname(__file__), *rnames)) as f: return f.read() long_description= read('README.rst') + '\n' + read('CHANGES.rst') entry_points = """ [console_scripts] buildout = %(name)s.buildout:main [zc.buildout] debug = %(name)s.testrecipes:Debug """ % dict(name=name) setup( name = name, version = version, author = "Jim Fulton", author_email = "jim@zope.com", description = "System for managing development buildouts", long_description=long_description, license = "ZPL 2.1", keywords = "development build", url='http://buildout.org', packages = [ 'zc', 'zc.buildout', 'zc.buildout._vendor', 'zc.buildout._vendor.pkg_resources', ], package_dir = {'': 'src'}, python_requires = '>=3.10', install_requires = [ 'setuptools>=75.8.2', 'packaging>=23.2', 'pip>=25.0', # platformdirs is used by the vendored pkg_resources copy, see # src/zc/buildout/_vendor/README.rst 'platformdirs', 'wheel', ], include_package_data = True, entry_points = entry_points, extras_require = dict( test=['zope.testing', 'manuel', 'bobo ==2.3.0', 'zdaemon', 'zc.zdaemonrecipe', 'zc.recipe.deployment', 'hatchling', 'build']), zip_safe=False, classifiers = [ 'Development Status :: 6 - Mature', 'Intended Audience :: Developers', 'License :: OSI Approved :: Zope Public License', 'Programming Language :: Python', 'Programming Language :: Python :: 3.10', 'Programming Language :: Python :: 3.11', 'Programming Language :: Python :: 3.12', 'Programming Language :: Python :: 3.13', 'Programming Language :: Python :: 3.14', 'Programming Language :: Python :: 3.15', 'Topic :: Software Development :: Build Tools', 'Topic :: Software Development :: Libraries :: Python Modules', ], ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2083683 zc_buildout-6.0.1/specifications/0000755000076500000240000000000015264141651016460 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.399021 zc_buildout-6.0.1/specifications/README.txt0000644000076500000240000000036315264141650020157 0ustar00mauritsstaffThis directory is (an experimental) place to manage Launchpad specification bodies. Files here are referenced (via svn.zope.org URLs) from the buildout project specifications in Launchpad, https://features.launchpad.net/products/zc.buildout. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3990793 zc_buildout-6.0.1/specifications/repeatable.txt0000644000076500000240000000562315264141650021332 0ustar00mauritsstaffRepeatable (taggable) buildouts =============================== It's important to be able to tag a buildout in a software repository in such a way that, months, or even years later, the buildout tag can be checked out and used to construct the same collection of parts, with the same versions. (Note that parts could still behave differently due to changes in parts of the environment, such as system libraries, not controlled by the buildout.) A feature of the buildout is it's use of eggs and the automatic resolution of dependencies. The latest versions of dependencies are automatically downloaded and installed. This is great during development or when using the buildout for casual software development, but it doesn't work very well for reproducing an old buildout. What's needed is some way to, when needed, record information about the versions of eggs (and any other bits) who's versions are determined dynamically. Proposal -------- We'll add a buildout option, create-repeatable. The option will specify a file into which option information should be saved to create a repeatable buildout. The data will be saved in a form that can be used by the buildout or recipes in a later run. To make a tagged buildout, a user would run the buildout with the create-repeatable option set to a file name and then modify the buildout to be extended-by this file. Consider the following example buildout.cfg:: [buildout] parts = foo [foo] recipe = zc.recipe.eggs eggs = foo eek Now assume that: - The current version of foo is 1.1 - Foo depends on bar =, which depends on baz. The current versions of bar and bas are 1.1 and 2.1. - The current version of eek is 1.5 - eek depends on ook, which is as version 1.3. - zc.recipe.egg is at version 1.0b5 If we run the buildout with the command-line option:: buildout:create-repeatable=reapeatable.cfg we'll get a repeatable.cfg file that looks something like:: [foo] recipe = zc.recipe.eggs ==1.0b5 static = true eggs = foo ==1.1 bar ==1.1 baz ==2.1 eek ==1.5 ook ==1.3 The file contains options for the foo part. The buildout software itself added an entry for the recipe that fixes the recipe version at the version used by the buildout. The zc.recipe.eggs recipe added the eggs option that lists the specific releases that were assembled. Finally the buildout.cfg file can be modified to use the repeatable.cfg file:: [buildout] parts = foo extended-by: repeatable.cfg [foo] recipe = zc.recipe.eggs eggs = foo eek When the buildout is run, the options in repeatable.cfg will override the ones in buildout.cfg, providing a repeatable buildout Python API ---------- The recipe API will grow a repeatable method that is called after the install method and is passed a dictionary that a recipe can store option data in. A recipe instance will only be able to provide repeatable data for it's part. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2066922 zc_buildout-6.0.1/src/0000755000076500000240000000000015264141651014244 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2067242 zc_buildout-6.0.1/src/zc/0000755000076500000240000000000015264141651014660 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2102838 zc_buildout-6.0.1/src/zc/buildout/0000755000076500000240000000000015264141651016507 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3992562 zc_buildout-6.0.1/src/zc/buildout/__init__.py0000644000076500000240000000603115264141650020617 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Buildout package """ # do not change the import order # deleting the spec_for_pip hack needs to be done before importing pip # see https://github.com/pypa/pip/issues/8761 to understand # the reason for the hack. # I think it is reasonable to assume we will not run into the race. import setuptools try: from _distutils_hack import DistutilsMetaFinder if hasattr(DistutilsMetaFinder, 'spec_for_pip'): del DistutilsMetaFinder.spec_for_pip except ImportError: pass import pip # NOQA import sys import warnings # zc.buildout ships its own copy of pkg_resources, taken from setuptools # 81.0.0 (the last release that contained it). Make it importable as # plain `pkg_resources` for buildout's own code and for recipes and # extensions running inside the buildout process. If some other code # already imported pkg_resources before us (only possible with # setuptools < 82), keep using that copy to preserve module identity # (isinstance checks etc.). # See src/zc/buildout/_vendor/README.rst and # https://github.com/buildout/buildout/issues/685 _replace_pkg_resources = False if 'pkg_resources' not in sys.modules: _replace_pkg_resources = True else: try: with warnings.catch_warnings(): # Don't show warnings from importing pkg_resources. # We know it is deprecated, but we have vendorized it. warnings.simplefilter("ignore") from pkg_resources import PkgResourcesDeprecationWarning del PkgResourcesDeprecationWarning except ImportError: # This may be a bare bones pkg_resources from horse-with-no-namespace _replace_pkg_resources = True with warnings.catch_warnings(): # Don't show warnings from importing pkg_resources. # We know it is deprecated, but we have vendorized it. warnings.simplefilter("ignore") if _replace_pkg_resources: from zc.buildout._vendor import ( pkg_resources as _vendored_pkg_resources) sys.modules['pkg_resources'] = _vendored_pkg_resources from pkg_resources import PkgResourcesDeprecationWarning warnings.filterwarnings('ignore', category=PkgResourcesDeprecationWarning) warnings.filterwarnings('ignore', message='Setuptools is replacing distutils.') import zc.buildout.patches # NOQA WINDOWS = sys.platform.startswith('win') class UserError(Exception): """Errors made by a user """ def __str__(self): return " ".join(map(str, self.args)) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3993785 zc_buildout-6.0.1/src/zc/buildout/_package_index.py0000644000076500000240000013353515264141650022013 0ustar00mauritsstaff"""PyPI and direct package downloading. This file was copied from setuptools v80.2.0, the file `package_index.py`, with minor changes. setuptools has the following license statement. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """ from __future__ import annotations import base64 import configparser import hashlib import html import http.client import io import itertools import os import re import shutil import socket import subprocess import sys import urllib.error import urllib.parse import urllib.request from fnmatch import translate from functools import wraps from typing import NamedTuple import setuptools from pkg_resources import ( BINARY_DIST, CHECKOUT_DIST, DEVELOP_DIST, EGG_DIST, SOURCE_DIST, Distribution, Environment, Requirement, find_distributions, normalize_path, parse_version, safe_name, safe_version, to_filename, ) from setuptools.wheel import Wheel from distutils import log from distutils.errors import DistutilsError EGG_FRAGMENT = re.compile(r'^egg=([-A-Za-z0-9_.+!]+)$') HREF = re.compile(r"""href\s*=\s*['"]?([^'"> ]+)""", re.I) PYPI_MD5 = re.compile( r'([^<]+)\n\s+\(md5\)' ) URL_SCHEME = re.compile('([-+.a-z0-9]{2,}):', re.I).match EXTENSIONS = ".tar.gz .tar.bz2 .tar .zip .tgz".split() __all__ = [ 'PackageIndex', 'distros_for_url', 'parse_bdist_wininst', 'interpret_distro_name', ] _SOCKET_TIMEOUT = 15 user_agent = f"setuptools/{setuptools.__version__} Python-urllib/{sys.version_info.major}.{sys.version_info.minor}" try: from setuptools.unicode_utils import _cfg_read_utf8_with_fallback from setuptools.unicode_utils import _read_utf8_with_fallback except ImportError: # BBB These functions were introduced in setuptools 70.0.0. from configparser import RawConfigParser from setuptools import SetuptoolsDeprecationWarning # Explicitly use the ``"locale"`` encoding in versions that support it, # otherwise just rely on the implicit handling of ``encoding=None``. # Since all platforms that support ``EncodingWarning`` also support # ``encoding="locale"``, this can be used to suppress the warning. # However, please try to use UTF-8 when possible # (.pth files are the notorious exception: python/cpython#77102, pypa/setuptools#3937). # This definition is copied from setuptools.compat.py39. LOCALE_ENCODING = "locale" if sys.version_info >= (3, 10) else None class _Utf8EncodingNeeded(SetuptoolsDeprecationWarning): _SUMMARY = """ `encoding="utf-8"` fails with {file!r}, trying `encoding={fallback_encoding!r}`. """ _DETAILS = """ Fallback behavior for UTF-8 is considered **deprecated** and future versions of `setuptools` may not implement it. Please encode {file!r} with "utf-8" to ensure future builds will succeed. If this file was produced by `setuptools` itself, cleaning up the cached files and re-building/re-installing the package with a newer version of `setuptools` (e.g. by updating `build-system.requires` in its `pyproject.toml`) might solve the problem. """ # TODO: Add a deadline? # Will we be able to remove this? # The question comes to mind mainly because of sdists that have been produced # by old versions of setuptools and published to PyPI... def _read_utf8_with_fallback(file: str, fallback_encoding=LOCALE_ENCODING) -> str: """ First try to read the file with UTF-8, if there is an error fallback to a different encoding ("locale" by default). Returns the content of the file. Also useful when reading files that might have been produced by an older version of setuptools. """ try: with open(file, "r", encoding="utf-8") as f: return f.read() except UnicodeDecodeError: # pragma: no cover _Utf8EncodingNeeded.emit(file=file, fallback_encoding=fallback_encoding) with open(file, "r", encoding=fallback_encoding) as f: return f.read() def _cfg_read_utf8_with_fallback( cfg: RawConfigParser, file: str, fallback_encoding=LOCALE_ENCODING ) -> None: """Same idea as :func:`_read_utf8_with_fallback`, but for the :meth:`RawConfigParser.read` method. This method may call ``cfg.clear()``. """ try: cfg.read(file, encoding="utf-8") except UnicodeDecodeError: # pragma: no cover _Utf8EncodingNeeded.emit(file=file, fallback_encoding=fallback_encoding) cfg.clear() cfg.read(file, encoding=fallback_encoding) def unique_everseen(iterable, key=None): """Yield unique elements, preserving order. This function is copied from `more_itertools` 10.7.0. I did not want to add `more_itertools` as dependency for just this function. `more-itertools` is under the MIT License. >>> list(unique_everseen('AAAABBBCCDAABBB')) ['A', 'B', 'C', 'D'] >>> list(unique_everseen('ABBCcAD', str.lower)) ['A', 'B', 'C', 'D'] Sequences with a mix of hashable and unhashable items can be used. The function will be slower (i.e., `O(n^2)`) for unhashable items. Remember that ``list`` objects are unhashable - you can use the *key* parameter to transform the list to a tuple (which is hashable) to avoid a slowdown. >>> iterable = ([1, 2], [2, 3], [1, 2]) >>> list(unique_everseen(iterable)) # Slow [[1, 2], [2, 3]] >>> list(unique_everseen(iterable, key=tuple)) # Faster [[1, 2], [2, 3]] Similarly, you may want to convert unhashable ``set`` objects with ``key=frozenset``. For ``dict`` objects, ``key=lambda x: frozenset(x.items())`` can be used. """ seenset = set() seenset_add = seenset.add seenlist = [] seenlist_add = seenlist.append use_key = key is not None for element in iterable: k = key(element) if use_key else element try: if k not in seenset: seenset_add(k) yield element except TypeError: if k not in seenlist: seenlist_add(k) yield element def parse_requirement_arg(spec): try: return Requirement.parse(spec) except ValueError as e: raise DistutilsError( f"Not a URL, existing file, or requirement spec: {spec!r}" ) from e def parse_bdist_wininst(name): """Return (base,pyversion) or (None,None) for possible .exe name""" lower = name.lower() base, py_ver, plat = None, None, None if lower.endswith('.exe'): if lower.endswith('.win32.exe'): base = name[:-10] plat = 'win32' elif lower.startswith('.win32-py', -16): py_ver = name[-7:-4] base = name[:-16] plat = 'win32' elif lower.endswith('.win-amd64.exe'): base = name[:-14] plat = 'win-amd64' elif lower.startswith('.win-amd64-py', -20): py_ver = name[-7:-4] base = name[:-20] plat = 'win-amd64' return base, py_ver, plat def egg_info_for_url(url): parts = urllib.parse.urlparse(url) _scheme, server, path, _parameters, _query, fragment = parts base = urllib.parse.unquote(path.split('/')[-1]) if server == 'sourceforge.net' and base == 'download': # XXX Yuck base = urllib.parse.unquote(path.split('/')[-2]) if '#' in base: base, fragment = base.split('#', 1) return base, fragment def distros_for_url(url, metadata=None): """Yield egg or source distribution objects that might be found at a URL""" base, fragment = egg_info_for_url(url) yield from distros_for_location(url, base, metadata) if fragment: match = EGG_FRAGMENT.match(fragment) if match: yield from interpret_distro_name( url, match.group(1), metadata, precedence=CHECKOUT_DIST ) def distros_for_location(location, basename, metadata=None): """Yield egg or source distribution objects based on basename""" if basename.endswith('.egg.zip'): basename = basename[:-4] # strip the .zip if basename.endswith('.egg') and '-' in basename: # only one, unambiguous interpretation return [Distribution.from_location(location, basename, metadata)] if basename.endswith('.whl') and '-' in basename: wheel = Wheel(basename) if not wheel.is_compatible(): return [] return [ Distribution( location=location, project_name=wheel.project_name, version=wheel.version, # Increase priority over eggs. precedence=EGG_DIST + 1, ) ] if basename.endswith('.exe'): win_base, py_ver, platform = parse_bdist_wininst(basename) if win_base is not None: return interpret_distro_name( location, win_base, metadata, py_ver, BINARY_DIST, platform ) # Try source distro extensions (.zip, .tgz, etc.) # for ext in EXTENSIONS: if basename.endswith(ext): basename = basename[: -len(ext)] return interpret_distro_name(location, basename, metadata) return [] # no extension matched def distros_for_filename(filename, metadata=None): """Yield possible egg or source distribution objects based on a filename""" return distros_for_location( normalize_path(filename), os.path.basename(filename), metadata ) def interpret_distro_name( location, basename, metadata, py_version=None, precedence=SOURCE_DIST, platform=None ): """Generate the interpretation of a source distro name Note: if `location` is a filesystem filename, you should call ``pkg_resources.normalize_path()`` on it before passing it to this routine! """ parts = basename.split('-') if not py_version and any(re.match(r'py\d\.\d$', p) for p in parts[2:]): # it is a bdist_dumb, not an sdist -- bail out return # find the pivot (p) that splits the name from the version. # infer the version as the first item that has a digit. for p in range(len(parts)): if parts[p][:1].isdigit(): break else: p = len(parts) yield Distribution( location, metadata, '-'.join(parts[:p]), '-'.join(parts[p:]), py_version=py_version, precedence=precedence, platform=platform, ) def unique_values(func): """ Wrap a function returning an iterable such that the resulting iterable only ever yields unique items. """ @wraps(func) def wrapper(*args, **kwargs): return unique_everseen(func(*args, **kwargs)) return wrapper REL = re.compile(r"""<([^>]*\srel\s{0,10}=\s{0,10}['"]?([^'" >]+)[^>]*)>""", re.I) """ Regex for an HTML tag with 'rel="val"' attributes. """ @unique_values def find_external_links(url, page): """Find rel="homepage" and rel="download" links in `page`, yielding URLs""" for match in REL.finditer(page): tag, rel = match.groups() rels = set(map(str.strip, rel.lower().split(','))) if 'homepage' in rels or 'download' in rels: for match in HREF.finditer(tag): yield urllib.parse.urljoin(url, htmldecode(match.group(1))) for tag in ("Home Page", "Download URL"): pos = page.find(tag) if pos != -1: match = HREF.search(page, pos) if match: yield urllib.parse.urljoin(url, htmldecode(match.group(1))) class ContentChecker: """ A null content checker that defines the interface for checking content """ def feed(self, block): """ Feed a block of data to the hash. """ return def is_valid(self): """ Check the hash. Return False if validation fails. """ return True def report(self, reporter, template): """ Call reporter with information about the checker (hash name) substituted into the template. """ return class HashChecker(ContentChecker): pattern = re.compile( r'(?Psha1|sha224|sha384|sha256|sha512|md5)=' r'(?P[a-f0-9]+)' ) def __init__(self, hash_name, expected) -> None: self.hash_name = hash_name self.hash = hashlib.new(hash_name) self.expected = expected @classmethod def from_url(cls, url): "Construct a (possibly null) ContentChecker from a URL" fragment = urllib.parse.urlparse(url)[-1] if not fragment: return ContentChecker() match = cls.pattern.search(fragment) if not match: return ContentChecker() return cls(**match.groupdict()) def feed(self, block): self.hash.update(block) def is_valid(self): return self.hash.hexdigest() == self.expected def report(self, reporter, template): msg = template % self.hash_name return reporter(msg) class PackageIndex(Environment): """A distribution index that scans web pages for download URLs""" def __init__( self, index_url: str = "https://pypi.org/simple/", hosts=('*',), ca_bundle=None, verify_ssl: bool = True, *args, **kw, ) -> None: super().__init__(*args, **kw) self.index_url = index_url + "/"[: not index_url.endswith('/')] self.scanned_urls: dict = {} self.fetched_urls: dict = {} self.package_pages: dict = {} self.allows = re.compile('|'.join(map(translate, hosts))).match self.to_scan: list = [] self.opener = urllib.request.urlopen def add(self, dist): # ignore invalid versions try: parse_version(dist.version) except Exception: return None return super().add(dist) # FIXME: 'PackageIndex.process_url' is too complex (14) def process_url(self, url, retrieve: bool = False) -> None: # noqa: C901 """Evaluate a URL as a possible download, and maybe retrieve it BEWARE: this method is patched by zc.buildout in patches.py. I thought of merging that patch in here, but the patch uses the `pip._internal` module, so it could easily break in a newer pip. In patches.py we make sure that in that case the patch is simply not implied. If we would merge the patch, it could be that Buildout cannot even start. """ if url in self.scanned_urls and not retrieve: return self.scanned_urls[url] = True if not URL_SCHEME(url): self.process_filename(url) return else: dists = list(distros_for_url(url)) if dists: if not self.url_ok(url): return self.debug("Found link: %s", url) if dists or not retrieve or url in self.fetched_urls: list(map(self.add, dists)) return # don't need the actual page if not self.url_ok(url): self.fetched_urls[url] = True return self.info("Reading %s", url) self.fetched_urls[url] = True # prevent multiple fetch attempts tmpl = "Download error on %s: %%s -- Some packages may not be found!" f = self.open_url(url, tmpl % url) if f is None: return if isinstance(f, urllib.error.HTTPError) and f.code == 401: self.info(f"Authentication error: {f.msg}") self.fetched_urls[f.url] = True if 'html' not in f.headers.get('content-type', '').lower(): f.close() # not html, we can't process it return base = f.url # handle redirects page = f.read() if not isinstance(page, str): # In Python 3 and got bytes but want str. if isinstance(f, urllib.error.HTTPError): # Errors have no charset, assume latin1: charset = 'latin-1' else: charset = f.headers.get_param('charset') or 'latin-1' page = page.decode(charset, "ignore") f.close() for match in HREF.finditer(page): link = urllib.parse.urljoin(base, htmldecode(match.group(1))) self.process_url(link) if url.startswith(self.index_url) and getattr(f, 'code', None) != 404: page = self.process_index(url, page) def process_filename(self, fn, nested: bool = False) -> None: # process filenames or directories if not os.path.exists(fn): self.warn("Not found: %s", fn) return if os.path.isdir(fn) and not nested: path = os.path.realpath(fn) for item in os.listdir(path): self.process_filename(os.path.join(path, item), True) dists = distros_for_filename(fn) if dists: self.debug("Found: %s", fn) list(map(self.add, dists)) def url_ok(self, url, fatal: bool = False) -> bool: """Is this url allowed by the allow-hosts option? Buildout has changed this to not log any warning when a url is not allowed, as this warning would be unavoidable. """ s = URL_SCHEME(url) is_file = s and s.group(1).lower() == 'file' return is_file or self.allows(urllib.parse.urlparse(url)[1]) def scan_egg_links(self, search_path) -> None: dirs = filter(os.path.isdir, search_path) egg_links = ( (path, entry) for path in dirs for entry in os.listdir(path) if entry.endswith('.egg-link') ) list(itertools.starmap(self.scan_egg_link, egg_links)) def scan_egg_link(self, path, entry) -> None: content = _read_utf8_with_fallback(os.path.join(path, entry)) # filter non-empty lines lines = list(filter(None, map(str.strip, content.splitlines()))) if len(lines) != 2: # format is not recognized; punt return egg_path, _setup_path = lines for dist in find_distributions(os.path.join(path, egg_path)): dist.location = os.path.join(path, *lines) dist.precedence = SOURCE_DIST self.add(dist) def _scan(self, link): # Process a URL to see if it's for a package page NO_MATCH_SENTINEL = None, None if not link.startswith(self.index_url): return NO_MATCH_SENTINEL parts = list(map(urllib.parse.unquote, link[len(self.index_url) :].split('/'))) if len(parts) != 2 or '#' in parts[1]: return NO_MATCH_SENTINEL # it's a package page, sanitize and index it pkg = safe_name(parts[0]) ver = safe_version(parts[1]) self.package_pages.setdefault(pkg.lower(), {})[link] = True return to_filename(pkg), to_filename(ver) def process_index(self, url, page): """Process the contents of a PyPI page""" # process an index page into the package-page index for match in HREF.finditer(page): try: self._scan(urllib.parse.urljoin(url, htmldecode(match.group(1)))) except ValueError: pass pkg, ver = self._scan(url) # ensure this page is in the page index if not pkg: return "" # no sense double-scanning non-package pages # process individual package page for new_url in find_external_links(url, page): # Process the found URL base, frag = egg_info_for_url(new_url) if base.endswith('.py') and not frag: if ver: new_url += f'#egg={pkg}-{ver}' else: self.need_version_info(url) self.scan_url(new_url) return PYPI_MD5.sub( lambda m: '{}'.format(*m.group(1, 3, 2)), page ) def need_version_info(self, url) -> None: self.scan_all( "Page at %s links to .py file(s) without version info; an index " "scan is required.", url, ) def scan_all(self, msg=None, *args) -> None: if self.index_url not in self.fetched_urls: if msg: self.warn(msg, *args) self.info("Scanning index of all packages (this may take a while)") self.scan_url(self.index_url) def find_packages(self, requirement) -> None: """Find packages. Note: Buildout had a patch for this, but that code is now merged in here. The change implements PEP 503 for using canonical package names. """ url_name = re.sub(r"[-_.]+", "-", requirement.unsafe_name).lower() self.scan_url(self.index_url + url_name + '/') if self.package_pages.get(url_name): # We have found a package page and don't need try to any other # package pages for this package. for url in list(self.package_pages.get(url_name)): # scan each page that might be related to the desired package self.scan_url(url) return if not self.package_pages.get(requirement.key) and requirement.key != url_name: # Fall back to safe version of the name self.scan_url(self.index_url + requirement.project_name + '/') if not self.package_pages.get(requirement.key): # We couldn't find the target package, so search the index page too self.not_found_in_index(requirement) for url in list(self.package_pages.get(requirement.key, ())): # scan each page that might be related to the desired package self.scan_url(url) def obtain(self, requirement, installer=None): self.prescan() self.find_packages(requirement) for dist in self[requirement.key]: if dist in requirement: return dist self.debug("%s does not match %s", requirement, dist) return super().obtain(requirement, installer) def check_hash(self, checker, filename, tfp) -> None: """ checker is a ContentChecker """ checker.report(self.debug, f"Validating %s checksum for {filename}") if not checker.is_valid(): tfp.close() os.unlink(filename) raise DistutilsError( f"{checker.hash.name} validation failed for {os.path.basename(filename)}; " "possible download problem?" ) def add_find_links(self, urls) -> None: """Add `urls` to the list that will be prescanned for searches""" for url in urls: if ( self.to_scan is None # if we have already "gone online" or not URL_SCHEME(url) # or it's a local file/directory or url.startswith('file:') or list(distros_for_url(url)) # or a direct package link ): # then go ahead and process it now self.scan_url(url) else: # otherwise, defer retrieval till later self.to_scan.append(url) def prescan(self): """Scan urls scheduled for prescanning (e.g. --find-links)""" if self.to_scan: list(map(self.scan_url, self.to_scan)) self.to_scan = None # from now on, go ahead and process immediately def not_found_in_index(self, requirement) -> None: if self[requirement.key]: # we've seen at least one distro meth, msg = self.info, "Couldn't retrieve index page for %r" else: # no distros seen for this name, might be misspelled meth, msg = self.warn, "Couldn't find index page for %r (maybe misspelled?)" meth(msg, requirement.unsafe_name) self.scan_all() def download(self, spec, tmpdir): """Locate and/or download `spec` to `tmpdir`, returning a local path `spec` may be a ``Requirement`` object, or a string containing a URL, an existing local filename, or a project/version requirement spec (i.e. the string form of a ``Requirement`` object). If it is the URL of a .py file with an unambiguous ``#egg=name-version`` tag (i.e., one that escapes ``-`` as ``_`` throughout), a trivial ``setup.py`` is automatically created alongside the downloaded file. If `spec` is a ``Requirement`` object or a string containing a project/version requirement spec, this method returns the location of a matching distribution (possibly after downloading it to `tmpdir`). If `spec` is a locally existing file or directory name, it is simply returned unchanged. If `spec` is a URL, it is downloaded to a subpath of `tmpdir`, and the local filename is returned. Various errors may be raised if a problem occurs during downloading. """ if not isinstance(spec, Requirement): scheme = URL_SCHEME(spec) if scheme: # It's a url, download it to tmpdir found = self._download_url(spec, tmpdir) base, fragment = egg_info_for_url(spec) if base.endswith('.py'): found = self.gen_setup(found, fragment, tmpdir) return found elif os.path.exists(spec): # Existing file or directory, just return it return spec else: spec = parse_requirement_arg(spec) return getattr(self.fetch_distribution(spec, tmpdir), 'location', None) def fetch_distribution( # noqa: C901 # is too complex (14) # FIXME self, requirement, tmpdir, force_scan: bool = False, source: bool = False, develop_ok: bool = False, local_index=None, ) -> Distribution | None: """Obtain a distribution suitable for fulfilling `requirement` `requirement` must be a ``pkg_resources.Requirement`` instance. If necessary, or if the `force_scan` flag is set, the requirement is searched for in the (online) package index as well as the locally installed packages. If a distribution matching `requirement` is found, the returned distribution's ``location`` is the value you would have gotten from calling the ``download()`` method with the matching distribution's URL or filename. If no matching distribution is found, ``None`` is returned. If the `source` flag is set, only source distributions and source checkout links will be considered. Unless the `develop_ok` flag is set, development and system eggs (i.e., those using the ``.egg-info`` format) will be ignored. """ # process a Requirement self.info("Searching for %s", requirement) skipped = set() dist = None def find(req, env: Environment | None = None): if env is None: env = self # Find a matching distribution; may be called more than once for dist in env[req.key]: if dist.precedence == DEVELOP_DIST and not develop_ok: if dist not in skipped: self.warn( "Skipping development or system egg: %s", dist, ) skipped.add(dist) continue test = dist in req and (dist.precedence <= SOURCE_DIST or not source) if test: loc = self.download(dist.location, tmpdir) dist.download_location = loc if os.path.exists(dist.download_location): return dist return None if force_scan: self.prescan() self.find_packages(requirement) dist = find(requirement) if not dist and local_index is not None: dist = find(requirement, local_index) if dist is None: if self.to_scan is not None: self.prescan() dist = find(requirement) if dist is None and not force_scan: self.find_packages(requirement) dist = find(requirement) if dist is None: self.warn( "No local packages or working download links found for %s%s", (source and "a source distribution of " or ""), requirement, ) return None else: self.info("Best match: %s", dist) return dist.clone(location=dist.download_location) def fetch( self, requirement, tmpdir, force_scan: bool = False, source: bool = False ) -> str | None: """Obtain a file suitable for fulfilling `requirement` DEPRECATED; use the ``fetch_distribution()`` method now instead. For backward compatibility, this routine is identical but returns the ``location`` of the downloaded distribution instead of a distribution object. """ dist = self.fetch_distribution(requirement, tmpdir, force_scan, source) if dist is not None: return dist.location return None def gen_setup(self, filename, fragment, tmpdir): match = EGG_FRAGMENT.match(fragment) dists = ( match and [ d for d in interpret_distro_name(filename, match.group(1), None) if d.version ] or [] ) if len(dists) == 1: # unambiguous ``#egg`` fragment basename = os.path.basename(filename) # Make sure the file has been downloaded to the temp dir. if os.path.dirname(filename) != tmpdir: dst = os.path.join(tmpdir, basename) if not (os.path.exists(dst) and os.path.samefile(filename, dst)): shutil.copy2(filename, dst) filename = dst with open(os.path.join(tmpdir, 'setup.py'), 'w', encoding="utf-8") as file: file.write( "from setuptools import setup\n" f"setup(name={dists[0].project_name!r}, version={dists[0].version!r}, py_modules=[{os.path.splitext(basename)[0]!r}])\n" ) return filename elif match: raise DistutilsError( f"Can't unambiguously interpret project/version identifier {fragment!r}; " "any dashes in the name or version should be escaped using " f"underscores. {dists!r}" ) else: raise DistutilsError( "Can't process plain .py files without an '#egg=name-version'" " suffix to enable automatic setup script generation." ) dl_blocksize = 8192 def _download_to(self, url, filename): self.info("Downloading %s", url) # Download the file fp = None try: checker = HashChecker.from_url(url) fp = self.open_url(url) if isinstance(fp, urllib.error.HTTPError): raise DistutilsError(f"Can't download {url}: {fp.code} {fp.msg}") headers = fp.info() blocknum = 0 bs = self.dl_blocksize size = -1 if "content-length" in headers: # Some servers return multiple Content-Length headers :( sizes = headers.get_all('Content-Length') size = max(map(int, sizes)) self.reporthook(url, filename, blocknum, bs, size) with open(filename, 'wb') as tfp: while True: block = fp.read(bs) if block: checker.feed(block) tfp.write(block) blocknum += 1 self.reporthook(url, filename, blocknum, bs, size) else: break self.check_hash(checker, filename, tfp) return headers finally: if fp: fp.close() def reporthook(self, url, filename, blocknum, blksize, size) -> None: pass # no-op # FIXME: def open_url(self, url, warning=None): # noqa: C901 # is too complex (12) if url.startswith('file:'): return local_open(url) try: return open_with_auth(url, self.opener) except (ValueError, http.client.InvalidURL) as v: msg = ' '.join([str(arg) for arg in v.args]) if warning: self.warn(warning, msg) else: raise DistutilsError(f'{url} {msg}') from v except urllib.error.HTTPError as v: return v except urllib.error.URLError as v: if warning: self.warn(warning, v.reason) else: raise DistutilsError(f"Download error for {url}: {v.reason}") from v except http.client.BadStatusLine as v: if warning: self.warn(warning, v.line) else: raise DistutilsError( f'{url} returned a bad status line. The server might be ' f'down, {v.line}' ) from v except (http.client.HTTPException, OSError) as v: if warning: self.warn(warning, v) else: raise DistutilsError(f"Download error for {url}: {v}") from v @staticmethod def _sanitize(name): r""" Replace unsafe path directives with underscores. >>> san = PackageIndex._sanitize >>> san('/home/user/.ssh/authorized_keys') '_home_user_.ssh_authorized_keys' >>> san('..\\foo\\bing') '__foo_bing' >>> san('D:bar') 'D_bar' >>> san('C:\\bar') 'C__bar' >>> san('foo..bar') 'foo..bar' >>> san('D:../foo') 'D___foo' """ pattern = '|'.join(( # drive letters r':', # path separators r'[/\\]', # parent dirs r'(?:(?<=([/\\]|:))\.\.(?=[/\\]|$))|(?:^\.\.(?=[/\\]|$))', )) return re.sub(pattern, r'_', name) @classmethod def _resolve_download_filename(cls, url, tmpdir): """ >>> import pathlib >>> du = PackageIndex._resolve_download_filename >>> root = getfixture('tmp_path') >>> url = 'https://files.pythonhosted.org/packages/a9/5a/0db.../setuptools-78.1.0.tar.gz' >>> str(pathlib.Path(du(url, root)).relative_to(root)) 'setuptools-78.1.0.tar.gz' """ name, _fragment = egg_info_for_url(url) # Sanitize, and use a default if URL has no path contents name = cls._sanitize(name or '__downloaded__') # strip any extra .zip before download name = re.sub(r'\.egg\.zip$', '.egg', name) return os.path.join(tmpdir, name) def _download_url(self, url, tmpdir): """ Determine the download filename. """ filename = self._resolve_download_filename(url, tmpdir) return self._download_vcs(url, filename) or self._download_other(url, filename) @staticmethod def _resolve_vcs(url): """ >>> rvcs = PackageIndex._resolve_vcs >>> rvcs('git+http://foo/bar') 'git' >>> rvcs('hg+https://foo/bar') 'hg' >>> rvcs('git:myhost') 'git' >>> rvcs('hg:myhost') >>> rvcs('http://foo/bar') """ scheme = urllib.parse.urlsplit(url).scheme pre, sep, _post = scheme.partition('+') # svn and git have their own protocol; hg does not allowed = set(['svn', 'git'] + ['hg'] * bool(sep)) return next(iter({pre} & allowed), None) def _download_vcs(self, url, spec_filename): vcs = self._resolve_vcs(url) if not vcs: return None if vcs == 'svn': raise DistutilsError( f"Invalid config, SVN download is not supported: {url}" ) filename, _, _ = spec_filename.partition('#') url, rev = self._vcs_split_rev_from_url(url) self.info(f"Doing {vcs} clone from {url} to {filename}") subprocess.check_call([vcs, 'clone', '--quiet', url, filename]) co_commands = dict( git=[vcs, '-C', filename, 'checkout', '--quiet', rev], hg=[vcs, '--cwd', filename, 'up', '-C', '-r', rev, '-q'], ) if rev is not None: self.info(f"Checking out {rev}") subprocess.check_call(co_commands[vcs]) return filename def _download_other(self, url, filename): scheme = urllib.parse.urlsplit(url).scheme if scheme == 'file': # pragma: no cover return urllib.request.url2pathname(urllib.parse.urlparse(url).path) # raise error if not allowed self.url_ok(url, True) return self._attempt_download(url, filename) def scan_url(self, url) -> None: self.process_url(url, True) def _attempt_download(self, url, filename): headers = self._download_to(url, filename) if 'html' in headers.get('content-type', '').lower(): return self._invalid_download_html(url, headers, filename) else: return filename def _invalid_download_html(self, url, headers, filename): os.unlink(filename) raise DistutilsError(f"Unexpected HTML page found at {url}") @staticmethod def _vcs_split_rev_from_url(url): """ Given a possible VCS URL, return a clean URL and resolved revision if any. >>> vsrfu = PackageIndex._vcs_split_rev_from_url >>> vsrfu('git+https://github.com/pypa/setuptools@v69.0.0#egg-info=setuptools') ('https://github.com/pypa/setuptools', 'v69.0.0') >>> vsrfu('git+https://github.com/pypa/setuptools#egg-info=setuptools') ('https://github.com/pypa/setuptools', None) >>> vsrfu('http://foo/bar') ('http://foo/bar', None) """ parts = urllib.parse.urlsplit(url) clean_scheme = parts.scheme.split('+', 1)[-1] # Some fragment identification fails no_fragment_path, _, _ = parts.path.partition('#') pre, sep, post = no_fragment_path.rpartition('@') clean_path, rev = (pre, post) if sep else (post, None) resolved = parts._replace( scheme=clean_scheme, path=clean_path, # discard the fragment fragment='', ).geturl() return resolved, rev def debug(self, msg, *args) -> None: log.debug(msg, *args) def info(self, msg, *args) -> None: log.info(msg, *args) def warn(self, msg, *args) -> None: log.warn(msg, *args) # This pattern matches a character entity reference (a decimal numeric # references, a hexadecimal numeric reference, or a named reference). entity_sub = re.compile(r'&(#(\d+|x[\da-fA-F]+)|[\w.:-]+);?').sub def decode_entity(match): what = match.group(0) return html.unescape(what) def htmldecode(text): """ Decode HTML entities in the given text. >>> htmldecode( ... 'https://../package_name-0.1.2.tar.gz' ... '?tokena=A&tokenb=B">package_name-0.1.2.tar.gz') 'https://../package_name-0.1.2.tar.gz?tokena=A&tokenb=B">package_name-0.1.2.tar.gz' """ return entity_sub(decode_entity, text) def socket_timeout(timeout=15): def _socket_timeout(func): def _socket_timeout(*args, **kwargs): old_timeout = socket.getdefaulttimeout() socket.setdefaulttimeout(timeout) try: return func(*args, **kwargs) finally: socket.setdefaulttimeout(old_timeout) return _socket_timeout return _socket_timeout def _encode_auth(auth): """ Encode auth from a URL suitable for an HTTP header. >>> str(_encode_auth('username%3Apassword')) 'dXNlcm5hbWU6cGFzc3dvcmQ=' Long auth strings should not cause a newline to be inserted. >>> long_auth = 'username:' + 'password'*10 >>> chr(10) in str(_encode_auth(long_auth)) False """ auth_s = urllib.parse.unquote(auth) # convert to bytes auth_bytes = auth_s.encode() encoded_bytes = base64.b64encode(auth_bytes) # convert back to a string encoded = encoded_bytes.decode() # strip the trailing carriage return return encoded.replace('\n', '') class Credential(NamedTuple): """ A username/password pair. Displayed separated by `:`. >>> str(Credential('username', 'password')) 'username:password' """ username: str password: str def __str__(self) -> str: return f'{self.username}:{self.password}' class PyPIConfig(configparser.RawConfigParser): def __init__(self): """ Load from ~/.pypirc """ defaults = dict.fromkeys(['username', 'password', 'repository'], '') super().__init__(defaults) rc = os.path.join(os.path.expanduser('~'), '.pypirc') if os.path.exists(rc): _cfg_read_utf8_with_fallback(self, rc) @property def creds_by_repository(self): sections_with_repositories = [ section for section in self.sections() if self.get(section, 'repository').strip() ] return dict(map(self._get_repo_cred, sections_with_repositories)) def _get_repo_cred(self, section): repo = self.get(section, 'repository').strip() return repo, Credential( self.get(section, 'username').strip(), self.get(section, 'password').strip(), ) def find_credential(self, url): """ If the URL indicated appears to be a repository defined in this config, return the credential for that repository. """ for repository, cred in self.creds_by_repository.items(): if url.startswith(repository): return cred return None def open_with_auth(url, opener=urllib.request.urlopen): """Open a urllib2 request, handling HTTP authentication""" parsed = urllib.parse.urlparse(url) scheme, netloc, path, params, query, frag = parsed # Double scheme does not raise on macOS as revealed by a # failing test. We would expect "nonnumeric port". Refs #20. if netloc.endswith(':'): raise http.client.InvalidURL("nonnumeric port: ''") if scheme in ('http', 'https'): auth, address = _splituser(netloc) else: auth, address = (None, None) if not auth: cred = PyPIConfig().find_credential(url) if cred: auth = str(cred) info = cred.username, url log.info('Authenticating as %s for %s (from .pypirc)', *info) if auth: auth = "Basic " + _encode_auth(auth) parts = scheme, address, path, params, query, frag new_url = urllib.parse.urlunparse(parts) request = urllib.request.Request(new_url) request.add_header("Authorization", auth) else: request = urllib.request.Request(url) request.add_header('User-Agent', user_agent) fp = opener(request) if auth: # Put authentication info back into request URL if same host, # so that links found on the page will work s2, h2, path2, param2, query2, frag2 = urllib.parse.urlparse(fp.url) if s2 == scheme and h2 == address: parts = s2, netloc, path2, param2, query2, frag2 fp.url = urllib.parse.urlunparse(parts) return fp # copy of urllib.parse._splituser from Python 3.8 # See https://github.com/python/cpython/issues/80072. def _splituser(host): """splituser('user[:passwd]@host[:port]') --> 'user[:passwd]', 'host[:port]'.""" user, delim, host = host.rpartition('@') return (user if delim else None), host # adding a timeout to avoid freezing package_index open_with_auth = socket_timeout(_SOCKET_TIMEOUT)(open_with_auth) def fix_sf_url(url): return url # backward compatibility def local_open(url): """Read a local path, with special support for directories""" _scheme, _server, path, _param, _query, _frag = urllib.parse.urlparse(url) filename = urllib.request.url2pathname(path) if os.path.isfile(filename): return urllib.request.urlopen(url) elif path.endswith('/') and os.path.isdir(filename): files = [] for f in os.listdir(filename): filepath = os.path.join(filename, f) if f == 'index.html': body = _read_utf8_with_fallback(filepath) break elif os.path.isdir(filepath): f += '/' files.append(f'{f}') else: tmpl = "{url}{files}" body = tmpl.format(url=url, files='\n'.join(files)) status, message = 200, "OK" else: status, message, body = 404, "Path not found", "Not found" headers = {'content-type': 'text/html'} body_stream = io.StringIO(body) return urllib.error.HTTPError(url, status, message, headers, body_stream) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2105446 zc_buildout-6.0.1/src/zc/buildout/_vendor/0000755000076500000240000000000015264141651020143 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3995047 zc_buildout-6.0.1/src/zc/buildout/_vendor/README.rst0000644000076500000240000000614715264141650021641 0ustar00mauritsstaffVendored code ============= This directory contains third-party code that zc.buildout ships itself. ``pkg_resources`` ----------------- ``pkg_resources/__init__.py`` is a copy of https://github.com/pypa/setuptools/blob/v81.0.0/pkg_resources/__init__.py — setuptools v81.0.0 is the last release that contained ``pkg_resources``; it was removed in setuptools 82.0.0. ``zc.buildout`` (and many recipes and extensions) use the ``pkg_resources`` API as their data model for distributions, requirements and working sets, so buildout ships its own copy and installs it as ``sys.modules ['pkg_resources']`` early in ``zc/buildout/__init__.py``. See https://github.com/buildout/buildout/issues/685 for the background. Note that this copy is only importable *inside* a buildout run (or after ``import zc.buildout``). Scripts generated by buildout for distributions that use pkg_resources-style namespace packages still need a real ``pkg_resources`` (i.e. setuptools < 82) at script run time. Deviations from upstream ~~~~~~~~~~~~~~~~~~~~~~~~ Each change is marked with a ``LOCAL CHANGE`` comment in the file: 1. A provenance and license note was added to the module docstring. 2. The ``sys.path`` extension with setuptools' private ``_vendor`` directory and the ``sys.modules.pop('backports', None)`` workaround were removed: the dependencies they made importable are provided differently here (see 3. and 4.), and vendored code must not mutate the host process' ``sys.path``/``sys.modules``. 3. ``from jaraco.text import ...`` was replaced by an import from ``zc.buildout._vendor.jaraco_text`` (see below). 4. (No source change) ``platformdirs`` is imported unchanged; it is a regular install requirement of zc.buildout instead of being vendored, because it is pure Python and has no dependencies of its own. Upstream's ``pkg_resources/tests/``, ``api_tests.txt`` and ``py.typed`` are not vendored: the first two are test-only, and the ``sys.modules`` alias is invisible to static type checkers anyway, so shipping type information would not help. ``jaraco_text.py`` ------------------ Contains the three helper functions (and their private helpers) that ``pkg_resources`` needs from ``jaraco.text``, copied unchanged from ``jaraco.text`` 4.0.0 as vendored by setuptools v81.0.0 (``setuptools/_vendor/jaraco/text/__init__.py``). Vendoring all of ``jaraco.text`` — or depending on it — would drag in its dependency tree (jaraco.functools, more-itertools, ...) for three small functions. Re-sync procedure ----------------- setuptools v81.0.0 is the final upstream release of ``pkg_resources``, so re-syncs from upstream are not expected; future changes will be local patches, which should be marked with ``LOCAL CHANGE`` comments and listed above. Should a re-sync ever be needed: 1. Download ``pkg_resources/__init__.py`` from the desired tag, e.g. https://raw.githubusercontent.com/pypa/setuptools/v81.0.0/pkg_resources/__init__.py 2. Re-apply the deviations listed above (diff against the current vendored copy to find them: look for ``LOCAL CHANGE``). 3. Update the version references in this file and in the module docstring. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3995576 zc_buildout-6.0.1/src/zc/buildout/_vendor/__init__.py0000644000076500000240000000017615264141650022257 0ustar00mauritsstaff"""Vendored third-party code used by zc.buildout. See README.rst in this directory for provenance and maintenance notes. """ ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.399615 zc_buildout-6.0.1/src/zc/buildout/_vendor/jaraco_text.py0000644000076500000240000000655415264141650023031 0ustar00mauritsstaff"""Text handling helpers needed by the vendored ``pkg_resources``. This file contains the functions ``yield_lines``, ``drop_comment`` and ``join_continuation`` (plus their helpers) copied unchanged from ``jaraco.text`` 4.0.0, as vendored by setuptools v81.0.0 in ``setuptools/_vendor/jaraco/text/__init__.py``. jaraco.text has the following license statement. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """ import functools import itertools from collections.abc import Iterable def _nonblank(str): return str and not str.startswith('#') @functools.singledispatch def yield_lines(iterable): r""" Yield valid lines of a string or iterable. >>> list(yield_lines('')) [] >>> list(yield_lines(['foo', 'bar'])) ['foo', 'bar'] >>> list(yield_lines('foo\nbar')) ['foo', 'bar'] >>> list(yield_lines('\nfoo\n#bar\nbaz #comment')) ['foo', 'baz #comment'] >>> list(yield_lines(['foo\nbar', 'baz', 'bing\n\n\n'])) ['foo', 'bar', 'baz', 'bing'] """ return itertools.chain.from_iterable(map(yield_lines, iterable)) @yield_lines.register(str) def _(text): return clean(text.splitlines()) def clean(lines: Iterable[str]): """ Yield non-blank, non-comment elements from lines. """ return filter(_nonblank, map(str.strip, lines)) def drop_comment(line): """ Drop comments. >>> drop_comment('foo # bar') 'foo' A hash without a space may be in a URL. >>> drop_comment('http://example.com/foo#bar') 'http://example.com/foo#bar' """ return line.partition(' #')[0] def join_continuation(lines): r""" Join lines continued by a trailing backslash. >>> list(join_continuation(['foo \\', 'bar', 'baz'])) ['foobar', 'baz'] >>> list(join_continuation(['foo \\', 'bar', 'baz'])) ['foobar', 'baz'] >>> list(join_continuation(['foo \\', 'bar \\', 'baz'])) ['foobarbaz'] Not sure why, but... The character preceding the backslash is also elided. >>> list(join_continuation(['goo\\', 'dly'])) ['godly'] A terrible idea, but... If no line is available to continue, suppress the lines. >>> list(join_continuation(['foo', 'bar\\', 'baz\\'])) ['foo'] """ lines = iter(lines) for item in lines: while item.endswith('\\'): try: item = item[:-2].strip() + next(lines) except StopIteration: return yield item ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2106397 zc_buildout-6.0.1/src/zc/buildout/_vendor/pkg_resources/0000755000076500000240000000000015264141651023016 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.3998432 zc_buildout-6.0.1/src/zc/buildout/_vendor/pkg_resources/__init__.py0000644000076500000240000037226615264141650025146 0ustar00mauritsstaff""" Package resource API -------------------- A resource is a logical file contained within a package, or a logical subdirectory thereof. The package resource API expects resource names to have their path parts separated with ``/``, *not* whatever the local path separator is. Do not use os.path operations to manipulate resource names being passed into the API. The package resource API is designed to work with normal filesystem packages, .egg files, and unpacked .egg files. It can also work in a limited way with .zip files and with custom PEP 302 loaders that support the ``get_data()`` method. This module is deprecated. Users are directed to :mod:`importlib.resources`, :mod:`importlib.metadata` and :pypi:`packaging` instead. --- LOCAL CHANGE --- This file was copied from setuptools v81.0.0 (the last release containing ``pkg_resources``), the file ``pkg_resources/__init__.py``, with minor changes. Each change is marked with a ``LOCAL CHANGE`` comment; see ``src/zc/buildout/_vendor/README.rst`` for the list of changes and the re-sync procedure. setuptools has the following license statement. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --- END LOCAL CHANGE --- """ from __future__ import annotations import sys if sys.version_info < (3, 9): # noqa: UP036 # Check for unsupported versions raise RuntimeError("Python 3.9 or later is required") import _imp import collections import email.parser import errno import functools import importlib import importlib.abc import importlib.machinery import inspect import io import ntpath import operator import os import pkgutil import platform import plistlib import posixpath import re import stat import tempfile import textwrap import time import types import warnings import zipfile import zipimport from collections.abc import Iterable, Iterator, Mapping, MutableSequence from pkgutil import get_importer from typing import ( TYPE_CHECKING, Any, BinaryIO, Callable, Literal, NamedTuple, NoReturn, Protocol, TypeVar, Union, overload, ) # --- LOCAL CHANGE --- # Upstream extends sys.path with setuptools' private ``_vendor`` directory # here (to make jaraco.text and platformdirs importable) and pops a # possibly half-imported ``backports`` module (workaround for # pypa/setuptools#4476). Neither is wanted when running vendored inside # zc.buildout: platformdirs is a regular dependency of zc.buildout and the # jaraco.text helpers are vendored alongside this file, so we must not # touch the host process' sys.path or sys.modules. # --- END LOCAL CHANGE --- # capture these to bypass sandboxing from os import open as os_open, utime # isort: skip from os.path import isdir, split # isort: skip try: from os import mkdir, rename, unlink WRITE_SUPPORT = True except ImportError: # no write support, probably under GAE WRITE_SUPPORT = False import packaging.markers import packaging.requirements import packaging.specifiers import packaging.utils import packaging.version # --- LOCAL CHANGE --- # Upstream imports these helpers from jaraco.text; zc.buildout vendors # them next to this file to avoid depending on jaraco.text. from zc.buildout._vendor.jaraco_text import ( drop_comment, join_continuation, yield_lines, ) # --- END LOCAL CHANGE --- from platformdirs import user_cache_dir as _user_cache_dir if TYPE_CHECKING: from _typeshed import BytesPath, StrOrBytesPath, StrPath from _typeshed.importlib import LoaderProtocol from typing_extensions import Self, TypeAlias warnings.warn( "pkg_resources is deprecated as an API. " "See https://setuptools.pypa.io/en/latest/pkg_resources.html. " "The pkg_resources package is slated for removal as early as " "2025-11-30. Refrain from using this package or pin to " "Setuptools<81.", UserWarning, stacklevel=2, ) _T = TypeVar("_T") _DistributionT = TypeVar("_DistributionT", bound="Distribution") # Type aliases _NestedStr: TypeAlias = Union[str, Iterable[Union[str, Iterable["_NestedStr"]]]] _StrictInstallerType: TypeAlias = Callable[["Requirement"], "_DistributionT"] _InstallerType: TypeAlias = Callable[["Requirement"], Union["Distribution", None]] _PkgReqType: TypeAlias = Union[str, "Requirement"] _EPDistType: TypeAlias = Union["Distribution", _PkgReqType] _MetadataType: TypeAlias = Union["IResourceProvider", None] _ResolvedEntryPoint: TypeAlias = Any # Can be any attribute in the module _ResourceStream: TypeAlias = Any # TODO / Incomplete: A readable file-like object # Any object works, but let's indicate we expect something like a module (optionally has __loader__ or __file__) _ModuleLike: TypeAlias = Union[object, types.ModuleType] # Any: Should be _ModuleLike but we end up with issues where _ModuleLike doesn't have _ZipLoaderModule's __loader__ _ProviderFactoryType: TypeAlias = Callable[[Any], "IResourceProvider"] _DistFinderType: TypeAlias = Callable[[_T, str, bool], Iterable["Distribution"]] _NSHandlerType: TypeAlias = Callable[[_T, str, str, types.ModuleType], Union[str, None]] _AdapterT = TypeVar( "_AdapterT", _DistFinderType[Any], _ProviderFactoryType, _NSHandlerType[Any] ) class _ZipLoaderModule(Protocol): __loader__: zipimport.zipimporter _PEP440_FALLBACK = re.compile( r"^v?(?P(?:[0-9]+!)?[0-9]+(?:\.[0-9]+)*)", re.IGNORECASE ) class PEP440Warning(RuntimeWarning): """ Used when there is an issue with a version or specifier not complying with PEP 440. """ parse_version = packaging.version.Version _state_vars: dict[str, str] = {} def _declare_state(vartype: str, varname: str, initial_value: _T) -> _T: _state_vars[varname] = vartype return initial_value def __getstate__() -> dict[str, Any]: state = {} g = globals() for k, v in _state_vars.items(): state[k] = g['_sget_' + v](g[k]) return state def __setstate__(state: dict[str, Any]) -> dict[str, Any]: g = globals() for k, v in state.items(): g['_sset_' + _state_vars[k]](k, g[k], v) return state def _sget_dict(val): return val.copy() def _sset_dict(key, ob, state) -> None: ob.clear() ob.update(state) def _sget_object(val): return val.__getstate__() def _sset_object(key, ob, state) -> None: ob.__setstate__(state) _sget_none = _sset_none = lambda *args: None def get_supported_platform(): """Return this platform's maximum compatible version. distutils.util.get_platform() normally reports the minimum version of macOS that would be required to *use* extensions produced by distutils. But what we want when checking compatibility is to know the version of macOS that we are *running*. To allow usage of packages that explicitly require a newer version of macOS, we must also know the current version of the OS. If this condition occurs for any other platform with a version in its platform strings, this function should be extended accordingly. """ plat = get_build_platform() m = macosVersionString.match(plat) if m is not None and sys.platform == "darwin": try: major_minor = '.'.join(_macos_vers()[:2]) build = m.group(3) plat = f'macosx-{major_minor}-{build}' except ValueError: # not macOS pass return plat __all__ = [ # Basic resource access and distribution/entry point discovery 'require', 'run_script', 'get_provider', 'get_distribution', 'load_entry_point', 'get_entry_map', 'get_entry_info', 'iter_entry_points', 'resource_string', 'resource_stream', 'resource_filename', 'resource_listdir', 'resource_exists', 'resource_isdir', # Environmental control 'declare_namespace', 'working_set', 'add_activation_listener', 'find_distributions', 'set_extraction_path', 'cleanup_resources', 'get_default_cache', # Primary implementation classes 'Environment', 'WorkingSet', 'ResourceManager', 'Distribution', 'Requirement', 'EntryPoint', # Exceptions 'ResolutionError', 'VersionConflict', 'DistributionNotFound', 'UnknownExtra', 'ExtractionError', # Warnings 'PEP440Warning', # Parsing functions and string utilities 'parse_requirements', 'parse_version', 'safe_name', 'safe_version', 'get_platform', 'compatible_platforms', 'yield_lines', 'split_sections', 'safe_extra', 'to_filename', 'invalid_marker', 'evaluate_marker', # filesystem utilities 'ensure_directory', 'normalize_path', # Distribution "precedence" constants 'EGG_DIST', 'BINARY_DIST', 'SOURCE_DIST', 'CHECKOUT_DIST', 'DEVELOP_DIST', # "Provider" interfaces, implementations, and registration/lookup APIs 'IMetadataProvider', 'IResourceProvider', 'FileMetadata', 'PathMetadata', 'EggMetadata', 'EmptyProvider', 'empty_provider', 'NullProvider', 'EggProvider', 'DefaultProvider', 'ZipProvider', 'register_finder', 'register_namespace_handler', 'register_loader_type', 'fixup_namespace_packages', 'get_importer', # Warnings 'PkgResourcesDeprecationWarning', # Deprecated/backward compatibility only 'run_main', 'AvailableDistributions', ] class ResolutionError(Exception): """Abstract base for dependency resolution errors""" def __repr__(self) -> str: return self.__class__.__name__ + repr(self.args) class VersionConflict(ResolutionError): """ An already-installed version conflicts with the requested version. Should be initialized with the installed Distribution and the requested Requirement. """ _template = "{self.dist} is installed but {self.req} is required" @property def dist(self) -> Distribution: return self.args[0] @property def req(self) -> Requirement: return self.args[1] def report(self): return self._template.format(**locals()) def with_context( self, required_by: set[Distribution | str] ) -> Self | ContextualVersionConflict: """ If required_by is non-empty, return a version of self that is a ContextualVersionConflict. """ if not required_by: return self args = self.args + (required_by,) return ContextualVersionConflict(*args) class ContextualVersionConflict(VersionConflict): """ A VersionConflict that accepts a third parameter, the set of the requirements that required the installed Distribution. """ _template = VersionConflict._template + ' by {self.required_by}' @property def required_by(self) -> set[str]: return self.args[2] class DistributionNotFound(ResolutionError): """A requested distribution was not found""" _template = ( "The '{self.req}' distribution was not found " "and is required by {self.requirers_str}" ) @property def req(self) -> Requirement: return self.args[0] @property def requirers(self) -> set[str] | None: return self.args[1] @property def requirers_str(self): if not self.requirers: return 'the application' return ', '.join(self.requirers) def report(self): return self._template.format(**locals()) def __str__(self) -> str: return self.report() class UnknownExtra(ResolutionError): """Distribution doesn't have an "extra feature" of the given name""" _provider_factories: dict[type[_ModuleLike], _ProviderFactoryType] = {} PY_MAJOR = f'{sys.version_info.major}.{sys.version_info.minor}' EGG_DIST = 3 BINARY_DIST = 2 SOURCE_DIST = 1 CHECKOUT_DIST = 0 DEVELOP_DIST = -1 def register_loader_type( loader_type: type[_ModuleLike], provider_factory: _ProviderFactoryType ) -> None: """Register `provider_factory` to make providers for `loader_type` `loader_type` is the type or class of a PEP 302 ``module.__loader__``, and `provider_factory` is a function that, passed a *module* object, returns an ``IResourceProvider`` for that module. """ _provider_factories[loader_type] = provider_factory @overload def get_provider(moduleOrReq: str) -> IResourceProvider: ... @overload def get_provider(moduleOrReq: Requirement) -> Distribution: ... def get_provider(moduleOrReq: str | Requirement) -> IResourceProvider | Distribution: """Return an IResourceProvider for the named module or requirement""" if isinstance(moduleOrReq, Requirement): return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0] try: module = sys.modules[moduleOrReq] except KeyError: __import__(moduleOrReq) module = sys.modules[moduleOrReq] loader = getattr(module, '__loader__', None) return _find_adapter(_provider_factories, loader)(module) @functools.cache def _macos_vers(): version = platform.mac_ver()[0] # fallback for MacPorts if version == '': plist = '/System/Library/CoreServices/SystemVersion.plist' if os.path.exists(plist): with open(plist, 'rb') as fh: plist_content = plistlib.load(fh) if 'ProductVersion' in plist_content: version = plist_content['ProductVersion'] return version.split('.') def _macos_arch(machine): return {'PowerPC': 'ppc', 'Power_Macintosh': 'ppc'}.get(machine, machine) def get_build_platform(): """Return this platform's string for platform-specific distributions""" from sysconfig import get_platform plat = get_platform() if sys.platform == "darwin" and not plat.startswith('macosx-'): try: version = _macos_vers() machine = _macos_arch(os.uname()[4].replace(" ", "_")) return f"macosx-{version[0]}.{version[1]}-{machine}" except ValueError: # if someone is running a non-Mac darwin system, this will fall # through to the default implementation pass return plat macosVersionString = re.compile(r"macosx-(\d+)\.(\d+)-(.*)") darwinVersionString = re.compile(r"darwin-(\d+)\.(\d+)\.(\d+)-(.*)") # XXX backward compat get_platform = get_build_platform def compatible_platforms(provided: str | None, required: str | None) -> bool: """Can code for the `provided` platform run on the `required` platform? Returns true if either platform is ``None``, or the platforms are equal. XXX Needs compatibility checks for Linux and other unixy OSes. """ if provided is None or required is None or provided == required: # easy case return True # macOS special cases reqMac = macosVersionString.match(required) if reqMac: provMac = macosVersionString.match(provided) # is this a Mac package? if not provMac: # this is backwards compatibility for packages built before # setuptools 0.6. All packages built after this point will # use the new macOS designation. provDarwin = darwinVersionString.match(provided) if provDarwin: dversion = int(provDarwin.group(1)) macosversion = f"{reqMac.group(1)}.{reqMac.group(2)}" if ( dversion == 7 and macosversion >= "10.3" or dversion == 8 and macosversion >= "10.4" ): return True # egg isn't macOS or legacy darwin return False # are they the same major version and machine type? if provMac.group(1) != reqMac.group(1) or provMac.group(3) != reqMac.group(3): return False # is the required OS major update >= the provided one? if int(provMac.group(2)) > int(reqMac.group(2)): return False return True # XXX Linux and other platforms' special cases should go here return False @overload def get_distribution(dist: _DistributionT) -> _DistributionT: ... @overload def get_distribution(dist: _PkgReqType) -> Distribution: ... def get_distribution(dist: Distribution | _PkgReqType) -> Distribution: """Return a current distribution object for a Requirement or string""" if isinstance(dist, str): dist = Requirement.parse(dist) if isinstance(dist, Requirement): dist = get_provider(dist) if not isinstance(dist, Distribution): raise TypeError("Expected str, Requirement, or Distribution", dist) return dist def load_entry_point(dist: _EPDistType, group: str, name: str) -> _ResolvedEntryPoint: """Return `name` entry point of `group` for `dist` or raise ImportError""" return get_distribution(dist).load_entry_point(group, name) @overload def get_entry_map( dist: _EPDistType, group: None = None ) -> dict[str, dict[str, EntryPoint]]: ... @overload def get_entry_map(dist: _EPDistType, group: str) -> dict[str, EntryPoint]: ... def get_entry_map(dist: _EPDistType, group: str | None = None): """Return the entry point map for `group`, or the full entry map""" return get_distribution(dist).get_entry_map(group) def get_entry_info(dist: _EPDistType, group: str, name: str) -> EntryPoint | None: """Return the EntryPoint object for `group`+`name`, or ``None``""" return get_distribution(dist).get_entry_info(group, name) class IMetadataProvider(Protocol): def has_metadata(self, name: str) -> bool: """Does the package's distribution contain the named metadata?""" ... def get_metadata(self, name: str) -> str: """The named metadata resource as a string""" ... def get_metadata_lines(self, name: str) -> Iterator[str]: """Yield named metadata resource as list of non-blank non-comment lines Leading and trailing whitespace is stripped from each line, and lines with ``#`` as the first non-blank character are omitted.""" ... def metadata_isdir(self, name: str) -> bool: """Is the named metadata a directory? (like ``os.path.isdir()``)""" ... def metadata_listdir(self, name: str) -> list[str]: """List of metadata names in the directory (like ``os.listdir()``)""" ... def run_script(self, script_name: str, namespace: dict[str, Any]) -> None: """Execute the named script in the supplied namespace dictionary""" ... class IResourceProvider(IMetadataProvider, Protocol): """An object that provides access to package resources""" def get_resource_filename( self, manager: ResourceManager, resource_name: str ) -> str: """Return a true filesystem path for `resource_name` `manager` must be a ``ResourceManager``""" ... def get_resource_stream( self, manager: ResourceManager, resource_name: str ) -> _ResourceStream: """Return a readable file-like object for `resource_name` `manager` must be a ``ResourceManager``""" ... def get_resource_string( self, manager: ResourceManager, resource_name: str ) -> bytes: """Return the contents of `resource_name` as :obj:`bytes` `manager` must be a ``ResourceManager``""" ... def has_resource(self, resource_name: str) -> bool: """Does the package contain the named resource?""" ... def resource_isdir(self, resource_name: str) -> bool: """Is the named resource a directory? (like ``os.path.isdir()``)""" ... def resource_listdir(self, resource_name: str) -> list[str]: """List of resource names in the directory (like ``os.listdir()``)""" ... class WorkingSet: """A collection of active distributions on sys.path (or a similar list)""" def __init__(self, entries: Iterable[str] | None = None) -> None: """Create working set from list of path entries (default=sys.path)""" self.entries: list[str] = [] self.entry_keys: dict[str | None, list[str]] = {} self.by_key: dict[str, Distribution] = {} self.normalized_to_canonical_keys: dict[str, str] = {} self.callbacks: list[Callable[[Distribution], object]] = [] if entries is None: entries = sys.path for entry in entries: self.add_entry(entry) @classmethod def _build_master(cls): """ Prepare the master working set. """ ws = cls() try: from __main__ import __requires__ except ImportError: # The main program does not list any requirements return ws # ensure the requirements are met try: ws.require(__requires__) except VersionConflict: return cls._build_from_requirements(__requires__) return ws @classmethod def _build_from_requirements(cls, req_spec): """ Build a working set from a requirement spec. Rewrites sys.path. """ # try it without defaults already on sys.path # by starting with an empty path ws = cls([]) reqs = parse_requirements(req_spec) dists = ws.resolve(reqs, Environment()) for dist in dists: ws.add(dist) # add any missing entries from sys.path for entry in sys.path: if entry not in ws.entries: ws.add_entry(entry) # then copy back to sys.path sys.path[:] = ws.entries return ws def add_entry(self, entry: str) -> None: """Add a path item to ``.entries``, finding any distributions on it ``find_distributions(entry, True)`` is used to find distributions corresponding to the path entry, and they are added. `entry` is always appended to ``.entries``, even if it is already present. (This is because ``sys.path`` can contain the same value more than once, and the ``.entries`` of the ``sys.path`` WorkingSet should always equal ``sys.path``.) """ self.entry_keys.setdefault(entry, []) self.entries.append(entry) for dist in find_distributions(entry, True): self.add(dist, entry, False) def __contains__(self, dist: Distribution) -> bool: """True if `dist` is the active distribution for its project""" return self.by_key.get(dist.key) == dist def find(self, req: Requirement) -> Distribution | None: """Find a distribution matching requirement `req` If there is an active distribution for the requested project, this returns it as long as it meets the version requirement specified by `req`. But, if there is an active distribution for the project and it does *not* meet the `req` requirement, ``VersionConflict`` is raised. If there is no active distribution for the requested project, ``None`` is returned. """ dist: Distribution | None = None candidates = ( req.key, self.normalized_to_canonical_keys.get(req.key), safe_name(req.key).replace(".", "-"), ) for candidate in filter(None, candidates): dist = self.by_key.get(candidate) if dist: req.key = candidate break if dist is not None and dist not in req: # XXX add more info raise VersionConflict(dist, req) return dist def iter_entry_points( self, group: str, name: str | None = None ) -> Iterator[EntryPoint]: """Yield entry point objects from `group` matching `name` If `name` is None, yields all entry points in `group` from all distributions in the working set, otherwise only ones matching both `group` and `name` are yielded (in distribution order). """ return ( entry for dist in self for entry in dist.get_entry_map(group).values() if name is None or name == entry.name ) def run_script(self, requires: str, script_name: str) -> None: """Locate distribution for `requires` and run `script_name` script""" ns = sys._getframe(1).f_globals name = ns['__name__'] ns.clear() ns['__name__'] = name self.require(requires)[0].run_script(script_name, ns) def __iter__(self) -> Iterator[Distribution]: """Yield distributions for non-duplicate projects in the working set The yield order is the order in which the items' path entries were added to the working set. """ seen = set() for item in self.entries: if item not in self.entry_keys: # workaround a cache issue continue for key in self.entry_keys[item]: if key not in seen: seen.add(key) yield self.by_key[key] def add( self, dist: Distribution, entry: str | None = None, insert: bool = True, replace: bool = False, ) -> None: """Add `dist` to working set, associated with `entry` If `entry` is unspecified, it defaults to the ``.location`` of `dist`. On exit from this routine, `entry` is added to the end of the working set's ``.entries`` (if it wasn't already present). `dist` is only added to the working set if it's for a project that doesn't already have a distribution in the set, unless `replace=True`. If it's added, any callbacks registered with the ``subscribe()`` method will be called. """ if insert: dist.insert_on(self.entries, entry, replace=replace) if entry is None: entry = dist.location keys = self.entry_keys.setdefault(entry, []) keys2 = self.entry_keys.setdefault(dist.location, []) if not replace and dist.key in self.by_key: # ignore hidden distros return self.by_key[dist.key] = dist normalized_name = packaging.utils.canonicalize_name(dist.key) self.normalized_to_canonical_keys[normalized_name] = dist.key if dist.key not in keys: keys.append(dist.key) if dist.key not in keys2: keys2.append(dist.key) self._added_new(dist) @overload def resolve( self, requirements: Iterable[Requirement], env: Environment | None, installer: _StrictInstallerType[_DistributionT], replace_conflicting: bool = False, extras: tuple[str, ...] | None = None, ) -> list[_DistributionT]: ... @overload def resolve( self, requirements: Iterable[Requirement], env: Environment | None = None, *, installer: _StrictInstallerType[_DistributionT], replace_conflicting: bool = False, extras: tuple[str, ...] | None = None, ) -> list[_DistributionT]: ... @overload def resolve( self, requirements: Iterable[Requirement], env: Environment | None = None, installer: _InstallerType | None = None, replace_conflicting: bool = False, extras: tuple[str, ...] | None = None, ) -> list[Distribution]: ... def resolve( self, requirements: Iterable[Requirement], env: Environment | None = None, installer: _InstallerType | None | _StrictInstallerType[_DistributionT] = None, replace_conflicting: bool = False, extras: tuple[str, ...] | None = None, ) -> list[Distribution] | list[_DistributionT]: """List all distributions needed to (recursively) meet `requirements` `requirements` must be a sequence of ``Requirement`` objects. `env`, if supplied, should be an ``Environment`` instance. If not supplied, it defaults to all distributions available within any entry or distribution in the working set. `installer`, if supplied, will be invoked with each requirement that cannot be met by an already-installed distribution; it should return a ``Distribution`` or ``None``. Unless `replace_conflicting=True`, raises a VersionConflict exception if any requirements are found on the path that have the correct name but the wrong version. Otherwise, if an `installer` is supplied it will be invoked to obtain the correct version of the requirement and activate it. `extras` is a list of the extras to be used with these requirements. This is important because extra requirements may look like `my_req; extra = "my_extra"`, which would otherwise be interpreted as a purely optional requirement. Instead, we want to be able to assert that these requirements are truly required. """ # set up the stack requirements = list(requirements)[::-1] # set of processed requirements processed = set() # key -> dist best: dict[str, Distribution] = {} to_activate: list[Distribution] = [] req_extras = _ReqExtras() # Mapping of requirement to set of distributions that required it; # useful for reporting info about conflicts. required_by = collections.defaultdict[Requirement, set[str]](set) while requirements: # process dependencies breadth-first req = requirements.pop(0) if req in processed: # Ignore cyclic or redundant dependencies continue if not req_extras.markers_pass(req, extras): continue dist = self._resolve_dist( req, best, replace_conflicting, env, installer, required_by, to_activate ) # push the new requirements onto the stack new_requirements = dist.requires(req.extras)[::-1] requirements.extend(new_requirements) # Register the new requirements needed by req for new_requirement in new_requirements: required_by[new_requirement].add(req.project_name) req_extras[new_requirement] = req.extras processed.add(req) # return list of distros to activate return to_activate def _resolve_dist( self, req, best, replace_conflicting, env, installer, required_by, to_activate ) -> Distribution: dist = best.get(req.key) if dist is None: # Find the best distribution and add it to the map dist = self.by_key.get(req.key) if dist is None or (dist not in req and replace_conflicting): ws = self if env is None: if dist is None: env = Environment(self.entries) else: # Use an empty environment and workingset to avoid # any further conflicts with the conflicting # distribution env = Environment([]) ws = WorkingSet([]) dist = best[req.key] = env.best_match( req, ws, installer, replace_conflicting=replace_conflicting ) if dist is None: requirers = required_by.get(req, None) raise DistributionNotFound(req, requirers) to_activate.append(dist) if dist not in req: # Oops, the "best" so far conflicts with a dependency dependent_req = required_by[req] raise VersionConflict(dist, req).with_context(dependent_req) return dist @overload def find_plugins( self, plugin_env: Environment, full_env: Environment | None, installer: _StrictInstallerType[_DistributionT], fallback: bool = True, ) -> tuple[list[_DistributionT], dict[Distribution, Exception]]: ... @overload def find_plugins( self, plugin_env: Environment, full_env: Environment | None = None, *, installer: _StrictInstallerType[_DistributionT], fallback: bool = True, ) -> tuple[list[_DistributionT], dict[Distribution, Exception]]: ... @overload def find_plugins( self, plugin_env: Environment, full_env: Environment | None = None, installer: _InstallerType | None = None, fallback: bool = True, ) -> tuple[list[Distribution], dict[Distribution, Exception]]: ... def find_plugins( self, plugin_env: Environment, full_env: Environment | None = None, installer: _InstallerType | None | _StrictInstallerType[_DistributionT] = None, fallback: bool = True, ) -> tuple[ list[Distribution] | list[_DistributionT], dict[Distribution, Exception], ]: """Find all activatable distributions in `plugin_env` Example usage:: distributions, errors = working_set.find_plugins( Environment(plugin_dirlist) ) # add plugins+libs to sys.path map(working_set.add, distributions) # display errors print('Could not load', errors) The `plugin_env` should be an ``Environment`` instance that contains only distributions that are in the project's "plugin directory" or directories. The `full_env`, if supplied, should be an ``Environment`` contains all currently-available distributions. If `full_env` is not supplied, one is created automatically from the ``WorkingSet`` this method is called on, which will typically mean that every directory on ``sys.path`` will be scanned for distributions. `installer` is a standard installer callback as used by the ``resolve()`` method. The `fallback` flag indicates whether we should attempt to resolve older versions of a plugin if the newest version cannot be resolved. This method returns a 2-tuple: (`distributions`, `error_info`), where `distributions` is a list of the distributions found in `plugin_env` that were loadable, along with any other distributions that are needed to resolve their dependencies. `error_info` is a dictionary mapping unloadable plugin distributions to an exception instance describing the error that occurred. Usually this will be a ``DistributionNotFound`` or ``VersionConflict`` instance. """ plugin_projects = list(plugin_env) # scan project names in alphabetic order plugin_projects.sort() error_info: dict[Distribution, Exception] = {} distributions: dict[Distribution, Exception | None] = {} if full_env is None: env = Environment(self.entries) env += plugin_env else: env = full_env + plugin_env shadow_set = self.__class__([]) # put all our entries in shadow_set list(map(shadow_set.add, self)) for project_name in plugin_projects: for dist in plugin_env[project_name]: req = [dist.as_requirement()] try: resolvees = shadow_set.resolve(req, env, installer) except ResolutionError as v: # save error info error_info[dist] = v if fallback: # try the next older version of project continue else: # give up on this project, keep going break else: list(map(shadow_set.add, resolvees)) distributions.update(dict.fromkeys(resolvees)) # success, no need to try any more versions of this project break sorted_distributions = list(distributions) sorted_distributions.sort() return sorted_distributions, error_info def require(self, *requirements: _NestedStr) -> list[Distribution]: """Ensure that distributions matching `requirements` are activated `requirements` must be a string or a (possibly-nested) sequence thereof, specifying the distributions and versions required. The return value is a sequence of the distributions that needed to be activated to fulfill the requirements; all relevant distributions are included, even if they were already activated in this working set. """ needed = self.resolve(parse_requirements(requirements)) for dist in needed: self.add(dist) return needed def subscribe( self, callback: Callable[[Distribution], object], existing: bool = True ) -> None: """Invoke `callback` for all distributions If `existing=True` (default), call on all existing ones, as well. """ if callback in self.callbacks: return self.callbacks.append(callback) if not existing: return for dist in self: callback(dist) def _added_new(self, dist) -> None: for callback in self.callbacks: callback(dist) def __getstate__( self, ) -> tuple[ list[str], dict[str | None, list[str]], dict[str, Distribution], dict[str, str], list[Callable[[Distribution], object]], ]: return ( self.entries[:], self.entry_keys.copy(), self.by_key.copy(), self.normalized_to_canonical_keys.copy(), self.callbacks[:], ) def __setstate__(self, e_k_b_n_c) -> None: entries, keys, by_key, normalized_to_canonical_keys, callbacks = e_k_b_n_c self.entries = entries[:] self.entry_keys = keys.copy() self.by_key = by_key.copy() self.normalized_to_canonical_keys = normalized_to_canonical_keys.copy() self.callbacks = callbacks[:] class _ReqExtras(dict["Requirement", tuple[str, ...]]): """ Map each requirement to the extras that demanded it. """ def markers_pass(self, req: Requirement, extras: tuple[str, ...] | None = None): """ Evaluate markers for req against each extra that demanded it. Return False if the req has a marker and fails evaluation. Otherwise, return True. """ return not req.marker or any( req.marker.evaluate({'extra': extra}) for extra in self.get(req, ()) + (extras or ("",)) ) class Environment: """Searchable snapshot of distributions on a search path""" def __init__( self, search_path: Iterable[str] | None = None, platform: str | None = get_supported_platform(), python: str | None = PY_MAJOR, ) -> None: """Snapshot distributions available on a search path Any distributions found on `search_path` are added to the environment. `search_path` should be a sequence of ``sys.path`` items. If not supplied, ``sys.path`` is used. `platform` is an optional string specifying the name of the platform that platform-specific distributions must be compatible with. If unspecified, it defaults to the current platform. `python` is an optional string naming the desired version of Python (e.g. ``'3.6'``); it defaults to the current version. You may explicitly set `platform` (and/or `python`) to ``None`` if you wish to map *all* distributions, not just those compatible with the running platform or Python version. """ self._distmap: dict[str, list[Distribution]] = {} self.platform = platform self.python = python self.scan(search_path) def can_add(self, dist: Distribution) -> bool: """Is distribution `dist` acceptable for this environment? The distribution must match the platform and python version requirements specified when this environment was created, or False is returned. """ py_compat = ( self.python is None or dist.py_version is None or dist.py_version == self.python ) return py_compat and compatible_platforms(dist.platform, self.platform) def remove(self, dist: Distribution) -> None: """Remove `dist` from the environment""" self._distmap[dist.key].remove(dist) def scan(self, search_path: Iterable[str] | None = None) -> None: """Scan `search_path` for distributions usable in this environment Any distributions found are added to the environment. `search_path` should be a sequence of ``sys.path`` items. If not supplied, ``sys.path`` is used. Only distributions conforming to the platform/python version defined at initialization are added. """ if search_path is None: search_path = sys.path for item in search_path: for dist in find_distributions(item): self.add(dist) def __getitem__(self, project_name: str) -> list[Distribution]: """Return a newest-to-oldest list of distributions for `project_name` Uses case-insensitive `project_name` comparison, assuming all the project's distributions use their project's name converted to all lowercase as their key. """ distribution_key = project_name.lower() return self._distmap.get(distribution_key, []) def add(self, dist: Distribution) -> None: """Add `dist` if we ``can_add()`` it and it has not already been added""" if self.can_add(dist) and dist.has_version(): dists = self._distmap.setdefault(dist.key, []) if dist not in dists: dists.append(dist) dists.sort(key=operator.attrgetter('hashcmp'), reverse=True) @overload def best_match( self, req: Requirement, working_set: WorkingSet, installer: _StrictInstallerType[_DistributionT], replace_conflicting: bool = False, ) -> _DistributionT: ... @overload def best_match( self, req: Requirement, working_set: WorkingSet, installer: _InstallerType | None = None, replace_conflicting: bool = False, ) -> Distribution | None: ... def best_match( self, req: Requirement, working_set: WorkingSet, installer: _InstallerType | None | _StrictInstallerType[_DistributionT] = None, replace_conflicting: bool = False, ) -> Distribution | None: """Find distribution best matching `req` and usable on `working_set` This calls the ``find(req)`` method of the `working_set` to see if a suitable distribution is already active. (This may raise ``VersionConflict`` if an unsuitable version of the project is already active in the specified `working_set`.) If a suitable distribution isn't active, this method returns the newest distribution in the environment that meets the ``Requirement`` in `req`. If no suitable distribution is found, and `installer` is supplied, then the result of calling the environment's ``obtain(req, installer)`` method will be returned. """ try: dist = working_set.find(req) except VersionConflict: if not replace_conflicting: raise dist = None if dist is not None: return dist for dist in self[req.key]: if dist in req: return dist # try to download/install return self.obtain(req, installer) @overload def obtain( self, requirement: Requirement, installer: _StrictInstallerType[_DistributionT], ) -> _DistributionT: ... @overload def obtain( self, requirement: Requirement, installer: Callable[[Requirement], None] | None = None, ) -> None: ... @overload def obtain( self, requirement: Requirement, installer: _InstallerType | None = None, ) -> Distribution | None: ... def obtain( self, requirement: Requirement, installer: Callable[[Requirement], None] | _InstallerType | None | _StrictInstallerType[_DistributionT] = None, ) -> Distribution | None: """Obtain a distribution matching `requirement` (e.g. via download) Obtain a distro that matches requirement (e.g. via download). In the base ``Environment`` class, this routine just returns ``installer(requirement)``, unless `installer` is None, in which case None is returned instead. This method is a hook that allows subclasses to attempt other ways of obtaining a distribution before falling back to the `installer` argument.""" return installer(requirement) if installer else None def __iter__(self) -> Iterator[str]: """Yield the unique project names of the available distributions""" for key in self._distmap.keys(): if self[key]: yield key def __iadd__(self, other: Distribution | Environment) -> Self: """In-place addition of a distribution or environment""" if isinstance(other, Distribution): self.add(other) elif isinstance(other, Environment): for project in other: for dist in other[project]: self.add(dist) else: raise TypeError(f"Can't add {other!r} to environment") return self def __add__(self, other: Distribution | Environment) -> Self: """Add an environment or distribution to an environment""" new = self.__class__([], platform=None, python=None) for env in self, other: new += env return new # XXX backward compatibility AvailableDistributions = Environment class ExtractionError(RuntimeError): """An error occurred extracting a resource The following attributes are available from instances of this exception: manager The resource manager that raised this exception cache_path The base directory for resource extraction original_error The exception instance that caused extraction to fail """ manager: ResourceManager cache_path: str original_error: BaseException | None class ResourceManager: """Manage resource extraction and packages""" extraction_path: str | None = None def __init__(self) -> None: # acts like a set self.cached_files: dict[str, Literal[True]] = {} def resource_exists( self, package_or_requirement: _PkgReqType, resource_name: str ) -> bool: """Does the named resource exist?""" return get_provider(package_or_requirement).has_resource(resource_name) def resource_isdir( self, package_or_requirement: _PkgReqType, resource_name: str ) -> bool: """Is the named resource an existing directory?""" return get_provider(package_or_requirement).resource_isdir(resource_name) def resource_filename( self, package_or_requirement: _PkgReqType, resource_name: str ) -> str: """Return a true filesystem path for specified resource""" return get_provider(package_or_requirement).get_resource_filename( self, resource_name ) def resource_stream( self, package_or_requirement: _PkgReqType, resource_name: str ) -> _ResourceStream: """Return a readable file-like object for specified resource""" return get_provider(package_or_requirement).get_resource_stream( self, resource_name ) def resource_string( self, package_or_requirement: _PkgReqType, resource_name: str ) -> bytes: """Return specified resource as :obj:`bytes`""" return get_provider(package_or_requirement).get_resource_string( self, resource_name ) def resource_listdir( self, package_or_requirement: _PkgReqType, resource_name: str ) -> list[str]: """List the contents of the named resource directory""" return get_provider(package_or_requirement).resource_listdir(resource_name) def extraction_error(self) -> NoReturn: """Give an error message for problems extracting file(s)""" old_exc = sys.exc_info()[1] cache_path = self.extraction_path or get_default_cache() tmpl = textwrap.dedent( """ Can't extract file(s) to egg cache The following error occurred while trying to extract file(s) to the Python egg cache: {old_exc} The Python egg cache directory is currently set to: {cache_path} Perhaps your account does not have write access to this directory? You can change the cache directory by setting the PYTHON_EGG_CACHE environment variable to point to an accessible directory. """ ).lstrip() err = ExtractionError(tmpl.format(**locals())) err.manager = self err.cache_path = cache_path err.original_error = old_exc raise err def get_cache_path(self, archive_name: str, names: Iterable[StrPath] = ()) -> str: """Return absolute location in cache for `archive_name` and `names` The parent directory of the resulting path will be created if it does not already exist. `archive_name` should be the base filename of the enclosing egg (which may not be the name of the enclosing zipfile!), including its ".egg" extension. `names`, if provided, should be a sequence of path name parts "under" the egg's extraction location. This method should only be called by resource providers that need to obtain an extraction location, and only for names they intend to extract, as it tracks the generated names for possible cleanup later. """ extract_path = self.extraction_path or get_default_cache() target_path = os.path.join(extract_path, archive_name + '-tmp', *names) try: _bypass_ensure_directory(target_path) except Exception: self.extraction_error() self._warn_unsafe_extraction_path(extract_path) self.cached_files[target_path] = True return target_path @staticmethod def _warn_unsafe_extraction_path(path) -> None: """ If the default extraction path is overridden and set to an insecure location, such as /tmp, it opens up an opportunity for an attacker to replace an extracted file with an unauthorized payload. Warn the user if a known insecure location is used. See Distribute #375 for more details. """ if os.name == 'nt' and not path.startswith(os.environ['windir']): # On Windows, permissions are generally restrictive by default # and temp directories are not writable by other users, so # bypass the warning. return mode = os.stat(path).st_mode if mode & stat.S_IWOTH or mode & stat.S_IWGRP: msg = ( "Extraction path is writable by group/others " "and vulnerable to attack when " "used with get_resource_filename ({path}). " "Consider a more secure " "location (set with .set_extraction_path or the " "PYTHON_EGG_CACHE environment variable)." ).format(**locals()) warnings.warn(msg, UserWarning) def postprocess(self, tempname: StrOrBytesPath, filename: StrOrBytesPath) -> None: """Perform any platform-specific postprocessing of `tempname` This is where Mac header rewrites should be done; other platforms don't have anything special they should do. Resource providers should call this method ONLY after successfully extracting a compressed resource. They must NOT call it on resources that are already in the filesystem. `tempname` is the current (temporary) name of the file, and `filename` is the name it will be renamed to by the caller after this routine returns. """ if os.name == 'posix': # Make the resource executable mode = ((os.stat(tempname).st_mode) | 0o555) & 0o7777 os.chmod(tempname, mode) def set_extraction_path(self, path: str) -> None: """Set the base path where resources will be extracted to, if needed. If you do not call this routine before any extractions take place, the path defaults to the return value of ``get_default_cache()``. (Which is based on the ``PYTHON_EGG_CACHE`` environment variable, with various platform-specific fallbacks. See that routine's documentation for more details.) Resources are extracted to subdirectories of this path based upon information given by the ``IResourceProvider``. You may set this to a temporary directory, but then you must call ``cleanup_resources()`` to delete the extracted files when done. There is no guarantee that ``cleanup_resources()`` will be able to remove all extracted files. (Note: you may not change the extraction path for a given resource manager once resources have been extracted, unless you first call ``cleanup_resources()``.) """ if self.cached_files: raise ValueError("Can't change extraction path, files already extracted") self.extraction_path = path def cleanup_resources(self, force: bool = False) -> list[str]: """ Delete all extracted resource files and directories, returning a list of the file and directory names that could not be successfully removed. This function does not have any concurrency protection, so it should generally only be called when the extraction path is a temporary directory exclusive to a single process. This method is not automatically called; you must call it explicitly or register it as an ``atexit`` function if you wish to ensure cleanup of a temporary directory used for extractions. """ # XXX return [] def get_default_cache() -> str: """ Return the ``PYTHON_EGG_CACHE`` environment variable or a platform-relevant user cache dir for an app named "Python-Eggs". """ return os.environ.get('PYTHON_EGG_CACHE') or _user_cache_dir(appname='Python-Eggs') def safe_name(name: str) -> str: """Convert an arbitrary string to a standard distribution name Any runs of non-alphanumeric/. characters are replaced with a single '-'. """ return re.sub('[^A-Za-z0-9.]+', '-', name) def safe_version(version: str) -> str: """ Convert an arbitrary string to a standard version string """ try: # normalize the version return str(packaging.version.Version(version)) except packaging.version.InvalidVersion: version = version.replace(' ', '.') return re.sub('[^A-Za-z0-9.]+', '-', version) def _forgiving_version(version) -> str: """Fallback when ``safe_version`` is not safe enough >>> parse_version(_forgiving_version('0.23ubuntu1')) >>> parse_version(_forgiving_version('0.23-')) >>> parse_version(_forgiving_version('0.-_')) >>> parse_version(_forgiving_version('42.+?1')) >>> parse_version(_forgiving_version('hello world')) """ version = version.replace(' ', '.') match = _PEP440_FALLBACK.search(version) if match: safe = match["safe"] rest = version[len(safe) :] else: safe = "0" rest = version local = f"sanitized.{_safe_segment(rest)}".strip(".") return f"{safe}.dev0+{local}" def _safe_segment(segment): """Convert an arbitrary string into a safe segment""" segment = re.sub('[^A-Za-z0-9.]+', '-', segment) segment = re.sub('-[^A-Za-z0-9]+', '-', segment) return re.sub(r'\.[^A-Za-z0-9]+', '.', segment).strip(".-") def safe_extra(extra: str) -> str: """Convert an arbitrary string to a standard 'extra' name Any runs of non-alphanumeric characters are replaced with a single '_', and the result is always lowercased. """ return re.sub('[^A-Za-z0-9.-]+', '_', extra).lower() def to_filename(name: str) -> str: """Convert a project or version name to its filename-escaped form Any '-' characters are currently replaced with '_'. """ return name.replace('-', '_') def invalid_marker(text: str) -> SyntaxError | Literal[False]: """ Validate text as a PEP 508 environment marker; return an exception if invalid or False otherwise. """ try: evaluate_marker(text) except SyntaxError as e: e.filename = None e.lineno = None return e return False def evaluate_marker(text: str, extra: str | None = None) -> bool: """ Evaluate a PEP 508 environment marker. Return a boolean indicating the marker result in this environment. Raise SyntaxError if marker is invalid. This implementation uses the 'pyparsing' module. """ try: marker = packaging.markers.Marker(text) return marker.evaluate() except packaging.markers.InvalidMarker as e: raise SyntaxError(e) from e class NullProvider: """Try to implement resources and metadata for arbitrary PEP 302 loaders""" egg_name: str | None = None egg_info: str | None = None loader: LoaderProtocol | None = None def __init__(self, module: _ModuleLike) -> None: self.loader = getattr(module, '__loader__', None) self.module_path = os.path.dirname(getattr(module, '__file__', '')) def get_resource_filename( self, manager: ResourceManager, resource_name: str ) -> str: return self._fn(self.module_path, resource_name) def get_resource_stream( self, manager: ResourceManager, resource_name: str ) -> BinaryIO: return io.BytesIO(self.get_resource_string(manager, resource_name)) def get_resource_string( self, manager: ResourceManager, resource_name: str ) -> bytes: return self._get(self._fn(self.module_path, resource_name)) def has_resource(self, resource_name: str) -> bool: return self._has(self._fn(self.module_path, resource_name)) def _get_metadata_path(self, name): return self._fn(self.egg_info, name) def has_metadata(self, name: str) -> bool: if not self.egg_info: return False path = self._get_metadata_path(name) return self._has(path) def get_metadata(self, name: str) -> str: if not self.egg_info: return "" path = self._get_metadata_path(name) value = self._get(path) try: return value.decode('utf-8') except UnicodeDecodeError as exc: # Include the path in the error message to simplify # troubleshooting, and without changing the exception type. exc.reason += f' in {name} file at path: {path}' raise def get_metadata_lines(self, name: str) -> Iterator[str]: return yield_lines(self.get_metadata(name)) def resource_isdir(self, resource_name: str) -> bool: return self._isdir(self._fn(self.module_path, resource_name)) def metadata_isdir(self, name: str) -> bool: return bool(self.egg_info and self._isdir(self._fn(self.egg_info, name))) def resource_listdir(self, resource_name: str) -> list[str]: return self._listdir(self._fn(self.module_path, resource_name)) def metadata_listdir(self, name: str) -> list[str]: if self.egg_info: return self._listdir(self._fn(self.egg_info, name)) return [] def run_script(self, script_name: str, namespace: dict[str, Any]) -> None: script = 'scripts/' + script_name if not self.has_metadata(script): raise ResolutionError( "Script {script!r} not found in metadata at {self.egg_info!r}".format( **locals() ), ) script_text = self.get_metadata(script).replace('\r\n', '\n') script_text = script_text.replace('\r', '\n') script_filename = self._fn(self.egg_info, script) namespace['__file__'] = script_filename if os.path.exists(script_filename): source = _read_utf8_with_fallback(script_filename) code = compile(source, script_filename, 'exec') exec(code, namespace, namespace) else: from linecache import cache cache[script_filename] = ( len(script_text), 0, script_text.split('\n'), script_filename, ) script_code = compile(script_text, script_filename, 'exec') exec(script_code, namespace, namespace) def _has(self, path) -> bool: raise NotImplementedError( "Can't perform this operation for unregistered loader type" ) def _isdir(self, path) -> bool: raise NotImplementedError( "Can't perform this operation for unregistered loader type" ) def _listdir(self, path) -> list[str]: raise NotImplementedError( "Can't perform this operation for unregistered loader type" ) def _fn(self, base: str | None, resource_name: str): if base is None: raise TypeError( "`base` parameter in `_fn` is `None`. Either override this method or check the parameter first." ) self._validate_resource_path(resource_name) if resource_name: return os.path.join(base, *resource_name.split('/')) return base @staticmethod def _validate_resource_path(path) -> None: """ Validate the resource paths according to the docs. https://setuptools.pypa.io/en/latest/pkg_resources.html#basic-resource-access >>> warned = getfixture('recwarn') >>> warnings.simplefilter('always') >>> vrp = NullProvider._validate_resource_path >>> vrp('foo/bar.txt') >>> bool(warned) False >>> vrp('../foo/bar.txt') >>> bool(warned) True >>> warned.clear() >>> vrp('/foo/bar.txt') >>> bool(warned) True >>> vrp('foo/../../bar.txt') >>> bool(warned) True >>> warned.clear() >>> vrp('foo/f../bar.txt') >>> bool(warned) False Windows path separators are straight-up disallowed. >>> vrp(r'\\foo/bar.txt') Traceback (most recent call last): ... ValueError: Use of .. or absolute path in a resource path \ is not allowed. >>> vrp(r'C:\\foo/bar.txt') Traceback (most recent call last): ... ValueError: Use of .. or absolute path in a resource path \ is not allowed. Blank values are allowed >>> vrp('') >>> bool(warned) False Non-string values are not. >>> vrp(None) Traceback (most recent call last): ... AttributeError: ... """ invalid = ( os.path.pardir in path.split(posixpath.sep) or posixpath.isabs(path) or ntpath.isabs(path) or path.startswith("\\") ) if not invalid: return msg = "Use of .. or absolute path in a resource path is not allowed." # Aggressively disallow Windows absolute paths if (path.startswith("\\") or ntpath.isabs(path)) and not posixpath.isabs(path): raise ValueError(msg) # for compatibility, warn; in future # raise ValueError(msg) issue_warning( msg[:-1] + " and will raise exceptions in a future release.", DeprecationWarning, ) def _get(self, path) -> bytes: if hasattr(self.loader, 'get_data') and self.loader: # Already checked get_data exists return self.loader.get_data(path) # type: ignore[attr-defined] raise NotImplementedError( "Can't perform this operation for loaders without 'get_data()'" ) register_loader_type(object, NullProvider) def _parents(path): """ yield all parents of path including path """ last = None while path != last: yield path last = path path, _ = os.path.split(path) class EggProvider(NullProvider): """Provider based on a virtual filesystem""" def __init__(self, module: _ModuleLike) -> None: super().__init__(module) self._setup_prefix() def _setup_prefix(self): # Assume that metadata may be nested inside a "basket" # of multiple eggs and use module_path instead of .archive. eggs = filter(_is_egg_path, _parents(self.module_path)) egg = next(eggs, None) egg and self._set_egg(egg) def _set_egg(self, path: str) -> None: self.egg_name = os.path.basename(path) self.egg_info = os.path.join(path, 'EGG-INFO') self.egg_root = path class DefaultProvider(EggProvider): """Provides access to package resources in the filesystem""" def _has(self, path) -> bool: return os.path.exists(path) def _isdir(self, path) -> bool: return os.path.isdir(path) def _listdir(self, path): return os.listdir(path) def get_resource_stream( self, manager: object, resource_name: str ) -> io.BufferedReader: return open(self._fn(self.module_path, resource_name), 'rb') def _get(self, path) -> bytes: with open(path, 'rb') as stream: return stream.read() @classmethod def _register(cls) -> None: loader_names = ( 'SourceFileLoader', 'SourcelessFileLoader', ) for name in loader_names: loader_cls = getattr(importlib.machinery, name, type(None)) register_loader_type(loader_cls, cls) DefaultProvider._register() class EmptyProvider(NullProvider): """Provider that returns nothing for all requests""" # A special case, we don't want all Providers inheriting from NullProvider to have a potentially None module_path module_path: str | None = None # type: ignore[assignment] _isdir = _has = lambda self, path: False def _get(self, path) -> bytes: return b'' def _listdir(self, path): return [] def __init__(self) -> None: pass empty_provider = EmptyProvider() class ZipManifests(dict[str, "MemoizedZipManifests.manifest_mod"]): """ zip manifest builder """ # `path` could be `StrPath | IO[bytes]` but that violates the LSP for `MemoizedZipManifests.load` @classmethod def build(cls, path: str) -> dict[str, zipfile.ZipInfo]: """ Build a dictionary similar to the zipimport directory caches, except instead of tuples, store ZipInfo objects. Use a platform-specific path separator (os.sep) for the path keys for compatibility with pypy on Windows. """ with zipfile.ZipFile(path) as zfile: items = ( ( name.replace('/', os.sep), zfile.getinfo(name), ) for name in zfile.namelist() ) return dict(items) load = build class MemoizedZipManifests(ZipManifests): """ Memoized zipfile manifests. """ class manifest_mod(NamedTuple): manifest: dict[str, zipfile.ZipInfo] mtime: float def load(self, path: str) -> dict[str, zipfile.ZipInfo]: # type: ignore[override] # ZipManifests.load is a classmethod """ Load a manifest at path or return a suitable manifest already loaded. """ path = os.path.normpath(path) mtime = os.stat(path).st_mtime if path not in self or self[path].mtime != mtime: manifest = self.build(path) self[path] = self.manifest_mod(manifest, mtime) return self[path].manifest class ZipProvider(EggProvider): """Resource support for zips and eggs""" eagers: list[str] | None = None _zip_manifests = MemoizedZipManifests() # ZipProvider's loader should always be a zipimporter or equivalent loader: zipimport.zipimporter def __init__(self, module: _ZipLoaderModule) -> None: super().__init__(module) self.zip_pre = self.loader.archive + os.sep def _zipinfo_name(self, fspath): # Convert a virtual filename (full path to file) into a zipfile subpath # usable with the zipimport directory cache for our target archive fspath = fspath.rstrip(os.sep) if fspath == self.loader.archive: return '' if fspath.startswith(self.zip_pre): return fspath[len(self.zip_pre) :] raise AssertionError(f"{fspath} is not a subpath of {self.zip_pre}") def _parts(self, zip_path): # Convert a zipfile subpath into an egg-relative path part list. # pseudo-fs path fspath = self.zip_pre + zip_path if fspath.startswith(self.egg_root + os.sep): return fspath[len(self.egg_root) + 1 :].split(os.sep) raise AssertionError(f"{fspath} is not a subpath of {self.egg_root}") @property def zipinfo(self): return self._zip_manifests.load(self.loader.archive) def get_resource_filename( self, manager: ResourceManager, resource_name: str ) -> str: if not self.egg_name: raise NotImplementedError( "resource_filename() only supported for .egg, not .zip" ) # no need to lock for extraction, since we use temp names zip_path = self._resource_to_zip(resource_name) eagers = self._get_eager_resources() if '/'.join(self._parts(zip_path)) in eagers: for name in eagers: self._extract_resource(manager, self._eager_to_zip(name)) return self._extract_resource(manager, zip_path) @staticmethod def _get_date_and_size(zip_stat): size = zip_stat.file_size # ymdhms+wday, yday, dst date_time = zip_stat.date_time + (0, 0, -1) # 1980 offset already done timestamp = time.mktime(date_time) return timestamp, size # FIXME: 'ZipProvider._extract_resource' is too complex (12) def _extract_resource(self, manager: ResourceManager, zip_path) -> str: # noqa: C901 if zip_path in self._index(): for name in self._index()[zip_path]: last = self._extract_resource(manager, os.path.join(zip_path, name)) # return the extracted directory name return os.path.dirname(last) timestamp, _size = self._get_date_and_size(self.zipinfo[zip_path]) if not WRITE_SUPPORT: raise OSError( '"os.rename" and "os.unlink" are not supported on this platform' ) try: if not self.egg_name: raise OSError( '"egg_name" is empty. This likely means no egg could be found from the "module_path".' ) real_path = manager.get_cache_path(self.egg_name, self._parts(zip_path)) if self._is_current(real_path, zip_path): return real_path outf, tmpnam = _mkstemp( ".$extract", dir=os.path.dirname(real_path), ) os.write(outf, self.loader.get_data(zip_path)) os.close(outf) utime(tmpnam, (timestamp, timestamp)) manager.postprocess(tmpnam, real_path) try: rename(tmpnam, real_path) except OSError: if os.path.isfile(real_path): if self._is_current(real_path, zip_path): # the file became current since it was checked above, # so proceed. return real_path # Windows, del old file and retry elif os.name == 'nt': unlink(real_path) rename(tmpnam, real_path) return real_path raise except OSError: # report a user-friendly error manager.extraction_error() return real_path def _is_current(self, file_path, zip_path): """ Return True if the file_path is current for this zip_path """ timestamp, size = self._get_date_and_size(self.zipinfo[zip_path]) if not os.path.isfile(file_path): return False stat = os.stat(file_path) if stat.st_size != size or stat.st_mtime != timestamp: return False # check that the contents match zip_contents = self.loader.get_data(zip_path) with open(file_path, 'rb') as f: file_contents = f.read() return zip_contents == file_contents def _get_eager_resources(self): if self.eagers is None: eagers = [] for name in ('native_libs.txt', 'eager_resources.txt'): if self.has_metadata(name): eagers.extend(self.get_metadata_lines(name)) self.eagers = eagers return self.eagers def _index(self): try: return self._dirindex except AttributeError: ind = {} for path in self.zipinfo: parts = path.split(os.sep) while parts: parent = os.sep.join(parts[:-1]) if parent in ind: ind[parent].append(parts[-1]) break else: ind[parent] = [parts.pop()] self._dirindex = ind return ind def _has(self, fspath) -> bool: zip_path = self._zipinfo_name(fspath) return zip_path in self.zipinfo or zip_path in self._index() def _isdir(self, fspath) -> bool: return self._zipinfo_name(fspath) in self._index() def _listdir(self, fspath): return list(self._index().get(self._zipinfo_name(fspath), ())) def _eager_to_zip(self, resource_name: str): return self._zipinfo_name(self._fn(self.egg_root, resource_name)) def _resource_to_zip(self, resource_name: str): return self._zipinfo_name(self._fn(self.module_path, resource_name)) register_loader_type(zipimport.zipimporter, ZipProvider) class FileMetadata(EmptyProvider): """Metadata handler for standalone PKG-INFO files Usage:: metadata = FileMetadata("/path/to/PKG-INFO") This provider rejects all data and metadata requests except for PKG-INFO, which is treated as existing, and will be the contents of the file at the provided location. """ def __init__(self, path: StrPath) -> None: self.path = path def _get_metadata_path(self, name): return self.path def has_metadata(self, name: str) -> bool: return name == 'PKG-INFO' and os.path.isfile(self.path) def get_metadata(self, name: str) -> str: if name != 'PKG-INFO': raise KeyError("No metadata except PKG-INFO is available") with open(self.path, encoding='utf-8', errors="replace") as f: metadata = f.read() self._warn_on_replacement(metadata) return metadata def _warn_on_replacement(self, metadata) -> None: replacement_char = '�' if replacement_char in metadata: tmpl = "{self.path} could not be properly decoded in UTF-8" msg = tmpl.format(**locals()) warnings.warn(msg) def get_metadata_lines(self, name: str) -> Iterator[str]: return yield_lines(self.get_metadata(name)) class PathMetadata(DefaultProvider): """Metadata provider for egg directories Usage:: # Development eggs: egg_info = "/path/to/PackageName.egg-info" base_dir = os.path.dirname(egg_info) metadata = PathMetadata(base_dir, egg_info) dist_name = os.path.splitext(os.path.basename(egg_info))[0] dist = Distribution(basedir, project_name=dist_name, metadata=metadata) # Unpacked egg directories: egg_path = "/path/to/PackageName-ver-pyver-etc.egg" metadata = PathMetadata(egg_path, os.path.join(egg_path,'EGG-INFO')) dist = Distribution.from_filename(egg_path, metadata=metadata) """ def __init__(self, path: str, egg_info: str) -> None: self.module_path = path self.egg_info = egg_info class EggMetadata(ZipProvider): """Metadata provider for .egg files""" def __init__(self, importer: zipimport.zipimporter) -> None: """Create a metadata provider from a zipimporter""" self.zip_pre = importer.archive + os.sep self.loader = importer if importer.prefix: self.module_path = os.path.join(importer.archive, importer.prefix) else: self.module_path = importer.archive self._setup_prefix() _distribution_finders: dict[type, _DistFinderType[Any]] = _declare_state( 'dict', '_distribution_finders', {} ) def register_finder( importer_type: type[_T], distribution_finder: _DistFinderType[_T] ) -> None: """Register `distribution_finder` to find distributions in sys.path items `importer_type` is the type or class of a PEP 302 "Importer" (sys.path item handler), and `distribution_finder` is a callable that, passed a path item and the importer instance, yields ``Distribution`` instances found on that path item. See ``pkg_resources.find_on_path`` for an example.""" _distribution_finders[importer_type] = distribution_finder def find_distributions(path_item: str, only: bool = False) -> Iterable[Distribution]: """Yield distributions accessible via `path_item`""" importer = get_importer(path_item) finder = _find_adapter(_distribution_finders, importer) return finder(importer, path_item, only) def find_eggs_in_zip( importer: zipimport.zipimporter, path_item: str, only: bool = False ) -> Iterator[Distribution]: """ Find eggs in zip files; possibly multiple nested eggs. """ if importer.archive.endswith('.whl'): # wheels are not supported with this finder # they don't have PKG-INFO metadata, and won't ever contain eggs return metadata = EggMetadata(importer) if metadata.has_metadata('PKG-INFO'): yield Distribution.from_filename(path_item, metadata=metadata) if only: # don't yield nested distros return for subitem in metadata.resource_listdir(''): if _is_egg_path(subitem): subpath = os.path.join(path_item, subitem) dists = find_eggs_in_zip(zipimport.zipimporter(subpath), subpath) yield from dists elif subitem.lower().endswith(('.dist-info', '.egg-info')): subpath = os.path.join(path_item, subitem) submeta = EggMetadata(zipimport.zipimporter(subpath)) submeta.egg_info = subpath yield Distribution.from_location(path_item, subitem, submeta) register_finder(zipimport.zipimporter, find_eggs_in_zip) def find_nothing( importer: object | None, path_item: str | None, only: bool | None = False ): return () register_finder(object, find_nothing) def find_on_path(importer: object | None, path_item, only=False): """Yield distributions accessible on a sys.path directory""" path_item = _normalize_cached(path_item) if _is_unpacked_egg(path_item): yield Distribution.from_filename( path_item, metadata=PathMetadata(path_item, os.path.join(path_item, 'EGG-INFO')), ) return entries = (os.path.join(path_item, child) for child in safe_listdir(path_item)) # scan for .egg and .egg-info in directory for entry in sorted(entries): fullpath = os.path.join(path_item, entry) factory = dist_factory(path_item, entry, only) yield from factory(fullpath) def dist_factory(path_item, entry, only): """Return a dist_factory for the given entry.""" lower = entry.lower() is_egg_info = lower.endswith('.egg-info') is_dist_info = lower.endswith('.dist-info') and os.path.isdir( os.path.join(path_item, entry) ) is_meta = is_egg_info or is_dist_info return ( distributions_from_metadata if is_meta else find_distributions if not only and _is_egg_path(entry) else resolve_egg_link if not only and lower.endswith('.egg-link') else NoDists() ) class NoDists: """ >>> bool(NoDists()) False >>> list(NoDists()('anything')) [] """ def __bool__(self) -> Literal[False]: return False def __call__(self, fullpath: object): return iter(()) def safe_listdir(path: StrOrBytesPath): """ Attempt to list contents of path, but suppress some exceptions. """ try: return os.listdir(path) except (PermissionError, NotADirectoryError): pass except OSError as e: # Ignore the directory if does not exist, not a directory or # permission denied if e.errno not in (errno.ENOTDIR, errno.EACCES, errno.ENOENT): raise return () def distributions_from_metadata(path: str): root = os.path.dirname(path) if os.path.isdir(path): if len(os.listdir(path)) == 0: # empty metadata dir; skip return metadata: _MetadataType = PathMetadata(root, path) else: metadata = FileMetadata(path) entry = os.path.basename(path) yield Distribution.from_location( root, entry, metadata, precedence=DEVELOP_DIST, ) def non_empty_lines(path): """ Yield non-empty lines from file at path """ for line in _read_utf8_with_fallback(path).splitlines(): line = line.strip() if line: yield line def resolve_egg_link(path): """ Given a path to an .egg-link, resolve distributions present in the referenced path. """ referenced_paths = non_empty_lines(path) resolved_paths = ( os.path.join(os.path.dirname(path), ref) for ref in referenced_paths ) dist_groups = map(find_distributions, resolved_paths) return next(dist_groups, ()) if hasattr(pkgutil, 'ImpImporter'): register_finder(pkgutil.ImpImporter, find_on_path) register_finder(importlib.machinery.FileFinder, find_on_path) _namespace_handlers: dict[type, _NSHandlerType[Any]] = _declare_state( 'dict', '_namespace_handlers', {} ) _namespace_packages: dict[str | None, list[str]] = _declare_state( 'dict', '_namespace_packages', {} ) def register_namespace_handler( importer_type: type[_T], namespace_handler: _NSHandlerType[_T] ) -> None: """Register `namespace_handler` to declare namespace packages `importer_type` is the type or class of a PEP 302 "Importer" (sys.path item handler), and `namespace_handler` is a callable like this:: def namespace_handler(importer, path_entry, moduleName, module): # return a path_entry to use for child packages Namespace handlers are only called if the importer object has already agreed that it can handle the relevant path item, and they should only return a subpath if the module __path__ does not already contain an equivalent subpath. For an example namespace handler, see ``pkg_resources.file_ns_handler``. """ _namespace_handlers[importer_type] = namespace_handler def _handle_ns(packageName, path_item): """Ensure that named package includes a subpath of path_item (if needed)""" importer = get_importer(path_item) if importer is None: return None # use find_spec (PEP 451) and fall-back to find_module (PEP 302) try: spec = importer.find_spec(packageName) except AttributeError: # capture warnings due to #1111 with warnings.catch_warnings(): warnings.simplefilter("ignore") loader = importer.find_module(packageName) else: loader = spec.loader if spec else None if loader is None: return None module = sys.modules.get(packageName) if module is None: module = sys.modules[packageName] = types.ModuleType(packageName) module.__path__ = [] _set_parent_ns(packageName) elif not hasattr(module, '__path__'): raise TypeError("Not a package:", packageName) handler = _find_adapter(_namespace_handlers, importer) subpath = handler(importer, path_item, packageName, module) if subpath is not None: path = module.__path__ path.append(subpath) importlib.import_module(packageName) _rebuild_mod_path(path, packageName, module) return subpath def _rebuild_mod_path(orig_path, package_name, module: types.ModuleType) -> None: """ Rebuild module.__path__ ensuring that all entries are ordered corresponding to their sys.path order """ sys_path = [_normalize_cached(p) for p in sys.path] def safe_sys_path_index(entry): """ Workaround for #520 and #513. """ try: return sys_path.index(entry) except ValueError: return float('inf') def position_in_sys_path(path): """ Return the ordinal of the path based on its position in sys.path """ path_parts = path.split(os.sep) module_parts = package_name.count('.') + 1 parts = path_parts[:-module_parts] return safe_sys_path_index(_normalize_cached(os.sep.join(parts))) new_path = sorted(orig_path, key=position_in_sys_path) new_path = [_normalize_cached(p) for p in new_path] if isinstance(module.__path__, list): module.__path__[:] = new_path else: module.__path__ = new_path def declare_namespace(packageName: str) -> None: """Declare that package 'packageName' is a namespace package""" msg = ( f"Deprecated call to `pkg_resources.declare_namespace({packageName!r})`.\n" "Implementing implicit namespace packages (as specified in PEP 420) " "is preferred to `pkg_resources.declare_namespace`. " "See https://setuptools.pypa.io/en/latest/references/" "keywords.html#keyword-namespace-packages" ) warnings.warn(msg, DeprecationWarning, stacklevel=2) _imp.acquire_lock() try: if packageName in _namespace_packages: return path: MutableSequence[str] = sys.path parent, _, _ = packageName.rpartition('.') if parent: declare_namespace(parent) if parent not in _namespace_packages: __import__(parent) try: path = sys.modules[parent].__path__ except AttributeError as e: raise TypeError("Not a package:", parent) from e # Track what packages are namespaces, so when new path items are added, # they can be updated _namespace_packages.setdefault(parent or None, []).append(packageName) _namespace_packages.setdefault(packageName, []) for path_item in path: # Ensure all the parent's path items are reflected in the child, # if they apply _handle_ns(packageName, path_item) finally: _imp.release_lock() def fixup_namespace_packages(path_item: str, parent: str | None = None) -> None: """Ensure that previously-declared namespace packages include path_item""" _imp.acquire_lock() try: for package in _namespace_packages.get(parent, ()): subpath = _handle_ns(package, path_item) if subpath: fixup_namespace_packages(subpath, package) finally: _imp.release_lock() def file_ns_handler( importer: object, path_item: StrPath, packageName: str, module: types.ModuleType, ): """Compute an ns-package subpath for a filesystem or zipfile importer""" subpath = os.path.join(path_item, packageName.split('.')[-1]) normalized = _normalize_cached(subpath) for item in module.__path__: if _normalize_cached(item) == normalized: break else: # Only return the path if it's not already there return subpath if hasattr(pkgutil, 'ImpImporter'): register_namespace_handler(pkgutil.ImpImporter, file_ns_handler) register_namespace_handler(zipimport.zipimporter, file_ns_handler) register_namespace_handler(importlib.machinery.FileFinder, file_ns_handler) def null_ns_handler( importer: object, path_item: str | None, packageName: str | None, module: _ModuleLike | None, ) -> None: return None register_namespace_handler(object, null_ns_handler) @overload def normalize_path(filename: StrPath) -> str: ... @overload def normalize_path(filename: BytesPath) -> bytes: ... def normalize_path(filename: StrOrBytesPath) -> str | bytes: """Normalize a file/dir name for comparison purposes""" return os.path.normcase(os.path.realpath(os.path.normpath(_cygwin_patch(filename)))) def _cygwin_patch(filename: StrOrBytesPath): # pragma: nocover """ Contrary to POSIX 2008, on Cygwin, getcwd (3) contains symlink components. Using os.path.abspath() works around this limitation. A fix in os.getcwd() would probably better, in Cygwin even more so, except that this seems to be by design... """ return os.path.abspath(filename) if sys.platform == 'cygwin' else filename if TYPE_CHECKING: # https://github.com/python/mypy/issues/16261 # https://github.com/python/typeshed/issues/6347 @overload def _normalize_cached(filename: StrPath) -> str: ... @overload def _normalize_cached(filename: BytesPath) -> bytes: ... def _normalize_cached(filename: StrOrBytesPath) -> str | bytes: ... else: @functools.cache def _normalize_cached(filename): return normalize_path(filename) def _is_egg_path(path): """ Determine if given path appears to be an egg. """ return _is_zip_egg(path) or _is_unpacked_egg(path) def _is_zip_egg(path): return ( path.lower().endswith('.egg') and os.path.isfile(path) and zipfile.is_zipfile(path) ) def _is_unpacked_egg(path): """ Determine if given path appears to be an unpacked egg. """ return path.lower().endswith('.egg') and os.path.isfile( os.path.join(path, 'EGG-INFO', 'PKG-INFO') ) def _set_parent_ns(packageName) -> None: parts = packageName.split('.') name = parts.pop() if parts: parent = '.'.join(parts) setattr(sys.modules[parent], name, sys.modules[packageName]) MODULE = re.compile(r"\w+(\.\w+)*$").match EGG_NAME = re.compile( r""" (?P[^-]+) ( -(?P[^-]+) ( -py(?P[^-]+) ( -(?P.+) )? )? )? """, re.VERBOSE | re.IGNORECASE, ).match class EntryPoint: """Object representing an advertised importable object""" def __init__( self, name: str, module_name: str, attrs: Iterable[str] = (), extras: Iterable[str] = (), dist: Distribution | None = None, ) -> None: if not MODULE(module_name): raise ValueError("Invalid module name", module_name) self.name = name self.module_name = module_name self.attrs = tuple(attrs) self.extras = tuple(extras) self.dist = dist def __str__(self) -> str: s = f"{self.name} = {self.module_name}" if self.attrs: s += ':' + '.'.join(self.attrs) if self.extras: extras = ','.join(self.extras) s += f' [{extras}]' return s def __repr__(self) -> str: return f"EntryPoint.parse({str(self)!r})" @overload def load( self, require: Literal[True] = True, env: Environment | None = None, installer: _InstallerType | None = None, ) -> _ResolvedEntryPoint: ... @overload def load( self, require: Literal[False], *args: Any, **kwargs: Any, ) -> _ResolvedEntryPoint: ... def load( self, require: bool = True, *args: Environment | _InstallerType | None, **kwargs: Environment | _InstallerType | None, ) -> _ResolvedEntryPoint: """ Require packages for this EntryPoint, then resolve it. """ if not require or args or kwargs: warnings.warn( "Parameters to load are deprecated. Call .resolve and " ".require separately.", PkgResourcesDeprecationWarning, stacklevel=2, ) if require: # We could pass `env` and `installer` directly, # but keeping `*args` and `**kwargs` for backwards compatibility self.require(*args, **kwargs) # type: ignore[arg-type] return self.resolve() def resolve(self) -> _ResolvedEntryPoint: """ Resolve the entry point from its module and attrs. """ module = __import__(self.module_name, fromlist=['__name__'], level=0) try: return functools.reduce(getattr, self.attrs, module) except AttributeError as exc: raise ImportError(str(exc)) from exc def require( self, env: Environment | None = None, installer: _InstallerType | None = None, ) -> None: if not self.dist: error_cls = UnknownExtra if self.extras else AttributeError raise error_cls("Can't require() without a distribution", self) # Get the requirements for this entry point with all its extras and # then resolve them. We have to pass `extras` along when resolving so # that the working set knows what extras we want. Otherwise, for # dist-info distributions, the working set will assume that the # requirements for that extra are purely optional and skip over them. reqs = self.dist.requires(self.extras) items = working_set.resolve(reqs, env, installer, extras=self.extras) list(map(working_set.add, items)) pattern = re.compile( r'\s*' r'(?P.+?)\s*' r'=\s*' r'(?P[\w.]+)\s*' r'(:\s*(?P[\w.]+))?\s*' r'(?P\[.*\])?\s*$' ) @classmethod def parse(cls, src: str, dist: Distribution | None = None) -> Self: """Parse a single entry point from string `src` Entry point syntax follows the form:: name = some.module:some.attr [extra1, extra2] The entry name and module name are required, but the ``:attrs`` and ``[extras]`` parts are optional """ m = cls.pattern.match(src) if not m: msg = "EntryPoint must be in 'name=module:attrs [extras]' format" raise ValueError(msg, src) res = m.groupdict() extras = cls._parse_extras(res['extras']) attrs = res['attr'].split('.') if res['attr'] else () return cls(res['name'], res['module'], attrs, extras, dist) @classmethod def _parse_extras(cls, extras_spec): if not extras_spec: return () req = Requirement.parse('x' + extras_spec) if req.specs: raise ValueError return req.extras @classmethod def parse_group( cls, group: str, lines: _NestedStr, dist: Distribution | None = None, ) -> dict[str, Self]: """Parse an entry point group""" if not MODULE(group): raise ValueError("Invalid group name", group) this: dict[str, Self] = {} for line in yield_lines(lines): ep = cls.parse(line, dist) if ep.name in this: raise ValueError("Duplicate entry point", group, ep.name) this[ep.name] = ep return this @classmethod def parse_map( cls, data: str | Iterable[str] | dict[str, str | Iterable[str]], dist: Distribution | None = None, ) -> dict[str, dict[str, Self]]: """Parse a map of entry point groups""" _data: Iterable[tuple[str | None, str | Iterable[str]]] if isinstance(data, dict): _data = data.items() else: _data = split_sections(data) maps: dict[str, dict[str, Self]] = {} for group, lines in _data: if group is None: if not lines: continue raise ValueError("Entry points must be listed in groups") group = group.strip() if group in maps: raise ValueError("Duplicate group name", group) maps[group] = cls.parse_group(group, lines, dist) return maps def _version_from_file(lines): """ Given an iterable of lines from a Metadata file, return the value of the Version field, if present, or None otherwise. """ def is_version_line(line): return line.lower().startswith('version:') version_lines = filter(is_version_line, lines) line = next(iter(version_lines), '') _, _, value = line.partition(':') return safe_version(value.strip()) or None class Distribution: """Wrap an actual or potential sys.path entry w/metadata""" PKG_INFO = 'PKG-INFO' def __init__( self, location: str | None = None, metadata: _MetadataType = None, project_name: str | None = None, version: str | None = None, py_version: str | None = PY_MAJOR, platform: str | None = None, precedence: int = EGG_DIST, ) -> None: self.project_name = safe_name(project_name or 'Unknown') if version is not None: self._version = safe_version(version) self.py_version = py_version self.platform = platform self.location = location self.precedence = precedence self._provider = metadata or empty_provider @classmethod def from_location( cls, location: str, basename: StrPath, metadata: _MetadataType = None, **kw: int, # We could set `precedence` explicitly, but keeping this as `**kw` for full backwards and subclassing compatibility ) -> Distribution: project_name, version, py_version, platform = [None] * 4 basename, ext = os.path.splitext(basename) if ext.lower() in _distributionImpl: cls = _distributionImpl[ext.lower()] match = EGG_NAME(basename) if match: project_name, version, py_version, platform = match.group( 'name', 'ver', 'pyver', 'plat' ) return cls( location, metadata, project_name=project_name, version=version, py_version=py_version, platform=platform, **kw, )._reload_version() def _reload_version(self): return self @property def hashcmp(self): return ( self._forgiving_parsed_version, self.precedence, self.key, self.location, self.py_version or '', self.platform or '', ) def __hash__(self) -> int: return hash(self.hashcmp) def __lt__(self, other: Distribution) -> bool: return self.hashcmp < other.hashcmp def __le__(self, other: Distribution) -> bool: return self.hashcmp <= other.hashcmp def __gt__(self, other: Distribution) -> bool: return self.hashcmp > other.hashcmp def __ge__(self, other: Distribution) -> bool: return self.hashcmp >= other.hashcmp def __eq__(self, other: object) -> bool: if not isinstance(other, self.__class__): # It's not a Distribution, so they are not equal return False return self.hashcmp == other.hashcmp def __ne__(self, other: object) -> bool: return not self == other # These properties have to be lazy so that we don't have to load any # metadata until/unless it's actually needed. (i.e., some distributions # may not know their name or version without loading PKG-INFO) @property def key(self): try: return self._key except AttributeError: self._key = key = self.project_name.lower() return key @property def parsed_version(self): if not hasattr(self, "_parsed_version"): try: self._parsed_version = parse_version(self.version) except packaging.version.InvalidVersion as ex: info = f"(package: {self.project_name})" if hasattr(ex, "add_note"): ex.add_note(info) # PEP 678 raise raise packaging.version.InvalidVersion(f"{str(ex)} {info}") from None return self._parsed_version @property def _forgiving_parsed_version(self): try: return self.parsed_version except packaging.version.InvalidVersion as ex: self._parsed_version = parse_version(_forgiving_version(self.version)) notes = "\n".join(getattr(ex, "__notes__", [])) # PEP 678 msg = f"""!!\n\n ************************************************************************* {str(ex)}\n{notes} This is a long overdue deprecation. For the time being, `pkg_resources` will use `{self._parsed_version}` as a replacement to avoid breaking existing environments, but no future compatibility is guaranteed. If you maintain package {self.project_name} you should implement the relevant changes to adequate the project to PEP 440 immediately. ************************************************************************* \n\n!! """ warnings.warn(msg, DeprecationWarning) return self._parsed_version @property def version(self): try: return self._version except AttributeError as e: version = self._get_version() if version is None: path = self._get_metadata_path_for_display(self.PKG_INFO) msg = f"Missing 'Version:' header and/or {self.PKG_INFO} file at path: {path}" raise ValueError(msg, self) from e return version @property def _dep_map(self): """ A map of extra to its list of (direct) requirements for this distribution, including the null extra. """ try: return self.__dep_map except AttributeError: self.__dep_map = self._filter_extras(self._build_dep_map()) return self.__dep_map @staticmethod def _filter_extras( dm: dict[str | None, list[Requirement]], ) -> dict[str | None, list[Requirement]]: """ Given a mapping of extras to dependencies, strip off environment markers and filter out any dependencies not matching the markers. """ for extra in list(filter(None, dm)): new_extra: str | None = extra reqs = dm.pop(extra) new_extra, _, marker = extra.partition(':') fails_marker = marker and ( invalid_marker(marker) or not evaluate_marker(marker) ) if fails_marker: reqs = [] new_extra = safe_extra(new_extra) or None dm.setdefault(new_extra, []).extend(reqs) return dm def _build_dep_map(self): dm = {} for name in 'requires.txt', 'depends.txt': for extra, reqs in split_sections(self._get_metadata(name)): dm.setdefault(extra, []).extend(parse_requirements(reqs)) return dm def requires(self, extras: Iterable[str] = ()) -> list[Requirement]: """List of Requirements needed for this distro if `extras` are used""" dm = self._dep_map deps: list[Requirement] = [] deps.extend(dm.get(None, ())) for ext in extras: try: deps.extend(dm[safe_extra(ext)]) except KeyError as e: raise UnknownExtra(f"{self} has no such extra feature {ext!r}") from e return deps def _get_metadata_path_for_display(self, name): """ Return the path to the given metadata file, if available. """ try: # We need to access _get_metadata_path() on the provider object # directly rather than through this class's __getattr__() # since _get_metadata_path() is marked private. path = self._provider._get_metadata_path(name) # Handle exceptions e.g. in case the distribution's metadata # provider doesn't support _get_metadata_path(). except Exception: return '[could not detect]' return path def _get_metadata(self, name): if self.has_metadata(name): yield from self.get_metadata_lines(name) def _get_version(self): lines = self._get_metadata(self.PKG_INFO) return _version_from_file(lines) def activate(self, path: list[str] | None = None, replace: bool = False) -> None: """Ensure distribution is importable on `path` (default=sys.path)""" if path is None: path = sys.path self.insert_on(path, replace=replace) if path is sys.path and self.location is not None: fixup_namespace_packages(self.location) for pkg in self._get_metadata('namespace_packages.txt'): if pkg in sys.modules: declare_namespace(pkg) def egg_name(self): """Return what this distribution's standard .egg filename should be""" filename = f"{to_filename(self.project_name)}-{to_filename(self.version)}-py{self.py_version or PY_MAJOR}" if self.platform: filename += '-' + self.platform return filename def __repr__(self) -> str: if self.location: return f"{self} ({self.location})" else: return str(self) def __str__(self) -> str: try: version = getattr(self, 'version', None) except ValueError: version = None version = version or "[unknown version]" return f"{self.project_name} {version}" def __getattr__(self, attr: str): """Delegate all unrecognized public attributes to .metadata provider""" if attr.startswith('_'): raise AttributeError(attr) return getattr(self._provider, attr) def __dir__(self): return list( set(super().__dir__()) | set(attr for attr in self._provider.__dir__() if not attr.startswith('_')) ) @classmethod def from_filename( cls, filename: StrPath, metadata: _MetadataType = None, **kw: int, # We could set `precedence` explicitly, but keeping this as `**kw` for full backwards and subclassing compatibility ) -> Distribution: return cls.from_location( _normalize_cached(filename), os.path.basename(filename), metadata, **kw ) def as_requirement(self): """Return a ``Requirement`` that matches this distribution exactly""" if isinstance(self.parsed_version, packaging.version.Version): spec = f"{self.project_name}=={self.parsed_version}" else: spec = f"{self.project_name}==={self.parsed_version}" return Requirement.parse(spec) def load_entry_point(self, group: str, name: str) -> _ResolvedEntryPoint: """Return the `name` entry point of `group` or raise ImportError""" ep = self.get_entry_info(group, name) if ep is None: raise ImportError(f"Entry point {(group, name)!r} not found") return ep.load() @overload def get_entry_map(self, group: None = None) -> dict[str, dict[str, EntryPoint]]: ... @overload def get_entry_map(self, group: str) -> dict[str, EntryPoint]: ... def get_entry_map(self, group: str | None = None): """Return the entry point map for `group`, or the full entry map""" if not hasattr(self, "_ep_map"): self._ep_map = EntryPoint.parse_map( self._get_metadata('entry_points.txt'), self ) if group is not None: return self._ep_map.get(group, {}) return self._ep_map def get_entry_info(self, group: str, name: str) -> EntryPoint | None: """Return the EntryPoint object for `group`+`name`, or ``None``""" return self.get_entry_map(group).get(name) # FIXME: 'Distribution.insert_on' is too complex (13) def insert_on( # noqa: C901 self, path: list[str], loc=None, replace: bool = False, ) -> None: """Ensure self.location is on path If replace=False (default): - If location is already in path anywhere, do nothing. - Else: - If it's an egg and its parent directory is on path, insert just ahead of the parent. - Else: add to the end of path. If replace=True: - If location is already on path anywhere (not eggs) or higher priority than its parent (eggs) do nothing. - Else: - If it's an egg and its parent directory is on path, insert just ahead of the parent, removing any lower-priority entries. - Else: add it to the front of path. """ loc = loc or self.location if not loc: return nloc = _normalize_cached(loc) bdir = os.path.dirname(nloc) npath = [(p and _normalize_cached(p) or p) for p in path] for p, item in enumerate(npath): if item == nloc: if replace: break else: # don't modify path (even removing duplicates) if # found and not replace return elif item == bdir and self.precedence == EGG_DIST: # if it's an .egg, give it precedence over its directory # UNLESS it's already been added to sys.path and replace=False if (not replace) and nloc in npath[p:]: return if path is sys.path: self.check_version_conflict() path.insert(p, loc) npath.insert(p, nloc) break else: if path is sys.path: self.check_version_conflict() if replace: path.insert(0, loc) else: path.append(loc) return # p is the spot where we found or inserted loc; now remove duplicates while True: try: np = npath.index(nloc, p + 1) except ValueError: break else: del npath[np], path[np] # ha! p = np return def check_version_conflict(self): if self.key == 'setuptools': # ignore the inevitable setuptools self-conflicts :( return nsp = dict.fromkeys(self._get_metadata('namespace_packages.txt')) loc = normalize_path(self.location) for modname in self._get_metadata('top_level.txt'): if ( modname not in sys.modules or modname in nsp or modname in _namespace_packages ): continue if modname in ('pkg_resources', 'setuptools', 'site'): continue fn = getattr(sys.modules[modname], '__file__', None) if fn and ( normalize_path(fn).startswith(loc) or fn.startswith(self.location) ): continue issue_warning( f"Module {modname} was already imported from {fn}, " f"but {self.location} is being added to sys.path", ) def has_version(self) -> bool: try: self.version except ValueError: issue_warning("Unbuilt egg for " + repr(self)) return False except SystemError: # TODO: remove this except clause when python/cpython#103632 is fixed. return False return True def clone(self, **kw: str | int | IResourceProvider | None) -> Self: """Copy this distribution, substituting in any changed keyword args""" names = 'project_name version py_version platform location precedence' for attr in names.split(): kw.setdefault(attr, getattr(self, attr, None)) kw.setdefault('metadata', self._provider) # Unsafely unpacking. But keeping **kw for backwards and subclassing compatibility return self.__class__(**kw) # type:ignore[arg-type] @property def extras(self): return [dep for dep in self._dep_map if dep] class EggInfoDistribution(Distribution): def _reload_version(self): """ Packages installed by distutils (e.g. numpy or scipy), which uses an old safe_version, and so their version numbers can get mangled when converted to filenames (e.g., 1.11.0.dev0+2329eae to 1.11.0.dev0_2329eae). These distributions will not be parsed properly downstream by Distribution and safe_version, so take an extra step and try to get the version number from the metadata file itself instead of the filename. """ md_version = self._get_version() if md_version: self._version = md_version return self class DistInfoDistribution(Distribution): """ Wrap an actual or potential sys.path entry w/metadata, .dist-info style. """ PKG_INFO = 'METADATA' EQEQ = re.compile(r"([\(,])\s*(\d.*?)\s*([,\)])") @property def _parsed_pkg_info(self): """Parse and cache metadata""" try: return self._pkg_info except AttributeError: metadata = self.get_metadata(self.PKG_INFO) self._pkg_info = email.parser.Parser().parsestr(metadata) return self._pkg_info @property def _dep_map(self): try: return self.__dep_map except AttributeError: self.__dep_map = self._compute_dependencies() return self.__dep_map def _compute_dependencies(self) -> dict[str | None, list[Requirement]]: """Recompute this distribution's dependencies.""" self.__dep_map: dict[str | None, list[Requirement]] = {None: []} reqs: list[Requirement] = [] # Including any condition expressions for req in self._parsed_pkg_info.get_all('Requires-Dist') or []: reqs.extend(parse_requirements(req)) def reqs_for_extra(extra): for req in reqs: if not req.marker or req.marker.evaluate({'extra': extra}): yield req common = types.MappingProxyType(dict.fromkeys(reqs_for_extra(None))) self.__dep_map[None].extend(common) for extra in self._parsed_pkg_info.get_all('Provides-Extra') or []: s_extra = safe_extra(extra.strip()) self.__dep_map[s_extra] = [ r for r in reqs_for_extra(extra) if r not in common ] return self.__dep_map _distributionImpl = { '.egg': Distribution, '.egg-info': EggInfoDistribution, '.dist-info': DistInfoDistribution, } def issue_warning(*args, **kw): level = 1 g = globals() try: # find the first stack frame that is *not* code in # the pkg_resources module, to use for the warning while sys._getframe(level).f_globals is g: level += 1 except ValueError: pass warnings.warn(stacklevel=level + 1, *args, **kw) def parse_requirements(strs: _NestedStr) -> map[Requirement]: """ Yield ``Requirement`` objects for each specification in `strs`. `strs` must be a string, or a (possibly-nested) iterable thereof. """ return map(Requirement, join_continuation(map(drop_comment, yield_lines(strs)))) class RequirementParseError(packaging.requirements.InvalidRequirement): "Compatibility wrapper for InvalidRequirement" class Requirement(packaging.requirements.Requirement): # prefer variable length tuple to set (as found in # packaging.requirements.Requirement) extras: tuple[str, ...] # type: ignore[assignment] def __init__(self, requirement_string: str) -> None: """DO NOT CALL THIS UNDOCUMENTED METHOD; use Requirement.parse()!""" super().__init__(requirement_string) self.unsafe_name = self.name project_name = safe_name(self.name) self.project_name, self.key = project_name, project_name.lower() self.specs = [(spec.operator, spec.version) for spec in self.specifier] self.extras = tuple(map(safe_extra, self.extras)) self.hashCmp = ( self.key, self.url, self.specifier, frozenset(self.extras), str(self.marker) if self.marker else None, ) self.__hash = hash(self.hashCmp) def __eq__(self, other: object) -> bool: return isinstance(other, Requirement) and self.hashCmp == other.hashCmp def __ne__(self, other: object) -> bool: return not self == other def __contains__( self, item: Distribution | packaging.specifiers.UnparsedVersion ) -> bool: if isinstance(item, Distribution): if item.key != self.key: return False version = item.version else: version = item # Allow prereleases always in order to match the previous behavior of # this method. In the future this should be smarter and follow PEP 440 # more accurately. return self.specifier.contains( version, prereleases=True, ) def __hash__(self) -> int: return self.__hash def __repr__(self) -> str: return f"Requirement.parse({str(self)!r})" @staticmethod def parse(s: str | Iterable[str]) -> Requirement: (req,) = parse_requirements(s) return req def _always_object(classes): """ Ensure object appears in the mro even for old-style classes. """ if object not in classes: return classes + (object,) return classes def _find_adapter(registry: Mapping[type, _AdapterT], ob: object) -> _AdapterT: """Return an adapter factory for `ob` from `registry`""" types = _always_object(inspect.getmro(getattr(ob, '__class__', type(ob)))) for t in types: if t in registry: return registry[t] # _find_adapter would previously return None, and immediately be called. # So we're raising a TypeError to keep backward compatibility if anyone depended on that behaviour. raise TypeError(f"Could not find adapter for {registry} and {ob}") def ensure_directory(path: StrOrBytesPath) -> None: """Ensure that the parent directory of `path` exists""" dirname = os.path.dirname(path) os.makedirs(dirname, exist_ok=True) def _bypass_ensure_directory(path) -> None: """Sandbox-bypassing version of ensure_directory()""" if not WRITE_SUPPORT: raise OSError('"os.mkdir" not supported on this platform.') dirname, filename = split(path) if dirname and filename and not isdir(dirname): _bypass_ensure_directory(dirname) try: mkdir(dirname, 0o755) except FileExistsError: pass def split_sections(s: _NestedStr) -> Iterator[tuple[str | None, list[str]]]: """Split a string or iterable thereof into (section, content) pairs Each ``section`` is a stripped version of the section header ("[section]") and each ``content`` is a list of stripped lines excluding blank lines and comment-only lines. If there are any such lines before the first section header, they're returned in a first ``section`` of ``None``. """ section = None content: list[str] = [] for line in yield_lines(s): if line.startswith("["): if line.endswith("]"): if section or content: yield section, content section = line[1:-1].strip() content = [] else: raise ValueError("Invalid section heading", line) else: content.append(line) # wrap up last segment yield section, content def _mkstemp(*args, **kw): old_open = os.open try: # temporarily bypass sandboxing os.open = os_open return tempfile.mkstemp(*args, **kw) finally: # and then put it back os.open = old_open # Silence the PEP440Warning by default, so that end users don't get hit by it # randomly just because they use pkg_resources. We want to append the rule # because we want earlier uses of filterwarnings to take precedence over this # one. warnings.filterwarnings("ignore", category=PEP440Warning, append=True) class PkgResourcesDeprecationWarning(Warning): """ Base class for warning about deprecations in ``pkg_resources`` This class is not derived from ``DeprecationWarning``, and as such is visible by default. """ # Ported from ``setuptools`` to avoid introducing an import inter-dependency: _LOCALE_ENCODING = "locale" if sys.version_info >= (3, 10) else None # This must go before calls to `_call_aside`. See https://github.com/pypa/setuptools/pull/4422 def _read_utf8_with_fallback(file: str, fallback_encoding=_LOCALE_ENCODING) -> str: """See setuptools.unicode_utils._read_utf8_with_fallback""" try: with open(file, "r", encoding="utf-8") as f: return f.read() except UnicodeDecodeError: # pragma: no cover msg = f"""\ ******************************************************************************** `encoding="utf-8"` fails with {file!r}, trying `encoding={fallback_encoding!r}`. This fallback behaviour is considered **deprecated** and future versions of `setuptools/pkg_resources` may not implement it. Please encode {file!r} with "utf-8" to ensure future builds will succeed. If this file was produced by `setuptools` itself, cleaning up the cached files and re-building/re-installing the package with a newer version of `setuptools` (e.g. by updating `build-system.requires` in its `pyproject.toml`) might solve the problem. ******************************************************************************** """ # TODO: Add a deadline? # See comment in setuptools.unicode_utils._Utf8EncodingNeeded warnings.warn(msg, PkgResourcesDeprecationWarning, stacklevel=2) with open(file, "r", encoding=fallback_encoding) as f: return f.read() # from jaraco.functools 1.3 def _call_aside(f, *args, **kwargs): f(*args, **kwargs) return f @_call_aside def _initialize(g=globals()) -> None: "Set up global resource manager (deliberately not state-saved)" manager = ResourceManager() g['_manager'] = manager g.update( (name, getattr(manager, name)) for name in dir(manager) if not name.startswith('_') ) @_call_aside def _initialize_master_working_set() -> None: """ Prepare the master working set and make the ``require()`` API available. This function has explicit effects on the global state of pkg_resources. It is intended to be invoked once at the initialization of this module. Invocation by other packages is unsupported and done at their own risk. """ working_set = _declare_state('object', 'working_set', WorkingSet._build_master()) require = working_set.require iter_entry_points = working_set.iter_entry_points add_activation_listener = working_set.subscribe run_script = working_set.run_script # backward compatibility run_main = run_script # Activate all distributions already on sys.path with replace=False and # ensure that all distributions added to the working set in the future # (e.g. by calling ``require()``) will get activated as well, # with higher priority (replace=True). tuple(dist.activate(replace=False) for dist in working_set) add_activation_listener( lambda dist: dist.activate(replace=True), existing=False, ) working_set.entries = [] # match order list(map(working_set.add_entry, sys.path)) globals().update(locals()) if TYPE_CHECKING: # All of these are set by the @_call_aside methods above __resource_manager = ResourceManager() # Won't exist at runtime resource_exists = __resource_manager.resource_exists resource_isdir = __resource_manager.resource_isdir resource_filename = __resource_manager.resource_filename resource_stream = __resource_manager.resource_stream resource_string = __resource_manager.resource_string resource_listdir = __resource_manager.resource_listdir set_extraction_path = __resource_manager.set_extraction_path cleanup_resources = __resource_manager.cleanup_resources working_set = WorkingSet() require = working_set.require iter_entry_points = working_set.iter_entry_points add_activation_listener = working_set.subscribe run_script = working_set.run_script run_main = run_script ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.400134 zc_buildout-6.0.1/src/zc/buildout/buildout.py0000644000076500000240000026251315264141650020720 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2005-2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Buildout main script """ from collections.abc import MutableMapping as DictMixin from functools import partial from hashlib import md5 as md5_original from packaging import utils as packaging_utils from zc.buildout.rmtree import rmtree import zc.buildout.easy_install import zc.buildout.utils import zc.buildout.configparser import copy import datetime import distutils.errors import glob import importlib import inspect import itertools import logging import os import pkg_resources import re import shutil import subprocess import sys import tempfile import zc.buildout import zc.buildout.download try: hashed = md5_original(b'test') md5 = md5_original except ValueError: md5 = partial(md5_original, usedforsecurity=False) def command(method): method.buildout_command = True return method def commands(cls): for name, method in cls.__dict__.items(): if hasattr(method, "buildout_command"): cls.COMMANDS.add(name) return cls def _print_options(sep=' ', end='\n', file=None): return sep, end, file def print_(*args, **kw): sep, end, file = _print_options(**kw) if file is None: file = sys.stdout file.write(sep.join(map(str, args))+end) realpath = zc.buildout.easy_install.realpath _isurl = re.compile('([a-zA-Z0-9+.-]+)://').match class MissingOption(zc.buildout.UserError, KeyError): """A required option was missing. """ class MissingSection(zc.buildout.UserError, KeyError): """A required section is missing. """ def __str__(self): return "The referenced section, %r, was not defined." % self.args[0] def _annotate_section(section, source): for key in section: section[key] = SectionKey(section[key], source) return section class SectionKey(object): def __init__(self, value, source): self.history = [] self.value = value self.addToHistory("SET", value, source) @property def source(self): return self.history[-1].source def overrideValue(self, sectionkey): self.value = sectionkey.value if sectionkey.history[-1].operation not in ['ADD', 'REMOVE']: self.addToHistory("OVERRIDE", sectionkey.value, sectionkey.source) else: self.history = copy.deepcopy(sectionkey.history) def setDirectory(self, value): self.value = value self.addToHistory("DIRECTORY", value, self.source) def addToValue(self, added, source): subvalues = self.value.split('\n') + added.split('\n') self.value = "\n".join(subvalues) self.addToHistory("ADD", added, source) def removeFromValue(self, removed, source): subvalues = [ v for v in self.value.split('\n') if v not in removed.split('\n') ] self.value = "\n".join(subvalues) self.addToHistory("REMOVE", removed, source) def addToHistory(self, operation, value, source): item = HistoryItem(operation, value, source) self.history.append(item) def printAll(self, key, basedir, verbose): self.printKeyAndValue(key) if verbose: self.printVerbose(basedir) else: self.printTerse(basedir) def printKeyAndValue(self, key): lines = self.value.splitlines() if len(lines) <= 1: args = [key, "="] if self.value: args.append(" ") args.append(self.value) print_(*args, sep='') else: print_(key, "= ", lines[0], sep='') for line in lines[1:]: print_(line) def printVerbose(self, basedir): print_() for item in reversed(self.history): item.printAll(basedir) print_() def printTerse(self, basedir): toprint = [] history = copy.deepcopy(self.history) while history: next = history.pop() if next.operation in ["ADD", "REMOVE"]: next.printShort(toprint, basedir) else: next.printShort(toprint, basedir) break for line in reversed(toprint): if line.strip(): print_(line) def __repr__(self): return "" % ( " ".join(self.value.split('\n')), self.source) class HistoryItem(object): def __init__(self, operation, value, source): self.operation = operation self.value = value self.source = source def printShort(self, toprint, basedir): source = self.source_for_human(basedir) if self.operation in ["OVERRIDE", "SET", "DIRECTORY"]: toprint.append(" " + source) elif self.operation == "ADD": toprint.append("+= " + source) elif self.operation == "REMOVE": toprint.append("-= " + source) def printOperation(self): lines = self.value.splitlines() if len(lines) <= 1: print_(" ", self.operation, "VALUE =", self.value) else: print_(" ", self.operation, "VALUE =") for line in lines: print_(" ", " ", line) def printSource(self, basedir): if self.source in ( 'DEFAULT_VALUE', 'COMPUTED_VALUE', 'COMMAND_LINE_VALUE' ): prefix = "AS" else: prefix = "IN" print_(" ", prefix, self.source_for_human(basedir)) def source_for_human(self, basedir): if self.source.startswith(basedir): return os.path.relpath(self.source, basedir) else: return self.source def printAll(self, basedir): self.printSource(basedir) self.printOperation() def __repr__(self): return "" % ( self.operation, " ".join(self.value.split('\n')), self.source) def _annotate(data, note): for key in data: data[key] = _annotate_section(data[key], note) return data def _print_annotate(data, verbose, chosen_sections, basedir): sections = list(data.keys()) sections.sort() print_() print_("Annotated sections") print_("="*len("Annotated sections")) for section in sections: if (not chosen_sections) or (section in chosen_sections): print_() print_('[%s]' % section) keys = list(data[section].keys()) keys.sort() for key in keys: sectionkey = data[section][key] sectionkey.printAll(key, basedir, verbose) def _unannotate_section(section): return {key: entry.value for key, entry in section.items()} def _unannotate(data): return {key: _unannotate_section(section) for key, section in data.items()} def _format_picked_versions(picked_versions, required_by): output = ['[versions]'] required_output = [] for dist_, version in picked_versions: if dist_ in required_by: required_output.append('') required_output.append('# Required by:') for req_ in sorted(required_by[dist_]): required_output.append('# '+req_) target = required_output else: target = output target.append("%s = %s" % (dist_, version)) output.extend(required_output) return output _buildout_default_options = _annotate_section({ 'allow-hosts': '*', 'allow-picked-versions': 'true', 'bin-directory': 'bin', 'develop-eggs-directory': 'develop-eggs', 'eggs-directory': 'eggs', 'eggs-directory-version': 'v5', 'executable': sys.executable, 'find-links': '', 'install-from-cache': 'false', 'installed': '.installed.cfg', 'log-format': '', 'log-level': 'INFO', 'newest': 'true', 'offline': 'false', 'parts-directory': 'parts', 'prefer-final': 'true', 'python': 'buildout', 'show-picked-versions': 'false', 'socket-timeout': '', 'update-versions-file': '', 'use-dependency-links': 'true', 'allow-unknown-extras': 'false', }, 'DEFAULT_VALUE') def _get_user_config(): buildout_home = os.path.join(os.path.expanduser('~'), '.buildout') buildout_home = os.environ.get('BUILDOUT_HOME', buildout_home) return os.path.join(buildout_home, 'default.cfg') @commands class Buildout(DictMixin): COMMANDS = set() def __init__(self, config_file, cloptions, use_user_defaults=True, command=None, args=()): __doing__ = 'Initializing.' # default options _buildout_default_options_copy = copy.deepcopy( _buildout_default_options) data = dict(buildout=_buildout_default_options_copy) self._buildout_dir = os.getcwd() if config_file and not _isurl(config_file): config_file = os.path.abspath(config_file) if not os.path.exists(config_file): if command == 'init': self._init_config(config_file, args) elif command == 'setup': # Sigh. This model of a buildout instance # with methods is breaking down. :( config_file = None data['buildout']['directory'] = SectionKey( '.', 'COMPUTED_VALUE') else: raise zc.buildout.UserError( "Couldn't open %s" % config_file) elif command == 'init': raise zc.buildout.UserError( "%r already exists." % config_file) if config_file: data['buildout']['directory'] = SectionKey( os.path.dirname(config_file), 'COMPUTED_VALUE') cloptions = dict( (section, dict((option, SectionKey(value, 'COMMAND_LINE_VALUE')) for (_, option, value) in v)) for (section, v) in itertools.groupby(sorted(cloptions), lambda v: v[0]) ) override = copy.deepcopy(cloptions.get('buildout', {})) # load user defaults, which override defaults user_config = _get_user_config() if use_user_defaults and os.path.exists(user_config): download_options = data['buildout'] user_defaults, _ = _open( os.path.dirname(user_config), user_config, [], download_options, override, set(), {} ) for_download_options = _update(data, user_defaults) else: user_defaults = {} for_download_options = copy.deepcopy(data) # load configuration files if config_file: download_options = for_download_options['buildout'] cfg_data, _ = _open( os.path.dirname(config_file), config_file, [], download_options, override, set(), user_defaults ) data = _update(data, cfg_data) # extends from command-line if 'buildout' in cloptions: cl_extends = cloptions['buildout'].pop('extends', None) if cl_extends: for extends in cl_extends.value.split(): download_options = for_download_options['buildout'] cfg_data, _ = _open( os.path.dirname(extends), os.path.basename(extends), [], download_options, override, set(), user_defaults ) data = _update(data, cfg_data) # apply command-line options data = _update(data, cloptions) # Set up versions section, if necessary if 'versions' not in data['buildout']: data['buildout']['versions'] = SectionKey( 'versions', 'DEFAULT_VALUE') if 'versions' not in data: data['versions'] = {} # Default versions: versions_section_name = data['buildout']['versions'].value if versions_section_name: versions = data[versions_section_name] else: versions = {} if 'zc.buildout' not in versions: # Prevent downgrading of zc.buildout itself due to prefer-final. ws = pkg_resources.working_set dist = ws.find( pkg_resources.Requirement.parse('zc-buildout') ) if dist is None: # older setuptools dist = ws.find( pkg_resources.Requirement.parse('zc.buildout') ) if dist is None: # This would be really strange, but I prefer an explicit # failure here over an unclear error later. raise ValueError( "Could not find distribution for zc.buildout in working set." ) minimum = dist.version versions['zc.buildout'] = SectionKey(f'>={minimum}', 'DEFAULT_VALUE') if 'zc.recipe.egg' not in versions: # zc.buildout and zc.recipe egg are closely linked, but zc.buildout # does NOT depend on it: we do not want to add it to our # install_requires. (One could debate why, although one answer # would be to avoid a circular dependency. Maybe we could merge them, # as I see no use case for Buildout without recipes. But we would # need to update the zc.recipe.egg test setup first.) # # Anyway: we use a different way to set a minimum version. # Originally (in 2013, zc.buildout 2.0.0b1) we made sure # zc.recipe.egg>=2.0.0a3 was pinned, mostly to avoid problems # when prefer-final is true. # Later (in 2018, zc.buildout 2.12.1) we updated the minimum version # to 2.0.6, to avoid a KeyError: 'allow-unknown-extras'. # See https://github.com/buildout/buildout/pull/461 # I wonder if we really need a minimum version, as older versions # are unlikely to even be installable by supported Python versions. # But if we ever really need a more recent minimum version, # it is easy to update a version here. versions['zc.recipe.egg'] = SectionKey('>=2.0.6', 'DEFAULT_VALUE') # Absolutize some particular directory, handling also the ~/foo form, # and considering the location of the configuration file that generated # the setting as the base path, falling back to the main configuration # file location for name in ('download-cache', 'eggs-directory', 'extends-cache'): if name in data['buildout']: sectionkey = data['buildout'][name] origdir = sectionkey.value src = sectionkey.source if '${' in origdir: continue if not os.path.isabs(origdir): if src in ('DEFAULT_VALUE', 'COMPUTED_VALUE', 'COMMAND_LINE_VALUE'): if 'directory' in data['buildout']: basedir = data['buildout']['directory'].value else: basedir = self._buildout_dir else: if _isurl(src): raise zc.buildout.UserError( 'Setting "%s" to a non absolute location ("%s") ' 'within a\n' 'remote configuration file ("%s") is ambiguous.' % ( name, origdir, src)) basedir = os.path.dirname(src) absdir = os.path.expanduser(origdir) if not os.path.isabs(absdir): absdir = os.path.join(basedir, absdir) absdir = os.path.abspath(absdir) sectionkey.setDirectory(absdir) self._annotated = copy.deepcopy(data) self._raw = _unannotate(data) self._data = {} self._parts = [] # provide some defaults before options are parsed # because while parsing options those attributes might be # used already (Gottfried Ganssauge) buildout_section = self._raw['buildout'] # Try to make sure we have absolute paths for standard # directories. We do this before doing substitutions, in case # a one of these gets read by another section. If any # variable references are used though, we leave it as is in # _buildout_path. if 'directory' in buildout_section: self._buildout_dir = buildout_section['directory'] for name in ('bin', 'parts', 'eggs', 'develop-eggs'): d = self._buildout_path(buildout_section[name+'-directory']) buildout_section[name+'-directory'] = d # Attributes on this buildout object shouldn't be used by # recipes in their __init__. It can cause bugs, because the # recipes will be instantiated below (``options = self['buildout']``) # before this has completed initializing. These attributes are # left behind for legacy support but recipe authors should # beware of using them. A better practice is for a recipe to # use the buildout['buildout'] options. links = buildout_section['find-links'] self._links = links and links.split() or () allow_hosts = buildout_section['allow-hosts'].split('\n') self._allow_hosts = tuple([host.strip() for host in allow_hosts if host.strip() != '']) self._logger = logging.getLogger('zc.buildout') self.offline = bool_option(buildout_section, 'offline') self.newest = ((not self.offline) and bool_option(buildout_section, 'newest') ) ################################################################## ## WARNING!!! ## ALL ATTRIBUTES MUST HAVE REASONABLE DEFAULTS AT THIS POINT ## OTHERWISE ATTRIBUTEERRORS MIGHT HAPPEN ANY TIME FROM RECIPES. ## RECIPES SHOULD GENERALLY USE buildout['buildout'] OPTIONS, NOT ## BUILDOUT ATTRIBUTES. ################################################################## # initialize some attrs and buildout directories. options = self['buildout'] # now reinitialize links = options.get('find-links', '') self._links = links and links.split() or () allow_hosts = options['allow-hosts'].split('\n') self._allow_hosts = tuple([host.strip() for host in allow_hosts if host.strip() != '']) self._buildout_dir = options['directory'] # Make sure we have absolute paths for standard directories. We do this # a second time here in case someone overrode these in their configs. for name in ('bin', 'parts', 'eggs', 'develop-eggs'): d = self._buildout_path(options[name+'-directory']) options[name+'-directory'] = d if options['installed']: options['installed'] = os.path.join(options['directory'], options['installed']) self._setup_logging() self._setup_socket_timeout() # finish w versions if versions_section_name: # refetching section name just to avoid a warning versions = self[versions_section_name] else: # remove annotations versions = dict((k, v.value) for (k, v) in versions.items()) options['versions'] # refetching section name just to avoid a warning self.versions = versions zc.buildout.easy_install.default_versions(versions) zc.buildout.easy_install.prefer_final( bool_option(options, 'prefer-final')) zc.buildout.easy_install.use_dependency_links( bool_option(options, 'use-dependency-links')) zc.buildout.easy_install.index_url(options.get('index', '').strip()) zc.buildout.easy_install.allow_picked_versions( bool_option(options, 'allow-picked-versions')) self.show_picked_versions = bool_option(options, 'show-picked-versions') self.update_versions_file = options['update-versions-file'] zc.buildout.easy_install.store_required_by(self.show_picked_versions or self.update_versions_file) download_cache = options.get('download-cache') extends_cache = options.get('extends-cache') # Since zc.buildout version 5 we maintain separate directories for each # buildout eggs format version. Current idea: we use v5 from zc.buildout # 5.x onwards. Later versions will likely also use v5, as the current # expectation is that they will be compatible, just like zc.buildout # 1.x through 4.x are compatible. # If you know what you are doing, you can set eggs-directory-version to # an empty string. This can be fine if you don't have any previous eggs # and only use zc.buildout 5 or later. It should also be fine in case # you don't use any namespace packages; but you would be wrong, because # you are using zc.buildout and probably zc.recipe.egg, so you use the # zc namespace. Still, if those are the only two packages, it might # possibly work. if options['eggs-directory-version']: options['eggs-directory'] = os.path.join( options['eggs-directory'], options['eggs-directory-version']) if bool_option(options, 'abi-tag-eggs', 'false'): from zc.buildout.pep425tags import get_abi_tag options['eggs-directory'] = os.path.join( options['eggs-directory'], get_abi_tag()) eggs_cache = options.get('eggs-directory') for cache in [download_cache, extends_cache, eggs_cache]: if cache: cache = os.path.join(options['directory'], cache) if not os.path.exists(cache): self._logger.info('Creating directory %r.', cache) os.makedirs(cache) if download_cache: # Actually, we want to use a subdirectory in there called 'dist'. download_cache = os.path.join(download_cache, 'dist') if not os.path.exists(download_cache): os.mkdir(download_cache) zc.buildout.easy_install.download_cache(download_cache) if bool_option(options, 'install-from-cache'): if self.offline: raise zc.buildout.UserError( "install-from-cache can't be used with offline mode.\n" "Nothing is installed, even from cache, in offline\n" "mode, which might better be called 'no-install mode'.\n" ) zc.buildout.easy_install.install_from_cache(True) # "Use" each of the defaults so they aren't reported as unused options. for name in _buildout_default_options: options[name] os.chdir(options['directory']) def _buildout_path(self, name): if '${' in name: return name return os.path.join(self._buildout_dir, name) @command def bootstrap(self, args): __doing__ = 'Bootstrapping.' if os.path.exists(self['buildout']['develop-eggs-directory']): if os.path.isdir(self['buildout']['develop-eggs-directory']): rmtree(self['buildout']['develop-eggs-directory']) self._logger.debug( "Removed existing develop-eggs directory") self._setup_directories() # Now copy buildout and setuptools eggs, and record destination eggs: entries = [] for dist in zc.buildout.easy_install.buildout_and_setuptools_dists: if dist.precedence == pkg_resources.DEVELOP_DIST: dest = os.path.join(self['buildout']['develop-eggs-directory'], dist.key + '.egg-link') with open(dest, 'w') as fh: fh.write(dist.location) entries.append(dist.location) else: dest = os.path.join(self['buildout']['eggs-directory'], os.path.basename(dist.location)) entries.append(dest) if not os.path.exists(dest): if os.path.isdir(dist.location): shutil.copytree(dist.location, dest) else: shutil.copy2(dist.location, dest) # Create buildout script ws = pkg_resources.WorkingSet(entries) ws.require('zc.buildout') options = self['buildout'] eggs_dir = options['eggs-directory'] develop_eggs_dir = options['develop-eggs-directory'] ws = zc.buildout.easy_install.sort_working_set( ws, eggs_dir=eggs_dir, develop_eggs_dir=develop_eggs_dir ) zc.buildout.easy_install.scripts( ['zc.buildout'], ws, sys.executable, options['bin-directory'], relative_paths = ( bool_option(options, 'relative-paths', False) and options['directory'] or ''), ) def _init_config(self, config_file, args): print_('Creating %r.' % config_file) f = open(config_file, 'w') sep = re.compile(r'[\\/]') if args: eggs = '\n '.join(a for a in args if not sep.search(a)) sepsub = os.path.sep == '/' and '/' or re.escape(os.path.sep) paths = '\n '.join( sep.sub(sepsub, a) for a in args if sep.search(a)) f.write('[buildout]\n' 'parts = py\n' '\n' '[py]\n' 'recipe = zc.recipe.egg\n' 'interpreter = py\n' 'eggs =\n' ) if eggs: f.write(' %s\n' % eggs) if paths: f.write('extra-paths =\n %s\n' % paths) for p in [a for a in args if sep.search(a)]: if not os.path.exists(p): os.mkdir(p) else: f.write('[buildout]\nparts =\n') f.close() @command def init(self, args): self.bootstrap(()) if args: self.install(()) def _update_sys_path(self): # Add develop-eggs directory to path so that it gets searched # for eggs: dev_eggs_dir = self['buildout']['develop-eggs-directory'] if os.path.exists(dev_eggs_dir): import site # site.addsitedir adds to the end of sys.path, and processes .pth files. # We want the resulting paths at the beginning of sys.path to maintain # buildout's priority (develop eggs first). # Ensure dev_eggs_dir is in sys.path so that .pth files in it can # import finders from the same directory. if dev_eggs_dir not in sys.path: sys.path.insert(0, dev_eggs_dir) old_path = list(sys.path) site.addsitedir(dev_eggs_dir) new_paths = [p for p in sys.path if p not in old_path] # Move newly added paths to the front for p in new_paths: if p in sys.path: sys.path.remove(p) for p in reversed(new_paths): sys.path.insert(0, p) @command def install(self, install_args): __doing__ = 'Installing.' self._update_sys_path() self._load_extensions() self._setup_directories() # Check for updates. This could cause the process to be restarted self._maybe_upgrade() # load installed data (installed_part_options, installed_exists )= self._read_installed_part_options() # Remove old develop eggs self._uninstall( installed_part_options['buildout'].get( 'installed_develop_eggs', '') ) # Build develop eggs installed_develop_eggs = self._develop() # Update sys.path again in case new develop eggs were created self._update_sys_path() installed_part_options['buildout']['installed_develop_eggs' ] = installed_develop_eggs if installed_exists: self._update_installed( installed_develop_eggs=installed_develop_eggs) # get configured and installed part lists conf_parts = self['buildout']['parts'] conf_parts = conf_parts and conf_parts.split() or [] installed_parts = installed_part_options['buildout']['parts'] installed_parts = installed_parts and installed_parts.split() or [] if install_args: install_parts = install_args uninstall_missing = False else: install_parts = conf_parts uninstall_missing = True # load and initialize recipes [self[part]['recipe'] for part in install_parts] if not install_args: install_parts = self._parts if self._log_level < logging.DEBUG: sections = list(self) sections.sort() print_() print_('Configuration data:') for section in sorted(self._data): _save_options(section, self[section], sys.stdout) print_() # compute new part recipe signatures self._compute_part_signatures(install_parts) # uninstall parts that are no-longer used or who's configs # have changed for part in reversed(installed_parts): if part in install_parts: old_options = installed_part_options[part].copy() installed_files = old_options.pop('__buildout_installed__') new_options = self.get(part) if old_options == new_options: # The options are the same, but are all of the # installed files still there? If not, we should # reinstall. if not installed_files: continue for f in installed_files.split('\n'): if not os.path.exists(self._buildout_path(f)): break else: continue # output debugging info if self._logger.getEffectiveLevel() < logging.DEBUG: for k in old_options: if k not in new_options: self._logger.debug("Part %s, dropped option %s.", part, k) elif old_options[k] != new_options[k]: self._logger.debug( "Part %s, option %s changed:\n%r != %r", part, k, new_options[k], old_options[k], ) for k in new_options: if k not in old_options: self._logger.debug("Part %s, new option %s.", part, k) elif not uninstall_missing: continue self._uninstall_part(part, installed_part_options) installed_parts = [p for p in installed_parts if p != part] if installed_exists: self._update_installed(parts=' '.join(installed_parts)) # Check for unused buildout options: _check_for_unused_options_in_section(self, 'buildout') # install new parts for part in install_parts: signature = self[part].pop('__buildout_signature__') saved_options = self[part].copy() recipe = self[part].recipe if part in installed_parts: # update need_to_save_installed = False __doing__ = 'Updating %s.', part self._logger.info(*__doing__) old_options = installed_part_options[part] old_installed_files = old_options['__buildout_installed__'] try: update = recipe.update except AttributeError: update = recipe.install self._logger.warning( "The recipe for %s doesn't define an update " "method. Using its install method.", part) try: installed_files = self[part]._call(update) except Exception: installed_parts.remove(part) self._uninstall(old_installed_files) if installed_exists: self._update_installed( parts=' '.join(installed_parts)) raise old_installed_files = old_installed_files.split('\n') if installed_files is None: installed_files = old_installed_files else: if isinstance(installed_files, str): installed_files = [installed_files] else: installed_files = list(installed_files) need_to_save_installed = [ p for p in installed_files if p not in old_installed_files] if need_to_save_installed: installed_files = (old_installed_files + need_to_save_installed) else: # install need_to_save_installed = True __doing__ = 'Installing %s.', part self._logger.info(*__doing__) installed_files = self[part]._call(recipe.install) if installed_files is None: self._logger.warning( "The %s install returned None. A path or " "iterable os paths should be returned.", part) installed_files = () elif isinstance(installed_files, str): installed_files = [installed_files] else: installed_files = list(installed_files) installed_part_options[part] = saved_options saved_options['__buildout_installed__' ] = '\n'.join(installed_files) saved_options['__buildout_signature__'] = signature installed_parts = [p for p in installed_parts if p != part] installed_parts.append(part) _check_for_unused_options_in_section(self, part) if need_to_save_installed: installed_part_options['buildout']['parts'] = ( ' '.join(installed_parts)) self._save_installed_options(installed_part_options) installed_exists = True else: assert installed_exists self._update_installed(parts=' '.join(installed_parts)) if installed_develop_eggs: if not installed_exists: self._save_installed_options(installed_part_options) elif (not installed_parts) and installed_exists: os.remove(self['buildout']['installed']) if self.show_picked_versions or self.update_versions_file: self._print_picked_versions() self._print_namespace_packages() self._unload_extensions() def _update_installed(self, **buildout_options): installed = self['buildout']['installed'] f = open(installed, 'a') f.write('\n[buildout]\n') for option, value in list(buildout_options.items()): _save_option(option, value, f) f.close() def _uninstall_part(self, part, installed_part_options): # uninstall part __doing__ = 'Uninstalling %s.', part self._logger.info(*__doing__) # run uninstall recipe recipe, entry = _recipe(installed_part_options[part]) try: uninstaller = _install_and_load( recipe, 'zc.buildout.uninstall', entry, self) self._logger.info('Running uninstall recipe.') uninstaller(part, installed_part_options[part]) except (ImportError, pkg_resources.DistributionNotFound): pass # remove created files and directories self._uninstall( installed_part_options[part]['__buildout_installed__']) def _setup_directories(self): __doing__ = 'Setting up buildout directories' # Create buildout directories for name in ('bin', 'parts', 'develop-eggs'): d = self['buildout'][name+'-directory'] if not os.path.exists(d): self._logger.info('Creating directory %r.', d) os.mkdir(d) def _develop(self): """Install sources by running in editable mode. Traditionally: run `setup.py develop` on them. Nowadays: run `pip install -e` on them, as there may not be a `setup.py`, but `pyproject.toml` instead, using for example `hatchling`. """ __doing__ = 'Processing directories listed in the develop option' develop = self['buildout'].get('develop') if not develop: return '' dest = self['buildout']['develop-eggs-directory'] old_files = os.listdir(dest) env = dict(os.environ, PYTHONPATH=zc.buildout.easy_install.setuptools_pythonpath) here = os.getcwd() try: try: for setup in develop.split(): setup = self._buildout_path(setup) files = glob.glob(setup) if not files: self._logger.warning("Couldn't develop %r (not found)", setup) else: files.sort() for setup in files: self._logger.info("Develop: %r", setup) __doing__ = 'Processing develop directory %r.', setup zc.buildout.easy_install.develop(setup, dest) except Exception: # if we had an error, we need to roll back changes, by # removing any files we created. self._sanity_check_develop_eggs_files(dest, old_files) self._uninstall('\n'.join( [os.path.join(dest, f) for f in os.listdir(dest) if f not in old_files ])) raise else: self._sanity_check_develop_eggs_files(dest, old_files) res = [os.path.join(dest, f) for f in os.listdir(dest) if f not in old_files ] res.sort() return '\n'.join(res) finally: os.chdir(here) def _sanity_check_develop_eggs_files(self, dest, old_files): for f in os.listdir(dest): if f in old_files or f == '__pycache__': continue if not (os.path.isfile(os.path.join(dest, f)) and (f.endswith('.egg-link') or f.endswith('.pth') or (f.startswith('__editable__') and f.endswith('.py'))) or (os.path.isdir(os.path.join(dest, f)) and (f.endswith('.dist-info') or f.endswith('.egg-info')))): self._logger.warning( "Unexpected entry, %r, in develop-eggs directory.", f) def _compute_part_signatures(self, parts): # Compute recipe signature and add to options for part in parts: options = self.get(part) if options is None: options = self[part] = {} recipe, entry = _recipe(options) req = pkg_resources.Requirement.parse(recipe) sig = _dists_sig(pkg_resources.working_set.resolve([req])) options['__buildout_signature__'] = ' '.join(sig) def _read_installed_part_options(self): old = self['buildout']['installed'] if old and os.path.isfile(old): fp = open(old) sections = zc.buildout.configparser.parse(fp, old) fp.close() result = {} for section, options in sections.items(): for option, value in options.items(): if '%(' in value: for k, v in _spacey_defaults: value = value.replace(k, v) options[option] = value result[section] = self.Options(self, section, options) return result, True else: return ({'buildout': self.Options(self, 'buildout', {'parts': ''})}, False, ) def _uninstall(self, installed): for f in installed.split('\n'): if not f: continue f = self._buildout_path(f) if os.path.isdir(f): rmtree(f) elif os.path.isfile(f): try: os.remove(f) except OSError: if not ( sys.platform == 'win32' and (realpath(os.path.join(os.path.dirname(sys.argv[0]), 'buildout.exe')) == realpath(f) ) # Sigh. This is the executable used to run the buildout # and, of course, it's in use. Leave it. ): raise def _install(self, part): options = self[part] recipe, entry = _recipe(options) recipe_class = pkg_resources.load_entry_point( recipe, 'zc.buildout', entry) installed = recipe_class(self, part, options).install() if installed is None: installed = [] elif isinstance(installed, str): installed = [installed] base = self._buildout_path('') installed = [d.startswith(base) and d[len(base):] or d for d in installed] return ' '.join(installed) def _save_installed_options(self, installed_options): installed = self['buildout']['installed'] if not installed: return f = open(installed, 'w') _save_options('buildout', installed_options['buildout'], f) for part in installed_options['buildout']['parts'].split(): print_(file=f) _save_options(part, installed_options[part], f) f.close() def _error(self, message, *args): raise zc.buildout.UserError(message % args) def _setup_socket_timeout(self): timeout = self['buildout']['socket-timeout'] if timeout != '': try: timeout = int(timeout) import socket self._logger.info( 'Setting socket time out to %d seconds.', timeout) socket.setdefaulttimeout(timeout) except ValueError: self._logger.warning("Default socket timeout is used !\n" "Value in configuration is not numeric: [%s].\n", timeout) def _setup_logging(self): root_logger = logging.getLogger() self._logger = logging.getLogger('zc.buildout') handler = logging.StreamHandler(sys.stdout) log_format = self['buildout']['log-format'] if not log_format: # No format specified. Use different formatter for buildout # and other modules, showing logger name except for buildout log_format = '%(name)s: %(message)s' buildout_handler = logging.StreamHandler(sys.stdout) buildout_handler.setFormatter(logging.Formatter('%(message)s')) self._logger.propagate = False self._logger.addHandler(buildout_handler) handler.setFormatter(logging.Formatter(log_format)) root_logger.addHandler(handler) level = self['buildout']['log-level'] if level in ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'): level = getattr(logging, level) else: try: level = int(level) except ValueError: self._error("Invalid logging level %s", level) verbosity = self['buildout'].get('verbosity', 0) try: verbosity = int(verbosity) except ValueError: self._error("Invalid verbosity %s", verbosity) level -= verbosity root_logger.setLevel(level) self._log_level = level def _maybe_upgrade(self): # See if buildout or setuptools or other dependencies need to be upgraded. # If they do, do the upgrade and restart the buildout process. __doing__ = 'Checking for upgrades.' if 'BUILDOUT_RESTART_AFTER_UPGRADE' in os.environ: return if not self.newest: return # We must install `wheel` before `setuptools`` to avoid confusion between # the true `wheel` package and the one vendorized by `setuptools`. # See https://github.com/buildout/buildout/issues/691 projects = ('zc.buildout', 'wheel', 'pip', 'setuptools') ws = zc.buildout.easy_install.install( projects, self['buildout']['eggs-directory'], links = self['buildout'].get('find-links', '').split(), index = self['buildout'].get('index'), path = [self['buildout']['develop-eggs-directory']], allow_hosts = self._allow_hosts ) upgraded = [] for project in projects: canonicalized_name = packaging_utils.canonicalize_name(project) req = pkg_resources.Requirement.parse(canonicalized_name) dist = ws.find(req) if dist is None and canonicalized_name != project: # Try with the original project name. Depending on which setuptools # version is used, this is either useless or a life saver. req = pkg_resources.Requirement.parse(project) dist = ws.find(req) importlib.import_module(project) if dist is None: # This is unexpected. This must be some problem with how we use # setuptools/pkg_resources. But since the import worked, it feels # safe to ignore. self._logger.warning( "Could not find %s in working set during upgrade check. Ignoring.", project, ) continue if not inspect.getfile(sys.modules[project]).startswith(dist.location): upgraded.append(dist) if not upgraded: return __doing__ = 'Upgrading.' should_run = realpath( os.path.join(os.path.abspath(self['buildout']['bin-directory']), 'buildout') ) if sys.platform == 'win32': should_run += '-script.py' if (realpath(os.path.abspath(sys.argv[0])) != should_run): self._logger.debug("Running %r.", realpath(sys.argv[0])) self._logger.debug("Local buildout is %r.", should_run) self._logger.warning("Not upgrading because not running a local " "buildout command.") return self._logger.info("Upgraded:\n %s;\nRestarting.", ",\n ".join([("%s version %s" % (dist.project_name, dist.version) ) for dist in upgraded ] ), ) # the new dist is different, so we've upgraded. # Update the scripts and return True options = self['buildout'] eggs_dir = options['eggs-directory'] develop_eggs_dir = options['develop-eggs-directory'] ws = zc.buildout.easy_install.sort_working_set( ws, eggs_dir=eggs_dir, develop_eggs_dir=develop_eggs_dir ) zc.buildout.easy_install.scripts( ['zc.buildout'], ws, sys.executable, options['bin-directory'], relative_paths = ( bool_option(options, 'relative-paths', False) and options['directory'] or ''), ) # Restart args = sys.argv[:] if not __debug__: args.insert(0, '-O') args.insert(0, sys.executable) env=dict(os.environ, BUILDOUT_RESTART_AFTER_UPGRADE='1') sys.exit(subprocess.call(args, env=env)) def _load_extensions(self): __doing__ = 'Loading extensions.' specs = self['buildout'].get('extensions', '').split() for superceded_extension in ['buildout-versions', 'buildout.dumppickedversions']: if superceded_extension in specs: msg = ("Buildout now includes 'buildout-versions' (and part " "of the older 'buildout.dumppickedversions').\n" "Remove the extension from your configuration and " "look at the 'show-picked-versions' option in " "buildout's documentation.") raise zc.buildout.UserError(msg) if specs: path = [self['buildout']['develop-eggs-directory']] if self.offline: dest = None path.append(self['buildout']['eggs-directory']) else: dest = self['buildout']['eggs-directory'] zc.buildout.easy_install.install( specs, dest, path=path, working_set=pkg_resources.working_set, links = self['buildout'].get('find-links', '').split(), index = self['buildout'].get('index'), newest=self.newest, allow_hosts=self._allow_hosts) # Clear cache because extensions might now let us read pages we # couldn't read before. zc.buildout.easy_install.clear_index_cache() for ep in pkg_resources.iter_entry_points('zc.buildout.extension'): ep.load()(self) def _unload_extensions(self): __doing__ = 'Unloading extensions.' specs = self['buildout'].get('extensions', '').split() if specs: for ep in pkg_resources.iter_entry_points( 'zc.buildout.unloadextension'): ep.load()(self) def _print_picked_versions(self): picked_versions, required_by = (zc.buildout.easy_install .get_picked_versions()) if not picked_versions: # Don't print empty output. return output = _format_picked_versions(picked_versions, required_by) if self.show_picked_versions: print_("Versions had to be automatically picked.") print_("The following part definition lists the versions picked:") print_('\n'.join(output)) if self.update_versions_file: # Write to the versions file. if os.path.exists(self.update_versions_file): output[:1] = [ '', '# Added by buildout at %s' % datetime.datetime.now() ] output.append('') f = open(self.update_versions_file, 'a') f.write(('\n'.join(output))) f.close() print_("Picked versions have been written to " + self.update_versions_file) def _print_namespace_packages(self): namespace_packages = zc.buildout.easy_install.get_namespace_packages() if not namespace_packages: # Don't print empty output. return print(""" ** WARNING ** Some development packages are using old style namespace packages. You should switch to native namespaces (PEP 420). If you get a ModuleNotFound or an ImportError when importing a package from one of these namespaces, you can try this as a temporary workaround: pip install horse-with-no-namespace Note: installing horse-with-no-namespace with buildout will not work. You must install it into the virtualenv with pip (or uv) before running the buildout. The following list shows the affected packages and their namespaces: """ ) for key, value in namespace_packages: print(f"* {key}: {', '.join(value.splitlines())}") @command def setup(self, args): if not args: raise zc.buildout.UserError( "The setup command requires the path to a setup script or \n" "directory containing a setup script, and its arguments." ) setup = args.pop(0) if os.path.isdir(setup): setup = os.path.join(setup, 'setup.py') self._logger.info("Running setup script %r.", setup) setup = os.path.abspath(setup) fd, tsetup = tempfile.mkstemp() try: os.write(fd, (zc.buildout.easy_install.runsetup_template % dict( setupdir=os.path.dirname(setup), setup=setup, __file__ = setup, extra="", )).encode()) args = [sys.executable, tsetup] + args zc.buildout.easy_install.call_subprocess(args) finally: os.close(fd) os.remove(tsetup) @command def runsetup(self, args): self.setup(args) @command def query(self, args=None): if args is None or len(args) != 1: _error('The query command requires a single argument.') option = args[0] option = option.split(':') if len(option) == 1: option = 'buildout', option[0] elif len(option) != 2: _error('Invalid option:', args[0]) section, option = option verbose = self['buildout'].get('verbosity', 0) != 0 if verbose: print_('${%s:%s}' % (section, option)) try: print_(self._raw[section][option]) except KeyError: if section in self._raw: _error('Key not found:', option) else: _error('Section not found:', section) @command def annotate(self, args=None): verbose = self['buildout'].get('verbosity', 0) != 0 section = None if args is None: sections = [] else: sections = args _print_annotate(self._annotated, verbose, sections, self._buildout_dir) def print_options(self, base_path=None): for section in sorted(self._data): if section == 'buildout' or section == self['buildout']['versions']: continue print_('['+section+']') for k, v in sorted(self._data[section].items()): if '\n' in v: v = '\n ' + v.replace('\n', '\n ') else: v = ' '+v if base_path: v = v.replace(os.getcwd(), base_path) print_("%s =%s" % (k, v)) def __getitem__(self, section): __doing__ = 'Getting section %s.', section try: return self._data[section] except KeyError: pass try: data = self._raw[section] except KeyError: raise MissingSection(section) options = self.Options(self, section, data) self._data[section] = options options._initialize() return options def __setitem__(self, name, data): if name in self._raw: raise KeyError("Section already exists", name) self._raw[name] = dict((k, str(v)) for (k, v) in data.items()) self[name] # Add to parts def parse(self, data): from io import StringIO import textwrap sections = zc.buildout.configparser.parse( StringIO(textwrap.dedent(data)), '', _default_globals) for name in sections: if name in self._raw: raise KeyError("Section already exists", name) self._raw[name] = dict((k, str(v)) for (k, v) in sections[name].items()) for name in sections: self[name] # Add to parts def __delitem__(self, key): raise NotImplementedError('__delitem__') def keys(self): return list(self._raw.keys()) def __iter__(self): return iter(self._raw) def __len__(self): return len(self._raw) def _install_and_load(spec, group, entry, buildout): __doing__ = 'Loading recipe %r.', spec try: req = pkg_resources.Requirement.parse(spec) buildout_options = buildout['buildout'] if pkg_resources.working_set.find(req) is None: __doing__ = 'Installing recipe %s.', spec if buildout.offline: dest = None path = [buildout_options['develop-eggs-directory'], buildout_options['eggs-directory'], ] else: dest = buildout_options['eggs-directory'] path = [buildout_options['develop-eggs-directory']] # Pin versions when processing the buildout section versions_section_name = buildout['buildout'].get('versions', 'versions') versions = buildout.get(versions_section_name, {}) zc.buildout.easy_install.allow_picked_versions( bool_option(buildout['buildout'], 'allow-picked-versions') ) zc.buildout.easy_install.install( [spec], dest, links=buildout._links, index=buildout_options.get('index'), path=path, working_set=pkg_resources.working_set, newest=buildout.newest, allow_hosts=buildout._allow_hosts, versions=versions, ) __doing__ = 'Loading %s recipe entry %s:%s.', group, spec, entry return pkg_resources.load_entry_point( req.project_name, group, entry) except Exception: v = sys.exc_info()[1] buildout._logger.log( 1, "Couldn't load %s entry point %s\nfrom %s:\n%s.", group, entry, spec, v) raise class Options(DictMixin): def __init__(self, buildout, section, data): self.buildout = buildout self.name = section self._raw = data self._cooked = {} self._data = {} def _initialize(self): name = self.name __doing__ = 'Initializing section %s.', name if '<' in self._raw: self._raw = self._do_extend_raw(name, self._raw, []) # force substitutions for k, v in sorted(self._raw.items()): if '${' in v: self._dosub(k, v) if name == 'buildout': return # buildout section can never be a part for dname in self.get('', '').split(): # force use of dependencies in buildout: self.buildout[dname] if self.get('recipe'): self.initialize() self.buildout._parts.append(name) def initialize(self): reqs, entry = _recipe(self._data) buildout = self.buildout recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout) name = self.name self.recipe = recipe_class(buildout, name, self) def _do_extend_raw(self, name, data, doing): if name == 'buildout': return data if name in doing: raise zc.buildout.UserError("Infinite extending loop %r" % name) doing.append(name) try: to_do = data.get('<', None) if to_do is None: return data __doing__ = 'Loading input sections for %r', name result = {} for iname in to_do.split('\n'): iname = iname.strip() if not iname: continue raw = self.buildout._raw.get(iname) if raw is None: raise zc.buildout.UserError("No section named %r" % iname) result.update(self._do_extend_raw(iname, raw, doing)) result = _annotate_section(result, "") data = _annotate_section(copy.deepcopy(data), "") result = _update_section(result, data) result = _unannotate_section(result) result.pop('<', None) return result finally: assert doing.pop() == name def _dosub(self, option, v): __doing__ = 'Getting option %s:%s.', self.name, option seen = [(self.name, option)] v = '$$'.join([self._sub(s, seen) for s in v.split('$$')]) self._cooked[option] = v def get(self, option, default=None, seen=None): try: return self._data[option] except KeyError: pass v = self._cooked.get(option) if v is None: v = self._raw.get(option) if v is None: return default __doing__ = 'Getting option %s:%s.', self.name, option if '${' in v: key = self.name, option if seen is None: seen = [key] elif key in seen: raise zc.buildout.UserError( "Circular reference in substitutions.\n" ) else: seen.append(key) v = '$$'.join([self._sub(s, seen) for s in v.split('$$')]) seen.pop() self._data[option] = v return v _template_split = re.compile('([$]{[^}]*})').split _simple = re.compile('[-a-zA-Z0-9 ._]+$').match _valid = re.compile(r'\${[-a-zA-Z0-9 ._]*:[-a-zA-Z0-9 ._]+}$').match def _sub(self, template, seen): value = self._template_split(template) subs = [] for ref in value[1::2]: s = tuple(ref[2:-1].split(':')) if not self._valid(ref): if len(s) < 2: raise zc.buildout.UserError("The substitution, %s,\n" "doesn't contain a colon." % ref) if len(s) > 2: raise zc.buildout.UserError("The substitution, %s,\n" "has too many colons." % ref) if not self._simple(s[0]): raise zc.buildout.UserError( "The section name in substitution, %s,\n" "has invalid characters." % ref) if not self._simple(s[1]): raise zc.buildout.UserError( "The option name in substitution, %s,\n" "has invalid characters." % ref) section, option = s if not section: section = self.name v = self.buildout[section].get(option, None, seen) if v is None: if option == '_buildout_section_name_': v = self.name else: raise MissingOption("Referenced option does not exist:", section, option) subs.append(v) subs.append('') return ''.join([''.join(v) for v in zip(value[::2], subs)]) def __getitem__(self, key): try: return self._data[key] except KeyError: pass v = self.get(key) if v is None: raise MissingOption("Missing option: %s:%s" % (self.name, key)) return v def __setitem__(self, option, value): if not isinstance(value, str): raise TypeError('Option values must be strings', value) self._data[option] = value def __delitem__(self, key): if key in self._raw: del self._raw[key] if key in self._data: del self._data[key] if key in self._cooked: del self._cooked[key] elif key in self._data: del self._data[key] else: raise KeyError(key) def keys(self): raw = self._raw return list(self._raw) + [k for k in self._data if k not in raw] def __iter__(self): return iter(self.keys()) def __len__(self): return len(self.keys()) def copy(self): result = copy.deepcopy(self._raw) result.update(self._cooked) result.update(self._data) return result def _call(self, f): buildout_directory = self.buildout['buildout']['directory'] self._created = [] try: try: os.chdir(buildout_directory) return f() except Exception: for p in self._created: if os.path.isdir(p): rmtree(p) elif os.path.isfile(p): os.remove(p) else: self.buildout._logger.warning("Couldn't clean up %r.", p) raise finally: self._created = None os.chdir(buildout_directory) def created(self, *paths): try: self._created.extend(paths) except AttributeError: raise TypeError( "Attempt to register a created path while not installing", self.name) return self._created def __repr__(self): return repr(dict(self)) Buildout.Options = Options _spacey_nl = re.compile('[ \t\r\f\v]*\n[ \t\r\f\v\n]*' '|' '^[ \t\r\f\v]+' '|' '[ \t\r\f\v]+$' ) _spacey_defaults = [ ('%(__buildout_space__)s', ' '), ('%(__buildout_space_n__)s', '\n'), ('%(__buildout_space_r__)s', '\r'), ('%(__buildout_space_f__)s', '\f'), ('%(__buildout_space_v__)s', '\v'), ] def _quote_spacey_nl(match): match = match.group(0).split('\n', 1) result = '\n\t'.join( [(s .replace(' ', '%(__buildout_space__)s') .replace('\r', '%(__buildout_space_r__)s') .replace('\f', '%(__buildout_space_f__)s') .replace('\v', '%(__buildout_space_v__)s') .replace('\n', '%(__buildout_space_n__)s') ) for s in match] ) return result def _save_option(option, value, f): value = _spacey_nl.sub(_quote_spacey_nl, value) if value.startswith('\n\t'): value = '%(__buildout_space_n__)s' + value[2:] if value.endswith('\n\t'): value = value[:-2] + '%(__buildout_space_n__)s' print_(option, '=', value, file=f) def _save_options(section, options, f): print_('[%s]' % section, file=f) items = list(options.items()) items.sort() for option, value in items: _save_option(option, value, f) def _default_globals(): """Return a mapping of default and precomputed expressions. These default expressions are convenience defaults available when eveluating section headers expressions. NB: this is wrapped in a function so that the computing of these expressions is lazy and done only if needed (ie if there is at least one section with an expression) because the computing of some of these expressions can be expensive. """ # partially derived or inspired from its.py # Copyright (c) 2012, Kenneth Reitz All rights reserved. # Redistribution and use in source and binary forms, with or without modification, # are permitted provided that the following conditions are met: # Redistributions of source code must retain the above copyright notice, this list # of conditions and the following disclaimer. Redistributions in binary form must # reproduce the above copyright notice, this list of conditions and the following # disclaimer in the documentation and/or other materials provided with the # distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND # CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A # PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, # OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING # IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY # OF SUCH DAMAGE. # default available modules, explicitly re-imported locally here on purpose import sys import os import platform import re globals_defs = {'sys': sys, 'os': os, 'platform': platform, 're': re,} # major python major_python_versions as python2 and python3 major_python_versions = tuple(map(str, platform.python_version_tuple())) globals_defs.update({'python2': major_python_versions[0] == '2', 'python3': major_python_versions[0] == '3'}) # minor python major_python_versions as python24, python25 ... python39 minor_python_versions = ( '24', '25', '26', '27', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '310', '311', '312', '313', '314', '315', '316', '317', '318', '319', ) for v in minor_python_versions: globals_defs['python' + v] = ''.join(major_python_versions[:2]) == v # interpreter type sys_version = sys.version.lower() pypy = 'pypy' in sys_version jython = 'java' in sys_version ironpython ='iron' in sys_version # assume CPython, if nothing else. cpython = not any((pypy, jython, ironpython,)) globals_defs.update({'cpython': cpython, 'pypy': pypy, 'jython': jython, 'ironpython': ironpython}) # operating system sys_platform = str(sys.platform).lower() globals_defs.update({'linux': 'linux' in sys_platform, 'windows': 'win32' in sys_platform, 'cygwin': 'cygwin' in sys_platform, 'solaris': 'sunos' in sys_platform, 'macosx': 'darwin' in sys_platform, 'posix': 'posix' in os.name.lower()}) #bits and endianness import struct void_ptr_size = struct.calcsize('P') * 8 globals_defs.update({'bits32': void_ptr_size == 32, 'bits64': void_ptr_size == 64, 'little_endian': sys.byteorder == 'little', 'big_endian': sys.byteorder == 'big'}) return globals_defs variable_template_split = re.compile('([$]{[^}]*})').split def _open( base, filename, seen, download_options, override, downloaded, user_defaults ): """Open a configuration file and return the result as a dictionary, Recursively open other files based on buildout options found. """ download_options = _update_section(download_options, override) raw_download_options = _unannotate_section(download_options) newest = bool_option(raw_download_options, 'newest', 'false') fallback = newest and not (filename in downloaded) extends_cache = raw_download_options.get('extends-cache') if extends_cache and variable_template_split(extends_cache)[1::2]: raise ValueError( "extends-cache '%s' may not contain ${section:variable} to expand." % extends_cache ) download = zc.buildout.download.Download( raw_download_options, cache=extends_cache, fallback=fallback, hash_name=True) is_temp = False downloaded_filename = None if _isurl(filename): downloaded_filename, is_temp = download(filename) fp = open(downloaded_filename) base = filename[:filename.rfind('/')] elif _isurl(base): if os.path.isabs(filename): fp = open(filename) base = os.path.dirname(filename) else: filename = base + '/' + filename downloaded_filename, is_temp = download(filename) fp = open(downloaded_filename) base = filename[:filename.rfind('/')] else: filename = os.path.join(base, filename) fp = open(filename) base = os.path.dirname(filename) downloaded.add(filename) if filename in seen: if is_temp: fp.close() os.remove(downloaded_filename) raise zc.buildout.UserError("Recursive file include", seen, filename) root_config_file = not seen seen.append(filename) filename_for_logging = filename if downloaded_filename: filename_for_logging = '%s (downloaded as %s)' % ( filename, downloaded_filename) result = zc.buildout.configparser.parse( fp, filename_for_logging, _default_globals) fp.close() if is_temp: os.remove(downloaded_filename) options = result.get('buildout', {}) extends = options.pop('extends', None) if 'extended-by' in options: raise zc.buildout.UserError( 'No-longer supported "extended-by" option found in %s.' % filename) result = _annotate(result, filename) if root_config_file and 'buildout' in result: download_options = _update_section( download_options, result['buildout'] ) # Process extends to handle nested += and -= eresults = [] if extends: extends = extends.split() for fname in extends: next_extend, user_defaults = _open( base, fname, seen, download_options, override, downloaded, user_defaults) eresults.extend(next_extend) else: if user_defaults: result = _update(user_defaults, result) user_defaults = {} optional_extends = options.pop('optional-extends', None) if optional_extends: optional_extends = optional_extends.value.split() for fname in optional_extends: if not os.path.exists(fname): print("optional-extends file not found: %s" % fname) continue next_extend, user_defaults = _open( base, fname, seen, download_options, override, downloaded, user_defaults) eresults.extend(next_extend) eresults.append(result) seen.pop() if root_config_file: final_result = {} for eresult in eresults: final_result = _update(final_result, eresult) return final_result, user_defaults else: return eresults, user_defaults ignore_directories = '.svn', 'CVS', '__pycache__', '.git' _dir_hashes = {} def _dir_hash(dir): dir_hash = _dir_hashes.get(dir, None) if dir_hash is not None: return dir_hash hash = md5() for (dirpath, dirnames, filenames) in os.walk(dir): dirnames[:] = sorted(n for n in dirnames if n not in ignore_directories) filenames[:] = sorted(f for f in filenames if (not (f.endswith('pyc') or f.endswith('pyo')) and os.path.exists(os.path.join(dirpath, f))) ) for_hash = ' '.join(dirnames + filenames) if isinstance(for_hash, str): for_hash = for_hash.encode() hash.update(for_hash) for name in filenames: path = os.path.join(dirpath, name) if name == 'entry_points.txt': f = open(path) # Entry points aren't written in stable order. :( try: sections = zc.buildout.configparser.parse(f, path) data = repr([(sname, sorted(sections[sname].items())) for sname in sorted(sections)]).encode('utf-8') except Exception: f.close() f = open(path, 'rb') data = f.read() else: f = open(path, 'rb') data = f.read() f.close() hash.update(data) _dir_hashes[dir] = dir_hash = hash.hexdigest() return dir_hash def _dists_sig(dists): seen = set() result = [] for dist in sorted(dists): if dist in seen: continue seen.add(dist) location = dist.location if dist.precedence == pkg_resources.DEVELOP_DIST: result.append(dist.project_name + '-' + _dir_hash(location)) else: result.append(os.path.basename(location)) return result def _update_section(in1, s2): s1 = copy.deepcopy(in1) # Base section 2 on section 1; section 1 is copied, with key-value pairs # in section 2 overriding those in section 1. If there are += or -= # operators in section 2, process these to add or subtract items (delimited # by newlines) from the preexisting values. s2 = copy.deepcopy(s2) # avoid mutating the second argument, which is unexpected # Sort on key, then on the addition or subtraction operator (+ comes first) for k, v in sorted(s2.items(), key=lambda x: (x[0].rstrip(' +'), x[0][-1])): if k.endswith('+'): key = k.rstrip(' +') implicit_value = SectionKey("", "IMPLICIT_VALUE") # Find v1 in s2 first; it may have been defined locally too. section_key = s2.get(key, s1.get(key, implicit_value)) section_key = copy.deepcopy(section_key) section_key.addToValue(v.value, v.source) s2[key] = section_key del s2[k] elif k.endswith('-'): key = k.rstrip(' -') implicit_value = SectionKey("", "IMPLICIT_VALUE") # Find v1 in s2 first; it may have been set by a += operation first section_key = s2.get(key, s1.get(key, implicit_value)) section_key = copy.deepcopy(section_key) section_key.removeFromValue(v.value, v.source) s2[key] = section_key del s2[k] _update_verbose(s1, s2) return s1 def _update_verbose(s1, s2): for key, v2 in s2.items(): if key in s1: v1 = s1[key] v1.overrideValue(v2) else: s1[key] = copy.deepcopy(v2) def _update(in1, d2): d1 = copy.deepcopy(in1) for section in d2: if section in d1: d1[section] = _update_section(d1[section], d2[section]) elif '<' not in d2[section].keys(): # Skip sections that extend in other sections (macros), as we don't # have all the data (these will be processed when the section is # extended) temp = copy.deepcopy(d2[section]) # 641 - Process base definitions done with += and -= for k, v in sorted(temp.items(), key=lambda item: item[0]): # Process + before -, configparser resolves conflicts if k[-1] == '+' and k[:-2] not in temp: # Turn += without a preceding = into an assignment temp[k[:-2]] = temp[k] del temp[k] elif k[-1] == '-' and k[:-2] not in temp: # Turn -= without a preceding = into an empty assignment temp[k[:-2]] = temp[k] temp[k[:-2]].removeFromValue( temp[k[:-2]].value, "IMPLICIT_VALUE" ) del temp[k] # 656 - Handle multiple option assignments/extensions/removals # in the same file, which can happen with conditional sections d1[section] = _update_section({}, temp) else: d1[section] = copy.deepcopy(d2[section]) return d1 def _recipe(options): recipe = options['recipe'] if ':' in recipe: recipe, entry = recipe.split(':') else: entry = 'default' return recipe, entry def _doing(): _, v, tb = sys.exc_info() message = str(v) doing = [] while tb is not None: d = tb.tb_frame.f_locals.get('__doing__') if d: doing.append(d) tb = tb.tb_next if doing: sys.stderr.write('While:\n') for d in doing: if not isinstance(d, str): d = d[0] % d[1:] sys.stderr.write(' %s\n' % d) def _error(*message): sys.stderr.write('Error: ' + ' '.join(message) +'\n') sys.exit(1) _internal_error_template = """ An internal error occurred due to a bug in either zc.buildout or in a recipe being used: """ def _check_for_unused_options_in_section(buildout, section): options = buildout[section] unused = [option for option in sorted(options._raw) if option not in options._data] if unused: buildout._logger.warning( "Section `%s` contains unused option(s): %s.\n" "This may be an indication for either a typo in the option's name " "or a bug in the used recipe." % (section, ' '.join(map(repr, unused))) ) _usage = """\ Usage: buildout [options] [assignments] [command [command arguments]] Options: -c config_file Specify the path to the buildout configuration file to be used. This defaults to the file named "buildout.cfg" in the current working directory. -D Debug errors. If an error occurs, then the post-mortem debugger will be started. This is especially useful for debugging recipe problems. -h, --help Print this message and exit. -N Run in non-newest mode. This is equivalent to the assignment buildout:newest=false. With this setting, buildout will not seek new distributions if installed distributions satisfy it's requirements. -q Decrease the level of verbosity. This option can be used multiple times. -t socket_timeout Specify the socket timeout in seconds. -U Don't read user defaults. -v Increase the level of verbosity. This option can be used multiple times. --version Print buildout version number and exit. Assignments are of the form: section:option=value and are used to provide configuration options that override those given in the configuration file. For example, to run the buildout in offline mode, use buildout:offline=true. Options and assignments can be interspersed. Commands: install Install the parts specified in the buildout configuration. This is the default command if no command is specified. bootstrap Create a new buildout in the current working directory, copying the buildout and setuptools eggs and, creating a basic directory structure and a buildout-local buildout script. init [requirements] Initialize a buildout, creating a minimal buildout.cfg file if it doesn't exist and then performing the same actions as for the bootstrap command. If requirements are supplied, then the generated configuration will include an interpreter script that requires them. This provides an easy way to quickly set up a buildout to experiment with some packages. setup script [setup command and options] Run a given setup script arranging that setuptools is in the script's path and and that it has been imported so that setuptools-provided commands (like bdist_egg) can be used even if the setup script doesn't import setuptools. The script can be given either as a script path or a path to a directory containing a setup.py script. annotate Display annotated sections. All sections are displayed, sorted alphabetically. For each section, all key-value pairs are displayed, sorted alphabetically, along with the origin of the value (file name or COMPUTED_VALUE, DEFAULT_VALUE, COMMAND_LINE_VALUE). query section:key Display value of given section key pair. """ def _help(): print_(_usage) sys.exit(0) def _version(): version = pkg_resources.working_set.find( pkg_resources.Requirement.parse('zc.buildout')).version print_("buildout version %s" % version) sys.exit(0) def main(args=None): if args is None: args = sys.argv[1:] config_file = 'buildout.cfg' verbosity = 0 options = [] use_user_defaults = True debug = False while args: if args[0][0] == '-': op = orig_op = args.pop(0) op = op[1:] while op and op[0] in 'vqhWUoOnNDA': if op[0] == 'v': verbosity += 10 elif op[0] == 'q': verbosity -= 10 elif op[0] == 'U': use_user_defaults = False elif op[0] == 'o': options.append(('buildout', 'offline', 'true')) elif op[0] == 'O': options.append(('buildout', 'offline', 'false')) elif op[0] == 'n': options.append(('buildout', 'newest', 'true')) elif op[0] == 'N': options.append(('buildout', 'newest', 'false')) elif op[0] == 'D': debug = True else: _help() op = op[1:] if op[:1] in ('c', 't'): op_ = op[:1] op = op[1:] if op_ == 'c': if op: config_file = op else: if args: config_file = args.pop(0) else: _error("No file name specified for option", orig_op) elif op_ == 't': try: timeout_string = args.pop(0) timeout = int(timeout_string) options.append( ('buildout', 'socket-timeout', timeout_string)) except IndexError: _error("No timeout value specified for option", orig_op) except ValueError: _error("Timeout value must be numeric", orig_op) elif op: if orig_op == '--help': _help() elif orig_op == '--version': _version() else: _error("Invalid option", '-'+op[0]) elif '=' in args[0]: option, value = args.pop(0).split('=', 1) option = option.split(':') if len(option) == 1: option = 'buildout', option[0] elif len(option) != 2: _error('Invalid option:', option) section, option = option options.append((section.strip(), option.strip(), value.strip())) else: # We've run out of command-line options and option assignments # The rest should be commands, so we'll stop here break if verbosity: options.append(('buildout', 'verbosity', str(verbosity))) if args: command = args.pop(0) if command not in Buildout.COMMANDS: _error('invalid command:', command) else: command = 'install' try: try: buildout = Buildout(config_file, options, use_user_defaults, command, args) getattr(buildout, command)(args) except SystemExit: logging.shutdown() # Make sure we properly propagate an exit code from a restarted # buildout process. raise except Exception: v = sys.exc_info()[1] _doing() exc_info = sys.exc_info() import pdb, traceback if debug: traceback.print_exception(*exc_info) sys.stderr.write('\nStarting pdb:\n') pdb.post_mortem(exc_info[2]) else: if isinstance(v, (zc.buildout.UserError, distutils.errors.DistutilsError ) ): _error(str(v)) else: sys.stderr.write(_internal_error_template) traceback.print_exception(*exc_info) sys.exit(1) finally: logging.shutdown() _bool_names = {'true': True, 'false': False, True: True, False: False} def bool_option(options, name, default=None): value = options.get(name, default) if value is None: raise KeyError(name) try: return _bool_names[value] except KeyError: raise zc.buildout.UserError( 'Invalid value for %r option: %r' % (name, value)) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4003627 zc_buildout-6.0.1/src/zc/buildout/configparser.py0000644000076500000240000002600215264141650021542 0ustar00mauritsstaff############################################################################## # # Copyright Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## # The following copied from Python 2 config parser because: # - The py3 configparser isn't backward compatible # - Both strip option values in undesirable ways # - dict of dicts is a much simpler api import re import textwrap import logging from packaging import markers Marker = markers.Marker InvalidMarker = markers.InvalidMarker logger = logging.getLogger('zc.buildout') class Error(Exception): """Base class for ConfigParser exceptions.""" def _get_message(self): """Getter for 'message'; needed only to override deprecation in BaseException.""" return self.__message def _set_message(self, value): """Setter for 'message'; needed only to override deprecation in BaseException.""" self.__message = value # BaseException.message has been deprecated since Python 2.6. To prevent # DeprecationWarning from popping up over this pre-existing attribute, use # a new property that takes lookup precedence. message = property(_get_message, _set_message) def __init__(self, msg=''): self.message = msg Exception.__init__(self, msg) def __repr__(self): return self.message __str__ = __repr__ class ParsingError(Error): """Raised when a configuration file does not follow legal syntax.""" def __init__(self, filename): Error.__init__(self, 'File contains parsing errors: %s' % filename) self.filename = filename self.errors = [] def append(self, lineno, line): self.errors.append((lineno, line)) self.message += '\n\t[line %2d]: %s' % (lineno, line) class MissingSectionHeaderError(ParsingError): """Raised when a key-value pair is found before any section header.""" def __init__(self, filename, lineno, line): Error.__init__( self, 'File contains no section headers.\nfile: %s, line: %d\n%r' % (filename, lineno, line)) self.filename = filename self.lineno = lineno self.line = line # This regex captures either sections headers with optional trailing comment # separated by a semicolon or a hash. Section headers can have an optional # expression. Expressions and comments can contain brackets but no verbatim '#' # and ';' : these need to be escaped. # A title line with an expression has the general form: # [section_name: some Python expression] #; some comment # This regex leverages the fact that the following is a valid Python expression: # [some Python expression] # some comment # and that section headers are also delimited by [brackets] that are also [list] # delimiters. # So instead of doing complex parsing to balance brackets in an expression, we # capture just enough from a header line to collect then remove the section_name # and colon expression separator keeping only a list-enclosed expression and # optional comments. The parsing and validation of this Python expression can be # entirely delegated to Python's eval. The result of the evaluated expression is # the always returned wrapped in a list with a single item that contains the # original expression section_header = re.compile( r'(?P\[)' r'\s*' r'(?P[^\s#[\]:;{}]+)' r'\s*' r'(:(?P[^#;]*))?' r'\s*' r'(?P]' r'\s*' r'([#;].*)?$)' ).match option_start = re.compile( r'(?P[^\s{}[\]=:]+\s*[-+]?)' r'=' r'(?P.*)$').match leading_blank_lines = re.compile(r"^(\s*\n)+") def parse(fp, fpname, exp_globals=dict): """Parse a sectioned setup file. The sections in setup files contain a title line at the top, indicated by a name in square brackets (`[]'), plus key/value options lines, indicated by `name: value' format lines. Continuations are represented by an embedded newline then leading whitespace. Blank lines, lines beginning with a '#', and just about everything else are ignored. The title line is in the form [name] followed by an optional trailing comment separated by a semicolon `;' or a hash `#' character. Optionally the title line can have the form `[name:expression]' where expression is an arbitrary Python expression. Sections with an expression that evaluates to False are ignored. Semicolon `;' an hash `#' characters must be string-escaped in expression literals. exp_globals is a callable returning a mapping of defaults used as globals during the evaluation of a section conditional expression. """ sections = {} # the current section condition, possibly updated from a section expression section_condition = True context = None cursect = None # None, or a dictionary blockmode = None optname = None lineno = 0 e = None # None, or an exception while True: line = fp.readline() if not line: break # EOF lineno = lineno + 1 if line[0] in '#;': continue # comment if line[0].isspace() and cursect is not None and optname: if not section_condition: #skip section based on its expression condition continue # continuation line if blockmode: line = line.rstrip() else: line = line.strip() if not line: continue cursect[optname] = "%s\n%s" % (cursect[optname], line) else: header = section_header(line) if header: # reset to True when starting a new section section_condition = True sectname = header.group('name') head = header.group('head') # the starting [ expression = header.group('expression') tail = header.group('tail') # closing ]and comment if expression: # normalize tail comments to Python style tail = tail.replace(';', '#') if tail else '' # un-escape literal # and ; . Do not use a # string-escape decode expr = expression.replace(r'\x23','#').replace(r'\x3b', ';') try: # new-style markers as used in pip constraints, e.g.: # 'python_version < "3.11" and platform_system == "Windows"' marker = Marker(expr) section_condition = marker.evaluate() except InvalidMarker: # old style buildout expression # rebuild a valid Python expression wrapped in a list expr = head + expr + tail # lazily populate context only expression if not context: context = exp_globals() # evaluated expression is in list: get first element section_condition = eval(expr, context)[0] # finally, ignore section when an expression # evaluates to false if not section_condition: logger.debug( 'Ignoring section %(sectname)r with [expression]:' ' %(expression)r' % locals()) continue if sectname in sections: cursect = sections[sectname] else: sections[sectname] = cursect = {} # So sections can't start with a continuation line optname = None elif cursect is None: if not line.strip(): continue # no section header in the file? raise MissingSectionHeaderError(fpname, lineno, line) else: if line[:2] == '=>': line = ' = ' + line[2:] mo = option_start(line) if mo: if not section_condition: # filter out options of conditionally ignored section continue # option start line optname, optval = mo.group('name', 'value') optname = optname.rstrip() optval = optval.strip() # Handle multiple extensions of the same value in the # same file. This happens with conditional sections. opt_op = optname[-1] if opt_op not in '+-': opt_op = '=' if optname in cursect and opt_op in '+-': # Strip any trailing \n, which happens when we have multiple # +=/-= in one file cursect[optname] = cursect[optname].rstrip() if optval: cursect[optname] = "%s\n%s" % (cursect[optname], optval) else: # If an assignment (=) comes after an extend (+=) / # remove (-=), it overrides and replaces the preceding # extend / remove if opt_op == '=': for suffix in '+-': tempname = "%s %s" % (optname, suffix) if tempname in cursect: del cursect[tempname] cursect[optname] = optval blockmode = not optval elif not (optname or line.strip()): # blank line after section start continue else: # a non-fatal parsing error occurred. set up the # exception but keep going. the exception will be # raised at the end of the file and will contain a # list of all bogus lines if not e: e = ParsingError(fpname) e.append(lineno, repr(line)) # if any parsing errors occurred, raise an exception if e: raise e for sectname in sections: section = sections[sectname] for name in section: value = section[name] if value[:1].isspace(): section[name] = leading_blank_lines.sub( '', textwrap.dedent(value.rstrip())) return sections ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.400452 zc_buildout-6.0.1/src/zc/buildout/download.py0000644000076500000240000002070415264141650020672 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Buildout download infrastructure""" from hashlib import md5 from urllib.request import urlretrieve from urllib.parse import urlparse from zc.buildout.easy_install import realpath import logging import os import os.path import re import shutil import sys import tempfile import zc.buildout class ChecksumError(zc.buildout.UserError): pass class Download(object): """Configurable download utility. Handles the download cache and offline mode. Download(options=None, cache=None, namespace=None, offline=False, fallback=False, hash_name=False, logger=None) options: mapping of buildout options (e.g. a ``buildout`` config section) cache: path to the download cache (excluding namespaces) namespace: namespace directory to use inside the cache offline: whether to operate in offline mode fallback: whether to use the cache as a fallback (try downloading first) hash_name: whether to use a hash of the URL as cache file name logger: an optional logger to receive download-related log messages """ def __init__(self, options=None, cache=-1, namespace=None, offline=-1, fallback=False, hash_name=False, logger=None): if options is None: options = {} self.directory = options.get('directory', '') self.cache = cache if cache == -1: self.cache = options.get('download-cache') self.namespace = namespace self.offline = offline if offline == -1: self.offline = (options.get('offline') == 'true' or options.get('install-from-cache') == 'true') self.fallback = fallback self.hash_name = hash_name self.logger = logger or logging.getLogger('zc.buildout') @property def download_cache(self): if self.cache is not None: return realpath(os.path.join(self.directory, self.cache)) @property def cache_dir(self): if self.download_cache is not None: return os.path.join(self.download_cache, self.namespace or '') def __call__(self, url, md5sum=None, path=None): """Download a file according to the utility's configuration. url: URL to download md5sum: MD5 checksum to match path: where to place the downloaded file Returns the path to the downloaded file. """ if self.cache: local_path, is_temp = self.download_cached(url, md5sum) else: local_path, is_temp = self.download(url, md5sum, path) return locate_at(local_path, path), is_temp def download_cached(self, url, md5sum=None): """Download a file from a URL using the cache. This method assumes that the cache has been configured. Optionally, it raises a ChecksumError if a cached copy of a file has an MD5 mismatch, but will not remove the copy in that case. """ if not os.path.exists(self.download_cache): raise zc.buildout.UserError( 'The directory:\n' '%r\n' "to be used as a download cache doesn't exist.\n" % self.download_cache) cache_dir = self.cache_dir if not os.path.exists(cache_dir): os.mkdir(cache_dir) cache_key = self.filename(url) cached_path = os.path.join(cache_dir, cache_key) self.logger.debug('Searching cache at %s' % cache_dir) if os.path.exists(cached_path): is_temp = False if self.fallback: try: _, is_temp = self.download(url, md5sum, cached_path) except ChecksumError: raise except Exception: pass if not check_md5sum(cached_path, md5sum): raise ChecksumError( 'MD5 checksum mismatch for cached download ' 'from %r at %r' % (url, cached_path)) self.logger.debug('Using cache file %s' % cached_path) else: self.logger.debug('Cache miss; will cache %s as %s' % (url, cached_path)) _, is_temp = self.download(url, md5sum, cached_path) return cached_path, is_temp def download(self, url, md5sum=None, path=None): """Download a file from a URL to a given or temporary path. An online resource is always downloaded to a temporary file and moved to the specified path only after the download is complete and the checksum (if given) matches. If path is None, the temporary file is returned and the client code is responsible for cleaning it up. """ # Make sure the drive letter in windows-style file paths isn't # interpreted as a URL scheme. if re.match(r"^[A-Za-z]:\\", url): url = 'file:' + url parsed_url = urlparse(url, 'file') url_scheme, _, url_path = parsed_url[:3] if url_scheme == 'file': self.logger.debug('Using local resource %s' % url) if not check_md5sum(url_path, md5sum): raise ChecksumError( 'MD5 checksum mismatch for local resource at %r.' % url_path) return locate_at(url_path, path), False if self.offline: raise zc.buildout.UserError( "Couldn't download %r in offline mode." % url) self.logger.info('Downloading %s' % url) handle, tmp_path = tempfile.mkstemp(prefix='buildout-') os.close(handle) try: tmp_path, headers = urlretrieve(url, tmp_path) if not check_md5sum(tmp_path, md5sum): raise ChecksumError( 'MD5 checksum mismatch downloading %r' % url) except IOError: e = sys.exc_info()[1] os.remove(tmp_path) raise zc.buildout.UserError("Error downloading extends for URL " "%s: %s" % (url, e)) except Exception: os.remove(tmp_path) raise if path: shutil.move(tmp_path, path) return path, False else: return tmp_path, True def filename(self, url): """Determine a file name from a URL according to the configuration. """ if self.hash_name: return md5(url.encode()).hexdigest() else: if re.match(r"^[A-Za-z]:\\", url): url = 'file:' + url parsed = urlparse(url, 'file') url_path = parsed[2] if parsed[0] == 'file': while True: url_path, name = os.path.split(url_path) if name: return name if not url_path: break else: for name in reversed(url_path.split('/')): if name: return name url_host, url_port = parsed[-2:] return '%s:%s' % (url_host, url_port) def check_md5sum(path, md5sum): """Tell whether the MD5 checksum of the file at path matches. No checksum being given is considered a match. """ if md5sum is None: return True f = open(path, 'rb') checksum = md5() try: chunk = f.read(2**16) while chunk: checksum.update(chunk) chunk = f.read(2**16) return checksum.hexdigest() == md5sum finally: f.close() def remove(path): if os.path.exists(path): os.remove(path) def locate_at(source, dest): if dest is None or realpath(dest) == realpath(source): return source if os.path.isdir(source): shutil.copytree(source, dest) else: try: os.link(source, dest) except (AttributeError, OSError): shutil.copyfile(source, dest) return dest ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4006488 zc_buildout-6.0.1/src/zc/buildout/easy_install.py0000644000076500000240000027503415264141650021562 0ustar00mauritsstaff############################################################################# # # Copyright (c) 2005 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Python easy_install API This module provides a high-level Python API for installing packages. It doesn't install scripts. It uses setuptools and requires it to be installed. """ import copy import distutils.errors import email import errno import glob import logging import operator import os import pkg_resources import posixpath import re import setuptools.archive_util import setuptools.command.setopt import shutil import subprocess import sys import tempfile import urllib.parse import zc.buildout import zc.buildout.rmtree import zipfile from . import _package_index from functools import cached_property from importlib import metadata from packaging import specifiers from packaging.utils import canonicalize_name from packaging.utils import is_normalized_name from pathlib import Path from pkg_resources import Distribution from setuptools.wheel import Wheel from zc.buildout import WINDOWS from zc.buildout.utils import normalize_name from zc.buildout.utils import get_pth_paths import warnings import csv BIN_SCRIPTS = 'Scripts' if WINDOWS else 'bin' warnings.filterwarnings( 'ignore', '.+is being parsed as a legacy, non PEP 440, version') _oprp = getattr(os.path, 'realpath', lambda path: path) def realpath(path): return os.path.normcase(os.path.abspath(_oprp(path))) default_index_url = os.environ.get( 'buildout_testing_index_url', 'https://pypi.org/simple', ) logger = logging.getLogger('zc.buildout.easy_install') macosVersionString = re.compile(r"macosx-(\d+)\.(\d+)-(.*)") url_match = re.compile('[a-z0-9+.-]+://').match is_source_encoding_line = re.compile(r'coding[:=]\s*([-\w.]+)').search # Source encoding regex from http://www.python.org/dev/peps/pep-0263/ is_win32 = sys.platform == 'win32' is_jython = sys.platform.startswith('java') if is_jython: import java.lang.System jython_os_name = (java.lang.System.getProperties()['os.name']).lower() # Include buildout and setuptools eggs in paths. We get this # initially from the entire working set. Later, we'll use the install # function to narrow to just the buildout and setuptools paths. buildout_and_setuptools_path = sorted({d.location for d in pkg_resources.working_set}) setuptools_path = buildout_and_setuptools_path pip_path = buildout_and_setuptools_path logger.debug('before restricting versions: pip_path %r', pip_path) FILE_SCHEME = re.compile('file://', re.I).match DUNDER_FILE_PATTERN = re.compile(r"__file__ = '(?P.+)'$") class EnvironmentMixin(object): """Mixin class for Environment and PackageIndex for canonicalized names. * pkg_resources defines the Environment class * setuptools defines a PackageIndex class that inherits from Environment * Buildout needs a few fixes that should be used by both. The fixes are needed for this issue, where distributions created by setuptools 69.3+ get a different name than with older versions: https://github.com/buildout/buildout/issues/647 """ def __getitem__(self, project_name): """Return a newest-to-oldest list of distributions for `project_name` Uses case-insensitive `project_name` comparison, assuming all the project's distributions use their project's name converted to all lowercase as their key. """ distribution_key = normalize_name(project_name) return self._distmap.get(distribution_key, []) def add(self, dist): """Add `dist` if we ``can_add()`` it and it has not already been added """ if self.can_add(dist) and dist.has_version(): # Instead of 'dist.key' we add a normalized version. distribution_key = normalize_name(dist.key) dists = self._distmap.setdefault(distribution_key, []) if dist not in dists: dists.append(dist) dists.sort(key=operator.attrgetter('hashcmp'), reverse=True) class Environment(EnvironmentMixin, pkg_resources.Environment): """Buildout version of Environment with canonicalized names. * pkg_resources defines the Environment class * setuptools defines a PackageIndex class that inherits from Environment * Buildout needs a few fixes that should be used by both. The fixes are needed for this issue, where distributions created by setuptools 69.3+ get a different name than with older versions: https://github.com/buildout/buildout/issues/647 And since May 2025 we override the can_add method to work better on Mac: accept distributions when the architecture (machine type) matches, instead of failing when the major or minor version do not match. See long explanation in https://github.com/buildout/buildout/pull/707 It boils down to this, depending on how you installed Python: % bin/zopepy >>> import pkg_resources >>> pkg_resources.get_platform() 'macosx-11.0-arm64' >>> pkg_resources.get_supported_platform() 'macosx-15.4-arm64' Here macosx-11.0 is the platform on which the Python was built/compiled. And macosx-15.4 is the current platform (my laptop). This gives problems when we get a Mac-specific wheel. We turn it into an egg that has the result of get_supported_platform() in its name. Then our code in easy_install._get_matching_dist_in_location creates a pkg_resources.Environment with the egg location. Under the hood, pkg_resources.compatible_platforms is called, and this does not find any matching dists because it compares the platform in the egg name with that of the system, which is pkg_resources.get_platform(). So an egg created on the current machine by the current Python may not be recognized. This is obviously wrong. """ @cached_property def _mac_machine_type(self): """Machine type (architecture) on Mac. Adapted from pkg_resources.compatible_platforms. If self.platform is something like 'macosx-15.4-arm64', we return 'arm64. """ match = macosVersionString.match(self.platform) if match is None: # no Mac return "" return match.group(3) def can_add(self, dist: Distribution) -> bool: """Is distribution `dist` acceptable for this environment? The distribution must match the platform and python version requirements specified when this environment was created, or False is returned. For Mac we make a change compared to the original. Platforms like 'macosx-11.0-arm64' and 'macosx-15.4-arm64' are considered compatible. """ if super().can_add(dist): return True if sys.platform != "darwin": # Our override is only useful on Mac OSX. return False # The rest of the code is a combination of the original # pkg_resources.Environment.can_add and pkg_resources.compatible_platforms. py_compat = ( self.python is None or dist.py_version is None or dist.py_version == self.python ) if not py_compat: return False provMac = macosVersionString.match(dist.platform) if not provMac: # The dist is not for Mac. return False provided_machine_type = provMac.group(3) if provided_machine_type != self._mac_machine_type: return False logger.debug( "Accepted dist %s although its provided platform %s does not " "match our supported platform %s.", dist, dist.platform, self.platform, ) return True class AllowHostsPackageIndex(EnvironmentMixin, _package_index.PackageIndex): """Will allow urls that are local to the system. This class had its own url_ok method, but we merged this into _package_index.py. """ pass _indexes = {} def _get_index(index_url, find_links, allow_hosts=('*',)): key = index_url, tuple(find_links) index = _indexes.get(key) if index is not None: return index if index_url is None: index_url = default_index_url if index_url.startswith('file://'): index_url = index_url[7:] index = AllowHostsPackageIndex(index_url, hosts=allow_hosts) if find_links: index.add_find_links(find_links) _indexes[key] = index return index clear_index_cache = _indexes.clear if is_win32: # work around spawn lamosity on windows # XXX need safe quoting (see the subproces.list2cmdline) and test def _safe_arg(arg): return '"%s"' % arg else: _safe_arg = str if is_win32: # In setuptools 80.3 the setuptools.command.easy_install module was first # removed, and later only partially restored as wrapper around the new # setuptools._scripts module. try: from setuptools._scripts import get_win_launcher except ImportError: from setuptools.command.easy_install import get_win_launcher else: get_win_launcher = None def call_subprocess(args, **kw): if subprocess.call(args, **kw) != 0: raise Exception( "Failed to run command:\n%s" % repr(args)[1:-1]) def get_subprocess_output(args, **kw): result = subprocess.run( args, **kw, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, ) stdout = result.stdout.decode("utf-8") if result.returncode: cmd = repr(args)[1:-1] msg = f"Failed to run command:\n{cmd}" logger.error(msg + "\nError output follows:") print(stdout) raise Exception(msg) return stdout def _execute_permission(): current_umask = os.umask(0o022) # os.umask only returns the current umask if you also give it one, so we # have to give it a dummy one and immediately set it back to the real # value... Distribute does the same. os.umask(current_umask) return 0o777 - current_umask def get_namespace_package_paths(dist): """ Generator of the expected pathname of each __init__.py file of the namespaces of a distribution. """ base = [dist.location] init = ['__init__.py'] for namespace in dist.get_metadata_lines('namespace_packages.txt'): yield os.path.join(*(base + namespace.split('.') + init)) def namespace_packages_need_pkg_resources(dist): if os.path.isfile(dist.location): # Zipped egg, with namespaces, surely needs setuptools return True # If they have `__init__.py` files that use pkg_resources and don't # fallback to using `pkgutil`, then they need setuptools/pkg_resources: for path in get_namespace_package_paths(dist): if os.path.isfile(path): with open(path, 'rb') as f: source = f.read() if (source and b'pkg_resources' in source and not b'pkgutil' in source): return True return False def dist_needs_pkg_resources(dist): """ A distribution needs setuptools/pkg_resources added as requirement if: * It has namespace packages declared with: - `pkg_resources.declare_namespace()` * Those namespace packages don't fall back to `pkgutil` Until and including zc.buildout 5.x we had an extra condition: * It doesn't have `setuptools/pkg_resources` as requirement already But this does not catch the case where a distribution *does* depend on setuptools, but we have a too new one (82+) which does not actually have the pkg_resources module anymore. So we removed this condition. """ return ( dist.has_metadata('namespace_packages.txt') and namespace_packages_need_pkg_resources(dist) ) class IncompatibleConstraintError(zc.buildout.UserError): """A specified version is incompatible with a given requirement. """ IncompatibleVersionError = IncompatibleConstraintError # Backward compatibility def _constrained_requirement(constraint, requirement): assert isinstance(requirement, pkg_resources.Requirement) if constraint[0] not in '<>': if constraint.startswith('='): assert constraint.startswith('==') version = constraint[2:] else: version = constraint constraint = '==' + constraint if version not in requirement: msg = ("The requirement (%r) is not allowed by your [versions] " "constraint (%s)" % (str(requirement), version)) raise IncompatibleConstraintError(msg) specifier = specifiers.SpecifierSet(constraint) else: specifier = requirement.specifier & constraint constrained = copy.deepcopy(requirement) constrained.specifier = specifier return pkg_resources.Requirement.parse(str(constrained)) class Installer(object): _versions = {} _required_by = {} _picked_versions = {} _download_cache = None _install_from_cache = False _prefer_final = True _use_dependency_links = True _allow_picked_versions = True _store_required_by = False _allow_unknown_extras = False _namespace_packages = {} _index_url = None def __init__(self, dest=None, links=(), index=None, executable=sys.executable, always_unzip=None, # Backward compat :/ path=None, newest=True, versions=None, use_dependency_links=None, allow_hosts=('*',), check_picked=True, allow_unknown_extras=False, ): assert executable == sys.executable, (executable, sys.executable) self._dest = dest if dest is None else pkg_resources.normalize_path(dest) self._allow_hosts = allow_hosts self._allow_unknown_extras = allow_unknown_extras if self._install_from_cache: if not self._download_cache: raise ValueError("install_from_cache set to true with no" " download cache") links = () index = 'file://' + self._download_cache if use_dependency_links is not None: self._use_dependency_links = use_dependency_links self._links = links = list(self._fix_file_links(links)) if self._download_cache and (self._download_cache not in links): links.insert(0, self._download_cache) if index: self._index_url = index path = (path and path[:] or []) + buildout_and_setuptools_path self._path = path if self._dest is None: newest = False self._newest = newest self._env = self._make_env() self._index = _get_index(index, links, self._allow_hosts) self._requirements_and_constraints = [] self._check_picked = check_picked if versions is not None: self._versions = normalize_versions(versions) def _make_env(self): full_path = self._get_dest_dist_paths() + self._path env = Environment(full_path) # this needs to be called whenever self._env is modified (or we could # make an Environment subclass): self._eggify_env_dest_dists(env, self._dest) return env def _env_rescan_dest(self): self._env.scan(self._get_dest_dist_paths()) self._eggify_env_dest_dists(self._env, self._dest) def _get_dest_dist_paths(self): dest = self._dest if dest is None: return [] eggs = glob.glob(os.path.join(dest, '*.egg')) dists = [os.path.dirname(dist_info) for dist_info in glob.glob(os.path.join(dest, '*', '*.dist-info'))] return list(set(eggs + dists)) @staticmethod def _eggify_env_dest_dists(env, dest): """ Make sure everything found under `dest` is seen as an egg, even if it's some other kind of dist. """ for project_name in env: for dist in env[project_name]: if os.path.dirname(dist.location) == dest: dist.precedence = pkg_resources.EGG_DIST def _version_conflict_information(self, name): """Return textual requirements/constraint information for debug purposes We do a very simple textual search, as that filters out most extraneous information without missing anything. """ output = [ "Version and requirements information containing %s:" % name] version_constraint = self._versions.get(canonicalize_name(name)) if version_constraint: output.append( "[versions] constraint on %s: %s" % (name, version_constraint)) output += [line for line in self._requirements_and_constraints if name.lower() in line.lower()] return '\n '.join(output) def _satisfied(self, req, source=None): dists = [dist for dist in self._env[req.project_name] if dist in req] if not dists: logger.debug('We have no distributions for %s that satisfies %r.', req.project_name, str(req)) return None, self._obtain(req, source) # Note that dists are sorted from best to worst, as promised by # env.__getitem__ for dist in dists: if (dist.precedence == pkg_resources.DEVELOP_DIST): logger.debug('We have a develop egg: %s', dist) return dist, None # Special common case, we have a specification for a single version: specs = req.specs if len(specs) == 1 and specs[0][0] == '==': logger.debug('We have the distribution that satisfies %r.', str(req)) return dists[0], None if self._prefer_final: fdists = [dist for dist in dists if self._final_version(dist.parsed_version) ] if fdists: # There are final dists, so only use those dists = fdists if not self._newest: # We don't need the newest, so we'll use the newest one we # find, which is the first returned by # Environment.__getitem__. return dists[0], None best_we_have = dists[0] # Because dists are sorted from best to worst # We have some installed distros. There might, theoretically, be # newer ones. Let's find out which ones are available and see if # any are newer. We only do this if we're willing to install # something, which is only true if dest is not None: best_available = self._obtain(req, source) if best_available is None: # That's a bit odd. There aren't any distros available. # We should use the best one we have that meets the requirement. logger.debug( 'There are no distros available that meet %r.\n' 'Using our best, %s.', str(req), best_we_have) return best_we_have, None if self._prefer_final: if self._final_version(best_available.parsed_version): if self._final_version(best_we_have.parsed_version): if (best_we_have.parsed_version < best_available.parsed_version ): return None, best_available else: return None, best_available else: if (not self._final_version(best_we_have.parsed_version) and (best_we_have.parsed_version < best_available.parsed_version ) ): return None, best_available else: if (best_we_have.parsed_version < best_available.parsed_version ): return None, best_available logger.debug( 'We have the best distribution that satisfies %r.', str(req)) return best_we_have, None def _call_pip_install(self, spec, dest, dist): tmp = tempfile.mkdtemp(dir=dest) try: paths = call_pip_install(spec, tmp) dists = [] env = Environment(paths) for project in env: dists.extend(env[project]) if not dists: raise zc.buildout.UserError("Couldn't install: %s" % dist) if len(dists) > 1: logger.warn("Installing %s\n" "caused multiple distributions to be installed:\n" "%s\n", dist, '\n'.join(map(str, dists))) else: d = dists[0] if d.project_name != dist.project_name: logger.warn("Installing %s\n" "Caused installation of a distribution:\n" "%s\n" "with a different project name.", dist, d) if d.version != dist.version: logger.warn("Installing %s\n" "Caused installation of a distribution:\n" "%s\n" "with a different version.", dist, d) result = [] for d in dists: result.append(_move_to_eggs_dir_and_compile(d, dest)) return result finally: zc.buildout.rmtree.rmtree(tmp) def _obtain(self, requirement, source=None): # initialize out index for this project: index = self._index if index.obtain(requirement) is None: # Nothing is available. return None # Filter the available dists for the requirement and source flag dists = [dist for dist in index[requirement.project_name] if ((dist in requirement) and ((not source) or (dist.precedence == pkg_resources.SOURCE_DIST) ) ) ] # If we prefer final dists, filter for final and use the # result if it is non empty. if self._prefer_final: fdists = [dist for dist in dists if self._final_version(dist.parsed_version) ] if fdists: # There are final dists, so only use those dists = fdists # Now find the best one: best = [] bestv = None for dist in dists: distv = dist.parsed_version if bestv is None or distv > bestv: best = [dist] bestv = distv elif distv == bestv: best.append(dist) if not best: return None if len(best) == 1: return best[0] if self._download_cache: for dist in best: if (realpath(os.path.dirname(dist.location)) == self._download_cache ): return dist best.sort() return best[-1] def _fetch(self, dist, tmp, download_cache): if (download_cache and (realpath(os.path.dirname(dist.location)) == download_cache) ): logger.debug("Download cache has %s at: %s", dist, dist.location) return dist logger.debug("Fetching %s from: %s", dist, dist.location) new_location = self._index.download(dist.location, tmp) if (download_cache and (realpath(new_location) == realpath(dist.location)) and os.path.isfile(new_location) ): # setuptools avoids making extra copies, but we want to copy # to the download cache shutil.copy2(new_location, tmp) new_location = os.path.join(tmp, os.path.basename(new_location)) return dist.clone(location=new_location) def _get_dist(self, requirement, ws): __doing__ = 'Getting distribution for %r.', str(requirement) # Maybe an existing dist is already the best dist that satisfies the # requirement. If not, get a link to an available distribution that # we could download. The method returns a tuple with an existing # dist or an available dist. Either 'dist' is None, or 'avail' # is None, or both are None. dist, avail = self._satisfied(requirement) if dist is None: if self._dest is None: raise zc.buildout.UserError( "We don't have a distribution for %s\n" "and can't install one in offline (no-install) mode.\n" % requirement) logger.info(*__doing__) if avail is None: # We have no existing dist, and none is available for download. raise MissingDistribution(requirement, ws) # We may overwrite distributions, so clear importer # cache. sys.path_importer_cache.clear() tmp = self._download_cache if tmp is None: tmp = tempfile.mkdtemp('get_dist') try: dist = self._fetch(avail, tmp, self._download_cache) if dist is None: raise zc.buildout.UserError( "Couldn't download distribution %s." % avail) dists = [_move_to_eggs_dir_and_compile(dist, self._dest)] for _d in dists: if _d not in ws: ws.add(_d, replace=True) finally: if tmp != self._download_cache: zc.buildout.rmtree.rmtree(tmp) self._env_rescan_dest() dist = self._env.best_match(requirement, ws) logger.info("Got %s.", dist) else: dists = [dist] if dist not in ws: ws.add(dist) if not self._install_from_cache and self._use_dependency_links: self._add_dependency_links_from_dists(dists) if self._check_picked: self._check_picked_requirement_versions(requirement, dists) return dists def _add_dependency_links_from_dists(self, dists): reindex = False links = self._links for dist in dists: if dist.has_metadata('dependency_links.txt'): for link in dist.get_metadata_lines('dependency_links.txt'): link = link.strip() if link not in links: logger.debug('Adding find link %r from %s', link, dist) links.append(link) reindex = True if reindex: self._index = _get_index(self._index_url, links, self._allow_hosts) def _check_picked_requirement_versions(self, requirement, dists): """ Check whether we picked a version and, if we did, report it """ for dist in dists: if not (dist.precedence == pkg_resources.DEVELOP_DIST or (len(requirement.specs) == 1 and requirement.specs[0][0] == '==') ): logger.debug('Picked: %s = %s', dist.project_name, dist.version) self._picked_versions[dist.project_name] = dist.version if not self._allow_picked_versions: msg = NOT_PICKED_AND_NOT_ALLOWED.format( name=dist.project_name, version=dist.version ) raise zc.buildout.UserError(msg) def _maybe_add_setuptools(self, ws, dist): if dist_needs_pkg_resources(dist): # We have a namespace package but no requirement for setuptools if dist.precedence == pkg_resources.DEVELOP_DIST: logger.warning( "Develop distribution: %s\n" "uses namespace packages but the distribution " "does not require setuptools.", dist) requirement = self._constrain( pkg_resources.Requirement.parse('setuptools') ) if not requirement.specs: # The user did not pin setuptools. Scripts using # pkg_resources-style namespace packages run # `import pkg_resources` at script run time, outside the # buildout process where the copy vendored by zc.buildout # is not importable, so prefer a setuptools version that # still ships pkg_resources (it was removed in 82). preferred = _constrained_requirement('<82', requirement) if ws.find(preferred) is not None: return try: self._get_dist(preferred, ws) return except zc.buildout.UserError: # No setuptools < 82 available, for example offline # with only a newer setuptools installed. Fall back # to any setuptools version: better a working set # with a pkg_resources-less setuptools than an # error. Buildout already warns loudly about # old-style namespace packages elsewhere. logger.debug( "Could not get setuptools<82 (the last version " "that ships pkg_resources) for %s; falling back " "to any setuptools version.", dist) if ws.find(requirement) is None: self._get_dist(requirement, ws) def _constrain(self, requirement): """Return requirement with optional [versions] constraint added.""" canonical_name = canonicalize_name(requirement.project_name) constraint = self._versions.get(canonical_name) if constraint: try: requirement = _constrained_requirement(constraint, requirement) except IncompatibleConstraintError: logger.info(self._version_conflict_information(canonical_name)) raise return requirement def install(self, specs, working_set=None): logger.debug('Installing %s.', repr(specs)[1:-1]) self._requirements_and_constraints.append( "Base installation request: %s" % repr(specs)[1:-1]) for_buildout_run = bool(working_set) requirements = [pkg_resources.Requirement.parse(spec) for spec in specs] requirements = [ self._constrain(requirement) for requirement in requirements if not requirement.marker or requirement.marker.evaluate() ] if working_set is None: ws = pkg_resources.WorkingSet([]) else: ws = working_set for requirement in requirements: for dist in self._get_dist(requirement, ws): self._maybe_add_setuptools(ws, dist) # OK, we have the requested distributions and they're in the working # set, but they may have unmet requirements. We'll resolve these # requirements. This is code modified from # pkg_resources.WorkingSet.resolve. We can't reuse that code directly # because we have to constrain our requirements (see # versions_section_ignored_for_dependency_in_favor_of_site_packages in # zc.buildout.tests). requirements.reverse() # Set up the stack. processed = {} # This is a set of processed requirements. best = {} # This is a mapping of package name -> dist. # Note that we don't use the existing environment, because we want # to look for new eggs unless what we have is the best that # matches the requirement. env = Environment(ws.entries) while requirements: # Process dependencies breadth-first. current_requirement = requirements.pop(0) req = self._constrain(current_requirement) if req in processed: # Ignore cyclic or redundant dependencies. continue dist = best.get(req.key) if dist is None: try: dist = env.best_match(req, ws) except pkg_resources.VersionConflict as err: logger.debug( "Version conflict while processing requirement %s " "(constrained to %s)", current_requirement, req) # Installing buildout itself and its extensions and # recipes requires the global # ``pkg_resources.working_set`` to be active, which also # includes all system packages. So there might be # conflicts, which are fine to ignore. We'll grab the # correct version a few lines down. if not for_buildout_run: raise VersionConflict(err, ws) if dist is None: if self._dest: logger.debug('Getting required %r', str(req)) else: logger.debug('Adding required %r', str(req)) self._log_requirement(ws, req) for dist in self._get_dist(req, ws): self._maybe_add_setuptools(ws, dist) if dist not in req: # Oops, the "best" so far conflicts with a dependency. logger.info(self._version_conflict_information(req.key)) raise VersionConflict( pkg_resources.VersionConflict(dist, req), ws) best[req.key] = dist missing_requested = sorted( set(req.extras) - set(dist.extras) ) for missing in missing_requested: logger.warning( '%s does not provide the extra \'%s\'', dist, missing ) if missing_requested: if not self._allow_unknown_extras: raise zc.buildout.UserError( "Couldn't find the required extra. " "This means the requirement is incorrect. " "If the requirement is itself from software you " "requested, then there might be a bug in " "requested software. You can ignore this by " "using 'allow-unknown-extras=true', however " "that may simply cause needed software to be omitted." ) extra_requirements = sorted( set(dist.extras) & set(req.extras) ) else: extra_requirements = dist.requires(req.extras)[::-1] for extra_requirement in extra_requirements: self._requirements_and_constraints.append( "Requirement of %s: %s" % ( current_requirement, extra_requirement)) requirements.extend(extra_requirements) processed[req] = True return ws def build(self, spec, build_ext): requirement = self._constrain(pkg_resources.Requirement.parse(spec)) dist, avail = self._satisfied(requirement, 1) if dist is not None: return [dist.location] # Retrieve the dist: if avail is None: raise zc.buildout.UserError( "Couldn't find a source distribution for %r." % str(requirement)) if self._dest is None: raise zc.buildout.UserError( "We don't have a distribution for %s\n" "and can't build one in offline (no-install) mode.\n" % requirement ) logger.debug('Building %r', spec) tmp = self._download_cache if tmp is None: tmp = tempfile.mkdtemp('get_dist') try: dist = self._fetch(avail, tmp, self._download_cache) build_tmp = tempfile.mkdtemp('build') try: setuptools.archive_util.unpack_archive(dist.location, build_tmp) base = build_tmp if not os.path.exists(os.path.join(build_tmp, 'setup.py')): setups = glob.glob( os.path.join(build_tmp, '*', 'setup.py')) if not setups: # We used to raise an error, but now we just log a warning. # Maybe there is a pyproject.toml file that pip can use. # Otherwise we let pip do the complaining. logger.warning( "Couldn't find a setup script to build in %s. " "Trying pip install anyway." % os.path.basename(dist.location) ) elif len(setups) > 1: raise distutils.errors.DistutilsError( "Multiple setup scripts in %s" % os.path.basename(dist.location) ) else: base = os.path.dirname(setups[0]) setup_cfg = os.path.join(base, 'setup.cfg') if not os.path.exists(setup_cfg): f = open(setup_cfg, 'w') f.close() setuptools.command.setopt.edit_config( setup_cfg, dict(build_ext=build_ext)) dists = self._call_pip_install(base, self._dest, dist) return [dist.location for dist in dists] finally: zc.buildout.rmtree.rmtree(build_tmp) finally: if tmp != self._download_cache: zc.buildout.rmtree.rmtree(tmp) def _fix_file_links(self, links): for link in links: if link.startswith('file://') and link[-1] != '/': if os.path.isdir(link[7:]): # work around excessive restriction in setuptools: link += '/' yield link def _log_requirement(self, ws, req): if (not logger.isEnabledFor(logging.DEBUG) and not Installer._store_required_by): # Sorting the working set and iterating over it's requirements # is expensive, so short circuit the work if it won't even be # logged. When profiling a simple buildout with 10 parts with # identical and large working sets, this resulted in a # decrease of run time from 93.411 to 15.068 seconds, about a # 6 fold improvement. return ws = list(ws) ws.sort() for dist in ws: if req in dist.requires(): logger.debug(" required by %s." % dist) req_ = str(req) if req_ not in Installer._required_by: Installer._required_by[req_] = set() Installer._required_by[req_].add(str(dist.as_requirement())) def _final_version(self, parsed_version): return not parsed_version.is_prerelease def normalize_versions(versions): """Return version dict with keys canonicalized. PyPI is case-insensitive and not all distributions are consistent in their own naming. Also, there are dashes, underscores, dots... """ return dict([(canonicalize_name(k), v) for (k, v) in versions.items()]) def default_versions(versions=None): old = Installer._versions if versions is not None: Installer._versions = normalize_versions(versions) return old def download_cache(path=-1): old = Installer._download_cache if path != -1: if path: path = realpath(path) Installer._download_cache = path return old def install_from_cache(setting=None): old = Installer._install_from_cache if setting is not None: Installer._install_from_cache = bool(setting) return old def prefer_final(setting=None): old = Installer._prefer_final if setting is not None: Installer._prefer_final = bool(setting) return old def use_dependency_links(setting=None): old = Installer._use_dependency_links if setting is not None: Installer._use_dependency_links = bool(setting) return old def index_url(setting=None): old = Installer._index_url if setting is not None: Installer._index_url = setting return old def allow_picked_versions(setting=None): old = Installer._allow_picked_versions if setting is not None: Installer._allow_picked_versions = bool(setting) return old def store_required_by(setting=None): old = Installer._store_required_by if setting is not None: Installer._store_required_by = bool(setting) return old def get_picked_versions(): picked_versions = sorted(Installer._picked_versions.items()) required_by = Installer._required_by return (picked_versions, required_by) def get_namespace_packages(): return sorted(Installer._namespace_packages.items()) def install(specs, dest, links=(), index=None, executable=sys.executable, always_unzip=None, # Backward compat :/ path=None, working_set=None, newest=True, versions=None, use_dependency_links=None, allow_hosts=('*',), include_site_packages=None, allowed_eggs_from_site_packages=None, check_picked=True, allow_unknown_extras=False, ): assert executable == sys.executable, (executable, sys.executable) assert include_site_packages is None assert allowed_eggs_from_site_packages is None installer = Installer(dest, links, index, sys.executable, always_unzip, path, newest, versions, use_dependency_links, allow_hosts=allow_hosts, check_picked=check_picked, allow_unknown_extras=allow_unknown_extras) return installer.install(specs, working_set) buildout_and_setuptools_dists = list(install(['zc.buildout'], None, check_picked=False)) buildout_and_setuptools_path = sorted({d.location for d in buildout_and_setuptools_dists}) pip_dists = [d for d in buildout_and_setuptools_dists if d.project_name != 'zc.buildout'] pip_path = sorted({d.location for d in pip_dists}) logger.debug('after restricting versions: pip_path %r', pip_path) pip_pythonpath = os.pathsep.join(pip_path) setuptools_path = pip_path setuptools_pythonpath = pip_pythonpath def build(spec, dest, build_ext, links=(), index=None, executable=sys.executable, path=None, newest=True, versions=None, allow_hosts=('*',)): assert executable == sys.executable, (executable, sys.executable) installer = Installer(dest, links, index, executable, True, path, newest, versions, allow_hosts=allow_hosts) return installer.build(spec, build_ext) def _rm(*paths): for path in paths: if os.path.isdir(path): zc.buildout.rmtree.rmtree(path) elif os.path.exists(path): os.remove(path) def _create_egg_link(directory, dest, egg_name): """Create egg-link file. setuptools 80 basically removes its own 'setup.py develop' code, and replaces it with 'pip install -e' (which then calls setuptools again, but okay). See https://github.com/pypa/setuptools/pull/4955 This leads to a different outcome. There is no longer an .egg-link file that we can copy. So we create it ourselves, based on the previous setuptools code. So what should be in the .egg-link file? Two lines: an egg path and a relative setup.py path. For example with setuptools 79 we may have a file zc.recipe.egg.egg-link with as contents two lines: /Users/maurits/community/buildout/zc.recipe.egg_/src ../ The relative setup.py path on the second line does not seem really used, but it should be there according to some checks, so let's try to get it right. There is only so much we can do, but we support two common cases: a src-layout and a layout with the code starting at the same level as the setup.py file. """ egg_path = os.path.realpath(directory) assert os.path.isdir(egg_path) if not egg_name: egg_name = os.path.basename(egg_path) if 'src' in os.listdir(egg_path): egg_path = os.path.join(egg_path, 'src') setup_path = '..' else: setup_path = '.' # Return TWO lines, so NO line ending on the last line. contents = f"{egg_path}\n{setup_path}" egg_link = os.path.join(dest, egg_name) + '.egg-link' with open(egg_link, "w") as myfile: myfile.write(contents) return egg_link def _copyeggs(src, dest, suffix, undo): """Copy eggs from src to dest. * 'src' is the temporary directory where eggs were built * 'dest' is the 'develop-eggs' directory * 'suffix' is '.egg-link' * 'undo' is a list of cleanup actions that will be undone automatically * after this function returns (or throws an exception). The only thing we need to do: find the file with the given suffix in src, and move it to dest. This works until and including setuptools 79. For setuptools 80+ we call _create_egg_link. """ egg_links = glob.glob(os.path.join(src, "*" + suffix)) if egg_links: assert len(egg_links) == 1, str(egg_links) egg_link = egg_links[0] name = os.path.basename(egg_link) new = os.path.join(dest, name) _rm(new) os.rename(egg_link, new) return new def _copy_metadata(src, dest, undo): for name in os.listdir(src): if name == '__pycache__': continue if not (name.endswith('.dist-info') or name.endswith('.egg-info') or name.endswith('.pth') or (name.startswith('__editable__') and name.endswith('.py'))): continue new = os.path.join(dest, name) _rm(new) old = os.path.join(src, name) os.rename(old, new) _develop_distutils_scripts = {} def _detect_distutils_scripts(directory): """Record detected distutils scripts from develop eggs ``setup.py develop`` doesn't generate metadata on distutils scripts, in contrast to ``setup.py install``. So we have to store the information for later. This won't find anything on setuptools 80.0.0+, because this does the editable install with pip, instead of its previous own code. The result is different. There is no egg-link file, so our code stops early. Maybe we could skip this check, use a different way of getting the proper egg_name, and still look for the 'EASY-INSTALL-DEV-SCRIPT' marker that setuptools adds. But after setuptools 80.3.0 this marker is not set anymore: the setuptools.command.easy_install module was first removed, and later only partially restored. So if we would change the logic here, it would only be potentially useful for a very short range of setuptools versions. Also, we look for distutils scripts, which sounds like something that is long deprecated. """ dir_contents = os.listdir(directory) # TODO For newer dists maybe just look for a 'bin' directory and get # any script in there. egginfo_filenames = [filename for filename in dir_contents if filename.endswith('.egg-link')] if not egginfo_filenames: return egg_name = egginfo_filenames[0].replace('.egg-link', '') marker = 'EASY-INSTALL-DEV-SCRIPT' scripts_found = [] for filename in dir_contents: if filename.endswith('.exe'): continue filepath = os.path.join(directory, filename) if not os.path.isfile(filepath): continue with open(filepath) as fp: dev_script_content = fp.read() if marker in dev_script_content: # The distutils bin script points at the actual file we need. for line in dev_script_content.splitlines(): match = DUNDER_FILE_PATTERN.search(line) if match: # The ``__file__ =`` line in the generated script points # at the actual distutils script we need. actual_script_filename = match.group('filename') with open(actual_script_filename) as fp: actual_script_content = fp.read() scripts_found.append([filename, actual_script_content]) if scripts_found: logger.debug( "Distutils scripts found for develop egg %s: %s", egg_name, scripts_found) _develop_distutils_scripts[egg_name] = scripts_found def develop(setup, dest, build_ext=None, executable=sys.executable): """Make a development/editable install of a package. This expects to get a path to a directory or a file as the first argument. If it is a file, we used to expect it to be a `setup.py` file. And then we would basically call `python setup.py develop`. Nowadays it could also be a `pyproject.toml` file. Calling `setup.py develop` is a deprecated way of installing a package. In setuptools 80 this still works, but setuptools has internally changed to call `pip install`. Since zc.buildout 5 we also do that. We basically ignore the file, and just get its directory instead. With the `build_ext` option you can influence how C extensions in the package are built. This may not be possible in a project that is using hatchling, unless you have some hatchling extensions. So the current code assumes you are using setuptools. It will create or edit a `setup.cfg` file in the package directory and put the build_ext options in there. """ assert executable == sys.executable, (executable, sys.executable) if os.path.isdir(setup): directory = setup else: directory = os.path.dirname(setup) # We will be calling `pip install -e directory` later on. This works when # directory is `src/something`. But if it is just `something`, pip will # try to get `something` from PyPI, even if there is a sub directory # `something`. So let's make it an absolute path. # See https://github.com/buildout/buildout/issues/734 # Let's also handle '~/'. directory = Path(directory).expanduser().resolve() logger.debug("Making editable install of %s", setup) undo = [] try: if build_ext: setup_cfg = os.path.join(directory, 'setup.cfg') if os.path.exists(setup_cfg): os.rename(setup_cfg, setup_cfg+'-develop-aside') def restore_old_setup(): if os.path.exists(setup_cfg): os.remove(setup_cfg) os.rename(setup_cfg+'-develop-aside', setup_cfg) undo.append(restore_old_setup) else: f = open(setup_cfg, 'w') f.close() undo.append(lambda: os.remove(setup_cfg)) setuptools.command.setopt.edit_config( setup_cfg, dict(build_ext=build_ext)) tmp3 = tempfile.mkdtemp('build', dir=dest) undo.append(lambda : zc.buildout.rmtree.rmtree(tmp3)) egg_name = call_pip_install(directory.as_uri(), tmp3, editable=True) # output = get_subprocess_output(args) # if log_level <= logging.DEBUG: # print(output) # This won't find anything on setuptools 80+. # Can't be helped, I think. _detect_distutils_scripts(tmp3) # Older setuptools versions (before 80) might have created an .egg-link # file in the temporary directory. egg_link = _copyeggs(tmp3, dest, '.egg-link', undo) if egg_link: logger.debug("Successfully made editable install: %s", egg_link) return egg_link # For newer setuptools (80+) or other PEP 660 backends, we copy # the metadata and finders to the destination. _copy_metadata(tmp3, dest, undo) _detect_distutils_scripts(tmp3) # We also create an .egg-link file for backward compatibility with # existing tests and tools that expect it. egg_link = _create_egg_link(directory, dest, egg_name) if egg_link: logger.debug("Successfully made editable install: %s", egg_link) return egg_link finally: undo.reverse() [f() for f in undo] def working_set(specs, executable, path=None, include_site_packages=None, allowed_eggs_from_site_packages=None): # Backward compat: if path is None: path = executable else: assert executable == sys.executable, (executable, sys.executable) assert include_site_packages is None assert allowed_eggs_from_site_packages is None return install(specs, None, path=path) def scripts(reqs, working_set, executable, dest=None, scripts=None, extra_paths=(), arguments='', interpreter=None, initialization='', relative_paths=False, ): assert executable == sys.executable, (executable, sys.executable) path = [dist.location for dist in working_set] # Collect additional paths from .pth files in distribution locations # (relevant for PEP 660 editable installs). for loc in list(path): for p in get_pth_paths(loc): if p not in path: path.append(p) path.extend(extra_paths) # order preserving unique unique_path = [] for p in path: if p not in unique_path: unique_path.append(p) path = [realpath(p) for p in unique_path] generated = [] if isinstance(reqs, str): raise TypeError('Expected iterable of requirements or entry points,' ' got string.') if initialization: initialization = '\n'+initialization+'\n' entry_points = [] distutils_scripts = [] for req in reqs: if isinstance(req, str): orig_req = pkg_resources.Requirement.parse(req) if orig_req.marker and not orig_req.marker.evaluate(): continue dist = None if is_normalized_name(orig_req.name): dist = working_set.find(orig_req) if dist is None: raise ValueError( f"Could not find requirement '{orig_req.name}' in working set. " ) else: # First try finding the package by its canonical name. canonicalized_name = canonicalize_name(orig_req.name) canonical_req = pkg_resources.Requirement.parse(canonicalized_name) dist = working_set.find(canonical_req) if dist is None: # Now try to find the package by the original name we got from # the requirements. This may succeed with setuptools versions # older than 75.8.2. dist = working_set.find(orig_req) if dist is None: raise ValueError( f"Could not find requirement '{orig_req.name}' in working " f"set. Could not find it with normalized " f"'{canonicalized_name}' either." ) # regular console_scripts entry points for name in pkg_resources.get_entry_map(dist, 'console_scripts'): entry_point = dist.get_entry_info('console_scripts', name) entry_points.append( (name, entry_point.module_name, '.'.join(entry_point.attrs)) ) # The metadata on "old-style" distutils scripts is not retained by # distutils/setuptools, except by placing the original scripts in # /EGG-INFO/scripts/. if dist.metadata_isdir('scripts'): # egg-info metadata from installed egg. for name in dist.metadata_listdir('scripts'): if dist.metadata_isdir('scripts/' + name): # Probably Python 3 __pycache__ directory. continue if name.lower().endswith('.exe'): # windows: scripts are implemented with 2 files # the .exe gets also into metadata_listdir # get_metadata chokes on the binary continue contents = dist.get_metadata('scripts/' + name) distutils_scripts.append((name, contents)) elif dist.key in _develop_distutils_scripts: # Development eggs don't have metadata about scripts, so we # collected it ourselves in develop()/ and # _detect_distutils_scripts(). for name, contents in _develop_distutils_scripts[dist.key]: distutils_scripts.append((name, contents)) else: entry_points.append(req) entry_points_names = [] for name, module_name, attrs in entry_points: entry_points_names.append(name) if scripts is not None: sname = scripts.get(name) if sname is None: continue else: sname = name sname = os.path.join(dest, sname) spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths) generated.extend( _script(module_name, attrs, spath, sname, arguments, initialization, rpsetup) ) # warn when a script name passed in 'scripts' argument # is not defined in an entry point. if scripts is not None: for name, target in scripts.items(): if name not in entry_points_names: if name == target: logger.warning("Could not generate script '%s' as it is not " "defined in the egg entry points.", name) else: logger.warning("Could not generate script '%s' as script " "'%s' is not defined in the egg entry points.", name, target) for name, contents in distutils_scripts: if scripts is not None: sname = scripts.get(name) if sname is None: continue else: sname = name sname = os.path.join(dest, sname) spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths) generated.extend( _distutils_script(spath, sname, contents, initialization, rpsetup) ) if interpreter: sname = os.path.join(dest, interpreter) spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths) generated.extend(_pyscript(spath, sname, rpsetup, initialization)) return generated def _relative_path_and_setup(sname, path, relative_paths): if relative_paths: relative_paths = os.path.normcase(relative_paths) sname = os.path.normcase(os.path.abspath(sname)) spath = ',\n '.join( [_relativitize(os.path.normcase(path_item), sname, relative_paths) for path_item in path] ) rpsetup = relative_paths_setup for i in range(_relative_depth(relative_paths, sname)): rpsetup += "base = os.path.dirname(base)\n" else: spath = repr(path)[1:-1].replace(', ', ',\n ') rpsetup = '' return spath, rpsetup def _relative_depth(common, path): n = 0 while 1: dirname = os.path.dirname(path) if dirname == path: raise AssertionError("dirname of %s is the same" % dirname) if dirname == common: break n += 1 path = dirname return n def _relative_path(common, path): r = [] while 1: dirname, basename = os.path.split(path) r.append(basename) if dirname == common: break if dirname == path: raise AssertionError("dirname of %s is the same" % dirname) path = dirname r.reverse() return os.path.join(*r) def _relativitize(path, script, relative_paths): if path == script: raise AssertionError("path == script") if path == relative_paths: return "base" common = os.path.dirname(os.path.commonprefix([path, script])) if (common == relative_paths or common.startswith(os.path.join(relative_paths, '')) ): return "join(base, %r)" % _relative_path(common, path) else: return repr(path) relative_paths_setup = """ import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) """ def _script(module_name, attrs, path, dest, arguments, initialization, rsetup): if is_win32: dest += '-script.py' python = _safe_arg(sys.executable) contents = script_template % dict( python = python, path = path, module_name = module_name, attrs = attrs, arguments = arguments, initialization = initialization, relative_paths_setup = rsetup, ) return _create_script(contents, dest) def _distutils_script(path, dest, script_content, initialization, rsetup): if is_win32: dest += '-script.py' lines = script_content.splitlines(True) if not ('#!' in lines[0]) and ('python' in lines[0]): # The script doesn't follow distutil's rules. Ignore it. return [] lines = lines[1:] # Strip off the first hashbang line. line_with_first_import = len(lines) for line_number, line in enumerate(lines): if not 'import' in line: continue if not (line.startswith('import') or line.startswith('from')): continue if '__future__' in line: continue line_with_first_import = line_number break before = ''.join(lines[:line_with_first_import]) after = ''.join(lines[line_with_first_import:]) python = _safe_arg(sys.executable) contents = distutils_script_template % dict( python = python, path = path, initialization = initialization, relative_paths_setup = rsetup, before = before, after = after ) return _create_script(contents, dest) def _file_changed(filename, old_contents, mode='r'): try: with open(filename, mode) as f: return f.read() != old_contents except EnvironmentError as e: if e.errno == errno.ENOENT: return True else: raise def _create_script(contents, dest): generated = [] script = dest changed = _file_changed(dest, contents) if is_win32: # generate exe file and give the script a magic name: win32_exe = os.path.splitext(dest)[0] # remove ".py" if win32_exe.endswith('-script'): win32_exe = win32_exe[:-7] # remove "-script" win32_exe = win32_exe + '.exe' # add ".exe" new_data = get_win_launcher('cli') if _file_changed(win32_exe, new_data, 'rb'): # Only write it if it's different. with open(win32_exe, 'wb') as f: f.write(new_data) generated.append(win32_exe) if changed: with open(dest, 'w') as f: f.write(contents) logger.info( "Generated script %r.", # Normalize for windows script.endswith('-script.py') and script[:-10] or script) try: os.chmod(dest, _execute_permission()) except (AttributeError, os.error): pass generated.append(dest) return generated if is_jython and jython_os_name == 'linux': script_header = '#!/usr/bin/env %(python)s' else: script_header = '#!%(python)s' script_template = script_header + '''\ %(relative_paths_setup)s import sys sys.path[0:0] = [ %(path)s, ] %(initialization)s import %(module_name)s if __name__ == '__main__': sys.exit(%(module_name)s.%(attrs)s(%(arguments)s)) ''' distutils_script_template = script_header + ''' %(before)s %(relative_paths_setup)s import sys sys.path[0:0] = [ %(path)s, ] %(initialization)s %(after)s''' def _pyscript(path, dest, rsetup, initialization=''): generated = [] script = dest if is_win32: dest += '-script.py' python = _safe_arg(sys.executable) if path: path += ',' # Courtesy comma at the end of the list. contents = py_script_template % dict( python = python, path = path, relative_paths_setup = rsetup, initialization=initialization, ) changed = _file_changed(dest, contents) if is_win32: # generate exe file and give the script a magic name: exe = script + '.exe' with open(exe, 'wb') as f: f.write( pkg_resources.resource_string('setuptools', 'cli.exe') ) generated.append(exe) if changed: with open(dest, 'w') as f: f.write(contents) try: os.chmod(dest, _execute_permission()) except (AttributeError, os.error): pass logger.info("Generated interpreter %r.", script) generated.append(dest) return generated py_script_template = script_header + '''\ %(relative_paths_setup)s import sys sys.path[0:0] = [ %(path)s ] %(initialization)s _interactive = True if len(sys.argv) > 1: # The Python interpreter wrapper allows only some of the options that a # "regular" Python interpreter accepts. _options, _args = __import__("getopt").getopt(sys.argv[1:], 'Iic:m:') _interactive = False for (_opt, _val) in _options: if _opt == '-i': _interactive = True elif _opt == '-c': exec(_val) elif _opt == '-m': sys.argv[1:] = _args _args = [] __import__("runpy").run_module( _val, {}, "__main__", alter_sys=True) elif _opt == '-I': # Allow yet silently ignore the `-I` option. The original behaviour # for this option is to create an isolated Python runtime. It was # deemed acceptable to allow the option here as this Python wrapper # is isolated from the system Python already anyway. # The specific use-case that led to this change is how the Python # language extension for Visual Studio Code calls the Python # interpreter when initializing the extension. pass if _args: sys.argv[:] = _args __file__ = _args[0] del _options, _args with open(__file__) as __file__f: exec(compile(__file__f.read(), __file__, "exec")) if _interactive: del _interactive __import__("code").interact(banner="", local=globals()) ''' runsetup_template = """ import sys sys.path.insert(0, %%(setupdir)r) sys.path[0:0] = %r import os, setuptools %%(extra)s __file__ = %%(__file__)r os.chdir(%%(setupdir)r) sys.argv[0] = %%(setup)r with open(%%(setup)r) as f: exec(compile(f.read(), %%(setup)r, 'exec')) """ % setuptools_path disable_root_logger = """ import logging root_logger = logging.getLogger() handler = logging.NullHandler() root_logger.addHandler(handler) """ class VersionConflict(zc.buildout.UserError): def __init__(self, err, ws): ws = list(ws) ws.sort() self.err, self.ws = err, ws def __str__(self): result = ["There is a version conflict."] if len(self.err.args) == 2: existing_dist, req = self.err.args result.append("We already have: %s" % existing_dist) for dist in self.ws: if req in dist.requires(): result.append("but %s requires %r." % (dist, str(req))) else: # The error argument is already a nice error string. result.append(self.err.args[0]) return '\n'.join(result) class MissingDistribution(zc.buildout.UserError): def __init__(self, req, ws): ws = list(ws) ws.sort() self.data = req, ws def __str__(self): req, ws = self.data return "Couldn't find a distribution for %r." % str(req) def call_pip_install(spec, dest, editable=False): """ Call `pip install` from a subprocess to install a distribution specified by `spec` into `dest`. For normal (non-editable) installs, returns all the paths inside `dest` created by the above. For editable installs, it returns the package name. These very different return values may seem strange, but it is because what needs to happen afterwards is very different for the two cases. """ args = [sys.executable, '-m', 'pip', 'install', '--no-deps', '-t', dest] package_index_url = index_url() if package_index_url: if not urllib.parse.urlsplit(package_index_url).scheme: # pip 25+ does not accept a directory as index, which buildout # does support. package_index_url = Path(package_index_url) if package_index_url.exists(): package_index_url = package_index_url.expanduser().resolve().as_uri() else: package_index_url = None if package_index_url: # We could pass the index in the '--index-url' parameter. # But then some of our tests start failing when we pass an index # with only a few zc.buildout distributions. Reason is that pip # will try to find setuptools there as well, as this is needed as # build-system for most packages. And this fails. # So we pass the index as *extra* url. args.extend(["--extra-index-url", package_index_url]) level = logger.getEffectiveLevel() if level >= logging.INFO: args.append('-q') else: args.append('-v') if editable: args.append('-e') args.append(spec) try: from pip._internal.cli.cmdoptions import no_python_version_warning HAS_WARNING_OPTION = True except ImportError: HAS_WARNING_OPTION = False if HAS_WARNING_OPTION: if not hasattr(call_pip_install, 'displayed'): call_pip_install.displayed = True else: args.append('--no-python-version-warning') env = os.environ.copy() python_path = pip_path[:] python_path.append(env.get('PYTHONPATH', '')) env['PYTHONPATH'] = os.pathsep.join(python_path) if level <= logging.DEBUG: logger.debug('Running pip install:\n"%s"\npath=%s\n', '" "'.join(args), pip_path) sys.stdout.flush() # We want any pending output first # This will quit the buildout process if there is an error. output = get_subprocess_output(list(args), env=env) if level <= logging.DEBUG: if output: logger.debug(output) logger.debug("Pip install completed successfully.") logger.debug("Contents of %s:", dest) for entry in os.listdir(dest): logger.debug("- %s", entry) split_entries = [os.path.splitext(entry) for entry in os.listdir(dest)] if editable: # On setuptools 79 and earlier, the egg-link file is created. package_name = "" for base, ext in split_entries: if ext == ".egg-link": logger.debug( "Found .egg-link file after successful pip install of %s", spec, ) package_name = base break if package_name: # For pkg_resources style namespaces a .pth file is created, # for example `plone.app.something-nspkg.pth`. We only need this # if the name was found. If name was not found, the namespaces # will be checked in a different way further on. for base, ext in split_entries: if ext != ".pth" or not base.endswith("-nspkg"): continue logger.debug( "Found -nspkg.pth file after successful pip install of %s", spec, ) logger.warning( "WARNING: Package %s at %s is using old style namespace packages. " "You should switch to native namespaces (PEP 420).", package_name, spec, ) # We don't want to analyze this file, so we can only make an # educated guess about the namespaces. # Assume at most two dots, that is enough info for our warning. # `a.b` -> `a` # `a.b.c` -> `a\na.b` # `a.b.c.d` -> `a\na.b` # Get all names except the last one, and then keep the first two: names = package_name.split(".")[:-1][:2] if len(names) > 1: names[1] = ".".join(names) namespaces = "\n".join(names) Installer._namespace_packages[package_name] = namespaces break return package_name # With normal (non-editable) installs, there won't be an egg-link file created. # The same is true for editable installs with setuptools 80+. # In both cases, we need to look for the .dist-info directory. try: distinfo_dir = [ base + ext for base, ext in split_entries if ext == ".dist-info" ][0] except IndexError: logger.error( "No .dist-info directory after successful pip install of %s", spec) raise full_distinfo_dir = os.path.join(dest, distinfo_dir) distrib = metadata.Distribution.at(full_distinfo_dir) # On Python 3.10 we could use `distrib.name`. name = distrib.metadata['Name'] if not name: logger.error( "Could not find package name in metadata after installing %s.", spec, ) sys.exit(1) if editable: namespaces_file = os.path.join(full_distinfo_dir, "namespace_packages.txt") if os.path.exists(namespaces_file): logger.warning( "WARNING: Package %s at %s is using old style namespace packages. " "You should switch to native namespaces (PEP 420).", name, spec, ) with open(namespaces_file) as myfile: Installer._namespace_packages[name] = myfile.read() if not editable: # TODO: we should no longer make an egg, but we still need some of this. return make_egg_after_pip_install(dest, distinfo_dir) return name def check_namespace_init_file(ns_file): """Look for namespace declaration in file. This can be spelled in different ways. It can be a one-liner: __import__('pkg_resources').declare_namespace(__name__) Or it can be a multi-line declaration, including comments, like this: # See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages try: __import__("pkg_resources").declare_namespace(__name__) except ImportError: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) """ logger.debug("Checking namespace __init__.py file: %s", ns_file) with open(ns_file, 'r') as myfile: contents = [line for line in myfile.readlines() if line.strip() and not line.strip().startswith('#')] if len(contents) == 0: logger.debug("Found too few lines to be a namespace declaration.") return False if len(contents) > 6: logger.debug("Found too many lines to be a namespace declaration.") return False for combo in [ ("pkg_resources", "declare_namespace"), ("pkgutil", "extend_path"), ]: found_first = False for line in contents: if combo[0] in line: found_first = True if found_first and combo[1] in line: logger.debug("Found namespace declaration in %s", ns_file) return True logger.debug("No namespace declaration found in %s", ns_file) return False def find_namespace_init_files(directory): logger.debug("Searching for namespace __init__.py files in %s", directory) found_files = [] for root, dirs, files in os.walk(directory): if root.endswith('__pycache__'): continue if len(files) != 1 or '__init__.py' not in files: continue ns_file = os.path.join(root, '__init__.py') logger.debug('Found possible namespace __init__.py file: %s', ns_file) if check_namespace_init_file(ns_file): found_files.append(ns_file) if found_files: logger.debug("Found namespace __init__.py files: %s", found_files) else: logger.debug("No namespace __init__.py files found.") return found_files def make_egg_after_pip_install(dest, distinfo_dir): """build properly named egg directory""" logger.debug('Making egg in %s from pip installation in %s', dest, distinfo_dir) # `pip install` does not build the namespace aware __init__.py files. # In the new situation we are happy with that. But actually, for some # packages, pip *does* include these files, so we need to remove them. # For example try `pip install zope.interface==4.1.3`. You get a warning: # "DEPRECATION: zope.interface is being installed using the legacy # 'setup.py install' method, because it does not have a 'pyproject.toml'"" # Apparently this has the side effect that the namespace files get installed. # zope.interface 7.0.3 does not have this problem. for ns_file in find_namespace_init_files(dest): os.remove(ns_file) logger.debug("Removed namespace __init__.py file: %s", ns_file) # Remove `bin` directory if needed # as there is no way to avoid script installation # when running `pip install` entry_points_file = os.path.join(dest, distinfo_dir, 'entry_points.txt') if os.path.isfile(entry_points_file): with open(entry_points_file, encoding='utf-8', errors="replace") as f: content = f.read() if "console_scripts" in content or "gui_scripts" in content: bin_dir = os.path.join(dest, BIN_SCRIPTS) if os.path.exists(bin_dir): shutil.rmtree(bin_dir) # Get actual project name from dist-info directory. metadata_path = posixpath.join(dest, distinfo_dir, "METADATA") # The encoding and errors arguments can be needed on Windows. # See https://github.com/buildout/buildout/issues/722 with open(metadata_path, encoding='utf-8', errors="replace") as fp: value = fp.read() metadata = email.parser.Parser().parsestr(value) project_name = metadata.get("Name") # Make properly named new egg dir distro = list(pkg_resources.find_distributions(dest))[0] if project_name: distro.project_name = project_name base = "{}-{}".format( distro.egg_name(), pkg_resources.get_supported_platform() ) egg_name = base + '.egg' new_distinfo_dir = base + '.dist-info' egg_dir = os.path.join(dest, egg_name) os.mkdir(egg_dir) # Move ".dist-info" dir into new egg dir os.rename( os.path.join(dest, distinfo_dir), os.path.join(egg_dir, new_distinfo_dir) ) top_level_file = os.path.join(egg_dir, new_distinfo_dir, 'top_level.txt') if os.path.isfile(top_level_file): with open(top_level_file, encoding='utf-8', errors="replace") as f: top_levels = filter( (lambda x: len(x) != 0), [line.strip() for line in f.readlines()] ) else: top_levels = () # Move all top_level modules or packages for top_level in top_levels: # as package top_level_dir = os.path.join(dest, top_level) if os.path.exists(top_level_dir): shutil.move(top_level_dir, egg_dir) continue # as module top_level_py = top_level_dir + '.py' if os.path.exists(top_level_py): shutil.move(top_level_py, egg_dir) top_level_pyc = top_level_dir + '.pyc' if os.path.exists(top_level_pyc): shutil.move(top_level_pyc, egg_dir) continue record_file = os.path.join(egg_dir, new_distinfo_dir, 'RECORD') if os.path.isfile(record_file): with open(record_file, newline='', encoding='utf-8', errors="replace") as f: all_files = [row[0] for row in csv.reader(f)] # There might be some c extensions left over for entry in all_files: if entry.endswith(('.pyc', '.pyo')): continue dest_entry = os.path.join(dest, entry) # work around pip install -t bug that leaves entries in RECORD # that starts with '../../' if not os.path.abspath(dest_entry).startswith(dest): continue egg_entry = os.path.join(egg_dir, entry) if os.path.exists(dest_entry) and not os.path.exists(egg_entry): egg_entry_dir = os.path.dirname(egg_entry) if not os.path.exists(egg_entry_dir): os.makedirs(egg_entry_dir) os.rename(dest_entry, egg_entry) return [egg_dir] def unpack_egg(location, dest): # Buildout 2 no longer installs zipped eggs, # so we always want to unpack it. # XXX The next line seems double now. # dest = os.path.join(dest, os.path.basename(location)) setuptools.archive_util.unpack_archive(location, dest) def unpack_wheel(location, dest): wheel = Wheel(location) # The egg_name method returns a string that includes: # platform = None if self.platform == 'any' else get_platform() # get_platform is imported from distutils.util, vendorized # by setuptools, but this is really just: sysconfig.get_platform() # This is the platform where Python got compiled. This may differ # from the current platform, and this trips up the logic in # pkg_resources.compatible_platforms. We have a patch for that. # See the docstring of the Environment class above. wheel.install_as_egg(os.path.join(dest, wheel.egg_name())) UNPACKERS = { '.egg': unpack_egg, # '.whl': setuptools.archive_util.unpack_zipfile, } def _get_matching_dist_in_location(dist, location): """ Check if `locations` contain only the one intended dist. Return the dist with metadata in the new location. """ # Getting the dist from the environment causes the distribution # meta data to be read. Cloning isn't good enough. We must compare # dist.parsed_version, not dist.version, because one or the other # may be normalized (e.g., 3.3 becomes 3.3.0 when downloaded from # PyPI.) env = Environment([location]) dists = [ d for project_name in env for d in env[project_name] ] dist_infos = [ (normalize_name(d.project_name), d.parsed_version) for d in dists ] if dist_infos == [(normalize_name(dist.project_name), dist.parsed_version)]: return dists.pop() class BuildoutWheel(Wheel): """Extension for Wheel class to get the actual project name.""" def get_project_name(self): """Get project name by looking in the .dist-info of the wheel. This is adapted from the Wheel.install_as_egg method and the methods it calls. Ideally, this would be the same as self.project_name. """ with zipfile.ZipFile(self.filename) as zf: dist_info = self.get_dist_info(zf) with zf.open(posixpath.join(dist_info, 'METADATA')) as fp: value = fp.read().decode('utf-8') metadata = email.parser.Parser().parsestr(value) return metadata.get("Name") def _maybe_copy_and_rename_wheel(dist, dest): """Maybe copy and rename wheel. Return the new dist or None. So why do we do this? We need to check a special case: - zest_releaser-9.4.0-py3-none-any.whl with an underscore results in: zest_releaser-9.4.0-py3.13.egg In the resulting `bin/fullrease` script the zest.releaser distribution is not found. - So in this function we copy and rename the wheel to: zest.releaser-9.4.0-py3-none-any.whl with a dot, which results in: zest.releaser-9.4.0-py3.13.egg The resulting `bin/fullrease` script works fine. See https://github.com/buildout/buildout/issues/686 So check if we should rename the wheel before handling it. At first, source dists seemed to not have this problem. Or not anymore, after some fixes in Buildout last year: - zest_releaser-9.4.0.tar.gz with an underscore results in (in my case): zest_releaser-9.4.0-py3.13-macosx-14.7-x86_64.egg And this works fine, despite having an underscore. - But: products_cmfplone-6.1.1.tar.gz with an underscore leads to products_cmfplone-6.1.1-py3.13-macosx-14.7-x86_64.egg and with this, a Plone instance totally fails to start. Ah, but this is only because the generated zope.conf contains a temporarystorage option which is added because plone.recipe.zope2instance could not determine the Products.CMFPlone version. If I work around that, the instance actually starts. The zest.releaser egg generated from the source dist has a dist-info directory: zest_releaser-9.4.0-py3.13-macosx-14.7-x86_64.dist-info The egg generated from any of the two wheels only has an EGG-INFO directory. I guess the dist-info directory somehow helps. It is there because our make_egg_after_pip_install function, which only gets called after installing a source dist, has its own home grown way of creating an egg. """ wheel = BuildoutWheel(dist.location) actual_project_name = wheel.get_project_name() if actual_project_name and wheel.project_name == actual_project_name: return filename = os.path.basename(dist.location) new_filename = filename.replace(wheel.project_name, actual_project_name) if filename == new_filename: return logger.debug("Renaming wheel %s to %s", dist.location, new_filename) tmp_wheeldir = tempfile.mkdtemp() try: new_location = os.path.join(tmp_wheeldir, new_filename) shutil.copy(dist.location, new_location) # Now we create a clone of the original distribution, # but with the new location and the wanted project name. new_dist = Distribution( new_location, project_name=actual_project_name, version=dist.version, py_version=dist.py_version, platform=dist.platform, precedence=dist.precedence, ) # We were called by _move_to_eggs_dir_and_compile. # Now we call it again with the new dist. # I tried simply returning new_dist, but then it immediately # got removed because we remove its temporary directory. return _move_to_eggs_dir_and_compile(new_dist, dest) finally: # Remember that temporary directories must be removed zc.buildout.rmtree.rmtree(tmp_wheeldir) def _move_to_eggs_dir_and_compile(dist, dest): """Move distribution to the eggs destination directory. Originally we compiled the py files if we actually moved the dist. But this was never updated for Python 3, so it had no effect. So we removed this part. See https://github.com/buildout/buildout/issues/699 Its new location is expected not to exist there yet, otherwise we would not be calling this function: the egg is already there. But the new location might exist at this point if another buildout is running in parallel. So we copy to a temporary directory first. See discussion at https://github.com/buildout/buildout/issues/307 We return the new distribution with properly loaded metadata. """ # First make sure the destination directory exists. This could suffer from # the same kind of race condition as the rest: if we check that it does not # exist, and we then create it, it will fail when a second buildout is # doing the same thing. try: os.makedirs(dest) except OSError: if not os.path.isdir(dest): # Unknown reason. Reraise original error. raise logger.debug( "Turning dist %s (%s) into egg, and moving to eggs dir (%s).", dist, dist.location, dest, ) if dist.location.endswith('site-packages'): return dist tmp_dest = tempfile.mkdtemp(dir=dest) try: if (os.path.isdir(dist.location) and dist.precedence >= pkg_resources.BINARY_DIST): # We got a pre-built directory. It must have been obtained locally. # Just copy it. logger.debug("dist is pre-built directory.") # TODO Can we still support this? Do we need to? Maybe warn, or let pip install this. tmp_loc = os.path.join(tmp_dest, os.path.basename(dist.location)) shutil.copytree(dist.location, tmp_loc) else: # It is an archive of some sort. # Figure out how to unpack it, or fall back to easy_install. basename, ext = os.path.splitext(dist.location) if ext == ".gz" and basename.endswith(".tar"): basename = basename[:-4] # Set new location with name ending in '.experimental'. # XXX TODO some our code or tests expects '.egg' at the end. # tmp_loc = os.path.join(tmp_dest, os.path.basename(basename) + ".experimental") tmp_loc = os.path.join(tmp_dest, os.path.basename(basename) + ".egg") if ext in UNPACKERS: # TODO Maybe simply always call pip install for all dists, without # checking for unpackers. # TODO Maybe never rename a wheel or other dist anymore. # if ext == '.whl': # logger.debug("Checking if wheel needs to be renamed.") # new_dist = _maybe_copy_and_rename_wheel(dist, dest) # if new_dist is not None: # logger.debug("Found dist after renaming wheel: %s", new_dist) # return new_dist # logger.debug("Renaming wheel was not needed or did not help.") unpacker = UNPACKERS[ext] logger.debug("Calling unpacker for %s on %s", ext, dist.location) unpacker(dist.location, tmp_loc) else: logger.debug("Calling pip install for %s on %s", ext, dist.location) [tmp_loc] = call_pip_install(dist.location, tmp_dest) # We have installed the dist. Now try to rename/move it. logger.debug("Egg for %s installed at %s", dist, tmp_loc) newloc = os.path.join(dest, os.path.basename(tmp_loc)) try: os.rename(tmp_loc, newloc) except OSError: logger.error( "Moving/renaming egg for %s (%s) to %s failed.", dist, dist.location, newloc, ) # Might be for various reasons. If it is because newloc already # exists, we can investigate. if not os.path.exists(newloc): # No, it is a different reason. Give up. logger.error("New location %s does not exist.", newloc) raise # Try to use it as environment and check if our project is in it. newdist = _get_matching_dist_in_location(dist, newloc) if newdist is None: # Path exists, but is not our package. We could # try something, but it seems safer to bail out # with the original error. logger.error( "New location %s exists, but has no distribution for %s", newloc, dist) raise # newloc looks okay to use. # This may happen more often on Mac, and is the reason why we # override Environment.can_add, see above. # Do print a warning. logger.warning( "Path %s unexpectedly already exists.\n" "It contains the expected distribution for %s.\n" "Maybe a buildout running in parallel has added it. " "We will accept it.\n" "If this contains a wrong package, please remove it yourself.", newloc, dist) else: # There were no problems during the rename. newdist = _get_matching_dist_in_location(dist, newloc) if newdist is None: raise AssertionError(f"{newloc} has no distribution for {dist}") # The new dist automatically has precedence DEVELOP_DIST, which sounds # wrong. And this interferes with a check for printing picked versions. # So set it to EGG_DIST. We already did this for a long time, then I # removed it because I thought it would no longer be needed, but it is. # Also, we used to do this only when the dist was installed by pip, # but it seems needed always, otherwise dists installed from eggs won't # be reported in picked versions either. It could be that we report # too much then, but we will see. newdist.precedence = pkg_resources.EGG_DIST finally: # Remember that temporary directories must be removed zc.buildout.rmtree.rmtree(tmp_dest) return newdist def sort_working_set(ws, eggs_dir, develop_eggs_dir): develop_paths = set() pattern = os.path.join(develop_eggs_dir, '*.egg-link') for egg_link in glob.glob(pattern): with open(egg_link, 'rt') as f: path = f.readline().strip() if path: develop_paths.add(path) sorted_paths = [] egg_paths = [] other_paths = [] for dist in ws: path = dist.location if path in develop_paths: sorted_paths.append(path) elif os.path.commonprefix([path, eggs_dir]) == eggs_dir: egg_paths.append(path) else: other_paths.append(path) sorted_paths.extend(egg_paths) sorted_paths.extend(other_paths) return pkg_resources.WorkingSet(sorted_paths) NOT_PICKED_AND_NOT_ALLOWED = """\ Picked: {name} = {version} The `{name}` egg does not have a version pin and `allow-picked-versions = false`. To resolve this, add {name} = {version} to the [versions] section, OR set `allow-picked-versions = true`.""" ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4007437 zc_buildout-6.0.1/src/zc/buildout/patches.py0000644000076500000240000001600115264141650020505 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## def patch_PackageIndex(): """Patch the package index from setuptools. Main goal: check the package urls on an index page to see if they are compatible with the Python version. """ try: import logging logging.getLogger('pip._internal.index.collector').setLevel(logging.ERROR) from ._package_index import PackageIndex from ._package_index import URL_SCHEME from ._package_index import distros_for_url try: # pip 22.2+ from pip._internal.index.collector import IndexContent except ImportError: # pip 22.1- from pip._internal.index.collector import HTMLPage as IndexContent from pip._internal.index.collector import parse_links from pip._internal.index.package_finder import _check_link_requires_python from pip._internal.models.target_python import TargetPython from urllib.error import HTTPError except ImportError: import logging logger = logging.getLogger('zc.buildout.patches') logger.warning( 'Requires-Python support missing and could not be patched into ' 'zc.buildout. \n\n', exc_info=True ) return PY_VERSION_INFO = TargetPython().py_version_info # method copied over from setuptools 46.1.3 # Unchanged in setuptools 70.0.0. def process_url(self, url, retrieve=False): """Evaluate a URL as a possible download, and maybe retrieve it""" if url in self.scanned_urls and not retrieve: return self.scanned_urls[url] = True if not URL_SCHEME(url): self.process_filename(url) return else: dists = list(distros_for_url(url)) if dists: if not self.url_ok(url): return self.debug("Found link: %s", url) if dists or not retrieve or url in self.fetched_urls: list(map(self.add, dists)) return # don't need the actual page if not self.url_ok(url): self.fetched_urls[url] = True return self.info("Reading %s", url) self.fetched_urls[url] = True # prevent multiple fetch attempts tmpl = "Download error on %s: %%s -- Some packages may not be found!" f = self.open_url(url, tmpl % url) if f is None: return # --- LOCAL CHANGES MADE HERE: --- if isinstance(f, HTTPError): if f.code == 401: self.info("Authentication error: %s" % f.msg) else: self.info("HTTP error: %s" % f.msg) # --- END OF LOCAL CHANGES --- self.fetched_urls[f.url] = True if 'html' not in f.headers.get('content-type', '').lower(): f.close() # not html, we can't process it return base = f.url # handle redirects page = f.read() # --- LOCAL CHANGES MADE HERE: --- if isinstance(page, str): page = page.encode('utf8') charset = 'utf8' else: if isinstance(f, HTTPError): # Errors have no charset, assume latin1: charset = 'latin-1' else: charset = f.headers.get_param('charset') or 'latin-1' try: content_type = f.getheader('content-type') except AttributeError: # On at least Python 2.7: # addinfourl instance has no attribute 'getheader' content_type = "text/html" try: # pip 22.2+ html_page = IndexContent( page, content_type=content_type, encoding=charset, url=base, cache_link_parsing=False, ) except TypeError: try: # pip 20.1-22.1 html_page = IndexContent(page, charset, base, cache_link_parsing=False) except TypeError: # pip 20.0 or older html_page = IndexContent(page, charset, base) # https://github.com/buildout/buildout/issues/598 # use_deprecated_html5lib is a required addition in pip 22.0/22.1 # and it is gone already in 22.2 try: plinks = parse_links(html_page, use_deprecated_html5lib=False) except TypeError: plinks = parse_links(html_page) plinks = list(plinks) # --- END OF LOCAL CHANGES --- if not isinstance(page, str): # In Python 3 and got bytes but want str. page = page.decode(charset, "ignore") f.close() # --- LOCAL CHANGES MADE HERE: --- for link in plinks: if _check_link_requires_python(link, PY_VERSION_INFO): self.process_url(link.url) # --- END OF LOCAL CHANGES --- if url.startswith(self.index_url) and getattr(f, 'code', None) != 404: page = self.process_index(url, page) setattr(PackageIndex, 'process_url', process_url) patch_PackageIndex() def patch_pkg_resources_requirement_contains(): """Patch pkg_resources.Requirement contains method. What this hopefully solves, is checking if a Requirement contains a Distribution, without the key needing to be exactly the same. We want to compare normalized names. """ try: from pkg_resources import Distribution from pkg_resources import Requirement from zc.buildout.utils import normalize_name except ImportError: return def __contains__(self, item): if isinstance(item, Distribution): # if item.key != self.key: if normalize_name(item.key) != normalize_name(self.key): return False item = item.version # Allow prereleases always in order to match the previous behavior of # this method. In the future this should be smarter and follow PEP 440 # more accurately. try: return self.specifier.contains(item, prereleases=True) except Exception: # For example on https://pypi.org/simple/zope-exceptions/ # the first distribution is zope.exceptions-3.4dev-r73107.tar.gz # I want to catch version.InvalidVersion, but it may # come from a different place then I think. return False Requirement.__contains__ = __contains__ patch_pkg_resources_requirement_contains() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4008217 zc_buildout-6.0.1/src/zc/buildout/pep425tags.py0000644000076500000240000001162415264141650020762 0ustar00mauritsstaff"""Generate and work with PEP 425 Compatibility Tags.""" import sys import warnings import sysconfig import distutils.util def get_config_var(var): try: return sysconfig.get_config_var(var) except IOError as e: # pip Issue #1074 warnings.warn("{0}".format(e), RuntimeWarning) return None def get_abbr_impl(): """Return abbreviated implementation name.""" if hasattr(sys, 'pypy_version_info'): pyimpl = 'pp' elif sys.platform.startswith('java'): pyimpl = 'jy' elif sys.platform == 'cli': pyimpl = 'ip' else: pyimpl = 'cp' return pyimpl def get_impl_ver(): """Return implementation version.""" impl_ver = get_config_var("py_version_nodot") if not impl_ver or get_abbr_impl() == 'pp': impl_ver = ''.join(map(str, get_impl_version_info())) return impl_ver def get_impl_version_info(): """Return sys.version_info-like tuple for use in decrementing the minor version.""" if get_abbr_impl() == 'pp': # as per https://github.com/pypa/pip/issues/2882 return (sys.version_info[0], sys.pypy_version_info.major, sys.pypy_version_info.minor) else: return sys.version_info[0], sys.version_info[1] def get_flag(var, fallback, expected=True, warn=True): """Use a fallback method for determining SOABI flags if the needed config var is unset or unavailable.""" val = get_config_var(var) if val is None: if warn: warnings.warn("Config variable '{0}' is unset, Python ABI tag may " "be incorrect".format(var), RuntimeWarning, 2) return fallback() return val == expected def get_abi_tag(): """Return the ABI tag based on SOABI (if available) or emulate SOABI (CPython 2, PyPy).""" soabi = get_config_var('SOABI') impl = get_abbr_impl() if not soabi and impl in ('cp', 'pp') and hasattr(sys, 'maxunicode'): d = '' m = '' if get_flag('Py_DEBUG', lambda: hasattr(sys, 'gettotalrefcount'), warn=(impl == 'cp')): d = 'd' if get_flag('WITH_PYMALLOC', lambda: impl == 'cp', warn=(impl == 'cp')): m = 'm' abi = '%s%s%s%s' % (impl, get_impl_ver(), d, m) elif soabi and soabi.startswith('cpython-'): abi = 'cp' + soabi.split('-')[1] elif soabi: abi = soabi.replace('.', '_').replace('-', '_') else: abi = None return abi def get_platform(): """Return our platform name 'win32', 'linux_x86_64'""" # XXX remove distutils dependency result = distutils.util.get_platform().replace('.', '_').replace('-', '_') if result == "linux_x86_64" and sys.maxsize == 2147483647: # pip pull request #3497 result = "linux_i686" return result def get_supported(versions=None, supplied_platform=None): """Return a list of supported tags for each version specified in `versions`. :param versions: a list of string versions, of the form ["33", "32"], or None. The first version will be assumed to support our ABI. """ supported = [] # Versions must be given with respect to the preference if versions is None: versions = [] version_info = get_impl_version_info() major = version_info[:-1] # Support all previous minor Python versions. for minor in range(version_info[-1], -1, -1): versions.append(''.join(map(str, major + (minor,)))) impl = get_abbr_impl() abis = [] abi = get_abi_tag() if abi: abis[0:0] = [abi] abi3s = set() import imp for suffix in imp.get_suffixes(): if suffix[0].startswith('.abi'): abi3s.add(suffix[0].split('.', 2)[1]) abis.extend(sorted(list(abi3s))) abis.append('none') platforms = [] if supplied_platform: platforms.append(supplied_platform) platforms.append(get_platform()) # Current version, current API (built specifically for our Python): for abi in abis: for arch in platforms: supported.append(('%s%s' % (impl, versions[0]), abi, arch)) # No abi / arch, but requires our implementation: for i, version in enumerate(versions): supported.append(('%s%s' % (impl, version), 'none', 'any')) if i == 0: # Tagged specifically as being cross-version compatible # (with just the major version specified) supported.append(('%s%s' % (impl, versions[0][0]), 'none', 'any')) # Major Python version + platform; e.g. binaries not using the Python API supported.append(('py%s' % (versions[0][0]), 'none', arch)) # No abi / arch, generic Python for i, version in enumerate(versions): supported.append(('py%s' % (version,), 'none', 'any')) if i == 0: supported.append(('py%s' % (version[0]), 'none', 'any')) return supported ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4010336 zc_buildout-6.0.1/src/zc/buildout/rmtree.py0000644000076500000240000000377315264141650020370 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import shutil import os import doctest import time def rmtree (path): """ A variant of shutil.rmtree which tries hard to be successful. On windows shutil.rmtree aborts when it tries to delete a read only file or a file which is still handled by another process (e.g. antivirus scanner). This tries to chmod the file to writeable and retries 10 times before giving up. >>> from tempfile import mkdtemp Let's make a directory ... >>> d = mkdtemp() and make sure it is actually there >>> os.path.isdir (d) 1 Now create a file ... >>> foo = os.path.join (d, 'foo') >>> with open (foo, 'w') as f: _ = f.write ('huhu') and make it unwriteable >>> os.chmod (foo, 256) # 0400 rmtree should be able to remove it: >>> rmtree (d) and now the directory is gone >>> os.path.isdir (d) 0 """ def retry_writeable (func, path, exc): os.chmod (path, 384) # 0600 for i in range(10): try: func (path) break except OSError: time.sleep(0.1) else: # tried 10 times without success, thus # finally rethrow the last exception raise shutil.rmtree (path, onerror = retry_writeable) def test_suite(): return doctest.DocTestSuite() if "__main__" == __name__: doctest.testmod() ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1792066472.40116 zc_buildout-6.0.1/src/zc/buildout/testing.py0000644000076500000240000005714215264141650020546 0ustar00mauritsstaff############################################################################# # # Copyright (c) 2004-2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Various test-support utility functions """ from http.server import HTTPServer, BaseHTTPRequestHandler from urllib.request import urlopen import errno import logging import multiprocessing import operator import os import pkg_resources import random import re import shutil import socket import subprocess import sys import tempfile import threading import time import zc.buildout.buildout import zc.buildout.easy_install from zc.buildout.rmtree import rmtree print_ = zc.buildout.buildout.print_ fsync = getattr(os, 'fsync', lambda fileno: None) is_win32 = sys.platform == 'win32' def read(path='out', *rest): with open(os.path.join(path, *rest)) as f: return f.read() def cat(dir, *names): path = os.path.join(dir, *names) if (not os.path.exists(path) and is_win32 and os.path.exists(path+'-script.py') ): path = path+'-script.py' with open(path) as f: print_(f.read(), end='') def eqs(a, *b): a = set(a); b = set(b) return None if a == b else (a - b, b - a) def clear_here(): for name in os.listdir('.'): if os.path.isfile(name) or os.path.islink(name): os.remove(name) else: shutil.rmtree(name) def ls(dir, *subs, lowercase_and_sort_output=False): if subs: dir = os.path.join(dir, *subs) if lowercase_and_sort_output: # Get the original names, but sorted lowercase. names = sorted(os.listdir(dir), key=operator.methodcaller("lower")) else: names = sorted(os.listdir(dir)) for name in names: # PEP 660 filter: hide extra metadata files in develop-eggs to keep # legacy tests passing. if (name.endswith('.dist-info') or name.endswith('.egg-info') or name.endswith('.pth') or (name.startswith('__editable__') and name.endswith('.py')) or name == '__pycache__'): # Only hide if we're not specifically wanting to show PEP 660 files. if os.environ.get('BUILDOUT_TESTING_SHOW_PEP660') != '1': continue # If we're running under coverage, elide coverage files if os.getenv("COVERAGE_PROCESS_START") and name.startswith('.coverage.'): continue if os.path.isdir(os.path.join(dir, name)): print_('d ', end=' ') elif os.path.islink(os.path.join(dir, name)): print_('l ', end=' ') else: print_('- ', end=' ') if lowercase_and_sort_output: name = name.lower() print_(name) def mkdir(*path): os.mkdir(os.path.join(*path)) def remove(*path): path = os.path.join(*path) if os.path.isdir(path): shutil.rmtree(path) else: os.remove(path) def rmdir(*path): shutil.rmtree(os.path.join(*path)) def write(dir, *args): path = os.path.join(dir, *(args[:-1])) f = open(path, 'w') f.write(args[-1]) f.flush() fsync(f.fileno()) f.close() def clean_up_pyc(*path): base, filename = os.path.join(*path[:-1]), path[-1] if filename.endswith('.py'): filename += 'c' # .py -> .pyc for path in ( os.path.join(base, filename), os.path.join(base, '__pycache__'), ): if os.path.isdir(path): rmdir(path) elif os.path.exists(path): remove(path) ## FIXME - check for other platforms MUST_CLOSE_FDS = not sys.platform.startswith('win') def system(command, input='', with_exit_code=False, env=None): # Some TERMinals, especially xterm and its variants, add invisible control # characters, which we do not want as they mess up doctests. See: # https://github.com/buildout/buildout/pull/311 # http://bugs.python.org/issue19884 sub_env = dict(os.environ, TERM='dumb') if env is not None: sub_env.update(env) # We used to pass for example 'buildout annotate' as command, and call Popen # with 'shell=True'. Since October 2024 this no longer works on Windows on GHA. # So we pass the command as a list. But then it breaks on POSIX when we have # 'shell=True', because args[1:] gets passed as options for '/bin/sh' instead # of options for our command. So let's let the value of 'shell' depend on # whether command is a list or a string. # See also https://stackoverflow.com/a/2401128/621201 # Actually, having the command as a string turns out to only be a problem on # Windows if there is a space in the path, like with 'Program Files'. if isinstance(command, list): shell = False else: shell = True p = subprocess.Popen(command, shell=shell, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=MUST_CLOSE_FDS, env=sub_env) i, o, e = (p.stdin, p.stdout, p.stderr) if input: i.write(input.encode()) i.close() result = o.read() + e.read() o.close() e.close() output = result.decode() if with_exit_code: # Use the with_exit_code=True parameter when you want to test the exit # code of the command you're running. output += 'EXIT CODE: %s' % p.wait() p.wait() return output def get(url): return str(urlopen(url).read().decode()) def _runsetup(setup, *args): if os.path.isdir(setup): setup = os.path.join(setup, 'setup.py') args = list(args) args.insert(0, '-q') here = os.getcwd() try: os.chdir(os.path.dirname(setup)) zc.buildout.easy_install.call_subprocess( [sys.executable, setup] + args, env=dict(os.environ, PYTHONPATH=zc.buildout.easy_install.pip_pythonpath, ), # Prevent showing several lines of output per created distribution, # especially with older setuptools versions. stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, ) if os.path.exists('build'): rmtree('build') finally: os.chdir(here) def sdist(setup, dest): _runsetup(setup, 'sdist', '-d', dest) def bdist_egg(setup, executable, dest=None): # Backward compat: if dest is None: dest = executable else: assert executable == sys.executable, (executable, sys.executable) _runsetup(setup, 'bdist_egg', '-d', dest) def bdist_wheel(setup, dest): _runsetup(setup, 'bdist_wheel', '-d', dest) def wait_until(label, func, *args, **kw): if 'timeout' in kw: kw = dict(kw) timeout = kw.pop('timeout') else: timeout = 30 deadline = time.time()+timeout while time.time() < deadline: if func(*args, **kw): return time.sleep(0.01) raise ValueError('Timed out waiting for: '+label) class TestOptions(zc.buildout.buildout.Options): def __init__(self, *args): zc.buildout.buildout.Options.__init__(self, *args) self._created = [] def initialize(self): pass class Buildout(zc.buildout.buildout.Buildout): def __init__(self): for name in os.path.join('eggs', 'v5'), 'parts': if not os.path.exists(name): os.makedirs(name) zc.buildout.buildout.Buildout.__init__( self, '', [('buildout', 'directory', os.getcwd())], False) Options = TestOptions def buildoutSetUp(test): test.globs['__tear_downs'] = __tear_downs = [] test.globs['register_teardown'] = register_teardown = __tear_downs.append prefer_final = zc.buildout.easy_install.prefer_final() register_teardown( lambda: zc.buildout.easy_install.prefer_final(prefer_final) ) here = os.getcwd() register_teardown(lambda: os.chdir(here)) handlers_before_set_up = logging.getLogger().handlers[:] def restore_root_logger_handlers(): root_logger = logging.getLogger() for handler in root_logger.handlers[:]: root_logger.removeHandler(handler) for handler in handlers_before_set_up: root_logger.addHandler(handler) bo_logger = logging.getLogger('zc.buildout') for handler in bo_logger.handlers[:]: bo_logger.removeHandler(handler) register_teardown(restore_root_logger_handlers) base = tempfile.mkdtemp('buildoutSetUp') base = os.path.realpath(base) register_teardown(lambda base=base: rmtree(base)) old_home = os.environ.get('HOME') os.environ['HOME'] = os.path.join(base, 'bbbBadHome') def restore_home(): if old_home is None: del os.environ['HOME'] else: os.environ['HOME'] = old_home register_teardown(restore_home) base = os.path.join(base, '_TEST_') os.mkdir(base) tmp = tempfile.mkdtemp('buildouttests') register_teardown(lambda: rmtree(tmp)) zc.buildout.easy_install.default_index_url = 'file://'+tmp os.environ['buildout_testing_index_url'] = ( zc.buildout.easy_install.default_index_url) def tmpdir(name): path = os.path.join(base, name) mkdir(path) return path sample = tmpdir('sample-buildout') os.chdir(sample) # Create a basic buildout.cfg to avoid a warning from buildout: with open('buildout.cfg', 'w') as f: f.write("[buildout]\nparts =\n") # Use the buildout bootstrap command to create a buildout zc.buildout.buildout.Buildout( 'buildout.cfg', [('buildout', 'log-level', 'WARNING'), # trick bootstrap into putting the buildout develop egg # in the eggs dir. ('buildout', 'develop-eggs-directory', os.path.join('eggs', 'v5')), ] ).bootstrap([]) # Create the develop-eggs dir, which didn't get created the usual # way due to the trick above: os.mkdir('develop-eggs') path_to_coveragerc = os.getenv("COVERAGE_PROCESS_START", None) if path_to_coveragerc is not None: # Before we return to the current directory and destroy the # temporary working directory, we need to copy all the coverage files # back so that they can be `coverage combine`d. def copy_coverage_files(): coveragedir = os.path.dirname(path_to_coveragerc) import glob for f in glob.glob('.coverage*'): shutil.copy(f, coveragedir) __tear_downs.insert(0, copy_coverage_files) # Now we must modify the newly created bin/buildout to # actually begin coverage. with open('bin/buildout') as f: import textwrap lines = f.read().splitlines() assert lines[1] == '', lines lines[1] = 'import coverage; coverage.process_startup()' with open('bin/buildout', 'w') as f: f.write('\n'.join(lines)) def start_server(path): port, thread = _start_server(path, name=path) url = 'http://localhost:%s/' % port register_teardown(lambda: stop_server(url, thread)) return url cdpaths = [] def cd(*path): path = os.path.join(*path) cdpaths.append(os.path.abspath(os.getcwd())) os.chdir(path) def uncd(): os.chdir(cdpaths.pop()) test.globs.update(dict( sample_buildout = sample, ls = ls, cat = cat, mkdir = mkdir, rmdir = rmdir, remove = remove, tmpdir = tmpdir, write = write, system = system, get = get, cd = cd, uncd = uncd, join = os.path.join, sdist = sdist, bdist_egg = bdist_egg, start_server = start_server, stop_server = stop_server, buildout = os.path.join(sample, 'bin', 'buildout'), wait_until = wait_until, print_ = print_, clean_up_pyc = clean_up_pyc, os = os, )) zc.buildout.easy_install.prefer_final(prefer_final) def buildoutTearDown(test): for f in test.globs['__tear_downs']: f() class Server(HTTPServer): def __init__(self, tree, *args): HTTPServer.__init__(self, *args) self.tree = os.path.abspath(tree) __run = True def serve_forever(self): while self.__run: self.handle_request() def handle_error(self, *_): self.__run = False class Handler(BaseHTTPRequestHandler): Server.__log = False def __init__(self, request, address, server): self.__server = server self.tree = server.tree BaseHTTPRequestHandler.__init__(self, request, address, server) def do_GET(self): if '__stop__' in self.path: self.__server.server_close() raise SystemExit def k(): self.send_response(200) out = 'k\n'.encode() self.send_header('Content-Length', str(len(out))) self.send_header('Content-Type', 'text/html') self.end_headers() self.wfile.write(out) if self.path == '/enable_server_logging': self.__server.__log = True return k() if self.path == '/disable_server_logging': self.__server.__log = False return k() path = os.path.abspath(os.path.join(self.tree, *self.path.split('/'))) if not ( ((path == self.tree) or path.startswith(self.tree+os.path.sep)) and os.path.exists(path) ): self.send_response(404, 'Not Found') #self.send_response(200) out = 'Not Found'.encode() #out = '\n'.join(self.tree, self.path, path) self.send_header('Content-Length', str(len(out))) self.send_header('Content-Type', 'text/html') self.end_headers() self.wfile.write(out) return self.send_response(200) if os.path.isdir(path): out = ['\n'] names = sorted(os.listdir(path)) for name in names: if os.path.isdir(os.path.join(path, name)): name += '/' out.append('%s
\n' % (name, name)) out.append('\n') out = ''.join(out).encode() self.send_header('Content-Length', str(len(out))) self.send_header('Content-Type', 'text/html') else: with open(path, 'rb') as f: out = f.read() self.send_header('Content-Length', len(out)) if path.endswith('.egg'): self.send_header('Content-Type', 'application/zip') elif path.endswith('.gz'): self.send_header('Content-Type', 'application/x-gzip') elif path.endswith('.zip'): self.send_header('Content-Type', 'application/x-gzip') elif path.endswith('.whl'): self.send_header('Content-Type', 'application/octet-stream') else: self.send_header('Content-Type', 'text/html') self.end_headers() self.wfile.write(out) def log_request(self, code): if self.__server.__log: print_('%s %s %s' % (self.command, code, self.path)) def _run(tree, port): server_address = ('localhost', port) httpd = Server(tree, server_address, Handler) httpd.serve_forever() httpd.server_close() def get_port(): for i in range(10): port = random.randrange(20000, 30000) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) try: try: s.connect(('localhost', port)) except socket.error: return port finally: s.close() raise RuntimeError("Can't find port") def _start_server(tree, name=''): port = get_port() thread = threading.Thread( target=_run, args=(tree, port), name=name, daemon=True) thread.start() wait(port, up=True) return port, thread def start_server(tree): return _start_server(tree)[0] def stop_server(url, thread=None): try: urlopen(url+'__stop__') except Exception: pass if thread is not None: thread.join() # wait for thread to stop def wait(port, up): addr = 'localhost', port for i in range(120): time.sleep(0.25) try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(addr) s.close() if up: break except socket.error: e = sys.exc_info()[1] if e[0] not in (errno.ECONNREFUSED, errno.ECONNRESET): raise s.close() if not up: break else: if up: raise else: raise SystemError("Couldn't stop server") def install(project, destination): if not isinstance(destination, str): destination = os.path.join(destination.globs['sample_buildout'], 'eggs') dist = pkg_resources.working_set.find( pkg_resources.Requirement.parse(project)) if dist.location.endswith('.egg'): destination = os.path.join(destination, os.path.basename(dist.location), ) if os.path.isdir(dist.location): shutil.copytree(dist.location, destination) else: shutil.copyfile(dist.location, destination) else: # copy link with open(os.path.join(destination, project+'.egg-link'), 'w') as f: f.write(dist.location) def install_develop(project, destination): if not isinstance(destination, str): destination = os.path.join(destination.globs['sample_buildout'], 'develop-eggs') dist = pkg_resources.working_set.find( pkg_resources.Requirement.parse(project)) with open(os.path.join(destination, project+'.egg-link'), 'w') as f: f.write(dist.location) def _normalize_path(match): path = match.group(1) if os.path.sep == '\\': path = path.replace('\\\\', '/') if path.startswith('\\'): path = path[1:] return '/' + path.replace(os.path.sep, '/') normalize_path = ( re.compile( r'''[^'" \t\n\r]+\%(sep)s_[Tt][Ee][Ss][Tt]_\%(sep)s([^"' \t\n\r]+)''' % dict(sep=os.path.sep)), _normalize_path, ) normalize_endings = re.compile('\r\n'), '\n' normalize_script = ( re.compile('(\n?)- ([a-zA-Z_.-]+)-script.py\n- \\2.exe\n'), '\\1- \\2\n') normalize___pycache__ = ( re.compile('(\n?)d __pycache__\n'), '\\1') normalize_egg_py = ( re.compile(r'-py\d[.]\d+(-\S+)?\.egg'), '-pyN.N.egg', ) normalize_exception_type_for_python_2_and_3 = ( re.compile(r'^(\w+\.)*([A-Z][A-Za-z0-9]+Error: )'), '\2') normalize_open_in_generated_script = ( re.compile(r"open\(__file__, 'U'\)"), 'open(__file__)') # Remove the complete line: a partial match would leave the line prefix # (e.g. a "root: " logging prefix) glued to the following line, where a # later normalizer like ignore_root_logger could then eat that line too. not_found = (re.compile(r'.*Not found: [^\n]+/(\w|\.|-)+/\r?\n'), '') easyinstall_deprecated = (re.compile(r'.*EasyInstallDeprecationWarning.*\n'),'') setuptools_deprecated = (re.compile(r'.*SetuptoolsDeprecationWarning.*\n'),'') pkg_resources_deprecated = (re.compile(r'.*PkgResourcesDeprecationWarning.*\n'),'') warnings_warn = (re.compile(r'.*warnings\.warn.*\n'),'') # Setuptools now pulls in dependencies when installed. adding_find_link = (re.compile(r"Adding find link '[^']+'" r" from setuptools .*\r?\n"), '') ignore_not_upgrading = ( re.compile( 'Not upgrading because not running a local buildout command.\n' ), '') # The root logger from setuptools prints all kinds of lines. # This might depend on which setuptools version, or something else, # because it did not happen before. Sample lines: # "root: Couldn't retrieve index page for 'zc.recipe.egg'" # "root: Scanning index of all packages. # "root: Found: /sample-buildout/recipe/dist/spam-2-pyN.N.egg" # I keep finding new lines like that, so let's ignore all. ignore_root_logger = (re.compile(r'root:.*'), '') # Now replace a multiline warning about that you should switch to native namespaces. ignore_native_namespace_warning_1 = (re.compile(r'!!'), '') ignore_native_namespace_warning_2 = (re.compile(r'\*' * 80), '') ignore_native_namespace_warning_3 = (re.compile( r'Please replace its usage with implicit namespaces \(PEP 420\).'), '' ) ignore_native_namespace_warning_4 = (re.compile( r'See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages for details.'), '' ) ignore_native_namespace_warning_5 = (re.compile( r'ep.load\(\)\(self, ep.name, value\)'), '' ) def run_buildout(command): # Make sure we don't get .buildout os.environ['HOME'] = os.path.join(os.getcwd(), 'home') args = command.split() buildout = pkg_resources.load_entry_point( 'zc.buildout', 'console_scripts', args[0]) buildout(args[1:]) def run_from_process(target, *args, **kw): sys.stdout = sys.stderr = open('out', 'w') target(*args, **kw) def run_in_process(*args, **kwargs): try: ctx = multiprocessing.get_context('fork') process = ctx.Process(target=run_from_process, args=args, kwargs=kwargs) except AttributeError: process = multiprocessing.Process(target=run_from_process, args=args, kwargs=kwargs) process.daemon = True process.start() process.join(99) if process.is_alive() or process.exitcode: with open('out') as f: print(f.read()) def run_buildout_in_process(command='buildout'): command = command.split(' ', 1) command.insert( 1, " use-dependency-links=false" # Leaving this here so we can uncomment to see what's going on. #" log-format=%(asctime)s____%(levelname)s_%(message)s -vvv" " index=" + __file__ + 'nonexistent' # hide index ) command = ' '.join(command) run_in_process(run_buildout, command) def setup_coverage(path_to_coveragerc): if 'RUN_COVERAGE' not in os.environ: return if not os.path.exists(path_to_coveragerc): raise ValueError('coveragerc file %s does not exist.' % path_to_coveragerc) os.environ['COVERAGE_PROCESS_START'] = path_to_coveragerc rootdir = os.path.dirname(path_to_coveragerc) def combine_report(): subprocess.call( [ sys.executable, '-m', 'coverage', 'combine', ], cwd=rootdir, ) subprocess.call( [ sys.executable, '-m', 'coverage', 'report', ], cwd=rootdir, ) if path_to_coveragerc: try: import coverage print("Coverage configured with %s" % path_to_coveragerc) if 'COVERAGE_REPORT' in os.environ: import atexit atexit.register(combine_report) coverage.process_startup() except ImportError: print( "You try to run coverage " "but coverage is not installed in your environment." ) sys.exit(1) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4013336 zc_buildout-6.0.1/src/zc/buildout/testing.txt0000644000076500000240000001615415264141650020733 0ustar00mauritsstaffTesting Support =============== The zc.buildout.testing module provides an API that can be used when writing recipe tests. This API is documented below. Many examples of using this API can be found in the zc.buildout, zc.recipe.egg, and zc.recipe.testrunner tests. zc.buildout.testing.buildoutSetUp(test) --------------------------------------- The buildoutSetup function can be used as a doctest setup function. It creates a sample buildout that can be used by tests, changing the current working directory to the sample_buildout. It also adds a number of names to the test namespace: ``sample_buildout`` This is the name of a buildout with a basic configuration. ``buildout`` This is the path of the buildout script in the sample buildout. ``ls(*path)`` List the contents of a directory. The directory path is provided as one or more strings, to be joined with os.path.join. ``cat(*path)`` Display the contents of a file. The file path is provided as one or more strings, to be joined with os.path.join. On Windows, if the file doesn't exist, the function will try adding a '-script.py' suffix. This helps to work around a difference in script generation on windows. ``mkdir(*path)`` Create a directory. The directory path is provided as one or more strings, to be joined with os.path.join. ``rmdir(*path)`` Remove a directory. The directory path is provided as one or more strings, to be joined with os.path.join. ``remove(*path)`` Remove a directory or file. The path is provided as one or more strings, to be joined with os.path.join. ``tmpdir(name)`` Create a temporary directory with the given name. The directory will be automatically removed at the end of the test. The path of the created directory is returned. Further, if the the normalize_path normalizing substitution (see below) is used, then any paths starting with this path will be normalized to:: /name/restofpath No two temporary directories can be created with the same name. A directory created with tmpdir can be removed with rmdir and recreated. Note that the sample_buildout directory is created by calling this function. ``write(*path_and_contents)`` Create a file. The file path is provided as one or more strings, to be joined with os.path.join. The last argument is the file contents. ``system(command, input='')`` Execute a system command with the given input passed to the command's standard input. The output (error and regular output) from the command is returned. ``get(url)`` Get a web page. ``cd(*path)`` Change to the given directory. The directory path is provided as one or more strings, to be joined with os.path.join. The directory will be reset at the end of the test. ``uncd()`` Change to the directory that was current prior to the previous call to ``cd``. You can call ``cd`` multiple times and then ``uncd`` the same number of times to return to the same location. ``join(*path)`` A convenient reference to os.path.join. ``register_teardown(func)`` Register a tear-down function. The function will be called with no arguments at the end of the test. ``start_server(path)`` Start a web server on the given path. The server will be shut down at the end of the test. The server URL is returned. You can cause the server to start and stop logging it's output using: >>> get(server_url+'enable_server_logging') and: >>> get(server_url+'disable_server_logging') This can be useful to see how buildout is interacting with a server. ``sdist(setup, dest)`` Create a source distribution by running the given setup file and placing the result in the given destination directory. If the setup argument is a directory, the setup.py file in that directory is used. ``bdist_egg(setup, dest)`` Create an egg by running the given setup file and placing the result in the given destination directory. If the setup argument is a directory, then the setup.py file in that directory is used. Code Coverage Measurement ~~~~~~~~~~~~~~~~~~~~~~~~~ ``buildoutSetUp`` prepares a new ``bin/buildout`` executable for doctests to run. Because this is will run in a subprocess, there are some extra steps needed to collect code coverage data for buildout and buildout recipes. This function does most of the heavy lifting for you, you only need to do a few things. First, you need to define a coverage configuration ('.coveragerc') which specifies that multiple different runs should be collected:: [run] source = my.egg.name parallel = true Second, when you invoke the test runner, you need to define the ``COVERAGE_PROCESS_START`` environment variable to point to this configuration and invoke it with coverage:: COVERAGE_PROCESS_START=.coveragerc coverage run -m zope.testrunner --test-path=src Finally, after the tests have finished, you need to combine the resulting coverage data files and then you can produce reports:: coverage combine coverage report .. caution:: You must be working in an environment (e.g., a virtual environment) that has coverage already installed. ``zc.buildout.testing.buildoutTearDown(test)`` ---------------------------------------------- Tear down everything set up by zc.buildout.testing.buildoutSetUp. Any functions passed to register_teardown are called as well. ``install(project, destination)`` --------------------------------- Install eggs for a given project into a destination. If the destination is a test object, then the eggs directory of the sample buildout (sample_buildout) defined by the test will be used. Tests will use this to install the distributions for the packages being tested (and their dependencies) into a sample buildout. The egg to be used should already be loaded, by importing one of the modules provided, before calling this function. ``install_develop(project, destination)`` ----------------------------------------- Like install, but a develop egg is installed even if the current egg if not a develop egg. ``Output normalization`` ------------------------ Recipe tests often generate output that is dependent on temporary file locations, operating system conventions or Python versions. To deal with these dependencies, we often use zope.testing.renormalizing.RENormalizing to normalize test output. zope.testing.renormalizing.RENormalizing takes pairs of regular expressions and substitutions. The zc.buildout.testing module provides a few helpful variables that define regular-expression/substitution pairs that you can pass to zope.testing.renormalizing.RENormalizing. ``normalize_path`` Converts tests paths, based on directories created with tmpdir(), to simple paths. ``normalize_script`` On Unix-like systems, scripts are implemented in single files without suffixes. On windows, scripts are implemented with 2 files, a -script.py file and a .exe file. This normalization converts directory listings of Windows scripts to the form generated on UNix-like systems. ``normalize_egg_py`` Normalize Python version and platform indicators, if specified, in egg names. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4013855 zc_buildout-6.0.1/src/zc/buildout/testrecipes.py0000644000076500000240000000062515264141650021415 0ustar00mauritsstafffrom zc.buildout.buildout import print_ class Debug: def __init__(self, buildout, name, options): self.buildout = buildout self.name = name self.options = options def install(self): items = list(self.options.items()) items.sort() for option, value in items: print_(" %s=%r" % (option, value)) return () update = install ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2138083 zc_buildout-6.0.1/src/zc/buildout/tests/0000755000076500000240000000000015264141651017651 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4015105 zc_buildout-6.0.1/src/zc/buildout/tests/__init__.py0000644000076500000240000001737515264141650021776 0ustar00mauritsstaff# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2020 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import re import os import sys import shutil import tempfile import zc.buildout def create_sample_eggs(test, executable=sys.executable): assert executable == sys.executable, (executable, sys.executable) write = test.globs['write'] dest = test.globs['sample_eggs'] tmp = tempfile.mkdtemp() try: write(tmp, 'README.txt', '') for i in (0, 1, 2): write(tmp, 'eggrecipedemoneeded.py', 'y=%s\ndef f():\n pass' % i) rc1 = 'rc1' if i==2 else '' write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='demoneeded', py_modules=['eggrecipedemoneeded']," " zip_safe=True, version='1.%s%s', author='bob', url='bob', " "author_email='bob')\n" % (i, rc1) ) zc.buildout.testing.sdist(tmp, dest) write( tmp, 'distutilsscript', '#!/usr/bin/python\n' '# -*- coding: utf-8 -*-\n' '"""Module docstring."""\n' 'import os\n' 'import sys; sys.stdout.write("distutils!\\n")\n' ) write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='other', zip_safe=False, version='1.0', " "scripts=['distutilsscript']," "py_modules=['eggrecipedemoneeded'])\n" ) zc.buildout.testing.bdist_wheel(tmp, dest) write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='du_zipped', zip_safe=True, version='1.0', " "scripts=['distutilsscript']," "py_modules=['eggrecipedemoneeded'])\n" ) # We still create an egg for this one, as we use it for testing # distutils scripts in a zipped egg. # TODO No, we should not support this anymore. zc.buildout.testing.bdist_egg(tmp, executable, dest) os.remove(os.path.join(tmp, 'distutilsscript')) os.remove(os.path.join(tmp, 'eggrecipedemoneeded.py')) for i in (1, 2, 3, 4): write( tmp, 'eggrecipedemo.py', 'import eggrecipedemoneeded, sys\n' 'def print_(*a):\n' ' sys.stdout.write(" ".join(map(str, a))+"\\n")\n' 'x=%s\n' 'def main():\n' ' print_(x, eggrecipedemoneeded.y)\n' % i) rc1 = 'rc1' if i==4 else '' write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='demo', py_modules=['eggrecipedemo']," " install_requires = 'demoneeded'," " entry_points={'console_scripts': " "['demo = eggrecipedemo:main']}," " zip_safe=True, version='0.%s%s')\n" % (i, rc1) ) zc.buildout.testing.bdist_wheel(tmp, dest) write(tmp, 'mixedcase.py', 'def f():\n pass') write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='MIXEDCASE', py_modules=['mixedcase']," " author='bob', url='bob', author_email='bob'," " install_requires = 'demoneeded'," " zip_safe=True, version='0.5')\n" ) zc.buildout.testing.sdist(tmp, dest) # rename file to lower case # to test issues between file and package name curdir = os.getcwd() os.chdir(dest) for file in os.listdir(dest): if "MIXEDCASE" in file: os.rename(file, file.lower()) os.chdir(curdir) write(tmp, 'eggrecipebigdemo.py', 'import eggrecipedemo') write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='bigdemo', " " install_requires = 'demo'," " py_modules=['eggrecipebigdemo'], " " zip_safe=True, version='0.1')\n" ) zc.buildout.testing.bdist_wheel(tmp, dest) finally: shutil.rmtree(tmp) extdemo_c = """ #include #include static PyMethodDef methods[] = {{NULL}}; #define MOD_DEF(ob, name, doc, methods) \ static struct PyModuleDef moduledef = { \ PyModuleDef_HEAD_INIT, name, doc, -1, methods, }; \ ob = PyModule_Create(&moduledef); #define MOD_INIT(name) PyMODINIT_FUNC PyInit_##name(void) MOD_INIT(extdemo) { PyObject *m; MOD_DEF(m, "extdemo", "", methods); #ifdef TWO PyModule_AddObject(m, "val", PyLong_FromLong(2)); #else PyModule_AddObject(m, "val", PyLong_FromLong(EXTDEMO)); #endif return m; } """ extdemo_setup_py = r""" import os, sys from distutils.core import setup, Extension if os.environ.get('test_environment_variable'): print( "Have environment test_environment_variable: %%s" %% os.environ['test_environment_variable'] ) setup(name = "extdemo", version = "%s", url="http://www.zope.org", author="Demo", author_email="demo@demo.com", ext_modules = [Extension('extdemo', ['extdemo.c'])], ) """ def add_source_dist(test, version=1.4): if 'extdemo' not in test.globs: test.globs['extdemo'] = test.globs['tmpdir']('extdemo') tmp = test.globs['extdemo'] write = test.globs['write'] try: write(tmp, 'extdemo.c', extdemo_c); write(tmp, 'setup.py', extdemo_setup_py % version); write(tmp, 'README', ""); write(tmp, 'MANIFEST.in', "include *.c\n"); test.globs['sdist'](tmp, test.globs['sample_eggs']) except Exception: shutil.rmtree(tmp) def easy_install_SetUp(test): zc.buildout.testing.buildoutSetUp(test) sample_eggs = test.globs['tmpdir']('sample_eggs') test.globs['sample_eggs'] = sample_eggs os.mkdir(os.path.join(sample_eggs, 'index')) create_sample_eggs(test) add_source_dist(test) test.globs['link_server'] = test.globs['start_server']( test.globs['sample_eggs']) test.globs['update_extdemo'] = lambda : add_source_dist(test, 1.5) zc.buildout.testing.install_develop('zc.recipe.egg', test) normalize_bang = ( re.compile(re.escape('#!'+ zc.buildout.easy_install._safe_arg(sys.executable)) + ".*"), '#!/usr/local/bin/python2.7', ) def create_wheel(name, version, dest, install_requires=None, dependency_links=None): d = tempfile.mkdtemp() if dest=='available': extras = dict(x=['x']) else: extras = {} if dependency_links: links = 'dependency_links = %s, ' % dependency_links else: links = '' if install_requires: requires = 'install_requires = %s, ' % install_requires else: requires = '' try: with open(os.path.join(d, 'setup.py'), 'w') as f: f.write( 'from setuptools import setup\n' 'setup(name=%r, version=%r, extras_require=%r, zip_safe=True,\n' ' %s %s py_modules=["setup"]\n)' % (name, str(version), extras, requires, links) ) zc.buildout.testing.bdist_wheel(d, os.path.abspath(dest)) finally: shutil.rmtree(d) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4015756 zc_buildout-6.0.1/src/zc/buildout/tests/allow-unknown-extras.txt0000644000076500000240000000351415264141650024533 0ustar00mauritsstaff====================== Allow Unknown Extras ====================== Sometimes we need to allow unknown extras. The ``allow-unknown-extras`` option lets us do that in a buildout configuration, just as we can directly calling ``easy_install`` works exactly like the one provided in ``easy_install``. Let's create a develop egg that requires a bogus extra. >>> mkdir(sample_buildout, 'allowdemo') >>> write(sample_buildout, 'allowdemo', 'dependencydemo.py', ... 'import eggrecipekss.core') >>> write(sample_buildout, 'allowdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='allowdemo', py_modules=['dependencydemo'], ... zip_safe=True, version='1') ... ''') Now let's configure the buildout to use the develop egg with a bogus extra. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = allowdemo ... parts = eggs ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = allowdemo[bad_extra] ... ''') Now we can run the buildout and see that it fails: >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/allowdemo' Installing eggs... ... While: Installing eggs. Error: Couldn't find the required extra... If we flip the option on, the buildout succeeds >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = allowdemo ... parts = eggs ... allow-unknown-extras = true ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = allowdemo[bad_extra] ... ''') Now we can run the buildout and only get a warning:: >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/allowdemo' Installing eggs... allowdemo 1 does not provide the extra 'bad_extra' ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4016345 zc_buildout-6.0.1/src/zc/buildout/tests/allowhosts.txt0000644000076500000240000000750215264141650022614 0ustar00mauritsstaffAllow hosts ----------- On some environments the links visited by `zc.buildout` can be forbidden by paranoiac firewalls. These URL might be on the chain of links visited by `zc.buildout` whether they are defined in the `find-links` option or by various eggs in their `url`, `download_url` and `dependency_links` metadata. It is even harder to track that package_index works like a spider and might visit links and go to other location. The `allow-hosts` option provides a way to prevent this, and works exactly like the one provided in `easy_install` (see `easy_install allow-hosts option`_). You can provide a list of allowed host, together with wildcards:: [buildout] ... allow-hosts = *.python.org example.com Let's create a develop egg in our buildout that specifies `dependency_links` which points to a server in the outside world:: >>> mkdir(sample_buildout, 'allowdemo') >>> write(sample_buildout, 'allowdemo', 'dependencydemo.py', ... 'import eggrecipekss.core') >>> write(sample_buildout, 'allowdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='allowdemo', py_modules=['dependencydemo'], ... install_requires = 'kss.core', ... dependency_links = ['http://dist.plone.org'], ... zip_safe=True, version='1') ... ''') Now let's configure the buildout to use the develop egg, together with some rules that disallow any web site but PyPI and local files:: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = allowdemo ... parts = eggs ... allow-hosts = ... pypi.org ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = allowdemo ... ''') Now we can run the buildout and make sure all attempts to dist.plone.org fails:: >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/allowdemo' Installing eggs... ... While: Installing eggs. Getting distribution for 'kss.core'. Error: Couldn't find a distribution for 'kss.core'. That's what we wanted : this will prevent any attempt to access unwanted domains. For instance, some packages are listing in their links `svn://` links. These can lead to error in some cases, and can therefore be protected like this:: XXX (showcase with a svn:// file) >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = allowdemo ... parts = eggs ... allow-hosts = ... ^(!svn://).* ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = allowdemo ... ''') Now we can run the buildout and make sure all attempts to dist.plone.org fails:: >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/allowdemo' Installing eggs... ... While: Installing eggs. Getting distribution for 'kss.core'. Error: Couldn't find a distribution for 'kss.core'. Test for issues --------------- Test for 1.0.5 breakage as in https://bugs.launchpad.net/zc.buildout/+bug/239212:: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... parts=python ... foo = ${python:interpreter} ... ... [python] ... recipe=zc.recipe.egg ... eggs=zc.buildout ... interpreter=python ... ''') >>> print_('XX'); print_(system(buildout), end='') # doctest: +ELLIPSIS X... Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Installing python... Generated interpreter '/sample-buildout/bin/python'. The bug 239212 above would have got us an *AttributeError* on *buildout._allow_hosts*. This was fixed in this changeset: http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/buildout.py?rev=87309&r1=87277&r2=87309 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4017549 zc_buildout-6.0.1/src/zc/buildout/tests/buildout.txt0000644000076500000240000005774415264141650022261 0ustar00mauritsstaffBuildouts ========= The word "buildout" refers to a description of a set of parts and the software to create and assemble them. It is often used informally to refer to an installed system based on a buildout definition. For example, if we are creating an application named "Foo", then "the Foo buildout" is the collection of configuration and application-specific software that allows an instance of the application to be created. We may refer to such an instance of the application informally as "a Foo buildout". This document describes how to define buildouts using buildout configuration files and recipes. There are three ways to set up the buildout software and create a buildout instance: 1. Install the ``zc.buildout`` egg with ``easy_install`` and use the buildout script installed in a Python scripts area. 2. Use the buildout bootstrap script to create a buildout that includes both the ``setuptools`` and ``zc.buildout`` eggs. This allows you to use the buildout software without modifying a Python install. The buildout script is installed into your buildout local scripts area. 3. Use a buildout command from an already installed buildout to bootstrap a new buildout. (See the section on bootstrapping later in this document.) Often, a software project will be managed in a software repository, such as a subversion repository, that includes some software source directories, buildout configuration files, and a copy of the buildout bootstrap script. To work on the project, one would check out the project from the repository and run the bootstrap script which installs ``setuptools`` and ``zc.buildout`` into the checkout as well as any parts defined. We have a sample buildout that we created using the bootstrap command of an existing buildout (method 3 above). It has the absolute minimum information. We have ``bin``, ``develop-eggs``, ``eggs`` and ``parts`` directories, and a configuration file:: >>> ls(sample_buildout) d bin - buildout.cfg d develop-eggs d eggs d parts d recipes The ``bin`` directory contains scripts:: >>> ls(sample_buildout, 'bin') - buildout The ``eggs`` directory has installed distributions, now in a directory per format version: >>> ls(sample_buildout, 'eggs') d v5 >>> ls(sample_buildout, 'eggs', 'v5') - packaging.egg-link - pip.egg-link - platformdirs.egg-link - setuptools.egg-link - wheel.egg-link - zc.buildout.egg-link The ``develop-eggs`` and ``parts`` directories are initially empty:: >>> ls(sample_buildout, 'develop-eggs') >>> ls(sample_buildout, 'parts') The ``develop-eggs`` directory holds egg links for software being developed in the buildout. We separate ``develop-eggs`` and other eggs to allow eggs directories to be shared across multiple buildouts. For example, a common developer technique is to define a common eggs directory in their home that all non-develop eggs are stored in. This allows larger buildouts to be set up much more quickly and saves disk space. The ``parts`` directory provides an area where recipes can install part data. For example, if we built a custom Python, we would install it in the ``parts`` directory. Part data is stored in a sub-directory of the parts directory with the same name as the part. Buildouts are defined using configuration files. These are in the format defined by the Python ``ConfigParser`` module, with extensions that we'll describe later. By default, when a buildout is run, it looks for the file ``buildout.cfg`` in the directory where the buildout is run. The minimal configuration file has a ``buildout`` section that defines no parts:: >>> cat(sample_buildout, 'buildout.cfg') [buildout] parts = A part is simply something to be created by a buildout. It can be almost anything, such as a Python package, a program, a directory, or even a configuration file. Recipes ------- A part is created by a recipe. Recipes are always installed as Python eggs. They can be downloaded from a package server, such as the Python Package Index, or they can be developed as part of a project using a "develop" egg. A develop egg is a special kind of egg that gets installed as an "egg link" that contains the name of a source directory. Develop eggs don't have to be packaged for distribution to be used and can be modified in place, which is especially useful while they are being developed. Let's create a recipe as part of the sample project. We'll create a recipe for creating directories. Our sample buildout has a recipes source directory for our local recipes:: >>> ls(sample_buildout, 'recipes') - README.txt - setup.py d src >>> ls(sample_buildout, 'recipes', 'src') - debug.py - environ.py - mkdir.py Here we will only use one of these files, for our ``mkdir`` recipe. But for demonstration purposes we start with a simpler version that we will gradually improve. >>> write(sample_buildout, 'recipes', 'src', 'mkdir.py', ... """ ... import logging, os, zc.buildout ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... options['path'] = os.path.join( ... buildout['buildout']['directory'], ... options['path'], ... ) ... if not os.path.isdir(os.path.dirname(options['path'])): ... logging.getLogger(self.name).error( ... 'Cannot create %s. %s is not a directory.', ... options['path'], os.path.dirname(options['path'])) ... raise zc.buildout.UserError('Invalid Path') ... ... ... def install(self): ... path = self.options['path'] ... logging.getLogger(self.name).info( ... 'Creating directory %s', os.path.basename(path)) ... os.mkdir(path) ... return path ... ... def update(self): ... pass ... """) Currently, recipes must define 3 methods: - a constructor, - an install method, and - an update method. The constructor is responsible for updating a parts options to reflect data read from other sections. The buildout system keeps track of whether a part specification has changed. A part specification has changed if it's options, after adjusting for data read from other sections, has changed, or if the recipe has changed. Only the options for the part are considered. If data are read from other sections, then that information has to be reflected in the parts options. In the ``mkdir`` example, the given path is interpreted relative to the buildout directory, and data from the buildout directory is read. The path option is updated to reflect this. If the directory option was changed in the buildout sections, we would know to update parts created using the ``mkdir`` recipe using relative path names. When buildout is run, it saves configuration data for installed parts in a file named ``.installed.cfg``. In subsequent runs, it compares part-configuration data stored in the ``.installed.cfg`` file and the part-configuration data loaded from the configuration files as modified by recipe constructors to decide if the configuration of a part has changed. If the configuration has changed, or if the recipe has changed, then the part is uninstalled and reinstalled. The buildout only looks at the part's options, so any data used to configure the part needs to be reflected in the part's options. It is the job of a recipe constructor to make sure that the options include all relevant data. Of course, parts are also uninstalled if they are no longer used. The recipe defines a constructor that takes a buildout object, a part name, and an options dictionary. It saves them in instance attributes. If the path is relative, we'll interpret it as relative to the buildout directory. The buildout object passed in is a mapping from section name to a mapping of options for that section. The buildout directory is available as the directory option of the buildout section. We normalize the path and save it back into the options directory. The install method is responsible for creating the part. In this case, we need the path of the directory to create. We'll use a path option from our options dictionary. The install method logs what it's doing using the Python ``logging`` call. We return the path that we installed. If the part is uninstalled or reinstalled, then the path returned will be removed by the buildout machinery. A recipe install method is expected to return a string, or an iterable of strings containing paths to be removed if a part is uninstalled. For most recipes, this is all of the uninstall support needed. For more complex uninstallation scenarios, use `Uninstall recipes`_. The update method is responsible for updating an already installed part. An empty method is often provided, as in this example, if parts can't be updated. An update method can return None, a string, or an iterable of strings. If a string or iterable of strings is returned, then the saved list of paths to be uninstalled is updated with the new information by adding any new files returned by the update method. We need to provide packaging information so that our recipe can be installed as a develop egg. The minimum information we need to specify is a name. For recipes, we also need to define the names of the recipe classes as entry points. Packaging information is provided via a ``setup.py`` script:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "recipes", ... entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']}, ... ) ... """) Our setup script defines an *entry point*. Entry points provide a way for an egg to define the services it provides. Here we've said that we define a ``zc.buildout`` entry point named ``mkdir``. Recipe classes must be exposed as entry points in the ``zc.buildout`` group. We give entry points names within the group. Now let's update our ``buildout.cfg``:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = mystuff ... """) Let's go through the changes one by one:: develop = recipes This tells the buildout to install a development egg for our recipes. Any number of paths can be listed. The paths can be relative or absolute. If relative, they are treated as relative to the buildout directory. They can be directory or file paths. If a file path is given, it should point to a Python setup script. If a directory path is given, it should point to a directory containing a ``setup.py`` file. Development eggs are installed before building any parts, as they may provide locally-defined recipes needed by the parts. :: parts = data-dir Here we've named a part to be "built". We can use any name we want, except that part names have to be unique and recipes will often use the part name to decide what to do. :: [data-dir] recipe = recipes:mkdir path = mystuff When we name a part, we also create a section of the same name that contains part data. In this section, we'll define the recipe to be used to install the part. In this case, we also specify the path to be created. Let's run the buildout. We do so by running the build script in the buildout:: >>> os.chdir(sample_buildout) >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory mystuff We see that the recipe created the directory, as expected:: >>> ls(sample_buildout) - .installed.cfg d bin - buildout.cfg d develop-eggs d eggs d mystuff d parts d recipes In addition, .installed.cfg has been created containing information about the part we installed:: >>> cat(sample_buildout, '.installed.cfg') [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = data-dir [data-dir] __buildout_installed__ = /sample-buildout/mystuff __buildout_signature__ = recipes-c7vHV6ekIDUPy/7fjAaYjg== path = /sample-buildout/mystuff recipe = recipes:mkdir Note that the directory we installed is included in .installed.cfg. In addition, the path option includes the actual destination directory. If we change the name of the directory in the configuration file, we'll see that the directory gets removed and recreated:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling data-dir. Installing data-dir. data-dir: Creating directory mydata >>> ls(sample_buildout) - .installed.cfg d bin - buildout.cfg d develop-eggs d eggs d mydata d parts d recipes If any of the files or directories created by a recipe are removed, the part will be reinstalled:: >>> rmdir(sample_buildout, 'mydata') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling data-dir. Installing data-dir. data-dir: Creating directory mydata Error reporting --------------- If a user makes an error the error needs to be reported, and work needs to stop. This is accomplished by logging a detailed error message and then raising a ``zc.buildout.UserError`` exception (or a subclass of this exception). Raising an error other than a ``UserError`` still displays the error, but labels it as a bug in the buildout software or recipe. In the sample above, if someone gives a non-existent directory to create the directory in:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = /xxx/mydata ... """) we'll get a user error, not a traceback:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' data-dir: Cannot create .../xxx/mydata. .../xxx is not a directory. While: Installing. Getting section data-dir. Initializing section data-dir. Error: Invalid Path Recipe Error Handling --------------------- If an error occurs during installation, it is up to the recipe to clean up any system side effects, such as files created. Let's update the ``mkdir`` recipe to support multiple paths:: >>> write(sample_buildout, 'recipes', 'src', 'mkdir.py', ... """ ... import logging, os, zc.buildout ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... ... # Normalize paths and check that their parent ... # directories exist: ... paths = [] ... for path in options['path'].split(): ... path = os.path.join(buildout['buildout']['directory'], path) ... if not os.path.isdir(os.path.dirname(path)): ... logging.getLogger(self.name).error( ... 'Cannot create %s. %s is not a directory.', ... options['path'], os.path.dirname(options['path'])) ... raise zc.buildout.UserError('Invalid Path') ... paths.append(path) ... options['path'] = ' '.join(paths) ... ... def install(self): ... paths = self.options['path'].split() ... for path in paths: ... logging.getLogger(self.name).info( ... 'Creating directory %s', os.path.basename(path)) ... os.mkdir(path) ... return paths ... ... def update(self): ... pass ... """) .. >>> clean_up_pyc(sample_buildout, 'recipes', 'src', 'mkdir.py') If there is an error creating a path, the install method will exit and leave previously created paths in place:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = foo bin ... """) >>> print_(system(buildout)) # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Uninstalling data-dir. Installing data-dir. data-dir: Creating directory foo data-dir: Creating directory bin While: Installing data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... exists... We meant to create a directory ``bins``, but typed ``bin``. Now ``foo`` was left behind:: >>> os.path.exists('foo') True If we fix the typo:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = foo bins ... """) >>> print_(system(buildout)) # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory foo While: Installing data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... exists... Now they fail because ``foo`` exists, because it was left behind:: >>> remove('foo') Let's fix the recipe:: >>> write(sample_buildout, 'recipes', 'src', 'mkdir.py', ... """ ... import logging, os, zc.buildout, sys ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... ... # Normalize paths and check that their parent ... # directories exist: ... paths = [] ... for path in options['path'].split(): ... path = os.path.join(buildout['buildout']['directory'], path) ... if not os.path.isdir(os.path.dirname(path)): ... logging.getLogger(self.name).error( ... 'Cannot create %s. %s is not a directory.', ... options['path'], os.path.dirname(options['path'])) ... raise zc.buildout.UserError('Invalid Path') ... paths.append(path) ... options['path'] = ' '.join(paths) ... ... def install(self): ... paths = self.options['path'].split() ... created = [] ... try: ... for path in paths: ... logging.getLogger(self.name).info( ... 'Creating directory %s', os.path.basename(path)) ... os.mkdir(path) ... created.append(path) ... except Exception: ... for d in created: ... os.rmdir(d) ... assert not os.path.exists(d) ... logging.getLogger(self.name).info( ... 'Removed %s due to error', ... os.path.basename(d)) ... sys.stderr.flush() ... sys.stdout.flush() ... raise ... ... return paths ... ... def update(self): ... pass ... """) .. >>> clean_up_pyc(sample_buildout, 'recipes', 'src', 'mkdir.py') And put back the typo:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = foo bin ... """) When we rerun the buildout:: >>> print_(system(buildout)) # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory foo data-dir: Creating directory bin data-dir: Removed foo due to error While: Installing data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... exists... we get the same error, but we don't get the directory left behind:: >>> os.path.exists('foo') False It's critical that recipes clean up partial effects when errors occur. Because recipes most commonly create files and directories, buildout provides a helper API for removing created files when an error occurs. Option objects have a ``created`` method that can be called to record files as they are created. If the ``install`` or ``update`` method returns with an error, then any registered paths are removed automatically. The method returns the files registered and can be used to return the files created. Let's use this API to simplify the recipe:: >>> write(sample_buildout, 'recipes', 'src', 'mkdir.py', ... """ ... import logging, os, zc.buildout ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... ... # Normalize paths and check that their parent ... # directories exist: ... paths = [] ... for path in options['path'].split(): ... path = os.path.join(buildout['buildout']['directory'], path) ... if not os.path.isdir(os.path.dirname(path)): ... logging.getLogger(self.name).error( ... 'Cannot create %s. %s is not a directory.', ... options['path'], os.path.dirname(options['path'])) ... raise zc.buildout.UserError('Invalid Path') ... paths.append(path) ... options['path'] = ' '.join(paths) ... ... def install(self): ... paths = self.options['path'].split() ... for path in paths: ... logging.getLogger(self.name).info( ... 'Creating directory %s', os.path.basename(path)) ... os.mkdir(path) ... self.options.created(path) ... ... return self.options.created() ... ... def update(self): ... pass ... """) .. >>> clean_up_pyc(sample_buildout, 'recipes', 'src', 'mkdir.py') We returned by calling ``created``, taking advantage of the fact that it returns the registered paths. We did this for illustrative purposes. It would be simpler to just return the paths as before. If we rerun the buildout again, we'll get the error and no directories will be created:: >>> print_(system(buildout)) # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory foo data-dir: Creating directory bin While: Installing data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... exists... >>> os.path.exists('foo') False Now, we'll fix the typo again and we'll get the directories we expect:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = foo bins ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory foo data-dir: Creating directory bins >>> os.path.exists('foo') True >>> os.path.exists('bins') True Originally this ``buildout.txt`` file contained much more tests. They have been split into multiple files. If you want to follow along in the narrative, the correct order is this: - buildout.txt - configuration.txt - extending.txt - options.txt - init.txt - extensions.txt ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4018455 zc_buildout-6.0.1/src/zc/buildout/tests/configparser.test0000644000076500000240000002646115264141650023244 0ustar00mauritsstaffSome tests of the basic config-file parser: First, an example that illustrates a well-formed configuration:: [s1] a = 1 [ s2 ] # a comment long = a b c l2 = a # not a comment # comment ; also a comment b c empty = c=1 b += 1 [s3]; comment x = a b .. -> text >>> from io import StringIO >>> from pprint import pprint >>> import zc.buildout.configparser >>> def parse(config, *args, **kw): ... return zc.buildout.configparser.parse(StringIO(config), ... 'test', *args, **kw) >>> pprint(parse(text)) {'s1': {'a': '1'}, 's2': {'b +': '1', 'c': '1', 'empty': '', 'l2': 'a\n\n\n# not a comment\n\n\nb\n\n c', 'long': 'a\nb\nc'}, 's3': {'x': 'a b'}} Here's an example with leading blank lines: >>> text = '\n\n[buildout]\nz=1\n\n' >>> pprint(parse(text)) {'buildout': {'z': '1'}} From email: "It fails when the first non-comment line after a section (even an otherwise empty section) is blank. For example:" [buildout] parts = hello versions = versions [versions] # Add any version pins here. [hello] recipe = collective.recipe.cmd on_install = true on_update = true cmds = echo Hello .. -> text >>> pprint(parse(text)) {'buildout': {'parts': 'hello', 'versions': 'versions'}, 'hello': {'cmds': 'echo Hello', 'on_install': 'true', 'on_update': 'true', 'recipe': 'collective.recipe.cmd'}, 'versions': {}} Sections headers can contain an optional arbitrary Python expression. When the expression evaluates to false the whole section is skipped. Several sections can have the same name with different expressions, enabling conditional exclusion of sections:: [s1: 2 + 2 == 4] # this expression is true [therefore "this section" _will_ be NOT skipped a = 1 [ s2 : 2 + 2 == 5 ] # comment: this expression is false, so this section will be ignored] long = a [ s2 : 41 + 1 == 42 ] # a comment: this expression is [true], so this section will be kept long = b [s3:2 in map(lambda i:i*2, [i for i in range(10)])] ;# Complex expressions are [possible!];, though they should not be (abused:) # this section will not be skipped long = c .. -> text >>> pprint(parse(text)) {'s1': {'a': '1'}, 's2': {'long': 'b'}, 's3': {'long': 'c'}} Title line optional trailing comments are separated by a hash '#' or semicolon ';' character. The expression is an arbitrary expression with one restriction: it cannot contain a literal hash '#' or semicolon ';' character: these need to be string-escaped. The comment can contain arbitrary characters, including brackets that are also used to mark the end of a section header and may be ambiguous to recognize in some cases. For example, valid sections lines include:: [ a ] a=1 [ b ] # [] b=1 [ c : True ] # ] c =1 [ d : True] # [] d=1 [ e ] # [] e = 1 [ f ] # ] f = 1 [g:2 in map(lambda i:i*2, ['''\x23\x3b)'''] + [i for i in range(10)] + list('\x23[]][\x3b\x23'))] # Complex #expressions; ][are [possible!] and can us escaped # and ; in literals g = 1 [ h : True ] ; ] h =1 [ i : True] ; [] i=1 [j:2 in map(lambda i:i*2, ['''\x23\x3b)'''] + [i for i in range(10)] + list('\x23[]][\x3b\x23'))] ; Complex #expressions; ][are [possible!] and can us escaped # and ; in literals j = 1 .. -> text >>> pprint(parse(text)) {'a': {'a': '1'}, 'b': {'b': '1'}, 'c': {'c': '1'}, 'd': {'d': '1'}, 'e': {'e': '1'}, 'f': {'f': '1'}, 'g': {'g': '1'}, 'h': {'h': '1'}, 'i': {'i': '1'}, 'j': {'j': '1'}} A title line optional trailing comment be separated by a hash or semicolon character. The following are valid semicolon-separated comments:: [ a ] ;semicolon comment are supported for lines without expressions ] a = 1 [ b ] ; [] b = 1 [ c ] ; ] c = 1 [ d ] ; [ d = 1 [ e: True ] ;semicolon comments are supported for lines with expressions ] e = 1 .. -> text >>> pprint(parse(text)) {'a': {'a': '1'}, 'b': {'b': '1'}, 'c': {'c': '1'}, 'd': {'d': '1'}, 'e': {'e': '1'}} The following sections with hash comment separators are valid too:: [ a ] #hash comment ] are supported for lines without expressions ] a = 1 [ b ] # [] b = 1 [ c ] # ] c = 1 [ d ] # [ d = 1 [ e: True ] #hash comments] are supported for lines with expressions ] e = 1 .. -> text >>> pprint(parse(text)) {'a': {'a': '1'}, 'b': {'b': '1'}, 'c': {'c': '1'}, 'd': {'d': '1'}, 'e': {'e': '1'}} However, explicit semicolon and hash characters are invalid in expressions and must be escaped or this triggers an error. In the rare case where a hash '#' or semicolon ';' would be needed in an expression literal, you can use the string-escaped representation of these characters: use '\x23' for hash '#' and '\x3b' for semicolon ';' to avoid evaluation errors. These expressions are valid and use escaped hash and semicolons in literals:: [a:2 in map(lambda i:i*2, ['''\x23\x3b)'''] + [i for i in range(10)] + list('\x23[]][\x3b\x23'))] # Complex #expressions; ][are [possible!] and can us escaped # and ; in literals a = 1 [b:2 in map(lambda i:i*2, ['''\x23\x3b)'''] + [i for i in range(10)] + list('\x23[]][\x3b\x23'))] ; Complex #expressions; ][are [possible!] and can us escaped # and ; in literals b = 1 .. -> text >>> pprint(parse(text)) {'a': {'a': '1'}, 'b': {'b': '1'}} And using unescaped semicolon and hash characters in expressions triggers an error:: [a:'#' in '#;'] # this is not a supported expression a = 1 .. -> text >>> try: parse(text) ... except zc.buildout.configparser.MissingSectionHeaderError: pass # success One of the typical usage of expression is to have buildout parts that are operating system or platform-specific. The configparser.parse function has an optional exp_globals argument. This is a callable returning a mapping of objects made available to the evaluation context of the expression. Here we add the platform and sys modules to the evaluation context, so we can access platform and sys modules functions and objects in our expressions :: [s1: str(platform.python_version_tuple()[0]) in ('2', '3',)] # this expression is true, the major versions of python are either 2 or 3 a = 1 [s2:sys.version[0] == '0'] # comment: this expression "is false", there no major version 0 of Python so this section will be ignored long = a [s2:len(platform.uname()) > 0] # a comment: this expression is likely always true, so this section will be kept long = b .. -> text >>> import platform, sys >>> globs = lambda: {'platform': platform, 'sys': sys} >>> pprint(parse(text, exp_globals=globs)) {'s1': {'a': '1'}, 's2': {'long': 'b'}} Some limited (but hopefully sane and sufficient) default modules and pre-computed common expressions available to an expression when the parser in called by buildout:: #imported modules [s1: sys and re and os and platform] # this expression is true: these modules are available a = 1 # major and minor python versions, yes even python 3.5 and 3.6 are there , prospectively # comment: this expression "is true" and not that long expression cannot span several lines [s2: any([python2, python3, python24 , python25 , python26 , python27 , python30 , python31 , python32 , python33 , python34 , python35 , python36, python37, python38, python39, python310, python311, python312, python313, python314, python315, python316, python317, python318, python319]) ] b = 1 # common python interpreter types [s3:cpython or pypy or jython or ironpython] # a comment: this expression is likely always true, so this section will be kept c = 1 # common operating systems [s4:linux or windows or cygwin or macosx or solaris or posix or True] d = 1 # common bitness and endianness [s5:bits32 or bits64 or little_endian or big_endian] e = 1 .. -> text >>> import zc.buildout.buildout >>> pprint(parse(text, zc.buildout.buildout._default_globals)) {'s1': {'a': '1'}, 's2': {'b': '1'}, 's3': {'c': '1'}, 's4': {'d': '1'}, 's5': {'e': '1'}} Preprocessing of implication and unicode cuteness:: [foo] => part1 part2 .. -> text >>> pprint(parse(text)) {'foo': {'': 'part1 part2'}} A recent addition is support for PEP 508 markers:: [section] # These are the values when no other section overrides them. a = 1 b = 1 [section: python_version < "2.6"] a = 26 [section: python_version < "4.0"] b = 40 .. -> text >>> pprint(parse(text)) {'section': {'a': '1', 'b': '40'}} You can use the platform. This is hard to test because the tests run on various platforms. But an unknown platform should never match:: [section] # These are the values when no other section overrides them. a = 1 [section: platform_system == "msdos"] a = 2 .. -> text >>> pprint(parse(text)) {'section': {'a': '1'}} You can make combinations:: [section] # These are the values when no other section overrides them. a = 1 b = 1 [section: python_version >= "2.0" and platform_system != "msdos"] a = 2 [section: python_version >= "2.0" or platform_system == "msdos"] b = 3 .. -> text >>> pprint(parse(text)) {'section': {'a': '2', 'b': '3'}} The old and new style conditional expressions can be used in the same file:: [section] # These are the values when no other section overrides them. a = 1 b = 1 [section: python_version >= "2.0"] a = 4 [section:linux or windows or cygwin or macosx or solaris or posix or True] b = 5 .. -> text >>> pprint(parse(text, zc.buildout.buildout._default_globals)) {'section': {'a': '4', 'b': '5'}} Issue 656: Because conditional sections mean we can have the same definition twice or more in a file, the parser needs to process them in the order they occur in the file, and handle conflicts appropriately. For example, a "+=" in a non-conditional section shouldn't be overwritten by a "+=" in a following conditional section. Instead the second "+=" should be appended to the first. Likewise, an "+=" in a section *should* be overwritten by an "=" in a following conditional section:: [section] # =, += a = a b c # =, -= b = a b c # =, = c = a b c # +=, = d += a b c # +=, += e += a b c # +=, -= f += a b c # -=, = g -= a b c # -=, += h -= a b c # -=, -= i -= a b c [section:True] a += d b -= b c = x y z d = x y z e += d e f f -= a g = d e f h += d e f i -= d .. -> text >>> pprint(parse(text)) {'section': {'a': 'a\nb\nc', 'a +': 'd', 'b': 'a\nb\nc', 'b -': 'b', 'c': 'x\ny\nz', 'd': 'x\ny\nz', 'e +': 'a\nb\nc\nd\ne\nf', 'f +': 'a\nb\nc', 'f -': 'a', 'g': 'd\ne\nf', 'h +': 'd\ne\nf', 'h -': 'a\nb\nc', 'i -': 'a\nb\nc\nd'}} ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4020343 zc_buildout-6.0.1/src/zc/buildout/tests/configuration.txt0000644000076500000240000004376715264141650023301 0ustar00mauritsstaffConfiguration file syntax ------------------------- A buildout configuration file consists of a sequence of sections. A section has a *section header* followed by 0 or more *section options*. (Buildout configuration files may be viewed as a variation on INI files.) A section header consists of a section name enclosed in square braces. A section name consists of one or more non-whitespace characters other than square braces (``[``, ``]``), curly braces (``{``, ``}``), colons (``:``) or equal signs (``=``). Whitespace surrounding section names is ignored. A section header can optionally have a condition expression separated by a colon. See `Conditional sections`_. Options consist of option names, followed by optional space or tab characters, an optional plus or minus sign and an equal sign and values. An option value may be spread over multiple lines as long as the lines after the first start with a whitespace character. An option name consists of one or more non-whitespace characters other than equal signs, square braces (``[``, ``]``), curly braces (``{``, ``}``), plus signs or colons (``:``). The option name ``<`` is reserved. An option's data consists of the characters following the equal sign on the start line, plus the continuation lines. Option values have extra whitespace stripped. How this is done depends on whether the value has non-whitespace characters on the first line. If an option value has non-whitespace characters on the first line, then each line is stripped and blank lines are removed. For example, in:: [foo] bar = 1 baz = a b c .. -> text >>> from io import StringIO >>> import pprint, zc.buildout.configparser >>> pprint.pprint(zc.buildout.configparser.parse(StringIO( ... text), 'test')) {'foo': {'bar': '1', 'baz': 'a\nb\nc'}} The value of of ``bar`` is ``'1'`` and the value of ``baz`` is ``'a\nb\nc'``. If the first line of an option does **not** contain whitespace, then the value is dedented (with ``textwrap.dedent``), trailing spaces in lines are removed, and leading and trailing blank lines are removed. For example, in:: [foo] bar = baz = a b c .. -> text >>> pprint.pprint(zc.buildout.configparser.parse(StringIO( ... text), 'test')) {'foo': {'bar': '', 'baz': 'a\n b\n\nc'}} The value of bar is ``''``, and the value of baz is ``'a\n b\n\nc'``. Lines starting with ``#`` or ``;`` characters are comments. Comments can also be placed after the closing square bracket (``]``) in a section header. Buildout configuration data are Python strings, which are bytes in Python 2 and unicode in Python 3. Sections and options within sections may be repeated. Multiple occurrences of a section are treated as if they were concatenated. The last option value for a given name in a section overrides previous values. In addition to the syntactic details above: - option names are case sensitive - option values can use a substitution syntax, described below, to refer to option values in specific sections. - option values can be appended or removed using the - and + operators. Annotated sections ------------------ When used with the ``annotate`` command, buildout displays annotated sections. All sections are displayed, sorted alphabetically. For each section, all key-value pairs are displayed, sorted alphabetically, along with the origin of the value (file name or ``COMPUTED_VALUE``, ``DEFAULT_VALUE``, ``COMMAND_LINE_VALUE``):: >>> print_(system([buildout, 'annotate']), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== [buildout] allow-hosts= * DEFAULT_VALUE allow-picked-versions= true DEFAULT_VALUE allow-unknown-extras= false DEFAULT_VALUE bin-directory= bin DEFAULT_VALUE develop-eggs-directory= develop-eggs DEFAULT_VALUE directory= /sample-buildout COMPUTED_VALUE eggs-directory= /sample-buildout/eggs DEFAULT_VALUE eggs-directory-version= v5 DEFAULT_VALUE executable= ... DEFAULT_VALUE find-links= DEFAULT_VALUE install-from-cache= false DEFAULT_VALUE installed= .installed.cfg DEFAULT_VALUE log-format= DEFAULT_VALUE log-level= INFO DEFAULT_VALUE newest= true DEFAULT_VALUE offline= false DEFAULT_VALUE parts= buildout.cfg parts-directory= parts DEFAULT_VALUE prefer-final= true DEFAULT_VALUE python= buildout DEFAULT_VALUE show-picked-versions= false DEFAULT_VALUE socket-timeout= DEFAULT_VALUE update-versions-file= DEFAULT_VALUE use-dependency-links= true DEFAULT_VALUE versions= versions DEFAULT_VALUE [versions] zc.buildout = >=1.99 DEFAULT_VALUE zc.recipe.egg = >=1.99 DEFAULT_VALUE The ``annotate`` command is sensitive to the verbosity flag. You get more information about the way values are computed:: >>> print_(system([buildout, '-v', 'annotate']), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== [buildout] allow-hosts= * AS DEFAULT_VALUE SET VALUE = * allow-picked-versions= true AS DEFAULT_VALUE SET VALUE = true allow-unknown-extras= false AS DEFAULT_VALUE SET VALUE = false bin-directory= bin AS DEFAULT_VALUE SET VALUE = bin develop-eggs-directory= develop-eggs AS DEFAULT_VALUE SET VALUE = develop-eggs directory= /sample-buildout AS COMPUTED_VALUE SET VALUE = /sample-buildout eggs-directory= /sample-buildout/eggs AS DEFAULT_VALUE DIRECTORY VALUE = /sample-buildout/eggs AS DEFAULT_VALUE SET VALUE = eggs eggs-directory-version= v5 AS DEFAULT_VALUE SET VALUE = v5 executable= ... AS DEFAULT_VALUE SET VALUE = ... find-links= AS DEFAULT_VALUE SET VALUE = install-from-cache= false AS DEFAULT_VALUE SET VALUE = false installed= .installed.cfg AS DEFAULT_VALUE SET VALUE = .installed.cfg log-format= AS DEFAULT_VALUE SET VALUE = log-level= INFO AS DEFAULT_VALUE SET VALUE = INFO newest= true AS DEFAULT_VALUE SET VALUE = true offline= false AS DEFAULT_VALUE SET VALUE = false parts= IN buildout.cfg SET VALUE = parts-directory= parts AS DEFAULT_VALUE SET VALUE = parts prefer-final= true AS DEFAULT_VALUE SET VALUE = true python= buildout AS DEFAULT_VALUE SET VALUE = buildout show-picked-versions= false AS DEFAULT_VALUE SET VALUE = false socket-timeout= AS DEFAULT_VALUE SET VALUE = update-versions-file= AS DEFAULT_VALUE SET VALUE = use-dependency-links= true AS DEFAULT_VALUE SET VALUE = true verbosity= 10 AS COMMAND_LINE_VALUE SET VALUE = 10 versions= versions AS DEFAULT_VALUE SET VALUE = versions [versions] ... The output of the ``annotate`` command can be very long. You can restrict the output to some sections by passing section names as arguments:: >>> print_(system([buildout, 'annotate', 'versions']), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== [versions] zc.buildout= >=1.99 DEFAULT_VALUE zc.recipe.egg= >=1.99 DEFAULT_VALUE Query values ------------ For continuous integration, it might be useful to query the buildout config. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = . ... ... [values] ... host = buildout.org ... multiline = ... first ... second ... """) >>> print_(system([buildout, 'query', 'buildout:develop']), end='') . >>> print_(system([buildout, 'query', 'values:host']), end='') buildout.org >>> print_(system([buildout, 'query', 'values:multiline']), end='') first second As with assignments, if the section is omitted, 'buildout' section is assumed. >>> print_(system([buildout, 'query', 'develop']), end='') . When used with -v option, the query command also displays section and key. >>> print_(system([buildout, '-v', 'query', 'develop']), end='') ${buildout:develop} . >>> print_(system([buildout, '-v', 'query', 'values:host']), end='') ${values:host} buildout.org The query commands outputs proper error messages. >>> print_(system([buildout, 'query', 'versions', 'parts']), end='') Error: The query command requires a single argument. >>> print_(system([buildout, 'query']), end='') Error: The query command requires a single argument. >>> print_(system([buildout, 'query', 'invalid:section:key']), end='') Error: Invalid option: invalid:section:key >>> print_(system([buildout, '-v', 'query', 'values:port']), end='') ${values:port} Error: Key not found: port >>> print_(system([buildout, '-v', 'query', 'versionx']), end='') ${buildout:versionx} Error: Key not found: versionx >>> print_(system([buildout, '-v', 'query', 'specific:port']), end='') ${specific:port} Error: Section not found: specific Variable substitutions ---------------------- Buildout configuration files support variable substitution. To illustrate this, we'll create an debug recipe to allow us to see interactions with the buildout. See the ``debug.py`` and ``environ.py`` files in the ``tests/recipes`` directory. This recipe doesn't actually create anything. The install method doesn't return anything, because it didn't create any files or directories. We also have to update our setup script:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... entry_points = ( ... ''' ... [zc.buildout] ... mkdir = mkdir:Mkdir ... debug = debug:Debug ... environ = environ:Environ ... ''') ... setup(name="recipes", entry_points=entry_points) ... """) We've rearranged the script a bit to make the entry points easier to edit. Specifically, entry points are now defined as a configuration string, rather than a dictionary. Let's update our configuration to provide variable substitution examples:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir debug ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... File-1 = ${data-dir:path}/file ... File-2 = ${debug:File-1}/log ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) We used a string-template substitution for ``File-1`` and ``File-2``. This type of substitution uses the ``string.Template`` syntax. Names substituted are qualified option names, consisting of a section name and option name joined by a colon. Now, if we run the buildout, we'll see the options with the values substituted:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory mydata Installing debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log recipe recipes:debug Note that the substitution of the ``data-dir`` path option reflects the update to the option performed by the ``mkdir`` recipe. It might seem surprising that ``mydata`` was created again. This is because we changed our ``recipes`` package by adding the ``debug`` module. The buildout system didn't know if this module could affect the ``mkdir`` recipe, so it assumed it could and reinstalled ``mydata``. If we rerun the buildout:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating data-dir. Updating debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log recipe recipes:debug we can see that ``mydata`` was not recreated. Note that, in this case, we didn't specify a log level, so we didn't get output about what the buildout was doing. Section and option names in variable substitutions are only allowed to contain alphanumeric characters, hyphens, periods and spaces. This restriction might be relaxed in future releases. We can omit the section name in a variable substitution to refer to the current section. We can also use the special option, ``_buildout_section_name_`` to get the current section name:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir debug ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... File-1 = ${data-dir:path}/file ... File-2 = ${:File-1}/log ... my_name = ${:_buildout_section_name_} ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Updating data-dir. Installing debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log my_name debug recipe recipes:debug Automatic part selection and ordering ------------------------------------- When a section with a recipe is referred to, either through variable substitution or by an initializing recipe, the section is treated as a part and added to the part list before the referencing part. For example, we can leave ``data-dir`` out of the parts list:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... File-1 = ${data-dir:path}/file ... File-2 = ${debug:File-1}/log ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) It will still be treated as a part:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Updating data-dir. Installing debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log recipe recipes:debug >>> cat('.installed.cfg') # doctest: +ELLIPSIS [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = data-dir debug ... Note that the ``data-dir`` part is included *before* the ``debug`` part, because the ``debug`` part refers to the ``data-dir`` part. Even if we list the ``data-dir`` part after the ``debug`` part, it will be included before:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug data-dir ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... File-1 = ${data-dir:path}/file ... File-2 = ${debug:File-1}/log ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) It will still be treated as a part:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating data-dir. Updating debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log recipe recipes:debug >>> cat('.installed.cfg') # doctest: +ELLIPSIS [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = data-dir debug ... Extending sections (macros) --------------------------- A section (other than the buildout section) can extend one or more other sections using the ``<`` option. Options from the referenced sections are copied to the referring section *before* variable substitution. This, together with the ability to refer to variables of the current section, allows sections to be used as macros:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = myfiles ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... ... [with_file1] ... <= debug ... file1 = ${:path}/file1 ... color = red ... ... [with_file2] ... <= debug ... file2 = ${:path}/file2 ... color = blue ... ... [myfiles] ... <= with_file1 ... with_file2 ... path = mydata ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Uninstalling data-dir. Installing myfiles. color blue file1 mydata/file1 file2 mydata/file2 path mydata recipe recipes:debug In this example, the ``debug``, ``with_file1`` and ``with_file2`` sections act as macros. In particular, the variable substitutions are performed relative to the ``myfiles`` section. .. note:: Don't be fooled by the appearance of the ``<= section`` lines --- though ``<=`` may look like a new operator, it's still just the familiar ``key = value`` syntax. .. cleanup buildout >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = ... """) >>> _ = system(buildout) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4020996 zc_buildout-6.0.1/src/zc/buildout/tests/debugging.txt0000644000076500000240000000556615264141650022360 0ustar00mauritsstaffDebugging buildouts =================== Buildouts can be pretty complex. When things go wrong, it isn't always obvious why. Errors can occur due to problems in user input or due to bugs in zc.buildout or recipes. When an error occurs, Python's post-mortem debugger can be used to inspect the state of the buildout or recipe code were there error occurred. To enable this, use the -D option to the buildout. Let's create a recipe that has a bug: >>> mkdir(sample_buildout, 'recipes') >>> write(sample_buildout, 'recipes', 'mkdir.py', ... """ ... import os, zc.buildout ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... options['path'] = os.path.join( ... buildout['buildout']['directory'], ... options['path'], ... ) ... ... def install(self): ... directory = self.options['directory'] ... os.mkdir(directory) ... return directory ... ... def update(self): ... pass ... """) >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... ... setup(name = "recipes", ... entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']}, ... ) ... """) And create a buildout that uses it: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = mystuff ... """) If we run the buildout, we'll get an error: >>> print_(system(buildout, with_exit_code=True), end='') Develop: '/sample-buildout/recipes' Installing data-dir. While: Installing data-dir. Error: Missing option: data-dir:directory EXIT CODE: 1 If we want to debug the error, we can add the -D option. Here's we'll supply some input: >>> print_(system(buildout+" -D", """\ ... up ... p sorted(self.options.keys()) ... q ... """, with_exit_code=True), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Installing data-dir. > /zc/buildout/buildout.py(925)__getitem__() -> raise MissingOption("Missing option: %s:%s" % (self.name, key)) (Pdb) > /sample-buildout/recipes/mkdir.py(14)install() -> directory = self.options['directory'] (Pdb) ['path', 'recipe'] ...While: Installing data-dir. Traceback (most recent call last): File "/zc/buildout/buildout.py", line 1352, in main ... File "/zc/buildout/buildout.py", line 925, in __getitem__ raise MissingOption("Missing option: %s:%s" % (self.name, key)) MissingOption: Missing option: data-dir:directory Starting pdb: EXIT CODE: 1 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4021647 zc_buildout-6.0.1/src/zc/buildout/tests/dependencylinks.txt0000644000076500000240000001402515264141650023572 0ustar00mauritsstaffDependency links ---------------- By default buildout will obey the setuptools dependency_links metadata when it looks for dependencies. This behavior can be controlled with the use-dependency-links buildout option. [buildout] ... use-dependency-links = false The option defaults to true. If you set it to false, then dependency links are only looked for in the locations specified by find-links. Let's see this feature in action. To begin, let's create a new egg repository. This repository uses the same sample eggs as the normal testing repository. >>> link_server2 = start_server(sample_eggs) Turn on logging on this server so that we can see when eggs are pulled from it. >>> _ = get(link_server2 + 'enable_server_logging') GET 200 /enable_server_logging Let's create a develop egg in our buildout that specifies dependency_links which point to the new server. >>> mkdir(sample_buildout, 'depdemo') >>> write(sample_buildout, 'depdemo', 'dependencydemo.py', ... 'import eggrecipedemoneeded') >>> write(sample_buildout, 'depdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='depdemo', py_modules=['dependencydemo'], ... install_requires = 'demoneeded', ... dependency_links = ['%s'], ... zip_safe=True, version='1') ... ''' % link_server2) Now let's configure the buildout to use the develop egg. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = depdemo ... parts = eggs ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = depdemo ... ''') Now we can run the buildout. >>> print_(system(buildout), end='') GET 200 / GET 200 /demoneeded-1.1.tar.gz Develop: '/sample-buildout/depdemo' Installing eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1... Notice that the egg was retrieved from the logging server. The output may have extra lines like this, at least on setuptools 63, which we ignore: Not found: /demoneeded/ Not found: /.VolumeIcon.icns Now let's change the egg so that it doesn't specify dependency links. >>> write(sample_buildout, 'depdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='depdemo', py_modules=['dependencydemo'], ... install_requires = 'demoneeded', ... zip_safe=True, version='1') ... ''') Now we'll remove the existing dependency egg, and rerunning the buildout to see where the egg comes from this time. >>> from glob import glob >>> from os.path import join >>> def remove_demoneeded_egg(): ... for egg in glob(join(sample_buildout, 'eggs', 'v5', 'demoneeded*.egg')): ... remove(sample_buildout, 'eggs', egg) >>> remove_demoneeded_egg() >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/depdemo' Updating eggs. ... While: Updating eggs. Getting distribution for 'demoneeded'. Error: Couldn't find a distribution for 'demoneeded'. Now it can't find the dependency since neither the buildout configuration nor setup specifies where to look. Let's change things so that the buildout configuration specifies where to look for eggs. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = depdemo ... parts = eggs ... find-links = %s ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = depdemo ... ''' % link_server) >>> print_(system(buildout), end='') Develop: '/sample-buildout/depdemo' Installing eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1. This time the dependency egg was found on the server without logging configured. Now let's change things once again so that both buildout and setup specify different places to look for the dependency egg. >>> write(sample_buildout, 'depdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='depdemo', py_modules=['dependencydemo'], ... install_requires = 'demoneeded', ... dependency_links = ['%s'], ... zip_safe=True, version='1') ... ''' % link_server2) >>> remove_demoneeded_egg() >>> print_(system(buildout), end='') #doctest: +ELLIPSIS GET 200 /... Develop: '/sample-buildout/depdemo' Updating eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1... So when both setuptools and buildout specify places to search for eggs, the dependency_links takes precedence over find-links. There is a buildout option that you can specify to change this behavior. It is the use-dependency-links option. This option defaults to true. When you specify false for this option, buildout will ignore dependency_links and only look for eggs using find-links. Here is an example of using this option to disable dependency_links. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = depdemo ... parts = eggs ... find-links = %s ... use-dependency-links = false ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = depdemo ... ''' % link_server) >>> remove_demoneeded_egg() >>> print_(system(buildout), end='') Develop: '/sample-buildout/depdemo' Updating eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Notice that this time the egg isn't downloaded from the logging server. If we set the option to true, things return to the way they were before. The dependency's are looked for first in the logging server. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = depdemo ... parts = eggs ... find-links = %s ... use-dependency-links = true ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = depdemo ... ''' % link_server) >>> remove_demoneeded_egg() >>> print_(system(buildout), end='') #doctest: +ELLIPSIS GET 200 /... Develop: '/sample-buildout/depdemo' Updating eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1... ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4022396 zc_buildout-6.0.1/src/zc/buildout/tests/download.txt0000644000076500000240000004252215264141650022225 0ustar00mauritsstaffUsing the download utility ========================== The ``zc.buildout.download`` module provides a download utility that handles the details of downloading files needed for a buildout run from the internet. It downloads files to the local file system, using the download cache if desired and optionally checking the downloaded files' MD5 checksum. We setup an HTTP server that provides a file we want to download: >>> server_data = tmpdir('sample_files') >>> write(server_data, 'foo.txt', 'This is a foo text.') >>> server_url = start_server(server_data) We also use a fresh directory for temporary files in order to make sure that all temporary files have been cleaned up in the end: >>> import tempfile >>> old_tempdir = tempfile.tempdir >>> tempfile.tempdir = tmpdir('tmp') Downloading without using the cache ----------------------------------- If no download cache should be used, the download utility is instantiated without any arguments: >>> from zc.buildout.download import Download >>> download = Download() >>> print_(download.cache_dir) None Downloading a file is achieved by calling the utility with the URL as an argument. A tuple is returned that consists of the path to the downloaded copy of the file and a boolean value indicating whether this is a temporary file meant to be cleaned up during the same buildout run: >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /.../buildout-... >>> cat(path) This is a foo text. As we aren't using the download cache and haven't specified a target path either, the download has ended up in a temporary file: >>> is_temp True >>> import tempfile >>> path.startswith(tempfile.gettempdir()) True We are responsible for cleaning up temporary files behind us: >>> remove(path) When trying to access a file that doesn't exist, we'll get an exception: >>> try: download(server_url+'not-there') # doctest: +ELLIPSIS ... except: print_('download error') ... else: print_('woops') download error Downloading a local file doesn't produce a temporary file but simply returns the local file itself: >>> download(join(server_data, 'foo.txt')) ('/sample_files/foo.txt', False) We can also have the downloaded file's MD5 sum checked: >>> from hashlib import md5 >>> path, is_temp = download(server_url+'foo.txt', ... md5('This is a foo text.'.encode()).hexdigest()) >>> is_temp True >>> remove(path) >>> download(server_url+'foo.txt', ... md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt' The error message in the event of an MD5 checksum mismatch for a local file reads somewhat differently: >>> download(join(server_data, 'foo.txt'), ... md5('This is a foo text.'.encode()).hexdigest()) ('/sample_files/foo.txt', False) >>> download(join(server_data, 'foo.txt'), ... md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch for local resource at '/sample_files/foo.txt'. Finally, we can download the file to a specified place in the file system: >>> target_dir = tmpdir('download-target') >>> path, is_temp = download(server_url+'foo.txt', ... path=join(target_dir, 'downloaded.txt')) >>> print_(path) /download-target/downloaded.txt >>> cat(path) This is a foo text. >>> is_temp False Trying to download a file in offline mode will result in an error: >>> download = Download(cache=None, offline=True) >>> download(server_url+'foo.txt') Traceback (most recent call last): UserError: Couldn't download 'http://localhost/foo.txt' in offline mode. As an exception to this rule, file system paths and URLs in the ``file`` scheme will still work: >>> cat(download(join(server_data, 'foo.txt'))[0]) This is a foo text. >>> cat(download('file:' + join(server_data, 'foo.txt'))[0]) This is a foo text. >>> remove(path) Downloading using the download cache ------------------------------------ In order to make use of the download cache, we need to configure the download utility differently. To do this, we pass a directory path as the ``cache`` attribute upon instantiation: >>> cache = tmpdir('download-cache') >>> download = Download(cache=cache) >>> print_(download.cache_dir) /download-cache/ Simple usage ~~~~~~~~~~~~ When using the cache, a file will be stored in the cache directory when it is first downloaded. The file system path returned by the download utility points to the cached copy: >>> ls(cache) >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/foo.txt >>> cat(path) This is a foo text. >>> is_temp False Whenever the file is downloaded again, the cached copy is used. Let's change the file on the server to see this: >>> write(server_data, 'foo.txt', 'The wrong text.') >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/foo.txt >>> cat(path) This is a foo text. If we specify an MD5 checksum for a file that is already in the cache, the cached copy's checksum will be verified: >>> download(server_url+'foo.txt', md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch for cached download from 'http://localhost/foo.txt' at '/download-cache/foo.txt' Trying to access another file at a different URL which has the same base name will result in the cached copy being used: >>> mkdir(server_data, 'other') >>> write(server_data, 'other', 'foo.txt', 'The wrong text.') >>> path, is_temp = download(server_url+'other/foo.txt') >>> print_(path) /download-cache/foo.txt >>> cat(path) This is a foo text. Given a target path for the download, the utility will provide a copy of the file at that location both when first downloading the file and when using a cached copy: >>> remove(cache, 'foo.txt') >>> ls(cache) >>> write(server_data, 'foo.txt', 'This is a foo text.') >>> path, is_temp = download(server_url+'foo.txt', ... path=join(target_dir, 'downloaded.txt')) >>> print_(path) /download-target/downloaded.txt >>> cat(path) This is a foo text. >>> is_temp False >>> ls(cache) - foo.txt >>> remove(path) >>> write(server_data, 'foo.txt', 'The wrong text.') >>> path, is_temp = download(server_url+'foo.txt', ... path=join(target_dir, 'downloaded.txt')) >>> print_(path) /download-target/downloaded.txt >>> cat(path) This is a foo text. >>> is_temp False In offline mode, downloads from any URL will be successful if the file is found in the cache: >>> download = Download(cache=cache, offline=True) >>> cat(download(server_url+'foo.txt')[0]) This is a foo text. Local resources will be cached just like any others since download caches are sometimes used to create source distributions: >>> remove(cache, 'foo.txt') >>> ls(cache) >>> write(server_data, 'foo.txt', 'This is a foo text.') >>> download = Download(cache=cache) >>> cat(download('file:' + join(server_data, 'foo.txt'), path=path)[0]) This is a foo text. >>> ls(cache) - foo.txt >>> remove(cache, 'foo.txt') >>> cat(download(join(server_data, 'foo.txt'), path=path)[0]) This is a foo text. >>> ls(cache) - foo.txt >>> remove(cache, 'foo.txt') However, resources with checksum mismatches will not be copied to the cache: >>> download(server_url+'foo.txt', md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt' >>> ls(cache) >>> remove(path) If the file is completely missing it should notify the user of the error: >>> download(server_url+'bar.txt') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Traceback (most recent call last): ... UserError: Error downloading extends for URL http://localhost/bar.txt: ...404... >>> ls(cache) Finally, let's see what happens if the download cache to be used doesn't exist as a directory in the file system yet: >>> Download(cache=join(cache, 'non-existent'))(server_url+'foo.txt') Traceback (most recent call last): UserError: The directory: '/download-cache/non-existent' to be used as a download cache doesn't exist. Using namespace sub-directories of the download cache ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is common to store cached copies of downloaded files within sub-directories of the download cache to keep some degree of order. For example, zc.buildout stores downloaded distributions in a sub-directory named "dist". Those sub-directories are also known as namespaces. So far, we haven't specified any namespaces to use, so the download utility stored files directly inside the download cache. Let's use a namespace "test" instead: >>> download = Download(cache=cache, namespace='test') >>> print_(download.cache_dir) /download-cache/test The namespace sub-directory hasn't been created yet: >>> ls(cache) Downloading a file now creates the namespace sub-directory and places a copy of the file inside it: >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/test/foo.txt >>> ls(cache) d test >>> ls(cache, 'test') - foo.txt >>> cat(path) This is a foo text. >>> is_temp False The next time we want to download that file, the copy from inside the cache namespace is used. To see this clearly, we put a file with the same name but different content both on the server and in the cache's root directory: >>> write(server_data, 'foo.txt', 'The wrong text.') >>> write(cache, 'foo.txt', 'The wrong text.') >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/test/foo.txt >>> cat(path) This is a foo text. >>> rmdir(cache, 'test') >>> remove(cache, 'foo.txt') >>> write(server_data, 'foo.txt', 'This is a foo text.') Using a hash of the URL as the filename in the cache ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ So far, the base name of the downloaded file read from the URL has been used for the name of the cached copy of the file. This may not be desirable in some cases, for example when downloading files from different locations that have the same base name due to some naming convention, or if the file content depends on URL parameters. In such cases, an MD5 hash of the complete URL may be used as the filename in the cache: >>> download = Download(cache=cache, hash_name=True) >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/09f5793fcdc1716727f72d49519c688d >>> cat(path) This is a foo text. >>> ls(cache) - 09f5793fcdc1716727f72d49519c688d The path was printed just to illustrate matters; we cannot know the real checksum since we don't know which port the server happens to listen at when the test is run, so we don't actually know the full URL of the file. Let's check that the checksum actually belongs to the particular URL used: >>> (path.lower() == ... join(cache, md5((server_url+'foo.txt').encode()).hexdigest()).lower()) True The cached copy is used when downloading the file again: >>> write(server_data, 'foo.txt', 'The wrong text.') >>> (path, is_temp) == download(server_url+'foo.txt') True >>> cat(path) This is a foo text. >>> ls(cache) - 09f5793fcdc1716727f72d49519c688d If we change the URL, even in such a way that it keeps the base name of the file the same, the file will be downloaded again this time and put in the cache under a different name: >>> path2, is_temp = download(server_url+'other/foo.txt') >>> print_(path2) /download-cache/537b6d73267f8f4447586989af8c470e >>> path == path2 False >>> (path2.lower() == ... join(cache, md5((server_url+'other/foo.txt').encode()).hexdigest() ... ).lower()) True >>> cat(path) This is a foo text. >>> cat(path2) The wrong text. >>> ls(cache) - 09f5793fcdc1716727f72d49519c688d - 537b6d73267f8f4447586989af8c470e >>> remove(path) >>> remove(path2) >>> write(server_data, 'foo.txt', 'This is a foo text.') Using the cache purely as a fall-back ------------------------------------- Sometimes it is desirable to try downloading a file from the net if at all possible, and use the cache purely as a fall-back option when a server is down or if we are in offline mode. This mode is only in effect if a download cache is configured in the first place: >>> download = Download(cache=cache, fallback=True) >>> print_(download.cache_dir) /download-cache/ A downloaded file will be cached: >>> ls(cache) >>> path, is_temp = download(server_url+'foo.txt') >>> ls(cache) - foo.txt >>> cat(cache, 'foo.txt') This is a foo text. >>> is_temp False If the file cannot be served, the cached copy will be used: >>> remove(server_data, 'foo.txt') >>> try: Download()(server_url+'foo.txt') # doctest: +ELLIPSIS ... except: print_('download error') ... else: print_('woops') download error >>> path, is_temp = download(server_url+'foo.txt') >>> cat(path) This is a foo text. >>> is_temp False Similarly, if the file is served but we're in offline mode, we'll fall back to using the cache: >>> write(server_data, 'foo.txt', 'The wrong text.') >>> get(server_url+'foo.txt') 'The wrong text.' >>> offline_download = Download(cache=cache, offline=True, fallback=True) >>> path, is_temp = offline_download(server_url+'foo.txt') >>> print_(path) /download-cache/foo.txt >>> cat(path) This is a foo text. >>> is_temp False However, when downloading the file normally with the cache being used in fall-back mode, the file will be downloaded from the net and the cached copy will be replaced with the new content: >>> cat(download(server_url+'foo.txt')[0]) The wrong text. >>> cat(cache, 'foo.txt') The wrong text. When trying to download a resource whose checksum does not match, the cached copy will neither be used nor overwritten: >>> write(server_data, 'foo.txt', 'This is a foo text.') >>> download(server_url+'foo.txt', md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt' >>> cat(cache, 'foo.txt') The wrong text. Configuring the download utility from buildout options ------------------------------------------------------ The configuration options explained so far derive from the build logic implemented by the calling code. Other options configure the download utility for use in a particular project or buildout run; they are read from the ``buildout`` configuration section. The latter can be passed directly as the first argument to the download utility's constructor. The location of the download cache is specified by the ``download-cache`` option: >>> download = Download({'download-cache': cache}, namespace='cmmi') >>> print_(download.cache_dir) /download-cache/cmmi If the ``download-cache`` option specifies a relative path, it is understood relative to the current working directory, or to the buildout directory if that is given: >>> download = Download({'download-cache': 'relative-cache'}) >>> print_(download.cache_dir) /sample-buildout/relative-cache/ >>> download = Download({'directory': join(sample_buildout, 'root'), ... 'download-cache': 'relative-cache'}) >>> print_(download.cache_dir) /sample-buildout/root/relative-cache/ Keyword parameters take precedence over the corresponding options: >>> download = Download({'download-cache': cache}, cache=None) >>> print_(download.cache_dir) None Whether to assume offline mode can be inferred from either the ``offline`` or the ``install-from-cache`` option. As usual with zc.buildout, these options must assume one of the values 'true' and 'false': >>> download = Download({'offline': 'true'}) >>> download.offline True >>> download = Download({'offline': 'false'}) >>> download.offline False >>> download = Download({'install-from-cache': 'true'}) >>> download.offline True >>> download = Download({'install-from-cache': 'false'}) >>> download.offline False These two options are combined using logical 'or': >>> download = Download({'offline': 'true', 'install-from-cache': 'false'}) >>> download.offline True >>> download = Download({'offline': 'false', 'install-from-cache': 'true'}) >>> download.offline True The ``offline`` keyword parameter takes precedence over both the ``offline`` and ``install-from-cache`` options: >>> download = Download({'offline': 'true'}, offline=False) >>> download.offline False >>> download = Download({'install-from-cache': 'false'}, offline=True) >>> download.offline True Regressions ----------- MD5 checksum calculation needs to be reliable on all supported systems, which requires text files to be treated as binary to avoid implicit line-ending conversions: >>> text = 'First line of text.\r\nSecond line.\r\n' >>> f = open(join(server_data, 'foo.txt'), 'wb') >>> _ = f.write(text.encode()) >>> f.close() >>> path, is_temp = Download()(server_url+'foo.txt', ... md5(text.encode()).hexdigest()) >>> remove(path) When "downloading" a directory given by file-system path or ``file:`` URL and using a download cache at the same time, the cached directory wasn't handled correctly. Consequently, the cache was defeated and an attempt to cache the directory a second time broke. This is how it should work: >>> download = Download(cache=cache) >>> dirpath = join(server_data, 'some_directory') >>> mkdir(dirpath) >>> dest, _ = download(dirpath) If we now modify the source tree, the second download will produce the original one from the cache: >>> mkdir(join(dirpath, 'foo')) >>> ls(dirpath) d foo >>> dest, _ = download(dirpath) >>> ls(dest) Clean up -------- We should have cleaned up all temporary files created by downloading things: >>> ls(tempfile.tempdir) Reset the global temporary directory: >>> tempfile.tempdir = old_tempdir ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.402309 zc_buildout-6.0.1/src/zc/buildout/tests/downloadcache.txt0000644000076500000240000001644415264141650023215 0ustar00mauritsstaffUsing a download cache ====================== Normally, when distributions are installed, if any processing is needed, they are downloaded from the internet to a temporary directory and then installed from there. A download cache can be used to avoid the download step. This can be useful to reduce network access and to create source distributions of an entire buildout. The buildout download-cache option can be used to specify a directory to be used as a download cache. In this example, we'll create a directory to hold the cache: >>> cache = tmpdir('cache') And set up a buildout that downloads some eggs: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... download-cache = %(cache)s ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = demo ==0.2 ... ''' % globals()) We specified a link server that has some distributions available for download: >>> print_(get(link_server), end='') bigdemo-0.1-py3-none-any.whl
demo-0.1-py3-none-any.whl
demo-0.2-py3-none-any.whl
demo-0.3-py3-none-any.whl
demo-0.4rc1-py3-none-any.whl
demoneeded-1.0.tar.gz
demoneeded-1.1.tar.gz
demoneeded-1.2rc1.tar.gz
du_zipped-1.0-pyN.N.egg
extdemo-1.4.tar.gz
index/
mixedcase-0.5.tar.gz
other-1.0-py3-none-any.whl
We'll enable logging on the link server so we can see what's going on: >>> _ = get(link_server+'enable_server_logging') GET 200 /enable_server_logging We also specified a download cache. If we run the buildout, we'll see the eggs installed from the link server as usual: >>> print_(system(buildout), end='') GET ... GET 200 /demo-0.2-py3-none-any.whl GET 200 /demoneeded-1.1.tar.gz Installing eggs. Getting distribution for 'demo==0.2'. Got demo 0.2. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Generated script '/sample-buildout/bin/demo'. We'll also get the download cache populated. The buildout doesn't put files in the cache directly. It creates an intermediate directory, dist: >>> ls(cache) d dist >>> ls(cache, 'dist') - demo-0.2-py3-none-any.whl - demoneeded-1.1.tar.gz If we remove the installed eggs from eggs directory and re-run the buildout: >>> import os >>> for f in os.listdir(os.path.join('eggs', 'v5')): ... if f.startswith('demo'): ... remove('eggs', 'v5', f) >>> print_(system(buildout), end='') GET ... Updating eggs. Getting distribution for 'demo==0.2'. Got demo 0.2. Getting distribution for 'demoneeded'. Got demoneeded 1.1. We see that the distributions aren't downloaded, because they're downloaded from the cache. Installing solely from a download cache --------------------------------------- A download cache can be used as the basis of application source releases. In an application source release, we want to distribute an application that can be built without making any network accesses. In this case, we distribute a buildout with download cache and tell the buildout to install from the download cache only, without making network accesses. The buildout install-from-cache option can be used to signal that packages should be installed only from the download cache. Let's remove our installed eggs and run the buildout with the install-from-cache option set to true: >>> for f in os.listdir(os.path.join('eggs', 'v5')): ... if f.startswith('demo'): ... remove('eggs', 'v5', f) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... download-cache = %(cache)s ... install-from-cache = true ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') Uninstalling eggs. Installing eggs. Getting distribution for 'demo'. Got demo 0.2. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Generated script '/sample-buildout/bin/demo'. Auto-creation of download cache directory ----------------------------------------- With zc.buildout version 2.2.2 or higher the cache directory is automatically created:: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... download-cache = %(cache)s/newdir ... ''' % globals()) >>> print_(system(buildout), end='') Creating directory '/cache/newdir'. Uninstalling eggs. >>> ls(cache) d dist d newdir Using relative paths -------------------- You can use a relative path for ``download-cache`` (the same logic is applied to ``eggs-directory`` and to ``extends-cache`` too) and in such case it is considered relative to the location of the configuration file that sets its value. As an example, we create a ``base.cfg`` configuration in a different directory:: >>> basedir = tmpdir('basecfg') >>> write(basedir, 'base.cfg', ... ''' ... [buildout] ... download-cache = cache ... ''') and a ``buildout.cfg`` that extends from there:: >>> write('buildout.cfg', ... ''' ... [buildout] ... extends = %(basedir)s/base.cfg ... parts = ... ''' % globals()) >>> dummy = system(buildout) >>> ls(basedir) - base.cfg d cache Of course this cannot be used when the base configuration is not on the local filesystem because it wouldn't make any sense having a remote cache:: >>> server_data = tmpdir('server_data') >>> server_url = start_server(server_data) >>> cd(sample_buildout) >>> write(server_data, 'base.cfg', """\ ... [buildout] ... download-cache = cache ... """) >>> write('buildout.cfg', ... ''' ... [buildout] ... extends = %(server_url)s/base.cfg ... parts = ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS While: Initializing. Error: Setting "download-cache" to a non absolute location ("cache") within a remote configuration file... Though, you can create the ``download-cache`` within a nested directory, so that you can group all your generated directories (like ``eggs-directory`` or ``extends-cache`` too) within a single directory: >>> test_nested = tmpdir('test_nested') >>> cd(test_nested) >>> write('buildout.cfg', ... ''' ... [buildout] ... download-cache = ${buildout:directory}/var/cache ... eggs-directory = ${buildout:directory}/var/eggs ... parts-directory = ${buildout:directory}/var/parts ... develop-eggs-directory = ${buildout:directory}/var/develop-eggs ... ''') >>> dummy = system(buildout) >>> ls(test_nested) d bin - buildout.cfg d var >>> ls(os.path.join(test_nested, 'var')) d cache d develop-eggs d eggs d parts ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4024088 zc_buildout-6.0.1/src/zc/buildout/tests/easy_install.txt0000644000076500000240000015145615264141650023114 0ustar00mauritsstaffPython API for egg and script installation ========================================== The easy_install module provides some functions to provide support for egg and script installation. It provides functionality at the python level that is similar to easy_install, with a few exceptions: - By default, we look for new packages *and* the packages that they depend on. This is somewhat like (and uses) the --upgrade option of easy_install, except that we also upgrade required packages. - If the highest-revision package satisfying a specification is already present, then we don't try to get another one. This saves a lot of search time in the common case that packages are pegged to specific versions. - If there is a develop egg that satisfies a requirement, we don't look for additional distributions. We always give preference to develop eggs. - Distutils options for building extensions can be passed. Distribution installation ------------------------- The easy_install module provides a function, install, for installing one or more packages and their dependencies. The install function takes 2 positional arguments: - An iterable of setuptools requirement strings for the distributions to be installed, and - A destination directory to install to and to satisfy requirements from. The destination directory can be None, in which case, no new distributions are downloaded and there will be an error if the needed distributions can't be found among those already installed. It supports a number of optional keyword arguments: links A sequence of URLs, file names, or directories to look for links to distributions. index The URL of an index server, or almost any other valid URL. :) If not specified, the Python Package Index, https://pypi.org/simple/, is used. You can specify an alternate index with this option. If you use the links option and if the links point to the needed distributions, then the index can be anything and will be largely ignored. In the examples, here, we'll just point to an empty directory on our link server. This will make our examples run a little bit faster. path A list of additional directories to search for locally-installed distributions. working_set An existing working set to be augmented with additional distributions, if necessary to satisfy requirements. This allows you to call install multiple times, if necessary, to gather multiple sets of requirements. newest A boolean value indicating whether to search for new distributions when already-installed distributions meet the requirement. When this is true, the default, and when the destination directory is not None, then the install function will search for the newest distributions that satisfy the requirements. versions A dictionary mapping project names to version numbers to be used when selecting distributions. This can be used to specify a set of distribution versions independent of other requirements. use_dependency_links A flag indicating whether to search for dependencies using the setup dependency_links metadata or not. If true, links are searched for using dependency_links in preference to other locations. Defaults to true. relative_paths Adjust egg paths so they are relative to the script path. This allows scripts to work when scripts and eggs are moved, as long as they are both moved in the same way. allow_unknown_extras Install the requirements, even if one of them specifies an extra not provided by the distribution. The install method returns a working set containing the distributions needed to meet the given requirements. We have a link server that has a number of eggs: >>> print_(get(link_server), end='') bigdemo-0.1-py3-none-any.whl
demo-0.1-py3-none-any.whl
demo-0.2-py3-none-any.whl
demo-0.3-py3-none-any.whl
demo-0.4rc1-py3-none-any.whl
demoneeded-1.0.tar.gz
demoneeded-1.1.tar.gz
demoneeded-1.2rc1.tar.gz
du_zipped-1.0-pyN.N.egg
extdemo-1.4.tar.gz
index/
mixedcase-0.5.tar.gz
other-1.0-py3-none-any.whl
Let's make a directory and install the demo egg to it, using the demo: >>> dest = tmpdir('sample-install') >>> import zc.buildout.easy_install >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=[link_server], index=link_server+'index/') We requested version 0.2 of the demo distribution to be installed into the destination server. We specified that we should search for links on the link server and that we should use the (empty) link server index directory as a package index. The working set contains the distributions we retrieved. >>> for dist in ws: ... print_(dist) demoneeded 1.1 demo 0.2 We got demoneeded because it was a dependency of demo. And the actual eggs were added to the eggs directory. >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg If we remove the version restriction on demo, but specify a false value for newest, no new distributions will be installed: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... newest=False) >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg If we leave off the newest option, we'll get an update for demo: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/') >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demoneeded-1.1-py2.4.egg Note that we didn't get the newest versions available. There were release candidates for newer versions of both packages. By default, final releases are preferred. We can change this behavior using the prefer_final function: >>> zc.buildout.easy_install.prefer_final(False) True The old setting is returned. >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/') >>> for dist in ws: ... print_(dist) demoneeded 1.2rc1 demo 0.4rc1 >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demo-0.4rc1-py2.4.egg d demoneeded-1.1-py2.4.egg d demoneeded-1.2rc1-py2.4.egg Let's put the setting back to the default. >>> zc.buildout.easy_install.prefer_final(True) False We can supply additional distributions. We can also supply specifications for distributions that would normally be found via dependencies. We might do this to specify a specific version. >>> ws = zc.buildout.easy_install.install( ... ['demo', 'other', 'demoneeded==1.0'], dest, ... links=[link_server], index=link_server+'index/') >>> for dist in ws: ... print_(dist) demoneeded 1.0 other 1.0 demo 0.3 >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demo-0.4rc1-py2.4.egg d demoneeded-1.0-py2.4.egg d demoneeded-1.1-py2.4.egg d demoneeded-1.2rc1-py2.4.egg d other-1.0-py2.4.egg >>> rmdir(dest) Unknown extras -------------- Attempting to install a requirement with an extra it doesn't provide is an error. >>> ws = zc.buildout.easy_install.install( ... ['demo[unknown_extra]'], dest, links=[link_server], ... index=link_server+'index/') Traceback (most recent call last): ... UserError: Couldn't find the required extra... We can pass the ``allow_unknown_extras`` argument to force the installation to proceed. >>> ws = zc.buildout.easy_install.install( ... ['demo[unknown_extra]'], dest, links=[link_server], ... index=link_server+'index/', ... allow_unknown_extras=True) >>> ls(dest) d demo-0.3-py2.4.egg >>> rmdir(dest) Case issues ----------- Let's install an egg with case naming issues. Specifically, the sdist file is lower case while the name of the package is uppercase. Let's enable server logging to check that the lower case file is downloaded. >>> _ = get(link_server + 'enable_server_logging') GET 200 /enable_server_logging >>> ws = zc.buildout.easy_install.install( ... ['MIXEDCASE'], dest, ... links=[link_server], index=link_server+'index/') GET 404 /index/mixedcase/ GET 200 /mixedcase-0.5.tar.gz GET 200 /demoneeded-1.1.tar.gz Let's check that the uppercase dist is installed. Depending on the setuptools version, the name is reported either in all lowercase or in the original uppercase. So we compare lowercase. >>> for dist in ws: ... print_(str(dist).lower()) demoneeded 1.1 mixedcase 0.5 >>> ls(dest, lowercase_and_sort_output=True) d demoneeded-1.1-py2.4.egg d mixedcase-0.5-pyN.N.egg And cleanup. >>> _ = get(link_server + 'disable_server_logging') >>> rmdir(dest) Specifying version information independent of requirements ---------------------------------------------------------- Sometimes it's useful to specify version information independent of normal requirements specifications. For example, a buildout may need to lock down a set of versions, without having to put put version numbers in setup files or part definitions. If a dictionary is passed to the install function, mapping project names to version numbers, then the versions numbers will be used. >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... versions = dict(demo='0.2', demoneeded='1.0')) >>> [d.version for d in ws] ['1.0', '0.2'] In this example, we specified a version for demoneeded, even though we didn't define a requirement for it. The versions specified apply to dependencies as well as the specified requirements. If we specify a version that's incompatible with a requirement, then we'll get an error: >>> from zope.testing.loggingsupport import InstalledHandler >>> handler = InstalledHandler('zc.buildout.easy_install') >>> import logging >>> logging.getLogger('zc.buildout.easy_install').propagate = False >>> ws = zc.buildout.easy_install.install( ... ['demo >0.2'], dest, links=[link_server], ... index=link_server+'index/', ... versions = dict(demo='0.2', demoneeded='1.0')) Traceback (most recent call last): ... IncompatibleConstraintError: The requirement ('demo>0.2') is not allowed by your [versions] constraint (0.2) >>> print_(handler) zc.buildout.easy_install DEBUG Installing 'demo >0.2'. zc.buildout.easy_install INFO Version and requirements information containing demo: [versions] constraint on demo: 0.2 Base installation request: 'demo >0.2' >>> handler.clear() If no versions are specified, a debugging message will be output reporting that a version was picked automatically: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... ) >>> print_(handler) # doctest: +ELLIPSIS zc.buildout.easy_install DEBUG Installing 'demo'. zc.buildout.easy_install INFO Getting distribution for 'demo'. zc.buildout.easy_install DEBUG Fetching demo 0.3 from: http://.../demo-0.3-py3-none-any.whl zc.buildout.easy_install DEBUG Turning dist demo 0.3 (.../demo-0.3-py3-none-any.whl) into egg, and moving to eggs dir /sample-install). zc.buildout.easy_install DEBUG Calling pip install for .whl on .../demo-0.3-py3-none-any.whl zc.buildout.easy_install DEBUG Running pip install:... zc.buildout.easy_install DEBUG Egg for demo 0.3 installed at .../demo-0.3-pyN.N.egg zc.buildout.easy_install INFO Got demo 0.3. zc.buildout.easy_install DEBUG Picked: demo = 0.3 zc.buildout.easy_install DEBUG Getting required 'demoneeded' zc.buildout.easy_install DEBUG required by demo 0.3. zc.buildout.easy_install INFO Getting distribution for 'demoneeded'. zc.buildout.easy_install DEBUG Fetching demoneeded 1.1 from: http://.../demoneeded-1.1.tar.gz zc.buildout.easy_install DEBUG Turning dist demoneeded 1.1 (.../demoneeded-1.1.tar.gz) into egg, and moving to eggs dir /sample-install). zc.buildout.easy_install DEBUG Calling pip install for .gz on .../demoneeded-1.1.tar.gz zc.buildout.easy_install DEBUG Running pip install:... zc.buildout.easy_install DEBUG Egg for demoneeded 1.1 installed at .../demoneeded-1.1-pyN.N.egg zc.buildout.easy_install INFO Got demoneeded 1.1. zc.buildout.easy_install DEBUG Picked: demoneeded = 1.1 >>> handler.uninstall() >>> logging.getLogger('zc.buildout.easy_install').propagate = True We can request that we get an error if versions are picked: >>> zc.buildout.easy_install.allow_picked_versions(False) True (The old setting is returned.) >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... ) Traceback (most recent call last): ... UserError: Picked: demo = 0.3 The `demo` egg does not have a version pin and `allow-picked-versions = false`. To resolve this, add demo = 0.3 to the [versions] section, OR set `allow-picked-versions = true`. >>> zc.buildout.easy_install.allow_picked_versions(True) False The function default_versions can be used to get and set default version information to be used when no version information is passes. If called with an argument, it sets the default versions: >>> zc.buildout.easy_install.default_versions(dict(demoneeded='1')) ... # doctest: +ELLIPSIS {...} It always returns the previous default versions. If called without an argument, it simply returns the default versions without changing them: >>> zc.buildout.easy_install.default_versions() {'demoneeded': '1'} So with the default versions set, we'll get the requested version even if the versions option isn't used: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... ) >>> [d.version for d in ws] ['0.3', '1.0'] Of course, we can unset the default versions by passing an empty dictionary: >>> zc.buildout.easy_install.default_versions({}) {'demoneeded': '1'} >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... ) >>> [d.version for d in ws] ['0.3', '1.1'] Dependency links ---------------- Setuptools allows metadata that describes where to search for package dependencies. This option is called dependency_links. Buildout has its own notion of where to look for dependencies, but it also uses the setup tools dependency_links information if it's available. Let's demo this by creating an egg that specifies dependency_links. To begin, let's create a new egg repository. This repository hold a newer version of the 'demoneeded' egg than the sample repository does. >>> repoloc = tmpdir('repo') >>> from zc.buildout.tests import create_wheel >>> create_wheel('demoneeded', '1.2', repoloc) >>> link_server2 = start_server(repoloc) Turn on logging on this server so that we can see when eggs are pulled from it. >>> _ = get(link_server2 + 'enable_server_logging') GET 200 /enable_server_logging Now we can create an egg that specifies that its dependencies are found on this server. >>> repoloc = tmpdir('repo2') >>> create_wheel('hasdeps', '1.0', repoloc, ... install_requires = "'demoneeded'", ... dependency_links = [link_server2]) Let's add the egg to another repository. >>> link_server3 = start_server(repoloc) Now let's install the egg. >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, ... links=[link_server3], index=link_server3+'index/') GET 200 / GET 200 /demoneeded-1.2-py3-none-any.whl The server logs show that the dependency was retrieved from the server specified in the dependency_links. Now let's see what happens if we provide two different ways to retrieve the dependencies. >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3]) GET 200 / GET 200 /demoneeded-1.2-py3-none-any.whl Once again the dependency is fetched from the logging server even though it is also available from the non-logging server. This is because the version on the logging server is newer and buildout normally chooses the newest egg available. If you wish to control where dependencies come from regardless of dependency_links setup metadata use the 'use_dependency_links' option to zc.buildout.easy_install.install(). >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3], ... use_dependency_links=False) Notice that this time the dependency egg is not fetched from the logging server. When you specify not to use dependency_links, eggs will only be searched for using the links you explicitly provide. Another way to control this option is with the zc.buildout.easy_install.use_dependency_links() function. This function sets the default behavior for the zc.buildout.easy_install() function. >>> zc.buildout.easy_install.use_dependency_links(False) True The function returns its previous setting. >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3]) It can be overridden by passing a keyword argument to the install function. >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3], ... use_dependency_links=True) GET 200 /demoneeded-1.2-py3-none-any.whl To return the dependency_links behavior to normal call the function again. >>> zc.buildout.easy_install.use_dependency_links(True) False >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3]) GET 200 /demoneeded-1.2-py3-none-any.whl Script generation ----------------- The easy_install module provides support for creating scripts from eggs. It provides a function similar to setuptools except that it provides facilities for baking a script's path into the script. This has two advantages: - The eggs to be used by a script are not chosen at run time, making startup faster and, more importantly, deterministic. - The script doesn't have to import pkg_resources because the logic that pkg_resources would execute at run time is executed at script-creation time. The scripts method can be used to generate scripts. Let's create a destination directory for it to place them in: >>> import tempfile >>> bin = tmpdir('bin') Now, we'll use the scripts method to generate scripts in this directory from the demo egg: >>> import sys >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin) the three arguments we passed were: 1. A sequence of distribution requirements. These are of the same form as setuptools requirements. Here we passed a single requirement, for the version 0.1 demo distribution. 2. A working set, 3. The destination directory. The bin directory now contains a generated script: >>> ls(bin) - demo The return value is a list of the scripts generated: >>> import os, sys >>> if sys.platform == 'win32': ... scripts == [os.path.join(bin, 'demo.exe'), ... os.path.join(bin, 'demo-script.py')] ... else: ... scripts == [os.path.join(bin, 'demo')] True Note that in Windows, 2 files are generated for each script. A script file, ending in '-script.py', and an exe file that allows the script to be invoked directly without having to specify the Python interpreter and without having to provide a '.py' suffix. The demo script run the entry point defined in the demo egg: >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main()) Some things to note: - The demo and demoneeded eggs are added to the beginning of sys.path. - The module for the script entry point is imported and the entry point, in this case, 'main', is run. Rather than requirement strings, you can pass tuples containing 3 strings: - A script name, - A module, - An attribute expression for an entry point within the module. For example, we could have passed entry point information directly rather than passing a requirement: >>> scripts = zc.buildout.easy_install.scripts( ... [('demo', 'eggrecipedemo', 'main')], ws, ... sys.executable, bin) >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main()) Passing entry-point information directly is handy when using eggs (or distributions) that don't declare their entry points, such as distributions that aren't based on setuptools. The interpreter keyword argument can be used to generate a script that can be used to invoke the Python interactive interpreter with the path set based on the working set. This generated script can also be used to run other scripts with the path set on the working set: >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, interpreter='py') >>> ls(bin) - demo - py >>> if sys.platform == 'win32': ... scripts == [os.path.join(bin, 'demo.exe'), ... os.path.join(bin, 'demo-script.py'), ... os.path.join(bin, 'py.exe'), ... os.path.join(bin, 'py-script.py')] ... else: ... scripts == [os.path.join(bin, 'demo'), ... os.path.join(bin, 'py')] True The py script simply runs the Python interactive interpreter with the path set: >>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-pyN.N.egg', '/sample-install/demoneeded-1.1-pyN.N.egg', ] _interactive = True if len(sys.argv) > 1: # The Python interpreter wrapper allows only some of the options that a # "regular" Python interpreter accepts. _options, _args = __import__("getopt").getopt(sys.argv[1:], 'Iic:m:') _interactive = False for (_opt, _val) in _options: if _opt == '-i': _interactive = True elif _opt == '-c': exec(_val) elif _opt == '-m': sys.argv[1:] = _args _args = [] __import__("runpy").run_module( _val, {}, "__main__", alter_sys=True) elif _opt == '-I': # Allow yet silently ignore the `-I` option. The original behaviour # for this option is to create an isolated Python runtime. It was # deemed acceptable to allow the option here as this Python wrapper # is isolated from the system Python already anyway. # The specific use-case that led to this change is how the Python # language extension for Visual Studio Code calls the Python # interpreter when initializing the extension. pass if _args: sys.argv[:] = _args __file__ = _args[0] del _options, _args with open(__file__, 'U') as __file__f: exec(compile(__file__f.read(), __file__, "exec")) if _interactive: del _interactive __import__("code").interact(banner="", local=globals()) If invoked with a script name and arguments, it will run that script, instead. >>> write('ascript', r''' ... "demo doc" ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\n') ... print_(sys.argv) ... print_((__name__, __file__, __doc__)) ... ''') >>> print_(system(join(bin, 'py')+' ascript a b c'), end='') ['ascript', 'a', 'b', 'c'] ('__main__', 'ascript', 'demo doc') You can also use the -m option to run a module: >>> print_(system(join(bin, 'py')+' -m pdb'), end='') usage: ...pdb... >>> print_(system(join(bin, 'py')+' -m pdb what'), end='') # if you have pdbpp installed, this may fail. Error: what does not exist An interpreter can also be generated without other eggs: >>> scripts = zc.buildout.easy_install.scripts( ... [], [], sys.executable, bin, interpreter='py') >>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ ] ... An additional argument can be passed to define which scripts to install and to provide script names. The argument is a dictionary mapping original script names to new script names. >>> bin = tmpdir('bin2') >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, dict(demo='run')) >>> if sys.platform == 'win32': ... scripts == [os.path.join(bin, 'run.exe'), ... os.path.join(bin, 'run-script.py')] ... else: ... scripts == [os.path.join(bin, 'run')] True >>> ls(bin) - run >>> print_(system(os.path.join(bin, 'run')), end='') 3 1 The scripts that are generated are made executable: >>> if sys.platform == 'win32': ... os.access(os.path.join(bin, 'run.exe'), os.X_OK) ... else: ... os.access(os.path.join(bin, 'run'), os.X_OK) True Including extra paths in scripts -------------------------------- We can pass a keyword argument, extra paths, to cause additional paths to be included in the a generated script: >>> foo = tmpdir('foo') >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, dict(demo='run'), ... extra_paths=[foo]) >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', '/foo', ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main()) Providing script arguments -------------------------- An "argument" keyword argument can be used to pass arguments to an entry point. The value passed is a source string to be placed between the parentheses in the call: >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, dict(demo='run'), ... arguments='1, 2') >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main(1, 2)) Passing initialization code --------------------------- You can also pass script initialization code: >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, dict(demo='run'), ... arguments='1, 2', ... initialization='import os\nos.chdir("foo")', ... interpreter='py') >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', ] import os os.chdir("foo") import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main(1, 2)) It will be included in interpreters too: >>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py3.3.egg', '/sample-install/demoneeded-1.1-py3.3.egg', ] import os os.chdir("foo") _interactive = True ... Relative paths -------------- Sometimes, you want to be able to move a buildout directory around and have scripts still work without having to rebuild them. We can control this using the relative_paths option to install. You need to pass a common base directory of the scripts and eggs: >>> bo = tmpdir('bo') >>> ba = tmpdir('ba') >>> mkdir(bo, 'eggs') >>> mkdir(bo, 'bin') >>> mkdir(bo, 'other') >>> ws = zc.buildout.easy_install.install( ... ['demo'], join(bo, 'eggs'), links=[link_server], ... index=link_server+'index/') >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, join(bo, 'bin'), dict(demo='run'), ... extra_paths=[ba, join(bo, 'bar'), bo], ... interpreter='py', ... relative_paths=bo) >>> cat(bo, 'bin', 'run') #!/usr/local/bin/python2.7 import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) base = os.path.dirname(base) import sys sys.path[0:0] = [ join(base, 'eggs/demoneeded-1.1-pyN.N.egg'), join(base, 'eggs/demo-0.3-pyN.N.egg'), '/ba', join(base, 'bar'), base, ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main()) Note that the extra path we specified that was outside the directory passed as relative_paths wasn't converted to a relative path. Of course, running the script works: >>> print_(system(join(bo, 'bin', 'run')), end='') 3 1 We specified an interpreter and its paths are adjusted too: >>> cat(bo, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF #!/usr/local/bin/python2.7 import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) base = os.path.dirname(base) import sys sys.path[0:0] = [ join(base, 'eggs/demoneeded-1.1-pyN.N.egg'), join(base, 'eggs/demo-0.3-pyN.N.egg'), '/ba', join(base, 'bar'), base, ] _interactive = True if len(sys.argv) > 1: # The Python interpreter wrapper allows only some of the options that a # "regular" Python interpreter accepts. _options, _args = __import__("getopt").getopt(sys.argv[1:], 'Iic:m:') _interactive = False for (_opt, _val) in _options: if _opt == '-i': _interactive = True elif _opt == '-c': exec(_val) elif _opt == '-m': sys.argv[1:] = _args _args = [] __import__("runpy").run_module( _val, {}, "__main__", alter_sys=True) elif _opt == '-I': # Allow yet silently ignore the `-I` option. The original behaviour # for this option is to create an isolated Python runtime. It was # deemed acceptable to allow the option here as this Python wrapper # is isolated from the system Python already anyway. # The specific use-case that led to this change is how the Python # language extension for Visual Studio Code calls the Python # interpreter when initializing the extension. pass if _args: sys.argv[:] = _args __file__ = _args[0] del _options, _args with open(__file__, 'U') as __file__f: exec(compile(__file__f.read(), __file__, "exec")) if _interactive: del _interactive __import__("code").interact(banner="", local=globals()) # Installing distutils-style scripts # ---------------------------------- # TODO Since zc.buildout 4.11 we can no longer detect distutils scripts in # development eggs. And now not in normal installs either. Might be fixable, # but for now it is causing too much trouble. # Most python libraries use the console_scripts entry point nowadays. But # several still have a ``scripts=['bin/something']`` in their setup() call. # Buildout also installs those: # >>> distdir = tmpdir('distutilsscriptdir') # >>> distbin = tmpdir('distutilsscriptbin') # >>> ws = zc.buildout.easy_install.install( # ... ['other'], distdir, # ... links=[link_server], index=link_server+'index/') # >>> scripts = zc.buildout.easy_install.scripts( # ... ['other'], ws, sys.executable, distbin) # >>> ls(distbin) # - distutilsscript # Like for console_scripts, the output is a list of the scripts # generated. Likewise, on windows two files, an ``.exe`` and a script with # ``-script.py`` appended, are generated: # >>> import os, sys # >>> if sys.platform == 'win32': # ... scripts == [os.path.join(distbin, 'distutilsscript.exe'), # ... os.path.join(distbin, 'distutilsscript-script.py')] # ... else: # ... scripts == [os.path.join(distbin, 'distutilsscript')] # True # It also works for zipped eggs: # >>> distdir2 = tmpdir('distutilsscriptdir2') # >>> distbin2 = tmpdir('distutilsscriptbin2') # >>> ws = zc.buildout.easy_install.install( # ... ['du_zipped'], distdir2, # ... links=[link_server], index=link_server+'index/') # >>> scripts = zc.buildout.easy_install.scripts( # ... ['du_zipped'], ws, sys.executable, distbin2) # >>> ls(distbin2) # - distutilsscript # Distutils copies the script files verbatim, apart from a line at the top that # looks like ``#!/usr/bin/python``, which gets replaced by the actual python # interpreter. Buildout does the same, but additionally also adds the sys.path # like for the console_scripts. # >>> cat(distbin, 'distutilsscript') # #!/usr/local/bin/python2.7 # # -*- coding: utf-8 -*- # """Module docstring.""" # # # import sys # sys.path[0:0] = [ # '/distutilsscriptdir/other-1.0-pyN.N.egg', # ] # # # import os # import sys; sys.stdout.write("distutils!\n") # Note that there are several items that need to come first in such a script # *before* buildout's ``sys.path`` statements: a source encoding hint, a module # docstring and ``__future__`` imports. Buildout retains them in their proper # place by looking at the first non-future import and placing its ``sys.path`` # statement before that. # Due to the nature of distutils scripts, buildout cannot pass arguments as # there's no specific method to pass them to. # In some cases, a python 3 ``__pycache__`` directory can end up in an internal # ``EGG-INFO`` metadata directory, next to the script information we're looking # for. Buildout doesn't crash on that: # >>> eggname = [name for name in os.listdir(distdir2) # ... if name.endswith('egg')][0] # >>> scripts_metadata_dir = os.path.join( # ... distdir2, eggname, 'EGG-INFO', 'scripts') # >>> os.mkdir(os.path.join(scripts_metadata_dir, '__dummy__')) # >>> scripts = zc.buildout.easy_install.scripts( # ... ['du_zipped'], ws, sys.executable, distbin2) # >>> ls(distbin2) # - distutilsscript # Installing develop eggs sadly means that setuptools doesn't record distutils # scripts in the metadata. We try to detect such scripts anyhow: # >>> dev_distutils_dir = tmpdir('dev_distutils_dir') # >>> dev_distutils_dest = tmpdir('dev_distutils_dest') # >>> dev_eggs_dir = os.path.join(dev_distutils_dest, 'develop-eggs') # >>> bin_dir = os.path.join(dev_distutils_dest, 'bin') # >>> os.mkdir(dev_eggs_dir) # >>> os.mkdir(bin_dir) # >>> write(dev_distutils_dir, 'distutilsscript2', # ... '#!/usr/bin/python\n' # ... '# -*- coding: utf-8 -*-\n' # ... '"""Module docstring."""\n' # ... 'import os\n' # ... 'import sys; sys.stdout.write("distutils!\\n")\n' # ... ) # >>> write(dev_distutils_dir, 'setup.py', # ... ''' # ... from setuptools import setup # ... setup(name="foo2", # ... scripts=['distutilsscript2']) # ... ''') # >>> zc.buildout.easy_install.develop( # ... dev_distutils_dir, dev_eggs_dir) # '/dev_distutils_dest/develop-eggs/foo2.egg-link' # >>> ws = zc.buildout.easy_install.working_set( # ... ['foo2'], sys.executable, [dev_eggs_dir]) # >>> scripts = zc.buildout.easy_install.scripts( # ... ['foo2'], ws, sys.executable, dest=bin_dir) # >>> from zc.buildout.utils import IS_SETUPTOOLS_80_PLUS # >>> if IS_SETUPTOOLS_80_PLUS: # ... # distutils scripts are no longer detected here # ... True # ... elif sys.platform == 'win32': # ... scripts == [os.path.join(dev_distutils_dest, 'bin', 'distutilsscript2.exe'), # ... os.path.join(dev_distutils_dest, 'bin', 'distutilsscript2-script.py')] # ... else: # ... scripts == [os.path.join(dev_distutils_dest, 'bin', 'distutilsscript2')] # True Handling custom build options for extensions provided in source distributions ----------------------------------------------------------------------------- Sometimes, we need to control how extension modules are built. The build function provides this level of control. It takes a single package specification, downloads a source distribution, and builds it with specified custom build options. The build function takes 3 positional arguments: spec A package specification for a source distribution dest A destination directory build_ext A dictionary of options to be passed to the distutils build_ext command when building extensions. It supports a number of optional keyword arguments: links a sequence of URLs, file names, or directories to look for links to distributions, index The URL of an index server, or almost any other valid URL. :) If not specified, the Python Package Index, https://pypi.org/simple/, is used. You can specify an alternate index with this option. If you use the links option and if the links point to the needed distributions, then the index can be anything and will be largely ignored. In the examples, here, we'll just point to an empty directory on our link server. This will make our examples run a little bit faster. path A list of additional directories to search for locally-installed distributions. newest A boolean value indicating whether to search for new distributions when already-installed distributions meet the requirement. When this is true, the default, and when the destination directory is not None, then the install function will search for the newest distributions that satisfy the requirements. versions A dictionary mapping project names to version numbers to be used when selecting distributions. This can be used to specify a set of distribution versions independent of other requirements. Our link server included a source distribution that includes a simple extension, extdemo.c:: #include #include static PyMethodDef methods[] = {}; PyMODINIT_FUNC initextdemo(void) { PyObject *m; m = Py_InitModule3("extdemo", methods, ""); #ifdef TWO PyModule_AddObject(m, "val", PyInt_FromLong(2)); #else PyModule_AddObject(m, "val", PyInt_FromLong(EXTDEMO)); #endif } The extension depends on a system-dependent include file, extdemo.h, that defines a constant, EXTDEMO, that is exposed by the extension. We'll add an include directory to our sample buildout and add the needed include file to it: >>> mkdir('include') >>> write('include', 'extdemo.h', ... """ ... #define EXTDEMO 42 ... """) Now, we can use the build function to create an egg from the source distribution: >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/') ['/sample-install/extdemo-1.4-py2.4-unix-i686.egg'] The function returns the list of eggs Now if we look in our destination directory, we see we have an extdemo egg: >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demoneeded-1.0-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.4-py2.4-unix-i686.egg Let's update our link server with a new version of extdemo: >>> update_extdemo() >>> print_(get(link_server), end='') bigdemo-0.1-py3-none-any.whl
demo-0.1-py3-none-any.whl
demo-0.2-py3-none-any.whl
demo-0.3-py3-none-any.whl
demo-0.4rc1-py3-none-any.whl
demoneeded-1.0.tar.gz
demoneeded-1.1.tar.gz
demoneeded-1.2rc1.tar.gz
du_zipped-1.0-pyN.N.egg
extdemo-1.4.tar.gz
extdemo-1.5.tar.gz
index/
mixedcase-0.5.tar.gz
other-1.0-py3-none-any.whl
The easy_install caches information about servers to reduce network access. To see the update, we have to call the clear_index_cache function to clear the index cache: >>> zc.buildout.easy_install.clear_index_cache() If we run build with newest set to False, we won't get an update: >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/', ... newest=False) ['/sample-install/extdemo-1.4-py2.4-linux-i686.egg'] >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demoneeded-1.0-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.4-py2.4-unix-i686.egg But if we run it with the default True setting for newest, then we'll get an updated egg: >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/') ['/sample-install/extdemo-1.5-py2.4-unix-i686.egg'] >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demoneeded-1.0-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.4-py2.4-unix-i686.egg d extdemo-1.5-py2.4-unix-i686.egg The versions option also influences the versions used. For example, if we specify a version for extdemo, then that will be used, even though it isn't the newest. Let's clean out the destination directory first: >>> import os >>> for name in os.listdir(dest): ... remove(dest, name) >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/', ... versions=dict(extdemo='1.4')) ['/sample-install/extdemo-1.4-py2.4-unix-i686.egg'] >>> ls(dest) d extdemo-1.4-py2.4-unix-i686.egg Handling custom build options for extensions in develop eggs ------------------------------------------------------------ The develop function is similar to the build function, except that, rather than building an egg from a source directory containing a setup.py script. The develop function takes 2 positional arguments: setup The path to a setup script, typically named "setup.py", or a directory containing a setup.py script. dest The directory to install the egg link to It supports some optional keyword argument: build_ext A dictionary of options to be passed to the distutils build_ext command when building extensions. We have a local directory containing the extdemo source: >>> contents = os.listdir(extdemo) >>> 'MANIFEST.in' in contents True >>> 'README' in contents True >>> 'extdemo.c' in contents True >>> 'setup.py' in contents True Now, we can use the develop function to create a develop egg from the source distribution: >>> zc.buildout.easy_install.develop( ... extdemo, dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}) '/sample-install/extdemo.egg-link' The name of the egg link created is returned. Now if we look in our destination directory, we see we have an extdemo egg link: >>> ls(dest) d extdemo-1.4-py2.4-unix-i686.egg - extdemo.egg-link And that the source directory contains the compiled extension: >>> contents = os.listdir(extdemo) >>> bool([f for f in contents if f.endswith('.so') or f.endswith('.pyd')]) True Download cache -------------- Normally, when distributions are installed, if any processing is needed, they are downloaded from the internet to a temporary directory and then installed from there. A download cache can be used to avoid the download step. This can be useful to reduce network access and to create source distributions of an entire buildout. A download cache is specified by calling the download_cache function. The function always returns the previous setting. If no argument is passed, then the setting is unchanged. If an argument is passed, the download cache is set to the given path, which must point to an existing directory. Passing None clears the cache setting. To see this work, we'll create a directory and set it as the cache directory: >>> cache = tmpdir('cache') >>> zc.buildout.easy_install.download_cache(cache) We'll recreate our destination directory: >>> remove(dest) >>> dest = tmpdir('sample-install') We'd like to see what is being fetched from the server, so we'll enable server logging: >>> _ = get(link_server+'enable_server_logging') GET 200 /enable_server_logging Now, if we install demo, and extdemo: >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=[link_server], index=link_server+'index/') GET 200 / GET 404 /index/demo/ GET 200 /index/ GET 200 /demo-0.2-py3-none-any.whl GET 404 /index/demoneeded/ GET 200 /demoneeded-1.1.tar.gz >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/') GET 404 /index/extdemo/ GET 200 /extdemo-1.5.tar.gz ['/sample-install/extdemo-1.5-py2.4-linux-i686.egg'] Not only will we get eggs in our destination directory: >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.5-py2.4-linux-i686.egg But we'll get distributions in the cache directory: >>> ls(cache) - demo-0.2-py3-none-any.whl - demoneeded-1.1.tar.gz - extdemo-1.5.tar.gz The cache directory contains uninstalled distributions, such as zipped eggs or source distributions. Let's recreate our destination directory and clear the index cache: >>> remove(dest) >>> dest = tmpdir('sample-install') >>> zc.buildout.easy_install.clear_index_cache() Now when we install the distributions: >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=[link_server], index=link_server+'index/') GET 200 / GET 404 /index/demo/ GET 200 /index/ GET 404 /index/demoneeded/ >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/') GET 404 /index/extdemo/ ['/sample-install/extdemo-1.5-py2.4-linux-i686.egg'] >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.5-py2.4-linux-i686.egg Note that we didn't download the distributions from the link server. If we remove the restriction on demo, we'll download a newer version from the link server: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, ... links=[link_server], index=link_server+'index/') GET 200 /demo-0.3-py3-none-any.whl Normally, the download cache is the preferred source of downloads, but not the only one. Installing solely from a download cache --------------------------------------- A download cache can be used as the basis of application source releases. In an application source release, we want to distribute an application that can be built without making any network accesses. In this case, we distribute a download cache and tell the easy_install module to install from the download cache only, without making network accesses. The install_from_cache function can be used to signal that packages should be installed only from the download cache. The function always returns the previous setting. Calling it with no arguments returns the current setting without changing it: >>> zc.buildout.easy_install.install_from_cache() False Calling it with a boolean value changes the setting and returns the previous setting: >>> zc.buildout.easy_install.install_from_cache(True) False Let's remove demo-0.3-py2.4.egg from the cache, clear the index cache, recreate the destination directory, and reinstall demo: >>> for f in os.listdir(cache): ... if f.startswith('demo-0.3-'): ... remove(cache, f) >>> zc.buildout.easy_install.clear_index_cache() >>> remove(dest) >>> dest = tmpdir('sample-install') >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, ... links=[link_server], index=link_server+'index/') >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg This time, we didn't download from or even query the link server. .. Disable the download cache: >>> zc.buildout.easy_install.download_cache(None) '/cache' >>> zc.buildout.easy_install.install_from_cache(False) True Environment markers ------------------- Specifications can include PEP 496 environment markers. >>> _ = get(link_server + 'disable_server_logging') >>> spec = ["demo ==0.1; python_version < '3.10'", ... "demo == 0.2; python_version >= '3.10'"] >>> ws = zc.buildout.easy_install.install( ... spec, dest, links=[link_server], index=link_server+'index/') >>> demo_version = None >>> for egg in ws: ... if egg.project_name == 'demo': ... demo_version = egg.version >>> demo_version is not None True >>> if (sys.version_info.minor < 10): ... demo_version ... else: ... '0.1' '0.1' >>> if (sys.version_info.minor >= 10): ... demo_version ... else: ... '0.2' '0.2' Conflicts properly with version spec. >>> ws = zc.buildout.easy_install.install( ... spec, dest, links=[link_server], index=link_server+'index/', ... versions = dict(demo='0.3')) Traceback (most recent call last): ... zc.buildout.easy_install.IncompatibleConstraintError: The requirement ('demo==0... ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.402506 zc_buildout-6.0.1/src/zc/buildout/tests/extending.txt0000644000076500000240000004732215264141650022406 0ustar00mauritsstaffExtending sections (macros) - Adding and removing options --------------------------------------------------------- .. >>> import os We can also add and remove options in extended sections. This is illustrated below; first we define a base configuration. >>> write(sample_buildout, 'base.cfg', ... """ ... [buildout] ... parts = part1 part2 part3 ... ... [part1] ... recipe = ... option = a1 ... a2 ... ... [part2] ... <= part1 ... option -= a1 ... option += c3 c4 ... ... [part3] ... <= part2 ... option += d2 ... c5 d1 d6 ... option -= a2 ... """) To verify that the options are adjusted correctly, we'll set up an extension that prints out the options. >>> mkdir(sample_buildout, 'demo') >>> write(sample_buildout, 'demo', 'demo.py', ... """ ... import sys ... def ext(buildout): ... sys.stdout.write(str( ... [part['option'] for name, part in sorted(buildout.items()) ... if name.startswith('part')])+'\\n') ... """) >>> write(sample_buildout, 'demo', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name="demo", ... entry_points={'zc.buildout.extension': ['ext = demo:ext']}, ... ) ... """) Set up a buildout configuration for this extension. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... parts = ... """) >>> os.chdir(sample_buildout) >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/demo' Verify option values. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... extensions = demo ... extends = base.cfg ... """) >>> print_(system(os.path.join('bin', 'buildout')), end='') ['a1/na2', 'a2/nc3 c4', 'c3 c4/nd2/nc5 d1 d6'] Develop: '/sample-buildout/demo' Cleanup. >>> os.remove(os.path.join(sample_buildout, 'base.cfg')) >>> rmdir(sample_buildout, 'demo') Adding and removing options --------------------------- We can append and remove values to an option by using the ``+`` and ``-`` operators. This is illustrated below; first we define a base configuration:: >>> write(sample_buildout, 'base.cfg', ... """ ... [buildout] ... parts = part1 part2 part3 ... ... [part1] ... recipe = ... option = a1 a2 ... ... [part2] ... recipe = ... option = b1 b2 b3 b4 ... ... [part3] ... recipe = ... option = c1 c2 ... ... [part4] ... recipe = ... option = d2 ... d3 ... d5 ... ... # Issue #641 - Properly handle options which are initially defined ... # using += / -= ... [part6] ... option += e1 ... ... [part7] ... option -= f1 ... ... """) Extending this configuration, we can "adjust" the values set in the base configuration file:: >>> write(sample_buildout, 'extension1.cfg', ... """ ... [buildout] ... extends = base.cfg ... ... # appending values ... [part1] ... option += a3 a4 ... ... # removing values ... [part2] ... option -= b1 b2 ... ... # alt. spelling ... [part3] ... option+=c3 c4 c5 ... ... # combining both adding and removing ... [part4] ... option += d1 ... d4 ... option -= d5 ... ... # normal assignment ... [part5] ... option = h1 h2 ... ... """) An additional extension:: >>> write(sample_buildout, 'extension2.cfg', ... """ ... [buildout] ... extends = extension1.cfg ... ... # appending values ... [part1] ... option += a5 ... ... # removing values ... [part2] ... option -= b1 b2 b3 ... ... """) To verify that the options are adjusted correctly, we'll set up an extension that prints out the options:: >>> mkdir(sample_buildout, 'demo') >>> write(sample_buildout, 'demo', 'demo.py', ... """ ... import sys ... def ext(buildout): ... sys.stdout.write(str( ... [part.get('option') for name, part in sorted(buildout.items()) ... if name.startswith('part')])+'\\n') ... """) >>> write(sample_buildout, 'demo', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name="demo", ... entry_points={'zc.buildout.extension': ['ext = demo:ext']}, ... ) ... """) Set up a buildout configuration for this extension:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... parts = ... """) >>> os.chdir(sample_buildout) >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), end='') ... # doctest: +ELLIPSIS Develop: '/sample-buildout/demo'... Verify option values:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... extensions = demo ... extends = extension2.cfg ... """) >>> print_(system(os.path.join('bin', 'buildout')), end='') ['a1 a2/na3 a4/na5', 'b1 b2 b3 b4', 'c1 c2/nc3 c4 c5', 'd2/nd3/nd1/nd4', 'h1 h2', 'e1', ''] Develop: '/sample-buildout/demo' Annotated sections output shows which files are responsible for which operations:: >>> print_(system(os.path.join('bin', 'buildout') + ' annotate'), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== ... [part1] option= a1 a2 a3 a4 a5 base.cfg += extension1.cfg += extension2.cfg recipe= base.cfg [part2] option= b1 b2 b3 b4 base.cfg -= extension1.cfg -= extension2.cfg recipe= base.cfg [part3] option= c1 c2 c3 c4 c5 base.cfg += extension1.cfg recipe= base.cfg [part4] option= d2 d3 d1 d4 base.cfg += extension1.cfg -= extension1.cfg recipe= base.cfg [part5] option= h1 h2 extension1.cfg [part6] option= e1 base.cfg [part7] option= base.cfg -= IMPLICIT_VALUE [versions] zc.buildout= >=1.99 DEFAULT_VALUE zc.recipe.egg= >=1.99 DEFAULT_VALUE With more verbosity:: >>> print_(system(os.path.join('bin', 'buildout') + ' -v annotate'), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== ... [part1] option= a1 a2 a3 a4 a5 IN extension2.cfg ADD VALUE = a5 IN extension1.cfg ADD VALUE = a3 a4 IN base.cfg SET VALUE = a1 a2 ... [part2] option= b1 b2 b3 b4 IN extension2.cfg REMOVE VALUE = b1 b2 b3 IN extension1.cfg REMOVE VALUE = b1 b2 IN base.cfg SET VALUE = b1 b2 b3 b4 ... [part3] option= c1 c2 c3 c4 c5 IN extension1.cfg ADD VALUE = c3 c4 c5 IN base.cfg SET VALUE = c1 c2 ... [part4] option= d2 d3 d1 d4 IN extension1.cfg REMOVE VALUE = d5 IN extension1.cfg ADD VALUE = d1 d4 IN base.cfg SET VALUE = d2 d3 d5 ... [part5] option= h1 h2 IN extension1.cfg SET VALUE = h1 h2 [part6] option= e1 IN base.cfg SET VALUE = e1 [part7] option= IN IMPLICIT_VALUE REMOVE VALUE = f1 IN base.cfg SET VALUE = f1 ... Issue #656 - If an option was defined in one file, then extended twice in another - once in a regular section and again in a conditonal, the original definition is lost, not extended:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = ... extension1.cfg ... extension2.cfg ... develop = demo ... extensions = demo ... parts = part1 ... """) >>> write(sample_buildout, 'extension1.cfg', ... """ ... [part1] ... recipe = ... option = ... a ... b ... """) >>> write(sample_buildout, 'extension2.cfg', ... """ ... [part1] ... option += ... c ... d ... ... [part1:False] ... option += ... z ... ... [part1:True] ... option += ... e ... """) ... parts = part1 >>> print_(system(os.path.join('bin', 'buildout')), end='') ['a/nb/nc/nd/ne'] Develop: '/sample-buildout/demo' Cleanup:: >>> os.remove(os.path.join(sample_buildout, 'base.cfg')) >>> os.remove(os.path.join(sample_buildout, 'extension1.cfg')) >>> os.remove(os.path.join(sample_buildout, 'extension2.cfg')) Multiple configuration files ---------------------------- A configuration file can *extend* another configuration file. Options are read from the other configuration file if they aren't already defined by your configuration file. The configuration files your file extends can extend other configuration files. The same file may be used more than once although, of course, cycles aren't allowed. To see how this works, we use an example:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = base.cfg ... ... [debug] ... op = buildout ... """) >>> write(sample_buildout, 'base.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... op = base ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing debug. op buildout recipe recipes:debug The example is pretty trivial, but the pattern it illustrates is pretty common. In a more practical example, the base buildout might represent a product and the extending buildout might be a customization. Here is a more elaborate example:: >>> other = tmpdir('other') >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg b2.cfg %(b3)s ... ... [debug] ... op = buildout ... """ % dict(b3=os.path.join(other, 'b3.cfg'))) >>> write(sample_buildout, 'b1.cfg', ... """ ... [buildout] ... extends = base.cfg ... ... [debug] ... op1 = b1 1 ... op2 = b1 2 ... """) >>> write(sample_buildout, 'b2.cfg', ... """ ... [buildout] ... extends = base.cfg ... ... [debug] ... op2 = b2 2 ... op3 = b2 3 ... """) >>> write(other, 'b3.cfg', ... """ ... [buildout] ... extends = b3base.cfg ... ... [debug] ... op4 = b3 4 ... """) >>> write(other, 'b3base.cfg', ... """ ... [debug] ... op5 = b3base 5 ... """) >>> write(sample_buildout, 'base.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... name = base ... ... [environ] ... recipe = recipes:environ ... name = base ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 recipe recipes:debug There are several things to note about this example: - We can name multiple files in an ``extends`` option. - We can reference files recursively. - Relative file names in extended options are interpreted relative to the directory containing the referencing configuration file. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg ... ... [debug] ... op = buildout ... """) >>> print_(system([buildout, 'buildout:extends=b2.cfg']), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 recipe recipes:debug >>> print_(system([buildout, 'buildout:extends=b2.cfg %(b3)s' ... % dict(b3=os.path.join(other, 'b3.cfg'))]), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 recipe recipes:debug >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg b2.cfg %(b3)s ... ... [debug] ... op = buildout ... """ % dict(b3=os.path.join(other, 'b3.cfg'))) Optional extends ---------------- With ``optional-extends`` you can specify an extends file that is only loaded when it exists. The main idea is that a Buildout project in a shared Git repository can specify an ``optional-extends`` line with ``local.cfg`` or ``custom.cfg``. This optional file would be added to the files ignored by Git, and would only exist on individual computers. Any developer who wants this, can then create such a file and for example use this to always add the ``pdbpp`` egg to one of the Buildout parts. Or they can add a part, or other options specific to their system. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg ... optional-extends = optional.cfg ... ... [debug] ... op = buildout ... """) >>> print_(system(buildout), end='') optional-extends file not found: optional.cfg Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b1 2 recipe recipes:debug Now add the optional config: >>> write('optional.cfg', ... """ ... [debug] ... op2 = optional2 2 ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 optional2 2 recipe recipes:debug Restore the previous ``buildout.cfg`` and remove the new file, so we do not interfere with existing tests:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg b2.cfg %(b3)s ... ... [debug] ... op = buildout ... """ % dict(b3=os.path.join(other, 'b3.cfg'))) >>> remove(sample_buildout, 'optional.cfg') Loading Configuration from URLs ------------------------------- Configuration files can be loaded from URLs. To see how this works, we'll set up a web server with some configuration files:: >>> server_data = tmpdir('server_data') >>> write(server_data, "r1.cfg", ... """ ... [debug] ... op1 = r1 1 ... op2 = r1 2 ... """) >>> write(server_data, "r2.cfg", ... """ ... [buildout] ... extends = r1.cfg ... ... [debug] ... op2 = r2 2 ... op3 = r2 3 ... """) >>> server_url = start_server(server_data) >>> write('client.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... extends = %(url)s/r2.cfg ... ... [debug] ... recipe = recipes:debug ... name = base ... """ % dict(url=server_url)) >>> print_(system([buildout, '-c', 'client.cfg']), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op1 r1 1 op2 r2 2 op3 r2 3 recipe recipes:debug Here we specified a URL for the file we extended. The file we downloaded itself referred to a file on the server using a relative URL reference. Relative references are interpreted relative to the base URL when they appear in configuration files loaded via URL. We can also specify a URL as the configuration file to be used by a buildout:: >>> os.remove('client.cfg') >>> write(server_data, 'remote.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... extends = r2.cfg ... ... [debug] ... recipe = recipes:debug ... name = remote ... """) >>> print_(system([buildout, '-c', server_url + '/remote.cfg']), end='') While: Initializing. Error: Missing option: buildout:directory Normally, the buildout directory defaults to a directory containing a configuration file. This won't work for configuration files loaded from URLs. In this case, the buildout directory would normally be defined on the command line:: >>> print_(system([buildout, ... '-c', server_url + '/remote.cfg', ... 'buildout:directory=' + sample_buildout, ... ]), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name remote op1 r1 1 op2 r2 2 op3 r2 3 recipe recipes:debug User defaults ------------- If the file ``$HOME/.buildout/default.cfg`` exists, it is read before reading the configuration file. (``$HOME`` is the value of the ``HOME`` environment variable. The ``/`` is replaced by the operating system file delimiter.):: >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ... """ ... [debug] ... op1 = 1 ... op7 = 7 ... """) >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout, env=env), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 op7 7 recipe recipes:debug A buildout command-line argument, ``-U``, can be used to suppress reading user defaults:: >>> print_(system([buildout, '-U'], env=env), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 recipe recipes:debug If the environment variable ``BUILDOUT_HOME`` is non-empty, that is used to locate ``default.cfg`` instead of looking in ``~/.buildout/``. Let's set up a configuration file in an alternate directory and verify that we get the appropriate set of defaults:: >>> alterhome = tmpdir('alterhome') >>> write(alterhome, 'default.cfg', ... """ ... [debug] ... op1 = 1' ... op7 = 7' ... op8 = eight! ... """) >>> env['BUILDOUT_HOME'] = alterhome >>> print_(system(buildout, env=env), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 op7 7' op8 eight! recipe recipes:debug The ``-U`` argument still suppresses reading of the ``default.cfg`` file from ``BUILDOUT_HOME``:: >>> print_(system([buildout, '-U'], env=env), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 recipe recipes:debug Log level --------- We can control the level of logging by specifying a log level in our configuration file. For example, to suppress info messages, we can set the logging level to *WARNING*:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... log-level = WARNING ... extends = b1.cfg b2.cfg ... """) >>> print_(system(buildout), end='') name base op1 b1 1 op2 b2 2 op3 b2 3 recipe recipes:debug .. >>> stop_server(server_url) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4025877 zc_buildout-6.0.1/src/zc/buildout/tests/extends-cache.txt0000644000076500000240000004405115264141650023130 0ustar00mauritsstaffCaching extended configuration ============================== As mentioned in the general buildout documentation, configuration files can extend each other, including the ability to download configuration being extended from a URL. If desired, zc.buildout caches downloaded configuration in order to be able to use it when run offline. As we're going to talk about downloading things, let's start an HTTP server. Also, all of the following will take place inside the sample buildout. >>> server_data = tmpdir('server_data') >>> server_url = start_server(server_data) >>> cd(sample_buildout) We also use a fresh directory for temporary files in order to make sure that all temporary files have been cleaned up in the end: >>> import tempfile >>> old_tempdir = tempfile.tempdir >>> tempfile.tempdir = tmpdir('tmp') Basic use of the extends cache ------------------------------ We put some base configuration on a server and reference it from a sample buildout: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... foo = bar ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... """ % server_url) When trying to run this buildout offline, we'll find that we cannot read all of the required configuration: >>> print_(system(buildout + ' -o')) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. Trying the same online, we can: >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. As long as we haven't said anything about caching downloaded configuration, nothing gets cached. Offline mode will still cause the buildout to fail: >>> print_(system(buildout + ' -o')) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. Let's now specify a cache for base configuration files. This cache is different from the download cache used by recipes for caching distributions and other files; one might, however, use a namespace subdirectory of the download cache for it. The configuration cache we specify will be created when running buildout and the base.cfg file will be put in it (with the file name being a hash of the complete URL): >>> mkdir('cache') >>> write('buildout.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... extends-cache = cache ... """ % server_url) >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> cache = join(sample_buildout, 'cache') >>> ls(cache) - 5aedc98d7e769290a29d654a591a3a45 >>> import os >>> cat(cache, os.listdir(cache)[0]) [buildout] parts = foo = bar We can now run buildout offline as it will read base.cfg from the cache: >>> print_(system(buildout + ' -o')) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. The cache is being used purely as a fall-back in case we are offline or don't have access to a configuration file to be downloaded. As long as we are online, buildout attempts to download a fresh copy of each file even if a cached copy of the file exists. To see this, we put different configuration in the same place on the server and run buildout in offline mode so it takes base.cfg from the cache: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... bar = baz ... """) >>> print_(system(buildout + ' -o')) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. .. verify same behavior for install-from-cache: >>> print_(system(buildout + ' install-from-cache=true download-cache=.')) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. In online mode, buildout will download and use the modified version: >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'bar'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Trying offline mode again, the new version will be used as it has been put in the cache now: >>> print_(system(buildout + ' -o')) Section `buildout` contains unused option(s): 'bar'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Clean up: >>> rmdir(cache) Specifying extends cache and offline mode ----------------------------------------- Normally, the values of buildout options such as the location of a download cache or whether to use offline mode are determined by first reading the user's default configuration, updating it with the project's configuration and finally applying command-line options. User and project configuration are assembled by reading a file such as ``~/.buildout/default.cfg``, ``buildout.cfg`` or a URL given on the command line, recursively (depth-first) downloading any base configuration specified by the ``buildout:extends`` option read from each of those config files, and finally evaluating each config file to provide default values for options not yet read. This works fine for all options that do not influence how configuration is downloaded in the first place. The ``extends-cache`` and ``offline`` options, however, are treated differently from the procedure described in order to make it simple and obvious to see where a particular configuration file came from under any particular circumstances. - Offline and extends-cache settings are read from the two root config files exclusively. Otherwise one could construct configuration files that, when read, imply that they should have been read from a different source than they have. Also, specifying the extends cache within a file that might have to be taken from the cache before being read wouldn't make a lot of sense. - Offline and extends-cache settings given by the user's defaults apply to the process of assembling the project's configuration. If no extends cache has been specified by the user's default configuration, the project's root config file must be available, be it from disk or from the net. - Offline mode turned on by the ``-o`` command line option is honored from the beginning even though command line options are applied to the configuration last. If offline mode is not requested by the command line, it may be switched on by either the user's or the project's config root. Extends cache ~~~~~~~~~~~~~ Let's see the above rules in action. We create a new home directory for our user and write user and project configuration that recursively extends online bases, using different caches: >>> mkdir('home') >>> mkdir('home', '.buildout') >>> mkdir('cache') >>> mkdir('user-cache') >>> home=join(sample_buildout, 'home') >>> env=dict(HOME=home, USERPROFILE=home) >>> import functools >>> system = functools.partial(system, env=env) >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... extends-cache = user-cache ... """) >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... """ % server_url) >>> write(server_data, 'base_default.cfg', """\ ... [buildout] ... foo = bar ... offline = false ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... extends-cache = cache ... """) >>> write('fancy.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... """ % server_url) >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... offline = false ... """) Buildout will now assemble its configuration from all of these 6 files, defaults first. The online resources end up in the respective extends caches: >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> ls('user-cache') - 10e772cf422123ef6c64ae770f555740 >>> cat('user-cache', os.listdir('user-cache')[0]) [buildout] foo = bar offline = false >>> ls('cache') - c72213127e6eb2208a3e1fc1dba771a7 >>> cat('cache', os.listdir('cache')[0]) [buildout] parts = offline = false If, on the other hand, the extends caches are specified in files that get extended themselves, they won't be used for assembling the configuration they belong to (user's or project's, resp.). The extends cache specified by the user's defaults does, however, apply to downloading project configuration. Let's rewrite the config files, clean out the caches and re-run buildout: >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... """) >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... extends-cache = user-cache ... """ % server_url) >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... """) >>> write('fancy.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... extends-cache = cache ... """ % server_url) >>> remove('user-cache', os.listdir('user-cache')[0]) >>> remove('cache', os.listdir('cache')[0]) >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> ls('user-cache') - 0548bad6002359532de37385bb532e26 >>> cat('user-cache', os.listdir('user-cache')[0]) [buildout] parts = offline = false >>> ls('cache') Clean up: >>> rmdir('user-cache') >>> rmdir('cache') Offline mode and installation from cache ---------------------------------------- If we run buildout in offline mode now, it will fail because it cannot get at the remote configuration file needed by the user's defaults: >>> print_(system(buildout + ' -o')) While: Initializing. Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. Let's now successively turn on offline mode by different parts of the configuration and see when buildout applies this setting in each case: >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... offline = true ... """) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... """) >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... offline = true ... """ % server_url) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... """ % server_url) >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... offline = true ... """) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... """) >>> write('fancy.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... offline = true ... """ % server_url) >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. The ``install-from-cache`` option is treated accordingly: >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... install-from-cache = true ... """) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... """) >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... install-from-cache = true ... """ % server_url) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... """ % server_url) >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... install-from-cache = true ... """) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... """) >>> write('fancy.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... install-from-cache = true ... """ % server_url) >>> print_(system(buildout)) While: Installing. Checking for upgrades. An internal error occurred ... ValueError: install_from_cache set to true with no download cache >>> rmdir('home', '.buildout') Newest and non-newest behavior for extends cache ------------------------------------------------- While offline mode forbids network access completely, 'newest' mode determines whether to look for updated versions of a resource even if some version of it is already present locally. If we run buildout in newest mode (``newest = true``), the configuration files are updated with each run: >>> mkdir("cache") >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends-cache = cache ... extends = %sbase.cfg ... """ % server_url) >>> print_(system(buildout)) >>> ls('cache') - 5aedc98d7e769290a29d654a591a3a45 >>> cat('cache', os.listdir(cache)[0]) [buildout] parts = A change to ``base.cfg`` is picked up on the next buildout run: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... foo = bar ... """) >>> print_(system(buildout + " -n")) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> cat('cache', os.listdir(cache)[0]) [buildout] parts = foo = bar In contrast, when not using ``newest`` mode (``newest = false``), the files already present in the extends cache will not be updated: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... """) >>> print_(system(buildout + " -N")) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> cat('cache', os.listdir(cache)[0]) [buildout] parts = foo = bar Even when updating base configuration files with a buildout run, any given configuration file will be downloaded only once during that particular run. If some base configuration file is extended more than once, its cached copy is used: >>> write(server_data, 'baseA.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... foo = bar ... """ % server_url) >>> write(server_data, 'baseB.cfg', """\ ... [buildout] ... extends-cache = cache ... extends = %sbase.cfg ... bar = foo ... """ % server_url) >>> write('buildout.cfg', """\ ... [buildout] ... extends-cache = cache ... newest = true ... extends = %sbaseA.cfg %sbaseB.cfg ... """ % (server_url, server_url)) >>> print_(system(buildout + " -n")) Section `buildout` contains unused option(s): 'bar' 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. (XXX We patch download utility's API to produce readable output for the test; a better solution would re-use the logging already done by the utility.) >>> import zc.buildout >>> old_download = zc.buildout.download.Download.download >>> def wrapper_download(self, url, md5sum=None, path=None): ... print_("The URL %s was downloaded." % url) ... return old_download(url, md5sum, path) >>> zc.buildout.download.Download.download = wrapper_download >>> zc.buildout.buildout.main([]) The URL http://localhost/baseA.cfg was downloaded. The URL http://localhost/base.cfg was downloaded. The URL http://localhost/baseB.cfg was downloaded. Not upgrading because not running a local buildout command. Section `buildout` contains unused option(s): 'bar' 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> zc.buildout.download.Download.download = old_download The deprecated ``extended-by`` option ------------------------------------- The ``buildout`` section used to recognize an option named ``extended-by`` that was deprecated at some point and removed in the 1.5 line. Since ignoring this option silently was considered harmful as a matter of principle, a UserError is raised if that option is encountered now: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... extended-by = foo.cfg ... """) >>> print_(system(buildout)) While: Initializing. Error: No-longer supported "extended-by" option found in http://localhost/base.cfg. Reporting cached locations for downloads of faulty config files --------------------------------------------------------------- A downloaded config file might be invalid. A cancelled buildout run, an accidentally gzip-encoded download and so on. >>> write(server_data, 'faulty.cfg', """\ ... This is definitively not ... a proper() config file. ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends = %sfaulty.cfg ... """ % server_url) >>> print_(system(buildout)) While: Initializing. ... File contains no section headers. file: http://localhost/faulty.cfg (downloaded as ...), line: 1 'This is definitively not\n' Failing if extends-cache contains ${section:variable} ----------------------------------------------------- Because extends-cache is used to download extends, it may not contain ${section:variable} as the state of all variables cannot be computed before all extends have been loaded. >>> write(server_data, 'proper.cfg', """\ ... [buildout] ... dummy = fjhfj ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends = %sproper.cfg ... extends-cache = ${buildout:dummy} ... """ % server_url) >>> print_(system(buildout)) While: Initializing. ... ValueError: extends-cache '${buildout:dummy}' may not contain ${section:variable} to expand. Clean up -------- We should have cleaned up all temporary files created by downloading things: >>> ls(tempfile.tempdir) Reset the global temporary directory: >>> tempfile.tempdir = old_tempdir ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4026473 zc_buildout-6.0.1/src/zc/buildout/tests/extensions.txt0000644000076500000240000000527315264141650022617 0ustar00mauritsstaffExtensions ---------- A feature allows code to be loaded and run *after* configuration files have been read, but *before* the buildout has begun any processing. The intent is to allow special plugins such as ``urllib2`` request handlers to be loaded. To load an extension we use the ``extensions`` option and list one or more distribution requirements, on separate lines. The distributions named will be loaded and any ``zc.buildout.extension`` entry points found will be called with the buildout as an argument. When buildout finishes processing, any ``zc.buildout.unloadextension`` entry points found will be called with the buildout as an argument. Let's create a sample extension in our sample buildout created in the previous section:: >>> mkdir(sample_buildout, 'demo') >>> write(sample_buildout, 'demo', 'demo.py', ... """ ... import sys ... def ext(buildout): ... sys.stdout.write('%s %s\\n' % ('ext', sorted(buildout))) ... def unload(buildout): ... sys.stdout.write('%s %s\\n' % ('unload', sorted(buildout))) ... """) >>> write(sample_buildout, 'demo', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "demo", ... entry_points = { ... 'zc.buildout.extension': ['ext = demo:ext'], ... 'zc.buildout.unloadextension': ['ext = demo:unload'], ... }, ... ) ... """) Our extension just prints out the word 'demo', and lists the sections found in the buildout passed to it. We'll update our ``buildout.cfg`` to list the demo directory as a develop egg to be built:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... parts = ... """) >>> os.chdir(sample_buildout) >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') Develop: '/sample-buildout/demo' Now we can add the ``extensions`` option. We were a bit tricky and ran the buildout once with the demo develop egg defined but without the extension option. This is because extensions are loaded before the buildout creates develop eggs. We needed to use a separate buildout run to create the develop egg. Normally, when eggs are loaded from the network, we wouldn't need to do anything special. :: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... extensions = demo ... parts = ... """) We see that our extension is loaded and executed:: >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') ext ['buildout', 'versions'] Develop: '/sample-buildout/demo' unload ['buildout', 'versions'] ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4027073 zc_buildout-6.0.1/src/zc/buildout/tests/init.txt0000644000076500000240000001714415264141650021363 0ustar00mauritsstaffCreating new buildouts and bootstrapping ---------------------------------------- If ``zc.buildout`` is installed, you can use it to create a new buildout with its own local copies of ``zc.buildout`` and ``setuptools`` and with local buildout scripts:: >>> sample_bootstrapped = tmpdir('sample-bootstrapped') >>> print_(system([buildout, ... '-c' + os.path.join(sample_bootstrapped, 'setup.cfg'), ... 'init']), end='') Creating '/sample-bootstrapped/setup.cfg'. Creating directory '/sample-bootstrapped/eggs/v5'. Creating directory '/sample-bootstrapped/bin'. Creating directory '/sample-bootstrapped/parts'. Creating directory '/sample-bootstrapped/develop-eggs'. Generated script '/sample-bootstrapped/bin/buildout'. Note that a basic ``setup.cfg`` was created for us. This is because we provided an ``init`` argument. By default, the generated ``setup.cfg`` is as minimal as can be:: >>> cat(sample_bootstrapped, 'setup.cfg') [buildout] parts = We also get other buildout artifacts:: >>> ls(sample_bootstrapped) d bin d develop-eggs d eggs d parts - setup.cfg >>> ls(sample_bootstrapped, 'bin') - buildout >>> _ = (ls(sample_bootstrapped, 'eggs', 'v5'), ... ls(sample_bootstrapped, 'develop-eggs')) - packaging.egg-link - pip.egg-link - platformdirs.egg-link - setuptools.egg-link - wheel.egg-link - zc.buildout.egg-link (We list both the ``eggs`` and ``develop-eggs`` directories because the buildout or setuptools egg could be installed in the ``develop-eggs`` directory if the original buildout had develop eggs for either buildout or setuptools.) Note that the buildout script was installed but not run. To run the buildout, we'd have to run the installed buildout script. If we have an existing buildout that already has a ``buildout.cfg``, we'll normally use the ``bootstrap`` command instead of ``init``. It will complain if there isn't a configuration file:: >>> sample_bootstrapped2 = tmpdir('sample-bootstrapped2') >>> print_(system([buildout, ... '-c' + os.path.join(sample_bootstrapped2, 'setup.cfg'), ... 'bootstrap']), end='') While: Initializing. Error: Couldn't open /sample-bootstrapped2/setup.cfg >>> write(sample_bootstrapped2, 'setup.cfg', ... """ ... [buildout] ... parts = ... """) >>> print_(system([buildout, ... '-c' + os.path.join(sample_bootstrapped2, 'setup.cfg'), ... 'bootstrap']), end='') Creating directory '/sample-bootstrapped2/eggs/v5'. Creating directory '/sample-bootstrapped2/bin'. Creating directory '/sample-bootstrapped2/parts'. Creating directory '/sample-bootstrapped2/develop-eggs'. Generated script '/sample-bootstrapped2/bin/buildout'. Similarly, if there is a configuration file and we use the ``init`` command, we'll get an error that the configuration file already exists:: >>> print_(system([buildout, ... '-c' + os.path.join(sample_bootstrapped, 'setup.cfg'), ... 'init']), end='') While: Initializing. Error: '/sample-bootstrapped/setup.cfg' already exists. Initial eggs ------------ When using the ``init`` command, you can specify distribution requirements or paths to use:: >>> cd(sample_bootstrapped) >>> remove('setup.cfg') >>> print_(system([buildout, '-csetup.cfg', 'init', 'demo', 'other', './src']), end='') Creating '/sample-bootstrapped/setup.cfg'. Creating directory '/sample-bootstrapped/develop-eggs'. Getting distribution for 'zc.recipe.egg>=2.0.6'. Got zc.recipe.egg Installing py. Getting distribution for 'demo'. Got demo 0.3. Getting distribution for 'other'. Got other 1.0. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Generated script '/sample-bootstrapped/bin/demo'. Generated interpreter '/sample-bootstrapped/bin/py'. This causes a ``py`` part to be included that sets up a custom Python interpreter with the given requirements or paths:: >>> cat('setup.cfg') [buildout] parts = py [py] recipe = zc.recipe.egg interpreter = py eggs = demo other extra-paths = ./src Passing requirements or paths causes the buildout to be run as part of initialization. In the example above, we got a number of distributions installed and 2 scripts generated. The first, ``demo``, was defined by the ``demo`` project. The second, ``py`` was defined by the generated configuration. It's a *custom interpreter* that behaves like a standard Python interpreter, except that it includes the specified eggs and extra paths in its Python path. We specified a source directory that didn't exist. Buildout created it for us:: >>> ls('.') - .installed.cfg d bin d develop-eggs d eggs d parts - setup.cfg d src >>> uncd() .. Make sure it works if the dir is already there: >>> cd(sample_bootstrapped) >>> _ = system([buildout, '-csetup.cfg', 'buildout:parts=']) >>> remove('setup.cfg') >>> print_(system([buildout, '-csetup.cfg', 'init', 'demo', 'other', './src']), end='') Creating '/sample-bootstrapped/setup.cfg'. Creating directory '/sample-bootstrapped/develop-eggs'. Installing py. Generated script '/sample-bootstrapped/bin/demo'. Generated interpreter '/sample-bootstrapped/bin/py'. .. cleanup >>> _ = system([buildout, '-csetup.cfg', 'buildout:parts=']) >>> uncd() Finding distributions --------------------- By default, buildout searches the Python Package Index when looking for distributions. You can, instead, specify your own index to search using the `index` option:: [buildout] ... index = https://index.example.com/ This index, or the default of https://pypi.org/simple/ if no index is specified, will always be searched for distributions unless running buildout with options that prevent searching for distributions. The latest version of the distribution that meets the requirements of the buildout will always be used. You can also specify more locations to search for distributions using the ``find-links`` option. See its description above. Controlling the installation database ------------------------------------- The buildout ``installed`` option is used to specify the file used to save information on installed parts. This option is initialized to ``.installed.cfg``, but it can be overridden in the configuration file or on the command line:: >>> write('buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... """) >>> print_(system([buildout, 'buildout:installed=inst.cfg']), end='') Develop: '/sample-buildout/recipes' Installing debug. recipe recipes:debug >>> ls(sample_buildout) d bin - buildout.cfg d develop-eggs d eggs - inst.cfg d parts d recipes The installation database can be disabled by supplying an empty buildout installed option:: >>> os.remove('inst.cfg') >>> print_(system([buildout, 'buildout:installed=']), end='') Develop: '/sample-buildout/recipes' Installing debug. recipe recipes:debug >>> ls(sample_buildout) d bin - buildout.cfg d develop-eggs d eggs d parts d recipes Note that there will be no installation database if there are no parts:: >>> write('buildout.cfg', ... """ ... [buildout] ... parts = ... """) >>> print_(system([buildout, 'buildout:installed=inst.cfg']), end='') >>> ls(sample_buildout) d bin - buildout.cfg d develop-eggs d eggs d parts d recipes ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.402788 zc_buildout-6.0.1/src/zc/buildout/tests/options.txt0000644000076500000240000007044615264141650022117 0ustar00mauritsstaffSocket timeout -------------- The timeout of the connections to egg and configuration servers can be configured in the buildout section. Its value is configured in seconds:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... socket-timeout = 5 ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... op = timeout ... """) >>> print_(system(buildout), end='') Setting socket time out to 5 seconds. Develop: '/sample-buildout/recipes' Installing debug. op timeout recipe recipes:debug If the ``socket-timeout`` is not numeric, a warning is issued and the default timeout of the Python socket module is used:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... socket-timeout = 5s ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... op = timeout ... """) >>> print_(system(buildout), end='') Default socket timeout is used ! Value in configuration is not numeric: [5s]. Develop: '/sample-buildout/recipes' Updating debug. op timeout recipe recipes:debug Uninstall recipes ----------------- As we've seen, when parts are installed, buildout keeps track of files and directories that they create. When the parts are uninstalled these files and directories are deleted. Sometimes more clean-up is needed. For example, a recipe might add a system service by calling ``chkconfig --add`` during installation. Later during uninstallation, ``chkconfig --del`` will need to be called to remove the system service. In order to deal with these uninstallation issues, you can register uninstall recipes. Uninstall recipes are registered using the ``zc.buildout.uninstall`` entry point. Parts specify uninstall recipes using the ``uninstall`` option. In comparison to regular recipes, uninstall recipes are much simpler. They are simply callable objects that accept the name of the part to be uninstalled and the part's options dictionary. Uninstall recipes don't have access to the part itself since it may be impossible to instantiate at uninstallation time. Here's a recipe that simulates installation of a system service, along with an uninstall recipe that simulates removing the service:: >>> write(sample_buildout, 'recipes', 'src', 'service.py', ... """ ... import sys ... class Service: ... ... def __init__(self, buildout, name, options): ... self.buildout = buildout ... self.name = name ... self.options = options ... ... def install(self): ... sys.stdout.write("chkconfig --add %s\\n" ... % self.options['script']) ... return () ... ... def update(self): ... pass ... ... ... def uninstall_service(name, options): ... sys.stdout.write("chkconfig --del %s\\n" % options['script']) ... """) To use these recipes we must register them using entry points. Make sure to use the same name for the recipe and uninstall recipe. This is required to let buildout know which uninstall recipe goes with which recipe:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... entry_points = ( ... ''' ... [zc.buildout] ... mkdir = mkdir:Mkdir ... debug = debug:Debug ... service = service:Service ... ... [zc.buildout.uninstall] ... service = service:uninstall_service ... ''') ... setup(name="recipes", entry_points=entry_points) ... """) Here's how these recipes could be used in a buildout:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = service ... ... [service] ... recipe = recipes:service ... script = /path/to/script ... """) When the buildout is run the service will be installed:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing service. chkconfig --add /path/to/script The service has been installed. If the buildout is run again with no changes, the service shouldn't be changed:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating service. Now we change the service part to trigger uninstallation and re-installation:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = service ... ... [service] ... recipe = recipes:service ... script = /path/to/a/different/script ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling service. Running uninstall recipe. chkconfig --del /path/to/script Installing service. chkconfig --add /path/to/a/different/script Now we remove the service part, and add another part:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling service. Running uninstall recipe. chkconfig --del /path/to/a/different/script Installing debug. recipe recipes:debug Uninstall recipes don't have to take care of removing all the files and directories created by the part. This is still done automatically, following the execution of the uninstall recipe. An upshot is that an uninstallation recipe can access files and directories created by a recipe before they are deleted. For example, here's an uninstallation recipe that simulates backing up a directory before it is deleted. It is designed to work with the ``mkdir`` recipe introduced earlier:: >>> write(sample_buildout, 'recipes', 'src', 'backup.py', ... """ ... import os, sys ... def backup_directory(name, options): ... path = options['path'] ... size = len(os.listdir(path)) ... sys.stdout.write("backing up directory %s of size %s\\n" ... % (path, size)) ... """) It must be registered with the ``zc.buildout.uninstall`` entry point. Notice how it is given the name ``mkdir`` to associate it with the ``mkdir`` recipe:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... entry_points = ( ... ''' ... [zc.buildout] ... mkdir = mkdir:Mkdir ... debug = debug:Debug ... service = service:Service ... ... [zc.buildout.uninstall] ... uninstall_service = service:uninstall_service ... mkdir = backup:backup_directory ... ''') ... setup(name="recipes", entry_points=entry_points) ... """) Now we can use it with a ``mkdir`` part:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = dir debug ... ... [dir] ... recipe = recipes:mkdir ... path = my_directory ... ... [debug] ... recipe = recipes:debug ... """) Run the buildout to install the part:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing dir. dir: Creating directory my_directory Installing debug. recipe recipes:debug Now we remove the part from the configuration file:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... """) When the buildout is run the part is removed, and the uninstall recipe is run before the directory is deleted:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling dir. Running uninstall recipe. backing up directory /sample-buildout/my_directory of size 0 Updating debug. recipe recipes:debug Now we will return the registration to normal for the benefit of the rest of the examples:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... entry_points = ( ... ''' ... [zc.buildout] ... mkdir = mkdir:Mkdir ... debug = debug:Debug ... ''') ... setup(name="recipes", entry_points=entry_points) ... """) Command-line usage ------------------ A number of arguments can be given on the buildout command line. The command usage is:: buildout [options and assignments] [command [command arguments]] The following options are supported: ``-h`` (or ``--help``) Print basic usage information. If this option is used, then all other options are ignored. ``-c`` filename The ``-c`` option can be used to specify a configuration file, rather than ``buildout.cfg`` in the current directory. ``-t`` socket_timeout Specify the socket timeout in seconds. ``-v`` Increment the verbosity by 10. The verbosity is used to adjust the logging level. The verbosity is subtracted from the numeric value of the log-level option specified in the configuration file. ``-q`` Decrement the verbosity by 10. ``-U`` Don't read user-default configuration. ``-o`` Run in off-line mode. This is equivalent to the assignment ``buildout:offline=true``. ``-O`` Run in non-off-line mode. This is equivalent to the assignment ``buildout:offline=false``. This is the default buildout mode. The ``-O`` option would normally be used to override a true offline setting in a configuration file. ``-n`` Run in newest mode. This is equivalent to the assignment ``buildout:newest=true``. With this setting, which is the default, buildout will try to find the newest versions of distributions available that satisfy its requirements. ``-N`` Run in non-newest mode. This is equivalent to the assignment ``buildout:newest=false``. With this setting, buildout will not seek new distributions if installed distributions satisfy its requirements. Assignments are of the form:: section_name:option_name=value or:: option_name=value which is equivalent to:: buildout:option_name=value Options and assignments can be given in any order. Here's an example:: >>> write(sample_buildout, 'other.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... installed = .other.cfg ... log-level = WARNING ... ... [debug] ... name = other ... recipe = recipes:debug ... """) Note that we used the installed buildout option to specify an alternate file to store information about installed parts:: >>> print_(system([buildout, '-c', 'other.cfg', 'debug:op1=foo', '-v']), end='') Develop: '/sample-buildout/recipes' Installing debug. name other op1 foo recipe recipes:debug Here we used the ``-c`` option to specify an alternate configuration file, and the ``-v`` option to increase the level of logging from the default, *WARNING*. Options can also be combined in the usual Unix way, as in:: >>> print_(system([buildout, '-vcother.cfg', 'debug:op1=foo']), end='') Develop: '/sample-buildout/recipes' Updating debug. name other op1 foo recipe recipes:debug Here we combined the ``-v`` and ``-c`` options with the configuration file name. Note that the ``-c`` option has to be last, because it takes an argument:: >>> os.remove(os.path.join(sample_buildout, 'other.cfg')) >>> os.remove(os.path.join(sample_buildout, '.other.cfg')) The most commonly used command is ``install``, and it takes a list of parts to install. If any parts are specified, only those parts are installed. To illustrate this, we'll update our configuration and run the buildout in the usual way:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug d1 d2 d3 ... ... [d1] ... recipe = recipes:mkdir ... path = d1 ... ... [d2] ... recipe = recipes:mkdir ... path = d2 ... ... [d3] ... recipe = recipes:mkdir ... path = d3 ... ... [debug] ... recipe = recipes:debug ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. recipe recipes:debug Installing d1. d1: Creating directory d1 Installing d2. d2: Creating directory d2 Installing d3. d3: Creating directory d3 >>> ls(sample_buildout) - .installed.cfg d bin - buildout.cfg d d1 d d2 d d3 d develop-eggs d eggs d parts d recipes >>> cat(sample_buildout, '.installed.cfg') ... # doctest: +NORMALIZE_WHITESPACE [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = debug d1 d2 d3 [debug] __buildout_installed__ = __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== recipe = recipes:debug [d1] __buildout_installed__ = /sample-buildout/d1 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d1 recipe = recipes:mkdir [d2] __buildout_installed__ = /sample-buildout/d2 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d2 recipe = recipes:mkdir [d3] __buildout_installed__ = /sample-buildout/d3 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d3 recipe = recipes:mkdir Now we'll update our configuration file:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug d2 d3 d4 ... ... [d2] ... recipe = recipes:mkdir ... path = data2 ... ... [d3] ... recipe = recipes:mkdir ... path = data3 ... ... [d4] ... recipe = recipes:mkdir ... path = ${d2:path}-extra ... ... [debug] ... recipe = recipes:debug ... x = 1 ... """) and run the buildout specifying just ``d3`` and ``d4``:: >>> print_(system([buildout, 'install', 'd3', 'd4']), end='') Develop: '/sample-buildout/recipes' Uninstalling d3. Installing d3. d3: Creating directory data3 Installing d4. d4: Creating directory data2-extra >>> ls(sample_buildout) - .installed.cfg d bin - buildout.cfg d d1 d d2 d data2-extra d data3 d develop-eggs d eggs d parts d recipes Only the ``d3`` and ``d4`` recipes ran. ``d3`` was removed and ``data3`` and ``data2-extra`` were created. The ``.installed.cfg`` is only updated for the recipes that ran:: >>> cat(sample_buildout, '.installed.cfg') ... # doctest: +NORMALIZE_WHITESPACE [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = debug d1 d2 d3 d4 [debug] __buildout_installed__ = __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== recipe = recipes:debug [d1] __buildout_installed__ = /sample-buildout/d1 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d1 recipe = recipes:mkdir [d2] __buildout_installed__ = /sample-buildout/d2 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d2 recipe = recipes:mkdir [d3] __buildout_installed__ = /sample-buildout/data3 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/data3 recipe = recipes:mkdir [d4] __buildout_installed__ = /sample-buildout/data2-extra __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/data2-extra recipe = recipes:mkdir Note that the installed data for ``debug``, ``d1``, and ``d2`` haven't changed, because we didn't install those parts, and that the ``d1`` and ``d2`` directories are still there. Now, if we run the buildout without the install command:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling d2. Uninstalling d1. Uninstalling debug. Installing debug. recipe recipes:debug x 1 Installing d2. d2: Creating directory data2 Updating d3. Updating d4. We see the output of the debug recipe, and that ``data2`` was created. We also see that ``d1`` and ``d2`` have gone away:: >>> ls(sample_buildout) - .installed.cfg d bin - buildout.cfg d data2 d data2-extra d data3 d develop-eggs d eggs d parts d recipes Alternate directory and file locations -------------------------------------- The buildout normally puts the ``bin``, ``eggs``, and ``parts`` directories in the directory containing the configuration file. You can provide alternate locations, and even names for these directories:: >>> alt = tmpdir('sample-alt') >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = ... develop-eggs-directory = %(developbasket)s ... eggs-directory = %(basket)s ... eggs-directory-version = v2 ... bin-directory = %(scripts)s ... parts-directory = %(work)s ... """ % dict( ... developbasket = os.path.join(alt, 'developbasket'), ... basket = os.path.join(alt, 'basket'), ... scripts = os.path.join(alt, 'scripts'), ... work = os.path.join(alt, 'work'), ... )) >>> print_(system(buildout), end='') Creating directory '/sample-alt/basket/v2'. Creating directory '/sample-alt/scripts'. Creating directory '/sample-alt/work'. Creating directory '/sample-alt/developbasket'. Develop: '/sample-buildout/recipes' Uninstalling d4. Uninstalling d3. Uninstalling d2. Uninstalling debug. >>> ls(alt) d basket d developbasket d scripts d work >>> ls(alt, 'developbasket') - recipes.egg-link You can also specify an alternate buildout directory:: >>> rmdir(alt) >>> alt = tmpdir('sample-alt') >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... directory = %(alt)s ... develop = %(recipes)s ... parts = ... """ % dict( ... alt=alt, ... recipes=os.path.join(sample_buildout, 'recipes'), ... )) >>> print_(system(buildout), end='') Creating directory '/sample-alt/eggs/v5'. Creating directory '/sample-alt/bin'. Creating directory '/sample-alt/parts'. Creating directory '/sample-alt/develop-eggs'. Develop: '/sample-buildout/recipes' >>> ls(alt) - .installed.cfg d bin d develop-eggs d eggs d parts >>> ls(alt, 'develop-eggs') - recipes.egg-link Logging control --------------- Three buildout options are used to control logging: ``log-level`` specifies the log level ``verbosity`` adjusts the log level ``log-format`` allows an alternate logging format to be specified We've already seen the log level and verbosity. Let's look at an example of changing the format:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = ... log-level = 25 ... verbosity = 5 ... log-format = %(levelname)s %(message)s ... """) Here, we've changed the format to include the log-level name, rather than the logger name. We've also illustrated, with a contrived example, that the log level can be a numeric value and that the verbosity can be specified in the configuration file. Because the verbosity is subtracted from the log level, we get a final log level of 20, which is the *INFO* level:: >>> print_(system(buildout), end='') INFO Develop: '/sample-buildout/recipes' Predefined buildout options --------------------------- Buildouts have a number of predefined options that recipes can use and that users can override in their configuration files. To see these, we'll run a minimal buildout configuration with a ``debug`` logging level. One of the features of ``debug`` logging is that the configuration database is shown:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = ... """) >>> print_(system([buildout, '-vv']), end='') # doctest: +NORMALIZE_WHITESPACE Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Configuration data: [buildout] allow-hosts = * allow-picked-versions = true allow-unknown-extras = false bin-directory = /sample-buildout/bin develop-eggs-directory = /sample-buildout/develop-eggs directory = /sample-buildout eggs-directory = /sample-buildout/eggs/v5 eggs-directory-version = v5 executable = python find-links = install-from-cache = false installed = /sample-buildout/.installed.cfg log-format = log-level = INFO newest = true offline = false parts = parts-directory = /sample-buildout/parts prefer-final = true python = buildout show-picked-versions = false socket-timeout = update-versions-file = use-dependency-links = true verbosity = 20 versions = versions [versions] zc.buildout = >=1.99 zc.recipe.egg = >=1.99 All of these options can be overridden by configuration files or by command-line assignments. We've discussed most of these options already, but let's review them and touch on some we haven't discussed: ``abi-tag-eggs`` Add an `ABI tag `_ to the directory name given by the ``eggs-directory`` option. This is useful when switching between python implementations when details of the implementation aren't reflected in egg names. It also has the side benefit of making eggs directories smaller, because eggs for different Python versions are in different directories. ``allow-hosts`` On some environments the links visited by ``zc.buildout`` can be forbidden by paranoid firewalls. These URLs might be in the chain of links visited by ``zc.buildout`` as defined by buildout's ``find-links`` option, or as defined by various eggs in their ``url``, ``download_url``, ``dependency_links`` metadata. The fact that ``package_index`` works like a spider and might visit links and go to other locations makes this even harder. The ``allow-hosts`` option provides a way to prevent this, and works exactly like the one provided in ``easy_install``. You can provide a list of allowed hosts, together with wildcards:: [buildout] ... allow-hosts = *.python.org example.com All URLs that do not match these hosts will not be visited. ``allow-picked-versions`` By default, the buildout will choose the best match for a given requirement if the requirement is not specified precisely (for instance, using the ``versions`` option. This behavior corresponds to the ``allow-picked-versions`` being set to its default value, ``true``. If ``allow-picked-versions`` is ``false``, instead of picking the best match, buildout will raise an error. This helps enforce repeatability. ``bin-directory`` The directory path where scripts are written. This can be a relative path, which is interpreted relative to the directory option. ``develop-eggs-directory`` The directory path where development egg links are created for software being created in the local project. This can be a relative path, which is interpreted relative to the directory option. ``directory`` The buildout directory. This is the base for other buildout file and directory locations, when relative locations are used. ``eggs-directory`` The directory path where downloaded eggs are put. It is common to share this directory across buildouts. This can be a relative path, which is interpreted relative to the directory option. .. warning:: Eggs in this directory should *never* be modified. ``find-links`` You can specify more locations to search for distributions using the ``find-links`` option. All locations specified will be searched for distributions along with the package index as described before. Locations can be URLs:: [buildout] ... find-links = http://download.zope.org/distribution/ They can also be directories on disk:: [buildout] ... find-links = /some/path Finally, they can also be direct paths to distributions:: [buildout] ... find-links = /some/path/someegg-1.0.0-py2.3.egg Any number of locations can be specified in the ``find-links`` option:: [buildout] ... find-links = http://download.zope.org/distribution/ /some/otherpath /some/path/someegg-1.0.0-py2.3.egg ``install-from-cache`` A download cache can be used as the basis of application source releases. In an application source release, we want to distribute an application that can be built without making any network accesses. In this case, we distribute a buildout along with a download cache, and tell the buildout to install from the download cache only, without making network accesses. The buildout ``install-from-cache`` option can be used to signal that packages should be installed only from the download cache. ``installed`` The file path where information about the results of the previous buildout run is written. This can be a relative path, which is interpreted relative to the directory option. This file provides an inventory of installed parts with information needed to decide which if any parts need to be uninstalled. ``log-format`` The format used for logging messages. ``log-level`` The log level before verbosity adjustment ``newest`` By default buildout and recipes will try to find the newest versions of distributions needed to satisfy requirements. This can be very time consuming, especially when incrementally working on setting up a buildout or working on a recipe. The buildout ``newest`` option can be used to to suppress this. If the ``newest`` option is set to false, then new distributions won't be sought if an installed distribution meets requirements. The ``newest`` option can also be set to false using the -N command-line option. See also the ``offline`` option. ``offline`` The ``offline`` option goes a bit further than the ``newest`` option. If the buildout ``offline`` option is given a value of ``true``, the buildout and recipes that are aware of the option will avoid doing network access. This is handy when running the buildout when not connected to the internet. It also makes buildouts run much faster. This option is typically set using the buildout ``-o`` option. ``parts`` A whitespace-separated list of parts to be installed. ``parts-directory`` A working directory that parts can used to store data. ``prefer-final`` Currently, when searching for new releases, the newest available release is used. This isn't usually ideal, as you may get a development release or alpha releases not ready to be widely used. You can request that final releases be preferred using the ``prefer-final`` option in the ``buildout`` section:: [buildout] ... prefer-final = true When the ``prefer-final`` option is set to ``true``, then when searching for new releases, final releases are preferred. If there are final releases that satisfy distribution requirements, then those releases are used even if newer non-final releases are available. The buildout ``prefer-final`` option can be used to override this behavior. In buildout version 2, final releases will be preferred by default. You will then need to use a ``false`` value for ``prefer-final`` to get the newest releases. ``use-dependency-links`` By default buildout will obey the setuptools ``dependency_links`` metadata when it looks for dependencies. This behavior can be controlled with the ``use-dependency-links`` buildout option:: [buildout] ... use-dependency-links = false The option defaults to ``true``. If you set it to ``false``, then dependency links are only looked for in the locations specified by ``find-links``. ``verbosity`` A log-level adjustment. Typically, this is set via the ``-q`` and ``-v`` command-line options. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2139876 zc_buildout-6.0.1/src/zc/buildout/tests/recipes/0000755000076500000240000000000015264141651021303 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4028885 zc_buildout-6.0.1/src/zc/buildout/tests/recipes/README.txt0000644000076500000240000000015015264141650022774 0ustar00mauritsstaffThis is a package used during testing. The tests that use this, may override files, including setup.py. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4029408 zc_buildout-6.0.1/src/zc/buildout/tests/recipes/setup.py0000644000076500000240000000030515264141650023012 0ustar00mauritsstafffrom setuptools import setup entry_points = ( ''' [zc.buildout] mkdir = mkdir:Mkdir debug = debug:Debug environ = environ:Environ ''') setup( name="recipes", entry_points=entry_points, ) ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066473.214254 zc_buildout-6.0.1/src/zc/buildout/tests/recipes/src/0000755000076500000240000000000015264141651022072 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4030216 zc_buildout-6.0.1/src/zc/buildout/tests/recipes/src/debug.py0000644000076500000240000000053215264141650023531 0ustar00mauritsstaffimport sys class Debug: def __init__(self, buildout, name, options): self.buildout = buildout self.name = name self.options = options def install(self): for option, value in sorted(self.options.items()): sys.stdout.write('%s %s\n' % (option, value)) return () update = install ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4030704 zc_buildout-6.0.1/src/zc/buildout/tests/recipes/src/environ.py0000644000076500000240000000070615264141650024126 0ustar00mauritsstaffimport os import sys class Environ: def __init__(self, buildout, name, options): self.buildout = buildout self.options = options def install(self): _ = self.options['name'] sys.stdout.write('HOME %s\\n' % os.environ['HOME']) sys.stdout.write('USERPROFILE %s\\n' % os.environ['USERPROFILE']) sys.stdout.write('expanduser %s\\n' % os.path.expanduser('~')) return () update = install ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4031153 zc_buildout-6.0.1/src/zc/buildout/tests/recipes/src/mkdir.py0000644000076500000240000000225715264141650023557 0ustar00mauritsstaff# This is the final version of the mkdir.py file that we created # in the buildout.txt tests. import logging import os import zc.buildout class Mkdir: def __init__(self, buildout, name, options): self.name, self.options = name, options # Normalize paths and check that their parent # directories exist: paths = [] for path in options['path'].split(): path = os.path.join(buildout['buildout']['directory'], path) if not os.path.isdir(os.path.dirname(path)): logging.getLogger(self.name).error( 'Cannot create %s. %s is not a directory.', options['path'], os.path.dirname(options['path'])) raise zc.buildout.UserError('Invalid Path') paths.append(path) options['path'] = ' '.join(paths) def install(self): paths = self.options['path'].split() for path in paths: logging.getLogger(self.name).info( 'Creating directory %s', os.path.basename(path)) os.mkdir(path) self.options.created(path) return self.options.created() def update(self): pass ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.403207 zc_buildout-6.0.1/src/zc/buildout/tests/repeatable.txt0000644000076500000240000003305715264141650022525 0ustar00mauritsstaffRepeatable buildouts: controlling eggs used =========================================== One of the goals of zc.buildout is to provide enough control to make buildouts repeatable. It should be possible to check the buildout configuration files for a project into a version control system and later use the checked in files to get the same buildout, subject to changes in the environment outside the buildout. An advantage of using Python eggs is that dependencies of eggs used are automatically determined and used. The automatic inclusion of dependent distributions is at odds with the goal of repeatable buildouts. To support repeatable buildouts, a versions section can be created with options for each distribution name who's version is to be fixed. The section can then be specified via the buildout versions option. To see how this works, we'll create two versions of a recipe egg: >>> mkdir('recipe') >>> write('recipe', 'recipe.py', ... ''' ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... class Recipe: ... def __init__(*a): pass ... def install(self): ... print_('recipe v1') ... return () ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='spam', version='1', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> write('recipe', 'README', '') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... >>> rmdir('recipe', 'build') >>> write('recipe', 'recipe.py', ... ''' ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... class Recipe: ... def __init__(*a): pass ... def install(self): ... print_('recipe v2') ... return () ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='spam', version='2', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... and we'll configure a buildout to use it: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) If we run the buildout, it will use version 2: >>> print_(system(buildout), end='') Getting distribution for 'spam'. Got spam 2. Installing foo. recipe v2 We can specify a versions section that lists our recipe and name it in the buildout section: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [versions] ... spam = 1 ... eggs = 2.2 ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) Here we created a versions section listing the version 1 for the spam distribution. We told the buildout to use it by specifying release-1 as in the versions option. Now, if we run the buildout, we'll use version 1 of the spam recipe: >>> print_(system(buildout), end='') Getting distribution for 'spam==1'. Got spam 1. Uninstalling foo. Installing foo. recipe v1 Running the buildout in verbose mode will help us get information about versions used. If we run the buildout in verbose mode without specifying a versions section: >>> print_(system(buildout+' buildout:versions= -v'), end='') Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Installing 'spam'. We have the best distribution that satisfies 'spam'. Picked: spam = 2. Uninstalling foo. Installing foo. recipe v2 We'll get output that includes lines that tell us what versions buildout chose a for us, like:: zc.buildout.easy_install.picked: spam = 2 This allows us to discover versions that are picked dynamically, so that we can fix them in a versions section. If we run the buildout with the versions section: >>> print_(system(buildout+' -v'), end='') Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Installing 'spam'. We have the distribution that satisfies 'spam==1'. Uninstalling foo. Installing foo. recipe v1 We won't get output for the spam distribution, which we didn't pick, but we will get output for setuptools, which we didn't specify versions for. .. Edge case: version applied to range requirement: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [versions] ... spam = 1 ... eggs = 2.2 ... ... [foo] ... recipe = spam >0 ... ''' % join('recipe', 'dist')) >>> print_(system(buildout+' -v'), end='') Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Installing 'spam >0'. We have the distribution that satisfies 'spam==1'. Uninstalling foo. Installing foo. recipe v1 Edge case (issue #577) where a substitution inside the buildout section which comes from a section with a recipe, then the versions versions specifications were ignored, and the latest version installed, even if allow-picked-versions is false. >>> write('buildout.cfg', ... ''' ... [buildout] ... allow-picked-versions = false ... #show-picked-versions = true ... parts = foo ... find-links = %s ... test = ${foo:option} ... ... [versions] ... spam = 1 ... ... [foo] ... recipe = spam ... option = TEST ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Uninstalling foo. Section `buildout` contains unused option(s): 'test'. ... Installing foo. recipe v1 You can request buildout to generate an error if it picks any versions: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... allow-picked-versions = false ... ... [versions] ... eggs = 2.2 ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS While: Installing. Getting section foo. Initializing section foo. Installing recipe spam. Getting distribution for 'spam'. Error: Picked: spam = 2 ... We can name a version something else, if we wish, using the versions option: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... versions = release1 ... ... [release1] ... spam = 1 ... eggs = 2.2 ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Uninstalling foo. Installing foo. recipe v1 We can also disable checking versions: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... versions = ... ... [versions] ... spam = 1 ... eggs = 2.2 ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Uninstalling foo. Installing foo. recipe v2 Easier reporting and managing of versions (new in buildout 2.0) --------------------------------------------------------------- Since buildout 2.0, the functionality of the `buildout-versions `_ extension is part of buildout itself. This makes reporting and managing versions easier. Buildout picks versions for pip and setuptools and for the tests, we need to grab the version number: >>> import pkg_resources >>> req = pkg_resources.Requirement.parse('setuptools') >>> setuptools_version = pkg_resources.working_set.find(req).version >>> req = pkg_resources.Requirement.parse('pip') >>> pip_version = pkg_resources.working_set.find(req).version If you set the ``show-picked-versions`` option, buildout will print versions it picked at the end of its run: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... show-picked-versions = true ... ... [versions] ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 Versions had to be automatically picked. The following part definition lists the versions picked: [versions] spam = 2 When everything is pinned, no output is generated: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... show-picked-versions = true ... ... [versions] ... pip = %s ... setuptools = %s ... spam = 2 ... ... [foo] ... recipe = spam ... ''' % (join('recipe', 'dist'), pip_version, setuptools_version)) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 The Python package index is case-insensitive. Both https://pypi.org/simple/Django/ and https://pypi.org/simple/dJaNgO/ work. And distributions aren't always naming themselves consistently case-wise. So all version names are normalized and case differences won't impact the pinning: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... show-picked-versions = true ... ... [versions] ... pip = %s ... setuptools = %s ... Spam = 2 ... ... [foo] ... recipe = spam ... ''' % (join('recipe', 'dist'), pip_version, setuptools_version)) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 Sometimes it is handy to have a separate file with versions. This is a regular buildout file with a single ``[versions]`` section. You include it by extending from that versions file: >>> write('my_versions.cfg', ... ''' ... [versions] ... pip = %s ... setuptools = %s ... spam = 2 ... ''' % (pip_version, setuptools_version)) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... extends = my_versions.cfg ... find-links = %s ... show-picked-versions = true ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 If not everything is pinned and buildout has to pick versions, you can tell buildout to append the versions to your versions file. It simply appends them at the end. >>> write('my_versions.cfg', ... ''' ... [versions] ... pip = %s ... setuptools = %s ... ''' % (pip_version, setuptools_version)) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... extends = my_versions.cfg ... update-versions-file = my_versions.cfg ... find-links = %s ... show-picked-versions = true ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 Versions had to be automatically picked. The following part definition lists the versions picked: [versions] spam = 2 Picked versions have been written to my_versions.cfg The versions file now contains the extra pin: >>> with open('my_versions.cfg') as f: print_(f.read()) # doctest: +ELLIPSIS ... # Added by buildout at YYYY-MM-DD hh:mm:ss.dddddd spam = 2 And re-running buildout doesn't report any picked versions anymore: >>> 'picked' in system(buildout) False If you've enabled ``update-versions-file`` but not ``show-picked-versions``, buildout will append the versions to your versions file anyway (without printing them to the console): >>> write('my_versions.cfg', ... ''' ... [versions] ... pip = %s ... setuptools = %s ... ''' % (pip_version, setuptools_version)) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... extends = my_versions.cfg ... update-versions-file = my_versions.cfg ... find-links = %s ... show-picked-versions = false ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 Picked versions have been written to my_versions.cfg The versions file contains the extra pin: >>> with open('my_versions.cfg') as f: print_(f.read()) # doctest: +ELLIPSIS [versions] ... # Added by buildout at YYYY-MM-DD hh:mm:ss.dddddd spam = 2 Because buildout now includes buildout-versions' (and part of the older buildout.dumppickedversions') functionality, it warns if these extensions are configured. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = foo ... extensions = buildout-versions ... ... [foo] ... recipe = spam ... """) >>> print_(system(buildout), end='') # doctest: +NORMALIZE_WHITESPACE While: Installing. Loading extensions. Error: Buildout now includes 'buildout-versions' (and part of the older 'buildout.dumppickedversions'). Remove the extension from your configuration and look at the 'show-picked-versions' option in buildout's documentation. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4032598 zc_buildout-6.0.1/src/zc/buildout/tests/runsetup.txt0000644000076500000240000000326215264141650022301 0ustar00mauritsstaffRunning setup scripts ===================== Buildouts are often used to work on packages that will be distributed as eggs. During development, we use develop eggs. When you've completed a development cycle, you'll need to run your setup script to generate a distribution and, perhaps, uploaded it to the Python package index. If your script uses setuptools, you'll need setuptools in your Python path, which may be an issue if you haven't installed setuptools into your Python installation. The buildout setup command is helpful in a situation like this. It can be used to run a setup script and it does so with the setuptools egg in the Python path and with setuptools already imported. The fact that setuptools is imported means that you can use setuptools-based commands, like bdist_egg even with packages that don't use setuptools. To illustrate this, we'll create a package in a sample buildout: >>> mkdir('hello') >>> write('hello', 'hello.py', ... 'import sys; sys.stdout.write("Hello World!\\n")\n') >>> write('hello', 'README', 'This is hello') >>> write('hello', 'setup.py', ... """ ... from distutils.core import setup ... setup(name="hello", ... version="1.0", ... py_modules=["hello"], ... author="Bob", ... author_email="bob@foo.com", ... ) ... """) We can use the buildout command to generate the hello egg: >>> print_(system(buildout +' setup hello -q bdist_egg'), end='') Running setup script 'hello/setup.py'. zip_safe flag not set; analyzing archive contents... The hello directory now has a hello egg in it's dist directory: >>> ls('hello', 'dist') - hello-1.0-py2.4.egg ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4033105 zc_buildout-6.0.1/src/zc/buildout/tests/setup.txt0000644000076500000240000000372115264141650021554 0ustar00mauritsstaffUsing zc.buildout to run setup scripts ====================================== zc buildout has a convenience command for running setup scripts. Why? There are two reasons. If a setup script doesn't import setuptools, you can't use any setuptools-provided commands, like bdist_egg. When buildout runs a setup script, it arranges to import setuptools before running the script so setuptools-provided commands are available. If you use a squeaky-clean Python to do your development, the setup script that would import setuptools because setuptools isn't in the path. Because buildout requires setuptools and knows where it has installed a setuptools egg, it adds the setuptools egg to the Python path before running the script. To run a setup script, use the buildout setup command, passing the name of a script or a directory containing a setup script and arguments to the script. Let's look at an example: >>> mkdir('test') >>> cd('test') >>> write('setup.py', ... ''' ... from distutils.core import setup ... setup(name='sample') ... ''') We've created a super simple (stupid) setup script. Note that it doesn't import setuptools. Let's try running it to create an egg. We'll use the buildout script from our sample buildout: >>> print_(system(buildout+' setup'), end='') ... # doctest: +NORMALIZE_WHITESPACE Creating directory '/sample-buildout/test/eggs/v5'. Error: The setup command requires the path to a setup script or directory containing a setup script, and its arguments. Oops, we forgot to give the name of the setup script: >>> print_(system(buildout+' setup setup.py bdist_egg')) ... # doctest: +ELLIPSIS Running setup script 'setup.py'. ... >>> ls('dist') - sample-0.0.0-py2.5.egg Note that we can specify a directory name. This is often shorter and preferred by the lazy :) >>> print_(system(buildout+' setup . bdist_egg')) # doctest: +ELLIPSIS Running setup script './setup.py'. ... ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4034786 zc_buildout-6.0.1/src/zc/buildout/tests/test_all.py0000644000076500000240000034022315264141650022035 0ustar00mauritsstaff# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2004-2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from __future__ import print_function import unittest from zc.buildout.buildout import print_ from zope.testing import renormalizing, setupstack import doctest import manuel.capture import manuel.doctest import manuel.testing import os import pkg_resources import re import shutil import subprocess import sys import tarfile import tempfile import zc.buildout.easy_install import zc.buildout.testing from build import ProjectBuilder from build.env import DefaultIsolatedEnv from pathlib import Path from zc.buildout.tests import easy_install_SetUp from zc.buildout.tests import normalize_bang from zc.buildout.tests import create_wheel from zc.buildout.tests import create_sample_eggs os_path_sep = os.path.sep if os_path_sep == '\\': os_path_sep *= 2 HERE = Path(__file__).parent class TestEasyInstall(unittest.TestCase): # The contents of a zipped egg, created by setuptools: # from setuptools import setup # setup( # name='TheProject', # version='3.3', # ) # # (we can't run setuptools at runtime, it may not be installed) EGG_DATA = ( b'PK\x03\x04\x14\x00\x00\x00\x08\x00q8\xa8Lg0\xb7ix\x00\x00\x00\xb6\x00' b'\x00\x00\x11\x00\x00\x00EGG-INFO/PKG-INFO\xf3M-ILI,I\xd4\rK-*' b'\xce\xcc\xcf\xb3R0\xd43\xe0\xf2K\xccM\xb5R\x08\xc9H\r(\xca\xcfJM' b'.\xe1\x82\xcb\x1a\xeb\x19s\x05\x97\xe6\xe6&\x16UZ)\x84\xfay\xfb\xf9\x87\xfb' b'qy\xe4\xe7\xa6\xea\x16$\xa6\xa7"\x84\x1cKK2\xf2\x8b\xd0\xf9\xba\xa9\xb9\x89' b'\x999\x08Q\x9f\xcc\xe4\xd4\xbcb$m.\xa9\xc5\xc9E\x99\x05%`\xbb`\x82\x019\x89%' b'i\xf9E\xb9\x08\x11\x00PK\x03\x04\x14\x00\x00\x00\x08\x00q8\xa8L61\xa1' b'XL\x00\x00\x00\x87\x00\x00\x00\x14\x00\x00\x00EGG-INFO/SOURCES.txt\x0b\xc9H' b'\r(\xca\xcfJM.\xd1KMO\xd7\xcd\xccK\xcb\xd7\x0f\xf0v\xd7\xf5\xf4s' b'\xf3\xe7\n\xc1"\x19\xec\x1f\x1a\xe4\xec\x1a\xacWRQ\x82U>%\xb5 5/%5/\xb92>\'3' b'/\xbb\x18\xa7\xc2\x92\xfc\x82\xf8\x9c\xd4\xb2\xd4\x1c\x90\n\x00PK\x03' b'\x04\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00\x00\x01' b'\x00\x00\x00\x1d\x00\x00\x00EGG-INFO/dependency_links.txt\xe3\x02\x00P' b'K\x03\x04\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00' b'\x00\x01\x00\x00\x00\x16\x00\x00\x00EGG-INFO/top_level.txt\xe3\x02\x00PK' b'\x03\x04\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00\x00' b'\x01\x00\x00\x00\x11\x00\x00\x00EGG-INFO/zip-safe\xe3\x02\x00PK\x01\x02' b'\x14\x03\x14\x00\x00\x00\x08\x00q8\xa8Lg0\xb7ix\x00\x00\x00\xb6\x00\x00\x00' b'\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xa4\x81\x00\x00\x00\x00EG' b'G-INFO/PKG-INFOPK\x01\x02\x14\x03\x14\x00\x00\x00\x08\x00q8\xa8L61\xa1XL' b'\x00\x00\x00\x87\x00\x00\x00\x14\x00\x00\x00\x00\x00\x00\x00\x00' b'\x00\x00\x00\xa4\x81\xa7\x00\x00\x00EGG-INFO/SOURCES.txtPK\x01' b'\x02\x14\x03\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00' b'\x00\x01\x00\x00\x00\x1d\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' b'\x00\xa4\x81%\x01\x00\x00EGG-INFO/dependency_links.txtPK\x01\x02' b'\x14\x03\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00\x00' b'\x01\x00\x00\x00\x16\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' b'\xa4\x81c\x01\x00\x00EGG-INFO/top_level.txtPK\x01\x02\x14\x03\x14\x00' b'\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00\x00\x01\x00\x00\x00' b'\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xa4\x81\x9a\x01\x00\x00EG' b'G-INFO/zip-safePK\x05\x06\x00\x00\x00\x00\x05\x00\x05\x00O\x01\x00\x00\xcc' b'\x01\x00\x00\x00\x00' ) def setUp(self): self.cwd = os.getcwd() self.temp_dir = tempfile.mkdtemp('.buildouttest') self.project_dir = os.path.join(self.temp_dir, 'TheProject') self.project_dist_dir = os.path.join(self.temp_dir, 'dist') os.mkdir(self.project_dist_dir) self.egg_path = os.path.join(self.project_dist_dir, 'TheProject.egg') os.mkdir(self.project_dir) self.setup_path = os.path.join(self.project_dir, 'setup.py') os.chdir(self.temp_dir) def tearDown(self): os.chdir(self.cwd) shutil.rmtree(self.temp_dir) def _make_egg(self): with open(self.egg_path, 'wb') as f: f.write(self.EGG_DATA) def _get_distro_and_egg_path(self): # Returns a distribution with a version of '3.3.0', # but an egg with a version of '3.3' self._make_egg() from distutils.dist import Distribution dist = Distribution() dist.project_name = 'TheProject' dist.version = '3.3.0' dist.parsed_version = pkg_resources.parse_version(dist.version) return dist, self.egg_path def test_get_matching_dist_in_location_uses_parsed_version(self): # https://github.com/buildout/buildout/pull/452 # An egg built with the version '3.3' should match a distribution # looking for '3.3.0' dist, location = self._get_distro_and_egg_path() result = zc.buildout.easy_install._get_matching_dist_in_location( dist, self.project_dist_dir ) self.assertIsNotNone(result) self.assertEqual(result.version, '3.3') def test_move_to_eggs_dir_and_compile(self): # https://github.com/buildout/buildout/pull/452 # An egg built with the version '3.3' should match a distribution # looking for '3.3.0' dist, location = self._get_distro_and_egg_path() dist.location = location dest = os.path.join(self.temp_dir, 'NewLoc') result = zc.buildout.easy_install._move_to_eggs_dir_and_compile( dist, dest ) self.assertIsNotNone(result) self.assertEqual(result.version, '3.3') if zc.buildout.WINDOWS: self.assertIn(dest.lower(), result.location) else: self.assertIn(dest, result.location) def develop_w_non_setuptools_setup_scripts(): """ We should be able to deal with setup scripts that aren't setuptools based. >>> mkdir('foo') >>> write('foo', 'setup.py', ... ''' ... from distutils.core import setup ... setup(name="foo") ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = foo ... parts = ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' >>> ls('develop-eggs') - foo.egg-link - zc.recipe.egg.egg-link """ def develop_verbose(): """Check how we deal with multiple times the verbose option. We don't really test the output anymore: this is just too different depending on which setuptools version you use. >>> mkdir('foo') >>> write('foo', 'setup.py', ... ''' ... from setuptools import setup ... setup(name="foo") ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = foo ... parts = ... ''') >>> print_(system(join('bin', 'buildout')+' -vv'), end='') ... # doctest: +ELLIPSIS Installing... Making editable install of /sample-buildout/foo ... Successfully made editable install: /sample-buildout/develop-eggs/foo.egg-link ... >>> ls('develop-eggs') - foo.egg-link - zc.recipe.egg.egg-link >>> print_(system(join('bin', 'buildout')+' -vvv'), end='') ... # doctest: +ELLIPSIS Installing... Making editable install of /sample-buildout/foo ... Successfully made editable install: /sample-buildout/develop-eggs/foo.egg-link ... """ def buildout_error_handling(): r"""Buildout error handling Asking for a section that doesn't exist, yields a missing section error: >>> import os >>> os.chdir(sample_buildout) >>> import zc.buildout.buildout >>> buildout = zc.buildout.buildout.Buildout('buildout.cfg', []) >>> buildout['eek'] Traceback (most recent call last): ... MissingSection: The referenced section, 'eek', was not defined. Asking for an option that doesn't exist, a MissingOption error is raised: >>> buildout['buildout']['eek'] Traceback (most recent call last): ... MissingOption: Missing option: buildout:eek It is an error to create a variable-reference cycle: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... parts = ... x = ${buildout:y} ... y = ${buildout:z} ... z = ${buildout:x} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Getting option buildout:y. Getting option buildout:z. Getting option buildout:x. Error: Circular reference in substitutions. It is an error to use funny characters in variable references: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data_dir debug ... x = ${bui$ldout:y} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Error: The section name in substitution, ${bui$ldout:y}, has invalid characters. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data_dir debug ... x = ${buildout:y{z} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Error: The option name in substitution, ${buildout:y{z}, has invalid characters. and too have too many or too few colons: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data_dir debug ... x = ${parts} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Error: The substitution, ${parts}, doesn't contain a colon. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data_dir debug ... x = ${buildout:y:z} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Error: The substitution, ${buildout:y:z}, has too many colons. All parts have to have a section: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... parts = x ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Installing. Getting section x. Error: The referenced section, 'x', was not defined. and all parts have to have a specified recipe: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... parts = x ... ... [x] ... foo = 1 ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Installing. Error: Missing option: x:recipe """ make_dist_that_requires_setup_py_template = """ from setuptools import setup setup(name=%r, version=%r, install_requires=%r, ) """ def make_dist_that_requires(dest, name, requires=None, version=1, egg=''): if requires is None: requires = [] os.mkdir(os.path.join(dest, name)) with open(os.path.join(dest, name, 'setup.py'), 'w') as f: f.write( make_dist_that_requires_setup_py_template % (name, version, requires) ) def show_who_requires_when_there_is_a_conflict(): """ It's a pain when we require eggs that have requirements that are incompatible. We want the error we get to tell us what is missing. Let's make a few develop distros, some of which have incompatible requirements. >>> make_dist_that_requires(sample_buildout, 'sampley', ... ['demoneeded ==1.0']) >>> make_dist_that_requires(sample_buildout, 'samplez', ... ['demoneeded ==1.1']) Now, let's create a buildout that requires y and z: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = sampley samplez ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = sampley ... samplez ... ''' % globals()) >>> print_(system(buildout), end='') Develop: '/sample-buildout/sampley' Develop: '/sample-buildout/samplez' Installing eggs. Getting distribution for 'demoneeded==1.1'. Got demoneeded 1.1. Version and requirements information containing demoneeded: Requirement of samplez: demoneeded==1.1 Requirement of sampley: demoneeded==1.0... While: Installing eggs. Error: There is a version conflict. We already have: demoneeded 1.1 but sampley 1 requires 'demoneeded==1.0'. Here, we see that sampley required an older version of demoneeded. What if we hadn't required sampley ourselves: >>> make_dist_that_requires(sample_buildout, 'samplea', ['sampleb']) >>> make_dist_that_requires(sample_buildout, 'sampleb', ... ['sampley', 'samplea']) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = sampley samplez samplea sampleb ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = samplea ... samplez ... ''' % globals()) If we use the verbose switch, we can see where requirements are coming from: >>> print_(system(buildout+' -v'), end='') # doctest: +ELLIPSIS Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Installing eggs. ... Error: There is a version conflict. We already have: demoneeded 1.1 but sampley 1 requires 'demoneeded==1.0'. """ def version_conflict_rendering(): """ We use the arguments passed by pkg_resources.VersionConflict to construct a nice error message: >>> error = pkg_resources.VersionConflict('pkg1 2.1', 'pkg1 1.0') >>> ws = [] # Not relevant for this test >>> print_(zc.buildout.easy_install.VersionConflict( ... error, ws)) # doctest: +ELLIPSIS There is a version conflict... But sometimes pkg_resources passes a nicely formatted string itself already. Extracting the original arguments fails in that case, so we just show the string. >>> error = pkg_resources.VersionConflict('pkg1 2.1 is simply wrong') >>> ws = [] # Not relevant for this test >>> print_(zc.buildout.easy_install.VersionConflict( ... error, ws)) # doctest: +ELLIPSIS There is a version conflict. pkg1 2.1 is simply wrong """ def show_who_requires_missing_distributions(): """ When working with a lot of eggs, which require eggs recursively, it can be hard to tell why we're requiring things we can't find. Fortunately, buildout will tell us who's asking for something that we can't find. when run in verbose mode >>> make_dist_that_requires(sample_buildout, 'sampley', ['demoneeded']) >>> make_dist_that_requires(sample_buildout, 'samplea', ['sampleb']) >>> make_dist_that_requires(sample_buildout, 'sampleb', ... ['sampley', 'samplea']) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = sampley samplea sampleb ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = samplea ... ''') >>> print_(system(buildout+' -v'), end='') # doctest: +ELLIPSIS Installing ... Installing 'samplea'. We have a develop egg: samplea 1 Getting required 'sampleb' required by samplea 1. We have a develop egg: sampleb 1 Getting required 'sampley' required by sampleb 1. We have a develop egg: sampley 1 Getting required 'demoneeded' required by sampley 1. We have no distributions for demoneeded that satisfies 'demoneeded'. ... While: Installing eggs. Getting distribution for 'demoneeded'. Error: Couldn't find a distribution for 'demoneeded'. """ def show_who_requires_picked_versions(): """ The show-picked-versions prints the versions, but it also prints who required the picked distributions. We do not need to run in verbose mode for that to work: >>> make_dist_that_requires(sample_buildout, 'sampley', ['demo']) >>> make_dist_that_requires(sample_buildout, 'samplea', ['sampleb']) >>> make_dist_that_requires(sample_buildout, 'sampleb', ... ['sampley', 'samplea']) >>> write('buildout.cfg', ... ''' ... [buildout] ... find-links = %(sample_eggs)s ... parts = eggs ... show-picked-versions = true ... develop = sampley samplea sampleb ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = samplea ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: ... Versions had to be automatically picked. The following part definition lists the versions picked: [versions] # Required by: # sampley==1 demo = 0.3 # Required by: # demo==0.3 demoneeded = 1.1 """ def test_comparing_saved_options_with_funny_characters(): """ If an option has newlines, extra/odd spaces or a %, we need to make sure the comparison with the saved value works correctly. >>> mkdir(sample_buildout, 'recipes') >>> write(sample_buildout, 'recipes', 'debug.py', ... ''' ... class Debug: ... def __init__(self, buildout, name, options): ... options['debug'] = \"\"\" ... ... ... path foo ... ... ... ... \"\"\" ... options['debug1'] = \"\"\" ... ... ... ... path foo ... ... ... ... \"\"\" ... options['debug2'] = ' x ' ... options['debug3'] = '42' ... options['format'] = '%3d' ... ... def install(self): ... with open('t', 'w') as f: f.write('t') ... return 't' ... ... update = install ... ''') >>> write(sample_buildout, 'recipes', 'setup.py', ... ''' ... from setuptools import setup ... setup( ... name = "recipes", ... entry_points = {'zc.buildout': ['default = debug:Debug']}, ... ) ... ''') >>> write(sample_buildout, 'recipes', 'README.txt', " ") >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes ... ''') >>> os.chdir(sample_buildout) >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing debug. If we run the buildout again, we shouldn't get a message about uninstalling anything because the configuration hasn't changed. >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating debug. """ def finding_eggs_as_local_directories(): r""" It is possible to set up find-links so that we could install from a local directory that may contained unzipped eggs. >>> src = tmpdir('src') >>> write(src, 'setup.py', ... ''' ... from setuptools import setup ... setup(name='demo', py_modules=[''], ... zip_safe=False, version='1.0', author='bob', url='bob', ... author_email='bob') ... ''') >>> write(src, 't.py', '#\n') >>> write(src, 'README.txt', '') >>> _ = system(join('bin', 'buildout')+' setup ' + src + ' bdist_egg') Install it so it gets unzipped: >>> d1 = tmpdir('d1') >>> ws = zc.buildout.easy_install.install( ... ['demo'], d1, links=[join(src, 'dist')], ... ) >>> ls(d1) d demo-1.0-py2.4.egg Then try to install it again: >>> d2 = tmpdir('d2') >>> ws = zc.buildout.easy_install.install( ... ['demo'], d2, links=[d1], ... ) >>> ls(d2) d demo-1.0-py2.4.egg """ def create_sections_on_command_line(): """ >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... x = ${foo:bar} ... ''') >>> print_(system(buildout + ' foo:bar=1 -vv'), end='') ... # doctest: +ELLIPSIS Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... [foo] bar = 1 ... """ def test_help(): """ >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')+' -h')) ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Usage: buildout [options] [assignments] [command [command arguments]] Options: -c config_file Specify the path to the buildout configuration file to be used. This defaults to the file named "buildout.cfg" in the current working directory. ... -h, --help ... >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout') ... +' --help')) ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Usage: buildout [options] [assignments] [command [command arguments]] Options: -c config_file Specify the path to the buildout configuration file to be used. This defaults to the file named "buildout.cfg" in the current working directory. ... -h, --help ... """ def test_version(): """ >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') >>> print_(system(buildout+' --version')) ... # doctest: +ELLIPSIS buildout version ... """ def test_bootstrap_with_extension(): """ We had a problem running a bootstrap with an extension. Let's make sure it is fixed. Basically, we don't load extensions when bootstrapping. >>> d = tmpdir('sample-bootstrap') >>> write(d, 'buildout.cfg', ... ''' ... [buildout] ... extensions = some_awsome_extension ... parts = ... ''') >>> os.chdir(d) >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout') ... + ' bootstrap'), end='') Creating directory '/sample-bootstrap/eggs/v5'. Creating directory '/sample-bootstrap/bin'. Creating directory '/sample-bootstrap/parts'. Creating directory '/sample-bootstrap/develop-eggs'. Generated script '/sample-bootstrap/bin/buildout'. """ def bug_92891_bootstrap_crashes_with_egg_recipe_in_buildout_section(): """ >>> d = tmpdir('sample-bootstrap') >>> write(d, 'buildout.cfg', ... ''' ... [buildout] ... parts = buildout ... eggs-directory = eggs ... ... [buildout] ... recipe = zc.recipe.egg ... eggs = zc.buildout ... scripts = buildout=buildout ... ''') >>> os.chdir(d) >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout') ... + ' bootstrap'), end='') Creating directory '/sample-bootstrap/eggs/v5'. Creating directory '/sample-bootstrap/bin'. Creating directory '/sample-bootstrap/parts'. Creating directory '/sample-bootstrap/develop-eggs'. Generated script '/sample-bootstrap/bin/buildout'. >>> print_(system(os.path.join('bin', 'buildout')), end='') Section `buildout` contains unused option(s): 'eggs' 'scripts'. This may be an indication for either a typo in the option's name or a bug in the used recipe. """ def removing_eggs_from_develop_section_causes_egg_link_to_be_removed(): ''' >>> cd(sample_buildout) Create a develop egg: >>> mkdir('foo') >>> write('foo', 'setup.py', ... """ ... from setuptools import setup ... setup(name='foox') ... """) >>> write('buildout.cfg', ... """ ... [buildout] ... develop = foo ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' >>> ls('develop-eggs') - foox.egg-link - zc.recipe.egg.egg-link Create another: >>> mkdir('bar') >>> write('bar', 'setup.py', ... """ ... from setuptools import setup ... setup(name='fooy') ... """) >>> write('buildout.cfg', ... """ ... [buildout] ... develop = foo bar ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' Develop: '/sample-buildout/bar' >>> ls('develop-eggs') - foox.egg-link - fooy.egg-link - zc.recipe.egg.egg-link Remove one: >>> write('buildout.cfg', ... """ ... [buildout] ... develop = bar ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/bar' It is gone >>> ls('develop-eggs') - fooy.egg-link - zc.recipe.egg.egg-link Remove the other: >>> write('buildout.cfg', ... """ ... [buildout] ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') All gone >>> ls('develop-eggs') - zc.recipe.egg.egg-link ''' def add_setuptools_to_dependencies_when_namespace_packages(): ''' Often, a package depends on setuptools solely by virtue of using namespace packages. In this situation, package authors often forget to declare setuptools as a dependency. This is a mistake, but, unfortunately, a common one that we need to work around. If an egg uses namespace packages and does not include setuptools as a dependency, we will still include setuptools in the working set. If we see this for a develop egg, we will also generate a warning. >>> mkdir('foo') >>> mkdir('foo', 'src') >>> mkdir('foo', 'src', 'stuff') >>> write('foo', 'src', 'stuff', '__init__.py', ... """__import__('pkg_resources').declare_namespace(__name__) ... """) >>> mkdir('foo', 'src', 'stuff', 'foox') >>> write('foo', 'src', 'stuff', 'foox', '__init__.py', '') >>> write('foo', 'setup.py', ... """ ... from setuptools import setup ... setup(name='foox', ... namespace_packages = ['stuff'], ... package_dir = {'': 'src'}, ... packages = ['stuff', 'stuff.foox'], ... ) ... """) >>> write('foo', 'README.txt', '') >>> write('buildout.cfg', ... """ ... [buildout] ... develop = foo ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' WARNING: Package foox at .../foo is using old style namespace packages. You should switch to native namespaces (PEP 420). ... Some development packages are using old style namespace packages. ... pip install horse-with-no-namespace ... The following list shows the affected packages and their namespaces: * foox:... Depending on the pip and setuptools versions used, the last line may either be `foox: stuff` or `foox:`. Now, if we generate a working set using the egg link, we will get a warning and we will get setuptools included in the working set. >>> import logging, zope.testing.loggingsupport >>> handler = zope.testing.loggingsupport.InstalledHandler( ... 'zc.buildout.easy_install', level=logging.WARNING) >>> logging.getLogger('zc.buildout.easy_install').propagate = False >>> def get_working_set(*project_names): ... paths = [join(sample_buildout, 'eggs', 'v5'), ... join(sample_buildout, 'develop-eggs')] ... return [ ... dist.project_name ... for dist in zc.buildout.easy_install.working_set( ... project_names, sys.executable, paths) ... ] >>> get_working_set('foox') # doctest: +ELLIPSIS ['foox'...] >>> print_(handler) # doctest: +ELLIPSIS zc.buildout.easy_install WARNING ... >>> handler.clear() On the other hand, if we have a zipped egg, rather than a develop egg: >>> os.remove(join('develop-eggs', 'foox.egg-link')) >>> _ = system(join('bin', 'buildout') + ' setup foo bdist_egg') >>> foox_dist = join('foo', 'dist') >>> import glob >>> [foox_egg] = glob.glob(join(foox_dist, 'foox-*.egg')) >>> _ = shutil.copy(foox_egg, join(sample_buildout, 'eggs', 'v5')) >>> ls('develop-eggs') - zc.recipe.egg.egg-link >>> ls('eggs') d v5 >>> ls('eggs', 'v5') - foox-0.0.0-py2.4.egg - packaging.egg-link - pip.egg-link - platformdirs.egg-link - setuptools.egg-link - wheel.egg-link - zc.buildout.egg-link We do not get a warning, but we do get setuptools included in the working set: >>> get_working_set('foox') # doctest: +ELLIPSIS ['foox'...] >>> print_(handler, end='') # doctest: +ELLIPSIS ... Likewise for an unzipped egg: >>> foox_egg_basename = os.path.basename(foox_egg) >>> os.remove(join(sample_buildout, 'eggs', 'v5', foox_egg_basename)) >>> _ = zc.buildout.easy_install.install( ... ['foox'], join(sample_buildout, 'eggs', 'v5'), links=[foox_dist], ... index='file://' + foox_dist) >>> ls('develop-eggs') - zc.recipe.egg.egg-link >>> get_working_set('foox') # doctest: +ELLIPSIS ['foox'...] >>> print_(handler, end='') # doctest: +ELLIPSIS ... We get the same behavior if it is a dependency that uses a namespace package. >>> mkdir('bar') >>> write('bar', 'setup.py', ... """ ... from setuptools import setup ... setup(name='bar', install_requires = ['foox']) ... """) >>> write('bar', 'README.txt', '') >>> write('buildout.cfg', ... """ ... [buildout] ... develop = foo bar ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' WARNING: Package foox at .../foo is using old style namespace packages. You should switch to native namespaces (PEP 420). Develop: '/sample-buildout/bar' ... Some development packages are using old style namespace packages. ... pip install horse-with-no-namespace ... The following list shows the affected packages and their namespaces: * foox:... >>> get_working_set('bar') # doctest: +ELLIPSIS ['bar'...] Strangely, on pip version 24.3.1 and older the handler shows a warning, but on newer versions not. It is just a warning, so let's ignore it. >>> handler.clear() On the other hand, if the distribution uses ``pkgutil.extend_path()`` to implement its namespaces, even if just as fallback from the absence of ``pkg_resources``, then ``setuptools`` should not be added as requirement to its unzipped egg: >>> foox_installed_egg = join(sample_buildout, 'eggs', 'v5', foox_egg_basename) >>> namespace_init = join(foox_installed_egg, 'stuff', '__init__.py') >>> write(namespace_init, ... """try: ... __import__('pkg_resources').declare_namespace(__name__) ... except ImportError: ... __path__ = __import__('pkgutil').extend_path(__path__, __name__) ... """) >>> os.remove(join('develop-eggs', 'foox.egg-link')) >>> os.remove(join('develop-eggs', 'bar.egg-link')) >>> get_working_set('foox') ['foox'] The same goes for packages using PEP420 namespaces >>> os.remove(namespace_init) >>> get_working_set('foox') ['foox'] Cleanup: >>> logging.getLogger('zc.buildout.easy_install').propagate = True >>> handler.uninstall() ''' def develop_preserves_existing_setup_cfg(): """ See "Handling custom build options for extensions in develop eggs" in easy_install.txt. This will be very similar except that we'll have an existing setup.cfg: >>> write(extdemo, "setup.cfg", ... ''' ... # sampe cfg file ... ... [foo] ... bar = 1 ... ... [build_ext] ... define = X,Y ... ''') >>> mkdir('include') >>> write('include', 'extdemo.h', ... ''' ... #define EXTDEMO 42 ... ''') >>> dest = tmpdir('dest') >>> zc.buildout.easy_install.develop( ... extdemo, dest, ... {'include-dirs': os.path.join(sample_buildout, 'include')}) '/dest/extdemo.egg-link' >>> ls(dest) - extdemo.egg-link >>> cat(extdemo, "setup.cfg") # sampe cfg file [foo] bar = 1 [build_ext] define = X,Y """ def uninstall_recipes_used_for_removal(): r""" Uninstall recipes need to be called when a part is removed too: >>> mkdir("recipes") >>> write("recipes", "setup.py", ... ''' ... from setuptools import setup ... setup(name='recipes', ... entry_points={ ... 'zc.buildout': ["demo=demo:Install"], ... 'zc.buildout.uninstall': ["demo=demo:uninstall"], ... }) ... ''') >>> write("recipes", "demo.py", ... r''' ... import sys ... class Install: ... def __init__(*args): pass ... def install(self): ... sys.stdout.write('installing\n') ... return () ... def uninstall(name, options): ... sys.stdout.write('uninstalling\n') ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... develop = recipes ... parts = demo ... [demo] ... recipe = recipes:demo ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/recipes' Installing demo. installing >>> write('buildout.cfg', ''' ... [buildout] ... develop = recipes ... parts = demo ... [demo] ... recipe = recipes:demo ... x = 1 ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/recipes' Uninstalling demo. Running uninstall recipe. uninstalling Installing demo. installing >>> write('buildout.cfg', ''' ... [buildout] ... develop = recipes ... parts = ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/recipes' Uninstalling demo. Running uninstall recipe. uninstalling """ def extensions_installed_as_eggs_work_in_offline_mode(): ''' >>> mkdir('demo') >>> write('demo', 'demo.py', ... r""" ... import sys ... def print_(*args): ... sys.stdout.write(' '.join(map(str, args)) + '\\n') ... def ext(buildout): ... print_('ext', sorted(buildout)) ... """) >>> write('demo', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "demo", ... py_modules=['demo'], ... entry_points = {'zc.buildout.extension': ['ext = demo:ext']}, ... ) ... """) >>> bdist_egg(join(sample_buildout, "demo"), sys.executable, ... join(sample_buildout, "eggs", "v5")) >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extensions = demo ... parts = ... offline = true ... """) >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') ext ['buildout', 'versions'] ''' def changes_in_svn_or_git_dont_affect_sig(): """ If we have a develop recipe, it's signature shouldn't be affected to changes in .git, .svn directories. CVS directories used to work as well but do not anymore. >>> mkdir('recipe') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', ... entry_points={'zc.buildout': ['default=foo:Foo']}) ... ''') >>> write('recipe', 'foo.py', ... ''' ... class Foo: ... def __init__(*args): pass ... def install(*args): return () ... update = install ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipe ... parts = foo ... ... [foo] ... recipe = recipe ... ''') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Installing foo. >>> mkdir('recipe', '.git') >>> mkdir('recipe', '.svn') >>> # deprecated CVS mkdir('recipe', 'CVS') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Updating foo. >>> write('recipe', '.git', 'x', '1') >>> write('recipe', '.svn', 'x', '1') >>> # deprecated CVS write('recipe', 'CVS', 'x', '1') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Updating foo. """ if hasattr(os, 'symlink'): def bug_250537_broken_symlink_doesnt_affect_sig(): """ If we have a develop recipe, it's signature shouldn't be affected by broken symlinks, and better yet, computing the hash should not break because of the missing target file. >>> mkdir('recipe') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', ... entry_points={'zc.buildout': ['default=foo:Foo']}) ... ''') >>> write('recipe', 'foo.py', ... ''' ... class Foo: ... def __init__(*args): pass ... def install(*args): return () ... update = install ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipe ... parts = foo ... ... [foo] ... recipe = recipe ... ''') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Installing foo. >>> write('recipe', 'some-file', '1') >>> os.symlink(join('recipe', 'some-file'), ... join('recipe', 'another-file')) >>> remove('recipe', 'some-file') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Updating foo. """ def unicode_filename_doesnt_break_hash(): """ Buildout's _dir_hash() used to break on non-ascii filenames on python 2. >>> mkdir('héhé') >>> write('héhé', 'héhé.py', ... ''' ... print('Example filename from pyramid tests') ... ''') >>> from zc.buildout.buildout import _dir_hash >>> dont_care = _dir_hash('héhé') """ def o_option_sets_offline(): """ >>> print_(system(join(sample_buildout, 'bin', 'buildout')+' -vvo'), end='') ... # doctest: +ELLIPSIS ... offline = true ... """ def recipe_upgrade(): r""" The buildout will upgrade recipes in newest (and non-offline) mode. Let's create a recipe egg >>> mkdir('recipe') >>> write('recipe', 'recipe.py', ... r''' ... import sys ... class Recipe: ... def __init__(*a): pass ... def install(self): ... sys.stdout.write('recipe v1\n') ... return () ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', version='1', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> write('recipe', 'README', '') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... >>> rmdir('recipe', 'build') And update our buildout to use it. >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [foo] ... recipe = recipe ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') Getting distribution for 'recipe'. Got recipe 1. Installing foo. recipe v1 Now, if we update the recipe egg: >>> write('recipe', 'recipe.py', ... r''' ... import sys ... class Recipe: ... def __init__(*a): pass ... def install(self): ... sys.stdout.write('recipe v2\n') ... return () ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', version='2', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... We won't get the update if we specify -N: >>> print_(system(buildout+' -N'), end='') Updating foo. recipe v1 or if we use -o: >>> print_(system(buildout+' -o'), end='') Updating foo. recipe v1 But we will if we use neither of these: >>> print_(system(buildout), end='') Getting distribution for 'recipe'. Got recipe 2. Uninstalling foo. Installing foo. recipe v2 We can also select a particular recipe version: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [foo] ... recipe = recipe ==1 ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') Uninstalling foo. Installing foo. recipe v1 """ def update_adds_to_uninstall_list(): """ Paths returned by the update method are added to the list of paths to uninstall >>> mkdir('recipe') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> write('recipe', 'recipe.py', ... ''' ... import os ... class Recipe: ... def __init__(*_): pass ... def install(self): ... r = ('a', 'b', 'c') ... for p in r: os.mkdir(p) ... return r ... def update(self): ... r = ('c', 'd', 'e') ... for p in r: ... if not os.path.exists(p): ... os.mkdir(p) ... return r ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipe ... parts = foo ... ... [foo] ... recipe = recipe ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipe' Installing foo. >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipe' Updating foo. >>> cat('.installed.cfg') # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE [buildout] ... [foo] __buildout_installed__ = a b c d e __buildout_signature__ = ... """ def log_when_there_are_not_local_distros(): """ >>> from zope.testing.loggingsupport import InstalledHandler >>> handler = InstalledHandler('zc.buildout.easy_install') >>> import logging >>> logger = logging.getLogger('zc.buildout.easy_install') >>> old_propogate = logger.propagate >>> logger.propagate = False >>> dest = tmpdir('sample-install') >>> import zc.buildout.easy_install >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=[link_server], index=link_server+'index/') >>> print_(handler) # doctest: +ELLIPSIS zc.buildout.easy_install DEBUG Installing 'demo==0.2'. zc.buildout.easy_install DEBUG We have no distributions for demo that satisfies 'demo==0.2'. ... >>> handler.uninstall() >>> logger.propagate = old_propogate """ def internal_errors(): """Internal errors are clearly marked and don't generate tracebacks: >>> mkdir(sample_buildout, 'recipes') >>> write(sample_buildout, 'recipes', 'mkdir.py', ... ''' ... class Mkdir: ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... options['path'] = os.path.join( ... buildout['buildout']['directory'], ... options['path'], ... ) ... ''') >>> write(sample_buildout, 'recipes', 'setup.py', ... ''' ... from setuptools import setup ... setup(name = "recipes", ... entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']}, ... ) ... ''') >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... ''') >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' While: Installing. Getting section data-dir. Initializing section data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... NameError: name 'os' is not defined... """ def whine_about_unused_options(): ''' >>> write('foo.py', ... """ ... class Foo: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... options['x'] ... ... def install(self): ... self.options['y'] ... return () ... """) >>> write('setup.py', ... """ ... from setuptools import setup ... setup(name = "foo", ... py_modules=['foo'], ... entry_points = {'zc.buildout': ['default = foo:Foo']}, ... ) ... """) >>> write('buildout.cfg', ... """ ... [buildout] ... develop = . ... parts = foo ... a = 1 ... ... [foo] ... recipe = foo ... x = 1 ... y = 1 ... z = 1 ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/.' Section `buildout` contains unused option(s): 'a'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Installing foo. Section `foo` contains unused option(s): 'z'. This may be an indication for either a typo in the option's name or a bug in the used recipe. ''' def abnormal_exit(): """ People sometimes hit control-c while running a builout. We need to make sure that the installed database Isn't corrupted. To test this, we'll create some evil recipes that exit uncleanly: >>> mkdir('recipes') >>> write('recipes', 'recipes.py', ... ''' ... import os ... ... class Clean: ... def __init__(*_): pass ... def install(_): return () ... def update(_): pass ... ... class EvilInstall(Clean): ... def install(_): os._exit(1) ... ... class EvilUpdate(Clean): ... def update(_): os._exit(1) ... ''') >>> write('recipes', 'setup.py', ... ''' ... import setuptools ... setuptools.setup(name='recipes', ... entry_points = { ... 'zc.buildout': [ ... 'clean = recipes:Clean', ... 'evil_install = recipes:EvilInstall', ... 'evil_update = recipes:EvilUpdate', ... 'evil_uninstall = recipes:Clean', ... ], ... }, ... ) ... ''') Now let's look at 3 cases: 1. We exit during installation after installing some other parts: >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:clean ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:evil_install ... ... [p4] ... recipe = recipes:clean ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing p1. Installing p2. Installing p3. >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating p1. Updating p2. Installing p3. >>> print_(system(buildout+' buildout:parts='), end='') Develop: '/sample-buildout/recipes' Uninstalling p2. Uninstalling p1. 2. We exit while updating: >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:clean ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:evil_update ... ... [p4] ... recipe = recipes:clean ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing p1. Installing p2. Installing p3. Installing p4. >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating p1. Updating p2. Updating p3. >>> print_(system(buildout+' buildout:parts='), end='') Develop: '/sample-buildout/recipes' Uninstalling p2. Uninstalling p1. Uninstalling p4. Uninstalling p3. 3. We exit while installing or updating after uninstalling: >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:evil_update ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:clean ... ... [p4] ... recipe = recipes:clean ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing p1. Installing p2. Installing p3. Installing p4. >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:evil_update ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:clean ... ... [p4] ... recipe = recipes:clean ... x = 1 ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling p4. Updating p1. >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:clean ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:clean ... ... [p4] ... recipe = recipes:clean ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling p1. Installing p1. Updating p2. Updating p3. Installing p4. """ def install_source_dist_with_bad_py(): r""" >>> mkdir('badegg') >>> mkdir('badegg', 'badegg') >>> write('badegg', 'badegg', '__init__.py', '#\\n') >>> mkdir('badegg', 'badegg', 'scripts') >>> write('badegg', 'badegg', 'scripts', '__init__.py', '#\\n') >>> write('badegg', 'badegg', 'scripts', 'one.py', ... ''' ... return 1 ... ''') >>> write('badegg', 'setup.py', ... ''' ... from setuptools import setup, find_packages ... setup( ... name='badegg', ... version='1', ... packages = find_packages('.'), ... zip_safe=False) ... ''') >>> print_(system(buildout+' setup badegg sdist'), end='') # doctest: +ELLIPSIS Running setup script 'badegg/setup.py'. ... >>> dist = join('badegg', 'dist') >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs bo ... find-links = %(dist)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = badegg ... ... [bo] ... recipe = zc.recipe.egg ... eggs = zc.buildout ... scripts = buildout=bo ... ''' % globals()) >>> print_(system(buildout));print_('X') # doctest: +ELLIPSIS Installing eggs. Getting distribution for 'badegg'. Got badegg 1. Installing bo. Generated script '/sample-buildout/bin/bo'. X >>> ls('eggs', 'v5') # doctest: +ELLIPSIS d badegg-1-py2.4.egg ... >>> ls('bin') - bo - buildout """ def version_requirements_in_build_honored(): ''' >>> update_extdemo() >>> dest = tmpdir('sample-install') >>> mkdir('include') >>> write('include', 'extdemo.h', ... """ ... #define EXTDEMO 42 ... """) >>> zc.buildout.easy_install.build( ... 'extdemo ==1.4', dest, ... {'include-dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/', ... newest=False) ['/sample-install/extdemo-1.4-py2.4-linux-i686.egg'] ''' def bug_105081_Specific_egg_versions_are_ignored_when_newer_eggs_are_around(): """ Buildout might ignore a specific egg requirement for a recipe: - Have a newer version of an egg in your eggs directory - Use 'recipe==olderversion' in your buildout.cfg to request an older version Buildout will go and fetch the older version, but it will *use* the newer version when installing a part with this recipe. >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = x ... find-links = %(sample_eggs)s ... ... [x] ... recipe = zc.recipe.egg ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') Installing x. Getting distribution for 'demo'. Got demo 0.3. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Generated script '/sample-buildout/bin/demo'. >>> print_(system(join('bin', 'demo')), end='') 3 1 >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = x ... find-links = %(sample_eggs)s ... ... [x] ... recipe = zc.recipe.egg ... eggs = demo ==0.1 ... ''' % globals()) >>> print_(system(buildout), end='') Uninstalling x. Installing x. Getting distribution for 'demo==0.1'. Got demo 0.1. Generated script '/sample-buildout/bin/demo'. >>> print_(system(join('bin', 'demo')), end='') 1 1 """ def test_exit_codes(): """ >>> import subprocess >>> def call(s): ... p = subprocess.Popen(s, stdin=subprocess.PIPE, ... stdout=subprocess.PIPE, stderr=subprocess.STDOUT) ... p.stdin.close() ... print_(p.stdout.read().decode()) ... print_('Exit:', bool(p.wait())) ... p.stdout.close() >>> call(buildout) Exit: False >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = x ... ''') >>> call(buildout) # doctest: +NORMALIZE_WHITESPACE While: Installing. Getting section x. Error: The referenced section, 'x', was not defined. Exit: True >>> write('setup.py', ... ''' ... from setuptools import setup ... setup(name='zc.buildout.testexit', ... py_modules=['testexitrecipe'], ... entry_points={'zc.buildout': ['default = testexitrecipe:x']}) ... ''') >>> write('testexitrecipe.py', ... ''' ... x y ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = x ... develop = . ... ... [x] ... recipe = zc.buildout.testexit ... ''') >>> call(buildout) # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Develop: '/sample-buildout/.' While: Installing. Getting section x. Initializing section x. Loading zc.buildout recipe entry zc.buildout.testexit:default. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... x y ...^... SyntaxError... Exit: True """ def bug_59270_recipes_always_start_in_buildout_dir(): r""" Recipes can rely on running from buildout directory >>> mkdir('bad_start') >>> write('bad_recipe.py', ... r''' ... import os, sys ... def print_(*args): ... sys.stdout.write(' '.join(map(str, args)) + '\n') ... class Bad: ... def __init__(self, *_): ... print_(os.getcwd()) ... def install(self): ... sys.stdout.write(os.getcwd()+'\n') ... os.chdir('bad_start') ... sys.stdout.write(os.getcwd()+'\n') ... return () ... ''') >>> write('setup.py', ... ''' ... from setuptools import setup ... setup(name='bad.test', ... py_modules=['bad_recipe'], ... entry_points={'zc.buildout': ['default=bad_recipe:Bad']},) ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = . ... parts = b1 b2 ... [b1] ... recipe = bad.test ... [b2] ... recipe = bad.test ... ''') >>> os.chdir('bad_start') >>> print_(system(join(sample_buildout, 'bin', 'buildout') ... +' -c '+join(sample_buildout, 'buildout.cfg')), end='') Develop: '/sample-buildout/.' /sample-buildout /sample-buildout Installing b1. /sample-buildout /sample-buildout/bad_start Installing b2. /sample-buildout /sample-buildout/bad_start """ def bug_61890_file_urls_dont_seem_to_work_in_find_dash_links(): """ This bug arises from the fact that setuptools is overly restrictive about file urls, requiring that file urls pointing at directories must end in a slash. >>> dest = tmpdir('sample-install') >>> import zc.buildout.easy_install >>> sample_eggs = sample_eggs.replace(os.path.sep, '/') >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=['file://'+sample_eggs], index=link_server+'index/') >>> for dist in ws: ... print_(dist) demoneeded 1.1 demo 0.2 >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg """ def bug_75607_buildout_should_not_run_if_it_creates_an_empty_buildout_cfg(): """ >>> remove('buildout.cfg') >>> print_(system(buildout), end='') While: Initializing. Error: Couldn't open /sample-buildout/buildout.cfg """ def dealing_with_extremely_insane_dependencies(): r""" There was a problem with analysis of dependencies taking a long time, in part because the analysis would get repeated every time a package was encountered in a dependency list. Now, we don't do the analysis any more: >>> import os >>> for i in range(5): ... p = 'pack%s' % i ... deps = [('pack%s' % j) for j in range(5) if j is not i] ... if i == 4: ... deps.append('pack5') ... mkdir(p) ... write(p, 'setup.py', ... 'from setuptools import setup\n' ... 'setup(name=%r, install_requires=%r,\n' ... ' url="u", author="a", author_email="e")\n' ... % (p, deps)) >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = pack0 pack1 pack2 pack3 pack4 ... parts = pack1 ... ... [pack1] ... recipe = zc.recipe.egg:eggs ... eggs = pack0 ... ''') >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/pack0' Develop: '/sample-buildout/pack1' Develop: '/sample-buildout/pack2' Develop: '/sample-buildout/pack3' Develop: '/sample-buildout/pack4' Installing pack1. ... While: Installing pack1. Getting distribution for 'pack5'. Error: Couldn't find a distribution for 'pack5'. However, if we run in verbose mode, we can see why packages were included: >>> print_(system(buildout+' -v'), end='') # doctest: +ELLIPSIS Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Installing pack1. ... Error: Couldn't find a distribution for 'pack5'. """ def read_find_links_to_load_extensions(): r""" We'll create a wacky buildout extension that just announces itself when used: >>> src = tmpdir('src') >>> write(src, 'wacky_handler.py', ... ''' ... import sys ... def install(buildout=None): ... sys.stdout.write("I am a wacky extension\\n") ... ''') >>> write(src, 'setup.py', ... ''' ... from setuptools import setup ... setup(name='wackyextension', version='1', ... py_modules=['wacky_handler'], ... entry_points = {'zc.buildout.extension': ... ['default = wacky_handler:install'] ... }, ... ) ... ''') >>> print_(system(buildout+' setup '+src+' bdist_egg'), end='') ... # doctest: +ELLIPSIS Running setup ... creating 'dist/wackyextension-1-... Now we'll create a buildout that uses this extension to load other packages: >>> dist = 'file://' + join(src, 'dist').replace(os.path.sep, '/') >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... extensions = wackyextension ... find-links = %(dist)s ... ''' % globals()) When we run the buildout. it will load the extension from the dist directory and then use the wacky extension to load the demo package >>> print_(system(buildout), end='') Getting distribution for 'wackyextension'. Got wackyextension 1. I am a wacky extension """ def distributions_from_local_find_links_make_it_to_download_cache(): """ If we specify a local directory in find links, distros found there need to make it to the download cache. >>> mkdir('test') >>> write('test', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='foo') ... ''') >>> print_(system(buildout+' setup test bdist_egg')) # doctest: +ELLIPSIS Running setup script 'test/setup.py'. ... >>> mkdir('cache') >>> old_cache = zc.buildout.easy_install.download_cache('cache') >>> list(zc.buildout.easy_install.install(['foo'], 'eggs', ... links=[join('test', 'dist')])) # doctest: +ELLIPSIS [foo 0.0.0 ... >>> ls('cache') - foo-0.0.0-py2.4.egg >>> _ = zc.buildout.easy_install.download_cache(old_cache) """ def prefer_final_permutation(existing, available): """Test different permutations of existing and available versions. We used to do this: * Call create_egg on the list of 'existing' versions and put this in a directory 'existing'. * Call create_egg on the list of 'available' versions and put this in a directory 'available'. * Call 'install' with as destination the 'existing' directory and as find-links the 'available' directory. We have switched to calling create_wheel instead of create_egg. This means the items in the 'existing' directory are not regarded as being installed: it is not an eggs directory, but a downloads cache. So we need to explicitly install them. For clarity we now use the 'installed' directory as destination. """ # Create the necessary empty directories. for d in ('existing', 'available', 'installed'): if os.path.exists(d): shutil.rmtree(d) os.mkdir(d) for version in existing: create_wheel('spam', version, 'existing') # Install the package (we expect only one version, really). zc.buildout.easy_install.clear_index_cache() [dist] = list( zc.buildout.easy_install.install(['spam'], 'installed', ['existing']) ) assert dist is not None for version in available: create_wheel('spam', version, 'available') zc.buildout.easy_install.clear_index_cache() [dist] = list( zc.buildout.easy_install.install(['spam'], 'installed', ['available']) ) if dist.extras: print_('downloaded', dist.version) else: print_('had', dist.version) sys.path_importer_cache.clear() def prefer_final(): """ This test tests several permutations: Using different version numbers to work around zip importer cache problems. :( - With prefer final: - Check that we indeed currently prefer final releases. >>> zc.buildout.easy_install.prefer_final() True - no existing and newer dev available >>> prefer_final_permutation((), [1, '2a1']) downloaded 1 - no existing and only dev available >>> prefer_final_permutation((), ['3a1']) downloaded 3a1 - final existing and only dev acailable >>> prefer_final_permutation([4], ['5a1']) had 4 - final existing and newer final available >>> prefer_final_permutation([6], [7]) downloaded 7 - final existing and same final available >>> prefer_final_permutation([8], [8]) had 8 - final existing and older final available >>> prefer_final_permutation([10], [9]) had 10 - only dev existing and final available >>> prefer_final_permutation(['12a1'], [11]) downloaded 11 - only dev existing and no final available newer dev available >>> prefer_final_permutation(['13a1'], ['13a2']) downloaded 13a2 - only dev existing and no final available older dev available >>> prefer_final_permutation(['15a1'], ['14a1']) had 15a1 - only dev existing and no final available same dev available >>> prefer_final_permutation(['16a1'], ['16a1']) had 16a1 - Without prefer final: >>> _ = zc.buildout.easy_install.prefer_final(False) - no existing and newer dev available >>> prefer_final_permutation((), [18, '19a1']) downloaded 19a1 - no existing and only dev available >>> prefer_final_permutation((), ['20a1']) downloaded 20a1 - final existing and only dev acailable >>> prefer_final_permutation([21], ['22a1']) downloaded 22a1 - final existing and newer final available >>> prefer_final_permutation([23], [24]) downloaded 24 - final existing and same final available >>> prefer_final_permutation([25], [25]) had 25 - final existing and older final available >>> prefer_final_permutation([27], [26]) had 27 - only dev existing and final available >>> prefer_final_permutation(['29a1'], [28]) had 29a1 - only dev existing and no final available newer dev available >>> prefer_final_permutation(['30a1'], ['30a2']) downloaded 30a2 - only dev existing and no final available older dev available >>> prefer_final_permutation(['32a1'], ['31a1']) had 32a1 - only dev existing and no final available same dev available >>> prefer_final_permutation(['33a1'], ['33a1']) had 33a1 >>> _ = zc.buildout.easy_install.prefer_final(True) """ def buildout_prefer_final_option(): """ The prefer-final buildout option can be used for override the default preference for newer distributions. The default is prefer-final = true: >>> zc.buildout.easy_install.prefer_final() True >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... update-versions-file = versions-picked.cfg ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Installing ... ... written to versions-picked.cfg >>> cat('versions-picked.cfg') [versions] demo = 0.3 # Required by: # demo==0.3 demoneeded = 1.1 >>> remove('versions-picked.cfg') Here we see that the final versions of demo and demoneeded are used. We get the same behavior if we add prefer-final = true >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... prefer-final = true ... update-versions-file = versions-picked.cfg ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating ... ... written to versions-picked.cfg >>> cat('versions-picked.cfg') [versions] demo = 0.3 # Required by: # demo==0.3 demoneeded = 1.1 >>> remove('versions-picked.cfg') If we specify prefer-final = false, we'll get the newest distributions: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... prefer-final = false ... update-versions-file = versions-picked.cfg ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating ... ... written to versions-picked.cfg >>> cat('versions-picked.cfg') [versions] demo = 0.4rc1 # Required by: # demo==0.4rc1 demoneeded = 1.2rc1 >>> remove('versions-picked.cfg') We get an error if we specify anything but true or false: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... prefer-final = no ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = demo ... ''' % globals()) >>> print_(system(buildout+' -v'), end='') # doctest: +ELLIPSIS While: Initializing. Error: Invalid value for 'prefer-final' option: 'no' """ def wont_downgrade_due_to_prefer_final(): r""" If we install a non-final buildout version, we don't want to downgrade just because we prefer-final. If a buildout version isn't specified using a versions entry, then buildout's version requirement gets set to >=CURRENT_VERSION. >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... ''') >>> [v] = [l.split('= >=', 1)[1].strip() ... for l in system(buildout+' -vv').split('\n') ... if l.startswith('zc.buildout = >=')] >>> v == pkg_resources.working_set.find( ... pkg_resources.Requirement.parse('zc.buildout') ... ).version True >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... [versions] ... zc.buildout = >0.1 ... ''') >>> [str(l.split('= >', 1)[1].strip()) ... for l in system(buildout+' -vv').split('\n') ... if l.startswith('zc.buildout =')] ['0.1'] >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... versions = versions ... [versions] ... zc.buildout = 43 ... ''') >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Getting distribution for 'zc.buildout==43'. ... """ def develop_with_modules(): """ Distribution setup scripts can import modules in the distribution directory: >>> mkdir('foo') >>> write('foo', 'bar.py', ... '''# empty ... ''') >>> write('foo', 'setup.py', ... ''' ... import bar ... from setuptools import setup ... setup(name="foo") ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = foo ... parts = ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' >>> ls('develop-eggs') - foo.egg-link - zc.recipe.egg.egg-link """ def dont_pick_setuptools_if_version_is_specified_when_required_by_src_dist(): r""" When installing a source distribution, we got setuptools without honoring our version specification. >>> mkdir('dist') >>> write('setup.py', ... ''' ... from setuptools import setup ... setup(name='foo', version='1', py_modules=['foo'], zip_safe=True) ... ''') >>> write('foo.py', '') >>> _ = system(buildout+' setup . sdist') >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = dist ... versions = versions ... allow-picked-versions = false ... ... [versions] ... wtf = %s ... foo = 1 ... ... [foo] ... recipe = zc.recipe.egg ... eggs = foo ... ''' % ('\n'.join( ... '%s = %s' % (d.key, d.version) ... for d in zc.buildout.easy_install.buildout_and_setuptools_dists))) >>> print_(system(buildout), end='') Installing foo. Getting distribution for 'foo==1'. Got foo 1. """ def pyc_and_pyo_files_have_correct_paths(): r""" >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = demo ... interpreter = py ... ''' % globals()) >>> _ = system(buildout) >>> write('t.py', ... r''' ... import eggrecipedemo, eggrecipedemoneeded, sys ... code = lambda f: f.__code__ ... sys.stdout.write(code(eggrecipedemo.main).co_filename+'\n') ... sys.stdout.write(code(eggrecipedemoneeded.f).co_filename+'\n') ... ''') >>> print_(system(join('bin', 'py')+ ' t.py'), end='') /sample-buildout/eggs/v5/demo-0.3-py2.4.egg/eggrecipedemo.py /sample-buildout/eggs/v5/demoneeded-1.1-py2.4.egg/eggrecipedemoneeded.py """ def dont_mess_with_standard_dirs_with_variable_refs(): """ >>> write('buildout.cfg', ... ''' ... [buildout] ... eggs-directory = ${buildout:directory}/develop-eggs ... eggs-directory-version = ... parts = ... ''' % globals()) >>> print_(system(buildout), end='') """ def expand_shell_patterns_in_develop_paths(): """ Sometimes we want to include a number of eggs in some directory as develop eggs, without explicitly listing all of them in our buildout.cfg >>> make_dist_that_requires(sample_buildout, 'sampley') >>> make_dist_that_requires(sample_buildout, 'samplez') Now, let's create a buildout that has a shell pattern that matches both: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = sample* ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = sampley ... samplez ... ''' % globals()) We can see that both eggs were found: >>> print_(system(buildout), end='') Develop: '/sample-buildout/sampley' Develop: '/sample-buildout/samplez' Installing eggs. """ def warn_users_when_expanding_shell_patterns_yields_no_results(): """ Sometimes shell patterns do not match anything, so we want to warn our users about it... >>> make_dist_that_requires(sample_buildout, 'samplea') So if we have 2 patterns, one that has a matching directory, and another one that does not >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = samplea grumble* ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = samplea ... ''' % globals()) We should get one of the eggs, and a warning for the pattern that did not match anything. >>> print_(system(buildout), end='') Develop: '/sample-buildout/samplea' Couldn't develop '/sample-buildout/grumble*' (not found) Installing eggs. """ def make_sure_versions_dont_cancel_extras(): """ There was a bug that caused extras in requirements to be lost. >>> with open('setup.py', 'w') as f: ... _ = f.write(''' ... from setuptools import setup ... setup(name='extraversiondemo', version='1.0', ... url='x', author='x', author_email='x', ... extras_require=dict(foo=['demo']), py_modules=['t']) ... ''') >>> open('README', 'w').close() >>> open('t.py', 'w').close() >>> sdist('.', sample_eggs) >>> mkdir('dest') >>> ws = zc.buildout.easy_install.install( ... ['extraversiondemo[foo]'], 'dest', links=[sample_eggs], ... versions = dict(extraversiondemo='1.0') ... ) >>> sorted(dist.key for dist in ws) ['demo', 'demoneeded', 'extraversiondemo'] """ def increment_buildout_options(): r""" >>> write('b1.cfg', ''' ... [buildout] ... parts = p1 ... x = 1 ... y = a ... b ... ... [p1] ... recipe = zc.buildout:debug ... foo = ${buildout:x} ${buildout:y} ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = b1.cfg ... parts += p2 ... x += 2 ... y -= a ... ... [p2] ... <= p1 ... ''') >>> print_(system(buildout), end='') Installing p1. foo='1\n2 b' recipe='zc.buildout:debug' Installing p2. foo='1\n2 b' recipe='zc.buildout:debug' """ def increment_buildout_with_multiple_extended_files_421022(): r""" >>> write('foo.cfg', ''' ... [buildout] ... foo-option = foo ... [other] ... foo-option = foo ... ''') >>> write('bar.cfg', ''' ... [buildout] ... bar-option = bar ... [other] ... bar-option = bar ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... parts = p other ... extends = bar.cfg foo.cfg ... bar-option += baz ... foo-option += ham ... ... [other] ... recipe = zc.buildout:debug ... bar-option += baz ... foo-option += ham ... ... [p] ... recipe = zc.buildout:debug ... x = ${buildout:bar-option} ${buildout:foo-option} ... ''') >>> print_(system(buildout), end='') Installing p. recipe='zc.buildout:debug' x='bar\nbaz foo\nham' Installing other. bar-option='bar\nbaz' foo-option='foo\nham' recipe='zc.buildout:debug' """ def increment_on_command_line(): r""" >>> write('buildout.cfg', ''' ... [buildout] ... parts = p1 ... x = 1 ... y = a ... b ... ... [p1] ... recipe = zc.buildout:debug ... foo = ${buildout:x} ${buildout:y} ... ... [p2] ... <= p1 ... ''') >>> print_(system(buildout+' buildout:parts+=p2 p1:foo+=bar'), end='') Installing p1. foo='1 a\nb\nbar' recipe='zc.buildout:debug' Installing p2. foo='1 a\nb\nbar' recipe='zc.buildout:debug' """ def test_constrained_requirement(): """ zc.buildout.easy_install._constrained_requirement(constraint, requirement) Transforms an environment by applying a constraint. Here's a table of examples: >>> from zc.buildout.easy_install import IncompatibleConstraintError >>> examples = [ ... # original, constraint, transformed ... ('x', '1', 'x==1'), ... ('x>1', '2', 'x==2'), ... ('x>3', '2', IncompatibleConstraintError), ... ('x>1', '>2', 'x>1,>2'), ... ] >>> from zc.buildout.easy_install import _constrained_requirement >>> for o, c, e in examples: ... try: ... o = pkg_resources.Requirement.parse(o) ... if isinstance(e, str): ... e = pkg_resources.Requirement.parse(e) ... g = _constrained_requirement(c, o) ... except IncompatibleConstraintError: ... g = IncompatibleConstraintError ... if str(g) != str(e): ... print_('failed', o, c, g, '!=', e) """ def test_distutils_scripts_using_import_are_properly_parsed(): """ zc.buildout.easy_install._distutils_script(path, dest, script_content, initialization, rsetup): Creates a script for a distutils based project. In this example for a hypothetical code quality checker called 'pyflint' that uses an import statement to import its code. >>> pyflint_script = '''#!/path/to/bin/python ... import pyflint.do_something ... pyflint.do_something() ... ''' >>> import sys >>> original_executable = sys.executable >>> sys.executable = 'python' >>> from zc.buildout.easy_install import _distutils_script >>> generated = _distutils_script('\\'/path/test/\\'', 'bin/pyflint', pyflint_script, '', '') >>> if sys.platform == 'win32': ... generated == ['bin/pyflint.exe', 'bin/pyflint-script.py'] ... else: ... generated == ['bin/pyflint'] True >>> if sys.platform == 'win32': ... cat('bin/pyflint-script.py') ... else: ... cat('bin/pyflint') #!python import sys sys.path[0:0] = [ '/path/test/', ] import pyflint.do_something pyflint.do_something() >>> sys.executable = original_executable """ def test_distutils_scripts_using_from_are_properly_parsed(): """ zc.buildout.easy_install._distutils_script(path, dest, script_content, initialization, rsetup): Creates a script for a distutils based project. In this example for a hypothetical code quality checker called 'pyflint' that uses a from statement to import its code. >>> pyflint_script = '''#!/path/to/bin/python ... from pyflint import do_something ... do_something() ... ''' >>> import sys >>> original_executable = sys.executable >>> sys.executable = 'python' >>> from zc.buildout.easy_install import _distutils_script >>> generated = _distutils_script('\\'/path/test/\\'', 'bin/pyflint', pyflint_script, '', '') >>> if sys.platform == 'win32': ... generated == ['bin/pyflint.exe', 'bin/pyflint-script.py'] ... else: ... generated == ['bin/pyflint'] True >>> if sys.platform == 'win32': ... cat('bin/pyflint-script.py') ... else: ... cat('bin/pyflint') #!python import sys sys.path[0:0] = [ '/path/test/', ] from pyflint import do_something do_something() >>> sys.executable = original_executable """ def want_new_zcrecipeegg(): """ >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = egg ... [egg] ... recipe = zc.recipe.egg <2dev ... eggs = demo ... ''') >>> print_(system(join('bin', 'buildout')), end='') # doctest: +ELLIPSIS ... """ def macro_inheritance_bug(): """ There was a bug preventing a section from using another section as a macro if that section was extended with macros, and both sections were listed as parts (phew!). The following contrived example demonstrates that this now works. >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo bar ... [base] ... recipe = zc.recipe.egg ... [foo] ... <=base ... eggs = zc.buildout ... interpreter = python ... [bar] ... <=foo ... interpreter = py ... ''') >>> print_(system(join('bin', 'buildout')), end='') # doctest: +ELLIPSIS Installing foo. ... Installing bar. ... >>> ls("./bin") - buildout - py - python """ def bootstrap_honors_relative_paths(): """ >>> working = tmpdir('working') >>> cd(working) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... relative-paths = true ... ''') >>> _ = system(buildout+' bootstrap') >>> cat('bin', 'buildout') # doctest: +ELLIPSIS #!/usr/local/bin/python2.7 import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) base = os.path.dirname(base) import sys sys.path[0:0] = [ ... ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) """ def cant_use_install_from_cache_and_offline_together(): r""" >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... offline = true ... install-from-cache = true ... ''') >>> print_(system(join('bin', 'buildout')), end='') # doctest: +ELLIPSIS While: Initializing. Error: install-from-cache can't be used with offline mode. Nothing is installed, even from cache, in offline mode, which might better be called 'no-install mode'. """ def error_installing_in_offline_mode_if_dont_have_needed_dist(): r""" >>> import zc.buildout.easy_install >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], None, ... links=[link_server], index=link_server+'index/') Traceback (most recent call last): ... UserError: We don't have a distribution for demo==0.2 and can't install one in offline (no-install) mode. """ def error_building_in_offline_mode_if_dont_have_needed_dist(): r""" >>> zc.buildout.easy_install.build( ... 'extdemo', None, ... {}, links=[link_server], index=link_server+'index/') Traceback (most recent call last): ... UserError: We don't have a distribution for extdemo and can't build one in offline (no-install) mode. """ def test_buildout_section_shorthand_for_command_line_assignments(): r""" >>> write('buildout.cfg', '') >>> print_(system(buildout+' parts='), end='') # doctest: +ELLIPSIS """ def buildout_honors_umask(): """ For setting the executable permission, the user's umask is honored: >>> orig_umask = os.umask(0o077) # Only user gets permissions. >>> zc.buildout.easy_install._execute_permission() == 0o700 True >>> tmp = os.umask(0o022) # User can write, the rest not. >>> zc.buildout.easy_install._execute_permission() == 0o755 True >>> tmp = os.umask(orig_umask) # Reset umask to the original value. """ def parse_with_section_expr(): r""" >>> class Recipe: ... def __init__(self, buildout, *_): ... buildout.parse(''' ... [foo : sys.version_info[0] > 0] ... x = 1 ... ''') >>> buildout = zc.buildout.testing.Buildout() >>> buildout.parse(''' ... [foo : sys.version_info[0] > 0] ... x = 1 ... ''') >>> buildout.print_options() [foo] x = 1 """ def test_abi_tag_eggs(): r""" >>> write('buildout.cfg', ... ''' ... [buildout] ... find-links = %(sample_eggs)s ... parts = abi ... abi-tag-eggs = true ... [abi] ... recipe = zc.recipe.egg ... eggs = demo ... ''' % globals()) >>> _ = system(join('bin', 'buildout')) >>> from zc.buildout.pep425tags import get_abi_tag >>> abi_tag = get_abi_tag() >>> abi_tag in os.listdir(join(sample_buildout, 'eggs')) False >>> abi_tag in os.listdir(join(sample_buildout, 'eggs', 'v5')) True >>> ls('eggs', 'v5', abi_tag) d demo-0.3-py3.7.egg d demoneeded-1.1-py3.7.egg """ def test_buildout_doesnt_keep_adding_itself_to_versions(): r""" We were constantly writing to versions.cfg for buildout and setuptools >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... extends = versions.cfg ... show-picked-versions = true ... update-versions-file = versions.cfg ... extends = versions.cfg ... ''') >>> write('versions.cfg', ... '''[versions] ... ''') >>> _ = system(join('bin', 'buildout')) >>> with open('versions.cfg') as f: ... versions = f.read() >>> _ = system(join('bin', 'buildout')) On the first run, some pins were added: >>> cat('versions.cfg') # doctest: +ELLIPSIS [versions] >>> _ = system(join('bin', 'buildout')) >>> _ = system(join('bin', 'buildout')) Subsequent runs didn't add additional text: >>> with open('versions.cfg') as f: ... versions == f.read() True """ if sys.platform == 'win32': del buildout_honors_umask # umask on dohs is academic ###################################################################### def buildout_txt_setup(test): zc.buildout.testing.buildoutSetUp(test) mkdir = test.globs['mkdir'] index = os.environ['buildout_testing_index_url'][7:] test.globs['sample_eggs'] = index create_sample_eggs(test) index = Path(index) for name in os.listdir(index): if '-' in name: pname = name.split('-')[0] if not (index / pname).exists(): mkdir(index, pname) shutil.move(index / name, index / pname / name) dist = pkg_resources.working_set.find( pkg_resources.Requirement.parse('zc.recipe.egg')) mkdir(index, 'zc.recipe.egg') zc.buildout.testing.bdist_wheel( os.path.dirname(dist.location), index / 'zc.recipe.egg', ) sample_buildout = test.globs['sample_buildout'] # Copy recipes directory recipes_dir = HERE / 'recipes' shutil.copytree(recipes_dir, Path(sample_buildout) / 'recipes') def _silent_project_builder_runner(cmd, cwd=None, extra_environ=None): """Run the build command silently. The default runner is `pyproject_hooks.default_subprocess_runner`, which currently does this: env = os.environ.copy() if extra_environ: env.update(extra_environ) check_call(cmd, cwd=cwd, env=env) We do the same, but ignore the output. """ env = os.environ.copy() if extra_environ: env.update(extra_environ) subprocess.check_output(cmd, cwd=cwd, env=env, stderr=subprocess.STDOUT) def _build_in_isolated_env(distribution, source_dir, dest_dir): """Build distribution in isolated env. This is our variant of the _build_in_isolated_env function from the `build` module. """ if distribution not in {"sdist", "wheel"}: raise ValueError(f"Invalid argument passed: {distribution=}") with DefaultIsolatedEnv(installer="pip") as env: builder = ProjectBuilder.from_isolated_env( env, source_dir=source_dir, runner=_silent_project_builder_runner, ) env.install(builder.build_system_requires) env.install(builder.get_requires_for_build(distribution)) builder.build(distribution, dest_dir) egg_parse = re.compile(r'([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d+)$' ).match def makeNewRelease(project, ws, dest, versions=['91.0', '99.99']): """Make a new release for a project. Theoretically this can work for various projects, but currently we are only considering the project being zc.buildout. We expect that it is a real development install, no egg, so dist.location is buildout-repo/src. """ if project != "zc.buildout": print( f"WARNING: Unexpected project: {project} instead of zc.buildout.\n" "If makeNewRelease works, great, otherwise you have been warned." ) dist = ws.find(pkg_resources.Requirement.parse(project)) oldver = dist.version if os.path.isfile(dist.location): # We used to handle this, but not anymore. raise ValueError(f"We expected a directory, but got a file: {dist.location}") elif dist.location.endswith('site-packages'): # We used to handle this, but not anymore. raise ValueError(f"We don't expect a dist in site-packages anymore: {dist.location}") # So we have a directory and it is not in site-packages. This is currently # expected to be the only case we really need to handle. Let's get a proper # Path object. location = Path(dist.location) # For zc.buildout we need the parent dir. if location.name == "src": location = location.parent # First create a source dist in a temporary directory. with tempfile.TemporaryDirectory() as tempdir: tempdir = Path(tempdir) _build_in_isolated_env("sdist", location, tempdir) # Get the tarball and extract it. tarball = os.listdir(tempdir)[0] with tarfile.open(os.path.join(tempdir, tarball)) as tar: tar.extractall(path=tempdir) os.remove(os.path.join(tempdir, tarball)) # Now the only directory enty is what was extracted. extracted = tempdir / os.listdir(tempdir)[0] setup_py_path = extracted / 'setup.py' assert setup_py_path.exists() old_line = f'version = "{oldver}"' for version in versions: # In the setup.py we expect a line like this: # version = "4.2.0.dev0" info = setup_py_path.read_text() new_line = f'version = "{version}"' if old_line not in info: raise ValueError(f"Expected line not found: {old_line}") info = info.replace(old_line, new_line) old_line = new_line setup_py_path.write_text(info) # Now we need to build the new wheel distribution. _build_in_isolated_env("wheel", extracted, dest) def updateSetup(test): zc.buildout.testing.buildoutSetUp(test) new_releases = test.globs['tmpdir']('new_releases') test.globs['new_releases'] = new_releases # ws = getWorkingSetWithBuildoutEgg(test) ws = pkg_resources.working_set # now let's make the new releases makeNewRelease('zc.buildout', ws, new_releases) # os.mkdir(os.path.join(new_releases, 'zc.buildout')) def ancestor(path, level): while level > 0: path = os.path.dirname(path) level -= 1 return path def test_suite(): test_suite = [ manuel.testing.TestSuite( manuel.doctest.Manuel() + manuel.capture.Manuel(), 'configparser.test'), manuel.testing.TestSuite( manuel.doctest.Manuel( optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, # (re.compile(r"Installing 'zc.buildout >=\S+"), ''), (re.compile(r'__buildout_signature__ = recipes-\S+'), '__buildout_signature__ = recipes-SSSSSSSSSSS'), (re.compile(r'executable = [\S ]+python\S*', re.I), 'executable = python'), (re.compile(r'[-d] (setuptools|setuptools)-\S+[.]egg'), 'setuptools.egg'), (re.compile(r'pip-\S+[.]egg'), 'pip.egg'), (re.compile(r'wheel-\S+[.]egg'), 'wheel.egg'), (re.compile(r'zc.buildout(-\S+)?[.]egg(-link)?'), 'zc.buildout.egg'), (re.compile(r'creating \S*setup.cfg'), 'creating setup.cfg'), (re.compile(r'hello\%ssetup' % os.path.sep), 'hello/setup'), (re.compile(r'Picked: (\S+) = \S+'), 'Picked: \\1 = V.V'), (re.compile(r'We have a develop egg: zc.buildout (\S+)'), 'We have a develop egg: zc.buildout X.X.'), (re.compile(r'\\[\\]?'), '/'), (re.compile('WindowsError'), 'OSError'), (re.compile(r'\[Error \d+\] Cannot create a file ' r'when that file already exists: '), '[Errno 17] File exists: ' ), (re.compile('setuptools'), 'setuptools'), (re.compile(r'Got zc.recipe.egg \S+'), 'Got zc.recipe.egg'), (re.compile(r'zc\.(buildout|recipe\.egg)\s*= >=\S+'), 'zc.\\1 = >=1.99'), (re.compile(r'installed_develop_eggs = (.*\n(\s+.*\n)*)'), lambda m: 'installed_develop_eggs = ' + ' '.join([f for f in m.group(1).split() if f.endswith('.egg-link')]) + '\n'), ]) ) + manuel.capture.Manuel(), 'buildout.txt', 'configuration.txt', 'extending.txt', 'options.txt', 'init.txt', 'extensions.txt', setUp=buildout_txt_setup, tearDown=zc.buildout.testing.buildoutTearDown, ), doctest.DocFileSuite( 'runsetup.txt', 'repeatable.txt', 'setup.txt', setUp=zc.buildout.testing.buildoutSetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, # (re.compile(r"Installing 'zc.buildout >=\S+"), ''), # (re.compile(r"Getting distribution for 'zc.buildout >=\S+"), # ''), (re.compile(r'__buildout_signature__ = recipes-\S+'), '__buildout_signature__ = recipes-SSSSSSSSSSS'), (re.compile(r'[-d] setuptools-\S+[.]egg'), 'setuptools.egg'), (re.compile(r'zc.buildout(-\S+)?[.]egg(-link)?'), 'zc.buildout.egg'), (re.compile(r'creating \S*setup.cfg'), 'creating setup.cfg'), (re.compile(r'hello\%ssetup' % os.path.sep), 'hello/setup'), (re.compile(r'Picked: (\S+) = \S+'), 'Picked: \\1 = V.V'), (re.compile(r'We have a develop egg: zc.buildout (\S+)'), 'We have a develop egg: zc.buildout X.X.'), (re.compile(r'\\[\\]?'), '/'), (re.compile(r'WindowsError'), 'OSError'), (re.compile(r'pip = \S+'), 'pip = 20.0.1'), (re.compile(r'setuptools = \S+'), 'setuptools = 46.1.3'), (re.compile(r'\[Error 17\] Cannot create a file ' r'when that file already exists: '), '[Errno 17] File exists: ' ), (re.compile('executable = %s' % re.escape(sys.executable)), 'executable = python'), (re.compile(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{6}'), 'YYYY-MM-DD hh:mm:ss.dddddd'), (re.compile(r'installed_develop_eggs = (.*\n(\s+.*\n)*)'), lambda m: 'installed_develop_eggs = ' + ' '.join([f for f in m.group(1).split() if f.endswith('.egg-link')]) + '\n'), ]), ), doctest.DocFileSuite( 'debugging.txt', setUp=zc.buildout.testing.buildoutSetUp, tearDown=zc.buildout.testing.buildoutTearDown, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, (re.compile('zc.buildout.buildout.MissingOption'), 'MissingOption'), (re.compile(r'\S+buildout.py'), 'buildout.py'), (re.compile(r'line \d+'), 'line NNN'), (re.compile(r'py\(\d+\)'), 'py(NNN)'), ]) ), doctest.DocFileSuite( 'easy_install.txt', 'downloadcache.txt', 'dependencylinks.txt', 'allowhosts.txt', 'allow-unknown-extras.txt', setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.normalize_open_in_generated_script, zc.buildout.testing.adding_find_link, zc.buildout.testing.not_found, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, normalize_bang, (re.compile(r'[-d] setuptools-\S+[.]egg'), 'setuptools.egg'), (re.compile(r'\\[\\]?'), '/'), (re.compile('(\n?)- ([a-zA-Z_.-]+)\n- \\2.exe\n'), '\\1- \\2\n'), ]), ), doctest.DocFileSuite( 'download.txt', 'extends-cache.txt', setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, (re.compile(' at -?0x[^>]+'), ''), (re.compile('http://localhost:[0-9]{4,5}/'), 'http://localhost/'), (re.compile('[0-9a-f]{32}'), ''), zc.buildout.testing.normalize_path, zc.buildout.testing.ignore_not_upgrading, ]), ), doctest.DocTestSuite( setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize___pycache__, zc.buildout.testing.not_found, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, zc.buildout.testing.ignore_native_namespace_warning_1, zc.buildout.testing.ignore_native_namespace_warning_2, zc.buildout.testing.ignore_native_namespace_warning_3, zc.buildout.testing.ignore_native_namespace_warning_4, zc.buildout.testing.ignore_native_namespace_warning_5, normalize_bang, (re.compile(r'^(\w+\.)*(Missing\w+: )'), '\2'), (re.compile(r"buildout: Running \S*setup.py"), 'buildout: Running setup.py'), (re.compile(r'pip-\S+-'), 'pip.egg'), (re.compile(r'setuptools-\S+-'), 'setuptools.egg'), (re.compile(r'zc.buildout-\S+-'), 'zc.buildout.egg'), (re.compile(r'pip = \S+'), 'pip = 20.0.0'), (re.compile(r'setuptools = \S+'), 'setuptools = 0.7.99'), (re.compile(r'File "\S+one.py"'), 'File "one.py"'), (re.compile(r'We have a develop egg: (\S+) (\S+)'), r'We have a develop egg: \1 V'), (re.compile(r'Picked: setuptools = \S+'), 'Picked: setuptools = V'), (re.compile('[-d] pip'), '- pip'), (re.compile('[-d] setuptools'), '- setuptools'), (re.compile(r'\\[\\]?'), '/'), (re.compile( '-q develop -mxN -d "/sample-buildout/develop-eggs'), '-q develop -mxN -d /sample-buildout/develop-eggs' ), (re.compile(r'^[*]...'), '...'), # for # bug_92891 # bootstrap_crashes_with_egg_recipe_in_buildout_section (re.compile(r"Unused options for buildout: 'eggs' 'scripts'\."), "Unused options for buildout: 'scripts' 'eggs'."), # fix for test_distutils_scripts_using_import_are_properly_parsed # and test_distutils_scripts_using_from_are_properly_parsed # win32 apparently adds a " around sys.executable (re.compile('#!"python"'), '#!python'), ]), ), zc.buildout.rmtree.test_suite(), doctest.DocFileSuite( 'windows.txt', setUp=zc.buildout.testing.buildoutSetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, (re.compile(r'__buildout_signature__ = recipes-\S+'), '__buildout_signature__ = recipes-SSSSSSSSSSS'), (re.compile(r'[-d] setuptools-\S+[.]egg'), 'setuptools.egg'), (re.compile(r'zc.buildout(-\S+)?[.]egg(-link)?'), 'zc.buildout.egg'), (re.compile(r'creating \S*setup.cfg'), 'creating setup.cfg'), (re.compile(r'hello\%ssetup' % os.path.sep), 'hello/setup'), (re.compile(r'Picked: (\S+) = \S+'), 'Picked: \\1 = V.V'), (re.compile(r'We have a develop egg: zc.buildout (\S+)'), 'We have a develop egg: zc.buildout X.X.'), (re.compile(r'\\[\\]?'), '/'), (re.compile('WindowsError'), 'OSError'), (re.compile(r'\[Error 17\] Cannot create a file ' r'when that file already exists: '), '[Errno 17] File exists: ' ), (re.compile(r'installed_develop_eggs = (.*\n(\s+.*\n)*)'), lambda m: 'installed_develop_eggs = ' + ' '.join([f for f in m.group(1).split() if f.endswith('.egg-link')]) + '\n'), ]) ), doctest.DocFileSuite('testing_bugfix.txt'), ] if not sys.platform.startswith('win'): # In the update.txt tests on Windows, instead of # "Upgraded: zc.buildout version NINETYNINE.NINETYNINE;" # we get: # "Not upgrading because not running a local buildout command." # I don't know why that is only the case on Windows. test_suite.append( doctest.DocFileSuite( 'update.txt', setUp=updateSetup, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ (re.compile(r'(zc.buildout|setuptools)-\d+[.]\d+\S*' r'-py\d.\d+.egg'), '\\1.egg'), zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, normalize_bang, (re.compile(r"Getting distribution for 'zc.buildout>=\S+"), ''), (re.compile('99[.]99'), 'NINETYNINE.NINETYNINE'), (re.compile( r'(zc.buildout|setuptools|pip)( version)? \d+[.]\d+\S*'), '\\1 V.V'), (re.compile('[-d] setuptools'), '- setuptools'), (re.compile('[-d] pip'), '- pip'), (re.compile('[-d] wheel'), '- wheel'), (re.compile(re.escape(os.path.sep)+'+'), '/'), (re.compile(r'installed_develop_eggs = (.*\n(\s+.*\n)*)'), lambda m: 'installed_develop_eggs = ' + ' '.join([f for f in m.group(1).split() if f.endswith('.egg-link')]) + '\n'), ]) ) ) docdir = os.path.join(ancestor(__file__, 5), 'doc') if os.path.exists(docdir) and not sys.platform.startswith('win'): # Note that the purpose of the documentation tests are mainly # to test the documentation, not to test buildout. def docSetUp(test): def write(text, *path): with open(os.path.join(*path), 'w') as f: f.write(text) def ls(d='.', *rest): names = os.listdir(os.path.join(d, *rest)) if os.environ.get('BUILDOUT_TESTING_SHOW_PEP660') == '1': return names return [n for n in names if not ( n.endswith('.dist-info') or n.endswith('.egg-info') or n.endswith('.pth') or (n.startswith('__editable__') and n.endswith('.py')) or n == '__pycache__' )] test.globs.update( run_buildout=zc.buildout.testing.run_buildout_in_process, yup=lambda cond, orelse='Nope': None if cond else orelse, nope=lambda cond, orelse='Nope': orelse if cond else None, eq=lambda a, b: None if a == b else (a, b), eqs=zc.buildout.testing.eqs, read=zc.buildout.testing.read, write=write, ls=ls, join=os.path.join, clear_here=zc.buildout.testing.clear_here, os=os, ) setupstack.setUpDirectory(test) test_suite.append( manuel.testing.TestSuite( manuel.doctest.Manuel( optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, ]), ) + manuel.capture.Manuel(), os.path.join(docdir, 'getting-started.rst'), os.path.join(docdir, 'reference.rst'), os.path.join(docdir, 'topics', 'bootstrapping.rst'), os.path.join(docdir, 'topics', 'implicit-parts.rst'), os.path.join( docdir, 'topics', 'variables-extending-and-substitutions.rst'), os.path.join(docdir, 'topics', 'writing-recipes.rst'), os.path.join(docdir, 'topics', 'optimizing.rst'), os.path.join(docdir, 'topics', 'meta-recipes.rst'), setUp=docSetUp, tearDown=setupstack.tearDown )) test_suite.append(unittest.defaultTestLoader.loadTestsFromName(__name__)) return unittest.TestSuite(test_suite) if __name__ == '__main__': unittest.main() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4035728 zc_buildout-6.0.1/src/zc/buildout/tests/test_extras.py0000644000076500000240000001144315264141650022572 0ustar00mauritsstaff# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2020 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest import re from zope.testing import renormalizing import zc.buildout.testing from zc.buildout.tests import easy_install_SetUp from zc.buildout.tests import normalize_bang def install_extras_with_greater_than_constrains(): """ There was a bug that caused extras in requirements to be lost. >>> working = tmpdir('working') >>> cd(working) >>> mkdir('dependency') >>> cd('dependency') >>> with open('setup.py', 'w') as f: ... _ = f.write(''' ... from setuptools import setup ... setup(name='dependency', version='1.0', ... url='x', author='x', author_email='x', ... py_modules=['t']) ... ''') >>> open('README', 'w').close() >>> open('t.py', 'w').close() >>> sdist('.', sample_eggs) >>> cd(working) >>> mkdir('extras') >>> cd('extras') >>> with open('setup.py', 'w') as f: ... _ = f.write(''' ... from setuptools import setup ... setup(name='extraversiondemo', version='1.0', ... url='x', author='x', author_email='x', ... extras_require=dict(foo=['dependency']), py_modules=['t']) ... ''') >>> open('README', 'w').close() >>> open('t.py', 'w').close() >>> sdist('.', sample_eggs) >>> mkdir('dest') >>> ws = zc.buildout.easy_install.install( ... ['extraversiondemo[foo]'], 'dest', links=[sample_eggs], ... versions = dict(extraversiondemo='1.0', dependency='>0.9') ... ) >>> sorted(dist.key for dist in ws) ['dependency', 'extraversiondemo'] """ def test_suite(): return doctest.DocTestSuite( setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize___pycache__, zc.buildout.testing.not_found, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, normalize_bang, (re.compile(r'^(\w+\.)*(Missing\w+: )'), '\2'), (re.compile(r"buildout: Running \S*setup.py"), 'buildout: Running setup.py'), (re.compile(r'pip-\S+-'), 'pip.egg'), (re.compile(r'setuptools-\S+-'), 'setuptools.egg'), (re.compile(r'zc.buildout-\S+-'), 'zc.buildout.egg'), (re.compile(r'pip = \S+'), 'pip = 20.0.0'), (re.compile(r'setuptools = \S+'), 'setuptools = 0.7.99'), (re.compile(r'File "\S+one.py"'), 'File "one.py"'), (re.compile(r'We have a develop egg: (\S+) (\S+)'), r'We have a develop egg: \1 V'), (re.compile(r'Picked: setuptools = \S+'), 'Picked: setuptools = V'), (re.compile('[-d] pip'), '- pip'), (re.compile('[-d] setuptools'), '- setuptools'), (re.compile(r'\\[\\]?'), '/'), (re.compile( '-q develop -mxN -d "/sample-buildout/develop-eggs'), '-q develop -mxN -d /sample-buildout/develop-eggs' ), (re.compile(r'^[*]...'), '...'), # for # bug_92891 # bootstrap_crashes_with_egg_recipe_in_buildout_section (re.compile(r"Unused options for buildout: 'eggs' 'scripts'\."), "Unused options for buildout: 'scripts' 'eggs'."), # Python 3.4 changed the wording of NameErrors (re.compile('NameError: global name'), 'NameError: name'), # fix for test_distutils_scripts_using_import_are_properly_parsed # and test_distutils_scripts_using_from_are_properly_parsed # win32 apparently adds a " around sys.executable (re.compile('#!"python"'), '#!python'), ]), ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4036586 zc_buildout-6.0.1/src/zc/buildout/tests/test_increment.py0000644000076500000240000003203615264141650023251 0ustar00mauritsstaff# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2020 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest import re from zope.testing import renormalizing import zc.buildout.testing from zc.buildout.tests import easy_install_SetUp from zc.buildout.tests import normalize_bang def default_cfg(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [debug] ... dec = 1 ... 2 ... inc = 1 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... ... [debug] ... dec -= 2 ... inc += 2 ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate debug', env=env), end='') Annotated sections ================== [debug] dec= 1 /home/.buildout/default.cfg -= buildout.cfg inc= 1 2 /home/.buildout/default.cfg += buildout.cfg """ def default_cfg_extensions(): r""" Add two extensions as develop eggs >>> mkdir('demo') >>> write('demo', 'demo.py', ''' ... import sys ... def ext(buildout): ... sys.stdout.write('demo %s %s\\n' % ('ext', sorted(buildout))) ... def unload(buildout): ... sys.stdout.write('demo %s %s\\n' % ('unload', sorted(buildout))) ... ''') >>> write('demo', 'setup.py', ''' ... from setuptools import setup ... ... setup( ... name = "demo", ... entry_points = { ... 'zc.buildout.extension': ['ext = demo:ext'], ... 'zc.buildout.unloadextension': ['ext = demo:unload'], ... }, ... ) ... ''') >>> mkdir('demo2') >>> write('demo2', 'demo2.py', ''' ... import sys ... def ext(buildout): ... sys.stdout.write('demo2 %s %s\\n' % ('ext', sorted(buildout))) ... def unload(buildout): ... sys.stdout.write('demo2 %s %s\\n' % ('unload', sorted(buildout))) ... ''') >>> write('demo2', 'setup.py', ''' ... from setuptools import setup ... ... setup( ... name = "demo2", ... entry_points = { ... 'zc.buildout.extension': ['ext = demo2:ext'], ... 'zc.buildout.unloadextension': ['ext = demo2:unload'], ... }, ... ) ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... develop = demo demo2 ... parts = ... ''') Run buildout once without extensions to actually develop the eggs. (Develop happens after loading extensions.) >>> print_(system(buildout), end='') Develop: '/sample-buildout/demo' Develop: '/sample-buildout/demo2' >>> ls("develop-eggs") - demo.egg-link - demo2.egg-link - zc.recipe.egg.egg-link extensions in .buildout/default.cfg incremented in buildout.cfg >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... develop = demo demo2 ... extensions += demo2 ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += buildout.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_base(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('base.cfg', ''' ... [buildout] ... extensions += demo2 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += base.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_base2(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('base.cfg', ''' ... [buildout] ... ''') >>> write('base2.cfg', ''' ... [buildout] ... extensions += demo2 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... base2.cfg ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += base2.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_base2_and_base3(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('base.cfg', ''' ... [buildout] ... ''') >>> write('base2.cfg', ''' ... [buildout] ... extensions += demo2 ... ''') >>> write('base3.cfg', ''' ... [buildout] ... extensions += demo3 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... base2.cfg ... base3.cfg ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 demo3 /home/.buildout/default.cfg += base2.cfg += base3.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_buildout(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('base.cfg', ''' ... [buildout] ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... extensions += demo2 ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += buildout.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_buildout_with_base_and_root(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('root.cfg', ''' ... [buildout] ... ''') >>> write('base.cfg', ''' ... [buildout] ... extends = root.cfg ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... extensions += demo2 ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += buildout.cfg ... versions= versions DEFAULT_VALUE """ def no_default_with_extends_increment_in_base2_and_base3(): r""" >>> write('base.cfg', ''' ... [buildout] ... ''') >>> write('base2.cfg', ''' ... [buildout] ... extensions += demo2 ... ''') >>> write('base3.cfg', ''' ... [buildout] ... extensions += demo3 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... base2.cfg ... base3.cfg ... parts = ... ''') >>> print_(system(buildout+' annotate buildout'), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== [buildout] ... extensions= demo2 demo3 IMPLICIT_VALUE += base2.cfg += base3.cfg ... versions= versions DEFAULT_VALUE """ def increment_buildout_with_multiple_extended_without_base_equals(): r""" >>> write('buildout.cfg', ''' ... [buildout] ... extends = base1.cfg base2.cfg ... parts += foo ... [foo] ... recipe = zc.buildout:debug ... [base1] ... recipe = zc.buildout:debug ... [base2] ... recipe = zc.buildout:debug ... ''') >>> write('base1.cfg', ''' ... [buildout] ... extends = base3.cfg ... parts += base1 ... ''') >>> write('base2.cfg', ''' ... [buildout] ... extends = base3.cfg ... parts += base2 ... ''') >>> write('base3.cfg', ''' ... [buildout] ... ''') >>> print_(system(buildout), end='') Installing base1. recipe='zc.buildout:debug' Installing base2. recipe='zc.buildout:debug' Installing foo. recipe='zc.buildout:debug' """ def test_suite(): return doctest.DocTestSuite( setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize___pycache__, zc.buildout.testing.not_found, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, normalize_bang, (re.compile(r'^(\w+\.)*(Missing\w+: )'), '\2'), (re.compile(r"buildout: Running \S*setup.py"), 'buildout: Running setup.py'), (re.compile(r'pip-\S+-'), 'pip.egg'), (re.compile(r'setuptools-\S+-'), 'setuptools.egg'), (re.compile(r'zc.buildout-\S+-'), 'zc.buildout.egg'), (re.compile(r'pip = \S+'), 'pip = 20.0.0'), (re.compile(r'setuptools = \S+'), 'setuptools = 0.7.99'), (re.compile(r'File "\S+one.py"'), 'File "one.py"'), (re.compile(r'We have a develop egg: (\S+) (\S+)'), r'We have a develop egg: \1 V'), (re.compile(r'Picked: setuptools = \S+'), 'Picked: setuptools = V'), (re.compile('[-d] pip'), '- pip'), (re.compile('[-d] setuptools'), '- setuptools'), (re.compile(r'\\[\\]?'), '/'), (re.compile( '-q develop -mxN -d "/sample-buildout/develop-eggs'), '-q develop -mxN -d /sample-buildout/develop-eggs' ), (re.compile(r'^[*]...'), '...'), # for # bug_92891 # bootstrap_crashes_with_egg_recipe_in_buildout_section (re.compile(r"Unused options for buildout: 'eggs' 'scripts'\."), "Unused options for buildout: 'scripts' 'eggs'."), # Python 3.4 changed the wording of NameErrors (re.compile('NameError: global name'), 'NameError: name'), # fix for test_distutils_scripts_using_import_are_properly_parsed # and test_distutils_scripts_using_from_are_properly_parsed # win32 apparently adds a " around sys.executable (re.compile('#!"python"'), '#!python'), ]), ) ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1792066472.40371 zc_buildout-6.0.1/src/zc/buildout/tests/test_pep660.py0000644000076500000240000000471515264141650022310 0ustar00mauritsstaffimport unittest import zc.buildout.testing import os class TestPEP660(unittest.TestCase): def setUp(self): self.globs = {} zc.buildout.testing.buildoutSetUp(self) self.mkdir = self.globs['mkdir'] self.write = self.globs['write'] self.system = self.globs['system'] def tearDown(self): zc.buildout.testing.buildoutTearDown(self) def test_develop_pep660(self): os.environ['BUILDOUT_TESTING_SHOW_PEP660'] = '1' try: # Create a PEP 660 compliant package (pyproject.toml only) self.mkdir('pep660_pkg') self.mkdir('pep660_pkg', 'src') self.mkdir('pep660_pkg', 'src', 'pep660_pkg') self.write('pep660_pkg', 'src', 'pep660_pkg', '__init__.py', 'def main(): print("Hello from PEP 660")') self.write('pep660_pkg', 'pyproject.toml', """ [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [project] name = "pep660_pkg" version = "0.1.0" """) self.write('buildout.cfg', """ [buildout] develop = pep660_pkg parts = """) output = self.system(os.path.join('bin', 'buildout')) self.assertIn("Develop: ", output) self.assertIn("pep660_pkg", output) # Check develop-eggs dev_eggs = os.listdir('develop-eggs') # Check if there is any .dist-info dist_infos = [d for d in dev_eggs if d.endswith('.dist-info')] self.assertGreater(len(dist_infos), 0, "Metadata should be present after fix") # Check if there is any .pth pths = [d for d in dev_eggs if d.endswith('.pth')] self.assertGreater(len(pths), 0, ".pth file should be present after fix") # Now verify that buildout process itself can import it if we add it to sys.path import sys sys.path.insert(0, os.path.abspath('develop-eggs')) # We need to process the .pth files import site site.addsitedir(os.path.abspath('develop-eggs')) try: import pep660_pkg # Success except ImportError: self.fail("Could not import pep660_pkg after processing .pth files") finally: if 'BUILDOUT_TESTING_SHOW_PEP660' in os.environ: del os.environ['BUILDOUT_TESTING_SHOW_PEP660'] if __name__ == '__main__': unittest.main() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4037764 zc_buildout-6.0.1/src/zc/buildout/tests/test_unit.py0000644000076500000240000002416015264141650022243 0ustar00mauritsstafffrom pathlib import Path import tempfile import unittest _oneliner1 = """import__('pkg_resources').declare_namespace(__name__)""" _oneliner2 = """__path__ = __import__("pkgutil").extend_path(__path__, __name__)""" _pkg_resources = """from pkg_resources import declare_namespace declare_namespace(__name__) """ _pkgutil = """from pkgutil import extend_path __path__ = extend_path(__path__, __name__) """ _both = """# See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages try: __import__("pkg_resources").declare_namespace(__name__) except ImportError: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) """ _comments = """# See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages # try: # __import__("pkg_resources").declare_namespace(__name__) # except ImportError: # from pkgutil import extend_path # # __path__ = extend_path(__path__, __name__) """ class TestFunctions(unittest.TestCase): """Test case for some functions.""" def test_check_namespace_init_file_and_find_namespace_init_files(self): from zc.buildout.easy_install import check_namespace_init_file from zc.buildout.easy_install import find_namespace_init_files with tempfile.TemporaryDirectory() as package_dir: init = Path(package_dir) / "__init__.py" str_init = str(init) # Test with empty __init__.py file init.write_text("") self.assertFalse(check_namespace_init_file(init)) self.assertListEqual(find_namespace_init_files(package_dir), []) # Test with non-empty __init__.py file init.write_text("# Non-empty init file") self.assertFalse(check_namespace_init_file(init)) self.assertListEqual(find_namespace_init_files(package_dir), []) # Test with one-line pkg_resources namespace declaration init.write_text(_oneliner1) self.assertTrue(check_namespace_init_file(init)) self.assertListEqual(find_namespace_init_files(package_dir), [str_init]) # Test with one-line pkgutil namespace declaration init.write_text(_oneliner2) self.assertTrue(check_namespace_init_file(init)) self.assertListEqual(find_namespace_init_files(package_dir), [str_init]) # Test with multiline pkg_resources namespace declaration init.write_text(_pkg_resources) self.assertTrue(check_namespace_init_file(init)) self.assertListEqual(find_namespace_init_files(package_dir), [str_init]) # Test with multiline pkgutil namespace declaration init.write_text(_pkgutil) self.assertTrue(check_namespace_init_file(init)) self.assertListEqual(find_namespace_init_files(package_dir), [str_init]) # Test with multiple namespace declarations init.write_text(_both) self.assertTrue(check_namespace_init_file(init)) self.assertListEqual(find_namespace_init_files(package_dir), [str_init]) # Test with commented-out namespace declarations init.write_text(_comments) self.assertFalse(check_namespace_init_file(init)) self.assertListEqual(find_namespace_init_files(package_dir), []) def test_find_namespace_init_files(self): from zc.buildout.easy_install import find_namespace_init_files with tempfile.TemporaryDirectory() as package_dir: # Create structure: src/plone/app/example # where plone and app are namespace packages. package_dir = Path(package_dir) (package_dir / "src" / "plone" / "app" / "example").mkdir(parents=True) plone_init = package_dir / "src" / "plone" / "__init__.py" plone_app_init = package_dir / "src" / "plone" / "app" / "__init__.py" example_init = package_dir / "src" / "plone" / "app" / "example" / "__init__.py" # In all cases, the example package is not a namespace package, # but it will have an __init__.py file. example_init.write_text("# Non-empty init file") # Native namespaces. self.assertListEqual(find_namespace_init_files(package_dir), []) # pkg_resources namespaces. plone_init.write_text(_oneliner1) plone_app_init.write_text(_oneliner1) self.assertListEqual( find_namespace_init_files(package_dir), [str(plone_init), str(plone_app_init)] ) # pkg_util namespaces. plone_init.write_text(_oneliner2) plone_app_init.write_text(_oneliner2) self.assertListEqual( find_namespace_init_files(package_dir), [str(plone_init), str(plone_app_init)] ) class TestVendoredPkgResources(unittest.TestCase): """Tests for the pkg_resources copy vendored in zc.buildout._vendor. See src/zc/buildout/_vendor/README.rst and https://github.com/buildout/buildout/issues/685 """ def test_pkg_resources_importable(self): # Importing zc.buildout guarantees that `import pkg_resources` # works afterwards, whatever setuptools version is installed: # either something imported a real pkg_resources first # (setuptools < 82) or the vendored copy was aliased. import sys import zc.buildout # noqa import pkg_resources self.assertIs(sys.modules['pkg_resources'], pkg_resources) self.assertTrue(hasattr(pkg_resources, 'WorkingSet')) def test_alias_in_fresh_process(self): # In a fresh process, importing zc.buildout installs the vendored # copy as `pkg_resources` - unless a real pkg_resources was # imported before, which old setuptools versions (< 66) do # themselves during `import setuptools`; then the guard keeps # that copy to preserve module identity. import os import subprocess import sys import textwrap import zc.buildout.easy_install env = dict(os.environ) env['PYTHONPATH'] = os.pathsep.join( zc.buildout.easy_install.buildout_and_setuptools_path) code = textwrap.dedent(""" import sys import setuptools pre_imported = 'pkg_resources' in sys.modules import zc.buildout import pkg_resources name = sys.modules['pkg_resources'].__name__ if pre_imported: assert name == 'pkg_resources', name else: assert name == 'zc.buildout._vendor.pkg_resources', name """) subprocess.check_call([sys.executable, '-c', code], env=env) def test_resource_string_finds_setuptools_cli_exe(self): # easy_install.py reads setuptools' cli.exe launcher via # pkg_resources.resource_string when generating interpreter # scripts on Windows. This must keep working with the vendored # pkg_resources, whatever the location of setuptools. import zc.buildout # noqa import pkg_resources data = pkg_resources.resource_string('setuptools', 'cli.exe') self.assertEqual(data[:2], b'MZ') def test_vendored_copy_is_functional(self): from zc.buildout._vendor import pkg_resources as vendored req = vendored.Requirement.parse('foo.bar>=1.0') self.assertEqual(req.project_name, 'foo.bar') self.assertEqual(req.key, 'foo.bar') ws = vendored.WorkingSet() self.assertIsNone(ws.find(vendored.Requirement.parse('does-not-exist'))) def test_jaraco_text_helpers(self): from zc.buildout._vendor.jaraco_text import drop_comment from zc.buildout._vendor.jaraco_text import join_continuation from zc.buildout._vendor.jaraco_text import yield_lines self.assertEqual(drop_comment('foo # bar'), 'foo') self.assertEqual( drop_comment('http://example.com/foo#bar'), 'http://example.com/foo#bar') self.assertEqual( list(join_continuation(['foo \\', 'bar', 'baz'])), ['foobar', 'baz']) self.assertEqual( list(yield_lines('\nfoo\n#bar\nbaz #comment')), ['foo', 'baz #comment']) self.assertEqual( list(yield_lines(['foo\nbar', 'baz', 'bing\n\n\n'])), ['foo', 'bar', 'baz', 'bing']) class TestSetuptoolsConstraint(unittest.TestCase): """Tests for how an unpinned setuptools requirement is constrained. Since zc.buildout vendors its own pkg_resources copy, an unpinned setuptools is no longer restricted for regular installs. Only distributions with pkg_resources-style namespace packages still get setuptools<82 (the last version shipping pkg_resources) added to their working set, because their scripts import pkg_resources at script run time. See https://github.com/buildout/buildout/issues/750 """ def test_constrain_leaves_unpinned_setuptools_alone(self): import pkg_resources from zc.buildout.easy_install import Installer installer = Installer() requirement = installer._constrain( pkg_resources.Requirement.parse('setuptools')) self.assertEqual(list(requirement.specs), []) def test_constrain_applies_user_pin_to_setuptools(self): import pkg_resources from zc.buildout.easy_install import Installer installer = Installer(versions={'setuptools': '81.0.0'}) requirement = installer._constrain( pkg_resources.Requirement.parse('setuptools')) self.assertEqual(str(requirement), 'setuptools==81.0.0') def test_namespace_dists_get_setuptools_below_82(self): # The building block used by Installer._maybe_add_setuptools when # the user did not pin setuptools. import pkg_resources from zc.buildout.easy_install import _constrained_requirement requirement = _constrained_requirement( '<82', pkg_resources.Requirement.parse('setuptools')) self.assertEqual(str(requirement), 'setuptools<82') self.assertIn('81.0.0', requirement) self.assertNotIn('82.0.0', requirement) self.assertNotIn('83.0.0', requirement) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4038267 zc_buildout-6.0.1/src/zc/buildout/tests/testing_bugfix.txt0000644000076500000240000000234115264141650023432 0ustar00mauritsstaffBug fixes in zc.buildout.testing ================================ Logging handler which did not get deleted ----------------------------------------- The buildout testing set up runs a buildout which adds a ``logging.StreamHandler`` to the root logger. But tear down did not remove it. This can disturb other tests of packages reusing zc.buildout.testing. The handlers before calling set up are: >>> import logging >>> count = len(logging.getLogger().handlers) >>> logging.getLogger().handlers # doctest: +ELLIPSIS [<...NullHandler...>] After calling it, a ``logging.StreamHandler`` was added: >>> import zc.buildout.testing >>> import doctest >>> test = doctest.DocTestParser().get_doctest( ... '>>> x', {}, 'foo', 'foo.py', 0) >>> zc.buildout.testing.buildoutSetUp(test) >>> len(logging.getLogger().handlers) == count + 1 True >>> logging.getLogger().handlers # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE [<...NullHandler...StreamHandler...>] But tear down removes the new logging handler: >>> zc.buildout.testing.buildoutTearDown(test) >>> len(logging.getLogger().handlers) == count True >>> logging.getLogger().handlers # doctest: +ELLIPSIS [<...NullHandler...>] ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792066472.403901 zc_buildout-6.0.1/src/zc/buildout/tests/update.txt0000644000076500000240000002155515264141650021703 0ustar00mauritsstaffAutomatic Buildout Updates ========================== When a buildout is run, one of the first steps performed is to check for updates to either zc.buildout or setuptools. To demonstrate this, we've created some "new releases" of buildout and setuptools in a new_releases folder: >>> ls(new_releases) - zc_buildout-91.0-py3-none-any.whl - zc_buildout-NINETYNINE.NINETYNINE-py3-none-any.whl Let's update the sample buildout.cfg to look in this area: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = show-versions ... develop = showversions ... ... [show-versions] ... recipe = showversions ... """ % dict(new_releases=new_releases)) We'll also include a recipe that echos the versions of setuptools and zc.buildout used: >>> mkdir(sample_buildout, 'showversions') >>> write(sample_buildout, 'showversions', 'showversions.py', ... """ ... import pkg_resources ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... ... class Recipe: ... ... def __init__(self, buildout, name, options): ... pass ... ... def install(self): ... for project in ['zc.buildout']: ... req = pkg_resources.Requirement.parse(project) ... print_(project, pkg_resources.working_set.find(req).version) ... return () ... update = install ... """) >>> write(sample_buildout, 'showversions', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "showversions", ... entry_points = {'zc.buildout': ['default = showversions:Recipe']}, ... ) ... """) The installed zc.buildout version is in development mode, so it won't upgrade itself. >>> print_(system(buildout), end='') Develop:... Installing show-versions. zc.buildout V.V But if we run the buildout and tell it to use version 50, the buildout will upgrade itself: >>> print_(system(buildout + ' versions:zc.buildout=91.0'), end='') Getting distribution for 'zc.buildout==91.0'. Got zc.buildout V.V Upgraded: zc.buildout V.V Restarting. Generated script '/sample-buildout/bin/buildout'. Develop: '/sample-buildout/showversions' Updating show-versions. zc.buildout V.V Now if we run the buildout without explicit version, the buildout will upgrade itself to the newest version found in new releases: >>> print_(system(buildout), end='') Got zc.buildout NINETYNINE.NINETYNINE. Upgraded: zc.buildout version NINETYNINE.NINETYNINE; Restarting. Generated script '/sample-buildout/bin/buildout'. Develop: '/sample-buildout/showversions' Updating show-versions. zc.buildout NINETYNINE.NINETYNINE Our buildout script has been updated to use the new eggs: >>> cat(sample_buildout, 'bin', 'buildout') #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-buildout/eggs/v5/zc.buildout-NINETYNINE.NINETYNINE-pyN.N.egg', ... ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) Now, let's recreate the sample buildout. If we specify constraints on the versions of zc.buildout and setuptools to use, running the buildout will install earlier versions of these packages: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = show-versions ... develop = showversions ... ... [versions] ... zc.buildout = < 99 ... ... [show-versions] ... recipe = showversions ... """ % dict(new_releases=new_releases)) Now we can see that we actually "upgrade" to an earlier version. >>> print_(system(buildout), end='') Upgraded: zc.buildout V.V Restarting. Generated script '/sample-buildout/bin/buildout'. Develop: '/sample-buildout/showversions' Updating show-versions. zc.buildout V.V There are a number of cases, described below, in which the updates don't happen. We won't upgrade in offline mode: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = show-versions ... develop = showversions ... ... [show-versions] ... recipe = showversions ... """ % dict(new_releases=new_releases)) >>> print_(system(buildout+' -o'), end='') Develop: '/sample-buildout/showversions' Updating show-versions. zc.buildout 1.0.0 Or in non-newest mode: >>> print_(system(buildout+' -N'), end='') Develop: '/sample-buildout/showversions' Updating show-versions. zc.buildout 1.0.0 We also won't upgrade if the buildout script being run isn't in the buildouts bin directory. To see this we'll create a new buildout directory: >>> sample_buildout2 = tmpdir('sample_buildout2') >>> write(sample_buildout2, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = ... ... [versions] ... zc.buildout = 99.99 ... """ % dict(new_releases=new_releases)) >>> cd(sample_buildout2) >>> print_(system(buildout), end='') Creating directory '/sample_buildout2/eggs/v5'. Creating directory '/sample_buildout2/bin'. Creating directory '/sample_buildout2/parts'. Creating directory '/sample_buildout2/develop-eggs'. Getting distribution for 'zc.buildout==NINETYNINE.NINETYNINE'. Got zc.buildout NINETYNINE.NINETYNINE. Not upgrading because not running a local buildout command. >>> ls('bin') .. The relative-paths option is honored: >>> cd(sample_buildout) >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = show-versions ... develop = showversions ... relative-paths = true ... ... [show-versions] ... recipe = showversions ... """ % dict(new_releases=new_releases)) >>> print_(system(buildout), end='') Upgraded: zc.buildout version NINETYNINE.NINETYNINE; Restarting. Generated script '/sample-buildout/bin/buildout'. Develop: '/sample-buildout/showversions' Section `buildout` contains unused option(s): 'relative-paths'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Updating show-versions. zc.buildout NINETYNINE.NINETYNINE >>> cat('bin', 'buildout') # doctest +ELL #!/usr/local/bin/python2.7 import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) base = os.path.dirname(base) import sys sys.path[0:0] = [ join(base, 'eggs/v5/zc.buildout-NINETYNINE.NINETYNINE-pyN.N.egg'), ... ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) When buildout restarts and the restarted buildout exits with an error code, the original buildout that called the second buildout also exits with that error code. Otherwise build scripts can erroneously detect a successful buildout run even if it failed. Make a recipe that fails: >>> mkdir(sample_buildout, 'failrecipe') >>> write(sample_buildout, 'failrecipe', 'failrecipe.py', ... """ ... import pkg_resources ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... ... class Recipe: ... ... def __init__(self, buildout, name, options): ... sys.exit('recipe sys-exits') ... ... def install(self): ... pass ... ... update = install ... """) >>> write(sample_buildout, 'failrecipe', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "failrecipe", ... entry_points = {'zc.buildout': ['default = failrecipe:Recipe']}, ... ) ... """) Let's downgrade again, triggering a restart. And use the failing recipe that gives us a sys.exit: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = fail ... develop = failrecipe ... ... [versions] ... zc.buildout = < 99 ... ... [fail] ... recipe = failrecipe ... """ % dict(new_releases=new_releases)) Run the buildout: >>> print_(system(buildout, with_exit_code=True), end='') Upgraded: zc.buildout V.V Restarting. Generated script '/sample-buildout/bin/buildout'. Develop: '/sample-buildout/failrecipe' recipe sys-exits EXIT CODE: 1 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4039521 zc_buildout-6.0.1/src/zc/buildout/tests/windows.txt0000644000076500000240000000375115264141650022111 0ustar00mauritsstaffzc.buildout on MS-Windows ========================= Certain aspects of every software project are dependent on the operating system used. The same - of course - applies to zc.buildout. To test that Windows doesn't get in the way, we'll test some system dependent aspects. The following recipe will create a read-only file which shutil.rmtree can't delete. >>> mkdir('recipe') >>> write('recipe', 'recipe.py', ... ''' ... import os ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... class Recipe: ... def __init__(self, buildout, name, options): ... self.location = os.path.join( ... buildout['buildout']['parts-directory'], ... name) ... ... def install(self): ... print_("can't remove read only files") ... if not os.path.exists (self.location): ... os.makedirs (self.location) ... ... name = os.path.join (self.location, 'readonly.txt') ... with open (name, 'w') as f: f.write ('this is a read only file') ... os.chmod(name, 256) ... return () ... ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='spam', version='1', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> write('recipe', 'README', '') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... and we'll configure a buildout to use it: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') Getting distribution for 'spam'. Got spam 1. Installing foo. can't remove read only files ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066472.4039998 zc_buildout-6.0.1/src/zc/buildout/utils.py0000644000076500000240000000300115264141650020212 0ustar00mauritsstafffrom importlib.metadata import version import packaging.version import re import os # In some cases we need to check the setuptools version to know what we can do. SETUPTOOLS_VERSION = packaging.version.parse(version("setuptools")) IS_SETUPTOOLS_80_PLUS = SETUPTOOLS_VERSION >= packaging.version.Version('80') def normalize_name(name): """PEP 503 normalization plus dashes as underscores. Taken over from importlib.metadata. I don't want to think about where to import this from in each Python version, or having it as extra dependency. Note that there is also packaging_utils.canonicalize_name which turns "foo.bar" into "foo-bar", so it is different. """ return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_') def get_pth_paths(loc): """Scan directory for .pth files and return referenced paths.""" paths = [] if os.path.isdir(loc): for name in os.listdir(loc): if name.endswith('.pth'): pth_path = os.path.join(loc, name) try: with open(pth_path, 'r') as f: for line in f: line = line.strip() if line and not line.startswith('#') and not line.startswith('import '): p = os.path.abspath(os.path.join(loc, line)) if p not in paths: paths.append(p) except Exception: pass return paths ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.2089791 zc_buildout-6.0.1/src/zc.buildout.egg-info/0000755000076500000240000000000015264141651020200 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.1992724 zc_buildout-6.0.1/src/zc.buildout.egg-info/PKG-INFO0000644000076500000240000005570315264141651021307 0ustar00mauritsstaffMetadata-Version: 2.4 Name: zc.buildout Version: 6.0.1 Summary: System for managing development buildouts Home-page: http://buildout.org Author: Jim Fulton Author-email: jim@zope.com License: ZPL 2.1 Keywords: development build Classifier: Development Status :: 6 - Mature Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Zope Public License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: 3.13 Classifier: Programming Language :: Python :: 3.14 Classifier: Programming Language :: Python :: 3.15 Classifier: Topic :: Software Development :: Build Tools Classifier: Topic :: Software Development :: Libraries :: Python Modules Requires-Python: >=3.10 License-File: LICENSE.txt Requires-Dist: setuptools>=75.8.2 Requires-Dist: packaging>=23.2 Requires-Dist: pip>=25.0 Requires-Dist: platformdirs Requires-Dist: wheel Provides-Extra: test Requires-Dist: zope.testing; extra == "test" Requires-Dist: manuel; extra == "test" Requires-Dist: bobo==2.3.0; extra == "test" Requires-Dist: zdaemon; extra == "test" Requires-Dist: zc.zdaemonrecipe; extra == "test" Requires-Dist: zc.recipe.deployment; extra == "test" Requires-Dist: hatchling; extra == "test" Requires-Dist: build; extra == "test" Dynamic: author Dynamic: author-email Dynamic: classifier Dynamic: description Dynamic: home-page Dynamic: keywords Dynamic: license Dynamic: license-file Dynamic: provides-extra Dynamic: requires-dist Dynamic: requires-python Dynamic: summary ******** Buildout ******** .. image:: https://github.com/buildout/buildout/actions/workflows/run-tests.yml/badge.svg :alt: GHA tests report :target: https://github.com/buildout/buildout/actions/workflows/run-tests.yml Buildout is a project designed to solve 2 problems: 1. Application-centric assembly and deployment *Assembly* runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.). Buildout might be confused with build tools like make or ant, but it is a little higher level and might invoke systems like make or ant to get its work done. Buildout might be confused with systems like puppet or chef, but it is more application focused. Systems like puppet or chef might use buildout to get their work done. Buildout is also somewhat Python-centric, even though it can be used to assemble and deploy non-python applications. It has some special features for assembling Python programs. It's scripted with Python, unlike, say puppet or chef, which are scripted with Ruby. 2. Repeatable assembly of programs from Python software distributions Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren't locked down, or in a deployment environment where dependency versions are fully specified. You should be able to check buildout into a VCS and later check it out. Two checkouts built at the same time in the same environment should always give the same result, regardless of their history. Among other things, after a buildout, all dependencies should be at the most recent version consistent with any version specifications expressed in the buildout. Buildout supports applications consisting of multiple programs, with different programs in an application free to use different versions of Python distributions. This is in contrast with a Python installation (real or virtual), where, for any given distribution, there can only be one installed. To learn more about buildout, including how to use it, see https://www.buildout.org/. Major changes in 6.x ******************** * Require ``setuptools >= 75.8.2``. * Require ``pip >= 25.0``. * Require Python 3.10 as a minimum. * Vendor ``pkg_resources`` from setuptools 81.0.0 (the last release that contained it). * Lifted the ``setuptools < 82`` restriction. This was there because we need ``pkg_resources``, but we now have our own copy. * Fix support for PEP 660 develop packages (``pyproject.toml`` only). See the change history below for more details. What does this mean in practice? * You can now develop packages that use other build systems than ``setuptools``. This means for example: `hatchling `_, `flit `_, `pdm `_. ``setuptools`` as build system still works of course And since ``zc.buildout`` 5 you already don't need a ``setup.py``, but can use ``pyproject.toml`` for all your package configuration. * This comes at a cost: packages that still use the deprecated ``pkg_resources`` style namespaces can cause problems: * When installed as standard package, all should still be well. * When used in development as package that simply gets pulled in by a recipe, it should still work. * But if it shares a namespace (for example ``collective.*``) with another package, and the other package uses the modern native namespaces, and one or both of the packages is a buildout extension or buildout recipe, then Buildout will most likely quit with a ``ModuleNotFound`` error. In that case, you should migrate the package to use native namespaces, or go back to ``zc.buildout`` 5, and maybe install ``horse-with-no-namespace``. See the next section. Since ``zc.buildout`` is used a lot in Plone, as one of the ways to install it: * Plone 6.2 will keep using ``zc.buildout`` 5 by default. * Plone 6.3 will most likely use ``zc.buildout`` 6. * In your own projects you are free to use ``zc.buildout`` 6 on Plone 6.2 already. * But beware: if you use ``collective.recipe.omelette`` and you develop a ``collective`` package that still uses ``pkg_resources`` namespaces, you will run into the problems described above. Native namespaces and breaking changes in 5.x ********************************************* Summary: ``zc.buildout`` 5.x installs most distributions with pip, even editable installs (with some remarks). It automatically uses native namespaces for all packages, except editable installs. Eggs go to directory ``eggs/v5`` to avoid compatibility problems. If a package name has a dot in it, it is a namespace package. ``zc.buildout`` has ``zc`` as namespace. Namespaces can be implemented in three ways: native namespaces (PEP 420, since Python 3.3), pkg_resources style (with ``__init__.py`` files that call ``pkg_resources.declare_namespace``) and pkgutil style (with ``__init__.py`` files that call ``pkgutil.extend_path``). Native namespaces are the modern way to do it. They work better with pip and other modern tools. The pkg_resources style depends on setuptools and is considered deprecated: setuptools 82 dropped support for it, removing its foundational ``pkg_resources`` module. ``zc.buildout`` ships its own copy of ``pkg_resources`` (taken from setuptools 81, see ``src/zc/buildout/_vendor/README.rst``), so buildout itself, recipes and extensions can keep using it inside a buildout run with any setuptools version. Scripts generated for distributions that use pkg_resources-style namespaces still need a real ``pkg_resources`` at script run time, so buildout prefers adding ``setuptools < 82`` to their working set (unless you pin setuptools yourself). When no such setuptools version can be found or installed, buildout falls back to the available setuptools; if that is 82 or later, those scripts will not find ``pkg_resources`` at run time. Problems start when you have multiple packages in the same namespace, that use different implementations. But it depends on what you use to install the packages. In the following examples, we have two packages in the same namespace, say ``ns.native`` (using native namespaces) and ``ns.deprecated`` (using pkg_resources style). * Make editable installs of both packages (``pip install -e`` or in buildout, ``develop =``): - This works neither in pip nor in buildout. - You can install the `horse-with-no-namespace `_ package to get this working. * Make a normal install of both packages: - This works fine in pip. - This fails in buildout 4.x. - This works fine in buildout 5.x. * Make a normal install of one package and an editable install of the other: - This works fine in pip. - This fails in buildout 4.x. - This fails in buildout 5.x as well. But again, you can use ``horse-with-no-namespace`` to get this working. So the big difference between buildout 4.x and 5.x is that with 5.x you can mix namespace styles when you do normal installs. This is possible due to the following major changes in 5.x: * The ``zc.buildout`` package itself uses native namespaces now. * ``zc.buildout`` 5.x installs most packages with pip, with only rare exceptions. * The tests have mostly been changed to use wheels instead of eggs, so they more closely resemble real life. * Buildout now automatically treats all namespace packages as having native namespaces. Previously, after we installed a ``pkg_resources-style`` namespace package with pip, the ``__init__.py`` files would be missing, so we would explicitly add them again. Now we no longer do this. In fact, in some cases these files *are* there after installation, and we explicitly *remove* them. * This means that "eggs" created by this Buildout version are not compatible with eggs from a previous Buildout version, at least for namespace packages. So Buildout now stores the eggs in a sub directory: ``eggs/v5``. Or with abi tags for example: ``eggs/v5/cp313``. So it should be fine to keep using the same shared eggs cache, even if you are using different zc.buildout versions. * Development eggs (editable installs) are now also installed by pip. Previously, zc.buildout would call ``python setup.py develop`` to install them, failing if there was no ``setup.py`` file. Now this file is no longer needed. Theoretically this means that you could develop a package that uses for example ``hatchling`` as build system. In practice this does not work yet, but the foundation is there. The Plone project is a major user of buildout, although alternative installation methods are becoming more popular. So what does this mean for Plone? * Plone 6.1 and earlier will keep using ``zc.buildout`` 4.x and ``pkg_resources``-style namespaces * Plone 6.2 will switch to ``zc.buildout`` 5.x. * Plone 6.2 will temporarily use ``horse-with-no-namespace`` both when installed with ``pip`` and with ``zc.buildout``. * This combination means that each package in Plone 6.2 can switch to native namespaces at its own pace. Previously, all packages in a namespace had to switch at the same time, and the tests would be broken until that happened. * In your own projects on any Plone 6 version, you are free to use any buildout and setuptools version you want. If Buildout 5 works on your Plone 6.0 project, or Buildout 4 works on your Plone 6.2 project, that is fine. Change History ************** .. You should *NOT* be adding new change log entries to this file. You should create a file in the news directory instead. For helpful instructions, please see: https://github.com/buildout/buildout/blob/master/doc/ADD-A-NEWS-ITEM.rst .. towncrier release notes start 6.0.1 (2026-10-15) ------------------ Bug fixes: - Also hide the deprecation warning when importing the vendored ``pkg_resources``. [icemac] - Fix ``DeprecationWarning`` about ``setDaemon()`` in ``zc.buildout.testing``. [icemac] 6.0.0 (2026-10-06) ------------------ New features: - Add support for python316-319 in conditional section expressions. [maurits] - Test on Python 3.15 and setuptools 84.0.0. [maurits] Bug fixes: - Don't show warnings from importing ``pkg_resources``. We know it is deprecated, but we have vendorized it. [maurits] Tests: - Add simple but real test projects. [maurits] 6.0.0a1 (2026-08-14) -------------------- Breaking changes: - Require ``setuptools >= 75.8.2``, up from ``61.0.0``. That release fixed ``pkg_resources.WorkingSet.find`` to consider several spellings of a requirement name, which is what most of buildout's remaining ``setuptools`` compatibility code worked around. Consequently the following patches are gone from ``zc.buildout.patches``: ``patch_pkg_resources_working_set_find`` (back-ported the 75.8.2 fix), ``patch_interpret_distro_name`` (only needed below setuptools 70, and already inoperative because it imported a module that had been renamed) and ``patch_Distribution`` (disabled for years). Require ``pip >= 25.0``, and test only against the latest pip release of each year. Require Python 3.10 as a minimum; 3.9 is out of security support. Use Python 3.13 instead of 3.10 for the CI jobs that vary something other than the Python version. See `issue 755 `_. (#755) New features: - Vendor ``pkg_resources`` from setuptools 81.0.0 (the last release that contained it) as ``zc.buildout._vendor.pkg_resources`` and install it as ``sys.modules['pkg_resources']`` when ``zc.buildout`` is imported. Buildout, recipes and extensions keep using the ``pkg_resources`` API inside a buildout run without needing setuptools to provide it. See `issue 685 `_. (#685) - Lifted the ``setuptools < 82`` restriction: both the ``setuptools`` requirement of ``zc.buildout`` and the automatic constraint applied to an unpinned ``setuptools`` during installs are gone, thanks to the vendored ``pkg_resources`` copy. Environments can now use setuptools 82 and later, including 83.0.0 which fixes the published vulnerability PYSEC-2026-3447 (GHSA-h35f-9h28-mq5c). Exception: distributions using pkg_resources-style namespace packages preferably get ``setuptools < 82`` added to their working set (unless setuptools is pinned), because their generated scripts import ``pkg_resources`` at script run time. When no such setuptools version is available, buildout falls back to the installed one. See `issue 750 `_. (#750) Bug fixes: - Fix support for PEP 660 develop packages (pyproject.toml only) by preserving .dist-info and processing .pth files. (pep660) Tests: - Fix detecting Windows in our ``prepare.sh`` script. [maurits] - Fix script tests. [maurits] - Test with ``pip`` 26.1.2. [maurits] - Update GitHub workflow action versions. [maurits] 5.2.0 (2026-04-29) ------------------ New features: - Restrict ``setuptools`` to less than 82 if it is not pinned. When ``zc.buildout`` checks if it should upgrade itself or ``setuptools``, under some circumstances this could lead to a too new ``setuptools`` version getting installed. [maurits] Tests: - Test with ``pip`` 26.1. [maurits] 5.1.3 (2026-03-06) ------------------ Bug fixes: - Replace 4 bare except clauses with except Exception. [haosenwang1018] 5.1.2 (2026-02-10) ------------------ Bug fixes: - Accept ``setuptools`` 81. Require ``setuptools<82``. [maurits] 5.1.1 (2025-11-25) ------------------ Bug fixes: - Store buildout index url in the installer, so pip can use it. [maurits] (#731) - When installing packages in development, treat the name as a file uri. Otherwise ``pip install -e package_name`` will look on PyPI, instead of a local ``package_name`` directory. [maurits] (#734) 5.1.0 (2025-11-20) ------------------ New features: - Support Python 3.14. No changes were needed. [maurits] (#314) - Warn when old-style namespaces are used in packages under development. [maurits] (#729) Bug fixes: - Require ``setuptools<81``. We need the ``pkg_resources`` module which is scheduled for removal in 81. [maurits] (#81) 5.0.0 (2025-11-12) ------------------ Tests: - Test with ``pip`` 25.3. (#727) 5.0.0a3 (2025-09-24) -------------------- Bug fixes: - Fix logic in detecting namespace init files for deletion. [maurits] (#720) 5.0.0a2 (2025-09-11) -------------------- Bug fixes: - Fix reading metadata files from dist-info in Windows. [maurits] (#722) 5.0.0a1 (2025-09-11) -------------------- Breaking changes: - Install development eggs (editable installs) using pip. Theoretically this means that you could develop a package that uses for example ``hatchling`` as build system. In practice this does not work yet, but the foundation is there. [maurits] (#676) - Install all namespace packages as native namespaces. [maurits] (#676) - Store eggs in a sub directory: ``eggs/v5``. Or with abi tags for example: ``eggs/v5/cp313``. [maurits] (#676) - Install most packages with pip, with only rare exceptions. [maurits] (#676) - The ``zc.buildout`` package itself uses native namespaces now. [maurits] (#676) - Require at least ``setuptools`` version 61.0.0. This is needed due to the changes in the test setup. [maurits] Tests: - The tests have mostly been changed to use wheels instead of eggs, so they more closely resemble real life. [maurits] (#676) - Removed inactive tests for no longer existing ``bootstrap.py``. [maurits] - Split the ``buildout.txt`` test file into multiple files. [maurits] 4.1.12 (2025-06-11) ------------------- Bug fixes: - Fix error ``get_win_launcher`` not found on Windows on setuptools 80.3+. [maurits] (#713) 4.1.11 (2025-06-11) ------------------- Bug fixes: - Fix development installs to still work when using setuptools 80.0.0. From then on, setuptools internally calls ``pip install --editable``. Note that "distutils scripts" can no longer be detected with setuptools 80. This seems an ancient technology, and probably hardly used. [maurits] (#708) - Use a copy of ``package_index.py`` from ``setuptools`` 80.2.0. This fixes compatibility with ``setuptools`` 80.3.0 where this module was removed. Merged some of our patches into this copy. [maurits] (#710) 4.1.10 (2025-05-21) ------------------- Bug fixes: - Override `pkg_resources.Environment.can_add` to have better results on Mac. Without this, a freshly created Mac-specific egg may not be considered compatible. This can happen when the Python you use was built on a different Mac OSX version. [maurits] (#609) 4.1.9 (2025-04-09) ------------------ Bug fixes: - Fix accidental changes to ``PYTHONPATH`` in ``os.environ`` when calling ``pip install``. [xavth] (#639) Tests - Use ``wheel`` 0.45.1 when testing with ``setuptools`` older than 70.1.0. Otherwise, when combining an older ``setuptools`` with a newer ``wheel`` version, the ``bdist_wheel`` command exists in neither of these packages. [maurits] (#705) 4.1.8 (2025-04-09) ------------------ Bug fixes: - Use the canonical name of a package when checking for a version constraint. [maurits] (#689) - Get actual project name from dist. Use this for naming the egg that gets created after installing a wheel or after doing a pip install of a source dist. [maurits] (#695) - Log all http errors when processing package url. [maurits] (#1013) 4.1.7 (2025-04-08) ------------------ Bug fixes: - Prevent getting package pages twice. Since version 4.1.5 we first request normalized package url on PyPI servers, but a subsequent check needed a fix. [maurits] (#634) - No longer recompile py files if we moved the dist. This code was never updated for Python 3, where the `.pyc` files are in a `__pycache__` directory, so it had no effect. [maurits] (#699) - Require at least `packaging` version 23.2. Needed because we use the `utils.is_normalized_name` function. [maurits] (#700) 4.1.6 (2025-04-03) ------------------ Tests - While creating sample packages for testing, mostly create wheels instead of eggs. For the sample source distributions, create ``tar.gz`` instead of ``zip`` files. Then our package index for testing is more like the actual PyPI. [maurits] (#675) 4.1.5 (2025-03-31) ------------------ Bug fixes: - Implement PEP 503: request normalized package url on PyPI servers. [andreclimaco] (#634) - Install ``wheel`` before ``setuptools`` when checking if an upgrade and restart are needed. [maurits] (#691) 4.1.4 (2025-03-07) ------------------ Bug fixes: - If needed, copy and rename wheels before making an egg out of them. This helps for wheels of namespace packages created with ``setuptools`` 75.8.1 or higher. For namespace package we need a dot instead of an underscore in the resulting egg name. [maurits] (#686) 4.1.3 (2025-03-05) ------------------ Bug fixes: - Patch the ``find`` method from ``pkg_resources.WorkingSet``. Let this use the code from ``setuptools`` 75.8.2, if the currently used version is older. This is better at finding installed distributions. But don't patch ``setuptools`` versions older than 61: the new version of the method would give an error there. [maurits] (#682) 4.1.2 (2025-03-05) ------------------ Bug fixes: - Fix error finding the ``zc.buildout`` distribution when checking if we need to upgrade/restart. This depends on your ``setuptools`` version. [maurits] (#681) 4.1.1 (2025-03-04) ------------------ Bug fixes: - Fix error adding minimum ``zc.buildout`` version as requirement. [maurits] (#679) 4.1 (2025-03-04) ---------------- New features: - In the ``ls`` testing method, add keyword argument ``lowercase_and_sort_output``. The default is False, so no change. When true, as the name says, it sorts the output by lowercase, and prints it lowercase. We need this in one test because with ``setuptools`` 75.8.1 we no longer have a filename ``MIXEDCASE-0.5-pyN.N.egg``, but ``mixedcase-0.5-pyN.N.egg``. [maurits] (#7581) Bug fixes: - When trying to find a distribution for ``package.name``, first try the normalized name (``package_name``). This fixes an error finding entry points for namespace packages. The error is: ``TypeError: ('Expected str, Requirement, or Distribution', None)``. [maurits] (#7581) Development: - Test with latest ``setuptools`` 75.8.2 and with ``pip`` 25.0.1. Note that ``setuptools`` 75.8.1 can be troublesome and should be avoided. [maurits] (#7581) 4.0 (2025-01-30) ---------------- Breaking changes: - Drop Python 3.8 support. Require 3.9 as minimum. (#38) Development: - Test against `setuptools == 75.6.0`. (#671) 4.0.0a1 (2024-10-22) -------------------- Breaking changes: - Add dependency on ``packaging``. This gets rid of ugly compatibility code. [maurits] (#38) - Require ``setuptools >= 49.0.0``. This is the first version that supports PEP 496 environment markers, for example ``demo ==0.1; python_version < '3.9'``. An earlier change had ``setuptools >= 42.0.2``, otherwise we got ImportErrors. Also, since this is higher than 38.2.3, we are sure to have support for wheels. Remove support for ``distribute``, which was probably already broken. [maurits] (#38) - Drop support for Python 2. Require Python 3.8 as minimum. [maurits] (#38) New features: - Support Python 3.12 and 3.13. This only needed a few test fixes. [maurits] (#38) ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1792066473.20512 zc_buildout-6.0.1/src/zc.buildout.egg-info/SOURCES.txt0000644000076500000240000000425115264141651022066 0ustar00mauritsstaff.coveragerc CHANGES.rst CONTRIBUTING.rst COPYRIGHT.txt HISTORY.rst LICENSE.txt MANIFEST.in README.rst dev.py pyproject.toml setup.cfg setup.py specifications/README.txt specifications/repeatable.txt src/zc.buildout.egg-info/PKG-INFO src/zc.buildout.egg-info/SOURCES.txt src/zc.buildout.egg-info/dependency_links.txt src/zc.buildout.egg-info/entry_points.txt src/zc.buildout.egg-info/not-zip-safe src/zc.buildout.egg-info/requires.txt src/zc.buildout.egg-info/top_level.txt src/zc/buildout/__init__.py src/zc/buildout/_package_index.py src/zc/buildout/buildout.py src/zc/buildout/configparser.py src/zc/buildout/download.py src/zc/buildout/easy_install.py src/zc/buildout/patches.py src/zc/buildout/pep425tags.py src/zc/buildout/rmtree.py src/zc/buildout/testing.py src/zc/buildout/testing.txt src/zc/buildout/testrecipes.py src/zc/buildout/utils.py src/zc/buildout/_vendor/README.rst src/zc/buildout/_vendor/__init__.py src/zc/buildout/_vendor/jaraco_text.py src/zc/buildout/_vendor/pkg_resources/__init__.py src/zc/buildout/tests/__init__.py src/zc/buildout/tests/allow-unknown-extras.txt src/zc/buildout/tests/allowhosts.txt src/zc/buildout/tests/buildout.txt src/zc/buildout/tests/configparser.test src/zc/buildout/tests/configuration.txt src/zc/buildout/tests/debugging.txt src/zc/buildout/tests/dependencylinks.txt src/zc/buildout/tests/download.txt src/zc/buildout/tests/downloadcache.txt src/zc/buildout/tests/easy_install.txt src/zc/buildout/tests/extending.txt src/zc/buildout/tests/extends-cache.txt src/zc/buildout/tests/extensions.txt src/zc/buildout/tests/init.txt src/zc/buildout/tests/options.txt src/zc/buildout/tests/repeatable.txt src/zc/buildout/tests/runsetup.txt src/zc/buildout/tests/setup.txt src/zc/buildout/tests/test_all.py src/zc/buildout/tests/test_extras.py src/zc/buildout/tests/test_increment.py src/zc/buildout/tests/test_pep660.py src/zc/buildout/tests/test_unit.py src/zc/buildout/tests/testing_bugfix.txt src/zc/buildout/tests/update.txt src/zc/buildout/tests/windows.txt src/zc/buildout/tests/recipes/README.txt src/zc/buildout/tests/recipes/setup.py src/zc/buildout/tests/recipes/src/debug.py src/zc/buildout/tests/recipes/src/environ.py src/zc/buildout/tests/recipes/src/mkdir.py././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.1995227 zc_buildout-6.0.1/src/zc.buildout.egg-info/dependency_links.txt0000644000076500000240000000000115264141651024246 0ustar00mauritsstaff ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.1996157 zc_buildout-6.0.1/src/zc.buildout.egg-info/entry_points.txt0000644000076500000240000000015415264141651023476 0ustar00mauritsstaff[console_scripts] buildout = zc.buildout.buildout:main [zc.buildout] debug = zc.buildout.testrecipes:Debug ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1792066473.11371 zc_buildout-6.0.1/src/zc.buildout.egg-info/not-zip-safe0000644000076500000240000000000115264141651022426 0ustar00mauritsstaff ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.1997867 zc_buildout-6.0.1/src/zc.buildout.egg-info/requires.txt0000644000076500000240000000024615264141651022602 0ustar00mauritsstaffsetuptools>=75.8.2 packaging>=23.2 pip>=25.0 platformdirs wheel [test] zope.testing manuel bobo==2.3.0 zdaemon zc.zdaemonrecipe zc.recipe.deployment hatchling build ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792066473.1998465 zc_buildout-6.0.1/src/zc.buildout.egg-info/top_level.txt0000644000076500000240000000000315264141651022723 0ustar00mauritsstaffzc