ArtifexSoftware-pipcl-04f69b2/0000775000175000017500000000000015245537643016216 5ustar epsilonepsilonArtifexSoftware-pipcl-04f69b2/pytest.ini0000664000175000017500000000010215245537643020240 0ustar epsilonepsilon[pytest] python_files = tests/test_*.py addopts = --tb=native ArtifexSoftware-pipcl-04f69b2/README.rst0000664000175000017500000002466315245537643017720 0ustar epsilonepsilonPipcl ===== .. contents:: :backlinks: entry :depth: 2 Overview -------- The ``pipcl`` package provides Python `build backend `_ operations (including PEP 517 support), for use by a ``setup.py`` script. * Designed to help build complex Python extension packages. * Can also be used to build simple pure-python packages. * Works on Linux, Windows, MacOS and OpenBSD. * Is a python module, not a framework, so does not impose any restrictions on usage. * Can be used with any external build system by running commands, for example with ``subprocess.run()`` or enhanced wrapper ``pipcl.run()``. The intention is to allow a ``setup.py`` script to use the full power of Python to do everything that is specific to the package, without having to worry about generic Python packaging issues such wheel formats etc (see https://packaging.python.org/en/latest/specifications/). Locations --------- * https://pypi.org/project/pipcl * https://github.com/ArtifexSoftware/pipcl Basic usage ----------- To build a pure-python package: * Create a ``setup.py`` file that imports pipcl and uses it to specify and build a package:: import pipcl def build(): # Creates a module called `mymodule`. return [ ('src/foo.py', 'mymodule/__init__.py'), ] def sdist(): return pipcl.git_items('.') p = pipcl.Package( 'mypackage', # Name of package. '1.2.3', # Package version. pure=True, # Pure python package. summary = 'My package', description = 'README.md', fn_build = build, fn_sdist = sdist, ) # PEP 517 support. build_wheel = p.build_wheel build_sdist = p.build_sdist * Create a ``pyproject.toml`` that tells pip (and other build frontends) to use setup.py:: [build-system] requires = ['pipcl'] build-backend = 'setup' # Use setup.py backend-path = ['.'] # in top-level of the checkout. To build a package containing a SWIG extension module, the ``build()`` function can be modified to use ``pipcl.build_extension()``:: import pipcl def build(): so_leaf = pipcl.build_extension( name = 'mymodule', # Name of extension module. path_i = 'src/foo.i', # SWIG input file. outdir = 'build', ) # will be '_mymodule.so' or similar. return [ ('build/mymodule.py', 'mymodule/__init__.py'), (f'build/{so_leaf}', 'mymodule/'), ] def sdist(): return pipcl.git_items('.') p = pipcl.Package( 'mypackage', # Name of package. '1.2.3', # Package version. summary = 'My package', description = 'README.md', fn_build = build, fn_sdist = sdist, ) # PEP 517 support. build_wheel = p.build_wheel build_sdist = p.build_sdist For more details, see doctest examples in file:src/pipcl.py. Documentation ------------- * Detailed documentation is in doc comments in Python code. * Convert into HTML with sphinx:: pip install sphinx sphinx-build -M html docs docs/_build View at: file:docs/_build/html/index.html * Convert ``README.rst`` to html with:: pip install docutils docutils -gdst --halt=3 --pep-references README.rst README.rst.html View at: file:README.rst.html API overview ------------ The ``pipcl.Package`` class ........................... This is similar in approach to setuptools/distutils in that one creates an instance of the class, passing the package name, version etc. * Most of the constructor arguments correspond exactly to metadata items in https://packaging.python.org/specifications/core-metadata/. * An explicit function ``fn_build()`` should be provided for building the package. This should return a list of files to be included in a wheel or install, typically as ``(from, to)`` pairs, where ```` is the path to a file and ```` is the path within a wheel or install. * An explicit function ``fn_sdist()`` can be provided for building an sdist, returning a list of files to include. * Unlike setuptools, pipcl does not use heuristics for finding files, instead everything must be specified exactly. Other functions and classes ........................... * ``pipcl.build_extension()`` - build a Python extension using `swig `_. Typically used by ``pipcl.Package``'s ``fn_build()`` callback. * Support for building with the `Limited C API `_. * ``pipcl.git_get()``: Create/update a clean git checkout for a particular branch, tag or sha of a remote repository. * ``pipcl.git_items()``: Get list of files within a git checkout. * ``pipcl.git_info*()``: Get git information such as sha, comment, diff, branch name, author etc. * ``pipcl.macos_add_brew_path()``: on MacOS, support for adding package binaries to ``PATH``. * ``pipcl.macos_patch()``: patching of MacOS shared libraries to avoid use of absolute paths. * Class ``pipcl.NewFiles``: Support for simple detection of changed files, for example around running of an external command. This can be useful when building a wheel with ``pip``, as the leafname of the wheel is not known in advance. * ``pipcl.run()``: enhances ``subprocess.run()``, providing dynamic output along with capture of output, prefixing of output, timeouts etc. * Also allows commands to be specified as multiple lines, which improves readability in logs. * ``pipcl.run_if()``: simple dependency checking to only run commands if prerequisites are newer than the output. * This can use information in Makefile-style ``.d`` dependency files. * ``pipcl.swig*()``: build and use swig from source; this can be useful on MacOS where some swig versions appear to generate incorrect code. * ``version_to_tuple()``: get numerical tuple from '.'-separated version string. * Class ``pipcl.wdev.WindowsVS``: on Windows, search for specific/latest versions of Visual Studio compiler (cl.exe) and linker (link.exe). * Has a command string that includes running of an appropriate vcvars, which can be used directly in compile and link command. Environment variables ..................... * ``PIPCL_CHANGE_VERSIONS`` This can be used to override package versions specified in various places: * The package version specified when creating a ``pipcl.Package()``. * Packages specified in the ``requires_dist`` specified when creating a ``pipcl.Package``. * Packages returned by setup.py's ``get_requires_for_build_wheel()``. * Note that this only works if ``get_requires_for_build_wheel()`` is defined in setup.py *before* it calls ``pipcl.Package()``. For more information see file:src/pipcl.py, class ``Package``, method ``__init__()``, args ``version`` and ``requires_dist``. * ``PIPCL_PREBUILT_WHEEL_`` Specifies a location of wheel to be used instead of building a wheel for ````. This can be used to get a wheel from an external location instead of building it. The value should be something that can be passed to an internal call to ``pip wheel``, for example a local path or a URL. ```` will be as used in wheel filenames - all lower-case with special characters replaced by underscores. Other ..... * Support for Pyodide. * Experimental support for Graal. Changelog --------- **Version 13** * Added missing ```` arg to ``pipcl.wdev.windows_vs()``. * ``pipcl.version_to_tuple()`` now also accepts a sequence as well as a string. * ``build_extension()``: optionally return ``(py, lib)`` instead of library leafname. * Minor improvements to indentation of compiler/linker commands. * Extend ``PIPCL_CHANGE_VERSIONS`` to also modify return from ``get_requires_for_build_wheel()``. * ``git_items()``: avoid occasional problem with windows ``/`` vs ``\``. * ``pipcl.run()``: added ```` arg. * Add support for getting wheel from external location - see `Environment variables`_ ``PIPCL_PREBUILT_WHEEL_``. **Version 12** (2026-07-09) * Fixed handling of ``None`` in ``Package.__init__()``'s ```` arg. * On Windows, improved searching for Visual Studio. * We now find Visual Studio 2026 correctly. * We now use Microsoft's ``vswhere.exe`` executable to find installations of Visual Studio. * Instances of class ``pipcl.wdev.WindowsVS`` are now primarily obtained using new functions ``pipcl.wdev.windows_vs()`` and ``pipcl.wdev.windows_vs_multiple()`` instead of direct construction. * Fixed bug in ``pipcl.build_extension()`` example. * Fixed compile/link commands on windows-arm - use ``vcvarsarm.bat`` or ``vcvarsarm64.bat``. **Version 11** (2026-07-01) * In ``Package`` class, make metadata version explicit. * Use license classifier to clarify that pipcl is GNU Affero v3 only. **Version 10** (2026-06-26) * Fixed ``pipcl.git_info_py()`` in a non-git-checkout. **Version 9** (2026-06-18) * Added added ``git_info_py()``. * Added ``version_to_tuple()``. **Version 8** (2026-06-17) * Improve support for ``PIPCL_CHANGE_VERSIONS``: * Convert package name to lower case before matching with ``PIPCL_CHANGE_VERSIONS``. * Provide ``pipcl.version_override()`` for use by a setup.py's ``get_requires_for_build_wheel()``. **Version 7** (2026-06-05) * Override matching package version numbers if ``PIPCL_CHANGE_VERSIONS`` is set. **Version 6** (2026-05-21) * In ``build_extension()``, added new arg ``compiler_extra_cpp``. **Version 5** (2026-05-18) * Automatically compile extension source with ``cc`` or ``c++`` depending on suffix. **Version 4** (2026-05-08) * Ignore encoding errors in ``pipcl.log()``, e.g. on Windows if default is not utf8. **Version 3** (2026-05-02): * Avoid unhelpful assert failures on Windows if paths differ only in upper/lower case. **Version 2** (2026-04-18): * Fixed bug in zip file generation on python<3.13. * Moved Python code into ``src/``. * Don't attempt to be usable in raw checkout. * Fixed doctest's to work on Windows. * Added ``python_version_tuple()``. * Avoid spurious differences between wheels built on different systems: * Sort lines in generated ``RECORD`` file. * Use ``--global core.autocrlf input`` when running ``git clone``. **Version 1** (2026-04-16): * First release to pypi.org ArtifexSoftware-pipcl-04f69b2/setup.py0000664000175000017500000000217115245537643017731 0ustar epsilonepsilonimport sys try: import pipcl except ImportError: # This only happens if we are run directly without use of pyproject.toml, # e.g. setup.py sdist. sys.path.insert(0, 'src') import pipcl def build(): return [ ('src/pipcl.py', 'pipcl/__init__.py'), ('src/wdev.py', 'pipcl/wdev.py'), ] def sdist(): return pipcl.git_items('.') p = pipcl.Package( 'pipcl', version = '13', pure = True, requires_dist=['pip', 'packaging'], description='README.rst', summary='Python packaging operations, including PEP-517 support, for use by a setup.py script.', author='Artifex', author_email='julian.smith@artifex.com', classifier='License :: OSI Approved :: GNU Affero General Public License v3', license='GNU AFFERO GPL 3.0', project_url=[ 'homepage, https://github.com/ArtifexSoftware/pipcl', ], fn_build = build, fn_sdist = sdist, ) build_wheel = p.build_wheel build_sdist = p.build_sdist if __name__ == '__main__': p.handle_argv(sys.argv) ArtifexSoftware-pipcl-04f69b2/pyproject.toml0000664000175000017500000000015015245537643021126 0ustar epsilonepsilon[build-system] requires = ['packaging'] build-backend = "setup" backend-path = [".", "src"] ArtifexSoftware-pipcl-04f69b2/tests/0000775000175000017500000000000015245537643017360 5ustar epsilonepsilonArtifexSoftware-pipcl-04f69b2/tests/test_doctest.py0000664000175000017500000000117415245537643022441 0ustar epsilonepsilonimport os import subprocess import sys def test_doctest_all(): root = os.path.normpath(f'{__file__}/../../') subprocess.run(f'pip install --upgrade swig', shell=1, check=1) subprocess.run(f'cd {root} && {sys.executable} -m doctest -f -v src/pipcl.py', shell=1, check=1) def _test_doctest_1(): root = os.path.normpath(f'{__file__}/../../') subprocess.run(f'pip install --upgrade swig', shell=1, check=1) command = f'cd {root} && {sys.executable} src/pipcl.py --doctest _run_if_test_scripting_windows' print(f'\ntest_doctest1(): running {command=}.', flush=1) subprocess.run(command, shell=1, check=1) ArtifexSoftware-pipcl-04f69b2/tests/test_visual_studio.py0000664000175000017500000000130315245537643023660 0ustar epsilonepsilonimport os import platform import pipcl def test_visual_studio(): ''' This is more a diagnostic tool than a test. ''' print() if platform.system() != 'Windows': print('test_visual_studio(): not running on non-Windows.') return vss = pipcl.wdev.windows_vs_multiple() print('Visual Studio installations are:') for vs in vss: print(vs.description_ml(indent=' ')) d = os.path.dirname(vs.vcvars) print(f' Contents of {d}:') for i in os.listdir(d): print(f' {i}') for year in 2017, 2019, 2022, 2026: vs = pipcl.wdev.windows_vs(year=year, check=0) print(f'{year=}: {vs=}') ArtifexSoftware-pipcl-04f69b2/tests/test_project.py0000664000175000017500000001420515245537643022441 0ustar epsilonepsilonimport glob import os import sys import textwrap g_root = os.path.normpath(f'{__file__}/../..') import pipcl pipcl.log(f'{sys.path=}') def test_project(): ''' We create a python project which builds a python extension with a setup.py file that uses pipcl, and check that pip installs pipcl in response to the pyproject.toml and builds the project. ''' path_test = f'{g_root}/temp_test' pipcl.fs_ensure_empty_dir(path_test) pipcl.log('## Prepare a piprepo containing a pipcl wheel.') pipcl.run(f'pip install --upgrade piprepo "setuptools<81"', prefix='pip install piprepo: ') pipcl.run(f'pip wheel -w {path_test}/wheelhouse {g_root}', prefix='pip wheel pipcl: ') pipcl.run(f'piprepo build {path_test}/wheelhouse', prefix='piprepo build: ') pip_index_url = f'file://{os.path.abspath(path_test)}/wheelhouse/simple' pip_index_url = pip_index_url.replace('\\', '/') pipcl.log('## Create package called `foo` that uses pipcl to build.') pipcl.run(f'pip uninstall -y foo') pipcl.fs_ensure_empty_dir(f'{path_test}/project') pipcl.fs_write( f'{path_test}/project/setup.py', textwrap.dedent(''' import subprocess def run(command): print(f'Running: {command}', flush=1) subprocess.run(command, shell=1, check=1) import pipcl def build(): so_leaf = pipcl.build_extension( name = 'pipcl_test_module', path_i = 'foo.i', outdir = 'build', source_extra = 'wibble.c', ) return [ ('build/pipcl_test_module.py', 'pipcl_test_module/__init__.py'), ('cli.py', 'pipcl_test_module/__main__.py'), (f'build/{so_leaf}', f'pipcl_test_module/'), ('README', '$dist-info/'), (b'Hello world', 'pipcl_test_module/hw.txt'), ] def sdist(): ret = list() for p in pipcl.git_items('.'): ret.append(p) return ret p = pipcl.Package( name = 'foo', version = '1.2.3', fn_build = build, fn_sdist = sdist, entry_points = ( { 'console_scripts': [ 'foo_cli = foo.__main__:main', ], } ), ) build_wheel = p.build_wheel build_sdist = p.build_sdist # Handle old-style setup.py command-line usage: if __name__ == '__main__': p.handle_argv(sys.argv) '''), ) pipcl.fs_write( f'{path_test}/project/pyproject.toml', textwrap.dedent(''' [build-system] requires = ["pipcl", "swig"] # See pep-517. build-backend = "setup" backend-path = ["."] ''') ) pipcl.fs_write( f'{path_test}/project/foo.i', textwrap.dedent(''' %include bar.i %{ #include #include int bar(const char* text) { printf("bar(): text: %s\\\\n", text); int len = (int) strlen(text); printf("bar(): len=%i\\\\n", len); fflush(stdout); return len; } %} int bar(const char* text); ''') ) pipcl.fs_write( f'{path_test}/project/bar.i', textwrap.dedent(''' ''') ) pipcl.fs_write( f'{path_test}/project/wibble.c', textwrap.dedent(''' ''') ) pipcl.fs_write( f'{path_test}/project/README', textwrap.dedent(''' This is Foo. ''') ) pipcl.fs_write( f'{path_test}/project/cli.py', textwrap.dedent(''' def main(): print('pipcl_test:main().') if __name__ == '__main__': main() ''') ) pipcl.log('## Build project foo, allowing pip to get pipcl from our piprepo\'d wheelhouse.') newfiles = pipcl.NewFiles(f'{path_test}/*.whl') pipcl.run(f''' pip wheel -v -w {path_test} --extra-index-url {pip_index_url} {path_test}/project ''', prefix=f'build {path_test}/project: ', ) pipcl.log('## Install project foo') path_foo_wheel = newfiles.get_one() pipcl.run(f'pip uninstall -y foo') pipcl.run(f'pip install {path_foo_wheel}', prefix=f'pip install {path_foo_wheel}: ') pipcl.log('## Check we can import pipcl_test_module.') pipcl.log(f'{sys.path=}') import pipcl_test_module # Test PIPCL_PREBUILT_WHEEL_*. env_name = f'PIPCL_PREBUILT_WHEEL_foo' env_value = path_foo_wheel wheelhouse_prebuilt = f'{path_test}/wheelhouse-prebuilt' out = pipcl.run(f'''pip wheel -v -w {wheelhouse_prebuilt} --extra-index-url {pip_index_url} {path_test}/project ''', env_extra={env_name: env_value}, capture=1, out='log', prefix=f'build with PIPCL_PREBUILT_WHEEL*: ', ) assert 'Returning prebuilt wheel_leaf=' in out wheel_path = glob.glob(f'{wheelhouse_prebuilt}/*') assert len(wheel_path) == 1 wheel_path = wheel_path[0] ArtifexSoftware-pipcl-04f69b2/tests/test_zip.py0000664000175000017500000000163115245537643021574 0ustar epsilonepsilonimport os import zipfile import pipcl def test_zipfile_writestr(): # Check pipcl.zipfile_writestr() creates file that extracts with correct # mode flags. root = os.path.normpath(f'{__file__}/../..') path_zf = f'{root}/tests/zipfile_writestr_test.zip' name = 'test_zip_out.txt' path_extracted = f'{root}/tests/{name}' mode = 0o644 with zipfile.ZipFile(path_zf, 'w') as zf: pipcl.zipfile_writestr(zf, name, 'hello world.', 0o644) pipcl.fs_remove(path_extracted) with zipfile.ZipFile(path_zf) as zf: zf.extractall(f'{root}/tests') assert os.path.isfile(path_extracted) st = os.stat(path_extracted) print(f'{st=}') if pipcl.windows(): # We don't expect extracted file to have specified unix permissions. pass else: # Extracted mode is modified by umask. assert (st.st_mode & mode) == mode ArtifexSoftware-pipcl-04f69b2/tests/test_change_versions.py0000664000175000017500000001773115245537643024157 0ustar epsilonepsilonimport glob import os import re import textwrap import pipcl g_root = os.path.normpath(f'{__file__}/../..') def test_change_versions(): ''' Check handling of PIPCL_CHANGE_VERSIONS. ''' path_test = f'{g_root}/temp_test_change_version' path_wheelhouse = f'{path_test}/wheelhouse' pipcl.fs_ensure_empty_dir(path_test) # Prepare a pip repo. pipcl.run(f'pip install --upgrade piprepo "setuptools<81"', prefix='pip install piprepo: ') pip_index_url = f'file://{os.path.abspath(path_wheelhouse)}/simple' pip_index_url = pip_index_url.replace('\\', '/') # Add pipcl wheel to pip repo. pipcl.run(f'pip wheel -w {path_wheelhouse} {g_root}', prefix='pip wheel pipcl: ') pipcl.run(f'piprepo build {path_wheelhouse}', prefix='piprepo build: ') def make_project(name, version, requires_dist=None, grfbw_version=None): ''' Makes minimal project for package , version and prerequisites '. If is specified: We also create a get_requires_for_build_wheel() fn that returns 'example-package-NAME==' (`example-package-NAME` is a dummy package on pypi.prg for which versions 0.0.1 and 0.0.3 are available). And at runtime our `build()` function asserts that the specified version of `example-package-NAME` is installed. Except if example_package_NAME_OVERRIDE is set, it asserts that version instead. This allow us to test that PIPCL_CHANGE_VERSIONS overrides get_requires_for_build_wheel() correctly. ''' requires_dist_str = f'requires_dist = {requires_dist!r},' if requires_dist else '' pipcl.fs_ensure_empty_dir(f'{path_test}/{name}') setup_py = '' setup_py += textwrap.dedent(f''' import pipcl ''') if name == 'pipcl_test_foo' and grfbw_version: setup_py += textwrap.dedent(f''' import importlib.metadata import os def build(): example_package_NAME_version = importlib.metadata.version('example-package-NAME') example_package_NAME_OVERRIDE = os.environ.get('example_package_NAME_OVERRIDE') if example_package_NAME_OVERRIDE: pipcl.log(f'Asserting that {{example_package_NAME_version=}} == {{example_package_NAME_OVERRIDE=}}.') assert example_package_NAME_version == example_package_NAME_OVERRIDE, ( f'{{example_package_NAME_version=}} should be {{example_package_NAME_OVERRIDE=}}.' ) else: grfbw_version = {grfbw_version!r} pipcl.log(f'Asserting that {{example_package_NAME_version=}} == {{grfbw_version=}}.') assert example_package_NAME_version == {grfbw_version!r}, ( f'{{example_package_NAME_version=}} should be {{grfbw_version=}}.' ) return [] ''') else: setup_py += textwrap.dedent(f''' def build(): return [] ''') if grfbw_version: setup_py += textwrap.dedent(f''' def get_requires_for_build_wheel(config_settings=None): ret = ['example-package-NAME=={grfbw_version}'] pipcl.log(f'get_requires_for_build_wheel(): Returning {{ret=}}.') return ret ''') setup_py += textwrap.dedent(f''' p = pipcl.Package( name = {name!r}, version = {version!r}, fn_build = build, {requires_dist_str} ) build_wheel = p.build_wheel ''') pipcl.fs_write( f'{path_test}/{name}/setup.py', setup_py, ) pipcl.fs_write( f'{path_test}/{name}/pyproject.toml', textwrap.dedent(''' [build-system] requires = ['pipcl'] build-backend = 'setup' backend-path = ['.'] ''') ) def build_wheel(name, env_extra=None): ''' Build wheel and add it to pip repo. We return the wheel version. ''' newfiles = pipcl.NewFiles(f'{path_wheelhouse}/*.whl') # We use --no-deps to avoid pip installing prerequisite packages - # i'm not sure why pip does this when merely building a wheel, but it # messes up our attempt to check that installing fails if we have used # PIPCL_CHANGE_VERSIONS to make wheels incompatible. pipcl.run( f''' pip wheel -v --no-deps -w {path_wheelhouse} --extra-index-url {pip_index_url} {path_test}/{name} ''', prefix=f'build {path_test}/{name}: ', env_extra=env_extra, ) wheel = newfiles.get_one() pipcl.log(f'{wheel=}') wheel_leaf = os.path.basename(wheel) m = re.match(f'{name}-([^-]+)-.*.whl', wheel_leaf) wheel_version = m.group(1) pipcl.run(f'piprepo build {path_wheelhouse}', prefix='piprepo build: ') return wheel_version def install(name, check=1): ''' Install package using pip repo. ''' return pipcl.run(f''' pip install -v --extra-index-url {pip_index_url} {name} ''', check=check, ) def clean(): pipcl.run(f'pip uninstall -y pipcl_test_foo pipcl_test_bar example-package-NAME') for p in ([] + glob.glob(f'{path_wheelhouse}/pipcl_test_*.whl') + glob.glob(f'{path_wheelhouse}/example_package_NAME*.whl') ): os.remove(p) make_project('pipcl_test_foo', '1.2.3', requires_dist='pipcl_test_bar==4.5.6', grfbw_version='0.0.3') make_project('pipcl_test_bar', '4.5.6') # Check operation without PIPCL_CHANGE_VERSIONS. if 1: clean() version_bar = build_wheel('pipcl_test_bar') version_foo = build_wheel('pipcl_test_foo') assert version_foo == '1.2.3' assert version_bar == '4.5.6' install('pipcl_test_foo') # Check PIPCL_CHANGE_VERSIONS changes version and requires_dist correctly. if 1: clean() env_extra=dict( PIPCL_CHANGE_VERSIONS=textwrap.dedent(''' ^pipcl_test_bar$ ==7.8.9 example-package-NAME ==0.0.1 '''), example_package_NAME_OVERRIDE='0.0.1', ) version_bar = build_wheel('pipcl_test_bar', env_extra=env_extra) version_foo = build_wheel('pipcl_test_foo', env_extra=env_extra) assert version_foo == '1.2.3' assert version_bar == '7.8.9' install('pipcl_test_foo') # Check that we get error if we specify different bar versions # when building foo vs bar. if 1: clean() env_extra=dict(PIPCL_CHANGE_VERSIONS='^pipcl_test_bar$ ==8.9') version_bar = build_wheel('pipcl_test_bar', env_extra=env_extra) env_extra=dict(PIPCL_CHANGE_VERSIONS='^pipcl_test_bar$ ==7.8') version_foo = build_wheel('pipcl_test_foo', env_extra=env_extra) assert version_foo == '1.2.3' assert version_bar == '8.9' e = install('pipcl_test_foo', check=0) assert e ArtifexSoftware-pipcl-04f69b2/tests/test_lint.py0000664000175000017500000001774115245537643021751 0ustar epsilonepsilonimport glob import os import re import platform import shlex import subprocess import sys import textwrap import pipcl def test_pylint(): subprocess.run(f'pip install -U pylint', shell=1, check=1) ignores = '' ignores += textwrap.dedent( ''' C0103: Variable name "ARTIFEX_SOFTWARE_SSH_KEY" doesn't conform to snake_case naming style (invalid-name) C0115: Missing class docstring (missing-class-docstring) C0116: Missing function or method docstring (missing-function-docstring) C0301: Line too long (131/100) (line-too-long) C0302: Too many lines in module (1652/1000) (too-many-lines) C0303: Trailing whitespace (trailing-whitespace) C3001: Lambda expression assigned to a variable. Define a function using the "def" keyword instead. (unnecessary-lambda-assignment) R0902: Too many instance attributes (31/7) (too-many-instance-attributes) R0903: Too few public methods (0/2) (too-few-public-methods) R0913: Too many arguments (7/5) (too-many-arguments) R0914: Too many local variables (108/15) (too-many-locals) R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return) R1713: Consider using str.join(sequence) for concatenating strings from an iterable (consider-using-join) R1726: Boolean condition ... may be simplified to ... (simplifiable-condition) R1734: Consider using [] instead of list() (use-list-literal) R1735: Consider using '{}' instead of a call to 'dict'. (use-dict-literal) W0108: Lambda may not be necessary (unnecessary-lambda) W0125: Using a conditional statement with a constant value (using-constant-test) W0201: Attribute 'cibw_name' defined outside __init__ (attribute-defined-outside-init) W0511: fixme: be able to set to '' for system install? (fixme) W0718: Catching too general exception Exception (broad-exception-caught) W0719: Raising too general exception: Exception (broad-exception-raised) W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation) C0325: Unnecessary parens after '=' keyword (superfluous-parens) W1514: Using open without explicitly specifying an encoding (unspecified-encoding) R0915: Too many statements (56/50) (too-many-statements) C0411: standard import "fnmatch" should be placed before first party import "pipcl" (wrong-import-order) R1707: Disallow trailing comma tuple (trailing-comma-tuple) R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements) R1723: Unnecessary "elif" after "break", remove the leading "el" from "elif" (no-else-break) R1714: Consider merging these comparisons with 'in' by using 'to_ in ('/', '')'. Use a set instead if elements are hashable. (consider-using-in) W0621: Redefining name 'verbose' from outer scope (line 3263) (redefined-outer-name) W0640: Cell variable write2 defined in loop (cell-var-from-loop) W0603: Using the global statement (global-statement) R0912: Too many branches (52/12) (too-many-branches) C0415: Import outside toplevel (pipcl) (import-outside-toplevel) C0114: Missing module docstring (missing-module-docstring) R0801: Similar lines in 2 files ''' ) ignores_list = list() for line in ignores.split('\n'): if not line or line.startswith('#'): continue m = re.match('^(.....): ', line) assert m, f'Failed to parse {line=}' ignores_list.append(m.group(1)) ignores = ','.join(ignores_list) root = os.path.normpath(f'{__file__}/../..') sys.path.insert(0, root) directory = root if 0: # 2026-04-13: do nothing because pylint fails to import. leafs = glob.glob(f'{root}/*.py') command = f'pylint -d {ignores}' for leaf in leafs: if leaf.endswith('.py'): command += f' {directory}/{leaf}' print(f'Running: {command}') subprocess.run(command, shell=1, check=1) def test_codespell(): ''' Check rebased Python code with codespell. ''' subprocess.run(f'pip install -U codespell', shell=1, check=1) root = os.path.abspath(f'{__file__}/../..') # For now we ignore files that we would ideally still look at, because it # is difficult to exclude some text sections. skips = textwrap.dedent(''' ''') skips = skips.strip().replace('\n', ',') command = textwrap.dedent(f''' cd {root} && codespell --skip {shlex.quote(skips)} --ignore-multiline-regex 'codespell:ignore-begin.*codespell:ignore-end' ''') git_files = pipcl.git_items(root) for p in git_files: _, ext = os.path.splitext(p) if ext in []: pass else: command += f' {p}\n' if platform.system() != 'Windows': command = command.replace('\n', ' \\\n') # Don't print entire command because very long, and will be displayed # anyway if there is an error. print(f'test_codespell(): Running: {command}') #print(f'Running codespell.') subprocess.run(command, shell=1, check=1) print('test_codespell(): codespell succeeded.') def test_flake8(): subprocess.run(f'pip install -U flake8', shell=1, check=1) ignores = textwrap.dedent(''' E123 closing bracket does not match indentation of opening bracket\'s line E124 closing bracket does not match visual indentation E126 continuation line over-indented for hanging indent E127 continuation line over-indented for visual indent E128 continuation line under-indented for visual indent E131 continuation line unaligned for hanging indent E201 whitespace after \'(\' E203 whitespace before \':\' E225 missing whitespace around operator E226 missing whitespace around arithmetic operator E227 missing whitespace around bitwise or shift operator E241 multiple spaces after \':\' E251 unexpected spaces around keyword / parameter equals E252 missing whitespace around parameter equals E261 at least two spaces before inline comment E265 block comment should start with \'# \' E271 multiple spaces after keyword E272 multiple spaces before keyword E302 expected 2 blank lines, found 1 E305 expected 2 blank lines after class or function definition, found 1 E306 expected 1 blank line before a nested definition, found 0 E402 module level import not at top of file E501 line too long E701 multiple statements on one line (colon) E731 do not assign a lambda expression, use a def E741 ambiguous variable name \'l\' F541 f-string is missing placeholders W293 blank line contains whitespace W503 line break before binary operator W504 line break after binary operator ''') ignores = ignores.strip().split('\n') ignores2 = list() for i in ignores: if not i.startswith('#'): ignores2.append(i.split(' ', 1)[0]) ignores = ','.join(ignores2) root = os.path.normpath(f'{__file__}/../..') print(f'{root=}') sys.path.insert(0, root) command = f'flake8 --ignore={ignores} --statistics' paths = ['src/pipcl.py', 'src/wdev.py'] for p in paths: if p.endswith('.py'): command += f' {root}/{p}' print(f'test_flake8(): Running: {command}') subprocess.run(command, shell=1, check=1) print(f'test_flake8(): flake8 succeeded.') ArtifexSoftware-pipcl-04f69b2/tests/test_nogit.py0000664000175000017500000000135115245537643022111 0ustar epsilonepsilonimport os import tempfile import pipcl def test_nogit_check(): ''' Check pipcl.git_info_py() in a non-git-checkout. ''' GITHUB_ACTIONS = os.environ.get('GITHUB_ACTIONS') if GITHUB_ACTIONS == 'true': print(f'test_nogit_check(): not running on Github because tempdir is within checkout.') return with tempfile.TemporaryDirectory() as path: try: text = pipcl.git_info_py(path) except Exception: text = None else: print(f'{text=}') assert 0, f'Expected exception for directory: {path}' print(f'{text=}') def test_nogit_nocheck(): path = tempfile.gettempdir() text = pipcl.git_info_py(path, check=0) print(f'{text}') ArtifexSoftware-pipcl-04f69b2/docs/0000775000175000017500000000000015245537643017146 5ustar epsilonepsilonArtifexSoftware-pipcl-04f69b2/docs/conf.py0000664000175000017500000000315715245537643020453 0ustar epsilonepsilon# Configuration file for the Sphinx documentation builder. # # For the full list of built-in configuration values, see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html # -- Project information ----------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information # To build: # # * Enter venv. # * pip install sphinx # * sphinx-build -M html docs docs/_build # * firefox docs/_build/html/index.html # project = 'pipcl' copyright = '2026, Artifex' author = 'Julian Smith' # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.autosummary', 'autodocsumm', # Understand Google-style doc-comments. See # https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html # 'sphinx.ext.napoleon', ] templates_path = ['_templates'] exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output html_theme = 'alabaster' html_static_path = ['_static'] import os import sys import subprocess root = os.path.normpath(f'{__file__}/../..') subprocess.run(f'pip install autodocsumm', shell=1, check=1) subprocess.run(f'pip install {root}', shell=1, check=1) autoclass_content = 'both' maximum_signature_line_length = 40 default_role = 'any' ArtifexSoftware-pipcl-04f69b2/docs/index.rst0000664000175000017500000000164515245537643021015 0ustar epsilonepsilon.. pipcl documentation master file, created by sphinx-quickstart on Wed Apr 8 12:11:07 2026. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. Pipcl documentation =================== [This is all auto-generated from pipcl doc-comments using `sphinx.ext.autodoc `_, `sphinx.ext.autosummary `_ and `autodocsumm `_. ] .. contents:: Contents :local: .. autosummary:: :toctree: generated pipcl ----- .. automodule:: pipcl :members: :undoc-members: :autosummary: pipcl.wdev ---------- .. automodule:: pipcl.wdev :members: :undoc-members: :autosummary: ArtifexSoftware-pipcl-04f69b2/src/0000775000175000017500000000000015245537643017005 5ustar epsilonepsilonArtifexSoftware-pipcl-04f69b2/src/pipcl.py0000664000175000017500000054505415245537643020503 0ustar epsilonepsilon''' Python packaging operations, including PEP-517 support, for use by a ``setup.py`` script. Overview: The intention is to take care of as many packaging details as possible so that setup.py contains only project-specific information, while also giving as much flexibility as possible. For example we provide a function `pipcl.build_extension()` that can be used to build a SWIG extension, but we also give access to the located compiler/linker so that a `setup.py` script can take over the details itself. Doctests: Doctest strings are provided in some comments. Test in the usual way with:: python -m doctest pipcl.py Test specific functions/classes with:: python pipcl.py --doctest run_if ... If no functions or classes are specified, this tests everything. Graal: For Graal we require that PIPCL_GRAAL_PYTHON is set to non-graal Python (we build for non-graal except with Graal Python's include paths and library directory). ''' import atexit import base64 import calendar import codecs import difflib import glob import hashlib import inspect import io import os import packaging.requirements import pickle import platform import re import selectors import shlex import shutil import subprocess import sys import sysconfig import tarfile import tempfile import textwrap import time import zipfile try: import wdev except ImportError: from . import wdev class Package: ''' Represents a Python package. The constructor takes a definition of a Python package similar to that passed to `distutils.core.setup()` or `setuptools.setup()` (name, version, summary etc) plus callbacks for building, getting a list of sdist filenames, and cleaning. Methods are provided that can be used to implement a Python package's `setup.py` supporting PEP-517. Basic command line handling is also supported for use with a legacy (pre-PEP-517) pip, as implemented by legacy distutils/setuptools and described in: https://pip.pypa.io/en/stable/reference/build-system/setup-py/ The file pyproject.toml must exist; this is checked if/when `fn_build()` is called. Here is a `doctest` example of using pipcl to create a SWIG extension module called 'foo', with a console-script command line tool called `cli`. Requires `swig`. Create an empty test directory: >>> import os >>> import shutil >>> shutil.rmtree('pipcl_test', ignore_errors=1) >>> os.mkdir('pipcl_test') Create a `setup.py` which uses `pipcl` to define an extension module. >>> import textwrap >>> with open('pipcl_test/setup.py', 'w') as f: ... _ = f.write(textwrap.dedent(""" ... import sys ... import pipcl ... ... def build(): ... so_leaf = pipcl.build_extension( ... name = 'foo', ... path_i = 'foo.i', ... outdir = 'build', ... source_extra = 'wibble.c', ... ) ... return [ ... ('build/foo.py', 'foo/__init__.py'), ... ('cli.py', 'foo/__main__.py'), ... (f'build/{so_leaf}', f'foo/'), ... (b'Hello world', 'foo/hw.txt'), ... ] ... ... def sdist(): ... return [ ... 'foo.i', ... 'bar.i', ... 'wibble.c', ... 'setup.py', ... 'pipcl.py', ... 'pyproject.toml', ... 'wdev.py', ... 'README', ... ] ... ... p = pipcl.Package( ... name = 'foo', ... version = '1.2.3', ... description = 'README', ... fn_build = build, ... fn_sdist = sdist, ... entry_points = ( ... { 'console_scripts': [ ... 'foo_cli = foo.__main__:main', ... ], ... }), ... ) ... ... build_wheel = p.build_wheel ... build_sdist = p.build_sdist ... ... # Handle old-style setup.py command-line usage: ... if __name__ == '__main__': ... p.handle_argv(sys.argv) ... """)) Create the files required by the above `setup.py` - pyproject.toml, the SWIG `.i` input file, the README file, and copies of `pipcl.py` and `wdev.py`. >>> with open('pipcl_test/pyproject.toml', 'w') as f: ... _ = f.write(textwrap.dedent(""" ... [build-system] ... requires = [] ... ... # See pep-517. ... # ... build-backend = "setup" ... backend-path = ["."] ... """)) >>> with open('pipcl_test/foo.i', 'w') as f: ... _ = f.write(textwrap.dedent(""" ... %include bar.i ... %{ ... #include ... #include ... int bar(const char* text) ... { ... printf("bar(): text: %s\\\\n", text); ... int len = (int) strlen(text); ... printf("bar(): len=%i\\\\n", len); ... fflush(stdout); ... return len; ... } ... %} ... int bar(const char* text); ... """)) >>> with open('pipcl_test/bar.i', 'w') as f: ... _ = f.write( '\\n') >>> with open('pipcl_test/wibble.c', 'w') as f: ... _ = f.write( '\\n') >>> with open('pipcl_test/README', 'w') as f: ... _ = f.write(textwrap.dedent(""" ... This is Foo. ... """)) >>> with open('pipcl_test/cli.py', 'w') as f: ... _ = f.write(textwrap.dedent(""" ... def main(): ... print('pipcl_test:main().') ... if __name__ == '__main__': ... main() ... """)) >>> root = os.path.dirname(__file__) >>> _ = shutil.copy2(f'{root}/pipcl.py', 'pipcl_test/pipcl.py') >>> _ = shutil.copy2(f'{root}/wdev.py', 'pipcl_test/wdev.py') Use `setup.py`'s command-line interface to build and install the extension module into root `pipcl_test/install`. >>> _ = subprocess.run( ... f'cd pipcl_test && {sys.executable} setup.py --root install install', ... shell=1, check=1) The actual install directory depends on `sysconfig.get_path('platlib')`: >>> if windows(): ... install_dir = 'pipcl_test/install' ... else: ... install_dir = f'pipcl_test/install/{sysconfig.get_path("platlib").lstrip(os.sep)}' >>> assert os.path.isfile( f'{install_dir}/foo/__init__.py') Create a test script which asserts that Python function call `foo.bar(s)` returns the length of `s`, and run it with `PYTHONPATH` set to the install directory: >>> with open('pipcl_test/test.py', 'w') as f: ... _ = f.write(textwrap.dedent(""" ... import sys ... import foo ... text = 'hello' ... print(f'test.py: calling foo.bar() with text={text!r}') ... sys.stdout.flush() ... l = foo.bar(text) ... print(f'test.py: foo.bar() returned: {l}') ... assert l == len(text) ... """)) >>> r = subprocess.run( ... f'{sys.executable} pipcl_test/test.py', ... shell=1, check=1, text=1, ... stdout=subprocess.PIPE, stderr=subprocess.STDOUT, ... env=os.environ | dict(PYTHONPATH=install_dir), ... ) >>> print(r.stdout) test.py: calling foo.bar() with text='hello' bar(): text: hello bar(): len=5 test.py: foo.bar() returned: 5 Check that building sdist and wheel succeeds. For now we don't attempt to check that the sdist and wheel actually work. >>> _ = subprocess.run( ... f'cd pipcl_test && {sys.executable} setup.py sdist', ... shell=1, check=1) >>> _ = subprocess.run( ... f'cd pipcl_test && {sys.executable} setup.py bdist_wheel', ... shell=1, check=1) Check that rebuild does nothing. >>> t0 = os.path.getmtime('pipcl_test/build/foo.py') >>> _ = subprocess.run( ... f'cd pipcl_test && {sys.executable} setup.py bdist_wheel', ... shell=1, check=1) >>> t = os.path.getmtime('pipcl_test/build/foo.py') >>> assert t == t0 Check that touching bar.i forces rebuild. >>> os.utime('pipcl_test/bar.i') >>> _ = subprocess.run( ... f'cd pipcl_test && {sys.executable} setup.py bdist_wheel', ... shell=1, check=1) >>> t = os.path.getmtime('pipcl_test/build/foo.py') >>> assert t > t0 Check that touching foo.i.cpp does not run swig, but does recompile/link. >>> so_suffix = 'pyd' if windows() else 'so' >>> t0 = time.time() >>> os.utime('pipcl_test/build/foo.i.cpp') >>> _ = subprocess.run( ... f'cd pipcl_test && {sys.executable} setup.py bdist_wheel', ... shell=1, check=1) >>> assert os.path.getmtime('pipcl_test/build/foo.py') <= t0 >>> so = glob.glob(f'pipcl_test/build/*.{so_suffix}') >>> assert len(so) == 1, f'{so_suffix=} {so=} {os.path.abspath(".")=}' >>> so = so[0] >>> assert os.path.getmtime(so) > t0 Check that touching wibbble.c does not run swig, but does recompile/link. >>> t0 = time.time() >>> os.utime('pipcl_test/wibble.c') >>> _ = subprocess.run( ... f'cd pipcl_test && {sys.executable} setup.py bdist_wheel', ... shell=1, check=1) >>> assert os.path.getmtime('pipcl_test/build/foo.py') <= t0 >>> so = glob.glob(f'pipcl_test/build/*.{so_suffix}') >>> assert len(so) == 1 >>> so = so[0] >>> assert os.path.getmtime(so) > t0 Check `entry_points` causes creation of command `foo_cli` when we install from our wheel using pip. [As of 2024-02-24 using pipcl's CLI interface directly with `setup.py install` does not support entry points.] >>> print('Creating venv.', file=sys.stderr) >>> _ = subprocess.run( ... f'cd pipcl_test && {sys.executable} -m venv pylocal', ... shell=1, check=1) >>> print('Installing from wheel into venv using pip.', file=sys.stderr) >>> whl = glob.glob(f'pipcl_test/dist/*.whl')[0] >>> _ = subprocess.run(f'{venv_enter_cmd("pipcl_test/pylocal")} && pip install {whl}', ... shell=1, check=1) >>> print('Running foo_cli.', file=sys.stderr) >>> _ = subprocess.run( ... f'{venv_enter_cmd("pipcl_test/pylocal")} && foo_cli', ... shell=1, check=1) Check that wheel contents' unix permissions are readable by all. >>> wheel_path = glob.glob('pipcl_test/dist/*.whl')[0] >>> with zipfile.ZipFile(wheel_path) as z: ... for zi in z.infolist(): ... external_attr_lo = zi.external_attr & 15 ... external_attr_hi = zi.external_attr >> 16 ... assert external_attr_lo + (external_attr_hi << 16) == zi.external_attr ... assert external_attr_hi & 0o444 == 0o444, ( ... f'info.external_attr not readable by all:' ... f' {zi.external_attr=:#x}' ... f' {external_attr_hi=:#o}' ... f' {external_attr_lo=:#o}' ... f' {zi.filename=}' ... ) Wheels and sdists Wheels: We generate wheels according to: https://packaging.python.org/specifications/binary-distribution-format/ * `{name}-{version}.dist-info/RECORD` uses sha256 hashes. * Other `RECORD*` files such as `RECORD.jws` or `RECORD.p7s` are not generated. * `{name}-{version}.dist-info/WHEEL` has: * `Wheel-Version: 1.0` * `Root-Is-Purelib: false` * No support for signed wheels. Sdists: We generate sdist's according to: https://packaging.python.org/specifications/source-distribution-format/ ''' _metadata_version = '2.1' def __init__(self, name, version, *, author = None, author_email = None, classifier = None, description = None, description_content_type = None, download_url = None, home_page = None, keywords = None, license = None, # pylint: disable=redefined-builtin maintainer = None, maintainer_email = None, platform = None, # pylint: disable=redefined-outer-name project_url = None, provides_extra = None, requires_dist = None, requires_external = None, requires_python = None, summary = None, supported_platform = None, entry_points = None, fn_build = None, fn_clean = None, fn_sdist = None, pure=False, py_limited_api = None, tag_python = None, tag_abi = None, tag_platform = None, root = None, wheel_compression = zipfile.ZIP_DEFLATED, wheel_compresslevel = None, ): ''' The initial args before `entry_points` define the package metadata and closely follow the definitions in: https://packaging.python.org/specifications/core-metadata/ Args: name: Used for metadata `Name`. A string, the name of the Python package. version: Used for metadata `Version`. A string, the version of the Python package. Also see PEP-440 `Version Identification and Dependency Specification`. Special handling of PIPCL_CHANGE_VERSIONS: If environment variable PIPCL_CHANGE_VERSIONS is set, it is used to override `version`. PIPCL_CHANGE_VERSIONS should contain one or more lines matching: If any line's matches , then is changed to the line's . Unlike with `requires_dist` it does not make sense to have non-exact version number here, so must start with '=='. For example if foo/ contains a package called foo that uses pipcl, then this: $ export PIPCL_CHANGE_VERSIONS=' > ^foo$ 1.2.3.4 > ^bar$ 3.4.5 ' $ pip wheel foo/ ... $ * Will create a foo wheel with version 1.2.3.4, regardless of what version is specified in foo/setup.py. * Any dependency of package foo on package bar, will be changed to `bar==3.4.5`. Note that if is not in the form `^...$`, it may match extra unintended package names. author: Used for metadata `Author`. Author name. author_email: Used for metadata `Author-email`. Author email. classifier: Used for metadata `Classifier`. A string or list of strings. Also see: * https://pypi.org/pypi?%3Aaction=list_classifiers * https://pypi.org/classifiers/ description: Used for metadata `Description`. A string. If contains newlines, a detailed description of the package. Otherwise the path of a file containing the detailed description of the package. description_content_type: Used for metadata `Description-Content-Type`. A string describing markup of `description` arg. For example `text/markdown; variant=GFM`. If not specified we try to infer a value from file suffix. download_url: Used for metadata `Download-URL`. Where this version can be downloaded from. Deprecated since metadata-1.2 - use project_url. home_page: Used for metadata `Home-page`. URL of home page. Deprecated since metadata-1.2 - use project_url. keywords: Used for metadata `Keywords`. A string containing comma-separated keywords. license: Used for metadata `License`. A string containing the license text. Written into metadata file `COPYING`. Is also written into metadata itself if not multi-line. maintainer: Used for metadata `Maintainer`. Maintainer. maintainer_email: Used for metadata `Maintainer-email`. Maintainer email. platform: Used for metadata `Platform`. A string or list of strings. project_url: Used for metadata `Project-URL`. A string or list of strings, each of the form: `{name}, {url}`. Well-known names include: homepage, source, documentation, issues; for more details, see: https://packaging.python.org/en/latest/specifications/well-known-project-urls/#well-known-project-urls provides_extra: Used for metadata `Provides-Extra`. A string or list of strings. requires_dist: Used for metadata `Requires-Dist`. A string or list of strings, Python packages required at runtime. None items are ignored. If environment variable PIPCL_CHANGE_VERSIONS is set, version numbers for matching items in will be overridden - see description of for details. requires_external: Used for metadata `Requires-External`. A string or list of strings. requires_python: Used for metadata `Requires-Python`. A string or list of strings. summary: Used for metadata `Summary`. A string, short description of the package. supported_platform: Used for metadata `Supported-Platform`. A string or list of strings. entry_points: A string specifying *.dist-info/entry_points.txt, for example:: [console_scripts] foo_cli = foo.__main__:main See: https://packaging.python.org/en/latest/specifications/entry-points/ Alternatively one can specify a dict similar to that supported by setuptools (see https://setuptools.pypa.io/en/latest/userguide/entry_point.html):: { 'console_scripts': [ 'foo_cli = foo.__main__:main', ], } fn_build: A function taking no args, or a single `config_settings` dict arg (as described in PEP-517). This should build all required files and return a list of items; each item should be a tuple `(from_, to_)`, or a single string `path` which is treated as the tuple `(path, path)`. `from_` can be a string or a `bytes`. If a string it should be the path to a file; a relative path is treated as relative to `root`. If a `bytes` it is the contents of the file to be added. `to_` identifies what the file should be called within a wheel or when installing. If `to_` is empty or `/` we set it to the leaf of `from_` (`from_` must not be a `bytes`) - i.e. we place the file in the root directory of the wheel; otherwise if `to_` ends with `/` the leaf of `from_` is appended to it (and `from_` must not be a `bytes`). Initial `$dist-info/` in `_to` is replaced by `{name}-{version}.dist-info/`; this is useful for license files etc. Initial `$data/` in `_to` is replaced by `{name}-{version}.data/`. We do not enforce particular subdirectories, instead it is up to `fn_build()` to specify specific subdirectories such as `purelib`, `headers`, `scripts`, `data` etc. If we are building a wheel (e.g. `python setup.py bdist_wheel`, or PEP-517 pip calls `self.build_wheel()`), we add file `from_` to the wheel archive with name `to_`. If we are installing (e.g. `install` command in the argv passed to `self.handle_argv()`), then we copy `from_` to `{sitepackages}/{to_}`, where `sitepackages` is the installation directory, the default being `sysconfig.get_path('platlib')` e.g. `myvenv/lib/python3.9/site-packages/`. When calling this function, we assert that the file pyproject.toml exists in the current directory. (We do this here rather than in pipcl.Package's constructor, as otherwise importing setup.py from non-package-related code could fail.) fn_clean: A function taking a single arg `all_` that cleans generated files. `all_` is true iff `--all` is in argv. For safety and convenience, can also returns a list of files/directory paths to be deleted. Relative paths are interpreted as relative to `root`. All paths are asserted to be within `root`. fn_sdist: A function taking no args, or a single `config_settings` dict arg (as described in PEP517), that returns a list of items to be copied into the sdist. The list should be in the same format as returned by `fn_build`. It can be convenient to use `pipcl.git_items()`. The specification for sdists requires that the list contains `pyproject.toml`; we enforce this with a Python assert. pure: If true we mark wheels as pure python - for example default wheel tag will be `py3-none-any`. py_limited_api: If true we build extension modules that use the Python Limited API. We use the version of `sys.executable` to define `Py_LIMITED_API` when compiling extensions, and use ABI tag `abi3` in the wheel name if argument `tag_abi` is None. tag_python: First element of wheel tag defined in PEP-425. If None we use `cp{version}`. tag_abi: Second element of wheel tag defined in PEP-425. If None we use `none`. tag_platform: Third element of wheel tag defined in PEP-425. Default is `os.environ('AUDITWHEEL_PLAT')` if set, otherwise derived from `sysconfig.get_platform()` (was `setuptools.distutils.util.get_platform()`, before that `distutils.util.get_platform()` as specified in the PEP), e.g. `openbsd_7_0_amd64`. root: Root of package, defaults to current directory. wheel_compression: Used as `zipfile.ZipFile()`'s `compression` parameter when creating wheels. Occurrences of `None` in lists are ignored. ''' assert name assert version def assert_str( v): if v is not None: assert isinstance( v, str), f'Not a string: {v!r}' def assert_str_or_multi( v): if v is not None: assert isinstance( v, (str, tuple, list)), f'Not a string, tuple or list: {v!r}' assert_str( name) assert_str( version) assert_str_or_multi( platform) assert_str_or_multi( supported_platform) assert_str( summary) assert_str( description) assert_str( description_content_type) assert_str( keywords) assert_str( home_page) assert_str( download_url) assert_str( author) assert_str( author_email) assert_str( maintainer) assert_str( maintainer_email) assert_str( license) assert_str_or_multi( classifier) assert_str_or_multi( requires_dist) assert_str_or_multi( requires_python) assert_str_or_multi( requires_external) assert_str_or_multi( project_url) assert_str_or_multi( provides_extra) assert re.match('^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])\\Z', name, re.IGNORECASE), ( f'Invalid package name' f' (https://packaging.python.org/en/latest/specifications/name-normalization/)' f': {name!r}' ) # https://packaging.python.org/en/latest/specifications/core-metadata/. assert re.match('([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$', name, re.IGNORECASE), \ f'Bad name: {name!r}' _assert_version_pep_440(version) # https://packaging.python.org/en/latest/specifications/binary-distribution-format/ if tag_python: assert '-' not in tag_python if tag_abi: assert '-' not in tag_abi if tag_platform: assert '-' not in tag_platform self.name = name self.version = version self.pure = pure self.platform = platform self.supported_platform = supported_platform self.summary = summary self.description = description self.description_content_type = description_content_type self.keywords = keywords self.home_page = home_page self.download_url = download_url self.author = author self.author_email = author_email self.maintainer = maintainer self.maintainer_email = maintainer_email self.license = license self.classifier = classifier self.requires_dist = requires_dist self.requires_python = requires_python self.requires_external = requires_external self.project_url = project_url self.provides_extra = provides_extra self.entry_points = entry_points self.root = os.path.abspath(root if root else os.getcwd()) self.fn_build = fn_build self.fn_clean = fn_clean self.fn_sdist = fn_sdist self.tag_python_ = tag_python self.tag_abi_ = tag_abi self.tag_platform_ = tag_platform self.py_limited_api = py_limited_api self.wheel_compression = wheel_compression self.wheel_compresslevel = wheel_compresslevel # If true and we are building for graal, we set PIPCL_PYTHON_CONFIG to # a command that will print includes/libs from graal_py's sysconfig. # self.graal_legacy_python_config = True # Override version numbers if PIPCL_CHANGE_VERSIONS is set. PIPCL_CHANGE_VERSIONS = os.environ.get('PIPCL_CHANGE_VERSIONS') log(f'{PIPCL_CHANGE_VERSIONS=}') if PIPCL_CHANGE_VERSIONS: # Override package version. name2, version2 = version_override(self.name, PIPCL_CHANGE_VERSIONS) if name2: assert name2 == self.name assert version2.startswith('==') version3 = version2[2:] log(f'Changing {self.version=} to {version3=}.') self.version = version3 _assert_version_pep_440(self.version) # Override requires_dist versions. # if self.requires_dist: requires_dist2 = self.requires_dist if isinstance(requires_dist2, str): requires_dist2 = [requires_dist2] else: requires_dist2 = list(requires_dist2) requires_dist3 = version_override_multiple(requires_dist2, PIPCL_CHANGE_VERSIONS) self.requires_dist = requires_dist3 # Override our caller's get_requires_for_build_wheel(). # frame = inspect.stack(context=0)[1] assert os.path.basename(frame.filename) == 'setup.py', f'{frame.filename=}' grfbw = frame.frame.f_globals.get('get_requires_for_build_wheel') if grfbw: def grfbw_wrap(config_settings=None): log(f'Wrapping get_requires_for_build_wheel().') ret1 = grfbw(config_settings=config_settings) ret2 = version_override_multiple(ret1, PIPCL_CHANGE_VERSIONS) log(f'{ret1=}.') log(f'{ret2=}.') if ret2 != ret1: log(f'Overriding return from get_requires_for_build_wheel():') for r1, r2 in zip(ret1, ret2, strict=1): if r1 != r2: log(f'Changing {r1!r} to {r2!r}') return ret2 frame.frame.f_globals['get_requires_for_build_wheel'] = grfbw_wrap else: log(f'Warning: PIPCL_CHANGE_VERSIONS is set, but unable to find get_requires_for_build_wheel().') if 0: self._caller_globals = frame.frame.f_globals for n, v in frame.frame.f_globals.items(): log(f' {n=} {v=}') def build_wheel(self, wheel_directory, config_settings=None, metadata_directory=None, ): ''' A PEP-517 `build_wheel()` function. Also called by `handle_argv()` to handle the `bdist_wheel` command. Returns leafname of generated wheel within `wheel_directory`. ''' log2( f' wheel_directory={wheel_directory!r}' f' config_settings={config_settings!r}' f' metadata_directory={metadata_directory!r}' ) prebuilt = f'PIPCL_PREBUILT_WHEEL_{_normalise2(self.name)}' prebuilt_location = os.environ.get(prebuilt) log(f'{prebuilt}={prebuilt_location!r}') if prebuilt_location: log(f'Getting pre-built wheel using pip, from {prebuilt_location!r}.') with tempfile.TemporaryDirectory(prefix='pipcl-') as dir_path: run(['pip', 'wheel', '--no-deps', '-w', dir_path, prebuilt_location]) wheel_path = glob.glob(f'{dir_path}/*.whl') assert len(wheel_path) == 1, f'{wheel_path=}' wheel_path = wheel_path[0] shutil.move(wheel_path, wheel_directory) wheel_leaf = os.path.basename(wheel_path) log(f'Returning prebuilt {wheel_leaf=}') return wheel_leaf if os.environ.get('CIBUILDWHEEL') == '1': # Don't special-case graal builds when running under cibuildwheel. pass elif sys.implementation.name == 'graalpy': # We build for Graal by building a native Python wheel with Graal # Python's include paths and library directory. We then rename the # wheel to contain graal's tag etc. # log0(f'### Graal build: deferring to cpython.') python_native = os.environ.get('PIPCL_GRAAL_PYTHON') assert python_native, f'Graal build requires that PIPCL_GRAAL_PYTHON is set.' env_extra = dict( PIPCL_SYSCONFIG_PATH_include = sysconfig.get_path('include'), PIPCL_SYSCONFIG_PATH_platinclude = sysconfig.get_path('platinclude'), PIPCL_SYSCONFIG_CONFIG_VAR_LIBDIR = sysconfig.get_config_var('LIBDIR'), ) # Tell native build to run pipcl.py itself to get python-config # information about include paths etc. if self.graal_legacy_python_config: env_extra['PIPCL_PYTHON_CONFIG'] = f'{python_native} {os.path.abspath(__file__)} --graal-legacy-python-config' # Create venv. venv_name = os.environ.get('PIPCL_GRAAL_NATIVE_VENV') if venv_name: log1(f'Graal using pre-existing {venv_name=}') else: venv_name = 'venv-pipcl-graal-native' run(f'{shlex.quote(python_native)} -m venv {venv_name}') log1(f'Graal using {venv_name=}') newfiles = NewFiles(f'{wheel_directory}/*.whl') run( f'. {venv_enter_cmd(venv_name)} && python setup.py --dist-dir {shlex.quote(wheel_directory)} bdist_wheel', env_extra = env_extra, prefix = f'pipcl.py graal {python_native}: ', ) wheel = newfiles.get_one() wheel_leaf = os.path.basename(wheel) python_major_minor = run(f'{shlex.quote(python_native)} -c "import platform; import sys; sys.stdout.write(str().join(platform.python_version_tuple()[:2]))"', capture=1) cpabi = f'cp{python_major_minor}-abi3' assert cpabi in wheel_leaf, f'Expected wheel to be for {cpabi=}, but {wheel=}.' graalpy_ext_suffix = sysconfig.get_config_var('EXT_SUFFIX') log1(f'{graalpy_ext_suffix=}') m = re.match(r'\.graalpy(\d+[^\-]*)-(\d+)', graalpy_ext_suffix) gpver = m[1] cpver = m[2] graalpy_wheel_tag = f'graalpy{cpver}-graalpy{gpver}_{cpver}_native' name = wheel_leaf.replace(cpabi, graalpy_wheel_tag) destination = f'{wheel_directory}/{name}' log0(f'### Graal build: copying {wheel=} to {destination=}') # Copying results in two wheels which appears to confuse pip, showing: # Found multiple .whl files; unspecified behaviour. Will call build_wheel. os.rename(wheel, destination) log1(f'Returning {name=}.') return name wheel_name = self.wheel_name() path = f'{wheel_directory}/{wheel_name}' # Do a build and get list of files to copy into the wheel. # items = list() if self.fn_build: items = self._call_fn_build(config_settings) log2(f'Creating wheel: {path}') os.makedirs(wheel_directory, exist_ok=True) to_maxlen = 0 for item in items: _, (_to_abs, to_rel) = self._fromto(item) if isinstance(to_rel, str): to_maxlen = max(to_maxlen, len(to_rel)) record = _Record() with zipfile.ZipFile(path, 'w', self.wheel_compression, self.wheel_compresslevel) as z: def add(from_, to_): if isinstance(from_, str): z.write(from_, to_) record.add_file(from_, to_, to_maxlen=to_maxlen) elif isinstance(from_, bytes): zipfile_writestr(z, to_, from_, 0o644) record.add_content(from_, to_) else: assert 0 def add_str(content, to_): add(content.encode('utf8'), to_) dist_info_dir = self._dist_info_dir() # Add the files returned by fn_build(). # for item in items: from_, (_to_abs, to_rel) = self._fromto(item) add(from_, to_rel) # Add -.dist-info/WHEEL. # add_str( f'Wheel-Version: 1.0\n' f'Generator: pipcl\n' f'Root-Is-Purelib: {"true" if self.pure else "false"}\n' f'Tag: {self.wheel_tag_string()}\n' , f'{dist_info_dir}/WHEEL', ) # Add -.dist-info/METADATA. # add_str(self._metainfo(), f'{dist_info_dir}/METADATA') # Add -.dist-info/COPYING. if self.license: add_str(self.license, f'{dist_info_dir}/COPYING') # Add -.dist-info/entry_points.txt. entry_points_text = self._entry_points_text() if entry_points_text: add_str(entry_points_text, f'{dist_info_dir}/entry_points.txt') # Update -.dist-info/RECORD. This must be last. # zipfile_writestr( z, f'{dist_info_dir}/RECORD', record.get(f'{dist_info_dir}/RECORD'), 0o644, ) st = os.stat(path) log1( f'Have created wheel size={st.st_size:,}: {path}') if g_verbose >= 2: with zipfile.ZipFile(path, compression=self.wheel_compression) as z: log2(f'Contents are:') for zi in sorted(z.infolist(), key=lambda z: z.filename): log2(f' {zi.file_size: 10,d} {zi.filename}') return os.path.basename(path) def build_sdist(self, sdist_directory, formats, config_settings=None, ): ''' A PEP-517 `build_sdist()` function. Also called by `handle_argv()` to handle the `sdist` command. Returns leafname of generated archive within `sdist_directory`. ''' assert self.fn_sdist, f'fn_sdist() not provided.' log2( f' sdist_directory={sdist_directory!r}' f' formats={formats!r}' f' config_settings={config_settings!r}' ) if formats and formats != 'gztar': raise Exception( f'Unsupported: formats={formats}') items = list() if inspect.signature(self.fn_sdist).parameters: items = self.fn_sdist(config_settings) else: items = self.fn_sdist() prefix = f'{_normalise2(self.name)}-{self.version}' os.makedirs(sdist_directory, exist_ok=True) tarpath = f'{sdist_directory}/{prefix}.tar.gz' log2(f'Creating sdist: {tarpath}') with tarfile.open(tarpath, 'w:gz') as tar: names_in_tar = list() def check_name(name): if name in names_in_tar: raise Exception(f'Name specified twice: {name}') names_in_tar.append(name) def add(from_, name): check_name(name) if isinstance(from_, str): log2( f'Adding file: {os.path.relpath(from_)} => {name}') tar.add( from_, f'{prefix}/{name}', recursive=False) elif isinstance(from_, bytes): log2( f'Adding: {name}') ti = tarfile.TarInfo(f'{prefix}/{name}') ti.size = len(from_) ti.mtime = time.time() tar.addfile(ti, io.BytesIO(from_)) else: assert 0 def add_string(text, name): textb = text.encode('utf8') return add(textb, name) found_pyproject_toml = False for item in items: from_, (_to_abs, to_rel) = self._fromto(item) if isinstance(from_, bytes): add(from_, to_rel) else: if from_.startswith(f'{os.path.abspath(sdist_directory)}/'): # Source files should not be inside . assert 0, f'Path is inside sdist_directory={sdist_directory}: {from_!r}' assert os.path.exists(from_), f'Path does not exist: {from_!r}' assert os.path.isfile(from_), f'Path is not a file: {from_!r}' add(from_, to_rel) if to_rel == 'pyproject.toml': found_pyproject_toml = True assert found_pyproject_toml, f'Cannot create sdist because file not specified: pyproject.toml' # Always add a PKG-INFO file. add_string(self._metainfo(), 'PKG-INFO') if self.license: if 'COPYING' in names_in_tar: log2(f'Not writing .license because file already in sdist: COPYING') else: add_string(self.license, 'COPYING') log1( f'Have created sdist: {tarpath}') return os.path.basename(tarpath) def wheel_tag_string(self): ''' Returns --. ''' return f'{self.tag_python()}-{self.tag_abi()}-{self.tag_platform()}' def tag_python(self): ''' Get two-digit python version, e.g. 'cp3.8' for python-3.8.6. ''' if self.tag_python_: ret = self.tag_python_ elif self.pure: return 'py3' else: ret = 'cp' + ''.join(platform.python_version().split('.')[:2]) assert '-' not in ret return ret def tag_abi(self): ''' ABI tag. ''' Py_GIL_DISABLED = sysconfig.get_config_var('Py_GIL_DISABLED') if self.tag_abi_: return self.tag_abi_ elif self.pure: return 'none' elif self.py_limited_api: assert Py_GIL_DISABLED != 1, \ f'{self.py_limited_api=} and {Py_GIL_DISABLED=} are not supported together as of 2026-02-20, e.g. see PEP 803 and PEP 809.' return 'abi3' elif Py_GIL_DISABLED == 1: ret = '' ret += 'cp' ret += ''.join(platform.python_version().split('.')[:2]) ret += 't' return ret else: return 'none' def tag_platform(self): ''' Find platform tag used in wheel filename. ''' ret = self.tag_platform_ log0(f'From self.tag_platform_: {ret=}.') if not ret and self.pure: return 'any' if not ret: # Prefer this to PEP-425. Appears to be undocumented, # but set in manylinux docker images and appears # to be used by cibuildwheel and auditwheel, e.g. # https://github.com/rapidsai/shared-action-workflows/issues/80 ret = os.environ.get( 'AUDITWHEEL_PLAT') log0(f'From AUDITWHEEL_PLAT: {ret=}.') if not ret: # Notes: # # PEP-425. On Linux gives `linux_x86_64` which is rejected by # pypi.org. # # On local MacOS/arm64 mac-mini have seen sysconfig.get_platform() # unhelpfully return `macosx-10.9-universal2` if `python3` is the # system Python /usr/bin/python3; this happens if we source `. # /etc/profile`. # ret = sysconfig.get_platform() ret = ret.replace('-', '_').replace('.', '_').lower() log0(f'From sysconfig.get_platform(): {ret=}.') ret = _macos_fixup_platform_tag(ret) log0( f'tag_platform(): returning {ret=}.') assert '-' not in ret return ret def wheel_name(self): ret = f'{_normalise2(self.name)}-{self.version}-{self.tag_python()}-{self.tag_abi()}-{self.tag_platform()}.whl' assert ret.count('-') == 4, f'Expected 4 dash characters in {ret=}.' return ret def wheel_name_match(self, wheel): ''' Returns true if `wheel` matches our wheel. We basically require the name to be the same, except that we accept platform tags that contain extra items (see pep-0600/), for example we return true with: self: foo-cp38-none-manylinux2014_x86_64.whl wheel: foo-cp38-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl ''' log2(f'{wheel=}') assert wheel.endswith('.whl') wheel2 = wheel[:-len('.whl')] name, version, tag_python, tag_abi, tag_platform = wheel2.split('-') py_limited_api_compatible = False if self.py_limited_api and tag_abi == 'abi3': # Allow lower tag_python number. m = re.match('cp([0-9]+)', tag_python) tag_python_int = int(m.group(1)) m = re.match('cp([0-9]+)', self.tag_python()) tag_python_int_self = int(m.group(1)) if tag_python_int <= tag_python_int_self: # This wheel uses Python stable ABI same or older than ours, so # we can use it. log2(f'py_limited_api; {tag_python=} compatible with {self.tag_python()=}.') py_limited_api_compatible = True log2(f'{_normalise2(self.name) == name=}') log2(f'{self.version == version=}') log2(f'{self.tag_python() == tag_python=} {self.tag_python()=} {tag_python=}') log2(f'{py_limited_api_compatible=}') log2(f'{self.tag_abi() == tag_abi=}') log2(f'{self.tag_platform() in tag_platform.split(".")=}') log2(f'{self.tag_platform()=}') log2(f'{tag_platform.split(".")=}') ret = (1 and _normalise2(self.name) == name and self.version == version and (self.tag_python() == tag_python or py_limited_api_compatible) and self.tag_abi() == tag_abi and self.tag_platform() in tag_platform.split('.') ) log2(f'Returning {ret=}.') return ret def _entry_points_text(self): if self.entry_points: if isinstance(self.entry_points, str): return self.entry_points ret = '' for key, values in self.entry_points.items(): ret += f'[{key}]\n' for value in values: ret += f'{value}\n' return ret def _call_fn_build( self, config_settings=None): assert self.fn_build assert os.path.isfile('pyproject.toml'), ( f'Cannot create package because file does not exist: {os.path.abspath("pyproject.toml")}' ) log2(f'calling self.fn_build={self.fn_build}') if inspect.signature(self.fn_build).parameters: ret = self.fn_build(config_settings) else: ret = self.fn_build() assert isinstance( ret, (list, tuple)), \ f'Expected list/tuple from {self.fn_build} but got: {ret!r}' # Check that any extensions that we have built, have same # py_limited_api value. If package is marked with py_limited_api=True # then non-py_limited_api extensions seem to fail at runtime on # Windows. # # (We could possibly allow package py_limited_api=False and extensions # py_limited_api=True, but haven't tested this, and it seems simpler to # be strict.) for item in ret: from_, (_to_abs, _to_rel) = self._fromto(item) from_abs = os.path.abspath(from_) is_py_limited_api = _extensions_to_py_limited_api.get(from_abs) if is_py_limited_api is not None: assert bool(self.py_limited_api) == bool(is_py_limited_api), ( f'Extension was built with' f' py_limited_api={is_py_limited_api} but pipcl.Package' f' name={self.name!r} has' f' py_limited_api={self.py_limited_api}:' f' {from_abs!r}' ) return ret def _argv_clean(self, all_): ''' Called by `handle_argv()`. ''' if not self.fn_clean: return paths = self.fn_clean(all_) if paths: if isinstance(paths, str): paths = paths, for path in paths: if not os.path.isabs(path): path = os.path.join(self.root, path) path = os.path.abspath(path) assert path.startswith(self.root+os.sep), \ f'path={path!r} does not start with root={self.root+os.sep!r}' log(f'Removing: {path}') fs_remove(path) def install(self, record_path=None, root=None): ''' Called by `handle_argv()` to handle `install` command.. ''' log2( f'{record_path=} {root=}') # Do a build and get list of files to install. # items = list() if self.fn_build: items = self._call_fn_build( dict()) root2 = install_dir(root) log2( f'{root2=}') log1( f'Installing into: {root2!r}') dist_info_dir = self._dist_info_dir() if not record_path: record_path = f'{root2}/{dist_info_dir}/RECORD' record = _Record() def add_file(from_, to_abs, to_rel): os.makedirs( os.path.dirname( to_abs), exist_ok=True) if isinstance(from_, bytes): log2(f'Copying content into {to_abs}.') with open(to_abs, 'wb') as f: f.write(from_) record.add_content(from_, to_rel) else: log0(f'{from_=}') log2(f'Copying from {os.path.relpath(from_, self.root)} to {to_abs}') shutil.copy2( from_, to_abs) record.add_file(from_, to_rel) def add_str(content, to_abs, to_rel): log2( f'Writing to: {to_abs}') os.makedirs( os.path.dirname( to_abs), exist_ok=True) with open( to_abs, 'w') as f: f.write( content) record.add_content(content, to_rel) for item in items: from_, (to_abs, to_rel) = self._fromto(item) log0(f'{from_=} {to_abs=} {to_rel=}') to_abs2 = f'{root2}/{to_rel}' add_file( from_, to_abs2, to_rel) add_str( self._metainfo(), f'{root2}/{dist_info_dir}/METADATA', f'{dist_info_dir}/METADATA') if self.license: add_str( self.license, f'{root2}/{dist_info_dir}/COPYING', f'{dist_info_dir}/COPYING') entry_points_text = self._entry_points_text() if entry_points_text: add_str( entry_points_text, f'{root2}/{dist_info_dir}/entry_points.txt', f'{dist_info_dir}/entry_points.txt', ) log2( f'Writing to: {record_path}') with open(record_path, 'w') as f: f.write(record.get()) log2(f'Finished.') def _argv_dist_info(self, root): ''' Called by `handle_argv()`. There doesn't seem to be any documentation for `setup.py dist_info`, but it appears to be like `egg_info` except it writes to a slightly different directory. ''' if root is None: root = f'{_normalise2(self.name)}-{self.version}.dist-info' self._write_info(f'{root}/METADATA') if self.license: with open( f'{root}/COPYING', 'w') as f: f.write( self.license) def _argv_egg_info(self, egg_base): ''' Called by `handle_argv()`. ''' if egg_base is None: egg_base = '.' self._write_info(f'{egg_base}/.egg-info') def _write_info(self, dirpath=None): ''' Writes egg/dist info to files in directory `dirpath` or `self.root` if `None`. ''' if dirpath is None: dirpath = self.root log2(f'Creating files in directory {dirpath}') os.makedirs(dirpath, exist_ok=True) with open(os.path.join(dirpath, 'PKG-INFO'), 'w') as f: f.write(self._metainfo()) # These don't seem to be required? # #with open(os.path.join(dirpath, 'SOURCES.txt', 'w') as f: # pass #with open(os.path.join(dirpath, 'dependency_links.txt', 'w') as f: # pass #with open(os.path.join(dirpath, 'top_level.txt', 'w') as f: # f.write(f'{self.name}\n') #with open(os.path.join(dirpath, 'METADATA', 'w') as f: # f.write(self._metainfo()) def handle_argv(self, argv): ''' Attempt to handle old-style (pre PEP-517) command line passed by old releases of pip to a `setup.py` script, and manual running of `setup.py`. This is partial support at best. ''' global g_verbose #log2(f'argv: {argv}') class ArgsRaise: pass class Args: ''' Iterates over argv items. ''' def __init__( self, argv): self.items = iter( argv) def next( self, eof=ArgsRaise): ''' Returns next arg. If no more args, we return or raise an exception if is ArgsRaise. ''' try: return next( self.items) except StopIteration as e: if eof is ArgsRaise: raise Exception('Not enough args') from e return eof command = None opt_all = None opt_dist_dir = 'dist' opt_egg_base = None opt_formats = None opt_install_headers = None # pylint: disable=unused-variable opt_record = None opt_root = None args = Args(argv[1:]) while 1: # pylint: disable=multiple-statements,consider-using-in arg = args.next(None) if arg is None: break if arg in ('-h', '--help', '--help-commands'): # codespell:ignore log0(textwrap.dedent(''' Usage: [...] [...] Commands: bdist_wheel Creates a wheel called /--
.whl, where is "dist" or as specified by --dist-dir, and
encodes ABI and platform etc. clean Cleans build files. dist_info Creates files in -.dist-info/ or directory specified by --egg-base. egg_info Creates files in .egg-info/ or directory directory specified by --egg-base. install Builds and installs. Writes installation information to if --record was specified. sdist Make a source distribution: /-.tar.gz Options: --all Used by "clean". --compile Ignored. --dist-dir | -d Default is "dist". --egg-base Used by "egg_info". --formats Used by "sdist". --install-headers Ignored. --python-tag Ignored. --record Used by "install". --root Used by "install". --single-version-externally-managed Ignored. --verbose -v Extra diagnostics. Other: windows-vs [-y ] [-v ] [-g ] [--verbose] Windows only; looks for matching Python. ''')) return elif arg in ('bdist_wheel', 'clean', 'dist_info', 'egg_info', 'install', 'sdist'): assert command is None, 'Two commands specified: {command} and {arg}.' command = arg elif arg in ('windows-vs', 'windows-python', 'show-sysconfig'): assert command is None, 'Two commands specified: {command} and {arg}.' command = arg elif arg == '--all': opt_all = True elif arg == '--compile': pass elif arg == '--dist-dir' or arg == '-d': opt_dist_dir = args.next() elif arg == '--egg-base': opt_egg_base = args.next() elif arg == '--formats': opt_formats = args.next() elif arg == '--install-headers': opt_install_headers = args.next() # noqa: F841 elif arg == '--python-tag': pass elif arg == '--record': opt_record = args.next() elif arg == '--root': opt_root = args.next() elif arg == '--single-version-externally-managed': pass elif arg == '--verbose' or arg == '-v': g_verbose += 1 else: raise Exception(f'Unrecognised arg: {arg}') assert command, 'No command specified' log1(f'Handling command={command}') # pylint: disable=multiple-statements if 0: pass elif command == 'bdist_wheel': self.build_wheel(opt_dist_dir) elif command == 'clean': self._argv_clean(opt_all) elif command == 'dist_info': self._argv_dist_info(opt_egg_base) elif command == 'egg_info': self._argv_egg_info(opt_egg_base) elif command == 'install': self.install(opt_record, opt_root) elif command == 'sdist': self.build_sdist(opt_dist_dir, opt_formats) elif command == 'windows-python': version = None while 1: arg = args.next(None) if arg is None: break elif arg == '-v': version = args.next() elif arg == '--verbose': g_verbose += 1 else: assert 0, f'Unrecognised {arg=}' python = wdev.WindowsPython(version=version) print(f'Python is:\n{python.description_ml(" ")}') elif command == 'windows-vs': grade = None version = None year = None while 1: arg = args.next(None) if arg is None: break elif arg == '-g': grade = args.next() elif arg == '-v': version = args.next() elif arg == '-y': year = args.next() elif arg == '--verbose': g_verbose += 1 else: assert 0, f'Unrecognised {arg=}' vs = wdev.windows_vs(year=year, grade=grade, version=version) print(f'Visual Studio is:\n{vs.description_ml(" ")}') # pylint: disable=too-many-nested-blocks elif command == 'show-sysconfig': show_sysconfig() for mod in platform, sys: log0(f'{mod.__name__}:') for n in dir(mod): if n.startswith('_'): continue log0(f'{mod.__name__}.{n}') if mod is platform and n == 'uname': continue if mod is platform and n == 'pdb': continue if mod is sys and n in ('breakpointhook', 'exit'): # We don't want to call these. continue v = getattr(mod, n) if callable(v): try: v = v() except Exception: pass else: #print(f'{n=}', flush=1) try: print(f' {mod.__name__}.{n}()={v!r}') except Exception: print(f' Failed to print value of {mod.__name__}.{n}().') else: try: print(f' {mod.__name__}.{n}={v!r}') except Exception: print(f' Failed to print value of {mod.__name__}.{n}.') else: assert 0, f'Unrecognised command: {command}' log2(f'Finished handling command: {command}') def __str__(self): return ('{' f'name={self.name!r}' f' version={self.version!r}' f' platform={self.platform!r}' f' supported_platform={self.supported_platform!r}' f' summary={self.summary!r}' f' description={self.description!r}' f' description_content_type={self.description_content_type!r}' f' keywords={self.keywords!r}' f' home_page={self.home_page!r}' f' download_url={self.download_url!r}' f' author={self.author!r}' f' author_email={self.author_email!r}' f' maintainer={self.maintainer!r}' f' maintainer_email={self.maintainer_email!r}' f' license={self.license!r}' f' classifier={self.classifier!r}' f' requires_dist={self.requires_dist!r}' f' requires_python={self.requires_python!r}' f' requires_external={self.requires_external!r}' f' project_url={self.project_url!r}' f' provides_extra={self.provides_extra!r}' f' root={self.root!r}' f' fn_build={self.fn_build!r}' f' fn_sdist={self.fn_sdist!r}' f' fn_clean={self.fn_clean!r}' f' tag_python={self.tag_python_!r}' f' tag_abi={self.tag_abi_!r}' f' tag_platform={self.tag_platform_!r}' '}' ) def _dist_info_dir( self): return f'{_normalise2(self.name)}-{self.version}.dist-info' def _metainfo(self): ''' Returns text for `.egg-info/PKG-INFO` file, or `PKG-INFO` in an sdist `.tar.gz` file, or `...dist-info/METADATA` in a wheel. ''' # 2021-04-30: Have been unable to get multiline content working on # test.pypi.org so we currently put the description as the body after # all the other headers. # ret = [''] def add(key, value): if value is None and key == 'Description-Content-Type': # Try to figure out from description suffix. if self.description and '\n' not in self.description: if self.description.endswith('.md'): value = 'text/markdown' elif self.description.endswith('.rst'): value = 'text/x-rst' if value: log(f'## Warning: description_content_type unspecified, inferring {value!r} from suffix of description={self.description!r}.') if value is None: return if isinstance( value, (tuple, list)): for v in value: if v is not None: add( key, v) return if key == 'License' and '\n' in value: # This is ok because we write `self.license` into # *.dist-info/COPYING. # log1( f'Omitting license because contains newline(s).') return assert '\n' not in value, f'key={key} value contains newline: {value!r}' if key == 'Project-URL': assert value.count(',') == 1, f'For {key=}, should have one comma in {value!r}.' ret[0] += f'{key}: {value}\n' #add('Description', self.description) add('Metadata-Version', self._metadata_version) # These names are from: # https://packaging.python.org/specifications/core-metadata/ # for name in ( 'Name', 'Version', 'Platform', 'Supported-Platform', 'Summary', 'Description-Content-Type', 'Keywords', 'Home-page', 'Download-URL', 'Author', 'Author-email', 'Maintainer', 'Maintainer-email', 'License', 'Classifier', 'Requires-Dist', 'Requires-Python', 'Requires-External', 'Project-URL', 'Provides-Extra', ): identifier = name.lower().replace( '-', '_') add( name, getattr( self, identifier)) ret = ret[0] # Append description as the body if self.description: if '\n' in self.description: description_text = self.description.strip() else: with open(self.description, encoding='utf8') as f: description_text = f.read() ret += '\n' # Empty line separates headers from body. ret += description_text ret += '\n' return ret def _path_relative_to_root(self, path, assert_within_root=True): ''' Returns `(path_abs, path_rel)`, where `path_abs` is absolute path and `path_rel` is relative to `self.root`. Interprets `path` as relative to `self.root` if not absolute. We use `os.path.realpath()` to resolve any links. if `assert_within_root` is true, assert-fails if `path` is not within `self.root`. ''' if os.path.isabs(path): p = path else: p = os.path.join(self.root, path) p = os.path.realpath(os.path.abspath(p)) if assert_within_root: # We use os.path.realpath() here because, on Windows, it transforms # paths to match the actual case on the case-insensitive but # case-preserving filesystem. Otherwise we would assert-fail due # to irrelevant differences in case on Windows' case-insensitive # filing system. root_real = os.path.realpath(self.root) p_real = os.path.realpath(p) assert p_real.startswith(root_real+os.sep) or p_real == root_real, \ f'Path not within root={self.root+os.sep!r}: {path=} {p=}' p_rel = os.path.relpath(p, self.root) return p, p_rel def _fromto(self, p): ''' Returns `(from_, (to_abs, to_rel))`. If `p` is a string we convert to `(p, p)`. Otherwise we assert that `p` is a tuple `(from_, to_)` where `from_` is str/bytes and `to_` is str. If `from_` is a bytes it is contents of file to add, otherwise the path of an existing file; non-absolute paths are assumed to be relative to `self.root`. If `to_` is empty or `/` we set it to the leaf of `from_` (which must be a str) - i.e. we place the file in the root directory of the wheel; otherwise if `to_` ends with `/` we append the leaf of `from_` (which must be a str). If `to_` starts with `$dist-info/`, we replace this with `self._dist_info_dir()`. If `to_` starts with `$data/`, we replace this with `{self.name}-{self.version}.data/`. We assert that `to_abs` is `within self.root`. `to_rel` is derived from the `to_abs` and is relative to self.root`. ''' if isinstance(p, str): p = p, p assert isinstance(p, tuple) and len(p) == 2 from_, to_ = p assert isinstance(from_, (str, bytes)) assert isinstance(to_, str) if to_ == '/' or to_ == '': to_ = os.path.basename(from_) elif to_.endswith('/'): to_ += os.path.basename(from_) prefix = '$dist-info/' if to_.startswith( prefix): to_ = f'{self._dist_info_dir()}/{to_[ len(prefix):]}' prefix = '$data/' if to_.startswith( prefix): to_ = f'{_normalise2(self.name)}-{self.version}.data/{to_[ len(prefix):]}' if isinstance(from_, str): from_, _ = self._path_relative_to_root( from_, assert_within_root=False) to_ = self._path_relative_to_root(to_) assert isinstance(from_, (str, bytes)) log2(f'returning {from_=} {to_=}') return from_, to_ _extensions_to_py_limited_api = dict() def make_overrides(overrides): ''' Returns if already a list, or a list of (regex, suffix) pairs obtained by splitting overrides into lines of ' ' pairs. ''' if isinstance(overrides, list): return overrides else: assert isinstance(overrides, str) ret = list() if not overrides: return ret for line in overrides.split('\n'): if not line: continue nv = line.split(' ') assert len(nv) == 2, f'Incorrect line in {overrides=}, should be ` `: {line!r}' regex, suffix = nv ret.append((regex, suffix)) return ret def version_override(name, overrides): ''' If has a match for , we return (name, suffix), e.g. ('foo', '==1.2.3'). Otherwise return (None, None). name: Package name, may contain suffix such as `==1.2.3`, which is ignored. overrides: Something that can be passed to make_overrides(). ''' overrides2 = make_overrides(overrides) name_requirement = packaging.requirements.Requirement(name) for regex, suffix in overrides2: m = re.match(regex, name_requirement.name) if m: return (name_requirement.name, suffix) return None, None def version_override_multiple(names, overrides): ''' Returns copy of list of names, overriding version numbers according to . ''' overrides2 = make_overrides(overrides) names2 = list() for name in names: name2, suffix = version_override(name, overrides2) if name2: names2.append(f'{name2}{suffix}') else: names2.append(name) if names2 != names: log(f'Changing {names=} to {names2=}.') return names2 def build_extension( name, path_i, outdir, *, builddir=None, includes=None, defines=None, libpaths=None, libs=None, optimise=True, debug=False, compiler_extra='', compiler_extra_cpp='', linker_extra='', swig=None, cpp=True, source_extra=None, prerequisites_swig=None, prerequisites_compile=None, prerequisites_link=None, infer_swig_includes=True, py_limited_api=False, nogil=False, return_paths=False, ): ''' Builds a Python extension module using SWIG. Works on Windows, Linux, MacOS and OpenBSD. On Unix, sets rpath when linking shared libraries. Args: name: Name of generated extension module. path_i: Path of input SWIG `.i` file. Internally we use swig to generate a corresponding `.c` or `.cpp` file. outdir: Output directory for generated files: * `{outdir}/{name}.py` * `{outdir}/_{name}.so` # Unix * `{outdir}/_{name}.*.pyd` # Windows We return the leafname of the `.so` or `.pyd` file. builddir: Where to put intermediate files, for example the .cpp file generated by swig and `.d` dependency files. Default is `outdir`. includes: A string, or a sequence of extra include directories to be prefixed with `-I`. defines: A string, or a sequence of extra preprocessor defines to be prefixed with `-D`. libpaths A string, or a sequence of library paths to be prefixed with `/LIBPATH:` on Windows or `-L` on Unix. libs A string, or a sequence of library names. Each item is prefixed with `-l` on non-Windows. optimise: Whether to use compiler optimisations and define NDEBUG. debug: Whether to build with debug symbols. compiler_extra: Extra compiler flags. Can be None. compiler_extra_cpp: Extra c++ compiler flags. Can be None. linker_extra: Extra linker flags. Can be None. swig: Swig command; if false we use 'swig'. cpp: If true we tell SWIG to generate C++ code instead of C. source_extra: Extra source files to build into the shared library, prerequisites_swig: prerequisites_compile: prerequisites_link: [These are mainly for use on Windows. On other systems we automatically generate dynamic dependencies using swig/compile/link commands' `-MD` and `-MF` args.] Sequences of extra input files/directories that should force running of swig, compile or link commands if they are newer than any existing generated SWIG `.i` file, compiled object file or shared library file. If present, the first occurrence of `True` or `False` forces re-run or no re-run. Any occurrence of None is ignored. If an item is a directory path we look for newest file within the directory tree. If not a sequence, we convert into a single-item list. prerequisites_swig We use swig's -MD and -MF args to generate dynamic dependencies automatically, so this is not usually required. prerequisites_compile prerequisites_link On non-Windows we use cc's -MF and -MF args to generate dynamic dependencies so this is not usually required. infer_swig_includes: If true, we extract `-I` and `-I ` args from `compile_extra` (also `/I` on windows) and use them with swig so that it can see the same header files as C/C++. This is useful when using enviromment variables such as `CC` and `CXX` to set `compile_extra`. py_limited_api: If true we build for current Python's limited API / stable ABI. Note that we will assert false if this extension is added to a pipcl.Package that has a different , because on Windows importing a non-py_limited_api extension inside a py_limited=True package fails. nogil: If true we use swig's `-nogil` flag which marks the extension as being thread safe. return_paths: If true we return `(py, lib)` where `py` is path of generated .py file and `lib` is path of generated shared library file. def build(): py, lib = pipcl.build_extension(...) return [ (py, 'mymodule/__init__.py'), (lib, 'mymodule/'), ] If `return_paths` is true: Returns `(py, lib)` where `py` is path of generated .py file and `lib` is path of generated shared library file; both of these will be in . This allows simpler usage like: def build(): py, lib = pipcl.build_extension(name='mymodule', ...) return [ (py, 'mymodule/__init__.py'), (lib, 'mymodule/'), ] Otherwise (the default): Returns the leafname of the generated library file within `outdir`, e.g. `_{name}.so` on Unix or `_{name}.cp311-win_amd64.pyd` on Windows. Usage is: def build(): so_leaf = pipcl.build_extension(name='mymodule', ...) return [ ('build/mymodule.py', 'mymodule/__init__.py'), (f'build/{so_leaf}', 'mymodule/'), ] Also creates `/.py`. ''' if builddir is None: builddir = outdir if not swig: swig = 'swig' if source_extra is None: source_extra = list() if isinstance(source_extra, str): source_extra = [source_extra] path_cpp = f'{builddir}/{os.path.basename(path_i)}' path_cpp += '.cpp' if cpp else '.c' os.makedirs(outdir, exist_ok=True) # Run SWIG. # if infer_swig_includes: # Extract include flags from `compiler_extra`. swig_includes_extra = '' compiler_extra_items = shlex.split(compiler_extra) i = 0 while i < len(compiler_extra_items): item = compiler_extra_items[i] # Swig doesn't seem to like a space after `I`. if item == '-I' or (windows() and item == '/I'): swig_includes_extra += f' -I{compiler_extra_items[i+1]}' i += 1 elif item.startswith('-I') or (windows() and item.startswith('/I')): swig_includes_extra += f' -I{compiler_extra_items[i][2:]}' i += 1 swig_includes_extra = swig_includes_extra.strip() deps_path = f'{path_cpp}.d' prerequisites_swig2 = _get_prerequisites( deps_path) run_if(f''' {swig} -Wall {'-c++' if cpp else ''} -python -module {name} -outdir {outdir} -o {path_cpp} -MD -MF {deps_path} {'-nogil' if nogil else ''} {_flags( includes, '-I')} {swig_includes_extra} {path_i} ''' , path_cpp, path_i, prerequisites_swig, prerequisites_swig2, ) if pyodide(): so_suffix = '.so' log0(f'pyodide: PEP-3149 suffix untested, so omitting. {_so_suffix()=}.') else: so_suffix = _so_suffix(use_so_versioning = not py_limited_api) path_so_leaf = f'_{name}{so_suffix}' path_so = f'{outdir}/{path_so_leaf}' # setuptools on Linux seems to use slightly different compile flags: # # -fwrapv -O3 -Wall -O2 -g0 -DPY_CALL_TRAMPOLINE # # Fun fact - on Linux, if the -L and -l options are before '{path_cpp}' # they seem to be ignored... # # Compile source files to object files. # path_os = list() for path_source in [path_cpp] + source_extra: path_o = f'{path_source}.obj' if windows() else f'{path_source}.o' path_os.append(path_o) prerequisites_path = f'{path_o}.d' command = compiler_command( includes=includes, defines=defines, optimise=optimise, debug=debug, compiler_extra=compiler_extra, compiler_extra_cpp=compiler_extra_cpp, python=True, py_limited_api=py_limited_api, source_paths=path_source, path_o=path_o, ) if not windows(): command += ' -fPIC\n' run_if( command, path_o, path_source, [path_source] + _get_prerequisites(prerequisites_path), prerequisites_compile, ) # Link object files to shared library. # prerequisites_path = f'{path_so}.d' command = linker_command( libpaths=libpaths, libs=libs, optimise=optimise, debug=debug, linker_extra=linker_extra, cpp=cpp, python=True, py_limited_api=py_limited_api, rpath=True, path_os=path_os, path_out=path_so, ) if windows(): base, _ = os.path.splitext(path_so_leaf) command += f' /DLL\n' command += f' /EXPORT:PyInit__{name}\n' command += f' /IMPLIB:{base}.lib # Overrides the default import library name.\n' else: command += f' -MD -MF {prerequisites_path}\n' command += f' -shared\n' link_was_run = run_if( command, path_so, path_cpp, *path_os, *_get_prerequisites(f'{path_so}.d'), prerequisites_link, ) if link_was_run and darwin(): macos_patch2(path_so, libpaths, libs) _extensions_to_py_limited_api[os.path.abspath(path_so)] = py_limited_api if return_paths: return f'{builddir}/{name}.py', path_so else: return path_so_leaf # Functions that might be useful. # def base_compiler(vs=None, pythonflags=None, cpp=False, use_env=True): ''' Returns basic compiler command and PythonFlags. Args: vs: Windows only. A `wdev.WindowsVS` instance or None to use default `wdev.WindowsVS` instance. pythonflags: A `pipcl.PythonFlags` instance or None to use default `pipcl.PythonFlags` instance. cpp: If true we return C++ compiler command instead of C. On Windows this has no effect - we always return `cl.exe`. use_env: If true we return '$CC' or '$CXX' if the corresponding environmental variable is set (without evaluating with `os.getenv()` or `os.environ`). Returns `(cc, pythonflags)`: cc: C or C++ command. On Windows this is of the form `{vs.vcvars}&&{vs.cl}`; otherwise it is typically `cc` or `c++`. pythonflags: The `pythonflags` arg or a new `pipcl.PythonFlags` instance. ''' if not pythonflags: pythonflags = PythonFlags() cc = None if use_env: if cpp: if os.environ.get( 'CXX'): cc = '$CXX' else: if os.environ.get( 'CC'): cc = '$CC' if cc: pass elif windows(): if not vs: vs = wdev.windows_vs() cc = f'"{vs.vcvars}"&&"{vs.cl}"' elif wasm(): cc = 'em++' if cpp else 'emcc' else: cc = 'c++' if cpp else 'cc' cc = macos_add_cross_flags( cc) return cc, pythonflags def base_linker(vs=None, pythonflags=None, cpp=False, use_env=True): ''' Returns basic linker command. Args: vs: Windows only. A `wdev.WindowsVS` instance or None to use default `wdev.WindowsVS` instance. pythonflags: A `pipcl.PythonFlags` instance or None to use default `pipcl.PythonFlags` instance. cpp: If true we return C++ linker command instead of C. On Windows this has no effect - we always return `link.exe`. use_env: If true we use `os.environ['LD']` if set. Returns `(linker, pythonflags)`: linker: Linker command. On Windows this is of the form `{vs.vcvars}&&{vs.link}`; otherwise it is typically `cc` or `c++`. pythonflags: The `pythonflags` arg or a new `pipcl.PythonFlags` instance. ''' if not pythonflags: pythonflags = PythonFlags() linker = None if use_env: if os.environ.get( 'LD'): linker = '$LD' if linker: pass elif windows(): if not vs: vs = wdev.windows_vs() linker = f'"{vs.vcvars}"&&"{vs.link}"' elif wasm(): linker = 'em++' if cpp else 'emcc' else: linker = 'c++' if cpp else 'cc' linker = macos_add_cross_flags( linker) return linker, pythonflags def any_cpp(source_paths): for source_path in source_paths: if os.path.splitext(source_path)[1] in ('.cpp', '.c++'): return True def compiler_command( *, includes=None, defines=None, optimise=True, debug=False, compiler_extra='', compiler_extra_cpp='', cpp=None, python=False, py_limited_api=False, rpath=False, source_paths=None, path_o=None, ): ''' Returns generic compiler and linker commands. Used by `build_extension()`. Args: includes: A string, or a sequence of extra include directories to be prefixed with `-I`. defines: A string, or a sequence of extra preprocessor defines to be prefixed with `-D`. optimise: Whether to use compiler optimisations and define NDEBUG. debug: Whether to build with debug symbols. compiler_extra: Extra compiler flags. Can be None. compiler_extra: Extra c++ compiler flags. Can be None. cpp: If true we tell SWIG to generate C++ code instead of C. If None we use suffix of items in . python: If true: * We include flags for using Python headers and libraries. * We predefine Py_GIL_DISABLED if sysconfig.get_config_var('Py_GIL_DISABLED')==1 (i.e. free-thread build). py_limited_api: If true we build for current Python's limited API / stable ABI. Note that we will assert false if this extension is added to a pipcl.Package that has a different , because on Windows importing a non-py_limited_api extension inside a py_limited=True package fails. ''' if compiler_extra is None: compiler_extra = '' if isinstance(source_paths, str): source_paths = source_paths, includes_text = _flags( includes, '-I') defines_text = _flags( defines, '-D') py_limited_api2 = current_py_limited_api() if py_limited_api else None if cpp is None: cpp = any_cpp(source_paths) compiler_command, pythonflags = base_compiler(cpp=cpp) # setuptools on Linux seems to use slightly different compile flags: # # -fwrapv -O3 -Wall -O2 -g0 -DPY_CALL_TRAMPOLINE # general_flags = '' if darwin(): general_flags += ' -undefined dynamic_lookup' if windows(): permissive = '/permissive-' EHsc = '/EHsc' T = '/Tp' if cpp else '/Tc' optimise2 = '/DNDEBUG /O2' if optimise else '/D_DEBUG' debug2 = '/Zi' if debug else '' py_limited_api3 = f'/DPy_LIMITED_API={py_limited_api2}' if py_limited_api2 else '' compiler_command = textwrap.dedent(f''' {compiler_command} # General: /c # Compiles without linking. {EHsc} # Enable "Standard C++ exception handling". #/MD # Creates a multithreaded DLL using MSVCRT.lib. {'/MDd' if debug else '/MD'} /Fo{path_o} # Output file. codespell:ignore # Include paths: {includes_text} {pythonflags.includes if python else ''} # Include path for Python headers. {'/D Py_GIL_DISABLED' if python and sysconfig.get_config_var('Py_GIL_DISABLED')==1 else ''} # Code generation: {optimise2} {debug2} {permissive} # Set standard-conformance mode. # Diagnostics: #/FC # Display full path of source code files passed to cl.exe in diagnostic text. /W3 # Sets which warning level to output. /W3 is IDE default. /diagnostics:caret # Controls the format of diagnostic messages. /nologo # {defines_text} {compiler_extra} {compiler_extra_cpp} {py_limited_api3} ''') for source_path in source_paths: compiler_command += f' {T}{source_path}\n' else: if debug: general_flags += ' -g' if optimise: general_flags += ' -O2 -DNDEBUG' py_limited_api3 = f'-DPy_LIMITED_API={py_limited_api2}' if py_limited_api2 else '' compiler_command = textwrap.dedent(f''' {compiler_command} {general_flags.strip()} {pythonflags.includes if python else ''} {'-D Py_GIL_DISABLED' if python and sysconfig.get_config_var('Py_GIL_DISABLED')==1 else ''} {includes_text} {defines_text} {compiler_extra} {compiler_extra_cpp if cpp else ''} {py_limited_api3} -c -o {path_o} ''') for source_path in source_paths: compiler_command += f' {source_path}\n' return compiler_command def linker_command( *, libpaths=None, libs=None, optimise=True, debug=False, linker_extra='', cpp=True, python=False, py_limited_api=False, rpath=False, path_os=None, path_out=None, ): ''' Returns generic compiler and linker commands. Used by `build_extension()`. Args: libpaths A string, or a sequence of library paths to be prefixed with `/LIBPATH:` on Windows or `-L` on Unix. libs A string, or a sequence of library names. Each item is prefixed with `-l` on non-Windows. optimise: Whether to use compiler optimisations and define NDEBUG. debug: Whether to build with debug symbols. linker_extra: Extra linker flags. Can be None. cpp: . python: If true we include flags for using Python headers and libraries. py_limited_api: If true we build for current Python's limited API / stable ABI. Note that we will assert false if this extension is added to a pipcl.Package that has a different , because on Windows importing a non-py_limited_api extension inside a py_limited=True package fails. rpath: If true we include an rpath arg to linker command, and patch up libraries on MacOS. ''' if linker_extra is None: linker_extra = '' libpaths_text = _flags( libpaths, '/LIBPATH:', '"') if windows() else _flags( libpaths, '-L') libs_text = _flags( libs, '' if windows() else '-l') py_limited_api2 = current_py_limited_api() if py_limited_api else None general_flags = '' if rpath: if windows(): pass elif darwin(): # MacOS's linker does not like `-z origin`. rpath_flag = "-Wl,-rpath,@loader_path/" # Avoid `Undefined symbols for ... "_PyArg_UnpackTuple" ...'. elif pyodide(): # Setting `-Wl,-rpath,'$ORIGIN',-z,origin` gives: # emcc: warning: ignoring unsupported linker flag: `-rpath` [-Wlinkflags] # wasm-ld: error: unknown -z value: origin # rpath_flag = "-Wl,-rpath,'$ORIGIN'" else: rpath_flag = "-Wl,-rpath,'$ORIGIN',-z,origin" if darwin(): general_flags += ' -undefined dynamic_lookup' linker_command, pythonflags = base_linker(cpp=cpp) if windows(): debug2 = '/Zi' if debug else '' py_limited_api3 = f'/DPy_LIMITED_API={py_limited_api2}' if py_limited_api2 else '' debug2 = '/DEBUG' if debug else '' linker_command = textwrap.dedent(f''' {linker_command} {libpaths_text} {pythonflags.ldflags if python else ''} {debug2} /nologo {libs_text} {linker_extra} {'/OUT:'+path_out if path_out else ''} ''') for path_o in path_os: linker_command += f' {path_o}\n' else: if debug: general_flags += ' -g' if optimise: general_flags += ' -O2 -DNDEBUG' py_limited_api3 = f'-DPy_LIMITED_API={py_limited_api2}' if py_limited_api2 else '' # Fun fact - on Linux, if the -L and -l options are before '{path_cpp}' # they seem to be ignored... # linker_command += f'\n' for path_o in path_os: linker_command += f' {path_o}\n' if path_out: linker_command += f' -o {path_out}\n' t = textwrap.dedent(f''' {general_flags.strip()} {libpaths_text} {libs_text} {linker_extra} {pythonflags.ldflags if python else ''} {rpath_flag} {py_limited_api3} ''') linker_command += textwrap.indent(t, ' ') return linker_command def macos_patch2(path_so, libpaths, libs): assert darwin() # We need to patch up references to shared libraries in `libs`. sublibraries = list() for lib in () if libs is None else libs: for libpath in libpaths: found = list() for suffix in '.so', '.dylib': path = f'{libpath}/lib{os.path.basename(lib)}{suffix}' if os.path.exists( path): found.append( path) if found: assert len(found) == 1, f'More than one file matches lib={lib!r}: {found}' sublibraries.append( found[0]) break else: log2(f'Warning: can not find path of lib={lib!r} in libpaths={libpaths}') macos_patch( path_so, *sublibraries) #run(f'ls -l {path_so}', check=0) #run(f'file {path_so}', check=0) def fs_ensure_empty_dir(path): ''' Ensures is an empty directory. ''' os.makedirs(path, exist_ok=True) fs_remove_dir_contents( path) def fs_ensure_dir(path): ''' Ensures directory exists. ''' os.makedirs(path, exist_ok=True) def fs_ensure_parent_dir(path): ''' Ensures parent directory of exists. ''' parent = os.path.dirname(path) if parent: fs_ensure_dir(parent) def fs_filesize( filename, default=0): ''' Returns size of or if error (e.g. doesn't exist). ''' try: return os.path.getsize( filename) except OSError: return default def fs_read(path, binary=False): ''' Returns string containing contents of file . Returns bytes if is true. ''' with open(path, 'rb' if binary else 'r') as f: return f.read() def fs_remove(path): ''' Removes file or directory, without raising exception if it doesn't exist. We assert-fail if the path still exists when we return, in case of permission problems etc. ''' try: os.remove(path) except Exception: pass shutil.rmtree(path, ignore_errors=1) assert not os.path.exists(path), f'Path still exists after deletion: {path!r}.' def fs_remove_dir_contents(path): ''' Removes all items in directory `path`; does not remove `path` itself. ''' for leaf in os.listdir(path): path2 = os.path.join(path, leaf) fs_remove(path2) def fs_write(path, data, binary=False): ''' Writes to file . ''' with open(path, 'wb' if binary else 'w') as f: return f.write( data) def git_info( directory): ''' Returns `(sha, comment, diff, branch)`, all items are str or None if not available. is the first line of the commit message. directory: Root of git checkout. ''' sha, comment, diff, branch = None, None, None, None def run_git(command): return run(command, capture=1, check=0, verbose=0, env_extra=dict(PAGER='')) # We are careful to run git with `--no-pager` to avoid problems on Windows where # Defender appears to cause problems. See: # https://www.reddit.com/r/sysadmin/comments/1msxc9o/git_commands_freezing_for_210_seconds_on_windows/ # e, out = run_git(f'cd {directory} && git --no-pager log -1 --pretty=oneline') if not e: line, _ = out.split('\n', 1) sha, comment = line.split(' ', 1) if 0 and platform.system() == 'Windows': # pylint: disable=condition-evals-to-constant # Have seen `git diff` sometimes hang when using a rsync'd # checkout. But it seems to work ok if we first diff a single file. #log(f'Doing dummy `git diff` of a single file, to avoid potential hang in later `git diff` on Windows.') e, text = run_git(f'cd {directory} && git ls-files') assert not e, f'git ls-files failed: {e=}' path0 = text.split('\n', 1)[0].strip() run_git(f'cd {directory} && git diff {path0}') e, out = run_git(f'cd {directory} && git --no-pager diff') if not e: diff = out e, out = run_git(f'cd {directory} && git --no-pager rev-parse --abbrev-ref HEAD') if not e: branch = out.strip() #log(f'git_info(): {directory=} {branch=} {sha=} {comment=}') return sha, comment, diff, branch def git_info_py( path, *, check = True, name_branch = 'branch', name_comment = 'comment', name_diff = 'diff', name_diff_n = 'diff_n', name_sha = 'sha', prefix = 'git_', ): ''' Returns python code that defines variables that contain information about a git checkout, typically the checkout that is being used to build a package. Args: path: Path of git checkout. check: If true we raise if we fail to get git info (e.g. is not a Git checkout), otherwise the returned text sets all variables to None. name_branch: Name of variable containing git branch, or false to omit. name_comment: Name of variable containing git comment, or false to omit. name_diff: Name of variable containing output of git diff, or false to omit. name_diff_n: Name of variable containing length of output of git diff. name_sha: Name of variable containing git sha, or false to omit. prefix: Prefix of each variable name. Example output: git_branch = 'main' comment = 'Add wibble' git_diff = '' git_diff_n = 0 git_sha = 'aa00d79773e1e3c4176b59ac76bfef89830bd289' ''' sha, comment, diff, branch = git_info(path) if None in (sha, comment, diff, branch) and check: raise Exception(f'Unable to get git info for: {path}') if isinstance(diff, int): diff_len = len(diff) else: diff_len = None ret = '' if name_branch: ret += f'{prefix}{name_branch} = {branch!r}\n' if name_comment: ret += f'{prefix}{name_comment} = {comment!r}\n' if name_diff: ret += f'{prefix}{name_diff} = {diff!r}\n' if name_sha: ret += f'{prefix}{name_diff_n} = {diff_len!r}\n' if name_sha: ret += f'{prefix}{name_sha} = {sha!r}\n' return ret def get_time_iso_8601(text): ''' Converts ISO-8601 text to Unix time. ''' t = time.strptime(text, '%Y-%m-%dT%H:%M:%S%z') t = calendar.timegm(t) return t def git_info_author_date(directory): ''' Returns author date (as a Unix time) of HEAD. ''' text = run(f'cd {directory} && git show -s --format=%aI HEAD', capture=1) return get_time_iso_8601(text.strip()) def git_info_committer_date(directory): ''' Returns committer date (as a Unix time) of HEAD. ''' text = run(f'cd {directory} && git show -s --format=%cI HEAD', capture=1) return get_time_iso_8601(text.strip()) def git_items( directory, submodules=False): ''' Returns list of paths for all files known to git within a `directory`. Args: directory: Must be somewhere within a git checkout. submodules: If true we also include git submodules. Returns: A list of paths for all files known to git within `directory`. Each path is relative to `directory`. `directory` must be somewhere within a git checkout. We run a `git ls-files` command internally. This function can be useful for the `fn_sdist()` callback. ''' command = 'git ls-files' cwd = directory.replace('/', os.sep) if submodules: command += ' --recurse-submodules' log1(f'Running with {cwd=}: {command=}.') text = subprocess.check_output(command, shell=True, cwd=cwd) ret = [] for path in text.decode('utf8').strip().split('\n'): path2 = os.path.join(directory, path) # Sometimes git ls-files seems to list empty/non-existent directories # within submodules. # if not os.path.exists(path2): log2(f'Ignoring git ls-files item that does not exist: {path2}') elif os.path.isdir(path2): log2(f'Ignoring git ls-files item that is actually a directory: {path2}') else: ret.append(path) return ret def fs_write_key(path, data): ''' Writes to , ensuring that is created with appropriate permissions. ''' fs_remove(path) if platform.system() == 'Windows': # For unknown reasons, the code below for non-Windows does not work # on Windows. # # Also for unknown reasons, using backslashes in # path doesn't work - we write the file but it # doesn't appear in the filesystem. path = path.replace('\\', '/') with open(path, 'wb') as f: f.write(data.encode('utf8').replace(b'\r', b'')) else: # Need to create file as read/write for current user only, so we have # to use `os.open()` instead of `open()`. # fd = os.open(path, os.O_WRONLY|os.O_CREAT|os.O_TRUNC|os.O_EXCL, 0o600) try: os.write(fd, data.encode('utf8')) finally: os.close(fd) def git_get_parse_text( text, *, branch = None, depth = None, remote = None, sha = None, tag = None, ): ''' Parses text following 'git:' for git_get(). ''' assert text.startswith('git:') args = iter(shlex.split(text[len('git:'):])) while 1: try: arg = next(args) except StopIteration: break if arg in ('-b', '--branch'): branch = next(args) tag = None sha = None elif arg in ('-t', '--tag'): tag = next(args) branch = None sha = None elif arg in ('-s', '--sha'): sha = next(args) branch = None tag = None elif arg == '--depth': depth = int(next(args)) elif arg.startswith('-'): assert 0, f'Unrecognised {arg=} in {text=}.' else: remote = arg return branch, depth, remote, sha, tag def git_get( local, *, remote=None, branch=None, tag=None, sha=None, text=None, depth=1, env_extra=None, update=True, submodules=True, devel=1, key=None, keyfile=None, clone_extra=None, ): ''' Creates/updates local checkout of remote repository and returns absolute path of . If is set but does not start with 'git:', it is assumed to be an up to date local checkout, and we return absolute path of without doing any git operations. Args: local: Local directory. Created and/or updated using `git clone` and `git fetch` etc. remote: Remote git repostitory, for example 'https://github.com/ArtifexSoftware/mupdf.git'. Can be overridden by . branch: Branch to use; can be overridden by . tag: Tag to use; can be overridden by . sha: Use this sha; must be the full length sha, e.g. `5ea4449c3edba8840d5c27e15095536cbfc7c1f` text: If None or empty: Ignored. If starts with 'git:': The remaining text should be a command-line style string containing some or all of these args: -b|--branch -t|-tag --depth These overrides , , and . Otherwise: is assumed to be a local directory, and we simply return it as an absolute path without doing any git operations. For example these all clone/update/branch master of https://foo.bar/qwerty.git to local checkout 'foo-local': git_get('foo-local', remote='https://foo.bar/qwerty.git', branch='master') git_get('foo-local', text='git:--branch master https://foo.bar/qwerty.git') git_get('foo-local', text='git:--branch master', remote='https://foo.bar/qwerty.git') git_get('foo-local', text='git:', branch='master', remote='https://foo.bar/qwerty.git') depth: Depth of local checkout when cloning and fetching, or None. env_extra: Dict of extra name=value environment variables to use whenever we run git. update: If false we do not update existing repository. Might be useful if testing without network access. submodules: If true, we clone with `--recursive --shallow-submodules` and run `git submodule update --init --recursive` before returning. key: Ssh key to use. keyfile: Ssh key file to use. Only used if is None. clone_extra: If true, is added to any `git clone` command. For example `clone_extra='--config core.autocrlf=input'` to avoid converting text files to Windows line endings. ''' log(f'{os.getcwd()=}') log(f'{remote=}') log(f'{local=}') log(f'{branch=}') log(f'{tag=}') log(f'{text=}') log(f'{env_extra=}') log(f'{update=}') log(f'{submodules=}') log(f'{devel=}') log(f'{bool(key)=}') log(f'{keyfile=}') if text: if text.startswith('git:'): branch, depth, remote, sha, tag = git_get_parse_text( text, branch=branch, depth=depth, remote=remote, sha=sha, tag=tag, ) assert remote, f' unset and no remote specified in {text=}.' assert branch or tag or sha, f' and unset and no branch/tag specified in {text=}.' else: log0(f'Using local directory {text!r}.') return os.path.abspath(text) assert bool(branch) ^ bool(tag) ^ bool(sha), \ f'Must specify exactly one of , or ; {branch=} {tag=} {sha=}.' depth_arg = f' --depth {depth}' if depth else '' paths_to_delete = list() if key: keyfile = os.path.abspath('pipcl-tmp-git-key') paths_to_delete.append(keyfile) fs_write_key(keyfile, key) if keyfile: GIT_SSH_COMMAND = f'ssh -i {os.path.abspath(keyfile)} -o StrictHostKeyChecking=no' GIT_SSH_COMMAND = GIT_SSH_COMMAND.replace('\\', '/') # Required on windows. log(f'{keyfile=}') log(f'{GIT_SSH_COMMAND=}') env_extra = (env_extra or dict()) | dict(GIT_SSH_COMMAND=GIT_SSH_COMMAND) if ':' not in remote: # Allow clone/fetch from local directory. remote = os.path.abspath(remote) try: # Convert local remotes to absolute path so it still works after we've # cd-ed into local directory. # if ':' not in remote: remote_0 = remote remote = os.path.abspath(remote) log(f'Changed local remote {remote_0!r} to {remote=}') def do_update(): # This seems to pull in the entire repository. log0(f'do_update(): attempting to update {local=}.') # Remove any local changes. run( f'cd {local} && git reset --hard', env_extra=env_extra, prefix='git reset: ', ) if tag: # `-u` avoids `fatal: Refusing to fetch into current branch`. # Using '+' and `revs/tags/` prefix seems to avoid errors like: # error: cannot update ref 'refs/heads/v3.16.44': # trying to write non-commit object # 06c4ae5fe39a03b37a25a8b95214d9f8f8a867b8 to branch # 'refs/heads/v3.16.44' # run( f'cd {local} && git fetch -fuv{depth_arg} {remote} +refs/tags/{tag}:refs/tags/{tag}', env_extra=env_extra, prefix='git fetch: ', ) run( f'cd {local} && git checkout -f {tag}', env_extra=env_extra, prefix='git checkout: ', ) if branch: # `-u` avoids `fatal: Refusing to fetch into current branch`. run( f'cd {local} && git fetch -fuv{depth_arg} {remote} {branch}:{branch}', env_extra=env_extra, prefix='git fetch: ', ) run( f'cd {local} && git checkout -f {branch}', env_extra=env_extra, prefix='git checkout: ', ) if sha: # `-u` avoids `fatal: Refusing to fetch into current branch`. run( f'cd {local} && git fetch -fuv{depth_arg} {remote} {sha}', env_extra=env_extra, prefix='git fetch: ', ) run( f'cd {local} && git checkout -f {sha}', env_extra=env_extra, prefix='git checkout: ', ) do_clone = True if os.path.isdir(f'{local}/.git'): if update: # Try to update existing checkout. try: do_update() do_clone = False except Exception as e: log0(f'Failed to update existing checkout {local}: {e}') else: do_clone = False if do_clone: # No existing git checkout, so do a fresh clone. #_fs_remove(local) log0(f'{os.getcwd()=}') log0(f'Cloning to: {os.path.abspath(local)}') command = f'git clone --config core.longpaths=true{depth_arg}' if clone_extra: command += f' {clone_extra}' if submodules: command += f' --recursive --shallow-submodules' if branch: command += f' -b {branch}' #if tag: # command += f' -t {tag}' command += f' {remote} {local}' run( command, env_extra=env_extra, prefix='git clone: ', ) do_update() if submodules: run( f'cd {local} && git submodule update --init --recursive', env_extra=env_extra, prefix='git submodule update: ', ) # Show sha of checkout. run( f'cd {local} && git --no-pager log -1 --pretty=oneline', check=False, prefix='git show: ', ) if 0: # Some extra checking that what we're doing is correct. if tag: diff = run(f'cd {local} && git --no-pager diff {tag}', capture=1) if branch: diff = run(f'cd {local} && git --no-pager diff {branch}', capture=1) if sha: diff = run(f'cd {local} && git --no-pager diff {sha}', capture=1) log(f'{diff=}') diff = diff.strip() assert not diff, f'Diff is not empty:\n{textwrap.indent(diff, " ")}' return os.path.abspath(local) finally: for path in paths_to_delete: fs_remove(path) def run( command, *, capture=False, check=1, cwd=None, verbose=1, env=None, env_extra=None, timeout=None, caller=1, prefix=None, encoding=None, # System default. errors='backslashreplace', ticker=0, out=None, tee=None, log=None, ): ''' Runs a command using `subprocess.run()`. Args: command: A string command to run or list of string args to run. If a string: Multiple lines in `command` are treated as a single command. * If a line starts with `#` it is discarded. * If a line contains ` #`, the trailing text is discarded. When running the command on Windows, newlines are replaced by spaces; otherwise each line is terminated by a backslash character. capture: If true, we include the command's output in our return value. check: If true we raise an exception on error; otherwise we include the command's returncode in our return value. cwd: If not None, the directory in which to run the command. verbose: If true we show the command. env: None or dict to use instead of . env_extra: None or dict to add to or . timeout: If not None, timeout in seconds; passed directly to `subprocess.run()`. Note that on MacOS `subprocess.run()` seems to leave processes running if timeout expires. prefix: String prefix for each line of output. If true: * We run command with stdout=subprocess.PIPE and stderr=subprocess.STDOUT, repetaedly reading the command's output and writing it with . * We do not support , which must be None. encoding: errors: Encoding of child process output. ticker: If non-zero, we show a rotating bar when waiting for child process output, with seconds between each rotation. out: Where to write output. Single item or list/tuple of items. Each item can be a callable or an object with .write() and .flush() members. Special value 'log' writes to `pipcl.log()`. Default is 'log', or None if is true. tee: Name of file to also write to. We refuse to overwrite existing file. log: If true we write to `pipcl.log()`. This is the default. On Windows: is not supported and ignored. and both being true is not supported, and will raise an exception. These restrictions are because the Python module `selectors` does not handle pipes. Returns: check capture Return -------------------------- false false returncode false true (returncode, output) true false None or raise exception true true output or raise exception ''' if env is None: env = os.environ if env_extra: env = env.copy() if env_extra: env.update(env_extra) lines = _command_lines( command) if verbose: text = f'Running:' nl = '\n ' text += f' {nl.join(lines)}\n' if env_extra: text += f'with:\n' for k in sorted(env_extra.keys()): text += f' {k}={shlex.quote(env_extra[k])}\n' if cwd: text += f'with: {cwd=}\n' log1(text, caller=caller+1) if isinstance(command, str): sep = ' ' if windows() else ' \\\n' command2 = sep.join( lines) else: command2 = command if platform.system() == 'Windows': # The `selectors` module does not support pipes. ticker = False cleanup = list() try: if out: if isinstance(out, tuple): out = list(out) elif not isinstance(out, list): out = [out] else: out = out.copy() else: out = list() if log: out.append('log') if not out and not capture: out = ['log'] if tee: assert not os.path.exists(tee), f'Tee file already exists: {tee}' tee_f = open(tee, 'w') # pylint: disable=consider-using-with cleanup.append(tee_f.close) out.append(tee_f) # Convert `out` into list of (write, flush) callables. for i, o in enumerate(out): if callable(o): out[i] = (o, lambda: None) elif o == 'log': write2 = lambda text: _log(text, level=0, caller=caller+3, raw=True) out[i] = (write2, None) def log_cleanup(): if not _log_text_line_start: write2('\n') cleanup.append(log_cleanup) else: out[i] = (o.write, o.flush) def write(text): for write2, _flush2 in out: write2(text) def flush(): for _write2, flush2 in out: if flush2: flush2() # pylint: disable=not-callable # pylint: disable=too-many-nested-blocks if prefix or ticker or out!=[sys.stdout]: # Use explicit read loop from child process. if platform.system() == 'Windows': # The `selectors` module does not support pipes. assert not timeout, 'timeout with (prefix or out) not supported on Windows' child = subprocess.Popen( # pylint: disable=consider-using-with command2, cwd=cwd, shell=isinstance(command2, str), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, encoding=encoding, errors=errors, env=env, ) if capture: capture_text = '' decoder = codecs.getincrementaldecoder(child.stdout.encoding)(errors) line_start = True timed_out = False waiter = _Wait(child.stdout.fileno(), ticker, timeout) cleanup.append(waiter.close) is_windows = windows() while 1: if waiter.wait(): child.terminate() timed_out = True break raw = os.read( child.stdout.fileno(), 10000) text = decoder.decode(raw, final=not raw) if is_windows: # Presence of \r bytes on windows causes them to # proliferate in nested runs, leading to logs being # difficult to read. So we remove them explicitly. (Our # subprocess.Popen instance will probably have text mode # stdout/stderr, but we read from child.stdout.fileno() # which will always be binary.) # text = text.replace('\r\n', '\n') if text: waiter.cancel_ticker() if capture: capture_text += text lines = text.split('\n') for i, line in enumerate(lines): if line_start: if prefix: write(prefix) line_start = False write(line) if i < len(lines) - 1: write('\n') line_start = True #else: # line_start = not line flush() if not raw: break waiter.cancel_ticker() if not line_start: write('\n') e = child.wait() if timed_out and check: raise subprocess.TimeoutExpired(e, command2, capture_text if capture else None) if check and e: raise subprocess.CalledProcessError(e, command2, capture_text if capture else None) if check: return capture_text if capture else None else: return (e, capture_text) if capture else e else: # Simple case - we can simply call subprocess.run(). cp = subprocess.run( command2, cwd=cwd, shell=True, stdout=subprocess.PIPE if capture else None, stderr=subprocess.STDOUT if capture else None, check=check, encoding=encoding, errors=errors, env=env, timeout=timeout, ) if check: return cp.stdout if capture else None else: return (cp.returncode, cp.stdout) if capture else cp.returncode finally: for fn in cleanup: fn() def darwin(): ''' Returns true of we are running on MacOS. ''' return sys.platform.startswith( 'darwin') def windows(): ''' Returns true of we are running on Windows. ''' return platform.system() == 'Windows' def wasm(): ''' Returns true of we are running on Wasm. ''' return os.environ.get( 'OS') in ('wasm', 'wasm-mt') def pyodide(): ''' Returns true of we are running on Pyodide. ''' return os.environ.get( 'PYODIDE') == '1' def linux(): ''' Returns true of we are running on Linux. ''' return platform.system() == 'Linux' def openbsd(): ''' Returns true of we are running on OpenBSD. ''' return platform.system() == 'OpenBSD' def show_system(): ''' Show useful information about the system plus argv and environ. Omits os.environ if $PIPCL_SHOW_ENV is '0'. ''' def log(text): log0(text, caller=3) #log(f'{__file__=}') #log(f'{__name__=}') log(f'{os.getcwd()=}') log(f'{platform.machine()=}') log(f'{platform.platform()=}') log(f'{platform.python_implementation()=}') log(f'{platform.python_version()=}') log(f'{platform.system()=}') if sys.implementation.name != 'graalpy': log(f'{platform.uname()=}') log(f'{sys.executable=}') log(f'{sys.version=}') log(f'{sys.version_info=}') log(f'{list(sys.version_info)=}') log(f'{sys.prefix=}') log(f'{sys.base_prefix=}') log(f'{sysconfig.get_config_var("Py_GIL_DISABLED")=}') try: log(f'{sys._is_gil_enabled()=}') # # pylint: disable=protected-access except AttributeError as e: log(f'sys._is_gil_enabled() => exception {e}') log(f'CPU bits: {cpu_bits()}') log(f'sys.argv ({len(sys.argv)}):') for i, arg in enumerate(sys.argv): log(f' {i}: {arg!r}') PIPCL_SHOW_ENV = os.environ.get('PIPCL_SHOW_ENV') if PIPCL_SHOW_ENV == '0': log(f'[Not showing os.environ because {PIPCL_SHOW_ENV=}.]') else: log(f'os.environ ({len(os.environ)}):') for k in sorted( os.environ.keys()): v = os.environ[ k] if 'BEGIN OPENSSH PRIVATE KEY' in v: # Don't show private keys. log(f' {k} ****') else: log( f' {k}: {v!r}') class PythonFlags: ''' Compile/link flags for the current python, for example the include path needed to get `Python.h`. The 'PIPCL_PYTHON_CONFIG' environment variable allows to override the location of the python-config executable. Members: .includes: String containing compiler flags for include paths. .ldflags: String containing linker flags for library paths. ''' def __init__(self): # Experimental detection of python flags from sysconfig.*() instead of # python-config command. includes_, ldflags_ = sysconfig_python_flags() if pyodide(): _include_dir = os.environ[ 'PYO3_CROSS_INCLUDE_DIR'] _lib_dir = os.environ[ 'PYO3_CROSS_LIB_DIR'] self.includes = f'-I {_include_dir}' self.ldflags = f'-L {_lib_dir}' elif 0: self.includes = includes_ self.ldflags = ldflags_ elif windows(): wp = wdev.WindowsPython() self.includes = f'/I"{wp.include}"' self.ldflags = f'/LIBPATH:"{wp.libs}"' elif pyodide(): _include_dir = os.environ[ 'PYO3_CROSS_INCLUDE_DIR'] _lib_dir = os.environ[ 'PYO3_CROSS_LIB_DIR'] self.includes = f'-I {_include_dir}' self.ldflags = f'-L {_lib_dir}' else: python_config = os.environ.get("PIPCL_PYTHON_CONFIG") if not python_config: # We use python-config which appears to work better than pkg-config # because it copes with multiple installed python's, e.g. # manylinux_2014's /opt/python/cp*-cp*/bin/python*. # # But... on non-macos it seems that we should not attempt to specify # libpython on the link command. The manylinux docker containers # don't actually contain libpython.so, and it seems that this # deliberate. And the link command runs ok. # python_exe = os.path.realpath( sys.executable) if darwin(): # Basic install of dev tools with `xcode-select --install` doesn't # seem to provide a `python3-config` or similar, but there is a # `python-config.py` accessible via sysconfig. # # We try different possibilities and use the last one that # works. # python_config = None for pc in ( f'python3-config', f'{sys.executable} {sysconfig.get_config_var("srcdir")}/python-config.py', f'{python_exe}-config', ): e = subprocess.run( f'{pc} --includes', shell=1, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=0, ).returncode log2(f'{e=} from {pc!r}.') if e == 0: python_config = pc assert python_config, f'Cannot find python-config' else: python_config = f'{python_exe}-config' log2(f'Using {python_config=}.') try: self.includes = run( f'{python_config} --includes', capture=1, verbose=0).strip() except Exception as e: raise Exception('We require python development tools to be installed.') from e self.ldflags = run( f'{python_config} --ldflags', capture=1, verbose=0).strip() if linux(): # It seems that with python-3.10 on Linux, we can get an # incorrect -lcrypt flag that on some systems (e.g. WSL) # causes: # # ImportError: libcrypt.so.2: cannot open shared object file: No such file or directory # ldflags2 = self.ldflags.replace(' -lcrypt ', ' ') if ldflags2 != self.ldflags: log2(f'### Have removed `-lcrypt` from ldflags: {self.ldflags!r} -> {ldflags2!r}') self.ldflags = ldflags2 if 0: log1(f'{self.includes=}') log1(f' {includes_=}') log1(f'{self.ldflags=}') log1(f' {ldflags_=}') def macos_add_cross_flags(command): ''' If running on MacOS and environment variables ARCHFLAGS is set (indicating we are cross-building, e.g. for arm64), returns `command` with extra flags appended. Otherwise returns unchanged `command`. ''' if darwin(): archflags = os.environ.get( 'ARCHFLAGS') if archflags: command = f'{command} {archflags}' log2(f'Appending ARCHFLAGS to command: {command}') return command return command def macos_patch( library, *sublibraries): ''' If running on MacOS, patches `library` so that all references to items in `sublibraries` are changed to `@rpath/{leafname}`. Does nothing on other platforms. library: Path of shared library. sublibraries: List of paths of shared libraries; these have typically been specified with `-l` when `library` was created. ''' log2( f'macos_patch(): library={library} sublibraries={sublibraries}') if not darwin(): return if not sublibraries: return subprocess.run( f'otool -L {library}', shell=1, check=1) command = 'install_name_tool' for sublibrary in sublibraries: name = subprocess.run( f'otool -D {sublibrary}', shell=1, check=1, capture_output=1, encoding='utf8', ).stdout.strip() name = name.split('\n') assert len(name) == 2 and name[0] == f'{sublibrary}:', f'{name=}' name = name[1] # strip trailing so_name. leaf = os.path.basename(name) m = re.match('^(.+[.]((so)|(dylib)))[0-9.]*$', leaf) assert m log2(f'Changing {leaf=} to {m.group(1)}') leaf = m.group(1) command += f' -change {name} @rpath/{leaf}' command += f' {library}' log2( f'Running: {command}') subprocess.run( command, shell=1, check=1) subprocess.run( f'otool -L {library}', shell=1, check=1) def _macos_fixup_platform_tag(tag): ''' Patch up platform tag on MacOS. E.g. `foo-1.2.3-cp311-none-macosx_13_x86_64.whl` causes `pip` to fail with: `not a supported wheel on this platform`. We seem to need to add `_0` to the OS version. (This is documented at https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#macos). And with graal we need to replace trailing `universal2` with x86_64 or arm64. On non-graal this causes problems because non-universal platform tags seem more restricted than platform tags from `sysconfig.get_platform()`. For example: pip install ...-macosx_10_13_arm64.whl ERROR: ...-macosx_10_13_arm64.whl is not a supported wheel on this platform. pip install ...-macosx_10_13_universal2.whl Ok. ''' m = re.match( '^macosx_([0-9_]+)_([^0-9].+)$', tag) if not m: return tag a = m.group(1) if '_' not in a: a += '_0' b = m.group(2) if sys.implementation.name == 'graalpy' and b == 'universal2': # Replace 'universal2' with x86_64 or arm64. b = platform.machine() ret = f'macosx_{a}_{b}' #log0(f'Changing from {tag=} to {ret=}.') return ret # Internal helpers. # class _Wait: def __init__(self, fd, ticker, timeout): ''' Helper when reading output from child process. fd: child's output. ticker: ticker period, or false for no ticker. timeout: . ''' self.fd = fd self.ticker = ticker if ticker: self.ticker_state = _TickerState() self.endtime = time.time() + timeout if timeout else None self.ticker_start = None self.selector = selectors.DefaultSelector() self.selector.register(fd, selectors.EVENT_READ) def wait(self): ''' Waits for output or timeout or ticker-related timeout. Updates ticker as required. Returns true if original timeout exceeded, else None. After we return, the caller should call .cancel_ticker() if it reads non-empty output. ''' if not self.endtime and not self.ticker: return while 1: # We need to use a timeout. t = time.time() if self.ticker: if self.ticker_start is None: self.ticker_start = t assert t >= self.ticker_start endtime_ticker = self.ticker_start + ((t - self.ticker_start) // self.ticker + 1) * self.ticker assert endtime_ticker > self.ticker_start assert endtime_ticker > t assert endtime_ticker <= t + self.ticker if self.ticker and self.endtime: endtime2 = min(self.endtime, endtime_ticker) # pylint: disable=possibly-used-before-assignment elif self.ticker: endtime2 = endtime_ticker else: endtime2 = self.endtime assert endtime2 > t, f'{t=} {endtime2=}' events = self.selector.select(endtime2 - t) if events: # Data is available. return else: # timeout t2 = time.time() if self.endtime and t2 >= self.endtime: return True if self.ticker and t2 >= endtime_ticker: self.ticker_state.increment(self.ticker_start, endtime_ticker) continue def cancel_ticker(self): self.ticker_start = None if self.ticker: self.ticker_state.cancel() def close(self): self.cancel_ticker() self.selector.close() class _TickerState: ''' Helper for ticker in `pipcl.run()`. ''' def __init__(self): self.current_length = 0 self.rotate_count = 0 self.time_t0 = None def increment(self, t1, t2): ''' Update the ticker display. t1: Start time; only used if we were not already showing ticker. t2: Current time to show. ''' self.rotate_count += 1 rotate_c = '\\|/-'[self.rotate_count % 4] if self.time_t0 is None: self.time_t0 = t1 dt = _duration(t2 - self.time_t0) dt = f'[{rotate_c} {dt}]' tail_len = max(0, self.current_length - len(dt)) tail = ' ' * tail_len + '\b' * tail_len sys.stdout.write('\b' * self.current_length + dt + tail) self.current_length = len(dt) sys.stdout.flush() def cancel(self): if self.rotate_count: sys.stdout.write('\b \b' * self.current_length) sys.stdout.flush() self.current_length = 0 self.rotate_count = 0 self.time_t0 = None def _command_lines( command): ''' Process multiline command by running through `textwrap.dedent()`, removes comments (lines starting with `#` or ` #` until end of line), removes entirely blank lines. Returns list of lines. ''' if isinstance(command, str): command = textwrap.dedent( command) lines = [] for line in command.split( '\n'): if line.startswith( '#'): h = 0 else: h = line.find( ' #') if h >= 0: line = line[:h] if line.strip(): lines.append(line.rstrip()) else: return ['['] + command + [']'] return lines def cpu_bits(): ''' Returns CPU wordsize in bits. ''' return int.bit_length(sys.maxsize+1) def _cpu_name(): ''' Returns `x32` or `x64` depending on Python build. ''' #log(f'sys.maxsize={hex(sys.maxsize)}') return f'x{32 if sys.maxsize == 2**31 - 1 else 64}' def run_if( command, out, *prerequisites, caller=1): ''' Runs a command only if the output file is not up to date. Args: command: The command to run. We write this and a hash of argv[0] into a file .cmd so that we know to run a command if the command itself has changed. out: Path of the output file. prerequisites: List of prerequisite paths or true/false/None items. If an item is None it is ignored, otherwise if an item is not a string we immediately return it cast to a bool. We recurse into directories, effectively using the newest file in the directory. Returns: True if we ran the command, otherwise None. If the output file does not exist, the command is run: >>> del _log_f[:] >>> _log_f.append(sys.stdout) >>> log_prefix('%f(): ') >>> out = 'tests/run_if_test_out' >>> fs_remove(out) >>> fs_remove( f'{out}.cmd') >>> run_if( f'touch {out}', out, caller=0) run_if(): Running command because: File does not exist: 'tests/run_if_test_out' run_if(): Running: touch tests/run_if_test_out True If we repeat, the output file will be up to date so the command is not run: >>> run_if( f'touch {out}', out, caller=0) run_if(): Not running command because up to date: 'tests/run_if_test_out' If we change the command, the command is run: >>> run_if( f'touch {out};', out, caller=0) run_if(): Running command because: Command has changed: run_if(): @@ -1,2 +1,2 @@ run_if(): touch run_if(): -tests/run_if_test_out run_if(): +tests/run_if_test_out; run_if(): Running: touch tests/run_if_test_out; True If we add a prerequisite that is newer than the output, the command is run: >>> time.sleep(1) >>> prerequisite = 'tests/run_if_test_prerequisite' >>> run( f'touch {prerequisite}', caller=0) run(): Running: touch tests/run_if_test_prerequisite >>> run_if( f'touch {out}', out, prerequisite, caller=0) run_if(): Running command because: Command has changed: run_if(): @@ -1,2 +1,2 @@ run_if(): touch run_if(): -tests/run_if_test_out; run_if(): +tests/run_if_test_out run_if(): Running: touch tests/run_if_test_out True If we repeat, the output will be newer than the prerequisite, so the command is not run: >>> run_if( f'touch {out}', out, prerequisite, caller=0) run_if(): Not running command because up to date: 'tests/run_if_test_out' Also see OS-specific doctest examples for `_run_if_test_scripting_windows()` and `_run_if_test_scripting_unix()`. ''' doit = False # Path of file containing pickle data for command and hash of command's # first arg. cmd_path = f'{out}.cmd' def hash_get(path): try: with open(path, 'rb') as f: return hashlib.md5(f.read()).hexdigest() except Exception: #log(f'Failed to get hash of {path=}: {e}') return None lines = _command_lines( command) command2 = ' '.join(lines) command_args = shlex.split(command2 or '') command_arg0_path = fs_find_in_paths(command_args[0]) command_arg0_hash = hash_get(command_arg0_path) cmd_args, cmd_arg0_hash = (None, None) if os.path.isfile(cmd_path): with open(cmd_path, 'rb') as f: try: cmd_args, cmd_arg0_hash = pickle.load(f) except Exception: pass if not doit: # Set doit if outfile does not exist. out_mtime = _fs_mtime( out) if out_mtime == 0: doit = f'File does not exist: {out!r}' if not doit: # Set doit if command has changed. if command_args != cmd_args: if cmd_args is None: doit = 'No previous command stored' else: doit = f'Command has changed' # Show diff based on commands split into pseudo lines by # shlex.split(). doit += ':\n' lines = difflib.unified_diff( cmd_args, command_args, lineterm='', ) # Skip initial lines. assert next(lines) == '--- ' assert next(lines) == '+++ ' for line in lines: doit += f' {line}\n' if not doit: # Set doit if argv[0] hash has changed. #print(f'{cmd_arg0_hash=} {command_arg0_hash=}', file=sys.stderr) if command_arg0_hash != cmd_arg0_hash: doit = f'arg0 hash has changed.' #doit = f'arg0 hash has changed from {cmd_arg0_hash=} to {command_arg0_hash=}..' if not doit: # See whether any prerequisites are newer than target. def _make_prerequisites(p): if isinstance( p, (list, tuple)): return list(p) else: return [p] prerequisites_all = list() for p in prerequisites: prerequisites_all += _make_prerequisites( p) if 0: log2( 'prerequisites_all:', caller=caller+1) for i in prerequisites_all: log2( f' {i!r}', caller=caller+1) pre_mtime = 0 pre_path = None for prerequisite in prerequisites_all: if isinstance( prerequisite, str): mtime = _fs_mtime_newest( prerequisite) if mtime >= pre_mtime: pre_mtime = mtime pre_path = prerequisite elif prerequisite is None: pass elif prerequisite: doit = str(prerequisite) break if not doit: if pre_mtime > out_mtime: doit = f'Prerequisite is new: {os.path.abspath(pre_path)!r}' if doit: # Remove `cmd_path` before we run the command, so any failure # will force rerun next time. # fs_remove(cmd_path) log1( f'Running command because: {doit}', caller=caller+1) run( command, caller=caller+1) # Write the command we ran, into `cmd_path`. with open(cmd_path, 'wb') as f: pickle.dump((command_args, command_arg0_hash), f) return True else: log1( f'Not running command because up to date: {out!r}', caller=caller+1) if windows(): def _run_if_test_scripting_windows(): ''' Dummy function to allow Windows-only doctests for pipcl.run_if(). We detect changes to the contents of argv[0]: Create a shell script and run it: >>> del _log_f[:] >>> _log_f.append(sys.stdout) >>> log_prefix('%f(): ') >>> for path in glob.glob('tests/run_if_test_argv0.*'): ... fs_remove(path) >>> with open('tests/run_if_test_argv0.bat', 'w') as f: ... print('@echo off', file=f) ... print('echo hello world > tests/run_if_test_argv0.out', file=f) >>> import stat >>> os.chmod('tests/run_if_test_argv0.bat', os.stat('tests/run_if_test_argv0.bat').st_mode | stat.S_IEXEC) >>> run_if('"tests/run_if_test_argv0.bat"', f'tests/run_if_test_argv0.out', caller=0) run_if(): Running command because: File does not exist: 'tests/run_if_test_argv0.out' run_if(): Running: "tests/run_if_test_argv0.bat" True Running it a second time does nothing: >>> run_if(f'"tests/run_if_test_argv0.bat"', f'tests/run_if_test_argv0.out', caller=0) run_if(): Not running command because up to date: 'tests/run_if_test_argv0.out' Modify the script. >>> with open('tests/run_if_test_argv0.bat', 'a') as f: ... print('echo hello >> tests/run_if_test_argv0.out', file=f) And now it is run because the hash of argv[0] has changed: >>> run_if(f'"tests/run_if_test_argv0.bat"', f'tests/run_if_test_argv0.out', caller=0) run_if(): Running command because: arg0 hash has changed. run_if(): Running: "tests/run_if_test_argv0.bat" True ''' pass else: def _run_if_test_scripting_unix(): ''' Dummy function to allow non-Windows doctests for pipcl.run_if(). We detect changes to the contents of argv[0]: Create a shell script and run it: >>> del _log_f[:] >>> _log_f.append(sys.stdout) >>> log_prefix('%f(): ') >>> for path in glob.glob('tests/run_if_test_argv0.*'): ... fs_remove(path) >>> with open('tests/run_if_test_argv0.sh', 'w') as f: ... print('#! /bin/sh', file=f) ... print('echo hello world > tests/run_if_test_argv0.out', file=f) >>> import stat >>> os.chmod('tests/run_if_test_argv0.sh', os.stat('tests/run_if_test_argv0.sh').st_mode | stat.S_IEXEC) >>> run_if('tests/run_if_test_argv0.sh', f'tests/run_if_test_argv0.out', caller=0) run_if(): Running command because: File does not exist: 'tests/run_if_test_argv0.out' run_if(): Running: tests/run_if_test_argv0.sh True Running it a second time does nothing: >>> run_if( f'tests/run_if_test_argv0.sh', f'tests/run_if_test_argv0.out', caller=0) run_if(): Not running command because up to date: 'tests/run_if_test_argv0.out' Modify the script. >>> with open('tests/run_if_test_argv0.sh', 'a') as f: ... print('echo hello >> tests/run_if_test_argv0.out', file=f) And now it is run because the hash of argv[0] has changed: >>> run_if( f'tests/run_if_test_argv0.sh', f'tests/run_if_test_argv0.out', caller=0) run_if(): Running command because: arg0 hash has changed. run_if(): Running: tests/run_if_test_argv0.sh True ''' pass def fs_find_in_paths( name, paths=None, verbose=False): ''' Looks for `name` in paths and returns complete path. `paths` is list/tuple or `os.pathsep`-separated string; if `None` we use `$PATH`. If `name` contains `/`, we return `name` itself if it is a file or None, regardless of . ''' if '/' in name: return name if os.path.isfile( name) else None if paths is None: paths = os.environ.get( 'PATH', '') if verbose: log('From os.environ["PATH"]: {paths=}') if isinstance( paths, str): paths = paths.split( os.pathsep) if verbose: log('After split: {paths=}') for path in paths: p = os.path.join( path, name) if verbose: log('Checking {p=}') if os.path.isfile( p): if verbose: log('Returning because is file: {p!r}') return p if verbose: log('Returning None because not found: {name!r}') def fs_symlink(from_, to_): ''' Makes symbolic link from to . ''' fs_remove(from_) os.symlink(to_, from_) def _get_prerequisites(path): ''' Returns list of prerequisites from Makefile-style dependency file, e.g. created by `cc -MD -MF `. ''' ret = list() if os.path.isfile(path): with open(path) as f: for line in f: for item in line.split(): if item.endswith( (':', '\\')): continue ret.append( item) return ret def _fs_mtime_newest( path): ''' path: If a file, returns mtime of the file. If a directory, returns mtime of newest file anywhere within directory tree. Otherwise returns 0. ''' ret = 0 if os.path.isdir( path): for dirpath, _dirnames, filenames in os.walk( path): for filename in filenames: path = os.path.join( dirpath, filename) ret = max( ret, _fs_mtime( path)) else: ret = _fs_mtime( path) return ret def _flags( items, prefix='', quote=''): ''' Turns sequence into string, prefixing/quoting each item. ''' if not items: return '' if isinstance( items, str): items = items, ret = '' for item in items: if ret: ret += ' ' ret += f'{prefix}{quote}{item}{quote}' return ret.strip() def _fs_mtime( filename, default=0): ''' Returns mtime of file, or `default` if error - e.g. doesn't exist. ''' try: return os.path.getmtime( filename) except OSError: return default def _normalise(name): ''' https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization ''' return re.sub(r"[-_.]+", "-", name).lower() def _normalise2(name): ''' https://packaging.python.org/en/latest/specifications/binary-distribution-format/ ''' return _normalise(name).replace('-', '_') def _assert_version_pep_440(version): ''' Asserts that cconforms with PEP 440. ''' assert re.match( r'^([1-9][0-9]*!)?(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*))*((a|b|rc)(0|[1-9][0-9]*))?(\.post(0|[1-9][0-9]*))?(\.dev(0|[1-9][0-9]*))?$', version, ), \ f'Bad version: {version!r}.' g_verbose = int(os.environ.get('PIPCL_VERBOSE', '1')) def verbose(level=None): ''' Sets verbose level if `level` is not None. Returns verbose level. ''' global g_verbose if level is not None: g_verbose = level return g_verbose def log(text='', caller=1): ''' Writes to log. Caller identifies the frame to use if file:line is included in prefix; default 1 identifies the caller. ''' _log(text, 0, caller+1) def log0(text='', caller=1): _log(text, 0, caller+1) def log1(text='', caller=1): _log(text, 1, caller+1) def log2(text='', caller=1): _log(text, 2, caller+1) def _duration(s): ''' Return representation of elapsed time +[[[DAYSd]HOURSh]MINSm]Ss, for example 1h03m45.3s. ''' m, s = divmod(s, 60) h, m = divmod(m, 60) d, h = divmod(h, 24) m = int(m) h = int(h) d = int(d) if d: return f'{d}d{h:02}h{m:02}m{s:04.1f}s' elif h: return f'{h:}h{m:02}m{s:04.1f}s' elif m: return f'{m}m{s:04.1f}s' else: return f'{s:.1f}s' # Log prefix pattern. # %d - elapsed time +[[[DAYSd]HOURSh]MINSm]Ss, for example 1h03m45.3s. # %f - function name # %l - line number # %p - path. # %t - current time YYYY-mm-dd-HH-MM-SS g_log_format = '[+%d] %p:%l: %f(): ' g_log_t0 = time.time() def log_prefix(pattern): ''' Sets log prefix pattern. ''' global g_log_format g_log_format = pattern _log_prefix_stack = list() class LogPrefix: ''' A context manager to add a temporary prefix to log output. ''' def __init__(self, prefix): self.prefix = prefix def __enter__(self): _log_prefix_stack.append(self.prefix) def __exit__(self, *args): del _log_prefix_stack[-1] def _log_prefix(format_, caller): ''' Evaluates log format for . ''' ret = '' _get_frame_cache = [None] def get_frame(): if _get_frame_cache[0] is None: frame = inspect.stack(context=0)[caller+1] path = relpath(frame.filename) _get_frame_cache[0] = frame, path return _get_frame_cache[0] # pylint: disable=unpacking-non-sequence pos = 0 while pos < len(format_): pos0 = pos if 0: pass elif format_[pos:].startswith('%d'): # Elapsed time. pos += 2 s = time.time() - g_log_t0 ret += _duration(s) elif format_[pos:].startswith('%f'): # Function name. pos += 2 frame, path = get_frame() ret += frame.function elif format_[pos:].startswith('%l'): # Line number. pos += 2 frame, path = get_frame() ret += str(frame.lineno) elif format_[pos:].startswith('%p'): # Path. pos += 2 frame, path = get_frame() ret += path elif format_[pos:].startswith('%t'): # Current date and time. pos += 2 ret += time.strftime(f'%F-%T') else: ret += format_[pos] pos += 1 assert pos > pos0 ret += ''.join(_log_prefix_stack) return ret _log_text_line_start = True _log_f = [sys.stdout] _log_tee_f = None _log_tee_exit_fn = None def log_tee(path, path_simple=None): ''' Copies log output to . If path_simple is specified, on exit we make a convenience symlink from to . ''' global _log_tee_f global _log_tee_exit_fn # Remove any current tee setting. if _log_tee_f: _log_f.remove(_log_tee_f) _log_tee_f = None if _log_tee_exit_fn: atexit.unregister(_log_tee_exit_fn) _log_tee_exit_fn = None if path: _log_tee_f = open(path, 'w') # pylint: disable=consider-using-with _log_f.append(_log_tee_f) if path_simple: def final(): fs_symlink(path_simple, path) _log_tee_exit_fn = final atexit.register(final) def _log2(text): #print(f'####### {_log_f=}') #sys.exit() if _log_f is None: f = [sys.stdout] elif hasattr(_log_f, '__len__'): f = _log_f else: f = [_log_f] for ff in f: try: ff.write(text) except UnicodeEncodeError: ff.write("[[pipcl:Ignoring encoding error]]") ff.flush() def _log(text, level, caller, *, raw=False, nl=True, format_=None): ''' Logs lines with prefix, if is lower or equal to . ''' if level > g_verbose: return if format_ is None: format_ = g_log_format prefix = None global _log_text_line_start pos = 0 while 1: if pos == len(text): break if not raw or _log_text_line_start: if prefix is None: prefix = _log_prefix(format_, caller+1) _log2(prefix) nlp = text.find('\n', pos) if nlp == -1: _log2(text[pos:]) if not raw and nl: _log2('\n') pos = len(text) else: _log2(text[pos:nlp+1]) pos = nlp+1 if raw: _log_text_line_start = (nlp >= 0) def relpath(path, start=None, allow_up=True): ''' A safe alternative to `os.path.relpath()`, avoiding an exception on Windows if the drive needs to change - in this case we use `os.path.abspath()`. Args: path: Path to be processed. start: Start directory or current directory if None. allow_up: If false we return absolute path is is not within . ''' if windows(): try: ret = os.path.relpath(path, start) except ValueError: # os.path.relpath() fails if trying to change drives. ret = os.path.abspath(path) else: ret = os.path.relpath(path, start) if not allow_up and ret.startswith('../') or ret.startswith('..\\'): ret = os.path.abspath(path) return ret def _so_suffix(use_so_versioning=True): ''' Filename suffix for shared libraries is defined in pep-3149. The pep claims to only address posix systems, but the recommended sysconfig.get_config_var('EXT_SUFFIX') also seems to give the right string on Windows. If use_so_versioning is false, we return only the last component of the suffix, which removes any version number, for example changing `.cp312-win_amd64.pyd` to `.pyd`. ''' # Example values: # linux: .cpython-311-x86_64-linux-gnu.so # macos: .cpython-311-darwin.so # openbsd: .cpython-310.so # windows .cp311-win_amd64.pyd # # Only Linux and Windows seem to identify the cpu. For example shared # libraries in numpy-1.25.2-cp311-cp311-macosx_11_0_arm64.whl are called # things like `numpy/core/_simd.cpython-311-darwin.so`. # ret = sysconfig.get_config_var('EXT_SUFFIX') if not use_so_versioning: # Use last component only. ret = os.path.splitext(ret)[1] return ret def get_soname(path): ''' If we are on Linux and `path` is softlink and points to a shared library for which `objdump -p` contains 'SONAME', return the pointee. Otherwise return `path`. Useful if Linux shared libraries have been created with `-Wl,-soname,...`, where we need to embed the versioned library. ''' if linux() and os.path.islink(path): path2 = os.path.realpath(path) if subprocess.run(f'objdump -p {path2}|grep SONAME', shell=1, check=0).returncode == 0: return path2 elif openbsd(): # Return newest .so with version suffix. sos = glob.glob(f'{path}.*') log1(f'{sos=}') sos2 = list() for so in sos: suffix = so[len(path):] if not suffix or re.match('^[.][0-9.]*[0-9]$', suffix): sos2.append(so) sos2.sort(key=lambda p: os.path.getmtime(p)) log1(f'{sos2=}') return sos2[-1] return path def current_py_limited_api(): ''' Returns value of PyLIMITED_API to build for current Python. ''' a, b = map(int, platform.python_version().split('.')[:2]) return f'0x{a:02x}{b:02x}0000' def install_dir(root=None): ''' Returns install directory used by `install()`. This will be `sysconfig.get_path('platlib')`, modified by `root` if not None. ''' # todo: for pure-python we should use sysconfig.get_path('purelib') ? root2 = sysconfig.get_path('platlib') if root: if windows(): # If we are in a venv, `sysconfig.get_path('platlib')` # can be absolute, e.g. # `C:\\...\\venv-pypackage-3.11.1-64\\Lib\\site-packages`, so it's # not clear how to append it to `root`. So we just use `root`. return root else: # E.g. if `root` is `install' and `sysconfig.get_path('platlib')` # is `/usr/local/lib/python3.9/site-packages`, we set `root2` to # `install/usr/local/lib/python3.9/site-packages`. # return os.path.join( root, root2.lstrip( os.sep)) else: return root2 def zipfile_writestr(zf, name, contents, unix_permissions=0o600): ''' Alternative to `zipfile.ZipFile.writestr()`, which allows control over permissions. Our default 0o600 matches `zipfile.ZipFile.writestr()`. ''' # There isn't a good way of setting permissions, zipfile seems very badly # designed here. zipinfo = zipfile.ZipInfo( filename = name, date_time = time.localtime(time.time())[:6], ) zipinfo.compress_type = zf.compression if python_version_tuple() >= (3, 13): zipinfo.compress_level = zf.compresslevel else: zipinfo._compresslevel = zf.compresslevel zipinfo.external_attr = unix_permissions << 16 zf.writestr(zipinfo, contents) class _Record: ''' Internal - builds up text suitable for writing to a RECORD item, e.g. within a wheel. ''' def __init__(self): self.lines = list() def add_content(self, content, to_, verbose=True): if isinstance(content, str): content = content.encode('utf8') # Specification for the line we write is supposed to be in # https://packaging.python.org/en/latest/specifications/binary-distribution-format # but it's not very clear. # h = hashlib.sha256(content) digest = h.digest() digest = base64.urlsafe_b64encode(digest) digest = digest.rstrip(b'=') digest = digest.decode('utf8') self.lines.append(f'{to_.replace(os.sep, "/")},sha256={digest},{len(content)}\n') if verbose: log2(f'Adding {to_}') def add_file(self, from_, to_, to_maxlen=0): log1(f'Adding file: {to_.ljust(to_maxlen)} <= {os.path.relpath(from_)}') with open(from_, 'rb') as f: content = f.read() self.add_content(content, to_, verbose=False) def get(self, record_path=None): ''' Returns contents of the RECORD file. If `record_path` is specified we append a final line `,,`; this can be used to include the RECORD file itself in the contents, with empty hash and size fields. ''' self.lines.sort() ret = ''.join(self.lines) if record_path: ret += f'{record_path},,\n' log(f'{record_path=}') log(f'ret is:\n{textwrap.indent(ret, " ")}') return ret class NewFiles: ''' Detects new/modified/updated files matching a glob pattern. Useful for detecting wheels created by pip or cubuildwheel etc. ''' def __init__(self, glob_pattern, recursive=None): ''' Sets things up to look for new files matching . is passed to `glob.glob()`, so for example set to true if using `**`. ''' # Find current matches of . self.glob_pattern = glob_pattern self.recursive = recursive self.items0 = self._items() def get(self): ''' Returns list of new matches of - paths of files that were not present previously, or have different mtimes or have different contents. ''' ret = list() items = self._items() for path, id_ in items.items(): id0 = self.items0.get(path) if id0 != id_: ret.append(path) return ret def get_n(self, n): ''' Returns new files matching , asserting that there are exactly . ''' ret = self.get() sep = '\n ' assert len(ret) == n, f'{len(ret)=} != {n=}:{sep}{sep.join(ret)}. {self.glob_pattern=}.' return ret def get_one(self): ''' Returns new match of , asserting that there is exactly one. ''' return self.get_n(1)[0] def _file_id(self, path): mtime = os.stat(path).st_mtime with open(path, 'rb') as f: content = f.read() hash_ = hashlib.md5(content).digest() # With python >= 3.11 we can do: #hash_ = hashlib.file_digest(f, hashlib.md5).digest() return mtime, hash_ def _items(self): ret = dict() for path in glob.glob(self.glob_pattern, recursive=self.recursive): if os.path.isfile(path): ret[path] = self._file_id(path) return ret def swig_prepare_build(swig_env_extra): ''' Builds required tools for building swig (not Windows). Adds to swig_env_extra['PATH']. ''' if darwin(): run(f'brew install automake') run(f'brew install pcre2') run(f'brew install bison') # Default bison doesn't work, and Brew's bison is not added to $PATH. # # > bison is keg-only, which means it was not symlinked into /opt/homebrew, # > because macOS already provides this software and installing another version in # > parallel can cause all kinds of trouble. # > # > If you need to have bison first in your PATH, run: # > echo 'export PATH="/opt/homebrew/opt/bison/bin:$PATH"' >> ~/.zshrc # macos_add_brew_path('bison', swig_env_extra) run(f'which bison') run(f'which bison', env_extra=swig_env_extra) # Building swig requires bison>=3.5. bison_ok = 0 e, text = run(f'bison --version', capture=1, check=0, env_extra=swig_env_extra) if not e: log(textwrap.indent(text, ' ')) m = re.search('bison (GNU Bison) ([0-9]+)[.]([0-9]+)', text) if m: assert m, f'Unexpected output from `bison --version`: {text!r}' version_tuple = int(m.group(1)), int(m.group2()) if version_tuple >= (3, 5): bison_ok = 1 if not bison_ok: if 0: # Use git checkout. Fails to find scan-code.c. Presumably # something wrong with ./bootstrap? log(f'Cloning/fetching/build/installing bison.') bison_git = git_get( 'pipcl-bison-git', remote='https://git.savannah.gnu.org/git/bison.git', #branch='master', tag='v3.5.4', submodules=0, # recursive update fails. ) run(f'cd {bison_git} && git submodule update --init', prefix='bison git submodule update --init: ') run(f'cd {bison_git} && ./bootstrap', prefix='bison bootstrap: ') run(f'cd {bison_git} && ./configure', prefix='bison configure: ') run(f'cd {bison_git} && make', prefix='bison make: ') run(f'cd {bison_git} && sudo make install', prefix='bison make install: ') else: bison_version = 'bison-3.5.4' if not os.path.exists(f'{bison_version}.tar.gz'): run( f'wget -O {bison_version}.tar.gz-0 http://www.mirrorservice.org/sites/ftp.gnu.org/gnu/bison/{bison_version}.tar.gz', prefix='bison wget: ', ) os.rename(f'{bison_version}.tar.gz-0', f'{bison_version}.tar.gz') if not os.path.exists(f'{bison_version}'): run(f'tar -xzf {bison_version}.tar.gz', prefix='bison extract: ') # 2026-02-23: builds have started failing, perhaps recent compilers have extra detection? run(f'cd {bison_version} && ./configure', prefix='bison configure: ', env_extra=dict(CFLAGS='-Wno-incompatible-function-pointer-types')) run(f'cd {bison_version} && make', prefix='bison make: ') run(f'cd {bison_version} && sudo make install', prefix='bison make install: ') def swig_get(swig, quick, swig_local='pipcl-swig-git'): ''' Returns or a new swig binary. If is true and starts with 'git:' (not Windows), the remaining text is passed to `git_get()` and we clone/update/build swig, and return the built binary. We default to the main swig repository, branch master, so for example 'git:' will return the latest swig from branch master. Otherwise we simply return . Args: swig: If starts with 'git:', passed as arg to `git_get()`, and we return the absolute path of the build binary. (Not windows.) If starts with 'pip:' we do: `pip install swig` and return 'swig'. E.g. 'pip:==4.3.1'. Otherwise we simply return . quick: If true, we do not update/build local checkout if the binary is already present. swig_local: path to use for checkout. ''' #log(f'{swig=}') if swig and swig.startswith('git:'): assert platform.system() != 'Windows', f'Cannot build swig on Windows.' # Note that {swig_local}/install/bin/swig doesn't work on MacOS because # {swig_local}/INSTALL is a file and the fs is case-insensitive. swig_binary = os.path.abspath(f'{swig_local}/install-dir/bin/swig') if quick and os.path.isfile(swig_binary): log1(f'{quick=} and {swig_binary=} already exists, so not downloading/building.') else: # Clone swig. swig_env_extra = None swig_local = git_get( swig_local, text=swig, remote='https://github.com/swig/swig.git', branch='master', ) swig_prepare_build(swig_env_extra) # Build swig. run(f'cd {swig_local} && ./autogen.sh', out=log, env_extra=swig_env_extra, prefix='build swig autogen.sh: ', ) run(f'cd {swig_local} && ./configure --prefix={swig_local}/install-dir', out=log, env_extra=swig_env_extra, prefix='build swig configure: ', ) run(f'cd {swig_local} && make', out=log, env_extra=swig_env_extra, prefix='build swig make: ', ) run(f'cd {swig_local} && make install', out=log, env_extra=swig_env_extra, prefix='build swig install: ', ) assert os.path.isfile(swig_binary) return swig_binary elif swig and swig.startswith('pip:'): run(f'pip uninstall -y swig', check=0) run(f'pip install swig{swig[4:]}') return 'swig' else: return swig def macos_add_brew_path(package, env=None, gnubin=True): ''' Adds path(s) for Brew 's binaries to env['PATH']. We assert-fail if the relevant directory does no exist. Args: package: Name of package. We get of installed package by running `brew --prefix `. env: The environment dict to modify. If None we use os.environ. If PATH is not in , we first copy os.environ['PATH'] into . gnubin: If true, we also add path to gnu binaries if it exists, /libexe/gnubin. ''' if not darwin(): return if env is None: env = os.environ if 'PATH' not in env: env['PATH'] = os.environ['PATH'] package_root = run(f'brew --prefix {package}', capture=1).strip() log(f'{package=} {package_root=}') def add(path): log(f'{path=}') if os.path.isdir(path): log(f'Prepending to $PATH: {path}') PATH = env['PATH'] env['PATH'] = f'{path}:{PATH}' return 1 else: log(f'Not a directory: {path=}') return 0 n = 0 n += add(f'{package_root}/bin') if gnubin: n += add(f'{package_root}/libexec/gnubin') assert n, f'Failed to add to $PATH, {package=} {gnubin=}.' def _show_dict(d): ret = '' for n in sorted(d.keys()): v = d[n] ret += f' {n}: {v!r}\n' return ret def show_sysconfig(): ''' Shows contents of `sysconfig.get_paths()` and `sysconfig.get_config_vars()` dicts. ''' log0(f'show_sysconfig().') log0(f'sysconfig.get_paths():\n{_show_dict(sysconfig.get_paths())}') log0(f'sysconfig.get_config_vars():\n{_show_dict(sysconfig.get_config_vars())}') def sysconfig_python_flags(): ''' Returns include paths and library directory for Python. Uses sysconfig.*(), overridden by environment variables PIPCL_SYSCONFIG_PATH_include, PIPCL_SYSCONFIG_PATH_platinclude and PIPCL_SYSCONFIG_CONFIG_VAR_LIBDIR if set. ''' include1_ = os.environ.get('PIPCL_SYSCONFIG_PATH_include') or sysconfig.get_path('include') include2_ = os.environ.get('PIPCL_SYSCONFIG_PATH_platinclude') or sysconfig.get_path('platinclude') ldflags_ = os.environ.get('PIPCL_SYSCONFIG_CONFIG_VAR_LIBDIR') or sysconfig.get_config_var('LIBDIR') includes_ = [include1_] if include2_ != include1_: includes_.append(include2_) if windows(): includes_ = [f'/I"{i}"' for i in includes_] ldflags_ = f'/LIBPATH:"{ldflags_}"' else: includes_ = [f'-I {i}' for i in includes_] ldflags_ = f'-L {ldflags_}' includes_ = ' '.join(includes_) return includes_, ldflags_ def _commands_similar(a, b): ''' Experimental debug fn for comparing command lines. ''' def canonalisee(command): args = shlex.split(command) i = 0 j = 1 lines = list() while 1: if j == len(args) or args[j].startswith(('-', '/')): line = shlex.join(args[i:j]) i = j lines.append(line) if j == len(args): break j += 1 lines = lines[:1] + sorted(lines[1:]) ret = '\n'.join(lines) return ret a = canonalisee(a) b = canonalisee(b) log(f'a:') log(f'{textwrap.indent(a, " ")}') log(f'b:') log(f'{textwrap.indent(b, " ")}') return a == b def python_version_tuple(): ''' Like platform.python_version_tuple() except converts items to integers if possible. ''' return version_to_tuple(platform.python_version_tuple()) def str_to_int(s, replace=str): ''' Returns string converted to int, with configurable behaviour if the string is not convertible. Args: s The input string. replace: Governs what we do if int(s) fails: None: Raise. str: Return the string . Otherwise: Return itself. ''' try: ret = int(s) except Exception: if replace is None: raise elif replace is str: return s else: ret = replace return ret def version_to_tuple(version, replace=str): ''' Returns a tuple containing each item from converted to an int. If an item does not convert to an int, behaviour depends on . Args: version A string containing items separated by '.', or a sequence of strings. replace Governs what we do if conversion of item to int fails: None: Raise. str: Use the string value. Otherwise: Use itself. `.`-separated string as tuple, with each item converted to an int, or kept as a string if the conversion failed (or to if not `str`. >>> version_to_tuple('1.2.3') (1, 2, 3) >>> version_to_tuple('1.2.3.alpha') (1, 2, 3, 'alpha') >>> version_to_tuple('1.2.3.alpha', replace=0) (1, 2, 3, 0) ''' if isinstance(version, str): version = version.split('.') return tuple(str_to_int(i, replace) for i in version) def venv_enter_cmd(venv_name): ''' Returns command that will enter venv, different on Windows vs Unix. ''' if windows(): return f'{venv_name}/Scripts/activate'.replace('/', '\\') else: return f'. {venv_name}/bin/activate' if __name__ == '__main__': # Internal-only limited command line support, used if # graal_legacy_python_config is true. # includes_, ldflags = sysconfig_python_flags() if len(sys.argv) == 1: run('sleep 4', ticker=0.5, timeout=0) run('sleep 3', ticker=0.25, timeout=0) elif sys.argv[1] == '--doctest': import doctest if sys.argv[2:]: for ff in sys.argv[2:]: fff = globals()[ff] doctest.run_docstring_examples(fff, globals()) # Unfortunately doctest.run_docstring_examples() always seems # to return None even if test(s) failed. else: doctest.testmod(None) elif sys.argv[1:] == ['--graal-legacy-python-config', '--includes']: print(includes_) elif sys.argv[1:] == ['--graal-legacy-python-config', '--ldflags']: print(ldflags) elif sys.argv[1:] == ['--check-git_info']: git_info('aptest') else: assert 0, f'Expected `--graal-legacy-python-config --includes|--ldflags` but {sys.argv=}' ArtifexSoftware-pipcl-04f69b2/src/wdev.py0000664000175000017500000005645015245537643020336 0ustar epsilonepsilon''' Finds locations of Windows command-line development tools. ''' import json import os import platform import glob import re import subprocess import sys import sysconfig import textwrap import pipcl class WindowsVS: r''' Windows only. Finds locations of Visual Studio command-line tools. Assumes VS2019-style paths. Members and example values:: .year: 2019 .grade: Community .version: 14.28.29910 .directory: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community .vcvars: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat .cl: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29910\bin\Hostx64\x64\cl.exe .link: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29910\bin\Hostx64\x64\link.exe .csc: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn\csc.exe .msbuild: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe .devenv: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\devenv.com .tools: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\devenv.com .toolvs: [142, 143] .vsversion: 17.14.37111.16 .product_line_version: 2017, 2019, 2022, 18 etc. .vswhere: Raw information from vswhere.exe. `.csc` is C# compiler; will be None if not found. ''' def __init__(self, *args, **kwargs): ''' Internal use only. Use windows_vs() or windows_vs_multiple() to create instance(s). ''' if args: assert len(args) == 2 and not kwargs, f'{args=} {kwargs=}' vswhere, cpu = args else: # Backwards compatibility only - callers should use windows_vs(). year = kwargs.pop('year', None) grade = kwargs.pop('grade', None) version = kwargs.pop('version', None) cpu = kwargs.pop('cpu', None) directory = kwargs.pop('directory', None) assert not kwargs, f'Unrecognised args: {kwargs=}' vs = windows_vs(year=year, grade=grade, version=version, directory=directory, cpu=cpu) # Want to set self = vs, but that's not going to work, so instead # we just construct from vs.vswhere and cpu. vswhere = vs.vswhere if not cpu: cpu = WindowsCpu() #pipcl.log(f'vswhere:\n{json.dumps(vswhere, indent=" ")}') product_line_version = vswhere['catalog']['productLineVersion'] # E.g. '2022'. vsversion = vswhere['catalog']['buildVersion'] # E.g. '17.14.37111.16'. grade = vswhere['productId'].split('.')[-1] # E.g. 'Professional'. directory = vswhere['installationPath'] # E.g. C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional version = None # Compiler/linker version is not specified in vswhere. year = None try: # Find `devenv`. # devenv = f'{directory}\\Common7\\IDE\\devenv.com' assert os.path.isfile( devenv), f'Does not exist: {devenv}' # Find vcvars.bat. # if cpu.name == 'x32': leaf = 'vcvars32.bat' elif cpu.name == 'x64': leaf = 'vcvars64.bat' elif cpu.name == 'arm32': leaf = 'vcvarsarm.bat' elif cpu.name == 'arm64': leaf = 'vcvarsarm64.bat' else: assert 0, f'Cannot specify vcvars*.bat because unrecognised {cpu.name=}.' vcvars = f'{directory}\\VC\\Auxiliary\\Build\\{leaf}' assert os.path.isfile( vcvars), f'No match for: {vcvars}' if 0: # List contents of . _root = f'{directory}\\VC\\Tools\\MSVC' pipcl.log(f'*.exe files in {_root=}:') for dirpath, _dirnames, filenames in os.walk(_root): for filename in filenames: if filename.endswith('.exe'): path = f'{dirpath}\\{filename}' pipcl.log(f' {path}') # Find cl.exe. # cl_pattern = f'{directory}\\VC\\Tools\\MSVC\\{version or "*"}\\bin\\Host{cpu.windows_name}\\{cpu.windows_name}\\cl.exe' cl_s = glob.glob( cl_pattern) assert cl_s, f'No match for: {cl_pattern}' cl_s.sort() cl = cl_s[-1] # Extract `version` from cl.exe's path. # m = re.search( rf'\\VC\\Tools\\MSVC\\([^\\]+)\\bin\\Host{cpu.windows_name}\\{cpu.windows_name}\\cl.exe$', cl) assert m version2 = m.group(1) #pipcl.log(f'Overriding {version=} to {version2=}.') version = version2 assert version # Find link.exe. # link_pattern = f'{directory}\\VC\\Tools\\MSVC\\{version or "*"}\\bin\\Host{cpu.windows_name}\\{cpu.windows_name}\\link.exe' link_s = glob.glob( link_pattern) assert link_s, f'No match for: {link_pattern}' link_s.sort() link = link_s[-1] # Find csc.exe. # csc = None for dirpath, _dirnames, filenames in os.walk(directory): for filename in filenames: if filename == 'csc.exe': csc = os.path.join(dirpath, filename) #_log(f'{csc=}') #break # Find MSBuild.exe. # msbuild = None for dirpath, _dirnames, filenames in os.walk(directory): for filename in filenames: if filename == 'MSBuild.exe': msbuild = os.path.join(dirpath, filename) #_log(f'{csc=}') #break # Find available build tool versions. # tools = list() toolvs = list() for p in glob.glob(f'{directory}\\VC/Auxiliary\\Build\\Microsoft.VCToolsVersion.v*.default.txt'): tools.append(p) leaf = os.path.basename(p) #pipcl.log(f'{p=}') #pipcl.log(f'{leaf=}') m = re.match('^Microsoft.VCToolsVersion.v([0-9]+).default.txt$', leaf) toolv = int(m.group(1)) toolvs.append(toolv) self.cl = cl self.devenv = devenv self.directory = directory self.grade = grade self.link = link self.csc = csc self.msbuild = msbuild self.vcvars = vcvars self.version = version self.year = year or vsversion_to_year(vsversion) self.cpu = cpu self.tools = tools self.toolvs = toolvs self.vsversion = vsversion self.product_line_version = product_line_version self.vswhere = vswhere except Exception as e: raise Exception( f'Unable to find Visual Studio {year=} {grade=} {version=} {cpu=} {directory=}') from e def description_ml( self, indent=''): ''' Return multiline description of `self`. ''' ret = textwrap.dedent(f''' year: {self.year} vsversion: {self.vsversion} grade: {self.grade} version: {self.version} directory: {self.directory} vcvars: {self.vcvars} cl: {self.cl} link: {self.link} csc: {self.csc} msbuild: {self.msbuild} devenv: {self.devenv} cpu: {self.cpu} ''').strip() + '\n' ret += 'tools:\n' for p in self.tools: ret += f' {p}\n' ret += 'toolvs:\n' for v in self.toolvs: ret += f' {v}\n' return textwrap.indent( ret, indent) def __repr__( self): items = list() for name in ( 'year', 'vsversion', 'grade', 'version', 'directory', 'vcvars', 'cl', 'link', 'csc', 'msbuild', 'devenv', 'cpu', ): items.append(f'{name}={getattr(self, name)!r}') return ' '.join(items) _vsversion_year = [ (2017, 15), (2019, 16), (2022, 17), (2026, 18), ] def vsversion_to_year(vsversion, check=0): if isinstance(vsversion, str): vsversion = pipcl.version_to_tuple(vsversion) for y, v in _vsversion_year: if vsversion[0] == v: return y if check: t = ', '.join([v for (y, v) in _vsversion_year]) assert 0, f'In {vsversion=}, unrecognised {vsversion[0]=}, must be one of {t}.' def year_to_vsversion(year, check=0): if isinstance(year, str): year = int(year) for y, v in _vsversion_year: if year == y: return v, if check: t = ', '.join([str(y) for (y, v) in _vsversion_year]) assert 0, f'Unrecognised {year=} must be one of {t}.' def version_startswith(v, prefix): if isinstance(v, str): v = pipcl.version_to_tuple(v) if isinstance(prefix, str): prefix = pipcl.version_to_tuple(prefix) return v[:len(prefix)] == prefix def windows_vs_multiple(*, year=None, vsversion=None, grade=None, version=None, directory=None, cpu=None, verbose=0): ''' Returns list of matching WindowsVS instances, using vswhere.exe Args: year: None or one of: 2017 2019 2022 2026 These map to visual studio versions 15, 16, 17 and 18. Must not be specified with . vsversion: None or Visual Studio build version, e.g. '17.14.37111.16'. Must not be specified with . grade: None or one of: * `Community` * `Professional` * `Enterprise` If None we use environment variable WDEV_VS_GRADE if set. version: None or, for example, `14.28.29910`. If None we use environment variable WDEV_VS_VERSION if set. This is not matched against output from vswhere.exe, instead identifies subdirectory in the cl.exe and link.exe paths. If None, we use the highest available version. directory: None or the installation directory. cpu: None or a `WindowsCpu` instance. Example vswhere.exe information about a vs install: { "instanceId": "c5be8a09", "installDate": "2025-11-27T16:58:25Z", "installationName": "VisualStudio/17.14.29+37111.16.-march.2026-", "installationPath": "C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional", "installationVersion": "17.14.37111.16", "productId": "Microsoft.VisualStudio.Product.Professional", "productPath": "C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\Common7\\IDE\\devenv.exe", "state": 4294967295, "isComplete": true, "isLaunchable": true, "isPrerelease": false, "isRebootRequired": false, "displayName": "Visual Studio Professional 2022", "description": "Professional IDE best suited to small teams", "channelId": "VisualStudio.17.Release", "channelUri": "https://aka.ms/vs/17/release/channel", "enginePath": "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\resources\\app\\ServiceHub\\Services\\Microsoft.VisualStudio.Setup.Service", "installedChannelId": "VisualStudio.17.Release", "installedChannelUri": "https://aka.ms/vs/17/release/channel", "releaseNotes": "https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.14#17.14.29", "resolvedInstallationPath": "C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional", "thirdPartyNotices": "https://go.microsoft.com/fwlink/?LinkId=661288", "updateDate": "2026-04-03T10:46:24.9483366Z", "catalog": { "buildBranch": "d17.14", "buildVersion": "17.14.37111.16", "id": "VisualStudio/17.14.29+37111.16.-march.2026-", "localBuild": "build-lab", "manifestName": "VisualStudio", "manifestType": "installer", "productDisplayVersion": "17.14.29 (March 2026)", "productLine": "Dev17", "productLineVersion": "2022", "productMilestone": "RTW", "productMilestoneIsPreRelease": "False", "productName": "Visual Studio", "productPatchVersion": "29", "productPreReleaseMilestoneSuffix": "1.0", "productReleaseNameSuffix": "(March 2026)", "productSemanticVersion": "17.14.29+37111.16.-march.2026-", "requiredEngineVersion": "3.14.2086.54749" }, "properties": { "appLocalWPF": "false", "campaignId": "", "channelManifestId": "VisualStudio.17.Release/17.14.29+37111.16.-march.2026-", "includeRecommended": "1", "nickname": "", "setupEngineFilePath": "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\setup.exe" } } ''' year = year or os.environ.get('WDEV_VS_YEAR') grade = grade or os.environ.get('WDEV_VS_GRADE') version = version or os.environ.get('WDEV_VS_VERSION') if year: # We always convert year to vsversion. assert not vsversion vsversion = year_to_vsversion(year, check=1) elif vsversion: assert not year vsversion = pipcl.version_to_tuple(vsversion) year = vsversion_to_year(vsversion, check=1) vss_json = pipcl.run(rf'"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -format json', capture=1) vss = json.loads(vss_json) #pipcl.log(f'vss:\n{json.dumps(vss, indent=" ")}') def select(wvs): if vsversion and not version_startswith(wvs.vsversion, vsversion): #pipcl.log(f'Returning false because {vsversion=} {wvs.vsversion=}') return False if grade and wvs.grade != grade: #pipcl.log(f'Returning false because {grade=} {wvs.grade=}') return False if directory and wvs.directory != directory: #pipcl.log(f'Returning false because {directory=} {wvs.directory=}') return False #pipcl.log(f'Returning true.') return True ret = list() for vs in vss: wvs = WindowsVS(vs, cpu) #pipcl.log(f'wvs:\n{wvs.description_ml(" ")}') if select(wvs): ret.append(wvs) return ret def windows_vs(*, year=None, grade=None, version=None, directory=None, cpu=None, verbose=0, check=1): ''' Returns best matching WindowsVS. Args are same as windows_vs_multiple(). check: If true (the default) we raise exception if not found, otherwise we return None. ''' wvss = windows_vs_multiple(year=year, grade=grade, version=version, directory=directory, cpu=cpu, verbose=verbose) if not wvss: if check: wvss = windows_vs_multiple() text = f'No Visual Studio found matching {year=} {grade=} {version=} {directory=}.\n' text += f'Available installations are ({len(wvss)}):\n' for wvs in wvss: text += f'\n' text += wvs.description_ml(' ') raise Exception(text) else: return None def key(wvs): grades = ['Community', 'Professional', 'Enterprise'] return ( pipcl.version_to_tuple(wvs.version), grades.index(wvs.grade), ) wvss.sort(key=key) return wvss[-1] class WindowsCpu: ''' For Windows only. Paths and names that depend on cpu. Members: .bits 32 or 64. .name One of: arm32 arm64 x32 x64. .windows_subdir Empty string, `x64/`, `arm64` etc. .windows_name `x86`,`x64`, `arm64` etc. .windows_config `x64`, `Win32`, `arm64` etc, e.g. for use in `/Build Release|x64`. .windows_suffix `64` or empty string. ARM members are not fixed yet. ''' def __init__(self, name=None): if not name: name = _cpu_name() self.name = name if name == 'x32': self.bits = 32 self.windows_subdir = '' self.windows_name = 'x86' self.windows_config = 'Win32' self.windows_suffix = '' elif name == 'x64': self.bits = 64 self.windows_subdir = 'x64/' self.windows_name = 'x64' self.windows_config = 'x64' self.windows_suffix = '64' elif name == 'arm32': # These are a guess. self.bits = 32 self.windows_subdir = 'arm/' self.windows_name = 'arm' self.windows_config = 'arm' self.windows_suffix = '' elif name == 'arm64': # These are a guess. self.bits = 64 self.windows_subdir = 'arm64/' self.windows_name = 'arm64' self.windows_config = 'arm64' self.windows_suffix = '64' else: assert 0, f'Unrecognised cpu name: {name}' def __repr__(self): return self.name class WindowsPython: ''' Windows only. Information about installed Python with specific word size and version. Defaults to the currently-running Python. Members: .path: Path of python binary. .version: `{major}.{minor}`, e.g. `3.9` or `3.11`. Same as `version` passed to `__init__()` if not None, otherwise the inferred version. .include: Python include path. .cpu: A `WindowsCpu` instance, same as `cpu` passed to `__init__()` if not None, otherwise the inferred cpu. We parse the output from `py -0p` to find all available python installations. ''' def __init__( self, cpu=None, version=None, verbose=True): ''' Args: cpu: A WindowsCpu instance. If None, we use whatever we are running on. version: Two-digit Python version as a string such as `3.8`. If None we use current Python's version. verbose: If true we show diagnostics. ''' if cpu is None: cpu = WindowsCpu(_cpu_name()) if version is None: version = '.'.join(platform.python_version().split('.')[:2]) _log(f'Looking for Python {version=} {cpu.bits=}.') if '.'.join(platform.python_version().split('.')[:2]) == version: # Current python matches, so use it directly. This avoids problems # on Github where experimental python-3.13 was not available via # `py`, and is kept here in case a similar problems happens with # future Python versions. _log(f'{cpu=} {version=}: using {sys.executable=}.') self.path = sys.executable self.version = version self.cpu = cpu self.include = sysconfig.get_path('include') else: command = 'py -0p' if verbose: _log(f'{cpu=} {version=}: Running: {command}') text = subprocess.check_output( command, shell=True, text=True) for line in text.split('\n'): #_log( f' {line}') if m := re.match( '^ *-V:([0-9.]+)(-32)? ([*])? +(.+)$', line): version2 = m.group(1) bits = 32 if m.group(2) else 64 path = m.group(4).strip() elif m := re.match( '^ *-([0-9.]+)-((32)|(64)) +(.+)$', line): version2 = m.group(1) bits = int(m.group(2)) path = m.group(5).strip() else: if verbose: _log( f'No match for {line=}') continue if verbose: _log( f'{version2=} {bits=} {path=} from {line=}.') if bits != cpu.bits or version2 != version: continue if not os.path.exists(path): # Sometimes it seems that the specified .../python.exe does not exist, # and we have to change it to .../python.exe. # assert path.endswith('.exe'), f'path={path!r}' path2 = f'{path[:-4]}{version}.exe' _log( f'Python {path!r} does not exist; changed to: {path2!r}') assert os.path.exists( path2) path = path2 self.path = path self.version = version self.cpu = cpu command = f'{self.path} -c "import sysconfig; print(sysconfig.get_path(\'include\'))"' _log(f'Finding Python include path by running {command=}.') self.include = subprocess.check_output(command, shell=True, text=True).strip() _log(f'Python include path is {self.include=}.') #_log( f'pipcl.py:WindowsPython():\n{self.description_ml(" ")}') break else: _log(f'Failed to find python matching cpu={cpu}.') _log(f'Output from {command!r} was:\n{text}') raise Exception( f'Failed to find python matching cpu={cpu} {version=}.') # Oddly there doesn't seem to be a # `sysconfig.get_path('libs')`, but it seems to be next # to `includes`: self.libs = os.path.abspath(f'{self.include}/../libs') _log( f'WindowsPython:\n{self.description_ml(" ")}') def description_ml(self, indent=''): ret = textwrap.dedent(f''' path: {self.path} version: {self.version} cpu: {self.cpu} include: {self.include} libs: {self.libs} ''') return textwrap.indent( ret, indent) def __repr__(self): return f'path={self.path!r} version={self.version!r} cpu={self.cpu!r} include={self.include!r} libs={self.libs!r}' # Internal helpers. # def _cpu_name(): ''' Returns `arm32`, `arm64`, `x32` or `x64` depending on Python build. ''' #log(f'sys.maxsize={hex(sys.maxsize)}') # bits = int.bit_length(sys.maxsize+1) if platform.machine().startswith('ARM'): return f'arm{bits}' else: return f'x{bits}' def _log(text='', caller=1): ''' Logs lines with prefix. ''' pipcl.log1(text, caller+1) ArtifexSoftware-pipcl-04f69b2/COPYING0000664000175000017500000010333015245537643017251 0ustar epsilonepsilon GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software. A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public. The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version. An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU Affero General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Remote Network Interaction; Use with the GNU General Public License. Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source. For example, if your program is a web application, its interface could display a "Source" link that leads users to an archive of the code. There are many ways you could offer source, and different solutions will be better for different programs; see section 13 for the specific requirements. You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU AGPL, see .