././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0895154 galaxy_tool_util-26.1.1/0000755000175100017510000000000015234436377014644 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/HISTORY.rst0000644000175100017510000021542715234436334016543 0ustar00runnerrunnerHistory ------- .. to_doc ------------------- 26.1.1 (2026-08-04) ------------------- No recorded changes since last release ------------------- 26.1.0 (2026-08-02) ------------------- ========= Bug fixes ========= * Avoid missing field value failure in data tables by `@guerler `_ in `#21008 `_ * Escape \`$\` in galaxy.xsd to fix output filter docs rendering by `@gdefazio `_ in `#21731 `_ * Fix ToolExpressionOutput.to_model() for boolean type by `@jmchilton `_ in `#22007 `_ * Convert math.inf to None in JSON-serialized tool dicts by `@dannon `_ in `#22152 `_ * Fix more paired_or_unpaired collection issues by `@jmchilton `_ in `#22170 `_ * Fix mypy and packaging issues; pin rocrate<0.15.0 by `@nsoranzo `_ in `#22469 `_ * Fix tool id handling for shed-installed tools by `@guerler `_ in `#22553 `_ * Don't install data table entries for non-DM tool installs by `@natefoo `_ in `#22679 `_ * Harden access control for user-defined tools by `@mvdbeek `_ in `#22704 `_ * [26.1 (spiritually)] Fix UDT collection outputs being silently dropped in YAML tool parser by `@jmchilton `_ in `#22761 `_ * Relax legacy doi:-prefixed citation handling (#22795) by `@mvdbeek `_ in `#22799 `_ * Add fixed color set from webcolors package by `@guerler `_ in `#22869 `_ * Fix input mangling when conditional parameter name ends in \`_N\` by `@mvdbeek `_ in `#22891 `_ * Always attempt to build tool request for new API endpoint by `@mvdbeek `_ in `#22894 `_ * Accept dce collection references in the tool request model by `@mvdbeek `_ in `#22926 `_ * Fix tool shed state serving for stock tools by `@jmchilton `_ in `#22963 `_ * Make the user-defined tool schema robust for LLM authoring (CustomToolAgent) by `@mvdbeek `_ in `#22968 `_ * Custom-tool agent: verified container resolution + authoring safeguards (with a reusable mulled-recommend CLI) by `@mvdbeek `_ in `#22981 `_ * Fix celery worker cold-start thundering herd (build app once, scope tool-data walk to load pass) by `@mvdbeek `_ in `#23048 `_ * Fix ToolPanelElements._section_by_tool being class-level shared state by `@mvdbeek `_ in `#23062 `_ * Skip RST help linting when help format is not restructuredtext by `@mvdbeek `_ in `#23073 `_ * Make verify_collection fail usefully on output-shape mismatches by `@mvdbeek `_ in `#23134 `_ * Handle sample_sheet and paired_or_unpaired collections in output_to_cwl_json by `@mvdbeek `_ in `#23152 `_ ============ Enhancements ============ * Add support for test credentials in tools by `@arash77 `_ in `#21643 `_ * Various YAML Tool Hardening and Progress toward Tool State Goals by `@jmchilton `_ in `#21828 `_ * Migrate tool execution request from api/tools to api/jobs by `@guerler `_ in `#21842 `_ * Add format_xml to galaxy-tool-util for shared XML formatting by `@dannon `_ in `#21903 `_ * Converge framework workflow test syntax toward Planemo syntax by `@jmchilton `_ in `#21907 `_ * Allow tools to specify time limits by `@jmchilton `_ in `#21927 `_ * Eliminate Twill from Tool Shed tests by `@jmchilton `_ in `#21953 `_ * Use \`\`os.path.join()\`\` or \`\`pathlib.Path\`\` instead of f-strings for filesystem paths by `@nsoranzo `_ in `#21954 `_ * Migrate Python packages to \`src\` layout and pure namespace packages by `@mr-c `_ in `#21977 `_ * Model subcollection mapping (map_over_type) and DCE in tool parameter schema by `@jmchilton `_ in `#21991 `_ * Enrich tool parameter JSON Schema with Galaxy metadata via model methods by `@mvdbeek `_ in `#22020 `_ * Replace mocked agent tests with static YAML backend for deterministic API/E2E testing by `@jmchilton `_ in `#22070 `_ * Extend the favorite tool panel concept by `@bgruening `_ in `#22212 `_ * Substantially expand tool form testing by `@jmchilton `_ in `#22216 `_ * Restore the #22116 fixes that only apply to dev by `@jmchilton `_ in `#22218 `_ * More Complete, Robust, and Tested JSON-Schema for Tool State Models by `@jmchilton `_ in `#22362 `_ * Clean up Pydantic v2 deprecation warnings by `@mvdbeek `_ in `#22463 `_ * Clean up remaining deprecation warnings by `@mvdbeek `_ in `#22473 `_ * Narrow YAML schema by `@jmchilton `_ in `#22507 `_ * Update Python dependencies by `@galaxybot `_ in `#22559 `_ * Tighten the workflow test schema by `@jmchilton `_ in `#22566 `_ * Add \`\`RetrySession\`\` to automatically retry idempotent HTTP requests by `@nsoranzo `_ in `#22567 `_ * Push UserToolSource semantic validation onto the pydantic model by `@mvdbeek `_ in `#22615 `_ * Update Python dependencies by `@galaxybot `_ in `#22619 `_ * Expose more metadata in ParsedTool abstraction by `@jmchilton `_ in `#22667 `_ * Add namespace stripping to \`\`CachedExplicitSingularityContainerResolver\`\` by `@mvdbeek `_ in `#22878 `_ * Enforce color format in JSON schema tests via FormatChecker by `@mvdbeek `_ in `#22909 `_ ============= Other changes ============= * Bump pydantic-ai from 1.44.0 to 1.56.0 in /lib/galaxy/dependencies by `@dependabot `_ in `#21782 `_ * Backport architecture-aware mulled builds by `@paperbenni `_ in `#23070 `_ ------------------- 26.0.1 (2026-06-04) ------------------- ========= Bug fixes ========= * Fix testing, publishing and dependencies of packages by `@nsoranzo `_ in `#22445 `_ * Accept bare ftype as valid test output check by `@jmchilton `_ in `#22483 `_ * Fix data manager URL for managers with '+' version suffix by `@mvdbeek `_ in `#22525 `_ ------------------- 26.0.0 (2026-04-08) ------------------- ========= Bug fixes ========= * When getting tool by uuid, get that exact tool by `@mvdbeek `_ in `#21844 `_ * Fix Pydantic UnsupportedFieldAttributeWarning for Field defaults in Annotated by `@mvdbeek `_ in `#21923 `_ * Backport paired or unpaired mapping fix by `@mvdbeek `_ in `#21936 `_ * Fix ToolExpressionOutput.to_model() for boolean type. by `@jmchilton `_ in `#22038 `_ * Model hidden_data params as optional data in tool meta-models. by `@jmchilton `_ in `#22116 `_ * Fix validation of certain classes of text validators in tools. by `@jmchilton `_ in `#22280 `_ ============ Enhancements ============ * Tool Request API by `@jmchilton `_ in `#20935 `_ * Remove Trackster by `@guerler `_ in `#20974 `_ * Add Playwright Backend Support to Galaxy Browser Automation Framework by `@jmchilton `_ in `#21102 `_ * Fix various bugs/hacks with tool_data_table handling. by `@jmchilton `_ in `#21163 `_ * Update Python dependencies by `@galaxybot `_ in `#21368 `_ * Enable attaching sample sheet to landing requests by `@mvdbeek `_ in `#21489 `_ * Tool testing: add min/max attributes to test output collections by `@bernt-matthias `_ in `#21495 `_ * Extend `image_diff` and image content assertions to handle boolean images by `@kostrykin `_ in `#21527 `_ * Apply 2026 black style by `@galaxybot `_ in `#21618 `_ * Fix slow startup regression by `@ksuderman `_ in `#21637 `_ * Remove sanitization from built-in grep tool by `@wm75 `_ in `#21648 `_ * Update Python dependencies by `@galaxybot `_ in `#21653 `_ * Add linter to validate required_files include paths exist by `@mvdbeek `_ in `#22061 `_ ============= Other changes ============= * Fix tool request API for multiple select parameters with defaults. by `@jmchilton `_ in `#21416 `_ ------------------- 25.1.2 (2026-03-09) ------------------- ========= Bug fixes ========= * Properly parse build info from requirement info by `@bernt-matthias `_ in `#21481 `_ * Fix loading non-tool files from watched tool directories by `@mvdbeek `_ in `#21880 `_ ------------------- 25.1.1 (2026-02-03) ------------------- ========= Bug fixes ========= * Fix data manager .loc file selection logic by `@jdavcs `_ in `#21664 `_ ============ Enhancements ============ * Document security considerations for using secrets by `@bernt-matthias `_ in `#21654 `_ ============= Other changes ============= * Improve tempdir cleanup in integration tests by `@mvdbeek `_ in `#21520 `_ ------------------- 25.1.0 (2025-12-12) ------------------- ========= Bug fixes ========= * Check for ``format="input"`` as well as ``format="auto"`` in datatype linter by `@bernt-matthias `_ in `#20177 `_ * Test and fix CORS on exceptions by `@mvdbeek `_ in `#21105 `_ * Apply tool test timeout only once by `@bernt-matthias `_ in `#21246 `_ * Update the mulled.py script to check json output by `@nilchia `_ in `#21276 `_ * Use anaconda API to find package download URL by `@nsoranzo `_ in `#21360 `_ ============ Enhancements ============ * Support credentials(secrets/variables) in tool requirements by `@arash77 `_ in `#19084 `_ * Implement Sample Sheets by `@jmchilton `_ in `#19305 `_ * Prepare ``ToolBox.dynamic_tool_to_tool()`` for CWL formats by `@nsoranzo `_ in `#20364 `_ * Type annotation fixes for mypy 1.16.0 by `@nsoranzo `_ in `#20424 `_ * Remove deprecated tool document cache by `@nsoranzo `_ in `#20510 `_ * Implement Data Landing Requests by `@jmchilton `_ in `#20592 `_ * Clean up code from pyupgrade by `@nsoranzo `_ in `#20642 `_ * Add markdown hint to schema by `@bgruening `_ in `#20654 `_ * Add configfiles support and various enhancements for user defined tools by `@mvdbeek `_ in `#20761 `_ * Add support for hierarchical module systems in lmod + module dependency resolvers by `@t1mk1k `_ in `#20866 `_ * Small cleanup of tool execution code. by `@jmchilton `_ in `#20899 `_ * Add support for picking ``from_work_dir`` directory by `@mvdbeek `_ in `#20916 `_ * Allow addressing user defined tools in job config by `@mvdbeek `_ in `#20932 `_ * Wire up and test resource requirement via tpv by `@mvdbeek `_ in `#20936 `_ * Allow specifying a command for determining a docker host port by `@natefoo `_ in `#20953 `_ * Use workflow-style payload in data landing request by `@mvdbeek `_ in `#21107 `_ * Update tool profile version for credentials by `@davelopez `_ in `#21273 `_ ============= Other changes ============= * Merge `release_25.0` into `dev` by `@davelopez `_ in `#20484 `_ ------------------- 25.0.4 (2025-11-18) ------------------- ========= Bug fixes ========= * Fix has_size assertion by `@bernt-matthias `_ in `#21081 `_ * Fix workflow run form input restrictions by `@mvdbeek `_ in `#21095 `_ * Fix direct tool execution not using the latest version by `@jmchilton `_ in `#21240 `_ ------------------- 25.0.3 (2025-09-23) ------------------- ========= Bug fixes ========= * Fix collection element sorting in extended_metadata by `@mvdbeek `_ in `#20928 `_ ------------------- 25.0.2 (2025-08-13) ------------------- ========= Bug fixes ========= * Fix restricting user defined tool input datasets extensions by `@mvdbeek `_ in `#20545 `_ * Don't fail volume mount construction for tools without tool directory by `@mvdbeek `_ in `#20593 `_ * Input linter: add missing attribute to `sort_by` by `@bernt-matthias `_ in `#20693 `_ * Fix ``test_base_image_for_targets`` mulled test to use mzmine by `@nsoranzo `_ in `#20713 `_ ------------------- 25.0.1 (2025-06-20) ------------------- ========= Bug fixes ========= * Fix single data element identifier to be a regular string by `@mvdbeek `_ in `#20438 `_ * Relax validation of XML test assertion parsing by `@mvdbeek `_ in `#20511 `_ * Fix import of ``galaxy.tool_util.cwl`` module by `@nsoranzo `_ in `#20529 `_ ------------------- 25.0.0 (2025-06-18) ------------------- ========= Bug fixes ========= * Use ``resource_path()`` to access datatypes_conf.xml.sample as a package resource by `@nsoranzo `_ in `#19331 `_ * Alternative `format_source` fix by `@mvdbeek `_ in `#19395 `_ * Fix `test_mulled_build.py::test_mulled_build_files_cli` with `use_mamba=True` by `@kysrpex `_ in `#19545 `_ * Fix incorrect type in tool_util.deps and fix package structure. by `@jmchilton `_ in `#19702 `_ * Add linter argument to linter report function calls by `@bernt-matthias `_ in `#19708 `_ * Fix linting with markdown in tool help by `@bgruening `_ in `#20058 `_ * Yaml parser fixes part2 by `@mvdbeek `_ in `#20162 `_ * Fix conda_link to use platform.machine() for architecture detection by `@chrisagrams `_ in `#20381 `_ * Add job config variable for singularity `--contain` by `@bernt-matthias `_ in `#20400 `_ ============ Enhancements ============ * Dynamic options: add data table filter by `@bernt-matthias `_ in `#12941 `_ * Isolate singularity containers more thoroughly for better reproducibility. by `@rhpvorderman `_ in `#18628 `_ * Implement tool markdown reports. by `@jmchilton `_ in `#19054 `_ * Fix UP031 errors - Part 4 by `@nsoranzo `_ in `#19235 `_ * Misc fixes 202412 by `@nsoranzo `_ in `#19341 `_ * Type annotation fixes for mypy 1.14.0 by `@nsoranzo `_ in `#19372 `_ * Make conditional discriminators literals instead of generic string/bool by `@mvdbeek `_ in `#19374 `_ * Empower Users to Build More Kinds of Collections, More Intelligently by `@jmchilton `_ in `#19377 `_ * Set safe default extraction filter for tar archives by `@nsoranzo `_ in `#19406 `_ * Allow controlling strict channel priority in mulled-build by `@bernt-matthias `_ in `#19425 `_ * Document `$__user_name__` by `@bernt-matthias `_ in `#19433 `_ * Add User-Defined Tools by `@mvdbeek `_ in `#19434 `_ * Type annotations improvements by `@nsoranzo `_ in `#19442 `_ * Type annotation improvements by `@nsoranzo `_ in `#19485 `_ * Format code with black 25.1.0 by `@nsoranzo `_ in `#19625 `_ * Mention default values for truevalue and falsevalue by `@pvanheus `_ in `#19657 `_ * Refactor dependencies for tool output actions. by `@jmchilton `_ in `#19662 `_ * Define simple models for job messages. by `@jmchilton `_ in `#19688 `_ * Add forgotten linter test by `@bernt-matthias `_ in `#19690 `_ * Speedup mulled build test by `@bernt-matthias `_ in `#19712 `_ * ToolShed2 - Add more context when navigating between tools an repositories. by `@jmchilton `_ in `#19738 `_ * Improvements to package decomposition. by `@jmchilton `_ in `#19759 `_ * Rename tool "Citations" to "References" by `@nsoranzo `_ in `#19770 `_ * Allow overriding datatypes for planemo lint by `@selten `_ in `#19780 `_ * xsd: use CollectionType for collections in tests by `@bernt-matthias `_ in `#19802 `_ * Add basic support for icons in tools by `@davelopez `_ in `#19850 `_ * Improve type annotations of ``ModelPersistenceContext`` and derived classes by `@nsoranzo `_ in `#19852 `_ * Improve docs for output filters by `@bernt-matthias `_ in `#19904 `_ * Fix local import in ``__resolvers_dict`` by `@mvdbeek `_ in `#19944 `_ * Update Python dependencies by `@galaxybot `_ in `#19964 `_ * Implement dataset collection support in workflow landing requests by `@mvdbeek `_ in `#20004 `_ * Enable ``warn_redundant_casts`` mypy option and drop redundant casts by `@nsoranzo `_ in `#20008 `_ * Additional type hints for ``toolbox.get_tool`` / ``toolbox.has_tool`` by `@mvdbeek `_ in `#20150 `_ * Yaml parser fixes and improvements (part 1) by `@mvdbeek `_ in `#20158 `_ * Add more descriptions to custom tool source schema by `@mvdbeek `_ in `#20266 `_ ------------------- 24.2.4 (2025-06-17) ------------------- ========= Bug fixes ========= * Fix #19515 - invalid citation handling changed with 24.2. by `@jmchilton `_ in `#19716 `_ * Drop unused alembic-utils from galaxy-data package requirements by `@nsoranzo `_ in `#19896 `_ * Report TestCaseValidation as linter error for 24.2 and above by `@bernt-matthias `_ in `#19928 `_ * Fix various parameter validation issues. by `@jmchilton `_ in `#19949 `_ * Fix `mulled-search --destination quay`, add index reuse by `@natefoo `_ in `#20107 `_ * Fix ``DataCollectionParameterModel`` factory by `@mvdbeek `_ in `#20164 `_ * Fix ``mull_targets()`` with mamba 2.x by `@nsoranzo `_ in `#20416 `_ ------------------- 24.2.3 (2025-03-16) ------------------- ========= Bug fixes ========= * Fix parameter model constructions with leading underscores, fixes converter linting by `@mvdbeek `_ in `#19790 `_ ------------------- 24.2.2 (2025-03-08) ------------------- ========= Bug fixes ========= * add missing linter name argument to linter report function calls by `@bernt-matthias `_ in `#19707 `_ * Assertion linter fixes by `@jmchilton `_ in `#19715 `_ * Fix dataypes linter: allow auto for output tags by `@bernt-matthias `_ in `#19757 `_ ============ Enhancements ============ * Add bwa_mem2_index directory datatype, framework enhancements for testing directories by `@mvdbeek `_ in `#19694 `_ ------------------- 24.2.1 (2025-02-28) ------------------- ========= Bug fixes ========= * Fix mulled tests by `@nsoranzo `_ in `#19626 `_ * xsd: fix format by `@bernt-matthias `_ in `#19683 `_ * Fix validation of required floats specified as empty strings in tool XML. by `@jmchilton `_ in `#19701 `_ * Fix container building by `@bernt-matthias `_ in `#19714 `_ ------------------- 24.2.0 (2025-02-11) ------------------- ========= Bug fixes ========= * Fix some deprecations by `@nsoranzo `_ in `#18433 `_ * Fix handling of v2 hashes by `@bernt-matthias `_ in `#18522 `_ * Fixes for errors reported by mypy 1.11.0 by `@nsoranzo `_ in `#18608 `_ * Handle output parsing when tool is not available for tool-centric API. by `@jmchilton `_ in `#18643 `_ * Fix new flake8-bugbear B039 and mypy type-var errors by `@nsoranzo `_ in `#18755 `_ * Fix commas that shouldn't be in biotools_mappings.tsv by `@jmchilton `_ in `#18919 `_ * Allow modes `z` and `Z` for docker volumes by `@bernt-matthias `_ in `#18998 `_ * Add requirement/resource to schema docs by `@bernt-matthias `_ in `#19172 `_ * Fix mulled_hash by `@bernt-matthias `_ in `#19199 `_ * Outputs filter linter fixes by `@bernt-matthias `_ in `#19208 `_ * Replace mambaforge Docker image with miniforge3 by `@nsoranzo `_ in `#19327 `_ * Partial backport of #19331 by `@nsoranzo `_ in `#19342 `_ * Backport fix for ``test_singularity_container_test`` test by `@nsoranzo `_ in `#19349 `_ * Remove unnecessary quoting in conda package specification by `@nsoranzo `_ in `#19382 `_ * Fix timeout handling for planemo / galaxy-tool-util by `@mvdbeek `_ in `#19384 `_ * Fix for empty string in regex validator message. by `@jmchilton `_ in `#19387 `_ * `mulled-build-tool` fix for packages wo version by `@bernt-matthias `_ in `#19450 `_ * Fix parsing of validators with inf limits by `@mvdbeek `_ in `#19459 `_ * Remove print statement by `@bernt-matthias `_ in `#19469 `_ * Downgrade 'InteractiveTools are not enabled' to warning by `@mvdbeek `_ in `#19518 `_ ============ Enhancements ============ * Add linters for datatypes by `@bernt-matthias `_ in `#17600 `_ * Update Python dependencies by `@galaxybot `_ in `#18406 `_ * Stray small changes from structured_tool_state branch. by `@jmchilton `_ in `#18453 `_ * Add Tool-Centric APIs to the Tool Shed 2.0 by `@jmchilton `_ in `#18524 `_ * Mixed enhancements from CWL branch by `@nsoranzo `_ in `#18539 `_ * Implement Declarative Testing for Workflow Behaviors by `@jmchilton `_ in `#18542 `_ * Improvements to Tool Test Parsing by `@jmchilton `_ in `#18560 `_ * More tool test typing. by `@jmchilton `_ in `#18590 `_ * Parameter Model Improvements by `@jmchilton `_ in `#18641 `_ * Another round of parameter model improvements. by `@jmchilton `_ in `#18673 `_ * De-couple XML from tool interface for test collections. by `@jmchilton `_ in `#18676 `_ * Another round of parameter model enhancements and tool tests by `@jmchilton `_ in `#18678 `_ * Dynamic Models for Tool Test Validation by `@jmchilton `_ in `#18679 `_ * Improvements to help terms & tool help. by `@jmchilton `_ in `#18722 `_ * Implement tool upgrade assistant. by `@jmchilton `_ in `#18728 `_ * Improvements to parameter models for test case inputs by `@jmchilton `_ in `#18743 `_ * Remove `defaults` channel for conda usage by `@bernt-matthias `_ in `#18757 `_ * Modelling for Test Assertions by `@jmchilton `_ in `#18787 `_ * Workflow Landing Requests by `@jmchilton `_ in `#18807 `_ * Add docs on indexing for filter column by `@bernt-matthias `_ in `#18844 `_ * Implement Pydantic model for workflow test format. by `@jmchilton `_ in `#18884 `_ * Improvements to Planemo test format based on feedback/discussions. by `@jmchilton `_ in `#18889 `_ * Add support for arbitrarily ordered image axes in image content assertions by `@kostrykin `_ in `#18891 `_ * Remove some unused dynamic drill down options. by `@jmchilton `_ in `#18892 `_ * Type annotations and fixes by `@nsoranzo `_ in `#18911 `_ * Add filter null collection operation tool by `@mvdbeek `_ in `#18928 `_ * Assert that `data_column` parameters have a valid `data_ref` by `@bernt-matthias `_ in `#18949 `_ * More concise, readable tool execution testing. by `@jmchilton `_ in `#18977 `_ * Workflow landing improvements by `@mvdbeek `_ in `#18979 `_ * Add hint to the tool schema about runtime conditional changes by `@martenson `_ in `#18996 `_ * A variety of improvements around tool parameter modeling (second try) by `@jmchilton `_ in `#19027 `_ * Better logging around tool loading by `@jmchilton `_ in `#19029 `_ * Update Python dependencies by `@galaxybot `_ in `#19031 `_ * Integrate Tool Parameter Modeling into Linting (for Planemo) by `@jmchilton `_ in `#19073 `_ * Allow deferred datasets to behave as URIs by `@davelopez `_ in `#19077 `_ * Implement workflow parameter validators. by `@jmchilton `_ in `#19092 `_ * Enable specifying dataset hash in test jobs by `@mvdbeek `_ in `#19113 `_ * Add linter to check validity of output filters by `@bernt-matthias `_ in `#19114 `_ * Also use miniforge docker for container building by `@bernt-matthias `_ in `#19128 `_ * Fix default value handling for parameters connected to required parameters by `@mvdbeek `_ in `#19219 `_ ------------------- 24.1.4 (2024-12-11) ------------------- ========= Bug fixes ========= * Handle all requests error in ``ApiBiotoolsMetadataSource._raw_get_metadata`` by `@nsoranzo `_ in `#18510 `_ * xsd: allow `change_format` and `actions` also in statically defined collection elements, and break recursion by `@bernt-matthias `_ in `#18605 `_ * Remove defaults channel for conda usage by `@bernt-matthias `_ in `#18859 `_ * Don't check availability of shellescape by `@nsoranzo `_ in `#18954 `_ * Backport 2 CI fixes by `@nsoranzo `_ in `#18973 `_ * Disable locking when opening h5 files, add missing ``with`` by `@mvdbeek `_ in `#18976 `_ * Fix extra call to test_data_path that requires an admin key by `@jmchilton `_ in `#19011 `_ * flip default value for use_mamba to false by `@bgruening `_ in `#19295 `_ * Linter: allow dynamic option definition by from_url by `@bernt-matthias `_ in `#19300 `_ ============ Enhancements ============ * Make `default_panel_view` a `_by_host` option by `@natefoo `_ in `#18471 `_ ------------------- 24.1.3 (2024-10-25) ------------------- ========= Bug fixes ========= * Handle all requests error in ``ApiBiotoolsMetadataSource._raw_get_metadata`` by `@nsoranzo `_ in `#18510 `_ * xsd: allow `change_format` and `actions` also in statically defined collection elements, and break recursion by `@bernt-matthias `_ in `#18605 `_ * Remove defaults channel for conda usage by `@bernt-matthias `_ in `#18859 `_ * Don't check availability of shellescape by `@nsoranzo `_ in `#18954 `_ * Backport 2 CI fixes by `@nsoranzo `_ in `#18973 `_ * Disable locking when opening h5 files, add missing ``with`` by `@mvdbeek `_ in `#18976 `_ * Fix extra call to test_data_path that requires an admin key by `@jmchilton `_ in `#19011 `_ ============ Enhancements ============ * Make `default_panel_view` a `_by_host` option by `@natefoo `_ in `#18471 `_ ------------------- 24.1.2 (2024-09-25) ------------------- ========= Bug fixes ========= * Handle all requests error in ``ApiBiotoolsMetadataSource._raw_get_metadata`` by `@nsoranzo `_ in `#18510 `_ * xsd: allow `change_format` and `actions` also in statically defined collection elements, and break recursion by `@bernt-matthias `_ in `#18605 `_ ============ Enhancements ============ * Make `default_panel_view` a `_by_host` option by `@natefoo `_ in `#18471 `_ ------------------- 24.1.1 (2024-07-02) ------------------- ========= Bug fixes ========= * Fix bug in galaxy.xsd by `@kostrykin `_ in `#17752 `_ * Fix bug in `assert_has_image_n_labels` by `@kostrykin `_ in `#17754 `_ * Remove linter for unstripped text content for tool xml leaves by `@bernt-matthias `_ in `#18252 `_ ============ Enhancements ============ * Add test and doc showing how dynamic selects are used by `@bernt-matthias `_ in `#16885 `_ * Add content assertion XML tags for test output verification using images by `@kostrykin `_ in `#17581 `_ * Set minimal metadata also for empty bed datasets by `@wm75 `_ in `#17586 `_ * Automatically bind `galaxy_data_manager_data_path` in containers by `@bernt-matthias `_ in `#17596 `_ * Type annotation improvements by `@nsoranzo `_ in `#17601 `_ * Type annotation and CWL-related improvements by `@nsoranzo `_ in `#17630 `_ * Update Python dependencies by `@galaxybot `_ in `#17653 `_ * Code cleanups from ruff and pyupgrade by `@nsoranzo `_ in `#17654 `_ * Tool linter: check for valid bio.tools entries by `@bernt-matthias `_ in `#17655 `_ * Tool linter: check for leaf nodes with unstripped text content by `@bernt-matthias `_ in `#17656 `_ * Issue #17631: Make it possible to use custom invfile.lua if needed by `@martin-g `_ in `#17693 `_ * SQLAlchemy 2.0 by `@jdavcs `_ in `#17778 `_ * Add support for floating point TIFF files in verification of image-based tool outputs by `@kostrykin `_ in `#17797 `_ * Add tool linting for valid EDAM terms by `@bernt-matthias `_ in `#17839 `_ * Add `pin_labels` attribute for `image_diff` comparison method by `@kostrykin `_ in `#17866 `_ * Enable ``warn_unused_ignores`` mypy option by `@nsoranzo `_ in `#17991 `_ * Add galaxy to user agent by `@mvdbeek `_ in `#18003 `_ * Document syntax for accessing nested parameters in `change_format` - `when` tags by `@bernt-matthias `_ in `#18018 `_ * Update Python dependencies by `@galaxybot `_ in `#18063 `_ * Enable flake8-implicit-str-concat ruff rules by `@nsoranzo `_ in `#18067 `_ ------------------- 24.0.3 (2024-06-28) ------------------- ========= Bug fixes ========= * Allow purge query param, deprecate purge body param by `@mvdbeek `_ in `#18105 `_ * Make sure that all Linter subclasses are imported for listing them by `@bernt-matthias `_ in `#18339 `_ * Assign default ``data`` extension on discovered collection output by `@mvdbeek `_ in `#18389 `_ * Allow in_range validator for selects by `@bernt-matthias `_ in `#18403 `_ ------------------- 24.0.2 (2024-05-07) ------------------- ========= Bug fixes ========= * Tool linters: allow to skip by old linter names (by allowing to skip linter modules) by `@bernt-matthias `_ in `#18091 `_ * tool linters: output filters should only consider child filter nodes by `@bernt-matthias `_ in `#18096 `_ ------------------- 24.0.1 (2024-05-02) ------------------- ========= Bug fixes ========= * Fix conditional Image imports by `@mvdbeek `_ in `#17899 `_ ------------------- 24.0.0 (2024-04-02) ------------------- ========= Bug fixes ========= * xsd: reorder choices for permissive boolean by `@bernt-matthias `_ in `#17187 `_ * Allow for upper case container tags by `@bernt-matthias `_ in `#17254 `_ * Fixes for flake8-bugbear 24.1.17 by `@nsoranzo `_ in `#17340 `_ * Escape pipe character in tool XSD docs by `@neoformit `_ in `#17359 `_ * XSD schema doc building: quote pipe characters in attribute tables by `@bernt-matthias `_ in `#17364 `_ * Fix IUC best practices links, mention data_source_async in XSD by `@wm75 `_ in `#17409 `_ * Fix data_source and data_source_async bugs by `@wm75 `_ in `#17422 `_ * Add tool XML schema documention for outputs - collection - data by `@bernt-matthias `_ in `#17465 `_ * has_size assertion: implement size (as synonym for value) by `@bernt-matthias `_ in `#17490 `_ * Yaml nested assertions: fix parsing by `@bernt-matthias `_ in `#17641 `_ ============ Enhancements ============ * build_mulled: also use namespace for building singularity images by `@bernt-matthias `_ in `#15657 `_ * Python 3.8 as minimum by `@mr-c `_ in `#16954 `_ * Extend regex groups in stdio regex matches by `@bernt-matthias `_ in `#17016 `_ * Split linters in separate classes by `@bernt-matthias `_ in `#17081 `_ * Add select parameter with options from remote resources by `@mvdbeek `_ in `#17087 `_ * Replace discouraged Mambaforge with Miniforge3 by `@nsoranzo `_ in `#17177 `_ * Clarify the meaning of lexical sorting of discovered datasets by `@bernt-matthias `_ in `#17190 `_ * Migrate models to pydantic 2 by `@mvdbeek `_ in `#17262 `_ * Add element_identifier and ext to inputs config file export by `@bernt-matthias `_ in `#17357 `_ * Enable ``warn_unreachable`` mypy option by `@mvdbeek `_ in `#17365 `_ * Fix type annotation of code using XML etree by `@nsoranzo `_ in `#17367 `_ * Update to black 2024 stable style by `@nsoranzo `_ in `#17391 `_ * Some additional typing for test interactor stuff. by `@jmchilton `_ in `#17398 `_ * Allow using tool data bundles as inputs to reference data select parameters by `@mvdbeek `_ in `#17435 `_ * `data_column` parameter: use `column_names` metadata if present by `@bernt-matthias `_ in `#17478 `_ * Fixing data_source tools and incrementing tool profile by `@wm75 `_ in `#17515 `_ * Add `image_diff` comparison method for test output verification using images by `@kostrykin `_ in `#17556 `_ * add shm_size based on ShmSize by `@richard-burhans `_ in `#17565 `_ * Record missing outputs as test errors by `@bernt-matthias `_ in `#17874 `_ ============= Other changes ============= * consistently compare profile versions by `@bernt-matthias `_ in `#16492 `_ ------------------- 23.2.1 (2024-02-21) ------------------- ========= Bug fixes ========= * Never consider `_galaxy_` conda env as unused by `@bernt-matthias `_ in `#16460 `_ * chore: fix typos by `@afuetterer `_ in `#16851 `_ * Ruff and flake8 fixes by `@nsoranzo `_ in `#16884 `_ * Quote singularity env parameters by `@bernt-matthias `_ in `#17055 `_ * Remove duplicates when copying sections for tool panel view by `@bernt-matthias `_ in `#17117 `_ * Display application fixes and tests by `@mvdbeek `_ in `#17233 `_ ============ Enhancements ============ * Implement default locations for data and collection parameters. by `@jmchilton `_ in `#14955 `_ * Add framework test for profile behavior of `format="input"` by `@bernt-matthias `_ in `#15482 `_ * Tool Shed 2.0 by `@jmchilton `_ in `#15639 `_ * Add ability to assert metadata properties on input dataset parameters by `@bernt-matthias `_ in `#15825 `_ * Migrate a part of the users API to Fast API by `@heisner-tillman `_ in `#16341 `_ * Update Python dependencies by `@galaxybot `_ in `#16436 `_ * Tweak tool memory use and optimize shared memory when using preload by `@mvdbeek `_ in `#16536 `_ * Document that required text parameters need a validator by `@bernt-matthias `_ in `#16538 `_ * Include `regex` when linting validators by `@davelopez `_ in `#16684 `_ * Refactor Tool Panel views structures and combine ToolBox and ToolBoxWorkflow into one component by `@ahmedhamidawan `_ in `#16739 `_ * Replace file_name property with get_file_name function by `@SergeyYakubov `_ in `#16783 `_ * Updated path-based interactive tools with entry point path injection, support for ITs with relative links, shortened URLs, doc and config updates including Podman job_conf by `@sveinugu `_ in `#16795 `_ * Remove remaining legacy backbone form input elements by `@guerler `_ in `#16834 `_ * Change `api/tool_panel` to `api/tool_panels/...` by `@ahmedhamidawan `_ in `#16928 `_ * optimize object store cache operations by `@SergeyYakubov `_ in `#17025 `_ * Enhance xsd schema and allow simpler assertion lists by `@mvdbeek `_ in `#17134 `_ ============= Other changes ============= * Merge 23.1 into dev by `@mvdbeek `_ in `#16534 `_ * Explicitly document default of multiple by `@bernt-matthias `_ in `#16625 `_ ------------------- 23.1.4 (2024-01-04) ------------------- ========= Bug fixes ========= * Separate collection and non-collection data element by `@mvdbeek `_ in `#17236 `_ ------------------- 23.1.3 (2023-12-01) ------------------- No recorded changes since last release ------------------- 23.1.2 (2023-11-29) ------------------- ========= Bug fixes ========= * Change log level for duplicate data table entries to warning by `@mvdbeek `_ in `#16988 `_ * Upgrade minimum conda to be compatible with latest conda-build by `@bernt-matthias `_ in `#17013 `_ * Fix duplicated tools in tool panel view section copying by `@jmchilton `_ in `#17036 `_ ============ Enhancements ============ * Adds `biii` as supported xref reference type by `@kostrykin `_ in `#16952 `_ ------------------- 23.1.1 (2023-10-23) ------------------- ========= Bug fixes ========= * Fixes for two framework test tools by `@bernt-matthias `_ in `#15483 `_ * add missing f for f-string by `@bernt-matthias `_ in `#15584 `_ * Fix call to `docker_cached_container_description` by `@bernt-matthias `_ in `#15598 `_ * Fix log message by `@bernt-matthias `_ in `#15838 `_ * add required_files to the tag list for linting by `@bernt-matthias `_ in `#16129 `_ * Handle appending to a results file that does not exists. by `@ksuderman `_ in `#16233 `_ * Improve container resolver documentation by `@mvdbeek `_ in `#16280 `_ * Add missing singularity_no_mount prop parsing by `@mvdbeek `_ in `#16367 `_ * Restore resolution of Conda environments generated from non-lowercase package names by `@nsoranzo `_ in `#16474 `_ * Fix up unit tests for local use by `@mvdbeek `_ in `#16483 `_ * Fix `multiple` remote test data by `@davelopez `_ in `#16542 `_ * Don't use ``docker run`` --user flag on OSX by `@mvdbeek `_ in `#16573 `_ * Backport tool mem fixes by `@mvdbeek `_ in `#16601 `_ * xsd: allow name attribute of test collections by `@bernt-matthias `_ in `#16663 `_ * Fix short ids in tool panel views. by `@jmchilton `_ in `#16800 `_ * Fix tool panel views for versionless tool ids by `@mvdbeek `_ in `#16809 `_ ============ Enhancements ============ * Decompress history data for testing assertions by `@bernt-matthias `_ in `#15085 `_ * OIDC tokens by `@SergeyYakubov `_ in `#15300 `_ * Fix for new style conda packages by `@bernt-matthias `_ in `#15446 `_ * Move database access code out of tool_util by `@jdavcs `_ in `#15467 `_ * Protection against problematic boolean parameters. by `@jmchilton `_ in `#15493 `_ * Implement initial tool/wf test assertions module for JSON data. by `@jmchilton `_ in `#15494 `_ * Explore tool remote test data by `@davelopez `_ in `#15510 `_ * xsd: add multiple to the list of attributes for all parameter types deriving from select by `@bernt-matthias `_ in `#15534 `_ * Update Python dependencies by `@galaxybot `_ in `#15564 `_ * Container resolvers: add docs, typing and tests by `@bernt-matthias `_ in `#15614 `_ * Migrate to MyST-Parser for Markdown docs by `@nsoranzo `_ in `#15844 `_ * Enable per-destination ``container_resolver_config_file`` by `@bernt-matthias `_ in `#15884 `_ * Updated doc and tests for attribute value filter by `@tuncK `_ in `#15929 `_ * Make container builders use mamba by `@bernt-matthias `_ in `#15953 `_ * Code cleanups from ruff and pyupgrade by `@nsoranzo `_ in `#16035 `_ * Vendorise ``packaging.versions.LegacyVersion`` by `@nsoranzo `_ in `#16058 `_ * Merge ``Target`` class with ``CondaTarget`` by `@nsoranzo `_ in `#16181 `_ * Small test decorator improvements. by `@jmchilton `_ in `#16220 `_ * tool_util: switch to mambaforge on non-32bit; add arm64 support by `@mr-c `_ in `#16223 `_ * Fix tool remote test data by `@davelopez `_ in `#16311 `_ ============= Other changes ============= * Restore tmp mounting by `@mvdbeek `_ in `#16533 `_ ------------------- 23.0.6 (2023-10-23) ------------------- ========= Bug fixes ========= * Fixes for extra files handling and cached object stores by `@mvdbeek `_ in `#16595 `_ * Fix create/install commands for conda 23.9.0 by `@nsoranzo `_ in `#16831 `_ ------------------- 23.0.5 (2023-07-29) ------------------- ========= Bug fixes ========= * Allow duplicate labels in linter if outputs contain filters by `@bernt-matthias `_ in `#15933 `_ * Fix parsing tool metadata from bio.tools by `@kysrpex `_ in `#16449 `_ * Linter: fix regex for profile version by `@bernt-matthias `_ in `#16480 `_ * Adjust test_data_download method in GalaxyInteractorApi so an admin user is not required by `@simonbray `_ in `#16482 `_ ------------------- 23.0.4 (2023-06-30) ------------------- No recorded changes since last release ------------------- 23.0.3 (2023-06-26) ------------------- ========= Bug fixes ========= * xsd: add missing `sep` attribute for `has_n_columns` by `@bernt-matthias `_ in `#16262 `_ * Missing init prevents models.py being bundled into tool_util by `@nuwang `_ in `#16308 `_ ============ Enhancements ============ * When importing tool data bundles, use the first loc file for the matching table by `@natefoo `_ in `#16247 `_ ------------------- 23.0.2 (2023-06-13) ------------------- ========= Bug fixes ========= * Don't fail CWL tool parsing when Cheetah not installed by `@mvdbeek `_ in `#16219 `_ * Allow skipping ``expect_num_outputs`` when ``expect_failure`` is set in tool test by `@bernt-matthias `_ in `#16237 `_ ------------------- 23.0.1 (2023-06-08) ------------------- ========= Bug fixes ========= * Fix assertion linting to not fail on byte suffixes by `@bernt-matthias `_ in `#15873 `_ * Fix ``get_test_from_anaconda()`` and ``base_image_for_targets()`` functions by `@nsoranzo `_ in `#16125 `_ * Fix test search for mulled container hashes by `@mvdbeek `_ in `#16170 `_ ============ Enhancements ============ * Allow setting auto_decompress property in staging interface by `@mvdbeek `_ in `#16014 `_ ------------------- 22.1.5 (2022-11-14) ------------------- * Set test status to success on expected failure ------------------- 22.1.4 (2022-10-28) ------------------- * Add missing unittest_utils package to galaxy-tool-util ------------------- 22.1.3 (2022-10-27) ------------------- * Pin minimum pyopenssl version when installing Conda * Add ``--strict-channel-priority`` to conda create/install commands if using conda >=4.7.5 ------------------- 22.1.2 (2022-08-29) ------------------- * Fix lint context error level * Pin galaxy-util to >= 22.1 * Fix biocontainer resolution without beaker cache ------------------- 22.1.1 (2022-08-22) ------------------- * First release from the 22.01 branch of Galaxy ------------------- 21.9.2 (2021-11-23) ------------------- * Fix linting of ``multiple="true"`` select inputs. ------------------- 21.9.1 (2021-11-03) ------------------- * Fix tool linting. ------------------- 21.9.0 (2021-11-03) ------------------- * First release from the 21.09 branch of Galaxy. ------------------- 21.1.2 (2021-06-23) ------------------- ------------------- 21.1.1 (2021-05-21) ------------------- ------------------- 21.1.0 (2021-03-19) ------------------- * First release from the 21.01 branch of Galaxy. ------------------- 20.9.1 (2020-10-28) ------------------- * Bugfixes to work around & annotate expected tool test failures. ------------------- 20.9.0 (2020-10-28) ------------------- * First release from the 20.09 branch of Galaxy. ------------------------ 20.9.0.dev2 (2020-08-02) ------------------------ * First release from the 20.09 branch of Galaxy. ------------------- 20.5.0 (2020-07-04) ------------------- * First release from the 20.05 branch of Galaxy. ------------------- 20.1.0 (2020-07-04) ------------------- * First release from the 20.01 branch of Galaxy. ------------------- 19.9.1 (2019-12-28) ------------------- * Fix declared dependency problem with package. ------------------- 19.9.0 (2019-12-16) ------------------- * Initial import from dev branch of Galaxy during 19.09 development cycle. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871579.0 galaxy_tool_util-26.1.1/LICENSE0000644000175100017510000000306615234436333015646 0ustar00runnerrunnerCopyright (c) 2005-2026 Galaxy Contributors (see CONTRIBUTORS.md) Galaxy is provided from 2026-02-25 onwards entirely under the MIT License. Some icons found in Galaxy are from the Silk Icons set, available under the Creative Commons Attribution 2.5 License, from: http://www.famfamfam.com/lab/icons/silk/ Other images and documentation are licensed under the Creative Commons Attribution 3.0 (CC BY 3.0) License. See: http://creativecommons.org/licenses/by/3.0/ -------------------------------------------------------------------------------- MIT License Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/Makefile0000644000175100017510000000545315234436334016304 0ustar00runnerrunner# Location of virtualenv used for development. VENV?=.venv # Open resource on Mac OS X or Linux OPEN_RESOURCE=bash -c 'open $$0 || xdg-open $$0' # Source virtualenv to execute command (flake8, sphinx, twine, etc...) IN_VENV=if [ -f $(VENV)/bin/activate ]; then . $(VENV)/bin/activate; fi; UPSTREAM?=galaxyproject SOURCE_DIR?=galaxy BUILD_SCRIPTS_DIR=scripts DEV_RELEASE?=0 VERSION?=$(shell DEV_RELEASE=$(DEV_RELEASE) python $(BUILD_SCRIPTS_DIR)/print_version_for_release.py) PROJECT_NAME?=galaxy-$(shell basename $(CURDIR)) PROJECT_NAME:=$(subst _,-,$(PROJECT_NAME)) BRANCH?=$(shell git rev-parse --abbrev-ref HEAD) TEST_DIR?=tests DIST=dist TESTS?=$(SOURCE_DIR) $(TEST_DIR) .PHONY: clean-pyc clean-build docs clean help: @echo "clean - remove all build, test, coverage and Python artifacts" @echo "clean-build - remove build artifacts" @echo "clean-pyc - remove Python file artifacts" @echo "clean-test - remove test and coverage artifacts" @echo "setup-venv - setup a development virtualenv in current directory." @echo "lint-dist - twine check dist results, including validating README content" @echo "dist - package project for PyPI distribution" clean: clean-build clean-pyc clean-tests clean-build: rm -fr build/ rm -fr $(DIST)/ find . -name "*.egg-info" -type d -exec rm -rf {} + clean-pyc: find . -name '*.pyc' -exec rm -f {} + find . -name '*.pyo' -exec rm -f {} + find . -name '*~' -exec rm -f {} + find . -name '__pycache__' -exec rm -fr {} + clean-tests: rm -fr .tox/ setup-venv: uv sync --inexact --all-extras _test: uv run pytest $(TESTS) test: setup-venv _test _dist: uv build -o $(DIST) ls -l $(DIST) dist: clean _dist _setup-mypy-venv: setup-venv uv pip install -r ../../lib/galaxy/dependencies/pinned-typecheck-requirements.txt _mypy: uv run mypy . mypy: _setup-mypy-venv _mypy _setup-lint-venv: setup-venv uv pip install -r ../../lib/galaxy/dependencies/pinned-lint-requirements.txt _lint: uv run ruff check . lint: _setup-lint-venv _lint lint-dist: uvx --with-requirements dev-requirements.txt --from twine twine check $(DIST)/* # black doesn't actually work on symlinked files because they are outside # the current directory #_setup-format-venv: setup-venv # uv pip install isort black #_isort: # uv run isort --sp ../../.isort.cfg . #_black: # uv run black --config ../pyproject.toml . #format: _setup-format-venv _isort _black commit-version: $(IN_VENV) DEV_RELEASE=$(DEV_RELEASE) python $(BUILD_SCRIPTS_DIR)/commit_version.py $(VERSION) new-version: $(IN_VENV) DEV_RELEASE=$(DEV_RELEASE) python $(BUILD_SCRIPTS_DIR)/new_version.py $(VERSION) release-local: commit-version release-artifacts new-version push-release: # git push $(UPSTREAM) $(BRANCH) # git push upstream $(PROJECT_NAME)-$(VERSION) echo "Makefile doesn't manually push release." release: release-local push-release ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0886352 galaxy_tool_util-26.1.1/PKG-INFO0000644000175100017510000022235015234436377015745 0ustar00runnerrunnerMetadata-Version: 2.4 Name: galaxy-tool-util Version: 26.1.1 Summary: Galaxy tool and tool dependency utilities Home-page: https://github.com/galaxyproject/galaxy Author: Galaxy Project and Community Author-email: galaxy-committers@lists.galaxyproject.org License: MIT Keywords: Galaxy Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console Classifier: Intended Audience :: Developers Classifier: Natural Language :: English Classifier: Operating System :: POSIX Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: 3.13 Classifier: Programming Language :: Python :: 3.14 Classifier: Topic :: Software Development Classifier: Topic :: Software Development :: Code Generators Classifier: Topic :: Software Development :: Testing Requires-Python: >=3.8 Description-Content-Type: text/x-rst License-File: LICENSE Requires-Dist: galaxy-tool-util-models>=26.0 Requires-Dist: galaxy-util[image-util,template]>=22.1 Requires-Dist: conda-package-streaming Requires-Dist: jsonschema Requires-Dist: lxml!=4.2.2 Requires-Dist: MarkupSafe Requires-Dist: packaging Requires-Dist: pydantic>=2.7.4 Requires-Dist: PyYAML Requires-Dist: requests Requires-Dist: sortedcontainers Requires-Dist: typing-extensions Provides-Extra: cwl Requires-Dist: cwl-utils; extra == "cwl" Requires-Dist: cwltool>=3.1.20230624081518; extra == "cwl" Provides-Extra: mulled Requires-Dist: jinja2; extra == "mulled" Requires-Dist: Whoosh; extra == "mulled" Provides-Extra: edam Requires-Dist: edam-ontology; extra == "edam" Provides-Extra: extended-assertions Requires-Dist: numpy; extra == "extended-assertions" Requires-Dist: pysam; extra == "extended-assertions" Requires-Dist: pillow; extra == "extended-assertions" Requires-Dist: tifffile; extra == "extended-assertions" Provides-Extra: test Requires-Dist: Beaker; extra == "test" Requires-Dist: pytest; extra == "test" Requires-Dist: pytest-mock; extra == "test" Dynamic: license-file .. image:: https://badge.fury.io/py/galaxy-tool-util.svg :target: https://pypi.org/project/galaxy-tool-util/ Overview -------- The Galaxy_ tool utilities. * Code: https://github.com/galaxyproject/galaxy/tree/dev/packages/tool_util .. _Galaxy: http://galaxyproject.org/ History ------- .. to_doc ------------------- 26.1.1 (2026-08-04) ------------------- No recorded changes since last release ------------------- 26.1.0 (2026-08-02) ------------------- ========= Bug fixes ========= * Avoid missing field value failure in data tables by `@guerler `_ in `#21008 `_ * Escape \`$\` in galaxy.xsd to fix output filter docs rendering by `@gdefazio `_ in `#21731 `_ * Fix ToolExpressionOutput.to_model() for boolean type by `@jmchilton `_ in `#22007 `_ * Convert math.inf to None in JSON-serialized tool dicts by `@dannon `_ in `#22152 `_ * Fix more paired_or_unpaired collection issues by `@jmchilton `_ in `#22170 `_ * Fix mypy and packaging issues; pin rocrate<0.15.0 by `@nsoranzo `_ in `#22469 `_ * Fix tool id handling for shed-installed tools by `@guerler `_ in `#22553 `_ * Don't install data table entries for non-DM tool installs by `@natefoo `_ in `#22679 `_ * Harden access control for user-defined tools by `@mvdbeek `_ in `#22704 `_ * [26.1 (spiritually)] Fix UDT collection outputs being silently dropped in YAML tool parser by `@jmchilton `_ in `#22761 `_ * Relax legacy doi:-prefixed citation handling (#22795) by `@mvdbeek `_ in `#22799 `_ * Add fixed color set from webcolors package by `@guerler `_ in `#22869 `_ * Fix input mangling when conditional parameter name ends in \`_N\` by `@mvdbeek `_ in `#22891 `_ * Always attempt to build tool request for new API endpoint by `@mvdbeek `_ in `#22894 `_ * Accept dce collection references in the tool request model by `@mvdbeek `_ in `#22926 `_ * Fix tool shed state serving for stock tools by `@jmchilton `_ in `#22963 `_ * Make the user-defined tool schema robust for LLM authoring (CustomToolAgent) by `@mvdbeek `_ in `#22968 `_ * Custom-tool agent: verified container resolution + authoring safeguards (with a reusable mulled-recommend CLI) by `@mvdbeek `_ in `#22981 `_ * Fix celery worker cold-start thundering herd (build app once, scope tool-data walk to load pass) by `@mvdbeek `_ in `#23048 `_ * Fix ToolPanelElements._section_by_tool being class-level shared state by `@mvdbeek `_ in `#23062 `_ * Skip RST help linting when help format is not restructuredtext by `@mvdbeek `_ in `#23073 `_ * Make verify_collection fail usefully on output-shape mismatches by `@mvdbeek `_ in `#23134 `_ * Handle sample_sheet and paired_or_unpaired collections in output_to_cwl_json by `@mvdbeek `_ in `#23152 `_ ============ Enhancements ============ * Add support for test credentials in tools by `@arash77 `_ in `#21643 `_ * Various YAML Tool Hardening and Progress toward Tool State Goals by `@jmchilton `_ in `#21828 `_ * Migrate tool execution request from api/tools to api/jobs by `@guerler `_ in `#21842 `_ * Add format_xml to galaxy-tool-util for shared XML formatting by `@dannon `_ in `#21903 `_ * Converge framework workflow test syntax toward Planemo syntax by `@jmchilton `_ in `#21907 `_ * Allow tools to specify time limits by `@jmchilton `_ in `#21927 `_ * Eliminate Twill from Tool Shed tests by `@jmchilton `_ in `#21953 `_ * Use \`\`os.path.join()\`\` or \`\`pathlib.Path\`\` instead of f-strings for filesystem paths by `@nsoranzo `_ in `#21954 `_ * Migrate Python packages to \`src\` layout and pure namespace packages by `@mr-c `_ in `#21977 `_ * Model subcollection mapping (map_over_type) and DCE in tool parameter schema by `@jmchilton `_ in `#21991 `_ * Enrich tool parameter JSON Schema with Galaxy metadata via model methods by `@mvdbeek `_ in `#22020 `_ * Replace mocked agent tests with static YAML backend for deterministic API/E2E testing by `@jmchilton `_ in `#22070 `_ * Extend the favorite tool panel concept by `@bgruening `_ in `#22212 `_ * Substantially expand tool form testing by `@jmchilton `_ in `#22216 `_ * Restore the #22116 fixes that only apply to dev by `@jmchilton `_ in `#22218 `_ * More Complete, Robust, and Tested JSON-Schema for Tool State Models by `@jmchilton `_ in `#22362 `_ * Clean up Pydantic v2 deprecation warnings by `@mvdbeek `_ in `#22463 `_ * Clean up remaining deprecation warnings by `@mvdbeek `_ in `#22473 `_ * Narrow YAML schema by `@jmchilton `_ in `#22507 `_ * Update Python dependencies by `@galaxybot `_ in `#22559 `_ * Tighten the workflow test schema by `@jmchilton `_ in `#22566 `_ * Add \`\`RetrySession\`\` to automatically retry idempotent HTTP requests by `@nsoranzo `_ in `#22567 `_ * Push UserToolSource semantic validation onto the pydantic model by `@mvdbeek `_ in `#22615 `_ * Update Python dependencies by `@galaxybot `_ in `#22619 `_ * Expose more metadata in ParsedTool abstraction by `@jmchilton `_ in `#22667 `_ * Add namespace stripping to \`\`CachedExplicitSingularityContainerResolver\`\` by `@mvdbeek `_ in `#22878 `_ * Enforce color format in JSON schema tests via FormatChecker by `@mvdbeek `_ in `#22909 `_ ============= Other changes ============= * Bump pydantic-ai from 1.44.0 to 1.56.0 in /lib/galaxy/dependencies by `@dependabot `_ in `#21782 `_ * Backport architecture-aware mulled builds by `@paperbenni `_ in `#23070 `_ ------------------- 26.0.1 (2026-06-04) ------------------- ========= Bug fixes ========= * Fix testing, publishing and dependencies of packages by `@nsoranzo `_ in `#22445 `_ * Accept bare ftype as valid test output check by `@jmchilton `_ in `#22483 `_ * Fix data manager URL for managers with '+' version suffix by `@mvdbeek `_ in `#22525 `_ ------------------- 26.0.0 (2026-04-08) ------------------- ========= Bug fixes ========= * When getting tool by uuid, get that exact tool by `@mvdbeek `_ in `#21844 `_ * Fix Pydantic UnsupportedFieldAttributeWarning for Field defaults in Annotated by `@mvdbeek `_ in `#21923 `_ * Backport paired or unpaired mapping fix by `@mvdbeek `_ in `#21936 `_ * Fix ToolExpressionOutput.to_model() for boolean type. by `@jmchilton `_ in `#22038 `_ * Model hidden_data params as optional data in tool meta-models. by `@jmchilton `_ in `#22116 `_ * Fix validation of certain classes of text validators in tools. by `@jmchilton `_ in `#22280 `_ ============ Enhancements ============ * Tool Request API by `@jmchilton `_ in `#20935 `_ * Remove Trackster by `@guerler `_ in `#20974 `_ * Add Playwright Backend Support to Galaxy Browser Automation Framework by `@jmchilton `_ in `#21102 `_ * Fix various bugs/hacks with tool_data_table handling. by `@jmchilton `_ in `#21163 `_ * Update Python dependencies by `@galaxybot `_ in `#21368 `_ * Enable attaching sample sheet to landing requests by `@mvdbeek `_ in `#21489 `_ * Tool testing: add min/max attributes to test output collections by `@bernt-matthias `_ in `#21495 `_ * Extend `image_diff` and image content assertions to handle boolean images by `@kostrykin `_ in `#21527 `_ * Apply 2026 black style by `@galaxybot `_ in `#21618 `_ * Fix slow startup regression by `@ksuderman `_ in `#21637 `_ * Remove sanitization from built-in grep tool by `@wm75 `_ in `#21648 `_ * Update Python dependencies by `@galaxybot `_ in `#21653 `_ * Add linter to validate required_files include paths exist by `@mvdbeek `_ in `#22061 `_ ============= Other changes ============= * Fix tool request API for multiple select parameters with defaults. by `@jmchilton `_ in `#21416 `_ ------------------- 25.1.2 (2026-03-09) ------------------- ========= Bug fixes ========= * Properly parse build info from requirement info by `@bernt-matthias `_ in `#21481 `_ * Fix loading non-tool files from watched tool directories by `@mvdbeek `_ in `#21880 `_ ------------------- 25.1.1 (2026-02-03) ------------------- ========= Bug fixes ========= * Fix data manager .loc file selection logic by `@jdavcs `_ in `#21664 `_ ============ Enhancements ============ * Document security considerations for using secrets by `@bernt-matthias `_ in `#21654 `_ ============= Other changes ============= * Improve tempdir cleanup in integration tests by `@mvdbeek `_ in `#21520 `_ ------------------- 25.1.0 (2025-12-12) ------------------- ========= Bug fixes ========= * Check for ``format="input"`` as well as ``format="auto"`` in datatype linter by `@bernt-matthias `_ in `#20177 `_ * Test and fix CORS on exceptions by `@mvdbeek `_ in `#21105 `_ * Apply tool test timeout only once by `@bernt-matthias `_ in `#21246 `_ * Update the mulled.py script to check json output by `@nilchia `_ in `#21276 `_ * Use anaconda API to find package download URL by `@nsoranzo `_ in `#21360 `_ ============ Enhancements ============ * Support credentials(secrets/variables) in tool requirements by `@arash77 `_ in `#19084 `_ * Implement Sample Sheets by `@jmchilton `_ in `#19305 `_ * Prepare ``ToolBox.dynamic_tool_to_tool()`` for CWL formats by `@nsoranzo `_ in `#20364 `_ * Type annotation fixes for mypy 1.16.0 by `@nsoranzo `_ in `#20424 `_ * Remove deprecated tool document cache by `@nsoranzo `_ in `#20510 `_ * Implement Data Landing Requests by `@jmchilton `_ in `#20592 `_ * Clean up code from pyupgrade by `@nsoranzo `_ in `#20642 `_ * Add markdown hint to schema by `@bgruening `_ in `#20654 `_ * Add configfiles support and various enhancements for user defined tools by `@mvdbeek `_ in `#20761 `_ * Add support for hierarchical module systems in lmod + module dependency resolvers by `@t1mk1k `_ in `#20866 `_ * Small cleanup of tool execution code. by `@jmchilton `_ in `#20899 `_ * Add support for picking ``from_work_dir`` directory by `@mvdbeek `_ in `#20916 `_ * Allow addressing user defined tools in job config by `@mvdbeek `_ in `#20932 `_ * Wire up and test resource requirement via tpv by `@mvdbeek `_ in `#20936 `_ * Allow specifying a command for determining a docker host port by `@natefoo `_ in `#20953 `_ * Use workflow-style payload in data landing request by `@mvdbeek `_ in `#21107 `_ * Update tool profile version for credentials by `@davelopez `_ in `#21273 `_ ============= Other changes ============= * Merge `release_25.0` into `dev` by `@davelopez `_ in `#20484 `_ ------------------- 25.0.4 (2025-11-18) ------------------- ========= Bug fixes ========= * Fix has_size assertion by `@bernt-matthias `_ in `#21081 `_ * Fix workflow run form input restrictions by `@mvdbeek `_ in `#21095 `_ * Fix direct tool execution not using the latest version by `@jmchilton `_ in `#21240 `_ ------------------- 25.0.3 (2025-09-23) ------------------- ========= Bug fixes ========= * Fix collection element sorting in extended_metadata by `@mvdbeek `_ in `#20928 `_ ------------------- 25.0.2 (2025-08-13) ------------------- ========= Bug fixes ========= * Fix restricting user defined tool input datasets extensions by `@mvdbeek `_ in `#20545 `_ * Don't fail volume mount construction for tools without tool directory by `@mvdbeek `_ in `#20593 `_ * Input linter: add missing attribute to `sort_by` by `@bernt-matthias `_ in `#20693 `_ * Fix ``test_base_image_for_targets`` mulled test to use mzmine by `@nsoranzo `_ in `#20713 `_ ------------------- 25.0.1 (2025-06-20) ------------------- ========= Bug fixes ========= * Fix single data element identifier to be a regular string by `@mvdbeek `_ in `#20438 `_ * Relax validation of XML test assertion parsing by `@mvdbeek `_ in `#20511 `_ * Fix import of ``galaxy.tool_util.cwl`` module by `@nsoranzo `_ in `#20529 `_ ------------------- 25.0.0 (2025-06-18) ------------------- ========= Bug fixes ========= * Use ``resource_path()`` to access datatypes_conf.xml.sample as a package resource by `@nsoranzo `_ in `#19331 `_ * Alternative `format_source` fix by `@mvdbeek `_ in `#19395 `_ * Fix `test_mulled_build.py::test_mulled_build_files_cli` with `use_mamba=True` by `@kysrpex `_ in `#19545 `_ * Fix incorrect type in tool_util.deps and fix package structure. by `@jmchilton `_ in `#19702 `_ * Add linter argument to linter report function calls by `@bernt-matthias `_ in `#19708 `_ * Fix linting with markdown in tool help by `@bgruening `_ in `#20058 `_ * Yaml parser fixes part2 by `@mvdbeek `_ in `#20162 `_ * Fix conda_link to use platform.machine() for architecture detection by `@chrisagrams `_ in `#20381 `_ * Add job config variable for singularity `--contain` by `@bernt-matthias `_ in `#20400 `_ ============ Enhancements ============ * Dynamic options: add data table filter by `@bernt-matthias `_ in `#12941 `_ * Isolate singularity containers more thoroughly for better reproducibility. by `@rhpvorderman `_ in `#18628 `_ * Implement tool markdown reports. by `@jmchilton `_ in `#19054 `_ * Fix UP031 errors - Part 4 by `@nsoranzo `_ in `#19235 `_ * Misc fixes 202412 by `@nsoranzo `_ in `#19341 `_ * Type annotation fixes for mypy 1.14.0 by `@nsoranzo `_ in `#19372 `_ * Make conditional discriminators literals instead of generic string/bool by `@mvdbeek `_ in `#19374 `_ * Empower Users to Build More Kinds of Collections, More Intelligently by `@jmchilton `_ in `#19377 `_ * Set safe default extraction filter for tar archives by `@nsoranzo `_ in `#19406 `_ * Allow controlling strict channel priority in mulled-build by `@bernt-matthias `_ in `#19425 `_ * Document `$__user_name__` by `@bernt-matthias `_ in `#19433 `_ * Add User-Defined Tools by `@mvdbeek `_ in `#19434 `_ * Type annotations improvements by `@nsoranzo `_ in `#19442 `_ * Type annotation improvements by `@nsoranzo `_ in `#19485 `_ * Format code with black 25.1.0 by `@nsoranzo `_ in `#19625 `_ * Mention default values for truevalue and falsevalue by `@pvanheus `_ in `#19657 `_ * Refactor dependencies for tool output actions. by `@jmchilton `_ in `#19662 `_ * Define simple models for job messages. by `@jmchilton `_ in `#19688 `_ * Add forgotten linter test by `@bernt-matthias `_ in `#19690 `_ * Speedup mulled build test by `@bernt-matthias `_ in `#19712 `_ * ToolShed2 - Add more context when navigating between tools an repositories. by `@jmchilton `_ in `#19738 `_ * Improvements to package decomposition. by `@jmchilton `_ in `#19759 `_ * Rename tool "Citations" to "References" by `@nsoranzo `_ in `#19770 `_ * Allow overriding datatypes for planemo lint by `@selten `_ in `#19780 `_ * xsd: use CollectionType for collections in tests by `@bernt-matthias `_ in `#19802 `_ * Add basic support for icons in tools by `@davelopez `_ in `#19850 `_ * Improve type annotations of ``ModelPersistenceContext`` and derived classes by `@nsoranzo `_ in `#19852 `_ * Improve docs for output filters by `@bernt-matthias `_ in `#19904 `_ * Fix local import in ``__resolvers_dict`` by `@mvdbeek `_ in `#19944 `_ * Update Python dependencies by `@galaxybot `_ in `#19964 `_ * Implement dataset collection support in workflow landing requests by `@mvdbeek `_ in `#20004 `_ * Enable ``warn_redundant_casts`` mypy option and drop redundant casts by `@nsoranzo `_ in `#20008 `_ * Additional type hints for ``toolbox.get_tool`` / ``toolbox.has_tool`` by `@mvdbeek `_ in `#20150 `_ * Yaml parser fixes and improvements (part 1) by `@mvdbeek `_ in `#20158 `_ * Add more descriptions to custom tool source schema by `@mvdbeek `_ in `#20266 `_ ------------------- 24.2.4 (2025-06-17) ------------------- ========= Bug fixes ========= * Fix #19515 - invalid citation handling changed with 24.2. by `@jmchilton `_ in `#19716 `_ * Drop unused alembic-utils from galaxy-data package requirements by `@nsoranzo `_ in `#19896 `_ * Report TestCaseValidation as linter error for 24.2 and above by `@bernt-matthias `_ in `#19928 `_ * Fix various parameter validation issues. by `@jmchilton `_ in `#19949 `_ * Fix `mulled-search --destination quay`, add index reuse by `@natefoo `_ in `#20107 `_ * Fix ``DataCollectionParameterModel`` factory by `@mvdbeek `_ in `#20164 `_ * Fix ``mull_targets()`` with mamba 2.x by `@nsoranzo `_ in `#20416 `_ ------------------- 24.2.3 (2025-03-16) ------------------- ========= Bug fixes ========= * Fix parameter model constructions with leading underscores, fixes converter linting by `@mvdbeek `_ in `#19790 `_ ------------------- 24.2.2 (2025-03-08) ------------------- ========= Bug fixes ========= * add missing linter name argument to linter report function calls by `@bernt-matthias `_ in `#19707 `_ * Assertion linter fixes by `@jmchilton `_ in `#19715 `_ * Fix dataypes linter: allow auto for output tags by `@bernt-matthias `_ in `#19757 `_ ============ Enhancements ============ * Add bwa_mem2_index directory datatype, framework enhancements for testing directories by `@mvdbeek `_ in `#19694 `_ ------------------- 24.2.1 (2025-02-28) ------------------- ========= Bug fixes ========= * Fix mulled tests by `@nsoranzo `_ in `#19626 `_ * xsd: fix format by `@bernt-matthias `_ in `#19683 `_ * Fix validation of required floats specified as empty strings in tool XML. by `@jmchilton `_ in `#19701 `_ * Fix container building by `@bernt-matthias `_ in `#19714 `_ ------------------- 24.2.0 (2025-02-11) ------------------- ========= Bug fixes ========= * Fix some deprecations by `@nsoranzo `_ in `#18433 `_ * Fix handling of v2 hashes by `@bernt-matthias `_ in `#18522 `_ * Fixes for errors reported by mypy 1.11.0 by `@nsoranzo `_ in `#18608 `_ * Handle output parsing when tool is not available for tool-centric API. by `@jmchilton `_ in `#18643 `_ * Fix new flake8-bugbear B039 and mypy type-var errors by `@nsoranzo `_ in `#18755 `_ * Fix commas that shouldn't be in biotools_mappings.tsv by `@jmchilton `_ in `#18919 `_ * Allow modes `z` and `Z` for docker volumes by `@bernt-matthias `_ in `#18998 `_ * Add requirement/resource to schema docs by `@bernt-matthias `_ in `#19172 `_ * Fix mulled_hash by `@bernt-matthias `_ in `#19199 `_ * Outputs filter linter fixes by `@bernt-matthias `_ in `#19208 `_ * Replace mambaforge Docker image with miniforge3 by `@nsoranzo `_ in `#19327 `_ * Partial backport of #19331 by `@nsoranzo `_ in `#19342 `_ * Backport fix for ``test_singularity_container_test`` test by `@nsoranzo `_ in `#19349 `_ * Remove unnecessary quoting in conda package specification by `@nsoranzo `_ in `#19382 `_ * Fix timeout handling for planemo / galaxy-tool-util by `@mvdbeek `_ in `#19384 `_ * Fix for empty string in regex validator message. by `@jmchilton `_ in `#19387 `_ * `mulled-build-tool` fix for packages wo version by `@bernt-matthias `_ in `#19450 `_ * Fix parsing of validators with inf limits by `@mvdbeek `_ in `#19459 `_ * Remove print statement by `@bernt-matthias `_ in `#19469 `_ * Downgrade 'InteractiveTools are not enabled' to warning by `@mvdbeek `_ in `#19518 `_ ============ Enhancements ============ * Add linters for datatypes by `@bernt-matthias `_ in `#17600 `_ * Update Python dependencies by `@galaxybot `_ in `#18406 `_ * Stray small changes from structured_tool_state branch. by `@jmchilton `_ in `#18453 `_ * Add Tool-Centric APIs to the Tool Shed 2.0 by `@jmchilton `_ in `#18524 `_ * Mixed enhancements from CWL branch by `@nsoranzo `_ in `#18539 `_ * Implement Declarative Testing for Workflow Behaviors by `@jmchilton `_ in `#18542 `_ * Improvements to Tool Test Parsing by `@jmchilton `_ in `#18560 `_ * More tool test typing. by `@jmchilton `_ in `#18590 `_ * Parameter Model Improvements by `@jmchilton `_ in `#18641 `_ * Another round of parameter model improvements. by `@jmchilton `_ in `#18673 `_ * De-couple XML from tool interface for test collections. by `@jmchilton `_ in `#18676 `_ * Another round of parameter model enhancements and tool tests by `@jmchilton `_ in `#18678 `_ * Dynamic Models for Tool Test Validation by `@jmchilton `_ in `#18679 `_ * Improvements to help terms & tool help. by `@jmchilton `_ in `#18722 `_ * Implement tool upgrade assistant. by `@jmchilton `_ in `#18728 `_ * Improvements to parameter models for test case inputs by `@jmchilton `_ in `#18743 `_ * Remove `defaults` channel for conda usage by `@bernt-matthias `_ in `#18757 `_ * Modelling for Test Assertions by `@jmchilton `_ in `#18787 `_ * Workflow Landing Requests by `@jmchilton `_ in `#18807 `_ * Add docs on indexing for filter column by `@bernt-matthias `_ in `#18844 `_ * Implement Pydantic model for workflow test format. by `@jmchilton `_ in `#18884 `_ * Improvements to Planemo test format based on feedback/discussions. by `@jmchilton `_ in `#18889 `_ * Add support for arbitrarily ordered image axes in image content assertions by `@kostrykin `_ in `#18891 `_ * Remove some unused dynamic drill down options. by `@jmchilton `_ in `#18892 `_ * Type annotations and fixes by `@nsoranzo `_ in `#18911 `_ * Add filter null collection operation tool by `@mvdbeek `_ in `#18928 `_ * Assert that `data_column` parameters have a valid `data_ref` by `@bernt-matthias `_ in `#18949 `_ * More concise, readable tool execution testing. by `@jmchilton `_ in `#18977 `_ * Workflow landing improvements by `@mvdbeek `_ in `#18979 `_ * Add hint to the tool schema about runtime conditional changes by `@martenson `_ in `#18996 `_ * A variety of improvements around tool parameter modeling (second try) by `@jmchilton `_ in `#19027 `_ * Better logging around tool loading by `@jmchilton `_ in `#19029 `_ * Update Python dependencies by `@galaxybot `_ in `#19031 `_ * Integrate Tool Parameter Modeling into Linting (for Planemo) by `@jmchilton `_ in `#19073 `_ * Allow deferred datasets to behave as URIs by `@davelopez `_ in `#19077 `_ * Implement workflow parameter validators. by `@jmchilton `_ in `#19092 `_ * Enable specifying dataset hash in test jobs by `@mvdbeek `_ in `#19113 `_ * Add linter to check validity of output filters by `@bernt-matthias `_ in `#19114 `_ * Also use miniforge docker for container building by `@bernt-matthias `_ in `#19128 `_ * Fix default value handling for parameters connected to required parameters by `@mvdbeek `_ in `#19219 `_ ------------------- 24.1.4 (2024-12-11) ------------------- ========= Bug fixes ========= * Handle all requests error in ``ApiBiotoolsMetadataSource._raw_get_metadata`` by `@nsoranzo `_ in `#18510 `_ * xsd: allow `change_format` and `actions` also in statically defined collection elements, and break recursion by `@bernt-matthias `_ in `#18605 `_ * Remove defaults channel for conda usage by `@bernt-matthias `_ in `#18859 `_ * Don't check availability of shellescape by `@nsoranzo `_ in `#18954 `_ * Backport 2 CI fixes by `@nsoranzo `_ in `#18973 `_ * Disable locking when opening h5 files, add missing ``with`` by `@mvdbeek `_ in `#18976 `_ * Fix extra call to test_data_path that requires an admin key by `@jmchilton `_ in `#19011 `_ * flip default value for use_mamba to false by `@bgruening `_ in `#19295 `_ * Linter: allow dynamic option definition by from_url by `@bernt-matthias `_ in `#19300 `_ ============ Enhancements ============ * Make `default_panel_view` a `_by_host` option by `@natefoo `_ in `#18471 `_ ------------------- 24.1.3 (2024-10-25) ------------------- ========= Bug fixes ========= * Handle all requests error in ``ApiBiotoolsMetadataSource._raw_get_metadata`` by `@nsoranzo `_ in `#18510 `_ * xsd: allow `change_format` and `actions` also in statically defined collection elements, and break recursion by `@bernt-matthias `_ in `#18605 `_ * Remove defaults channel for conda usage by `@bernt-matthias `_ in `#18859 `_ * Don't check availability of shellescape by `@nsoranzo `_ in `#18954 `_ * Backport 2 CI fixes by `@nsoranzo `_ in `#18973 `_ * Disable locking when opening h5 files, add missing ``with`` by `@mvdbeek `_ in `#18976 `_ * Fix extra call to test_data_path that requires an admin key by `@jmchilton `_ in `#19011 `_ ============ Enhancements ============ * Make `default_panel_view` a `_by_host` option by `@natefoo `_ in `#18471 `_ ------------------- 24.1.2 (2024-09-25) ------------------- ========= Bug fixes ========= * Handle all requests error in ``ApiBiotoolsMetadataSource._raw_get_metadata`` by `@nsoranzo `_ in `#18510 `_ * xsd: allow `change_format` and `actions` also in statically defined collection elements, and break recursion by `@bernt-matthias `_ in `#18605 `_ ============ Enhancements ============ * Make `default_panel_view` a `_by_host` option by `@natefoo `_ in `#18471 `_ ------------------- 24.1.1 (2024-07-02) ------------------- ========= Bug fixes ========= * Fix bug in galaxy.xsd by `@kostrykin `_ in `#17752 `_ * Fix bug in `assert_has_image_n_labels` by `@kostrykin `_ in `#17754 `_ * Remove linter for unstripped text content for tool xml leaves by `@bernt-matthias `_ in `#18252 `_ ============ Enhancements ============ * Add test and doc showing how dynamic selects are used by `@bernt-matthias `_ in `#16885 `_ * Add content assertion XML tags for test output verification using images by `@kostrykin `_ in `#17581 `_ * Set minimal metadata also for empty bed datasets by `@wm75 `_ in `#17586 `_ * Automatically bind `galaxy_data_manager_data_path` in containers by `@bernt-matthias `_ in `#17596 `_ * Type annotation improvements by `@nsoranzo `_ in `#17601 `_ * Type annotation and CWL-related improvements by `@nsoranzo `_ in `#17630 `_ * Update Python dependencies by `@galaxybot `_ in `#17653 `_ * Code cleanups from ruff and pyupgrade by `@nsoranzo `_ in `#17654 `_ * Tool linter: check for valid bio.tools entries by `@bernt-matthias `_ in `#17655 `_ * Tool linter: check for leaf nodes with unstripped text content by `@bernt-matthias `_ in `#17656 `_ * Issue #17631: Make it possible to use custom invfile.lua if needed by `@martin-g `_ in `#17693 `_ * SQLAlchemy 2.0 by `@jdavcs `_ in `#17778 `_ * Add support for floating point TIFF files in verification of image-based tool outputs by `@kostrykin `_ in `#17797 `_ * Add tool linting for valid EDAM terms by `@bernt-matthias `_ in `#17839 `_ * Add `pin_labels` attribute for `image_diff` comparison method by `@kostrykin `_ in `#17866 `_ * Enable ``warn_unused_ignores`` mypy option by `@nsoranzo `_ in `#17991 `_ * Add galaxy to user agent by `@mvdbeek `_ in `#18003 `_ * Document syntax for accessing nested parameters in `change_format` - `when` tags by `@bernt-matthias `_ in `#18018 `_ * Update Python dependencies by `@galaxybot `_ in `#18063 `_ * Enable flake8-implicit-str-concat ruff rules by `@nsoranzo `_ in `#18067 `_ ------------------- 24.0.3 (2024-06-28) ------------------- ========= Bug fixes ========= * Allow purge query param, deprecate purge body param by `@mvdbeek `_ in `#18105 `_ * Make sure that all Linter subclasses are imported for listing them by `@bernt-matthias `_ in `#18339 `_ * Assign default ``data`` extension on discovered collection output by `@mvdbeek `_ in `#18389 `_ * Allow in_range validator for selects by `@bernt-matthias `_ in `#18403 `_ ------------------- 24.0.2 (2024-05-07) ------------------- ========= Bug fixes ========= * Tool linters: allow to skip by old linter names (by allowing to skip linter modules) by `@bernt-matthias `_ in `#18091 `_ * tool linters: output filters should only consider child filter nodes by `@bernt-matthias `_ in `#18096 `_ ------------------- 24.0.1 (2024-05-02) ------------------- ========= Bug fixes ========= * Fix conditional Image imports by `@mvdbeek `_ in `#17899 `_ ------------------- 24.0.0 (2024-04-02) ------------------- ========= Bug fixes ========= * xsd: reorder choices for permissive boolean by `@bernt-matthias `_ in `#17187 `_ * Allow for upper case container tags by `@bernt-matthias `_ in `#17254 `_ * Fixes for flake8-bugbear 24.1.17 by `@nsoranzo `_ in `#17340 `_ * Escape pipe character in tool XSD docs by `@neoformit `_ in `#17359 `_ * XSD schema doc building: quote pipe characters in attribute tables by `@bernt-matthias `_ in `#17364 `_ * Fix IUC best practices links, mention data_source_async in XSD by `@wm75 `_ in `#17409 `_ * Fix data_source and data_source_async bugs by `@wm75 `_ in `#17422 `_ * Add tool XML schema documention for outputs - collection - data by `@bernt-matthias `_ in `#17465 `_ * has_size assertion: implement size (as synonym for value) by `@bernt-matthias `_ in `#17490 `_ * Yaml nested assertions: fix parsing by `@bernt-matthias `_ in `#17641 `_ ============ Enhancements ============ * build_mulled: also use namespace for building singularity images by `@bernt-matthias `_ in `#15657 `_ * Python 3.8 as minimum by `@mr-c `_ in `#16954 `_ * Extend regex groups in stdio regex matches by `@bernt-matthias `_ in `#17016 `_ * Split linters in separate classes by `@bernt-matthias `_ in `#17081 `_ * Add select parameter with options from remote resources by `@mvdbeek `_ in `#17087 `_ * Replace discouraged Mambaforge with Miniforge3 by `@nsoranzo `_ in `#17177 `_ * Clarify the meaning of lexical sorting of discovered datasets by `@bernt-matthias `_ in `#17190 `_ * Migrate models to pydantic 2 by `@mvdbeek `_ in `#17262 `_ * Add element_identifier and ext to inputs config file export by `@bernt-matthias `_ in `#17357 `_ * Enable ``warn_unreachable`` mypy option by `@mvdbeek `_ in `#17365 `_ * Fix type annotation of code using XML etree by `@nsoranzo `_ in `#17367 `_ * Update to black 2024 stable style by `@nsoranzo `_ in `#17391 `_ * Some additional typing for test interactor stuff. by `@jmchilton `_ in `#17398 `_ * Allow using tool data bundles as inputs to reference data select parameters by `@mvdbeek `_ in `#17435 `_ * `data_column` parameter: use `column_names` metadata if present by `@bernt-matthias `_ in `#17478 `_ * Fixing data_source tools and incrementing tool profile by `@wm75 `_ in `#17515 `_ * Add `image_diff` comparison method for test output verification using images by `@kostrykin `_ in `#17556 `_ * add shm_size based on ShmSize by `@richard-burhans `_ in `#17565 `_ * Record missing outputs as test errors by `@bernt-matthias `_ in `#17874 `_ ============= Other changes ============= * consistently compare profile versions by `@bernt-matthias `_ in `#16492 `_ ------------------- 23.2.1 (2024-02-21) ------------------- ========= Bug fixes ========= * Never consider `_galaxy_` conda env as unused by `@bernt-matthias `_ in `#16460 `_ * chore: fix typos by `@afuetterer `_ in `#16851 `_ * Ruff and flake8 fixes by `@nsoranzo `_ in `#16884 `_ * Quote singularity env parameters by `@bernt-matthias `_ in `#17055 `_ * Remove duplicates when copying sections for tool panel view by `@bernt-matthias `_ in `#17117 `_ * Display application fixes and tests by `@mvdbeek `_ in `#17233 `_ ============ Enhancements ============ * Implement default locations for data and collection parameters. by `@jmchilton `_ in `#14955 `_ * Add framework test for profile behavior of `format="input"` by `@bernt-matthias `_ in `#15482 `_ * Tool Shed 2.0 by `@jmchilton `_ in `#15639 `_ * Add ability to assert metadata properties on input dataset parameters by `@bernt-matthias `_ in `#15825 `_ * Migrate a part of the users API to Fast API by `@heisner-tillman `_ in `#16341 `_ * Update Python dependencies by `@galaxybot `_ in `#16436 `_ * Tweak tool memory use and optimize shared memory when using preload by `@mvdbeek `_ in `#16536 `_ * Document that required text parameters need a validator by `@bernt-matthias `_ in `#16538 `_ * Include `regex` when linting validators by `@davelopez `_ in `#16684 `_ * Refactor Tool Panel views structures and combine ToolBox and ToolBoxWorkflow into one component by `@ahmedhamidawan `_ in `#16739 `_ * Replace file_name property with get_file_name function by `@SergeyYakubov `_ in `#16783 `_ * Updated path-based interactive tools with entry point path injection, support for ITs with relative links, shortened URLs, doc and config updates including Podman job_conf by `@sveinugu `_ in `#16795 `_ * Remove remaining legacy backbone form input elements by `@guerler `_ in `#16834 `_ * Change `api/tool_panel` to `api/tool_panels/...` by `@ahmedhamidawan `_ in `#16928 `_ * optimize object store cache operations by `@SergeyYakubov `_ in `#17025 `_ * Enhance xsd schema and allow simpler assertion lists by `@mvdbeek `_ in `#17134 `_ ============= Other changes ============= * Merge 23.1 into dev by `@mvdbeek `_ in `#16534 `_ * Explicitly document default of multiple by `@bernt-matthias `_ in `#16625 `_ ------------------- 23.1.4 (2024-01-04) ------------------- ========= Bug fixes ========= * Separate collection and non-collection data element by `@mvdbeek `_ in `#17236 `_ ------------------- 23.1.3 (2023-12-01) ------------------- No recorded changes since last release ------------------- 23.1.2 (2023-11-29) ------------------- ========= Bug fixes ========= * Change log level for duplicate data table entries to warning by `@mvdbeek `_ in `#16988 `_ * Upgrade minimum conda to be compatible with latest conda-build by `@bernt-matthias `_ in `#17013 `_ * Fix duplicated tools in tool panel view section copying by `@jmchilton `_ in `#17036 `_ ============ Enhancements ============ * Adds `biii` as supported xref reference type by `@kostrykin `_ in `#16952 `_ ------------------- 23.1.1 (2023-10-23) ------------------- ========= Bug fixes ========= * Fixes for two framework test tools by `@bernt-matthias `_ in `#15483 `_ * add missing f for f-string by `@bernt-matthias `_ in `#15584 `_ * Fix call to `docker_cached_container_description` by `@bernt-matthias `_ in `#15598 `_ * Fix log message by `@bernt-matthias `_ in `#15838 `_ * add required_files to the tag list for linting by `@bernt-matthias `_ in `#16129 `_ * Handle appending to a results file that does not exists. by `@ksuderman `_ in `#16233 `_ * Improve container resolver documentation by `@mvdbeek `_ in `#16280 `_ * Add missing singularity_no_mount prop parsing by `@mvdbeek `_ in `#16367 `_ * Restore resolution of Conda environments generated from non-lowercase package names by `@nsoranzo `_ in `#16474 `_ * Fix up unit tests for local use by `@mvdbeek `_ in `#16483 `_ * Fix `multiple` remote test data by `@davelopez `_ in `#16542 `_ * Don't use ``docker run`` --user flag on OSX by `@mvdbeek `_ in `#16573 `_ * Backport tool mem fixes by `@mvdbeek `_ in `#16601 `_ * xsd: allow name attribute of test collections by `@bernt-matthias `_ in `#16663 `_ * Fix short ids in tool panel views. by `@jmchilton `_ in `#16800 `_ * Fix tool panel views for versionless tool ids by `@mvdbeek `_ in `#16809 `_ ============ Enhancements ============ * Decompress history data for testing assertions by `@bernt-matthias `_ in `#15085 `_ * OIDC tokens by `@SergeyYakubov `_ in `#15300 `_ * Fix for new style conda packages by `@bernt-matthias `_ in `#15446 `_ * Move database access code out of tool_util by `@jdavcs `_ in `#15467 `_ * Protection against problematic boolean parameters. by `@jmchilton `_ in `#15493 `_ * Implement initial tool/wf test assertions module for JSON data. by `@jmchilton `_ in `#15494 `_ * Explore tool remote test data by `@davelopez `_ in `#15510 `_ * xsd: add multiple to the list of attributes for all parameter types deriving from select by `@bernt-matthias `_ in `#15534 `_ * Update Python dependencies by `@galaxybot `_ in `#15564 `_ * Container resolvers: add docs, typing and tests by `@bernt-matthias `_ in `#15614 `_ * Migrate to MyST-Parser for Markdown docs by `@nsoranzo `_ in `#15844 `_ * Enable per-destination ``container_resolver_config_file`` by `@bernt-matthias `_ in `#15884 `_ * Updated doc and tests for attribute value filter by `@tuncK `_ in `#15929 `_ * Make container builders use mamba by `@bernt-matthias `_ in `#15953 `_ * Code cleanups from ruff and pyupgrade by `@nsoranzo `_ in `#16035 `_ * Vendorise ``packaging.versions.LegacyVersion`` by `@nsoranzo `_ in `#16058 `_ * Merge ``Target`` class with ``CondaTarget`` by `@nsoranzo `_ in `#16181 `_ * Small test decorator improvements. by `@jmchilton `_ in `#16220 `_ * tool_util: switch to mambaforge on non-32bit; add arm64 support by `@mr-c `_ in `#16223 `_ * Fix tool remote test data by `@davelopez `_ in `#16311 `_ ============= Other changes ============= * Restore tmp mounting by `@mvdbeek `_ in `#16533 `_ ------------------- 23.0.6 (2023-10-23) ------------------- ========= Bug fixes ========= * Fixes for extra files handling and cached object stores by `@mvdbeek `_ in `#16595 `_ * Fix create/install commands for conda 23.9.0 by `@nsoranzo `_ in `#16831 `_ ------------------- 23.0.5 (2023-07-29) ------------------- ========= Bug fixes ========= * Allow duplicate labels in linter if outputs contain filters by `@bernt-matthias `_ in `#15933 `_ * Fix parsing tool metadata from bio.tools by `@kysrpex `_ in `#16449 `_ * Linter: fix regex for profile version by `@bernt-matthias `_ in `#16480 `_ * Adjust test_data_download method in GalaxyInteractorApi so an admin user is not required by `@simonbray `_ in `#16482 `_ ------------------- 23.0.4 (2023-06-30) ------------------- No recorded changes since last release ------------------- 23.0.3 (2023-06-26) ------------------- ========= Bug fixes ========= * xsd: add missing `sep` attribute for `has_n_columns` by `@bernt-matthias `_ in `#16262 `_ * Missing init prevents models.py being bundled into tool_util by `@nuwang `_ in `#16308 `_ ============ Enhancements ============ * When importing tool data bundles, use the first loc file for the matching table by `@natefoo `_ in `#16247 `_ ------------------- 23.0.2 (2023-06-13) ------------------- ========= Bug fixes ========= * Don't fail CWL tool parsing when Cheetah not installed by `@mvdbeek `_ in `#16219 `_ * Allow skipping ``expect_num_outputs`` when ``expect_failure`` is set in tool test by `@bernt-matthias `_ in `#16237 `_ ------------------- 23.0.1 (2023-06-08) ------------------- ========= Bug fixes ========= * Fix assertion linting to not fail on byte suffixes by `@bernt-matthias `_ in `#15873 `_ * Fix ``get_test_from_anaconda()`` and ``base_image_for_targets()`` functions by `@nsoranzo `_ in `#16125 `_ * Fix test search for mulled container hashes by `@mvdbeek `_ in `#16170 `_ ============ Enhancements ============ * Allow setting auto_decompress property in staging interface by `@mvdbeek `_ in `#16014 `_ ------------------- 22.1.5 (2022-11-14) ------------------- * Set test status to success on expected failure ------------------- 22.1.4 (2022-10-28) ------------------- * Add missing unittest_utils package to galaxy-tool-util ------------------- 22.1.3 (2022-10-27) ------------------- * Pin minimum pyopenssl version when installing Conda * Add ``--strict-channel-priority`` to conda create/install commands if using conda >=4.7.5 ------------------- 22.1.2 (2022-08-29) ------------------- * Fix lint context error level * Pin galaxy-util to >= 22.1 * Fix biocontainer resolution without beaker cache ------------------- 22.1.1 (2022-08-22) ------------------- * First release from the 22.01 branch of Galaxy ------------------- 21.9.2 (2021-11-23) ------------------- * Fix linting of ``multiple="true"`` select inputs. ------------------- 21.9.1 (2021-11-03) ------------------- * Fix tool linting. ------------------- 21.9.0 (2021-11-03) ------------------- * First release from the 21.09 branch of Galaxy. ------------------- 21.1.2 (2021-06-23) ------------------- ------------------- 21.1.1 (2021-05-21) ------------------- ------------------- 21.1.0 (2021-03-19) ------------------- * First release from the 21.01 branch of Galaxy. ------------------- 20.9.1 (2020-10-28) ------------------- * Bugfixes to work around & annotate expected tool test failures. ------------------- 20.9.0 (2020-10-28) ------------------- * First release from the 20.09 branch of Galaxy. ------------------------ 20.9.0.dev2 (2020-08-02) ------------------------ * First release from the 20.09 branch of Galaxy. ------------------- 20.5.0 (2020-07-04) ------------------- * First release from the 20.05 branch of Galaxy. ------------------- 20.1.0 (2020-07-04) ------------------- * First release from the 20.01 branch of Galaxy. ------------------- 19.9.1 (2019-12-28) ------------------- * Fix declared dependency problem with package. ------------------- 19.9.0 (2019-12-16) ------------------- * Initial import from dev branch of Galaxy during 19.09 development cycle. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/README.rst0000644000175100017510000000042615234436334016326 0ustar00runnerrunner .. image:: https://badge.fury.io/py/galaxy-tool-util.svg :target: https://pypi.org/project/galaxy-tool-util/ Overview -------- The Galaxy_ tool utilities. * Code: https://github.com/galaxyproject/galaxy/tree/dev/packages/tool_util .. _Galaxy: http://galaxyproject.org/ ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/dev-requirements.txt0000644000175100017510000000006115234436334020672 0ustar00runnerrunner# For dev mypy # For release build twine==6.2.0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/mypy.ini0000644000175100017510000006230515234436334016342 0ustar00runnerrunner[mypy] enable_error_code = ignore-without-code plugins = pydantic.mypy show_error_codes = True ignore_missing_imports = True check_untyped_defs = True exclude = (?x)( ^build/ | .*test/functional/tools/cwl_tools/ | .*tool_shed/test/test_data/repos/ ) pretty = True no_implicit_reexport = True no_implicit_optional = True strict_equality = True warn_redundant_casts = True warn_unreachable = True warn_unused_ignores = True platform = linux explicit_package_bases = True # green list - work on growing these please! [mypy-galaxy.jobs.job_destination] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True [mypy-galaxy.tool_util.toolbox.lineages.interface] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True [mypy-galaxy.tools.cache] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True [mypy-galaxy.tools.search] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True [mypy-galaxy.util.compression_utils] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True [mypy-galaxy.util.custom_logging] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True [mypy-galaxy.util.unittest_utils] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True [mypy-galaxy.util.xml_macros] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True [mypy-galaxy.managers.secured] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True # orange list - still need to enable some of the checks [mypy-galaxy.tools.wrappers] disallow_any_generics = True disallow_subclassing_any = True disallow_untyped_calls = False disallow_untyped_decorators = True disallow_untyped_defs = True warn_return_any = True warn_unreachable = False # red list - work on reducing these please! [mypy-galaxy.celery] warn_unreachable = False [mypy-galaxy.config] warn_unreachable = False [mypy-galaxy.datatypes.constructive_solid_geometry] warn_unreachable = False [mypy-galaxy.datatypes.data] warn_unreachable = False [mypy-galaxy.datatypes.mothur] warn_unreachable = False [mypy-galaxy.datatypes.phylip] warn_unreachable = False [mypy-galaxy.datatypes.qiime2] warn_unreachable = False [mypy-galaxy.managers.jobs] warn_unreachable = False [mypy-galaxy.tool_shed.metadata.metadata_generator] warn_unreachable = False [mypy-galaxy.tool_util.cwl.parser] warn_unreachable = False [mypy-galaxy.tool_util.cwl.cwltool_deps] warn_unreachable = False [mypy-galaxy.tool_util.data] warn_unreachable = False [mypy-galaxy.tool_util.toolbox.base] warn_unreachable = False [mypy-galaxy.tool_util.verify.interactor] warn_unreachable = False [mypy-galaxy.tools] warn_unreachable = False [mypy-galaxy.tools.parameters.basic] warn_unreachable = False [mypy-galaxy.webapps.galaxy.api.library_datasets] warn_unreachable = False [mypy-galaxy.webapps.galaxy.services.jobs] warn_unreachable = False [mypy-galaxy.workflow.run_request] warn_unreachable = False [mypy-galaxy_test.driver.driver_util] warn_unreachable = False [mypy-tool_shed.metadata.repository_metadata_manager] warn_unreachable = False [mypy-galaxy.util.oset] # lots of tricky code in here... check_untyped_defs = False [mypy-galaxy.webapps.reports.controllers.*] check_untyped_defs = False [mypy-galaxy.managers.deletable] check_untyped_defs = False [mypy-galaxy.datatypes.converters.interval_to_bedstrict_converter] check_untyped_defs = False [mypy-galaxy_test.base.ssh_util] check_untyped_defs = False [mypy-galaxy.util.heartbeat] check_untyped_defs = False [mypy-galaxy.tools.bundled.stats.aggregate_scores_in_intervals] check_untyped_defs = False [mypy-galaxy.tools.bundled.sr_mapping.srma_wrapper] check_untyped_defs = False [mypy-galaxy.tools.bundled.sr_mapping.bfast_wrapper] check_untyped_defs = False [mypy-galaxy.tools.bundled.sr_assembly.velveth_wrapper] check_untyped_defs = False [mypy-galaxy.tools.bundled.sr_assembly.velvetg_wrapper] check_untyped_defs = False [mypy-galaxy.tools.bundled.solid_tools.solid_qual_stats] check_untyped_defs = False [mypy-galaxy.tools.bundled.solid_tools.maq_cs_wrapper] check_untyped_defs = False [mypy-galaxy.tools.bundled.phenotype_association.pagetag] check_untyped_defs = False [mypy-galaxy.tools.bundled.maf.maf_to_bed] check_untyped_defs = False [mypy-galaxy.tools.bundled.filters.trimmer] check_untyped_defs = False [mypy-galaxy.tools.bundled.filters.sff_extract] check_untyped_defs = False [mypy-galaxy.tools.bundled.filters.lav_to_bed] check_untyped_defs = False [mypy-galaxy.tools.bundled.evolution.add_scores] check_untyped_defs = False [mypy-galaxy.tools.bundled.data_source.microbial_import_code] check_untyped_defs = False [mypy-galaxy.datatypes.converters.pileup_to_interval_index_converter] check_untyped_defs = False [mypy-galaxy.datatypes.converters.bgzip] check_untyped_defs = False [mypy-tool_shed.grids.repository_grid_filter_manager] check_untyped_defs = False [mypy-galaxy.web_stack.message] check_untyped_defs = False [mypy-galaxy.web.framework.middleware.translogger] check_untyped_defs = False [mypy-galaxy.web.framework.middleware.profile] check_untyped_defs = False [mypy-galaxy.web.framework.middleware.error] check_untyped_defs = False [mypy-galaxy.util.topsort] check_untyped_defs = False [mypy-galaxy.util.submodules] check_untyped_defs = False [mypy-galaxy.util.monitors] check_untyped_defs = False [mypy-galaxy.tools.util.maf_utilities] check_untyped_defs = False [mypy-galaxy.tools.parameters.wrapped_json] check_untyped_defs = False [mypy-galaxy.tools.bundled.stats.grouping] check_untyped_defs = False [mypy-galaxy.tools.bundled.ngs_simulation.ngs_simulation] check_untyped_defs = False [mypy-galaxy.tools.bundled.metag_tools.shrimp_wrapper] check_untyped_defs = False [mypy-galaxy.tools.bundled.filters.random_lines_two_pass] check_untyped_defs = False [mypy-galaxy.tool_util.deps.brew_exts] check_untyped_defs = False [mypy-galaxy.model.item_attrs] check_untyped_defs = False [mypy-galaxy.model.base] check_untyped_defs = False [mypy-galaxy.model.dataset_collections.structure] check_untyped_defs = False [mypy-galaxy.managers.annotatable] check_untyped_defs = False [mypy-galaxy.jobs.runners.util.cli] check_untyped_defs = False [mypy-galaxy.datatypes.util.maf_utilities] check_untyped_defs = False [mypy-galaxy.datatypes.dataproviders.decorators] check_untyped_defs = False [mypy-galaxy.tools.bundled.maf.maf_to_interval] check_untyped_defs = False [mypy-galaxy.tools.bundled.maf.maf_stats] check_untyped_defs = False [mypy-galaxy.tools.bundled.maf.interval_maf_to_merged_fasta] check_untyped_defs = False [mypy-galaxy.tools.bundled.maf.interval2maf] check_untyped_defs = False [mypy-galaxy.util.yaml_util] check_untyped_defs = False [mypy-galaxy.util.rules_dsl] check_untyped_defs = False [mypy-test.functional.webhooks.phdcomics] check_untyped_defs = False [mypy-galaxy.util.path] check_untyped_defs = False [mypy-galaxy.util] check_untyped_defs = False no_implicit_reexport = False [mypy-galaxy.tools.bundled.maf.vcf_to_maf_customtrack] check_untyped_defs = False [mypy-tool_shed.webapp.security] check_untyped_defs = False [mypy-tool_shed.util.hgweb_config] check_untyped_defs = False [mypy-galaxy.workflow.resources] check_untyped_defs = False [mypy-galaxy.webapps.reports.config] check_untyped_defs = False [mypy-galaxy.web.framework.base] check_untyped_defs = False [mypy-galaxy.web.framework.middleware.remoteuser] check_untyped_defs = False [mypy-galaxy.util.task] check_untyped_defs = False [mypy-galaxy.util.sanitize_html] check_untyped_defs = False [mypy-galaxy.util.commands] check_untyped_defs = False [mypy-galaxy.util.tool_shed.xml_util] check_untyped_defs = False [mypy-galaxy.tools.parameters.sanitize] check_untyped_defs = False [mypy-galaxy.tools.bundled.filters.join] check_untyped_defs = False [mypy-galaxy.tool_util.output_checker] check_untyped_defs = False [mypy-galaxy.tool_util.verify.test_data] check_untyped_defs = False [mypy-galaxy.tool_util.toolbox.parser] check_untyped_defs = False [mypy-galaxy.tool_util.toolbox.filters] check_untyped_defs = False [mypy-galaxy.tool_util.parser.output_collection_def] check_untyped_defs = False [mypy-galaxy.tool_util.deps.mulled.get_tests] check_untyped_defs = False [mypy-galaxy.security.object_wrapper] check_untyped_defs = False [mypy-galaxy.quota] check_untyped_defs = False [mypy-galaxy.model.custom_types] check_untyped_defs = False [mypy-galaxy.datatypes.util.gff_util] check_untyped_defs = False [mypy-galaxy.datatypes.dataproviders.line] check_untyped_defs = False [mypy-galaxy.datatypes.dataproviders.external] check_untyped_defs = False [mypy-galaxy.datatypes.dataproviders.dataset] check_untyped_defs = False [mypy-galaxy.tools.bundled.filters.gff.gff_filter_by_feature_count] check_untyped_defs = False [mypy-galaxy.tools.bundled.extract.extract_genomic_dna] check_untyped_defs = False [mypy-galaxy.tool_util.deps.resolvers] check_untyped_defs = False [mypy-galaxy.tool_util.deps.mulled.mulled_update_singularity_containers] check_untyped_defs = False [mypy-galaxy.tool_util.deps.mulled.mulled_search] check_untyped_defs = False [mypy-galaxy.tool_shed.util.hg_util] check_untyped_defs = False [mypy-galaxy.jobs.runners.util.cli.shell.rsh] check_untyped_defs = False [mypy-galaxy.datatypes.display_applications.parameters] check_untyped_defs = False [mypy-tool_shed.webapp.search.tool_search] check_untyped_defs = False [mypy-tool_shed.webapp.search.repo_search] check_untyped_defs = False [mypy-tool_shed.util.hg_util] check_untyped_defs = False [mypy-tool_shed.repository_types.util] check_untyped_defs = False [mypy-tool_shed.managers.repositories] check_untyped_defs = False [mypy-tool_shed.managers.groups] check_untyped_defs = False [mypy-galaxy.workflow.trs_proxy] check_untyped_defs = False [mypy-galaxy.web_stack.handlers] check_untyped_defs = False [mypy-galaxy.web.proxy] check_untyped_defs = False [mypy-galaxy.web.framework.decorators] check_untyped_defs = False [mypy-galaxy.util.tool_shed.common_util] check_untyped_defs = False [mypy-galaxy.tools.expressions.evaluation] check_untyped_defs = False [mypy-galaxy.tool_util.verify] check_untyped_defs = False [mypy-galaxy.tool_util.toolbox.watcher] check_untyped_defs = False [mypy-galaxy.tool_util.toolbox.integrated_panel] check_untyped_defs = False [mypy-galaxy.tool_util.deps.resolvers.resolver_mixins] check_untyped_defs = False [mypy-galaxy.tool_util.deps.resolvers.modules] check_untyped_defs = False [mypy-galaxy.tool_util.cwl.representation] check_untyped_defs = False [mypy-galaxy.tool_shed.util.tool_dependency_util] check_untyped_defs = False [mypy-galaxy.tool_shed.tools.data_table_manager] check_untyped_defs = False [mypy-galaxy.objectstore.s3_multipart_upload] check_untyped_defs = False [mypy-galaxy.model.database_utils] check_untyped_defs = False [mypy-galaxy.model.dataset_collections.matching] check_untyped_defs = False [mypy-galaxy.datatypes.sniff] check_untyped_defs = False [mypy-galaxy.config.schema] check_untyped_defs = False [mypy-galaxy.auth.providers.ldap_ad] check_untyped_defs = False [mypy-galaxy.auth.util] check_untyped_defs = False [mypy-tool_shed.webapp.model.migrate.check] check_untyped_defs = False [mypy-galaxy.web_stack] check_untyped_defs = False [mypy-galaxy.web.legacy_framework.grids] check_untyped_defs = False [mypy-galaxy.web.framework.helpers.grids] check_untyped_defs = False [mypy-galaxy.util.script] check_untyped_defs = False [mypy-galaxy.tools.repositories] check_untyped_defs = False [mypy-galaxy.objectstore.s3] check_untyped_defs = False [mypy-galaxy.objectstore.pithos] check_untyped_defs = False [mypy-galaxy.objectstore.irods] check_untyped_defs = False [mypy-galaxy.objectstore.azure_blob] check_untyped_defs = False [mypy-galaxy.objectstore.cloud] check_untyped_defs = False [mypy-galaxy.model.tool_shed_install] check_untyped_defs = False [mypy-galaxy.model.metadata] check_untyped_defs = False [mypy-galaxy.model] check_untyped_defs = False [mypy-tool_shed.util.admin_util] check_untyped_defs = False [mypy-tool_shed.repository_types.registry] check_untyped_defs = False [mypy-galaxy.tools.errors] check_untyped_defs = False [mypy-galaxy.tools.parameters.validation] check_untyped_defs = False [mypy-galaxy.tools.parameters.dataset_matcher] check_untyped_defs = False [mypy-galaxy.tool_util.verify.script] check_untyped_defs = False [mypy-galaxy.tool_util.deps.resolvers.unlinked_tool_shed_packages] check_untyped_defs = False [mypy-galaxy.tool_shed.util.repository_util] check_untyped_defs = False [mypy-galaxy.model.security] check_untyped_defs = False [mypy-galaxy.model.search] check_untyped_defs = False [mypy-galaxy.model.database_heartbeat] check_untyped_defs = False [mypy-galaxy.model.tool_shed_install.mapping] check_untyped_defs = False [mypy-galaxy.model.store] warn_unreachable = False check_untyped_defs = False [mypy-galaxy.model.orm.scripts] check_untyped_defs = False [mypy-galaxy.managers.taggable] check_untyped_defs = False [mypy-galaxy.jobs.splitters.multi] check_untyped_defs = False [mypy-galaxy.datatypes.display_applications.application] check_untyped_defs = False [mypy-tool_shed.util.metadata_util] check_untyped_defs = False [mypy-galaxy.tools.error_reports.plugins.base_git] check_untyped_defs = False [mypy-galaxy.tool_util.deps.mulled.mulled_build] check_untyped_defs = False [mypy-galaxy.tool_util.deps] check_untyped_defs = False [mypy-galaxy.model.dataset_collections.registry] check_untyped_defs = False [mypy-galaxy.metadata] check_untyped_defs = False [mypy-galaxy.jobs.runners.util.cli.job.lsf] check_untyped_defs = False [mypy-galaxy.authnz.managers] check_untyped_defs = False [mypy-tool_shed.util.repository_util] check_untyped_defs = False [mypy-tool_shed.grids.util] check_untyped_defs = False [mypy-galaxy.tools.error_reports.plugins.gitlab] check_untyped_defs = False [mypy-galaxy.tools.error_reports.plugins.github] check_untyped_defs = False [mypy-galaxy.tool_util.parser.yaml] check_untyped_defs = False [mypy-galaxy.tool_util.parser.xml] check_untyped_defs = False [mypy-galaxy.tool_util.deps.views] check_untyped_defs = False [mypy-galaxy.tool_util.deps.container_classes] check_untyped_defs = False [mypy-galaxy.tool_util.deps.resolvers.conda] check_untyped_defs = False [mypy-galaxy.jobs.rule_helper] check_untyped_defs = False [mypy-galaxy.datatypes.isa] check_untyped_defs = False [mypy-galaxy.datatypes.binary] check_untyped_defs = False [mypy-tool_shed.utility_containers.utility_container_manager] check_untyped_defs = False [mypy-tool_shed.dependencies.attribute_handlers] warn_unreachable = False check_untyped_defs = False [mypy-galaxy.tool_util.parser.cwl] check_untyped_defs = False [mypy-galaxy.tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager] check_untyped_defs = False [mypy-galaxy.datatypes.triples] check_untyped_defs = False [mypy-galaxy.datatypes.tabular] warn_unreachable = False check_untyped_defs = False [mypy-galaxy.datatypes.sequence] check_untyped_defs = False [mypy-test.manual.workflows_scaling] check_untyped_defs = False [mypy-test.manual.gen_history_export_test_artifacts] check_untyped_defs = False [mypy-tool_shed.dependencies.repository.relation_builder] check_untyped_defs = False [mypy-galaxy.visualization.data_providers.basic] check_untyped_defs = False [mypy-galaxy.datatypes.proteomics] check_untyped_defs = False [mypy-galaxy.datatypes.molecules] check_untyped_defs = False [mypy-galaxy.datatypes.interval] warn_unreachable = False check_untyped_defs = False [mypy-galaxy.datatypes.genetics] warn_unreachable = False check_untyped_defs = False [mypy-tool_shed.webapp.model] check_untyped_defs = False [mypy-tool_shed.utility_containers] check_untyped_defs = False [mypy-galaxy.datatypes.registry] check_untyped_defs = False [mypy-galaxy.tools.parameters.dynamic_options] check_untyped_defs = False [mypy-galaxy.tools.parameters] check_untyped_defs = False [mypy-galaxy.tools.bundled.data_source.upload] check_untyped_defs = False [mypy-galaxy.tool_util.lint] check_untyped_defs = False [mypy-test.functional.tools.upload] check_untyped_defs = False [mypy-tool_shed.util.shed_index] check_untyped_defs = False [mypy-tool_shed.grids.repository_review_grids] check_untyped_defs = False [mypy-tool_shed.grids.admin_grids] check_untyped_defs = False [mypy-tool_shed.galaxy_install.dependency_display] check_untyped_defs = False [mypy-galaxy.tools.parameters.meta] check_untyped_defs = False [mypy-galaxy.tools.data_manager.manager] check_untyped_defs = False [mypy-galaxy.tool_util.deps.container_resolvers.mulled] check_untyped_defs = False [mypy-galaxy.managers.context] check_untyped_defs = False [mypy-galaxy.managers.citations] check_untyped_defs = False [mypy-galaxy.config.script] check_untyped_defs = False [mypy-galaxy.tool_shed.cache] check_untyped_defs = False [mypy-galaxy.tool_util.deps.containers] check_untyped_defs = False [mypy-galaxy.managers.users] check_untyped_defs = False [mypy-galaxy.managers.ratable] check_untyped_defs = False [mypy-galaxy.tools.actions.upload] check_untyped_defs = False [mypy-galaxy.tools.actions.model_operations] check_untyped_defs = False [mypy-galaxy.job_execution.output_collect] check_untyped_defs = False [mypy-galaxy.actions.library] check_untyped_defs = False [mypy-galaxy.tool_util.cwl.runtime_actions] check_untyped_defs = False [mypy-galaxy.metadata.set_metadata] check_untyped_defs = False [mypy-galaxy.tools.error_reports] check_untyped_defs = False [mypy-galaxy.jobs.dynamic_tool_destination] warn_unreachable = False check_untyped_defs = False [mypy-galaxy.jobs.runners.state_handler_factory] check_untyped_defs = False [mypy-galaxy.queue_worker] check_untyped_defs = False [mypy-galaxy.jobs.mapper] check_untyped_defs = False [mypy-galaxy.jobs.runners] warn_unreachable = False check_untyped_defs = False [mypy-galaxy.jobs] check_untyped_defs = False [mypy-galaxy.jobs.handler] check_untyped_defs = False [mypy-galaxy.workflow.scheduling_manager] check_untyped_defs = False [mypy-galaxy.jobs.manager] check_untyped_defs = False [mypy-galaxy.tool_shed.galaxy_install.installed_repository_manager] check_untyped_defs = False [mypy-tool_shed.webapp.config] check_untyped_defs = False [mypy-galaxy.main] check_untyped_defs = False [mypy-galaxy.web.framework.middleware.sqldebug] check_untyped_defs = False [mypy-galaxy.tools.recommendations] check_untyped_defs = False [mypy-galaxy.tool_shed.galaxy_install.install_manager] check_untyped_defs = False [mypy-galaxy.jobs.runners.pulsar] check_untyped_defs = False [mypy-galaxy.jobs.runners.pbs] check_untyped_defs = False [mypy-galaxy.jobs.runners.local] check_untyped_defs = False [mypy-galaxy.jobs.runners.kubernetes] check_untyped_defs = False [mypy-galaxy.jobs.runners.godocker] check_untyped_defs = False [mypy-galaxy.jobs.runners.drmaa] check_untyped_defs = False [mypy-galaxy.jobs.runners.condor] check_untyped_defs = False [mypy-galaxy.jobs.runners.cli] check_untyped_defs = False [mypy-galaxy.jobs.runners.chronos] check_untyped_defs = False [mypy-galaxy.jobs.runners.state_handlers.resubmit] check_untyped_defs = False [mypy-integration.objectstore_selection_rules.rules] check_untyped_defs = False [mypy-galaxy.webapps.base.controller] check_untyped_defs = False [mypy-galaxy.jobs.runners.univa] check_untyped_defs = False [mypy-integration.resubmission_runners] check_untyped_defs = False [mypy-tool_shed.webapp.controllers.upload] check_untyped_defs = False [mypy-tool_shed.webapp.controllers.repository_review] check_untyped_defs = False [mypy-tool_shed.webapp.api.repository_revisions] check_untyped_defs = False [mypy-tool_shed.webapp.api.repositories] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.workflow] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.forms] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.data_manager] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.async] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.admin] check_untyped_defs = False [mypy-galaxy.webapps.base.webapp] warn_unreachable = False check_untyped_defs = False [mypy-starlette_context.*] no_implicit_reexport = False [mypy-tool_shed.webapp.buildapp] check_untyped_defs = False [mypy-galaxy.webapps.reports.buildapp] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.buildapp] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.visualization] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.user] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.page] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.history] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.dataset] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.controllers.admin_toolshed] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.users] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.tool_shed_repositories] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.provenance] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.plugins] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.library_contents] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.jobs] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.item_tags] warn_unreachable = False check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.history_contents] warn_unreachable = False check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.genomes] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.datasets] check_untyped_defs = False [mypy-galaxy.webapps.galaxy.api.authenticate] check_untyped_defs = False [mypy-tool_shed.webapp.controllers.user] check_untyped_defs = False [mypy-test.functional.test_toolbox] check_untyped_defs = False [mypy-tool_shed.test.base.test_db_util] check_untyped_defs = False [mypy-integration.test_save_job_id_on_datasets] check_untyped_defs = False [mypy-integration.test_config_defaults] check_untyped_defs = False [mypy-test.functional.test_toolbox_pytest] check_untyped_defs = False [mypy-galaxy_test.selenium.test_histories_published] check_untyped_defs = False [mypy-galaxy_test.selenium.test_histories_list] check_untyped_defs = False [mypy-galaxy_test.api.test_history_contents] check_untyped_defs = False [mypy-galaxy_test.api.test_histories] check_untyped_defs = False [mypy-galaxy_test.api.test_folder_contents] check_untyped_defs = False [mypy-integration.test_workflow_sync] check_untyped_defs = False [mypy-integration.test_tool_data_delete] check_untyped_defs = False [mypy-integration.test_shed_tool_tests] check_untyped_defs = False [mypy-integration.test_scripts] check_untyped_defs = False [mypy-integration.test_pulsar_embedded_mq] check_untyped_defs = False [mypy-integration.test_local_job_cancellation] check_untyped_defs = False [mypy-integration.test_job_environments] check_untyped_defs = False [mypy-integration.test_handler_assignment_methods] check_untyped_defs = False [mypy-integration.test_default_permissions] check_untyped_defs = False [mypy-integration.test_data_manager_refgenie] check_untyped_defs = False [mypy-integration.test_data_manager] check_untyped_defs = False [mypy-integration.objectstore.test_swift_objectstore] check_untyped_defs = False [mypy-integration.objectstore._base] check_untyped_defs = False [mypy-test.functional.conftest] check_untyped_defs = False [mypy-integration.test_containerized_jobs] check_untyped_defs = False [mypy-integration.test_chained_dynamic_destinations] check_untyped_defs = False [mypy-integration.test_pulsar_embedded_containers] check_untyped_defs = False [mypy-integration.test_kubernetes_runner] check_untyped_defs = False [mypy-integration.objectstore.test_objectstore_datatype_upload] check_untyped_defs = False [mypy-integration.test_coexecution] check_untyped_defs = False [mypy-integration.test_interactivetools_api] check_untyped_defs = False [mypy-functional.*] check_untyped_defs = False [mypy-manual.*] check_untyped_defs = False # unit tests with prickly typing issues still that need to be worked through carefully [mypy-*.tools.test_tool_dependency_description] check_untyped_defs = False [mypy-*.tools.test_dataset_matcher] check_untyped_defs = False [mypy-*.datatypes.test_vcf] check_untyped_defs = False [mypy-*.datatypes.test_qiime2] check_untyped_defs = False [mypy-*.test_galaxy_mapping] check_untyped_defs = False [mypy-*.workflows.workflow_support] check_untyped_defs = False [mypy-*.workflows.test_modules] check_untyped_defs = False [mypy-*.managers.test_HDAManager] check_untyped_defs = False [mypy-*.managers.test_DatasetManager] check_untyped_defs = False [mypy-*.config.test_path_resolves_to] check_untyped_defs = False [mypy-*.config.test_config_values] check_untyped_defs = False [mypy-*.datatypes.test_storm] check_untyped_defs = False [mypy-*.datatypes.test_bcsl] check_untyped_defs = False ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/pyproject.toml0000644000175100017510000000401315234436334017547 0ustar00runnerrunner[build-system] build-backend = "setuptools.build_meta" requires = ["setuptools", "setuptools-scm>=8"] [project] dynamic = [ "authors", "dependencies", "description", "license", "optional-dependencies", "readme", "requires-python", "version", ] name = "galaxy-tool-util" classifiers = [ "Development Status :: 5 - Production/Stable", "Environment :: Console", "Intended Audience :: Developers", "Natural Language :: English", "Operating System :: POSIX", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", "Topic :: Software Development", "Topic :: Software Development :: Code Generators", "Topic :: Software Development :: Testing", ] keywords = ["Galaxy"] [project.scripts] galaxy-tool-format = "galaxy.tool_util.format:main" galaxy-tool-test = "galaxy.tool_util.verify.script:main" galaxy-tool-test-case-validation = "galaxy.tool_util.parameters.scripts.validate_test_cases:main" galaxy-tool-upgrade-advisor = "galaxy.tool_util.upgrade.script:main" validate-test-format = "galaxy.tool_util.validate_test_format:main" mulled-build = "galaxy.tool_util.deps.mulled.mulled_build:main" mulled-build-channel = "galaxy.tool_util.deps.mulled.mulled_build_channel:main" mulled-build-files = "galaxy.tool_util.deps.mulled.mulled_build_files:main" mulled-build-tool = "galaxy.tool_util.deps.mulled.mulled_build_tool:main" mulled-hash = "galaxy.tool_util.deps.mulled.mulled_hash:main" mulled-list = "galaxy.tool_util.deps.mulled.mulled_list:main" mulled-recommend = "galaxy.tool_util.deps.mulled.recommend:main" mulled-search = "galaxy.tool_util.deps.mulled.mulled_search:main" mulled-update-singularity-containers = "galaxy.tool_util.deps.mulled.mulled_update_singularity_containers:main" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/pytest.ini0000644000175100017510000000101015234436334016656 0ustar00runnerrunner[pytest] # Prevent execution of alembic/env.py at test collection stage (alembic.context not set). # Also ignore functional tests (galaxy_test/ and tool_shed/test/). addopts = --doctest-modules --ignore=src/galaxy/model/migrations/alembic/ --ignore=src/galaxy/tools/bundled/ --ignore=src/galaxy_test/ --ignore=src/tool_shed/test/ --ignore=src/tool_shed/webapp/model/migrations/alembic/ --doctest-continue-on-failure --verbosity=1 asyncio_mode = auto log_level = DEBUG consider_namespace_packages = true pythonpath = src ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0360568 galaxy_tool_util-26.1.1/scripts/0000755000175100017510000000000015234436377016333 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/scripts/commit_version.py0000644000175100017510000000266615234436334021745 0ustar00runnerrunner#!/usr/bin/env python # Modify version... import datetime import os import re import subprocess import sys def main(argv): DEV_RELEASE = os.environ.get("DEV_RELEASE", None) == "1" PROJECT_DIRECTORY = os.getcwd() PROJECT_DIRECTORY_NAME = os.path.basename(os.path.abspath(PROJECT_DIRECTORY)) PROJECT_NAME = PROJECT_DIRECTORY_NAME.replace("_", "-") version = argv[1] setup_cfg_path = os.path.join(PROJECT_DIRECTORY, "setup.cfg") if not DEV_RELEASE: history_path = os.path.join(PROJECT_DIRECTORY, "HISTORY.rst") with open(history_path) as f: history = f.read() today = datetime.datetime.today() today_str = today.strftime("%Y-%m-%d") history = history.replace(".dev0", f" ({today_str})") with open(history_path, "w") as f: f.write(history) with open(setup_cfg_path) as f: setup_cfg = f.read() setup_cfg = re.sub( r"^version = [\d\.]+\.dev\d+", f"version = {version}", setup_cfg, count=1, flags=re.MULTILINE ) with open(setup_cfg_path, "w") as f: f.write(setup_cfg) tag = f"galaxy-{PROJECT_NAME}-{version}" shell(["git", "commit", "-m", f"Version {version} of {PROJECT_NAME} (tag {tag}).", "HISTORY.rst", setup_cfg_path]) shell(["git", "tag", tag]) def shell(cmds, **kwds): p = subprocess.Popen(cmds, **kwds) return p.wait() if __name__ == "__main__": main(sys.argv) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/scripts/new_version.py0000644000175100017510000000433515234436334021241 0ustar00runnerrunner#!/usr/bin/env python # Modify version... import os import re import subprocess import sys from packaging.version import Version DEV_RELEASE = os.environ.get("DEV_RELEASE", None) == "1" PROJECT_DIRECTORY = os.getcwd() PROJECT_DIRECTORY_NAME = os.path.basename(os.path.abspath(PROJECT_DIRECTORY)) PROJECT_NAME = PROJECT_DIRECTORY_NAME.replace("_", "-") def main(argv): version = argv[1] version_obj = Version(version) if not DEV_RELEASE: # Discard epoch, release numbers after micro, pre-, post-, dev-release segments and local version new_version_tuple = list(version_obj.release[:3]) if len(new_version_tuple) < 3: new_version_tuple.append(0) else: new_version_tuple[2] += 1 new_version = ".".join(map(str, new_version_tuple)) else: dev_number = version_obj.dev assert dev_number is not None new_version = f"{version_obj.base_version}.dev{dev_number + 1}" history_path = os.path.join(PROJECT_DIRECTORY, "HISTORY.rst") if not DEV_RELEASE: with open(history_path) as f: history = f.read() def extend(from_str, line): from_str += "\n" return history.replace(from_str, from_str + line + "\n") history = extend( ".. to_doc", f""" --------------------- {new_version}.dev0 --------------------- """, ) with open(history_path, "w") as f: f.write(history) setup_cfg_path = os.path.join(PROJECT_DIRECTORY, "setup.cfg") with open(setup_cfg_path) as f: setup_cfg = f.read() if not DEV_RELEASE: setup_cfg = re.sub(r"^version = .*$", f"version = {new_version}.dev0", setup_cfg, count=1, flags=re.MULTILINE) else: setup_cfg = re.sub( r"^version = .*$", rf"version = {new_version}", setup_cfg, count=1, flags=re.MULTILINE, ) with open(setup_cfg_path, "w") as f: f.write(setup_cfg) shell(["git", "commit", "-m", f"Starting work on {PROJECT_NAME} {new_version}", "HISTORY.rst", setup_cfg_path]) def shell(cmds, **kwds): p = subprocess.Popen(cmds, **kwds) return p.wait() if __name__ == "__main__": main(sys.argv) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/scripts/print_version_for_release.py0000644000175100017510000000125415234436334024147 0ustar00runnerrunnerimport os import re from packaging.version import Version _version_re = re.compile(r"^version\s+=\s+(.*)", flags=re.MULTILINE) def main(): DEV_RELEASE = os.environ.get("DEV_RELEASE", None) == "1" PROJECT_DIRECTORY = os.getcwd() setup_cfg_path = os.path.join(PROJECT_DIRECTORY, "setup.cfg") with open(setup_cfg_path) as f: setup_cfg = f.read() version_match = _version_re.search(setup_cfg) assert version_match version = version_match.group(1) if not DEV_RELEASE: version_obj = Version(version) # Strip .devN print(version_obj.base_version) else: print(version) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1785871615.090092 galaxy_tool_util-26.1.1/setup.cfg0000644000175100017510000000177215234436377016474 0ustar00runnerrunner[metadata] author = Galaxy Project and Community author_email = galaxy-committers@lists.galaxyproject.org description = Galaxy tool and tool dependency utilities license = MIT license_files = LICENSE long_description = file: README.rst, HISTORY.rst long_description_content_type = text/x-rst name = galaxy-tool-util url = https://github.com/galaxyproject/galaxy version = 26.1.1 [options] include_package_data = True install_requires = galaxy-tool-util-models>=26.0 galaxy-util[template,image-util]>=22.1 conda-package-streaming jsonschema lxml!=4.2.2 MarkupSafe packaging pydantic>=2.7.4 PyYAML requests sortedcontainers typing-extensions package_dir = =src packages = find_namespace: python_requires = >=3.8 [options.extras_require] cwl = cwl-utils cwltool>=3.1.20230624081518 mulled = jinja2 Whoosh edam = edam-ontology extended-assertions = numpy pysam pillow tifffile test = Beaker pytest pytest-mock [options.packages.find] where = src [egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0306358 galaxy_tool_util-26.1.1/src/0000755000175100017510000000000015234436377015433 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1785871615.026636 galaxy_tool_util-26.1.1/src/galaxy/0000755000175100017510000000000015234436377016720 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0387747 galaxy_tool_util-26.1.1/src/galaxy/tool_util/0000755000175100017510000000000015234436377020732 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/__init__.py0000644000175100017510000000000015234436334023022 0ustar00runnerrunner././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0393052 galaxy_tool_util-26.1.1/src/galaxy/tool_util/biotools/0000755000175100017510000000000015234436377022564 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/biotools/__init__.py0000644000175100017510000000046315234436334024671 0ustar00runnerrunnerfrom .interface import ParsedBiotoolsEntry from .source import ( BiotoolsMetadataSource, BiotoolsMetadataSourceConfig, get_biotools_metadata_source, ) __all__ = ( "BiotoolsMetadataSource", "BiotoolsMetadataSourceConfig", "get_biotools_metadata_source", "ParsedBiotoolsEntry", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/biotools/interface.py0000644000175100017510000000324315234436334025071 0ustar00runnerrunnerimport re from typing import ( Any, Dict, List, Optional, ) TERM_PATTERN = re.compile(r"https?://edamontology.org/(.*)") class ParsedBiotoolsEntry: """Provide XML wrapper relevant entities from a bio.tool entry - topics and operations.""" biotoolsID: str edam_topics: List[str] edam_operations: List[str] class BiotoolsEntry: """Parse the RAW entries of interest for Galaxy from a bio.tools entry.""" biotoolsID: str topic: List[dict] function: List[dict] @staticmethod def from_json(from_json: Dict[str, Any]) -> "BiotoolsEntry": entry = BiotoolsEntry() entry.biotoolsID = from_json["biotoolsID"] entry.topic = from_json.get("topic", []) entry.function = from_json.get("function", []) return entry @property def edam_info(self) -> ParsedBiotoolsEntry: parsed = ParsedBiotoolsEntry() parsed.biotoolsID = self.biotoolsID parsed.edam_topics = list(set(simplify_edam_dicts(self.topic))) operations = [] for function in self.function: if "operation" in function: operations.extend(simplify_edam_dicts(function["operation"])) parsed.edam_operations = list(set(operations)) return parsed def simplify_edam_dicts(a_list: List[Dict[str, str]]): terms = [] for term in map(simplify_edam_dict, a_list): if term: terms.append(term) return terms def simplify_edam_dict(as_dict: Dict[str, str]) -> Optional[str]: uri = as_dict["uri"] match = TERM_PATTERN.match(uri) if match: return match.group(1) else: # TODO: log problem... return None ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/biotools/source.py0000644000175100017510000000703115234436334024430 0ustar00runnerrunnerimport functools import json import os from typing import ( Callable, Dict, List, Optional, ) from galaxy.util import ( DEFAULT_SOCKET_TIMEOUT, requests, ) from .interface import BiotoolsEntry class BiotoolsMetadataSource: def get_biotools_metadata(self, biotools_reference: str) -> Optional[BiotoolsEntry]: """Return a BiotoolsEntry if available.""" class GitContentBiotoolsMetadataSource(BiotoolsMetadataSource): """Parse entries from a repository clone of https://github.com/bio-tools/content.""" def __init__(self, content_directory): self._content_directory = content_directory def get_biotools_metadata(self, biotools_reference: str) -> Optional[BiotoolsEntry]: """Return a BiotoolsEntry if available.""" path = os.path.join(self._content_directory, "data", biotools_reference, f"{biotools_reference}.biotools.json") if not os.path.exists(path): return None with open(path) as f: content_json = json.load(f) return BiotoolsEntry.from_json(content_json) class InMemoryCache: backend: Dict[str, Optional[str]] = {} def get(self, key: str, createfunc: Callable[[], Optional[str]]): backend = self.backend if key not in backend: backend[key] = createfunc() return backend.get(key) class ApiBiotoolsMetadataSource(BiotoolsMetadataSource): """Parse entries from bio.tools API.""" def __init__(self, cache=None): self._cache = cache or InMemoryCache() def _raw_get_metadata(self, biotools_reference) -> Optional[str]: api_url = f"https://bio.tools/api/tool/{biotools_reference}?format=json" try: req = requests.get(api_url, timeout=DEFAULT_SOCKET_TIMEOUT) req.raise_for_status() req.encoding = req.apparent_encoding return req.text except Exception: return None def get_biotools_metadata(self, biotools_reference: str) -> Optional[BiotoolsEntry]: createfunc = functools.partial(self._raw_get_metadata, biotools_reference) content = self._cache.get(key=biotools_reference, createfunc=createfunc) if content is not None: return BiotoolsEntry.from_json(json.loads(content)) else: return None class CascadingBiotoolsMetadataSource(BiotoolsMetadataSource): def __init__(self, use_api=False, cache=None, content_directory: Optional[str] = None): sources: List[BiotoolsMetadataSource] = [] if content_directory: git_content_source = GitContentBiotoolsMetadataSource(content_directory) sources.append(git_content_source) if use_api: api_metadata_source = ApiBiotoolsMetadataSource(cache=cache) sources.append(api_metadata_source) self._sources = sources def get_biotools_metadata(self, biotools_reference: str) -> Optional[BiotoolsEntry]: for source in self._sources: entry = source.get_biotools_metadata(biotools_reference) if entry is not None: return entry return None class BiotoolsMetadataSourceConfig: use_api: bool = False content_directory: Optional[str] = None cache = None def get_biotools_metadata_source(metadata_source_config: BiotoolsMetadataSourceConfig) -> BiotoolsMetadataSource: return CascadingBiotoolsMetadataSource( use_api=metadata_source_config.use_api, content_directory=metadata_source_config.content_directory, cache=metadata_source_config.cache, ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0399432 galaxy_tool_util-26.1.1/src/galaxy/tool_util/client/0000755000175100017510000000000015234436377022210 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/client/__init__.py0000644000175100017510000000000015234436334024300 0ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/client/landing.py0000644000175100017510000000761315234436334024176 0ustar00runnerrunner#!/usr/bin/env python # . .venv/bin/activate; PYTHONPATH=lib python lib/galaxy/tool_util/client/landing.py -g http://localhost:8081 simple_workflow import argparse import random import string import sys from dataclasses import dataclass from typing import Optional import requests import yaml from galaxy.util.resources import resource_string DESCRIPTION = """ A small utility for demoing creation of tool and workflow landing endpoints. This allows an external developer to create tool and workflow forms with pre-populated parameters and handing them off with URLs to their clients/users. """ RANDOM_SECRET_LENGTH = 10 def load_default_catalog(): name = __name__ if __name__ == "__main__": name = "galaxy.tool_util.client.landing" catalog_yaml = resource_string(name, "landing_library.catalog.yml") return yaml.safe_load(catalog_yaml) @dataclass class Request: template_id: str catalog: str public: bool client_secret: Optional[str] galaxy_url: str @dataclass class Response: landing_url: str def generate_claim_url(request: Request) -> Response: template_id = request.template_id catalog_path = request.catalog galaxy_url = request.galaxy_url client_secret = request.client_secret if client_secret == "__GEN__": client_secret = "".join( random.choice(string.ascii_lowercase + string.digits) for _ in range(RANDOM_SECRET_LENGTH) ) if catalog_path: with open(catalog_path) as f: catalog = yaml.safe_load(f) else: catalog = load_default_catalog() template = catalog[template_id] if "tool_id" in template: template_type = "tool" elif "workflow_id" in template: template_type = "workflow" elif isinstance(template["request_state"], list): template_type = "file" else: template_type = "data" if client_secret: template["client_secret"] = client_secret if request.public: template["public"] = True landing_request_url = f"{galaxy_url}/api/{template_type}_landings" raw_response = requests.post( landing_request_url, json=template, ) try: raw_response.raise_for_status() except Exception: raise Exception("Request failed: %s", raw_response.json()) response = raw_response.json() response_type = "workflow" if template_type == "workflow" else "tool" url = f"{galaxy_url}/{response_type}_landings/{response['uuid']}" if client_secret: url = url + f"?secret={client_secret}" return Response(url) def arg_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(description=DESCRIPTION) parser.add_argument("template_id") parser.add_argument( "-g", "--galaxy-url", dest="galaxy_url", default="https://usegalaxy.org/", help="Galxy target for the landing request", ) parser.add_argument( "-c", "--catalog", dest="catalog", default=None, help="YAML catalog to load landing request templates from.", ) parser.add_argument( "-s", "--secret", dest="secret", default=None, help="An optional client secret to verify the request against, set to __GEN__ to generate one at random for this request.", ) parser.add_argument( "-p", "--public", dest="public", default=False, action="store_true", help="Declare the landing as public.", ) return parser def main(argv=None) -> None: if argv is None: argv = sys.argv[1:] args = arg_parser().parse_args(argv) request = Request( args.template_id, args.catalog, args.public, args.secret, args.galaxy_url, ) response = generate_claim_url(request) print(f"Your customized form is located at {response.landing_url}") if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/client/landing_library.catalog.yml0000644000175100017510000003561315234436334027505 0ustar00runnerrunner# Catalog of artifacts that the landing script can generate landing URLs for against # the target Galaxy. simple_workflow: # Encoded ID for a workflow to target - if workflow_target_type is stored_workflow this # will always be the latest version of that workflow and stored workflow ID should be provided. # If instead workflow_target_type is workflow - this is a particular version of a workflow and # the workflow ID should be provided. workflow_id: f2db41e1fa331b3e workflow_target_type: stored_workflow # request state provides prepopulated parameters for this workflow or stored workflow when the # user navigates to the landing URL. request_state: myinput: src: url url: https://raw.githubusercontent.com/galaxyproject/galaxy/dev/test-data/1.bed ext: txt int_workflow: workflow_id: f597429621d6eb2b workflow_target_type: stored_workflow request_state: int_input: 8 sheet_workflow: workflow_id: f2db41e1fa331b3e workflow_target_type: stored_workflow request_state: sample_sheet: class: Collection collection_type: sample_sheet name: test sample sheet elements: - class: File identifier: sample1 url: base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: txt deferred: False - class: File identifier: sample2 url: base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: txt deferred: False column_definitions: - type: int name: replicate optional: False - type: string name: condition optional: False rows: sample1: - 1 - control sample2: - 2 - treatment upload_file: request_state: - class: File name: Reference information location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "txt" - class: File name: Reference sequence location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" - class: File name: Read 1 location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq.gz" - class: Collection collection_type: list name: "my collection" elements: - class: File name: sample1 location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: File name: sample2 location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: File name: sample3 location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: Collection collection_type: list:paired name: "The List of Dataset Pairs" elements: - class: Collection collection_type: paired name: sample1 elements: - class: File name: forward location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: File name: reverse location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - name: sample2 class: Collection collection_type: paired elements: - class: File name: forward location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: File name: reverse location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: Collection collection_type: list:list:paired name: "Nested List of Dataset Pairs" elements: - class: Collection collection_type: list:paired name: treatment1 elements: - class: Collection collection_type: paired name: replicate1 elements: - class: File name: forward location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: File name: reverse location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: Collection collection_type: paired name: replicate2 elements: - class: File name: forward location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: File name: reverse location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: Collection collection_type: list:paired name: treatment2 elements: - class: Collection collection_type: paired name: replicate1 elements: - class: File name: forward location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: File name: reverse location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: Collection collection_type: paired name: replicate2 elements: - class: File name: forward location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: File name: reverse location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - class: File name: "sample1.fasta" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" tags: ['name:sample1'] - class: File name: "sample2.fasta" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" tags: ['name:sample2'] - class: File name: "sample1.fastq" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment1', 'group:replicate:replicate1'] - class: File name: "sample2.fastq" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment1', 'group:replicate:replicate2'] - class: File name: "sample2.fastq" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment1', 'group:replicate:replicate2'] - class: File name: "sample3.fastq" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment1', 'group:replicate:replicate2'] - class: File name: "sample3.fastq" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment2', 'group:replicate:replicate1'] - class: File name: "sample4.fastq" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment2', 'group:replicate:replicate2'] - class: File name: "Convert spaces and not newlines" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "txt" space_to_tab: true to_posix_lines: false - class: File name: "Convert newlines and not spaces" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" space_to_tab: false to_posix_lines: true - class: File name: "Reference genome" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq.gz" - class: File name: "Reference information" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "txt" dbkey: hg19 - class: File name: "Reference sequence" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" dbkey: hg19 - class: File name: "Reference annotation" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "gff" dbkey: hg19 - class: Collection collection_type: paired:paired name: "esoteric collection type" elements: - class: Collection collection_type: paired name: "forward" elements: - class: File location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" name: "forward" - class: File location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" name: "reverse" - class: Collection collection_type: paired name: "reverse" elements: - class: File location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" name: "forward" - class: File location: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" name: "reverse" upload_one_file: request_state: - class: Collection collection_type: list name: "my collection" elements: - class: File name: "Reference information" location: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "txt" upload: request_state: targets: - destination: {type: "hdas"} items: - src: url name: "Reference information" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "txt" - src: url name: "Reference sequence" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" - src: url url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq.gz" - destination: {type: "hdca"} collection_type: list name: "my collection" items: - src: url name: "sample1" url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - src: url name: sample2 url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - src: url name: sample3 url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - destination: {type: "hdca"} collection_type: list:paired name: "The List of Dataset Pairs" items: - name: sample1 items: - src: url name: forward url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - src: url name: reverse url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - name: sample2 items: - src: url name: forward url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - src: url name: reverse url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - destination: {type: "hdca"} collection_type: list:list:paired name: "Nested List of Dataset Pairs" items: - name: treatment1 items: - name: replicate1 items: - src: url name: forward url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - src: url name: reverse url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - name: replicate2 items: - src: url name: forward url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - src: url name: reverse url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - name: treatment2 items: - name: replicate1 items: - src: url name: forward url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - src: url name: reverse url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - name: replicate2 items: - src: url name: forward url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - src: url name: reverse url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" - destination: {type: "hdas"} items: - src: url name: "sample1.fasta" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" tags: ['name:sample1'] - src: url name: "sample2.fasta" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" tags: ['name:sample2'] - destination: {type: "hdas"} items: - src: url name: "sample1.fastq" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment1', 'group:replicate:replicate1'] - src: url name: "sample2.fastq" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment1', 'group:replicate:replicate2'] - src: url name: "sample3.fastq" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment2', 'group:replicate:replicate1'] - src: url name: "sample4.fastq" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq" tags: ['group:treatment:treatment2', 'group:replicate:replicate2'] - destination: {type: "hdas"} items: - src: url name: "Convert spaces and not newlines" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "txt" space_to_tab: true to_posix_lines: false - src: url name: "Convert newlines and not spaces" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" space_to_tab: false to_posix_lines: true - src: url url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq.gz" - destination: {type: "hdas"} items: - src: url name: "Reference information" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "txt" dbkey: hg19 - src: url name: "Reference sequence" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fasta" dbkey: hg19 - src: url url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "fastq.gz" - destination: {type: "hdca"} collection_type: paired:paired name: "esoteric collection type" items: - name: "forward" items: - src: url url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" name: "forward" items: - src: url url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" name: "reverse" - name: "reverse" items: - src: url url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" name: "forward" items: - src: url url: "base64://eyJ0c3JjIjogInRlc3QifQ==" ext: "txt" name: "reverse" upload_one: request_state: targets: - destination: {type: "hdas"} items: - src: url name: "Reference information" url: "base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: "txt" upload_sheet: request_state: - class: Collection collection_type: sample_sheet name: test sample sheet elements: - class: File identifier: sample1 url: base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: txt deferred: False - class: File identifier: sample2 url: base64://eyJ0ZXN0IjogInRlc3QifQ==" ext: txt deferred: False column_definitions: - type: int name: replicate optional: False - type: string name: condition optional: False rows: sample1: - 1 - control sample2: - 2 - treatment././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/client/staging.py0000644000175100017510000003727115234436334024221 0ustar00runnerrunner"""Client for staging inputs for Galaxy Tools and Workflows. Implement as a connector to serve a bridge between galactic_job_json utility and a Galaxy API library. """ import abc import json import logging import os from typing import ( Any, BinaryIO, Callable, Dict, List, Optional, Tuple, TYPE_CHECKING, Union, ) import yaml from typing_extensions import Literal from galaxy.tool_util.cwl.util import ( DirectoryUploadTarget, FileLiteralTarget, FileUploadTarget, galactic_job_json, ObjectUploadTarget, path_or_uri_to_uri, UploadTarget, ) if TYPE_CHECKING: from galaxy_test.base.api import ApiTestInteractor log = logging.getLogger(__name__) UPLOAD_TOOL_ID = "upload1" LOAD_TOOLS_FROM_PATH = True DEFAULT_USE_FETCH_API = True DEFAULT_FILE_TYPE = "auto" DEFAULT_DBKEY = "?" DEFAULT_DECOMPRESS = False class StagingInterface(metaclass=abc.ABCMeta): """Client that parses a job input and populates files into the Galaxy API. Abstract class that must override _post (and optionally other things such _attach_file, _log, etc..) to adapt to bioblend (for Planemo) or using the tool test interactor infrastructure. """ @abc.abstractmethod def _post(self, api_path: str, payload: Dict[str, Any]) -> Dict[str, Any]: """Make a post to the Galaxy API along supplied path.""" def _attach_file(self, path: str) -> BinaryIO: return open(path, "rb") def _tools_post(self, payload: Dict[str, Any]) -> Dict[str, Any]: tool_response = self._post("tools", payload) for job in tool_response.get("jobs", []): self._handle_job(job) return tool_response def _fetch_post(self, payload: Dict[str, Any]) -> Dict[str, Any]: tool_response = self._post("tools/fetch", payload) for job in tool_response.get("jobs", []): self._handle_job(job) return tool_response @abc.abstractmethod def _handle_job(self, job_response: Dict[str, Any]): """Implementer can decide if to wait for job(s) individually or not here.""" def stage( self, tool_or_workflow: Literal["tool", "workflow"], history_id: str, job: Optional[Dict[str, Any]] = None, job_path: Optional[str] = None, use_path_paste: bool = LOAD_TOOLS_FROM_PATH, to_posix_lines: bool = True, job_dir: str = ".", resolve_data: Optional[Callable[[str], Optional[str]]] = None, ) -> Tuple[Dict[str, Any], List[Dict[str, Any]]]: def upload_func_fetch(upload_target: UploadTarget) -> Dict[str, Any]: def _attach_file(upload_payload: Dict[str, Any], uri: str, index: int = 0) -> Dict[str, Union[str, bool]]: uri = path_or_uri_to_uri(uri) is_path = uri.startswith("file://") if not is_path or use_path_paste: return {"src": "url", "url": uri} else: path = uri[len("file://") :] upload_payload["__files"][f"files_{index}|file_data"] = self._attach_file(path) return {"src": "files"} fetch_payload = None if isinstance(upload_target, FileUploadTarget): file_path = upload_target.path file_type = upload_target.properties.get("filetype", None) or DEFAULT_FILE_TYPE dbkey = upload_target.properties.get("dbkey", None) or DEFAULT_DBKEY fetch_payload = _fetch_payload( history_id, file_type=file_type, dbkey=dbkey, to_posix_lines=to_posix_lines, decompress=upload_target.properties.get("decompress") or DEFAULT_DECOMPRESS, hashes=upload_target.properties.get("hashes"), ) name = upload_target.properties.get("name") or _file_path_to_name(file_path) if file_path is not None: src = _attach_file(fetch_payload, file_path) fetch_payload["targets"][0]["elements"][0].update(src) if upload_target.composite_data: composite_items = [] for i, composite_data in enumerate(upload_target.composite_data): composite_item_src = _attach_file(fetch_payload, composite_data, index=i) composite_items.append(composite_item_src) if "metadata" in upload_target.properties: fetch_payload["targets"][0]["elements"][0]["metadata"] = upload_target.properties["metadata"] fetch_payload["targets"][0]["elements"][0]["src"] = "composite" fetch_payload["targets"][0]["elements"][0]["composite"] = { "items": composite_items, } tags = upload_target.properties.get("tags") if tags: fetch_payload["targets"][0]["elements"][0]["tags"] = tags fetch_payload["targets"][0]["elements"][0]["name"] = name elif isinstance(upload_target, FileLiteralTarget): file_type = upload_target.properties.get("filetype", None) or DEFAULT_FILE_TYPE dbkey = upload_target.properties.get("dbkey", None) or DEFAULT_DBKEY fetch_payload = _fetch_payload(history_id, file_type=file_type, dbkey=dbkey) # For file literals - take them as is - never convert line endings. fetch_payload["targets"][0]["elements"][0].update( { "src": "pasted", "paste_content": upload_target.contents, "to_posix_lines": False, } ) tags = upload_target.properties.get("tags") if tags: fetch_payload["targets"][0]["elements"][0]["tags"] = tags name = upload_target.properties.get("name") if name: fetch_payload["targets"][0]["elements"][0]["name"] = name elif isinstance(upload_target, DirectoryUploadTarget): fetch_payload = _fetch_payload(history_id, file_type=upload_target.file_type) element = fetch_payload["targets"][0]["elements"][0] element["name"] = upload_target.name tar_path = upload_target.tar_path extra_files = _attach_file(fetch_payload, tar_path) extra_files["fuzzy_root"] = False extra_files["items_from"] = "archive" # {"src": "pasted", "paste_content": ""} because # we need some primary file even if we don't have one element["src"] = "pasted" element["paste_content"] = "" element["extra_files"] = extra_files elif isinstance(upload_target, ObjectUploadTarget): content = json.dumps(upload_target.object) fetch_payload = _fetch_payload(history_id, file_type="expression.json") fetch_payload["targets"][0]["elements"][0].update( { "src": "pasted", "paste_content": content, } ) tags = upload_target.properties.get("tags") if tags: fetch_payload["targets"][0]["elements"][0]["tags"] = tags else: raise ValueError(f"Unsupported type for upload_target: {type(upload_target)}") return self._fetch_post(fetch_payload) # Save legacy upload_func to target older Galaxy servers def upload_func(upload_target: UploadTarget) -> Dict[str, Any]: def _attach_file(upload_payload: Dict[str, Any], uri: str, index: int = 0) -> None: uri = path_or_uri_to_uri(uri) is_path = uri.startswith("file://") if not is_path or use_path_paste: upload_payload["inputs"][f"files_{index}|url_paste"] = uri else: path = uri[len("file://") :] upload_payload["__files"][f"files_{index}|file_data"] = self._attach_file(path) if isinstance(upload_target, FileUploadTarget): file_path = upload_target.path file_type = upload_target.properties.get("filetype", None) or DEFAULT_FILE_TYPE dbkey = upload_target.properties.get("dbkey", None) or DEFAULT_DBKEY upload_payload = _upload_payload(history_id, file_type=file_type, to_posix_lines=dbkey) name = upload_target.properties.get("name") or _file_path_to_name(file_path) upload_payload["inputs"]["files_0|auto_decompress"] = False upload_payload["inputs"]["auto_decompress"] = False if file_path is not None: _attach_file(upload_payload, file_path) upload_payload["inputs"]["files_0|NAME"] = name if upload_target.secondary_files: _attach_file(upload_payload, upload_target.secondary_files, index=1) upload_payload["inputs"]["files_1|type"] = "upload_dataset" upload_payload["inputs"]["files_1|auto_decompress"] = True upload_payload["inputs"]["file_count"] = "2" upload_payload["inputs"]["force_composite"] = "True" # galaxy.exceptions.RequestParameterInvalidException: Not input source type # defined for input '{'class': 'File', 'filetype': 'imzml', 'composite_data': # ['Example_Continuous.imzML', 'Example_Continuous.ibd']}'.\n"}]] if upload_target.composite_data: for i, composite_data in enumerate(upload_target.composite_data): upload_payload["inputs"][f"files_{i}|type"] = "upload_dataset" _attach_file(upload_payload, composite_data, index=i) self._log(f"upload_payload is {upload_payload}") return self._tools_post(upload_payload) elif isinstance(upload_target, FileLiteralTarget): # For file literals - take them as is - never convert line endings. file_type = upload_target.properties.get("filetype", None) or DEFAULT_FILE_TYPE payload = _upload_payload(history_id, file_type=file_type, auto_decompress=False, to_posix_lines=False) payload["inputs"]["files_0|url_paste"] = upload_target.contents return self._tools_post(payload) elif isinstance(upload_target, DirectoryUploadTarget): tar_path = upload_target.tar_path upload_payload = _upload_payload( history_id, file_type="tar", ) upload_payload["inputs"]["files_0|auto_decompress"] = False _attach_file(upload_payload, tar_path) tar_upload_response = self._tools_post(upload_payload) convert_payload = dict( tool_id="CONVERTER_tar_to_directory", tool_inputs={"input1": {"src": "hda", "id": tar_upload_response["outputs"][0]["id"]}}, history_id=history_id, ) convert_response = self._tools_post(convert_payload) assert "outputs" in convert_response, convert_response return convert_response elif isinstance(upload_target, ObjectUploadTarget): content = json.dumps(upload_target.object) payload = _upload_payload(history_id, file_type="expression.json") payload["files_0|url_paste"] = content return self._tools_post(payload) else: raise ValueError(f"Unsupported type for upload_target: {type(upload_target)}") def create_collection_func( element_identifiers: List[Dict[str, Any]], collection_type: str, rows: Optional[Dict[str, Any]] = None, name: Optional[str] = None, ) -> Dict[str, Any]: payload = { "name": name or "dataset collection", "instance_type": "history", "history_id": history_id, "element_identifiers": element_identifiers, "collection_type": collection_type, "fields": None if collection_type != "record" else "auto", "rows": rows, } return self._post("dataset_collections", payload) if job_path is not None: assert job is None with open(job_path) as f: job = yaml.safe_load(f) job_dir = os.path.dirname(os.path.abspath(job_path)) else: assert job is not None if self.use_fetch_api: upload = upload_func_fetch else: upload = upload_func return galactic_job_json( job, job_dir, upload, create_collection_func, tool_or_workflow, resolve_data=resolve_data, ) # extension point for planemo to override logging def _log(self, message): log.debug(message) @property @abc.abstractmethod def use_fetch_api(self): """Return true is this should use (modern) data fetch API.""" class InteractorStaging(StagingInterface): def __init__(self, galaxy_interactor: "ApiTestInteractor", use_fetch_api: bool = DEFAULT_USE_FETCH_API) -> None: self.galaxy_interactor = galaxy_interactor self._use_fetch_api = use_fetch_api def _post(self, api_path: str, payload: Dict[str, Any]) -> Dict[str, Any]: response = self.galaxy_interactor._post(api_path, payload, json=True) assert response.status_code == 200, response.text return response.json() def _handle_job(self, job_response: Dict[str, Any]): self.galaxy_interactor.wait_for_job(job_response["id"]) @property def use_fetch_api(self): return self._use_fetch_api def _file_path_to_name(file_path: Optional[str]) -> str: if file_path is not None: name = os.path.basename(file_path) else: name = "defaultname" return name def _upload_payload( history_id: str, file_type: str = DEFAULT_FILE_TYPE, dbkey: str = DEFAULT_DBKEY, **kwd ) -> Dict[str, Any]: """Adapted from BioBlend tools client.""" payload: Dict[str, Any] = {} payload["history_id"] = history_id payload["tool_id"] = UPLOAD_TOOL_ID tool_input: Dict[str, Any] = {} tool_input["file_type"] = file_type tool_input["dbkey"] = dbkey if not kwd.get("to_posix_lines", True): tool_input["files_0|to_posix_lines"] = False elif kwd.get("space_to_tab", False): tool_input["files_0|space_to_tab"] = "Yes" if "file_name" in kwd: tool_input["files_0|NAME"] = kwd["file_name"] tool_input["files_0|type"] = "upload_dataset" payload["inputs"] = tool_input payload["__files"] = {} return payload def _fetch_payload(history_id, file_type=DEFAULT_FILE_TYPE, dbkey=DEFAULT_DBKEY, **kwd): element = { "ext": file_type, "dbkey": dbkey, } for arg in ["to_posix_lines", "space_to_tab"]: if arg in kwd: element[arg] = kwd[arg] if kwd.get("hashes"): element["hashes"] = kwd["hashes"] if "file_name" in kwd: element["name"] = kwd["file_name"] if "decompress" in kwd: element["auto_decompress"] = kwd["decompress"] target = { "destination": {"type": "hdas"}, "elements": [element], "auto_decompress": False, } targets = [target] payload = {"history_id": history_id, "targets": targets, "__files": {}} return payload ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0414445 galaxy_tool_util-26.1.1/src/galaxy/tool_util/cwl/0000755000175100017510000000000015234436377021517 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/cwl/__init__.py0000644000175100017510000000072615234436334023626 0ustar00runnerrunnerfrom .cwltool_deps import needs_shell_quoting from .parser import ( tool_proxy, tool_proxy_from_persistent_representation, workflow_proxy, ) from .representation import ( to_cwl_job, to_galaxy_parameters, ) from .runtime_actions import handle_outputs __all__ = ( "tool_proxy", "tool_proxy_from_persistent_representation", "workflow_proxy", "handle_outputs", "to_cwl_job", "to_galaxy_parameters", "needs_shell_quoting", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/cwl/cwltool_deps.py0000644000175100017510000001061515234436334024563 0ustar00runnerrunner"""Logic for dealing with cwltool as an optional dependency. Use this as the import interface for cwltool and just call :func:`ensure_cwltool_available` before using any of the imported functionality at runtime. """ import re import warnings warnings.filterwarnings("ignore", message=r"[\n.]DEPRECATION: Python 2", module="cwltool") from galaxy.util import requests try: from cwl_utils.types import CWLObjectType except ImportError: try: from cwltool.utils import CWLObjectType # type: ignore[assignment, attr-defined, unused-ignore] except ImportError: CWLObjectType = object # type: ignore[assignment, misc] try: from cwltool import ( main, pathmapper, process, workflow, ) except ImportError: main = None # type: ignore[assignment] workflow = None # type: ignore[assignment] process = None # type: ignore[assignment] pathmapper = None # type: ignore[assignment] try: from cwltool.context import ( getdefault, LoadingContext, RuntimeContext, ) from cwltool.job import relink_initialworkdir from cwltool.stdfsaccess import StdFsAccess except ImportError: getdefault = None # type: ignore[assignment] LoadingContext = None # type: ignore[assignment,misc] relink_initialworkdir = None # type: ignore[assignment] RuntimeContext = None # type: ignore[assignment,misc] StdFsAccess = None # type: ignore[assignment,misc] try: from cwltool import load_tool except ImportError: load_tool = None # type: ignore[assignment] try: from cwltool import command_line_tool except ImportError: command_line_tool = None # type: ignore[assignment] try: from cwltool.load_tool import ( default_loader, resolve_and_validate_document, ) except ImportError: default_loader = None # type: ignore[assignment] resolve_and_validate_document = None # type: ignore[assignment] try: from cwltool.process import Process except ImportError: Process = None # type: ignore[assignment, misc] try: from cwltool.utils import ( JobsType, normalizeFilesDirs, OutputCallbackType, visit_class, ) except ImportError: JobsType = object # type: ignore[misc, unused-ignore] visit_class = None # type: ignore[assignment] normalizeFilesDirs = None # type: ignore[assignment] OutputCallbackType = None # type: ignore[misc] try: import schema_salad from schema_salad import ( ref_resolver, sourceline, ) from schema_salad.utils import yaml_no_ts except ImportError: schema_salad = None # type: ignore[assignment] ref_resolver = None # type: ignore[assignment] sourceline = None # type: ignore[assignment] yaml_no_ts = None # type: ignore[assignment] try: from ruamel.yaml.comments import CommentedMap except ImportError: CommentedMap = None # type: ignore[assignment,misc] needs_shell_quoting = re.compile(r"""(^$|[\s|&;()<>\'"$@])""").search # if set to True, file format checking is not performed. beta_relaxed_fmt_check = True def ensure_cwltool_available(): """Assert optional dependencies proxied via this module are available at runtime. Throw an ImportError with a description of the problem if they do not exist. """ if main is None or workflow is None: message = "This feature requires cwltool and dependencies to be available, they are not." if main is None: message += " cwltool is not unavailable." elif resolve_and_validate_document is None: message += " cwltool.load_tool.resolve_and_validate_document is unavailable - cwltool version is too old." if requests is None: message += " Library 'requests' unavailable." if schema_salad is None: message += " Library 'schema_salad' unavailable." raise ImportError(message) __all__ = ( "CommentedMap", "CWLObjectType", "default_loader", "ensure_cwltool_available", "getdefault", "JobsType", "load_tool", "LoadingContext", "main", "needs_shell_quoting", "normalizeFilesDirs", "OutputCallbackType", "pathmapper", "process", "Process", "ref_resolver", "relink_initialworkdir", "resolve_and_validate_document", "RuntimeContext", "schema_salad", "sourceline", "StdFsAccess", "visit_class", "workflow", "yaml_no_ts", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/cwl/parser.py0000644000175100017510000013424215234436334023364 0ustar00runnerrunner"""This module provides proxy objects around objects from the common workflow language reference implementation library cwltool. These proxies adapt cwltool to Galaxy features and abstract the library away from the rest of the framework. """ import copy import json import logging import os from abc import ( ABCMeta, abstractmethod, ) from typing import ( Any, cast, Dict, List, Optional, overload, TYPE_CHECKING, Union, ) from uuid import ( UUID, uuid4, ) from typing_extensions import ( Literal, TypedDict, ) from galaxy.exceptions import MessageException from galaxy.util import ( listify, safe_makedirs, ) from galaxy.util.bunch import Bunch from .cwltool_deps import ( beta_relaxed_fmt_check, ensure_cwltool_available, getdefault, normalizeFilesDirs, pathmapper, process, ref_resolver, relink_initialworkdir, RuntimeContext, sourceline, StdFsAccess, visit_class, yaml_no_ts, ) from .representation import ( field_to_field_type, FIELD_TYPE_REPRESENTATION, INPUT_TYPE, type_descriptions_for_field_types, USE_FIELD_TYPES, USE_STEP_PARAMETERS, ) from .schema import ( non_strict_non_validating_schema_loader, schema_loader, ) from .util import SECONDARY_FILES_EXTRA_PREFIX if TYPE_CHECKING: from .cwltool_deps import ( CWLObjectType, JobsType, OutputCallbackType, Process, workflow, ) from .schema import RawProcessReference log = logging.getLogger(__name__) JOB_JSON_FILE = ".cwl_job.json" DOCKER_REQUIREMENT = "DockerRequirement" SUPPORTED_TOOL_REQUIREMENTS = [ "CreateFileRequirement", "DockerRequirement", "EnvVarRequirement", "InitialWorkDirRequirement", "InlineJavascriptRequirement", "ResourceRequirement", "ShellCommandRequirement", "ScatterFeatureRequirement", "SchemaDefRequirement", "SubworkflowFeatureRequirement", "StepInputExpressionRequirement", "MultipleInputFeatureRequirement", "CredentialsRequirement", ] SUPPORTED_WORKFLOW_REQUIREMENTS = SUPPORTED_TOOL_REQUIREMENTS + [] ToolStateType = Dict[str, Union[None, str, bool, Dict[str, str]]] class InputInstanceDict(TypedDict, total=False): type: str name: str title: str label: str help: str optional: bool area: bool value: str collection_type: str class InputInstanceArrayDict(TypedDict): type: str name: str title: str blocks: List[InputInstanceDict] class ToolProxy(metaclass=ABCMeta): _class: str def __init__( self, tool: "Process", uuid: Union[UUID, str], raw_process_reference: Optional["RawProcessReference"] = None, tool_path: Optional[str] = None, ): self._tool = tool self.uuid = uuid self._tool_path = tool_path self._raw_process_reference = raw_process_reference # remove input parameter formats from CWL files so that cwltool # does not complain they are missing in the input data assert isinstance(self._tool.inputs_record_schema, dict) for input_field in self._tool.inputs_record_schema["fields"]: if "format" in input_field: del input_field["format"] def job_proxy(self, input_dict: Dict[str, Any], output_dict, job_directory: str = "."): """Build a cwltool.job.Job describing computation using a input_json Galaxy will generate mapping the Galaxy description of the inputs into a cwltool compatible variant. """ return JobProxy(self, input_dict, output_dict, job_directory=job_directory) @property def id(self): raw_id = self._tool.tool.get("id", None) return raw_id def galaxy_id(self) -> str: raw_id = self.id tool_id = None # don't reduce "search.cwl#index" to search if raw_id: tool_id = os.path.basename(raw_id) # tool_id = os.path.splitext(os.path.basename(raw_id))[0] if not tool_id: return str(self.uuid) if tool_id.startswith("#"): tool_id = tool_id[1:] return tool_id @abstractmethod def input_fields(self) -> list: """Return InputInstance objects describing mapping to Galaxy inputs.""" @abstractmethod def input_instances(self): """Return InputInstance objects describing mapping to Galaxy inputs.""" @abstractmethod def output_instances(self) -> List["OutputInstance"]: """Return OutputInstance objects describing mapping to Galaxy inputs.""" @abstractmethod def docker_identifier(self): """Return docker identifier for embedding in tool description.""" @abstractmethod def description(self): """Return description to tool.""" @abstractmethod def label(self): """Return label for tool.""" def to_persistent_representation(self): """Return a JSON representation of this tool. Not for serialization over the wire, but serialization in a database.""" persisted_obj = self._tool.tool persisted_obj["requirements"] = self.requirements if not persisted_obj.get("cwlVersion"): # This happens for any inline process, but getting it from metadata is correct for inline processes at least persisted_obj["cwlVersion"] = self._tool.metadata["cwlVersion"] return { "class": self._class, # Should maybe be yaml instead "raw_process_reference": persisted_obj, "uuid": self.uuid, } @staticmethod def from_persistent_representation( as_object: Dict[str, Any], strict_cwl_validation: bool = True, tool_directory: Optional[str] = None ) -> "ToolProxy": """Recover an object serialized with to_persistent_representation.""" if "class" not in as_object: raise Exception("Failed to deserialize tool proxy from JSON object - no class found.") loaded_object = tool_proxy( tool_object=as_object["raw_process_reference"], strict_cwl_validation=strict_cwl_validation, tool_directory=tool_directory, uuid=as_object.get("uuid"), ) return loaded_object @property def requirements(self) -> List: return getattr(self._tool, "requirements", []) def hints_or_requirements_of_class(self, class_name: str) -> List: reqs_and_hints = self.requirements + getattr(self._tool, "hints", []) return [hint for hint in reqs_and_hints if hint["class"] == class_name] def software_requirements(self) -> List: # Roughest imaginable pass at parsing requirements, really need to take in specs, handle # multiple versions, etc... requirements = [] for hint in self.hints_or_requirements_of_class("SoftwareRequirement"): packages = hint.get("packages", []) for package in packages: versions = package.get("version", []) first_version = None if not versions else versions[0] requirements.append((package["package"], first_version)) return requirements def resource_requirements(self) -> List: return self.hints_or_requirements_of_class("ResourceRequirement") def credentials_requirements(self) -> List: return self.hints_or_requirements_of_class("CredentialsRequirement") class CommandLineToolProxy(ToolProxy): _class = "CommandLineTool" def description(self): # Don't use description - typically too verbose. return "" def doc(self): # TODO: parse multiple lines and merge - valid in cwl-1.1 doc = self._tool.tool.get("doc") return doc def label(self): label = self._tool.tool.get("label") if label is not None: return label.partition(":")[0] # return substring before ':' else: return "" def input_fields(self) -> list: input_records_schema = self._eval_schema(self._tool.inputs_record_schema) if input_records_schema["type"] != "record": raise Exception("Unhandled CWL tool input structure") # TODO: handle this somewhere else? # schemadef_req_tool_param rval = [] for input in input_records_schema["fields"]: input_copy = copy.deepcopy(input) input_type = input.get("type") if isinstance(input_type, list) or isinstance(input_type, dict): rval.append(input_copy) continue if input_type in self._tool.schemaDefs: input_copy["type"] = self._tool.schemaDefs[input_type] rval.append(input_copy) return rval def _eval_schema(self, io_schema): schema_type = io_schema.get("type") if schema_type in self._tool.schemaDefs: io_schema = self._tool.schemaDefs[schema_type] return io_schema def input_instances(self): return [_outer_field_to_input_instance(_) for _ in self.input_fields()] def output_instances(self): outputs_schema = self._eval_schema(self._tool.outputs_record_schema) if outputs_schema["type"] != "record": raise Exception("Unhandled CWL tool output structure") return [_simple_field_to_output(output) for output in outputs_schema["fields"]] def docker_identifier(self): for hint in self.hints_or_requirements_of_class("DockerRequirement"): if "dockerImageId" in hint: return hint["dockerImageId"] else: return hint["dockerPull"] return None class ExpressionToolProxy(CommandLineToolProxy): _class = "ExpressionTool" class JobProxy: _is_command_line_job: bool def __init__(self, tool_proxy: ToolProxy, input_dict: Dict[str, Any], output_dict, job_directory: str): assert RuntimeContext is not None, "cwltool is not installed, cannot run CWL jobs" self._tool_proxy = tool_proxy self._input_dict = input_dict self._output_dict = output_dict self._job_directory = job_directory self._final_output: Optional[CWLObjectType] = None self._ok = True self._cwl_job: Optional[JobsType] = None self._normalize_job() def cwl_job(self): self._ensure_cwl_job_initialized() return self._cwl_job @property def is_command_line_job(self): self._ensure_cwl_job_initialized() return self._is_command_line_job def _ensure_cwl_job_initialized(self): if self._cwl_job is None: job_args = dict( basedir=self._job_directory, select_resources=self._select_resources, outdir=os.path.join(self._job_directory, "working"), tmpdir=os.path.join(self._job_directory, "cwltmp"), stagedir=os.path.join(self._job_directory, "cwlstagedir"), use_container=False, beta_relaxed_fmt_check=beta_relaxed_fmt_check, ) runtimeContext = RuntimeContext(job_args) self._cwl_job = next( self._tool_proxy._tool.job( self._input_dict, cast("OutputCallbackType", self._output_callback), runtimeContext ) ) self._is_command_line_job = hasattr(self._cwl_job, "command_line") def _normalize_job(self): # Somehow reuse whatever causes validate in cwltool... maybe? def pathToLoc(p): if "location" not in p and "path" in p: p["location"] = p["path"] del p["path"] runtime_context = RuntimeContext({}) make_fs_access = getdefault(runtime_context.make_fs_access, StdFsAccess) fs_access = make_fs_access(runtime_context.basedir) process.fill_in_defaults(self._tool_proxy._tool.tool["inputs"], self._input_dict, fs_access) visit_class(self._input_dict, ("File", "Directory"), pathToLoc) # TODO: Why doesn't fillInDefault fill in locations instead of paths? normalizeFilesDirs(self._input_dict) # TODO: validate like cwltool process _init_job. # validate.validate_ex(self.names.get_name("input_record_schema", ""), builder.job, # strict=False, logger=_logger_validation_warnings) def rewrite_inputs_for_staging(self): if hasattr(self._cwl_job, "pathmapper"): pass # DO SOMETHING LIKE THE FOLLOWING? # path_rewrites = {} # for f, p in self._cwl_job.pathmapper.items(): # if not p.staged: # continue # if p.type in ("File", "Directory"): # path_rewrites[p.resolved] = p.target # for key, value in self._input_dict.items(): # if key in path_rewrites: # self._input_dict[key]["location"] = path_rewrites[value] else: stagedir = os.path.join(self._job_directory, "cwlstagedir") safe_makedirs(stagedir) def stage_recursive(value): is_list = isinstance(value, list) is_dict = isinstance(value, dict) log.info(f"handling value {value}, is_list {is_list}, is_dict {is_dict}") if is_list: for val in value: stage_recursive(val) elif is_dict: if "location" in value and "basename" in value: location = value["location"] basename = value["basename"] if not location.endswith(basename): # TODO: sep()[-1] staged_loc = os.path.join(stagedir, basename) if not os.path.exists(staged_loc): os.symlink(location, staged_loc) value["location"] = staged_loc for dict_value in value.values(): stage_recursive(dict_value) else: log.info("skipping simple value...") stage_recursive(self._input_dict) def _select_resources(self, request, runtime_context=None): new_request = request.copy() new_request["cores"] = "$GALAXY_SLOTS" return new_request @property def command_line(self): if self.is_command_line_job: return self.cwl_job().command_line else: return ["true"] @property def stdin(self): if self.is_command_line_job: return self.cwl_job().stdin else: return None @property def stdout(self): if self.is_command_line_job: return self.cwl_job().stdout else: return None @property def stderr(self): if self.is_command_line_job: return self.cwl_job().stderr else: return None @property def environment(self): if self.is_command_line_job: return self.cwl_job().environment else: return {} @property def generate_files(self): if self.is_command_line_job: return self.cwl_job().generatefiles else: return {} def _output_callback(self, out: Optional["CWLObjectType"], process_status: str) -> None: self._process_status = process_status if process_status == "success": self._final_output = out else: self._ok = False log.info(f"Output are {out}, status is {process_status}") def collect_outputs(self, tool_working_directory: str, rcode: int): if not self.is_command_line_job: cwl_job = self.cwl_job() if RuntimeContext is not None: cwl_job.run(RuntimeContext({})) else: cwl_job.run() if not self._ok: raise Exception(f"Final process state not ok, [{self._process_status}]") return self._final_output else: return self.cwl_job().collect_outputs(tool_working_directory, rcode) def save_job(self) -> None: job_file = self._job_file(self._job_directory) job_objects = { # "tool_path": os.path.abspath(self._tool_proxy._tool_path), "tool_representation": self._tool_proxy.to_persistent_representation(), "job_inputs": self._input_dict, "output_dict": self._output_dict, } json.dump(job_objects, open(job_file, "w")) def _output_extra_files_dir(self, output_name): output_id = self.output_id(output_name) return os.path.join(self._job_directory, "outputs", f"dataset_{output_id}_files") def output_id(self, output_name): output_id = self._output_dict[output_name]["id"] return output_id def output_path(self, output_name): output_id = self._output_dict[output_name]["path"] return output_id def output_directory_contents_dir(self, output_name, create=False): extra_files_dir = self._output_extra_files_dir(output_name) return extra_files_dir def output_secondary_files_dir(self, output_name, create=False): extra_files_dir = self._output_extra_files_dir(output_name) secondary_files_dir = os.path.join(extra_files_dir, SECONDARY_FILES_EXTRA_PREFIX) if create and not os.path.exists(secondary_files_dir): safe_makedirs(secondary_files_dir) return secondary_files_dir def stage_files(self): cwl_job = self.cwl_job() def stageFunc(resolved_path, target_path): log.info(f"resolving {resolved_path} to {target_path}") try: os.symlink(resolved_path, target_path) except OSError: pass if hasattr(cwl_job, "pathmapper"): process.stage_files(cwl_job.pathmapper, stageFunc, ignore_writable=True, symlink=False) if hasattr(cwl_job, "generatefiles"): outdir = os.path.join(self._job_directory, "working") # TODO: Why doesn't cwl_job.generatemapper work? generate_mapper = pathmapper.PathMapper( cwl_job.generatefiles["listing"], outdir, outdir, separateDirs=False ) # TODO: figure out what inplace_update should be. inplace_update = cwl_job.inplace_update process.stage_files(generate_mapper, stageFunc, ignore_writable=inplace_update, symlink=False) relink_initialworkdir(generate_mapper, outdir, outdir, inplace_update=inplace_update) # else: expression tools do not have a path mapper. @staticmethod def _job_file(job_directory): return os.path.join(job_directory, JOB_JSON_FILE) class WorkflowProxy: def __init__(self, workflow: "workflow.Workflow", workflow_path: Optional[str] = None): self._workflow = workflow self._workflow_path = workflow_path self._step_proxies: Optional[List[Union[SubworkflowStepProxy, ToolStepProxy]]] = None @property def cwl_id(self): return self._workflow.tool["id"] def get_outputs_for_label(self, label): outputs = [] for output in self._workflow.tool["outputs"]: step, output_name = split_step_references( output["outputSource"], multiple=False, workflow_id=self.cwl_id, ) if step == label: output_id = output["id"] if "#" not in self.cwl_id: _, output_label = output_id.rsplit("#", 1) else: _, output_label = output_id.rsplit("/", 1) outputs.append( { "output_name": output_name, "label": output_label, } ) return outputs def tool_reference_proxies(self): """Fetch tool source definitions for all referenced tools.""" references: List[ToolProxy] = [] for step in self.step_proxies(): references.extend(step.tool_reference_proxies()) return references def step_proxies(self): if self._step_proxies is None: proxies = [] num_input_steps = len(self._workflow.tool["inputs"]) for i, step in enumerate(self._workflow.steps): proxies.append(build_step_proxy(self, step, i + num_input_steps)) self._step_proxies = proxies return self._step_proxies @property def runnables(self): runnables = [] for step in self._workflow.steps: if "run" in step.tool: runnables.append(step.tool["run"]) return runnables def cwl_ids_to_index(self, step_proxies): index = 0 cwl_ids_to_index = {} for input_dict in self._workflow.tool["inputs"]: cwl_ids_to_index[input_dict["id"]] = index index += 1 for step_proxy in step_proxies: cwl_ids_to_index[step_proxy.cwl_id] = index index += 1 return cwl_ids_to_index @property def output_labels(self): return [self.jsonld_id_to_label(o["id"]) for o in self._workflow.tool["outputs"]] def input_connections_by_step(self, step_proxies): cwl_ids_to_index = self.cwl_ids_to_index(step_proxies) input_connections_by_step = [] for step_proxy in step_proxies: input_connections_step: Dict[str, List[Dict[str, str]]] = {} for input_proxy in step_proxy.input_proxies: cwl_source_id = input_proxy.cwl_source_id input_name = input_proxy.input_name # Consider only allow multiple if MultipleInputFeatureRequirement is enabled for output_step_name, output_name in split_step_references(cwl_source_id, workflow_id=self.cwl_id): if "#" in self.cwl_id: sep_on = "/" else: sep_on = "#" output_step_id = self.cwl_id + sep_on + output_step_name if output_step_id not in cwl_ids_to_index: template = "Output [%s] does not appear in ID-to-index map [%s]." msg = template % (output_step_id, cwl_ids_to_index.keys()) raise AssertionError(msg) if input_name not in input_connections_step: input_connections_step[input_name] = [] input_connections_step[input_name].append( {"id": cwl_ids_to_index[output_step_id], "output_name": output_name, "input_type": "dataset"} ) input_connections_by_step.append(input_connections_step) return input_connections_by_step def to_dict(self): name = os.path.basename( self._workflow.tool.get("label") or self._workflow_path or "TODO - derive a name from ID" ) steps = {} step_proxies = self.step_proxies() input_connections_by_step = self.input_connections_by_step(step_proxies) index = 0 for i, input_dict in enumerate(self._workflow.tool["inputs"]): steps[index] = self.cwl_input_to_galaxy_step(input_dict, i) index += 1 for i, step_proxy in enumerate(step_proxies): input_connections = input_connections_by_step[i] steps[index] = step_proxy.to_dict(input_connections) index += 1 return { "name": name, "steps": steps, "annotation": self.cwl_object_to_annotation(self._workflow.tool), } def find_inputs_step_index(self, label): for i, input in enumerate(self._workflow.tool["inputs"]): if self.jsonld_id_to_label(input["id"]) == label: return i raise Exception(f"Failed to find index for label {label}") def jsonld_id_to_label(self, id): if "#" in self.cwl_id: return id.rsplit("/", 1)[-1] else: return id.rsplit("#", 1)[-1] def cwl_input_to_galaxy_step(self, input, i): input_type = input["type"] label = self.jsonld_id_to_label(input["id"]) input_as_dict = { "id": i, "label": label, "position": {"left": 0, "top": 0}, "annotation": self.cwl_object_to_annotation(input), "input_connections": {}, # Should the Galaxy API really require this? - Seems to. "workflow_outputs": self.get_outputs_for_label(label), } if input_type == "File" and "default" not in input: input_as_dict["type"] = "data_input" elif isinstance(input_type, dict) and input_type.get("type") == "array": input_as_dict["type"] = "data_collection_input" input_as_dict["collection_type"] = "list" elif isinstance(input_type, dict) and input_type.get("type") == "record": input_as_dict["type"] = "data_collection_input" input_as_dict["collection_type"] = "record" else: if USE_STEP_PARAMETERS: input_as_dict["type"] = "parameter_input" # TODO: dispatch on actual type so this doesn't always need # to be field - simpler types could be simpler inputs. tool_state: ToolStateType = {} tool_state["parameter_type"] = "field" default_set = "default" in input default_value = input.get("default") optional = default_set if isinstance(input_type, list) and "null" in input_type: optional = True if not optional and isinstance(input_type, dict) and "type" in input_type: raise ValueError("'type' detected in non-optional input dictionary.") if default_set: tool_state["default"] = {"src": "json", "value": default_value} tool_state["optional"] = optional input_as_dict["tool_state"] = tool_state else: input_as_dict["type"] = "data_input" # TODO: format = expression.json return input_as_dict def cwl_object_to_annotation(self, cwl_obj): return cwl_obj.get("doc", None) def tool_proxy( tool_path: Optional[str] = None, tool_object=None, strict_cwl_validation: bool = True, tool_directory: Optional[str] = None, uuid: Optional[Union[UUID, str]] = None, ) -> ToolProxy: """Provide a proxy object to cwltool data structures to just grab relevant data. """ ensure_cwltool_available() return _to_cwl_tool_object( tool_path=tool_path, tool_object=tool_object, strict_cwl_validation=strict_cwl_validation, tool_directory=tool_directory, uuid=uuid, ) def tool_proxy_from_persistent_representation( persisted_tool: Dict[str, Any], strict_cwl_validation: bool = True, tool_directory: Optional[str] = None ) -> ToolProxy: """Load a ToolProxy from a previously persisted representation.""" ensure_cwltool_available() return ToolProxy.from_persistent_representation( persisted_tool, strict_cwl_validation=strict_cwl_validation, tool_directory=tool_directory ) def workflow_proxy(workflow_path: str, strict_cwl_validation: bool = True) -> WorkflowProxy: ensure_cwltool_available() return _to_cwl_workflow_object(workflow_path, strict_cwl_validation=strict_cwl_validation) def load_job_proxy(job_directory: str, strict_cwl_validation: bool = True) -> JobProxy: ensure_cwltool_available() job_objects_path = os.path.join(job_directory, JOB_JSON_FILE) job_objects = json.load(open(job_objects_path)) job_inputs = job_objects["job_inputs"] output_dict = job_objects["output_dict"] persisted_tool = job_objects["tool_representation"] cwl_tool = tool_proxy_from_persistent_representation( persisted_tool=persisted_tool, strict_cwl_validation=strict_cwl_validation ) return cwl_tool.job_proxy(job_inputs, output_dict, job_directory=job_directory) def _to_cwl_tool_object( tool_path: Optional[str] = None, tool_object=None, strict_cwl_validation: bool = False, tool_directory: Optional[str] = None, uuid: Optional[Union[UUID, str]] = None, ) -> ToolProxy: if uuid is None: uuid = str(uuid4()) schema_loader = _schema_loader(strict_cwl_validation) if tool_path is not None: assert tool_object is None raw_process_reference = schema_loader.raw_process_reference(tool_path) cwl_tool = schema_loader.tool( raw_process_reference=raw_process_reference, ) elif tool_object is not None: # Allow loading tools from YAML... as_str = json.dumps(tool_object) tool_object = yaml_no_ts().load(as_str) path = tool_directory if path is None: path = os.getcwd() uri = f"{ref_resolver.file_uri(path)}/" sourceline.add_lc_filename(tool_object, uri) raw_process_reference = schema_loader.raw_process_reference_for_object(tool_object, uri=uri) cwl_tool = schema_loader.tool( raw_process_reference=raw_process_reference, ) else: raise ValueError("Either tool_path or tool_object should be defined") if isinstance(cwl_tool, int): raise Exception("Failed to load tool.") raw_tool = cwl_tool.tool # Apply Galaxy hacks to CWL tool representation to bridge semantic differences # between Galaxy and cwltool. _hack_cwl_requirements(cwl_tool) check_requirements(raw_tool) return _cwl_tool_object_to_proxy(cwl_tool, uuid, raw_process_reference=raw_process_reference, tool_path=tool_path) def _cwl_tool_object_to_proxy( cwl_tool: "Process", uuid: Union[UUID, str], raw_process_reference: Optional["RawProcessReference"] = None, tool_path: Optional[str] = None, ) -> ToolProxy: raw_tool = cwl_tool.tool if "class" not in raw_tool: raise Exception("File does not declare a class, not a valid Draft 3+ CWL tool.") process_class = raw_tool["class"] if process_class == "CommandLineTool": proxy_class = CommandLineToolProxy elif process_class == "ExpressionTool": proxy_class = ExpressionToolProxy else: raise Exception("File not a CWL CommandLineTool.") top_level_object = tool_path is not None if top_level_object and ("cwlVersion" not in raw_tool): raise Exception("File does not declare a CWL version, pre-draft 3 CWL tools are not supported.") return proxy_class(cwl_tool, uuid, raw_process_reference, tool_path) def _to_cwl_workflow_object(workflow_path: str, strict_cwl_validation: bool = True) -> WorkflowProxy: cwl_workflow = _schema_loader(strict_cwl_validation).tool(path=workflow_path) raw_workflow = cwl_workflow.tool check_requirements(raw_workflow, tool=False) return WorkflowProxy(cwl_workflow, workflow_path) def _schema_loader(strict_cwl_validation: bool): return schema_loader if strict_cwl_validation else non_strict_non_validating_schema_loader def _hack_cwl_requirements(cwl_tool): move_to_hints: List[int] = [] for i, requirement in enumerate(cwl_tool.requirements): if requirement["class"] == DOCKER_REQUIREMENT: move_to_hints.insert(0, i) for i in move_to_hints: del cwl_tool.requirements[i] cwl_tool.hints.append(requirement) def check_requirements(rec, tool=True): if isinstance(rec, dict): if "requirements" in rec: for r in rec["requirements"]: if tool: possible = SUPPORTED_TOOL_REQUIREMENTS else: possible = SUPPORTED_WORKFLOW_REQUIREMENTS if r["class"] not in possible: raise Exception(f"Unsupported requirement {r['class']}") for d in rec: check_requirements(rec[d], tool=tool) if isinstance(rec, list): for d in rec: check_requirements(d, tool=tool) def split_step_references(step_references, workflow_id=None, multiple=True): """Split a CWL step input or output reference into step id and name.""" # Trim off the workflow id part of the reference. step_references = listify(step_references) split_references = [] for step_reference in step_references: if workflow_id is None: # This path works fine for some simple workflows - but not so much # for subworkflows (maybe same for $graph workflows?) assert "#" in step_reference _, step_reference = step_reference.split("#", 1) else: if "#" in workflow_id: sep_on = "/" else: sep_on = "#" expected_prefix = workflow_id + sep_on if not step_reference.startswith(expected_prefix): raise AssertionError(f"step_reference [{step_reference}] doesn't start with {expected_prefix}") step_reference = step_reference[len(expected_prefix) :] # Now just grab the step name and input/output name. assert "#" not in step_reference if "/" in step_reference: step_name, io_name = step_reference.split("/", 1) else: # Referencing an input, not a step. # In Galaxy workflows input steps have an implicit output named # "output" for consistency with tools - in cwl land # just the input name is referenced. step_name = step_reference io_name = "output" split_references.append((step_name, io_name)) if multiple: return split_references else: assert len(split_references) == 1, split_references return split_references[0] def build_step_proxy(workflow_proxy: WorkflowProxy, step: "workflow.WorkflowStep", index: int): step_type = step.embedded_tool.tool["class"] if step_type == "Workflow": return SubworkflowStepProxy(workflow_proxy, step, index) else: return ToolStepProxy(workflow_proxy, step, index) class InputProxy: def __init__(self, step_proxy, cwl_input): self._cwl_input = cwl_input self.step_proxy = step_proxy self.workflow_proxy = step_proxy._workflow_proxy cwl_input_id = cwl_input["id"] cwl_source_id = cwl_input.get("source", None) if cwl_source_id is None: if "valueFrom" not in cwl_input and "default" not in cwl_input: msg = f"Workflow step input must define a source, a valueFrom, or a default value. Obtained [{cwl_input}]." raise MessageException(msg) assert cwl_input_id step_name, input_name = split_step_references( cwl_input_id, multiple=False, workflow_id=step_proxy.cwl_workflow_id ) self.step_name = step_name self.input_name = input_name self.cwl_input_id = cwl_input_id self.cwl_source_id = cwl_source_id scatter_inputs = [ split_step_references(i, multiple=False, workflow_id=step_proxy.cwl_workflow_id)[1] for i in listify(step_proxy._step.tool.get("scatter", [])) ] scatter = self.input_name in scatter_inputs self.scatter = scatter def to_dict(self): as_dict = { "name": self.input_name, } if "linkMerge" in self._cwl_input: as_dict["merge_type"] = self._cwl_input["linkMerge"] if "scatterMethod" in self.step_proxy._step.tool: as_dict["scatter_type"] = self.step_proxy._step.tool.get("scatterMethod", "dotproduct") else: as_dict["scatter_type"] = "dotproduct" if self.scatter else "disabled" if "valueFrom" in self._cwl_input: # TODO: Add a table for expressions - mark the type as CWL 1.0 JavaScript. as_dict["value_from"] = self._cwl_input["valueFrom"] if "default" in self._cwl_input: as_dict["default"] = self._cwl_input["default"] return as_dict class BaseStepProxy: def __init__(self, workflow_proxy: WorkflowProxy, step: "workflow.WorkflowStep", index: int): self._workflow_proxy = workflow_proxy self._step = step self._index = index self._uuid = str(uuid4()) cwl_inputs = self._step.tool["inputs"] self.input_proxies = [InputProxy(self, cwl_input) for cwl_input in cwl_inputs] @property def step_class(self): return self.cwl_tool_object.tool["class"] @property def cwl_id(self): return self._step.id @property def cwl_workflow_id(self): return self._workflow_proxy.cwl_id @property def requirements(self): return self._step.requirements @property def hints(self): return self._step.hints @property def label(self): label = self._workflow_proxy.jsonld_id_to_label(self._step.id) return label def galaxy_workflow_outputs_list(self): return self._workflow_proxy.get_outputs_for_label(self.label) @property def cwl_tool_object(self): return self._step.embedded_tool def inputs_to_dicts(self): inputs_as_dicts = [] for input_proxy in self.input_proxies: inputs_as_dicts.append(input_proxy.to_dict()) return inputs_as_dicts class ToolStepProxy(BaseStepProxy): def __init__(self, workflow_proxy, step, index): super().__init__(workflow_proxy, step, index) self._tool_proxy = None @property def tool_proxy(self): # Needs to be cached so UUID that is loaded matches UUID generated with to_dict. if self._tool_proxy is None: self._tool_proxy = _cwl_tool_object_to_proxy(self.cwl_tool_object, uuid=str(uuid4())) return self._tool_proxy def tool_reference_proxies(self): # Return a list so we can handle subworkflows recursively. return [self.tool_proxy] def to_dict(self, input_connections): # We need to stub out null entries for things getting replaced by # connections. This doesn't seem ideal - consider just making Galaxy # handle this. tool_state: ToolStateType = {} for input_name in input_connections.keys(): tool_state[input_name] = None outputs = self.galaxy_workflow_outputs_list() when_expression = self._step.tool.get("when") rval = { "id": self._index, "tool_uuid": self.tool_proxy.uuid, # TODO: make sure this is respected... "label": self.label, "position": {"left": 0, "top": 0}, "type": "tool", "annotation": self._workflow_proxy.cwl_object_to_annotation(self._step.tool), "input_connections": input_connections, "inputs": self.inputs_to_dicts(), "workflow_outputs": outputs, } if when_expression: rval["when"] = when_expression return rval class SubworkflowStepProxy(BaseStepProxy): def __init__(self, workflow_proxy, step, index): super().__init__(workflow_proxy, step, index) self._subworkflow_proxy = None def to_dict(self, input_connections): outputs = self.galaxy_workflow_outputs_list() for key, input_connection_list in input_connections.items(): input_subworkflow_step_id = self.subworkflow_proxy.find_inputs_step_index(key) for input_connection in input_connection_list: input_connection["input_subworkflow_step_id"] = input_subworkflow_step_id return { "id": self._index, "label": self.label, "position": {"left": 0, "top": 0}, "type": "subworkflow", "subworkflow": self.subworkflow_proxy.to_dict(), "annotation": self.subworkflow_proxy.cwl_object_to_annotation(self._step.tool), "input_connections": input_connections, "inputs": self.inputs_to_dicts(), "workflow_outputs": outputs, } def tool_reference_proxies(self): return self.subworkflow_proxy.tool_reference_proxies() @property def subworkflow_proxy(self): if self._subworkflow_proxy is None: self._subworkflow_proxy = WorkflowProxy(self.cwl_tool_object) return self._subworkflow_proxy def _outer_field_to_input_instance(field): field_type = field_to_field_type(field) # Must be a list if in here? if not isinstance(field_type, list): field_type = [field_type] name, label, description = _field_metadata(field) case_name = "_cwl__type_" case_label = f"Specify Parameter {label} As" def value_input(type_description): value_name = "_cwl__value_" value_label = label value_description = description return InputInstance( value_name, value_label, value_description, input_type=type_description.galaxy_param_type, collection_type=type_description.collection_type, ) select_options = [] case_options = [] type_descriptions = type_descriptions_for_field_types(field_type) for type_description in type_descriptions: select_options.append({"value": type_description.name, "label": type_description.label}) input_instances = [] if type_description.uses_param: input_instances.append(value_input(type_description)) case_options.append((type_description.name, input_instances)) # If there is more than one way to represent this parameter - produce a conditional # requesting user to ask for what form they want to submit the data in, else just map # a simple Galaxy parameter. if len(case_options) > 1 and not USE_FIELD_TYPES: case_input = SelectInputInstance( name=case_name, label=case_label, description=False, options=select_options, ) return ConditionalInstance(name, case_input, case_options) else: if len(case_options) > 1: only_type_description = FIELD_TYPE_REPRESENTATION else: only_type_description = type_descriptions[0] return InputInstance( name, label, description, input_type=only_type_description.galaxy_param_type, collection_type=only_type_description.collection_type, ) # Older array to repeat handling, now we are just representing arrays as # dataset collections - we should offer a blended approach in the future. # if field_type in simple_map_type_map.keys(): # input_type = simple_map_type_map[field_type] # return {"input_type": input_type, "array": False} # elif field_type == "array": # if isinstance(field["type"], dict): # array_type = field["type"]["items"] # else: # array_type = field["items"] # if array_type in simple_map_type_map.keys(): # input_type = simple_map_type_map[array_type] # return {"input_type": input_type, "array": True} # else: # raise Exception("Unhandled simple field type encountered - [%s]." % field_type) def _field_metadata(field): name = field["name"] label = field.get("label", None) description = field.get("doc", None) return name, label, description def _simple_field_to_output(field): name = field["name"] output_data_class = field["type"] output_instance = OutputInstance(name, output_data_type=output_data_class, output_type=OUTPUT_TYPE.GLOB) return output_instance class ConditionalInstance: def __init__(self, name, case, whens): self.input_type = INPUT_TYPE.CONDITIONAL self.name = name self.case = case self.whens = whens def to_dict(self): as_dict = dict( name=self.name, type=INPUT_TYPE.CONDITIONAL, test=self.case.to_dict(), when={}, ) for value, block in self.whens: as_dict["when"][value] = [i.to_dict() for i in block] return as_dict class SelectInputInstance: def __init__(self, name, label, description, options): self.input_type = INPUT_TYPE.SELECT self.name = name self.label = label self.description = description self.options = options def to_dict(self): # TODO: serialize options... as_dict = dict( name=self.name, label=self.label or self.name, help=self.description, type=self.input_type, options=self.options, ) return as_dict class InputInstance: def __init__(self, name, label, description, input_type, array=False, area=False, collection_type=None): self.input_type = input_type self.collection_type = collection_type self.name = name self.label = label self.description = description self.required = True self.array = array self.area = area @overload def to_dict(self, itemwise: Literal[False]) -> InputInstanceDict: ... @overload def to_dict(self, itemwise: Literal[True]) -> Union[InputInstanceDict, InputInstanceArrayDict]: ... def to_dict(self, itemwise: bool = True) -> Union[InputInstanceDict, InputInstanceArrayDict]: if itemwise and self.array: return InputInstanceArrayDict( type="repeat", name=f"{self.name}_repeat", title=f"{self.name}", blocks=[self.to_dict(itemwise=False)] ) else: as_dict = InputInstanceDict( name=self.name, label=self.label or self.name, help=self.description, type=self.input_type, optional=not self.required, ) if self.area: as_dict["area"] = True if self.input_type == INPUT_TYPE.INTEGER: as_dict["value"] = "0" if self.input_type == INPUT_TYPE.FLOAT: as_dict["value"] = "0.0" elif self.input_type == INPUT_TYPE.DATA_COLLECTION: as_dict["collection_type"] = self.collection_type return as_dict OUTPUT_TYPE = Bunch( GLOB="glob", STDOUT="stdout", ) # TODO: Different subclasses - this is representing different types of things. class OutputInstance: def __init__(self, name: str, output_data_type, output_type, path=None, fields=None): self.name = name self.output_data_type = output_data_type self.output_type = output_type self.path = path self.fields = fields __all__ = ( "tool_proxy", "load_job_proxy", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/cwl/representation.py0000644000175100017510000004412715234436334025134 0ustar00runnerrunner"""This module is responsible for converting between Galaxy's tool input description and the CWL description for a job json.""" import json import logging import os from enum import Enum from typing import ( Any, NamedTuple, Optional, ) from galaxy.exceptions import RequestParameterInvalidException from galaxy.util import ( safe_makedirs, string_as_bool, ) from .util import set_basename_and_derived_properties log = logging.getLogger(__name__) NOT_PRESENT = object() NO_GALAXY_INPUT = object() class INPUT_TYPE(str, Enum): DATA = "data" INTEGER = "integer" FLOAT = "float" TEXT = "text" BOOLEAN = "boolean" SELECT = "select" FIELD = "field" CONDITIONAL = "conditional" DATA_COLLECTION = "data_collection" # There are two approaches to mapping CWL tool state to Galaxy tool state # one is to map CWL types to compound Galaxy tool parameters combinations # with conditionals and the other is to use a new Galaxy parameter type that # allows unions, optional specifications, etc.... The problem with the former # is that it doesn't work with the workflow parameters for instance and is # very complex on the backend. The problem with the latter is that the GUI # for this parameter type is undefined curently. USE_FIELD_TYPES = True # There are two approaches to mapping CWL workflow inputs to Galaxy workflow # steps. The first is to simply map everything to expressions and stick them into # files and use data inputs - the second is to use parameter_input steps with # fields types. We are dispatching on USE_FIELD_TYPES for now - to choose but # may diverge later? # There are open issues with each approach: # - Mapping everything to files makes the GUI harder to imagine but the backend # easier to manage in someways. USE_STEP_PARAMETERS = USE_FIELD_TYPES class TypeRepresentation(NamedTuple): name: str galaxy_param_type: Any label: str collection_type: Optional[str] @property def uses_param(self): return self.galaxy_param_type is not NO_GALAXY_INPUT TYPE_REPRESENTATIONS = [ TypeRepresentation("null", NO_GALAXY_INPUT, "no input", None), TypeRepresentation("integer", INPUT_TYPE.INTEGER, "an integer", None), TypeRepresentation("float", INPUT_TYPE.FLOAT, "a decimal number", None), TypeRepresentation("double", INPUT_TYPE.FLOAT, "a decimal number", None), TypeRepresentation("file", INPUT_TYPE.DATA, "a dataset", None), TypeRepresentation("directory", INPUT_TYPE.DATA, "a directory", None), TypeRepresentation("boolean", INPUT_TYPE.BOOLEAN, "a boolean", None), TypeRepresentation("text", INPUT_TYPE.TEXT, "a simple text field", None), TypeRepresentation("record", INPUT_TYPE.DATA_COLLECTION, "record as a dataset collection", "record"), TypeRepresentation("json", INPUT_TYPE.TEXT, "arbitrary JSON structure", None), TypeRepresentation("array", INPUT_TYPE.DATA_COLLECTION, "as a dataset list", "list"), TypeRepresentation("enum", INPUT_TYPE.TEXT, "enum value", None), # TODO: make this a select... TypeRepresentation("field", INPUT_TYPE.FIELD, "arbitrary JSON structure", None), ] FIELD_TYPE_REPRESENTATION = TYPE_REPRESENTATIONS[-1] if not USE_FIELD_TYPES: CWL_TYPE_TO_REPRESENTATIONS = { "Any": ["integer", "float", "file", "boolean", "text", "record", "json"], "org.w3id.cwl.salad.Any": ["integer", "float", "file", "boolean", "text", "record", "json"], "array": ["array"], "string": ["text"], "boolean": ["boolean"], "int": ["integer"], "float": ["float"], "File": ["file"], "org.w3id.cwl.cwl.File": ["file"], "Directory": ["directory"], "org.w3id.cwl.cwl.Directory": ["directory"], "null": ["null"], "record": ["record"], } else: CWL_TYPE_TO_REPRESENTATIONS = { "Any": ["field"], "org.w3id.cwl.salad.Any": ["field"], "array": ["array"], "string": ["text"], "boolean": ["boolean"], "int": ["integer"], "float": ["float"], "File": ["file"], "org.w3id.cwl.cwl.File": ["file"], "Directory": ["directory"], "org.w3id.cwl.cwl.Directory": ["directory"], "null": ["null"], "record": ["record"], "enum": ["enum"], "double": ["double"], } def type_representation_from_name(type_representation_name): for type_representation in TYPE_REPRESENTATIONS: if type_representation.name == type_representation_name: return type_representation else: raise ValueError(f"No type representation for {type_representation_name}") def type_descriptions_for_field_types(field_types): type_representation_names = set() for field_type in field_types: if isinstance(field_type, dict) and field_type.get("type"): field_type = field_type.get("type") try: type_representation_names_for_field_type = CWL_TYPE_TO_REPRESENTATIONS.get(field_type) except TypeError: raise Exception(f"Failed to convert field_type {field_type}") if type_representation_names_for_field_type is None: raise Exception(f"Failed to convert type {field_type}") type_representation_names.update(type_representation_names_for_field_type) type_representations = [] for type_representation in TYPE_REPRESENTATIONS: if type_representation.name in type_representation_names: type_representations.append(type_representation) return type_representations def dataset_wrapper_to_file_json(inputs_dir, dataset_wrapper): if dataset_wrapper.ext == "expression.json": with open(dataset_wrapper.get_file_name()) as f: return json.load(f) if dataset_wrapper.ext == "directory": return dataset_wrapper_to_directory_json(inputs_dir, dataset_wrapper) extra_files_path = dataset_wrapper.extra_files_path secondary_files_path = os.path.join(extra_files_path, "__secondary_files__") path = str(dataset_wrapper) raw_file_object = {"class": "File"} if os.path.exists(secondary_files_path): safe_makedirs(inputs_dir) name = os.path.basename(path) new_input_path = os.path.join(inputs_dir, name) os.symlink(path, new_input_path) secondary_files = [] for secondary_file_name in os.listdir(secondary_files_path): secondary_file_path = os.path.join(secondary_files_path, secondary_file_name) target = os.path.join(inputs_dir, secondary_file_name) log.info(f"linking [{secondary_file_path}] to [{target}]") os.symlink(secondary_file_path, target) is_dir = os.path.isdir(os.path.realpath(secondary_file_path)) secondary_files.append({"class": "File" if not is_dir else "Directory", "location": target}) raw_file_object["secondaryFiles"] = secondary_files path = new_input_path raw_file_object["location"] = path # Verify it isn't a NoneDataset if dataset_wrapper.unsanitized: raw_file_object["size"] = int(dataset_wrapper.get_size()) set_basename_and_derived_properties( raw_file_object, str(dataset_wrapper.created_from_basename or dataset_wrapper.name) ) return raw_file_object def dataset_wrapper_to_directory_json(inputs_dir, dataset_wrapper): assert dataset_wrapper.ext == "directory" # get directory name archive_name = str(dataset_wrapper.created_from_basename or dataset_wrapper.name) nameroot, nameext = os.path.splitext(archive_name) directory_name = nameroot # assume archive file name contains the directory name # get archive location try: archive_location = dataset_wrapper.unsanitized.get_file_name() except Exception: archive_location = None directory_json = { "location": dataset_wrapper.extra_files_path, "class": "Directory", "name": directory_name, "archive_location": archive_location, "archive_nameext": nameext, "archive_nameroot": nameroot, } return directory_json def collection_wrapper_to_array(inputs_dir, wrapped_value): rval = [] for value in wrapped_value: rval.append(dataset_wrapper_to_file_json(inputs_dir, value)) return rval def collection_wrapper_to_record(inputs_dir, wrapped_value): rval = {} for key, value in wrapped_value.items(): rval[key] = dataset_wrapper_to_file_json(inputs_dir, value) return rval def to_cwl_job(tool, param_dict, local_working_directory): """tool is Galaxy's representation of the tool and param_dict is the parameter dictionary with wrapped values. """ tool_proxy = tool._cwl_tool_proxy input_fields = tool_proxy.input_fields() inputs = tool.inputs input_json = {} inputs_dir = os.path.join(local_working_directory, "_inputs") def simple_value(input, param_dict_value, type_representation_name=None): type_representation = type_representation_from_name(type_representation_name) # Hmm... cwl_type isn't really the cwl type in every case, # like in the case of json for instance. if type_representation.galaxy_param_type == NO_GALAXY_INPUT: assert param_dict_value is None return None if type_representation.name == "file": dataset_wrapper = param_dict_value return dataset_wrapper_to_file_json(inputs_dir, dataset_wrapper) elif type_representation.name == "directory": dataset_wrapper = param_dict_value return dataset_wrapper_to_directory_json(inputs_dir, dataset_wrapper) elif type_representation.name == "integer": return int(str(param_dict_value)) elif type_representation.name == "long": return int(str(param_dict_value)) elif type_representation.name in ["float", "double"]: return float(str(param_dict_value)) elif type_representation.name == "boolean": return string_as_bool(param_dict_value) elif type_representation.name == "text": return str(param_dict_value) elif type_representation.name == "enum": return str(param_dict_value) elif type_representation.name == "json": raw_value = param_dict_value.value return json.loads(raw_value) elif type_representation.name == "field": if param_dict_value is None: return None if hasattr(param_dict_value, "value"): # Is InputValueWrapper rval = param_dict_value.value if isinstance(rval, dict) and "src" in rval and rval["src"] == "json": # needed for wf_step_connect_undeclared_param, so non-file defaults? return rval["value"] return rval elif not param_dict_value.is_collection: # Is DatasetFilenameWrapper return dataset_wrapper_to_file_json(inputs_dir, param_dict_value) else: # Is DatasetCollectionWrapper hdca_wrapper = param_dict_value if hdca_wrapper.collection_type == "list": # TODO: generalize to lists of lists and lists of non-files... return collection_wrapper_to_array(inputs_dir, hdca_wrapper) elif hdca_wrapper.collection_type.collection_type == "record": return collection_wrapper_to_record(inputs_dir, hdca_wrapper) elif type_representation.name == "array": # TODO: generalize to lists of lists and lists of non-files... return collection_wrapper_to_array(inputs_dir, param_dict_value) elif type_representation.name == "record": return collection_wrapper_to_record(inputs_dir, param_dict_value) else: return str(param_dict_value) for input_name, input in inputs.items(): if input.type == "repeat": only_input = next(iter(input.inputs.values())) array_value = [] for instance in param_dict[input_name]: array_value.append(simple_value(only_input, instance[input_name[: -len("_repeat")]])) input_json[input_name[: -len("_repeat")]] = array_value elif input.type == "conditional": assert input_name in param_dict, f"No value for {input_name} in {param_dict}" current_case = param_dict[input_name]["_cwl__type_"] if str(current_case) != "null": # str because it is a wrapped... case_index = input.get_current_case(current_case) case_input = input.cases[case_index].inputs["_cwl__value_"] case_value = param_dict[input_name]["_cwl__value_"] input_json[input_name] = simple_value(case_input, case_value, current_case) else: matched_field = None for field in input_fields: if field["name"] == input_name: matched_field = field field_type = field_to_field_type(matched_field) if isinstance(field_type, list): assert USE_FIELD_TYPES type_descriptions = [FIELD_TYPE_REPRESENTATION] else: type_descriptions = type_descriptions_for_field_types([field_type]) assert len(type_descriptions) == 1 type_description_name = type_descriptions[0].name input_json[input_name] = simple_value(input, param_dict[input_name], type_description_name) log.debug(f"Galaxy Tool State is CWL State is {input_json}") return input_json def to_galaxy_parameters(tool, as_dict): """Tool is Galaxy's representation of the tool and as_dict is a Galaxified representation of the input json (no paths, HDA references for instance). """ inputs = tool.inputs galaxy_request = {} def from_simple_value(input, param_dict_value, type_representation_name=None): if type_representation_name == "json": return json.dumps(param_dict_value) else: return param_dict_value for input_name, input in inputs.items(): as_dict_value = as_dict.get(input_name, NOT_PRESENT) galaxy_input_type = input.type if galaxy_input_type == "repeat": if input_name not in as_dict: continue only_input = next(iter(input.inputs.values())) for value in as_dict_value: key = f"{input_name}_repeat_0|{only_input.name}" galaxy_value = from_simple_value(only_input, value) galaxy_request[key] = galaxy_value elif galaxy_input_type == "conditional": case_strings = input.case_strings # TODO: less crazy handling of defaults... if (as_dict_value is NOT_PRESENT or as_dict_value is None) and "null" in case_strings: type_representation_name = "null" elif as_dict_value is NOT_PRESENT or as_dict_value is None: raise RequestParameterInvalidException( f"Cannot translate CWL datatype - value [{as_dict_value}] of type [{type(as_dict_value)}] with case_strings [{case_strings}]. Non-null property must be set." ) elif isinstance(as_dict_value, bool) and "boolean" in case_strings: type_representation_name = "boolean" elif isinstance(as_dict_value, int) and "integer" in case_strings: type_representation_name = "integer" elif isinstance(as_dict_value, int) and "long" in case_strings: type_representation_name = "long" elif isinstance(as_dict_value, (int, float)) and "float" in case_strings: type_representation_name = "float" elif isinstance(as_dict_value, (int, float)) and "double" in case_strings: type_representation_name = "double" elif isinstance(as_dict_value, str) and "string" in case_strings: type_representation_name = "string" elif ( isinstance(as_dict_value, dict) and "src" in as_dict_value and "id" in as_dict_value and "file" in case_strings ): type_representation_name = "file" elif ( isinstance(as_dict_value, dict) and "src" in as_dict_value and "id" in as_dict_value and "directory" in case_strings ): # TODO: can't disambiuate with above if both are available... type_representation_name = "directory" elif "field" in case_strings: type_representation_name = "field" elif "json" in case_strings and as_dict_value is not None: type_representation_name = "json" else: raise RequestParameterInvalidException( f"Cannot translate CWL datatype - value [{as_dict_value}] of type [{type(as_dict_value)}] with case_strings [{case_strings}]." ) galaxy_request[f"{input_name}|_cwl__type_"] = type_representation_name if type_representation_name != "null": current_case_index = input.get_current_case(type_representation_name) current_case_inputs = input.cases[current_case_index].inputs current_case_input = current_case_inputs["_cwl__value_"] galaxy_value = from_simple_value(current_case_input, as_dict_value, type_representation_name) galaxy_request[f"{input_name}|_cwl__value_"] = galaxy_value elif as_dict_value is NOT_PRESENT: continue else: galaxy_value = from_simple_value(input, as_dict_value) galaxy_request[input_name] = galaxy_value log.info(f"Converted galaxy_request is {galaxy_request}") return galaxy_request def field_to_field_type(field): field_type = field["type"] if isinstance(field_type, dict): field_type = field_type["type"] if isinstance(field_type, list): field_type_length = len(field_type) if field_type_length == 0: raise Exception("Zero-length type list encountered, invalid CWL?") elif len(field_type) == 1: field_type = field_type[0] return field_type ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/cwl/runnable.py0000644000175100017510000000166115234436334023674 0ustar00runnerrunner"""Lighter-weight variant of Planemo runnable outputs.""" from galaxy.tool_util.parser import get_tool_source from .parser import workflow_proxy from .util import guess_artifact_type def get_outputs(path: str): tool_or_workflow = guess_artifact_type(path) if tool_or_workflow == "tool": tool_source = get_tool_source(path) output_datasets, _ = tool_source.parse_outputs(None) outputs = [ToolOutput(o) for o in output_datasets.values()] return outputs else: workflow = workflow_proxy(path, strict_cwl_validation=False) return [CwlWorkflowOutput(label) for label in workflow.output_labels] class CwlWorkflowOutput: def __init__(self, label): self._label = label def get_id(self): return self._label class ToolOutput: def __init__(self, tool_output): self._tool_output = tool_output def get_id(self): return self._tool_output.name ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/cwl/runtime_actions.py0000644000175100017510000002234515234436334025273 0ustar00runnerrunnerimport json import os import shutil from typing import Optional from galaxy.util import safe_makedirs from .cwltool_deps import ref_resolver from .parser import ( JOB_JSON_FILE, load_job_proxy, ) from .util import ( SECONDARY_FILES_INDEX_PATH, STORE_SECONDARY_FILES_WITH_BASENAME, ) def file_dict_to_description(file_dict): output_class = file_dict["class"] assert output_class in ["File", "Directory"], file_dict location = file_dict["location"] if location.startswith("_:"): assert output_class == "File" return LiteralFileDescription(file_dict["contents"]) elif output_class == "File": return PathFileDescription(_possible_uri_to_path(location)) else: return PathDirectoryDescription(_possible_uri_to_path(location)) class FileDescription: pass class PathFileDescription: def __init__(self, path): self.path = path def write_to(self, destination): # TODO: Move if we can be sure this is in the working directory for instance... shutil.copy(self.path, destination) class PathDirectoryDescription: def __init__(self, path): self.path = path def write_to(self, destination): shutil.copytree(self.path, destination) class LiteralFileDescription: def __init__(self, content): self.content = content def write_to(self, destination): with open(destination, "wb") as f: f.write(self.content.encode("UTF-8")) def _possible_uri_to_path(location): if location.startswith("file://"): path = ref_resolver.uri_file_path(location) else: path = location return path def handle_outputs(job_directory: Optional[str] = None): # Relocate dynamically collected files to pre-determined locations # registered with ToolOutput objects via from_work_dir handling. if job_directory is None: job_directory = os.path.join(os.getcwd(), os.path.pardir) cwl_job_file = os.path.join(job_directory, JOB_JSON_FILE) if not os.path.exists(cwl_job_file): # Not a CWL job, just continue return # So we only need to do strict validation when the tool was loaded, # no reason to do it again during job execution - so this shortcut # allows us to not need Galaxy's full configuration on job nodes. job_proxy = load_job_proxy(job_directory, strict_cwl_validation=False) tool_working_directory = os.path.join(job_directory, "working") cwl_metadata_params_path = os.path.join(job_directory, "cwl_params.json") with open(cwl_metadata_params_path) as f: cwl_metadata_params = json.load(f) job_id_tag = cwl_metadata_params["job_id_tag"] from galaxy.job_execution.output_collect import ( default_exit_code_file, read_exit_code_from, ) exit_code_file = default_exit_code_file(".", job_id_tag) tool_exit_code = read_exit_code_from(exit_code_file, job_id_tag) outputs = job_proxy.collect_outputs(tool_working_directory, tool_exit_code) # Build galaxy.json file. provided_metadata = {} def move_directory(output, target_path, output_name=None): assert output["class"] == "Directory" output_path = _possible_uri_to_path(output["location"]) if output_path.startswith("_:"): assert "listing" in output, "Do not know how to handle output, no 'listing' found." listing = output["listing"] # No a real path, just copy listing to target path. safe_makedirs(target_path) for listed_file in listing: # TODO: handle directories assert listed_file["class"] == "File" file_description = file_dict_to_description(listed_file) file_description.write_to(os.path.join(target_path, listed_file["basename"])) else: shutil.move(output_path, target_path) return {"created_from_basename": output["basename"]} def move_output(output, target_path, output_name=None): assert output["class"] == "File" file_description = file_dict_to_description(output) file_description.write_to(target_path) secondary_files = output.get("secondaryFiles", []) if secondary_files: order = [] index_contents = {"order": order} for secondary_file in secondary_files: if output_name is None: raise NotImplementedError("secondaryFiles are unimplemented for dynamic list elements") # TODO: handle nested files... secondary_file_description = file_dict_to_description(secondary_file) # assert secondary_file_path.startswith(output_path), "[%s] does not start with [%s]" % (secondary_file_path, output_path) secondary_file_basename = secondary_file["basename"] if not STORE_SECONDARY_FILES_WITH_BASENAME: output_basename = output["basename"] prefix = "" while True: if secondary_file_basename.startswith(output_basename): secondary_file_name = prefix + secondary_file_basename[len(output_basename) :] break prefix = f"^{prefix}" if "." not in output_basename: secondary_file_name = prefix + secondary_file_name break else: output_basename = output_basename.rsplit(".", 1)[0] else: secondary_file_name = secondary_file_basename # Convert to ^ format.... secondary_files_dir = job_proxy.output_secondary_files_dir(output_name, create=True) extra_target = os.path.join(secondary_files_dir, secondary_file_name) secondary_file_description.write_to(extra_target) order.append(secondary_file_name) with open(os.path.join(secondary_files_dir, "..", SECONDARY_FILES_INDEX_PATH), "w") as f: json.dump(index_contents, f) return {"created_from_basename": output["basename"]} def handle_known_output(output, output_key, output_name): # if output["class"] != "File": # # This case doesn't seem like it would be reached - why is this here? # provided_metadata[output_name] = { # "ext": "expression.json", # } # else: assert output_name if output["class"] == "File": target_path = job_proxy.output_path(output_name) file_metadata = move_output(output, target_path, output_name=output_name) elif output["class"] == "Directory": target_path = job_proxy.output_directory_contents_dir(output_name) file_metadata = move_directory(output, target_path, output_name=output_name) else: raise Exception(f"Unknown output type [{output}] encountered") provided_metadata[output_name] = file_metadata def handle_known_output_json(output, output_name): target_path = job_proxy.output_path(output_name) with open(target_path, "w") as f: f.write(json.dumps(output)) provided_metadata[output_name] = { "ext": "expression.json", } handled_outputs = [] for output_name, output in outputs.items(): handled_outputs.append(output_name) if isinstance(output, dict) and "location" in output: handle_known_output(output, output_name, output_name) elif isinstance(output, dict): prefix = f"{output_name}|__part__|" for record_key, record_value in output.items(): record_value_output_key = f"{prefix}{record_key}" if isinstance(record_value, dict) and "class" in record_value: handle_known_output(record_value, record_value_output_key, output_name) else: # param_evaluation_noexpr handle_known_output_json(output, output_name) elif isinstance(output, list): elements = [] for index, el in enumerate(output): if isinstance(el, dict) and el["class"] == "File": output_path = _possible_uri_to_path(el["location"]) elements.append( {"name": str(index), "filename": output_path, "created_from_basename": el["basename"]} ) else: target_path = f"{output_name}____{str(index)}" with open(target_path, "w") as f: f.write(json.dumps(el)) elements.append({"name": str(index), "filename": target_path, "ext": "expression.json"}) provided_metadata[output_name] = {"elements": elements} else: handle_known_output_json(output, output_name) for output_instance in job_proxy._tool_proxy.output_instances(): output_name = output_instance.name if output_name not in handled_outputs: handle_known_output_json(None, output_name) job_metadata = os.path.join(job_directory, cwl_metadata_params["job_metadata"]) with open(job_metadata, "w") as f: json.dump(provided_metadata, f) __all__ = ("handle_outputs",) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/cwl/schema.py0000644000175100017510000000775415234436334023337 0ustar00runnerrunner"""Abstraction around cwltool and related libraries for loading a CWL artifact.""" import os import tempfile from typing import NamedTuple from .cwltool_deps import ( CommentedMap, default_loader, ensure_cwltool_available, load_tool, LoadingContext, resolve_and_validate_document, ) class RawProcessReference(NamedTuple): loading_context: LoadingContext process_object: CommentedMap uri: str class ResolvedProcessDefinition(NamedTuple): loading_context: LoadingContext uri: str raw_process_reference: RawProcessReference REWRITE_EXPRESSIONS = False class SchemaLoader: def __init__(self, strict=True, validate=True): self._strict = strict self._validate = validate @property def raw_document_loader(self): ensure_cwltool_available() return default_loader(None) def loading_context(self): loading_context = LoadingContext() loading_context.strict = self._strict loading_context.do_validate = self._validate loading_context.loader = self.raw_document_loader loading_context.do_update = True loading_context.relax_path_checks = True return loading_context def raw_process_reference(self, path, loading_context=None): suffix = "" if "#" in path: path, suffix = path.split("#") path = os.path.abspath(path) uri = f"file://{path}" loading_context = loading_context or self.loading_context() with tempfile.TemporaryDirectory() as output_dir: processed_path = os.path.join(output_dir, os.path.basename(path)) # noqa: F841 # if REWRITE_EXPRESSIONS: # The cwl_expression_refactor import below doesn't work any more # because there are now 3 cwl_v1_X_expression_refactor modules. # from cwl_utils import cwl_expression_refactor # exit_code = cwl_expression_refactor.main([output_dir, path, "--skip-some1", "--skip-some2"]) # if exit_code == 0: # uri = f"file://{processed_path}" if suffix: uri = f"{uri}#{suffix}" loading_context, process_object, uri = load_tool.fetch_document(uri, loadingContext=loading_context) return RawProcessReference(loading_context, process_object, uri) def raw_process_reference_for_object(self, process_object, uri=None, loading_context=None): if uri is None: uri = "galaxy://" loading_context = loading_context or self.loading_context() process_object["id"] = uri loading_context, process_object, uri = load_tool.fetch_document(process_object, loadingContext=loading_context) return RawProcessReference(loading_context, process_object, uri) def process_definition(self, raw_process_reference: RawProcessReference) -> ResolvedProcessDefinition: assert raw_process_reference.loading_context is not None, "No loading context found for raw_process_reference" loading_context, uri = resolve_and_validate_document( raw_process_reference.loading_context, raw_process_reference.process_object, raw_process_reference.uri, ) return ResolvedProcessDefinition( loading_context, uri, raw_process_reference, ) def tool(self, **kwds): process_definition = kwds.get("process_definition", None) if process_definition is None: raw_process_reference = kwds.get("raw_process_reference", None) if raw_process_reference is None: raw_process_reference = self.raw_process_reference(kwds["path"]) process_definition = self.process_definition(raw_process_reference) tool = load_tool.make_tool( process_definition.uri, process_definition.loading_context, ) return tool schema_loader = SchemaLoader() non_strict_non_validating_schema_loader = SchemaLoader(strict=False, validate=False) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/cwl/util.py0000644000175100017510000006544615234436334023056 0ustar00runnerrunner"""Client-centric CWL-related utilities. Used to share code between the Galaxy test framework and other Galaxy CWL clients (e.g. Planemo).""" import abc import hashlib import io import json import os import tarfile import tempfile import urllib.parse from collections import namedtuple from typing import ( Any, BinaryIO, Callable, Dict, List, Optional, Tuple, ) import yaml from typing_extensions import ( Literal, Protocol, TypedDict, ) from galaxy.util import ( str_removeprefix, unicodify, ) STORE_SECONDARY_FILES_WITH_BASENAME = True SECONDARY_FILES_EXTRA_PREFIX = "__secondary_files__" SECONDARY_FILES_INDEX_PATH = "__secondary_files_index.json" def set_basename_and_derived_properties(properties, basename): properties["basename"] = basename properties["nameroot"], properties["nameext"] = os.path.splitext(basename) return properties OutputPropertiesType = TypedDict( "OutputPropertiesType", { "class": str, "location": Optional[str], "path": Optional[str], "listing": Optional[List[Any]], "basename": Optional[str], "nameroot": Optional[str], "nameext": Optional[str], "secondaryFiles": List[Any], "checksum": str, "size": int, }, total=False, ) def output_properties( path: Optional[str] = None, content: Optional[bytes] = None, basename=None, pseudo_location=False, ) -> OutputPropertiesType: checksum = hashlib.sha1() properties: OutputPropertiesType = {"class": "File", "checksum": "", "size": 0} f: BinaryIO if path is not None: properties["path"] = path f = open(path, "rb") else: if content is None: raise Exception("If no 'path', must provide 'content'.") f = io.BytesIO(content) try: contents = f.read(1024 * 1024) filesize = 0 while contents: checksum.update(contents) filesize += len(contents) contents = f.read(1024 * 1024) finally: f.close() properties["checksum"] = f"sha1${checksum.hexdigest()}" properties["size"] = filesize set_basename_and_derived_properties(properties, basename) _handle_pseudo_location(properties, pseudo_location) return properties def _handle_pseudo_location(properties, pseudo_location): if pseudo_location: properties["location"] = properties["basename"] def abs_path_or_uri(path_or_uri: str, relative_to: str, resolve_data: Optional[Callable[[str], Optional[str]]]) -> str: """Return the absolute path if this isn't a URI, otherwise keep the URI the same.""" if "://" in path_or_uri: return path_or_uri abs_path_ = os.path.abspath(os.path.join(relative_to, path_or_uri)) if resolve_data and not os.path.exists(abs_path_): resolved_data = resolve_data(path_or_uri) if resolved_data: abs_path_ = resolved_data _ensure_file_exists(abs_path_) return abs_path_ def abs_path(path_or_uri: str, relative_to: str) -> str: """Return the absolute path if this is a file:// URI or a local path.""" if path_or_uri.startswith("file://"): # The path after file:// must be absolute abs_path_ = path_or_uri[len("file://") :] else: index = path_or_uri.find("://") if index != -1: raise ValueError(f"Unsupported URI scheme: {path_or_uri[: index + 3]}") abs_path_ = os.path.abspath(os.path.join(relative_to, path_or_uri)) _ensure_file_exists(abs_path_) return abs_path_ def path_or_uri_to_uri(path_or_uri: str) -> str: if "://" not in path_or_uri: return f"file://{path_or_uri}" else: return path_or_uri class CollectionCreateFunc(Protocol): def __call__( self, element_identifiers: List[Dict[str, Any]], collection_type: str, rows: Optional[Dict[str, Any]] = None, name: Optional[str] = None, ) -> Dict[str, Any]: """Create a collection from these identifiers.""" def galactic_job_json( job: Dict[str, Any], test_data_directory: str, upload_func: Callable[["UploadTarget"], Dict[str, Any]], collection_create_func: CollectionCreateFunc, tool_or_workflow: Literal["tool", "workflow"] = "workflow", resolve_data: Optional[Callable[[str], Optional[str]]] = None, ) -> Tuple[Dict[str, Any], List[Dict[str, Any]]]: """Adapt a CWL job object to the Galaxy API. CWL derived tools in Galaxy can consume a job description sort of like CWL job objects via the API but paths need to be replaced with datasets and records and arrays with collection references. This function will stage files and modify the job description to adapt to these changes for Galaxy. """ datasets: List[Dict[str, Any]] = [] dataset_collections: List[Dict[str, Any]] = [] def response_to_hda(target: UploadTarget, upload_response: Dict[str, Any]) -> Dict[str, str]: assert isinstance(upload_response, dict), upload_response assert "outputs" in upload_response, upload_response assert len(upload_response["outputs"]) > 0, upload_response dataset = upload_response["outputs"][0] datasets.append(dataset) dataset_id = dataset["id"] return {"src": "hda", "id": dataset_id} def upload_file(file_path: str, secondary_files: Optional[str], **kwargs) -> Dict[str, str]: file_path = abs_path_or_uri(file_path, test_data_directory, resolve_data=resolve_data) target = FileUploadTarget(file_path, secondary_files, **kwargs) upload_response = upload_func(target) return response_to_hda(target, upload_response) def upload_file_literal(contents: str, **kwd) -> Dict[str, str]: target = FileLiteralTarget(contents, **kwd) upload_response = upload_func(target) return response_to_hda(target, upload_response) def upload_tar(file_path: str, file_type: str = "directory", name: str = "uploaded tar file") -> Dict[str, str]: file_path = abs_path_or_uri(file_path, test_data_directory, resolve_data=resolve_data) target = DirectoryUploadTarget(file_path, file_type=file_type, name=name) upload_response = upload_func(target) return response_to_hda(target, upload_response) def upload_file_with_composite_data(file_path: Optional[str], composite_data, **kwargs) -> Dict[str, str]: if file_path is not None: file_path = abs_path_or_uri(file_path, test_data_directory, resolve_data=resolve_data) composite_data_resolved = [] for cd in composite_data: composite_data_resolved.append(abs_path_or_uri(cd, test_data_directory, resolve_data=resolve_data)) target = FileUploadTarget(file_path, composite_data=composite_data_resolved, **kwargs) upload_response = upload_func(target) return response_to_hda(target, upload_response) def upload_object(the_object: Any) -> Dict[str, str]: target = ObjectUploadTarget(the_object) upload_response = upload_func(target) return response_to_hda(target, upload_response) def replacement_item(value, force_to_file: bool = False): is_dict = isinstance(value, dict) item_class = None if not is_dict else value.get("class", None) is_file = item_class == "File" is_directory = item_class == "Directory" is_collection = item_class == "Collection" # Galaxy extension. if force_to_file: if is_file: return replacement_file(value) else: return upload_object(value) if isinstance(value, list): return replacement_list(value) elif not isinstance(value, dict): if tool_or_workflow == "workflow": # All inputs represented as dataset or collection parameters return upload_object(value) else: return value if is_file: return replacement_file(value) elif is_directory: return replacement_directory(value) elif is_collection: return replacement_collection(value) else: return replacement_record(value) def replacement_file(value): if value.get("galaxy_id"): return {"src": "hda", "id": str(value["galaxy_id"])} file_path = value.get("location") or value.get("path") # format to match output definitions in tool, where did filetype come from? filetype = value.get("filetype", None) or value.get("format", None) composite_data_raw = value.get("composite_data", None) kwd = {} if "tags" in value: kwd["tags"] = value.get("tags") if "dbkey" in value: kwd["dbkey"] = value.get("dbkey") if "decompress" in value: kwd["decompress"] = value["decompress"] if value.get("hashes"): kwd["hashes"] = value["hashes"] if value.get("metadata"): kwd["metadata"] = value["metadata"] if "name" in value: kwd["name"] = value["name"] if composite_data_raw: composite_data = [] for entry in composite_data_raw: path = None if isinstance(entry, dict): path = entry.get("location", None) or entry.get("path", None) else: path = entry composite_data.append(path) rval_c = upload_file_with_composite_data(None, composite_data, filetype=filetype, **kwd) return rval_c if file_path is None: contents = value.get("contents") if contents is not None: return upload_file_literal(contents, filetype=filetype, **kwd) return value secondary_files = value.get("secondaryFiles", []) secondary_files_tar_path = None if secondary_files: tmp = tempfile.NamedTemporaryFile(delete=False) tf = tarfile.open(fileobj=tmp, mode="w:") order: List[str] = [] index_contents = {"order": order} for secondary_file in secondary_files: secondary_file_path = secondary_file.get("location", None) or secondary_file.get("path", None) assert secondary_file_path, f"Invalid secondaryFile entry found [{secondary_file}]" full_secondary_file_path = os.path.join(test_data_directory, secondary_file_path) basename = secondary_file.get("basename") or os.path.basename(secondary_file_path) order.append(unicodify(basename)) tf.add(full_secondary_file_path, os.path.join(SECONDARY_FILES_EXTRA_PREFIX, basename)) tmp_index = tempfile.NamedTemporaryFile(delete=False, mode="w") json.dump(index_contents, tmp_index) tmp_index.close() tf.add(tmp_index.name, SECONDARY_FILES_INDEX_PATH) tf.close() secondary_files_tar_path = tmp.name return upload_file(file_path, secondary_files_tar_path, filetype=filetype, **kwd) def replacement_directory(value: Dict[str, Any]) -> Dict[str, Any]: file_path = value.get("location", None) or value.get("path", None) if file_path is None: return value file_path = abs_path_or_uri(file_path, test_data_directory, resolve_data=resolve_data) file_type = value.get("filetype", None) or value.get("format", None) or "directory" tmp = tempfile.NamedTemporaryFile(delete=False) tf = tarfile.open(fileobj=tmp, mode="w:") tf.add(file_path, ".") tf.close() return upload_tar(tmp.name, file_type=file_type, name=os.path.basename(file_path)) def replacement_list(value) -> Dict[str, str]: collection_element_identifiers = [] for i, item in enumerate(value): dataset = replacement_item(item, force_to_file=True) collection_element = dataset.copy() collection_element["name"] = str(i) collection_element_identifiers.append(collection_element) # TODO: handle nested lists/arrays collection = collection_create_func(collection_element_identifiers, "list") dataset_collections.append(collection) hdca_id = collection["id"] return {"src": "hdca", "id": hdca_id} def to_elements(value, rank_collection_type: str) -> List[Dict[str, Any]]: collection_element_identifiers = [] assert "elements" in value elements = value["elements"] is_nested_collection = ":" in rank_collection_type for element in elements: if not is_nested_collection: # flat collection dataset = replacement_item(element, force_to_file=True) collection_element = dataset.copy() collection_element["name"] = element["identifier"] collection_element_identifiers.append(collection_element) else: # nested collection sub_collection_type = rank_collection_type[rank_collection_type.find(":") + 1 :] collection_element = { "name": element["identifier"], "src": "new_collection", "collection_type": sub_collection_type, "element_identifiers": to_elements(element, sub_collection_type), } collection_element_identifiers.append(collection_element) return collection_element_identifiers def replacement_collection(value: Dict[str, Any]) -> Dict[str, str]: if value.get("galaxy_id"): return {"src": "hdca", "id": str(value["galaxy_id"])} assert "collection_type" in value collection_type = value["collection_type"] elements = to_elements(value, collection_type) kwds = {} if collection_type.startswith("sample_sheet"): kwds["rows"] = value.get("rows") if "name" in value: kwds["name"] = value["name"] collection = collection_create_func(elements, collection_type, **kwds) dataset_collections.append(collection) hdca_id = collection["id"] return {"src": "hdca", "id": hdca_id} def replacement_record(value): collection_element_identifiers = [] for record_key, record_value in value.items(): if not isinstance(record_value, dict) or record_value.get("class") != "File": dataset = replacement_item(record_value, force_to_file=True) collection_element = dataset.copy() else: dataset = upload_file(record_value["location"], None) collection_element = dataset.copy() collection_element["name"] = record_key collection_element_identifiers.append(collection_element) collection = collection_create_func(collection_element_identifiers, "record") dataset_collections.append(collection) hdca_id = collection["id"] return {"src": "hdca", "id": hdca_id} replace_keys = {} for key, value in job.items(): replace_keys[key] = replacement_item(value) job.update(replace_keys) return job, datasets def _ensure_file_exists(file_path: str) -> None: if not os.path.exists(file_path): template = "File [%s] does not exist - parent directory [%s] does %sexist, cwd is [%s]" parent_directory = os.path.dirname(file_path) message = template % ( file_path, parent_directory, "" if os.path.exists(parent_directory) else "not ", os.getcwd(), ) raise Exception(message) class UploadTarget: @abc.abstractmethod def __str__(self) -> str: pass class FileLiteralTarget(UploadTarget): def __init__(self, contents: str, **kwargs) -> None: self.contents = contents self.properties = kwargs def __str__(self) -> str: return f"FileLiteralTarget[contents={self.contents}] with {self.properties}" class FileUploadTarget(UploadTarget): def __init__( self, path: Optional[str], secondary_files: Optional[str] = None, composite_data: Optional[List[str]] = None, **kwargs, ) -> None: self.path = path self.secondary_files = secondary_files self.composite_data = composite_data self.properties = kwargs def __str__(self) -> str: return f"FileUploadTarget[path={self.path}] with {self.properties}" class ObjectUploadTarget(UploadTarget): def __init__(self, the_object: Any) -> None: self.object = the_object self.properties: Dict = {} def __str__(self) -> str: return f"ObjectUploadTarget[object={self.object}] with {self.properties}" class DirectoryUploadTarget(UploadTarget): def __init__(self, tar_path: str, file_type: str = "directory", name: str = "uploaded directory") -> None: self.tar_path = tar_path self.file_type = file_type self.name = name def __str__(self) -> str: return f"DirectoryUploadTarget[tar_path={self.tar_path}]" GalaxyOutput = namedtuple("GalaxyOutput", ["history_id", "history_content_type", "history_content_id", "metadata"]) def tool_response_to_output(tool_response, history_id, output_name): for output in tool_response["outputs"]: if output["output_name"] == output_name: return GalaxyOutput(history_id, "dataset", output["id"], None) for output_collection in tool_response["output_collections"]: if output_collection["output_name"] == output_name: return GalaxyOutput(history_id, "dataset_collection", output_collection["id"], None) raise Exception(f"Failed to find output with label [{output_name}]") def invocation_to_output(invocation, history_id, output_name): if output_name in invocation["outputs"]: dataset = invocation["outputs"][output_name] return GalaxyOutput(history_id, "dataset", dataset["id"], None) elif output_name in invocation["output_collections"]: collection = invocation["output_collections"][output_name] return GalaxyOutput(history_id, "dataset_collection", collection["id"], None) elif output_name in invocation["output_values"]: output_value = invocation["output_values"][output_name] return GalaxyOutput(None, "raw_value", output_value, None) raise Exception(f"Failed to find output with label [{output_name}] in [{invocation}]") def output_to_cwl_json( galaxy_output, get_metadata, get_dataset, get_extra_files, pseudo_location=False, ): """Convert objects in a Galaxy history into a CWL object. Useful in running conformance tests and implementing the cwl-runner interface via Galaxy. """ def element_to_cwl_json(element): object = element["object"] content_type = object.get("history_content_type") metadata = None if content_type is None: content_type = "dataset_collection" metadata = element["object"] metadata["history_content_type"] = content_type element_output = GalaxyOutput( galaxy_output.history_id, content_type, object["id"], metadata, ) return output_to_cwl_json( element_output, get_metadata, get_dataset, get_extra_files, pseudo_location=pseudo_location ) output_metadata = galaxy_output.metadata if output_metadata is None: output_metadata = get_metadata(galaxy_output.history_content_type, galaxy_output.history_content_id) def dataset_dict_to_json_content(dataset_dict): if "content" in dataset_dict: return json.loads(dataset_dict["content"]) else: with open(dataset_dict["path"]) as f: return json.load(f) if galaxy_output.history_content_type == "raw_value": return galaxy_output.history_content_id elif output_metadata["history_content_type"] == "dataset": ext = output_metadata["file_ext"] if ext == "expression.json": dataset_dict = get_dataset(output_metadata) return dataset_dict_to_json_content(dataset_dict) else: file_or_directory = "Directory" if ext == "directory" else "File" secondary_files = [] if file_or_directory == "File": dataset_dict = get_dataset(output_metadata) properties = output_properties(pseudo_location=pseudo_location, **dataset_dict) basename = properties["basename"] extra_files = get_extra_files(output_metadata) found_index = False for extra_file in extra_files: if extra_file["class"] == "File": path = extra_file["path"] if path == SECONDARY_FILES_INDEX_PATH: found_index = True if found_index: ec = get_dataset(output_metadata, filename=SECONDARY_FILES_INDEX_PATH) index = dataset_dict_to_json_content(ec) def dir_listing(dir_path): listing = [] for extra_file in extra_files: path = extra_file["path"] extra_file_class = extra_file["class"] extra_file_basename = os.path.basename(path) if os.path.join(dir_path, extra_file_basename) != path: continue if extra_file_class == "File": ec = get_dataset(output_metadata, filename=path) ec["basename"] = extra_file_basename ec_properties = output_properties(pseudo_location=pseudo_location, **ec) elif extra_file_class == "Directory": ec_properties = {} ec_properties["class"] = "Directory" ec_properties["location"] = ec_basename ec_properties["listing"] = dir_listing(path) else: raise Exception("Unknown output type encountered....") listing.append(ec_properties) return listing for basename in index["order"]: for extra_file in extra_files: path = extra_file["path"] if path != os.path.join(SECONDARY_FILES_EXTRA_PREFIX, basename or ""): continue extra_file_class = extra_file["class"] # This is wrong... if not STORE_SECONDARY_FILES_WITH_BASENAME: ec_basename = basename + os.path.basename(path) else: ec_basename = os.path.basename(path) if extra_file_class == "File": ec = get_dataset(output_metadata, filename=path) ec["basename"] = ec_basename ec_properties = output_properties(pseudo_location=pseudo_location, **ec) elif extra_file_class == "Directory": ec_properties = {} ec_properties["class"] = "Directory" ec_properties["location"] = ec_basename ec_properties["listing"] = dir_listing(path) else: raise Exception("Unknown output type encountered....") secondary_files.append(ec_properties) else: basename = output_metadata.get("created_from_basename") if not basename: basename = output_metadata.get("name") listing: List[OutputPropertiesType] = [] properties = { "class": "Directory", "basename": basename, "listing": listing, } extra_files = get_extra_files(output_metadata) for extra_file in extra_files: if extra_file["class"] == "File": path = extra_file["path"] ec = get_dataset(output_metadata, filename=path) ec["basename"] = os.path.basename(path) ec_properties = output_properties(pseudo_location=pseudo_location, **ec) listing.append(ec_properties) if secondary_files: properties["secondaryFiles"] = secondary_files return properties elif output_metadata["history_content_type"] == "dataset_collection": collection_type = output_metadata["collection_type"].split(":", 1)[0] if collection_type in ["list", "paired", "sample_sheet", "paired_or_unpaired"]: rval_l = [] for element in output_metadata["elements"]: rval_l.append(element_to_cwl_json(element)) return rval_l elif collection_type == "record": rval_d = {} for element in output_metadata["elements"]: rval_d[element["element_identifier"]] = element_to_cwl_json(element) return rval_d raise NotImplementedError(f"Unsupported collection type encountered ({collection_type})") else: raise NotImplementedError("Unknown history content type encountered") def download_output(galaxy_output, get_metadata, get_dataset, get_extra_files, output_path): output_metadata = get_metadata(galaxy_output.history_content_type, galaxy_output.history_content_id) dataset_dict = get_dataset(output_metadata) with open(output_path, "wb") as fh: fh.write(dataset_dict["content"]) def guess_artifact_type(path): tool_or_workflow = "workflow" path, object_id = urllib.parse.urldefrag(path) with open(path) as f: document = yaml.safe_load(f) if "$graph" in document: # Packed document without a process object at the root objects = document["$graph"] if not object_id: object_id = "main" # default object id # Have to use str_removeprefix() instead of rstrip() because only the # first '#' should be removed from the object id matching_objects = [o for o in objects if str_removeprefix(o["id"], "#") == object_id] if len(matching_objects) == 0: raise Exception(f"No process object with id [{object_id}]") if len(matching_objects) > 1: raise Exception(f"Multiple process objects with id [{object_id}]") object_ = matching_objects[0] else: object_ = document tool_or_workflow = "tool" if object_["class"] != "Workflow" else "workflow" return tool_or_workflow ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0417972 galaxy_tool_util-26.1.1/src/galaxy/tool_util/data/0000755000175100017510000000000015234436377021643 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/data/__init__.py0000644000175100017510000020256415234436334023756 0ustar00runnerrunner""" Manage tool data tables, which store (at the application level) data that is used by tools, for example in the generation of dynamic options. Tables are loaded and stored by names which tools use to refer to them. This allows users to configure data tables for a local Galaxy instance without needing to modify the tool configurations. """ import errno import hashlib import json import logging import os import os.path import re import string from contextlib import contextmanager from dataclasses import dataclass from glob import glob from tempfile import NamedTemporaryFile from typing import ( Any, BinaryIO, Callable, Dict, Iterator, List, Optional, overload, Set, Tuple, Type, TYPE_CHECKING, Union, ) from typing_extensions import ( Protocol, TypedDict, ) from galaxy import util from galaxy.exceptions import MessageException from galaxy.util import ( Element, requests, RW_R__R__, ) from galaxy.util.compression_utils import decompress_path_to_directory from galaxy.util.dictifiable import Dictifiable from galaxy.util.filelock import FileLock from galaxy.util.path import StrPath from galaxy.util.renamed_temporary_file import RenamedTemporaryFile from galaxy.util.template import fill_template from ._schema import ( ToolDataEntry, ToolDataEntryList, ) from .bundles.models import ( DataTableBundle, DataTableBundleProcessorDescription, RepoInfo, ) if TYPE_CHECKING: from galaxy.tools.data_manager.manager import DataManager log = logging.getLogger(__name__) BUNDLE_INDEX_FILE_NAME = "_gx_data_bundle_index.json" DEFAULT_TABLE_TYPE = "tabular" TOOL_DATA_TABLE_CONF_XML = """ """ # Internally just the first two - but tool shed code (data_manager_manual) will still # pass DataManager in. EntrySource = Optional[Union[dict, RepoInfo, "DataManager"]] class StoresConfigFilePaths(Protocol): def get(self, key: Any, default: Optional[Any]) -> Optional[Any]: ... class ToolDataFilesystem(Protocol): """The two filesystem operations ``ToolDataPathFiles`` performs. Injected so tests can supply an in-memory implementation and assert caching behaviour (e.g. one walk per load pass) without monkeypatching ``os``. """ def walk(self, path: str) -> Iterator[Tuple[str, List[str], List[str]]]: ... def exists(self, path: str) -> bool: ... class _OsFilesystem: def walk(self, path: str) -> Iterator[Tuple[str, List[str], List[str]]]: return os.walk(path) def exists(self, path: str) -> bool: return os.path.exists(path) class ToolDataPathFiles: # ``None`` means no directory listing is cached, in which case ``exists()`` # resolves each path directly via the filesystem. A listing is only cached # for the duration of a load pass (see ``cached``), so it can never outlive # the on-disk state it was taken from. _tool_data_path_files: Optional[Set[str]] def __init__(self, tool_data_path, filesystem: Optional[ToolDataFilesystem] = None): self.tool_data_path = os.path.abspath(tool_data_path) self._fs = filesystem or _OsFilesystem() self._tool_data_path_files = None self._cache_depth = 0 @property def tool_data_path_files(self) -> Set[str]: return self._tool_data_path_files if self._tool_data_path_files is not None else set() @contextmanager def cached(self) -> Iterator[None]: # Walk the tool-data tree once and share the listing for the duration of # a load pass, so the many ``exists()`` checks a pass performs don't each # re-walk (a full walk of a production tool-data tree can take longer # than a single table takes to load). The snapshot is dropped on exit so # it can never go stale relative to disk. Re-entrant: only the outermost # ``cached`` builds and clears the listing. if self._cache_depth == 0: self.update_files() self._cache_depth += 1 try: yield finally: self._cache_depth -= 1 if self._cache_depth == 0: self.invalidate() def invalidate(self) -> None: self._tool_data_path_files = None def update_files(self) -> None: try: content = self._fs.walk(self.tool_data_path) self._tool_data_path_files = set( filter( self._fs.exists, [ os.path.join(dirpath, fn) for dirpath, _, fn_list in content for fn in fn_list if fn and fn.endswith(".loc") or fn.endswith(".loc.sample") ], ) ) except Exception: log.exception("Failed to update _tool_data_path_files") # Leave the listing uncached so exists() falls back to # os.path.exists(); the walk is retried on the next load pass. self.invalidate() def exists(self, path: str) -> bool: path = os.path.abspath(path) if path in self.tool_data_path_files: return True else: return self._fs.exists(path) ErrorListT = List[str] class FileNameInfoT(TypedDict): found: bool filename: str from_shed_config: bool tool_data_path: Optional[StrPath] config_element: Optional[Element] tool_shed_repository: Optional[Dict[str, Any]] errors: ErrorListT LoadInfoT = Tuple[Tuple[Element, Optional[StrPath]], Dict[str, Any]] class DataTableColumnMismatch(Exception): """Two data tables share a name but declare different columns.""" def __init__(self, table_name: str, existing_columns: Dict[str, int], incoming_columns: Dict[str, int]): self.table_name = table_name self.existing_columns = existing_columns self.incoming_columns = incoming_columns super().__init__( f"Data table {table_name!r} is already registered with columns {existing_columns}, " f"refusing to register conflicting columns {incoming_columns}." ) class ToolDataTable(Dictifiable): type_key: str data: List[List[str]] empty_field_value: str empty_field_values: Dict[Optional[str], str] filenames: Dict[str, FileNameInfoT] _load_info: LoadInfoT _merged_load_info: List[Tuple[Type["ToolDataTable"], LoadInfoT]] @classmethod def from_dict(cls, d): data_table_class = globals()[d["model_class"]] data_table = data_table_class.__new__(data_table_class) for attr, val in d.items(): if not attr == "model_class": setattr(data_table, attr, val) data_table._loaded_content_version = 1 return data_table def __init__( self, config_element: Element, tool_data_path: Optional[StrPath], tool_data_path_files: ToolDataPathFiles, from_shed_config: bool = False, filename: Optional[StrPath] = None, other_config_dict: Optional[StoresConfigFilePaths] = None, ) -> None: name = config_element.get("name") assert name self.name = name self.empty_field_value = config_element.get("empty_field_value", "") self.empty_field_values: Dict[str, str] = {} self.allow_duplicate_entries = util.asbool(config_element.get("allow_duplicate_entries", True)) self.here = os.path.dirname(filename) if filename else None self.filenames: Dict[str, FileNameInfoT] = {} self.tool_data_path = tool_data_path self.tool_data_path_files = tool_data_path_files self.other_config_dict = other_config_dict or {} self.missing_index_file: Optional[str] = None # increment this variable any time a new entry is added, or when the table is totally reloaded # This value has no external meaning, and does not represent an abstract version of the underlying data self._loaded_content_version = 1 self._load_info = ( (config_element, tool_data_path), { "from_shed_config": from_shed_config, "tool_data_path_files": self.tool_data_path_files, "other_config_dict": other_config_dict, "filename": filename, }, ) self._merged_load_info: List[Tuple[Type[ToolDataTable], Tuple[Tuple[Element, StrPath], Dict[str, Any]]]] = [] def _update_version(self, version: Optional[int] = None) -> int: if version is not None: self._loaded_content_version = version else: self._loaded_content_version += 1 return self._loaded_content_version def get_empty_field_by_name(self, name: Optional[str]) -> str: return self.empty_field_values.get(name, self.empty_field_value) def _add_entry( self, entry: Union[List[str], Dict[str, str]], allow_duplicates: bool = True, persist: bool = False, entry_source: EntrySource = None, tool_data_file_path: Optional[str] = None, bundle_mode: bool = False, **kwd, ) -> None: raise NotImplementedError("Abstract method") def add_entry( self, entry: Union[List[str], Dict[str, str]], allow_duplicates: bool = True, persist: bool = False, entry_source: EntrySource = None, tool_data_file_path: Optional[str] = None, bundle_mode: bool = False, **kwd, ) -> int: self._add_entry( entry, allow_duplicates=allow_duplicates, persist=persist, entry_source=entry_source, tool_data_file_path=tool_data_file_path, bundle_mode=bundle_mode, **kwd, ) return self._update_version() def add_entries( self, entries: List[List[str]], allow_duplicates: bool = True, persist: bool = False, entry_source: EntrySource = None, **kwd, ) -> int: for entry in entries: try: self.add_entry( entry, allow_duplicates=allow_duplicates, persist=persist, entry_source=entry_source, **kwd ) except MessageException as e: if e.type == "warning": log.warning(str(e)) else: log.error(str(e)) except Exception as e: log.error(str(e)) return self._loaded_content_version def _remove_entry(self, values): raise NotImplementedError("Abstract method") def remove_entry(self, values): self._remove_entry(values) return self._update_version() def is_current_version(self, other_version): return self._loaded_content_version == other_version def merge_tool_data_table( self, other_table: "ToolDataTable", allow_duplicates: bool = True, persist: bool = False, **kwd, ) -> int: raise NotImplementedError("Abstract method") def reload_from_files(self) -> int: new_version = self._update_version() merged_info = self._merged_load_info self.__init__(*self._load_info[0], **self._load_info[1]) # type: ignore[misc] self._update_version(version=new_version) for tool_data_table_class, load_info in merged_info: self.merge_tool_data_table(tool_data_table_class(*load_info[0], **load_info[1]), allow_duplicates=False) return self._update_version() class TabularToolDataTable(ToolDataTable): """ Data stored in a tabular / separated value format on disk, allows multiple files to be merged but all must have the same column definitions: .. code-block:: xml
""" dict_collection_visible_keys = ["name"] dict_element_visible_keys = ["name", "fields"] dict_export_visible_keys = ["name", "data", "largest_index", "columns", "missing_index_file"] type_key = "tabular" def __init__( self, config_element: Element, tool_data_path: Optional[StrPath], tool_data_path_files: ToolDataPathFiles, from_shed_config: bool = False, filename: Optional[StrPath] = None, other_config_dict: Optional[StoresConfigFilePaths] = None, ) -> None: super().__init__( config_element, tool_data_path, tool_data_path_files, from_shed_config, filename, other_config_dict=other_config_dict, ) self.config_element = config_element self.data = [] self.configure_and_load(config_element, tool_data_path, from_shed_config) def configure_and_load( self, config_element: Element, tool_data_path: Optional[StrPath], from_shed_config: bool = False, url_timeout: float = 10, ) -> None: """ Configure and load table from an XML element. """ self.separator = config_element.get("separator", "\t") self.comment_char = config_element.get("comment_char", "#") # Configure columns self.parse_column_spec(config_element) # store repo info if available: repo_elem = config_element.find("tool_shed_repository") if repo_elem is not None: tool_shed_elem = repo_elem.find("tool_shed") assert tool_shed_elem is not None repository_name_elem = repo_elem.find("repository_name") assert repository_name_elem is not None repository_owner_elem = repo_elem.find("repository_owner") assert repository_owner_elem is not None installed_changeset_revision_elem = repo_elem.find("installed_changeset_revision") assert installed_changeset_revision_elem is not None repo_info = dict( tool_shed=tool_shed_elem.text, name=repository_name_elem.text, owner=repository_owner_elem.text, installed_changeset_revision=installed_changeset_revision_elem.text, ) else: repo_info = None # Read every file for file_element in config_element.findall("file"): tmp_file = None filename = file_element.get("path", None) if filename is None: # Handle URLs as files filename = file_element.get("url", None) if filename: tmp_file = NamedTemporaryFile(prefix=f"TTDT_URL_{self.name}-", mode="w") try: tmp_file.write(requests.get(filename, timeout=url_timeout).text) except Exception as e: log.error('Error loading Data Table URL "%s": %s', filename, e) continue log.debug('Loading Data Table URL "%s" as filename "%s".', filename, tmp_file.name) filename = tmp_file.name tmp_file.flush() else: # Pull the filename from a global config filename = file_element.get("from_config") if filename: filename = self.other_config_dict.get(filename, None) if filename: filename = _expand_here_template(filename, here=self.here) found = False if filename is None: log.debug( "Encountered a file element (%s) that does not contain a path value when loading tool data table '%s'.", util.xml_to_string(file_element), self.name, ) continue # FIXME: splitting on and merging paths from a configuration file when loading is wonky # Data should exist on disk in the state needed, i.e. the xml configuration should # point directly to the desired file to load. Munging of the tool_data_tables_conf.xml.sample # can be done during installing / testing / metadata resetting with the creation of a proper # tool_data_tables_conf.xml file, containing correct attributes. Allowing a # path.join with a different root should be allowed, but splitting should not be necessary. if tool_data_path and from_shed_config: # Must identify with from_shed_config as well, because the # regular galaxy app has and uses tool_data_path. # We're loading a tool in the tool shed, so we cannot use the Galaxy tool-data # directory which is hard-coded into the tool_data_table_conf.xml entries. filename = os.path.split(filename)[1] filename = os.path.join(tool_data_path, filename) if self.tool_data_path_files.exists(filename): found = True elif not os.path.isabs(filename): # Since the path attribute can include a hard-coded path to a specific directory # (e.g., ) which may not be the same value # as self.tool_data_path, we'll parse the path to get the filename and see if it is # in self.tool_data_path. file_path, file_name = os.path.split(filename) if file_path != self.tool_data_path: assert self.tool_data_path corrected_filename = os.path.join(self.tool_data_path, file_name) if self.tool_data_path_files.exists(corrected_filename): filename = corrected_filename found = True elif not from_shed_config and self.tool_data_path_files.exists(f"{corrected_filename}.sample"): log.info(f"Could not find tool data {corrected_filename}, reading sample") filename = f"{corrected_filename}.sample" found = True errors: ErrorListT = [] if found: self.extend_data_with(filename, errors=errors) self._update_version() else: self.missing_index_file = filename # TODO: some data tables need to exist (even if they are empty) # for tools to load. In an installed Galaxy environment and the # default tool_data_table_conf.xml, this will emit spurious # warnings about missing location files that would otherwise be # empty and we don't care about unless the admin chooses to # populate them. log.warning(f"Cannot find index file '{filename}' for tool data table '{self.name}'") if filename not in self.filenames or not self.filenames[filename]["found"]: self.filenames[filename] = dict( found=found, filename=filename, from_shed_config=from_shed_config, tool_data_path=tool_data_path, config_element=config_element, tool_shed_repository=repo_info, errors=errors, ) else: log.debug( "Filename '%s' already exists in filenames (%s), not adding", filename, list(self.filenames.keys()) ) # Remove URL tmp file if tmp_file is not None: tmp_file.close() def merge_tool_data_table(self, other_table, allow_duplicates=True, persist=False, **kwd): assert ( self.columns == other_table.columns ), f"Merging tabular data tables with non matching columns is not allowed: {self.name}:{self.columns} != {other_table.name}:{other_table.columns}" # merge filename info for filename, info in other_table.filenames.items(): if filename not in self.filenames: self.filenames[filename] = info # save info about table self._merged_load_info.append((other_table.__class__, other_table._load_info)) # If we are merging in a data table that does not allow duplicates, enforce that upon the data table if self.allow_duplicate_entries and not other_table.allow_duplicate_entries: log.debug( 'While attempting to merge tool data table "%s", the other instance of the table specified that duplicate entries are not allowed, now deduplicating all previous entries.', self.name, ) self.allow_duplicate_entries = False self._deduplicate_data() # add data entries and return current data table version return self.add_entries(other_table.data, allow_duplicates=allow_duplicates, persist=persist, **kwd) def handle_found_index_file(self, filename): self.missing_index_file = None self.extend_data_with(filename) # This method is used in tools, so need to keep its API stable def get_fields(self) -> List[List[str]]: return self.data.copy() def get_field(self, value): rval = None for i in self.get_named_fields_list(): if i["value"] == value: rval = TabularToolDataField(i) return rval # This method is used in tools, so need to keep its API stable def get_named_fields_list(self) -> List[Dict[Union[str, int], str]]: rval = [] named_columns = self.get_column_name_list() for fields in self.get_fields(): field_dict = {} for i, field in enumerate(fields): if i == len(named_columns): break field_name: Optional[Union[str, int]] = named_columns[i] if field_name is None: field_name = i # check that this is supposed to be 0 based. field_dict[field_name] = field rval.append(field_dict) return rval def get_version_fields(self): return (self._loaded_content_version, self.get_fields()) @staticmethod def parse_column_spec_element( config_element: Element, ) -> Tuple[Dict[str, int], int, Dict[str, str]]: """ Parse column definitions into ``(columns, largest_index, empty_field_values)``. Does not mutate or assert — callers layer their own validation. """ columns: Dict[str, int] = {} empty_field_values: Dict[str, str] = {} largest_index = 0 columns_elem = config_element.find("columns") if columns_elem is not None: column_names = util.xml_text(columns_elem) for index, name in enumerate(n.strip() for n in column_names.split(",")): columns[name] = index largest_index = index else: for column_elem in config_element.findall("column"): name = column_elem.get("name") index_attr = column_elem.get("index") if name is None or index_attr is None: continue index = int(index_attr) columns[name] = index if index > largest_index: largest_index = index empty_field_value = column_elem.get("empty_field_value", None) if empty_field_value is not None: empty_field_values[name] = empty_field_value return columns, largest_index, empty_field_values def parse_column_spec(self, config_element: Element) -> None: """ Parse column definitions, which can either be a set of 'column' elements with a name and index (as in dynamic options config), or a shorthand comma separated list of names in order as the text of a 'column_names' element. A column named 'value' is required. """ if config_element.find("columns") is None: for column_elem in config_element.findall("column"): assert column_elem.get("name") is not None, "Required 'name' attribute missing from column def" assert column_elem.get("index") is not None, "Required 'index' attribute missing from column def" self.columns, self.largest_index, parsed_empty_field_values = self.parse_column_spec_element(config_element) self.empty_field_values.update(parsed_empty_field_values) assert "value" in self.columns, "Required 'value' column missing from column def" def extend_data_with(self, filename: str, errors: Optional[ErrorListT] = None) -> None: here = os.path.dirname(os.path.abspath(filename)) self.data.extend(self.parse_file_fields(filename, errors=errors, here=here)) if not self.allow_duplicate_entries: self._deduplicate_data() def parse_file_fields( self, filename: str, errors: Optional[ErrorListT] = None, here: str = "__HERE__" ) -> List[List[str]]: """ Parse separated lines from file and return a list of tuples. TODO: Allow named access to fields using the column names. """ separator_char = "" if self.separator == "\t" else self.separator rval = [] with open(filename) as fh: for i, line in enumerate(fh): if line.lstrip().startswith(self.comment_char): continue line = line.rstrip("\n\r") if line: line = _expand_here_template(line, here=here) fields = line.split(self.separator) if self.largest_index < len(fields): rval.append(fields) else: line_error = f"Line {i + 1} in tool data table '{self.name}' is invalid (HINT: '{separator_char}' characters must be used to separate fields):\n{line}" if errors is not None: errors.append(line_error) log.warning(line_error) log.debug("Loaded %i lines from '%s' for '%s'", len(rval), filename, self.name) return rval # This method is used in tools, so need to keep its API stable def get_column_name_list(self) -> List[Union[str, None]]: rval: List[Union[str, None]] = [] for i in range(self.largest_index + 1): found_column = False for name, index in self.columns.items(): if index == i: if not found_column: rval.append(name) elif name == "value": # the column named 'value' always has priority over other named columns rval[-1] = name found_column = True if not found_column: rval.append(None) return rval # This method is used in tools, so need to keep its API stable def get_entry(self, query_attr: str, query_val: str, return_attr: str, default: None = None): """ Returns table entry associated with a col/val pair. """ rval = self.get_entries(query_attr, query_val, return_attr, limit=1) if rval: return rval[0] return default def get_entries(self, query_attr: str, query_val: str, return_attr: str, limit=None) -> List: """ Returns table entries associated with a col/val pair. """ query_col = self.columns.get(query_attr, None) if query_col is None: return [] if return_attr is not None: return_col = self.columns.get(return_attr, None) if return_col is None: return [] rval = [] # Look for table entry. for fields in self.get_fields(): if fields[query_col] == query_val: if return_attr is None: field_dict = {} for i, col_name in enumerate(self.get_column_name_list()): field_dict[col_name or i] = fields[i] rval.append(field_dict) else: rval.append(fields[return_col]) if limit is not None and len(rval) == limit: break return rval # This method is used in tools, so need to keep its API stable def get_filename_for_source(self, source: EntrySource, default: Optional[str] = None) -> Optional[str]: source_repo_info: Optional[dict] = None if source: # if dict, assume is compatible info dict, otherwise call method if isinstance(source, dict): source_repo_info = source else: source_repo_info_model: Optional[RepoInfo] if source is None or isinstance(source, RepoInfo): source_repo_info_model = source else: # we have a data manager, use its repo_info method source_repo_info_model = source.repo_info source_repo_info = source_repo_info_model.model_dump() if source_repo_info_model else None filename = default shared_fallback: Optional[str] = None for name, value in self.filenames.items(): repo_info = value.get("tool_shed_repository") if (not source_repo_info and not repo_info) or ( source_repo_info and repo_info and source_repo_info == repo_info ): filename = name break if source_repo_info and not repo_info and shared_fallback is None: # Loc file registered without repo_info (shared install) — use as fallback # when a tool-shed source has no exact match. Preserves legacy behavior where # a ``-tagged filename matches exactly. shared_fallback = name else: if shared_fallback is not None: filename = shared_fallback return filename def _add_entry( self, entry: Union[List[str], Dict[str, str]], allow_duplicates: bool = True, persist: bool = False, entry_source: EntrySource = None, tool_data_file_path: Optional[str] = None, bundle_mode: bool = False, **kwd, ) -> None: # accepts dict or list of columns if isinstance(entry, dict): fields = [] for column_name in self.get_column_name_list(): if column_name not in entry: log.debug( "Using default column value for column '%s' when adding data table entry (%s) to table '%s'.", column_name, entry, self.name, ) field_value = self.get_empty_field_by_name(column_name) else: field_value = entry[column_name] fields.append(field_value) else: fields = entry if self.largest_index < len(fields): fields = self._replace_field_separators(fields) if (allow_duplicates and self.allow_duplicate_entries) or fields not in self.get_fields(): self.data.append(fields) else: raise MessageException( f"Attempted to add fields ({fields}) to data table '{self.name}', but this entry already exists and allow_duplicates is False.", type="warning", ) else: raise MessageException( f"Attempted to add fields ({fields}) to data table '{self.name}', but there were not enough fields specified ( {len(fields)} < {self.largest_index + 1} )." ) if persist: filename = self._locate_filename(tool_data_file_path, entry_source, bundle_mode) if filename is None: # If we reach this point, there is no data table with a corresponding .loc file. raise MessageException( f"Unable to determine filename for persisting data table '{self.name}' values: '{fields}'." ) else: log.debug("Persisting changes to file: %s", filename) data_table_fh: BinaryIO with FileLock(filename): try: if os.path.exists(filename): data_table_fh = open(filename, "r+b") if os.stat(filename).st_size > 0: # ensure last existing line ends with new line data_table_fh.seek(-1, 2) # last char in file last_char = data_table_fh.read(1) if last_char not in [b"\n", b"\r"]: data_table_fh.write(b"\n") else: data_table_fh = open(filename, "wb") except OSError as e: log.exception("Error opening data table file (%s): %s", filename, e) raise fields_collapsed = f"{self.separator.join(fields)}\n" data_table_fh.write(fields_collapsed.encode("utf-8")) def append_entries_with_attribution( self, entries: List[List[str]], attribution: str, allow_duplicates: bool = False, ) -> int: """Append ``entries`` to this table's loc file with an attribution comment line. When ``allow_duplicates`` is False, dedupes entries by the ``value`` column (the table's primary key) against existing in-memory rows and the pending batch. Writes a single ``{comment_char} {attribution}`` line before the first new row. No-op if no rows survive the dedup. """ filename: Optional[str] = self.get_filename_for_source(None) if filename is None: for name in self.filenames: filename = name break if filename is None: raise MessageException(f"Unable to determine filename for appending entries to data table '{self.name}'.") value_index = self.columns.get("value", 0) existing_values: Optional[Set[str]] = None if not allow_duplicates: existing_values = {row[value_index] for row in self.data if value_index < len(row)} new_rows: List[List[str]] = [] for entry in entries: fields = self._replace_field_separators(list(entry)) if self.largest_index >= len(fields): log.warning( "Skipping fields (%s) for data table '%s': only %d field(s) provided.", fields, self.name, len(fields), ) continue if existing_values is not None and fields[value_index] in existing_values: continue new_rows.append(fields) if existing_values is not None: existing_values.add(fields[value_index]) if not new_rows: return self._loaded_content_version with FileLock(filename): if os.path.exists(filename) and os.stat(filename).st_size > 0: with open(filename, "rb+") as fh: fh.seek(-1, 2) if fh.read(1) not in (b"\n", b"\r"): fh.write(b"\n") with open(filename, "ab") as fh: fh.write(f"{self.comment_char} {attribution}\n".encode()) for fields in new_rows: fh.write(f"{self.separator.join(fields)}\n".encode()) for fields in new_rows: self.data.append(fields) return self._update_version() def _locate_filename(self, tool_data_file_path, entry_source, bundle_mode): if tool_data_file_path is not None: filename = tool_data_file_path if os.path.realpath(filename) not in [os.path.realpath(n) for n in self.filenames]: raise MessageException(f"Path '{tool_data_file_path}' is not a known data table file path.") elif not bundle_mode: filename = self.get_filename_for_source(entry_source) else: for name in self.filenames: filename = name break return filename def _remove_entry(self, values): # update every file for filename in self.filenames: if os.path.exists(filename): values = self._replace_field_separators(values) self.filter_file_fields(filename, values) else: log.warning(f"Cannot find index file '{filename}' for tool data table '{self.name}'") self.reload_from_files() def filter_file_fields(self, loc_file, values): """ Reads separated lines from file and print back only the lines that pass a filter. """ with open(loc_file) as reader: rval = "" for line in reader: if line.lstrip().startswith(self.comment_char): rval += line else: line_s = line.rstrip("\n\r") if line_s: fields = line_s.split(self.separator) if fields != values: rval += line with open(loc_file, "w") as writer: writer.write(rval) return rval def _replace_field_separators(self, fields, separator=None, replace=None, comment_char=None): # make sure none of the fields contain separator # make sure separator replace is different from comment_char, # due to possible leading replace if separator is None: separator = self.separator if replace is None: if separator == " ": if comment_char == "\t": replace = "_" else: replace = "\t" else: if comment_char == " ": replace = "_" else: replace = " " return [x.replace(separator, replace) for x in fields] def _deduplicate_data(self): # Remove duplicate entries, without recreating self.data object dup_lines = [] hash_set = set() for i, fields in enumerate(self.data): fields_hash = hash(self.separator.join(fields)) if fields_hash in hash_set: dup_lines.append(i) log.debug( 'Found duplicate entry in tool data table "%s", but duplicates are not allowed, removing additional entry for: "%s"', self.name, fields, ) else: hash_set.add(fields_hash) for i in reversed(dup_lines): self.data.pop(i) @property def xml_string(self): return util.xml_to_string(self.config_element) def to_dict(self, view: str = "collection", value_mapper: Optional[Dict[str, Callable]] = None) -> Dict[str, Any]: rval = super().to_dict(view, value_mapper) if view == "element": rval["columns"] = sorted(self.columns.keys(), key=lambda x: self.columns[x]) rval["fields"] = self.get_fields() return rval class TabularToolDataField(Dictifiable): dict_collection_visible_keys: List[str] = [] def __init__(self, data: Dict): self.data = data def __getitem__(self, key): return self.data[key] def get_base_path(self): return os.path.normpath(os.path.abspath(self.data["path"])) def get_base_dir(self): path = self.get_base_path() if not os.path.isdir(path): path = os.path.dirname(path) return path def clean_base_dir(self, path): return re.sub(f"^{self.get_base_dir()}/*", "", path) def get_files(self): return glob(f"{self.get_base_path()}*") def get_filesize_map(self, rm_base_dir=False): out = {} for path in self.get_files(): if rm_base_dir: out[self.clean_base_dir(path)] = os.path.getsize(path) else: out[path] = os.path.getsize(path) return out def get_fingerprint(self): sha1 = hashlib.sha1() fmap = self.get_filesize_map(True) for k in sorted(fmap.keys()): sha1.update(util.smart_str(k)) sha1.update(util.smart_str(fmap[k])) return sha1.hexdigest() def to_dict(self, view: str = "collection", value_mapper: Optional[Dict[str, Callable]] = None) -> Dict[str, Any]: rval = super().to_dict(view, value_mapper) rval["name"] = self.data["value"] rval["fields"] = self.data rval["base_dir"] = [self.get_base_dir()] rval["files"] = self.get_filesize_map(True) rval["fingerprint"] = self.get_fingerprint() return rval @overload def _expand_here_template(content: str, here: Optional[str]) -> str: ... @overload def _expand_here_template(content: None, here: Optional[str]) -> None: ... def _expand_here_template(content: Optional[str], here: Optional[str]) -> Optional[str]: if here and content: content = string.Template(content).safe_substitute({"__HERE__": here}) return content # Registry of tool data types by type_key tool_data_table_types_list: List[Type[ToolDataTable]] = [TabularToolDataTable] class HasExtraFiles(Protocol): extra_files_path: str def extra_files_path_exists(self) -> bool: ... class DirectoryAsExtraFiles(HasExtraFiles): def __init__(self, directory): self.extra_files_path = directory def extra_files_path_exists(self) -> bool: return True class OutputDataset(HasExtraFiles, Protocol): ext: str def get_file_name(self, sync_cache=True) -> str: ... class ToolDataTableManager(Dictifiable): """Manages a collection of tool data tables""" data_tables: Dict[str, ToolDataTable] tool_data_table_types = {cls.type_key: cls for cls in tool_data_table_types_list} def __init__( self, tool_data_path: str, config_filename: Optional[Union[StrPath, List[StrPath]]] = None, tool_data_table_config_path_set=None, other_config_dict: Optional[StoresConfigFilePaths] = None, filesystem: Optional[ToolDataFilesystem] = None, ) -> None: self.tool_data_path = tool_data_path # This stores all defined data table entries from both the tool_data_table_conf.xml file and the shed_tool_data_table_conf.xml file # at server startup. If tool shed repositories are installed that contain a valid file named tool_data_table_conf.xml.sample, entries # from that file are inserted into this dict at the time of installation. self.data_tables = {} self.tool_data_path_files = ToolDataPathFiles(self.tool_data_path, filesystem=filesystem) self.other_config_dict = other_config_dict or {} for single_config_filename in util.listify(config_filename): if not single_config_filename: continue self.load_from_config_file(single_config_filename, self.tool_data_path, from_shed_config=False) def index(self) -> ToolDataEntryList: data_tables = [ToolDataEntry(**table.to_dict()) for table in self.data_tables.values()] return ToolDataEntryList.model_construct(root=data_tables) def __getitem__(self, key: str) -> ToolDataTable: return self.data_tables.__getitem__(key) def __setitem__(self, key: str, value) -> None: return self.data_tables.__setitem__(key, value) def __contains__(self, key: str) -> bool: return self.data_tables.__contains__(key) def get(self, name: str, default=None): try: return self[name] except KeyError: return default def set(self, name: str, value: ToolDataTable) -> None: self[name] = value def get_tables(self) -> Dict[str, "ToolDataTable"]: return self.data_tables def assert_data_table_consistency( self, candidate_name: str, candidate_columns: Dict[str, int], ) -> None: """Raise if ``candidate_name`` is already registered with different columns.""" existing = self.data_tables.get(candidate_name) if existing is not None: existing_columns = getattr(existing, "columns", None) if existing_columns is not None and existing_columns != candidate_columns: raise DataTableColumnMismatch(candidate_name, existing_columns, candidate_columns) def to_dict( self, view: str = "collection", value_mapper: Optional[Dict[str, Callable]] = None ) -> Dict[str, Dict[str, Any]]: return { name: data_table.to_dict(view="export", value_mapper=value_mapper) for name, data_table in self.data_tables.items() } def to_json(self, path: StrPath) -> None: with open(path, "w") as out: out.write(json.dumps(self.to_dict())) def load_from_config_file( self, config_filename: StrPath, tool_data_path: Optional[StrPath], from_shed_config: bool = False ) -> List[Element]: """ This method is called under 3 conditions: 1. When the ToolDataTableManager is initialized (see __init__ above). 2. Just after the ToolDataTableManager is initialized and the additional entries defined by shed_tool_data_table_conf.xml are being loaded into the ToolDataTableManager.data_tables. 3. When a tool shed repository that includes a tool_data_table_conf.xml.sample file is being installed into a local Galaxy instance. In this case, we have 2 entry types to handle, files whose root tag is , for example: """ table_elems = [] tree = util.parse_xml(config_filename) root = tree.getroot() # Share a single directory listing across every table in this config so # the per-table exists() checks don't each re-walk the tool-data tree. with self.tool_data_path_files.cached(): for table_elem in root.findall("table"): table = self.from_elem( table_elem, tool_data_path, from_shed_config, filename=config_filename, tool_data_path_files=self.tool_data_path_files, other_config_dict=self.other_config_dict, ) table_elems.append(table_elem) if table.name not in self.data_tables: self.data_tables[table.name] = table log.debug("Loaded tool data table '%s' from file '%s'", table.name, config_filename) else: log.debug( "Loading another instance of data table '%s' from file '%s', attempting to merge content.", table.name, config_filename, ) self.data_tables[table.name].merge_tool_data_table( table, allow_duplicates=False ) # only merge content, do not persist to disk, do not allow duplicate rows when merging # FIXME: This does not account for an entry with the same unique build ID, but a different path. return table_elems def from_elem( self, table_elem: Element, tool_data_path: Optional[StrPath], from_shed_config: bool, filename: StrPath, tool_data_path_files: ToolDataPathFiles, other_config_dict: Optional[StoresConfigFilePaths] = None, ) -> ToolDataTable: table_type = table_elem.get("type", "tabular") assert table_type in self.tool_data_table_types, f"Unknown data table type '{table_type}'" return self.tool_data_table_types[table_type]( table_elem, tool_data_path, tool_data_path_files=tool_data_path_files, from_shed_config=from_shed_config, filename=filename, other_config_dict=other_config_dict, ) def add_new_entries_from_config_file( self, config_filename: StrPath, tool_data_path: Optional[StrPath], shed_tool_data_table_config: StrPath, persist: bool = False, ) -> Tuple[List[Element], str]: """ This method is called when a tool shed repository that includes a tool_data_table_conf.xml.sample file is being installed into a local galaxy instance. We have 2 cases to handle, files whose root tag is , for example:: value, dbkey, name, path
and files whose root tag is , for example::
value, dbkey, name, path
""" error_message = "" try: table_elems = self.load_from_config_file( config_filename=config_filename, tool_data_path=tool_data_path, from_shed_config=True ) except Exception as e: error_message = ( f"Error attempting to parse file {str(os.path.split(config_filename)[1])}: {util.unicodify(e)}" ) log.debug(error_message, exc_info=True) table_elems = [] if persist: # Persist Galaxy's version of the changed tool_data_table_conf.xml file. self.to_xml_file(shed_tool_data_table_config, table_elems) return table_elems, error_message def to_xml_file( self, shed_tool_data_table_config: StrPath, new_elems: Optional[List[Element]] = None, remove_elems: Optional[List[Element]] = None, ) -> None: """ Write the current in-memory version of the shed_tool_data_table_conf.xml file to disk. remove_elems are removed before new_elems are added. """ if not (new_elems or remove_elems): log.debug("ToolDataTableManager.to_xml_file called without any elements to add or remove.") return # no changes provided, no need to persist any changes if not new_elems: new_elems = [] if not remove_elems: remove_elems = [] full_path = os.path.abspath(shed_tool_data_table_config) # FIXME: we should lock changing this file by other threads / head nodes try: try: tree = util.parse_xml(full_path) except OSError as e: if e.errno == errno.ENOENT: with open(full_path, "w") as fh: fh.write(TOOL_DATA_TABLE_CONF_XML) tree = util.parse_xml(full_path) else: raise root = tree.getroot() out_elems = list(root) except Exception as e: out_elems = [] log.debug("Could not parse existing tool data table config, assume no existing elements: %s", e) out_elems = [elem for elem in out_elems if elem not in remove_elems] # add new elems out_elems.extend(new_elems) root = util.parse_xml_string('\n') for elem in out_elems: root.append(elem) with RenamedTemporaryFile(full_path, mode="w") as out: out.write(util.xml_to_string(root, pretty=True)) os.chmod(full_path, RW_R__R__) def reload_tables( self, table_names: Optional[Union[List[str], str]] = None, path: Optional[str] = None ) -> List[str]: """ Reload tool data tables. If neither table_names nor path is given, reloads all tool data tables. """ tables = self.get_tables() if not table_names: if path: table_names = self.get_table_names_by_path(path) else: table_names = list(tables.keys()) elif not isinstance(table_names, list): table_names = [table_names] # Share a single, freshly walked directory listing across every table # reloaded in this pass, then drop it so it can't go stale on disk. with self.tool_data_path_files.cached(): for table_name in table_names: tables[table_name].reload_from_files() log.debug("Reloaded tool data table '%s' from files.", table_name) return table_names def get_table_names_by_path(self, path: str) -> List[str]: """Returns a list of table names given a path""" table_names = set() for name, data_table in self.data_tables.items(): if path in data_table.filenames: table_names.add(name) return list(table_names) def process_bundle( self, out_data: Dict[str, OutputDataset], bundle_description: DataTableBundleProcessorDescription, repo_info: Optional[RepoInfo], options: "BundleProcessingOptions", ) -> List[str]: data_manager_dict: Dict[str, Any] = _data_manager_dict(out_data) bundle = DataTableBundle( processor_description=bundle_description, data_tables=data_manager_dict.get("data_tables", {}), repo_info=repo_info, ) return _process_bundle(out_data, bundle, options, self) def import_bundle( self, target: str, options: "BundleProcessingOptions", ) -> List[str]: if not os.path.isdir(target): target_directory = decompress_path_to_directory(target) else: target_directory = target index_json = os.path.join(target_directory, BUNDLE_INDEX_FILE_NAME) with open(index_json) as f: index = json.load(f) bundle = DataTableBundle(**index) assert bundle.output_name out_data = {bundle.output_name: DirectoryAsExtraFiles(target_directory)} return _process_bundle(out_data, bundle, options, self, bundle_mode=True) def write_bundle( self, out_data: Dict[str, OutputDataset], bundle_description: DataTableBundleProcessorDescription, repo_info: Optional[RepoInfo], ) -> Dict[str, OutputDataset]: """Writes bundle and returns bundle path.""" data_manager_dict = _data_manager_dict(out_data, ensure_single_output=True) bundle_datasets: Dict[str, OutputDataset] = {} for output_name, dataset in out_data.items(): if dataset.ext != "data_manager_json": continue bundle = DataTableBundle( data_tables=data_manager_dict.get("data_tables", {}), output_name=output_name, processor_description=bundle_description, repo_info=repo_info, ) extra_files_path = dataset.extra_files_path bundle_path = os.path.join(extra_files_path, BUNDLE_INDEX_FILE_NAME) with open(bundle_path, "w") as fw: fw.write(bundle.model_dump_json()) bundle_datasets[bundle_path] = dataset return bundle_datasets SUPPORTED_DATA_TABLE_TYPES = TabularToolDataTable @dataclass class BundleProcessingOptions: what: str data_manager_path: str target_config_file: str tool_data_file_path: Optional[str] = None def _data_manager_dict(out_data: Dict[str, OutputDataset], ensure_single_output: bool = False) -> Dict[str, Any]: data_manager_dict: Dict[str, Any] = {} found_output = False for output_name, output_dataset in out_data.items(): if output_dataset.ext != "data_manager_json": continue if found_output and ensure_single_output: raise Exception("Galaxy can only write bundles for data managers with a single output data_manager_json.") found_output = True try: output_dict = json.loads(open(output_dataset.get_file_name()).read()) except Exception as e: log.warning(f'Error reading DataManagerTool json for "{output_name}": {e}') continue for key, value in output_dict.items(): if key not in data_manager_dict: data_manager_dict[key] = {} data_manager_dict[key].update(value) data_manager_dict.update(output_dict) return data_manager_dict from typing import Mapping def _process_bundle( out_data: Mapping[str, HasExtraFiles], bundle: DataTableBundle, options: BundleProcessingOptions, tool_data_tables: ToolDataTableManager, bundle_mode: bool = False, ): updated_data_tables = [] data_tables_dict = bundle.data_tables bundle_description = bundle.processor_description for data_table_name in bundle_description.data_table_names: data_table_values = data_tables_dict.pop(data_table_name, None) if not data_table_values: log.warning(f'No values for data table "{data_table_name}" were returned by "{options.what}".') continue # next data table data_table_remove_values = None if isinstance(data_table_values, dict): values_to_add = data_table_values.get("add") data_table_remove_values = data_table_values.get("remove") if values_to_add or data_table_remove_values: # We don't have an old style data table definition data_table_values = values_to_add data_table = tool_data_tables.get(data_table_name, None) if data_table is None: log.error( f'Processing by {options.what} returned an unknown data table "{data_table_name}" with new entries "{data_table_values}". These entries will not be created. Please confirm that an entry for "{data_table_name}" exists in your "tool_data_table_conf.xml" file.' ) continue # next table name if not isinstance(data_table, SUPPORTED_DATA_TABLE_TYPES): log.error( f'Processing by {options.what} returned an unsupported data table "{data_table_name}" with type "{type(data_table)}" with new entries "{data_table_values}". These entries will not be created. Please confirm that the data table is of a supported type ({SUPPORTED_DATA_TABLE_TYPES}).' ) continue # next table name output_ref_values = {} output_ref_by_data_table = bundle_description.output_ref_by_data_table if data_table_name in output_ref_by_data_table: for data_table_column, output_ref in output_ref_by_data_table[data_table_name].items(): output_ref_dataset = out_data.get(output_ref, None) assert output_ref_dataset is not None, "Referenced output was not found." output_ref_values[data_table_column] = output_ref_dataset if not isinstance(data_table_values, list): data_table_values = [data_table_values] if data_table_values else [] if not isinstance(data_table_remove_values, list): data_table_remove_values = [data_table_remove_values] if data_table_remove_values else [] for data_table_row in data_table_values: data_table_value = dict(**data_table_row) # keep original values here for ( name ) in ( data_table_row.keys() ): # FIXME: need to loop through here based upon order listed in data_manager config if name in output_ref_values: _process_move( data_table_name, name, output_ref_values[name].extra_files_path, bundle_description, options, **data_table_value, ) data_table_value[name] = _process_value_translations( data_table_name, name, bundle_description, options, **data_table_value ) data_table.add_entry( data_table_value, persist=True, entry_source=bundle.repo_info, tool_data_file_path=options.tool_data_file_path, bundle_mode=bundle_mode, ) # Removes data table entries for data_table_row in data_table_remove_values: data_table_value = dict(**data_table_row) # keep original values here data_table.remove_entry(list(data_table_value.values())) updated_data_tables.append(data_table_name) if bundle_description.undeclared_tables and data_tables_dict: # We handle the data move, by just moving all the data out of the extra files path # moving a directory and the target already exists, we move the contents instead log.debug("Attempting to add entries for undeclared tables: %s.", ", ".join(data_tables_dict.keys())) for ref_file in out_data.values(): if ref_file.extra_files_path_exists(): util.move_merge(ref_file.extra_files_path, options.data_manager_path) path_column_names = ["path"] for data_table_name, data_table_values in data_tables_dict.items(): data_table = tool_data_tables.get(data_table_name, None) if not isinstance(data_table_values, list): data_table_values = [data_table_values] for data_table_row in data_table_values: data_table_value = dict(**data_table_row) # keep original values here for name, value in data_table_row.items(): if name in path_column_names: data_table_value[name] = os.path.abspath(os.path.join(options.data_manager_path, value)) data_table.add_entry( data_table_value, persist=True, entry_source=bundle.repo_info, bundle_mode=bundle_mode, ) updated_data_tables.append(data_table_name) else: for data_table_name, data_table_values in data_tables_dict.items(): # tool returned extra data table entries, but data table was not declared in data manager # do not add these values, but do provide messages log.warning( f'Processing by {options.what} returned an undeclared data table "{data_table_name}" with new entries "{data_table_values}". These entries will not be created. Please confirm that an entry for "{data_table_name}" exists in your "{options.target_config_file}" file.' ) return updated_data_tables def _process_move( data_table_name: str, column_name: str, source_base_path: str, bundle_description: DataTableBundleProcessorDescription, options: BundleProcessingOptions, **kwd, ): move_by_data_table_column = bundle_description.move_by_data_table_column if data_table_name in move_by_data_table_column and column_name in move_by_data_table_column[data_table_name]: move = move_by_data_table_column[data_table_name][column_name] source = move.source_base if source is None: source = source_base_path else: source = fill_template( source, GALAXY_DATA_MANAGER_DATA_PATH=options.data_manager_path, **kwd, ).strip() assert source if move.source_value: source = os.path.join( source, fill_template( move.source_value, GALAXY_DATA_MANAGER_DATA_PATH=options.data_manager_path, **kwd, ).strip(), ) target = move.target_base if target is None: target = options.data_manager_path else: target = fill_template( target, GALAXY_DATA_MANAGER_DATA_PATH=options.data_manager_path, **kwd, ).strip() assert target if move.target_value: target = os.path.join( target, fill_template( move.target_value, GALAXY_DATA_MANAGER_DATA_PATH=options.data_manager_path, **kwd, ).strip(), ) if move.type == "file": dirs = os.path.split(target)[0] try: os.makedirs(dirs) except OSError as e: if e.errno != errno.EEXIST: raise e # moving a directory and the target already exists, we move the contents instead if os.path.exists(source): util.move_merge(source, target) if move.relativize_symlinks: util.relativize_symlinks(target) return True return False def _process_value_translations( data_table_name: str, column_name: str, bundle_description: DataTableBundleProcessorDescription, options: BundleProcessingOptions, **kwd, ) -> str: value_translation_by_data_table_column = bundle_description.value_translation_by_data_table_column value = kwd.get(column_name) if ( data_table_name in value_translation_by_data_table_column and column_name in value_translation_by_data_table_column[data_table_name] ): for value_translation in value_translation_by_data_table_column[data_table_name][column_name]: if isinstance(value_translation, str): value = fill_template( value_translation, GALAXY_DATA_MANAGER_DATA_PATH=options.data_manager_path, **kwd, ).strip() else: value = value_translation(value) assert value return value ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/data/_schema.py0000644000175100017510000000606015234436334023607 0ustar00runnerrunnerfrom typing import ( Dict, List, Optional, ) from pydantic import ( BaseModel, ConfigDict, Field, RootModel, ) class Model(BaseModel): model_config = ConfigDict(protected_namespaces=()) class ToolDataEntry(Model): name: str = Field( ..., # Mark this field as required title="Name", description="The name of this tool data entry", examples=["all_fasta"], ) model_class: str = Field( ..., # Mark this field as required title="Model class", description="The name of class modelling this tool data", examples=["TabularToolDataTable"], ) class ToolDataEntryList(RootModel): root: List[ToolDataEntry] = Field( title="A list with details on individual data tables.", ) def find_entry(self, name: str) -> Optional[ToolDataEntry]: for entry in self.root: if entry.name == name: return entry return None class ToolDataDetails(ToolDataEntry): columns: List[str] = Field( ..., # Mark this field as required title="Columns", description="A list of column names", examples=["value", "dbkey", "name", "path"], ) # We must use an alias since the name 'fields' # shadows a Model attribute fields_value: List[List[str]] = Field( alias="fields", default=[], title="Fields", description="", # TODO add documentation ) class ToolDataField(Model): name: str = Field( ..., # Mark this field as required title="Name", description="The name of the field", ) model_class: str = Field( ..., # Mark this field as required title="Model class", description="The name of class modelling this tool data field", examples=["TabularToolDataField"], ) # We must use an alias since the name 'fields' # shadows a Model attribute fields_value: Dict[str, str] = Field( ..., # Mark this field as required alias="fields", title="Fields", description="", # TODO add documentation ) base_dir: List[str] = Field( ..., # Mark this field as required title="Base directories", description="A list of directories where the data files are stored", ) files: Dict[str, int] = Field( ..., # Mark this field as required title="Files", description="A dictionary of file names and their size in bytes", examples=[{"file.txt": 136}], ) fingerprint: str = Field( ..., # Mark this field as required title="Fingerprint", description="SHA1 Hash", examples=["22b45237a85c2b3f474bf66888c534387ffe0ced"], ) class ToolDataItem(Model): values: str = Field( ..., # Mark this field as required title="Values", description=( "A `\\t` (TAB) separated list of column __contents__." " You must specify a value for each of the columns of the data table." ), examples=["value\tdbkey\tname\tpath"], ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0422392 galaxy_tool_util-26.1.1/src/galaxy/tool_util/data/bundles/0000755000175100017510000000000015234436377023277 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/data/bundles/__init__.py0000644000175100017510000000000015234436334025367 0ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/data/bundles/models.py0000644000175100017510000002277315234436334025140 0ustar00runnerrunnerimport os from typing import ( Callable, Dict, Iterator, List, Optional, Tuple, Union, ) from pydantic import ( BaseModel, ConfigDict, model_validator, ) from galaxy.util import ( asbool, Element, ) DEFAULT_VALUE_TRANSLATION_TYPE = "template" VALUE_TRANSLATION_FUNCTIONS: Dict[str, Callable] = dict(abspath=os.path.abspath) DEFAULT_VALUE_TRANSLATION_TYPE = "template" class DataTableBundleProcessorDataTableOutputColumnTranslation(BaseModel): type: str value: str model_config = ConfigDict(extra="forbid") class DataTableBundleProcessorDataTableOutputColumnMove(BaseModel): type: str source_base: Optional[str] = None source_value: str = "" target_base: Optional[str] = None target_value: Optional[str] = None relativize_symlinks: bool model_config = ConfigDict(extra="forbid") class DataTableBundleProcessorDataTableOutputColumn(BaseModel): name: str data_table_name: str output_ref: Optional[str] = None value_translations: List[DataTableBundleProcessorDataTableOutputColumnTranslation] = [] moves: List[DataTableBundleProcessorDataTableOutputColumnMove] = [] model_config = ConfigDict(extra="forbid") @model_validator(mode="before") @classmethod def fill_in_default_data_table_name(cls, values): data_table_name = values.get("data_table_name") if data_table_name is None: values["data_table_name"] = values["name"] return values class DataTableBundleProcessorDataTableOutput(BaseModel): columns: List[DataTableBundleProcessorDataTableOutputColumn] model_config = ConfigDict(extra="forbid") class DataTableBundleProcessorDataTable(BaseModel): name: str output: Optional[DataTableBundleProcessorDataTableOutput] = None model_config = ConfigDict(extra="forbid") class DataTableBundleProcessorDescription(BaseModel): undeclared_tables: bool = False data_tables: List[DataTableBundleProcessorDataTable] model_config = ConfigDict(extra="forbid") @property def data_table_names(self) -> List[str]: names = [] for data_table in self.data_tables: data_table_name = data_table.name names.append(data_table_name) return names def _walk_columns(self) -> Iterator[Tuple[str, DataTableBundleProcessorDataTableOutputColumn]]: for data_table in self.data_tables: data_table_name = data_table.name output = data_table.output if output: for column in output.columns: yield (data_table_name, column) @property def output_ref_by_data_table(self) -> Dict[str, Dict[str, str]]: output_refs: Dict[str, Dict[str, str]] = {} for data_table_name, column in self._walk_columns(): data_table_column_name = column.data_table_name output_ref = column.output_ref if output_ref is not None: if data_table_name not in output_refs: output_refs[data_table_name] = {} output_refs[data_table_name][data_table_column_name] = output_ref return output_refs @property def move_by_data_table_column(self) -> Dict[str, Dict[str, DataTableBundleProcessorDataTableOutputColumnMove]]: by_column: Dict[str, Dict[str, DataTableBundleProcessorDataTableOutputColumnMove]] = {} for data_table_name, column in self._walk_columns(): data_table_column_name = column.data_table_name for move in column.moves: if data_table_name not in by_column: by_column[data_table_name] = {} by_column[data_table_name][data_table_column_name] = move return by_column @property def value_translation_by_data_table_column(self) -> Dict[str, Dict[str, List[Union[str, Callable]]]]: by_column: Dict[str, Dict[str, List[Union[str, Callable]]]] = {} for data_table_name, column in self._walk_columns(): data_table_column_name = column.data_table_name for value_translation_model in column.value_translations: value_translation_str = value_translation_model.value value_translation_type = value_translation_model.type if data_table_name not in by_column: by_column[data_table_name] = {} if data_table_column_name not in by_column[data_table_name]: by_column[data_table_name][data_table_column_name] = [] value_translation: Union[str, Callable] if value_translation_type == "function": if value_translation_str in VALUE_TRANSLATION_FUNCTIONS: value_translation = VALUE_TRANSLATION_FUNCTIONS[value_translation_str] else: raise ValueError(f"Unsupported value translation function: '{value_translation}'") else: assert value_translation_type == DEFAULT_VALUE_TRANSLATION_TYPE, ValueError( f"Unsupported value translation type: '{value_translation_type}'" ) value_translation = value_translation_str by_column[data_table_name][data_table_column_name].append(value_translation) return by_column class RepoInfo(BaseModel): tool_shed: str name: str owner: str installed_changeset_revision: str model_config = ConfigDict(extra="forbid") class DataTableBundle(BaseModel): processor_description: DataTableBundleProcessorDescription data_tables: dict output_name: Optional[str] = None repo_info: Optional[RepoInfo] = None def _xml_to_data_table_output_column_move(move_elem: Element) -> DataTableBundleProcessorDataTableOutputColumnMove: move_type = move_elem.get("type", "directory") relativize_symlinks = move_elem.get( "relativize_symlinks", False ) # TODO: should we instead always relativize links? source_elem = move_elem.find("source") if source_elem is None: source_base = None source_value = "" else: source_base = source_elem.get("base", None) source_value = source_elem.text or "" target_elem = move_elem.find("target") if target_elem is None: target_base = None target_value: Optional[str] = "" else: target_base = target_elem.get("base", None) target_value = target_elem.text return DataTableBundleProcessorDataTableOutputColumnMove( type=move_type, source_base=source_base, source_value=source_value, target_base=target_base, target_value=target_value, relativize_symlinks=relativize_symlinks, ) def _xml_to_data_table_output_column_translation( value_translation_elem: Element, ) -> Optional[DataTableBundleProcessorDataTableOutputColumnTranslation]: value_translation = value_translation_elem.text if value_translation is not None: value_translation_type = value_translation_elem.get("type", DEFAULT_VALUE_TRANSLATION_TYPE) return DataTableBundleProcessorDataTableOutputColumnTranslation( value=value_translation, type=value_translation_type ) else: return None def _xml_to_data_table_output_column(column_elem: Element) -> DataTableBundleProcessorDataTableOutputColumn: column_name = column_elem.get("name", None) assert column_name is not None, "Name is required for column entry" data_table_column_name = column_elem.get("data_table_name", column_name) output_ref = column_elem.get("output_ref", None) value_translation_elems = column_elem.findall("value_translation") value_translations = [] if value_translation_elems is not None: for value_translation_elem in value_translation_elems: value_translation = _xml_to_data_table_output_column_translation(value_translation_elem) if value_translation is None: continue value_translations.append(value_translation) moves = [] for move_elem in column_elem.findall("move"): moves.append(_xml_to_data_table_output_column_move(move_elem)) return DataTableBundleProcessorDataTableOutputColumn( name=column_name, data_table_name=data_table_column_name, output_ref=output_ref, value_translations=value_translations, moves=moves, ) def _xml_to_data_table_output(output_elem: Optional[Element]) -> Optional[DataTableBundleProcessorDataTableOutput]: if output_elem is not None: columns = [] for column_elem in output_elem.findall("column"): columns.append(_xml_to_data_table_output_column(column_elem)) return DataTableBundleProcessorDataTableOutput(columns=columns) else: return None def _xml_to_data_table(data_table_elem: Element) -> DataTableBundleProcessorDataTable: data_table_name = data_table_elem.get("name") assert data_table_name is not None, "A name is required for a data table entry" output_elem = data_table_elem.find("output") output = _xml_to_data_table_output(output_elem) return DataTableBundleProcessorDataTable(name=data_table_name, output=output) def convert_data_tables_xml(elem: Element) -> DataTableBundleProcessorDescription: undeclared_tables = asbool(elem.get("undeclared_tables", False)) data_tables = [] for data_table_elem in elem.findall("data_table"): data_tables.append(_xml_to_data_table(data_table_elem)) return DataTableBundleProcessorDescription(undeclared_tables=undeclared_tables, data_tables=data_tables) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1785871615.0450246 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/0000755000175100017510000000000015234436377021665 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/__init__.py0000644000175100017510000005067615234436334024005 0ustar00runnerrunner""" Dependency management for tools. """ import json import logging import os.path import shutil from typing import ( Any, Dict, List, Optional, Type, TYPE_CHECKING, ) from typing_extensions import Protocol from galaxy.util import ( hash_util, plugin_config, string_as_bool, ) from galaxy.util.oset import OrderedSet from .container_resolvers import ContainerResolver from .dependencies import ToolInfo from .requirements import ( ContainerDescription, ToolRequirement, ToolRequirements, ) from .resolvers import ( ContainerDependency, NullDependency, ) from .resolvers.tool_shed_packages import ToolShedPackageDependencyResolver if TYPE_CHECKING: from galaxy.util.path import StrPath log = logging.getLogger(__name__) CONFIG_VAL_NOT_FOUND = object() def build_dependency_manager( app_config_dict: Optional[Dict[str, Any]] = None, resolution_config_dict: Optional[Dict[str, Any]] = None, conf_file: Optional[str] = None, default_tool_dependency_dir: Optional[str] = None, ) -> "DependencyManager": """Build a DependencyManager object from app and/or resolution config. If app_config_dict is specified, it should be application configuration information and configuration options are generally named to identify the context of dependency management (e.g. conda_prefix not prefix or use_cached_dependency_manager not cache). resolution_config_dict if specified is assumed to be the to_dict() version of a DependencyManager and should only contain dependency configuration options. """ if app_config_dict is None: app_config_dict = {} else: app_config_dict = app_config_dict.copy() tool_dependency_dir = app_config_dict.get("tool_dependency_dir", default_tool_dependency_dir) if tool_dependency_dir and tool_dependency_dir.lower() == "none": app_config_dict["tool_dependency_dir"] = None if resolution_config_dict is None and "dependency_resolution" in app_config_dict: resolution_config_dict = app_config_dict["dependency_resolution"] if resolution_config_dict: # Convert local to_dict options into global ones. # to_dict() has "cache", "cache_dir", "use", "default_base_path", "resolvers", "precache" app_config_props_from_resolution_config = { "use_tool_dependencies": resolution_config_dict.get("use", None), "tool_dependency_dir": resolution_config_dict.get("default_base_path", None), "dependency_resolvers": resolution_config_dict.get("resolvers", None), "tool_dependency_cache_dir": resolution_config_dict.get("cache_dir", None), "precache_dependencies": resolution_config_dict.get("precache", None), "use_cached_dependency_manager": resolution_config_dict.get("cache", None), } for key, value in app_config_props_from_resolution_config.items(): if value is not None: app_config_dict[key] = value # Re-evaluate tool_dependency_dir as app_config_dict may have been modified tool_dependency_dir = app_config_dict.get("tool_dependency_dir", default_tool_dependency_dir) # Default use_tool_dependencies to True but check that tool_dependency_dir is set in the following if use_tool_dependencies = app_config_dict.get("use_tool_dependencies", True) if use_tool_dependencies and tool_dependency_dir: if string_as_bool(app_config_dict.get("use_cached_dependency_manager")): return CachedDependencyManager(tool_dependency_dir, conf_file=conf_file, app_config=app_config_dict) return DependencyManager(tool_dependency_dir, conf_file=conf_file, app_config=app_config_dict) return NullDependencyManager() ContainerType = str DestinationId = str DestinationParametersType = Dict[str, Any] class DestinationProtocol(Protocol): id: Optional[DestinationId] params: DestinationParametersType class DependencyManager: """ A DependencyManager attempts to resolve named and versioned dependencies by searching for them under a list of directories. Directories should be of the form: $BASE/name/version/... and should each contain a file 'env.sh' which can be sourced to make the dependency available in the current shell environment. """ _destination_for_container_type: Dict[ContainerType, List[DestinationProtocol]] cached = False def __init__( self, default_base_path: str, conf_file: Optional[str] = None, app_config: Optional[Dict[str, Any]] = None ) -> None: """ Create a new dependency manager looking for packages under the paths listed in `base_paths`. The default base path is app.config.tool_dependency_dir. """ if app_config is None: app_config = {} if not os.path.exists(default_base_path): log.warning("Path '%s' does not exist, ignoring", default_base_path) if not os.path.isdir(default_base_path): log.warning("Path '%s' is not directory, ignoring", default_base_path) self.__app_config = app_config self.default_base_path = os.path.abspath(default_base_path) self.resolver_classes = self.__resolvers_dict() plugin_source = None dependency_resolver_dicts = app_config.get("dependency_resolvers") if dependency_resolver_dicts is not None: plugin_source = plugin_config.PluginConfigSource("dict", dependency_resolver_dicts) else: plugin_source = self.__build_dependency_resolvers_plugin_source(conf_file) self.dependency_resolvers = self.__parse_resolver_conf_plugins(plugin_source) self._enabled_container_types: List[str] = [] self._destination_for_container_type = {} def set_enabled_container_types( self, container_types_to_destinations: Dict[ContainerType, List[DestinationProtocol]] ): """Set the union of all enabled container types.""" self._enabled_container_types = list(container_types_to_destinations.keys()) # Just pick first enabled destination for a container type, probably covers the most common deployment scenarios self._destination_for_container_type = container_types_to_destinations def get_destination_info_for_container_type( self, container_type: ContainerType, destination_id: Optional[DestinationId] = None ) -> Optional[DestinationParametersType]: if destination_id is None: return next(iter(self._destination_for_container_type[container_type])).params else: for destination in self._destination_for_container_type[container_type]: if destination.id == destination_id: return destination.params return None @property def enabled_container_types(self): """Returns the union of enabled container types.""" return self._enabled_container_types def get_resolver_option(self, resolver, key, explicit_resolver_options=None): """Look in resolver-specific settings for option and then fallback to global settings.""" if explicit_resolver_options is None: explicit_resolver_options = {} default = resolver.config_options.get(key) config_prefix = resolver.resolver_type global_key = f"{config_prefix}_{key}" value = explicit_resolver_options.get(key, CONFIG_VAL_NOT_FOUND) if value is CONFIG_VAL_NOT_FOUND: value = self.get_app_option(global_key, default) return value def get_app_option(self, key, default=None): value = CONFIG_VAL_NOT_FOUND if isinstance(self.__app_config, dict): value = self.__app_config.get(key, CONFIG_VAL_NOT_FOUND) else: value = getattr(self.__app_config, key, CONFIG_VAL_NOT_FOUND) if value is CONFIG_VAL_NOT_FOUND and hasattr(self.__app_config, "config_dict"): value = self.__app_config.config_dict.get(key, CONFIG_VAL_NOT_FOUND) if value is CONFIG_VAL_NOT_FOUND: value = default return value @property def precache(self): return string_as_bool(self.get_app_option("precache_dependencies", True)) def dependency_shell_commands(self, requirements: ToolRequirements, **kwds: Any) -> List[str]: requirements_to_dependencies = self.requirements_to_dependencies(requirements, **kwds) ordered_dependencies = OrderedSet(requirements_to_dependencies.values()) return [ dependency.shell_commands() for dependency in ordered_dependencies if not isinstance(dependency, ContainerDependency) ] def requirements_to_dependencies(self, requirements, **kwds): """ Takes a list of requirements and returns a dictionary with requirements as key and dependencies as value caching these on the tool instance if supplied. """ requirement_to_dependency = self._requirements_to_dependencies_dict(requirements, **kwds) if "tool_instance" in kwds: kwds["tool_instance"].dependencies = [dep.to_dict() for dep in requirement_to_dependency.values()] return requirement_to_dependency def _requirements_to_dependencies_dict(self, requirements, search=False, **kwds): """Build simple requirements to dependencies dict for resolution.""" requirement_to_dependency = {} index = kwds.get("index") install = kwds.get("install", False) resolver_type = kwds.get("resolver_type") include_containers = kwds.get("include_containers", False) container_type = kwds.get("container_type") require_exact = kwds.get("exact", False) return_null_dependencies = kwds.get("return_null", False) resolvable_requirements = requirements.resolvable tool_info_kwds = dict(requirements=resolvable_requirements) if "tool_instance" in kwds: tool = kwds["tool_instance"] tool_info_kwds["tool_id"] = tool.id tool_info_kwds["tool_version"] = tool.version tool_info_kwds["container_descriptions"] = tool.containers tool_info_kwds["requires_galaxy_python_environment"] = tool.requires_galaxy_python_environment tool_info = ToolInfo(**tool_info_kwds) for i, resolver in enumerate(self.dependency_resolvers): if index is not None and i != index: continue if resolver_type is not None and resolver.resolver_type != resolver_type: continue if container_type is not None and getattr(resolver, "container_type", None) != container_type: continue _requirement_to_dependency = { k: v for k, v in requirement_to_dependency.items() if not isinstance(v, NullDependency) } if len(_requirement_to_dependency) == len(resolvable_requirements): # Shortcut - resolution complete. break # Check requirements all at once all_unmet = len(_requirement_to_dependency) == 0 if hasattr(resolver, "resolve_all"): resolve = resolver.resolve_all elif isinstance(resolver, ContainerResolver): if not include_containers: continue if not install and resolver.builds_on_resolution: # don't want to build images here continue if not resolver.resolver_type.startswith(("cached", "explicit", "fallback")) and not ( search or install ): # These would look up available containers using the quay API, # we only want to do this if we search for containers continue resolve = resolver.resolve else: resolve = None if all_unmet and resolve is not None: # TODO: Handle specs. dependencies = resolve( requirements=resolvable_requirements, enabled_container_types=self.enabled_container_types, destination_for_container_type=self.get_destination_info_for_container_type, tool_info=tool_info, **kwds, ) if dependencies: if isinstance(dependencies, ContainerDescription): dependencies = [ ContainerDependency( dependencies, name=r.name, version=r.version, container_resolver=resolver ) for r in resolvable_requirements ] assert len(dependencies) == len(resolvable_requirements) for requirement, dependency in zip(resolvable_requirements, dependencies): log.debug(dependency.resolver_msg) requirement_to_dependency[requirement] = dependency # Shortcut - resolution complete. break if not isinstance(resolver, ContainerResolver): # Check individual requirements for requirement in resolvable_requirements: if requirement in _requirement_to_dependency: continue dependency = resolver.resolve(requirement, **kwds) if require_exact and not dependency.exact: continue if not isinstance(dependency, NullDependency): log.debug(dependency.resolver_msg) requirement_to_dependency[requirement] = dependency elif return_null_dependencies: log.debug(dependency.resolver_msg) dependency.version = requirement.version requirement_to_dependency[requirement] = dependency return requirement_to_dependency def uses_tool_shed_dependencies(self): return any(isinstance(r, ToolShedPackageDependencyResolver) for r in self.dependency_resolvers) def find_dep(self, name: str, version: Optional[str] = None, type: str = "package", **kwds): log.debug(f"Find dependency {name} version {version}") requirements = ToolRequirements([ToolRequirement(name=name, version=version, type=type)]) dep_dict = self._requirements_to_dependencies_dict(requirements, **kwds) if len(dep_dict) > 0: return next(iter(dep_dict.values())) # get first dep else: return NullDependency(name=name, version=version) def __build_dependency_resolvers_plugin_source( self, conf_file: Optional["StrPath"] ) -> plugin_config.PluginConfigSource: if not conf_file: return self.__default_dependency_resolvers_source() if not os.path.exists(conf_file): log.debug("Unable to find config file '%s'", conf_file) return self.__default_dependency_resolvers_source() plugin_source = plugin_config.plugin_source_from_path(conf_file) return plugin_source def __default_dependency_resolvers_source(self): return plugin_config.PluginConfigSource( "dict", [ {"type": "tool_shed_packages"}, {"type": "galaxy_packages"}, {"type": "conda"}, {"type": "galaxy_packages", "versionless": True}, {"type": "conda", "versionless": True}, ], ) def __parse_resolver_conf_plugins(self, plugin_source: plugin_config.PluginConfigSource) -> List: """ """ extra_kwds = dict(dependency_manager=self) # Use either 'type' from YAML definition or 'resolver_type' from to_dict definition. return plugin_config.load_plugins( self.resolver_classes, plugin_source, extra_kwds, plugin_type_keys=["type", "resolver_type"] ) def __resolvers_dict(self) -> Dict[str, Type]: import galaxy.tool_util.deps.resolvers return plugin_config.plugins_dict(galaxy.tool_util.deps.resolvers, "resolver_type") def to_dict(self): return { "use": True, "cache": self.cached, "precache": self.precache, "cache_dir": getattr(self, "tool_dependency_cache_dir", None), "default_base_path": self.default_base_path, "resolvers": [m.to_dict() for m in self.dependency_resolvers], } class CachedDependencyManager(DependencyManager): cached = True def __init__( self, default_base_path: str, conf_file: Optional[str] = None, app_config: Optional[Dict[str, Any]] = None ) -> None: super().__init__(default_base_path, conf_file, app_config) self.tool_dependency_cache_dir = self.get_app_option("tool_dependency_cache_dir") or os.path.join( default_base_path, "_cache" ) def build_cache(self, requirements, **kwds): resolved_dependencies = self.requirements_to_dependencies(requirements, **kwds) cacheable_dependencies = [dep for dep in resolved_dependencies.values() if dep.cacheable] hashed_dependencies_dir = self.get_hashed_dependencies_path(cacheable_dependencies) if os.path.exists(hashed_dependencies_dir): if kwds.get("force_rebuild", False): try: shutil.rmtree(hashed_dependencies_dir) except Exception: log.warning(f"Could not delete cached dependencies directory '{hashed_dependencies_dir}'") raise else: log.debug("Cached dependencies directory '%s' already exists, skipping build", hashed_dependencies_dir) return [dep.build_cache(hashed_dependencies_dir) for dep in cacheable_dependencies] def dependency_shell_commands(self, requirements, **kwds): """ Runs a set of requirements through the dependency resolvers and returns a list of commands required to activate the dependencies. If dependencies are cacheable and the cache does not exist, will try to create it. If cached environment exists or is successfully created, will generate commands to activate it. """ resolved_dependencies = self.requirements_to_dependencies(requirements, **kwds) cacheable_dependencies = [dep for dep in resolved_dependencies.values() if dep.cacheable] hashed_dependencies_dir = self.get_hashed_dependencies_path(cacheable_dependencies) if not os.path.exists(hashed_dependencies_dir) and self.precache: # Cache not present, try to create it self.build_cache(requirements, **kwds) if os.path.exists(hashed_dependencies_dir): [dep.set_cache_path(hashed_dependencies_dir) for dep in cacheable_dependencies] commands = [dep.shell_commands() for dep in resolved_dependencies.values()] return commands def hash_dependencies(self, resolved_dependencies): """Return hash for dependencies""" resolved_dependencies = [ (dep.name, dep.version, dep.exact, dep.dependency_type) for dep in resolved_dependencies ] hash_str = json.dumps(sorted(resolved_dependencies)) return hash_util.new_insecure_hash(hash_str)[:8] # short hash def get_hashed_dependencies_path(self, resolved_dependencies): """ Returns the path to the hashed dependencies directory (but does not evaluate whether the path exists). :param resolved_dependencies: list of resolved dependencies :type resolved_dependencies: list :return: path :rtype: str """ req_hashes = self.hash_dependencies(resolved_dependencies) return os.path.abspath(os.path.join(self.tool_dependency_cache_dir, req_hashes)) class NullDependencyManager(DependencyManager): cached = False def __init__(self, default_base_path=None, conf_file=None, app_config=None): if app_config is None: app_config = {} self.__app_config = app_config self.resolver_classes = set() self.dependency_resolvers = [] self._enabled_container_types = [] self._destination_for_container_type = {} self.default_base_path = None def uses_tool_shed_dependencies(self): return False def dependency_shell_commands(self, requirements, **kwds): return [] def find_dep(self, name, version=None, type="package", **kwds): return NullDependency(version=version, name=name) def to_dict(self): return {"use": False} ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/brew_exts.py0000755000175100017510000004612615234436334024246 0ustar00runnerrunner#!/usr/bin/env python # % brew vinstall samtools 1.0 # % brew vinstall samtools 0.1.19 # % brew vinstall samtools 1.1 # % brew env samtools 1.1 # PATH=/home/john/.linuxbrew/Cellar/htslib/1.1/bin:/home/john/.linuxbrew/Cellar/samtools/1.1/bin:$PATH # export PATH # LD_LIBRARY_PATH=/home/john/.linuxbrew/Cellar/htslib/1.1/lib:/home/john/.linuxbrew/Cellar/samtools/1.1/lib:$LD_LIBRARY_PATH # export LD_LIBRARY_PATH # % . <(brew env samtools 1.1) # % which samtools # /home/john/.linuxbrew/Cellar/samtools/1.1/bin/samtools # % . <(brew env samtools 0.1.19) # % which samtools # /home/john/.linuxbrew/Cellar/samtools/0.1.19/bin/samtools # % brew vuninstall samtools 1.0 # % brew vdeps samtools 1.1 # htslib@1.1 # % brew vdeps samtools 0.1.19 import argparse import contextlib import glob import json import os import re import string import subprocess import sys from typing import ( List, Tuple, ) WHITESPACE_PATTERN = re.compile(r"[\s]+") DESCRIPTION = "Script built on top of linuxbrew to operate on isolated, versioned brew installed environments." if sys.platform == "darwin": DEFAULT_HOMEBREW_ROOT = "/usr/local" else: DEFAULT_HOMEBREW_ROOT = os.path.join(os.path.expanduser("~"), ".linuxbrew") NO_BREW_ERROR_MESSAGE = "Could not find brew on PATH, please place on path or pass to script with --brew argument." CANNOT_DETERMINE_TAP_ERROR_MESSAGE = ( "Cannot determine tap of specified recipe - please use fully qualified recipe (e.g. homebrew/science/samtools)." ) VERBOSE = False RELAXED = False BREW_ARGS: List[str] = [] class BrewContext: def __init__(self, args=None): ensure_brew_on_path(args) raw_config = brew_execute(["config"]) config_lines = [line.strip().split(":", 1) for line in raw_config.split("\n") if line] config = {p[0].strip(): p[1].strip() for p in config_lines} # unset if "/usr/local" -> https://github.com/Homebrew/homebrew/blob/master/Library/Homebrew/cmd/config.rb homebrew_prefix = config.get("HOMEBREW_PREFIX", "/usr/local") homebrew_cellar = config.get("HOMEBREW_CELLAR", os.path.join(homebrew_prefix, "Cellar")) self.homebrew_prefix = homebrew_prefix self.homebrew_cellar = homebrew_cellar class RecipeContext: @staticmethod def from_args(args, brew_context=None): return RecipeContext(args.recipe, args.version, brew_context) def __init__(self, recipe, version, brew_context=None): self.recipe = recipe self.version = version self.brew_context = brew_context or BrewContext() @property def cellar_path(self): return recipe_cellar_path(self.brew_context.homebrew_cellar, self.recipe, self.version) @property def tap_path(self) -> str: return os.path.join(self.brew_context.homebrew_prefix, "Library", "Taps", self.__tap_path(self.recipe)) def __tap_path(self, recipe): parts = recipe.split("/") if len(parts) == 1: info = brew_info(self.recipe) from_url = info["from_url"] if not from_url: raise Exception(CANNOT_DETERMINE_TAP_ERROR_MESSAGE) from_url_parts = from_url.split("/") blob_index = from_url_parts.index("blob") # comes right after username and repository if blob_index < 2: raise Exception(CANNOT_DETERMINE_TAP_ERROR_MESSAGE) username = from_url_parts[blob_index - 2] repository = from_url_parts[blob_index - 1] else: assert len(parts) == 3 parts = recipe.split("/") username = parts[0] repository = f"homebrew-{parts[1]}" path = os.path.join(username, repository) return path def main(): global VERBOSE global RELAXED global BREW_ARGS parser = argparse.ArgumentParser(description=DESCRIPTION) parser.add_argument("--brew", help="Path to linuxbrew 'brew' executable to target") actions = ["vinstall", "vuninstall", "vdeps", "vinfo", "env"] action = __action(sys) if not action: parser.add_argument("action", metavar="action", help="Versioned action to perform.", choices=actions) parser.add_argument( "recipe", metavar="recipe", help="Recipe for action - should be absolute (e.g. homebrew/science/samtools)." ) parser.add_argument("version", metavar="version", help="Version for action (e.g. 0.1.19).") parser.add_argument( "--relaxed", action="store_true", help="Relaxed processing - for instance allow use of env on non-vinstall-ed recipes.", ) parser.add_argument("--verbose", action="store_true", help="Verbose output") parser.add_argument("restargs", nargs=argparse.REMAINDER) args = parser.parse_args() if args.verbose: VERBOSE = True if args.relaxed: RELAXED = True BREW_ARGS = args.restargs if not action: action = args.action brew_context = BrewContext(args) recipe_context = RecipeContext.from_args(args, brew_context) if action == "vinstall": versioned_install(recipe_context, args.recipe, args.version) elif action == "vuninstall": brew_execute(["switch", args.recipe, args.version]) brew_execute(["uninstall", args.recipe]) elif action == "vdeps": print_versioned_deps(recipe_context, args.recipe, args.version) elif action == "env": env_statements = build_env_statements_from_recipe_context(recipe_context) print(env_statements) elif action == "vinfo": with brew_head_at_version(recipe_context, args.recipe, args.version): print(brew_info(args.recipe)) else: raise NotImplementedError() class CommandLineException(Exception): def __init__(self, command, stdout, stderr): self.command = command self.stdout = stdout self.stderr = stderr self.message = ( f"Failed to execute command-line {command}, stderr was:\n" "-------->>begin stderr<<--------\n" f"{stderr}\n" "-------->>end stderr<<--------\n" "-------->>begin stdout<<--------\n" f"{stdout}\n" "-------->>end stdout<<--------\n" ) def __str__(self): return self.message def versioned_install(recipe_context, package=None, version=None, installed_deps=None): if installed_deps is None: installed_deps = [] if package is None: package = recipe_context.recipe version = recipe_context.version attempt_unlink(package) with brew_head_at_version(recipe_context, package, version): deps = brew_deps(package) deps_metadata = [] dep_to_version = {} for dep in deps: version_info = brew_versions_info(dep, recipe_context.tap_path)[0] dep_version = version_info[0] dep_to_version[dep] = dep_version versioned = version_info[2] if versioned: dep_to_version[dep] = dep_version if dep in installed_deps: continue versioned_install(recipe_context, dep, dep_version) installed_deps.append(dep) else: # Install latest. dep_to_version[dep] = None if dep in installed_deps: continue unversioned_install(dep) try: for dep in deps: dep_version = dep_to_version[dep] if dep_version: brew_execute(["switch", dep, dep_version]) else: brew_execute(["link", dep]) # dep_version obtained from brew versions doesn't # include revision. This linked_keg attribute does. keg_verion = brew_info(dep)["linked_keg"] dep_metadata = {"name": dep, "version": keg_verion, "versioned": versioned} deps_metadata.append(dep_metadata) cellar_root = recipe_context.brew_context.homebrew_cellar cellar_path = recipe_context.cellar_path env_actions = build_env_actions(deps_metadata, cellar_root, cellar_path, custom_only=True) env = EnvAction.build_env(env_actions) args = ["install"] if VERBOSE: args.append("--verbose") args.extend(BREW_ARGS) args.append(package) brew_execute(args, env=env) deps = brew_execute(["deps", package]) deps = [d.strip() for d in deps.split("\n") if d] metadata = {"deps": deps_metadata} cellar_root = recipe_context.brew_context.homebrew_cellar cellar_path = recipe_cellar_path(cellar_root, package, version) v_metadata_path = os.path.join(cellar_path, "INSTALL_RECEIPT_VERSIONED.json") with open(v_metadata_path, "w") as f: json.dump(metadata, f) finally: attempt_unlink_all(package, deps) def commit_for_version(recipe_context: RecipeContext, package, version): tap_path = recipe_context.tap_path commit = None with brew_head_at_commit("master", tap_path): version_to_commit = brew_versions_info(package, tap_path) if version is None: version = version_to_commit[0][0] commit = version_to_commit[0][1] else: for mapping in version_to_commit: if mapping[0] == version: commit = mapping[1] if commit is None: raise Exception(f"Failed to find commit for version {version}") return commit def print_versioned_deps(recipe_context, recipe, version): deps = load_versioned_deps(recipe_context.cellar_path) for dep in deps: val = dep["name"] if dep["versioned"]: val += f"@{dep['version']}" print(val) def load_versioned_deps(cellar_path, relaxed=None): if relaxed is None: relaxed = RELAXED v_metadata_path = os.path.join(cellar_path, "INSTALL_RECEIPT_VERSIONED.json") if not os.path.isfile(v_metadata_path): if RELAXED: return [] else: raise OSError(f"Could not locate versioned receipt file: {v_metadata_path}") with open(v_metadata_path) as f: metadata = json.load(f) return metadata["deps"] def unversioned_install(package): try: deps = brew_deps(package) for dep in deps: brew_execute(["link", dep]) brew_execute(["install", package]) finally: attempt_unlink_all(package, deps) def attempt_unlink_all(package, deps): for dep in deps: attempt_unlink(dep) attempt_unlink(package) def attempt_unlink(package): try: brew_execute(["unlink", package]) except Exception: # TODO: warn pass def brew_execute(args, env=None): os.environ["HOMEBREW_NO_EMOJI"] = "1" # simplify brew parsing. cmds = ["brew"] + args return execute(cmds, env=env) def build_env_statements_from_recipe_context(recipe_context, **kwds): cellar_root = recipe_context.brew_context.homebrew_cellar env_statements = build_env_statements(cellar_root, recipe_context.cellar_path, **kwds) return env_statements def build_env_statements(cellar_root, cellar_path, relaxed=None, custom_only=False): deps = load_versioned_deps(cellar_path, relaxed=relaxed) actions = build_env_actions(deps, cellar_root, cellar_path, relaxed, custom_only) env_statements = [] for action in actions: env_statements.extend(action.to_statements()) return "\n".join(env_statements) def build_env_actions(deps, cellar_root, cellar_path, relaxed=None, custom_only=False): path_appends = [] ld_path_appends = [] actions = [] def handle_keg(cellar_path): bin_path = os.path.join(cellar_path, "bin") if os.path.isdir(bin_path): path_appends.append(bin_path) lib_path = os.path.join(cellar_path, "lib") if os.path.isdir(lib_path): ld_path_appends.append(lib_path) env_path = os.path.join(cellar_path, "platform_environment.json") if os.path.exists(env_path): with open(env_path) as f: env_metadata = json.load(f) if "actions" in env_metadata: def to_action(desc): return EnvAction(cellar_path, desc) actions.extend(map(to_action, env_metadata["actions"])) for dep in deps: package = dep["name"] version = dep["version"] dep_cellar_path = recipe_cellar_path(cellar_root, package, version) handle_keg(dep_cellar_path) handle_keg(cellar_path) if not custom_only: if path_appends: actions.append( EnvAction(cellar_path, {"action": "prepend", "variable": "PATH", "value": ":".join(path_appends)}) ) if ld_path_appends: actions.append( EnvAction( cellar_path, {"action": "prepend", "variable": "LD_LIBRARY_PATH", "value": ":".join(path_appends)} ) ) return actions class EnvAction: def __init__(self, keg_root, action_description): self.variable = action_description["variable"] self.action = action_description["action"] self.value = string.Template(action_description["value"]).safe_substitute( { "KEG_ROOT": keg_root, } ) @staticmethod def build_env(env_actions): new_env = os.environ.copy() (env_action.modify_environ(new_env) for env_action in env_actions) return new_env def modify_environ(self, environ): if self.action == "set" or not environ.get(self.variable, ""): environ[self.variable] = self.__eval("${value}") elif self.action == "prepend": environ[self.variable] = self.__eval(f"${{value}}:{environ[self.variable]}") else: environ[self.variable] = self.__eval(f"{environ[self.variable]}:${{value}}") def __eval(self, template): return string.Template(template).safe_substitute( variable=self.variable, value=self.value, ) def to_statements(self): if self.action == "set": template = '''${variable}="${value}"''' elif self.action == "prepend": template = '''${variable}="${value}:$$${variable}"''' else: template = '''${variable}="$$${variable}:${value}"''' return [self.__eval(template), f"export {self.variable}"] @contextlib.contextmanager def brew_head_at_version(recipe_context, package, version): commit = commit_for_version(recipe_context, package, version) tap_path = recipe_context.tap_path with brew_head_at_commit(commit, tap_path): yield @contextlib.contextmanager def brew_head_at_commit(commit, tap_path): try: os.chdir(tap_path) current_commit = git_execute(["rev-parse", "HEAD"]).strip() try: git_execute(["checkout", commit]) yield finally: git_execute(["checkout", current_commit]) finally: # TODO: restore chdir - or better yet just don't chdir # shouldn't be needed. pass def git_execute(args): cmds = ["git"] + args return execute(cmds) def execute(cmds, env=None): subprocess_kwds = dict( shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) if env: subprocess_kwds["env"] = env p = subprocess.Popen(cmds, **subprocess_kwds) # log = p.stdout.read() stdout, stderr = p.communicate() if p.returncode != 0: raise CommandLineException(" ".join(cmds), stdout, stderr) if VERBOSE: print(stdout) return stdout def brew_deps(package): args = ["deps"] args.extend(BREW_ARGS) args.append(package) stdout = brew_execute(args) return [p.strip() for p in stdout.split("\n") if p] def brew_info(recipe): info_json = brew_execute(["info", "--json=v1", recipe]) info = json.loads(info_json)[0] info.update(extended_brew_info(recipe)) return info def extended_brew_info(recipe): # Extract more info from non-json variant. JSON variant should # include this in a backward compatible way (TODO: Open PR). raw_info = brew_execute(["info", recipe]) extra_info = dict( from_url=None, build_dependencies=[], required_dependencies=[], recommended_dependencies=[], optional_dependencies=[], ) for line in raw_info.split("\n"): if line.startswith("From: "): extra_info["from_url"] = line[len("From: ") :].strip() for dep_type in ["Build", "Required", "Recommended", "Optional"]: if line.startswith(f"{dep_type}: "): key = f"{dep_type.lower()}_dependencies" raw_val = line[len(f"{dep_type}: ") :] extra_info[key].extend(raw_val.split(", ")) return extra_info def brew_versions_info(package, tap_path: str) -> List[Tuple[str, str, bool]]: def versioned(recipe_path: str): if not os.path.isabs(recipe_path): recipe_path = os.path.join(os.getcwd(), recipe_path) # Dependencies in the same repository should be versioned, # core dependencies (presumably in base homebrew) are not # versioned. return tap_path in recipe_path # TODO: Also use tags. stdout = brew_execute(["versions", package]) version_parts = [line for line in stdout.split("\n") if line and "git checkout" in line] version_parts = [WHITESPACE_PATTERN.split(line) for line in version_parts] info = [(p[0], p[3], versioned(p[4])) for p in version_parts] return info def __action(sys): script_name = os.path.basename(sys.argv[0]) if script_name.startswith("brew-"): return script_name[len("brew-") :] else: return None def recipe_cellar_path(cellar_path, recipe, version): recipe_base = recipe.split("/")[-1] recipe_base_path = os.path.join(cellar_path, recipe_base, version) revision_paths = glob.glob(f"{recipe_base_path}_*") if revision_paths: revisions = (int(x.rsplit("_", 1)[-1]) for x in revision_paths) max_revision = max(revisions) recipe_path = f"{recipe_base_path}_{max_revision}" else: recipe_path = recipe_base_path return recipe_path def ensure_brew_on_path(args): brew_on_path = which("brew") if brew_on_path: brew_on_path = os.path.abspath(brew_on_path) def ensure_on_path(brew): if brew != brew_on_path: os.environ["PATH"] = f"{os.path.dirname(brew)}:{os.environ['PATH']}" default_brew_path = os.path.join(DEFAULT_HOMEBREW_ROOT, "bin", "brew") if args and args.brew: user_brew_path = os.path.abspath(args.brew) ensure_on_path(user_brew_path) elif brew_on_path: return brew_on_path elif os.path.exists(default_brew_path): ensure_on_path(default_brew_path) else: raise Exception(NO_BREW_ERROR_MESSAGE) def which(file): # http://stackoverflow.com/questions/5226958/which-equivalent-function-in-python for path in os.environ["PATH"].split(":"): if os.path.exists(f"{path}/{file}"): return f"{path}/{file}" return None if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/brew_util.py0000644000175100017510000000213715234436334024227 0ustar00runnerrunner"""brew_exts defines generic extensions to Homebrew this file builds on those abstraction and provides Galaxy specific functionality not useful to the brew external commands. """ from . import brew_exts DEFAULT_TAP = "homebrew/science" class HomebrewRecipe: def __init__(self, recipe, version, tap): self.recipe = recipe self.version = version self.tap = tap def requirements_to_recipes(requirements): return filter(None, map(requirement_to_recipe, requirements)) def requirement_to_recipe(requirement): if requirement.type != "package": return None # TOOD: Allow requirements to annotate optionalbrew specific # adaptions. recipe_name = requirement.name recipe_version = requirement.version return HomebrewRecipe(recipe_name, recipe_version, tap=DEFAULT_TAP) def requirements_to_recipe_contexts(requirements, brew_context): def to_recipe_context(homebrew_recipe): return brew_exts.RecipeContext(homebrew_recipe.recipe, homebrew_recipe.version, brew_context) return map(to_recipe_context, requirements_to_recipes(requirements)) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/commands.py0000644000175100017510000000061415234436334024032 0ustar00runnerrunnerimport warnings from galaxy.util.commands import ( # noqa: F401 argv_to_str, CommandLineException, download_command, execute, redirect_aware_commmunicate, redirecting_io, shell, shell_process, which, ) warnings.warn( "Importing galaxy.tool_util.deps.commands is deprecated, use galaxy.util.commands instead", DeprecationWarning, stacklevel=2, ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/conda_compat.py0000644000175100017510000000523715234436334024666 0ustar00runnerrunner"""Compat. layer with conda_build/verify if Galaxy/galaxy-lib not installed through conda. In general there are utilities available for Conda building and parsing that are high-quality and should be utilized when available but that are only available in conda channels and not in PyPI. This module serves as a PyPI capable interface to these utilities. """ import os import yaml try: from conda_build.metadata import MetaData except ImportError: MetaData = None try: from anaconda_verify.recipe import ( parse, render_jinja2, ) except ImportError: render_jinja2 = None parse = None class _Memoized: def __init__(self, func): self.func = func self.cache = {} def __call__(self, *args): if args in self.cache: return self.cache[args] else: value = self.func(*args) self.cache[args] = value return value def _parse(data, cfg): """Parse metadata YAML.""" assert cfg is None, "Conda utilities for evaluating cfg are not available." return dict(yamlize(data)) def _render_jinja2(recipe_dir): """Evaluate Conda recipe as a jinja template.""" try: import jinja2 except ImportError: raise Exception("Failed to import jinja2 for evaluating Conda recipe templates.") loaders = [jinja2.FileSystemLoader(recipe_dir)] env = jinja2.Environment(loader=jinja2.ChoiceLoader(loaders)) template = env.get_or_select_template("meta.yaml") return template.render(environment=env) @_Memoized def yamlize(data): res = yaml.safe_load(data) # ensure the result is a dict if res is None: res = {} return res if render_jinja2 is None: render_jinja2 = _render_jinja2 if parse is None: parse = _parse def raw_metadata(recipe_dir): """Evaluate Conda template if needed and return raw metadata for supplied recipe directory.""" meta_path = os.path.join(recipe_dir, "meta.yaml") with open(meta_path, "rb") as fi: data = fi.read() if b"{{" in data: data = render_jinja2(recipe_dir) meta = parse(data, None) return meta class _MetaData: def __init__(self, input_dir): self.meta = raw_metadata(input_dir) def get_value(self, field, default=None): """Get nested field value or supplied default is not present.""" section, key = field.split("/") submeta = self.meta.get(section) if submeta is None: submeta = {} res = submeta.get(key) if res is None: res = default return res if MetaData is None: MetaData = _MetaData __all__ = ( "MetaData", "raw_metadata", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/conda_util.py0000644000175100017510000006611315234436334024360 0ustar00runnerrunnerimport functools import hashlib import json import logging import os import platform import re import shutil import sys import tempfile from typing import ( Any, Callable, Dict, Iterable, Iterator, List, Optional, Tuple, TYPE_CHECKING, Union, ) from packaging.version import Version from galaxy.tool_util.version import parse_version from galaxy.util import ( commands, download_to_file, listify, shlex_join, smart_str, which, ) from . import installable if TYPE_CHECKING: from galaxy.tool_util.deps.requirements import ToolRequirement log = logging.getLogger(__name__) # Not sure there are security concerns, lets just fail fast if we are going # break shell commands we are building. SHELL_UNSAFE_PATTERN = re.compile(r"[\s\"']") IS_OS_X = sys.platform == "darwin" VERSIONED_ENV_DIR_NAME = re.compile(r"__(.*)@(.*)") UNVERSIONED_ENV_DIR_NAME = re.compile(r"__(.*)@_uv_") USE_PATH_EXEC_DEFAULT = False CONDA_PACKAGE_SPECS = ("conda>=23.7.0", "conda-libmamba-solver", "pyopenssl>=22.1.0") CONDA_BUILD_SPECS = ("conda-build>=3.22.0",) USE_LOCAL_DEFAULT = False def conda_link() -> str: arch = platform.machine() if IS_OS_X: if "arm64" in arch: url = "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh" else: url = "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-x86_64.sh" else: if "arm64" in arch or "aarch64" in arch: url = "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh" else: url = "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh" return url def find_conda_prefix() -> str: """If supplied conda_prefix is not set, default to the default location for Miniconda installs. """ home = os.path.expanduser("~") destinations = ["miniforge3", "miniconda3", "miniconda2", "anaconda3", "anaconda2"] for destination in destinations: destination = os.path.join(home, destination) if os.path.exists(destination): return destination return os.path.join(home, "miniforge3") class CondaContext(installable.InstallableContext): installable_description = "Conda" _conda_build_available: Optional[bool] _conda_version: Optional[Version] _libmamba_solver_available: Optional[bool] def __init__( self, conda_prefix: Optional[str] = None, conda_exec: Optional[Union[str, List[str]]] = None, shell_exec: Optional[Callable[..., int]] = None, debug: bool = False, ensure_channels: Union[str, List[str]] = "", condarc_override: Optional[str] = None, use_path_exec: bool = USE_PATH_EXEC_DEFAULT, copy_dependencies: bool = False, use_local: bool = USE_LOCAL_DEFAULT, ) -> None: self.condarc_override = condarc_override if not conda_exec and use_path_exec: conda_exec = which("conda") if conda_exec and isinstance(conda_exec, str): conda_exec = os.path.normpath(conda_exec) self.debug = debug self.shell_exec = shell_exec or commands.shell self.copy_dependencies = copy_dependencies if conda_exec is not None: self.conda_exec = conda_exec if conda_prefix is None: info = self.conda_info() conda_prefix = info.get("default_prefix") if conda_prefix is None: conda_prefix = find_conda_prefix() self.conda_prefix = conda_prefix if conda_exec is None: self.conda_exec = self._bin("conda") self.ensure_channels: List[str] = listify(ensure_channels) self.use_local = use_local self._reset_conda_properties() def _reset_conda_properties(self) -> None: self._conda_version = None self._conda_build_available = None self._libmamba_solver_available = None @property def conda_version(self) -> Version: if self._conda_version is None: self._guess_conda_properties() assert isinstance(self._conda_version, Version) return self._conda_version @property def conda_build_available(self) -> bool: if self._conda_build_available is None: self._guess_conda_properties() assert isinstance(self._conda_build_available, bool) return self._conda_build_available def _guess_conda_properties(self) -> None: info = self.conda_info() self._conda_version = Version(info["conda_version"]) self._conda_build_available = False conda_build_version = info.get("conda_build_version") if conda_build_version and conda_build_version != "not installed": try: Version(conda_build_version) self._conda_build_available = True except Exception: pass @property def _override_channels_args(self) -> List[str]: override_channels_args = [] if self.ensure_channels: override_channels_args.append("--override-channels") for channel in self.ensure_channels: override_channels_args.extend(["--channel", channel]) return override_channels_args @property def _solver_args(self) -> List[str]: if self._libmamba_solver_available is None: self._libmamba_solver_available = self.conda_version >= Version("4.12.0") and self.is_package_installed( "conda-libmamba-solver" ) if self._libmamba_solver_available: # The "--solver" option was introduced in conda 22.11.0, when the # "--experimental-solver" option was deprecated. # The "--experimental-solver" option was removed in conda 23.9.0 . solver_option = "--solver" if self.conda_version >= Version("22.11.0") else "--experimental-solver" return [solver_option, "libmamba"] else: return [] def ensure_conda_build_installed_if_needed(self) -> int: if self.use_local and not self.conda_build_available: # Cannot use --use-local during installation of conda-build. return self.exec_install(CONDA_BUILD_SPECS, allow_local=False) else: return 0 def conda_info(self) -> Dict[str, Any]: cmd = listify(self.conda_exec) + ["info", "--json"] info_out = commands.execute(cmd) info = json.loads(info_out) return info def is_conda_installed(self) -> bool: """ Check if conda_info() works """ try: self.conda_info() return True except Exception: return False def can_install_conda(self) -> bool: """ If conda_exec is set to a path outside of conda_prefix, there is no use installing conda into conda_prefix, since it can't be used by galaxy. If conda_exec equals conda_prefix/bin/conda, we can install conda if either conda_prefix does not exist or is empty. """ assert isinstance(self.conda_exec, str), "conda_exec is not a str" conda_exec = os.path.abspath(self.conda_exec) conda_prefix_plus_exec = os.path.abspath(os.path.join(self.conda_prefix, "bin/conda")) if conda_exec == conda_prefix_plus_exec: if not os.path.exists(self.conda_prefix): return True elif os.listdir(self.conda_prefix) == []: os.rmdir(self.conda_prefix) # Conda's install script fails if path exists (even if empty). return True else: log.warning( "Cannot install Conda because conda_prefix '%s' exists and is not empty.", self.conda_prefix ) return False else: log.warning( "Skipping installation of Conda into conda_prefix '%s', " "since conda_exec '%s' is set to a path outside of conda_prefix.", self.conda_prefix, self.conda_exec, ) return False def exec_command(self, operation: str, args: List[str], stdout_path: Optional[str] = None) -> int: """ Execute the requested command. Return the process exit code (i.e. 0 in case of success). """ cmd = listify(self.conda_exec) + operation.split() if self.debug: cmd.append("--debug") cmd.extend(args) env = {} if self.condarc_override: env["CONDARC"] = self.condarc_override cmd_string = shlex_join(cmd) kwds: Dict[str, Any] = {} conda_exec_home: Optional[str] = None try: if stdout_path: kwds["stdout"] = open(stdout_path, "w") cmd_string += f" > '{stdout_path}'" conda_exec_home = env["HOME"] = tempfile.mkdtemp( prefix="conda_exec_home_" ) # We don't want to pollute ~/.conda, which may not even be writable log.debug("Executing command: %s", cmd_string) return self.shell_exec(cmd, env=env, **kwds) except Exception: log.exception("Failed to execute command: %s", cmd_string) return 1 finally: if kwds.get("stdout"): kwds["stdout"].close() if conda_exec_home: shutil.rmtree(conda_exec_home, ignore_errors=True) def is_package_installed(self, pkg_name: str, version: Optional[str] = None) -> bool: list_args = ["-f", "--json", pkg_name] with tempfile.NamedTemporaryFile("r") as temp: ret = self.exec_command("list", list_args, stdout_path=temp.name) if ret != 0: log.error("Failed to execute 'conda list'") return False out = json.load(temp) if not out: return False if not version: return True return any(match["version"] == version for match in out) def exec_create(self, args: Iterable[str], allow_local: bool = True, stdout_path: Optional[str] = None) -> int: """ Return the process exit code (i.e. 0 in case of success). """ for try_strict in [True, False]: create_args = ["-y", "--quiet"] if try_strict: if self.conda_version >= Version("4.7.5"): create_args.append("--strict-channel-priority") else: continue if allow_local and self.use_local: create_args.append("--use-local") create_args.extend(self._solver_args) create_args.extend(self._override_channels_args) create_args.extend(args) ret = self.exec_command("create", create_args, stdout_path=stdout_path) if ret == 0: break return ret def exec_remove(self, args: List[str]) -> int: """ Remove a conda environment using conda env remove -y --name `args`. Return the process exit code (i.e. 0 in case of success). """ remove_args = ["-y", "--name"] remove_args.extend(args) return self.exec_command("env remove", remove_args) def exec_install(self, args: Iterable[str], allow_local: bool = True, stdout_path: Optional[str] = None) -> int: """ Return the process exit code (i.e. 0 in case of success). """ for try_strict in [True, False]: install_args = ["-y"] if try_strict: if self.conda_version >= Version("4.7.5"): install_args.append("--strict-channel-priority") else: continue if allow_local and self.use_local: install_args.append("--use-local") install_args.extend(self._solver_args) install_args.extend(self._override_channels_args) install_args.extend(args) ret = self.exec_command("install", install_args, stdout_path=stdout_path) if ret == 0: break if ret == 0: self._reset_conda_properties() return ret def exec_clean(self, args: Optional[List[str]] = None, quiet: bool = False) -> int: """ Clean up after conda installation. Return the process exit code (i.e. 0 in case of success). """ clean_args = ["--tarballs", "-y"] if args: clean_args.extend(args) stdout_path = None if quiet: stdout_path = "/dev/null" return self.exec_command("clean", clean_args, stdout_path=stdout_path) def exec_search( self, args: List[str], json: bool = False, offline: bool = False, platform: Optional[str] = None ) -> str: """ Search conda channels for a package Return the standard output of the conda process. """ cmd = listify(self.conda_exec)[:] cmd.append("search") cmd.extend(self._override_channels_args) if json: cmd.append("--json") if offline: cmd.append("--offline") if platform: cmd.extend(["--platform", platform]) cmd.extend(args) return commands.execute(cmd) def export_list(self, name: str, path: str) -> int: """ Return the process exit code (i.e. 0 in case of success). """ return self.exec_command("list", ["--name", name, "--export"], stdout_path=path) def env_path(self, env_name: str) -> str: return os.path.join(self.envs_path, env_name) @property def envs_path(self) -> str: return os.path.join(self.conda_prefix, "envs") def has_env(self, env_name: str) -> bool: env_path = self.env_path(env_name) return os.path.isdir(env_path) def get_conda_target_installed_path(self, conda_target: "CondaTarget") -> Optional[str]: for env_name in (conda_target.install_environment, conda_target.capitalized_install_environment): if self.has_env(env_name): return self.env_path(env_name) return None @property def deactivate(self) -> str: return self._bin("deactivate") @property def activate(self) -> str: return self._bin("activate") def is_installed(self) -> bool: return self.is_conda_installed() def can_install(self) -> bool: return self.can_install_conda() @property def parent_path(self) -> str: return os.path.dirname(os.path.abspath(self.conda_prefix)) def _bin(self, name: str) -> str: return os.path.join(self.conda_prefix, "bin", name) def installed_conda_targets(conda_context: CondaContext) -> Iterator["CondaTarget"]: envs_path = conda_context.envs_path dir_contents = os.listdir(envs_path) if os.path.exists(envs_path) else [] for name in dir_contents: versioned_match = VERSIONED_ENV_DIR_NAME.match(name) if versioned_match: yield CondaTarget(versioned_match.group(1), version=versioned_match.group(2)) unversioned_match = UNVERSIONED_ENV_DIR_NAME.match(name) if unversioned_match: yield CondaTarget(unversioned_match.group(1)) class CondaTarget: def __init__( self, package: str, version: Optional[str] = None, build: Optional[str] = None, channel: Optional[str] = None ) -> None: if SHELL_UNSAFE_PATTERN.search(package) is not None or not package: raise ValueError(f"Invalid package [{package}] encountered.") self.capitalized_package = package self.package = package.lower() if version and SHELL_UNSAFE_PATTERN.search(version) is not None: raise ValueError(f"Invalid version [{version}] encountered.") self.version = version if build is not None and SHELL_UNSAFE_PATTERN.search(build) is not None: raise ValueError(f"Invalid build [{build}] encountered.") self.build = build if channel and SHELL_UNSAFE_PATTERN.search(channel) is not None: raise ValueError(f"Invalid version [{channel}] encountered.") self.channel = channel def __str__(self) -> str: attributes = f"package={self.package}" if self.version is not None: attributes += f",version={self.version}" if self.build is not None: attributes += f",build={self.build}" if self.channel: attributes += f",channel={self.channel}" return f"CondaTarget[{attributes}]" __repr__ = __str__ @property def package_specifier(self) -> str: """Return a package specifier as consumed by conda install/create.""" if self.version: spec = f"{self.package}={self.version}" else: spec = f"{self.package}=*" if self.build: spec += f"={self.build}" return spec @property def install_environment(self) -> str: """The dependency resolution and installation frameworks will expect each target to be installed it its own environment with a fixed and predictable name given package and version. Since Galaxy 23.1 the package name is lowercased as all Conda package names must be lowercase. """ if self.version: return f"__{self.package}@{self.version}" else: return f"__{self.package}@_uv_" @property def capitalized_install_environment(self) -> str: """Same as install_environment() but using the original capitalized package name for backward compatibility with environments created before Galaxy 23.1 . """ if self.version: return f"__{self.capitalized_package}@{self.version}" else: return f"__{self.capitalized_package}@_uv_" def __hash__(self) -> int: return hash((self.package, self.version, self.build, self.channel)) def __eq__(self, other: Any) -> bool: if isinstance(other, self.__class__): return (self.package, self.version, self.build, self.channel) == ( other.package, other.version, other.build, other.channel, ) return False def hash_conda_packages(conda_packages: Iterable[CondaTarget], capitalized_package_names: bool = False) -> str: """Produce a unique hash on supplied packages. TODO: Ideally we would do this in such a way that preserved environments. """ h = hashlib.new("sha256") for conda_package in conda_packages: h.update( smart_str( conda_package.capitalized_install_environment if capitalized_package_names else conda_package.install_environment ) ) return h.hexdigest() # shell makes sense for planemo, in Galaxy this should just execute # these commands as Python def install_conda(conda_context: CondaContext, force_conda_build: bool = False) -> int: with tempfile.NamedTemporaryFile(suffix=".sh", prefix="conda_install", delete=False) as temp: script_path = temp.name install_cmd = ["bash", script_path, "-b", "-p", conda_context.conda_prefix] package_targets = list(CONDA_PACKAGE_SPECS) if force_conda_build or conda_context.use_local: package_targets.extend(CONDA_BUILD_SPECS) log.info("Installing conda, this may take several minutes.") try: download_to_file(conda_link(), script_path) exit_code = conda_context.shell_exec(install_cmd) except Exception: log.exception("Failed to install conda") return 1 finally: if os.path.exists(script_path): os.remove(script_path) if exit_code: return exit_code return conda_context.exec_install(package_targets, allow_local=False) def install_conda_targets( conda_targets: Iterable[CondaTarget], conda_context: CondaContext, env_name: Optional[str] = None, allow_local: bool = True, ) -> int: """ Return the process exit code (i.e. 0 in case of success). """ if env_name is not None: create_args = [ "--name", env_name, # environment for package ] for conda_target in conda_targets: create_args.append(conda_target.package_specifier) return conda_context.exec_create(create_args, allow_local=allow_local) else: return conda_context.exec_install([t.package_specifier for t in conda_targets], allow_local=allow_local) def install_conda_target(conda_target: CondaTarget, conda_context: CondaContext, skip_environment: bool = False) -> int: """ Install specified target into a its own environment. Return the process exit code (i.e. 0 in case of success). """ if not skip_environment: create_args = [ "--name", conda_target.install_environment, # environment for package conda_target.package_specifier, ] return conda_context.exec_create(create_args) else: return conda_context.exec_install([conda_target.package_specifier]) def cleanup_failed_install_of_environment(env: str, conda_context: CondaContext) -> int: if conda_context.has_env(env): return conda_context.exec_remove([env]) return 0 def cleanup_failed_install(conda_target: CondaTarget, conda_context: CondaContext) -> int: return cleanup_failed_install_of_environment(conda_target.install_environment, conda_context=conda_context) def best_search_result( conda_target: CondaTarget, conda_context: CondaContext, offline: bool = False, platform: Optional[str] = None ) -> Union[Tuple[None, None], Tuple[Dict[str, Any], bool]]: """Find best "conda search" result for specified target. Return (``None``, ``None``) if no results match. """ # Cannot specify the version here (i.e. conda_target.package_specifier) # because if the version is not found, the exec_search() call would fail. search_args = [conda_target.package] try: res = conda_context.exec_search(search_args, json=True, offline=offline, platform=platform) # Use python's stable list sorting to sort by date, # then build_number, then version. The top of the list # then is the newest version with the newest build and # the latest update time. hits = json.loads(res).get(conda_target.package, [])[::-1] hits = sorted(hits, key=lambda hit: hit["build_number"], reverse=True) hits = sorted(hits, key=lambda hit: parse_version(hit["version"]), reverse=True) except commands.CommandLineException as e: log.error(f"Could not execute: '{e.command}'\n{e}") hits = [] if len(hits) == 0: return (None, None) best_result = (hits[0], False) for hit in hits: if is_search_hit_exact(conda_target, hit): best_result = (hit, True) break return best_result def is_search_hit_exact(conda_target: CondaTarget, search_hit: Dict[str, Any]) -> bool: # It'd be nice to make request verson of 1.0 match available # version of 1.0.3 or something like that. target_version = conda_target.version if target_version and search_hit["version"] != target_version: return False target_build = conda_target.build if target_build and search_hit["build"] != target_build: return False return True def is_conda_target_installed(conda_target: CondaTarget, conda_context: CondaContext) -> bool: return conda_context.get_conda_target_installed_path(conda_target) is not None def filter_installed_targets(conda_targets: Iterable[CondaTarget], conda_context: CondaContext) -> List[CondaTarget]: installed = functools.partial(is_conda_target_installed, conda_context=conda_context) return list(filter(installed, conda_targets)) def build_isolated_environment( conda_packages: Union[CondaTarget, List[CondaTarget]], conda_context: CondaContext, path: Optional[str] = None, copy: bool = False, quiet: bool = False, ) -> Tuple[str, int]: """Build a new environment (or reuse an existing one from hashes) for specified conda packages. """ if not isinstance(conda_packages, list): conda_packages = [conda_packages] # Lots we could do in here, hashing, checking revisions, etc... tempdir = None try: hash = hash_conda_packages(conda_packages) tempdir = tempfile.mkdtemp(prefix="jobdeps", suffix=hash) tempdir_name = os.path.basename(tempdir) export_paths = [] for conda_package in conda_packages: name = conda_package.install_environment export_path = os.path.join(tempdir, name) conda_context.export_list(name, export_path) export_paths.append(export_path) create_args = ["--unknown"] # Works in 3.19, 4.0 - 4.2 - not in 4.3. # Adjust fix if they fix Conda - xref # - https://github.com/galaxyproject/galaxy/issues/3635 # - https://github.com/conda/conda/issues/2035 offline_works = (conda_context.conda_version < Version("4.3")) or ( conda_context.conda_version >= Version("4.4") ) if offline_works: create_args.append("--offline") else: create_args.append("--use-index-cache") if path is None: create_args.extend(["--name", tempdir_name]) else: create_args.extend(["--prefix", path]) if copy: create_args.append("--copy") for export_path in export_paths: create_args.extend(["--file", export_path]) stdout_path = None if quiet: stdout_path = "/dev/null" if path is not None and os.path.exists(path): exit_code = conda_context.exec_install(create_args, stdout_path=stdout_path) else: exit_code = conda_context.exec_create(create_args, stdout_path=stdout_path) return (path or tempdir_name, exit_code) finally: conda_context.exec_clean(quiet=quiet) if tempdir is not None: shutil.rmtree(tempdir) def split_version_build(version: Optional[str]) -> Tuple[Optional[str], Optional[str]]: """Split version string into version and build. Handles '=' separator (conda format) and '--' separator (mulled format). """ if not version: return version, None build = None if "=" in version: version, build = version.split("=") elif "--" in version: version, build = version.split("--") return version, build def requirement_to_conda_targets(requirement: "ToolRequirement") -> Optional[CondaTarget]: conda_target = None if requirement.type == "package": assert requirement.name version, build = split_version_build(requirement.version) conda_target = CondaTarget(requirement.name, version=version, build=build) return conda_target def requirements_to_conda_targets(requirements: Iterable["ToolRequirement"]) -> List[CondaTarget]: conda_targets = (requirement_to_conda_targets(_) for _ in requirements) return [c for c in conda_targets if c is not None] __all__ = ( "CondaContext", "CondaTarget", "install_conda", "install_conda_target", "requirements_to_conda_targets", "split_version_build", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/container_classes.py0000644000175100017510000005760315234436334025742 0ustar00runnerrunnerimport os import string from abc import ( ABCMeta, abstractmethod, ) from logging import getLogger from typing import ( Any, Dict, List, Optional, Tuple, Type, TYPE_CHECKING, ) from uuid import uuid4 from packaging.version import Version from typing_extensions import Protocol from galaxy.util import ( asbool, in_directory, ) from . import ( docker_util, singularity_util, ) from .container_volumes import DockerVolume from .requirements import ( DEFAULT_CONTAINER_RESOLVE_DEPENDENCIES, DEFAULT_CONTAINER_SHELL, ) if TYPE_CHECKING: from .dependencies import ( AppInfo, JobInfo, ToolInfo, ) from .requirements import ContainerDescription log = getLogger(__name__) DOCKER_CONTAINER_TYPE = "docker" SINGULARITY_CONTAINER_TYPE = "singularity" TRAP_KILL_CONTAINER = "trap _on_exit EXIT" LOAD_CACHED_IMAGE_COMMAND_TEMPLATE = r""" python << EOF from __future__ import print_function import json import subprocess import tarfile t = tarfile.TarFile("${cached_image_file}") meta_str = t.extractfile('repositories').read() meta = json.loads(meta_str) repository, tag_value = next(iter(meta.items())) tag, id = next(iter(tag_value.items())) cmd = "${images_cmd}" proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, text=True) stdo, stde = proc.communicate() found = False for line in stdo.split("\n"): line = line.strip() if line: try: img = json.loads(line) if img.get("Repository") == repository and img.get("Tag") == tag and img.get("ID") == id: found = True break except (json.JSONDecodeError, ValueError): pass if not found: print("Loading image") cmd = "cat ${cached_image_file} | ${load_cmd}" subprocess.check_call(cmd, shell=True) EOF """ SOURCE_CONDA_ACTIVATE = """ # Check if container was created by installing conda packages, # and if so, source scripts to populate environment variables # that would be set by activating the conda environment. if [ -d /usr/local/etc/conda/activate.d ]; then export CONDA_PREFIX=/usr/local for f in /usr/local/etc/conda/activate.d/*.sh; do case "$f" in "/usr/local/etc/conda/activate.d/activate-"*) :;; *) . "$f" ;; esac; done fi """ class ContainerProtocol(Protocol): """ Helper class to allow typing for the HasDockerLikeVolumes mixin """ @property def app_info(self) -> "AppInfo": ... @property def tool_info(self) -> "ToolInfo": ... @property def job_info(self) -> Optional["JobInfo"]: ... class Container(metaclass=ABCMeta): """ TODO The container resolvers currently initialize job_info as None, ContainerFinder.find_container fixes this by constructing a new container with job_info (see __destination_container) """ container_type: str def __init__( self, container_id: str, app_info: "AppInfo", tool_info: "ToolInfo", destination_info: Dict[str, Any], job_info: Optional["JobInfo"], container_description: Optional["ContainerDescription"], container_name: Optional[str] = None, ) -> None: self.container_id = container_id self.app_info = app_info self.tool_info = tool_info self.destination_info = destination_info self.job_info = job_info self.container_description = container_description self.container_name = container_name or uuid4().hex self.container_info: Dict[str, Any] = {} def prop(self, name: str, default: Any) -> Any: destination_name = f"{self.container_type}_{name}" return self.destination_info.get(destination_name, default) @property def resolve_dependencies(self) -> bool: return ( DEFAULT_CONTAINER_RESOLVE_DEPENDENCIES if not self.container_description else self.container_description.resolve_dependencies ) @property def shell(self) -> str: return DEFAULT_CONTAINER_SHELL if not self.container_description else self.container_description.shell @property def source_environment(self) -> str: if self.container_description and not self.container_description.explicit: return SOURCE_CONDA_ACTIVATE return "" @abstractmethod def containerize_command(self, command: str) -> str: """ Use destination supplied container configuration parameters, container_id, and command to build a new command that runs input command in container. """ class Volume: """ helper class to manage a container volume string """ def __init__(self, rawstr: str, container_type: str): self.source, self.target, self.mode = Volume.parse_volume_str(rawstr) self.container_type = container_type @staticmethod def parse_volume_str(rawstr: str) -> Tuple[str, str, str]: """ >>> Volume.parse_volume_str('A:B:rw') ('A', 'B', 'rw') >>> Volume.parse_volume_str('A : B: ro') ('A', 'B', 'ro') >>> Volume.parse_volume_str('A:B') ('A', 'B', 'rw') >>> Volume.parse_volume_str('A:ro') ('A', 'A', 'ro') >>> Volume.parse_volume_str('A:z') ('A', 'A', 'z') >>> Volume.parse_volume_str('A:Z') ('A', 'A', 'Z') >>> Volume.parse_volume_str('A') ('A', 'A', 'rw') >>> Volume.parse_volume_str(' ') Traceback (most recent call last): ... Exception: Unparsable volumes string in configuration [ ] >>> Volume.parse_volume_str('A:B:C:D') Traceback (most recent call last): ... Exception: Unparsable volumes string in configuration [A:B:C:D] """ if rawstr.strip() == "": raise Exception(f"Unparsable volumes string in configuration [{rawstr}]") volume_parts = rawstr.split(":") if len(volume_parts) > 3: raise Exception(f"Unparsable volumes string in configuration [{rawstr}]") if len(volume_parts) == 3: source = volume_parts[0] target = volume_parts[1] mode = volume_parts[2] elif len(volume_parts) == 2: # not really parsing/checking mode here, just figuring out if the 2nd component is target or mode mode_parts = volume_parts[1].split(",") if any(mode_part not in ("rw", "ro", "default_ro", "z", "Z") for mode_part in mode_parts): source = volume_parts[0] target = volume_parts[1] mode = "rw" else: source = volume_parts[0] target = volume_parts[0] mode = volume_parts[1] elif len(volume_parts) == 1: source = volume_parts[0] target = volume_parts[0] mode = "rw" source = source.strip() target = target.strip() mode = mode.strip() return source, target, mode def __str__(self): """ >>> str(Volume('A:A:rw', 'docker')) 'A:rw' >>> str(Volume('A:B:rw', 'docker')) 'A:B:rw' >>> str(Volume('A:A:ro', 'singularity')) 'A:ro' >>> str(Volume('A:B:ro', 'singularity')) 'A:B:ro' >>> str(Volume('A:A:rw', 'singularity')) 'A' >>> str(Volume('A:B:rw', 'singularity')) 'A:B' """ if self.source == self.target: path = self.source else: path = f"{self.source}:{self.target}" # TODO remove this, we require quite recent singularity anyway # for a while singularity did not allow to specify the bind type rw # (which is the default). so we omit this default # see https://github.com/hpcng/singularity/pull/5487 if self.container_type == SINGULARITY_CONTAINER_TYPE and self.mode == "rw": return path else: return f"{path}:{self.mode}" def preprocess_volumes(volumes_raw_str: str, container_type: str) -> List[str]: """Process Galaxy volume specification string to either Docker or Singularity specification. Galaxy allows the mount try "default_ro" which translates to ro for Docker and ro for Singularity iff no subdirectories are rw (Singularity does not allow ro parent directories with rw subdirectories). Removes volumes that have the same target directory which is not allowed (for docker and singularity). Volumes that are specified later in the volumes_raw_str are favoured which allows admins to overwrite defaults. >>> preprocess_volumes("", DOCKER_CONTAINER_TYPE) [] >>> preprocess_volumes("/a/b", DOCKER_CONTAINER_TYPE) ['/a/b:rw'] >>> preprocess_volumes("/a/b:ro,/a/b/c:rw", DOCKER_CONTAINER_TYPE) ['/a/b:ro', '/a/b/c:rw'] >>> preprocess_volumes("/a/b:/a:ro,/a/b/c:/a/b:rw", DOCKER_CONTAINER_TYPE) ['/a/b:/a:ro', '/a/b/c:/a/b:rw'] >>> preprocess_volumes("/a/b:default_ro,/a/b/c:rw", DOCKER_CONTAINER_TYPE) ['/a/b:ro', '/a/b/c:rw'] >>> preprocess_volumes("/a/b:default_ro,/a/b/c:ro", SINGULARITY_CONTAINER_TYPE) ['/a/b:ro', '/a/b/c:ro'] >>> preprocess_volumes("/a/b:default_ro,/a/b/c:rw", SINGULARITY_CONTAINER_TYPE) ['/a/b', '/a/b/c'] >>> preprocess_volumes("/x:/a/b:default_ro,/y:/a/b/c:ro", SINGULARITY_CONTAINER_TYPE) ['/x:/a/b:ro', '/y:/a/b/c:ro'] >>> preprocess_volumes("/x:/a/b:default_ro,/y:/a/b/c:rw", SINGULARITY_CONTAINER_TYPE) ['/x:/a/b', '/y:/a/b/c'] >>> preprocess_volumes("/x:/x,/y:/x", SINGULARITY_CONTAINER_TYPE) ['/y:/x'] """ if not volumes_raw_str: return [] # filter out empty strings, this happens for tools without tool directories. volumes = [Volume(v, container_type) for v in volumes_raw_str.split(",") if v] rw_paths = [v.target for v in volumes if v.mode == "rw"] for volume in volumes: mode = volume.mode if volume.mode == "default_ro": mode = "ro" if container_type == SINGULARITY_CONTAINER_TYPE: for rw_path in rw_paths: if in_directory(rw_path, volume.target): mode = "rw" volume.mode = mode # remove duplicate targets target_to_volume = {v.target: str(v) for v in volumes} return list(target_to_volume.values()) class HasDockerLikeVolumes: """Mixin to share functionality related to Docker volume handling. Singularity seems to have a fairly compatible syntax for volume handling. """ def _expand_volume_str(self: ContainerProtocol, value: str) -> str: if not value: return value template = string.Template(value) variables = {} def add_var(name, value): if value: if not value.startswith("$"): value = os.path.abspath(value) variables[name] = value assert self.job_info is not None add_var("working_directory", self.job_info.working_directory) add_var("tmp_directory", self.job_info.tmp_directory) add_var("job_directory", self.job_info.job_directory) add_var("tool_directory", self.job_info.tool_directory) add_var("home_directory", self.job_info.home_directory) add_var("galaxy_root", self.app_info.galaxy_root_dir) add_var("default_file_path", self.app_info.default_file_path) add_var("library_import_dir", self.app_info.library_import_dir) add_var("tool_data_path", self.app_info.tool_data_path) add_var("galaxy_data_manager_data_path", self.app_info.galaxy_data_manager_data_path) add_var("shed_tool_data_path", self.app_info.shed_tool_data_path) if self.job_info.job_directory and self.job_info.job_directory_type == "pulsar": # We have a Pulsar job directory, so everything needed (excluding index # files) should be available in job_directory... defaults = "$job_directory:default_ro" if self.job_info.tool_directory: defaults += ",$tool_directory:default_ro" defaults += ",$job_directory/outputs:rw,$working_directory:rw" else: if self.job_info.tmp_directory is not None: defaults = "$tmp_directory:rw" # If a tool definitely has a temp directory available set it to /tmp in container for compat. # with CWL. This is part of that spec and should make it easier to share containers between CWL # and Galaxy. defaults += ",$tmp_directory:/tmp:rw" else: defaults = "$_GALAXY_JOB_TMP_DIR:rw,$TMPDIR:rw,$TMP:rw,$TEMP:rw" defaults += ",$galaxy_root:default_ro" if self.job_info.tool_directory: defaults += ",$tool_directory:default_ro" if self.job_info.job_directory: defaults += ",$job_directory:default_ro,$job_directory/outputs:rw" if Version(str(self.tool_info.profile)) <= Version("19.09"): defaults += ",$job_directory/configs:rw" if self.job_info.home_directory is not None: defaults += ",$home_directory:rw" if self.app_info.outputs_to_working_directory: # Should need default_file_path (which is of course an estimate given # object stores anyway). defaults += ",$working_directory:rw,$default_file_path:default_ro" else: defaults += ",$working_directory:rw,$default_file_path:rw" if self.app_info.library_import_dir: defaults += ",$library_import_dir:default_ro" if self.app_info.tool_data_path: defaults += ",$tool_data_path:default_ro" if self.app_info.galaxy_data_manager_data_path: defaults += ",$galaxy_data_manager_data_path:default_ro" if self.app_info.shed_tool_data_path: defaults += ",$shed_tool_data_path:default_ro" # Define $defaults that can easily be extended with external library and # index data without deployer worrying about above details. variables["defaults"] = string.Template(defaults).safe_substitute(variables) volumes_str = template.safe_substitute(variables) # Not all tools have a tool_directory - strip this out if supplied by # job_conf. tool_directory_index = volumes_str.find("$tool_directory") if tool_directory_index > 0: end_index = volumes_str.find(",", tool_directory_index) if end_index < 0: end_index = len(volumes_str) volumes_str = volumes_str[0:tool_directory_index] + volumes_str[end_index : len(volumes_str)] return volumes_str def _parse_volumes(volumes_raw: str, container_type: str) -> List[DockerVolume]: """ >>> volumes_raw = "$galaxy_root:ro,$tool_directory:ro,$job_directory:ro,$working_directory:z,$default_file_path:z" >>> volumes = _parse_volumes(volumes_raw, "docker") >>> [str(v) for v in volumes] ['"$galaxy_root:$galaxy_root:ro"', '"$tool_directory:$tool_directory:ro"', '"$job_directory:$job_directory:ro"', '"$working_directory:$working_directory:z"', '"$default_file_path:$default_file_path:z"'] """ preprocessed_volumes_list = preprocess_volumes(volumes_raw, container_type) # TODO: Remove redundant volumes... return [DockerVolume.from_str(v) for v in preprocessed_volumes_list] class DockerContainer(Container, HasDockerLikeVolumes): container_type = DOCKER_CONTAINER_TYPE @property def docker_host_props(self) -> Dict[str, Any]: docker_host_props = dict( docker_cmd=self.prop("cmd", docker_util.DEFAULT_DOCKER_COMMAND), sudo=asbool(self.prop("sudo", docker_util.DEFAULT_SUDO)), sudo_cmd=self.prop("sudo_cmd", docker_util.DEFAULT_SUDO_COMMAND), host=self.prop("host", docker_util.DEFAULT_HOST), ) return docker_host_props @property def connection_configuration(self) -> Dict[str, Any]: return self.docker_host_props def build_pull_command(self) -> List[str]: return docker_util.build_pull_command(self.container_id, **self.docker_host_props) def containerize_command(self, command: str) -> str: env_directives = [] for pass_through_var in self.tool_info.env_pass_through: env_directives.append(f'"{pass_through_var}=${pass_through_var}"') # Allow destinations to explicitly set environment variables just for # docker container. Better approach is to set for destination and then # pass through only what tool needs however. (See todo in ToolInfo.) for key, value in self.destination_info.items(): if key.startswith("docker_env_"): env = key[len("docker_env_") :] env_directives.append(f'"{env}={value}"') assert self.job_info is not None working_directory = self.job_info.working_directory if not working_directory: raise Exception(f"Cannot containerize command [{working_directory}] without defined working directory.") volumes_raw = self._expand_volume_str(self.destination_info.get("docker_volumes", "$defaults")) volumes = _parse_volumes(volumes_raw, self.container_type) volumes_from = self.destination_info.get("docker_volumes_from", docker_util.DEFAULT_VOLUMES_FROM) docker_host_props = self.docker_host_props cached_image_file = self.__get_cached_image_file() if not cached_image_file: # TODO: Add option to cache it once here and create cached_image_file. cache_command = docker_util.build_docker_cache_command(self.container_id, **docker_host_props) else: cache_command = self.__cache_from_file_command(cached_image_file, docker_host_props) run_command = docker_util.build_docker_run_command( command, self.container_id, volumes=volumes, volumes_from=volumes_from, env_directives=env_directives, working_directory=working_directory, net=self.prop("net", None), # By default, docker instance has networking disabled auto_rm=asbool(self.prop("auto_rm", docker_util.DEFAULT_AUTO_REMOVE)), set_user=self.prop("set_user", docker_util.DEFAULT_SET_USER), run_extra_arguments=self.prop("run_extra_arguments", docker_util.DEFAULT_RUN_EXTRA_ARGUMENTS), guest_ports=self.tool_info.guest_ports, host_port_cmd=self.prop("host_port_cmd", None), container_name=self.container_name, **docker_host_props, ) kill_command = docker_util.build_docker_simple_command( "kill", container_name=self.container_name, **docker_host_props ) # Suppress standard error below in the kill command because it can cause jobs that otherwise would work # to fail. Likely, in these cases the container has been stopped normally and so cannot be stopped again. # A less hacky approach might be to check if the container is running first before trying to kill. # https://stackoverflow.com/questions/34228864/stop-and-delete-docker-container-if-its-running # Standard error is: # Error response from daemon: Cannot kill container: 2b0b961527574ebc873256b481bbe72e: No such container: 2b0b961527574ebc873256b481bbe72e return f""" _on_exit() {{ {kill_command} &> /dev/null }} {TRAP_KILL_CONTAINER} {cache_command} {run_command}""" def __cache_from_file_command(self, cached_image_file: str, docker_host_props: Dict[str, Any]) -> str: images_cmd = docker_util.build_docker_images_command(truncate=False, format="json", **docker_host_props) load_cmd = docker_util.build_docker_load_command(**docker_host_props) return string.Template(LOAD_CACHED_IMAGE_COMMAND_TEMPLATE).safe_substitute( cached_image_file=cached_image_file, images_cmd=images_cmd, load_cmd=load_cmd ) def __get_cached_image_file(self) -> Optional[str]: container_id = self.container_id cache_directory = os.path.abspath(self.__get_destination_overridable_property("container_image_cache_path")) cache_path = docker_cache_path(cache_directory, container_id) return cache_path if os.path.exists(cache_path) else None def __get_destination_overridable_property(self, name: str) -> Any: prop_name = f"docker_{name}" if prop_name in self.destination_info: return self.destination_info[prop_name] else: return getattr(self.app_info, name) def docker_cache_path(cache_directory: str, container_id: str) -> str: file_container_id = container_id.replace("/", "_slash_") cache_file_name = f"docker_{file_container_id}.tar" return os.path.join(cache_directory, cache_file_name) class SingularityContainer(Container, HasDockerLikeVolumes): container_type = SINGULARITY_CONTAINER_TYPE def get_singularity_target_kwds(self) -> Dict[str, Any]: return dict( singularity_cmd=self.prop("cmd", singularity_util.DEFAULT_SINGULARITY_COMMAND), sudo=asbool(self.prop("sudo", singularity_util.DEFAULT_SUDO)), sudo_cmd=self.prop("sudo_cmd", singularity_util.DEFAULT_SUDO_COMMAND), ) @property def connection_configuration(self) -> Dict[str, Any]: return self.get_singularity_target_kwds() def build_mulled_singularity_pull_command( self, cache_directory: str, namespace: str = "biocontainers" ) -> List[str]: return singularity_util.pull_mulled_singularity_command( docker_image_identifier=self.container_id, cache_directory=cache_directory, namespace=namespace, **self.get_singularity_target_kwds(), ) def build_singularity_pull_command(self, cache_path: str) -> List[str]: return singularity_util.pull_singularity_command( image_identifier=self.container_id, cache_path=cache_path, **self.get_singularity_target_kwds() ) def containerize_command(self, command: str) -> str: env = [] for pass_through_var in self.tool_info.env_pass_through: env.append((pass_through_var, f"${pass_through_var}")) # Allow destinations to explicitly set environment variables just for # docker container. Better approach is to set for destination and then # pass through only what tool needs however. (See todo in ToolInfo.) for key, value in self.destination_info.items(): if key.startswith("singularity_env_"): real_key = key[len("singularity_env_") :] env.append((real_key, value)) assert self.job_info is not None working_directory = self.job_info.working_directory if not working_directory: raise Exception(f"Cannot containerize command [{working_directory}] without defined working directory.") volumes_raw = self._expand_volume_str(self.destination_info.get("singularity_volumes", "$defaults")) volumes = _parse_volumes(volumes_raw, self.container_type) run_command = singularity_util.build_singularity_run_command( command, self.container_id, volumes=volumes, env=env, working_directory=working_directory, run_extra_arguments=self.prop("run_extra_arguments", singularity_util.DEFAULT_RUN_EXTRA_ARGUMENTS), guest_ports=self.tool_info.guest_ports, container_name=self.container_name, cleanenv=asbool(self.prop("cleanenv", singularity_util.DEFAULT_CLEANENV)), ipc=asbool(self.prop("ipc", singularity_util.DEFAULT_IPC)), pid=asbool(self.prop("pid", singularity_util.DEFAULT_PID)), contain=asbool(self.prop("contain", singularity_util.DEFAULT_CONTAIN)), no_mount=self.prop("no_mount", singularity_util.DEFAULT_NO_MOUNT), **self.get_singularity_target_kwds(), ) return run_command CONTAINER_CLASSES: Dict[str, Type[Container]] = dict( docker=DockerContainer, singularity=SingularityContainer, ) ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1785871615.045749 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/container_resolvers/0000755000175100017510000000000015234436377025753 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/container_resolvers/__init__.py0000644000175100017510000000516615234436334030065 0ustar00runnerrunner"""The module defines the abstract interface for resolving container images for tool execution.""" from abc import ( ABCMeta, abstractmethod, ) from typing import ( Any, Container, Optional, TYPE_CHECKING, ) from galaxy.util.bunch import Bunch from galaxy.util.dictifiable import Dictifiable if TYPE_CHECKING: from beaker.cache import Cache from ..dependencies import ( AppInfo, ToolInfo, ) from ..requirements import ContainerDescription class ResolutionCache(Bunch): """Simple cache for duplicated computation created once per set of requests (likely web request in Galaxy context). This should not be assumed to be thread safe - resolution using a given cache should all occur one resolution at a time in a single thread. """ mulled_resolution_cache: Optional["Cache"] = None class ContainerResolver(Dictifiable, metaclass=ABCMeta): """Description of a technique for resolving container images for tool execution.""" # Keys for dictification. dict_collection_visible_keys = ["resolver_type", "can_uninstall_dependencies", "builds_on_resolution"] can_uninstall_dependencies = False builds_on_resolution = False read_only = True # not used for containers, but set for when they are used like dependency resolvers def __init__(self, app_info: "AppInfo", **kwds) -> None: """Default initializer for ``ContainerResolver`` subclasses.""" self.app_info = app_info self.resolver_kwds = kwds def _get_config_option(self, key: str, default: Any = None) -> Any: """Look in resolver-specific settings for option and then fallback to global settings. """ return getattr(self.app_info, key, default) @abstractmethod def resolve( self, enabled_container_types: Container[str], tool_info: "ToolInfo", **kwds ) -> Optional["ContainerDescription"]: """Find a container matching all supplied requirements for tool. The supplied argument is a :class:`galaxy.tool_util.deps.dependencies.ToolInfo` description of the tool and its requirements. """ @property @abstractmethod def resolver_type(self) -> str: """Short label for the type of container resolution.""" def _container_type_enabled( self, container_description: "ContainerDescription", enabled_container_types: Container[str] ) -> bool: """Return a boolean indicating if the specified container type is enabled.""" return container_description.type in enabled_container_types def __str__(self) -> str: return f"{self.__class__.__name__}[]" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/container_resolvers/explicit.py0000644000175100017510000003116515234436334030145 0ustar00runnerrunner"""This module describes the :class:`ExplicitContainerResolver` ContainerResolver plugin.""" import copy import logging import os from typing import ( Container, Optional, TYPE_CHECKING, ) from galaxy.util.commands import shell from . import ContainerResolver from .mulled import ( CacheDirectory, CachedMulledImageSingleTarget, CliContainerResolver, get_cache_directory_cacher, identifier_to_cached_target, ) from ..container_classes import SingularityContainer from ..requirements import ContainerDescription if TYPE_CHECKING: from ..dependencies import ( AppInfo, ToolInfo, ) log = logging.getLogger(__name__) DEFAULT_SHELL = "/bin/bash" class ExplicitContainerResolver(ContainerResolver): """Find explicit containers referenced in the tool description (e.g. tool XML file) if present.""" resolver_type = "explicit" def resolve( self, enabled_container_types: Container[str], tool_info: "ToolInfo", **kwds ) -> Optional[ContainerDescription]: """Find a container explicitly mentioned in tool description. This ignores the tool requirements and assumes the tool author crafted a correct container. """ for container_description in tool_info.container_descriptions: if self._container_type_enabled(container_description, enabled_container_types): container_description.explicit = True return container_description return None class ExplicitSingularityContainerResolver(ExplicitContainerResolver): resolver_type = "explicit_singularity" container_type = "singularity" def resolve( self, enabled_container_types: Container[str], tool_info: "ToolInfo", **kwds ) -> Optional[ContainerDescription]: """Find a container explicitly mentioned in tool description. This ignores the tool requirements and assumes the tool author crafted a correct container. We use singularity here to fetch docker containers, hence the container_description hack here. """ for container_description in tool_info.container_descriptions: if container_description.type == "docker": desc_dict = container_description.to_dict() desc_dict["type"] = self.container_type desc_dict["identifier"] = f"docker://{container_description.identifier}" container_description = container_description.from_dict(desc_dict) if self._container_type_enabled(container_description, enabled_container_types): return container_description return None # TODO: should this derive from SingularityCliContainerResolver? class CachedExplicitSingularityContainerResolver(CliContainerResolver): resolver_type = "cached_explicit_singularity" container_type = "singularity" cli = "singularity" def __init__(self, app_info: "AppInfo", **kwargs) -> None: super().__init__(app_info=app_info, **kwargs) cache_directory_path = kwargs.get("cache_directory") if not cache_directory_path: assert self.app_info.container_image_cache_path cache_directory_path = os.path.join(self.app_info.container_image_cache_path, "singularity", "explicit") self.cache_directory_path = cache_directory_path self.namespace = kwargs.get("namespace") cache_directory_cacher_type = kwargs.get("cache_directory_cacher_type") cacher_class = get_cache_directory_cacher(cache_directory_cacher_type) self.cache_directory: CacheDirectory = cacher_class(self.cache_directory_path) os.makedirs(self.cache_directory_path, exist_ok=True) def resolve( self, enabled_container_types: Container[str], tool_info: "ToolInfo", install: bool = False, **kwds ) -> Optional[ContainerDescription]: """Find a container explicitly mentioned in tool description. This ignores the tool requirements and assumes the tool author crafted a correct container. We use singularity here to fetch docker containers, hence the container_description hack here. """ for container_description in tool_info.container_descriptions: container_description = copy.copy(container_description) if container_description.type == "docker": container_description.type = self.container_type container_description.identifier = f"docker://{container_description.identifier}" if not self._container_type_enabled(container_description, enabled_container_types): return None image_id = container_description.identifier if self.namespace: prefix = f"docker://quay.io/{self.namespace}/" if image_id.startswith(prefix): image_id = image_id[len(prefix) :] parsed = identifier_to_cached_target(image_id, "v2") if parsed and isinstance(parsed, CachedMulledImageSingleTarget): cached_images = self.cache_directory.list_cached_mulled_images_from_path() for cached in cached_images: if ( isinstance(cached, CachedMulledImageSingleTarget) and cached.package_name == parsed.package_name and cached.version == parsed.version ): container_description.identifier = os.path.join( self.cache_directory_path, cached.image_identifier ) return container_description if not install: return None else: if not install and not os.path.exists(os.path.join(self.cache_directory_path, image_id)): return None cache_path = os.path.join(self.cache_directory_path, image_id) else: if not self.cli_available: return container_description cache_path = os.path.normpath(os.path.join(self.cache_directory_path, image_id)) if install and not os.path.exists(cache_path): if not self.cli_available: return None destination_info = {} destination_for_container_type = kwds.get("destination_for_container_type") if destination_for_container_type: destination_info = destination_for_container_type(self.container_type) container = SingularityContainer( container_id=container_description.identifier, app_info=self.app_info, tool_info=tool_info, destination_info=destination_info, job_info=None, container_description=container_description, ) command = container.build_singularity_pull_command(cache_path=cache_path) shell(command) self.cache_directory.invalidate_cache() # Point to container in the cache in stead. container_description.identifier = cache_path return container_description else: # No container descriptions found return None def __str__(self): return ( f"CachedExplicitSingularityContainerResolver[" f"cache_directory={self.cache_directory_path}," f"namespace={self.namespace}," f"cache_directory_cacher_type={self.cache_directory.cacher_type}]" ) class BaseAdminConfiguredContainerResolver(ContainerResolver): def __init__(self, app_info: "AppInfo", shell: str = DEFAULT_SHELL, **kwds) -> None: super().__init__(app_info=app_info, **kwds) self.shell = shell def _container_description(self, identifier: str, container_type: str) -> ContainerDescription: container_description = ContainerDescription( identifier, type=container_type, shell=self.shell, ) return container_description class FallbackContainerResolver(BaseAdminConfiguredContainerResolver): """Specify an explicit, identified container as a Docker container resolver.""" resolver_type = "fallback" container_type = "docker" def __init__(self, app_info: "AppInfo", identifier: str = "", **kwds) -> None: super().__init__(app_info=app_info, **kwds) assert identifier, "fallback container resolver must be specified with non-empty identifier" self.identifier = identifier def _match( self, enabled_container_types: Container[str], tool_info: "ToolInfo", container_description: ContainerDescription, ) -> bool: if self._container_type_enabled(container_description, enabled_container_types): return True return False def resolve( self, enabled_container_types: Container[str], tool_info: "ToolInfo", **kwds ) -> Optional[ContainerDescription]: container_description = self._container_description(self.identifier, self.container_type) if self._match(enabled_container_types, tool_info, container_description): return container_description return None class FallbackSingularityContainerResolver(FallbackContainerResolver): """Specify an explicit, identified container as a Singularity container resolver.""" resolver_type = "fallback_singularity" container_type = "singularity" class FallbackNoRequirementsContainerResolver(FallbackContainerResolver): resolver_type = "fallback_no_requirements" def _match( self, enabled_container_types: Container[str], tool_info: "ToolInfo", container_description: ContainerDescription, ) -> bool: type_matches = super()._match(enabled_container_types, tool_info, container_description) return type_matches and (tool_info.requirements is None or len(tool_info.requirements) == 0) class FallbackNoRequirementsSingularityContainerResolver(FallbackNoRequirementsContainerResolver): resolver_type = "fallback_no_requirements_singularity" container_type = "singularity" class RequiresGalaxyEnvironmentContainerResolver(FallbackContainerResolver): resolver_type = "requires_galaxy_environment" def _match( self, enabled_container_types: Container[str], tool_info: "ToolInfo", container_description: ContainerDescription, ) -> bool: type_matches = super()._match(enabled_container_types, tool_info, container_description) return type_matches and tool_info.requires_galaxy_python_environment class RequiresGalaxyEnvironmentSingularityContainerResolver(RequiresGalaxyEnvironmentContainerResolver): resolver_type = "requires_galaxy_environment_singularity" container_type = "singularity" class MappingContainerResolver(BaseAdminConfiguredContainerResolver): resolver_type = "mapping" def __init__(self, app_info: "AppInfo", **kwds) -> None: super().__init__(app_info=app_info, **kwds) mappings = self.resolver_kwds["mappings"] assert isinstance(mappings, list), "mapping container resolver must be specified with mapping list" self.mappings = mappings def resolve( self, enabled_container_types: Container[str], tool_info: "ToolInfo", **kwds ) -> Optional[ContainerDescription]: tool_id = tool_info.tool_id # If resolving against dependencies and not a specific tool, skip over this resolver if not tool_id: return None tool_version = tool_info.tool_version for mapping in self.mappings: if mapping.get("tool_id") != tool_id: continue mapping_tool_version = mapping.get("tool_version") if mapping_tool_version is not None and tool_version != mapping_tool_version: continue container_description = self._container_description(mapping["identifier"], mapping.get("container_type")) if not self._container_type_enabled(container_description, enabled_container_types): continue return container_description return None __all__ = ( "ExplicitContainerResolver", "ExplicitSingularityContainerResolver", "CachedExplicitSingularityContainerResolver", "FallbackContainerResolver", "FallbackSingularityContainerResolver", "FallbackNoRequirementsContainerResolver", "FallbackNoRequirementsSingularityContainerResolver", "MappingContainerResolver", "RequiresGalaxyEnvironmentContainerResolver", "RequiresGalaxyEnvironmentSingularityContainerResolver", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/container_resolvers/mulled.py0000644000175100017510000007037315234436334027612 0ustar00runnerrunner"""This module describes the :class:`MulledContainerResolver` ContainerResolver plugin.""" import json import logging import os import subprocess from abc import ( ABCMeta, abstractmethod, ) from typing import ( Any, Callable, Container as TypingContainer, Dict, List, NamedTuple, Optional, Type, TYPE_CHECKING, Union, ) from requests import Session from typing_extensions import Literal from galaxy.util import ( safe_makedirs, string_as_bool, unicodify, which, ) from galaxy.util.commands import shell from . import ( ContainerResolver, ResolutionCache, ) from ..conda_util import CondaTarget from ..container_classes import ( Container, CONTAINER_CLASSES, DockerContainer, SingularityContainer, ) from ..docker_util import build_docker_images_command from ..mulled.mulled_build import ( ensure_installed, InvolucroContext, mull_targets, ) from ..mulled.mulled_build_tool import requirements_to_mulled_targets from ..mulled.util import ( DEFAULT_CHANNELS, default_mulled_conda_channels_from_env, find_remote_mulled_name, split_tag, v1_image_name, v2_image_name, version_sorted, ) from ..requirements import ( ContainerDescription, DEFAULT_CONTAINER_SHELL, ) if TYPE_CHECKING: from ..dependencies import ( AppInfo, ToolInfo, ) log = logging.getLogger(__name__) class CachedMulledImageSingleTarget(NamedTuple): package_name: str version: Optional[str] build: Optional[str] image_identifier: str class CachedV1MulledImageMultiTarget(NamedTuple): hash: str build: Optional[str] image_identifier: str class CachedV2MulledImageMultiTarget(NamedTuple): image_name: str version_hash: Optional[str] build: Optional[str] image_identifier: str @property def package_hash(self) -> str: # Make this work for Singularity file name or fully qualified Docker repository # image names. image_name = self.image_name if "/" not in image_name: return image_name else: return image_name.rsplit("/")[-1] CachedTarget = Union[CachedMulledImageSingleTarget, CachedV1MulledImageMultiTarget, CachedV2MulledImageMultiTarget] class CacheDirectory(metaclass=ABCMeta): cacher_type: str def __init__(self, path: str, hash_func: Literal["v1", "v2"] = "v2") -> None: self.path = path self.hash_func = hash_func def _list_cached_mulled_images_from_path(self) -> List[CachedTarget]: contents = os.listdir(self.path) sorted_images = version_sorted(contents) raw_images = (identifier_to_cached_target(name, self.hash_func) for name in sorted_images) return [i for i in raw_images if i is not None] @abstractmethod def list_cached_mulled_images_from_path(self) -> List[CachedTarget]: """Generate a list of cached, mulled images in the cache.""" @abstractmethod def invalidate_cache(self) -> None: """Invalidate the cache.""" class UncachedCacheDirectory(CacheDirectory): cacher_type = "uncached" def list_cached_mulled_images_from_path( self, ) -> List[CachedTarget]: return self._list_cached_mulled_images_from_path() def invalidate_cache(self) -> None: pass class DirMtimeCacheDirectory(CacheDirectory): cacher_type = "dir_mtime" def __init__(self, path: str, **kwargs): super().__init__(path, **kwargs) self.invalidate_cache() def __get_mtime(self) -> float: return os.stat(self.path).st_mtime def __cache(self) -> None: self.__contents = self._list_cached_mulled_images_from_path() self.__mtime = self.__get_mtime() log.debug(f"Cached images in path {self.path} at directory mtime {self.__mtime}") def list_cached_mulled_images_from_path( self, ) -> List[CachedTarget]: mtime = self.__get_mtime() if mtime != self.__mtime: if mtime < self.__mtime: log.warning( f"Modification time '{mtime}' of cache directory '{self.path}' is older than previous " f"modification time '{self.__mtime}'! Cache directory will be recached" ) self.__cache() return self.__contents def invalidate_cache(self) -> None: self.__mtime = -1.0 self.__contents = [] def get_cache_directory_cacher(cacher_type: Optional[str]) -> Type[CacheDirectory]: # these can become a separate module and use plugin_config if we need more cachers: Dict[str, Type[CacheDirectory]] = { UncachedCacheDirectory.cacher_type: UncachedCacheDirectory, DirMtimeCacheDirectory.cacher_type: DirMtimeCacheDirectory, } cacher_type = cacher_type or "uncached" return cachers[cacher_type] def list_docker_cached_mulled_images( namespace: Optional[str] = None, hash_func: Literal["v1", "v2"] = "v2", resolution_cache: Optional[ResolutionCache] = None, ) -> List[CachedTarget]: cache_key = "galaxy.tool_util.deps.container_resolvers.mulled:cached_images" if resolution_cache is not None and cache_key in resolution_cache: images_and_versions = resolution_cache.get(cache_key) else: command = build_docker_images_command(truncate=True, format="json", sudo=False, to_str=False) try: output = unicodify(subprocess.check_output(command)).strip() except subprocess.CalledProcessError: log.info("Call to `docker images` failed, configured container resolution may be broken") return [] # Parse JSON output from docker images images_and_versions = [] for line in output.splitlines(): if line.strip(): try: image_info = json.loads(line) repository = image_info.get("Repository", "") tag = image_info.get("Tag", "") if repository and tag: images_and_versions.append(f"{repository}:{tag}") except json.JSONDecodeError: log.warning(f"Failed to parse docker image JSON: {line}") continue if resolution_cache is not None: resolution_cache[cache_key] = images_and_versions def output_line_to_image(line: str) -> Optional[CachedTarget]: image = identifier_to_cached_target(line, hash_func, namespace=namespace) return image name_filter = get_filter(namespace) sorted_images = version_sorted(list(filter(name_filter, images_and_versions))) raw_images = (output_line_to_image(_) for _ in sorted_images) return [i for i in raw_images if i is not None] def identifier_to_cached_target( identifier: str, hash_func: Literal["v1", "v2"], namespace: Optional[str] = None ) -> Optional[CachedTarget]: if ":" in identifier: image_name, version = identifier.rsplit(":", 1) else: image_name = identifier version = None if not version or version == "latest": version = None image: Optional[CachedTarget] = None prefix = "" if namespace is not None: prefix = f"quay.io/{namespace}/" if image_name.startswith(f"{prefix}mulled-v1-"): if hash_func == "v2": return None hash = image_name build = None if version and version.isdigit(): build = version image = CachedV1MulledImageMultiTarget(hash, build, identifier) elif image_name.startswith(f"{prefix}mulled-v2-"): if hash_func == "v1": return None version_hash = None build = None if version: if "-" in version: version_hash, build = version.rsplit("-", 1) elif version.isdigit(): version_hash, build = None, version else: log.debug(f"Unparsable mulled image tag encountered [{version}]") image = CachedV2MulledImageMultiTarget(image_name, version_hash, build, identifier) else: build = None if version and "--" in version: version, build = split_tag(version) if prefix and image_name.startswith(prefix): image_name = image_name[len(prefix) :] image = CachedMulledImageSingleTarget(image_name, version, build, identifier) return image def get_filter(namespace: Optional[str]) -> Callable[[str], bool]: prefix = "quay.io/" if namespace is None else f"quay.io/{namespace}" return lambda name: name.startswith(prefix) and name.count("/") == 2 def find_best_matching_cached_image( targets: List[CondaTarget], cached_images: List[CachedTarget], hash_func: Literal["v1", "v2"] ) -> Optional[CachedTarget]: if len(targets) == 0: return None image: Optional[CachedTarget] = None cached_image: CachedTarget if len(targets) == 1: target = targets[0] for cached_image in cached_images: if not isinstance(cached_image, CachedMulledImageSingleTarget): continue if not cached_image.package_name == target.package: continue if not target.version or target.version == cached_image.version: image = cached_image break elif hash_func == "v2": name = v2_image_name(targets) if ":" in name: package_hash, version_hash = name.split(":", 2) else: package_hash, version_hash = name, None for cached_image in cached_images: if not isinstance(cached_image, CachedV2MulledImageMultiTarget): continue if version_hash is None: # Just match on package hash... if package_hash == cached_image.package_hash: image = cached_image break else: # Match on package and version hash... if package_hash == cached_image.package_hash and version_hash == cached_image.version_hash: image = cached_image break elif hash_func == "v1": name = v1_image_name(targets) for cached_image in cached_images: if not isinstance(cached_image, CachedV1MulledImageMultiTarget): continue if name == cached_image.hash: image = cached_image break return image def docker_cached_container_description( targets: List[CondaTarget], namespace: str, hash_func: Literal["v1", "v2"] = "v2", shell: str = DEFAULT_CONTAINER_SHELL, resolution_cache: Optional[ResolutionCache] = None, ) -> Optional[ContainerDescription]: if len(targets) == 0: return None cached_images = list_docker_cached_mulled_images(namespace, hash_func=hash_func, resolution_cache=resolution_cache) image = find_best_matching_cached_image(targets, cached_images, hash_func) container = None if image: container = ContainerDescription( image.image_identifier, type="docker", shell=shell, ) return container def singularity_cached_container_description( targets: List[CondaTarget], cache_directory: CacheDirectory, hash_func: Literal["v1", "v2"] = "v2", shell: str = DEFAULT_CONTAINER_SHELL, ) -> Optional[ContainerDescription]: if len(targets) == 0: return None if not os.path.exists(cache_directory.path): return None cached_images = cache_directory.list_cached_mulled_images_from_path() image = find_best_matching_cached_image(targets, cached_images, hash_func) container = None if image: container = ContainerDescription( os.path.abspath(os.path.join(cache_directory.path, image.image_identifier)), type="singularity", shell=shell, ) return container def targets_to_mulled_name( targets: List[CondaTarget], hash_func: Literal["v1", "v2"], namespace: str, resolution_cache: Optional[ResolutionCache] = None, session: Optional[Session] = None, ) -> Optional[str]: unresolved_cache_key = "galaxy.tool_util.deps.container_resolvers.mulled:unresolved" if resolution_cache is not None: if unresolved_cache_key not in resolution_cache: resolution_cache[unresolved_cache_key] = set() unresolved_cache = resolution_cache.get(unresolved_cache_key) else: unresolved_cache = set() mulled_resolution_cache = None if resolution_cache and resolution_cache.mulled_resolution_cache: mulled_resolution_cache = resolution_cache.mulled_resolution_cache name = None def cached_name(cache_key: str) -> Optional[str]: if mulled_resolution_cache: try: return resolution_cache.get(cache_key) # type: ignore[union-attr] # mulled_resolution_cache not None implies resolution_cache not None except KeyError: return None return None if len(targets) == 1: target = targets[0] cache_key = f"ns[{namespace}]__single__{target.package}__@__{target.version}" elif hash_func == "v2": cache_key = f"ns[{namespace}]__{hash_func}__{v2_image_name(targets)}" elif hash_func == "v1": cache_key = f"ns[{namespace}]__{hash_func}__{v1_image_name(targets)}" else: raise Exception(f"Unimplemented mulled hash_func [{hash_func}]") if cache_key in unresolved_cache: return None name = cached_name(cache_key) if name: return name # Shared with the recommender: resolve targets to an exact remote mulled image name # (or None). allow_newest_fallback stays off here -- the resolver wants exact matches only. match = find_remote_mulled_name(targets, namespace, hash_func, resolution_cache=resolution_cache, session=session) name = match.name if match else None if name and mulled_resolution_cache: mulled_resolution_cache.put(cache_key, name) if name is None: unresolved_cache.add(name) return name class CliContainerResolver(ContainerResolver): container_type = "docker" cli = "docker" def __init__(self, app_info: "AppInfo", **kwargs) -> None: super().__init__(app_info=app_info, **kwargs) self._cli_available = bool(which(self.cli)) @property def cli_available(self) -> bool: return self._cli_available @cli_available.setter def cli_available(self, value: bool) -> None: if not value: log.info( f"{self.cli} CLI not available, cannot list or pull images in Galaxy process. Does not impact kubernetes." ) self._cli_available = value class SingularityCliContainerResolver(CliContainerResolver): container_type = "singularity" cli = "singularity" def __init__(self, app_info: "AppInfo", hash_func: Literal["v1", "v2"] = "v2", **kwargs) -> None: super().__init__(app_info=app_info, **kwargs) self.hash_func = hash_func self.cache_directory_cacher_type = kwargs.get("cache_directory_cacher_type") cacher_class = get_cache_directory_cacher(self.cache_directory_cacher_type) cache_directory_path = kwargs.get("cache_directory") if not cache_directory_path: assert self.app_info.container_image_cache_path cache_directory_path = os.path.join(self.app_info.container_image_cache_path, "singularity", "mulled") self.cache_directory = cacher_class(cache_directory_path, hash_func=self.hash_func) safe_makedirs(self.cache_directory.path) class CachedMulledDockerContainerResolver(CliContainerResolver): resolver_type = "cached_mulled" shell = "/bin/bash" def __init__( self, app_info: "AppInfo", namespace: str = "biocontainers", hash_func: Literal["v1", "v2"] = "v2", **kwds ): super().__init__(app_info=app_info, **kwds) self.namespace = namespace self.hash_func = hash_func def resolve( self, enabled_container_types: TypingContainer[str], tool_info: "ToolInfo", **kwds ) -> Optional[ContainerDescription]: if ( not self.cli_available or tool_info.requires_galaxy_python_environment or self.container_type not in enabled_container_types ): return None targets = mulled_targets(tool_info) log.debug(f"Image name for tool {tool_info.tool_id}: {image_name(targets, self.hash_func)}") resolution_cache = kwds.get("resolution_cache") return docker_cached_container_description( targets, self.namespace, hash_func=self.hash_func, shell=self.shell, resolution_cache=resolution_cache ) def __str__(self): return f"CachedMulledDockerContainerResolver[namespace={self.namespace}]" class CachedMulledSingularityContainerResolver(SingularityCliContainerResolver): resolver_type = "cached_mulled_singularity" shell = "/bin/bash" def resolve( self, enabled_container_types: TypingContainer[str], tool_info: "ToolInfo", **kwds ) -> Optional[ContainerDescription]: if tool_info.requires_galaxy_python_environment or self.container_type not in enabled_container_types: return None targets = mulled_targets(tool_info) log.debug(f"Image name for tool {tool_info.tool_id}: {image_name(targets, self.hash_func)}") return singularity_cached_container_description( targets, self.cache_directory, hash_func=self.hash_func, shell=self.shell ) def __str__(self) -> str: return f"CachedMulledSingularityContainerResolver[cache_directory={self.cache_directory.path}]" class MulledDockerContainerResolver(CliContainerResolver): """Look for mulled images matching tool dependencies.""" resolver_type = "mulled" shell = "/bin/bash" protocol: Optional[str] = None def __init__( self, app_info: "AppInfo", namespace: str = "biocontainers", hash_func: Literal["v1", "v2"] = "v2", auto_install: bool = True, **kwds, ) -> None: super().__init__(app_info=app_info, **kwds) self.namespace = namespace self.hash_func = hash_func self.auto_install = string_as_bool(auto_install) def cached_container_description( self, targets: List[CondaTarget], namespace: str, hash_func: Literal["v1", "v2"], resolution_cache: Optional[ResolutionCache] = None, ) -> Optional[ContainerDescription]: try: return docker_cached_container_description( targets, namespace, hash_func=hash_func, resolution_cache=resolution_cache ) except subprocess.CalledProcessError: # We should only get here if a docker binary is available, but command quits with a non-zero exit code, # e.g if the docker daemon is not available log.exception("An error occured while listing cached docker image. Docker daemon may need to be restarted.") return None def pull(self, container: Container) -> None: if self.cli_available: assert isinstance(container, DockerContainer) command = container.build_pull_command() shell(command) @property def can_list_containers(self) -> bool: return self.cli_available def resolve( self, enabled_container_types: TypingContainer[str], tool_info: "ToolInfo", install: bool = False, session: Optional[Session] = None, **kwds, ) -> Optional[ContainerDescription]: resolution_cache = kwds.get("resolution_cache") if tool_info.requires_galaxy_python_environment or self.container_type not in enabled_container_types: return None targets = mulled_targets(tool_info) log.debug(f"Image name for tool {tool_info.tool_id}: {image_name(targets, self.hash_func)}") if len(targets) == 0: return None name = targets_to_mulled_name( targets=targets, hash_func=self.hash_func, namespace=self.namespace, resolution_cache=resolution_cache, session=session, ) if not name: return None container_id = f"quay.io/{self.namespace}/{name}" if self.protocol: container_id = f"{self.protocol}{container_id}" container_description = ContainerDescription( container_id, type=self.container_type, shell=self.shell, ) if self.can_list_containers: if install and not self.cached_container_description( targets, namespace=self.namespace, hash_func=self.hash_func, resolution_cache=resolution_cache, ): destination_info = {} destination_for_container_type = kwds.get("destination_for_container_type") if destination_for_container_type: destination_info = destination_for_container_type(self.container_type) container = CONTAINER_CLASSES[self.container_type]( container_description.identifier, self.app_info, tool_info, destination_info, None, container_description, ) self.pull(container) if not self.auto_install: container_description = ( self.cached_container_description( targets, namespace=self.namespace, hash_func=self.hash_func, resolution_cache=resolution_cache, ) or container_description ) return container_description def __str__(self) -> str: return f"MulledDockerContainerResolver[namespace={self.namespace}]" class MulledSingularityContainerResolver(SingularityCliContainerResolver, MulledDockerContainerResolver): resolver_type = "mulled_singularity" protocol = "docker://" def __init__( self, app_info: "AppInfo", hash_func: Literal["v1", "v2"] = "v2", namespace: str = "biocontainers", auto_install: bool = True, **kwds, ) -> None: super().__init__(app_info=app_info, hash_func=hash_func, **kwds) self.namespace = namespace self.auto_install = string_as_bool(auto_install) def cached_container_description( self, targets: List[CondaTarget], namespace: str, hash_func: Literal["v1", "v2"], resolution_cache: Optional[ResolutionCache] = None, ) -> Optional[ContainerDescription]: return singularity_cached_container_description( targets, cache_directory=self.cache_directory, hash_func=hash_func ) @property def can_list_containers(self) -> bool: # Only needs access to path, doesn't require CLI return True def pull(self, container: Container) -> None: if self.cli_available: assert isinstance(container, SingularityContainer) cmds = container.build_mulled_singularity_pull_command( cache_directory=self.cache_directory.path, namespace=self.namespace ) shell(cmds=cmds) self.cache_directory.invalidate_cache() def __str__(self) -> str: return f"MulledSingularityContainerResolver[namespace={self.namespace}]" class BuildMulledDockerContainerResolver(CliContainerResolver): """Build for Docker mulled images matching tool dependencies.""" resolver_type = "build_mulled" shell = "/bin/bash" builds_on_resolution = True def __init__( self, app_info: "AppInfo", namespace: str = "local", hash_func: Literal["v1", "v2"] = "v2", auto_install: bool = True, **kwds, ) -> None: super().__init__(app_info=app_info, **kwds) self._involucro_context_kwds = {"involucro_bin": self._get_config_option("involucro_path", None)} self.namespace = namespace self.hash_func = hash_func self.auto_install = string_as_bool(auto_install) self._mulled_kwds: Dict[str, Any] = { "namespace": namespace, "hash_func": self.hash_func, "command": "build-and-test", "use_mamba": False, } self._mulled_kwds["channels"] = default_mulled_conda_channels_from_env() or self._get_config_option( "mulled_channels", DEFAULT_CHANNELS ) self.involucro_context = InvolucroContext(**self._involucro_context_kwds) auto_init = self._get_config_option("involucro_auto_init", True) self.enabled = ensure_installed(self.involucro_context, auto_init) def resolve( self, enabled_container_types: TypingContainer[str], tool_info: "ToolInfo", install: bool = False, **kwds ) -> Optional[ContainerDescription]: if tool_info.requires_galaxy_python_environment or self.container_type not in enabled_container_types: return None targets = mulled_targets(tool_info) log.debug(f"Image name for tool {tool_info.tool_id}: {image_name(targets, self.hash_func)}") if len(targets) == 0: return None if self.auto_install or install: mull_targets(targets, involucro_context=self.involucro_context, **self._mulled_kwds) return docker_cached_container_description(targets, self.namespace, hash_func=self.hash_func, shell=self.shell) def __str__(self) -> str: return f"BuildDockerContainerResolver[namespace={self.namespace}]" class BuildMulledSingularityContainerResolver(SingularityCliContainerResolver): """Build for Singularity mulled images matching tool dependencies.""" resolver_type = "build_mulled_singularity" shell = "/bin/bash" builds_on_resolution = True def __init__( self, app_info: "AppInfo", hash_func: Literal["v1", "v2"] = "v2", auto_install: bool = True, **kwds, ) -> None: super().__init__(app_info=app_info, hash_func=hash_func, **kwds) self._involucro_context_kwds = {"involucro_bin": self._get_config_option("involucro_path", None)} self.auto_install = string_as_bool(auto_install) self._mulled_kwds = { "channels": self._get_config_option("mulled_channels", DEFAULT_CHANNELS), "hash_func": self.hash_func, "command": "build-and-test", "namespace": "local", "singularity": True, "singularity_image_dir": self.cache_directory.path, "use_mamba": False, } self.involucro_context = InvolucroContext(**self._involucro_context_kwds) auto_init = self._get_config_option("involucro_auto_init", True) self.enabled = ensure_installed(self.involucro_context, auto_init) def resolve( self, enabled_container_types: TypingContainer[str], tool_info: "ToolInfo", install: bool = False, **kwds ) -> Optional[ContainerDescription]: if tool_info.requires_galaxy_python_environment or self.container_type not in enabled_container_types: return None targets = mulled_targets(tool_info) log.debug(f"Image name for tool {tool_info.tool_id}: {image_name(targets, self.hash_func)}") if len(targets) == 0: return None if self.auto_install or install: mull_targets(targets, involucro_context=self.involucro_context, **self._mulled_kwds) return singularity_cached_container_description( targets, self.cache_directory, hash_func=self.hash_func, shell=self.shell ) def __str__(self) -> str: return f"BuildSingularityContainerResolver[cache_directory={self.cache_directory.path}]" def mulled_targets(tool_info: "ToolInfo") -> List[CondaTarget]: return requirements_to_mulled_targets(tool_info.requirements) def image_name(targets: List[CondaTarget], hash_func: Literal["v1", "v2"]) -> str: if len(targets) == 0: return "no targets" elif hash_func == "v2": return v2_image_name(targets) else: return v1_image_name(targets) __all__ = ( "CachedMulledDockerContainerResolver", "CachedMulledSingularityContainerResolver", "MulledDockerContainerResolver", "MulledSingularityContainerResolver", "BuildMulledDockerContainerResolver", "BuildMulledSingularityContainerResolver", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/container_resolvers/test.py0000644000175100017510000000111315234436334027271 0ustar00runnerrunnerfrom typing import ( Container, Optional, TYPE_CHECKING, ) from . import ContainerResolver if TYPE_CHECKING: from ..dependencies import ToolInfo from ..requirements import ContainerDescription class NullContainerResolver(ContainerResolver): """ helper ContainerResolver that does not resolve ever for testing """ resolver_type = "null" def resolve( self, enabled_container_types: Container[str], tool_info: "ToolInfo", **kwds ) -> Optional["ContainerDescription"]: return None __all__ = ("NullContainerResolver",) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/container_volumes.py0000644000175100017510000000472215234436334025771 0ustar00runnerrunnerimport shlex from abc import ( ABCMeta, abstractmethod, ) from typing import Optional class ContainerVolume(metaclass=ABCMeta): valid_modes = frozenset({"ro", "rw", "z", "Z"}) def __init__(self, path: str, host_path: Optional[str] = None, mode: Optional[str] = None): self.path = path self.host_path = host_path self.mode = mode if mode and not self.mode_is_valid: raise ValueError(f"Invalid container volume mode: {mode}") @abstractmethod def from_str(cls, as_str: str) -> "ContainerVolume": """Classmethod to convert from this container type's string representation. :param as_str: string representation of volume :type as_str: str """ @abstractmethod def __str__(self) -> str: """Return this container type's string representation of the volume.""" @property def mode_is_valid(self) -> bool: return self.mode in self.valid_modes class DockerVolume(ContainerVolume): @classmethod def from_str(cls, as_str: str) -> "DockerVolume": """Construct an instance from a string as would be passed to `docker run --volume`. A string in the format ``:`` is supported for legacy purposes even though it is not valid Docker volume syntax. """ if not as_str: raise ValueError(f"Failed to parse Docker volume from {as_str}") parts = as_str.split(":", 2) kwds = dict(host_path=parts[0]) if len(parts) == 1: # auto-generated volume kwds["path"] = kwds["host_path"] elif len(parts) == 2: # /host_path:mode is not (or is no longer?) valid Docker volume syntax if all(mode_part in DockerVolume.valid_modes for mode_part in parts[1].split(",")): kwds["mode"] = parts[1] kwds["path"] = kwds["host_path"] else: kwds["path"] = parts[1] elif len(parts) == 3: kwds["path"] = parts[1] kwds["mode"] = parts[2] return cls(**kwds) def __str__(self) -> str: volume_str = ":".join(x for x in (self.host_path, self.path, self.mode) if x is not None) if "$" not in volume_str: volume_for_cmd_line = shlex.quote(volume_str) else: # e.g. $_GALAXY_JOB_TMP_DIR:$_GALAXY_JOB_TMP_DIR:rw so don't single quote. volume_for_cmd_line = f'"{volume_str}"' return volume_for_cmd_line ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/containers.py0000644000175100017510000004333415234436334024404 0ustar00runnerrunnerimport collections import logging import os from typing import ( Any, Container as TypingContainer, Dict, List, Optional, Type, TYPE_CHECKING, ) from requests import Session from typing_extensions import Literal from galaxy.util import ( asbool, plugin_config, ) from . import container_resolvers from .container_classes import ( Container, CONTAINER_CLASSES, DOCKER_CONTAINER_TYPE, SINGULARITY_CONTAINER_TYPE, ) from .container_resolvers import ResolutionCache from .container_resolvers.explicit import ( ExplicitContainerResolver, ExplicitSingularityContainerResolver, ) from .container_resolvers.mulled import ( BuildMulledDockerContainerResolver, BuildMulledSingularityContainerResolver, CachedMulledDockerContainerResolver, CachedMulledSingularityContainerResolver, MulledDockerContainerResolver, MulledSingularityContainerResolver, ) from .requirements import ContainerDescription if TYPE_CHECKING: from beaker.cache import Cache from galaxy.util.plugin_config import PluginConfigSource from .container_resolvers import ContainerResolver from .dependencies import ( AppInfo, JobInfo, ToolInfo, ) log = logging.getLogger(__name__) DEFAULT_CONTAINER_TYPE = DOCKER_CONTAINER_TYPE ALL_CONTAINER_TYPES = [DOCKER_CONTAINER_TYPE, SINGULARITY_CONTAINER_TYPE] ResolvedContainerDescription = collections.namedtuple( "ResolvedContainerDescription", ["container_resolver", "container_description"] ) class ContainerFinder: def __init__(self, app_info: "AppInfo", mulled_resolution_cache: Optional["Cache"] = None) -> None: self.app_info = app_info self.mulled_resolution_cache = mulled_resolution_cache self.default_container_registry = ContainerRegistry(app_info, mulled_resolution_cache=mulled_resolution_cache) self.destination_container_registeries: Dict[str, ContainerRegistry] = {} def _enabled_container_types(self, destination_info: Dict[str, Any]) -> List[str]: return [t for t in ALL_CONTAINER_TYPES if self.__container_type_enabled(t, destination_info)] def find_best_container_description( self, enabled_container_types: TypingContainer[str], tool_info: "ToolInfo", **kwds ) -> Optional[ContainerDescription]: """Regardless of destination properties - find best container for tool. Given container types and container.ToolInfo description of the tool.""" return self.default_container_registry.find_best_container_description( enabled_container_types, tool_info, **kwds ) def resolve( self, enabled_container_types: TypingContainer[str], tool_info: "ToolInfo", **kwds ) -> Optional[ResolvedContainerDescription]: """Regardless of destination properties - find ResolvedContainerDescription for tool.""" return self.default_container_registry.resolve(enabled_container_types, tool_info, **kwds) def _container_registry_for_destination(self, destination_info: Dict[str, Any]) -> "ContainerRegistry": destination_id = destination_info.get("id") # Probably not the way to get the ID? destination_container_registry = None if destination_id and destination_id not in self.destination_container_registeries: if "container_resolvers" in destination_info or "container_resolvers_config_file" in destination_info: destination_container_registry = ContainerRegistry( self.app_info, destination_info=destination_info, mulled_resolution_cache=self.mulled_resolution_cache, ) self.destination_container_registeries[destination_id] = destination_container_registry elif not destination_id and ( "container_resolvers" in destination_info or "container_resolvers_config_file" in destination_info ): destination_container_registry = ContainerRegistry( self.app_info, destination_info=destination_info, mulled_resolution_cache=self.mulled_resolution_cache ) if ( destination_container_registry is None and destination_id and destination_id in self.destination_container_registeries ): destination_container_registry = self.destination_container_registeries[destination_id] return destination_container_registry or self.default_container_registry def find_container( self, tool_info: "ToolInfo", destination_info: Dict[str, Any], job_info: "JobInfo" ) -> Optional[Container]: enabled_container_types = self._enabled_container_types(destination_info) # Short-cut everything else and just skip checks if no container type is enabled. if not enabled_container_types: return None def __destination_container( container_description: Optional[ContainerDescription] = None, container_id: Optional[str] = None, container_type: Optional[str] = None, ) -> Optional[Container]: """ either container_description or container_id and container_type must me given """ if container_description: container_id = container_description.identifier container_type = container_description.type if container_type is None: return None else: assert container_id container = self.__destination_container( container_id, container_type, tool_info, destination_info, job_info, container_description, ) return container def container_from_description_from_dicts( destination_container_dicts: List[Dict[str, Any]], ) -> Optional[Container]: for destination_container_dict in destination_container_dicts: container_description = ContainerDescription.from_dict(destination_container_dict) if container_description: container = __destination_container(container_description) if container: return container return None if "container_override" in destination_info: container = container_from_description_from_dicts(destination_info["container_override"]) if container: return container # If destination forcing Galaxy to use a particular container do it, # this is likely kind of a corner case. For instance if deployers # do not trust the containers annotated in tools. for container_type in CONTAINER_CLASSES.keys(): container_id = self.__overridden_container_id(container_type, destination_info) if container_id: container = __destination_container(container_type=container_type, container_id=container_id) if container: return container # Otherwise lets see if we can find container for the tool. container_registry = self._container_registry_for_destination(destination_info) container_description = container_registry.find_best_container_description(enabled_container_types, tool_info) container = __destination_container(container_description) if container: return container # If we still don't have a container, check to see if any container # types define a default container id and use that. if "container" in destination_info: container = container_from_description_from_dicts(destination_info["container"]) if container: return container for container_type in CONTAINER_CLASSES.keys(): container_id = self.__default_container_id(container_type, destination_info) if container_id: container = __destination_container(container_type=container_type, container_id=container_id) if container: return container return None def resolution_cache(self) -> ResolutionCache: return self.default_container_registry.get_resolution_cache() def __overridden_container_id(self, container_type: str, destination_info: Dict[str, Any]) -> Optional[str]: if not self.__container_type_enabled(container_type, destination_info): return None if f"{container_type}_container_id_override" in destination_info: return destination_info.get(f"{container_type}_container_id_override") if f"{container_type}_image_override" in destination_info: return self.__build_container_id_from_parts(container_type, destination_info, mode="override") return None def __build_container_id_from_parts( self, container_type: str, destination_info: Dict[str, Any], mode: Literal["default", "override"] ) -> str: repo = "" owner = "" repo_key = f"{container_type}_repo_{mode}" owner_key = f"{container_type}_owner_{mode}" if repo_key in destination_info: repo = f"{destination_info[repo_key]}/" if owner_key in destination_info: owner = f"{destination_info[owner_key]}/" cont_id = repo + owner + destination_info[f"{container_type}_image_{mode}"] tag_key = f"{container_type}_tag_{mode}" if tag_key in destination_info: cont_id += f":{destination_info[tag_key]}" return cont_id def __default_container_id(self, container_type: str, destination_info: Dict[str, Any]) -> Optional[str]: if not self.__container_type_enabled(container_type, destination_info): return None key = f"{container_type}_default_container_id" # Also allow docker_image... if key not in destination_info: key = f"{container_type}_image" if key in destination_info: return destination_info.get(key) elif f"{container_type}_image_default" in destination_info: return self.__build_container_id_from_parts(container_type, destination_info, mode="default") return None def __destination_container( self, container_id: str, container_type: str, tool_info: "ToolInfo", destination_info: Dict[str, Any], job_info: "JobInfo", container_description: Optional[ContainerDescription] = None, ) -> Optional[Container]: # TODO: ensure destination_info is dict-like if not self.__container_type_enabled(container_type, destination_info): return None # TODO: Right now this assumes all containers available when a # container type is - there should be more thought put into this. # Checking which are available - settings policies for what can be # auto-fetched, etc.... return CONTAINER_CLASSES[container_type]( container_id, self.app_info, tool_info, destination_info, job_info, container_description ) def __container_type_enabled(self, container_type: str, destination_info: Dict[str, Any]) -> bool: return asbool(destination_info.get(f"{container_type}_enabled", False)) class NullContainerFinder: def find_container(self, tool_info: "ToolInfo", destination_info: Dict[str, Any], job_info: "JobInfo") -> None: return None class ContainerRegistry: """Loop through enabled ContainerResolver plugins and find first match.""" def __init__( self, app_info: "AppInfo", destination_info: Optional[Dict[str, Any]] = None, mulled_resolution_cache: Optional["Cache"] = None, ) -> None: self.resolver_classes = self.__resolvers_dict() self.enable_mulled_containers = app_info.enable_mulled_containers self.app_info = app_info self.container_resolvers = self.__build_container_resolvers(app_info, destination_info) self.mulled_resolution_cache = mulled_resolution_cache def __build_container_resolvers( self, app_info: "AppInfo", destination_info: Optional[Dict[str, Any]] ) -> List["ContainerResolver"]: app_conf_file = getattr(app_info, "container_resolvers_config_file", None) app_conf_dict = getattr(app_info, "container_resolvers_config_dict", None) if destination_info is not None and ( "container_resolvers" in destination_info or "container_resolvers_config_file" in destination_info ): conf_file = destination_info.get("container_resolvers_config_file") conf_dict = destination_info.get("container_resolvers") else: conf_file = app_conf_file conf_dict = app_conf_dict plugin_source = None if conf_dict: log.debug("Loading container resolution config inline from Galaxy or job configuration file") plugin_source = plugin_config.plugin_source_from_dict(conf_dict) elif conf_file and not os.path.exists(conf_file): log.warning(f"Unable to find config file '{conf_file}'") elif conf_file: log.debug(f"Loading container resolution config from file '{conf_file}'") plugin_source = plugin_config.plugin_source_from_path(conf_file) if plugin_source: return self._parse_resolver_conf(plugin_source) return self.__default_container_resolvers() def _parse_resolver_conf(self, plugin_source: "PluginConfigSource") -> List["ContainerResolver"]: extra_kwds = {"app_info": self.app_info} return plugin_config.load_plugins(self.resolver_classes, plugin_source, extra_kwds) def __default_container_resolvers(self) -> List["ContainerResolver"]: default_resolvers: List[ContainerResolver] = [ ExplicitContainerResolver(self.app_info), ExplicitSingularityContainerResolver(self.app_info), ] if self.enable_mulled_containers: default_resolvers.extend( [ CachedMulledDockerContainerResolver(self.app_info, namespace="biocontainers"), CachedMulledDockerContainerResolver(self.app_info, namespace="local"), CachedMulledSingularityContainerResolver(self.app_info, namespace="biocontainers"), CachedMulledSingularityContainerResolver(self.app_info, namespace="local"), MulledDockerContainerResolver(self.app_info, namespace="biocontainers"), MulledSingularityContainerResolver(self.app_info, namespace="biocontainers"), ] ) # BuildMulledDockerContainerResolver and BuildMulledSingularityContainerResolver both need the docker daemon to build images. # If docker is not available, we don't load them. build_mulled_docker_container_resolver = BuildMulledDockerContainerResolver(self.app_info) if build_mulled_docker_container_resolver.cli_available: default_resolvers.extend( [ build_mulled_docker_container_resolver, BuildMulledSingularityContainerResolver(self.app_info), ] ) return default_resolvers def __resolvers_dict(self) -> Dict[str, Type["ContainerResolver"]]: return plugin_config.plugins_dict(container_resolvers, "resolver_type") def get_resolution_cache(self) -> ResolutionCache: cache = ResolutionCache() if self.mulled_resolution_cache is not None: cache.mulled_resolution_cache = self.mulled_resolution_cache return cache def find_best_container_description( self, enabled_container_types: TypingContainer[str], tool_info: "ToolInfo", **kwds: Any ) -> Optional[ContainerDescription]: """Yield best container description of supplied types matching tool info.""" try: resolved_container_description = self.resolve(enabled_container_types, tool_info, **kwds) except Exception: log.exception("Could not get container description for tool '%s'", tool_info.tool_id) return None return None if resolved_container_description is None else resolved_container_description.container_description def resolve( self, enabled_container_types: TypingContainer[str], tool_info: "ToolInfo", index: Optional[int] = None, resolver_type: Optional[str] = None, install: bool = True, resolution_cache: Optional[ResolutionCache] = None, session: Optional[Session] = None, ) -> Optional[ResolvedContainerDescription]: resolution_cache = resolution_cache or self.get_resolution_cache() for i, container_resolver in enumerate(self.container_resolvers): if index is not None and i != index: continue if resolver_type is not None and resolver_type != container_resolver.resolver_type: continue if hasattr(container_resolver, "container_type"): if container_resolver.container_type not in enabled_container_types: continue if not install and container_resolver.builds_on_resolution: continue container_description = container_resolver.resolve( enabled_container_types, tool_info, install=install, resolution_cache=resolution_cache, session=session ) log.info( f"Checking with container resolver [{container_resolver}] found description [{container_description}]" ) if container_description: assert container_resolver._container_type_enabled(container_description, enabled_container_types) return ResolvedContainerDescription(container_resolver, container_description) return None ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/dependencies.py0000644000175100017510000001535615234436334024670 0ustar00runnerrunnerfrom typing import ( Any, Dict, List, Optional, Union, ) from galaxy.tool_util.deps.requirements import ( ContainerDescription, ToolRequirement, ToolRequirements, ) from galaxy.util import bunch from .mulled.util import DEFAULT_CHANNELS class AppInfo: def __init__( self, galaxy_root_dir: Optional[str] = None, default_file_path: Optional[str] = None, tool_data_path: Optional[str] = None, galaxy_data_manager_data_path: Optional[str] = None, shed_tool_data_path: Optional[str] = None, outputs_to_working_directory: bool = False, container_image_cache_path: Optional[str] = None, library_import_dir: Optional[str] = None, enable_mulled_containers: bool = False, container_resolvers_config_file: Optional[str] = None, container_resolvers_config_dict: Optional[Dict[str, Any]] = None, involucro_path: Optional[str] = None, involucro_auto_init: bool = True, mulled_channels: List[str] = DEFAULT_CHANNELS, ) -> None: self.galaxy_root_dir = galaxy_root_dir self.default_file_path = default_file_path self.tool_data_path = tool_data_path self.galaxy_data_manager_data_path = galaxy_data_manager_data_path self.shed_tool_data_path = shed_tool_data_path # TODO: Vary default value for docker_volumes based on this... self.outputs_to_working_directory = outputs_to_working_directory self.container_image_cache_path = container_image_cache_path self.library_import_dir = library_import_dir self.enable_mulled_containers = enable_mulled_containers self.container_resolvers_config_file = container_resolvers_config_file self.container_resolvers_config_dict = container_resolvers_config_dict self.involucro_path = involucro_path self.involucro_auto_init = involucro_auto_init self.mulled_channels = mulled_channels class ToolInfo: # TODO: Introduce tool XML syntax to annotate the optional environment # variables they can consume (e.g. JVM options, license keys, etc..) # and add these to env_path_through def __init__( self, container_descriptions: Optional[List["ContainerDescription"]] = None, requirements: Optional[Union["ToolRequirements", List["ToolRequirement"]]] = None, requires_galaxy_python_environment: bool = False, env_pass_through=None, guest_ports=None, tool_id: Optional[str] = None, tool_version: Optional[str] = None, profile: float = -1, ): if env_pass_through is None: env_pass_through = ["GALAXY_SLOTS", "GALAXY_MEMORY_MB", "GALAXY_MEMORY_MB_PER_SLOT"] if container_descriptions is None: container_descriptions = [] if requirements is None: requirements = [] self.container_descriptions = container_descriptions self.requirements = requirements self.requires_galaxy_python_environment = requires_galaxy_python_environment self.env_pass_through = env_pass_through self.guest_ports = guest_ports self.tool_id = tool_id self.tool_version = tool_version self.profile = profile class JobInfo: def __init__( self, working_directory, tool_directory, job_directory, tmp_directory, home_directory, job_directory_type, ): self.working_directory = working_directory # Tool files may be remote staged - so this is unintuitively a property # of the job not of the tool. self.tool_directory = tool_directory self.job_directory = job_directory self.tmp_directory = tmp_directory self.home_directory = home_directory self.job_directory_type = job_directory_type # "galaxy" or "pulsar" class DependenciesDescription: """Capture (in a readily serializable way) context related a tool dependencies - both the tool's listed requirements and the tool shed related context required to resolve dependencies via the ToolShedPackageDependencyResolver. This is meant to enable remote resolution of dependencies, by the Pulsar or other potential remote execution mechanisms. """ def __init__(self, requirements=None, installed_tool_dependencies=None): requirements = requirements or ToolRequirements() if installed_tool_dependencies is None: installed_tool_dependencies = [] self.requirements = requirements # tool shed installed tool dependencies... self.installed_tool_dependencies = installed_tool_dependencies def to_dict(self): return dict( requirements=[r.to_dict() for r in self.requirements], installed_tool_dependencies=[ self._toolshed_install_dependency_to_dict(d) for d in self.installed_tool_dependencies ], ) @classmethod def from_dict(cls, as_dict): if as_dict is None: return None requirements_dicts = as_dict.get("requirements", []) requirements = ToolRequirements.from_list(requirements_dicts) installed_tool_dependencies_dicts = as_dict.get("installed_tool_dependencies", []) installed_tool_dependencies = list( map(cls._toolshed_install_dependency_from_dict, installed_tool_dependencies_dicts) ) return cls(requirements=requirements, installed_tool_dependencies=installed_tool_dependencies) @staticmethod def _toolshed_install_dependency_from_dict(as_dict): # Rather than requiring full models in Pulsar, just use simple objects # containing only properties and associations used to resolve # dependencies for tool execution. repository_object = bunch.Bunch( name=as_dict["repository_name"], owner=as_dict["repository_owner"], installed_changeset_revision=as_dict["repository_installed_changeset"], ) dependency_object = bunch.Bunch( name=as_dict["dependency_name"], version=as_dict["dependency_version"], type=as_dict["dependency_type"], tool_shed_repository=repository_object, ) return dependency_object @staticmethod def _toolshed_install_dependency_to_dict(tool_dependency): tool_shed_repository = tool_dependency.tool_shed_repository return dict( dependency_name=tool_dependency.name, dependency_version=tool_dependency.version, dependency_type=tool_dependency.type, repository_name=tool_shed_repository.name, repository_owner=tool_shed_repository.owner, repository_installed_changeset=tool_shed_repository.installed_changeset_revision, ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/docker_util.py0000644000175100017510000002031115234436334024531 0ustar00runnerrunner"""Utilities for building up Docker commands... ...using common defaults and configuration mechanisms. """ import os import shlex import sys from typing import ( List, Optional, TYPE_CHECKING, Union, ) if TYPE_CHECKING: from .container_volumes import DockerVolume from galaxy.util.commands import argv_to_str DEFAULT_DOCKER_COMMAND = "docker" DEFAULT_SUDO = False DEFAULT_SUDO_COMMAND = "sudo" DEFAULT_HOST = None DEFAULT_VOLUME_MOUNT_TYPE = "rw" DEFAULT_WORKING_DIRECTORY = None DEFAULT_NET = None DEFAULT_MEMORY = None DEFAULT_VOLUMES_FROM = None DEFAULT_AUTO_REMOVE = True DEFAULT_SET_USER = None if sys.platform == "darwin" else "$UID" DEFAULT_RUN_EXTRA_ARGUMENTS = None def kill_command(container: str, signal: Optional[str] = None, **kwds) -> List[str]: args = (["-s", signal] if signal else []) + [container] return command_list("kill", args, **kwds) def logs_command(container: str, **kwds) -> List[str]: return command_list("logs", [container], **kwds) def build_command(image: str, docker_build_path: str, **kwds) -> List[str]: if os.path.isfile(docker_build_path): docker_build_path = os.path.dirname(os.path.abspath(docker_build_path)) return command_list("build", ["-t", image, docker_build_path], **kwds) def build_save_image_command(image: str, destination: str, **kwds) -> List[str]: return command_list("save", ["-o", destination, image], **kwds) def build_pull_command(tag: str, **kwds) -> List[str]: return command_list("pull", [tag], **kwds) def build_docker_cache_command(image: str, **kwds) -> str: assert kwds.get("to_str", True) inspect_image_command = command_shell("inspect", [image], **kwds) pull_image_command = command_shell("pull", [image], **kwds) cache_command = f"{inspect_image_command} > /dev/null 2>&1\n[ $? -ne 0 ] && {pull_image_command} > /dev/null 2>&1\n" return cache_command def build_docker_images_command(truncate=True, format: Optional[str] = None, **kwds) -> Union[str, List[str]]: args = [] if not truncate: args.append("--no-trunc") if format: args.extend(["--format", format]) return command_shell("images", args, **kwds) def build_docker_load_command(**kwds) -> Union[str, List[str]]: return command_shell("load", []) def build_docker_simple_command( command: str, docker_cmd: str = DEFAULT_DOCKER_COMMAND, sudo: bool = DEFAULT_SUDO, sudo_cmd: str = DEFAULT_SUDO_COMMAND, container_name: Optional[str] = None, **kwd, ) -> str: command_parts = _docker_prefix( docker_cmd=docker_cmd, sudo=sudo, sudo_cmd=sudo_cmd, ) command_parts.append(command) command_parts.append(container_name or "{CONTAINER_NAME}") return " ".join(command_parts) def build_docker_run_command( container_command: str, image: str, interactive: bool = False, terminal: bool = False, tag: Optional[str] = None, volumes: Optional[List["DockerVolume"]] = None, volumes_from: Optional[str] = DEFAULT_VOLUMES_FROM, memory: Optional[str] = DEFAULT_MEMORY, env_directives: Optional[List[str]] = None, working_directory: Optional[str] = DEFAULT_WORKING_DIRECTORY, name: Optional[str] = None, net: Optional[str] = DEFAULT_NET, run_extra_arguments: Optional[str] = DEFAULT_RUN_EXTRA_ARGUMENTS, docker_cmd: str = DEFAULT_DOCKER_COMMAND, sudo: bool = DEFAULT_SUDO, sudo_cmd: str = DEFAULT_SUDO_COMMAND, auto_rm: bool = DEFAULT_AUTO_REMOVE, set_user: Optional[str] = DEFAULT_SET_USER, host: Optional[str] = DEFAULT_HOST, guest_ports: Union[bool, str, List[str]] = False, host_port_cmd: Optional[str] = None, container_name: Optional[str] = None, ) -> str: env_directives = env_directives or [] volumes = volumes or [] command_parts = _docker_prefix(docker_cmd=docker_cmd, sudo=sudo, sudo_cmd=sudo_cmd, host=host) command_parts.append("run") if interactive: command_parts.append("-i") if terminal: command_parts.append("-t") for env_directive in env_directives: # e.g. -e "GALAXY_SLOTS=$GALAXY_SLOTS" # These are environment variable expansions so we don't quote these. command_parts.extend(["-e", env_directive]) if guest_ports is True: # When is True, expose all ports command_parts.append("-P") elif guest_ports: if host_port_cmd: host_port_cmd = f"$({host_port_cmd}):" else: host_port_cmd = "" if not isinstance(guest_ports, list): guest_ports = [guest_ports] for guest_port in guest_ports: command_parts.extend(["-p", f"{host_port_cmd}{guest_port}"]) if container_name: command_parts.extend(["--name", container_name]) for volume in volumes: command_parts.extend(["-v", str(volume)]) if volumes_from: command_parts.extend(["--volumes-from", shlex.quote(str(volumes_from))]) if memory: command_parts.extend(["-m", shlex.quote(memory)]) command_parts.extend(["--cpus", "${GALAXY_SLOTS:-1}"]) if name: command_parts.extend(["--name", shlex.quote(name)]) if working_directory: command_parts.extend(["-w", shlex.quote(working_directory)]) if net: command_parts.extend(["--net", shlex.quote(net)]) if auto_rm: command_parts.append("--rm") if run_extra_arguments: command_parts.append(run_extra_arguments) if set_user: user = set_user if set_user == DEFAULT_SET_USER: # If future-us is ever in here and fixing this for docker-machine just # use cwltool.docker_id - it takes care of this default nicely. euid = os.geteuid() egid = os.getgid() user = f"{euid}:{egid}" command_parts.extend(["--user", user]) full_image = image if tag: full_image = f"{full_image}:{tag}" command_parts.append(shlex.quote(full_image)) command_parts.append(container_command) return " ".join(command_parts) def command_list(command: str, command_args: Optional[List[str]] = None, **kwds) -> List[str]: """Return Docker command as an argv list.""" command_args = command_args or [] command_parts = _docker_prefix(**kwds) command_parts.append(command) command_parts.extend(command_args) return command_parts def command_shell(command: str, command_args: Optional[List[str]] = None, **kwds) -> Union[str, List[str]]: """Return Docker command as a string for a shell or command-list.""" command_args = command_args or [] cmd = command_list(command, command_args, **kwds) to_str = kwds.get("to_str", True) if to_str: return argv_to_str(cmd) else: return cmd def _docker_prefix( docker_cmd: str = DEFAULT_DOCKER_COMMAND, sudo: bool = DEFAULT_SUDO, sudo_cmd: str = DEFAULT_SUDO_COMMAND, host: Optional[str] = DEFAULT_HOST, **kwds, ) -> List[str]: """Prefix to issue a docker command.""" command_parts = [] if sudo: command_parts.append(sudo_cmd) command_parts.append(docker_cmd) if host: command_parts.extend(["-H", host]) return command_parts def parse_port_text(port_text): """ >>> slurm_ports = parse_port_text("8888/tcp -> 0.0.0.0:32769") >>> slurm_ports[8888]['host'] '0.0.0.0' >>> ports = parse_port_text("5432/tcp -> :::5432") >>> len(ports) 0 >>> ports_new = parse_port_text("5432/tcp -> [::]:5432") >>> len(ports_new) 0 """ ports = None if port_text is not None: ports = {} for line in port_text.strip().split("\n"): if " -> " not in line: raise Exception(f"Cannot parse host and port from line [{line}]") tool, host = line.split(" -> ", 1) hostname, port = host.rsplit(":", 1) if hostname in ["::", "[::]"]: # Skip unspecified IPv6 address, which is also specified as 0:0:0:0 in another line. # This is brittle of course, but so is parsing the container ports like this. continue port = int(port) tool_p, tool_prot = tool.split("/") tool_p = int(tool_p) ports[tool_p] = dict(tool_port=tool_p, host=hostname, port=port, protocol=tool_prot) return ports ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/dockerfiles.py0000644000175100017510000000516115234436334024525 0ustar00runnerrunnerimport logging import os from galaxy.tool_util.deps import docker_util from galaxy.tool_util.deps.container_classes import docker_cache_path from galaxy.tool_util.deps.requirements import parse_requirements_from_xml from galaxy.tool_util.loader_directory import load_tool_elements_from_path from galaxy.util import commands log = logging.getLogger(__name__) def docker_host_args(**kwds): return dict( docker_cmd=kwds["docker_cmd"], sudo=kwds["docker_sudo"], sudo_cmd=kwds["docker_sudo_cmd"], host=kwds["docker_host"], ) def dockerfile_build(path, dockerfile=None, error=log.error, **kwds): expected_container_names = set() tool_directories = set() for tool_path, tool_xml in load_tool_elements_from_path(path): requirements, containers = parse_requirements_from_xml(tool_xml) for container in containers: if container.type == "docker": expected_container_names.add(container.identifier) tool_directories.add(os.path.dirname(tool_path)) break if len(expected_container_names) == 0: error("Could not find any docker identifiers to generate.") if len(expected_container_names) > 1: error("Multiple different docker identifiers found for selected tools [%s]", expected_container_names) image_identifier = expected_container_names.pop() dockerfile = __find_dockerfile(dockerfile, tool_directories) if dockerfile is not None: docker_command_parts = docker_util.build_command(image_identifier, dockerfile, **docker_host_args(**kwds)) else: docker_command_parts = docker_util.build_pull_command(image_identifier, **docker_host_args(**kwds)) commands.execute(docker_command_parts) commands.execute(docker_command_parts) docker_image_cache = kwds["docker_image_cache"] if docker_image_cache: destination = docker_cache_path(docker_image_cache, image_identifier) save_image_command_parts = docker_util.build_save_image_command( image_identifier, destination, **docker_host_args(**kwds) ) commands.execute(save_image_command_parts) def __find_dockerfile(dockerfile, tool_directories): if dockerfile is not None: return dockerfile search_directories = ["."] if len(tool_directories) == 1: tool_directory = tool_directories.pop() search_directories.insert(0, tool_directory) for directory in search_directories: potential_dockerfile = os.path.join(directory, "Dockerfile") if os.path.exists(potential_dockerfile): return potential_dockerfile return None ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/installable.py0000644000175100017510000000466615234436334024536 0ustar00runnerrunner"""Abstractions for installing local software managed and required by Galaxy/galaxy-lib.""" import abc import logging import os from galaxy.util.filelock import ( FileLock, FileLockException, ) log = logging.getLogger(__name__) class InstallableContext(metaclass=abc.ABCMeta): """Represent a directory/configuration of something that can be installed.""" @abc.abstractmethod def is_installed(self): """Return bool indicating if the configured software is installed.""" @abc.abstractmethod def can_install(self): """Check preconditions for installation.""" @property @abc.abstractmethod def installable_description(self): """Short description of thing being installed for log statements.""" @property @abc.abstractmethod def parent_path(self): """Return parent path of the location the installable will be created within.""" def ensure_installed(installable_context: InstallableContext, install_func, auto_init): """Make sure target is installed - handle multiple processes potentially attempting installation.""" parent_path = installable_context.parent_path desc = installable_context.installable_description def _check(): if not installable_context.is_installed(): if auto_init: if installable_context.can_install(): if install_func(installable_context): installed = False log.warning(f"{desc} installation requested and failed.") else: installed = installable_context.is_installed() if not installed: log.warning(f"{desc} installation requested, seemed to succeed, but not found.") else: installed = False else: installed = False log.warning("%s not installed and auto-installation disabled.", desc) else: installed = True return installed if not os.path.lexists(parent_path): os.mkdir(parent_path) try: if auto_init and os.access(parent_path, os.W_OK): with FileLock(os.path.join(parent_path, desc.lower()), timeout=300): return _check() else: return _check() except FileLockException: raise Exception(f"Failed to get file lock for {os.path.join(parent_path, desc.lower())}") ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1785871615.048229 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/0000755000175100017510000000000015234436377023147 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/__init__.py0000644000175100017510000000000015234436334025237 0ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/_cli.py0000644000175100017510000000064615234436334024426 0ustar00runnerrunner"""CLI helpers for mulled command-line tools.""" import argparse def arg_parser(argv, globals): """Build an argparser for this CLI tool.""" doc = globals["__doc__"] description, epilog = doc.split("\n", 1) parser = argparse.ArgumentParser( description=description, epilog=epilog, formatter_class=argparse.RawTextHelpFormatter, ) return parser __all__ = ("arg_parser",) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/get_tests.py0000644000175100017510000003201415234436334025513 0ustar00runnerrunner#!/usr/bin/env python """ searches for tests for packages in the bioconda-recipes repo and on Anaconda, looking in different file locations. If no test can be found for the specified version, it will look for tests for other versions of the same package. A shallow search (default for singularity and conda generation scripts) just checks once on Anaconda for the specified version. """ import json import logging import os.path from pathlib import Path from typing import ( Any, Dict, List, Optional, ) import yaml from galaxy.util import requests try: from jinja2 import Template from jinja2.exceptions import UndefinedError except ImportError: Template = None # type: ignore[assignment,misc] UndefinedError = Exception # type: ignore[assignment,misc] from galaxy.tool_util.deps.conda_util import ( best_search_result, CondaTarget, ) from galaxy.util import ( check_github_api_response_rate_limit, unicodify, ) from galaxy.util.commands import argv_to_str from .mulled_build import conda_platform from .util import ( CondaInDockerContext, get_files_from_conda_package, MULLED_SOCKET_TIMEOUT, split_container_name, ) INSTALL_JINJA_EXCEPTION = ( "This mulled functionality required jinja2 but it is unavailable, install condatesting extras." ) def get_commands_from_yaml(yaml_content: bytes) -> Optional[Dict[str, Any]]: """ Parse tests from Conda's meta.yaml file contents """ if Template is None: raise Exception(INSTALL_JINJA_EXCEPTION) package_tests = {} try: # we expect to get an input in bytes, so first decode to string; run the file through the jinja processing; load as yaml meta_yaml = yaml.safe_load(Template(yaml_content.decode("utf-8")).render()) except (yaml.scanner.ScannerError, UndefinedError) as e: # what about things like {{ compiler('cxx') }} logging.info(e, exc_info=True) return None try: if meta_yaml["test"]["commands"] != [None] and meta_yaml["test"]["commands"] is not None: package_tests["commands"] = meta_yaml["test"]["commands"] except (KeyError, TypeError): logging.info("Error reading commands") try: if meta_yaml["test"]["imports"] != [None] and meta_yaml["test"]["imports"] is not None: package_tests["imports"] = meta_yaml["test"]["imports"] except (KeyError, TypeError): logging.info("Error reading imports") if len(package_tests.get("commands", []) + package_tests.get("imports", [])) == 0: return None # need to know what scripting languages are needed to run the container package_tests["import_lang"] = "python -c" # python by default try: requirements = list(meta_yaml["requirements"]["run"]) except (KeyError, TypeError): logging.info("Error reading requirements", exc_info=True) else: for requirement in requirements: if requirement.split()[0] == "perl": package_tests["import_lang"] = "perl -e" break # elif ... : # other languages if necessary ... hopefully python and perl should suffice though return package_tests def get_run_test(file: str) -> Dict[str, Any]: r""" Get tests from a run_test.sh file """ package_tests = {} package_tests["commands"] = [file.replace("\n", " && ")] return package_tests def get_anaconda_url( container: str, anaconda_channel: str = "bioconda", conda_platform_str: Optional[str] = None ) -> str: """ Download tarball from anaconda for test """ if conda_platform_str is None: conda_platform_str = conda_platform() name = split_container_name(container) return f"https://anaconda.org/{anaconda_channel}/{name[0]}/{name[1]}/download/{conda_platform_str}/{'-'.join(name)}.tar.bz2" def get_test_from_anaconda(url: str) -> Optional[Dict[str, Any]]: """ Given the URL of an anaconda tarball, return tests """ content_dict = get_files_from_conda_package( url, ["info/recipe/meta.yaml", "info/recipe/meta.yaml.template", "info/recipe/run_test.sh"] ) content = content_dict.get("info/recipe/meta.yaml", content_dict.get("info/recipe/meta.yaml.template")) if content: package_tests = get_commands_from_yaml(content) if package_tests: return package_tests if "info/recipe/run_test.sh" in content_dict: return get_run_test(unicodify(content_dict["info/recipe/run_test.sh"])) return None def find_anaconda_download_url( name: str, version: str, build: Optional[str] = None, anaconda_channel: str = "bioconda", conda_platform_str: Optional[str] = None, ) -> Optional[str]: """ Find the anaconda download url for a given package. """ if conda_platform_str is None: conda_platform_str = conda_platform() r = requests.get( f"https://api.anaconda.org/package/{anaconda_channel}/{name}/files", timeout=MULLED_SOCKET_TIMEOUT, ) r.raise_for_status() package_files = r.json() for package_file in reversed(package_files): if ( package_file["version"] == version and (build is None or package_file["attrs"]["build"] == build) and package_file["attrs"]["subdir"] in [conda_platform_str, "noarch"] ): return f"https:{package_file['download_url']}" return None def open_recipe_file(file, recipes_path=None, github_repo="bioconda/bioconda-recipes"): """ Open a file at a particular location and return contents as string """ if recipes_path: return open(os.path.join(recipes_path, file)).read() else: # if no clone of the repo is available locally, download from GitHub r = requests.get( f"https://raw.githubusercontent.com/{github_repo}/master/{file}", timeout=MULLED_SOCKET_TIMEOUT, ) if r.status_code == 404: raise OSError else: return r.content def get_alternative_versions(filepath, filename, recipes_path=None, github_repo="bioconda/bioconda-recipes"): """ Return files that match ``filepath/*/filename`` in the bioconda-recipes repository """ if recipes_path: return [str(p.relative_to(recipes_path)) for p in Path(recipes_path).glob(f"{filepath}/*/{filename}")] # else use the GitHub API: versions = [] r = requests.get( f"https://api.github.com/repos/{github_repo}/contents/{filepath}", timeout=MULLED_SOCKET_TIMEOUT, ) check_github_api_response_rate_limit(r) r.raise_for_status() for subfile in json.loads(r.text): if subfile["type"] == "dir": if ( requests.get( f"https://raw.githubusercontent.com/{github_repo}/master/{subfile['path']}/{filename}", timeout=MULLED_SOCKET_TIMEOUT, ).status_code == 200 ): versions.append(f"{subfile['path']}/{filename}") return versions def try_a_func(func1, func2, param, container): """ Try to perform a function (or actually a combination of two functions: first getting the file and then processing it) """ try: result = func1(func2(*param)) except OSError: return None if result: result["container"] = container return result def deep_test_search( container: str, recipes_path: Optional[str] = None, anaconda_channel: str = "bioconda", github_repo: str = "bioconda/bioconda-recipes", conda_platform_str: Optional[str] = None, ) -> Dict[str, Any]: """ Look in bioconda-recipes repo as well as anaconda for the tests, checking in multiple possible locations. If no test is found for the specified version, search if other package versions have a test available. """ if conda_platform_str is None: conda_platform_str = conda_platform() name_tuple = split_container_name(container) assert len(name_tuple) in (2, 3) name = name_tuple[0] version = name_tuple[1] build = name_tuple[2] if len(name_tuple) == 3 else None for f in [ ( get_commands_from_yaml, open_recipe_file, (f"recipes/{name}/{version}/meta.yaml", recipes_path, github_repo), container, ), ( get_run_test, open_recipe_file, (f"recipes/{name}/{version}/run_test.sh", recipes_path, github_repo), container, ), ( get_commands_from_yaml, open_recipe_file, (f"recipes/{name}/meta.yaml", recipes_path, github_repo), container, ), (get_run_test, open_recipe_file, (f"recipes/{name}/run_test.sh", recipes_path, github_repo), container), (get_test_from_anaconda, get_anaconda_url, (container, anaconda_channel, conda_platform_str), container), ]: result = try_a_func(*f) if result: return result alt_versions = get_alternative_versions(f"recipes/{name}", "meta.yaml", recipes_path, github_repo) for alt_version in alt_versions: result = try_a_func( get_commands_from_yaml, open_recipe_file, (alt_version, recipes_path, github_repo), container ) if result: return result alt_versions = get_alternative_versions(f"recipes/{name}", "run_test.sh", recipes_path, github_repo) for alt_version in alt_versions: result = try_a_func(get_run_test, open_recipe_file, (alt_version, recipes_path, github_repo), container) if result: return result url = find_anaconda_download_url( name, version, build=build, anaconda_channel=anaconda_channel, conda_platform_str=conda_platform_str ) result = try_a_func(get_test_from_anaconda, lambda x: x, (url,), container) if result: return result # if everything fails return {"container": container} def main_test_search( container: str, recipes_path: Optional[str] = None, deep: bool = False, anaconda_channel: str = "bioconda", github_repo: str = "bioconda/bioconda-recipes", conda_platform_str: Optional[str] = None, ) -> Dict[str, Any]: """ Download tarball from anaconda for test """ if conda_platform_str is None: conda_platform_str = conda_platform() if deep: # do a deep search return deep_test_search(container, recipes_path, anaconda_channel, github_repo, conda_platform_str) # else shallow result = try_a_func( get_test_from_anaconda, get_anaconda_url, (container, anaconda_channel, conda_platform_str), container ) if result: return result return {"container": container} def import_test_to_command_list(import_lang: str, import_: str) -> List[str]: if import_lang == "python -c": return ["python", "-c", f"import {import_}"] elif import_lang == "perl -e": return ["perl", "-e", f"use {import_}"] else: raise ValueError(f"Unsupported import_lang '{import_lang}'") def hashed_test_search( container: str, recipes_path: Optional[str] = None, deep: bool = False, anaconda_channel: str = "bioconda", github_repo: str = "bioconda/bioconda-recipes", conda_platform_str: Optional[str] = None, ) -> Dict[str, Any]: """ Get test for hashed containers """ if conda_platform_str is None: conda_platform_str = conda_platform() package_tests: Dict[str, Any] = {"commands": [], "imports": [], "container": container, "import_lang": "python -c"} response = requests.get( f"https://raw.githubusercontent.com/BioContainers/multi-package-containers/master/combinations/{container}.tsv", timeout=MULLED_SOCKET_TIMEOUT, ) response.raise_for_status() for line in response.text.splitlines(): if not line.startswith("#"): break concatenated_targets = line.split("\t")[0] targets = concatenated_targets.split(",") packages = [target.split("=") for target in targets] conda_context = CondaInDockerContext(ensure_channels=[anaconda_channel]) containers = [] for package_name, package_version in packages: conda_target = CondaTarget(package_name, package_version) hit, exact = best_search_result(conda_target, conda_context, platform=conda_platform_str) if not hit or not exact: raise Exception(f"Could not find {conda_target}") build = hit["build"] if build: containers.append(f"{package_name}:{package_version}--{build}") else: containers.append(f"{package_name}:{package_version}") for container in containers: tests = main_test_search(container, recipes_path, deep, anaconda_channel, github_repo, conda_platform_str) package_tests["commands"] += tests.get("commands", []) # not a very nice solution but probably the simplest # Given that this could be a mix of Python and Perl packages, translate imports to commands for imp in tests.get("imports", []): package_tests["commands"].append(argv_to_str(import_test_to_command_list(tests["import_lang"], imp))) return package_tests ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/invfile.lua0000644000175100017510000001022715234436334025301 0ustar00runnerrunner-- http://stackoverflow.com/questions/19262761/lua-need-to-split-at-comma/19263313#19263313 function string:split( inSplitPattern, outResults ) if not outResults then outResults = { } end local theStart = 1 local theSplitStart, theSplitEnd = string.find( self, inSplitPattern, theStart ) while theSplitStart do table.insert( outResults, string.sub( self, theStart, theSplitStart-1 ) ) theStart = theSplitEnd + 1 theSplitStart, theSplitEnd = string.find( self, inSplitPattern, theStart ) end table.insert( outResults, string.sub( self, theStart ) ) return outResults end local repo = VAR.REPO local channel_args = '' local channels = VAR.CHANNELS:split(",") for i = 1, #channels do channel_args = channel_args .. " -c '" .. channels[i] .. "'" end local strict_channel_priority = VAR.STRICT_CHANNEL_PRIORITY if strict_channel_priority == '' then strict_channel_priority = '' else strict_channel_priority = '--strict-channel-priority' end local target_args = '' local targets = VAR.TARGETS:split(",") for i = 1, #targets do target_args = target_args .. " '" .. targets[i] .. "'" end local bind_args = {} local binds_table = VAR.BINDS:split(",") for i = 1, #binds_table do table.insert(bind_args, binds_table[i]) end local test_bind_args = {} local test_binds_table = VAR.TEST_BINDS:split(",") for i = 1, #test_binds_table do table.insert(test_bind_args, test_binds_table[i]) end local conda_image = VAR.CONDA_IMAGE if conda_image == '' then conda_image = 'quay.io/condaforge/miniforge3:latest' end local conda_bin = VAR.CONDA_BIN local singularity_image = VAR.SINGULARITY_IMAGE if singularity_image == '' then singularity_image = 'quay.io/singularity/singularity:v3.10.4' end local singularity_image_dir = VAR.SINGULARITY_IMAGE_DIR if singularity_image_dir == '' then singularity_image_dir = 'singularity_import' end local destination_base_image = VAR.DEST_BASE_IMAGE if destination_base_image == '' then destination_base_image = 'quay.io/bioconda/base-glibc-busybox-bash:latest' end local verbose = VAR.VERBOSE if verbose == '' then verbose = '--quiet' else verbose = '--verbose' end local preinstall = VAR.PREINSTALL if preinstall ~= '' then preinstall = preinstall .. ' && ' end local postinstall = VAR.POSTINSTALL if postinstall ~= '' then postinstall = '&&' .. postinstall end inv.task('build') .using(conda_image) .withHostConfig({binds = {"build:/data"}}) .run('rm', '-rf', '/data/dist') .using(conda_image) .withHostConfig({binds = bind_args}) .run('/bin/sh', '-c', preinstall .. conda_bin .. ' create ' .. channel_args .. ' ' .. strict_channel_priority .. ' ' .. target_args .. ' -p /usr/local --copy --yes ' .. verbose .. postinstall) .wrap('build/dist') .at('/usr/local') .inImage(destination_base_image) .as(repo) if VAR.SINGULARITY ~= '' then inv.task('singularity') .using(singularity_image) .withHostConfig({binds = {"build:/data", singularity_image_dir .. ":/import"}, privileged = true}) .withConfig({entrypoint = {'/bin/sh', '-c'}}) .run('mkdir -p /usr/local/var/singularity/mnt/container && ' .. 'singularity build /import/' .. VAR.SINGULARITY_IMAGE_NAME .. ' /import/Singularity.def && ' .. 'chown ' .. VAR.USER_ID .. ' /import/' .. VAR.SINGULARITY_IMAGE_NAME) end inv.task('cleanup') .using(conda_image) .withHostConfig({binds = {"build:/data"}}) .run('rm', '-rf', '/data/dist') if VAR.TEST_BINDS == '' then inv.task('test') .using(repo) .withConfig({entrypoint = {'/bin/sh', '-c'}}) .run(VAR.TEST) else inv.task('test') .using(repo) .withHostConfig({binds = test_bind_args}) .withConfig({entrypoint = {'/bin/sh', '-c'}}) .run(VAR.TEST) end inv.task('push') .push(repo) inv.task('build-and-test') .runTask('build') .runTask('singularity') .runTask('cleanup') .runTask('test') inv.task('all') .runTask('build') .runTask('singularity') .runTask('cleanup') .runTask('test') .runTask('push') ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/mulled_build.py0000644000175100017510000006424115234436334026162 0ustar00runnerrunner#!/usr/bin/env python """Build a mulled image for specified conda targets. Examples Build a mulled image with: mulled-build build 'samtools=1.3.1--4,bedtools=2.22' """ import json import logging import os import platform as _platform_module import shutil import stat import string import subprocess import sys from sys import platform as _platform from typing import ( Any, Callable, Dict, Iterable, List, NoReturn, Optional, TYPE_CHECKING, ) import yaml from typing_extensions import Literal from galaxy.tool_util.deps import installable from galaxy.tool_util.deps.conda_util import ( best_search_result, CondaContext, CondaTarget, split_version_build, ) from galaxy.util import ( commands, download_to_file, safe_makedirs, shlex_join, unicodify, ) from ._cli import arg_parser from .util import ( build_target, conda_build_target_str, CONDA_IMAGE, CondaInDockerContext, create_repository, DEFAULT_CHANNELS, get_files_from_conda_package, PrintProgress, quay_repository, v1_image_name, v2_image_name, ) from ..conda_compat import MetaData if TYPE_CHECKING: from galaxy.util.path import StrPath log = logging.getLogger(__name__) INVFILE = os.environ.get("INVFILE", os.path.join(os.path.dirname(__file__), "invfile.lua")) DEFAULT_BASE_IMAGE = os.environ.get("DEFAULT_BASE_IMAGE", "quay.io/bioconda/base-glibc-busybox-bash:latest") DEFAULT_EXTENDED_BASE_IMAGE = os.environ.get( "DEFAULT_EXTENDED_BASE_IMAGE", "quay.io/bioconda/base-glibc-debian-bash:latest" ) DEFAULT_REPOSITORY_TEMPLATE = "quay.io/${namespace}/${image}" DEFAULT_BINDS = ["build/dist:/usr/local/"] DEFAULT_WORKING_DIR = "/source/" IS_OS_X = _platform == "darwin" INVOLUCRO_VERSION = "1.2.0" DEST_BASE_IMAGE = os.environ.get("DEST_BASE_IMAGE", None) # Explicit Docker build targets use OCI platform notation. This is separate # from conda_platform(), which detects the native host OS and architecture. DockerPlatform = Literal[ "linux/amd64", "linux/arm64", "linux/arm/v7", "linux/ppc64le", "linux/riscv64", ] DOCKER_TO_CONDA_PLATFORM: Dict[DockerPlatform, str] = { "linux/amd64": "linux-64", "linux/arm64": "linux-aarch64", "linux/arm/v7": "linux-armv7l", "linux/ppc64le": "linux-ppc64le", "linux/riscv64": "linux-riscv64", } MACHINE_TO_DOCKER_PLATFORM: Dict[str, DockerPlatform] = { "x86_64": "linux/amd64", "amd64": "linux/amd64", "aarch64": "linux/arm64", "arm64": "linux/arm64", "armv7l": "linux/arm/v7", "ppc64le": "linux/ppc64le", "riscv64": "linux/riscv64", } SINGULARITY_TEMPLATE = """Bootstrap: docker From: %(base_image)s %%setup echo "Copying conda environment" mkdir -p /tmp/conda cp -r /data/dist/* /tmp/conda/ %%post rm -R /usr/local || true mkdir -p /usr/local cp -R /tmp/conda/* /usr/local/ %%test %(container_test)s """ def involucro_link(): url_start = f"https://github.com/involucro/involucro/releases/download/v{INVOLUCRO_VERSION}/" if IS_OS_X: url = f"{url_start}involucro.darwin" else: machine = _platform_module.machine() arch_map = { "x86_64": "involucro.linux-amd64", "amd64": "involucro.linux-amd64", "aarch64": "involucro.linux-arm64", "arm64": "involucro.linux-arm64", "armv7l": "involucro.linux-armv7", } asset = arch_map.get(machine, "involucro") url = f"{url_start}{asset}" return url def get_tests(args, pkg_path): """Extract test cases given a recipe's meta.yaml file.""" recipes_dir = args.recipes_dir tests = [] input_dir = os.path.dirname(os.path.join(recipes_dir, pkg_path)) recipe_meta = MetaData(input_dir) tests_commands = recipe_meta.get_value("test/commands") tests_imports = recipe_meta.get_value("test/imports") requirements = recipe_meta.get_value("requirements/run") if tests_imports or tests_commands: if tests_commands: tests.append(" && ".join(tests_commands)) if tests_imports and "python" in requirements: tests.append(" && ".join(f'python -c "import {imp}"' for imp in tests_imports)) elif tests_imports and ("perl" in requirements or "perl-threaded" in requirements): tests.append(" && ".join(f'''perl -e "use {imp};\"''' for imp in tests_imports)) tests = " && ".join(tests) tests = tests.replace("$R ", "Rscript ") return tests def get_pkg_name(args, pkg_path): """Extract the package name from a given meta.yaml file.""" recipes_dir = args.recipes_dir input_dir = os.path.dirname(os.path.join(recipes_dir, pkg_path)) recipe_meta = MetaData(input_dir) return recipe_meta.get_value("package/name") def get_affected_packages(args): """Return a list of all meta.yaml file that where modified/created recently. Length of time to check for indicated by the ``hours`` parameter. """ recipes_dir = args.recipes_dir hours = args.diff_hours cmd = ["git", "log", "--diff-filter=ACMRTUXB", "--name-only", '--pretty=""', f'--since="{hours} hours ago"'] changed_files = unicodify(subprocess.check_output(cmd, cwd=recipes_dir)).splitlines() pkg_list = {x for x in changed_files if x.startswith("recipes/") and x.endswith("meta.yaml")} for pkg in pkg_list: if pkg and os.path.exists(os.path.join(recipes_dir, pkg)): yield (get_pkg_name(args, pkg), get_tests(args, pkg)) def conda_versions(pkg_name, file_name): """Return all conda version strings for a specified package name.""" j = json.load(open(file_name)) ret = [] for pkg in j["packages"].values(): if pkg["name"] == pkg_name: ret.append(f"{pkg['version']}--{pkg['build']}") return ret def conda_platform() -> str: """Return the conda subdir for the native host OS and architecture.""" machine = _platform_module.machine().lower() if IS_OS_X: conda_arch_map = { "x86_64": "osx-64", "arm64": "osx-arm64", } default_platform = "osx-64" else: conda_arch_map = { "x86_64": "linux-64", "amd64": "linux-64", "aarch64": "linux-aarch64", "arm64": "linux-aarch64", "armv7l": "linux-armv7l", "ppc64le": "linux-ppc64le", } default_platform = "linux-64" return conda_arch_map.get(machine, default_platform) def docker_platform_to_conda_subdir(target_docker_platform: Optional[DockerPlatform]) -> str: """Return the conda subdir for an explicit Docker target, or for the host when unset.""" if target_docker_platform is None: return conda_platform() try: return DOCKER_TO_CONDA_PLATFORM[target_docker_platform] except KeyError: raise ValueError(f"Unsupported target platform '{target_docker_platform}'") from None def docker_platform_tag_suffix(target_platform: Optional[DockerPlatform]) -> Optional[str]: """Return an image-tag suffix, preserving unsuffixed tags for legacy amd64 images.""" target_platform = target_platform or MACHINE_TO_DOCKER_PLATFORM.get( _platform_module.machine().lower(), "linux/amd64" ) if target_platform == "linux/amd64": return None return target_platform[len("linux/") :].replace("/", "-") def apply_platform_tag_suffix(image: str, target_platform: Optional[DockerPlatform]) -> str: suffix = docker_platform_tag_suffix(target_platform) if suffix is None: return image last_slash = image.rfind("/") last_colon = image.rfind(":") if last_colon > last_slash: image_name, tag = image.rsplit(":", 1) else: image_name, tag = image, "latest" return f"{image_name}:{tag}-{suffix}" def get_conda_hits_for_targets( targets: Iterable[CondaTarget], conda_context: CondaContext, conda_platform_str: Optional[str] = None ) -> List[Dict[str, Any]]: platform = conda_platform_str or conda_platform() search_results = (best_search_result(t, conda_context, platform=platform)[0] for t in targets) return [r for r in search_results if r] def base_image_for_targets( targets: Iterable[CondaTarget], conda_context: CondaContext, conda_platform_str: Optional[str] = None ) -> str: """ determine base image (DEFAULT_BASE_IMAGE/DEFAULT_EXTENDED_BASE_IMAGE) for a list of targets by inspecting the conda package (i.e. if the use of an extended image is indicated in info/about.json or info/recipe/meta.yaml """ hits = get_conda_hits_for_targets(targets, conda_context, conda_platform_str) for hit in hits: try: content_dict = get_files_from_conda_package(hit["url"], ["info/about.json", "info/recipe/meta.yaml"]) if "info/about.json" in content_dict and json.loads(unicodify(content_dict["info/about.json"])).get( "extra", {} ).get("container", {}).get("extended-base", False): return DEFAULT_EXTENDED_BASE_IMAGE elif "info/recipe/meta.yaml" in content_dict and ( yaml.safe_load(unicodify(content_dict["info/recipe/meta.yaml"])) .get("extra", {}) .get("container", {}) .get("extended-base", False) ): return DEFAULT_EXTENDED_BASE_IMAGE except Exception: log.warning( "Could not load metadata.yaml for '%s', version '%s'", hit["name"], hit["version"], exc_info=True ) return DEFAULT_BASE_IMAGE class BuildExistsException(Exception): """Exception indicating mull_targets is skipping an existing build. If mull_targets is called with rebuild=False and the target built is already published an instance of this exception is thrown. """ def mull_targets( targets: List[CondaTarget], involucro_context: Optional["InvolucroContext"] = None, command: str = "build", channels: List[str] = DEFAULT_CHANNELS, namespace: str = "biocontainers", test: str = "true", test_files: Optional[List[str]] = None, image_build: Optional[str] = None, name_override: Optional[str] = None, repository_template: str = DEFAULT_REPOSITORY_TEMPLATE, dry_run: bool = False, conda_version: Optional[str] = None, mamba_version: Optional[str] = None, use_mamba: bool = False, verbose: bool = False, binds: List[str] = DEFAULT_BINDS, rebuild: bool = True, oauth_token: Optional[str] = None, hash_func: Literal["v1", "v2"] = "v2", singularity: bool = False, singularity_image_dir: "StrPath" = "singularity_import", base_image: Optional[str] = None, determine_base_image: bool = True, invfile: str = INVFILE, strict_channel_priority: bool = True, target_platform: Optional[DockerPlatform] = None, ) -> int: conda_platform_str = docker_platform_to_conda_subdir(target_platform) if singularity and target_platform: # Singularity build (invfile.lua:110-118) runs `singularity build` locally from a # .def file. Singularity has `--arch` only for remote builds (--remote); there is # no --platform flag for local builds. The resulting .sif is always the host arch, # so cross-platform builds would produce a .sif for the wrong architecture. raise ValueError("--target-platform cannot be used with --singularity") if involucro_context is None: involucro_context = InvolucroContext() image_function = v1_image_name if hash_func == "v1" else v2_image_name if len(targets) > 1 and image_build is None: # Force an image build in this case - this seems hacky probably # shouldn't work this way but single case broken else wise. image_build = "0" repo_template_kwds = { "namespace": namespace, "image": apply_platform_tag_suffix( image_function(targets, image_build=image_build, name_override=name_override), target_platform ), } repo = string.Template(repository_template).safe_substitute(repo_template_kwds) if not rebuild or "push" in command: repo_name = repo_template_kwds["image"].split(":", 1)[0] repo_data = quay_repository(repo_template_kwds["namespace"], repo_name) if not rebuild: tags = repo_data.get("tags", []) target_tag = None if ":" in repo_template_kwds["image"]: image_name_parts = repo_template_kwds["image"].split(":") assert len(image_name_parts) == 2, f": not allowed in image name [{repo_template_kwds['image']}]" target_tag = image_name_parts[1] if tags and (target_tag is None or target_tag in tags): raise BuildExistsException() if "push" in command and "error_type" in repo_data and oauth_token: # Explicitly create the repository so it can be built as public. create_repository(repo_template_kwds["namespace"], repo_name, oauth_token) for channel in channels: if channel.startswith("file://"): bind_path = channel[7:] binds.append(f"{bind_path}:{bind_path}") channels_str = ",".join(channels) target_str = ",".join(map(conda_build_target_str, targets)) bind_str = ",".join(binds) involucro_args = [ "-f", invfile, "-set", f"CHANNELS={channels_str}", "-set", f"TARGETS={target_str}", "-set", f"REPO={repo}", "-set", f"BINDS={bind_str}", ] if target_platform: involucro_args = ["--platform", target_platform] + involucro_args dest_base_image = None if base_image: dest_base_image = base_image elif DEST_BASE_IMAGE: dest_base_image = DEST_BASE_IMAGE elif determine_base_image: conda_context = CondaInDockerContext(ensure_channels=channels) dest_base_image = base_image_for_targets(targets, conda_context, conda_platform_str) if dest_base_image: involucro_args.extend(["-set", f"DEST_BASE_IMAGE={dest_base_image}"]) if CONDA_IMAGE: involucro_args.extend(["-set", f"CONDA_IMAGE={CONDA_IMAGE}"]) if verbose: involucro_args.extend(["-set", "VERBOSE=1"]) if singularity: singularity_image_name = repo_template_kwds["image"] involucro_args.extend(["-set", "SINGULARITY=1"]) involucro_args.extend(["-set", f"SINGULARITY_IMAGE_NAME={singularity_image_name}"]) involucro_args.extend(["-set", f"SINGULARITY_IMAGE_DIR={singularity_image_dir}"]) involucro_args.extend(["-set", f"USER_ID={os.getuid()}:{os.getgid()}"]) if test: involucro_args.extend(["-set", f"TEST={test}"]) verbose_opt = "--verbose" if verbose else "--quiet" specs: List[str] = [] if conda_version is not None: specs.append(f"conda={conda_version}") conda_bin = "conda" if use_mamba: conda_bin = "mamba" if mamba_version is not None: specs.append(f"mamba={mamba_version}") else: # For https://github.com/mamba-org/mamba/pull/3919 specs.append("mamba>=2.2.0") involucro_args.extend(["-set", f"CONDA_BIN={conda_bin}"]) if specs: conda_install = f"""conda install {verbose_opt} --yes {" ".join(f"'{spec}'" for spec in specs)}""" involucro_args.extend(["-set", f"PREINSTALL={conda_install}"]) if test_files: test_bind = [] for test_file in test_files: if ":" not in test_file: if os.path.exists(test_file): test_bind.append(f"{test_file}:{DEFAULT_WORKING_DIR}/{test_file}") else: if os.path.exists(test_file.split(":")[0]): test_bind.append(test_file) if test_bind: involucro_args.append("-set") involucro_args.append(f"TEST_BINDS={','.join(test_bind)}") if strict_channel_priority: involucro_args.extend(["-set", "STRICT_CHANNEL_PRIORITY=1"]) involucro_args.append(command) if dry_run: cmd = involucro_context.build_command(involucro_args) print(f"Executing: {shlex_join(cmd)}") return 0 ensure_installed(involucro_context, True) if singularity: if not os.path.exists(singularity_image_dir): safe_makedirs(singularity_image_dir) with open(os.path.join(singularity_image_dir, "Singularity.def"), "w+") as sin_def: fill_template = SINGULARITY_TEMPLATE % { "container_test": test, "base_image": dest_base_image or DEFAULT_BASE_IMAGE, } sin_def.write(fill_template) cmd = involucro_context.build_command(involucro_args) with PrintProgress(): ret = involucro_context.exec_command(involucro_args) if singularity: # we can not remove this folder as it contains the image wich is owned by root pass # shutil.rmtree('./singularity_import') return ret def context_from_args(args): verbose = "2" if not args.verbose else "3" return InvolucroContext(involucro_bin=args.involucro_path, verbose=verbose) class InvolucroContext(installable.InstallableContext): installable_description = "Involucro" def __init__( self, involucro_bin: Optional[str] = None, shell_exec: Optional[Callable[[List[str]], int]] = None, verbose: str = "3", ) -> None: if involucro_bin is None: if os.path.exists("./involucro"): self.involucro_bin = "./involucro" else: self.involucro_bin = "involucro" else: self.involucro_bin = involucro_bin self.shell_exec = shell_exec or commands.shell self.verbose = verbose def build_command(self, involucro_args: List[str]) -> List[str]: cmd = [self.involucro_bin, f"-v={self.verbose}"] return cmd + involucro_args def exec_command(self, involucro_args: List[str]) -> int: cmd = self.build_command(involucro_args) # Create ./build dir manually, otherwise Docker will do it as root created_build_dir = False if not os.path.exists("build"): created_build_dir = True os.mkdir("./build") try: res = self.shell_exec(cmd) finally: # delete build directory in any case if created_build_dir: shutil.rmtree("./build") return res def is_installed(self): return os.path.exists(self.involucro_bin) def can_install(self): return True @property def parent_path(self): return os.path.dirname(os.path.abspath(self.involucro_bin)) def ensure_installed(involucro_context, auto_init): return installable.ensure_installed(involucro_context, install_involucro, auto_init) def install_involucro(involucro_context): install_path = os.path.abspath(involucro_context.involucro_bin) involucro_context.involucro_bin = install_path try: download_to_file(involucro_link(), install_path) except Exception: log.exception(f"Failed to download involucro from url '{involucro_link()}'") return 1 try: os.chmod(install_path, os.stat(install_path).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH) return 0 except Exception: log.exception("Failed to make file '%s' executable", install_path) return 1 def add_build_arguments(parser): """Base arguments describing how to 'mull'.""" parser.add_argument( "--involucro-path", dest="involucro_path", default=None, help="Path to involucro (if not set will look in working directory and on PATH).", ) parser.add_argument( "--dry-run", dest="dry_run", action="store_true", help="Just print commands instead of executing them." ) parser.add_argument("--verbose", dest="verbose", action="store_true", help="Cause process to be verbose.") parser.add_argument( "--target-platform", dest="target_platform", default=None, choices=sorted(DOCKER_TO_CONDA_PLATFORM), help="Target platform for Docker/Involucro builds (e.g. 'linux/arm64'). Requires binfmt/QEMU support on the Docker daemon host.", ) parser.add_argument("--singularity", action="store_true", help="Additionally build a singularity image.") parser.add_argument( "--singularity-image-dir", dest="singularity_image_dir", help="Directory to write singularity images too." ) parser.add_argument("--involucro-lua-file", dest="invfile", default=INVFILE, help="Path to invfile.lua") parser.add_argument("-n", "--namespace", dest="namespace", default="biocontainers", help="quay.io namespace.") parser.add_argument( "-r", "--repository-template", dest="repository_template", default=DEFAULT_REPOSITORY_TEMPLATE, help="Docker repository target for publication (only quay.io or compat. API is currently supported).", ) parser.add_argument( "-c", "--channels", dest="channels", default=",".join(DEFAULT_CHANNELS), help="Comma separated list of target conda channels.", ) parser.add_argument( "--disable-strict-channel-priority", dest="strict_channel_priority", default=True, action="store_false", help="Disable strict channel priority. Will decrease speed of the resolver and should only be used in exceptional cases.", ) parser.add_argument( "--conda-version", dest="conda_version", default=None, help="Change to specified version of Conda before installing packages.", ) parser.add_argument( "--mamba-version", dest="mamba_version", default=None, help="Change to specified version of Mamba before installing packages.", ) parser.add_argument( "--use-mamba", dest="use_mamba", action="store_true", help="Use Mamba instead of Conda for package installation.", ) parser.add_argument( "--oauth-token", dest="oauth_token", default=None, help="If set, use this token when communicating with quay.io API.", ) parser.add_argument("--check-published", dest="rebuild", action="store_false") parser.add_argument("--hash", dest="hash", choices=["v1", "v2"], default="v2") def add_single_image_arguments(parser): parser.add_argument( "--name-override", dest="name_override", default=None, help="Override mulled image name - this is not recommended since metadata will not be detectable from the name of resulting images", ) parser.add_argument( "--image-build", dest="image_build", default=None, help="Build a versioned variant of this image." ) def target_str_to_targets(targets_raw: str) -> List[CondaTarget]: def parse_target(target_str: str) -> CondaTarget: if "=" in target_str: package_name, version_str = target_str.split("=", 1) version, build = split_version_build(version_str) target = build_target(package_name, version, build) else: target = build_target(target_str) return target if targets_raw.strip() == "": return [] else: return [parse_target(_) for _ in targets_raw.split(",")] def args_to_mull_targets_kwds(args): kwds = {} if hasattr(args, "image_build"): kwds["image_build"] = args.image_build if hasattr(args, "name_override"): kwds["name_override"] = args.name_override if hasattr(args, "namespace"): kwds["namespace"] = args.namespace if hasattr(args, "dry_run"): kwds["dry_run"] = args.dry_run if hasattr(args, "singularity"): kwds["singularity"] = args.singularity if hasattr(args, "test"): kwds["test"] = args.test if hasattr(args, "test_files"): if args.test_files: kwds["test_files"] = args.test_files.split(",") if hasattr(args, "channels"): kwds["channels"] = args.channels.split(",") if hasattr(args, "command"): kwds["command"] = args.command if hasattr(args, "repository_template"): kwds["repository_template"] = args.repository_template if hasattr(args, "conda_version"): kwds["conda_version"] = args.conda_version if hasattr(args, "mamba_version"): kwds["mamba_version"] = args.mamba_version if hasattr(args, "use_mamba"): kwds["use_mamba"] = args.use_mamba if hasattr(args, "oauth_token"): kwds["oauth_token"] = args.oauth_token if hasattr(args, "rebuild"): kwds["rebuild"] = args.rebuild if hasattr(args, "hash"): kwds["hash_func"] = args.hash if hasattr(args, "singularity_image_dir") and args.singularity_image_dir: kwds["singularity_image_dir"] = args.singularity_image_dir if hasattr(args, "invfile"): kwds["invfile"] = args.invfile if hasattr(args, "verbose"): kwds["verbose"] = args.verbose if hasattr(args, "target_platform"): kwds["target_platform"] = args.target_platform kwds["strict_channel_priority"] = args.strict_channel_priority kwds["involucro_context"] = context_from_args(args) return kwds def main(argv=None) -> NoReturn: """Main entry-point for the CLI tool.""" parser = arg_parser(argv, globals()) add_build_arguments(parser) add_single_image_arguments(parser) parser.add_argument("command", metavar="COMMAND", help="Command (build-and-test, build, all)") parser.add_argument( "targets", metavar="TARGETS", default=None, help="Build a single container with specific package(s)." ) parser.add_argument( "--repository-name", dest="repository_name", default=None, help="Name of mulled container (leave blank to auto-generate based on packages - recommended).", ) parser.add_argument("--test", help="Provide a test command for the container.") parser.add_argument( "--test-files", help="Provide test-files that may be required to run the test command. Individual mounts are separated by comma." "The source:dest docker syntax is respected. If relative file paths are given, files will be mounted in /source/", ) args = parser.parse_args() targets = target_str_to_targets(args.targets) sys.exit(mull_targets(targets, **args_to_mull_targets_kwds(args))) __all__ = ( "main", "build_target", ) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/mulled_build_channel.py0000644000175100017510000001065415234436334027651 0ustar00runnerrunner#!/usr/bin/env python """Build a mulled images for all recent conda recipe updates that don't have existing images. Examples: Build mulled images for recent bioconda changes with: mulled-build-channel build Build, test, and publish images with the follow command: mulled-build-channel all See recent changes that would be built with: mulled-build-channel list """ import os import subprocess import time from galaxy.util import requests from ._cli import arg_parser from .mulled_build import ( add_build_arguments, args_to_mull_targets_kwds, build_target, conda_versions, docker_platform_tag_suffix, docker_platform_to_conda_subdir, get_affected_packages, mull_targets, ) from .util import ( quay_versions, version_sorted, ) def _fetch_repo_data(args): repo_data = args.repo_data channel = args.channel if not os.path.exists(repo_data): platform_tag = docker_platform_to_conda_subdir(getattr(args, "target_platform", None)) subprocess.check_call( [ "wget", "--quiet", f"https://conda.anaconda.org/{channel}/{platform_tag}/repodata.json.bz2", "-O", f"{repo_data}.bz2", ] ) subprocess.check_call(["bzip2", "-d", f"{repo_data}.bz2"]) return repo_data def _unpublished_versions(quay, conda, platform_suffix=None): """Calculate the versions that are in conda but not on quay.io.""" squay = set(quay) if quay else set() if platform_suffix is None: return [v for v in conda if v not in squay] suffix = f"-{platform_suffix}" # Unsuffixed legacy tags represent amd64 builds and must not suppress # publication of the requested non-amd64 variant. return [v for v in conda if f"{v}{suffix}" not in squay] def run_channel(args, build_last_n_versions: int = 1) -> None: """Build list of involucro commands (as shell snippet) to run.""" session = requests.Session() for pkg_name, pkg_tests in get_affected_packages(args): repo_data = _fetch_repo_data(args) c = conda_versions(pkg_name, repo_data) # only package the most recent N versions c = version_sorted(c)[:build_last_n_versions] if not args.force_rebuild: time.sleep(1) q = quay_versions(args.namespace, pkg_name, session) versions = _unpublished_versions(q, c, docker_platform_tag_suffix(getattr(args, "target_platform", None))) else: versions = c for tag in versions: target = build_target(pkg_name, tag=tag) targets = [target] mull_targets(targets, test=pkg_tests, **args_to_mull_targets_kwds(args)) def get_pkg_names(args): """Print package names that would be affected.""" print("\n".join(pkg_name for pkg_name, pkg_tests in get_affected_packages(args))) def add_channel_arguments(parser): """Add arguments only used if running mulled over a whole conda channel.""" parser.add_argument( "--repo-data", dest="repo_data", required=True, help='Published repository data. If you want to build all containers for bioconda, this parameter needs to be set to "bioconda"', ) parser.add_argument( "--diff-hours", dest="diff_hours", default="25", help="If finding all recently changed recipes, use this number of hours.", ) parser.add_argument("--recipes-dir", dest="recipes_dir", default="./bioconda-recipes") parser.add_argument( "--force-rebuild", dest="force_rebuild", action="store_true", help="Rebuild package even if already published." ) def main(argv=None): """Main entry-point for the CLI tool.""" parser = arg_parser(argv, globals()) add_channel_arguments(parser) add_build_arguments(parser) parser.add_argument("command", metavar="COMMAND", help="Command (list, build-and-test, build, all)") parser.add_argument( "--targets", dest="targets", default=None, help="Build a single container with specific package(s)." ) parser.add_argument( "--repository-name", dest="repository_name", default=None, help="Name of a single container (leave blank to auto-generate based on packages).", ) args = parser.parse_args() if args.command == "list": get_pkg_names(args) else: run_channel(args) __all__ = ("main",) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/mulled_build_files.py0000644000175100017510000001004015234436334027330 0ustar00runnerrunner"""Build all composite mulled recipes discovered in TSV files. Use mulled-build-channel to build images for single recipes for a whole conda channel. This script instead builds images for combinations of recipes. This script can be given a single TSV file or a directory of TSV files to process. Examples: Build all recipes discovered in tsv files in a single directory. mulled-build-files build """ import sys from dataclasses import dataclass from pathlib import Path from typing import ( Any, Iterator, List, Optional, Sequence, ) from galaxy.tool_util.deps.conda_util import CondaTarget from ._cli import arg_parser from .mulled_build import ( add_build_arguments, args_to_mull_targets_kwds, BuildExistsException, mull_targets, target_str_to_targets, ) KNOWN_FIELDS = ["targets", "image_build", "name_override", "base_image"] FALLBACK_FIELD_ORDER = ("targets", "image_build", "name_override", "base_image") @dataclass class Target: targets: List[CondaTarget] image_build: Optional[str] name_override: Optional[str] base_image: Optional[str] def main(argv=None): """Main entry-point for the CLI tool.""" parser = arg_parser(argv, globals()) add_build_arguments(parser) parser.add_argument("command", metavar="COMMAND", help="Command (build-and-test, build, all)") parser.add_argument( "files", metavar="FILES", default=".", help="Path to directory (or single file) of TSV files describing composite recipes.", ) args = parser.parse_args() for target in generate_targets(args.files): try: ret = mull_targets( target.targets, image_build=target.image_build, name_override=target.name_override, base_image=target.base_image, determine_base_image=False, **args_to_mull_targets_kwds(args), ) except BuildExistsException: continue if ret > 0: sys.exit(ret) def generate_targets(target_source) -> Iterator[Target]: """Generate all targets from TSV files in specified file or directory.""" target_source = Path(target_source).resolve() if target_source.is_dir(): target_source_files = target_source.glob("*.tsv") else: target_source_files = [target_source] for target_source_file in target_source_files: # If no headers are defined we use the 4 default fields in the order # that has been used in galaxy-tool-util / galaxy-lib < 20.01 field_order: Sequence[str] = FALLBACK_FIELD_ORDER with open(target_source_file) as f: for line in f.readlines(): if line: line = line.strip() if line.startswith("#"): # headers can define a different column order field_order = field_order_from_header(line) else: yield line_to_targets(line, field_order) def field_order_from_header(header: str) -> List[str]: fields = header[1:].split("\t") for field in fields: assert field in KNOWN_FIELDS, f"'{field}' is not one of {KNOWN_FIELDS}" # Make sure tuple contains all fields for field in KNOWN_FIELDS: if field not in fields: fields.append(field) return fields def line_to_targets(line_str: str, field_order: Sequence[str]) -> Target: """Parse a line so that some columns can remain unspecified.""" line_parts: List[Any] = line_str.split("\t") n_fields = len(field_order) targets_column = field_order.index("targets") assert ( len(line_parts) <= n_fields ), f"Too many fields in line [{line_str}], expect at most {n_fields} - targets, image build number, and name override." line_parts += [None] * (n_fields - len(line_parts)) line_parts[targets_column] = target_str_to_targets(line_parts[targets_column]) return Target(**dict(zip(field_order, line_parts))) __all__ = ("main",) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/mulled_build_tool.py0000644000175100017510000000362615234436334027217 0ustar00runnerrunner#!/usr/bin/env python """Build a mulled images for a tool source (Galaxy or CWL tool). Examples: Build mulled images for requirements defined in a tool: mulled-build-tool build path/to/tool_file.xml """ from typing import ( List, TYPE_CHECKING, ) from galaxy.tool_util.parser import get_tool_source from ._cli import arg_parser from .mulled_build import ( add_build_arguments, add_single_image_arguments, args_to_mull_targets_kwds, mull_targets, target_str_to_targets, ) if TYPE_CHECKING: from galaxy.tool_util.deps.conda_util import CondaTarget from galaxy.util.path import StrPath def _mulled_build_tool(tool: "StrPath", args): tool_source = get_tool_source(tool) requirements, *_ = tool_source.parse_requirements() targets = requirements_to_mulled_targets(requirements) kwds = args_to_mull_targets_kwds(args) mull_targets(targets, **kwds) def main(argv=None) -> None: """Main entry-point for the CLI tool.""" parser = arg_parser(argv, globals()) add_build_arguments(parser) add_single_image_arguments(parser) parser.add_argument("command", metavar="COMMAND", help="Command (build-and-test, build, all)") parser.add_argument("tool", metavar="TOOL", default=None, help="Path to tool to build mulled image for.") args = parser.parse_args() _mulled_build_tool(args.tool, args) def requirements_to_mulled_targets(requirements) -> List["CondaTarget"]: """Convert Galaxy's representation of requirements into a list of CondaTarget objects. Only package requirements are retained. """ package_requirements = [r for r in requirements if r.type == "package"] target_str = ",".join(f"{r.name}={r.version}" if r.version else r.name for r in package_requirements) targets = target_str_to_targets(target_str) return targets __all__ = ("main", "requirements_to_mulled_targets") if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/mulled_hash.py0000644000175100017510000000254615234436334026006 0ustar00runnerrunner#!/usr/bin/env python """Produce a mulled hash for specified conda targets. Examples Produce a mulled hash with: mulled-hash samtools=1.3.1,bedtools=2.22 """ from typing_extensions import Literal from ._cli import arg_parser from .mulled_build import target_str_to_targets from .util import ( v1_image_name, v2_image_name, ) def _mulled_hash(hash: Literal["v1", "v2"], targets_str: str): """ >>> _mulled_hash("v2", "samtools=1.3.1,bedtools=2.26.0") 'mulled-v2-8186960447c5cb2faa697666dc1e6d919ad23f3e:a6419f25efff953fc505dbd5ee734856180bb619' >>> _mulled_hash("v2", "samtools=1.3.1=h9071d68_10,bedtools=2.26.0=0") 'mulled-v2-8186960447c5cb2faa697666dc1e6d919ad23f3e:a6419f25efff953fc505dbd5ee734856180bb619' """ targets = target_str_to_targets(targets_str) image_name = v2_image_name if hash == "v2" else v1_image_name return image_name(targets) def main(argv=None): """Main entry-point for the CLI tool.""" parser = arg_parser(argv, globals()) parser.add_argument( "targets", metavar="TARGETS", default=None, help="Comma-separated packages for calculating the mulled hash." ) parser.add_argument("--hash", dest="hash", choices=["v1", "v2"], default="v2") args = parser.parse_args() print(_mulled_hash(args.hash, args.targets)) __all__ = ("main",) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/mulled_list.py0000644000175100017510000001174115234436334026033 0ustar00runnerrunner#!/usr/bin/env python import argparse import logging from html.parser import HTMLParser from pathlib import Path from galaxy.util import requests from .util import MULLED_SOCKET_TIMEOUT QUAY_API_ENDPOINT = "https://quay.io/api/v1/repository" def get_quay_containers(repository="biocontainers"): """ Get all quay containers in the biocontainers repo """ containers = [] repos_parameters = {"public": "true", "namespace": repository} repos_headers = {"Accept-encoding": "gzip", "Accept": "application/json"} repos_response = requests.get(QUAY_API_ENDPOINT, headers=repos_headers, params=repos_parameters, timeout=12) repos = repos_response.json()["repositories"] for repo in repos: logging.info(repo) tags_response = requests.get(f"{QUAY_API_ENDPOINT}/{repository}/{repo['name']}", timeout=MULLED_SOCKET_TIMEOUT) tags = tags_response.json()["tags"] for tag in tags: containers.append(f"{repo['name']}:{tag}") return containers def get_singularity_containers(): """ Get all existing singularity containers from "https://depot.galaxyproject.org/singularity/" """ class GetContainerNames(HTMLParser): # small parser which gets list of containers def __init__(self): HTMLParser.__init__(self) self.containers = [] def handle_starttag(self, tag, attrs): try: for attr in attrs: if attr[0] == "href" and attr[1] != "../": self.containers.append(attr[1].replace("%3A", ":")) except IndexError: pass parser = GetContainerNames() index = requests.get("https://depot.galaxyproject.org/singularity/", timeout=MULLED_SOCKET_TIMEOUT) parser.feed(index.text) return parser.containers def get_conda_envs(filepath): """ Get list of already existing envs """ return [p.name.split("__")[-1].replace("@", ":") for p in Path(filepath).iterdir()] def get_missing_containers(quay_list, singularity_list, blocklist_file=None): r""" Return list of quay containers that do not exist as singularity containers. Files stored in a blocklist will be ignored """ blocklist = [] if blocklist_file: with open(blocklist_file) as fh: blocklist = fh.read().split("\n") return [n for n in quay_list if n not in singularity_list and n not in blocklist] def get_missing_envs(quay_list, conda_list, blocklist_file=None): r""" Compares list of conda envs and docker containers and returns missing conda envs """ blocklist = [] if blocklist_file: with open(blocklist_file) as fh: blocklist = fh.read().split("\n") return [n for n in quay_list if n.split("--")[0] not in conda_list and n.split("--")[0] not in blocklist] def main(): parser = argparse.ArgumentParser( description="Returns list of Docker containers in the quay.io biocontainers repository." ) parser.add_argument("--source", "-s", help="Docker, Singularity or Conda.") parser.add_argument( "--not-singularity", dest="not_singularity", action="store_true", help="Exclude Docker containers from which Singularity containers have already been built.", ) parser.add_argument( "--not-conda", dest="not_conda", action="store_true", help="Exclude Docker containers from which Conda environments have already been extracted.", ) parser.add_argument( "--conda-filepath", dest="conda_filepath", default=None, help="If searching for conda environments or employing the --not-conda option, a filepath where the environments are located.", ) parser.add_argument( "-b", "--blocklist", "--blacklist", dest="blocklist", default=None, help="Provide a 'blocklist file' containing containers which should be excluded from the list.", ) parser.add_argument( "-f", "--file", dest="output", default=None, help="File to write list to. If not given output will be returned on the command line.", ) args = parser.parse_args() if args.source == "docker": containers = get_quay_containers() if args.not_singularity: containers = get_missing_containers(containers, get_singularity_containers(), args.blocklist) if args.not_conda: containers = get_missing_envs(containers, get_conda_envs(args.conda_filepath), args.blocklist) elif args.source == "singularity": containers = get_singularity_containers() elif args.source == "conda": containers = get_conda_envs(args.conda_filepath) else: print("The 'source' argument was not understood.") return if args.output: with open(args.output, "a") as f: for container in containers: f.write(f"{container}\n") else: print(containers) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/mulled_search.py0000755000175100017510000004245015234436334026331 0ustar00runnerrunner#!/usr/bin/env python import argparse import json import logging import os import sys import tempfile import time from typing import ( Dict, List, ) from galaxy.tool_util.deps.conda_util import CondaContext from galaxy.util import ( check_github_api_response_rate_limit, requests, which, ) from .mulled_list import get_singularity_containers from .util import ( build_target, MULLED_SOCKET_TIMEOUT, v2_image_name, ) try: from whoosh.fields import ( Schema, STORED, TEXT, ) from whoosh.index import ( create_in, open_dir, ) from whoosh.qparser import QueryParser except ImportError: Schema = TEXT = STORED = create_in = QueryParser = None QUAY_API_URL = "https://quay.io/api/v1/repository" conda_path = which("conda") class QuaySearch: """ Tool to search within a quay organization for a given software name. """ tmp_dir_prefix = "mulled_search.quay." def __init__(self, organization, cache_time=900): self.index = None self.organization = organization self.cache_time = cache_time def paginator(self): # download all information about the repositories from the # given organization in self.organization json_decoder = json.JSONDecoder() parameters = {"public": "true", "namespace": self.organization} next_page = "" while next_page is not None: if next_page: parameters["next_page"] = next_page r = requests.get( QUAY_API_URL, headers={"Accept-encoding": "gzip"}, params=parameters, timeout=MULLED_SOCKET_TIMEOUT ) r.raise_for_status() decoded_request = json_decoder.decode(r.text) next_page = decoded_request.get("next_page") yield decoded_request def build_index(self): """ Create an index to quickly examine the repositories of a given quay.io organization. """ entries = [] uid = os.getuid() now = time.time() for entry in os.scandir(tempfile.gettempdir()): if not entry.name.startswith(QuaySearch.tmp_dir_prefix): continue if not entry.is_dir() or entry.is_symlink(): continue entry_st = entry.stat() if entry_st.st_uid != uid: continue if entry_st.st_mtime < (now - self.cache_time): continue entries.append((entry_st.st_mtime, entry.path)) if entries: tmp_dir = sorted(entries)[-1][1] print( f"Using existing quay.io index, remove or decrease --cache-time to rebuild: {tmp_dir}", file=sys.stderr ) self.index = open_dir(tmp_dir) else: tmp_dir = tempfile.mkdtemp(prefix=QuaySearch.tmp_dir_prefix) schema = Schema(title=TEXT(stored=True), content=STORED) self.index = create_in(tmp_dir, schema) writer = self.index.writer() for decoded_request in self.paginator(): for repository in decoded_request["repositories"]: writer.add_document(title=repository["name"], content=repository["description"]) writer.commit() def search_repository(self, search_string, non_strict): """ Search Docker containers on quay.io. Results are displayed with all available versions, including the complete image name. """ # with statement closes searcher after usage. with self.index.searcher() as searcher: query = QueryParser("title", self.index.schema).parse(search_string) results = searcher.search(query) if non_strict: # look for spelling errors and use suggestions as a search term too corrector = searcher.corrector("title") suggestions = corrector.suggest(search_string, limit=2) # get all repositories with suggested keywords for suggestion in suggestions: search_string = f"*{suggestion}*" query = QueryParser("title", self.index.schema).parse(search_string) results_tmp = searcher.search(query) results.extend(results_tmp) out = [] for result in results: title = result["title"] for version in self.get_additional_repository_information(title): out.append( { "package": title, "version": version, } ) return out def get_additional_repository_information(self, repository_string): """ Function downloads additional information from quay.io to get the tag-field which includes the version number. """ url = f"{QUAY_API_URL}/{self.organization}/{repository_string}" r = requests.get(url, headers={"Accept-encoding": "gzip"}, timeout=MULLED_SOCKET_TIMEOUT) json_decoder = json.JSONDecoder() decoded_request = json_decoder.decode(r.text) return decoded_request["tags"] class CondaSearch: """ Tool to search the bioconda channel """ def __init__(self, channel): self.channel = channel def get_json(self, search_string) -> List[Dict[str, str]]: """ Function takes search_string variable and returns results from the bioconda channel in JSON format """ if not conda_path: raise Exception("Invalid search destination. Required dependency [conda] is not in your PATH.") try: conda_context = CondaContext(conda_exec=conda_path, ensure_channels=self.channel) raw_out = conda_context.exec_search([search_string]) except Exception as e: logging.info(f"Search failed with: {e}") return [] header_found = False lines_fields: List[List[str]] = [] for line in raw_out.splitlines(): if line.startswith("#"): header_found = True elif header_found: lines_fields.append(line.split()) return [ {"package": line_fields[0], "version": line_fields[1], "build": line_fields[2]} for line_fields in lines_fields ] class GitHubSearch: """ Tool to search the GitHub bioconda-recipes repo """ def get_json(self, search_string): """ Takes search_string variable and return results from the bioconda-recipes github repository in JSON format DEPRECATED: this method is currently unreliable because the API query sometimes succeeds but returns no items. """ response = requests.get( f"https://api.github.com/search/code?q={search_string}+in:path+repo:bioconda/bioconda-recipes+path:recipes", timeout=MULLED_SOCKET_TIMEOUT, ) check_github_api_response_rate_limit(response) response.raise_for_status() return response.json() def process_json(self, json_response, search_string): """ Take JSON input and process it, returning the required data """ top_10_items = json_response["items"][0:10] # get top ten results return [{"name": result["name"], "path": result["path"]} for result in top_10_items] def recipe_present(self, search_string): """ Check if a recipe exists in bioconda-recipes which matches search_string exactly """ response = requests.get( f"https://api.github.com/repos/bioconda/bioconda-recipes/contents/recipes/{search_string}", timeout=MULLED_SOCKET_TIMEOUT, ) check_github_api_response_rate_limit(response) return response.status_code == 200 def get_package_hash(packages, versions): """ Take packages and versions (if the latter are given) and returns a hash for each. Also checks github to see if the container is already present. """ hash_results = {} targets = [] if versions: for p in packages: targets.append(build_target(p, version=versions[p])) else: # if versions are not given only calculate the package hash for p in packages: targets.append(build_target(p)) # make the hash from the processed targets package_hash = v2_image_name(targets) hash_results["package_hash"] = package_hash.split(":")[0] if versions: hash_results["version_hash"] = package_hash.split(":")[1] r = requests.get( f"https://quay.io/api/v1/repository/biocontainers/{hash_results['package_hash']}", timeout=MULLED_SOCKET_TIMEOUT ) if r.status_code == 200: hash_results["container_present"] = True if versions: # now test if the version hash is listed in the repository tags # remove -0, -1, etc from end of the tag tags = [n[:-2] for n in r.json()["tags"]] if hash_results["version_hash"] in tags: hash_results["container_present_with_version"] = True else: hash_results["container_present_with_version"] = False else: hash_results["container_present"] = False return hash_results def singularity_search(search_string): """ Check if a singularity package is present and return the link. """ results = [] containers = get_singularity_containers() for container in containers: if search_string in container: name = container.split(":")[0] version = container.split(":")[1] results.append({"package": name, "version": version}) return results def readable_output(json, organization="biocontainers", channel="bioconda"): # if json is empty: if sum(len(json[destination][results]) for destination in json for results in json[destination]) == 0: sys.stdout.write("No results found for that query.\n") return # return results for quay, conda and singularity together if ( sum( len(json[destination][results]) for destination in [ "quay", "conda", "singularity", ] for results in json.get(destination, []) ) > 0 ): sys.stdout.write("The query returned the following result(s).\n") # put quay, conda etc results as lists in lines lines = [["LOCATION", "NAME", "VERSION", "COMMAND\n"]] for results in json.get("quay", {}).values(): for result in results: lines.append( [ "quay", result["package"], result["version"], f"docker pull quay.io/{organization}/{result['package']}:{result['version']}\n", ] ) # NOT a real solution for results in json.get("conda", {}).values(): for result in results: lines.append( [ "conda", result["package"], f"{result['version']}--{result['build']}", f"conda install -c {channel} {result['package']}={result['version']}={result['build']}\n", ] ) for results in json.get("singularity", {}).values(): for result in results: lines.append( [ "singularity", result["package"], result["version"], f"wget https://depot.galaxyproject.org/singularity/{result['package']}:{result['version']}\n", ] ) col_width0, col_width1, col_width2 = ( max(len(line[n]) for line in lines) + 2 for n in (0, 1, 2) ) # def max col widths for the output # create table for line in lines: sys.stdout.write( "".join((line[0].ljust(col_width0), line[1].ljust(col_width1), line[2].ljust(col_width2), line[3])) ) # output if json.get("github_recipe_present", False): sys.stdout.write("\n" if "lines" in locals() else "") sys.stdout.write( "The following recipes were found in the bioconda-recipes repository which exactly matched one of the search terms:\n" ) lines = [["QUERY", "LOCATION\n"]] for recipe in json["github_recipe_present"]["recipes"]: lines.append( [recipe, f"https://api.github.com/repos/bioconda/bioconda-recipes/contents/recipes/{recipe}\n"] ) col_width0 = max(len(line[0]) for line in lines) + 2 for line in lines: sys.stdout.write("".join((line[0].ljust(col_width0), line[1]))) # output if sum(len(json["github"][results]) for results in json.get("github", [])) > 0: sys.stdout.write("\n" if "lines" in locals() else "") sys.stdout.write("Other result(s) on the bioconda-recipes GitHub repository:\n") lines = [["QUERY", "FILE", "URL\n"]] for search_string, results in json.get("github", {}).items(): for result in results: lines.append( [ search_string, result["name"], f"https://github.com/bioconda/bioconda-recipes/tree/master/{result['path']}\n", ] ) # def max col widths for the output col_width0, col_width1 = (max(len(line[n]) for line in lines) + 2 for n in (0, 1)) for line in lines: sys.stdout.write("".join((line[0].ljust(col_width0), line[1].ljust(col_width1), line[2]))) # output def deps_error_message(package): return f"Required dependency [{package}] is not installed. Run 'pip install galaxy-tool-util[mulled]'." def main(argv=None): if Schema is None: sys.stdout.write(deps_error_message("Whoosh")) return destination_defaults = ["quay", "singularity", "github"] if conda_path: destination_defaults.append("conda") parser = argparse.ArgumentParser(description="Searches in a given quay organization for a repository") parser.add_argument( "-d", "--destination", dest="search_dest", nargs="+", default=destination_defaults, help="Choose where to search. Options are 'conda', 'quay', 'singularity' and 'github'. If no option are given, all will be searched.", ) parser.add_argument( "-o", "--organization", dest="organization_string", default="biocontainers", help="Change quay organization to search; default is biocontainers.", ) parser.add_argument( "-c", "--channel", dest="channel_string", default="bioconda", help="Change conda channels to search; default is bioconda.", ) parser.add_argument( "--non-strict", dest="non_strict", action="store_true", help="Autocorrection of typos activated. Lists more results but can be confusing.\ For too many queries quay.io blocks the request and the results can be incomplete.", ) parser.add_argument("--cache-time", type=int, default=900, help="Number of seconds to reuse cached results for") parser.add_argument("-j", "--json", dest="json", action="store_true", help="Returns results as JSON.") parser.add_argument("-s", "--search", required=True, nargs="+", help="The name of the tool(s) to search for.") args = parser.parse_args() json_results = {dest: {} for dest in args.search_dest} versions = {} if len(args.search) > 1: # get hash if multiple packages are searched args.search.append(get_package_hash(args.search, versions)["package_hash"]) if "conda" in args.search_dest: conda_results = {} conda = CondaSearch(args.channel_string) for item in args.search: conda_results[item] = conda.get_json(item) json_results["conda"] = conda_results if "github" in args.search_dest: github_results = {} github_recipe_present = [] github = GitHubSearch() for item in args.search: if github.recipe_present(item): github_recipe_present.append(item) else: github_json = github.get_json(item) github_results[item] = github.process_json(github_json, item) json_results["github"] = github_results json_results["github_recipe_present"] = {"recipes": github_recipe_present} if "quay" in args.search_dest: quay_results = {} quay = QuaySearch(args.organization_string, cache_time=args.cache_time) quay.build_index() for item in args.search: quay_results[item] = quay.search_repository(item, args.non_strict) json_results["quay"] = quay_results if "singularity" in args.search_dest: singularity_results = {} for item in args.search: singularity_results[item] = singularity_search(item) json_results["singularity"] = singularity_results if args.json: print(json_results) else: readable_output(json_results, args.organization_string, args.channel_string) __all__ = ("main",) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/mulled_update_singularity_containers.py0000644000175100017510000002354115234436334033222 0ustar00runnerrunner#!/usr/bin/env python import argparse import os.path import subprocess import tempfile from glob import glob from subprocess import check_output from typing import ( Any, Dict, List, ) from galaxy.util import unicodify from galaxy.util.path import StrPath from .get_tests import ( hashed_test_search, import_test_to_command_list, main_test_search, ) def get_list_from_file(filename): """ Returns a list of containers stored in a file (one on each line) """ with open(filename) as fh: return [_ for _ in fh.read().splitlines() if _] # if blank lines are in the file empty strings must be removed def docker_to_singularity(container, installation, filepath, no_sudo=False): """ Convert docker to singularity container. """ cmd = [installation, "build", os.path.join(filepath, container), f"docker://quay.io/biocontainers/{container}"] try: if no_sudo: check_output(cmd, stderr=subprocess.STDOUT) else: check_output(cmd.insert(0, "sudo"), stderr=subprocess.STDOUT) check_output(["sudo", "rm", "-rf", "/root/.singularity/docker/"], stderr=subprocess.STDOUT) except subprocess.CalledProcessError as e: raise Exception(f"Docker to Singularity conversion failed.\nOutput was:\n{unicodify(e.output)}") def singularity_container_test( tests: Dict[str, Dict[str, Any]], installation: str, filepath: StrPath ) -> Dict[str, List]: """ Run tests, record if they pass or fail """ test_results: Dict[str, List] = {"passed": [], "failed": [], "notest": []} # create a 'sanitised home' directory in which the containers may be mounted - see http://singularity.lbl.gov/faq#solution-1-specify-the-home-to-mount with tempfile.TemporaryDirectory() as tmpdirname: for container, test in tests.items(): if "commands" not in test and "imports" not in test: test_results["notest"].append(container) else: exec_command = [installation, "exec", "-H", tmpdirname, os.path.join(filepath, container)] test_passed = True errors = [] for test_command in test.get("commands", []): test_command = test_command.replace("$PREFIX", "/usr/local/") test_command = test_command.replace("${PREFIX}", "/usr/local/") test_command = test_command.replace("$R ", "Rscript ") try: check_output(exec_command + ["bash", "-c", test_command], stderr=subprocess.STDOUT) except subprocess.CalledProcessError: try: check_output(exec_command + [test_command], stderr=subprocess.STDOUT) except subprocess.CalledProcessError as e: errors.append({"command": test_command, "output": unicodify(e.output)}) test_passed = False for imp in test.get("imports", []): try: check_output( exec_command + import_test_to_command_list(test["import_lang"], imp), stderr=subprocess.STDOUT, ) except subprocess.CalledProcessError as e: errors.append({"import": imp, "output": unicodify(e.output)}) test_passed = False if test_passed: test_results["passed"].append(container) else: test["errors"] = errors test_results["failed"].append(test) return test_results def main(): parser = argparse.ArgumentParser(description="Updates index of singularity containers.") parser.add_argument( "-c", "--containers", dest="containers", nargs="+", default=None, help="Containers to be generated. If the number of containers is large, it may be simpler to use the --containers-list option.", ) parser.add_argument( "-l", "--container-list", dest="container_list", default=None, help="Name of file containing list of containers to be generated. Alternative to --containers.", ) parser.add_argument( "-f", "--filepath", dest="filepath", help="File path where newly-built Singularity containers are placed." ) parser.add_argument("-i", "--installation", dest="installation", help="File path of Singularity installation.") parser.add_argument("--no-sudo", dest="no_sudo", action="store_true", help="Build containers without sudo.") parser.add_argument( "--testing", "-t", dest="testing", default=None, help="Performs testing automatically - a name for the output file should be provided. (Alternatively, testing may be done using the separate testing tool.", ) args = parser.parse_args() if args.containers: containers = args.containers elif args.container_list: containers = get_list_from_file(args.container_list) else: print("Either --containers or --container-list should be selected.") return for container in containers: docker_to_singularity(container, args.installation, args.filepath, args.no_sudo) if args.testing: container_testing( { "anaconda_channel": "bioconda", "installation": args.installation, "filepath": args.filepath, "github_repo": "bioconda/bioconda-recipes", "deep_search": False, "github_local_path": None, "logfile": args.testing, "containers": containers, } ) def container_testing(args=None): if not args: # i.e. if testing is called directly from CLI and not via main() parser = argparse.ArgumentParser(description="Tests.") parser.add_argument( "-c", "--containers", dest="containers", nargs="+", default=None, help="Containers to be tested. If the number of containers is large, it may be simpler to use the --containers-list option.", ) parser.add_argument( "-l", "--container-list", dest="container_list", default=None, help="Name of file containing list of containers to be tested. Alternative to --containers.", ) parser.add_argument( "-f", "--filepath", dest="filepath", help="File path where the containers to be tested are located." ) parser.add_argument( "-o", "--logfile", dest="logfile", default="singularity.log", help="Filename for a log to be written to." ) parser.add_argument("-i", "--installation", dest="installation", help="File path of Singularity installation.") parser.add_argument( "--deep-search", dest="deep_search", action="store_true", help="Perform a more extensive, but probably slower, search for tests.", ) parser.add_argument( "--anaconda-channel", dest="anaconda_channel", default="bioconda", help="Anaconda channel to search for tests (default: bioconda).", ) parser.add_argument( "--github-repo", dest="github_repo", default="bioconda/bioconda-recipes", help="Github repository to search for tests - only relevant if --deep-search is activated (default: bioconda/bioconda-recipes", ) parser.add_argument( "--github-local-path", dest="github_local_path", default=None, help="If the bioconda-recipes repository (or other repository containing tests) is available locally, provide the path here. Only relevant if --deep-search is activated.", ) args = vars(parser.parse_args()) if args["containers"]: containers = args["containers"] elif args["container_list"]: containers = get_list_from_file(args["container_list"]) else: # if no containers are specified, test everything in the filepath containers = [n.split(args["filepath"])[1] for n in glob(f"{args['filepath']}*")] with open(args["logfile"], "w") as f: f.write("SINGULARITY CONTAINERS GENERATED:") tests = {} for container in containers: if container[0:6] == "mulled": # if it is a 'hashed container' tests[container] = hashed_test_search( container, args["github_local_path"], args["deep_search"], args["anaconda_channel"], args["github_repo"], ) else: tests[container] = main_test_search( container, args["github_local_path"], args["deep_search"], args["anaconda_channel"], args["github_repo"], ) test_results = singularity_container_test(tests, args["installation"], args["filepath"]) f.write("\n\tTEST PASSED:") for container in test_results["passed"]: f.write(f"\n\t\t{container}") f.write("\n\tTEST FAILED:") for container in test_results["failed"]: f.write(f"\n\t\t{container['container']}") for error in container["errors"]: f.write( "\n\t\t\tCOMMAND: {}\n\t\t\t\tERROR:{}".format( error.get("command", f"import{error.get('import', 'nothing found')}"), error["output"] ) ) f.write("\n\tNO TEST AVAILABLE:") for container in test_results["notest"]: f.write(f"\n\t\t{container}") if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/recommend.py0000644000175100017510000004245015234436334025470 0ustar00runnerrunner#!/usr/bin/env python """Recommend the most appropriate biocontainer for a list of conda packages. Given a list of required packages (name + optional version) this resolves the best matching ``quay.io/biocontainers`` image, verified against quay.io rather than guessed. It reuses the existing mulled machinery (:mod:`.util` and :mod:`.mulled_search`) and is deliberately free of any LLM/agent dependency so it can be reused as a plain library or from the command line. Examples:: mulled-recommend samtools=1.17 mulled-recommend bwa,samtools mulled-recommend --json "samtools=1.3.1,bedtools=2.26.0" """ import argparse import itertools import json import logging import sys import threading import time from dataclasses import ( dataclass, field, ) from enum import Enum from typing import ( Any, Dict, List, Optional, Sequence, Tuple, ) from requests import ( HTTPError, RequestException, Session, ) from galaxy.tool_util.deps.conda_util import CondaTarget from galaxy.util import requests from .util import ( build_target, find_remote_mulled_name, mulled_tags_for, MulledNameMatch, select_mulled_v2_tag, split_tag, v2_image_name, ) log = logging.getLogger(__name__) BIOCONTAINERS_NAMESPACE = "biocontainers" QUAY_BIOCONTAINERS_PREFIX = "quay.io/biocontainers" RECOMMENDATION_CACHE_EXPIRY = 300 # seconds, mirrors QUAY_VERSIONS_CACHE_EXPIRY # Bounds for version resolution: how many recent versions to try per unpinned # package, and a hard cap on total combinations to test (keeps the search cheap # even with several unpinned packages). MAX_VERSION_CANDIDATES = 8 MAX_VERSION_COMBOS = 256 class RecommendationSource(str, Enum): QUAY_SINGLE = "quay_single" QUAY_MULLED_V2 = "quay_mulled_v2" NONE = "none" class MatchQuality(str, Enum): EXACT_VERSION = "exact_version" NAME_ONLY = "name_only" NOT_FOUND = "not_found" @dataclass(frozen=True) class PackageSpec: """A single required package: a name and an optional pinned version.""" name: str version: Optional[str] = None @dataclass(frozen=True) class ContainerRecommendation: """Result of a biocontainer lookup. ``image`` is a fully-qualified container reference (e.g. ``quay.io/biocontainers/samtools:1.17--h00cdaf9_0``) or ``None`` when nothing was found. ``match_quality`` distinguishes an exact-version hit from a name-only fallback; callers should only auto-apply an ``EXACT_VERSION`` recommendation. """ image: Optional[str] source: RecommendationSource match_quality: MatchQuality packages: Tuple[PackageSpec, ...] multi_package: bool tag: Optional[str] = None notes: Tuple[str, ...] = field(default_factory=tuple) @property def found(self) -> bool: return self.image is not None def _no_recommendation(packages: Sequence[PackageSpec], multi_package: bool, note: str) -> ContainerRecommendation: return ContainerRecommendation( image=None, source=RecommendationSource.NONE, match_quality=MatchQuality.NOT_FOUND, packages=tuple(packages), multi_package=multi_package, notes=(note,), ) # --- public entry point ----------------------------------------------------- # Per-process, bounded TTL memoization (no cross-worker sharing). Bounds repeated # identical lookups (e.g. refine round-trips); pass use_cache=False to bypass. class _TTLCache: """Minimal bounded TTL cache, hand-rolled so the lean ``galaxy-tool-util`` package needs no ``cachetools`` runtime dependency. Not thread-safe; guarded by ``_cache_lock``. """ def __init__(self, maxsize: int, ttl: float) -> None: self._maxsize = maxsize self._ttl = ttl self._entries: Dict[Any, Tuple[float, ContainerRecommendation]] = {} def get(self, key: Any) -> Optional[ContainerRecommendation]: entry = self._entries.get(key) if entry is None: return None expires_at, value = entry if time.monotonic() >= expires_at: self._entries.pop(key, None) return None return value def __setitem__(self, key: Any, value: ContainerRecommendation) -> None: if key not in self._entries and len(self._entries) >= self._maxsize: # Evict the entry closest to expiry to stay within ``maxsize``. soonest = min(self._entries, key=lambda k: self._entries[k][0]) self._entries.pop(soonest, None) self._entries[key] = (time.monotonic() + self._ttl, value) def clear(self) -> None: self._entries.clear() _cache = _TTLCache(maxsize=128, ttl=RECOMMENDATION_CACHE_EXPIRY) _cache_lock = threading.Lock() def _cache_key(packages: Sequence[PackageSpec]) -> Tuple[Tuple[str, str], ...]: return tuple(sorted((p.name, p.version or "") for p in packages)) def _normalize(packages: Sequence[PackageSpec]) -> List[PackageSpec]: normalized = [] for p in packages: name = p.name.strip().lower() version = p.version.strip() if p.version else None normalized.append(PackageSpec(name, version or None)) return normalized def recommend_container( packages: Sequence[PackageSpec], *, session: Optional[Session] = None, use_cache: bool = True, resolve_versions: bool = True, ) -> ContainerRecommendation: """Resolve the best ``quay.io/biocontainers`` image for ``packages``. For multi-package requests where some (or all) packages are unpinned, ``resolve_versions`` (default on) looks up candidate versions for the unpinned packages and searches for a version combination that is actually built, so a mixed request like ``samtools=1.17,bwa`` can still resolve to an exact mulled-v2 image instead of a name-only match. Set it false to skip the extra lookups. Never raises for a missing container or a transient network failure -- a ``NONE`` recommendation (with an explanatory note) is returned instead, so callers can degrade gracefully. """ normalized = _normalize([p for p in packages if p.name]) if not normalized: return _no_recommendation(normalized, False, "no packages supplied") key = (_cache_key(normalized), resolve_versions) if use_cache: with _cache_lock: hit = _cache.get(key) # None once the TTLCache entry has expired if hit is not None: return hit owns_session = session is None if session is None: session = requests.Session() try: if len(normalized) == 1: recommendation = _recommend_single(normalized[0], session) else: recommendation = _recommend_multi(normalized, session, resolve_versions=resolve_versions) except RequestException as e: log.info("biocontainer lookup failed for %s: %s", key, e) recommendation = _no_recommendation(normalized, len(normalized) > 1, f"lookup failed: {e}") finally: if owns_session: session.close() if use_cache: with _cache_lock: _cache[key] = recommendation return recommendation def biocontainer_tag_built(image: str, *, session: Optional[Session] = None) -> Optional[bool]: """Tri-state check of whether a ``quay.io/biocontainers`` image's exact tag is built. Returns: - ``True`` -- the tag is among the repository's built tags. - ``False`` -- the repository has tags, but not this one (the reference names a build/version that was never published, e.g. a hallucinated ``--py311h1128e8f_0`` build suffix). - ``None`` -- the answer can't be established: the reference isn't a ``quay.io/biocontainers`` image, it carries no tag, or the tag list couldn't be fetched (missing repo or transient network failure). Callers should only treat a ``False`` as grounds to override a container -- ``None`` is deliberately conservative so a network blip or a non-biocontainer image is never mistaken for a broken reference. Never raises. """ if not image: return None prefix = f"{QUAY_BIOCONTAINERS_PREFIX}/" if not image.startswith(prefix): return None ref = image[len(prefix) :] if ":" not in ref: return None repo, tag = ref.rsplit(":", 1) if not repo or not tag: return None owns_session = session is None if session is None: session = requests.Session() try: tags = mulled_tags_for(BIOCONTAINERS_NAMESPACE, repo, session=session) except (HTTPError, RequestException) as e: log.info("biocontainer tag verification failed for %s: %s", image, e) return None finally: if owns_session: session.close() # An empty list can't distinguish "repo absent" from a transient failure, so # don't claim the tag is missing -- only a populated list that excludes the # tag is positive evidence the reference is broken. if not tags: return None return tag in tags def _lookup_exact_or_newest(targets: List[CondaTarget], session: Session) -> Optional[MulledNameMatch]: """find_remote_mulled_name with the recommender's error handling. A 404 (HTTPError -- repo absent) is treated as "not found" (None); other network errors propagate to recommend_container's outer handler as a lookup failure. """ try: return find_remote_mulled_name( targets, BIOCONTAINERS_NAMESPACE, "v2", allow_newest_fallback=True, session=session ) except HTTPError: return None def _recommend_single(spec: PackageSpec, session: Session) -> ContainerRecommendation: match = _lookup_exact_or_newest([build_target(spec.name, version=spec.version)], session) if match is None: return _no_recommendation([spec], False, f"no biocontainer found for '{spec.name}'") tag = match.name.split(":", 1)[1] notes: Tuple[str, ...] = () if spec.version and not match.exact: notes = (f"requested version {spec.version} not found; newest available tag is {tag}",) return ContainerRecommendation( image=f"{QUAY_BIOCONTAINERS_PREFIX}/{match.name}", source=RecommendationSource.QUAY_SINGLE, match_quality=MatchQuality.EXACT_VERSION if match.exact else MatchQuality.NAME_ONLY, packages=(spec,), multi_package=False, tag=tag, notes=notes, ) def _recommend_multi( specs: List[PackageSpec], session: Session, resolve_versions: bool = True ) -> ContainerRecommendation: versioned = [p for p in specs if p.version] # Fully pinned: the shared resolver gives the exact image (or newest fallback) in one probe. if len(versioned) == len(specs): match = _lookup_exact_or_newest([build_target(p.name, version=p.version) for p in specs], session) if match is None: return _no_recommendation(specs, True, f"no multi-package biocontainer built for {[p.name for p in specs]}") notes = () if match.exact else ("exact version combination not built; using newest available mulled-v2 build",) return ContainerRecommendation( image=f"{QUAY_BIOCONTAINERS_PREFIX}/{match.name}", source=RecommendationSource.QUAY_MULLED_V2, match_quality=MatchQuality.EXACT_VERSION if match.exact else MatchQuality.NAME_ONLY, packages=tuple(specs), multi_package=True, tag=match.name.split(":", 1)[1], notes=notes, ) # Partial / unpinned: the mulled-v2 repo depends only on package names. Fetch its tags once, # then (optionally) resolve the unpinned versions against the tags that are actually built. repo = v2_image_name([build_target(p.name) for p in specs]).split(":")[0] try: tags = mulled_tags_for(BIOCONTAINERS_NAMESPACE, repo, session=session) except HTTPError: tags = [] if not tags: return _no_recommendation(specs, True, f"no multi-package biocontainer built for {[p.name for p in specs]}") notes_list: List[str] = [] version_hash: Optional[str] = None if resolve_versions: resolved, version_hash = _resolve_versions(specs, tags, session) if resolved: unpinned = [p.name for p in specs if not p.version] notes_list.append("resolved version(s): " + ", ".join(f"{n}={resolved[n]}" for n in unpinned)) else: notes_list.append("could not resolve a built version combination; using newest available build") elif versioned: notes_list.append("not all packages were versioned; resolved by package names only") if version_hash: tag, _ = select_mulled_v2_tag(tags, version_hash, allow_newest_fallback=True) quality = MatchQuality.EXACT_VERSION else: tag, quality = tags[0], MatchQuality.NAME_ONLY return ContainerRecommendation( image=f"{QUAY_BIOCONTAINERS_PREFIX}/{repo}:{tag}", source=RecommendationSource.QUAY_MULLED_V2, match_quality=quality, packages=tuple(specs), multi_package=True, tag=tag, notes=tuple(notes_list), ) def _candidate_versions(name: str, session: Session, limit: int) -> List[str]: """Recent biocontainer versions for a single package, newest first.""" try: tags = mulled_tags_for(BIOCONTAINERS_NAMESPACE, name, session=session) except (HTTPError, RequestException): return [] versions: List[str] = [] for tag in tags: version = split_tag(tag)[0] if version not in versions: versions.append(version) if len(versions) >= limit: break return versions def _resolve_versions( specs: List[PackageSpec], existing_tags: List[str], session: Session, ) -> Tuple[Optional[Dict[str, str]], Optional[str]]: """Find a version combination whose mulled-v2 image is actually built. Pinned versions are fixed; unpinned packages contribute their recent versions. Returns the full ``{name: version}`` combination and its version-hash for the first combination present in ``existing_tags``, or ``(None, None)`` if none of the tried combinations is built. """ existing_hashes = {tag.rsplit("-", 1)[0] for tag in existing_tags if "-" in tag} if not existing_hashes: return None, None candidate_lists: List[List[str]] = [] for spec in specs: if spec.version: candidate_lists.append([spec.version]) else: candidate_lists.append(_candidate_versions(spec.name, session, MAX_VERSION_CANDIDATES)) if any(not candidates for candidates in candidate_lists): return None, None # itertools.product yields the all-newest combination first. for count, combo in enumerate(itertools.product(*candidate_lists)): if count >= MAX_VERSION_COMBOS: break targets = [build_target(spec.name, version=version) for spec, version in zip(specs, combo)] version_hash = v2_image_name(targets).split(":")[1] if version_hash in existing_hashes: return {spec.name: version for spec, version in zip(specs, combo)}, version_hash return None, None # --- CLI -------------------------------------------------------------------- def _specs_from_arg(targets_raw: str) -> List[PackageSpec]: # Lazy import: mulled_build pulls in heavy build tooling. Keeping it out of # module scope lets the agent import path stay lightweight; it's only needed # for the CLI's spec parsing. from .mulled_build import target_str_to_targets return [PackageSpec(t.package, t.version) for t in target_str_to_targets(targets_raw)] def main(argv: Optional[List[str]] = None) -> None: """Main entry-point for the ``mulled-recommend`` CLI tool.""" parser = argparse.ArgumentParser( description="Recommend the best quay.io/biocontainers image for a list of conda packages." ) parser.add_argument( "targets", metavar="TARGETS", help="Comma-separated packages, each 'name' or 'name=version' (e.g. 'samtools=1.17,bedtools').", ) parser.add_argument("-j", "--json", dest="as_json", action="store_true", help="Emit the recommendation as JSON.") parser.add_argument( "--no-resolve-versions", dest="resolve_versions", action="store_false", help="Don't look up versions for unpinned packages in multi-package requests.", ) args = parser.parse_args(argv) recommendation = recommend_container( _specs_from_arg(args.targets), use_cache=False, resolve_versions=args.resolve_versions ) if args.as_json: print( json.dumps( { "image": recommendation.image, "source": recommendation.source.value, "match_quality": recommendation.match_quality.value, "multi_package": recommendation.multi_package, "tag": recommendation.tag, "notes": list(recommendation.notes), } ) ) return if recommendation.found: print(recommendation.image) print(f" match: {recommendation.match_quality.value} (source: {recommendation.source.value})") else: print("No biocontainer found.", file=sys.stderr) for note in recommendation.notes: print(f" note: {note}", file=sys.stderr) __all__ = ( "ContainerRecommendation", "MatchQuality", "PackageSpec", "RecommendationSource", "main", "recommend_container", ) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/mulled/util.py0000644000175100017510000005470515234436334024502 0ustar00runnerrunner"""Utilities for working with mulled abstractions outside the mulled package.""" import collections import hashlib import logging import os import re import sys import threading from typing import ( Any, Callable, Dict, Iterable, List, Literal, NamedTuple, Optional, Tuple, TYPE_CHECKING, Union, ) from conda_package_streaming.package_streaming import stream_conda_info from conda_package_streaming.url import stream_conda_info as stream_conda_info_from_url from packaging.version import Version from requests import Session from galaxy.tool_util.deps.conda_util import ( CondaContext, CondaTarget, ) from galaxy.tool_util.deps.docker_util import command_list as docker_command_list from galaxy.tool_util.version import ( LegacyVersion, parse_version, ) from galaxy.util import requests if TYPE_CHECKING: from galaxy.tool_util.deps.container_resolvers import ResolutionCache log = logging.getLogger(__name__) QUAY_REPOSITORY_API_ENDPOINT = "https://quay.io/api/v1/repository" BUILD_NUMBER_REGEX = re.compile(r"\d+$") MULLED_SOCKET_TIMEOUT = 12 QUAY_VERSIONS_CACHE_EXPIRY = 300 NAMESPACE_HAS_REPO_NAME_KEY = "galaxy.tool_util.deps.container_resolvers.mulled.util:namespace_repo_names" TAG_CACHE_KEY = "galaxy.tool_util.deps.container_resolvers.mulled.util:tag_cache" CONDA_IMAGE = os.environ.get("CONDA_IMAGE", "quay.io/condaforge/miniforge3:latest") class PARSED_TAG(NamedTuple): tag: str version: Union[LegacyVersion, Version] build_string: Union[LegacyVersion, Version] build_number: int def default_mulled_conda_channels_from_env() -> Optional[List[str]]: if "DEFAULT_MULLED_CONDA_CHANNELS" in os.environ: return os.environ["DEFAULT_MULLED_CONDA_CHANNELS"].split(",") else: return None DEFAULT_CHANNELS = default_mulled_conda_channels_from_env() or ["conda-forge", "bioconda"] class CondaInDockerContext(CondaContext): def __init__( self, conda_prefix: Optional[str] = None, conda_exec: Optional[Union[str, List[str]]] = None, shell_exec: Optional[Callable[..., int]] = None, debug: bool = False, ensure_channels: Union[str, List[str]] = DEFAULT_CHANNELS, condarc_override: Optional[str] = None, ): if not conda_exec: binds = [] for channel in ensure_channels: if channel.startswith("file://"): bind_path = channel[7:] binds.extend(["-v", f"{bind_path}:{bind_path}"]) conda_exec = docker_command_list("run", binds + [CONDA_IMAGE, "conda"]) super().__init__( conda_prefix=conda_prefix, conda_exec=conda_exec, shell_exec=shell_exec, debug=debug, ensure_channels=ensure_channels, condarc_override=condarc_override, ) def create_repository(namespace: str, repo_name: str, oauth_token: str) -> None: assert oauth_token headers = {"Authorization": f"Bearer {oauth_token}"} data = { "repository": repo_name, "namespace": namespace, "description": "", "visibility": "public", } response = requests.post(QUAY_REPOSITORY_API_ENDPOINT, json=data, headers=headers, timeout=MULLED_SOCKET_TIMEOUT) response.raise_for_status() def quay_versions(namespace: str, pkg_name: str, session: Optional[Session] = None) -> List[str]: """Get all version tags for a Docker image stored on quay.io for supplied package name.""" data = quay_repository(namespace, pkg_name, session=session) if "error_type" in data and data["error_type"] == "invalid_token": return [] if "tags" not in data: raise Exception(f"Unexpected response from quay.io - no tags description found [{data}]") return [tag for tag in data["tags"].keys() if tag != "latest"] def quay_repository(namespace: str, pkg_name: str, session: Optional[Session] = None) -> Dict[str, Any]: assert namespace is not None assert pkg_name is not None url = f"{QUAY_REPOSITORY_API_ENDPOINT}/{namespace}/{pkg_name}" if not session: session = requests.Session() response = session.get(url, timeout=MULLED_SOCKET_TIMEOUT) response.raise_for_status() data = response.json() return data def _get_namespace(namespace: str) -> List[str]: log.debug(f"Querying {QUAY_REPOSITORY_API_ENDPOINT} for repos within {namespace}") next_page = None repo_names = [] repos_headers = {"Accept-encoding": "gzip", "Accept": "application/json"} while True: repos_parameters = {"public": "true", "namespace": namespace, "next_page": next_page} repos_response = requests.get( QUAY_REPOSITORY_API_ENDPOINT, headers=repos_headers, params=repos_parameters, timeout=MULLED_SOCKET_TIMEOUT ) repos_response.raise_for_status() repos_response_json = repos_response.json() repos = repos_response_json["repositories"] repo_names += [r["name"] for r in repos] next_page = repos_response_json.get("next_page") if not next_page: break return repo_names def _namespace_has_repo_name(namespace: str, repo_name: str, resolution_cache: "ResolutionCache") -> bool: """ Get all quay containers in the biocontainers repo """ # resolution_cache.mulled_resolution_cache is the persistent variant of the resolution cache preferred_resolution_cache = resolution_cache.mulled_resolution_cache or resolution_cache cache_key = NAMESPACE_HAS_REPO_NAME_KEY if preferred_resolution_cache is not None: try: cached_namespace = preferred_resolution_cache.get(cache_key) if cached_namespace: return repo_name in cached_namespace except KeyError: # preferred_resolution_cache may be a beaker Cache instance, which # raises KeyError if key is not present on `.get` pass repo_names = _get_namespace(namespace) if preferred_resolution_cache is not None: preferred_resolution_cache[cache_key] = repo_names return repo_name in repo_names def mulled_tags_for( namespace: str, image: str, tag_prefix: Optional[str] = None, resolution_cache: Optional["ResolutionCache"] = None, session: Optional[Session] = None, expire: float = QUAY_VERSIONS_CACHE_EXPIRY, ) -> List[str]: """Fetch remote tags available for supplied image name. The result will be sorted so newest tags are first. """ if resolution_cache is not None: # Following check is pretty expensive against biocontainers... don't even bother doing it # if can't cache the response. if not _namespace_has_repo_name(namespace, image, resolution_cache): log.info(f"skipping mulled_tags_for [{image}] no repository") return [] cache_key = TAG_CACHE_KEY if resolution_cache is not None: if resolution_cache.mulled_resolution_cache is not None: # Use persistent cache if possible. Since tags query is lightweight use a relatively short expiry time. resolution_cache = resolution_cache.mulled_resolution_cache._get_cache( "mulled_tag_cache", {"expire": expire} ) assert resolution_cache is not None if cache_key not in resolution_cache: resolution_cache[cache_key] = collections.defaultdict(dict) tag_cache = resolution_cache.get(cache_key) else: tag_cache = collections.defaultdict(dict) tags_cached = False try: tags = tag_cache[namespace][image] tags_cached = True except KeyError: pass if not tags_cached: tags = quay_versions(namespace, image, session) tag_cache[namespace][image] = tags if tag_prefix is not None: tags = [t for t in tags if t.startswith(tag_prefix)] tags = version_sorted(tags) return tags def split_tag(tag: str) -> List[str]: """Split mulled image tag into conda version and conda build.""" return tag.rsplit("--", 1) def parse_tag(tag: str) -> PARSED_TAG: """Decompose tag of mulled images into version, build string and build number.""" version = tag.rsplit(":")[-1] build_string = "-1" build_number = -1 match = BUILD_NUMBER_REGEX.search(version) if match: build_number = int(match.group(0)) if "--" in version: version, build_string = version.rsplit("--", 1) elif "-" in version: # Should be mulled multi-container image tag version, build_string = version.rsplit("-", 1) else: # We don't have a build number, and the BUILD_NUMBER_REGEX above is only accurate for build strings, # so set build number to -1. Any matching image:version combination with a build number # will be considered newer. build_number = -1 return PARSED_TAG( tag=tag, version=parse_version(version), build_string=parse_version(build_string), build_number=build_number, ) def version_sorted(elements: Iterable[str]) -> List[str]: """Sort iterable based on loose description of "version" from newest to oldest.""" parsed_tags_iter = (parse_tag(tag) for tag in elements) sorted_tags = sorted(parsed_tags_iter, key=lambda tag: tag.build_string, reverse=True) sorted_tags = sorted(sorted_tags, key=lambda tag: tag.build_number, reverse=True) sorted_tags = sorted(sorted_tags, key=lambda tag: tag.version, reverse=True) return [e.tag for e in sorted_tags] def build_target( package_name: str, version: Optional[str] = None, build: Optional[str] = None, tag: Optional[str] = None ) -> CondaTarget: """Use supplied arguments to build a :class:`CondaTarget` object.""" if tag is not None: assert version is None assert build is None version, build = split_tag(tag) # conda package and quay image names are lowercase return CondaTarget(package_name, version=version, build=build) def conda_build_target_str(target: CondaTarget) -> str: rval = target.package if target.version: rval += f"={target.version}" if target.build: rval += f"={target.build}" return rval def _simple_image_name(targets: List[CondaTarget], image_build: Optional[str] = None) -> str: target = targets[0] suffix = "" if target.version is not None: build = target.build if build is None and image_build is not None and image_build != "0": # Special case image_build == "0", which has been built without a suffix print("WARNING: Hard-coding image build instead of using Conda build - this is not recommended.") build = image_build suffix += f":{target.version}" if build is not None: suffix += f"--{build}" return f"{target.package}{suffix}" def v1_image_name( targets: Iterable[CondaTarget], image_build: Optional[str] = None, name_override: Optional[str] = None ) -> str: """Generate mulled hash version 1 container identifier for supplied arguments. If a single target is specified, simply use the supplied name and version as the repository name and tag respectively. If multiple targets are supplied, hash the package names and versions together as the repository name. For mulled version 1 containers the image build is the repository tag (if supplied). >>> single_targets = [build_target("samtools", version="1.3.1")] >>> v1_image_name(single_targets) 'samtools:1.3.1' >>> multi_targets = [build_target("samtools", version="1.3.1"), build_target("bwa", version="0.7.13")] >>> v1_image_name(multi_targets) 'mulled-v1-b06ecbd9141f0dbbc0c287375fc0813adfcbdfbd' >>> multi_targets_on_versionless = [build_target("samtools", version="1.3.1"), build_target("bwa")] >>> v1_image_name(multi_targets_on_versionless) 'mulled-v1-bda945976caa5734347fbf7f35066d9f58519e0c' >>> multi_targets_versionless = [build_target("samtools"), build_target("bwa")] >>> v1_image_name(multi_targets_versionless) 'mulled-v1-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40' """ if name_override is not None: print( "WARNING: Overriding mulled image name, auto-detection of 'mulled' package attributes will fail to detect result." ) return name_override targets = list(targets) if len(targets) == 1: return _simple_image_name(targets, image_build=image_build) else: targets_order = sorted(targets, key=lambda t: t.package) requirements_buffer = "\n".join(map(conda_build_target_str, targets_order)) m = hashlib.sha1() m.update(requirements_buffer.encode()) suffix = "" if not image_build else f":{image_build}" return f"mulled-v1-{m.hexdigest()}{suffix}" def v2_image_name( targets: Iterable[CondaTarget], image_build: Optional[str] = None, name_override: Optional[str] = None ) -> str: """Generate mulled hash version 2 container identifier for supplied arguments. If a single target is specified, simply use the supplied name and version as the repository name and tag respectively. If multiple targets are supplied, hash the package names as the repository name and hash the package versions (if set) as the tag. >>> single_targets = [build_target("samtools", version="1.3.1")] >>> v2_image_name(single_targets) 'samtools:1.3.1' >>> single_targets = [build_target("samtools", version="1.3.1", build="py_1")] >>> v2_image_name(single_targets) 'samtools:1.3.1--py_1' >>> single_targets = [build_target("samtools", version="1.3.1")] >>> v2_image_name(single_targets, image_build="0") 'samtools:1.3.1' >>> single_targets = [build_target("samtools", version="1.3.1", build="py_1")] >>> v2_image_name(single_targets, image_build="0") 'samtools:1.3.1--py_1' >>> multi_targets = [build_target("samtools", version="1.3.1"), build_target("bwa", version="0.7.13")] >>> v2_image_name(multi_targets) 'mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:4d0535c94ef45be8459f429561f0894c3fe0ebcf' >>> multi_targets_on_versionless = [build_target("samtools", version="1.3.1"), build_target("bwa")] >>> v2_image_name(multi_targets_on_versionless) 'mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:b0c847e4fb89c343b04036e33b2daa19c4152cf5' >>> multi_targets_versionless = [build_target("samtools"), build_target("bwa")] >>> v2_image_name(multi_targets_versionless) 'mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40' >>> targets_version_with_build = [build_target("samtools", version="1.3.1", build="h9071d68_10"), build_target("bedtools", version="2.26.0", build="0")] >>> v2_image_name(targets_version_with_build) 'mulled-v2-8186960447c5cb2faa697666dc1e6d919ad23f3e:a6419f25efff953fc505dbd5ee734856180bb619' """ if name_override is not None: print( "WARNING: Overriding mulled image name, auto-detection of 'mulled' package attributes will fail to detect result." ) return name_override targets = list(targets) if len(targets) == 1: return _simple_image_name(targets, image_build=image_build) else: targets_order = sorted(targets, key=lambda t: t.package) package_name_buffer = "\n".join(t.package for t in targets_order) package_hash = hashlib.sha1() package_hash.update(package_name_buffer.encode()) versions = (t.version for t in targets_order) if any(versions): # Only hash versions if at least one package has versions... version_name_buffer = "\n".join(t.version or "null" for t in targets_order) version_hash = hashlib.sha1() version_hash.update(version_name_buffer.encode()) version_hash_str = version_hash.hexdigest() else: version_hash_str = "" if not image_build: build_suffix = "" elif version_hash_str: # tagged verson is - build_suffix = f"-{image_build}" else: # tagged version is simply the build build_suffix = image_build suffix = "" if version_hash_str or build_suffix: suffix = f":{version_hash_str}{build_suffix}" return f"mulled-v2-{package_hash.hexdigest()}{suffix}" class MulledNameMatch(NamedTuple): """A resolved remote mulled image name and whether it matched the request exactly. ``name`` is the unnamespaced image name (e.g. ``samtools:1.17--h0_0`` or ``mulled-v2-:-0``). ``exact`` is True when the tag matched the requested version / version-hash, False when it is a newest-available fallback. """ name: str exact: bool def select_single_package_tag( tags: List[str], version: Optional[str], *, allow_newest_fallback: bool = False ) -> Tuple[Optional[str], bool]: """Pick the best tag for a single-package repo (``tags`` newest-first). Returns ``(tag, exact)``: an exact version match when ``version`` is found, otherwise the newest tag with ``exact=False`` if ``allow_newest_fallback`` else ``(None, False)``. """ if not tags: return None, False if version is not None: for tag in tags: if split_tag(tag)[0] == version: return tag, True if allow_newest_fallback: return tags[0], False return None, False def select_mulled_v2_tag( tags: List[str], version_hash: Optional[str], *, allow_newest_fallback: bool = False ) -> Tuple[Optional[str], bool]: """Pick the best tag for a ``mulled-v2`` repo (``tags`` newest-first). A mulled-v2 tag is ``-``. With a ``version_hash`` an exact match is the newest tag carrying that hash; without one (e.g. v1 / unversioned) the newest tag *is* the canonical result (``exact=True`` -- there is no finer version to mismatch). When a requested ``version_hash`` isn't built, returns the newest tag with ``exact=False`` if ``allow_newest_fallback`` else ``(None, False)``. """ if not tags: return None, False if version_hash: for tag in tags: if tag == version_hash or tag.startswith(f"{version_hash}-"): return tag, True if allow_newest_fallback: return tags[0], False return None, False return tags[0], True def find_remote_mulled_name( targets: List[CondaTarget], namespace: str, hash_func: Literal["v1", "v2"] = "v2", *, allow_newest_fallback: bool = False, resolution_cache: Optional["ResolutionCache"] = None, session: Optional[Session] = None, ) -> Optional[MulledNameMatch]: """Resolve conda targets to a remote quay mulled image name (unnamespaced). Single target: the repo is the package name, matched by version. Multiple targets: the repo and version-hash come from :func:`v1_image_name` / :func:`v2_image_name`, matched by version-hash. With ``allow_newest_fallback`` the newest available tag is returned (with ``exact=False``) when there's no exact match; otherwise ``None``. Network errors from :func:`mulled_tags_for` propagate to the caller. """ if len(targets) == 1: target = targets[0] tags = mulled_tags_for(namespace, target.package, resolution_cache=resolution_cache, session=session) tag, exact = select_single_package_tag(tags, target.version, allow_newest_fallback=allow_newest_fallback) return MulledNameMatch(f"{target.package}:{tag}", exact) if tag is not None else None if hash_func == "v2": base_image_name = v2_image_name(targets) elif hash_func == "v1": base_image_name = v1_image_name(targets) else: raise Exception(f"Unimplemented mulled hash_func [{hash_func}]") if ":" in base_image_name: repo_name, version_hash = base_image_name.split(":", 1) else: repo_name, version_hash = base_image_name, None tags = mulled_tags_for(namespace, repo_name, resolution_cache=resolution_cache, session=session) tag, exact = select_mulled_v2_tag(tags, version_hash, allow_newest_fallback=allow_newest_fallback) return MulledNameMatch(f"{repo_name}:{tag}", exact) if tag is not None else None def get_files_from_conda_package(url: str, filepaths: Iterable[str]) -> Dict[str, bytes]: """ Get content of specified files in a conda package. The url can be a path to a local file or an url. The filepaths is an iterable of paths to extract from the conda package, if found in it. Return a dictionary mapping each found filepath to the corresponding content (as bytes). >>> content_dict = get_files_from_conda_package("https://anaconda.org/conda-forge/chopin2/1.0.6/download/noarch/chopin2-1.0.6-pyhd8ed1ab_0.tar.bz2", ["info/recipe/meta.yaml"]) >>> assert "info/recipe/meta.yaml" in content_dict, content_dict >>> assert isinstance(content_dict["info/recipe/meta.yaml"], bytes) >>> content_dict = get_files_from_conda_package("https://anaconda.org/conda-forge/chopin2/1.0.7/download/noarch/chopin2-1.0.7-pyhd8ed1ab_1.conda", ["info/about.json", "info/recipe/meta.yaml", "foo/bar"]) >>> assert sorted(content_dict.keys()) == ["info/about.json", "info/recipe/meta.yaml"], content_dict """ try: stream = stream_conda_info(url) except FileNotFoundError: stream = stream_conda_info_from_url(url) ret = {} for tar, member in stream: if member.name in filepaths: ret[member.name] = tar.extractfile(member).read() return ret def split_container_name(name: str) -> List[str]: """ Takes a container name (e.g. samtools:1.7--1) and returns a list (e.g. ['samtools', '1.7', '1']) >>> split_container_name('samtools:1.7--1') ['samtools', '1.7', '1'] """ return name.replace("--", ":").split(":") class PrintProgress: def __init__(self) -> None: self.thread = threading.Thread(target=self.progress) self.stop = threading.Event() def progress(self) -> None: while not self.stop.is_set(): print(".", end="") sys.stdout.flush() self.stop.wait(60) print("") def __enter__(self) -> "PrintProgress": self.thread.start() return self def __exit__(self, exc_type, exc_val, exc_tb) -> None: self.stop.set() self.thread.join() image_name = v1_image_name # deprecated __all__ = ( "build_target", "CONDA_IMAGE", "conda_build_target_str", "CondaInDockerContext", "DEFAULT_CHANNELS", "get_files_from_conda_package", "image_name", "mulled_tags_for", "quay_versions", "split_container_name", "split_tag", "v1_image_name", "v2_image_name", "version_sorted", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/requirements.py0000644000175100017510000004420115234436334024754 0ustar00runnerrunnerimport copy import os from typing import ( Any, Callable, cast, Dict, Iterable, Iterator, List, Optional, Tuple, Union, ) from typing_extensions import ( get_args, Literal, ) from galaxy.tool_util_models.tool_source import JavascriptRequirement from galaxy.util import ( asbool, string_as_bool, xml_text, ) from galaxy.util.oset import OrderedSet DEFAULT_REQUIREMENT_TYPE = "package" DEFAULT_REQUIREMENT_VERSION = None class ToolRequirement: """ Represents an external requirement that must be available for the tool to run (for example, a program, package, or library). Requirements can optionally assert a specific version. """ def __init__( self, name: str, type: Optional[str] = None, version: Optional[str] = None, specs: Optional[Iterable["RequirementSpecification"]] = None, ) -> None: if specs is None: specs = [] self.name = name self.type = type self.version = version self.specs = specs def to_dict(self) -> Dict[str, Any]: specs = [s.to_dict() for s in self.specs] return dict(name=self.name, type=self.type, version=self.version, specs=specs) def copy(self) -> "ToolRequirement": return copy.deepcopy(self) @classmethod def from_dict(cls, d: Dict[str, Any]) -> "ToolRequirement": version = d.get("version") name = d["name"] type = d.get("type") specs = [RequirementSpecification.from_dict(s) for s in d.get("specs", [])] return cls(name=name, type=type, version=version, specs=specs) def __eq__(self, other: Any) -> bool: return ( self.name == other.name and self.type == other.type and self.version == other.version and self.specs == other.specs ) def __hash__(self) -> int: return hash((self.name, self.type, self.version, frozenset(self.specs))) def __str__(self) -> str: return f"ToolRequirement[{self.name},version={self.version},type={self.type},specs={self.specs}]" __repr__ = __str__ class RequirementSpecification: """Refine a requirement using a URI.""" def __init__(self, uri: str, version: Optional[str] = None) -> None: self.uri = uri self.version = version @property def specifies_version(self) -> bool: return self.version is not None @property def short_name(self) -> str: return self.uri.split("/")[-1] def to_dict(self) -> Dict[str, Any]: return dict(uri=self.uri, version=self.version) @classmethod def from_dict(cls, dict: Dict) -> "RequirementSpecification": uri = dict["uri"] version = dict.get("version", None) return cls(uri=uri, version=version) def __eq__(self, other: Any) -> bool: return self.uri == other.uri and self.version == other.version def __hash__(self) -> int: return hash((self.uri, self.version)) class ToolRequirements: """ Represents all requirements (packages, env vars) needed to run a tool. """ def __init__(self, tool_requirements: Optional[List[Union[ToolRequirement, Dict[str, Any]]]] = None) -> None: if tool_requirements: if not isinstance(tool_requirements, list): raise ToolRequirementsException("ToolRequirements Constructor expects a list") self.tool_requirements = OrderedSet( r if isinstance(r, ToolRequirement) else ToolRequirement.from_dict(r) for r in tool_requirements ) else: self.tool_requirements = OrderedSet() @classmethod def from_list(cls, requirements: List[Union[ToolRequirement, Dict[str, Any]]]) -> "ToolRequirements": return cls(requirements) @property def resolvable(self) -> "ToolRequirements": return ToolRequirements([r for r in self.tool_requirements if r.type in {"package", "set_environment"}]) @property def packages(self) -> "ToolRequirements": return ToolRequirements([r for r in self.tool_requirements if r.type == "package"]) def to_list(self): return [r.to_dict() for r in self.tool_requirements] def append(self, requirement: Union[ToolRequirement, Dict[str, Any]]) -> None: if not isinstance(requirement, ToolRequirement): requirement = ToolRequirement.from_dict(requirement) self.tool_requirements.add(requirement) def __eq__(self, other: Any) -> bool: return ( len(self.tool_requirements & other.tool_requirements) == len(self.tool_requirements) == len(other.tool_requirements) ) def __iter__(self) -> Iterator[ToolRequirement]: yield from self.tool_requirements def __getitem__(self, ii) -> ToolRequirement: return list(self.tool_requirements)[ii] def __len__(self) -> int: return len(self.tool_requirements) def __hash__(self) -> int: return sum(r.__hash__() for r in self.tool_requirements) def to_dict(self) -> List[Dict[str, Any]]: return [r.to_dict() for r in self.tool_requirements] class ToolRequirementsException(Exception): pass DEFAULT_CONTAINER_TYPE = "docker" DEFAULT_CONTAINER_RESOLVE_DEPENDENCIES = False DEFAULT_CONTAINER_SHELL = "/bin/sh" # Galaxy assumes bash, but containers are usually thinner. class ContainerDescription: def __init__( self, identifier: str, type: str = DEFAULT_CONTAINER_TYPE, resolve_dependencies: bool = DEFAULT_CONTAINER_RESOLVE_DEPENDENCIES, shell: str = DEFAULT_CONTAINER_SHELL, ) -> None: # Force to lowercase because container image names must be lowercase. # Cached singularity images include the path on disk, so only lowercase # the image identifier portion. Note, the tag can also contain upper case # letters. self.identifier = identifier if identifier: parts = identifier.rsplit(os.sep, 1) if ":" in parts[-1]: name, tag = parts[-1].rsplit(":", 1) parts[-1] = f"{name.lower()}:{tag}" else: parts[-1] = parts[-1].lower() self.identifier = os.sep.join(parts) self.type = type self.resolve_dependencies = resolve_dependencies self.shell = shell self.explicit = False def to_dict(self, *args, **kwds) -> Dict[str, Any]: return dict( identifier=self.identifier, type=self.type, resolve_dependencies=self.resolve_dependencies, shell=self.shell, ) @classmethod def from_dict(cls, dict: Dict[str, Any]) -> "ContainerDescription": identifier = dict["identifier"] type = dict.get("type", DEFAULT_CONTAINER_TYPE) resolve_dependencies = dict.get("resolve_dependencies", DEFAULT_CONTAINER_RESOLVE_DEPENDENCIES) shell = dict.get("shell", DEFAULT_CONTAINER_SHELL) return cls( identifier=identifier, type=type, resolve_dependencies=resolve_dependencies, shell=shell, ) def __str__(self) -> str: return f"ContainerDescription[identifier={self.identifier},type={self.type}]" ResourceType = Literal[ "cores_min", "cores_max", "ram_min", "ram_max", "tmpdir_min", "tmpdir_max", "cuda_version_min", "cuda_compute_capability", "gpu_memory_min", "cuda_device_count_min", "cuda_device_count_max", "shm_size", "timelimit", ] VALID_RESOURCE_TYPES = get_args(ResourceType) class ResourceRequirement: def __init__(self, value_or_expression: Union[int, float, str], resource_type: ResourceType) -> None: self.value_or_expression = value_or_expression if not resource_type: raise ValueError("Missing resource requirement type") if resource_type not in VALID_RESOURCE_TYPES: raise ValueError(f"Invalid resource requirement type '{resource_type}'") self.resource_type = resource_type try: float(self.value_or_expression) self.runtime_required = False except ValueError: self.runtime_required = True def to_dict(self) -> Dict[str, Any]: return {"resource_type": self.resource_type, "value_or_expression": self.value_or_expression} def get_value(self, runtime: Optional[Dict] = None, js_evaluator: Optional[Callable] = None) -> float: if self.runtime_required: # TODO: hook up evaluator # return js_evaluator(self.value_or_expression, runtime) raise NotImplementedError( f"{self.value_or_expression} is not an integer or float value, expressions currently not implemented" ) return float(self.value_or_expression) def resource_requirements_from_list(requirements: Iterable[Dict[str, Any]]) -> List[ResourceRequirement]: cwl_to_galaxy = { "coresMin": "cores_min", "coresMax": "cores_max", "ramMin": "ram_min", "ramMax": "ram_max", "tmpdirMin": "tmpdir_min", "tmpdirMax": "tmpdir_max", "cudaVersionMin": "cuda_version_min", "cudaComputeCapability": "cuda_compute_capability", "gpuMemoryMin": "gpu_memory_min", "cudaDeviceCountMin": "cuda_device_count_min", "cudaDeviceCountMax": "cuda_device_count_max", "ShmSize": "shm_size", # timelimit is not a CWL ResourceRequirement field - CWL uses a separate # ToolTimeLimit requirement class. This entry is needed because Galaxy-format # resource dicts (type=resource) validate keys against cwl_to_galaxy.values(). "timelimit": "timelimit", } rr = [] for r in requirements: if r.get("class") == "ResourceRequirement": valid_key_set = set(cwl_to_galaxy.keys()) elif r.get("type") == "resource": valid_key_set = set(cwl_to_galaxy.values()) else: continue for key in valid_key_set.intersection(set(r.keys())): value = r[key] key = cast(ResourceType, cwl_to_galaxy.get(key, key)) if value is not None: # all resoure requirement fields are optional rr.append(ResourceRequirement(value_or_expression=value, resource_type=key)) return rr class BaseCredential: def __init__( self, name: str, inject_as_env: str, optional: bool = False, label: str = "", description: str = "", ) -> None: self.name = name self.inject_as_env = inject_as_env self.optional = optional self.label = label self.description = description if not self.name: raise ValueError("Missing credential (secret/variable) name") if not self.inject_as_env: raise ValueError("Missing inject_as_env") def to_dict(self) -> Dict[str, Any]: return { "name": self.name, "optional": self.optional, "label": self.label, "description": self.description, } class Secret(BaseCredential): @classmethod def from_element(cls, elem) -> "Secret": return cls( name=elem.get("name"), inject_as_env=elem.get("inject_as_env"), optional=string_as_bool(elem.get("optional", "false")), label=elem.get("label", ""), description=elem.get("description", ""), ) class Variable(BaseCredential): @classmethod def from_element(cls, elem) -> "Variable": return cls( name=elem.get("name"), inject_as_env=elem.get("inject_as_env"), optional=string_as_bool(elem.get("optional", "false")), label=elem.get("label", ""), description=elem.get("description", ""), ) class CredentialsRequirement: def __init__( self, name: str, version: str, label: str = "", description: str = "", optional: bool = False, secrets: Optional[List[Secret]] = None, variables: Optional[List[Variable]] = None, ) -> None: self.name = name self.version = version self.label = label self.description = description self.optional = optional self.secrets = secrets if secrets is not None else [] self.variables = variables if variables is not None else [] if not self.name: raise ValueError("Missing user credentials name") if not self.version: raise ValueError("Missing version") def to_dict(self) -> Dict[str, Any]: return { "name": self.name, "version": self.version, "label": self.label, "description": self.description, "optional": self.optional, "secrets": [s.to_dict() for s in self.secrets], "variables": [v.to_dict() for v in self.variables], } @classmethod def from_dict(cls, dict: Dict[str, Any]) -> "CredentialsRequirement": name = dict["name"] version = dict["version"] label = dict.get("label", "") description = dict.get("description", "") optional = dict.get("optional", False) secrets = [Secret.from_element(s) for s in dict.get("secrets", [])] variables = [Variable.from_element(v) for v in dict.get("variables", [])] return cls( name=name, version=version, label=label, description=description, optional=optional, secrets=secrets, variables=variables, ) def parse_requirements_from_lists( software_requirements: List[Union[ToolRequirement, Dict[str, Any]]], containers: Iterable[Dict[str, Any]], resource_requirements: Iterable[Dict[str, Any]], javascript_requirements: List[Dict[str, Any]], credentials: Iterable[Dict[str, Any]], ) -> Tuple[ ToolRequirements, List[ContainerDescription], List[ResourceRequirement], List[JavascriptRequirement], List[CredentialsRequirement], ]: return ( ToolRequirements.from_list(software_requirements), [ContainerDescription.from_dict(c) for c in containers], resource_requirements_from_list(resource_requirements), [JavascriptRequirement(**r) for r in javascript_requirements], [CredentialsRequirement.from_dict(s) for s in credentials], ) def parse_requirements_from_xml(xml_root, parse_resources_and_credentials: bool = False): """ Parses requirements, containers and optionally resource requirements from Xml tree. >>> from galaxy.util import parse_xml_string >>> def load_requirements(contents, parse_resources_and_credentials=False): ... contents_document = '''%s''' ... root = parse_xml_string(contents_document % contents) ... return parse_requirements_from_xml(root, parse_resources_and_credentials=parse_resources_and_credentials) >>> reqs, containers = load_requirements('''bwa''') >>> reqs[0].name 'bwa' >>> reqs[0].version is None True >>> reqs[0].type 'package' >>> reqs, containers = load_requirements('''cufflinks''') >>> reqs[0].name 'cufflinks' >>> reqs[0].version '1.3.3' >>> reqs[0].type 'binary' """ requirements_elem = xml_root.find("requirements") requirement_elems = [] container_elems = [] if requirements_elem is not None: requirement_elems = requirements_elem.findall("requirement") container_elems = requirements_elem.findall("container") requirements = ToolRequirements() for requirement_elem in requirement_elems: name = xml_text(requirement_elem) type = requirement_elem.get("type", DEFAULT_REQUIREMENT_TYPE) version = requirement_elem.get("version", DEFAULT_REQUIREMENT_VERSION) requirement = ToolRequirement(name=name, type=type, version=version) requirements.append(requirement) containers = [container_from_element(c) for c in container_elems] if parse_resources_and_credentials: resource_elems = requirements_elem.findall("resource") if requirements_elem is not None else [] resources = [resource_from_element(r) for r in resource_elems] javascript_requirements: List[Dict[str, Any]] = [] credentials_elems = requirements_elem.findall("credentials") if requirements_elem is not None else [] credentials = [credentials_from_element(s) for s in credentials_elems] return requirements, containers, resources, javascript_requirements, credentials return requirements, containers def resource_from_element(resource_elem) -> ResourceRequirement: value_or_expression = xml_text(resource_elem) resource_type = resource_elem.get("type") return ResourceRequirement(value_or_expression=value_or_expression, resource_type=resource_type) def container_from_element(container_elem) -> ContainerDescription: identifier = xml_text(container_elem) type = container_elem.get("type", DEFAULT_CONTAINER_TYPE) resolve_dependencies = asbool(container_elem.get("resolve_dependencies", DEFAULT_CONTAINER_RESOLVE_DEPENDENCIES)) shell = container_elem.get("shell", DEFAULT_CONTAINER_SHELL) container = ContainerDescription( identifier=identifier, type=type, resolve_dependencies=resolve_dependencies, shell=shell, ) return container def credentials_from_element(credentials_elem) -> CredentialsRequirement: name = credentials_elem.get("name") version = credentials_elem.get("version") label = credentials_elem.get("label", "") description = credentials_elem.get("description", "") optional = string_as_bool(credentials_elem.get("optional", "false")) secrets = [Secret.from_element(elem) for elem in credentials_elem.findall("secret")] variables = [Variable.from_element(elem) for elem in credentials_elem.findall("variable")] return CredentialsRequirement( name=name, version=version, label=label, description=description, optional=optional, secrets=secrets, variables=variables, ) ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1785871615.050191 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/0000755000175100017510000000000015234436377023711 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/__init__.py0000644000175100017510000002571415234436334026024 0ustar00runnerrunner"""The module defines the abstract interface for dealing tool dependency resolution plugins.""" import errno import os.path from abc import ( ABCMeta, abstractmethod, ) from typing import ( Any, Dict, List, ) import yaml from galaxy.util import listify from galaxy.util.dictifiable import Dictifiable from ..requirements import ( ToolRequirement, ToolRequirements, ) class DependencyResolver(Dictifiable, metaclass=ABCMeta): """Abstract description of a technique for resolving container images for tool execution.""" # Keys for dictification. dict_collection_visible_keys = [ "resolver_type", "resolves_simple_dependencies", "can_uninstall_dependencies", "read_only", ] # A "simple" dependency is one that does not depend on the tool # resolving the dependency. Classic tool shed dependencies are non-simple # because the repository install context is used in dependency resolution # so the same requirement tags in different tools will have very different # resolution. disabled = False resolves_simple_dependencies = True config_options: Dict[str, Any] = {} read_only = True @abstractmethod def resolve(self, requirement: ToolRequirement, **kwds) -> "Dependency": """Given inputs describing dependency in the abstract yield a Dependency object. The Dependency object describes various attributes (script, bin, version) used to build scripts with the dependency availble. Here script is the env.sh file to source before running a job, if that is not found the bin directory will be appended to the path (if it is not ``None``). Finally, version is the resolved tool dependency version (which may differ from requested version for instance if the request version is 'default'.) """ def install_dependency(self, name, version, type, **kwds): if self.read_only: return False else: return self._install_dependency(name, version, type, **kwds) def _install_dependency(self, name, version, type, **kwds): """Attempt to install this dependency if a recipe to do so has been registered in some way. """ return False @property def can_uninstall_dependencies(self): return not self.read_only class MultipleDependencyResolver: """Variant of DependencyResolver that can optionally resolve multiple dependencies together.""" @abstractmethod def resolve_all(self, requirements: ToolRequirements, **kwds) -> List["Dependency"]: """ Given multiple requirements yields a list of Dependency objects if and only if they may all be resolved together. Unsuccessfull attempts should return an empty list. :param requirements: list of tool requirements :param type: [ToolRequirement] or ToolRequirements :returns: list of resolved dependencies :rtype: [Dependency] """ class ListableDependencyResolver(metaclass=ABCMeta): """Mix this into a ``DependencyResolver`` and implement to indicate the dependency resolver can iterate over its dependencies and generate requirements. """ @abstractmethod def list_dependencies(self): """List the "simple" requirements that may be resolved "exact"-ly by this dependency resolver. """ def _to_requirement(self, name, version=None): return ToolRequirement(name=name, type="package", version=version) class MappableDependencyResolver: """Mix this into a ``DependencyResolver`` to allow mapping files. Mapping files allow adapting generic requirements to specific local implementations. """ def _setup_mapping(self, dependency_manager, **kwds): mapping_files = dependency_manager.get_resolver_option(self, "mapping_files", explicit_resolver_options=kwds) mappings = [] if mapping_files: search_dirs = [os.getcwd()] if isinstance(dependency_manager.default_base_path, str): search_dirs.append(dependency_manager.default_base_path) def candidates(path): if os.path.isabs(path): yield path else: for search_dir in search_dirs: yield os.path.join(search_dir, path) mapping_files = listify(mapping_files) for mapping_file in mapping_files: for full_path in candidates(mapping_file): if os.path.exists(full_path): mappings.extend(MappableDependencyResolver._mapping_file_to_list(full_path)) break self._mappings = mappings @staticmethod def _mapping_file_to_list(mapping_file): raw_mapping = [] try: with open(mapping_file) as f: raw_mapping = yaml.safe_load(f) except OSError as exc: if exc.errno != errno.ENOENT: raise return list(map(RequirementMapping.from_dict, raw_mapping)) def _expand_mappings(self, requirement): for mapping in self._mappings: if mapping.matches_requirement(requirement): requirement = mapping.apply(requirement) break return requirement FROM_UNVERSIONED = object() class RequirementMapping: def __init__(self, from_name, from_version, to_name, to_version): self.from_name = from_name self.from_version = from_version self.to_name = to_name self.to_version = to_version def matches_requirement(self, requirement): """Check if supplied ToolRequirement matches this mapping description. For it to match - the names must match. Additionally if the requirement is created with a version or with unversioned being set to True additional checks are needed. If a version is specified, it must match the supplied version exactly. If ``unversioned`` is True, then the supplied requirement must be unversioned (i.e. its version must be set to ``None``). """ if requirement.name != self.from_name: return False elif self.from_version is None: return True elif self.from_version is FROM_UNVERSIONED: return requirement.version is None else: return requirement.version == self.from_version def apply(self, requirement): requirement = requirement.copy() requirement.name = self.to_name if self.to_version is not None: requirement.version = self.to_version return requirement @staticmethod def from_dict(raw_mapping): from_raw = raw_mapping.get("from") if isinstance(from_raw, dict): from_name = from_raw.get("name") raw_version = from_raw.get("version", None) unversioned = from_raw.get("unversioned", False) if unversioned and raw_version: raise Exception("Cannot define both version and set unversioned to True.") if unversioned: from_version = FROM_UNVERSIONED else: from_version = str(raw_version) if raw_version is not None else raw_version else: from_name = from_raw from_version = None to_raw = raw_mapping.get("to") if isinstance(to_raw, dict): to_name = to_raw.get("name", from_name) to_version = str(to_raw.get("version")) else: to_name = to_raw to_version = None return RequirementMapping(from_name, from_version, to_name, to_version) class SpecificationAwareDependencyResolver(metaclass=ABCMeta): """Mix this into a :class:`DependencyResolver` to implement URI specification matching. Allows adapting generic requirements to more specific URIs - to tailor name or version to specified resolution system. """ @abstractmethod def _expand_specs(self, requirement): """Find closest matching specification for discovered resolver and return new concrete requirement.""" class SpecificationPatternDependencyResolver(SpecificationAwareDependencyResolver): """Implement the :class:`SpecificationAwareDependencyResolver` with a regex pattern.""" @property @abstractmethod def _specification_pattern(self): """Pattern of URI to match against.""" def _find_specification(self, specs): pattern = self._specification_pattern for spec in specs: if pattern.match(spec.uri): return spec return None def _expand_specs(self, requirement): name = requirement.name version = requirement.version specs = requirement.specs spec = self._find_specification(specs) if spec is not None: name = spec.short_name version = spec.version or version requirement = requirement.copy() requirement.name = name requirement.version = version return requirement class Dependency(Dictifiable, metaclass=ABCMeta): dict_collection_visible_keys = ["dependency_type", "exact", "name", "version", "cacheable"] cacheable = False @abstractmethod def shell_commands(self): """ Return shell commands to enable this dependency. """ @property @abstractmethod def exact(self): """Return true if version information wasn't discarded to resolve the dependency. """ @property def resolver_msg(self): """ Return a message describing this dependency """ return f"Using dependency {self.name} version {self.version} of type {self.dependency_type}" class ContainerDependency(Dependency): dict_collection_visible_keys = Dependency.dict_collection_visible_keys + [ "environment_path", "container_description", "container_resolver", ] def __init__(self, container_description, name=None, version=None, container_resolver=None): self.container_description = container_description self.dependency_type = container_description.type self._name = name self._version = version self.environment_path = container_description.identifier self.container_resolver = container_resolver @property def name(self): return self._name @property def version(self): return self._version @property def exact(self): return True @property def shell_commands(self): return None class NullDependency(Dependency): dependency_type = None exact = True def __init__(self, version=None, name=None): self.version = version self.name = name @property def resolver_msg(self): """ Return a message describing this dependency """ return f"Dependency {self.name} not found." def shell_commands(self): return None class DependencyException(Exception): pass ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/brewed_tool_shed_packages.py0000644000175100017510000001240315234436334031422 0ustar00runnerrunner""" This dependency resolver resolves tool shed dependencies (those defined tool_dependencies.xml) installed using Platform Homebrew and converted via shed2tap (e.g. https://github.com/jmchilton/homebrew-toolshed). """ import logging import os from galaxy.util import parse_xml from . import ( DependencyResolver, NullDependency, ) from .resolver_mixins import ( UsesHomebrewMixin, UsesInstalledRepositoriesMixin, UsesToolDependencyDirMixin, ) log = logging.getLogger(__name__) class HomebrewToolShedDependencyResolver( DependencyResolver, UsesHomebrewMixin, UsesToolDependencyDirMixin, UsesInstalledRepositoriesMixin, ): resolver_type = "tool_shed_tap" def __init__(self, dependency_manager, **kwds): self._init_homebrew(**kwds) self._init_base_path(dependency_manager, **kwds) def resolve(self, requirement, **kwds): name, version, type = requirement.name, requirement.version, requirement.type if type != "package": return NullDependency(version=version, name=name) if version is None: return NullDependency(version=version, name=name) return self._find_tool_dependencies(name, version, type, **kwds) def _find_tool_dependencies(self, name, version, type, **kwds): installed_tool_dependency = self._get_installed_dependency(name, type, version=version, **kwds) if installed_tool_dependency: return self._resolve_from_installed_tool_dependency(name, version, installed_tool_dependency) if "tool_dir" in kwds: tool_directory = os.path.abspath(kwds["tool_dir"]) tool_depenedencies_path = os.path.join(tool_directory, "tool_dependencies.xml") if os.path.exists(tool_depenedencies_path): return self._resolve_from_tool_dependencies_path(name, version, tool_depenedencies_path) return NullDependency(version=version, name=name) def _resolve_from_installed_tool_dependency(self, name, version, installed_tool_dependency): tool_shed_repository = installed_tool_dependency.tool_shed_repository recipe_name = build_recipe_name( package_name=name, package_version=version, repository_owner=tool_shed_repository.owner, repository_name=tool_shed_repository.name, ) return self._find_dep_default(recipe_name, None) def _resolve_from_tool_dependencies_path(self, name, version, tool_dependencies_path): try: raw_dependencies = RawDependencies(tool_dependencies_path) except Exception: log.debug(f"Failed to parse dependencies in file {tool_dependencies_path}") return NullDependency(version=version, name=name) raw_dependency = raw_dependencies.find(name, version) if not raw_dependency: return NullDependency(version=version, name=name) recipe_name = build_recipe_name( package_name=name, package_version=version, repository_owner=raw_dependency.repository_owner, repository_name=raw_dependency.repository_name, ) dep = self._find_dep_default(recipe_name, None) return dep class RawDependencies: def __init__(self, dependencies_file): self.root = parse_xml(dependencies_file).getroot() dependencies = [] package_els = self.root.findall("package") or [] for package_el in package_els: repository_el = package_el.find("repository") if repository_el is None: continue dependency = RawDependency(self, package_el, repository_el) dependencies.append(dependency) self.dependencies = dependencies def find(self, package_name, package_version): target_dependency = None for dependency in self.dependencies: if dependency.package_name == package_name and dependency.package_version == package_version: target_dependency = dependency break return target_dependency class RawDependency: def __init__(self, dependencies, package_el, repository_el): self.dependencies = dependencies self.package_el = package_el self.repository_el = repository_el def __repr__(self): temp = "Dependency[package_name=%s,version=%s,dependent_package=%s]" return temp % ( self.package_el.attrib["name"], self.package_el.attrib["version"], self.repository_el.attrib["name"], ) @property def repository_owner(self): return self.repository_el.attrib["owner"] @property def repository_name(self): return self.repository_el.attrib["name"] @property def package_name(self): return self.package_el.attrib["name"] @property def package_version(self): return self.package_el.attrib["version"] def build_recipe_name(package_name, package_version, repository_owner, repository_name): # TODO: Consider baking package_name and package_version into name? (would be more "correct") owner = repository_owner.replace("-", "") name = repository_name name = name.replace("_", "").replace("-", "") base = f"{owner}_{name}" return base __all__ = ("HomebrewToolShedDependencyResolver",) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/conda.py0000644000175100017510000005016615234436334025350 0ustar00runnerrunner""" This is still an experimental module and there will almost certainly be backward incompatible changes coming. """ import logging import os import re from typing import ( List, Optional, ) import galaxy.tool_util.deps.installable from galaxy.tool_util.deps.requirements import ( ToolRequirement, ToolRequirements, ) from . import ( Dependency, DependencyException, DependencyResolver, ListableDependencyResolver, MappableDependencyResolver, MultipleDependencyResolver, NullDependency, SpecificationPatternDependencyResolver, ) from ..conda_util import ( build_isolated_environment, cleanup_failed_install, cleanup_failed_install_of_environment, CondaContext, CondaTarget, hash_conda_packages, install_conda, install_conda_target, install_conda_targets, installed_conda_targets, is_conda_target_installed, USE_PATH_EXEC_DEFAULT, ) DEFAULT_BASE_PATH_DIRECTORY = "_conda" DEFAULT_CONDARC_OVERRIDE = "_condarc" # Conda channel order from highest to lowest, following the one used in # https://github.com/bioconda/bioconda-recipes/blob/master/config.yml DEFAULT_ENSURE_CHANNELS = "conda-forge,bioconda" CONDA_SOURCE_CMD = """[ "$(basename "$CONDA_DEFAULT_ENV")" = "$(basename '{environment_path}')" ] || {{ MAX_TRIES=3 COUNT=0 while [ $COUNT -lt $MAX_TRIES ]; do . '{activate_path}' '{environment_path}' > conda_activate.log 2>&1 if [ $? -eq 0 ];then break else let COUNT=COUNT+1 if [ $COUNT -eq $MAX_TRIES ];then echo "Failed to activate conda environment! Error was:" cat conda_activate.log exit 1 fi sleep 10s fi done }} """ log = logging.getLogger(__name__) class CondaDependencyResolver( DependencyResolver, MultipleDependencyResolver, ListableDependencyResolver, SpecificationPatternDependencyResolver, MappableDependencyResolver, ): dict_collection_visible_keys = DependencyResolver.dict_collection_visible_keys + [ "prefix", "versionless", "ensure_channels", "auto_install", "auto_init", "use_local", ] resolver_type = "conda" config_options = { "prefix": None, "exec": None, "debug": None, "ensure_channels": DEFAULT_ENSURE_CHANNELS, "auto_install": False, "auto_init": True, "copy_dependencies": False, "use_local": False, } _specification_pattern = re.compile(r"https\:\/\/anaconda.org\/\w+\/\w+") def __init__(self, dependency_manager, **kwds): read_only = _string_as_bool(kwds.get("read_only", "false")) self.read_only = read_only self._setup_mapping(dependency_manager, **kwds) self.versionless = _string_as_bool(kwds.get("versionless", "false")) self.dependency_manager = dependency_manager def get_option(name): return dependency_manager.get_resolver_option(self, name, explicit_resolver_options=kwds) # Conda context options (these define the environment) conda_prefix = get_option("prefix") if conda_prefix is None: conda_prefix = os.path.join(dependency_manager.default_base_path, DEFAULT_BASE_PATH_DIRECTORY) conda_prefix = os.path.abspath(conda_prefix) self.conda_prefix_parent = os.path.dirname(conda_prefix) condarc_override = get_option("condarc_override") if condarc_override is None: condarc_override = os.path.join(dependency_manager.default_base_path, DEFAULT_CONDARC_OVERRIDE) copy_dependencies = _string_as_bool(get_option("copy_dependencies")) use_local = _string_as_bool(get_option("use_local")) conda_exec = get_option("exec") debug = _string_as_bool(get_option("debug")) ensure_channels = get_option("ensure_channels") use_path_exec = get_option("use_path_exec") if use_path_exec is None: use_path_exec = USE_PATH_EXEC_DEFAULT else: use_path_exec = _string_as_bool(use_path_exec) if ensure_channels is None: ensure_channels = DEFAULT_ENSURE_CHANNELS conda_context = CondaContext( conda_prefix=conda_prefix, conda_exec=conda_exec, debug=debug, ensure_channels=ensure_channels, condarc_override=condarc_override, use_path_exec=use_path_exec, copy_dependencies=copy_dependencies, use_local=use_local, ) self.use_local = use_local self.ensure_channels = ensure_channels # Conda operations options (these define how resolution will occur) auto_install = _string_as_bool(get_option("auto_install")) self.auto_init = _string_as_bool(get_option("auto_init")) self.conda_context = conda_context self.disabled = not galaxy.tool_util.deps.installable.ensure_installed( conda_context, install_conda, self.auto_init ) if self.auto_init and not self.disabled: self.conda_context.ensure_conda_build_installed_if_needed() self.auto_install = auto_install self.copy_dependencies = copy_dependencies def clean(self, **kwds): return self.conda_context.exec_clean() def uninstall(self, requirements): """Uninstall requirements installed by install_all or multiple install statements.""" all_resolved = [r for r in self.resolve_all(requirements) if r.dependency_type] if not all_resolved: all_resolved = [self.resolve(requirement) for requirement in requirements] all_resolved = [r for r in all_resolved if r.dependency_type] if not all_resolved: return None environments = {os.path.basename(dependency.environment_path) for dependency in all_resolved} return self.uninstall_environments(environments) def uninstall_environments(self, environments): environments = [ env if not env.startswith(self.conda_context.envs_path) else os.path.basename(env) for env in environments ] return_codes = [self.conda_context.exec_remove([env]) for env in environments] final_return_code = 0 for env, return_code in zip(environments, return_codes): if return_code == 0: log.debug(f"Conda environment '{env}' successfully removed.") else: log.debug(f"Conda environment '{env}' could not be removed.") final_return_code = return_code return final_return_code def install_all(self, conda_targets: List[CondaTarget], env: str) -> bool: if self.read_only: return False return_code = install_conda_targets(conda_targets, conda_context=self.conda_context, env_name=env) if return_code != 0: is_installed = False else: # Recheck if installed is_installed = self.conda_context.has_env(env) if not is_installed: log.debug(f"Removing failed conda install of {str(conda_targets)}") cleanup_failed_install_of_environment(env, conda_context=self.conda_context) return is_installed def resolve_all(self, requirements: ToolRequirements, **kwds) -> List[Dependency]: """ Some combinations of tool requirements need to be resolved all at once, so that Conda can select a compatible combination of dependencies. This method returns a list of MergedCondaDependency instances (one for each requirement) if all requirements have been successfully resolved, or an empty list if any of the requirements could not be resolved. Parameters specific to this resolver are: preserve_python_environment: Boolean, controls whether the python environment should be maintained during job creation for tools that rely on galaxy being importable. install: Controls if `requirements` should be installed. If `install` is True and the requirements are not installed an attempt is made to install the requirements. If `install` is None requirements will only be installed if `conda_auto_install` has been activated and the requirements are not yet installed. If `install` is False will not install requirements. """ if len(requirements) == 0: return [] if not os.path.isdir(self.conda_context.conda_prefix): return [] for requirement in requirements: if requirement.type != "package": return [] expanded_requirements = ToolRequirements([self._expand_requirement(r) for r in requirements]) if self.versionless: conda_targets = [CondaTarget(r.name, version=None) for r in expanded_requirements] else: conda_targets = [CondaTarget(r.name, version=r.version) for r in expanded_requirements] preserve_python_environment = kwds.get("preserve_python_environment", False) for capitalized_package_names in (True, False): env = self.merged_environment_name(conda_targets, capitalized_package_names) is_installed = self.conda_context.has_env(env) if is_installed: break install = kwds.get("install", None) if install is None: # Default behavior, install dependencies if conda_auto_install is active. install = not is_installed and self.auto_install elif install: # Install has been set to True, install if not yet installed. install = not is_installed if install: is_installed = self.install_all(conda_targets, env) dependencies: List[Dependency] = [] if is_installed: for requirement in requirements: dependency = MergedCondaDependency( self.conda_context, self.conda_context.env_path(env), exact=not self.versionless or requirement.version is None, name=requirement.name, version=requirement.version, preserve_python_environment=preserve_python_environment, dependency_resolver=self, ) dependencies.append(dependency) return dependencies def merged_environment_name(self, conda_targets: List[CondaTarget], capitalized_package_names: bool = False) -> str: if len(conda_targets) > 1: # For continuity with mulled containers this is kind of nice. return f"mulled-v1-{hash_conda_packages(conda_targets, capitalized_package_names)}" else: assert len(conda_targets) == 1 return conda_targets[0].install_environment def resolve(self, requirement: ToolRequirement, **kwds) -> Dependency: requirement = self._expand_requirement(requirement) name, version, type = requirement.name, requirement.version, requirement.type # Check for conda just not being there, this way we can enable # conda by default and just do nothing in not configured. if not os.path.isdir(self.conda_context.conda_prefix): return NullDependency(version=version, name=name) if type != "package": return NullDependency(version=version, name=name) exact = not self.versionless or version is None if self.versionless: version = None conda_target = CondaTarget(name, version=version) is_installed = is_conda_target_installed(conda_target, conda_context=self.conda_context) preserve_python_environment = kwds.get("preserve_python_environment", False) job_directory = kwds.get("job_directory", None) install = kwds.get("install", None) if install is None: install = not is_installed and self.auto_install elif install: install = not is_installed if install: is_installed = self.install_dependency(name=name, version=version, type=type) if not is_installed: return NullDependency(version=version, name=name) # Have installed conda_target and job_directory to send it to. # If dependency is for metadata generation, store environment in conda-metadata-env if kwds.get("metadata", False): conda_env = "conda-metadata-env" else: conda_env = "conda-env" if job_directory: conda_environment = os.path.join(job_directory, conda_env) else: conda_environment = self.conda_context.get_conda_target_installed_path(conda_target) return CondaDependency( self.conda_context, conda_environment, exact, name, version, preserve_python_environment=preserve_python_environment, dependency_resolver=self, ) def _expand_requirement(self, requirement): return self._expand_specs(self._expand_mappings(requirement)) def unused_dependency_paths(self, toolbox_requirements_status): """ Identify all local environments that are not needed to build requirements_status. We try to resolve the requirements, and we note every environment_path that has been taken. """ used_paths = set() for dependencies in toolbox_requirements_status.values(): for dependency in dependencies: if dependency.get("dependency_type") == "conda": path = os.path.basename(dependency["environment_path"]) used_paths.add(path) dir_contents = set( os.listdir(self.conda_context.envs_path) if os.path.exists(self.conda_context.envs_path) else [] ) # never delete Galaxy's conda env used_paths.add("_galaxy_") unused_paths = dir_contents.difference(used_paths) # New set with paths in dir_contents but not in used_paths unused_paths = [os.path.join(self.conda_context.envs_path, p) for p in unused_paths] return unused_paths def list_dependencies(self): for install_target in installed_conda_targets(self.conda_context): name = install_target.package version = install_target.version yield self._to_requirement(name, version) def _install_dependency(self, name, version, type, **kwds): "Returns True on (seemingly) successfull installation" # should be checked before called assert not self.read_only if type != "package": log.warning(f"Cannot install dependencies of type '{type}'") return False if self.versionless: version = None conda_target = CondaTarget(name, version=version) is_installed = is_conda_target_installed(conda_target, conda_context=self.conda_context) if is_installed: return is_installed return_code = install_conda_target(conda_target, conda_context=self.conda_context) if return_code != 0: is_installed = False else: # Recheck if installed is_installed = is_conda_target_installed(conda_target, conda_context=self.conda_context) if not is_installed: log.debug(f"Removing failed conda install of {name}, version '{version}'") cleanup_failed_install(conda_target, conda_context=self.conda_context) return is_installed @property def prefix(self): return self.conda_context.conda_prefix class MergedCondaDependency(Dependency): dict_collection_visible_keys = Dependency.dict_collection_visible_keys + [ "environment_path", "name", "version", "dependency_resolver", ] dependency_type = "conda" def __init__( self, conda_context: CondaContext, environment_path: str, exact: bool, name: str, version: Optional[str] = None, preserve_python_environment: bool = False, dependency_resolver: Optional[DependencyResolver] = None, ) -> None: self.activate = conda_context.activate self.conda_context = conda_context self.environment_path = environment_path self._exact = exact self._name = name self._version = version self.cache_path = None self._preserve_python_environment = preserve_python_environment self.dependency_resolver = dependency_resolver @property def exact(self): return self._exact @property def name(self): return self._name @property def version(self): return self._version def shell_commands(self): if self._preserve_python_environment: # On explicit testing the only such requirement I am aware of is samtools - and it seems to work # fine with just appending the PATH as done below. Other tools may require additional # variables in the future. return f"""export PATH=$PATH:'{self.environment_path}/bin' """ else: return CONDA_SOURCE_CMD.format(activate_path=self.activate, environment_path=self.environment_path) class CondaDependency(Dependency): dict_collection_visible_keys = Dependency.dict_collection_visible_keys + [ "environment_path", "name", "version", "dependency_resolver", ] dependency_type = "conda" cacheable = True def __init__( self, conda_context: CondaContext, environment_path: str, exact: bool, name: str, version: Optional[str] = None, preserve_python_environment: bool = False, dependency_resolver: Optional[DependencyResolver] = None, ) -> None: self.activate = conda_context.activate self.conda_context = conda_context self.environment_path = environment_path self._exact = exact self._name = name self._version = version self.cache_path = None self._preserve_python_environment = preserve_python_environment self.dependency_resolver = dependency_resolver @property def exact(self): return self._exact @property def name(self): return self._name @property def version(self): return self._version def build_cache(self, cache_path): self.set_cache_path(cache_path) self.build_environment() def set_cache_path(self, cache_path): self.cache_path = cache_path self.environment_path = cache_path def build_environment(self) -> None: env_path, exit_code = build_isolated_environment( CondaTarget(self.name, version=self.version), conda_context=self.conda_context, path=self.environment_path, copy=self.conda_context.copy_dependencies, ) if exit_code: if len(os.path.abspath(self.environment_path)) > 79: # TODO: remove this once conda_build version 2 is released and packages have been rebuilt. raise DependencyException( "Conda dependency failed to build job environment. " "This is most likely a limitation in conda. " "You can try to shorten the path to the job_working_directory." ) raise DependencyException("Conda dependency seemingly installed but failed to build job environment.") def shell_commands(self): if not self.cache_path: # Build an isolated environment if not using a cached dependency manager self.build_environment() if self._preserve_python_environment: # On explicit testing the only such requirement I am aware of is samtools - and it seems to work # fine with just appending the PATH as done below. Other tools may require additional # variables in the future. return f"""export PATH=$PATH:'{self.environment_path}/bin' """ else: return CONDA_SOURCE_CMD.format(activate_path=self.activate, environment_path=self.environment_path) def _string_as_bool(value): return str(value).lower() == "true" __all__ = ("CondaDependencyResolver", "DEFAULT_ENSURE_CHANNELS") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/default_conda_mapping.yml0000644000175100017510000000127615234436334030736 0ustar00runnerrunner- from: R to: r-base - from: blast+ to: blast - from: name: samtools unversioned: true to: name: samtools version: 1.3.1 - from: name: ucsc_tools unversioned: true to: name: ucsc_tools version: 357 - from: name: bedtools unversioned: true to: name: bedtools version: 2.26.0gx - from: name: gnuplot unversioned: true to: name: gnuplot version: 4.6 - from: name: openbabel unversioned: true to: name: openbabel version: 2.3.2 - from: name: gnuplot-py unversioned: true to: name: gnuplot-py version: 1.8 - from: name: bowtie unversioned: true to: name: bowtie version: 1.2.0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/galaxy_packages.py0000644000175100017510000001241515234436334027402 0ustar00runnerrunnerimport logging from os import listdir from os.path import ( basename, exists, isdir, islink, join, realpath, ) from . import ( Dependency, DependencyResolver, ListableDependencyResolver, MappableDependencyResolver, NullDependency, ) from .resolver_mixins import UsesToolDependencyDirMixin log = logging.getLogger(__name__) class GalaxyPackageDependency(Dependency): dict_collection_visible_keys = Dependency.dict_collection_visible_keys + [ "script", "path", "version", "name", "dependency_resolver", ] dependency_type = "galaxy_package" def __init__(self, script, path, name, type, version, exact=True, dependency_resolver=None): self.script = script self.path = path self.name = name self.type = type self.version = version self._exact = exact assert self.script is not None or self.path is not None self.dependency_resolver = dependency_resolver @property def exact(self): return self._exact def shell_commands(self): base_path = self.path if self.type == "package" and self.script is None: commands = f'PACKAGE_BASE={base_path}; export PACKAGE_BASE; PATH="{base_path}/bin:$PATH"; export PATH' else: commands = f"PACKAGE_BASE={base_path}; export PACKAGE_BASE; . {self.script}" return commands class ToolShedDependency(GalaxyPackageDependency): dependency_type = "tool_shed_package" class BaseGalaxyPackageDependencyResolver(DependencyResolver, UsesToolDependencyDirMixin): dict_collection_visible_keys = DependencyResolver.dict_collection_visible_keys + ["base_path", "versionless"] dependency_type = GalaxyPackageDependency def __init__(self, dependency_manager, **kwds): # Galaxy tool shed requires explicit versions on XML elements, # this in inconvient for testing or Galaxy instances not utilizing # the tool shed so allow a fallback version of the Galaxy package # resolver that will just grab 'default' version of exact version # unavailable. self.versionless = str(kwds.get("versionless", "false")).lower() == "true" self._init_base_path(dependency_manager, **kwds) def resolve(self, requirement, **kwds): """ Attempt to find a dependency named `name` at version `version`. If version is None, return the "default" version as determined using a symbolic link (if found). Returns a triple of: env_script, base_path, real_version """ name, version, type = requirement.name, requirement.version, requirement.type if version is None or self.versionless: exact = not self.versionless or version is None return self._find_dep_default(name, type=type, exact=exact, **kwds) else: return self._find_dep_versioned(name, version, type=type, **kwds) def _find_dep_versioned(self, name, version, type="package", **kwds): base_path = self.base_path path = join(base_path, name, version) return self._galaxy_package_dep(path, version, name, type, True) def _find_dep_default(self, name, type="package", exact=True, **kwds): base_path = self.base_path path = join(base_path, name, "default") if islink(path): real_path = realpath(path) real_version = basename(real_path) return self._galaxy_package_dep(real_path, real_version, name, type, exact) else: return NullDependency(version=None, name=name) def _galaxy_package_dep(self, path, version, name, type, exact): script = join(path, "env.sh") if exists(script): return self.dependency_type(script, path, name, type, version, exact, dependency_resolver=self) elif exists(join(path, "bin")): return self.dependency_type(None, path, name, type, version, exact, dependency_resolver=self) return NullDependency(version=version, name=name) class GalaxyPackageDependencyResolver( BaseGalaxyPackageDependencyResolver, ListableDependencyResolver, MappableDependencyResolver ): resolver_type = "galaxy_packages" def __init__(self, dependency_manager, **kwds): super().__init__(dependency_manager, **kwds) self._setup_mapping(dependency_manager, **kwds) def resolve(self, requirement, **kwds): requirement = self._expand_mappings(requirement) return super().resolve(requirement, **kwds) def list_dependencies(self): base_path = self.base_path for package_name in listdir(base_path): package_dir = join(base_path, package_name) if isdir(package_dir): for version in listdir(package_dir): version_dir = join(package_dir, version) if version == "default": version = None valid_dependency = _is_dependency_directory(version_dir) if valid_dependency: yield self._to_requirement(package_name, version) def _is_dependency_directory(directory): return exists(join(directory, "env.sh")) or exists(join(directory, "bin")) __all__ = ("GalaxyPackageDependency", "GalaxyPackageDependencyResolver", "ToolShedDependency") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/homebrew.py0000644000175100017510000000376115234436334026073 0ustar00runnerrunner""" This file implements a brew resolver for Galaxy requirements. In order for Galaxy to pick up on recursively defined and versioned brew dependencies recipes should be installed using the experimental `brew-vinstall` external command. More information here: https://github.com/jmchilton/brew-tests https://github.com/Homebrew/homebrew-science/issues/1191 This is still an experimental module and there will almost certainly be backward incompatible changes coming. """ from . import ( DependencyResolver, NullDependency, ) from .resolver_mixins import UsesHomebrewMixin # TODO: Implement prefer version linked... PREFER_VERSION_LINKED = "linked" PREFER_VERSION_LATEST = "latest" UNKNOWN_PREFER_VERSION_MESSAGE_TEMPLATE = "HomebrewDependencyResolver prefer_version must be %s" UNKNOWN_PREFER_VERSION_MESSAGE = UNKNOWN_PREFER_VERSION_MESSAGE_TEMPLATE % (PREFER_VERSION_LATEST) DEFAULT_PREFER_VERSION = PREFER_VERSION_LATEST class HomebrewDependencyResolver(DependencyResolver, UsesHomebrewMixin): resolver_type = "homebrew" def __init__(self, dependency_manager, **kwds): self.versionless = _string_as_bool(kwds.get("versionless", "false")) self.prefer_version = kwds.get("prefer_version", None) if self.prefer_version is None: self.prefer_version = DEFAULT_PREFER_VERSION if self.versionless and self.prefer_version not in [PREFER_VERSION_LATEST]: raise Exception(UNKNOWN_PREFER_VERSION_MESSAGE) self._init_homebrew(**kwds) def resolve(self, requirement, **kwds): name, version, type = requirement.name, requirement.version, requirement.type if type != "package": return NullDependency(version=version, name=name) if version is None or self.versionless: return self._find_dep_default(name, version) else: return self._find_dep_versioned(name, version) def _string_as_bool(value): return str(value).lower() == "true" __all__ = ("HomebrewDependencyResolver",) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/lmod.py0000644000175100017510000002010015234436334025200 0ustar00runnerrunner""" This is a prototype dependency resolver to be able to use the "LMOD environment modules system" from TACC to solve package requirements LMOD official website: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod LMOD @ Github: https://github.com/TACC/Lmod """ import logging from io import StringIO from os import getenv from os.path import exists from subprocess import ( PIPE, Popen, ) from . import ( Dependency, DependencyResolver, MappableDependencyResolver, NullDependency, ) log = logging.getLogger(__name__) DEFAULT_LMOD_PATH = getenv("LMOD_CMD") DEFAULT_SETTARG_PATH = getenv("LMOD_SETTARG_CMD") DEFAULT_MODULEPATH = getenv("MODULEPATH") DEFAULT_MAPPING_FILE = "config/lmod_modules_mapping.yml" INVALID_LMOD_PATH_MSG = "The following LMOD executable could not be found: %s. Either your LMOD Dependency Resolver is misconfigured or LMOD is improperly installed on your system !" EMPTY_MODULEPATH_MSG = "No valid LMOD MODULEPATH defined ! Either your LMOD Dependency Resolver is misconfigured or LMOD is improperly installed on your system !" class LmodDependencyResolver(DependencyResolver, MappableDependencyResolver): """Dependency resolver based on the LMOD environment modules system""" dict_collection_visible_keys = DependencyResolver.dict_collection_visible_keys + ["base_path", "modulepath"] resolver_type = "lmod" def __init__(self, dependency_manager, **kwds): # Mapping file management self._set_default_mapping_file(kwds) self._setup_mapping(dependency_manager, **kwds) # Other attributes self.versionless = _string_as_bool(kwds.get("versionless", "false")) self.lmodexec = kwds.get("lmodexec", DEFAULT_LMOD_PATH) self.settargexec = kwds.get("settargexec", DEFAULT_SETTARG_PATH) self.modulepath = kwds.get("modulepath", DEFAULT_MODULEPATH) self.module_checker = AvailModuleChecker(self, self.modulepath) self.skip_availability_check = _string_as_bool(kwds.get("skip_availability_check", "false")) def _set_default_mapping_file(self, resolver_attributes): if "mapping_files" not in resolver_attributes: if exists(DEFAULT_MAPPING_FILE): resolver_attributes["mapping_files"] = DEFAULT_MAPPING_FILE def resolve(self, requirement, **kwds): requirement = self._expand_mappings(requirement) name, version, type = requirement.name, requirement.version, requirement.type if type != "package": return NullDependency(version=version, name=name) if self.skip_availability_check: return LmodDependency(self, name, version, exact=True, dependency_resolver=self) if self.__has_module(name, version): return LmodDependency(self, name, version, exact=True, dependency_resolver=self) elif self.versionless and self.__has_module(name, None): return LmodDependency(self, name, None, exact=False, dependency_resolver=self) return NullDependency(version=version, name=name) def __has_module(self, name, version): return self.module_checker.has_module(name, version) class AvailModuleChecker: """Parses the output of Lmod 'module avail' command to get the list of available modules.""" def __init__(self, lmod_dependency_resolver, modulepath): self.lmod_dependency_resolver = lmod_dependency_resolver self.modulepath = modulepath def has_module(self, module, version): # When version is None (No specific version required by the wrapper -or- versionless is set to 'true'), we only get the list of default modules # We get the full list of modules otherwise if version is None: available_modules = self.__get_list_of_available_modules(True) else: available_modules = self.__get_list_of_available_modules(False) # Is the required module in the list of avaialable modules ? for module_name, module_version in available_modules: names_match = module == module_name module_match = names_match and (version is None or module_version == version) if module_match: return True return False def __get_list_of_available_modules(self, default_version_only=False): # Get the results of the "module avail" command in an easy to parse format # Note that since "module" is actually a bash function, we are directy executing the underlying executable instead raw_output = self.__get_module_avail_command_output(default_version_only).decode("utf-8") # Parse the result for line in StringIO(raw_output): # Clean line and discard non-module lines line = line and line.strip() if not line or line.startswith("/"): continue # Split module lines by / to separate the module name from the module version # Module without version are discarded module_parts = line.split("/") if len(module_parts) == 2: yield module_parts[0], module_parts[1] def __get_module_avail_command_output(self, default_version_only=False): # Check if the LMOD executable is available (ie. if both LMOD and the lmod dependency resolver are both setup properly) lmodexec = self.lmod_dependency_resolver.lmodexec if not exists(lmodexec): raise Exception(INVALID_LMOD_PATH_MSG % lmodexec) # Check if the MODULEPATH environment if self.modulepath == "" or self.modulepath is None: raise Exception(EMPTY_MODULEPATH_MSG) # Build command line if default_version_only: module_avail_command = [lmodexec, "-t", "-d", "avail"] else: module_avail_command = [lmodexec, "-t", "avail"] # The list of avaialable modules is actually printed on stderr and not stdout for module commands return Popen( module_avail_command, stdout=PIPE, stderr=PIPE, env={"MODULEPATH": self.modulepath}, close_fds=True ).communicate()[1] class LmodDependency(Dependency): """Prepare the commands required to solve the dependency and add them to the script used to run a tool in Galaxy.""" dict_collection_visible_keys = Dependency.dict_collection_visible_keys + [ "module_name", "module_version", "dependency_resolver", ] dependency_type = "lmod" def __init__( self, lmod_dependency_resolver, module_name, module_version=None, exact=True, dependency_resolver=None ): self.lmod_dependency_resolver = lmod_dependency_resolver self.module_name = module_name self.module_version = module_version self._exact = exact self.dependency_resolver = dependency_resolver @property def name(self): return self.module_name @property def version(self): return self.module_version @property def exact(self): return self._exact def shell_commands(self): # Get the full module name in the form "tool_name/tool_version" module_to_load = self.module_name if self.module_version: module_to_load = f"{self.module_name}/{self.module_version}" # Build the list of command to add to run script # Note that since "module" is actually a bash function, we are directy executing the underlying executable instead # - Set and/or prepend the MODULEPATH environment variable command = f"MODULEPATH={self.lmod_dependency_resolver.modulepath}${{MODULEPATH:+:${{MODULEPATH}}}}; " command += "export MODULEPATH; " # - Execute the "module load" command (or rather the "/path/to/lmod load" command) command += f"eval `{self.lmod_dependency_resolver.lmodexec} load {module_to_load}` " # - Execute the "settarg" command in addition if needed if self.lmod_dependency_resolver.settargexec is not None: command += f"&& eval `{self.lmod_dependency_resolver.settargexec} -s sh`" return command def _string_as_bool(value): return str(value).lower() == "true" __all__ = ("LmodDependencyResolver",) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/modules.py0000644000175100017510000002255715234436334025737 0ustar00runnerrunner""" This file contains the outline of an implementation to load environment modules (http://modules.sourceforge.net/). This is a community contributed feature and the core Galaxy team does utilize it, hence support for it will be minimal. The Galaxy team eagerly welcomes community contribution and maintenance however. """ import logging from io import StringIO from os import ( environ, pathsep, ) from os.path import ( exists, isdir, join, ) from subprocess import ( PIPE, Popen, ) from . import ( Dependency, DependencyResolver, MappableDependencyResolver, NullDependency, ) log = logging.getLogger(__name__) DEFAULT_MODULECMD_PATH = "modulecmd" # Just check path DEFAULT_MODULE_PATH = "/usr/share/modules/modulefiles" DEFAULT_INDICATOR = "(default)" DEFAULT_MODULE_PREFETCH = "true" DEFAULT_MAPPING_FILE = "config/environment_modules_mapping.yml" UNKNOWN_FIND_BY_MESSAGE = ( "ModuleDependencyResolver does not know how to find modules by [%s], find_by should be one of %s" ) class ModuleDependencyResolver(DependencyResolver, MappableDependencyResolver): dict_collection_visible_keys = DependencyResolver.dict_collection_visible_keys + [ "base_path", "modulepath", "modulecmd", "prefetch", "default_indicator", "find_by", ] resolver_type = "modules" def __init__(self, dependency_manager, **kwds): # Mapping file management self._set_default_mapping_file(kwds) self._setup_mapping(dependency_manager, **kwds) self.versionless = _string_as_bool(kwds.get("versionless", "false")) find_by = kwds.get("find_by", "avail") self.find_by = find_by prefetch = _string_as_bool(kwds.get("prefetch", DEFAULT_MODULE_PREFETCH)) self.modulecmd = kwds.get("modulecmd", DEFAULT_MODULECMD_PATH) self.modulepath = kwds.get("modulepath", self.__default_modulespath()) self.default_indicator = kwds.get("default_indicator", DEFAULT_INDICATOR) self.skip_availability_check = _string_as_bool(kwds.get("skip_availability_check", "false")) if find_by == "directory": self.module_checker = DirectoryModuleChecker(self, self.modulepath, prefetch) elif find_by == "avail": self.module_checker = AvailModuleChecker(self, self.modulepath, prefetch, self.default_indicator) else: raise Exception(UNKNOWN_FIND_BY_MESSAGE % (find_by, ["avail", "directory"])) def __default_modulespath(self): if "MODULEPATH" in environ: module_path = environ["MODULEPATH"] elif "MODULESHOME" in environ: module_path = join(environ["MODULESHOME"], "modulefiles") else: module_path = DEFAULT_MODULE_PATH return module_path def _set_default_mapping_file(self, resolver_attributes): if "mapping_files" not in resolver_attributes: if exists(DEFAULT_MAPPING_FILE): resolver_attributes["mapping_files"] = DEFAULT_MAPPING_FILE def resolve(self, requirement, **kwds): requirement = self._expand_mappings(requirement) name, version, type = requirement.name, requirement.version, requirement.type if type != "package": return NullDependency(version=version, name=name) if self.skip_availability_check: return ModuleDependency(self, name, version, exact=True, dependency_resolver=self) if self.__has_module(name, version): return ModuleDependency(self, name, version, exact=True, dependency_resolver=self) elif self.versionless and self.__has_module(name, None): return ModuleDependency(self, name, None, exact=False, dependency_resolver=self) return NullDependency(version=version, name=name) def __has_module(self, name, version): return self.module_checker.has_module(name, version) class DirectoryModuleChecker: """Finds module by path. Searches the paths listed in modulepath to for a file or directory matching the module name. If the version=True, searches for files named module/version.""" def __init__(self, module_dependency_resolver, modulepath, prefetch): self.module_dependency_resolver = module_dependency_resolver self.directories = modulepath.split(pathsep) if prefetch: log.warning( "Created module dependency resolver with prefetch enabled, but directory module checker does not support this." ) def has_module(self, module, version): has_module = False for directory in self.directories: module_directory = join(directory, module) has_module_directory = isdir(module_directory) if not version: has_module = has_module_directory or exists(module_directory) # could be a bare modulefile else: modulefile = join(module_directory, version) has_modulefile = exists(modulefile) has_module = has_module_directory and has_modulefile if has_module: break return has_module class AvailModuleChecker: """Finds modules by searching output of 'module avail'. Parses the Environment Modules 'module avail' output, splitting module names into module and version on '/' and discarding a postfix matching default_indicator (by default '(default)'. Matching is done using the module and (if version=True) the module version.""" def __init__(self, module_dependency_resolver, modulepath, prefetch, default_indicator=DEFAULT_INDICATOR): self.module_dependency_resolver = module_dependency_resolver self.modulepath = modulepath self.default_indicator = default_indicator if prefetch: prefetched_modules = [] for module in self.__modules(): prefetched_modules.append(module) else: prefetched_modules = None self.prefetched_modules = prefetched_modules def has_module(self, module, version): module_generator = self.prefetched_modules if module_generator is None: module_generator = self.__modules() for module_name, module_version in module_generator: names_match = module == module_name module_match = names_match and (version is None or module_version == version) if module_match: return True return False def __modules(self): raw_output = self.__module_avail_output().decode("utf-8") for line in StringIO(raw_output): line = line and line.strip() if not line or line.startswith("-"): continue line_modules = line.split() for module in line_modules: if module.endswith(self.default_indicator): module = module[0 : -len(self.default_indicator)].strip() module_parts = module.split("/") module_version = None if len(module_parts) == 2: module_version = module_parts[1] module_name = module_parts[0] yield module_name, module_version def __module_avail_output(self): avail_command = [self.module_dependency_resolver.modulecmd, "sh", "avail"] return Popen(avail_command, stderr=PIPE, env={"MODULEPATH": self.modulepath}).communicate()[1] class ModuleDependency(Dependency): """Converts module dependencies into shell expressions using modulecmd. Using Environment Modules' 'modulecmd' (specifically 'modulecmd sh load') to convert module specifications into shell expressions for inclusion in the script used to run a tool in Galaxy.""" dict_collection_visible_keys = Dependency.dict_collection_visible_keys + [ "module_name", "module_version", "dependency_resolver", ] dependency_type = "module" def __init__( self, module_dependency_resolver, module_name, module_version=None, exact=True, dependency_resolver=None ): self.module_dependency_resolver = module_dependency_resolver self.module_name = module_name self.module_version = module_version self._exact = exact self.dependency_resolver = dependency_resolver @property def name(self): return self.module_name @property def version(self): return self.module_version @property def exact(self): return self._exact def shell_commands(self): # Get the full module name in the form "tool_name/tool_version" module_to_load = self.module_name if self.module_version: module_to_load = f"{self.module_name}/{self.module_version}" # Build the list of command to add to run script # Note that since "module" is actually a bash function, we are directy executing the underlying executable instead # - Set and/or prepend the MODULEPATH environment variable command = f"MODULEPATH={self.module_dependency_resolver.modulepath}${{MODULEPATH:+:${{MODULEPATH}}}}; " command += "export MODULEPATH; " # - Execute the "module load" command (or rather the "/path/to/module load" command) command += f"eval `{self.module_dependency_resolver.modulecmd} sh load {module_to_load}`" return command def _string_as_bool(value): return str(value).lower() == "true" __all__ = ("ModuleDependencyResolver",) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/resolver_mixins.py0000644000175100017510000000527415234436334027514 0ustar00runnerrunnerimport os from . import ( Dependency, NullDependency, ) from ..brew_exts import ( build_env_statements, DEFAULT_HOMEBREW_ROOT, recipe_cellar_path, ) class UsesHomebrewMixin: def _init_homebrew(self, **kwds): cellar_root = kwds.get("cellar", None) if cellar_root is None: cellar_root = os.path.join(DEFAULT_HOMEBREW_ROOT, "Cellar") self.cellar_root = cellar_root def _find_dep_versioned(self, name, version): recipe_path = recipe_cellar_path(self.cellar_root, name, version) if not os.path.exists(recipe_path) or not os.path.isdir(recipe_path): return NullDependency(version=version, name=name) commands = build_env_statements(self.cellar_root, recipe_path, relaxed=True) return HomebrewDependency(commands) def _find_dep_default(self, name, version): installed_versions = self._installed_versions(name) if not installed_versions: return NullDependency(version=version, name=name) # Just grab newest installed version - may make sense some day to find # the linked version instead. default_version = sorted(installed_versions, reverse=True)[0] return self._find_dep_versioned(name, default_version, exact=version is None) def _installed_versions(self, recipe): recipe_base_path = os.path.join(self.cellar_root, recipe) if not os.path.exists(recipe_base_path): return [] names = os.listdir(recipe_base_path) return [n for n in names if os.path.isdir(os.path.join(recipe_base_path, n))] class UsesToolDependencyDirMixin: def _init_base_path(self, dependency_manager, **kwds): self.base_path = os.path.abspath(kwds.get("base_path", dependency_manager.default_base_path)) class UsesInstalledRepositoriesMixin: def _get_installed_dependency(self, name, type, version=None, **kwds): installed_tool_dependencies = kwds.get("installed_tool_dependencies") or [] for installed_tool_dependency in installed_tool_dependencies: if installed_tool_dependency.name == name and installed_tool_dependency.type == type: if not version or installed_tool_dependency.version == version: return installed_tool_dependency return None class HomebrewDependency(Dependency): def __init__(self, commands, exact=True): self.commands = commands self._exact = exact @property def exact(self): return self._exact def shell_commands(self): raw_commands = self.commands.replace("\n", ";") return raw_commands def __repr__(self): return f"PlatformBrewDependency[commands={self.commands}]" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/tool_shed_packages.py0000644000175100017510000000677715234436334030113 0ustar00runnerrunnerfrom os.path import ( abspath, exists, join, ) from . import NullDependency from .galaxy_packages import ( BaseGalaxyPackageDependencyResolver, ToolShedDependency, ) from .resolver_mixins import UsesInstalledRepositoriesMixin class ToolShedPackageDependencyResolver(BaseGalaxyPackageDependencyResolver, UsesInstalledRepositoriesMixin): resolver_type = "tool_shed_packages" # Resolution of these dependencies depends on more than just the requirement # tag, it depends on the tool installation context - therefore these are # non-simple. dependency_type = ToolShedDependency resolves_simple_dependencies = False def __init__(self, dependency_manager, **kwds): super().__init__(dependency_manager, **kwds) def _find_dep_versioned(self, name, version, type="package", **kwds): installed_tool_dependency = self._get_installed_dependency(name, type, version=version, **kwds) if installed_tool_dependency: path = self._get_package_installed_dependency_path(installed_tool_dependency, name, version) return self._galaxy_package_dep(path, version, name, type, True) else: return NullDependency(version=version, name=name) def _find_dep_default(self, name, type="package", exact=True, **kwds): if type == "set_environment" and kwds.get("installed_tool_dependencies", None): installed_tool_dependency = self._get_installed_dependency(name, type, version=None, **kwds) if installed_tool_dependency: dependency = self._get_set_environment_installed_dependency_script_path(installed_tool_dependency, name) is_galaxy_dep = isinstance(dependency, ToolShedDependency) has_script_dep = is_galaxy_dep and dependency.script and dependency.path if has_script_dep: # Environment settings do not use versions. return ToolShedDependency( dependency.script, dependency.path, name, "set_environment", None, True, dependency_resolver=self, ) return NullDependency(version=None, name=name) def _get_package_installed_dependency_path(self, installed_tool_dependency, name, version): tool_shed_repository = installed_tool_dependency.tool_shed_repository base_path = self.base_path return join( base_path, name, version, tool_shed_repository.owner, tool_shed_repository.name, tool_shed_repository.installed_changeset_revision, ) def _get_set_environment_installed_dependency_script_path(self, installed_tool_dependency, name): tool_shed_repository = installed_tool_dependency.tool_shed_repository base_path = self.base_path path = abspath( join( base_path, "environment_settings", name, tool_shed_repository.owner, tool_shed_repository.name, tool_shed_repository.installed_changeset_revision, ) ) if exists(path): script = join(path, "env.sh") return ToolShedDependency(script, path, name, "set_environment", None, True, dependency_resolver=self) return NullDependency(version=None, name=name) __all__ = ("ToolShedPackageDependencyResolver",) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/resolvers/unlinked_tool_shed_packages.py0000644000175100017510000001656715234436334032002 0ustar00runnerrunner""" Backup resolvers for when dependencies can not be loaded from the database. Mainly suited for testing stage. Ideally all dependencies will be stored in the database when a tool is added from a Tool Shed. That should remain the preferred way of locating dependencies. In cases where that is not possible for example during testing this resolver can act as a backup. This resolver looks not just for manually added dependencies but also ones added from a Tool Shed. This tool is still under development so the default behaviour could change. It has been tested when placed in the same directory as galaxy_packages.py At the time of writing July 3 2015 this resolver has to be plugged in. See bottom for instructions on how to add this resolver. """ import logging from os import listdir from os.path import ( exists, getmtime, join, ) from . import ( Dependency, NullDependency, ) from .galaxy_packages import BaseGalaxyPackageDependencyResolver log = logging.getLogger(__name__) MANUAL = "manual" PREFERRED_OWNERS = f"{MANUAL},iuc,devteam" class UnlinkedToolShedPackageDependencyResolver(BaseGalaxyPackageDependencyResolver): dict_collection_visible_keys = BaseGalaxyPackageDependencyResolver.dict_collection_visible_keys + [ "preferred_owners", "select_by_owner", ] resolver_type = "unlinked_tool_shed_packages" def __init__(self, dependency_manager, **kwds): super().__init__(dependency_manager, **kwds) # Provide a list of preferred owners whose dependency to use self.preferred_owners = kwds.get("preferred_owners", PREFERRED_OWNERS).split(",") # Option to ignore owner and just use last modified time self.select_by_owner = str(kwds.get("select_by_owner", "true")).lower() != "false" def _find_dep_versioned(self, name, version, type="package", **kwds): try: possibles = self._find_possible_dependencies(name, version, type) if len(possibles) == 0: log.debug("Unable to find dependency,'%s' '%s' '%s'", name, version, type) return NullDependency(version=version, name=name) elif len(possibles) == 1: # Only one candidate found so ignore any preference rules return possibles[0].dependency else: # Pick the preferred one return self._select_preferred_dependency(possibles).dependency except Exception: log.exception("Unexpected error hunting for dependency '%s' '%s''%s'", name, version, type) return NullDependency(version=version, name=name) # Finds all possible dependency to use # Should be extended as required # Returns CandidateDependency objects with data for preference picking def _find_possible_dependencies(self, name, version, type): possibles = [] if exists(self.base_path): path = join(self.base_path, name, version) if exists(path): # First try the way without owner/name/revision package = self._galaxy_package_dep(path, version, name, type, True) if not isinstance(package, NullDependency): log.debug("Found dependency '%s' '%s' '%s' at '%s'", name, version, type, path) possibles.append(CandidateDependency(package, path)) # now try with an owner/name/revision for owner in listdir(path): owner_path = join(path, owner) for package_name in listdir(owner_path): if package_name.lower().startswith(f"package_{name.lower()}"): package_path = join(owner_path, package_name) for revision in listdir(package_path): revision_path = join(package_path, revision) package = self._galaxy_package_dep(revision_path, version, name, type, True) if not isinstance(package, NullDependency): log.debug( "Found dependency '%s' '%s' '%s' at '%s'", name, version, type, revision_path ) possibles.append(CandidateDependency(package, package_path, owner)) return possibles def _select_preferred_dependency(self, possibles, by_owner=None): if by_owner is None: by_owner = self.select_by_owner preferred = [] if by_owner: for owner in self.preferred_owners: for candidate in possibles: if candidate.owner == owner: preferred.append(candidate) if len(preferred) == 1: log.debug("Picked dependency based on owner '%s'", owner) return preferred[0] elif len(preferred) > 1: log.debug("Multiple dependency found with owner '%s'", owner) break if len(preferred) == 0: preferred = possibles latest_modified = 0 for candidate in preferred: modified = getmtime(candidate.path) if latest_modified < modified: latest_candidate = candidate latest_modified = modified log.debug("Picking dependency at '%s' as it was the last modified", latest_candidate.path) return latest_candidate class CandidateDependency(Dependency): dict_collection_visible_keys = Dependency.dict_collection_visible_keys + ["dependency", "path", "owner"] dependency_type = "unlinked_tool_shed_package" @property def exact(self): return self.dependency.exact def __init__(self, dependency, path, owner=MANUAL): self.dependency = dependency self.path = path self.owner = owner def shell_commands(self): """ Return shell commands to enable this dependency. """ return self.dependency.shell_commands() __all__ = ("UnlinkedToolShedPackageDependencyResolver",) """ At the time of writing July 3 2015 this resolver has to be plugged in. Adding resolver instructions: 1. create a dependency_resolvers_config.xml file 1a. ALWAYS add first!!!! 1b. is optional as this resolver will also find dependency found by that resolver 1bi Current default is to use a dependency to find that way first! 1bii So an alternative version of dependency_resolvers_config.xml 1c. See __init__ for optional config values 1ci versionless currently is handled by the super class GalaxyPackageDependencyResolver 2. Add a parameter to config.ini dependency_resolvers_config_file = ./config/dependency_resolvers_config.xml 2a. File name/path can be different 2b. config key must be dependency_resolvers_config_file 3. For planemo it may be required to specify: --dependency_resolvers_config_file (xml file described in 1 above) --tool_dependency_dir (root of dependencies typically galaxy/dependency_dir) See planemo test --help for more information """ ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/singularity_util.py0000644000175100017510000001055215234436334025642 0ustar00runnerrunnerimport os import shlex from typing import ( List, Optional, Tuple, TYPE_CHECKING, Union, ) if TYPE_CHECKING: from .container_volumes import DockerVolume DEFAULT_WORKING_DIRECTORY = None DEFAULT_SINGULARITY_COMMAND = "singularity" # --pid isolates the pid namespace. --ipc isolates # the ipc namespace, this fixes some issues with python multiprocessing. # --cleanenv makes sure the current environment is not inherited. DEFAULT_CLEANENV = True DEFAULT_IPC = True DEFAULT_PID = True DEFAULT_CONTAIN = True DEFAULT_NO_MOUNT = ["tmp"] DEFAULT_SUDO = False DEFAULT_SUDO_COMMAND = "sudo" DEFAULT_RUN_EXTRA_ARGUMENTS = None def pull_mulled_singularity_command( docker_image_identifier: str, cache_directory: str, namespace: Optional[str] = None, singularity_cmd: str = DEFAULT_SINGULARITY_COMMAND, sudo: bool = DEFAULT_SUDO, sudo_cmd: str = DEFAULT_SUDO_COMMAND, ) -> List[str]: command_parts = [] command_parts += _singularity_prefix( singularity_cmd=singularity_cmd, sudo=sudo, sudo_cmd=sudo_cmd, ) save_path = docker_image_identifier if namespace: prefix = f"docker://quay.io/{namespace}/" if docker_image_identifier.startswith(prefix): save_path = docker_image_identifier[len(prefix) :] command_parts.extend(["build", os.path.join(cache_directory, save_path), docker_image_identifier]) return command_parts def pull_singularity_command( image_identifier: str, cache_path: str, singularity_cmd: str = DEFAULT_SINGULARITY_COMMAND, sudo: bool = DEFAULT_SUDO, sudo_cmd: str = DEFAULT_SUDO_COMMAND, ) -> List[str]: # Make sure cache dir exists dirname = os.path.dirname(os.path.normpath(cache_path)) os.makedirs(dirname, exist_ok=True) command_parts = _singularity_prefix(singularity_cmd, sudo, sudo_cmd) command_parts.extend(["build", cache_path, image_identifier]) return command_parts def build_singularity_run_command( container_command: str, image: str, volumes: Optional[List["DockerVolume"]] = None, env: Optional[List[Tuple[str, str]]] = None, working_directory: Optional[str] = DEFAULT_WORKING_DIRECTORY, singularity_cmd: str = DEFAULT_SINGULARITY_COMMAND, run_extra_arguments: Optional[str] = DEFAULT_RUN_EXTRA_ARGUMENTS, sudo: bool = DEFAULT_SUDO, sudo_cmd: str = DEFAULT_SUDO_COMMAND, guest_ports: Union[bool, List[str]] = False, container_name: Optional[str] = None, cleanenv: bool = DEFAULT_CLEANENV, ipc: bool = DEFAULT_IPC, pid: bool = DEFAULT_PID, contain: bool = DEFAULT_CONTAIN, no_mount: Optional[List[str]] = DEFAULT_NO_MOUNT, ) -> str: volumes = volumes or [] env = env or [] command_parts = [] # http://singularity.lbl.gov/docs-environment-metadata home = None for key, value in env: if key == "HOME": home = value command_parts.extend([f'SINGULARITYENV_{key}="{value}"']) command_parts += _singularity_prefix( singularity_cmd=singularity_cmd, sudo=sudo, sudo_cmd=sudo_cmd, ) command_parts.append("-s") command_parts.append("exec") if contain: command_parts.append("--contain") if working_directory: command_parts.extend(["--pwd", shlex.quote(working_directory)]) if cleanenv: command_parts.append("--cleanenv") if ipc: command_parts.append("--ipc") if pid: command_parts.append("--pid") if no_mount: command_parts.extend(["--no-mount", ",".join(no_mount)]) for volume in volumes: command_parts.extend(["-B", str(volume)]) if home is not None: command_parts.extend(["--home", f"{home}:{home}"]) if run_extra_arguments: command_parts.append(run_extra_arguments) full_image = image command_parts.append(shlex.quote(full_image)) command_parts.append(container_command) return " ".join(command_parts) def _singularity_prefix( singularity_cmd: str = DEFAULT_SINGULARITY_COMMAND, sudo: bool = DEFAULT_SUDO, sudo_cmd: str = DEFAULT_SUDO_COMMAND, **kwds, ) -> List[str]: """Prefix to issue a singularity command.""" command_parts = [] if sudo: command_parts.append(sudo_cmd) command_parts.append(singularity_cmd) return command_parts __all__ = ("build_singularity_run_command", "pull_mulled_singularity_command", "pull_singularity_command") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/deps/views.py0000644000175100017510000004401115234436334023365 0ustar00runnerrunnerfrom typing import ( Any, Dict, List, Optional, TYPE_CHECKING, ) from requests import Session from typing_extensions import ( NotRequired, TypedDict, ) from galaxy import exceptions from galaxy.util import ( asbool, listify, ) from .dependencies import ToolInfo from .resolvers import ( ContainerDependency, NullDependency, ) if TYPE_CHECKING: from galaxy.structured_app import StructuredApp from galaxy.tool_util.deps.container_resolvers import ResolutionCache from .container_resolvers import ContainerResolver class DependencyResolversView: """Provide a RESTfulish/JSONy interface to a galaxy.tool_util.deps.DependencyResolver object. This can be adapted by the Galaxy web framework or other web apps. """ def __init__(self, app): self._app = app def index(self): return [r.to_dict() for r in self._dependency_resolvers] def show(self, index): return self._dependency_resolver(index).to_dict() def reload(self): self.toolbox.reload_dependency_manager() def manager_requirements(self): requirements = [] for index, resolver in enumerate(self._dependency_resolvers): if not hasattr(resolver, "list_dependencies"): continue for requirement in resolver.list_dependencies(): requirements.append({"index": index, "requirement": requirement.to_dict()}) return requirements def resolver_requirements(self, index): requirements = [] resolver = self._dependency_resolver(index) if not hasattr(resolver, "list_dependencies"): raise exceptions.NotImplemented() for requirement in resolver.list_dependencies(): requirements.append(requirement.to_dict()) return requirements def manager_dependency(self, **kwds): return self._dependency(**kwds) def resolver_dependency(self, index, **kwds): return self._dependency(**kwds) def show_dependencies(self, tool_requirements_d, installed_tool_dependencies=None, **kwd): """ Resolves dependencies to build a requirements status in the admin panel/API """ to_deps_kwds = { "install": False, "return_null": True, "installed_tool_dependencies": installed_tool_dependencies, } to_deps_kwds.update(kwd) dependencies_per_tool = { tool: self._dependency_manager.requirements_to_dependencies(requirements, **to_deps_kwds) for tool, requirements in tool_requirements_d.items() } return dependencies_per_tool def uninstall_dependencies(self, index=None, resolver_type=None, container_type=None, **payload): """Attempt to uninstall requirements. Returns 0 if successfull, else None.""" requirements = payload.get("requirements") if not requirements: return None if index: resolver = self._dependency_resolvers[index] if resolver.can_uninstall_dependencies: return resolver.uninstall(requirements) elif resolver_type: for resolver in self._dependency_resolvers: if resolver.resolver_type == resolver_type and resolver.can_uninstall_dependencies: return resolver.uninstall(requirements) elif container_type: for resolver in self._dependency_resolvers: if getattr(resolver, "container_type", None) == container_type and resolver.can_uninstall_dependencies: return resolver.uninstall(requirements) else: for index in self.uninstallable_resolvers: return_code = self._dependency_resolvers[index].uninstall(requirements) if return_code == 0: return return_code return None @property def unused_dependency_paths(self): """List dependencies that are not currently installed.""" unused_dependencies = [] toolbox_requirements_status = self.toolbox_requirements_status for resolver in self._dependency_resolvers: if hasattr(resolver, "unused_dependency_paths"): unused_dependencies.extend(resolver.unused_dependency_paths(toolbox_requirements_status)) return set(unused_dependencies) def remove_unused_dependency_paths(self, envs): """ Remove dependencies that are not currently used. Returns a list of all environments that have been successfully removed. """ envs_to_remove = set(envs) toolbox_requirements_status = self.toolbox_requirements_status removed_environments = set() for resolver in self._dependency_resolvers: if hasattr(resolver, "unused_dependency_paths") and hasattr(resolver, "uninstall_environments"): unused_dependencies = resolver.unused_dependency_paths(toolbox_requirements_status) can_remove = envs_to_remove & set(unused_dependencies) exit_code = resolver.uninstall_environments(can_remove) if exit_code == 0: removed_environments = removed_environments.union(can_remove) envs_to_remove = envs_to_remove.difference(can_remove) return list(removed_environments) def install_dependencies(self, requirements, **kwds): kwds["install"] = True return self._dependency_manager._requirements_to_dependencies_dict(requirements, **kwds) def install_dependency(self, index=None, **payload): """ Installs dependency using highest priority resolver that supports dependency installation (Currently only the conda resolver supports this). If index is given, attempt installation directly using the corresponding resolver. Returns True on success, False on failure. payload is dictionary that must container name, version and type, e.g. {'name': 'numpy', version='1.9.1', type='package'} """ if index: return self._install_dependency(index, **payload) else: for index in self.installable_resolvers: success = self._install_dependency(index, **payload) if success: return success return False def _install_dependency(self, index, **payload): """ Resolver install dependency should return True when installation succeeds, False if not successful """ resolver = self._dependency_resolver(index) if resolver.read_only: raise exceptions.RequestParameterInvalidException( "Attempted to install on a read_only dependency resolver." ) if resolver.disabled: raise exceptions.RequestParameterInvalidException("Attempted to install on a disabled dependency resolver.") name, version, type, extra_kwds = self._parse_dependency_info(payload) return resolver.install_dependency(name=name, version=version, type=type, **extra_kwds) def _dependency(self, index=None, **kwds): if index is not None: index = int(index) name, version, type, extra_kwds = self._parse_dependency_info(kwds) resolve_kwds = dict(job_directory=None, index=index, **extra_kwds) dependency = self._dependency_manager.find_dep(name, version=version, type=type, **resolve_kwds) return dependency.to_dict() def _parse_dependency_info(self, kwds): extra_kwds = kwds.copy() name = extra_kwds.pop("name", None) if name is None: raise exceptions.RequestParameterMissingException("Missing 'name' parameter required for resolution.") version = extra_kwds.pop("version", None) type = extra_kwds.pop("type", "package") return name, version, type, extra_kwds def _dependency_resolver(self, index): index = int(index) return self._dependency_resolvers[index] @property def _dependency_manager(self): return self._app.toolbox.dependency_manager @property def _dependency_resolvers(self): dependency_manager = self._dependency_manager dependency_resolvers = dependency_manager.dependency_resolvers return dependency_resolvers @property def installable_resolvers(self): """ List index for all active resolvers that have the 'install_dependency' attribute. """ return [ index for index, resolver in enumerate(self._dependency_resolvers) if not resolver.read_only and not resolver.disabled ] @property def uninstallable_resolvers(self): """ List index for all active resolvers that can uninstall dependencies that have been installed through this resolver. """ return [ index for index, resolver in enumerate(self._dependency_resolvers) if resolver.can_uninstall_dependencies and not resolver.disabled ] @property def tool_ids_by_requirements(self): """Dictionary with requirements as keys, and tool_ids as values.""" tool_ids_by_requirements = {} if not self._app.toolbox.tools_by_id: return {} for tid, tool in self._app.toolbox.tools_by_id.items(): if tool.tool_requirements not in tool_ids_by_requirements: tool_ids_by_requirements[tool.tool_requirements] = [tid] else: tool_ids_by_requirements[tool.tool_requirements].append(tid) return tool_ids_by_requirements @property def toolbox_requirements_status(self): return self.summarize_requirements() def summarize_requirements(self, **kwds): summary_kwds = {} if "index" in kwds: summary_kwds["index"] = int(kwds["index"]) if "include_containers" in kwds: summary_kwds["include_containers"] = asbool(kwds["include_containers"]) if "container_type" in kwds: summary_kwds["container_type"] = kwds["container_type"] if "resolver_type" in kwds: summary_kwds["resolver_type"] = kwds["resolver_type"] if "search" in kwds: summary_kwds["search"] = asbool(kwds["search"]) if "install" in kwds: summary_kwds["install"] = asbool(kwds["install"]) tool_ids_by_requirements = self.tool_ids_by_requirements statuses = { r: self.get_requirements_status( tool_requirements_d={tids[0]: r}, installed_tool_dependencies=self._app.toolbox.tools_by_id[tids[0]].installed_tool_dependencies, **summary_kwds, ) for r, tids in tool_ids_by_requirements.items() } if kwds.get("for_json", False): # All public attributes of this class should be returning JSON - this is meant to mimic a restful API. rval = [] for requirements, status in statuses.items(): item = {} item["requirements"] = requirements.to_dict() item["status"] = status item["tool_ids"] = tool_ids_by_requirements[requirements] rval.append(item) statuses = rval return statuses def summarize_tools(self, **kwds): summary_kwds = {} if "index" in kwds: summary_kwds["index"] = int(kwds["index"]) if "include_containers" in kwds: summary_kwds["include_containers"] = asbool(kwds["include_containers"]) if "container_type" in kwds: summary_kwds["container_type"] = kwds["container_type"] if "resolver_type" in kwds: summary_kwds["resolver_type"] = kwds["resolver_type"] if "search" in kwds: summary_kwds["search"] = asbool(kwds["search"]) if "install" in kwds: summary_kwds["install"] = asbool(kwds["install"]) tool_ids = pop_tool_ids(kwds) rval = [] for tid, tool in self._app.toolbox.tools_by_id.items(): if tool_ids and tid not in tool_ids: continue requirements = tool.tool_requirements status = self.get_requirements_status( tool_requirements_d={tid: requirements}, installed_tool_dependencies=tool.installed_tool_dependencies, tool_instance=tool, **summary_kwds, ) item = {} item["requirements"] = requirements.to_dict() item["status"] = status item["tool_ids"] = [tid] rval.append(item) return rval def get_requirements_status(self, tool_requirements_d, installed_tool_dependencies=None, **kwd): dependencies = self.show_dependencies(tool_requirements_d, installed_tool_dependencies, **kwd) # dependencies is a dict keyed on tool_ids, value is a ToolRequirements object for that tool. # We use the union of resolvable ToolRequirements to get resolved dependencies without duplicates. requirements = [r.resolvable for r in tool_requirements_d.values()] flat_tool_requirements = set().union(*requirements) flat_dependencies = [] for requirements_odict in dependencies.values(): for requirement in requirements_odict: if requirement in flat_tool_requirements: flat_dependencies.append(requirements_odict[requirement]) flat_tool_requirements.remove(requirement) return [d.to_dict() for d in flat_dependencies] def clean(self, index=None, **kwds): if index: resolver = self._dependency_resolver(index) if not hasattr(resolver, "clean"): raise exceptions.NotImplemented() else: resolver.clean() return "OK" else: [resolver.clean(**kwds) for resolver in self._dependency_resolvers if hasattr(resolver, "clean")] return "OK" class ContainerResolutionView: """ """ def __init__(self, app: "StructuredApp"): self._app = app def index(self) -> List[Dict[str, Any]]: return [r.to_dict() for r in self._container_resolvers] def show(self, index: str) -> Dict[str, Any]: return self._container_resolver(int(index)).to_dict() def resolve(self, index: Optional[str] = None, **kwds) -> Dict[str, Any]: class ResolveKwds(TypedDict): install: bool enabled_container_types: List["str"] tool_info: "ToolInfo" resolution_cache: NotRequired["ResolutionCache"] session: NotRequired["Session"] index: NotRequired[int] resolver_type: NotRequired[str] tool_info_kwds = {} tool_id = kwds["tool_id"] tool = self._app.toolbox.tools_by_id[tool_id] requirements = tool.tool_requirements tool_info_kwds = dict(requirements=requirements) requirements_only = asbool(kwds.get("requirements_only", False)) # If requirements_only, simply use the tool to load a requirement set from, # mimics the default behavior of searching for containers through the dependency resolution # component. Not useful for tool execution but perhaps when summarizing mulled containers for requirements. if not requirements_only: tool_info_kwds["container_descriptions"] = tool.containers tool_info_kwds["requires_galaxy_python_environment"] = tool.requires_galaxy_python_environment tool_info_kwds["tool_id"] = tool.id tool_info_kwds["tool_version"] = tool.version find_best_kwds = ResolveKwds( install=False, enabled_container_types=["docker", "singularity"], tool_info=ToolInfo(**tool_info_kwds) ) if "resolution_cache" in kwds: find_best_kwds["resolution_cache"] = kwds["resolution_cache"] if "session" in kwds: find_best_kwds["session"] = kwds["session"] if index is not None: find_best_kwds["index"] = int(index) if "container_type" in kwds: find_best_kwds["enabled_container_types"] = [kwds["container_type"]] if "resolver_type" in kwds: find_best_kwds["resolver_type"] = kwds["resolver_type"] if "install" in kwds: find_best_kwds["install"] = asbool(kwds["install"]) # Consider implementing 'search' to match dependency resolution API. resolved_container_description = self._app.container_finder.resolve(**find_best_kwds) if resolved_container_description: status = ContainerDependency( resolved_container_description.container_description, container_resolver=resolved_container_description.container_resolver, ).to_dict() else: status = NullDependency().to_dict() return {"tool_id": kwds["tool_id"], "status": status, "requirements": requirements.to_dict()} def resolve_toolbox(self, **kwds) -> List[Dict[str, Any]]: rval = [] resolve_kwds = kwds.copy() tool_ids = pop_tool_ids(resolve_kwds) resolve_kwds["resolution_cache"] = self._app.container_finder.resolution_cache() if tool_ids is not None: tool_ids = listify(tool_ids) for tool_id, tool in self._app.toolbox.tools_by_id.items(): if tool_ids is not None and tool_id not in tool_ids: continue if tool.tool_action.produces_real_jobs: rval.append(self.resolve(tool_id=tool_id, **resolve_kwds)) return rval @property def _container_resolvers(self) -> List["ContainerResolver"]: return self._app.container_finder.default_container_registry.container_resolvers def _container_resolver(self, index: int): return self._container_resolvers[index] def pop_tool_ids(kwds: Dict[str, Any]) -> Optional[List[str]]: tool_ids = None if "tool_ids" in kwds: tool_ids = listify(kwds.pop("tool_ids")) if "tool_ids[]" in kwds: tool_ids = listify(kwds.pop("tool_ids[]")) if "tool_id" in kwds: tool_ids = [kwds.pop("tool_id")] return tool_ids ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/edam_util.py0000644000175100017510000000503115234436334023237 0ustar00runnerrunnerimport os from typing import ( Dict, Optional, TextIO, ) try: from edam_ontology.streams import tabular_stream except ImportError: tabular_stream = None EDAM_PREFIX = "http://edamontology.org/" ROOT_OPERATION = "operation_0004" ROOT_TOPIC = "topic_0003" def load_edam_tree(path: Optional[str] = None, *included_terms: str): if path is not None: assert os.path.exists(path), f"Failed to load EDAM tabular data at [{path}] path does not exist." handle = open(path) else: assert ( tabular_stream is not None ), "Failed to load optional import from edam-ontology package, install using [pip install edam-ontology]." handle = tabular_stream() return load_edam_tree_from_tsv_stream(handle, *included_terms) def load_edam_tree_from_tsv_stream(tsv_stream: TextIO, *included_terms: str): edam: Dict[str, Dict] = {} def _recurse_edam_parents(term, path=None): if edam[term]["parents"] and len(edam[term]["parents"]) > 0: for parent in edam[term]["parents"]: yield from _recurse_edam_parents(parent, path + [parent]) else: yield path is_first = True for line in tsv_stream.readlines(): fields = line.split("\t") if is_first: columns = {} for i, field in enumerate(fields): columns[field] = i is_first = False definition_column = columns["http://www.geneontology.org/formats/oboInOwl#hasDefinition"] term_column = columns["Class ID"] label_column = columns["Preferred Label"] parents_column = columns["Parents"] continue term = fields[term_column] if not term.startswith(EDAM_PREFIX): continue term_id = term[len(EDAM_PREFIX) :] # Only care about included terms if included_terms and not (term_id.startswith(included_terms)): continue parents = fields[parents_column].split("|") edam[term_id] = { "label": fields[label_column].strip('"'), "definition": fields[definition_column].strip('"'), "parents": [x[len(EDAM_PREFIX) :] for x in parents if x.startswith(EDAM_PREFIX)], } for term in sorted(edam.keys()): tails = [] for x in _recurse_edam_parents(term, path=[]): if x[-2:] not in tails: tails.append(x[-2:]) edam[term]["path"] = tails return edam __all__ = ("load_edam_tree", "ROOT_OPERATION", "ROOT_TOPIC") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/fetcher.py0000644000175100017510000000226615234436334022723 0ustar00runnerrunnerimport os from typing import ( Dict, Type, TYPE_CHECKING, ) from galaxy.util import plugin_config if TYPE_CHECKING: from galaxy.tool_util.locations import ToolLocationResolver from galaxy.util.path import StrPath class ToolLocationFetcher: def __init__(self): self.resolver_classes = self.__resolvers_dict() def __resolvers_dict(self) -> Dict[str, Type["ToolLocationResolver"]]: import galaxy.tool_util.locations return plugin_config.plugins_dict(galaxy.tool_util.locations, "scheme") def to_tool_path(self, path_or_uri_like: "StrPath", **kwds) -> "StrPath": if isinstance(path_or_uri_like, os.PathLike) or "://" not in path_or_uri_like: path = path_or_uri_like else: uri_like = path_or_uri_like if ":" not in path_or_uri_like: raise Exception("Invalid URI passed to get_tool_source") scheme = uri_like.split(":", 2)[0] if scheme not in self.resolver_classes: raise Exception(f"Unknown tool scheme [{scheme}] for URI [{uri_like}]") path = self.resolver_classes[scheme]().get_tool_source_path(uri_like) return path ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/format.py0000644000175100017510000000424315234436334022570 0ustar00runnerrunnerimport argparse import difflib import sys from lxml import etree DESCRIPTION = """Format Galaxy tool XML files with consistent indentation.""" def format_xml(content: str, tab_size: int = 4) -> str: """Format XML content with consistent indentation. Used by planemo's format command and the Galaxy Language Server to apply uniform formatting to Galaxy tool XML files. """ try: parser = etree.XMLParser(strip_cdata=False) xml = etree.fromstring(content, parser=parser) etree.indent(xml, space=" " * tab_size) return etree.tostring(xml, pretty_print=True, encoding=str) except etree.XMLSyntaxError: return content def arg_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(description=DESCRIPTION) parser.add_argument("xml_files", nargs="+", metavar="FILE", help="Galaxy tool XML file(s) to format") parser.add_argument( "-t", "--tab-size", type=int, default=4, help="Number of spaces per indentation level (default: 4)" ) parser.add_argument("-d", "--diff", action="store_true", help="Show diff of changes instead of formatting in-place") return parser def main(argv=None) -> None: if argv is None: argv = sys.argv[1:] args = arg_parser().parse_args(argv) has_diff = False for path in args.xml_files: with open(path) as f: original = f.read() formatted = format_xml(original, tab_size=args.tab_size) changed = original != formatted if args.diff: if changed: has_diff = True diff = difflib.unified_diff( original.splitlines(keepends=True), formatted.splitlines(keepends=True), fromfile=path, tofile=path, ) sys.stdout.writelines(diff) else: if changed: with open(path, "w") as f: f.write(formatted) print(f"reformatted {path}") else: print(f"{path} already formatted") if args.diff and has_diff: sys.exit(1) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/identifiers.py0000644000175100017510000000044415234436334023604 0ustar00runnerrunnerfrom urllib.parse import quote def uri_safe_tool_id(tool_id: str) -> str: # Shed tool ids contain ``+`` in their version suffix (e.g. # ``.../1.3.1+galaxy1``). In a query string the ``+`` decodes to a # literal space unless percent-encoded as ``%2B``. return quote(tool_id) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/lint.py0000644000175100017510000003536415234436334022256 0ustar00runnerrunner"""This modules contains the functions that drive the tool linting framework. LintContext: a container for LintMessages LintMessage: the actual message and level The idea is to define a LintContext and to apply a linting function ``foo`` on a ``target``. The ``level`` (defined by ``LintLevel``) determines which linting messages are shown:: lint_ctx = LintContext(level) # level is the reporting level lint_ctx.lint(..., lint_func = foo, lint_target = target, ...) The ``lint`` function essentially calls ``foo(target, self)``. Hence the function ``foo`` must have two parameters: 1. the object to lint 2. the lint context In ``foo`` the lint context can be used to add LintMessages to the lint context by using its ``valid``, ``info``, ``warn``, and ``error`` functions:: lint_foo(target, lint_ctx): lint_ctx.error("target is screwed") Calling ``lint`` prints out the messages emmited by the linter function immediately. Which messages are shown can be determined with the ``level`` argument of the ``LintContext`` constructor. If set to ``SILENT``, no messages will be printed. For special lint targets it might be useful to have additional information in the lint messages. This can be achieved by subclassing ``LintMessage``. See for instance ``XMLLintMessageLine`` which has an additional argument ``node`` in its constructor which is used to determine the line and filename in an XML document that caused the message. In order to use this. - the lint context needs to be initialized with the additional parameter ``lint_message_class=XMLLintMessageLine`` - the additional parameter needs to be added as well to calls adding messages to the lint context, e.g. ``lint_ctx.error("some message", node=X)``. Note that the additional properties must be given as keyword arguments. """ import inspect from abc import ( ABC, abstractmethod, ) from enum import IntEnum from typing import ( Callable, List, Optional, Type, TYPE_CHECKING, TypeVar, Union, ) import galaxy.tool_util.linters from galaxy.tool_util.parser import get_tool_source from galaxy.tool_util.parser.yaml import YamlToolSource from galaxy.util import ( Element, submodules, ) if TYPE_CHECKING: from galaxy.tool_util.parser.interface import ToolSource from galaxy.tool_util_models import UserToolSource class LintLevel(IntEnum): SILENT = 5 ERROR = 4 WARN = 3 INFO = 2 VALID = 1 ALL = 0 class Linter(ABC): """ a linter. needs to define a lint method and the code property. optionally a fix method can be given """ @classmethod @abstractmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): """ should add at most one message to the lint context """ pass @classmethod def name(cls) -> str: """ get the linter name """ return cls.__name__ @classmethod def list_linters(cls) -> List[str]: """ list the names of all linter derived from Linter """ submodules.import_submodules(galaxy.tool_util.linters) return [s.__name__ for s in cls.__subclasses__()] list_listers: Callable[[], List[str]] # deprecated alias # Define the `list_listers` alias outside of the `Linter` class so that # @classmethod's change to `list_linters`s signature has taken effect and mypy # doesn't report an [assignment] error Linter.list_listers = Linter.list_linters class LintMessage: """ a message from the linter """ def __init__(self, level: str, message: str, linter: Optional[str] = None, **kwargs): self.level = level self.message = message self.linter = linter def __eq__(self, other) -> bool: """ add equal operator to easy lookup of a message in a List[LintMessage] which is useful in tests. If the other object is a string, it is loosely checked if the string is contained in the message. """ if isinstance(other, str): return other in self.message if isinstance(other, LintMessage): return self.message == other.message return False def __str__(self) -> str: if self.linter: linter = f" ({self.linter})" else: linter = "" return f".. {self.level.upper()}{linter}: {self.message}" def __repr__(self) -> str: return f"LintMessage({self.message})" class XMLLintMessageLine(LintMessage): def __init__(self, level: str, message: str, linter: Optional[str] = None, node: Optional[Element] = None): super().__init__(level, message, linter) self.line = None if node is not None: self.line = node.sourceline def __str__(self) -> str: rval = super().__str__() if self.line is not None: rval += " (" rval += str(self.line) rval += ")" return rval class XMLLintMessageXPath(LintMessage): def __init__(self, level: str, message: str, linter: Optional[str] = None, node: Optional[Element] = None): super().__init__(level, message, linter) self.xpath = None if node is not None: tool_xml = node.getroottree() self.xpath = tool_xml.getpath(node) def __str__(self) -> str: rval = super().__str__() if self.xpath is not None: rval += f" [{self.xpath}]" return rval LintTargetType = TypeVar("LintTargetType") # TODO: Nothing inherently tool-y about LintContext and in fact # it is reused for repositories in planemo. Therefore, it should probably # be moved to galaxy.util.lint. class LintContext: skip_types: List[str] level: LintLevel lint_message_class: Type[LintMessage] object_name: Optional[str] message_list: List[LintMessage] def __init__( self, level: Union[LintLevel, str], lint_message_class: Type[LintMessage] = LintMessage, skip_types: Optional[List[str]] = None, object_name: Optional[str] = None, ): self.skip_types = skip_types or [] if isinstance(level, str): self.level = LintLevel[level.upper()] else: self.level = level self.lint_message_class = lint_message_class self.object_name = object_name self.message_list = [] @property def found_errors(self) -> bool: return len(self.error_messages) > 0 @property def found_warns(self) -> bool: return len(self.warn_messages) > 0 def lint( self, name: str, lint_func: Callable[[LintTargetType, "LintContext"], None], lint_target: LintTargetType, module_name: Optional[str] = None, ): if name.startswith("lint_"): name = name[len("lint_") :] if name in self.skip_types: return if module_name and module_name in self.skip_types: return if self.level < LintLevel.SILENT: # this is a relict from the past where the lint context # was reset when called with a new lint_func, as workaround # we save the message list, apply the lint_func (which then # adds to the message_list) and restore the message list # at the end (+ append the new messages) tmp_message_list = list(self.message_list) self.message_list = [] # call linter lint_func(lint_target, self) if self.level < LintLevel.SILENT: for message in self.error_messages: print(f"{message}") if self.level <= LintLevel.WARN: for message in self.warn_messages: print(f"{message}") if self.level <= LintLevel.INFO: for message in self.info_messages: print(f"{message}") if self.level <= LintLevel.VALID: for message in self.valid_messages: print(f"{message}") self.message_list = tmp_message_list + self.message_list @property def valid_messages(self) -> List[LintMessage]: return [x for x in self.message_list if x.level == "check"] @property def info_messages(self) -> List[LintMessage]: return [x for x in self.message_list if x.level == "info"] @property def warn_messages(self) -> List[LintMessage]: return [x for x in self.message_list if x.level == "warning"] @property def error_messages(self) -> List[LintMessage]: return [x for x in self.message_list if x.level == "error"] def __handle_message(self, level: str, message: str, linter: Optional[str] = None, *args, **kwargs) -> None: if args: message = message % args self.message_list.append(self.lint_message_class(level=level, message=message, linter=linter, **kwargs)) def valid(self, message: str, linter: Optional[str] = None, *args, **kwargs) -> None: self.__handle_message("check", message, linter, *args, **kwargs) def info(self, message: str, linter: Optional[str] = None, *args, **kwargs) -> None: self.__handle_message("info", message, linter, *args, **kwargs) def error(self, message: str, linter: Optional[str] = None, *args, **kwargs) -> None: self.__handle_message("error", message, linter, *args, **kwargs) def warn(self, message: str, linter: Optional[str] = None, *args, **kwargs) -> None: self.__handle_message("warning", message, linter, *args, **kwargs) def failed(self, fail_level: Union[LintLevel, str]) -> bool: if isinstance(fail_level, str): fail_level = LintLevel[fail_level.upper()] found_warns = self.found_warns found_errors = self.found_errors if fail_level == LintLevel.WARN: lint_fail = found_warns or found_errors elif fail_level >= LintLevel.ERROR: lint_fail = found_errors return lint_fail # Linters that reach out to external services (bio.tools API, EDAM ontology). # Skipped when linting a UserToolSource during create/edit so the API call # doesn't hang or fail when those services are unreachable. NETWORK_LINTERS = ("BioToolsValid", "EDAMTermsValid") def lint_user_tool_source(user_tool_source: "UserToolSource") -> List[str]: """Run the lint pipeline against a ``UserToolSource`` pydantic value. Returns a list of formatted ``": "`` bullets at WARN level or above, suitable for surfacing through ``format_validation_errors``- style consumers (the agent's bullet list, an API 4xx body). Network-touching linters (``NETWORK_LINTERS``) are skipped; callers create/edit tools synchronously and should not block on third-party services. """ root_dict = user_tool_source.model_dump(by_alias=True, exclude_none=True) tool_source = YamlToolSource(root_dict) lint_context = get_lint_context_for_tool_source(tool_source, skip_types=list(NETWORK_LINTERS)) bullets: List[str] = [] for message in lint_context.error_messages + lint_context.warn_messages: prefix = f"{message.linter}: " if message.linter else "" bullets.append(f"{prefix}{message.message}") return bullets def lint_tool_source( tool_source, level=LintLevel.ALL, fail_level=LintLevel.WARN, extra_modules=None, skip_types=None, name=None ) -> bool: """ apply all (applicable) linters from the linters submodule and the ones in extramodules immediately print linter messages (wrt level) and return if linting failed (wrt fail_level) """ extra_modules = extra_modules or [] skip_types = skip_types or [] lint_context = LintContext(level=level, skip_types=skip_types, object_name=name) lint_tool_source_with(lint_context, tool_source, extra_modules) return not lint_context.failed(fail_level) def get_lint_context_for_tool_source(tool_source, extra_modules=None, skip_types=None, name=None) -> LintContext: """ this is the silent variant of lint_tool_source it returns the LintContext from which all linter messages and the status can be obtained """ extra_modules = extra_modules or [] skip_types = skip_types or [] lint_context = LintContext(level=LintLevel.SILENT, skip_types=skip_types, object_name=name) lint_tool_source_with(lint_context, tool_source, extra_modules) return lint_context def lint_xml( tool_xml, level=LintLevel.ALL, fail_level=LintLevel.WARN, lint_message_class=LintMessage, extra_modules=None, skip_types=None, name=None, ) -> bool: """ lint an xml tool """ extra_modules = extra_modules or [] skip_types = skip_types or [] lint_context = LintContext( level=level, lint_message_class=lint_message_class, skip_types=skip_types, object_name=name ) lint_xml_with(lint_context, tool_xml, extra_modules) return not lint_context.failed(fail_level) def lint_tool_source_with(lint_context, tool_source, extra_modules=None) -> LintContext: extra_modules = extra_modules or [] linter_modules = submodules.import_submodules(galaxy.tool_util.linters) linter_modules.extend(extra_modules) return lint_tool_source_with_modules(lint_context, tool_source, linter_modules) def lint_tool_source_with_modules(lint_context: LintContext, tool_source, linter_modules) -> LintContext: tool_xml = getattr(tool_source, "xml_tree", None) tool_type = tool_source.parse_tool_type() or "default" for module in linter_modules: module_name = module.__name__.split(".")[-1] lint_tool_types = getattr(module, "lint_tool_types", ["default", "manage_data"]) if not ("*" in lint_tool_types or tool_type in lint_tool_types): continue for name, value in inspect.getmembers(module): if callable(value) and name.startswith("lint_"): # Look at the first argument to the linter to decide # if we should lint the XML description or the abstract # tool parser object. first_arg = inspect.getfullargspec(value).args[0] if first_arg == "tool_xml": if tool_xml is None: # XML linter and non-XML tool, skip for now continue else: lint_context.lint(name, value, tool_xml) else: lint_context.lint(name, value, tool_source) elif inspect.isclass(value) and issubclass(value, Linter) and not inspect.isabstract(value): lint_context.lint(name, value.lint, tool_source, module_name=module_name) return lint_context def lint_xml_with(lint_context, tool_xml, extra_modules=None) -> LintContext: extra_modules = extra_modules or [] tool_source = get_tool_source(xml_tree=tool_xml) return lint_tool_source_with(lint_context, tool_source, extra_modules=extra_modules) ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1785871615.053486 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/0000755000175100017510000000000015234436377022412 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/__init__.py0000644000175100017510000000010015234436334024503 0ustar00runnerrunner"""This package contains linting functions for Galaxy tools.""" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/_util.py0000644000175100017510000000054415234436334024074 0ustar00runnerrunnerimport re def is_datasource(tool_xml): """Returns true if the tool is a datasource tool""" return tool_xml.getroot().attrib.get("tool_type", "") in ["data_source", "data_source_async"] def is_valid_cheetah_placeholder(name): """Returns true if name is a valid Cheetah placeholder""" return re.match(r"^[a-zA-Z_]\w*$", name) is not None ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/citations.py0000644000175100017510000001160415234436334024754 0ustar00runnerrunner"""This module contains citation linters. Citations describe references that should be used when consumers of the tool publish results. """ from typing import TYPE_CHECKING from pydantic import ValidationError from galaxy.tool_util.lint import Linter from galaxy.tool_util.version import parse_version from galaxy.tool_util_models.tool_source import Citation # Profile at which invalid citations stop loading fatally rather than being # dropped; see galaxyproject/galaxy#22795. CITATION_FATAL_PROFILE = parse_version("26.1") if TYPE_CHECKING: from galaxy.tool_util.lint import LintContext from galaxy.tool_util.parser.interface import ToolSource class CitationsMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return root = tool_xml.find("./citations") if root is None: root = tool_xml.getroot() citations = tool_xml.findall("citations") if len(citations) == 0: lint_ctx.warn("No citations found, consider adding citations to your tool.", linter=cls.name(), node=root) class CitationsNoText(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return citations = tool_xml.find("citations") if citations is None: return for citation in citations: citation_type = citation.attrib.get("type") if citation_type in ["doi", "bibtex"] and (citation.text is None or not citation.text.strip()): lint_ctx.error(f"Empty {citation_type} citation.", linter=cls.name(), node=citation) class CitationsFound(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return root = tool_xml.find("./citations") if root is None: root = tool_xml.getroot() citations = tool_xml.find("citations") if citations is not None and len(citations) > 0: lint_ctx.valid(f"Found {len(citations)} citations.", linter=cls.name(), node=root) class CitationsInvalid(Linter): """Surface citations that the ``Citation`` model would reject at tool load. Reuses the same pydantic validation the parser uses so the linter reports exactly what fails to load, plus the legacy ``doi:`` prefix that is accepted but normalized. For tools targeting profile >= 26.1 these are errors (an invalid citation prevents loading and the legacy prefix is no longer tolerated); for older profiles they are warnings (the citation is dropped). """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return citations = tool_xml.find("citations") if citations is None: return report = ( lint_ctx.error if parse_version(tool_source.parse_profile()) >= CITATION_FATAL_PROFILE else lint_ctx.warn ) for citation in citations: content = (citation.text or "").strip() if not content: # Empty citations are reported by CitationsNoText. continue # An absent type routes to the model's content-shape check (a useful # message) rather than failing on the required str field. citation_type = citation.attrib.get("type") or "" try: parsed = Citation(type=citation_type, content=content) except ValidationError as e: report( f"Citation '{content}' is invalid and will not load for tools with profile >= 26.1 " f"(and is dropped for older profiles): {e.errors()[0]['msg']}", linter=cls.name(), node=citation, ) continue if parsed.content != content: report( f"Citation '{content}' uses the legacy 'doi:' prefix; use the bare DOI '{parsed.content}' instead.", linter=cls.name(), node=citation, ) class CitationsNoValid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return root = tool_xml.find("./citations") if root is None: root = tool_xml.getroot() citations = tool_xml.findall("citations") if len(citations) != 1: return if len(citations[0]) == 0: lint_ctx.warn("Found no valid citations.", linter=cls.name(), node=root) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/command.py0000644000175100017510000000575315234436334024405 0ustar00runnerrunner"""This module contains linters for a tool's command description. A command description describes how to build the command-line to execute from supplied inputs. """ from typing import TYPE_CHECKING from galaxy.tool_util.lint import Linter if TYPE_CHECKING: from galaxy.tool_util.lint import LintContext from galaxy.tool_util.parser.interface import ToolSource class CommandMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return root = tool_xml.find("./command") if root is None: root = tool_xml.getroot() command = tool_xml.find("./command") if command is None: lint_ctx.error( "No command tag found, must specify a command template to execute.", linter=cls.name(), node=root ) class CommandEmpty(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return root = tool_xml.find("./command") if root is None: root = tool_xml.getroot() command = tool_xml.find("./command") if command is not None and command.text is None: lint_ctx.error("Command is empty.", linter=cls.name(), node=root) class CommandTODO(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return command = tool_xml.find("./command") if command is not None and command.text is not None and "TODO" in command.text: lint_ctx.warn("Command template contains TODO text.", linter=cls.name(), node=command) class CommandInterpreterDeprecated(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return command = tool_xml.find("./command") if command is None: return interpreter_type = command.attrib.get("interpreter", None) if interpreter_type is not None: lint_ctx.warn("Command uses deprecated 'interpreter' attribute.", linter=cls.name(), node=command) class CommandInfo(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return command = tool_xml.find("./command") if command is None: return interpreter_type = command.attrib.get("interpreter", None) interpreter_info = "" if interpreter_type: interpreter_info = f" with interpreter of type [{interpreter_type}]" lint_ctx.info(f"Tool contains a command{interpreter_info}.", linter=cls.name(), node=command) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/containers.py0000644000175100017510000000330315234436334025121 0ustar00runnerrunner"""Linter rules covering container references on a tool source.""" import re from typing import ( Iterator, Tuple, TYPE_CHECKING, ) from galaxy.tool_util.lint import Linter if TYPE_CHECKING: from galaxy.tool_util.lint import LintContext from galaxy.tool_util.parser.interface import ToolSource lint_tool_types = ["*"] CONTAINER_PREFIXES: Tuple[str, ...] = ("quay.io/biocontainers/", "docker://", "oras://") DOCKER_IMAGE_RE = re.compile(r"^[a-zA-Z0-9][a-zA-Z0-9._-]*(/[a-zA-Z0-9._-]+)*(:[\w][\w.-]*)?$") def _iter_container_identifiers(tool_source: "ToolSource") -> Iterator[str]: try: _, containers, _, _, _ = tool_source.parse_requirements() except Exception: return for container in containers or (): identifier = getattr(container, "identifier", None) if identifier: yield identifier class ContainerImageShape(Linter): """Container identifiers should match a recognized shape. Recognized: a `quay.io/biocontainers/...`, `docker://...`, or `oras://...` prefix; or a Docker-Hub-style `[:]` reference. """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext") -> None: for identifier in _iter_container_identifiers(tool_source): stripped = identifier.strip() if stripped.startswith(CONTAINER_PREFIXES): continue if DOCKER_IMAGE_RE.match(stripped): continue lint_ctx.warn( f"container '{identifier}' does not match a recognized shape " "(quay.io/biocontainers/..., docker://..., oras://..., or [:])", linter=cls.name(), ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/cwl.py0000644000175100017510000000700315234436334023542 0ustar00runnerrunner"""Linter for CWL tools.""" from typing import TYPE_CHECKING lint_tool_types = ["cwl"] from galaxy.tool_util.cwl.schema import schema_loader from galaxy.tool_util.lint import Linter if TYPE_CHECKING: from galaxy.tool_util.lint import LintContext from galaxy.tool_util.parser import ToolSource class CWLValid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): raw_reference = schema_loader.raw_process_reference(tool_source.source_path) try: schema_loader.process_definition(raw_reference) except Exception: return lint_ctx.info("CWL appears to be valid.", linter=cls.name()) class CWLInValid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): raw_reference = schema_loader.raw_process_reference(tool_source.source_path) try: schema_loader.process_definition(raw_reference) except Exception as e: lint_ctx.error(f"Failed to valdiate CWL artifact: {e}", linter=cls.name()) class CWLVersionMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): raw_reference = schema_loader.raw_process_reference(tool_source.source_path) cwl_version = raw_reference.process_object.get("cwlVersion", None) if cwl_version is None: lint_ctx.error("CWL file does not contain a 'cwlVersion'", linter=cls.name()) class CWLVersionUnknown(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): raw_reference = schema_loader.raw_process_reference(tool_source.source_path) cwl_version = raw_reference.process_object.get("cwlVersion", None) if cwl_version not in ["v1.0"]: lint_ctx.warn( f"CWL version [{cwl_version}] is unknown, we recommend the v1.0 the stable release.", linter=cls.name() ) class CWLVersionGood(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): raw_reference = schema_loader.raw_process_reference(tool_source.source_path) cwl_version = raw_reference.process_object.get("cwlVersion", None) if cwl_version in ["v1.0"]: lint_ctx.info(f"Modern CWL version [{cwl_version}].", linter=cls.name()) class CWLDockerMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, containers, *_ = tool_source.parse_requirements() if len(containers) == 0: lint_ctx.warn("Tool does not specify a DockerPull source.", linter=cls.name()) class CWLDockerGood(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, containers, *_ = tool_source.parse_requirements() if len(containers) > 0: identifier = containers[0].identifier lint_ctx.info(f"Tool will run in Docker image [{identifier}].", linter=cls.name()) class CWLDescriptionMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): help = tool_source.parse_help() if not help: lint_ctx.warn("Description of tool is empty or absent.", linter=cls.name()) class CWLHelpTODO(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): help = tool_source.parse_help() if help and "TODO" in help.content: lint_ctx.warn("Help contains TODO text.", linter=cls.name()) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/datatypes.py0000644000175100017510000000760715234436334024765 0ustar00runnerrunnerimport os.path from os import getenv from typing import ( Set, TYPE_CHECKING, Union, ) from packaging.version import Version # from galaxy import config from galaxy.tool_util.lint import Linter from galaxy.util import ( listify, parse_xml, ) from galaxy.util.resources import resource_path if TYPE_CHECKING: from galaxy.tool_util.lint import LintContext from galaxy.tool_util.parser import ToolSource from galaxy.util.resources import Traversable DATATYPES_CONF = getenv("DATATYPES_CONF", resource_path(__name__, "datatypes_conf.xml.sample")) def _parse_datatypes(datatype_conf_path: Union[str, "Traversable"]) -> Set[str]: datatypes = set() tree = parse_xml(datatype_conf_path) root = tree.getroot() for elem in root.findall("./registration/datatype"): extension = elem.get("extension", "") datatypes.add(extension) auto_compressed_types = listify(elem.get("auto_compressed_types", "")) for act in auto_compressed_types: datatypes.add(f"{extension}.{act}") return datatypes class DatatypesCustomConf(Linter): """ Check if a custom datatypes_conf.xml is present """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): if not tool_source.source_path: return tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return tool_node = tool_xml.getroot() tool_dir = os.path.dirname(tool_source.source_path) datatypes_conf_path = os.path.join(tool_dir, "datatypes_conf.xml") if os.path.exists(datatypes_conf_path): lint_ctx.warn( "Tool uses a custom datatypes_conf.xml which is discouraged", linter=cls.name(), node=tool_node, ) class ValidDatatypes(Linter): """ Check that used datatypes are available """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return profile = tool_source.parse_profile() # get Galaxy built-in dataypes datatypes = _parse_datatypes(DATATYPES_CONF) # add custom tool data types if tool_source.source_path: tool_dir = os.path.dirname(tool_source.source_path) datatypes_conf_path = os.path.join(tool_dir, "datatypes_conf.xml") if os.path.exists(datatypes_conf_path): datatypes |= _parse_datatypes(datatypes_conf_path) for attrib in ["format", "ftype", "ext"]: for elem in tool_xml.findall(f".//*[@{attrib}]"): # skip help section, "format" in help has a different meaning if elem.tag == "help": continue formats = elem.get(attrib, "").split(",") # Certain elements (e.g. `data`) can only have one format. This # is checked separately by linting against the XSD. if elem.tag == "param": if any(format in ["auto", "input"] for format in formats): lint_ctx.error( "Invalid format (auto or input) in tool or tool test inputs", linter=cls.name(), node=elem, ) continue for format in formats: if Version(str(profile)) <= Version("16.04") and format == "input": continue if format == "auto": continue if format not in datatypes: lint_ctx.error( f"Unknown datatype [{format}] used in {elem.tag} element", linter=cls.name(), node=elem, ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871579.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/datatypes_conf.xml.sample0000644000175100017510000043576415234436333027432 0ustar00runnerrunner The BAM index format "BAI" is automatically created by Galaxy when a BAM file is processed. Usually you don't need to upload this file. SOLID Color-Space sequence data is a rare data type. Consider selecting fasta instead. Generally a more specific fastq datatype should be specified, please consider selecting fastqsanger${auto_compressed_type} instead. Sequencing data produced since February 2011 is generally using Sanger encoding. Please consider selecting fastqsanger${auto_compressed_type} instead. Color space data is rarely used in modern analyses, please consider selecting fastqsanger${auto_compressed_type} instead. Sequencing data produced since February 2011 is generally using Sanger encoding. Please consider selecting fastqsanger${auto_compressed_type} instead. SAM files are uncompressed and large, it is recommended to upload the compressed version called BAM. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/general.py0000644000175100017510000002270515234436334024400 0ustar00runnerrunner"""This module contains linting functions for general aspects of the tool.""" import re from typing import ( Tuple, TYPE_CHECKING, ) from packaging.version import Version from galaxy.tool_util.biotools.source import ApiBiotoolsMetadataSource from galaxy.tool_util.edam_util import load_edam_tree from galaxy.tool_util.lint import Linter from galaxy.tool_util.version import ( LegacyVersion, parse_version, ) if TYPE_CHECKING: from galaxy.tool_util.lint import LintContext from galaxy.tool_util.parser.interface import ToolSource from galaxy.util.etree import ( Element, ElementTree, ) PROFILE_PATTERN = re.compile(r"^[12]\d\.\d{1,2}$") lint_tool_types = ["*"] def _tool_xml_and_root(tool_source: "ToolSource") -> Tuple["ElementTree", "Element"]: tool_xml = getattr(tool_source, "xml_tree", None) if tool_xml: tool_node = tool_xml.getroot() else: tool_node = None return tool_xml, tool_node class ToolVersionMissing(Linter): """ Tools must have a version """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml, tool_node = _tool_xml_and_root(tool_source) version = tool_source.parse_version() or "" if not version: lint_ctx.error("Tool version is missing or empty.", linter=cls.name(), node=tool_node) class ToolVersionPEP404(Linter): """ Tools should have a PEP404 compliant version. """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml, tool_node = _tool_xml_and_root(tool_source) version = tool_source.parse_version() or "" parsed_version = parse_version(version) if version and isinstance(parsed_version, LegacyVersion): lint_ctx.warn(f"Tool version [{version}] is not compliant with PEP 440.", linter=cls.name(), node=tool_node) class ToolVersionWhitespace(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml, tool_node = _tool_xml_and_root(tool_source) version = tool_source.parse_version() or "" if version != version.strip(): lint_ctx.warn( f"Tool version is pre/suffixed by whitespace, this may cause errors: [{version}].", linter=cls.name(), node=tool_node, ) class ToolVersionValid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml, tool_node = _tool_xml_and_root(tool_source) version = tool_source.parse_version() or "" parsed_version = parse_version(version) if version and not isinstance(parsed_version, LegacyVersion) and version == version.strip(): lint_ctx.valid(f"Tool defines a version [{version}].", linter=cls.name(), node=tool_node) class ToolNameMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) name = tool_source.parse_name() if not name: lint_ctx.error("Tool name is missing or empty.", linter=cls.name(), node=tool_node) class ToolNameWhitespace(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) name = tool_source.parse_name() if name and name != name.strip(): lint_ctx.warn( f"Tool name is pre/suffixed by whitespace, this may cause errors: [{name}].", linter=cls.name(), node=tool_node, ) class ToolNameValid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) name = tool_source.parse_name() if name and name == name.strip(): lint_ctx.valid(f"Tool defines a name [{name}].", linter=cls.name(), node=tool_node) class ToolIDMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) tool_id = tool_source.parse_id() if not tool_id: lint_ctx.error("Tool does not define an id attribute.", linter=cls.name(), node=tool_node) class ToolIDWhitespace(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) tool_id = tool_source.parse_id() if tool_id and re.search(r"\s", tool_id): lint_ctx.warn( f"Tool ID contains whitespace - this is discouraged: [{tool_id}].", linter=cls.name(), node=tool_node ) class ToolIDValid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) tool_id = tool_source.parse_id() if tool_id and not re.search(r"\s", tool_id): lint_ctx.valid(f"Tool defines an id [{tool_id}].", linter=cls.name(), node=tool_node) class ToolProfileInvalid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) profile = tool_source.parse_profile() profile_valid = PROFILE_PATTERN.match(profile) is not None if not profile_valid: lint_ctx.error(f"Tool specifies an invalid profile version [{profile}].", linter=cls.name(), node=tool_node) class ToolProfileLegacy(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) profile = tool_source.parse_profile() profile_valid = PROFILE_PATTERN.match(profile) is not None if profile_valid and Version(profile) == Version("16.01"): lint_ctx.valid("Tool targets 16.01 Galaxy profile.", linter=cls.name(), node=tool_node) class ToolProfileValid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) profile = tool_source.parse_profile() profile_valid = PROFILE_PATTERN.match(profile) is not None if profile_valid and Version(profile) != Version("16.01"): lint_ctx.valid(f"Tool specifies profile version [{profile}].", linter=cls.name(), node=tool_node) class RequirementNameMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) requirements, *_ = tool_source.parse_requirements() for r in requirements: if r.type != "package": continue if not r.name: lint_ctx.error("Requirement without name found", linter=cls.name(), node=tool_node) class RequirementVersionMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) requirements, *_ = tool_source.parse_requirements() for r in requirements: if r.type != "package": continue if not r.version: lint_ctx.warn(f"Requirement {r.name} defines no version", linter=cls.name(), node=tool_node) class RequirementVersionWhitespace(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) requirements, *_ = tool_source.parse_requirements() for r in requirements: if r.type != "package": continue if r.version and r.version != r.version.strip(): lint_ctx.warn( f"Requirement version contains whitespace, this may cause errors: [{r.version}].", linter=cls.name(), node=tool_node, ) class ResourceRequirementExpression(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) requirements, containers, resource_requirements, *_ = tool_source.parse_requirements() for rr in resource_requirements: if rr.runtime_required: lint_ctx.warn( "Expressions in resource requirement not supported yet", linter=cls.name(), node=tool_node ) class BioToolsValid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) xrefs = tool_source.parse_xrefs() for xref in xrefs: if xref["type"] != "bio.tools": continue metadata_source = ApiBiotoolsMetadataSource() if not metadata_source.get_biotools_metadata(xref["value"]): lint_ctx.warn(f'No entry {xref["value"]} in bio.tools.', linter=cls.name(), node=tool_node) class EDAMTermsValid(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): _, tool_node = _tool_xml_and_root(tool_source) edam = load_edam_tree(None, "operation_", "topic_") terms = tool_source.parse_edam_operations() + tool_source.parse_edam_topics() for term in terms: if term not in edam: lint_ctx.warn(f"No entry '{term}' in EDAM.", linter=cls.name(), node=tool_node) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/help.py0000644000175100017510000001045215234436334023707 0ustar00runnerrunner"""This module contains a linting function for a tool's help.""" from typing import ( Optional, Tuple, TYPE_CHECKING, Union, ) from galaxy.tool_util.lint import Linter from galaxy.util import ( rst_to_html, unicodify, ) if TYPE_CHECKING: from galaxy.tool_util.lint import LintContext from galaxy.tool_util.parser.interface import ToolSource from galaxy.util.etree import Element class HelpMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return root = tool_xml.find("./help") if root is None: root = tool_xml.getroot() help = tool_xml.find("./help") if help is None: lint_ctx.warn( "No help section found, consider adding a help section to your tool.", linter=cls.name(), node=root ) class HelpEmpty(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return help = tool_xml.find("./help") if help is None: return help_text = help.text or "" if not help_text.strip(): lint_ctx.warn("Help section appears to be empty.", linter=cls.name(), node=help) class HelpPresent(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return help = tool_xml.find("./help") if help is None: return help_text = help.text or "" if help_text.strip(): lint_ctx.valid("Tool contains help section.", linter=cls.name(), node=help) class HelpTODO(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return help = tool_xml.find("./help") if help is None: return help_text = help.text or "" if "TODO" in help_text: lint_ctx.warn("Help contains TODO text.", linter=cls.name(), node=help) class HelpInvalidRST(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): help_text, node = _help_rst(tool_source) if help_text is None: return invalid_rst = rst_invalid(help_text) if invalid_rst: lint_ctx.warn(f"Invalid reStructuredText found in help - [{invalid_rst}].", linter=cls.name(), node=node) class HelpValidRST(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): help_text, node = _help_rst(tool_source) if help_text is None: return invalid_rst = rst_invalid(help_text) if not invalid_rst: lint_ctx.valid("Help contains valid reStructuredText.", linter=cls.name(), node=node) def _help_rst(tool_source: "ToolSource") -> "Tuple[Optional[str], Optional[Element]]": """Return the help text to RST-validate and the XML node to anchor messages to. The text is ``None`` (i.e. RST validation should be skipped) when there is no help, when the help is empty, or when its format is not ``restructuredtext`` (e.g. Markdown help, which YAML tool sources default to). ``node`` is the ```` element for XML tool sources and ``None`` for others. """ help_content = tool_source.parse_help() tool_xml = getattr(tool_source, "xml_tree", None) node = tool_xml.find("./help") if tool_xml is not None else None if help_content is None or not help_content.content.strip(): return None, node if help_content.format != "restructuredtext": return None, node return help_content.content, node def rst_invalid(text: str) -> Union[bool, str]: """ Predicate to determine if text is invalid reStructuredText. Return False if the supplied text is valid reStructuredText or a string indicating the problem. """ invalid_rst: Union[bool, str] = False try: rst_to_html(text, error=True) except Exception as e: invalid_rst = unicodify(e) return invalid_rst ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1785871580.0 galaxy_tool_util-26.1.1/src/galaxy/tool_util/linters/inputs.py0000644000175100017510000016573315234436334024316 0ustar00runnerrunner"""This module contains a linting functions for tool inputs.""" import ast import re import warnings from copy import deepcopy from typing import ( Iterator, Optional, Tuple, TYPE_CHECKING, ) from packaging.version import Version from galaxy.tool_util.lint import Linter from galaxy.util import string_as_bool from ._util import ( is_datasource, is_valid_cheetah_placeholder, ) from ..parser.util import _parse_name if TYPE_CHECKING: from galaxy.tool_util.lint import LintContext from galaxy.tool_util.parser import ToolSource from galaxy.util.etree import ( Element, ElementTree, ) lint_tool_types = ["*"] ATTRIB_VALIDATOR_COMPATIBILITY = { "check": ["metadata"], "expression": ["substitute_value_in_message"], "table_name": [ "dataset_metadata_in_data_table", "dataset_metadata_not_in_data_table", "value_in_data_table", "value_not_in_data_table", ], "filename": ["dataset_metadata_in_file"], "metadata_name": [ "dataset_metadata_equal", "dataset_metadata_in_data_table", "dataset_metadata_not_in_data_table", "dataset_metadata_in_file", "dataset_metadata_in_range", ], "metadata_column": [ "dataset_metadata_in_data_table", "dataset_metadata_not_in_data_table", "value_in_data_table", "value_not_in_data_table", "dataset_metadata_in_file", ], "line_startswith": ["dataset_metadata_in_file"], "min": ["in_range", "length", "dataset_metadata_in_range"], "max": ["in_range", "length", "dataset_metadata_in_range"], "exclude_min": ["in_range", "dataset_metadata_in_range"], "exclude_max": ["in_range", "dataset_metadata_in_range"], "split": ["dataset_metadata_in_file"], "skip": ["metadata"], "value": ["dataset_metadata_equal"], "value_json": ["dataset_metadata_equal"], } PARAMETER_VALIDATOR_TYPE_COMPATIBILITY = { "integer": ["in_range", "expression"], "float": ["in_range", "expression"], "data": [ "metadata", "no_options", "unspecified_build", "dataset_ok_validator", "dataset_metadata_equal", "dataset_metadata_in_range", "dataset_metadata_in_file", "dataset_metadata_in_data_table", "dataset_metadata_not_in_data_table", "expression", ], "data_collection": [ "metadata", "no_options", "unspecified_build", "dataset_ok_validator", "dataset_metadata_equal", "dataset_metadata_in_range", "dataset_metadata_in_file", "dataset_metadata_in_data_table", "dataset_metadata_not_in_data_table", "expression", ], "text": ["regex", "length", "empty_field", "value_in_data_table", "value_not_in_data_table", "expression"], "select": [ "in_range", "no_options", "regex", "length", "empty_field", "value_in_data_table", "value_not_in_data_table", "expression", ], "drill_down": [ "no_options", "regex", "length", "empty_field", "value_in_data_table", "value_not_in_data_table", "expression", ], "data_column": [ "no_options", "regex", "length", "empty_field", "value_in_data_table", "value_not_in_data_table", "expression", ], } PARAM_TYPE_CHILD_COMBINATIONS = [ ("./options", ["data", "select", "drill_down"]), ("./options/option", ["drill_down"]), ("./options/column", ["data", "select"]), ] # TODO lint for valid param type - attribute combinations # TODO check if ref input param is present for from_dataset class InputsNum(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return tool_node = tool_xml.find("./inputs") if tool_node is None: tool_node = tool_xml.getroot() num_inputs = len(tool_xml.findall("./inputs//param")) if num_inputs: lint_ctx.info(f"Found {num_inputs} input parameters.", linter=cls.name(), node=tool_node) class InputsMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return tool_node = tool_xml.find("./inputs") if tool_node is None: tool_node = tool_xml.getroot() num_inputs = len(tool_xml.findall("./inputs//param")) if num_inputs == 0 and not is_datasource(tool_xml): lint_ctx.warn("Found no input parameters.", linter=cls.name(), node=tool_node) class InputsMissingDataSource(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return tool_node = tool_xml.find("./inputs") if tool_node is None: tool_node = tool_xml.getroot() num_inputs = len(tool_xml.findall("./inputs//param")) if num_inputs == 0 and is_datasource(tool_xml): lint_ctx.info("No input parameters, OK for data sources", linter=cls.name(), node=tool_node) class InputsDatasourceTags(Linter): """ Lint that datasource tools have display and uihints tags """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return tool_node = tool_xml.find("./inputs") if tool_node is None: tool_node = tool_xml.getroot() inputs = tool_xml.findall("./inputs//param") if is_datasource(tool_xml): # TODO only display is subtag of inputs, uihints is a separate top level tag (supporting only attrib minwidth) for datasource_tag in ("display", "uihints"): if not any(param.tag == datasource_tag for param in inputs): lint_ctx.info( f"{datasource_tag} tag usually present in data sources", linter=cls.name(), node=tool_node ) def _iter_param(tool_xml: "ElementTree") -> Iterator[Tuple["Element", str]]: for param in tool_xml.findall("./inputs//param"): if "name" not in param.attrib and "argument" not in param.attrib: continue param_name = _parse_name(param.attrib.get("name"), param.attrib.get("argument")) yield param, param_name def _iter_param_type(tool_xml: "ElementTree") -> Iterator[Tuple["Element", str, str]]: for param, param_name in _iter_param(tool_xml): if "type" not in param.attrib: continue param_type = param.attrib["type"] if param_type == "": continue yield param, param_name, param_type class InputsName(Linter): """ Lint params define a name """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param in tool_xml.findall("./inputs//param"): if "name" not in param.attrib and "argument" not in param.attrib: lint_ctx.error("Found param input with no name specified.", linter=cls.name(), node=param) class InputsNameRedundantArgument(Linter): """ Lint params define a name """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param in tool_xml.findall("./inputs//param"): if "name" in param.attrib and "argument" in param.attrib: param_name = _parse_name(param.attrib.get("name"), param.attrib.get("argument")) if param.attrib.get("name") == _parse_name(None, param.attrib.get("argument")): lint_ctx.warn( f"Param input [{param_name}] 'name' attribute is redundant if argument implies the same name.", linter=cls.name(), node=param, ) # TODO redundant with InputsNameValid class InputsNameEmpty(Linter): """ Lint params define a non-empty name """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name in _iter_param(tool_xml): if param_name.strip() == "": lint_ctx.error("Param input with empty name.", linter=cls.name(), node=param) class InputsNameValid(Linter): """ Lint params define valid cheetah placeholder """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name in _iter_param(tool_xml): if param_name != "" and not is_valid_cheetah_placeholder(param_name): lint_ctx.warn( f"Param input [{param_name}] is not a valid Cheetah placeholder.", linter=cls.name(), node=param ) def _param_path(param: "Element", param_name: str) -> str: path = [param_name] parent = param while True: parent = parent.getparent() if parent.tag == "inputs": break # parameters of the same name in different when branches are allowed # just add the value of the when branch to the path (this also allows # that the conditional select has the same name as params in the whens) if parent.tag == "when": path.append(str(parent.attrib.get("value"))) else: path.append(str(parent.attrib.get("name"))) return ".".join(reversed(path)) class InputsNameDuplicate(Linter): """ Lint params with duplicate names """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return input_names = set() for param, param_name in _iter_param(tool_xml): # check for parameters with duplicate names path = _param_path(param, param_name) if path in input_names: lint_ctx.error( f"Tool defines multiple parameters with the same name: '{path}'", linter=cls.name(), node=param ) input_names.add(path) class InputsNameDuplicateOutput(Linter): """ Lint params names that are also used in outputs """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return input_names = set() for param, param_name in _iter_param(tool_xml): input_names.add(_param_path(param, param_name)) # check if there is an output with the same name as an input outputs = tool_xml.findall("./outputs/*") for output in outputs: if output.get("name") in input_names: lint_ctx.error( f'Tool defines an output with a name equal to the name of an input: \'{output.get("name")}\'', linter=cls.name(), node=output, ) class InputsTypeChildCombination(Linter): """ Lint for invalid parameter type child combinations """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): # lint for valid param type - child node combinations for ptcc in PARAM_TYPE_CHILD_COMBINATIONS: if param.find(ptcc[0]) is not None and param_type not in ptcc[1]: lint_ctx.error( f"Parameter [{param_name}] '{ptcc[0]}' tags are only allowed for parameters of type {ptcc[1]}", linter=cls.name(), node=param, ) class InputsDataFormat(Linter): """ Lint for data params wo format """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "data": continue if "format" not in param.attrib: lint_ctx.warn( f"Param input [{param_name}] with no format specified - 'data' format will be assumed.", linter=cls.name(), node=param, ) class InputsDataOptionsMultiple(Linter): """ Lint for data params with multiple options tags """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "data": continue options = param.findall("./options") if len(options) > 1: lint_ctx.error( f"Data parameter [{param_name}] contains multiple options elements.", linter=cls.name(), node=options[1], ) class InputsDataOptionsAttrib(Linter): """ Lint for data params with options that have invalid attributes """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "data": continue options = param.find("./options") if options is None: continue for oa in options.attrib: if oa != "options_filter_attribute": lint_ctx.error( f"Data parameter [{param_name}] uses invalid attribute: {oa}", linter=cls.name(), node=param ) class InputsDataOptionsFilterAttribFiltersType(Linter): """ Lint for valid filter types for data parameters if options set options_filter_attribute """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "data": continue options = param.find("./options") if options is None: continue for filter in param.findall("./options/filter"): if "options_filter_attribute" in options.attrib: if filter.get("type") != "data_meta": lint_ctx.error( f'Data parameter [{param_name}] for filters only type="data_meta" is allowed, found type="{filter.get("type")}"', linter=cls.name(), node=filter, ) class InputsDataOptionsFiltersType(Linter): """ Lint for valid filter types for data parameters if options do not set options_filter_attribute """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "data": continue options = param.find("./options") if options is None: continue for filter in param.findall("./options/filter"): if "options_filter_attribute" not in options.attrib: if filter.get("key") != "dbkey" or filter.get("type") != "data_meta": lint_ctx.error( f'Data parameter [{param_name}] for filters only type="data_meta" and key="dbkey" are allowed, found type="{filter.get("type")}" and key="{filter.get("key")}"', linter=cls.name(), node=filter, ) FILTER_REQUIRED_ATTRIBUTES = { "data_meta": ["type", "ref", "key"], # column needs special treatment "param_value": ["type", "ref", "column"], "static_value": ["type", "column", "value"], "regexp": ["type", "column", "value"], "unique_value": ["type", "column"], "multiple_splitter": ["type", "column"], "attribute_value_splitter": ["type", "column"], "add_value": ["type", "value"], "remove_value": ["type"], # this is handled separately in InputsOptionsRemoveValueFilterRequiredAttributes "sort_by": ["type", "column"], "data_table": ["type", "column", "table_name", "data_table_column"], } class InputsOptionsFiltersRequiredAttributes(Linter): """ check required attributes of filters """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name in _iter_param(tool_xml): options = param.find("./options") if options is None: continue for filter in param.findall("./options/filter"): filter_type = filter.get("type", None) if filter_type is None or filter_type not in FILTER_ALLOWED_ATTRIBUTES: continue for attrib in FILTER_REQUIRED_ATTRIBUTES[filter_type]: if attrib not in filter.attrib: lint_ctx.error( f"Select parameter [{param_name}] '{filter_type}' filter misses required attribute '{attrib}'", linter=cls.name(), node=filter, ) class InputsOptionsRemoveValueFilterRequiredAttributes(Linter): """ check required attributes of remove_value filter """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name in _iter_param(tool_xml): options = param.find("./options") if options is None: continue for filter in param.findall("./options/filter"): filter_type = filter.get("type", None) # check for required attributes for filter (remove_value needs a bit more logic here) if filter_type != "remove_value": continue if not ( ( "value" in filter.attrib and "ref" not in filter.attrib and "meta_ref" not in filter.attrib and "key" not in filter.attrib ) or ( "value" not in filter.attrib and "ref" in filter.attrib and "meta_ref" not in filter.attrib and "key" not in filter.attrib ) or ( "value" not in filter.attrib and "ref" not in filter.attrib and "meta_ref" in filter.attrib and "key" in filter.attrib ) ): lint_ctx.error( f"Select parameter [{param_name}] '{filter_type}'' filter needs either the 'value'; 'ref'; or 'meta' and 'key' attribute(s)", linter=cls.name(), node=filter, ) FILTER_ALLOWED_ATTRIBUTES = deepcopy(FILTER_REQUIRED_ATTRIBUTES) FILTER_ALLOWED_ATTRIBUTES["static_value"].append("keep") FILTER_ALLOWED_ATTRIBUTES["regexp"].append("keep") FILTER_ALLOWED_ATTRIBUTES["data_meta"].extend(["column", "multiple", "separator"]) FILTER_ALLOWED_ATTRIBUTES["param_value"].extend(["keep", "ref_attribute"]) FILTER_ALLOWED_ATTRIBUTES["multiple_splitter"].append("separator") FILTER_ALLOWED_ATTRIBUTES["attribute_value_splitter"].extend(["pair_separator", "name_val_separator"]) FILTER_ALLOWED_ATTRIBUTES["add_value"].extend(["name", "index"]) FILTER_ALLOWED_ATTRIBUTES["remove_value"].extend(["value", "ref", "meta_ref", "key"]) FILTER_ALLOWED_ATTRIBUTES["data_table"].append("keep") FILTER_ALLOWED_ATTRIBUTES["sort_by"].append("reverse_sort_order") class InputsOptionsFiltersAllowedAttributes(Linter): """ check allowed attributes of filters """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name in _iter_param(tool_xml): options = param.find("./options") if options is None: continue for filter in param.findall("./options/filter"): filter_type = filter.get("type", None) if filter_type is None or filter_type not in FILTER_ALLOWED_ATTRIBUTES: continue for attrib in filter.attrib: if attrib not in FILTER_ALLOWED_ATTRIBUTES[filter_type]: lint_ctx.warn( f"Select parameter [{param_name}] '{filter_type}' filter specifies unnecessary attribute '{attrib}'", linter=cls.name(), node=filter, ) class InputsOptionsRegexFilterExpression(Linter): """ Check the regular expression of regexp filters """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name in _iter_param(tool_xml): options = param.find("./options") if options is None: continue for filter in param.findall("./options/filter"): filter_type = filter.get("type", None) if filter_type == "regexp" and "value" in filter.attrib: try: re.compile(filter.attrib["value"]) except re.error as re_error: lint_ctx.error( f"Select parameter [{param_name}] '{filter_type}'' filter 'value' is not a valid regular expression ({re_error})'", linter=cls.name(), node=filter, ) class InputsOptionsFiltersCheckReferences(Linter): """ Check the references used in filters """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return # get the set of param names param_names = {param_name for _, param_name in _iter_param(tool_xml)} for param, param_name in _iter_param(tool_xml): options = param.find("./options") if options is None: continue for filter in param.findall("./options/filter"): filter_type = filter.get("type", None) if filter_type is not None: # check for references to other inputs # TODO: currently ref and metaref seem only to work for top level params, # once this is fixed the linter needs to be extended, e.g. `f.attrib[ref_attrib].split('|')[-1]` for ref_attrib in ["meta_ref", "ref"]: if ref_attrib in filter.attrib and filter.attrib[ref_attrib] not in param_names: lint_ctx.error( f"Select parameter [{param_name}] '{filter_type}'' filter attribute '{ref_attrib}' refers to non existing parameter '{filter.attrib[ref_attrib]}'", linter=cls.name(), node=filter, ) class InputsDataOptionsFiltersRef(Linter): """ Lint for set ref for filters of data parameters """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "data": continue options = param.find("./options") if options is None: continue for filter in param.findall("./options/filter"): if not filter.get("ref"): lint_ctx.error( f"Data parameter [{param_name}] filter needs to define a ref attribute", linter=cls.name(), node=filter, ) class InputsSelectDynamicOptions(Linter): """ Lint for select with deprecated dynamic_options attribute """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue dynamic_options = param.get("dynamic_options", None) if dynamic_options is not None: lint_ctx.warn( f"Select parameter [{param_name}] uses deprecated 'dynamic_options' attribute.", linter=cls.name(), node=param, ) class InputsSelectOptionsDef(Linter): """ Lint for valid ways to define select options """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue dynamic_options = param.get("dynamic_options", None) options = param.findall("./options") select_options = param.findall("./option") if param.getparent().tag != "conditional": if (dynamic_options is not None) + (len(options) > 0) + (len(select_options) > 0) != 1: lint_ctx.error( f"Select parameter [{param_name}] options have to be defined by either 'option' children elements, a 'options' element or the 'dynamic_options' attribute.", linter=cls.name(), node=param, ) class InputsSelectOptionsDefConditional(Linter): """ Lint for valid ways to define select options (for a select in a conditional) """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue dynamic_options = param.get("dynamic_options", None) options = param.findall("./options") select_options = param.findall("./option") if param.getparent().tag == "conditional": if len(select_options) == 0 or dynamic_options is not None or len(options) > 0: lint_ctx.error( f"Select parameter of a conditional [{param_name}] options have to be defined by 'option' children elements.", linter=cls.name(), node=param, ) # TODO xsd class InputsSelectOptionValueMissing(Linter): """ Lint for select option tags without value """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue select_options = param.findall("./option") if any("value" not in option.attrib for option in select_options): lint_ctx.error( f"Select parameter [{param_name}] has option without value", linter=cls.name(), node=param ) class InputsSelectOptionDuplicateValue(Linter): """ Lint for select option with same value """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue select_options = param.findall("./option") select_options_values = [] for option in select_options: value = option.attrib.get("value", "") select_options_values.append((value, option.attrib.get("selected", "false"))) if len(set(select_options_values)) != len(select_options_values): lint_ctx.error( f"Select parameter [{param_name}] has multiple options with the same value", linter=cls.name(), node=param, ) class InputsSelectOptionDuplicateText(Linter): """ Lint for select option with same text """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue select_options = param.findall("./option") select_options_texts = [] for option in select_options: if option.text is None: text = option.attrib.get("value", "").capitalize() else: text = option.text select_options_texts.append((text, option.attrib.get("selected", "false"))) if len(set(select_options_texts)) != len(select_options_texts): lint_ctx.error( f"Select parameter [{param_name}] has multiple options with the same text content", linter=cls.name(), node=param, ) class InputsSelectOptionsMultiple(Linter): """ Lint dynamic options select for multiple options tags """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue options = param.findall("./options") if len(options) > 1: lint_ctx.error( f"Select parameter [{param_name}] contains multiple options elements.", linter=cls.name(), node=options[1], ) class InputsSelectOptionsDefinesOptions(Linter): """ Lint dynamic options select for the potential to define options """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue options = param.find("./options") if options is None: continue filter_adds_options = any( f.get("type", None) in ["add_value", "data_meta"] for f in param.findall("./options/filter") ) from_file = options.get("from_file", None) from_parameter = options.get("from_parameter", None) # TODO check if input param is present for from_dataset from_dataset = options.get("from_dataset", None) from_data_table = options.get("from_data_table", None) from_url = options.get("from_url", None) if ( from_file is None and from_parameter is None and from_dataset is None and from_data_table is None and from_url is None and not filter_adds_options ): lint_ctx.error( f"Select parameter [{param_name}] options tag defines no options. Use 'from_dataset', 'from_data_table', or a filter that adds values.", linter=cls.name(), node=options, ) class InputsSelectOptionsDeprecatedAttr(Linter): """ Lint dynamic options select deprecated attributes """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue options = param.find("./options") if options is None: continue for deprecated_attr in ["from_file", "from_parameter", "options_filter_attribute", "transform_lines"]: if options.get(deprecated_attr) is not None: lint_ctx.warn( f"Select parameter [{param_name}] options uses deprecated '{deprecated_attr}' attribute.", linter=cls.name(), node=options, ) class InputsSelectOptionsFromDatasetAndDatatable(Linter): """ Lint dynamic options select for from_dataset and from_data_table """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue options = param.find("./options") if options is None: continue from_dataset = options.get("from_dataset", None) from_data_table = options.get("from_data_table", None) if from_dataset is not None and from_data_table is not None: lint_ctx.error( f"Select parameter [{param_name}] options uses 'from_dataset' and 'from_data_table' attribute.", linter=cls.name(), node=options, ) class InputsSelectOptionsMetaFileKey(Linter): """ Lint dynamic options select: meta_file_key attribute can only be used with from_dataset """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "select": continue options = param.find("./options") if options is None: continue from_dataset = options.get("from_dataset", None) if options.get("meta_file_key", None) is not None and from_dataset is None: lint_ctx.error( f"Select parameter [{param_name}] 'meta_file_key' is only compatible with 'from_dataset'.", linter=cls.name(), node=options, ) class InputsBoolDistinctValues(Linter): """ Lint booleans for distinct true/false value """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "boolean": continue profile = tool_source.parse_profile() truevalue = param.attrib.get("truevalue", "true") falsevalue = param.attrib.get("falsevalue", "false") problematic_booleans_allowed = Version(profile) < Version("23.1") lint_level = lint_ctx.warn if problematic_booleans_allowed else lint_ctx.error if truevalue == falsevalue: lint_level( f"Boolean parameter [{param_name}] needs distinct 'truevalue' and 'falsevalue' values.", linter=cls.name(), node=param, ) class InputsBoolProblematic(Linter): """ Lint booleans for problematic values, i.e. truevalue being false and vice versa """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type != "boolean": continue profile = tool_source.parse_profile() truevalue = param.attrib.get("truevalue", "true") falsevalue = param.attrib.get("falsevalue", "false") problematic_booleans_allowed = Version(profile) < Version("23.1") lint_level = lint_ctx.warn if problematic_booleans_allowed else lint_ctx.error if truevalue.lower() == "false": lint_level( f"Boolean parameter [{param_name}] has invalid truevalue [{truevalue}].", linter=cls.name(), node=param, ) if falsevalue.lower() == "true": lint_level( f"Boolean parameter [{param_name}] has invalid falsevalue [{falsevalue}].", linter=cls.name(), node=param, ) class InputsSelectSingleCheckboxes(Linter): """ Lint selects that allow only a single selection but display as checkboxes """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type not in ["select", "data_column", "drill_down"]: continue multiple = string_as_bool(param.attrib.get("multiple", "false")) if param.attrib.get("display") == "checkboxes": if not multiple: lint_ctx.error( f'Select [{param_name}] `display="checkboxes"` is incompatible with `multiple="false"`, remove the `display` attribute', linter=cls.name(), node=param, ) class InputsSelectMandatoryCheckboxes(Linter): """ Lint selects that are mandatory but display as checkboxes """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type not in ["select", "data_column", "drill_down"]: continue multiple = string_as_bool(param.attrib.get("multiple", "false")) optional = string_as_bool(param.attrib.get("optional", multiple)) if param.attrib.get("display") == "checkboxes": if not optional: lint_ctx.error( f'Select [{param_name}] `display="checkboxes"` is incompatible with `optional="false"`, remove the `display` attribute', linter=cls.name(), node=param, ) class InputsSelectMultipleRadio(Linter): """ Lint selects that allow only multiple selections but display as radio """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type not in ["select", "data_column", "drill_down"]: continue multiple = string_as_bool(param.attrib.get("multiple", "false")) if param.attrib.get("display") == "radio": if multiple: lint_ctx.error( f'Select [{param_name}] display="radio" is incompatible with multiple="true"', linter=cls.name(), node=param, ) class InputsSelectOptionalRadio(Linter): """ Lint selects that are optional but display as radio """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param, param_name, param_type in _iter_param_type(tool_xml): if param_type not in ["select", "data_column", "drill_down"]: continue multiple = string_as_bool(param.attrib.get("multiple", "false")) optional = string_as_bool(param.attrib.get("optional", multiple)) if param.attrib.get("display") == "radio": if optional: lint_ctx.error( f'Select [{param_name}] display="radio" is incompatible with optional="true"', linter=cls.name(), node=param, ) def _iter_param_validator(tool_xml: "ElementTree") -> Iterator[Tuple[str, str, "Element", str]]: input_params = tool_xml.findall("./inputs//param[@type]") for param in input_params: try: param_name = _parse_name(param.attrib.get("name"), param.attrib.get("argument")) except ValueError: continue param_type = param.attrib["type"] validators = param.findall("./validator[@type]") for validator in validators: vtype = validator.attrib["type"] yield (param_name, param_type, validator, vtype) class ValidatorParamIncompatible(Linter): """ Lint for validator type - parameter type incompatibilities """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, param_type, validator, vtype in _iter_param_validator(tool_xml): if param_type in PARAMETER_VALIDATOR_TYPE_COMPATIBILITY: if vtype not in PARAMETER_VALIDATOR_TYPE_COMPATIBILITY[param_type]: lint_ctx.error( f"Parameter [{param_name}]: validator with an incompatible type '{vtype}'", linter=cls.name(), node=validator, ) class ValidatorAttribIncompatible(Linter): """ Lint for incompatibilities between validator type and given attributes """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, _, validator, vtype in _iter_param_validator(tool_xml): for attrib in ATTRIB_VALIDATOR_COMPATIBILITY: if attrib in validator.attrib and vtype not in ATTRIB_VALIDATOR_COMPATIBILITY[attrib]: lint_ctx.error( f"Parameter [{param_name}]: attribute '{attrib}' is incompatible with validator of type '{vtype}'", linter=cls.name(), node=validator, ) class ValidatorHasText(Linter): """ Lint that parameters that need text have text """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if vtype in ["expression", "regex"]: if validator.text is None: lint_ctx.error( f"Parameter [{param_name}]: {vtype} validators are expected to contain text", linter=cls.name(), node=validator, ) class ValidatorHasNoText(Linter): """ Lint that parameters that need no text have no text """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if vtype not in ["expression", "regex"] and validator.text is not None: lint_ctx.warn( f"Parameter [{param_name}]: '{vtype}' validators are not expected to contain text (found '{validator.text}')", linter=cls.name(), node=validator, ) class ValidatorExpression(Linter): """ Lint that checks expressions / regexp (ignoring FutureWarning) """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return with warnings.catch_warnings(): warnings.simplefilter("error", FutureWarning) for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if vtype in ["expression", "regex"] and validator.text is not None: try: if vtype == "regex": re.compile(validator.text) else: ast.parse(validator.text, mode="eval") except FutureWarning: pass except Exception as e: lint_ctx.error( f"Parameter [{param_name}]: '{validator.text}' is no valid {vtype}: {str(e)}", linter=cls.name(), node=validator, ) class ValidatorExpressionFuture(Linter): """ Lint that checks expressions / regexp FutureWarnings """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return with warnings.catch_warnings(): warnings.simplefilter("error", FutureWarning) for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if vtype in ["expression", "regex"] and validator.text is not None: try: if vtype == "regex": re.compile(validator.text) else: ast.parse(validator.text, mode="eval") except FutureWarning as e: lint_ctx.warn( f"Parameter [{param_name}]: '{validator.text}' is marked as deprecated {vtype}: {str(e)}", linter=cls.name(), node=validator, ) except Exception: pass class ValidatorMinMax(Linter): """ Lint for min/max for validator that need it """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if vtype in ["in_range", "length", "dataset_metadata_in_range"] and ( "min" not in validator.attrib and "max" not in validator.attrib ): lint_ctx.error( f"Parameter [{param_name}]: '{vtype}' validators need to define the 'min' or 'max' attribute(s)", linter=cls.name(), node=validator, ) class ValidatorDatasetMetadataEqualValue(Linter): """ Lint dataset_metadata_equal needs value or value_json """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if vtype in ["dataset_metadata_equal"]: if ( not ("value" in validator.attrib or "value_json" in validator.attrib) or "metadata_name" not in validator.attrib ): lint_ctx.error( f"Parameter [{param_name}]: '{vtype}' validators need to define the 'value'/'value_json' and 'metadata_name' attributes", linter=cls.name(), node=validator, ) class ValidatorDatasetMetadataEqualValueOrJson(Linter): """ Lint dataset_metadata_equal needs either value or value_json """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if vtype in ["dataset_metadata_equal"]: if "value" in validator.attrib and "value_json" in validator.attrib: lint_ctx.error( f"Parameter [{param_name}]: '{vtype}' validators must not define the 'value' and the 'value_json' attributes", linter=cls.name(), node=validator, ) class ValidatorMetadataCheckSkip(Linter): """ Lint metadata needs check or skip """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if vtype in ["metadata"] and ("check" not in validator.attrib and "skip" not in validator.attrib): lint_ctx.error( f"Parameter [{param_name}]: '{vtype}' validators need to define the 'check' or 'skip' attribute(s)", linter=cls.name(), node=validator, ) class ValidatorTableName(Linter): """ Lint table_name is present if needed """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if ( vtype in [ "value_in_data_table", "value_not_in_data_table", "dataset_metadata_in_data_table", "dataset_metadata_not_in_data_table", ] and "table_name" not in validator.attrib ): lint_ctx.error( f"Parameter [{param_name}]: '{vtype}' validators need to define the 'table_name' attribute", linter=cls.name(), node=validator, ) class ValidatorMetadataName(Linter): """ Lint metadata_name is present if needed """ @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for param_name, _, validator, vtype in _iter_param_validator(tool_xml): if ( vtype in [ "dataset_metadata_in_data_table", "dataset_metadata_not_in_data_table", "dataset_metadata_in_file", "dataset_metadata_in_range", ] and "metadata_name" not in validator.attrib ): lint_ctx.error( f"Parameter [{param_name}]: '{vtype}' validators need to define the 'metadata_name' attribute", linter=cls.name(), node=validator, ) def _iter_conditional(tool_xml: "ElementTree") -> Iterator[Tuple["Element", Optional[str], "Element", Optional[str]]]: conditionals = tool_xml.findall("./inputs//conditional") for conditional in conditionals: conditional_name = conditional.get("name") if conditional.get("value_from"): # Probably only the upload tool use this, no children elements continue first_param = conditional.find("param") if first_param is None: continue first_param_type = first_param.get("type") yield conditional, conditional_name, first_param, first_param_type class ConditionalParamTypeBool(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for _, conditional_name, first_param, first_param_type in _iter_conditional(tool_xml): if first_param_type == "boolean": lint_ctx.warn( f'Conditional [{conditional_name}] first param of type="boolean" is discouraged, use a select', linter=cls.name(), node=first_param, ) class ConditionalParamType(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for _, conditional_name, first_param, first_param_type in _iter_conditional(tool_xml): if first_param_type not in ["boolean", "select"]: lint_ctx.error( f'Conditional [{conditional_name}] first param should have type="select"', linter=cls.name(), node=first_param, ) class ConditionalParamIncompatibleAttributes(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for _, conditional_name, first_param, first_param_type in _iter_conditional(tool_xml): if first_param_type not in ["boolean", "select"]: continue for incomp in ["optional", "multiple"]: if string_as_bool(first_param.get(incomp, False)): lint_ctx.warn( f'Conditional [{conditional_name}] test parameter cannot be {incomp}="true"', linter=cls.name(), node=first_param, ) class ConditionalWhenMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for conditional, conditional_name, first_param, first_param_type in _iter_conditional(tool_xml): if first_param_type not in ["boolean", "select"]: continue if first_param_type == "select": options = first_param.findall("./option[@value]") option_ids = {option.get("value") for option in options} else: # boolean option_ids = {first_param.get("truevalue", "true"), first_param.get("falsevalue", "false")} whens = conditional.findall("./when[@value]") when_ids = {w.get("value") for w in whens if w.get("value") is not None} for option_id in option_ids - when_ids: lint_ctx.warn( f"Conditional [{conditional_name}] no block found for {first_param_type} option '{option_id}'", linter=cls.name(), node=conditional, ) class ConditionalOptionMissing(Linter): @classmethod def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"): tool_xml = getattr(tool_source, "xml_tree", None) if not tool_xml: return for conditional, conditional_name, first_param, first_param_type in _iter_conditional(tool_xml): if first_param_type != "select": continue options = first_param.findall("./option[@value]") option_ids = {option.get("value") for option in options} whens = conditional.findall("./when[@value]") when_ids = {w.get("value") for w in whens if w.get("value") is not None} for when_id in when_ids - option_ids: lint_ctx.warn( f"Conditional [{conditional_name}] no