././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6651685 datacache-1.27.1/0000755000076500000000000000000015264225206013221 5ustar00iskanderwheel././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092720.7502215 datacache-1.27.1/CHANGELOG.md0000644000076500000000000006362715264225061015047 0ustar00iskanderwheel# Changelog ## 1.27.1 - Drain bounded zlib output when compressed input ends at an expansion boundary, avoiding false truncation errors for valid highly compressible ZIP members. Size, CRC, hash, EOF and trailing-byte checks remain enforced. ## 1.27.0 - Add `install_zip_member` for bounded HTTP range retrieval of individual ZIP members, including ZIP64. Publish expanded and compressed bytes with a source receipt as one atomic bundle; verify transport consistency, CRC32, hashes and size limits. Reuse verified cached bytes offline and retain previous bundles. ## 1.26.1 - Shorten the README, clarify download and cache guides, and keep maintainer design notes outside the published user documentation. ## 1.26.0 - `install_archive(..., validator=...)` checks application requirements on the extracted tree before publishing it and on cache reuse. A rejected refresh leaves the previously installed bundle active, even when the archive itself is structurally valid but lacks required data. ## 1.25.1 - Read a padded `XDG_CACHE_HOME` the same way whichever platformdirs version is installed: surrounding whitespace is stripped, as platformdirs 4.6 and later do. Before, platformdirs 3.0 to 4.5 kept the padding and DataCache fell back to `~/.cache`, so 1.25.0's location still depended on the version there. ## 1.25.0 - Find the platform cache directory with platformdirs instead of appdirs, which is deprecated and unmaintained since 2020. Windows and Linux paths are unchanged, except that an empty or relative `XDG_CACHE_HOME` now falls back to `~/.cache`; appdirs made the cache relative to the working directory. macOS keeps `~/Library/Caches` even when `XDG_CACHE_HOME` is set, which platformdirs 4.6 and later would honour, so no existing cache moves and the location doesn't depend on which platformdirs version is installed. Requires `platformdirs>=3.0`. ## 1.24.0 - `get_cache_root(name, *envkeys, override=None, legacy=())` resolves a package's cache root in one call: an explicit override, then the first environment variable that is set, then the platform cache directory if it already holds data, then the first legacy location that does, then the platform cache directory. Empty folders and files operating systems leave behind don't count as data. This replaces the root resolvers several OpenVax packages keep. The shared-caches guide shows the pattern. - Drop the `typechecks` dependency: three small checks in the SQLite code replace it, raising the same `TypeError`s. One fewer package to install for libraries that only download. - `fetch_file(..., resume="auto")`, also accepted by `Cache.fetch` and the installs' `download_options`, resumes interrupted transfers when it can and otherwise downloads normally: on Windows, without a known size, with decompression or a validator, or when the server turns out not to support resuming (the partial is discarded and the file downloaded in full). This replaces `os.name` checks and error-message matching downstream. `resume=True` still raises in those cases, now with `CannotResume`, a new `FileValidationError` subclass exported from `datacache`, when the server or the destination's filesystem is the reason. With `"auto"`, installs keep partial downloads between attempts only when every file is pinned by hash and size. ## 1.23.0 - Add `prune_bundles(destination, keep=1)` and `list_bundles(destination)`, and `prune` on both registries, to list the bundles in any store and delete all but the newest (#91). The current bundle is always kept, and pruning holds the store's lock, so it never races an install. - Bundle assets can name a local file with `path` instead of `url`, and `install_bundle` and `VersionedDatasetRegistry.download` take `source_paths` to read named assets from files already downloaded, keeping their declared URLs as identity (#94). - Add design notes for caller-declared source identity (#92) and shared storage across consumers (#95) under `docs/design/`. ## 1.22.0 - `install_bundle`, `VersionedDatasetRegistry` and `materialize` work on Windows, as archive installs already did (#89). Resumable downloads (`resume=True`) still need POSIX. Private working directories are named for the user's login name where there are no numeric user ids. ## 1.21.0 - Add `max_bytes` to `fetch_file`, `Cache.fetch` and `fetch_bytes`, and to the `download_options` of `install_bundle`, `install_archive` and `materialize`, for downloads whose exact size isn't known but whose largest acceptable size is (#108). It is checked before every write, so no file a fetch writes ever holds more: the download itself, which is the body after HTTP transfer decoding whatever `Content-Length` says, and the installed file after `decompress=True`. A `Content-Length` over the limit is refused before the body is read. A larger transfer raises `FileValidationError` without retrying, keeps any existing file and removes temporary files. It works on every platform, unlike `resume=True`. `expected_size` keeps its exact meaning and must not exceed `max_bytes`; HTML-to-CSV conversion can't be bounded and is rejected with it. Installs check every size they already know against `max_bytes` before creating anything. ## 1.20.0 - Bundles, archive trees and materializations share one simple store layout, implemented once in the new `datacache.bundle_store` module (#103): ```text / .datacache-store.json {"format": 2, "kind": "bundle"} bundles/ 2026-09-30T14-11-05Z/ an older bundle 2026-10-08T17-02-42Z/ the current bundle: always the newest ``` A bundle is a complete set of files installed together, named for its UTC install time. Installing checks a bundle completely, then renames it into `bundles/`, so readers see the old bundle or the new one, never a mix. There is no `current.json` pointer any more, so the `recovery-required` status and recovery are gone. - Breaking changes, all to stores and APIs that no library uses yet: - Stores written by 1.13 to 1.19, with `generations/` and `current.json`, aren't read. Delete them and install again. - `BundleInspection`, `ArchiveInspection` and `MaterializationInspection` have `bundle` in place of `generation`, as do `VersionedArchiveRegistry.status` rows. - `VersionedDatasetRegistry.bundle_path` is now `store_path`, matching `VersionedArchiveRegistry`. - `force=True` on an invalid bundle or archive always installs a new bundle; it no longer falls back to an older one. - Every kind of install, archives included, turns an existing empty directory into the store in place, so it keeps its owner, group and permissions. A directory with files in it that isn't a store of the same kind is never taken over, and the error says so instead of suggesting `force=True`. - All three kinds of install lock the same way: one lock file beside the store, named without regard to letter case, using `flock` on POSIX and the `filelock` package on Windows. - Registries reject dataset or archive names, and versions, that differ only by letter case, since case-insensitive filesystems would give them one store. - Archive and materialization checks report missing and unexpected files and directories together, and reject a file that changes while it is hashed. - A file in an inspection is `verified` only when a hash the caller supplied vouched for it, for every kind of store; a bundle's own manifest checks the bytes but vouches for nothing. - Name checks reject paths whose parent directories differ only by letter case, such as `A/x` and `a/y`. - `install_bundle` and `install_archive` check that resumable sources can resume before creating anything, and a forced install no longer hashes the bundle it is about to replace. - `install_bundle`, `install_archive` and `materialize` check `download_options` exactly as `fetch_file` checks the same arguments, before creating anything (#103). - Remove `DatabaseTable.from_fasta_dict`. It was deprecated in 1.11.0 for removal in 2.0, and nothing uses it. Parse FASTA in the consuming library and pass a DataFrame to `db_from_dataframe` (#67). - Remove the private `download._decompress_to_file`, which nothing called (#67). - Cap retry backoff in one place, `retries.retry_delay`; delays are unchanged (#67). ## 1.19.1 - Identify every HTTP and HTTPS request as `datacache/ (+https://github.com/openvax/datacache)` instead of Requests' default `python-requests/`, which some hosts refuse. IEDB returned 403 for pepdata's allele list downloads (#104). ## 1.19.0 - Add `materialize` and offline `inspect_materialization` for caller-built multi-file artifacts with complete raw-input/transform/output dependency receipts. Preserve completed inputs and HTTP resume state across failures, publish immutable output generations atomically, and discard owned inputs only after successful publication unless retention is requested. Local sources stay caller-owned; scientific validation stays in the builder (#90). `force=True` always rebuilds, copies local sources again while the original exists, and repairs corrupt private input receipts. - Recover the newest complete generation when a bundle, archive or materialization pointer is lost. Generation names are random, so recovery previously chose an arbitrary matching generation and could roll back to older content; it now orders by the creation time each receipt records. - Add a `store_path(name, version)` callback to `VersionedDatasetRegistry` for exact consumer-chosen managed-store locations. It is mutually exclusive with root configuration and does not run at construction; the first lookup resolves every dataset version once and rejects two versions sharing a store. A populated foreign directory is never taken over, with or without `force=True` (#93). - Add opt-in `verify_files=False` to `inspect_bundle`, `install_bundle` and `VersionedDatasetRegistry`'s `inspect`, `local_path`, `is_cached`, `status`, `download` and `ensure`: checks ownership, source identity, required file types, readability and recorded sizes without reading payloads, so it misses same-size corruption. Fast results never claim hash verification. Full verification stays the default everywhere, so all registry methods agree about a corrupted bundle. New generations and recovery always validate bytes before publication (#88). - `VersionedDatasetRegistry.ensure` returns the paths its download validated instead of inspecting the bundle a second time. - `VersionedArchiveRegistry.status` takes `verify_files` like the archive registry's other lookups; it stays metadata-only by default. ## 1.18.1 - Reorganize the documentation into a site (`./docs.sh`, published from master): installation and an offline verified-cache example on the home page, guides ordered by common tasks, and an interface and guarantees reference. The README links to the guides by absolute URL, so they work on PyPI, and old README section links land on the guides that replaced them. - Run the documentation's examples and check internal links in the documentation build. ## 1.18.0 - Rename `fetch_file` and `Cache.fetch`'s `stale_if_error`, added in 1.17.0, to `return_stale_on_error`, which says what it does: return the cached copy when a refresh fails. The old name is no longer accepted. ## 1.17.0 - Add `fetch_bytes` for small resources such as directory listings: the body in memory, transferred and retried exactly like `fetch_file` downloads, with nothing written to disk. Downloads and `fetch_bytes` share one retry loop. - Add `expire_after` to `fetch_file` and `Cache.fetch`, named as in requests-cache: a cached file older than the given seconds or `timedelta`, or one that no longer validates, is downloaded again (also with `resume=True`). Age comes from the provenance record's fetch time, or else the file's modification time. - Add `stale_if_error`: when a refresh fails, return the valid cached file with a redacted warning instead of raising, like HTTP `stale-if-error`. Progress-callback cancellations still propagate. - Add `validator`: a callable that rejects wrong content, such as an HTTP 200 error page, by raising or returning `False`, before it replaces the cached file and on cache hits. Generalizes the empty-response check (#74). ## 1.16.1 - Always install `tqdm>=4.64` as a required dependency. Progress displays remain opt-in with `show_progress=True` and tqdm stays lazily imported. Keep the `progress` extra as a compatibility alias for existing installation commands. ## 1.16.0 - Add `install_archive` and `inspect_archive` for safely installing a complete tar archive tree as one immutable generation. Ordered split archives are concatenated byte-for-byte; URL and local-path sources share the same API. Extraction rejects traversal, links, special files, path collisions and reserved metadata names, and ignores archive ownership and permission bits. Standard `.` / `./` tar paths are normalized safely; resource limits stop header scanning at the first violation, and local FIFOs cannot block a writer. - Add `VersionedArchiveRegistry`, a catalogue facade with pinned defaults, explicit versions, reusable status rows, fast local path resolution, consumer layout callbacks, local predownloaded-part overrides, and the same progress, retry, verification and recovery behavior as `install_archive`. - Include the recorded URL and observed SHA-256 in `VersionedFileRegistry` status rows so fixed-file consumers can report the same download provenance. - Publish consumer metadata such as MHCflurry's `DOWNLOAD_INFO.csv` inside the same transaction with `extra_files`. Archive receipts record the ordered source identities, assembled and per-part observed hashes, and every extracted file. Trusted expectations remain distinct from observed consistency checks. Reuse checks the complete consumer metadata inventory, including removals. Large manifests are read without the small control-record size cap and validated in staging before the generation is published. - Serialize archive writers with a cross-platform file lock, retain immutable generations across refreshes, and recover a complete generation after an interrupted pointer publication. Legacy or otherwise foreign directories are never claimed, including with `force=True` (#85). ## 1.15.0 - Add `VersionedFileRegistry` for established single-file caches with fixed version paths and root provenance manifests. Legacy files are reused offline without relocation; transfers use the shared downloader, and new receipts use bounded-memory hashing and atomic JSON publication (#83). Writers serialize per root to prevent the legacy registry’s lost manifest-update race. ## 1.14.0 - Allow `resume=True` with `expected_size` alone when the server supplies a strong ETag. Resume uses `If-Range` and validates the returned range and validator; servers without a strong ETag still require `expected_sha256`. Complete size-only partials restart instead of being published without a fresh response. Size-only validation does not verify content checksums (#80). - Add `raw=True` to `fetch_file` and `Cache.fetch` to disable archive decompression and HTML conversion at arbitrary output names. It composes with integrity checks, atomic replacement, progress, provenance, and resume; existing suffix inference is unchanged when raw mode is omitted (#81). ## 1.13.0 - Add integrity-pinned resumable raw HTTP downloads with `resume=True`, private persistent partials, validated ranges, bounded retries, and explicit `discard_partial` cleanup (#64). - Add `VersionedDatasetRegistry`, `install_bundle`, and `inspect_bundle` for single-file and multi-file datasets. Immutable generations and an atomic pointer preserve readers and old installations through refreshes; completed local generations can be recovered offline. Supports hitlist's registry mapping and independently versioned source bundles (#59). - Check full source URL fingerprints and decompression settings before reusing bundle assets without trusted hashes. Hash-pinned generations remain shareable across libraries using different mirrors or compression settings. - Accept precreated empty bundle directories without forcing an install, keep their access modes, and reject asset/provenance-sidecar collisions before writes. Publish generations with their final sharing permissions while keeping failed or interrupted resumable work inside a private staging directory. - Remove the redundant test-local FASTA downloader and unused private `ext` option; use urllib3's public exception import. Keep the published 2.0 deprecation deadline for `DatabaseTable.from_fasta_dict` (#67). - Report `size` and `mtime` for an available file from `inspect_file`, `Cache.inspect`, and `inspect_files`, taken from the file that was validated, so a cache listing needs no extra `stat` (#75). - Add `record_provenance=True` to `fetch_file` and `Cache.fetch`. After publishing, a hidden `..datacache.json` records the source URL (without user name, password, query string, or fragment), fetch time, size, and the SHA-256 when `expected_sha256` verified it, with the file's permissions. Inspection reports `source_url`, `fetched_at`, and `recorded_sha256` while the file is unchanged; `verified` still means a supplied digest matched just now. Any new download removes a stale record. Recording is off by default (#75). - Reject empty downloads. A complete but empty response, such as a withdrawn upstream record, was published and then reused as a valid cache entry. An empty HTTP response is now retried like a transient failure, then raises `FileValidationError` without publishing; an empty decompressed archive member is rejected too. An empty file already in a cache is an invalid hit that `force=True` replaces. Pass `allow_empty=True` to `fetch_file` or `Cache.fetch`, or `expected_size=0`, when an empty file is expected (#74). ## 1.12.0 - Add `get_cache_root(name, *envkeys)`, which selects a cache root shared by several packages. The first environment variable that is set is the root itself; otherwise the platform cache directory for `name` is used. Unlike `get_data_dir(subdir, envkey)`, nothing is appended to the variable's value. OpenVax packages use `get_cache_root("openvax", "_CACHE", "OPENVAX_DATA_CACHE")` so that shared data is downloaded once. - Import pandas, numpy, and requests only in the functions that use them. `import datacache` now takes tens of milliseconds instead of hundreds, so command-line tools can resolve cache paths cheaply. `download.pd` and `download.requests` still refer to those modules. - Map numpy's fixed-width types in `db_type` on Python 3: `str_` (`dtype('U…')`) becomes `TEXT`, and `bytes_` (`dtype('S…')`) and `bytes` become `BLOB`, instead of raising `ValueError`. Databases built from DataFrames are unchanged, since pandas stores such arrays with the `object` dtype. - Remove leftover Travis, pylint, and Python 2 configuration and syntax. ## 1.11.1 - Shorten cache names longer than 255 UTF-8 bytes. Names were shortened by character count, so a long non-ASCII filename (for example 100 CJK characters) exceeded Linux's 255-byte limit, and `fetch_file`, `fetch_csv_dataframe`, and `fetch_csv_db` failed with `OSError`. Such names now become a digest followed by as much of their end, including the extension, as fits. Every name within the limit is unchanged. On macOS and Windows, which limit characters rather than bytes, a file already cached under such a name is downloaded once more under the shorter name. ## 1.11.0 - Keep `fetch_csv_db` database names safe. An inferred name spells out each column, so a header containing `/` created subdirectories (and `..` could place the database outside the cache), characters such as `:` and `?` from headers or the CSV filename are invalid on Windows, and wide CSVs exceeded the 255-byte file name limit and failed with `OSError`. Names are now spelled out only when every character is valid on all supported platforms and SQLite's journal still fits, measured the same way everywhere; otherwise the schema is named by a digest. A matching database stored under the historical name is still reused in place. A new `version` rebuilds it under the new name and leaves the older copy alone, since another process may still have it open, and logs a warning naming it. - Explain why a database cannot be rebuilt when the filesystem has no room for its SQLite `-journal` file name: a `version` change or `overwrite=True` now raises a clear `ValueError` instead of `unable to open database file`. Creating and reusing such a database still work. - Reject the reserved `_datacache_metadata` and `sqlite_` table names before reusing a database. Requesting `_datacache_metadata` for an existing database returned the version table instead of storing the DataFrame. - Compare table and column names the way SQLite does, ignoring the case of ASCII letters only: `records` now reuses an existing `Records` table instead of rebuilding the database, and `Éclair` and `éclair` are accepted as distinct. - Report non-string column labels (for example `header=None` without `names=`) with the same `ValueError` whether or not `db_filename` is given, instead of an `AttributeError`. - Choose ZIP members more carefully: a member with the output's name inside a folder, or differing only in letter case, is installed instead of the largest member, preferring the copy nearest the archive root, then an exact-case match. When nothing matches, the largest member is still installed, now with a logged warning if the output was named explicitly and the archive has several. New downloads of such archives install different bytes, so an `expected_sha256` pinned to the previously installed member fails validation; files already in a cache are unchanged. - Suggest `force=True` only when a regular file is present; it cannot replace a directory at the cache path. - Store `float16` DataFrame columns as `FLOAT`. - Let `ensure_dir` accept a directory that another process creates at the same moment. - Declare MD5 cache-key digests as not used for security, so naming works on FIPS-mode Python. Existing cache keys are unchanged. - Deprecate `DatabaseTable.from_fasta_dict`; it will be removed in datacache 2.0. - Correct `fetch_file`'s `subdir` and `timeout` documentation, and document `Cache`, `ensure_dir`, `get_data_dir`, and `clear_cache`. ## 1.10.2 - Add a complete [public API reference](docs/api.md) covering every name in `datacache.__all__` and every public `Cache` method, with signatures, defaults, return values, exceptions, and runnable offline examples. - Lead the README with a copy-pasteable download quickstart and explain how to locate, inspect, and clear a cache directory on each platform. - Use absolute documentation links so the PyPI project page resolves them. - Check the reference in CI: its version, its coverage of `datacache.__all__` and the public `Cache` methods, its signatures, and its examples are all tested. No API, behavior, or cache-format changes. ## 1.10.1 - Rebuild databases containing SQLite full-text-search tables without trying to drop already-removed shadow tables. Failed rebuilds still restore the original tables, full-text indexes, data, and version. - Create every requested SQLite index when generated names collide, including across tables. Keep historical names where available and reuse equivalent non-partial indexes instead of duplicating them. - Keep cache hits unchanged. Existing databases with missing indexes can be rebuilt explicitly with `overwrite=True` or a new database version. ## 1.10.0 - Publish new downloads and decompressed files with normal creation permissions for shared caches; keep staging private and preserve existing modes (#68). - Roll back failed SQLite rebuilds, including explicit overwrites, and publish new databases only after successful construction. Close rejected version lookup connections and support explicit read-only lookup. - Remove existing views during explicit SQLite overwrites in the same transaction, restoring views and their triggers if rebuilding fails. - Create databases through dangling symlinks by staging and publishing at their targets, preserving the links and cleaning up failed builds. - Preserve integer types and precision; support nullable pandas scalars and empty tables. Normalize column constraints consistently and reject ambiguous names. Insert rows incrementally in bounded batches. - Publish custom transformations only after success, honor their source cache directory, and add explicit rebuild/download options. - Fix CSV database filename inference and separate download settings from pandas parsing options. - Keep legacy filenames, database versions, schemas, and cached file modes unchanged on reuse. Open matching databases before validating rebuild inputs, and reuse explicitly named CSV databases without their source file. Honor source integrity and refresh requests when supplied. - Reuse transform sources stored in the old default directory without moving them, while placing new sources in the requested directory. - Keep the existing pandas dependency range and cover pandas 1.4.4 in CI. - Add explicit, single-file `make_file_readable` / `Cache.make_readable` maintenance for owners who want to share older private cache files. - Add optional tqdm progress for downloads, decompression, hash verification, and database insertion, with exception-safe cleanup and per-attempt counts. - Replace stale API documentation with an offline quickstart and guides for shared caches, databases, progress, and downstream integration. There is no cache migration or required rebuild on upgrade. Existing private files remain usable by their owner; use the explicit permission helper to share selected files. Databases already containing corrupted numbers require a deliberate rebuild from source to recover those values. Existing caches are not silently modified. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7039723 datacache-1.27.1/LICENSE0000644000076500000000000002607315264220442014233 0ustar00iskanderwheelApache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright {yyyy} {name of copyright owner} Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7040606 datacache-1.27.1/MANIFEST.in0000644000076500000000000000023515264220442014754 0ustar00iskanderwheelinclude CHANGELOG.md include requirements.txt recursive-include docs *.md recursive-include examples *.py recursive-include tests/fixtures *.sql *.json *.md ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6649191 datacache-1.27.1/PKG-INFO0000644000076500000000000001310015264225206014311 0ustar00iskanderwheelMetadata-Version: 2.4 Name: datacache Version: 1.27.1 Summary: Helpers for transparently downloading datasets Author-email: Alex Rubinsteyn Project-URL: Homepage, https://github.com/openvax/datacache Project-URL: Bug Tracker, https://github.com/openvax/datacache/issues Project-URL: Documentation, https://github.com/openvax/datacache#readme Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console Classifier: Operating System :: OS Independent Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: Apache Software License Classifier: Programming Language :: Python Classifier: Topic :: Scientific/Engineering :: Bio-Informatics Requires-Python: >=3.9 Description-Content-Type: text/markdown License-File: LICENSE Requires-Dist: pandas>=0.15.2 Requires-Dist: platformdirs>=3.0 Requires-Dist: requests>=2.5.1 Requires-Dist: filelock>=3.13 Requires-Dist: tqdm>=4.64 Provides-Extra: progress Provides-Extra: html Requires-Dist: lxml>=4.9; extra == "html" Provides-Extra: test Requires-Dist: pytest>=7; extra == "test" Requires-Dist: pytest-cov; extra == "test" Requires-Dist: ruff; extra == "test" Requires-Dist: lxml>=4.9; extra == "test" Dynamic: license-file # DataCache [![Tests](https://github.com/openvax/datacache/actions/workflows/tests.yml/badge.svg)](https://github.com/openvax/datacache/actions/workflows/tests.yml) [![PyPI](https://img.shields.io/pypi/v/datacache.svg)](https://pypi.org/project/datacache/) Download a dataset once, verify it, and reuse the local file on later runs. DataCache handles streaming downloads, gzip/ZIP decompression, tar-tree installation, offline inspection and SQLite caches built from pandas DataFrames, for Python applications including OpenVax libraries such as pyensembl. ## Install Python 3.9 or newer is required. ```sh python -m pip install datacache # Optional HTML-table conversion: python -m pip install "datacache[html]" ``` ## Quickstart Download a file once, then reuse its local path on later calls: ```python from datacache import Cache cache = Cache("my-project") url = "https://raw.githubusercontent.com/openvax/datacache/master/LICENSE" path = cache.fetch(url, filename="LICENSE", timeout=30) print(path) # Reuses the cached file without a network request, even in a later process. assert cache.fetch(url, filename="LICENSE") == path ``` Replace the URL and filename with your dataset. Existing files are reused until you refresh them with `force=True`, or periodically with `expire_after`; DataCache doesn't ask the server whether a file has changed. Add `show_progress=True` to display a download bar; no extra installation is needed. ## Guides The guides are also published at . - [Working offline example, verification and interface choices](https://github.com/openvax/datacache/blob/master/docs/index.md) - [Downloads, inspection, integrity, freshness, retries and resumption](https://github.com/openvax/datacache/blob/master/docs/downloads.md) - [Tables, SQLite and transformations](https://github.com/openvax/datacache/blob/master/docs/data.md) - [Shared caches](https://github.com/openvax/datacache/blob/master/docs/shared-caches.md) - [Progress and logging](https://github.com/openvax/datacache/blob/master/docs/progress.md) - [Pinned bundles](https://github.com/openvax/datacache/blob/master/docs/bundles.md), [archive trees](https://github.com/openvax/datacache/blob/master/docs/archives.md) and the [fixed-path file registry](https://github.com/openvax/datacache/blob/master/docs/file_registry.md) - [Interface selection and guarantees](https://github.com/openvax/datacache/blob/master/docs/reference/choosing.md) - [Derived-artifact materialization](https://github.com/openvax/datacache/blob/master/docs/materialization.md): caller-owned builders, outputs published together as one bundle, dependency receipts and resumable inputs. - [Complete API reference](https://github.com/openvax/datacache/blob/master/docs/api.md) - [Individual HTTP ZIP members](https://github.com/openvax/datacache/blob/master/docs/zip-members.md): bounded range downloads with compressed and expanded checksums. - [Downstream integration](https://github.com/openvax/datacache/blob/master/docs/integration.md): contracts for consuming libraries. - [Release notes](https://github.com/openvax/datacache/blob/master/CHANGELOG.md) and [release procedure](https://github.com/openvax/datacache/blob/master/RELEASING.md). ## Dependencies and progress display tqdm is installed by default; existing download APIs enable displays with `show_progress=True`, while `materialize` enables them by default. Quiet use does not import tqdm or configure your application's logging. The old `datacache[progress]` install syntax remains supported as a compatibility alias. The existing pandas dependency range is unchanged; upgrading DataCache does not introduce a pandas 1.5 requirement. CI covers pandas 1.4.4, 1.5.3, and current releases on supported Python versions. ## Development ```sh python -m pip install -e ".[test]" ./lint-and-test.sh python -m examples.basic_usage ``` Tests use local files, mocked responses, and local HTTP servers. They do not depend on external dataset servers. See CI for the Python, dependency, and operating-system combinations exercised. Build the site with `python -m pip install -r requirements-docs.txt` and `./docs.sh`. ## License Apache 2.0; see [LICENSE](LICENSE). ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090568.4722145 datacache-1.27.1/README.md0000644000076500000000000001054515264220710014500 0ustar00iskanderwheel# DataCache [![Tests](https://github.com/openvax/datacache/actions/workflows/tests.yml/badge.svg)](https://github.com/openvax/datacache/actions/workflows/tests.yml) [![PyPI](https://img.shields.io/pypi/v/datacache.svg)](https://pypi.org/project/datacache/) Download a dataset once, verify it, and reuse the local file on later runs. DataCache handles streaming downloads, gzip/ZIP decompression, tar-tree installation, offline inspection and SQLite caches built from pandas DataFrames, for Python applications including OpenVax libraries such as pyensembl. ## Install Python 3.9 or newer is required. ```sh python -m pip install datacache # Optional HTML-table conversion: python -m pip install "datacache[html]" ``` ## Quickstart Download a file once, then reuse its local path on later calls: ```python from datacache import Cache cache = Cache("my-project") url = "https://raw.githubusercontent.com/openvax/datacache/master/LICENSE" path = cache.fetch(url, filename="LICENSE", timeout=30) print(path) # Reuses the cached file without a network request, even in a later process. assert cache.fetch(url, filename="LICENSE") == path ``` Replace the URL and filename with your dataset. Existing files are reused until you refresh them with `force=True`, or periodically with `expire_after`; DataCache doesn't ask the server whether a file has changed. Add `show_progress=True` to display a download bar; no extra installation is needed. ## Guides The guides are also published at . - [Working offline example, verification and interface choices](https://github.com/openvax/datacache/blob/master/docs/index.md) - [Downloads, inspection, integrity, freshness, retries and resumption](https://github.com/openvax/datacache/blob/master/docs/downloads.md) - [Tables, SQLite and transformations](https://github.com/openvax/datacache/blob/master/docs/data.md) - [Shared caches](https://github.com/openvax/datacache/blob/master/docs/shared-caches.md) - [Progress and logging](https://github.com/openvax/datacache/blob/master/docs/progress.md) - [Pinned bundles](https://github.com/openvax/datacache/blob/master/docs/bundles.md), [archive trees](https://github.com/openvax/datacache/blob/master/docs/archives.md) and the [fixed-path file registry](https://github.com/openvax/datacache/blob/master/docs/file_registry.md) - [Interface selection and guarantees](https://github.com/openvax/datacache/blob/master/docs/reference/choosing.md) - [Derived-artifact materialization](https://github.com/openvax/datacache/blob/master/docs/materialization.md): caller-owned builders, outputs published together as one bundle, dependency receipts and resumable inputs. - [Complete API reference](https://github.com/openvax/datacache/blob/master/docs/api.md) - [Individual HTTP ZIP members](https://github.com/openvax/datacache/blob/master/docs/zip-members.md): bounded range downloads with compressed and expanded checksums. - [Downstream integration](https://github.com/openvax/datacache/blob/master/docs/integration.md): contracts for consuming libraries. - [Release notes](https://github.com/openvax/datacache/blob/master/CHANGELOG.md) and [release procedure](https://github.com/openvax/datacache/blob/master/RELEASING.md). ## Dependencies and progress display tqdm is installed by default; existing download APIs enable displays with `show_progress=True`, while `materialize` enables them by default. Quiet use does not import tqdm or configure your application's logging. The old `datacache[progress]` install syntax remains supported as a compatibility alias. The existing pandas dependency range is unchanged; upgrading DataCache does not introduce a pandas 1.5 requirement. CI covers pandas 1.4.4, 1.5.3, and current releases on supported Python versions. ## Development ```sh python -m pip install -e ".[test]" ./lint-and-test.sh python -m examples.basic_usage ``` Tests use local files, mocked responses, and local HTTP servers. They do not depend on external dataset servers. See CI for the Python, dependency, and operating-system combinations exercised. Build the site with `python -m pip install -r requirements-docs.txt` and `./docs.sh`. ## License Apache 2.0; see [LICENSE](LICENSE). ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792092805.647323 datacache-1.27.1/datacache/0000755000076500000000000000000015264225206015116 5ustar00iskanderwheel././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090568.4725053 datacache-1.27.1/datacache/__init__.py0000644000076500000000000000535115264220710017226 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from .download import ( expected_path, file_exists, fetch_bytes, fetch_file, fetch_and_transform, fetch_csv_dataframe, ) from .integrity import FileValidationError, validate_file from .inspection import CacheInspection, FileInspection, inspect_file, inspect_files from .permissions import make_file_readable from .database_helpers import ( db_from_dataframe, db_from_dataframes, db_from_dataframes_with_absolute_path, fetch_csv_db, connect_if_correct_version ) from .common import ( ensure_dir, get_data_dir, get_cache_root, build_path, resolve_path, clear_cache, build_local_filename ) from .cache import Cache from .resume import CannotResume, discard_partial from .bundles import BundleInspection, VersionedDatasetRegistry, inspect_bundle, install_bundle from .bundle_store import list_bundles, prune_bundles from .archives import ( ArchiveInspection, VersionedArchiveRegistry, inspect_archive, install_archive) from .file_registry import VersionedFileRegistry from .materialization import MaterializationInspection, inspect_materialization, materialize from .version import __version__ from .zip_members import install_zip_member __all__ = [ 'install_zip_member', '__version__', 'fetch_file', 'fetch_bytes', 'discard_partial', 'CannotResume', 'BundleInspection', 'MaterializationInspection', 'inspect_materialization', 'materialize', 'ArchiveInspection', 'VersionedArchiveRegistry', 'VersionedDatasetRegistry', 'VersionedFileRegistry', 'inspect_bundle', 'install_bundle', 'list_bundles', 'prune_bundles', 'inspect_archive', 'install_archive', 'expected_path', 'file_exists', 'resolve_path', 'FileInspection', 'CacheInspection', 'inspect_file', 'inspect_files', 'FileValidationError', 'validate_file', 'make_file_readable', 'fetch_and_transform', 'fetch_csv_dataframe', 'db_from_dataframe', 'db_from_dataframes', 'db_from_dataframes_with_absolute_path', 'fetch_csv_db', 'connect_if_correct_version', 'ensure_dir', 'get_data_dir', 'get_cache_root', 'build_path', 'clear_cache', 'build_local_filename', 'Cache', ] ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7044752 datacache-1.27.1/datacache/_filesystem.py0000644000076500000000000000557215264220442020021 0ustar00iskanderwheel"""Small local-filesystem primitives shared by resumable and bundle installs.""" from contextlib import contextmanager import json import os from pathlib import Path import stat import tempfile from .integrity import FileValidationError def path_present(path): """Like lexists, but never turn permission errors into absence.""" try: os.lstat(path) except FileNotFoundError: return False return True def open_regular(path, flags=os.O_RDONLY, mode=0o600, *, private=False): """Open without following links or blocking on special files.""" flags |= getattr(os, 'O_NOFOLLOW', 0) | getattr(os, 'O_NONBLOCK', 0) before = None try: before = os.lstat(path) except FileNotFoundError: pass if before is not None and not stat.S_ISREG(before.st_mode): raise FileValidationError(path, 'expected a regular file, not a link or special file') fd = os.open(path, flags, mode) try: info = os.fstat(fd) if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1: raise FileValidationError(path, 'expected a regular file with a single link') if private and (info.st_uid != os.getuid() or stat.S_IMODE(info.st_mode) & 0o077): raise FileValidationError(path, 'resume state must be owner-only') return fd except BaseException: os.close(fd) raise @contextmanager def file_lock(path, *, private=False): """Permanent inode, advisory POSIX lock; never unlink a live lock file.""" if os.name != 'posix': raise NotImplementedError('Resumable and bundle installs require a POSIX local filesystem') import fcntl fd = open_regular(path, os.O_RDWR | os.O_CREAT, 0o600 if private else 0o666, private=private) try: fcntl.flock(fd, fcntl.LOCK_EX) yield finally: os.close(fd) def read_json(path, limit=1024 * 1024): """Read a regular JSON file; ``limit=None`` permits growing tree manifests.""" with os.fdopen(open_regular(path), 'rb') as handle: data = handle.read() if limit is None else handle.read(limit + 1) if limit is not None and len(data) > limit: raise ValueError('JSON record is too large') return json.loads(data) def write_json(path, value, mode=0o600): path = Path(path) fd, temporary = tempfile.mkstemp(prefix='.datacache-json-', dir=path.parent) try: with os.fdopen(fd, 'w', encoding='utf-8') as handle: json.dump(value, handle, sort_keys=True) handle.write('\n') handle.flush() os.fsync(handle.fileno()) if hasattr(os, 'fchmod'): os.fchmod(handle.fileno(), mode) else: # Windows: no descriptor chmod; staging is still private. os.chmod(temporary, mode) os.replace(temporary, path) finally: Path(temporary).unlink(missing_ok=True) ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792090401.704823 datacache-1.27.1/datacache/archives.py0000644000076500000000000011376015264220442017301 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Safe, transactional installation of complete directory trees from tar archives.""" from dataclasses import dataclass, field from datetime import datetime, timezone import hashlib import json import os from pathlib import Path import shutil import tarfile from uuid import uuid4 from ._filesystem import open_regular, read_json, write_json from .bundle_store import ( CHUNK_SIZE, INVALID_STORE_ERRORS, MANIFEST, BundleStore, check_tree, discard_private_directory, hash_file, list_tree, local_file_identity, private_directory, prune_bundles, source_fingerprint, validate_distinct_paths, validate_file_record, validate_path_component, validate_relative_name, ) from .download import _run_validator, normal_creation_mode, validate_download_options, validate_limit, validate_size_within_limit from .integrity import FileValidationError, _validate_expectations from .provenance import redact_url FORMAT = 1 @dataclass(frozen=True) class ArchiveInspection: """Read-only status of an archive store's current bundle, the extracted tree. ``status`` is available, missing, invalid or inaccessible. ``bundle`` is the extracted tree's directory. ``verified`` means caller-supplied trusted archive or part hashes identified the installed bytes. Receipt-only inspection still hashes every installed file, but an observed receipt is not an independent source of trust. """ path: str status: str verified: bool = False bundle: object = None files: dict = field(default_factory=dict) error: object = None source_urls: tuple = () fetched_at: object = None archive_size: object = None recorded_sha256: object = None def _normalize_sources(sources): """Normalize one source or an explicitly ordered sequence of archive parts.""" if isinstance(sources, (str, os.PathLike, dict)): sources = [sources] elif not isinstance(sources, (list, tuple)): raise ValueError("sources must be a URL, path, mapping, or ordered sequence") if not sources: raise ValueError("sources must contain at least one archive source") result = [] for source in sources: if isinstance(source, os.PathLike): source = {"path": source} elif isinstance(source, str): source = {"url": source} elif isinstance(source, dict): source = dict(source) else: raise ValueError("each archive source must be a URL, path, or mapping") unknown = set(source) - {"url", "path", "sha256", "size"} if unknown: raise ValueError("unknown archive source options: %s" % sorted(unknown)) url, path = source.get("url"), source.get("path") if url is not None and (not isinstance(url, str) or not url): raise ValueError("archive source url must be a nonempty string") if path is not None: try: path = Path(path) except TypeError as error: raise ValueError("archive source path must be path-like") from error if url is None and path is None: raise ValueError("each archive source requires url or path") identity = url if url is not None else local_file_identity(path) digest, size = source.get("sha256"), source.get("size") _validate_expectations(digest, size) result.append({ "url": url, "path": path, "identity": identity, "fingerprint": source_fingerprint(identity), "sha256": digest.lower() if digest else None, "size": size, }) return tuple(result) def _normalize_extra_files(extra_files): if extra_files is None: return {} if not isinstance(extra_files, dict): raise ValueError("extra_files must be a mapping of relative paths to text or bytes") result = {} for name, value in extra_files.items(): validate_relative_name(name) if isinstance(value, str): value = value.encode("utf-8") elif not isinstance(value, bytes): raise ValueError("extra file contents must be text or bytes") result[name] = value validate_distinct_paths(result) return result def _definition(sources, expected_sha256, expected_size, extra_files, require_verified=False): normalized = _normalize_sources(sources) _validate_expectations(expected_sha256, expected_size) expected_sha256 = expected_sha256.lower() if expected_sha256 else None extras = _normalize_extra_files(extra_files) trusted = bool(expected_sha256 and expected_size is not None) or all( source["sha256"] and source["size"] is not None for source in normalized) if require_verified and not trusted: raise ValueError( "verified archive installs require expected_sha256 and expected_size " "for the assembled archive, or sha256 and size for every part") return { "sources": normalized, "expected_sha256": expected_sha256, "expected_size": expected_size, "extra_files": extras, "trusted": trusted, } def _manifest(receipt): if (not isinstance(receipt, dict) or receipt.get("format") != FORMAT or receipt.get("kind") != "archive-tree"): raise ValueError("unrecognized archive manifest") archive = receipt.get("archive") sources = receipt.get("sources") files = receipt.get("files") extra_files = receipt.get("extra_files") directories = receipt.get("directories") fetched_at = receipt.get("fetched_at") if (not isinstance(archive, dict) or not isinstance(sources, list) or not sources or not isinstance(files, dict) or not files or not isinstance(extra_files, dict) or not isinstance(directories, list) or not isinstance(fetched_at, str)): raise ValueError("invalid archive manifest") validate_file_record(archive) normalized_sources = [] for source in sources: if not isinstance(source, dict): raise ValueError("invalid archive source receipt") if set(source) != {"url", "fingerprint", "sha256", "size"}: raise ValueError("invalid archive source receipt") if (not isinstance(source["url"], str) or not isinstance(source["fingerprint"], str) or len(source["fingerprint"]) != 64 or any(char not in "0123456789abcdef" for char in source["fingerprint"])): raise ValueError("invalid archive source receipt") validate_file_record({"sha256": source["sha256"], "size": source["size"]}) normalized_sources.append(source) normalized_files = {} for name, spec in files.items(): normalized_files[validate_relative_name(name)] = validate_file_record(spec) for name, spec in extra_files.items(): if name not in normalized_files or spec != normalized_files[name]: raise ValueError("consumer extra file receipt disagrees with tree manifest") normalized_directories = [] folded = set() for name in directories: validate_relative_name(name) key = name.casefold() if key in folded: raise ValueError("duplicate directory in archive manifest") folded.add(key) normalized_directories.append(name) file_keys = {name.casefold() for name in normalized_files} if len(file_keys) != len(normalized_files) or file_keys & folded: raise ValueError("colliding paths in archive manifest") return archive, normalized_sources, normalized_files, normalized_directories, fetched_at, extra_files def _open_local_source(path): descriptor = open_regular(path, os.O_RDONLY | getattr(os, "O_BINARY", 0)) try: return os.fdopen(descriptor, "rb") except BaseException: os.close(descriptor) raise def _matches_definition(receipt_archive, receipt_sources, receipt_extras, definition): expected_sha256 = definition["expected_sha256"] expected_size = definition["expected_size"] if expected_sha256 is not None and receipt_archive["sha256"] != expected_sha256: raise FileValidationError("archive", "assembled archive hash disagrees with requested source") if expected_size is not None and receipt_archive["size"] != expected_size: raise FileValidationError("archive", "assembled archive size disagrees with requested source") if len(receipt_sources) != len(definition["sources"]): raise FileValidationError("archive", "archive part count disagrees with requested source") for recorded, expected in zip(receipt_sources, definition["sources"]): if expected["sha256"] is not None and recorded["sha256"] != expected["sha256"]: raise FileValidationError("archive", "archive part hash disagrees with requested source") if expected["size"] is not None and recorded["size"] != expected["size"]: raise FileValidationError("archive", "archive part size disagrees with requested source") if not definition["trusted"]: fingerprints = [source["fingerprint"] for source in receipt_sources] expected = [source["fingerprint"] for source in definition["sources"]] if fingerprints != expected: raise FileValidationError("archive", "ordered archive source identities disagree") if set(receipt_extras) != set(definition["extra_files"]): raise FileValidationError("archive", "consumer extra file names disagree with requested definition") for name, value in definition["extra_files"].items(): expected = {"sha256": hashlib.sha256(value).hexdigest(), "size": len(value)} if receipt_extras[name] != expected: raise FileValidationError(name, "extra file disagrees with requested content") def _check_bundle(store, directory, definition, verify_files=True): """Check one bundle, or a tree staged to become one, against its manifest.""" # Tree inventories grow with the archive; the writer has no fixed size cap. receipt = read_json(directory / MANIFEST, limit=None) archive, sources, files, directories, fetched_at, extras = _manifest(receipt) if definition is not None: _matches_definition(archive, sources, extras, definition) metadata = { "source_urls": tuple(source["url"] for source in sources), "fetched_at": fetched_at, "archive_size": archive["size"], "recorded_sha256": archive["sha256"], } if not verify_files: # The manifest alone: path lookups stay one read, however large the tree. return ArchiveInspection(str(store.path), "available", False, str(directory), **metadata) hashed = check_tree(directory, files, directories, ignore=(MANIFEST,)) # Only hashes the caller supplied verify files; the manifest's own don't. verified = bool(definition and definition["trusted"]) inspections = {name: value.inspection(verified=verified) for name, value in hashed.items()} return ArchiveInspection( str(store.path), "available", verified, str(directory), inspections, **metadata) def inspect_archive( destination, sources=None, *, expected_sha256=None, expected_size=None, extra_files=None, verify_files=True): """Check the current archive bundle without writes, locks, or network. Omit ``sources`` for receipt-only consistency checking. Supplying sources checks the requested ordered archive identity or trusted content hashes. ``verify_files=False`` checks the manifest and source metadata without reading the extracted tree; its result is never marked verified. """ if not isinstance(verify_files, bool): raise ValueError("verify_files must be a boolean") definition = None if sources is not None: definition = _definition( sources, expected_sha256, expected_size, extra_files, require_verified=False) elif any(value is not None for value in (expected_sha256, expected_size, extra_files)): raise ValueError("sources are required with archive expectations or extra_files") path = Path(destination) store = BundleStore(path, "archive") try: bundle = store.current_bundle() if bundle is None: return ArchiveInspection(str(path), "missing") return _check_bundle(store, bundle, definition, verify_files=verify_files) except PermissionError as error: return ArchiveInspection(str(path), "inaccessible", error=error) except INVALID_STORE_ERRORS as error: return ArchiveInspection(str(path), "invalid", error=error) def _archive_members( archive, max_members=None, max_extracted_size=None, extra_names=()): nodes = {} planned = [] total = 0 regular_files = 0 member_count = 0 for member in archive: member_count += 1 if max_members is not None and member_count > max_members: raise FileValidationError(archive.name, "archive has more than %d members" % max_members) if member.isdir(): name, kind = member.name.rstrip("/"), "directory" elif member.isfile(): name, kind = member.name, "file" if member.size < 0: raise FileValidationError(archive.name, "archive member has negative size") total += member.size if max_extracted_size is not None and total > max_extracted_size: raise FileValidationError( archive.name, "archive expands to more than %d bytes" % max_extracted_size) regular_files += 1 else: raise FileValidationError( archive.name, "archive contains a link or special member: %s" % member.name) # Standard tar tools use '.' for the root and './' for relative paths. while name.startswith("./"): name = name[2:] if kind == "directory" and name == ".": continue try: validate_relative_name(name) except ValueError as error: raise FileValidationError( archive.name, "unsafe archive member path: %s" % member.name) from error parts = name.split("/") for index in range(1, len(parts)): parent = "/".join(parts[:index]) key = parent.casefold() previous = nodes.get(key) if previous is None: nodes[key] = (parent, "directory", False) elif previous[0] != parent or previous[1] != "directory": raise FileValidationError(archive.name, "archive member paths collide: %s" % name) key = name.casefold() previous = nodes.get(key) if kind == "directory" and previous is not None and previous[1] == "directory" and not previous[2]: if previous[0] != name: raise FileValidationError(archive.name, "archive member paths collide: %s" % name) nodes[key] = (name, kind, True) elif previous is not None: raise FileValidationError(archive.name, "archive member paths collide: %s" % name) else: nodes[key] = (name, kind, True) planned.append((member, name, kind)) if not member_count: raise FileValidationError(archive.name, "archive is empty") for name in extra_names: parts = name.split("/") for index in range(1, len(parts)): parent = "/".join(parts[:index]) key = parent.casefold() previous = nodes.get(key) if previous is None: nodes[key] = (parent, "directory", False) elif previous[0] != parent or previous[1] != "directory": raise FileValidationError( archive.name, "extra file path collides with archive content: %s" % name) if name.casefold() in nodes: raise FileValidationError( archive.name, "extra file path collides with archive content: %s" % name) nodes[name.casefold()] = (name, "file", True) if not regular_files: raise FileValidationError(archive.name, "archive contains no regular files") return planned def _extract_tar( archive_path, destination, max_members=None, max_extracted_size=None, extra_names=()): try: with tarfile.open(archive_path, "r:*") as archive: planned = _archive_members( archive, max_members, max_extracted_size, extra_names) directories = sorted( {name for _, name, kind in planned if kind == "directory"} | {"/".join(name.split("/")[:index]) for _, name, _ in planned for index in range(1, len(name.split("/")))}, key=lambda name: (name.count("/"), name.casefold())) for name in directories: (destination / Path(*name.split("/"))).mkdir(exist_ok=True) for member, name, kind in planned: if kind == "directory": continue target = destination / Path(*name.split("/")) source = archive.extractfile(member) if source is None: raise FileValidationError(archive_path, "could not read archive member %s" % name) flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_BINARY", 0) flags |= getattr(os, "O_NOFOLLOW", 0) descriptor = os.open(target, flags, 0o666) count = 0 try: output = os.fdopen(descriptor, "wb") except BaseException: os.close(descriptor) source.close() raise with source, output: for chunk in iter(lambda: source.read(CHUNK_SIZE), b""): output.write(chunk) count += len(chunk) if count != member.size: raise FileValidationError( archive_path, "archive member %s is truncated" % name) except FileValidationError: raise except (tarfile.TarError, EOFError) as error: raise FileValidationError(archive_path, "invalid tar archive: %s" % error) from error def _working_key(definition): serializable = { "sources": [{key: source[key] for key in ("identity", "sha256", "size")} for source in definition["sources"]], "expected_sha256": definition["expected_sha256"], "expected_size": definition["expected_size"], "extra_files": {name: hashlib.sha256(value).hexdigest() for name, value in definition["extra_files"].items()}, } return hashlib.sha256(json.dumps(serializable, sort_keys=True).encode("utf-8")).hexdigest() def _part_expectations(definition, index): source = definition["sources"][index] digest, size = source["sha256"], source["size"] if len(definition["sources"]) == 1: digest = digest or definition["expected_sha256"] size = size if size is not None else definition["expected_size"] return digest, size def _assemble_archive(working, definition, options, keep_parts): from .download import fetch_file archive_path = working / "archive.tar" aggregate = hashlib.sha256() aggregate_size = 0 recorded = [] with archive_path.open("wb") as assembled: for index, source in enumerate(definition["sources"]): expected_sha256, expected_size = _part_expectations(definition, index) temporary = None if source["path"] is not None: handle = _open_local_source(source["path"]) else: temporary = working / ("part-%06d" % index) fetch_file( source["url"], destination=temporary, raw=True, expected_sha256=expected_sha256, expected_size=expected_size, **options) handle = _open_local_source(temporary) digest, size = hashlib.sha256(), 0 with handle: for chunk in iter(lambda: handle.read(CHUNK_SIZE), b""): assembled.write(chunk) aggregate.update(chunk) digest.update(chunk) size += len(chunk) aggregate_size += len(chunk) actual = digest.hexdigest() if expected_size is not None and size != expected_size: raise FileValidationError(source["path"] or source["url"], "archive part size mismatch") if expected_sha256 is not None and actual != expected_sha256: raise FileValidationError(source["path"] or source["url"], "archive part SHA-256 mismatch") recorded.append({ "url": redact_url(source["identity"]), "fingerprint": source["fingerprint"], "sha256": actual, "size": size, }) if temporary is not None and not keep_parts: temporary.unlink() actual_sha256 = aggregate.hexdigest() if (definition["expected_size"] is not None and aggregate_size != definition["expected_size"]): raise FileValidationError(archive_path, "assembled archive size mismatch") if (definition["expected_sha256"] is not None and actual_sha256 != definition["expected_sha256"]): raise FileValidationError(archive_path, "assembled archive SHA-256 mismatch") return archive_path, {"sha256": actual_sha256, "size": aggregate_size}, recorded def install_archive( destination, sources, *, expected_sha256=None, expected_size=None, extra_files=None, force=False, verified=True, download_options=None, max_members=None, max_extracted_size=None, validator=None): """Safely extract a complete tar archive and publish the tree as a new bundle. ``sources`` is one URL/path or an ordered sequence. Source mappings accept ``url``, ``path``, ``sha256``, and ``size``; supplying both URL and path uses local bytes while retaining the URL as source identity. Split parts are concatenated byte-for-byte in the supplied order. ``extra_files`` are written after extraction and before the tree receipt, which lets consumers publish their own source record in the same transaction. ``validator(tree)`` may inspect the extracted tree (including extra files) and reject it by raising or returning False. It must not modify the tree. Validation runs before publication and on reuse; rejection raises FileValidationError and leaves the current bundle unchanged. It does not change source identity or establish cryptographic verification. """ if not isinstance(force, bool) or not isinstance(verified, bool): raise ValueError("force and verified must be booleans") if validator is not None and not callable(validator): raise ValueError("validator must be callable") validate_limit("max_members", max_members) validate_limit("max_extracted_size", max_extracted_size) definition = _definition( sources, expected_sha256, expected_size, extra_files, require_verified=verified) options = validate_download_options(download_options, "archive") for index, source in enumerate(definition["sources"]): if source["path"] is None: size = _part_expectations(definition, index)[1] validate_size_within_limit(size, options.get("max_bytes"), "part %d size" % index) if options.get("resume") is True: # "auto" downloads normally where it can't resume. # Check before creating anything: a resumable install keeps its staging. from .resume import validate_resume for index, source in enumerate(definition["sources"]): digest, size = _part_expectations(definition, index) if source["path"] is None: if digest is None or size is None: raise ValueError("resumable archive parts require sha256 and size") validate_resume(source["url"], digest, size) path = Path(destination) store = BundleStore(path, "archive") inspection = inspect_archive( path, sources, expected_sha256=expected_sha256, expected_size=expected_size, extra_files=extra_files, verify_files=not force) # A forced install replaces it unhashed. if store.can_reuse(inspection, force=force): _run_validator(validator, Path(inspection.bundle), path, "archive tree") return Path(inspection.bundle) path.parent.mkdir(parents=True, exist_ok=True) with store.lock(): store.create() inspection = inspect_archive( path, sources, expected_sha256=expected_sha256, expected_size=expected_size, extra_files=extra_files, verify_files=not force) if store.can_reuse(inspection, force=force): _run_validator(validator, Path(inspection.bundle), path, "archive tree") return Path(inspection.bundle) # Parts are kept across attempts only when each is pinned, so a kept # part can never be stale; "auto" doesn't require pinning, True does. resumable = options.get("resume") is True or (options.get("resume") == "auto" and all( source["path"] is not None or None not in _part_expectations(definition, index) for index, source in enumerate(definition["sources"]))) working = private_directory( path / (".staging-" + (_working_key(definition) if resumable else uuid4().hex))) tree = working / "tree" if tree.exists(): shutil.rmtree(tree) tree.mkdir() published = False try: archive_path, archive_record, source_records = _assemble_archive( working, definition, options, keep_parts=resumable) _extract_tar( archive_path, tree, max_members, max_extracted_size, definition["extra_files"]) for name, value in definition["extra_files"].items(): target = tree / Path(*name.split("/")) target.parent.mkdir(parents=True, exist_ok=True) flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_BINARY", 0) flags |= getattr(os, "O_NOFOLLOW", 0) try: descriptor = os.open(target, flags, 0o666) except FileExistsError as error: raise FileValidationError(target, "extra file collides with archive content") from error with os.fdopen(descriptor, "wb") as output: output.write(value) _run_validator(validator, tree, path, "archive tree") listing = list_tree(tree) files = {name: hash_file(file_path).record for name, file_path in listing.files.items()} receipt = { "format": FORMAT, "kind": "archive-tree", "fetched_at": datetime.now(timezone.utc).isoformat(), "archive": archive_record, "sources": source_records, "files": files, "extra_files": {name: files[name] for name in definition["extra_files"]}, "directories": listing.directories, } write_json(tree / MANIFEST, receipt, mode=normal_creation_mode(tree)) # Every file was just hashed into the manifest: read it back and # check it against the tree, without hashing again. _check_bundle(store, tree, definition, verify_files=False) bundle = store.publish(tree) published = True return bundle finally: if published or not resumable: discard_private_directory(working) class VersionedArchiveRegistry: """Versioned catalogue of complete archive-tree downloads. Each item has ``default_version`` and a ``versions`` mapping. A version is one source, an ordered source sequence, or a mapping containing ``sources`` plus archive expectations, consumer ``extra_files``, and extraction limits. ``cache_root`` uses ``//`` stores; ``store_path`` can instead preserve a consumer's established ``(name, version)`` layout. Construction validates definitions without touching the filesystem. """ def __init__( self, archives, *, cache_root=None, store_path=None, verified=True): if (cache_root is None) == (store_path is None): raise ValueError("provide exactly one of cache_root or store_path") if store_path is not None and not callable(store_path): raise ValueError("store_path must be callable") if not isinstance(verified, bool): raise ValueError("verified must be a boolean") if not isinstance(archives, dict) or not archives: raise ValueError("archives must be a nonempty mapping") self.verified = verified self._path = (store_path if store_path is not None else lambda name, version: Path(cache_root) / name / version) self._archives = {} version_options = { "sources", "expected_sha256", "expected_size", "extra_files", "max_members", "max_extracted_size", } for name, archive in archives.items(): validate_path_component(name) if not isinstance(archive, dict): raise ValueError("archive definitions must be mappings") unknown = set(archive) - {"default_version", "versions", "description"} if unknown: raise ValueError("unknown archive definition options: %s" % sorted(unknown)) versions = archive.get("versions") if not isinstance(versions, dict) or not versions: raise ValueError("each archive requires a nonempty versions mapping") normalized_versions = {} for version, value in versions.items(): validate_path_component(version) if isinstance(value, dict) and "sources" in value: options = dict(value) unknown = set(options) - version_options if unknown: raise ValueError("unknown archive version options: %s" % sorted(unknown)) else: options = {"sources": value} validate_limit("max_members", options.get("max_members")) validate_limit("max_extracted_size", options.get("max_extracted_size")) definition = _definition( options["sources"], options.get("expected_sha256"), options.get("expected_size"), options.get("extra_files"), require_verified=verified) normalized_versions[version] = { "sources": tuple({key: source[key] for key in ("url", "path", "sha256", "size") if source[key] is not None} for source in definition["sources"]), "expected_sha256": definition["expected_sha256"], "expected_size": definition["expected_size"], "extra_files": definition["extra_files"], "max_members": options.get("max_members"), "max_extracted_size": options.get("max_extracted_size"), } default = archive.get("default_version") if default not in normalized_versions: raise ValueError("default_version must name a concrete archive version") # Versions are directories: case-insensitive filesystems would give # two that differ only by case one store. validate_distinct_paths(list(normalized_versions)) self._archives[name] = { "default_version": default, "versions": normalized_versions, "description": archive.get("description", ""), } validate_distinct_paths(list(self._archives)) def resolve_version(self, name, version=None): if name not in self._archives: raise ValueError( "unknown archive %r; known: %s" % ( name, ", ".join(sorted(self._archives)))) definition = self._archives[name] version = definition["default_version"] if version is None else version if version not in definition["versions"]: raise ValueError("unknown version %r for %s" % (version, name)) return version def store_path(self, name, version=None): """Return the managed store path without creating or inspecting it.""" version = self.resolve_version(name, version) return Path(self._path(name, version)) def _version(self, name, version): version = self.resolve_version(name, version) return version, self._archives[name]["versions"][version] @staticmethod def _local_sources(sources, source_paths): if source_paths is None: return sources if isinstance(source_paths, (str, os.PathLike)): source_paths = [source_paths] elif not isinstance(source_paths, (list, tuple)): raise ValueError("source_paths must be a path or ordered sequence") if len(source_paths) != len(sources): raise ValueError("source_paths must have one path for every archive part") result = [] for source, path in zip(sources, source_paths): item = dict(source) if "url" not in item: item["url"] = local_file_identity(item["path"]) item["path"] = Path(path) result.append(item) return tuple(result) def inspect(self, name, version=None, *, verify_files=True): """Inspect one version offline; no destination or lock is created.""" version, definition = self._version(name, version) return inspect_archive( self.store_path(name, version), definition["sources"], expected_sha256=definition["expected_sha256"], expected_size=definition["expected_size"], extra_files=definition["extra_files"], verify_files=verify_files) def download( self, name, version=None, *, force=False, source_paths=None, **download_options): """Install/reuse one version and return its current bundle, the extracted tree.""" version, definition = self._version(name, version) sources = self._local_sources(definition["sources"], source_paths) return install_archive( self.store_path(name, version), sources, expected_sha256=definition["expected_sha256"], expected_size=definition["expected_size"], extra_files=definition["extra_files"], force=force, verified=self.verified, download_options=download_options, max_members=definition["max_members"], max_extracted_size=definition["max_extracted_size"]) def prune(self, name, version=None, *, keep=1): """Delete all but the newest keep bundles of one version; see prune_bundles.""" return prune_bundles(self.store_path(name, version), keep=keep) def local_path(self, name, version=None, *, verify_files=False): """Return the current bundle without downloading or repairing.""" inspected = self.inspect(name, version, verify_files=verify_files) if inspected.status == "missing": raise FileNotFoundError(inspected.path) if inspected.status == "inaccessible": raise inspected.error if inspected.status != "available": raise FileValidationError( inspected.path, "archive installation is %s" % inspected.status) return Path(inspected.bundle) def ensure(self, name, version=None, **download_options): """Install if needed, then return the current bundle's path.""" return self.download(name, version, **download_options) def is_cached(self, name, version=None, *, verify_files=False): """Whether one version has an available current bundle.""" return self.inspect( name, version, verify_files=verify_files).status == "available" def status(self, name=None, *, verify_files=False): """Return one offline status row per concrete archive version. Rows check publication and source metadata without reading payloads; verify_files=True hashes every file as inspect does by default. """ if name is not None: self.resolve_version(name) names = [name] else: names = sorted(self._archives) rows = [] for archive_name in names: archive = self._archives[archive_name] for version, definition in archive["versions"].items(): inspection = self.inspect( archive_name, version, verify_files=verify_files) identities = [source.get("url") or local_file_identity(source["path"]) for source in definition["sources"]] rows.append({ "name": archive_name, "version": version, "default": version == archive["default_version"], "description": archive["description"], "sources": [redact_url(identity) for identity in identities], "downloaded_sources": list(inspection.source_urls), "path": str(self.store_path(archive_name, version)), "bundle": inspection.bundle, "status": inspection.status, "cached": inspection.status == "available", "fetched_at": inspection.fetched_at, "archive_size": inspection.archive_size, "recorded_sha256": inspection.recorded_sha256, "inspection": inspection, }) return rows ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090568.5055842 datacache-1.27.1/datacache/bundle_store.py0000644000076500000000000005510615264220711020160 0ustar00iskanderwheel"""Bundle stores: directories that keep complete sets of files. install_bundle, install_archive and materialize each keep one store directory per dataset version, for example /ensembl/110:: /ensembl/110/ .datacache-store.json {"format": 2, "kind": "bundle"} bundles/ 2026-09-30T14-11-05Z/ an older bundle 2026-10-08T17-02-42Z/ the current bundle: always the newest genes.gtf .datacache-manifest.json what the bundle contains A bundle is a complete set of files that belong together. An install builds the new bundle in a hidden staging directory, checks it, then renames it into bundles/. A rename happens completely or not at all, so everything in bundles/ is complete, and readers see the old bundle or the new one, never a mix. Old bundles are kept because a running program may still be reading them; nothing deletes them automatically. The functions after BundleStore check the files inside a bundle: their names, their hashes, and that a bundle holds exactly the files it should. """ from dataclasses import dataclass from datetime import datetime, timedelta, timezone import hashlib import logging import os from pathlib import Path import re import shutil import stat from typing import Optional import unicodedata from filelock import FileLock from ._filesystem import file_lock, open_regular, path_present, read_json, write_json from .download import normal_creation_mode from .inspection import FileInspection from .integrity import FileValidationError, _validate_expectations from .progress import Progress from .provenance import sidecar_path FORMAT = 2 MARKER = '.datacache-store.json' MANIFEST = '.datacache-manifest.json' CHUNK_SIZE = 2 ** 20 # UTC install time. Windows forbids ':', and this sorts oldest to newest. BUNDLE_NAME_FORMAT = '%Y-%m-%dT%H-%M-%SZ' KINDS = ('bundle', 'archive', 'materialization', 'zip-member') # Exceptions meaning a store or bundle is unusable as found: corrupt, # incomplete, foreign or unreadable. PermissionError is one of them, so callers # that report it separately must catch it first. INVALID_STORE_ERRORS = (OSError, ValueError, KeyError, TypeError, RecursionError) logger = logging.getLogger(__name__) class BundleStore: """The store directory for one dataset version, and the bundles in it. kind is 'bundle', 'archive' or 'materialization', recorded in the marker so that no kind of install takes over another's store. Reading never writes or locks. Installs hold lock() while they call create and publish. """ def __init__(self, path, kind): self.path = Path(path) self.kind = kind self.bundles = self.path / 'bundles' # Beside the store, so it exists before the store does. Named for the # store's own name, case-folded and Unicode-normalized, so spellings a # filesystem treats as one directory share a lock. absolute = Path(os.path.abspath(self.path)) name = unicodedata.normalize('NFC', absolute.name.casefold()) key = hashlib.sha256(os.fsencode(name)).hexdigest()[:32] self.lock_path = absolute.parent / ('.datacache-lock-' + key) def __repr__(self): return 'BundleStore(%r, %r)' % (str(self.path), self.kind) def lock(self): """The lock installs into this store take turns on.""" if os.name == 'posix': return file_lock(self.lock_path) return FileLock(str(self.lock_path)) def can_reuse(self, inspection, *, force): """Whether an install can return the bundle it just inspected. Raises when the install can't go ahead either: the store is unreadable, or its current bundle is invalid and force=True wasn't given. A directory this store may never take over says so, rather than suggesting force=True. """ if inspection.status == 'inaccessible': raise inspection.error if inspection.status == 'invalid' and not force: self.refuse_takeover() raise FileValidationError( self.path, "the current bundle is damaged or doesn't match this request; " "use force=True to install a new one") from inspection.error return not force and inspection.status == 'available' def check(self): """Raise unless path holds this kind's store marker.""" require_directory(self.path) if read_json(self.path / MARKER) != {'format': FORMAT, 'kind': self.kind}: raise FileValidationError(self.path, 'not a datacache %s store' % self.kind) def bundle_names(self): """Names of every bundle in the store, oldest first. Other entries are ignored.""" return sorted(entry.name for entry in self.bundles.iterdir() if is_bundle_name(entry.name)) def current_bundle(self): """The newest bundle's directory, or None when nothing is installed. Raises one of INVALID_STORE_ERRORS when path exists but isn't a store of this kind. An empty directory counts as nothing installed: it's where the first install will create the store. """ if not path_present(self.path): return None try: self.check() except FileNotFoundError: if not holds_files(self.path): return None # Empty: the first install will create the store here. self.check() # Another installer may have placed the marker since. if not path_present(self.bundles): return None # create() places the marker first, then bundles/. require_directory(self.bundles) names = self.bundle_names() if not names: return None bundle = self.bundles / names[-1] require_directory(bundle) return bundle def create(self): """Make path a store of this kind, never taking over a directory. A missing or empty directory becomes the store in place, so it keeps its owner, group and permissions: first the marker, then bundles/. Readers see nothing installed until a bundle is published. An existing store of this kind gets back anything that is missing. Anything else raises FileValidationError (see refuse_takeover). """ self.path.mkdir(exist_ok=True) self.refuse_takeover() if not path_present(self.path / MARKER): write_json(self.path / MARKER, {'format': FORMAT, 'kind': self.kind}, mode=normal_creation_mode(self.path)) self.check() self.bundles.mkdir(exist_ok=True) require_directory(self.bundles) def refuse_takeover(self): """Raise if path can never become, or be repaired as, this kind's store. That is anything but a real directory, a directory holding files without this kind's marker, or a store whose bundles/ isn't a real directory. force=True never changes this, so installs check it before suggesting force. """ if not path_present(self.path): return if not stat.S_ISDIR(self.path.lstat().st_mode): raise FileValidationError(self.path, 'not a directory, and never taken over') if not holds_files(self.path): return try: marker = read_json(self.path / MARKER) except (FileNotFoundError, ValueError, RecursionError): marker = None if marker != {'format': FORMAT, 'kind': self.kind}: raise FileValidationError( self.path, 'not a datacache %s store; a directory with files in it is never taken over' % self.kind) if path_present(self.bundles) and not stat.S_ISDIR(self.bundles.lstat().st_mode): raise FileValidationError(self.bundles, 'not a directory, and never taken over') def publish(self, staged): """Rename a complete, checked directory into bundles/ as the newest. Returns the new bundle's directory. It is named for the current UTC time, or one second after the newest bundle if the clock is behind, so the newest name is always the current bundle. """ name = datetime.now(timezone.utc).strftime(BUNDLE_NAME_FORMAT) names = self.bundle_names() if names and name <= names[-1]: newest = datetime.strptime(names[-1], BUNDLE_NAME_FORMAT) try: name = (newest + timedelta(seconds=1)).strftime(BUNDLE_NAME_FORMAT) except OverflowError: raise FileValidationError( self.bundles / names[-1], 'this bundle is dated so late that no newer name exists') from None bundle = self.bundles / name os.replace(staged, bundle) return bundle def store_at(destination): """The store at destination, of whichever kind its marker records. None when nothing is installed there yet: no directory, or an empty one. Raises FileValidationError for a directory that isn't a datacache store. """ path = Path(destination) if not path_present(path): return None require_directory(path) if not path_present(path / MARKER): if holds_files(path): raise FileValidationError(path, 'not a datacache store') return None marker = read_json(path / MARKER) kind = marker.get('kind') if isinstance(marker, dict) else None if kind not in KINDS or marker != {'format': FORMAT, 'kind': kind}: raise FileValidationError(path, 'not a datacache store') return BundleStore(path, kind) def list_bundles(destination): """Every bundle in a store, oldest first; the last is the current one. Works for bundle, archive and materialization stores alike, and only reads. A store with nothing installed has no bundles. """ store = store_at(destination) if store is None or not path_present(store.bundles): return [] require_directory(store.bundles) return [store.bundles / name for name in store.bundle_names()] def prune_bundles(destination, *, keep=1): """Delete all but the newest keep bundles from a store; return the deleted paths. The newest bundle is the current one, so it is always kept: keep must be at least 1. Paths into deleted bundles stop working, so prune only when no running program still uses them. The store's lock is held throughout, so pruning never races an install. Works for bundle, archive and materialization stores alike. """ if isinstance(keep, bool) or not isinstance(keep, int) or keep < 1: raise ValueError('keep must be a positive integer; the current bundle is always kept') store = store_at(destination) if store is None: return [] with store.lock(): deleted = list_bundles(store.path)[:-keep] for bundle in deleted: require_directory(bundle) # Never follow a link out of the store. shutil.rmtree(bundle) return deleted def holds_files(directory): """Whether directory holds anything besides DataCache's temporary files. Writing a file such as the store marker briefly leaves a temporary .datacache-json-* or .datacache-mode-* file beside it, and an interrupted write can leave one behind. Neither makes a directory someone else's. """ return any(not entry.name.startswith(('.datacache-json-', '.datacache-mode-')) for entry in Path(directory).iterdir()) def user_key(): """Identifies the current user in the names of private working directories. The numeric user id on POSIX, the login name elsewhere, so two users of a shared store never use each other's unfinished files. """ if hasattr(os, 'getuid'): return str(os.getuid()) import getpass return getpass.getuser() def private_directory(path): """Create path if needed as a directory only its owner can use, and check it. Installs keep unfinished files here until they publish them. Windows has no such modes, so there it only checks that path is a directory. """ path = Path(path) if not path_present(path): path.mkdir(mode=0o700) require_directory(path) if os.name == 'posix': info = path.lstat() if info.st_uid != os.getuid() or stat.S_IMODE(info.st_mode) & 0o077: raise FileValidationError(path, 'working directory must be private and owner-owned') return path def discard_private_directory(path): """Remove a private working directory, logging rather than raising on failure. Cleanup never hides an install's real outcome: its error, or the bundle it published. """ try: if path_present(path): private_directory(path) # Only ever remove a directory of our own. shutil.rmtree(path) except (OSError, ValueError) as error: logger.warning('Could not remove the working directory %s: %s', path, error) def is_bundle_name(name): """Whether name is a bundle's UTC install time, exactly as publish writes it.""" try: return datetime.strptime(name, BUNDLE_NAME_FORMAT).strftime(BUNDLE_NAME_FORMAT) == name except ValueError: return False def source_fingerprint(source): """SHA-256 identifying a source URL or path without storing its text.""" return hashlib.sha256(source.encode('utf-8')).hexdigest() @dataclass(frozen=True) class HashedFile: """What was read from one regular file. sha256 is None when the contents weren't hashed. info is the os.stat_result of the open file that was read, so it describes exactly those bytes. """ path: str sha256: Optional[str] info: os.stat_result @property def size(self): return self.info.st_size @property def record(self): """The {'sha256', 'size'} entry a manifest stores for these bytes.""" return {'sha256': self.sha256, 'size': self.size} def inspection(self, *, verified): """This file as an available FileInspection.""" return FileInspection(self.path, 'available', verified=verified, size=self.info.st_size, mtime=self.info.st_mtime) def hash_file(path, expected=None, *, hash_contents=True, show_progress=False): """Hash one regular file, optionally checking it against expected values. The file is opened without following links; a link, a special file or a file with several hard links raises FileValidationError, as does a file that changes while it is read. hash_contents=False skips reading the bytes: sha256 is None and only the size is checked. expected is a mapping with sha256 and size; None values aren't checked. """ # O_BINARY: Windows would otherwise translate line endings as it reads. with os.fdopen(open_regular(path, os.O_RDONLY | getattr(os, 'O_BINARY', 0)), 'rb') as handle: info = os.fstat(handle.fileno()) sha256 = None if hash_contents: digest, size = hashlib.sha256(), 0 with Progress(show_progress, 'Verifying', info.st_size) as progress: for chunk in iter(lambda: handle.read(CHUNK_SIZE), b''): digest.update(chunk) size += len(chunk) progress(size, info.st_size) if size != info.st_size or os.fstat(handle.fileno()).st_mtime_ns != info.st_mtime_ns: raise FileValidationError(path, 'file changed while it was being read') sha256 = digest.hexdigest() hashed = HashedFile(os.fspath(path), sha256, info) if expected is not None: check_expected(path, hashed.record, expected) return hashed def check_expected(path, record, expected): """Raise FileValidationError if record's bytes disagree with expected. Both are mappings with sha256 and size. A None value on either side isn't compared, and other keys in expected are ignored. """ if expected.get('size') is not None and record['size'] != expected['size']: raise FileValidationError(path, 'size is %d bytes, expected %d' % (record['size'], expected['size'])) if (expected.get('sha256') is not None and record['sha256'] is not None and record['sha256'] != expected['sha256']): raise FileValidationError(path, 'SHA-256 disagrees with the expected digest') def validate_file_record(record): """Return record if it is a manifest entry for one file, else raise. A file record is exactly {'sha256': <64 hex digits>, 'size': }; neither may be missing, because a manifest records what was seen. """ if not isinstance(record, dict) or set(record) != {'sha256', 'size'}: raise ValueError('invalid file record: %r' % (record,)) _validate_expectations(record['sha256'], record['size']) if record['sha256'] is None or record['size'] is None: raise ValueError('a file record needs both sha256 and size') return record @dataclass(frozen=True) class TreeListing: """Every file and directory below a root, by relative POSIX name. files maps each regular file's name to its path; directories lists every subdirectory, parents included. Both are sorted by name. """ files: dict directories: list def list_tree(root, *, ignore=()): """List a directory tree without following links or reading any file. Names in ignore are skipped at the root, such as a bundle's manifest. Raises FileValidationError for a link or special file, and ValueError for a name that validate_relative_name rejects. """ files, directories, pending = {}, [], [(Path(root), '')] while pending: directory, prefix = pending.pop() require_directory(directory) with os.scandir(directory) as entries: entries = list(entries) for entry in entries: name = prefix + entry.name if name in ignore: continue validate_relative_name(name) mode = entry.stat(follow_symlinks=False).st_mode if stat.S_ISDIR(mode): directories.append(name) pending.append((Path(entry.path), name + '/')) elif stat.S_ISREG(mode): files[name] = Path(entry.path) else: raise FileValidationError(entry.path, 'expected only regular files and directories, not a link or special file') return TreeListing(dict(sorted(files.items())), sorted(directories)) def check_tree(root, files, directories, *, ignore=(), hash_contents=True, show_progress=False): """Check that root holds exactly the recorded files and directories. files maps relative names to records with sha256 and size (None values aren't checked); directories names every directory the tree should hold. The whole tree is listed before any file is read, so an unexpected file, directory, link or special file is rejected without hashing anything. Then every file is hashed (see hash_file). Returns a dict of names to HashedFile. """ listing = list_tree(root, ignore=ignore) for what, recorded, found in (('files', set(files), set(listing.files)), ('directories', set(directories), set(listing.directories))): if recorded != found: raise FileValidationError(root, '%s differ from the record: missing %s, unexpected %s' % ( what, sorted(recorded - found), sorted(found - recorded))) return {name: hash_file(path, files[name], hash_contents=hash_contents, show_progress=show_progress) for name, path in listing.files.items()} def parent_directories(names): """Every directory that relative names imply: 'a/b/c' implies 'a' and 'a/b'.""" return {'/'.join(parts[:index]) for parts in (name.split('/') for name in names) for index in range(1, len(parts))} def validate_relative_name(name): """Return name if it is a safe relative POSIX path inside a bundle. Rejects empty names and components, backslashes, '.' and '..', names Windows can't store (reserved device names, control and punctuation characters, trailing dots or spaces) and DataCache's own '.datacache-*' names. Raises ValueError otherwise. """ if not isinstance(name, str) or not name or '\\' in name: raise ValueError('names must be nonempty relative POSIX paths') for part in name.split('/'): if (part in ('', '.', '..') or part.endswith((' ', '.')) or re.search(r'[\x00-\x1f<>:"|?*]', part) or part.lower().startswith('.datacache-') or re.fullmatch(r'(?i:con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\..*)?', part)): raise ValueError('unsafe relative path: %r' % name) return name def validate_path_component(name): """Return name if it is a safe relative name without '/', such as a version.""" validate_relative_name(name) if '/' in name: raise ValueError('names and versions must be single path components') return name def validate_distinct_paths(names): """Raise ValueError if names can't all exist together in one directory tree. That is when two paths, or any of their parent directories, differ only by letter case, which case-insensitive filesystems store as one; or when one name is a directory of another. """ seen = {} # Case-folded path: (path as written, whether it is a file). for name in names: parts = name.split('/') for depth in range(1, len(parts) + 1): path, is_file = '/'.join(parts[:depth]), depth == len(parts) key = path.casefold() # Only the same directory, spelled the same way, may appear twice. if key in seen and (is_file or seen[key] != (path, False)): raise ValueError('paths collide as files/directories or ignoring case') seen[key] = (path, is_file) def validate_no_sidecar_collisions(names): """Raise ValueError if a name is at another name's provenance record path. fetch_file(..., record_provenance=True) writes a hidden record beside each download, so those paths must stay free. """ occupied = {name.casefold() for name in names} | {name.casefold() for name in parent_directories(names)} if {Path(sidecar_path(name)).as_posix().casefold() for name in names} & occupied: raise ValueError('paths collide with automatic provenance sidecars') def local_file_identity(path): """The file:// URL identifying a local source, such as file:///data/a.fa. Relative paths are made absolute first, so the identity doesn't depend on the working directory of whichever process records it. """ return Path(path).absolute().as_uri() def require_directory(path): """Raise FileValidationError unless path is a directory, not a link to one.""" if not stat.S_ISDIR(Path(path).lstat().st_mode): raise FileValidationError(path, 'expected a directory, not a link') ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7051806 datacache-1.27.1/datacache/bundles.py0000644000076500000000000005515215264220442017131 0ustar00iskanderwheel"""Versioned datasets installed as bundles of separately downloaded files.""" from dataclasses import dataclass, field from datetime import datetime, timezone import hashlib import json import os from pathlib import Path import shutil import unicodedata from uuid import uuid4 from ._filesystem import path_present, read_json, write_json from .bundle_store import ( INVALID_STORE_ERRORS, MANIFEST, BundleStore, discard_private_directory, hash_file, local_file_identity, private_directory, prune_bundles, require_directory, source_fingerprint, user_key, validate_distinct_paths, validate_no_sidecar_collisions, validate_path_component, validate_relative_name, ) from .download import normal_creation_mode, validate_download_options, validate_size_within_limit from .integrity import FileValidationError, _validate_expectations from .provenance import redact_url FORMAT = 1 def _assets(assets, verified=True): if not isinstance(assets, dict) or not assets: raise ValueError('assets must be a nonempty mapping of relative names to metadata') result = {} for name, spec in assets.items(): validate_relative_name(name) if isinstance(spec, str): spec = {'url': spec} if not isinstance(spec, dict): raise ValueError('asset metadata must be a mapping') unknown = set(spec) - {'url', 'path', 'sha256', 'size', 'decompress'} if unknown: raise ValueError('unknown asset options: %s' % sorted(unknown)) if ('url' in spec) == ('path' in spec): raise ValueError('each asset needs exactly one of url or path') if 'path' in spec and (not isinstance(spec['path'], (str, os.PathLike)) or not os.fspath(spec['path'])): raise ValueError('asset path must be a nonempty path') # A local file is identified by its file:// URL, like any other source. url = spec['url'] if 'url' in spec else local_file_identity(spec['path']) if not isinstance(url, str) or not url: raise ValueError('each asset requires a nonempty URL') digest, size = spec.get('sha256'), spec.get('size') _validate_expectations(digest, size) if verified and (digest is None or size is None): raise ValueError('verified bundles require sha256 and size for every asset') decompress = spec.get('decompress', False) if not isinstance(decompress, bool): raise ValueError('decompress must be a boolean') result[name] = dict(url=url, sha256=digest.lower() if digest else None, size=size, decompress=decompress) validate_distinct_paths(result) validate_no_sidecar_collisions(result) return result def _receipt_assets(receipt): if not isinstance(receipt, dict) or receipt.get('format') != FORMAT: raise ValueError('unrecognized bundle manifest') assets = receipt.get('assets') # Every local receipt pins the bytes, even in unverified acquisition mode. return _assets(assets, verified=True) @dataclass(frozen=True) class BundleInspection: """Read-only status of a store's current bundle; paths belong to that bundle. status is available, missing, invalid or inaccessible. bundle is the current bundle's directory. verified means every file matched caller-supplied trusted hashes now; receipt-only full inspection checks recorded hashes without asserting trust. Metadata-only inspection checks readability and recorded sizes, not hashes. """ path: str status: str verified: bool = False bundle: object = None files: dict = field(default_factory=dict) error: object = None def _check_bundle(store, directory, assets, *, verify_files=True): # Manifests grow with the number of assets; the writer has no size cap. receipt = read_json(directory / MANIFEST, limit=None) recorded = _receipt_assets(receipt) if assets is not None: if set(recorded) != set(assets): raise FileValidationError(directory, 'manifest asset names disagree with registry') for name, spec in assets.items(): if spec['sha256'] is None: fingerprints = receipt.get('source_fingerprints') if (not isinstance(fingerprints, dict) or fingerprints.get(name) != source_fingerprint(spec['url'])): raise FileValidationError(directory, 'manifest source identity is missing or disagrees with registry') if spec['decompress'] != recorded[name]['decompress']: raise FileValidationError(directory, 'manifest decompression setting disagrees with registry') for key in ('sha256', 'size'): if spec[key] is not None and recorded[name][key] != spec[key]: raise FileValidationError(directory, 'manifest expectations disagree with registry') files = {} for name, spec in recorded.items(): target = directory / name for parent in target.relative_to(directory).parents: require_directory(directory / parent) hashed = hash_file(target, spec, hash_contents=verify_files) # Only a hash the caller supplied verifies a file; the manifest's own doesn't. supplied = assets is not None and assets[name]['sha256'] is not None files[name] = hashed.inspection(verified=verify_files and supplied) trusted = verify_files and assets is not None and all(spec['sha256'] for spec in assets.values()) return BundleInspection(str(store.path), 'available', bool(trusted), str(directory), files) def inspect_bundle(destination, assets=None, *, verify_files=True): """Check the current bundle offline, without writes, locks or repair. Optional assets is the caller's trusted mapping (url, sha256, size). Without it, check consistency with the bundle's own manifest, verified=False. verify_files=False checks the receipt, source expectations, required file types, readability and sizes without reading payloads; verified stays False. """ if not isinstance(verify_files, bool): raise ValueError('verify_files must be a boolean') path = Path(destination) store = BundleStore(path, 'bundle') expected = _assets(assets, verified=False) if assets is not None else None try: bundle = store.current_bundle() if bundle is None: return BundleInspection(str(path), 'missing') return _check_bundle(store, bundle, expected, verify_files=verify_files) except PermissionError as error: return BundleInspection(str(path), 'inaccessible', error=error) except INVALID_STORE_ERRORS as error: return BundleInspection(str(path), 'invalid', error=error) def _local_sources(source_paths, assets): """source_paths as absolute Paths, checked against the assets they replace.""" if source_paths is None: return {} if not isinstance(source_paths, dict): raise ValueError('source_paths must map asset names to local files') unknown = set(source_paths) - set(assets) if unknown: raise ValueError('source_paths names unknown assets: %s' % sorted(unknown)) result = {} for name, path in source_paths.items(): if not isinstance(path, (str, os.PathLike)) or not os.fspath(path): raise ValueError('source_paths[%r] must be a path' % name) path = Path(path).absolute() if assets[name]['decompress'] and path.suffix.lower() not in ('.gz', '.zip'): raise ValueError('source_paths[%r] must end in .gz or .zip to be decompressed' % name) result[name] = path return result def _paths(inspection): return {name: value.path for name, value in inspection.files.items()} def install_bundle(destination, assets, *, force=False, verified=True, verify_files=True, download_options=None, source_paths=None): """Download every asset, then publish them together as a new bundle. assets maps relative names to {url, sha256, size, decompress?}; path in place of url names a local file. Trusted sha256 and size are mandatory unless verified=False is explicit. source_paths maps asset names to local files to read instead of downloading, such as files already downloaded by hand; each asset keeps its declared URL as its identity, so later installs treat the bundle exactly as if it had been downloaded. On a valid cache hit this is read-only, including on a read-only filesystem. An invalid current bundle requires force=True, which installs a new one. Returns a dict of asset names to paths in the current bundle. Old bundles are kept, so these paths survive later force installs. Generated outputs belong outside this store. Installation works on local filesystems with atomic sibling renames, on Linux, macOS and Windows. verify_files controls checks of an existing bundle only; new downloads are always checked. """ from .download import fetch_file if not all(isinstance(value, bool) for value in (verified, force, verify_files)): raise ValueError('verified, force and verify_files must be booleans') expected = _assets(assets, verified) options = validate_download_options(download_options, 'bundle') local_sources = _local_sources(source_paths, expected) for name, spec in expected.items(): validate_size_within_limit(spec['size'], options.get('max_bytes'), '%s size' % name) if options.get('resume') is True: # "auto" downloads normally where it can't resume. # Check before creating anything: a resumable install keeps its staging. from .resume import validate_resume for name, spec in expected.items(): if name in local_sources: continue # Copied, not downloaded. validate_resume(spec['url'], spec['sha256'], spec['size']) if spec['decompress']: raise ValueError('resume=True supports raw downloads only; %s has decompress=True' % name) path = Path(destination) store = BundleStore(path, 'bundle') # A forced install replaces the bundle whatever it holds: don't hash it. inspection = inspect_bundle(path, expected, verify_files=verify_files and not force) if store.can_reuse(inspection, force=force): return _paths(inspection) path.parent.mkdir(parents=True, exist_ok=True) with store.lock(): store.create() inspection = inspect_bundle(path, expected, verify_files=verify_files and not force) if store.can_reuse(inspection, force=force): return _paths(inspection) # A resumable bundle keeps its private working directory across calls, # including completed assets. The registry identity selects it, and # different users never inherit each other's private partials. With # "auto", only when every asset is pinned, so a kept file can never be # stale: an unpinned upstream file may change between attempts. resumable = options.get('resume') is True or ( options.get('resume') == 'auto' and all(spec['sha256'] is not None and spec['size'] is not None for spec in expected.values())) staging_key = (hashlib.sha256(json.dumps(expected, sort_keys=True).encode()).hexdigest() + '-' + user_key()) if resumable else uuid4().hex working = private_directory(path / ('.staging-' + staging_key)) # The private parent protects unfinished bytes. The inner directory # already has its final sharing mode, so the bundle is readable by # others the moment it is renamed into place. staged = working / 'files' staged.mkdir(exist_ok=resumable) require_directory(staged) published = False try: recorded = {} for name, spec in expected.items(): target = staged / name fetch_options = dict(options, destination=target, decompress=spec['decompress'], expected_sha256=spec['sha256'], expected_size=spec['size']) source = spec['url'] if name in local_sources: # Read the local copy; the manifest keeps the declared URL. source = local_sources[name].as_uri() fetch_options.pop('resume', None) fetch_options['raw'] = not spec['decompress'] try: fetch_file(source, **fetch_options) except FileValidationError: if not resumable or not target.is_file(): raise # This is private, unpublished working state, not a user's # installed bundle. Explicit installation may repair it. fetch_file(source, force=True, **fetch_options) # fetch_file already checked a trusted hash; hash only the rest. hashed = hash_file(target, hash_contents=spec['sha256'] is None) recorded[name] = dict(url=redact_url(spec['url']), sha256=hashed.sha256 or spec['sha256'], size=hashed.size, decompress=spec['decompress']) receipt = dict(format=FORMAT, fetched_at=datetime.now(timezone.utc).isoformat(), assets=recorded, source_fingerprints={name: source_fingerprint(spec['url']) for name, spec in expected.items()}) # Remove DataCache's own temporary files: resume state and anything # an interrupted earlier attempt left. No asset is named .datacache-*. for leftover in sorted(staged.rglob('.datacache-*'), reverse=True): if leftover.is_dir() and not leftover.is_symlink(): shutil.rmtree(leftover) else: leftover.unlink(missing_ok=True) write_json(staged / MANIFEST, receipt, mode=normal_creation_mode(staged)) # Check the complete bundle, hashes included, before publishing it. _check_bundle(store, staged, expected) bundle = store.publish(staged) published = True return {name: str(bundle / name) for name in expected} finally: if published or not resumable: discard_private_directory(working) class VersionedDatasetRegistry: """A plain mapping of dataset names, pinned defaults, versions and assets. Each dataset is {default_version, versions: {version: {asset: metadata}}}. The single-file shape {filename, urls: {version: url}, default_version} is also accepted with verified=False; its file is installed with decompress=True. Give the root as cache_root, a path, or as cache_dir, a zero-argument callable consulted on every lookup. Alternatively, store_path is a two-argument (name, version) callback selecting an exact managed store. Construction never writes or invokes either callback. """ def __init__(self, datasets, *, cache_root=None, cache_dir=None, store_path=None, verified=True): if sum(value is not None for value in (cache_root, cache_dir, store_path)) != 1: raise ValueError('provide exactly one of cache_root, cache_dir or store_path') if not isinstance(verified, bool): raise ValueError('verified must be a boolean') if cache_dir is not None and not callable(cache_dir): raise ValueError('cache_dir must be callable') if store_path is not None and not callable(store_path): raise ValueError('store_path must be callable') if store_path is None: # A root callable is consulted on every lookup, so it can change. root = cache_dir if cache_dir is not None else lambda: cache_root self._store_path = lambda name, version: Path(root()) / name / version else: self._store_path = store_path # Custom store paths, resolved and checked once on first use. self._custom_stores = None if store_path is None else {} self.verified = verified self._datasets = {} for name, spec in datasets.items(): validate_path_component(name) versions = spec.get('versions') if versions is None: versions = {version: {spec['filename']: {'url': url, 'decompress': True}} for version, url in spec['urls'].items()} normalized = {validate_path_component(version): _assets(assets, verified) for version, assets in versions.items()} # Versions are directories: case-insensitive filesystems would give # two that differ only by case one store. validate_distinct_paths(list(normalized)) default = spec['default_version'] if default not in normalized: raise ValueError('default_version must name a pinned version') self._datasets[name] = dict(default_version=default, versions=normalized, description=spec.get('description', '')) validate_distinct_paths(list(self._datasets)) def resolve_version(self, name, version=None): if name not in self._datasets: raise ValueError('unknown dataset %r; known: %s' % (name, ', '.join(sorted(self._datasets)))) spec = self._datasets[name] version = spec['default_version'] if version is None else version if version not in spec['versions']: raise ValueError('unknown version %r for %s' % (version, name)) return version def store_path(self, name, version=None): """Resolve the store path without checking it or creating directories.""" version = self.resolve_version(name, version) if self._custom_stores is None: path = self._store_path(name, version) # / is DataCache's own directory: never follow a link there. if path_present(path.parent): require_directory(path.parent) return path # A custom store's parent belongs to the application and may be a link # (e.g. to another disk); the store itself is still never one. if not self._custom_stores: self._custom_stores = self._resolve_custom_stores() return self._custom_stores[(name, version)] def _resolve_custom_stores(self): """Every (name, version)'s store_path result: one store per version.""" stores, owners = {}, {} for name in sorted(self._datasets): for version in sorted(self._datasets[name]['versions']): path = self._store_path(name, version) if not isinstance(path, (str, os.PathLike)) or not os.fspath(path): raise ValueError('store_path(%r, %r) returned %r, not a path' % (name, version, path)) shown = os.path.normpath(os.path.abspath(path)) # Case-folded: case-insensitive filesystems store these as one. key = unicodedata.normalize('NFC', shown.casefold()) if key in owners: raise ValueError('store_path gives %s for both %s %s and %s %s; ' 'each version needs its own store' % ((shown,) + owners[key] + (name, version))) owners[key] = (name, version) stores[(name, version)] = Path(path) return stores def inspect(self, name, version=None, *, verify_files=True): version = self.resolve_version(name, version) return inspect_bundle(self.store_path(name, version), self._datasets[name]['versions'][version], verify_files=verify_files) def download(self, name, version=None, *, force=False, verify_files=True, source_paths=None, **download_options): """Install if needed (or always, with force=True); return asset paths in the current bundle. source_paths maps asset names to local files to read instead of downloading; the declared URLs remain the assets' identity. """ version = self.resolve_version(name, version) return install_bundle(self.store_path(name, version), self._datasets[name]['versions'][version], force=force, verified=self.verified, verify_files=verify_files, download_options=download_options, source_paths=source_paths) def prune(self, name, version=None, *, keep=1): """Delete all but the newest keep bundles of one version; see prune_bundles.""" return prune_bundles(self.store_path(name, version), keep=keep) def local_path(self, name, version=None, *, asset=None, verify_files=True): """Resolve the current bundle; no writes or network. Missing raises. For one asset, return its Path; for multiple assets return the bundle directory, or select an individual asset with asset=. Assets are hashed by default; verify_files=False checks metadata and sizes only, which cannot detect same-size corruption. """ inspected = self.inspect(name, version, verify_files=verify_files) if inspected.status == 'missing': raise FileNotFoundError(inspected.path) if inspected.status != 'available': raise FileValidationError(inspected.path, inspected.status) from inspected.error if asset is not None: return Path(inspected.files[asset].path) if len(inspected.files) == 1: return Path(next(iter(inspected.files.values())).path) return Path(inspected.bundle) def ensure(self, name, version=None, **download_options): """Download/reuse, then return what local_path would: the single asset's Path, or the bundle directory of several assets.""" paths = self.download(name, version, **download_options) if len(paths) == 1: return Path(next(iter(paths.values()))) # The paths download validated: no second inspection, so a concurrent # refresh cannot swap the bundle between the two. asset_name, asset_path = next(iter(paths.items())) return Path(asset_path).parents[len(Path(asset_name).parts) - 1] def is_cached(self, name, version=None, *, verify_files=True): """Whether inspection reports available; verify_files=False skips hashing.""" return self.inspect(name, version, verify_files=verify_files).status == 'available' def status(self, *, verify_files=True): """One read-only status row per pinned default; verify_files=False skips hashing.""" return [dict(name=name, version=self.resolve_version(name), description=self._datasets[name]['description'], available_versions=sorted(self._datasets[name]['versions']), inspection=self.inspect(name, verify_files=verify_files)) for name in sorted(self._datasets)] ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7052927 datacache-1.27.1/datacache/cache.py0000644000076500000000000002300715264220442016532 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from os.path import isabs, join, realpath from os import fspath, getcwd, remove, scandir, stat from shutil import rmtree from . import common from . import download from . import provenance from .download import expected_path from .database_helpers import db_from_dataframe from .inspection import inspect_file, path_exists from .permissions import make_file_readable class Cache: """Download, inspect, and delete files under one cache directory. subdir names the application whose platform cache directory is used; cache_root selects an exact directory instead. A relative cache_root is resolved against the working directory on every call, not at construction. """ def __init__(self, subdir="datacache", *, cache_root=None): """Select a cache directory without creating it. cache_root overrides the platform cache location selected by subdir. """ if not subdir: raise ValueError("Cache subdir must be a non-empty string") self.subdir = subdir self.cache_directory_path = ( common.get_data_dir(subdir) if cache_root is None else fspath(cache_root)) # Track explicit filenames as well as inferred names for deletion. # TODO: handle decompression separately from download, # so we can use copies of compressed files we've already downloaded self._local_paths = {} def delete_url(self, url): """Delete this root's local copies of a URL; missing files are ignored. Removes files this instance fetched from url, including under explicit filenames, and the URL-derived raw and decompressed paths under this root whichever instance created them, with their provenance records. Explicit filenames used by other instances are not recorded and are left in place. """ keys = [key for key in self._local_paths if key[0] == url] paths = {self._local_paths[key] for key in keys} # Include inferred paths created by another Cache instance or fetch_file. paths.update(self.local_path(url, decompress=value) for value in (False, True)) for path in paths: try: remove(path) except FileNotFoundError: pass provenance.remove(path) for key in keys: del self._local_paths[key] def delete_all(self): """Clear cached contents while preserving the root and its permissions.""" # Validate before resolving: missing/.. must not select an existing # parent. Resolve once so deleting a child in a root like child/.. # cannot invalidate the paths used for the remaining entries. stat(self.cache_directory_path) directory = realpath(self.cache_directory_path) with scandir(directory) as entries: for entry in entries: if entry.is_dir(follow_symlinks=False): rmtree(entry.path) else: remove(entry.path) self._local_paths.clear() def exists(self, url=None, filename=None, decompress=False): """ Check presence without writes or network. Permission errors propagate. """ return path_exists(self.local_path(url, filename, decompress)) def inspect(self, url=None, filename=None, decompress=False, *, expected_sha256=None, expected_size=None): """Report availability/integrity without writes, network, or repair.""" return inspect_file( self.local_path(url, filename, decompress), expected_sha256=expected_sha256, expected_size=expected_size) def make_readable(self, url=None, filename=None, decompress=False, *, group=True, others=False): """Explicitly add read access to one cached regular file on POSIX. Defaults to group read access; does not download, replace, or recurse. The file owner must invoke this to share an older private cache file. """ return make_file_readable( self.local_path(url, filename, decompress), group=group, others=others) def fetch( self, url, filename=None, decompress=False, force=False, timeout=None, use_wget_if_available=None, *, chunk_size=download.DEFAULT_CHUNK_SIZE, progress_callback=None, expected_sha256=None, expected_size=None, max_bytes=None, max_retries=download.DEFAULT_MAX_RETRIES, retry_backoff=download.DEFAULT_RETRY_BACKOFF, retry_max_delay=download.DEFAULT_RETRY_MAX_DELAY, show_progress=False, record_provenance=False, allow_empty=False, resume=False, raw=False, expire_after=None, return_stale_on_error=False, validator=None): """ Return the local path to the downloaded copy of a given URL. Don't download the file again if it's already present, unless `force` is True. Retry options have the same meanings as in fetch_file; max_retries=0 disables automatic retries of transient HTTP failures. show_progress=True enables optional tqdm displays; callbacks remain supported independently. Existing cache hits are quiet. record_provenance=True records where the file came from, for inspect_file, as described for fetch_file. allow_empty=True accepts an empty file, which is otherwise rejected. max_bytes bounds every file the fetch writes, as described for fetch_file. raw=True disables decompression and HTML conversion even for an explicit filename. It cannot be combined with decompress=True. Resume options have the same size, hash, and strong-ETag requirements as fetch_file. expire_after, return_stale_on_error and validator refresh expired copies, fall back to a cached copy when a refresh fails, and reject wrong content, as described for fetch_file. `use_wget_if_available` is deprecated and ignored (datacache always uses its streaming Python downloader now); passing it emits a warning. """ key = (url, filename, decompress) path = download.fetch_file( url, filename=filename, decompress=decompress, subdir=self.subdir, cache_root=self.cache_directory_path, force=force, timeout=timeout, use_wget_if_available=use_wget_if_available, chunk_size=chunk_size, progress_callback=progress_callback, expected_sha256=expected_sha256, expected_size=expected_size, max_bytes=max_bytes, max_retries=max_retries, retry_backoff=retry_backoff, retry_max_delay=retry_max_delay, show_progress=show_progress, record_provenance=record_provenance, allow_empty=allow_empty, resume=resume, raw=raw, expire_after=expire_after, return_stale_on_error=return_stale_on_error, validator=validator) self._local_paths[key] = path return path def local_filename( self, url=None, filename=None, decompress=False): """ What local filename will we use within the cache directory for the given URL/filename/decompress options. """ return common.build_local_filename(url, filename, decompress) def local_path(self, url=None, filename=None, decompress=False, download=False): """ What will the full local path be if we download the given file? """ if download: return self.fetch(url=url, filename=filename, decompress=decompress) else: return expected_path( url, filename, decompress, cache_root=self.cache_directory_path) def db_from_dataframe( self, db_filename, table_name, df, key_column_name=None, *, overwrite=False, version=1, show_progress=False): """Build or reuse a database in this cache and return its connection. The caller must close the connection. Change version or set overwrite to rebuild; failed rebuilds preserve the previous database. """ db_path = join(self.cache_directory_path, db_filename) if not isabs(db_path): # Prefix the cwd without abspath's lexical removal of symlink/.. # components; the filesystem must determine their destination. db_path = join(getcwd(), db_path) return db_from_dataframe( db_filename=db_path, table_name=table_name, df=df, primary_key=key_column_name, subdir=self.subdir, overwrite=overwrite, version=version, show_progress=show_progress) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7054048 datacache-1.27.1/datacache/common.py0000644000076500000000000002715215264220442016764 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import hashlib import os from os import makedirs, environ from os.path import join, exists, split, splitext import re from urllib.parse import parse_qsl, urlsplit from shutil import rmtree import platformdirs from platformdirs.macos import MacOS from .version import __version__ COMPRESSION_SUFFIXES = (".gz", ".zip") # Sent on every HTTP(S) request. Some data hosts (IEDB among them) reject any # User-Agent containing Requests' default "python-requests" token, so this # names datacache alone rather than extending that default. USER_AGENT = "datacache/%s (+https://github.com/openvax/datacache)" % __version__ # Longest file name ext4 accepts, in UTF-8 bytes. APFS and NTFS instead limit # characters or UTF-16 units, which a UTF-8 byte count never undercounts. MAX_NAME_BYTES = 255 def _source_suffix(download_url): """Prefer a supported path suffix, then a filename in the final query value. This supports IEDB/pepdata download endpoints without interpreting bare format hints such as ?format=.gz or fragments as a file format. """ supported = COMPRESSION_SUFFIXES + (".html", ".htm") parsed = urlsplit(download_url) path_suffix = splitext(parsed.path)[1].lower() if path_suffix in supported: return path_suffix query = parse_qsl(parsed.query, keep_blank_values=True) query_suffix = splitext(query[-1][1])[1].lower() if query else "" return query_suffix if query_suffix in supported else path_suffix def ensure_dir(path): """Create path and its parents unless something already exists there. An existing path is left alone, even if it is not a directory. A directory another process creates at the same moment is fine; a broken symlink raises FileExistsError, since no directory can be created in its place. """ if not exists(path): makedirs(path, exist_ok=True) def get_data_dir(subdir=None, envkey=None): """Return the platform cache directory for an application, without creating it. subdir is the application name, "datacache" when omitted or empty. If the environment variable named by envkey is set and nonempty, its value is the root instead, with subdir appended when supplied. To use a variable's value as the root itself, see get_cache_root. """ if envkey and environ.get(envkey): envdir = environ[envkey] if subdir: return join(envdir, subdir) else: return envdir return _platform_cache_dir(subdir if subdir else "datacache") def _platform_cache_dir(name): """name's directory in the platform's user cache, as platformdirs gives it. Two exceptions keep the location the same whichever platformdirs version another package pulled in: - macOS always uses ~/Library/Caches, as appdirs did before DataCache 1.25.0, so existing caches stay put. platformdirs 4.6 and later honour XDG_CACHE_HOME there instead. - XDG_CACHE_HOME is read as platformdirs 4.11.8 and later read it: padding is stripped, and a relative value falls back to ~/.cache. Earlier versions kept the padding (before 4.6) and used a relative value as given, so the cache followed the working directory. """ if issubclass(platformdirs.PlatformDirs, MacOS): return join(os.path.expanduser("~/Library/Caches"), name) path = platformdirs.user_cache_dir(name) if not os.path.isabs(path): xdg = environ.get("XDG_CACHE_HOME", "").strip() return join(xdg if os.path.isabs(xdg) else os.path.expanduser("~/.cache"), name) return path def get_cache_root(name, *envkeys, override=None, legacy=()): """Return where name's cached data lives on this machine. The first of these that applies, with ~ expanded: 1. override, when not None: an explicit choice, such as a command-line flag; 2. the first environment variable in envkeys set to a non-blank value; 3. the platform's cache directory for name, when it already holds data; 4. the first path in legacy that already holds data, so a cache made before a move keeps working; 5. the platform's cache directory for name. A directory holds data when it contains a file anywhere inside, other than files operating systems or DataCache leave on their own (.DS_Store, Thumbs.db, desktop.ini, .datacache-*). Empty folders don't count, and unreadable entries are skipped. legacy paths must be absolute; a relative one would depend on the working directory. Unlike get_data_dir(subdir, envkey), nothing is appended to an environment value, so every package reading the same variable agrees on one location. Nothing is created. The legacy check reads the disk, so resolve the root once and pass it on. """ if not isinstance(name, str) or not name: raise ValueError("name must be a non-empty string") if not all(isinstance(envkey, str) and envkey for envkey in envkeys): raise ValueError("environment variable names must be non-empty strings") if legacy is None: legacy = () elif isinstance(legacy, (str, os.PathLike)): legacy = (legacy,) legacy = [_path_text(path, "legacy") for path in legacy] if any(not os.path.isabs(path) for path in legacy): raise ValueError("legacy locations must be absolute paths") if override is not None: return _path_text(override, "override") for envkey in envkeys: value = environ.get(envkey, "").strip() if value: return os.path.expanduser(value) platform = _platform_cache_dir(name) if legacy and not _holds_data(platform): for path in legacy: if _holds_data(path): return path return platform def _path_text(value, what): """value as a stripped, ~-expanded text path, or ValueError naming what.""" if not isinstance(value, (str, os.PathLike)) or not isinstance(os.fspath(value), str): raise ValueError("%s must be a text path, not %r" % (what, value)) text = os.fspath(value).strip() if not text: raise ValueError("%s must be a non-empty path" % what) return os.path.expanduser(text) # Files operating systems and DataCache leave in a folder on their own. _INCIDENTAL_FILES = (".DS_Store", "Thumbs.db", "desktop.ini") def _holds_data(directory): """Whether directory has a file anywhere inside, besides incidental ones.""" pending = [directory] while pending: try: with os.scandir(pending.pop()) as entries: for entry in entries: try: if entry.is_dir(follow_symlinks=False): pending.append(entry.path) elif (entry.is_file() and entry.name not in _INCIDENTAL_FILES and not entry.name.startswith(".datacache-")): return True except OSError: continue # An unreadable entry says nothing either way. except OSError: continue return False def resolve_path(filename, subdir=None, *, cache_root=None): """Resolve a cache path without filesystem access or directory creation. cache_root is the directory containing cached files, overriding the platform location selected by subdir. Relative roots remain relative to the cwd. """ data_dir = get_data_dir(subdir) if cache_root is None else os.fspath(cache_root) return join(data_dir, os.fspath(filename)) def build_path(filename, subdir=None, *, cache_root=None): """Resolve a writable cache path, creating its parent when necessary.""" full_path = resolve_path(filename, subdir, cache_root=cache_root) os.makedirs(os.path.dirname(full_path) or ".", exist_ok=True) return full_path def clear_cache(subdir=None): """Recursively delete an application's entire platform cache directory. The directory itself is removed too; subdir=None selects the default "datacache" cache. Use Cache.delete_all to empty an explicit cache_root while keeping it. Raises FileNotFoundError if the directory is absent. """ data_dir = get_data_dir(subdir) rmtree(data_dir) def name_digest(text): """MD5 hex digest used in cache names: a key, not a security measure.""" return hashlib.md5(text.encode("utf-8", "surrogatepass"), usedforsecurity=False).hexdigest() def name_length(name): """Measure a file name in UTF-8 bytes, so a name within limits fits anywhere.""" return len(name.encode("utf-8", "surrogatepass")) def _fit_name(filename): """Shorten a cache name the filesystem could not hold, keeping its end. normalize_filename shortens by characters, so a non-ASCII name can still exceed ext4's 255 bytes and could never be created on Linux. Keep a digest of such a name plus as much of its end, and so its extension, as fits. Every name within the limit is unchanged, so existing cache keys still work. """ if name_length(filename) <= MAX_NAME_BYTES: return filename digest = name_digest(filename) budget = MAX_NAME_BYTES - len(digest) start, used = len(filename), 0 while start > 0 and used + name_length(filename[start - 1]) <= budget: start -= 1 used += name_length(filename[start]) return digest + filename[start:] def normalize_filename(filename): """ Remove special characters and shorten if name is too long """ # if the url pointed to a directory then just replace all the special chars filename = re.sub(r"/|\\|;|:|\?|=", "_", filename) if len(filename) > 150: prefix = name_digest(filename) filename = prefix + filename[-140:] return filename def build_local_filename(download_url=None, filename=None, decompress=False): """ Determine which local filename to use based on the file's source URL, an optional desired filename, and whether a compression suffix needs to be removed """ if not (download_url or filename): raise ValueError("Either filename or URL must be specified") inferred = not filename # if no filename provided, use the original filename on the server if not filename: digest = name_digest(download_url) filename_url = download_url if decompress: parsed = urlsplit(download_url) if splitext(parsed.path)[1].lower() in COMPRESSION_SUFFIXES: # Keep the full URL in the digest, but let the compression # suffix be stripped from the display name. Default archive # keys remain unchanged, including their query/fragment text. filename_url = download_url.split("?", 1)[0].split("#", 1)[0] parts = split(filename_url) filename = digest + "." + "_".join(parts) filename = normalize_filename(filename) if decompress: (base, ext) = splitext(filename) if ext.lower() in COMPRESSION_SUFFIXES: filename = base elif inferred and _source_suffix(download_url) in COMPRESSION_SUFFIXES: # Endpoint filenames may be encoded or followed by a fragment. # Keep raw and decompressed contents under different cache keys. filename += ".decompressed" return _fit_name(filename) ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1792090401.70553 datacache-1.27.1/datacache/database.py0000644000076500000000000003343315264220442017237 0ustar00iskanderwheel# Copyright (c) 2015-2018 Mount Sinai School of Medicine # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import logging import sqlite3 import string from pathlib import Path from itertools import chain, islice from .progress import Progress def require_integer(value, name): """Raise TypeError unless value is an int.""" if not isinstance(value, int): raise TypeError("%s: expected an integer, got %r" % (name, value)) def require_string(value, name): """Raise TypeError unless value is a str.""" if not isinstance(value, str): raise TypeError("%s: expected a string, got %r" % (name, value)) def require_iterable_of(values, types, name="values"): """Raise TypeError unless values is iterable and every item is one of types.""" try: wrong = [value for value in values if not isinstance(value, types)] except TypeError: raise TypeError("%s: expected an iterable, got %r" % (name, values)) from None if wrong: raise TypeError("%s: expected items of type %s, got %r" % (name, types, wrong[0])) logger = logging.getLogger(__name__) METADATA_TABLE_NAME = "_datacache_metadata" _ASCII_LOWERCASE = str.maketrans(string.ascii_uppercase, string.ascii_lowercase) def fold_identifier(name): """Fold an identifier's case the way SQLite compares names: ASCII only. SQLite treats "Records" and "records" as one table but "Éclair" and "éclair" as two, so str.lower() would merge names SQLite keeps apart. Use this to validate names before a database exists; with a connection, let SQLite compare them (see tables_exist). """ return name.translate(_ASCII_LOWERCASE) def tables_exist(connection, table_names): """Are all table_names present? SQLite's NOCASE compares them as it does.""" return all(connection.execute( "SELECT 1 FROM sqlite_master WHERE type='table' AND name = ? COLLATE NOCASE", (name,)).fetchone() for name in table_names) def quote_identifier(identifier): """ Wrap a SQL identifier (table, column, or index name) in double quotes so that names which contain spaces or punctuation, or which collide with SQL keywords, don't need to be sanitized down to a restricted character set. Embedded double quotes are escaped by doubling them, per the SQL standard. See https://github.com/openvax/datacache/issues/17 """ return '"%s"' % str(identifier).replace('"', '""') class Database: """ Wrapper object for sqlite3 database which provides helpers for querying and constructing the datacache metadata table, as well as creating and checking for existence of particular table names. create() commits a complete build, rolling back on failure. close() commits pending work before closing; exception handlers should roll back and close the underlying connection directly. drop_all_tables() commits by default. """ def __init__(self, path, *, must_exist=False, read_only=False): self.path = path # check_same_thread=False allows a cached database connection to be # reused across threads, which is needed by long-running / interactive # consumers (e.g. pyensembl) that may issue queries from different # threads than the one which opened the connection. Applications still # need to coordinate simultaneous use of one connection; independent # workers should have their own. See https://github.com/openvax/datacache/issues/45 if must_exist or read_only: uri = Path(path).absolute().as_uri() + ("?mode=ro" if read_only else "?mode=rw") self.connection = sqlite3.connect(uri, uri=True, check_same_thread=False) else: self.connection = sqlite3.connect(path, check_same_thread=False) def _commit(self): self.connection.commit() def close(self): """Commit changes and close database connection""" self._commit() self.connection.close() def table_names(self): """Returns names of all tables in the database""" query = "SELECT name FROM sqlite_master WHERE type='table'" cursor = self.connection.execute(query) results = cursor.fetchall() return [result_tuple[0] for result_tuple in results] def has_table(self, table_name): """Does a table named `table_name` exist, comparing case as SQLite does?""" return tables_exist(self.connection, [table_name]) def drop_all_tables(self, *, commit=True, include_views=False): """Drop tables, optionally removing views for a complete overwrite. commit=False lets the caller roll back the entire schema replacement. Ordinary version rebuilds retain views for compatibility. """ if include_views: views = self.connection.execute( "SELECT name FROM sqlite_master WHERE type='view'").fetchall() for (view_name,) in views: self.execute_sql("DROP VIEW %s" % quote_identifier(view_name)) for table_name in self.table_names(): if not table_name.startswith("sqlite_"): # Dropping a virtual table also drops its shadow tables. They # may still appear later in this snapshot of sqlite_master. self.execute_sql("DROP TABLE IF EXISTS %s" % quote_identifier(table_name)) if commit: self.connection.commit() def execute_sql(self, sql, commit=False): """Log and then execute a SQL query""" logger.info("Running sqlite query: \"%s\"", sql) self.connection.execute(sql) if commit: self.connection.commit() def has_tables(self, table_names): """Are all of the given table names present in the database?""" return tables_exist(self.connection, table_names) def has_version(self): """Does this database have version information? The absence of version information indicates that this database was either not created by datacache or is incomplete. """ return self.has_table(METADATA_TABLE_NAME) def version(self): """What's the version of this database? Found in metadata attached by datacache when creating this database.""" query = "SELECT %s FROM %s" % ( quote_identifier("version"), quote_identifier(METADATA_TABLE_NAME)) cursor = self.connection.execute(query) version = cursor.fetchone() if not version: return 0 else: return int(version[0]) def _finalize_database(self, version): """ Create metadata table for database with version number. Parameters ---------- version : int Tag created database with user-specified version number """ require_integer(version, "version") create_metadata_sql = \ "CREATE TABLE %s (%s INT)" % ( quote_identifier(METADATA_TABLE_NAME), quote_identifier("version")) self.execute_sql(create_metadata_sql) insert_version_sql = \ "INSERT INTO %s VALUES (%s)" % ( quote_identifier(METADATA_TABLE_NAME), version) self.execute_sql(insert_version_sql) def _create_table(self, table_name, column_types, primary=None, nullable=()): """Creates a sqlite3 table from the given metadata. Parameters ---------- column_types : list of (str, str) pairs First element of each tuple is the column name, second element is the sqlite3 type primary : str, optional Which column is the primary key nullable : iterable, optional Names of columns which have null values """ require_string(table_name, "table name") require_iterable_of(column_types, tuple, name="rows") if primary is not None: require_string(primary, "primary") require_iterable_of(nullable, str, name="nullable") column_decls = [] for column_name, column_type in column_types: decl = "%s %s" % (quote_identifier(column_name), column_type) if column_name == primary: decl += " UNIQUE PRIMARY KEY" if column_name == primary or column_name not in nullable: decl += " NOT NULL" column_decls.append(decl) column_decl_str = ", ".join(column_decls) create_table_sql = \ "CREATE TABLE %s (%s)" % ( quote_identifier(table_name), column_decl_str) self.execute_sql(create_table_sql) def _fill_table(self, table_name, rows, *, show_progress=False, total=None): require_string(table_name, "table_name") if not self.has_table(table_name): raise ValueError( "Table '%s' does not exist in database" % (table_name,)) rows = iter(rows) empty = object() first_row = next(rows, empty) if first_row is empty: return if not isinstance(first_row, tuple): raise TypeError("Rows must be tuples") n_columns = len(first_row) blank_slots = ", ".join("?" for _ in range(n_columns)) logger.info("Inserting rows into table %s", table_name) sql = "INSERT INTO %s VALUES (%s)" % ( quote_identifier(table_name), blank_slots) rows = chain((first_row,), rows) with Progress(show_progress, "Writing " + table_name, total, unit="rows") as progress: completed = 0 while True: batch = list(islice(rows, 1000)) if not batch: break if not all(isinstance(row, tuple) and len(row) == n_columns for row in batch): raise ValueError("Rows must all be tuples with %d values" % n_columns) self.connection.executemany(sql, batch) completed += len(batch) progress(completed, total) def create(self, tables, version, *, show_progress=False): """Do the actual work of creating the database, filling its tables with values, creating indices, and setting the datacache version metadata. Parameters ---------- tables : list List of datacache.DatabaseTable objects version : int """ # Explicit BEGIN includes DDL in the transaction on all supported # Python sqlite3 versions. A caller's pending drops join this transaction. with self.connection: if not self.connection.in_transaction: self.connection.execute("BEGIN") for table in tables: self._create_table( table_name=table.name, column_types=table.column_types, primary=table.primary_key, nullable=table.nullable) self._fill_table(table.name, table.iter_rows(), show_progress=show_progress, total=table.row_count) self._create_indices(table.name, table.indices) self._finalize_database(version) def _create_index(self, table_name, index_columns): """ Creates an index over multiple columns of a given table. Parameters ---------- table_name : str index_columns : iterable of str Which columns should be indexed """ logger.info( "Creating index on %s (%s)", table_name, ", ".join(index_columns)) index_columns = tuple(index_columns) base_name = "%s_index_%s" % ( table_name, "_".join(index_columns)) index_name = base_name suffix = 2 table_indices = {row[1]: row[4] for row in self.connection.execute( "PRAGMA index_list(%s)" % quote_identifier(table_name))} while True: existing = self.connection.execute( "SELECT name FROM sqlite_master WHERE name = ? COLLATE NOCASE " "AND type IN ('table', 'view', 'index')", (index_name,)).fetchone() if existing is None: break existing_name = existing[0] if existing_name in table_indices and not table_indices[existing_name]: columns = tuple(row[2] for row in self.connection.execute( "PRAGMA index_info(%s)" % quote_identifier(existing_name))) if columns == index_columns: return # Keep the legacy name when available, but never silently skip a # different index (even on another table) that shares that name. index_name = "%s_%d" % (base_name, suffix) suffix += 1 self.connection.execute( "CREATE INDEX %s ON %s (%s)" % ( quote_identifier(index_name), quote_identifier(table_name), ", ".join(quote_identifier(c) for c in index_columns))) def _create_indices(self, table_name, indices): """ Create multiple indices (each over multiple columns) on a given table. Parameters ---------- table_name : str indices : iterable of tuples Multiple groups of columns, each of which should be indexed. """ require_string(table_name, "table_name") require_iterable_of(indices, (tuple, list)) for index_column_set in indices: self._create_index(table_name, index_column_set) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7058055 datacache-1.27.1/datacache/database_helpers.py0000644000076500000000000005131315264220442020756 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import os import errno import json import re import sqlite3 import stat from os.path import splitext, lexists import logging from .common import MAX_NAME_BYTES, build_local_filename, name_digest, name_length, resolve_path from .download import ( fetch_csv_dataframe, _open_staging_file, normal_creation_mode, _remove_staging_file, ) from .database import ( METADATA_TABLE_NAME, Database, fold_identifier, require_integer, require_iterable_of, require_string, tables_exist, ) from .database_table import DatabaseTable, validate_column_names from .database_types import db_type from .inspection import path_exists logger = logging.getLogger(__name__) # Rebuilding a database creates "-journal" beside it, so database names # need that much room to spare within the filesystem's name limit. _MAX_DB_NAME_LENGTH = MAX_NAME_BYTES - len("-journal") # Characters some supported platform forbids in a file name, separators included. _UNSAFE_NAME_CHARACTERS = re.compile(r'[\x00-\x1f<>:"/\\|?*\ud800-\udfff]') # Separators this platform's filesystem interprets. _SEPARATORS = re.compile("[%s]" % re.escape(os.sep + (os.altsep or ""))) def connect_if_correct_version(db_path, version, *, read_only=False): """Return a sqlite3 database connection if the version in the database's metadata matches the version argument. Also implicitly checks for whether the data in this database has been completely filled, since we set the version last. Missing files and version mismatches return None, without creating files. The caller owns the returned connection. Use read_only=True to explicitly open a shared installation without write access. """ try: os.stat(db_path) except FileNotFoundError: return None db = Database(db_path, must_exist=True, read_only=read_only) try: if db.has_version() and db.version() == version: return db.connection except BaseException: db.connection.close() raise db.connection.close() return None def _validate_table_names(table_names): """Reject table names that no build could create, compared as SQLite does.""" if not all(isinstance(name, str) and name for name in table_names): raise ValueError("Database table names must be non-empty strings") folded = [fold_identifier(name) for name in table_names] if not folded or len(set(folded)) != len(folded): raise ValueError("Database requires non-empty, distinct table names") if any(name.startswith("sqlite_") or name == METADATA_TABLE_NAME for name in folded): raise ValueError("Database table name is reserved for metadata") def _cached_connection(db_path, table_names, version): """Reuse a matching installation before parsing or validating new data. Table names are checked first, so reuse can never hand back a table that no build could create, such as the version metadata. """ require_integer(version, "version") table_names = list(table_names) if table_names: # an empty request has always reused any matching database _validate_table_names(table_names) connection = connect_if_correct_version(db_path, version) if connection is None: return None try: if tables_exist(connection, table_names): return connection except BaseException: connection.close() raise connection.close() return None def _database_target_path(db_path): """Follow final symlinks so publication creates their target, not a new link. Keep parent components intact: normalizing ``symlink/..`` or ``missing/..`` lexically could select a different directory than the filesystem would. """ original = db_path for depth in range(41): try: info = os.lstat(db_path) except FileNotFoundError: return db_path if not stat.S_ISLNK(info.st_mode): return db_path if depth == 40: break target = os.readlink(db_path) db_path = target if os.path.isabs(target) else os.path.join(os.path.dirname(db_path), target) raise OSError(errno.ELOOP, "Too many symbolic links", original) def _explain_rebuild_failure(db_path, error): """Replace SQLite's vague error when its journal name is too long to create. Rebuilding writes "-journal" beside the database. Staged creation works for a name without room for that suffix, but a rebuild never can. Name limits differ by filesystem (bytes on ext4, characters on APFS), so ask this one rather than predict it. """ try: os.stat(db_path + "-journal") except OSError as stat_error: # Windows reports ERROR_FILENAME_EXCED_RANGE rather than ENAMETOOLONG. if stat_error.errno == errno.ENAMETOOLONG or getattr(stat_error, "winerror", None) == 206: name = os.path.basename(db_path) raise ValueError( "Cannot rebuild database %r: this filesystem has no room for SQLite's " "%r file beside it. Use a shorter filename." % (name, name + "-journal")) from error def _create_cached_db( db_path, tables, version=1, *, overwrite=False, show_progress=False): """ Either create or retrieve sqlite database. Parameters -------- db_path : str Path to sqlite3 database file tables : iterable datacache.DatabaseTable objects version : int, optional Version acceptable as cached data. Returns sqlite3 connection """ db_path = os.fspath(db_path) require_string(db_path, "db_path") tables = list(tables) require_iterable_of(tables, DatabaseTable) require_integer(version, "version") table_names = [table.name for table in tables] _validate_table_names(table_names) db_path = _database_target_path(db_path) if not lexists(db_path): # Readers never see an empty/partial new database. Hard-link publication # is atomic and does not clobber a concurrent creator's database. directory = os.path.dirname(db_path) or "." staged_path = None db = None try: with _open_staging_file(directory=directory, suffix=".db") as staged: staged_path = staged.name db = Database(staged_path) db.create(tables, version, show_progress=show_progress) db.connection.close() db = None os.chmod(staged_path, normal_creation_mode(directory)) try: os.link(staged_path, db_path) except FileExistsError: pass # Another creator won; use the existing-file path below. else: return Database(db_path, must_exist=True).connection finally: if db is not None: db.connection.close() if staged_path is not None: _remove_staging_file(staged_path) db = Database(db_path, must_exist=True) def reusable(): return (not overwrite and db.has_tables(table_names) and db.has_version() and db.version() == version) try: if reusable(): logger.info("Found existing table in database %s", db_path) else: # Serialize rebuilds and recheck after acquiring SQLite's write # lock. Never unlink an existing database or commit drops alone. try: with db.connection: db.connection.execute("BEGIN IMMEDIATE") if not reusable(): db.drop_all_tables(commit=False, include_views=overwrite) logger.info("Creating database %s containing: %s", db_path, ", ".join(table_names)) db.create(tables, version, show_progress=show_progress) except sqlite3.OperationalError as error: if not _is_lock_error(error): _explain_rebuild_failure(db_path, error) raise except BaseException: logger.warning( "Failed to create tables %s in database %s", table_names, db_path) db.connection.rollback() db.connection.close() raise return db.connection def build_tables( table_names_to_dataframes, table_names_to_primary_keys=None, table_names_to_indices=None): """ Parameters ---------- table_names_to_dataframes : dict Dictionary mapping each table name to a DataFrame table_names_to_primary_keys : dict Dictionary mapping each table to its primary key table_names_to_indices : dict Dictionary mapping each table to a set of indices Returns list of DatabaseTable objects """ table_names_to_primary_keys = table_names_to_primary_keys or {} table_names_to_indices = table_names_to_indices or {} tables = [] for table_name, df in table_names_to_dataframes.items(): table_indices = table_names_to_indices.get(table_name, []) primary_key = table_names_to_primary_keys.get(table_name) table = DatabaseTable.from_dataframe( name=table_name, df=df, indices=table_indices, primary_key=primary_key) tables.append(table) return tables def db_from_dataframes_with_absolute_path( db_path, table_names_to_dataframes, table_names_to_primary_keys=None, table_names_to_indices=None, overwrite=False, version=1, *, show_progress=False): """ Create a sqlite3 database from a collection of DataFrame objects. Return an open connection owned by the caller. The parent directory must already exist. Rebuilds, including overwrite=True, roll back on failure; existing databases are not unlinked. show_progress enables tqdm row counts. Parameters ---------- db_path : str Path to database file to create table_names_to_dataframes : dict Dictionary from table names to DataFrame objects table_names_to_primary_keys : dict, optional Name of primary key column for each table table_names_to_indices : dict, optional Dictionary from table names to list of column name tuples overwrite : bool, optional If the database already exists, overwrite it? version : int, optional """ if not overwrite: connection = _cached_connection(db_path, table_names_to_dataframes, version) if connection is not None: return connection return _build_database( db_path, table_names_to_dataframes, table_names_to_primary_keys, table_names_to_indices, overwrite, version, show_progress) def _build_database(db_path, dataframes, primary_keys, indices, overwrite, version, show_progress): """Build (or, after a race, reuse) a database once reuse was ruled out.""" tables = build_tables(dataframes, primary_keys, indices) return _create_cached_db( db_path, tables=tables, version=version, overwrite=overwrite, show_progress=show_progress) def db_from_dataframes( db_filename, dataframes, primary_keys=None, indices=None, subdir=None, overwrite=False, version=1, *, cache_root=None, show_progress=False): """ Create a sqlite3 database from a collection of DataFrame objects. Return an open connection owned by the caller. cache_root selects an exact cache directory; show_progress enables optional tqdm insertion progress. Bump version or set overwrite=True to refresh a matching cached database. Parameters ---------- db_filename : str Name of database file to create dataframes : dict Dictionary from table names to DataFrame objects primary_keys : dict, optional Name of primary key column for each table indices : dict, optional Dictionary from table names to list of column name tuples subdir : str, optional Application name selecting a platform cache directory; ignored when cache_root is supplied. overwrite : bool, optional If the database already exists, overwrite it? version : int, optional """ if not (subdir is None or isinstance(subdir, str)): raise TypeError("Expected subdir to be None or str, got %s : %s" % ( subdir, type(subdir))) db_path = resolve_path(db_filename, subdir, cache_root=cache_root) if not overwrite: connection = _cached_connection(db_path, dataframes, version) if connection is not None: return connection os.makedirs(os.path.dirname(db_path) or ".", exist_ok=True) return _build_database(db_path, dataframes, primary_keys, indices, overwrite, version, show_progress) def db_from_dataframe( db_filename, table_name, df, primary_key=None, subdir=None, overwrite=False, indices=(), version=1, *, cache_root=None, show_progress=False): """ Given a dataframe `df`, turn it into a sqlite3 database. Store values in a table called `table_name`. Returns an open sqlite3.Connection owned by the caller. Close it when done. Rebuild failures preserve the previous database, including on overwrite. """ return db_from_dataframes( db_filename=db_filename, dataframes={table_name: df}, primary_keys={table_name: primary_key}, indices={table_name: indices}, subdir=subdir, overwrite=overwrite, version=version, cache_root=cache_root, show_progress=show_progress) def _truncate_name(name, limit): """Return the longest prefix of name within limit UTF-8 bytes.""" return name.encode("utf-8", "surrogatepass")[:limit].decode("utf-8", "ignore") def _db_filenames_from_dataframe(base_filename, df): """ Generate database filename for a sqlite3 database we're going to fill with the contents of a DataFrame, using the DataFrame's column names and types. Returns the filename and, when it differs, the historical name older releases used for the same schema (otherwise None). The historical spelling is kept only when every character of the file name is allowed on all supported platforms and SQLite's journal still fits beside it. Other names use a digest, so column names from downloaded data can never add directories or leave the cache directory. Lengths are measured the same way on every platform (see common.name_length); only an explicit CSV filename's path separators are interpreted by the local platform. """ validate_column_names(df.columns) columns = [(column_name, db_type(df[column_name].dtype)) for column_name in df.columns] schema = "".join(".%s_%s" % (name.replace(" ", "_"), kind) for name, kind in columns) prefix = base_filename + ("_nrows%d" % len(df)) historical = prefix + schema + ".db" # An explicit CSV filename may name directories; those are the caller's. cut = max((match.end() for match in _SEPARATORS.finditer(prefix)), default=0) directory, stem = prefix[:cut], prefix[cut:] if (not _UNSAFE_NAME_CHARACTERS.search(stem + schema) and name_length(stem + schema + ".db") <= _MAX_DB_NAME_LENGTH): return historical, None # The spelled-out schema is ambiguous (column "a_INT.b" of TEXT reads like # two columns), so digest an unambiguous encoding instead. digest = name_digest(json.dumps([stem, columns])) safe_stem = _UNSAFE_NAME_CHARACTERS.sub("_", stem) safe_stem = _truncate_name(safe_stem, _MAX_DB_NAME_LENGTH - len(".%s.db" % digest)) # A historical name with a ".." component could lie outside the cache, and # one with a NUL could never have been created, so neither is reused. reusable = ".." not in _SEPARATORS.split(stem + schema) and "\x00" not in historical return "%s%s.%s.db" % (directory, safe_stem, digest), historical if reusable else None def _is_lock_error(error): """Is this SQLite's busy or locked error, rather than a failure to open?""" code = getattr(error, "sqlite_errorcode", None) # Python 3.11+ if code is not None: return code & 0xFF in (5, 6) # SQLITE_BUSY, SQLITE_LOCKED return "locked" in str(error) def _open_legacy_database(path, table_name, version): """Reuse a matching database an older release stored at path. Returns (connection, present): the connection when it can be reused, and whether a SQLite database is there at all. A name this platform rejects, a file SQLite cannot open, or anything that is not a SQLite database only means there is nothing to reuse. A busy or locked database raises instead: building a second copy beside one that is in use would let the two diverge. """ try: if not stat.S_ISREG(os.stat(path).st_mode): return None, False except (OSError, UnicodeError): return None, False try: return _cached_connection(path, [table_name], version), True except sqlite3.DatabaseError as error: if _is_lock_error(error): raise return None, False def fetch_csv_db( table_name, download_url, csv_filename=None, db_filename=None, subdir=None, version=1, *, download_options=None, show_progress=False, **pandas_kwargs): """ Download CSV data and return an open SQLite connection owned by the caller. Omitted CSV/database filenames are inferred from the URL and schema. Parser settings go in pandas_kwargs; fetch_file settings go in download_options. Its cache_root applies to both files. show_progress enables optional download and row-insertion displays. Bump version when the data or schema changes to rebuild an existing matching database. """ options = download_options or {} cache_root = options.get("cache_root") superseded = False check_source = (options.get("force") or options.get("expected_sha256") is not None or options.get("expected_size") is not None) if db_filename is not None and not check_source: connection = _cached_connection( resolve_path(db_filename, subdir, cache_root=cache_root), [table_name], version) if connection is not None: return connection df = fetch_csv_dataframe( download_url=download_url, filename=csv_filename, subdir=subdir, download_options=download_options, show_progress=show_progress, **pandas_kwargs) if db_filename is None: # Explicit filenames have always used the caller's spelling here, # including the last compressed suffix and any path components, and # still do whenever that spelling is a valid name on every platform. source_filename = (os.fspath(csv_filename) if csv_filename is not None else build_local_filename(download_url, decompress=True)) base_filename = splitext(source_filename)[0] db_filename, legacy_filename = _db_filenames_from_dataframe(base_filename, df) if legacy_filename is not None and not path_exists( resolve_path(db_filename, subdir, cache_root=cache_root)): # Reuse a matching database an older release stored under its # historical name, without moving it; a new version rebuilds under # the new name. The older copy is never deleted: another process, # perhaps running an older release, may still have it open. legacy_path = resolve_path(legacy_filename, subdir, cache_root=cache_root) connection, superseded = _open_legacy_database(legacy_path, table_name, version) if connection is not None: return connection connection = db_from_dataframe( db_filename, table_name, df, subdir=subdir, version=version, cache_root=cache_root, show_progress=show_progress) if superseded: # Only now that the replacement exists is the older copy truly unused. logger.warning( "Built %s under a new name; the older database at %s is no longer used " "and can be deleted once no older datacache release uses this cache.", db_filename, legacy_path) return connection ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1792090401.70604 datacache-1.27.1/datacache/database_table.py0000644000076500000000000001217015264220442020401 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import datetime from .database import fold_identifier from .database_types import db_type def _sqlite_converter(): """Return a converter keeping scalar types and integer precision, with missing values as NULL. numpy and pandas are imported only once a DataFrame is converted, so importing datacache stays fast.""" import numpy as np import pandas as pd # pd.NA was introduced after the original supported pandas versions. # Missing nullable scalars only exist when that feature is installed. pandas_na = getattr(pd, "NA", object()) def convert(value): if value is None or value is pandas_na or value is pd.NaT: return None if isinstance(value, np.generic): value = value.item() if isinstance(value, float) and pd.isnull(value): return None if isinstance(value, (datetime.datetime, datetime.date)): return value.isoformat() return value return convert def validate_column_names(columns): """Require the non-empty string column names a database table needs.""" if len(columns) == 0 or not all(isinstance(column, str) and column for column in columns): raise ValueError("DataFrame columns must be non-empty strings") class DatabaseTable: """Converts between a DataFrame and a sqlite3 database table""" def __init__( self, name, column_types, make_rows, indices=None, nullable=None, primary_key=None, row_count=None): self.name = name self.column_types = column_types self.make_rows = make_rows self.indices = [] if indices is None else indices self.nullable = set() if nullable is None else nullable self.primary_key = primary_key self.row_count = row_count @property def rows(self): """Delay constructing list of row tuples""" return list(self.iter_rows()) def iter_rows(self): """Iterate rows without materializing the full table.""" return iter(self.make_rows()) @classmethod def from_dataframe(cls, name, df, indices, primary_key=None): """Infer types and normalize column/constraint names consistently.""" validate_column_names(df.columns) normalized = [column.replace(" ", "_") for column in df.columns] if len(set(fold_identifier(column) for column in normalized)) != len(normalized): raise ValueError("DataFrame column names collide after normalization") names = dict(zip(df.columns, normalized)) def column_name(column): # Accept original names and their legacy normalized spellings. if column in names: return names[column] if column in normalized: return column raise ValueError("Unknown column %r in table %r" % (column, name)) primary_key = column_name(primary_key) if primary_key is not None else None indices = list(indices) if any(isinstance(group, str) for group in indices): raise ValueError("Each index must be a sequence of column names, not a string") indices = [tuple(column_name(column) for column in group) for group in indices] if any(not group for group in indices): raise ValueError("Index column groups must be non-empty") # ordered list (column_name, column_type) pairs column_types = [] # which columns have nullable values nullable = set() # tag cached database by dataframe's number of rows and columns for column_name in df.columns: values = df[column_name] if values.isnull().any(): nullable.add(names[column_name]) column_db_type = db_type(values.dtype) column_types.append((column_name.replace(" ", "_"), column_db_type)) convert = _sqlite_converter() def make_rows(): # df.values coerces mixed numeric columns to floats and exposes # numpy integers that sqlite3 binds as blobs. Read column scalars. # Column iteration also works on pandas versions predating the # name=None option of itertuples; zip remains lazy on Python 3. return (tuple(convert(value) for value in row) for row in zip(*(df[column] for column in df.columns))) return cls( name=name, column_types=column_types, make_rows=make_rows, indices=indices, nullable=nullable, primary_key=primary_key, row_count=len(df)) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7061436 datacache-1.27.1/datacache/database_types.py0000644000076500000000000000553415264220442020464 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Convert from Python type names to sqlite3 column types""" _dtype_to_db_type_dict = { 'int': 'INT', 'int8': 'INT', 'int16': 'INT', 'int32': 'INT', 'int64': 'INT', 'uint8': 'INT', 'uint16': 'INT', 'uint32': 'INT', 'uint64': 'INT', 'bool': 'INT', 'boolean': 'INT', 'float': 'FLOAT', 'float16': 'FLOAT', 'float32': 'FLOAT', 'float64': 'FLOAT', 'object': 'TEXT', 'object_': 'TEXT', 'str': 'TEXT', 'str_': 'TEXT', 'string': 'TEXT', 'category': 'TEXT', 'bytes': 'BLOB', 'bytes_': 'BLOB', } def _lookup_type_name(type_name): type_name = type_name.lower() if type_name.startswith("datetime64["): return "TEXT" if type_name in _dtype_to_db_type_dict: return _dtype_to_db_type_dict[type_name] else: return None def _candidate_type_names(python_type_representation): """Generator which yields possible type names to look up in the conversion dictionary. Parameters ---------- python_type_representation : object Any Python object which represents a type, such as `int`, `dtype('int8')`, `np.int8`, or `"int8"`. """ # if we get a dtype object i.e. dtype('int16'), then pull out its name if hasattr(python_type_representation, 'name'): yield python_type_representation.name # convert Python types by adding their type's name if hasattr(python_type_representation, '__name__'): yield python_type_representation.__name__ # for a fixed-width dtype like dtype('S3') or dtype('U5'), the scalar # type's name ('bytes_' or 'str_') identifies it if hasattr(python_type_representation, 'type'): if hasattr(python_type_representation.type, '__name__'): yield python_type_representation.type.__name__ yield str(python_type_representation) def db_type(python_type_representation): """ Converts from any of: (1) Python type (2) NumPy/Pandas dtypes (3) string names of types ...to a sqlite3 type name """ for type_name in _candidate_type_names(python_type_representation): db_type_name = _lookup_type_name(type_name) if db_type_name: return db_type_name raise ValueError("Failed to find sqlite3 column type for %s" % ( python_type_representation)) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7064884 datacache-1.27.1/datacache/download.py0000644000076500000000000015543515264220442017311 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from contextlib import closing from datetime import datetime, timedelta, timezone import gzip import io import logging import math import os import stat import time import warnings from shutil import copyfileobj from tempfile import gettempdir, TemporaryDirectory from uuid import uuid4 import zipfile import urllib.parse import urllib.request from . import common, provenance from .common import _source_suffix, build_local_filename from .integrity import FileValidationError, _validate_expectations, validate_file from .inspection import path_exists from .progress import Progress from .retries import ( DEFAULT_MAX_RETRIES, DEFAULT_RETRY_BACKOFF, DEFAULT_RETRY_MAX_DELAY, error_description, is_retryable_http_error, retry_delay, validate_retry_options, ) logger = logging.getLogger(__name__) def __getattr__(name): """Keep download.pd and download.requests, now imported only when needed. pandas and requests are imported inside the functions that use them, so importing datacache stays fast. Existing references to these module attributes, such as test patches, still reach the same modules. """ if name == "pd": import pandas return pandas if name == "requests": import requests return requests raise AttributeError("module %r has no attribute %r" % (__name__, name)) # Number of bytes to read/write at a time when streaming a download to disk. DEFAULT_CHUNK_SIZE = 2 ** 20 # 1 MB class EmptyResponse(Exception): """A transfer completed without error but delivered no bytes. A withdrawn upstream record or a misbehaving edge server can answer with a clean, empty 200. HTTP attempts retry it like a transient failure. """ class MaxBytesExceeded(Exception): """Writing more would make a file larger than the caller's max_bytes. Never retried: the same request would deliver the same oversized bytes. """ def validate_limit(name, value): """Raise ValueError unless value is None (no limit) or a non-negative integer.""" if value is not None and (isinstance(value, bool) or not isinstance(value, int) or value < 0): raise ValueError("%s must be a non-negative integer" % name) def validate_size_within_limit(size, max_bytes, what="expected_size"): """Raise ValueError when a known size can never fit within max_bytes.""" if max_bytes is not None and size is not None and size > max_bytes: raise ValueError("%s %d is larger than max_bytes %d" % (what, size, max_bytes)) def refuse_if_over(size, max_bytes): """Raise MaxBytesExceeded if size is more than max_bytes.""" if max_bytes is not None and size > max_bytes: raise MaxBytesExceeded("more than max_bytes=%d bytes" % max_bytes) class LimitedWriter: """A file that refuses any write that would take it past max_bytes. The refused write raises MaxBytesExceeded before anything is written, so the file never holds more than max_bytes bytes. """ def __init__(self, file, max_bytes): self.file = file self.max_bytes = max_bytes self.written = 0 def write(self, data): refuse_if_over(self.written + len(data), self.max_bytes) self.written += len(data) return self.file.write(data) def _content_length(header_value): """Parse a Content-Length header value into an int, or None if it's absent or not a valid integer.""" if header_value is None: return None try: length = int(header_value) return length if length >= 0 else None except (TypeError, ValueError): return None def _stream_to_file( download_url, file_handle, timeout=None, chunk_size=DEFAULT_CHUNK_SIZE, progress_callback=None, max_bytes=None): """ Stream the contents of `download_url` into an already-open binary file handle, one chunk at a time, so the entire payload never has to be held in memory at once. This performs one attempt. _download_to_temp_file owns retries and creates a fresh staging file for each attempt. If `progress_callback` is given it is called as ``progress_callback(bytes_downloaded, total_bytes)`` after each chunk is written, where `total_bytes` is taken from the server's Content-Length header (or None when the server doesn't report a size). This lets callers drive an application's own progress display instead of the built-in bar. max_bytes, if given, is checked before each write, so no more than that many bytes are ever written; a larger Content-Length is refused before the body is read. Exceeding it raises MaxBytesExceeded. Returns the total number of bytes written. """ bytes_downloaded = 0 if max_bytes is not None: file_handle = LimitedWriter(file_handle, max_bytes) # Never read, and so never decode, much more than the limit at once. chunk_size = min(chunk_size, max_bytes + 1) def report(total_bytes): if progress_callback is not None: progress_callback(bytes_downloaded, total_bytes) if urllib.parse.urlsplit(download_url).scheme.lower() in ("http", "https"): import requests with closing(requests.get(download_url, headers={"User-Agent": common.USER_AGENT}, timeout=timeout, stream=True)) as response: response.raise_for_status() # Requests decodes content encoding before yielding chunks; a wire # length is not a valid total for the bytes we write in that case. encoding = response.headers.get("Content-Encoding", "identity").lower() total_bytes = (_content_length(response.headers.get("Content-Length")) if encoding == "identity" else None) # Chunked transfer makes Content-Length meaningless (RFC 9112). if "Transfer-Encoding" not in response.headers: refuse_if_over(total_bytes or 0, max_bytes) for chunk in response.iter_content(chunk_size=chunk_size): if not chunk: # skip keep-alive chunks that carry no data continue file_handle.write(chunk) bytes_downloaded += len(chunk) report(total_bytes) else: req = urllib.request.Request(download_url) with urllib.request.urlopen(req, data=None, timeout=timeout) as response: total_bytes = _content_length(response.headers.get("Content-Length")) refuse_if_over(total_bytes or 0, max_bytes) while True: chunk = response.read(chunk_size) if not chunk: break file_handle.write(chunk) bytes_downloaded += len(chunk) report(total_bytes) return bytes_downloaded def _with_retries( download_url, attempt, *, max_retries=DEFAULT_MAX_RETRIES, retry_backoff=DEFAULT_RETRY_BACKOFF, retry_max_delay=DEFAULT_RETRY_MAX_DELAY, may_retry=None): """Return attempt(), retrying transient HTTP(S) failures. Connection errors, 408/429/5xx responses and empty bodies (EmptyResponse) are retried with doubling backoff capped by retry_max_delay; Retry-After is honored within that cap. Other schemes and other errors are not retried, nor is any failure for which may_retry(error) is false. Options must already be validated. """ http = urllib.parse.urlsplit(download_url).scheme.lower() in ("http", "https") backoff = retry_backoff for number in range(max_retries + 1): try: return attempt() except BaseException as error: retryable = isinstance(error, EmptyResponse) or is_retryable_http_error(error) if not http or not retryable or (may_retry is not None and not may_retry(error)): raise if number == max_retries: logger.warning("HTTP download failed after %d attempt(s): %s", number + 1, error_description(error)) raise delay = retry_delay(error, backoff, retry_max_delay) if delay is None: logger.warning("HTTP download attempt %d/%d failed (%s); Retry-After exceeds retry_max_delay", number + 1, max_retries + 1, error_description(error)) raise logger.warning("HTTP download attempt %d/%d failed (%s); retrying in %.3g seconds", number + 1, max_retries + 1, error_description(error), delay) if delay: time.sleep(delay) backoff *= 2 # retry_delay caps it at retry_max_delay. def _download_to_temp_file( download_url, timeout=None, base_name="download", chunk_size=DEFAULT_CHUNK_SIZE, progress_callback=None, directory=None, *, max_retries=DEFAULT_MAX_RETRIES, retry_backoff=DEFAULT_RETRY_BACKOFF, retry_max_delay=DEFAULT_RETRY_MAX_DELAY, show_progress=False, allow_empty=True, max_bytes=None): retry_backoff, retry_max_delay = validate_retry_options(max_retries, retry_backoff, retry_max_delay) if not download_url: raise ValueError("URL not provided") callback_failed = False def attempt(): nonlocal callback_failed callback_failed = False tmp_path = None def report(done, total): nonlocal callback_failed try: progress(done, total) if progress_callback is not None: progress_callback(done, total) except BaseException: callback_failed = True raise try: with Progress(show_progress, "Downloading") as progress, _open_staging_file( directory=directory, suffix='.tmp', prefix=base_name) as tmp: tmp_path = tmp.name count = _stream_to_file( download_url, tmp, timeout=timeout, chunk_size=chunk_size, progress_callback=report if progress_callback is not None or show_progress else None, max_bytes=max_bytes) if count == 0: progress(0, 0) if not allow_empty: raise EmptyResponse("the transfer delivered no bytes") return tmp_path except BaseException: if tmp_path is not None: _remove_staging_file(tmp_path) raise return _with_retries( download_url, attempt, max_retries=max_retries, retry_backoff=retry_backoff, retry_max_delay=retry_max_delay, may_retry=lambda error: not callback_failed) def fetch_bytes( download_url, *, timeout=None, max_retries=DEFAULT_MAX_RETRIES, retry_backoff=DEFAULT_RETRY_BACKOFF, retry_max_delay=DEFAULT_RETRY_MAX_DELAY, allow_empty=False, max_bytes=None): """ Return a remote resource's bytes in memory, transferred and retried exactly as fetch_file downloads are. Nothing is written to disk. Meant for small resources such as directory listings and metadata: the whole body is held in memory. It also works where a cache is read-only. Parameters ---------- download_url : str HTTP, HTTPS, FTP, or file:// URL. timeout, max_retries, retry_backoff, retry_max_delay : optional As for fetch_file: a per-attempt timeout (None waits indefinitely), and retries of connection errors, 408/429/5xx responses and empty bodies with capped backoff that honors Retry-After. Only HTTP(S) transfers are retried. allow_empty : bool, optional Accept an empty body, default False. An empty HTTP response is otherwise retried as transient, then rejected with ValueError. max_bytes : int, optional The largest body to accept, after HTTP transfer decoding. It is checked before each chunk is kept, so the returned bytes never grow past it, and the body is read in chunks no larger than the limit. A larger body raises ValueError without retrying. Default None: no limit. Raises ValueError for invalid options, an empty body or one larger than max_bytes; Requests exceptions for HTTP failures, preserved after retry exhaustion; and urllib.error.URLError for file and FTP failures. """ retry_backoff, retry_max_delay = validate_retry_options(max_retries, retry_backoff, retry_max_delay) if not isinstance(download_url, str) or not download_url: raise ValueError("URL not provided") if not isinstance(allow_empty, bool): raise ValueError("allow_empty must be a boolean") validate_limit("max_bytes", max_bytes) if max_bytes == 0 and not allow_empty: raise ValueError("max_bytes=0 allows only an empty body; pass allow_empty=True") def attempt(): buffer = io.BytesIO() _stream_to_file(download_url, buffer, timeout=timeout, max_bytes=max_bytes) if not buffer.getbuffer().nbytes and not allow_empty: raise EmptyResponse("the transfer delivered no bytes") return buffer.getvalue() try: return _with_retries( download_url, attempt, max_retries=max_retries, retry_backoff=retry_backoff, retry_max_delay=retry_max_delay) except EmptyResponse as error: raise ValueError( "%s returned no bytes; pass allow_empty=True if an empty body is expected" % provenance.redact_url(download_url)) from error except MaxBytesExceeded as error: raise ValueError("%s returned %s" % (provenance.redact_url(download_url), error)) from error def _open_staging_file(directory=None, prefix=".datacache-", suffix="", mode=0o600): """Create a unique sibling file exclusively, honoring umask for its mode. Unlike chmod after NamedTemporaryFile, exclusive creation with the desired mode lets the OS apply umask without reading/changing process-global state. Callers own cleanup after closing the returned binary file. """ directory = gettempdir() if directory is None else directory for _ in range(100): path = os.path.join(directory, prefix + uuid4().hex + suffix) try: return open(path, "x+b", opener=lambda name, flags: os.open(name, flags, mode)) except FileExistsError: continue raise FileExistsError("Could not create a unique staging file in %s" % directory) def _remove_staging_file(path): try: os.remove(path) except FileNotFoundError: pass def normal_creation_mode(directory): """Measure ordinary creation permissions using an empty, disposable file. Never put data in this file: another user might open it before removal. Actual download/conversion files remain private through validation. """ probe_path = None try: with _open_staging_file( directory=directory, prefix=".datacache-mode-", mode=0o666) as probe: probe_path = probe.name return stat.S_IMODE(os.fstat(probe.fileno()).st_mode) & 0o777 finally: if probe_path is not None: _remove_staging_file(probe_path) def _publish_staged_file(staged_path, full_path, mode=None): """Apply normal creation or existing access permissions before publication. Call only after writing and validation, so staging data stays private until it is ready to publish. New files honor the destination's creation mode; replacements preserve the existing regular file's access permissions. An explicit mode overrides both. Returns the mode applied. """ try: existing = os.stat(full_path) except FileNotFoundError: if mode is None: mode = normal_creation_mode(os.path.dirname(full_path) or ".") else: if not stat.S_ISREG(existing.st_mode): raise FileValidationError(full_path, "expected a regular file") if mode is None: # Preserve rwx permissions, not setuid/setgid/sticky bits on new content. mode = stat.S_IMODE(existing.st_mode) & 0o777 os.chmod(staged_path, mode) os.replace(staged_path, full_path) return mode def _publish_file(staged_path, full_path, record=None): """Publish a staged file, keeping its provenance record truthful. Any record of the previous bytes is removed first, so an interruption leaves the new file unrecorded, never misdescribed. record (JSON text from provenance.describe) is then written with the file's own permissions, without execute bits, so a private file keeps a private record. Records are best effort: failing to write one, or to remove an old one, never fails the publication. """ try: provenance.remove(full_path) except OSError as error: logger.warning("Could not remove the provenance record for %s: %s", full_path, error) mode = _publish_staged_file(staged_path, full_path) if record is None: return staged_record = None try: with _open_staging_file( directory=os.path.dirname(full_path) or ".", prefix=".datacache-provenance-") as output: staged_record = output.name output.write(record.encode("utf-8")) _publish_staged_file(staged_record, provenance.sidecar_path(full_path), mode & 0o666) except (OSError, ValueError) as error: logger.debug("Could not record provenance for %s: %s", full_path, error) finally: if staged_record is not None: _remove_staging_file(staged_record) def _copy_with_progress(source, destination, show_progress, total=None, max_bytes=None): if max_bytes is not None: destination = LimitedWriter(destination, max_bytes) if not show_progress: return copyfileobj(source, destination) with Progress(True, "Decompressing", total) as progress: completed = 0 while True: chunk = source.read(DEFAULT_CHUNK_SIZE) if not chunk: break destination.write(chunk) completed += len(chunk) progress(completed, total) def _choose_zip_member(infos, filename, warn=True): """Pick the archive member to install as filename. Prefer the member stored at exactly that name, then a member with that name, ignoring case, in any folder: nearest the archive root first, then exact case, then the largest. Otherwise install the largest member, warning (if warn) when that was a guess among several. """ chosen = next((info for info in infos if info.filename == filename), None) if chosen is not None: return chosen paths = {info: info.filename.replace("\\", "/") for info in infos} names = {info: path.rsplit("/", 1)[-1] for info, path in paths.items()} depths = {info: sum(part not in ("", ".") for part in path.split("/")[:-1]) for info, path in paths.items()} named = [info for info in infos if names[info].casefold() == filename.casefold()] if named: return min(named, key=lambda info: ( depths[info], names[info] != filename, -info.file_size)) chosen = max(infos, key=lambda info: info.file_size) if warn and len(infos) > 1: logger.warning("No ZIP member is named %s; installing the largest of %d members, %s", filename, len(infos), chosen.filename) return chosen def _download_and_decompress_if_necessary( full_path, download_url, timeout=None, chunk_size=DEFAULT_CHUNK_SIZE, progress_callback=None, *, decompress=None, convert_html=None, explicit_output=None, expected_sha256=None, expected_size=None, max_retries=DEFAULT_MAX_RETRIES, retry_backoff=DEFAULT_RETRY_BACKOFF, retry_max_delay=DEFAULT_RETRY_MAX_DELAY, show_progress=False, record_provenance=False, allow_empty=False, validator=None, max_bytes=None): """ Download, transform and validate in sibling staging files, then publish with one atomic replace. Expectations always describe installed bytes. Unspecified transform flags retain the pre-1.8 literal-URL heuristics for downstream callers (including pyensembl) using this private entry point. Explicit flags use the parsed URL format, including download endpoints. explicit_output=False marks an inferred cache key, which no archive member can match, so installing the largest ZIP member is not reported as a guess. Publishing always removes a stale provenance record; record_provenance writes a new one describing the staged bytes. An empty download or installed file is rejected unless allow_empty is true or expected_size is 0; empty HTTP responses are retried first. """ logger.info("Downloading %s to %s", download_url, full_path) full_path = os.fspath(full_path) filename = os.path.basename(full_path) out_dir = os.path.dirname(full_path) or "." source_suffix = _source_suffix(download_url) output_suffix = os.path.splitext(filename)[1].lower() if decompress is None: unzip = download_url.endswith("zip") and not filename.endswith("zip") gunzip = download_url.endswith("gz") and not filename.endswith("gz") else: unzip = source_suffix == ".zip" and decompress gunzip = source_suffix == ".gz" and decompress if convert_html is None: html = download_url.endswith(("html", "htm")) and full_path.endswith(".csv") else: html = convert_html and source_suffix in (".html", ".htm") and output_suffix == ".csv" if html and max_bytes is not None: raise ValueError("max_bytes can't bound HTML-to-CSV conversion; use raw=True for the HTML itself") # expected_size=0 states that an empty file is the right result. reject_empty = not allow_empty and expected_size != 0 try: tmp_path = _download_to_temp_file( download_url=download_url, timeout=timeout, base_name=".datacache-download-", directory=out_dir, chunk_size=chunk_size, progress_callback=progress_callback, max_retries=max_retries, retry_backoff=retry_backoff, retry_max_delay=retry_max_delay, show_progress=show_progress, allow_empty=not reject_empty, max_bytes=max_bytes) except EmptyResponse as error: raise FileValidationError(full_path, "downloaded file is empty; pass allow_empty=True if an empty file is expected") from error except MaxBytesExceeded as error: raise FileValidationError(full_path, "download has %s" % error) from error staged_path = tmp_path try: if unzip or gunzip or html: with _open_staging_file( directory=out_dir, prefix=".datacache-install-") as tmp: staged_path = tmp.name if unzip: with zipfile.ZipFile(tmp_path) as z: infos = [info for info in z.infolist() if not info.is_dir()] if not infos: raise ValueError("Empty zip archive") # Never extract stored paths: stream one member's contents. chosen = _choose_zip_member(infos, filename, warn=explicit_output is not False) refuse_if_over(chosen.file_size, max_bytes) with z.open(chosen) as src, open(staged_path, "wb") as dst: _copy_with_progress(src, dst, show_progress, chosen.file_size, max_bytes) elif gunzip: with gzip.GzipFile(tmp_path) as src, open(staged_path, "wb") as dst: _copy_with_progress(src, dst, show_progress, max_bytes=max_bytes) else: import pandas as pd df = pd.read_html(tmp_path, header=0)[0] df.to_csv(staged_path, sep=',', index=False, encoding='utf-8') if reject_empty and os.path.getsize(staged_path) == 0: raise FileValidationError(full_path, "installed file is empty; pass allow_empty=True if an empty file is expected") try: if show_progress: validate_file(staged_path, expected_sha256, expected_size, show_progress=True) else: validate_file(staged_path, expected_sha256, expected_size) except FileValidationError as error: raise FileValidationError(full_path, "downloaded file " + error.reason) from error _run_validator(validator, staged_path, full_path, "downloaded file") record = None if record_provenance: # Describe the private staged bytes, not whatever is at full_path # after publication, which a concurrent writer may have replaced. # os.replace keeps their size and modification time. try: record = provenance.describe(download_url, os.stat(staged_path), expected_sha256) except OSError as error: logger.debug("Could not describe provenance for %s: %s", full_path, error) _publish_file(staged_path, full_path, record) except MaxBytesExceeded as error: # Only decompression writes here; the download was limited above. raise FileValidationError(full_path, "decompressed file has %s" % error) from error finally: if staged_path != tmp_path: _remove_staging_file(staged_path) _remove_staging_file(tmp_path) def _run_validator(validator, path, reported_path, description): """Raise FileValidationError for reported_path if validator(path) raises or returns False.""" if validator is None: return try: accepted = validator(path) except Exception as error: raise FileValidationError( reported_path, "%s failed validation: %s" % (description, error)) from error if accepted is False: raise FileValidationError(reported_path, "%s failed validation" % description) def _expiry_seconds(expire_after): """expire_after as seconds, or None for no expiry; ValueError if invalid.""" if expire_after is None: return None if isinstance(expire_after, timedelta): expire_after = expire_after.total_seconds() if (isinstance(expire_after, bool) or not isinstance(expire_after, (int, float)) or not math.isfinite(expire_after) or expire_after < 0): raise ValueError("expire_after must be a non-negative number of seconds or a timedelta") return expire_after def _check_cached_file(full_path, expected_sha256, expected_size, reject_empty, validator): """Raise unless full_path is a usable cache hit; FileNotFoundError if absent.""" validate_file(full_path, expected_sha256, expected_size) if reject_empty and os.stat(full_path).st_size == 0: raise FileValidationError(full_path, "cached file is empty (pass allow_empty=True if an empty file is expected)") _run_validator(validator, full_path, full_path, "cached file") def _age_seconds(full_path): """Seconds since full_path was fetched: from its provenance record when one describes it, otherwise from its modification time, which atomic publication sets when the download is written.""" info = os.stat(full_path) fetched = info.st_mtime record = provenance.read(full_path, info) if record is not None: try: recorded = datetime.fromisoformat(record["fetched_at"]) if recorded.tzinfo is None: recorded = recorded.replace(tzinfo=timezone.utc) # Records are UTC. fetched = recorded.timestamp() except (TypeError, ValueError, OverflowError): pass # A modification time in the future (clock skew, rsync) counts as new. return max(0.0, time.time() - fetched) def expected_path( download_url=None, filename=None, decompress=False, subdir=None, *, destination=None, cache_root=None): """Resolve the fetch destination without filesystem access or mutations. cache_root overrides the platform location selected by subdir. destination is an exact path and cannot be combined with filename, subdir, or cache_root. """ if destination is not None: if filename is not None or subdir is not None or cache_root is not None: raise ValueError("destination cannot be combined with filename, subdir, or cache_root") result = os.fspath(destination) if not isinstance(result, str) or not result: raise ValueError("destination must be a non-empty text path") return result filename = build_local_filename(download_url, filename, decompress) return common.resolve_path(filename, subdir, cache_root=cache_root) def file_exists( download_url=None, filename=None, decompress=False, subdir=None, *, destination=None, cache_root=None): """Check presence without writes/network; permission errors propagate. This does not check file type, readability, or integrity. Use inspect_file or validate_file on expected_path(...) for those checks. """ return path_exists(expected_path( download_url, filename, decompress, subdir, destination=destination, cache_root=cache_root)) DOWNLOAD_OPTIONS = frozenset(( "timeout", "chunk_size", "progress_callback", "show_progress", "max_retries", "retry_backoff", "retry_max_delay", "resume", "max_bytes")) def validate_download_options(options, kind): """Check the download_options an install passes to each fetch_file call. Bundle, archive and materialization installs accept the transfer settings named in DOWNLOAD_OPTIONS and nothing else. Each is checked as fetch_file checks it, so a bad value fails before an install creates anything rather than at its first download. options may be None. kind names the install in messages, such as "bundle". Returns a new dict of the options. """ options = dict(options or {}) unsupported = set(options) - DOWNLOAD_OPTIONS if unsupported: raise ValueError("unsupported %s download options: %s" % (kind, sorted(unsupported))) if options.get("max_bytes") == 0: raise ValueError("max_bytes=0 allows only empty downloads") validate_transfer_settings( chunk_size=options.get("chunk_size", DEFAULT_CHUNK_SIZE), progress_callback=options.get("progress_callback"), show_progress=options.get("show_progress", False), resume=options.get("resume", False), max_retries=options.get("max_retries", DEFAULT_MAX_RETRIES), retry_backoff=options.get("retry_backoff", DEFAULT_RETRY_BACKOFF), retry_max_delay=options.get("retry_max_delay", DEFAULT_RETRY_MAX_DELAY), max_bytes=options.get("max_bytes")) return options def validate_transfer_settings( *, chunk_size, progress_callback, show_progress, resume, max_retries, retry_backoff, retry_max_delay, max_bytes=None): """Check fetch_file's transfer settings; return the retry delays as floats.""" validate_limit("max_bytes", max_bytes) if not isinstance(resume, bool) and resume != "auto": raise ValueError('resume must be True, False or "auto"') if not isinstance(show_progress, bool): raise ValueError("show_progress must be a boolean") if progress_callback is not None and not callable(progress_callback): raise ValueError("progress_callback must be callable") if not isinstance(chunk_size, int) or isinstance(chunk_size, bool) or chunk_size <= 0: raise ValueError("chunk_size must be a positive integer") return validate_retry_options(max_retries, retry_backoff, retry_max_delay) def fetch_file( download_url, filename=None, decompress=False, subdir=None, force=False, timeout=None, use_wget_if_available=None, chunk_size=DEFAULT_CHUNK_SIZE, progress_callback=None, *, destination=None, cache_root=None, expected_sha256=None, expected_size=None, max_bytes=None, max_retries=DEFAULT_MAX_RETRIES, retry_backoff=DEFAULT_RETRY_BACKOFF, retry_max_delay=DEFAULT_RETRY_MAX_DELAY, show_progress=False, record_provenance=False, allow_empty=False, resume=False, raw=False, expire_after=None, return_stale_on_error=False, validator=None): """ Download a remote file and store it locally in a cache directory. Don't download it again if it's already present (unless `force` is True, or the cached copy is older than `expire_after`). Parameters ---------- download_url : str Remote URL of file to download. filename : str, optional Local filename, used as cache key. If omitted, then determine the local filename from the URL. decompress : bool, optional With an inferred filename, archives are retained by default, including for URLs with query strings or fragments. Set True to decompress them under a distinct cache key. An explicit filename or destination lacking the source's .zip/.gz suffix still implies decompression for compatibility. raw : bool, optional Disable archive decompression and HTML-to-CSV conversion regardless of the output name. Incompatible with decompress=True. Defaults to False, preserving legacy output-name inference. Integrity expectations then describe the unchanged payload, after HTTP transfer decoding. subdir : str, optional Application name selecting a platform cache directory, "datacache" by default. It is not nested inside the default cache. Ignored when cache_root is supplied. force : bool, optional By default, a remote file is not downloaded if it's already present. However, with this argument set to True, it will be overwritten. timeout : float or (float, float), optional Per-attempt connect/read timeout in seconds. The default None waits indefinitely. HTTP(S) also accepts a Requests (connect, read) tuple. use_wget_if_available : bool, optional Deprecated and ignored. datacache now always uses its streaming Python downloader (which handles http(s) and ftp); the legacy `wget` path was removed. Passing this argument emits a DeprecationWarning. chunk_size : int, optional Number of bytes to stream from the server to disk at a time. Defaults to 1 MB. progress_callback : callable, optional If provided, called as ``progress_callback(bytes_downloaded, total_bytes)`` after each chunk is written, where `total_bytes` is the server-reported size or None if unknown. Lets callers render a progress display in their application's own UI instead of the built-in bar. show_progress : bool, optional Show tqdm download, decompression, and hash-verification progress. tqdm is installed with datacache. Defaults to False; cache hits are quiet. Can be combined with progress_callback. Each retry starts a fresh bar. destination : str or os.PathLike, optional Exact output path, including filename, instead of the default cache. Mutually exclusive with filename, subdir, and cache_root. Parent directories are created only when downloading. With decompress=True, the destination name is kept exactly as supplied while archive contents are installed. cache_root : str or os.PathLike, optional Directory containing cached files, overriding the location selected by subdir. Cannot be combined with destination. expected_sha256 : str, optional Trusted SHA-256 hex digest of the installed bytes, after decompression or HTML conversion (not of the compressed archive or HTTP wire bytes). Checked both on cache reuse and before publishing a replacement. expected_size : int, optional Expected non-negative byte count of the same installed bytes. max_bytes : int, optional The most bytes any file this fetch writes may hold, checked before every write. It bounds the download itself, which is the body after HTTP transfer decoding such as gzip Content-Encoding (whether or not Content-Length is sent), and the installed file after decompress=True, so allow for the compressed download too; together they never need more than twice max_bytes. A larger transfer raises FileValidationError without retrying: an existing file stays and temporary files are removed. Cache hits write nothing and are unaffected. expected_size must not be larger, and HTML-to-CSV conversion can't be bounded; options that can't work together raise ValueError, even on a cache hit. Default None: no limit. max_retries : int, optional Additional attempts after transient HTTP/transport failures, default 2. Set 0 to disable retries. File and FTP transfers are not retried. retry_backoff : float, optional Initial retry delay in seconds, default 1. Doubles after each retry, capped by retry_max_delay. Progress counts restart for each attempt. retry_max_delay : float, optional Maximum delay between attempts, default 30 seconds. Retry-After is honored when within this limit; longer server waits stop retries. timeout still applies per attempt, not as a total download deadline. resume : bool or "auto", optional Keep private partials and resume raw HTTP(S) transfers, default False. "auto" resumes whenever it can and otherwise downloads normally: when the platform, URL, size or options don't allow it, and when the server turns out not to support it (the partial is then discarded and the file downloaded in full). resume=True instead raises in those cases. Requires expected_size and a POSIX local filesystem. If expected_sha256 is omitted, the server must supply a strong ETag on every accepted response; it is sent as If-Range when resuming. ETags prevent mixing representations, but do not verify a trusted checksum. Cache hits with size alone check only their byte count. Does not support decompression or HTML conversion; raw=True permits arbitrary output names. Retries report cumulative bytes, resetting only when the server requires a fresh transfer. discard_partial(destination) explicitly drops partials. Interrupted downloads retain at most expected_size partial bytes. record_provenance : bool, optional After publishing a download, also write a hidden "..datacache.json" record of the source URL (without user name, password, query, or fragment; the path is kept as is), fetch time, size, and the SHA-256 when expected_sha256 verified it, which inspect_file reports offline. The record has the file's permissions. Off by default: a caller that downloads to a temporary name and then moves the file would leave the record behind. Cache hits never write one, and any new download removes a previous record first. allow_empty : bool, optional Accept an empty file, default False. A complete but empty response, such as a withdrawn upstream record, is otherwise never published: HTTP retries it as transient, then FileValidationError is raised. An empty cached file is likewise invalid. expected_size=0 also allows it. expire_after : float or datetime.timedelta, optional How long a cached file stays fresh, in seconds or as a timedelta, like requests-cache's option of the same name. An older cached file is downloaded again, as is one that no longer validates; 0 refreshes every time. Its age comes from the provenance record's fetch time when one describes the file (see record_provenance), otherwise from its modification time; a future time counts as new. Default None reuses a valid file however old and raises for an invalid one. return_stale_on_error : bool, optional When a refresh (force=True or an expired expire_after) fails with an exception and a valid cached file exists, log a warning and return the cached path instead, as HTTP's stale-if-error directive allows. Without a valid cached file the error propagates, as does an exception from progress_callback, which cancels the fetch. Default False. validator : callable, optional validator(path) checks content that a successful transfer can still get wrong, such as an HTTP 200 error page. It rejects the file by raising (normally ValueError) or returning False. It runs on the installed bytes before publication, so a rejected download never replaces the cached file, and on cache hits. Rejection raises FileValidationError, chained to any exception. Not supported with resume=True. A corrupt cache hit raises FileValidationError; use force=True for an explicit repair. Missing files are downloaded. Transport, decompression and filesystem exceptions propagate. Failed downloads leave an existing destination unchanged and remove their staging files. Atomic replacement requires a local filesystem supporting os.replace; concurrent writers publish complete files with the last successful replacement winning. Staging files remain private until publication. New files use normal creation permissions (0666 filtered by umask); replacements preserve the existing file's read/write/execute permission bits. Returns the local path, which is relative when destination or cache_root is. """ retry_backoff, retry_max_delay = validate_transfer_settings( chunk_size=chunk_size, progress_callback=progress_callback, show_progress=show_progress, resume=resume, max_retries=max_retries, retry_backoff=retry_backoff, retry_max_delay=retry_max_delay, max_bytes=max_bytes) # "auto" is decided once the download's shape is known, below. resume_if_possible = resume == "auto" if resume_if_possible: resume = False if not isinstance(raw, bool): raise ValueError("raw must be a boolean") if raw and decompress: raise ValueError("raw=True cannot be combined with decompress=True") if resume: from .resume import validate_resume validate_resume(download_url, expected_sha256, expected_size) _validate_expectations(expected_sha256, expected_size) validate_size_within_limit(expected_size, max_bytes) if max_bytes == 0 and not allow_empty and expected_size != 0: raise ValueError("max_bytes=0 allows only an empty file; pass allow_empty=True") if not isinstance(record_provenance, bool): raise ValueError("record_provenance must be a boolean") if not isinstance(allow_empty, bool): raise ValueError("allow_empty must be a boolean") expiry = _expiry_seconds(expire_after) if not isinstance(return_stale_on_error, bool): raise ValueError("return_stale_on_error must be a boolean") if validator is not None and not callable(validator): raise ValueError("validator must be callable") if validator is not None and resume: raise ValueError("validator cannot be combined with resume=True") reject_empty = not allow_empty and expected_size != 0 # Query/fragment text in an inferred cache key is not an output-format request. explicit_output = destination is not None or bool(filename) if use_wget_if_available is not None: warnings.warn( "use_wget_if_available is deprecated and ignored; datacache now always " "uses its streaming Python downloader (handles http(s) and ftp).", DeprecationWarning, stacklevel=2) full_path = expected_path( download_url, filename, decompress, subdir, destination=destination, cache_root=cache_root) source_suffix = _source_suffix(download_url) output_suffix = os.path.splitext(full_path)[1].lower() archive_decompression = not raw and bool( decompress or (explicit_output and output_suffix != source_suffix)) html_conversion = (not raw and explicit_output and source_suffix in (".htm", ".html") and output_suffix == ".csv") if html_conversion and max_bytes is not None: raise ValueError("max_bytes can't bound HTML-to-CSV conversion; use raw=True for the HTML itself") transformed = bool(decompress or (source_suffix in (".gz", ".zip") and archive_decompression) or html_conversion) if resume and transformed: raise ValueError("resume=True supports raw downloads only; use raw=True or retain the archive suffix") if resume_if_possible: from .resume import can_resume resume = can_resume(download_url, expected_sha256, expected_size) and not transformed and validator is None # Whether the cached file is known valid, for a failed refresh to fall # back to (return_stale_on_error); with force=True it is checked only on failure. cached = False refresh = force if not force: try: _check_cached_file(full_path, expected_sha256, expected_size, reject_empty, validator) except FileNotFoundError: pass except FileValidationError as error: # force=True replaces a mismatched file, but never a directory or # other non-regular path, so only suggest it when it can help. try: replaceable = stat.S_ISREG(os.stat(full_path).st_mode) except OSError: replaceable = False if not replaceable: raise if expiry is None: raise FileValidationError( full_path, error.reason + "; use force=True to explicitly replace it") from error # An expiring cache replaces content that no longer validates. logger.info("Cached file %s is invalid (%s); fetching it again", full_path, error.reason) refresh = True else: if expiry is None or (expiry > 0 and _age_seconds(full_path) < expiry): logger.info("Cached file %s from URL %s", full_path, download_url) return full_path cached = True refresh = True logger.info("Cached file %s has expired; fetching it again", full_path) # A progress callback's exception cancels the fetch; never fall back. callback_errors = [] if progress_callback is not None: caller_callback = progress_callback def progress_callback(done, total): try: caller_callback(done, total) except BaseException as error: callback_errors.append(error) raise try: os.makedirs(os.path.dirname(full_path) or ".", exist_ok=True) logger.info("Fetching %s from URL %s", full_path, download_url) def download_in_full(): _download_and_decompress_if_necessary( full_path=full_path, download_url=download_url, timeout=timeout, chunk_size=chunk_size, progress_callback=progress_callback, decompress=archive_decompression, convert_html=html_conversion, explicit_output=explicit_output, expected_sha256=expected_sha256, expected_size=expected_size, max_retries=max_retries, retry_backoff=retry_backoff, retry_max_delay=retry_max_delay, show_progress=show_progress, record_provenance=record_provenance, allow_empty=allow_empty, validator=validator, max_bytes=max_bytes) if resume: from .resume import CannotResume, discard_partial, download_resumable try: download_resumable( download_url, full_path, expected_sha256=expected_sha256, expected_size=expected_size, timeout=timeout, chunk_size=chunk_size, progress_callback=progress_callback, show_progress=show_progress, max_retries=max_retries, retry_backoff=retry_backoff, retry_max_delay=retry_max_delay, record_provenance=record_provenance, force=refresh) except CannotResume as error: if not resume_if_possible: raise logger.info("Cannot resume %s (%s); downloading it in full", full_path, error.reason) try: discard_partial(full_path) except (OSError, FileValidationError) as discard_error: logger.debug("Could not discard resume state for %s: %s", full_path, discard_error) download_in_full() else: download_in_full() except Exception as error: if not return_stale_on_error or any(error is cancel for cancel in callback_errors): raise if not cached: try: # A failed download leaves the destination untouched. _check_cached_file(full_path, expected_sha256, expected_size, reject_empty, validator) cached = True except (OSError, ValueError): pass if not cached: raise # The refresh's own error, with its cause. logger.warning("Could not refresh %s (%s); using the cached copy", full_path, error_description(error)) return full_path def fetch_and_transform( transformed_filename, transformer, loader, source_filename, source_url, subdir=None, *, force=False, cache_root=None, download_options=None, show_progress=False): """ Download a source and cache a successful single-file transformation. transformer(source_path, output_path) must create and close output_path. It receives an absent path in a private sibling directory, with the final basename and extension. Its result is returned on success; cache hits call loader(final_path). A directly returned output path is remapped to the final path. Prefer in-memory results over objects containing staged paths. force rebuilds the transformed output while reusing the source. Set download_options={"force": True} to refresh the source too. Other fetch_file settings (timeouts, retries, hashes, callbacks) go in download_options. cache_root selects the root for both artifacts. For legacy compatibility, a nonempty subdir also defaults source decompression to True; override it explicitly in download_options. Existing sources misplaced in the default cache by older versions can be reused without moving them. Failed transformations preserve previous output and remove their partial files. """ transformed_path = common.resolve_path(transformed_filename, subdir, cache_root=cache_root) try: if force: raise FileNotFoundError(transformed_path) validate_file(transformed_path) except FileNotFoundError: options = dict(download_options or {}) options.setdefault("cache_root", cache_root) options.setdefault("show_progress", show_progress) options.setdefault("decompress", bool(subdir)) source_path = None # Older versions accidentally passed subdir as decompress and cached # the source in the default root. Check only that exact legacy path, # after preferring the requested root, and never move/chmod it. if (subdir and options["cache_root"] is None and not options.get("force") and "destination" not in options): current = expected_path(source_url, source_filename, options["decompress"], subdir) legacy = expected_path(source_url, source_filename, options["decompress"]) if not path_exists(current): try: validate_file(legacy, options.get("expected_sha256"), options.get("expected_size")) except FileNotFoundError: pass else: source_path = legacy if source_path is None: source_path = fetch_file(source_url, filename=source_filename, subdir=subdir, **options) logger.info("Generating data file %s from %s", transformed_path, source_path) os.makedirs(os.path.dirname(transformed_path) or ".", exist_ok=True) # A private directory keeps arbitrary transformer output private while # preserving the filename/extension and the absent-output contract. with TemporaryDirectory( dir=os.path.dirname(transformed_path) or ".", prefix=".datacache-transform-") as staging_directory: staged_path = os.path.join(staging_directory, os.path.basename(transformed_path)) result = transformer(source_path, staged_path) try: validate_file(staged_path) except FileNotFoundError as error: raise RuntimeError("Transformer did not create %s" % transformed_path) from error _publish_file(staged_path, transformed_path) if isinstance(result, (str, os.PathLike)) and os.fspath(result) == staged_path: result = type(result)(transformed_path) else: logger.info("Cached data file: %s", transformed_path) result = loader(transformed_path) return result def fetch_csv_dataframe( download_url, filename=None, subdir=None, *, download_options=None, show_progress=False, **pandas_kwargs): """ Download `download_url` (cached under the key `filename`, if given) and load it with pandas.read_csv, passing extra keyword arguments such as sep='\t'. Archives are decompressed before parsing. Pass fetch_file settings such as cache_root, timeout, expected_sha256, and progress_callback in a separate download_options dictionary. show_progress enables optional tqdm displays. The remaining keyword arguments are passed only to pandas.read_csv. """ options = dict(download_options or {}) options.setdefault("show_progress", show_progress) path = fetch_file( download_url=download_url, filename=filename, decompress=True, subdir=subdir, **options) import pandas as pd return pd.read_csv(path, **pandas_kwargs) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7066064 datacache-1.27.1/datacache/file_registry.py0000644000076500000000000001533515264220442020343 0ustar00iskanderwheel"""Fixed-path single-file registry for applications with established caches. Unlike bundle stores, this preserves legacy files and a root manifest. The caller owns trusted dataset definitions; writers serialize per root. Use bundles for transactional multi-file data. """ from __future__ import annotations import hashlib import json from datetime import datetime, timezone from pathlib import Path from filelock import FileLock from ._filesystem import write_json from .download import fetch_file class VersionedFileRegistry: """Download + cache for versioned, version-pinned external datasets. Parameters ---------- datasets Mapping of ``name -> spec`` where each spec has:: { "filename": "local_name.tsv", # name on disk (post-decompress) "urls": {"v23": "https://...zip", "latest": "https://..."}, "default_version": "v23", # used when caller passes version=None "description": "...", # optional, for status() } cache_dir Zero-arg callable returning the cache root :class:`~pathlib.Path` (created on demand by the caller). The on-disk layout is ``///`` plus a ``/manifest.json`` provenance file. error_cls Exception type raised for unknown datasets/versions and download failures. Defaults to :class:`RuntimeError`; consumers may pass their own subclass to preserve their public error type. """ def __init__(self, datasets, *, cache_dir, error_cls=RuntimeError): self._datasets = datasets self._cache_dir = cache_dir self._error_cls = error_cls # -- dataset / version resolution -- def _dataset(self, name: str) -> dict: try: return self._datasets[name] except KeyError: known = ", ".join(sorted(self._datasets)) raise self._error_cls(f"unknown dataset {name!r}; known: {known}") from None def resolve_version(self, name: str, version: str | None = None) -> str: """Return the concrete version for *name*, applying its default.""" spec = self._dataset(name) if version is None: version = spec["default_version"] if version not in spec["urls"]: avail = ", ".join(sorted(spec["urls"])) raise self._error_cls(f"{name!r} has no version {version!r}; available: {avail}") return version # -- cache paths / manifest -- def _manifest_path(self) -> Path: return Path(self._cache_dir()) / "manifest.json" @staticmethod def _read_manifest_at(path: Path) -> dict: try: manifest = json.loads(path.read_text()) return manifest if isinstance(manifest, dict) else {} except (json.JSONDecodeError, OSError): return {} def local_path(self, name: str, version: str | None = None) -> Path: """Expected cache path for *name*/*version* (may not exist yet).""" version = self.resolve_version(name, version) spec = self._dataset(name) return Path(self._cache_dir()) / name / version / spec["filename"] def is_cached(self, name: str, version: str | None = None) -> bool: return self.local_path(name, version).exists() # -- fetch -- def download( self, name: str, version: str | None = None, *, force: bool = False, **download_options ) -> Path: """Fetch a fixed path, serializing writers and forwarding fetch_file options. Ordinary cache hits neither hash nor write. Explicit integrity options validate cache hits. New bytes and the root receipt publish separately. """ version = self.resolve_version(name, version) spec = self._dataset(name) root = Path(self._cache_dir()) dest = root / name / version / spec["filename"] url = spec["urls"][version] def reuse(): if force or not dest.exists(): return False if any(download_options.get(key) is not None for key in ('expected_sha256', 'expected_size')): acquire() return True def acquire(): try: fetch_file(url, destination=dest, force=force, **download_options) except Exception as error: raise self._error_cls(f"failed to download {name} ({url}): {error}") from error if reuse(): return dest root.mkdir(parents=True, exist_ok=True) with FileLock(str(root / '.datacache-file-registry.lock')): if reuse(): return dest acquire() digest = hashlib.sha256() with dest.open("rb") as handle: for chunk in iter(lambda: handle.read(2 ** 20), b""): digest.update(chunk) manifest_path = root / 'manifest.json' manifest = self._read_manifest_at(manifest_path) manifest[name] = { "version": version, "url": url, "path": str(dest), "bytes": dest.stat().st_size, "sha256": digest.hexdigest(), "downloaded_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), } write_json(manifest_path, manifest) return dest def ensure(self, name: str, version: str | None = None, **download_options) -> Path: """Return a local path to *name*/*version*, downloading if absent.""" return self.download(name, version, **download_options) def status(self) -> list[dict]: """Return one status row per dataset (for a ``... list`` CLI command).""" root = Path(self._cache_dir()) manifest = self._read_manifest_at(root / "manifest.json") rows = [] for name, spec in sorted(self._datasets.items()): default_v = spec["default_version"] path = root / name / default_v / spec["filename"] record = manifest.get(name, {}) rows.append( { "name": name, "description": spec.get("description", ""), "default_version": default_v, "available_versions": sorted(spec["urls"]), "cached": path.exists(), "cached_version": record.get("version") if record else None, "url": record.get("url") if path.exists() else None, "bytes": record.get("bytes") if path.exists() else None, "sha256": record.get("sha256") if path.exists() else None, "downloaded_at": record.get("downloaded_at") if path.exists() else None, "path": str(path), } ) return rows ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792090401.706712 datacache-1.27.1/datacache/inspection.py0000644000076500000000000001361415264220442017645 0ustar00iskanderwheel"""Read-only presence and integrity checks; creation and repair belong to fetch.""" from dataclasses import dataclass import os from pathlib import PurePosixPath, PureWindowsPath import stat from typing import Dict, Optional from . import provenance from .integrity import FileValidationError, _validate_expectations, _validate_file @dataclass(frozen=True) class FileInspection: """File status: available, missing, corrupt, or inaccessible. available means readable, regular, and matching any supplied expectations. verified is True only when a supplied SHA-256 expectation was checked. error retains the validation or OS exception for unavailable files. size (bytes) and mtime (seconds since the epoch) describe an available file. source_url (without user name, password, query, or fragment), fetched_at (UTC, ISO 8601), and recorded_sha256 are what fetch_file recorded when it downloaded the file, reported only while the file's size and mtime are unchanged since. recorded_sha256 is the digest verified at that download; it is a record, not a check of the current bytes. None means unknown. """ path: str status: str verified: bool = False error: Optional[Exception] = None size: Optional[int] = None mtime: Optional[float] = None source_url: Optional[str] = None fetched_at: Optional[str] = None recorded_sha256: Optional[str] = None @dataclass(frozen=True) class CacheInspection: """Status of a directory and its required files, without manifest parsing. An absent root is missing; a present directory missing required files is corrupt/incomplete. files contains per-file results when the root is readable. verified requires supplied SHA-256 expectations for every required file. """ path: str status: str files: Dict[str, FileInspection] verified: bool = False error: Optional[Exception] = None def path_exists(path): """Check presence without creating directories or hiding permission errors. Presence is independent of file type, readability, and integrity. A broken symlink is absent; other OS errors propagate to the caller. """ try: os.stat(path) except FileNotFoundError: return False return True def inspect_file(path, expected_sha256=None, expected_size=None): """Report file availability without writes, network, locks, or recovery. Invalid expectation arguments raise ValueError. Filesystem and integrity problems are returned as distinct statuses with their original exceptions. An available file's size and mtime are reported, taken from the same open file that was validated, along with any provenance that fetch_file(..., record_provenance=True) recorded for those bytes. """ _validate_expectations(expected_sha256, expected_size) path = os.fspath(path) try: _, info = _validate_file(path, expected_sha256, expected_size) except FileNotFoundError as error: return FileInspection(path, "missing", error=error) except (FileValidationError, NotADirectoryError, IsADirectoryError) as error: return FileInspection(path, "corrupt", error=error) except OSError as error: return FileInspection(path, "inaccessible", error=error) record = provenance.read(path, info) or {} return FileInspection( path, "available", verified=expected_sha256 is not None, size=info.st_size, mtime=info.st_mtime, source_url=record.get("url"), fetched_at=record.get("fetched_at"), recorded_sha256=record.get("sha256")) def inspect_files(cache_root, files): """Inspect required relative files in a cache directory, entirely offline. files maps relative names to dicts containing expected_sha256/expected_size; use {} or None when integrity expectations are unavailable. Include a manifest as a required file to detect incomplete installations. This does not parse manifests or provide a snapshot across concurrent external edits. """ root = os.fspath(cache_root) if not files: raise ValueError("files must contain at least one required file") expectations = {} for name, metadata in files.items(): if (not isinstance(name, str) or not name or "\\" in name or ":" in name or PurePosixPath(name).is_absolute() or PureWindowsPath(name).drive or ".." in PurePosixPath(name).parts or "/".join(PurePosixPath(name).parts) != name): raise ValueError("Required file names must be normalized relative paths: %r" % name) metadata = {} if metadata is None else dict(metadata) if set(metadata) - {"expected_sha256", "expected_size"}: raise ValueError("Unexpected integrity metadata for %s" % name) _validate_expectations(metadata.get("expected_sha256"), metadata.get("expected_size")) expectations[name] = metadata try: info = os.stat(root) if not stat.S_ISDIR(info.st_mode): raise FileValidationError(root, "expected a cache directory") except FileNotFoundError as error: return CacheInspection(root, "missing", {}, error=error) except (FileValidationError, NotADirectoryError) as error: return CacheInspection(root, "corrupt", {}, error=error) except OSError as error: return CacheInspection(root, "inaccessible", {}, error=error) results = { name: inspect_file(os.path.join(root, name), **metadata) for name, metadata in expectations.items() } for status in ("inaccessible", "corrupt", "missing"): failed = next((result for result in results.values() if result.status == status), None) if failed is not None: return CacheInspection( root, "inaccessible" if status == "inaccessible" else "corrupt", results, error=failed.error) return CacheInspection(root, "available", results, verified=all(r.verified for r in results.values())) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7068129 datacache-1.27.1/datacache/integrity.py0000644000076500000000000000737115264220442017513 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import hashlib import os import re import stat from .progress import Progress class FileValidationError(ValueError): """A local file does not match its supplied integrity expectations.""" def __init__(self, path, reason): self.path = os.fspath(path) self.reason = reason super().__init__("%s: %s" % (self.path, reason)) def _validate_expectations(expected_sha256, expected_size): if expected_sha256 is not None and ( not isinstance(expected_sha256, str) or re.fullmatch(r"[0-9a-fA-F]{64}", expected_sha256) is None): raise ValueError("expected_sha256 must be a 64-character hexadecimal digest") if expected_size is not None and ( isinstance(expected_size, bool) or not isinstance(expected_size, int) or expected_size < 0): raise ValueError("expected_size must be a non-negative integer") def validate_file(path, expected_sha256=None, expected_size=None, *, show_progress=False): """Check a readable regular file without writes or network access. Expectations describe the bytes at ``path`` (after any decompression). Return the path as a string on success. Missing files raise ``FileNotFoundError``; permission errors propagate; non-regular files and size/hash mismatches raise ``FileValidationError``. Without expectations, this only checks that the file is readable and regular, not its integrity. show_progress=True displays optional tqdm progress during SHA-256 hashing. """ return _validate_file(path, expected_sha256, expected_size, show_progress=show_progress)[0] def _validate_file(path, expected_sha256=None, expected_size=None, *, show_progress=False): """validate_file, also returning the stat of the bytes that were checked. The stat comes from the open file, so callers reporting size or modification time describe exactly the bytes validated, even if the path is replaced. """ _validate_expectations(expected_sha256, expected_size) path = os.fspath(path) if not stat.S_ISREG(os.stat(path).st_mode): raise FileValidationError(path, "expected a regular file") with open(path, "rb") as source: info = os.fstat(source.fileno()) if not stat.S_ISREG(info.st_mode): raise FileValidationError(path, "expected a regular file") if expected_size is not None and info.st_size != expected_size: raise FileValidationError( path, "size mismatch: expected %d bytes, found %d" % ( expected_size, info.st_size)) if expected_sha256 is not None: digest = hashlib.sha256() with Progress(show_progress, "Verifying", info.st_size) as progress: completed = 0 for chunk in iter(lambda: source.read(2 ** 20), b""): digest.update(chunk) completed += len(chunk) progress(completed, info.st_size) actual_sha256 = digest.hexdigest() if actual_sha256 != expected_sha256.lower(): raise FileValidationError( path, "SHA-256 mismatch: expected %s, found %s" % ( expected_sha256.lower(), actual_sha256)) return path, info ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7069867 datacache-1.27.1/datacache/materialization.py0000644000076500000000000005145715264220442020675 0ustar00iskanderwheel"""Transactional caller-built artifacts with durable raw-input dependencies.""" from dataclasses import dataclass, field from datetime import datetime, timezone import hashlib import json import os from pathlib import Path from urllib.parse import urlsplit from urllib.request import url2pathname from uuid import uuid4 from ._filesystem import open_regular, path_present, read_json, write_json from .bundle_store import ( CHUNK_SIZE, INVALID_STORE_ERRORS, MANIFEST, BundleStore, check_expected, check_tree, discard_private_directory, hash_file, local_file_identity, private_directory, user_key, parent_directories, require_directory, validate_distinct_paths, validate_file_record, validate_no_sidecar_collisions, validate_relative_name, ) from .download import normal_creation_mode, validate_download_options, validate_size_within_limit from .integrity import FileValidationError, _validate_expectations from .progress import Progress from . import provenance INPUTS = '.datacache-inputs.json' FORMAT = 1 def _json_copy(value): # Reject non-JSON identities and NaNs, and detach from caller mutation. return json.loads(json.dumps(value, sort_keys=True, allow_nan=False)) def _fingerprint(value): return hashlib.sha256(json.dumps(value, sort_keys=True, allow_nan=False).encode()).hexdigest() def _inventory(mapping): if not isinstance(mapping, dict) or not mapping: raise ValueError('sources and outputs must be nonempty mappings') for name in mapping: validate_relative_name(name) validate_distinct_paths(mapping) return mapping def _expectations(spec): if not isinstance(spec, dict) or set(spec) - {'sha256', 'size'}: raise ValueError('output expectations must contain only sha256 and size') digest, size = spec.get('sha256'), spec.get('size') _validate_expectations(digest, size) return dict(sha256=digest.lower() if digest else None, size=size) def _transform(value): if (not isinstance(value, dict) or set(value) - {'version', 'options'} or not isinstance(value.get('version'), str) or not value['version']): raise ValueError('transform requires a nonempty version and optional JSON options') return dict(version=value['version'], options=_json_copy(value.get('options', {}))) def _definition(sources, transform, outputs): normalized, acquisition = {}, {} for name, spec in _inventory(sources).items(): if not isinstance(spec, dict) or set(spec) - {'url', 'path', 'sha256', 'size'}: raise ValueError('source metadata supports url or path, sha256 and size') if ('url' in spec) == ('path' in spec): raise ValueError('each source requires exactly one of url or path') expected = _expectations({key: spec[key] for key in ('sha256', 'size') if key in spec}) if 'path' in spec: origin = local_file_identity(spec['path']) acquisition[name] = ('path', str(Path(spec['path']).absolute())) else: origin = spec['url'] if not isinstance(origin, str) or not origin: raise ValueError('source URL must be a nonempty string') parts = urlsplit(origin) if parts.scheme not in ('http', 'https', 'ftp', 'file'): raise ValueError('source URL must use HTTP(S), FTP or file') if parts.scheme == 'file': if parts.netloc not in ('', 'localhost') or parts.query or parts.fragment: raise ValueError('local file URLs cannot have remote hosts, queries or fragments') acquisition[name] = ('path', str(Path(url2pathname(parts.path)).absolute())) else: acquisition[name] = ('url', origin) normalized[name] = dict(expected, identity=_fingerprint(origin), origin=provenance.redact_url(origin)) validate_no_sidecar_collisions(normalized) output_specs = {name: _expectations(spec) for name, spec in _inventory(outputs).items()} return dict(sources=normalized, transform=_transform(transform), outputs=output_specs), acquisition def _recorded_definition(value): if not isinstance(value, dict) or set(value) != {'sources', 'transform', 'outputs'}: raise ValueError('invalid materialization dependency definition') for spec in _inventory(value['sources']).values(): if not isinstance(spec, dict) or set(spec) != {'origin', 'identity', 'sha256', 'size'}: raise ValueError('invalid source dependency definition') _expectations({key: spec[key] for key in ('sha256', 'size')}) _validate_expectations(spec['identity'], None) if spec['identity'] is None or not isinstance(spec['origin'], str): raise ValueError('source identity and origin are required') for spec in _inventory(value['outputs']).values(): _expectations(spec) if _transform(value['transform']) != value['transform']: raise ValueError('invalid transform definition') return value def _source_records(value, definition): if not isinstance(value, dict) or set(value) != set(definition): raise ValueError('source receipt inventory disagrees with definition') for name, record in value.items(): if not isinstance(record, dict): raise ValueError('invalid source receipt') observed = {key: record.get(key) for key in ('sha256', 'size')} check_expected(name, validate_file_record(observed), definition[name]) trusted = definition[name]['sha256'] is not None if (record.get('identity') != definition[name]['identity'] or record.get('origin') != definition[name]['origin'] or record.get('verified') is not trusted or not isinstance(record.get('fetched_at'), str)): raise ValueError('source receipt identity or trust disagrees with definition') transport = record.get('transport') if transport is not None: from .resume import _valid_validator if not _valid_validator(transport): raise ValueError('invalid validated transport metadata') return value @dataclass(frozen=True) class MaterializationInspection: """Read-only status of the current bundle of built outputs, with its sources. status is available, missing, invalid or inaccessible. bundle is the current bundle's directory. verified refers only to outputs matched against caller-trusted hashes now. Source records distinguish acquisition-time trusted hashes from observation. """ path: str status: str verified: bool = False bundle: object = None files: dict = field(default_factory=dict) sources: dict = field(default_factory=dict) transform: object = None error: object = None def _check_bundle(store, directory, expected=None, *, verify_files=True): receipt = read_json(directory / MANIFEST, limit=None) if not isinstance(receipt, dict) or receipt.get('format') != FORMAT: raise ValueError('unrecognized materialization receipt') definition = _recorded_definition(receipt.get('definition')) if expected is not None and definition != expected: raise FileValidationError(directory, 'dependency identity changed; use force=True to refresh') sources = _source_records(receipt.get('sources'), definition['sources']) outputs = receipt.get('outputs') if not isinstance(outputs, dict) or set(outputs) != set(definition['outputs']): raise ValueError('output receipt inventory disagrees with definition') for name, record in outputs.items(): check_expected(name, validate_file_record(record), definition['outputs'][name]) trusted = expected['outputs'] if expected is not None else None hashed = check_tree(directory, outputs, parent_directories(outputs), ignore=(MANIFEST,), hash_contents=verify_files) # Only a hash the caller supplied verifies an output; the receipt's own doesn't. files = {name: value.inspection(verified=bool(verify_files and trusted and trusted[name]['sha256'])) for name, value in hashed.items()} verified = verify_files and trusted is not None and all(spec['sha256'] for spec in trusted.values()) return MaterializationInspection(str(store.path), 'available', bool(verified), str(directory), files, sources, definition['transform']) def inspect_materialization(destination, sources=None, *, transform=None, outputs=None, verify_files=True): """Check the current bundle offline, without writes, locks or source access. Supply all of sources/transform/outputs to check dependency identity; omit all three for receipt-only consistency checks. Metadata-only inspection checks regular files and sizes, never claims verification of payload hashes. """ if not isinstance(verify_files, bool): raise ValueError('verify_files must be a boolean') supplied = sum(value is not None for value in (sources, transform, outputs)) if supplied not in (0, 3): raise ValueError('provide sources, transform and outputs together, or omit all three') expected = _definition(sources, transform, outputs)[0] if supplied else None return _inspect_materialization(destination, expected, verify_files=verify_files) def _inspect_materialization(destination, expected, *, verify_files=True): path = Path(destination) store = BundleStore(path, 'materialization') try: bundle = store.current_bundle() if bundle is None: return MaterializationInspection(str(path), 'missing') return _check_bundle(store, bundle, expected, verify_files=verify_files) except PermissionError as error: return MaterializationInspection(str(path), 'inaccessible', error=error) except INVALID_STORE_ERRORS as error: return MaterializationInspection(str(path), 'invalid', error=error) def _input_directory(store, definition): return store / ('.inputs-%s-%s' % (_fingerprint(definition['sources']), user_key())) def _cleanup_inputs(store, definition, retain_sources): if not retain_sources: discard_private_directory(_input_directory(store, definition)) def _acquire_inputs(directory, definition, acquisition, options, *, force=False): from .download import fetch_file private_directory(directory) receipt_path = directory / INPUTS try: records = read_json(receipt_path, limit=None) if path_present(receipt_path) else {} if not isinstance(records, dict) or set(records) - set(definition): raise FileValidationError(receipt_path, 'invalid private input receipt') except (ValueError, RecursionError) as error: if not force: raise FileValidationError( receipt_path, 'invalid private input receipt; use force=True to repair') from error # Unrecorded inputs are checked or acquired again below. records = {} write_json(receipt_path, records) paths = {} for name, spec in definition.items(): target = directory / name for parent in reversed(target.relative_to(directory).parents): current = directory / parent if not path_present(current): current.mkdir(mode=0o700) require_directory(current) kind, source = acquisition[name] if force and kind == 'path' and name in records and path_present(Path(source)): # The caller may have corrected a local source in place without # changing its path, so a forced refresh copies it again while the # original exists; once it's gone, the retained copy still serves. del records[name] write_json(receipt_path, records) if path_present(target): os.close(open_regular(target)) # Never remove a planted link/special file. target.unlink() provenance.remove(target) if name in records: try: _source_records({name: records[name]}, {name: spec}) hash_file(target, records[name]) except (FileNotFoundError, ValueError) as error: if not force: raise FileValidationError( target, 'invalid private input; use force=True to repair') from error if path_present(target): fd = open_regular(target) # Never repair a planted link/special file. os.close(fd) target.unlink() provenance.remove(target) del records[name] write_json(receipt_path, records) else: paths[name] = str(target) continue if path_present(target): # Only installer-created, private completed bytes can exist here. # Validate their type before fetch_file's ordinary cache-hit path. fd = open_regular(target) os.close(fd) # A hash-pinned acquisition is independently reusable. A bare # size-only/unverified file has no durable completion receipt yet. if spec['sha256'] is None: target.unlink() else: try: hash_file(target, spec) except FileValidationError: if not force: raise FileValidationError(target, 'invalid private input; use force=True to repair') target.unlink() provenance.remove(target) copied = None if kind == 'url': fetch_file(source, destination=target, raw=True, record_provenance=True, expected_sha256=spec['sha256'], expected_size=spec['size'], **options) elif not path_present(target): temporary = directory / ('.copy-' + uuid4().hex) try: with os.fdopen(open_regular(source), 'rb') as handle, temporary.open('xb') as output: digest = hashlib.sha256() with Progress(options['show_progress'], 'Copying', os.fstat(handle.fileno()).st_size) as progress: size = 0 for chunk in iter(lambda: handle.read(options.get('chunk_size', CHUNK_SIZE)), b''): output.write(chunk) digest.update(chunk) # Hash while copying: one read. size += len(chunk) progress(size, progress.total) if options.get('progress_callback') is not None: options['progress_callback'](size, progress.total) output.flush() os.fsync(output.fileno()) copied = dict(sha256=digest.hexdigest(), size=size) check_expected(temporary, copied, spec) os.replace(temporary, target) finally: temporary.unlink(missing_ok=True) if copied is not None: observed, info = copied, os.stat(target) else: hashed = hash_file(target, spec, show_progress=options['show_progress']) observed, info = hashed.record, hashed.info origin_record = provenance.read(target, info) or {} record = dict(observed, origin=spec['origin'], identity=spec['identity'], verified=spec['sha256'] is not None, fetched_at=origin_record.get('fetched_at', datetime.now(timezone.utc).isoformat())) if origin_record.get('transport') is not None: record['transport'] = origin_record['transport'] records[name] = record write_json(receipt_path, records) paths[name] = str(target) return paths, records def materialize(destination, sources, *, transform, outputs, builder, force=False, retain_sources=False, verify_files=True, download_options=None): """Build all declared outputs, then publish them together as a new bundle. builder(source_paths, output_paths) must create/close exactly the declared outputs and leave inputs unchanged. Source definitions contain url or path plus optional raw sha256/size; outputs map names to optional sha256/size. transform is {version: nonempty string, options: opaque JSON value}. Changed dependencies and a corrupt bundle require explicit force=True. Force rebuilds outputs while reusing any complete matching private inputs. Inputs survive failures, including without HTTP resume. resume=True enables existing raw HTTP resume rules for remote sources; local inputs are copied. Successful publication removes owned inputs unless retain_sources=True. Caller-owned inputs and old bundles are never deleted. Cache hits are offline and read-only; verify_files=False skips payload hashing only on reuse. New outputs are always hashed before publication. Download/copy/verification progress is on by default; builder progress is caller-owned. Installation needs atomic local renames, on Linux, macOS and Windows; resume=True needs POSIX. """ if not all(isinstance(value, bool) for value in (force, retain_sources, verify_files)): raise ValueError('force, retain_sources and verify_files must be booleans') if not callable(builder): raise ValueError('builder must be callable') definition, acquisition = _definition(sources, transform, outputs) options = validate_download_options(download_options, 'materialization') options.setdefault('show_progress', True) for name, (kind, source) in acquisition.items(): if kind == 'url': validate_size_within_limit(definition['sources'][name]['size'], options.get('max_bytes'), '%s size' % name) if options.get('resume') is True: # "auto" downloads normally where it can't resume. from .resume import validate_resume for name, (kind, source) in acquisition.items(): if kind == 'url': validate_resume(source, definition['sources'][name]['sha256'], definition['sources'][name]['size']) path = Path(destination).absolute() store = BundleStore(path, 'materialization') def inspect_current(): # A forced refresh rebuilds whatever is there; it only needs to know # whether the store is readable, not to hash the old outputs. return _inspect_materialization(path, definition, verify_files=verify_files and not force) inspection = inspect_current() if store.can_reuse(inspection, force=force): return {name: value.path for name, value in inspection.files.items()} path.parent.mkdir(parents=True, exist_ok=True) with store.lock(): store.create() inspection = inspect_current() if store.can_reuse(inspection, force=force): return {name: value.path for name, value in inspection.files.items()} inputs = _input_directory(path, definition) source_paths, records = _acquire_inputs(inputs, definition['sources'], acquisition, options, force=force) working = path / ('.staging-%s-%s' % (_fingerprint(definition), user_key())) # Start from nothing: discard this user's own outputs from a failed build. discard_private_directory(working) private_directory(working) staged = working / 'files' staged.mkdir() output_paths = {name: str(staged / name) for name in definition['outputs']} for target in output_paths.values(): Path(target).parent.mkdir(parents=True, exist_ok=True) try: builder(dict(source_paths), dict(output_paths)) # Builder callbacks are trusted code, not a sandbox; enforce their # read-only-input contract before creating a dependency receipt. for name, source in source_paths.items(): hash_file(source, records[name]) # Reject extras/links/specials before hashing any declared output. declared = definition['outputs'] check_tree(staged, declared, parent_directories(declared), hash_contents=False) mode = normal_creation_mode(staged) observed = {} for name, target in output_paths.items(): observed[name] = hash_file(target, declared[name], show_progress=options['show_progress']).record os.chmod(target, mode) receipt = dict(format=FORMAT, created_at=datetime.now(timezone.utc).isoformat(), definition=definition, sources=records, outputs=observed) write_json(staged / MANIFEST, receipt, mode=mode) # The outputs were just hashed into the manifest; check its # structure, inventory and sizes without hashing them again. _check_bundle(store, staged, definition, verify_files=False) bundle = store.publish(staged) _cleanup_inputs(path, definition, retain_sources) return {name: str(bundle / name) for name in declared} finally: discard_private_directory(working) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7070823 datacache-1.27.1/datacache/permissions.py0000644000076500000000000000451015264220442020040 0ustar00iskanderwheel"""Explicit, narrowly scoped maintenance of shared-cache access.""" import os import stat from . import provenance from .integrity import FileValidationError def make_file_readable(path, *, group=True, others=False): """Add read access to one existing regular file, without replacing it. On POSIX, its owner can grant group read access (default), or explicitly grant others read access. No write/execute bits are added or removed, and contents are unchanged. Symlinks and non-regular files are rejected. Downloads and cache hits never call this function automatically. The file's provenance record, if any, gets the same access, best effort. Return the path on success. Permission errors propagate: this cannot grant access to a file the caller cannot open or chmod. Parent directories and owning groups must already permit the intended readers to reach the file. """ if not isinstance(group, bool) or not isinstance(others, bool): raise ValueError("group and others must be booleans") if not hasattr(os, "fchmod") or not hasattr(os, "O_NOFOLLOW"): raise NotImplementedError("Explicit shared-file permissions require POSIX file descriptors") path = os.fspath(path) _add_read_access(path, group, others) try: _add_read_access(provenance.sidecar_path(path), group, others) except (OSError, FileValidationError): pass # no record, or one that is not ours to share return path def _add_read_access(path, group, others): if not stat.S_ISREG(os.lstat(path).st_mode): raise FileValidationError(path, "expected a regular file, not a symlink or directory") # Operate on the opened inode rather than a path that could be swapped to a # symlink between checking and chmod. Nonblocking open also avoids a FIFO # substituted after lstat hanging this maintenance operation. descriptor = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK) try: info = os.fstat(descriptor) if not stat.S_ISREG(info.st_mode): raise FileValidationError(path, "expected a regular file") old_mode = stat.S_IMODE(info.st_mode) mode = old_mode | (stat.S_IRGRP if group else 0) | (stat.S_IROTH if others else 0) if mode != old_mode: os.fchmod(descriptor, mode) finally: os.close(descriptor) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7071726 datacache-1.27.1/datacache/progress.py0000644000076500000000000000241715264220442017335 0ustar00iskanderwheel"""Opt-in progress displays using the required, lazily imported tqdm dependency.""" class Progress: """Own one tqdm bar for one operation or download attempt.""" def __init__(self, enabled=False, description="", total=None, unit="B"): self.enabled = enabled self.description = description self.total = total self.unit = unit self.bar = None def __enter__(self): if self.enabled: try: from tqdm.auto import tqdm except ImportError as error: raise ImportError( "tqdm is a required datacache dependency; install 'tqdm>=4.64' " "or reinstall datacache") from error self.bar = tqdm( desc=self.description, total=self.total, unit=self.unit, unit_scale=self.unit == "B", unit_divisor=1024, dynamic_ncols=True, leave=False) return self def __call__(self, completed, total): if self.bar is not None: if completed < self.bar.n: self.bar.reset(total=total) self.bar.total = total self.bar.update(completed - self.bar.n) def __exit__(self, *exc): if self.bar is not None: self.bar.close() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7072704 datacache-1.27.1/datacache/provenance.py0000644000076500000000000001270215264220442017627 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Where a downloaded file came from, recorded beside it for offline inspection. fetch_file(..., record_provenance=True) writes a small hidden record, "..datacache.json", beside the file it publishes. Any later replacement of the file removes the record first, so a record never describes bytes it was not written for. Inspection only reads records: a missing, unreadable, or malformed record means no recorded provenance. """ from datetime import datetime, timezone import errno import json import os import re import stat from urllib.parse import urlsplit, urlunsplit SUFFIX = ".datacache.json" FORMAT = 1 # A record is a few hundred bytes; anything much larger is not one of ours. _MAX_RECORD_BYTES = 64 * 1024 # Never block on a FIFO or follow a link planted at a record's name. _OPEN_FLAGS = (os.O_RDONLY | getattr(os, "O_BINARY", 0) | getattr(os, "O_NONBLOCK", 0) | getattr(os, "O_NOFOLLOW", 0)) def sidecar_path(path): """Path of the provenance record kept for the file at path (str or bytes).""" directory, name = os.path.split(os.fspath(path)) if isinstance(name, bytes): return os.path.join(directory, b"." + name + os.fsencode(SUFFIX)) return os.path.join(directory, "." + name + SUFFIX) def redact_url(url): """Drop the user name, password, query string, and fragment. Those are where URLs usually carry secrets, such as signed-URL signatures. The path is kept as is, so a secret embedded in the path, as in some share links, would be recorded. """ parts = urlsplit(url) host = parts.hostname or "" if ":" in host: host = "[%s]" % host # IPv6 literal if parts.port is not None: host = "%s:%d" % (host, parts.port) return urlunsplit((parts.scheme, host, parts.path, "", "")) def describe(download_url, info, sha256=None, *, transport=None): """Provenance, as JSON text, for the bytes that stat info describes. info must come from the exact bytes being published (the staged file), not from whatever is at the destination afterwards, which a concurrent writer may have replaced. sha256 is recorded only when those bytes were verified against a trusted expectation: a digest of what a server sent proves nothing. transport optionally records the validator accepted by resumable transport; it is evidence of representation consistency, not a trusted content hash. """ record = { "format": FORMAT, "url": redact_url(download_url), "fetched_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), "size": info.st_size, "mtime_ns": info.st_mtime_ns, "sha256": sha256.lower() if sha256 else None, } if transport is not None: record['transport'] = transport return json.dumps(record, sort_keys=True) def remove(path): """Remove the record kept for path, if there is one. A record name longer than the filesystem allows cannot exist, so it counts as absent. Other errors propagate. """ try: os.remove(sidecar_path(path)) except FileNotFoundError: pass except OSError as error: if error.errno != errno.ENAMETOOLONG: raise def _valid(record): def integer(value): return isinstance(value, int) and not isinstance(value, bool) and value >= 0 return (isinstance(record, dict) and record.get("format") == FORMAT and isinstance(record.get("url"), str) and isinstance(record.get("fetched_at"), str) and integer(record.get("size")) and integer(record.get("mtime_ns")) and (record.get("sha256") is None or (isinstance(record.get("sha256"), str) and re.fullmatch(r"[0-9a-f]{64}", record["sha256"])))) def read(path, info): """The record for path if it still describes the file with stat info. Returns None when there is no usable record, or when the file's size or modification time changed since it was recorded. Never writes and never blocks; any problem with the record means no recorded provenance. """ try: descriptor = os.open(sidecar_path(path), _OPEN_FLAGS) except OSError: return None try: # Check the opened object itself, so it cannot be swapped after a check. record_info = os.fstat(descriptor) if not stat.S_ISREG(record_info.st_mode) or record_info.st_size > _MAX_RECORD_BYTES: return None chunks, remaining = [], _MAX_RECORD_BYTES + 1 while remaining > 0: chunk = os.read(descriptor, remaining) if not chunk: break chunks.append(chunk) remaining -= len(chunk) record = json.loads(b"".join(chunks)) except (OSError, ValueError, RecursionError): return None finally: os.close(descriptor) if not _valid(record) or record["size"] != info.st_size or record["mtime_ns"] != info.st_mtime_ns: return None return record ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7073662 datacache-1.27.1/datacache/resume.py0000644000076500000000000003514015264220442016770 0ustar00iskanderwheel"""Hash- or strong-ETag-validated HTTP downloads with private partials.""" from contextlib import closing, contextmanager import hashlib import logging import os from pathlib import Path import re import stat import time from urllib.parse import urlsplit from ._filesystem import file_lock, open_regular, path_present, read_json, write_json from .common import USER_AGENT from .integrity import FileValidationError, _validate_expectations, validate_file from .progress import Progress from .retries import error_description, is_retryable_http_error, retry_delay logger = logging.getLogger(__name__) class CannotResume(FileValidationError): """A resumable transfer isn't possible here, though a normal one may be. The server may send no strong ETag when no trusted SHA-256 was given, encode the response, or answer ranges inconsistently; or the destination's filesystem can't keep private resume state or locks. fetch_file(resume= "auto") then downloads the file in full instead. """ def can_resume(download_url, expected_sha256, expected_size): """Whether resume=True would accept these arguments; see validate_resume.""" try: validate_resume(download_url, expected_sha256, expected_size) except (ValueError, NotImplementedError): return False return True def validate_resume(download_url, expected_sha256, expected_size): _validate_expectations(expected_sha256, expected_size) if expected_size is None: raise ValueError('resume=True requires expected_size') if urlsplit(download_url).scheme.lower() not in ('http', 'https'): raise ValueError('resume=True supports only raw HTTP/HTTPS downloads') if os.name != 'posix': raise NotImplementedError('Resumable downloads require a POSIX local filesystem') def _state_directory(destination): destination = Path(destination) key = hashlib.sha256(os.fsencode(destination.name)).hexdigest()[:32] return destination.parent / ('.datacache-resume-%d-%s' % (os.getuid(), key)) def _prepare_directory(destination): directory = _state_directory(destination) directory.mkdir(mode=0o700, exist_ok=True) info = directory.lstat() if (not stat.S_ISDIR(info.st_mode) or info.st_uid != os.getuid() or stat.S_IMODE(info.st_mode) & 0o077): raise CannotResume(directory, 'resume directory must be owner-only and not a link') return directory @contextmanager def _state_lock(directory): """The private lock on resume state; CannotResume if this filesystem can't hold it.""" try: lock = file_lock(directory / 'lock', private=True) lock.__enter__() except (OSError, FileValidationError) as error: raise CannotResume(directory, 'cannot lock resume state on this filesystem: %s' % error) from error try: yield finally: lock.__exit__(None, None, None) def discard_partial(destination): """Discard this user's resumable bytes for an exact destination, under lock. Does not change the installed file. Missing state is a no-op. The small private directory and permanent lock remain to coordinate concurrent calls. """ if os.name != 'posix': raise NotImplementedError('Resumable downloads require a POSIX local filesystem') directory = _state_directory(destination) if not path_present(directory): return directory = _prepare_directory(destination) with file_lock(directory / 'lock', private=True): for name in ('partial', 'metadata.json'): (directory / name).unlink(missing_ok=True) def _valid_validator(validator): """Whether a recorded transport validator is a usable If-Range value: {header: ETag (strong) or Last-Modified, value: one header line}.""" return ( isinstance(validator, dict) and set(validator) == {'header', 'value'} and validator['header'] in ('ETag', 'Last-Modified') and isinstance(validator['value'], str) and '\r' not in validator['value'] and '\n' not in validator['value'] and (validator['header'] != 'ETag' or _strong_etag(validator['value']))) def _strong_etag(value): # RFC 9110 entity-tag syntax, without the weak W/ prefix. Embedded quotes, # whitespace, and control characters cannot authorize an If-Range request. return isinstance(value, str) and re.fullmatch(r'"[\x21\x23-\x7e\x80-\xff]*"', value) is not None def _validator(headers): etag = headers.get('ETag', '') if _strong_etag(etag): return {'header': 'ETag', 'value': etag} # Last-Modified is not always strong, but changes still invalidate a partial. modified = headers.get('Last-Modified') return {'header': 'Last-Modified', 'value': modified} if modified else None def download_resumable(download_url, destination, *, expected_sha256, expected_size, timeout, chunk_size, progress_callback, show_progress, max_retries, retry_backoff, retry_max_delay, record_provenance, force): from . import download, provenance import requests directory = _prepare_directory(destination) partial = directory / 'partial' metadata_path = directory / 'metadata.json' identity = {'sha256': expected_sha256.lower() if expected_sha256 is not None else None, 'size': expected_size, 'url_hash': hashlib.sha256(download_url.encode()).hexdigest()} with _state_lock(directory): # Another installer may have completed while we waited for its lock. if not force: try: validate_file(destination, expected_sha256, expected_size) except FileNotFoundError: pass else: return try: metadata = read_json(metadata_path) except (OSError, ValueError, RecursionError): metadata = {} if not isinstance(metadata, dict) or metadata.get('identity') != identity: metadata = {'identity': identity, 'validator': None} # Unlink, never truncate an unknown path planted in the state dir. partial.unlink(missing_ok=True) validator = metadata.get('validator') valid_validator = _valid_validator(validator) strong_validator = (valid_validator and validator['header'] == 'ETag') if ((validator is not None and not valid_validator) or (expected_sha256 is None and not strong_validator)): metadata['validator'] = None partial.unlink(missing_ok=True) write_json(metadata_path, metadata) try: fd = open_regular(partial, os.O_RDWR | os.O_CREAT, private=True) except FileValidationError as error: raise CannotResume(partial, error.reason) from error with os.fdopen(fd, 'r+b') as output: backoff = retry_backoff with Progress(show_progress, 'Downloading', expected_size) as progress: def report(): done = output.tell() progress(done, expected_size) if progress_callback is not None: progress_callback(done, expected_size) for attempt in range(max_retries + 1): output.seek(0, os.SEEK_END) offset = output.tell() if offset >= expected_size: output.flush() if expected_sha256 is not None: try: validate_file(partial, expected_sha256, expected_size) except FileValidationError: pass else: report() break # Size alone cannot validate a completed persistent # partial, including an empty one. Request fresh bytes. output.seek(0) output.truncate() offset = 0 metadata['validator'] = None callback_failed = False try: # One protocol restart per attempt, independent of retries. for restart in range(2): headers = {'Accept-Encoding': 'identity', 'User-Agent': USER_AGENT} validator = metadata.get('validator') if offset: headers['Range'] = 'bytes=%d-' % offset if validator and validator['header'] == 'ETag': headers['If-Range'] = validator['value'] with closing(requests.get(download_url, headers=headers, timeout=timeout, stream=True)) as response: if offset and response.status_code == 416: output.seek(0) output.truncate() offset = 0 metadata['validator'] = None continue response.raise_for_status() if response.headers.get('Content-Encoding', 'identity').lower() != 'identity': raise CannotResume(destination, 'resume requires identity Content-Encoding') current_validator = _validator(response.headers) if response.status_code == 206: match = re.fullmatch(r'bytes (\d+)-(\d+)/(\d+)', response.headers.get('Content-Range', '')) valid_range = (match is not None and tuple(map(int, match.groups())) == (offset, expected_size - 1, expected_size)) same_identity = not offset or not validator or validator == current_validator if not valid_range or not same_identity: output.seek(0) output.truncate() offset = 0 metadata['validator'] = None if restart == 0: continue raise CannotResume(destination, 'incompatible HTTP Content-Range or validator') elif response.status_code == 200: output.seek(0) output.truncate() offset = 0 else: raise CannotResume(destination, 'expected HTTP 200 or 206') if expected_sha256 is None and ( current_validator is None or current_validator['header'] != 'ETag'): raise CannotResume( destination, 'resume without expected_sha256 requires a strong ETag') metadata['validator'] = current_validator write_json(metadata_path, metadata) for chunk in response.iter_content(chunk_size=chunk_size): if not chunk: continue if output.tell() + len(chunk) > expected_size: output.seek(0) output.truncate() raise FileValidationError(destination, 'download exceeds expected_size') output.write(chunk) try: report() except BaseException: callback_failed = True raise if output.tell() != expected_size: raise requests.ConnectionError('incomplete resumable response') break else: raise CannotResume(destination, 'server could not restart the download') output.flush() try: validate_file(partial, expected_sha256, expected_size, show_progress=show_progress) except FileValidationError: output.seek(0) output.truncate() raise break except BaseException as error: output.flush() if callback_failed or not is_retryable_http_error(error) or attempt == max_retries: raise delay = retry_delay(error, backoff, retry_max_delay) if delay is None: raise logger.warning('Resumable HTTP attempt %d/%d failed (%s); retrying in %.3g seconds', attempt + 1, max_retries + 1, error_description(error), delay) if delay: time.sleep(delay) backoff *= 2 # retry_delay caps it at retry_max_delay. output.flush() os.fsync(output.fileno()) # Keep the resumable inode private even if publication fails after chmod. # Copy into ordinary sibling staging; never expose persistent partials. staged = None try: with download._open_staging_file(directory=Path(destination).parent) as handle: staged = handle.name with partial.open('rb') as source: download.copyfileobj(source, handle) record = (provenance.describe(download_url, os.stat(staged), expected_sha256, transport=metadata.get('validator')) if record_provenance else None) download._publish_file(staged, destination, record) finally: if staged is not None: download._remove_staging_file(staged) partial.unlink() metadata_path.unlink(missing_ok=True) ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792090401.707461 datacache-1.27.1/datacache/retries.py0000644000076500000000000001014315264220442017141 0ustar00iskanderwheel"""Bounded retry policy for idempotent HTTP downloads.""" from datetime import timezone from email.utils import parsedate_to_datetime import math from numbers import Real import ssl import time DEFAULT_MAX_RETRIES = 2 DEFAULT_RETRY_BACKOFF = 1.0 DEFAULT_RETRY_MAX_DELAY = 30.0 RETRYABLE_HTTP_STATUSES = frozenset((408, 429, 500, 502, 503, 504)) def validate_retry_options(max_retries, retry_backoff, retry_max_delay): """Validate settings and return built-in float delays for arithmetic/sleep.""" if isinstance(max_retries, bool) or not isinstance(max_retries, int) or max_retries < 0: raise ValueError("max_retries must be a non-negative integer") delays = [] for name, value in (("retry_backoff", retry_backoff), ("retry_max_delay", retry_max_delay)): message = "%s must be a finite non-negative number" % name if isinstance(value, bool) or not isinstance(value, Real) or value < 0: raise ValueError(message) try: delay = float(value) except (OverflowError, TypeError, ValueError) as error: raise ValueError(message) from error if not math.isfinite(delay): raise ValueError(message) delays.append(delay) return tuple(delays) def _contains_tls_error(error): """Follow active causes and Requests/urllib3 wrappers, guarding cycles.""" import requests from urllib3.exceptions import SSLError as Urllib3SSLError pending, seen = [error], set() while pending: current = pending.pop() if not isinstance(current, BaseException) or id(current) in seen: continue seen.add(id(current)) if isinstance(current, (ssl.SSLError, requests.exceptions.SSLError, Urllib3SSLError)): return True pending.extend(current.args) pending.extend((getattr(current, "reason", None), getattr(current, "original_error", None))) if current.__cause__ is not None: pending.append(current.__cause__) elif not current.__suppress_context__: pending.append(current.__context__) return False def is_retryable_http_error(error): import requests if isinstance(error, requests.HTTPError): return error.response is not None and error.response.status_code in RETRYABLE_HTTP_STATUSES transient = isinstance(error, ( requests.ConnectionError, requests.Timeout, requests.exceptions.ChunkedEncodingError)) return transient and not _contains_tls_error(error) def retry_delay(error, backoff, max_delay): """Return a delay, or None if Retry-After exceeds the caller's wait limit. backoff is the uncapped exponential delay; this is the one place it is capped at max_delay, so retry loops simply double it. Invalid Retry-After values are ignored. A server's minimum wait is never shortened to fit the cap: in that case the original failure must propagate. """ response = getattr(error, "response", None) header = response.headers.get("Retry-After") if response is not None else None retry_after = 0 if header: header = header.strip() try: if header.isascii() and header.isdigit(): digits = header.lstrip("0") or "0" # Avoid integer-conversion limits for an enormous server wait. if len(digits) > len(str(int(max_delay))): return None retry_after = int(digits) else: date = parsedate_to_datetime(header) if date.tzinfo is None: date = date.replace(tzinfo=timezone.utc) retry_after = max(0, date.timestamp() - time.time()) except (TypeError, ValueError, OverflowError, OSError): pass if retry_after > max_delay: return None return float(max(min(backoff, max_delay), retry_after)) def error_description(error): """Describe a failure without adding credentials or URL query text to logs.""" import requests if isinstance(error, requests.HTTPError) and error.response is not None: return "HTTP %s" % error.response.status_code return type(error).__name__ ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092720.7518952 datacache-1.27.1/datacache/version.py0000644000076500000000000000002715264225061017153 0ustar00iskanderwheel__version__ = "1.27.1" ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092720.7573152 datacache-1.27.1/datacache/zip_members.py0000644000076500000000000004101415264225061020003 0ustar00iskanderwheel"""Bounded HTTP ZIP member acquisition, stored as an atomic evidence bundle.""" from datetime import datetime, timezone from email.utils import parsedate_to_datetime import io import os from pathlib import Path import re import struct from urllib.parse import urlsplit from uuid import uuid4 import zipfile import zlib from ._filesystem import read_json, write_json from .bundle_store import ( BundleStore, CHUNK_SIZE, INVALID_STORE_ERRORS, MANIFEST, check_expected, check_tree, discard_private_directory, hash_file, private_directory, source_fingerprint, validate_file_record, validate_relative_name, ) from .bundles import BundleInspection from .download import normal_creation_mode from .integrity import _validate_expectations from .provenance import redact_url from .resume import _strong_etag def _positive(value, name): if isinstance(value, bool) or not isinstance(value, int) or value <= 0: raise ValueError("%s must be a positive integer" % name) return value class _HTTPFile(io.RawIOBase): """Seekable metadata reads; data transfers explicitly stream one range.""" def __init__(self, url, timeout, max_metadata_bytes): import requests self.url = url self.timeout = timeout self.max_metadata_bytes = max_metadata_bytes self.metadata_bytes = 0 self.position = 0 self.size = None self.validator = None self.session = requests.Session() try: self.read(1) except BaseException: self.close() raise self.position = 0 def close(self): self.session.close() super().close() def seekable(self): return True def readable(self): return True def tell(self): return self.position def seek(self, offset, whence=0): if whence not in (0, 1, 2): raise ValueError("invalid seek origin") position = offset + ( self.position if whence == 1 else self.size if whence == 2 else 0 ) if position < 0: raise ValueError("negative seek") self.position = position return position def chunks(self, start, size): if not size: return end = start + size - 1 headers = {"Range": "bytes=%d-%d" % (start, end), "Accept-Encoding": "identity"} if self.validator: headers["If-Range"] = self.validator["value"] with self.session.get( self.url, headers=headers, stream=True, timeout=self.timeout ) as response: if response.status_code != 206: raise ValueError("ZIP source must honor byte ranges with HTTP 206") if ( response.headers.get("Content-Encoding", "identity").lower() != "identity" ): raise ValueError("encoded range responses are not supported") match = re.fullmatch( r"bytes (\d+)-(\d+)/(\d+)", response.headers.get("Content-Range", "") ) if not match or tuple(map(int, match.groups()[:2])) != (start, end): raise ValueError("incorrect HTTP Content-Range") total = int(match.group(3)) if total <= end or (self.size is not None and self.size != total): raise ValueError("archive length changed or range exceeds archive") length = response.headers.get("Content-Length") if length is not None and length != str(size): raise ValueError("incorrect HTTP Content-Length") if self.validator is None: etag = response.headers.get("ETag") modified = response.headers.get("Last-Modified") if _strong_etag(etag): validator = {"header": "ETag", "value": etag} elif modified: # Last-Modified is a transport consistency check, not a hash. if parsedate_to_datetime(modified).tzinfo is None: raise ValueError("Last-Modified must include a timezone") validator = {"header": "Last-Modified", "value": modified} else: raise ValueError( "ZIP source requires a strong ETag or Last-Modified" ) self.validator = validator elif ( response.headers.get(self.validator["header"]) != self.validator["value"] ): raise ValueError("archive validator changed between ranges") self.size = total remaining = size while remaining: chunk = response.raw.read( min(CHUNK_SIZE, remaining), decode_content=False ) if not chunk: raise ValueError("truncated HTTP range") remaining -= len(chunk) yield chunk if response.raw.read(1, decode_content=False): raise ValueError("HTTP range exceeds requested size") def read(self, size=-1): if self.size is not None: size = ( max(0, self.size - self.position) if size < 0 else min(size, max(0, self.size - self.position)) ) if size < 0 or self.metadata_bytes + size > self.max_metadata_bytes: raise ValueError("ZIP metadata exceeds max_metadata_bytes") self.metadata_bytes += size result = b"".join(self.chunks(self.position, size)) self.position += len(result) return result def _definition(url, member, expected_sha256, expected_size, compressed_sha256): if not isinstance(url, str) or urlsplit(url).scheme not in ("http", "https"): raise ValueError("ZIP source must be an HTTP(S) URL") validate_relative_name(member) _validate_expectations(expected_sha256, expected_size) _validate_expectations(compressed_sha256, None) return dict( url=redact_url(url), url_identity=source_fingerprint(url), member=member, sha256=expected_sha256.lower() if expected_sha256 else None, size=expected_size, compressed_sha256=compressed_sha256.lower() if compressed_sha256 else None, ) def _inspect(destination, definition, max_compressed_bytes, max_bytes): store = BundleStore(destination, "zip-member") try: directory = store.current_bundle() if directory is None: return BundleInspection(str(destination), "missing") manifest = read_json(directory / MANIFEST) if manifest.get("format") != 1 or manifest.get("definition") != definition: raise ValueError("ZIP member dependency identity changed") files = manifest.get("files") if not isinstance(files, dict) or set(files) != { "payload", "compressed", "receipt.json", }: raise ValueError("invalid ZIP member inventory") for record in files.values(): validate_file_record(record) if ( files["payload"]["size"] > max_bytes or files["compressed"]["size"] > max_compressed_bytes ): raise ValueError("cached ZIP member exceeds requested size limits") check_expected(directory, files["payload"], definition) check_expected( directory, files["compressed"], {"sha256": definition["compressed_sha256"]} ) hashed = check_tree(directory, files, [], ignore=(MANIFEST,)) receipt = read_json(directory / "receipt.json") if ( receipt.get("definition") != definition or receipt.get("payload") != files["payload"] or receipt.get("compressed") != files["compressed"] ): raise ValueError("ZIP member acquisition receipt disagrees with inventory") return BundleInspection( str(destination), "available", bool(definition["sha256"]), str(directory), { name: value.inspection( verified=bool(definition["sha256"]) and name == "payload" ) for name, value in hashed.items() }, ) except PermissionError as error: return BundleInspection(str(destination), "inaccessible", error=error) except INVALID_STORE_ERRORS as error: return BundleInspection(str(destination), "invalid", error=error) def _extract(remote, info, offset, paths, max_bytes): """Retain exact compressed bytes and validate the complete deflate stream.""" decompressor = ( zlib.decompressobj(-15) if info.compress_type == zipfile.ZIP_DEFLATED else None ) crc, size = 0, 0 with ( open(paths["payload"], "xb") as output, open(paths["compressed"], "xb") as compressed, ): def write(chunk): nonlocal crc, size size += len(chunk) if size > min(max_bytes, info.file_size): raise ValueError( "expanded ZIP member exceeds declared size or max_bytes" ) crc = zlib.crc32(chunk, crc) output.write(chunk) for chunk in remote.chunks(offset, info.compress_size): compressed.write(chunk) if decompressor is None: write(chunk) else: while True: expanded = decompressor.decompress(chunk, CHUNK_SIZE) write(expanded) chunk = decompressor.unconsumed_tail if decompressor.unused_data: raise ValueError("trailing bytes after compressed ZIP member") # zlib can consume all input while output remains buffered. # Drain with bounded empty-input calls, never an unbounded flush. if not chunk and len(expanded) < CHUNK_SIZE: break if decompressor is not None and not decompressor.eof: raise ValueError("truncated deflate stream") if size != info.file_size or crc != info.CRC: raise ValueError("ZIP member size or CRC mismatch") def install_zip_member( destination, url, member, *, expected_sha256=None, expected_size=None, compressed_sha256=None, max_compressed_bytes=512 * 2**20, max_bytes=4 * 2**30, max_metadata_bytes=8 * 2**20, timeout=60, force=False, ): """Fetch one named HTTP ZIP member into a shared, atomic bundle. Return paths named payload, compressed and receipt.json. The receipt records the archive URL/validator/size, member metadata, CRC and both SHA-256 hashes. Expected hashes are optional during discovery; observed hashes are not independent verification. Pin expected_sha256 (and preferably compressed_sha256) for subsequent reproduction. Last-Modified is weaker than a strong ETag. Only stored and deflated, unencrypted regular members are supported. Require exact byte ranges; never fall back to a whole-archive transfer. Metadata and compressed/expanded payloads have separate finite limits. Failed attempts preserve previous generations; retries restart the selected member. Cache hits verify all retained bytes offline. Corruption or changed identity requires explicit force=True. No source is refreshed automatically. """ definition = _definition( url, member, expected_sha256, expected_size, compressed_sha256 ) for name, value in [ ("max_compressed_bytes", max_compressed_bytes), ("max_bytes", max_bytes), ("max_metadata_bytes", max_metadata_bytes), ]: _positive(value, name) if not isinstance(force, bool): raise ValueError("force must be a boolean") if expected_size is not None and expected_size > max_bytes: raise ValueError("expected_size exceeds max_bytes") destination = Path(destination).absolute() store = BundleStore(destination, "zip-member") def reuse(): inspection = _inspect(destination, definition, max_compressed_bytes, max_bytes) if store.can_reuse(inspection, force=force): return {name: entry.path for name, entry in inspection.files.items()} return None existing = reuse() if existing: return existing destination.parent.mkdir(parents=True, exist_ok=True) with store.lock(): store.create() existing = reuse() if existing: return existing working = private_directory(destination / (".staging-" + uuid4().hex)) staged = working / "files" staged.mkdir() paths = { name: staged / name for name in ("payload", "compressed", "receipt.json") } try: with ( _HTTPFile(url, timeout, max_metadata_bytes) as remote, zipfile.ZipFile(remote) as archive, ): matches = [ info for info in archive.infolist() if info.filename == member ] if len(matches) != 1: raise ValueError( "ZIP must contain exactly one member named %r" % member ) info = matches[0] if ( info.is_dir() or info.flag_bits & 1 or info.compress_type not in (zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED) ): raise ValueError( "ZIP member must be unencrypted, stored or deflated" ) mode = (info.external_attr >> 16) & 0o170000 if mode not in (0, 0o100000): raise ValueError("ZIP member must be a regular file") if ( info.compress_size > max_compressed_bytes or info.file_size > max_bytes ): raise ValueError( "ZIP member exceeds compressed or expanded size limits" ) check_expected( member, {"size": info.file_size, "sha256": None}, definition ) remote.seek(info.header_offset) header = remote.read(30) if len(header) != 30 or header[:4] != b"PK\x03\x04": raise ValueError("invalid ZIP local header") flags, method = struct.unpack_from(" Project-URL: Homepage, https://github.com/openvax/datacache Project-URL: Bug Tracker, https://github.com/openvax/datacache/issues Project-URL: Documentation, https://github.com/openvax/datacache#readme Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console Classifier: Operating System :: OS Independent Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: Apache Software License Classifier: Programming Language :: Python Classifier: Topic :: Scientific/Engineering :: Bio-Informatics Requires-Python: >=3.9 Description-Content-Type: text/markdown License-File: LICENSE Requires-Dist: pandas>=0.15.2 Requires-Dist: platformdirs>=3.0 Requires-Dist: requests>=2.5.1 Requires-Dist: filelock>=3.13 Requires-Dist: tqdm>=4.64 Provides-Extra: progress Provides-Extra: html Requires-Dist: lxml>=4.9; extra == "html" Provides-Extra: test Requires-Dist: pytest>=7; extra == "test" Requires-Dist: pytest-cov; extra == "test" Requires-Dist: ruff; extra == "test" Requires-Dist: lxml>=4.9; extra == "test" Dynamic: license-file # DataCache [![Tests](https://github.com/openvax/datacache/actions/workflows/tests.yml/badge.svg)](https://github.com/openvax/datacache/actions/workflows/tests.yml) [![PyPI](https://img.shields.io/pypi/v/datacache.svg)](https://pypi.org/project/datacache/) Download a dataset once, verify it, and reuse the local file on later runs. DataCache handles streaming downloads, gzip/ZIP decompression, tar-tree installation, offline inspection and SQLite caches built from pandas DataFrames, for Python applications including OpenVax libraries such as pyensembl. ## Install Python 3.9 or newer is required. ```sh python -m pip install datacache # Optional HTML-table conversion: python -m pip install "datacache[html]" ``` ## Quickstart Download a file once, then reuse its local path on later calls: ```python from datacache import Cache cache = Cache("my-project") url = "https://raw.githubusercontent.com/openvax/datacache/master/LICENSE" path = cache.fetch(url, filename="LICENSE", timeout=30) print(path) # Reuses the cached file without a network request, even in a later process. assert cache.fetch(url, filename="LICENSE") == path ``` Replace the URL and filename with your dataset. Existing files are reused until you refresh them with `force=True`, or periodically with `expire_after`; DataCache doesn't ask the server whether a file has changed. Add `show_progress=True` to display a download bar; no extra installation is needed. ## Guides The guides are also published at . - [Working offline example, verification and interface choices](https://github.com/openvax/datacache/blob/master/docs/index.md) - [Downloads, inspection, integrity, freshness, retries and resumption](https://github.com/openvax/datacache/blob/master/docs/downloads.md) - [Tables, SQLite and transformations](https://github.com/openvax/datacache/blob/master/docs/data.md) - [Shared caches](https://github.com/openvax/datacache/blob/master/docs/shared-caches.md) - [Progress and logging](https://github.com/openvax/datacache/blob/master/docs/progress.md) - [Pinned bundles](https://github.com/openvax/datacache/blob/master/docs/bundles.md), [archive trees](https://github.com/openvax/datacache/blob/master/docs/archives.md) and the [fixed-path file registry](https://github.com/openvax/datacache/blob/master/docs/file_registry.md) - [Interface selection and guarantees](https://github.com/openvax/datacache/blob/master/docs/reference/choosing.md) - [Derived-artifact materialization](https://github.com/openvax/datacache/blob/master/docs/materialization.md): caller-owned builders, outputs published together as one bundle, dependency receipts and resumable inputs. - [Complete API reference](https://github.com/openvax/datacache/blob/master/docs/api.md) - [Individual HTTP ZIP members](https://github.com/openvax/datacache/blob/master/docs/zip-members.md): bounded range downloads with compressed and expanded checksums. - [Downstream integration](https://github.com/openvax/datacache/blob/master/docs/integration.md): contracts for consuming libraries. - [Release notes](https://github.com/openvax/datacache/blob/master/CHANGELOG.md) and [release procedure](https://github.com/openvax/datacache/blob/master/RELEASING.md). ## Dependencies and progress display tqdm is installed by default; existing download APIs enable displays with `show_progress=True`, while `materialize` enables them by default. Quiet use does not import tqdm or configure your application's logging. The old `datacache[progress]` install syntax remains supported as a compatibility alias. The existing pandas dependency range is unchanged; upgrading DataCache does not introduce a pandas 1.5 requirement. CI covers pandas 1.4.4, 1.5.3, and current releases on supported Python versions. ## Development ```sh python -m pip install -e ".[test]" ./lint-and-test.sh python -m examples.basic_usage ``` Tests use local files, mocked responses, and local HTTP servers. They do not depend on external dataset servers. See CI for the Python, dependency, and operating-system combinations exercised. Build the site with `python -m pip install -r requirements-docs.txt` and `./docs.sh`. ## License Apache 2.0; see [LICENSE](LICENSE). ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6388342 datacache-1.27.1/datacache.egg-info/SOURCES.txt0000644000076500000000000000452115264225206020476 0ustar00iskanderwheelCHANGELOG.md LICENSE MANIFEST.in README.md pyproject.toml requirements.txt datacache/__init__.py datacache/_filesystem.py datacache/archives.py datacache/bundle_store.py datacache/bundles.py datacache/cache.py datacache/common.py datacache/database.py datacache/database_helpers.py datacache/database_table.py datacache/database_types.py datacache/download.py datacache/file_registry.py datacache/inspection.py datacache/integrity.py datacache/materialization.py datacache/permissions.py datacache/progress.py datacache/provenance.py datacache/resume.py datacache/retries.py datacache/version.py datacache/zip_members.py datacache.egg-info/PKG-INFO datacache.egg-info/SOURCES.txt datacache.egg-info/dependency_links.txt datacache.egg-info/requires.txt datacache.egg-info/top_level.txt docs/api.md docs/archives.md docs/bundles.md docs/data.md docs/downloads.md docs/file_registry.md docs/index.md docs/integration.md docs/materialization.md docs/progress.md docs/shared-caches.md docs/zip-members.md docs/dev/documentation-style.md docs/reference/choosing.md examples/basic_usage.py examples/reference_dna_materialization.py examples/versioned_datasets.py tests/test_archives.py tests/test_bundle_lifecycle.py tests/test_bundle_metadata.py tests/test_bundle_store.py tests/test_bundle_store_paths.py tests/test_bundles.py tests/test_byte_limits.py tests/test_cache_inspection.py tests/test_cache_object.py tests/test_cache_paths.py tests/test_cross_platform_stores.py tests/test_database_objects.py tests/test_database_safety.py tests/test_database_symlinks.py tests/test_database_types.py tests/test_db_from_dataframes.py tests/test_docs_api_reference.py tests/test_download.py tests/test_download_compatibility.py tests/test_downstream_compatibility.py tests/test_fetch_freshness.py tests/test_file_registry.py tests/test_http_retries.py tests/test_import_cost.py tests/test_legacy_cache_compatibility.py tests/test_materialization.py tests/test_names.py tests/test_packaging.py tests/test_permissions.py tests/test_progress.py tests/test_provenance.py tests/test_raw_downloads.py tests/test_resume.py tests/test_sqlite_schema_regressions.py tests/test_streaming_download.py tests/test_transform_helpers.py tests/test_user_agent.py tests/test_verified_download.py tests/test_zip_members.py tests/fixtures/README.md tests/fixtures/legacy_v191.sql tests/fixtures/legacy_v191_names.json././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6333668 datacache-1.27.1/datacache.egg-info/dependency_links.txt0000644000076500000000000000000115264225206022656 0ustar00iskanderwheel ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6336656 datacache-1.27.1/datacache.egg-info/requires.txt0000644000076500000000000000022515264225206021207 0ustar00iskanderwheelpandas>=0.15.2 platformdirs>=3.0 requests>=2.5.1 filelock>=3.13 tqdm>=4.64 [html] lxml>=4.9 [progress] [test] pytest>=7 pytest-cov ruff lxml>=4.9 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6337683 datacache-1.27.1/datacache.egg-info/top_level.txt0000644000076500000000000000001215264225206021333 0ustar00iskanderwheeldatacache ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6513546 datacache-1.27.1/docs/0000755000076500000000000000000015264225206014151 5ustar00iskanderwheel././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092720.7582338 datacache-1.27.1/docs/api.md0000644000076500000000000022110315264225061015242 0ustar00iskanderwheel# Public API reference This reference covers every name exported in `datacache.__all__` and every public `Cache` method in DataCache 1.27.1. Import these names from `datacache`. Signatures below show all defaults; arguments after `*` are keyword-only. Method signatures omit `self` and are called on a `Cache` instance. Start with the [quickstart](index.md#cache-and-verify-a-file) for a first download. The [download guide](downloads.md), [archive guide](archives.md), [data guide](data.md), [progress guide](progress.md), and [shared-cache guide](shared-caches.md) explain the longer workflows and compatibility guarantees. ## Index | Area | APIs | | --- | --- | | Downloading | [fetch_file](#fetch_file), [fetch_bytes](#fetch_bytes), [fetch_csv_dataframe](#fetch_csv_dataframe), [fetch_and_transform](#fetch_and_transform), [discard_partial](#discard_partial) | | Paths and presence | [expected_path](#expected_path), [file_exists](#file_exists), [build_local_filename](#build_local_filename), [get_data_dir](#get_data_dir), [get_cache_root](#get_cache_root), [resolve_path](#resolve_path), [build_path](#build_path), [ensure_dir](#ensure_dir), [clear_cache](#clear_cache) | | Integrity and permissions | [validate_file](#validate_file), [inspect_file](#inspect_file), [inspect_files](#inspect_files), [make_file_readable](#make_file_readable) | | Results and exceptions | [FileInspection](#fileinspection), [CacheInspection](#cacheinspection), [FileValidationError](#filevalidationerror), [CannotResume](#cannotresume) | | SQLite | [db_from_dataframe](#db_from_dataframe), [db_from_dataframes](#db_from_dataframes), [db_from_dataframes_with_absolute_path](#db_from_dataframes_with_absolute_path), [fetch_csv_db](#fetch_csv_db), [connect_if_correct_version](#connect_if_correct_version) | | Cache object | [Cache](#cache), [fetch](#cachefetch), [local_filename](#cachelocal_filename), [local_path](#cachelocal_path), [exists](#cacheexists), [inspect](#cacheinspect), [make_readable](#cachemake_readable), [db_from_dataframe](#cachedb_from_dataframe), [delete_url](#cachedelete_url), [delete_all](#cachedelete_all) | | Complete archive trees | [install_archive](#install_archive), [inspect_archive](#inspect_archive), [ArchiveInspection](#archiveinspection), [VersionedArchiveRegistry](#versionedarchiveregistry) | | Versioned file bundles | [install_bundle](#install_bundle), [inspect_bundle](#inspect_bundle), [BundleInspection](#bundleinspection), [list_bundles](#list_bundles), [prune_bundles](#prune_bundles), [VersionedDatasetRegistry](#versioneddatasetregistry) | | Derived artifacts | [materialize](#materialize), [inspect_materialization](#inspect_materialization), [MaterializationInspection](#materializationinspection) | | Fixed-path file registry | [VersionedFileRegistry](#versionedfileregistry) | | Package version | [`__version__`](#__version__) | ## Example setup The examples in this page use this shared setup. Run it once, then run the examples in order in the same Python session. All downloads use a local file; no external dataset server is needed. Explicit roots keep example writes in a temporary directory. Cleanup is shown at the end of the page. ```python from contextlib import closing from pathlib import Path from tempfile import TemporaryDirectory import hashlib import io import os import tarfile import pandas as pd import datacache as dc temporary = TemporaryDirectory() root = Path(temporary.name) cache_root = root / "cache" source = root / "source.csv" contents = b"id,value\n1,10\n2,20\n" source.write_bytes(contents) url = source.as_uri() sha256 = hashlib.sha256(contents).hexdigest() frame = pd.DataFrame({"id": [1, 2], "value": [10, 20]}) archive_source = root / "models.tar.bz2" with tarfile.open(archive_source, "w:bz2") as archive: member_contents = b'{"model": "example"}\n' member = tarfile.TarInfo("models/model.json") member.size = len(member_contents) archive.addfile(member, io.BytesIO(member_contents)) archive_contents = archive_source.read_bytes() archive_sha256 = hashlib.sha256(archive_contents).hexdigest() ``` ## Downloading ### `fetch_file` ```text fetch_file( download_url, filename=None, decompress=False, subdir=None, force=False, timeout=None, use_wget_if_available=None, chunk_size=1048576, progress_callback=None, *, destination=None, cache_root=None, expected_sha256=None, expected_size=None, max_bytes=None, max_retries=2, retry_backoff=1.0, retry_max_delay=30.0, show_progress=False, record_provenance=False, allow_empty=False, resume=False, raw=False, expire_after=None, return_stale_on_error=False, validator=None ) ``` Download a missing file or reuse a readable regular file at its cache path. On reuse, supplied integrity expectations are checked without a request. An invalid hit raises an error; `force=True` explicitly downloads a replacement. Successful new downloads are validated and published atomically. A failed replacement leaves the previous file intact. | Parameter | Meaning | | --- | --- | | `download_url` | Source URL string: HTTP, HTTPS, FTP, or `file://`. | | `filename` | Optional string cache key. Omitted names are inferred from the full URL. Names are normalized by [build_local_filename](#build_local_filename), not treated as exact paths. | | `decompress` | Request gzip/ZIP contents. With inferred filenames, `False` retains the archive and `True` selects a separate decompressed key. An explicit output lacking the source's compression suffix also implies decompression. | | `raw` | Boolean disabling archive decompression and HTML conversion regardless of the output name. Default `False` preserves legacy inference; cannot be combined with `decompress=True`. Expectations describe the unchanged payload after HTTP transfer decoding. | | `subdir` | Application name selecting a platform cache directory; `None` selects `datacache`. See [get_data_dir](#get_data_dir). | | `force` | Download even when a cached file exists. Does not automatically broaden its permissions. | | `timeout` | Per-attempt timeout in seconds; `None` supplies no timeout. HTTP also accepts a Requests `(connect, read)` tuple. Timeouts limit connection/read inactivity, not overall elapsed time. | | `use_wget_if_available` | Deprecated and ignored. Passing a value other than `None` emits `DeprecationWarning`. | | `chunk_size` | Positive integer transfer chunk size in bytes; the default is 1 MiB. | | `progress_callback` | Optional callable receiving `(completed_bytes, total_bytes)` after each transfer chunk. The total can be `None`; retries restart counts. Callback exceptions propagate without retrying. | | `destination` | Exact output path, as a string or `Path`. Mutually exclusive with `filename`, `subdir`, and `cache_root`. Its spelling is kept even when decompressing. | | `cache_root` | Exact directory containing cached files, as a string or `Path`. Overrides `subdir`; relative roots stay relative to the working directory. | | `expected_sha256` | Optional trusted 64-character hexadecimal SHA-256 digest of the **installed bytes**, after decompression or conversion. Case-insensitive. | | `expected_size` | Optional non-negative integer size of those installed bytes; booleans are rejected. | | `max_bytes` | Optional non-negative integer: the most bytes any file this fetch writes may hold, checked before every write. It bounds the download (the body after HTTP transfer decoding, such as gzip `Content-Encoding`, whatever `Content-Length` says) and the installed file after `decompress=True`. A larger transfer raises `FileValidationError` without retrying, keeps any existing file and removes temporary files. Cache hits write nothing and are unaffected. `expected_size` must not exceed it; HTML-to-CSV conversion can't be bounded. See [byte limits](downloads.md#byte-limits). | | `max_retries` | Non-negative integer number of additional attempts after transient HTTP failures. Default `2` allows three total attempts; `0` disables retries. File and FTP downloads are not retried. | | `retry_backoff` | Initial retry delay in seconds, doubled for subsequent retries. Must be finite and non-negative. | | `retry_max_delay` | Maximum retry delay in seconds, finite and non-negative. A server `Retry-After` exceeding this limit stops retries. | | `show_progress` | Boolean enabling tqdm download, decompression, and hash bars. tqdm is installed with datacache; no extra is needed. Displays are opt-in and cache hits are quiet. | | `resume` | `True`, `False` or `"auto"`, retaining private partials for raw HTTP transfers. `"auto"` resumes whenever it can and otherwise downloads normally, including when the server turns out not to support resuming (no strong ETag, encoded responses); `True` raises in those cases. Requires size and a POSIX local filesystem. Without an expected SHA-256, every accepted response must supply a strong ETag. Size-only cache hits check only byte count. See [resumable downloads](downloads.md#resumable-http-downloads). | | `record_provenance` | Boolean; after publishing a download, also write a hidden `..datacache.json` record of the source URL (without user name, password, query string, or fragment; the path is kept as is, so avoid recording URLs with secrets in their path), the fetch time, the size, and the SHA-256 when `expected_sha256` verified it. [inspect_file](#inspect_file) reports these offline. The record has the file's permissions. Default `False`: a caller that downloads to a temporary name and then moves the file would leave the record behind. Cache hits never write one, any new download removes a previous record first, and a failure to write one never fails the download. | | `allow_empty` | Boolean accepting an empty installed file; default `False`. A complete but empty response, such as a withdrawn upstream record, is otherwise never published: HTTP retries it as transient, then raises `FileValidationError`. An empty cached file is likewise an invalid hit. `expected_size=0` also allows an empty file. | | `expire_after` | How long a cached file stays fresh: non-negative seconds or a `datetime.timedelta`, as in requests-cache. An older cached file is downloaded again, as is one that no longer validates; `0` refreshes every time. Age comes from the provenance record's fetch time when one describes the file, otherwise from its modification time; a future time counts as new. Default `None` reuses a valid file however old and raises for an invalid one. | | `return_stale_on_error` | Boolean; when a refresh (`force=True` or an expired `expire_after`) raises and a valid cached file exists, log a warning (without URL secrets) and return the cached path, like HTTP `stale-if-error`. Without a valid cached file the error propagates, as does an exception from `progress_callback`, which cancels the fetch. The cached file is checked only after a failed refresh. Default `False`. | | `validator` | Optional callable `validator(path)` for content a successful transfer can still get wrong, such as an HTTP 200 error page. It rejects the file by raising (normally `ValueError`) or returning `False`. It checks the installed bytes before publication, so a rejected download never replaces the cached file, and checks cache hits. Rejection raises `FileValidationError`, chained to any exception. Not supported with `resume=True`. | ZIP downloads install the member stored at the output name; otherwise a member with that name, ignoring letter case, in any folder: the one nearest the archive root, then an exact-case match, then the largest. If none matches, the largest non-directory member is installed, with a logged warning when the output was named explicitly and the archive has several. HTML-to-CSV conversion is enabled by an explicit `.csv` output for an HTML source and requires `datacache[html]`. See [format selection](downloads.md#archives-and-decompression) and [HTTP retry behavior](downloads.md#transient-http-failures). **Returns:** local path string. It is not necessarily absolute when given a relative `destination` or `cache_root`. **Raises:** `ValueError` for conflicting paths or invalid expectations, chunk size, byte limit, retry options, callback, or progress flag; [`FileValidationError`](#filevalidationerror) for non-regular files or integrity mismatches; `OSError` subclasses for filesystem failures; Requests exceptions for HTTP failures; `urllib.error.URLError` for URL-handler failures; archive errors such as `zipfile.BadZipFile`, `gzip.BadGzipFile`, or `EOFError` for damaged input. Empty ZIPs raise `ValueError`. Missing optional dependencies raise `ImportError`. Parser and callback exceptions propagate. Transport exceptions are preserved after retry exhaustion. ```python path = dc.fetch_file( url, filename="records.csv", cache_root=cache_root, expected_sha256=sha256, expected_size=len(contents), timeout=30, ) assert dc.fetch_file(url, filename="records.csv", cache_root=cache_root) == path def has_header(path): if not Path(path).read_bytes().startswith(b"id,value"): raise ValueError("not the records table") # Refresh copies older than a day, keep the cached copy if the source is # unreachable, and never cache content without the expected header. fresh = dc.fetch_file( url, filename="records.csv", cache_root=cache_root, expire_after=86400, return_stale_on_error=True, validator=has_header, ) assert fresh == path ``` ### `fetch_bytes` ```text fetch_bytes( download_url, *, timeout=None, max_retries=2, retry_backoff=1.0, retry_max_delay=30.0, allow_empty=False, max_bytes=None ) ``` Return a resource's bytes in memory, transferred and retried exactly as [fetch_file](#fetch_file) downloads are, without writing anything to disk. Use it for small resources such as directory listings and metadata, or where the cache is read-only. | Parameter | Meaning | | --- | --- | | `download_url` | Source URL string: HTTP, HTTPS, FTP, or `file://`. | | `timeout`, `max_retries`, `retry_backoff`, `retry_max_delay` | As for [fetch_file](#fetch_file). Only HTTP(S) transfers are retried. | | `allow_empty` | Boolean accepting an empty body; default `False`. An empty HTTP response is otherwise retried as transient, then rejected. | | `max_bytes` | Optional non-negative integer: the largest body to accept, after HTTP transfer decoding. Checked before each chunk is kept, so the returned bytes never grow past it, and the body is read in chunks no larger than the limit; a larger body raises `ValueError` without retrying. `0` requires `allow_empty=True`. | **Returns:** `bytes`. **Raises:** `ValueError` for invalid options, an empty body, or a body larger than `max_bytes`; Requests exceptions for HTTP failures, preserved after retry exhaustion; `urllib.error.URLError` for file and FTP failures. ```python assert dc.fetch_bytes(url, timeout=30) == contents ``` ### `fetch_csv_dataframe` ```text fetch_csv_dataframe( download_url, filename=None, subdir=None, *, download_options=None, show_progress=False, **pandas_kwargs ) ``` Fetch and decompress the source, then call `pandas.read_csv` on its local path. `download_url`, `filename`, and `subdir` have the meanings above. Pass parser arguments such as `sep`, `dtype`, and `usecols` through `**pandas_kwargs`. `download_options` is an optional dictionary of other `fetch_file` arguments (for example `cache_root`, `timeout`, `force`, retry settings, and expectations). It is copied, not mutated. Keep `download_url`, `filename`, `subdir`, and `decompress` out of this dictionary: the helper supplies them itself, with `decompress=True`. Exact destinations still obey `fetch_file`'s exclusions. An inner `show_progress` overrides the outer setting for the download. **Returns:** a pandas `DataFrame` normally; a pandas `TextFileReader` when `chunksize` or `iterator=True` is supplied. Close a reader when finished. **Raises:** [fetch_file errors](#fetch_file), pandas parser errors such as `pandas.errors.ParserError` and `pandas.errors.EmptyDataError`, decoding errors, and `TypeError` for duplicated or unsupported keyword arguments. ```python loaded = dc.fetch_csv_dataframe( url, filename="records.csv", dtype={"id": "int64"}, download_options={"cache_root": cache_root, "expected_sha256": sha256}, ) assert loaded["id"].tolist() == [1, 2] ``` ### `fetch_and_transform` ```text fetch_and_transform( transformed_filename, transformer, loader, source_filename, source_url, subdir=None, *, force=False, cache_root=None, download_options=None, show_progress=False ) ``` Cache one transformed file. `transformed_filename` is resolved with [`resolve_path`](#resolve_path); `source_filename` is a download cache key and `source_url` is the source URL. `subdir` and `cache_root` select the output root and the default source root. `transformer(source_path, output_path)` must write and close one regular file at the supplied, initially absent path. Its basename and extension match the final output, but it lives in a private staging directory. The result is published only after successful validation. `loader(final_path)` loads an existing readable output. A hit calls only the loader, ignoring source options. `force=True` rebuilds the output while reusing the source; also pass `download_options={"force": True}` to refresh the source. The optional options dictionary is copied and passed to `fetch_file`; do not duplicate its supplied `download_url`, `filename`, or `subdir` arguments. Integrity expectations in this dictionary validate the downloaded source, not the transformed output. `show_progress` sets the source-download default; arbitrary transformer work has no automatic bar. Inner options can override source `cache_root`, `show_progress`, and `decompress`. For compatibility, a nonempty `subdir` defaults source decompression to `True`; otherwise it defaults to `False`. An explicit `download_options["decompress"]` overrides this. Without an explicit root, sources misplaced in the default cache by older releases can be reused without moving or changing them. See the [transformation guide](data.md#custom-single-file-transformations). **Returns:** the transformer's result after a build, or the loader's result on reuse. A transformer returning its output path directly as a string or `Path` receives the final path back. Paths nested inside other objects are not rewritten; prefer in-memory results and close files before returning. **Raises:** [fetch_file errors](#fetch_file), `RuntimeError` if the transformer does not create its output, `FileValidationError` for invalid output, and any transformer/loader exception. Failures preserve the previous transformed file. ```python def double_values(source_path, output_path): result = pd.read_csv(source_path) result["value"] *= 2 result.to_csv(output_path, index=False) return result doubled = dc.fetch_and_transform( "doubled.csv", double_values, pd.read_csv, "records.csv", url, cache_root=cache_root, ) assert doubled["value"].tolist() == [20, 40] ``` ### `discard_partial` ```text discard_partial(destination) ``` Explicitly discard this user's private resumable bytes for an exact destination, under its writer lock. The installed file is unchanged; absent state is a no-op. See [resumable downloads](downloads.md#resumable-http-downloads). ## Paths and presence ### `expected_path` ```text expected_path( download_url=None, filename=None, decompress=False, subdir=None, *, destination=None, cache_root=None ) ``` Compute the path that `fetch_file` would use, without filesystem access or directory creation. The parameters follow [fetch_file](#fetch_file); either `download_url`, `filename`, or `destination` must identify the output. A `destination` is used exactly and cannot be combined with `filename`, `subdir`, or `cache_root`. Otherwise, the name is normalized by `build_local_filename`. **Returns:** path string, possibly relative. **Raises:** `ValueError` for no URL/filename, an empty destination, or mutually exclusive path arguments. ```python assert dc.expected_path(filename="records.csv", cache_root=cache_root) == path ``` ### `file_exists` ```text file_exists( download_url=None, filename=None, decompress=False, subdir=None, *, destination=None, cache_root=None ) ``` Check the presence of the path selected by [expected_path](#expected_path), with the same parameters. Follows symlinks. Directories count as present and dangling symlinks as absent. Does not validate readability or integrity. **Returns:** `bool`. **Raises:** `expected_path` argument errors and filesystem errors other than `FileNotFoundError`, such as `PermissionError`. No directories are created, even when the cache root is absent. ```python assert dc.file_exists(filename="records.csv", cache_root=cache_root) ``` ### `build_local_filename` ```text build_local_filename(download_url=None, filename=None, decompress=False) ``` Choose a string cache key from a URL string or an explicit filename string. At least one must be nonempty. An explicit name takes precedence. URL-derived names include an MD5 digest of the full URL for naming, not integrity checking. Special characters `/`, `\`, `;`, `:`, `?`, and `=` are replaced with underscores; long names are shortened with a digest, as is any name over 255 UTF-8 bytes, which keeps as much of its end (including the extension) as fits. This is a cache key, not an exact path. `decompress=True` strips a final `.gz` or `.zip` suffix. Inferred archive endpoints without a removable suffix instead get `.decompressed`, so archive and decompressed keys differ. No filesystem or network access occurs. **Returns:** filename string. **Raises:** `ValueError` if neither a URL nor a filename is supplied. ```python assert dc.build_local_filename(filename="records.csv.gz", decompress=True) == "records.csv" ``` ### `get_data_dir` ```text get_data_dir(subdir=None, envkey=None) ``` Select a platform cache directory without creating it. `subdir` is an application name, defaulting to `datacache` when omitted or empty. It is not a directory nested under the default datacache cache. | Platform | Path for an application named `my-project` | | --- | --- | | Linux / Unix | `$XDG_CACHE_HOME/my-project`, or `~/.cache/my-project` when it is unset, empty or relative | | macOS | `~/Library/Caches/my-project`, even when `XDG_CACHE_HOME` is set | | Windows | `%LOCALAPPDATA%\my-project\my-project\Cache` | Paths follow `platformdirs.user_cache_dir`, including Windows' repeated application name, with two exceptions that keep a location the same whichever platformdirs version is installed. macOS ignores `XDG_CACHE_HOME`, as appdirs did before DataCache 1.25.0, so existing caches stay put; platformdirs 4.6 and later would honour it. `XDG_CACHE_HOME` is read as platformdirs 4.11.8 and later read it: surrounding whitespace is stripped, and a relative value is ignored, so the cache never depends on the working directory. The function's return value is authoritative for the current platform. If `envkey` names a nonempty environment variable, use that variable's value as the root and append `subdir` if supplied. `envkey` is opt-in for this helper; there is no implicit `DATACACHE_*` environment override for all APIs. Pass the returned path as `cache_root` to use that selection elsewhere. **Returns:** directory path string. **Raises:** platform directory lookup errors from platformdirs if it cannot determine the location, such as `RuntimeError` when there is no home directory; invalid argument types can raise `TypeError`. Does not require the directory to exist. ```python print(dc.get_data_dir()) # Platform cache directory for datacache. print(dc.get_data_dir("my-project")) # For an application-specific override: selected_root = dc.get_data_dir("my-project", envkey="MY_PROJECT_CACHE_ROOT") ``` ### `get_cache_root` ```text get_cache_root(name, *envkeys, override=None, legacy=()) ``` Find where a package's cached data lives, without creating anything. The first of these that applies is the root, with `~` expanded: 1. `override`, when not `None`: an explicit choice, such as a command-line flag; 2. the first environment variable in `envkeys` set to a non-blank value; 3. the platform cache directory for `name` (the one `get_data_dir(name)` selects), when it already holds data; 4. the first path in `legacy` that already holds data, so a cache made before a move keeps working; 5. the platform cache directory for `name`. A directory holds data when it contains a file anywhere inside, other than files operating systems or DataCache leave on their own (`.DS_Store`, `Thumbs.db`, `desktop.ini`, `.datacache-*`); empty folders don't count, and unreadable entries are skipped. `legacy` paths must be absolute. The legacy check reads the disk, so resolve the root once and pass it on. Nothing is appended to an environment value, unlike `get_data_dir(subdir, envkey)`, so every package that reads the same variable agrees on one location. List a package-specific variable before a shared one so it can override it. Pass the result as `cache_root`. A relative value stays relative to the working directory. See [choosing where data lives](shared-caches.md#choosing-where-data-lives). **Returns:** directory path string. **Raises:** `ValueError` for an empty `name`, a non-string or empty variable name, an empty or non-text `override`, or a non-text, empty or relative `legacy` path (checked whichever rule applies), and platform directory lookup errors. ```python os.environ["EXAMPLE_SHARED"] = str(root / "shared") shared_root = dc.get_cache_root("example", "EXAMPLE_OWN", "EXAMPLE_SHARED") assert shared_root == str(root / "shared") del os.environ["EXAMPLE_SHARED"] print(dc.get_cache_root("example")) # Platform cache directory for example. ``` ### `resolve_path` ```text resolve_path(filename, subdir=None, *, cache_root=None) ``` Join a filename/path with `cache_root`, or with `get_data_dir(subdir)` if no root is supplied. Accepts string or `Path` values for `filename` and `cache_root`. Relative roots remain relative. Unlike `expected_path`, does not normalize filenames; nested components and absolute filenames are supported. An absolute filename overrides the root. This is ordinary path joining, not confinement to a cache directory. Performs no filesystem access. **Returns:** path string. **Raises:** path argument `TypeError` for unsupported types, or platform lookup errors from `get_data_dir`. ```python nested_path = dc.resolve_path("nested/records.csv", cache_root=cache_root) assert nested_path == str(cache_root / "nested" / "records.csv") ``` ### `build_path` ```text build_path(filename, subdir=None, *, cache_root=None) ``` Resolve a path with the same parameters as [resolve_path](#resolve_path), and create its parent directories if needed. Does not create or validate the file. Use `resolve_path` for lookup without writes. **Returns:** path string. **Raises:** `resolve_path` errors and `OSError` subclasses from directory creation, including permission or path-type errors. ```python writable_path = dc.build_path("nested/output.txt", cache_root=cache_root) assert Path(writable_path).parent.is_dir() ``` ### `ensure_dir` ```text ensure_dir(path) ``` Create `path` and its parents unless something already exists there. Accepts a string or `Path`. An existing path is left alone; this helper does not verify that an existing path is a directory. A directory another process creates at the same moment is accepted. **Returns:** `None`. **Raises:** `OSError` subclasses for directory creation failures, including `PermissionError`, and `FileExistsError` for a broken symlink or a file created at `path` concurrently. ```python dc.ensure_dir(root / "extra" / "nested") assert (root / "extra" / "nested").is_dir() ``` ### `clear_cache` ```text clear_cache(subdir=None) ``` Recursively delete the **entire directory** selected by `get_data_dir(subdir)`, including its root. `subdir=None` selects the default `datacache` cache. This function has no `cache_root` argument. For an explicit root, use [`Cache.delete_all`](#cachedelete_all), which preserves the root directory. Deletion is explicit, is not transactional, and may partially complete on error. **Returns:** `None`. **Raises:** `OSError` subclasses such as `FileNotFoundError` when the root is missing, `PermissionError` on denied access, or a path-type error if the root is not a directory. A root symlink is rejected by `shutil.rmtree`; it is not followed. This example is commented out so running this page does not delete a real cache: ```python # dc.clear_cache("my-project") # Delete this application's platform cache and root. ``` ## Integrity, inspection, and permissions ### `validate_file` ```text validate_file(path, expected_sha256=None, expected_size=None, *, show_progress=False) ``` Read-only validation of a string or `Path` pointing to a readable regular file; symlinks are followed. `expected_sha256` and `expected_size` have the same types and installed-byte meanings as in [fetch_file](#fetch_file). Without expectations, checks only readability and file type. `show_progress=True` enables a tqdm bar during SHA-256 hashing, including for an existing file. No download, directory creation, lock, permission change, or repair occurs. **Returns:** the supplied path as a string. **Raises:** `ValueError` for invalid expectations, `FileValidationError` for non-regular files or mismatched size/hash, `FileNotFoundError` for a missing file, other `OSError` subclasses for access failures, and `ImportError` if a requested hash bar requires missing tqdm. ```python assert dc.validate_file(path, expected_sha256=sha256, expected_size=len(contents)) == path ``` ### `inspect_file` ```text inspect_file(path, expected_sha256=None, expected_size=None) ``` Inspect a string or `Path` using the same optional expectations as `validate_file`, entirely offline and without writes. Filesystem and validation failures become result statuses instead of being raised. An available file's size and modification time are reported, taken from the same open file that was validated. If `fetch_file(..., record_provenance=True)` recorded where it came from, and the file's size and modification time are unchanged since, its source URL, fetch time, and recorded SHA-256 are reported too. The recorded digest never sets `verified`, which still means a supplied SHA-256 matched the bytes now; pass it as `expected_sha256` to check them. A missing, unreadable, or malformed record is ignored, and a record is never opened in a way that could block. **Returns:** [`FileInspection`](#fileinspection). **Raises:** `ValueError` for invalid expectation arguments. Unsupported path types can raise `TypeError`. ```python inspection = dc.inspect_file(path, expected_sha256=sha256) assert inspection.status == "available" and inspection.verified assert dc.inspect_file(root / "absent.csv").status == "missing" assert inspection.size == len(contents) ``` ### `inspect_files` ```text inspect_files(cache_root, files) ``` Inspect a required-file inventory under `cache_root` (string or `Path`), with no writes or network access. `files` must be a nonempty mapping of normalized relative names to dictionaries containing `expected_sha256` and/or `expected_size`. Use `{}` or `None` when expectations are unavailable. Nested names use `/`; absolute paths, `..`, redundant separators, backslashes, and colons (including drive-qualified names) are rejected. Symlinks are followed during inspection. This checks a caller-supplied inventory; it does not parse a manifest or take a snapshot across concurrent changes. **Returns:** [`CacheInspection`](#cacheinspection), including individual file results when the root is an accessible directory. A missing root is `missing`; an existing root missing a required file is `corrupt`. Inaccessibility takes precedence over corrupt or missing files. `verified=True` requires a matching SHA-256 for every required file. **Raises:** `ValueError` for an empty mapping, invalid names, unexpected metadata keys, or invalid expectations. Malformed mapping/metadata types can raise `TypeError` or `AttributeError`. Filesystem failures are returned in the result. ```python inventory = dc.inspect_files(cache_root, { "records.csv": {"expected_sha256": sha256, "expected_size": len(contents)}, }) assert inventory.status == "available" and inventory.verified assert inventory.files["records.csv"].path == path ``` ### `FileInspection` ```text FileInspection( path: str, status: str, verified: bool = False, error: Optional[Exception] = None, size: Optional[int] = None, mtime: Optional[float] = None, source_url: Optional[str] = None, fetched_at: Optional[str] = None, recorded_sha256: Optional[str] = None ) ``` Frozen dataclass returned by `inspect_file` and `Cache.inspect`. | Field | Meaning | | --- | --- | | `path` | Inspected path string. | | `status` | `available` (readable regular file matching supplied expectations), `missing` (absent), `corrupt` (wrong type or failed expectations), or `inaccessible` (other filesystem error). | | `verified` | `True` only when a supplied SHA-256 matched. Readability or matching size alone leaves it `False`, and so does a recorded digest. | | `error` | Original exception for an unavailable file, or `None` on success. | | `size` | Size in bytes of an available file; otherwise `None`. | | `mtime` | Modification time of an available file, in seconds since the epoch; otherwise `None`. | | `source_url` | Source URL recorded by `fetch_file(..., record_provenance=True)`, without user name, password, query string, or fragment, while the file's size and modification time are unchanged since it was recorded; otherwise `None`. | | `fetched_at` | Recorded fetch time, UTC in ISO 8601, under the same condition; otherwise `None`. | | `recorded_sha256` | SHA-256 that `expected_sha256` verified when the file was downloaded, under the same condition; otherwise `None`. It is a record, not a check of the current bytes: a file edited in place without changing its size or modification time, or a record forged in a shared cache, still reports it. To check the bytes now, pass it back as `expected_sha256`. | **Construction:** returns a `FileInspection` instance; fields are assigned as given, with no validation of manually supplied status/type combinations. Incorrect constructor arguments raise `TypeError`. Assigning a field afterward raises `dataclasses.FrozenInstanceError`. ```python assert isinstance(inspection, dc.FileInspection) manual_result = dc.FileInspection(path=path, status="available") assert not manual_result.verified ``` ### `CacheInspection` ```text CacheInspection( path: str, status: str, files: Dict[str, FileInspection], verified: bool = False, error: Optional[Exception] = None ) ``` Frozen dataclass returned by `inspect_files`. | Field | Meaning | | --- | --- | | `path` | Inspected root path string. | | `status` | `available` (all required files available), `missing` (root absent), `corrupt` (wrong root type or incomplete/corrupt files), or `inaccessible` (root or required file access error). | | `files` | Dictionary from required relative names to `FileInspection` results. Empty if the root could not be inspected as a directory. | | `verified` | `True` only when every required file has a matching SHA-256. | | `error` | Root failure or a representative failing file's exception; `None` on success. Inspect `files` for all individual failures. | **Construction:** returns a `CacheInspection` instance; manually supplied fields are not validated. Incorrect constructor arguments raise `TypeError`. Field assignment raises `dataclasses.FrozenInstanceError`; the contained `files` dictionary itself is not frozen. ```python assert isinstance(inventory, dc.CacheInspection) manual_inventory = dc.CacheInspection( path=str(cache_root), status="available", files={"records.csv": manual_result}, ) assert not manual_inventory.verified ``` ### `FileValidationError` ```text FileValidationError(path, reason) ``` `ValueError` subclass for a non-regular file, an integrity mismatch, or another file validation failure. `path` accepts a string or `Path`; `reason` describes the failure. Attributes are `path` (converted with `os.fspath`) and `reason`. The message is `": "`. No filesystem access occurs in construction. **Construction:** returns an exception instance; an invalid path type raises `TypeError`. It is raised by validation/download helpers and retained in inspection results when applicable. ```python try: dc.validate_file(path, expected_size=0) except dc.FileValidationError as error: assert error.path == path assert "size mismatch" in error.reason ``` ### `CannotResume` A `FileValidationError` subclass raised by `resume=True` when a resumable transfer isn't possible but a normal one may be: the server sends no strong ETag (without a trusted SHA-256), encodes the response, answers ranges inconsistently or with an unexpected status, or the destination's filesystem can't keep private resume state or locks. `resume="auto"` catches it and downloads the file in full instead. The installed file is unchanged. ### `make_file_readable` ```text make_file_readable(path, *, group=True, others=False) ``` Explicitly add group read permission (`group=True`) and/or other-user read permission (`others=True`) to one existing regular file, identified by a string or `Path`. Both flags must be booleans. Adds no write/execute bits, removes no permissions, and changes no contents. A `False` flag leaves that class's existing access unchanged. The file's provenance record, if any, gets the same access, best effort. Requires POSIX file-descriptor support. The caller must be able to open the file and change its mode; parent directories and group ownership must already allow the intended readers to reach it. **Returns:** path string. **Raises:** `ValueError` for non-boolean flags, `NotImplementedError` on unsupported platforms, `FileValidationError` for a symlink or non-regular file, and `OSError` subclasses for missing files or open/chmod failures. This is never called automatically on cache reuse. ```python if hasattr(os, "fchmod") and hasattr(os, "O_NOFOLLOW"): dc.make_file_readable(path, group=True, others=False) ``` ## SQLite ### Shared database behavior All database builders return an open `sqlite3.Connection` owned by the caller. Use `contextlib.closing` or call `.close()` explicitly: `with connection:` commits/rolls back transactions but does not close the connection. Connections are opened with `check_same_thread=False`, so Python does not reject cross-thread use: applications must serialize simultaneous use of one connection and should give independent concurrent workers their own. A matching version and all requested tables permit reuse without parsing or validating new DataFrames, changing old constraints, or rewriting the file. Table names match as SQLite compares them, ignoring the case of ASCII letters. Data changes are not detected automatically. Change `version` (an integer) or use `overwrite=True` where available to rebuild. A rebuild replaces the database's tables, not just the named table, and rolls back on failure. Explicit overwrites also remove views; version-only rebuilds retain views. New databases are staged before publication. A rebuild writes SQLite's `-journal` file beside the database; where the filesystem has no room for that name, rebuilding raises a `ValueError` that says so, while creating and reusing the database still work. See [reuse and replacement](data.md#reuse-and-replacement) for locking, symlink, and filesystem requirements. Table names must be nonempty strings, distinct ignoring the case of ASCII letters (as SQLite compares names), and must not be the reserved metadata name `_datacache_metadata` or start with `sqlite_`, in any case. They are checked before any reuse. DataFrame column names must be nonempty strings; spaces become underscores and names must remain distinct ignoring the case of ASCII letters. Primary keys name one column; index specifications are sequences of nonempty column-name sequences, such as `[("id",), ("id", "value")]`, not bare strings. Original and normalized column spellings are accepted. The pandas row index is not stored. `show_progress=True` enables optional tqdm row-insertion bars; reuse is quiet. Signed integers preserve precision, missing values become SQL `NULL`, and dates/datetimes become ISO text. See [types and constraints](data.md) for details. **Shared errors:** `ValueError` for invalid names, constraints, versions, or unsupported dtypes; `TypeError` for unsupported argument types; `sqlite3.IntegrityError` for duplicate or null primary keys; `OverflowError` for integers outside SQLite's signed 64-bit range; other `sqlite3.Error` subclasses for binding, database, schema, or locking failures; `OSError` subclasses for filesystem/publication failures; and `ImportError` for missing tqdm when a progress bar is needed. Existing malformed databases are not silently repaired. ### `db_from_dataframe` ```text db_from_dataframe( db_filename, table_name, df, primary_key=None, subdir=None, overwrite=False, indices=(), version=1, *, cache_root=None, show_progress=False ) ``` Build or reuse one table named `table_name` from pandas DataFrame `df`. `db_filename`, `subdir`, and `cache_root` are resolved with `resolve_path`; parents are created if a build is needed. `primary_key=None` means no primary key; `indices=()` means no extra indexes. `overwrite`, `version`, column naming, and progress follow [shared database behavior](#shared-database-behavior). **Returns:** open `sqlite3.Connection`. **Raises:** the shared database errors. ```python with closing(dc.db_from_dataframe( "records.db", "records", frame, primary_key="id", indices=[("value",)], cache_root=cache_root, version=1, )) as connection: assert connection.execute("SELECT value FROM records ORDER BY id").fetchall() == [(10,), (20,)] ``` ### `db_from_dataframes` ```text db_from_dataframes( db_filename, dataframes, primary_keys=None, indices=None, subdir=None, overwrite=False, version=1, *, cache_root=None, show_progress=False ) ``` Build or reuse a database containing the tables in the nonempty mapping `dataframes`, from table names to pandas DataFrames. `primary_keys` optionally maps table names to a primary-key column; omitted entries have no primary key. `indices` optionally maps table names to index specifications; omitted entries have no extra indexes. `db_filename`, `subdir`, and `cache_root` select the path as in `db_from_dataframe`, creating parents when needed. `overwrite`, `version`, and `show_progress` follow the shared behavior above. **Returns:** open `sqlite3.Connection`. **Raises:** the shared database errors. ```python with closing(dc.db_from_dataframes( "multiple.db", {"records": frame, "other_records": frame}, primary_keys={"records": "id"}, indices={"records": [("value",)]}, cache_root=cache_root, )) as connection: assert connection.execute("SELECT COUNT(*) FROM other_records").fetchone() == (2,) ``` ### `db_from_dataframes_with_absolute_path` ```text db_from_dataframes_with_absolute_path( db_path, table_names_to_dataframes, table_names_to_primary_keys=None, table_names_to_indices=None, overwrite=False, version=1, *, show_progress=False ) ``` Use `db_path` directly (string or `Path`), without selecting a platform cache directory. Despite the name, relative paths also work relative to the current directory. The parent directory must already exist. For dangling symlinks, the target's parent must exist; publication preserves the symlink. `table_names_to_dataframes`, `table_names_to_primary_keys`, and `table_names_to_indices` have the same mapping meanings as `dataframes`, `primary_keys`, and `indices` in `db_from_dataframes`. `overwrite`, `version`, and `show_progress` follow the shared database behavior. **Returns:** open `sqlite3.Connection`. **Raises:** the shared database errors, including filesystem errors if the parent is absent. ```python with closing(dc.db_from_dataframes_with_absolute_path( root / "direct.db", {"records": frame}, table_names_to_primary_keys={"records": "id"}, )) as connection: assert connection.execute("SELECT COUNT(*) FROM records").fetchone() == (2,) ``` ### `fetch_csv_db` ```text fetch_csv_db( table_name, download_url, csv_filename=None, db_filename=None, subdir=None, version=1, *, download_options=None, show_progress=False, **pandas_kwargs ) ``` Download/parse a CSV and build or reuse `table_name` in SQLite. `download_url`, `csv_filename`, `subdir`, `download_options`, and `**pandas_kwargs` follow `fetch_csv_dataframe`. `csv_filename=None` infers a download key from the URL; `db_filename=None` infers a database name from the CSV name, row count, column names, and dtypes. That name is spelled out only when every character is allowed in file names on all supported platforms (no `/`, `\`, `:`, `*`, `?`, `"`, `<`, `>`, `|`, or control characters) and the name leaves room for SQLite's `-journal` file within 255 UTF-8 bytes, measured the same way on every platform. Otherwise the schema is named by a digest, forbidden characters from the CSV filename become `_`, and a very long CSV filename is shortened, so column names never add directories or leave the cache directory; directories in an explicit `csv_filename` are kept. A matching database an older release stored under the historical name is still reused in place. A new `version` rebuilds it under the new name and leaves the older copy alone, since another process may still have it open, logging a warning that names it. An explicit `db_filename` works independently of `csv_filename`. A `cache_root` inside `download_options` applies to both files. Parser options must produce a DataFrame with non-empty string column names: do not pass `chunksize` or `iterator`, and pair `header=None` with `names=`. With an explicit database filename, matching tables/version can be reused without downloading or parsing the source. Supplying `force`, `expected_sha256`, or `expected_size` in `download_options` requests source work even on such a hit. Refreshing a source alone does not replace a matching database: increment `version` when its data or schema changes. There is no `overwrite` parameter. `show_progress` enables both download and insertion displays; an inner setting overrides only the download display. **Returns:** open `sqlite3.Connection`. **Raises:** `fetch_csv_dataframe` errors and the shared database errors. Incompatible parser return types can raise `AttributeError` or `TypeError` during database construction. ```python with closing(dc.fetch_csv_db( "records", url, csv_filename="records.csv", db_filename="from-csv.db", download_options={"cache_root": cache_root}, version=1, )) as connection: assert connection.execute("SELECT SUM(value) FROM records").fetchone() == (30,) ``` ### `connect_if_correct_version` ```text connect_if_correct_version(db_path, version, *, read_only=False) ``` Open an existing `db_path` (string or `Path`) when its DataCache metadata matches the requested integer `version`. `read_only=True` uses SQLite's read-only open mode. Never creates a missing database; closes connections it rejects. This checks metadata only, not application tables, data correctness, or a checksum. **Returns:** an open caller-owned `sqlite3.Connection` on a match, or `None` when the path is missing, metadata is absent, or the version differs. **Raises:** `OSError` subclasses for filesystem access failures, `sqlite3.Error` subclasses for invalid/unreadable databases or malformed metadata tables, and conversion errors such as `ValueError` for malformed stored versions. ```python connection = dc.connect_if_correct_version(cache_root / "records.db", 1, read_only=True) assert connection is not None with closing(connection): assert connection.execute("SELECT COUNT(*) FROM records").fetchone() == (2,) assert dc.connect_if_correct_version(root / "absent.db", 1) is None ``` ## Cache object ### `Cache` ```text Cache(subdir='datacache', *, cache_root=None) ``` Select a root once for file, inspection, database, and deletion operations. `subdir` must be a nonempty application name; `cache_root` optionally supplies the exact string/`Path` root, overriding the platform location selected by `subdir`. Construction performs no directory creation or download. **Returns:** a `Cache` instance. Public attributes are `subdir` (the supplied name) and `cache_directory_path` (the selected root as a string). Relative roots remain relative to the working directory; use an absolute root if that directory may change. **Raises:** `ValueError` for an empty `subdir`, invalid path-type errors, or platform directory lookup errors from `get_data_dir`. ```python cache = dc.Cache("reference-example", cache_root=root / "object-cache") assert cache.cache_directory_path == str(root / "object-cache") ``` ### `Cache.fetch` ```text Cache.fetch( url, filename=None, decompress=False, force=False, timeout=None, use_wget_if_available=None, *, chunk_size=1048576, progress_callback=None, expected_sha256=None, expected_size=None, max_bytes=None, max_retries=2, retry_backoff=1.0, retry_max_delay=30.0, show_progress=False, record_provenance=False, allow_empty=False, resume=False, raw=False, expire_after=None, return_stale_on_error=False, validator=None ) ``` Instance form of [fetch_file](#fetch_file), with `url` as the source URL and the instance's root as `cache_root`. Every other parameter has the same meaning and default as its `fetch_file` counterpart. Explicit names are remembered for this instance's `delete_url` calls; this association is not written to disk. **Returns:** local path string. **Raises:** the errors documented for `fetch_file`. ```python cached_path = cache.fetch(url, filename="records.csv", expected_sha256=sha256) assert Path(cached_path).read_bytes() == contents ``` ### `Cache.local_filename` ```text Cache.local_filename(url=None, filename=None, decompress=False) ``` Instance form of [build_local_filename](#build_local_filename). `url` is its `download_url`; `filename` and `decompress` have identical meanings. Does not access the filesystem or include the root directory in the result. **Returns:** filename string. **Raises:** `ValueError` if neither a URL nor a filename is supplied. ```python assert cache.local_filename(filename="records.csv.gz", decompress=True) == "records.csv" ``` ### `Cache.local_path` ```text Cache.local_path(url=None, filename=None, decompress=False, download=False) ``` Instance form of [expected_path](#expected_path). `url`, `filename`, and `decompress` select a cache key under this instance's root. With the default `download=False`, only computes the path. With `download=True`, delegates to `Cache.fetch` with its defaults; use `Cache.fetch` directly for timeouts, integrity expectations, or other download options. **Returns:** path string. **Raises:** naming/path argument errors, plus `Cache.fetch` errors when `download=True`. ```python assert cache.local_path(filename="records.csv") == cached_path ``` ### `Cache.exists` ```text Cache.exists(url=None, filename=None, decompress=False) ``` Check presence at `Cache.local_path(url, filename, decompress)`, using the [file_exists](#file_exists) contract: no download, repair, or directory creation; directories count as present and dangling symlinks as absent. **Returns:** `bool`. **Raises:** naming/path argument errors and filesystem errors other than missing files, such as `PermissionError`. ```python assert cache.exists(filename="records.csv") ``` ### `Cache.inspect` ```text Cache.inspect( url=None, filename=None, decompress=False, *, expected_sha256=None, expected_size=None ) ``` Call [inspect_file](#inspect_file) on the path selected by `url`, `filename`, and `decompress`. Both expectations have the same meanings as in `inspect_file`. No network access or writes occur. **Returns:** `FileInspection`. **Raises:** naming/path argument errors and `ValueError` for invalid expectations; filesystem failures are in the result. ```python assert cache.inspect(filename="records.csv", expected_sha256=sha256).verified ``` ### `Cache.make_readable` ```text Cache.make_readable( url=None, filename=None, decompress=False, *, group=True, others=False ) ``` Call [make_file_readable](#make_file_readable) on the path selected by `url`, `filename`, and `decompress`. `group` and `others` have that function's meanings. Explicit POSIX maintenance only: does not download or recurse into directories. **Returns:** path string. **Raises:** naming/path argument errors and the errors documented for `make_file_readable`. ```python if hasattr(os, "fchmod") and hasattr(os, "O_NOFOLLOW"): cache.make_readable(filename="records.csv", group=True) ``` ### `Cache.db_from_dataframe` ```text Cache.db_from_dataframe( db_filename, table_name, df, key_column_name=None, *, overwrite=False, version=1, show_progress=False ) ``` Build or reuse `table_name` from `df` at `db_filename` under this cache's root. An absolute filename overrides the root. `key_column_name` is the primary-key column, equivalent to `primary_key` in the top-level builder. `overwrite`, `version`, and `show_progress` follow [db_from_dataframe](#db_from_dataframe). This method has no index specification argument; use the top-level function with `cache_root=cache.cache_directory_path` to request indexes. **Returns:** open caller-owned `sqlite3.Connection`. **Raises:** the [shared database errors](#shared-database-behavior). ```python with closing(cache.db_from_dataframe( "records.db", "records", frame, key_column_name="id", )) as connection: assert connection.execute("SELECT COUNT(*) FROM records").fetchone() == (2,) ``` ### `Cache.delete_url` ```text Cache.delete_url(url) ``` Remove local files for a URL string. Includes paths successfully fetched by this `Cache` instance (including explicit filenames), plus the URL-derived raw/decompressed paths that may have been created by other instances, along with their provenance records. Missing files are ignored. Explicit filenames used by earlier instances are not recorded persistently and cannot be rediscovered by URL alone. Files unrelated to those paths are left alone. A symlink at a selected file path is unlinked without deleting its target. **Returns:** `None`. **Raises:** naming errors and `OSError` subclasses such as `PermissionError` or a directory-removal error. Deletion is not transactional: some files may already have been removed when another removal fails. ```python cache.delete_url(url) assert not cache.exists(filename="records.csv") ``` ### `Cache.delete_all` ```text Cache.delete_all() ``` Remove **all contents** of this cache root recursively, including files not created by DataCache. Use only a dedicated directory. Preserves the root and its permissions. A root symlink is followed to the selected directory, while symlinks inside the cache are unlinked without clearing their external targets. The instance's remembered download paths are cleared after success. **Returns:** `None`. **Raises:** `FileNotFoundError` if the root is absent, `NotADirectoryError` for a non-directory root, and other `OSError` subclasses on access/removal failure. Deletion is not transactional and may partially complete on error. Unlike `clear_cache`, this method preserves the root and works with an explicit `cache_root`. ```python cache.delete_all() # Safe here: object-cache is inside our temporary example directory. assert Path(cache.cache_directory_path).is_dir() assert not list(Path(cache.cache_directory_path).iterdir()) ``` ## Complete archive trees ### `install_archive` ```text install_archive( destination, sources, *, expected_sha256=None, expected_size=None, extra_files=None, force=False, verified=True, download_options=None, max_members=None, max_extracted_size=None, validator=None ) ``` Safely extract a complete tar archive into a private directory and publish the tree as a new bundle. `sources` is one URL, one `Path`, or an explicitly ordered sequence. A mapping may contain `url`, `path`, `sha256`, and `size`; supplying both `url` and `path` reads local bytes while retaining the URL as the logical source identity. Multiple sources are concatenated byte-for-byte, which supports historical split archives. By default the assembled archive needs trusted `expected_sha256` and `expected_size`, or every part needs `sha256` and `size`. Set `verified=False` explicitly for historical sources without published integrity metadata. Observed hashes are still recorded and checked, but do not authenticate the source. `extra_files` maps safe relative names to text or bytes written after extraction and before publication. It is intended for consumer receipts such as `DOWNLOAD_INFO.csv`; collisions with archive content fail the installation. The complete metadata inventory is part of the archive's identity: additions, removals, and content changes require an explicit refresh. `download_options` accepts timeout, chunk size, progress, retry, resume, and `max_bytes` settings from `fetch_file`. Resumable URL parts each require a trusted hash and size. Optional `max_members` and `max_extracted_size` limits are checked before files are extracted, as headers arrive; scanning stops when a limit is exceeded. Benign leading `./` paths are normalized and root `.` directory entries ignored. Local special files such as FIFOs are rejected without blocking. `validator(tree)` optionally checks application requirements after extraction and extra-file creation, before publication, and on cache reuse. It receives a `Path` and must leave the tree unchanged. Raising or returning `False` rejects the tree with `FileValidationError`; a failed refresh leaves the old bundle active. A cached tree rejected by the callback requires an explicit `force=True` repair. The callback does not alter source identity or establish cryptographic verification. Non-callable validators raise `ValueError` before any filesystem writes. **Returns:** the `Path` of the extracted tree, the store's current bundle; not the store path. Existing returned paths remain usable across forced refreshes. **Raises:** `ValueError` for invalid definitions or options; `FileValidationError` for integrity failures, unsafe members, malformed tar archives, or invalid installations; transport and filesystem errors otherwise. An invalid current installation requires `force=True`. A legacy or otherwise foreign destination is never claimed, even with force. ```python archive_store = root / "archive-store" archive_bundle = dc.install_archive( archive_store, archive_source, expected_sha256=archive_sha256, expected_size=len(archive_contents), extra_files={"DOWNLOAD_INFO.csv": "url\n" + archive_source.as_uri() + "\n"}, ) assert (archive_bundle / "models/model.json").is_file() ``` See the [archive installation guide](archives.md) for split sources, safe-member rules, publication semantics, and an MHCflurry adapter. ### `inspect_archive` ```text inspect_archive( destination, sources=None, *, expected_sha256=None, expected_size=None, extra_files=None, verify_files=True ) ``` Inspect the current archive bundle without writes, locks, repair, or network access. Omit `sources` for receipt-only consistency checking, which returns `verified=False`. Supply the same source definition, expectations, and consumer metadata used for installation to validate the requested identity. Set `verify_files=False` for a fast check of the manifest and source identity that reads none of the extracted tree. Fast results have an empty `files` mapping and `verified=False`; use the default before asserting content integrity. ### `ArchiveInspection` A frozen record with `path`, `status`, `verified`, `bundle`, `files`, `error`, `source_urls`, `fetched_at`, `archive_size`, and `recorded_sha256`. Status is `available`, `missing`, `invalid` or `inaccessible`. `bundle` is the extracted tree applications should use; `files` maps every installed relative file name to a `FileInspection`; a file is `verified` only when a hash the caller supplied vouched for it. Source URLs are redacted for display. The recorded digest is observed receipt data, not trusted verification. ### `VersionedArchiveRegistry` ```text VersionedArchiveRegistry( archives, *, cache_root=None, store_path=None, verified=True ) ``` Reusable version catalogue for archive-tree consumers. Each named archive has a `default_version`, optional description, and `versions` mapping. A concrete version may be one source, an ordered source sequence, or a mapping containing `sources`, `expected_sha256`, `expected_size`, `extra_files`, `max_members`, and `max_extracted_size`. Definitions are validated without filesystem access. Select exactly one destination strategy. `cache_root` stores versions at `//`. A two-argument `store_path(name, version)` callback lets an adapter retain an established layout such as `//` without putting layout policy in DataCache. `verified=False` explicitly enables historical unpinned catalogues. | Method | Result | | --- | --- | | `resolve_version(name, version=None)` | Concrete version, applying the pinned default. | | `store_path(name, version=None)` | Store `Path`, without creating or inspecting it. | | `inspect(name, version=None, *, verify_files=True)` | Offline `ArchiveInspection`. | | `download(name, version=None, *, force=False, source_paths=None, **download_options)` | Install/reuse and return the current bundle's `Path`; `source_paths` overlays ordered local files while retaining catalogue identities. | | `local_path(name, version=None, *, verify_files=False)` | Resolve the current bundle without network, writes, or repair. | | `ensure(name, version=None, **download_options)` | Install if needed and return the current bundle. | | `is_cached(name, version=None, *, verify_files=False)` | Whether the current bundle is available; full verification is optional. | | `status(name=None, *, verify_files=False)` | One read-only row per concrete version, including catalogue and downloaded source URLs, store and bundle paths, default flag, status, fetch time, observed archive size/hash and inspection. Metadata-only by default; `verify_files=True` hashes every file. | | `prune(name, version=None, *, keep=1)` | Delete all but the newest `keep` bundles of one version, as [prune_bundles](#prune_bundles) does. | ## Versioned file bundles ### `install_bundle` ```text install_bundle(destination, assets, *, force=False, verified=True, verify_files=True, download_options=None, source_paths=None) ``` Download a mapping of relative asset names to `{url, sha256, size, decompress?}` metadata and publish the files together as a new bundle. `path` in place of `url` names a local file, identified by its `file://` URL. `source_paths` maps asset names to local files to read instead of downloading, such as files downloaded by hand; each asset keeps its declared URL as its identity, so later installs treat the bundle as if it had been downloaded. Returns a dict of asset names to paths in the current bundle. An invalid current bundle requires `force=True`. `verified=False` explicitly permits unpinned sources; their observed hashes are recorded without authenticating them. `verify_files=False` avoids payload reads when checking an existing bundle; it checks readability, file types and recorded sizes but cannot detect same-size corruption. New bundles are always hashed before publication, regardless of this cache-hit setting. `download_options` accepts timeout, chunk size, progress, retry, resume and `max_bytes` settings. Works on Linux, macOS and Windows local filesystems. [Complete guide](bundles.md). ### `inspect_bundle` ```text inspect_bundle(destination, assets=None, *, verify_files=True) ``` Return a `BundleInspection` of the current bundle without writes, network, locks or repair. Supply trusted assets to verify against the registry; omit them to check consistency against the bundle's own recorded hashes (`verified=False`). `verify_files=False` validates metadata, source expectations, required file types, readability and recorded sizes without reading payloads. Its bundle and file results always have `verified=False`, even with trusted hashes in the registry. ### `BundleInspection` A frozen record with `path`, `status`, `verified`, `bundle`, `files`, and `error`. Status is `available`, `missing`, `invalid` or `inaccessible`. `bundle` is the current bundle's directory, and `files` maps asset names to `FileInspection` values from it. Paths remain usable across refreshes until explicitly removed. ### `list_bundles` ```text list_bundles(destination) ``` Every bundle in a bundle, archive or materialization store, as `Path`s ordered oldest first; the last is the current bundle. Read-only. A store with nothing installed, or no directory at all, has none. A directory that isn't a datacache store raises `FileValidationError`. ### `prune_bundles` ```text prune_bundles(destination, *, keep=1) ``` Delete all but the newest `keep` bundles from a bundle, archive or materialization store, and return the deleted `Path`s, oldest first. The newest bundle is the current one, so it is always kept: `keep` must be a positive integer. Paths into deleted bundles stop working, so prune only when no running program still uses them. Pruning holds the store's lock, so it never races an install. See [removing old bundles](bundles.md#removing-old-bundles). ### `VersionedDatasetRegistry` ```text VersionedDatasetRegistry(datasets, *, cache_root=None, cache_dir=None, store_path=None, verified=True) ``` Select exactly one root path, zero-argument `cache_dir` callable, or `store_path(name, version)` callback selecting the exact store path; it is called once per dataset version on first lookup, and each version needs its own store. Root strategies keep `//`; the exact-path callback owns the consumer layout but not the layout inside each store. Each dataset specifies a `default_version` and `versions`, mapping concrete versions to asset mappings. Construction validates metadata and performs no writes or networking, and does not invoke callbacks. Path callbacks should be pure path computations. The [bundle guide](bundles.md) includes an example; notes for specific libraries are in the [integration guide](integration.md#notes-for-specific-libraries). | Method | Result | | --- | --- | | `resolve_version(name, version=None)` | Concrete version label, applying the pinned default. Unknown names/versions raise `ValueError`. | | `store_path(name, version=None)` | Store `Path`, without creating it. | | `inspect(name, version=None, *, verify_files=True)` | Read-only `BundleInspection`; full payload verification by default. | | `download(name, version=None, *, force=False, verify_files=True, source_paths=None, **download_options)` | Install/reuse and return a dict of asset paths in the current bundle; the flag selects cache-hit cost only. `source_paths` reads named assets from local files while keeping their declared URLs as identity. | | `local_path(name, version=None, *, asset=None, verify_files=True)` | Installed single asset's `Path`, or the current bundle's directory for several assets; select one asset by name. Assets are hashed by default; `verify_files=False` checks metadata and sizes only. Missing raises `FileNotFoundError`; invalid raises `FileValidationError`. | | `ensure(name, version=None, **download_options)` | Download/reuse, then return what `local_path` would, from the paths the download validated. `verify_files=False` among the options skips hashing on reuse. | | `is_cached(name, version=None, *, verify_files=True)` | Whether inspection reports `available`; `verify_files=False` skips hashing. | | `status(*, verify_files=True)` | One row per pinned default: name, version, description, available_versions and inspection. `verify_files=False` skips hashing. | | `prune(name, version=None, *, keep=1)` | Delete all but the newest `keep` bundles of one version, as [prune_bundles](#prune_bundles) does. | ## Derived artifacts ### `materialize` ```text materialize(destination, sources, *, transform, outputs, builder, force=False, retain_sources=False, verify_files=True, download_options=None) ``` Build outputs with a caller's builder and publish them, with their dependency receipt, as a new bundle. Sources map relative names to `{url: ...}` or `{path: ...}`, plus optional raw-byte `sha256`/`size`; outputs map names to optional output-byte `sha256`/`size`. Transform is `{version: nonempty_string, options: JSON_value}`. The builder receives private `(source_paths, output_paths)` mappings, must leave inputs unchanged and create/close exactly the declared regular output files. Returns name-to-string paths in the new bundle; builder return values are ignored. Dependency mismatch/corruption requires explicit `force=True`. Completed inputs survive failures; successful publication removes owned inputs unless retention is requested. Caller-owned local inputs and old bundles are never deleted. Cache hits are offline/read-only. Fast reuse is opt-in; new outputs are always hashed. Download/copy/verification progress defaults on, configurable through download options. Works on Linux, macOS and Windows local storage. See the [materialization guide](materialization.md) for receipts, resume, callback contracts, retention and peak/retained disk use. ### `inspect_materialization` ```text inspect_materialization(destination, sources=None, *, transform=None, outputs=None, verify_files=True) ``` Return `MaterializationInspection` offline without writes, locks, original input access or repair. Supply all of sources/transform/outputs for requested dependency checks, or omit all three for receipt-only consistency checks. `verify_files=False` validates inventory, types and sizes without payload reads and never claims trusted verification. ### `MaterializationInspection` A read-only record with `path`, `status`, `verified`, `bundle`, `files`, `sources`, `transform` and `error`. Status is available, missing, invalid or inaccessible. `bundle` is the current bundle's directory. `verified` means all output files matched caller-trusted hashes now. Source records separately report raw observed hashes, sizes, redacted origins, identity, acquisition time, acquisition-time trusted verification and available validated transport metadata. ## Fixed-path file registry ### `VersionedFileRegistry` `VersionedFileRegistry(datasets, *, cache_dir, error_cls=RuntimeError)` Fixed-path single-file compatibility registry. Definitions contain `filename`, `urls` (version to URL), `default_version`, and optional `description`. The zero-argument root callable may return a string or Path. Construction creates nothing. See the [fixed-path guide](file_registry.md) for legacy receipt semantics, trust boundaries, and the distinction from bundles. | Method | Result | | --- | --- | | `resolve_version(name, version=None)` | Concrete version; unknown names/versions raise the caller's error_cls. | | `local_path(name, version=None)` | Expected fixed Path, even when absent, without writes. | | `is_cached(name, version=None)` | Presence only, not verified integrity. | | `download(name, version=None, *, force=False, **download_options)` | One fixed Path; fetch_file options control acquisition. Ordinary cache hits do not hash or write; explicit size/hash expectations are checked. | | `ensure(name, version=None, **download_options)` | Same download/reuse behavior and Path result. | | `status()` | Legacy status dicts with name, description, default_version, available_versions, cached, cached_version, URL, bytes, observed SHA-256, downloaded_at and path. | ## Individual HTTP ZIP members ### `install_zip_member` ```text install_zip_member(destination, url, member, *, expected_sha256=None, expected_size=None, compressed_sha256=None, max_compressed_bytes=536870912, max_bytes=4294967296, max_metadata_bytes=8388608, timeout=60, force=False) ``` Download one explicitly named, regular, unencrypted stored/deflated ZIP member. Returns a mapping with paths `payload`, `compressed` and `receipt.json` in one atomic bundle. The receipt records source URL, archive length, HTTP validator, member offset, compression method, CRC32, sizes and SHA-256 hashes. ZIP64 is supported. Duplicate names, malformed ranges, changed objects, CRC mismatches, unsupported members and size-limit violations fail before publication. The server must support exact HTTP byte ranges, identity encoding and a strong ETag or Last-Modified. Last-Modified is weaker than a strong ETag. There is no whole-archive fallback. Metadata, compressed input and expanded output have separate finite byte limits. Downloads restart the selected member after failure. Hashes recorded during discovery describe observed bytes; only caller-supplied hashes verify independently expected content. Pin them for subsequent builds. Cache hits verify all retained bytes offline. Changed dependency identity or corruption requires `force=True`, which publishes a new generation and retains old paths. `list_bundles` and `prune_bundles` also support these stores. See the [member guide](zip-members.md) for a complete example and storage costs. ## Package version ### `__version__` String containing the installed DataCache package version. This is a constant, not a callable; it has no parameters, defaults, or API-specific exceptions. It is independent of the integer `version` used to identify a SQLite cache. ```python print(dc.__version__) ``` ## Cleanup After finishing the examples, remove their temporary files: ```python temporary.cleanup() ``` ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7085824 datacache-1.27.1/docs/archives.md0000644000076500000000000002172415264220442016302 0ustar00iskanderwheel# Archive directory installation `install_archive` safely publishes an entire tar archive tree. It is intended for applications such as MHCflurry whose released weights and datasets already have nested paths inside `.tar.bz2` files. DataCache owns transfer, extraction, integrity receipts, concurrency and publication; the caller continues to own catalogue versions and domain-specific validation. For a reusable catalogue, wrap it in `VersionedArchiveRegistry`; the lower-level functions remain useful when a downstream library already owns version lookup. ## One archive ```python from datacache import install_archive, inspect_archive tree = install_archive( cache_root / "models_class1_presentation", "https://example.org/models.tar.bz2", expected_sha256=trusted_archive_sha256, expected_size=trusted_archive_size, download_options={"timeout": 60, "max_retries": 3, "show_progress": True}, ) models = tree / "models" state = inspect_archive( cache_root / "models_class1_presentation", "https://example.org/models.tar.bz2", expected_sha256=trusted_archive_sha256, expected_size=trusted_archive_size, ) assert state.status == "available" and state.verified ``` The destination is a store, not the extracted directory. Installation returns the current bundle: the directory holding the extracted tree. Resolve it once and use that returned path for an operation; never construct `/` directly. ## Ordered split archives and local inputs An ordered list is concatenated byte-for-byte before tar parsing: ```python urls = [ "https://example.org/data.tar.bz2.part.aa", "https://example.org/data.tar.bz2.part.ab", ] tree = install_archive( cache_root / "data_evaluation", urls, expected_sha256=trusted_concatenated_sha256, expected_size=trusted_concatenated_size, ) ``` Each item can instead be `{url, sha256, size}` to verify parts independently. If a command already has downloaded files, supply `path` as well as the logical catalogue URL. The local bytes are not fetched again, while reuse still compares the catalogue identities: ```python sources = [ {"url": url, "path": already_downloaded / url.rsplit("/", 1)[-1]} for url in urls ] ``` Strings are URLs; use a `Path` object for a local-only source. The full logical URL is hashed for identity, while receipts omit credentials, query strings and fragments. Avoid secrets in URL paths because paths are retained for display. ## MHCflurry compatibility Historical MHCflurry catalogues lack published hashes and record the exact, ordered URL list in `DOWNLOAD_INFO.csv`. An adapter can preserve that contract: ```python import csv import io buffer = io.StringIO(newline="") writer = csv.writer(buffer) writer.writerow(["url"]) writer.writerows([url] for url in urls) tree = install_archive( release_directory / download_name, sources, verified=False, extra_files={"DOWNLOAD_INFO.csv": buffer.getvalue()}, download_options={"timeout": timeout, "max_retries": max_retries}, ) ``` MHCflurry should treat a fast `inspect_archive(..., verify_files=False).status == "available"` using the same sources and `extra_files` as installed, not mere destination existence. Its `get_path` adapter should append member paths to `state.bundle`. The fast check validates the manifest and requested source identity without reading large model files on every path lookup. Explicit diagnostics can use the default `verify_files=True` for complete tree verification. This prevents an initialized store or interrupted install from being mistaken for a complete bundle. For a catalogue-wide adapter, `VersionedArchiveRegistry` supplies consistent version resolution, download, path, inspection and status methods while allowing MHCflurry's existing directory order: ```python from datacache import VersionedArchiveRegistry registry = VersionedArchiveRegistry( archive_definitions, store_path=lambda name, release: release_directory(release) / name, verified=False, # Historical catalogue entries have no published hashes. ) tree = registry.download( "models_class1_presentation", "2.3.0", show_progress=True, timeout=60, max_retries=3, ) rows = registry.status("models_class1_presentation") ``` `source_paths=` on `download` accepts one local file per ordered catalogue URL, covering MHCflurry's `--already-downloaded-dir` mode without changing source identity or bypassing transactional extraction. Existing MHCflurry directories with files in them are never adopted or overwritten, whether complete, partial, or unreceipted. A migration can continue to read a legacy directory and use archive stores only for new installations, or explicitly validate/import legacy content in application code. DataCache never claims it silently, including with `force=True`. ## Extraction policy DataCache parses tar, gzip-compressed tar, bzip2-compressed tar and xz-compressed tar archives using format detection. Leading `./` components are normalized and the `.` root directory entry is ignored, so archives created with `tar -cf archive.tar -C tree .` work. Before writing members it rejects: - absolute, parent-traversing, non-portable, and reserved DataCache paths; - symbolic links, hard links, devices, FIFOs and every other special member; - duplicate names, file/directory conflicts, and collisions ignoring case; - archives without a regular file; and - consumer `extra_files` that collide with archive content. Extraction writes regular files itself rather than calling `extractall`. Archive ownership, timestamps and permission bits—including executable and set-ID bits— are not applied. New files and directories use ordinary umask-derived modes. `max_members` and `max_extracted_size` can impose application-specific resource limits. They are enforced as each header arrives, before traversing the payload of a member that exceeds the limit; scanning stops at the first violation. The member count includes root directory entries even though they are not extracted. Local sources must be regular files; links and special inputs such as FIFOs are rejected without waiting for a writer. After extraction, the bundle's manifest records the assembled archive's observed hash and size, every part's observed hash, ordered source fingerprints, and a sorted hash/size manifest of every installed file. The complete consumer `extra_files` inventory is recorded separately. Manifest reading supports the same sizes as writing, including inventories larger than 1 MiB. Inspection rejects missing, changed, additional, linked or special files and directory changes. `ArchiveInspection` also exposes the redacted source URLs, fetch time, assembled size and recorded digest for downstream `list` and `info` commands. ## Trust and reuse Trusted verification requires either an assembled archive hash and size or a hash and size for every part. Hash-pinned content can be reused across different mirrors. `verified=False` permits historical unpinned sources; it verifies local consistency against observed receipts and requires the same ordered full source identities, but `ArchiveInspection.verified` remains false. A valid cache hit is entirely read-only. `inspect_archive` hashes the complete tree by default; `verify_files=False` checks only the manifest and source identity, reading none of the extracted tree, and never reports `verified=True`. Invalid installations require an explicit `force=True`; inaccessible installations propagate their permission error. Consumer metadata is part of the requested archive's identity, so a different, added, or removed metadata entry requires an explicit refresh. ## Publication and platforms Archive stores use the same layout as [bundles](bundles.md#how-bundles-are-stored): each install is a bundle in `bundles/`, named for its UTC install time, and the newest is the current one. Installs into one store take turns on a lock file beside the store. Downloads, concatenation, extraction, consumer metadata and the manifest are completed in private staging and read back and checked there. Only then is the tree renamed into `bundles/`, which publishes it in one step. A failed or interrupted install leaves the previous bundle current. Old bundles are kept so paths already returned to readers remain valid, until you delete them with `prune_bundles` or `registry.prune`; see [removing old bundles](bundles.md#removing-old-bundles). Installs lock a file beside the store (`flock` on POSIX, the `filelock` package on Windows) and publish with same-filesystem atomic renames, supporting ordinary local filesystems on Linux, macOS and Windows. Distributed coordination, arbitrary network-filesystem semantics, and durability after sudden power loss are outside the contract. With `download_options={"resume": True}`, remote part files persist privately across attempts and each remote part must have a trusted hash and size. A single part may use the assembled archive expectations. Retry, timeout and progress options otherwise have the same meanings as `fetch_file`; progress covers network transfer, not concatenation or extraction. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7087069 datacache-1.27.1/docs/bundles.md0000644000076500000000000003230215264220442016124 0ustar00iskanderwheel# Versioned datasets and bundles `VersionedDatasetRegistry` gives libraries a common way to resolve pinned data versions, install their files together, and inspect them offline. Dataset names, versions, URLs, and biological interpretation belong to the caller. DataCache owns the transfer, verification, and publication. Existing `Cache` and database APIs work independently; adopting a registry is optional. A dataset can contain one file, a records/manifest pair, or many source files: ```python from pathlib import Path from datacache import VersionedDatasetRegistry, get_cache_root registry = VersionedDatasetRegistry( { "reference": { "default_version": "2026-09", "versions": { "2026-09": { "records.fa": { "url": "https://data.example.org/2026-09/records.fa.gz", "decompress": True, "sha256": trusted_records_sha256, "size": trusted_records_size, }, "manifest.json": { "url": "https://data.example.org/2026-09/manifest.json", "sha256": trusted_manifest_sha256, "size": trusted_manifest_size, }, }, }, }, }, cache_root=Path(get_cache_root("openvax", "OPENVAX_DATA_CACHE")) / "datasets", ) # No directory creation, locks, network requests, or repairs: store = registry.store_path("reference") state = registry.inspect("reference") # Download if missing; verify and reuse if already installed: paths = registry.download("reference", timeout=60) records = paths["records.fa"] # Explicit repair or refresh; old returned paths remain usable: paths = registry.download("reference", force=True, timeout=60) ``` SHA-256 and size describe the installed bytes, after decompression. Use opaque, concrete data-version labels independent of Python package versions. A default must name a version in the supplied mapping; DataCache never resolves a remote `latest` alias. `get_cache_root` returns a string; wrap it in `Path` when joining paths with `/`. For a caller that already resolves versions, the lower-level operations are: ```python from datacache import install_bundle, inspect_bundle paths = install_bundle(source_directory, assets, download_options={"timeout": 60}) state = inspect_bundle(source_directory, assets) ``` `assets` is the same mapping of relative names to metadata. Names cannot escape the bundle, collide as files/directories or by letter case, or occupy DataCache's reserved metadata names. A file or directory at another asset's automatic provenance sidecar path is also rejected before installation starts. Source directories are stores: an existing empty directory is reported as `missing` and becomes the store on the first install, in place, so it keeps its owner, group and permissions. Installation never takes over a directory with files in it unless it is already a bundle store, even with `force=True`. Store generated indices and other derived outputs under a separate application-owned directory. Reinstalling sources does not visit those outputs. ## Consumer-selected store paths Select exactly one of `cache_root`, a zero-argument `cache_dir` root callable, or an exact `store_path(name, version)` callback. Root strategies retain `//`; the callback can keep an application's own layout, for example a version directory per release: ```python registry = VersionedDatasetRegistry( datasets, store_path=lambda name, version: application_root / version / "sources" / name, ) store = registry.store_path("reference") ``` The callback receives a validated concrete version, including when the caller omits it and selects the pinned default. Construction never calls it. The first lookup calls it once for every dataset version, without filesystem inspection or mutation, and the paths are reused afterwards, so callbacks should only compute paths. Each version needs its own store: a callback that gives two versions the same path raises `ValueError`, as does one that returns something other than a path. Inspection, installation and refresh all use the chosen store; DataCache owns everything inside it. The store's parent belongs to the application and may be a link, for example to another disk; the store itself is never a link. Installation creates missing parent directories and keeps a lock file in the parent, beside the store, so give the stores a parent of their own rather than a directory the application lists or cleans. Unfinished downloads, including resumable ones, stay in hidden `.staging-*` directories inside the store. Do not point this callback at a populated legacy data/index directory: even `force=True` cannot adopt a foreign directory, and installation raises `FileValidationError` saying so. Keep those old caches readable and deliberately install into a new managed source subdirectory instead. Model, biological naming and migration policy remain the application's responsibility. ## Status and trust `BundleInspection.status` is `available`, `missing`, `invalid` or `inaccessible`. `bundle` is the current bundle's directory, and `files` maps asset names to `FileInspection` objects from it. The `error` field retains the cause of a failed inspection. Explicit inspection hashes every asset by default. With trusted registry metadata, `verified=True` means every file matched the supplied SHA-256 just now. Every bundle also has its own manifest recording sizes, observed SHA-256 digests, source URLs, and fetch time. `inspect_bundle(directory)` can check those recorded hashes without a registry or network, but returns `verified=False`, for the bundle and for each file: a local receipt is not an independent authority. Display URLs omit credentials, queries and fragments; a separate SHA-256 fingerprint identifies the full source URL without storing that omitted text. Paths are retained, so do not use secret-bearing URL paths. Use `inspect_bundle(directory, assets, verify_files=False)` or `registry.inspect(name, verify_files=False)` for metadata-only checks. This checks ownership, the bundle's manifest, registry expectations, the required inventory, regular file types, readability and recorded sizes without reading payloads. Both bundle and per-file results have `verified=False`. Same-size corruption requires full verification to detect. Every registry method hashes by default, so `inspect`, `local_path`, `is_cached`, `status`, `download` and `ensure` agree about a corrupted bundle. Pass `verify_files=False` to any of them for metadata-only checks where speed matters more than detecting same-size corruption. On `install_bundle`, `registry.download` or `ensure` it skips payload reads on reuse only: new downloads are always hashed before they are published. All these checks remain offline and read-only. Archive registries differ: their `local_path`, `is_cached` and `status` have been metadata-only by default since they were added, and take `verify_files=True` for full checks. `verified=False` on installation or registry construction explicitly permits acquiring assets without trusted hashes/sizes. Their observed hashes are still recorded and checked on full verification, but cannot authenticate the original download. Use this mode only for upstream data without immutable integrity metadata. Without a trusted hash, reuse requires the same full URL fingerprint and decompression setting. Changing either reports an invalid installation; use `force=True` to acquire the newly requested data. Older receipts without source fingerprints still support offline consistency checks and trusted-hash reuse, but need an explicit refresh before reuse without trusted hashes. Trusted SHA-256 expectations identify the installed bytes. Libraries using the same root, dataset name, version and asset names can therefore share a verified bundle even when they use different mirrors, signed URLs or compression settings. DataCache checks the supplied hashes and sizes without downloading or rewriting a matching bundle. Cache reuse and inspection are read-only. A corrupt bundle requires `force=True`, which installs a new one. ## How bundles are stored ```text /// the store for one dataset version .datacache-store.json marks the directory as a DataCache store bundles/ 2026-09-30T14-11-05Z/ an older bundle 2026-10-08T17-02-42Z/ the current bundle: always the newest records.fa manifest.json .datacache-manifest.json what DataCache installed, and from where ``` A bundle is a complete set of files that belong together. Each bundle is named for the UTC time it was installed, and the newest is always the current one. Installing downloads and checks every file in a hidden staging directory, then renames that directory into `bundles/`. A rename happens completely or not at all, so readers see the old bundle or the new one, never a mix, and need no locks or write permission. Installs into one store take turns, using a lock file beside the store. Old bundles are kept, so paths returned by one install keep working after later installs. Forced refreshes therefore use more disk space until you [remove old bundles](#removing-old-bundles); DataCache never deletes them on its own. Never modify files in a bundle. Resolve paths once for a multi-file operation: a later lookup can return a newer bundle. Installation works on local filesystems with atomic sibling renames, on Linux, macOS and Windows; resumable downloads (`resume=True`) need POSIX. Shared caches use normal umask-derived permissions; readers need only read/search access. The staging directory stays private while files download, but the bundle inside it already has its final sharing permissions, so others can read it the moment it is renamed into place. A failed rename leaves resumable work in the private staging directory. Atomic publication covers process interruption, not guaranteed durability after power loss. Distributed coordination and arbitrary network filesystem semantics are outside this API. An unhandled termination can leave staging directories; only complete bundles are ever returned to callers. ## Removing old bundles Every forced refresh adds a complete bundle and keeps the old ones, so a multi-GB reference refreshed three times takes four times the space. Delete the old ones explicitly when nothing still uses them: ```python from datacache import list_bundles, prune_bundles list_bundles(store) # Every bundle, oldest first; the last is current. prune_bundles(store) # Keep only the current bundle. prune_bundles(store, keep=2) # Keep the current bundle and the one before it. registry.prune("reference", "110") # The same, through a registry. ``` `prune_bundles` returns the paths it deleted. The current bundle is always kept. Paths into deleted bundles stop working, so prune when no running program still uses them, for example from a cleanup command rather than during an analysis. Pruning takes the store's lock, so it never races an install. It works the same for archive and materialization stores. ## Local and predownloaded files An asset can name a local file with `path` instead of `url`; its identity is the file's `file://` URL: ```python assets = {"genes.gtf": {"path": "/data/custom/genes.gtf", "sha256": genes_sha256, "size": genes_size}} ``` To install from files someone already downloaded, while keeping the declared URLs as the assets' identity, pass `source_paths`: ```python paths = registry.download( "reference", "110", source_paths={"genes.gtf": "/downloads/Homo_sapiens.GRCh38.110.gtf"}, ) ``` Each local file is checked against the asset's trusted hash and size exactly as a download would be, and copied in; the original stays where it is. The manifest records the declared URL, so later installs and inspections treat the bundle as if it had been downloaded, including assets without trusted hashes. For an asset with `decompress=True`, the local file must keep its `.gz` or `.zip` suffix. ## Large raw assets For integrity-pinned HTTP assets that need no decompression or conversion, use `registry.download(name, resume=True)` or `install_bundle(..., download_options={"resume": True})`. An interrupted bundle keeps a private working directory per user and asset mapping. A later call with the same mapping reuses completed assets and resumes partial ones; the current bundle stays in place. Persistent partials require both hash and size. Changing the mapping chooses a different working directory. The normal non-resumable mode cleans its staging directory on handled failures. See [resumable downloads](downloads.md#resumable-http-downloads) for protocol, progress, disk-space and partial-discard details. ## Example and library notes The [runnable offline example](https://github.com/openvax/datacache/blob/master/examples/versioned_datasets.py) demonstrates single-file, paired-file and multi-file registries, generated SQLite indices, shared reuse, and inspection with all upstream sources removed. Notes for specific OpenVax libraries are in [integrating a consuming library](integration.md#notes-for-specific-libraries). ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7088094 datacache-1.27.1/docs/data.md0000644000076500000000000002504215264220442015404 0ustar00iskanderwheel# SQLite caches and custom transformations See the [API reference](api.md) for signatures, defaults, return values, and errors. ## DataFrames to SQLite Database helpers return open `sqlite3.Connection` objects, not file paths. The caller owns them. Use `contextlib.closing`: a connection's own context manager commits or rolls back, but does not close the connection. ```python from contextlib import closing import pandas as pd from datacache import db_from_dataframe frame = pd.DataFrame({"sample id": [1, 2], "label": ["case", "control"]}) with closing(db_from_dataframe( "samples.db", "samples", frame, primary_key="sample id", indices=[("label",)], cache_root="references", version=1, )) as connection: rows = connection.execute("SELECT sample_id, label FROM samples").fetchall() ``` `db_from_dataframes` accepts a mapping of table names to DataFrames, with `primary_keys` and `indices` mappings for each table. The `db_from_dataframes_with_absolute_path` helper takes an exact database path whose parent already exists. `Cache.db_from_dataframe` uses the selected cache root and also supports `version`, `overwrite`, and `show_progress`. ### Types and names - Integer and boolean columns use SQLite integers. Mixed numeric DataFrames retain integer precision instead of being coerced through a float array. - Nullable pandas integer, boolean, floating, and string dtypes are supported. `None`, `pd.NA`, `NaT`, and floating NaN values become SQL NULL. - Strings and categorical columns use TEXT. Datetimes are ISO-formatted text. Unsupported dtypes raise an error rather than guessing a representation. - SQLite integer values must fit a signed 64-bit integer. Oversized unsigned values raise an error and roll back; they are not silently rounded. - Column names must be nonempty strings. For compatibility, spaces become underscores. Primary-key and index names accept original or normalized names; nullability uses the same mapping. Colliding names and unknown constraint columns are rejected. - Primary keys cannot be NULL. Duplicate keys fail the build. Empty DataFrames with a supported column schema produce valid empty tables. Index names retain their historical spelling when available. If two requested indexes would share a name, the later one gets a numeric suffix so both are created. Matching cache hits do not add or rename indexes in existing databases; rebuild explicitly to restore an index omitted by an older release. DataFrame rows are converted as individual scalars and inserted in batches of up to 1,000. Building a database does not create another full list of all rows. The lower-level `DatabaseTable.rows` property still materializes a list for compatibility; use `iter_rows()` to consume it incrementally. ### Reuse and replacement A database is reused when its requested tables and version match, even if the supplied DataFrame's contents have changed. Table names match as SQLite compares them, so `Records` reuses `records`, but `Éclair` and `éclair` differ. Increment `version` when changing the schema or dataset, or pass `overwrite=True` to rebuild explicitly. Version metadata is written in the same transaction after rows and indices succeed. Existing databases rebuild in one SQLite transaction. A schema error, duplicate key, insertion failure, or handled interruption rolls back the old tables, rows, and version. The inode and its permissions remain unchanged, and existing connections can observe the committed update on their next read. SQLite locks serialize rebuilds; long-lived transactions can block a writer or cause a lock-timeout error, which leaves the previous database intact. Connections may be used across threads, but applications must coordinate simultaneous use of the same connection and should use separate connections for concurrent work. An explicit `overwrite=True` also removes existing views and their triggers so their names can be used for replacement tables. These removals are part of the transaction: failure restores the previous views and triggers as well as the tables. Ordinary cache hits and version rebuilds retain existing views. Rebuilds also handle virtual tables, such as SQLite FTS tables, whose internal shadow tables disappear automatically when the parent table is dropped. New databases are built in private sibling files, closed, and published using an atomic hard link that cannot overwrite a concurrent creator. A losing creator checks the winner's tables/version and reuses or transactionally rebuilds it. Failed builds remove their staging files. This requires a local filesystem supporting hard links and SQLite locking; unsupported publication fails without installing a partial database. A database path may be a symlink, including one whose target does not yet exist. Creation stages and publishes the database beside that target without replacing the symlink; the target's parent directory must already exist. Relative and chained links keep their filesystem meaning. Failed creation leaves the symlink intact and does not install a partial target database. `connect_if_correct_version(path, version)` returns `None` for a missing file or mismatched metadata, without creating an empty database. It closes rejected connections. `read_only=True` explicitly opens a matching database read-only. Version matching is a metadata check, not a checksum or proof of data validity. ### Upgrading existing caches No migration or blanket rebuild is required. Existing cache keys, SQLite metadata, versions, and table layouts remain readable. A matching database is opened before new-build input validation: its old schema and constraints are left intact, even if the current builder would create them differently. Reuse requires no directory creation or write access, and preserves file contents, inode, and permissions. New databases retain the same metadata format so older readers can open them too. Only an explicit `overwrite=True`, a changed version, or missing requested tables triggers rebuilding. Stronger constraints and numeric conversion rules apply then. If a rebuild rejects old inputs, the previous database remains intact; correct the source inputs before retrying. Older releases could store NumPy integers as BLOBs or round integers in mixed numeric DataFrames. Such values are not silently decoded or rewritten on upgrade: their intended type cannot always be inferred, and lost precision cannot be recovered from the database. If an affected database needs correction, rebuild it from the original data with `overwrite=True` or a new cache version. Unaffected databases need no action. ## CSV to SQLite `fetch_csv_db("records", url)` infers both filenames when omitted. An explicit `db_filename` also works without `csv_filename`. Supply parser options directly and download options in `download_options`, as described in the [progress guide](progress.md#csv-options). A `cache_root` in that dictionary is used for both the downloaded CSV and its database. The inferred database name records the row count and each column's name and type. Because column names come from the downloaded data, they are spelled out only when every character is valid in file names on all supported platforms and the name leaves room for SQLite's `-journal` file; otherwise the schema is named by a digest. Column names therefore never add directories or leave the cache directory (earlier releases turned a column named `m/z` into a subdirectory). A matching database an older release stored under its historical name is still reused in place. A new `version` rebuilds it under the new name and leaves the older copy alone, since another process may still have it open; a warning names it, and once no older release uses the cache you can remove it by hand or with `Cache.delete_all`. Explicit CSV filenames keep their historical database key, including names ending in `.csv.gz`, whenever that key is a valid file name on every supported platform. If `db_filename` is supplied and its tables/version match, the database can be reused offline without a source CSV or parsing it again. Supplying `expected_sha256`, `expected_size`, or `force` in `download_options` still requests source validation or refresh. Refreshing a source alone does not rebuild a matching database; also increment `version` when its data changes. ## Custom single-file transformations For versioned dependencies, multiple outputs, resumable compressed inputs or automatic post-publication source cleanup, use the [derived-artifact materialization API](materialization.md). The existing helper below retains its compatible single-file behavior. ```python from pathlib import Path from datacache import fetch_and_transform def uppercase(source_path, output_path): text = Path(source_path).read_text().upper() Path(output_path).write_text(text) return text text = fetch_and_transform( transformed_filename="uppercase.txt", transformer=uppercase, loader=lambda path: Path(path).read_text(), source_filename="source.txt", source_url="https://example.org/source.txt", subdir="my-library", ) ``` The transformer receives an absent output path inside a private temporary directory on the destination filesystem. Its basename and extension match the final output. It must write and close one regular file at that path. A no-op transformer raises an error; an intentionally written empty file is valid. Only successful output is published. Failures remove partial files and leave any previous transformed output intact. On a cache hit, only the loader runs. `force=True` reruns the transformation; it does not implicitly redownload its source. Use `download_options={"force": True}` to refresh the source as well. New source downloads and transformed outputs honor `subdir` and `cache_root`. Other settings can be passed through `download_options` without changing parser/transformer code. Older versions accidentally stored the source in the default cache when a `subdir` was supplied. Without an explicit `cache_root`, the helper reuses that exact legacy source if the requested directory has no source of its own. It validates supplied integrity expectations and does not move or chmod the old file. New downloads go into the requested directory. For compatibility, a nonempty `subdir` continues to default source decompression to enabled; set `download_options={"decompress": False}` to override it explicitly. Return in-memory results from the transformer and loader. A transformer that returns its output path directly receives the final path back; embedded paths inside arbitrary objects cannot be rewritten. This contract covers one file, not directory trees, multiple sidecar files, or open database connections. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6516237 datacache-1.27.1/docs/dev/0000755000076500000000000000000015264225206014727 5ustar00iskanderwheel././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7089787 datacache-1.27.1/docs/dev/documentation-style.md0000644000076500000000000000274715264220442021267 0ustar00iskanderwheel# Documentation style Follow the [PyEnsembl writing guide](https://github.com/openvax/pyensembl/blob/edd183d/docs/dev/documentation-style.md) and the [mhctools example](https://github.com/openvax/mhctools/tree/master/docs). Put installation and the first useful example on the home page. Give complete imports and concrete inputs, explain the result, and state prerequisites before the reader runs a command. Show output only when it has been checked. Label replaceable paths and templates. Order guides by common tasks before advanced setup and implementation details. Keep desktop sidebar groups expanded initially and individually collapsible. Split a page when a substantial task or reference needs its own explanation. Preserve useful old links and anchors when moving content. Use ordinary text for concepts and library names; reserve inline code for literal commands, API names, parameters, paths and values. Keep qualifications, experimental status and scientific limitations beside the claims they qualify. Use system fonts, a readable line length and restrained headings. Let wide tables and code blocks scroll inside their containers. Avoid repetitive navigation lists in prose and excessive callouts. Install requirements-docs.txt and run ./docs.sh before opening a PR. Review the home page and a reference page at desktop and narrow widths. Run the repository's required lint and test commands and execute complete examples. Documentation dependencies are separate from runtime dependencies. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7091272 datacache-1.27.1/docs/downloads.md0000644000076500000000000005525415264220442016475 0ustar00iskanderwheel# Download and cache reference See the [API reference](api.md) for signatures, defaults, return values, and errors. ## Verified downloads Use `destination` to install a single file at an exact path, including its filename. Supply trusted integrity metadata to check both cached files and new downloads: ```python from datacache import fetch_file, validate_file, FileValidationError path = fetch_file( "https://example.org/releases/v1/records.tsv.gz", destination="/data/references/v1/records.tsv", decompress=True, expected_sha256=release_metadata["installed_sha256"], expected_size=release_metadata["installed_size"], timeout=30, ) # Read-only validation: no requests, directory creation, locks, or repair. validate_file(path, expected_sha256=release_metadata["installed_sha256"]) ``` `destination` accepts a string or `pathlib.Path` and cannot be combined with `filename`, `subdir`, or `cache_root`. Existing calls using the default cache continue to work and can also supply `expected_sha256` and `expected_size`. Parent directories are created only when fetching a missing file or explicitly refreshing it. A valid cached file can be reused offline in a readable, non-writable installation. Both expectations always describe **installed bytes**, after decompression or HTML-to-CSV conversion. They do not describe HTTP wire bytes or a compressed archive when its contents are being installed. To verify and retain an archive at any output name, pass `raw=True`. This also disables HTML-to-CSV conversion: ```python path = fetch_file( archive_url, destination="reference.bin", raw=True, expected_sha256=release_metadata["archive_sha256"], expected_size=release_metadata["archive_size"], record_provenance=True, ) ``` `Cache.fetch` accepts the same `raw=True` option with `filename`. Raw mode is incompatible with `decompress=True`. It preserves the downloaded payload after normal HTTP transfer decoding, not encoded HTTP wire bytes. Hashes and sizes refer to that payload. A cache hit still uses the supplied local validation checks; changing transformation options does not replace an existing output. Use distinct names, trusted expectations, or `force=True` when switching modes. ### Archives and decompression Without `raw=True`, keeping the source's `.gz` or `.zip` suffix at the destination and leaving `decompress=False` retains an archive as before. Decompression also depends on how the output is named: - With an inferred filename, archives are retained by default, including URLs with query strings or fragments; their existing cache keys are preserved. `decompress=True` uses a distinct key for the decompressed contents, keeping the full URL in the key's digest. When a download endpoint's inferred filename has no removable archive suffix, `.decompressed` distinguishes its output. - With an explicit `filename` or `destination`, a missing compression suffix still implies decompression for compatibility unless `raw=True` is supplied. `decompress=True` explicitly requests decompression while preserving an explicit destination's exact name. Format detection prefers a supported extension on the URL path (case-insensitively). For download endpoints without a supported path extension, a filename in the final query parameter is also recognized for compatibility, such as IEDB's `downloader.php?file_name=doc/data.zip`. Bare format hints such as `?format=.gz` and fragments do not select a format. A ZIP file installs one member; no archive paths are extracted. DataCache selects, in order: 1. The member stored at the output filename. 2. A member with that name in any folder, ignoring letter case (such as `release/data.csv` or `Data.CSV` for `data.csv`), preferring the one nearest the archive root and then an exact-case match. 3. The largest non-directory member. A warning is logged when the output was named explicitly and this was a guess among several. HTML-to-CSV conversion requires an explicit `filename` or `destination` ending in `.csv`. Query strings and fragments in inferred cache keys never request conversion; those downloads retain their original HTML bytes. ### Validation errors and empty files A size or SHA-256 mismatch raises `FileValidationError`, with the path and expected/actual values in its message. A corrupt cache hit does not trigger a download: call `fetch_file(..., force=True)` to explicitly attempt replacement. `validate_file` raises `FileNotFoundError` for missing files and propagates permission errors; it rejects non-regular files. Fetching propagates transport, decompression, and filesystem errors so applications can translate them. Expectations are optional; omitting them provides no integrity guarantee. An empty installed file is rejected even without expectations, because a complete but empty response, such as a withdrawn upstream record, would otherwise be cached as a valid file and parse to zero records much later. An empty HTTP response is retried like a transient failure, then raises `FileValidationError`; nothing is published and an existing file stays in place. After decompression, an empty archive member is rejected the same way. An empty file already in the cache is an invalid hit, which `force=True` replaces. Pass `allow_empty=True`, or `expected_size=0`, when an empty file is the correct result. ## Provenance Pass `record_provenance=True` to `fetch_file` or `Cache.fetch` to record where a download came from. After publishing, datacache writes a hidden record, `..datacache.json`, beside the file. It holds the source URL, the fetch time, the size, and the SHA-256 when `expected_sha256` verified it. The URL is stored without its user name, password, query string, or fragment, since signed URLs carry secrets there; its path is kept as is, so do not record URLs that embed a secret in the path, as some share links do. The record has the file's permissions, and sharing the file with `make_file_readable` shares it too. `inspect_file`, `Cache.inspect`, and `inspect_files` report `source_url`, `fetched_at`, and `recorded_sha256`, but only while the file's size and modification time match the record. `recorded_sha256` is a record, not a check of the current bytes: an in-place edit that keeps the size and modification time, or a forged record in a shared cache, would still report it. `verified` therefore keeps its meaning, a supplied SHA-256 matching the bytes now; pass `recorded_sha256` back as `expected_sha256` to check them. A missing, unreadable, or malformed record reads as unrecorded, and inspection never writes. Recording is off by default because a caller that downloads to a temporary name and then moves the file would leave the record behind. A record is written only by a download, never by a cache hit, and any later replacement of the file removes the previous record first, whether or not it records a new one. `Cache.delete_url` removes records with their files. Recording is best effort: failing to write a record never fails the download. ## Transient HTTP failures HTTP and HTTPS downloads retry temporary failures by default, with at most three attempts (the initial request plus two retries). This includes HTTP 408, 429, 500, 502, 503, and 504 responses, connection failures, timeouts, and interrupted response streams. Configure the policy on `fetch_file` or `Cache.fetch`: ```python path = fetch_file( "https://example.org/releases/v1/records.tsv.gz", destination="/data/references/v1/records.tsv", decompress=True, expected_sha256=release_metadata["installed_sha256"], timeout=30, max_retries=2, retry_backoff=1.0, retry_max_delay=30.0, ) ``` `max_retries` counts additional attempts; set it to `0` to disable retries. `retry_backoff` is the first delay in seconds and doubles after each retry, capped by `retry_max_delay`. Defaults therefore wait one second and then two seconds. `Retry-After` supports both integer seconds and HTTP dates. When its wait fits within `retry_max_delay`, the larger of that wait and the backoff is used. If the server requests a longer wait, the original error is raised immediately instead of retrying sooner than requested. Malformed header values are ignored. Both delay settings must be finite non-negative real numbers; accepted values are normalized to Python floats before use. Each attempt downloads from the beginning into a fresh private staging file. Failed responses are closed and partial files are removed before waiting or retrying. An existing destination remains intact until a successful transfer has been transformed, validated, and published. Retry warnings report the attempt count, failure category, and delay; exhaustion raises the original Requests exception, preserving its response and cause. Permanent HTTP failures such as 404, TLS errors (including proxy-wrapped TLS failures), local filesystem errors, progress callback exceptions, and transformation/integrity/publication errors are not retried. Local file and FTP transfers remain single attempts. Cache reuse and read-only inspection make no requests and do not wait. Every HTTP and HTTPS request, including resumed range requests and `fetch_bytes`, sends `User-Agent: datacache/ (+https://github.com/openvax/datacache)`. Some hosts, IEDB among them, refuse Requests' default `python-requests` User-Agent. Progress byte counts describe the current attempt and may decrease after a restart. The same `timeout` is passed to each request; Requests timeouts govern connection/read inactivity, not an overall elapsed-time deadline. Retry count and waiting time are bounded independently of the transfer duration. ## Byte limits Use `max_bytes` when a download's exact size isn't known yet but its largest acceptable size is, for example to keep a first download of a metadata file within a disk budget: ```python from datacache import fetch_file path = fetch_file( "https://data.example.org/inventory.tsv", destination=cache / "inventory.tsv", raw=True, max_bytes=30 * 2 ** 20, # Refuse anything over 30 MiB. ) ``` The limit is checked before every write, so no file this fetch writes ever holds more than `max_bytes` bytes: - The download is the body after HTTP transfer decoding, such as gzip `Content-Encoding`, counted as it arrives, so it holds whether or not the server sends `Content-Length`. A `Content-Length` over the limit is refused before the body is read, unless chunked transfer makes it meaningless. - With `decompress=True`, the installed file is limited too, so a download and its decompressed file together never need more than twice `max_bytes`. The compressed download must fit as well; leave room for it when the data barely compresses. - A larger transfer raises `FileValidationError` and isn't retried. Any existing file stays as it was, and temporary files are removed. - Cache hits write nothing, so the limit doesn't apply to them. `expected_size` keeps its exact meaning and must not exceed `max_bytes`. HTML-to-CSV conversion can't be bounded while it writes, so `max_bytes` rejects it; pass `raw=True` to keep the HTML itself. Like other options that can't work together, these raise `ValueError` before anything else, cache hits included. `max_bytes=0` needs `allow_empty=True`. `fetch_bytes` accepts `max_bytes` for bodies held in memory: the result never grows past it, and the body is read in chunks no larger than the limit. `install_bundle`, `install_archive` and `materialize` accept it in `download_options` for each download they make, and check every size they already know against it before creating anything. This works the same on every platform, unlike `resume=True`, which needs a POSIX filesystem. ## Small metadata and freshness `fetch_bytes` returns a resource's bytes in memory with the same transport, TLS trust and retry policy as `fetch_file`, without writing to disk. It suits small resources such as directory listings, and caches that are read-only. Metadata that changes upstream, such as a list of releases, can be cached and refreshed periodically. `expire_after` is named as in requests-cache, and `return_stale_on_error` behaves like the HTTP `stale-if-error` directive: ```python path = fetch_file( "https://example.org/releases/", destination="/data/references/releases.html", raw=True, timeout=10, expire_after=86400, # seconds or a datetime.timedelta return_stale_on_error=True, validator=require_release_links, ) ``` - `expire_after` downloads a cached copy again once it is older than the given time, or once it no longer validates. Age comes from the provenance record's fetch time when there is one (`record_provenance=True`), otherwise from the file's modification time, which publication sets when the download is written. - `return_stale_on_error` returns the cached copy, with a logged warning, when a refresh fails, including a rejected validation. Without a valid cached copy the error propagates; so does a progress callback's exception, which cancels the fetch. - `validator(path)` rejects content that a successful transfer can still get wrong, such as an HTTP 200 error page from a proxy, by raising or returning `False`. It runs on the staged bytes before publication, so a rejected download never replaces the cached copy, and on cache hits. ## Read-only cache inspection Path lookup, presence, and integrity checks have different contracts: ```python from datacache import Cache, expected_path, file_exists, inspect_file, inspect_files root = "/data/references/v1" path = expected_path(filename="records.tsv", cache_root=root) present = file_exists(filename="records.tsv", cache_root=root) result = inspect_file(path, expected_sha256=release_metadata["installed_sha256"]) if result.status == "available" and result.verified: process_records(result.path) # The object API uses the same paths and validation contract. cache = Cache("references", cache_root=root) result = cache.inspect(filename="records.tsv", expected_sha256=release_metadata["installed_sha256"]) # Inventory required files using trusted, caller-supplied metadata. installation = inspect_files(root, { "records.tsv": {"expected_sha256": release_metadata["installed_sha256"]}, "manifest.json": {"expected_sha256": release_metadata["manifest_sha256"]}, }) ``` `expected_path`, `resolve_path`, and `Cache.local_path(download=False)` only compute paths, with no filesystem access. `file_exists` and `Cache.exists` check presence without requiring a readable regular file: directories count as present, broken symlinks count as absent, and permission errors propagate. None of these operations creates a directory or a lock file. `inspect_file`, `inspect_files`, and `Cache.inspect` return results with `path`, `status`, `verified`, and `error` attributes: | Status | File inspection | Required-file inventory | | --- | --- | --- | | `available` | Readable regular file matching supplied metadata | Every required file is available | | `missing` | File absent | Cache root absent | | `corrupt` | Wrong file type or size/hash mismatch | Wrong root type, corrupt file, or incomplete installation | | `inaccessible` | Permission or other filesystem error | Root or required file cannot be inspected | `verified` is true only when a supplied SHA-256 digest matched; presence, readability, or a size check alone does not establish verified integrity. `error` retains the original filesystem or validation exception when unavailable. Invalid integrity arguments raise `ValueError` rather than reporting a cache problem. `inspect_files` also returns a `files` mapping of individual results; for example, a missing manifest makes the installation `corrupt` while that manifest's individual status is `missing`. Inaccessible files take precedence over corrupt or missing files in the aggregate result. Required names must be normalized relative paths; nested names are supported. Use `{}` or `None` for an entry without integrity metadata. The required mapping must be nonempty. Inspection only reads local files. It never downloads, writes manifests, creates locks, or attempts recovery, so a valid read-only version and a missing sibling version can be inspected independently. The caller supplies required files and trusted integrity metadata; these individual-file helpers do not discover versions. Symlinks are followed as in ordinary file access. Inventory is not a snapshot across concurrent external changes or a multi-file installation mechanism. Use [versioned bundles](bundles.md) for atomic multi-file installation and inspection of one complete bundle. `cache_root` accepts a string or `pathlib.Path` and names the actual directory containing cached files, overriding the platform location selected by `subdir`. Relative roots remain relative to the current working directory. It is supported by `fetch_file`, `expected_path`, `file_exists`, `resolve_path`, `build_path`, and `Cache`. An exact `destination` is mutually exclusive with `cache_root`. Default cache locations and filename normalization remain the same. `build_path` still creates parents; use `resolve_path` for pure lookup. Creation and repair remain explicit operations: use `fetch_file` or `Cache.fetch`, supplying integrity metadata and `force=True` for replacement. `Cache.fetch` validates every reuse and respects different filenames for the same URL. Its database and deletion methods also use the selected cache root. Database paths preserve the filesystem meaning of symlinks followed by `..`. `Cache.delete_all()` clears the root's contents while preserving the directory and its permissions, including when the root is `.` or a symlink. Symlinks inside the cache are removed without clearing their external targets. ## Resumable HTTP downloads The simplest choice is `resume="auto"`: it resumes interrupted transfers when it can, and otherwise downloads normally, so callers need no `os.name` checks or error-message matching. It downloads in full, discarding any partial, on Windows, when the size isn't known, with decompression or a `validator`, when the server turns out not to support resuming, and when the filesystem can't hold private resume state or locks. Learning that a server can't resume costs one extra request, whose body isn't read: ```python path = fetch_file(url, destination=dest, raw=True, expected_size=size, resume="auto") ``` The rest of this section describes `resume=True`, which instead raises whenever resuming isn't possible. Use `fetch_file(..., resume=True, expected_sha256=sha256, expected_size=size)` (or the same options on `Cache.fetch`) for large immutable raw HTTP/HTTPS files. `expected_size` is required. Prefer a trusted `expected_sha256` when available. For sources such as Ensembl that do not publish SHA-256 digests, omit the hash: ```python path = fetch_file( ensembl_dna_url, destination="reference.download", raw=True, resume=True, expected_size=ensembl_dna_size, ) ``` Without `expected_sha256`, every accepted response must supply a syntactically valid strong ETag. Missing or weak ETags (including a Last-Modified value alone) raise `FileValidationError` before response bytes are retained or published. Strong ETags prevent joining bytes from different server representations; they are not independently verified checksums. Callers can still apply their source's checksum to the completed archive. Size-only cache hits remain offline and check only the local byte count, not remote freshness or same-size corruption. Optional provenance records no verified SHA-256 unless the caller supplied one. Use `raw=True` for an arbitrary destination name, or keep the source archive's suffix. Resumable decompression and HTML conversion are not supported. Resume is off by default and requires a POSIX local filesystem. Each destination gets a private, owner-only working directory beside it, with one bounded partial file, metadata and a permanent advisory lock. Threads and processes serialize on that destination for the same user; other users retain separate private state. A valid destination remains intact throughout transfer, verification, and publication. With an expected SHA-256, complete partials are verified and reused without network; oversized or corrupt ones restart. Without one, a complete-size partial restarts to obtain a fresh validated response, and an incomplete partial without a saved strong ETag is discarded. Interruptions and exhausted transport retries keep safely written partial bytes for the next call. Range requests use `Accept-Encoding: identity`. DataCache checks the exact `Content-Range` offset and total, carries `If-Range` for a strong ETag, and only appends bytes from a matching representation. A 200 response replaces the partial; incompatible ranges, changed validators, or 416 responses restart it. When a trusted SHA-256 is supplied, downloads can also resume without strong ETags, with Last-Modified changes triggering restarts and the final digest authorizing publication. The final byte count is always checked. HTTP range behavior follows [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-range-requests). Progress reports bytes present in the current transfer, including the retained prefix, and the trusted total size. A resumed retry continues its byte count; a server-required restart resets it. Callback failures propagate without being retried. The existing bounded HTTP retry/backoff options apply. Call `discard_partial(destination)` to explicitly discard this user's partial bytes and metadata. It waits for any active transfer, preserves the installed file, and retains the small lock directory for coordination. Missing state is a no-op. `force=True` refreshes the installed file but can reuse an independently verified complete partial; call `discard_partial` first to require a new transfer. Allow space for the old destination, up to `expected_size` partial bytes, and a second file of that size during publication. Copying verified bytes into ordinary staging keeps persistent partials private even when publication fails after setting shared file permissions. No credentials or raw signed URLs are stored in the private metadata; a URL hash identifies the source. Changing the URL or integrity expectations starts a new transfer. ## Downstream compatibility The private `_download_and_decompress_if_necessary` entry point, used by pyensembl, retains its pre-1.8 literal-URL format inference when transform flags are omitted. In particular, query/fragment-bearing archive URLs retain the same bytes under pyensembl's existing cache keys. Explicit transform flags and the public `fetch_file` API retain the parsed-URL behavior documented above. Existing pyensembl calls through this entry point receive the same default [retry policy](#transient-http-failures) without changes to downstream code. The IEDB download endpoints used by pepdata continue to decompress archives named at the end of the URL query into the requested CSV filenames. New integrations should use the public download and inspection APIs. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7092173 datacache-1.27.1/docs/file_registry.md0000644000076500000000000000725215264220442017345 0ustar00iskanderwheel# Fixed-path versioned files `VersionedFileRegistry` supports applications whose single-file datasets already live at `///` with a root `manifest.json`. It reuses those files in place. Use `VersionedDatasetRegistry` and bundles when you need several files installed together as one bundle. ```python from pathlib import Path from datacache import VersionedFileRegistry registry = VersionedFileRegistry({ "reference": { "filename": "records.tsv", "default_version": "2026-09", "urls": {"2026-09": "https://data.example.org/2026-09/records.tsv.gz"}, "description": "Reference records", }, }, cache_dir=lambda: Path("existing-cache")) expected = registry.local_path("reference") # works before installation; no writes status = registry.status() # presence, URL/hash receipt, offline path = registry.ensure("reference", timeout=60, record_provenance=True) ``` The dataset mapping and cache-root callable remain application-owned. The root is resolved for each operation, so environment-backed callables may change it. Use trusted definitions. Writers serialize per root using a cross-platform file lock; cache hits and inspection never acquire or create that lock. Dataset names, version labels and filenames must describe paths within the selected root. Ordinary cache hits check presence and reuse the old path without reading its bytes, networking, hashing or rewriting receipts. This deliberately preserves legacy behavior and is not integrity verification. Explicit `expected_sha256` or `expected_size` options validate reused files through `fetch_file`; invalid files require `force=True`. `ensure` accepts the same options. New acquisition forwards `fetch_file` options, including raw/decompression, timeout, bounded retries, progress, provenance and resume. Integrity expectations describe the installed bytes. Human cache-status messages belong in the calling application. After downloading, the registry hashes the installed file in bounded chunks and atomically updates its root JSON receipt. Entries are keyed by dataset name and retain `version`, `url`, `path`, `bytes`, `sha256` and `downloaded_at` (UTC). The URL is the caller's original URL for legacy compatibility; avoid credentials or signed URLs in these definitions. Local receipts describe observed bytes, not independently trusted checksums. Missing, unreadable or malformed JSON receipts are treated as empty, matching the legacy registry. `status()` returns `name`, `description`, `default_version`, `available_versions`, `cached`, `cached_version`, `url`, `bytes`, `sha256`, `downloaded_at` and `path`. Presence and path refer to the pinned default. `cached_version` comes from the most recent download recorded for that dataset, which may be another version; `url`, `bytes`, `sha256` and `downloaded_at` are filled in only while the pinned default's file exists, and are otherwise `None`. This compatibility view does not certify freshness or integrity; use `inspect_file` for the selected path. Unknown dataset/version errors and acquisition failures use `error_cls` (`RuntimeError` by default). Acquisition failures retain their original cause. Filesystem failures while hashing or writing the receipt propagate directly. Failed acquisition leaves the previous file and receipt intact. The file and receipt are separate publications: if receipt publication fails after a valid download, the new file remains installed and the prior receipt remains intact. Concurrent registry writers serialize acquisition and receipt updates, so downloading different datasets cannot discard each other’s receipts. These are the established single-file semantics, not the guarantees of bundles. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7093048 datacache-1.27.1/docs/index.md0000644000076500000000000001033215264220442015576 0ustar00iskanderwheel# DataCache Download a dataset once, verify its contents, and reuse the local file on later runs. DataCache also manages pinned bundles, archive trees and SQLite caches for applications such as PyEnsembl. ## Install ```sh python -m pip install datacache ``` Python 3.9 or later is required. Add `datacache[html]` if you need HTML-table conversion. Progress bars are available with `show_progress=True`. ## Cache and verify a file This complete example uses a local file and a temporary cache, so it runs offline and cleans up after itself. HTTP, HTTPS and FTP URLs work through the same `fetch()` interface. ```python import gzip import hashlib from pathlib import Path from tempfile import TemporaryDirectory from datacache import Cache with TemporaryDirectory() as directory: root = Path(directory) contents = b">reference\nACGT\n" source = root / "reference.fa.gz" source.write_bytes(gzip.compress(contents)) cache = Cache("references", cache_root=root / "cache") path = cache.fetch( source.as_uri(), filename="reference.fa", expected_sha256=hashlib.sha256(contents).hexdigest(), ) print(Path(path).read_text().strip()) print(cache.inspect(filename="reference.fa").status) source.unlink() print(cache.fetch(source.as_uri(), filename="reference.fa") == path) ``` ```text >reference ACGT available True ``` The output is decompressed because the explicit filename omits the gzip suffix. The hash describes the installed, decompressed bytes. The final call reuses the cached file after the source has been removed. For real data, obtain the expected hash from trusted release metadata. Computing a hash from an untrusted download does not establish its authenticity. ## Use your dataset This template requires your own dataset URL and destination name: ```python from datacache import Cache cache = Cache("my-project") path = cache.fetch( "https://example.org/releases/v1/records.tsv", filename="records.tsv", timeout=30, show_progress=True, ) ``` Existing files are reused until you request `force=True`, or once they are older than `expire_after`. DataCache doesn't ask the server whether a file has changed. Supply trusted integrity expectations or a `validator` when you need to validate a cached hit; invalid hits raise an error rather than silently downloading replacements, unless `expire_after` asks for periodic refreshes. Read [download and inspection options](downloads.md) for exact destinations, decompression, retries and resumable transfers. Use [pinned bundles](bundles.md) when a dataset contains several assets that must be installed together. ## Choose an interface Pick by what you need: | You need | Use | You get back | | --- | --- | --- | | One file, downloaded once and reused | `fetch_file()` or `Cache.fetch()` ([guide](downloads.md)) | Its local path | | One file that changes upstream, refreshed now and then | `fetch_file(expire_after=...)` ([guide](downloads.md#small-metadata-and-freshness)) | Its local path | | One file per version, at a fixed path | `VersionedFileRegistry` ([guide](file_registry.md)) | Its local path | | A small resource held only in memory, such as a listing | `fetch_bytes()` | Its bytes; nothing is cached | | Several files that belong together, pinned per version | `VersionedDatasetRegistry` ([guide](bundles.md)) | Their paths, in one bundle | | A tar archive unpacked into a directory | `VersionedArchiveRegistry` ([guide](archives.md)) | The unpacked directory | | Files your own code builds from downloads | `materialize()` ([guide](materialization.md)) | The built files' paths | | Tables or SQLite databases | `fetch_csv_dataframe()`, `db_from_dataframe()` ([guide](data.md)) | A DataFrame or a connection | Installed files can be checked offline without changing them: `inspect_file()` for a file, `inspect()` on a dataset or archive registry, and `inspect_materialization()` for built files. Several files, archives and built files are installed all at once or not at all, and old copies are kept until you delete them with `prune_bundles()`. The [interface selection reference](reference/choosing.md) covers the main entry points and the publication guarantees. The [API reference](api.md) gives every signature, default, result and error. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7094083 datacache-1.27.1/docs/integration.md0000644000076500000000000002046715264220442017024 0ustar00iskanderwheel# Building downstream libraries on DataCache See the [API reference](api.md) for signatures, defaults, return values, and errors. Use the public `Cache`, `fetch_file`, inspection, registry and database APIs for new integrations. Private helpers can change between releases, except the compatibility entry points named under [notes for specific libraries](#notes-for-specific-libraries). ## Recommended contracts 1. Let callers choose a cache root. Do not create it merely to check whether a dataset is installed: use `expected_path`, `Cache.local_path`, or inspection. 2. Keep presence separate from integrity. `exists` answers whether a path is present; `inspect` and `validate_file` check readability and supplied trusted hash/size. Inspect every required member of a release. 3. Pass trusted metadata for installed bytes, after transformation. Reuse the same expectations across concurrent writers to a destination. 4. Make installation and repair explicit. Do not automatically force a refresh merely because a cache is corrupt, inaccessible, or unavailable offline. 5. Forward `timeout`, retry settings, `progress_callback`, and `show_progress` where your library exposes downloads. Keep logging configuration in the top-level application. 6. Use database versions to identify data/schema changes. Close connections when finished; use separate connections for independent concurrent work. 7. Let original network and filesystem exceptions propagate, or retain them as causes when adding domain-specific context. `FileValidationError.path` and `.reason` provide structured validation details. 8. Use `install_archive` when a released tar file owns a directory tree. Treat only `inspect_archive(...).status == "available"` as installed and resolve member paths from the returned bundle directory, never from the store. ## Reusable downstream toolkit Downstream libraries should keep domain catalogues, CLI wording and generated indexes, while delegating acquisition and source-store lifecycle to DataCache: | Downstream shape | DataCache API | Shared behavior | | --- | --- | --- | | One established file path, as in PyEnsembl annotation/FASTA sources | `fetch_file` or `VersionedFileRegistry` | Atomic download, decompression, retries, resume, progress, integrity and provenance | | A released tar tree, as in MHCflurry weights/data | `VersionedArchiveRegistry` or `install_archive` | Ordered parts, safe extraction, consumer receipts, all-or-nothing publication and status | | Several separately published files that belong together | `VersionedDatasetRegistry` or `install_bundle` | Pinned versions, all-or-nothing publication and inspection | All explicit download methods forward `show_progress`, callback, timeout and bounded retry options to the shared transport. Registry construction, status, inspection and local-path resolution are offline. Applications can therefore offer consistent `list`/`info`/`download` behavior without maintaining private network or extraction implementations. ## Shared OpenVax cache OpenVax packages share one cache so that published data, such as test datasets, is downloaded once for all of them. Select its root with `get_cache_root`, listing the package's own variable before the shared one: ```python root = datacache.get_cache_root("openvax", "OSTEOSARC_CACHE", "OPENVAX_DATA_CACHE") ``` `OPENVAX_DATA_CACHE`, when set, is the root itself; otherwise the platform cache directory named `openvax` is used. Store content-addressed files under `/objects/sha256/`, as osteosarc and Vaxrank do, so identical files are shared. Keep each package's own receipts and derived files in a subdirectory named after the package. Importing datacache does not import pandas, numpy, or requests, so command-line tools can resolve the root cheaply; those are loaded only by the functions that need them. ## Compatibility and scope Existing positional download and database arguments remain supported. New settings are keyword-only. Filename normalization and archive-selection rules remain compatible; see the [download reference](downloads.md). Cache hits preserve existing bytes, names, schemas, versions, and permissions. New-build validation runs only when a database actually needs rebuilding. The [upgrade guide](data.md#upgrading-existing-caches) explains legacy source reuse and deliberate repair of previously corrupted data. Optional `make_file_readable` maintenance lets file owners share selected old `0600` files without changing them during normal cache access. The existing pandas dependency range remains unchanged. CI exercises pandas 1.4.4, 1.5.3, and current versions; older versions in the historical dependency range are not all tested on modern Python. Nullable dtype support depends on the installed pandas version. DataCache caches by path or database version, not by tracking remote content changes. Refreshing a URL does not invalidate arbitrary downstream artifacts. Libraries own dataset versions and their dependency relationships. tqdm is installed by default; enable displays with `show_progress=True`. HTML table conversion requires the `html` extra. The library does not install logging handlers or change the process umask. Download retry counts and waits are bounded, while `timeout` limits connection/read inactivity per attempt, not total elapsed time. Use [versioned bundles](bundles.md) when named files must be installed together, or [archive installation](archives.md) when a tar archive defines the complete tree. Both work on Linux, macOS and Windows; resumable downloads need POSIX. Individual atomic downloads do not provide multi-file transactions or distributed locking. Unhandled termination can leave staging files behind. SQLite operations depend on SQLite's filesystem locking and journaling. New database publication needs hard-link support. Validate deployment-specific network filesystems and ACL requirements in the consuming application. The runnable [offline example](https://github.com/openvax/datacache/blob/master/examples/basic_usage.py) exercises the public download, transform, inspection, CSV, and database APIs together. ## Notes for specific libraries - **MHCflurry:** use [`VersionedArchiveRegistry`](archives.md) (or `install_archive`) for released `.tar.bz2` trees and ordered historical parts. Its `store_path` callback preserves release selection and existing paths. Resolve member paths from the returned bundle; preserve `DOWNLOAD_INFO.csv` with `extra_files`. Do not enumerate model files as bundle assets or treat the managed store's existence as a completed download. - **hitlist / tsarina:** use [VersionedFileRegistry](file_registry.md) to retain fixed paths, single-Path returns and legacy root manifests without moving old caches. For a deliberate migration to bundles, the existing `{filename, urls, default_version}` mapping is accepted with `verified=False`, as is the `cache_dir` root callable. This is mapping compatibility, not a drop-in filesystem or return-value migration: `download` returns asset paths, `local_path` requires an installed bundle, and status rows contain a `BundleInspection`. A downstream adapter can preserve its public return types and errors. Keep old paths readable during migration; install into a new managed root instead of overwriting legacy directories. - **mhcseqs:** express each records/manifest pair or multi-file source bundle as one version's assets. Keep schema checks and biological validation in mhcseqs. Generated outputs stay outside the source store. The library no longer needs to compose lock, backup, rename and rollback helpers. - **Vaxrank / Isovar / Varcode / Topiary:** share a root, dataset name and data revision to share a complete installation. Existing digest-addressed individual assets can remain under `/objects/sha256/`; these APIs do not relocate them or change prediction-cache identities. - **pyensembl:** keeps its fixed paths and derived SQLite/FASTA indexes while using `fetch_file`; `record_provenance=True` and `inspect_file` show where files came from. Bundles are optional for references with trustworthy multi-file metadata. Existing calls to the private `_download_and_decompress_if_necessary` keep their legacy literal-URL inference when no transform flags are passed (explicit flags use the public parsed-URL rules), with the current permissions and retry behavior. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7095149 datacache-1.27.1/docs/materialization.md0000644000076500000000000002404415264220442017670 0ustar00iskanderwheel# Derived-artifact materialization Use `materialize` when an application must download raw inputs, build one or more derived files, validate them, and publish them together. For example, PyEnsembl can keep its FASTA parsing and biological validation in a builder while DataCache owns reference-DNA download, retry, receipts and publication. Existing `fetch_and_transform` keeps its compatible single-file behavior. ```python from datacache import materialize, inspect_materialization sources = { "dna.fa.gz": { "url": compressed_dna_url, "sha256": compressed_dna_sha256, "size": compressed_dna_size, }, } outputs = {"dna.fa": {}, "index.json": {}} transform = {"version": "dna-parser-1", "options": {"assembly": "GRCh38"}} def build(source_paths, output_paths): # These are private staged paths, not the caller's source or public output. # Stream gzip into output_paths["dna.fa"], validate sequence/contig contents, # and create output_paths["index.json"]. Scientific policy stays here. parse_and_validate_dna(source_paths["dna.fa.gz"], output_paths) paths = materialize( artifact_store, sources, transform=transform, outputs=outputs, builder=build, download_options={"resume": True, "timeout": 60}, ) dna_path = paths["dna.fa"] index_path = paths["index.json"] state = inspect_materialization( artifact_store, sources, transform=transform, outputs=outputs, ) ``` The [offline reference-DNA example](https://github.com/openvax/datacache/blob/master/examples/reference_dna_materialization.py) is runnable with `python -m examples.reference_dna_materialization` and uses tiny gzip data without any biological-library dependency. ## Definitions and builder contract Choose an exact managed store path, independently of the Python package version. Sources are a nonempty mapping from relative names to either `url` or `path`, plus optional `sha256` and `size`. Local paths and `file://` URLs are copied into private working storage; their original bytes and permissions are never modified, and caller-owned inputs are never deleted. Links, hard-linked files, FIFOs and other nonregular local inputs are rejected without blocking. Source expectations describe the **raw input bytes**, including compression. Outputs are a nonempty mapping from relative names to optional `sha256` and `size`, describing **builder-produced bytes**. Empty `{}` expectations are supported; observation and a local consistency receipt do not authenticate these bytes. An expected size of zero supports explicitly empty inputs/outputs. Names cannot escape the store, collide by case or as files/directories, or occupy reserved metadata names. Sources also cannot collide with their automatic provenance-sidecar paths. `transform` contains a nonempty opaque `version` string and optional JSON `options`. Bump this identity when parsing, validation, index format or other scientific decisions change. DataCache does not infer code changes or decide biological compatibility. Do not put secrets in options: they are recorded. `builder(source_paths, output_paths)` receives name-to-string mappings. Output paths are initially absent, with required parent directories already present. The builder must create exactly those outputs, close all handles, and leave inputs unchanged before returning. Only regular, single-link files and the declared output parent directories are accepted. Return values are ignored; `materialize` returns name-to-absolute-path mappings in the newly published bundle. The callback is trusted application code, not a sandbox. Builder progress, streamed decompression and biological validation are caller-owned. ## Identity, receipts and trust Each bundle's manifest records the complete source/transform/output dependency definition and observed sizes/SHA-256 hashes. Source receipts also record redacted origins, acquisition time, acquisition-time trusted-hash verification, and validated transport metadata when available. Resumable HTTP acquisitions record the accepted strong ETag or Last-Modified validator. These validators establish transport consistency, not scientific correctness or authenticity. Ordinary transfers without validated transport metadata do not invent it. Full source references are represented by SHA-256 identity fingerprints. Display origins omit URL credentials, query text and fragments, but retain the path; avoid secret-bearing paths. Changing the full URL/path (including signed query text), integrity expectations, transform identity or output inventory makes the current bundle `invalid` for that request. Refresh is explicit: `materialize(..., force=True)` builds a replacement. This API does not claim different mirror URLs are the same dependency; caller-declared source identity policy is separate. No remote `latest` lookup occurs. `inspect_materialization` reports `available`, `missing`, `invalid` or `inaccessible`, with the current bundle's path, output inspections, source receipts, transform identity and an error cause. Supply all three of `sources`, `transform`, and `outputs` to check the requested identity, or omit all three for receipt-only consistency checks. Inspection never accesses the original inputs, performs network requests, writes, locks or repairs. Full inspection and cache-hit checks hash outputs by default. `verified=True` means all outputs matched caller-supplied trusted hashes **now**; source receipt verification is explicitly historical. Without trusted output hashes, even fully consistent results have `verified=False`. `verify_files=False` checks ownership, dependency identity, exact inventory, types, readability and sizes, but cannot detect same-size corruption and always reports unverified outputs. This fast flag applies only to reuse. New outputs are always hashed before they are published. ## Interruption, publication and retention Writers serialize per artifact; unrelated artifacts have independent locks. Completed inputs persist in owner-only source-keyed directories even without HTTP resume. `download_options={"resume": True}` also preserves partial raw HTTP(S) downloads, using existing hash/size or size/strong-ETag resume rules. Remote resumable inputs require `size`; local inputs are simply copied. The supported transport options are timeout, chunk size, progress callback, progress display, retry count/backoff/delay, resume and `max_bytes`, which limits each remote download. Download, copy and hash progress is enabled by default; set `show_progress=False` in download options for noninteractive use. Valid cache hits remain quiet. Download, gzip, validation and builder failures leave the current bundle untouched. Failed builder outputs are discarded; completed inputs survive for retry without another download. A killed builder can leave private partial outputs, which the next transaction with the same definition discards. Changing only the transform can reuse retained matching inputs. `force=True` always rebuilds outputs, and it doesn't hash the outputs it replaces. It reuses remote inputs that still match, copies a local source again while the original exists (so an in-place correction is picked up; once the original is gone, a retained copy still serves), and repairs corrupted private inputs or input receipts. It never adopts unsafe paths or foreign directories. Materialization stores use the same layout as [bundles](bundles.md#how-bundles-are-stored). All declared outputs and `.datacache-manifest.json` are staged and checked as one tree, which is then renamed into `bundles/` under its UTC install time; the newest bundle is the current one. Readers see the complete old or complete new tree, and returned paths keep working across later refreshes. Do not add indexes or edit files in a published bundle; declare them as outputs or place independently mutable files outside the store. A failure to remove the private staging directory is logged; it never replaces a builder's error or fails a published build. After a successful publication, owned inputs for that source definition are removed by default. `retain_sources=True` keeps them privately for future builds. Cache hits never perform cleanup: a crash between publication and cleanup may leave inputs for an explicit later refresh. Inputs for other source definitions/users and old bundles are never removed implicitly; use `prune_bundles` to delete old bundles (see [removing old bundles](bundles.md#removing-old-bundles)). Abandoned input/staging variants require deliberate operator cleanup; general inventory/pruning is separate work. Retention is installation policy, not output identity, so toggling it on a cache hit does not mutate disk. The store marker, `.datacache-store.json`, records that this is a materialization store, so DataCache never adopts a populated legacy directory or another kind of store, even with force. An empty precreated directory becomes the store in place, keeping its owner, group and permissions. Installation works on local filesystems with atomic sibling renames, on Linux, macOS and Windows; `resume=True` needs POSIX. Read-only existing hits require no lock. This is atomic visibility, not a guarantee against filesystem/power-loss faults. ## Disk use Let `C` be total completed raw inputs, `D` the new bundle of outputs, and `O` all old bundles. During a build, storage is approximately `O + C + D`, plus scratch files created by the builder. Publication is a rename and does not duplicate `D`. After successful default cleanup, retained storage is `O + D`; opt-in source retention uses `O + C + D`. The existing resumable transport copies a completed private partial into file publication staging before removing the partial. That phase can need an extra compressed-input-sized copy: peak is approximately `O + max(C + largest_input, C + D + builder_scratch)`, excluding any previously abandoned input/staging variants. For one 1 GB compressed / 3 GB decompressed human-DNA input, a first build needs roughly 4 GB plus builder scratch, then retains roughly 3 GB (or 4 GB with source retention). A refresh retaining an old 3 GB output needs roughly 7 GB peak and retains 6 GB afterward. Caller-owned local originals occupy additional storage outside this managed store. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7096138 datacache-1.27.1/docs/progress.md0000644000076500000000000000774315264220442016347 0ustar00iskanderwheel# Progress and logging See the [API reference](api.md) for signatures, defaults, return values, and errors. Set `show_progress=True` on `fetch_file`, `Cache.fetch`, or the CSV and database helpers to display progress bars. Displays are quiet by default and close on success, exceptions, and handled cancellation. The exception is `materialize`, which shows download, copy and hash progress unless you pass `show_progress=False` in its download options. DataCache uses [tqdm.auto](https://tqdm.github.io/docs/shortcuts/), which selects a terminal or notebook display. Terminal bars use stderr; stdout remains available for application output. tqdm is a required dependency, installed by `python -m pip install datacache`; the old `datacache[progress]` extra remains a supported compatibility alias. Downloads display bytes and a total when the server provides a usable size. Compressed HTTP content encoding can make the wire size differ from the decoded data; in that case the total is unknown. gzip decompression also uses an unknown total rather than guessing from its compressed size. ZIP entries use their declared uncompressed size. SHA-256 verification displays bytes read; database insertion displays rows completed in batches of up to 1,000. Each retry starts a fresh download bar. A finished transfer is followed by any decompression and hash-verification work before publication. HTML parsing and arbitrary user transformations do not expose incremental progress. Ordinary cache reuse shows no bar; an explicit `validate_file(..., show_progress=True)` can show a checksum pass over an already cached file. ## Application callbacks The existing `progress_callback(completed, total)` API remains supported and can be used with or without built-in progress bars. `completed` is cumulative for the current download attempt; `total` may be `None`. Retries restart from zero, so counts may decrease. Callbacks cover transfer bytes only, not transformation or database work. Empty downloads do not invoke the callback. ```python from datacache import fetch_file def report(completed, total): # Replace with your UI's progress update. print(f"Downloaded {completed} bytes; expected total: {total}") path = fetch_file( "https://example.org/data.tsv", destination="references/data.tsv", progress_callback=report, chunk_size=1024 * 1024, ) ``` Callback exceptions abort the operation, clean up partial downloads, and are not retried. Keep callbacks inexpensive. Treat `fetch_file` returning successfully as the publication signal, rather than relying on the byte count. ## CSV options CSV helpers separate pandas options from download options. Parser options such as `sep`, `dtype`, and `usecols` stay as keyword arguments. Put `timeout`, `cache_root`, `expected_sha256`, `expected_size`, retry settings, and callbacks inside `download_options`: ```python from datacache import fetch_csv_dataframe frame = fetch_csv_dataframe( "https://example.org/records.tsv.gz", filename="records.tsv", sep="\t", dtype={"sample_id": "string"}, show_progress=True, download_options={"cache_root": "references", "timeout": 30, "max_retries": 2}, ) ``` `fetch_csv_db` accepts the same options, and `show_progress=True` also covers database insertion. If `download_options` contains `show_progress`, it overrides the outer setting for the download only. Keep filename and subdirectory selection in the helper's own arguments; CSV helpers decompress archives before parsing. The options dictionary is copied and never mutated. ## Logging DataCache uses Python logging without adding handlers or changing global configuration. Applications can enable informational messages: ```python import logging logging.basicConfig(level=logging.INFO) logging.getLogger("datacache").setLevel(logging.INFO) ``` Retry warnings include attempt counts, failure category, and delay. HTTP errors retain their original Requests exception and response. See the [retry reference](downloads.md#transient-http-failures) for precise behavior. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6518776 datacache-1.27.1/docs/reference/0000755000076500000000000000000015264225206016107 5ustar00iskanderwheel././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7097664 datacache-1.27.1/docs/reference/choosing.md0000644000076500000000000001040715264220442020241 0ustar00iskanderwheel# Choose an interface and understand guarantees ## Choose the right API The [complete API reference](../api.md) lists every public signature, default, return value, exception, and example. | Task | API | Result | | --- | --- | --- | | Download or reuse one file | `fetch_file(...)`, `Cache.fetch(...)` | Local path string | | Refresh a cached file periodically, keeping it when the source is unreachable | `fetch_file(..., expire_after=..., return_stale_on_error=True, validator=...)` | Local path string | | Read a small resource into memory | `fetch_bytes(...)` | `bytes` | | Install or reuse a versioned dataset | `VersionedDatasetRegistry`, `install_bundle(...)` | Mapping of asset names to paths in the current bundle | | Build derived files from versioned dependencies | `materialize(...)` | Mapping of output names to paths in the current bundle | | Install versioned archive trees | `VersionedArchiveRegistry`, `install_archive(...)` | `Path` of the extracted tree, the current bundle | | List or delete old bundles in any store | `list_bundles(...)`, `prune_bundles(...)` | Bundle paths, oldest first | | Reuse an established fixed-path versioned file cache | `VersionedFileRegistry` | One Path and a legacy-compatible root receipt | | Inspect a dataset's current bundle | `inspect_bundle(...)` | `BundleInspection` | | Inspect an installed archive tree | `inspect_archive(...)` | `ArchiveInspection` | | Discard retained partial download bytes | `discard_partial(destination)` | Installed file unchanged | | Compute a path without filesystem access | `expected_path(...)`, `Cache.local_path(...)` | Path string | | Choose where a package keeps its data, shared or not | `get_cache_root(name, *envkeys, override=..., legacy=...)` | Path string | | Check presence | `file_exists(...)`, `Cache.exists(...)` | Boolean; does not establish integrity | | Validate bytes, raising on failure | `validate_file(...)` | Path string | | Inspect without repair or network access | `inspect_file(...)`, `Cache.inspect(...)` | `FileInspection` | | Inspect a set of required files | `inspect_files(root, files)` | `CacheInspection` | | Explicitly share an existing private file | `make_file_readable(...)`, `Cache.make_readable(...)` | Path string; POSIX only | | Download and parse CSV/TSV | `fetch_csv_dataframe(...)` | pandas DataFrame | | Cache a custom file transformation | `fetch_and_transform(...)` | Transformer/loader result | | Create a SQLite cache | `db_from_dataframe(...)`, `db_from_dataframes(...)` | Open SQLite connection | | Download CSV into SQLite | `fetch_csv_db(...)` | Open SQLite connection | | Reopen a database with matching metadata | `connect_if_correct_version(...)` | Connection or `None` | The library does not export `fetch_fasta_dict` or `fetch_fasta_db`. Download FASTA files with `fetch_file`, then parse them in the consuming library. ## Guarantees and limits Downloads and complete archive trees are staged privately and published atomically after validation. New files respect the process umask; replacements preserve existing access permissions. This includes pyensembl's private download helpers. Custom single-file transformations publish only successful output. Existing SQLite caches rebuild in a transaction: failure rolls back both schema and rows. New databases are built privately before publication. Cached data is reused by path or database version. DataCache doesn't ask servers whether files changed; `expire_after` refreshes files after a set time and also replaces cached files that no longer validate. Otherwise invalid caches are reported, not repaired. Upgrades keep existing cache names and database metadata compatible. Valid cache hits do not rewrite files, change permissions, or apply new schema constraints. See [upgrading existing caches](../data.md#upgrading-existing-caches) and [sharing old private files](../shared-caches.md#files-already-downloaded-as-0600). File publication requires local filesystem support for atomic replacement; new SQLite database publication also requires hard links. SQLite locking and transactions govern database rebuilds. Use `install_bundle` for atomic multi-file publication. Bundle, archive and materialization installs work on local filesystems on Linux, macOS and Windows; resumable transfers need POSIX. Distributed coordination is outside their scope. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7098742 datacache-1.27.1/docs/shared-caches.md0000644000076500000000000001342715264220442017171 0ustar00iskanderwheel# Shared caches See the [API reference](api.md) for signatures, defaults, return values, and errors. ## Choosing where data lives Most packages let users move their data with an option or an environment variable, keep using a location from an older release, and otherwise use the platform's cache directory. `get_cache_root` does all of that in one call: ```python from datacache import fetch_file, get_cache_root root = get_cache_root( "hitlist", # Platform cache directory name, the last resort. "HITLIST_DATA_DIR", # Environment variables, checked in order. override=data_dir_option, # An explicit choice wins, when not None. legacy=["~/.hitlist"], # Older locations, used only if they hold data. ) path = fetch_file(url, cache_root=root) ``` The root is the first of these that applies: 1. The override. 2. The first environment variable that is set. 3. The platform cache directory, if it already holds data. 4. The first legacy location holding data. 5. The platform cache directory. A location holds data when it contains a file anywhere inside other than the `.DS_Store`-style files operating systems leave behind. Empty leftover folders therefore never pin a user to an old location, and a user who has already downloaded into the platform directory stays there. Legacy paths must be absolute. The legacy check reads the disk, so resolve the root once, for example at startup, and pass it along. Environment values are used exactly as given, never with a package name added, so several packages reading one variable agree on one directory. Nothing is created until something is downloaded. Pass the root to `fetch_file`, `Cache` or a registry as `cache_root`; `fetch_file`'s `subdir` alone reads no environment variables. ## Sharing one root Select a common root explicitly: ```python from datacache import Cache cache = Cache("references", cache_root="/srv/references/v1") status = cache.inspect(filename="annotations.gtf") print(status.status) if status.error is not None: print(status.error) ``` Inspection never downloads, creates directories, or repairs files. The root is the actual directory containing files, not a parent to which `subdir` is added. For pyensembl, configure its own `PYENSEMBL_CACHE_DIR`; that environment setting is interpreted by pyensembl, not globally by DataCache. ## New downloads Files remain private while downloading and validating. New final files use normal creation permissions measured in the destination directory, without temporarily changing the process-wide umask: | Umask | Typical new file mode | Access | | --- | --- | --- | | `022` | `0644` | Owner writes; others read | | `002` | `0664` | Owner/group write; others read | | `007` | `0660` | Owner/group read and write | | `077` | `0600` | Owner only | These modes assume no default ACL modifies creation. Choose the umask in the shell or application launcher, before starting worker threads. Shared group directories can use setgid so new files inherit the directory's group; have the cache administrator configure the intended group and directory access. Replacing a downloaded file preserves its existing read/write/execute bits. It creates a new inode, so existing ownership, ACLs, and extended attributes are not copied. Existing SQLite databases rebuild transactionally in place and retain their inode and permissions. ## Files already downloaded as `0600` The fix for [#68](https://github.com/openvax/datacache/issues/68) affects new files. Reusing or force-refreshing an existing file does not broaden its permissions. Its owner or an administrator must explicitly grant access to the files intended for sharing. For example, to grant the owning group read access to two selected files: ```python from datacache import Cache, make_file_readable make_file_readable("/srv/references/v1/annotations.gtf") # Add group read. cache = Cache(cache_root="/srv/references/v1") cache.make_readable(filename="transcripts.fa") # Add others=True only if everyone should be able to read this file. ``` These POSIX helpers add only the requested read bits, preserve contents and the inode, and reject symlinks, directories, and other non-regular files. They do not recurse, change ownership, grant write access, or run implicitly during normal cache use. Calling them again is harmless. Permission errors propagate if the caller cannot open or chmod the file. The equivalent shell command is: ```sh chmod g+r /srv/references/v1/annotations.gtf /srv/references/v1/transcripts.fa ``` Group membership and directory traversal permissions must also permit access. Use `ls -ld` on the cache and parent directories, and `ls -l` on the affected files. A readable file inside an inaccessible parent is still inaccessible. ## Troubleshooting | Symptom | What to check | | --- | --- | | `inaccessible` / `PermissionError` | File mode, owning group, ACLs, and traversal permissions on every parent | | File is present but `corrupt` | Expected hash/size, file type, and whether a previous tool wrote incomplete data | | Cache root is `available` but not `verified` | Every required file needs a trusted SHA-256 expectation to mark the inventory verified | | Read-only cache can be read but cannot refresh | Refresh requires write access to the destination directory; database rebuilds also need SQLite journal access | | Old private files stay private after refresh | Existing file modes are preserved; adjust the selected files explicitly | | Previously generated SQLite numbers are wrong | Rebuild from source with a new version or `overwrite=True` | An intact downloaded file can be reused from a read-only installation. For SQLite, use `connect_if_correct_version(path, version, read_only=True)`. Keep published releases in distinct versioned directories when readers must continue using a stable release while another one is being prepared. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090568.5105245 datacache-1.27.1/docs/zip-members.md0000644000076500000000000000505615264220711016727 0ustar00iskanderwheel# Fetch an individual ZIP member Use `install_zip_member` when a large remote ZIP contains a small subset of files you need. It requests only ZIP metadata and the named member's compressed bytes. The server must support exact byte ranges and supply a strong ETag or Last-Modified. A server that ignores ranges is rejected before its body is read. ```python import json from pathlib import Path from datacache import install_zip_member paths = install_zip_member( "/shared/research/search-2026/peptides", "https://example.org/results.zip", "txt/peptides.txt", max_compressed_bytes=512 * 2**20, max_bytes=4 * 2**30, ) receipt = json.loads(Path(paths["receipt.json"]).read_text()) print(receipt["payload"], receipt["compressed"]) ``` Replace the example URL and member with your source. The result contains: | Key | Content | | --- | --- | | `payload` | Complete expanded member bytes | | `compressed` | Exact compressed bytes, without ZIP headers | | `receipt.json` | Source/member identity, HTTP validator, archive size, offset, compression method, CRC32, both SHA-256 hashes and sizes | The first acquisition records observed hashes. Reproduce it in a new destination with `expected_sha256`, `expected_size` and `compressed_sha256` from an independently trusted receipt. Those expectations describe the expanded member and exact compressed stream, respectively. CRC32 detects accidental corruption; it is not a cryptographic provenance guarantee. Last-Modified can miss changes within its timestamp resolution, so pin content hashes when reproducibility matters. Only stored or deflated regular, unencrypted members are supported. ZIP64 is supported; member names must be safe relative POSIX paths and unique. Metadata reads default to an 8 MiB total limit. Compressed input and expanded output have separate limits, checked against the directory and again while streaming. Memory usage during extraction is bounded; both byte representations remain on disk, so allow space for their sum plus any retained old generations. Every acquisition publishes all three files together. Interrupted or rejected attempts leave the previous generation usable. Another call with identical arguments verifies retained bytes and returns offline without checking upstream freshness. Changed expectations or damaged files require explicit `force=True`. A failed attempt restarts the selected member on retry; it never downloads other members or falls back to downloading the complete ZIP. Use `list_bundles` to inspect generations and explicit `prune_bundles` when old paths are no longer used. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6525354 datacache-1.27.1/examples/0000755000076500000000000000000015264225206015037 5ustar00iskanderwheel././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7101398 datacache-1.27.1/examples/basic_usage.py0000644000076500000000000000410015264220442017646 0ustar00iskanderwheel"""Run with `python -m examples.basic_usage`; no network or persistent writes.""" from contextlib import closing import gzip import hashlib from pathlib import Path from tempfile import TemporaryDirectory from datacache import Cache, fetch_and_transform, fetch_csv_db def main(): with TemporaryDirectory() as directory: root = Path(directory) data = b"id,label\n1,case\n2,control\n" source = root / "records.csv.gz" source.write_bytes(gzip.compress(data)) cache = Cache(cache_root=root / "cache") digest = hashlib.sha256(data).hexdigest() path = cache.fetch(source.as_uri(), filename="records.csv", expected_sha256=digest) assert Path(path).read_bytes() == data assert cache.inspect(filename="records.csv", expected_sha256=digest).verified archive = source.read_bytes() raw_path = cache.fetch(source.as_uri(), filename="archive.bin", raw=True, expected_sha256=hashlib.sha256(archive).hexdigest(), expected_size=len(archive)) assert Path(raw_path).read_bytes() == archive with closing(fetch_csv_db( "records", source.as_uri(), csv_filename="records.csv", download_options={"cache_root": root / "cache", "expected_sha256": digest})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(1, "case"), (2, "control")] def transform(source_path, output_path): text = Path(source_path).read_text().upper() Path(output_path).write_text(text) return text transformed = fetch_and_transform( "uppercase.csv", transform, lambda p: Path(p).read_text(), "records.csv", source.as_uri(), cache_root=root / "cache") assert transformed == data.decode().upper() source.unlink() assert cache.fetch(source.as_uri(), filename="records.csv") == path print("Verified download, SQLite cache, transformation, and offline reuse succeeded.") if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7102213 datacache-1.27.1/examples/reference_dna_materialization.py0000644000076500000000000000513115264220442023442 0ustar00iskanderwheel"""Tiny offline reference-DNA transaction; scientific work stays in the builder.""" import gzip from hashlib import sha256 import json from pathlib import Path import shutil from tempfile import TemporaryDirectory from datacache import inspect_materialization, materialize def build_reference(source_paths, output_paths): with gzip.open(source_paths['dna.fa.gz'], 'rb') as source, open(output_paths['dna.fa'], 'wb') as output: shutil.copyfileobj(source, output, length=2 ** 20) # A toy caller-side biological check, not a DataCache FASTA parser. with open(output_paths['dna.fa'], 'rb') as fasta: if fasta.readline() != b'>chr1\n': raise ValueError('unexpected contig header') length = sum(len(line.strip()) for line in fasta) Path(output_paths['index.json']).write_text(json.dumps({'length': length})) def main(): with TemporaryDirectory() as temporary: root = Path(temporary) dna = b'>chr1\nACGTACGT\n' compressed = gzip.compress(dna, mtime=0) original = root / 'upstream.fa.gz' original.write_bytes(compressed) sources = {'dna.fa.gz': {'path': original, 'sha256': sha256(compressed).hexdigest(), 'size': len(compressed)}} outputs = {'dna.fa': {'sha256': sha256(dna).hexdigest(), 'size': len(dna)}, 'index.json': {}} transform = {'version': 'toy-fasta-1', 'options': {'contig': 'chr1'}} store = root / 'reference' / 'release-110' paths = materialize(store, sources, transform=transform, outputs=outputs, builder=build_reference, download_options={'show_progress': False}) assert Path(paths['dna.fa']).read_bytes() == dna assert json.loads(Path(paths['index.json']).read_text()) == {'length': 8} assert not list(store.glob('.inputs-*')) original.unlink() # Neither the upstream source nor a builder is needed on an offline hit. def forbidden(*args): raise AssertionError('offline reuse attempted to rebuild') assert materialize(store, sources, transform=transform, outputs=outputs, builder=forbidden, verify_files=False) == paths state = inspect_materialization(store, sources, transform=transform, outputs=outputs) assert state.status == 'available' assert not state.verified # Index has observed metadata, not a trusted hash. assert state.sources['dna.fa.gz']['verified'] print('Reference DNA: atomic pair, compressed provenance, input cleanup and offline reuse pass') if __name__ == '__main__': main() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7103083 datacache-1.27.1/examples/versioned_datasets.py0000644000076500000000000000522515264220442021300 0ustar00iskanderwheel"""Run with python -m examples.versioned_datasets; no network required.""" from hashlib import sha256 from pathlib import Path from tempfile import TemporaryDirectory from datacache import VersionedDatasetRegistry, db_from_dataframe, inspect_bundle def main(): import pandas as pd with TemporaryDirectory() as temporary: root = Path(temporary) upstream = root / 'upstream' upstream.mkdir() assets = {} for name, content in [('records.csv', b'id,sequence\n1,ACGT\n'), ('manifest.json', b'{"release":"2026-09"}\n'), ('source.txt', b'example source\n')]: source = upstream / name source.write_bytes(content) assets[name] = dict(url=source.as_uri(), size=len(content), sha256=sha256(content).hexdigest()) datasets = { name: dict(default_version='2026-09', versions={'2026-09': selected}) for name, selected in [ ('single', {'records.csv': assets['records.csv']}), ('pair', {k: assets[k] for k in ('records.csv', 'manifest.json')}), ('multi', assets), ] } registry = VersionedDatasetRegistry(datasets, cache_root=root / 'shared') assert registry.inspect('single').status == 'missing' assert not (root / 'shared').exists() for name in datasets: paths = registry.download(name) assert set(paths) == set(datasets[name]['versions']['2026-09']) assert registry.inspect(name).verified generated = root / 'generated' generated.mkdir() table = pd.read_csv(registry.local_path('pair', asset='records.csv')) connection = db_from_dataframe(str(generated / 'sequences.db'), 'sequences', table, primary_key='id') try: assert connection.execute('SELECT sequence FROM sequences WHERE id=1').fetchone() == ('ACGT',) finally: connection.close() registry.download('pair', force=True) assert (generated / 'sequences.db').is_file() for source in upstream.iterdir(): source.unlink() consumer = VersionedDatasetRegistry(datasets, cache_root=root / 'shared') for name in datasets: consumer.download(name) # Verified reuse with no upstream available. assert consumer.inspect(name).verified assert inspect_bundle(consumer.store_path(name)).status == 'available' print('Three dataset shapes installed, indexed, shared, and inspected offline.') if __name__ == '__main__': main() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7111707 datacache-1.27.1/pyproject.toml0000644000076500000000000000256615264220442016143 0ustar00iskanderwheel[build-system] requires = ["setuptools>=64"] build-backend = "setuptools.build_meta" [project] name = "datacache" requires-python = ">=3.9" authors = [{name="Alex Rubinsteyn", email="alex.rubinsteyn@unc.edu"}] description = "Helpers for transparently downloading datasets" classifiers = [ "Development Status :: 4 - Beta", "Environment :: Console", "Operating System :: OS Independent", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python", "Topic :: Scientific/Engineering :: Bio-Informatics", ] readme = "README.md" dynamic = ["version", "dependencies"] [project.optional-dependencies] # Compatibility alias: tqdm is now installed with the base package. progress = [] html = ["lxml>=4.9"] test = ["pytest>=7", "pytest-cov", "ruff", "lxml>=4.9"] [tool.setuptools.dynamic] version = {attr = "datacache.version.__version__"} dependencies = {file = ["requirements.txt"]} [tool.setuptools] packages = ["datacache"] [project.urls] "Homepage" = "https://github.com/openvax/datacache" "Bug Tracker" = "https://github.com/openvax/datacache/issues" "Documentation" = "https://github.com/openvax/datacache#readme" [tool.ruff.lint] select = ["E", "F"] ignore = ["F821", "E501", "F841", "E731", "E741", "E722", "E721"] [tool.ruff.lint.per-file-ignores] "__init__.py" = ["F401"] "test*/*" = ["F401"] ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7113235 datacache-1.27.1/requirements.txt0000644000076500000000000000011315264220442016475 0ustar00iskanderwheelpandas>=0.15.2 platformdirs>=3.0 requests>=2.5.1 filelock>=3.13 tqdm>=4.64 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6652148 datacache-1.27.1/setup.cfg0000644000076500000000000000004615264225206015042 0ustar00iskanderwheel[egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092805.6637638 datacache-1.27.1/tests/0000755000076500000000000000000015264225206014363 5ustar00iskanderwheel././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792092805.664372 datacache-1.27.1/tests/fixtures/0000755000076500000000000000000015264225206016234 5ustar00iskanderwheel././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792090401.712354 datacache-1.27.1/tests/fixtures/README.md0000644000076500000000000000175415264220442017517 0ustar00iskanderwheel# Legacy cache fixtures These fixtures were generated using the unmodified DataCache 1.9.1 source at commit `ee20b5a`, imported from a separate extraction of `git archive`. - `legacy_v191.sql` is SQLite's `iterdump()` of a database built with `db_from_dataframe`, table `records`, version `7`, and the DataFrame `{"sample id": [1, 2], "label": ["case", "control"]}`. The original `primary_key="sample id"` was ignored after column normalization; retaining that exact old schema tests reuse without applying new constraints. - `legacy_v191_names.json` records `build_local_filename` for raw, gzip, ZIP, query-string, and fragment URLs, plus the database filename used by `fetch_csv_db` for `records.csv.gz` with integer `id` and text `label` columns and two rows. The tests create databases from the SQL dump to avoid committing opaque binary files. Do not regenerate these fixtures with the new implementation: they represent existing user caches, not the expected output of new builds. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7124336 datacache-1.27.1/tests/fixtures/legacy_v191.sql0000644000076500000000000000053315264220442020777 0ustar00iskanderwheel-- Generated by datacache 1.9.1 at ee20b5a, before these changes. BEGIN TRANSACTION; CREATE TABLE "_datacache_metadata" ("version" INT); INSERT INTO "_datacache_metadata" VALUES(7); CREATE TABLE "records" ("sample_id" INT NOT NULL, "label" TEXT NOT NULL); INSERT INTO "records" VALUES(1,'case'); INSERT INTO "records" VALUES(2,'control'); COMMIT; ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7125223 datacache-1.27.1/tests/fixtures/legacy_v191_names.json0000644000076500000000000000302215264220442022330 0ustar00iskanderwheel{ "filenames": [ { "url": "https://example.org/records.csv.gz", "decompress": false, "filename": "9a93c4e951e0435f9f8d2c8581dd3781.https___example.org_records.csv.gz" }, { "url": "https://example.org/records.csv.gz", "decompress": true, "filename": "9a93c4e951e0435f9f8d2c8581dd3781.https___example.org_records.csv" }, { "url": "https://example.org/records.csv.gz?token=release", "decompress": false, "filename": "c78256efb732d73b536dcb23168c0276.https___example.org_records.csv.gz_token_release" }, { "url": "https://example.org/records.csv.gz?token=release", "decompress": true, "filename": "c78256efb732d73b536dcb23168c0276.https___example.org_records.csv" }, { "url": "https://example.org/records.zip#release", "decompress": false, "filename": "05c7653f9eb755fc99fd432248bd948d.https___example.org_records.zip#release" }, { "url": "https://example.org/records.zip#release", "decompress": true, "filename": "05c7653f9eb755fc99fd432248bd948d.https___example.org_records" }, { "url": "https://example.org/records.csv", "decompress": false, "filename": "87fe5f7f54237b64f37559e773d70008.https___example.org_records.csv" }, { "url": "https://example.org/records.csv", "decompress": true, "filename": "87fe5f7f54237b64f37559e773d70008.https___example.org_records.csv" } ], "csv_database_filename": "records.csv_nrows2.id_INT.label_TEXT.db" } ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7126899 datacache-1.27.1/tests/test_archives.py0000644000076500000000000010263715264220442017606 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Transactional whole-archive installation for directory-tree consumers.""" from concurrent.futures import ThreadPoolExecutor from hashlib import sha256 import gzip import io import json import os from pathlib import Path import subprocess import sys import tarfile import threading import pytest from datacache import ( FileValidationError, VersionedArchiveRegistry, inspect_archive, install_archive, ) from datacache import archives, bundle_store, download def make_tar(path, members, *, mode="w:bz2"): with tarfile.open(path, mode) as archive: for name, value in members: if isinstance(value, tarfile.TarInfo): archive.addfile(value) continue info = tarfile.TarInfo(name) if value is None: info.type = tarfile.DIRTYPE archive.addfile(info) else: info.size = len(value) archive.addfile(info, io.BytesIO(value)) return path @pytest.fixture def archive_file(tmp_path): path = make_tar(tmp_path / "models.tar.bz2", [ ("models", None), ("models/a/model.json", b'{"model": "a"}\n'), ("models/b/model.json", b'{"model": "b"}\n'), ("README.txt", b"released weights\n"), ]) data = path.read_bytes() return path, data, sha256(data).hexdigest() def test_install_inspect_and_read_only_reuse(tmp_path, archive_file, monkeypatch): source, data, digest = archive_file destination = tmp_path / "download" receipt = "url\nhttps://example.test/models.tar.bz2\n" sources = [{"url": "https://example.test/models.tar.bz2", "path": source}] bundle = install_archive( destination, sources, expected_sha256=digest, expected_size=len(data), extra_files={"DOWNLOAD_INFO.csv": receipt}) assert (bundle / "models/a/model.json").read_text() == '{"model": "a"}\n' assert (bundle / "DOWNLOAD_INFO.csv").read_text() == receipt inspected = inspect_archive( destination, sources, expected_sha256=digest, expected_size=len(data), extra_files={"DOWNLOAD_INFO.csv": receipt}) assert inspected.status == "available" and inspected.verified assert Path(inspected.bundle) == bundle assert inspected.source_urls == ("https://example.test/models.tar.bz2",) assert inspected.archive_size == len(data) assert inspected.recorded_sha256 == digest assert inspected.fetched_at.endswith("+00:00") assert inspect_archive(destination).status == "available" assert not inspect_archive(destination).verified # Files are verified only by hashes the caller supplied, never the manifest's own. assert all(item.verified for item in inspected.files.values()) assert not any(item.verified for item in inspect_archive(destination).files.values()) fast = inspect_archive( destination, sources, expected_sha256=digest, expected_size=len(data), extra_files={"DOWNLOAD_INFO.csv": receipt}, verify_files=False) assert fast.status == "available" and not fast.verified assert fast.files == {} and Path(fast.bundle) == bundle def forbidden(*args, **kwargs): raise AssertionError("fast inspection read the extracted tree") # The fast check reads the manifest alone, however large the tree. monkeypatch.setattr(archives, "check_tree", forbidden) assert inspect_archive( destination, sources, extra_files={"DOWNLOAD_INFO.csv": receipt}, verify_files=False).status == "available" monkeypatch.undo() refreshed = install_archive( destination, sources, expected_sha256=digest, expected_size=len(data), extra_files={"DOWNLOAD_INFO.csv": receipt}, force=True) assert refreshed != bundle assert (bundle / "models/a/model.json").read_text() == '{"model": "a"}\n' def forbidden(*args, **kwargs): raise AssertionError("a valid cache hit attempted a write, lock, or transfer") monkeypatch.setattr(bundle_store.BundleStore, "lock", forbidden) monkeypatch.setattr(download, "fetch_file", forbidden) monkeypatch.setattr(archives, "write_json", forbidden) monkeypatch.setattr(bundle_store, "write_json", forbidden) assert install_archive( destination, sources, expected_sha256=digest, expected_size=len(data), extra_files={"DOWNLOAD_INFO.csv": receipt}) == refreshed def test_line_endings_and_control_bytes_are_installed_unchanged(tmp_path): # Windows translates line endings, and stops reading at 0x1A, in files # that aren't opened as binary. contents = b"a,b\r\n1,2\r\n\x1a after the control byte\n" source = make_tar(tmp_path / "table.tar.bz2", [("table.csv", contents)]) data = source.read_bytes() destination = tmp_path / "download" bundle = install_archive( destination, source, expected_sha256=sha256(data).hexdigest(), expected_size=len(data), extra_files={"DOWNLOAD_INFO.csv": "url\r\nhttps://example.test/table.tar.bz2\r\n"}) assert (bundle / "table.csv").read_bytes() == contents manifest = json.loads((bundle / archives.MANIFEST).read_text()) assert manifest["files"]["table.csv"] == {"sha256": sha256(contents).hexdigest(), "size": len(contents)} assert inspect_archive(destination).status == "available" def test_install_hashes_the_extracted_tree_once(tmp_path, archive_file, monkeypatch): source, data, digest = archive_file destination = tmp_path / "download" def forbidden(*args, **kwargs): raise AssertionError("hashed the extracted tree a second time") monkeypatch.setattr(archives, "check_tree", forbidden) install_archive(destination, source, expected_sha256=digest, expected_size=len(data)) monkeypatch.undo() assert inspect_archive(destination, source, expected_sha256=digest, expected_size=len(data)).verified def test_standard_dot_prefixed_tar_tree_installs_and_inspects(tmp_path): source = make_tar(tmp_path / "standard.tar.bz2", [ (".", None), ("././", None), ("./models/", None), ("./models/model.json", b"model"), ("././README.txt", b"readme"), ]) data = source.read_bytes() destination = tmp_path / "download" bundle = install_archive( destination, source, expected_sha256=sha256(data).hexdigest(), expected_size=len(data)) assert (bundle / "models/model.json").read_bytes() == b"model" inspected = inspect_archive(destination) assert inspected.status == "available" assert set(inspected.files) == {"models/model.json", "README.txt"} @pytest.mark.parametrize("members", [ [("./file", b"one"), ("file", b"two")], [("./Models/file", b"one"), ("models/other", b"two")], [("./../escape", b"unsafe")], [("./.datacache-archive-manifest.json", b"forged")], ]) def test_dot_prefix_normalization_preserves_path_safety(tmp_path, members): source = make_tar(tmp_path / "unsafe.tar.bz2", members) destination = tmp_path / "download" with pytest.raises(FileValidationError): install_archive(destination, source, verified=False) assert inspect_archive(destination).status == "missing" assert not (tmp_path / "escape").exists() def test_large_tree_manifest_installs_and_is_reused_offline(tmp_path): source = make_tar(tmp_path / "large.tar.bz2", [ ("file-%05d" % index, b"") for index in range(12000) ]) data = source.read_bytes() options = {"expected_sha256": sha256(data).hexdigest(), "expected_size": len(data)} destination = tmp_path / "download" bundle = install_archive(destination, source, **options) assert (bundle / archives.MANIFEST).stat().st_size > 1024 * 1024 inspected = inspect_archive(destination, source, **options) assert inspected.status == "available" and inspected.verified assert len(inspected.files) == 12000 source.unlink() assert install_archive(destination, source, **options) == bundle assert inspect_archive(destination, source, verify_files=False, **options).status == "available" @pytest.mark.skipif(os.name != "posix", reason="POSIX FIFO sources") def test_fifo_source_fails_promptly_and_releases_writer_lock(tmp_path, archive_file): fifo = tmp_path / "source.fifo" os.mkfifo(fifo) destination = tmp_path / "download" # A subprocess timeout makes a blocking open a bounded test failure. script = """ import sys from pathlib import Path from datacache import FileValidationError, install_archive try: install_archive(Path(sys.argv[1]), Path(sys.argv[2]), verified=False) except FileValidationError: pass else: raise AssertionError("FIFO archive source was accepted") """ subprocess.run( [sys.executable, "-c", script, str(destination), str(fifo)], timeout=5, check=True, capture_output=True, text=True) assert inspect_archive(destination).status == "missing" source, data, digest = archive_file assert install_archive( destination, source, expected_sha256=digest, expected_size=len(data)).is_dir() @pytest.mark.parametrize("new_extras", [ {}, {"DOWNLOAD_INFO.csv": "url\noriginal\n"}, {"DOWNLOAD_INFO.csv": "url\nchanged\n", "INFO.txt": "old"}, {"NEW_INFO.txt": "new"}, ]) def test_changed_or_removed_consumer_metadata_requires_refresh( tmp_path, archive_file, new_extras): source, data, digest = archive_file options = {"expected_sha256": digest, "expected_size": len(data)} destination = tmp_path / "download" old_extras = {"DOWNLOAD_INFO.csv": "url\noriginal\n", "INFO.txt": "old"} old = install_archive(destination, source, extra_files=old_extras, **options) manifest = json.loads((old / archives.MANIFEST).read_text()) assert set(manifest["extra_files"]) == set(old_extras) for verify_files in (True, False): assert inspect_archive( destination, source, extra_files=new_extras, verify_files=verify_files, **options).status == "invalid" with pytest.raises(FileValidationError, match="force=True"): install_archive(destination, source, extra_files=new_extras, **options) refreshed = install_archive( destination, source, extra_files=new_extras, force=True, **options) assert refreshed != old assert inspect_archive(destination, source, extra_files=new_extras, **options).verified assert {name for name in old_extras if (refreshed / name).exists()} == set(old_extras) & set(new_extras) assert all((old / name).read_text() == value for name, value in old_extras.items()) def test_ordered_split_archive_and_logical_source_urls(tmp_path, archive_file): _, data, _ = archive_file cuts = (len(data) // 3, 2 * len(data) // 3) pieces = (data[:cuts[0]], data[cuts[0]:cuts[1]], data[cuts[1]:]) sources, urls = [], [] for index, piece in enumerate(pieces): path = tmp_path / ("models.part.%d" % index) path.write_bytes(piece) url = "https://example.test/models.tar.bz2.part.%d?token=secret" % index urls.append(url) sources.append({"url": url, "path": path}) csv_receipt = "url\n" + "".join(url.split("?", 1)[0] + "\n" for url in urls) destination = tmp_path / "split" bundle = install_archive( destination, sources, verified=False, extra_files={"DOWNLOAD_INFO.csv": csv_receipt}) assert (bundle / "models/b/model.json").is_file() assert (bundle / "DOWNLOAD_INFO.csv").read_text() == csv_receipt state = inspect_archive(destination, sources, extra_files={"DOWNLOAD_INFO.csv": csv_receipt}) assert state.status == "available" and not state.verified manifest = json.loads((bundle / archives.MANIFEST).read_text()) assert [item["url"] for item in manifest["sources"]] == [url.split("?", 1)[0] for url in urls] assert [item["fingerprint"] for item in manifest["sources"]] != [ sha256(url.split("?", 1)[0].encode()).hexdigest() for url in urls] assert inspect_archive( destination, list(reversed(sources)), extra_files={"DOWNLOAD_INFO.csv": csv_receipt}).status == "invalid" with pytest.raises(FileValidationError, match="force=True"): install_archive( destination, list(reversed(sources)), verified=False, extra_files={"DOWNLOAD_INFO.csv": csv_receipt}) def test_trusted_part_hashes_allow_mirror_reuse(tmp_path, archive_file): _, data, _ = archive_file middle = len(data) // 2 pieces = [data[:middle], data[middle:]] sources = [] for index, piece in enumerate(pieces): path = tmp_path / ("part-%d" % index) path.write_bytes(piece) sources.append({ "url": "https://one.example/part-%d" % index, "path": path, "sha256": sha256(piece).hexdigest(), "size": len(piece), }) destination = tmp_path / "download" bundle = install_archive(destination, sources) mirrored = [dict(source, url=source["url"].replace("one", "two")) for source in sources] state = inspect_archive(destination, mirrored) assert state.status == "available" and state.verified assert install_archive(destination, mirrored) == bundle def test_url_parts_use_raw_fetch_and_forward_download_options(tmp_path, archive_file, monkeypatch): source, data, digest = archive_file calls = [] original = download.fetch_file def recording(url, **options): calls.append((url, dict(options))) return original(url, **options) monkeypatch.setattr(download, "fetch_file", recording) bundle = install_archive( tmp_path / "download", source.as_uri(), expected_sha256=digest, expected_size=len(data), download_options={"timeout": 12, "max_retries": 4}) assert bundle.is_dir() assert len(calls) == 1 assert calls[0][1]["raw"] is True assert calls[0][1]["timeout"] == 12 assert calls[0][1]["max_retries"] == 4 assert calls[0][1]["expected_sha256"] == digest @pytest.mark.parametrize("bad_member", [ tarfile.TarInfo("../escape"), tarfile.TarInfo("/absolute"), tarfile.TarInfo("models/link"), tarfile.TarInfo("models/fifo"), ]) def test_unsafe_members_are_rejected_without_publication(tmp_path, bad_member): if bad_member.name.endswith("link"): bad_member.type = tarfile.SYMTYPE bad_member.linkname = "../outside" elif bad_member.name.endswith("fifo"): bad_member.type = tarfile.FIFOTYPE path = make_tar(tmp_path / "bad.tar.bz2", [(bad_member.name, bad_member)]) outside = tmp_path / "escape" destination = tmp_path / "download" with pytest.raises(FileValidationError): install_archive(destination, path, verified=False) assert not outside.exists() assert inspect_archive(destination).status == "missing" assert not list((destination / "bundles").iterdir()) @pytest.mark.parametrize("members", [ [("A.txt", b"a"), ("a.txt", b"b")], [("models/file", b"a"), ("models/file", b"b")], [("models/file", b"a"), ("models/file/child", b"b")], [(".datacache-archive-manifest.json", b"forged")], ]) def test_colliding_and_reserved_member_names_are_rejected(tmp_path, members): path = make_tar(tmp_path / "bad.tar.bz2", members) with pytest.raises(FileValidationError): install_archive(tmp_path / "download", path, verified=False) def test_extra_file_collision_is_not_published(tmp_path, archive_file): source, _, _ = archive_file destination = tmp_path / "download" with pytest.raises(FileValidationError, match="collides"): install_archive( destination, source, verified=False, extra_files={"README.txt": "consumer receipt"}) assert inspect_archive(destination).status == "missing" def test_extra_file_casefolded_parent_collision_is_rejected(tmp_path): archive = make_tar(tmp_path / "archive.tar.bz2", [("Models/model.json", b"model")]) with pytest.raises(FileValidationError, match="collides"): install_archive( tmp_path / "download", archive, verified=False, extra_files={"models/DOWNLOAD_INFO.csv": "url\n"}) def test_failed_refresh_preserves_old_bundle(tmp_path, archive_file, monkeypatch): old_source, old_data, old_digest = archive_file destination = tmp_path / "download" old = install_archive( destination, old_source, expected_sha256=old_digest, expected_size=len(old_data)) old_bundle = inspect_archive(destination).bundle replacement = make_tar(tmp_path / "replacement.tar.bz2", [("new.txt", b"new")]) replacement_data = replacement.read_bytes() def fail(*args, **kwargs): raise OSError("injected extraction failure") monkeypatch.setattr(archives, "_extract_tar", fail) with pytest.raises(OSError, match="injected"): install_archive( destination, replacement, expected_sha256=sha256(replacement_data).hexdigest(), expected_size=len(replacement_data), force=True) assert inspect_archive(destination).bundle == old_bundle assert inspect_archive( destination, old_source, expected_sha256=old_digest, expected_size=len(old_data)).status == "available" assert (old / "README.txt").is_file() def test_interrupted_first_extraction_is_hidden_and_retryable( tmp_path, archive_file, monkeypatch): source, data, digest = archive_file destination = tmp_path / "download" original = archives._extract_tar def interrupt(archive_path, tree, *args, **kwargs): (tree / "partial.txt").write_text("not complete") raise OSError("interrupted extraction") monkeypatch.setattr(archives, "_extract_tar", interrupt) with pytest.raises(OSError, match="interrupted extraction"): install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) assert inspect_archive(destination).status == "missing" assert not list((destination / "bundles").iterdir()) monkeypatch.setattr(archives, "_extract_tar", original) bundle = install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) assert (bundle / "models/a/model.json").is_file() def test_tree_is_installed_once_renamed(tmp_path, archive_file, monkeypatch): source, data, digest = archive_file destination = tmp_path / "download" replace = os.replace def interrupt(source_path, target): replace(source_path, target) if Path(target).parent == destination / "bundles": raise KeyboardInterrupt("right after the bundle rename") monkeypatch.setattr(os, "replace", interrupt) with pytest.raises(KeyboardInterrupt): install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) monkeypatch.setattr(os, "replace", replace) source.unlink() def forbidden(*args, **kwargs): raise AssertionError("an installed archive was acquired again") monkeypatch.setattr(archives, "_assemble_archive", forbidden) bundle = install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) assert bundle.is_dir() assert inspect_archive( destination, source, expected_sha256=digest, expected_size=len(data)).verified def test_failed_manifest_write_leaves_no_apparently_installed_tree( tmp_path, archive_file, monkeypatch): source, data, digest = archive_file destination = tmp_path / "download" original = archives.write_json def fail(path, value, **kwargs): if Path(path).name == archives.MANIFEST: raise OSError("receipt publication failed") return original(path, value, **kwargs) monkeypatch.setattr(archives, "write_json", fail) with pytest.raises(OSError, match="receipt publication"): install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) assert inspect_archive(destination).status == "missing" assert not list((destination / "bundles").iterdir()) monkeypatch.setattr(archives, "write_json", original) bundle = install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) assert bundle.is_dir() def test_unreadable_generated_manifest_is_rejected_before_the_bundle_rename( tmp_path, archive_file, monkeypatch): source, data, digest = archive_file destination = tmp_path / "download" original = archives.write_json def corrupt(path, value, **kwargs): original(path, value, **kwargs) if Path(path).name == archives.MANIFEST: Path(path).write_text("invalid JSON") monkeypatch.setattr(archives, "write_json", corrupt) with pytest.raises(ValueError): install_archive(destination, source, expected_sha256=digest, expected_size=len(data)) assert inspect_archive(destination).status == "missing" assert not list((destination / "bundles").iterdir()) def test_populated_directories_are_never_claimed_even_with_force(tmp_path, archive_file): source, data, digest = archive_file destination = tmp_path / "legacy" destination.mkdir() (destination / "legacy.txt").write_text("keep me") for force in (False, True): with pytest.raises(FileValidationError, match="never taken over"): install_archive( destination, source, expected_sha256=digest, expected_size=len(data), force=force) assert [path.name for path in destination.iterdir()] == ["legacy.txt"] def test_an_empty_directory_becomes_the_store(tmp_path, archive_file): source, data, digest = archive_file destination = tmp_path / "precreated" destination.mkdir() assert inspect_archive(destination).status == "missing" bundle = install_archive(destination, source, expected_sha256=digest, expected_size=len(data)) assert (bundle / "README.txt").is_file() def test_tree_changes_and_extra_files_are_detected(tmp_path, archive_file): source, data, digest = archive_file destination = tmp_path / "download" bundle = install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) (bundle / "README.txt").write_text("changed") assert inspect_archive(destination).status == "invalid" (bundle / "unexpected.txt").write_text("extra") assert inspect_archive(destination).status == "invalid" with pytest.raises(FileValidationError, match="force=True"): install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) repaired = install_archive( destination, source, expected_sha256=digest, expected_size=len(data), force=True) assert repaired != bundle assert inspect_archive(destination).verified is False assert inspect_archive( destination, source, expected_sha256=digest, expected_size=len(data)).verified def test_links_added_after_installation_are_invalid_and_never_followed( tmp_path, archive_file): source, data, digest = archive_file destination = tmp_path / "download" bundle = install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) outside = tmp_path / "outside.txt" outside.write_text("outside") target = bundle / "README.txt" target.unlink() try: target.symlink_to(outside) except (OSError, NotImplementedError): pytest.skip("symlinks are unavailable") assert inspect_archive(destination).status == "invalid" assert outside.read_text() == "outside" @pytest.mark.skipif(os.name != "posix", reason="POSIX permission modes") @pytest.mark.parametrize("creation_mask", [0o022, 0o002, 0o077]) def test_published_tree_uses_normal_creation_permissions( tmp_path, archive_file, creation_mask): source, data, digest = archive_file previous = os.umask(creation_mask) try: destination = tmp_path / "download" bundle = install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) finally: os.umask(previous) assert destination.stat().st_mode & 0o777 == 0o777 & ~creation_mask assert bundle.stat().st_mode & 0o777 == 0o777 & ~creation_mask assert (bundle / "README.txt").stat().st_mode & 0o777 == 0o666 & ~creation_mask assert (bundle / archives.MANIFEST).stat().st_mode & 0o777 == 0o666 & ~creation_mask assert (destination / bundle_store.MARKER).stat().st_mode & 0o777 == 0o666 & ~creation_mask def test_concurrent_installers_converge_on_one_bundle(tmp_path, archive_file): source, data, digest = archive_file destination = tmp_path / "download" barrier = threading.Barrier(4) def install(_): barrier.wait(timeout=5) return install_archive( destination, source, expected_sha256=digest, expected_size=len(data)) with ThreadPoolExecutor(max_workers=4) as pool: results = list(pool.map(install, range(4))) assert len(set(results)) == 1 assert len(list((destination / "bundles").iterdir())) == 1 with ThreadPoolExecutor(max_workers=2) as pool: refreshed = list(pool.map( lambda _: install_archive( destination, source, expected_sha256=digest, expected_size=len(data), force=True), range(2))) assert len(set(refreshed)) == 2 assert all(path.is_dir() for path in results + refreshed) assert inspect_archive( destination, source, expected_sha256=digest, expected_size=len(data)).verified def test_limits_and_verification_requirements_fail_before_publication(tmp_path, archive_file): source, data, digest = archive_file with pytest.raises(ValueError, match="verified archive"): install_archive(tmp_path / "unverified", source) assert not (tmp_path / "unverified").exists() with pytest.raises(FileValidationError, match="members"): install_archive( tmp_path / "members", source, expected_sha256=digest, expected_size=len(data), max_members=1) with pytest.raises(FileValidationError, match="expands"): install_archive( tmp_path / "size", source, expected_sha256=digest, expected_size=len(data), max_extracted_size=1) def test_member_limit_stops_before_parsing_later_headers(tmp_path): # Parsing the deliberately invalid third header must never be necessary. source = tmp_path / "many.tar" source.write_bytes( tarfile.TarInfo("first").tobuf() + tarfile.TarInfo("second").tobuf() + b"invalid header".ljust(512, b"\0")) with pytest.raises(FileValidationError, match="more than 1 members"): install_archive(tmp_path / "download", source, verified=False, max_members=1) def test_expansion_limit_stops_before_traversing_compressed_member_data(tmp_path): member = tarfile.TarInfo("huge") member.size = 2 ** 30 # The advertised payload is absent: scanning to the next header would fail. source = tmp_path / "huge.tar.gz" source.write_bytes(gzip.compress(member.tobuf())) with pytest.raises(FileValidationError, match="expands to more than 1024 bytes"): install_archive( tmp_path / "download", source, verified=False, max_extracted_size=1024) def test_resumable_split_downloads_require_part_integrity(tmp_path, archive_file): source, data, digest = archive_file half = len(data) // 2 first, second = tmp_path / "one", tmp_path / "two" first.write_bytes(data[:half]) second.write_bytes(data[half:]) with pytest.raises(ValueError, match="resumable archive parts"): install_archive( tmp_path / "download", [first.as_uri(), second.as_uri()], expected_sha256=digest, expected_size=len(data), download_options={"resume": True}) def test_versioned_archive_registry_supports_consumer_layout_and_status( tmp_path, archive_file): source, data, digest = archive_file url = "https://example.test/models.tar.bz2" receipt = "url\n%s\n" % url paths = [] def consumer_path(name, version): paths.append((name, version)) return tmp_path / "releases" / version / name registry = VersionedArchiveRegistry({ "models": { "default_version": "2.3.0", "description": "Presentation models", "versions": { "2.3.0": { "sources": url, "expected_sha256": digest, "expected_size": len(data), "extra_files": {"DOWNLOAD_INFO.csv": receipt}, }, "2.2.0": { "sources": "https://example.test/older.tar.bz2", "expected_sha256": digest, "expected_size": len(data), }, }, }, }, store_path=consumer_path) assert registry.resolve_version("models") == "2.3.0" assert registry.store_path("models") == tmp_path / "releases/2.3.0/models" assert not (tmp_path / "releases").exists() rows = registry.status() assert [(row["version"], row["status"]) for row in rows] == [ ("2.3.0", "missing"), ("2.2.0", "missing")] assert rows[0]["default"] and rows[0]["sources"] == [url] assert rows[0]["downloaded_sources"] == [] bundle = registry.download("models", source_paths=source) assert (bundle / "models/a/model.json").is_file() assert registry.is_cached("models") assert registry.local_path("models") == bundle assert registry.inspect("models").verified assert not registry.is_cached("models", "2.2.0") installed = registry.status("models")[0] assert installed["bundle"] == str(bundle) assert installed["archive_size"] == len(data) assert installed["downloaded_sources"] == [url] assert ("models", "2.3.0") in paths source.unlink() assert registry.local_path("models") == bundle with pytest.raises(FileNotFoundError): registry.local_path("models", "2.2.0") with pytest.raises(ValueError, match="unknown archive"): registry.resolve_version("unknown") def test_versioned_archive_registry_default_layout_and_unverified_history( tmp_path, archive_file): source, _, _ = archive_file registry = VersionedArchiveRegistry({ "data": { "default_version": "historical", "versions": {"historical": source}, }, }, cache_root=tmp_path / "cache", verified=False) assert registry.store_path("data") == tmp_path / "cache/data/historical" bundle = registry.ensure("data") assert bundle.is_dir() assert registry.inspect("data").status == "available" assert not registry.inspect("data").verified @pytest.mark.parametrize('warm', [False, True]) @pytest.mark.parametrize('raises', [False, True]) def test_validator_rejects_before_publication(tmp_path, archive_file, warm, raises): source, _, _ = archive_file destination = tmp_path / 'download' previous = install_archive(destination, source, verified=False) if warm else None def reject(tree): assert (tree / 'models/a/model.json').is_file() assert (tree / 'source.txt').read_text() == 'extra metadata' assert inspect_archive(destination).bundle == (str(previous) if warm else None) if raises: raise ValueError('required model absent') return False with pytest.raises(FileValidationError, match='archive tree failed validation') as failure: install_archive(destination, source, verified=False, force=True, extra_files={'source.txt': 'extra metadata'}, validator=reject) if raises: assert isinstance(failure.value.__cause__, ValueError) state = inspect_archive(destination) assert state.status == ('available' if warm else 'missing') assert state.bundle == (str(previous) if warm else None) assert not list(destination.glob('.staging-*')) repaired = install_archive(destination, source, verified=False, force=True, validator=lambda tree: (tree / 'models/a/model.json').is_file()) assert inspect_archive(destination).bundle == str(repaired) def test_validator_checks_cached_tree_without_writes(tmp_path, archive_file, monkeypatch): source, _, _ = archive_file destination = tmp_path / 'download' previous = install_archive(destination, source, verified=False) before = {p: p.stat().st_mtime_ns for p in destination.rglob('*')} def forbidden(*args, **kwargs): pytest.fail('cached validator should not download or acquire a write lock') monkeypatch.setattr(archives, '_assemble_archive', forbidden) monkeypatch.setattr(bundle_store.BundleStore, 'lock', forbidden) checked = [] assert install_archive(destination, source, verified=False, validator=lambda tree: checked.append(tree)) == previous assert checked == [previous] with pytest.raises(FileValidationError, match='archive tree failed validation'): install_archive(destination, source, verified=False, validator=lambda tree: False) assert {p: p.stat().st_mtime_ns for p in destination.rglob('*')} == before assert inspect_archive(destination).bundle == str(previous) def test_noncallable_archive_validator_rejected_before_writes(tmp_path, archive_file): source, _, _ = archive_file destination = tmp_path / 'absent' / 'download' with pytest.raises(ValueError, match='validator must be callable'): install_archive(destination, source, verified=False, validator=True) assert not destination.parent.exists() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7127903 datacache-1.27.1/tests/test_bundle_lifecycle.py0000644000076500000000000002175515264220442021273 0ustar00iskanderwheel"""Listing and pruning bundles (#91), and local or predownloaded bundle files (#94).""" import gzip from hashlib import sha256 import io from pathlib import Path import tarfile import pytest from datacache import ( FileValidationError, VersionedArchiveRegistry, VersionedDatasetRegistry, inspect_bundle, install_archive, install_bundle, list_bundles, materialize, prune_bundles, ) from datacache import download REMOTE = "https://data.example.invalid/release-1/genes.gtf" def describe(data): return {"sha256": sha256(data).hexdigest(), "size": len(data)} @pytest.fixture def local_file(tmp_path): path = tmp_path / "genes.gtf" path.write_bytes(b"gene\n") return path @pytest.fixture def never_downloads(monkeypatch): """Fail if anything but a local file:// source is fetched.""" fetch_file = download.fetch_file def local_only(url, *args, **kwargs): assert url.startswith("file:"), "downloaded %s" % url return fetch_file(url, *args, **kwargs) monkeypatch.setattr(download, "fetch_file", local_only) def test_bundles_are_listed_oldest_first_and_pruned_to_the_newest(tmp_path, local_file): store = tmp_path / "store" assets = {"genes.gtf": dict(describe(b"gene\n"), path=local_file)} assert list_bundles(store) == [] and prune_bundles(store) == [] first = Path(install_bundle(store, assets)["genes.gtf"]).parent second = Path(install_bundle(store, assets, force=True)["genes.gtf"]).parent third = Path(install_bundle(store, assets, force=True)["genes.gtf"]).parent assert list_bundles(store) == [first, second, third] assert Path(inspect_bundle(store).bundle) == third assert prune_bundles(store, keep=2) == [first] assert prune_bundles(store) == [second] assert prune_bundles(store) == [] assert list_bundles(store) == [third] and inspect_bundle(store, assets).verified for keep in (0, -1, True, 1.5): with pytest.raises(ValueError, match="current bundle is always kept"): prune_bundles(store, keep=keep) def test_every_kind_of_store_can_be_pruned(tmp_path, local_file): archive = io.BytesIO() with tarfile.open(fileobj=archive, mode="w") as tar: info = tarfile.TarInfo("README") info.size = 5 tar.addfile(info, io.BytesIO(b"hello")) tar_path = tmp_path / "data.tar" tar_path.write_bytes(archive.getvalue()) registry = VersionedArchiveRegistry( {"data": {"default_version": "v1", "versions": {"v1": {"path": tar_path, **describe(tar_path.read_bytes())}}}}, cache_root=tmp_path / "archives") old = registry.download("data") new = registry.download("data", force=True) assert registry.prune("data") == [old] and list_bundles(registry.store_path("data")) == [new] options = dict(sources={"in": {"path": local_file}}, transform={"version": "1"}, outputs={"out": {}}, builder=lambda inputs, outputs: Path(outputs["out"]).write_bytes(b"x"), download_options={"show_progress": False}) store = tmp_path / "derived" old = Path(materialize(store, **options)["out"]).parent new = Path(materialize(store, force=True, **options)["out"]).parent assert prune_bundles(store) == [old] and list_bundles(store) == [new] datasets = VersionedDatasetRegistry( {"genes": {"default_version": "v1", "versions": {"v1": {"genes.gtf": dict(describe(b"gene\n"), path=local_file)}}}}, cache_root=tmp_path / "datasets") old = datasets.ensure("genes").parent datasets.download("genes", force=True) assert datasets.prune("genes") == [old] def test_only_datacache_stores_are_pruned(tmp_path): foreign = tmp_path / "foreign" (foreign / "bundles" / "2026-01-01T00-00-00Z").mkdir(parents=True) (foreign / "keep.txt").write_text("keep") with pytest.raises(FileValidationError, match="not a datacache store"): prune_bundles(foreign) assert (foreign / "bundles" / "2026-01-01T00-00-00Z").is_dir() def test_predownloaded_files_keep_the_declared_source(tmp_path, local_file, never_downloads): assets = {"genes.gtf": dict(describe(b"gene\n"), url=REMOTE)} store = tmp_path / "store" paths = install_bundle(store, assets, source_paths={"genes.gtf": local_file}) assert Path(paths["genes.gtf"]).read_bytes() == b"gene\n" # Recorded as if downloaded, so later installs reuse it without the file. local_file.unlink() assert install_bundle(store, assets) == paths assert inspect_bundle(store, assets).verified def test_predownloaded_files_without_trusted_hashes(tmp_path, local_file, never_downloads): registry = VersionedDatasetRegistry( {"genes": {"default_version": "v1", "versions": {"v1": {"genes.gtf": {"url": REMOTE}}}}}, cache_root=tmp_path / "cache", verified=False) path = registry.ensure("genes", source_paths={"genes.gtf": local_file}) assert path.read_bytes() == b"gene\n" # The declared URL's identity was recorded, so this is a cache hit. assert registry.download("genes")["genes.gtf"] == str(path) def test_predownloaded_archives_are_decompressed(tmp_path, never_downloads): compressed = tmp_path / "genes.gtf.gz" compressed.write_bytes(gzip.compress(b"gene\n", mtime=0)) assets = {"genes.gtf": dict(describe(b"gene\n"), url=REMOTE + ".gz", decompress=True)} paths = install_bundle(tmp_path / "store", assets, source_paths={"genes.gtf": compressed}) assert Path(paths["genes.gtf"]).read_bytes() == b"gene\n" @pytest.mark.parametrize("source_paths, message", [ ({"other.gtf": "x"}, "unknown assets"), ({"genes.gtf": 7}, "must be a path"), ([], "must map asset names"), ]) def test_invalid_source_paths_are_rejected_before_anything_is_created(tmp_path, source_paths, message): assets = {"genes.gtf": dict(describe(b"gene\n"), url=REMOTE)} with pytest.raises(ValueError, match=message): install_bundle(tmp_path / "store", assets, source_paths=source_paths) assets["genes.gtf"]["decompress"] = True with pytest.raises(ValueError, match=r"\.gz or \.zip"): install_bundle(tmp_path / "store", assets, source_paths={"genes.gtf": tmp_path / "genes.gtf"}) with pytest.raises(ValueError, match="exactly one of url or path"): install_bundle(tmp_path / "store", {"genes.gtf": {"url": REMOTE, "path": "x"}}, verified=False) assert not (tmp_path / "store").exists() def test_stores_are_recognized_only_by_a_known_marker(tmp_path): from datacache.bundle_store import MARKER (tmp_path / "empty").mkdir() assert list_bundles(tmp_path / "empty") == [] and prune_bundles(tmp_path / "empty") == [] (tmp_path / "odd").mkdir() (tmp_path / "odd" / MARKER).write_text('{"format": 2, "kind": "something-else"}') with pytest.raises(FileValidationError, match="not a datacache store"): list_bundles(tmp_path / "odd") def test_users_are_told_apart_without_numeric_ids(monkeypatch): import getpass import os from datacache.bundle_store import user_key monkeypatch.delattr(os, "getuid", raising=False) # As on Windows. assert user_key() == getpass.getuser() def test_local_files_need_no_resume_support(tmp_path, local_file, never_downloads): assets = {"genes.gtf": dict(describe(b"gene\n"), url=REMOTE)} paths = install_bundle(tmp_path / "store", assets, download_options={"resume": True}, source_paths={"genes.gtf": local_file}) assert Path(paths["genes.gtf"]).read_bytes() == b"gene\n" def test_auto_resume_never_keeps_unpinned_files_between_attempts(tmp_path): first, second = tmp_path / "a.txt", tmp_path / "b.txt" first.write_bytes(b"release 1\n") assets = {"a.txt": first.as_uri(), "b.txt": second.as_uri()} # Unpinned; b is missing. store = tmp_path / "store" with pytest.raises(Exception): install_bundle(store, assets, verified=False, download_options={"resume": "auto"}) first.write_bytes(b"release 2\n") # Upstream moves on before the retry. second.write_bytes(b"release 2\n") paths = install_bundle(store, assets, verified=False, download_options={"resume": "auto"}) assert Path(paths["a.txt"]).read_bytes() == b"release 2\n" def test_auto_resume_never_keeps_unpinned_archive_parts(tmp_path): archive = io.BytesIO() with tarfile.open(fileobj=archive, mode="w") as tar: info = tarfile.TarInfo("README") info.size = 5 tar.addfile(info, io.BytesIO(b"hello")) data = archive.getvalue() parts = [tmp_path / "part0", tmp_path / "part1"] parts[0].write_bytes(data[:100]) parts[1].write_bytes(b"an error page instead of the second part") sources = [part.as_uri() for part in parts] # Unpinned parts; the whole is pinned. options = dict(expected_sha256=sha256(data).hexdigest(), expected_size=len(data), download_options={"resume": "auto"}) with pytest.raises(FileValidationError): install_archive(tmp_path / "store", sources, **options) parts[1].write_bytes(data[100:]) # The upstream recovers. bundle = install_archive(tmp_path / "store", sources, **options) assert (bundle / "README").read_bytes() == b"hello" ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7129006 datacache-1.27.1/tests/test_bundle_metadata.py0000644000076500000000000002744315264220442021114 0ustar00iskanderwheel"""Fast bundle reads retain snapshot, ownership and explicit trust boundaries.""" import builtins from concurrent.futures import ThreadPoolExecutor from hashlib import sha256 import io import json import os from pathlib import Path import threading import pytest from datacache import FileValidationError, VersionedDatasetRegistry, inspect_bundle, install_bundle from datacache import bundle_store, bundles, download pytestmark = pytest.mark.skipif(os.name != 'posix', reason='POSIX bundle installation') @pytest.fixture def installed(tmp_path): upstream = tmp_path / 'upstream' upstream.mkdir() assets = {} for name, payload in [('records.txt', b'records'), ('nested/metadata.txt', b'metadata')]: source = upstream / name source.parent.mkdir(exist_ok=True) source.write_bytes(payload) assets[name] = dict(url=source.as_uri(), sha256=sha256(payload).hexdigest(), size=len(payload)) registry = VersionedDatasetRegistry( {'reference': dict(default_version='v1', versions={'v1': assets})}, cache_root=tmp_path / 'cache') paths = registry.download('reference') return registry, assets, paths @pytest.mark.parametrize('trusted', [True, False]) def test_fast_resolution_and_cache_hits_read_no_payloads_or_write(installed, monkeypatch, trusted): registry, assets, paths = installed if not trusted: assets = {name: {'url': spec['url']} for name, spec in assets.items()} registry = VersionedDatasetRegistry( {'reference': dict(default_version='v1', versions={'v1': assets})}, cache_root=registry.store_path('reference').parents[1], verified=False) payloads = set(map(Path, paths.values())) original_open, original_os_open, original_read = builtins.open, os.open, os.read payload_fds = set() def guarded_open(path, *args, **kwargs): if isinstance(path, (str, os.PathLike)) and Path(path) in payloads: raise AssertionError('fast lookup opened a payload for content reading') return original_open(path, *args, **kwargs) def tracked_os_open(path, *args, **kwargs): # Opening a payload to check its type and readability is allowed; # reading from it is not. fd = original_os_open(path, *args, **kwargs) if isinstance(path, (str, os.PathLike)) and Path(path) in payloads: payload_fds.add(fd) return fd def guarded_read(fd, *args, **kwargs): if fd in payload_fds: raise AssertionError('fast lookup read payload bytes') return original_read(fd, *args, **kwargs) def forbidden(*args, **kwargs): raise AssertionError('fast lookup tried full inspection, mutation or acquisition') monkeypatch.setattr(builtins, 'open', guarded_open) monkeypatch.setattr(io, 'open', guarded_open) monkeypatch.setattr(os, 'open', tracked_os_open) monkeypatch.setattr(os, 'read', guarded_read) hash_file = bundles.hash_file def sizes_only(*args, **kwargs): if kwargs.get('hash_contents', True): raise AssertionError('fast lookup tried full inspection') return hash_file(*args, **kwargs) monkeypatch.setattr(bundles, 'hash_file', sizes_only) monkeypatch.setattr(bundle_store.BundleStore, 'lock', forbidden) monkeypatch.setattr(bundles, 'write_json', forbidden) monkeypatch.setattr(bundle_store, 'write_json', forbidden) monkeypatch.setattr(download, 'fetch_file', forbidden) monkeypatch.setattr(Path, 'mkdir', forbidden) state = registry.inspect('reference', verify_files=False) assert state.status == 'available' and not state.verified assert {name: item.path for name, item in state.files.items()} == paths assert all(not item.verified and item.size is not None for item in state.files.values()) assert registry.local_path('reference', verify_files=False) == Path(state.bundle) assert registry.local_path( 'reference', asset='records.txt', verify_files=False) == Path(paths['records.txt']) assert registry.is_cached('reference', verify_files=False) assert not registry.status(verify_files=False)[0]['inspection'].verified assert registry.download('reference', verify_files=False) == paths assert registry.ensure('reference', verify_files=False) == Path(state.bundle) assert inspect_bundle(registry.store_path('reference'), verify_files=False).status == 'available' with pytest.raises(AssertionError, match='full inspection'): registry.inspect('reference') def test_defaults_agree_on_same_size_corruption_that_only_fast_checks_miss(installed): registry, assets, paths = installed path = Path(paths['records.txt']) path.write_bytes(b'changed') assert len(b'changed') == assets['records.txt']['size'] # Every lookup verifies by default, so they agree the bundle is invalid. assert registry.inspect('reference').status == 'invalid' assert not registry.is_cached('reference') assert registry.status()[0]['inspection'].status == 'invalid' with pytest.raises(FileValidationError): registry.local_path('reference') with pytest.raises(FileValidationError, match='force=True'): registry.download('reference') with pytest.raises(FileValidationError, match='force=True'): registry.ensure('reference') # Opting into metadata-only checks trades that detection for speed. assert registry.inspect('reference', verify_files=False).status == 'available' assert registry.is_cached('reference', verify_files=False) assert registry.download('reference', verify_files=False) == paths def test_fast_resolution_on_a_read_only_store(installed): registry, assets, paths = installed store = registry.store_path('reference') entries = [store, *store.rglob('*')] modes = {path: path.stat().st_mode & 0o777 for path in entries} try: for path in entries: path.chmod(0o555 if path.is_dir() else 0o444) state = registry.inspect('reference', verify_files=False) assert state.status == 'available' and not state.verified assert registry.local_path('reference', verify_files=False) == Path(state.bundle) assert registry.is_cached('reference', verify_files=False) assert registry.status(verify_files=False)[0]['inspection'].status == 'available' assert registry.download('reference', verify_files=False) == paths finally: for path, mode in modes.items(): path.chmod(mode) @pytest.mark.parametrize('damage', ['missing', 'size', 'symlink', 'hardlink', 'fifo', 'parent-link', 'newest-link', 'manifest', 'inventory', 'expectations']) def test_fast_lookup_rejects_invalid_metadata_and_unsafe_files(installed, tmp_path, damage): registry, assets, paths = installed store = registry.store_path('reference') target = Path(paths['records.txt']) bundle = target.parent manifest = bundle / bundles.MANIFEST if damage == 'missing': target.unlink() elif damage == 'size': target.write_bytes(b'short') elif damage == 'symlink': target.unlink() target.symlink_to(tmp_path / 'upstream/records.txt') elif damage == 'hardlink': os.link(target, tmp_path / 'hardlink') elif damage == 'fifo': target.unlink() os.mkfifo(target) elif damage == 'parent-link': nested = bundle / 'nested' nested.rename(bundle / 'original-nested') nested.symlink_to(bundle / 'original-nested', target_is_directory=True) elif damage == 'newest-link': (store / 'bundles' / '2999-01-01T00-00-00Z').symlink_to(tmp_path / 'upstream', target_is_directory=True) elif damage == 'manifest': manifest.write_text('{') elif damage == 'inventory': receipt = json.loads(manifest.read_text()) del receipt['assets']['records.txt'] manifest.write_text(json.dumps(receipt)) else: assets['records.txt']['sha256'] = '0' * 64 assert inspect_bundle(store, assets, verify_files=False).status == 'invalid' def test_fast_lookup_retains_unverified_source_identity_checks(installed): registry, assets, paths = installed untrusted = {name: dict(url=spec['url']) for name, spec in assets.items()} untrusted['records.txt']['url'] = 'https://different.invalid/records.txt' assert inspect_bundle(registry.store_path('reference'), untrusted, verify_files=False).status == 'invalid' def test_fast_lookup_reports_permission_errors(installed, monkeypatch): registry, assets, paths = installed original = bundle_store.open_regular def denied(path, *args, **kwargs): if str(path) == paths['records.txt']: raise PermissionError('payload is not readable') return original(path, *args, **kwargs) monkeypatch.setattr(bundle_store, 'open_regular', denied) state = registry.inspect('reference', verify_files=False) assert state.status == 'inaccessible' assert isinstance(state.error, PermissionError) def test_fast_install_flag_never_weakens_new_bundle_validation(installed, monkeypatch): registry, assets, paths = installed store = registry.store_path('reference') old_bundle = inspect_bundle(store).bundle def bad_download(url, *, destination, **kwargs): destination.parent.mkdir(parents=True, exist_ok=True) destination.write_bytes(b'x' * kwargs['expected_size']) return str(destination) monkeypatch.setattr(download, 'fetch_file', bad_download) with pytest.raises(FileValidationError, match='SHA-256'): registry.download('reference', force=True, verify_files=False) assert inspect_bundle(store).bundle == old_bundle assert registry.local_path('reference', asset='records.txt') == Path(paths['records.txt']) def test_fast_readers_resolve_one_bundle_during_refresh(installed, monkeypatch): registry, assets, paths = installed entered, release = threading.Event(), threading.Event() replace = os.replace bundles_directory = registry.store_path('reference') / 'bundles' def paused(source, target): if Path(target).parent == bundles_directory: entered.set() assert release.wait(5) return replace(source, target) monkeypatch.setattr(os, 'replace', paused) with ThreadPoolExecutor() as pool: future = pool.submit(registry.download, 'reference', force=True) assert entered.wait(5) try: state = registry.inspect('reference', verify_files=False) assert {name: item.path for name, item in state.files.items()} == paths finally: release.set() refreshed = future.result() state = registry.inspect('reference', verify_files=False) assert {name: item.path for name, item in state.files.items()} == refreshed assert all(Path(path).exists() for path in paths.values()) @pytest.mark.parametrize('flag', [None, 0, 'false']) def test_nonboolean_verification_modes_fail_without_creating_paths(tmp_path, flag): dest = tmp_path / 'missing' with pytest.raises(ValueError, match='verify_files'): inspect_bundle(dest, verify_files=flag) with pytest.raises(ValueError, match='verify_files'): install_bundle(dest, {'file': {'url': 'https://example.invalid/data'}}, verified=False, verify_files=flag) assert not dest.exists() def test_ensure_returns_the_downloaded_snapshot_without_inspecting_again(installed, monkeypatch): registry, assets, paths = installed bundle = Path(paths['records.txt']).parent monkeypatch.setattr(registry, 'inspect', lambda *args, **kwargs: pytest.fail('inspected again')) monkeypatch.setattr(bundles, 'install_bundle', lambda *args, **kwargs: paths) assert registry.ensure('reference') == bundle single = {'records.txt': paths['records.txt']} monkeypatch.setattr(bundles, 'install_bundle', lambda *args, **kwargs: single) assert registry.ensure('reference') == Path(paths['records.txt']) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7130377 datacache-1.27.1/tests/test_bundle_store.py0000644000076500000000000003420015264220442020455 0ustar00iskanderwheel"""The store layout shared by bundles, archive trees and materializations.""" from datetime import datetime, timezone from hashlib import sha256 import json import os from pathlib import Path import shutil import pytest from datacache import FileValidationError, install_archive, install_bundle, materialize from datacache import bundle_store from datacache.bundle_store import ( MARKER, BundleStore, check_tree, hash_file, is_bundle_name, list_tree, local_file_identity, validate_distinct_paths, validate_file_record, validate_no_sidecar_collisions, validate_path_component, validate_relative_name, ) from datacache.download import validate_download_options pytestmark = pytest.mark.skipif(os.name != 'posix', reason='POSIX permissions and links') def publish(store, payload=b'data'): staged = store.path / '.staging' staged.mkdir() (staged / 'file.txt').write_bytes(payload) return store.publish(staged) def test_the_newest_bundle_is_current(tmp_path): store = BundleStore(tmp_path / 'store', 'bundle') assert store.current_bundle() is None store.create() assert json.loads((store.path / MARKER).read_text()) == {'format': 2, 'kind': 'bundle'} assert store.current_bundle() is None first = publish(store, b'first') second = publish(store, b'second') # Two publishes in one second still sort in order. assert store.bundle_names() == [first.name, second.name] and first.name < second.name assert store.current_bundle() == second assert (first / 'file.txt').read_bytes() == b'first' def test_bundle_names_are_utc_times_that_never_go_backwards(tmp_path, monkeypatch): store = BundleStore(tmp_path / 'store', 'bundle') store.create() now = datetime.now(timezone.utc) assert abs(datetime.strptime(publish(store).name, '%Y-%m-%dT%H-%M-%SZ').replace(tzinfo=timezone.utc) - now).total_seconds() < 5 (store.bundles / '2999-01-01T00-00-00Z').mkdir() # The clock is behind the newest bundle: the next one still sorts after it. assert publish(store).name == '2999-01-01T00-00-01Z' assert store.current_bundle().name == '2999-01-01T00-00-01Z' @pytest.mark.parametrize('name, expected', [ ('2026-10-08T17-02-42Z', True), ('\u0662\u0660\u0662\u0666-10-08T17-02-42Z', False), # Non-ASCII digits. ('2026-1-08T17-02-42Z', False), ('2026-13-08T17-02-42Z', False), ('2026-10-08T17:02:42Z', False), ('notes', False), ]) def test_only_canonical_utc_times_are_bundle_names(name, expected): assert is_bundle_name(name) is expected def test_a_bundle_dated_too_late_to_follow_is_reported(tmp_path): store = BundleStore(tmp_path / 'store', 'bundle') store.create() (store.bundles / '9999-12-31T23-59-59Z').mkdir() (store.path / '.staging').mkdir() with pytest.raises(FileValidationError, match='no newer name'): store.publish(store.path / '.staging') def test_other_entries_in_bundles_are_ignored(tmp_path): store = BundleStore(tmp_path / 'store', 'bundle') store.create() bundle = publish(store) (store.bundles / '.DS_Store').write_text('') (store.bundles / 'notes').mkdir() assert store.bundle_names() == [bundle.name] assert store.current_bundle() == bundle def test_a_linked_newest_bundle_is_rejected(tmp_path): store = BundleStore(tmp_path / 'store', 'bundle') store.create() publish(store) (tmp_path / 'elsewhere').mkdir() (store.bundles / '2999-01-01T00-00-00Z').symlink_to(tmp_path / 'elsewhere', target_is_directory=True) with pytest.raises(FileValidationError, match='not a link'): store.current_bundle() def test_an_empty_directory_becomes_the_store_in_place(tmp_path): (tmp_path / 'store').mkdir(mode=0o750) before = (tmp_path / 'store').stat() store = BundleStore(tmp_path / 'store', 'archive') assert store.current_bundle() is None store.create() # The same directory, so its owner, group and permissions are kept. after = store.path.stat() assert (after.st_ino, after.st_dev) == (before.st_ino, before.st_dev) assert (after.st_mode & 0o777) == 0o750 assert sorted(path.name for path in store.path.iterdir()) == [MARKER, 'bundles'] assert [path.name for path in tmp_path.iterdir()] == ['store'] # No leftover staged marker. def test_a_store_missing_bundles_has_nothing_installed_and_is_repaired(tmp_path): store = BundleStore(tmp_path / 'store', 'bundle') store.create() publish(store) shutil.rmtree(store.bundles) # Also what readers see between create() placing the marker and bundles/. assert store.current_bundle() is None store.create() assert store.current_bundle() is None and store.bundles.is_dir() assert publish(store) == store.current_bundle() def test_spellings_of_one_directory_share_a_lock_beside_it(tmp_path, monkeypatch): def lock(name): return BundleStore(tmp_path / name, 'bundle').lock_path assert lock('GRCh38') == lock('grch38') assert lock('caf\u00e9') == lock('cafe\u0301') # NFC and NFD, as macOS may report them. assert lock('a') != lock('b') and lock('a').parent == tmp_path # A store named "." still locks beside itself, never inside. (tmp_path / 'store').mkdir() monkeypatch.chdir(tmp_path / 'store') assert BundleStore('.', 'archive').lock_path.parent == tmp_path def test_directories_with_files_are_never_taken_over(tmp_path): (tmp_path / 'legacy').mkdir() (tmp_path / 'legacy' / 'keep.txt').write_text('keep me') store = BundleStore(tmp_path / 'legacy', 'bundle') with pytest.raises(FileValidationError, match='never taken over'): store.create() assert [path.name for path in store.path.iterdir()] == ['keep.txt'] @pytest.mark.parametrize('make', [ lambda path: path.write_text('a file'), lambda path: path.symlink_to(path.parent / 'elsewhere', target_is_directory=True), ], ids=['file', 'link']) def test_anything_but_a_directory_is_never_taken_over(tmp_path, make): (tmp_path / 'elsewhere').mkdir() make(tmp_path / 'store') store = BundleStore(tmp_path / 'store', 'bundle') with pytest.raises(FileValidationError, match='never taken over'): store.refuse_takeover() with pytest.raises(FileValidationError, match='never taken over'): install_bundle(tmp_path / 'store', {'a': {'url': 'https://example.test/a', 'sha256': '0' * 64, 'size': 1}}) def test_datacache_temporary_files_dont_make_a_directory_foreign(tmp_path, monkeypatch): (tmp_path / 'store').mkdir() (tmp_path / 'store' / '.datacache-json-interrupted').write_text('{') store = BundleStore(tmp_path / 'store', 'bundle') assert store.current_bundle() is None store.create() assert store.current_bundle() is None # A store named "." is created in place, with nothing left beside it. (tmp_path / 'here').mkdir() monkeypatch.chdir(tmp_path / 'here') BundleStore('.', 'archive').create() assert sorted(path.name for path in (tmp_path / 'here').iterdir()) == [MARKER, 'bundles'] assert sorted(path.name for path in tmp_path.iterdir()) == ['here', 'store'] def test_one_kind_never_uses_another_kinds_store(tmp_path): BundleStore(tmp_path / 'store', 'archive').create() other = BundleStore(tmp_path / 'store', 'bundle') with pytest.raises(FileValidationError, match='not a datacache bundle store'): other.current_bundle() # Installs check this before suggesting force=True, which can't help. with pytest.raises(FileValidationError, match='never taken over'): other.refuse_takeover() with pytest.raises(FileValidationError, match='never taken over'): other.create() def test_trees_are_listed_completely_before_any_file_is_read(tmp_path, monkeypatch): root = tmp_path / 'tree' (root / 'a' / 'b').mkdir(parents=True) (root / 'a' / 'b' / 'c.txt').write_bytes(b'abc') (root / 'extra.txt').write_bytes(b'x') (root / '.datacache-manifest.json').write_text('{}') listing = list_tree(root, ignore=('.datacache-manifest.json',)) assert list(listing.files) == ['a/b/c.txt', 'extra.txt'] assert listing.directories == ['a', 'a/b'] def forbidden(*args, **kwargs): raise AssertionError('read a file before checking the inventory') monkeypatch.setattr(bundle_store, 'hash_file', forbidden) record = {'sha256': sha256(b'abc').hexdigest(), 'size': 3} with pytest.raises(FileValidationError, match=r"missing \[\], unexpected \['extra.txt'\]"): check_tree(root, {'a/b/c.txt': record}, ['a', 'a/b'], ignore=('.datacache-manifest.json',)) with pytest.raises(FileValidationError, match=r"directories differ.*unexpected \['a/b'\]"): check_tree(root, {'a/b/c.txt': record, 'extra.txt': record}, ['a'], ignore=('.datacache-manifest.json',)) monkeypatch.undo() (root / 'extra.txt').unlink() hashed = check_tree(root, {'a/b/c.txt': record}, ['a', 'a/b'], ignore=('.datacache-manifest.json',)) assert hashed['a/b/c.txt'].record == record with pytest.raises(FileValidationError, match='SHA-256'): check_tree(root, {'a/b/c.txt': dict(record, sha256='0' * 64)}, ['a', 'a/b'], ignore=('.datacache-manifest.json',)) @pytest.mark.parametrize('make', [ lambda root: (root / 'link').symlink_to(root / 'file.txt'), lambda root: (root / 'directory-link').symlink_to(root, target_is_directory=True), lambda root: os.mkfifo(root / 'fifo'), ], ids=['file-link', 'directory-link', 'fifo']) def test_trees_hold_only_regular_files_and_directories(tmp_path, make): root = tmp_path / 'tree' root.mkdir() (root / 'file.txt').write_bytes(b'data') make(root) with pytest.raises(FileValidationError, match='not a link or special file'): list_tree(root) def test_hashing_can_check_only_the_size(tmp_path): path = tmp_path / 'file.txt' path.write_bytes(b'data') hashed = hash_file(path, {'sha256': '0' * 64, 'size': 4}, hash_contents=False) assert hashed.sha256 is None and hashed.size == 4 with pytest.raises(FileValidationError, match='size is 4 bytes, expected 5'): hash_file(path, {'sha256': None, 'size': 5}, hash_contents=False) assert hash_file(path).record == {'sha256': sha256(b'data').hexdigest(), 'size': 4} os.link(path, tmp_path / 'linked.txt') with pytest.raises(FileValidationError, match='single link'): hash_file(path) @pytest.mark.parametrize('record', [ None, {}, {'sha256': None, 'size': 1}, {'sha256': '0' * 64, 'size': None}, {'sha256': 'xyz', 'size': 1}, {'sha256': '0' * 64, 'size': -1}, {'sha256': '0' * 64, 'size': 1, 'url': ''}, ]) def test_file_records_must_identify_the_bytes(record): with pytest.raises(ValueError): validate_file_record(record) @pytest.mark.parametrize('name', [ '', '/absolute', 'a//b', './a', 'a/..', 'back\\slash', 'trailing.', 'trailing ', 'colon:', 'CON', 'lpt1.txt', '.datacache-manifest.json', 'nested/.DataCache-x', 7, ]) def test_unsafe_relative_names_are_rejected(name): with pytest.raises(ValueError): validate_relative_name(name) def test_name_validation(): assert validate_relative_name('a/b.c') == 'a/b.c' assert validate_path_component('2026-09') == '2026-09' with pytest.raises(ValueError, match='single path components'): validate_path_component('a/b') validate_distinct_paths(['a/b', 'a/c', 'd']) for names in (['a', 'A'], ['a', 'a/b'], ['A', 'a/b'], ['A/x', 'a/y'], ['x', 'x']): with pytest.raises(ValueError, match='collide'): validate_distinct_paths(names) validate_no_sidecar_collisions(['a.txt', 'dir/b.txt']) with pytest.raises(ValueError, match='sidecars'): validate_no_sidecar_collisions(['a.txt', '.A.txt.datacache.json']) def test_local_identity_is_an_absolute_file_url(tmp_path, monkeypatch): monkeypatch.chdir(tmp_path) assert local_file_identity('data/a.fa') == (tmp_path / 'data' / 'a.fa').as_uri() assert local_file_identity(Path('/data/a b.fa')) == 'file:///data/a%20b.fa' def test_download_options_are_checked_like_fetch_file(): assert validate_download_options(None, 'bundle') == {} options = {'timeout': 5, 'resume': True} assert validate_download_options(options, 'bundle') == options assert validate_download_options(options, 'bundle') is not options with pytest.raises(ValueError, match=r"unsupported archive download options: \['force'\]"): validate_download_options({'force': True}, 'archive') for bad in ({'chunk_size': 0}, {'chunk_size': True}, {'progress_callback': 'print'}, {'show_progress': 1}, {'resume': 'yes'}, {'max_retries': -1}, {'retry_backoff': float('inf')}, {'retry_max_delay': -1}): with pytest.raises(ValueError): validate_download_options(bad, 'bundle') def test_installs_reject_bad_download_options_before_creating_anything(tmp_path): source = tmp_path / 'source.txt' source.write_bytes(b'data') spec = {'url': source.as_uri(), 'sha256': sha256(b'data').hexdigest(), 'size': 4} bad = {'chunk_size': 0} with pytest.raises(ValueError, match='chunk_size'): install_bundle(tmp_path / 'bundle', {'data.txt': spec}, download_options=bad) with pytest.raises(ValueError, match='chunk_size'): install_archive(tmp_path / 'archive', dict(spec), download_options=bad) with pytest.raises(ValueError, match='chunk_size'): materialize(tmp_path / 'derived', {'data.txt': spec}, transform={'version': '1'}, outputs={'out.txt': {}}, builder=lambda inputs, outputs: None, download_options=bad) # Resumable installs keep their staging, so unsuitable sources fail first too. with pytest.raises(ValueError, match='HTTP'): install_bundle(tmp_path / 'bundle', {'data.txt': spec}, download_options={'resume': True}) with pytest.raises(ValueError, match='raw downloads only'): install_bundle(tmp_path / 'bundle', {'data.txt': dict(spec, url='https://example.test/data.gz', decompress=True)}, download_options={'resume': True}) with pytest.raises(ValueError, match='HTTP'): install_archive(tmp_path / 'archive', dict(spec), download_options={'resume': True}) assert sorted(path.name for path in tmp_path.iterdir()) == ['source.txt'] ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792090401.713152 datacache-1.27.1/tests/test_bundle_store_paths.py0000644000076500000000000002044215264220442021657 0ustar00iskanderwheel"""Consumer-selected paths use the ordinary owned bundle transaction.""" from hashlib import sha256 import os from pathlib import Path import pytest from datacache import FileValidationError, VersionedDatasetRegistry from datacache import bundle_store, bundles, download pytestmark = pytest.mark.skipif(os.name != 'posix', reason='POSIX bundle installation') @pytest.fixture def datasets(tmp_path): source = tmp_path / 'source.fa' payload = b'>reference\nACGT\n' source.write_bytes(payload) assets = {'records.fa': dict(url=source.as_uri(), sha256=sha256(payload).hexdigest(), size=len(payload))} return {'reference': dict(default_version='110', versions={'110': assets, '109': assets})} def test_custom_path_resolution_is_pure_and_uses_concrete_versions(tmp_path, datasets, monkeypatch): calls = [] def store_path(name, version): calls.append((name, version)) return str(tmp_path / 'GRCh38' / ('ensembl-' + version) / 'sources' / name) registry = VersionedDatasetRegistry(datasets, store_path=store_path) assert calls == [] def forbidden(*args, **kwargs): raise AssertionError('path resolution inspected or mutated the filesystem') monkeypatch.setattr(bundles, 'path_present', forbidden) monkeypatch.setattr(Path, 'mkdir', forbidden) assert registry.store_path('reference') == tmp_path / 'GRCh38/ensembl-110/sources/reference' # Every version's store is resolved once, on first use, and then reused. assert sorted(calls) == [('reference', '109'), ('reference', '110')] assert registry.store_path('reference', '109') == tmp_path / 'GRCh38/ensembl-109/sources/reference' with pytest.raises(ValueError): registry.store_path('reference', 'latest') assert len(calls) == 2 assert not (tmp_path / 'GRCh38').exists() def test_custom_layout_install_versions_refresh_and_read_only_reuse(tmp_path, datasets, monkeypatch): root = tmp_path / 'application' index = root / 'GRCh38' / 'ensembl-110' / 'index.sqlite' index.parent.mkdir(parents=True) index.write_bytes(b'application-owned derived index') registry = VersionedDatasetRegistry( datasets, store_path=lambda name, version: root / 'GRCh38' / ('ensembl-' + version) / 'sources' / name) assert registry.inspect('reference').status == 'missing' assert not registry.store_path('reference').exists() current = registry.download('reference') older = registry.download('reference', '109') assert current != older assert registry.local_path('reference') == Path(current['records.fa']) assert registry.local_path('reference', '109') == Path(older['records.fa']) refreshed = registry.download('reference', force=True) assert refreshed != current assert Path(current['records.fa']).read_bytes() == Path(refreshed['records.fa']).read_bytes() assert index.read_bytes() == b'application-owned derived index' store = registry.store_path('reference') entries = [store, *store.rglob('*')] modes = {path: path.stat().st_mode & 0o777 for path in entries} def forbidden(*args, **kwargs): raise AssertionError('offline reuse attempted a write or acquisition') monkeypatch.setattr(download, 'fetch_file', forbidden) monkeypatch.setattr(bundle_store.BundleStore, 'lock', forbidden) monkeypatch.setattr(bundles, 'write_json', forbidden) monkeypatch.setattr(bundle_store, 'write_json', forbidden) try: for path in entries: path.chmod(0o555 if path.is_dir() else 0o444) assert registry.download('reference') == refreshed assert registry.local_path('reference') == Path(refreshed['records.fa']) assert registry.is_cached('reference') assert registry.inspect('reference').verified finally: for path, mode in modes.items(): path.chmod(mode) def test_custom_layout_bundle_is_installed_once_renamed(tmp_path, datasets, monkeypatch): registry = VersionedDatasetRegistry( datasets, store_path=lambda name, version: tmp_path / ('sources-' + version) / name) store = registry.store_path('reference') replace = os.replace def interrupted(source, target): replace(source, target) if Path(target).parent == store / 'bundles': raise KeyboardInterrupt('interrupted right after the bundle rename') monkeypatch.setattr(os, 'replace', interrupted) with pytest.raises(KeyboardInterrupt): registry.download('reference') monkeypatch.setattr(os, 'replace', replace) def forbidden(*args, **kwargs): raise AssertionError('an installed bundle was downloaded again') monkeypatch.setattr(download, 'fetch_file', forbidden) paths = registry.download('reference') assert registry.inspect('reference').verified assert registry.local_path('reference') == Path(paths['records.fa']) @pytest.mark.parametrize('force', [False, True]) def test_custom_paths_never_take_over_foreign_directories(tmp_path, datasets, force): foreign = tmp_path / 'foreign' foreign.mkdir() precious = foreign / 'index.sqlite' precious.write_bytes(b'keep me') registry = VersionedDatasetRegistry( datasets, store_path=lambda name, version: foreign if version == '110' else tmp_path / version) with pytest.raises(FileValidationError, match='never taken over'): registry.download('reference', force=force) assert list(foreign.iterdir()) == [precious] assert precious.read_bytes() == b'keep me' def test_custom_paths_reject_symlinked_stores(tmp_path, datasets): foreign = tmp_path / 'foreign' foreign.mkdir() store = tmp_path / 'link' store.symlink_to(foreign, target_is_directory=True) registry = VersionedDatasetRegistry( datasets, store_path=lambda name, version: store if version == '110' else tmp_path / version) assert registry.inspect('reference').status == 'invalid' with pytest.raises(FileValidationError): registry.download('reference', force=True) assert list(foreign.iterdir()) == [] @pytest.mark.parametrize('options', [{}, {'cache_root': '/cache', 'cache_dir': lambda: '/cache'}, {'cache_root': '/cache', 'store_path': lambda name, version: '/store'}, {'cache_dir': lambda: '/cache', 'store_path': lambda name, version: '/store'}, {'cache_root': '/cache', 'cache_dir': lambda: '/cache', 'store_path': lambda name, version: '/store'}]) def test_destination_strategies_are_mutually_exclusive(datasets, options): with pytest.raises(ValueError, match='exactly one'): VersionedDatasetRegistry(datasets, **options) @pytest.mark.parametrize('key', ['cache_dir', 'store_path']) def test_path_callbacks_must_be_callable(datasets, key): with pytest.raises(ValueError, match=key + ' must be callable'): VersionedDatasetRegistry(datasets, **{key: '/not-a-callback'}) def test_existing_root_strategies_remain_compatible(tmp_path, datasets): explicit = VersionedDatasetRegistry(datasets, cache_root=tmp_path / 'root') selected = [tmp_path / 'root'] dynamic = VersionedDatasetRegistry(datasets, cache_dir=lambda: selected[0]) assert explicit.store_path('reference') == dynamic.store_path('reference') selected[0] = tmp_path / 'different-root' assert dynamic.store_path('reference') == tmp_path / 'different-root/reference/110' assert not selected[0].exists() def test_versions_sharing_a_custom_store_are_rejected(tmp_path, datasets): registry = VersionedDatasetRegistry(datasets, store_path=lambda name, version: tmp_path / name) with pytest.raises(ValueError, match='each version needs its own store'): registry.store_path('reference') assert not (tmp_path / 'reference').exists() @pytest.mark.parametrize('result', [None, '', 7, b'/bytes']) def test_custom_store_paths_must_be_paths(datasets, result): registry = VersionedDatasetRegistry(datasets, store_path=lambda name, version: result) with pytest.raises(ValueError, match='not a path'): registry.store_path('reference') def test_custom_store_parent_may_be_a_link(tmp_path, datasets): disk = tmp_path / 'disk' disk.mkdir() (tmp_path / 'data').symlink_to(disk, target_is_directory=True) registry = VersionedDatasetRegistry( datasets, store_path=lambda name, version: tmp_path / 'data' / version / name) paths = registry.download('reference') assert Path(paths['records.fa']).resolve().is_relative_to(disk) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7133017 datacache-1.27.1/tests/test_bundles.py0000644000076500000000000006252415264220442017436 0ustar00iskanderwheel"""Atomic multi-file installs with tiny offline fixtures and failure injection.""" from concurrent.futures import ThreadPoolExecutor from hashlib import sha256 import json import os from pathlib import Path import threading import pytest from datacache import ( FileValidationError, VersionedDatasetRegistry, inspect_bundle, install_bundle, ) from datacache import bundle_store, bundles, download pytestmark = pytest.mark.skipif(os.name != 'posix', reason='POSIX bundle locks') @pytest.fixture def assets(tmp_path): root = tmp_path / 'upstream' root.mkdir() result = {} for name, data in [('records.json', b'{"sequence":"ACGT"}\n'), ('release/manifest.json', b'{"release":"2026-09"}\n')]: path = root / name path.parent.mkdir(exist_ok=True) path.write_bytes(data) result[name] = dict(url=path.as_uri(), size=len(data), sha256=sha256(data).hexdigest()) return result def test_install_inspect_offline_and_generated_outputs_survive(tmp_path, assets, monkeypatch): destination = tmp_path / 'source' / '2026-09' outputs = tmp_path / 'generated' outputs.mkdir() (outputs / 'index.sqlite').write_bytes(b'derived data') paths = install_bundle(destination, assets) inspected = inspect_bundle(destination, assets) assert inspected.status == 'available' and inspected.verified assert set(paths) == set(assets) assert all(Path(path).is_file() for path in paths.values()) assert inspect_bundle(destination).status == 'available' assert not inspect_bundle(destination).verified old = {name: Path(path).read_bytes() for name, path in paths.items()} refreshed = install_bundle(destination, assets, force=True) assert paths != refreshed assert {name: Path(path).read_bytes() for name, path in paths.items()} == old assert (outputs / 'index.sqlite').read_bytes() == b'derived data' def forbidden(*args, **kwargs): raise AssertionError('read-only reuse attempted a mutation or download') monkeypatch.setattr(download, 'fetch_file', forbidden) monkeypatch.setattr(bundle_store.BundleStore, 'lock', forbidden) monkeypatch.setattr(bundles, 'write_json', forbidden) monkeypatch.setattr(bundle_store, 'write_json', forbidden) monkeypatch.setattr(Path, 'mkdir', forbidden) assert install_bundle(destination, assets) == refreshed assert inspect_bundle(destination, assets).verified assert inspect_bundle(destination.parent / 'absent', assets).status == 'missing' @pytest.mark.parametrize('failure', ['last-download', 'hash', 'publication']) def test_failed_refresh_preserves_old_bundle(tmp_path, assets, monkeypatch, failure): destination = tmp_path / 'data' paths = install_bundle(destination, assets) old_bundle = inspect_bundle(destination).bundle replacement = {k: dict(v) for k, v in assets.items()} if failure == 'last-download': replacement['release/manifest.json']['url'] = (tmp_path / 'missing').as_uri() elif failure == 'hash': replacement['release/manifest.json']['sha256'] = '0' * 64 else: replace = os.replace def fail(source, target): if Path(target).parent == destination / 'bundles': raise OSError('publication failed') return replace(source, target) monkeypatch.setattr(os, 'replace', fail) with pytest.raises((OSError, FileValidationError)): install_bundle(destination, replacement, force=True) assert inspect_bundle(destination).bundle == old_bundle assert inspect_bundle(destination, assets).verified assert all(Path(path).exists() for path in paths.values()) assert not list(destination.glob('.staging-*')) def test_concurrent_first_and_forced_installs(tmp_path, assets): destination = tmp_path / 'data' barrier = threading.Barrier(4) def install(_): barrier.wait(timeout=5) return install_bundle(destination, assets) with ThreadPoolExecutor(max_workers=4) as pool: initial = list(pool.map(install, range(4))) assert all(value == initial[0] for value in initial) with ThreadPoolExecutor(max_workers=3) as pool: forced = list(pool.map(lambda _: install_bundle(destination, assets, force=True), range(3))) assert len({tuple(p.values()) for p in forced}) == 3 assert all(Path(path).exists() for paths in forced + initial for path in paths.values()) assert inspect_bundle(destination, assets).verified def test_reader_sees_complete_old_bundle_during_publish(tmp_path, assets, monkeypatch): destination = tmp_path / 'data' old = install_bundle(destination, assets) entered, release = threading.Event(), threading.Event() replace = os.replace def paused(source, target): if Path(target).parent == destination / 'bundles': entered.set() assert release.wait(5) return replace(source, target) monkeypatch.setattr(os, 'replace', paused) with ThreadPoolExecutor() as pool: future = pool.submit(install_bundle, destination, assets, force=True) assert entered.wait(5) try: inspected = inspect_bundle(destination, assets) assert inspected.verified assert {name: value.path for name, value in inspected.files.items()} == old finally: release.set() assert future.result() != old assert all(Path(path).exists() for path in old.values()) @pytest.mark.parametrize('name', ['/absolute', '../escape', 'a/../escape', 'a//b', 'C:/file', 'a\\b', '.datacache-manifest.json', 'CON', 'a/', 'a\x00b']) def test_unsafe_asset_names_rejected_without_creating_root(tmp_path, assets, name): dest = tmp_path / 'missing' / 'bundle' with pytest.raises(ValueError): install_bundle(dest, {name: next(iter(assets.values()))}) assert not dest.parent.exists() @pytest.mark.parametrize('names', [('a', 'a/b'), ('A', 'a'), ('A', 'a/b')]) def test_colliding_asset_paths_are_rejected(tmp_path, assets, names): with pytest.raises(ValueError): install_bundle(tmp_path / 'bundle', {name: next(iter(assets.values())) for name in names}) def test_foreign_directory_is_never_replaced_even_with_force(tmp_path, assets): dest = tmp_path / 'foreign' dest.mkdir() precious = dest / 'precious.txt' precious.write_text('keep me') with pytest.raises((FileValidationError, FileNotFoundError)): install_bundle(dest, assets, force=True) assert precious.read_text() == 'keep me' def test_integrity_mismatch_requires_explicit_repair(tmp_path, assets): dest = tmp_path / 'data' paths = install_bundle(dest, assets) Path(paths['records.json']).write_bytes(b'corrupt') assert inspect_bundle(dest, assets).status == 'invalid' with pytest.raises(FileValidationError, match='force=True'): install_bundle(dest, assets) assert install_bundle(dest, assets, force=True) != paths assert inspect_bundle(dest, assets).verified def test_missing_manifest_and_symlinks_cannot_escape_store(tmp_path, assets): dest = tmp_path / 'data' paths = install_bundle(dest, assets) inspected = inspect_bundle(dest, assets) target = Path(paths['records.json']) target.unlink() target.symlink_to(Path(assets['records.json']['url'][7:])) assert inspect_bundle(dest, assets).status == 'invalid' manifest = Path(inspected.bundle) / bundles.MANIFEST manifest.unlink() assert inspect_bundle(dest).status == 'invalid' # A newest "bundle" that links elsewhere is never followed. (dest / 'bundles' / '2999-01-01T00-00-00Z').symlink_to(tmp_path / 'upstream', target_is_directory=True) assert inspect_bundle(dest).status == 'invalid' def test_permission_error_is_inaccessible_without_writes(tmp_path, assets, monkeypatch): dest = tmp_path / 'data' install_bundle(dest, assets) def forbidden(*args, **kwargs): raise PermissionError('unreadable') monkeypatch.setattr(bundles, 'read_json', forbidden) assert inspect_bundle(dest, assets).status == 'inaccessible' with pytest.raises(PermissionError): install_bundle(dest, assets) def test_registry_versions_are_independent_and_pinned(tmp_path, assets): reg = VersionedDatasetRegistry({'example': dict(default_version='2026-09', versions={'2026-09': assets, '2025-01': {'records.json': assets['records.json']}})}, cache_root=tmp_path / 'cache') assert reg.resolve_version('example') == '2026-09' assert reg.inspect('example').status == 'missing' assert not reg.store_path('example').parent.exists() reg.download('example') assert reg.is_cached('example') assert reg.inspect('example', '2025-01').status == 'missing' assert reg.local_path('example', asset='records.json').read_bytes().startswith(b'{') reg.download('example', '2025-01') assert reg.local_path('example', '2025-01').is_file() assert reg.status()[0]['inspection'].verified assert not reg.status(verify_files=False)[0]['inspection'].verified with pytest.raises(ValueError): reg.resolve_version('example', 'latest') def test_hitlist_registry_shape_and_explicit_unverified_mode(tmp_path, assets): mapping = {'example': {'filename': 'data.json', 'default_version': 'v1', 'urls': {'v1': assets['records.json']['url']}}} with pytest.raises(ValueError, match='sha256'): VersionedDatasetRegistry(mapping, cache_root=tmp_path / 'cache') reg = VersionedDatasetRegistry(mapping, cache_dir=lambda: tmp_path / 'cache', verified=False) path = reg.ensure('example') assert path.is_file() assert reg.inspect('example').status == 'available' assert not reg.inspect('example').verified assert not (tmp_path / 'cache' / 'manifest.json').exists() def test_parent_permission_error_is_not_missing(tmp_path, assets, monkeypatch): dest = tmp_path / 'unreadable' / 'data' original = os.lstat def denied(path, *args, **kwargs): if Path(path) == dest: raise PermissionError('parent not searchable') return original(path, *args, **kwargs) monkeypatch.setattr(os, 'lstat', denied) assert inspect_bundle(dest, assets).status == 'inaccessible' with pytest.raises(PermissionError): install_bundle(dest, assets) def test_separate_processes_converge_on_one_bundle(tmp_path, assets): import subprocess import sys dest = tmp_path / 'data' script = ('import json, sys; from datacache import install_bundle; ' 'print(json.dumps(install_bundle(sys.argv[1], json.loads(sys.argv[2])), sort_keys=True))') processes = [subprocess.Popen([sys.executable, '-c', script, str(dest), json.dumps(assets)], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) for _ in range(3)] results = [process.communicate(timeout=10) for process in processes] assert all(process.returncode == 0 for process in processes), results assert len({stdout for stdout, stderr in results}) == 1 assert len(list((dest / 'bundles').iterdir())) == 1 def test_single_manifest_tampering_does_not_override_trusted_registry(tmp_path, assets): dest = tmp_path / 'data' paths = install_bundle(dest, assets) bundle = Path(inspect_bundle(dest).bundle) Path(paths['records.json']).write_bytes(b'forged') manifest = bundle / bundles.MANIFEST receipt = json.loads(manifest.read_text()) receipt['assets']['records.json'].update(sha256=sha256(b'forged').hexdigest(), size=6) manifest.write_text(json.dumps(receipt)) assert inspect_bundle(dest).status == 'available' assert not inspect_bundle(dest).verified assert inspect_bundle(dest, assets).status == 'invalid' def test_registry_rejects_symlinked_dataset_directory(tmp_path, assets): outside = tmp_path / 'outside' outside.mkdir() root = tmp_path / 'cache' root.mkdir() (root / 'example').symlink_to(outside) reg = VersionedDatasetRegistry({'example': dict(default_version='v1', versions={'v1': assets})}, cache_root=root) with pytest.raises(FileValidationError): reg.download('example') assert list(outside.iterdir()) == [] @pytest.mark.parametrize('changed_url', [ 'https://reader:old-secret@example.test/download?id=new#original', 'https://reader:new-secret@example.test/download?id=old#original', 'https://reader:old-secret@example.test/download?id=old#changed', ]) def test_unverified_reuse_checks_full_source_identity(tmp_path, monkeypatch, changed_url): original_url = 'https://reader:old-secret@example.test/download?id=old#original' bodies = {original_url: b'old data', changed_url: b'new data'} requests = [] def stream(url, handle, **kwargs): requests.append(url) return handle.write(bodies[url]) monkeypatch.setattr(download, '_stream_to_file', stream) dest = tmp_path / 'data' original = {'data': {'url': original_url, 'size': 8}} changed = {'data': {'url': changed_url, 'size': 8}} old_paths = install_bundle(dest, original, verified=False) assert install_bundle(dest, original, verified=False) == old_paths assert requests == [original_url] assert inspect_bundle(dest, changed).status == 'invalid' with pytest.raises(FileValidationError, match='force=True'): install_bundle(dest, changed, verified=False) assert requests == [original_url] new_paths = install_bundle(dest, changed, verified=False, force=True) assert Path(new_paths['data']).read_bytes() == b'new data' assert Path(old_paths['data']).read_bytes() == b'old data' assert requests == [original_url, changed_url] receipt = json.loads((Path(inspect_bundle(dest).bundle) / bundles.MANIFEST).read_text()) assert receipt['source_fingerprints']['data'] == sha256(changed_url.encode()).hexdigest() assert receipt['assets']['data']['url'] == 'https://example.test/download' serialized = json.dumps(receipt) assert 'secret' not in serialized and 'id=' not in serialized and 'reader' not in serialized @pytest.mark.parametrize('decompress', [False, True]) def test_unverified_reuse_checks_requested_decompression(tmp_path, decompress): import gzip payload = b'plain dataset bytes\n' compressed = gzip.compress(payload) source = tmp_path / 'source.gz' source.write_bytes(compressed) dest = tmp_path / 'bundle' original = {'data.gz': dict(url=source.as_uri(), decompress=decompress)} changed = {'data.gz': dict(url=source.as_uri(), decompress=not decompress)} old_paths = install_bundle(dest, original, verified=False) assert Path(old_paths['data.gz']).read_bytes() == (payload if decompress else compressed) assert inspect_bundle(dest, changed).status == 'invalid' with pytest.raises(FileValidationError, match='force=True'): install_bundle(dest, changed, verified=False) new_paths = install_bundle(dest, changed, verified=False, force=True) assert Path(new_paths['data.gz']).read_bytes() == (compressed if decompress else payload) def test_trusted_hashes_allow_cross_library_reuse_across_mirrors(tmp_path, assets, monkeypatch): root = tmp_path / 'shared' mapping = {'reference': dict(default_version='v1', versions={'v1': assets})} first = VersionedDatasetRegistry(mapping, cache_root=root) paths = first.download('reference') mirrored = {name: dict(spec, url='https://mirror.example.test/' + name + '.gz?token=renewed', decompress=True) for name, spec in assets.items()} second = VersionedDatasetRegistry( {'reference': dict(default_version='v1', versions={'v1': mirrored})}, cache_root=root) def forbidden(*args, **kwargs): raise AssertionError('trusted shared reuse must not download or write') monkeypatch.setattr(download, 'fetch_file', forbidden) monkeypatch.setattr(bundle_store.BundleStore, 'lock', forbidden) monkeypatch.setattr(bundles, 'write_json', forbidden) monkeypatch.setattr(bundle_store, 'write_json', forbidden) assert second.download('reference') == paths assert second.inspect('reference').verified def test_first_install_into_empty_directory_is_missing_and_preserves_mode(tmp_path, assets): dest = tmp_path / 'precreated' dest.mkdir(mode=0o700) assert inspect_bundle(dest, assets).status == 'missing' assert list(dest.iterdir()) == [] paths = install_bundle(dest, assets) assert inspect_bundle(dest, assets).verified assert all(Path(path).is_file() for path in paths.values()) assert dest.stat().st_mode & 0o777 == 0o700 def test_empty_symlink_is_not_an_uninitialized_bundle(tmp_path, assets): target = tmp_path / 'outside' target.mkdir() dest = tmp_path / 'symlink' dest.symlink_to(target) assert inspect_bundle(dest, assets).status == 'invalid' with pytest.raises(FileValidationError): install_bundle(dest, assets) assert list(target.iterdir()) == [] @pytest.mark.parametrize('creation_mask', [0o022, 0o002, 0o077]) def test_bundle_is_shared_and_current_once_renamed(tmp_path, assets, monkeypatch, creation_mask): dest = tmp_path / 'bundle' replace = os.replace installed_modes = [] def interrupt_after_rename(source, target): replace(source, target) if Path(target).parent == dest / 'bundles': installed_modes.append(Path(target).stat().st_mode & 0o777) raise KeyboardInterrupt('interrupted immediately after the bundle rename') monkeypatch.setattr(os, 'replace', interrupt_after_rename) previous_mask = os.umask(creation_mask) try: with pytest.raises(KeyboardInterrupt): install_bundle(dest, assets) assert installed_modes == [0o777 & ~creation_mask] monkeypatch.setattr(os, 'replace', replace) def forbidden(*args, **kwargs): raise AssertionError('an installed bundle must not be downloaded again') monkeypatch.setattr(download, 'fetch_file', forbidden) assert inspect_bundle(dest, assets).verified paths = install_bundle(dest, assets) assert Path(inspect_bundle(dest).bundle).stat().st_mode & 0o777 == 0o777 & ~creation_mask assert all(Path(path).is_file() for path in paths.values()) finally: os.umask(previous_mask) @pytest.mark.parametrize('names', [ ('data', '.data.datacache.json'), ('dir/data', 'dir/.data.datacache.json'), ('dir/DATA', 'dir/.data.datacache.JSON'), ('data', '.data.datacache.json/nested'), ]) @pytest.mark.parametrize('reverse', [False, True]) def test_provenance_sidecar_collisions_fail_before_any_writes(tmp_path, assets, monkeypatch, names, reverse): if reverse: names = names[::-1] dest = tmp_path / 'missing' / 'bundle' def forbidden(*args, **kwargs): raise AssertionError('invalid asset mapping must not download or create files') monkeypatch.setattr(download, 'fetch_file', forbidden) monkeypatch.setattr(Path, 'mkdir', forbidden) with pytest.raises(ValueError, match='provenance'): install_bundle(dest, {name: next(iter(assets.values())) for name in names}) assert not dest.parent.exists() def test_existing_bundle_inspection_lists_only_the_bundles_directory(tmp_path, assets, monkeypatch): dest = tmp_path / 'bundle' paths = install_bundle(dest, assets) iterdir = Path.iterdir def only_bundles(path): # Finding the newest bundle lists bundles/; known files need no listing. assert path == dest / 'bundles', 'listed %s' % path return iterdir(path) monkeypatch.setattr(Path, 'iterdir', only_bundles) assert inspect_bundle(dest, assets).verified assert install_bundle(dest, assets) == paths def test_legacy_receipts_still_verify_but_cannot_prove_unpinned_source_identity(tmp_path, assets): dest = tmp_path / 'bundle' paths = install_bundle(dest, assets) manifest = Path(inspect_bundle(dest).bundle) / bundles.MANIFEST receipt = json.loads(manifest.read_text()) del receipt['source_fingerprints'] manifest.write_text(json.dumps(receipt)) assert inspect_bundle(dest).status == 'available' assert install_bundle(dest, assets) == paths unpinned = {name: {'url': spec['url']} for name, spec in assets.items()} assert inspect_bundle(dest, unpinned).status == 'invalid' with pytest.raises(FileValidationError, match='force=True'): install_bundle(dest, unpinned, verified=False) refreshed = install_bundle(dest, unpinned, verified=False, force=True) assert refreshed != paths assert inspect_bundle(dest, unpinned).status == 'available' def test_sidecar_like_asset_without_collision_remains_supported(tmp_path, assets): mapping = {'.data.datacache.json': assets['records.json'], 'other/data': assets['release/manifest.json']} dest = tmp_path / 'bundle' paths = install_bundle(dest, mapping) assert set(paths) == set(mapping) assert inspect_bundle(dest, mapping).verified @pytest.mark.parametrize('force', [False, True]) def test_hidden_foreign_files_are_never_treated_as_an_empty_destination(tmp_path, assets, force): dest = tmp_path / 'bundle' dest.mkdir() precious = dest / '.user-owned' precious.write_text('keep me') assert inspect_bundle(dest, assets).status == 'invalid' with pytest.raises((FileNotFoundError, FileValidationError)): install_bundle(dest, assets, force=force) assert list(dest.iterdir()) == [precious] assert precious.read_text() == 'keep me' def test_concurrent_initialization_of_precreated_directory_reuses_winner(tmp_path, assets, monkeypatch): dest = tmp_path / 'bundle' dest.mkdir() missing_marker, resume_inspection = threading.Event(), threading.Event() read_json = bundle_store.read_json def pause_after_missing_marker(path, *args, **kwargs): try: return read_json(path, *args, **kwargs) except FileNotFoundError: if Path(path) == dest / bundle_store.MARKER and not missing_marker.is_set(): missing_marker.set() assert resume_inspection.wait(5) raise monkeypatch.setattr(bundle_store, 'read_json', pause_after_missing_marker) with ThreadPoolExecutor(max_workers=1) as pool: future = pool.submit(install_bundle, dest, assets) assert missing_marker.wait(5) try: winner = install_bundle(dest, assets) finally: resume_inspection.set() assert future.result() == winner assert len(list((dest / 'bundles').iterdir())) == 1 def test_large_manifests_are_read(tmp_path, assets): dest = tmp_path / 'bundle' paths = install_bundle(dest, assets) manifest = Path(inspect_bundle(dest).bundle) / bundles.MANIFEST receipt = json.loads(manifest.read_text()) # Manifests grow with the number of assets; this one is over 1 MiB. receipt['assets']['records.json']['url'] += '/' + 'x' * (2 ** 21) manifest.write_text(json.dumps(receipt)) assert inspect_bundle(dest).status == 'available' assert all(Path(path).is_file() for path in paths.values()) def test_files_are_verified_only_by_hashes_the_caller_supplied(tmp_path, assets): dest = tmp_path / 'bundle' install_bundle(dest, assets) assert all(item.verified for item in inspect_bundle(dest, assets).files.values()) # The bundle's own manifest checks the bytes but vouches for nothing. receipt_only = inspect_bundle(dest) assert receipt_only.status == 'available' assert not any(item.verified for item in receipt_only.files.values()) def test_cleanup_failure_after_publishing_is_logged_not_raised(tmp_path, assets, monkeypatch, caplog): import shutil rmtree = shutil.rmtree def failing_rmtree(path, *args, **kwargs): if Path(path).name.startswith('.staging-'): raise OSError('busy') return rmtree(path, *args, **kwargs) monkeypatch.setattr(shutil, 'rmtree', failing_rmtree) paths = install_bundle(tmp_path / 'bundle', assets) assert all(Path(path).is_file() for path in paths.values()) assert 'Could not remove the working directory' in caplog.text def test_force_never_hashes_the_bundle_it_replaces(tmp_path, assets, monkeypatch): dest = tmp_path / 'bundle' old = install_bundle(dest, assets) hash_file = bundles.hash_file def unhashed_old_bundle(path, *args, **kwargs): if str(path) in old.values() and kwargs.get('hash_contents', True): raise AssertionError('hashed the bundle being replaced') return hash_file(path, *args, **kwargs) monkeypatch.setattr(bundles, 'hash_file', unhashed_old_bundle) assert install_bundle(dest, assets, force=True) != old @pytest.mark.parametrize('datasets', [ {'reference': dict(default_version='GRCh38', versions={'GRCh38': 'A', 'grch38': 'A'})}, {'Reference': dict(default_version='v1', versions={'v1': 'A'}), 'reference': dict(default_version='v1', versions={'v1': 'A'})}, ]) def test_registry_names_that_differ_only_by_case_are_rejected(tmp_path, assets, datasets): datasets = {name: dict(spec, versions={version: assets for version in spec['versions']}) for name, spec in datasets.items()} with pytest.raises(ValueError, match='collide'): VersionedDatasetRegistry(datasets, cache_root=tmp_path) def test_the_newest_bundle_is_current(tmp_path, assets): dest = tmp_path / 'bundle' first = install_bundle(dest, assets) second = install_bundle(dest, assets, force=True) names = sorted(entry.name for entry in (dest / 'bundles').iterdir()) assert len(names) == 2 assert Path(inspect_bundle(dest).bundle).name == names[-1] assert install_bundle(dest, assets) == second != first assert all(Path(path).is_file() for path in first.values()) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7134044 datacache-1.27.1/tests/test_byte_limits.py0000644000076500000000000002670315264220442020325 0ustar00iskanderwheel"""max_bytes: no file a fetch writes ever holds more than the caller allows (#108).""" import gzip from hashlib import sha256 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import io import os from pathlib import Path import threading import zipfile import pytest from datacache import Cache, FileValidationError, fetch_bytes, fetch_file, install_bundle from datacache import download from datacache.download import validate_download_options BODY = bytes(range(256)) * 256 # 64 KiB that doesn't compress to nothing. @pytest.fixture(autouse=True) def local_requests(monkeypatch): monkeypatch.setenv("NO_PROXY", "127.0.0.1,localhost") monkeypatch.setattr(download.time, "sleep", lambda seconds: None) @pytest.fixture def serve(): """serve(body, how) -> (url, request paths); how is how the length is sent.""" servers = [] def start(body=BODY, how="length", path="/data"): requests = [] class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def log_message(self, *args): pass def do_GET(self): requests.append(self.path) self.send_response(200) self.send_header("Connection", "close") payload = body if how == "gzip": payload = gzip.compress(body, mtime=0) self.send_header("Content-Encoding", "gzip") if how in ("length", "gzip"): self.send_header("Content-Length", str(len(payload))) if how == "overstated": # Chunked transfer overrides any Content-Length (RFC 9112). self.send_header("Content-Length", str(100 * len(payload))) if how in ("chunked", "overstated"): self.send_header("Transfer-Encoding", "chunked") self.end_headers() if how in ("chunked", "overstated"): for start in range(0, len(payload), 4096): piece = payload[start:start + 4096] self.wfile.write(b"%x\r\n%s\r\n" % (len(piece), piece)) self.wfile.write(b"0\r\n\r\n") else: self.wfile.write(payload) self.wfile.flush() self.close_connection = True server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) worker = threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True) worker.start() servers.append((server, worker)) return "http://127.0.0.1:%d%s" % (server.server_port, path), requests yield start for server, worker in servers: server.shutdown() server.server_close() worker.join(timeout=5) def leftovers(directory, keep): return sorted(path.name for path in Path(directory).iterdir() if path.name not in keep) def test_a_content_length_over_the_limit_is_refused_before_reading(serve, tmp_path): url, requests = serve(how="length") destination = tmp_path / "data" written = [] with pytest.raises(FileValidationError, match="max_bytes=1000"): fetch_file(url, destination=destination, raw=True, max_bytes=1000, progress_callback=lambda done, total: written.append(done)) assert written == [] # Nothing was written. assert requests == ["/data"] # Not retried. assert leftovers(tmp_path, keep=()) == [] @pytest.mark.parametrize("how", ["none", "chunked", "gzip"]) def test_bodies_of_unknown_length_never_write_past_the_limit(serve, tmp_path, how): # With no Content-Length, or one that counts gzip-encoded bytes, the # limit has to be enforced chunk by chunk as the decoded body arrives. url, requests = serve(how=how) destination = tmp_path / "data" destination.write_bytes(b"the previous file") written = [] with pytest.raises(FileValidationError, match="max_bytes=10000"): fetch_file(url, destination=destination, raw=True, force=True, max_bytes=10000, chunk_size=1024, progress_callback=lambda done, total: written.append(done)) assert written and max(written) <= 10000 assert requests == ["/data"] assert destination.read_bytes() == b"the previous file" assert leftovers(tmp_path, keep=("data",)) == [] @pytest.mark.parametrize("how", ["length", "none", "chunked", "gzip", "overstated"]) def test_a_body_of_exactly_max_bytes_is_accepted(serve, tmp_path, how): url, _ = serve(how=how) destination = tmp_path / "data" fetch_file(url, destination=destination, raw=True, max_bytes=len(BODY), expected_size=len(BODY)) assert destination.read_bytes() == BODY def test_cache_hits_are_unaffected(serve, tmp_path): url, requests = serve() destination = tmp_path / "data" destination.write_bytes(BODY) assert fetch_file(url, destination=destination, raw=True, max_bytes=1) == str(destination) assert requests == [] def test_decompressed_files_are_limited_too(serve, tmp_path): # 64 KiB of zeros compresses to well under the limit, then expands past it. zeros = bytes(len(BODY)) url, _ = serve(gzip.compress(zeros, mtime=0), path="/data.gz") destination = tmp_path / "data.txt" destination.write_bytes(b"the previous file") with pytest.raises(FileValidationError, match="decompressed file has more than max_bytes=10000"): fetch_file(url, destination=destination, decompress=True, force=True, max_bytes=10000) assert destination.read_bytes() == b"the previous file" assert leftovers(tmp_path, keep=("data.txt",)) == [] archive = io.BytesIO() with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zipped: zipped.writestr("data.txt", zeros) url, _ = serve(archive.getvalue(), path="/data.zip") with pytest.raises(FileValidationError, match="decompressed file has more than max_bytes=10000"): fetch_file(url, destination=destination, decompress=True, force=True, max_bytes=10000) assert destination.read_bytes() == b"the previous file" fetch_file(url, destination=destination, decompress=True, force=True, max_bytes=len(zeros)) assert destination.read_bytes() == zeros def test_local_files_over_the_limit_are_refused(tmp_path): source = tmp_path / "source" source.write_bytes(BODY) with pytest.raises(FileValidationError, match="max_bytes"): fetch_file(source.as_uri(), destination=tmp_path / "copy", raw=True, max_bytes=100) assert leftovers(tmp_path, keep=("source",)) == [] @pytest.mark.parametrize("how", ["length", "chunked", "gzip"]) def test_fetch_bytes_keeps_no_more_than_max_bytes(serve, how): url, requests = serve(how=how) with pytest.raises(ValueError, match="more than max_bytes=100"): fetch_bytes(url, max_bytes=100) assert requests == ["/data"] assert fetch_bytes(url, max_bytes=len(BODY)) == BODY @pytest.mark.parametrize("options, message", [ (dict(max_bytes=-1), "non-negative"), (dict(max_bytes=True), "non-negative"), (dict(max_bytes=1.5), "non-negative"), (dict(max_bytes=10, expected_size=11), "larger than max_bytes"), ]) def test_invalid_limits_are_rejected_before_any_request(serve, tmp_path, options, message): url, requests = serve() with pytest.raises(ValueError, match=message): fetch_file(url, destination=tmp_path / "data", raw=True, **options) assert requests == [] def test_html_conversion_cant_be_bounded(serve, tmp_path): url, requests = serve(path="/table.html") with pytest.raises(ValueError, match="HTML-to-CSV"): fetch_file(url, destination=tmp_path / "table.csv", max_bytes=10) # The converter guards itself too, for direct callers of the old entry point. with pytest.raises(ValueError, match="HTML-to-CSV"): download._download_and_decompress_if_necessary(str(tmp_path / "table.csv"), url, max_bytes=10) assert requests == [] def test_a_zero_limit_needs_allow_empty(serve, tmp_path): url, requests = serve(b"", how="length") with pytest.raises(ValueError, match="allow_empty"): fetch_file(url, destination=tmp_path / "empty", raw=True, max_bytes=0) with pytest.raises(ValueError, match="allow_empty"): fetch_bytes(url, max_bytes=0) with pytest.raises(ValueError, match="empty"): validate_download_options({"max_bytes": 0}, "bundle") assert requests == [] fetch_file(url, destination=tmp_path / "empty", raw=True, max_bytes=0, allow_empty=True) assert (tmp_path / "empty").read_bytes() == b"" assert fetch_bytes(url, max_bytes=0, allow_empty=True) == b"" def test_installs_check_known_sizes_before_creating_anything(serve, tmp_path): from datacache import install_archive, materialize url, requests = serve() small = {"url": url, "sha256": "0" * 64, "size": 10} large = {"url": url, "sha256": "0" * 64, "size": 10 ** 9} options = {"max_bytes": 10000} if os.name == "posix": with pytest.raises(ValueError, match="b size 1000000000 is larger than max_bytes 10000"): install_bundle(tmp_path / "bundle", {"a": small, "b": large}, download_options=options) with pytest.raises(ValueError, match="b size 1000000000 is larger than max_bytes 10000"): materialize(tmp_path / "derived", {"a": small, "b": large}, transform={"version": "1"}, outputs={"out": {}}, builder=lambda inputs, outputs: None, download_options=options) with pytest.raises(ValueError, match="part 1 size 1000000000 is larger than max_bytes 10000"): install_archive(tmp_path / "archive", [small, large], download_options=options) assert requests == [] assert sorted(path.name for path in tmp_path.iterdir()) == [] def test_cache_fetch_and_install_download_options_take_max_bytes(serve, tmp_path): url, _ = serve() with pytest.raises(FileValidationError, match="max_bytes=100"): Cache(cache_root=tmp_path / "cache").fetch(url, filename="data", raw=True, max_bytes=100) assert validate_download_options({"max_bytes": 100}, "bundle") == {"max_bytes": 100} with pytest.raises(ValueError, match="max_bytes"): validate_download_options({"max_bytes": -1}, "bundle") if os.name != "posix": return # Bundle installation is POSIX-only (#89). store = tmp_path / "bundle" with pytest.raises(FileValidationError, match="max_bytes=100"): install_bundle(store, {"data": {"url": url}}, verified=False, download_options={"max_bytes": 100}) assert not list((store / "bundles").iterdir()) @pytest.mark.skipif(os.name != "posix", reason="resume=True requires POSIX") def test_resumable_downloads_check_their_size_against_the_limit(serve, tmp_path): url, _ = serve() with pytest.raises(ValueError, match="larger than max_bytes"): fetch_file(url, destination=tmp_path / "data", raw=True, resume=True, expected_size=len(BODY), max_bytes=len(BODY) - 1) path = fetch_file(url, destination=tmp_path / "data", raw=True, resume=True, expected_sha256=sha256(BODY).hexdigest(), expected_size=len(BODY), max_bytes=len(BODY)) assert Path(path).read_bytes() == BODY @pytest.mark.parametrize("how", ["length", "chunked"]) def test_auto_resume_works_everywhere(serve, tmp_path, how): # Runs on Windows CI too: there "auto" downloads normally, and on POSIX # this server's missing ETag makes it fall back to a full download. url, _ = serve(how=how) path = fetch_file(url, destination=tmp_path / "data", raw=True, expected_size=len(BODY), resume="auto") assert Path(path).read_bytes() == BODY ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7134945 datacache-1.27.1/tests/test_cache_inspection.py0000644000076500000000000002564315264220442021301 0ustar00iskanderwheel"""Inspection must work offline without even attempting a filesystem write.""" import builtins from contextlib import contextmanager import hashlib import io import os from pathlib import Path import pandas as pd import pytest from datacache import ( Cache, FileValidationError, common, download, expected_path, fetch_file, file_exists, inspect_file, inspect_files, resolve_path, ) DATA = b"id,value\n1,example\n" DIGEST = hashlib.sha256(DATA).hexdigest() METADATA = {"expected_sha256": DIGEST, "expected_size": len(DATA)} @contextmanager def read_only(monkeypatch): """Reject writes explicitly, even when chmod would not constrain root.""" original_open = builtins.open original_io_open = io.open original_os_open = os.open def reject(*args, **kwargs): pytest.fail("inspection attempted a write or network operation") def guard_opener(original): def guarded(path, mode="r", *args, **kwargs): if any(flag in mode for flag in "wax+"): reject() return original(path, mode, *args, **kwargs) return guarded def guarded_os_open(path, flags, *args, **kwargs): if flags & (os.O_WRONLY | os.O_RDWR | os.O_CREAT | os.O_TRUNC | os.O_APPEND): reject() return original_os_open(path, flags, *args, **kwargs) with monkeypatch.context() as patch: patch.setattr(builtins, "open", guard_opener(original_open)) patch.setattr(io, "open", guard_opener(original_io_open)) patch.setattr(os, "open", guarded_os_open) for operation in ("mkdir", "makedirs", "remove", "unlink", "rename", "replace", "chmod", "rmdir"): patch.setattr(os, operation, reject) patch.setattr(download, "_stream_to_file", reject) patch.setattr(download.requests, "get", reject) patch.setattr(download.urllib.request, "urlopen", reject) yield def test_absent_paths_do_not_create_cache_or_lock(tmp_path, monkeypatch): root = tmp_path / "absent" / "v2" monkeypatch.setattr(common, "get_data_dir", lambda subdir=None: str(root)) with read_only(monkeypatch): for cache in (Cache(), Cache(cache_root=root)): assert cache.local_path(filename="records") == str(root / "records") assert not cache.exists(filename="records") assert cache.inspect(filename="records", **METADATA).status == "missing" assert expected_path(filename="records") == str(root / "records") assert not file_exists(filename="records") assert not file_exists(destination=root / "records") assert not file_exists(filename="records", cache_root=root) assert inspect_files(root, {"records": METADATA}).status == "missing" assert list(tmp_path.iterdir()) == [] def test_resolution_does_not_access_filesystem(tmp_path, monkeypatch): def reject(*args, **kwargs): pytest.fail("path resolution inspected the filesystem") with monkeypatch.context() as patch: patch.setattr(os, "stat", reject) patch.setattr(os, "lstat", reject) assert resolve_path("file", cache_root=tmp_path) == str(tmp_path / "file") assert expected_path(filename="file", cache_root=tmp_path) == str(tmp_path / "file") assert expected_path(destination=tmp_path / "file") == str(tmp_path / "file") assert Cache(cache_root=tmp_path).local_path(filename="file") == str(tmp_path / "file") @pytest.mark.parametrize("kw", [{"filename": "file"}, {"subdir": "app"}, {"cache_root": "cache"}]) def test_destination_conflicts_rejected_without_writes(tmp_path, monkeypatch, kw): with read_only(monkeypatch): with pytest.raises(ValueError, match="cannot be combined"): expected_path(destination=tmp_path / "file", **kw) with pytest.raises(ValueError, match="cannot be combined"): fetch_file("https://host/file", destination=tmp_path / "file", **kw) def test_read_only_installed_version_and_missing_sibling(tmp_path, monkeypatch): installed = tmp_path / "v1" installed.mkdir() records = installed / "records" records.write_bytes(DATA) records.chmod(0o444) installed.chmod(0o555) tmp_path.chmod(0o555) try: with read_only(monkeypatch): cache = Cache(cache_root=installed) assert cache.exists(filename="records") result = cache.inspect(filename="records", **METADATA) assert (result.status, result.verified, result.error) == ("available", True, None) assert inspect_files(installed, {"records": METADATA}).verified missing = inspect_files(tmp_path / "v2", {"records": METADATA}) assert (missing.status, missing.verified) == ("missing", False) # Ordinary fetch reuses the valid file without needing write access. assert cache.fetch("https://host/records", filename="records", **METADATA) == str(records) assert cache.local_path("https://host/records", filename="records", download=True) == str(records) finally: tmp_path.chmod(0o755) installed.chmod(0o755) @pytest.mark.parametrize("condition", ["valid", "truncated", "missing-manifest", "unreadable"]) def test_required_file_inventory(tmp_path, monkeypatch, condition): records = tmp_path / "records" manifest = tmp_path / "manifest.json" records.write_bytes(DATA[:3] if condition == "truncated" else DATA) manifest.write_bytes(b"{}") if condition == "missing-manifest": manifest.unlink() elif condition == "unreadable": original_open = builtins.open def unreadable(path, *args, **kwargs): if os.fspath(path) == str(records): raise PermissionError("denied by fixture") return original_open(path, *args, **kwargs) monkeypatch.setattr(builtins, "open", unreadable) required = { "records": METADATA, "manifest.json": {"expected_sha256": hashlib.sha256(b"{}").hexdigest()}, } with read_only(monkeypatch): result = inspect_files(tmp_path, required) assert result.status == { "valid": "available", "truncated": "corrupt", "missing-manifest": "corrupt", "unreadable": "inaccessible", }[condition] assert result.verified == (condition == "valid") if condition == "missing-manifest": assert result.files["manifest.json"].status == "missing" assert result.files["records"].verified elif condition == "unreadable": assert isinstance(result.error, PermissionError) assert file_exists(destination=records) # Presence does not imply readability. assert set(p.name for p in tmp_path.iterdir()) == ( {"records"} if condition == "missing-manifest" else {"records", "manifest.json"}) def test_presence_readability_and_verified_integrity_are_distinct(tmp_path, monkeypatch): path = tmp_path / "file" path.write_bytes(DATA) with read_only(monkeypatch): assert file_exists(destination=tmp_path) # Directories are present. assert inspect_file(tmp_path).status == "corrupt" assert inspect_files(path, {"records": METADATA}).status == "corrupt" assert inspect_file(path).status == "available" assert not inspect_file(path).verified assert not inspect_file(path, expected_size=len(DATA)).verified assert not inspect_files(tmp_path, {"file": None}).verified assert inspect_file(path, expected_sha256=DIGEST.upper()).verified result = inspect_file(path, expected_sha256="0" * 64) assert result.status == "corrupt" assert isinstance(result.error, FileValidationError) def test_permission_errors_are_never_missing(tmp_path, monkeypatch): def denied(*args, **kwargs): raise PermissionError("denied by fixture") with read_only(monkeypatch), monkeypatch.context() as patch: patch.setattr(os, "stat", denied) with pytest.raises(PermissionError): file_exists(destination=tmp_path / "records") with pytest.raises(PermissionError): Cache(cache_root=tmp_path).exists(filename="records") assert inspect_file(tmp_path / "records").status == "inaccessible" assert inspect_files(tmp_path, {"records": METADATA}).status == "inaccessible" @pytest.mark.parametrize("metadata", [ {"expected_sha256": "invalid"}, {"expected_size": -1}, {"expected_size": True}, ]) def test_invalid_expectations_raise_even_for_absent_cache(tmp_path, monkeypatch, metadata): with read_only(monkeypatch): with pytest.raises(ValueError): inspect_file(tmp_path / "missing", **metadata) with pytest.raises(ValueError): inspect_files(tmp_path / "missing", {"records": metadata}) @pytest.mark.parametrize("name", ["", ".", "..", "../outside", "a/../b", "/absolute", "C:/file", "a\\b", "a//b", "./a"]) def test_inventory_rejects_ambiguous_asset_paths(tmp_path, monkeypatch, name): with read_only(monkeypatch): with pytest.raises(ValueError, match="normalized relative"): inspect_files(tmp_path / "missing", {name: METADATA}) def test_explicit_root_is_used_for_fetch_inspection_and_deletion(tmp_path, monkeypatch): root = tmp_path / "custom" default_root = tmp_path / "unused-default" monkeypatch.setattr(common, "get_data_dir", lambda subdir=None: str(default_root)) source = tmp_path / "source" source.write_bytes(DATA) cache = Cache("app", cache_root=root) url = source.as_uri() for filename in ("one", "two"): path = cache.fetch(url, filename=filename, **METADATA) assert path == str(root / filename) assert cache.local_path(url, filename=filename) == path assert cache.inspect(filename=filename, **METADATA).verified assert expected_path(url, filename=filename, subdir="app", cache_root=root) == path assert fetch_file(url, filename=filename, subdir="app", cache_root=root, **METADATA) == path cache.delete_url(url) assert list(root.iterdir()) == [] cache.delete_url(url) # Already absent is harmless. connection = cache.db_from_dataframe("table.db", "records", pd.DataFrame({"id": ["example"]})) try: assert connection.execute("select id from records").fetchall() == [("example",)] assert (root / "table.db").is_file() finally: connection.close() cache.delete_all() assert list(root.iterdir()) == [] assert not default_root.exists() def test_cache_fetch_rechecks_integrity_on_every_use(tmp_path, monkeypatch): source = tmp_path / "source" source.write_bytes(DATA) cache = Cache(cache_root=tmp_path / "cache") path = Path(cache.fetch(source.as_uri(), filename="records", **METADATA)) path.write_bytes(b"truncated") with read_only(monkeypatch): with pytest.raises(FileValidationError): cache.fetch(source.as_uri(), filename="records", **METADATA) assert path.read_bytes() == b"truncated" cache.fetch(source.as_uri(), filename="records", force=True, **METADATA) assert path.read_bytes() == DATA ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7135832 datacache-1.27.1/tests/test_cache_object.py0000644000076500000000000000517715264220442020374 0ustar00iskanderwheelfrom os import remove from os.path import exists from unittest.mock import patch import pytest from datacache import Cache, common CACHE_DIR = "datacache_test" TEST_URL = "http://www.google.com" TEST_FILENAME = "google" @pytest.fixture(autouse=True) def offline_cache(tmp_path, monkeypatch): source = tmp_path / "source" source.write_text("local cache fixture") monkeypatch.setitem(globals(), "TEST_URL", source.as_uri()) monkeypatch.setattr( common, "get_data_dir", lambda subdir=None, envkey=None: str(tmp_path / (subdir or "datacache"))) def eq_(x, y): assert x == y def test_cache_object_path(): cache = Cache(CACHE_DIR) assert cache.cache_directory_path.endswith(CACHE_DIR), \ "Expected directory path to end with %s but got %s" % ( CACHE_DIR, cache.cache_directory_path) def test_cache_object_local_filename(): filename = Cache(CACHE_DIR).local_filename(filename="test") assert filename.endswith("test") def test_cache_fetch_google(): cache = Cache(CACHE_DIR) path = cache.fetch(TEST_URL, filename=TEST_FILENAME) assert path.endswith(TEST_FILENAME), \ "Expected local file to be named %s but got %s" % ( TEST_FILENAME, path) assert exists(path), "File not found: %s" % path eq_(path, cache.local_path(TEST_URL, filename=TEST_FILENAME)) @patch('datacache.cache.download._download_and_decompress_if_necessary') def test_cache_fetch_force(mock_download): cache = Cache("datacache_test") cache.fetch("http://www.google.com", filename="google", force=True) cache.fetch("http://www.google.com", filename="google", force=True) assert len(mock_download.call_args_list) == 2, \ "Expected two separate calls to _download, given force=True" def test_cache_delete_url(): cache = Cache(CACHE_DIR) path = cache.fetch(TEST_URL, filename=TEST_FILENAME) assert exists(path), "Expected %s to exist after download" % path cache.delete_url(TEST_URL) assert not exists(path), \ "Expected %s to be deleted after call to delete_url" % path def test_cache_missing_file(): """test_cache_missing_file : Files can be deleted from the file system, Cache should be aware that these files no longer exist """ cache = Cache(CACHE_DIR) path = cache.fetch(TEST_URL, filename=TEST_FILENAME) # does the filename exist? assert exists(path) # does the cache know the URL has been downloaded? assert cache.exists(TEST_URL, filename=TEST_FILENAME) remove(path) assert not cache.exists(TEST_URL, filename=TEST_FILENAME), \ "Local file for %s has been deleted from the file system" % TEST_URL ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7144938 datacache-1.27.1/tests/test_cache_paths.py0000644000076500000000000002706315264220442020243 0ustar00iskanderwheel"""Cache mutations must respect the filesystem meaning of selected paths.""" import os from pathlib import Path import stat import sys import pandas as pd import platformdirs import platformdirs.macos import platformdirs.unix import platformdirs.windows import pytest from datacache import Cache, common, ensure_dir, get_cache_root, get_data_dir @pytest.mark.parametrize("root_kind", ["dot", "path-dot", "dot-slash", "absolute", "parent", "symlink"]) def test_delete_all_preserves_current_directory(tmp_path, monkeypatch, root_kind): directory = tmp_path / "cache" directory.mkdir(mode=0o750) outside = tmp_path / "outside" outside.mkdir() source = outside / "source" source.write_text("keep outside data") alias = tmp_path / "alias" alias.symlink_to(directory, target_is_directory=True) monkeypatch.chdir(directory) root = { "dot": ".", "path-dot": Path("."), "dot-slash": "./", "absolute": directory, "parent": "nested/..", "symlink": alias, }[root_kind] (directory / "nested").mkdir() (directory / "nested" / "records").write_text("cached data") (directory / ".hidden").write_text("cached metadata") (directory / "linked-directory").symlink_to(outside, target_is_directory=True) (directory / "linked-file").symlink_to(source) (directory / "dangling").symlink_to(outside / "missing") cache = Cache(cache_root=root) cache.fetch(source.as_uri(), filename="downloaded") previous = directory.stat() cache.delete_all() assert list(directory.iterdir()) == [] assert os.path.samefile(".", directory) assert directory.stat().st_ino == previous.st_ino assert stat.S_IMODE(directory.stat().st_mode) == stat.S_IMODE(previous.st_mode) assert source.read_text() == "keep outside data" assert alias.is_symlink() # A path through nested/.. becomes absent when nested itself is cleared. if root_kind != "parent": cache.delete_all() assert Path(cache.fetch(source.as_uri(), filename="downloaded")).read_text() == "keep outside data" def test_delete_all_rejects_missing_parent_before_resolving_dotdot(tmp_path): sentinel = tmp_path / "keep" sentinel.write_text("not a cache") with pytest.raises(FileNotFoundError): Cache(cache_root=tmp_path / "missing" / "..").delete_all() assert sentinel.read_text() == "not a cache" @pytest.mark.parametrize("location", ["root", "filename", "absolute-filename"]) @pytest.mark.parametrize("absolute_root", [False, True]) def test_database_paths_follow_symlinks_before_parent_components( tmp_path, monkeypatch, location, absolute_root): monkeypatch.chdir(tmp_path) cache_directory = tmp_path / "cache" cache_directory.mkdir() target = tmp_path / "actual" / "child" target.mkdir(parents=True) (cache_directory / "link").symlink_to(target, target_is_directory=True) root = cache_directory if absolute_root else Path("cache") filename = "records.db" if location == "root": root = root / "link" / ".." elif location == "filename": filename = "link/../records.db" else: filename = str(cache_directory / "link" / ".." / "records.db") cache = Cache(cache_root=root) expected = target.parent / "records.db" wrong = cache_directory / "records.db" connection = cache.db_from_dataframe(filename, "records", pd.DataFrame({"value": ["correct location"]})) try: assert connection.execute("select value from records").fetchall() == [("correct location",)] assert expected.is_file() assert not wrong.exists() finally: connection.close() if location == "root": assert cache.inspect(filename="records.db").status == "available" cache.delete_all() assert not expected.exists() assert not wrong.exists() def test_ensure_dir_tolerates_a_concurrent_creator(tmp_path, monkeypatch): target = tmp_path / "made-elsewhere" target.mkdir() # Another process creates the directory between the check and makedirs. monkeypatch.setattr(common, "exists", lambda path: False) ensure_dir(target) assert target.is_dir() def test_ensure_dir_leaves_existing_paths_but_rejects_a_broken_symlink(tmp_path): existing = tmp_path / "file.txt" existing.write_text("kept") ensure_dir(existing) assert existing.read_text() == "kept" broken = tmp_path / "broken" broken.symlink_to(tmp_path / "missing") with pytest.raises(FileExistsError): ensure_dir(broken) def test_cache_root_uses_the_first_set_variable_as_the_root(tmp_path, monkeypatch): monkeypatch.setenv("SHARED_ROOT", str(tmp_path / "shared")) monkeypatch.setenv("OWN_ROOT", " ") # blank counts as unset root = get_cache_root("openvax", "OWN_ROOT", "SHARED_ROOT") # The value is the root itself; get_data_dir would append "openvax". assert root == str(tmp_path / "shared") assert get_data_dir("openvax", envkey="SHARED_ROOT") == str(tmp_path / "shared" / "openvax") monkeypatch.setenv("OWN_ROOT", " ~/own-root ") assert get_cache_root("openvax", "OWN_ROOT", "SHARED_ROOT") == os.path.expanduser("~/own-root") assert not (tmp_path / "shared").exists() def test_cache_root_falls_back_to_the_platform_directory(monkeypatch): monkeypatch.delenv("UNSET_ROOT", raising=False) assert get_cache_root("openvax", "UNSET_ROOT") == get_data_dir("openvax") assert get_cache_root("openvax") == get_data_dir("openvax") @pytest.fixture def platform_home(tmp_path, monkeypatch): """Simulate a platform through platformdirs, with tmp_path as the home directory.""" for home in ("HOME", "USERPROFILE"): # expanduser reads USERPROFILE on Windows. monkeypatch.setenv(home, str(tmp_path)) monkeypatch.delenv("XDG_CACHE_HOME", raising=False) def simulate(kind): monkeypatch.setattr(platformdirs, "PlatformDirs", kind) monkeypatch.setattr(platformdirs.windows, "get_win_folder", lambda csidl: str(tmp_path / "Local")) return simulate @pytest.mark.skipif(os.name != "posix", reason="POSIX path separators") @pytest.mark.parametrize("xdg", [None, "/elsewhere", "", "relative/cache"]) def test_macos_caches_stay_in_library_caches(platform_home, tmp_path, monkeypatch, xdg): # appdirs ignored XDG_CACHE_HOME on macOS; platformdirs 4.6+ honours it. platform_home(platformdirs.macos.MacOS) if xdg is not None: monkeypatch.setenv("XDG_CACHE_HOME", xdg) expected = str(tmp_path / "Library" / "Caches" / "example") assert get_data_dir("example") == expected assert get_cache_root("example") == expected @pytest.mark.skipif(os.name != "posix", reason="POSIX path separators") @pytest.mark.parametrize("xdg, under", [ (None, ".cache"), ("", ".cache"), ("relative/cache", ".cache"), # Never relative to the working directory. ("~/cache", ".cache"), # Not expanded: relative, as XDG specifies. ("{tmp}/xdg", "xdg"), (" {tmp}/xdg \t", "xdg"), # Stripped, which platformdirs before 4.6 didn't do. ]) def test_linux_caches_follow_an_absolute_xdg_cache_home(platform_home, tmp_path, monkeypatch, xdg, under): platform_home(platformdirs.unix.Unix) if xdg is not None: monkeypatch.setenv("XDG_CACHE_HOME", xdg.format(tmp=tmp_path)) assert get_data_dir("example") == str(tmp_path / under / "example") assert get_cache_root("example") == str(tmp_path / under / "example") @pytest.mark.skipif(os.name != "posix", reason="POSIX path separators") def test_windows_caches_repeat_the_application_name(platform_home, tmp_path): platform_home(platformdirs.windows.Windows) local = tmp_path / "Local" assert get_data_dir("example") == str(local / "example" / "example" / "Cache") # pyensembl's per-genome caches from before get_cache_root nest the name twice. assert get_data_dir("pyensembl/GRCh38/ensembl82") == str( local / "pyensembl" / "GRCh38" / "ensembl82" / "pyensembl" / "GRCh38" / "ensembl82" / "Cache") @pytest.mark.skipif(sys.platform != "darwin", reason="real macOS lookup") def test_this_mac_ignores_xdg_cache_home(tmp_path, monkeypatch): monkeypatch.setenv("HOME", str(tmp_path)) monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "xdg")) assert get_data_dir("example") == str(tmp_path / "Library" / "Caches" / "example") @pytest.mark.parametrize("arguments", [("",), (None,), ("openvax", ""), ("openvax", None)]) def test_cache_root_rejects_empty_names(arguments): with pytest.raises(ValueError): get_cache_root(*arguments) def test_cache_root_prefers_an_override_then_variables_then_legacy_data(tmp_path, monkeypatch): for home in ("HOME", "USERPROFILE"): # expanduser reads USERPROFILE on Windows. monkeypatch.setenv(home, str(tmp_path)) monkeypatch.delenv("EXAMPLE_ROOT", raising=False) platform = tmp_path / "platform" monkeypatch.setattr(common, "_platform_cache_dir", lambda name: str(platform)) old, older = tmp_path / ".example", tmp_path / ".older" assert get_cache_root("example", "EXAMPLE_ROOT", legacy=["~/.example"]) == str(platform) # Empty folders and incidental files don't count; a file at any depth does. (old / "proteomes" / "human").mkdir(parents=True) (old / ".DS_Store").write_text("") (old / ".datacache-file-registry.lock").write_text("") assert get_cache_root("example", legacy=["~/.example"]) == str(platform) (older / "a" / "b" / "c").mkdir(parents=True) (older / "a" / "b" / "c" / "data.fa").write_text(">p\n") assert get_cache_root("example", legacy=["~/.example", "~/.older"]) == str(older) (old / "proteomes" / "human" / "human.fa").write_text(">p\n") assert get_cache_root("example", legacy=["~/.example", "~/.older"]) == str(old) assert get_cache_root("example", legacy="~/.example") == str(old) # Once the platform directory holds data, it stays the root. platform.mkdir() (platform / "downloaded.fa").write_text(">p\n") assert get_cache_root("example", legacy=["~/.example"]) == str(platform) monkeypatch.setenv("EXAMPLE_ROOT", "~/chosen") assert get_cache_root("example", "EXAMPLE_ROOT", legacy=["~/.example"]) == str(tmp_path / "chosen") assert get_cache_root("example", "EXAMPLE_ROOT", override=tmp_path / "flag") == str(tmp_path / "flag") assert get_cache_root("example", override=" ~/flag ") == str(tmp_path / "flag") @pytest.mark.parametrize("options, message", [ (dict(override=""), "override"), (dict(override=" "), "override"), (dict(override=b"/bytes"), "text path"), (dict(legacy=[None]), "text path"), (dict(legacy=["relative/old"]), "absolute"), (dict(legacy=[""]), "non-empty"), ]) def test_cache_root_rejects_bad_locations_wherever_they_are(monkeypatch, options, message): monkeypatch.setenv("SET_ROOT", "/somewhere") # Checked up front, so a bad argument fails even when a variable is set. with pytest.raises(ValueError, match=message): get_cache_root("example", "SET_ROOT", **options) assert get_cache_root("example", "SET_ROOT", legacy=None) == "/somewhere" @pytest.mark.skipif(os.name != "posix", reason="POSIX symlinks and permissions") def test_unreadable_legacy_entries_are_skipped(tmp_path, monkeypatch): monkeypatch.setattr(common, "_platform_cache_dir", lambda name: str(tmp_path / "platform")) old = tmp_path / "old" old.mkdir() (old / "loop").symlink_to(old / "loop") (old / "locked").mkdir() (old / "locked").chmod(0) try: assert get_cache_root("example", legacy=[str(old)]) == str(tmp_path / "platform") (old / "data.fa").write_text(">p\n") assert get_cache_root("example", legacy=[str(old)]) == str(old) finally: (old / "locked").chmod(0o700) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7146666 datacache-1.27.1/tests/test_cross_platform_stores.py0000644000076500000000000000421315264220442022425 0ustar00iskanderwheel"""Bundle and materialization installs that also run on Windows CI (#89).""" from hashlib import sha256 from pathlib import Path from datacache import ( VersionedDatasetRegistry, inspect_bundle, inspect_materialization, install_bundle, materialize, ) def asset(path, data): path.write_bytes(data) return {"url": path.as_uri(), "sha256": sha256(data).hexdigest(), "size": len(data)} def test_bundles_install_reuse_and_refresh(tmp_path): upstream = tmp_path / "upstream" upstream.mkdir() assets = {"genes.gtf": asset(upstream / "genes.gtf", b"gene\r\n"), "dna/chr1.fa": asset(upstream / "chr1.fa", b">chr1\nACGT\n")} store = tmp_path / "store" paths = install_bundle(store, assets) assert Path(paths["genes.gtf"]).read_bytes() == b"gene\r\n" assert inspect_bundle(store, assets).verified assert install_bundle(store, assets) == paths refreshed = install_bundle(store, assets, force=True) assert refreshed != paths and all(Path(path).is_file() for path in paths.values()) registry = VersionedDatasetRegistry( {"reference": {"default_version": "v1", "versions": {"v1": assets}}}, cache_root=tmp_path / "cache") bundle = registry.ensure("reference") assert registry.local_path("reference", asset="genes.gtf") == bundle / "genes.gtf" assert registry.is_cached("reference") def test_materializations_build_and_reuse(tmp_path): source = tmp_path / "input.txt" source.write_bytes(b"acgt") built = [] def build(inputs, outputs): built.append(True) Path(outputs["upper.txt"]).write_bytes(Path(inputs["input.txt"]).read_bytes().upper()) options = dict(sources={"input.txt": {"path": source}}, transform={"version": "upper-1"}, outputs={"upper.txt": {}}, builder=build, download_options={"show_progress": False}) store = tmp_path / "derived" paths = materialize(store, **options) assert Path(paths["upper.txt"]).read_bytes() == b"ACGT" assert materialize(store, **options) == paths and built == [True] assert inspect_materialization(store).status == "available" assert materialize(store, force=True, **options) != paths ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792090401.714795 datacache-1.27.1/tests/test_database_objects.py0000644000076500000000000000643115264220442021252 0ustar00iskanderwheel""" Test that datacache constructs databases correctly (separately from downloading/caching them) """ import tempfile import threading import datacache def eq_(x, y): assert x == y TABLE_NAME = "test" INT_COL_NAME = "int_col" STR_COL_NAME = "str_col" COL_TYPES = [(INT_COL_NAME, "INT"), (STR_COL_NAME, "STR")] KEY_COLUMN_NAME = "int_col" NULLABLE = {STR_COL_NAME} ROWS = [(1, "darkness"), (2, "light"), (3, None)] INDICES = [["str_col"]] VERSION = 2 def make_table_object(): return datacache.database_table.DatabaseTable( name=TABLE_NAME, column_types=COL_TYPES, make_rows=lambda: ROWS, indices=INDICES, nullable=NULLABLE, primary_key=INT_COL_NAME) def test_database_table_object(): table = make_table_object() eq_(table.name, TABLE_NAME) eq_(table.indices, INDICES) eq_(table.nullable, NULLABLE) eq_(table.rows, ROWS) eq_(table.indices, INDICES) def test_create_db(): with tempfile.NamedTemporaryFile(suffix="test.db") as f: db = datacache.database.Database(f.name) table = make_table_object() db.create(tables=[table], version=VERSION) assert db.has_table(TABLE_NAME) assert db.has_version() assert db.version() == VERSION sql = """ SELECT %s from %s WHERE %s = '%s' """ % (INT_COL_NAME, TABLE_NAME, STR_COL_NAME, "light") cursor = db.connection.execute(sql) int_result_tuple = cursor.fetchone() int_result = int_result_tuple[0] eq_(int_result, 2) def test_query_db_from_another_thread(): # regression test for https://github.com/openvax/datacache/issues/45: # a connection opened in one thread must remain usable from another thread. with tempfile.NamedTemporaryFile(suffix="test.db") as f: db = datacache.database.Database(f.name) table = make_table_object() db.create(tables=[table], version=VERSION) results = {} def query_in_thread(): cursor = db.connection.execute( "SELECT %s FROM %s WHERE %s = '%s'" % ( INT_COL_NAME, TABLE_NAME, STR_COL_NAME, "light")) results["value"] = cursor.fetchone()[0] thread = threading.Thread(target=query_in_thread) thread.start() thread.join() eq_(results["value"], 2) def test_create_db_with_reserved_and_spaced_identifiers(): # regression test for https://github.com/openvax/datacache/issues/17: # table/column names that are SQL keywords or contain spaces must be # quoted so they don't have to be sanitized to alphanumeric + underscore. table_name = "weird table" column_types = [("order", "INT"), ("group by", "STR")] rows = [(1, "a"), (2, "b")] table = datacache.database_table.DatabaseTable( name=table_name, column_types=column_types, make_rows=lambda: rows, indices=[["group by"]], nullable={"group by"}, primary_key="order") with tempfile.NamedTemporaryFile(suffix="test.db") as f: db = datacache.database.Database(f.name) db.create(tables=[table], version=VERSION) assert db.has_table(table_name) cursor = db.connection.execute( 'SELECT "order" FROM "weird table" WHERE "group by" = ?', ("b",)) eq_(cursor.fetchone()[0], 2) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7149653 datacache-1.27.1/tests/test_database_safety.py0000644000076500000000000005266415264220442021125 0ustar00iskanderwheel"""Data fidelity, rollback, publication, and downstream SQLite contracts.""" from concurrent.futures import ThreadPoolExecutor from contextlib import closing import errno import os import sqlite3 import stat import threading import pandas as pd import pytest from datacache import connect_if_correct_version, db_from_dataframe, db_from_dataframes from datacache.database import Database, fold_identifier, quote_identifier from datacache import database_helpers from datacache.database_helpers import _create_cached_db, db_from_dataframes_with_absolute_path from datacache.database_table import DatabaseTable @pytest.mark.parametrize("dtype", ["int8", "int32", "int64", "uint32", "Int64", "boolean"]) def test_numeric_scalars_are_stored_as_integers(tmp_path, dtype): values = [True, False] if dtype == "boolean" else [1, 2] frame = pd.DataFrame({"value": pd.Series(values, dtype=dtype)}) with closing(db_from_dataframe(tmp_path / "data.db", "records", frame)) as connection: assert connection.execute("SELECT value, typeof(value) FROM records").fetchall() == [ (int(value), "integer") for value in values] def test_mixed_numbers_keep_integer_precision(tmp_path): value = 2**53 + 1 frame = pd.DataFrame({"id": [value], "weight": [1.5]}) with closing(db_from_dataframe(tmp_path / "data.db", "records", frame)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(value, 1.5)] assert connection.execute("SELECT id FROM records WHERE id = ?", (value,)).fetchone() == (value,) def test_missing_scalars_and_datetime_roundtrip(tmp_path): frame = pd.DataFrame({ "int": pd.Series([1, None], dtype="Int64"), "flag": pd.Series([True, None], dtype="boolean"), "float": pd.Series([1.5, None], dtype="Float64"), "string": pd.Series(["hello", None], dtype="string"), "category": pd.Series(["value", None], dtype="category"), "date": pd.to_datetime(["2026-01-02", None]), }) with closing(db_from_dataframe(tmp_path / "data.db", "records", frame)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [ (1, 1, 1.5, "hello", "value", "2026-01-02T00:00:00"), (None,) * 6] @pytest.mark.parametrize("key", ["sample id", "sample_id"]) def test_spaced_names_apply_to_keys_nullability_and_indices(tmp_path, key): frame = pd.DataFrame({"sample id": [1, 2], "group by": ["a", None]}) with closing(db_from_dataframe( tmp_path / "data.db", "records", frame, primary_key=key, indices=[("group by",)])) as connection: assert connection.execute("SELECT sample_id, group_by FROM records").fetchall() == [(1, "a"), (2, None)] with pytest.raises(sqlite3.IntegrityError): connection.execute("INSERT INTO records VALUES (1, 'duplicate')") indices = connection.execute("PRAGMA index_info(records_index_group_by)").fetchall() assert indices[0][2] == "group_by" @pytest.mark.parametrize("frame,options", [ (pd.DataFrame([[1, 2]], columns=["a b", "a_b"]), {}), (pd.DataFrame([[1, 2]], columns=["value", "VALUE"]), {}), (pd.DataFrame({"value": [1]}), {"primary_key": "missing"}), (pd.DataFrame({"value": [1]}), {"indices": [("missing",)]}), (pd.DataFrame({"value": [1]}), {"indices": ["value"]}), ]) def test_invalid_schema_does_not_destroy_existing_database(tmp_path, frame, options): path = tmp_path / "data.db" with closing(db_from_dataframe(path, "records", pd.DataFrame({"value": [42]}))): pass previous = path.read_bytes() with pytest.raises(ValueError): db_from_dataframe(path, "records", frame, overwrite=True, **options) assert path.read_bytes() == previous @pytest.mark.parametrize("overwrite", [False, True]) @pytest.mark.parametrize("failure", ["duplicate", "overflow", "interrupt"]) def test_failed_rebuild_preserves_data_version_and_permissions(tmp_path, monkeypatch, overwrite, failure): path = tmp_path / "data.db" with closing(db_from_dataframe(path, "records", pd.DataFrame({"value": [42]}), version=1)): pass path.chmod(0o640) mode = stat.S_IMODE(path.stat().st_mode) before = path.stat().st_ino values = list(range(1001)) + [1] error = sqlite3.IntegrityError if failure == "overflow": values = [2**63] error = OverflowError elif failure == "interrupt": values = list(range(1001)) original = Database._create_indices def interrupt(self, *args, **kwargs): original(self, *args, **kwargs) raise KeyboardInterrupt() monkeypatch.setattr(Database, "_create_indices", interrupt) error = KeyboardInterrupt with pytest.raises(error): db_from_dataframe(path, "records", pd.DataFrame({"value": values}), primary_key="value", version=2, overwrite=overwrite) with closing(connect_if_correct_version(path, 1)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(42,)] assert path.stat().st_ino == before assert stat.S_IMODE(path.stat().st_mode) == mode assert list(tmp_path.iterdir()) == [path] def test_empty_dataframe_is_a_complete_database(tmp_path): path = tmp_path / "empty.db" with closing(db_from_dataframe(path, "records", pd.DataFrame({"id": pd.Series([], dtype="int64")}))) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [] with closing(connect_if_correct_version(path, 1)) as connection: assert connection is not None def test_failed_new_database_is_not_published(tmp_path): with pytest.raises(sqlite3.IntegrityError): db_from_dataframe(tmp_path / "bad.db", "records", pd.DataFrame({"id": [1, 1]}), primary_key="id") assert list(tmp_path.iterdir()) == [] @pytest.mark.parametrize("overwrite", [False, True]) def test_successful_rebuild_keeps_existing_inode_and_readers(tmp_path, overwrite): path = tmp_path / "data.db" with closing(db_from_dataframe(path, "records", pd.DataFrame({"value": [1]}))) as reader: inode = path.stat().st_ino with closing(db_from_dataframe(path, "records", pd.DataFrame({"value": [2]}), version=1 if overwrite else 2, overwrite=overwrite)): pass assert reader.execute("SELECT * FROM records").fetchall() == [(2,)] assert path.stat().st_ino == inode def database_with_views(path, view_name="records"): quoted_view = quote_identifier(view_name) with closing(db_from_dataframe(path, "legacy", pd.DataFrame({"value": [42]}))) as connection: connection.execute("CREATE VIEW %s AS SELECT value FROM legacy" % quoted_view) connection.execute("CREATE VIEW summary AS SELECT * FROM %s" % quoted_view) connection.execute("CREATE TRIGGER remove_summary INSTEAD OF DELETE ON summary " "BEGIN DELETE FROM legacy WHERE value = OLD.value; END") connection.commit() path.chmod(0o640) return path.stat() @pytest.mark.parametrize("view_name", ["records", 'records "view"']) def test_explicit_overwrite_replaces_views_with_tables(tmp_path, view_name): path = tmp_path / "data.db" before = database_with_views(path, view_name) with closing(db_from_dataframe(path, view_name, pd.DataFrame({"value": [99]}), overwrite=True, version=2)) as connection: assert connection.execute("SELECT * FROM %s" % quote_identifier(view_name)).fetchall() == [(99,)] assert connection.execute("SELECT type, name FROM sqlite_master ORDER BY name").fetchall() == [ ("table", "_datacache_metadata"), ("table", view_name)] assert connection.execute("SELECT version FROM _datacache_metadata").fetchone() == (2,) assert path.stat().st_ino == before.st_ino assert stat.S_IMODE(path.stat().st_mode) == 0o640 @pytest.mark.parametrize("failure", ["constraint", "interrupt"]) def test_failed_overwrite_restores_views_and_triggers(tmp_path, monkeypatch, failure): path = tmp_path / "data.db" before = database_with_views(path) original_bytes = path.read_bytes() error = sqlite3.IntegrityError if failure == "interrupt": def interrupt(*args, **kwargs): raise KeyboardInterrupt() monkeypatch.setattr(Database, "_create_indices", interrupt) error = KeyboardInterrupt with pytest.raises(error): db_from_dataframe(path, "records", pd.DataFrame({"value": [1, 1]}), primary_key="value" if failure == "constraint" else None, overwrite=True, version=2) assert path.read_bytes() == original_bytes assert path.stat().st_ino == before.st_ino assert stat.S_IMODE(path.stat().st_mode) == 0o640 with closing(connect_if_correct_version(path, 1)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(42,)] assert connection.execute("SELECT * FROM summary").fetchall() == [(42,)] connection.execute("DELETE FROM summary") assert connection.execute("SELECT * FROM legacy").fetchall() == [] connection.rollback() assert list(tmp_path.iterdir()) == [path] def test_cache_reuse_and_version_rebuild_preserve_views(tmp_path): path = tmp_path / "data.db" database_with_views(path) original_bytes = path.read_bytes() with closing(db_from_dataframe(path, "legacy", None)) as connection: assert connection.execute("SELECT * FROM summary").fetchall() == [(42,)] assert path.read_bytes() == original_bytes with closing(db_from_dataframe(path, "legacy", pd.DataFrame({"value": [99]}), version=2)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(99,)] assert connection.execute("SELECT * FROM summary").fetchall() == [(99,)] @pytest.mark.parametrize("symlink", [False, True]) def test_concurrent_new_database_creators_reuse_winner(tmp_path, symlink): path = tmp_path / "data.db" target = tmp_path / "target.db" if symlink: path.symlink_to(target.name) barrier = threading.Barrier(2, timeout=10) def create(value): def rows(): assert not path.exists() barrier.wait() yield (value,) table = DatabaseTable("records", [("value", "INT")], rows) with closing(_create_cached_db(path, [table])) as connection: return connection.execute("SELECT value FROM records").fetchone()[0] with ThreadPoolExecutor(max_workers=2) as executor: first = executor.submit(create, 1) second = executor.submit(create, 2) assert first.result(timeout=15) == second.result(timeout=15) assert set(tmp_path.iterdir()) == ({path, target} if symlink else {path}) if symlink: assert path.is_symlink() assert os.readlink(path) == target.name def test_version_lookup_does_not_create_and_can_open_read_only(tmp_path): path = tmp_path / "data.db" assert connect_if_correct_version(path, 1) is None assert not path.exists() with closing(db_from_dataframe(path, "records", pd.DataFrame({"value": [1]}))): pass assert connect_if_correct_version(path, 2) is None with closing(connect_if_correct_version(path, 1, read_only=True)) as connection: assert connection.execute("SELECT value FROM records").fetchall() == [(1,)] with pytest.raises(sqlite3.OperationalError, match="readonly"): connection.execute("DELETE FROM records") def test_direct_database_create_rolls_back_partial_tables(tmp_path): db = Database(tmp_path / "data.db") table = DatabaseTable("records", [("id", "INT")], lambda: [(1,), (1,)], primary_key="id") try: with pytest.raises(sqlite3.IntegrityError): db.create([table], 1) assert db.table_names() == [] assert not db.connection.in_transaction finally: db.connection.close() def test_primary_key_cannot_be_null(tmp_path): with pytest.raises(sqlite3.IntegrityError): db_from_dataframe(tmp_path / "data.db", "records", pd.DataFrame({"sample id": pd.Series([1, None], dtype="Int64")}), primary_key="sample id") assert list(tmp_path.iterdir()) == [] @pytest.mark.parametrize("operation", ["link", "chmod"]) def test_failed_new_database_publication_cleans_up(tmp_path, monkeypatch, operation): def reject(*args): raise PermissionError("injected publication failure") monkeypatch.setattr(os, operation, reject) with pytest.raises(PermissionError): db_from_dataframe(tmp_path / "data.db", "records", pd.DataFrame({"id": [1]})) assert list(tmp_path.iterdir()) == [] @pytest.mark.skipif(os.name != "posix", reason="POSIX umask semantics") @pytest.mark.parametrize("mask", [0o022, 0o002, 0o077]) def test_new_database_honors_umask(tmp_path, mask): path = tmp_path / "data.db" previous = os.umask(mask) try: with closing(db_from_dataframe(path, "records", pd.DataFrame({"id": [1]}))): pass finally: os.umask(previous) assert stat.S_IMODE(path.stat().st_mode) == 0o666 & ~mask def test_mismatched_version_closes_rejected_connection(tmp_path, monkeypatch): from datacache import database_helpers path = tmp_path / "data.db" with closing(db_from_dataframe(path, "records", pd.DataFrame({"id": [1]}))): pass opened = [] def observed(*args, **kwargs): db = Database(*args, **kwargs) opened.append(db.connection) return db monkeypatch.setattr(database_helpers, "Database", observed) assert connect_if_correct_version(path, 2) is None assert len(opened) == 1 with pytest.raises(sqlite3.ProgrammingError, match="closed"): opened[0].execute("SELECT 1") def test_new_failed_creator_does_not_remove_concurrent_winner(tmp_path): path = tmp_path / "data.db" started = threading.Event() published = threading.Event() def failed_rows(): started.set() assert published.wait(timeout=10) yield (2,) raise RuntimeError("failed after another creator published") def fail(): table = DatabaseTable("records", [("id", "INT")], failed_rows) with pytest.raises(RuntimeError, match="another creator"): _create_cached_db(path, [table]) with ThreadPoolExecutor(max_workers=1) as executor: failed = executor.submit(fail) try: assert started.wait(timeout=10) with closing(db_from_dataframe(path, "records", pd.DataFrame({"id": [1]}))): pass finally: published.set() failed.result(timeout=10) with closing(connect_if_correct_version(path, 1)) as connection: assert connection.execute("SELECT id FROM records").fetchall() == [(1,)] assert list(tmp_path.iterdir()) == [path] def test_reuse_matches_table_names_ignoring_ascii_case(tmp_path): # SQLite treats "Records" and "records" as one table, so a caller spelling # it differently must reuse the database instead of rebuilding it. with closing(db_from_dataframe("cased.db", "Records", pd.DataFrame({"id": [1]}), cache_root=tmp_path)): pass with closing(db_from_dataframe( "cased.db", "records", pd.DataFrame({"id": [2]}), cache_root=tmp_path)) as connection: assert connection.execute("SELECT id FROM records").fetchall() == [(1,)] def test_reuse_keeps_non_ascii_case_distinct_like_sqlite(tmp_path): # SQLite folds ASCII letters only: "Éclair" and "éclair" are two tables. # Matching them with str.lower() would reuse a table SQLite cannot find. with closing(db_from_dataframe("accented.db", "Éclair", pd.DataFrame({"id": [1]}), cache_root=tmp_path)): pass with closing(db_from_dataframe( "accented.db", "éclair", pd.DataFrame({"id": [2]}), cache_root=tmp_path)) as connection: assert connection.execute('SELECT id FROM "éclair"').fetchall() == [(2,)] @pytest.mark.parametrize("name", ["_datacache_metadata", "_Datacache_Metadata", "SQLite_Sequence"]) def test_reserved_table_names_are_rejected_even_when_a_database_exists(tmp_path, name): # Reuse must never hand back the version metadata as if it were data. with closing(db_from_dataframe("reserved.db", "records", pd.DataFrame({"id": [1]}), cache_root=tmp_path)): pass with pytest.raises(ValueError, match="reserved"): db_from_dataframe("reserved.db", name, pd.DataFrame({"id": [2]}), cache_root=tmp_path) def test_names_are_distinct_exactly_when_sqlite_says_so(tmp_path): frames = {"Éclair": pd.DataFrame({"id": [1]}), "éclair": pd.DataFrame({"id": [2]})} with closing(db_from_dataframes("tables.db", frames, cache_root=tmp_path)) as connection: assert connection.execute('SELECT id FROM "Éclair"').fetchall() == [(1,)] assert connection.execute('SELECT id FROM "éclair"').fetchall() == [(2,)] frame = pd.DataFrame({"Éclair": [1], "éclair": [2]}) with closing(db_from_dataframe("columns.db", "t", frame, cache_root=tmp_path)) as connection: assert connection.execute('SELECT "Éclair", "éclair" FROM t').fetchall() == [(1, 2)] with pytest.raises(ValueError, match="distinct"): db_from_dataframes("ascii.db", {"Records": frame, "records": frame}, cache_root=tmp_path) with pytest.raises(ValueError, match="collide"): db_from_dataframe("ascii-columns.db", "t", pd.DataFrame({"ID": [1], "id": [2]}), cache_root=tmp_path) def test_names_without_room_for_the_journal_are_built_and_reused_but_not_rebuilt(tmp_path): # Staged creation works at any valid name, but SQLite rebuilds through # "-journal", which does not fit here. Explain that clearly. frame = pd.DataFrame({"id": [1]}) long_name = "r" * 250 + ".db" with closing(db_from_dataframe(long_name, "t", frame, cache_root=tmp_path)) as connection: assert connection.execute("SELECT id FROM t").fetchall() == [(1,)] with closing(db_from_dataframe(long_name, "t", frame, cache_root=tmp_path)) as connection: assert connection.execute("SELECT id FROM t").fetchall() == [(1,)] with pytest.raises(ValueError, match="Cannot rebuild.*journal"): db_from_dataframe(long_name, "t", frame, cache_root=tmp_path, version=2) def test_rebuilds_are_allowed_wherever_the_filesystem_fits_the_journal(tmp_path): # 93 characters but 273 UTF-8 bytes: APFS counts characters and accepts it, # ext4 counts bytes and does not. Where it exists, it must rebuild. name = "數據" * 45 + ".db" try: created = db_from_dataframe(name, "t", pd.DataFrame({"id": [1]}), cache_root=tmp_path) except OSError as error: if error.errno != errno.ENAMETOOLONG: raise pytest.skip("this filesystem limits names in bytes") created.close() with closing(db_from_dataframe( name, "t", pd.DataFrame({"id": [2]}), cache_root=tmp_path, version=2)) as connection: assert connection.execute("SELECT id FROM t").fetchall() == [(2,)] def test_an_empty_request_still_reuses_an_existing_database(tmp_path): path = tmp_path / "existing.db" with closing(db_from_dataframes_with_absolute_path(path, {"t": pd.DataFrame({"id": [1]})})): pass with closing(db_from_dataframes_with_absolute_path(path, {})) as connection: assert connection.execute("SELECT id FROM t").fetchall() == [(1,)] def test_python_and_sqlite_compare_names_alike(): # Validation folds names in Python before a database exists; lookups ask # SQLite. Both must agree on which names are the same. names = ["Records", "records", "RECORDS", "Éclair", "éclair", "ÉCLAIR", "\u212a", "k", "K", "straße", "STRASSE", "İstanbul", "istanbul"] with closing(sqlite3.connect(":memory:")) as connection: for left in names: for right in names: same_in_sqlite = connection.execute( "SELECT ? = ? COLLATE NOCASE", (left, right)).fetchone()[0] assert (fold_identifier(left) == fold_identifier(right)) == bool(same_in_sqlite), (left, right) def test_a_lock_during_a_rebuild_is_reported_as_a_lock(tmp_path, monkeypatch): # Even when the name leaves no room for the journal, contention for the # lock must surface as SQLite's own error so callers can retry it. real_connect = sqlite3.connect monkeypatch.setattr(sqlite3, "connect", lambda *args, **kwargs: real_connect(*args, **{"timeout": 0.1, **kwargs})) long_name = "r" * 250 + ".db" db_from_dataframe(long_name, "t", pd.DataFrame({"id": [1]}), cache_root=tmp_path).close() holder = sqlite3.connect(tmp_path / long_name, isolation_level=None) holder.execute("BEGIN IMMEDIATE") try: with pytest.raises(sqlite3.OperationalError, match="locked"): db_from_dataframe(long_name, "t", pd.DataFrame({"id": [2]}), cache_root=tmp_path, version=2) finally: holder.execute("ROLLBACK") holder.close() def test_windows_reports_an_overlong_journal_name_too(tmp_path, monkeypatch): # Windows raises ERROR_FILENAME_EXCED_RANGE (206) instead of ENAMETOOLONG. real_stat = os.stat def windows_stat(path, *args, **kwargs): if os.fspath(path).endswith("-journal"): error = OSError(errno.ENOENT, "The filename or extension is too long") error.winerror = 206 raise error return real_stat(path, *args, **kwargs) monkeypatch.setattr(database_helpers.os, "stat", windows_stat) with pytest.raises(ValueError, match="Cannot rebuild"): database_helpers._explain_rebuild_failure( str(tmp_path / "x.db"), sqlite3.OperationalError("unable to open database file")) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7150786 datacache-1.27.1/tests/test_database_symlinks.py0000644000076500000000000000755315264220442021500 0ustar00iskanderwheel"""Preserve linked cache locations when creating or rebuilding SQLite files.""" from contextlib import closing import errno import os import sqlite3 import stat import pandas as pd import pytest from datacache import connect_if_correct_version, db_from_dataframe @pytest.mark.parametrize("target_kind", ["relative", "absolute", "chain", "parent"]) def test_dangling_database_symlink_creates_target_and_keeps_link(tmp_path, target_kind): directory = tmp_path / "data" directory.mkdir() target = directory / "records.db" path = tmp_path / "cache.db" link_text = "data/records.db" if target_kind == "absolute": link_text = str(target) elif target_kind == "chain": (tmp_path / "alias.db").symlink_to(link_text) link_text = "alias.db" elif target_kind == "parent": (directory / "child").mkdir() (tmp_path / "alias").symlink_to(directory / "child", target_is_directory=True) link_text = "alias/../records.db" path.symlink_to(link_text) before = path.lstat() assert connect_if_correct_version(path, 1) is None assert not target.exists() with closing(db_from_dataframe(path, "records", pd.DataFrame({"value": [42]}))) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(42,)] assert target.is_file() target.chmod(0o640) inode = target.stat().st_ino with closing(db_from_dataframe(path, "records", None)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(42,)] with closing(db_from_dataframe(path, "records", pd.DataFrame({"value": [99]}), overwrite=True)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(99,)] assert target.stat().st_ino == inode assert stat.S_IMODE(target.stat().st_mode) == 0o640 assert path.is_symlink() assert path.lstat().st_ino == before.st_ino assert os.readlink(path) == link_text if target_kind == "chain": assert (tmp_path / "alias.db").is_symlink() assert not (tmp_path / "records.db").exists() assert not list(directory.glob(".datacache-*")) @pytest.mark.parametrize("failure", ["constraint", "publication"]) def test_failed_creation_keeps_dangling_link_without_partial_target(tmp_path, monkeypatch, failure): directory = tmp_path / "data" directory.mkdir() path = tmp_path / "cache.db" path.symlink_to("data/records.db") before = path.lstat() error = sqlite3.IntegrityError if failure == "publication": def reject(*args, **kwargs): raise PermissionError("injected publication failure") monkeypatch.setattr(os, "link", reject) error = PermissionError with pytest.raises(error): db_from_dataframe(path, "records", pd.DataFrame({"id": [1, 1]}), primary_key="id" if failure == "constraint" else None) assert list(directory.iterdir()) == [] assert path.is_symlink() assert path.lstat().st_ino == before.st_ino assert os.readlink(path) == "data/records.db" def test_dangling_link_does_not_normalize_away_missing_parent(tmp_path): path = tmp_path / "cache.db" path.symlink_to("missing/../records.db") with pytest.raises(FileNotFoundError): db_from_dataframe(path, "records", pd.DataFrame({"value": [42]})) assert list(tmp_path.iterdir()) == [path] assert path.is_symlink() def test_cyclic_database_symlinks_are_not_replaced(tmp_path): path = tmp_path / "cache.db" path.symlink_to("./alias.db") alias = tmp_path / "alias.db" alias.symlink_to("./cache.db") with pytest.raises(OSError) as error: db_from_dataframe(path, "records", pd.DataFrame({"value": [42]}), overwrite=True) assert error.value.errno == errno.ELOOP assert path.is_symlink() assert alias.is_symlink() assert set(tmp_path.iterdir()) == {path, alias} ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792090401.715171 datacache-1.27.1/tests/test_database_types.py0000644000076500000000000000342115264220442020761 0ustar00iskanderwheelfrom contextlib import closing import numpy as np import pandas as pd from datacache import db_from_dataframe from datacache.database_types import db_type def test_db_types(): for int_type in [ int, np.int8, np.int16, np.int32, np.int64, np.uint8, np.uint16, np.uint32, np.uint64]: assert db_type(int_type) == "INT" for float_type in [float, np.float32, np.float64]: assert db_type(float) == "FLOAT" assert db_type(str) == "TEXT" def test_float16_columns_are_stored_as_floats(tmp_path): assert db_type(np.float16) == "FLOAT" frame = pd.DataFrame({"value": np.array([1.5], dtype=np.float16)}) with closing(db_from_dataframe("half.db", "halves", frame, cache_root=tmp_path)) as connection: assert connection.execute("SELECT value FROM halves").fetchall() == [(1.5,)] def test_numpy_fixed_width_and_bytes_types_are_mapped(): # dtype('S3').type is bytes_ and dtype('U5').type is str_ on Python 3; # the Python 2 name string_ never matched them. assert db_type(np.dtype("U5")) == "TEXT" assert db_type(np.str_) == "TEXT" assert db_type(np.dtype("S3")) == "BLOB" assert db_type(np.bytes_) == "BLOB" assert db_type(bytes) == "BLOB" def test_dataframes_of_fixed_width_arrays_keep_text_columns(tmp_path): # pandas stores such arrays with the object dtype, so existing schemas # built from DataFrames are unchanged. frame = pd.DataFrame({"b": np.array([b"ab"], dtype="S2"), "u": np.array(["x"], dtype="U1")}) with closing(db_from_dataframe("fixed.db", "t", frame, cache_root=tmp_path)) as connection: assert [row[2] for row in connection.execute("PRAGMA table_info(t)")] == ["TEXT", "TEXT"] assert connection.execute("SELECT * FROM t").fetchall() == [(b"ab", "x")] ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7152586 datacache-1.27.1/tests/test_db_from_dataframes.py0000644000076500000000000000251715264220442021575 0ustar00iskanderwheelimport pandas as pd from tempfile import NamedTemporaryFile from datacache import db_from_dataframes, db_from_dataframe dfA = pd.DataFrame({"numbers": [1, 2, 3], "strings": ["a", "b", "c"]}) dfB = pd.DataFrame({"wuzzles": ["nuzzle", "ruzzle"]}) def eq_(x, y): assert x == y def test_database_from_dataframes(): with NamedTemporaryFile(suffix="test.db") as f: db = db_from_dataframes( db_filename=f.name, dataframes={"A": dfA, "B": dfB}, primary_keys={"A": "numbers"}, indices={"A": [("numbers", "strings")]}, subdir="test_datacache") cursor_A = db.execute("SELECT * FROM A") results_A = cursor_A.fetchall() eq_(results_A, [(1, "a"), (2, "b"), (3, "c")]) cursor_B = db.execute("SELECT * FROM B") results_B = cursor_B.fetchall() eq_(results_B, [("nuzzle",), ("ruzzle",)]) def test_database_from_single_dataframe(): with NamedTemporaryFile(suffix="test.db") as f: db = db_from_dataframe( db_filename=f.name, table_name="A", df=dfA, primary_key="numbers", indices=[("numbers", "strings")], subdir="test_datacache") cursor = db.execute("SELECT * FROM A") results = cursor.fetchall() eq_(results, [(1, "a"), (2, "b"), (3, "c")]) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7153726 datacache-1.27.1/tests/test_docs_api_reference.py0000644000076500000000000001205015264220442021566 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Keep docs/api.md true: its version, its coverage, and its runnable examples. The reference page hardcodes a release version, claims to cover every name in ``datacache.__all__`` plus every public ``Cache`` method, and tells readers to run its examples in order in one session. Nothing but these checks stops those promises from drifting away from the code. """ import inspect import re from pathlib import Path import pytest import datacache from datacache.version import __version__ REFERENCE = Path(__file__).parent.parent / "docs" / "api.md" def reference_text(): if not REFERENCE.is_file(): pytest.skip("docs/api.md is not available in this installation") return REFERENCE.read_text() def canonical_annotations(text): """Compare annotations the same way on every supported Python version. inspect.signature renders typing.Optional[X] as "Optional[X]" before Python 3.14 and as "X | None" from 3.14 on, and qualifies dataclass references with their module. Reduce both spellings to one form so the reference does not have to name a particular interpreter's formatting. """ text = re.sub(r"\s+", "", text).replace("datacache.inspection.", "") while True: start = text.find("Optional[") if start == -1: return text opening = start + len("Optional[") - 1 depth = 0 for index in range(opening, len(text)): if text[index] == "[": depth += 1 elif text[index] == "]": depth -= 1 if depth == 0: inner = text[opening + 1:index] text = text[:start] + inner + "|None" + text[index + 1:] break else: return text def test_documented_version_matches_package(): text = reference_text() documented = re.findall(r"in DataCache (\d+\.\d+\.\d+)\.", text) assert documented == [__version__], ( "docs/api.md names DataCache %s but datacache.version.__version__ is %s; " "update the reference when bumping the version." % (documented, __version__)) def test_every_public_name_is_documented(): text = reference_text() headings = set(re.findall(r"^#+\s+`?([A-Za-z_][\w.]*)", text, re.M)) missing = sorted(name for name in datacache.__all__ if name not in headings) assert not missing, "docs/api.md has no section for: %s" % missing methods = sorted( name for name in vars(datacache.Cache) if not name.startswith("_") and callable(getattr(datacache.Cache, name))) undocumented = sorted("Cache.%s" % name for name in methods if "Cache.%s" % name not in headings) assert not undocumented, "docs/api.md has no section for: %s" % undocumented def test_documented_signatures_match_code(): text = reference_text() mismatched = [] for block in re.findall(r"```text\n(.*?)```", text, re.S): collapsed = " ".join(block.split()) match = re.match(r"^([A-Za-z_][\w.]*)\((.*)\)$", collapsed) if match is None: continue name, documented = match.group(1), match.group(2) if name.startswith("Cache."): signature = inspect.signature(getattr(datacache.Cache, name.split(".", 1)[1])) parameters = list(signature.parameters.values())[1:] else: signature = inspect.signature(getattr(datacache, name)) parameters = list(signature.parameters.values()) actual = str(signature.replace( parameters=parameters, return_annotation=inspect.Signature.empty))[1:-1] if canonical_annotations(actual) != canonical_annotations(documented): mismatched.append("%s\n documented: %s\n actual: %s" % (name, documented, actual)) assert not mismatched, "docs/api.md signatures are stale:\n" + "\n".join(mismatched) def test_examples_run_in_order(): """The page's examples share one namespace and must run offline, in order.""" pytest.importorskip("pandas") blocks = re.findall(r"```python\n(.*?)```", reference_text(), re.S) assert len(blocks) > 1, "docs/api.md should contain runnable examples" namespace = {"__name__": "docs_api_examples"} try: for index, block in enumerate(blocks): source = compile(block, "docs/api.md[block %d]" % index, "exec") exec(source, namespace) finally: # The page cleans up in its last block; do it again if a block failed. temporary = namespace.get("temporary") if temporary is not None: temporary.cleanup() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7154891 datacache-1.27.1/tests/test_download.py0000644000076500000000000002544715264220442017614 0ustar00iskanderwheel# Copyright (c) 2014. Mount Sinai School of Medicine # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Tests for ``fetch_file`` download + decompression. These serve a local ``.gz`` over a ``file://`` URL and redirect the cache to an isolated tmp dir, so they don't depend on a live FTP server or on cross-run cached state -- the two sources of flakiness in the old live-Ensembl test (#42). """ import gzip import logging import os import zipfile import pytest from datacache import common, fetch_file @pytest.fixture def isolated_cache(tmp_path, monkeypatch): """Point datacache's cache dir at an isolated tmp dir (so the test neither depends on nor pollutes the user's real cache). Fixtures live in ``tmp_path``; the cache lives in ``tmp_path/cache``.""" cache = tmp_path / "cache" monkeypatch.setattr(common, "get_data_dir", lambda subdir=None, envkey=None: str(cache)) return tmp_path def _write_gz(path, text): with gzip.open(str(path), "wb") as f: f.write(text.encode()) return "file://" + str(path) def test_fetch_decompress(isolated_cache): url = _write_gz(isolated_cache / "seq.fa.gz", "ACGTACGT TCAATTTCGTGCCAG\n") path = fetch_file(url, filename="seq.fa.gz", decompress=True) assert path.endswith("seq.fa") with open(path) as f: assert "TCAATTTCGTGCCAG" in f.read() def test_fetch_decompress_caches_then_force(isolated_cache): # Deterministic cache behaviour. The old test looped over # use_wget_if_available/timeout but the file was downloaded once and reused, # so the result depended on iteration order (#42). Here we assert it # explicitly: without force the cached copy is reused even if the source # changes; with force it's re-fetched. src = isolated_cache / "data.fa.gz" url = _write_gz(src, "FIRST\n") p1 = fetch_file(url, filename="data.fa.gz", decompress=True) with open(p1) as f: assert f.read() == "FIRST\n" _write_gz(src, "SECOND\n") # change source; cached copy should win p2 = fetch_file(url, filename="data.fa.gz", decompress=True) assert p2 == p1 with open(p2) as f: assert f.read() == "FIRST\n" p3 = fetch_file(url, filename="data.fa.gz", decompress=True, force=True) with open(p3) as f: assert f.read() == "SECOND\n" def test_fetch_decompress_zip_picks_named_member(isolated_cache): # Multi-member zip: the member matching the local filename is chosen and # streamed into the cache -- never extracted into the working directory # (the cwd-pollution / path-traversal footgun of the old ZipFile.extract). archive = isolated_cache / "table.tsv.zip" with zipfile.ZipFile(str(archive), "w") as z: z.writestr("table.tsv", "the wanted member\n") z.writestr("readme.txt", "ignore me\n") url = "file://" + str(archive) path = fetch_file(url, filename="table.tsv.zip", decompress=True) assert path.endswith("table.tsv") with open(path) as f: assert f.read() == "the wanted member\n" # Nothing leaked into the current working directory. assert not os.path.exists("table.tsv") assert not os.path.exists("readme.txt") def test_fetch_decompress_zip_matches_member_inside_a_folder(isolated_cache): # Archives often wrap their files in a top-level folder. The member named # like the output must win over a larger sibling rather than having the # largest member's contents silently installed under the requested name. archive = isolated_cache / "release.zip" with zipfile.ZipFile(str(archive), "w") as z: z.writestr("release/wanted.csv", "the wanted member\n") z.writestr("release/other.csv", "a much larger member\n" * 100) path = fetch_file("file://" + str(archive), filename="wanted.csv", decompress=True) with open(path) as f: assert f.read() == "the wanted member\n" def test_fetch_decompress_zip_prefers_the_exact_member_path(isolated_cache): archive = isolated_cache / "release.zip" with zipfile.ZipFile(str(archive), "w") as z: z.writestr("wanted.csv", "top level\n") z.writestr("nested/wanted.csv", "a larger nested copy\n" * 100) path = fetch_file("file://" + str(archive), filename="wanted.csv", decompress=True) with open(path) as f: assert f.read() == "top level\n" def test_corrupt_gz_leaves_no_partial_cache(isolated_cache): # A truncated gzip decompresses partway then fails its trailing CRC check. # fetch_file must surface the error and leave NO file at the destination, # so the next call re-fetches instead of serving a silent partial. good = gzip.compress(b"the full payload that must not be half-cached\n" * 50) corrupt = good[:-8] # drop the CRC32 + ISIZE trailer archive = isolated_cache / "trunc.fa.gz" archive.write_bytes(corrupt) url = "file://" + str(archive) with pytest.raises(Exception): fetch_file(url, filename="trunc.fa.gz", decompress=True) dest = isolated_cache / "cache" / "trunc.fa" assert not dest.exists(), "corrupt decompress must not leave a cached partial" def test_fetch_subdirs(tmp_path, monkeypatch): # Different subdirs resolve to different cache locations. Hermetic: route # each subdir to its own tmp dir. src = tmp_path / "seq.fa.gz" url = _write_gz(src, "ACGT\n") dirs = {"datacache": tmp_path / "a", "datacache_test": tmp_path / "b"} monkeypatch.setattr( common, "get_data_dir", lambda subdir=None, envkey=None: str(dirs.get(subdir, tmp_path / "default")), ) path = fetch_file(url, filename="seq.fa.gz", decompress=True, subdir="datacache") assert path.endswith("seq.fa") other_path = fetch_file(url, filename="seq.fa.gz", decompress=True, subdir="datacache_test") assert other_path.endswith("seq.fa") assert other_path != path def test_use_wget_if_available_is_deprecated(isolated_cache): # The legacy wget path was removed; passing the argument is accepted for # backwards compatibility but warns and is otherwise ignored. url = _write_gz(isolated_cache / "w.fa.gz", "ACGT\n") with pytest.warns(DeprecationWarning, match="use_wget_if_available is deprecated"): path = fetch_file(url, filename="w.fa.gz", decompress=True, use_wget_if_available=True) assert path.endswith("w.fa") def test_fetch_decompress_zip_matches_member_names_ignoring_case(isolated_cache, caplog): archive = isolated_cache / "cased.zip" with zipfile.ZipFile(str(archive), "w") as z: z.writestr("Data.CSV", "the wanted member\n") z.writestr("notes.txt", "a much larger member\n" * 100) with caplog.at_level(logging.WARNING, logger="datacache.download"): path = fetch_file("file://" + str(archive), filename="data.csv", decompress=True) with open(path) as f: assert f.read() == "the wanted member\n" assert not caplog.records def test_fetch_decompress_zip_prefers_the_copy_nearest_the_root(isolated_cache): archive = isolated_cache / "nested.zip" with zipfile.ZipFile(str(archive), "w") as z: z.writestr("release/data.csv", "current\n") z.writestr("release/archive/2019/data.csv", "older and larger\n" * 100) path = fetch_file("file://" + str(archive), filename="data.csv", decompress=True) with open(path) as f: assert f.read() == "current\n" def test_fetch_decompress_zip_warns_only_when_guessing_among_members(isolated_cache, caplog): several = isolated_cache / "several.zip" with zipfile.ZipFile(str(several), "w") as z: z.writestr("readme.txt", "short\n") z.writestr("table.tsv", "the largest member\n" * 10) single = isolated_cache / "single.zip" with zipfile.ZipFile(str(single), "w") as z: z.writestr("anything.bin", "the only member\n") with caplog.at_level(logging.WARNING, logger="datacache.download"): guessed = fetch_file("file://" + str(several), filename="guessed.csv", decompress=True) assert "table.tsv" in caplog.text caplog.clear() only = fetch_file("file://" + str(single), filename="only.csv", decompress=True) assert not caplog.records with open(guessed) as f: assert f.read().startswith("the largest member") with open(only) as f: assert f.read() == "the only member\n" def test_fetch_decompress_zip_is_quiet_for_inferred_names(isolated_cache, caplog): # An inferred cache key can never match a member name, so installing the # largest member is the expected behavior, not a guess worth reporting. archive = isolated_cache / "inferred.zip" with zipfile.ZipFile(str(archive), "w") as z: z.writestr("data.csv", "the largest member\n" * 10) z.writestr("README", "short\n") with caplog.at_level(logging.WARNING, logger="datacache.download"): path = fetch_file("file://" + str(archive), decompress=True) assert not caplog.records with open(path) as f: assert f.read().startswith("the largest member") def test_fetch_decompress_zip_prefers_the_root_over_letter_case(isolated_cache): # The dataset at the root wins over a nested sample, even though only the # sample matches the requested name's letter case exactly. archive = isolated_cache / "rooted.zip" with zipfile.ZipFile(str(archive), "w") as z: z.writestr("DATA.CSV", "the dataset\n") z.writestr("docs/examples/data.csv", "a sample\n") path = fetch_file("file://" + str(archive), filename="data.csv", decompress=True) with open(path) as f: assert f.read() == "the dataset\n" @pytest.mark.parametrize("root_name", ["./DATA.CSV", "/DATA.CSV"]) def test_fetch_decompress_zip_treats_dot_and_slash_prefixes_as_the_root(isolated_cache, root_name): archive = isolated_cache / "prefixed.zip" with zipfile.ZipFile(str(archive), "w") as z: z.writestr(zipfile.ZipInfo(root_name), "the dataset\n") z.writestr("docs/data.csv", "a sample\n") path = fetch_file("file://" + str(archive), filename="data.csv", decompress=True) with open(path) as f: assert f.read() == "the dataset\n" def test_fetch_file_accepts_names_longer_than_255_utf8_bytes(isolated_cache): source = isolated_cache / "source.csv" source.write_text("id\n1\n") path = fetch_file("file://" + str(source), filename="測" * 100 + ".csv") assert os.path.basename(path).endswith(".csv") assert len(os.path.basename(path).encode("utf-8")) <= 255 with open(path) as f: assert f.read() == "id\n1\n" ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7156448 datacache-1.27.1/tests/test_download_compatibility.py0000644000076500000000000004351115264220442022535 0ustar00iskanderwheel"""Offline compatibility checks for URL formats and publication permissions.""" import gzip import hashlib import io import os from pathlib import Path import stat from uuid import UUID import zipfile import pandas as pd import pytest from datacache import common, download, fetch_file CONTENTS = b"complete contents\n" @pytest.fixture(autouse=True) def offline_cache(tmp_path, monkeypatch): monkeypatch.setattr(common, "get_data_dir", lambda subdir=None: str(tmp_path / "cache")) def reject_network(*args, **kwargs): pytest.fail("unexpected network access") monkeypatch.setattr(download.requests, "get", reject_network) monkeypatch.setattr(download.urllib.request, "urlopen", reject_network) def serve(monkeypatch, contents): def stream(url, output, **kwargs): output.write(contents) return len(contents) monkeypatch.setattr(download, "_stream_to_file", stream) def archive_bytes(extension): if extension.lower() == "gz": return gzip.compress(CONTENTS) output = io.BytesIO() with zipfile.ZipFile(output, "w") as archive: archive.writestr("data", CONTENTS) archive.writestr("other", b"second member") return output.getvalue() @pytest.mark.parametrize("extension", ["gz", "zip"]) @pytest.mark.parametrize("decoration,legacy_suffix", [ ("?token=abc", "_token_abc"), ("#download", "#download"), ("?token=abc#download", "_token_abc#download"), ]) def test_default_archive_retention_and_cache_key(monkeypatch, extension, decoration, legacy_suffix): url = "https://host/data." + extension + decoration payload = archive_bytes(extension) serve(monkeypatch, payload) kwargs = dict(expected_sha256=hashlib.sha256(payload).hexdigest()) path = Path(fetch_file(url, **kwargs)) # Preserve the key used by master, as well as the bytes stored under it. assert path.name == (hashlib.md5(url.encode()).hexdigest() + ".https___host_data." + extension + legacy_suffix) assert path.read_bytes() == payload assert fetch_file(url, **kwargs) == str(path) assert fetch_file(url, force=True, **kwargs) == str(path) if extension == "zip": with zipfile.ZipFile(path) as archive: assert archive.namelist() == ["data", "other"] else: with gzip.open(path, "rb") as archive: assert archive.read() == CONTENTS @pytest.mark.parametrize("extension", ["gz", "zip", "GZ", "ZIP"]) @pytest.mark.parametrize("decoration", ["", "?token=abc", "#download", "?next=file.zip#download"]) def test_inferred_archive_and_decompressed_paths_are_distinct(monkeypatch, extension, decoration): url = "https://host/data." + extension + decoration payload = archive_bytes(extension) serve(monkeypatch, payload) archive_path = Path(fetch_file(url)) installed_path = Path(fetch_file( url, decompress=True, expected_sha256=hashlib.sha256(CONTENTS).hexdigest())) assert archive_path != installed_path assert archive_path.read_bytes() == payload assert installed_path.read_bytes() == CONTENTS assert not installed_path.name.lower().endswith((".gz", ".zip")) assert fetch_file(url) == str(archive_path) assert fetch_file(url, decompress=True) == str(installed_path) @pytest.mark.parametrize("explicit", ["filename", "destination"]) @pytest.mark.parametrize("extension", ["gz", "zip"]) @pytest.mark.parametrize("keep_suffix,decompress", [(True, False), (False, False), (True, True)]) def test_explicit_archive_output_with_query_string( tmp_path, monkeypatch, explicit, extension, keep_suffix, decompress): payload = archive_bytes(extension) serve(monkeypatch, payload) filename = "data." + extension if keep_suffix else "data" kwargs = {explicit: tmp_path / filename if explicit == "destination" else filename} expected = payload if keep_suffix and not decompress else CONTENTS path = fetch_file( "https://host/data." + extension + "?token=abc", decompress=decompress, expected_sha256=hashlib.sha256(expected).hexdigest(), **kwargs) assert Path(path).read_bytes() == expected @pytest.mark.parametrize("source_name", ["notgz", "notzip", "nothtml", "data.txt?format=.gz"]) def test_transform_detection_uses_actual_extensions(tmp_path, monkeypatch, source_name): serve(monkeypatch, CONTENTS) destination = tmp_path / "data.csv" fetch_file("https://host/" + source_name, destination=destination) assert destination.read_bytes() == CONTENTS @pytest.mark.parametrize("decoration", ["?filename=table.csv", "#table.csv", "?token=abc#table.csv"]) @pytest.mark.parametrize("decompress", [False, True]) def test_inferred_html_is_not_converted_from_query_text(monkeypatch, decoration, decompress): url = "https://host/table.html" + decoration html = b"
private data
" serve(monkeypatch, html) def reject_parser(*args, **kwargs): pytest.fail("inferred HTML must not require an HTML parser") monkeypatch.setattr(download.pd, "read_html", reject_parser) kwargs = dict(decompress=decompress, expected_sha256=hashlib.sha256(html).hexdigest()) path = Path(fetch_file(url, **kwargs)) assert path.name == common.build_local_filename(url, decompress=decompress) assert path.read_bytes() == html assert fetch_file(url, **kwargs) == str(path) assert fetch_file(url, force=True, **kwargs) == str(path) assert path.read_bytes() == html @pytest.mark.parametrize("output_argument", ["filename", "destination"]) def test_explicit_csv_conversion_ignores_query_text(tmp_path, monkeypatch, output_argument): html = b"
fixture
" frame = pd.DataFrame({"value": [1]}) expected = frame.to_csv(index=False).encode() serve(monkeypatch, html) monkeypatch.setattr(download.pd, "read_html", lambda *a, **kw: [frame]) output = tmp_path / "table.csv" if output_argument == "destination" else "table.csv" path = fetch_file( "https://host/table.html?filename=source.html#raw", **{output_argument: output}, expected_sha256=hashlib.sha256(expected).hexdigest()) assert Path(path).read_bytes() == expected @pytest.fixture def output_source(monkeypatch): def prepare(kind): if kind == "html": frame = pd.DataFrame({"value": [1]}) monkeypatch.setattr(download.pd, "read_html", lambda *a, **kw: [frame]) serve(monkeypatch, b"fixture
") return "https://host/table.html", frame.to_csv(index=False).encode() serve(monkeypatch, CONTENTS if kind == "raw" else archive_bytes(kind)) return "https://host/data." + kind, CONTENTS return prepare @pytest.mark.skipif(os.name != "posix", reason="POSIX file-mode semantics") @pytest.mark.parametrize("kind", ["raw", "gz", "zip", "html"]) @pytest.mark.parametrize("mode", [0o600, 0o640, 0o664, 0o440]) def test_refresh_preserves_existing_permissions(tmp_path, output_source, kind, mode): url, expected = output_source(kind) destination = tmp_path / "data.csv" destination.write_bytes(b"old contents") destination.chmod(mode) fetch_file(url, destination=destination, force=True) assert destination.read_bytes() == expected assert stat.S_IMODE(destination.stat().st_mode) == mode @pytest.mark.skipif(os.name != "posix", reason="POSIX umask semantics") @pytest.mark.parametrize("creation_mask", [0o022, 0o002, 0o027, 0o077]) @pytest.mark.parametrize("kind", ["raw", "gz", "zip", "html"]) @pytest.mark.parametrize("entry_point", ["public", "private"]) def test_new_files_use_normal_creation_permissions( tmp_path, monkeypatch, output_source, creation_mask, kind, entry_point): url, expected = output_source(kind) destination = tmp_path / "data.csv" previous_mask = os.umask(creation_mask) try: # Implementation must not read/reset the process-wide umask, even briefly. with monkeypatch.context() as guard: def reject_umask(*args): pytest.fail("download changed the process umask") guard.setattr(download.os, "umask", reject_umask) if entry_point == "public": fetch_file(url, destination=destination) else: # pyensembl downloads GTF and FASTA files through this helper. download._download_and_decompress_if_necessary(str(destination), url) finally: os.umask(previous_mask) assert destination.read_bytes() == expected assert stat.S_IMODE(destination.stat().st_mode) == 0o666 & ~creation_mask assert list(tmp_path.iterdir()) == [destination] @pytest.mark.skipif(os.name != "posix", reason="POSIX file-mode semantics") @pytest.mark.parametrize("existing_mode", [None, 0o600, 0o640, 0o644]) @pytest.mark.parametrize("failure", [None, "conversion", "validation"]) def test_csv_contents_remain_private_until_validated( tmp_path, monkeypatch, output_source, existing_mode, failure): url, expected = output_source("html") destination = tmp_path / "private.csv" if existing_mode is not None: destination.write_bytes(b"old private data") destination.chmod(existing_mode) original_csv = pd.DataFrame.to_csv original_validate = download.validate_file original_replace = os.replace observations = [] def check_private(path): assert stat.S_IMODE(Path(path).stat().st_mode) & 0o077 == 0 for sibling in tmp_path.glob(".datacache-*"): assert stat.S_IMODE(sibling.stat().st_mode) & 0o077 == 0 def checked_csv(frame, path, **kwargs): check_private(path) Path(path).write_bytes(b"partial private data") check_private(path) if failure == "conversion": raise RuntimeError("injected conversion failure") result = original_csv(frame, path, **kwargs) check_private(path) observations.append("converted") return result def checked_validate(path, *args, **kwargs): check_private(path) assert Path(path).read_bytes() == expected if failure == "validation": raise RuntimeError("injected validation failure") result = original_validate(path, *args, **kwargs) observations.append("validated") return result def checked_replace(source, target): assert observations == ["converted", "validated"] assert Path(source).read_bytes() == expected final_mode = 0o644 if existing_mode is None else existing_mode assert stat.S_IMODE(Path(source).stat().st_mode) == final_mode return original_replace(source, target) monkeypatch.setattr(pd.DataFrame, "to_csv", checked_csv) monkeypatch.setattr(download, "validate_file", checked_validate) monkeypatch.setattr(download.os, "replace", checked_replace) previous_mask = os.umask(0o022) try: if failure is None: fetch_file(url, destination=destination, force=True) assert destination.read_bytes() == expected else: with pytest.raises(RuntimeError, match="injected"): fetch_file(url, destination=destination, force=True) if existing_mode is not None: assert destination.read_bytes() == b"old private data" assert stat.S_IMODE(destination.stat().st_mode) == existing_mode else: assert not destination.exists() finally: os.umask(previous_mask) assert not list(tmp_path.glob(".datacache-*")) @pytest.mark.parametrize("failed", [False, True]) @pytest.mark.parametrize("kind", ["raw", "gz", "zip", "html"]) def test_creation_mode_probe_never_contains_data(tmp_path, monkeypatch, output_source, failed, kind): url, expected = output_source(kind) destination = tmp_path / "data.csv" original_open = download._open_staging_file original_fstat = os.fstat readers = [] probe_fd = None def observed_open(*args, **kwargs): nonlocal probe_fd result = original_open(*args, **kwargs) if kwargs.get("prefix") == ".datacache-mode-": probe_fd = result.fileno() # Model a reader that opened the publicly readable empty file and # retains its descriptor even after it has been unlinked. readers.append(open(result.name, "rb")) assert readers[-1].read() == b"" return result def checked_fstat(fd): if failed and fd == probe_fd: raise OSError("injected mode-probe failure") return original_fstat(fd) monkeypatch.setattr(download, "_open_staging_file", observed_open) monkeypatch.setattr(download.os, "fstat", checked_fstat) try: if failed: with pytest.raises(OSError, match="mode-probe failure"): fetch_file(url, destination=destination) assert not destination.exists() else: fetch_file(url, destination=destination) assert destination.read_bytes() == expected assert len(readers) == 1 assert readers[0].read() == b"" assert not list(tmp_path.glob(".datacache-*")) finally: for reader in readers: reader.close() @pytest.mark.parametrize("kind", ["raw", "gz", "zip", "html"]) @pytest.mark.parametrize("failure", ["chmod", "replace"]) @pytest.mark.parametrize("existing", [False, True]) def test_permission_or_publication_failure_preserves_file_and_mode( tmp_path, monkeypatch, output_source, kind, failure, existing): url, expected = output_source(kind) destination = tmp_path / "data.csv" if existing: destination.write_bytes(b"old complete file") destination.chmod(0o640) original_mode = stat.S_IMODE(destination.stat().st_mode) def fail(path, *args): assert Path(path).read_bytes() == expected if failure == "replace" and existing: assert stat.S_IMODE(Path(path).stat().st_mode) == original_mode raise PermissionError("injected " + failure + " failure") monkeypatch.setattr(download.os, failure, fail) with pytest.raises(PermissionError, match="injected"): fetch_file(url, destination=destination, force=True) if existing: assert destination.read_bytes() == b"old complete file" assert stat.S_IMODE(destination.stat().st_mode) == original_mode assert list(tmp_path.iterdir()) == [destination] else: assert list(tmp_path.iterdir()) == [] @pytest.mark.skipif(os.name != "posix", reason="POSIX file-mode semantics") @pytest.mark.parametrize("kind", ["raw", "gz", "zip"]) @pytest.mark.parametrize("failed", [False, True]) def test_binary_staging_remains_private_until_validated(tmp_path, monkeypatch, output_source, kind, failed): url, expected = output_source(kind) destination = tmp_path / "data" original_stream = download._stream_to_file original_copy = download.copyfileobj original_validate = download.validate_file original_replace = os.replace observations = [] def check_private(): assert not destination.exists() for sibling in tmp_path.glob(".datacache-*"): assert stat.S_IMODE(sibling.stat().st_mode) & 0o077 == 0 def checked_stream(*args, **kwargs): check_private() result = original_stream(*args, **kwargs) check_private() observations.append("downloaded") return result def checked_copy(*args, **kwargs): check_private() result = original_copy(*args, **kwargs) check_private() observations.append("decompressed") return result def checked_validate(path, *args, **kwargs): check_private() assert Path(path).read_bytes() == expected result = original_validate(path, *args, **kwargs) observations.append("validated") return result def checked_replace(source, target): assert observations[-1] == "validated" assert Path(source).read_bytes() == expected assert stat.S_IMODE(Path(source).stat().st_mode) == 0o644 return original_replace(source, target) monkeypatch.setattr(download, "_stream_to_file", checked_stream) monkeypatch.setattr(download, "copyfileobj", checked_copy) monkeypatch.setattr(download, "validate_file", checked_validate) monkeypatch.setattr(download.os, "replace", checked_replace) previous_mask = os.umask(0o022) try: kwargs = dict(destination=destination, force=True, expected_sha256="0" * 64 if failed else hashlib.sha256(expected).hexdigest()) if failed: with pytest.raises(download.FileValidationError, match="SHA-256 mismatch"): fetch_file(url, **kwargs) assert not destination.exists() else: fetch_file(url, **kwargs) assert destination.read_bytes() == expected finally: os.umask(previous_mask) assert observations == (["downloaded"] + ([] if kind == "raw" else ["decompressed"]) + ([] if failed else ["validated"])) assert not list(tmp_path.glob(".datacache-*")) @pytest.mark.parametrize("exhausted", [False, True]) def test_staging_collisions_do_not_overwrite_unrelated_files(tmp_path, monkeypatch, exhausted): first, second = UUID(int=1), UUID(int=2) collision = tmp_path / (".datacache-download-" + first.hex + ".tmp") collision.write_bytes(b"unrelated file") candidates = iter([first, second, UUID(int=3)]) monkeypatch.setattr(download, "uuid4", lambda: first if exhausted else next(candidates)) serve(monkeypatch, CONTENTS) destination = tmp_path / "data" if exhausted: with pytest.raises(FileExistsError, match="unique staging file"): fetch_file("https://host/data", destination=destination) assert list(tmp_path.iterdir()) == [collision] else: fetch_file("https://host/data", destination=destination) assert destination.read_bytes() == CONTENTS assert set(tmp_path.iterdir()) == {collision, destination} assert collision.read_bytes() == b"unrelated file" ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1792090401.71576 datacache-1.27.1/tests/test_downstream_compatibility.py0000644000076500000000000001116015264220442023104 0ustar00iskanderwheel"""Offline fixtures matching private/public call patterns in downstream packages.""" import gzip import io import zipfile import pytest from datacache import Cache, build_local_filename, download DATA = b">reference\nACGT\n" def archive_bytes(suffix): if suffix == "gz": return gzip.compress(DATA) output = io.BytesIO() with zipfile.ZipFile(output, "w") as archive: archive.writestr("reference.fa", DATA) return output.getvalue() @pytest.mark.parametrize("suffix", ["gz", "zip"]) @pytest.mark.parametrize("decoration", ["", "?token=abc", "#download", "?token=abc#download"]) @pytest.mark.parametrize("decompress_on_download", [False, True]) def test_pyensembl_private_download_contract(tmp_path, monkeypatch, suffix, decoration, decompress_on_download): url = "https://host/reference.fa." + suffix + decoration # pyensembl passes the remote basename explicitly and strips a trailing # compression suffix itself, before calling the helper without flags. filename = build_local_filename(url, filename=url.rsplit("/", 1)[-1]) if decompress_on_download and filename.endswith("." + suffix): filename = filename[:-(len(suffix) + 1)] destination = tmp_path / filename payload = archive_bytes(suffix) def stream(url, output, **kwargs): output.write(payload) monkeypatch.setattr(download, "_stream_to_file", stream) expected = DATA if decompress_on_download and not decoration else payload for previous in (None, b"previous artifact"): if previous is not None: destination.write_bytes(previous) download._download_and_decompress_if_necessary(str(destination), url, timeout=3600) assert destination.read_bytes() == expected assert list(tmp_path.iterdir()) == [destination] @pytest.mark.parametrize("url", ["https://host/table.html?name=table.csv", "https://host/table.htm#table.csv"]) def test_private_helper_preserves_legacy_html_with_query_or_fragment(tmp_path, monkeypatch, url): payload = b"original HTML
" destination = tmp_path / "table.csv" monkeypatch.setattr(download, "_stream_to_file", lambda url, output, **kw: output.write(payload)) def reject_parser(*args, **kwargs): pytest.fail("legacy query/fragment URL must retain HTML") monkeypatch.setattr(download.pd, "read_html", reject_parser) download._download_and_decompress_if_necessary(str(destination), url) assert destination.read_bytes() == payload @pytest.mark.parametrize("decompress", [False, True]) def test_private_helper_explicit_decompression_overrides_legacy_inference(tmp_path, monkeypatch, decompress): payload = gzip.compress(DATA) destination = tmp_path / "reference.gz_token_abc" monkeypatch.setattr(download, "_stream_to_file", lambda url, output, **kw: output.write(payload)) download._download_and_decompress_if_necessary( destination, "https://host/reference.gz?token=abc", decompress=decompress) assert destination.read_bytes() == (DATA if decompress else payload) @pytest.mark.parametrize("suffix", ["gz", "zip"]) @pytest.mark.parametrize("decompress", [False, True]) def test_pepdata_download_endpoint_contract(tmp_path, monkeypatch, suffix, decompress): # IEDB's real URLs name the archive in a query, while the path ends in .php. url = "https://www.iedb.org/downloader.php?file_name=doc/records." + suffix archive = archive_bytes(suffix) calls = [] def stream(url, output, **kwargs): calls.append(url) output.write(archive) monkeypatch.setattr(download, "_stream_to_file", stream) cache = Cache("pepdata", cache_root=tmp_path) for force in (False, False, True): path = cache.fetch(url, filename="records.csv", decompress=decompress, force=force) with open(path, "rb") as source: assert source.read() == DATA assert len(calls) == 2 @pytest.mark.parametrize("suffix", ["gz", "zip"]) @pytest.mark.parametrize("template", ["?file=records.{}", "?file=records.{}#download", "?file=records%2E{}"]) def test_endpoint_default_retention_and_explicit_decompression(tmp_path, monkeypatch, suffix, template): url = "https://host/download" + template.format(suffix) archive = archive_bytes(suffix) monkeypatch.setattr(download, "_stream_to_file", lambda url, output, **kw: output.write(archive)) retained = download.fetch_file(url, cache_root=tmp_path) expanded = download.fetch_file(url, decompress=True, cache_root=tmp_path) assert retained != expanded with open(retained, "rb") as source: assert source.read() == archive with open(expanded, "rb") as source: assert source.read() == DATA ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7158706 datacache-1.27.1/tests/test_fetch_freshness.py0000644000076500000000000003247315264220442021153 0ustar00iskanderwheel"""fetch_bytes, and fetch_file's expire_after, return_stale_on_error and validator.""" from datetime import datetime, timedelta, timezone from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import logging import os import threading import pytest import requests from datacache import Cache, FileValidationError, fetch_bytes, fetch_file from datacache import download, provenance @pytest.fixture(autouse=True) def waits(monkeypatch): delays = [] monkeypatch.setattr(download.time, "sleep", delays.append) monkeypatch.setenv("NO_PROXY", "127.0.0.1,localhost") return delays @pytest.fixture def server(): """A local server answering each GET with the next of its responses (the last repeats); a response is a body, or a (status, body) pair.""" class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def log_message(self, *args): pass def do_GET(self): index = len(self.server.requests) self.server.requests.append(self.path) responses = self.server.responses response = responses[min(index, len(responses) - 1)] status, body = response if isinstance(response, tuple) else (200, response) self.send_response(status) self.send_header("Content-Length", str(len(body))) self.send_header("Connection", "close") self.end_headers() self.wfile.write(body) self.close_connection = True httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler) httpd.requests = [] httpd.responses = [b"first\n"] httpd.url = "http://127.0.0.1:%d/listing.html" % httpd.server_port worker = threading.Thread(target=httpd.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True) worker.start() yield httpd httpd.shutdown() httpd.server_close() worker.join(timeout=5) def age(path, seconds): """Make path look fetched `seconds` ago by its modification time.""" past = datetime.now().timestamp() - seconds os.utime(path, (past, past)) # fetch_bytes def test_fetch_bytes_returns_the_body_without_writing(server, tmp_path, monkeypatch): monkeypatch.chdir(tmp_path) assert fetch_bytes(server.url, timeout=5) == b"first\n" assert list(tmp_path.iterdir()) == [] def test_fetch_bytes_retries_like_downloads(server, waits): server.responses = [(503, b"busy"), (429, b"slow down"), b"done"] assert fetch_bytes(server.url) == b"done" assert len(server.requests) == 3 and len(waits) == 2 def test_fetch_bytes_does_not_retry_permanent_errors(server): server.responses = [(404, b"missing")] with pytest.raises(requests.HTTPError): fetch_bytes(server.url) assert len(server.requests) == 1 def test_fetch_bytes_rejects_empty_bodies_unless_allowed(server): server.responses = [b""] with pytest.raises(ValueError, match="returned no bytes"): fetch_bytes(server.url) assert len(server.requests) == 3 # Retried as transient first. assert fetch_bytes(server.url, allow_empty=True) == b"" def test_fetch_bytes_reads_file_urls(tmp_path): source = tmp_path / "local.txt" source.write_bytes(b"local\n") assert fetch_bytes(source.as_uri()) == b"local\n" @pytest.mark.parametrize("options", [ {"max_retries": -1}, {"allow_empty": "yes"}, {"retry_backoff": float("nan")}, ]) def test_fetch_bytes_validates_options(server, options): with pytest.raises(ValueError): fetch_bytes(server.url, **options) assert server.requests == [] # expire_after def test_fresh_copy_is_reused_without_a_request(server, tmp_path): path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True) server.responses = [b"second\n"] assert fetch_file(server.url, destination=path, raw=True, expire_after=3600) == path assert len(server.requests) == 1 @pytest.mark.parametrize("expire_after", [60, timedelta(minutes=1), 0]) def test_expired_copy_is_fetched_again(server, tmp_path, expire_after): path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True) age(path, 120) server.responses = [b"second\n"] fetch_file(server.url, destination=path, raw=True, expire_after=expire_after) assert open(path, "rb").read() == b"second\n" def test_age_comes_from_the_provenance_record(server, tmp_path): path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True, record_provenance=True) # The file is new by its modification time, but the record says it was # fetched two hours ago. record_path = provenance.sidecar_path(path) record = json.loads(open(record_path).read()) record["fetched_at"] = (datetime.now(timezone.utc) - timedelta(hours=2)).isoformat() with open(record_path, "w") as handle: json.dump(record, handle) server.responses = [b"second\n"] fetch_file(server.url, destination=path, raw=True, expire_after=3600) assert open(path, "rb").read() == b"second\n" @pytest.mark.parametrize("expire_after", [-1, True, float("inf"), "60"]) def test_invalid_expiry_is_rejected(server, tmp_path, expire_after): with pytest.raises(ValueError, match="expire_after"): fetch_file(server.url, destination=tmp_path / "x", expire_after=expire_after) assert server.requests == [] # return_stale_on_error def test_failed_refresh_returns_the_cached_copy(server, tmp_path, caplog): path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True) age(path, 120) server.responses = [(503, b"down")] with caplog.at_level(logging.WARNING, logger="datacache"): assert fetch_file(server.url, destination=path, raw=True, expire_after=60, return_stale_on_error=True) == path assert open(path, "rb").read() == b"first\n" assert "using the cached copy" in caplog.text def test_forced_refresh_can_fall_back_too(server, tmp_path): path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True) server.responses = [(500, b"error")] assert fetch_file(server.url, destination=path, raw=True, force=True, return_stale_on_error=True) == path def test_errors_propagate_without_a_cached_copy_or_the_option(server, tmp_path): server.responses = [(503, b"down")] with pytest.raises(requests.HTTPError): fetch_file(server.url, destination=tmp_path / "missing.html", raw=True, return_stale_on_error=True) server.responses = [b"first\n"] path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True) age(path, 120) server.responses = [(503, b"down")] with pytest.raises(requests.HTTPError): fetch_file(server.url, destination=path, raw=True, expire_after=60) assert open(path, "rb").read() == b"first\n" # validator def has_first(path): if b"first" not in open(path, "rb").read(): raise ValueError("not a listing") def test_validator_keeps_wrong_content_out_of_the_cache(server, tmp_path): server.responses = [b"proxy error page"] destination = tmp_path / "listing.html" with pytest.raises(FileValidationError, match="failed validation: not a listing") as raised: fetch_file(server.url, destination=destination, raw=True, validator=has_first) assert isinstance(raised.value.__cause__, ValueError) assert not destination.exists() assert sorted(p.name for p in tmp_path.iterdir()) == [] # No staging files left. def test_validator_returning_false_rejects(server, tmp_path): with pytest.raises(FileValidationError, match="failed validation"): fetch_file(server.url, destination=tmp_path / "x", raw=True, validator=lambda path: False) def test_rejected_refresh_keeps_and_can_return_the_cached_copy(server, tmp_path): path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True, validator=has_first) server.responses = [b"proxy error page"] with pytest.raises(FileValidationError): fetch_file(server.url, destination=path, raw=True, force=True, validator=has_first) assert fetch_file(server.url, destination=path, raw=True, force=True, validator=has_first, return_stale_on_error=True) == path assert open(path, "rb").read() == b"first\n" def test_validator_checks_cache_hits(server, tmp_path): destination = tmp_path / "listing.html" destination.write_bytes(b"something else") with pytest.raises(FileValidationError, match="cached file failed validation.*force=True"): fetch_file(server.url, destination=destination, raw=True, validator=has_first) assert server.requests == [] def test_validator_is_not_combined_with_resume(server, tmp_path): with pytest.raises(ValueError, match="validator"): fetch_file(server.url, destination=tmp_path / "x", raw=True, resume=True, expected_size=6, validator=has_first) def test_cache_fetch_forwards_the_options(server, tmp_path): cache = Cache("freshness", cache_root=tmp_path) path = cache.fetch(server.url, filename="listing.html", raw=True, validator=has_first) age(path, 120) server.responses = [(503, b"down")] assert cache.fetch(server.url, filename="listing.html", raw=True, expire_after=60, return_stale_on_error=True, validator=has_first) == path assert len(server.requests) == 4 # One download, then three refresh attempts. # Review fixes def test_expired_resumable_download_is_forced(server, tmp_path, monkeypatch): from datacache import resume path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True) age(path, 120) calls = [] monkeypatch.setattr(resume, "download_resumable", lambda *args, **kwargs: calls.append(kwargs["force"])) fetch_file(server.url, destination=path, raw=True, resume=True, expected_size=os.path.getsize(path), expire_after=60) assert calls == [True] def test_fallback_warning_does_not_log_url_secrets(server, tmp_path, caplog): url = server.url + "?token=SECRET" path = fetch_file(url, destination=tmp_path / "listing.html", raw=True) server.responses = [(503, b"down")] with caplog.at_level(logging.WARNING, logger="datacache"): fetch_file(url, destination=path, raw=True, force=True, return_stale_on_error=True) assert "using the cached copy" in caplog.text and "SECRET" not in caplog.text def test_cancelling_progress_callback_is_not_a_fallback(server, tmp_path): class Cancelled(Exception): pass def cancel(done, total): raise Cancelled() path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True) with pytest.raises(Cancelled): fetch_file(server.url, destination=path, raw=True, force=True, return_stale_on_error=True, progress_callback=cancel) def test_forced_refresh_checks_the_cached_copy_only_on_failure(server, tmp_path): path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True) checked = [] def validator(path): checked.append(path) fetch_file(server.url, destination=path, raw=True, force=True, return_stale_on_error=True, validator=validator) assert len(checked) == 1 # The new download only. def test_zero_expiry_refreshes_even_a_future_dated_file(server, tmp_path): path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True) age(path, -3600) # Modified an hour in the future. server.responses = [b"second\n"] fetch_file(server.url, destination=path, raw=True, expire_after=0) assert open(path, "rb").read() == b"second\n" age(path, -3600) server.responses = [b"third\n"] fetch_file(server.url, destination=path, raw=True, expire_after=60) # Counts as new. assert open(path, "rb").read() == b"second\n" def test_expiring_cache_replaces_content_that_no_longer_validates(server, tmp_path): destination = tmp_path / "listing.html" destination.write_bytes(b"old error page") path = fetch_file(server.url, destination=destination, raw=True, expire_after=3600, validator=has_first) assert open(path, "rb").read() == b"first\n" destination.write_bytes(b"old error page") server.responses = [(503, b"down")] with pytest.raises(requests.HTTPError): # Invalid content is no fallback. fetch_file(server.url, destination=destination, raw=True, expire_after=3600, return_stale_on_error=True, validator=has_first) def test_rejection_keeps_its_cause_without_a_cached_copy(server, tmp_path): server.responses = [b"proxy error page"] with pytest.raises(FileValidationError) as raised: fetch_file(server.url, destination=tmp_path / "listing.html", raw=True, return_stale_on_error=True, validator=has_first) assert isinstance(raised.value.__cause__, ValueError) def test_timezone_free_fetch_time_is_utc(server, tmp_path, monkeypatch): path = fetch_file(server.url, destination=tmp_path / "listing.html", raw=True, record_provenance=True) record_path = provenance.sidecar_path(path) record = json.loads(open(record_path).read()) two_hours_ago = datetime.now(timezone.utc) - timedelta(hours=2) record["fetched_at"] = two_hours_ago.replace(tzinfo=None).isoformat() with open(record_path, "w") as handle: json.dump(record, handle) assert 7100 < download._age_seconds(path) < 7300 ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792090401.715981 datacache-1.27.1/tests/test_file_registry.py0000644000076500000000000002154315264220442020645 0ustar00iskanderwheel"""Released hitlist contracts for the shared fixed-path registry.""" import gzip import hashlib import json import multiprocessing from pathlib import Path import threading import pytest from datacache import VersionedFileRegistry from datacache import file_registry @pytest.fixture def registry(tmp_path): source = tmp_path / 'source' source.write_bytes(b'original') mapping = {'thing': { 'filename': 'thing.tsv', 'urls': {'v1': source.as_uri(), 'v2': source.as_uri()}, 'default_version': 'v2', 'description': 'reference', }} return VersionedFileRegistry(mapping, cache_dir=lambda: tmp_path / 'cache'), source def test_missing_lookup_and_status_are_read_only(registry, tmp_path): reg, _ = registry assert reg.resolve_version('thing') == 'v2' assert reg.resolve_version('thing', 'v1') == 'v1' assert reg.local_path('thing') == tmp_path / 'cache/thing/v2/thing.tsv' assert not reg.is_cached('thing') assert reg.status() == [{ 'name': 'thing', 'description': 'reference', 'default_version': 'v2', 'available_versions': ['v1', 'v2'], 'cached': False, 'cached_version': None, 'url': None, 'bytes': None, 'sha256': None, 'downloaded_at': None, 'path': str(reg.local_path('thing')), }] assert not (tmp_path / 'cache').exists() def test_download_receipt_and_multiple_versions(registry, tmp_path): reg, source = registry first = reg.download('thing', 'v1', timeout=2, record_provenance=True) assert isinstance(first, Path) assert first == tmp_path / 'cache/thing/v1/thing.tsv' assert first.read_bytes() == b'original' manifest_path = tmp_path / 'cache/manifest.json' receipt = json.loads(manifest_path.read_text())['thing'] assert receipt['version'] == 'v1' assert receipt['path'] == str(first) assert receipt['url'] == source.as_uri() assert receipt['bytes'] == 8 assert receipt['sha256'] == hashlib.sha256(b'original').hexdigest() assert receipt['downloaded_at'].endswith('+00:00') second = reg.ensure('thing') assert first.exists() and second.exists() assert first != second status = reg.status()[0] assert status['cached_version'] == 'v2' assert status['url'] == source.as_uri() assert status['sha256'] == hashlib.sha256(b'original').hexdigest() assert not list(manifest_path.parent.glob('.datacache-json-*')) def test_legacy_reuse_is_offline_without_hash_or_manifest_rewrite(registry, tmp_path, monkeypatch): reg, source = registry path = reg.local_path('thing') path.parent.mkdir(parents=True) path.write_bytes(b'legacy file') manifest = tmp_path / 'cache/manifest.json' manifest.write_text(json.dumps({'thing': { 'version': 'v2', 'path': str(path), 'bytes': 11, 'downloaded_at': '2020-01-01', }})) before = {p: (p.read_bytes(), p.stat().st_mtime_ns) for p in (path, manifest)} source.unlink() monkeypatch.setattr(file_registry, 'fetch_file', lambda *a, **k: pytest.fail('fetch')) monkeypatch.setattr(file_registry.hashlib, 'sha256', lambda *a: pytest.fail('hash')) assert reg.download('thing') == path assert reg.ensure('thing') == path assert reg.is_cached('thing') assert reg.status()[0]['downloaded_at'] == '2020-01-01' assert before == {p: (p.read_bytes(), p.stat().st_mtime_ns) for p in (path, manifest)} def test_forced_failure_preserves_legacy_file_and_receipt(registry, tmp_path): reg, source = registry path = reg.download('thing') manifest = tmp_path / 'cache/manifest.json' before = manifest.read_bytes() source.unlink() with pytest.raises(RuntimeError, match='failed to download') as raised: reg.download('thing', force=True) assert raised.value.__cause__ is not None assert path.read_bytes() == b'original' assert manifest.read_bytes() == before def test_refresh_and_raw_archive_override(registry): reg, source = registry archive = source.with_suffix('.gz') archive.write_bytes(gzip.compress(b'expanded')) reg._datasets['thing']['urls']['v2'] = archive.as_uri() path = reg.download('thing') assert path.read_bytes() == b'expanded' assert reg.download('thing', force=True, raw=True).read_bytes() == archive.read_bytes() def test_callable_root_remains_dynamic(registry, tmp_path): reg, _ = registry selected = [tmp_path / 'first'] reg._cache_dir = lambda: str(selected[0]) first = reg.download('thing') selected[0] = tmp_path / 'second' second = reg.ensure('thing') assert first != second assert first.read_bytes() == second.read_bytes() def test_custom_errors_and_cause(registry, monkeypatch): reg, _ = registry class DomainError(RuntimeError): pass reg._error_cls = DomainError with pytest.raises(DomainError, match='unknown dataset'): reg.resolve_version('absent') with pytest.raises(DomainError, match='available: v1, v2'): reg.resolve_version('thing', 'v3') error = OSError('offline') def fail(*a, **k): raise error monkeypatch.setattr(file_registry, 'fetch_file', fail) with pytest.raises(DomainError) as raised: reg.download('thing') assert raised.value.__cause__ is error @pytest.mark.parametrize('method', ['download', 'ensure']) def test_explicit_integrity_expectations_are_not_ignored_on_reuse(registry, method): reg, _ = registry path = reg.download('thing') with pytest.raises(RuntimeError, match='size'): getattr(reg, method)('thing', expected_size=99) assert path.read_bytes() == b'original' def test_failed_manifest_publication_preserves_prior_receipt(registry, tmp_path, monkeypatch): reg, source = registry reg.download('thing') manifest = tmp_path / 'cache/manifest.json' before = manifest.read_bytes() source.write_bytes(b'new file') import datacache._filesystem as filesystem replace = filesystem.os.replace def fail_manifest(src, dest): if Path(dest) == manifest: raise OSError('receipt publication failed') return replace(src, dest) monkeypatch.setattr(filesystem.os, 'replace', fail_manifest) with pytest.raises(OSError, match='receipt publication failed'): reg.download('thing', force=True) assert manifest.read_bytes() == before # Legacy single-file contract: file publication and receipt are separate. assert reg.local_path('thing').read_bytes() == b'new file' assert not list(manifest.parent.glob('.datacache-json-*')) def _download_in_process(root, source, name, start, read_barrier): """Widen the old manifest race deterministically, in separate interpreters.""" registry = VersionedFileRegistry({name: { 'filename': 'data', 'urls': {'v1': Path(source).as_uri()}, 'default_version': 'v1', }}, cache_dir=lambda: Path(root)) read = registry._read_manifest_at def overlapping_read(path): result = read(path) try: read_barrier.wait(timeout=0.5) except threading.BrokenBarrierError: pass # a correct writer lock prevents the other reader entering return result registry._read_manifest_at = overlapping_read start.wait(timeout=20) registry.download(name) def test_concurrent_processes_preserve_both_manifest_entries(tmp_path): ctx = multiprocessing.get_context('spawn') source = tmp_path / 'source' source.write_bytes(b'data') root = tmp_path / 'cache' start, read_barrier = ctx.Barrier(2), ctx.Barrier(2) processes = [ctx.Process(target=_download_in_process, args=(str(root), str(source), name, start, read_barrier)) for name in ('first', 'second')] try: for process in processes: process.start() for process in processes: process.join(timeout=30) assert [process.exitcode for process in processes] == [0, 0] finally: for process in processes: if process.is_alive(): process.terminate() process.join(timeout=5) manifest = json.loads((root / 'manifest.json').read_text()) assert set(manifest) == {'first', 'second'} assert all(Path(record['path']).read_bytes() == b'data' for record in manifest.values()) def test_download_resolves_dynamic_root_once(registry, tmp_path): reg, _ = registry calls = [] def changing_root(): calls.append(len(calls)) return tmp_path / str(len(calls)) reg._cache_dir = changing_root path = reg.download('thing') assert len(calls) == 1 receipt = json.loads((tmp_path / '1/manifest.json').read_text()) assert receipt['thing']['path'] == str(path) def test_atomic_receipt_supports_platform_without_fchmod(tmp_path, monkeypatch): import datacache._filesystem as filesystem monkeypatch.delattr(filesystem.os, 'fchmod') path = tmp_path / 'manifest.json' filesystem.write_json(path, {'data': 'receipt'}) assert json.loads(path.read_text()) == {'data': 'receipt'} ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7161167 datacache-1.27.1/tests/test_http_retries.py0000644000076500000000000004642415264220442020517 0ustar00iskanderwheel"""Exercise retries against a local HTTP server, with deterministic fake waits.""" from datetime import datetime, timezone from email.utils import format_datetime from fractions import Fraction import gzip import hashlib from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path import ssl import stat import threading from time import sleep as real_sleep import numpy as np import pytest import requests from urllib3 import exceptions as urllib3_errors from datacache import Cache, FileValidationError, fetch_file from datacache import download, retries PAYLOAD = b"complete downloaded data\n" @pytest.fixture(autouse=True) def waits(monkeypatch): delays = [] monkeypatch.setattr(download.time, "sleep", delays.append) monkeypatch.setenv("NO_PROXY", "127.0.0.1,localhost") return delays @pytest.fixture def http_server(): servers = [] def start(*responses): class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def log_message(self, *args): pass def do_GET(self): index = len(self.server.requests) self.server.requests.append(self.path) response = responses[min(index, len(responses) - 1)] body = response.get("body", PAYLOAD) self.send_response(response.get("status", 200)) self.send_header("Content-Length", str(len(body))) self.send_header("Connection", "close") for name, value in response.get("headers", {}).items(): self.send_header(name, value) self.end_headers() self.wfile.write(body[:response.get("cut_after", len(body))]) self.wfile.flush() self.close_connection = True server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) server.requests = [] server.url = "http://127.0.0.1:%d/file" % server.server_port worker = threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True) worker.start() servers.append((server, worker)) return server yield start for server, worker in servers: server.shutdown() server.server_close() worker.join(timeout=5) assert not worker.is_alive() @pytest.mark.parametrize("status", [408, 429, 500, 502, 503, 504]) def test_transient_http_error_then_success(http_server, tmp_path, waits, status, caplog): server = http_server({"status": status}, {}) destination = tmp_path / "data" fetch_file(server.url, destination=destination, timeout=2, expected_sha256=hashlib.sha256(PAYLOAD).hexdigest()) assert destination.read_bytes() == PAYLOAD assert len(server.requests) == 2 assert waits == [1] assert "attempt 1/3 failed (HTTP %d)" % status in caplog.text assert list(tmp_path.iterdir()) == [destination] @pytest.mark.parametrize("existing", [False, True]) def test_perpetual_504_is_bounded_and_preserves_destination(http_server, tmp_path, waits, existing, caplog): server = http_server({"status": 504}) destination = tmp_path / "data" if existing: destination.write_bytes(PAYLOAD) with pytest.raises(requests.HTTPError) as caught: fetch_file(server.url, destination=destination, force=True, expected_sha256=hashlib.sha256(PAYLOAD).hexdigest()) assert caught.value.response.status_code == 504 assert caught.value.request.url == server.url assert caught.value.response.raw.closed assert len(server.requests) == 3 assert waits == [1, 2] assert "failed after 3 attempt(s): HTTP 504" in caplog.text assert list(tmp_path.iterdir()) == ([destination] if existing else []) if existing: assert destination.read_bytes() == PAYLOAD def test_partial_stream_restarts_in_fresh_private_staging(http_server, tmp_path, monkeypatch, waits): server = http_server({"body": b"partial" * 100, "cut_after": 80}, {}) destination = tmp_path / "data" destination.write_bytes(b"previous verified artifact") real_open = download._open_staging_file real_get = download.requests.get paths, responses, progress = [], [], [] def open_staging(*args, **kwargs): assert destination.read_bytes() == b"previous verified artifact" assert list(tmp_path.iterdir()) == [destination] stream = real_open(*args, **kwargs) paths.append(stream.name) assert stat.S_IMODE(Path(stream.name).stat().st_mode) == 0o600 return stream def get(*args, **kwargs): assert kwargs["timeout"] == 2 response = real_get(*args, **kwargs) responses.append(response) return response monkeypatch.setattr(download, "_open_staging_file", open_staging) monkeypatch.setattr(download.requests, "get", get) fetch_file(server.url, destination=destination, force=True, chunk_size=8, timeout=2, expected_sha256=hashlib.sha256(PAYLOAD).hexdigest(), progress_callback=lambda done, total: progress.append((done, total))) assert len(server.requests) == len(set(paths)) == 2 assert all(response.raw.closed for response in responses) assert waits == [1] assert progress[0] == (8, 700) assert progress[-1] == (len(PAYLOAD), len(PAYLOAD)) assert (8, len(PAYLOAD)) in progress assert destination.read_bytes() == PAYLOAD assert list(tmp_path.iterdir()) == [destination] @pytest.mark.parametrize("header,expected", [ ("4", 4), ("0", 1), ("future-date", 4), ("past-date", 1), ("invalid", 1), ("-1", 1), ("1.5", 1), ("NaN", 1), ]) def test_retry_after_seconds_dates_and_invalid_values(http_server, tmp_path, monkeypatch, waits, header, expected): now = 1700000000 monkeypatch.setattr(retries.time, "time", lambda: now) if header in ("future-date", "past-date"): date = datetime.fromtimestamp(now + (4 if header == "future-date" else -4), tz=timezone.utc) header = format_datetime(date, usegmt=True) server = http_server({"status": 429, "headers": {"Retry-After": header}}, {}) fetch_file(server.url, destination=tmp_path / "data") assert waits == [expected] assert len(server.requests) == 2 @pytest.mark.parametrize("header", ["31", pytest.param("9" * 5000, id="huge-retry-after")]) def test_excessive_retry_after_stops_without_retrying_early(http_server, tmp_path, waits, header, caplog): server = http_server({"status": 503, "headers": {"Retry-After": header}}, {}) with pytest.raises(requests.HTTPError) as caught: fetch_file(server.url, destination=tmp_path / "data") assert caught.value.response.status_code == 503 assert len(server.requests) == 1 assert waits == [] assert list(tmp_path.iterdir()) == [] assert "Retry-After exceeds retry_max_delay" in caplog.text @pytest.mark.parametrize("retry_backoff, expected", [(2, [2, 3, 3, 3]), (10, [3, 3, 3, 3])]) def test_configurable_backoff_is_capped(http_server, tmp_path, waits, retry_backoff, expected): server = http_server(*([{"status": 504}] * 4), {}) fetch_file(server.url, destination=tmp_path / "data", max_retries=4, retry_backoff=retry_backoff, retry_max_delay=3) assert waits == expected assert len(server.requests) == 5 @pytest.mark.parametrize("number", [Fraction, np.float32], ids=["fraction", "numpy-float32"]) @pytest.mark.parametrize("option", ["retry_backoff", "retry_max_delay"]) def test_real_valued_delays_work_with_actual_sleep(http_server, tmp_path, monkeypatch, number, option): server = http_server({"status": 504}, {"status": 504}, {}) options = dict(retry_backoff=0.001, retry_max_delay=0.002) options[option] = number("0.001") delays = [] def sleep(delay): # Exercise the real sleep's numeric conversion for Fraction and # numpy.float32. Waits stay under 2ms. real_sleep(delay) delays.append(delay) monkeypatch.setattr(download.time, "sleep", sleep) fetch_file(server.url, destination=tmp_path / "data", **options) expected = [0.001, 0.002] if option == "retry_backoff" else [0.001, 0.001] assert delays == pytest.approx(expected) assert len(server.requests) == 3 assert (tmp_path / "data").read_bytes() == PAYLOAD @pytest.mark.parametrize("status,max_retries", [(504, 0), (401, 2), (403, 2), (404, 2), (410, 2), (501, 2)]) def test_disabled_retries_and_permanent_http_errors(http_server, tmp_path, waits, status, max_retries): server = http_server({"status": status}) with pytest.raises(requests.HTTPError): fetch_file(server.url, destination=tmp_path / "data", max_retries=max_retries) assert len(server.requests) == 1 assert waits == [] assert list(tmp_path.iterdir()) == [] @pytest.mark.parametrize("error_type", [requests.ConnectTimeout, requests.ReadTimeout, requests.ConnectionError]) def test_transport_failure_then_success(http_server, tmp_path, monkeypatch, waits, error_type): server = http_server({}) original_get = download.requests.get attempts = [] def get(*args, **kwargs): attempts.append(1) if len(attempts) == 1: raise error_type("temporary failure") return original_get(*args, **kwargs) monkeypatch.setattr(download.requests, "get", get) fetch_file(server.url, destination=tmp_path / "data") assert len(attempts) == 2 assert waits == [1] def test_exhaustion_preserves_original_transport_exception(tmp_path, monkeypatch, waits): error = requests.ConnectionError("original transport failure") attempts = [] def get(*args, **kwargs): attempts.append(1) raise error monkeypatch.setattr(download.requests, "get", get) with pytest.raises(requests.ConnectionError) as caught: fetch_file("https://host/file", destination=tmp_path / "data", max_retries=1) assert caught.value is error assert len(attempts) == 2 assert waits == [1] @pytest.mark.parametrize("error_type", [ requests.exceptions.SSLError, requests.exceptions.InvalidURL, requests.exceptions.ContentDecodingError, requests.TooManyRedirects, requests.HTTPError, ]) def test_non_transient_request_failures_are_not_retried(tmp_path, monkeypatch, waits, error_type): attempts = [] def get(*args, **kwargs): attempts.append(1) raise error_type("permanent failure") monkeypatch.setattr(download.requests, "get", get) with pytest.raises(error_type): fetch_file("https://host/file", destination=tmp_path / "data") assert len(attempts) == 1 assert waits == [] @pytest.mark.parametrize("cause_type", [ ssl.SSLCertVerificationError, urllib3_errors.SSLError, requests.exceptions.SSLError, ConnectionRefusedError, ]) def test_proxy_retry_policy_uses_wrapped_cause(tmp_path, monkeypatch, waits, cause_type): cause = cause_type("HTTPS proxy connection failed") # The actual Requests / urllib3 layout: args -> reason -> original_error. proxy = urllib3_errors.ProxyError("Unable to connect to proxy", cause) wrapped = urllib3_errors.MaxRetryError(None, "https://host/file", proxy) error = requests.exceptions.ProxyError(wrapped) attempts = [] destination = tmp_path / "data" destination.write_bytes(PAYLOAD) def get(*args, **kwargs): attempts.append(1) raise error monkeypatch.setattr(download.requests, "get", get) with pytest.raises(requests.exceptions.ProxyError) as caught: fetch_file("https://host/file", destination=destination, force=True) transient = cause_type is ConnectionRefusedError assert len(attempts) == (3 if transient else 1) assert waits == ([1, 2] if transient else []) assert caught.value is error assert destination.read_bytes() == PAYLOAD assert list(tmp_path.iterdir()) == [destination] @pytest.mark.parametrize("chain", ["cause", "context", "suppressed", "overridden", "cycle"]) def test_tls_cause_inspection_respects_exception_chaining(chain): error = requests.ConnectionError("outer transport failure") tls = ssl.SSLError("certificate verification failed") if chain == "cause": error.__cause__ = tls elif chain == "cycle": wrapped = requests.exceptions.ProxyError(error) error.__cause__ = wrapped else: error.__context__ = tls if chain == "suppressed": error.__suppress_context__ = True elif chain == "overridden": error.__cause__ = ConnectionRefusedError("active cause") assert retries.is_retryable_http_error(error) == (chain in ("suppressed", "overridden", "cycle")) @pytest.mark.parametrize("error_type", [requests.ConnectionError, RuntimeError, KeyboardInterrupt]) def test_callback_failures_are_not_transfer_retries(http_server, tmp_path, waits, error_type): server = http_server({}) error = error_type("callback failed") def callback(*args): raise error with pytest.raises(error_type) as caught: fetch_file(server.url, destination=tmp_path / "data", progress_callback=callback) assert caught.value is error assert len(server.requests) == 1 assert waits == [] assert list(tmp_path.iterdir()) == [] @pytest.mark.parametrize("failure", ["hash", "gzip", "publish"]) def test_validation_transform_and_publication_are_not_retried(http_server, tmp_path, monkeypatch, waits, failure): server = http_server({"body": gzip.compress(PAYLOAD)[:-8] if failure == "gzip" else PAYLOAD}) destination = tmp_path / "data" destination.write_bytes(b"previous verified artifact") kwargs = {} url = server.url if failure == "hash": kwargs["expected_sha256"] = "0" * 64 error_type = FileValidationError elif failure == "gzip": url += ".gz" error_type = EOFError else: error_type = PermissionError def denied(*args, **kwargs): raise PermissionError("publication denied") monkeypatch.setattr(download, "_publish_staged_file", denied) with pytest.raises(error_type): fetch_file(url, destination=destination, force=True, **kwargs) assert destination.read_bytes() == b"previous verified artifact" assert len(server.requests) == 1 assert waits == [] assert list(tmp_path.iterdir()) == [destination] @pytest.mark.parametrize("scheme", ["file", "ftp"]) def test_non_http_transfers_are_not_retried(tmp_path, monkeypatch, waits, scheme): attempts = [] def stream(*args, **kwargs): attempts.append(1) raise requests.ConnectionError("non-HTTP failure") monkeypatch.setattr(download, "_stream_to_file", stream) with pytest.raises(requests.ConnectionError): fetch_file(scheme + "://host/file", destination=tmp_path / "data") assert len(attempts) == 1 assert waits == [] @pytest.mark.parametrize("api", ["fetch", "cache", "private"]) def test_default_retries_reach_public_and_legacy_private_callers(http_server, tmp_path, waits, api): payload = gzip.compress(PAYLOAD) server = http_server({"status": 504}, {"body": payload}) url = server.url + ".gz" destination = tmp_path / "file.gz" if api == "fetch": fetch_file(url, destination=destination) elif api == "cache": Cache(cache_root=tmp_path).fetch(url, filename="file.gz") else: # Matches pyensembl.DownloadCache's call with no new flags. download._download_and_decompress_if_necessary(str(destination), url, timeout=3600) assert destination.read_bytes() == payload assert waits == [1] assert len(server.requests) == 2 @pytest.mark.parametrize("api", ["cache", "private"]) def test_retry_configuration_reaches_wrappers(http_server, tmp_path, waits, api): server = http_server({"status": 504}) options = dict(max_retries=1, retry_backoff=7, retry_max_delay=0) with pytest.raises(requests.HTTPError): if api == "cache": Cache(cache_root=tmp_path).fetch(server.url, filename="file", **options) else: download._download_and_decompress_if_necessary(tmp_path / "file", server.url, **options) assert len(server.requests) == 2 assert waits == [] def test_interruption_during_backoff_preserves_cache(http_server, tmp_path, monkeypatch): server = http_server({"status": 504}) destination = tmp_path / "file" destination.write_bytes(PAYLOAD) def interrupt(delay): assert list(tmp_path.iterdir()) == [destination] raise KeyboardInterrupt() monkeypatch.setattr(download.time, "sleep", interrupt) with pytest.raises(KeyboardInterrupt): fetch_file(server.url, destination=destination, force=True) assert len(server.requests) == 1 assert destination.read_bytes() == PAYLOAD assert list(tmp_path.iterdir()) == [destination] def test_verified_cache_reuse_never_requests_or_waits(tmp_path, monkeypatch): destination = tmp_path / "file" destination.write_bytes(PAYLOAD) def reject(*args, **kwargs): pytest.fail("cache reuse must not make a request or sleep") monkeypatch.setattr(download.requests, "get", reject) monkeypatch.setattr(download.time, "sleep", reject) assert fetch_file("https://host/file", destination=destination, expected_sha256=hashlib.sha256(PAYLOAD).hexdigest()) == str(destination) @pytest.mark.parametrize("options", [ {"max_retries": -1}, {"max_retries": True}, {"max_retries": 1.5}, {"retry_backoff": -1}, {"retry_backoff": True}, {"retry_backoff": "1"}, {"retry_backoff": float("nan")}, {"retry_backoff": 10 ** 1000}, {"retry_max_delay": -1}, {"retry_max_delay": float("inf")}, ]) def test_invalid_retry_options_fail_before_creating_cache(tmp_path, options): with pytest.raises(ValueError): fetch_file("https://host/file", destination=tmp_path / "missing" / "data", **options) assert list(tmp_path.iterdir()) == [] def test_clean_empty_response_is_retried_then_rejected(http_server, tmp_path, waits): # A withdrawn upstream record answers a clean 200 with no body. It must not # be published as a permanently valid cache entry. server = http_server({"body": b""}) destination = tmp_path / "withdrawn.fasta" with pytest.raises(FileValidationError, match="downloaded file is empty"): fetch_file(server.url, destination=destination, max_retries=2) assert len(server.requests) == 3 and len(waits) == 2 assert not destination.exists() assert not list(tmp_path.glob(".datacache-*")) def test_briefly_empty_server_recovers_on_retry(http_server, tmp_path): server = http_server({"body": b""}, {}) destination = tmp_path / "data" fetch_file(server.url, destination=destination, max_retries=1) assert destination.read_bytes() == PAYLOAD assert len(server.requests) == 2 def test_empty_response_leaves_an_existing_file_in_place(http_server, tmp_path): destination = tmp_path / "data" destination.write_bytes(b"previous release") server = http_server({"body": b""}) with pytest.raises(FileValidationError, match="empty"): fetch_file(server.url, destination=destination, force=True, max_retries=0) assert destination.read_bytes() == b"previous release" @pytest.mark.parametrize("consent", [{"allow_empty": True}, {"expected_size": 0}]) def test_an_expected_empty_response_is_published(http_server, tmp_path, consent): server = http_server({"body": b""}) destination = tmp_path / "empty" fetch_file(server.url, destination=destination, **consent) assert destination.read_bytes() == b"" assert len(server.requests) == 1 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7162135 datacache-1.27.1/tests/test_import_cost.py0000644000076500000000000000161715264220442020340 0ustar00iskanderwheel"""Keep ``import datacache`` cheap enough for command-line tools.""" import subprocess import sys import pytest from datacache import download def test_importing_datacache_loads_no_heavy_dependencies(): # Command-line tools resolve cache paths on every run; pandas alone costs # hundreds of milliseconds. Import in a fresh interpreter to measure it. loaded = subprocess.run( [sys.executable, "-c", "import sys, datacache; " "print(sorted(m for m in ('pandas', 'numpy', 'requests', 'urllib3', 'tqdm') if m in sys.modules))"], capture_output=True, text=True, check=True).stdout.strip() assert loaded == "[]" def test_download_module_still_exposes_pandas_and_requests(): import pandas import requests assert download.pd is pandas assert download.requests is requests with pytest.raises(AttributeError): download.not_an_attribute ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7163146 datacache-1.27.1/tests/test_legacy_cache_compatibility.py0000644000076500000000000002232115264220442023311 0ustar00iskanderwheel"""Keep caches generated by released datacache usable without rewriting them.""" from contextlib import closing import gzip import json import os from pathlib import Path import sqlite3 import stat import pandas as pd import pytest from datacache import ( Cache, build_local_filename, connect_if_correct_version, db_from_dataframe, db_from_dataframes_with_absolute_path, fetch_and_transform, fetch_csv_db, fetch_file, ) from datacache import database_helpers, database_table, download from datacache import common FIXTURES = Path(__file__).parent / "fixtures" NAMES = json.loads((FIXTURES / "legacy_v191_names.json").read_text()) def legacy_database(path): with closing(sqlite3.connect(path)) as connection: connection.executescript((FIXTURES / "legacy_v191.sql").read_text()) return path def reject(*args, **kwargs): pytest.fail("a legacy cache hit attempted a write, download, or rebuild") @pytest.mark.parametrize("api", ["dataframe", "absolute", "cache", "version", "csv"]) def test_legacy_database_reuse_is_read_only_and_keeps_schema(tmp_path, monkeypatch, api): path = legacy_database(tmp_path / "records.db") before = path.read_bytes() inode = path.stat().st_ino # Old builds ignored this unnormalized primary key. Reuse must not apply # new-build constraints or reconstruct a frame just to open the database. path.chmod(0o444) tmp_path.chmod(0o555) cache = Cache(cache_root=tmp_path) try: with monkeypatch.context() as guard: guard.setattr(database_helpers, "build_tables", reject) guard.setattr(database_helpers, "fetch_csv_dataframe", reject) guard.setattr(os, "makedirs", reject) guard.setattr(download, "_download_to_temp_file", reject) if api == "dataframe": connection = db_from_dataframe(path, "records", None, primary_key="sample id", version=7) elif api == "absolute": connection = db_from_dataframes_with_absolute_path(path, {"records": None}, version=7) elif api == "cache": connection = cache.db_from_dataframe("records.db", "records", None, version=7) elif api == "csv": connection = fetch_csv_db("records", "https://offline.invalid/source.csv", db_filename="records.db", version=7, download_options={"cache_root": tmp_path}) else: connection = connect_if_correct_version(path, 7, read_only=True) with closing(connection): assert connection.execute("SELECT * FROM records").fetchall() == [(1, "case"), (2, "control")] # Keep the old schema exactly, including its missing PK. assert not any(row[-1] for row in connection.execute("PRAGMA table_info(records)")) assert connection.execute("SELECT version FROM _datacache_metadata").fetchone() == (7,) assert path.read_bytes() == before assert path.stat().st_ino == inode assert stat.S_IMODE(path.stat().st_mode) == 0o444 assert list(tmp_path.iterdir()) == [path] finally: tmp_path.chmod(0o755) path.chmod(0o644) def test_legacy_blob_values_are_not_guessed_or_migrated(tmp_path): path = legacy_database(tmp_path / "records.db") blob = b"\x01\x00\x00\x00\x00\x00\x00\x00" with closing(sqlite3.connect(path)) as connection: connection.execute("UPDATE records SET sample_id = ? WHERE label = 'case'", (blob,)) connection.commit() before = path.read_bytes() with closing(db_from_dataframe(path, "records", None, version=7)) as connection: assert connection.execute("SELECT sample_id FROM records WHERE label = 'case'").fetchone() == (blob,) assert path.read_bytes() == before def test_legacy_csv_database_filename_is_preserved(tmp_path, monkeypatch): name = NAMES["csv_database_filename"] path = tmp_path / name with closing(sqlite3.connect(path)) as connection: connection.executescript("CREATE TABLE records (id INT, label TEXT);" "INSERT INTO records VALUES (42, 'legacy');" "CREATE TABLE _datacache_metadata (version INT);" "INSERT INTO _datacache_metadata VALUES (1);") # Source has the same schema/count used to generate the old key. Contents # differ so rebuilding or looking under a new key is observable. (tmp_path / "records.csv").write_text("id,label\n1,case\n2,control\n") before = path.read_bytes() monkeypatch.setattr(download, "_download_to_temp_file", reject) with closing(fetch_csv_db("records", "https://offline.invalid/records.csv.gz", csv_filename="records.csv.gz", download_options={"cache_root": tmp_path})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(42, "legacy")] assert path.read_bytes() == before assert list(tmp_path.glob("*.db")) == [path] @pytest.mark.parametrize("case", NAMES["filenames"]) def test_released_download_keys_and_private_files_are_reused(tmp_path, monkeypatch, case): assert build_local_filename(case["url"], decompress=case["decompress"]) == case["filename"] path = tmp_path / case["filename"] path.write_bytes(b"old installed bytes") path.chmod(0o600) before = path.stat() monkeypatch.setattr(download, "_download_to_temp_file", reject) monkeypatch.setattr(os, "chmod", reject) assert fetch_file(case["url"], decompress=case["decompress"], cache_root=tmp_path) == str(path) assert path.read_bytes() == b"old installed bytes" assert path.stat().st_ino == before.st_ino assert stat.S_IMODE(path.stat().st_mode) == 0o600 def test_existing_custom_transform_works_without_source(tmp_path, monkeypatch): path = tmp_path / "output.txt" path.write_text("old transformation") path.chmod(0o600) monkeypatch.setattr(download, "fetch_file", reject) assert fetch_and_transform("output.txt", reject, lambda p: Path(p).read_text(), "missing.txt", "https://offline.invalid/missing.txt", cache_root=tmp_path) == "old transformation" assert stat.S_IMODE(path.stat().st_mode) == 0o600 def test_build_does_not_require_new_pandas_scalar_or_row_apis(tmp_path, monkeypatch): # Also run this module on pandas <1.5 in CI. Feature removal catches # accidental reintroduction of newer APIs in the ordinary numeric path. frame = pd.DataFrame({"id": [2**53 + 1], "weight": [1.5], "missing": [float("nan")]}) monkeypatch.delattr(pd, "NA", raising=False) monkeypatch.delattr(pd, "isna", raising=False) monkeypatch.setattr(pd.DataFrame, "itertuples", reject) with closing(db_from_dataframe(tmp_path / "data.db", "records", frame)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(2**53 + 1, 1.5, None)] def test_new_database_is_readable_with_legacy_sql(tmp_path): path = tmp_path / "new.db" frame = pd.DataFrame({"sample id": [1, 2], "label": ["case", "control"]}) with closing(db_from_dataframe(path, "records", frame, primary_key="sample id", version=7)): pass with closing(sqlite3.connect(path)) as old_reader: assert old_reader.execute("SELECT version FROM _datacache_metadata").fetchone() == (7,) assert old_reader.execute("SELECT sample_id, label FROM records").fetchall() == [(1, "case"), (2, "control")] @pytest.mark.parametrize("new_source", [False, True]) def test_transform_reuses_misplaced_legacy_source_without_moving_it(tmp_path, monkeypatch, new_source): monkeypatch.setattr(common, "get_data_dir", lambda subdir=None: str(tmp_path / (subdir or "default"))) default = tmp_path / "default" default.mkdir() legacy = default / "source.txt" legacy.write_text("old source") legacy.chmod(0o600) before = legacy.stat() if new_source: current = tmp_path / "project" / "source.txt" current.parent.mkdir() current.write_text("new source") monkeypatch.setattr(download, "_download_to_temp_file", reject) def transform(source_path, output_path): text = Path(source_path).read_text() Path(output_path).write_text(text.upper()) return text.upper() result = fetch_and_transform("output.txt", transform, lambda p: Path(p).read_text(), "source.txt.gz", "https://offline.invalid/source.txt.gz", subdir="project") assert result == ("NEW SOURCE" if new_source else "OLD SOURCE") assert legacy.read_text() == "old source" assert legacy.stat().st_ino == before.st_ino assert stat.S_IMODE(legacy.stat().st_mode) == 0o600 def test_explicit_csv_database_still_honors_source_integrity(tmp_path): path = legacy_database(tmp_path / "records.db") source = tmp_path / "source.csv" source.write_text("sample id,label\n1,case\n2,control\n") before = path.read_bytes() with pytest.raises(download.FileValidationError, match="SHA-256"): fetch_csv_db("records", source.as_uri(), db_filename="records.db", version=7, download_options={"cache_root": tmp_path, "expected_sha256": "0" * 64}) assert path.read_bytes() == before ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7165012 datacache-1.27.1/tests/test_materialization.py0000644000076500000000000010762315264220442021176 0ustar00iskanderwheel"""Caller-built bundles, private retained inputs, and raw gzip dependencies.""" from concurrent.futures import ThreadPoolExecutor import gzip from hashlib import sha256 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import os from pathlib import Path import socket import subprocess import sys import threading import time import pytest from datacache import FileValidationError, inspect_bundle, inspect_materialization, materialize from datacache import bundle_store, download, materialization as module from datacache.resume import _state_directory pytestmark = pytest.mark.skipif(os.name != 'posix', reason='POSIX materialization locks') DATA = b'>chr1\nACGTACGT\n' COMPRESSED = gzip.compress(DATA, mtime=0) TRANSFORM = {'version': 'fasta-1', 'options': {'validate': True}} def metadata(data): return {'sha256': sha256(data).hexdigest(), 'size': len(data)} @pytest.fixture def sources(tmp_path): source = tmp_path / 'caller-owned.fa.gz' source.write_bytes(COMPRESSED) return {'dna.fa.gz': dict(path=source, **metadata(COMPRESSED))} @pytest.fixture def outputs(): return {'dna.fa': metadata(DATA), 'index/info.json': metadata(b'{"length":8}\n')} def build(sources, outputs): assert all(Path(path).parent.stat().st_mode & 0o077 == 0 for path in sources.values()) with gzip.open(sources['dna.fa.gz'], 'rb') as source, open(outputs['dna.fa'], 'wb') as target: while chunk := source.read(4): target.write(chunk) Path(outputs['index/info.json']).write_bytes(b'{"length":8}\n') def install(store, sources, outputs, **options): defaults = dict(transform=TRANSFORM, outputs=outputs, builder=build, download_options={'show_progress': False}) defaults.update(options) return materialize(store, sources, **defaults) def inspected(store, sources, outputs, **options): return inspect_materialization(store, sources, transform=TRANSFORM, outputs=outputs, **options) def input_dir(store): return next(store.glob('.inputs-*')) def test_transaction_receipt_cleanup_and_immutable_paths(tmp_path, sources, outputs): store = tmp_path / 'artifacts' / 'dna-110' paths = install(store, sources, outputs) assert Path(paths['dna.fa']).read_bytes() == DATA assert Path(paths['index/info.json']).read_bytes() == b'{"length":8}\n' state = inspected(store, sources, outputs) assert state.status == 'available' and state.verified assert all(file.verified for file in state.files.values()) assert state.transform == TRANSFORM record = state.sources['dna.fa.gz'] assert record['verified'] is True assert {key: record[key] for key in ('sha256', 'size')} == metadata(COMPRESSED) receipt = json.loads((Path(state.bundle) / module.MANIFEST).read_text()) assert receipt['definition']['transform'] == TRANSFORM assert set(receipt['outputs']) == set(outputs) assert Path(sources['dna.fa.gz']['path']).read_bytes() == COMPRESSED assert not list(store.glob('.inputs-*')) assert not list(store.glob('.staging-*')) assert inspect_bundle(store).status == 'invalid' # Distinct store ownership. assert not inspect_materialization(store).verified new_paths = install(store, sources, outputs, force=True) assert paths != new_paths assert all(Path(path).exists() for path in paths.values()) def forbidden(*args, **kwargs): raise AssertionError('offline cache hit tried to write, build, or acquire') def test_readonly_offline_hit_never_touches_sources(tmp_path, sources, outputs, monkeypatch): store = tmp_path / 'dna' paths = install(store, sources, outputs) Path(sources['dna.fa.gz']['path']).unlink() changed_modes = [] for path in [*store.rglob('*'), store]: changed_modes.append((path, path.stat().st_mode & 0o777)) os.chmod(path, 0o555 if path.is_dir() else 0o444) try: monkeypatch.setattr(download, 'fetch_file', forbidden) monkeypatch.setattr(bundle_store.BundleStore, 'lock', forbidden) monkeypatch.setattr(module, 'write_json', forbidden) monkeypatch.setattr(bundle_store, 'write_json', forbidden) monkeypatch.setattr(Path, 'mkdir', forbidden) assert install(store, sources, outputs, builder=forbidden) == paths assert inspected(store, sources, outputs).verified fast = inspected(store, sources, outputs, verify_files=False) assert fast.status == 'available' and not fast.verified assert not any(file.verified for file in fast.files.values()) finally: for path, mode in reversed(changed_modes): os.chmod(path, mode) @pytest.mark.parametrize('failure', ['builder', 'keyboard', 'last-output', 'validation', 'rename']) def test_failed_refresh_preserves_old_and_keeps_complete_input(tmp_path, sources, outputs, monkeypatch, failure): store = tmp_path / 'dna' old_paths = install(store, sources, outputs) old_bundle = inspect_materialization(store).bundle def failing_build(inputs, targets): build(inputs, targets) assert inspect_materialization(store).bundle == old_bundle if failure == 'builder': raise RuntimeError('biological validation failed') if failure == 'keyboard': raise KeyboardInterrupt if failure == 'last-output': Path(targets['index/info.json']).unlink() if failure == 'validation': Path(targets['index/info.json']).write_bytes(b'wrong') original_replace = module.os.replace if failure == 'rename': def fail_replace(source, target): if Path(target).parent.name == 'bundles': raise OSError('bundle rename failed') return original_replace(source, target) monkeypatch.setattr(module.os, 'replace', fail_replace) with pytest.raises((RuntimeError, KeyboardInterrupt, FileValidationError, OSError)): install(store, sources, outputs, builder=failing_build, force=True) assert inspect_materialization(store).bundle == old_bundle assert inspected(store, sources, outputs).verified assert all(Path(path).exists() for path in old_paths.values()) assert (input_dir(store) / 'dna.fa.gz').read_bytes() == COMPRESSED assert not list(store.glob('.staging-*')) Path(sources['dna.fa.gz']['path']).unlink() monkeypatch.setattr(module.os, 'replace', original_replace) assert install(store, sources, outputs, force=True) != old_paths assert not list(store.glob('.inputs-*')) def test_outputs_are_installed_once_renamed(tmp_path, sources, outputs, monkeypatch): store = tmp_path / 'dna' replace = os.replace def interrupted(source, target): replace(source, target) if Path(target).parent == store / 'bundles': raise KeyboardInterrupt monkeypatch.setattr(os, 'replace', interrupted) with pytest.raises(KeyboardInterrupt): install(store, sources, outputs) monkeypatch.setattr(os, 'replace', replace) Path(sources['dna.fa.gz']['path']).unlink() monkeypatch.setattr(download, 'fetch_file', forbidden) paths = install(store, sources, outputs, builder=forbidden) assert Path(paths['dna.fa']).read_bytes() == DATA assert inspected(store, sources, outputs).verified @pytest.mark.parametrize('change', ['source', 'transform-version', 'transform-options', 'output-inventory']) def test_dependency_mismatch_is_explicit_and_does_not_touch_old(tmp_path, sources, outputs, change): store = tmp_path / 'dna' paths = install(store, sources, outputs, retain_sources=True) new_sources = {name: dict(spec) for name, spec in sources.items()} options = {} new_outputs = outputs if change == 'source': replacement = tmp_path / 'another.fa.gz' replacement.write_bytes(COMPRESSED) new_sources['dna.fa.gz']['path'] = replacement elif change == 'transform-version': options['transform'] = {'version': 'fasta-2', 'options': TRANSFORM['options']} elif change == 'transform-options': options['transform'] = {'version': TRANSFORM['version'], 'options': {'validate': False}} else: new_outputs = {'dna.fa': outputs['dna.fa']} options['builder'] = lambda inputs, targets: Path(targets['dna.fa']).write_bytes(DATA) with pytest.raises(FileValidationError, match='force=True'): install(store, new_sources, new_outputs, **options) assert inspected(store, sources, outputs).verified assert install(store, new_sources, new_outputs, force=True, **options) != paths def test_transform_change_can_reuse_retained_inputs_after_upstream_removed(tmp_path, sources, outputs): store = tmp_path / 'dna' install(store, sources, outputs, retain_sources=True) owned = input_dir(store) Path(sources['dna.fa.gz']['path']).unlink() paths = install(store, sources, outputs, force=True, transform={'version': 'fasta-2'}) assert Path(paths['dna.fa']).read_bytes() == DATA assert not owned.exists() def test_untrusted_observation_is_not_trusted_verification(tmp_path, sources, outputs): store = tmp_path / 'dna' untrusted_sources = {'dna.fa.gz': {'path': sources['dna.fa.gz']['path']}} untrusted_outputs = {name: {} for name in outputs} paths = install(store, untrusted_sources, untrusted_outputs) state = inspected(store, untrusted_sources, untrusted_outputs) assert state.status == 'available' and not state.verified assert state.sources['dna.fa.gz']['verified'] is False assert state.sources['dna.fa.gz']['sha256'] == sha256(COMPRESSED).hexdigest() Path(paths['dna.fa']).write_bytes(b'x' * len(DATA)) assert inspected(store, untrusted_sources, untrusted_outputs, verify_files=False).status == 'available' assert inspected(store, untrusted_sources, untrusted_outputs).status == 'invalid' with pytest.raises(FileValidationError): install(store, untrusted_sources, untrusted_outputs) assert install(store, untrusted_sources, untrusted_outputs, force=True) != paths @pytest.mark.parametrize('kind', ['symlink', 'hardlink', 'fifo', 'extra', 'extra-directory', 'input-mutation']) def test_unsafe_or_undeclared_builder_results_are_never_published(tmp_path, sources, outputs, kind): store = tmp_path / 'dna' def unsafe(inputs, targets): build(inputs, targets) target = Path(targets['dna.fa']) if kind in ('symlink', 'hardlink', 'fifo'): target.unlink() if kind == 'symlink': target.symlink_to(sources['dna.fa.gz']['path']) elif kind == 'hardlink': os.link(inputs['dna.fa.gz'], target) else: os.mkfifo(target) elif kind == 'extra': (target.parent / 'undeclared').write_text('extra') elif kind == 'extra-directory': (target.parent / 'unused').mkdir() else: Path(inputs['dna.fa.gz']).write_bytes(b'changed') with pytest.raises(FileValidationError): install(store, sources, outputs, builder=unsafe) assert inspected(store, sources, outputs).status == 'missing' assert list((store / 'bundles').iterdir()) == [] assert Path(sources['dna.fa.gz']['path']).read_bytes() == COMPRESSED @pytest.mark.parametrize('kind', ['symlink', 'hardlink', 'fifo', 'directory']) def test_nonregular_local_sources_rejected_without_blocking(tmp_path, sources, outputs, kind): target = tmp_path / 'unsafe' if kind == 'symlink': target.symlink_to(sources['dna.fa.gz']['path']) elif kind == 'hardlink': os.link(sources['dna.fa.gz']['path'], target) elif kind == 'fifo': os.mkfifo(target) else: target.mkdir() sources['dna.fa.gz']['path'] = target with pytest.raises(FileValidationError): install(tmp_path / 'dna', sources, outputs) @pytest.mark.parametrize('force', [False, True]) def test_foreign_directory_never_adopted(tmp_path, sources, outputs, force): store = tmp_path / 'foreign' store.mkdir() (store / 'precious').write_text('keep') # Never a suggestion to use force=True, which can't help here. with pytest.raises(FileValidationError, match='never taken over'): install(store, sources, outputs, force=force) assert (store / 'precious').read_text() == 'keep' def test_complete_gzip_but_failed_decompression_reuses_input(tmp_path, sources, outputs): invalid_gzip = b'not-gzip' source = sources['dna.fa.gz']['path'] source.write_bytes(invalid_gzip) sources['dna.fa.gz'].update(metadata(invalid_gzip)) store = tmp_path / 'dna' with pytest.raises(gzip.BadGzipFile): install(store, sources, outputs) source.unlink() owned = input_dir(store) / 'dna.fa.gz' assert owned.read_bytes() == invalid_gzip calls = [] def failure(inputs, targets): calls.append(Path(inputs['dna.fa.gz']).read_bytes()) raise RuntimeError('retry reached builder with complete input') with pytest.raises(RuntimeError, match='retry reached'): install(store, sources, outputs, builder=failure) assert calls == [invalid_gzip] and owned.exists() def test_same_artifact_serializes_but_unrelated_builders_can_run(tmp_path, sources, outputs): store = tmp_path / 'dna' entered, release = threading.Event(), threading.Event() calls = [] def paused(inputs, targets): calls.append(1) entered.set() assert release.wait(8) build(inputs, targets) with ThreadPoolExecutor(max_workers=3) as pool: first = pool.submit(install, store, sources, outputs, builder=paused) assert entered.wait(8) second = pool.submit(install, store, sources, outputs, builder=paused) unrelated = pool.submit(install, tmp_path / 'other', sources, outputs) try: assert unrelated.result(timeout=8) assert not second.done() finally: release.set() assert first.result(timeout=8) == second.result(timeout=8) assert calls == [1] def test_reader_sees_whole_old_pair_until_the_new_bundle_is_renamed_in(tmp_path, sources, outputs, monkeypatch): store = tmp_path / 'dna' old = install(store, sources, outputs) entered, release = threading.Event(), threading.Event() replace = os.replace def paused(source, target): if Path(target).parent == store / 'bundles': entered.set() assert release.wait(8) return replace(source, target) monkeypatch.setattr(os, 'replace', paused) with ThreadPoolExecutor() as pool: future = pool.submit(install, store, sources, outputs, force=True) assert entered.wait(8) try: state = inspected(store, sources, outputs) assert state.verified assert {name: file.path for name, file in state.files.items()} == old assert (input_dir(store) / 'dna.fa.gz').exists() finally: release.set() new = future.result(timeout=8) assert new != old assert Path(new['index/info.json']).parents[1] == Path(new['dna.fa']).parent def test_large_dependency_receipts_are_readable(tmp_path, sources, outputs): store = tmp_path / 'dna' transform = {'version': '1', 'options': 'x' * (1024 * 1024 + 1)} paths = install(store, sources, outputs, transform=transform) receipt = Path(paths['dna.fa']).parent / module.MANIFEST assert receipt.stat().st_size > 1024 * 1024 assert inspect_materialization(store, sources, transform=transform, outputs=outputs).verified @pytest.fixture def server(): requests = [] actions = [] class Handler(BaseHTTPRequestHandler): def do_GET(self): requests.append(dict(self.headers)) action = actions.pop(0) if actions else {} offset = int(self.headers.get('Range', 'bytes=0-')[6:-1]) body = COMPRESSED[offset:] self.send_response(206 if offset else 200) self.send_header('Content-Length', str(len(body))) self.send_header('ETag', '"dna-110"') if offset: self.send_header('Content-Range', 'bytes %d-%d/%d' % (offset, len(COMPRESSED) - 1, len(COMPRESSED))) self.end_headers() try: self.wfile.write(body[:action.get('drop', len(body))]) self.wfile.flush() if 'drop' in action: self.connection.shutdown(socket.SHUT_RDWR) except OSError: pass def log_message(self, *args): pass srv = ThreadingHTTPServer(('127.0.0.1', 0), Handler) thread = threading.Thread(target=srv.serve_forever, daemon=True) thread.start() yield 'http://127.0.0.1:%d/dna.gz?token=secret' % srv.server_port, requests, actions srv.shutdown() srv.server_close() thread.join() @pytest.mark.parametrize('trusted', [True, False]) def test_resume_gzip_and_record_validated_transport(server, tmp_path, outputs, trusted): url, requests, actions = server actions.append({'drop': 12}) sources = {'dna.fa.gz': dict(url=url, size=len(COMPRESSED))} if trusted: sources['dna.fa.gz']['sha256'] = sha256(COMPRESSED).hexdigest() store = tmp_path / 'dna' paths = install(store, sources, outputs, download_options={'resume': True, 'chunk_size': 4, 'retry_backoff': 0, 'show_progress': False}) assert Path(paths['dna.fa']).read_bytes() == DATA assert requests[1]['Range'] == 'bytes=12-' assert requests[1]['If-Range'] == '"dna-110"' state = inspected(store, sources, outputs) record = state.sources['dna.fa.gz'] assert record['verified'] is trusted assert record['transport'] == {'header': 'ETag', 'value': '"dna-110"'} assert record['origin'] == url.split('?')[0] assert 'secret' not in (Path(state.bundle) / module.MANIFEST).read_text() assert not list(store.glob('.inputs-*')) def test_interrupted_download_and_builder_retry_without_network(server, tmp_path, outputs, monkeypatch): url, requests, actions = server sources = {'dna.fa.gz': dict(url=url, **metadata(COMPRESSED))} store = tmp_path / 'dna' def interrupt(done, total): if done >= 8: raise KeyboardInterrupt options = {'resume': True, 'chunk_size': 4, 'progress_callback': interrupt, 'show_progress': False} with pytest.raises(KeyboardInterrupt): install(store, sources, outputs, download_options=options) owned = input_dir(store) / 'dna.fa.gz' assert (_state_directory(owned) / 'partial').read_bytes() == COMPRESSED[:8] assert inspected(store, sources, outputs).status == 'missing' del options['progress_callback'] def interrupted_build(inputs, targets): with gzip.open(inputs['dna.fa.gz'], 'rb') as source, open(targets['dna.fa'], 'wb') as target: target.write(source.read(4)) raise KeyboardInterrupt with pytest.raises(KeyboardInterrupt): install(store, sources, outputs, builder=interrupted_build, download_options=options) assert requests[1]['Range'] == 'bytes=8-' assert owned.read_bytes() == COMPRESSED assert not (_state_directory(owned) / 'partial').exists() monkeypatch.setattr(download, 'fetch_file', forbidden) assert install(store, sources, outputs, download_options=options) assert len(requests) == 2 assert not owned.exists() def test_killed_builder_retries_with_completed_local_input(tmp_path, sources, outputs): store = tmp_path / 'dna' ready = tmp_path / 'builder-ready' code = ''' import gzip, json, sys, time from pathlib import Path from datacache import materialize store, source, ready, outputs = sys.argv[1:] def builder(inputs, targets): with gzip.open(inputs['dna.fa.gz'], 'rb') as compressed: Path(targets['dna.fa']).write_bytes(compressed.read(4)) Path(ready).write_text('ready') time.sleep(30) materialize(store, {'dna.fa.gz': {'path': source}}, transform={'version':'fasta-1','options':{'validate':True}}, outputs=json.loads(outputs), builder=builder, download_options={'show_progress':False}) ''' process = subprocess.Popen([sys.executable, '-c', code, str(store), str(sources['dna.fa.gz']['path']), str(ready), json.dumps(outputs)], stdout=subprocess.PIPE, stderr=subprocess.PIPE) try: deadline = time.monotonic() + 8 while not ready.exists() and process.poll() is None and time.monotonic() < deadline: time.sleep(0.02) assert ready.exists(), process.communicate(timeout=2) process.kill() process.communicate(timeout=5) assert list(store.glob('.staging-*')) Path(sources['dna.fa.gz']['path']).unlink() untrusted = {'dna.fa.gz': {'path': sources['dna.fa.gz']['path']}} paths = install(store, untrusted, outputs) assert Path(paths['dna.fa']).read_bytes() == DATA assert not list(store.glob('.staging-*')) assert not list(store.glob('.inputs-*')) finally: if process.poll() is None: process.kill() process.communicate(timeout=5) @pytest.mark.parametrize('options', [ {'force': 1}, {'retain_sources': 'yes'}, {'verify_files': None}, {'builder': None}, {'transform': {'version': ''}}, {'transform': {'version': '1', 'options': float('nan')}}, {'outputs': {'../escape': {}}}, {'outputs': {'A': {}, 'a/b': {}}}, {'download_options': {'raw': False}}, {'download_options': {'show_progress': 1}}, {'download_options': {'resume': 'yes'}}, {'download_options': {'chunk_size': 0}}, {'download_options': {'progress_callback': 'yes'}}, {'download_options': {'max_retries': -1}}, ]) def test_invalid_options_fail_before_creating_store(tmp_path, sources, outputs, options): store = tmp_path / 'missing' / 'dna' with pytest.raises((ValueError, TypeError)): install(store, sources, outputs, **options) assert not store.parent.exists() def test_fast_reuse_does_not_hash_payloads(tmp_path, sources, outputs, monkeypatch): store = tmp_path / 'dna' paths = install(store, sources, outputs) original = bundle_store.hash_file def no_hash(*args, **options): assert options.get('hash_contents') is False return original(*args, **options) monkeypatch.setattr(bundle_store, 'hash_file', no_hash) monkeypatch.setattr(module, 'hash_file', no_hash) assert install(store, sources, outputs, verify_files=False, builder=forbidden) == paths def test_input_hash_failure_and_bad_final_output_never_publish(tmp_path, sources, outputs): store = tmp_path / 'dna' sources['dna.fa.gz']['sha256'] = '0' * 64 with pytest.raises(FileValidationError): install(store, sources, outputs, verify_files=False) assert not list((store / 'bundles').iterdir()) sources['dna.fa.gz'].update(metadata(COMPRESSED)) outputs['dna.fa']['sha256'] = '0' * 64 with pytest.raises(FileValidationError): install(store, sources, outputs, verify_files=False) assert not list((store / 'bundles').iterdir()) def test_source_file_url_is_private_copy_and_not_deleted(tmp_path, sources, outputs): source = sources['dna.fa.gz']['path'] sources['dna.fa.gz'] = dict(url=source.as_uri(), **metadata(COMPRESSED)) assert install(tmp_path / 'dna', sources, outputs) assert source.read_bytes() == COMPRESSED def test_private_input_corruption_requires_explicit_repair(tmp_path, sources, outputs): store = tmp_path / 'dna' def interrupted(inputs, targets): raise KeyboardInterrupt with pytest.raises(KeyboardInterrupt): install(store, sources, outputs, builder=interrupted) owned = input_dir(store) / 'dna.fa.gz' owned.write_bytes(b'x' * len(COMPRESSED)) with pytest.raises(FileValidationError, match='force=True'): install(store, sources, outputs) assert install(store, sources, outputs, force=True) assert not owned.exists() def test_bare_complete_input_without_receipt_can_be_explicitly_repaired(tmp_path, sources, outputs): store = tmp_path / 'dna' def interrupted(inputs, targets): raise KeyboardInterrupt with pytest.raises(KeyboardInterrupt): install(store, sources, outputs, builder=interrupted) directory = input_dir(store) (directory / module.INPUTS).unlink() (directory / 'dna.fa.gz').write_bytes(b'x' * len(COMPRESSED)) with pytest.raises(FileValidationError, match='force=True'): install(store, sources, outputs) assert install(store, sources, outputs, force=True) def test_untrusted_bare_input_requires_new_acquisition(tmp_path, sources, outputs): store = tmp_path / 'dna' sources = {'dna.fa.gz': {'path': sources['dna.fa.gz']['path']}} def interrupted(inputs, targets): raise KeyboardInterrupt with pytest.raises(KeyboardInterrupt): install(store, sources, outputs, builder=interrupted) directory = input_dir(store) (directory / module.INPUTS).unlink() (directory / 'dna.fa.gz').write_bytes(b'unverified unknown') assert install(store, sources, outputs) def test_inspection_requires_complete_definition_or_none(tmp_path, sources, outputs): with pytest.raises(ValueError): inspect_materialization(tmp_path / 'dna', sources, outputs=outputs) with pytest.raises(ValueError): inspect_materialization(tmp_path / 'dna', verify_files='yes') def test_precreated_empty_store_preserves_mode(tmp_path, sources, outputs): store = tmp_path / 'dna' store.mkdir(mode=0o750) assert inspected(store, sources, outputs).status == 'missing' install(store, sources, outputs) assert store.stat().st_mode & 0o777 == 0o750 def test_retention_toggle_on_hit_does_not_cleanup(tmp_path, sources, outputs): store = tmp_path / 'dna' paths = install(store, sources, outputs, retain_sources=True) owned = input_dir(store) / 'dna.fa.gz' assert install(store, sources, outputs) == paths assert owned.exists() assert install(store, sources, outputs, force=True) != paths assert not owned.exists() @pytest.mark.parametrize('kind', ['symlink', 'fifo', 'shared-directory']) def test_planted_private_state_is_not_followed_or_adopted(tmp_path, sources, outputs, kind): store = tmp_path / 'dna' def interrupted(inputs, targets): raise KeyboardInterrupt with pytest.raises(KeyboardInterrupt): install(store, sources, outputs, builder=interrupted) owned_directory = input_dir(store) owned = owned_directory / 'dna.fa.gz' if kind == 'shared-directory': os.chmod(owned_directory, 0o755) else: owned.unlink() if kind == 'symlink': owned.symlink_to(sources['dna.fa.gz']['path']) else: os.mkfifo(owned) with pytest.raises(FileValidationError): install(store, sources, outputs, force=True) assert Path(sources['dna.fa.gz']['path']).read_bytes() == COMPRESSED @pytest.mark.parametrize('malformation', [ 'missing-output', 'extra-output', 'source-size', 'source-identity', 'source-trust', 'source-inventory', 'transform', 'output-digest', 'transport', 'definition', ]) def test_malformed_manifest_is_invalid_even_for_fast_inspection(tmp_path, sources, outputs, malformation): store = tmp_path / 'dna' paths = install(store, sources, outputs) manifest = Path(paths['dna.fa']).parent / module.MANIFEST receipt = json.loads(manifest.read_text()) if malformation == 'missing-output': del receipt['outputs']['dna.fa'] elif malformation == 'extra-output': receipt['outputs']['extra'] = metadata(b'') elif malformation == 'source-size': receipt['sources']['dna.fa.gz']['size'] += 1 elif malformation == 'source-identity': receipt['sources']['dna.fa.gz']['identity'] = '0' * 64 elif malformation == 'source-trust': receipt['sources']['dna.fa.gz']['verified'] = False elif malformation == 'source-inventory': receipt['sources'] = {} elif malformation == 'transform': receipt['definition']['transform'] = {'version': ''} elif malformation == 'output-digest': receipt['outputs']['dna.fa']['sha256'] = None elif malformation == 'transport': receipt['sources']['dna.fa.gz']['transport'] = {'header': 'ETag', 'value': 'W/"weak"'} else: receipt['definition'] = None manifest.write_text(json.dumps(receipt)) assert inspected(store, sources, outputs, verify_files=False).status == 'invalid' assert inspect_materialization(store, verify_files=False).status == 'invalid' def test_inspection_permission_error_is_inaccessible(tmp_path, sources, outputs, monkeypatch): store = tmp_path / 'dna' install(store, sources, outputs) def permission_denied(*args, **kwargs): raise PermissionError('cannot read receipt') monkeypatch.setattr(module, 'read_json', permission_denied) assert inspected(store, sources, outputs).status == 'inaccessible' with pytest.raises(PermissionError): install(store, sources, outputs) @pytest.mark.parametrize('bad_sources', [ {}, {'a': {}}, {'a': {'url': 'https://example.test', 'path': 'local'}}, {'a': {'url': ''}}, {'a': {'url': 'unsupported://example.test'}}, {'a': {'url': 'file://remote/file'}}, {'a': {'url': 'file:///file?query'}}, {'a': {'url': 'https://example.test', 'decompress': True}}, {'a': {'path': 'local'}, '.a.datacache.json': {'path': 'local'}}, {'a': {'path': 'local'}, '.a.datacache.json/extra': {'path': 'local'}}, ]) def test_invalid_source_definitions_fail_without_writes(tmp_path, outputs, bad_sources): store = tmp_path / 'missing' / 'dna' with pytest.raises(ValueError): install(store, bad_sources, outputs) assert not store.parent.exists() def test_multiple_inputs_keep_completed_first_source_on_later_failure(tmp_path, sources, outputs, monkeypatch): store = tmp_path / 'dna' original = Path(sources['dna.fa.gz']['path']) sources['auxiliary'] = {'path': tmp_path / 'missing'} with pytest.raises(FileNotFoundError): install(store, sources, outputs) completed = input_dir(store) / 'dna.fa.gz' assert completed.read_bytes() == COMPRESSED original.unlink() Path(sources['auxiliary']['path']).write_bytes(b'auxiliary') paths = install(store, sources, outputs) assert Path(paths['dna.fa']).read_bytes() == DATA assert set(inspected(store, sources, outputs).sources) == {'dna.fa.gz', 'auxiliary'} def test_explicitly_empty_input_and_output(tmp_path): source = tmp_path / 'empty' source.touch() sources = {'empty': dict(path=source, **metadata(b''))} outputs = {'empty-result': metadata(b'')} def empty(inputs, targets): Path(targets['empty-result']).touch() store = tmp_path / 'artifact' materialize(store, sources, outputs=outputs, transform={'version': '1'}, builder=empty, download_options={'show_progress': False}) assert inspect_materialization(store, sources, transform={'version': '1'}, outputs=outputs).verified assert source.exists() def test_progress_defaults_on_and_existing_hits_stay_quiet(tmp_path, sources, outputs, monkeypatch): enabled = [] original = module.Progress def progress(enabled_flag, *args, **options): enabled.append(enabled_flag) return original(False, *args, **options) monkeypatch.setattr(module, 'Progress', progress) store = tmp_path / 'dna' paths = install(store, sources, outputs, download_options={}) assert any(enabled) enabled.clear() assert install(store, sources, outputs, download_options={}) == paths assert not any(enabled) def test_builder_paths_stay_absolute_after_working_directory_change(tmp_path, sources, outputs, monkeypatch): monkeypatch.chdir(tmp_path) other = tmp_path / 'other' other.mkdir() def changing_directory(inputs, targets): assert all(Path(path).is_absolute() for path in [*inputs.values(), *targets.values()]) monkeypatch.chdir(other) build(inputs, targets) paths = install(Path('relative') / 'dna', sources, outputs, builder=changing_directory) assert Path(paths['dna.fa']).read_bytes() == DATA def test_local_copy_honors_chunk_size_and_progress_callback(tmp_path, sources, outputs): events = [] install(tmp_path / 'dna', sources, outputs, download_options={'chunk_size': 4, 'progress_callback': lambda *event: events.append(event), 'show_progress': False}) assert events == [(count, len(COMPRESSED)) for count in range(4, len(COMPRESSED) + 1, 4)] def test_interrupted_local_copy_never_modifies_original(tmp_path, sources, outputs): store = tmp_path / 'dna' def interrupt(done, total): raise KeyboardInterrupt with pytest.raises(KeyboardInterrupt): install(store, sources, outputs, download_options={'chunk_size': 4, 'progress_callback': interrupt, 'show_progress': False}) assert Path(sources['dna.fa.gz']['path']).read_bytes() == COMPRESSED assert not (input_dir(store) / 'dna.fa.gz').exists() assert not list(input_dir(store).glob('.copy-*')) assert install(store, sources, outputs) # Review fixes def unpinned(tmp_path, data): source = tmp_path / 'local.fa.gz' source.write_bytes(gzip.compress(data, mtime=0)) return source, {'dna.fa.gz': dict(path=source)} def unpinned_outputs(): return {'dna.fa': {}, 'index/info.json': {}} def test_forced_refresh_copies_an_edited_local_source_again(tmp_path): source, sources = unpinned(tmp_path, DATA) store = tmp_path / 'dna' install(store, sources, unpinned_outputs(), retain_sources=True) source.write_bytes(gzip.compress(b'>chr1\nTTTTTTTT\n', mtime=0)) paths = install(store, sources, unpinned_outputs(), force=True) assert Path(paths['dna.fa']).read_bytes() == b'>chr1\nTTTTTTTT\n' state = inspect_materialization(store) assert state.sources['dna.fa.gz']['sha256'] == sha256(source.read_bytes()).hexdigest() def test_force_repairs_a_corrupt_private_input_receipt(tmp_path, sources, outputs): import shutil store = tmp_path / 'dna' install(store, sources, outputs, retain_sources=True) # No bundles left, so the next call has to acquire inputs again. for bundle in (store / 'bundles').iterdir(): shutil.rmtree(bundle) (input_dir(store) / module.INPUTS).write_text('{') with pytest.raises(FileValidationError, match='use force=True to repair'): install(store, sources, outputs) paths = install(store, sources, outputs, force=True) assert Path(paths['dna.fa']).read_bytes() == DATA def test_force_always_rebuilds(tmp_path, sources, outputs): store = tmp_path / 'dna' old = install(store, sources, outputs) built = [] def counting_build(inputs, targets): built.append(True) build(inputs, targets) new = install(store, sources, outputs, builder=counting_build, force=True) assert built == [True] and new != old assert install(store, sources, outputs, builder=counting_build) == new assert built == [True] def test_forced_refresh_does_not_hash_the_old_outputs(tmp_path, sources, outputs, monkeypatch): store = tmp_path / 'dna' install(store, sources, outputs) modes = [] original = module._inspect_materialization def recording(path, expected, *, verify_files=True): modes.append(verify_files) return original(path, expected, verify_files=verify_files) monkeypatch.setattr(module, '_inspect_materialization', recording) install(store, sources, outputs, force=True) assert modes and not any(modes) @pytest.mark.parametrize('builder_fails', [True, False]) def test_cleanup_failure_never_masks_the_outcome(tmp_path, sources, outputs, monkeypatch, caplog, builder_fails): store = tmp_path / 'dna' import shutil original = shutil.rmtree def failing_rmtree(path, *args, **kwargs): if Path(path).name.startswith('.staging-'): raise OSError('busy') return original(path, *args, **kwargs) monkeypatch.setattr(shutil, 'rmtree', failing_rmtree) def failing_build(inputs, targets): raise RuntimeError('builder failed') if builder_fails: with pytest.raises(RuntimeError, match='builder failed'): install(store, sources, outputs, builder=failing_build) else: assert Path(install(store, sources, outputs)['dna.fa']).read_bytes() == DATA assert 'Could not remove the working directory' in caplog.text ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7165802 datacache-1.27.1/tests/test_names.py0000644000076500000000000000356715264220442017107 0ustar00iskanderwheelfrom datacache import build_local_filename def test_url_without_filename(): filename = build_local_filename(download_url="http://www.google.com/") assert filename assert "google" in filename def test_multiple_domains_same_file(): filename_google = build_local_filename( download_url="http://www.google.com/index.html") filename_yahoo = build_local_filename( download_url="http://www.yahoo.com/index.html") assert "index" in filename_google assert "index" in filename_yahoo assert filename_yahoo != filename_google def test_long_url_can_be_used_as_cache_key(): url = "file:///" + "long-directory/" * 20 + "sequences.fa.gz" filename = build_local_filename(download_url=url) assert filename == build_local_filename(download_url=url) assert filename.endswith("sequences.fa.gz") assert len(filename) < len(url) def test_names_over_255_utf8_bytes_are_shortened_keeping_their_end(): # 104 characters but 304 bytes: too long for ext4, which counts bytes. long_name = "測" * 100 + ".csv" key = build_local_filename(filename=long_name) assert len(key.encode("utf-8")) <= 255 assert key.endswith("測.csv") assert key == build_local_filename(filename=long_name) assert key != build_local_filename(filename="數" + long_name[1:]) def test_names_within_255_utf8_bytes_are_unchanged(): within = "é" * 125 + ".csv" # 254 bytes assert build_local_filename(filename=within) == within at_limit = "é" * 125 + "x.csv" # 255 bytes assert build_local_filename(filename=at_limit) == at_limit over = "é" * 126 + ".csv" # 256 bytes assert build_local_filename(filename=over) != over def test_shortened_decompressed_names_keep_their_extension(): key = build_local_filename(filename="測" * 100 + ".csv.gz", decompress=True) assert key.endswith("測.csv") and len(key.encode("utf-8")) <= 255 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7166524 datacache-1.27.1/tests/test_packaging.py0000644000076500000000000000132515264220442017716 0ustar00iskanderwheel"""Check the dependencies consumers receive from a normal installation.""" from importlib.metadata import metadata, requires from packaging.requirements import Requirement def test_tqdm_is_an_unconditional_runtime_dependency(): dependencies = [Requirement(value) for value in requires("datacache")] tqdm_dependencies = [item for item in dependencies if item.name == "tqdm"] assert len(tqdm_dependencies) == 1 dependency = tqdm_dependencies[0] assert dependency.marker is None assert "4.64.0" in dependency.specifier assert "4.63.0" not in dependency.specifier def test_progress_extra_remains_a_compatible_alias(): assert "progress" in metadata("datacache").get_all("Provides-Extra", []) ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1792090401.71673 datacache-1.27.1/tests/test_permissions.py0000644000076500000000000000366515264220442020356 0ustar00iskanderwheel"""Sharing old caches is explicit and never changes their bytes or identity.""" import os import stat import pytest from datacache import Cache, FileValidationError, make_file_readable pytestmark = pytest.mark.skipif(os.name != "posix", reason="POSIX sharing permissions") @pytest.mark.parametrize("group,others,expected", [ (True, False, 0o640), (True, True, 0o644), (False, True, 0o604), (False, False, 0o600), ]) def test_explicit_read_access_preserves_cached_file(tmp_path, group, others, expected): path = tmp_path / "data" path.write_bytes(b"legacy data") path.chmod(0o600) before = path.stat() cache = Cache(cache_root=tmp_path) for _ in range(2): assert cache.make_readable(filename="data", group=group, others=others) == str(path) assert stat.S_IMODE(path.stat().st_mode) == expected assert path.read_bytes() == b"legacy data" assert path.stat().st_ino == before.st_ino assert path.stat().st_mtime_ns == before.st_mtime_ns @pytest.mark.parametrize("kind", ["directory", "symlink", "fifo"]) def test_sharing_rejects_non_regular_files(tmp_path, kind): target = tmp_path / "target" target.write_bytes(b"unrelated") target.chmod(0o600) path = tmp_path / "data" if kind == "directory": path.mkdir() elif kind == "symlink": path.symlink_to(target) else: os.mkfifo(path) with pytest.raises(FileValidationError): make_file_readable(path) assert stat.S_IMODE(target.stat().st_mode) == 0o600 def test_sharing_missing_file_does_not_create_it(tmp_path): with pytest.raises(FileNotFoundError): make_file_readable(tmp_path / "missing") assert list(tmp_path.iterdir()) == [] def test_sharing_does_not_remove_existing_write_or_execute_bits(tmp_path): path = tmp_path / "data" path.write_bytes(b"data") path.chmod(0o701) make_file_readable(path) assert stat.S_IMODE(path.stat().st_mode) == 0o741 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7168195 datacache-1.27.1/tests/test_progress.py0000644000076500000000000001462215264220442017642 0ustar00iskanderwheel"""Progress must be accurate, optional, retry-aware, and exception-safe.""" import builtins from contextlib import closing, redirect_stderr import gzip import hashlib import io from pathlib import Path from types import SimpleNamespace import zipfile import pandas as pd import pytest import requests import tqdm.auto from datacache import Cache, db_from_dataframe, download, fetch_file @pytest.fixture def bars(monkeypatch): created = [] class Bar: def __init__(self, **options): self.options = options self.total = options["total"] self.n = 0 self.closed = False self.counts = [] created.append(self) def update(self, count): assert count >= 0 self.n += count self.counts.append((self.n, self.total)) def reset(self, total=None): self.n = 0 self.total = total def close(self): self.closed = True monkeypatch.setattr(tqdm.auto, "tqdm", Bar) return created @pytest.mark.parametrize("kind", ["raw", "gz", "zip"]) def test_download_decompression_and_verification_progress(tmp_path, bars, kind): data = b"complete data\n" * 100 source = tmp_path / ("source." + kind) if kind == "gz": source.write_bytes(gzip.compress(data)) elif kind == "zip": with zipfile.ZipFile(source, "w") as archive: archive.writestr("data", data) else: source.write_bytes(data) reports = [] cache = Cache(cache_root=tmp_path / "cache") path = cache.fetch(source.as_uri(), filename="data", show_progress=True, chunk_size=10, expected_sha256=hashlib.sha256(data).hexdigest(), progress_callback=lambda done, total: reports.append((done, total))) assert Path(path).read_bytes() == data descriptions = [bar.options["desc"] for bar in bars] assert descriptions == ["Downloading"] + ([] if kind == "raw" else ["Decompressing"]) + ["Verifying"] assert bars[0].n == source.stat().st_size assert bars[-1].n == len(data) assert all(bar.closed for bar in bars) assert reports[-1] == (source.stat().st_size, source.stat().st_size) count = len(bars) cache.fetch(source.as_uri(), filename="data", show_progress=True) assert len(bars) == count def test_each_retry_gets_fresh_progress_even_with_larger_first_chunk(tmp_path, bars, monkeypatch): calls = [] def stream(url, output, progress_callback, **kwargs): calls.append(url) if len(calls) == 1: output.write(b"part") progress_callback(4, 10) raise requests.ConnectionError("injected interruption") output.write(b"complete") progress_callback(8, 8) return 8 monkeypatch.setattr(download, "_stream_to_file", stream) destination = tmp_path / "data" fetch_file("https://host/file", destination=destination, show_progress=True, retry_backoff=0) assert destination.read_bytes() == b"complete" assert [bar.counts for bar in bars] == [[(4, 10)], [(8, 8)]] assert all(bar.closed for bar in bars) assert list(tmp_path.iterdir()) == [destination] @pytest.mark.parametrize("failure", ["callback", "hash", "gzip"]) def test_failure_closes_bars_and_preserves_old_file(tmp_path, bars, failure): source = tmp_path / ("source.gz" if failure == "gzip" else "source") source.write_bytes(b"new data") destination = tmp_path / "data" destination.write_bytes(b"old data") options = {} if failure == "callback": def reject(*args): raise RuntimeError("callback failed") options["progress_callback"] = reject elif failure == "hash": options["expected_sha256"] = "0" * 64 with pytest.raises((RuntimeError, ValueError, OSError)): fetch_file(source.as_uri(), destination=destination, force=True, show_progress=True, **options) assert bars and all(bar.closed for bar in bars) assert destination.read_bytes() == b"old data" assert not list(tmp_path.glob(".datacache-*")) def test_empty_download_reports_zero_total(tmp_path, bars): source = tmp_path / "source" source.touch() fetch_file(source.as_uri(), destination=tmp_path / "output", show_progress=True, allow_empty=True) assert bars[0].counts == [(0, 0)] assert bars[0].closed def test_http_content_encoding_uses_unknown_decoded_total(tmp_path, bars, monkeypatch): data = b"uncompressed payload" * 100 response = SimpleNamespace( headers={"Content-Length": "20", "Content-Encoding": "gzip"}, raise_for_status=lambda: None, iter_content=lambda **kw: iter([data]), close=lambda: None) monkeypatch.setattr(download.requests, "get", lambda *a, **kw: response) fetch_file("https://host/data", destination=tmp_path / "data", show_progress=True) assert bars[0].counts == [(len(data), None)] def test_database_progress_reports_completed_batches(tmp_path, bars): with closing(db_from_dataframe(tmp_path / "data.db", "records", pd.DataFrame({"id": range(2501)}), show_progress=True)) as connection: assert connection.execute("SELECT COUNT(*) FROM records").fetchone() == (2501,) assert bars[0].options["unit"] == "rows" assert bars[0].counts == [(1000, 2501), (2000, 2501), (2501, 2501)] assert bars[0].closed def test_missing_required_tqdm_has_actionable_error(tmp_path, monkeypatch): original_import = builtins.__import__ def no_tqdm(name, *args, **kwargs): if name.startswith("tqdm"): raise ImportError("not installed") return original_import(name, *args, **kwargs) monkeypatch.setattr(builtins, "__import__", no_tqdm) source = tmp_path / "source" source.write_bytes(b"data") fetch_file(source.as_uri(), destination=tmp_path / "quiet") with pytest.raises(ImportError, match=r"tqdm is a required datacache dependency"): fetch_file(source.as_uri(), destination=tmp_path / "progress", show_progress=True) assert not (tmp_path / "progress").exists() assert not list(tmp_path.glob(".datacache-*")) def test_real_tqdm_writes_only_to_stderr(tmp_path, capsys): source = tmp_path / "source" source.write_bytes(b"data") captured = io.StringIO() with redirect_stderr(captured): fetch_file(source.as_uri(), destination=tmp_path / "output", show_progress=True) assert "Downloading" in captured.getvalue() assert capsys.readouterr().out == "" ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7169423 datacache-1.27.1/tests/test_provenance.py0000644000076500000000000002664615264220442020147 0ustar00iskanderwheel"""Inspection reports size, mtime, and opt-in download provenance (#75).""" import hashlib import json import os from pathlib import Path import stat import threading from types import SimpleNamespace import pytest from datacache import ( Cache, download, fetch_file, inspect_file, inspect_files, make_file_readable, provenance) from datacache.integrity import _validate_file DATA = b"id,value\n1,10\n" SHA256 = hashlib.sha256(DATA).hexdigest() posix_permissions = pytest.mark.skipif( not hasattr(os, "geteuid") or os.geteuid() == 0, reason="needs POSIX permissions") @pytest.fixture def source(tmp_path): path = tmp_path / "source.csv" path.write_bytes(DATA) return path def fetch(source, destination, **options): return Path(fetch_file(source.as_uri(), destination=destination, **options)) def record_of(path): return json.loads(Path(provenance.sidecar_path(path)).read_text()) def test_available_files_report_size_and_mtime(source, tmp_path): inspection = inspect_file(source) assert inspection.size == len(DATA) assert inspection.mtime == os.stat(source).st_mtime missing = inspect_file(tmp_path / "absent") assert missing.status == "missing" and missing.size is None and missing.mtime is None def test_size_and_mtime_come_from_the_validated_bytes(source): # The stat is taken from the open file that was checked, not a second stat. _, info = _validate_file(source, SHA256) assert info.st_size == len(DATA) @posix_permissions def test_an_unreadable_parent_reports_no_size(tmp_path): folder = tmp_path / "locked" folder.mkdir() (folder / "file").write_bytes(DATA) folder.chmod(0) try: inspection = inspect_file(folder / "file") finally: folder.chmod(0o755) assert inspection.status == "inaccessible" and inspection.size is None def test_recording_is_opt_in(source, tmp_path): path = fetch(source, tmp_path / "data.csv", expected_sha256=SHA256) assert sorted(os.listdir(tmp_path)) == ["data.csv", "source.csv"] inspection = inspect_file(path) assert inspection.recorded_sha256 is None and inspection.source_url is None def test_a_recorded_digest_is_reported_but_never_counts_as_verified(source, tmp_path): path = fetch(source, tmp_path / "data.csv", expected_sha256=SHA256, record_provenance=True) inspection = inspect_file(path) # no expectation supplied # verified keeps its meaning: a trusted digest matched the bytes just now. assert inspection.status == "available" and not inspection.verified assert inspection.recorded_sha256 == SHA256 assert inspection.source_url == source.as_uri() assert inspection.fetched_at.endswith("+00:00") # A caller that trusts the record can check the current bytes against it. assert inspect_file(path, expected_sha256=inspection.recorded_sha256).verified def test_an_unverified_download_records_provenance_but_no_digest(source, tmp_path): path = fetch(source, tmp_path / "data.csv", record_provenance=True) inspection = inspect_file(path) assert inspection.source_url == source.as_uri() and inspection.recorded_sha256 is None @pytest.mark.parametrize("change", ["size", "same-size"]) def test_a_file_changed_after_its_download_is_unrecorded(source, tmp_path, change): path = fetch(source, tmp_path / "data.csv", expected_sha256=SHA256, record_provenance=True) before = os.stat(path) if change == "size": path.write_bytes(DATA + b"2,20\n") else: path.write_bytes(DATA.replace(b"10", b"99")) os.utime(path, ns=(before.st_atime_ns, before.st_mtime_ns + 1_000_000_000)) inspection = inspect_file(path) assert inspection.status == "available" assert inspection.recorded_sha256 is None and inspection.source_url is None def test_a_concurrent_replacement_is_not_described_by_the_record(source, tmp_path, monkeypatch): # The record describes the bytes this call staged, not whatever another # writer publishes at the destination before the record is written. destination = tmp_path / "data.csv" real_replace = os.replace def replace_then_overwrite(source_path, target): real_replace(source_path, target) if os.fspath(target) == str(destination): destination.write_bytes(DATA.replace(b"10", b"77")) # same size info = os.stat(destination) os.utime(destination, ns=(info.st_atime_ns, info.st_mtime_ns + 5_000_000_000)) monkeypatch.setattr(download.os, "replace", replace_then_overwrite) fetch(source, destination, expected_sha256=SHA256, record_provenance=True) monkeypatch.undo() assert inspect_file(destination).recorded_sha256 is None def test_a_new_download_without_recording_removes_the_old_record(source, tmp_path): path = fetch(source, tmp_path / "data.csv", expected_sha256=SHA256, record_provenance=True) fetch(source, path, force=True) # records nothing this time assert not os.path.exists(provenance.sidecar_path(path)) assert inspect_file(path).recorded_sha256 is None def test_a_new_recorded_download_replaces_the_record(source, tmp_path): path = fetch(source, tmp_path / "data.csv", expected_sha256=SHA256, record_provenance=True) fetch(source, path, force=True, record_provenance=True) # no expectation this time assert record_of(path)["sha256"] is None @pytest.mark.parametrize("record", ["", "{not json", "[]", '{"format": 99}', "[" * 60000, "directory"]) def test_unusable_records_degrade_to_no_provenance(source, tmp_path, record): path = fetch(source, tmp_path / "data.csv", expected_sha256=SHA256, record_provenance=True) sidecar = Path(provenance.sidecar_path(path)) sidecar.unlink() if record == "directory": sidecar.mkdir() else: sidecar.write_text(record) inspection = inspect_file(path) assert inspection.status == "available" assert inspection.recorded_sha256 is None and inspection.source_url is None @pytest.mark.skipif(not hasattr(os, "mkfifo"), reason="needs FIFOs") def test_a_fifo_at_the_record_name_never_blocks_inspection(source, tmp_path): path = fetch(source, tmp_path / "data.csv") os.mkfifo(provenance.sidecar_path(path)) results = [] worker = threading.Thread(target=lambda: results.append(inspect_file(path)), daemon=True) worker.start() worker.join(timeout=10) assert not worker.is_alive(), "inspection blocked on a FIFO" assert results[0].status == "available" and results[0].source_url is None @posix_permissions def test_an_unreadable_record_degrades_to_no_provenance(source, tmp_path): path = fetch(source, tmp_path / "data.csv", expected_sha256=SHA256, record_provenance=True) sidecar = Path(provenance.sidecar_path(path)) sidecar.chmod(0) try: inspection = inspect_file(path) finally: sidecar.chmod(0o644) assert inspection.status == "available" and inspection.recorded_sha256 is None def test_bytes_paths_are_inspected(source, tmp_path): path = fetch(source, tmp_path / "data.csv", expected_sha256=SHA256, record_provenance=True) inspection = inspect_file(os.fsencode(path)) assert inspection.status == "available" and inspection.recorded_sha256 == SHA256 def test_inventories_report_records_but_verify_only_expectations(source, tmp_path): root = tmp_path / "cache" fetch(source, root / "data.csv", expected_sha256=SHA256, record_provenance=True) inventory = inspect_files(root, {"data.csv": None}) assert list(inventory.files) == ["data.csv"] assert inventory.status == "available" and not inventory.verified assert inventory.files["data.csv"].recorded_sha256 == SHA256 assert inspect_files(root, {"data.csv": {"expected_sha256": SHA256}}).verified @posix_permissions def test_inspection_writes_nothing_on_a_read_only_cache(source, tmp_path): root = tmp_path / "cache" path = fetch(source, root / "data.csv", expected_sha256=SHA256, record_provenance=True) before = sorted(os.listdir(root)) root.chmod(0o555) try: assert inspect_file(path).recorded_sha256 == SHA256 assert inspect_files(root, {"data.csv": None}).files["data.csv"].recorded_sha256 == SHA256 finally: root.chmod(0o755) assert sorted(os.listdir(root)) == before def test_cache_hits_never_write_a_record(source, tmp_path): path = fetch(source, tmp_path / "data.csv", record_provenance=True) Path(provenance.sidecar_path(path)).unlink() fetch(source, path, record_provenance=True) # cache hit assert not os.path.exists(provenance.sidecar_path(path)) @posix_permissions def test_records_have_their_files_permissions(source, tmp_path): previous = os.umask(0o022) try: path = fetch(source, tmp_path / "data.csv", record_provenance=True) assert stat.S_IMODE(os.stat(provenance.sidecar_path(path)).st_mode) == 0o644 # A file made private keeps a private record when it is refetched. path.chmod(0o600) fetch(source, path, force=True, record_provenance=True) assert stat.S_IMODE(os.stat(path).st_mode) == 0o600 assert stat.S_IMODE(os.stat(provenance.sidecar_path(path)).st_mode) == 0o600 # Sharing the file explicitly shares its record too. make_file_readable(path, group=True) assert stat.S_IMODE(os.stat(provenance.sidecar_path(path)).st_mode) == 0o640 finally: os.umask(previous) def test_a_record_name_too_long_for_the_filesystem_counts_as_absent(tmp_path): # A 244-character name fits, but its record name (16 more) cannot exist. provenance.remove(tmp_path / ("n" * 244)) def test_names_too_long_for_a_record_still_download_and_delete(source, tmp_path): # 244 UTF-8 bytes fits ext4's 255-byte limit, but the record name does not # (on filesystems that count characters, the record simply fits). cache = Cache("provenance", cache_root=tmp_path / "cache") name = "é" * 120 + ".csv" path = cache.fetch(source.as_uri(), filename=name, record_provenance=True) assert Path(path).read_bytes() == DATA cache.delete_url(source.as_uri()) assert not os.path.exists(path) and not os.path.exists(provenance.sidecar_path(path)) def test_secrets_in_the_source_url_are_not_recorded(tmp_path, monkeypatch): response = SimpleNamespace( headers={"Content-Length": str(len(DATA))}, raise_for_status=lambda: None, iter_content=lambda **kw: iter([DATA]), close=lambda: None) monkeypatch.setattr(download.requests, "get", lambda *a, **kw: response) url = "https://user:secret@example.org:8443/data.csv?X-Amz-Signature=abc#part" path = fetch_file(url, destination=tmp_path / "data.csv", record_provenance=True) assert inspect_file(path).source_url == "https://example.org:8443/data.csv" assert "secret" not in Path(provenance.sidecar_path(path)).read_text() def test_redaction_keeps_ipv6_hosts_and_ports(): assert provenance.redact_url("http://[::1]:8000/a?b=c") == "http://[::1]:8000/a" assert provenance.redact_url("file:///tmp/data.csv") == "file:///tmp/data.csv" def test_delete_url_removes_records(source, tmp_path): cache = Cache("provenance", cache_root=tmp_path / "cache") path = cache.fetch(source.as_uri(), filename="data.csv", record_provenance=True) assert os.path.exists(provenance.sidecar_path(path)) cache.delete_url(source.as_uri()) assert not os.path.exists(path) and not os.path.exists(provenance.sidecar_path(path)) def test_record_provenance_must_be_a_boolean(source, tmp_path): with pytest.raises(ValueError, match="record_provenance"): fetch(source, tmp_path / "data.csv", record_provenance="yes") ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7170298 datacache-1.27.1/tests/test_raw_downloads.py0000644000076500000000000001025315264220442020635 0ustar00iskanderwheel"""Raw downloads keep payload bytes at exact destinations without conversion.""" import gzip from hashlib import sha256 from pathlib import Path import zipfile import pytest from datacache import Cache, FileValidationError, fetch_file, inspect_file from datacache import download, provenance @pytest.fixture(params=['gz', 'zip', 'html']) def source(tmp_path, request): path = tmp_path / ('source.' + request.param) payload = b'
name
gene
' if request.param == 'gz': path.write_bytes(gzip.compress(payload)) elif request.param == 'zip': with zipfile.ZipFile(path, 'w') as archive: archive.writestr('inside.txt', payload) else: path.write_bytes(payload) return path @pytest.mark.parametrize('api', ['destination', 'filename', 'cache', 'inferred']) def test_raw_payload_validation_progress_provenance_and_reuse(source, tmp_path, api): payload = source.read_bytes() digest = sha256(payload).hexdigest() events = [] options = dict(raw=True, expected_sha256=digest, expected_size=len(payload), record_provenance=True, chunk_size=7, progress_callback=lambda *args: events.append(args)) if api == 'destination': call = lambda: fetch_file(source.as_uri(), destination=tmp_path / 'result.csv', **options) elif api == 'filename': call = lambda: fetch_file(source.as_uri(), filename='result.csv', cache_root=tmp_path / 'cache', **options) elif api == 'cache': cache = Cache(cache_root=tmp_path / 'cache') call = lambda: cache.fetch(source.as_uri(), filename='result.csv', **options) else: call = lambda: fetch_file(source.as_uri(), cache_root=tmp_path / 'cache', **options) path = Path(call()) assert path.read_bytes() == payload assert api == 'inferred' or path.name == 'result.csv' assert events[-1] == (len(payload), len(payload)) info = inspect_file(path, expected_sha256=digest) assert info.verified assert info.recorded_sha256 == digest record = Path(provenance.sidecar_path(path)).read_bytes() events.clear() source.unlink() assert call() == str(path) assert not events assert Path(provenance.sidecar_path(path)).read_bytes() == record if api == 'cache': cache.delete_url(source.as_uri()) assert not path.exists() assert not Path(provenance.sidecar_path(path)).exists() def test_raw_validation_failure_preserves_destination_and_provenance(source, tmp_path): dest = tmp_path / 'result.csv' fetch_file(source.as_uri(), destination=dest, raw=True, record_provenance=True) original = dest.read_bytes() record = Path(provenance.sidecar_path(dest)).read_bytes() source.write_bytes(b'changed source') with pytest.raises(FileValidationError, match='SHA-256'): fetch_file(source.as_uri(), destination=dest, raw=True, force=True, expected_sha256=sha256(original).hexdigest(), record_provenance=True) assert dest.read_bytes() == original assert Path(provenance.sidecar_path(dest)).read_bytes() == record assert not list(tmp_path.glob('.datacache-*')) def test_raw_html_does_not_invoke_parser(tmp_path, monkeypatch): source = tmp_path / 'not-a-table.html' source.write_bytes(b'

No table to convert

') def fail(*args, **kwargs): pytest.fail('raw mode attempted HTML conversion') monkeypatch.setattr(download.pd, 'read_html', fail) path = fetch_file(source.as_uri(), destination=tmp_path / 'result.csv', raw=True) assert Path(path).read_bytes() == source.read_bytes() @pytest.mark.parametrize('options', [dict(raw=True, decompress=True), dict(raw='yes')]) @pytest.mark.parametrize('api', ['fetch_file', 'cache']) def test_invalid_raw_options_have_no_side_effects(tmp_path, options, api): root = tmp_path / 'missing' with pytest.raises(ValueError, match='raw'): if api == 'fetch_file': fetch_file('https://example.invalid/data.gz', destination=root / 'data', **options) else: Cache(cache_root=root).fetch('https://example.invalid/data.gz', **options) assert not root.exists() ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792090401.717166 datacache-1.27.1/tests/test_resume.py0000644000076500000000000006055415264220442017303 0ustar00iskanderwheel"""Real local HTTP transfers, interrupted on deterministic byte boundaries.""" from concurrent.futures import ThreadPoolExecutor from hashlib import sha256 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path import os import json import socket import threading import pytest from datacache import Cache, FileValidationError, discard_partial, fetch_file, inspect_file from datacache.resume import _state_directory PAYLOAD = b'0123456789abcdef' * 8 DIGEST = sha256(PAYLOAD).hexdigest() pytestmark = pytest.mark.skipif(os.name != 'posix', reason='POSIX resume locks') @pytest.fixture def server(): servers = [] def start(*actions): actions = list(actions) requests = [] class Handler(BaseHTTPRequestHandler): def do_GET(self): requests.append(dict(self.headers)) action = actions.pop(0) if actions else {} offset = int(self.headers.get('Range', 'bytes=0-')[6:-1]) status = action.get('status', 206 if offset else 200) if status == 200: offset = 0 body = action.get('body', PAYLOAD)[offset:] self.send_response(status) self.send_header('Content-Length', str(len(body))) if action.get('etag', '"v1"') is not None: self.send_header('ETag', action.get('etag', '"v1"')) if action.get('modified'): self.send_header('Last-Modified', action['modified']) if status == 206: self.send_header('Content-Range', action.get( 'range', 'bytes %d-%d/%d' % (offset, len(PAYLOAD) - 1, len(PAYLOAD)))) if action.get('encoding'): self.send_header('Content-Encoding', action['encoding']) self.end_headers() try: self.wfile.write(body[:action.get('drop', len(body))]) self.wfile.flush() if 'drop' in action: self.connection.shutdown(socket.SHUT_RDWR) except (BrokenPipeError, ConnectionResetError, OSError): pass def log_message(self, *args): pass srv = ThreadingHTTPServer(('127.0.0.1', 0), Handler) thread = threading.Thread(target=srv.serve_forever, daemon=True) thread.start() servers.append((srv, thread)) return 'http://127.0.0.1:%d/data' % srv.server_port, requests yield start for srv, thread in servers: srv.shutdown() srv.server_close() thread.join() def fetch(url, dest, **options): defaults = dict(resume=True, expected_sha256=DIGEST, expected_size=len(PAYLOAD), chunk_size=8, retry_backoff=0) defaults.update(options) return fetch_file(url, destination=dest, **defaults) def interrupt(url, dest, **options): def callback(done, total): if done >= 16: raise KeyboardInterrupt with pytest.raises(KeyboardInterrupt): fetch(url, dest, progress_callback=callback, **options) @pytest.mark.parametrize('expected_sha256', [DIGEST, None]) def test_dropped_connection_resumes_and_records(server, tmp_path, expected_sha256): url, requests = server({'drop': 32}) events = [] dest = tmp_path / 'data' fetch(url, dest, progress_callback=lambda *args: events.append(args), record_provenance=True, expected_sha256=expected_sha256) assert requests[1]['Range'] == 'bytes=32-' assert requests[1]['If-Range'] == '"v1"' assert all(r['Accept-Encoding'] == 'identity' for r in requests) assert events[-1] == (len(PAYLOAD), len(PAYLOAD)) assert [e[0] for e in events] == list(range(8, len(PAYLOAD) + 1, 8)) assert dest.read_bytes() == PAYLOAD assert inspect_file(dest).recorded_sha256 == expected_sha256 assert not (_state_directory(dest) / 'partial').exists() @pytest.mark.parametrize('action', [ {'status': 200}, {'range': 'bytes 0-127/128'}, {'range': 'nonsense'}, {'status': 416}, {'etag': '"v2"'}, ]) @pytest.mark.parametrize('expected_sha256', [DIGEST, None]) def test_protocol_changes_restart_safely(server, tmp_path, action, expected_sha256): url, requests = server({}, action) dest = tmp_path / 'data' interrupt(url, dest, expected_sha256=expected_sha256) fetch(url, dest, max_retries=0, expected_sha256=expected_sha256) assert requests[1]['Range'] == 'bytes=16-' assert dest.read_bytes() == PAYLOAD if action.get('status') != 200: assert 'Range' not in requests[2] @pytest.mark.parametrize('expected_sha256', [DIGEST, None]) def test_interruption_preserves_private_partial_and_old_destination(server, tmp_path, expected_sha256): url, requests = server() dest = tmp_path / 'data' dest.write_bytes(b'old') def callback(done, total): raise KeyboardInterrupt with pytest.raises(KeyboardInterrupt): fetch(url, dest, force=True, progress_callback=callback, expected_sha256=expected_sha256) assert dest.read_bytes() == b'old' partial = _state_directory(dest) / 'partial' assert partial.read_bytes() == PAYLOAD[:8] assert partial.stat().st_mode & 0o077 == 0 fetch(url, dest, force=True, expected_sha256=expected_sha256) assert requests[1]['Range'] == 'bytes=8-' assert dest.read_bytes() == PAYLOAD def test_discard_is_explicit_and_does_not_touch_destination(server, tmp_path): dest = tmp_path / 'data' discard_partial(dest) assert list(tmp_path.iterdir()) == [] url, requests = server() interrupt(url, dest) discard_partial(dest) fetch(url, dest) assert 'Range' not in requests[1] discard_partial(dest) assert dest.read_bytes() == PAYLOAD @pytest.mark.parametrize('content', [PAYLOAD, b'x' * len(PAYLOAD), PAYLOAD + b'oversized']) def test_complete_partials_validate_or_restart_without_range(server, tmp_path, content): url, requests = server() dest = tmp_path / 'data' interrupt(url, dest) (_state_directory(dest) / 'partial').write_bytes(content) fetch(url, dest) assert len(requests) == (1 if content == PAYLOAD else 2) assert all('Range' not in request for request in requests) assert dest.read_bytes() == PAYLOAD def test_integrity_failure_never_replaces_old_file(server, tmp_path): url, requests = server({'body': b'x' * len(PAYLOAD)}) dest = tmp_path / 'data' dest.write_bytes(b'old') with pytest.raises(FileValidationError, match='SHA-256'): fetch(url, dest, force=True) assert dest.read_bytes() == b'old' assert (_state_directory(dest) / 'partial').stat().st_size == 0 assert len(requests) == 1 def test_partial_persists_after_exhausted_retries(server, tmp_path): url, requests = server({'drop': 24}) dest = tmp_path / 'data' import requests as http with pytest.raises(http.exceptions.ChunkedEncodingError): fetch(url, dest, max_retries=0) assert not dest.exists() fetch(url, dest) assert requests[1]['Range'] == 'bytes=24-' @pytest.mark.parametrize('expected_sha256', [DIGEST, None]) def test_concurrent_first_calls_download_once(server, tmp_path, expected_sha256): url, requests = server() dest = tmp_path / 'data' with ThreadPoolExecutor(max_workers=4) as pool: paths = list(pool.map(lambda _: fetch(url, dest, expected_sha256=expected_sha256), range(4))) assert paths == [str(dest)] * 4 assert dest.read_bytes() == PAYLOAD assert len(requests) == 1 @pytest.mark.parametrize('kind', ['symlink', 'fifo', 'hardlink']) def test_unsafe_partial_is_rejected(server, tmp_path, kind): url, requests = server() dest = tmp_path / 'data' interrupt(url, dest) partial = _state_directory(dest) / 'partial' partial.unlink() target = tmp_path / 'precious' target.write_bytes(b'precious') if kind == 'symlink': partial.symlink_to(target) elif kind == 'fifo': os.mkfifo(partial) else: os.link(target, partial) with pytest.raises(FileValidationError): fetch(url, dest) assert target.read_bytes() == b'precious' assert len(requests) == 1 @pytest.mark.parametrize('options', [ {'expected_size': None}, {'decompress': True}, {'resume': 'yes'}, ]) def test_invalid_resume_options_have_no_side_effects(tmp_path, options): dest = tmp_path / 'missing' / 'file' with pytest.raises(ValueError): fetch('https://example.test/file', dest, **options) assert not dest.parent.exists() def test_cache_forwards_resume(server, tmp_path): url, requests = server({'drop': 16}) path = Cache(cache_root=tmp_path).fetch(url, filename='data', resume=True, expected_sha256=DIGEST, expected_size=len(PAYLOAD), chunk_size=8, retry_backoff=0) assert Path(path).read_bytes() == PAYLOAD assert requests[1]['Range'] == 'bytes=16-' def test_resumable_bundle_keeps_work_across_calls(server, tmp_path): from datacache import install_bundle, inspect_bundle url, requests = server() dest = tmp_path / 'bundle' assets = {'data': dict(url=url, sha256=DIGEST, size=len(PAYLOAD))} def callback(done, total): if done >= 16: raise KeyboardInterrupt with pytest.raises(KeyboardInterrupt): install_bundle(dest, assets, download_options=dict( resume=True, chunk_size=8, progress_callback=callback)) assert inspect_bundle(dest, assets).status == 'missing' paths = install_bundle(dest, assets, download_options=dict(resume=True, chunk_size=8)) assert requests[1]['Range'] == 'bytes=16-' assert Path(paths['data']).read_bytes() == PAYLOAD assert inspect_bundle(dest, assets).verified @pytest.mark.parametrize('expected_sha256', [DIGEST, None]) def test_encoded_http_body_is_rejected(server, tmp_path, expected_sha256): url, _ = server({'encoding': 'gzip'}) dest = tmp_path / 'data' with pytest.raises(FileValidationError, match='identity'): fetch(url, dest, expected_sha256=expected_sha256) assert not dest.exists() def test_publication_failure_keeps_partial_private_and_reusable(server, tmp_path, monkeypatch): from datacache import download url, requests = server() dest = tmp_path / 'data' original = download._publish_file def fail(*args, **kwargs): raise OSError('publication failure') monkeypatch.setattr(download, '_publish_file', fail) with pytest.raises(OSError): fetch(url, dest) partial = _state_directory(dest) / 'partial' assert partial.read_bytes() == PAYLOAD assert partial.stat().st_mode & 0o077 == 0 monkeypatch.setattr(download, '_publish_file', original) fetch(url, dest) assert len(requests) == 1 assert dest.read_bytes() == PAYLOAD def test_changed_integrity_metadata_discards_partial(server, tmp_path): url, requests = server() dest = tmp_path / 'data' interrupt(url, dest) with pytest.raises(FileValidationError): fetch(url, dest, expected_sha256='0' * 64) assert 'Range' not in requests[1] def test_server_exceeding_size_keeps_disk_usage_bounded(server, tmp_path): url, requests = server({'body': PAYLOAD * 2}) dest = tmp_path / 'data' with pytest.raises(FileValidationError, match='exceeds'): fetch(url, dest) assert not dest.exists() assert (_state_directory(dest) / 'partial').stat().st_size == 0 def test_callback_transport_error_is_not_retried(server, tmp_path): import requests as http url, requests = server() def fail(done, total): raise http.ConnectionError('callback failed') with pytest.raises(http.ConnectionError, match='callback'): fetch(url, tmp_path / 'data', progress_callback=fail) assert len(requests) == 1 def test_transient_status_uses_bounded_retry_policy(server, tmp_path, monkeypatch): from datacache import resume url, requests = server({'status': 429}, {'status': 503}) waits = [] monkeypatch.setattr(resume.time, 'sleep', waits.append) fetch(url, tmp_path / 'data', retry_backoff=2, retry_max_delay=3) assert len(requests) == 3 assert waits == [2, 3] def test_repeated_wrong_range_is_rejected(server, tmp_path): url, requests = server({}, {'range': 'invalid'}, {'status': 206, 'range': 'invalid'}) dest = tmp_path / 'data' interrupt(url, dest) with pytest.raises(FileValidationError, match='Content-Range'): fetch(url, dest) assert not dest.exists() assert len(requests) == 3 def test_changed_last_modified_restarts_without_using_a_weak_if_range(server, tmp_path): old = 'Tue, 01 Sep 2026 00:00:00 GMT' new = 'Wed, 02 Sep 2026 00:00:00 GMT' url, requests = server({'etag': None, 'modified': old}, {'etag': None, 'modified': new}, {'etag': None, 'modified': new}) dest = tmp_path / 'data' interrupt(url, dest) fetch(url, dest) assert 'Range' in requests[1] and 'If-Range' not in requests[1] assert 'Range' not in requests[2] assert dest.read_bytes() == PAYLOAD def test_resumable_bundle_repairs_corrupt_unpublished_asset(server, tmp_path): from datacache import install_bundle url, requests = server() dest = tmp_path / 'bundle' assets = {name: dict(url=url, sha256=DIGEST, size=len(PAYLOAD)) for name in ('first', 'second')} completed = [] def callback(done, total): if completed: raise KeyboardInterrupt if done == total: completed.append(done) with pytest.raises(KeyboardInterrupt): install_bundle(dest, assets, download_options=dict( resume=True, chunk_size=8, progress_callback=callback)) working = next(dest.glob('.staging-*')) (working / 'files' / 'first').write_bytes(b'corrupted private working state') paths = install_bundle(dest, assets, download_options=dict(resume=True, chunk_size=8)) assert all(Path(path).read_bytes() == PAYLOAD for path in paths.values()) assert len(requests) == 4 assert requests[-1]['Range'] == 'bytes=8-' def test_failed_bundle_rename_keeps_private_resumable_work(server, tmp_path, monkeypatch): from datacache import install_bundle url, requests = server() dest = tmp_path / 'bundle' assets = {'data': dict(url=url, sha256=DIGEST, size=len(PAYLOAD))} replace = os.replace def fail_bundle_rename(source, target): if Path(target).parent == dest / 'bundles': assert Path(source).parent.stat().st_mode & 0o077 == 0 raise OSError('cannot publish bundle') return replace(source, target) monkeypatch.setattr(os, 'replace', fail_bundle_rename) with pytest.raises(OSError, match='cannot publish bundle'): install_bundle(dest, assets, download_options=dict(resume=True, chunk_size=8)) working = next(dest.glob('.staging-*')) assert working.stat().st_mode & 0o077 == 0 assert (working / 'files' / 'data').read_bytes() == PAYLOAD monkeypatch.setattr(os, 'replace', replace) paths = install_bundle(dest, assets, download_options=dict(resume=True, chunk_size=8)) assert Path(paths['data']).read_bytes() == PAYLOAD assert len(requests) == 1 assert not working.exists() # The private resume lock stays out of the shared bundle. assert not list(Path(paths['data']).parent.rglob('.datacache-resume-*')) @pytest.mark.parametrize('etag', [None, 'W/"v1"', 'v1', '"v1", "v2"', '"has space"', '"has\ttab"']) def test_size_only_requires_strong_etag_before_reading_body(server, tmp_path, etag): url, requests = server({'etag': etag, 'modified': 'Tue, 01 Sep 2026 00:00:00 GMT'}) dest = tmp_path / 'data' dest.write_bytes(b'old') with pytest.raises(FileValidationError, match='requires a strong ETag'): fetch(url, dest, expected_sha256=None, force=True) assert dest.read_bytes() == b'old' assert len(requests) == 1 assert (_state_directory(dest) / 'partial').stat().st_size == 0 @pytest.mark.parametrize('status', [200, 206]) def test_size_only_changed_representation_restarts_with_new_bytes(server, tmp_path, status): new = b'x' * len(PAYLOAD) url, requests = server({}, {'status': status, 'etag': '"v2"', 'body': new}, {'etag': '"v2"', 'body': new}) dest = tmp_path / 'data' interrupt(url, dest, expected_sha256=None) fetch(url, dest, expected_sha256=None, max_retries=0) assert requests[1]['If-Range'] == '"v1"' assert dest.read_bytes() == new if status == 206: assert 'Range' not in requests[2] @pytest.mark.parametrize('etag', [None, 'W/"v1"']) def test_size_only_missing_resumed_validator_restarts_then_rejects(server, tmp_path, etag): url, requests = server({}, {'etag': etag}, {'etag': etag}) dest = tmp_path / 'data' dest.write_bytes(b'old') interrupt(url, dest, expected_sha256=None, force=True) with pytest.raises(FileValidationError, match='requires a strong ETag'): fetch(url, dest, expected_sha256=None, force=True, max_retries=0) assert requests[1]['If-Range'] == '"v1"' assert 'Range' not in requests[2] assert dest.read_bytes() == b'old' assert (_state_directory(dest) / 'partial').stat().st_size == 0 @pytest.mark.parametrize('validator', [ None, {}, 'not a validator', {'header': 'Last-Modified', 'value': 'Tue, 01 Sep 2026 00:00:00 GMT'}, {'header': 'ETag', 'value': 'W/"v1"'}, {'header': 'ETag', 'value': '"v1"\r\nInjected: yes'}, ]) def test_size_only_unvalidated_saved_partial_restarts(server, tmp_path, validator): url, requests = server() dest = tmp_path / 'data' interrupt(url, dest, expected_sha256=None) state = _state_directory(dest) metadata = json.loads((state / 'metadata.json').read_text()) metadata['validator'] = validator (state / 'metadata.json').write_text(json.dumps(metadata)) (state / 'partial').write_bytes(b'x' * 16) fetch(url, dest, expected_sha256=None) assert 'Range' not in requests[1] assert dest.read_bytes() == PAYLOAD @pytest.mark.parametrize('content', [PAYLOAD, b'x' * len(PAYLOAD), PAYLOAD + b'oversized']) def test_size_only_complete_partial_always_requests_fresh_bytes(server, tmp_path, content): new = b'y' * len(PAYLOAD) url, requests = server({}, {'etag': '"v2"', 'body': new}) dest = tmp_path / 'data' interrupt(url, dest, expected_sha256=None) (_state_directory(dest) / 'partial').write_bytes(content) fetch(url, dest, expected_sha256=None) assert len(requests) == 2 assert 'Range' not in requests[1] assert dest.read_bytes() == new @pytest.mark.parametrize('etag', ['"empty"', None]) def test_size_only_empty_download_still_requires_response_validator(server, tmp_path, etag): url, requests = server({'body': b'', 'etag': etag}) dest = tmp_path / 'data' if etag is None: with pytest.raises(FileValidationError, match='strong ETag'): fetch(url, dest, expected_sha256=None, expected_size=0) assert not dest.exists() else: fetch(url, dest, expected_sha256=None, expected_size=0) assert dest.read_bytes() == b'' assert len(requests) == 1 def test_hash_pinned_download_can_resume_without_validator(server, tmp_path): url, requests = server({'drop': 24, 'etag': None}, {'etag': None}) dest = tmp_path / 'data' fetch(url, dest) assert requests[1]['Range'] == 'bytes=24-' assert 'If-Range' not in requests[1] assert dest.read_bytes() == PAYLOAD @pytest.mark.parametrize('api', ['fetch_file', 'cache']) @pytest.mark.parametrize('suffix', ['.gz', '.zip', '.html']) @pytest.mark.parametrize('expected_sha256', [DIGEST, None]) def test_raw_resume_at_arbitrary_output_name(server, tmp_path, api, suffix, expected_sha256): url, requests = server({'drop': 16}) url += suffix + '?token=secret' options = dict(raw=True, resume=True, expected_sha256=expected_sha256, expected_size=len(PAYLOAD), chunk_size=8, retry_backoff=0, record_provenance=True) if api == 'fetch_file': call = lambda: fetch_file(url, destination=tmp_path / 'data.csv', **options) else: cache = Cache(cache_root=tmp_path) call = lambda: cache.fetch(url, filename='data.csv', **options) path = Path(call()) assert path.name == 'data.csv' assert path.read_bytes() == PAYLOAD assert requests[1]['Range'] == 'bytes=16-' assert requests[1]['If-Range'] == '"v1"' assert inspect_file(path).recorded_sha256 == expected_sha256 assert call() == str(path) assert len(requests) == 2 def test_leftovers_from_an_interrupted_attempt_are_never_published(server, tmp_path, monkeypatch): from datacache import download, install_bundle url, requests = server() dest = tmp_path / 'bundle' assets = {'data': dict(url=url, sha256=DIGEST, size=len(PAYLOAD))} fetch_file = download.fetch_file def killed(url, *, destination, **options): (Path(destination).parent / '.datacache-install-deadbeef').write_bytes(b'partial') raise KeyboardInterrupt monkeypatch.setattr(download, 'fetch_file', killed) with pytest.raises(KeyboardInterrupt): install_bundle(dest, assets, download_options=dict(resume=True)) monkeypatch.setattr(download, 'fetch_file', fetch_file) paths = install_bundle(dest, assets, download_options=dict(resume=True)) bundle = Path(paths['data']).parent assert sorted(path.name for path in bundle.iterdir()) == ['.datacache-manifest.json', 'data'] def test_auto_resume_resumes_an_interrupted_transfer(server, tmp_path): url, requests = server({'drop': 10}, {}) dest = tmp_path / 'data' fetch(url, dest, resume='auto') assert dest.read_bytes() == PAYLOAD assert requests[1]['Range'] == 'bytes=8-' # The complete chunks before the drop. def test_auto_resume_downloads_in_full_when_the_server_cant_resume(server, tmp_path): from datacache.resume import CannotResume url, requests = server({'etag': None}, {'etag': None}) dest = tmp_path / 'data' # resume=True refuses; "auto" downloads the file in full instead. with pytest.raises(CannotResume, match='strong ETag'): fetch(url, dest, expected_sha256=None) fetch(url, dest, expected_sha256=None, resume='auto') assert dest.read_bytes() == PAYLOAD assert not (_state_directory(dest) / 'partial').exists() @pytest.mark.parametrize('options', [ dict(expected_size=None), # Size unknown. dict(validator=lambda path: True), # Validators need the plain path. ]) def test_auto_resume_downloads_normally_when_resuming_is_impossible(server, tmp_path, options): url, requests = server({'etag': None}) dest = tmp_path / 'data' fetch(url, dest, resume='auto', **options) assert dest.read_bytes() == PAYLOAD assert 'Range' not in requests[0] def test_auto_resume_accepts_decompression_and_local_files(tmp_path): import gzip source = tmp_path / 'data.gz' source.write_bytes(gzip.compress(PAYLOAD, mtime=0)) out = fetch_file(source.as_uri(), destination=tmp_path / 'data', decompress=True, resume='auto') assert Path(out).read_bytes() == PAYLOAD with pytest.raises(ValueError, match='resume must be True, False or "auto"'): fetch_file(source.as_uri(), destination=tmp_path / 'other', resume='yes') def test_installs_accept_auto_resume_for_any_source(tmp_path): from hashlib import sha256 from datacache import install_bundle source = tmp_path / 'local' source.write_bytes(PAYLOAD) assets = {'data': dict(url=source.as_uri(), sha256=sha256(PAYLOAD).hexdigest(), size=len(PAYLOAD))} paths = install_bundle(tmp_path / 'bundle', assets, download_options={'resume': 'auto'}) assert Path(paths['data']).read_bytes() == PAYLOAD def test_auto_resume_falls_back_when_this_filesystem_cant_hold_resume_state(server, tmp_path, monkeypatch): import errno from datacache import CannotResume, resume def no_locks(*args, **kwargs): raise OSError(errno.ENOLCK, 'No locks available') # As on some NFS mounts. monkeypatch.setattr(resume, 'file_lock', no_locks) url, requests = server({}, {}) with pytest.raises(CannotResume, match='cannot lock'): fetch(url, tmp_path / 'strict') fetch(url, tmp_path / 'data', resume='auto') assert (tmp_path / 'data').read_bytes() == PAYLOAD def test_auto_resume_falls_back_on_unexpected_success_statuses(server, tmp_path): from datacache import CannotResume url, requests = server({'status': 203}, {'status': 203}, {'status': 203}) with pytest.raises(CannotResume, match='200 or 206'): fetch(url, tmp_path / 'strict') fetch(url, tmp_path / 'data', resume='auto') assert (tmp_path / 'data').read_bytes() == PAYLOAD ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7172575 datacache-1.27.1/tests/test_sqlite_schema_regressions.py0000644000076500000000000001363415264220442023244 0ustar00iskanderwheel"""Full-text cache rebuilds and distinct indexes must survive schema cleanup.""" from contextlib import closing import sqlite3 import stat import pandas as pd import pytest from datacache import connect_if_correct_version, db_from_dataframe, db_from_dataframes_with_absolute_path from datacache.database import Database, quote_identifier def index_columns(connection, table_name): return { row[1]: tuple(info[2] for info in connection.execute( "PRAGMA index_info(%s)" % quote_identifier(row[1]))) for row in connection.execute("PRAGMA index_list(%s)" % quote_identifier(table_name)) } @pytest.fixture(params=["fts3", "fts4", "fts5"]) def full_text_database(tmp_path, request): path = tmp_path / "records.db" with closing(db_from_dataframe(path, "records", pd.DataFrame({"value": [42]}))) as connection: try: connection.execute('CREATE VIRTUAL TABLE "full text" USING %s(text)' % request.param) except sqlite3.OperationalError as error: if "no such module" in str(error): pytest.skip("SQLite build does not provide " + request.param) raise connection.execute('INSERT INTO "full text" VALUES (?)', ("original cached text",)) connection.commit() path.chmod(0o640) return path @pytest.mark.parametrize("overwrite", [False, True]) def test_rebuild_with_full_text_tables(full_text_database, overwrite): path = full_text_database before = path.stat() with closing(db_from_dataframe(path, "records", pd.DataFrame({"value": [99]}), overwrite=overwrite, version=2)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(99,)] assert connection.execute("SELECT version FROM _datacache_metadata").fetchone() == (2,) assert connection.execute("SELECT name FROM sqlite_master ORDER BY name").fetchall() == [ ("_datacache_metadata",), ("records",)] assert path.stat().st_ino == before.st_ino assert stat.S_IMODE(path.stat().st_mode) == 0o640 @pytest.mark.parametrize("failure", ["constraint", "interrupt"]) def test_failed_full_text_rebuild_restores_search(full_text_database, monkeypatch, failure): path = full_text_database before = path.stat() original_bytes = path.read_bytes() error = sqlite3.IntegrityError if failure == "interrupt": def interrupt(*args, **kwargs): raise KeyboardInterrupt() monkeypatch.setattr(Database, "_create_indices", interrupt) error = KeyboardInterrupt with pytest.raises(error): db_from_dataframe(path, "records", pd.DataFrame({"value": [1, 1]}), primary_key="value" if failure == "constraint" else None, overwrite=True, version=2) with closing(connect_if_correct_version(path, 1)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(42,)] assert connection.execute( 'SELECT text FROM "full text" WHERE "full text" MATCH ?', ("original",) ).fetchall() == [("original cached text",)] assert path.read_bytes() == original_bytes assert path.stat().st_ino == before.st_ino assert stat.S_IMODE(path.stat().st_mode) == 0o640 def test_colliding_composite_indexes_are_all_created(tmp_path): frame = pd.DataFrame({"a_b": [1], "c": [2], "a": [3], "b_c": [4]}) groups = [("a_b", "c"), ("a", "b_c"), ("a_b", "c"), ("a", "b_c")] with closing(db_from_dataframe(tmp_path / "records.db", "records", frame, indices=groups)) as connection: indexes = index_columns(connection, "records") assert len(indexes) == 2 assert set(indexes.values()) == {("a_b", "c"), ("a", "b_c")} assert indexes["records_index_a_b_c"] == ("a_b", "c") def test_index_names_colliding_across_tables(tmp_path): # Both legacy names are a_index_b_index_c; different tables still need # separate indexes because index names share the database namespace. frames = {"a": pd.DataFrame({"b_index_c": [1]}), "a_index_b": pd.DataFrame({"c": [2]})} groups = {"a": [("b_index_c",)], "a_index_b": [("c",)]} with closing(db_from_dataframes_with_absolute_path( tmp_path / "records.db", frames, table_names_to_indices=groups)) as connection: assert list(index_columns(connection, "a").values()) == [("b_index_c",)] assert list(index_columns(connection, "a_index_b").values()) == [("c",)] def test_index_collision_checks_tables_views_and_partial_indexes(tmp_path): db = Database(tmp_path / "records.db") try: db.connection.executescript(''' CREATE TABLE "odd table" ("quoted\"\"column" INT); CREATE VIEW "odd table_index_quoted\"\"column" AS SELECT 1; CREATE TABLE "odd table_index_quoted\"\"column_2" (value INT); CREATE INDEX "odd table_index_quoted\"\"column_3" ON "odd table" ("quoted\"\"column") WHERE "quoted\"\"column" > 0; ''') db._create_index("odd table", ['quoted"column']) db._create_index("odd table", ['quoted"column']) indexes = index_columns(db.connection, "odd table") assert len(indexes) == 2 # Original partial index plus a full index. assert set(indexes.values()) == {('quoted"column',)} assert sum(row[4] == 0 for row in db.connection.execute('PRAGMA index_list("odd table")')) == 1 finally: db.connection.close() def test_legacy_cache_hit_does_not_add_missing_index(tmp_path): path = tmp_path / "records.db" frame = pd.DataFrame({"a_b": [1], "c": [2], "a": [3], "b_c": [4]}) with closing(db_from_dataframe(path, "records", frame, indices=[("a_b", "c")])): pass before = path.read_bytes() with closing(db_from_dataframe(path, "records", frame, indices=[("a_b", "c"), ("a", "b_c")])) as connection: assert list(index_columns(connection, "records").values()) == [("a_b", "c")] assert path.read_bytes() == before ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7173464 datacache-1.27.1/tests/test_streaming_download.py0000644000076500000000000000646015264220442021657 0ustar00iskanderwheel# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Tests for streaming downloads and the progress_callback hook (regression tests for https://github.com/openvax/datacache/issues/49). These use local file:// URLs so they don't depend on the network. """ import os from tempfile import NamedTemporaryFile, mkdtemp from datacache.download import ( _stream_to_file, _download_to_temp_file, _download_and_decompress_if_necessary, ) def _make_temp_file(contents=b""): with NamedTemporaryFile(delete=False) as f: f.write(contents) return f.name def _file_url(path): return "file://" + path def test_stream_to_file_writes_all_bytes_and_reports_progress(): contents = b"abc" * 5000 # 15000 bytes src_path = _make_temp_file(contents) dst_path = _make_temp_file() progress = [] try: with open(dst_path, "wb") as dst: total = _stream_to_file( _file_url(src_path), dst, chunk_size=4096, progress_callback=lambda done, total: progress.append((done, total))) assert total == len(contents) with open(dst_path, "rb") as f: assert f.read() == contents # callback fired and the final report equals the full size assert progress, "progress_callback was never called" assert progress[-1][0] == len(contents) # byte counts are monotonically non-decreasing counts = [done for (done, _total) in progress] assert counts == sorted(counts) # more than one chunk was written for a 15kB file at 4kB chunks assert len(progress) > 1 finally: os.remove(src_path) os.remove(dst_path) def test_download_to_temp_file_streams_local_file(): contents = b"hello streaming world" src_path = _make_temp_file(contents) tmp_path = None try: tmp_path = _download_to_temp_file(_file_url(src_path)) with open(tmp_path, "rb") as f: assert f.read() == contents finally: os.remove(src_path) if tmp_path and os.path.exists(tmp_path): os.remove(tmp_path) def test_download_and_decompress_threads_progress_callback(): contents = b"x" * 1234 src_path = _make_temp_file(contents) out_dir = mkdtemp() out_path = os.path.join(out_dir, "out.bin") progress = [] try: _download_and_decompress_if_necessary( full_path=out_path, download_url=_file_url(src_path), chunk_size=100, progress_callback=lambda done, _total: progress.append(done)) with open(out_path, "rb") as f: assert f.read() == contents assert progress and progress[-1] == len(contents) finally: os.remove(src_path) if os.path.exists(out_path): os.remove(out_path) os.rmdir(out_dir) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7174883 datacache-1.27.1/tests/test_transform_helpers.py0000644000076500000000000005711415264220442021536 0ustar00iskanderwheel"""Offline regression tests for composed download/transform helpers.""" from contextlib import closing import gzip import hashlib import logging import os from pathlib import Path import re import sqlite3 import stat import pytest import pandas as pd from datacache import common, fetch_and_transform, fetch_csv_dataframe, fetch_csv_db from datacache.common import build_local_filename, name_length from datacache import database_helpers from datacache.database_helpers import _db_filenames_from_dataframe, _truncate_name @pytest.fixture def source(tmp_path, monkeypatch): monkeypatch.setattr(common, "get_data_dir", lambda subdir=None: str(tmp_path / (subdir or "default"))) source = tmp_path / "source.txt" source.write_text("complete data") return source @pytest.mark.parametrize("failure", [RuntimeError, KeyboardInterrupt]) @pytest.mark.parametrize("existing", [False, True]) def test_failed_transform_preserves_cache_and_cleans_up(source, tmp_path, failure, existing): destination = tmp_path / "project" / "result.txt" if existing: destination.parent.mkdir() destination.write_text("old contents") destination.chmod(0o640) def transformer(source_path, output_path): assert Path(source_path).parent == destination.parent assert Path(output_path).name == "result.txt" assert not Path(output_path).exists() assert stat.S_IMODE(Path(output_path).parent.stat().st_mode) & 0o077 == 0 Path(output_path).write_text("partial") raise failure("injected failure") with pytest.raises(failure): fetch_and_transform("result.txt", transformer, lambda p: Path(p).read_text(), "download.txt", source.as_uri(), subdir="project", force=existing) if existing: assert destination.read_text() == "old contents" assert stat.S_IMODE(destination.stat().st_mode) == 0o640 else: assert not destination.exists() assert not list(destination.parent.glob(".datacache-*")) def test_transform_success_reuse_and_force(source, tmp_path): root = tmp_path / "cache" calls = [] def transform(source_path, output_path): calls.append(source_path) Path(output_path).write_text(Path(source_path).read_text().upper()) return Path(output_path) options = dict(transformed_filename="result.txt", transformer=transform, loader=lambda p: Path(p), source_filename="input.txt", source_url=source.as_uri(), cache_root=root) result = fetch_and_transform(**options) assert result == root / "result.txt" assert result.read_text() == "COMPLETE DATA" source.unlink() assert fetch_and_transform(**options) == result assert len(calls) == 1 assert fetch_and_transform(**options, force=True) == result assert len(calls) == 2 assert set(root.iterdir()) == {root / "input.txt", result} def test_noop_transform_does_not_cache_empty_placeholder(source, tmp_path): with pytest.raises(RuntimeError, match="did not create"): fetch_and_transform("result.txt", lambda *args: None, lambda p: p, "input.txt", source.as_uri(), cache_root=tmp_path / "cache") assert not (tmp_path / "cache" / "result.txt").exists() assert not list((tmp_path / "cache").glob(".datacache-*")) @pytest.mark.parametrize("explicit_csv", [False, True]) @pytest.mark.parametrize("explicit_db", [False, True]) def test_csv_database_filename_defaults(source, tmp_path, explicit_csv, explicit_db): source.write_text("id;value\n1;alpha\n2;beta\n") options = dict(csv_filename="records.csv" if explicit_csv else None, db_filename="records.db" if explicit_db else None, subdir="project", sep=";") with closing(fetch_csv_db("records", source.as_uri(), **options)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(1, "alpha"), (2, "beta")] assert len(list((tmp_path / "project").glob("*.db"))) == 1 def test_dataframe_download_options_do_not_leak_to_pandas(tmp_path): source = tmp_path / "records.csv.gz" data = b"id;value\n1;hello\n" source.write_bytes(gzip.compress(data)) options = dict(cache_root=tmp_path / "cache", expected_sha256=hashlib.sha256(data).hexdigest(), expected_size=len(data), timeout=2, max_retries=0) frame = fetch_csv_dataframe(source.as_uri(), download_options=options, sep=";") assert frame.to_dict("list") == {"id": [1], "value": ["hello"]} with closing(fetch_csv_db("records", source.as_uri(), download_options=options, sep=";")) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(1, "hello")] assert list((tmp_path / "cache").glob("*.db")) def test_real_html_table_conversion(tmp_path): from datacache import fetch_file source = tmp_path / "records.html" source.write_text("" "
idlabel
1case
") expected = b"id,label\n1,case\n" destination = tmp_path / "records.csv" fetch_file(source.as_uri(), destination=destination, expected_sha256=hashlib.sha256(expected).hexdigest()) assert destination.read_bytes() == expected def test_inferred_database_names_keep_their_historical_spelling(): # Existing databases are found by name, so ordinary schemas must keep # producing exactly the old key. frame = pd.DataFrame({"id": [1], "label": ["x"], "with space": [1.5], "Éclair μ (%)": [2]}) assert _db_filenames_from_dataframe("records", frame) == ( "records_nrows1.id_INT.label_TEXT.with_space_FLOAT.Éclair_μ_(%)_INT.db", None) def test_csv_headers_cannot_place_the_database_outside_the_cache(tmp_path): cache_root = tmp_path / "deep" / "cache" source = tmp_path / "headers.csv" source.write_text('id,"/../../../escaped/x"\n1,2\n') with closing(fetch_csv_db("records", source.as_uri(), csv_filename="headers.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(1, 2)] [database] = tmp_path.rglob("*.db") assert database.parent == cache_root def test_wide_csv_infers_a_database_name_the_filesystem_accepts(tmp_path): columns = ["measurement_%02d" % i for i in range(40)] source = tmp_path / "wide.csv" source.write_text(",".join(columns) + "\n" + ",".join("1" for _ in columns) + "\n") with closing(fetch_csv_db("records", source.as_uri(), csv_filename="wide.csv", download_options={"cache_root": tmp_path / "cache"})) as connection: assert connection.execute("SELECT COUNT(*) FROM records").fetchone() == (1,) [database] = (tmp_path / "cache").glob("*.db") assert len(database.name) < 255 def test_shortened_database_names_still_distinguish_schemas(): frame = pd.DataFrame({"a": [1], "/../x": [2]}) name, historical = _db_filenames_from_dataframe("records", frame) assert re.fullmatch(r"records_nrows1\.[0-9a-f]{32}\.db", name) assert historical is None # a name that could leave the cache is never reused assert _db_filenames_from_dataframe("records", frame)[0] == name assert _db_filenames_from_dataframe("records", frame.astype({"a": float}))[0] != name @pytest.mark.parametrize("db_filename", [None, "unnamed.db"]) def test_unnamed_csv_columns_raise_the_same_clear_error(tmp_path, db_filename): source = tmp_path / "unnamed.csv" source.write_text("1,2\n3,4\n") with pytest.raises(ValueError, match="non-empty strings"): fetch_csv_db("records", source.as_uri(), csv_filename="unnamed.csv", db_filename=db_filename, download_options={"cache_root": tmp_path / "cache"}, header=None) def _write_legacy_database(path, version, rows): path.parent.mkdir(parents=True, exist_ok=True) with closing(sqlite3.connect(path)) as connection: connection.executescript( 'CREATE TABLE records (peptide TEXT, "m/z" FLOAT);' "CREATE TABLE _datacache_metadata (version INT);" "INSERT INTO _datacache_metadata VALUES (%d);" % version) connection.executemany("INSERT INTO records VALUES (?, ?)", rows) connection.commit() @pytest.fixture def mz_source(tmp_path): source = tmp_path / "mz.csv" source.write_text("peptide,m/z\nAAA,1.5\n") return source def test_slash_in_a_column_name_does_not_nest_directories(tmp_path, mz_source): cache_root = tmp_path / "cache" with closing(fetch_csv_db("records", mz_source.as_uri(), csv_filename="mz.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute('SELECT "m/z" FROM records').fetchall() == [(1.5,)] assert [path for path in cache_root.iterdir() if path.is_dir()] == [] assert len(list(cache_root.glob("*.db"))) == 1 def test_nested_database_from_an_older_release_is_reused_in_place(tmp_path, mz_source): cache_root = tmp_path / "cache" legacy = cache_root / "mz_nrows1.peptide_TEXT.m" / "z_FLOAT.db" _write_legacy_database(legacy, 1, [("OLD", 9.5)]) before = legacy.read_bytes() with closing(fetch_csv_db("records", mz_source.as_uri(), csv_filename="mz.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [("OLD", 9.5)] assert legacy.read_bytes() == before assert list(cache_root.glob("*.db")) == [] def test_new_version_of_a_nested_database_is_built_flat(tmp_path, mz_source): # The older copy is left alone: a process running an older release may # still have it open, and deleting an open SQLite file can corrupt data. cache_root = tmp_path / "cache" legacy = cache_root / "mz_nrows1.peptide_TEXT.m" / "z_FLOAT.db" _write_legacy_database(legacy, 1, [("OLD", 9.5)]) before = legacy.read_bytes() with closing(fetch_csv_db("records", mz_source.as_uri(), csv_filename="mz.csv", version=2, download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [("AAA", 1.5)] assert legacy.read_bytes() == before assert len(list(cache_root.glob("*.db"))) == 1 def test_legacy_lookup_never_opens_a_database_outside_the_cache(tmp_path): # Even when the directories a hostile header names already exist, an old # nested name is reused only if it stays inside the cache directory. cache_root = tmp_path / "deep" / "cache" (cache_root / "headers_nrows1.a_INT.").mkdir(parents=True) outside = tmp_path / "escaped" / "x_INT.db" outside.parent.mkdir() with closing(sqlite3.connect(outside)) as connection: connection.executescript("CREATE TABLE records (a INT, b INT);" "INSERT INTO records VALUES (99, 99);" "CREATE TABLE _datacache_metadata (version INT);" "INSERT INTO _datacache_metadata VALUES (1);") before = outside.read_bytes() source = tmp_path / "headers.csv" source.write_text('a,"/../../../escaped/x"\n1,2\n') with closing(fetch_csv_db("records", source.as_uri(), csv_filename="headers.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(1, 2)] assert outside.read_bytes() == before def test_rebuilds_leave_room_for_the_sqlite_journal(tmp_path): # SQLite creates "-journal" to rebuild. A 250-byte name could be # built once but never rebuilt, so such names are shortened up front, even # when a long CSV filename leaves no room for the usual digest name. schema = ".id_INT.value_INT.db" base = "r" * (250 - len("_nrows1") - len(schema)) source = tmp_path / "source.csv" source.write_text("id,value\n1,10\n") options = dict(csv_filename=base + ".csv", download_options={"cache_root": tmp_path / "cache"}) for version in (1, 2): with closing(fetch_csv_db("records", source.as_uri(), version=version, **options)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(1, 10)] [database] = (tmp_path / "cache").glob("*.db") assert len(database.name) <= 255 - len("-journal") assert database.name.startswith("rrrr") def test_database_at_a_name_too_long_to_rebuild_is_still_reused(tmp_path): # Older releases could create a 248-255 byte name. It is still reused in # place; only a rebuild moves to the shorter name. cache_root = tmp_path / "cache" source = tmp_path / "mass.csv" source.write_text("peptide,mass\nAAA,1.5\n") base = "m" * (250 - len("_nrows1") - len(".peptide_TEXT.mass_FLOAT.db")) name, historical = _db_filenames_from_dataframe(base, pd.read_csv(source)) assert len(historical) == 250 and len(name) <= 255 - len("-journal") # SQLite cannot write at this name (its journal would not fit), so, like # older releases, build under a short name and publish it with a link. _write_legacy_database(cache_root / "staged.db", 1, [("OLD", 9.5)]) os.link(cache_root / "staged.db", cache_root / historical) os.remove(cache_root / "staged.db") with closing(fetch_csv_db("records", source.as_uri(), csv_filename=base + ".csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [("OLD", 9.5)] def test_characters_some_platform_forbids_are_kept_out_of_names(tmp_path): source = tmp_path / "times.csv" source.write_text('id,time:s,"a?b",c*d,e|f\n1,2,3,4,5\n') cache_root = tmp_path / "cache" with closing(fetch_csv_db("records", source.as_uri(), csv_filename="times.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute('SELECT "time:s" FROM records').fetchall() == [(2,)] [database] = cache_root.glob("*.db") assert not set(':?*|<>"') & set(database.name) @pytest.mark.skipif(os.name == "nt", reason="':' names an alternate data stream on Windows") def test_database_with_a_colon_from_an_older_release_is_reused(tmp_path): cache_root = tmp_path / "cache" legacy = cache_root / "times_nrows1.peptide_TEXT.m:z_FLOAT.db" _write_legacy_database(legacy, 1, [("OLD", 9.5)]) source = tmp_path / "times.csv" source.write_text("peptide,m:z\nAAA,1.5\n") with closing(fetch_csv_db("records", source.as_uri(), csv_filename="times.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [("OLD", 9.5)] @pytest.mark.skipif(os.name == "nt", reason="backslash separates paths on Windows") def test_backslashes_in_an_older_name_are_ordinary_characters_on_posix(tmp_path): # On POSIX, "a\..\b" was one harmless file name, so it is reused in place. cache_root = tmp_path / "cache" legacy = cache_root / "slashes_nrows1.peptide_TEXT.a\\..\\b_FLOAT.db" _write_legacy_database(legacy, 1, [("OLD", 9.5)]) source = tmp_path / "slashes.csv" source.write_text("peptide,a\\..\\b\nAAA,1.5\n") with closing(fetch_csv_db("records", source.as_uri(), csv_filename="slashes.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [("OLD", 9.5)] @pytest.mark.parametrize("obstacle", ["file", "directory"]) def test_anything_else_at_an_older_name_is_not_reused(tmp_path, mz_source, obstacle): # Looking for an older release's database must never stop a new build. cache_root = tmp_path / "cache" cache_root.mkdir() if obstacle == "file": # the old name's parent directory is a regular file (cache_root / "mz_nrows1.peptide_TEXT.m").write_text("not a directory") else: # a directory sits where the old database would be (cache_root / "mz_nrows1.peptide_TEXT.m" / "z_FLOAT.db").mkdir(parents=True) with closing(fetch_csv_db("records", mz_source.as_uri(), csv_filename="mz.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [("AAA", 1.5)] assert len(list(cache_root.glob("*.db"))) == 1 def test_cache_names_work_where_md5_is_disabled_for_security(monkeypatch): # FIPS-mode OpenSSL rejects md5 unless it is declared not to be for security. real_md5 = hashlib.md5 def fips_md5(*args, usedforsecurity=True, **kwargs): if usedforsecurity: raise ValueError("[digital envelope routines] unsupported") return real_md5(*args, usedforsecurity=False, **kwargs) monkeypatch.setattr(hashlib, "md5", fips_md5) assert build_local_filename("https://example.org/data?id=1") assert len(build_local_filename(filename="x" * 200)) < 200 assert _db_filenames_from_dataframe("records", pd.DataFrame({"m/z": [1.5]}))[0].endswith(".db") @pytest.mark.parametrize("message", ["database is locked", "unable to open database file"]) def test_only_a_locked_older_database_stops_a_new_build(tmp_path, mz_source, monkeypatch, message): # An older release may be writing to its database; building a second copy # beside it would let the two diverge, so a lock error propagates. A file # SQLite cannot open at all just means there is nothing to reuse. cache_root = tmp_path / "cache" legacy = cache_root / "mz_nrows1.peptide_TEXT.m" / "z_FLOAT.db" _write_legacy_database(legacy, 1, [("OLD", 9.5)]) real_cached_connection = database_helpers._cached_connection def failing(path, *args): if os.fspath(path) == str(legacy): raise sqlite3.OperationalError(message) return real_cached_connection(path, *args) monkeypatch.setattr(database_helpers, "_cached_connection", failing) options = dict(csv_filename="mz.csv", download_options={"cache_root": cache_root}) if "locked" in message: with pytest.raises(sqlite3.OperationalError, match="locked"): fetch_csv_db("records", mz_source.as_uri(), **options) assert list(cache_root.glob("*.db")) == [] else: with closing(fetch_csv_db("records", mz_source.as_uri(), **options)) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [("AAA", 1.5)] def test_csv_filename_characters_some_platform_forbids_stay_out_of_names(tmp_path): source = tmp_path / "run.csv" source.write_text("id\n1\n") cache_root = tmp_path / "cache" with closing(fetch_csv_db("records", source.as_uri(), csv_filename="run:2.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(1,)] [database] = cache_root.glob("*.db") assert database.name.startswith("run_2_nrows1.") and ":" not in database.name @pytest.mark.skipif(os.name == "nt", reason="':' names an alternate data stream on Windows") def test_database_named_after_a_csv_filename_with_a_colon_is_reused(tmp_path): cache_root = tmp_path / "cache" _write_legacy_database(cache_root / "run:2_nrows1.peptide_TEXT.mass_FLOAT.db", 1, [("OLD", 9.5)]) source = tmp_path / "run.csv" source.write_text("peptide,mass\nAAA,1.5\n") with closing(fetch_csv_db("records", source.as_uri(), csv_filename="run:2.csv", download_options={"cache_root": cache_root})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [("OLD", 9.5)] def test_database_names_need_named_columns(): with pytest.raises(ValueError, match="non-empty strings"): _db_filenames_from_dataframe("records", pd.DataFrame()) def test_truncation_counts_utf8_bytes_on_every_platform(): assert _truncate_name("abc", 5) == "abc" assert _truncate_name("é" * 10, 5) == "éé" # two bytes each assert name_length("é" * 130) == 260 def test_inferred_names_measure_utf8_bytes_on_every_platform(): # 130 accented characters are 260 UTF-8 bytes but only 130 characters or # UTF-16 units. Every platform must still pick the same (digest) name. name, historical = _db_filenames_from_dataframe("records", pd.DataFrame({"é" * 130: [1]})) assert re.fullmatch(r"records_nrows1\.[0-9a-f]{32}\.db", name) assert historical == "records_nrows1.%s_INT.db" % ("é" * 130) def test_schemas_that_read_alike_get_different_digest_names(): # One column "a_INT.b:c" of text and two columns "a" and "b:c" spell out # the same way; their digest names must still differ. one = pd.DataFrame({"a_INT.b:c": ["x"]}) two = pd.DataFrame({"a": [1], "b:c": ["x"]}) assert _db_filenames_from_dataframe("records", one)[1] == _db_filenames_from_dataframe("records", two)[1] assert _db_filenames_from_dataframe("records", one)[0] != _db_filenames_from_dataframe("records", two)[0] def test_a_superseded_older_database_is_reported_once(tmp_path, mz_source, caplog): cache_root = tmp_path / "cache" legacy = cache_root / "mz_nrows1.peptide_TEXT.m" / "z_FLOAT.db" _write_legacy_database(legacy, 1, [("OLD", 9.5)]) options = dict(csv_filename="mz.csv", download_options={"cache_root": cache_root}) with caplog.at_level(logging.WARNING, logger="datacache.database_helpers"): fetch_csv_db("records", mz_source.as_uri(), version=2, **options).close() assert str(legacy) in caplog.text and "can be deleted" in caplog.text caplog.clear() fetch_csv_db("records", mz_source.as_uri(), version=3, **options).close() assert not caplog.records assert legacy.exists() def test_no_report_without_an_older_database(tmp_path, mz_source, caplog): with caplog.at_level(logging.WARNING, logger="datacache.database_helpers"): fetch_csv_db("records", mz_source.as_uri(), csv_filename="mz.csv", download_options={"cache_root": tmp_path / "cache"}).close() assert not caplog.records def test_the_older_copy_is_called_deletable_only_after_a_successful_build( tmp_path, mz_source, caplog, monkeypatch): cache_root = tmp_path / "cache" _write_legacy_database(cache_root / "mz_nrows1.peptide_TEXT.m" / "z_FLOAT.db", 1, [("OLD", 9.5)]) def failing_build(*args, **kwargs): raise OSError("disk full") monkeypatch.setattr(database_helpers, "db_from_dataframe", failing_build) with caplog.at_level(logging.WARNING, logger="datacache.database_helpers"): with pytest.raises(OSError, match="disk full"): fetch_csv_db("records", mz_source.as_uri(), csv_filename="mz.csv", version=2, download_options={"cache_root": cache_root}) assert "can be deleted" not in caplog.text def test_older_names_whose_dots_stay_in_the_cache_are_still_reused(): # "records_nrows1../x_INT.db" has components "records_nrows1.." and # "x_INT.db": it never leaves the cache, so an existing copy is reused. name, historical = _db_filenames_from_dataframe("records", pd.DataFrame({"./x": [1]})) assert historical == "records_nrows1../x_INT.db" and name != historical assert _db_filenames_from_dataframe("records", pd.DataFrame({"/../x": [1]}))[1] is None def test_csv_filenames_longer_than_255_utf8_bytes_build_a_database(tmp_path): source = tmp_path / "source.csv" source.write_text("id\n1\n") with closing(fetch_csv_db("records", source.as_uri(), csv_filename="測" * 100 + ".csv", download_options={"cache_root": tmp_path / "cache"})) as connection: assert connection.execute("SELECT * FROM records").fetchall() == [(1,)] assert all(len(path.name.encode("utf-8")) <= 255 for path in (tmp_path / "cache").iterdir()) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7175796 datacache-1.27.1/tests/test_user_agent.py0000644000076500000000000000431315264220442020126 0ustar00iskanderwheel"""Every HTTP request names datacache instead of Requests' default User-Agent.""" from hashlib import sha256 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import os from pathlib import Path import threading import pytest from datacache import fetch_bytes, fetch_file from datacache.version import __version__ PAYLOAD = b"0123456789abcdef" * 4 @pytest.fixture def server(): agents = [] class Handler(BaseHTTPRequestHandler): def do_GET(self): agent = self.headers.get("User-Agent", "") agents.append(agent) # Like IEDB, refuse any client that announces python-requests. status = 403 if "python-requests" in agent else 200 body = b"" if status == 403 else PAYLOAD self.send_response(status) self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) def log_message(self, *args): pass httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler) thread = threading.Thread(target=httpd.serve_forever, daemon=True) thread.start() yield "http://127.0.0.1:%d/data" % httpd.server_port, agents httpd.shutdown() httpd.server_close() thread.join() def plain(url, tmp_path): return Path(fetch_file(url, destination=tmp_path / "plain", raw=True, max_retries=0)).read_bytes() def resumable(url, tmp_path): return Path(fetch_file(url, destination=tmp_path / "resumed", raw=True, resume=True, expected_size=len(PAYLOAD), expected_sha256=sha256(PAYLOAD).hexdigest(), max_retries=0)).read_bytes() def in_memory(url, tmp_path): return fetch_bytes(url, max_retries=0) @pytest.mark.parametrize("fetch", [ plain, pytest.param(resumable, marks=pytest.mark.skipif( os.name != "posix", reason="POSIX resume locks")), in_memory, ]) def test_requests_name_datacache(server, tmp_path, fetch): url, agents = server assert fetch(url, tmp_path) == PAYLOAD assert len(agents) == 1 assert agents[0].startswith("datacache/%s " % __version__) assert "python-requests" not in agents[0] ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792090401.7178676 datacache-1.27.1/tests/test_verified_download.py0000644000076500000000000004467515264220442021475 0ustar00iskanderwheel"""Offline integrity/publication tests, including deterministic concurrent writers.""" import builtins from concurrent.futures import ThreadPoolExecutor import errno import gzip import hashlib import os from pathlib import Path import tempfile import threading import zipfile import pandas as pd import pytest import requests from datacache import FileValidationError, fetch_file, validate_file from datacache import common, download PAYLOAD = b"the complete installed bytes\n" * 100 def sha256(data): return hashlib.sha256(data).hexdigest() @pytest.fixture(autouse=True) def no_network(monkeypatch): def reject(*args, **kwargs): pytest.fail("unexpected network request") urlopen = download.urllib.request.urlopen def local_only(request, *args, **kwargs): assert request.full_url.startswith("file://") return urlopen(request, *args, **kwargs) monkeypatch.setattr(download.requests, "get", reject) monkeypatch.setattr(download.urllib.request, "urlopen", local_only) @pytest.fixture def source(tmp_path): source = tmp_path / "source.bin" source.write_bytes(PAYLOAD) return source def reject_mutation(*args, **kwargs): pytest.fail("inspection attempted a filesystem mutation") def test_explicit_destination_and_verified_offline_reuse(source, tmp_path, monkeypatch): destination = tmp_path / "custom" / "version-1" / "exact name;with spaces.bin" progress = [] assert fetch_file( source.as_uri(), destination=destination, expected_sha256=sha256(PAYLOAD).upper(), expected_size=len(PAYLOAD), timeout=5, chunk_size=7, progress_callback=lambda done, total: progress.append((done, total)), ) == str(destination) assert destination.read_bytes() == PAYLOAD assert progress[-1] == (len(PAYLOAD), len(PAYLOAD)) assert len(progress) > 1 source.unlink() destination.chmod(0o444) destination.parent.chmod(0o555) original_open = builtins.open def read_only_open(file, mode="r", *args, **kwargs): assert not any(flag in mode for flag in "wax+") return original_open(file, mode, *args, **kwargs) try: # Reject write attempts explicitly, even when running as root. with monkeypatch.context() as readonly: readonly.setattr(builtins, "open", read_only_open) for operation in ("makedirs", "mkdir", "replace", "remove"): readonly.setattr(download.os, operation, reject_mutation) readonly.setattr(download, "_open_staging_file", reject_mutation) assert fetch_file( "https://unavailable.invalid/file", destination=destination, expected_sha256=sha256(PAYLOAD), expected_size=len(PAYLOAD), ) == str(destination) assert validate_file(destination, sha256(PAYLOAD), len(PAYLOAD)) == str(destination) with pytest.raises(FileNotFoundError): validate_file(destination.parent / "absent" / "file") finally: destination.parent.chmod(0o755) destination.chmod(0o644) assert list(destination.parent.iterdir()) == [destination] @pytest.mark.parametrize("metadata,match", [ ({"expected_size": len(PAYLOAD) + 1}, "size mismatch"), ({"expected_sha256": "0" * 64}, "SHA-256 mismatch"), ]) def test_corrupt_cache_requires_explicit_repair(source, tmp_path, metadata, match): destination = tmp_path / "cache" / "file" destination.parent.mkdir() destination.write_bytes(PAYLOAD) with pytest.raises(FileValidationError, match=match + ".*force=True") as caught: fetch_file("https://unavailable.invalid/file", destination=destination, **metadata) assert caught.value.path == str(destination) assert destination.read_bytes() == PAYLOAD destination.write_bytes(b"truncated") fetch_file( source.as_uri(), destination=destination, force=True, expected_sha256=sha256(PAYLOAD), expected_size=len(PAYLOAD)) assert destination.read_bytes() == PAYLOAD def test_expected_metadata_with_legacy_cache_arguments(source, tmp_path, monkeypatch): monkeypatch.setattr(common, "get_data_dir", lambda subdir: str(tmp_path / subdir)) kwargs = dict(filename="legacy.bin", subdir="release-v1", expected_sha256=sha256(PAYLOAD), expected_size=len(PAYLOAD)) destination = tmp_path / "release-v1" / "legacy.bin" assert fetch_file(source.as_uri(), **kwargs) == str(destination) assert fetch_file("https://unavailable.invalid/file", **kwargs) == str(destination) destination.write_bytes(b"incomplete") with pytest.raises(FileValidationError, match="size mismatch"): fetch_file("https://unavailable.invalid/file", **kwargs) @pytest.mark.parametrize("metadata", [ {"expected_sha256": "bad"}, {"expected_sha256": 123}, {"expected_size": -1}, {"expected_size": 1.5}, {"expected_size": True}, {"chunk_size": 0}, {"chunk_size": -1}, {"chunk_size": True}, {"filename": "conflict"}, {"subdir": "conflict"}, ]) def test_invalid_arguments_do_not_create_directories(tmp_path, metadata): destination = tmp_path / "absent" / "file" with pytest.raises(ValueError): fetch_file("https://unavailable.invalid/file", destination=destination, **metadata) assert not destination.parent.exists() def test_empty_file_and_relative_destination(tmp_path, monkeypatch): source = tmp_path / "empty" source.touch() monkeypatch.chdir(tmp_path) assert fetch_file( source.as_uri(), destination="relative", expected_size=0, expected_sha256=sha256(b"")) == "relative" assert (tmp_path / "relative").read_bytes() == b"" @pytest.mark.parametrize("kind", ["gzip", "zip", "stored-gzip", "named-gzip", "html"]) def test_integrity_describes_installed_bytes(tmp_path, monkeypatch, kind): if kind == "html": source = tmp_path / "source.html" source.write_text("
value
1
") # Avoid requiring an optional HTML parser in the test environment. monkeypatch.setattr(download.pd, "read_html", lambda *a, **k: [pd.DataFrame({"value": [1]})]) destination = tmp_path / "cache" / "file.csv" installed = pd.DataFrame({"value": [1]}).to_csv(index=False).encode("utf-8") else: source = tmp_path / ("source.zip" if kind == "zip" else "source.gz") if kind == "zip": with zipfile.ZipFile(source, "w") as archive: archive.writestr("file", PAYLOAD) archive.writestr("larger", b"ignore" * len(PAYLOAD)) else: source.write_bytes(gzip.compress(PAYLOAD)) destination = tmp_path / "cache" / ("file.gz" if kind in ("stored-gzip", "named-gzip") else "file") installed = source.read_bytes() if kind == "stored-gzip" else PAYLOAD fetch_file( source.as_uri(), destination=destination, decompress=kind == "named-gzip", expected_sha256=sha256(installed), expected_size=len(installed)) assert destination.read_bytes() == installed assert list(destination.parent.iterdir()) == [destination] if kind != "stored-gzip": with pytest.raises(FileValidationError): fetch_file( source.as_uri(), destination=destination, force=True, decompress=kind == "named-gzip", expected_sha256=sha256(source.read_bytes())) assert destination.read_bytes() == installed assert list(destination.parent.iterdir()) == [destination] @pytest.mark.parametrize("existing", [False, True]) @pytest.mark.parametrize("failure", ["transfer", "callback", "gzip", "zip", "html", "size", "hash", "publish", "cancel"]) def test_failed_download_preserves_destination_and_cleans_staging( source, tmp_path, monkeypatch, existing, failure): destination = tmp_path / "cache" / ("file.csv" if failure == "html" else "file") destination.parent.mkdir() if existing: destination.write_bytes(b"previous complete artifact") kwargs = {} expected_error = Exception if failure in ("transfer", "cancel"): expected_error = requests.ConnectionError if failure == "transfer" else KeyboardInterrupt def interrupted(url, stream, **kwargs): stream.write(b"partial transfer") raise expected_error("injected failure") monkeypatch.setattr(download, "_stream_to_file", interrupted) elif failure == "callback": def interrupted_callback(*args): raise RuntimeError("progress failed") kwargs["progress_callback"] = interrupted_callback elif failure in ("gzip", "zip"): source = tmp_path / ("bad.gz" if failure == "gzip" else "bad.zip") if failure == "gzip": source.write_bytes(gzip.compress(PAYLOAD)[:-8]) else: source.write_bytes(b"invalid zip") elif failure == "html": source = tmp_path / "source.html" source.write_bytes(b"html fixture") monkeypatch.setattr(download.pd, "read_html", lambda *a, **k: [pd.DataFrame()]) def interrupted_csv(self, path, **kwargs): Path(path).write_bytes(b"partial conversion") raise RuntimeError("conversion failed") monkeypatch.setattr(download.pd.DataFrame, "to_csv", interrupted_csv) elif failure == "size": kwargs["expected_size"] = len(PAYLOAD) + 1 expected_error = FileValidationError elif failure == "hash": kwargs["expected_sha256"] = "0" * 64 expected_error = FileValidationError else: def interrupted_publish(src, dst): raise PermissionError("publication denied") monkeypatch.setattr(download.os, "replace", interrupted_publish) expected_error = PermissionError with pytest.raises(expected_error): fetch_file(source.as_uri(), destination=destination, force=True, **kwargs) if existing: assert destination.read_bytes() == b"previous complete artifact" assert list(destination.parent.iterdir()) == [destination] else: assert list(destination.parent.iterdir()) == [] @pytest.mark.parametrize("compressed", [False, True]) def test_staging_uses_destination_filesystem(source, tmp_path, monkeypatch, compressed): system_temp = tmp_path / "different-filesystem" system_temp.mkdir() monkeypatch.setattr(tempfile, "tempdir", str(system_temp)) if compressed: source = tmp_path / "source.gz" source.write_bytes(gzip.compress(PAYLOAD)) destination = tmp_path / "cache" / "file" real_temp = download._open_staging_file real_replace = os.replace staging_paths = [] def checked_temp(*args, **kwargs): assert Path(kwargs["directory"]) == destination.parent result = real_temp(*args, **kwargs) staging_paths.append(result.name) return result def checked_replace(src, dst): # Model EXDEV deterministically without requiring two mounted devices. if Path(src).parent != Path(dst).parent: raise OSError(errno.EXDEV, "cross-device rename") assert Path(src).read_bytes() == PAYLOAD real_replace(src, dst) monkeypatch.setattr(download, "_open_staging_file", checked_temp) monkeypatch.setattr(download.os, "replace", checked_replace) fetch_file(source.as_uri(), destination=destination, expected_sha256=sha256(PAYLOAD)) # Download, optional decompression, and an empty creation-mode probe. assert len(staging_paths) == (3 if compressed else 2) assert destination.read_bytes() == PAYLOAD assert list(system_temp.iterdir()) == [] assert list(destination.parent.iterdir()) == [destination] @pytest.mark.parametrize("existing", [False, True]) @pytest.mark.parametrize("compressed", [False, True]) def test_simultaneous_fetches_publish_only_complete_files( source, tmp_path, monkeypatch, existing, compressed): if compressed: source = tmp_path / "source.gz" source.write_bytes(gzip.compress(PAYLOAD)) destination = tmp_path / "cache" / "file" if existing: destination.parent.mkdir() destination.write_bytes(b"old complete file") started = threading.Barrier(3, timeout=10) release = threading.Event() ready_to_publish = threading.Barrier(2, timeout=10) real_stream = download._stream_to_file real_replace = os.replace staging_paths = [] def paused_stream(url, stream, **kwargs): staging_paths.append(stream.name) stream.write(b"partial") stream.flush() started.wait() assert release.wait(timeout=10) stream.seek(0) stream.truncate() return real_stream(url, stream, **kwargs) def checked_replace(src, dst): assert Path(src).read_bytes() == PAYLOAD ready_to_publish.wait() real_replace(src, dst) assert Path(dst).read_bytes() == PAYLOAD monkeypatch.setattr(download, "_stream_to_file", paused_stream) monkeypatch.setattr(download.os, "replace", checked_replace) with ThreadPoolExecutor(max_workers=2) as executor: futures = [executor.submit( fetch_file, source.as_uri(), destination=destination, force=existing, expected_sha256=sha256(PAYLOAD), expected_size=len(PAYLOAD), ) for _ in range(2)] try: started.wait() assert len(set(staging_paths)) == 2 if existing: assert destination.read_bytes() == b"old complete file" else: assert not destination.exists() finally: release.set() assert [future.result(timeout=10) for future in futures] == [str(destination)] * 2 assert destination.read_bytes() == PAYLOAD assert list(destination.parent.iterdir()) == [destination] def test_permission_errors_are_not_cache_misses(source, tmp_path, monkeypatch): destination = tmp_path / "unreadable" destination.write_bytes(PAYLOAD) original_open = builtins.open def denied_open(file, *args, **kwargs): if os.fspath(file) == str(destination): raise PermissionError("unreadable installation") return original_open(file, *args, **kwargs) monkeypatch.setattr(builtins, "open", denied_open) monkeypatch.setattr(download.os, "makedirs", reject_mutation) with pytest.raises(PermissionError, match="unreadable installation"): validate_file(destination) with pytest.raises(PermissionError, match="unreadable installation"): fetch_file(source.as_uri(), destination=destination) def test_directory_is_not_a_cached_file(tmp_path): with pytest.raises(FileValidationError, match="regular file"): fetch_file("https://unavailable.invalid/file", destination=tmp_path) @pytest.mark.parametrize("failed", [False, True]) def test_http_timeout_progress_and_response_cleanup(tmp_path, monkeypatch, failed): destination = tmp_path / "cache" / "file" class Response: headers = {"Content-Length": str(len(PAYLOAD))} closed = False def close(self): self.closed = True def raise_for_status(self): pass def iter_content(self, chunk_size): assert chunk_size == 8 yield b"" yield PAYLOAD[:8] if failed: raise requests.ConnectionError("interrupted HTTP") yield PAYLOAD[8:] response = Response() def get(url, headers, timeout, stream): assert timeout == 3 assert stream is True return response monkeypatch.setattr(download.requests, "get", get) progress = [] kwargs = dict(destination=destination, timeout=3, chunk_size=8, max_retries=0, progress_callback=lambda done, total: progress.append((done, total))) if failed: with pytest.raises(requests.ConnectionError): fetch_file("https://example.invalid/file", **kwargs) assert list(destination.parent.iterdir()) == [] else: fetch_file("https://example.invalid/file", **kwargs) assert destination.read_bytes() == PAYLOAD assert progress == [(8, len(PAYLOAD)), (len(PAYLOAD), len(PAYLOAD))] assert response.closed def test_force_is_only_suggested_when_it_can_replace_the_path(tmp_path): source = tmp_path / "source.txt" source.write_bytes(b"data") cache_root = tmp_path / "cache" cache_root.mkdir() (cache_root / "mismatched.txt").write_bytes(b"old") with pytest.raises(FileValidationError, match="use force=True"): fetch_file(source.as_uri(), filename="mismatched.txt", cache_root=cache_root, expected_size=4) (cache_root / "directory.txt").mkdir() with pytest.raises(FileValidationError) as error: fetch_file(source.as_uri(), filename="directory.txt", cache_root=cache_root) assert "force=True" not in str(error.value) def test_empty_local_source_is_rejected_without_publishing(tmp_path): source = tmp_path / "source" source.touch() destination = tmp_path / "output" with pytest.raises(FileValidationError, match="downloaded file is empty"): fetch_file(source.as_uri(), destination=destination) assert not destination.exists() @pytest.mark.parametrize("archive", ["gz", "zip"]) def test_an_empty_archive_member_is_rejected(tmp_path, archive): # The check applies to installed bytes: the archive itself is not empty. source = tmp_path / ("data.csv." + archive) if archive == "gz": source.write_bytes(gzip.compress(b"")) else: with zipfile.ZipFile(source, "w") as z: z.writestr("data.csv", b"") destination = tmp_path / "data.csv" with pytest.raises(FileValidationError, match="installed file is empty"): fetch_file(source.as_uri(), destination=destination, decompress=True) assert not destination.exists() fetch_file(source.as_uri(), destination=destination, decompress=True, allow_empty=True) assert destination.read_bytes() == b"" def test_an_empty_cached_file_is_an_invalid_hit(tmp_path): # Older releases could cache an empty response; it must stop being reused. source = tmp_path / "source" source.write_bytes(b"real data") cached = tmp_path / "cached" cached.touch() with pytest.raises(FileValidationError, match="cached file is empty.*force=True"): fetch_file(source.as_uri(), destination=cached) assert fetch_file(source.as_uri(), destination=cached, allow_empty=True) == str(cached) assert cached.read_bytes() == b"" fetch_file(source.as_uri(), destination=cached, force=True) assert cached.read_bytes() == b"real data" def test_allow_empty_must_be_a_boolean(tmp_path): with pytest.raises(ValueError, match="allow_empty"): fetch_file((tmp_path / "x").as_uri(), destination=tmp_path / "y", allow_empty="yes") ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792092720.7653558 datacache-1.27.1/tests/test_zip_members.py0000644000076500000000000003361215264225061020314 0ustar00iskanderwheelfrom concurrent.futures import ThreadPoolExecutor import hashlib from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import io import json from pathlib import Path import re import struct import threading import warnings import zipfile import zlib import pytest from urllib3.exceptions import ProtocolError from datacache import FileValidationError, install_zip_member, list_bundles PAYLOAD = b"peptide\tintensity\nEXAMPLE\t25\n" * 10000 def archive_bytes(method=zipfile.ZIP_DEFLATED, *, duplicate=False, zip64=False): handle = io.BytesIO() with zipfile.ZipFile(handle, "w", compression=method) as archive: archive.writestr("other", b"x" * 2**20) if zip64: with archive.open("txt/peptides.txt", "w", force_zip64=True) as member: member.write(PAYLOAD) else: archive.writestr("txt/peptides.txt", PAYLOAD) if duplicate: with warnings.catch_warnings(): warnings.simplefilter("ignore", UserWarning) archive.writestr("txt/peptides.txt", b"ambiguous") return handle.getvalue() @pytest.fixture def server(): running = [] def start(payload=None, **options): state = dict( payload=archive_bytes() if payload is None else payload, requests=[], **options, ) class Handler(BaseHTTPRequestHandler): def log_message(self, *_): pass def do_GET(self): state["requests"].append(dict(self.headers)) match = re.fullmatch(r"bytes=(\d+)-(\d+)", self.headers["Range"]) first, last = map(int, match.groups()) data = state["payload"][first : last + 1] bad = ( state.get("bad") if len(state["requests"]) > state.get("after", 0) else None ) self.send_response(200 if bad == "status" else 206) if bad != "validator": if state.get("last_modified"): self.send_header( "Last-Modified", "Thu, 18 Nov 2021 01:40:15 GMT" ) else: self.send_header("ETag", '"v2"' if bad == "changed" else '"v1"') total = len(state["payload"]) + (1 if bad == "size" else 0) self.send_header( "Content-Range", "bytes %d-%d/%d" % (first + (1 if bad == "range" else 0), last, total), ) self.send_header( "Content-Length", str(len(data) + (1 if bad == "length" else 0)) ) if bad == "encoding": self.send_header("Content-Encoding", "gzip") self.end_headers() try: self.wfile.write(data[:-1] if bad == "truncated" else data) except (BrokenPipeError, ConnectionResetError): pass httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler) thread = threading.Thread(target=httpd.serve_forever, daemon=True) thread.start() running.append((httpd, thread)) return "http://127.0.0.1:%d/archive.zip" % httpd.server_port, state yield start for httpd, thread in running: httpd.shutdown() httpd.server_close() thread.join() def install(tmp_path, url, **kwargs): return install_zip_member(tmp_path / "store", url, "txt/peptides.txt", **kwargs) @pytest.mark.parametrize("extra", [-1, 0, 1, 2]) def test_drain_buffered_output_at_expansion_boundary(tmp_path, server, extra): from datacache.zip_members import CHUNK_SIZE payload = b"a" * (CHUNK_SIZE + extra) + b"\n" stream = io.BytesIO() with zipfile.ZipFile(stream, "w", compression=zipfile.ZIP_DEFLATED) as archive: archive.writestr("txt/peptides.txt", payload) url, _ = server(stream.getvalue()) paths = install( tmp_path, url, expected_size=len(payload), expected_sha256=hashlib.sha256(payload).hexdigest(), ) assert Path(paths["payload"]).read_bytes() == payload @pytest.mark.parametrize("method", [zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED]) @pytest.mark.parametrize("zip64", [False, True]) def test_member_integrity_and_offline_reuse( tmp_path, server, method, zip64, monkeypatch ): url, state = server(archive_bytes(method, zip64=zip64)) digest = hashlib.sha256(PAYLOAD).hexdigest() paths = install(tmp_path, url, expected_sha256=digest, expected_size=len(PAYLOAD)) assert Path(paths["payload"]).read_bytes() == PAYLOAD receipt = json.loads(Path(paths["receipt.json"]).read_text()) assert receipt["payload_verified"] and not receipt["compressed_verified"] assert receipt["payload"] == dict(sha256=digest, size=len(PAYLOAD)) assert ( receipt["compressed"]["sha256"] == hashlib.sha256(Path(paths["compressed"]).read_bytes()).hexdigest() ) assert receipt["archive_size_bytes"] == len(state["payload"]) assert all(item["Accept-Encoding"] == "identity" for item in state["requests"]) assert all(item["If-Range"] == '"v1"' for item in state["requests"][1:]) assert len(list_bundles(tmp_path / "store")) == 1 monkeypatch.setattr( "requests.Session.get", lambda *args, **kwargs: pytest.fail("cache hit used network"), ) assert ( install(tmp_path, url, expected_sha256=digest, expected_size=len(PAYLOAD)) == paths ) def test_last_modified_and_metadata_budget(tmp_path, server): url, state = server(last_modified=True) paths = install(tmp_path, url) receipt = json.loads(Path(paths["receipt.json"]).read_text()) assert receipt["validator"]["header"] == "Last-Modified" assert not receipt["payload_verified"] assert receipt["metadata_size_bytes"] < 100000 @pytest.mark.parametrize( "bad,after", [ ("status", 0), ("validator", 0), ("changed", 1), ("size", 1), ("range", 0), ("length", 0), ("encoding", 0), ("truncated", 0), ], ) def test_bad_http_never_publishes(tmp_path, server, bad, after): url, state = server(bad=bad, after=after) with pytest.raises((ValueError, zipfile.BadZipFile, ProtocolError)): install(tmp_path, url) assert list_bundles(tmp_path / "store") == [] assert not list((tmp_path / "store").glob(".staging-*")) @pytest.mark.parametrize( "options", [ dict(max_bytes=5), dict(max_compressed_bytes=5), dict(max_metadata_bytes=5), dict(expected_size=1), dict(expected_sha256="0" * 64), dict(compressed_sha256="0" * 64), ], ) def test_integrity_and_resource_limits(tmp_path, server, options): url, state = server() with pytest.raises((ValueError, FileValidationError, zipfile.BadZipFile)): install(tmp_path, url, **options) assert not list_bundles(tmp_path / "store") @pytest.mark.parametrize( "kind", ["duplicate", "missing", "unsupported", "crc", "encrypted", "local_name"] ) def test_invalid_members(tmp_path, server, kind): data = archive_bytes( duplicate=kind == "duplicate", method=zipfile.ZIP_BZIP2 if kind == "unsupported" else zipfile.ZIP_STORED, ) if kind in ("crc", "encrypted", "local_name"): with zipfile.ZipFile(io.BytesIO(data)) as archive: info = archive.getinfo("txt/peptides.txt") data = bytearray(data) if kind == "crc": data[info.header_offset + 30 + len(info.filename)] ^= 1 elif kind == "local_name": data[info.header_offset + 30] ^= 1 else: offset = data.index( b"PK\x01\x02", data.index(b"PK\x01\x02", info.header_offset) + 4 ) struct.pack_into("