pax_global_header 0000666 0000000 0000000 00000000064 15244036076 0014521 g ustar 00root root 0000000 0000000 52 comment=f928c11f3950c126217a2089616e265849ffba3c
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/ 0000775 0000000 0000000 00000000000 15244036076 0022047 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/.gitignore 0000664 0000000 0000000 00000000411 15244036076 0024033 0 ustar 00root root 0000000 0000000 # Hidden files
.*
!.gitignore
!.gitlab-ci.yml
!.readthedocs.yaml
!.flake8
# Byte / compiled / optimized
*.py[cod]
__pycache__/
*.so
# Distribution / packaging
/build/
/dist/
*.egg-info/
.eggs/
.venv/
/doc/reference/_public_api
/doc/reference/_private_api
uv.lock
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/.gitlab-ci.yml 0000664 0000000 0000000 00000004106 15244036076 0024504 0 ustar 00root root 0000000 0000000 include:
- project: dau/ci/pyci
file: .gitlab-ci-template.yml
variables:
WARNING1: "ignore::DeprecationWarning:ansiwrap.core"
WARNING2: "ignore::DeprecationWarning:jupyter_client.session"
WARNING3: "ignore::DeprecationWarning:traitlets.traitlets"
WARNING4: "ignore::DeprecationWarning:papermill.parameterize"
WARNING5: "ignore::DeprecationWarning:papermill.engines"
WARNINGS: "-W ${WARNING1} -W ${WARNING2} -W ${WARNING3} -W ${WARNING4} -W ${WARNING5}"
DISABLE_BLACK_FLAKE8: "yes"
ruff:
extends: .ruff
test-3.8:
extends: .test-3.8
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test-3.9:
extends: .test-3.9
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test-3.9-win:
extends: .test-3.9-win
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test-3.10:
extends: .test-3.10
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test-3.11:
extends: .test-3.11
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test-3.12:
extends: .test-3.12
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
PYTEST_COV: "true"
test-3.13:
extends: .test-3.13
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
build_sdist:
extends: .build_sdist
test_sdist-3.8:
extends: .test_sdist-3.8
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test_sdist-3.9:
extends: .test_sdist-3.9
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test_sdist-3.10:
extends: .test_sdist-3.10
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test_sdist-3.10-win:
extends: .test_sdist-3.10-win
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test_sdist-3.11:
extends: .test_sdist-3.11
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
test_sdist-3.12:
extends: .test_sdist-3.12
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
PYTEST_COV: "true"
test_sdist-3.13:
extends: .test_sdist-3.13
variables:
PYTEST_WARNINGS: "-W error ${WARNINGS}"
build_doc:
extends: .build_doc
pypi:
extends: .pypi
testpypi:
extends: .testpypi
pages:
extends: .pages
assets:
extends: .assets
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/.readthedocs.yaml 0000664 0000000 0000000 00000000347 15244036076 0025302 0 ustar 00root root 0000000 0000000 # Read the Docs configuration file
version: 2
build:
os: ubuntu-24.04
tools:
python: "3.12"
sphinx:
configuration: doc/conf.py
python:
install:
- method: pip
path: .
extra_requirements:
- doc esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/CHANGELOG.md 0000664 0000000 0000000 00000020017 15244036076 0023660 0 ustar 00root root 0000000 0000000 # Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [1.2.0rc6] - 2026-08-27
### Deprecated
- Tomo template names (the old names still work but are deprecated):
- `volumes_file` -> `volumes_technique_path`
- `volumes_custom_type_file` -> `volumes_custom_type_technique_path`
- `volumes_link_collection_file` -> `volumes_link_collection_technique_path`
- `volumes_link_root_file` -> `volumes_link_root_technique_path`
### Added
- `esrf_v4` schema: adds the `AUXILIARY_DATA` data type directory, in analogy to
`SCRIPTS`, `GALLERY` and `NOTES`.
- `tomo_v2` schema: same as `tomo_v1` but derived from `esrf_v4` instead of `esrf_v3`.
- `tomoproc_v1` schema: tomography processed data of a single dataset rooted at an
arbitrary directory (`dataset_path` template) instead of an ESRF session directory
(`tomo_v1` schema).
- `tomo_dataset` field: the dataset directory name which is also the prefix of the
tomography result file names. A path concept for `tomoproc_v1` (the
collection/dataset split of a local directory name is unknown) and a derived
concept `{collection}_{dataset}` for `tomo_v1`.
- `ESRFPath.schema_description`: short description of the matched schema (e.g. when it
was introduced), taken from its module docstring. Also shown by the `esrf-pathlib`
CLI.
- Schemas with same-named templates and concepts can be used together in a single
registry: template names resolve within the schema that matched the path,
same-named concepts must have the same definition (their default value may differ:
the first schema added to the registry wins) and a concept of one schema can be a
derived concept of another schema. Paths of `tomoproc_v1` can therefore be rendered
from `collection` and `dataset` when combined with `tomo_v1`.
- Fields (concepts, derived concepts and templates) can be deprecated.
### Changed
- Path matching prefers a template that identifies the path itself over a template
that only identifies its parent directories. A path of one schema which is not
identified by the schema it is rooted in (for example a tomography result inside an
ESRF session directory with a name that is not `{collection}_{dataset}_...`) is
therefore matched by the schema that does identify it.
- Match score ties are broken deterministically:
- path matching prefers literal segments over catch-all segments
- path rendering prefers the template fully described by the provided concepts.
- `ESRFPath.replace_fields(template_name=...)` renders the template of the schema that
matched the path.
- `tomo_angle` field: rendered with two decimals (`90.00deg` instead of `90deg`) and
negative angles are accepted. Existing names without decimals are still parsed.
- The `data_root` and `unknown_part` fields are `pathlib.Path` objects instead of
strings. Field values provided to `ESRFPath.from_fields` and
`ESRFPath.replace_fields` can be strings or `pathlib.Path` objects.
### Fixed
- Path fields of an `ESRFPath` subclass are instances of that subclass instead of
`ESRFPath` itself.
- A path obtained from a template attribute (e.g. `path.slices_link_root_path`) is
matched by that template instead of by the template that best matches the rendered
path.
- Instantiating from an existing instance (e.g. `TomoPath(path)`) reuses the path
template that was matched before instead of matching the path again.
- A path field spanning path separators (e.g. `data_root`) rendered as the filesystem
root no longer parses back as an empty value.
## [1.1.0] - 2026-08-03
### Added
- `ESRFPath.glob`: without a `pattern` argument, glob the path's own string representation,
supporting field wildcards from `replace_fields(..., field="*")`.
## [1.0.0] - 2026-03-30
### Added
- `ESRFPath.strip_mount_point`: remove leading segments typically introduced by `ESRFPath.resolve`.
## [0.7.2] - 2026-03-12
### Fixed
- Improve regex to match session date 'YYYYMMDD'.
## [0.7.1] - 2026-03-06
### Fixed
- Fixed bad folder level for volume gallery in `tomo_v1`.
## [0.7.0] - 2026-03-03
### Added
- `esrf-pathlib` CLI to print information to path schemas.
- `tomo_v1` schema.
- `ESRFPath` arguments
- `ESRFPath` class arguments `fallback_depth` and names arguments such as `esrf=3` and `tomo=1`.
- `ESRFPath.replace_fields` argument `template_name`.
- `ESRFPath.from_fields` arguments `schema_version` and `template_name`.
- `ESRFPath` attributes
- Function `ESRFPath.from_path`: instantiate `ESRFPath` with a specific schema, schema version and template.
- Property `ESRFPath.template_name`: matching path template name of the current instance.
- `esrf_pathlib.errors.EsrfPathlibError` exception type.
### Removed
- `esrf_pathlib.schemas.detect_schema` is removed and has no replacement.
## [0.6.0] - 2025-09-16
### Changed
- Show path and schema in attribute errors.
## [0.5.0] - 2025-09-05
### Added
- `ESRFPath.raw_metadata_path` for schema esrf_v3, esrf_v2 and esrf_v1.
- `ESRFPath.raw_metadata_file` for schema esrf_v3, esrf_v2 and esrf_v1.
## [0.4.1] - 2025-09-03
### Changed
- Default value for `data_root` in `ESRFPath` is set to `/data/visitor` on non-Windows platforms.
## [0.4.0] - 2025-08-25
### Added
- `ESRFPath.beamline_normalized` for schema esrf_v3, esrf_v2 and esrf_v1.
## [0.3.0] - 2025-08-13
### Added
- `ESRFPath.processed_dataset_path` for schema esrf_v3, esrf_v2.
- `ESRFPath.processed_dataset_file` for schema esrf_v3, esrf_v2.
- `ESRFPath.processed_collection_file` for schema esrf_v3, esrf_v2.
- `ESRFPath.processed_proposal_file` for schema esrf_v3, esrf_v2.
- `ESRFPath.nobackup_dataset_path` for schema esrf_v3, esrf_v2.
- `ESRFPath.nobackup_dataset_file` for schema esrf_v3, esrf_v2.
- `ESRFPath.nobackup_collection_file` for schema esrf_v3, esrf_v2.
- `ESRFPath.nobackup_proposal_file` for schema esrf_v3, esrf_v2.
## [0.2.0] - 2025-08-11
### Removed
- `esrf_pathlib.Path` is replaced by `esrf_pathlib.ESRFPath`.
- `ESRFPath.esrf_info` is replaced by `ESRFPath.schema_name`.
### Added
- `ESRFPath.replace_fields` provides a copy with the option to modify fields.
- `ESRFPath.schema_name` path schema name - if is an known ESRF path - else None.
- `ESRFPath.from_fields` allows construction an ESRF map from the `dict` of fields.
- `ESRFPath.raw_data_path` for schema esrf_v3, esrf_v2, esrf_v1.
- `ESRFPath.processed_data_path` for schema esrf_v3, esrf_v2.
- `ESRFPath.scripts_path` for schema esrf_v3, esrf_v2.
- `ESRFPath.nobackup_path` for schema esrf_v3, esrf_v2.
- `ESRFPath.raw_dataset_path` for schema esrf_v3, esrf_v2, esrf_v1.
- `ESRFPath.raw_dataset_file` for schema esrf_v3, esrf_v2, esrf_v1.
- `ESRFPath.raw_collection_file` for schema esrf_v3, esrf_v2.
- `ESRFPath.raw_proposal_file` for schema esrf_v3, esrf_v2.
## [0.1.0] - 2025-06-02
### Added
- Initial project scaffold for esrf-pathlib
- Basic `Path` subclass and `proposal` method
[Unreleased]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v1.2.0rc6...HEAD
[1.2.0rc6]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v1.1.0...v1.2.0rc6
[1.1.0]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v1.0.0...v1.1.0
[1.0.0]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.7.2...v1.0.0
[0.7.2]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.7.1...v0.7.2
[0.7.1]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.7.0...v0.7.1
[0.7.0]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.6.0...v0.7.0
[0.6.0]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.5.0...v0.6.0
[0.5.0]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.4.1...v0.5.0
[0.4.1]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.4.0...v0.4.1
[0.4.0]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.3.0...v0.4.0
[0.3.0]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.2.0...v0.3.0
[0.2.0]: https://gitlab.esrf.fr/workflow/esrf-pathlib/compare/v0.1.0...v0.2.0
[0.1.0]: https://gitlab.esrf.fr/workflow/esrf-pathlib/-/tags/v0.1.0
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/CONTRIBUTING.md 0000664 0000000 0000000 00000000154 15244036076 0024300 0 ustar 00root root 0000000 0000000 CONTRIBUTING.md esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/LICENSE.md 0000664 0000000 0000000 00000002116 15244036076 0023453 0 ustar 00root root 0000000 0000000 # MIT License
**Copyright (c) 2025 European Synchrotron Radiation Facility**
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/README.md 0000664 0000000 0000000 00000002043 15244036076 0023325 0 ustar 00root root 0000000 0000000 # esrf-pathlib
`esrf-pathlib` is a drop-in replacement for Python’s built-in [`pathlib.Path`](https://docs.python.org/3/library/pathlib.html),
extended with ESRF-specific attributes.
It detects and interprets known ESRF data directory structures and provides high-level access to proposal names, processed data locations, and more.
## Installation
```bash
pip install esrf-pathlib
```
## Getting Started
```python
from esrf_pathlib import ESRFPath
path = ESRFPath("/data/visitor/ma1234/id00/20250509/RAW_DATA/sample/sample_dataset")
print(path)
print("Proposal:", path.proposal)
print("Processed results:", path.processed_dataset_file)
```
**Output:**
```bash
Proposal: ma1234
Processed results: /data/visitor/ma1234/id00/20250509/PROCESSED_DATA/sample/sample_dataset/sample_dataset.h5
```
The available attributes depend on the ESRF path template matching the path.
## Documentation
https://esrf-pathlib.readthedocs.io/
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/ 0000775 0000000 0000000 00000000000 15244036076 0022614 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/_ext/ 0000775 0000000 0000000 00000000000 15244036076 0023553 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/_ext/esrfpath_autodoc.py 0000664 0000000 0000000 00000004230 15244036076 0027456 0 ustar 00root root 0000000 0000000 import pathlib
from typing import Any
from sphinx.util import logging
import esrf_pathlib
from esrf_pathlib._schemas.definitions import load
logger = logging.getLogger(__name__)
def make_schema_stub_class(schema) -> type:
"""
Dynamically create a stub class for a given ESRFPath schema.
Attributes correspond to concepts, derived concepts, and templates.
"""
stub_attrs = {}
for cname, concept in schema.concepts.items():
stub_attrs[cname] = make_property(concept.annotation(), concept.docstring)
for dname, derived in schema.derived_concepts.items():
stub_attrs[dname] = make_property(derived.annotation(), derived.docstring)
for tname, template in schema.templates.items():
stub_attrs[tname] = make_property(template.annotation(), template.docstring)
docstring = f"""{schema.docstring}
Derive a class from :class:`ESRFPath` to restrict path matching, parsing and rendering to this schema:
.. code-block:: python
from esrf_pathlib import ESRFPath
class {schema.identifier.name.capitalize()}Path(ESRFPath, {schema.identifier.name}={schema.identifier.version}, fallback_depth=0):
pass
"""
stub_attrs["__doc__"] = docstring
class_name = f"ESRFPath_{schema.identifier}"
return type(class_name, (pathlib.Path,), stub_attrs)
def make_property(annotation: Any, doc: str):
def prop(self):
pass
prop.__annotations__ = {"return": annotation}
prop.__doc__ = doc
return property(prop)
def setup(app):
for sname, sversions in load._ALL_VERSIONS.items():
for sversion in sversions:
schema = load.get_schema(sname, sversion)
stub_cls = make_schema_stub_class(schema)
setattr(esrf_pathlib, stub_cls.__name__, stub_cls)
logger.info(
f"Created stub class esrf_pathlib.{stub_cls.__name__} for schema '{schema.identifier}'"
)
# Add config for the new module path if needed
app.add_css_file("custom.css")
# No directives or roles to add here, just generating classes
return {
"version": "0.1",
"parallel_read_safe": True,
"parallel_write_safe": True,
}
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/_static/ 0000775 0000000 0000000 00000000000 15244036076 0024242 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/_static/custom.css 0000664 0000000 0000000 00000000117 15244036076 0026265 0 ustar 00root root 0000000 0000000 button.copybtn {
opacity: 0.3;
}
button.copybtn:hover {
opacity: 1;
}
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/_templates/ 0000775 0000000 0000000 00000000000 15244036076 0024751 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/_templates/footer_end.html 0000664 0000000 0000000 00000000033 15244036076 0027757 0 ustar 00root root 0000000 0000000 {{ project }} {{ release }} esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/conf.py 0000664 0000000 0000000 00000003741 15244036076 0024120 0 ustar 00root root 0000000 0000000 # -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
import importlib.metadata
import os
import sys
sys.path.append(os.path.abspath("_ext"))
release = importlib.metadata.version("esrf-pathlib")
project = "esrf-pathlib"
version = ".".join(release.split(".")[:2])
copyright = "2025-2026, ESRF"
author = "ESRF"
docstitle = f"{project} {version}"
# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
extensions = [
"sphinxarg.ext",
"sphinx.ext.autodoc",
"sphinx.ext.autosummary",
"sphinx.ext.viewcode",
"sphinx_autodoc_typehints",
"sphinx_copybutton",
"esrfpath_autodoc",
]
templates_path = ["_templates"]
exclude_patterns = []
always_document_param_types = True
autodoc_default_options = {
"members": True,
"member-order": "groupwise",
"undoc-members": True,
"show-inheritance": True,
}
copybutton_prompt_text = r">>> |\.\.\. |\$ |In \[\d*\]: | {2,5}\.\.\.: | {5,8}: "
copybutton_prompt_is_regexp = True
# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
html_theme = "pydata_sphinx_theme"
html_title = docstitle
# html_logo = "_static/logo.png"
html_static_path = ["_static"]
html_template_path = ["_templates"]
html_css_files = ["custom.css"]
html_theme_options = {
"icon_links": [
{
"name": "gitlab",
"url": "https://gitlab.esrf.fr/workflow/esrf-pathlib",
"icon": "fa-brands fa-gitlab",
},
{
"name": "pypi",
"url": "https://pypi.org/project/esrf-pathlib",
"icon": "fa-brands fa-python",
},
],
"logo": {
"text": docstitle,
},
"footer_start": ["copyright"],
"footer_end": ["footer_end"],
}
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/explanations.rst 0000664 0000000 0000000 00000000141 15244036076 0026047 0 ustar 00root root 0000000 0000000 Explanations
============
.. toctree::
:maxdepth: 1
explanations/schemas_and_templates
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/explanations/ 0000775 0000000 0000000 00000000000 15244036076 0025321 5 ustar 00root root 0000000 0000000 schemas_and_templates.rst 0000664 0000000 0000000 00000010434 15244036076 0032321 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/explanations Schemas and templates
=======================
A *schema* (for example ``esrf`` version 3) defines *concepts* (`proposal`,
`beamline`, `collection`, `dataset`, ...) and *templates*: named directory or
file patterns built from those concepts, such as ``raw_collection_path``
(``{data_root}/{proposal}/{beamline}/{session_date}/RAW_DATA/{collection}``).
Constructing an :class:`~esrf_pathlib.ESRFPath` matches the string against
every template of every registered schema. The first match determines
`schema_name` and `template_name`, and which concept attributes become
available
.. code-block:: python-console
>>> path = ESRFPath("/data/visitor/ma1234/id00/20250509/RAW_DATA/sample/sample_dataset")
>>> path.schema_name, path.template_name
('esrf_v3', 'raw_dataset_path')
Schema versions and fallback
------------------------------
Path layouts change between ESRF data policy versions. A class only
recognizes the schema versions it was declared with, plus a configurable
number of older versions (`fallback_depth`) - see
:doc:`../howtoguides/select_schema_version`.
Picking a template without ``template_name``
----------------------------------------------
Without ``template_name``, :meth:`~esrf_pathlib.ESRFPath.replace_fields` and
:meth:`~esrf_pathlib.ESRFPath.from_fields` render the *most specific* template
fully described by the known concepts - a file is always at least as
specific as its parent directory, so it wins whenever both are fully known
.. code-block:: python-console
>>> path.replace_fields(data_type="PROCESSED_DATA")
This is why searching for collections or datasets, see
:doc:`../howtoguides/search_with_glob`, resolves to a *file* template by
default: there is no set of concept values for which a directory template is
ever the most specific match.
The directory can be retrieved in several ways
.. code-block:: python-console
>>> path.replace_fields(data_type="PROCESSED_DATA", template_name="processed_dataset_path")
>>> path.replace_fields(data_type="PROCESSED_DATA").processed_dataset_path
Note that since `path` already contains all concepts needed for the dataset path,
`replace_fields` can be omitted entirely
.. code-block:: python-console
>>> path.processed_dataset_path
Wildcards do not survive re-parsing
--------------------------------------
A field value like ``"*"`` is written into the rendered path as-is, but
parsing a path *back* into concept values validates it against the concept's
expected format - so a wildcard generally fails to parse back
.. code-block:: python-console
>>> wildcarded = path.replace_fields(collection="*")
>>> list(wildcarded.glob()) # OK: only needs the literal string
[...]
>>> wildcarded.collection # fails: re-derives concept values first
Traceback (most recent call last):
...
esrf_pathlib._schemas.errors.SchemaAttributeError: has no attribute 'collection'
Treat a wildcarded ``ESRFPath`` as write-only: build it, glob it, then work
with the results, which are real, fully parsed paths.
Deprecated names
------------------
Concepts, derived concepts and templates are sometimes renamed between schema
revisions. The former name keeps working - as an ``ESRFPath`` attribute and
as a ``template_name``/field argument of :meth:`~esrf_pathlib.ESRFPath.replace_fields`,
:meth:`~esrf_pathlib.ESRFPath.from_fields` and :meth:`~esrf_pathlib.ESRFPath.from_path`
- but emits a ``DeprecationWarning`` pointing to the current name
.. code-block:: python-console
>>> path.volumes_file
:1: DeprecationWarning: Template 'volumes_file' is deprecated, use 'volumes_technique_path' instead.
A field can accumulate several deprecated names over time; each one resolves
to the current canonical name.
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/howtoguides.rst 0000664 0000000 0000000 00000000304 15244036076 0025704 0 ustar 00root root 0000000 0000000 How-to Guides
=============
.. toctree::
:maxdepth: 1
howtoguides/select_schema_version
howtoguides/replace_fields
howtoguides/search_with_glob
howtoguides/strip_mount_point
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/howtoguides/ 0000775 0000000 0000000 00000000000 15244036076 0025155 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/howtoguides/replace_fields.rst 0000664 0000000 0000000 00000003354 15244036076 0030655 0 ustar 00root root 0000000 0000000 Replace fields in a path
=========================
ESRF path fields (`proposal`, `collection`, `dataset`, ...) are immutable
.. code-block:: python-console
>>> from esrf_pathlib import ESRFPath
>>> path = ESRFPath("/data/visitor/ma1234/id00/20250509/RAW_DATA/sample/sample_dataset")
>>> path.dataset = "other_dataset"
Traceback (most recent call last):
...
esrf_pathlib._schemas.errors.SchemaAttributeError: Attribute 'dataset' is immutable. Create a new path instance with `ESRFPath.replace_fields(dataset='other_dataset')`.
Use :meth:`~esrf_pathlib.ESRFPath.replace_fields` to derive a new path with one
or more fields changed
.. code-block:: python-console
>>> path.replace_fields(dataset="other_dataset")
When the new set of fields no longer matches the current template,
``replace_fields`` picks the *best matching* template automatically: the
deepest template fully described by the known fields. Pass ``template_name``
to pick a specific one instead
.. code-block:: python-console
>>> path.replace_fields(data_type="PROCESSED_DATA")
>>> path.replace_fields(data_type="PROCESSED_DATA", template_name="processed_dataset_path")
See :doc:`../explanations/schemas_and_templates` for how templates are scored,
and :doc:`search_with_glob` for using ``replace_fields`` with wildcard values.
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/howtoguides/search_with_glob.rst0000664 0000000 0000000 00000005072 15244036076 0031216 0 ustar 00root root 0000000 0000000 Search collections and datasets
=================================
Collections and datasets are not enumerated by the schema - they are
discovered on the filesystem by rendering a field as a wildcard with
:meth:`~esrf_pathlib.ESRFPath.replace_fields` and searching for matches with
:meth:`~esrf_pathlib.ESRFPath.glob`.
List all collections in a session
----------------------------------
.. code-block:: python-console
>>> from esrf_pathlib import ESRFPath
>>> session_path = ESRFPath("/data/visitor/blc17255/id13/20260715")
>>> collections = list(session_path.replace_fields(collection="*").glob())
>>> sorted(c.collection for c in collections)
['sample1', 'sample2']
Each result is a fully parsed ``ESRFPath``, so `collection` and every other
schema attribute are available directly on it - no need for
``template_name`` just to read a field back. See
:doc:`../explanations/schemas_and_templates` for which template
``replace_fields`` picks by default.
List all datasets in a collection
-----------------------------------
.. code-block:: python-console
>>> one_collection = collections[0]
>>> datasets = list(one_collection.replace_fields(dataset="*").glob())
>>> sorted(d.dataset for d in datasets)
['dataset0001', 'dataset0002']
Other glob patterns
--------------------
A field value is used verbatim in the rendered path, so any `fnmatch`_-style
pattern works, not just ``"*"``:
* ``"sample?"`` matches a single unknown character
* ``"sample[12]"`` or ``"sample[1-3]"`` matches one of a set of characters
* ``"sample[!12]"`` matches anything except a set of characters
.. code-block:: python-console
>>> session_path.replace_fields(collection="sample[12]").glob()
Wildcard more than one field to search across collections and datasets in a
single call
.. code-block:: python-console
>>> session_path.replace_fields(collection="*", dataset="*").glob()
Without a ``pattern`` argument, :meth:`~esrf_pathlib.ESRFPath.glob` globs the
path's own string representation against the filesystem - this is what makes
wildcarded fields searchable, since ``pathlib.Path.glob`` otherwise requires
a pattern relative to an existing directory. Calling ``glob(pattern)`` with
an explicit pattern still behaves like ``pathlib.Path.glob``.
Pick a specific template - such as a directory rather than a file inside it -
by passing ``template_name`` explicitly
.. code-block:: python-console
>>> session_path.replace_fields(
... collection="*", template_name="raw_collection_path"
... ).glob()
.. _fnmatch: https://docs.python.org/3/library/fnmatch.html
select_schema_version.rst 0000664 0000000 0000000 00000003607 15244036076 0032202 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/howtoguides Select a schema version
========================
By default, :class:`~esrf_pathlib.ESRFPath` recognizes path templates from the
latest ``esrf`` schema, falling back one version when a path does not match
(equivalent to ``esrf=3, fallback_depth=1``).
Derive a class to change the default schema and fallback depth
(``None`` denotes the latest version of a schema)
.. code-block:: python
from esrf_pathlib import ESRFPath
class TomoPath(ESRFPath, tomo=None, fallback_depth=0):
pass
Schemas can be mixed provided same-named concepts have the same definition and the
schemas do not extend different versions of the same base schema
.. code-block:: python
class MyPath(ESRFPath, schemaA=1, schemaB=2, fallback_depth=0):
pass
Mixed schemas may define path templates with the same name, for example the *tomo* and
*tomoproc* schemas which describe the same tomography results rooted at different
directories. Path matching considers the templates of all schemas while template names
resolve within the schema that matched the path. When no schema matched, template names
resolve to the first schema that defines them (the order of the schemas matters)
.. code-block:: python
class TomoPath(ESRFPath, tomo=None, tomoproc=None):
pass
To enforce a schema on a single path without deriving a class, use
:meth:`~esrf_pathlib.ESRFPath.from_path`
.. code-block:: python-console
>>> from esrf_pathlib import ESRFPath
>>> path = ESRFPath.from_path(
... "/data/visitor/ma1234/id00/20250509/raw/sample/sample_dataset",
... schema_name="esrf",
... schema_version=2,
... )
>>> path.processed_dataset_file
See :doc:`../reference/public_api` for the full list of supported schemas and
their versions.
strip_mount_point.rst 0000664 0000000 0000000 00000001753 15244036076 0031432 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/howtoguides Strip a resolved mount point
=============================
Calling ``Path.resolve()`` on an ESRF path typically replaces the symbolic
`/data` root with the real, mount-specific path used on that machine
.. code-block:: python-console
>>> from esrf_pathlib import ESRFPath
>>> path = ESRFPath("/mnt/multipath-shares/data/visitor/ma1234/id00/20250509/RAW_DATA/sample/sample_dataset")
>>> path
Use :meth:`~esrf_pathlib.ESRFPath.strip_mount_point` to get back the portable,
symbolic form
.. code-block:: python-console
>>> path.strip_mount_point()
This is useful when persisting a path (for example in a database or a
workflow definition) that should remain valid regardless of which mount
point served it.
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/index.rst 0000664 0000000 0000000 00000001032 15244036076 0024451 0 ustar 00root root 0000000 0000000 esrf-pathlib |version|
=======================
*esrf-pathlib* is a drop-in replacement for Python's built-in `pathlib.Path`_,
extended with ESRF-specific attributes.
It detects and interprets known ESRF data directory structures and provides
high-level access to proposals, collections, datasets, processed data
locations, and more through the :ref:`ESRFPath` class.
.. toctree::
:hidden:
tutorials
howtoguides
explanations
reference
.. _pathlib.Path: https://docs.python.org/3/library/pathlib.html
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference.rst 0000664 0000000 0000000 00000000462 15244036076 0025306 0 ustar 00root root 0000000 0000000 .. _esrfpath-api:
Reference
=========
Command-Line Interface (CLI), Python API and per-schema attribute reference.
See :doc:`howtoguides/select_schema_version` for how to select or mix schema versions.
.. toctree::
:maxdepth: 2
reference/cli
reference/public_api
reference/private_api
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/ 0000775 0000000 0000000 00000000000 15244036076 0024552 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/cli.rst 0000664 0000000 0000000 00000000732 15244036076 0026055 0 ustar 00root root 0000000 0000000 .. _cli:
CLI reference
=============
esrf-pathlib
------------
.. argparse::
:module: esrf_pathlib.__main__
:func: create_argument_parser
:prog: esrf-pathlib
Prints the concepts, templates and directory tree of the given schema
versions, with no arguments defaulting to :class:`~esrf_pathlib.ESRFPath`'s
own schemas
.. code-block:: bash
esrf-pathlib
esrf-pathlib esrf=3 tomo=1
esrf-pathlib esrf=3 --fallback-depth=1
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/esrf_v1.rst 0000664 0000000 0000000 00000000130 15244036076 0026643 0 ustar 00root root 0000000 0000000 ``ESRFPath_esrf_v1``
====================
.. autoclass:: esrf_pathlib.ESRFPath_esrf_v1
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/esrf_v2.rst 0000664 0000000 0000000 00000000130 15244036076 0026644 0 ustar 00root root 0000000 0000000 ``ESRFPath_esrf_v2``
====================
.. autoclass:: esrf_pathlib.ESRFPath_esrf_v2
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/esrf_v3.rst 0000664 0000000 0000000 00000000130 15244036076 0026645 0 ustar 00root root 0000000 0000000 ``ESRFPath_esrf_v3``
====================
.. autoclass:: esrf_pathlib.ESRFPath_esrf_v3
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/esrf_v4.rst 0000664 0000000 0000000 00000000130 15244036076 0026646 0 ustar 00root root 0000000 0000000 ``ESRFPath_esrf_v4``
====================
.. autoclass:: esrf_pathlib.ESRFPath_esrf_v4
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/private_api.rst 0000664 0000000 0000000 00000000160 15244036076 0027604 0 ustar 00root root 0000000 0000000 Private API
===========
.. autosummary::
:toctree: _private_api
:recursive:
esrf_pathlib._schemas
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/public_api.rst 0000664 0000000 0000000 00000000467 15244036076 0027422 0 ustar 00root root 0000000 0000000 Public API
==========
.. autosummary::
:toctree: _public_api
:recursive:
esrf_pathlib
Refer to the following pages for details on each supported schema version:
.. toctree::
:maxdepth: 1
esrf_v4
esrf_v3
esrf_v2
esrf_v1
tomo_v2
tomo_v1
tomoproc_v1
unknown_v1
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/tomo_v1.rst 0000664 0000000 0000000 00000000130 15244036076 0026662 0 ustar 00root root 0000000 0000000 ``ESRFPath_tomo_v1``
====================
.. autoclass:: esrf_pathlib.ESRFPath_tomo_v1
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/tomo_v2.rst 0000664 0000000 0000000 00000000130 15244036076 0026663 0 ustar 00root root 0000000 0000000 ``ESRFPath_tomo_v2``
====================
.. autoclass:: esrf_pathlib.ESRFPath_tomo_v2
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/tomoproc_v1.rst 0000664 0000000 0000000 00000000144 15244036076 0027553 0 ustar 00root root 0000000 0000000 ``ESRFPath_tomoproc_v1``
========================
.. autoclass:: esrf_pathlib.ESRFPath_tomoproc_v1
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/reference/unknown_v1.rst 0000664 0000000 0000000 00000000141 15244036076 0027405 0 ustar 00root root 0000000 0000000 ``ESRFPath_unknown_v1``
=======================
.. autoclass:: esrf_pathlib.ESRFPath_unknown_v1
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/tutorials.rst 0000664 0000000 0000000 00000000122 15244036076 0025367 0 ustar 00root root 0000000 0000000 Tutorials
=========
.. toctree::
:maxdepth: 1
tutorials/getting_started
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/tutorials/ 0000775 0000000 0000000 00000000000 15244036076 0024642 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/doc/tutorials/getting_started.rst 0000664 0000000 0000000 00000003574 15244036076 0030574 0 ustar 00root root 0000000 0000000 Getting started
===============
Install *esrf-pathlib*
.. code-block:: bash
pip install esrf-pathlib
Parse an ESRF path
-------------------
:class:`~esrf_pathlib.ESRFPath` behaves like `pathlib.Path`_. When it matches a
recognized ESRF *path template*, it exposes additional attributes such as
`proposal` and `processed_dataset_file`
.. code-block:: python-console
>>> from esrf_pathlib import ESRFPath
>>>
>>> path = ESRFPath("/data/visitor/ma1234/id00/20250509/RAW_DATA/sample/sample_dataset")
>>> path
>>> path.proposal
'ma1234'
>>> path.processed_dataset_file
The available attributes depend on which ESRF path template the path matches.
Use it like a regular path
---------------------------
``ESRFPath`` remains a fully compatible `pathlib.Path`_
.. code-block:: python-console
>>> path.is_absolute()
True
>>> path.root
'/'
and ESRF-specific attributes stay available after manipulating the path
.. code-block:: python-console
>>> path = ESRFPath("/data/visitor/ma1234/id00/20250509/PROCESSED_DATA")
>>> new_path = path / "result.h5"
>>> new_path
>>> new_path.data_type
Next steps
----------
* :doc:`../howtoguides/search_with_glob` to search collections and datasets.
* :doc:`../howtoguides/replace_fields` to derive related paths.
* :doc:`../explanations/schemas_and_templates` to understand how matching works.
.. _pathlib.Path: https://docs.python.org/3/library/pathlib.html
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/pyproject.toml 0000664 0000000 0000000 00000004232 15244036076 0024764 0 ustar 00root root 0000000 0000000 [build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"
[project]
name = "esrf-pathlib"
version = "1.2.0rc6"
authors = [{ name = "ESRF", email = "dau-pydev@esrf.fr" }]
description = "A drop-in pathlib clone with ESRF-specific path utilities"
readme = { file = "README.md", content-type = "text/markdown" }
license = { file = "LICENSE.md" }
keywords = ["ESRF", "pathlib"]
classifiers = [
"Intended Audience :: Science/Research",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
]
requires-python = ">=3.8"
dependencies = []
[project.urls]
Homepage = "https://gitlab.esrf.fr/workflow/esrf-pathlib"
Documentation = "https://esrf-pathlib.readthedocs.io/"
Repository = "https://gitlab.esrf.fr/workflow/esrf-pathlib"
Issues = "https://gitlab.esrf.fr/workflow/esrf-pathlib/issues"
Changelog = "https://gitlab.esrf.fr/workflow/esrf-pathlib/-/blob/main/CHANGELOG.md"
[project.optional-dependencies]
test = ["pytest >=7.0"]
dev = ["esrf-pathlib[test]", "ruff"]
doc = [
"esrf-pathlib[test]",
"sphinx >=6.0",
"sphinx-argparse",
"sphinx-autodoc-typehints",
"pydata-sphinx-theme",
"sphinx-copybutton",
]
[tool.setuptools]
package-dir = { "" = "src" }
[tool.setuptools.packages.find]
where = ["src"]
[tool.setuptools.package-data]
"*" = ["*.json", "*.ipynb"]
[tool.coverage.run]
omit = ["*/tests/*"]
[tool.ruff]
line-length = 88
[tool.ruff.lint]
extend-select = [
"E", # pycodestyle errors
"F", # pyflakes
"I", # isort
"S", # flake8-bandit
]
ignore = [
"E501", # line-too-long: many docstrings contain ASCII path-tree diagrams
"FA100", # would rewrite typing.Optional/Union to `X | Y`, which breaks
# runtime introspection (`typing.get_type_hints`) on Python <3.10; this
# package supports Python 3.8
"PYI019", # would rewrite the self-type TypeVar to `typing_extensions.Self`,
# which is not a dependency of this package (`typing.Self` requires Python 3.11)
]
[tool.ruff.lint.per-file-ignores]
"**/test_*.py" = ["S101"]
[tool.ruff.lint.isort]
force-single-line = true
known-first-party = ["esrf_pathlib"]
[project.scripts]
esrf-pathlib = "esrf_pathlib.__main__:main"
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/scripts/ 0000775 0000000 0000000 00000000000 15244036076 0023536 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/scripts/examples.py 0000664 0000000 0000000 00000010505 15244036076 0025727 0 ustar 00root root 0000000 0000000 """
.. code-block:: bash
python scripts/examples.py
python scripts/examples.py esrf=3 tomo=1
python scripts/examples.py esrf=2
python scripts/examples.py esrf=1
"""
import argparse
import logging
from esrf_pathlib._schemas.errors import PathSchemaMatchError
from esrf_pathlib._schemas.registry import PathSchemaRegistry
from esrf_pathlib._schemas.registry import get_schema_registry
def example(registry: PathSchemaRegistry):
paths = [
"/data/visitor/ma1234/id23eh1/20250509/RAW_DATA/sample/sample_dataset/scan0001/master.h5",
"/data/visitor/ma1234/id23eh1/20250509/PROCESSED_DATA/sample/sample_dataset/sample_dataset.h5",
"/data/visitor/ma1234/id23eh1/20250509/RAW_DATA/__icat__/sample_dataset.xml",
"/data/visitor/ma1234/id23eh1/20250509/PROCESSED_DATA/sample/sample_dataset/projections/gallery/sample_dataset_90deg.png",
"/data/visitor/ma1234/id23eh1/20250509/PROCESSED_DATA/sample/sample_dataset/slices/gallery/sample_dataset_absorption_xz_00001.png",
"/data/visitor/ma1234/id23eh1/20250509/raw/sample/sample_dataset/sample_dataset.h5",
"/data/visitor/ma1234/id23eh1/20250509/processed/sample/sample_dataset/sample_dataset.h5",
"/data/visitor/ma1234/id23eh1/20250509/_nobackup/sample/sample_dataset/sample_dataset.h5",
"/data/visitor/ma1234/id23eh1/20250509/raw/sample/sample_dataset/scan0001/master.h5",
"/data/visitor/ma1234/id23eh1/20250509/sample/sample_dataset/scan0001/master.h5",
"/home/user/mydata/sample_dataset/slices/gallery/sample_dataset_absorption_xz_00001.png",
"/home/user/mydata/sample_dataset/volumes/sample_dataset_absorption_16Bit_vol",
"/some/random/path",
]
for path in paths:
print()
try:
schema = registry.parse_path(path)
except PathSchemaMatchError:
print(f"NOT parsed: {path}")
else:
print(f"Parsed: {path}")
print(schema.__info__())
def examples_esrf_v3_tomo_v1(registry: PathSchemaRegistry):
schema = registry.parse_path(
"/data/visitor/ma1234/id23eh1/20250509/PROCESSED_DATA/sample/sample_dataset/sample_dataset.h5"
)
print()
print(schema.__info__())
slice_schema = schema.replace_concepts(
tomo_technique="absorption",
tomo_slicing_direction="xz",
tomo_slice_number=1,
thumbnail_file_type="png",
)
print()
print(slice_schema.__info__())
print("Slice thumbnail path:", slice_schema.slices_thumbnail)
preview_schema = schema.replace_concepts(tomo_angle=90, thumbnail_file_type="jpg")
print()
print(preview_schema.__info__())
print("Projection thumbnail:", preview_schema.projections_thumbnail)
unknown_schema = registry.parse_path(
"/data/visitor/ma1234/id23eh1/20250509/RAW_DATA/sample/sample_dataset/scan0001/master.h5"
)
print()
print(unknown_schema.__info__())
print("Unknown part:", unknown_schema.unknown_part)
fields = {
"beamline": "id23eh1",
"proposal": "ma1234",
"session_date": "20250509",
"collection": "sample",
}
print()
print("Path from fields:", registry.render_path(fields))
print("Path from fields:", registry.render_path(fields, "raw_data_path"))
fields = {
"beamline": "id23eh1",
"proposal": "*",
"session_date": "*",
"collection": "sample",
}
print()
print("Path from fields for glob:", registry.render_path(fields))
print("Path from fields for glob:", registry.render_path(fields, "raw_data_path"))
def parse_schemas(schemas_list):
"""Convert list like ['esrf=3', 'tomo=1'] to dict."""
result = {}
for s in schemas_list or []:
key, val = s.split("=")
result[key] = int(val)
return result
def main():
parser = argparse.ArgumentParser(description="Run ESRF path examples.")
parser.add_argument(
"schemas", nargs="*", default=[], help="Schema versions, e.g. esrf=3 tomo=1"
)
args = parser.parse_args()
logging.basicConfig(level=logging.DEBUG)
schema_kwargs = parse_schemas(args.schemas)
registry = get_schema_registry(**schema_kwargs)
print(registry.__info__())
example(registry)
if schema_kwargs.get("esrf") == 3 and schema_kwargs.get("tomo") == 1:
examples_esrf_v3_tomo_v1(registry)
if __name__ == "__main__":
main()
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/ 0000775 0000000 0000000 00000000000 15244036076 0022636 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/ 0000775 0000000 0000000 00000000000 15244036076 0025300 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/__init__.py 0000664 0000000 0000000 00000000050 15244036076 0027404 0 ustar 00root root 0000000 0000000 from .core import ESRFPath # noqa F401
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/__main__.py 0000664 0000000 0000000 00000002607 15244036076 0027377 0 ustar 00root root 0000000 0000000 """Print ESRF path information
.. code-block:: bash
esrf-pathlib esrf=3 tomo=1
esrf-pathlib esrf=3 --fallback-depth=1
"""
import argparse
import json
import logging
from typing import List
from ._schemas.registry import get_schema_registry
from .core import ESRFPath
def _parse_cli_schemas(schemas_list: List[str]):
"""Convert list like ['esrf=3', 'tomo=1'] to dict."""
result = {}
for s in schemas_list or []:
key, value = s.split("=")
result[key] = json.loads(value)
if not result:
return ESRFPath._SCHEMAS
return result
def create_argument_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="esrf-pathlib", description="Print ESRF path information."
)
parser.add_argument(
"schemas",
nargs="*",
default=[],
help="Schema versions, e.g. esrf=3 tomo=1",
)
parser.add_argument(
"--fallback-depth",
type=int,
default=0,
help="Include previous schema versions as fallback",
)
return parser
def main():
parser = create_argument_parser()
args = parser.parse_args()
logging.basicConfig(level=logging.DEBUG)
schema_kwargs = _parse_cli_schemas(args.schemas)
registry = get_schema_registry(**schema_kwargs, fallback_depth=args.fallback_depth)
print(registry.__info__())
if __name__ == "__main__":
main()
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/ 0000775 0000000 0000000 00000000000 15244036076 0027062 5 ustar 00root root 0000000 0000000 __init__.py 0000664 0000000 0000000 00000000200 15244036076 0031104 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas """Path schema system for matching, parsing and rendering ESRF filesystem paths.
A *schema registry* is the top-level API.
"""
constants.py 0000664 0000000 0000000 00000000357 15244036076 0031376 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas import re
import sys
IS_WINDOWS = sys.platform == "win32"
UNKNOWN_SCHEMA_NAME = "unknown"
UNKNOWN_PART_NAME = "unknown_part"
PLACEHOLDER_REGEX = re.compile(
r"\{([^\}=]+)(?:=([^\}]+))?\}"
) # matches {} or {=}
definitions/ 0000775 0000000 0000000 00000000000 15244036076 0031316 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas __init__.py 0000664 0000000 0000000 00000000057 15244036076 0033431 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions """ESRF filesystem path schema definitions."""
esrf/ 0000775 0000000 0000000 00000000000 15244036076 0032255 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions __init__.py 0000664 0000000 0000000 00000000106 15244036076 0034363 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/esrf """ESRF Data Policy schema definitions (``esrf`` schema versions)."""
esrf_utils.py 0000664 0000000 0000000 00000013070 15244036076 0035007 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/esrf import datetime
import os
import pathlib
import re
from typing import Any
from typing import Dict
from typing import Union
from ... import constants
from ...typing import TemplateTree
from ..types import ConceptValueType
from ..types import DataType
from ..utils import as_path_string
from ..utils import as_pathlib_path
from ..utils import deserializer_enum_type
from ..utils import named_directory
from ..utils import serializer_enum_type
DEFAULT_DATA_ROOT = (
None if constants.IS_WINDOWS else pathlib.Path(os.sep, "data", "visitor")
)
_BEAMLINE_NAME_TO_DIR = {
"id23-1": "id23eh1",
"id23-2": "id23eh2",
"id30a-1": "id30a1",
"id30a-2": "id30a2",
"id30a-3": "id30a3",
}
_BEAMLINE_DIR_TO_NAME = {v: k for k, v in _BEAMLINE_NAME_TO_DIR.items()}
# Names of proposal, beamline, sample, dataset
_FORBIDDEN_CHARS_BASE = {os.sep, "{", "}", "%", "\x00"}
_FORBIDDEN_CHARS_WINDOWS = {"<", ">", ":", '"', "'", "/", "\\", "|", "?", "*"} | {
chr(i) for i in range(32)
}
_FORBIDDEN_CHARS_URL = {"@", "#", ":", "?"}
_FORBIDDEN_CHARS = (
_FORBIDDEN_CHARS_BASE | _FORBIDDEN_CHARS_WINDOWS | _FORBIDDEN_CHARS_URL
)
_BLISS_ALLOWED_CHARS = f"[^{re.escape(''.join(_FORBIDDEN_CHARS))}]"
# BLISS_NAME_PATTERN = f"{_BLISS_ALLOWED_CHARS}+"
BLISS_NAME_PATTERN = rf"{_BLISS_ALLOWED_CHARS}+(? datetime.date:
if isinstance(session_date, datetime.datetime):
return session_date.date()
if isinstance(session_date, datetime.date):
return session_date
return datetime.datetime.strptime( # noqa: DTZ007 -- date-only format, no time/tz component
session_date, "%Y%m%d"
).date()
def serialize_session_date(
session_date: Union[str, datetime.date, datetime.datetime],
) -> str:
if isinstance(session_date, str):
return session_date
if isinstance(session_date, datetime.datetime):
session_date = session_date.date()
return session_date.strftime("%Y%m%d")
def serialize_beamline(normalized_beamline: str) -> str:
name = normalized_beamline.lower()
return _BEAMLINE_NAME_TO_DIR.get(name, name)
def derive_normalized_beamline(values: Dict[str, ConceptValueType]) -> str:
name = values["beamline"].lower()
return _BEAMLINE_DIR_TO_NAME.get(name, name)
def data_type_tree(
prefix: str, path_name: str, *, include_icat: bool = False
) -> TemplateTree:
"""Directory content shared by the ``RAW_DATA``, ``PROCESSED_DATA`` and
``NOBACKUP`` data types: a proposal file, a ``{collection}`` breakdown and,
for raw data, an ``__icat__`` metadata directory.
"""
tree: TemplateTree = {
named_directory(path_name): None,
("{proposal}_{beamline}.h5", f"{prefix}_proposal_file"): None,
"{collection}": {
named_directory(f"{prefix}_collection_path"): None,
("{proposal}_{collection}.h5", f"{prefix}_collection_file"): None,
"{collection}_{dataset}": {
named_directory(f"{prefix}_dataset_path"): None,
("{collection}_{dataset}.h5", f"{prefix}_dataset_file"): None,
"gallery": {
named_directory(f"{prefix}_gallery_path"): None,
},
},
},
}
if include_icat:
tree["__icat__"] = {
named_directory(f"{prefix}_metadata_path"): None,
("{collection}_{dataset}.xml", f"{prefix}_metadata_file"): None,
}
return tree
BASE_CONCEPTS = {
"data_root": {
"description": "Root directory",
"regex": r".*",
"default_value": DEFAULT_DATA_ROOT,
"deserializer": as_pathlib_path,
"serializer": as_path_string,
},
"proposal": {
"description": "ESRF proposal name",
"regex": BLISS_NAME_PATTERN,
"examples": ["ma1234", "blc1234"],
},
"beamline": {
"description": "Beamline name",
"regex": BLISS_NAME_PATTERN,
"serializer": serialize_beamline,
"examples": ["id21", "id23eh1", "id30a1"],
},
"session_date": {
"description": "Start date of the experimental session (YYYYMMDD)",
"regex": SESSION_DATE_PATTERN,
"deserializer": deserialize_session_date,
"serializer": serialize_session_date,
},
"collection": {
"description": "Collection of datasets",
"regex": BLISS_NAME_PATTERN,
},
"dataset": {
"description": "Dataset name (smallest entity in the data portal)",
"regex": BLISS_NAME_PATTERN,
},
}
BEAMLINE_NORMALIZED_DERIVED_CONCEPT = {
"beamline_normalized": {
"description": "Normalized beamline name for the Data Portal",
"examples": ["id21", "id23-1", "id30a-1"],
"derive_func": derive_normalized_beamline,
"derived_from": ["beamline"],
}
}
def data_type_concept(string_to_enum: Dict[str, DataType]) -> Dict[str, Any]:
"""The ``data_type`` concept for a given ``{string: DataType}`` mapping."""
def _deserializer(data_type: Union[str, DataType]) -> DataType:
return deserializer_enum_type(
data_type, enum_type=DataType, string_to_enum=string_to_enum
)
def _serializer(data_type: Union[str, DataType]) -> str:
return serializer_enum_type(
data_type, enum_type=DataType, string_to_enum=string_to_enum
)
return {
"description": "Data type",
"regex": "|".join(list(string_to_enum)),
"deserializer": _deserializer,
"serializer": _serializer,
}
esrf_v1.py 0000664 0000000 0000000 00000007715 15244036076 0034206 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/esrf """
ESRF Data Policy v1
Introduced with BLISS 1.2.0 for beamline data on Feb 11, 2020.
Path hierarchy::
{data_root}/{proposal}/{beamline}/ [TID]
└── {session_date}/ [BLISS]
├── {proposal}_{beamline}.h5
├── {collection}/
| ├── {proposal}_{collection}.h5
| └── {dataset}/
| └── {collection}_{dataset}.h5
└── __icat__/
└── {collection}_{dataset}.xml
"""
from ...schema.path import PathSchema
from ...typing import TemplateTree
from ..types import DataType
from ..unknown import unknown_v1
from ..utils import as_path_string
from ..utils import as_pathlib_path
from .esrf_utils import BLISS_NAME_PATTERN
from .esrf_utils import DEFAULT_DATA_ROOT
from .esrf_utils import SESSION_DATE_PATTERN
from .esrf_utils import derive_normalized_beamline
from .esrf_utils import deserialize_session_date
from .esrf_utils import named_directory
from .esrf_utils import serialize_beamline
from .esrf_utils import serialize_session_date
_SESSION_TREE: TemplateTree = {
named_directory("session_path"): None,
("{proposal}_{beamline}.h5", "raw_proposal_file"): None,
"{collection}": {
named_directory("raw_collection_path"): None,
("{proposal}_{collection}.h5", "raw_collection_file"): None,
"{collection}_{dataset}": {
named_directory("raw_dataset_path"): None,
("{collection}_{dataset}.h5", "raw_dataset_file"): None,
"gallery": {
named_directory("raw_gallery_path"): None,
},
},
},
"__icat__": {
named_directory("raw_metadata_path"): None,
("{collection}_{dataset}.xml", "raw_metadata_file"): None,
},
}
_ROOT_TREE: TemplateTree = {
"{data_root}": {
"{proposal}": {
"{beamline}": {
"{session_date}": _SESSION_TREE,
},
},
},
}
_CONCEPTS = {
"data_root": {
"description": "Root directory",
"regex": r".*",
"default_value": DEFAULT_DATA_ROOT,
"deserializer": as_pathlib_path,
"serializer": as_path_string,
},
"proposal": {
"description": "ESRF proposal name",
"regex": BLISS_NAME_PATTERN,
"examples": ["ma1234", "blc1234"],
},
"beamline": {
"description": "Beamline name",
"regex": BLISS_NAME_PATTERN,
"serializer": serialize_beamline,
"examples": ["id21", "id23eh1", "id30a1"],
},
"session_date": {
"description": "Start date of the experimental session (YYYYMMDD)",
"regex": SESSION_DATE_PATTERN,
"deserializer": deserialize_session_date,
"serializer": serialize_session_date,
},
"collection": {
"description": "Collection identifier grouping datasets",
"regex": BLISS_NAME_PATTERN,
},
"dataset": {
"description": "Dataset name (smallest entity in the data portal)",
"regex": BLISS_NAME_PATTERN,
},
}
def _derive_data_type(_) -> DataType:
return DataType.RAW
_DERIVED_CONCEPTS = {
"beamline_normalized": {
"description": "Normalized beamline name for the Data Portal",
"examples": ["id21", "id23-1", "id30a-1"],
"derive_func": derive_normalized_beamline,
"derived_from": ["beamline"],
},
"data_type": {
"description": "Data type. Fixed to 'raw' for this schema.",
"derive_func": _derive_data_type,
"derived_from": [],
},
}
_SCHEMA = None
def get_schema() -> PathSchema:
global _SCHEMA
if _SCHEMA is None:
_SCHEMA = PathSchema(
name="esrf",
version=1,
description=__doc__,
concepts=_CONCEPTS,
derived_concepts=_DERIVED_CONCEPTS,
template_tree=_ROOT_TREE,
extends=[unknown_v1.get_schema()],
symbolic_root_segment="data",
)
return _SCHEMA
esrf_v2.py 0000664 0000000 0000000 00000005301 15244036076 0034174 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/esrf """
ESRF Data Policy v2
Introduced with BLISS 1.10.1 for beamline data on Dec 16, 2022.
Path hierarchy::
{data_root}/{proposal}/{beamline}/{session_date}/{data_type}/ [TID]
├── {data_type=raw}/ [BLISS]
│ ├── {proposal}_{beamline}.h5
│ ├── {collection}/
│ │ ├── {proposal}_{collection}.h5
│ | └── {dataset}/
│ | └── {collection}_{dataset}.h5
| └── __icat__/
| └── {collection}_{dataset}.xml
├── {data_type=processed}/ [esrf-pathlib]
│ ├── {proposal}_{beamline}.h5
│ └── {collection}/
│ ├── {proposal}_{collection}.h5
│ └── {dataset}/
│ └── {collection}_{dataset}.h5
└── {data_type=_nobackup}/ [esrf-pathlib]
├── {proposal}_{beamline}.h5
└── {collection}/
├── {proposal}_{collection}.h5
└── {dataset}/
└── {collection}_{dataset}.h5
"""
from ...schema.path import PathSchema
from ...typing import TemplateTree
from ..types import DataType
from ..unknown import unknown_v1
from . import esrf_utils
_STRING_TO_DATA_TYPE = {
"raw": DataType.RAW,
"processed": DataType.PROCESSED,
"_nobackup": DataType.NOBACKUP,
}
_SESSION_TREE: TemplateTree = {
esrf_utils.named_directory("session_path"): None,
"{data_type=raw}": esrf_utils.data_type_tree(
"raw", "raw_data_path", include_icat=True
),
"{data_type=processed}": esrf_utils.data_type_tree(
"processed", "processed_data_path"
),
"{data_type=_nobackup}": esrf_utils.data_type_tree("nobackup", "nobackup_path"),
}
_ROOT_TREE: TemplateTree = {
"{data_root}": {
"{proposal}": {
"{beamline}": {
"{session_date}": _SESSION_TREE,
},
},
}
}
_CONCEPTS = {
**esrf_utils.BASE_CONCEPTS,
"data_type": esrf_utils.data_type_concept(_STRING_TO_DATA_TYPE),
}
_DERIVED_CONCEPTS = esrf_utils.BEAMLINE_NORMALIZED_DERIVED_CONCEPT
_SCHEMA = None
def get_schema() -> PathSchema:
global _SCHEMA
if _SCHEMA is None:
_SCHEMA = PathSchema(
name="esrf",
version=2,
description=__doc__,
concepts=_CONCEPTS,
derived_concepts=_DERIVED_CONCEPTS,
template_tree=_ROOT_TREE,
extends=[unknown_v1.get_schema()],
symbolic_root_segment="data",
)
return _SCHEMA
esrf_v3.py 0000664 0000000 0000000 00000006251 15244036076 0034202 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/esrf """
ESRF Data Policy v3
Introduced with BLISS 1.11.1 for beamline data on Mar 24, 2023.
Path hierarchy::
{data_root}/{proposal}/{beamline}/{session_date}/{data_type}/ [TID]
├── {data_type=RAW_DATA}/
│ ├── {proposal}_{beamline}.h5 [BLISS]
│ ├── {collection}/
│ │ ├── {proposal}_{collection}.h5
│ | └── {dataset}/
│ | └── {collection}_{dataset}.h5
| └── __icat__/
| └── {collection}_{dataset}.xml
├── {data_type=PROCESSED_DATA}/
│ ├── {proposal}_{beamline}.h5 [esrf-pathlib]
│ └── {collection}/
│ ├── {proposal}_{collection}.h5
│ └── {dataset}/
│ └── {collection}_{dataset}.h5
├── {data_type=NOBACKUP}/
│ ├── {proposal}_{beamline}.h5 [esrf-pathlib]
│ └── {collection}/
│ ├── {proposal}_{collection}.h5
│ └── {dataset}/
│ └── {collection}_{dataset}.h5
├── {data_type=SCRIPTS}/
├── {data_type=GALLERY}/
└── {data_type=NOTES}/
"""
from ...schema.path import PathSchema
from ...typing import TemplateTree
from ..types import DataType
from ..unknown import unknown_v1
from . import esrf_utils
_STRING_TO_DATA_TYPE = {
"RAW_DATA": DataType.RAW,
"PROCESSED_DATA": DataType.PROCESSED,
"NOBACKUP": DataType.NOBACKUP,
"SCRIPTS": DataType.SCRIPT,
"GALLERY": DataType.GALLERY,
"NOTES": DataType.NOTE,
}
_SESSION_TREE: TemplateTree = {
esrf_utils.named_directory("session_path"): None,
"{data_type=RAW_DATA}": esrf_utils.data_type_tree(
"raw", "raw_data_path", include_icat=True
),
"{data_type=PROCESSED_DATA}": esrf_utils.data_type_tree(
"processed", "processed_data_path"
),
"{data_type=NOBACKUP}": esrf_utils.data_type_tree("nobackup", "nobackup_path"),
"{data_type=SCRIPTS}": {
esrf_utils.named_directory("scripts_path"): None,
},
"{data_type=GALLERY}": {
esrf_utils.named_directory("gallery_path"): None,
},
"{data_type=NOTES}": {
esrf_utils.named_directory("notes_path"): None,
},
}
_ROOT_TREE: TemplateTree = {
"{data_root}": {
"{proposal}": {
"{beamline}": {
"{session_date}": _SESSION_TREE,
},
},
},
}
_CONCEPTS = {
**esrf_utils.BASE_CONCEPTS,
"data_type": esrf_utils.data_type_concept(_STRING_TO_DATA_TYPE),
}
_DERIVED_CONCEPTS = esrf_utils.BEAMLINE_NORMALIZED_DERIVED_CONCEPT
_SCHEMA = None
def get_schema() -> PathSchema:
global _SCHEMA
if _SCHEMA is None:
_SCHEMA = PathSchema(
name="esrf",
version=3,
description=__doc__,
concepts=_CONCEPTS,
derived_concepts=_DERIVED_CONCEPTS,
template_tree=_ROOT_TREE,
extends=[unknown_v1.get_schema()],
symbolic_root_segment="data",
)
return _SCHEMA
esrf_v4.py 0000664 0000000 0000000 00000006530 15244036076 0034203 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/esrf """
ESRF Data Policy v4
Introduced for beamline data on Jan 1, 2026.
Path hierarchy::
{data_root}/{proposal}/{beamline}/{session_date}/{data_type}/ [TID]
├── {data_type=RAW_DATA}/
│ ├── {proposal}_{beamline}.h5 [BLISS]
│ ├── {collection}/
│ │ ├── {proposal}_{collection}.h5
│ | └── {dataset}/
│ | └── {collection}_{dataset}.h5
| └── __icat__/
| └── {collection}_{dataset}.xml
├── {data_type=PROCESSED_DATA}/
│ ├── {proposal}_{beamline}.h5 [esrf-pathlib]
│ └── {collection}/
│ ├── {proposal}_{collection}.h5
│ └── {dataset}/
│ └── {collection}_{dataset}.h5
├── {data_type=NOBACKUP}/
│ ├── {proposal}_{beamline}.h5 [esrf-pathlib]
│ └── {collection}/
│ ├── {proposal}_{collection}.h5
│ └── {dataset}/
│ └── {collection}_{dataset}.h5
├── {data_type=AUXILIARY_DATA}/
├── {data_type=SCRIPTS}/
├── {data_type=GALLERY}/
└── {data_type=NOTES}/
"""
from ...schema.path import PathSchema
from ...typing import TemplateTree
from ..types import DataType
from ..unknown import unknown_v1
from . import esrf_utils
_STRING_TO_DATA_TYPE = {
"RAW_DATA": DataType.RAW,
"PROCESSED_DATA": DataType.PROCESSED,
"NOBACKUP": DataType.NOBACKUP,
"AUXILIARY_DATA": DataType.AUXILIARY,
"SCRIPTS": DataType.SCRIPT,
"GALLERY": DataType.GALLERY,
"NOTES": DataType.NOTE,
}
_SESSION_TREE: TemplateTree = {
esrf_utils.named_directory("session_path"): None,
"{data_type=RAW_DATA}": esrf_utils.data_type_tree(
"raw", "raw_data_path", include_icat=True
),
"{data_type=PROCESSED_DATA}": esrf_utils.data_type_tree(
"processed", "processed_data_path"
),
"{data_type=NOBACKUP}": esrf_utils.data_type_tree("nobackup", "nobackup_path"),
"{data_type=AUXILIARY_DATA}": {
esrf_utils.named_directory("auxiliary_data_path"): None,
},
"{data_type=SCRIPTS}": {
esrf_utils.named_directory("scripts_path"): None,
},
"{data_type=GALLERY}": {
esrf_utils.named_directory("gallery_path"): None,
},
"{data_type=NOTES}": {
esrf_utils.named_directory("notes_path"): None,
},
}
_ROOT_TREE: TemplateTree = {
"{data_root}": {
"{proposal}": {
"{beamline}": {
"{session_date}": _SESSION_TREE,
},
},
},
}
_CONCEPTS = {
**esrf_utils.BASE_CONCEPTS,
"data_type": esrf_utils.data_type_concept(_STRING_TO_DATA_TYPE),
}
_DERIVED_CONCEPTS = esrf_utils.BEAMLINE_NORMALIZED_DERIVED_CONCEPT
_SCHEMA = None
def get_schema() -> PathSchema:
global _SCHEMA
if _SCHEMA is None:
_SCHEMA = PathSchema(
name="esrf",
version=4,
description=__doc__,
concepts=_CONCEPTS,
derived_concepts=_DERIVED_CONCEPTS,
template_tree=_ROOT_TREE,
extends=[unknown_v1.get_schema()],
symbolic_root_segment="data",
)
return _SCHEMA
load.py 0000664 0000000 0000000 00000005523 15244036076 0032614 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions import importlib
from itertools import product
from typing import Dict
from typing import List
from typing import Optional
from ..errors import UnknownPathSchema
from ..schema.path import PathSchema
_ALL_VERSIONS = {"unknown": [1], "esrf": [1, 2, 3, 4], "tomo": [1, 2], "tomoproc": [1]}
# Schemas are grouped in sub-packages of `definitions`. The module holding a
# given schema version is named after this template with `{version}` filled in.
_MODULE_TEMPLATES = {
"unknown": "unknown.unknown_v{version}",
"esrf": "esrf.esrf_v{version}",
"tomo": "tomo.tomo_v{version}",
"tomoproc": "tomo.tomoproc_v{version}",
}
def default_schema_version(name: str) -> int:
"""
:raises UnknownPathSchema
"""
if name not in _ALL_VERSIONS:
raise UnknownPathSchema(f"Schema {name!r} is not known")
return max(_ALL_VERSIONS[name])
def fallback_schema_versions(
start_versions: Dict[str, int],
fallback_depth: Optional[int] = None,
) -> List[Dict[str, int]]:
"""
Return fallback schema versions starting from ``start_versions``.
- Excludes ``start_versions``.
- Versions are sorted from high to low.
- If fallback_depth is None -> include all older versions.
- If fallback_depth is set -> include up to that many older versions.
:raises ValueError:
:raises UnknownPathSchema:
"""
fallback_versions = {}
if fallback_depth and fallback_depth < 0:
raise ValueError("'fallback_depth' must be >=0")
for sname, start_version in start_versions.items():
if sname not in _ALL_VERSIONS:
raise UnknownPathSchema(f"Schema {sname!r} is not known")
all_versions = sorted(_ALL_VERSIONS.get(sname), reverse=True)
if start_version not in all_versions:
raise UnknownPathSchema(
f"Schema {sname!r} does not have version {start_version}"
)
# Exclude current version, take older ones
idx = all_versions.index(start_version)
fallbacks = all_versions[idx + 1 :]
# Limit fallback depth
if fallback_depth is not None:
fallbacks = fallbacks[:fallback_depth]
fallback_versions[sname] = fallbacks
return [
dict(zip(start_versions.keys(), combination))
for combination in product(
*(fallback_versions[sname] for sname in start_versions)
)
]
def get_schema(name: str, version: int) -> PathSchema:
"""
:raises UnknownPathSchema:
"""
if name not in _MODULE_TEMPLATES:
raise UnknownPathSchema(f"Schema {name!r} is not known")
module_name = _MODULE_TEMPLATES[name].format(version=version)
try:
mod = importlib.import_module(f".{module_name}", package=__package__)
except ImportError as ex:
raise UnknownPathSchema(f"{name!r} version {version} does not exist") from ex
return mod.get_schema()
tomo/ 0000775 0000000 0000000 00000000000 15244036076 0032274 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions __init__.py 0000664 0000000 0000000 00000000126 15244036076 0034404 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/tomo """Tomography data schema definitions (``tomo`` and ``tomoproc`` schema versions)."""
tomo_utils.py 0000664 0000000 0000000 00000027670 15244036076 0035060 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/tomo """Definitions shared between the *tomo* and *tomoproc* schemas.
The *tomo* schema describes tomography processed data rooted at the ESRF
session directory
.. code::
{data_root}/{proposal}/{beamline}/{session_date}/{data_type=PROCESSED_DATA}/{collection}/{tomo_dataset}
while the *tomoproc* schema describes the same processed data rooted at an
arbitrary directory
.. code::
{data_root}
Everything below the dataset directory (:data:`RESULT_TREE`) and the concepts
used by it (:data:`CONCEPTS`) are defined here once so that both schemas
describe the dataset results in exactly the same way.
The dataset directory name is also the prefix of the result file names. It is
the ``tomo_dataset`` concept which is a path concept for *tomoproc*
(:data:`DATASET_CONCEPT`) because the collection/dataset split of a local
directory name is unknown, and a derived concept for *tomo*
(:data:`DATASET_DERIVED_CONCEPT`).
"""
from typing import Union
from ...schema.base import BasePathSchema
from ...schema.path import PathSchema
from ...typing import TemplateTree
from ..esrf.esrf_utils import BLISS_NAME_PATTERN
from ..types import TomoResultType
from ..utils import deserializer_enum_type
from ..utils import named_directory
from ..utils import serializer_enum_type
RESULT_TREE: TemplateTree = {
"references": {
named_directory("references_path"): None,
("{tomo_dataset}_darks.hdf5", "references_dark"): None,
("{tomo_dataset}_flats.hdf5", "references_flat"): None,
},
"projections": {
named_directory("projections_path"): None,
("{tomo_dataset}.nx", "projections_file"): None,
"gallery": {
named_directory("projections_gallery_path"): None,
("metadata.json", "projections_metadata_file"): None,
(
"{tomo_dataset}_{tomo_angle}_large.{thumbnail_file_type}",
"projections_thumbnail",
): None,
(
"{tomo_dataset}_{tomo_angle}.{thumbnail_file_type}",
"projections_preview",
): None,
},
},
"slices": {
named_directory("slices_path"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}",
"slices_file",
): None,
"gallery": {
named_directory("slices_gallery_path"): None,
("metadata.json", "slices_metadata_file"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}",
"slices_thumbnail",
): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}",
"slices_preview",
): None,
},
},
"volumes": {
named_directory("volumes_path"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}",
"volumes_technique_path",
True,
("volumes_file",),
): {
(
"{tomo_dataset}_{tomo_slice_number}.{tomo_extension}",
"volumes_slice_file",
): None,
"gallery": {
named_directory("volumes_gallery_path"): None,
("metadata.json", "volumes_metadata_file"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}",
"volumes_thumbnail",
): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}",
"volumes_preview",
): None,
},
},
},
"volumes_{tomo_type}": {
named_directory("volumes_custom_type_path"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_type}_{tomo_dtype}_{tomo_extension}",
"volumes_custom_type_technique_path",
True,
("volumes_custom_type_file",),
): {
(
"{tomo_dataset}_{tomo_slice_number}.{tomo_extension}",
"volumes_custom_type_slice_file",
): None,
"gallery": {
named_directory("volumes_custom_type_gallery_path"): None,
("metadata.json", "volumes_custom_type_metadata_file"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_type}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}",
"volumes_custom_type_thumbnail",
): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_type}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}",
"volumes_custom_type_preview",
): None,
},
},
},
"logs": {
named_directory("tomo_log_path"): None,
("{tomo_dataset}_{tomo_application}.log", "tomo_log_file"): None,
},
"workflows": {
named_directory("workflows_path"): None,
"gallery": {
named_directory("workflows_gallery_path"): None,
("{tomo_dataset}.json", "workflows_file"): None,
},
},
}
def _deserializer_angle(string_value: str) -> Union[int, float]:
string_value = string_value.replace("deg", "")
if string_value.lstrip("-").isdigit():
return int(string_value)
return float(string_value)
def _serializer_angle(py_value: Union[str, float]) -> str:
return f"{float(py_value):.2f}deg"
def _deserializer_slicing_direction(string_value: str) -> str:
value = string_value.lower()
if value not in {"xy", "yz", "xz"}:
raise ValueError(f"Invalid slicing direction: {string_value!r}")
return value
def _serializer_slicing_direction(py_value: str) -> str:
return _deserializer_slicing_direction(py_value)
def _serializer_slice_number(py_value: Union[str, int]) -> str:
return f"{int(py_value):05d}"
ALPHA_NUMERIC_REGEX = r"[a-zA-Z0-9]+"
_DATASET_DESCRIPTION = "Dataset directory name and result file name prefix"
_DATASET_EXAMPLES = ["sample_0001"]
DATASET_CONCEPT = {
"tomo_dataset": {
"description": _DATASET_DESCRIPTION,
"regex": BLISS_NAME_PATTERN,
"examples": _DATASET_EXAMPLES,
}
}
DATASET_DERIVED_CONCEPT = {
"tomo_dataset": {
"description": _DATASET_DESCRIPTION,
"template": "{collection}_{dataset}",
"examples": _DATASET_EXAMPLES,
}
}
CONCEPTS = {
"tomo_angle": {
"description": "Projection angle in deg notation, two decimals (e.g. 180.00deg, -90.50deg)",
"regex": r"-?\d+(\.\d+)?deg",
"deserializer": _deserializer_angle,
"serializer": _serializer_angle,
"examples": ["180.00deg"],
},
"tomo_technique": {
"description": "Tomography technique (e.g. absorption, phase)",
"regex": ALPHA_NUMERIC_REGEX,
"examples": ["absorption", "phase"],
},
"tomo_type": {
"description": "Processing type/context (e.g. stitched, denoised, segmented)",
"regex": ALPHA_NUMERIC_REGEX,
"examples": ["stitched", "denoised", "segmented"],
},
"tomo_slicing_direction": {
"description": "Slicing direction in ESRF coordinates (xy, yz or xz)",
"regex": r"(xy|yz|xz)",
"deserializer": _deserializer_slicing_direction,
"serializer": _serializer_slicing_direction,
},
"tomo_slice_number": {
"description": "Slice index (0-based) formatted with 5 digits minimum (e.g. 00000, 00001)",
"regex": r"\d{5,}",
"deserializer": int,
"serializer": _serializer_slice_number,
},
"tomo_dtype": {
"description": "Output numeric data type / bit depth (e.g. 16Bit, 32Bit)",
"regex": ALPHA_NUMERIC_REGEX,
"examples": ["16Bit", "32Bit"],
},
"tomo_extension": {
"description": "File extension (e.g. hdf5, tiff, vol)",
"regex": ALPHA_NUMERIC_REGEX,
},
"tomo_application": {
"description": "Application/workflow name that produced the result (e.g. nabu)",
"regex": ALPHA_NUMERIC_REGEX,
},
"thumbnail_file_type": {
"description": "Thumbnail file type (e.g. png, jpg)",
"regex": ALPHA_NUMERIC_REGEX,
},
}
# The `tomo` schema (unlike `tomoproc`) roots the dataset results under the
# ESRF session directory, with `{collection}`/`{tomo_result_type}` shortcuts
# next to the dataset directory. This part is shared between all `tomo`
# schema versions; only the `esrf` schema version they extend differs.
TOMO_SESSION_TREE: TemplateTree = {
"{data_type=PROCESSED_DATA}": {
"{collection}": {
"{tomo_dataset}": RESULT_TREE,
"{tomo_result_type=SLICES}": {
named_directory("slices_link_collection_path"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}",
"slices_link_collection_file",
): None,
},
"{tomo_result_type=VOLUMES}": {
named_directory("volumes_link_collection_path"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}",
"volumes_link_collection_technique_path",
True,
("volumes_link_collection_file",),
): {
(
"{tomo_dataset}_{tomo_slice_number}.{tomo_extension}",
"volumes_link_collection_slice_file",
): None,
},
},
},
"{tomo_result_type=SLICES}": {
named_directory("slices_link_root_path"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}",
"slices_link_root_file",
): None,
},
"{tomo_result_type=VOLUMES}": {
named_directory("volumes_link_root_path"): None,
(
"{tomo_dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}",
"volumes_link_root_technique_path",
True,
("volumes_link_root_file",),
): {
(
"{tomo_dataset}_{tomo_slice_number}.{tomo_extension}",
"volumes_link_root_slice_file",
): None,
},
},
},
}
TOMO_ROOT_TREE: TemplateTree = {
"{data_root}": {
"{proposal}": {
"{beamline}": {
"{session_date}": TOMO_SESSION_TREE,
},
},
},
}
_STRING_TO_RESULT_TYPE = {
"SLICES": TomoResultType.SLICE,
"VOLUMES": TomoResultType.VOLUME,
}
def _deserializer_result_type(data_type: Union[str, TomoResultType]) -> TomoResultType:
return deserializer_enum_type(
data_type,
enum_type=TomoResultType,
string_to_enum=_STRING_TO_RESULT_TYPE,
)
def _serializer_result_type(data_type: Union[str, TomoResultType]) -> str:
return serializer_enum_type(
data_type,
enum_type=TomoResultType,
string_to_enum=_STRING_TO_RESULT_TYPE,
)
TOMO_CONCEPTS = {
**CONCEPTS,
"tomo_result_type": {
"description": "Tomography result type.",
"regex": "|".join(list(_STRING_TO_RESULT_TYPE)),
"deserializer": _deserializer_result_type,
"serializer": _serializer_result_type,
},
}
TOMO_DERIVED_CONCEPTS = DATASET_DERIVED_CONCEPT
def build_tomo_schema(
version: int, description: str, extends: BasePathSchema
) -> PathSchema:
"""Build a `tomo` schema for the given `esrf` schema version it extends."""
return PathSchema(
name="tomo",
version=version,
description=description,
concepts=TOMO_CONCEPTS,
derived_concepts=TOMO_DERIVED_CONCEPTS,
template_tree=TOMO_ROOT_TREE,
extends=[extends],
)
tomo_v1.py 0000664 0000000 0000000 00000011254 15244036076 0034235 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/tomo """
Tomography Processed Data Policy v1
Introduced on Feb 23, 2026.
The dataset directory name ``{collection}_{dataset}`` is also available as the
derived concept ``tomo_dataset``.
Path hierarchy::
{data_root}/{proposal}/{beamline}/{session_date}/ [TID]
└── {data_type=PROCESSED_DATA}/
├── {tomo_result_type=SLICES}/
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}
├── {tomo_result_type=VOLUMES}/
| └── {collection}_{dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}/
| └── {collection}_{dataset}_{tomo_slice_number}.{tomo_extension}
└── {collection}/ [esrf-pathlib]
├── {tomo_result_type=SLICES}/
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}
├── {tomo_result_type=VOLUMES}/
| └── {collection}_{dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}/
| └── {collection}_{dataset}_{tomo_slice_number}.{tomo_extension}
└── {collection}_{dataset}/ [esrf-pathlib]
├── references/
│ ├── {collection}_{dataset}_darks.hdf5
│ └── {collection}_{dataset}_flats.hdf5
├── projections/
│ ├── {collection}_{dataset}.nx
│ └── gallery/
│ ├── metadata.json
│ ├── {collection}_{dataset}_{tomo_angle}_large.{thumbnail_file_type}
│ └── {collection}_{dataset}_{tomo_angle}.{thumbnail_file_type}
├── slices/
│ ├── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}
│ └── gallery/
│ ├── metadata.json
│ ├── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}
├── volumes/
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}/
│ ├── {collection}_{dataset}_{tomo_slice_number}.{tomo_extension}
│ └── gallery/
│ ├── metadata.json
│ ├── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}
├── volumes_{tomo_type}/
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_type}_{tomo_dtype}_{tomo_extension}/
│ ├── {collection}_{dataset}_{tomo_slice_number}.{tomo_extension}
│ └── gallery/
│ ├── metadata.json
│ ├── {collection}_{dataset}_{tomo_technique}_{tomo_type}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_type}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}
├── logs/
│ └── {collection}_{dataset}_{tomo_application}.log
└── workflows/
└── gallery/
└── {collection}_{dataset}.json
"""
from ...schema.path import PathSchema
from ..esrf import esrf_v3
from . import tomo_utils
_SCHEMA = None
def get_schema() -> PathSchema:
global _SCHEMA
if _SCHEMA is None:
_SCHEMA = tomo_utils.build_tomo_schema(
version=1, description=__doc__, extends=esrf_v3.get_schema()
)
return _SCHEMA
tomo_v2.py 0000664 0000000 0000000 00000011352 15244036076 0034235 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/tomo """
Tomography Processed Data Policy v2
Introduced on Aug 24, 2026.
Derived from `esrf_v4` instead of `esrf_v3` (:mod:`tomo_v1`). The dataset
directory name ``{collection}_{dataset}`` is also available as the derived
concept ``tomo_dataset``.
Path hierarchy::
{data_root}/{proposal}/{beamline}/{session_date}/ [TID]
└── {data_type=PROCESSED_DATA}/
├── {tomo_result_type=SLICES}/
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}
├── {tomo_result_type=VOLUMES}/
| └── {collection}_{dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}/
| └── {collection}_{dataset}_{tomo_slice_number}.{tomo_extension}
└── {collection}/ [esrf-pathlib]
├── {tomo_result_type=SLICES}/
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}
├── {tomo_result_type=VOLUMES}/
| └── {collection}_{dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}/
| └── {collection}_{dataset}_{tomo_slice_number}.{tomo_extension}
└── {collection}_{dataset}/ [esrf-pathlib]
├── references/
│ ├── {collection}_{dataset}_darks.hdf5
│ └── {collection}_{dataset}_flats.hdf5
├── projections/
│ ├── {collection}_{dataset}.nx
│ └── gallery/
│ ├── metadata.json
│ ├── {collection}_{dataset}_{tomo_angle}_large.{thumbnail_file_type}
│ └── {collection}_{dataset}_{tomo_angle}.{thumbnail_file_type}
├── slices/
│ ├── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}
│ └── gallery/
│ ├── metadata.json
│ ├── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}
├── volumes/
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}/
│ ├── {collection}_{dataset}_{tomo_slice_number}.{tomo_extension}
│ └── gallery/
│ ├── metadata.json
│ ├── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}
├── volumes_{tomo_type}/
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_type}_{tomo_dtype}_{tomo_extension}/
│ ├── {collection}_{dataset}_{tomo_slice_number}.{tomo_extension}
│ └── gallery/
│ ├── metadata.json
│ ├── {collection}_{dataset}_{tomo_technique}_{tomo_type}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}
│ └── {collection}_{dataset}_{tomo_technique}_{tomo_type}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}
├── logs/
│ └── {collection}_{dataset}_{tomo_application}.log
└── workflows/
└── gallery/
└── {collection}_{dataset}.json
"""
from ...schema.path import PathSchema
from ..esrf import esrf_v4
from . import tomo_utils
_SCHEMA = None
def get_schema() -> PathSchema:
global _SCHEMA
if _SCHEMA is None:
_SCHEMA = tomo_utils.build_tomo_schema(
version=2, description=__doc__, extends=esrf_v4.get_schema()
)
return _SCHEMA
tomoproc_v1.py 0000664 0000000 0000000 00000010040 15244036076 0035111 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/tomo """
Local Tomography Processed Data Policy v1
Introduced on Jul 17, 2026.
Local version of the *tomo* schema: it describes the tomography processed data
of a single dataset rooted at an arbitrary directory ``{data_root}`` instead of
the dataset directory inside an ESRF session directory
.. code::
{data_root}/{proposal}/{beamline}/{session_date}/{data_type=PROCESSED_DATA}/{collection}/{collection}_{dataset}
Use this schema when tomography results are not located on the ESRF shared
file system. The ``{collection}`` and ``{dataset}`` concepts are not part of this
schema: the dataset directory name is the ``{tomo_dataset}`` concept because the
collection/dataset split of a local directory name is unknown.
Path hierarchy::
{data_root}/ [esrf-pathlib, matches any path]
├── references/
│ ├── {tomo_dataset}_darks.hdf5
│ └── {tomo_dataset}_flats.hdf5
├── projections/
│ ├── {tomo_dataset}.nx
│ └── gallery/
│ ├── metadata.json
│ ├── {tomo_dataset}_{tomo_angle}_large.{thumbnail_file_type}
│ └── {tomo_dataset}_{tomo_angle}.{thumbnail_file_type}
├── slices/
│ ├── {tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}
│ └── gallery/
│ ├── metadata.json
│ ├── {tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}
│ └── {tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}
├── volumes/
│ └── {tomo_dataset}_{tomo_technique}_{tomo_dtype}_{tomo_extension}/
│ ├── {tomo_dataset}_{tomo_slice_number}.{tomo_extension}
│ └── gallery/
│ ├── metadata.json
│ ├── {tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}
│ └── {tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}
├── volumes_{tomo_type}/
│ └── {tomo_dataset}_{tomo_technique}_{tomo_type}_{tomo_dtype}_{tomo_extension}/
│ ├── {tomo_dataset}_{tomo_slice_number}.{tomo_extension}
│ └── gallery/
│ ├── metadata.json
│ ├── {tomo_dataset}_{tomo_technique}_{tomo_type}_{tomo_slicing_direction}_{tomo_slice_number}_large.{thumbnail_file_type}
│ └── {tomo_dataset}_{tomo_technique}_{tomo_type}_{tomo_slicing_direction}_{tomo_slice_number}.{thumbnail_file_type}
├── logs/
│ └── {tomo_dataset}_{tomo_application}.log
└── workflows/
└── gallery/
└── {tomo_dataset}.json
"""
from ...schema.path import PathSchema
from ...typing import TemplateTree
from ..unknown import unknown_v1
from ..utils import as_path_string
from ..utils import as_pathlib_path
from ..utils import named_directory
from . import tomo_utils
_ROOT_TREE: TemplateTree = {
"{data_root}": {
named_directory("dataset_path"): None,
**tomo_utils.RESULT_TREE,
},
}
_CONCEPTS = {
"data_root": {
"description": "Root directory",
"regex": r".*",
"deserializer": as_pathlib_path,
"serializer": as_path_string,
},
**tomo_utils.DATASET_CONCEPT,
**tomo_utils.CONCEPTS,
}
_SCHEMA = None
def get_schema() -> PathSchema:
global _SCHEMA
if _SCHEMA is None:
_SCHEMA = PathSchema(
name="tomoproc",
version=1,
description=__doc__,
concepts=_CONCEPTS,
template_tree=_ROOT_TREE,
extends=[unknown_v1.get_schema()],
)
return _SCHEMA
types.py 0000664 0000000 0000000 00000000675 15244036076 0033044 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions import datetime
import pathlib
from enum import Enum
from typing import Union
class DataType(str, Enum):
RAW = "raw"
PROCESSED = "processed"
NOBACKUP = "nobackup"
AUXILIARY = "auxiliary"
SCRIPT = "script"
GALLERY = "gallery"
NOTE = "note"
class TomoResultType(str, Enum):
SLICE = "slice"
VOLUME = "volume"
ConceptValueType = Union[
str, pathlib.Path, datetime.date, DataType, TomoResultType, None
]
unknown/ 0000775 0000000 0000000 00000000000 15244036076 0033015 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions __init__.py 0000664 0000000 0000000 00000000133 15244036076 0035123 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/unknown """Placeholder schema definitions for unrecognized paths (``unknown`` schema versions)."""
unknown_v1.py 0000664 0000000 0000000 00000002024 15244036076 0035472 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions/unknown """
Unknown Path Schema v1
Represents a placeholder for paths or path segments that are not yet known.
Path hierarchy::
/
└── {unknown_part}
"""
from ... import constants
from ...schema.path import PathSchema
from ...typing import TemplateTree
from ..utils import as_path_string
from ..utils import as_pathlib_path
_ROOT_TREE: TemplateTree = {
(f"{{{constants.UNKNOWN_PART_NAME}}}", "unknown_path"): None,
}
_CONCEPTS = {
constants.UNKNOWN_PART_NAME: {
"description": "Unknown path or path segment (placeholder for unmapped directories or files)",
"regex": r".+",
"score": 0,
"deserializer": as_pathlib_path,
"serializer": as_path_string,
}
}
_SCHEMA = None
def get_schema() -> PathSchema:
global _SCHEMA
if _SCHEMA is None:
_SCHEMA = PathSchema(
name=constants.UNKNOWN_SCHEMA_NAME,
version=1,
description=__doc__,
concepts=_CONCEPTS,
template_tree=_ROOT_TREE,
)
return _SCHEMA
utils.py 0000664 0000000 0000000 00000002371 15244036076 0033033 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/definitions """General-purpose helpers shared by schema definitions."""
import os
import pathlib
from enum import Enum
from typing import Dict
from typing import Tuple
from typing import Type
from typing import Union
def as_pathlib_path(value: Union[str, pathlib.Path]) -> pathlib.Path:
"""Path concept value as a `pathlib` path."""
return pathlib.Path(value)
def as_path_string(value: Union[str, pathlib.Path]) -> str:
"""Path concept value as a path string for the current OS."""
return str(value).replace("/", os.sep)
def named_directory(name: str) -> Tuple[str, str, bool]:
"""Use for a named directory with optionally unknown content."""
return ("{unknown_part}", name, False)
def deserializer_enum_type(
data_type: Union[str, Enum], enum_type: Type[Enum], string_to_enum: Dict[str, Enum]
) -> Enum:
if isinstance(data_type, str) and data_type in string_to_enum:
return string_to_enum[data_type]
return enum_type(data_type)
def serializer_enum_type(
data_type: Union[str, Enum], enum_type: Type[Enum], string_to_enum: Dict[str, Enum]
) -> str:
data_type = deserializer_enum_type(data_type, enum_type, string_to_enum)
enum_to_string = {v: k for k, v in string_to_enum.items()}
return enum_to_string[data_type]
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/errors.py 0000664 0000000 0000000 00000002600 15244036076 0030746 0 ustar 00root root 0000000 0000000 class EsrfPathlibError(Exception):
pass
class FieldNameError(EsrfPathlibError):
"""Field (concept or template) cannot be called this way."""
class ForbiddenFieldName(FieldNameError):
"""Field (concept or template) has a reserved name."""
class FieldNameCollision(FieldNameError):
"""Field (concept or template) already exists."""
class PathSchemaCollision(EsrfPathlibError):
"""Schema already exists."""
class UnknownPathConcept(EsrfPathlibError):
"""Path concept is not known."""
class PathConceptValueError(EsrfPathlibError):
"""Path concept is known but its value is not known."""
class PathConceptMatchError(PathConceptValueError):
"""Path concept is known but its value is wrong."""
class PathConceptWithoutValue(PathConceptValueError):
"""Path concept is known but its value is not known."""
class UnknownPathTemplate(EsrfPathlibError):
"""Path template is not known."""
class UnknownField(EsrfPathlibError):
"""Path field (concept or template) is not known."""
class UnknownPathSchema(EsrfPathlibError):
"""Schema is not known."""
class PathTemplateMatchError(EsrfPathlibError):
"""Path does not match a path template."""
class PathSchemaMatchError(PathTemplateMatchError):
"""Path does not match the schema. It did not match any of the enabled path templates."""
class SchemaAttributeError(AttributeError):
pass
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/fields/ 0000775 0000000 0000000 00000000000 15244036076 0030330 5 ustar 00root root 0000000 0000000 __init__.py 0000664 0000000 0000000 00000000412 15244036076 0032357 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/fields """Fields are components of an ESRF filesystem path schema.
- **Path concepts**: typed components extracted from paths.
- **Derived concepts**: computed from one or more path concepts.
- **Path templates**: patterns of concepts that describe a filesystem path.
"""
base.py 0000664 0000000 0000000 00000007221 15244036076 0031537 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/fields from enum import Enum
from typing import Any
from typing import Callable
from typing import List
from typing import Optional
from typing import Tuple
from typing import Type
from typing import Union
from typing import get_type_hints
from ..identifier import SchemaIdentifier
class Field:
def __init__(
self,
name: str,
description: str,
schema_identifier: SchemaIdentifier,
value_generator: Union[Callable[..., Any], Type[Any]],
value_serializer: Union[Callable[..., Any], Type[Any], None] = None,
examples: Optional[List[str]] = None,
deprecated_names: Optional[List[str]] = None,
) -> None:
self._name = name
self._description = description
self._examples = examples
self._schema_identifier = schema_identifier
self._value_generator = value_generator
self._value_serializer = value_serializer
self._deprecated_names = tuple(deprecated_names or ())
def __repr__(self) -> str:
return f"<{type(self).__name__} {self._name!r} schema={str(self._schema_identifier)!r}>"
@property
def name(self) -> str:
return self._name
@property
def deprecated_names(self) -> Tuple[str, ...]:
"""Former names of this field that are still accepted (with a
``DeprecationWarning``) wherever the field's name is used."""
return self._deprecated_names
@property
def description(self) -> str:
return self._description
@property
def docstring(self) -> str:
description = self._description or "Path concept"
description = _ensure_sentence(description)
category, values = self.value_info()
if category is None:
return description
return f"{description}\n\n{category}: ``{', '.join(values)}``"
@property
def schema_identifier(self) -> SchemaIdentifier:
return self._schema_identifier
def annotation(self) -> Any:
"""More specific type annotation of the return value of ``deserialize``."""
return _get_return_annotation(self._value_generator)
def value_info(self) -> Tuple[Optional[str], Optional[str]]:
enum_values = self._enum_values()
if enum_values:
return "Enumeration", enum_values
if self._examples:
return "Examples", list(map(repr, self._examples))
return None, None
def _enum_values(self) -> List[str]:
enum_type = None
if isinstance(self._value_generator, Enum):
enum_type = self._value_generator
else:
return_type = self.annotation()
if isinstance(return_type, type) and issubclass(return_type, Enum):
enum_type = return_type
if enum_type is None or self._value_serializer is None:
return []
values = []
for field in enum_type:
try:
value = repr(self._value_serializer(field))
except Exception: # noqa: BLE001, S112 -- best-effort doc generation
continue
values.append(value)
return values
def _get_return_annotation(obj: Union[Callable[..., Any], Type[Any]]) -> Any:
"""Extract the return annotation of a function or a type."""
if isinstance(obj, type):
return obj
if callable(obj):
hints = get_type_hints(obj)
return hints.get("return", None)
raise TypeError(f"{obj!r} must be a type or callable")
def _ensure_sentence(text: str) -> str:
text = text.strip()
if not text:
return ""
if text[0].isalpha():
text = text[0].upper() + text[1:]
if text[-1] not in ".!?":
text += "."
return text
concept.py 0000664 0000000 0000000 00000010670 15244036076 0032262 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/fields import re
from typing import Any
from typing import Callable
from typing import List
from typing import Optional
from typing import Type
from typing import Union
from .. import errors
from ..definitions.types import ConceptValueType
from ..identifier import SchemaIdentifier
from .base import Field
class PathConcept(Field):
"""Concept that is part of an ESRF path. For example beamline, proposal, collection, etc."""
def __init__(
self,
name: str,
description: str,
schema_identifier: SchemaIdentifier,
regex: str,
deserializer: Union[Callable[[str], ConceptValueType], Type[Any]] = str,
serializer: Union[Callable[[ConceptValueType], str], Type[Any]] = str,
score: int = 1,
default_value: Any = None,
examples: Optional[List[str]] = None,
deprecated_names: Optional[List[str]] = None,
):
"""
:param name: name used in path template placeholders
:param description: for documentation
:param schema_identifier: schema that defines the concept (could be used in other schemas)
:param regex: value use in path template placeholders
:param deserializer: convert the string value used in the path to a python object (default: ``str``)
:param serializer: convert a python object to the string value used in the path (default: ``str``)
:param score: score used when calculating a matching score between a path template and a path string or concept value dict
:param default_value: used when generating a path from a concept value dict
:param deprecated_names: former names of this concept that are still accepted (with a ``DeprecationWarning``)
"""
super().__init__(
name=name,
description=description,
schema_identifier=schema_identifier,
value_generator=deserializer,
value_serializer=serializer,
examples=examples,
deprecated_names=deprecated_names,
)
self._description = description
self._regex = regex
self._deserializer = deserializer
self._serializer = serializer
self._score = score
self._default_value = default_value
self._spans_path_separator = bool(re.fullmatch(regex, "a/b"))
@property
def regex(self) -> str:
return self._regex
@property
def score(self) -> int:
return self._score
@property
def spans_path_separator(self) -> bool:
"""The value can span several path segments (`{data_root}`,
`{unknown_part}`), so it does not identify the segments it matches."""
return self._spans_path_separator
def is_same_definition(self, other: "PathConcept") -> bool:
"""Concepts with the same definition can be defined by different schemas
(only the schema identifier, and possibly the default value used when
rendering, differ)."""
return (
self._name == other._name
and self._description == other._description
and self._regex == other._regex
and self._deserializer is other._deserializer
and self._serializer is other._serializer
and self._score == other._score
and self._examples == other._examples
)
def deserialize(self, string_value: Optional[str]) -> ConceptValueType:
"""String value used in the path to Python object.
:raises PathConceptMatchError:
"""
if string_value is None:
return None
try:
return self._deserializer(string_value)
except Exception as ex:
raise errors.PathConceptMatchError(
f"{self.name!r} deserialization failed for {string_value!r}"
) from ex
def serialize(
self, py_value: ConceptValueType, raise_on_missing: bool = True
) -> str:
"""Python object to string value used in the path.
When missing and ``raise_on_missing=False`` the value ``"*"`` is returned.
:raises PathConceptValueError:
"""
if py_value is None:
if raise_on_missing:
raise errors.PathConceptWithoutValue(f"{self.name!r} is not defined")
return "*"
try:
return self._serializer(py_value)
except Exception as ex:
raise errors.PathConceptMatchError(
f"{self.name!r} serialization failed for {py_value}"
) from ex
derived.py 0000664 0000000 0000000 00000006675 15244036076 0032263 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/fields from typing import Callable
from typing import Dict
from typing import List
from typing import Optional
from typing import Tuple
from .. import errors
from ..constants import PLACEHOLDER_REGEX
from ..definitions.types import ConceptValueType
from ..identifier import SchemaIdentifier
from .base import Field
from .concept import PathConcept
class DerivedConcept(Field):
"""A concept not parsed from the path, but computed from other concepts."""
def __init__(
self,
name: str,
description: str,
schema_identifier: SchemaIdentifier,
derive_func: Callable[[Dict[str, ConceptValueType]], ConceptValueType],
derived_from: List[str],
template: Optional[str] = None,
examples: Optional[List[str]] = None,
deprecated_names: Optional[List[str]] = None,
):
"""
:param template: path template fragment the concept is a shorthand for
(see :func:`template_derive_func`). Placeholders of the
concept are replaced by the fragment in path templates.
:param deprecated_names: former names of this concept that are still
accepted (with a ``DeprecationWarning``)
"""
super().__init__(
name=name,
description=description,
schema_identifier=schema_identifier,
value_generator=derive_func,
examples=examples,
deprecated_names=deprecated_names,
)
self._derive_func = derive_func
self._derived_from = derived_from
self._template = template
@property
def template(self) -> Optional[str]:
return self._template
@property
def docstring(self) -> str:
if self._template is None:
return super().docstring
return f"{super().docstring}\n\nPath template: ``{self._template}``"
def derive(
self, concept_values: Dict[str, ConceptValueType], raise_on_missing: bool = True
) -> ConceptValueType:
"""
:raises PathConceptValueError:
"""
try:
values = {}
for cname in self._derived_from:
if cname not in concept_values:
raise errors.PathConceptWithoutValue(cname)
values[cname] = concept_values[cname]
return self._derive_func(values)
except Exception as ex:
if raise_on_missing:
raise errors.PathConceptMatchError(self.name) from ex
return "*"
def template_derive_func(
template: str, concepts: Dict[str, PathConcept]
) -> Tuple[Callable[[Dict[str, ConceptValueType]], str], List[str]]:
"""Derive function and dependencies of a concept that is a shorthand for the
path template fragment ``template`` (for example ``"{collection}_{dataset}"``).
:raises UnknownPathConcept:
"""
derived_from: List[str] = []
for cname, _ in PLACEHOLDER_REGEX.findall(template):
if cname not in concepts:
raise errors.UnknownPathConcept(
f"Placeholder {cname!r} of template {template!r} is not a path concept"
)
if cname not in derived_from:
derived_from.append(cname)
def derive_func(values: Dict[str, ConceptValueType]) -> str:
return PLACEHOLDER_REGEX.sub(
lambda match: concepts[match.group(1)].serialize(values[match.group(1)]),
template,
)
return derive_func, derived_from
path.py 0000664 0000000 0000000 00000035126 15244036076 0031566 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/fields import os
import pathlib
import re
from typing import Dict
from typing import List
from typing import Optional
from typing import Tuple
from typing import Union
from .. import errors
from ..constants import PLACEHOLDER_REGEX
from ..definitions.types import ConceptValueType
from ..identifier import SchemaIdentifier
from .base import Field
from .concept import PathConcept
from .tree import SegmentTemplate
class PathTemplate(Field):
"""Class for matching, parsing, and rendering paths from segment templates."""
def __init__(
self,
name: str,
schema_identifier: SchemaIdentifier,
segments: List[SegmentTemplate],
concepts: Dict[str, PathConcept],
deprecated_names: Optional[List[str]] = None,
):
"""
:raises UnknownPathConcept:
"""
# String representation of the templates
template_string = ""
suffix = ""
for segment in segments:
if segment.mandatory:
if template_string:
template_string = f"{template_string}/{segment.template}"
else:
template_string = segment.template
else:
if template_string:
template_string = f"{template_string}[/{segment.template}"
else:
template_string = f"[{segment.template}"
suffix = f"{suffix}]"
template_string += suffix
# Count the required levels
num_required_segments = 0
for segment in segments:
if not segment.mandatory:
break
num_required_segments += 1
# Extract the used concepts and calculate template score
used_concepts = {}
required_concepts = []
required = True
concept_score = 0 # total concept score in required segments, multiple occurrences do NOT count
num_placeholders = 0 # total number of placeholders in all required segments, multiple occurrences count
for segment in segments:
if not segment.mandatory:
required = False
for placeholder_name, _ in PLACEHOLDER_REGEX.findall(segment.template):
if placeholder_name not in concepts:
raise errors.UnknownPathConcept(
f"Placeholder {placeholder_name!r} is not a concept of schema {str(schema_identifier)!r}"
)
concept = concepts[placeholder_name]
if required:
num_placeholders += 1
if placeholder_name not in required_concepts:
required_concepts.append(placeholder_name)
if concept.name not in used_concepts:
concept_score += concept.score
used_concepts[placeholder_name] = concept
super().__init__(
name=name,
description=template_string,
schema_identifier=schema_identifier,
value_generator=str,
deprecated_names=deprecated_names,
)
self._concepts: Dict[str, PathConcept] = used_concepts
self._segments: List[SegmentTemplate] = segments
self._regex: re.Pattern = self._compile_regex(segments)
self._concept_score: int = concept_score
self._num_placeholders: int = num_placeholders
self._num_required_segments: int = num_required_segments
self._required_concepts: List[str] = required_concepts
self._identifies_last_segment: bool = self._last_segment_is_identified(
segments[:num_required_segments], used_concepts
)
@staticmethod
def _last_segment_is_identified(
required_segments: List[SegmentTemplate], concepts: Dict[str, PathConcept]
) -> bool:
"""The last required segment identifies the path it matches when it is
not swallowed by concepts that span path separators (`{data_root}`,
`{unknown_part}`).
"""
if not required_segments:
return False
template = required_segments[-1].template
placeholders = PLACEHOLDER_REGEX.findall(template)
if PLACEHOLDER_REGEX.sub("", template):
return True # the segment contains a literal
return not all(
concepts[cname].spans_path_separator for cname, _ in placeholders
)
def __repr__(self) -> str:
return f"<{type(self).__name__} path={self._name!r} schema={str(self._schema_identifier)!r} template={self._description!r}>"
@property
def docstring(self) -> str:
return f"Filesystem path template\n\n``{self._description}``"
def parse(self, path: Union[str, pathlib.Path]) -> Dict[str, ConceptValueType]:
"""Match and convert a path to a dictionary of path concept values.
:raises PathTemplateMatchError:
"""
return self._parse(str(path))
def parse_with_score(
self, path: str
) -> Tuple[Optional[Dict[str, ConceptValueType]], Tuple[int, ...]]:
"""Returns an empty tuple if it does not match.
If it does match it returns the concept values and the match score.
The score is a tuple compared lexicographically (higher wins) with
1. whether the template identifies the path itself: the last path
segment is matched by a required segment which is not swallowed by
concepts that span path separators. A template of one schema which
identifies the path wins from a template of another schema which
only identifies parent directories,
2. the total score of the unique concepts in required segments
(multiple occurrences of a concept do not count),
3. the total number of placeholders in required segments (multiple
occurrences of a concept do count),
4. the number of required segments: breaks the tie between templates
matching the same concepts by preferring path segments matched by
literals over segments swallowed by concepts or optional segments.
"""
try:
concept_values = self._parse(path)
except errors.PathTemplateMatchError:
return None, ()
identifies_path = self._identifies_last_segment and not self.contains_optionals(
concept_values
)
score = (
int(identifies_path),
self._concept_score,
self._num_placeholders,
self._num_required_segments,
)
return concept_values, score
def _parse(self, path: str) -> Dict[str, ConceptValueType]:
"""
:raises PathTemplateMatchError:
"""
norm_path = path.replace(os.sep, "/")
m = self._regex.match(norm_path)
if not m:
raise errors.PathTemplateMatchError(f"Path {path!r} does not match {self}")
is_absolute = bool(re.match(r"^(?:[A-Za-z]:)?/", norm_path))
result = {}
for cname, cvalue in m.groupdict().items():
# A leading concept spanning path separators (e.g. `{data_root}`)
# that is rendered as the filesystem root is indistinguishable
# from an empty value once joined with the segment after it (the
# separators collapse into one). Recover the root in that case.
if is_absolute and cvalue == "" and m.start(cname) == 0:
cvalue = "/"
result[cname] = self._concepts[cname].deserialize(cvalue)
return result
def render(
self,
concept_values: Dict[str, ConceptValueType],
raise_on_missing: bool = True,
raise_on_mismatch: bool = True,
) -> str:
"""Convert a dictionary of path concept values to a path.
:raises PathConceptValueError:
"""
parts, _ = self._match(
concept_values,
raise_on_missing=raise_on_missing,
raise_on_mismatch=raise_on_mismatch,
)
return os.path.join(*parts)
def render_score(
self, concept_values: Dict[str, ConceptValueType]
) -> Tuple[int, ...]:
"""Returns an empty tuple if it does not match.
If it does match it returns the match score.
"""
try:
_, score = self._match(concept_values)
except errors.PathConceptValueError:
return ()
return score
def _match(
self,
concept_values: Dict[str, ConceptValueType],
raise_on_missing: bool = True,
raise_on_mismatch: bool = True,
) -> Tuple[List[str], Tuple[int, ...]]:
"""Render the segments from the concept values and calculate the
match score.
The score is a tuple compared lexicographically (higher wins) with
1. the total score of the unique concepts in required segments
(multiple occurrences of a concept do not count),
2. the total number of placeholders in required segments (multiple
occurrences of a concept do count),
3. minus the number of provided concepts not used by the template,
4. minus the number of required segments: breaks the tie between
templates rendered from the same concepts by preferring the
template fully described by the concepts over templates with
additional literal segments.
:raises PathConceptValueError:
"""
used = set()
def serialize_concept(match: re.Match):
concept, cvalue = self._serialize_concept(
match,
concept_values=concept_values,
raise_on_missing=raise_on_missing or not segment.mandatory,
raise_on_mismatch=raise_on_mismatch,
)
used.add(concept.name)
return cvalue
parts = []
for segment in self._segments:
try:
rendered = PLACEHOLDER_REGEX.sub(serialize_concept, segment.template)
except errors.PathConceptValueError:
if segment.mandatory:
raise
continue
parts.append(rendered)
num_missing = len(set(concept_values) - used)
score = (
self._concept_score,
self._num_placeholders,
-num_missing,
-self._num_required_segments,
)
return parts, score
def _compile_regex(self, segments: List[SegmentTemplate]) -> re.Pattern:
"""
:raises UnknownPathConcept:
"""
used_groups = []
def segment_as_regex(match):
return self._segment_as_regex(match, used_groups)
regex_parts = []
for segment in segments:
seg_regex = PLACEHOLDER_REGEX.sub(segment_as_regex, segment.template)
if regex_parts:
# Not the first segment
if segment.mandatory:
seg_regex = f"/{seg_regex}"
else:
seg_regex = f"(?:/{seg_regex})?"
else:
# First segment
if not segment.mandatory:
seg_regex = f"(?:{seg_regex})?"
regex_parts.append(seg_regex)
pattern = "".join(regex_parts)
return re.compile(f"^{pattern}$")
def _segment_as_regex(self, match: re.Match, used_groups: List[str]) -> str:
"""
:raises UnknownPathConcept:
"""
cname, literal = match.groups()
if cname not in self._concepts:
raise errors.UnknownPathConcept(cname)
if cname in used_groups:
return f"(?P={cname})"
used_groups.append(cname)
concept = self._concepts[cname]
if literal:
return f"(?P<{cname}>{re.escape(literal)})"
else:
return f"(?P<{cname}>{concept.regex})"
def _serialize_concept(
self,
match: re.Match,
concept_values: Dict[str, ConceptValueType],
raise_on_missing: bool = True,
raise_on_mismatch: bool = True,
) -> Tuple[PathConcept, ConceptValueType]:
"""
:raises PathConceptValueError:
"""
cname, literal = match.groups()
concept = self._concepts[cname]
has_literal = literal is not None
provided = concept_values.get(cname)
if has_literal:
literal = concept.serialize(literal, raise_on_missing=raise_on_missing)
if provided is None:
provided = literal
has_literal = False
result = concept.serialize(provided, raise_on_missing=raise_on_missing)
if not has_literal:
return concept, result
literal = concept.serialize(literal, raise_on_missing=raise_on_missing)
if provided is not None and result != literal and raise_on_mismatch:
raise errors.PathConceptMatchError(
f"{self.name!r}={result!r} instead of {literal!r}"
)
return concept, literal
def contains_optionals(self, concept_values: Dict[str, ConceptValueType]) -> bool:
return any(
cname not in self._required_concepts and cvalue is not None
for cname, cvalue in concept_values.items()
)
def remove_optionals(
self, concept_values: Dict[str, ConceptValueType]
) -> Dict[str, ConceptValueType]:
names = self._required_concepts or self._concepts
return {
cname: cvalue
for cname, cvalue in concept_values.items()
if cname in names and cvalue is not None
}
@property
def concept_names(self) -> List[str]:
return list(self._concepts)
@property
def required_concept_names(self) -> List[str]:
return self._required_concepts
def strip_mount_point(
self, concept_values: Dict[str, ConceptValueType]
) -> Dict[str, ConceptValueType]:
"""
Strip all segments from the root concept before a specific marker segment.
"""
concept_values = dict(concept_values)
if not self._segments or not self._segments[0].symbolic_root_segment:
return concept_values
root_concept_name = self._segments[0].template.strip("{}")
if root_concept_name not in concept_values:
return concept_values
concept_values[root_concept_name] = _strip_before_segment(
concept_values[root_concept_name],
segment=self._segments[0].symbolic_root_segment,
)
return concept_values
def _strip_before_segment(
path: pathlib.Path, segment: Optional[str] = None
) -> pathlib.Path:
if segment is None:
return path
parts = path.parts
try:
idx = parts.index(segment)
except ValueError:
return path
return pathlib.Path(path.anchor or os.sep, *parts[idx:])
tree.py 0000664 0000000 0000000 00000011520 15244036076 0031561 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/fields import re
from typing import Dict
from typing import List
from typing import NamedTuple
from typing import Optional
from typing import Tuple
from ..constants import PLACEHOLDER_REGEX
from ..typing import TemplateTree
class _ValidatedRawSegment(NamedTuple):
template: str
path_name: Optional[str] = None
mandatory: bool = True
deprecated_names: Tuple[str, ...] = ()
class SegmentTemplate(NamedTuple):
template: str
mandatory: bool
symbolic_root_segment: Optional[str] = None
class PathTemplateArguments(NamedTuple):
name: str
segments: List[SegmentTemplate]
deprecated_names: Tuple[str, ...] = ()
def parse_template_tree(
template_tree: TemplateTree,
symbolic_root_segment: Optional[str] = None,
_parent_segments: Optional[List[PathTemplateArguments]] = None,
) -> List[PathTemplateArguments]:
"""Convert a nested dictionary with :class:`RawSegmentType` keys to a
flat list of :class:`PathTemplate` arguments.
"""
if _parent_segments is None:
_parent_segments = []
templates: List[PathTemplateArguments] = []
for key, subtree in template_tree.items():
if isinstance(key, str):
key = (key,)
segment = _ValidatedRawSegment(*key)
current_segments = _parent_segments + [
SegmentTemplate(
template=segment.template,
mandatory=segment.mandatory,
symbolic_root_segment=symbolic_root_segment,
)
]
symbolic_root_segment = None
if segment.path_name:
templates.append(
PathTemplateArguments(
name=segment.path_name,
segments=current_segments,
deprecated_names=segment.deprecated_names,
)
)
if subtree is not None:
templates.extend(
parse_template_tree(subtree, _parent_segments=current_segments)
)
return templates
def expand_placeholders(
segments: List[SegmentTemplate], concept_templates: Dict[str, str]
) -> List[SegmentTemplate]:
"""Replace the placeholder of each concept in ``concept_templates`` by its
template fragment. Fragments only contain placeholders of path concepts so a
single pass is enough.
"""
if not concept_templates:
return segments
def expand(match: re.Match) -> str:
return concept_templates.get(match.group(1), match.group(0))
return [
segment._replace(template=PLACEHOLDER_REGEX.sub(expand, segment.template))
for segment in segments
]
def tree_as_strings(segments: List[List[SegmentTemplate]]) -> List[str]:
"""Convert several segment lists to a stringified representation of a directory structure.
For example
.. code::
{data_root}/{proposal}/{beamline}/{session_date}/
├── {proposal}_{beamline}.h5
└── {collection}/
├── {proposal}_{collection}.h5
└── {dataset}/
└── {collection}_{dataset}.h5
"""
tree: Dict[str, dict] = {}
for seg_path in segments:
_insert_path(tree, seg_path)
# Collapse common root prefix only if not a leaf
collapsed = []
node = tree
while len(node) == 1:
key, subnode = next(iter(node.items()))
if not subnode: # stop collapsing at a leaf
break
collapsed.append(key)
node = subnode
collapsed_node = "/".join(collapsed) + "/" if collapsed else "/"
if node:
return [collapsed_node] + _render_tree(node)
return [collapsed_node]
_BRANCH = " ├──"
_LAST_BRANCH = " └──"
_PREFIX = " │ "
_LAST_PREFIX = " "
_OPTIONAL = " [*]"
def _insert_path(tree: Dict[str, dict], path: List[SegmentTemplate]):
"""Recursively insert a segment path into the nested tree dict."""
if not path:
return
current_segment, *sub_path = path
label = current_segment.template + (
_OPTIONAL if not current_segment.mandatory else ""
)
subtree = tree.setdefault(label, {})
_insert_path(subtree, sub_path)
def _render_tree(node: Dict[str, dict], prefix: str = "") -> List[str]:
"""Render a nested dict tree into pretty tree lines."""
lines = []
keys = list(node.keys())
for i, key in enumerate(keys):
is_last = i == len(keys) - 1
branch = _LAST_BRANCH if is_last else _BRANCH
is_dir = bool(node[key])
optional = key.endswith(_OPTIONAL)
clean_key = key[: -len(_OPTIONAL)] if optional else key
slash = "/" if is_dir else ""
opt_marker = _OPTIONAL if optional else ""
lines.append(f"{prefix}{branch} {clean_key}{slash}{opt_marker}")
if is_dir:
next_prefix = prefix + (_LAST_PREFIX if is_last else _PREFIX)
lines.extend(_render_tree(node[key], next_prefix))
return lines
identifier.py 0000664 0000000 0000000 00000003206 15244036076 0031500 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas from collections.abc import Hashable
from typing import Union
class SchemaIdentifier(Hashable):
"""Identifier of a single schema or the merger of several schemas."""
def __init__(self, **schema_versions: int) -> None:
if not schema_versions:
raise ValueError("At least one schema must be provided")
self._schema_versions = schema_versions
names = list(schema_versions)
versions = list(schema_versions.values())
full_names = [
f"{sname}_v{sversion}" for sname, sversion in schema_versions.items()
]
if len(names) == 1:
self._name = names[0]
self._version = versions[0]
self._full_name = full_names[0]
else:
self._name = "+".join(names)
self._version = 0
self._full_name = "+".join(full_names)
self._names = names
self._full_names = full_names
self._hash = hash(tuple(sorted(schema_versions.items())))
def __hash__(self) -> int:
return self._hash
def __eq__(self, value: object) -> bool:
if isinstance(value, (str, SchemaIdentifier)):
return str(self) == str(value)
return super().__eq__(value)
@property
def name(self) -> str:
return self._name
@property
def version(self) -> int:
return self._version
def __repr__(self) -> str:
return f"<{type(self).__name__} {self._full_name!r}>"
def __str__(self) -> str:
return self._full_name
def has_identifier(self, identifier: Union[str, "SchemaIdentifier"]) -> bool:
return str(identifier) in self._full_names
registry.py 0000664 0000000 0000000 00000065266 15244036076 0031244 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas import logging
import os
import warnings
from pathlib import Path
from typing import Any
from typing import Dict
from typing import Iterator
from typing import List
from typing import Optional
from typing import Tuple
from typing import Union
from . import constants
from . import errors
from .definitions.load import default_schema_version
from .definitions.load import fallback_schema_versions
from .definitions.load import get_schema
from .definitions.types import ConceptValueType
from .fields.concept import PathConcept
from .fields.derived import DerivedConcept
from .fields.path import PathTemplate
from .identifier import SchemaIdentifier
from .schema.merged import MergedPathSchema
from .schema.path import BasePathSchema
from .schema.path import PathSchema
_logger = logging.getLogger(__name__)
def _warn_deprecated_name(kind: str, old_name: str, canonical_name: str) -> None:
warnings.warn(
f"{kind} {old_name!r} is deprecated, use {canonical_name!r} instead.",
DeprecationWarning,
stacklevel=3,
)
class PathSchemaRegistry:
"""Use one or more path schemas to find the best matching *path template* for a path
string or a dictionary of *path concepts*. With the template path strings are *parsed*
to a dictionary of concept values (wrapped by a ``ParsedPath`` class). Inversally
a dictionary of concept values can be *rendered* as a path string.
"""
def __init__(self, schemas: List[BasePathSchema]):
"""The order of the schemas is the order of field fallback resolution."""
if not schemas:
raise ValueError("At least one schema is required")
self._schemas = schemas
self._name = " > ".join([str(schema.identifier) for schema in schemas])
_logger.debug("Created registry %r", self.name)
@property
def name(self) -> str:
return self._name
def __info__(self) -> str:
lines = [f"PathSchemaRegistry: {self.name}"]
for _schema in self._schemas:
lines.extend(_schema.__info__().split("\n"))
return "\n ".join(lines)
def __repr__(self) -> str:
return f"<{type(self).__name__} {self.name}>"
def description(self, schema_identifier: Optional[SchemaIdentifier] = None) -> str:
schema = self._get_schema(schema_identifier=schema_identifier)
return schema.description
def concept_names(
self, schema_identifier: Optional[SchemaIdentifier] = None
) -> List[str]:
schema = self._get_schema(schema_identifier=schema_identifier)
return list(schema.concepts)
def derived_concept_names(
self, schema_identifier: Optional[SchemaIdentifier] = None
) -> List[str]:
schema = self._get_schema(schema_identifier=schema_identifier)
return list(schema.derived_concepts)
def template_names(
self, schema_identifier: Optional[SchemaIdentifier] = None
) -> List[str]:
schema = self._get_schema(schema_identifier=schema_identifier)
return list(schema.templates)
def deprecated_concept_names(
self, schema_identifier: Optional[SchemaIdentifier] = None
) -> List[str]:
schema = self._get_schema(schema_identifier=schema_identifier)
return list(schema.deprecated_concepts)
def deprecated_derived_concept_names(
self, schema_identifier: Optional[SchemaIdentifier] = None
) -> List[str]:
schema = self._get_schema(schema_identifier=schema_identifier)
return list(schema.deprecated_derived_concepts)
def deprecated_template_names(
self, schema_identifier: Optional[SchemaIdentifier] = None
) -> List[str]:
schema = self._get_schema(schema_identifier=schema_identifier)
return list(schema.deprecated_templates)
def get_concept(
self, name: str, schema_identifier: Optional[SchemaIdentifier] = None
) -> PathConcept:
"""
:raises UnknownPathConcept:
"""
schema = self._get_schema(schema_identifier=schema_identifier)
if name not in schema.concepts:
canonical_name = schema.deprecated_concepts.get(name)
if canonical_name is None:
raise errors.UnknownPathConcept(name)
_warn_deprecated_name("Concept", name, canonical_name)
name = canonical_name
return schema.concepts[name]
def get_derived_concept(
self, name: str, schema_identifier: Optional[SchemaIdentifier] = None
) -> DerivedConcept:
"""
:raises UnknownPathConcept:
"""
schema = self._get_schema(schema_identifier=schema_identifier)
if name not in schema.derived_concepts:
canonical_name = schema.deprecated_derived_concepts.get(name)
if canonical_name is None:
raise errors.UnknownPathConcept(name)
_warn_deprecated_name("Derived concept", name, canonical_name)
name = canonical_name
return schema.derived_concepts[name]
def find_derived_concept(
self, name: str, schema_identifier: Optional[SchemaIdentifier] = None
) -> Optional[DerivedConcept]:
"""Return the derived concept of the schema resolved by ``schema_identifier``
or of any other schema in the registry. A concept of one schema can be a
derived concept of another schema."""
resolved = self._get_schema(schema_identifier=schema_identifier)
for schema in [resolved] + self._schemas:
derived = schema.derived_concepts.get(name)
if derived is not None:
return derived
return None
def get_template(
self, name: str, schema_identifier: Optional[SchemaIdentifier] = None
) -> PathTemplate:
"""
:raises UnknownPathTemplate:
"""
schema = self._get_schema(schema_identifier=schema_identifier)
if name not in schema.templates:
canonical_name = schema.deprecated_templates.get(name)
if canonical_name is None:
raise errors.UnknownPathTemplate(name)
_warn_deprecated_name("Template", name, canonical_name)
name = canonical_name
return schema.templates[name]
def _get_schema(
self, schema_identifier: Optional[SchemaIdentifier] = None
) -> BasePathSchema:
if schema_identifier is None:
return self._schemas[0]
for schema in self._schemas:
resolved = schema.resolve_schema(schema_identifier)
if resolved is not None:
return resolved
return self._schemas[0]
def iter_templates(
self,
schema_identifier: Optional[SchemaIdentifier] = None,
include_shadowed: bool = False,
) -> Iterator[PathTemplate]:
"""Yield all path templates, optionally including same-named templates from
different schemas."""
schema = self._get_schema(schema_identifier=schema_identifier)
yield from schema.iter_templates(include_shadowed=include_shadowed)
def parse_path(self, path: Union[str, Path]) -> "ParsedPath":
"""
:raises PathSchemaMatchError:
"""
norm_path = str(path).replace(os.sep, "/")
# Keep the commented prints until we find a better we of debugging matching
# print("---")
# print("Scores for", norm_path)
best_template = None
best_score = None
best_concept_values = None
best_has_optionals = None
for _schema in self._schemas:
for template in _schema.iter_templates(include_shadowed=True):
concept_values, score = template.parse_with_score(norm_path)
if not score:
continue
# print(" ", score, template.name, "from", template.schema_identifier)
if best_score is None or score > best_score:
best_template = template
best_score = score
best_concept_values = concept_values
best_has_optionals = template.contains_optionals(concept_values)
if best_template is None:
continue
if not best_has_optionals:
break
if best_template is None:
raise errors.PathSchemaMatchError(f"No template matched path {path!r}")
# print(
# "Final:",
# best_score,
# best_template.name,
# "from",
# best_template.schema_identifier,
# )
return ParsedPath(self, best_concept_values, template=best_template)
def render_path(
self,
concept_values: Dict[str, ConceptValueType],
template_name: Optional[str] = None,
schema_identifier: Optional[SchemaIdentifier] = None,
) -> str:
"""
:raises PathSchemaMatchError:
:raises PathTemplateMatchError:
:raises UnknownPathTemplate:
"""
concept_values = self.resolve_concept_values(
concept_values, schema_identifier=schema_identifier
)
concept_values = self._fill_default_concepts(
concept_values, schema_identifier=schema_identifier
)
parsed = ParsedPath(self, concept_values)
# Note: do not keep the `parsed` instance because the concept values
# not to go through a serialize+deserialize cycle for normalization.
return parsed.render(template_name=template_name)
def resolve_concept_values(
self,
concept_values: Dict[str, ConceptValueType],
schema_identifier: Optional[SchemaIdentifier] = None,
) -> Dict[str, ConceptValueType]:
"""Translate deprecated concept-name keys of ``concept_values`` to their
canonical name, emitting a ``DeprecationWarning`` for each one translated.
Keys that are not a deprecated concept name (including unknown ones) are
kept as-is.
"""
schema = self._get_schema(schema_identifier=schema_identifier)
if not schema.deprecated_concepts:
return concept_values
resolved = {}
for key, value in concept_values.items():
canonical_name = schema.deprecated_concepts.get(key)
if canonical_name is not None:
_warn_deprecated_name("Concept", key, canonical_name)
key = canonical_name
resolved[key] = value
return resolved
def _fill_default_concepts(
self,
concept_values: Dict[str, ConceptValueType],
schema_identifier: Optional[SchemaIdentifier] = None,
) -> None:
provided_names = set(concept_values)
schema = self._get_schema(schema_identifier=schema_identifier)
all_names = set(schema.concepts)
if not (provided_names & all_names):
return concept_values
concept_values = {
cname: cvalue
for cname, cvalue in concept_values.items()
if cvalue is not None
}
known_names = set(concept_values)
unknown_names = all_names - known_names
for concept_name in unknown_names:
concept = schema.concepts[concept_name]
default_value = concept._default_value
if default_value is not None:
concept_values[concept_name] = default_value
return concept_values
_CACHE: Dict[Tuple[Tuple[str, int]], PathSchemaRegistry] = {}
def get_schema_registry(
fallback_depth: Optional[int] = None, **schemas: Optional[int]
) -> PathSchemaRegistry:
"""
:raises FieldNameError:
:raises PathSchemaCollision:
:raises UnknownPathSchema:
"""
if not schemas:
schemas = {constants.UNKNOWN_SCHEMA_NAME: 1}
# Resolve version for each schema: 0 or None -> latest available
main_versions = {
sname: sversion if sversion is not None else default_schema_version(sname)
for sname, sversion in schemas.items()
}
# Return when cached. The order of the schemas matters: it is the order
# of field name resolution when schemas share field names.
cache_key = (tuple(main_versions.items()), fallback_depth)
registry = _CACHE.get(cache_key)
if registry is not None:
return registry
# Build main merged schema
main_schemas: List[PathSchema] = []
for sname, sversion in main_versions.items():
main_schemas.append(get_schema(sname, sversion))
main_schema = MergedPathSchema(main_schemas)
schemas = [main_schema]
# Build secondary merged schemas from older versions
fallback_versions = fallback_schema_versions(
main_versions, fallback_depth=fallback_depth
)
if fallback_versions:
for older_versions in fallback_versions:
older_schemas = [
get_schema(sname, sversion)
for sname, sversion in older_versions.items()
]
try:
schemas.append(MergedPathSchema(older_schemas))
except (errors.FieldNameError, errors.PathSchemaCollision):
# Schema versions are not compatible
pass
registry = PathSchemaRegistry(schemas)
_CACHE[cache_key] = registry
return registry
class ParsedPath:
"""Container of path concept values which are either provided by the users
or extracted from a path string by a schema registry. The corresponding path
template is provided or can be derived using the registry.
`Fields` are exposed as instance attributes. A field can be a *path concept*,
*derived concept* or *rendered path template*.
"""
def __init__(
self,
registry: PathSchemaRegistry,
concept_values: Dict[str, ConceptValueType],
template: Optional[PathTemplate] = None,
):
self._concept_values = {
k: v for k, v in concept_values.items() if v is not None
}
self._template = template
self._registry = registry
@property
def concept_names(self) -> List[str]:
return list(self._concept_values)
@property
def required_concept_names(self) -> List[str]:
return self.template.required_concept_names
@property
def template(self) -> PathTemplate:
"""
:raises UnknownPathTemplate:
:raises PathSchemaMatchError:
"""
return self._get_template()
@property
def template_name(self) -> str:
try:
return self.template.name
except errors.UnknownPathTemplate:
return ""
except errors.PathSchemaMatchError:
return ""
except Exception:
# Used in __repr__ and avoid circular calls in case we get an unexpected error
_logger.exception("Failed unexpectidly")
return ""
@property
def schema_identifier(self) -> SchemaIdentifier:
try:
return self.template.schema_identifier
except errors.UnknownPathTemplate:
return ""
except errors.PathSchemaMatchError:
return ""
except Exception:
# Used in __repr__ and avoid circular calls in case we get an unexpected error
_logger.exception("Failed unexpectidly")
return ""
@property
def schema_description(self) -> str:
try:
return self._registry.description(self.template.schema_identifier)
except errors.UnknownPathTemplate:
return ""
except errors.PathSchemaMatchError:
return ""
except Exception:
# Used in __repr__ and avoid circular calls in case we get an unexpected error
_logger.exception("Failed unexpectidly")
return ""
@property
def _optional_schema_identifier(self) -> Optional[str]:
try:
return self.template.schema_identifier
except (errors.UnknownPathTemplate, errors.PathSchemaMatchError):
return None
def __repr__(self) -> str:
return f"<{type(self).__name__} schema={str(self.schema_identifier)!r} template={self.template_name!r} concepts={self._concept_values}>"
def __info__(self, include_missing: bool = False) -> str:
lines = [
f"Template name: {self.template_name} (schema={self.schema_identifier})",
f"Path: {self.render()}",
]
fields = self.field_values(include_missing=include_missing)
for fname, fvalue in fields.items():
lines.append(f" {fname}: {fvalue}")
return "\n".join(lines)
def __eq__(self, value: object) -> bool:
if isinstance(value, ParsedPath):
return self._concept_values == value._concept_values
return super().__eq__(value)
def __getattr__(self, item: str) -> Any:
"""
:raises SchemaAttributeError:
"""
try:
return self.get_field_value(item, raise_on_mismatch=False)
except (errors.PathConceptValueError, errors.UnknownField) as ex:
raise errors.SchemaAttributeError(item) from ex
def get_field_value(
self, item: str, raise_on_missing: bool = True, raise_on_mismatch: bool = True
) -> Any:
"""
:raises UnknownField:
:raises PathConceptValueError:
"""
item = str(item)
schema_identifier = self._optional_schema_identifier
canonical_name = None
if item in self._registry.concept_names(schema_identifier):
canonical_name = item
elif item in self._registry.deprecated_concept_names(schema_identifier):
# `get_concept` also emits the `DeprecationWarning`.
canonical_name = self._registry.get_concept(item, schema_identifier).name
if canonical_name is not None:
if canonical_name in self._concept_values:
value = self._concept_values[canonical_name]
if value is not None:
return value
raise errors.PathConceptWithoutValue(
f"{canonical_name!r} not known for {self.template_name!r}"
)
if item in self._registry.template_names(
schema_identifier
) or item in self._registry.deprecated_template_names(schema_identifier):
template = self._registry.get_template(item, schema_identifier)
concept_values = template.remove_optionals(
self._template_concept_values(template)
)
return template.render(
concept_values,
raise_on_missing=raise_on_missing,
raise_on_mismatch=raise_on_mismatch,
)
if item in self._registry.derived_concept_names(
schema_identifier
) or item in self._registry.deprecated_derived_concept_names(schema_identifier):
concept = self._registry.get_derived_concept(item, schema_identifier)
return concept.derive(
self._concept_values, raise_on_missing=raise_on_missing
)
raise errors.UnknownField(
f"{item!r} is not a known field: {self.field_names()}"
)
def field_names(self, include_missing: bool = False) -> List[str]:
"""Return all field names. A field is either a path concept,
derived path concept or a path template."""
schema_identifier = self._optional_schema_identifier
if include_missing:
concept_names = self._registry.concept_names(schema_identifier)
else:
concept_names = list(self._concept_values)
return (
concept_names
+ self._registry.derived_concept_names(schema_identifier)
+ self._registry.template_names(schema_identifier)
)
def field_values(self, include_missing: bool = False) -> Dict[str, Any]:
"""Return all fields that can be derived from the known concepts.
Use "*" for missing concepts.
"""
field_values = {}
for fname in self.field_names(include_missing=include_missing):
try:
field_values[fname] = self.get_field_value(
fname, raise_on_missing=not include_missing
)
except (errors.PathConceptValueError, errors.UnknownField):
pass
return field_values
def resolve_template_name(self, template_name: str) -> str:
"""Canonical name of ``template_name``, emitting a ``DeprecationWarning``
if it is a deprecated template name.
:raises UnknownPathTemplate:
"""
return self._registry.get_template(
template_name, self._optional_schema_identifier
).name
def path_field_names(self) -> List[str]:
"""Return all field names that are paths (path templates)."""
schema_identifier = self._optional_schema_identifier
return self._registry.template_names(schema_identifier)
def deprecated_path_field_names(self) -> List[str]:
"""Return all deprecated field names that are paths (path templates)."""
schema_identifier = self._optional_schema_identifier
return self._registry.deprecated_template_names(schema_identifier)
def nonpath_field_names(self, include_missing: bool = False) -> List[str]:
"""Return all field names that are not paths (path concepts and derived path concepts)."""
schema_identifier = self._optional_schema_identifier
if include_missing:
concept_names = self._registry.concept_names(schema_identifier)
else:
concept_names = list(self._concept_values)
return concept_names + self._registry.derived_concept_names(schema_identifier)
def deprecated_nonpath_field_names(self) -> List[str]:
"""Return all deprecated field names that are not paths (deprecated path
concepts and derived path concepts)."""
schema_identifier = self._optional_schema_identifier
return self._registry.deprecated_concept_names(
schema_identifier
) + self._registry.deprecated_derived_concept_names(schema_identifier)
def parse_as_template(self, path: str, template_name: str) -> "ParsedPath":
"""Concept values of `path` extracted with a specific path template.
:raises PathTemplateMatchError:
:raises UnknownPathTemplate:
"""
template = self._registry.get_template(
template_name, self._optional_schema_identifier
)
return ParsedPath(self._registry, template.parse(path), template=template)
def render(self, template_name: Optional[str] = None) -> str:
"""Convert schema to path.
:raises PathSchemaMatchError:
:raises PathTemplateMatchError:
:raises UnknownPathTemplate:
"""
template = self._get_template(template_name=template_name)
try:
return template.render(self._template_concept_values(template))
except errors.PathConceptValueError as ex:
raise errors.PathTemplateMatchError(
f"Path is not a {template.name!r}"
) from ex
def _template_concept_values(
self, template: PathTemplate
) -> Dict[str, ConceptValueType]:
"""Concept values completed with the concepts of ``template`` that are
not known but can be derived (a concept of one schema can be a derived
concept of another schema).
"""
schema_identifier = template.schema_identifier
derived_values = {}
for cname in template.concept_names:
if cname in self._concept_values:
continue
derived = self._registry.find_derived_concept(cname, schema_identifier)
if derived is None:
continue
try:
derived_values[cname] = derived.derive(self._concept_values)
except errors.PathConceptValueError:
pass
if not derived_values:
return self._concept_values
return {**self._concept_values, **derived_values}
def _get_template(self, template_name: Optional[str] = None) -> PathTemplate:
"""
:raises UnknownPathTemplate:
:raises PathSchemaMatchError:
"""
if template_name:
# Resolve the name within the schema of the matched template
# in case several schemas define the same template name.
schema_identifier = None
if self._template is not None:
schema_identifier = self._template.schema_identifier
return self._registry.get_template(template_name, schema_identifier)
if self._template is not None:
return self._template
# Keep the commented prints until we find a better we of debugging matching
# print("---")
# print("Scores for", self._concept_values)
best_template = None
best_score = None
for template in self._registry.iter_templates(include_shadowed=True):
score = template.render_score(self._template_concept_values(template))
if not score:
continue
# print(" ", score, template.name, "from", template.schema_identifier)
if best_score is None or score > best_score:
best_template = template
best_score = score
if best_template is None:
raise errors.PathSchemaMatchError(
f"No path template can be found that is fully described by the fields {self._concept_values}"
)
# print(
# "Best:",
# best_score,
# best_template.name,
# "from",
# best_template.schema_identifier,
# )
self._template = best_template
return best_template
def replace_concepts(
self,
template_name: Optional[str] = None,
**concept_values: ConceptValueType,
) -> "ParsedPath":
"""Returns a new ``ParsedPath`` instance with additional or replaces concepts.
:raises PathSchemaMatchError:
:raises PathTemplateMatchError:
"""
concept_values = self._registry.resolve_concept_values(
concept_values, self._optional_schema_identifier
)
new_concept_values = self._concept_values.copy()
new_concept_values.update(concept_values)
if template_name:
template = self._get_template(template_name)
else:
template = None
parsed = ParsedPath(self._registry, new_concept_values, template=template)
path = parsed.render()
return self._registry.parse_path(path)
def strip_mount_point(self) -> "ParsedPath":
"""Returns a new ``ParsedPath`` instance with stripped leading segments.
:raises PathSchemaMatchError:
:raises PathTemplateMatchError:
"""
template = self._get_template()
new_concept_values = self._template.strip_mount_point(self._concept_values)
parsed = ParsedPath(self._registry, new_concept_values, template=template)
path = parsed.render()
return self._registry.parse_path(path)
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/schema/ 0000775 0000000 0000000 00000000000 15244036076 0030322 5 ustar 00root root 0000000 0000000 __init__.py 0000664 0000000 0000000 00000000556 15244036076 0032362 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/schema """An ESRF filesystem path schema is defined by several *path templates* describing a hierarchical
directory structure and *path concepts* that are the components that make up a template.
Schemas are used to match path strings, *parse* path strings (extract path concept values)
and *render* path strings from a dictionary of concepts (the inverse of parsing).
"""
base.py 0000664 0000000 0000000 00000031146 15244036076 0031534 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/schema from typing import Dict
from typing import Iterator
from typing import List
from typing import Optional
from typing import Tuple
from typing import Union
from .. import errors
from ..fields.base import Field
from ..fields.concept import PathConcept
from ..fields.derived import DerivedConcept
from ..fields.path import PathTemplate
from ..fields.tree import tree_as_strings
from ..identifier import SchemaIdentifier
class BasePathSchema:
"""A versioned collection of path concepts, derived concepts and path templates."""
_FORBIDDEN_FIELD_NAMES = ("schema_name", "schema_version", "template_name")
def __init__(self, identifier: SchemaIdentifier, description: str) -> None:
self._identifier = identifier
self._docstring = description
self._description = _extract_short_description(description)
self._concepts: Dict[str, PathConcept] = {}
self._derived_concepts: Dict[str, DerivedConcept] = {}
self._templates: Dict[str, PathTemplate] = {}
# Deprecated name -> canonical name of the field it was renamed to.
self._deprecated_concepts: Dict[str, str] = {}
self._deprecated_derived_concepts: Dict[str, str] = {}
self._deprecated_templates: Dict[str, str] = {}
# Shadowed name -> templates matched by path but not addressable by name
# (same-named templates from a merged schema, beyond the first one added).
self._shadowed_templates: Dict[str, List[PathTemplate]] = {}
self._all_path_segments = []
@property
def identifier(self) -> SchemaIdentifier:
return self._identifier
@property
def description(self) -> str:
"""Short summary of the schema, e.g. when it was introduced."""
return self._description
@property
def docstring(self) -> str:
"""Full module docstring the schema was defined with."""
return self._docstring
@property
def concepts(self) -> Dict[str, PathConcept]:
return self._concepts
@property
def derived_concepts(self) -> Dict[str, DerivedConcept]:
return self._derived_concepts
@property
def templates(self) -> Dict[str, PathTemplate]:
return self._templates
@property
def deprecated_concepts(self) -> Dict[str, str]:
"""Deprecated concept name -> canonical concept name."""
return self._deprecated_concepts
@property
def deprecated_derived_concepts(self) -> Dict[str, str]:
"""Deprecated derived concept name -> canonical derived concept name."""
return self._deprecated_derived_concepts
@property
def deprecated_templates(self) -> Dict[str, str]:
"""Deprecated template name -> canonical template name."""
return self._deprecated_templates
def iter_templates(self, include_shadowed: bool = False) -> Iterator[PathTemplate]:
"""Yield all path templates, optionally including templates shadowed by a
same-named template from another schema (merged schemas only)."""
yield from self._templates.values()
if not include_shadowed:
return
for shadowed in self._shadowed_templates.values():
yield from shadowed
def __info__(self) -> str:
lines = [f"Schema: {self.identifier}"]
if self.description:
lines.append(f" Description: {self.description}")
lines.append(" Tree:")
tree_lines = tree_as_strings(self._all_path_segments)
lines.extend([f" {s}" for s in tree_lines])
def field_string(field: Field):
category, values = field.value_info()
if category is None:
return f" - {field.name}: {field.description}"
return f" - {field.name}: {field.description} ({category}: {', '.join(values)})"
if self.concepts:
lines.append(" Concepts:")
lines.extend(field_string(concept) for concept in self.concepts.values())
else:
lines.append(" Concepts: (none)")
if self.derived_concepts:
lines.append(" Derived concepts:")
lines.extend(
field_string(derived) for derived in self.derived_concepts.values()
)
else:
lines.append(" Derived concepts: (none)")
if self.templates:
lines.append(" Paths:")
lines.extend(field_string(template) for template in self.templates.values())
else:
lines.append(" Paths: (none)")
return "\n".join(lines)
def __repr__(self) -> str:
return f"<{type(self).__name__} {self.identifier}>"
def has_schema(self, schema_identifier: str) -> bool:
return self.identifier.has_identifier(schema_identifier)
def resolve_schema(
self, schema_identifier: Union[str, SchemaIdentifier]
) -> Optional["BasePathSchema"]:
"""Return the schema providing the field namespace for ``schema_identifier``
or ``None`` when this schema does not know the identifier."""
if self.has_schema(schema_identifier):
return self
return None
@staticmethod
def _create_identifier(
name: Optional[str] = None,
version: Optional[int] = None,
schemas: Optional[List["BasePathSchema"]] = None,
) -> None:
"""
:raises PathSchemaCollision:
"""
if name is not None and version is not None:
identifier_kwargs = {name: version}
schema_versions = {}
else:
identifier_kwargs = schema_versions = {}
for schema in schemas or []:
for (
schema_name,
schema_version,
) in schema.identifier._schema_versions.items():
if schema_name not in schema_versions:
schema_versions[schema_name] = schema_version
elif schema_version != schema_versions[schema_name]:
raise errors.PathSchemaCollision(
f"Cannot mix different versions ({schema_version} and {schema_versions[schema_name]}) of the same schema {schema_name!r}"
)
return SchemaIdentifier(**identifier_kwargs)
def _merge_fields_from(
self, schema: "BasePathSchema", collision: str = "raise"
) -> None:
"""
:raises FieldNameError:
"""
for concept in schema.concepts.values():
self._add_field(concept, collision=collision)
for derived in schema.derived_concepts.values():
self._add_field(derived, collision=collision)
for template in schema.templates.values():
self._add_field(template, collision=collision)
for shadowed in schema._shadowed_templates.values():
for template in shadowed:
self._add_shadowed_template(template)
def _add_field(self, field: Field, collision: str = "raise") -> None:
"""``collision`` selects the behavior when adding a path template with the
name of an already existing path template, or a concept with the name of
an already existing derived concept (and vice versa):
- ``"raise"``: raise :class:`FieldNameError`,
- ``"keep"``: a colliding path template becomes a shadowed template (used
for path matching but not for name-based field access) while a concept
and a derived concept with the same name are both kept.
:raises FieldNameError:
"""
if isinstance(field, PathConcept):
container = self.concepts
others = (self.derived_concepts, self.templates)
deprecated_container = self._deprecated_concepts
other_deprecated = (
self._deprecated_derived_concepts,
self._deprecated_templates,
)
elif isinstance(field, DerivedConcept):
container = self.derived_concepts
others = (self.concepts, self.templates)
deprecated_container = self._deprecated_derived_concepts
other_deprecated = (self._deprecated_concepts, self._deprecated_templates)
elif isinstance(field, PathTemplate):
container = self.templates
others = (self.concepts, self.derived_concepts)
deprecated_container = self._deprecated_templates
other_deprecated = (
self._deprecated_concepts,
self._deprecated_derived_concepts,
)
else:
raise TypeError(type(field))
fname = field.name
# Field of the same type already defined?
if fname in container:
existing = container[fname]
if id(existing) == id(field):
return
if (
isinstance(field, PathConcept)
and isinstance(existing, PathConcept)
and existing.is_same_definition(field)
):
return
if isinstance(field, PathTemplate) and collision == "keep":
self._add_shadowed_template(field)
return
raise errors.FieldNameError(f"Field {fname!r} already exists as {existing}")
# Field of a diffrent type already defined?
for other in others:
if fname not in other:
continue
existing = other[fname]
if id(existing) == id(field):
return
if collision == "keep" and _is_concept(field) and _is_concept(existing):
# A concept of one schema can be a derived concept of
# another schema
continue
raise errors.FieldNameError(
f"{field} of schema {str(field.schema_identifier)!r} already exists as "
f"{existing} of schema {str(existing.schema_identifier)!r}"
)
# Add field
container[fname] = field
# Add its deprecated names
self._add_deprecated_names(
field,
fname,
container,
others,
deprecated_container,
other_deprecated,
collision,
)
@staticmethod
def _add_deprecated_names(
field: Field,
fname: str,
container: Dict[str, Field],
others: Tuple[Dict[str, Field], ...],
deprecated_container: Dict[str, str],
other_deprecated: Tuple[Dict[str, str], ...],
collision: str,
) -> None:
"""Register the deprecated names of a newly added field as aliases for
``fname``. A deprecated name that collides with an existing (canonical or
deprecated) name is dropped when ``collision == "keep"`` (merged schemas)
and otherwise raises.
:raises FieldNameError:
"""
for old_name in field.deprecated_names:
if old_name == fname:
continue
colliding = (
old_name in container
or any(old_name in other for other in others)
or any(old_name in other for other in other_deprecated)
)
existing_target = deprecated_container.get(old_name)
if existing_target is not None and existing_target != fname:
colliding = True
if colliding:
if collision == "keep":
continue
raise errors.FieldNameError(
f"Deprecated name {old_name!r} of {field} collides with an existing field name"
)
deprecated_container[old_name] = fname
def _add_shadowed_template(self, template: PathTemplate) -> None:
"""Add a path template that is used for path matching but not for
name-based field access."""
if self._templates.get(template.name) is template:
return
shadowed = self._shadowed_templates.setdefault(template.name, [])
if not any(existing is template for existing in shadowed):
shadowed.append(template)
def _is_concept(field: Field) -> bool:
return isinstance(field, (PathConcept, DerivedConcept))
def _extract_short_description(docstring: str) -> str:
"""Return the second paragraph of a module docstring, e.g. for::
\"\"\"
ESRF Data Policy v2
Introduced with BLISS 1.10.1 for beamline data on Dec 16, 2022.
Path hierarchy::
...
\"\"\"
this returns ``"Introduced with BLISS 1.10.1 for beamline data on Dec 16, 2022."``.
Falls back to the first paragraph when there is no second one.
"""
paragraphs = [p.strip() for p in docstring.strip().split("\n\n")]
paragraphs = [p for p in paragraphs if p]
paragraph = paragraphs[1] if len(paragraphs) > 1 else next(iter(paragraphs), "")
return " ".join(paragraph.split())
merged.py 0000664 0000000 0000000 00000004157 15244036076 0032067 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/schema from typing import List
from typing import Optional
from typing import Union
from ..identifier import SchemaIdentifier
from .base import BasePathSchema
from .path import PathSchema
class MergedPathSchema(BasePathSchema):
"""A collection of path concepts, derived concepts and path templates merged from different path schema's.
Schemas may define path templates with the same name (for example the same
logical path rooted at a different directory) or a concept that is a derived
concept in another schema. Path matching considers all templates while
name-based field access resolves to the first schema unless a schema
identifier is provided (see :meth:`resolve_schema`).
"""
def __init__(self, schemas: List[PathSchema]):
"""The order of the schemas is the order of field name resolution.
:raises FieldNameError:
:raises PathSchemaCollision:
"""
identifier = self._create_identifier(schemas=schemas)
super().__init__(identifier, "Merged schema")
self._component_schemas: List[PathSchema] = list(schemas)
for schema in schemas:
self._merge_fields_from(schema, collision="keep")
self._all_path_segments = [
lst for schema in schemas for lst in schema._all_path_segments
]
@property
def description(self) -> str:
if len(self._component_schemas) == 1:
return self._component_schemas[0].description
return "; ".join(
f"{schema.identifier}: {schema.description}"
for schema in self._component_schemas
)
def resolve_schema(
self, schema_identifier: Union[str, SchemaIdentifier]
) -> Optional[BasePathSchema]:
"""Resolve to the merged schema component so that same-named fields
from different components resolve within the component identified by
``schema_identifier``."""
for schema in self._component_schemas:
resolved = schema.resolve_schema(schema_identifier)
if resolved is not None:
return resolved
return super().resolve_schema(schema_identifier)
path.py 0000664 0000000 0000000 00000007676 15244036076 0031571 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/schema from typing import Any
from typing import Callable
from typing import Dict
from typing import List
from typing import Optional
from ..definitions.types import ConceptValueType
from ..fields.concept import PathConcept
from ..fields.derived import DerivedConcept
from ..fields.derived import template_derive_func
from ..fields.path import PathTemplate
from ..fields.tree import SegmentTemplate
from ..fields.tree import expand_placeholders
from ..fields.tree import parse_template_tree
from ..typing import TemplateTree
from .base import BasePathSchema
class PathSchema(BasePathSchema):
"""A versioned collection of path concepts, derived concepts and path templates."""
def __init__(
self,
name: str,
version: int,
description: str,
concepts: Optional[Dict[str, Dict[str, Any]]] = None,
derived_concepts: Optional[Dict[str, Dict[str, Any]]] = None,
template_tree: Optional[TemplateTree] = None,
extends: Optional[List["PathSchema"]] = None,
symbolic_root_segment: Optional[str] = None,
) -> None:
"""
:raises FieldNameError:
:raises PathSchemaCollision:
"""
identifier = self._create_identifier(
name=name, version=version, schemas=extends
)
super().__init__(identifier, description)
for schema in extends or []:
self._merge_fields_from(schema)
symbolic_root_segment = (
symbolic_root_segment or schema._symbolic_root_segment
)
for concept_name, kwargs in (concepts or {}).items():
concept = PathConcept(
name=concept_name, schema_identifier=self.identifier, **kwargs
)
self._add_field(concept)
for concept_name, kwargs in (derived_concepts or {}).items():
self._add_field(self._create_derived_concept(concept_name, **kwargs))
# Derived concepts that are a shorthand for a path template fragment
concept_templates = {
dname: derived.template
for dname, derived in self.derived_concepts.items()
if derived.template is not None
}
arguments = parse_template_tree(
template_tree or {}, symbolic_root_segment=symbolic_root_segment
)
all_path_segments: List[List[SegmentTemplate]] = []
for args in arguments:
segments = expand_placeholders(args.segments, concept_templates)
all_path_segments.append(segments)
template = PathTemplate(
name=args.name,
schema_identifier=self.identifier,
segments=segments,
concepts=self.concepts,
deprecated_names=args.deprecated_names,
)
self._add_field(template)
self._all_path_segments = all_path_segments
self._symbolic_root_segment = symbolic_root_segment
def _create_derived_concept(
self,
name: str,
description: str,
template: Optional[str] = None,
derive_func: Optional[
Callable[[Dict[str, ConceptValueType]], ConceptValueType]
] = None,
derived_from: Optional[List[str]] = None,
examples: Optional[List[str]] = None,
deprecated_names: Optional[List[str]] = None,
) -> DerivedConcept:
"""``template`` is an alternative to ``derive_func`` and ``derived_from``
for a concept that is a shorthand for a path template fragment.
:raises UnknownPathConcept:
"""
if template is not None:
derive_func, derived_from = template_derive_func(template, self.concepts)
return DerivedConcept(
name=name,
description=description,
schema_identifier=self.identifier,
derive_func=derive_func,
derived_from=derived_from,
template=template,
examples=examples,
deprecated_names=deprecated_names,
)
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/_schemas/typing.py 0000664 0000000 0000000 00000000633 15244036076 0030750 0 ustar 00root root 0000000 0000000 """Structural type aliases shared across the schema framework."""
from typing import Dict
from typing import Optional
from typing import Tuple
from typing import Union
RawSegmentType = Union[
str,
Tuple[str],
Tuple[str, str],
Tuple[str, str, bool],
Tuple[str, str, bool, Tuple[str, ...]],
]
# Parsed as `_ValidatedRawSegment`
TemplateTree = Dict[RawSegmentType, Optional["TemplateTree"]]
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/core.py 0000664 0000000 0000000 00000025442 15244036076 0026611 0 ustar 00root root 0000000 0000000 import glob as _glob
import pathlib
import sys
from typing import Any
from typing import ClassVar
from typing import Iterator
from typing import List
from typing import Optional
from typing import Type
from typing import TypeVar
from typing import Union
from ._schemas.definitions.types import ConceptValueType as _ConceptValueType
from ._schemas.errors import PathSchemaMatchError as _PathSchemaMatchError
from ._schemas.errors import PathTemplateMatchError as _PathTemplateMatchError
from ._schemas.errors import SchemaAttributeError as _SchemaAttributeError
from ._schemas.registry import PathSchemaRegistry as _Registry
from ._schemas.registry import get_schema_registry as _get_registry
_T = TypeVar("_T", bound="ESRFPath")
if sys.version_info >= (3, 12):
# https://docs.python.org/3/whatsnew/3.12.html
# -> The pathlib.Path class now supports subclassing
class _BaseESRFPath(pathlib.Path):
__slots__ = ("_esrf_parsed",)
def __init__(self, *args: Any) -> None:
super().__init__(*args)
_add_esrf_parsed(self, *args)
else:
class _BaseESRFPath(type(pathlib.Path())):
__slots__ = ("_esrf_parsed",)
def __new__(cls, *args: Any) -> "_BaseESRFPath":
self = super().__new__(cls, *args)
if not hasattr(self, "_esrf_parsed"):
# `Path.__new__` instantiates through `_from_parts` which parsed
# the path already
_add_esrf_parsed(self, *args)
return self
@classmethod
def _from_parts(cls, args: Any, **kwargs: Any) -> "_BaseESRFPath":
self = super()._from_parts(args, **kwargs)
_add_esrf_parsed(self, *args)
return self
@classmethod
def _from_parsed_parts(
cls, drv: str, root: str, parts: List[str], **kwargs: Any
) -> "_BaseESRFPath":
self = super()._from_parsed_parts(drv, root, parts, **kwargs)
_add_esrf_parsed(self)
return self
class ESRFPath(_BaseESRFPath):
"""Provides all features of ``pathlib.Path`` with additional attributes
depending on the matching ESRF filesystem path template.
"""
_SCHEMAS: ClassVar[dict] = {"esrf": None} # Latest version
_FALLBACK_DEPTH = 1 # Fall back one version by default
_REGISTRY = None
def __init_subclass__(cls, fallback_depth: Optional[int] = None, **schemas):
"""Allow subclasses to declare their own schemas like:
class TomoPath(ESRFPath, tomo=1):
pass
"""
super().__init_subclass__()
if schemas != cls._SCHEMAS:
cls._SCHEMAS = schemas
cls._REGISTRY = None
if fallback_depth != cls._FALLBACK_DEPTH:
cls._FALLBACK_DEPTH = fallback_depth
cls._REGISTRY = None
def __getattr__(self, attr_name: str) -> Any:
"""
:raises SchemaAttributeError:
"""
if attr_name in self._get_schema_path_attributes():
# Resolved once so a deprecated `attr_name` only warns once.
template_name = self._esrf_parsed.resolve_template_name(attr_name)
path = getattr(self._esrf_parsed, template_name)
# The rendered path is a path of `template_name`, even when another
# template of the registry is a better match for it.
instance = self.__class__(path)
instance._esrf_parsed = self._esrf_parsed.parse_as_template(
path, template_name
)
return instance
if attr_name in self._get_schema_nonpath_attributes():
return getattr(self._esrf_parsed, attr_name)
if not attr_name.startswith("_"):
raise _SchemaAttributeError(f"{self!r} has no attribute {attr_name!r}")
raise _SchemaAttributeError(
f"{type(self).__name__!r} object has no attribute {attr_name!r}"
)
def __setattr__(self, attr_name: str, value: Any) -> None:
if not attr_name.startswith("_") and attr_name in self._get_schema_attributes():
raise _SchemaAttributeError(
f"Attribute {attr_name!r} is immutable. Create a new path instance with `{type(self).__name__}.replace_fields({attr_name}={value!r})`."
)
return super().__setattr__(attr_name, value)
def _get_schema_path_attributes(self) -> List[str]:
if self._esrf_parsed is None:
return []
return (
self._esrf_parsed.path_field_names()
+ self._esrf_parsed.deprecated_path_field_names()
)
def _get_schema_nonpath_attributes(self) -> List[str]:
if self._esrf_parsed is None:
return []
return (
self._esrf_parsed.nonpath_field_names()
+ self._esrf_parsed.deprecated_nonpath_field_names()
)
def _get_schema_attributes(self) -> List[str]:
if self._esrf_parsed is None:
return []
return self._esrf_parsed.field_names()
def __dir__(self) -> List[str]:
return sorted(set(super().__dir__()) | set(self._get_schema_attributes()))
def __repr__(self) -> str:
return f"<{type(self).__name__} {str(self)!r} schema={self.schema_name!r} template={self.template_name!r}>"
@property
def schema_name(self) -> str:
"""Name of the ESRF path schema if it matches a registered schema."""
if self._esrf_parsed:
return str(self._esrf_parsed.schema_identifier)
else:
return ""
@property
def schema_description(self) -> str:
"""Short description of the ESRF path schema if it matches a registered schema."""
if self._esrf_parsed:
return self._esrf_parsed.schema_description
else:
return ""
@property
def template_name(self) -> str:
"""Name of the ESRF path template if it matches a template from a registered schema."""
if self._esrf_parsed:
return self._esrf_parsed.template_name
else:
return ""
def replace_fields(
self: _T,
template_name: Optional[str] = None,
**fields: _ConceptValueType,
) -> _T:
"""Creates a new path of the same type, replacing fields with values from `fields`.
:raises PathSchemaMatchError:
:raises PathTemplateMatchError:
"""
if self._esrf_parsed is None:
raise _PathSchemaMatchError("Path does not match any ESRF schema")
current_parsed = self._esrf_parsed
# Add/replace concept values
new_parsed = current_parsed.replace_concepts(
template_name=template_name, **fields
)
# Render the new path with the existing template
new_path = None
if template_name is None and set(current_parsed.concept_names) == set(
new_parsed.concept_names
):
try:
new_path = new_parsed.render(template_name=current_parsed.template_name)
except _PathTemplateMatchError:
pass
# Render the new path with a new template (best match)
if new_path is None:
new_path = new_parsed.render()
# New ESRFPath instance
new_instance = self.__class__(new_path)
# Append part not defined by the schema
original_schema_base = current_parsed.render()
part_not_in_template = self.relative_to(original_schema_base)
return new_instance / part_not_in_template
def strip_mount_point(self: _T) -> _T:
"""
Remove leading segments typically introduced by ``ESRFPath.resolve()``.
"""
if self._esrf_parsed is None:
return self.__class__(self)
new_parsed = self._esrf_parsed.strip_mount_point()
new_path = new_parsed.render()
return self.__class__(new_path)
def glob(self: _T, pattern: Optional[str] = None, **kwargs: Any) -> Iterator[_T]:
"""Without `pattern`, glob this path's own string representation,
supporting field wildcards from `replace_fields(..., field="*")`.
"""
if pattern is None:
return (self.__class__(match) for match in _glob.glob(str(self)))
return super().glob(pattern, **kwargs)
@classmethod
def from_fields(
cls: Type[_T],
schema_name: Optional[str] = None,
schema_version: Optional[int] = None,
template_name: Optional[str] = None,
**fields: _ConceptValueType,
) -> _T:
"""Create an instance from schema fields.
:raises PathSchemaMatchError:
:raises PathTemplateMatchError:
:raises UnknownPathTemplate:
"""
if schema_name is not None:
registry = _get_registry(**{schema_name: schema_version})
else:
registry = cls._get_registry()
path = registry.render_path(fields, template_name=template_name)
instance = cls(path)
instance._esrf_parsed = registry.parse_path(path)
return instance
@classmethod
def from_path(
cls: Type[_T],
path: Union[str, pathlib.Path],
schema_name: Optional[str] = None,
schema_version: Optional[int] = None,
template_name: Optional[str] = None,
) -> _T:
"""Instantiate with a specific schema, schema version and template.
:raises UnknownPathSchema:
:raises PathSchemaCollision:
:raises UnknownPathTemplate:
:raises PathSchemaMatchError:
:raises PathTemplateMatchError:
"""
if schema_name is not None:
registry = _get_registry(**{schema_name: schema_version})
else:
registry = cls._get_registry()
parsed_path = registry.parse_path(path)
if template_name:
_ = parsed_path.render(template_name)
instance = cls(path)
instance._esrf_parsed = parsed_path
return instance
@classmethod
def _get_registry(cls) -> _Registry:
"""Get default registry of the class.
:raises UnknownPathSchema:
:raises PathSchemaCollision:
"""
if cls._REGISTRY is None:
cls._REGISTRY = _get_registry(
fallback_depth=cls._FALLBACK_DEPTH, **cls._SCHEMAS
)
return cls._REGISTRY
def _add_esrf_parsed(self: _BaseESRFPath, *args: Any) -> None:
self._esrf_parsed = None
if len(args) == 1 and isinstance(args[0], ESRFPath):
# Inherit the parsed state of the original path: there might be
# another template that is a better match.
original = args[0]
if (
original._esrf_parsed is not None
and original._get_registry() is self._get_registry()
and str(original) == str(self)
):
self._esrf_parsed = original._esrf_parsed
return
try:
self._esrf_parsed = self._get_registry().parse_path(self)
except _PathSchemaMatchError:
pass
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/errors.py 0000664 0000000 0000000 00000000172 15244036076 0027166 0 ustar 00root root 0000000 0000000 from ._schemas.errors import EsrfPathlibError # noqa F401
from ._schemas.errors import SchemaAttributeError # noqa F401
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/ 0000775 0000000 0000000 00000000000 15244036076 0026442 5 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/__init__.py 0000664 0000000 0000000 00000000000 15244036076 0030541 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf/ 0000775 0000000 0000000 00000000000 15244036076 0027401 5 ustar 00root root 0000000 0000000 __init__.py 0000664 0000000 0000000 00000000173 15244036076 0031434 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf """*esrf* schema tests."""
from ...core import ESRFPath
class EsrfPath(ESRFPath, esrf=3, fallback_depth=None):
pass
test_caveats.py 0000664 0000000 0000000 00000007161 15244036076 0032366 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf """Consequences of `EsrfPath` combining multiple *esrf* schema versions via version
fallback which are easy to trip over. These tests assert documented behavior, not
desired behavior: a failure here means the behavior changed, which is an API
decision rather than a bug fix.
"""
import pytest
from ..._schemas import errors
from ..._schemas.definitions.types import DataType
from ...core import ESRFPath
from ..utils import make_path
from . import EsrfPath
class EsrfPathShallowFallback(ESRFPath, esrf=3, fallback_depth=1):
"""Same schema as `EsrfPath` but with fallback limited to one older version
(esrf_v3, esrf_v2)."""
class EsrfPathNoFallback(ESRFPath, esrf=3, fallback_depth=0):
"""Same schema as `EsrfPath` but without any older-version fallback."""
def test_caveat_1_fallback_depth_limits_recognized_formats():
"""Parsing tries the main schema version first and only falls back to older
versions when it does not fully identify the path. A 'fallback_depth' too
shallow to reach the version that actually matches does not raise: it silently
degrades to a much less specific template ('session_path'), because esrf_v2
(the only fallback within reach) does not match this esrf_v1-only path either."""
v1_path = make_path(
"visitor", "ma6658", "id21", "20250509", "foo", "foo_bar", "foo_bar.h5"
)
path = EsrfPath(v1_path)
assert path.schema_name == "esrf_v1"
assert path.template_name == "raw_dataset_file"
assert path.collection == "foo"
assert path.dataset == "bar"
for cls in (EsrfPathShallowFallback, EsrfPathNoFallback):
path = cls(v1_path)
assert path.schema_name == "esrf_v3"
assert path.template_name == "session_path"
with pytest.raises(errors.SchemaAttributeError):
_ = path.collection
def test_caveat_2_data_type_marker_collides_with_older_collection_names():
"""A directory literally named 'processed' is either the esrf_v2 data_type marker
or, for esrf_v1 (which has no data_type level and always derives 'data_type' as
RAW), an ordinary collection name. Which interpretation wins depends on what
follows it, not on which is semantically intended, because the highest scoring
(most specific) match wins."""
# 'processed_bar' is a valid esrf_v1 '{collection}_{dataset}' directory
# (collection 'processed', dataset 'bar'): esrf_v1 fully identifies the path and
# wins, so 'data_type' comes out as RAW despite the directory name.
path = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"processed",
"processed_bar",
"processed_bar.h5",
)
)
assert path.schema_name == "esrf_v1"
assert path.template_name == "raw_dataset_file"
assert path.collection == "processed"
assert path.dataset == "bar"
assert path.data_type == DataType.RAW
# 'foo' is not a valid esrf_v1 dataset directory for collection 'processed'
# ('processed_foo' would be), so esrf_v1 cannot fully identify this path. esrf_v2
# wins instead: 'processed' becomes the data_type marker and 'foo' the
# collection. The dataset name ('foo_bar.h5') is left unparsed and 'dataset' is
# unavailable.
path = EsrfPath(
make_path(
"visitor", "ma6658", "id21", "20250509", "processed", "foo", "foo_bar.h5"
)
)
assert path.schema_name == "esrf_v2"
assert path.template_name == "processed_collection_path"
assert path.collection == "foo"
assert path.data_type == DataType.PROCESSED
with pytest.raises(errors.SchemaAttributeError):
_ = path.dataset
test_construction.py 0000664 0000000 0000000 00000010424 15244036076 0033466 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf """Tests for constructing `EsrfPath` instances."""
import re
import pytest
from ..._schemas import errors
from ..._schemas.constants import IS_WINDOWS
from ..._schemas.definitions.esrf.esrf_utils import DEFAULT_DATA_ROOT
from ..._schemas.registry import PathSchemaRegistry
from ..utils import make_path
from ..utils import make_realpath
from ..utils import make_unrelated_path
from . import EsrfPath
def test_inherit_from_existing_path():
p1 = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
"foo_bar.h5",
)
)
p2 = EsrfPath(p1)
assert p2._esrf_parsed is p1._esrf_parsed
def test_from_fields():
data_root = make_path("visitor")
with pytest.raises(
errors.PathSchemaMatchError,
match=re.escape(
"No path template can be found that is fully described by the fields {"
),
):
_ = EsrfPath.from_fields(
data_root=data_root,
proposal="ma6658",
beamline="id00",
)
path = EsrfPath.from_fields(
data_root=data_root,
proposal="ma6658",
beamline="id00",
session_date="20250509",
)
assert path.template_name == "raw_proposal_file"
assert path.schema_name == "esrf_v3"
assert path == make_path(
"visitor",
"ma6658",
"id00",
"20250509",
"RAW_DATA",
"ma6658_id00.h5",
)
path = EsrfPath.from_fields(
data_root=data_root,
proposal="ma6658",
beamline="id00",
session_date="20250509",
data_type="processed",
)
assert path.template_name == "processed_proposal_file"
assert path.schema_name == "esrf_v3"
assert path == make_path(
"visitor",
"ma6658",
"id00",
"20250509",
"PROCESSED_DATA",
"ma6658_id00.h5",
)
_CUSTOM_ROOT = make_unrelated_path("custom", "root")
@pytest.mark.parametrize(
"data_root, expected_data_root",
[
(make_path("visitor"), make_path("visitor")),
(str(make_path("visitor")), make_path("visitor")),
(None, DEFAULT_DATA_ROOT),
(_CUSTOM_ROOT, _CUSTOM_ROOT),
],
)
def test_data_root_default_value(data_root, expected_data_root):
if IS_WINDOWS and data_root is None:
with pytest.raises(
errors.PathSchemaMatchError,
match=re.escape(
"No path template can be found that is fully described by the fields {"
),
):
_ = EsrfPath.from_fields(
data_root=data_root,
proposal="ma6658",
beamline="id00",
session_date="20250828",
)
else:
path = EsrfPath.from_fields(
data_root=data_root,
proposal="ma6658",
beamline="id00",
session_date="20250828",
)
assert path.data_root == expected_data_root
def test_strip_mount_point():
realpath = EsrfPath.from_fields(
data_root=make_realpath("visitor"),
proposal="ma6658",
beamline="id00",
session_date="20250509",
collection="sample",
)
path = EsrfPath.from_fields(
data_root=make_path("visitor"),
proposal="ma6658",
beamline="id00",
session_date="20250509",
collection="sample",
)
assert path == realpath.strip_mount_point()
def test_parsed_once(monkeypatch):
"""Constructing a path is expensive: `pathlib` instantiates through
different methods depending on the Python version."""
parsed = []
original = PathSchemaRegistry.parse_path
def parse_path(self, path):
parsed.append(str(path))
return original(self, path)
monkeypatch.setattr(PathSchemaRegistry, "parse_path", parse_path)
dataset_dir = make_path(
"visitor", "ma6658", "id21", "20250509", "RAW_DATA", "foo", "foo_bar"
)
path = EsrfPath(dataset_dir)
assert parsed == [str(dataset_dir)]
del parsed[:]
_ = path.parent
assert len(parsed) == 1
del parsed[:]
_ = path / "foo_bar.h5"
assert len(parsed) == 1
del parsed[:]
_ = EsrfPath(path)
assert not parsed # the parsed state of `path` is inherited
test_fields.py 0000664 0000000 0000000 00000010115 15244036076 0032177 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf """Tests for reading, replacing, and deriving `EsrfPath` field values."""
import re
import pytest
from ..._schemas import errors
from ..utils import make_path
from . import EsrfPath
def test_replace_fields():
path1 = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
"foo_bar.h5",
)
)
path2 = path1.replace_fields(data_type="processed")
assert path2 == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"PROCESSED_DATA",
"foo",
"foo_bar",
"foo_bar.h5",
)
path1 = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
"foo_bar.h5",
)
)
path2 = path1.replace_fields(collection="baz")
assert path2 == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"baz",
"baz_bar",
"baz_bar.h5",
)
path1 = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
)
)
path2 = path1.replace_fields(collection="baz")
assert path2 == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"baz",
"baz_bar",
)
path1 = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
"something",
"unknown.txt",
)
)
path2 = path1.replace_fields(collection="baz")
assert path2 == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"baz",
"baz_bar",
"baz_bar.h5", # appears because we need to make glob pattterns work
)
path2 = path1.replace_fields(collection="baz", template_name=path1.template_name)
assert path2 == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"baz",
"baz_bar",
"something",
"unknown.txt",
)
def test_immutable_schema_fields():
path = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
)
)
with pytest.raises(
errors.SchemaAttributeError,
match=re.escape(
"Attribute 'proposal' is immutable. Create a new path instance with `EsrfPath.replace_fields(proposal='ma6659')`."
),
):
path.proposal = "ma6659"
@pytest.mark.parametrize(
"beamline_raw, beamline_normalized",
[
("id21", "id21"),
("id23eh1", "id23-1"),
],
)
def test_derived_concepts(beamline_raw, beamline_normalized):
path1 = EsrfPath(
make_path("visitor", "ma6658", beamline_raw, "20250509", "RAW_DATA", "foo")
)
path2 = EsrfPath.from_fields(
data_root=make_path("visitor"),
proposal="ma6658",
beamline=beamline_raw,
session_date="20250509",
data_type="raw",
collection="foo",
)
assert path1 == path2.raw_collection_path
assert path2.beamline == beamline_raw
assert path2.beamline_normalized == beamline_normalized
path1 = EsrfPath(
make_path(
"visitor", "ma6658", beamline_raw, "20250509", "RAW_DATA", "foo", "foo_bar"
)
)
path2 = EsrfPath.from_fields(
data_root=make_path("visitor"),
proposal="ma6658",
beamline=beamline_normalized,
session_date="20250509",
data_type="raw",
collection="foo",
dataset="bar",
)
assert path1 == path2.raw_dataset_path
assert path2.beamline == beamline_raw
assert path2.beamline_normalized == beamline_normalized
test_glob.py 0000664 0000000 0000000 00000011135 15244036076 0031657 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf """Tests for globbing `EsrfPath` collections and datasets."""
import pytest
from ..utils import make_path
from . import EsrfPath
def _make_session_with_collections(
tmp_path, collections=("sampleA", "sampleB", "other")
):
for collection in collections:
collection_dir = (
tmp_path
/ "visitor"
/ "blc17255"
/ "id13"
/ "20260715"
/ "RAW_DATA"
/ collection
)
collection_dir.mkdir(parents=True)
(collection_dir / f"blc17255_{collection}.h5").touch()
for dataset in ("0001", "0002"):
dataset_dir = collection_dir / f"{collection}_{dataset}"
dataset_dir.mkdir()
(dataset_dir / f"{collection}_{dataset}.h5").touch()
return EsrfPath(tmp_path / "visitor" / "blc17255" / "id13" / "20260715")
def test_glob_pattern():
path = EsrfPath.from_fields(
data_root=make_path("visitor"),
proposal="ma6658",
beamline="id00",
session_date="20250509",
collection="sample",
)
expected = make_path(
"visitor",
"ma6658",
"id00",
"20250509",
"RAW_DATA",
"sample",
"sample_*",
"sample_*.h5",
)
assert path.replace_fields(dataset="*") == expected
expected = make_path(
"visitor", "ma6658", "id00", "20250509", "RAW_DATA", "sample", "sample_*"
)
assert (
path.replace_fields(dataset="*", template_name="raw_dataset_path") == expected
)
path = EsrfPath.from_fields(
data_root=make_path("visitor"),
proposal="ma6658",
beamline="id00",
session_date="20250509",
collection="sample",
data_type="processed",
)
expected = make_path(
"visitor",
"ma6658",
"id00",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_*",
"sample_*.h5",
)
assert path.replace_fields(dataset="*") == expected
path = EsrfPath.from_fields(
data_root=make_path("visitor"),
proposal="ma6658",
beamline="id00",
session_date="20250509",
)
expected = make_path("visitor", "ma6658", "id00", "*")
assert (
path.replace_fields(session_date="*", template_name="session_path") == expected
)
def test_glob_no_pattern(tmp_path):
session_path = _make_session_with_collections(tmp_path)
# Without a `template_name`, `replace_fields` renders the deepest template
# fully described by the known fields, here `raw_collection_file`.
collection_files = list(session_path.replace_fields(collection="*").glob())
assert sorted(p.collection for p in collection_files) == [
"other",
"sampleA",
"sampleB",
]
assert all(p.template_name == "raw_collection_file" for p in collection_files)
one_collection = next(p for p in collection_files if p.collection == "sampleA")
dataset_files = list(one_collection.replace_fields(dataset="*").glob())
assert sorted(p.dataset for p in dataset_files) == ["0001", "0002"]
assert all(p.collection == "sampleA" for p in dataset_files)
assert all(p.template_name == "raw_dataset_file" for p in dataset_files)
# regular glob(pattern) usage is unaffected
assert list(session_path.glob("RAW_DATA")) == [
session_path / "RAW_DATA",
]
@pytest.mark.parametrize(
"pattern,expected",
[
("*", ["other", "sampleA", "sampleB"]),
("sample*", ["sampleA", "sampleB"]),
("sample?", ["sampleA", "sampleB"]),
("sample[AB]", ["sampleA", "sampleB"]),
("[so]*", ["other", "sampleA", "sampleB"]),
("*A", ["sampleA"]),
],
)
def test_glob_collection_patterns(tmp_path, pattern, expected):
session_path = _make_session_with_collections(tmp_path)
collection_files = list(session_path.replace_fields(collection=pattern).glob())
assert sorted(p.collection for p in collection_files) == expected
assert all(p.template_name == "raw_collection_file" for p in collection_files)
@pytest.mark.parametrize(
"pattern,expected",
[
("*", ["0001", "0002"]),
("000?", ["0001", "0002"]),
("0001", ["0001"]),
("000[12]", ["0001", "0002"]),
],
)
def test_glob_dataset_patterns(tmp_path, pattern, expected):
session_path = _make_session_with_collections(tmp_path)
collection_file = session_path.replace_fields(collection="sampleA")
dataset_files = list(collection_file.replace_fields(dataset=pattern).glob())
assert sorted(p.dataset for p in dataset_files) == expected
assert all(p.collection == "sampleA" for p in dataset_files)
test_parsing.py 0000664 0000000 0000000 00000007715 15244036076 0032410 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf """Tests for parsing paths into `EsrfPath` concept values."""
import datetime
import pathlib
from typing import Dict
import pytest
from ..._schemas import errors
from ..._schemas.definitions.types import DataType
from ..._schemas.registry import ParsedPath
from ..._schemas.registry import get_schema_registry
from ..utils import make_path
from . import EsrfPath
_NOT_ESRF_PATH = make_path("some", "other", "path", "that", "does", "not", "match")
_PATH_TO_CONCEPTS: Dict[pathlib.Path, dict] = {
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
"foo_bar.h5",
): {
"schemas": {"esrf": 3},
"template_name": "raw_dataset_file",
"data_root": make_path("visitor"),
"proposal": "ma6658",
"beamline": "id21",
"session_date": datetime.date(2025, 5, 9),
"data_type": DataType.RAW,
"collection": "foo",
"dataset": "bar",
},
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"raw",
"foo",
"foo_bar",
"foo_bar.h5",
): {
"schemas": {"esrf": 2},
"template_name": "raw_dataset_file",
"data_root": make_path("visitor"),
"proposal": "ma6658",
"beamline": "id21",
"session_date": datetime.date(2025, 5, 9),
"data_type": DataType.RAW,
"collection": "foo",
"dataset": "bar",
},
make_path(
"visitor", "ma6658", "id21", "20250509", "foo", "foo_bar", "foo_bar.h5"
): {
"schemas": {"esrf": 1},
"template_name": "raw_dataset_file",
"data_root": make_path("visitor"),
"proposal": "ma6658",
"beamline": "id21",
"session_date": datetime.date(2025, 5, 9),
"collection": "foo",
"dataset": "bar",
},
_NOT_ESRF_PATH: {
"schemas": {"unknown": 1},
"template_name": "unknown_path",
"unknown_part": _NOT_ESRF_PATH,
},
}
@pytest.mark.parametrize("input_path", _PATH_TO_CONCEPTS.keys())
def test_esrf_path_esrf_schema(input_path):
path = EsrfPath(input_path)
expected_parsed = _expected_parsed_path(_PATH_TO_CONCEPTS[input_path])
assert path._esrf_parsed == expected_parsed
@pytest.mark.parametrize("input_path", _PATH_TO_CONCEPTS.keys())
def test_esrf_path_attributes(input_path):
path = EsrfPath(input_path)
expected_parsed = _expected_parsed_path(_PATH_TO_CONCEPTS[input_path])
if expected_parsed.schema_identifier == "unknown_v1":
expected_to_raise = ()
else:
expected_to_raise = ("thumbnail_file", "note_file", "unknown_path")
# Access concept values
print(path._esrf_parsed.__info__(include_missing=True))
for name in expected_parsed.nonpath_field_names():
actual = getattr(path, name)
expected = getattr(expected_parsed, name)
assert actual == expected, name
# Access paths
for name in expected_parsed.path_field_names():
if name in expected_to_raise:
with pytest.raises(errors.SchemaAttributeError, match=name):
_ = getattr(path, name)
else:
actual = getattr(path, name)
expected = pathlib.Path(getattr(expected_parsed, name))
assert actual == expected, name
# Access non-existing attribute
with pytest.raises(errors.SchemaAttributeError):
_ = path.not_an_attribute
@pytest.mark.parametrize("input_path", _PATH_TO_CONCEPTS.keys())
def test_reconstruct_path(input_path):
path = EsrfPath(input_path)
assert path._esrf_parsed.render() == str(input_path)
def _expected_parsed_path(data: dict) -> ParsedPath:
concept_values = dict(data)
schemas = concept_values.pop("schemas")
template_name = concept_values.pop("template_name")
registry = get_schema_registry(**schemas, fallback_depth=0)
template = registry.get_template(template_name)
return ParsedPath(registry, concept_values, template=template)
test_paths.py 0000664 0000000 0000000 00000012367 15244036076 0032063 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf """Tests for `EsrfPath` derived paths, filenames, and templates."""
import pytest
from ..._schemas import errors
from ..utils import make_path
from . import EsrfPath
def test_raw_dataset_path():
path1 = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"PROCESSED_DATA",
"foo",
"foo_bar",
"process1",
"foo_bar_process1.h5",
)
)
path2 = path1.raw_dataset_path
assert path2 == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
)
with pytest.raises(errors.SchemaAttributeError, match="raw_dataset_path"):
_ = path2.parent.raw_dataset_path
def test_most_specific_template_wins():
"""A path matched by literal segments wins over a path matched by an
optional catch-all segment."""
path = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
"gallery",
)
)
assert path.template_name == "raw_gallery_path"
path = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
"unmapped",
"content",
)
)
assert path.template_name == "raw_dataset_path"
def test_data_type_root_paths():
path = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"PROCESSED_DATA",
"foo",
"foo_bar",
"process1",
"foo_bar_process1.h5",
)
)
assert path.raw_data_path == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
)
assert path.processed_data_path == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"PROCESSED_DATA",
)
assert path.scripts_path == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"SCRIPTS",
)
assert path.nobackup_path == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"NOBACKUP",
)
path = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
)
)
with pytest.raises(errors.SchemaAttributeError, match="raw_data_path"):
_ = path.raw_data_path
def test_filenames():
path = EsrfPath(
make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"PROCESSED_DATA",
"foo",
"foo_bar",
"process1",
"foo_bar_process1.h5",
)
)
assert path.raw_dataset_file == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"foo_bar",
"foo_bar.h5",
)
assert path.raw_collection_file == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
"ma6658_foo.h5",
)
assert path.raw_proposal_file == make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"ma6658_id21.h5",
)
def test_raw_metadata_path():
base_path = make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
)
path = EsrfPath(base_path)
assert path.raw_metadata_path == base_path / "__icat__"
base_path = make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"raw",
)
path = EsrfPath.from_path(base_path, schema_name="esrf", schema_version=2)
assert path.raw_metadata_path == base_path / "__icat__"
base_path = make_path(
"visitor",
"ma6658",
"id21",
"20250509",
)
path = EsrfPath.from_path(base_path, schema_name="esrf", schema_version=1)
assert path.raw_metadata_path == base_path / "__icat__"
def test_raw_metadata_file():
dataset_path = make_path(
"visitor", "ma6658", "id21", "20250509", "RAW_DATA", "foo", "foo_bar"
)
path = EsrfPath(dataset_path)
assert path.raw_metadata_file == path.raw_metadata_path / "foo_bar.xml"
dataset_path = make_path(
"visitor", "ma6658", "id21", "20250509", "raw", "foo", "foo_bar"
)
path = EsrfPath.from_path(dataset_path, schema_name="esrf", schema_version=2)
assert path.raw_metadata_file == path.raw_metadata_path / "foo_bar.xml"
dataset_path = make_path("visitor", "ma6658", "id21", "20250509", "foo", "foo_bar")
path = EsrfPath.from_path(dataset_path, schema_name="esrf", schema_version=1)
assert path.raw_metadata_file == path.raw_metadata_path / "foo_bar.xml"
collection_path = make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"RAW_DATA",
"foo",
)
path = EsrfPath(collection_path)
with pytest.raises(errors.SchemaAttributeError, match="raw_metadata_file"):
_ = path.raw_metadata_file
test_representation.py 0000664 0000000 0000000 00000006034 15244036076 0034000 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf """Tests for `EsrfPath` string, repr, and dir representations."""
from ..utils import make_path
from . import EsrfPath
def test_parent():
native_path = make_path(
"visitor",
"ma6658",
"id21",
"20250509",
"raw",
"foo",
"foo_bar",
"foo_bar.h5",
)
path = EsrfPath(native_path)
assert (
repr(path)
== f""
)
parent = path.parent
native_parent = native_path.parent
assert (
repr(parent)
== f""
)
parent = parent.parent
native_parent = native_parent.parent
assert (
repr(parent)
== f""
)
parent = parent.parent
native_parent = native_parent.parent
assert (
repr(parent)
== f""
)
parent = parent.parent
native_parent = native_parent.parent
assert (
repr(parent)
== f""
)
parent = parent.parent
native_parent = native_parent.parent
assert (
repr(parent)
== f""
)
parent = parent.parent
parent = EsrfPath(parent)
native_parent = native_parent.parent
assert (
repr(parent)
== f""
)
parent = parent.parent
native_parent = native_parent.parent
assert (
repr(parent)
== f""
)
def test_str():
native_path = make_path(
"visitor", "ma6658", "id21", "20250509", "raw", "foo", "foo_bar", "foo_bar.h5"
)
path = EsrfPath(native_path)
assert str(path) == str(native_path)
assert "foo_bar.h5" in str(path)
assert "foo_bar.h5" in str(native_path)
def test_repr():
native_path = make_path(
"visitor", "ma6658", "id21", "20250509", "raw", "foo", "foo_bar", "foo_bar.h5"
)
path = EsrfPath(native_path)
assert (
repr(path)
== f""
)
assert "foo_bar.h5" in repr(path)
assert "foo_bar.h5" in repr(native_path)
def test_dir():
dataset_file = make_path(
"visitor", "ma6658", "id21", "20250509", "raw", "foo", "foo_bar", "foo_bar.h5"
)
path = EsrfPath(dataset_file)
expected = set(path._esrf_parsed.field_names())
assert expected.issubset(set(dir(path)))
def test_dir_without_esrf_schema():
not_esrf_path = make_path("some", "other", "path", "that", "does", "not", "match")
esrf_path = EsrfPath(not_esrf_path)
assert set(dir(not_esrf_path)).issubset(set(dir(esrf_path)))
test_session_date.py 0000664 0000000 0000000 00000004352 15244036076 0033417 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/esrf """Tests for disambiguating the session date from other 8-digit segments."""
import datetime
from pathlib import Path
from ...core import ESRFPath
from ..utils import make_path
def test_session_date():
original_path = make_path(
"data",
"visitor",
"me1783",
"id19",
"20251114", # 8 digits and a valid date
"RAW_DATA",
"20251115", # 8 digits and a valid date
"test_C001H001S0001",
)
path = ESRFPath(original_path)
assert path == original_path
assert path.template_name == "raw_collection_path"
assert path.session_date == datetime.date(2025, 11, 14)
assert path.collection == "20251115"
assert path.unknown_part == Path("test_C001H001S0001")
original_path = make_path(
"data",
"visitor",
"me1783",
"id19",
"20251114", # 8 digits and a valid date
"RAW_DATA",
"14112025", # 8 digits but not a valid date
"test_C001H001S0001",
)
path = ESRFPath(original_path)
assert path == original_path
assert path.template_name == "raw_collection_path"
assert path.session_date == datetime.date(2025, 11, 14)
assert path.collection == "14112025"
assert path.unknown_part == Path("test_C001H001S0001")
original_path = make_path(
"data",
"visitor",
"me1783",
"id19",
"14112025", # 8 digits but not a valid date
"RAW_DATA",
"20251114", # 8 digits and a valid date
"test_C001H001S0001",
)
path = ESRFPath(original_path)
assert path == original_path
assert path.template_name == "session_path"
assert path.proposal == "14112025"
assert path.beamline == "RAW_DATA"
assert path.session_date == datetime.date(2025, 11, 14)
assert path.unknown_part == Path("test_C001H001S0001")
original_path = make_path(
"data",
"visitor",
"me1783",
"id19",
"14112025", # 8 digits but not a valid date
"RAW_DATA",
"14112025", # 8 digits but not a valid date
"test_C001H001S0001",
)
path = ESRFPath(original_path)
assert path == original_path
assert path.template_name == "unknown_path"
assert path.unknown_part == original_path
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/internals/ 0000775 0000000 0000000 00000000000 15244036076 0030441 5 ustar 00root root 0000000 0000000 __init__.py 0000664 0000000 0000000 00000000000 15244036076 0032461 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/internals test_concept.py 0000664 0000000 0000000 00000004565 15244036076 0033440 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/internals from typing import Union
import pytest
from ..._schemas import errors
from ..._schemas.fields.concept import PathConcept
from ..._schemas.identifier import SchemaIdentifier
def test_init_and_properties():
concept = PathConcept("test", "A test concept", SchemaIdentifier(test=1), r"[a-z]+")
assert concept.name == "test"
assert concept.description == "A test concept"
assert concept.schema_identifier == SchemaIdentifier(test=1)
assert concept.regex == r"[a-z]+"
def test_deserialize_success():
concept = PathConcept(
"test", "A test concept", SchemaIdentifier(test=1), r"\d+deg", deserializer=int
)
assert concept.deserialize("42") == 42
def test_serialize_success():
def _serializer_tomo_angle(py_value: Union[str, float]) -> str:
return f"{py_value}deg"
concept = PathConcept(
"test",
"A test concept",
SchemaIdentifier(test=1),
r"\d+deg",
serializer=_serializer_tomo_angle,
)
assert concept.serialize(10) == "10deg"
def test_deserialize_none_returns_none():
concept = PathConcept("test", "A test concept", SchemaIdentifier(test=1), r"\w+")
assert concept.deserialize(None) is None
def test_serialize_none_wildcard():
concept = PathConcept("test", "A test concept", SchemaIdentifier(test=1), r"\w+")
assert concept.serialize(None, raise_on_missing=False) == "*"
def test_serialize_none_raises():
concept = PathConcept("test", "A test concept", SchemaIdentifier(test=1), r"\w+")
with pytest.raises(errors.PathConceptWithoutValue):
concept.serialize(None)
def test_serialize_failure():
def bad_serializer(_):
raise RuntimeError("nope")
concept = PathConcept(
"test",
"A test concept",
SchemaIdentifier(test=1),
r"\w+",
serializer=bad_serializer,
)
with pytest.raises(errors.PathConceptMatchError) as exc:
concept.serialize("bad")
assert "serialization failed" in str(exc.value)
def test_deserialize_failure():
def bad_deserializer(_):
raise RuntimeError("boom")
concept = PathConcept(
"test",
"A test concept",
SchemaIdentifier(test=1),
r"\w+",
deserializer=bad_deserializer,
)
with pytest.raises(errors.PathConceptMatchError) as exc:
concept.deserialize("bad")
assert "deserialization failed" in str(exc.value)
test_definitions.py 0000664 0000000 0000000 00000061715 15244036076 0034320 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/internals import datetime
from pathlib import Path
import pytest
from ..._schemas.definitions import load
from ..._schemas.definitions import types
from ..utils import make_local_path
from ..utils import make_path
@pytest.mark.parametrize(
"name,version",
[
(name, version)
for name, versions in load._ALL_VERSIONS.items()
for version in versions
],
)
def test_info(name, version):
schema = load.get_schema(name, version)
info = schema.__info__()
assert str(schema.identifier) in info
for fields in (schema.concepts, schema.derived_concepts, schema.templates):
for field_name in fields:
assert f"{field_name}:" in info, field_name
_CONCEPT_VALUES = {
"unknown_part": Path("something"),
"data_root": make_path("visitor"),
"proposal": "hg123",
"beamline": "id00",
"session_date": datetime.date(2025, 5, 9),
"collection": "sample",
"dataset": "0001",
"tomo_dataset": "sample_0001",
"data_type": types.DataType.RAW,
"tomo_angle": 90,
"tomo_technique": "phase",
"tomo_type": "stitched",
"tomo_slicing_direction": "xy",
"tomo_slice_number": 10,
"tomo_dtype": "16Bit",
"tomo_result_type": types.TomoResultType.SLICE,
"tomo_extension": "tiff",
"tomo_application": "ewokstomo",
"thumbnail_file_type": "jpg",
"note_type": "txt",
}
@pytest.mark.parametrize(
"name,version",
[
(name, version)
for name, versions in load._ALL_VERSIONS.items()
for version in versions
],
)
def test_concepts(name, version):
schema = load.get_schema(name, version)
concepts = schema.concepts
for cname, concept in concepts.items():
if cname not in _CONCEPT_VALUES:
raise RuntimeError(f"Concept {cname!r} does not have a test value")
py_value = _CONCEPT_VALUES[cname]
# Round-trip
string_value = concept.serialize(py_value)
py_value2 = concept.deserialize(string_value)
assert py_value == py_value2, cname
# None pass-through
assert concept.serialize(None, raise_on_missing=False) == "*", cname
assert concept.deserialize(None) is None, cname
_PATH_VALUES_UNKNOWN_V1 = {
"unknown_path": make_path("visitor", "hg123", "id00"),
}
_PATH_VALUES_ESRF_V1 = {
**_PATH_VALUES_UNKNOWN_V1,
"session_path": make_path("visitor", "hg123", "id00", "20251201"),
"raw_data_path": make_path("visitor", "hg123", "id00", "20251201"),
"raw_proposal_file": make_path(
"visitor", "hg123", "id00", "20251201", "hg123_id00.h5"
),
"raw_collection_path": make_path("visitor", "hg123", "id00", "20251201", "sample"),
"raw_collection_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"sample",
"hg123_sample.h5",
),
"raw_dataset_path": make_path(
"visitor", "hg123", "id00", "20251201", "sample", "sample_0001"
),
"raw_dataset_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"sample",
"sample_0001",
"sample_0001.h5",
),
"raw_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"sample",
"sample_0001",
"gallery",
),
"raw_metadata_path": make_path("visitor", "hg123", "id00", "20251201", "__icat__"),
"raw_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"__icat__",
"sample_0001.xml",
),
}
_PATH_VALUES_ESRF_V2 = {
**_PATH_VALUES_UNKNOWN_V1,
"session_path": make_path("visitor", "hg123", "id00", "20251201"),
"raw_data_path": make_path("visitor", "hg123", "id00", "20251201", "raw"),
"raw_proposal_file": make_path(
"visitor", "hg123", "id00", "20251201", "raw", "hg123_id00.h5"
),
"raw_collection_path": make_path(
"visitor", "hg123", "id00", "20251201", "raw", "sample"
),
"raw_collection_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"raw",
"sample",
"hg123_sample.h5",
),
"raw_dataset_path": make_path(
"visitor", "hg123", "id00", "20251201", "raw", "sample", "sample_0001"
),
"raw_dataset_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"raw",
"sample",
"sample_0001",
"sample_0001.h5",
),
"raw_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"raw",
"sample",
"sample_0001",
"gallery",
),
"raw_metadata_path": make_path(
"visitor", "hg123", "id00", "20251201", "raw", "__icat__"
),
"raw_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"raw",
"__icat__",
"sample_0001.xml",
),
"processed_data_path": make_path(
"visitor", "hg123", "id00", "20251201", "processed"
),
"processed_proposal_file": make_path(
"visitor", "hg123", "id00", "20251201", "processed", "hg123_id00.h5"
),
"processed_collection_path": make_path(
"visitor", "hg123", "id00", "20251201", "processed", "sample"
),
"processed_collection_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"processed",
"sample",
"hg123_sample.h5",
),
"processed_dataset_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"processed",
"sample",
"sample_0001",
),
"processed_dataset_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"processed",
"sample",
"sample_0001",
"sample_0001.h5",
),
"processed_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"processed",
"sample",
"sample_0001",
"gallery",
),
"processed_metadata_path": make_path(
"visitor", "hg123", "id00", "20251201", "processed", "__icat__"
),
"processed_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"processed",
"__icat__",
"sample_0001.xml",
),
"nobackup_path": make_path("visitor", "hg123", "id00", "20251201", "_nobackup"),
"nobackup_proposal_file": make_path(
"visitor", "hg123", "id00", "20251201", "_nobackup", "hg123_id00.h5"
),
"nobackup_collection_path": make_path(
"visitor", "hg123", "id00", "20251201", "_nobackup", "sample"
),
"nobackup_collection_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"_nobackup",
"sample",
"hg123_sample.h5",
),
"nobackup_dataset_path": make_path(
"visitor", "hg123", "id00", "20251201", "_nobackup", "sample", "sample_0001"
),
"nobackup_dataset_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"_nobackup",
"sample",
"sample_0001",
"sample_0001.h5",
),
"nobackup_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"_nobackup",
"sample",
"sample_0001",
"gallery",
),
"nobackup_metadata_path": make_path(
"visitor", "hg123", "id00", "20251201", "_nobackup", "__icat__"
),
"nobackup_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"_nobackup",
"__icat__",
"sample_0001.xml",
),
"scripts_path": make_path("visitor", "hg123", "id00", "20251201", "SCRIPTS"),
"gallery_path": make_path("visitor", "hg123", "id00", "20251201", "GALLERY"),
"notes_path": make_path("visitor", "hg123", "id00", "20251201", "NOTES"),
}
_PATH_VALUES_ESRF_V3 = {
**_PATH_VALUES_UNKNOWN_V1,
"session_path": make_path("visitor", "hg123", "id00", "20251201"),
"raw_data_path": make_path("visitor", "hg123", "id00", "20251201", "RAW_DATA"),
"raw_proposal_file": make_path(
"visitor", "hg123", "id00", "20251201", "RAW_DATA", "hg123_id00.h5"
),
"raw_collection_path": make_path(
"visitor", "hg123", "id00", "20251201", "RAW_DATA", "sample"
),
"raw_collection_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"RAW_DATA",
"sample",
"hg123_sample.h5",
),
"raw_dataset_path": make_path(
"visitor", "hg123", "id00", "20251201", "RAW_DATA", "sample", "sample_0001"
),
"raw_dataset_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"RAW_DATA",
"sample",
"sample_0001",
"sample_0001.h5",
),
"raw_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"RAW_DATA",
"sample",
"sample_0001",
"gallery",
),
"raw_metadata_path": make_path(
"visitor", "hg123", "id00", "20251201", "RAW_DATA", "__icat__"
),
"raw_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"RAW_DATA",
"__icat__",
"sample_0001.xml",
),
"processed_data_path": make_path(
"visitor", "hg123", "id00", "20251201", "PROCESSED_DATA"
),
"processed_proposal_file": make_path(
"visitor", "hg123", "id00", "20251201", "PROCESSED_DATA", "hg123_id00.h5"
),
"processed_collection_path": make_path(
"visitor", "hg123", "id00", "20251201", "PROCESSED_DATA", "sample"
),
"processed_collection_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"hg123_sample.h5",
),
"processed_dataset_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
),
"processed_dataset_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"sample_0001.h5",
),
"processed_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"gallery",
),
"processed_metadata_path": make_path(
"visitor", "hg123", "id00", "20251201", "PROCESSED_DATA", "__icat__"
),
"processed_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"__icat__",
"sample_0001.xml",
),
"nobackup_path": make_path("visitor", "hg123", "id00", "20251201", "NOBACKUP"),
"nobackup_proposal_file": make_path(
"visitor", "hg123", "id00", "20251201", "NOBACKUP", "hg123_id00.h5"
),
"nobackup_collection_path": make_path(
"visitor", "hg123", "id00", "20251201", "NOBACKUP", "sample"
),
"nobackup_collection_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"NOBACKUP",
"sample",
"hg123_sample.h5",
),
"nobackup_dataset_path": make_path(
"visitor", "hg123", "id00", "20251201", "NOBACKUP", "sample", "sample_0001"
),
"nobackup_dataset_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"NOBACKUP",
"sample",
"sample_0001",
"sample_0001.h5",
),
"nobackup_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"NOBACKUP",
"sample",
"sample_0001",
"gallery",
),
"nobackup_metadata_path": make_path(
"visitor", "hg123", "id00", "20251201", "NOBACKUP", "__icat__"
),
"nobackup_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"NOBACKUP",
"__icat__",
"sample_0001.xml",
),
"scripts_path": make_path("visitor", "hg123", "id00", "20251201", "SCRIPTS"),
"gallery_path": make_path("visitor", "hg123", "id00", "20251201", "GALLERY"),
"notes_path": make_path("visitor", "hg123", "id00", "20251201", "NOTES"),
}
_PATH_VALUES_ESRF_V4 = {
**_PATH_VALUES_ESRF_V3,
"auxiliary_data_path": make_path(
"visitor", "hg123", "id00", "20251201", "AUXILIARY_DATA"
),
}
_PATH_VALUES_TOMO_V1 = {
**_PATH_VALUES_ESRF_V3,
"references_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"references",
),
"references_dark": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"references",
"sample_0001_darks.hdf5",
),
"references_flat": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"references",
"sample_0001_flats.hdf5",
),
"projections_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"projections",
),
"projections_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"projections",
"sample_0001.nx",
),
"projections_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"projections",
"gallery",
),
"projections_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"projections",
"gallery",
"metadata.json",
),
"projections_thumbnail": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"projections",
"gallery",
"sample_0001_90.00deg_large.jpg",
),
"projections_preview": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"projections",
"gallery",
"sample_0001_90.00deg.jpg",
),
"slices_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"slices",
),
"slices_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"slices",
"sample_0001_phase_xy_00001.tiff",
),
"slices_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"slices",
"gallery",
),
"slices_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"slices",
"gallery",
"metadata.json",
),
"slices_thumbnail": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"slices",
"gallery",
"sample_0001_phase_xy_00001_large.jpg",
),
"slices_preview": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"slices",
"gallery",
"sample_0001_phase_xy_00001.jpg",
),
"volumes_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes",
),
"volumes_technique_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes",
"sample_0001_phase_16Bit_vol",
),
"volumes_slice_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes",
"sample_0001_phase_16Bit_vol",
"sample_0001_00001.vol",
),
"volumes_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes",
"sample_0001_phase_16Bit_vol",
"gallery",
),
"volumes_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes",
"sample_0001_phase_16Bit_vol",
"gallery",
"metadata.json",
),
"volumes_thumbnail": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes",
"sample_0001_phase_16Bit_vol",
"gallery",
"sample_0001_phase_xy_00001_large.jpg",
),
"volumes_preview": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes",
"sample_0001_phase_16Bit_vol",
"gallery",
"sample_0001_phase_xy_00001.jpg",
),
"volumes_custom_type_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes_stitched",
),
"volumes_custom_type_technique_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes_stitched",
"sample_0001_phase_stitched_16Bit_vol",
),
"volumes_custom_type_slice_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes_stitched",
"sample_0001_phase_stitched_16Bit_vol",
"sample_0001_00001.vol",
),
"volumes_custom_type_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes_stitched",
"sample_0001_phase_stitched_16Bit_vol",
"gallery",
),
"volumes_custom_type_metadata_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes_stitched",
"sample_0001_phase_stitched_16Bit_vol",
"gallery",
"metadata.json",
),
"volumes_custom_type_thumbnail": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes_stitched",
"sample_0001_phase_stitched_16Bit_vol",
"gallery",
"sample_0001_phase_stitched_xy_00001_large.jpg",
),
"volumes_custom_type_preview": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes_stitched",
"sample_0001_phase_stitched_16Bit_vol",
"gallery",
"sample_0001_phase_stitched_xy_00001.jpg",
),
"tomo_log_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"logs",
),
"tomo_log_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"logs",
"sample_0001_ewokstomo.log",
),
"workflows_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"workflows",
),
"workflows_gallery_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"workflows",
"gallery",
),
"workflows_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"sample_0001",
"workflows",
"gallery",
"sample_0001.json",
),
"slices_link_collection_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"SLICES",
),
"slices_link_root_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"SLICES",
),
"slices_link_collection_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"SLICES",
"sample_0001_phase_xy_00001.tiff",
),
"slices_link_root_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"SLICES",
"sample_0001_phase_xy_00001.tiff",
),
"volumes_link_collection_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"VOLUMES",
),
"volumes_link_root_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"VOLUMES",
),
"volumes_link_collection_technique_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"VOLUMES",
"sample_0001_phase_16Bit_vol",
),
"volumes_link_root_technique_path": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"VOLUMES",
"sample_0001_phase_16Bit_vol",
),
"volumes_link_collection_slice_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"sample",
"VOLUMES",
"sample_0001_phase_16Bit_vol",
"sample_0001_00001.vol",
),
"volumes_link_root_slice_file": make_path(
"visitor",
"hg123",
"id00",
"20251201",
"PROCESSED_DATA",
"VOLUMES",
"sample_0001_phase_16Bit_vol",
"sample_0001_00001.vol",
),
}
_PATH_VALUES_TOMO_V2 = {
**_PATH_VALUES_ESRF_V4,
**{
tname: tpath
for tname, tpath in _PATH_VALUES_TOMO_V1.items()
if tname not in _PATH_VALUES_ESRF_V3
},
}
# The `tomoproc_v1` templates describe the same dataset results as the
# `tomo_v1` templates but rooted at an arbitrary directory instead of the
# dataset directory inside an ESRF session directory.
_TOMO_DATASET_DIR = make_path(
"visitor", "hg123", "id00", "20251201", "PROCESSED_DATA", "sample", "sample_0001"
)
_LOCAL_DATASET_DIR = make_local_path("sample_0001")
_PATH_VALUES_TOMOPROC_V1 = {
**_PATH_VALUES_UNKNOWN_V1,
"dataset_path": _LOCAL_DATASET_DIR,
**{
tname: _LOCAL_DATASET_DIR / tpath.relative_to(_TOMO_DATASET_DIR)
for tname, tpath in _PATH_VALUES_TOMO_V1.items()
if tname not in _PATH_VALUES_ESRF_V3 and _TOMO_DATASET_DIR in tpath.parents
},
}
_PATH_VALUES = {
"unknown_v1": _PATH_VALUES_UNKNOWN_V1,
"esrf_v1": _PATH_VALUES_ESRF_V1,
"esrf_v2": _PATH_VALUES_ESRF_V2,
"esrf_v3": _PATH_VALUES_ESRF_V3,
"esrf_v4": _PATH_VALUES_ESRF_V4,
"tomo_v1": _PATH_VALUES_TOMO_V1,
"tomo_v2": _PATH_VALUES_TOMO_V2,
"tomoproc_v1": _PATH_VALUES_TOMOPROC_V1,
}
@pytest.mark.parametrize(
"name,version",
[
(name, version)
for name, versions in load._ALL_VERSIONS.items()
for version in versions
],
)
def test_path_templates(name, version):
schema = load.get_schema(name, version)
key = str(schema.identifier)
if key not in _PATH_VALUES:
raise RuntimeError(f"Schema {key!r} does not have test values")
paths = _PATH_VALUES[key]
for tname, template in schema.templates.items():
if tname not in paths:
raise RuntimeError(
f"Template {tname!r} does not have a test value for schema '{schema.identifier}'"
)
path = str(paths[tname])
# Round-trip
concept_values = template.parse(path)
path2 = template.render(concept_values)
assert path == path2, tname
test_deprecated_fields.py 0000664 0000000 0000000 00000014535 15244036076 0035431 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/internals import pytest
from ..._schemas import errors
from ..._schemas.registry import PathSchemaRegistry
from ..._schemas.schema.merged import MergedPathSchema
from ..._schemas.schema.path import PathSchema
def _make_schema(name="deprschema", version=1):
"""Schema with deprecated fields."""
return PathSchema(
name=name,
version=version,
description="Schema with deprecated fields.\n\nSchema with deprecated fields.",
concepts={
"new_concept": {
"description": "A concept",
"regex": r"[a-z]+",
"deprecated_names": ["old_concept", "older_concept"],
},
},
derived_concepts={
"new_derived": {
"description": "A derived concept",
"template": "{new_concept}_suffix",
"deprecated_names": ["old_derived"],
},
},
template_tree={
(
"{new_concept}.txt",
"new_template",
True,
("old_template", "older_template"),
): None,
},
)
def test_schema_exposes_deprecated_names():
schema = _make_schema()
assert schema.deprecated_concepts == {
"old_concept": "new_concept",
"older_concept": "new_concept",
}
assert schema.deprecated_derived_concepts == {"old_derived": "new_derived"}
assert schema.deprecated_templates == {
"old_template": "new_template",
"older_template": "new_template",
}
# Canonical names are not affected
assert list(schema.concepts) == ["new_concept"]
assert list(schema.derived_concepts) == ["new_derived"]
assert list(schema.templates) == ["new_template"]
def test_deprecated_name_colliding_with_existing_field_raises():
with pytest.raises(errors.FieldNameError):
PathSchema(
name="collide1",
version=1,
description="d\n\nd",
concepts={
"a": {"description": "a", "regex": "."},
"b": {"description": "b", "regex": ".", "deprecated_names": ["a"]},
},
)
def test_two_fields_cannot_share_a_deprecated_name():
with pytest.raises(errors.FieldNameError):
PathSchema(
name="collide2",
version=1,
description="d\n\nd",
concepts={
"a": {"description": "a", "regex": ".", "deprecated_names": ["x"]},
"b": {"description": "b", "regex": ".", "deprecated_names": ["x"]},
},
)
def test_registry_resolves_deprecated_concept_and_warns():
registry = PathSchemaRegistry([_make_schema()])
assert registry.concept_names() == ["new_concept"]
assert set(registry.deprecated_concept_names()) == {"old_concept", "older_concept"}
with pytest.deprecated_call():
concept = registry.get_concept("old_concept")
assert concept.name == "new_concept"
with pytest.deprecated_call():
concept = registry.get_concept("older_concept")
assert concept.name == "new_concept"
# Canonical name resolves without a warning
assert registry.get_concept("new_concept").name == "new_concept"
with pytest.raises(errors.UnknownPathConcept):
registry.get_concept("not_a_concept")
def test_registry_resolves_deprecated_derived_concept_and_warns():
registry = PathSchemaRegistry([_make_schema()])
with pytest.deprecated_call():
derived = registry.get_derived_concept("old_derived")
assert derived.name == "new_derived"
with pytest.raises(errors.UnknownPathConcept):
registry.get_derived_concept("not_a_derived_concept")
def test_registry_resolves_deprecated_template_and_warns():
registry = PathSchemaRegistry([_make_schema()])
assert registry.template_names() == ["new_template"]
assert set(registry.deprecated_template_names()) == {
"old_template",
"older_template",
}
with pytest.deprecated_call():
template = registry.get_template("old_template")
assert template.name == "new_template"
with pytest.raises(errors.UnknownPathTemplate):
registry.get_template("not_a_template")
def test_render_and_parse_with_deprecated_template_name():
registry = PathSchemaRegistry([_make_schema()])
with pytest.deprecated_call():
path = registry.render_path(
{"new_concept": "abc"}, template_name="old_template"
)
assert path == "abc.txt"
parsed = registry.parse_path(path)
assert parsed.template_name == "new_template"
with pytest.deprecated_call():
rendered_again = parsed.render(template_name="older_template")
assert rendered_again == "abc.txt"
def test_render_with_deprecated_concept_kwarg():
"""A deprecated concept name used as a keyword argument (as done by
`ESRFPath.from_fields`/`replace_fields`) is translated to the canonical
concept name before rendering."""
registry = PathSchemaRegistry([_make_schema()])
with pytest.deprecated_call():
path = registry.render_path({"old_concept": "abc"})
assert path == "abc.txt"
parsed = registry.parse_path(path)
with pytest.deprecated_call():
replaced = parsed.replace_concepts(old_concept="xyz")
assert replaced.render() == "xyz.txt"
def test_parsed_path_attribute_access_with_deprecated_names():
registry = PathSchemaRegistry([_make_schema()])
parsed = registry.parse_path("abc.txt")
assert parsed.new_concept == "abc"
with pytest.deprecated_call():
assert parsed.old_concept == "abc"
with pytest.deprecated_call():
assert parsed.old_derived == "abc_suffix"
with pytest.deprecated_call():
assert parsed.old_template == "abc.txt"
def test_merged_schema_keeps_deprecated_names_from_first_schema():
"""Same-named deprecated aliases colliding across merged schemas are
dropped (like colliding canonical names) instead of raising."""
schema_a = _make_schema(name="mergea")
schema_b = PathSchema(
name="mergeb",
version=1,
description="d\n\nd",
concepts={
"unrelated": {
"description": "unrelated",
"regex": r"[a-z]+",
"deprecated_names": ["old_concept"], # collides with schema_a
},
},
)
merged = MergedPathSchema([schema_a, schema_b])
assert merged.deprecated_concepts["old_concept"] == "new_concept"
test_derived.py 0000664 0000000 0000000 00000003454 15244036076 0033423 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/internals import pytest
from ..._schemas import errors
from ..._schemas.fields.concept import PathConcept
from ..._schemas.fields.derived import DerivedConcept
from ..._schemas.fields.derived import template_derive_func
from ..._schemas.identifier import SchemaIdentifier
def test_init_and_properties():
def derive_func(d):
return None
concept = DerivedConcept(
"test", "A test concept", SchemaIdentifier(test=1), derive_func, []
)
assert concept.name == "test"
assert concept.description == "A test concept"
assert concept.schema_identifier == SchemaIdentifier(test=1)
def test_derive():
def derive_func(d) -> str:
return d["first"] + " " + d["last"]
concept = DerivedConcept(
"test",
"A test concept",
SchemaIdentifier(test=1),
derive_func,
["first", "last"],
)
assert concept.derive({"first": "John", "last": "Doe", "unused": 11}) == "John Doe"
def test_derive_from_template():
identifier = SchemaIdentifier(test=1)
concepts = {
"first": PathConcept("first", "First name", identifier, regex=r"\w+"),
"last": PathConcept("last", "Last name", identifier, regex=r"\w+"),
}
derive_func, derived_from = template_derive_func("{first}_{last}", concepts)
concept = DerivedConcept(
"test",
"A test concept",
identifier,
derive_func,
derived_from,
template="{first}_{last}",
)
assert derived_from == ["first", "last"]
assert concept.derive({"first": "John", "last": "Doe", "unused": 11}) == "John_Doe"
assert concept.annotation() is str
assert "{first}_{last}" in concept.docstring
def test_derive_from_template_unknown_concept():
with pytest.raises(errors.UnknownPathConcept):
template_derive_func("{first}_{last}", {})
test_path_schema.py 0000664 0000000 0000000 00000020744 15244036076 0034256 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/internals import pytest
from ..._schemas import errors
from ..._schemas.definitions.esrf import esrf_v3
from ..._schemas.definitions.tomo import tomo_v1
from ..._schemas.definitions.tomo import tomoproc_v1
from ..._schemas.registry import get_schema_registry
from ..._schemas.schema.merged import MergedPathSchema
from ..._schemas.schema.path import PathSchema
from ..utils import make_local_path
from ..utils import make_path
def test_merged_schema_shadowed_templates():
"""Schemas can define path templates with the same name. Name-based access
resolves to the first schema while path matching considers all of them."""
tomo = tomo_v1.get_schema()
tomoproc = tomoproc_v1.get_schema()
merged = MergedPathSchema([tomo, tomoproc])
# Name-based access resolves to the first schema
assert merged.templates["slices_file"] is tomo.templates["slices_file"]
# Path matching considers all templates
all_templates = list(merged.iter_templates(include_shadowed=True))
assert tomo.templates["slices_file"] in all_templates
assert tomoproc.templates["slices_file"] in all_templates
# Reversing the order reverses the name resolution
merged = MergedPathSchema([tomoproc, tomo])
assert merged.templates["slices_file"] is tomoproc.templates["slices_file"]
def test_merged_schema_resolve_schema():
tomo = tomo_v1.get_schema()
tomoproc = tomoproc_v1.get_schema()
merged = MergedPathSchema([tomo, tomoproc])
assert merged.resolve_schema("tomo_v1") is tomo
assert merged.resolve_schema("tomoproc_v1") is tomoproc
assert merged.resolve_schema("esrf_v3") is None
assert merged.resolve_schema("not_a_schema") is None
def test_merged_schema_concept_collision():
"""Same-named concepts must have the same definition."""
concepts = {
"myconcept": {"description": "Some concept", "regex": r"[a-z]+"},
}
schema_a = PathSchema(
name="schemaa",
version=1,
description="Schema A",
concepts=concepts,
template_tree={("{myconcept}", "mypath"): None},
)
schema_b = PathSchema(
name="schemab",
version=1,
description="Schema B",
concepts=concepts,
template_tree={("prefix", "otherpath"): {"{myconcept}": None}},
)
# Same definition: allowed although the concept instances differ
assert schema_a.concepts["myconcept"] is not schema_b.concepts["myconcept"]
merged = MergedPathSchema([schema_a, schema_b])
assert merged.concepts["myconcept"] is schema_a.concepts["myconcept"]
# Different definition: not allowed
schema_c = PathSchema(
name="schemac",
version=1,
description="Schema C",
concepts={"myconcept": {"description": "Some concept", "regex": r"\d+"}},
template_tree={("{myconcept}", "thirdpath"): None},
)
with pytest.raises(errors.FieldNameError):
MergedPathSchema([schema_a, schema_c])
def test_merged_schema_concept_and_derived_concept():
"""A concept of one schema can be a derived concept of another schema."""
tomo = tomo_v1.get_schema()
tomoproc = tomoproc_v1.get_schema()
assert "tomo_dataset" in tomo.derived_concepts
assert "tomo_dataset" in tomoproc.concepts
for schemas in ([tomo, tomoproc], [tomoproc, tomo]):
merged = MergedPathSchema(schemas)
assert merged.concepts["tomo_dataset"] is tomoproc.concepts["tomo_dataset"]
assert (
merged.derived_concepts["tomo_dataset"]
is tomo.derived_concepts["tomo_dataset"]
)
def test_extending_schema_cannot_redefine_derived_concept():
"""Concepts and derived concepts with the same name are only supported when
merging schemas in a registry, not when extending a schema."""
with pytest.raises(errors.FieldNameError):
PathSchema(
name="mytomo",
version=1,
description="Schema redefining a derived concept as a concept",
concepts={
"tomo_dataset": {"description": "Dataset", "regex": r"[a-z0-9_]+"}
},
extends=[tomo_v1.get_schema()],
)
def test_derived_concept_template():
"""Placeholders of a derived concept with a template are replaced by the
template fragment in the path templates."""
schema = PathSchema(
name="schemad",
version=1,
description="Schema D",
concepts={
"first": {"description": "First name", "regex": r"[a-z]+"},
"last": {"description": "Last name", "regex": r"[a-z]+"},
},
derived_concepts={
"fullname": {"description": "Full name", "template": "{first}_{last}"}
},
template_tree={("{fullname}", "mypath"): None},
)
template = schema.templates["mypath"]
assert [segment.template for segment in template._segments] == ["{first}_{last}"]
concept_values = template.parse("john_doe")
assert concept_values == {"first": "john", "last": "doe"}
assert schema.derived_concepts["fullname"].derive(concept_values) == "john_doe"
def test_derived_concept_template_unknown_concept():
with pytest.raises(errors.UnknownPathConcept):
PathSchema(
name="schemae",
version=1,
description="Schema E",
derived_concepts={
"fullname": {"description": "Full name", "template": "{first}_{last}"}
},
)
def test_registry_resolves_template_per_schema():
registry = get_schema_registry(tomo=1, tomoproc=1, fallback_depth=0)
tomo_template = registry.get_template("slices_file", "tomo_v1")
tomoproc_template = registry.get_template("slices_file", "tomoproc_v1")
assert tomo_template is tomo_v1.get_schema().templates["slices_file"]
assert tomoproc_template is tomoproc_v1.get_schema().templates["slices_file"]
# Without a schema identifier the first schema wins
assert registry.get_template("slices_file") is tomo_template
# Identifiers that do not resolve to a component fall back to the merged pool
assert registry.get_template("slices_file", "esrf_v3") is tomo_template
def test_registry_parses_across_schemas():
registry = get_schema_registry(tomo=1, tomoproc=1, fallback_depth=0)
esrf_path = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
"slices",
"sample_0001_phase_xy_00001.tiff",
)
parsed = registry.parse_path(esrf_path)
assert str(parsed.schema_identifier) == "tomo_v1"
assert parsed.template_name == "slices_file"
local_path = make_local_path(
"sample_0001", "slices", "sample_0001_phase_xy_00001.tiff"
)
parsed = registry.parse_path(local_path)
assert str(parsed.schema_identifier) == "tomoproc_v1"
assert parsed.template_name == "slices_file"
def test_registry_order_matters():
registry1 = get_schema_registry(tomo=1, tomoproc=1, fallback_depth=0)
registry2 = get_schema_registry(tomoproc=1, tomo=1, fallback_depth=0)
assert registry1 is not registry2
assert (
registry1.get_template("slices_file")
is tomo_v1.get_schema().templates["slices_file"]
)
assert (
registry2.get_template("slices_file")
is tomoproc_v1.get_schema().templates["slices_file"]
)
def test_extending_schema_cannot_shadow_templates():
"""Shadowing same-named templates is only supported when merging schemas
in a registry, not when extending a schema."""
with pytest.raises(errors.FieldNameError):
PathSchema(
name="mytomo",
version=1,
description="Schema with conflicting template names",
template_tree={
"{data_root}": {
"slices": {
(
"{tomo_dataset}_{tomo_technique}_{tomo_slicing_direction}_{tomo_slice_number}.{tomo_extension}",
"slices_file",
): None,
},
},
},
extends=[tomoproc_v1.get_schema()],
)
def test_single_schema_registry_resolves_own_templates():
"""A registry backed by a single schema resolves `get_template` to that
schema's own template objects, with or without an explicit schema
identifier."""
registry = get_schema_registry(esrf=3, fallback_depth=0)
template = registry.get_template("raw_dataset_file", "esrf_v3")
assert template is esrf_v3.get_schema().templates["raw_dataset_file"]
assert registry.get_template("raw_dataset_file") is template
test_path_template.py 0000664 0000000 0000000 00000034142 15244036076 0034626 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/internals import os
from pathlib import Path
import pytest
from ..._schemas import errors
from ..._schemas.fields.concept import PathConcept
from ..._schemas.fields.path import PathTemplate
from ..._schemas.fields.tree import parse_template_tree
from ..._schemas.identifier import SchemaIdentifier
def _os_path(*segments: str) -> str:
"""Rendered path for the current OS."""
return str(Path(*segments))
@pytest.fixture()
def path_tree():
schema_identifier = SchemaIdentifier(test=1)
template_tree = {
("{data_root}", "root_path"): {
"{experiment_id}": {
("{unknown_part}", "experiment_path", False): None,
("{optional_subdir1}", "optional1_path", False): {
("fileA.txt", "file_patha"): None,
("fileB.txt", "file_pathb"): None,
("{optional_subdir2}", None, False): {
("fileC.txt", "file_pathc"): None,
("fileD.txt", "file_pathd"): None,
},
},
("{data_id=data2}", "mandatory2_path"): {
("{file_name}.txt", "file_pathe"): None,
},
}
}
}
concepts = {
"unknown_part": PathConcept(
name="unknown_part",
description="Unknown path or path segment (placeholder for unmapped directories or files)",
schema_identifier=schema_identifier,
regex=r".+",
score=0,
),
"data_root": PathConcept(
name="data_root",
description="Top-level data directory",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"experiment_id": PathConcept(
name="experiment_id",
description="Experiment or proposal identifier",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"data_id": PathConcept(
name="data_id",
description="Data identifier",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"optional_subdir1": PathConcept(
name="optional_subdir1",
description="Optional subdirectory for processed data",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"optional_subdir2": PathConcept(
name="optional_subdir2",
description="Optional subdirectory for processed data",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"file_name": PathConcept(
name="file_name",
description="Base name of a generated file",
schema_identifier=schema_identifier,
regex=r"\w+",
),
}
return schema_identifier, template_tree, concepts
def test_init_and_str(path_tree):
schema_identifier, template_tree, concepts = path_tree
templates = parse_template_tree(template_tree)
reprs = {}
for args in templates:
pt = PathTemplate(args.name, schema_identifier, args.segments, concepts)
reprs[args.name] = str(pt)
expected = {
"root_path": "",
"experiment_path": "",
"mandatory2_path": "",
"optional1_path": "",
"file_patha": "",
"file_pathb": "",
"file_pathc": "",
"file_pathd": "",
"file_pathe": "",
}
assert reprs == expected
def test_render(path_tree):
schema_identifier, template_tree, concepts = path_tree
concept_values = {
"data_root": "DATA",
"experiment_id": "EXP001",
"optional_subdir1": "data1",
"file_name": "summary",
}
templates = parse_template_tree(template_tree)
paths = {}
for args in templates:
pt = PathTemplate(args.name, schema_identifier, args.segments, concepts)
paths[args.name] = pt.render(concept_values)
expected = {
"root_path": "DATA",
"experiment_path": _os_path("DATA", "EXP001"),
"mandatory2_path": _os_path("DATA", "EXP001", "data2"),
"optional1_path": _os_path("DATA", "EXP001", "data1"),
"file_patha": _os_path("DATA", "EXP001", "data1", "fileA.txt"),
"file_pathb": _os_path("DATA", "EXP001", "data1", "fileB.txt"),
"file_pathc": _os_path("DATA", "EXP001", "data1", "fileC.txt"),
"file_pathd": _os_path("DATA", "EXP001", "data1", "fileD.txt"),
"file_pathe": _os_path("DATA", "EXP001", "data2", "summary.txt"),
}
assert paths == expected
def test_render_missing_literal(path_tree):
schema_identifier, template_tree, concepts = path_tree
concept_values = {
"data_root": "DATA",
"experiment_id": "EXP001",
"optional_subdir1": "data1",
"file_name": "summary",
}
templates = parse_template_tree(template_tree)
paths = {}
for args in templates:
pt = PathTemplate(args.name, schema_identifier, args.segments, concepts)
paths[args.name] = pt.render(concept_values)
expected = {
"root_path": "DATA",
"experiment_path": _os_path("DATA", "EXP001"),
"mandatory2_path": _os_path("DATA", "EXP001", "data2"),
"optional1_path": _os_path("DATA", "EXP001", "data1"),
"file_patha": _os_path("DATA", "EXP001", "data1", "fileA.txt"),
"file_pathb": _os_path("DATA", "EXP001", "data1", "fileB.txt"),
"file_pathc": _os_path("DATA", "EXP001", "data1", "fileC.txt"),
"file_pathd": _os_path("DATA", "EXP001", "data1", "fileD.txt"),
"file_pathe": _os_path("DATA", "EXP001", "data2", "summary.txt"),
}
assert paths == expected
def test_render_missing(path_tree):
schema_identifier, template_tree, concepts = path_tree
concept_values = {
"data_root": "DATA",
"experiment_id": "EXP001",
"optional_subdir1": "data1",
}
templates = parse_template_tree(template_tree)
paths = {}
for args in templates:
pt = PathTemplate(args.name, schema_identifier, args.segments, concepts)
if args.name == "file_pathe":
with pytest.raises(
errors.PathConceptWithoutValue, match="'file_name' is not defined"
):
pt.render(concept_values)
else:
paths[args.name] = pt.render(concept_values)
expected = {
"root_path": "DATA",
"experiment_path": _os_path("DATA", "EXP001"),
"mandatory2_path": _os_path("DATA", "EXP001", "data2"),
"optional1_path": _os_path("DATA", "EXP001", "data1"),
"file_patha": _os_path("DATA", "EXP001", "data1", "fileA.txt"),
"file_pathb": _os_path("DATA", "EXP001", "data1", "fileB.txt"),
"file_pathc": _os_path("DATA", "EXP001", "data1", "fileC.txt"),
"file_pathd": _os_path("DATA", "EXP001", "data1", "fileD.txt"),
}
assert paths == expected
def test_render_missing_wildcard(path_tree):
schema_identifier, template_tree, concepts = path_tree
concept_values = {
"data_root": "DATA",
"optional_subdir1": "data1",
}
templates = parse_template_tree(template_tree)
paths = {}
for args in templates:
pt = PathTemplate(args.name, schema_identifier, args.segments, concepts)
paths[args.name] = pt.render(concept_values, raise_on_missing=False)
expected = {
"root_path": "DATA",
"experiment_path": _os_path("DATA", "*"),
"mandatory2_path": _os_path("DATA", "*", "data2"),
"optional1_path": _os_path("DATA", "*", "data1"),
"file_patha": _os_path("DATA", "*", "data1", "fileA.txt"),
"file_pathb": _os_path("DATA", "*", "data1", "fileB.txt"),
"file_pathc": _os_path("DATA", "*", "data1", "fileC.txt"),
"file_pathd": _os_path("DATA", "*", "data1", "fileD.txt"),
"file_pathe": _os_path("DATA", "*", "data2", "*.txt"),
}
assert paths == expected
@pytest.fixture()
def simple_path_tree():
schema_identifier = SchemaIdentifier(test=1)
template_tree = {
("{root}", "root_path"): {
"{experiment_id}": {
("{data_id=data2}", "data_path"): {
("{file_name}.txt", "file_path"): None,
},
}
}
}
concepts = {
"root": PathConcept(
name="root",
description="root folder",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"experiment_id": PathConcept(
name="experiment_id",
description="Experiment ID",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"data_id": PathConcept(
name="data_id",
description="Data ID",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"file_name": PathConcept(
name="file_name",
description="Filename",
schema_identifier=schema_identifier,
regex=r"\w+",
),
}
return schema_identifier, template_tree, concepts
def test_parse_basic_match(simple_path_tree):
schema_identifier, template_tree, concepts = simple_path_tree
templates = parse_template_tree(template_tree)
pt = PathTemplate("file_path", schema_identifier, templates[-1].segments, concepts)
path = pt.render(
{
"root": "DATA",
"experiment_id": "EXP001",
"data_id": "data2",
"file_name": "summary",
}
)
parsed = pt.parse(path)
assert parsed == {
"root": "DATA",
"experiment_id": "EXP001",
"data_id": "data2",
"file_name": "summary",
}
def test_parse_fails_on_mismatch(simple_path_tree):
schema_identifier, template_tree, concepts = simple_path_tree
templates = parse_template_tree(template_tree)
pt = PathTemplate("file_path", schema_identifier, templates[-1].segments, concepts)
bad_path = _os_path("DATA", "WRONG", "data3", "summary.txt")
with pytest.raises(errors.PathTemplateMatchError):
pt.parse(bad_path)
def test_parse_respects_literals(simple_path_tree):
schema_identifier, template_tree, concepts = simple_path_tree
templates = parse_template_tree(template_tree)
pt = PathTemplate("data_path", schema_identifier, templates[-2].segments, concepts)
path = _os_path("ROOT", "EXP001", "data2")
parsed = pt.parse(path)
assert parsed["data_id"] == "data2"
def test_render_and_parse_roundtrip(simple_path_tree):
schema_identifier, template_tree, concepts = simple_path_tree
templates = parse_template_tree(template_tree)
pt = PathTemplate("file_path", schema_identifier, templates[-1].segments, concepts)
values = {
"root": "ROOT",
"experiment_id": "EXP001",
"data_id": "data2",
"file_name": "summary",
}
rendered = pt.render(values)
parsed = pt.parse(rendered)
assert parsed == values
def test_render_and_parse_with_optional_segments():
schema_identifier = SchemaIdentifier(test=1)
template_tree = {
("{root}", "root_path"): {
("{optional}", "opt_path", False): {("{file_name}.txt", "file_path"): None}
}
}
concepts = {
"root": PathConcept(
name="root",
description="Root",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"optional": PathConcept(
name="optional",
description="Optional segment",
schema_identifier=schema_identifier,
regex=r"\w+",
),
"file_name": PathConcept(
name="file_name",
description="File",
schema_identifier=schema_identifier,
regex=r"\w+",
),
}
templates = parse_template_tree(template_tree)
pt = PathTemplate("file_path", schema_identifier, templates[-1].segments, concepts)
values = {"root": "ROOT", "optional": "SUB", "file_name": "data"}
path = pt.render(values)
assert pt.parse(path) == values
values = {"root": "ROOT", "file_name": "data"}
path = pt.render(values)
assert pt.parse(path) == {"root": "ROOT", "optional": None, "file_name": "data"}
def test_render_and_parse_roundtrip_from_filesystem_root():
"""A leading concept spanning path separators (like `{data_root}`) rendered
as the bare filesystem root must still round-trip."""
schema_identifier = SchemaIdentifier(test=1)
template_tree = {
("{root}", "root_path"): {
("{file_name}.txt", "file_path"): None,
}
}
concepts = {
"root": PathConcept(
name="root",
description="Root directory",
schema_identifier=schema_identifier,
regex=r".*",
),
"file_name": PathConcept(
name="file_name",
description="Filename",
schema_identifier=schema_identifier,
regex=r"\w+",
),
}
templates = parse_template_tree(template_tree)
pt = PathTemplate("file_path", schema_identifier, templates[-1].segments, concepts)
values = {"root": os.sep, "file_name": "summary"}
rendered = pt.render(values)
assert rendered == _os_path(os.sep, "summary.txt")
assert pt.parse(rendered) == values
test_tree.py 0000664 0000000 0000000 00000016566 15244036076 0032750 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/internals from ..._schemas.fields.tree import PathTemplateArguments
from ..._schemas.fields.tree import SegmentTemplate
from ..._schemas.fields.tree import parse_template_tree
from ..._schemas.fields.tree import tree_as_strings
_STRING_TREE = """
{root}/{noname}/
├── {project}/
│ └── {category}/
│ ├── {name}.txt
│ ├── metadata.json
│ └── {dataset}/
│ ├── images/
│ │ └── {image_id}.png
│ └── logs/
│ └── {log_name}.txt
└── archive/ [*]
└── {year}/
└── {month}/
└── {file_name}.zip
"""
_DICT_TREE = {
("{root}", "root_path"): {
"{noname}": {
("{project}", "project_path"): {
("{category}", "category_path"): {
("{name}.txt", "text_file_path"): None,
("metadata.json", "metadata_file_path"): None,
("{dataset}", "dataset_path"): {
("images", "images_path"): {
("{image_id}.png", "image_file_path"): None,
},
("logs", "logs_path"): {
("{log_name}.txt", "log_file_path"): None,
},
},
},
},
("archive", "archive_path", False): {
("{year}", "year_path"): {
("{month}", "month_path"): {
("{file_name}.zip", "archive_file_path"): None,
},
},
},
}
}
}
def test_parse_template_tree():
segments = parse_template_tree(_DICT_TREE)
nodes = {
"{root}": SegmentTemplate(template="{root}", mandatory=True),
"{noname}": SegmentTemplate(template="{noname}", mandatory=True),
"{project}": SegmentTemplate(template="{project}", mandatory=True),
"{category}": SegmentTemplate(template="{category}", mandatory=True),
"{name}.txt": SegmentTemplate(template="{name}.txt", mandatory=True),
"metadata.json": SegmentTemplate(template="metadata.json", mandatory=True),
"{dataset}": SegmentTemplate(template="{dataset}", mandatory=True),
"images": SegmentTemplate(template="images", mandatory=True),
"{image_id}.png": SegmentTemplate(template="{image_id}.png", mandatory=True),
"logs": SegmentTemplate(template="logs", mandatory=True),
"{log_name}.txt": SegmentTemplate(template="{log_name}.txt", mandatory=True),
"archive": SegmentTemplate(template="archive", mandatory=False),
"{year}": SegmentTemplate(template="{year}", mandatory=True),
"{month}": SegmentTemplate(template="{month}", mandatory=True),
"{file_name}.zip": SegmentTemplate(template="{file_name}.zip", mandatory=True),
}
expected = [
PathTemplateArguments(name="root_path", segments=[nodes["{root}"]]),
PathTemplateArguments(
name="project_path",
segments=[nodes["{root}"], nodes["{noname}"], nodes["{project}"]],
),
PathTemplateArguments(
name="category_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["{project}"],
nodes["{category}"],
],
),
PathTemplateArguments(
name="text_file_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["{project}"],
nodes["{category}"],
nodes["{name}.txt"],
],
),
PathTemplateArguments(
name="metadata_file_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["{project}"],
nodes["{category}"],
nodes["metadata.json"],
],
),
PathTemplateArguments(
name="dataset_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["{project}"],
nodes["{category}"],
nodes["{dataset}"],
],
),
PathTemplateArguments(
name="images_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["{project}"],
nodes["{category}"],
nodes["{dataset}"],
nodes["images"],
],
),
PathTemplateArguments(
name="image_file_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["{project}"],
nodes["{category}"],
nodes["{dataset}"],
nodes["images"],
nodes["{image_id}.png"],
],
),
PathTemplateArguments(
name="logs_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["{project}"],
nodes["{category}"],
nodes["{dataset}"],
nodes["logs"],
],
),
PathTemplateArguments(
name="log_file_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["{project}"],
nodes["{category}"],
nodes["{dataset}"],
nodes["logs"],
nodes["{log_name}.txt"],
],
),
PathTemplateArguments(
name="archive_path",
segments=[nodes["{root}"], nodes["{noname}"], nodes["archive"]],
),
PathTemplateArguments(
name="year_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["archive"],
nodes["{year}"],
],
),
PathTemplateArguments(
name="month_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["archive"],
nodes["{year}"],
nodes["{month}"],
],
),
PathTemplateArguments(
name="archive_file_path",
segments=[
nodes["{root}"],
nodes["{noname}"],
nodes["archive"],
nodes["{year}"],
nodes["{month}"],
nodes["{file_name}.zip"],
],
),
]
assert segments == expected
def test_parse_template_tree_string():
arguments = parse_template_tree(_DICT_TREE)
lines = tree_as_strings([arg.segments for arg in arguments])
text = "\n".join(lines)
assert f"\n{text}\n" == _STRING_TREE
def test_parse_template_tree_single_branch():
segments = [
SegmentTemplate(template="{root}", mandatory=True),
SegmentTemplate(template="archive", mandatory=False),
SegmentTemplate(template="{year}", mandatory=True),
SegmentTemplate(template="{month}", mandatory=True),
SegmentTemplate(template="{file_name}.zip", mandatory=True),
]
lines = tree_as_strings([segments])
text = "\n".join(lines)
expected = """
{root}/archive [*]/{year}/{month}/
└── {file_name}.zip
"""
assert f"\n{text}\n" == expected
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/tomo/ 0000775 0000000 0000000 00000000000 15244036076 0027420 5 ustar 00root root 0000000 0000000 __init__.py 0000664 0000000 0000000 00000000717 15244036076 0031457 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/tomo """*tomo* and *tomoproc* schema tests."""
from ...core import ESRFPath
class TomoPath(ESRFPath, tomo=1, fallback_depth=0):
"""Tomography processed data on the ESRF file system."""
class TomoprocPath(ESRFPath, tomoproc=1, fallback_depth=0):
"""Tomography processed data outside the ESRF file system."""
class MixedTomoPath(ESRFPath, tomo=1, tomoproc=1, fallback_depth=0):
"""Tomography processed data inside and outside the ESRF file system."""
test_caveats.py 0000664 0000000 0000000 00000031623 15244036076 0032405 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/tomo """Consequences of combining the *tomo* and *tomoproc* schemas which are easy to trip
over. These tests assert documented behavior, not desired behavior: a failure here
means the behavior changed, which is an API decision rather than a bug fix.
"""
from pathlib import Path
import pytest
from ..._schemas import errors
from ..._schemas.constants import IS_WINDOWS
from ..._schemas.definitions.esrf.esrf_utils import DEFAULT_DATA_ROOT
from ...core import ESRFPath
from ..utils import make_local_path
from ..utils import make_path
from ..utils import make_unrelated_path
from . import MixedTomoPath
from . import TomoPath
from . import TomoprocPath
class TomoprocFirstPath(ESRFPath, tomoproc=1, tomo=1, fallback_depth=0):
"""Same schemas as `MixedTomoPath` in the opposite order."""
def test_caveat_1_data_root():
"""'data_root' means the ESRF root directory for *tomo* but the dataset directory
for *tomoproc*. Code that interprets 'data_root' uniformly will be surprised."""
esrf_dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
local_dataset_dir = make_local_path("sample_0001")
esrf_slices_file = esrf_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
local_slices_file = local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
esrf = MixedTomoPath(esrf_slices_file)
local = MixedTomoPath(local_slices_file)
assert esrf.data_root == make_path("visitor")
assert local.data_root == local_dataset_dir
def test_caveat_2_every_path_matches():
"""Every path matches *tomoproc* ('dataset_path' template): any directory is
potentially the root directory of local tomography processed data. The *unknown*
schema never applies and 'schema_name == "tomoproc_v1"' is a much weaker statement
than 'schema_name == "tomo_v1"': check the template name instead."""
unrelated = make_unrelated_path("etc", "passwd")
path = MixedTomoPath(unrelated)
assert path.schema_name == "tomoproc_v1"
assert path.template_name == "dataset_path"
assert path.slices_path == unrelated / "slices"
def test_caveat_3_false_positives():
"""Unrelated paths match specific *tomoproc* templates when they contain a marker
directory (references, projections, slices, volumes, logs, workflows) and a
conforming name, because '{data_root}' matches any prefix."""
path = MixedTomoPath(
make_unrelated_path(
"home", "john_doe", "Pictures", "slices", "holiday_photo_beach_xy_00001.png"
)
)
assert path.template_name == "slices_file"
assert path.tomo_technique == "beach"
# '{tomo_dataset}' is a single concept so one name part before the technique is
# enough to match
path = MixedTomoPath(
make_unrelated_path(
"home", "john_doe", "Pictures", "slices", "holiday_beach_xy_00001.png"
)
)
assert path.template_name == "slices_file"
assert (path.tomo_dataset, path.tomo_technique) == ("holiday", "beach")
path = MixedTomoPath(make_unrelated_path("var", "logs", "syslog.1"))
assert path.template_name == "tomo_log_path"
def test_caveat_4_collection_dataset_split():
"""The collection/dataset split is unknown for local paths. In an ESRF session
directory '{collection}' is anchored by a directory. For local data the dataset
directory name is the single concept '{tomo_dataset}'. Local paths can be rendered
from the collection and dataset names (the *tomo* schema derives '{tomo_dataset}'
from them) but a local path does not provide them."""
esrf_dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_a_b",
)
local_dataset_dir = make_local_path("sample_a_b")
slice_name = "sample_a_b_phase_xy_00001.tiff"
esrf = MixedTomoPath(esrf_dataset_dir / "slices" / slice_name)
local = MixedTomoPath(local_dataset_dir / "slices" / slice_name)
assert (esrf.collection, esrf.dataset) == ("sample", "a_b")
assert esrf.tomo_dataset == "sample_a_b"
assert local.tomo_dataset == "sample_a_b"
with pytest.raises(errors.SchemaAttributeError):
_ = local.collection
# The local path can be rendered from the collection and dataset names
path = MixedTomoPath.from_fields(
data_root=local_dataset_dir,
collection="sample",
dataset="a_b",
tomo_technique="phase",
tomo_slicing_direction="xy",
tomo_slice_number=1,
tomo_extension="tiff",
)
assert path == local
def test_caveat_5_session_fields():
"""Session concepts (proposal, beamline, ...) and session-level templates
(slices_link_root_file, ...) are unavailable for *tomoproc* paths. Code handling
both schemas must be prepared for these exceptions."""
local_dataset_dir = make_local_path("sample_0001")
slices_file = local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
path = MixedTomoPath(slices_file)
with pytest.raises(errors.SchemaAttributeError):
_ = path.proposal
with pytest.raises(errors.UnknownPathTemplate):
_ = path.replace_fields(template_name="slices_link_root_file")
def test_caveat_6_schema_order():
"""The schema order matters for template name lookups when no path was matched: the
name resolves to the first schema that defines it. Path matching itself and
rendering without an explicit template name are order independent (best score
wins)."""
local_dataset_dir = make_local_path("sample_0001")
fields = {
"data_root": local_dataset_dir,
"collection": "sample",
"dataset": "0001",
"tomo_technique": "phase",
"tomo_slicing_direction": "xy",
"tomo_slice_number": 1,
"tomo_extension": "tiff",
}
expected = local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
# 'slices_file' resolves to the tomo template which needs the session concepts
with pytest.raises(errors.PathTemplateMatchError):
_ = MixedTomoPath.from_fields(template_name="slices_file", **fields)
# 'slices_file' resolves to the tomoproc template
path = TomoprocFirstPath.from_fields(template_name="slices_file", **fields)
assert path == expected
# Without a template name the best matching template wins in both cases
assert MixedTomoPath.from_fields(**fields) == expected
assert TomoprocFirstPath.from_fields(**fields) == expected
def test_caveat_7_data_root_default():
"""Unlike *esrf* and *tomo*, *tomoproc* has no default 'data_root': rendering
tomoproc-only concepts without it raises. But a merged registry can only keep one
definition per concept name (`test_caveat_6`), so combining *tomo* and *tomoproc*
makes 'data_root' default to the ESRF root '/data/visitor' whenever *tomo* is
merged first, silently yielding a bogus ESRF-like local path. Always provide
'data_root' when rendering tomoproc paths."""
fields = {
"collection": "sample",
"dataset": "0001",
"tomo_technique": "phase",
"tomo_slicing_direction": "xy",
"tomo_slice_number": 1,
"tomo_extension": "tiff",
}
with pytest.raises(errors.PathSchemaMatchError):
_ = TomoPath.from_fields(**fields)
with pytest.raises(errors.PathSchemaMatchError):
_ = TomoprocPath.from_fields(**fields)
# 'tomoproc' merged after 'tomo': 'tomo' has already claimed 'data_root' with
# its ESRF default, so tomoproc's own (default-less) definition is discarded.
with pytest.raises(errors.PathSchemaMatchError):
_ = TomoprocFirstPath.from_fields(**fields)
if IS_WINDOWS:
# Windows has no default data root
with pytest.raises(errors.PathSchemaMatchError):
_ = MixedTomoPath.from_fields(**fields)
else:
path = MixedTomoPath.from_fields(**fields)
assert path == Path(
DEFAULT_DATA_ROOT, "slices", "sample_0001_phase_xy_00001.tiff"
)
def test_caveat_8_tomoproc_only():
"""The *tomoproc* schema does not describe the ESRF session directories: a
tomoproc-only registry gives paths inside an ESRF session directory the local
interpretation. Combine with the *tomo* schema when paths on the ESRF file system
must be recognized."""
esrf_dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
esrf_slice = esrf_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
path = TomoprocPath(esrf_slice)
assert path.schema_name == "tomoproc_v1"
assert path.data_root == esrf_dataset_dir
raw_file = TomoprocPath(
make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"RAW_DATA",
"sample",
"sample_0001",
"sample_0001.h5",
)
)
assert raw_file.template_name == "dataset_path" # not 'raw_dataset_file'
assert MixedTomoPath(esrf_slice).schema_name == "tomo_v1"
def test_caveat_9_dataset_name_fields():
"""The dataset name is a different field per schema: 'tomo_dataset' for *tomoproc*
paths and 'collection'/'dataset' for *tomo* paths. Replacing a field that is not
used by the matched template is silently ignored."""
esrf_dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
local_dataset_dir = make_local_path("sample_0001")
esrf = MixedTomoPath(
esrf_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
)
local = MixedTomoPath(
local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
)
# Both provide the dataset name as 'tomo_dataset' (a derived concept for *tomo*)
assert esrf.tomo_dataset == local.tomo_dataset == "sample_0001"
# ... but only the concepts of the matched schema can be replaced
assert (
local.replace_fields(tomo_dataset="sample_0002").name
== "sample_0002_phase_xy_00001.tiff"
)
assert local.replace_fields(dataset="0002") == local # silently ignored
assert esrf.replace_fields(dataset="0002").name == "sample_0002_phase_xy_00001.tiff"
assert (
esrf.replace_fields(tomo_dataset="other_0002").name # silently ignored
== "sample_0001_phase_xy_00001.tiff"
)
def test_caveat_10_tomo_dataset_derivation():
"""Rendering *tomoproc* paths from 'collection' and 'dataset' relies on the derived
concept of the *tomo* schema. A tomoproc-only registry does not know these concepts
and silently renders 'dataset_path' instead ('{data_root}' matches any path).
Provide 'tomo_dataset' when the *tomo* schema is not enabled."""
local_dataset_dir = make_local_path("sample_0001")
expected = local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
fields = {
"data_root": local_dataset_dir,
"tomo_technique": "phase",
"tomo_slicing_direction": "xy",
"tomo_slice_number": 1,
"tomo_extension": "tiff",
}
# The *tomo* schema derives 'tomo_dataset' from the collection and dataset names
path = MixedTomoPath.from_fields(collection="sample", dataset="0001", **fields)
assert path == expected
# Without it the requested slice file silently becomes the dataset directory
path = TomoprocPath.from_fields(collection="sample", dataset="0001", **fields)
assert path.template_name == "dataset_path"
assert path == local_dataset_dir
assert TomoprocPath.from_fields(tomo_dataset="sample_0001", **fields) == expected
def test_caveat_11_tomo_dataset_conflict():
"""'tomo_dataset' and 'collection'/'dataset' are not cross-checked when both are
provided. Each template uses the concepts of its own schema: the *tomoproc*
templates use 'tomo_dataset' and the *tomo* templates use '{collection}_{dataset}',
so conflicting values silently give different dataset names."""
fields = {
"collection": "other",
"dataset": "0002",
"tomo_dataset": "sample_0001",
"tomo_technique": "phase",
"tomo_slicing_direction": "xy",
"tomo_slice_number": 1,
"tomo_extension": "tiff",
}
local_dataset_dir = make_local_path("sample_0001")
local = MixedTomoPath.from_fields(data_root=local_dataset_dir, **fields)
assert local == local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
esrf = MixedTomoPath.from_fields(
data_root=make_path("visitor"),
proposal="ma1234",
beamline="id23eh1",
session_date="20250509",
data_type="processed",
tomo_result_type="slice",
**fields,
)
assert esrf == make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"other",
"SLICES",
"other_0002_phase_xy_00001.tiff",
)
test_mixed.py 0000664 0000000 0000000 00000036163 15244036076 0032071 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/tomo """Tests for a registry with both the *tomo* and *tomoproc* schemas."""
import pytest
from ..._schemas.definitions import types
from ..._schemas.errors import SchemaAttributeError
from ..utils import make_local_path
from ..utils import make_path
from . import MixedTomoPath
@pytest.mark.parametrize(
"template_name,relpath",
[
("slices_file", ("slices", "sample_0001_phase_xy_00001.tiff")),
(
"slices_thumbnail",
("slices", "gallery", "sample_0001_phase_xy_00001_large.jpg"),
),
("volumes_technique_path", ("volumes", "sample_0001_phase_16Bit_vol")),
("tomo_log_file", ("logs", "sample_0001_nabu.log")),
("tomo_log_path", ("logs",)),
("projections_file", ("projections", "sample_0001.nx")),
],
)
def test_local_path_matches_tomoproc(template_name, relpath):
local_dataset_dir = make_local_path("sample_0001")
path = MixedTomoPath(local_dataset_dir.joinpath(*relpath))
assert path.schema_name == "tomoproc_v1"
assert path.template_name == template_name
assert path.data_root == local_dataset_dir
if template_name != "tomo_log_path":
assert path.tomo_dataset == "sample_0001"
# The collection/dataset split of a local directory name is unknown
with pytest.raises(SchemaAttributeError):
_ = path.collection
with pytest.raises(SchemaAttributeError):
_ = path.dataset
@pytest.mark.parametrize(
"template_name,relpath",
[
("slices_file", ("slices", "sample_0001_phase_xy_00001.tiff")),
(
"slices_thumbnail",
("slices", "gallery", "sample_0001_phase_xy_00001_large.jpg"),
),
("volumes_technique_path", ("volumes", "sample_0001_phase_16Bit_vol")),
("tomo_log_file", ("logs", "sample_0001_nabu.log")),
("tomo_log_path", ("logs",)),
("projections_file", ("projections", "sample_0001.nx")),
],
)
def test_esrf_path_matches_tomo(template_name, relpath):
esrf_dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
path = MixedTomoPath(esrf_dataset_dir.joinpath(*relpath))
assert path.schema_name == "tomo_v1"
assert path.template_name == template_name
assert path.data_root == make_path("visitor")
assert path.proposal == "ma1234"
assert path.collection == "sample"
assert path.dataset == "0001"
assert path.tomo_dataset == "sample_0001"
@pytest.mark.parametrize("kind", ["esrf", "local"])
def test_template_names_resolve_per_schema(kind):
"""Both schemas define the same template names: they resolve within the
schema that matched the path."""
if kind == "esrf":
dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
else:
dataset_dir = make_local_path("sample_0001")
slices_file = dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
path = MixedTomoPath(slices_file)
assert path.slices_path == dataset_dir / "slices"
assert path.slices_gallery_path == dataset_dir / "slices" / "gallery"
assert path.volumes_path == dataset_dir / "volumes"
assert path.tomo_log_path == dataset_dir / "logs"
@pytest.mark.parametrize("kind", ["esrf", "local"])
def test_replace_fields(kind):
if kind == "esrf":
dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
else:
dataset_dir = make_local_path("sample_0001")
slices_file = dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
path = MixedTomoPath(slices_file)
thumbnail = path.replace_fields(
template_name="slices_thumbnail", thumbnail_file_type="jpg"
)
assert (
thumbnail
== dataset_dir / "slices" / "gallery" / "sample_0001_phase_xy_00001_large.jpg"
)
preview = path.replace_fields(
template_name="slices_preview", thumbnail_file_type="jpg"
)
assert (
preview == dataset_dir / "slices" / "gallery" / "sample_0001_phase_xy_00001.jpg"
)
@pytest.mark.parametrize("kind", ["esrf", "local"])
def test_replace_fields_volume_gallery(kind):
if kind == "esrf":
dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
else:
dataset_dir = make_local_path("sample_0001")
volume_result_dir = dataset_dir / "volumes" / "sample_0001_phase_16Bit_vol"
path = MixedTomoPath(volume_result_dir)
assert path.template_name == "volumes_technique_path"
preview = path.replace_fields(
template_name="volumes_preview",
tomo_slicing_direction="xy",
tomo_slice_number=5,
thumbnail_file_type="png",
)
assert (
preview
== dataset_dir
/ "volumes"
/ "sample_0001_phase_16Bit_vol"
/ "gallery"
/ "sample_0001_phase_xy_00005.png"
)
def test_from_fields():
local_dataset_dir = make_local_path("sample_0001")
path = MixedTomoPath.from_fields(
data_root=local_dataset_dir,
tomo_dataset="sample_0001",
tomo_technique="phase",
tomo_slicing_direction="xy",
tomo_slice_number=1,
tomo_extension="tiff",
)
assert path == local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
assert path.schema_name == "tomoproc_v1"
assert path.template_name == "slices_file"
def test_from_fields_collection_dataset():
"""The `tomoproc_v1` templates can be rendered from the ESRF collection and
dataset names because `tomo_dataset` is a derived concept of `tomo_v1`."""
local_dataset_dir = make_local_path("sample_0001")
path = MixedTomoPath.from_fields(
data_root=local_dataset_dir,
collection="sample",
dataset="0001",
tomo_technique="phase",
tomo_slicing_direction="xy",
tomo_slice_number=1,
tomo_extension="tiff",
)
assert path == local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
assert path.schema_name == "tomoproc_v1"
assert path.template_name == "slices_file"
def test_from_fields_esrf_session():
"""When the ESRF session concepts are provided, the `tomo_v1` templates
win over the `tomoproc_v1` templates. Same result as `TomoPath(tomo=1)`
in `test_tomo.py::test_from_fields`."""
path = MixedTomoPath.from_fields(
data_root=make_path("visitor"),
proposal="ma1234",
beamline="id23eh1",
session_date="20250509",
data_type="processed",
collection="sample",
dataset="0001",
tomo_technique="absorption",
tomo_slicing_direction="xz",
tomo_slice_number=1,
tomo_result_type="slice",
tomo_extension="tiff",
)
expected = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"SLICES",
"sample_0001_absorption_xz_00001.tiff",
)
assert path == expected
assert path.schema_name == "tomo_v1"
def test_dataset_directory():
"""Any path that does not match anything else is a potential root
directory of local tomography processed data."""
local_dataset_dir = make_local_path("sample_0001")
path = MixedTomoPath(local_dataset_dir)
assert path.schema_name == "tomoproc_v1"
assert path.template_name == "dataset_path"
assert path.data_root == local_dataset_dir
assert path.slices_path == local_dataset_dir / "slices"
path = MixedTomoPath(make_path("some", "random", "path.txt"))
assert path.schema_name == "tomoproc_v1"
assert path.template_name == "dataset_path"
def test_from_fields_dataset_directory():
local_dataset_dir = make_local_path("sample_0001")
path = MixedTomoPath.from_fields(data_root=local_dataset_dir)
assert path == local_dataset_dir
assert path.schema_name == "tomoproc_v1"
assert path.template_name == "dataset_path"
def test_dataset_directory_field():
"""The dataset directory template of `tomoproc_v1` is the local
equivalent of the `processed_dataset_path` template of `esrf_v3`."""
esrf_dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
esrf_path = MixedTomoPath(
esrf_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
)
assert esrf_path.processed_dataset_path == esrf_dataset_dir
local_dataset_dir = make_local_path("sample_0001")
local_path = MixedTomoPath(
local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
)
assert local_path.dataset_path == local_dataset_dir
@pytest.mark.parametrize(
"template_name,relpath",
[
("slices_file", ("slices", "sample_0001_phase_xy_00001.tiff")),
("volumes_technique_path", ("volumes", "sample_0001_phase_16Bit_vol")),
],
)
def test_esrf_result_name_without_dataset_prefix(template_name, relpath):
"""The `tomo_v1` result names start with `{collection}_{dataset}`. An ESRF
result with another name (the results of a single sequence of a
multi-sequence dataset) is identified by `tomoproc_v1` instead of being
swallowed by the parent directory template of `tomo_v1`."""
esrf_dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
result_path = esrf_dataset_dir.joinpath(*relpath)
result_path = result_path.with_name(
result_path.name.replace("sample_0001_phase", "sample_0001_0004_phase")
)
path = MixedTomoPath(result_path)
assert path.schema_name == "tomoproc_v1"
assert path.template_name == template_name
assert path.tomo_dataset == "sample_0001_0004"
assert path.data_root == esrf_dataset_dir
def test_esrf_result_of_unknown_type():
"""A path that no schema identifies is still matched by the closest parent
directory template of `tomo_v1`."""
esrf_dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
path = MixedTomoPath(esrf_dataset_dir / "logs" / "nabu" / "nabu.log")
assert path.schema_name == "tomo_v1"
assert path.template_name == "tomo_log_path"
assert path.collection == "sample"
def test_template_attribute_keeps_its_template():
"""The session-level link directories of `tomo_v1` are named after the result
type, which is also a valid collection name of the `esrf` schema."""
esrf_dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
path = MixedTomoPath(esrf_dataset_dir / "projections" / "sample_0001.nx")
link_root = path.slices_link_root_path
assert link_root == make_path(
"visitor", "ma1234", "id23eh1", "20250509", "PROCESSED_DATA", "SLICES"
)
assert link_root.template_name == "slices_link_root_path"
assert link_root.tomo_result_type is types.TomoResultType.SLICE
# The parsed state is passed on to derived paths
assert MixedTomoPath(link_root).template_name == "slices_link_root_path"
# The rendered path itself is matched by the collection directory template
assert MixedTomoPath(str(link_root)).template_name == "processed_collection_path"
@pytest.mark.parametrize(
"deprecated_name,canonical_name,relpath,extra_fields",
[
(
"volumes_file",
"volumes_technique_path",
(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes",
"sample_0001_phase_16Bit_vol",
),
{},
),
(
"volumes_custom_type_file",
"volumes_custom_type_technique_path",
(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
"volumes_stitched",
"sample_0001_phase_stitched_16Bit_vol",
),
{"tomo_type": "stitched"},
),
(
"volumes_link_collection_file",
"volumes_link_collection_technique_path",
(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"VOLUMES",
"sample_0001_phase_16Bit_vol",
),
{},
),
(
"volumes_link_root_file",
"volumes_link_root_technique_path",
(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"VOLUMES",
"sample_0001_phase_16Bit_vol",
),
{},
),
],
)
def test_deprecated_tomo_template_names(
deprecated_name, canonical_name, relpath, extra_fields
):
"""The `*_file` tomo volume templates were renamed to `*_technique_path`.
The old names still work (as attributes and as `template_name`) but warn."""
path = MixedTomoPath(make_path(*relpath))
assert path.template_name == canonical_name
with pytest.deprecated_call():
deprecated_attr = getattr(path, deprecated_name)
canonical_attr = getattr(path, canonical_name)
assert deprecated_attr == canonical_attr
assert deprecated_attr.template_name == canonical_name
with pytest.deprecated_call():
replaced = path.replace_fields(
template_name=deprecated_name, tomo_technique="absorption"
)
assert replaced.template_name == canonical_name
with pytest.deprecated_call():
from_fields_path = MixedTomoPath.from_fields(
template_name=deprecated_name,
data_root=make_path("visitor"),
proposal="ma1234",
beamline="id23eh1",
session_date="20250509",
collection="sample",
dataset="0001",
tomo_technique="phase",
tomo_dtype="16Bit",
tomo_extension="vol",
**extra_fields,
)
assert from_fields_path.template_name == canonical_name
def test_tomo_only_fields_raise_for_local_paths():
"""Session-level templates cannot be derived from local processed data."""
local_dataset_dir = make_local_path("sample_0001")
slices_file = local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
path = MixedTomoPath(slices_file)
with pytest.raises(SchemaAttributeError):
_ = path.slices_link_root_file
with pytest.raises(SchemaAttributeError):
_ = path.proposal
test_shared_definitions.py 0000664 0000000 0000000 00000005172 15244036076 0034620 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/tomo """Tests for the definitions shared by the *tomo* and *tomoproc* schemas."""
from ..._schemas.definitions.tomo import tomo_v1
from ..._schemas.definitions.tomo import tomoproc_v1
from ..._schemas.definitions.tomo.tomo_utils import RESULT_TREE
from ..._schemas.registry import get_schema_registry
def test_common_part_defined_in_the_same_way():
"""The dataset result templates and their concepts are shared between
`tomo_v1` and `tomoproc_v1`."""
tomo = tomo_v1.get_schema()
tomoproc = tomoproc_v1.get_schema()
# Same concept definitions (the session concepts are tomo-only and the
# collection/dataset split is unknown for tomoproc)
session_concepts = {"proposal", "beamline", "session_date", "data_type"}
assert set(tomo.concepts) - set(tomoproc.concepts) == session_concepts | {
"tomo_result_type",
"collection",
"dataset",
}
assert set(tomoproc.concepts) - set(tomo.concepts) == {"tomo_dataset"}
for cname, concept in tomoproc.concepts.items():
if cname == "tomo_dataset":
continue
assert tomo.concepts[cname].is_same_definition(concept), cname
# The dataset directory name is a path concept for tomoproc and a derived
# concept for tomo
assert tomo.derived_concepts["tomo_dataset"].template == "{collection}_{dataset}"
assert "tomo_dataset" not in tomoproc.derived_concepts
# Template names of the dataset results
result_template_names = set()
stack = [RESULT_TREE]
while stack:
tree = stack.pop()
for key, subtree in tree.items():
if isinstance(key, tuple) and len(key) >= 2:
result_template_names.add(key[1])
if subtree is not None:
stack.append(subtree)
assert result_template_names <= set(tomo.templates)
assert result_template_names <= set(tomoproc.templates)
# Same relative path with respect to the dataset directory
registry = get_schema_registry(tomo=1, tomoproc=1, fallback_depth=0)
for template_name in sorted(result_template_names):
tomo_template = registry.get_template(template_name, "tomo_v1")
tomoproc_template = registry.get_template(template_name, "tomoproc_v1")
assert tomo_template is not tomoproc_template, template_name
tomo_segments = [seg.template for seg in tomo_template._segments]
tomoproc_segments = [
seg.template.replace("{tomo_dataset}", "{collection}_{dataset}")
for seg in tomoproc_template._segments
]
assert tomoproc_segments[0] == "{data_root}", template_name
assert tomo_segments[7:] == tomoproc_segments[1:], template_name
test_subclass.py 0000664 0000000 0000000 00000005514 15244036076 0032576 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/tomo """Tests for preserving the `ESRFPath` subclass in derived paths."""
from pathlib import Path
import pytest
from ...core import ESRFPath
from ..utils import make_local_path
from ..utils import make_path
from . import MixedTomoPath
from . import TomoprocPath
def _slices_file(kind: str) -> Path:
if kind == "esrf":
dataset_dir = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"sample_0001",
)
else:
dataset_dir = make_local_path("sample_0001")
return dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
@pytest.mark.parametrize("kind", ["esrf", "local"])
def test_field_is_subclass(kind):
path = MixedTomoPath(_slices_file(kind))
gallery = path.slices_gallery_path
assert type(gallery) is MixedTomoPath
assert gallery.schema_name == path.schema_name
assert gallery.template_name == "slices_gallery_path"
def test_field_uses_subclass_schemas():
path = TomoprocPath(make_local_path("sample_0001", "projections", "sample_0001.nx"))
assert path.template_name == "projections_file"
same_path = path.projections_file
assert same_path == path
assert same_path.schema_name == "tomoproc_v1"
assert same_path.template_name == "projections_file"
# The *tomoproc* schema is not enabled for `ESRFPath` itself
assert ESRFPath(same_path).schema_name == "unknown_v1"
@pytest.mark.parametrize("kind", ["esrf", "local"])
def test_derived_paths_are_subclass(kind):
path = MixedTomoPath(_slices_file(kind))
assert type(path.parent) is MixedTomoPath
assert type(path.parent / path.name) is MixedTomoPath
assert type(path.with_name("sample_0001_phase_xy_00002.tiff")) is MixedTomoPath
assert type(path.replace_fields(tomo_slice_number=2)) is MixedTomoPath
assert (
type(path.replace_fields(template_name="slices_gallery_path")) is MixedTomoPath
)
assert type(path.strip_mount_point()) is MixedTomoPath
assert type(MixedTomoPath.from_path(path)) is MixedTomoPath
def test_from_fields_is_subclass():
path = TomoprocPath.from_fields(
data_root=make_local_path("sample_0001"), tomo_dataset="sample_0001"
)
assert type(path) is TomoprocPath
assert path.schema_name == "tomoproc_v1"
def test_glob_is_subclass(tmp_path):
slices_dir = tmp_path / "sample_0001" / "slices"
slices_dir.mkdir(parents=True)
for technique in ("phase", "absorption"):
(slices_dir / f"sample_0001_{technique}_xy_00001.tiff").touch()
path = TomoprocPath(slices_dir / "sample_0001_phase_xy_00001.tiff")
matches = list(path.replace_fields(tomo_technique="*").glob())
assert len(matches) == 2
for match in matches:
assert type(match) is TomoprocPath
assert match.template_name == "slices_file"
test_tomo.py 0000664 0000000 0000000 00000006260 15244036076 0031734 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/tomo """Tests for a registry with the *tomo* schema only."""
from ..._schemas.constants import IS_WINDOWS
from ...core import ESRFPath
from ..utils import make_local_path
from ..utils import make_path
from ..utils import make_realpath
from . import TomoPath
def test_replace_fields():
path_in = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"RAW_DATA",
"sample",
"sample_dataset",
"sample_dataset.h5",
)
expected_out = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"SLICES",
"sample_dataset_absorption_xz_00001.tiff",
)
path = TomoPath(path_in)
xz_slice_1 = path.replace_fields(
data_type="processed",
tomo_technique="absorption",
tomo_type="denoised",
tomo_slicing_direction="xz",
tomo_slice_number=1,
tomo_result_type="slice",
tomo_extension="tiff",
)
assert xz_slice_1 == expected_out, xz_slice_1
def test_from_fields():
if IS_WINDOWS:
data_root = "X:\\visitor"
else:
data_root = None
xz_slice_1 = ESRFPath.from_fields(
data_root=data_root,
schema_name="tomo",
proposal="ma1234",
beamline="id23eh1",
session_date="20250509",
data_type="processed",
collection="sample",
dataset="dataset",
tomo_technique="absorption",
tomo_type="denoised",
tomo_slicing_direction="xz",
tomo_slice_number=1,
tomo_result_type="slice",
tomo_extension="tiff",
)
expected_out = make_path(
"visitor",
"ma1234",
"id23eh1",
"20250509",
"PROCESSED_DATA",
"sample",
"SLICES",
"sample_dataset_absorption_xz_00001.tiff",
)
assert xz_slice_1 == expected_out, xz_slice_1
def test_strip_mount_point():
realpath = TomoPath.from_fields(
data_root=make_realpath("visitor"),
proposal="ma1234",
beamline="id23eh1",
session_date="20250509",
data_type="processed",
collection="sample",
dataset="dataset",
tomo_technique="absorption",
tomo_type="denoised",
tomo_slicing_direction="xz",
tomo_slice_number=1,
tomo_result_type="slice",
tomo_extension="tiff",
)
path = TomoPath.from_fields(
data_root=make_path("visitor"),
proposal="ma1234",
beamline="id23eh1",
session_date="20250509",
data_type="processed",
collection="sample",
dataset="dataset",
tomo_technique="absorption",
tomo_type="denoised",
tomo_slicing_direction="xz",
tomo_slice_number=1,
tomo_result_type="slice",
tomo_extension="tiff",
)
assert path == realpath.strip_mount_point()
def test_tomo_only_registry_does_not_match_local_paths():
local_dataset_dir = make_local_path("sample_0001")
slices_file = local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
path = TomoPath(slices_file)
assert path.schema_name == "unknown_v1"
assert path.template_name == "unknown_path"
test_tomoproc.py 0000664 0000000 0000000 00000002176 15244036076 0032622 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/tomo """Tests for a registry with the *tomoproc* schema only."""
from ...core import ESRFPath
from ..utils import make_local_path
from . import TomoprocPath
def test_tomoproc_only_registry():
local_dataset_dir = make_local_path("sample_0001")
tomo_log_file = local_dataset_dir / "logs" / "sample_0001_nabu.log"
path = TomoprocPath(tomo_log_file)
assert path.schema_name == "tomoproc_v1"
assert path.template_name == "tomo_log_file"
assert path.tomo_application == "nabu"
log_dir = TomoprocPath(local_dataset_dir / "logs" / "nested" / "dir")
assert log_dir.schema_name == "tomoproc_v1"
assert log_dir.template_name == "tomo_log_path"
def test_from_fields_schema_name():
local_dataset_dir = make_local_path("sample_0001")
slices_file = local_dataset_dir / "slices" / "sample_0001_phase_xy_00001.tiff"
path = ESRFPath.from_fields(
schema_name="tomoproc",
data_root=local_dataset_dir,
tomo_dataset="sample_0001",
tomo_technique="phase",
tomo_slicing_direction="xy",
tomo_slice_number=1,
tomo_extension="tiff",
)
assert path == slices_file
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/unknown/ 0000775 0000000 0000000 00000000000 15244036076 0030141 5 ustar 00root root 0000000 0000000 __init__.py 0000664 0000000 0000000 00000000201 15244036076 0032164 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/unknown """*unknown* schema tests."""
from ...core import ESRFPath
class UnknownPath(ESRFPath, unknown=1, fallback_depth=0):
pass
test_caveats.py 0000664 0000000 0000000 00000005007 15244036076 0033123 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/unknown """Consequences of the *unknown* schema being merged into every other schema which are
easy to trip over. These tests assert documented behavior, not desired behavior: a
failure here means the behavior changed, which is an API decision rather than a bug
fix.
"""
import pytest
from ..._schemas import errors
from ...core import ESRFPath
from ..utils import make_unrelated_path
from . import UnknownPath
class EsrfOnly(ESRFPath, esrf=3, fallback_depth=0):
"""Same schema as `esrf.EsrfPath` but without any older-version fallback and
without requesting the *unknown* schema."""
class EsrfPlusUnknown(ESRFPath, esrf=3, unknown=1, fallback_depth=0):
"""Same schema as `EsrfOnly` but with the *unknown* schema requested
explicitly."""
def test_caveat_1_unknown_is_always_available():
"""*esrf_v3* (like every other real schema) extends the *unknown* schema, so its
'unknown_part' concept and 'unknown_path' template are already part of the
registry even when 'unknown' is never requested. Requesting it explicitly changes
nothing: unrelated paths resolve to 'schema_name == "unknown_v1"' either way."""
path = make_unrelated_path("etc", "passwd")
without_unknown = EsrfOnly(path)
with_unknown = EsrfPlusUnknown(path)
assert without_unknown.schema_name == "unknown_v1"
assert without_unknown.template_name == "unknown_path"
assert with_unknown.schema_name == without_unknown.schema_name
assert with_unknown.template_name == without_unknown.template_name
def test_caveat_2_replacing_unknown_part_can_drop_the_root():
"""'unknown_part' is a single concept spanning the whole path: there is no
separate root concept enforcing that the path stays absolute. Replacing it with a
relative-looking value silently turns an absolute path into a relative one."""
path = UnknownPath(make_unrelated_path("etc", "passwd"))
assert path.is_absolute()
relative = path.replace_fields(unknown_part="not/an/absolute/path")
assert str(relative) == "not/an/absolute/path"
assert not relative.is_absolute()
def test_caveat_3_no_default_value_for_unknown_part():
"""Unlike 'data_root' for *esrf*/*tomo*, 'unknown_part' has no default value.
Rendering without it does not silently fall back to anything: it raises
'PathTemplateMatchError' because the single template of the schema cannot be
filled in, not 'PathSchemaMatchError' because there is no other schema to
consider."""
with pytest.raises(errors.PathTemplateMatchError):
_ = UnknownPath.from_fields()
test_unknown.py 0000664 0000000 0000000 00000004117 15244036076 0033175 0 ustar 00root root 0000000 0000000 esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/unknown """Tests for a registry with the *unknown* schema only."""
import re
import pytest
from ..._schemas import errors
from ..utils import make_unrelated_path
from . import UnknownPath
def test_any_path_matches():
path = UnknownPath(make_unrelated_path("etc", "passwd"))
assert path.schema_name == "unknown_v1"
assert path.template_name == "unknown_path"
assert path.unknown_part == path
def test_parent_stays_unknown():
path = UnknownPath(make_unrelated_path("etc", "foo", "bar"))
for _ in range(3):
assert path.schema_name == "unknown_v1"
assert path.template_name == "unknown_path"
path = path.parent
def test_from_fields():
path = UnknownPath.from_fields(unknown_part=make_unrelated_path("etc", "passwd"))
assert path == make_unrelated_path("etc", "passwd")
def test_replace_fields():
path = UnknownPath(make_unrelated_path("etc", "passwd"))
path = path.replace_fields(unknown_part=make_unrelated_path("etc", "shadow"))
assert path == make_unrelated_path("etc", "shadow")
def test_immutable_schema_fields():
path = UnknownPath(make_unrelated_path("etc", "passwd"))
with pytest.raises(
errors.SchemaAttributeError,
match=re.escape(
"Attribute 'unknown_part' is immutable. Create a new path instance with "
"`UnknownPath.replace_fields(unknown_part='/other/path')`."
),
):
path.unknown_part = "/other/path"
def test_dir():
path = UnknownPath(make_unrelated_path("etc", "passwd"))
assert {"unknown_part", "unknown_path"}.issubset(set(dir(path)))
def test_strip_mount_point_is_a_no_op():
"""The *unknown* schema has no root concept to strip a mount point from."""
path = UnknownPath(make_unrelated_path("mnt", "multipath-shares", "data", "foo"))
assert path.strip_mount_point() == path
def test_str_and_repr():
path_str = str(make_unrelated_path("etc", "passwd"))
path = UnknownPath(path_str)
assert str(path) == path_str
assert (
repr(path)
== f""
)
esrf-pathlib-v1.2.0rc6-f928c11f3950c126217a2089616e265849ffba3c/src/esrf_pathlib/tests/utils.py 0000664 0000000 0000000 00000002120 15244036076 0030147 0 ustar 00root root 0000000 0000000 from pathlib import Path
from .._schemas import constants
if constants.IS_WINDOWS:
_ROOT = Path("X:\\")
_SYMBOLIC_BASE = _ROOT
_REAL_BASE = _ROOT
_LOCAL_BASE = _ROOT / "Users" / "john_doe" / "mydata"
else:
_ROOT = Path("/")
_SYMBOLIC_BASE = _ROOT / "data"
_REAL_BASE = _ROOT / "mnt" / "multipath-shares" / "data"
_LOCAL_BASE = _ROOT / "home" / "john_doe" / "mydata"
def make_path(*segments: str) -> Path:
"""Construct a path under the symbolic data root for the current OS."""
return _SYMBOLIC_BASE.joinpath(*segments)
def make_realpath(*segments: str) -> Path:
"""Construct a path under the real filesystem data root for the current OS."""
return _REAL_BASE.joinpath(*segments)
def make_local_path(*segments: str) -> Path:
"""Construct a path under a dataset directory outside the ESRF data root
for the current OS."""
return _LOCAL_BASE.joinpath(*segments)
def make_unrelated_path(*segments: str) -> Path:
"""Construct an absolute path unrelated to any data directory for the current OS."""
return _ROOT.joinpath(*segments)