pax_global_header 0000666 0000000 0000000 00000000064 15241025157 0014514 g ustar 00root root 0000000 0000000 52 comment=8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/ 0000775 0000000 0000000 00000000000 15241025157 0021540 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/.gitignore 0000664 0000000 0000000 00000000121 15241025157 0023522 0 ustar 00root root 0000000 0000000 __pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
build/
dist/
html
doc/generated volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/.gitlab-ci.yml 0000664 0000000 0000000 00000004442 15241025157 0024200 0 ustar 00root root 0000000 0000000 include:
- project: dau/ci/pyci
file: .gitlab-ci-template.yml
variables:
TOMOBASE_DEP: "git+https://gitlab.esrf.fr/tomotools/tomobase.git"
DISABLE_BLACK_FLAKE8: "yes"
# style/linting
ruff:
extends: .ruff
# tests with dev deps
# tomobase has no PyPI release yet, so every source-based test job must
# install it from git instead of letting pip resolve it from PyPI.
test_python3_12_dev:
extends: .test-3.12_glx
before_script:
- !reference [.test-3.12_glx, before_script]
script:
- echo "Installing $TOMOBASE_DEP"
- python -m pip install $TOMOBASE_DEP
- !reference [.test-3.12_glx, script]
after_script:
- !reference [.test-3.12_glx, after_script]
variables:
COVERAGE_SOURCE: "./$INTERNAL_PROJECT_NAME"
PYTEST_WARNINGS: "-W error ${IGNORE_WARNINGS}"
PYTEST_COV: "true"
# tests with PyPI deps
# NOTE: allowed to fail until `tomobase` gets its first PyPI release - remove
# `allow_failure` once it is published.
test_sdist-3.10_glx:
extends: .test_sdist-3.10_glx
allow_failure: true
variables:
PYTEST_WARNINGS: "-W error ${IGNORE_WARNINGS}"
test_sdist-3.12_glx:
extends: .test_sdist-3.12_glx
allow_failure: true
variables:
PYTEST_WARNINGS: "-W error ${IGNORE_WARNINGS}"
PYTEST_COV: "true"
test_sdist-3.13_glx:
extends: .test_sdist-3.13_glx
allow_failure: true
variables:
PYTEST_WARNINGS: "-W error ${IGNORE_WARNINGS}"
# doc
build_doc:
extends: .build_doc
before_script:
- !reference [.build_doc, before_script]
script:
- echo "Installing $TOMOBASE_DEP"
- python -m pip install $TOMOBASE_DEP
- !reference [.build_doc, script]
after_script:
- !reference [.build_doc, after_script]
build_sdist:
extends: .build_sdist
# test_notebooks
test_tutorials:
extends: .base-lin
stage: test
image: gitlab-registry.esrf.fr/dau/ci/pyci/python_3.12_glx:latest
before_script:
- !reference [.base-lin, before_script]
script:
- !reference [.base-lin, script]
- python -m pip install $TOMOBASE_DEP
- python -m pip install .[doc,test]
- jupyter nbconvert --to notebook --execute doc/tutorials/volume.ipynb
after_script:
- !reference [.base-lin, after_script]
allow_failure: true
# deploy
assets:
extends: .assets
pages:
extends: .pages
testpypi:
extends: .testpypi
pypi:
extends: .pypi
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/.readthedocs.yaml 0000664 0000000 0000000 00000001144 15241025157 0024767 0 ustar 00root root 0000000 0000000 # Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the version of Python and other tools you might need
build:
os: ubuntu-24.04
tools:
python: "3.12"
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: doc/conf.py
# If using Sphinx, optionally build your docs in additional formats such as PDF
# formats:
# - pdf
# Optionally declare the Python requirements required to build your docs
python:
install:
- method: pip
path: .
extra_requirements:
- doc
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/CHANGELOG.md 0000664 0000000 0000000 00000003052 15241025157 0023351 0 ustar 00root root 0000000 0000000 # Changelog
All notable changes to this project will be documented in this file.
The format follows the [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) conventions and this project adheres to [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html).
## [0.0.0dev0] - Unreleased
### Added
- Initial extraction of `volio` from `tomoscan` (2.5.1): volume load/save code, now decoupled from scan reading and depending only on `tomobase`.
- `VolumeBase`, `SliceTuple` (from `tomoscan.volumebase`).
- `VolumeSingleFrameBase`, `TIFFLikeDiskAccessor` (from `tomoscan.esrf.volume.singleframebase`).
- `EDFVolume`, `HDF5Volume`, `JP2KVolume`, `RawVolume`, `TIFFVolume`, `MultiTIFFVolume` and their identifiers (`EDFVolumeIdentifier`, `HDF5VolumeIdentifier`, `JP2KVolumeIdentifier`, `RawVolumeIdentifier`, `TIFFVolumeIdentifier`, `MultiTiffVolumeIdentifier`).
- `mock.create_volume` volume-mocking helper.
- `utils` (merged `tomoscan.esrf.volume.utils` and `tomoscan.utils.volume`): `guess_hdf5_volume_data_paths`, `guess_volumes`, `get_most_common_extension`, `group_slices_by_axis`, `concatenate`, `update_metadata`, `rescale_data`.
- `drac` (from `tomoscan.utils.drac`): `flatten_vol_metadata`, `map_nabu_keys_to_drac`.
- `source_scan` (on `VolumeBase` and subclasses) is now typed as `tomobase.tomoobject.TomoObject` instead of `tomoscan.scanbase.TomoScanBase`, so `volio` does not depend on `tomoscan`. `HDF5Volume`'s `get_default_data_path_for_volume` now duck-types on `hasattr(scan, "entry")` instead of an `isinstance(scan, TomoScanBase)` check.
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/LICENSE 0000664 0000000 0000000 00000002317 15241025157 0022550 0 ustar 00root root 0000000 0000000
The volio library provides volume load/save for tomography data at ESRF.
volio is distributed under the MIT license.
The MIT license follows:
Copyright (c) European Synchrotron Radiation Facility (ESRF)
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/README.md 0000664 0000000 0000000 00000001571 15241025157 0023023 0 ustar 00root root 0000000 0000000 # volio
Volume load/save for tomography data at ESRF.
`volio` reads and writes tomography reconstruction volumes stored in several formats:
* HDF5 (`HDF5Volume`)
* single frame EDF (`EDFVolume`)
* single frame or multi-frame TIFF (`TIFFVolume`, `MultiTIFFVolume`)
* JPEG2000 (`JP2KVolume`)
* raw / `.vol` (`RawVolume`)
It also provides utilities to guess volumes from a file or folder, concatenate volumes,
and map reconstruction metadata to the drac/icat data-portal metadata model.
## installation
### using pypi
To install the latest 'volio' pip package
``` bash
pip install volio
```
### using gitlab repository
``` bash
pip install git+https://gitlab.esrf.fr/tomotools/volio.git
```
## documentation
General documentation can be found here: [https://tomotools.gitlab-pages.esrf.fr/volio/](https://tomotools.gitlab-pages.esrf.fr/volio/)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/ 0000775 0000000 0000000 00000000000 15241025157 0022305 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/_static/ 0000775 0000000 0000000 00000000000 15241025157 0023733 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/_static/custom.css 0000664 0000000 0000000 00000000323 15241025157 0025755 0 ustar 00root root 0000000 0000000 /* compact code blocks nested in table cells (e.g. the identifier examples table) */
.compact-code-table pre {
margin: 0;
padding: 0.35em 0.6em;
}
.compact-code-table td {
vertical-align: middle;
}
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/_templates/ 0000775 0000000 0000000 00000000000 15241025157 0024442 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/_templates/autosummary/ 0000775 0000000 0000000 00000000000 15241025157 0027030 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/_templates/autosummary/class.rst 0000664 0000000 0000000 00000001142 15241025157 0030665 0 ustar 00root root 0000000 0000000 {{ fullname | escape | underline}}
.. currentmodule:: {{ module }}
.. autoclass:: {{ objname }}
:members:
:show-inheritance:
:inherited-members:
{% block methods %}
.. automethod:: __init__
{% if methods %}
.. rubric:: {{ _('Methods') }}
.. autosummary::
{% for item in methods %}
~{{ name }}.{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block attributes %}
{% if attributes %}
.. rubric:: {{ _('Attributes') }}
.. autosummary::
{% for item in attributes %}
~{{ name }}.{{ item }}
{%- endfor %}
{% endif %}
{% endblock %} volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/_templates/autosummary/module.rst 0000664 0000000 0000000 00000002145 15241025157 0031051 0 ustar 00root root 0000000 0000000 {{ fullname | escape | underline}}
.. automodule:: {{ fullname }}
{% block attributes %}
{% if attributes %}
.. rubric:: Module Attributes
.. autosummary::
:toctree:
{% for item in attributes %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block functions %}
{% if functions %}
.. rubric:: {{ _('Functions') }}
.. autosummary::
:toctree:
{% for item in functions %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block classes %}
{% if classes %}
.. rubric:: {{ _('Classes') }}
.. autosummary::
:toctree:
{% for item in classes %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block exceptions %}
{% if exceptions %}
.. rubric:: {{ _('Exceptions') }}
.. autosummary::
:toctree:
{% for item in exceptions %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block modules %}
{% if modules %}
.. rubric:: Modules
.. autosummary::
:toctree:
:recursive:
{% for item in modules %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %} volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/_templates/version.html 0000664 0000000 0000000 00000000100 15241025157 0027004 0 ustar 00root root 0000000 0000000
{{ version }} volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/api.rst 0000664 0000000 0000000 00000000136 15241025157 0023610 0 ustar 00root root 0000000 0000000 API reference
=============
.. autosummary::
:toctree: generated
:recursive:
volio
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/conf.py 0000664 0000000 0000000 00000003337 15241025157 0023612 0 ustar 00root root 0000000 0000000 from importlib.metadata import version as get_version
extensions = [
"sphinx.ext.autodoc",
"sphinx.ext.intersphinx",
"sphinx.ext.autosummary",
"sphinx_design",
"nbsphinx",
"sphinx_autodoc_typehints",
"myst_parser",
"sphinx_copybutton",
]
nbsphinx_timeout = 60
nbsphinx_allow_errors = True
templates_path = ["_templates"]
source_suffix = {
".rst": "restructuredtext",
".md": "markdown",
}
master_doc = "index"
project = "volio"
copyright = "2026, ESRF"
author = "ESRF"
release = get_version(project)
version = ".".join(release.split(".")[:2])
language = "en"
exclude_patterns = [
"build",
"**.ipynb_checkpoints",
]
pygments_style = "sphinx"
todo_include_todos = False
html_theme = "pydata_sphinx_theme"
html_logo = "img/volio.png"
html_static_path = ["_static"]
html_css_files = ["custom.css"]
htmlhelp_basename = "volio"
man_pages = [(master_doc, "volio", "volio documentation", [author], 1)]
autosummary_generate = True
autodoc_default_flags = [
"members",
"undoc-members",
"show-inheritance",
]
html_theme_options = {
"icon_links": [
{
"name": "pypi",
"url": "https://pypi.org/project/volio",
"icon": "_static/navbar_icons/pypi.svg",
"type": "local",
},
{
"name": "gitlab",
"url": "https://gitlab.esrf.fr/tomotools/volio",
"icon": "_static/navbar_icons/gitlab.svg",
"type": "local",
},
],
"show_toc_level": 1,
"navbar_align": "left",
"show_version_warning_banner": True,
"navbar_start": ["navbar-logo", "version"],
"navbar_center": ["navbar-nav"],
"footer_start": ["copyright"],
"footer_center": ["sphinx-version"],
}
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/img/ 0000775 0000000 0000000 00000000000 15241025157 0023061 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/img/volio-lockup.svg 0000664 0000000 0000000 00000001102 15241025157 0026217 0 ustar 00root root 0000000 0000000
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/img/volio.png 0000664 0000000 0000000 00000003655 15241025157 0024730 0 ustar 00root root 0000000 0000000 PNG
IHDR d d pT pHYs od tEXtSoftware www.inkscape.org< :IDATxOl ?u)
uQr!ErR`w
!&*zQd!%FKeM[&Dz Rm)Q"P2^{fޛw=cQU݁$8Fcmw (àk}S S$|a+ +" |As|n\?v݈AU 29مQ;s͎)E 21юʛi<٫3ڱp:& ; 1S_V88g^~<ۈ iw \ΣmPS 2ێBm D&';Qu-,"ɜݕ "{jG`*bgby Ume,v b@lXyMbJ !mBKD*v I,bOlcV,+\0f2SngXՄrd6l3y,Eɶ1òƒ.K3Sx=su͚͞@"9~_J u Z@HpK8cuå GYLUbY(p[X3
U ^VKU
|}_~ }4)3bkːWtKiܺ~TwTO{pPwwXg>2Ғ@uD:HG'rog+EJ:s `z "Kv~Y~S_ۋH7'!wf^E T !˙CdwtYCq<~<ԅQ( FH,0^<^>Jl+q r_zv).^} ADWאH$%SrU Y' !fQ@J^@%+ e'%92^S֊
Ē//}DbjrjWJ_c3+U$SVCI I a0yX:E1S3W;q|7_mzݦ>rwu|߲4ދ"iކԐΜYmDk{# DNyzmOJa4^9?,k,ӅQ4p)]Ϝ+ h;q1ɱ^7ٓQ6_!>8)e x
O#k{Lק3ˑN EEߤK břIJ@I,
$| Ӂ,PXV
$B J,YX HbgjA,]Pb5BS&b]@'wm%rxeq"4ZC$Hq C8 IENDB` volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/img/volio.svg 0000664 0000000 0000000 00000000602 15241025157 0024730 0 ustar 00root root 0000000 0000000
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/index.rst 0000664 0000000 0000000 00000007316 15241025157 0024155 0 ustar 00root root 0000000 0000000 volio
=====
Volume load/save for tomography data at ESRF: read and write tomography reconstruction
volumes stored as HDF5, EDF, TIFF, JP2K or raw (.vol) files.
Usage
-----
Creating and saving a volume
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Every volume format (:class:`~volio.HDF5Volume`, :class:`~volio.EDFVolume`,
:class:`~volio.TIFFVolume`, :class:`~volio.MultiTIFFVolume`,
:class:`~volio.JP2KVolume`) shares the same API: provide some ``data``
(a 3D numpy array) and optional ``metadata`` (a dict), then call ``save()``.
.. code-block:: python
import numpy
from volio import HDF5Volume
data = numpy.random.random((10, 100, 100))
metadata = {"processing_options": {"reconstruction": {"voxel_size_cm": 0.1}}}
volume = HDF5Volume(
file_path="reconstruction.hdf5",
data_path="entry0000/reconstruction",
data=data,
metadata=metadata,
)
volume.save()
Reading a volume
~~~~~~~~~~~~~~~~
Instantiate the volume from the same location and call ``load()`` (or
``load_data()`` / ``load_metadata()`` to load only part of it):
.. code-block:: python
from volio import HDF5Volume
volume = HDF5Volume(
file_path="reconstruction.hdf5",
data_path="entry0000/reconstruction",
)
volume.load()
print(volume.data.shape)
print(volume.metadata)
.. warning::
``load()`` (and ``load_data()``) reads the **entire** volume into memory
as a single numpy array. For large volumes this can be expensive.
To avoid this, you can:
* browse the volume slice by slice with ``browse_slices()``, which reads
one slice from disk at a time:
.. code-block:: python
for data_slice in volume.browse_slices():
... # process a single 2D numpy array without loading the full volume
* read a single, specific slice with ``get_slice()``:
.. code-block:: python
middle_slice = volume.get_slice(axis=0, index="middle")
Retrieving a volume from its identifier
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A volume can also be retrieved directly from its identifier string, without
knowing its class in advance, using :class:`~volio.factory.Factory`:
.. code-block:: python
from volio.factory import Factory
identifier = "hdf5:volume:/path/to/reconstruction.hdf5?path=entry0000/reconstruction"
volume = Factory.create_tomo_object_from_identifier(identifier)
volume.load()
print(volume.data.shape)
print(volume.metadata)
Identifiers
~~~~~~~~~~~
Any volume can be represented as a compact string, called an *identifier*
(``scheme:tomo_type:...``), which can be used to recreate the volume through
:class:`~volio.factory.Factory` without knowing its class in advance:
.. code-block:: python
from volio.factory import Factory
identifier = volume.get_identifier().to_str()
same_volume = Factory.create_tomo_object_from_identifier(identifier)
Example identifier for each format:
.. list-table::
:header-rows: 1
:class: compact-code-table
:widths: 25 75
* - Format
- Example identifier
* - HDF5Volume
- .. code-block:: text
hdf5:volume:/path/to/file_path?path=entry0000
* - EDFVolume
- .. code-block:: text
edf:volume:/path/to/my/my_folder
* - TIFFVolume
- .. code-block:: text
tiff:volume:/path/to/my/my_folder
* - MultiTIFFVolume
- .. code-block:: text
tiff3d:volume:/path/to/tiff_file.tif
* - JP2KVolume
- .. code-block:: text
jp2k:volume:/path/to/my/my_folder
* - RawVolume
- .. code-block:: text
raw:volume:/path/to/my/my_volume.vol
See the :doc:`tutorials/index` for a complete, runnable walk-through of each
volume format.
.. toctree::
:maxdepth: 2
tutorials/index.rst
api.rst
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/tutorials/ 0000775 0000000 0000000 00000000000 15241025157 0024333 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/tutorials/img/ 0000775 0000000 0000000 00000000000 15241025157 0025107 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/tutorials/img/volumes.svg 0000664 0000000 0000000 00000004062 15241025157 0027324 0 ustar 00root root 0000000 0000000
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/tutorials/index.rst 0000664 0000000 0000000 00000000244 15241025157 0026174 0 ustar 00root root 0000000 0000000 Tutorials
---------
.. toctree::
:maxdepth: 1
:hidden:
volume
.. grid:: 3
.. grid-item-card::
:img-top: img/volumes.svg
:doc:`volume`
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/doc/tutorials/volume.ipynb 0000664 0000000 0000000 00000037141 15241025157 0026713 0 ustar 00root root 0000000 0000000 {
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Volumes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Context"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`volio` provides classes dedicated to volumes: a common way of saving and loading tomography reconstruction volumes across the tomotools suite.\n",
"From it we intend to simplify volume handling for creating them but also to do some post processing operation on them such as stitching but also ease handling from gui."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## overview"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Volumes contains two main parts:\n",
"* `data`: the volume itself. It is expected to be a 3D numpy array\n",
"* `metadata`: metadata associated to the volume such as information regarding the reconstruction parameters..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For now the following volumes exists:\n",
" \n",
"* HDF5Volume: save data and metadata to an hdf5 file.\n",
"* EDFVolume: save data to single frame EDF files and metadata to a text file\n",
"* JP2KVolume: save data to single frame jp2k files and metadata to a text file\n",
"* TIFFVolume: save data to single frame tiff files and metadata to a text file\n",
"* MultiTIFFVolume: save data to a single 3D tiff file and metadata to a text file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Volume API"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"from h5glance import H5Glance\n",
"\n",
"from volio import ( # noqa: F401\n",
" EDFVolume,\n",
" HDF5Volume,\n",
" JP2KVolume,\n",
" MultiTIFFVolume,\n",
" TIFFVolume, # noqa: F401\n",
" has_glymur,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### create some dataset to test"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from volio.tests.mock import create_volume\n\ndata = create_volume(frame_dims=(100, 100), z_size=5)\nmetadata = {\n \"reconstruction_params\": {\n \"dataset\": {\n \"location\": \"toto.hdf5\",\n \"entry\": \"entry0000\",\n },\n \"phase\": {\n \"method\": \"None\",\n \"padding_type\": \"edge\",\n },\n },\n \"processing_option\": {\n \"build_sino\": {\n \"axis_correction\": \"None\",\n \"enable_halftomo\": True,\n },\n \"flatfield\": {\n \"binning\": \"11\",\n \"do_flat_distortion\": False,\n },\n },\n}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# %pylab"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# imshow(data[0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### HDF5Volume\n",
"#### constructor\n",
"Here we will only focus on providing a `file_path` and a `data_path` to the constructor. From it it will deduce default data and metadata `data_path`.\n",
"\n",
"This way allow us to get a identifier that can be reused. But users can also provide directly an url for data and one for the metadata. But this is for advance usage and it will not be detailled here."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# users can define the volume will all the information\n",
"volume = HDF5Volume(\n",
" file_path=\"test_volume.hdf5\",\n",
" data_path=\"entry0000/reconstruction\",\n",
" data=data,\n",
" metadata=metadata,\n",
")\n",
"# or define them from properties\n",
"volume = HDF5Volume()\n",
"volume.file_path = \"test_volume.hdf5\"\n",
"volume.data_path = \"entry0000/reconstruction\"\n",
"volume.data = data # data must be None or 3D numpy array\n",
"volume.metadata = metadata # metadata must be None or an instance of dict"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# data can be access from the data property\n",
"# imshow(volume.data[0])\n",
"# metadata from the metadata property.\n",
"volume.metadata"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### saving"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# save the volume\n",
"volume.save()\n",
"# display the contents of the volume file\n",
"\n",
"H5Glance(\"test_volume.hdf5\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### loading\n",
"once the data is saved on disk then we can retrieve it from the same class"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"volume_loaded = HDF5Volume(\n",
" file_path=\"test_volume.hdf5\", data_path=\"entry0000/reconstruction\"\n",
")\n",
"volume_loaded.load()\n",
"# imshow(volume_loaded.data[0])\n",
"print(volume_loaded.metadata)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To avoid heavy memory consumption the data and metadata will not be loaded automatically.\n",
"In order to purge cache you can also call `volume.clear_cache()` or set data and metadata to None like:\n",
"```\n",
"volume.data = None\n",
"volume.metadata = None\n",
"```\n",
"Note: if data or metadata is None call to saving function can raise ValueError exception."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### identifier\n",
"Each subclass of `VolumeBase` define what we call an identifier. The goal is that from this identifier a user can retrieve a Volume.\n",
"An identifier can be save as an instance of `Identifier` or as it string representation.\n",
"An identifier looks like:\n",
"\n",
"```\n",
"scheme:tomo_type:data_path@data_file\n",
"```\n",
"For now:\n",
"* scheme can be `hdf5`, `edf`, `jp2k`, `tiff` and `tiff_3d`\n",
"* tomo_type can be `scan` or `volume`\n",
"\n",
"*Note: This is pretty convienient for gui for example or to 'copy/paste' a reference to a volume from one application to another.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### get identifier"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"identifier = volume_loaded.get_identifier()\n",
"print(identifier, type(identifier))\n",
"print(identifier.to_str(), type(identifier.to_str()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### use identifier to retrieve a volume"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from volio.factory import Factory\n\nretrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier)\nassert isinstance(retrieve_volume, HDF5Volume)\nretrieve_volume = Factory.create_tomo_object_from_identifier(\n identifier=identifier.to_str()\n)\nassert isinstance(retrieve_volume, HDF5Volume)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# clean workspace\n",
"if os.path.exists(\"test_volume.hdf5\"):\n",
" os.remove(\"test_volume.hdf5\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### EDFVolume\n",
"#### constructor\n",
"\n",
"EDFVolume will store each frame as `folder_prefix_index.edf` and metadata to `folder_prefix_infos.txt`.\n",
"\n",
"So the default constructor only require a path to a folder where to save the data.\n",
"\n",
"This folder path will also be the identifier"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"volume = EDFVolume(folder=\"edf_volume\", data=data, metadata=metadata)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or as for the HDF5Volume you can provide data and metadata once the Volume has been defined"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"volume = EDFVolume(folder=\"edf_volume\")\n",
"volume.data = data\n",
"volume.metadata = metadata"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"access to data and metadata is the same for any instance of VolumeBase, from `data` and `metadata` properties"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# imshow(volume.data[0])\n",
"print(volume.metadata)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### saving"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# save the volume\n",
"volume.save()\n",
"\n",
"# if you want to overwrite existing data you can set the overwrite property\n",
"volume.overwrite = True\n",
"\n",
"# display the contents of the folder\n",
"print(os.listdir(\"edf_volume\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# open one file containing a frame\n",
"\n",
"file_frame_0 = os.path.join(\"edf_volume\", \"edf_volume_000000.edf\")\n",
"# imshow(fabio.open(file_frame_0).data)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# display contents of the infos.txt file\n",
"metadata_file = os.path.join(\"edf_volume\", \"edf_volume_infos.txt\")\n",
"with open(metadata_file, \"r\") as of:\n",
" print(\"\".join(of.readlines()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### loading"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"volume_loaded = EDFVolume(folder=\"edf_volume\")\n",
"volume_loaded.load()\n",
"# imshow(volume_loaded.data[1])\n",
"print(volume_loaded.metadata)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### identifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### get_identifier"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"identifier = volume_loaded.get_identifier()\n",
"print(identifier, type(identifier))\n",
"print(identifier.to_str(), type(identifier.to_str()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### use identifier to retrieve a volume"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from volio.factory import Factory\n\nretrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier)\nassert isinstance(retrieve_volume, EDFVolume)\nretrieve_volume = Factory.create_tomo_object_from_identifier(\n identifier=identifier.to_str()\n)\nassert isinstance(retrieve_volume, EDFVolume)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# clean\n",
"import shutil\n",
"\n",
"if os.path.exists(\"edf_volume\"):\n",
" shutil.rmtree(\"edf_volume\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### JP2KVolume\n",
"\n",
"The API of `JP2KVolume` is the same as `EDFVolume`. So, please have a look at the `EDFVolume` tutorial.\n",
"\n",
"`JP2KVolume` requires `glymur` to be installed. You can check availability with `has_glymur`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TIFFVolume\n",
"\n",
"The API of `TIFFVolume` is the same as `EDFVolume`. So, please have a look at the `EDFVolume` tutorial."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### MultiTIFFVolume"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### constructor"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"volume = MultiTIFFVolume(file_path=\"multitiff_file.tiff\", data=data, metadata=metadata)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# access data and metadata\n",
"# imshow(volume.data[0])\n",
"print(volume.metadata)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### saving"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"volume.save()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### loading"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"volume_loaded = MultiTIFFVolume(file_path=\"multitiff_file.tiff\")\n",
"assert volume_loaded.data is None\n",
"assert volume_loaded.metadata is None\n",
"volume_loaded.load()\n",
"# imshow(volume_loaded.data[0])\n",
"print(volume_loaded.metadata)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### identifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### get_identifier"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"identifier = volume_loaded.get_identifier()\n",
"print(identifier, type(identifier))\n",
"print(identifier.to_str(), type(identifier.to_str()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### use identifier to retrieve a volume"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from volio.factory import Factory\n\nretrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier)\nassert isinstance(retrieve_volume, MultiTIFFVolume)\nretrieve_volume = Factory.create_tomo_object_from_identifier(\n identifier=identifier.to_str()\n)\nassert isinstance(retrieve_volume, MultiTIFFVolume)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# clean\n",
"if os.path.exists(\"multitiff_file.tiff\"):\n",
" os.remove(\"multitiff_file.tiff\")\n",
"if os.path.exists(retrieve_volume.metadata_url.file_path()):\n",
" os.remove(retrieve_volume.metadata_url.file_path())"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/pyproject.toml 0000664 0000000 0000000 00000005075 15241025157 0024463 0 ustar 00root root 0000000 0000000 [project]
name = "volio"
version = "0.0.0dev1"
authors = [
{ name = "Henri Payno", email = "henri.payno@esrf.fr" },
{ name = "Pierre Paleo", email = "pierre.paleo@esrf.fr" },
{ name = "Pierre-Olivier Autran", email = "pierre-olivier.autran@esrf.fr" },
{ name = "Jérôme Lesaint", email = "jerome.lesaint@esrf.fr" },
]
description = "Volume load/save for tomography data at ESRF"
readme = { file = "README.md", content-type = "text/markdown" }
license = "MIT"
requires-python = ">=3.10"
keywords = ["tomography", "esrf", "volume", "reconstruction"]
classifiers = [
"Intended Audience :: Education",
"Intended Audience :: Science/Research",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Environment :: Console",
"Operating System :: POSIX",
"Natural Language :: English",
"Topic :: Scientific/Engineering :: Physics",
"Topic :: Software Development :: Libraries :: Python Modules"
]
dependencies = [
"tomobase>=0.0.0dev0",
"h5py>=3.0",
"silx>=2.0",
"numpy",
"pint",
"packaging>=22.0",
"fabio",
"defusedxml",
"dicttoxml",
"tifffile>=2023.7.4",
]
[project.optional-dependencies]
compression_plugins = [
"hdf5plugin",
"blosc2-grok",
]
full_no_compression_plugins = [
"glymur",
"tqdm",
"tifffile[zarr]>=2023.7.4",
"zarr",
]
full = [
"volio[full_no_compression_plugins]",
"volio[compression_plugins]",
]
test = [
"volio[full]",
"pytest",
"ruff",
]
doc = [
"Sphinx>=4.0.0, <5.2.0",
"nbsphinx",
"pandoc",
"ipykernel",
"jupyter_client",
"nbconvert",
"h5glance",
"pydata_sphinx_theme",
"sphinx-design",
"sphinx-autodoc-typehints",
"myst-parser",
"sphinx-copybutton",
]
[build-system]
requires = [
"setuptools>=61.0",
"wheel",
"numpy"
]
build-backend = "setuptools.build_meta"
[tool.sphinx]
source-dir = "./doc"
[tool.ruff]
line-length = 88
target-version = "py310"
[tool.ruff.lint]
# `select` (not `extend-select`): pin the rule set explicitly so a new ruff
# release changing its defaults cannot break the CI.
select = [
"E", # pycodestyle errors
"F", # pyflakes
"I", # isort
"S", # flake8-bandit
]
ignore = [
"E501", # line length: `ruff format` is the authority
"S101", # assert
]
[tool.ruff.lint.isort]
known-first-party = ["volio"]
[tool.setuptools.packages.find]
where = ["."]
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/ 0000775 0000000 0000000 00000000000 15241025157 0022670 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/__init__.py 0000664 0000000 0000000 00000000503 15241025157 0024777 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""volio: volume load/save for tomography data at ESRF"""
from importlib.metadata import version as _get_version
from .volumes import ( # noqa: F401
EDFVolume,
HDF5Volume,
JP2KVolume,
MultiTIFFVolume,
RawVolume,
TIFFVolume,
has_glymur,
)
__version__ = _get_version("volio")
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/drac/ 0000775 0000000 0000000 00000000000 15241025157 0023601 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/drac/__init__.py 0000664 0000000 0000000 00000000236 15241025157 0025713 0 ustar 00root root 0000000 0000000 """drac utils (icat successor). Used for data publication to the data portal"""
from .mapper import flatten_vol_metadata, map_nabu_keys_to_drac # noqa F401
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/drac/mapper.py 0000664 0000000 0000000 00000004501 15241025157 0025437 0 ustar 00root root 0000000 0000000 """mapper functions to convert from raw data fields (like nabu reconstructed volume metadata) to drac / icat fields"""
from __future__ import annotations
import numpy
from .recons_vol_mapping import DRAC_NABU_MAPPING, DracRawDataMapping
__all__ = [
"flatten_vol_metadata",
"map_nabu_keys_to_drac",
]
def flatten_vol_metadata(metadata: dict, parent_key: str = "", sep="/") -> dict:
"""
Flatten a nested dictionary (expected to come from nabu volume metadata) and converting arrays to lists
and numpy types to serializable types. The resulting dictionary is JSON-serializable.
:param metadata: The dictionary to flatten.
:param parent_key: The base key string used for recursive calls.
:param sep: The separator used between parent and child keys.
:return: The flattened and type-converted dictionary.
"""
# Exclusion list
items = []
for k, v in metadata.items():
# Construct the new key
new_key = f"{parent_key}{sep}{k}" if parent_key else k
if isinstance(v, dict):
items.extend(flatten_vol_metadata(v, new_key, sep=sep).items())
else:
# Convert numpy arrays to lists
if isinstance(v, numpy.ndarray):
v = v.tolist()
# Convert numpy boolean to native boolean
elif isinstance(v, numpy.bool_):
v = bool(v)
# Convert numpy integers to native int
elif isinstance(v, (numpy.int64, numpy.int32)):
v = int(v)
# Convert numpy floats to native float
elif isinstance(v, (numpy.float64, numpy.float32)):
v = float(v)
items.append((new_key, v))
return dict(items)
def map_nabu_keys_to_drac(flatten_dict: dict) -> dict:
"""
map a set of **nabu** keys to drac 'TOMOReconstruction' definition (https://gitlab.esrf.fr/icat/hdf5-master-config/-/blob/master/hdf5_cfg.xml)
"""
drac_dict = {}
for map in DRAC_NABU_MAPPING:
assert isinstance(map, DracRawDataMapping), (
f"map should be an instance of {DracRawDataMapping}. Got {type(map)}"
)
if map.raw_data_key in flatten_dict:
raw_data = flatten_dict[map.raw_data_key]
drac_value = map.to_drac(raw_data)
drac_dict[map.drac_key] = drac_value
return drac_dict
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/drac/recons_vol_mapping.py 0000664 0000000 0000000 00000017055 15241025157 0030047 0 ustar 00root root 0000000 0000000 """Defines mapping rule to be applied for reconstructed volume to drac"""
from __future__ import annotations
import numpy
import pint
_ureg = pint.get_application_registry()
class DracRawDataMapping:
"""
Map drac/icat metadata from / to raw data. Raw data in this context is expected to be a reconstructed volume (by nabu)
This layer ease handling of unit conversion as data type (from a single field to multi field structures.)
A raw data / nabu field can be split into several drac/icat field. And several drac/icat field can be concatenate to a single raw_data/nabu field.
This is the purpose of the 'raw_data_field_index', 'raw_data_field_size' and 'raw_data_type_constructor' parameters.
The over way around is not forseen
"""
def __init__(
self,
drac_key: str,
raw_data_key: str,
raw_data_field_index: int | None = None,
raw_data_type_constructor=None,
raw_data_unit: str | pint.Unit = None,
drac_unit: str | pint.Unit = None,
) -> None:
self.drac_key = drac_key
self.raw_data_key = raw_data_key
self.raw_data_field_index = raw_data_field_index
self.raw_data_type_constructor = raw_data_type_constructor
self._raw_data_unit = raw_data_unit
self._drac_unit = drac_unit
def to_drac(self, raw_value):
"convert a raw_data (nabu) field to an icat field"
if self.raw_data_field_index is not None:
assert isinstance(self.raw_data_field_index, int)
assert isinstance(raw_value, (list, tuple, numpy.array))
raw_value = raw_value[self.raw_data_field_index]
if self._raw_data_unit is not None and self._drac_unit is not None:
return (raw_value * self._raw_data_unit).to(self._drac_unit).magnitude
else:
return raw_value
DRAC_NABU_MAPPING = (
DracRawDataMapping(
drac_key="TOMOReconstruction_angle_offset", # float
raw_data_key="nabu_config/reconstruction/angle_offset",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_angles_file", # str - file path
raw_data_key="nabu_config/reconstruction/angles_file",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_axis_correction_file", # str - file path
raw_data_key="nabu_config/reconstruction/axis_correction_file",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_centered_axis", # bool
raw_data_key="nabu_config/reconstruction/centered_axis",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_clip_outer_circle", # bool
raw_data_key="nabu_config/reconstruction/clip_outer_circle",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_cor_options", # str
raw_data_key="nabu_config/reconstruction/cor_options",
),
DracRawDataMapping(
drac_key="endTime", # str
raw_data_key="date",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_enable_halftomo", # bool
raw_data_key="nabu_config/reconstruction/enable_halftomo",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_end_x", # int (px)
raw_data_key="nabu_config/reconstruction/end_x",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_end_y", # int (px)
raw_data_key="nabu_config/reconstruction/end_y",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_end_z", # int (px)
raw_data_key="nabu_config/reconstruction/end_z",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_fbp_filter_cutoff", # float
raw_data_key="nabu_config/reconstruction/fbp_filter_cutoff",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_fbp_filter_type", # str
raw_data_key="nabu_config/reconstruction/fbp_filter_type",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_method", # str
raw_data_key="nabu_config/reconstruction/method",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_optim_algorithm", # str
raw_data_key="nabu_config/reconstruction/optim_algorithm",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_padding_type", # str
raw_data_key="nabu_config/reconstruction/padding_type",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_preconditioning_filter", # str
raw_data_key="nabu_config/reconstruction/preconditioning_filter",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_rotation_axis_position", # float
raw_data_key="nabu_config/reconstruction/rotation_axis_position",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_start_x", # int
raw_data_key="nabu_config/reconstruction/start_x",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_start_y", # int
raw_data_key="nabu_config/reconstruction/start_y",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_start_z", # int
raw_data_key="nabu_config/reconstruction/start_z",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_translation_movements_file", # str
raw_data_key="nabu_config/reconstruction/translation_movements_file",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_weight_tv", # float
raw_data_key="nabu_config/reconstruction/weight_tv",
),
DracRawDataMapping(
drac_key="TOMOReconstruction_voxel_size_x", # float
raw_data_key="processing_options/reconstruction/voxel_size_cm",
raw_data_field_index=0,
raw_data_unit=_ureg.centimeter,
drac_unit=_ureg.micrometer,
),
DracRawDataMapping(
drac_key="TOMOReconstruction_voxel_size_y", # float
raw_data_key="processing_options/reconstruction/voxel_size_cm",
raw_data_field_index=1,
raw_data_unit=_ureg.centimeter,
drac_unit=_ureg.micrometer,
),
DracRawDataMapping(
drac_key="TOMOReconstruction_voxel_size_z", # float
raw_data_key="processing_options/reconstruction/voxel_size_cm",
raw_data_field_index=2,
raw_data_unit=_ureg.centimeter,
drac_unit=_ureg.micrometer,
),
# phase
DracRawDataMapping(
drac_key="TOMOReconstructionPhase_ctf_advanced_params", # str
raw_data_key="nabu_config/phase/ctf_advanced_params",
),
DracRawDataMapping(
drac_key="TOMOReconstructionPhase_ctf_geometry", # str
raw_data_key="nabu_config/phase/ctf_geometry",
),
DracRawDataMapping(
drac_key="TOMOReconstructionPhase_delta_beta", # float
raw_data_key="nabu_config/phase/delta_beta",
),
DracRawDataMapping(
drac_key="TOMOReconstructionPhase_detector_sample_distance", # float
raw_data_key="nabu_config/phase/distance_m",
raw_data_unit=_ureg.meter,
drac_unit=_ureg.millimeter,
),
DracRawDataMapping(
drac_key="TOMOReconstructionPhase_method", # str
raw_data_key="nabu_config/phase/method",
),
DracRawDataMapping(
drac_key="TOMOReconstructionPhase_padding_type", # ste
raw_data_key="nabu_config/phase/padding_type",
),
DracRawDataMapping(
drac_key="TOMOReconstructionPhase_unsharp_coeff", # float
raw_data_key="nabu_config/phase/unsharp_coeff",
),
DracRawDataMapping(
drac_key="TOMOReconstructionPhase_unsharp_method", # str
raw_data_key="nabu_config/phase/unsharp_method",
),
DracRawDataMapping(
drac_key="TOMOReconstructionPhase_unsharp_sigma", # float
raw_data_key="nabu_config/phase/unsharp_sigma",
),
)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/factory.py 0000664 0000000 0000000 00000011333 15241025157 0024712 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""Contains the Factory class, to build volio volumes from identifiers"""
from __future__ import annotations
from urllib.parse import urlparse
from tomobase.identifiers import BaseIdentifier
from tomobase.tomoobject import TomoObject
from tomobase.utils.url import split_path
from .identifiers.base import VolumeIdentifier
from .identifiers.edfidentifier import EDFVolumeIdentifier
from .identifiers.hdf5identifier import HDF5VolumeIdentifier
from .identifiers.jp2kidentifier import JP2KVolumeIdentifier
from .identifiers.rawidentifier import RawVolumeIdentifier
from .identifiers.tiffidentifier import MultiTiffVolumeIdentifier, TIFFVolumeIdentifier
from .volumes.edfvolume import EDFVolume
from .volumes.hdf5volume import HDF5Volume
from .volumes.jp2kvolume import JP2KVolume
from .volumes.rawvolume import RawVolume
from .volumes.tiffvolume import MultiTIFFVolume, TIFFVolume
__all__ = [
"Factory",
]
class Factory:
"""
Factory to build volio volumes (`VolumeBase` subclasses) from identifiers.
Example identifier for each volume format:
* HDF5Volume: "hdf5:volume:/path/to/file_path?path=entry0000"
* EDFVolume: "edf:volume:/path/to/my/my_folder"
* TIFFVolume: "tiff:volume:/path/to/my/my_folder"
* MultiTIFFVolume: "tiff3d:volume:/path/to/tiff_file.tif"
* JP2KVolume: "jp2k:volume:/path/to/my/my_folder"
* RawVolume: "raw:volume:/path/to/my/my_volume.vol"
.. code-block:: python
from volio.factory import Factory
volume = Factory.create_tomo_object_from_identifier(
"hdf5:volume:/path/to/reconstruction.hdf5?path=entry0000/reconstruction"
)
"""
@staticmethod
def create_tomo_object_from_identifier(
identifier: str | VolumeIdentifier,
) -> TomoObject:
"""
Create an instance of VolumeBase from its identifier if possible
:param identifier: identifier of the volume
:raises TypeError: if identifier is not a str or a VolumeIdentifier
:raises ValueError: if identifier cannot be converted back to an instance of VolumeBase
"""
if not isinstance(identifier, (str, BaseIdentifier)):
raise TypeError(
f"identifier is expected to be a str or an instance of {BaseIdentifier} not {type(identifier)}. {type(identifier)} provided"
)
# step 1: convert identifier to an instance of BaseIdentifier if necessary
if isinstance(identifier, str):
info = urlparse(identifier)
paths = split_path(info.path)
scheme = info.scheme
if len(paths) == 2:
tomo_type, _ = paths
if tomo_type != VolumeIdentifier.TOMO_TYPE:
raise ValueError(
f"identifier fits tomo_type '{tomo_type}', expected '{VolumeIdentifier.TOMO_TYPE}'"
)
elif len(paths) != 1:
raise ValueError("Failed to parse path string:", info.path)
if scheme == "edf":
identifier = EDFVolumeIdentifier.from_str(identifier=identifier)
elif scheme == "hdf5":
identifier = HDF5VolumeIdentifier.from_str(identifier=identifier)
elif scheme == "tiff":
identifier = TIFFVolumeIdentifier.from_str(identifier=identifier)
elif scheme == "tiff3d":
identifier = MultiTiffVolumeIdentifier.from_str(identifier=identifier)
elif scheme == "jp2k":
identifier = JP2KVolumeIdentifier.from_str(identifier=identifier)
elif scheme == "raw":
identifier = RawVolumeIdentifier.from_str(identifier=identifier)
else:
raise ValueError(f"Scheme {scheme} is not recognized")
# step 2: convert identifier to a VolumeBase
assert isinstance(identifier, BaseIdentifier)
if identifier.tomo_type != VolumeIdentifier.TOMO_TYPE:
raise ValueError(
f"identifier is a '{identifier.tomo_type}', expected '{VolumeIdentifier.TOMO_TYPE}'"
)
scheme = identifier.scheme
if scheme == "edf":
return EDFVolume.from_identifier(identifier=identifier)
elif scheme == "hdf5":
return HDF5Volume.from_identifier(identifier=identifier)
elif scheme == "jp2k":
return JP2KVolume.from_identifier(identifier=identifier)
elif scheme == "tiff":
return TIFFVolume.from_identifier(identifier=identifier)
elif scheme == "tiff3d":
return MultiTIFFVolume.from_identifier(identifier=identifier)
elif scheme == "raw":
return RawVolume.from_identifier(identifier=identifier)
else:
raise ValueError(f"Scheme {scheme} not recognized")
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/identifiers/ 0000775 0000000 0000000 00000000000 15241025157 0025175 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/identifiers/__init__.py 0000664 0000000 0000000 00000000703 15241025157 0027306 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""identifiers: string-convertible identifiers used to reference volio volumes"""
from .edfidentifier import EDFVolumeIdentifier # noqa: F401
from .hdf5identifier import HDF5VolumeIdentifier # noqa: F401
from .jp2kidentifier import JP2KVolumeIdentifier # noqa: F401
from .rawidentifier import RawVolumeIdentifier # noqa: F401
from .tiffidentifier import ( # noqa: F401
MultiTiffVolumeIdentifier,
TIFFVolumeIdentifier,
)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/identifiers/base.py 0000664 0000000 0000000 00000000214 15241025157 0026456 0 ustar 00root root 0000000 0000000 from tomobase.identifiers.base import BaseIdentifier as _BaseIdentifier
class VolumeIdentifier(_BaseIdentifier):
TOMO_TYPE = "volume"
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/identifiers/edfidentifier.py 0000664 0000000 0000000 00000001742 15241025157 0030354 0 ustar 00root root 0000000 0000000 # coding: utf-8
from tomobase.identifiers import BaseIdentifier
from tomobase.identifiers.folderidentifiers import (
BaseFolderAndfilePrefixIdentifierMixIn,
)
from tomobase.utils import docstring
from .base import VolumeIdentifier
__all__ = ["EDFVolumeIdentifier"]
class EDFVolumeIdentifier(BaseFolderAndfilePrefixIdentifierMixIn, VolumeIdentifier):
"""Identifier specific to EDF volume"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs, tomo_type=VolumeIdentifier.TOMO_TYPE)
@property
@docstring(BaseIdentifier)
def scheme(self) -> str:
return "edf"
@staticmethod
def from_str(identifier):
from volio.volumes.edfvolume import EDFVolume
return (
BaseFolderAndfilePrefixIdentifierMixIn._from_str_to_single_frame_identifier(
identifier=identifier,
SingleFrameIdentifierClass=EDFVolumeIdentifier,
ObjClass=EDFVolume,
)
)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/identifiers/hdf5identifier.py 0000664 0000000 0000000 00000002552 15241025157 0030444 0 ustar 00root root 0000000 0000000 # coding: utf-8
from urllib.parse import urlparse
from tomobase.identifiers.hdf5identifier import HDF5IdentifierMixIn
from tomobase.utils.url import UrlSettings, split_path, split_query
from volio.identifiers.base import VolumeIdentifier
__all__ = ["HDF5VolumeIdentifier"]
class HDF5VolumeIdentifier(HDF5IdentifierMixIn, VolumeIdentifier):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs, tomo_type=VolumeIdentifier.TOMO_TYPE)
@staticmethod
def from_str(identifier):
info = urlparse(identifier)
paths = split_path(info.path)
if len(paths) == 1:
hdf5_file = paths[0]
tomo_type = None
elif len(paths) == 2:
tomo_type, hdf5_file = paths
else:
raise ValueError("Failed to parse path string:", info.path)
if tomo_type is not None and tomo_type != VolumeIdentifier.TOMO_TYPE:
raise TypeError(
f"provided identifier fits {tomo_type} and not {VolumeIdentifier.TOMO_TYPE}"
)
queries = split_query(info.query)
entry = queries.get(UrlSettings.DATA_PATH_KEY, None)
if entry is None:
raise ValueError("expects to get a data_path")
from volio.volumes.hdf5volume import HDF5Volume
return HDF5VolumeIdentifier(object=HDF5Volume, hdf5_file=hdf5_file, entry=entry)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/identifiers/jp2kidentifier.py 0000664 0000000 0000000 00000001724 15241025157 0030464 0 ustar 00root root 0000000 0000000 # coding: utf-8
from tomobase.identifiers.folderidentifiers import (
BaseFolderAndfilePrefixIdentifierMixIn,
)
from tomobase.utils import docstring
from volio.identifiers.base import VolumeIdentifier
__all__ = [
"JP2KVolumeIdentifier",
]
class JP2KVolumeIdentifier(BaseFolderAndfilePrefixIdentifierMixIn, VolumeIdentifier):
"""Identifier specific to JP2K volume"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs, tomo_type=VolumeIdentifier.TOMO_TYPE)
@property
@docstring(VolumeIdentifier)
def scheme(self) -> str:
return "jp2k"
@staticmethod
def from_str(identifier):
from volio.volumes.jp2kvolume import JP2KVolume
return (
BaseFolderAndfilePrefixIdentifierMixIn._from_str_to_single_frame_identifier(
identifier=identifier,
SingleFrameIdentifierClass=JP2KVolumeIdentifier,
ObjClass=JP2KVolume,
)
)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/identifiers/rawidentifier.py 0000664 0000000 0000000 00000002603 15241025157 0030404 0 ustar 00root root 0000000 0000000 # coding: utf-8
import os
from tomobase.utils import docstring
from volio.identifiers.base import VolumeIdentifier
__all__ = [
"RawVolumeIdentifier",
]
class RawVolumeIdentifier(VolumeIdentifier):
"""Identifier for the .vol volume"""
def __init__(self, object, file_path):
super().__init__(object)
self._file_path = os.path.abspath(os.path.abspath(file_path))
@docstring(VolumeIdentifier)
def short_description(self) -> str:
return f"{self.scheme}:{self.tomo_type}:{os.path.basename(self._file_path)}"
@property
def file_path(self):
return self._file_path
@property
@docstring(VolumeIdentifier)
def scheme(self) -> str:
return "raw"
def __str__(self):
return f"{self.scheme}:{self.tomo_type}:{self._file_path}"
def __eq__(self, other):
if isinstance(other, RawVolumeIdentifier):
return (
self.tomo_type == other.tomo_type
and self._file_path == other._file_path
)
else:
return False
def __hash__(self):
return hash(self._file_path)
@staticmethod
def from_str(identifier):
identifier_no_scheme = identifier.split(":")[-1]
vol_file = identifier_no_scheme
from volio.volumes.rawvolume import RawVolume
return RawVolumeIdentifier(object=RawVolume, file_path=vol_file)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/identifiers/tiffidentifier.py 0000664 0000000 0000000 00000004411 15241025157 0030542 0 ustar 00root root 0000000 0000000 # coding: utf-8
import os
from tomobase.identifiers.folderidentifiers import (
BaseFolderAndfilePrefixIdentifierMixIn,
)
from tomobase.utils import docstring
from volio.identifiers.base import VolumeIdentifier
__all__ = ["TIFFVolumeIdentifier", "MultiTiffVolumeIdentifier"]
class TIFFVolumeIdentifier(BaseFolderAndfilePrefixIdentifierMixIn, VolumeIdentifier):
"""Identifier specific to (single frame) tiff volume"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs, tomo_type=VolumeIdentifier.TOMO_TYPE)
@property
@docstring(VolumeIdentifier)
def scheme(self) -> str:
return "tiff"
@staticmethod
def from_str(identifier):
from volio.volumes.tiffvolume import TIFFVolume
return (
BaseFolderAndfilePrefixIdentifierMixIn._from_str_to_single_frame_identifier(
identifier=identifier,
SingleFrameIdentifierClass=TIFFVolumeIdentifier,
ObjClass=TIFFVolume,
)
)
class MultiTiffVolumeIdentifier(VolumeIdentifier):
def __init__(self, object, tiff_file):
super().__init__(object)
self._file_path = os.path.abspath(os.path.abspath(tiff_file))
@docstring(VolumeIdentifier)
def short_description(self) -> str:
return f"{self.scheme}:{self.tomo_type}:{os.path.basename(self._file_path)}"
@property
def file_path(self):
return self._file_path
@property
@docstring(VolumeIdentifier)
def scheme(self) -> str:
return "tiff3d"
def __str__(self):
return f"{self.scheme}:{self.tomo_type}:{self._file_path}"
def __eq__(self, other):
if isinstance(other, MultiTiffVolumeIdentifier):
return (
self.tomo_type == other.tomo_type
and self._file_path == other._file_path
)
else:
return super().__eq__(other)
def __hash__(self):
return hash(self._file_path)
@staticmethod
def from_str(identifier):
identifier_no_scheme = identifier.split(":")[-1]
# TODO: check tomo_type ?
tiff_file = identifier_no_scheme
from volio.volumes.tiffvolume import TIFFVolume
return MultiTiffVolumeIdentifier(object=TIFFVolume, tiff_file=tiff_file)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/ 0000775 0000000 0000000 00000000000 15241025157 0024032 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/__init__.py 0000664 0000000 0000000 00000000000 15241025157 0026131 0 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/conftest.py 0000664 0000000 0000000 00000000401 15241025157 0026224 0 ustar 00root root 0000000 0000000 from tempfile import TemporaryDirectory
import pytest
@pytest.fixture(scope="session", autouse=True)
def changetmp(request):
with TemporaryDirectory(prefix="pytest-volio-") as temp_dir:
request.config.option.basetemp = temp_dir
yield
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/mock.py 0000664 0000000 0000000 00000003355 15241025157 0025343 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""module to mock volume"""
from __future__ import annotations
from enum import Enum
from typing import Sized
import numpy
from silx.image.phantomgenerator import PhantomGenerator
__all__ = ["Scene", "create_volume"]
class Scene(Enum):
SHEPP_LOGAN = "Shepp-Logan"
def create_volume(
frame_dims: int | tuple, z_size: int, scene: Scene = Scene.SHEPP_LOGAN
) -> numpy.ndarray:
"""
create a numpy array of the requested scheme for a total of frames_dimes*z_size elements
:param tuple frame_dims: 2d tuple of frame dimensions
:param z_size: number of elements on the volume on z axis
:param Scene scene: scene to compose
"""
scene = Scene(scene)
if not isinstance(z_size, int):
raise TypeError(
f"z_size is expected to be an instance of int not {type(z_size)}"
)
if scene is Scene.SHEPP_LOGAN:
if isinstance(frame_dims, Sized):
if not len(frame_dims) == 2:
raise ValueError(
f"frame_dims is expected to be an integer or a list of two integers. Not {frame_dims}"
)
if frame_dims[0] != frame_dims[1]:
raise ValueError(
f"{scene} only handle square frame. Frame width and height should be the same"
)
else:
dim = frame_dims[0]
elif isinstance(frame_dims, int):
dim = frame_dims
else:
raise TypeError(
f"frame_dims is expected to be a list of two integers or an integer. Not {frame_dims}"
)
return numpy.asarray(
[PhantomGenerator.get2DPhantomSheppLogan(dim) * 10000.0] * z_size
)
else:
raise NotImplementedError
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_drac_utils.py 0000664 0000000 0000000 00000025464 15241025157 0027607 0 ustar 00root root 0000000 0000000 import json
import numpy
import pint
from volio.drac.mapper import (
flatten_vol_metadata,
map_nabu_keys_to_drac,
)
_ureg = pint.get_application_registry()
_NABU_VOL_RAW_METADATA = {
"@NX_class": "NXcollection",
"date": "2024-10-02T17:42:01",
"nabu_config": {
"dataset": {
"binning": 1,
"binning_z": 1,
"darks_flats_dir": "None",
"exclude_projections": "None",
"hdf5_entry": "/entry0000",
"location": "/mnt/multipath-shares/tmp_14_days/payno/test_deployment/bambou_hercules_0001.nx",
"nexus_version": 1.0,
"overwrite_metadata": "",
"projections_subsampling": numpy.array([1, 0]),
},
"output": {
"file_format": "hdf5",
"file_prefix": "bambou_hercules_0001_vol",
"float_clip_values": "None",
"jpeg2000_compression_ratio": "None",
"location": "/mnt/multipath-shares/tmp_14_days/payno/test_deployment/reconstructed_volumes",
"overwrite_results": True,
"tiff_single_file": False,
},
"phase": {
"distance_cm": 210,
"distance_m": 2.1,
"ctf_advanced_params": "length_scale=1e-05; lim1=1e-05; lim2=0.2; normalize_by_mean=True",
"ctf_geometry": "z1_v=None; z1_h=None; detec_pixel_size=None; magnification=True",
"delta_beta": 100.0,
"method": "paganin",
"padding_type": "edge",
"unsharp_coeff": 0.0,
"unsharp_method": "gaussian",
"unsharp_sigma": 0.0,
"pixel_size_m": 3.0325e-06,
"pixel_size_microns": 3.0325,
"energy_kev": 130,
},
"pipeline": {
"resume_from_step": "None",
"save_steps": "None",
"steps_file": "None",
"verbosity": "info",
},
"postproc": {"histogram_bins": 1000000, "output_histogram": True},
"preproc": {
"autotilt_options": "None",
"ccd_filter_enabled": False,
"ccd_filter_threshold": 0.04,
"detector_distortion_correction": "None",
"detector_distortion_correction_options": "None",
"dff_sigma": 0.0,
"double_flatfield_enabled": False,
"flat_distortion_correction_enabled": False,
"flat_distortion_params": "tile_size=100; "
"interpolation_kind='linear'; "
"padding_mode='edge'; "
"correction_spike_threshold=None",
"flatfield": True,
"log_max_clip": 10.0,
"log_min_clip": 1e-06,
"normalize_srcurrent": True,
"processes_file": "None",
"rotate_projections": "None",
"rotate_projections_center": "None",
"sino_normalization": "None",
"sino_normalization_file": "",
"sino_rings_correction": "None",
"sino_rings_options": "sigma=1.0 ; levels=10 ; padding=False",
"take_logarithm": True,
"tilt_correction": "None",
},
"reconstruction": {
"angle_offset": 0.0,
"angles_file": "None",
"axis_correction_file": "None",
"centered_axis": True,
"clip_outer_circle": True,
"cor_options": "side='from_file'",
"cor_slice": "None",
"enable_halftomo": True,
"end_x": -1,
"end_y": -1,
"end_z": 180,
"fbp_filter_cutoff": 1.0,
"fbp_filter_type": "ramlak",
"iterations": 200,
"method": "FBP",
"optim_algorithm": "chambolle-pock",
"padding_type": "edge",
"positivity_constraint": True,
"preconditioning_filter": True,
"rotation_axis_position": 2060.0,
"sample_detector_dist": "None",
"source_sample_dist": "None",
"start_x": 0,
"start_y": 0,
"start_z": 150,
"translation_movements_file": "None",
"weight_tv": 0.01,
},
"resources": {
"cpu_workers": 0,
"gpu_id": numpy.array([], dtype=numpy.float64),
"gpus": 1,
"memory_per_node": numpy.array([90.0, 1.0]),
"method": "local",
"queue": "gpu",
"threads_per_node": numpy.array([100.0, 1.0]),
"walltime": numpy.array([1, 0, 0]),
},
},
"processing_options": {
"flatfield": {
"binning": numpy.array([1, 1]),
"do_flat_distortion": False,
"flat_distortion_params": {
"correction_spike_threshold": "None",
"interpolation_kind": "linear",
"padding_mode": "edge",
"tile_size": 100,
},
"flats_srcurrent": numpy.array([0.19458202]),
"normalize_srcurrent": True,
"projs_indices": numpy.array([81, 82, 83, 4478, 4479, 4480]),
"radios_srcurrent": numpy.array(
[0.19458, 0.1945795, 0.19457899, 0.19364, 0.19364, 0.19364],
dtype=numpy.float32,
),
},
"histogram": {"histogram_bins": 1000000},
"read_chunk": {
"binning": numpy.array([1, 1]),
"dataset_subsampling": numpy.array([1, 0]),
"files": {
"100": "silx:///mnt/multipath-shares/tmp_14_days/payno/test_deployment/bambou_hercules_0001.nx?path=/entry0000/instrument/detector/data&slice=100",
"999": "silx:///mnt/multipath-shares/tmp_14_days/payno/test_deployment/bambou_hercules_0001.nx?path=/entry0000/instrument/detector/data&slice=999",
},
"sub_region": "None",
},
"reconstruction": {
"angles": numpy.array(
[
0.00000000e00,
1.43989663e-03,
2.85934019e-03,
...,
6.27849473e00,
6.27991963e00,
6.28134453e00,
]
),
"axis_correction": "None",
"centered_axis": True,
"clip_outer_circle": True,
"cor_estimated_auto": False,
"enable_halftomo": True,
"end_x": 4121,
"end_y": 4121,
"end_z": 180,
"fbp_filter_cutoff": 1.0,
"fbp_filter_type": "ramlak",
"method": "FBP",
"padding_type": "edge",
"position": numpy.array([0.026, 0.0013, 0.0007]),
"rotation_axis_position": 2060.0,
"sample_detector_dist": "None",
"source_sample_dist": "None",
"start_x": 0,
"start_y": 0,
"start_z": 150,
"voxel_size_cm": numpy.array([1.25e-05, 2.25e-05, 3.25e-05]),
},
"save": {
"file_format": "hdf5",
"file_prefix": "bambou_hercules_0001_vol",
"float_clip_values": "None",
"jpeg2000_compression_ratio": "None",
"location": "/mnt/multipath-shares/tmp_14_days/payno/test_deployment/reconstructed_volumes",
"overwrite": True,
"overwrite_results": True,
"tiff_single_file": False,
},
"take_log": {"log_max_clip": 10.0, "log_min_clip": 1e-06},
},
"reconstruction_stages": {
"((0, 4400), (150, 181), (0, 2560))": "bambou_hercules_0001_vol/bambou_hercules_0001_vol_00150.hdf5"
},
}
# here we only compare keys because values have been cast (list to numpy array...) but this shouldn't prevent from creating the nabu config file
def compare_keys(dict_1: dict, dict_2: dict):
"""
compare two nested dictionaries.
Only keys are checked.
Because we want to make sure the structure of the dict are equivalent. Some values might have been casted (from numpy.array to list for example)
"""
all_keys = set(dict_1.keys()) | set(dict_2.keys())
for key in all_keys:
assert key in dict_1
assert key in dict_2
if isinstance(dict_1[key], dict):
assert isinstance(dict_2[key], dict), (
f"fail for key ({key}), values are {type(dict_1[key]), {type(dict_2[key])}}"
)
compare_keys(dict_1[key], dict_2[key])
def test_nabu_and_drac_compatibility():
"""
test that the nabu metadata can be publish to drac/icat and retrieve from drac/icat to nabu config (for remote processing)
"""
flatten_dict = flatten_vol_metadata(_NABU_VOL_RAW_METADATA)
drac_dict = map_nabu_keys_to_drac(flatten_dict=flatten_dict)
# make sure the flatten dict is serializable
json.dumps(drac_dict)
assert drac_dict == {
"TOMOReconstructionPhase_ctf_advanced_params": "length_scale=1e-05; "
"lim1=1e-05; lim2=0.2; "
"normalize_by_mean=True",
"TOMOReconstructionPhase_ctf_geometry": "z1_v=None; z1_h=None; "
"detec_pixel_size=None; "
"magnification=True",
"TOMOReconstructionPhase_delta_beta": 100.0,
"TOMOReconstructionPhase_detector_sample_distance": 2100.0,
"TOMOReconstructionPhase_method": "paganin",
"TOMOReconstructionPhase_padding_type": "edge",
"TOMOReconstructionPhase_unsharp_coeff": 0.0,
"TOMOReconstructionPhase_unsharp_method": "gaussian",
"TOMOReconstructionPhase_unsharp_sigma": 0.0,
"TOMOReconstruction_angle_offset": 0.0,
"TOMOReconstruction_angles_file": "None",
"TOMOReconstruction_axis_correction_file": "None",
"TOMOReconstruction_centered_axis": True,
"TOMOReconstruction_clip_outer_circle": True,
"TOMOReconstruction_cor_options": "side='from_file'",
"TOMOReconstruction_enable_halftomo": True,
"TOMOReconstruction_end_x": -1,
"TOMOReconstruction_end_y": -1,
"TOMOReconstruction_end_z": 180,
"TOMOReconstruction_fbp_filter_cutoff": 1.0,
"TOMOReconstruction_fbp_filter_type": "ramlak",
"TOMOReconstruction_method": "FBP",
"TOMOReconstruction_optim_algorithm": "chambolle-pock",
"TOMOReconstruction_padding_type": "edge",
"TOMOReconstruction_preconditioning_filter": True,
"TOMOReconstruction_rotation_axis_position": 2060.0,
"TOMOReconstruction_start_x": 0,
"TOMOReconstruction_start_y": 0,
"TOMOReconstruction_start_z": 150,
"TOMOReconstruction_translation_movements_file": "None",
"TOMOReconstruction_weight_tv": 0.01,
"endTime": "2024-10-02T17:42:01",
"TOMOReconstruction_voxel_size_x": (1.25e-05 * _ureg.centimeter)
.to(_ureg.micrometer)
.magnitude,
"TOMOReconstruction_voxel_size_y": (2.25e-05 * _ureg.centimeter)
.to(_ureg.micrometer)
.magnitude,
"TOMOReconstruction_voxel_size_z": (3.25e-05 * _ureg.centimeter)
.to(_ureg.micrometer)
.magnitude,
}
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_edfvolume.py 0000664 0000000 0000000 00000003562 15241025157 0027437 0 ustar 00root root 0000000 0000000 # coding: utf-8
import os
from datetime import datetime
import fabio
import numpy
from volio.tests.mock import create_volume
from volio.volumes.edfvolume import EDFVolume
_data = create_volume(
frame_dims=(20, 20), z_size=11
) # z_size need to be at least 10 to check loading from file name works
for i in range(len(_data)):
_data[i] += 1
def test_edf_writer_header(tmp_path):
"""
Test writing an edf volume from several EDFVolume and reading the entire volume back
"""
acquisition_dir = tmp_path / "acquisition"
os.makedirs(acquisition_dir)
volume_dir = str(acquisition_dir / "volume")
os.makedirs(volume_dir)
sub_data = _data[:5]
header = {"time": datetime.now().timestamp()}
volume = EDFVolume(
folder=volume_dir,
data=sub_data,
header=header,
)
volume.save()
first_frame_path = os.path.join(volume_dir, "volume_000000.edf")
assert os.path.exists(first_frame_path)
assert "time" in fabio.open(first_frame_path).header
# test get_slice function
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=0),
sub_data[2],
)
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=1),
sub_data[:, 2, :],
)
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=2),
sub_data[:, :, 2],
)
volume.data = None
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=0),
sub_data[2],
)
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=1),
sub_data[:, 2, :],
)
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=2),
sub_data[:, :, 2],
)
assert volume.get_min_max_values() == (_data.min(), _data.max())
assert volume.data_extension == "edf"
assert volume.metadata_extension == "txt"
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_factory.py 0000664 0000000 0000000 00000003511 15241025157 0027112 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""test of the volio.factory module"""
import os
import numpy
import pytest
from volio.factory import Factory
from volio.tests.mock import Scene, create_volume
from volio.volumes.edfvolume import EDFVolume
from volio.volumes.hdf5volume import HDF5Volume
_data = create_volume(frame_dims=12, z_size=4, scene=Scene.SHEPP_LOGAN)
def test_factory_hdf5(tmp_path):
"""Factory should recreate a HDF5Volume saved to disk from its identifier"""
file_path = os.path.join(tmp_path, "volume.hdf5")
volume = HDF5Volume(
file_path=file_path,
data_path="entry/reconstruction",
data=_data,
)
volume.save()
identifier = volume.get_identifier()
obj_from_identifier = Factory.create_tomo_object_from_identifier(identifier)
assert isinstance(obj_from_identifier, HDF5Volume)
obj_from_identifier.load()
numpy.testing.assert_array_almost_equal(obj_from_identifier.data, _data)
obj_from_str = Factory.create_tomo_object_from_identifier(identifier.to_str())
assert isinstance(obj_from_str, HDF5Volume)
def test_factory_edf(tmp_path):
"""Factory should recreate an EDFVolume saved to disk from its identifier"""
folder = os.path.join(tmp_path, "edf_volume")
volume = EDFVolume(folder=folder, data=_data)
volume.save()
identifier = volume.get_identifier()
obj_from_identifier = Factory.create_tomo_object_from_identifier(identifier)
assert isinstance(obj_from_identifier, EDFVolume)
obj_from_str = Factory.create_tomo_object_from_identifier(identifier.to_str())
assert isinstance(obj_from_str, EDFVolume)
def test_factory_invalid_input():
with pytest.raises(TypeError):
Factory.create_tomo_object_from_identifier(12)
with pytest.raises(ValueError):
Factory.create_tomo_object_from_identifier("unknownscheme:volume:/some/path")
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_hdf5volume.py 0000664 0000000 0000000 00000037454 15241025157 0027536 0 ustar 00root root 0000000 0000000 # coding: utf-8
import os
import h5py
import numpy
import pytest
from silx.io.url import DataUrl
from volio.identifiers.base import VolumeIdentifier
from volio.identifiers.hdf5identifier import HDF5VolumeIdentifier
from volio.tests.mock import create_volume
from volio.volumes.hdf5volume import HDF5Volume, get_default_data_path_for_volume
_data = create_volume(frame_dims=(20, 20), z_size=5)
_metadata = {
"nabu_config": {
"dataset": {
"location": "toto.hdf5",
"entry": "entry0000",
},
"phase": {
"method": "None",
"padding_type": "edge",
},
},
"processing_option": {
"build_sino": {
"axis_correction": "None",
"enable_halftomo": True,
},
"flatfield": {
"binning": "11",
"do_flat_distortion": False,
},
},
}
def test_hdf5volume_file_path(tmp_path):
"""test creation of an hdf5volume by providing url to the constructor"""
test_dir = tmp_path / "test_volume_url"
os.makedirs(test_dir)
volume_file = os.path.join(test_dir, "volume.hdf5")
assert not os.path.exists(volume_file)
volume = HDF5Volume(
file_path=volume_file,
data_path="/entry0000/reconstruction",
data=_data,
metadata=_metadata,
)
assert volume.get_min_max_values() == (_data.min(), _data.max())
url = DataUrl(file_path=volume_file, data_path="/entry0000/reconstruction")
assert volume.url.file_path() == url.file_path()
assert volume.url.data_path() == url.data_path()
assert volume.data_url is not None
assert volume.metadata_url is not None
volume_identifier = volume.get_identifier()
assert isinstance(volume_identifier, VolumeIdentifier)
# insure data_url and metadata_url cannot be write
with pytest.raises(AttributeError):
volume.data_url = None
with pytest.raises(AttributeError):
volume.metadata_url = None
volume.save()
assert os.path.exists(volume_file)
volume_loaded = HDF5Volume(
file_path=volume_file, data_path="/entry0000/reconstruction"
)
assert volume_loaded._data is None
assert volume_loaded._metadata is None
# check loading function won't save loaded data in cache
data_no_cache = volume_loaded.load_data(store=False)
metadata_no_cache = volume_loaded.load_metadata(store=False)
assert data_no_cache is not None
assert metadata_no_cache is not None
assert volume_loaded._data is None
assert volume_loaded._metadata is None
numpy.testing.assert_array_almost_equal(_data, data_no_cache)
assert _metadata == metadata_no_cache
# check data and metadata properties will load and set data in cache
volume_loaded.load()
data_cache = volume_loaded.data
assert data_cache is not None
metadata_cache = volume_loaded.metadata
assert metadata_cache is not None
numpy.testing.assert_array_almost_equal(_data, data_cache)
assert _metadata == metadata_cache
volume_loaded.data = None
volume_loaded.metadata = None
with pytest.raises(ValueError):
volume_loaded.save_data()
with pytest.raises(ValueError):
volume_loaded.save_metadata()
with pytest.raises(TypeError):
volume_loaded.data = "toto"
volume_loaded.data = create_volume(frame_dims=(20, 20), z_size=2)
with pytest.raises(ValueError):
volume_loaded.data = create_volume(frame_dims=(20, 20), z_size=2)[0]
other_hdf5_file = os.path.join(test_dir, "other_hdf5_file.h5")
new_volume_url = DataUrl(file_path=other_hdf5_file, data_path="data", scheme="silx")
new_volume = HDF5Volume(
file_path=new_volume_url.file_path(),
data_path=new_volume_url.data_path(),
data=volume_loaded.data,
metadata=volume_loaded.metadata,
)
new_volume.save_data()
with pytest.raises(OSError):
new_volume.save_data(
url=DataUrl(file_path=other_hdf5_file, data_path="data", scheme="silx")
)
new_volume.overwrite = True
numpy.testing.assert_array_almost_equal(
volume_loaded.load_data(), volume_loaded.data
)
with pytest.raises(ValueError):
volume_loaded.save_metadata()
# get the volume from the identifier
volume_from_id = HDF5Volume.from_identifier(identifier=volume_identifier)
assert volume_from_id.data is None
assert volume_from_id.metadata is None
# get the volume from the identifier str
volume_from_str = HDF5Volume.from_identifier(
identifier=HDF5VolumeIdentifier.from_str(volume_identifier.to_str())
)
assert volume_from_str.data is None
assert volume_from_str.metadata is None
# test browsing frames
for l_frame, o_frame in zip(volume.browse_slices(), _data):
numpy.testing.assert_allclose(l_frame, o_frame)
# test hash
hash(volume_from_str)
# test get_min_max values
assert volume_from_str.get_min_max() == (_data.min(), _data.max())
volume_from_str.load_data()
assert volume_from_str.get_min_max() == (_data.min(), _data.max())
assert volume.data_extension == "hdf5"
assert volume.metadata_extension == "hdf5"
def test_hdf5volume_data_url_and_metadata_url(tmp_path):
"""test creation of an hdf5volume by providing a data url and a metadata url"""
test_dir = tmp_path / "test_volume_data_url_metadata_url"
os.makedirs(test_dir)
hdf5_data_file = os.path.join(test_dir, "my_data_file.hdf5")
hdf5_metadata_file = os.path.join(test_dir, "my_metadata_file.hdf5")
data_url = DataUrl(
file_path=hdf5_data_file,
data_path="data",
scheme="silx",
)
metadata_url = DataUrl(
file_path=hdf5_metadata_file,
data_path="metadata",
scheme="silx",
)
volume = HDF5Volume(
data_url=data_url, metadata_url=metadata_url, data=_data, metadata=_metadata
)
with pytest.raises(ValueError):
assert volume.get_identifier()
assert tuple(volume.browse_data_files()) == tuple()
assert tuple(volume.browse_data_urls()) == tuple()
assert tuple(volume.browse_metadata_files()) == tuple()
assert not os.path.exists(hdf5_data_file)
volume.save_data()
assert os.path.exists(hdf5_data_file)
assert not os.path.exists(hdf5_metadata_file)
volume.save_metadata()
assert os.path.exists(hdf5_metadata_file)
volume_loaded = HDF5Volume(data_url=data_url, metadata_url=metadata_url)
assert volume_loaded.metadata is None
volume_loaded.load_metadata()
assert volume_loaded.metadata is not None
assert volume_loaded.data is None
# test get_slice
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=0), _data[2, :, :]
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=1), _data[:, 2, :]
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=2), _data[:, :, 2]
)
# test old API (deprecated)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=0), volume_loaded.get_slice(xy=2)
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=1), volume_loaded.get_slice(xz=2)
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=2), volume_loaded.get_slice(yz=2)
)
volume_loaded.load_data()
assert volume_loaded.data is not None
# test get_slice
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=0), _data[2, :, :]
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=1), _data[:, 2, :]
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=2), _data[:, :, 2]
)
numpy.testing.assert_array_equal(
volume_loaded.data,
_data,
)
assert tuple(volume_loaded.browse_data_files()) == (hdf5_data_file,)
assert tuple(volume_loaded.browse_data_urls()) == (volume_loaded.data_url,)
assert tuple(volume_loaded.browse_metadata_files()) == (hdf5_metadata_file,)
# test load_chunk
chunks_to_test = [
(slice(None, None), slice(0, 5), slice(None, None)),
(slice(0, 4), slice(0, 15), slice(None, None)),
]
for chunk in chunks_to_test:
data_chunk = volume.load_chunk(chunk)
assert data_chunk.shape == _data[chunk].shape
assert numpy.allclose(data_chunk, _data[chunk])
def test_write_hdf5_volume_with_parameter(tmp_path):
test_dir = tmp_path / "test_volume_data_url_metadata_url"
os.makedirs(test_dir)
hdf5_data_file = os.path.join(test_dir, "my_data_file.hdf5")
hdf5_metadata_file = os.path.join(test_dir, "my_metadata_file.hdf5")
data_url = DataUrl(
file_path=hdf5_data_file,
data_path="data",
scheme="silx",
)
metadata_url = DataUrl(
file_path=hdf5_metadata_file,
data_path="metadata",
scheme="silx",
)
volume = HDF5Volume(
data_url=data_url,
metadata_url=metadata_url,
data=_data,
metadata=_metadata,
overwrite=True,
)
# check chunks
volume.save(chunks=True)
with h5py.File(hdf5_data_file, mode="r") as h5f:
assert h5f["data"].chunks is not None
# check compression
volume.save(compression="gzip", compression_opts=9)
with h5py.File(hdf5_data_file, mode="r") as h5f:
assert h5f["data"].compression == "gzip"
def test_data_file_saver_generator(tmp_path):
"""
data_file_saver_generator (dumping frame by frame a volume to disk)
"""
volume_file_path = str(tmp_path / "volume.h5")
volume = HDF5Volume(file_path=volume_file_path, data_path="volume")
for slice_, slice_saver in zip(
_data,
volume.data_file_saver_generator(n_frames=_data.shape[0], overwrite=False),
):
slice_saver[:] = slice_
assert volume.data is None
numpy.testing.assert_array_equal(
volume.load_data(),
_data,
)
def test_write_hdf5_with_virtual_layout_as_data(tmp_path):
"""
insure HDF5Volume.data is safely handling virtual layout
"""
test_dir = tmp_path / "test_volume_data_url_metadata_url"
os.makedirs(test_dir)
n_file = 5
virtual_layout = h5py.VirtualLayout(shape=(n_file * 10, 100, 100), dtype=float)
# create raw data
for i_file in range(n_file):
file_path = os.path.join(test_dir, f"file{i_file}.hdf5")
data_path = f"path_to_dataset_{i_file}"
with h5py.File(file_path, mode="w") as h5f:
data = numpy.arange(0, 10 * 100 * 100).reshape(10, 100, 100)
h5f[data_path] = data
vs = h5py.VirtualSource(h5f[data_path])
virtual_layout[i_file * 10 : (i_file + 1) * 10] = vs
hdf5_data_file = os.path.join(test_dir, "my_data_file.hdf5")
volume = HDF5Volume(
file_path=hdf5_data_file,
data_path="/",
data=virtual_layout,
metadata=_metadata,
overwrite=True,
)
volume.save()
with h5py.File(hdf5_data_file, mode="r") as h5f:
assert h5f["results/data"].is_virtual
numpy.testing.assert_array_equal(
h5f["results/data"][:10],
numpy.arange(0, 10 * 100 * 100).reshape(10, 100, 100),
)
volume.data = None
assert volume.get_volume_shape() == (n_file * 10, 100, 100)
def test_utils(tmp_path):
"""test utils functions of the hdf5volume module"""
class MockScanWithEntry:
"""minimal duck-typed stand-in for a scan exposing an 'entry' attribute
(e.g. tomoscan's NXtomoScan), used here to avoid a dependency on tomoscan"""
entry = "entry0000"
assert isinstance(get_default_data_path_for_volume(MockScanWithEntry()), str)
with pytest.raises(TypeError):
get_default_data_path_for_volume("toto")
def test_get_volume_size(tmp_path):
"""
'get_volume_size' must return the exact size when data is loaded, and an
estimated (but equal) size when computed from the hdf5 dataset without
loading the full data.
"""
volume_file = str(tmp_path / "volume_size.hdf5")
volume = HDF5Volume(
file_path=volume_file,
data_path="/entry0000/reconstruction",
data=_data,
)
volume.save()
assert volume.get_volume_size() == _data.nbytes
volume_unloaded = HDF5Volume(
file_path=volume_file, data_path="/entry0000/reconstruction"
)
assert volume_unloaded.data is None
assert volume_unloaded.get_volume_size() == _data.nbytes
def test_example():
"""test static function 'example'"""
assert isinstance(HDF5Volume.example_defined_from_str_identifier(), str)
def test_get_histogram(tmp_path):
"""test 'get_histogram' API on a HDF5Volume"""
test_dir = str(tmp_path / "test_get_histogram")
os.makedirs(test_dir)
volume_file = os.path.join(test_dir, "volume.hdf5")
volume = HDF5Volume(
file_path=volume_file,
data_path="entry0000/reconstruction",
data=_data,
)
volume.save()
# no histogram written yet
assert volume.get_histogram_url() is None
assert volume.get_histogram() is None
# histogram is stored as a sibling NXprocess group in the same file/entry
stored_counts = numpy.arange(6, dtype="f")
stored_bins = numpy.linspace(_data.min(), _data.max(), 6, dtype="f")
stored_histogram = numpy.array([stored_counts, stored_bins])
with h5py.File(volume_file, mode="a") as h5f:
h5f.create_dataset("entry0000/histogram/results/data", data=stored_histogram)
url = volume.get_histogram_url()
assert url is not None
assert url.file_path() == volume_file
assert url.data_path() == "entry0000/histogram/results/data"
counts, bins = volume.get_histogram()
numpy.testing.assert_array_equal(counts, stored_counts)
numpy.testing.assert_array_equal(bins, stored_bins)
def test_compute_histogram(tmp_path):
"""test 'compute_histogram' / recompute-if-missing on a HDF5Volume"""
test_dir = str(tmp_path / "test_compute_histogram")
os.makedirs(test_dir)
volume_file = os.path.join(test_dir, "volume.hdf5")
data = _data.astype(numpy.float32)
volume = HDF5Volume(
file_path=volume_file,
data_path="entry0000/reconstruction",
data=data,
)
volume.save()
assert volume.get_histogram() is None
counts, bin_edges = volume.get_histogram(recompute_if_missing=True, n_bins=16)
expected_counts, expected_edges = numpy.histogram(
data.ravel(), bins=16, range=(data.min(), data.max())
)
numpy.testing.assert_array_equal(counts, expected_counts)
numpy.testing.assert_allclose(bin_edges, expected_edges)
@pytest.mark.parametrize("save_volume_to_file", (True, False))
def test_get_slices(tmp_path, save_volume_to_file: bool):
"""test the volume 'get_slices' function"""
test_dir = tmp_path / "test_get_slices"
test_dir.mkdir()
volume_file = os.path.join(test_dir, "volume.hdf5")
assert not os.path.exists(volume_file)
data = numpy.linspace(0, 1000, num=1000, endpoint=False).reshape(10, 10, 10)
volume = HDF5Volume(
file_path=volume_file,
data_path="/entry0000/reconstruction",
data=data,
metadata=_metadata,
)
if save_volume_to_file:
volume.save_data()
volume.clear_cache()
slices_axis_0 = volume.get_slices(((0, 2), (0, 5)))
numpy.testing.assert_array_almost_equal(
slices_axis_0[(0, 2)],
data[2],
)
slices_axis_1 = volume.get_slices(((1, 0), (1, 6)))
numpy.testing.assert_array_almost_equal(
slices_axis_1[(1, 6)],
data[:, 6, :],
)
assert isinstance(slices_axis_1[(1, 6)], numpy.ndarray)
slices_axis_2 = volume.get_slices(((2, 8), (2, 9)))
numpy.testing.assert_array_almost_equal(
slices_axis_2[(2, 9)],
data[:, :, 9],
)
with pytest.raises(TypeError):
volume.get_slices(
"toto",
)
with pytest.raises(TypeError):
volume.get_slices(
(0, 10),
)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_identifiers.py 0000664 0000000 0000000 00000001617 15241025157 0027755 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""
Volume-identifier-specific test cases ported from tomoscan's
tomoscan/esrf/identifier/tests/ (test_edfidentifier.py, test_hdf5identifier.py,
test_rawidentifier.py). Those source files mostly exercised scan identifiers
(EDFTomoScanIdentifier, NXtomoScanIdentifier) which are out of scope for volio;
only the RawVolumeIdentifier case applied to a volume-only identifier.
EDFVolumeIdentifier and HDF5VolumeIdentifier are already exercised end-to-end
in test_single_frame_volume.py and test_hdf5volume.py.
"""
from volio.identifiers.rawidentifier import RawVolumeIdentifier
def test_vol_identifier():
"""insure identifier is working for .vol files"""
from volio.volumes.rawvolume import RawVolume
identifier = RawVolumeIdentifier(
object=RawVolume,
file_path="/dsad/test.vol",
)
assert str(identifier).startswith(f"raw:{RawVolumeIdentifier.TOMO_TYPE}:")
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_jp2kvolume.py 0000664 0000000 0000000 00000002504 15241025157 0027542 0 ustar 00root root 0000000 0000000 """specific test for the jp2kvolume. the large part is in test_single_frame_volume as most of the processing is common"""
import os
import numpy
import pytest
from volio.tests.mock import create_volume
from volio.volumes.jp2kvolume import JP2KVolume
@pytest.fixture
def raw_data(dtype: numpy.dtype):
data = create_volume(
frame_dims=(100, 100), z_size=11
) # z_size need to be at least 10 to check loading from file name works
for i in range(len(data)):
data[i] += 1
return data.astype(dtype)
@pytest.mark.parametrize("dtype", (numpy.uint8, numpy.uint16))
def test_jp2kvolume_rescale(raw_data, dtype, tmp_path):
"""
Test that rescale is correctly applied by default by the JP2KVolume
"""
acquisition_dir = tmp_path / "acquisition"
os.makedirs(acquisition_dir)
volume_dir = str(acquisition_dir / "volume")
os.makedirs(volume_dir)
assert raw_data.dtype == dtype
volume = JP2KVolume(folder=volume_dir, data=raw_data, metadata={})
assert volume.data.dtype == dtype
volume.save()
volume.clear_cache()
volume.load()
assert volume.data.dtype == dtype
assert volume.data.min() == 0
assert volume.data.max() in (numpy.iinfo(dtype).max, numpy.iinfo(dtype).max - 1)
assert volume.data_extension == "jp2"
assert volume.metadata_extension == "txt"
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_raw_volume.py 0000664 0000000 0000000 00000032022 15241025157 0027622 0 ustar 00root root 0000000 0000000 """test of the .vol Volume"""
import os
import numpy
import pytest
from h5py import File
from volio.identifiers.base import VolumeIdentifier
from volio.identifiers.rawidentifier import RawVolumeIdentifier
from volio.tests.mock import create_volume
from volio.volumes.rawvolume import RawVolume
_data = create_volume(frame_dims=(20, 20), z_size=60)
_data = _data.astype(numpy.uint8)
metadata_info = {
"NUM_X": 20,
"NUM_Y": 20,
"NUM_Z": 60,
"voxelSize": 2.3e-2,
"BYTEORDER": "LOWBYTEFIRST",
"ValMin": _data.min(),
"ValMax": _data.max(),
"s1": -1.13,
"s2": 8.19,
"S1": -4.13,
"S2": 5.15,
}
metadata_xml = {
"idAc": "N_A_",
"listSubVolume": {
"SUBVOLUME_NAME": "toto_volume",
"SIZEX": metadata_info["NUM_X"],
"SIZEY": metadata_info["NUM_Y"],
"SIZEZ": metadata_info["NUM_Z"],
"ORIGINX": 1,
"ORIGINY": 1,
"ORIGINZ": 1,
"DIM_REC": 14155776000,
"voxelsize": metadata_info["voxelSize"],
"BYTE_ORDER": metadata_info["BYTEORDER"],
"ValMin": metadata_info["ValMin"],
"ValMax": metadata_info["ValMax"],
"s1": metadata_info["s1"],
"s2": metadata_info["s2"],
"S1": metadata_info["S1"],
"S2": metadata_info["S2"],
},
}
def test_raw_volume(tmp_path):
test_dir = str(tmp_path / "test_volume_url")
os.makedirs(test_dir)
volume_file = os.path.join(test_dir, "volume.vol")
metadata_info_volume_file = os.path.join(test_dir, "volume.vol.info")
metadata_xml_volume_file = os.path.join(test_dir, "volume.vol.xml")
volume = RawVolume(
file_path=volume_file,
data=_data,
metadata=metadata_info,
)
assert volume.url.file_path() == volume_file
assert volume.url.data_path() is None
assert volume.data_url.scheme() == "raw"
assert volume.data_url.file_path() == volume_file
assert volume.data_url.data_path() is None
assert volume.metadata_url.scheme() == "info"
assert volume.metadata_url.file_path() == metadata_info_volume_file
assert volume.metadata_url.data_path() is None
volume_identifier = volume.get_identifier()
assert isinstance(volume_identifier.short_description(), str)
assert isinstance(volume_identifier, VolumeIdentifier)
assert not os.path.exists(volume_file)
assert not os.path.exists(metadata_info_volume_file)
assert not os.path.exists(metadata_xml_volume_file)
# make sure saving other file format than float32 make it fails
with pytest.raises(TypeError):
volume.save()
data_cast = volume.data.astype(numpy.float32)
volume.data = data_cast
volume.save()
# try to save it twice to make sure we are not appending data to it
assert os.path.exists(volume_file)
assert os.path.exists(metadata_info_volume_file)
assert not os.path.exists(metadata_xml_volume_file)
assert tuple(volume.browse_data_urls()) == (volume.data_url,)
assert tuple(volume.browse_data_files()) == (volume.data_url.file_path(),)
assert tuple(volume.browse_metadata_files()) == (volume.metadata_url.file_path(),)
volume_loaded = RawVolume(file_path=volume_file)
assert volume.url.file_path() == volume_file
assert volume.metadata_url.file_path() == metadata_info_volume_file
data_no_cache = volume_loaded.load_data(store=False)
metadata_no_cache = volume_loaded.load_metadata(store=False)
assert data_no_cache is not None
assert metadata_no_cache is not None
assert volume_loaded.data is None
assert volume_loaded.metadata is None
data_float_32 = _data.astype(numpy.float32)
numpy.testing.assert_array_almost_equal(data_float_32, data_no_cache)
# only check keys because values are not cast (once read everything is a str...)
assert metadata_info.keys() == metadata_no_cache.keys()
assert volume.get_min_max_values() == (data_float_32.min(), data_float_32.max())
# check error is raised if byte order is incorrect
highbytefirst = numpy.dtype(">" + volume.data.dtype.char)
volume.data = volume.data.astype(highbytefirst)
with pytest.raises(TypeError):
volume.save_data()
# get the volume from the identifier
volume_from_id = RawVolume.from_identifier(identifier=volume_identifier)
assert volume_from_id.data is None
assert volume_from_id.metadata is None
# get the volume from the identifier str
volume_from_str = RawVolume.from_identifier(
identifier=RawVolumeIdentifier.from_str(volume_identifier.to_str())
)
assert volume_from_str.data is None
assert volume_from_str.metadata is None
assert volume_from_str.get_identifier() == volume_from_id.get_identifier()
# test browsing frames
# TODO
# test hash
hash(volume_from_str)
assert volume.data_extension == "vol"
assert volume.metadata_extension == "info"
def test_data_file_saver_generator(tmp_path):
"""
data_file_saver_generator (dumping frame by frame a volume to disk)
"""
volume_file_path = str(tmp_path / "volume_for_file_generator.vol")
volume = RawVolume(file_path=volume_file_path)
data_cast = _data.astype(numpy.float32)
for slice_, slice_saver in zip(
data_cast,
volume.data_file_saver_generator(n_frames=_data.shape[0], overwrite=False),
):
slice_saver[:] = slice_
assert volume.data is None
volume.metadata = metadata_info
volume.save_metadata()
numpy.testing.assert_array_equal(
volume.load_data(),
data_cast,
)
def test_raw_volume_slice_getter(tmp_path):
"""
tset slice getters
"""
volume_file = str(tmp_path / "volume_test_getter.vol")
data_cast = _data.astype(numpy.float32)
volume = RawVolume(
file_path=volume_file,
data=data_cast,
metadata=metadata_info,
overwrite=True,
)
volume.save()
assert os.path.exists(volume_file)
volume_loaded = RawVolume(
file_path=volume_file,
)
volume_loaded.load()
numpy.testing.assert_array_equal(
data_cast,
volume_loaded.data,
)
# test get_slice function
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=0),
data_cast[2],
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=1),
data_cast[:, 2, :],
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=2),
data_cast[:, :, 2],
)
volume_loaded.data = None
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=0),
data_cast[2],
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=1),
data_cast[:, 2, :],
)
numpy.testing.assert_array_equal(
volume_loaded.get_slice(index=2, axis=2),
data_cast[:, :, 2],
)
def test_append_writes(tmp_path):
"""
Test trying to mimic PyHST2 output files that can be opened with imagej:
- One single .vol, written in several steps (append mode)
- One .info file (for users ?)
- One .xml file (for ImageJ)
"""
def generate_metadata(data, **kwargs):
n_z, n_y, n_x = data.shape
metadata = {
"NUM_X": n_x,
"NUM_Y": n_y,
"NUM_Z": n_z,
"voxelSize": kwargs.get("voxelSize", 40.0),
"BYTEORDER": "LOWBYTEFIRST",
"ValMin": kwargs.get("ValMin", 0.0),
"ValMax": kwargs.get("ValMin", 1.0),
"s1": 0.0,
"s2": 0.0,
"S1": 0.0,
"S2": 0.0,
}
return metadata
def sanitize_metadata(metadata):
# To be fixed in RawVolume
for what in ["NUM_X", "NUM_Y", "NUM_Z"]:
metadata[what] = int(metadata[what])
for what in ["voxelSize", "ValMin", "ValMax", "s1", "s2", "S1", "S2"]:
metadata[what] = float(metadata[what])
def write_data(volume, data):
existing_metadata = volume.load_metadata()
new_metadata = generate_metadata(data)
if len(existing_metadata) == 0:
# first write
metadata = new_metadata
else:
# append write ; update metadata
metadata = existing_metadata.copy()
sanitize_metadata(metadata)
metadata["NUM_Z"] += new_metadata["NUM_Z"]
volume.data = data
volume.metadata = metadata
volume.save()
volume_file = str(tmp_path / "volume_append.vol")
volume = RawVolume(
file_path=volume_file,
append=True,
)
# First write
data_1 = numpy.arange(10 * 32 * 32, dtype="f").reshape((10, 32, 32))
write_data(volume, data_1)
# Second write - metadata should be updated
data_2 = numpy.arange(11 * 32 * 32, dtype="f").reshape((11, 32, 32))
write_data(volume, data_2)
# Third write - metadata should be updated
data_3 = numpy.arange(12 * 32 * 32, dtype="f").reshape((12, 32, 32))
write_data(volume, data_3)
assert int(volume.load_metadata()["NUM_Z"]) == 10 + 11 + 12
assert os.path.getsize(volume_file) == data_1.nbytes + data_2.nbytes + data_3.nbytes
def test_get_volume_size(tmp_path):
"""
'get_volume_size' must return the exact size when data is loaded, and an
estimated (but equal) size when computed from disk metadata without
loading the full data.
"""
volume_file = str(tmp_path / "volume_size.vol")
data_cast = _data.astype(numpy.float32)
volume = RawVolume(file_path=volume_file, data=data_cast, metadata=metadata_info)
volume.save()
assert volume.get_volume_size() == data_cast.nbytes
volume_unloaded = RawVolume(file_path=volume_file)
assert volume_unloaded.data is None
assert volume_unloaded.get_volume_size() == data_cast.nbytes
assert volume_unloaded.get_volume_size(unit="Mbyte") == pytest.approx(
data_cast.nbytes / 1e6
)
def test_browse_slices_from_disk_does_not_load_full_volume(tmp_path):
"""
'browse_slices' must stream frames from disk (memory-mapped) instead of loading
the full volume into memory when 'self.data' isn't already set.
"""
volume_file = str(tmp_path / "volume_browse.vol")
data_cast = _data.astype(numpy.float32)
volume = RawVolume(
file_path=volume_file,
data=data_cast,
metadata=metadata_info,
)
volume.save()
volume_loaded = RawVolume(file_path=volume_file)
slices = list(volume_loaded.browse_slices())
assert volume_loaded.data is None
numpy.testing.assert_array_equal(numpy.array(slices), data_cast)
# browsing from an in-memory volume (no data on disk needed) must still work
volume_in_memory = RawVolume(data=data_cast)
slices_in_memory = list(volume_in_memory.browse_slices())
numpy.testing.assert_array_equal(numpy.array(slices_in_memory), data_cast)
def test_browse_slices_respects_byte_order(tmp_path):
"""
'browse_slices' must honor the 'BYTEORDER' metadata the same way 'load_data' does.
"""
volume_file = str(tmp_path / "volume_byteorder.vol")
data_cast = _data.astype(numpy.float32)
# write raw bytes in big-endian order, simulating a HIGHBYTEFIRST .vol file
with open(volume_file, "wb") as fdesc:
data_cast.astype(">f4").tofile(fdesc)
metadata = dict(metadata_info)
metadata["BYTEORDER"] = "HIGHBYTEFIRST"
volume = RawVolume(file_path=volume_file, metadata=metadata)
volume.save_metadata()
volume_loaded = RawVolume(file_path=volume_file)
slices = list(volume_loaded.browse_slices())
numpy.testing.assert_array_equal(numpy.array(slices), data_cast)
def test_example():
"""test static function 'example'"""
assert isinstance(RawVolume.example_defined_from_str_identifier(), str)
def test_get_histogram(tmp_path):
"""test 'get_histogram' API on a RawVolume"""
volume_file = str(tmp_path / "volume_histogram.vol")
data = _data.astype(numpy.float32)
volume = RawVolume(file_path=volume_file, data=data)
volume.save()
# no histogram file exists yet
assert volume.get_histogram_url() is None
assert volume.get_histogram() is None
# recomputing it must match numpy.histogram applied on the full volume
counts, bin_edges = volume.get_histogram(recompute_if_missing=True, n_bins=32)
expected_counts, expected_edges = numpy.histogram(
data.ravel(), bins=32, range=(data.min(), data.max())
)
numpy.testing.assert_array_equal(counts, expected_counts)
numpy.testing.assert_allclose(bin_edges, expected_edges)
# if a nabu-like histogram file exists next to the volume it should be picked up instead
histogram_file = str(tmp_path / "volume_histogram_histogram.hdf5")
stored_counts = numpy.arange(10, dtype="f")
stored_bins = numpy.linspace(data.min(), data.max(), 10, dtype="f")
stored_histogram = numpy.array([stored_counts, stored_bins])
with File(histogram_file, mode="w") as h5f:
h5f.create_dataset("entry0000/histogram/results/data", data=stored_histogram)
url = volume.get_histogram_url()
assert url is not None
counts, bins = volume.get_histogram()
numpy.testing.assert_array_equal(counts, stored_counts)
numpy.testing.assert_array_equal(bins, stored_bins)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_single_frame_volume.py 0000664 0000000 0000000 00000056362 15241025157 0031501 0 ustar 00root root 0000000 0000000 # coding: utf-8
import os
from glob import glob
import numpy
import pytest
from h5py import File
from tomobase.identifiers.folderidentifiers import BaseFolderIdentifierMixIn
from volio.identifiers.base import VolumeIdentifier
from volio.tests.mock import create_volume
from volio.volumes.edfvolume import EDFVolume
from volio.volumes.jp2kvolume import JP2KVolume, has_minimal_openjpeg
from volio.volumes.tiffvolume import TIFFVolume
from volio.volumes.volumebase import VolumeBase
_data = create_volume(
frame_dims=(100, 100), z_size=11
) # z_size need to be at least 10 to check loading from file name works
for i in range(len(_data)):
_data[i] += 1
_data = _data.astype(numpy.uint16)
_metadata = {
"nabu_config": {
"dataset": {
"location": "toto.hdf5",
"entry": "entry0000",
},
"phase": {
"method": "None",
"padding_type": "edge",
},
},
"processing_option": {
"build_sino": {
"axis_correction": "None",
"enable_halftomo": True,
},
"flatfield": {
"binning": "11",
"do_flat_distortion": False,
},
},
}
volume_constructors = [
EDFVolume,
TIFFVolume,
]
if has_minimal_openjpeg:
volume_constructors.append(JP2KVolume)
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_create_volume_from_folder(tmp_path, volume_constructor):
"""Test a volume can be weel defined from it path"""
acquisition_dir = tmp_path / "acquisition"
os.makedirs(acquisition_dir)
volume_dir = str(acquisition_dir / "volume")
os.makedirs(volume_dir)
volume = volume_constructor(folder=volume_dir, data=_data, metadata=_metadata)
assert (
len(
glob(
os.path.join(
volume_dir, f"*.{volume_constructor.DEFAULT_DATA_EXTENSION}"
)
)
)
== 0
)
assert (
len(
glob(
os.path.join(
volume_dir, f"*.{volume_constructor.DEFAULT_METADATA_EXTENSION}"
)
)
)
== 0
)
assert tuple(volume.browse_data_files()) == tuple()
assert tuple(volume.browse_data_urls()) == tuple()
assert tuple(volume.browse_metadata_files()) == tuple()
volume.save()
assert (
len(
glob(
os.path.join(
volume_dir, f"*.{volume_constructor.DEFAULT_DATA_EXTENSION}"
)
)
)
== _data.shape[0]
)
assert (
len(
glob(
os.path.join(
volume_dir, f"*.{volume_constructor.DEFAULT_METADATA_EXTENSION}"
)
)
)
== 1
)
# check overwrite parameter
volume.data = numpy.random.random((45, 45, 45))
with pytest.raises(OSError):
volume.save()
volume.overwrite = True
if isinstance(volume, JP2KVolume):
volume.rescale_data = False
volume.save()
assert volume.get_volume_shape() == (45, 45, 45)
volume.data = _data
volume.save()
# check load data and metadata
volume.clear_cache()
assert volume.get_min_max_values() == (_data.min(), _data.max())
volume.load()
numpy.testing.assert_array_almost_equal(_data, volume.data)
assert _metadata == volume.metadata
# test get_slice function
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=0),
_data[2],
)
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=1),
_data[:, 2, :],
)
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=2),
_data[:, :, 2],
)
# test deprecated API
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=0),
volume.get_slice(xy=2),
)
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=1),
volume.get_slice(xz=2),
)
numpy.testing.assert_array_equal(
volume.get_slice(index=2, axis=2),
volume.get_slice(yz=2),
)
assert len(list(volume.browse_data_files())) == _data.shape[0]
assert len(list(volume.browse_data_urls())) == _data.shape[0]
assert len(list(volume.browse_metadata_files())) == 1
# test browsing frames
for l_frame, o_frame in zip(volume.browse_slices(), _data):
numpy.testing.assert_allclose(l_frame, o_frame)
# test get_min_max values
volume.data = None
assert volume.get_min_max() == (_data.min(), _data.max())
volume.load_data()
assert volume.get_min_max() == (_data.min(), _data.max())
if isinstance(volume, JP2KVolume):
volume.data_extension == "jp2k"
elif isinstance(volume, EDFVolume):
volume.data_extension == "edf"
elif isinstance(volume, TIFFVolume):
volume.data_extension == "tiff"
else:
raise NotImplementedError
assert volume.metadata_extension == "txt"
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_get_start_index(tmp_path, volume_constructor):
"""Test setting start_index parameter"""
acquisition_dir = tmp_path / "acquisition"
os.makedirs(acquisition_dir)
volume_dir = str(acquisition_dir / "volume")
os.makedirs(volume_dir)
# use data which has a different value for each frame
data = _data.copy()
data += numpy.arange(data.shape[0], dtype=_data.dtype).reshape((-1, 1, 1))
#
start_index_list = [0, 7] # one default and one non-default
for start_index in start_index_list:
volume_basename = f"test_startindex{start_index}"
volume = volume_constructor(
folder=volume_dir, volume_basename=volume_basename, start_index=start_index
)
assert volume.get_first_file() is None
volume.data = _data
volume.save()
expected_first_file_name = os.path.join(
volume_dir,
"%s_%06d.%s"
% (volume_basename, start_index, volume_constructor.DEFAULT_DATA_EXTENSION),
)
assert volume.get_first_file() == expected_first_file_name
assert volume.get_file_slice_index(volume.get_first_file()) == start_index
# Now create another XXVolume object using the same basename, while there are existing files
volume2 = volume_constructor(
folder=volume_dir, volume_basename=volume_basename, start_index=start_index
)
assert volume2.get_first_file() == expected_first_file_name
assert (
volume2.get_file_slice_index(volume.get_first_file()) == start_index
) # This used to be False before this patch
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_reset_slice_index(tmp_path, volume_constructor):
"""
Test a case that might happen during volume casting: existing files have start_index > 0,
so we should re-define start_index for the new Volume object before writing
"""
acquisition_dir = tmp_path / "acquisition"
os.makedirs(acquisition_dir)
volume_dir = str(acquisition_dir / "volume")
os.makedirs(volume_dir)
volume_dir2 = str(acquisition_dir / "volume2")
os.makedirs(volume_dir2)
# use data which has a different value for each frame
data = _data.copy()
data += numpy.arange(data.shape[0], dtype=_data.dtype).reshape((-1, 1, 1))
#
# Step 1: a Volume is created during reconstruction, with start_z > 0
volume = volume_constructor(
folder=volume_dir, volume_basename="test_reset_slice_index", start_index=500
)
if isinstance(volume, JP2KVolume):
volume.rescale_data = False
volume.data = data
volume.save()
# Step 2: after reconstruction, volume casting is triggered.
# The "input_volume" object has to detect the actual start_index
input_volume = volume_constructor(
folder=volume_dir, volume_basename="test_reset_slice_index"
)
output_volume = volume_constructor(
folder=volume_dir2, volume_basename="test_reset_slice_index"
)
if isinstance(output_volume, JP2KVolume):
output_volume.rescale_data = False
input_vol_start_idx = input_volume.get_file_slice_index(
input_volume.get_first_file()
)
assert input_vol_start_idx == 500
output_volume.start_index = input_vol_start_idx
output_volume.data = data + 10
output_volume.save()
# Check !
vol_check = volume_constructor(
folder=volume_dir2, volume_basename="test_reset_slice_index", start_index=500
)
assert numpy.allclose(vol_check.load_data(), volume.load_data() + 10)
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_data_file_saver_generator(tmp_path, volume_constructor):
"""
data_file_saver_generator (dumping frame by frame a volume to disk)
"""
volume_dir = str(tmp_path / "volume")
os.makedirs(volume_dir)
volume = volume_constructor(folder=volume_dir)
if isinstance(volume, JP2KVolume):
volume.rescale_data = False
for slice_, slice_saver in zip(
_data,
volume.data_file_saver_generator(n_frames=_data.shape[0], overwrite=False),
):
slice_saver[:] = slice_
assert volume.data is None
numpy.testing.assert_array_equal(
volume.load_data(),
_data,
)
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_decreasing_index(tmp_path, volume_constructor):
"""
Test the 'index_step' parameter
"""
volume_dir = str(tmp_path / "volume")
os.makedirs(volume_dir)
volume = volume_constructor(folder=volume_dir, start_index=200)
volume.write_in_descending_order = True
if isinstance(volume, JP2KVolume):
volume.rescale_data = False
data = numpy.arange(32 * 33 * 34, dtype="f").reshape(
(32, 33, 34)
) # put a not-too-small number for jpeg2000
volume.data = data
volume.save()
# Check that the indices are correct (this will not check the write index order)
files_list = sorted(
glob(os.path.join(volume_dir, f"*.{volume.DEFAULT_DATA_EXTENSION}"))
)
indices = [
int(os.path.splitext(os.path.basename(fname))[0].split("_")[-1])
for fname in files_list
]
assert sorted(indices, reverse=True) == list(
range(volume.start_index, volume.start_index - data.shape[0], -1)
)
# Check that the files order was reversed at write time
volume_reader = volume_constructor(folder=volume_dir)
data_read = volume_reader.load_data()
assert numpy.allclose(data_read, data[::-1, ...])
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_several_writer(tmp_path, volume_constructor):
"""
Test writing a volume from several instance of VolumeSingleFrameBase and reading the entire volume back
"""
acquisition_dir = tmp_path / "acquisition"
os.makedirs(acquisition_dir)
volume_dir = acquisition_dir / "volume"
os.makedirs(volume_dir)
volume_1 = volume_constructor(
folder=volume_dir,
data=_data[:5],
metadata=_metadata,
)
volume_2 = volume_constructor(folder=volume_dir, data=_data[5:], start_index=5)
if isinstance(volume_1, JP2KVolume):
volume_1.rescale_data = (
False # keep coherence between all the volumes. Simplify test
)
volume_2.rescale_data = False
volume_1.save()
volume_2.skip_existing_data_files_removal = True
volume_2.save()
full_volume = volume_constructor(folder=volume_dir)
full_volume.load()
numpy.testing.assert_array_almost_equal(full_volume.data, _data)
assert full_volume.metadata == _metadata
full_volume.data = None
assert full_volume.get_volume_shape() == _data.shape
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_volume_identifier(tmp_path, volume_constructor):
"""
Insure each type of volume can provide an identifier and recover the data store from it
"""
acquisition_dir = tmp_path / "acquisition"
os.makedirs(acquisition_dir)
volume_dir = acquisition_dir / "volume"
os.makedirs(volume_dir)
volume = volume_constructor(
folder=volume_dir,
data=_data,
metadata=_metadata,
)
if isinstance(volume, JP2KVolume):
volume.rescale_data = (
False # keep coherence between all the volumes. Simplify test
)
volume.save()
identifier = volume.get_identifier()
assert isinstance(identifier, VolumeIdentifier)
del volume
volume_loaded = volume_constructor.from_identifier(identifier)
volume_loaded.load()
numpy.testing.assert_array_almost_equal(volume_loaded.data, _data)
assert volume_loaded.metadata == _metadata
# check API Identifier.to() and identifier.from_str()
identifier_from_str = identifier.to_str()
duplicate_id = identifier.from_str(identifier_from_str)
assert duplicate_id == identifier
assert identifier_from_str == identifier.to_str()
assert identifier != object()
assert identifier != "toto"
assert isinstance(identifier.short_description(), str)
# check it can be reconstructed via 'recreate_object', the low-level primitive
# that volio.factory.Factory (and tomoscan's own Factory, for the volume case)
# builds on
assert isinstance(identifier, VolumeIdentifier)
tomo_obj_from_identifier_instance = identifier.recreate_object()
assert isinstance(tomo_obj_from_identifier_instance, type(volume_loaded))
tomo_obj_from_str = type(identifier).from_str(identifier.to_str()).recreate_object()
assert isinstance(tomo_obj_from_str, type(volume_loaded))
# test hash
hash(tomo_obj_from_str)
hash(tomo_obj_from_str.get_identifier())
def test_folder_mix_in():
"""simple test of the BaseFolderIdentifierMixIn class"""
class FolderIdentifierTest(BaseFolderIdentifierMixIn, VolumeIdentifier):
pass
class VolumeTest(VolumeBase):
pass
obj = FolderIdentifierTest(object=VolumeTest, folder="toto", tomo_type="Volume")
with pytest.raises(NotImplementedError):
obj.scheme
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_volume_with_prefix(tmp_path, volume_constructor):
"""
Test writing and reading volume with a file_prefix and using the identifier
"""
acquisition_dir = tmp_path / "acquisition"
os.makedirs(acquisition_dir)
volume_dir = acquisition_dir / "volume"
os.makedirs(volume_dir)
file_prefix = "test"
volume_1 = volume_constructor(
folder=volume_dir,
data=_data[:5],
metadata=_metadata,
volume_basename=file_prefix,
)
if isinstance(volume_1, JP2KVolume):
volume_1.rescale_data = (
False # keep coherence between all the volumes. Simplify test
)
volume_1.save()
full_volume = volume_constructor(folder=volume_dir, volume_basename=file_prefix)
full_volume.load()
numpy.testing.assert_array_almost_equal(full_volume.data, _data[:5])
assert full_volume.metadata == _metadata
obj_recreated = full_volume.get_identifier().recreate_object()
assert isinstance(obj_recreated, volume_constructor)
obj_recreated.load()
numpy.testing.assert_array_equal(
obj_recreated.data,
volume_1.data,
)
numpy.testing.assert_array_equal(
obj_recreated.metadata,
volume_1.metadata,
)
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_example(volume_constructor):
"""test static function 'example'"""
assert isinstance(volume_constructor.example_defined_from_str_identifier(), str)
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_get_histogram(tmp_path, volume_constructor):
"""test 'get_histogram' API on single frame volumes (edf, tiff, jp2k)"""
volume_dir = tmp_path / "volume_get_histogram"
os.makedirs(volume_dir)
file_prefix = "volume"
volume = volume_constructor(
folder=volume_dir,
data=_data[:5],
volume_basename=file_prefix,
)
if isinstance(volume, JP2KVolume):
volume.rescale_data = False
volume.save()
assert volume.get_histogram_url() is None
assert volume.get_histogram() is None
stored_counts = numpy.arange(6, dtype="f")
stored_bins = numpy.linspace(0, 1, 6, dtype="f")
stored_histogram = numpy.array([stored_counts, stored_bins])
# nabu default naming: '{volume_basename}_histogram.hdf5'
histogram_file = os.path.join(volume_dir, f"{file_prefix}_histogram.hdf5")
with File(histogram_file, mode="w") as h5f:
h5f.create_dataset("entry0000/histogram/results/data", data=stored_histogram)
counts, bins = volume.get_histogram()
numpy.testing.assert_array_equal(counts, stored_counts)
numpy.testing.assert_array_equal(bins, stored_bins)
# ambiguous histogram file (several entries): a 'scan' with an 'entry' attribute must be provided
with File(histogram_file, mode="a") as h5f:
h5f.create_dataset("entry0001/histogram/results/data", data=stored_histogram)
assert volume.get_histogram() is None
class MockScan:
entry = "entry0001"
assert volume.get_histogram(scan=MockScan()) is not None
# legacy naming (no underscore): '{volume_basename}histogram.hdf5'
os.remove(histogram_file)
legacy_histogram_file = os.path.join(volume_dir, f"{file_prefix}histogram.hdf5")
with File(legacy_histogram_file, mode="w") as h5f:
h5f.create_dataset("entry0000/histogram/results/data", data=stored_histogram)
counts, bins = volume.get_histogram()
numpy.testing.assert_array_equal(counts, stored_counts)
numpy.testing.assert_array_equal(bins, stored_bins)
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_compute_histogram(tmp_path, volume_constructor):
"""test recomputing the histogram from slices when none can be found on disk"""
volume_dir = tmp_path / "volume_compute_histogram"
os.makedirs(volume_dir)
data = _data[:5]
volume = volume_constructor(folder=volume_dir, data=data, volume_basename="volume")
if isinstance(volume, JP2KVolume):
volume.rescale_data = False
volume.save()
assert volume.get_histogram() is None
counts, bin_edges = volume.get_histogram(recompute_if_missing=True, n_bins=16)
expected_counts, expected_edges = numpy.histogram(
data.ravel(), bins=16, range=(data.min(), data.max())
)
numpy.testing.assert_array_equal(counts, expected_counts)
numpy.testing.assert_allclose(bin_edges, expected_edges)
def test_backward_compatibility_z_index_4(tmp_path):
"""
make sure we are still able to read old volumes with the new class
"""
acquisition_dir = tmp_path / "acquisition"
os.makedirs(acquisition_dir)
volume_dir = str(acquisition_dir / "volume")
os.makedirs(volume_dir)
class LegacyEDFVolume(EDFVolume):
DEFAULT_DATA_PATH_PATTERN = "{volume_basename}_{index_zfill4}.{data_extension}"
legacy_volume = LegacyEDFVolume(
folder=volume_dir,
data=_data,
)
legacy_volume.save()
first_frame_path = os.path.join(volume_dir, "volume_0000.edf")
assert os.path.exists(first_frame_path)
# test loading with the legacy class
legacy_volume.clear_cache()
legacy_volume.load()
numpy.testing.assert_array_almost_equal(_data, legacy_volume.data)
# test loading with the updated EDFVolume class
volume = EDFVolume(
folder=volume_dir,
)
volume.load()
numpy.testing.assert_array_almost_equal(_data, legacy_volume.data)
@pytest.mark.parametrize("save_volume_to_file", (True, False))
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_get_slices(tmp_path, volume_constructor, save_volume_to_file):
"""test the volume 'get_slices' function"""
test_dir = tmp_path / "test_get_slices"
test_dir.mkdir()
volume_dir = os.path.join(test_dir, "volume")
volume_shape = 20, 100, 50
volume_n_elemts = volume_shape[0] * volume_shape[1] * volume_shape[2]
data = numpy.linspace(
0,
volume_n_elemts,
num=volume_n_elemts,
endpoint=False,
).reshape(volume_shape)
volume = volume_constructor(folder=volume_dir, data=data, metadata={})
if save_volume_to_file:
volume.save()
volume.clear_cache()
# test raising error
with pytest.raises(TypeError):
volume.get_slices(
"toto",
)
with pytest.raises(TypeError):
volume.get_slices(
(0, 10),
)
# test retrieving results
slices_axis_0 = volume.get_slices(((0, 2), (0, 5)))
slices_axis_1 = volume.get_slices(((1, 0), (1, 6)))
slices_axis_2 = volume.get_slices(((2, 8), (2, 9)))
slices_several_axis = volume.get_slices(((0, 2), (1, 0), (2, 9)))
if volume_constructor is JP2KVolume:
# JP2K is modifying original slice values. So we cannot compare them with
# the original one.
return
numpy.testing.assert_array_almost_equal(
slices_axis_0[(0, 2)],
data[2],
)
numpy.testing.assert_array_almost_equal(
slices_axis_1[(1, 6)],
data[:, 6, :],
)
numpy.testing.assert_array_almost_equal(
slices_axis_2[(2, 9)],
data[:, :, 9],
)
numpy.testing.assert_array_almost_equal(
slices_several_axis[(0, 2)],
slices_axis_0[(0, 2)],
)
numpy.testing.assert_array_almost_equal(
slices_several_axis[(1, 0)],
slices_axis_1[(1, 0)],
)
numpy.testing.assert_array_almost_equal(
slices_several_axis[(2, 9)],
slices_axis_2[(2, 9)],
)
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_load_chunk(tmp_path, volume_constructor):
volume_dir = str(tmp_path / "volume")
os.makedirs(volume_dir)
volume = volume_constructor(folder=volume_dir)
if isinstance(volume, JP2KVolume): # bis repetita (non) placent
volume.rescale_data = False
volume.data = numpy.arange(32 * 33 * 34, dtype="f").reshape(
(32, 33, 34)
) # put a not-too-small number for jpeg2000
volume.save_data()
volume_reader = volume_constructor(folder=volume_dir)
volume_reader.load_data()
data_all = volume_reader.data
chunks_to_test = [
(slice(1, 3), slice(None), slice(None)),
(slice(-7, None), slice(None), slice(None)),
(slice(None), slice(0, 3), slice(None)),
(slice(None), slice(None), slice(5, 9)),
(slice(None), slice(None), slice(None)),
]
for chunk in chunks_to_test:
data = volume.load_chunk(chunk)
assert numpy.allclose(data, data_all[chunk])
@pytest.mark.parametrize("volume_constructor", volume_constructors)
def test_get_volume_size(tmp_path, volume_constructor):
"""
'get_volume_size' must return the exact size once data is loaded, and an
estimated size - computed by peeking at a single frame instead of loading
the full volume - beforehand. Both must agree.
"""
volume_dir = str(tmp_path / "volume")
os.makedirs(volume_dir)
volume = volume_constructor(folder=volume_dir, data=_data)
if isinstance(volume, JP2KVolume):
volume.rescale_data = False
volume.save()
volume_reader = volume_constructor(folder=volume_dir)
assert volume_reader.data is None
estimated_size = volume_reader.get_volume_size()
volume_reader.load_data()
assert volume_reader.get_volume_size() == volume_reader.data.nbytes
assert estimated_size == volume_reader.data.nbytes
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_tiffvolume.py 0000664 0000000 0000000 00000031010 15241025157 0027616 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""
Test of the tiffmodule.
Note: some test are also perform on the test_single_frame_volume
"""
import os
import numpy
import pytest
from h5py import File
from silx.io.url import DataUrl
from tifffile import TiffFile
from volio.identifiers.base import VolumeIdentifier
from volio.identifiers.tiffidentifier import MultiTiffVolumeIdentifier
from volio.tests.mock import create_volume
from volio.volumes.tiffvolume import MultiTIFFVolume, TIFFVolume
_data = create_volume(frame_dims=(20, 20), z_size=5)
_data = _data.astype(numpy.uint8)
_metadata = {
"nabu_config": {
"dataset": {
"location": "toto.hdf5",
"entry": "entry0000",
},
"phase": {
"method": "None",
"padding_type": "edge",
},
},
"processing_option": {
"build_sino": {
"axis_correction": "None",
"enable_halftomo": True,
},
"flatfield": {
"binning": "11",
"do_flat_distortion": False,
},
},
}
def test_multi_tiff_volume(tmp_path):
test_dir = str(tmp_path / "test_volume_url")
os.makedirs(test_dir)
volume_file = os.path.join(test_dir, "volume.tiff")
volume = MultiTIFFVolume(
file_path=volume_file,
data=_data,
metadata=_metadata,
)
assert volume.url.file_path() == volume_file
assert volume.url.data_path() is None
assert volume.data_url.scheme() == "tifffile"
assert volume.data_url.file_path() == volume_file
assert volume.data_url.data_path() is None
assert volume.metadata_url.scheme() == "ini"
expected_metadata_file = os.path.join(test_dir, "volume_infos.txt")
assert volume.metadata_url.file_path() == expected_metadata_file
assert volume.metadata_url.data_path() is None
volume_identifier = volume.get_identifier()
assert isinstance(volume_identifier.short_description(), str)
assert isinstance(volume_identifier, VolumeIdentifier)
assert not os.path.exists(volume_file)
assert tuple(volume.browse_data_files()) == tuple()
assert tuple(volume.browse_data_urls()) == tuple()
assert tuple(volume.browse_metadata_files()) == tuple()
volume.save()
assert os.path.exists(volume_file)
assert tuple(volume.browse_metadata_files()) == (expected_metadata_file,)
assert tuple(volume.browse_data_files()) == (volume_file,)
assert len(list(volume.browse_data_urls())) == len((volume.data_url,))
volume_loaded = MultiTIFFVolume(
file_path=volume_file,
)
assert volume_loaded.metadata_url.file_path() == expected_metadata_file
assert volume_loaded.metadata_url.data_path() is None
assert volume_loaded._data is None
assert volume_loaded._metadata is None
# check loading function won't save loaded data in cache
data_no_cache = volume_loaded.load_data(store=False)
metadata_no_cache = volume_loaded.load_metadata(store=False)
assert data_no_cache is not None
assert metadata_no_cache is not None
assert volume_loaded._data is None
assert volume_loaded._metadata is None
numpy.testing.assert_array_almost_equal(_data, data_no_cache)
assert _metadata == metadata_no_cache
assert volume.get_min_max_values() == (_data.min(), _data.max())
# check data and metadata properties will load and set data in cache
volume_loaded.load()
data_cache = volume_loaded.data
assert data_cache is not None
metadata_cache = volume_loaded.metadata
assert metadata_cache is not None
numpy.testing.assert_array_almost_equal(_data, data_cache)
assert _metadata == metadata_cache
volume_loaded.data = None
volume_loaded.metadata = None
with pytest.raises(ValueError):
volume_loaded.save_data()
with pytest.raises(ValueError):
volume_loaded.save_metadata()
with pytest.raises(TypeError):
volume_loaded.data = "toto"
volume_loaded.data = create_volume(frame_dims=(20, 20), z_size=2)
with pytest.raises(ValueError):
volume_loaded.data = create_volume(frame_dims=(20, 20), z_size=2)[0]
# get the volume from the identifier
volume_from_id = MultiTIFFVolume.from_identifier(identifier=volume_identifier)
assert volume_from_id.data is None
assert volume_from_id.metadata is None
# get the volume from the identifier str
volume_from_str = MultiTIFFVolume.from_identifier(
identifier=MultiTiffVolumeIdentifier.from_str(volume_identifier.to_str())
)
assert volume_from_str.data is None
assert volume_from_str.metadata is None
assert volume_from_str.get_identifier() == volume_from_id.get_identifier()
assert volume_from_str.get_identifier() != "toto"
# test browsing frames
for l_frame, o_frame in zip(volume_from_str.browse_slices(), _data):
numpy.testing.assert_allclose(l_frame, o_frame)
# test hash
hash(volume_from_str)
# test append param
volume_file_append = os.path.join(test_dir, "volume_append.tiff")
volume_append = MultiTIFFVolume(
file_path=volume_file_append, data=_data, metadata=_metadata, append=True
)
volume_append.data = create_volume(frame_dims=(20, 20), z_size=2)
volume_append.save()
volume_append2 = MultiTIFFVolume(
file_path=volume_file_append, data=_data, metadata=_metadata, append=True
)
volume_append2.data = create_volume(frame_dims=(20, 20), z_size=3) + 10
volume_append2.save()
with TiffFile(volume_file_append) as tiff_append:
assert len(tiff_append.series) == 5
# test load_chunk
chunks_to_test = [
(slice(None, None), slice(0, 5), slice(None, None)),
(slice(0, 4), slice(0, 15), slice(None, None)),
]
for chunk in chunks_to_test:
data_chunk = volume.load_chunk(chunk)
assert data_chunk.shape == _data[chunk].shape
assert numpy.allclose(data_chunk, _data[chunk])
assert volume.data_extension == "tiff"
assert volume.metadata_extension == "txt"
def test_multi_tiff_volume_get_histogram(tmp_path):
"""test 'get_histogram' API on a MultiTIFFVolume"""
test_dir = str(tmp_path / "test_get_histogram")
os.makedirs(test_dir)
volume_file = os.path.join(test_dir, "volume.tiff")
data = _data.astype(numpy.float32)
volume = MultiTIFFVolume(file_path=volume_file, data=data)
volume.save()
assert volume.get_histogram_url() is None
assert volume.get_histogram() is None
stored_counts = numpy.arange(6, dtype="f")
stored_bins = numpy.linspace(0, 1, 6, dtype="f")
stored_histogram = numpy.array([stored_counts, stored_bins])
histogram_file = os.path.join(test_dir, "volume_histogram.hdf5")
with File(histogram_file, mode="w") as h5f:
h5f.create_dataset("entry0000/histogram/results/data", data=stored_histogram)
counts, bins = volume.get_histogram()
numpy.testing.assert_array_equal(counts, stored_counts)
numpy.testing.assert_array_equal(bins, stored_bins)
# no histogram file found: recompute it from slices
os.remove(histogram_file)
counts, bin_edges = volume.get_histogram(recompute_if_missing=True, n_bins=8)
expected_counts, expected_edges = numpy.histogram(
data.ravel(), bins=8, range=(data.min(), data.max())
)
numpy.testing.assert_array_equal(counts, expected_counts)
numpy.testing.assert_allclose(bin_edges, expected_edges)
def test_data_file_saver_generator(tmp_path):
"""
data_file_saver_generator (dumping frame by frame a volume to disk)
"""
volume_file_path = str(tmp_path / "volume.h5")
volume = MultiTIFFVolume(file_path=volume_file_path)
for slice_, slice_saver in zip(
_data,
volume.data_file_saver_generator(n_frames=_data.shape[0], overwrite=False),
):
slice_saver[:] = slice_
assert volume.data is None
numpy.testing.assert_array_equal(
volume.load_data(),
_data,
)
def test_tiff_volume_nabu_single_frame_pattern(tmp_path):
"""
Insure we can save and load using a specific pattermn (single frame volume) that can happen in nabu for example
"""
test_dir = str(tmp_path / "test_volume_url")
os.makedirs(test_dir)
output_file = os.path.join(test_dir, "frame.tiff")
class SingleFrameTIFFVolume(TIFFVolume):
# we are not insure that the output directory name is the base name of the file_path
DEFAULT_DATA_PATH_PATTERN = "frame.tiff"
frame = _data[0].reshape(1, _data.shape[1], _data.shape[2])
volume = SingleFrameTIFFVolume(
folder=test_dir,
data=frame,
metadata=_metadata,
overwrite=True,
)
volume.save()
assert os.path.exists(output_file)
volume_loaded_1 = SingleFrameTIFFVolume(
folder=test_dir,
)
volume_loaded_1.load()
numpy.testing.assert_array_equal(
frame,
volume_loaded_1.data,
)
volume_loaded_2 = TIFFVolume(
data_url=DataUrl(
file_path=output_file,
data_path="frame.tiff",
scheme="tifffile",
),
)
volume_loaded_2.load_data(store=True)
numpy.testing.assert_array_equal(
frame,
volume_loaded_2.data,
)
# test get_slice function
with pytest.raises(IndexError):
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=0),
_data[2],
)
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=0, axis=0),
_data[0],
)
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=1),
frame[:, 2, :],
)
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=2),
frame[:, :, 2],
)
volume_loaded_2.data = None
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=1),
frame[:, 2, :],
)
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=2),
frame[:, :, 2],
)
def test_tiff_volume_nabu_multi_frame_pattern(tmp_path):
"""
Insure we can save and load using a specific pattermn (multi frame volume) that can happen in nabu for example
"""
test_dir = str(tmp_path / "test_volume_url")
os.makedirs(test_dir)
class MultiFrameTIFFVolume(TIFFVolume):
# we are not insure that the output directory name is the base name of the file_path
DEFAULT_DATA_PATH_PATTERN = "frame_{index_zfill6}.tiff"
volume = MultiFrameTIFFVolume(
folder=test_dir,
data=_data,
metadata=_metadata,
overwrite=True,
)
volume.save()
output_file_first_frame = os.path.join(test_dir, "frame_000000.tiff")
assert os.path.exists(output_file_first_frame)
volume_loaded_1 = MultiFrameTIFFVolume(
folder=test_dir,
)
volume_loaded_1.load()
numpy.testing.assert_array_equal(
_data,
volume_loaded_1.data,
)
volume_loaded_2 = TIFFVolume(
data_url=DataUrl(
file_path=test_dir,
data_path="frame_{index_zfill6}.tiff",
scheme="tifffile",
),
)
volume_loaded_2.load_data(store=True)
numpy.testing.assert_array_equal(
_data,
volume_loaded_2.data,
)
# test get_slice function
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=0),
_data[2],
)
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=1),
_data[:, 2, :],
)
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=2),
_data[:, :, 2],
)
volume_loaded_2.data = None
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=0),
_data[2],
)
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=1),
_data[:, 2, :],
)
numpy.testing.assert_array_equal(
volume_loaded_2.get_slice(index=2, axis=2),
_data[:, :, 2],
)
def test_get_volume_size(tmp_path):
"""
'get_volume_size' must return the exact size when data is loaded, and an
estimated (but equal) size when computed from the tiff file without
loading the full data.
"""
volume_file = str(tmp_path / "volume_size.tiff")
volume = MultiTIFFVolume(file_path=volume_file, data=_data)
volume.save()
assert volume.get_volume_size() == _data.nbytes
volume_unloaded = MultiTIFFVolume(file_path=volume_file)
assert volume_unloaded.data is None
assert volume_unloaded.get_volume_size() == _data.nbytes
def test_example():
"""test static function 'example'"""
assert isinstance(MultiTIFFVolume.example_defined_from_str_identifier(), str)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_utils.py 0000664 0000000 0000000 00000025006 15241025157 0026606 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""tests for volio.utils, merging what used to be two separate "volume utils" test
modules in tomoscan: test_volume_esrf_utils.py (guessing volumes on disk) and
test_volume_utils.py (concatenation, metadata merging)."""
import os
import shutil
from copy import deepcopy
import numpy
import pytest
from volio.utils import (
concatenate,
get_most_common_extension,
guess_hdf5_volume_data_paths,
guess_volumes,
update_metadata,
)
from volio.volumes.edfvolume import EDFVolume
from volio.volumes.hdf5volume import HDF5Volume
from volio.volumes.jp2kvolume import JP2KVolume, has_minimal_openjpeg
from volio.volumes.tiffvolume import MultiTIFFVolume, TIFFVolume
@pytest.mark.parametrize("file_extension", ("h5", "hdf5"))
def test_guess_volumes_hdf5(tmp_path, file_extension):
"""
Test `guess_volumes` function within hdf5 file
"""
assert guess_volumes(path=tmp_path) == tuple()
file_path = os.path.join(tmp_path, ".".join(["my_file", file_extension]))
# check error if path does not exists
with pytest.raises(OSError):
guess_volumes(path=file_path)
volume = HDF5Volume(
file_path=file_path,
data_path="entry",
data=numpy.linspace(1, 10, 100).reshape(1, 10, 10),
)
volume.save()
guess_vols = guess_volumes(path=file_path)
assert len(guess_vols) == 1
guess_vol = guess_vols[0]
assert isinstance(guess_vol, HDF5Volume)
assert guess_vol.file_path == file_path
assert guess_vol.data_path == "/entry"
HDF5Volume(
file_path=file_path,
data_path="entry0002",
data=numpy.linspace(1, 10, 100).reshape(1, 10, 10),
).save()
assert len(guess_volumes(path=file_path)) == 2
# a few more test on 'guess_hdf5_volume_data_paths'
assert len(guess_hdf5_volume_data_paths(file_path=file_path, depth=0)) == 0
assert (
len(
guess_hdf5_volume_data_paths(
file_path=file_path, data_path="/entry", depth=0
)
)
== 1
)
_constructors = [EDFVolume, TIFFVolume]
if has_minimal_openjpeg:
_constructors.append(JP2KVolume)
@pytest.mark.parametrize("Constructor", _constructors)
def test_guess_volumes_single_frame_file(tmp_path, Constructor):
"""
Test `guess_volumes` function for single frame volume (like EDFVolume, TIFFVolume and Jp2KVolume)
"""
assert guess_volumes(path=tmp_path) == tuple()
folder = os.path.join(tmp_path, "my_volume_folder")
# check error if path does not exists
with pytest.raises(OSError):
guess_volumes(path=folder)
volume = Constructor(
folder=folder,
data=numpy.linspace(1, 10, 30000, dtype=numpy.uint8).reshape(3, 100, 100),
)
volume.save()
guess_vols = guess_volumes(path=folder)
assert len(guess_vols) == 1
guess_vol = guess_vols[0]
assert isinstance(guess_vol, Constructor)
assert guess_vol.data_url.file_path() == folder
assert guess_vol.get_volume_basename() == os.path.basename(folder)
# test adding some noise in the folder
for _ in range(15):
file_name = os.path.join(folder, "my_volume_folder.txt")
with open(file_name, mode="w") as f:
f.write("almost empty text file")
for other_file in ("other_file.tif", "other_file.tiff"):
with open(os.path.join(folder, other_file), mode="w") as f:
f.write("none")
guess_vols = guess_volumes(path=folder)
assert len(guess_vols) == 1
shutil.rmtree(folder)
# test with a basename
volume = Constructor(
folder=folder,
volume_basename="other_basename",
data=numpy.linspace(1, 10, 10000, dtype=numpy.uint8).reshape(1, 100, 100),
)
volume.save()
guess_vols = guess_volumes(path=folder)
assert len(guess_vols) == 1
assert guess_vols[0].get_volume_basename() == "other_basename"
def test_guess_volumes_multitiff(tmp_path):
"""
Test `guess_volumes` function for MultiTiffVolume
"""
assert guess_volumes(path=tmp_path) == tuple()
file_path = os.path.join(tmp_path, "my_multitiff.tiff")
# check error if path does not exists
with pytest.raises(OSError):
guess_volumes(path=file_path)
volume = MultiTIFFVolume(
file_path=file_path, data=numpy.linspace(1, 10, 200).reshape(2, 10, 10)
)
volume.save()
guess_vols = guess_volumes(path=file_path)
assert len(guess_vols) == 1
guess_vol = guess_vols[0]
assert isinstance(guess_vol, MultiTIFFVolume)
assert guess_vol.data_url.file_path() == file_path
def test_get_most_common_extension(tmp_path):
"""test 'get_most_common_extension' raises when given a non-folder path"""
with pytest.raises(ValueError):
get_most_common_extension(str(tmp_path / "not_a_folder.txt"))
_clases_to_test = [EDFVolume, HDF5Volume, TIFFVolume]
if has_minimal_openjpeg:
_clases_to_test.append(JP2KVolume)
def test_concatenate_volume_errors():
"""test some error raised by volio.utils.concatenate function"""
vol = HDF5Volume(
file_path="toto",
data_path="test",
)
with pytest.raises(TypeError):
(concatenate(output_volume=1, volumes=(), axis=1),)
with pytest.raises(TypeError):
concatenate(output_volume=vol, volumes=(), axis="1")
with pytest.raises(TypeError):
concatenate(output_volume=vol, volumes=(), axis="1")
with pytest.raises(ValueError):
concatenate(output_volume=vol, volumes=(), axis=6)
with pytest.raises(ValueError):
concatenate(output_volume=vol, volumes=(1,), axis=1)
with pytest.raises(TypeError):
concatenate(output_volume=vol, volumes="toto", axis=1)
@pytest.mark.parametrize("axis", (0, 1, 2))
@pytest.mark.parametrize("volume_class", _clases_to_test)
def test_concatenate_volume(tmp_path, volume_class, axis):
"""
test concatenation of 3 volumes into a single one
"""
# create folder to save data (and debug)
raw_data_dir = tmp_path / "raw_data"
raw_data_dir.mkdir()
output_dir = tmp_path / "output_dir"
output_dir.mkdir()
param_set_1 = {
"data": numpy.ones((100, 100, 100), dtype=numpy.uint16),
"metadata": {
"this": {
"is": {"metadata": 1},
},
},
}
param_set_2 = {
"data": numpy.arange(100 * 100 * 100, dtype=numpy.uint16).reshape(
100, 100, 100
),
"metadata": {
"this": {
"is": {"metadata": 2},
"isn't": {
"something": 12.3,
},
},
},
}
param_set_3 = {
"data": numpy.zeros((100, 100, 100), dtype=numpy.uint16),
"metadata": {
"yet": {
"another": {
"metadata": 12,
},
},
},
}
volumes = []
param_sets = (param_set_1, param_set_2, param_set_3)
for i_vol, vol_params in enumerate(param_sets):
if volume_class == HDF5Volume:
vol_params.update(
{
"file_path": os.path.join(raw_data_dir, f"volume_{i_vol}.hdf5"),
"data_path": "volume",
}
)
else:
vol_params.update({"folder": os.path.join(raw_data_dir, f"volume_{i_vol}")})
volume = volume_class(**vol_params)
if isinstance(volume, JP2KVolume):
volume.rescale_data = False # simplify test
volume.save()
volumes.append(volume)
volume.data = None
volume.metadata = None
volumes = tuple(volumes)
# apply concatenation
if volume_class == HDF5Volume:
final_volume = HDF5Volume(
file_path=os.path.join(output_dir, "final_vol.hdf5"),
data_path="volume",
)
else:
final_volume = volume_class(
folder=os.path.join(output_dir, "final_vol"),
)
if isinstance(final_volume, JP2KVolume):
final_volume.rescale_data = False
concatenate(output_volume=final_volume, volumes=volumes, axis=axis)
if axis == 0:
expected_final_shape = (300, 100, 100)
elif axis == 1:
expected_final_shape = (100, 300, 100)
elif axis == 2:
expected_final_shape = (100, 100, 300)
else:
raise RuntimeError("axis should be in (0, 1, 2)")
assert final_volume.data is None
assert final_volume.get_volume_shape() == expected_final_shape
final_volume.load()
assert "this" in final_volume.metadata
[volume.load() for volume in volumes]
if axis == 0:
numpy.testing.assert_almost_equal(final_volume.data[0:100], volumes[0].data)
numpy.testing.assert_almost_equal(final_volume.data[100:200], volumes[1].data)
numpy.testing.assert_almost_equal(final_volume.data[200:300], volumes[2].data)
elif axis == 1:
numpy.testing.assert_almost_equal(final_volume.data[:, 0:100], volumes[0].data)
numpy.testing.assert_almost_equal(
final_volume.data[:, 100:200], volumes[1].data
)
numpy.testing.assert_almost_equal(
final_volume.data[:, 200:300], volumes[2].data
)
elif axis == 2:
numpy.testing.assert_almost_equal(
final_volume.data[:, :, 0:100], volumes[0].data
)
numpy.testing.assert_almost_equal(
final_volume.data[:, :, 100:200], volumes[1].data
)
numpy.testing.assert_almost_equal(
final_volume.data[:, :, 200:300], volumes[2].data
)
final_volume.overwrite = False
with pytest.raises(OSError):
concatenate(output_volume=final_volume, volumes=volumes, axis=axis)
final_volume.overwrite = True
concatenate(output_volume=final_volume, volumes=volumes, axis=axis)
def test_update_metadata():
ddict_1 = {
"key": {
"sub_key_1": "toto",
"sub_key_2": "tata",
},
"second_key": "test",
}
ddict_2 = {
"key": {
"sub_key_1": "test",
"sub_key_3": "test",
},
"third_key": "test",
}
assert update_metadata(deepcopy(ddict_1), deepcopy(ddict_2)) == {
"key": {
"sub_key_1": "test",
"sub_key_2": "tata",
"sub_key_3": "test",
},
"second_key": "test",
"third_key": "test",
}
assert update_metadata(deepcopy(ddict_2), deepcopy(ddict_1)) == {
"key": {
"sub_key_1": "toto",
"sub_key_2": "tata",
"sub_key_3": "test",
},
"second_key": "test",
"third_key": "test",
}
with pytest.raises(TypeError):
update_metadata(1, 2)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/tests/test_volume_base.py 0000664 0000000 0000000 00000005065 15241025157 0027752 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""Module containing validators"""
import numpy
import pytest
from volio.volumes.volumebase import VolumeBase
def test_volume_base():
"""Test VolumeBase file"""
class UnimplementedVolumeBase(VolumeBase):
def deduce_data_and_metadata_urls(self, url):
return None, None
volume_base = UnimplementedVolumeBase()
with pytest.raises(NotImplementedError):
volume_base.example_defined_from_str_identifier()
with pytest.raises(NotImplementedError):
volume_base.get_identifier()
with pytest.raises(NotImplementedError):
VolumeBase.from_identifier("")
with pytest.raises(NotImplementedError):
volume_base.save_data()
with pytest.raises(NotImplementedError):
volume_base.save_metadata()
with pytest.raises(NotImplementedError):
volume_base.save()
with pytest.raises(NotImplementedError):
volume_base.load_data()
with pytest.raises(NotImplementedError):
volume_base.load_metadata()
with pytest.raises(NotImplementedError):
volume_base.load()
with pytest.raises(NotImplementedError):
volume_base.browse_data_files()
with pytest.raises(NotImplementedError):
volume_base.browse_metadata_files()
with pytest.raises(NotImplementedError):
volume_base.browse_data_urls()
with pytest.raises(NotImplementedError):
volume_base.get_volume_shape()
with pytest.raises(NotImplementedError):
volume_base.get_volume_dtype()
with pytest.raises(NotImplementedError):
# no data set and no way to get shape / dtype from disk (base class)
volume_base.get_volume_size()
data = numpy.arange(2 * 3 * 4, dtype=numpy.float32).reshape((2, 3, 4))
volume_base.data = data
assert volume_base.get_volume_dtype() == data.dtype
assert volume_base.get_volume_size() == data.nbytes
assert volume_base.get_volume_size(unit="kilobyte") == pytest.approx(
data.nbytes / 1000
)
volume_base.data = None
volume_base.position = (0, 1, 2)
assert isinstance(volume_base.position, tuple)
assert volume_base.position == (0, 1, 2)
volume_base.voxel_size = (12.3, 2.5, 6.9)
assert volume_base.voxel_size == (12.3, 2.5, 6.9)
assert type(volume_base.voxel_size[0]) is float
# by default no histogram location is known and none should be recomputed
assert volume_base.get_histogram_url() is None
assert volume_base.get_histogram() is None
with pytest.raises(NotImplementedError):
volume_base.get_histogram(recompute_if_missing=True)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/utils.py 0000664 0000000 0000000 00000043613 15241025157 0024411 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""utils function for volumes
This module merges what used to be two separate "volume utils" modules in tomoscan:
`tomoscan.esrf.volume.utils` (guessing volumes on disk) and `tomoscan.utils.volume`
(concatenation, metadata merging, data rescaling).
"""
from __future__ import annotations
import logging
import os
from collections.abc import Mapping
import h5py
import numpy
from silx.io.utils import open as hdf5_open
from tomobase.utils.hdf5 import DatasetReader
from volio.identifiers.edfidentifier import EDFVolumeIdentifier
from volio.identifiers.hdf5identifier import HDF5VolumeIdentifier
from volio.identifiers.jp2kidentifier import JP2KVolumeIdentifier
from volio.identifiers.rawidentifier import RawVolumeIdentifier
from volio.identifiers.tiffidentifier import (
MultiTiffVolumeIdentifier,
TIFFVolumeIdentifier,
)
from volio.volumes.edfvolume import EDFVolume
from volio.volumes.hdf5volume import HDF5Volume
from volio.volumes.jp2kvolume import JP2KVolume
from volio.volumes.rawvolume import RawVolume
from volio.volumes.tiffvolume import MultiTIFFVolume, TIFFVolume
from volio.volumes.volumebase import SliceTuple, VolumeBase
_logger = logging.getLogger(__name__)
__all__ = [
"guess_hdf5_volume_data_paths",
"guess_volumes",
"get_most_common_extension",
"group_slices_by_axis",
"concatenate",
"update_metadata",
"rescale_data",
]
_DEFAULT_SCHEME_TO_VOL = {
EDFVolumeIdentifier.scheme: EDFVolume,
HDF5VolumeIdentifier.scheme: HDF5Volume,
TIFFVolumeIdentifier.scheme: TIFFVolume,
MultiTiffVolumeIdentifier.scheme: MultiTIFFVolume,
JP2KVolumeIdentifier.scheme: JP2KVolume,
RawVolumeIdentifier.scheme: RawVolume,
}
def guess_hdf5_volume_data_paths(file_path, data_path="/", depth=3) -> tuple:
"""
browse hdf5 file 'file_path' from 'data_path' on 'depth' level and check for possible defined volumes.
:param file_path: file path to the hdf5 file to browse
:param data_path: path in the file to start research
:param depth: on which layer we should apply research
:return: tuple of data_path that could fit a volume
"""
if not h5py.is_hdf5(file_path):
raise ValueError(f"{file_path} is not a hdf5 file path")
with hdf5_open(filename=file_path) as h5f:
group = h5f[data_path]
if isinstance(group, h5py.Group):
if HDF5Volume.DATA_DATASET_NAME in group:
return (data_path,)
elif depth > 0:
res = []
for key in group.keys():
res.extend(
guess_hdf5_volume_data_paths(
file_path=file_path,
data_path="/".join((data_path, key)).replace("//", "/"),
depth=depth - 1,
)
)
return tuple(res)
return tuple()
def guess_volumes(
path, scheme_to_vol: dict | None = None, filter_histograms=True
) -> tuple:
"""
from a file path or a folder path try to guess volume(s)
:param path: file or folder path
:param scheme_to_vol: dict to know which constructor to call. Key if the scheme, value if the volume constructor.
usefull for libraries redefining volume or adding some like tomwer.
If none provided will take the volio default one
:return: tuple of volume
"""
if not os.path.exists(path):
raise OSError("path doesn't exists")
if scheme_to_vol is None:
scheme_to_vol = _DEFAULT_SCHEME_TO_VOL
if os.path.isfile(path):
if h5py.is_hdf5(path):
res = []
for data_path in guess_hdf5_volume_data_paths(path):
assert isinstance(data_path, str)
res.append(
scheme_to_vol[HDF5VolumeIdentifier.scheme](
file_path=path,
data_path=data_path,
)
)
# filter potential 'nabu histogram'
# as nabu histograms looks like volume simply look at the name
# could also be on data ndim
if filter_histograms is True:
def is_not_histogram(vol_identifier):
return not (
hasattr(vol_identifier, "data_path")
and vol_identifier.data_path.endswith("histogram")
)
res = tuple(filter(is_not_histogram, res))
return tuple(res)
elif path.lower().endswith((".tif", ".tiff")):
return (scheme_to_vol[MultiTiffVolumeIdentifier.scheme](file_path=path),)
elif path.lower().endswith((".vol", ".raw")):
return (scheme_to_vol[RawVolumeIdentifier.scheme](file_path=path),)
elif os.path.isdir(path):
most_common_extension = get_most_common_extension(path)
if most_common_extension is None:
return tuple()
basename = _guess_volume_basename(path, extension=most_common_extension)
if most_common_extension in ("tiff", "tif"):
return (
scheme_to_vol[TIFFVolumeIdentifier.scheme](
folder=path,
volume_basename=basename,
data_extension=most_common_extension,
),
)
elif most_common_extension in ("jp2", "jp2k"):
return (
scheme_to_vol[JP2KVolumeIdentifier.scheme](
folder=path,
volume_basename=basename,
data_extension=most_common_extension,
),
)
elif most_common_extension == "edf":
return (
scheme_to_vol[EDFVolumeIdentifier.scheme](
folder=path,
volume_basename=basename,
data_extension=most_common_extension,
),
)
else:
_logger.warning(
f"most common extension is {most_common_extension}. Unable to create a volume from it"
)
return tuple()
else:
raise NotImplementedError("guess_volumes only handle file and folder...")
def get_most_common_extension(folder_path):
if not os.path.isdir(folder_path):
raise ValueError(f"a folder path is expected. {folder_path} isn't")
extensions = {}
for file_path in os.listdir(folder_path):
_, ext = os.path.splitext(file_path)
ext = ext.lower().lstrip(".")
if ext in extensions:
extensions[ext] += 1
else:
extensions[ext] = 1
# filter not handled extensions
def is_valid_extension(extension):
return extension in ("edf", "tif", "tiff", "jp2", "jp2k")
extensions = {
key: value for (key, value) in extensions.items() if is_valid_extension(key)
}
if len(extensions) == 0:
_logger.warning(f"no valid extensions found in {folder_path}")
else:
sort_extensions = sorted(extensions.items(), key=lambda x: x[1], reverse=True)
return sort_extensions[0][0]
def _guess_volume_basename(folder_path, extension):
# list all the files matching the file and guessing the file parttern
files_to_check = []
possible_basenames = {}
for file_path in os.listdir(folder_path):
if file_path.lower().endswith(extension):
files_to_check.append(os.path.splitext(file_path)[0])
# the expected way to save those files is basename_XXXX with XXXX is the index over 4 char
basename = "_".join(file_path.split("_")[:-1])
if basename in possible_basenames:
possible_basenames[basename] += 1
else:
possible_basenames[basename] = 1
if len(possible_basenames) == 0:
_logger.warning(f"no valid basename found in {folder_path}")
else:
sort_basenames = sorted(
possible_basenames.items(), key=lambda x: x[1], reverse=True
)
if len(sort_basenames) > 1:
_logger.warning(
f"more than one basename found. Take the most probable one ({sort_basenames[0][0]})"
)
return sort_basenames[0][0]
def group_slices_by_axis(
slices: tuple[SliceTuple], volume_shape: tuple[int] | None
) -> dict[int, set]:
"""
group a tuple of slices provided as SliceTuple (axis, slice) to a dict.
Dict contains predetermined keys (0, 1, 2). For each key with have a set of int (slice indices to be read)
volume_shape: shape of the reference volume to perform test on slice indices (if provided)
"""
indices_to_read_per_axis: dict[int, set[int]] = {
0: set(),
1: set(),
2: set(),
}
for slice_ in slices:
if volume_shape is not None and (slice_.index >= volume_shape[slice_.axis]):
raise IndexError(
f"index {slice_.index} is out of bounds for axis {slice_.axis}. (Volume shape is {volume_shape})"
)
indices_to_read_per_axis[slice_.axis].add(slice_.index)
return indices_to_read_per_axis
def concatenate(output_volume: VolumeBase, volumes: tuple, axis: int) -> None:
"""
Function to do 'raw' concatenation on volumes.
This is agnostic of any metadata. So if you want to ensure about coherence of metadata (and data) you must do it yourself
data will be concatenate in the order volumes are provided. Volumes data must be 3D. Concatenate data will be 3D and concatenation will be done
over the axis `axis`
concatenation will be done with a virtual dataset if input volumes and output_volume are HDF5Volume instances.
warning: concatenation enforce writing data and metadata to disk
:param output_volume VolumeBase: volume to create
:param tuple volumes: tuple of VolumeBase instances
:param axis: axis to use for doing the concatenation. must be in 0, 1, 2
"""
# 0. do some check
if not isinstance(output_volume, VolumeBase):
raise TypeError(
f"output_volume is expected to be an instance of {VolumeBase}. {type(output_volume)} provided"
)
if not isinstance(axis, int):
raise TypeError(f"axis must be an int. {type(axis)} provided")
elif axis not in (0, 1, 2):
raise ValueError(f"axis must be in (0, 1, 2). {axis} provided")
if not isinstance(volumes, tuple):
raise TypeError(f"volumes must be a tuple. {type(volumes)} provided")
else:
is_invalid = lambda y: not isinstance(y, VolumeBase) # noqa: E731
invalids = tuple(filter(is_invalid, volumes))
if len(invalids) > 0:
raise ValueError(f"Several non-volumes found. ({invalids})")
if isinstance(output_volume, JP2KVolume) and output_volume.rescale_data is True:
_logger.warning(
"concatenation will rescale data frame. If you want to avoid this please set output volume 'rescale_data' to False"
)
# 1. compute final shape
def get_volume_shape():
if axis == 0:
new_shape = [0, None, None]
elif axis == 1:
new_shape = [None, 0, None]
else:
new_shape = [None, None, 0]
for vol in volumes:
vol_shape = vol.get_volume_shape()
if vol_shape is None:
raise ValueError(
f"Unable to find shape for volume {vol.get_identifier().to_str()}"
)
new_shape[axis] += vol_shape[axis]
if axis == 0:
if new_shape[1] is None:
new_shape[1], new_shape[2] = vol_shape[1], vol_shape[2]
elif new_shape[1] != vol_shape[1] or new_shape[2] != vol_shape[2]:
raise ValueError("Found incoherent shapes. Unable to concatenate")
elif axis == 1:
if new_shape[0] is None:
new_shape[0], new_shape[2] = vol_shape[0], vol_shape[2]
elif new_shape[0] != vol_shape[0] or new_shape[2] != vol_shape[2]:
raise ValueError("Found incoherent shapes. Unable to concatenate")
else:
if new_shape[0] is None:
new_shape[0], new_shape[1] = vol_shape[0], vol_shape[1]
elif new_shape[0] != vol_shape[0] or new_shape[1] != vol_shape[1]:
raise ValueError("Found incoherent shapes. Unable to concatenate")
return tuple(new_shape)
final_shape = get_volume_shape()
if final_shape is None:
# should never be raised. Other error type is expected to be raised first
raise RuntimeError("Unable to get final volume shape")
# 2. Handle volume data (concatenation)
if isinstance(output_volume, HDF5Volume) and numpy.all(
[isinstance(vol, HDF5Volume)] for vol in volumes
):
# 2.1 in the case of HDF5 we can short cut this by creating a virtual dataset. Would highly speed up processing avoid copy
# note: in theory this could be done for any input_volume type using external dataset but we don't want to spend ages on
# this use case for now. Some work around this (using EDf) has been done in nxtomomill for information. See https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/115
_logger.info("start creation of external dataset")
with DatasetReader(volumes[0].data_url) as dataset:
data_type = dataset.dtype
# FIXME: avoid keeping some file open. not clear why this is needed
dataset = None
with h5py.File(output_volume.data_url.file_path(), mode="a") as h5s:
# 2.1.1 check data path
if output_volume.data_url.data_path() in h5s:
if output_volume.overwrite:
del h5s[output_volume.data_url.data_path()]
else:
raise OSError(
f"Unable to save data to {output_volume.data_url.data_path()}. This path already exists in {output_volume.data_url.file_path()}. If you want you can ask to overwrite it (from the output volume)."
)
# 2.1.2 create virtual layout
v_layout = h5py.VirtualLayout(
shape=final_shape,
dtype=data_type,
)
# 2.1.3 create virtual source
start_index = 0
for volume in volumes:
# provide relative path
rel_file_path = os.path.relpath(
volume.data_url.file_path(),
os.path.dirname(output_volume.data_url.file_path()),
)
rel_file_path = "./" + rel_file_path
data_path = volume.data_url.data_path()
vol_shape = volume.get_volume_shape()
vs = h5py.VirtualSource(
rel_file_path,
name=data_path,
shape=vol_shape,
)
stop_index = start_index + vol_shape[axis]
if axis == 0:
v_layout[start_index:stop_index] = vs
elif axis == 1:
v_layout[:, start_index:stop_index, :] = vs
elif axis == 2:
v_layout[:, :, start_index:stop_index] = vs
start_index = stop_index
# 2.1.4 create virtual dataset
h5s.create_virtual_dataset(
name=output_volume.data_url.data_path(), layout=v_layout
)
else:
# 2.1 default case (duplicate all input data slice by slice)
# 2.1.1 special case of the concatenation other axis 0
if axis == 0:
def iter_input():
for vol in volumes:
for data_slice in vol.browse_slices():
yield data_slice
for frame_dumper, input_slice in zip(
output_volume.data_file_saver_generator(
n_frames=final_shape[0],
overwrite=output_volume.overwrite,
),
iter_input(),
):
frame_dumper[:] = input_slice
else:
# 2.1.2 concatenation with data duplication over axis 1 or 2
for i_z, frame_dumper in enumerate(
output_volume.data_file_saver_generator(
n_frames=final_shape[0],
overwrite=output_volume.overwrite,
)
):
if axis == 1:
frame_dumper[:] = numpy.concatenate(
[vol.get_slice(axis=0, index=i_z) for vol in volumes],
axis=0,
)
elif axis == 2:
frame_dumper[:] = numpy.concatenate(
[vol.get_slice(axis=0, index=i_z) for vol in volumes],
axis=1,
)
else:
raise RuntimeError
# 3. handle metadata
for vol in volumes:
if vol.metadata is None:
try:
vol.load_metadata(store=True)
except Exception as e:
_logger.error(f"fail to load metadata for {vol}. Error is {e}")
output_volume.metadata = {}
[update_metadata(output_volume.metadata, vol.metadata) for vol in volumes]
output_volume.save_metadata()
def update_metadata(ddict_1: dict, ddict_2: dict) -> dict:
"""
update metadata ddict_1 from ddict_2
metadata are dict. And those dicts
warning: will modify ddict_1
"""
if not isinstance(ddict_1, dict) or not isinstance(ddict_2, dict):
raise TypeError(f"ddict_1 and ddict_2 are expected to be instances of {dict}")
for key, value in ddict_2.items():
if isinstance(value, Mapping):
ddict_1[key] = update_metadata(ddict_1.get(key, {}), value)
else:
ddict_1[key] = value
return ddict_1
def rescale_data(data, new_min, new_max, data_min=None, data_max=None):
if data_min is None:
data_min = numpy.min(data)
if data_max is None:
data_max = numpy.max(data)
return (new_max - new_min) / (data_max - data_min) * (data - data_min) + new_min
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/volumes/ 0000775 0000000 0000000 00000000000 15241025157 0024362 5 ustar 00root root 0000000 0000000 volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/volumes/__init__.py 0000664 0000000 0000000 00000001014 15241025157 0026467 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""volumes: classes to load and save tomography reconstruction volumes"""
from .edfvolume import EDFVolume # noqa: F401
from .hdf5volume import HDF5Volume # noqa: F401
from .jp2kvolume import JP2KVolume, has_glymur # noqa: F401
from .rawvolume import RawVolume # noqa: F401
from .singleframebase import TIFFLikeDiskAccessor, VolumeSingleFrameBase # noqa: F401
from .tiffvolume import MultiTIFFVolume, TIFFVolume, has_tifffile # noqa: F401
from .volumebase import SliceTuple, VolumeBase # noqa: F401
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/volumes/edfvolume.py 0000664 0000000 0000000 00000016465 15241025157 0026736 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""module defining utils for an edf volume"""
from __future__ import annotations
import logging
import mmap
import os
import fabio
import fabio.edfimage
import numpy
from silx.io.url import DataUrl
from tomobase.tomoobject import TomoObject
from tomobase.utils import docstring
from ..identifiers.edfidentifier import EDFVolumeIdentifier
from .singleframebase import TIFFLikeDiskAccessor, VolumeSingleFrameBase
_logger = logging.getLogger(__name__)
__all__ = [
"EDFVolume",
]
class EDFVolume(TIFFLikeDiskAccessor, VolumeSingleFrameBase):
"""
Save volume data to single frame edf and metadata to .txt files
:warning: each file saved under {volume_basename}_{index_zfill6}.edf is considered to be a slice of the volume.
Example:
.. code-block:: python
import numpy
from volio import EDFVolume
data = numpy.random.random((10, 100, 100))
metadata = {"processing_options": {"reconstruction": {"voxel_size_cm": 0.1}}}
volume = EDFVolume(folder="edf_volume", data=data, metadata=metadata)
volume.save()
volume_loaded = EDFVolume(folder="edf_volume")
volume_loaded.load()
print(volume_loaded.data.shape)
print(volume_loaded.metadata)
"""
DEFAULT_DATA_SCHEME = "fabio"
DEFAULT_DATA_EXTENSION = "edf"
def __init__(
self,
folder: str | None = None,
volume_basename: str | None = None,
data: numpy.ndarray | None = None,
source_scan: TomoObject | None = None,
metadata: dict | None = None,
data_url: DataUrl | None = None,
metadata_url: DataUrl | None = None,
overwrite: bool = False,
header: dict | None = None,
start_index=0,
data_extension=DEFAULT_DATA_EXTENSION,
metadata_extension=VolumeSingleFrameBase.DEFAULT_METADATA_EXTENSION,
) -> None:
if folder is not None:
url = DataUrl(
file_path=str(folder),
data_path=None,
)
else:
url = None
TIFFLikeDiskAccessor.__init__(self)
super().__init__(
volume_basename=volume_basename,
url=url,
data=data,
source_scan=source_scan,
metadata=metadata,
data_url=data_url,
metadata_url=metadata_url,
overwrite=overwrite,
start_index=start_index,
data_extension=data_extension,
metadata_extension=metadata_extension,
)
self._header = header
@property
def header(self) -> dict | None:
"""possible header for the edf files"""
return self._header
@docstring(VolumeSingleFrameBase)
def save_frame(self, frame, file_name, scheme):
if scheme == "fabio":
header = self.header or {}
edf_writer = fabio.edfimage.EdfImage(
data=frame,
header=header,
)
parent_dir = os.path.dirname(file_name)
if parent_dir != "":
os.makedirs(parent_dir, exist_ok=True)
edf_writer.write(file_name)
else:
raise ValueError(f"scheme {scheme} is not handled")
@docstring(VolumeSingleFrameBase)
def load_frame(self, file_name, scheme):
if scheme == "fabio":
return fabio.open(file_name).data
else:
raise ValueError(f"scheme {scheme} is not handled")
@staticmethod
@docstring(VolumeSingleFrameBase)
def from_identifier(identifier):
"""Return the Dataset from a identifier"""
if not isinstance(identifier, EDFVolumeIdentifier):
raise TypeError(
f"identifier should be an instance of {EDFVolumeIdentifier}"
)
return EDFVolume(
folder=identifier.folder,
volume_basename=identifier.file_prefix,
)
@docstring(VolumeSingleFrameBase)
def get_identifier(self) -> EDFVolumeIdentifier:
if self.url is None:
raise ValueError("no file_path provided. Cannot provide an identifier")
return EDFVolumeIdentifier(
object=self, folder=self.url.file_path(), file_prefix=self._volume_basename
)
@staticmethod
def example_defined_from_str_identifier() -> str:
return " ; ".join(
[
f"{EDFVolume(folder='/path/to/my/my_folder').get_identifier().to_str()}",
f"{EDFVolume(folder='/path/to/my/my_folder', volume_basename='mybasename').get_identifier().to_str()} (if mybasename != folder name)",
]
)
# implementation of TIFFLikeAccessor
def read_n_lines_in_file(self, file_name, line_indices) -> tuple:
reader = _EDFImageWithReadMMap()
reader.read(file_name)
volume_shape = self.get_volume_shape()
result = []
for line in line_indices:
data = reader.read_mmap(
file_name,
coords=(
slice(0, volume_shape[1]),
slice(line, line + 1),
),
).ravel()
if data is None:
# an error might have occur ('fast_read_roi' only accept un-compressed). In this case try the 'good old' .data directly which is safer
data = reader.data[:, line]
result.append(data)
return tuple(result)
def read_n_columns_in_file(self, file_name, column_indices) -> tuple:
image = fabio.open(file_name)
volume_shape = self.get_volume_shape()
result = []
for column in column_indices:
data = image.fast_read_roi(
file_name,
coords=[
slice(column, column + 1),
slice(0, volume_shape[2]),
],
).ravel()
if data is None:
# an error might have occur ('fast_read_roi' only accept un-compressed). In this case try the 'good old' .data directly which is safer
data = image.data[column,]
result.append(data)
return tuple(result)
def read_file(self, file_name) -> tuple:
return fabio.open(file_name).data
class _EDFImageWithReadMMap(fabio.edfimage.EdfImage):
def read_mmap(self, fname: str, coords: tuple) -> numpy.ndarray:
if len(coords) == 4:
slice1 = self.make_slice(coords) # and fix that FIXME ...
elif (
len(coords) == 2
and isinstance(coords[0], slice)
and isinstance(coords[1], slice)
):
slice1 = coords
else:
_logger.warning(
"readROI: Unable to understand Region Of Interest: got %s", coords
)
return
with open(fname, "rb") as fin:
fin.seek(0)
count = self.shape[0] * self.shape[1]
offset = self._frames[self.currentframe].start
nbytes = self.bpp * count
with mmap.mmap(
fin.fileno(), offset + nbytes, access=mmap.ACCESS_READ
) as mem:
buf = numpy.frombuffer(
mem, offset=offset, count=count, dtype=self.bytecode
)
buf = buf.reshape(self.shape)
ret = buf[slice1].copy()
del buf
if self.swap_needed():
ret.byteswap(True)
return ret
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/volumes/hdf5volume.py 0000664 0000000 0000000 00000050736 15241025157 0027025 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""module defining utils for an hdf5 volume"""
from __future__ import annotations
import logging
import os
import h5py
import numpy
from silx.io.dictdump import dicttonx, nxtodict
from silx.io.url import DataUrl
from silx.io.utils import open as open_hdf5
from tomobase.tomoobject import TomoObject
from tomobase.utils import docstring
from tomobase.utils.io import deprecated_warning
from ..identifiers.hdf5identifier import HDF5VolumeIdentifier
from .volumebase import SliceTuple, VolumeBase
_logger = logging.getLogger(__name__)
__all__ = ["HDF5Volume", "get_default_data_path_for_volume"]
class HDF5Volume(VolumeBase):
"""
Volume where both data and metadata are store in a HDF5 file but at a different location.
Example:
.. code-block:: python
import numpy
from volio import HDF5Volume
data = numpy.random.random((10, 100, 100))
metadata = {"processing_options": {"reconstruction": {"voxel_size_cm": 0.1}}}
volume = HDF5Volume(
file_path="reconstruction.hdf5",
data_path="entry0000/reconstruction",
data=data,
metadata=metadata,
)
volume.save()
volume_loaded = HDF5Volume(
file_path="reconstruction.hdf5",
data_path="entry0000/reconstruction",
)
volume_loaded.load()
print(volume_loaded.data.shape)
print(volume_loaded.metadata)
"""
DATA_DATASET_NAME = "results/data"
METADATA_GROUP_NAME = "configuration"
def __init__(
self,
file_path: str | None = None,
data_path: str | None = None,
data: numpy.ndarray | None = None,
source_scan: TomoObject | None = None,
metadata: dict | None = None,
data_url: DataUrl | None = None,
metadata_url: DataUrl | None = None,
overwrite: bool = False,
) -> None:
url = self._get_url_from_file_path_data_path(
file_path=file_path, data_path=data_path
)
self._file_path = file_path
self._data_path = data_path
super().__init__(
url=url,
data=data,
source_scan=source_scan,
metadata=metadata,
data_url=data_url,
metadata_url=metadata_url,
overwrite=overwrite,
)
@property
def data_extension(self) -> None | str:
if self.data_url is not None and self.data_url.file_path() is not None:
_, extension = os.path.splitext(self.data_url.file_path())
if extension:
return extension[1:]
@property
def metadata_extension(self) -> None | str:
if self.metadata_url is not None and self.metadata_url.file_path() is not None:
_, extension = os.path.splitext(self.metadata_url.file_path())
if extension:
return extension[1:]
@staticmethod
def _get_url_from_file_path_data_path(
file_path: str | None, data_path: str | None
) -> DataUrl | None:
if file_path is not None and data_path is not None:
return DataUrl(file_path=file_path, data_path=data_path, scheme="silx")
else:
return None
@VolumeBase.data.setter
def data(self, data):
if not isinstance(data, (numpy.ndarray, type(None), h5py.VirtualLayout)):
raise TypeError(
f"data is expected to be None or a numpy array not {type(data)}"
)
if isinstance(data, numpy.ndarray) and data.ndim != 3:
raise ValueError(f"data is expected to be 3D and not {data.ndim}D.")
self._data = data
@property
def file_path(self):
return self._file_path
@file_path.setter
def file_path(self, file_path: str | None):
if not (file_path is None or isinstance(file_path, str)):
raise TypeError
self._file_path = file_path
self.url = self._get_url_from_file_path_data_path(
self.file_path, self.data_path
)
@property
def data_path(self):
return self._data_path
@data_path.setter
def data_path(self, data_path: str | None):
if not (data_path is None or isinstance(data_path, str)):
raise TypeError
self._data_path = data_path
self.url = self._get_url_from_file_path_data_path(
self.file_path, self.data_path
)
@docstring(VolumeBase)
def deduce_data_and_metadata_urls(self, url: DataUrl | None) -> tuple:
if url is None:
return None, None
else:
if url.data_slice() is not None:
raise ValueError(f"data_slice is not handled by the {HDF5Volume}")
file_path = url.file_path()
data_path = url.data_path()
if data_path is None:
raise ValueError(
"data_path not provided from the DataUrl. Please provide one."
)
scheme = url.scheme() or "silx"
return (
# data url
DataUrl(
file_path=file_path,
data_path="/".join([data_path, self.DATA_DATASET_NAME]),
scheme=scheme,
),
# medata url
DataUrl(
file_path=file_path,
data_path="/".join([data_path, self.METADATA_GROUP_NAME]),
scheme=scheme,
),
)
@docstring(VolumeBase)
def save_data(self, mode="a", **kwargs) -> None:
"""
:raises KeyError: if data path already exists and overwrite set to False
:raises ValueError: if data is None
"""
# to be discussed. Not sure we should raise an error in this case. Could be usefull but this could also be double edged knife
if self.data is None:
raise ValueError("No data to be saved")
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to save this."
)
else:
_logger.info(f"save data to {url.path()}")
if url.file_path() is not None and os.path.dirname(url.file_path()) != "":
os.makedirs(os.path.dirname(url.file_path()), exist_ok=True)
with h5py.File(url.file_path(), mode=mode) as h5s:
if url.data_path() in h5s:
if self.overwrite:
_logger.debug(
f"overwrite requested. Will remove {url.data_path()} entry"
)
del h5s[url.data_path()]
else:
raise OSError(
f"Unable to save data to {url.data_path()}. This path already exists in {url.file_path()}. If you want you can ask to overwrite it."
)
if isinstance(self.data, h5py.VirtualLayout):
h5s.create_virtual_dataset(name=url.data_path(), layout=self.data)
else:
h5s.create_dataset(url.data_path(), data=self.data, **kwargs)
@docstring(VolumeBase)
def data_file_saver_generator(
self,
n_frames,
overwrite: bool = False,
mode: str = "a",
**kwargs,
):
"""
warning: the file will be open until the generator exists
"""
class _FrameDumper:
"""
will not work for VirtualLayout
"""
Dataset = None
# shared dataset
def __init__(
self,
root_group,
data_path,
create_dataset,
n_frames,
i_frame,
overwrite,
mode,
) -> None:
self.data_path = data_path
self.root_group = root_group
self.create_dataset = create_dataset
self.n_frames = n_frames
self.mode = mode
self.overwrite = overwrite
self.i_frame = i_frame
self.__kwargs = kwargs # keep chunk arguments for example
def __setitem__(self, key, value):
frame = value
if _FrameDumper.Dataset is None:
if self.data_path in self.root_group:
if self.overwrite:
_logger.debug(
f"overwrite requested. Will remove {data_url.data_path()} entry"
)
del h5s[data_url.data_path()]
else:
raise OSError(
f"Unable to save data to {data_url.data_path()}. This path already exists in {data_url.file_path()}. If you want you can ask to overwrite it."
)
_FrameDumper.Dataset = h5s.create_dataset( # pylint: disable=E1137
name=data_url.data_path(),
shape=(n_frames, frame.shape[0], frame.shape[1]),
dtype=frame.dtype,
**self.__kwargs,
)
if key != slice(None, None, None):
raise ValueError("item setting only handle ':' for now")
_FrameDumper.Dataset[i_frame] = frame # pylint: disable=E1137
data_url = self.data_url
if (
self.data_url.file_path() is not None
and os.path.dirname(self.data_url.file_path()) != ""
):
os.makedirs(os.path.dirname(self.data_url.file_path()), exist_ok=True)
with h5py.File(self.data_url.file_path(), mode=mode) as h5s:
for i_frame in range(n_frames):
yield _FrameDumper(
create_dataset=i_frame == 0,
data_path=self.data_url.data_path(),
root_group=h5s,
n_frames=n_frames,
i_frame=i_frame,
overwrite=overwrite,
mode=mode,
)
@docstring(VolumeBase)
def save_metadata(self) -> None:
"""
:raises KeyError: if data path already exists and overwrite set to False
:raises ValueError: if data is None
"""
if self.metadata is None:
raise ValueError("No metadata to be saved")
url = self.metadata_url
if url is None:
raise ValueError(
"Cannot get metadata_url. An url should be provided. Don't know where to save this."
)
else:
_logger.info(f"save metadata to {url.path()}")
if url.file_path() is not None and os.path.dirname(url.file_path()) != "":
os.makedirs(os.path.dirname(url.file_path()), exist_ok=True)
dicttonx(
self.metadata,
h5file=url.file_path(),
h5path=url.data_path(),
update_mode="replace",
mode="a",
)
@docstring(VolumeBase)
def load_data(self, store: bool = True) -> numpy.ndarray:
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to save this."
)
with open_hdf5(filename=url.file_path()) as h5s:
if url.data_path() in h5s:
data = h5s[url.data_path()][()]
else:
raise KeyError(f"Data path {url.data_path()} not found.")
if store:
self.data = data
return data
def get_slice(
self,
index: int | str = None,
axis=None,
xy=None,
xz=None,
yz=None,
):
if xy is yz is xz is None and (index is None or axis is None):
raise ValueError("index and axis should be provided")
if xy is not None:
deprecated_warning(
type_="parameter",
name="xy",
replacement="axis and index",
)
if axis is None and index is None:
axis = 0
index = xy
else:
raise ValueError("several axis (previously xy, xz, yz requested")
elif xz is not None:
deprecated_warning(
type_="parameter",
name="xz",
replacement="axis and index",
)
if axis is None and index is None:
axis = 1
index = xz
else:
raise ValueError("several axis (previously xy, xz, yz requested")
elif yz is not None:
deprecated_warning(
type_="parameter",
name="yz",
replacement="axis and index",
)
if axis is None and index is None:
axis = 2
index = yz
else:
raise ValueError("several axis (previously xy, xz, yz requested")
if isinstance(index, str):
if index == "first":
index = 0
elif index == "middle":
index = self.get_volume_shape()[axis] // 2
elif index == "last":
index = -1
else:
raise ValueError(f"index '{index}' is not handled")
if self.data is not None:
return self.select(volume=self.data, axis=axis, index=index)
else:
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to save this."
)
with open_hdf5(filename=url.file_path()) as h5s:
if url.data_path() in h5s:
return self.select(
volume=h5s[url.data_path()], axis=axis, index=index
)
else:
raise KeyError(f"Data path {url.data_path()} not found.")
def _get_slices_from_disk(
self, slices: tuple[SliceTuple]
) -> dict[SliceTuple, numpy.ndarray]:
"""
read from files a couple of slices along any axis.
"""
url = self.data_url
with open_hdf5(filename=url.file_path()) as h5s:
if url.data_path() in h5s:
return self.select_slices(volume=h5s[url.data_path()], slices=slices)
@docstring(VolumeBase)
def load_metadata(self, store: bool = True) -> dict:
url = self.metadata_url
if url is None:
raise ValueError(
"Cannot get metadata_url. An url should be provided. Don't know where to save this."
)
try:
metadata = nxtodict(
h5file=url.file_path(), path=url.data_path(), asarray=False
)
except KeyError:
_logger.warning(f"no metadata found in {url.data_path()}")
metadata = {}
if store:
self.metadata = metadata
return metadata
def browse_metadata_files(self):
"""
return a generator go through all the existing files associated to the data volume
"""
url = self.metadata_url
if url is None:
return
elif url.file_path() is not None and os.path.exists(url.file_path()):
yield url.file_path()
def browse_data_files(self):
"""
return a generator go through all the existing files associated to the data volume
"""
url = self.data_url
if url is None:
return
elif url.file_path() is not None and os.path.exists(url.file_path()):
yield url.file_path()
def browse_data_urls(self):
url = self.data_url
if url is not None and os.path.exists(url.file_path()):
yield url
@staticmethod
@docstring(VolumeBase)
def from_identifier(identifier):
"""Return the Dataset from a identifier"""
if not isinstance(identifier, HDF5VolumeIdentifier):
raise TypeError(
f"identifier should be an instance of {HDF5VolumeIdentifier}"
)
return HDF5Volume(
file_path=identifier.file_path,
data_path=identifier.data_path,
)
@docstring(VolumeBase)
def get_identifier(self) -> HDF5VolumeIdentifier:
if self.url is None:
raise ValueError("no file_path provided. Cannot provide an identifier")
return HDF5VolumeIdentifier(
object=self, hdf5_file=self.url.file_path(), entry=self.url.data_path()
)
@staticmethod
def example_defined_from_str_identifier() -> str:
return (
HDF5Volume(file_path="/path/to/file_path", data_path="entry0000")
.get_identifier()
.to_str()
)
@docstring(VolumeBase)
def browse_slices(self):
# load slices from volume is in memory
if self.data is not None:
for data_slice in self.data:
yield data_slice
return
# Else load slices from disk
url = self.data_url
if url is None:
raise ValueError(
"No data and data_url know and no url provided. Uanble to browse slices"
)
with open_hdf5(filename=url.file_path()) as h5s:
if url.data_path() in h5s:
for data_slice in h5s[url.data_path()]:
yield data_slice
else:
raise KeyError(f"Data path {url.data_path()} not found.")
@docstring(VolumeBase)
def load_chunk(self, chunk):
url = self.data_url
if url is None:
raise ValueError("Cannot get data_url. An url should be provided.")
with open_hdf5(filename=url.file_path()) as h5s:
if url.data_path() in h5s:
return h5s[url.data_path()][chunk]
else:
raise KeyError(f"Data path {url.data_path()} not found.")
def get_volume_shape(self):
if self.data is not None:
return self.data.shape
url = self.data_url
if url is None:
raise ValueError("Cannot get data_url. An url should be provided.")
else:
with open_hdf5(filename=url.file_path()) as h5s:
if url.data_path() in h5s:
return h5s[url.data_path()].shape
else:
return None
@docstring(VolumeBase)
def get_volume_dtype(self):
if self.data is not None:
return self.data.dtype
url = self.data_url
if url is None:
raise ValueError("Cannot get data_url. An url should be provided.")
else:
with open_hdf5(filename=url.file_path()) as h5s:
if url.data_path() in h5s:
return h5s[url.data_path()].dtype
else:
return None
def get_histogram_url(self, scan: TomoObject | None = None) -> DataUrl | None:
if self.data_url is None:
return None
histogram_file = self.data_url.file_path()
if not os.path.exists(histogram_file):
return None
# the histogram is expected to be stored as a sibling NXprocess group ('histogram') of the
# ('reconstruction') group the volume data belongs to, in the same NXentry
if self.url is not None:
data_path = self.url.data_path()
if data_path.endswith("reconstruction"):
data_path = "/".join(
[*data_path.split("/")[:-1], "histogram/results/data"]
)
else:
data_path = f"{data_path}/histogram/results/data"
else:
data_path = self.data_url.data_path().replace(
self.DATA_DATASET_NAME, "histogram/results/data"
)
with h5py.File(histogram_file, mode="r") as h5f:
if data_path not in h5f:
_logger.info(f"{data_path} in {histogram_file} not found")
return None
return DataUrl(file_path=histogram_file, data_path=data_path, scheme="silx")
def get_default_data_path_for_volume(scan) -> str:
"""
:param scan: object expected to expose an 'entry' attribute (e.g. a tomoscan NXtomoScan).
Note: this check is duck-typed (``hasattr(scan, "entry")``) instead of an
``isinstance`` check against a scan base class, to avoid a hard dependency on
``tomoscan`` from ``volio``.
"""
if not hasattr(scan, "entry"):
raise TypeError(
f"scan is expected to expose an 'entry' attribute (e.g. a NXtomoScan) not {type(scan)}"
)
entry = getattr(scan, "entry", "entry")
return "/".join([entry, "reconstruction"])
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/volumes/jp2kvolume.py 0000664 0000000 0000000 00000025222 15241025157 0027035 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""module defining utils for a jp2k volume"""
from __future__ import annotations
import logging
import os
import numpy
from packaging.version import parse as parse_version
from silx.io.url import DataUrl
from tomobase.tomoobject import TomoObject
from tomobase.utils import docstring
from ..identifiers.jp2kidentifier import JP2KVolumeIdentifier
from .singleframebase import VolumeSingleFrameBase
try:
import glymur # noqa #F401 needed for later possible lazy loading
except ImportError:
has_glymur = False
has_minimal_openjpeg = False
glymur_version = None
openjpeg_version = None
else:
has_glymur = True
from glymur import set_option as glymur_set_option
from glymur.version import openjpeg_version
from glymur.version import version as glymur_version
if openjpeg_version < "2.3.0":
has_minimal_openjpeg = False
else:
has_minimal_openjpeg = True
_logger = logging.getLogger(__name__)
_MISSING_GLYMUR_MSG = "Fail to import glymur. won't be able to load / save volume to jp2k. You can install it by calling pip."
__all__ = [
"JP2KVolume",
"has_glymur",
"has_minimal_openjpeg",
"glymur_version",
"openjpeg_version",
"get_available_threads",
]
class JP2KVolume(VolumeSingleFrameBase):
"""
Save volume data to single frame jp2k files and metadata to .txt file
:param cratios: list of ints. compression ratio for each jpeg2000 layer
:param psnr: list of int. The PSNR (Peak Signal-to-Noise ratio) for each jpeg2000 layer.
This defines a quality metric for lossy compression.
The number "0" stands for lossless compression.
:param n_threads: number of thread to use for writing. If None will try to get as much as possible
:param clip_values: optional tuple of two float (min, max) to clamp volume value
:param rescale_data: rescale data before dumping each frame. Expected to be True when dump a new volume and False
when save volume cast for example (and when histogram is know...)
:warning: each file saved under {volume_basename}_{index_zfill6}.jp2 is considered to be a slice of the volume.
Example:
.. code-block:: python
import numpy
from volio import JP2KVolume
data = numpy.random.random((10, 100, 100))
metadata = {"processing_options": {"reconstruction": {"voxel_size_cm": 0.1}}}
volume = JP2KVolume(folder="jp2k_volume", data=data, metadata=metadata)
volume.save()
volume_loaded = JP2KVolume(folder="jp2k_volume")
volume_loaded.load()
print(volume_loaded.data.shape)
print(volume_loaded.metadata)
"""
DEFAULT_DATA_EXTENSION = "jp2"
DEFAULT_DATA_SCHEME = "glymur"
def __init__(
self,
folder: str | None = None,
volume_basename: str | None = None,
data: numpy.ndarray | None = None,
source_scan: TomoObject | None = None,
metadata: dict | None = None,
data_url: DataUrl | None = None,
metadata_url: DataUrl | None = None,
overwrite: bool = False,
start_index=0,
data_extension=DEFAULT_DATA_EXTENSION,
metadata_extension=VolumeSingleFrameBase.DEFAULT_METADATA_EXTENSION,
cratios: list | None = None,
psnr: list | None = None,
n_threads: int | None = None,
clip_values: tuple | None = None,
rescale_data: bool = True,
) -> None:
if folder is not None:
url = DataUrl(
file_path=str(folder),
data_path=None,
)
else:
url = None
super().__init__(
url=url,
data=data,
volume_basename=volume_basename,
source_scan=source_scan,
metadata=metadata,
data_url=data_url,
metadata_url=metadata_url,
overwrite=overwrite,
start_index=start_index,
data_extension=data_extension,
metadata_extension=metadata_extension,
)
if not has_glymur:
_logger.warning(_MISSING_GLYMUR_MSG)
else:
if not has_minimal_openjpeg:
_logger.warning(
"You must have at least version 2.3.0 of OpenJPEG "
"in order to write jp2k images."
)
self._cratios = cratios
self._psnr = psnr
self._clip_values = None
self._cast_already_logged = False
"""bool used to avoid logging potential data cast for each frame when save a volume"""
self._rescale_data = rescale_data
"""should we rescale data before dumping the frame"""
self.setup_multithread_encoding(n_threads=n_threads)
self.clip_values = clip_values # execute test about the type...
@property
def cratios(self) -> list | None:
return self._cratios
@cratios.setter
def cratios(self, cratios: list | None):
self._cratios = cratios
@property
def psnr(self) -> list | None:
return self._psnr
@psnr.setter
def psnr(self, psnr: list | None):
self._psnr = psnr
@property
def rescale_data(self) -> bool:
return self._rescale_data
@rescale_data.setter
def rescale_data(self, rescale: bool) -> None:
if not isinstance(rescale, bool):
raise TypeError
self._rescale_data = rescale
@property
def clip_values(self) -> tuple | None:
"""
:return: optional min and max value to clip - as float.
"""
return self._clip_values
@clip_values.setter
def clip_values(self, values: tuple | None) -> None:
if values is None:
self._clip_values = None
elif not isinstance(values, (tuple, list)):
raise TypeError
elif not len(values) == 2:
raise ValueError("clip values are expected to be two floats")
elif not values[1] >= values[0]:
raise ValueError
else:
self._clip_values = values
@docstring(VolumeSingleFrameBase)
def save_data(self) -> None:
self._cast_already_logged = False
super().save_data()
@docstring(VolumeSingleFrameBase)
def save_frame(self, frame, file_name, scheme):
if not has_glymur:
raise RuntimeError(_MISSING_GLYMUR_MSG)
expected_output_dtype = frame.dtype
if self.clip_values is not None:
frame = numpy.clip(frame, self.clip_values[0], self.clip_values[1])
if expected_output_dtype not in (numpy.uint8, numpy.uint16):
_logger.info(
f"{self.get_identifier().to_str()} get {frame.dtype}. Cast it as {numpy.uint16}"
)
expected_output_dtype = numpy.uint16
if self.rescale_data:
from volio.utils import rescale_data # avoid cyclic import
max_uint = numpy.iinfo(expected_output_dtype).max
frame = rescale_data(
data=frame,
new_min=0,
new_max=max_uint,
data_min=self.clip_values[0] if self.clip_values is not None else None,
data_max=self.clip_values[1] if self.clip_values is not None else None,
)
frame = frame.astype(expected_output_dtype)
if scheme == "glymur":
glymur.Jp2k(file_name, data=frame, psnr=self.psnr, cratios=self.cratios)
else:
raise ValueError(f"Scheme {scheme} is not handled")
@docstring(VolumeSingleFrameBase)
def load_frame(self, file_name, scheme):
if not has_glymur:
raise RuntimeError(_MISSING_GLYMUR_MSG)
if scheme == "glymur":
jp2_file = glymur.Jp2k(file_name)
return jp2_file[:]
else:
raise ValueError(f"Scheme {scheme} is not handled")
def read_file(self, file_name) -> tuple:
jp2_file = glymur.Jp2k(file_name)
return jp2_file[:]
@staticmethod
def setup_multithread_encoding(
n_threads: int | None = None, what_if_not_available: str = "ignore"
):
"""
Setup OpenJpeg multi-threaded encoding.
:param n_threads: Number of threads. If not provided, all available threads are used.
:param what_if_not_available: What to do if requirements are not fulfilled. Possible values are:
- "ignore": do nothing, proceed
- "print": show an information message
- "raise": raise an error
"""
required_glymur_version = "0.9.3"
required_openjpeg_version = "2.4.0"
def not_available(msg):
if what_if_not_available == "raise":
raise ValueError(msg)
elif what_if_not_available == "print":
print(msg)
if not has_glymur:
not_available(f"glymur not installed. {required_glymur_version} required")
return
elif parse_version(glymur_version) < parse_version(required_glymur_version):
not_available(
f"glymur >= {required_glymur_version} is required for multi-threaded encoding (current version: {glymur_version})"
)
return
elif not has_minimal_openjpeg:
not_available(
f"libopenjpeg >= {required_openjpeg_version} is required for multi-threaded encoding (current version: {openjpeg_version})"
)
return
if n_threads is None:
n_threads = get_available_threads()
glymur_set_option("lib.num_threads", n_threads)
@staticmethod
@docstring(VolumeSingleFrameBase)
def from_identifier(identifier):
"""Return the Dataset from a identifier"""
if not isinstance(identifier, JP2KVolumeIdentifier):
raise TypeError(
f"identifier should be an instance of {JP2KVolumeIdentifier}"
)
return JP2KVolume(
folder=identifier.folder,
volume_basename=identifier.file_prefix,
)
@docstring(VolumeSingleFrameBase)
def get_identifier(self) -> JP2KVolumeIdentifier:
if self.url is None:
raise ValueError("no file_path provided. Cannot provide an identifier")
return JP2KVolumeIdentifier(
object=self, folder=self.url.file_path(), file_prefix=self._volume_basename
)
@staticmethod
def example_defined_from_str_identifier() -> str:
return " ; ".join(
[
f"{JP2KVolume(folder='/path/to/my/my_folder').get_identifier().to_str()}",
f"{JP2KVolume(folder='/path/to/my/my_folder', volume_basename='mybasename').get_identifier().to_str()} (if mybasename != folder name)",
]
)
def get_available_threads():
return len(os.sched_getaffinity(0))
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/volumes/rawvolume.py 0000664 0000000 0000000 00000046461 15241025157 0026770 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""module defining utils for an .vol volume (also known as raw)"""
from __future__ import annotations
import logging
import os
import sys
import h5py
import numpy
from defusedxml.minidom import parseString as parse_xml_string
from dicttoxml import dicttoxml
from silx.io.dictdump import dicttoini
from silx.io.url import DataUrl
from tomobase.tomoobject import TomoObject
from tomobase.utils import docstring
from tomobase.utils.io import cast_to_default_types
from ..identifiers.rawidentifier import RawVolumeIdentifier
from .volumebase import VolumeBase
_logger = logging.getLogger(__name__)
__all__ = ["RawVolume"]
class RawVolume(VolumeBase):
"""
Volume where data si saved under .vol binary file and metadata are saved in .vol.info and or .vol.xml
Note: for now reading information from the .xml is not managed. We expect to write one or both and read from the text file (.vol.info)
Warning: meant as legacy for pyhst .vol file and existing post processing tool. We mostly expect software to write .vol file.
Example:
.. code-block:: python
import numpy
from volio import RawVolume
# the .vol format only ever stores float32 data, and has no shape of its
# own: NUM_X, NUM_Y and NUM_Z must be provided in the metadata to be able
# to reload it afterwards.
data = numpy.random.random((10, 100, 100)).astype(numpy.float32)
metadata = {
"NUM_Z": data.shape[0],
"NUM_Y": data.shape[1],
"NUM_X": data.shape[2],
}
volume = RawVolume(file_path="volume.vol", data=data, metadata=metadata)
volume.save()
volume_loaded = RawVolume(file_path="volume.vol")
volume_loaded.load()
print(volume_loaded.data.shape)
print(volume_loaded.metadata)
"""
DEFAULT_DATA_SCHEME = "raw"
DEFAULT_DATA_EXTENSION = "vol"
DEFAULT_METADATA_SCHEME = "info"
DEFAULT_METADATA_EXTENSION = "vol.info"
def __init__(
self,
file_path: str | None = None,
data: numpy.ndarray | None = None,
source_scan: TomoObject | None = None,
metadata: dict | None = None,
data_url: DataUrl | None = None,
metadata_url: DataUrl | None = None,
overwrite: bool = False,
append: bool = False,
data_extension=DEFAULT_DATA_EXTENSION,
metadata_extension=DEFAULT_METADATA_EXTENSION,
) -> None:
if file_path is not None:
url = DataUrl(file_path=file_path, data_path=None, scheme="raw")
else:
url = None
self._file_path = file_path
super().__init__(
url=url,
data=data,
source_scan=source_scan,
metadata=metadata,
data_url=data_url,
metadata_url=metadata_url,
overwrite=overwrite,
data_extension=data_extension,
metadata_extension=metadata_extension,
)
self.append = append
@property
def data_extension(self) -> str | None:
print(self.data_url.file_path())
if self.data_url is not None and self.data_url.file_path() is not None:
_, ext = os.path.splitext(self.data_url.file_path())
if ext:
return ext[1:]
@property
def metadata_extension(self):
if self.metadata_url is not None and self.metadata_url.file_path() is not None:
_, ext = os.path.splitext(self.metadata_url.file_path())
if ext:
return ext[1:]
@VolumeBase.data.setter
def data(self, data):
if not isinstance(data, (numpy.ndarray, type(None), h5py.VirtualLayout)):
raise TypeError(
f"data is expected to be None or a numpy array not {type(data)}"
)
if isinstance(data, numpy.ndarray) and data.ndim != 3:
raise ValueError(f"data is expected to be 3D and not {data.ndim}D.")
self._data = data
@property
def file_path(self):
return self._file_path
@file_path.setter
def file_path(self, file_path: str | None):
if not (file_path is None or isinstance(file_path, str)):
raise TypeError
self._file_path = file_path
self.url = DataUrl(file_path=file_path, data_path=None, scheme="raw")
@docstring(VolumeBase)
def deduce_data_and_metadata_urls(self, url: DataUrl | None) -> tuple:
if url is None:
return None, None
else:
if url.data_slice() is not None:
raise ValueError(f"data_slice is not handled by the {RawVolume}")
file_path = url.file_path()
data_path = url.data_path()
if data_path is not None:
raise ValueError("data_path is not handle by the .vol volume.")
scheme = url.scheme() or "raw"
metadata_info_file = os.path.splitext(url.file_path())[0] + ".vol.info"
return (
# data url
DataUrl(
file_path=file_path,
data_path=None,
scheme=scheme,
),
# medata url
DataUrl(
file_path=metadata_info_file,
data_path=None,
scheme=self.DEFAULT_METADATA_SCHEME,
),
)
@docstring(VolumeBase)
def save_data(self, **kwargs) -> None:
if self.data is None:
return
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to save this."
)
if url.scheme() != "raw":
raise ValueError("Unsupported scheme - please use scheme='raw'")
if url.data_path() is not None:
raise ValueError("No data path expected. Unagleto save data")
_logger.info(f"save data to {url.path()}")
if self.data.dtype != numpy.float32:
raise TypeError(".vol format only takes float32 as data type")
# check endianness: make sure data is lowbytefirst
if self.data.dtype.byteorder == ">" or (
self.data.dtype.byteorder == "=" and sys.byteorder != "little"
):
# lowbytefirst
raise TypeError("data is expected to be byteorder: low byte first")
if self.data.ndim == 3:
data = self.data
elif self.data.ndim == 2:
data = self.data.reshape(1, self.data.shape[0], self.data.shape[1])
else:
raise ValueError(f"data should be 3D and not {self.data.ndim}D")
file_mode = "ab" if self.append else "wb"
with open(url.file_path(), file_mode) as fdesc:
if self.append:
n_bytes = os.path.getsize(url.file_path())
fdesc.seek(n_bytes)
data.tofile(fdesc)
@docstring(VolumeBase)
def load_data(self, store: bool = True) -> numpy.array:
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to save this."
)
if self.metadata is None:
# for .vol file we need metadata to get shape - expected in a .vol.info file
metadata = self.load_metadata(store=False)
else:
metadata = self.metadata
dimX = metadata.get("NUM_X", None)
dimY = metadata.get("NUM_Y", None)
dimZ = metadata.get("NUM_Z", None)
byte_order = metadata.get("BYTEORDER", "LOWBYTEFIRST")
if byte_order.lower() == "highbytefirst":
byte_order = ">"
elif byte_order.lower() == "lowbytefirst":
byte_order = "<"
else:
raise ValueError(f"Unable to interpret byte order value: {byte_order}")
if dimX is None or dimY is None or dimZ is None:
_logger.error(f"Unable to get volume shape (get: {dimZ, dimY, dimZ} )")
data = None
else:
shape = (int(dimZ), int(dimY), int(dimX))
try:
data_type = numpy.dtype(byte_order + "f")
data = numpy.fromfile(
url.file_path(), dtype=data_type, count=-1, sep=""
)
except Exception as e:
_logger.warning(
f"Fail to load data from {url.file_path()}. Error is {e}."
)
data = None
else:
data = data.reshape(shape)
if store is True:
self.data = data
return data
@docstring(VolumeBase)
def save_metadata(self, store: bool = True) -> None:
"""
:raises KeyError: if data path already exists and overwrite set to False
:raises ValueError: if data is None
"""
if self.metadata is None:
raise ValueError("No metadata to be saved")
url = self.metadata_url
if url is None:
raise ValueError(
"Cannot get metadata_url. An url should be provided. Don't know where to save this."
)
_logger.info(f"save metadata to {url.path()}")
if url.scheme() == "info":
metadata_file = url.file_path()
_logger.info(f"save data to {metadata_file}")
if len(self.metadata) > 0:
# same as ini but no section. Write works but read fails
metadata = cast_to_default_types(self.metadata)
dicttoini(metadata, metadata_file)
elif url.scheme() == "lxml":
metadata_file = url.file_path()
_logger.info(f"save data to {metadata_file}")
if len(self.metadata) > 0:
# Format metadata to a XML file, with a format that can be read by imagej.
# Does not make sense to you ? For us neither!
size_xyz = [
int(self.metadata.get(key, 0))
for key in ["NUM_X", "NUM_Y", "NUM_Z"]
]
if size_xyz == 0:
_logger.error(
"Something wrong with NUM_X, NUM_Y or NUM_X: missing or zero ?"
)
metadata_for_xml = {
"reconstruction": {
"idAc": "N_A_",
"listSubVolume": {
"subVolume": {
"SUBVOLUME_NAME": os.path.basename(
self.data_url.file_path()
),
"SIZEX": size_xyz[0],
"SIZEY": size_xyz[1],
"SIZEZ": size_xyz[2],
"ORIGINX": 1,
"ORIGINY": 1,
"ORIGINZ": 1,
"DIM_REC": numpy.prod(size_xyz),
"BYTE_ORDER": "LOWBYTEFIRST", # !
}
},
}
}
for what in ["voxelSize", "ValMin", "ValMax", "s1", "s2", "S1", "S2"]:
metadata_for_xml["reconstruction"]["listSubVolume"]["subVolume"][
what
] = float(self.metadata.get(what, 0.0))
xml_str = dicttoxml(
metadata_for_xml,
custom_root="tomodb2",
xml_declaration=False,
attr_type=False,
return_bytes=False,
)
xml_str_pretty = parse_xml_string(xml_str).toprettyxml(indent=" ")
with open(metadata_file, mode="w") as file_:
file_.write(xml_str_pretty)
else:
raise ValueError(f"scheme {url.scheme()} is not handled")
@docstring(VolumeBase)
def load_metadata(self, store: bool = True) -> dict:
url = self.metadata_url
if url is None:
raise ValueError(
"Cannot get metadata_url. An url should be provided. Don't know where to save this."
)
if url.scheme() == "info":
def info_file_to_dict(info_file):
ddict = {}
with open(info_file, "r") as _file:
lines = _file.readlines()
for line in lines:
if "=" not in line:
# section headers (e.g. "[cast_volume]") are not handled by this
# flat key=value format, skip them
continue
_line = line.rstrip().replace(" ", "")
_line = _line.split("#")[0]
# only split on the first '=' so values containing '=' (e.g. urls
# like 'hdf5:volume:/path/file.hdf5?path=entry') are kept intact
key, value = _line.split("=", 1)
ddict[key] = value
return ddict
metadata_file = url.file_path()
if url.data_path() is not None:
raise ValueError("data_path is not handled by ini scheme")
else:
try:
metadata = info_file_to_dict(metadata_file)
except FileNotFoundError:
_logger.warning(f"unable to load metadata from {metadata_file}")
metadata = {}
else:
raise ValueError(f"scheme {url.scheme()} is not handled")
if store:
self.metadata = metadata
return metadata
def get_volume_shape(self):
metadata = self.metadata or self.load_metadata()
from_metadata = (
metadata.get("SIZEZ", metadata.get("NUM_Z")),
metadata.get("SIZEY", metadata.get("NUM_Y")),
metadata.get("SIZEX", metadata.get("NUM_X")),
)
if (
from_metadata[0] is not None
and from_metadata[1] is not None
and from_metadata[2] is not None
):
return tuple(int(dim) for dim in from_metadata)
else:
if self.data is None:
data = self.load_data()
else:
data = self.data
if data is not None:
return data.shape
else:
return None
@docstring(VolumeBase)
def get_volume_dtype(self):
if self.data is not None:
return self.data.dtype
# the .vol format only ever stores float32 (see 'save_data')
return numpy.dtype(numpy.float32)
def browse_metadata_files(self):
"""
return a generator go through all the existing files associated to the data volume
"""
url = self.metadata_url
if url is None:
return
elif url.file_path() is not None and os.path.exists(url.file_path()):
yield url.file_path()
def browse_data_files(self):
"""
return a generator go through all the existing files associated to the data volume
"""
url = self.data_url
if url is None:
return
elif url.file_path() is not None and os.path.exists(url.file_path()):
yield url.file_path()
def browse_slices(self):
"""
Browse volume slices without loading the full volume in memory:
the data file is memory-mapped and each frame is only paged in
from disk when it is actually accessed.
"""
# if data loaded
if self.data is not None:
for vol_slice in self.data:
yield vol_slice
return
# else access slice one by one from disk
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to load this."
)
shape = self.get_volume_shape()
if shape is None:
raise ValueError(f"Unable to get volume shape from {url.file_path()}.")
n_z, n_y, n_x = shape
metadata = self.metadata or self.load_metadata()
byte_order = metadata.get("BYTEORDER", "LOWBYTEFIRST")
if byte_order.lower() == "highbytefirst":
byte_order = ">"
elif byte_order.lower() == "lowbytefirst":
byte_order = "<"
else:
raise ValueError(f"Unable to interpret byte order value: {byte_order}")
data = numpy.memmap(
url.file_path(),
dtype=numpy.dtype(byte_order + "f4"),
mode="r",
shape=(n_z, n_y, n_x),
)
for vol_slice in data:
yield vol_slice
def browse_data_urls(self):
url = self.data_url
if url is not None and os.path.exists(url.file_path()):
yield url
@docstring(VolumeBase)
def get_histogram_url(self, scan=None) -> DataUrl | None:
if self.data_url is None or self.data_url.file_path() is None:
return None
file_path, _ = os.path.splitext(self.data_url.file_path())
histogram_file = file_path + "_histogram.hdf5"
return self._get_histogram_url_from_file(histogram_file, scan=scan)
@docstring(VolumeBase)
def data_file_saver_generator(
self,
n_frames,
overwrite: bool = False,
mode: str = "a",
**kwargs,
):
"""
warning: the file will be open until the generator exists
"""
class _FrameDumper:
"""
will not work for VirtualLayout
"""
Dataset = None
# shared dataset
def __init__(
self,
fid,
) -> None:
self._fid = fid
def __setitem__(self, key, value):
if key != slice(None, None, None):
raise ValueError("item setting only handle ':' for now")
if not isinstance(value, numpy.ndarray):
raise TypeError(
"value is expected to be an instance of numpy.ndarray"
)
value.tofile(self._fid)
data_url = self.data_url
if (
data_url.file_path() is not None
and os.path.dirname(data_url.file_path()) != ""
):
os.makedirs(os.path.dirname(data_url.file_path()), exist_ok=True)
with open(data_url.file_path(), "wb") as fid:
for _ in range(n_frames):
yield _FrameDumper(fid=fid)
@staticmethod
@docstring(VolumeBase)
def from_identifier(identifier):
"""Return the Dataset from a identifier"""
if not isinstance(identifier, RawVolumeIdentifier):
raise TypeError(
f"identifier should be an instance of {RawVolumeIdentifier}"
)
return RawVolume(
file_path=identifier.file_path,
)
@docstring(VolumeBase)
def get_identifier(self) -> RawVolumeIdentifier:
if self.url is None:
raise ValueError("no file_path provided. Cannot provide an identifier")
return RawVolumeIdentifier(object=self, file_path=self.url.file_path())
@staticmethod
def example_defined_from_str_identifier() -> str:
"""example as string to explain how users can defined identifiers from a string"""
return " ; ".join(
[
f"{RawVolume(file_path='/path/to/my/my_volume.vol').get_identifier().to_str()}",
]
)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/volumes/singleframebase.py 0000664 0000000 0000000 00000061667 15241025157 0030103 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""module defining the base class for volumes storing data as one file per slice (edf, jp2k, tiff)"""
from __future__ import annotations
import logging
import os
import re
from warnings import warn
import numpy
from silx.io.dictdump import dicttoini
from silx.io.dictdump import load as load_ini
from silx.io.url import DataUrl
from tomobase.tomoobject import TomoObject
from tomobase.utils import docstring
from tomobase.utils.io import (
cast_to_default_types,
catch_log_messages,
deprecated_warning,
)
from .volumebase import SliceTuple, VolumeBase
_logger = logging.getLogger(__name__)
__all__ = ["VolumeSingleFrameBase", "TIFFLikeDiskAccessor"]
class VolumeSingleFrameBase(VolumeBase):
"""
Base class for Volume where each slice is saved in a separate file like edf, jp2k or tiff.
:param volume_basename: prefix used to name the data and metadata files. If not provided it is deduced from the folder name.
:param start_index: users can provide a shift on the file name index when saving the data. This is interesting if you want to
create a volume from several writers.
"""
DEFAULT_DATA_SCHEME = None
DEFAULT_DATA_PATH_PATTERN = "{volume_basename}_{index_zfill6}.{data_extension}"
DEFAULT_METADATA_EXTENSION = "txt"
# information regarding metadata
DEFAULT_METADATA_SCHEME = "ini"
DEFAULT_METADATA_PATH_PATTERN = "{volume_basename}_infos.{metadata_extension}"
def __init__(
self,
url: DataUrl | None = None,
data: numpy.ndarray | None = None,
source_scan: TomoObject | None = None,
metadata: dict | None = None,
data_url: DataUrl | None = None,
metadata_url: DataUrl | None = None,
overwrite: bool = False,
start_index: int = 0,
volume_basename: str | None = None,
data_extension=None,
metadata_extension="txt",
) -> None:
self._volume_basename = volume_basename
super().__init__(
url,
data,
source_scan,
metadata,
data_url,
metadata_url,
overwrite,
data_extension,
metadata_extension,
)
self._start_index = start_index
self.write_in_descending_order = False
self._skip_existing_data_files_removal = False
def get_first_file(self):
try:
first_file = next(self.browse_data_files())
except StopIteration:
first_file = None
return first_file
def get_file_slice_index(self, filename):
if (
self.DEFAULT_DATA_PATH_PATTERN
!= VolumeSingleFrameBase.DEFAULT_DATA_PATH_PATTERN
):
raise ValueError("Non-default file name pattern not handled")
return int(filename.split(".")[0].split("_")[-1])
@property
def start_index(self) -> int:
return self._start_index
@start_index.setter
def start_index(self, start_idx: int) -> None:
self._start_index = start_idx
def get_volume_basename(self):
if self._volume_basename is not None:
return self._volume_basename
else:
return self._deduce_volume_basename(self.data_url)
@classmethod
def _deduce_volume_basename(cls, data_url: DataUrl) -> str:
return os.path.basename(data_url.file_path())
@property
def skip_existing_data_files_removal(self) -> bool:
"""The loading of the volume for single frame base is done by loading all the file
contained in a folder data_url.file_path(). When saving the data we make sure there is no 'remaining' of
any previous saving by using the file pattern.
But when we want to save a volume from several thread (one thread save the n first frame, second the n next frame ...) this
could be a limitation.
So in this case we can use the 'ignore_existing_files' that will avoid calling '_remove_existing_data_files'
"""
return self._skip_existing_data_files_removal
@skip_existing_data_files_removal.setter
def skip_existing_data_files_removal(self, ignore: bool) -> None:
if not isinstance(ignore, bool):
raise TypeError(f"ignore should be a bool. Got {type(ignore)}")
self._skip_existing_data_files_removal = ignore
@docstring(VolumeBase)
def deduce_data_and_metadata_urls(self, url: DataUrl | None) -> tuple:
"""
Deduce automatically data and metadata url.
Default data will be saved as single frame edf.
Default metadata will be saved as a text file
"""
if url is None:
return None, None
else:
metadata_keywords = {
"volume_basename": self._deduce_volume_basename(data_url=url),
"metadata_extension": self.metadata_extension,
}
metadata_data_path = self.DEFAULT_METADATA_PATH_PATTERN.format(
**metadata_keywords
)
return (
# data url
DataUrl(
file_path=url.file_path(),
data_path=self.DEFAULT_DATA_PATH_PATTERN,
scheme=url.scheme() or self.DEFAULT_DATA_SCHEME,
data_slice=url.data_slice(),
),
# medata url
DataUrl(
file_path=url.file_path(),
data_path=metadata_data_path,
scheme=url.scheme() or self.DEFAULT_METADATA_SCHEME,
),
)
@docstring(VolumeBase)
def load_metadata(self, store: bool = True) -> dict:
url = self.metadata_url
if url is None:
raise ValueError(
"Cannot get metadata_url. An url should be provided. Don't know where to save this."
)
if url.scheme() == "ini":
metadata_file = url.file_path()
if url.data_path() is not None:
metadata_file = os.path.join(metadata_file, url.data_path())
_logger.info(f"load data to {metadata_file}")
try:
metadata = load_ini(metadata_file, "ini")
except FileNotFoundError:
_logger.warning(
f"unable to load metadata from {metadata_file} - File not found"
)
metadata = {}
except Exception as e:
_logger.error(
f"Failed to load metadata from {metadata_file}. Error is {e}"
)
metadata = {}
else:
raise ValueError(f"scheme {url.scheme()} is not handled")
if store:
self.metadata = metadata
return metadata
@docstring(VolumeBase)
def save_metadata(self) -> None:
if self.metadata is None:
raise ValueError("No data to be saved")
url = self.metadata_url
if url is None:
raise ValueError(
"Cannot get metadata_url. An url should be provided. Don't know where to save this."
)
else:
if url.scheme() == "ini":
metadata_file = url.file_path()
if url.data_path() is not None:
metadata_file = os.path.join(metadata_file, url.data_path())
_logger.info(f"save data to {metadata_file}")
if len(self.metadata) > 0:
metadata = cast_to_default_types(self.metadata)
dicttoini(metadata, metadata_file)
else:
raise ValueError(f"scheme {url.scheme()} is not handled")
# utils to format file path
def format_data_path_for_data(
self, data_path: str, index: int, volume_basename: str
) -> str:
"""
Return file path to save the frame at `index` of the current volume
"""
keywords = {
"index_zfill4": str(index + self.start_index).zfill(4),
"index_zfill6": str(index + self.start_index).zfill(6),
"volume_basename": volume_basename,
"data_extension": self.data_extension,
}
return data_path.format(**keywords)
def get_data_path_pattern_for_data(
self, data_path: str, volume_basename: str
) -> str:
"""
Return file path **pattern** (and not full path) to load data.
For example in edf it can return 'myacquisition_*.edf' in order to be handled by
"""
keywords = {
"index_zfill4": "[0-9]{3,4}",
"index_zfill6": "[0-9]{3,6}",
"volume_basename": volume_basename,
"data_extension": self.data_extension,
}
return data_path.format(**keywords)
def remove_existing_data_files(self) -> None:
"""Clean any existing files (if overwrite and rights) that must be used for saving"""
url = self.data_url
if not os.path.exists(url.file_path()):
return
research_pattern = self.get_data_path_pattern_for_data(
data_path=url.data_path(),
volume_basename=self.get_volume_basename(),
)
try:
research_pattern = re.compile(research_pattern)
except Exception:
_logger.error(
f"Fail to compute regular expression for {research_pattern}. Unable to load data"
)
return None
for file_ in sorted(os.listdir(url.file_path())):
if research_pattern.match(file_):
full_file_path = os.path.join(url.file_path(), file_)
if self.overwrite:
os.remove(full_file_path)
else:
raise OSError(
f"Output file '{full_file_path}' already exists. No overwrite required. Skip writting"
)
@docstring(VolumeBase)
def save_data(self) -> None:
if self.data is None:
raise ValueError("No data to be saved")
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to save this."
)
else:
if not self.skip_existing_data_files_removal:
self.remove_existing_data_files()
_logger.info(f"save data to {url.path()}")
# if necessary create output directory (some third part writer does not do it for us)
try:
os.makedirs(url.file_path(), exist_ok=True)
except FileNotFoundError:
# can raise FileNotFoundError if file path is '.' for example
pass
assert self.data.ndim == 3
for frame, frame_dumper in zip(
self.data,
self.data_file_saver_generator(
n_frames=self.data.shape[0], overwrite=self.overwrite
),
):
frame_dumper[:] = frame
def data_file_name_generator(self, n_frames):
"""
browse output files for n_frames
"""
indices = range(n_frames)
if self.write_in_descending_order:
indices = list(range(0, -n_frames, -1))
for i_frame in indices:
file_name = self.format_data_path_for_data(
self.data_url.data_path(),
index=i_frame,
volume_basename=self.get_volume_basename(),
)
file_name = os.path.join(self.data_url.file_path(), file_name)
yield file_name
@docstring(VolumeBase)
def data_file_saver_generator(self, n_frames, overwrite: bool = False):
class _FrameDumper:
def __init__(self, url_scheme, file_name, callback) -> None:
self.url_scheme = url_scheme
self.file_name = file_name
self.overwrite = overwrite
self.__callback = callback
def __setitem__(self, key, value):
if not self.overwrite and os.path.exists(self.file_name):
raise OSError(
f"{self.file_name} already exists. If you want you can ask for the volume to overwriting existing files."
)
if key != slice(None, None, None):
raise ValueError("item setting only handle ':' for now")
self.__callback(
frame=value, file_name=self.file_name, scheme=self.url_scheme
)
data_url = self.data_url
os.makedirs(data_url.file_path(), exist_ok=True)
for file_name in self.data_file_name_generator(n_frames=n_frames):
yield _FrameDumper(
file_name=file_name,
url_scheme=data_url.scheme(),
callback=self.save_frame,
)
def get_volume_shape(self):
if self.data is not None:
return self.data.shape
else:
first_slice = next(self.browse_slices())
n_slices = len(tuple(self.browse_data_urls()))
return n_slices, first_slice.shape[0], first_slice.shape[1]
def get_volume_dtype(self):
if self.data is not None:
return self.data.dtype
else:
first_slice = next(self.browse_slices())
return first_slice.dtype
@docstring(VolumeBase)
def get_histogram_url(self, scan=None) -> DataUrl | None:
if self.data_url is None or self.data_url.file_path() is None:
return None
volume_basename = self.get_volume_basename()
histogram_file = os.path.join(
self.data_url.file_path(), volume_basename + "_histogram.hdf5"
)
if not os.path.exists(histogram_file):
# legacy naming - without underscore
legacy_histogram_file = os.path.join(
self.data_url.file_path(), volume_basename + "histogram.hdf5"
)
if os.path.exists(legacy_histogram_file):
histogram_file = legacy_histogram_file
return self._get_histogram_url_from_file(histogram_file, scan=scan)
@docstring(VolumeBase)
def load_data(self, store: bool = True) -> numpy.ndarray:
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to save this."
)
data = list(self.browse_slices())
if data == []:
data = None
_logger.warning(
f"Failed to load any data for {self.get_identifier().short_description}"
)
else:
data = numpy.asarray(data)
if data.ndim != 3:
raise ValueError(f"data is expected to be 3D not {data.ndim}.")
if store:
self.data = data
return data
def _read_frame_from_fast_axis(self, index: int) -> numpy.ndarray:
"""
Read a single frame along the 'fast axis' (0 aka Z).
Raise "IndexError if not accessible
"""
data_files = tuple(self.browse_data_files())
file_path = data_files[index]
return self.load_frame(file_name=file_path, scheme=self.data_url.scheme())
def get_slice(
self,
index: int | str = None,
axis=None,
xy=None,
xz=None,
yz=None,
):
# if we are on the 'fast' read axis then we can directly read a single file. Else we must fall back on the default reading mechanism
if axis == 0 or xy is not None:
if xy is not None:
deprecated_warning(
type_="parameter",
name="xy",
replacement="axis and index",
)
if axis is None and index is None:
axis = 0
index = xy
else:
raise ValueError("several axis (previously xy, xz, yz requested")
return self._read_frame_from_fast_axis(index=index)
else:
return super().get_slice(index=index, axis=axis, xy=xy, xz=xz, yz=yz)
def save_frame(self, frame: numpy.ndarray, file_name: str, scheme: str):
"""
Function dedicated for volune saving each frame on a single file
:param frame: frame to be save
:param file_name: path to store the data
:param scheme: scheme to save the data
"""
raise NotImplementedError("Base class")
def load_frame(self, file_name: str, scheme: str) -> numpy.ndarray:
"""
Function dedicated for volume saving each frame on a single file
:param file_name: path to store the data
:param scheme: scheme to save the data
"""
raise NotImplementedError("Base class")
@docstring(VolumeBase)
def load_chunk(self, chunk):
if (not numpy.iterable(chunk)) or len(chunk) != 3:
raise ValueError(
f"Expected 'chunk' to be in the form (slice(start0, stop0), slice(start1, end1), slice(start2, end2)) but got: {chunk}"
)
url = self.data_url
scheme = url.scheme() or self.DEFAULT_DATA_SCHEME
files = list(self.browse_data_files())
files_selection = files[chunk[0]]
if files_selection == []:
warn(
f"load_chunk(): empty files selection returned by chunk={chunk[0]}",
RuntimeWarning,
)
return numpy.array([], dtype="f")
# Pre-allocate chunk
first_frame = self.load_frame(files_selection[0], scheme=scheme)
first_frame = first_frame[chunk[1], chunk[2]]
result_chunk_shape = (len(files_selection),) + first_frame.shape
result_chunk = numpy.zeros(result_chunk_shape, dtype=first_frame.dtype)
# Browse files, populate chunk
for i, filename in enumerate(files_selection):
data = self.load_frame(filename, scheme=scheme)
result_chunk[i] = data[chunk[1], chunk[2]]
return result_chunk
@docstring(VolumeBase)
def browse_metadata_files(self):
url = self.metadata_url
if url is None:
return
elif url.file_path() is not None:
if url.scheme() == "ini":
metadata_file = url.file_path()
if url.data_path() is not None:
metadata_file = os.path.join(metadata_file, url.data_path())
if os.path.exists(metadata_file):
yield metadata_file
else:
raise ValueError(f"scheme {url.scheme()} is not handled")
@docstring(VolumeBase)
def browse_data_files(self):
url = self.data_url
if url is None:
return
research_pattern = self.get_data_path_pattern_for_data(
url.data_path(), volume_basename=self.get_volume_basename()
)
try:
research_pattern = re.compile(research_pattern)
except Exception:
_logger.error(
f"Fail to compute regular expression for {research_pattern}. Unable to load data"
)
return None
# use case of a single file
if not os.path.exists(url.file_path()):
return
elif os.path.isfile(url.file_path()):
yield url.file_path()
else:
for file_ in sorted(os.listdir(url.file_path())):
if research_pattern.match(file_):
full_file_path = os.path.join(url.file_path(), file_)
yield full_file_path
@docstring(VolumeBase)
def browse_data_urls(self):
url_ = self.data_url
with catch_log_messages():
for data_file in self.browse_data_files():
yield DataUrl(
file_path=data_file,
scheme=url_.scheme(),
)
@docstring(VolumeBase)
def browse_slices(self):
# if volume in memory
if self.data is not None:
for data_slice in self.data:
yield data_slice
return
# if load slice by slice
url = self.data_url
if url is None:
raise ValueError(
"No data and data_url know and no url provided. Unable to browse slices"
)
with catch_log_messages():
for file_path in self.browse_data_files():
yield self.load_frame(file_name=file_path, scheme=url.scheme())
@property
def write_in_descending_order(self):
"""
Return True if the data is saved with decreasing indices:
- data[0] is written to file with index `start_index`
- data[1] is written to file with index `start_index - 1`
- and so on
"""
return self._write_in_descending_order
@write_in_descending_order.setter
def write_in_descending_order(self, val):
self._write_in_descending_order = bool(val)
class TIFFLikeDiskAccessor:
"""
Class to define a common implementation of '_get_slices_from_disk' for 'tiff like' format (!!! single frame per file !!!).
(format unable to implement 'read_n_lines_in_file', 'read_n_columns_in_file' and 'read_file' functions
.. warning:: a class that want to inherit from TIFFLikeDiskAccessor should also inherit from VolumeSingleFrameBase.
a separated class has been created instead of a direct inheritance to clearly separate the '_get_slices_from_disk' instantiation
"""
def _get_slices_from_disk(
self, slices: tuple[SliceTuple]
) -> dict[SliceTuple, numpy.ndarray]:
_logger.debug("Try to read %s from disk", slices)
volume_shape = self.get_volume_shape() # pylint: disable=E1101
_logger.debug("volume shape is %s", volume_shape)
def get_output_slice_shape(axis):
if axis == 0:
return (volume_shape[1], volume_shape[2])
elif axis == 1:
return (volume_shape[0], volume_shape[2])
elif axis == 2:
return (volume_shape[0], volume_shape[1])
else:
raise ValueError(f"unknown requested axis ({axis})")
# init resulting data
result = {
slice_: numpy.empty(shape=get_output_slice_shape(slice_.axis))
for slice_ in slices
}
_logger.debug(
"Reserve numpy arrays for reading %s",
{slice_: (data.shape, data.dtype) for (slice_, data) in result.items()},
)
from volio.utils import (
group_slices_by_axis,
) # avoid cyclic import
indices_to_read_per_axis: dict[int, set[int]] = group_slices_by_axis(
slices=slices, volume_shape=volume_shape
)
_logger.debug("slices ordered by axis %s", indices_to_read_per_axis)
for i_axis_0, file_name in enumerate(
self.browse_data_files() # pylint: disable=E1101
):
# browse all files. Each file represent a slice along the axis 0
data_cache = None
# used to cache data in the case a full slice
# needs to be read along axis 0. Then we can reuse it for over axis
# handle axis 0
if i_axis_0 in indices_to_read_per_axis[0]:
data_cache = self.read_file(file_name=file_name)
result[(0, i_axis_0)][:] = data_cache
indices_to_read_per_axis[0].remove(i_axis_0)
# test: if all slices are read avoid going through all the slices
if numpy.sum([len(indices_to_read_per_axis[i]) for i in range(3)]) == 0:
break
# if data has been read already let's benefit from it
if data_cache is not None:
for index in indices_to_read_per_axis[1]:
result[(1, index)][i_axis_0:] = data_cache[index, :]
for index in indices_to_read_per_axis[2]:
result[(2, index)][i_axis_0, :] = data_cache[:, index]
else:
# else read lines and / or columns directly
if len(indices_to_read_per_axis[1]) > 0:
data = self.read_n_columns_in_file(
file_name=file_name, column_indices=indices_to_read_per_axis[1]
)
assert len(data) == len(indices_to_read_per_axis[1])
for column, index in zip(data, indices_to_read_per_axis[1]):
result[(1, index)][i_axis_0, :] = column
if len(indices_to_read_per_axis[2]) > 0:
data = self.read_n_lines_in_file(
file_name=file_name, line_indices=indices_to_read_per_axis[2]
)
assert len(data) == len(indices_to_read_per_axis[2])
for line, index in zip(data, indices_to_read_per_axis[2]):
result[(2, index)][i_axis_0, :] = line
return result
def read_n_lines_in_file(self, file_name, line_indices) -> tuple:
raise NotImplementedError("Base class")
def read_n_columns_in_file(self, file_name, column_indices) -> tuple:
raise NotImplementedError("Base class")
def read_file(self, file_name) -> tuple:
raise NotImplementedError("Base class")
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/volumes/tiffvolume.py 0000664 0000000 0000000 00000045440 15241025157 0027123 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""module defining utils for a tiff volume"""
from __future__ import annotations
import logging
import os
import warnings
import numpy
import tifffile
from silx.io.dictdump import dicttoini
from silx.io.dictdump import load as load_ini
from silx.io.url import DataUrl
from tifffile import TiffFile, TiffWriter
from tomobase.geometry import get_subvolume_shape
from tomobase.tomoobject import TomoObject
from tomobase.utils import docstring
from tomobase.utils.io import (
cast_to_default_types,
catch_log_messages,
)
from ..identifiers.tiffidentifier import MultiTiffVolumeIdentifier, TIFFVolumeIdentifier
from .singleframebase import TIFFLikeDiskAccessor, VolumeSingleFrameBase
from .volumebase import SliceTuple, VolumeBase
_logger = logging.getLogger(__name__)
__all__ = ["TIFFVolume", "MultiTIFFVolume", "has_tifffile"]
@property
def has_tifffile():
warnings.warn(
"'has_tifffile' is deprecated and will be removed in a future version. tifffile is now a core dependency of volio.",
DeprecationWarning,
)
return True
class TIFFVolume(TIFFLikeDiskAccessor, VolumeSingleFrameBase):
"""
Save volume data to single frame tiff and metadata to .txt files
:warning: each file saved under {volume_basename}_{index_zfill6}.tiff is considered to be a slice of the volume.
Example:
.. code-block:: python
import numpy
from volio import TIFFVolume
data = numpy.random.random((10, 100, 100))
metadata = {"processing_options": {"reconstruction": {"voxel_size_cm": 0.1}}}
volume = TIFFVolume(folder="tiff_volume", data=data, metadata=metadata)
volume.save()
volume_loaded = TIFFVolume(folder="tiff_volume")
volume_loaded.load()
print(volume_loaded.data.shape)
print(volume_loaded.metadata)
"""
DEFAULT_DATA_EXTENSION = "tiff"
DEFAULT_DATA_SCHEME = "tifffile"
def __init__(
self,
folder: str | None = None,
volume_basename: str | None = None,
data: numpy.ndarray | None = None,
source_scan: TomoObject | None = None,
metadata: dict | None = None,
data_url: DataUrl | None = None,
metadata_url: DataUrl | None = None,
overwrite: bool = False,
start_index=0,
data_extension=DEFAULT_DATA_EXTENSION,
metadata_extension=VolumeSingleFrameBase.DEFAULT_METADATA_EXTENSION,
) -> None:
if folder is not None:
url = DataUrl(
file_path=str(folder),
data_path=None,
)
else:
url = None
TIFFLikeDiskAccessor.__init__(self)
super().__init__(
url=url,
volume_basename=volume_basename,
data=data,
source_scan=source_scan,
metadata=metadata,
data_url=data_url,
metadata_url=metadata_url,
overwrite=overwrite,
start_index=start_index,
data_extension=data_extension,
metadata_extension=metadata_extension,
)
@docstring(VolumeSingleFrameBase)
def save_frame(self, frame, file_name, scheme):
if scheme == "tifffile":
with TiffWriter(file_name) as tiff_writer:
tiff_writer.write(frame)
else:
raise ValueError(f"scheme {scheme} is not handled")
@docstring(VolumeSingleFrameBase)
def load_frame(self, file_name, scheme) -> numpy.ndarray:
if scheme == "tifffile":
return tifffile.imread(file_name)
else:
raise ValueError(f"scheme {scheme} is not handled")
def _get_slices_from_disk(
self, slices: tuple[SliceTuple]
) -> dict[SliceTuple, numpy.ndarray]:
"""
read from files a couple of slices along any axis.
"""
return super()._get_slices_from_disk(
slices=slices,
)
# identifier section
@staticmethod
@docstring(VolumeSingleFrameBase)
def from_identifier(identifier):
"""Return the Dataset from a identifier"""
if not isinstance(identifier, TIFFVolumeIdentifier):
raise TypeError(
f"identifier should be an instance of {TIFFVolumeIdentifier} not {type(identifier)}"
)
return TIFFVolume(
folder=identifier.folder,
volume_basename=identifier.file_prefix,
)
@docstring(VolumeSingleFrameBase)
def get_identifier(self) -> TIFFVolumeIdentifier:
if self.url is None:
raise ValueError("no file_path provided. Cannot provide an identifier")
return TIFFVolumeIdentifier(
object=self, folder=self.url.file_path(), file_prefix=self._volume_basename
)
@staticmethod
def example_defined_from_str_identifier() -> str:
return " ; ".join(
[
f"{TIFFVolume(folder='/path/to/my/my_folder').get_identifier().to_str()}",
f"{TIFFVolume(folder='/path/to/my/my_folder', volume_basename='mybasename').get_identifier().to_str()} (if mybasename != folder name)",
]
)
# TIFFLikeAccessor implementation
def read_n_lines_in_file(self, file_name, line_indices) -> tuple:
return tuple(
[
tifffile.imread(file_name, selection=(slice(None, None), index))
for index in line_indices
]
)
def read_n_columns_in_file(self, file_name, column_indices):
return tuple(
[
tifffile.imread(file_name, selection=(index, slice(None, None)))
for index in column_indices
]
)
def read_file(self, file_name) -> tuple:
return tifffile.imread(file_name)
class MultiTIFFVolume(VolumeBase):
"""
Save tiff into a single tiff file
:param file_path: path to the multiframe tiff file
Example:
.. code-block:: python
import numpy
from volio import MultiTIFFVolume
data = numpy.random.random((10, 100, 100))
metadata = {"processing_options": {"reconstruction": {"voxel_size_cm": 0.1}}}
volume = MultiTIFFVolume(file_path="volume.tiff", data=data, metadata=metadata)
volume.save()
volume_loaded = MultiTIFFVolume(file_path="volume.tiff")
volume_loaded.load()
print(volume_loaded.data.shape)
print(volume_loaded.metadata)
"""
DEFAULT_DATA_EXTENSION = "tiff"
DEFAULT_METADATA_EXTENSION = "txt"
def __init__(
self,
file_path: str | None = None,
data: numpy.ndarray | None = None,
source_scan: TomoObject | None = None,
metadata: dict | None = None,
data_url: DataUrl | None = None,
metadata_url: DataUrl | None = None,
overwrite: bool = False,
append: bool = False,
data_extension=DEFAULT_DATA_EXTENSION,
metadata_extension=DEFAULT_METADATA_EXTENSION,
) -> None:
if file_path is not None:
url = DataUrl(file_path=file_path)
else:
url = None
self._file_path = file_path
super().__init__(
url,
data,
source_scan,
metadata,
data_url,
metadata_url,
overwrite,
data_extension=data_extension,
metadata_extension=metadata_extension,
)
self.append = append
@docstring(VolumeBase)
def deduce_data_and_metadata_urls(self, url: DataUrl | None) -> tuple:
# convention for tiff multiframe:
# expect the url to provide a path to a the tiff multiframe file. so data_url will be the same as url
# and the metadata_url will target a prefix_info.txt file with prefix is the tiff file prefix
if url is None:
return None, None
else:
if url.data_slice() is not None:
raise ValueError(f"data_slice is not handled by the {MultiTIFFVolume}")
file_path = url.file_path()
if url.data_path() is not None:
raise ValueError("data_path is not handled")
scheme = url.scheme() or "tifffile"
metadata_file = "_".join([os.path.splitext(file_path)[0], "infos.txt"])
return (
# data url
DataUrl(
file_path=url.file_path(),
scheme=scheme,
),
# medata url
DataUrl(
file_path=metadata_file,
scheme="ini",
),
)
@docstring(VolumeBase)
def save_data(self) -> None:
"""
:raises KeyError: if data path already exists and overwrite set to False
:raises ValueError: if data is None
"""
# to be discussed. Not sure we should raise an error in this case. Could be useful but this could also be double edged knife
if self.data is None:
raise ValueError("No data to be saved")
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to save this."
)
if url.file_path() is not None and os.path.dirname(url.file_path()) != "":
os.makedirs(os.path.dirname(url.file_path()), exist_ok=True)
if url.scheme() == "tifffile":
if url.data_path() is not None:
raise ValueError("No data path expected. Unagleto save data")
else:
_logger.info(f"save data to {url.path()}")
with TiffWriter(url.file_path(), bigtiff=True, append=self.append) as tif:
if self.data.ndim == 2:
tif.write(self.data)
elif self.data.ndim == 3:
for data_slice in self.data:
tif.write(data_slice)
else:
raise ValueError(f"data should be 3D and not {self.data.ndim}D")
else:
raise ValueError(f"Scheme {url.scheme()} is not handled")
@docstring(VolumeBase)
def data_file_saver_generator(self, n_frames, overwrite: bool = False):
"""
warning: the file will be open until the generator exists
"""
class _FrameDumper:
"""
will not work for VirtualLayout
"""
def __init__(self, url, append) -> None:
self.url = url
self.append = append
def __setitem__(self, key, value):
if self.url.scheme() == "tifffile":
if self.url.data_path() is not None:
raise ValueError("No data path expected. Unagleto save data")
else:
_logger.info(f"save data to {self.url.path()}")
if key != slice(None, None, None):
raise ValueError("item setting only handle ':' for now")
with TiffWriter(
self.url.file_path(), bigtiff=True, append=self.append
) as tif:
tif.write(value)
else:
raise ValueError(f"Scheme {self.url.scheme()} is not handled")
data_url = self.data_url
if (
data_url.file_path() is not None
and os.path.dirname(data_url.file_path()) != ""
):
os.makedirs(os.path.dirname(data_url.file_path()), exist_ok=True)
for i_frame in range(n_frames):
yield _FrameDumper(data_url, append=self.append if i_frame == 0 else True)
@docstring(VolumeBase)
def save_metadata(self) -> None:
"""
:raises KeyError: if data path already exists and overwrite set to False
:raises ValueError: if data is None
"""
if self.metadata is None:
raise ValueError("No metadata to be saved")
url = self.metadata_url
if url is None:
raise ValueError(
"Cannot get metadata_url. An url should be provided. Don't know where to save this."
)
if url.file_path() is not None and os.path.dirname(url.file_path()) != "":
os.makedirs(os.path.dirname(url.file_path()), exist_ok=True)
_logger.info(f"save metadata to {url.path()}")
if url.scheme() == "ini":
if url.data_path() is not None:
raise ValueError("data_path is not handled by 'ini' scheme")
else:
metadata = cast_to_default_types(self.metadata)
dicttoini(
metadata,
url.file_path(),
)
else:
raise ValueError(f"Scheme {url.scheme()} is not handled by multiframe tiff")
@property
def file_path(self):
return self._file_path
@file_path.setter
def file_path(self, file_path: str | None):
if not (file_path is None or isinstance(file_path, str)):
raise TypeError
self._file_path = file_path
@docstring(VolumeBase)
def load_data(self, store: bool = True) -> numpy.ndarray:
url = self.data_url
if url is None:
raise ValueError(
"Cannot get data_url. An url should be provided. Don't know where to save this."
)
data = numpy.asarray([slice for slice in self.browse_slices()])
if store:
self.data = data
return data
@docstring(VolumeBase)
def load_metadata(self, store: bool = True) -> dict:
url = self.metadata_url
if url is None:
raise ValueError(
"Cannot get metadata_url. An url should be provided. Don't know where to save this."
)
if url.scheme() == "ini":
metadata_file = url.file_path()
if url.data_path() is not None:
raise ValueError("data_path is not handled by ini scheme")
else:
try:
metadata = load_ini(metadata_file, "ini")
except FileNotFoundError:
_logger.warning(f"unable to load metadata from {metadata_file}")
metadata = {}
else:
raise ValueError(f"Scheme {url.scheme()} is not handled by multiframe tiff")
if store:
self.metadata = metadata
return metadata
@staticmethod
@docstring(VolumeBase)
def from_identifier(identifier):
"""Return the Dataset from a identifier"""
if not isinstance(identifier, MultiTiffVolumeIdentifier):
raise TypeError(
f"identifier should be an instance of {MultiTiffVolumeIdentifier}"
)
return MultiTIFFVolume(
file_path=identifier.file_path,
)
@docstring(VolumeBase)
def get_identifier(self) -> MultiTiffVolumeIdentifier:
if self.url is None:
raise ValueError("no file_path provided. Cannot provide an identifier")
return MultiTiffVolumeIdentifier(object=self, tiff_file=self.url.file_path())
def browse_metadata_files(self):
"""
return a generator go through all the existing files associated to the data volume
"""
url = self.metadata_url
if url is None:
return
elif url.file_path() is not None and os.path.exists(url.file_path()):
yield url.file_path()
def browse_data_files(self, url=None):
"""
return a generator go through all the existing files associated to the data volume
"""
url = self.data_url
if url is None:
return
elif url.file_path() is not None and os.path.exists(url.file_path()):
yield url.file_path()
def browse_data_urls(self):
url = self.data_url
with catch_log_messages():
for data_file in self.browse_data_files():
yield DataUrl(
file_path=data_file,
scheme=url.scheme(),
)
@docstring(VolumeBase)
def browse_slices(self):
# load slice from in-memory volume when possible
if self.data is not None:
for data_slice in self.data:
yield data_slice
return
# load slice one by one from disk
url = self.data_url
if url is None:
raise ValueError(
"No data and data_url know and no url provided. Unable to browse slices"
)
if url.scheme() == "tifffile":
if url.data_path() is not None:
raise ValueError("data_path is not handle by multiframe tiff")
with TiffFile(url.file_path()) as reader:
for series in reader.series:
data = series.asarray()
if data.ndim == 3:
for data_slice in data:
yield data_slice
elif data.ndim == 2:
yield data
else:
raise ValueError("series is expected to be 2D or 3D")
else:
raise ValueError(f"Scheme {url.scheme()} is not handled by multiframe tiff")
def get_volume_shape(self):
if self.data is not None:
return self.data.shape
url = self.data_url
with tifffile.TiffFile(url.file_path()) as t:
shapes = [series.shape for series in t.series]
# assume that all series have the same dimensions for axis 1 and 2
vol_shape = (len(t.series), shapes[0][0], shapes[0][1])
return vol_shape
def get_volume_dtype(self):
if self.data is not None:
return self.data.dtype
with tifffile.TiffFile(self.url.file_path()) as t:
dtype = t.series[0].dtype
# assume that dtype is the same for all series
return dtype
@docstring(VolumeBase)
def get_histogram_url(self, scan=None) -> DataUrl | None:
if self.data_url is None or self.data_url.file_path() is None:
return None
file_path, _ = os.path.splitext(self.data_url.file_path())
histogram_file = file_path + "_histogram.hdf5"
return self._get_histogram_url_from_file(histogram_file, scan=scan)
@docstring(VolumeBase)
def load_chunk(self, chunk):
vol_shape = self.get_volume_shape()
vol_dtype = self.get_volume_dtype()
chunk_shape = get_subvolume_shape(chunk, vol_shape)
data_chunk = numpy.zeros(chunk_shape, dtype=vol_dtype)
start_z = chunk[0].start or 0
with catch_log_messages():
for i, image in enumerate(self.browse_slices()):
if i >= start_z and i - start_z < chunk_shape[0]:
data_chunk[i - start_z, ...] = image[chunk[1:]]
return data_chunk
@staticmethod
def example_defined_from_str_identifier() -> str:
return (
MultiTIFFVolume(file_path="/path/to/tiff_file.tif")
.get_identifier()
.to_str()
)
volio-v0.0.0.dev1-8e6bbd971ead7ce64b5a3a9708a3184c7c31ac2c/volio/volumes/volumebase.py 0000664 0000000 0000000 00000070642 15241025157 0027107 0 ustar 00root root 0000000 0000000 # coding: utf-8
"""module to define base class for a volume"""
from __future__ import annotations
import logging
import os
from typing import NamedTuple
import numpy
import pint
from h5py import File
from silx.io.url import DataUrl
from silx.io.utils import get_data
from silx.math.combo import ( # pylint: disable=E0611 (not found from the pyx file)
min_max,
)
from tomobase.geometry import BoundingBox1D, BoundingBox3D
from tomobase.tomoobject import TomoObject
from tomobase.utils.io import (
catch_log_messages,
deprecated,
deprecated_warning,
)
from volio import drac as drac_utils
from volio.identifiers.base import VolumeIdentifier
_ureg = pint.get_application_registry()
_logger = logging.getLogger(__name__)
__all__ = ["SliceTuple", "VolumeBase"]
class SliceTuple(NamedTuple):
axis: int
index: int
class VolumeBase(TomoObject):
"""
Base class defining a common way of saving and loading tomography reconstruction
volumes across the tomotools suite, so volumes can be created, read and processed
(e.g. stitched) the same way regardless of their storage format.
Concrete formats are implemented as subclasses (see :class:`~volio.HDF5Volume`,
:class:`~volio.EDFVolume`, :class:`~volio.TIFFVolume`,
:class:`~volio.MultiTIFFVolume`, :class:`~volio.JP2KVolume` and
:class:`~volio.RawVolume`), which all share this same API: build them from
``data`` / ``metadata`` and a location, then use :meth:`save` / :meth:`load`.
:param url: url of the volume. Could be path to a master file if we can provide one per each volume. Otherwise could be a pattern of edf files or tiff file with a data range
:param source_scan: potential instance exposing the `TomoObject` interface (e.g. a tomoscan scan) in order to get extra information. This could be saved in the volume file to (external link)
:param data: volume data. Expected to be 3D
:param metadata: metadata associated to the volume. Must be a dict of serializable object
:param data_url: url to save the data. If provided url must not be provided. If an object is constructed from data and metadata url then no rule to create a VolumeIdentifier can be created and call to get_identifier will raise an error.
:param metadata_url: url to save the metadata. If provided url must not be provided. If an object is constructed from data and metadata url then no rule to create a VolumeIdentifier can be created and call to get_identifier will raise an error.
:param overwrite: when save the data if encounter a resource already existing overwrite it (if True) or not.
:param data_extension: extension to use for the data file(s) if it can vary for this format (e.g. 'edf', 'tif'). If None the format default is used.
:param metadata_extension: extension to use for the metadata file if it can vary for this format (e.g. 'txt'). If None the format default is used.
:raises TypeError: if data, metadata, source_scan or url are of an unexpected type.
:raises ValueError: * if data is a numpy array and not 3D.
* if both url and (data_url or metadata_url) are provided.
:raises OSError: if an existing file cannot be read or written to.
"""
EXTENSION = None
def __init__(
self,
url: DataUrl | None = None,
data: numpy.ndarray | None = None,
source_scan: TomoObject | None = None,
metadata: dict | None = None,
data_url: DataUrl | None = None,
metadata_url: DataUrl | None = None,
overwrite: bool = False,
data_extension: str | None = None,
metadata_extension: str | None = None,
) -> None:
super().__init__()
if url is not None and (data_url is not None or metadata_url is not None):
raise ValueError(
"Either url or (data_url and / or metadata_url) can be provided not both"
)
# warning on source_scan: should be defined before the url because deduce_data_and_metadata_urls might need it
# Then as source scan can imply several modification of url... we can only define it during construction and this
# must not involve with object life
if not isinstance(source_scan, (TomoObject, type(None))):
raise TypeError(
f"source scan is expected to be None or an instance of TomoObject. Not {type(source_scan)}"
)
self.__source_scan = source_scan
self._data_extension = data_extension
self._metadata_extension = metadata_extension
self.overwrite = overwrite
self.url = url
self.metadata = metadata
self.data = data
if url is None:
self._data_url = data_url
self._metadata_url = metadata_url
else:
# otherwise have be set from call to 'deduce_data_and_metadata_urls'
pass
@property
def url(self):
return self._url
@url.setter
def url(self, url: DataUrl | None) -> None:
if url is not None and not isinstance(url, DataUrl):
raise TypeError
self._url = url
self._data_url, self._metadata_url = self.deduce_data_and_metadata_urls(url)
def deduce_data_and_metadata_urls(self, url: DataUrl | None) -> tuple:
"""
compute data and metadata urls from 'parent url'
:return: data_url: DataUrl | None, metadata_url: DataUrl | None
"""
raise NotImplementedError("Base class")
@property
def data_extension(self):
return self._data_extension
@property
def metadata_extension(self):
return self._metadata_extension
@property
def data_url(self):
return self._data_url
@property
def metadata_url(self):
return self._metadata_url
@property
def data(self) -> numpy.ndarray | None:
return self._data
@data.setter
def data(self, data):
if not isinstance(data, (numpy.ndarray, type(None))):
raise TypeError(
f"data is expected to be None or a numpy array not {type(data)}"
)
if isinstance(data, numpy.ndarray) and data.ndim != 3:
raise ValueError(f"data is expected to be 3D and not {data.ndim}D.")
self._data = data
def get_slice(
self,
index: str | int = None,
axis=None,
xy=None,
xz=None,
yz=None,
) -> None | numpy.ndarray:
"read a single slice of the volume"
if xy is yz is xz is axis is None:
raise ValueError("axis should be provided")
if self.data is None:
with catch_log_messages():
self.load_data(store=True)
if isinstance(index, str):
if index == "middle":
index = self.get_volume_shape()[axis] // 2
elif index == "first":
index = 0
elif index == "last":
index = -1
if self.data is not None:
return self.select(
volume=self.data, xy=xy, xz=xz, yz=yz, axis=axis, index=index
)
else:
return None
def get_slices(
self, slices: tuple[SliceTuple | tuple]
) -> dict[SliceTuple, numpy.ndarray]:
"""
retrieve a couple of slices along any axis:
For example, if you want to retrieve slice number 2 of axis 0 and slice number 56 of axis 1:
.. code-block:: python
slices = volume.get_slices(
(0, 2),
(1, 56),
)
for (axis, slice), data in slices:
...
"""
if not isinstance(slices, tuple):
raise TypeError(f"slices should be a tuple. Got {type(slices)}")
for elmt in slices:
if not isinstance(elmt, (tuple, NamedTuple)):
raise TypeError(
f"slices is expected to be a tuple of tuple or a tuple of SliceTuple. Get {type(elmt)} as an element of the tuple"
)
# cast 2 element tuple to SliceTuple
slices = tuple([SliceTuple(*slice_) for slice_ in slices])
if self.data is not None:
return self.select_slices(volume=self.data, slices=slices)
else:
return self._get_slices_from_disk(slices=slices)
def _get_slices_from_disk(
self, slices: tuple[SliceTuple]
) -> dict[SliceTuple, numpy.ndarray]:
"""
read from files a couple of slices along any axis.
"""
with catch_log_messages():
data = self.load_data(store=False)
if data is None:
raise RuntimeError(
f"Enable to load data from disk for {self.get_identifier()}"
)
return self.select_slices(volume=data, slices=slices)
@property
def metadata(self) -> dict | None:
return self._metadata
@metadata.setter
def metadata(self, metadata: dict | None):
if not isinstance(metadata, (dict, type(None))):
raise TypeError(
f"metadata is expected to be None or a dict not {type(metadata)}"
)
self._metadata = metadata
@staticmethod
def example_defined_from_str_identifier() -> str:
"""example as string to explain how users can defined identifiers from a string"""
raise NotImplementedError("Base class")
def clear_cache(self):
"""remove object stored in data and metadata"""
self.data = None
self.metadata = None
# generic function requested
@property
def source_scan(self) -> TomoObject | None:
return self.__source_scan
@property
def overwrite(self) -> bool:
return self._overwrite
@overwrite.setter
def overwrite(self, overwrite: bool) -> None:
if not isinstance(overwrite, bool):
raise TypeError
self._overwrite = overwrite
# function to be loaded to an url
@staticmethod
def from_identifier(identifier: str | VolumeIdentifier):
"""Return the Dataset from a identifier"""
raise NotImplementedError("Base class")
def get_identifier(self) -> VolumeIdentifier:
"""dataset unique identifier. Can be for example a hdf5 and
en entry from which the dataset can be rebuild"""
raise NotImplementedError("Base class")
# utils required for operations like stitching
@staticmethod
def _insure_reconstruction_dict_exists(ddict):
if "processing_options" not in ddict:
ddict["processing_options"] = {}
if "reconstruction" not in ddict["processing_options"]:
ddict["processing_options"]["reconstruction"] = {}
@property
def position(self) -> tuple | None:
"""position are provided as a tuple using the same reference for axis as the volume data.
position is returned as (axis_0_pos, axis_1_pos, axis_2_pos). Can also be see as (z_position, y_position, x_position)
"""
metadata = self.metadata or self.load_metadata()
position = (
metadata.get("processing_options", {})
.get("reconstruction", {})
.get("position", None)
)
if position is None:
return None
else:
return tuple(position)
@position.setter
def position(self, position) -> None:
if self.metadata is None:
self.metadata = {}
self._insure_reconstruction_dict_exists(self.metadata)
self.metadata["processing_options"]["reconstruction"]["position"] = numpy.array(
position
)
@property
@deprecated(replacement="voxel_size", since_version="1.3.0")
def pixel_size(self):
return self.voxel_size
@pixel_size.setter
@deprecated(replacement="voxel_size", since_version="1.3.0")
def pixel_size(self, pixel_size) -> None:
if numpy.isscalar(pixel_size):
pixel_size = [pixel_size] * 3
_logger.warning(
"pixel_size is expected to be a tuple. Conversion will be removed soon. Please update your code"
)
self.voxel_size = pixel_size
@property
def voxel_size(self) -> tuple[float] | None:
"""
voxel size as (axis 0 dim - aka z, axis 1 dim - aka y, axis 2 dim aka z)
Returned in meter
"""
metadata = self.metadata or self.load_metadata()
voxel_size = (
metadata.get("processing_options", {})
.get("reconstruction", {})
.get("voxel_size_cm", None)
)
if voxel_size is None:
# ensure backward compatibility with old volumes (before nabu 2023.1)
# try fall back on pixel_size_cm for old volumes
voxel_size = (
metadata.get("processing_options", {})
.get("reconstruction", {})
.get("pixel_size_cm", None)
)
if voxel_size is not None:
if numpy.isscalar(voxel_size):
# be safer and handle the case voxel size is a scalar
voxel_size = [voxel_size] * 3
# voxel are in cm -> convert them to meter
voxel_size = (voxel_size * _ureg.centimeter).to(_ureg.meter).magnitude
# convert to float
return tuple([float(value) for value in voxel_size])
return None
@voxel_size.setter
def voxel_size(self, voxel_size: tuple) -> None:
if self.metadata is None:
self.metadata = {}
if numpy.isscalar(voxel_size):
raise TypeError("voxel is expected to be a tuple of three values")
self._insure_reconstruction_dict_exists(self.metadata)
voxel_size_m = voxel_size * _ureg.meter
voxel_size_cm = voxel_size_m.to(_ureg.centimeter)
self.metadata["processing_options"]["reconstruction"]["voxel_size_cm"] = (
numpy.array(voxel_size_cm.magnitude)
)
def get_volume_shape(self) -> tuple:
"""
return volume shape as a tuple
"""
raise NotImplementedError("Base class")
def get_volume_dtype(self) -> numpy.dtype | None:
"""
return the dtype of the volume data, avoiding to load the full data when possible.
"""
if self.data is not None:
return self.data.dtype
raise NotImplementedError("Base class")
def get_volume_size(self, unit: str = "byte") -> float:
"""
estimate the size the volume would take once fully loaded in memory (as a numpy array).
If `data` is already loaded then the exact size is returned (`data.nbytes`).
Otherwise the size is estimated from the volume shape and dtype, avoiding to load the full data.
:param unit: unit to express the size in. Any 'byte' compatible unit handled by `pint` (e.g. 'byte', 'kilobyte', 'Mbyte', 'Gbyte'...)
"""
if self.data is not None:
n_bytes = self.data.nbytes
else:
shape = self.get_volume_shape()
if shape is None:
raise ValueError(
"Unable to determine volume shape. Cannot estimate volume size"
)
dtype = self.get_volume_dtype()
if dtype is None:
raise ValueError(
"Unable to determine volume dtype. Cannot estimate volume size"
)
n_bytes = int(numpy.prod(shape)) * numpy.dtype(dtype).itemsize
return (n_bytes * _ureg.byte).to(unit).magnitude
def get_bounding_box(self, axis: str | int | None = None):
if axis is None:
x_bb = self.get_bounding_box(axis=2)
y_bb = self.get_bounding_box(axis=1)
z_bb = self.get_bounding_box(axis=0)
return BoundingBox3D(
(z_bb.min, y_bb.min, x_bb.min),
(z_bb.max, y_bb.max, x_bb.max),
)
position = self.position
shape = self.get_volume_shape()
# TODO: does it make sense that pixel size is a scalar ?
voxel_size = self.voxel_size
missing = []
if position is None:
missing.append("position")
if shape is None:
missing.append("shape")
raise ValueError("Unable to find volume shape")
if voxel_size is None:
missing.append("pixel_size")
if len(missing) > 0:
raise ValueError(
f"Unable to get bounding box. Missing information: {'; '.join(missing)}"
)
assert axis is not None
if isinstance(axis, str):
if axis == "x":
axis = 2
elif axis == "y":
axis = 1
elif axis == "z":
axis = 0
else:
raise ValueError(f"axis '{axis}' is not handled")
min_pos_in_meter = position[axis] - voxel_size[axis] * shape[axis] / 2.0
max_pos_in_meter = position[axis] + voxel_size[axis] * shape[axis] / 2.0
return BoundingBox1D(min_pos_in_meter, max_pos_in_meter)
def get_min_max(self) -> tuple:
"""
compute min max of the volume. Can take some time but avoid to load the full volume in memory
"""
if self.data is not None:
return self.data.min(), self.data.max()
else:
min_v, max_v = None, None
for s in self.browse_slices():
min_v = min(min_v, s.min()) if min_v is not None else s.min()
max_v = max(max_v, s.max()) if max_v is not None else s.max()
return min_v, max_v
# load / save stuff
@property
def extension(self) -> str:
return self.EXTENSION
def load(self):
self.load_metadata(store=True)
# always load metadata first because we might expect to get some information from
# it in order to load data next
self.load_data(store=True)
def save(self, **kwargs):
"""
save volume data and metadata to disk
"""
# catch warning from using data_url until 'url' removed from the signature. See https://gitlab.esrf.fr/tomotools/tomwer/-/issues/1488
with catch_log_messages():
self.save_data(**kwargs)
if self.metadata is not None:
# a volume is not force to have metadata to save. But calling save_metadata direclty might raise an error
# if no metadata found
self.save_metadata()
def save_data(self, **kwargs) -> None:
"""
save data to the provided url or existing one if none is provided
"""
raise NotImplementedError("Base class")
def save_metadata(self) -> None:
"""
save metadata to the provided url or existing one if none is provided
"""
raise NotImplementedError("Base class")
def load_data(self, store: bool = True) -> numpy.ndarray:
"""
Load volume data from disk.
"""
raise NotImplementedError("Base class")
def load_metadata(self, store: bool = True) -> dict:
"""
Load volume metadata from disk
"""
raise NotImplementedError("Base class")
def check_can_provide_identifier(self):
if self.url is None:
raise ValueError(
"Unable to provide an identifier. No url has been provided"
)
@staticmethod
def select_slices(volume: numpy.ndarray, slices: tuple[SliceTuple]) -> dict:
return {
SliceTuple(axis=slice_.axis, index=slice_.index): VolumeBase.select(
volume=volume, index=slice_.index, axis=slice_.axis
)
for slice_ in slices
}
@staticmethod
def select(
volume: numpy.ndarray,
xy: int | None = None,
xz: int | None = None,
yz: int | None = None,
axis: int | None = None,
index: int | None = None,
) -> numpy.array:
"""
select a slice at 'index' along an axis (axis)
"""
if xy is not None:
deprecated_warning(
type_="parameter",
name="xy",
replacement="axis and index",
)
if axis is None and index is None:
axis = 0
index = xy
else:
raise ValueError("several axis (previously xy, xz, yz requested")
elif xz is not None:
deprecated_warning(
type_="parameter",
name="xz",
replacement="axis and index",
)
if axis is None and index is None:
axis = 1
index = xz
else:
raise ValueError("several axis (previously xy, xz, yz requested")
elif yz is not None:
deprecated_warning(
type_="parameter",
name="yz",
replacement="axis and index",
)
if axis is None and index is None:
axis = 2
index = yz
else:
raise ValueError("several axis (previously xy, xz, yz requested")
if not volume.ndim == 3:
raise TypeError(f"volume is expected to be 3D. {volume.ndim}D provided")
if axis == 0:
return volume[index]
elif axis == 1:
return volume[:, index]
elif axis == 2:
return volume[:, :, index]
else:
raise ValueError(f"axis {axis} is not handled")
def browse_data_files(self):
"""
return a generator go through all the existing files associated to the data volume
"""
raise NotImplementedError("Base class")
def browse_metadata_files(self):
"""
return a generator go through all the existing files associated to the data volume
"""
raise NotImplementedError("Base class")
def browse_data_urls(self):
"""
generator on data urls used.
"""
raise NotImplementedError("Base class")
def browse_slices(self):
"""
generator of 2D numpy array representing a slice
:warning: this will get the slice from the data on disk and never use `data` property.
so before browsing slices you might want to check if data is already loaded
"""
raise NotImplementedError("Base class")
def load_chunk(self, chunk):
"""
Load a sub-volume.
:param chunk: tuple of slice objects indicating which chunk of the volume has to be loaded.
"""
raise NotImplementedError("Base class")
def get_min_max_values(self) -> tuple:
"""
compute min max over 'data' if exists else browsing the volume slice by slice
"""
min_v = None
max_v = None
if self.data is not None:
data = self.data
else:
with catch_log_messages():
data = self.browse_slices()
for data_slice in data:
if min_v is None:
min_v = data_slice.min()
max_v = data_slice.max()
else:
min_lv, max_lv = min_max(data_slice, finite=True)
min_v = min(min_v, min_lv)
max_v = max(max_v, max_lv)
return min_v, max_v
def get_histogram_url(self, scan: TomoObject | None = None) -> DataUrl | None:
"""
Try to locate the histogram of this volume, as computed by nabu at reconstruction time.
:param scan: scan the volume has been reconstructed from. Can help figuring out the HDF5 entry
containing the histogram when it cannot be deduced from the volume itself.
:return: url to the histogram dataset if one can be found on disk, else None. Base class has no
known histogram location and always returns None - overridden by volume formats nabu can write a histogram for.
"""
return None
@staticmethod
def _get_histogram_entry(
histogram_file: str, scan: TomoObject | None = None
) -> str | None:
"""
resolve the HDF5 entry (NXentry name) expected to contain the histogram inside 'histogram_file'
"""
if scan is not None:
# an object exposing an 'entry' attribute (like NXtomoScan). Other scan-like objects are expected to use 'entry' as default
return getattr(scan, "entry", "entry")
with File(histogram_file, mode="r") as h5f:
entries = tuple(h5f.keys())
if len(entries) == 1:
return entries[0]
_logger.info(
f"Unable to find a single relevant entry in {histogram_file} (found {entries}). "
"Please provide 'scan' or recompute the histogram."
)
return None
def _get_histogram_url_from_file(
self, histogram_file: str | None, scan: TomoObject | None = None
) -> DataUrl | None:
"""
build the DataUrl of the histogram stored in 'histogram_file' (a separate hdf5 file next to the volume)
"""
if histogram_file is None or not os.path.exists(histogram_file):
return None
entry = self._get_histogram_entry(histogram_file=histogram_file, scan=scan)
if entry is None:
return None
data_path = f"{entry}/histogram/results/data"
with File(histogram_file, mode="r") as h5f:
if data_path not in h5f:
_logger.info(f"{data_path} in {histogram_file} not found")
return None
return DataUrl(file_path=histogram_file, data_path=data_path, scheme="silx")
def get_histogram(
self,
scan: TomoObject | None = None,
recompute_if_missing: bool = False,
n_bins: int = 256,
) -> tuple | None:
"""
Try to retrieve the histogram of this volume, as computed by nabu at reconstruction time.
:param scan: scan the volume has been reconstructed from (see `get_histogram_url`)
:param recompute_if_missing: if True and no histogram can be found on disk, compute it by
browsing the volume slice by slice. This can be time-consuming
as it requires reading the full volume.
:param n_bins: number of bins to use when recomputing the histogram. Ignored if an existing
histogram is found on disk.
:return: (counts, bins) as two numpy arrays, or None if no histogram could be found nor computed.
"""
try:
url = self.get_histogram_url(scan=scan)
except Exception as e:
_logger.warning(f"Failed looking for an existing histogram. Error is {e}")
url = None
if url is not None:
try:
histogram = get_data(url)
except Exception as e:
_logger.warning(f"Failed to load histogram from {url}. Error is {e}")
else:
return histogram[0], histogram[1]
if not recompute_if_missing:
return None
return self.compute_histogram(n_bins=n_bins)
def compute_histogram(self, n_bins: int = 256) -> tuple:
"""
Compute the volume histogram by browsing it slice by slice, to avoid loading the full volume in memory.
:param n_bins: number of bins of the returned histogram
:return: (counts, bin_edges) as two numpy arrays
"""
min_value, max_value = self.get_min_max_values()
if min_value is None or max_value is None:
raise ValueError("Unable to find volume value range (empty volume?)")
bin_edges = numpy.histogram_bin_edges(
numpy.array([]), bins=n_bins, range=(min_value, max_value)
)
counts = numpy.zeros(n_bins, dtype=numpy.float64)
for data_slice in self.browse_slices():
slice_counts, _ = numpy.histogram(
data_slice.ravel(), bins=bin_edges, range=(min_value, max_value)
)
counts += slice_counts
return counts, bin_edges
def data_file_saver_generator(self, n_frames, overwrite: bool = False):
"""
Provide a helper class to dump data frame by frame. For know the only possible interaction is
Helper[:] = frame
:param n_frames: number of frame the final volume will contain
:param DataUrl data_url: **Deprecated** url to dump data
:param overwrite: overwrite existing file ?
"""
raise NotImplementedError("Base class")
def build_drac_metadata(self) -> dict:
"""
build the drac (successor of icat) metadata dict from existing volume metadata.
"""
metadata = self.metadata or self.load_metadata()
flatten_metadata = drac_utils.flatten_vol_metadata(metadata)
icat_metadata = drac_utils.map_nabu_keys_to_drac(flatten_metadata)
return icat_metadata