Volumes#
Context#
the aim of classes dedicated to volumes is to have a common way of saving and loading volumes through the tomotools suite. From it we intend to simplify volume handling for creating them but also to do some post processing operation on them such as stitching but also ease handling from gui.*
overview#
Volumes contains two main parts: * data: the volume itself. It is expected to be a 3D numpy array * metadata: metadata associated to the volume such as information regarding the reconstruction parameters…
For now the following volumes exists:
HDF5Volume: save data and metadata to an hdf5 file.
EDFVolume: save data to single frame EDF files and metadata to a text file
JP2KVolume: save data to single frame jp2k files and metadata to a text file
TIFFVolume: save data to single frame tiff files and metadata to a text file
MultiTIFFVolume: save data to a single 3D tiff file and metadata to a text file
Volume API#
[1]:
from tomoscan.esrf import HDF5Volume
from tomoscan.esrf import EDFVolume
from tomoscan.esrf import TIFFVolume, MultiTIFFVolume, has_tifffile
from tomoscan.esrf import JP2KVolume, has_glymur
import numpy
from silx.io.url import DataUrl
from tempfile import TemporaryDirectory
import os
from h5glance import H5Glance
/usr/local/lib/python3.8/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
create some dataset to test#
[2]:
from tomoscan.esrf.volume.mock import create_volume
data = create_volume(frame_dims=(100, 100), z_size=5)
metadata = {
"reconstruction_params": {
"dataset": {
"location": "toto.hdf5",
"entry": "entry0000",
},
"phase": {
"method": "None",
"padding_type": "edge",
},
},
"processing_option": {
"build_sino": {
"axis_correction": "None",
"enable_halftomo": True,
},
"flatfield": {
"binning": "11",
"do_flat_distortion": False,
},
},
}
[3]:
#%pylab
[4]:
# imshow(data[0])
HDF5Volume#
constructor#
Here we will only focus on providing a file_path and a data_path to the constructor. From it it will deduce default data and metadata data_path.
This way allow us to get a identifier that can be reused. But users can also provide directly an url for data and one for the metadata. But this is for advance usage and it will not be detailled here.
[5]:
# users can define the volume will all the information
volume = HDF5Volume(file_path="test_volume.hdf5", data_path="entry0000/reconstruction", data=data, metadata=metadata)
# or define them from properties
volume = HDF5Volume()
volume.file_path = "test_volume.hdf5"
volume.data_path = "entry0000/reconstruction"
volume.data = data # data must be None or 3D numpy array
volume.metadata = metadata # metadata must be None or an instance of dict
[6]:
# data can be access from the data property
# imshow(volume.data[0])
# metadata from the metadata property.
volume.metadata
[6]:
{'reconstruction_params': {'dataset': {'location': 'toto.hdf5',
'entry': 'entry0000'},
'phase': {'method': 'None', 'padding_type': 'edge'}},
'processing_option': {'build_sino': {'axis_correction': 'None',
'enable_halftomo': True},
'flatfield': {'binning': '11', 'do_flat_distortion': False}}}
saving#
[7]:
# save the volume
volume.save()
# display the contents of the volume file
H5Glance("test_volume.hdf5")
[7]:
- data [📋]: 5 × 100 × 100 entries, dtype: float64
loadding#
once the data is saved on disk then we can retrieve it from the same class
[8]:
volume_loaded = HDF5Volume(file_path="test_volume.hdf5", data_path="entry0000/reconstruction")
volume_loaded.load()
# imshow(volume_loaded.data[0])
print(volume_loaded.metadata)
{'processing_option': {'build_sino': {'axis_correction': 'None', 'enable_halftomo': True}, 'flatfield': {'binning': '11', 'do_flat_distortion': False}}, 'reconstruction_params': {'dataset': {'entry': 'entry0000', 'location': 'toto.hdf5'}, 'phase': {'method': 'None', 'padding_type': 'edge'}}}
To avoid heavy memory consumption the data and metadata will not be loaded automatically. In order to purge cache you can also call volume.clear_cache() or set data and metadata to None like:
volume.data = None
volume.metadata = None
Note: if data or metadata is None call to saving function can raise ValueError exception.
identifier#
Each subclass of VolumeBase define what we call an identifier. The goal is that from this identifier a user can retrieve a Volume. An identifier can be save as an instance of Identifier or as it string representation. An identifier looks like:
scheme:tomo_type:data_path@data_file
For now: * scheme can be hdf5, edf, jp2k, tiff and tiff_3d * tomo_type can be scan or volume
Note: This is pretty convienient for gui for example or to ‘copy/paste’ a reference to a volume from one application to another.
get identifier#
[9]:
identifier = volume_loaded.get_identifier()
print(identifier, type(identifier))
print(identifier.to_str(), type(identifier.to_str()))
hdf5:volume:/builds/tomotools/tomoscan/doc/tutorials/test_volume.hdf5?path=entry0000/reconstruction <class 'tomoscan.esrf.identifier.hdf5Identifier.HDF5VolumeIdentifier'>
hdf5:volume:/builds/tomotools/tomoscan/doc/tutorials/test_volume.hdf5?path=entry0000/reconstruction <class 'str'>
use identifier to retrieve a volume#
[10]:
from tomoscan.factory import Factory
retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier)
assert isinstance(retrieve_volume, HDF5Volume)
retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier.to_str())
assert isinstance(retrieve_volume, HDF5Volume)
[11]:
# clean workspace
if os.path.exists("test_volume.hdf5"):
os.remove("test_volume.hdf5")
EDFVolume#
constructor#
EDFVolume will store each frame as folder_prefix_index.edf and metadata to folder_prefix_infos.txt.
So the default constructor only require a path to a folder where to save the data.
This folder path will also be the identifier
[12]:
volume = EDFVolume(folder="edf_volume", data=data, metadata=metadata)
or as for the HDF5Volume you can provide data and metadata once the Volume has been defined
[13]:
volume = EDFVolume(folder="edf_volume")
volume.data = data
volume.metadata = metadata
access to data and metadata is the same for any instance of VolumeBase, from data and metadata properties
[14]:
# imshow(volume.data[0])
print(volume.metadata)
{'reconstruction_params': {'dataset': {'location': 'toto.hdf5', 'entry': 'entry0000'}, 'phase': {'method': 'None', 'padding_type': 'edge'}}, 'processing_option': {'build_sino': {'axis_correction': 'None', 'enable_halftomo': True}, 'flatfield': {'binning': '11', 'do_flat_distortion': False}}}
saving#
[15]:
# save the volume
volume.save()
# if you want to overwrite existing data you can set the overwrite property
volume.overwrite = True
# display the contents of the folder
print(os.listdir("edf_volume"))
['edf_volume_000001.edf', 'edf_volume_000003.edf', 'edf_volume_000000.edf', 'edf_volume_infos.txt', 'edf_volume_000002.edf', 'edf_volume_000004.edf']
[16]:
# open one file containing a frame
import fabio
file_frame_0 = os.path.join("edf_volume", "edf_volume_000000.edf")
# imshow(fabio.open(file_frame_0).data)
[17]:
# display contents of the infos.txt file
metadata_file = os.path.join("edf_volume", "edf_volume_infos.txt")
with open(metadata_file, "r") as of:
print("".join(of.readlines()))
[reconstruction_params]
[reconstruction_params.dataset]
location = toto.hdf5
entry = entry0000
[reconstruction_params.phase]
method = \None
padding_type = edge
[processing_option]
[processing_option.build_sino]
axis_correction = \None
enable_halftomo = True
[processing_option.flatfield]
binning = \11
do_flat_distortion = False
loading#
[18]:
volume_loaded = EDFVolume(folder="edf_volume")
volume_loaded.load()
#imshow(volume_loaded.data[1])
print(volume_loaded.metadata)
{'reconstruction_params': {'dataset': {'location': 'toto.hdf5', 'entry': 'entry0000'}, 'phase': {'method': 'None', 'padding_type': 'edge'}}, 'processing_option': {'build_sino': {'axis_correction': 'None', 'enable_halftomo': True}, 'flatfield': {'binning': '11', 'do_flat_distortion': False}}}
identifier#
get_identifier#
[19]:
identifier = volume_loaded.get_identifier()
print(identifier, type(identifier))
print(identifier.to_str(), type(identifier.to_str()))
edf:volume:/builds/tomotools/tomoscan/doc/tutorials/edf_volume <class 'tomoscan.esrf.identifier.edfidentifier.EDFVolumeIdentifier'>
edf:volume:/builds/tomotools/tomoscan/doc/tutorials/edf_volume <class 'str'>
use identifier to retrieve a volume#
[20]:
from tomoscan.factory import Factory
retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier)
assert isinstance(retrieve_volume, EDFVolume)
retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier.to_str())
assert isinstance(retrieve_volume, EDFVolume)
[21]:
# clean
import shutil
if os.path.exists("edf_volume"):
shutil.rmtree("edf_volume")
JP2KVolume#
The API of JP2KVolume is the same as EDFVolume. So for tutorial please have a look at this one.
It requires glymur to be install
TIFFVolume#
The API of TIFFVolume is the same as EDFVolume. So for tutorial please have a look at this one.
It requires tifffile to be install
MultiTIFFVolume#
The MultiTIFFVolume requires tifffile to be install
constructor#
[22]:
volume = MultiTIFFVolume(file_path="multitiff_file.tiff", data=data, metadata=metadata)
[23]:
# access data and metadata
# imshow(volume.data[0])
print(volume.metadata)
{'reconstruction_params': {'dataset': {'location': 'toto.hdf5', 'entry': 'entry0000'}, 'phase': {'method': 'None', 'padding_type': 'edge'}}, 'processing_option': {'build_sino': {'axis_correction': 'None', 'enable_halftomo': True}, 'flatfield': {'binning': '11', 'do_flat_distortion': False}}}
saving#
[24]:
from tomoscan.esrf.volume.tiffvolume import has_tifffile
if has_tifffile:
volume.save()
else:
print("tifffile must be installed to use MultiTIFFVolume")
loading#
[25]:
if has_tifffile:
volume_loaded = MultiTIFFVolume(file_path="multitiff_file.tiff")
assert volume_loaded.data is None
assert volume_loaded.metadata is None
volume_loaded.load()
# imshow(volume_loaded.data[0])
print(volume_loaded.metadata)
else:
print("tifffile must be installed to use MultiTIFFVolume")
{'reconstruction_params': {'dataset': {'location': 'toto.hdf5', 'entry': 'entry0000'}, 'phase': {'method': 'None', 'padding_type': 'edge'}}, 'processing_option': {'build_sino': {'axis_correction': 'None', 'enable_halftomo': True}, 'flatfield': {'binning': '11', 'do_flat_distortion': False}}}
identifier#
[26]:
if has_tifffile:
identifier = volume_loaded.get_identifier()
print(identifier, type(identifier))
print(identifier.to_str(), type(identifier.to_str()))
tiff3d:volume:/builds/tomotools/tomoscan/doc/tutorials/multitiff_file.tiff <class 'tomoscan.esrf.identifier.tiffidentifier.MultiTiffVolumeIdentifier'>
tiff3d:volume:/builds/tomotools/tomoscan/doc/tutorials/multitiff_file.tiff <class 'str'>
[27]:
from tomoscan.factory import Factory
if has_tifffile:
retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier)
assert isinstance(retrieve_volume, MultiTIFFVolume)
retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier.to_str())
assert isinstance(retrieve_volume, MultiTIFFVolume)
[28]:
# clean
if os.path.exists("multitiff_file.tiff"):
os.remove("multitiff_file.tiff")
if os.path.exists(retrieve_volume.metadata_url.file_path()):
os.remove(retrieve_volume.metadata_url.file_path())