edit an NXtomo#

The general pipeline to edit an NXtomo is:

load it from disk -> modify it 'in memory' -> save it to disk

In this example we will edit the dummy_nxtomo.nx file. which is the outcome of the ‘create_from_scratch’ tutorial

[1]:

import os
from nxtomo import NXtomo

nx_tomo_file_path = os.path.join("resources", "dummy_nxtomo.nx")
nx_tomo = NXtomo().load(nx_tomo_file_path, "entry", detector_data_as="as_numpy_array")
print("nx_tomo type is", type(nx_tomo))
print("nx_tomo energy is", nx_tomo.energy)

nx_tomo type is <class 'nxtomo.application.nxtomo.NXtomo'>
nx_tomo energy is None keV

Then you can modify your values as it was presented previously and overwrite the file.

[2]:

nx_tomo.energy = 13.6
nx_tomo.save(
    file_path=nx_tomo_file_path,
    data_path="entry",
    overwrite=True,
)
print("new energy is", NXtomo().load(nx_tomo_file_path, "entry").energy)

new energy is 13.6 keV

The detector data is usually saved as a h5py virtual dataset. The amount of data assocaited can be heavy according to the acquisition and to the available memory. In order to allow a ‘smooth’ edition detector data can be load according to several strategies:

“as_data_url” (default): in this case each Virtual Source will be saved as a DataUrl in order to ease it handling (see later on the tutorial)
“as_virtual_source”: retrieve original VirtualSource to allow edition of it
“as_numpy_array”: load all data in memory in order to modify it (and will dump the entire data). to avoid in case of “real” dataset. Can trigger huge IO.

clean#

[3]:

if os.path.exists(nx_tomo_file_path):
    os.remove(nx_tomo_file_path)
if os.path.exists("nxtomo_reconstruction.hdf5"):
    os.remove("nxtomo_reconstruction.hdf5")

Advance usage: Provide DataUrl to instrument.detector.data#

The issue of using NXtomo we presented above is that the memory to handle data can be used (if you have a large number of projections and / or large detector).

The way to work around for now it to use provide DataUrl (that can be pointing to an external file). Then this is the job to the FrameAppender to handle those.

For this example we will first save metadata to the hdf5 (and maybe some frame) then you can append to the dataset series of frames sequentially with there rotation angle, image key…

create some dataset to external files#

Here we simply create datasets on external files and record the DataUrls

those datasets must be 3D otherwise the virtual dataset creation will fail.

[4]:

import h5py
import numpy
from silx.io.url import DataUrl
from nxtomo.utils.frameappender import FrameAppender


detector_data_urls = []
for i_file in range(5):
    os.makedirs("output/external_files", exist_ok=True)
    external_file = os.path.join(f"output/external_files/file_{i_file}.nx")
    with h5py.File(external_file, mode="w") as h5f:
        h5f["data"] = numpy.arange(
            start=(5 * 100 * 100 * i_file),
            stop=(5 * 100 * 100 * (i_file + 1))
        ).reshape([5, 100, 100])  # of course here this is most likely that you will load data from another file

    detector_data_urls.append(
        DataUrl(
            file_path=external_file,
            data_path="data",
            scheme="silx",
        )
    )

create a simple nxtomo but this time provide the list of DataUrl to the instrument.detector.data attribute#

[5]:

my_large_nxtomo = NXtomo()

provide all information at the exception of frames. Here lets say we will have a dataset with only 180 projections

[6]:

my_large_nxtomo.instrument.detector.distance = 0.2
my_large_nxtomo.instrument.detector.x_pixel_size = my_large_nxtomo.instrument.detector.y_pixel_size = 1e-7
my_large_nxtomo.energy = 12.3
# ...
my_large_nxtomo.sample.rotation_angle = numpy.linspace(0, 180, 180, endpoint=False)
my_large_nxtomo.instrument.detector.image_key_control = [0] * 180  # 0 == Projection

provide the list of DataUrl to instrument.detector.data

[7]:

my_large_nxtomo.instrument.detector.data = detector_data_urls

[8]:

os.makedirs("output", exist_ok=True)
my_large_nxtomo.save("output/my_large_nxtomo.nx", data_path="entry0000", overwrite=True)

this will create a virtual dataset under ‘instrument/detector/data’ containing relative links from “my_large_nxtomo.nx” to other files.

Then you can see that the ‘data’ dataset now contains 180 frames (if you run several time the previous cell then it will continue appending data to it).

If an url is provided instead of a numpy array then it will create be used to create a virtual dataset and avoid duplicating data. But be carreful in this case you must keep relative position of the two files.

append frames must have the same dimensions otherwise the operation will fail

[9]:

from h5glance import H5Glance
H5Glance("output/my_large_nxtomo.nx")

[9]:

output/my_large_nxtomo.nx
- entry0000
  - beam
    - incident_energy → /entry0000/instrument/beam/incident_energy
  - data
    - data → /entry0000/instrument/detector/data
    - image_key → /entry0000/instrument/detector/image_key
    - image_key_control → /entry0000/instrument/detector/image_key_control
    - rotation_angle → /entry0000/sample/rotation_angle
  - instrument
    - beam
      - incident_energy [📋]: scalar entries, dtype: float64
    - detector
      - data [📋]: 25 × 100 × 100 entries, dtype: int64
      - distance [📋]: scalar entries, dtype: float64
      - field_of_view [📋]: scalar entries, dtype: UTF-8 string
      - image_key [📋]: 180 entries, dtype: int64
      - image_key_control [📋]: 180 entries, dtype: int64
      - x_pixel_size [📋]: scalar entries, dtype: float64
      - y_pixel_size [📋]: scalar entries, dtype: float64
    - source
      - name [📋]: scalar entries, dtype: UTF-8 string
      - probe [📋]: scalar entries, dtype: UTF-8 string
      - type [📋]: scalar entries, dtype: UTF-8 string
  - sample
    - rotation_angle [📋]: 180 entries, dtype: float64
  - definition [📋]: scalar entries, dtype: UTF-8 string

check path of VirtualSources are relative (must start with ‘./’ string):

[10]:

with h5py.File("output/my_large_nxtomo.nx", mode="r") as h5f:
    dataset = h5f["entry0000/instrument/detector/data"]
    print("dataset is virtual:", dataset.is_virtual)
    for vs_info in dataset.virtual_sources():
        print("file name is", vs_info.file_name)
        assert vs_info.file_name.startswith("./")

dataset is virtual: True
file name is ./external_files/file_0.nx
file name is ./external_files/file_1.nx
file name is ./external_files/file_2.nx
file name is ./external_files/file_3.nx
file name is ./external_files/file_4.nx

tip

you can also provide a list of h5py.VirtualSource to the detector.data attribute.
To append frame to an existing dataset you can also use the FrameAppender class from nxtomo directly.

Advanced use cases#

provide NXtransformations to NXdetector (detector flip, rotation, translation…)#

Detector can have image flip (up-down / left-right) created by the acquisition (BLISS-tango) but also some manual flips (rotation along an axis most likely).

To specify those the NXdetector has a TRANSFORMATIONS group defining the transformation chain to be applied. As of today (2023) only image flip are taking into account by nabu (for stitching).

To provide such transformations you can provide a set of transformations like:

[11]:

from nxtomo import NXtomo
my_nxtomo = NXtomo()

[12]:

from nxtomo.utils.transformation import Transformation
my_nxtomo.instrument.detector.transformations.add_transformation(
    Transformation(
        axis_name="rx",    # axis name must be unique
        transformation_type="rotation",
        value=180,         # default unit for rotation is 'degree'
        vector=(1, 0, 0),  # warning: transformation are provided as (x, y, z) which is different of the usual numpy ref used (z, y, x)
    )
)

There is several utils class to provide directly detector up-down / left-right flips, basic transformation axis… Please consider using them

[13]:

from nxtomo.utils.transformation import Transformation, UDDetTransformation, LRDetTransformation, TransformationAxis, TransformationType
from nxtomo.nxobject.nxtransformations import NXtransformations

nx_transformations = NXtransformations()
nx_transformations.transformations = (
    UDDetTransformation(),                                  # vertical flip of the detector
    LRDetTransformation(depends_on="ry"),                   # horizontal flip of the detector. Applied after the vertical flip
    Transformation(                                         # some translation over x axis. Applied after the horizontal flip
        axis_name="tx",
        value=0.02,                                         # value can be a scalar - static value - of an array of value (one per frame expected)
        transformation_type=TransformationType.TRANSLATION, # default unit for translation is SI 'meter'
        depends_on="rz",                                    #  Applied after the horizontal flip in the transformation chain
        vector=TransformationAxis.AXIS_X,
    ),
)
my_nxtomo.instrument.detector.transformations = nx_transformations

NXtransformations and NXsample

As of today the NXtomo application is using a set of dataset (‘x_translation’, ‘y_translation’, ‘z_translation’, ‘rotation_angle’ to determine the sample transformation). It might evolve in the future to the benefit of an NXtransformations. But this is not implemented at the moment. So please use the ‘original’ datasets for transformations..

[ ]: