edit an NXtomo

The general pipeline to edit an NXtomo is:

load it from disk -> modify it 'in memory' -> save it to disk

In this example we will edit the dummy_nxtomo.nx file. which is the outcome of the 'create_from_scratch' tutorial

import os
from nxtomo import NXtomo

nx_tomo_file_path = os.path.join("resources", "dummy_nxtomo.nx")
nx_tomo = NXtomo().load(nx_tomo_file_path, "entry", detector_data_as="as_numpy_array")
print("nx_tomo type is", type(nx_tomo))
print("nx_tomo energy is", nx_tomo.energy)

Then you can modify your values as it was presented previously and overwrite the file.

nx_tomo.energy = 13.6
nx_tomo.save(
    file_path=nx_tomo_file_path,
    data_path="entry",
    overwrite=True,
)
print("new energy is", NXtomo().load(nx_tomo_file_path, "entry").energy)

The detector data is usually saved as a h5py virtual dataset. The amount of data assocaited can be heavy according to the acquisition and to the available memory. In order to allow a 'smooth' edition detector data can be load according to several strategies:

• "as_data_url" (default): in this case each Virtual Source will be saved as a DataUrl in order to ease it handling (see later on the tutorial)
• "as_virtual_source": retrieve original VirtualSource to allow edition of it
• "as_numpy_array": load all data in memory in order to modify it (and will dump the entire data). to avoid in case of "real" dataset. Can trigger huge IO.

clean

if os.path.exists(nx_tomo_file_path):
    os.remove(nx_tomo_file_path)
if os.path.exists("nxtomo_reconstruction.hdf5"):
    os.remove("nxtomo_reconstruction.hdf5")

Exercise A: fix crayon_broken.nx

You can find in /data/projects/tomo-sample-data/training/edit_nxtomo an nxtomo with the following issue:

• rotation angles are in radians when they should be provided as degree.
• 'fov' is set as Half when it should be Full
• sample-detector distance is 1m when it should be 0.012 meter

Fix the file using nxtomo

Advance usage: Provide DataUrl to instrument.detector.data

The issue of using NXtomo we presented above is that the memory to handle data can be used (if you have a large number of projections and / or large detector).

The way to work around for now it to use provide DataUrl (that can be pointing to an external file). Then this is the job to the FrameAppender to handle those.

For this example we will first save metadata to the hdf5 (and maybe some frame) then you can append to the dataset series of frames sequentially with there rotation angle, image key...

create some dataset to external files

Here we simply create datasets on external files and record the DataUrls

those datasets must be 3D otherwise the virtual dataset creation will fail.

import h5py
import numpy
from silx.io.url import DataUrl
from nxtomo.utils.frameappender import FrameAppender


detector_data_urls = []
for i_file in range(5):
    os.makedirs("output/external_files", exist_ok=True)
    external_file = os.path.join(f"output/external_files/file_{i_file}.nx")
    with h5py.File(external_file, mode="w") as h5f:
        h5f["data"] = numpy.arange(
            start=(5 * 100 * 100 * i_file),
            stop=(5 * 100 * 100 * (i_file + 1))
        ).reshape([5, 100, 100])  # of course here this is most likely that you will load data from another file

    detector_data_urls.append(
        DataUrl(
            file_path=external_file,
            data_path="data",
            scheme="silx",
        )
    )

create a simple nxtomo but this time provide the list of DataUrl to the instrument.detector.data attribute

my_large_nxtomo = NXtomo()

provide all information at the exception of frames. Here lets say we will have a dataset with only 180 projections

my_large_nxtomo.instrument.detector.distance = 0.2
my_large_nxtomo.instrument.detector.x_pixel_size = my_large_nxtomo.instrument.detector.y_pixel_size = 1e-7
my_large_nxtomo.energy = 12.3
# ...
my_large_nxtomo.sample.rotation_angle = numpy.linspace(0, 180, 180, endpoint=False)
my_large_nxtomo.instrument.detector.image_key_control = [0] * 180  # 0 == Projection

provide the list of DataUrl to instrument.detector.data

my_large_nxtomo.instrument.detector.data = detector_data_urls

os.makedirs("output", exist_ok=True)
my_large_nxtomo.save("output/my_large_nxtomo.nx", data_path="entry0000", overwrite=True)

this will create a virtual dataset under 'instrument/detector/data' containing relative links from "my_large_nxtomo.nx" to other files.

Then you can see that the 'data' dataset now contains 180 frames (if you run several time the previous cell then it will continue appending data to it).

If an url is provided instead of a numpy array then it will create be used to create a virtual dataset and avoid duplicating data. But be carreful in this case you must keep relative position of the two files.

append frames must have the same dimensions otherwise the operation will fail

from h5glance import H5Glance
H5Glance("output/my_large_nxtomo.nx")

check path of VirtualSources are relative (must start with './' string):

with h5py.File("output/my_large_nxtomo.nx", mode="r") as h5f:
    dataset = h5f["entry0000/instrument/detector/data"]
    print("dataset is virtual:", dataset.is_virtual)
    for vs_info in dataset.virtual_sources():
        print("file name is", vs_info.file_name)
        assert vs_info.file_name.startswith("./")
    

tip

• you can also provide a list of `h5py.VirtualSource` to the `detector.data` attribute.
• To append frame to an existing dataset you can also use the FrameAppender class from nxtomo directly.

Advanced use cases

provide NXtransformations to NXdetector (detector flip, rotation, translation...)

Detector can have image flip (up-down / left-right) created by the acquisition (BLISS-tango) but also some manual flips (rotation along an axis most likely).

To specify those the NXdetector has a TRANSFORMATIONS group defining the transformation chain to be applied. As of today (2023) only image flip are taking into account by nabu (for stitching).

To provide such transformations you can provide a set of transformations like:

from nxtomo import NXtomo
my_nxtomo = NXtomo()

from nxtomo.utils.transformation import Transformation
my_nxtomo.instrument.detector.transformations.add_transformation(
    Transformation(
        axis_name="rx",    # axis name must be unique
        transformation_type="rotation",
        value=180,         # default unit for rotation is 'degree'
        vector=(1, 0, 0),  # warning: transformation are provided as (x, y, z) which is different of the usual numpy ref used (z, y, x)
    )
)

There is several utils class to provide directly detector up-down / left-right flips, basic transformation axis... Please consider using them

from nxtomo.utils.transformation import Transformation, UDDetTransformation, LRDetTransformation, TransformationAxis, TransformationType
from nxtomo.nxobject.nxtransformations import NXtransformations

nx_transformations = NXtransformations()
nx_transformations.transformations = (
    UDDetTransformation(),                                  # vertical flip of the detector
    LRDetTransformation(depends_on="ry"),                   # horizontal flip of the detector. Applied after the vertical flip
    Transformation(                                         # some translation over x axis. Applied after the horizontal flip
        axis_name="tx",
        value=0.02,                                         # value can be a scalar - static value - of an array of value (one per frame expected)
        transformation_type=TransformationType.TRANSLATION, # default unit for translation is SI 'meter'
        depends_on="rz",                                    #  Applied after the horizontal flip in the transformation chain
        vector=TransformationAxis.AXIS_X,
    ),
)
my_nxtomo.instrument.detector.transformations = nx_transformations

NXtransformations and NXsample

As of today the NXtomo application is using a set of dataset ('x_translation', 'y_translation', 'z_translation', 'rotation_angle' to determine the sample transformation). It might evolve in the future to the benefit of an NXtransformations. But this is not implemented at the moment. So please use the 'original' datasets for transformations..