Converting from BLISS raw data to NXtomo

There is 3 differrent ways to convert a raw file:

• from the Command Line Interface (CLI)
  • provide all options to the CLI
  • providing a configuration file
• from the python API

from the command line interface

nxtomomill h52nx [...]

access help

%%bash
nxtomomill h52nx --help

Note: all tomotools software have --help option

usage example

If we have a bliss file: 'bliss_file.h5' that we want to convert to a NXtomo entry.

nxtomomill h52nx bliss_file.h5 output_file.nx

Once converted you can have a look at the output file and insure the conversion want smoothly:

silx view output_file.nx

For each file it will generate one NXtomo per acquisition (at the exception of `pcotomo`).

tip

• To ease processing by default it will create one file per NXtomo and one master file connecting each NXtomo entry.
• You can ask for a single file using the `--single-file` option.
• All available entries will be converted. If you want to limit it to some entries you can use the `--entries` option.

output file extension can be '.nx', '.nxs', '.h5', '.hdf5' or '.nexus'. We encourage to use '.nx' for 'readibility' and 'simplicity'.

If the output is not provided the default behavior is to create the NXtomo to the PROCESSED_DATA directory when possible. If an output directory is provided then it will picked.

hand on - exercise A

convert bambou_hercules_0001.h5 to bambou_hercules_0001.nx

%%bash
# TODO 1: convert bambou_hercules_0001.h5 to bambou_hercules_0001.nx
# TODO 2: browse the file using silx view

conversion using a configuration file

You can also convert raw data by providing a configuration file.

The configuration file will contain all the details of the conversion to apply.

You can generate a default configuration file by running

nxtomomill h5-config my_configuration_file.cfg

configuration file overview.

The configuration file look like:

The configuration file is split into several sections:

GENERAL_SECTION section

Mandatory section.

Define information regarding I/O and extra information to the acquisition (format, FOV...)

KEYS_SECTION section

Optional section if FRAME_TYPE_SECTION is provided

Define dataset location of the different information to get.

FRAME_TYPE_SECTION section

We will present this section at the end of the notebook. For now it can be ignored / removed

EXTRA_PARAMS_SECTION section

From it you can provide some values for energy or pixel size for example ignoring the existing value (if any)

We will see on the next slides the possible usage of each section.

But for this simple use case we can only provide input_file and output_file in the GENERAL_SECTION and execute it.

And to execute it we can go for:

nxtomomill h52nx --config my_configuration_file.cfg

If you provide a relative path be meticulous regarding your working directory.

hand on - exercise B

create a default configuration file and convert bamboo_hercules_0001.h5 to NXtomo

%%bash
# TODO 1: create a nxtomomill configuration file
# TODO 2: convert `bamboo_hercules_0001.h5` to NXtomo

conversion from the python API

the h52nx command call nxtomomill.converter.from_h5_to_nx function.

To do the same operation from a python script you could have execute

from nxtomomill.converter import from_h5_to_nx
from nxtomomill.io.config import TomoHDF5Config

input_file_path = "bambou_hercules_0001.h5"

configuration = TomoHDF5Config()
configuration.input_file = input_file_path
configuration.output_file = "output_file.nx"

# ...

res = from_h5_to_nx(configuration=configuration)

The TomoHDF5Config is containing the same information as the configuration file.

hand on - exercise C

convert bamboo_hercules_0001.h5 to NXtomo from a python script

What to do if some bliss entry is skipped / unrecognized ?

Currently the deduction of a bliss scan type (dark, flat, projection...) is done by:

• looking at the 'technique' group (image_key dataset)
• else look at the title. Title mapping is defined in settings.py file of nxtomomill.

If the titles have a specific naming convention then you can provide updated information to one of the following:

• modify it from the settings.py file (if this is a local installation
• provide different name to be used
  • from the CLI
  • from a configuration file (see later)
  • from the python API

You can 'ignore' the bliss 'technique' dataset (aka bliss tomo config) by using a configuration and turning the `ignore_bliss_tomo_config` to True

from the CLI without a configuration file

If you look at the help you can see how to redefine title names.

nxtomomill h52nx --help

• --init_titles: mark the beginning of a Bliss sequence (eq acquisition). Use for example to retrieve energy.
• --dark_titles: specify that this Bliss entry is relative to dark field
• --flat_titles: specify that this Bliss entry is relative to flat field
• --proj_titles: specify that this Bliss entry is relative to projection
• --align_titles: specify that this Bliss entry is relative to alignment (some time called 'return')
• --init_zserie_titles, --init_pcotomo_titles same as init-titles but dedicated to zseries and pcotomo (behavior of NXtomo creation is a bit different)

from the CLI with the configuration file

The same information can be provided to ENTRIES_AND_TITLES_SECTION section

from the python API

you can also provide this information to the TomoHDF5Config class like:

configuration = TomoHDF5Config()
configuration.init_titles = ("mytomo:basic", "mytomo:fullturn")

hand on - exercise D

the bambou_hercules_0001_different_titles.h5 file is the same as bambou_hercules_0001.h5 but with different titles.

Convert this file to an NXtomo. Depending on how you feel you can do it fully from the CLI, use a configuration file or do it from python

%%bash
# TODO: convert bambou_hercules_0001_exotic_rotation_location.h5

What to do if there is some mandatory information missing (energy, distance...)

from the CLI without a configuration file

There is a limited number of information that the user can provide manually like energy or pixel size. Those can be provided from the set-params option like:

In this case you can provide it from the --set-params option from the CLI like:

nxtomomill h52nx ... --set-params energy 0.5

the `--set-params` option should always be put at the end of the command. Because it can take a full list of sub-options

from the CLI with a configuration file

You can also provide this information to the configuration file under the EXTRA_PARAMS_SECTION section like:

from the python API

Or provide this from a python script when defining the configuration:

configuration = TomoHDF5Config()
configuration.param_already_defined = {
    "energy_kev": 19.2,
}

What to do if a field cannot be found

For some limited fields ("detector name", "translation_x", "translation_y", "translation_z" and "rotation") we try to read this information from the 'technique' dataset. Else we fallback on the generic behavior.

The generic behavior is to look for each field to a set of pre-defined location / path and pick it from there if the dataset shape fit the expectation.

You can tune those locations (as for titles) from.

• settings.py file of nxtomomill (if this is your own installation).
• command line options
• configuration file

from the CLI without a configuration file

For the CLI we can get them using ̀nxtomomill h52nx --help again:

• --x_trans_keys: x translation key in bliss HDF5 file.
• --y_trans_keys: y translation key in bliss HDF5 file.
• --z_trans_keys: z translation key in bliss HDF5 file.
• --sample-detector-distance: sample detector distance.
• --valid_camera_names: Valid NXDetector dataset name to be considered. If None will try to deduce the NXdetector from attributes and shape of the dataset
• --rot_angle_keys: Valid dataset name for rotation angle. If None, look name from "technique/scan/motor"
• --acq_expo_time_keys: acquisition exposure time in bliss HDF5 file.
• --x_pixel_size_key x pixel size key in bliss HDF5 file.
• --y_pixel_size_key y pixel size key in bliss HDF5 file.

from the CLI with a configuration file

the same field are available on the KEYS_SECTION of the configuration file.

dataset discovery is done as follow: converter will first look at positioner group to resolve key then at root group aka bliss entry. Collect is done by a look like code:

for parse_group in (positioner, root_aka_bliss_entry):
   # first step: look for existing dataset with expected number of elmt
   for key_checked in parse_group:
      if find_dataset_with_expected_nb_elment:
          return it
   # second step: look for existing dataset and adapt it if not enought elmt
   for key_checked in parse_group:
      if find_dataset:
       return it
return None

hand on - exercise E

• browse bambou_hercules_0001_exotic_rot_angle.h5 and locate the dataset containing the rotation angle of the sample
• sucessfully convert bambou_hercules_0001_exotic_rot_angle.h5 by providing the new dataset location either from the CLI or from python.

%%bash
# TODO: convert bambou_hercules_0001_exotic_rotation_location.h5 into a valid NXtomo containing expected rotation angle

Advance usage: define your own sequence to build an NXtomo

design

The conversion process is done as follow:

The first step this can be done two ways

Until now we were only using the title to deduce the acquisition and the type of frames of each Bliss entry.

But the FRAME_TYPE_SECTION allow us to ignore those titles and define manually the sequence of the acquisition.

Coming back to the FRAME_TYPE_SECTION section

FRAME_TYPE_SECTION section

If this section is fill then the ENTRIES_AND_TITLES_SECTION will be ignored. Those are mutually exclusive sections.

From it we can define data_scans that allow us to define a sequence of scan defining an acquisition using url. Like:

data_scans = (
    (frame_type=projections, entry=silx:///path/to/file?/path/to/scan/node,),
    (frame_type=projections, entry=/path_relative_to_file),
)

Here we will create one acquisition from silx:///path/to/file?/path/to/scan/node to be used as a set of projections and /path_relative_to_file as a set of projections to.

Url can be relative to different file

the created acquisition will follow the provided order

example: create an NXtomo using the data_scans field

using the configuration file

You can find a file conversion_using_data_scans.cfg in the solution folder that create an acquisition using data_scans:

It can be executed by calling:

nxtomomill h52nx --config conversion_using_data_scans.cfg

using python directly

from nxtomomill.converter import from_h5_to_nx
from nxtomomill.io.config import TomoHDF5Config
from nxtomomill.io.framegroup import FrameGroup
from silx.io.url import DataUrl

input_file_path = "bambou_hercules_0001.h5"

configuration = TomoHDF5Config()
configuration.input_file = input_file_path
configuration.output_file = "bambou_hercules_0001.nx"
configuration.data_frame_grps = (
    FrameGroup(
        url=DataUrl(
            file_path=input_file_path,
            data_path="1.1",
            scheme="silx",
        ),
        frame_type="initialization",
    ),
    FrameGroup(
        url=DataUrl(
            file_path=input_file_path,
            data_path="2.1",
            scheme="silx",
        ),
        frame_type="darks",
    ),
    FrameGroup(
        url=DataUrl(
            file_path=input_file_path,
            data_path="3.1",
            scheme="silx",
        ),
        frame_type="flats",
    ),
    FrameGroup(
        url=DataUrl(
            file_path=input_file_path,
            data_path="4.1",
            scheme="silx",
        ),
        frame_type="projections",
    ),            
)

res = from_h5_to_nx(configuration=configuration)

note: you will see another way to create an NXtomo from scratch that could be another alternative.

Provide to tomwer a configuration file.

Today there is two widgets that can call nxtomomill: the datalistener and the nxtomomill widgets.

You can define this from the configuration menu

Then select a configuration file

When a configuration file is active then we notify it by a "green point"

You can do the same from the data listener as well:

It can useful to know the version of the application you are using. For nxtomomill you can access it :

• from the command line interface:

nxtomomill --version

• from python

import nxtomomill
nxtomomill.__version__