Reduced Darks and Flats

To perform the (optional) flat field correction, you must associate an NXtomo (or Spec‑EDF) with its corresponding reduced darks and flats.

• Reduced darks: usually the mean of dark frames
• Reduced flats: usually the median of flat frames

Reduced Dark and Flat Field Widget

This widget computes the reduction of darks and flats. It should be defined at the beginning of the workflow to ensure that reduced frames are available as soon as possible (for Nabu and for displaying normalized projections, among other uses).

Reduced darks and flats are saved in files named {dataset_prefix}_darks.hdf5 and {dataset_prefix}_flats.hdf5. You can also create them directly from Python.

from IPython.display import Video

Video("video/part_3_dark_flat/reduced_darks_flats_widget.mp4", embed=True, height=500)

Here is a screenshot of a reduced flats file containing a single series of flats at the beginning

---

Copying Darks and Flats

In some cases, users may need to reuse reduced darks or flats. There are several methods to achieve this:

• Method 1: Manually setting reduced darks and flats

  • Use case: Offline processing

• Method 2: Activate copy and "auto" mode

  • Use case: Online processing

• Method 3: Generate reduced darks and reduced flats using Python

  • Use case: When unable to produce reduced darks/flats through simpler methods

Method 1: Manually Setting Reduced Darks and Flats

In this scenario, assume you have the following datasets:

• Dataset 'A': Contains reduced darks
• Dataset 'B': Contains reduced flats (note: datasets 'A' and 'B' can be the same)
• Dataset 'C': Does not contain dark or flat frames (or reduced frames)

To retrieve and set the URLs for the reduced darks or flats, you can use one of the following sub-methods:

• Sub-method 1: Use the Object Selector linked to the reduced darks and flats widget (recommended).

  This is demonstrated in the 'examples' → 'darks and flats manipulations' → darks and flats copy – Method 1.

  

• Python Script: Provide the reduced_darks and reduced_flats directly to the widget as a dictionary. (An example is provided in Method 3.)

• Then, select the URLs from the 'reduced darks/flats → copy' tab, uncheck 'auto', and choose the appropriate URL.

Once the reduced frames are defined, you can trigger processing on your dataset that lacks darks or flats.

Warning: The frame shape must be the same between the reduced frames and the projections.

Note 1: We assume that the reduced darks and flats for datasets 'A' and 'B' have already been computed. If not, you can add a 'dark flat' widget upstream.

Note 2: If dataset 'C' already contains dark or flat frames and you wish to ignore them, you can modify the frame key to mark them as invalid.

Video Tutorial for Method 1

from IPython.display import Video

Video("video/part_3_dark_flat/method1.mp4", embed=True, height=500)

Method 2: Activate Copy and "Auto" Mode

If the reduced darks and flats already exist, you can simply use the copy option from the reduced dark and flat widget.

!!! The copy option is activated by default !!!

• Auto mode: When enabled, this mode will cache reduced darks or flats from any dataset it encounters. When it finds a dataset missing darks or flats, it will automatically copy the cached frames to that dataset.

• Manual mode: In this mode, the user can provide a URL to a well-formed HDF5 dataset containing reduced darks and flats.

Note: The cache file for darks and flats is provided at the bottom of the widget. This is useful for verifying that the registration process is proceeding as expected.

Method 3: Create reduced darks and reduced flats from Python

You can create the reduced frames directly from Python using the Tomoscan API. You can either save them directly or provide them to the reduced darks and flats widget.

• Provide reduced darks and flats as a dictionary (the key is the index and the value is a 2D numpy array):

from tomwer.core.scan.hdf5scan import HDF5TomoScan
# from tomwer.core.scan.edfscan import EDFTomoScan  # Same API for EDF or HDF5

scan = HDF5TomoScan(file_path, data_path)
darks = {
    0: numpy.array(...),  # darks at start
}
flats = {
    0: numpy.array(...),  # flats at start
    3000: numpy.array(...),  # flats at end
}

scan.save_reduced_darks(darks)
scan.save_reduced_flats(flats)
# To provide information about 'machine_electric_current' for normalization, you must also provide `darks_infos` and `flats_infos`

• Provide reduced darks and flats from an already existing HDF5 dataset:

from tomoscan.esrf.scan.utils import copy_darks_to, copy_flats_to
import os
import numpy
from dicttoh5 import dicttoh5
from some_module import DataUrl  # Replace with the correct import

# Create darks and flats as numpy arrays
darks = {
    0: numpy.ones((100, 100), dtype=numpy.float32),
}
flats = {
    1: numpy.ones((100, 100), dtype=numpy.float32) * 2.0,
    100: numpy.ones((100, 100), dtype=numpy.float32) * 2.0,
}

original_dark_flat_file = os.path.join(tmp_path, "originals.hdf5")

dicttoh5(darks, h5file=original_dark_flat_file, h5path="darks", mode="a")
dicttoh5(flats, h5file=original_dark_flat_file, h5path="flats", mode="a")

# Create darks and flats URLs
darks_url = DataUrl(
    file_path=original_dark_flat_file,
    data_path="/darks",
    scheme="silx"
)
flats_url = DataUrl(
    file_path=original_dark_flat_file,
    data_path="/flats",
    scheme="silx"
)

# Apply the copy
scan = HDF5TomoScan(...)
copy_darks_to(scan=scan, darks_url=darks_url, save=True)
copy_flats_to(scan=scan, flats_url=flats_url, save=True)

Hands‑on Exercise A

Copy datasets from /data/projects/tomo-sample-data/training/part3_dark_flat/WGN_01_0000_P_110_8128_D_129 to a local folder. This folder contains two datasets:

• dataset_with_darks_flats_frames.nx – contains darks and flats
• dataset_without_darks_flats_frames.nx – does not contain any darks or flats

Reconstruct dataset_without_darks_flats_frames.nx using the method most convenient for you.

Note 1: The COR can be hard to determine; it is typically around -12.3.

Note 2: If the processing is done before the flat copy is complete or if the copy fails, flat field correction will fail. You might encounter the following error:

2023-03-06 16:04:45,967 [ERROR] cannot make flat field correction, flat not found [tomwer.core.scan.scanbase](scanbase.py:177)
2023-03-06 16:04:45,967:ERROR:tomwer.core.scan.scanbase: cannot make flat field correction, flat not found

---

Advanced Use Cases: Computing Reduced Flats from Raw Projections

Sometimes you have a BLISS dataset containing projections that need to be used as flats to compute reduced flats (dataset 1), and these reduced flats must then be used by other datasets (dataset 2, etc.).

In this case, you can perform the following actions:

• Preprocessing (computing reduced flats from dataset 1):
  • Convert dataset 1 NXtomo image_key projections to flats (using the image-key-editor or image-key-upgrader widget).
  • Compute reduced flats from these raw flats (using the reduced dark and flat widget).
  • Copy these reduced flats to dataset 2 (by providing a URL to the reduced flats or by setting them directly via the reduced flats input as shown in the video).
• Processing (reconstructing dataset 2):
  • Compute reduced darks (from raw) and copy reduced flats from dataset 1 using the reduced dark and flat widget (the reduced flats should have been set during preprocessing).
  • Then create the workflow you want to process (using the default center of rotation, nabu slice, and data viewer widgets as shown in the video).

from IPython.display import YouTubeVideo

YouTubeVideo("vJOo0rHHUYk")