Data processing roadmap

Here we define preliminary work packages and related milestones that should ideally be reached at the end of two years of development.

List of work packages

Here is a list of the initial work packages for Nabu (WP):

  • Base architecture : infrastructural work that does not enable features "per se", but that supports features from other work packages.
  • Plain fullfield : enabling basic normalizations and corrections in fullfield imaging
  • Induced emission (XRD/XRF-CT) : enabling and fostering multi-channel reconstructions and regularizations, and projector corrections which account for physical corrections (attenuation, solid angles of detection, photon incidence angle on pixels, etc), and sample shape corrections.
  • Phase : enabling reconstructions from phase imaging, including single and multiple distance phase retrieval and ptychography.
  • Prior knowledge : developing advanced methods for solving the following problems: local tomography, missing wedge, and super-resolution.
  • Multi-scale : enabling multi-resolution reconstructions, and handling of the related data-size problem. This work package implies the development of the previous one.
  • Alignment : providing advanced methods for projection alignment.
  • Streamlining : including topics like online, automation and diagnostics. Thus, it deals with: enabling online reconstructions, enabling semi-automatic estimation of certain reconstruction parameters (e.g. rotation axis, etc), and providing useful feedback on the data to the user.

List of features (from all work packages)

Here we list the currently desired features for the computational core (F).

  1. v MR09 - FBP reconstruction (WPa): acts on projections or sinograms (after contrast specific preprocessing), ram-lak filtering (with alternative filters support!) and backprojection.
  2. v MR03 - Simple flat-fielding and absorption correction (log) (WPb).
  3. v MR__ - GPU implementation of the basic full-field pipeline (WPa).
  4. v MR__ - Distributed computation of the basic full-field pipeline (WPa).
  5. v MR__ - Image distortion correction (WPb).
  6. v MR26 - Projection based ring correction (e.g. double flat-fielding) (WPb).
  7. v MR10 - Sinogram based ring correction (WPa).
  8. v MR__ - Volume based ring correction (WPb).
  9. v Paganin filtering, with modifications (WPd) (1 v out of 2):
    • v MR02 - unsharp mask.
    • v deconvolution.
  10. v MR104 - CTF (contrast transfer function) filtering (WPd).
  11. v MR__ - Vertical volume stitching (WPa).
  12. v MR27 - Simple rotation-axis finding algorithms (accurate, manual) (WPa).
  13. v MR33,44 - Sample shifts (WPa/WPg): it could be part of a broader concept like flexible / arbitrary projection geometry. Useful for correcting sample motion (see F22), reducing ring artefacts and for helical scans.
  14. v MR__ - Integration of basic XRD-CT support (WPc): definition and implementation of the basic interfaces for preprocessing with PyFAI.
  15. v MR__ - Integration of basic XRF-CT support (WPc): definition and implementation of the basic interfaces for preprocessing with PyMCA.
  16. v MR__ - Loading/saving compressed data (WPa).
  17. v MR__ - Integration of Holotomography/Ptychography support (WPc): definition and implementation of the basic interfaces for preprocessing with external code (PyNX, Ptypy, etc).
  18. v MR61 - Data conversion (8/16 bits tif/jpeg2000/raw/EDF) with manual and automatic conversion values strategies (per scan / per series / per list of scans or series) (WPa).
  19. v MR61 - Compression of native tomographic data in jpeg2000 format or equivalent for permanent storage (WPa).
  20. v MR39 - Handling half-acquisition (WPb).
  21. v Center-of-rotation (CoR) finding (WPh):
    • v MR27 - On projections of centered axis acquisitions.
    • v MR59,93 - On projections of lateral axis acquisitions (half-acquisitions).
    • v MR93 - On sinogram of centered axis acquisitions.
    • v MR93 - On sinogram of lateral axis acquisitions (half-acquisitions).
  22. v MR__ - Automatic sample shifts finding (WPh).
  23. v Normalization of projection intesities (WPb) (1 v out of 3):
    • v Using a monitor (e.g. diode readings).
    • v Using a fitting polynomial (Chebyshev or lsq spline).
    • v Using a region of the direct beam that never intersects the sample (low priority).
  24. v MR__ - Add DICOM file format support.
  25. v Support for projection rotation (compensate detector tilt) (WPh) (2 v out of 3):
    • v MR102 - Rotate projections in processing pipeline
    • v MR113 - Automatically detect projection tilt (with centered rotation axis)
    • v MR__ - Automatically detect projection tilt (with rotation axis on the side)
  26. v MR__ - Helical scans support.
  27. v MR__ - On-the_fly reconstruction support.

Here we list the currently desired features for the graphical pipeline design system (G).

  1. v Manually specify a smaller number of projections (for a rough quick initial reconstruction) (WPh).
  2. v Show reconstruction parameters (WPh).
  3. v Launch, monitor, delete and re-launch jobs on SLURM (WPh).
  4. v Interactive feedback from processing to acquisition system (WPe,g).

Milestones

Here we list the initial milestones for the development of Nabu, composed by the features indicated here above.

By March 2020 (v2020.1.0):

  • Milestone 0 (M00): v basic FBP reconstruction of a preprocessed dataset. Includes: F01 .

Preview release in early June 2020 (v2020.2.0), and autumn / winter stable releases (v2020.3.0), (v2020.4.0), (v2020.4.1), (v2020.5.0):

  • Milestone 1.1 (M01.1): v Basic pre-processing plain fullfield. Includes: F02 , F06 , F09 , F12 , F13 , F18 , F20 .

By April 2021 (v2021.1.0):

  • Milestone 1.1.5 (M01.1.5): v Extended full-field processing. Includes: F10 , F25 (basic support).

By January 2022 (v2021.2.0) (Tentative, subject to changes!):

  • Milestone 1.2 (M01.2): v Basic pre-processing XRD. Includes: (optional F07 ), F14 .

By March 2022 (Tentative, subject to changes!):

  • Milestone 2 (M02): v GPU implementation of the basic pipeline and distributed computation. Includes: F03 , F27 , (optional F04 ).
  • Milestone 3 (M03): v Interactive feedback from processing to acquisition system (G04).

By October 2022 (Tentative, subject to changes!):

  • Milestone 4 (M04): Pre-processing extended to XRF-CT. Includes: F15 .
  • Milestone 5 (M05): Pre-processing extended to holography and ptychography. Includes: F17 .

By March 2023 (Tentative, subject to changes!):

  • Milestone 6 (M06): Complete integration of the full-field tomography workflow. Includes: F05 , F08 , F11 , F16 , F19 .