{ "cells": [ { "cell_type": "markdown", "id": "9acfb438-b43c-4042-a9eb-982da0528bff", "metadata": {}, "source": [ "# edit an NXtomo\n", "\n", "The general pipeline to edit an NXtomo is:\n", "\n", "```\n", "load it from disk -> modify it 'in memory' -> save it to disk\n", "```\n", "In this example we will edit the `dummy_nxtomo.nx` file. which is the outcome of the 'create_from_scratch' tutorial" ] }, { "cell_type": "code", "execution_count": null, "id": "4ccc9c2c-cacf-423d-aa44-c8958c9b8f57", "metadata": {}, "outputs": [], "source": [ "import os\n", "from nxtomo import NXtomo\n", "\n", "nx_tomo_file_path = os.path.join(\"resources\", \"dummy_nxtomo.nx\")\n", "nx_tomo = NXtomo().load(nx_tomo_file_path, \"entry\", detector_data_as=\"as_numpy_array\")\n", "print(\"nx_tomo type is\", type(nx_tomo))\n", "print(\"nx_tomo energy is\", nx_tomo.energy)" ] }, { "cell_type": "markdown", "id": "5978b2db-900f-4b46-ae6b-28d8d4c01958", "metadata": {}, "source": [ "Then you can modify your values as it was presented previously and overwrite the file." ] }, { "cell_type": "code", "execution_count": null, "id": "b35f9b9b-d80d-4b51-814c-bb829d7d6b94", "metadata": {}, "outputs": [], "source": [ "nx_tomo.energy = 13.6\n", "nx_tomo.save(\n", " file_path=nx_tomo_file_path,\n", " data_path=\"entry\",\n", " overwrite=True,\n", ")\n", "print(\"new energy is\", NXtomo().load(nx_tomo_file_path, \"entry\").energy)" ] }, { "cell_type": "markdown", "id": "2c6ace38-4ab9-4efb-961c-ec7afb6b4751", "metadata": {}, "source": [ "
\n", "
\n", " \n", "
\n", " The detector data is usually saved as a h5py virtual dataset. The amount of data assocaited can be heavy according to the acquisition and to the available memory. In order to allow a 'smooth' edition detector data can be load according to several strategies:\n", "
    \n", "
  • \"as_data_url\" (default): in this case each Virtual Source will be saved as a DataUrl in order to ease it handling (see later on the tutorial)
    • \n", "
  • \"as_virtual_source\": retrieve original VirtualSource to allow edition of it
    • \n", "
  • \"as_numpy_array\": load all data in memory in order to modify it (and will dump the entire data). to avoid in case of \"real\" dataset. Can trigger huge IO.
    • \n", "
\n", "
\n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "55a76aa3-6053-4682-ac5c-e08fc4e5bd55", "metadata": {}, "source": [ "#### clean" ] }, { "cell_type": "code", "execution_count": null, "id": "8395a7cf-6f41-4fb6-9c7a-bb181f7aa32d", "metadata": {}, "outputs": [], "source": [ "if os.path.exists(nx_tomo_file_path):\n", " os.remove(nx_tomo_file_path)\n", "if os.path.exists(\"nxtomo_reconstruction.hdf5\"):\n", " os.remove(\"nxtomo_reconstruction.hdf5\")" ] }, { "cell_type": "markdown", "id": "ab38514e-ed51-4760-8786-080fb546622a", "metadata": {}, "source": [ "## Advance usage: Provide DataUrl to instrument.detector.data\n", "The issue of using NXtomo we presented above is that the memory to handle data can be used (if you have a large number of projections and / or large detector).\n", "\n", "The way to work around for now it to use provide DataUrl (that can be pointing to an external file). Then this is the job to the FrameAppender to handle those.\n", "\n", "For this example we will first save metadata to the hdf5 (and maybe some frame) then you can append to the dataset series of frames sequentially with there rotation angle, image key..." ] }, { "cell_type": "markdown", "id": "3bf3b4af-0fcd-4cd4-8482-ac191c4db45a", "metadata": {}, "source": [ "### create some dataset to external files\n", "\n", "Here we simply create datasets on external files and record the DataUrls" ] }, { "cell_type": "markdown", "id": "179b77d7-50bd-4656-a93f-94d6384cf7bd", "metadata": {}, "source": [ "
\n", " \n", "
\n", " those datasets must be 3D otherwise the virtual dataset creation will fail.\n", "
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "b726786a-93de-4ae1-82b9-82ba7da3a5da", "metadata": {}, "outputs": [], "source": [ "import h5py\n", "import numpy\n", "from silx.io.url import DataUrl\n", "from nxtomo.utils.frameappender import FrameAppender\n", "\n", "\n", "detector_data_urls = []\n", "for i_file in range(5):\n", " os.makedirs(\"output/external_files\", exist_ok=True)\n", " external_file = os.path.join(f\"output/external_files/file_{i_file}.nx\")\n", " with h5py.File(external_file, mode=\"w\") as h5f:\n", " h5f[\"data\"] = numpy.arange(\n", " start=(5 * 100 * 100 * i_file),\n", " stop=(5 * 100 * 100 * (i_file + 1))\n", " ).reshape([5, 100, 100]) # of course here this is most likely that you will load data from another file\n", "\n", " detector_data_urls.append(\n", " DataUrl(\n", " file_path=external_file,\n", " data_path=\"data\",\n", " scheme=\"silx\",\n", " )\n", " )\n" ] }, { "cell_type": "markdown", "id": "3f8200bb-ae51-419c-b8db-13690835e6c9", "metadata": {}, "source": [ "### create a simple nxtomo but this time provide the list of DataUrl to the instrument.detector.data attribute" ] }, { "cell_type": "code", "execution_count": null, "id": "0191a47b-10a5-4af3-b639-1608e0fd2c29", "metadata": {}, "outputs": [], "source": [ "my_large_nxtomo = NXtomo()" ] }, { "cell_type": "markdown", "id": "9df7b9a8-e5e4-4ffe-bebf-e83cf5573de3", "metadata": {}, "source": [ "provide all information at the exception of frames. Here lets say we will have a dataset with only 180 projections" ] }, { "cell_type": "code", "execution_count": null, "id": "d219ec05-b8a4-49c9-add2-cc4f15e4f7df", "metadata": {}, "outputs": [], "source": [ "my_large_nxtomo.instrument.detector.distance = 0.2\n", "my_large_nxtomo.instrument.detector.x_pixel_size = my_large_nxtomo.instrument.detector.y_pixel_size = 1e-7\n", "my_large_nxtomo.energy = 12.3\n", "# ...\n", "my_large_nxtomo.sample.rotation_angle = numpy.linspace(0, 180, 180, endpoint=False)\n", "my_large_nxtomo.instrument.detector.image_key_control = [0] * 180 # 0 == Projection" ] }, { "cell_type": "markdown", "id": "c4e5f8fc-ca0c-47dc-a0e1-1067fa1e2a1f", "metadata": {}, "source": [ "provide the list of DataUrl to `instrument.detector.data`" ] }, { "cell_type": "code", "execution_count": null, "id": "cda2cbed-2b0d-48d5-bab8-272fc8e0b80c", "metadata": {}, "outputs": [], "source": [ "my_large_nxtomo.instrument.detector.data = detector_data_urls" ] }, { "cell_type": "code", "execution_count": null, "id": "6c853847-0158-4fc0-82d6-2eb01129d043", "metadata": {}, "outputs": [], "source": [ "os.makedirs(\"output\", exist_ok=True)\n", "my_large_nxtomo.save(\"output/my_large_nxtomo.nx\", data_path=\"entry0000\", overwrite=True)" ] }, { "cell_type": "markdown", "id": "559875fd-637e-46df-86e0-7d442da5e50d", "metadata": {}, "source": [ "
\n", " \n", "
\n", " this will create a virtual dataset under 'instrument/detector/data' containing relative links from \"my_large_nxtomo.nx\" to other files.\n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "bfae56a3-60a0-4d07-b7ec-1a505e3d1bcd", "metadata": {}, "source": [ "Then you can see that the 'data' dataset now contains 180 frames (if you run several time the previous cell then it will continue appending data to it).\n", "\n", "If an url is provided instead of a numpy array then it will create be used to create a virtual dataset and avoid duplicating data. But be carreful in this case you must keep relative position of the two files.\n", "\n", "**append frames must have the same dimensions otherwise the operation will fail**" ] }, { "cell_type": "code", "execution_count": null, "id": "5bd78498-59d3-45b9-88b9-7c9a80346f30", "metadata": {}, "outputs": [], "source": [ "from h5glance import H5Glance\n", "H5Glance(\"output/my_large_nxtomo.nx\")" ] }, { "cell_type": "markdown", "id": "f9e885eb-111e-40b8-9655-7e1a7ef391f2", "metadata": {}, "source": [ "check path of VirtualSources are relative (must start with './' string):" ] }, { "cell_type": "code", "execution_count": null, "id": "9ca2c9d9-5881-4fa6-ac39-47394d44d7b3", "metadata": { "tags": [] }, "outputs": [], "source": [ "with h5py.File(\"output/my_large_nxtomo.nx\", mode=\"r\") as h5f:\n", " dataset = h5f[\"entry0000/instrument/detector/data\"]\n", " print(\"dataset is virtual:\", dataset.is_virtual)\n", " for vs_info in dataset.virtual_sources():\n", " print(\"file name is\", vs_info.file_name)\n", " assert vs_info.file_name.startswith(\"./\")\n", " " ] }, { "cell_type": "markdown", "id": "561b423a-3e00-4004-a5b9-7d9a474ba6d1", "metadata": {}, "source": [ "
\n", " \n", "
\n", " tip
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "4f0b1b10-b3b8-488f-89f5-24fae10f91d5", "metadata": {}, "source": [ "## Advanced use cases" ] }, { "cell_type": "markdown", "id": "fd246a40-88a0-452d-b327-52145b9fddde", "metadata": {}, "source": [ "### provide NXtransformations to NXdetector (detector flip, rotation, translation...)\n", "\n", "Detector can have image flip (up-down / left-right) created by the acquisition (BLISS-tango) but also some manual flips (rotation along an axis most likely).\n", "\n", "To specify those the `NXdetector` has a `TRANSFORMATIONS` group defining the transformation chain to be applied.\n", "As of today (2023) only image flip are taking into account by nabu (for stitching).\n", "\n", "To provide such transformations you can provide a set of transformations like:" ] }, { "cell_type": "code", "execution_count": null, "id": "f77ede82-0c75-4a8f-b428-3813f8f389d2", "metadata": { "tags": [] }, "outputs": [], "source": [ "from nxtomo import NXtomo\n", "my_nxtomo = NXtomo()" ] }, { "cell_type": "code", "execution_count": null, "id": "5fd411dc-f337-4c50-8231-c0d73b01fef0", "metadata": { "tags": [] }, "outputs": [], "source": [ "from nxtomo.utils.transformation import Transformation\n", "my_nxtomo.instrument.detector.transformations.add_transformation(\n", " Transformation(\n", " axis_name=\"rx\", # axis name must be unique\n", " transformation_type=\"rotation\",\n", " value=180, # default unit for rotation is 'degree'\n", " vector=(1, 0, 0), # warning: transformation are provided as (x, y, z) which is different of the usual numpy ref used (z, y, x)\n", " )\n", ")" ] }, { "cell_type": "markdown", "id": "aaa28d86-8a32-4f8a-a073-90e857ed6fb3", "metadata": {}, "source": [ "There is several utils class to provide directly detector up-down / left-right flips, basic transformation axis...\n", "Please consider using them" ] }, { "cell_type": "code", "execution_count": null, "id": "77832746-948b-4e97-8fab-4bb8e71c607a", "metadata": { "tags": [] }, "outputs": [], "source": [ "from nxtomo.utils.transformation import Transformation, DetYFlipTransformation, DetZFlipTransformation, TransformationAxis, TransformationType\n", "from nxtomo.nxobject.nxtransformations import NXtransformations\n", "\n", "nx_transformations = NXtransformations()\n", "nx_transformations.transformations = (\n", " DetYFlipTransformation(flip=True), # vertical flip of the detector\n", " DetZFlipTransformation(flip=True, depends_on=\"ry\"), # horizontal flip of the detector. Applied after the vertical flip\n", " Transformation( # some translation over x axis. Applied after the horizontal flip\n", " axis_name=\"tx\",\n", " value=0.02, # value can be a scalar - static value - of an array of value (one per frame expected)\n", " transformation_type=TransformationType.TRANSLATION, # default unit for translation is SI 'meter'\n", " depends_on=\"rz\", # Applied after the horizontal flip in the transformation chain\n", " vector=TransformationAxis.AXIS_X,\n", " ),\n", ")\n", "my_nxtomo.instrument.detector.transformations = nx_transformations\n" ] }, { "cell_type": "markdown", "id": "e818a401-a3ee-472a-adb2-0f2e11c1d3cb", "metadata": {}, "source": [ "
\n", " \n", " NXtransformations and NXsample\n", "
\n", " As of today the NXtomo application is using a set of dataset ('x_translation', 'y_translation', 'z_translation', 'rotation_angle' to determine the sample transformation). It might evolve in the future to the benefit of an NXtransformations. But this is not implemented at the moment. So please use the 'original' datasets for transformations..\n", "
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "0be99b25-e62e-4194-a10a-e935eeaf8b68", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.2" } }, "nbformat": 4, "nbformat_minor": 5 }