{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "287b637a-4a01-4c11-81b8-68404d4c0727",
   "metadata": {},
   "source": [
    "# How to publish reconstructed volume to the data portal?\n",
    "\n",
    "This tutorial explains how to publish a reconstructed volume (done by nabu) to the (ESRF) data portal.\n",
    "\n",
    "Today the ESRF data portal catalog is based on DRAC (successor of ICAT). As switching is fresh, you should consider ICAT == DRAC as both names can be used interchangeably."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dae0f793-0a6b-440c-8bae-d6781b144bad",
   "metadata": {},
   "source": [
    "## DRAC processed dataset\n",
    "\n",
    "There are two types of datasets in DRAC: raw datasets (published automatically by Bliss-tomo), and processed datasets (the ones we will publish in this tutorial)\n",
    "\n",
    "A **DRAC processed dataset** is related to:\n",
    "\n",
    "* one or several raw datasets (usually one, but it can be several in the case of a stitching, for example) (*a*)\n",
    "* a set of metadata keys (voxel size, phase retrieval options, etc.) (*b*)\n",
    "* one beamline (*c*)\n",
    "* one proposal (*d*)\n",
    "* one dataset (*e*)\n",
    "* one folder (the folder containing the reconstructed volume) (*f*)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "868b2c98-0bb3-43de-b0ce-c14daa2fbbd2",
   "metadata": {},
   "source": [
    "## Retrieve all the data needed for ICAT\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7254799-34f7-474e-9674-9a02874cd1ba",
   "metadata": {},
   "source": [
    "### (*a*) `raw` parameter\n",
    "\n",
    "Path to the raw datasets. Source(s) of the processed dataset. It should be a tuple, but it can be a tuple of a single element.\n",
    "\n",
    "You can get the original dataset path from an instance of `TomoScanBaseInstance` by calling `get_bliss_original_files()`.\n",
    "\n",
    "Warning: this path can contain some '/mnt/multipath-shares' prefix that shouldn't be passed to ICAT/DRAC. To filter this you can use the 'from_bliss_original_file_to_raw' helper function.\n",
    "``` python\n",
    "from tomoscan.esrf.scan.utils import from_bliss_original_file_to_raw\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0b764b9-93bd-46f5-bdda-ae93f1a3af40",
   "metadata": {},
   "source": [
    "### (*b*) `metadata` parameter\n",
    "\n",
    "The metadata to be published to ICAT can be obtained from an instance of `VolumeBase` by calling the `build_drac_metadata` function.\n",
    "\n",
    "For example, for an `HDF5Volume` you can have:\n",
    "\n",
    "```\n",
    "volume = HDF5Volume(\n",
    "    file_path=...,\n",
    "    data_path=...,\n",
    ")\n",
    "\n",
    "drac_metadata = volume.build_drac_metadata()\n",
    "```\n",
    "\n",
    "Note: there is a tutorial on volumes for more information.\n",
    "\n",
    "**Warning**: at the moment, the DRAC metadata will not contain the 'Sample_name' field, which is mandatory (without it, there will be no processing done). So you will need to add it.\n",
    "\n",
    "```\n",
    "drac_metadata[\"Sample_name\"] = ...\n",
    "```\n",
    "\n",
    "It can be obtained from the `TomoScanBaseInstance` by calling `scan.sample_name`.\n",
    "\n",
    "*Note*: Available DRAC keys are defined [here](https://gitlab.esrf.fr/icat/hdf5-master-config/-/blob/master/hdf5_cfg.xml?ref_type=heads) (see `Tomo` group, `reconstruction` section)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c10b63a-b52e-4cff-8dea-e3b6f9105f74",
   "metadata": {},
   "source": [
    "### (*c*) `beamline` parameter\n",
    "\n",
    "This is the name of the beamline, like 'bm05', 'bm18'... (in lower case)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ceb617-b64f-46c9-bfa4-8fa888f50b6b",
   "metadata": {},
   "source": [
    "### (*d*) `proposal` parameter\n",
    "\n",
    "Name of the proposal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45a211a4-e203-4517-8edb-1e7515beab01",
   "metadata": {},
   "source": [
    "### (*e*) `dataset` parameter\n",
    "\n",
    "Name of the dataset. This is the (processed) dataset in the DRAC context.\n",
    "\n",
    "This dataset will create a key with the folder path at the DRAC level and it must be unique.\n",
    "\n",
    "The default value we propose is 'reconstructed_volumes'."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e6ce848-cc68-47bf-bb4f-65083bbf789c",
   "metadata": {},
   "source": [
    "### (*f*) `path` parameter\n",
    "\n",
    "This is the path to the folder containing the reconstructed volume (by Nabu).\n",
    "\n",
    "**Warning 1**: path should be cleaned of any 'esrf mounting points' like '/mnt/multipath-shares' or '/gpfs/easy'. If needed you can use the 'filter_esrf_mounting_points' from tomoscan.esrf.scan.utils.\n",
    "\n",
    "**Warning 2**: All files contained in this folder will be published to ICAT. There is no mechanism to publish a single file or a set of files.\n",
    "\n",
    "Here is the recommended structure if path == 'reconstructed_volumes' and for an HDF5 reconstruction:\n",
    "\n",
    "```\n",
    "reconstructed_volumes\n",
    "    |\n",
    "    |------ nabu_rec.hdf5                                - nabu reconstructed volume master file  (1)\n",
    "    |------ nabu_rec \n",
    "    |         |---------- nabu_rec_0000_0256.hdf5        - nabu reconstructed volume sub file 1\n",
    "    |------ gallery                                      - gallery related to the processed dataset (2) \n",
    "    |         |------ screenshot_1.png\n",
    "    |         |------ screenshot_2.png\n",
    "    |------ nabu_cfg_files                               - folder containing nabu configuration files (3)\n",
    "              |------ nabu_config.cfg\n",
    "``` \n",
    "(1) The Nabu reconstructions. It can be replaced by a folder containing a volume with .tiff files.\n",
    "\n",
    "(2) **Optional**. A set of images (.png or .jpg) linked to the reconstructed volume, like 3 slices along each axis.\n",
    "\n",
    "(3) nabu_cfg_files: location of the configuration used to obtain the volume(s). In the future, it should be used to reprocess a volume."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28d6a1e3-2616-4112-a90d-736d6793ddd5",
   "metadata": {},
   "source": [
    "## Publication to DRAC / ICAT\n",
    "\n",
    "To publish a **processed dataset** to ICAT, we use [pyicat_plus](https://gitlab.esrf.fr/icat/pyicat-plus)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eee28397-a05f-4aaa-a748-5bfa2d3e7944",
   "metadata": {},
   "source": [
    "### Instantiate the `IcatClient`\n",
    "\n",
    "``` python\n",
    "from pyicat_plus.client.main import IcatClient \n",
    "icat_client = IcatClient(\n",
    "    metadata_urls=(\"bcu-mq-01.esrf.fr:61613\", \"bcu-mq-02.esrf.fr:61613\")\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "047debaa-bac2-431a-8c99-d1411aa73c71",
   "metadata": {},
   "source": [
    "### Publish to ICAT\n",
    "\n",
    "``` python\n",
    "icat_client.store_processed_data(\n",
    "    raw=raw,  # (a)\n",
    "    metadata=metadata,  # (b)\n",
    "    beamline=\"id16a\",  # (c)\n",
    "    proposal=self.inputs.proposal, # (d)\n",
    "    dataset=\"reconstructed_volumes\",\n",
    "    path=path,\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d4bee56-b109-4f38-9b34-66171c45b661",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}