{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Volumes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the aim of classes dedicated to volumes is to have a common way of saving and loading volumes through the tomotools suite.\n",
    "From it we intend to simplify volume handling for creating them but also to do some post processing operation on them such as stitching but also ease handling from gui.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Volumes contains two main parts:\n",
    "* `data`: the volume itself. It is expected to be a 3D numpy array\n",
    "* `metadata`: metadata associated to the volume such as information regarding the reconstruction parameters..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now the following volumes exists:\n",
    "   \n",
    "* HDF5Volume: save data and metadata to an hdf5 file.\n",
    "* EDFVolume: save data to single frame EDF files and metadata to a text file\n",
    "* JP2KVolume: save data to single frame jp2k files and metadata to a text file\n",
    "* TIFFVolume: save data to single frame tiff files and metadata to a text file\n",
    "* MultiTIFFVolume: save data to a single 3D tiff file and metadata to a text file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Volume API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tomoscan.esrf import HDF5Volume\n",
    "from tomoscan.esrf import EDFVolume\n",
    "from tomoscan.esrf import TIFFVolume, MultiTIFFVolume, has_tifffile\n",
    "from tomoscan.esrf import JP2KVolume, has_glymur\n",
    "import numpy\n",
    "from silx.io.url import DataUrl\n",
    "from tempfile import TemporaryDirectory\n",
    "import os\n",
    "from h5glance import H5Glance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### create some dataset to test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tomoscan.esrf.volume.mock import create_volume\n",
    "\n",
    "data = create_volume(frame_dims=(100, 100), z_size=5)\n",
    "metadata = {\n",
    "    \"reconstruction_params\": {\n",
    "        \"dataset\": {\n",
    "            \"location\": \"toto.hdf5\",\n",
    "            \"entry\": \"entry0000\",\n",
    "        },\n",
    "        \"phase\": {\n",
    "            \"method\": \"None\",\n",
    "            \"padding_type\": \"edge\",\n",
    "        },\n",
    "    },\n",
    "    \"processing_option\": {\n",
    "        \"build_sino\": {\n",
    "            \"axis_correction\": \"None\",\n",
    "            \"enable_halftomo\": True,\n",
    "        },\n",
    "        \"flatfield\": {\n",
    "            \"binning\": \"11\",\n",
    "            \"do_flat_distortion\": False,\n",
    "        },\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pylab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imshow(data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HDF5Volume\n",
    "#### constructor\n",
    "Here we will only focus on providing a `file_path` and a `data_path` to the constructor. From it it will deduce default data and metadata `data_path`.\n",
    "\n",
    "This way allow us to get a identifier that can be reused. But users can also provide directly an url for data and one for the metadata. But this is for advance usage and it will not be detailled here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# users can define the volume will all the information\n",
    "volume = HDF5Volume(file_path=\"test_volume.hdf5\", data_path=\"entry0000/reconstruction\", data=data, metadata=metadata)\n",
    "# or define them from properties\n",
    "volume = HDF5Volume()\n",
    "volume.file_path = \"test_volume.hdf5\"\n",
    "volume.data_path = \"entry0000/reconstruction\"\n",
    "volume.data = data  # data must be None or 3D numpy array\n",
    "volume.metadata = metadata  # metadata must be None or an instance of dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data can be access from the data property\n",
    "# imshow(volume.data[0])\n",
    "# metadata from the metadata property.\n",
    "volume.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### saving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# save the volume\n",
    "volume.save()\n",
    "# display the contents of  the volume file\n",
    "\n",
    "H5Glance(\"test_volume.hdf5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### loadding\n",
    "once the data is saved on disk then we can retrieve it from the same class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "volume_loaded = HDF5Volume(file_path=\"test_volume.hdf5\", data_path=\"entry0000/reconstruction\")\n",
    "volume_loaded.load()\n",
    "# imshow(volume_loaded.data[0])\n",
    "print(volume_loaded.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To avoid heavy memory consumption the data and metadata will not be loaded automatically.\n",
    "In order to purge cache you can also call `volume.clear_cache()` or set data and metadata to None like:\n",
    "```\n",
    "volume.data = None\n",
    "volume.metadata = None\n",
    "```\n",
    "Note: if data or metadata is None call to saving function can raise ValueError exception."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### identifier\n",
    "Each subclass of `VolumeBase` define what we call an identifier. The goal is that from this identifier a user can retrieve a Volume.\n",
    "An identifier can be save as an instance of `Identifier` or as it string representation.\n",
    "An identifier looks like:\n",
    "\n",
    "```\n",
    "scheme:tomo_type:data_path@data_file\n",
    "```\n",
    "For now:\n",
    "* scheme can be `hdf5`, `edf`, `jp2k`, `tiff` and `tiff_3d`\n",
    "* tomo_type can be `scan` or `volume`\n",
    "\n",
    "*Note: This is pretty convienient for gui for example or to 'copy/paste' a reference to a volume from one application to another.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### get identifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "identifier = volume_loaded.get_identifier()\n",
    "print(identifier, type(identifier))\n",
    "print(identifier.to_str(), type(identifier.to_str()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### use identifier to retrieve a volume"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tomoscan.factory import Factory\n",
    "retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier)\n",
    "assert isinstance(retrieve_volume, HDF5Volume)\n",
    "retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier.to_str())\n",
    "assert isinstance(retrieve_volume, HDF5Volume)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clean workspace\n",
    "if os.path.exists(\"test_volume.hdf5\"):\n",
    "    os.remove(\"test_volume.hdf5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EDFVolume\n",
    "#### constructor\n",
    "\n",
    "EDFVolume will store each frame as `folder_prefix_index.edf` and metadata to `folder_prefix_infos.txt`.\n",
    "\n",
    "So the default constructor only require a path to a folder where to save the data.\n",
    "\n",
    "This folder path will also be the identifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "volume = EDFVolume(folder=\"edf_volume\", data=data, metadata=metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or as for the HDF5Volume you can provide data and metadata once the Volume has been defined"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "volume = EDFVolume(folder=\"edf_volume\")\n",
    "volume.data = data\n",
    "volume.metadata = metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "access to data and metadata is the same for any instance of VolumeBase, from `data` and `metadata` properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imshow(volume.data[0])\n",
    "print(volume.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### saving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the volume\n",
    "volume.save()\n",
    "\n",
    "# if you want to overwrite existing data you can set the overwrite property\n",
    "volume.overwrite = True\n",
    "\n",
    "# display the contents of the folder\n",
    "print(os.listdir(\"edf_volume\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# open one file containing a frame\n",
    "import fabio\n",
    "file_frame_0 = os.path.join(\"edf_volume\", \"edf_volume_000000.edf\")\n",
    "# imshow(fabio.open(file_frame_0).data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display contents of the infos.txt file\n",
    "metadata_file = os.path.join(\"edf_volume\", \"edf_volume_infos.txt\")\n",
    "with open(metadata_file, \"r\") as of:\n",
    "    print(\"\".join(of.readlines()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "volume_loaded = EDFVolume(folder=\"edf_volume\")\n",
    "volume_loaded.load()\n",
    "#imshow(volume_loaded.data[1])\n",
    "print(volume_loaded.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### identifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### get_identifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "identifier = volume_loaded.get_identifier()\n",
    "print(identifier, type(identifier))\n",
    "print(identifier.to_str(), type(identifier.to_str()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### use identifier to retrieve a volume"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tomoscan.factory import Factory\n",
    "retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier)\n",
    "assert isinstance(retrieve_volume, EDFVolume)\n",
    "retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier.to_str())\n",
    "assert isinstance(retrieve_volume, EDFVolume)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clean\n",
    "import shutil\n",
    "if os.path.exists(\"edf_volume\"):\n",
    "    shutil.rmtree(\"edf_volume\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JP2KVolume\n",
    "\n",
    "The API of `JP2KVolume` is the same as `EDFVolume`. So for tutorial please have a look at this one.\n",
    "\n",
    "It requires `glymur` to be install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TIFFVolume\n",
    "\n",
    "The API of `TIFFVolume` is the same as `EDFVolume`. So for tutorial please have a look at this one.\n",
    "\n",
    "It requires `tifffile to be install`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MultiTIFFVolume\n",
    "\n",
    "The MultiTIFFVolume requires `tifffile` to be install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### constructor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "volume = MultiTIFFVolume(file_path=\"multitiff_file.tiff\", data=data, metadata=metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# access data and metadata\n",
    "# imshow(volume.data[0])\n",
    "print(volume.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### saving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tomoscan.esrf.volume.tiffvolume import has_tifffile\n",
    "if has_tifffile:\n",
    "    volume.save()\n",
    "else:\n",
    "    print(\"tifffile must be installed to use MultiTIFFVolume\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if has_tifffile:\n",
    "    volume_loaded = MultiTIFFVolume(file_path=\"multitiff_file.tiff\")\n",
    "    assert volume_loaded.data is None\n",
    "    assert volume_loaded.metadata is None\n",
    "    volume_loaded.load()\n",
    "    # imshow(volume_loaded.data[0])\n",
    "    print(volume_loaded.metadata)\n",
    "else:\n",
    "    print(\"tifffile must be installed to use MultiTIFFVolume\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### identifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### get_identifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if has_tifffile:\n",
    "    identifier = volume_loaded.get_identifier()\n",
    "    print(identifier, type(identifier))\n",
    "    print(identifier.to_str(), type(identifier.to_str()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### use identifier to retrieve a volume"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tomoscan.factory import Factory\n",
    "if has_tifffile:\n",
    "    retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier)\n",
    "    assert isinstance(retrieve_volume, MultiTIFFVolume)\n",
    "    retrieve_volume = Factory.create_tomo_object_from_identifier(identifier=identifier.to_str())\n",
    "    assert isinstance(retrieve_volume, MultiTIFFVolume)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clean\n",
    "if os.path.exists(\"multitiff_file.tiff\"):\n",
    "    os.remove(\"multitiff_file.tiff\")\n",
    "if os.path.exists(retrieve_volume.metadata_url.file_path()):\n",
    "    os.remove(retrieve_volume.metadata_url.file_path())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "est_venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  },
  "vscode": {
   "interpreter": {
    "hash": "32d4b0e4dd0d191fdf6b0e78dccee44d0b22a4c0143a249787f4200ca7a7dc85"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}