Ewoks Tutorial 2 - Launching a Workflow Using ewoks submit#

Sometimes it can be convenient to trigger a workflow over Slurm to run some processing.

Requirements#

  • This tutorial assumes you have already created your workflow file (.ows). Since we will submit the entire workflow to Slurm, there should be no ‘direct’ Slurm job submission in this workflow (i.e., no ‘Slurm cluster’ widget).

  • We also assume you already have a valid Slurm access token. If not, you can create one with the scontrol token lifespan=xxxx command, like:

    ../_images/slurm_token.png

For more information, here is the official Confluence page about Slurm REST.

Submit a (Tomo) Job Over Slurm#

In this example, we will use the simple slice reconstruction workflow. For now, no dataset is defined within the workflow. Only the widget settings are defined (such as the algorithm to be used for the center of rotation, Paganin value, etc.).

  • Create a client on a node of the cluster (on a node with GPU).

module load ewoks
slurm-token-env

Then trigger the worker.

You will need a message broker.

Create a config file for ewoksjob and provide the URI:

export EWOKS_CONFIG_URI=config

Then run the worker:

ewoksjob worker --pool=slurm -Q slurm -n slurm@payno \
    --slurm-url=http://slurm-api.esrf.fr:6820 \
    --slurm-user=payno \
    --slurm-token=eyJhbGciOiJIUzI1NiIsInR5cCI6[.....]cQRf1VSdNsO1WyR19jlzoN1dLtGzD7SdwTQNcrZHPmM \
    --slurm-pre-script='module load tomotools/stable' \
    --slurm-log-directory='/tmp_14_days/payno/slurm_logs' \
    -sp time_limit=240 \
    -sp current_working_directory=/tmp_14_days/payno/test_workflow_submission