This demo is part of A Example Pipeline for Unattended Pipelines, Data Transfer, and Computation Using Globus. This demo should be watched after Configure a Globus Compute Endpoint.

In this example, we see…

• The Compute Endpoint is still running, and we have its UUID (b72ee151-a5d9-4cfc-97b3-86e0664730b2).

• We remembered the UUID of the Compute Function that we registered (it’s e381b154-abe5-48d2-956f-4c61ee02adf0).

• On the cluster, we are storing files temporarily at path /ruthm/akkornel on the SRCC SCG Lab Storage Collection. The Collection’s UUID is part of the Flow Definition, and so is not specified on the command line.

• We are sending the results to a path in Karl’s home directory in the SRCC SCG Home Collection. The Collection’s UUID is 2e23906b-0608-45bb-b344-393b8706e862.

All that’s left is to run the script! The script logs us in to Globus, validates the information we provided, asks for final confirmation (and a label for the Run), and then runs the Flow.

Once the Flow run is submitted, the script shows us the progress of the Run’s progression through each state of the Flow. (Read the section on The Flow Definition for more information about the Flow’s states.)

Once the Run is complete, the script exits, and we can see the checksums file has been uploaded.

Meanwhile, monitoring the SLURM queue shows the Globus Compute job in queue and running. You can see this by opening Run the Flow: SLURM job. If you are watching both demos at the same time, wait until timestamp 2:24 before starting the SLURM demo, then both should be (roughly) in sync.

Here are the commands to run, to reproduce the demo in your own session:

NOTE: On SCG, Python versions are provided through Lmod modules. Every example assumes that you have already loaded the python/3.11.1 module.

# If needed, activate the venv
cd globus_compute_endpoint
. bin/activate
which python
python -V

# Download the script that will submit the Flow run
# Also, install the Python packages needed by the script
pip install globus-sdk click
curl -L -o run_and_monitor.py https://github.com/stanford-rc/globus-example-flow/raw/refs/heads/flow/run_and_monitor.py

# Make a directory to hold the results of the Compute Function
mkdir results

# Confirm the Compute Endpoint is still running
globus-compute-endpoint --config-dir /home/akkornel/globus_compute_demo/config list 

# Run the Flow, and monitor the Run.  Note that you need to provide:
# * A path on SCG Lab storage, to hold the data temporarily
# * A Collecton UUID and path to receive the results
# * The UUID of your Compute Endpoint
# * The UUID of your Compute Function
python run_and_monitor.py
--cluster-temp-path /ruthm/akkornel/ \
--destination-uuid 2e23906b-0608-45bb-b344-393b8706e862 \
--destination-path /akkornel/globus_compute_endpoint/results/ \
b72ee151-a5d9-4cfc-97b3-86e0664730b2 \
e381b154-abe5-48d2-956f-4c61ee02adf0  

# Go into the results directory, and confirm the Run created a directory
cd results
ls

# Go into the directory created by the Run
cd karl-demo*

# Confirm we have a checksums file, and examine it
ls
head checksums.txt

This is the last demo in the series.