Configure a Globus Compute Endpoint — A Example Pipeline for Unattended Pipelines, Data Transfer, and Computation Using Globus

by akkornel 7 months ago

GNU/Linux • xterm-256color • bash 194 views

This demo is part of A Example Pipeline for Unattended Pipelines, Data Transfer, and Computation Using Globus. This demo should be watched after Register the Function with Globus Compute.

Now, we configure the Globus Compute Endpoint. This involves telling the Compute Endpoint about our job scheduler, and how jobs should be submitted.

The demonstration assumes that you just logged in to SCG, and that you are a member of the Full-Tier ruthm Lab. You should change the SLURM account to your own PI’s SUNetID; and if your Lab is a Free-Tier Lab, you should change the SLURM partition from batch to nih_s10.

The demo stores the Compute Endpoint configuration in the globus_compute_demo directory. Normally configuration is stored in the hidden .globus_compute directory in your home directory; the explicit configuration path makes cleanup easier.

Here are the commands to run, to reproduce the demo in your own session:

NOTE: On SCG, Python versions are provided through Lmod modules. Every example assumes that you have already loaded the python/3.11.1 module.

# If needed, activate the venv
cd globus_compute_endpoint
. bin/activate
which python
python -V

# Create a new Compute Endpoint
globus-compute-endpoint \
--config-dir=/home/akkornel/globus_compute_demo/config \
configure --display-name "Globus Flow Demo on SCG" globus_flow_demo

# Edit the file at path `config/globus_flow_demo/config.yaml`,
# configuring the provider as follows:

provider:
  type: SlurmProvider
  partition: batch
  account: ruthm
  launcher:
    type: SrunLauncher

# After saving the `config.yaml` file, start the Endpoint
globus-compute-endpoint --config-dir=/home/akkornel/globus_compute_demo/config start globus_flow_demo

# List all endpoints, confirming that the Endpoint is started
globus-compute-endpoint --config-dir=/home/akkornel/globus_compute_demo/config list

In the above example, the Compute Endpoint has been registered and assigned UUID b72ee151-a5d9-4cfc-97b3-86e0664730b2.

The next demo is Run the Flow.

More recordings by akkornel

Browse all

Singularity Bootstrap with package install inside the container 2:01

by akkornel 8 years ago

Installing and running mpi4.py on a single host 0:38

by akkornel 6 years ago

Bootstrapping CentOS without yum 0:43

by akkornel 8 years ago

Building an MGP container image 3:26

by akkornel 7 months ago

https://asciinema.org/a/725819

Copied!

Append ?t=30 to start the playback at 30s, ?t=3:20 to start the playback at 3m 20s.

See sharing docs for more link customization options.

Embed as image link

Use snippets below to display a preview image linking to this recording.
Ideal for places where scripts are not allowed, such as project README files.

HTML:

<a href="https://asciinema.org/a/725819" target="_blank"><img src="https://asciinema.org/a/725819.svg" /></a>

Copied!

Markdown:

[![asciicast](https://asciinema.org/a/725819.svg)](https://asciinema.org/a/725819)

Copied!

Embed the player

If you're embedding on your own page or on a site which permits script tags, you can use the full player widget:

<script src="https://asciinema.org/a/725819.js" id="asciicast-725819" async="true"></script>

Copied!

Paste the above script tag where you want the player to be displayed on your page.

See embedding docs for more player customization options.

You can download this recording in asciicast v2 format, as a .cast file.

Download

Replay in terminal

You can replay the downloaded recording in your terminal using the asciinema play command:

asciinema play 725819.cast

Copied!

If you don't have asciinema CLI installed then see installation instructions.

Use with stand-alone player on your website

Download asciinema player from the releases page (you only need .js and .css file), then use it like this:

<!DOCTYPE html>
<html>
<head>
  <link rel="stylesheet" type="text/css" href="/assets/asciinema-player.css" />
</head>
<body>
  <div id="player"></div>
  <script src="/assets/asciinema-player.min.js"></script>
  <script>
    AsciinemaPlayer.create(
      '/assets/725819.cast',
      document.getElementById('player'),
      { cols: 134, rows: 24 }
    );
  </script>
</body>
</html>

See asciinema player quick-start guide for full usage instructions.

While this site doesn't provide GIF conversion at the moment, you can still do it yourself with the help of asciinema GIF generator utility - agg.

Once you have it installed, generate a GIF with the following command:

agg https://asciinema.org/a/725819 demo.gif

Copied!

Or, if you already downloaded the recording file:

agg demo.cast demo.gif

Copied!

Check agg --help for all available options. You can change font family and size, select color theme, adjust speed and more.

See agg manual for full usage instructions.