Run Pegasus Workflow across Multiple HTCondor Pools in Flocking mode

Install workflow submitter

The workflow submitter consists of HTCondor master and submitter that run in the flocking mode. Pegasus is installed on the HTCondor submitter. The workflow submitter flock workflow jobs to other HTCondor pools with available resources.

To install the workflow submitter, follow the instructions on the main page to install the HTCondor master and submitter using Docker. To enable flocking,

1. Set TCP_FORWARDING_HOST in condor_config.local.submitter with the public/external IP address of the host
2. Set FLOCK_TO as a comma-separated list of HTCondor pool addresses where the workflow jobs will flock to
3. Provide proper credentials, certificates and keys for flocking to secured HTCondor pools
4. Run docker_run_htcondor.sh -f and wait for 60 seconds for the submitter to be ready

Deploy a HTCondor pool

The HTCondor pool will offer resources for running flocking jobs. As of writing, it can be installed on a single node with Ubuntu 14.04 using condor_setup/ubuntu_install.sh <flock-from-list>. The <flock-from-list> should be a comma-separated list of host addresses where the workflow jobs will flock from.

Note that the default HTCondor deployment defaults to a low security level (CLAIMTOBE) and requires no authentication for flocking jobs. The default security configuration should only be used for development and test. For more sophisticated security configuration, please refer to this page.

Submit a workflow

There is an example Pegasus workflow under the home directory of condor_pool in the HTCondor submitter. The workflow consists of a configurable number of parallel jobs, each of which sleeps for 120 seconds and writes its finishing time and hostname as output. To submit the workflow,

1. Log in the HTCondor submitter: docker exec -it condor-submitter /bin/bash
2. Switch to user condor_pool and enter the workflow directory:

su - condor_pool
cd ~/example