Is there a way to do startup and cleanup tasks with an SGE task array?

Q&A
, , , , ,
1

Is there a way to run one time commands at the beginning before the first task in the array or after the last task?

Doing things like creating a new directory before hand to put all the output files in, or being able to tar and move them to a long term storage location would be very helpful.

2

There are different, but combinable options for doing so by either when the first or last task starts
and for when all tasks finish:

When only concerned about when tasks start:
The key is that SGE array jobs set some additional variables
$SGE_TASK_FIRST, $SGE_TASK_LAST, $SGE_STEP_SIZE.

#!/bin/bash
#$ -t 1-100
if [ $SGE_TASK_ID -eq $SGE_TASK_FIRST ] ; then
# do first-task stuff here
fi

# do normal processing here

if [ $SGE_TASK_ID -q $SGE_TASK_LAST ]; then
# do last-task stuff here
fi

When concerned about when the last job finishes:
you can use the --sync flag in a script

qsub --sync y jobscript.sh >> exitstatus.log
# do things after

Will wait on the submitted job(s) to complete and report its exit code before closing.
With an array job it will wait on, and report on all of them.

3

This seems rather clunky are there better ways?