5.2.4. gromacs.qsub – utilities for batch submission systems

The module helps writing submission scripts for various batch submission queuing systems. The known ones are listed stored as QueuingSystem instances in queuing_systems; append new ones to this list.

The working paradigm is that template scripts are provided (see gromacs.config.templates) and only a few place holders are substituted (using gromacs.cbook.edit_txt()).

User-supplied template scripts can be stored in gromacs.config.qscriptdir (by default ~/.gromacswrapper/qscripts) and they will be picked up before the package-supplied ones.

At the moment, some of the functions in gromacs.setup use this module but it is fairly independent and could conceivably be used for a wider range of projects.

5.2.4.1. Queuing system templates

The queuing system scripts are highly specific and you will need to add your own. Templates should be shell scripts. Some parts of the templates are modified by the generate_submit_scripts() function. The “place holders” that can be replaced are shown in the table below. Typically, the place holders are either shell variable assignments or batch submission system commands. The table shows SGE commands but PBS and LoadLeveler have similar constructs; e.g. PBS commands start with #PBS and LoadLeveller uses #@ with its own command keywords).

Substitutions in queuing system templates.

place holder	default	replacement	description	regex
#$ -N	GMX_MD	sgename	job name	/^#.*(-N|job_name)/
#$ -l walltime=	00:20:00	walltime	max run time	/^#.*(-l walltime|wall_clock_limit)/
#$ -A	BUDGET	budget	account	/^#.*(-A|account_no)/
DEFFNM=	md	deffnm	default gmx name	/^ *DEFFNM=/
STARTDIR=	.	startdir	remote jobdir	/^ *STARTDIR=/
WALL_HOURS=	0.33	walltime h	mdrun’s -maxh	/^ *WALL_HOURS=/
NPME=		npme	PME nodes	/^ *NPME=/
MDRUN_OPTS=	“”	mdrun_opts	more options	/^ *MDRUN_OPTS=/

Lines with place holders should not have any white space at the beginning. The regular expression pattern (“regex”) is used to find the lines for the replacement and the literal default values (“default”) are replaced. (Exception: any value that follows an equals sign “=” is replaced, regardless of the default value in the table except for MDRUN_OPTS where only “” will be replace.) Not all place holders have to occur in a template; for instance, if a queue has no run time limitation then one would probably not include walltime and WALL_HOURS place holders.

The line # JOB_ARRAY_PLACEHOLDER can be replaced by generate_submit_array() to produce a “job array” (also known as a “task array”) script that runs a large number of related simulations under the control of a single queuing system job. The individual array tasks are run from different sub directories. Only queuing system scripts that are using the bash shell are supported for job arrays at the moment.

A queuing system script must have the appropriate suffix to be properly recognized, as shown in the table below.

Suffices for queuing system templates. Pure shell-scripts are only used to run locally.

Queuing system	suffix	notes
Sun Gridengine	.sge	Sun’s Sun Gridengine
Portable Batch queuing system	.pbs	OpenPBS and PBS Pro
LoadLeveler	.ll	IBM’s LoadLeveler
bash script	.bash, .sh	Advanced bash scripting
csh script	.csh	avoid csh

5.2.4.1.1. Example queuing system script template for PBS

The following script is a usable PBS script for a super computer. It contains almost all of the replacement tokens listed in the table (indicated by ++++++).

#!/bin/bash
# File name: ~/.gromacswrapper/qscripts/supercomputer.somewhere.fr_64core.pbs
#PBS -N GMX_MD
#       ++++++
#PBS -j oe
#PBS -l select=8:ncpus=8:mpiprocs=8
#PBS -l walltime=00:20:00
#                ++++++++

# host: supercomputer.somewhere.fr
# queuing system: PBS

# set this to the same value as walltime; mdrun will stop cleanly
# at 0.99 * WALL_HOURS
WALL_HOURS=0.33
#          ++++

# deffnm line is possibly modified by gromacs.setup
# (leave it as it is in the template)
DEFFNM=md
#      ++

TPR=${DEFFNM}.tpr
OUTPUT=${DEFFNM}.out
PDB=${DEFFNM}.pdb

MDRUN_OPTS=""
#          ++

# If you always want to add additional MDRUN options in this script then
# you can either do this directly in the mdrun commandline below or by
# constructs such as the following:
## MDRUN_OPTS="-npme 24 $MDRUN_OPTS"

# JOB_ARRAY_PLACEHOLDER
#++++++++++++++++++++++   leave the full commented line intact!

# avoids some failures
export MPI_GROUP_MAX=1024
# use hard coded path for time being
GMXBIN="/opt/software/SGI/gromacs/4.0.3/bin"
MPIRUN=/usr/pbs/bin/mpiexec
APPLICATION=$GMXBIN/mdrun_mpi

$MPIRUN $APPLICATION -stepout 1000 -deffnm ${DEFFNM} -s ${TPR} -c ${PDB} -cpi                         $MDRUN_OPTS                         -maxh ${WALL_HOURS} > $OUTPUT
rc=$?

# dependent jobs will only start if rc == 0
exit $rc

Save the above script in ~/.gromacswrapper/qscripts under the name supercomputer.somewhere.fr_64core.pbs. This will make the script immediately usable. For example, in order to set up a production MD run with gromacs.setup.MD() for this super computer one would use

gromacs.setup.MD(..., qscripts=['supercomputer.somewhere.fr_64core.pbs', 'local.sh'])

This will generate submission scripts based on supercomputer.somewhere.fr_64core.pbs and also the default local.sh that is provided with GromacsWrapper.

In order to modify MDRUN_OPTS one would use the additonal mdrun_opts argument, for instance:

gromacs.setup.MD(..., qscripts=['supercomputer.somewhere.fr_64core.pbs', 'local.sh'],
                 mdrun_opts="-v -npme 20 -dlb yes -nosum")

Currently there is no good way to specify the number of processors when creating run scripts. You will need to provide scripts with different numbers of cores hard coded or set them when submitting the scripts with command line options to qsub.

5.2.4.2. Classes and functions

class gromacs.qsub.QueuingSystem(name, suffix, qsub_prefix, array_variable=None, array_option=None)

Class that represents minimum information about a batch submission system.

Define a queuing system’s functionality

Arguments:

name

name of the queuing system, e.g. ‘Sun Gridengine’

suffix

suffix of input files, e.g. ‘sge’

qsub_prefix

prefix string that starts a qsub flag in a script, e.g. ‘#$’

Keywords:

array_variable

environment variable exported for array jobs, e.g. ‘SGE_TASK_ID’

array_option

qsub option format string to launch an array (e.g. ‘-t %d-%d’)

array(directories)

Return multiline string for simple array jobs over directories.

Warning

The string is in bash and hence the template must also be bash (and not csh or sh).

array_flag(directories)

Return string to embed the array launching option in the script.

flag(*args)

Return string for qsub flag args prefixed with appropriate inscript prefix.

has_arrays()

True if known how to do job arrays.

isMine(scriptname)

Primitive queuing system detection; only looks at suffix at the moment.

gromacs.qsub.generate_submit_scripts(templates, prefix=None, deffnm='md', jobname='MD', budget=None, mdrun_opts=None, walltime=1.0, jobarray_string=None, startdir=None, npme=None, **kwargs)

Write scripts for queuing systems.

This sets up queuing system run scripts with a simple search and replace in templates. See gromacs.cbook.edit_txt() for details. Shell scripts are made executable.

Arguments:

templates

Template file or list of template files. The “files” can also be names or symbolic names for templates in the templates directory. See gromacs.config for details and rules for writing templates.

prefix

Prefix for the final run script filename; by default the filename will be the same as the template. [None]

dirname

Directory in which to place the submit scripts. [.]

deffnm

Default filename prefix for mdrun -deffnm [md]

jobname

Name of the job in the queuing system. [MD]

budget

Which budget to book the runtime on [None]

startdir

Explicit path on the remote system (for run scripts that need to cd into this directory at the beginning of execution) [None]

mdrun_opts

String of additional options for mdrun.

walltime

Maximum runtime of the job in hours. [1]

npme

number of PME nodes

jobarray_string

Multi-line string that is spliced in for job array functionality (see gromacs.qsub.generate_submit_array(); do not use manually)

kwargs

all other kwargs are ignored

Returns:

list of generated run scripts

gromacs.qsub.generate_submit_array(templates, directories, **kwargs)

Generate a array job.

For each work_dir in directories, the array job will

1. cd into work_dir

2. run the job as detailed in the template

It will use all the queuing system directives found in the template. If more complicated set ups are required, then this function cannot be used.

Arguments:

templates

Basic template for a single job; the job array logic is spliced into the position of the line

# JOB_ARRAY_PLACEHOLDER

The appropriate commands for common queuing systems (Sun Gridengine, PBS) are hard coded here. The queuing system is detected from the suffix of the template.

directories

List of directories under dirname. One task is set up for each directory.

dirname

The array script will be placed in this directory. The directories must be located under dirname.

kwargs

See gromacs.setup.generate_submit_script() for details.

gromacs.qsub.detect_queuing_system(scriptfile)

Return the queuing system for which scriptfile was written.

gromacs.qsub.queuing_systems = [<Sun Gridengine QueuingSystem instance>, <PBS QueuingSystem instance>, <LoadLeveler QueuingSystem instance>, <Slurm QueuingSystem instance>]

Pre-defined queuing systems (SGE, PBS). Add your own here.