BLAH Introduction

BLAHPD is a light component accepting commands according to the BLAH (Batch Local Ascii Helper) protocol to manage jobs on different Local Resources Management Systems (LRMS). The BLAH service provides a minimal, pragmatically designed common interface for job submission and control to a set of batch systems. Interaction with BLAH is achieved via text commands whose syntax and semantics is described below. Interaction with each one of the various supported batch system is achieved via customized scripts (the existing scripts are all Bourne shell scripts) that take care of executing the following functions:

- job submit - job hold - job resume - job status - job cancel

BLAH Portability

The purpose of this writeup is to:

• describe the assumptions that are currently made on the minimal functionality that batch systems must provide;
• describe the details of the command line arguments that are passed to each one of the five scripts that implement the fuctionality described above for a specific batch system;
• provide guidelines for porting of the existing scripts, especially the submit script for which helper shell functions were defined.

As the universe of different batch systems that need to be supported is very small, the most efficient way to get help beyond the information collected in these notes is to interact with the BLAH developers directly via the e-mail address blah-help@mi.infn.it

The string XXX, in the following text, is meant to be replaced with the name of the specific batch system being interfaced (pbs, lsf, condor, etc.).

Assumptions on batch system functionality.

BLAH assumes that batch systems are capable to

• Identify jobs for subsequent operations via a unique, constant identifier that is returned at the time of job submission.
• Transfer and optionally rename a configurable set of files from a given location on the submit node to the initial working directory on the selected worker node before the job execution starts.
• Transfer and optionally rename a configurable set of files from the initial working directory of jobs on the worker node that is selected for execution to a given location on the submit node.
• Provide status of currently running jobs via an appropriate command.
• Provide a historical track of jobs that were accepted and ran in the past via one or more log files.

BLAH can use this optional batch system functionality:

a) Hold (suspend) and resume jobs via appropriate commands.

BLAH doesn't require batch systems to

a) Be able to renew X509 proxies and/or transfer files to and from the worker node while the job is being executed. Proxy renewal is taken care of by a proxy renewal daemon that runs at the side of the running job and receives delegations of refreshed proxies during the lifetime of the job.

BLAH Scripts

XXX_submit.sh script

Submit a new job request to the batch system.

NOTE: most of the functionality for parsing and handling the submit script arguments is provided by a set of shell functions that are described further below (see SUBMIT SCRIPT HELPER FUNCTIONS). The argument description is provided for reference only.

SYNOPSIS:

XXX_submit.sh -c -q [-i ] [-o ] [-e ] [-x ] [-v | -V ] [-s ] [-d ] [-w ] [-n <# of MPI nodes>] [-r ] [-p ] [-l ] [-j ] [-T ] [-I ] [-O ] [-R ] [-C ] [-- command_arguments]

Any argument after '--' will be passed to the user job ("command").

The command line switches have the following meaning (switches listed in alphabetical order):

[-C ]: When this argument is present, the local script XXX_local_submit_attributes.sh is called, and its output is pasted to the batch system submit file. The XXX_local_submit_attributes.sh script is called after sourcing the contents of the 'CE requirements file' first. This file is composed by the BLAH main daemon and sets shell variables that specify attribute bounds derived from the CERequirements attribute in the BLAH submit command. Example format for the 'CE requirements file':

GlueHostMainMemoryRAMSize_Min=1000 GlueCEPolicyMaxCPUTime_Max=30 UserVO_Eq="EGEE"

-c command: (shell) command to be executed as a batch job.

[-d ]: Debug option. If set to 'yes', it causes the submit file to be sent to /dev/tty, and no actual submission to the batch system.

[-e ]: Location of the local file that should receive the job STDERR upon job termination.

[-I ]: points to an optional, temporary file containing a list of additional files (one per line) that need to be transferred from the submission node to the initial working directory on the worker node. This temp file is removed by the submit script.

[-i ]: Location of a local file to be transferred to the worker node and connected to the STDIN of the job.

[-j ]: String for unique identification of the job by the calling layer. As this string is unique, it is used to name the submit script, that will be created as 'cream_jobID'.

[-l ]: Minimum remaining lifetime of the user proxy (expressed in seconds) before the proxy renewal damon will kill the user job. Defaults to 180 seconds, or 3 minutes.

[-n <# of MPI nodes>]: Number of MPI nodes to be reserved for the job, in case this feature is supported by the underlying batch system.

[-O ]: points to an optional, temporary file containing a list of additional files (one per line) that need to be transferred from the working directory on the worker node back to the submit node. This temp file is removed by the submit script. The -R arguments can be used to establish file name remapping.

[-o ]: Location of the local file that should receive the job STDOUT upon job termination.

[-p ]: time interval (expressed in seconds) between attempts of the proxy renewal daemon to check that the use process is still alive. Defaults to 60 seconds.

-q Batch system queue to run the job in.

[-R ]: Points to a file containing a list of names (one for each line of the file passed with the -O argument) that contain the local name of each each output file that is transferred from the worker node to the submit (local) node. The list file will be deleted by the submit script.

[-r ]: Disables the entire BLAH proxy renewal machinery when set to 'no'. Defaults to 'yes' if missing.

[-T ]: sets the directory location for storing temporary files.

[-v | -V ]: Environment variables to be set for the user job. Two formats are possible: 1) semicolon-separated assignments -v "ENV1=val1;ENV2=val2;..." 2) space-separated assignments -V "ENV1=val1 ENV2=val2 ..."

[-x ]: Location of the initial X509 user proxy to associate with teh job. This file is initially transferred with the job, then (optionally, see -r) renewed via proxy delegation.

[-w ]: Directory location pre-pended to relative file paths and used as CWD by the submit script.

RETURN VALUE:

Termination code is zero on success, nonzero on error. Job identifier, with the identifier string 'BLAHP_JOBID_PREFIX' is returned on STDOUT on success. The job identifier must start with the batch system name followed by a slash (XXX/id).

SUBMIT SCRIPT HELPER FUNCTIONS:

A set of shell functions was written to ease the parsing and handling of the submit script options. They allow to write a submit script along the following template:

#!/bin/bash # 1. Source definition of helper functions. . `dirname $0`/blah_common_submit_functions.sh

# 2. Parse the submit options, that are used to set # a list of bls_opt_XYZ shell variables. bls_parse_submit_options $@

# 3. Set up temporary files. Set a variable with the name of # an environment variable that holds the batch system job ID # at runtime. bls_setup_all_files bls_job_id_for_renewal=XXX_JOBID

# 4. Start writing the submit script to $bls_tmp_file

cat > $bls_tmp_file << end_of_preamble #!/bin/bash # PBS job wrapper generated by `basename $0` # on `/bin/date` # proxy_string = $bls_opt_proxy_string # proxy_local_file = $bls_proxy_local_file # Etc. Etc. Etc. end_of_preamble

# 5. Handle script local customisations according to -C option # as appropriate for the batch system at hand. if [ ! -z $bls_opt_req_file ] ; then echo \#\!/bin/sh >> ${bls_opt_req_file}-temp_req_script cat $bls_opt_req_file >> ${bls_opt_req_file}-temp_req_script echo "source ${GLITE_LOCATION:-/opt/glite}/bin/XXX_local_submit_attributes.sh" >> ${bls_opt_req_file}-temp_req_script chmod +x ${bls_opt_req_file}-temp_req_script ${bls_opt_req_file}-temp_req_script >> $bls_tmp_file 2> /dev/null rm -f ${bls_opt_req_file}-temp_req_script rm -f $bls_opt_req_file fi

# 6. Add specific directives to select queue ($bls_opt_queue) and # MPI node request ($bls_opt_mpinodes)

# 7. Add directives to transfer and rename input and output files. # These are stored as # $bls_inputsand_local_0...$bls_inputsand_local_n-1 # $bls_inputsand_remote_0...$bls_inputsand_remote_n-1 # $bls_outputsand_local_0...$bls_outputsand_local_n-1 # $bls_outputsand_remote_0...$bls_outputsand_remote_n-1 # # Two shell functions can help here. # a: # bls_fl_subst_and_accumulate inputsand "@@F_REMOTE/@@F_LOCAL" "sep" # bls_fl_subst_and_accumulate outputsand "@@F_REMOTE/@@F_LOCAL" "sep" # fill $bls_fl_subst_and_accumulate_result with a list of "sep" # separated strings formatted as shown in the second argument. # The submit node full file path is substituted to @@F_LOCAL # and the worker node path relative to the initial working dir # is substituted to @@F_REMOTE. # b: # bls_fl_subst_and_dump inputsand "@@F_LOCAL>@@F_REMOTE" $bls_tmp_file # bls_fl_subst_and_dump outputsand "@@F_LOCAL<@@F_REMOTE" $bls_tmp_file # append to $bls_tmp_file a line for each input and output file, # where @@F_REMOTE and @@F_LOCAL are substituted as above.

# 8. Append job wrapper as a shell script to $bls_tmp_file bls_add_job_wrapper

# 9. Send the submit file $bls_tmp_file to the batch system and # try making sure it doesn't get lost.

# 10. Echo to STDOUT the unique job ID to be used by subsequent scripts # (with BLAHP_JOBID_PREFIX) and wrap up. The job ID must be # properly understood by subsequent commands. echo "BLAHP_JOBID_PREFIXXXX?$jobID" bls_wrap_up_submit exit $retcode

XXX_hold.sh script

Suspend the execution of a job.

SYNOPSIS:

XXX_hold.sh The job identifier must be the same string returned by the submit script (without the leading BLAHP_JOBID_PREFIX). Any leading part up to the first slash '/' will be ignored by the script.

RETURN VALUE:

Termination code is zero on success, nonzero on error.

XXX_resume.sh script

Resume the execution of a (previously suspended) job.

SYNOPSIS:

XXX_resume.sh The job identifier must be the same string returned by the submit script (without the leading BLAHP_JOBID_PREFIX). Any leading part up to the first slash '/' will be ignored by the script.

RETURN VALUE:

Termination code is zero on success, nonzero on error.

XXX_status.sh script

Get current status of a job.

SYNOPSIS:

XXX_status.sh [-w] [-n] The job identifier must be the same string returned by the submit script (without the leading BLAHP_JOBID_PREFIX). Any leading part up to the first slash '/' will be ignored by the script.

[-n] Option specific to the CREAM caller. Return the port used by the BLParser (see description below) to communicate with the CREAM service

[-w] Return the worker node where a job is running as an attribute of the output classad (WorkerNode="host.domain")

RETURN VALUE:

The script must return a string-formatted classad (see http://www.cs.wisc.edu/condor/classad/refman/ for the complete reference on classad syntax) containing at least the following attributes:

BatchjobId = "jobId (without the leading batch system name)" JobStatus = status_code (1 = IDLE, 2 = RUNNING, 3 = REMOVED, 4 = COMPLETED 5 = HELD) (only for COMPLETED jobs): ExitCode = code

Example of status script results:

(job queued) [ BatchjobId = "26526.atlfarm006.mi.infn.it"; JobStatus = 1 ]

(job completed) [ ExitCode = 0; BatchjobId = "26526.atlfarm006.mi.infn.it"; JobStatus = 4 ]

XXX_cancel.sh script

Remove a job from the batch system queue.

SYNOPSIS:

XXX_cancel.sh The job identifier must be the same string returned by the submit script (without the leading BLAHP_JOBID_PREFIX). Any leading part up to the first slash '/' will be ignored by the script.

RETURN VALUE:

Termination code is zero on success, nonzero on error.

BLAH Commands syntax and semantics

BLAH Commands

The following list of commands represents the set of commands required for interaction with the BLAHP server, interfacing to a given Local Resource Management system. This is based on the minimum set of commands used in the original GAHP (v1.0.0) specification removing commands that are specific to the operation of the GRAM protocol (INITIALIZE_FROM_FILE, GASS_SERVER_INIT, GRAM_CALLBACK_ALLOW, GRAM_JOB_CALLBACK_REGISTER, GRAM_PING). The JOB_SIGNAL command may be initially left unimplemented for some of the batch systems (and in that case will return an error -E- state and will not be returned by COMMANDS).

• BLAH_JOB_CANCEL
• BLAH_JOB_SIGNAL
• BLAH_JOB_HOLD
• BLAH_JOB_REFRESH_PROXY
• BLAH_JOB_RESUME
• BLAH_JOB_STATUS
• BLAH_JOB_STATUS_ALL
• BLAH_JOB_STATUS_SELECT
• BLAH_JOB_SUBMIT
• BLAH_SET_GLEXEC_DN
• BLAH_SET_GLEXEC_OFF
• COMMANDS
• CACHE_PROXY_FROM_FILE
• QUIT
• RESULTS
• USE_CACHED_PROXY
• UNCACHE_PROXY
• VERSION

Optionally, the following two commands may also be implemented:

• ASYNC_MODE_ON
• ASYNC_MODE_OFF

BLAHP Commands structure

Conventions and Terms used

Below are definitions for the terms used in the sections to follow:

<CRLF>The characters carriage return and line feed (in that order), or solely the line feed character.

<SP>The space character.

line A sequence of ASCII characters ending with a <SP>

Request Line A request for action on the part of the BLAHP server.

Return Line A line immediately returned by the BLAHP server upon receiving a Request Line.

Result Line A line sent by the BLAHP server in response to a RESULTS request, which communicates the results of a previous asynchronous command Request.

S: and R: In the Example sections for the commands below, the prefix "S: " is used to signify what the client sends to the BLAHP server. The prefix "R: " is used to signify what the client receives from the BLAHP server. Note that the "S: " or "R: " should not actually be sent or received.

Commands structure

BLAHP commands consist of three parts:

• Request Line
• Return Line
• Result Line

Each of these "Lines" consists of a variable length character string ending with the character sequence <CRLF>.

A Request Line is a request from the client for action on the part of the BLAHP server. Each Request Line consists of a command code followed by argument field(s). Command codes are a string of alphabetic characters. Upper and lower case alphabetic characters are to be treated identically with respect to command codes. Thus, any of the following may represent the blah_job_submit command: blah_job_submit Blah_Job_Submit blAh_joB_suBMit BLAH_JOB_SUBMIT In contrast, the argument fields of a Request Line are _case sensitive_.

The Return Line is always generated by the server as an immediate response to a Request Line. The first character of a Return Line will contain one the following characters: S - for Success F - for Failure E - for a syntax or parse Error Any Request Line which contains an unrecognized or unsupported command, or a command with an insufficient number of arguments, will generate an "E" response.

The Result Line is used to support commands that would otherwise block. Any BLAHP command which may require the implementation to block on network communication require a "request id" as part of the Request Line. For such commands, the Result Line just communicates if the request has been successfully parsed and queued for service by the BLAHP server. At this point, the BLAHP server would typically dispatch a new thread to actually service the request. Once the request has completed, the dispatched thread should create a Result Line and enqueue it until the client issues a RESULT command.

Transparency

Arguments on a particular Line (be it Request, Return, or Result) are typically separated by a <SP>. In the event that a string argument needs to contain a <SP> within the string itself, it may be escaped by placing a backslash ("\") in front of the <SP> character. Thus, the character sequence "\ " (no quotes) must not be treated as a separator between arguments, but instead as a space character within a string argument.

Sequence of Events

Upon startup, the BLAHP server should output to stdout a banner string which is identical to the output from the VERSION command without the beginning "S " sequence (see example below). Next, the BLAHP server should wait for a complete Request Line from the client (e.g. stdin). The server is to take no action until a Request Line sequence is received.

Example:

R: $GahpVersion: x.y.z Feb 31 2004 INFN\ Blahpd $
      S: COMMANDS
      R: S COMMANDS BLAH_JOB_CANCEL BLAH_JOB_SIGNAL BLAH_JOB_STATUS BLAH_JOB_SUBMIT COMMANDS QUIT RESULTS VERSION
      S: VERSION
      R: S $GahpVersion: x.y.z Feb 31 2004 INFN\ Blahpd $
                (other commands)
      S: QUIT
      R: S

BLAH Commands syntax

This section contains the syntax for the Request, Return, and Result line for each command.

• COMMANDS: List all the commands from this protocol specification which are implemented by this BLAHP server.
  + Request Line: COMMANDS <CRLF>
  + Return Line: S <SP> <SP> <SP> ... <CRLF>
  + Result Line: None.

• VERSION: Return the version string for this BLAHP. The version string follows a specified format (see below). Ideally, the version entire version string, including the starting and ending dollar sign ($) delimiters, should be a literal string in the text of the BLAHP server executable. This way, the Unix/RCS "ident" command can produce the version string. The version returned should correspond to the version of the protocol supported.
  + Request Line: VERSION <CRLF>
  + Return Line:
  • S <SP> $GahpVesion: <SP> .. <SP> <build-month> <SP> <build-day-of-month> <SP> <build-year> <SP> <general-descrip> <SP>$ <CRLF>
  • major.minor.subminor = for this version of the protocol, use version 1.0.0.
  • build-month = string with the month abbreviation when this BLAHP server was built or released. Permitted values are: "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", and "Dec".
  • build-day-of-month = day of the month when BLAHP server was built or released; an integer between 1 and 31 inclusive.
  • build-year = four digit integer specifying the year in which the BLAHP server was built or released.
  • general-descrip = a string identifying a particular BLAHP server implementation.

+ Result Line: None.
+ Example:

S: VERSION R: S $GahpVersion: x.y.z Feb 31 2004 INFN\ Blahpd $

• QUIT: Free any/all system resources (close all sockets, etc) and terminate as quickly as possible.
  + Request Line: QUIT <CRLF>
  + Return Line: S <CRLF>
  Immediately afterwards, the command pipe should be closed and the BLAHP server should terminate.
  + Result Line: None.

• RESULTS: Display all of the Result Lines which have been queued since the last RESULTS command was issued. Upon success, the first return line specifies the number of subsequent Result Lines which will be displayed. Then each result line appears (one per line) -- each starts with the request ID which corresponds to the request ID supplied when the corresponding command was submitted. The exact format of the Result Line varies based upon which corresponding Request command was issued.
  IMPORTANT: Result Lines must be displayed in the exact order in which they were queued!!! In other words, the Result Lines displayed must be sorted in the order by which they were placed into the BLAHP's result line queue, from earliest to most recent.
  + Request Line: RESULTS
  + Return Line(s): S <SP><num-of-subsequent-result-lines> <CRLF> <SP> ... <CRLF> <SP> ... <CRLF>...
  * reqid = integer Request ID, set to the value specified in the corresponding Request Line.
  + Result Line: None.
  + Example:

S: RESULTS
      R: S 1
      R: 100 0

BLParser

To obtain efficient access and prompt update to the current status of BLAH active jobs, a 'log parser' daemon was developed for LSF and PBS. The submit and status scripts can make optional use of the batch log parser to increase their efficiency.

Here's a description of the semantics of the batch log parser (BLParser).

The BLParser listens on a network socket and replies to a set of mandatory and a set of optional queries (optional queries can be used for info and debug).

Here is the list of mandatory queries and their replies:

- BLAHJOB/<blahjob-id> => <lrms-jobid>

- <date-YYYYmmdd>/<lrms-jobid> => a classad like this: [BatchJobId=<lrms-jobid>; Workernode=; JobStatus=; LRMSSubmissionTime=; LRMSStartRunningTime=; LRMSCompletedTime=; ExitReason=; ExitCode=;]/pr_removal

where pr_removal is a flag that told the status script if the proxy file has to be removed (when job is killed or finished) or not.

- CREAMPORT => port where cream can connect to the parser.

- TEST => print Y (e.g. YLSF YPBS). This query is used to know if a parser is active. This is useful when in config file is specified more than one parser to try: if the first does not reply to the TEST query the next one is tried.

These are optional queries and their replies:

- HELP => print command list

- VERSION => print version

- TOTAL => print total number of jobs cached in the parser

- LISTALL => print jobid of all jobs cached in the parser

- LISTF[/<first-n-jobid>] => print first n jobid

- LISTL[/<last-n-jobid>] => print last n jobid

The BLParser includes an option to notify to a client (typically the CREAM service) every state change in CREAM-managed jobs. CREAM sends to the parser a string like this:

STARTNOTIFY/<date-YYYYmmdd>

atfer that string the parser sends back to cream every state change for cream job since the date contained in the string with lines like this:

NTFDATE/

where is a classad like the one sent in reply to <date-YYYYmmdd>/<lrms-jobid> query.

After that parser will sent to cream every new state change for cream jobs.

Cream can change the default prefix (cream_) used to identify tracked jobs among all other jobs with this command:

CREAMFILTER/

where should be a string like crXXX_ (XXX can be any character [a-z],[A-Z],[0-9]). This command has to sent before STARTNOTIFY.

There is a startup script for the BLParser that reads all the parameters from a config file. It is possible to set the debug level, the debug log file, the log file location; it is also possible to start different parsers listening on different ports with the same startup script.

-- ElisabettaMolinari - 24 Jun 2008