Difference: JobWrapperGeneration (1 vs. 3)

Revision 32007-12-13 - MarcoCecchi

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How the Job Wrapper is Generated

Line: 44 to 44
 

Why the change?

Changed:
<
<
We changed the approach on how to generate the job wrapper script because the complete generation from code is very inflexible: any change would require a change to the code, with all the obvious consequences (rebuilding other components depending on this one, testing, certification, etc.). With the current approach, instead, changing the way the job wrapper is generated most of the times requires only a change in the template file.
>
>
We changed the approach on how to generate the job wrapper script because the earlier hard-coded generation revealed to be pretty inflexible: any change would have required a change to the C++ code, with all the obvious consequences (rebuilding other components depending on this one, testing, certification, etc.). With the current approach, instead, changing the way the job wrapper is generated most of the times requires only a change in the template file.
  For example, in order to fix bug #29604 ("WMS job wrapper must not reset umask"), waiting for an official patch, a WMS sysadmin could simply remove the following line
Line: 54 to 54
  from the template file.
Added:
>
>
As another example, to enable the limit mechanism on the output sandbox in the slc3 release - which does not work because of bug #27215 - applying the following patch would fix it:

diff -u -r1.37.2.45 template.sh
--- template.sh 15 Jun 2007 10:47:55 -0000      1.37.2.45
+++ template.sh 13 Dec 2007 14:40:04 -0000
Line: 870 to 870
Added:
>
>
if [ ${max_osb_size} -ge 0 ]; then # TODO #if hostname=wms - file_size=`stat -t $f | awk '{print $2}'` + file_size=`stat -t $s | awk '{print $2}'` file_size_acc=`expr $file_size_acc + $file_size` #fi if [ $file_size_acc -le ${max_osb_size} ]; then
Line: 883 to 883
Added:
>
>
fi else jw_echo "OSB quota exceeded for $s, truncating needed" - remaining_files=`expr $total_files \- $current_file + 2` + file_size_acc=`expr $file_size_acc - $file_size` + remaining_files=`expr $total_files \- $current_file` remaining_space=`expr $max_osb_size \- $file_size_acc` - trunc_len=`expr $remaining_space / $remaining_files`||0 + trunc_len=`expr $remaining_space / $remaining_files || echo 0` + file_size_acc=`expr $file_size_acc + $trunc_len` if [ $trunc_len -lt 10 ]; then # non trivial truncation jw_echo "Not enough room for a significant truncation on file ${f}, not sending" else
 -- FrancescoGiacomini - 15 Nov 2007 \ No newline at end of file

Revision 22007-12-13 - FrancescoGiacomini

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How the Job Wrapper is Generated

Line: 44 to 44
 

Why the change?

Changed:
<
<
We changed the approach on how to generate the job wrapper script because the complete generation from code is very inflexible: any change would require a change to the code, with all the obvious consequences (rebuilding other components depending on this one, testing, certification, etc.). With the current approach, instead, changing the way the job wrapper is generated most of the times requires only a change in the template file. For example, in order to fix bug #29604 ("WMS job wrapper must not reset umask"), waiting for an official patch, a WMS sysadmin could simply remove the following line
>
>
We changed the approach on how to generate the job wrapper script because the complete generation from code is very inflexible: any change would require a change to the code, with all the obvious consequences (rebuilding other components depending on this one, testing, certification, etc.). With the current approach, instead, changing the way the job wrapper is generated most of the times requires only a change in the template file.

For example, in order to fix bug #29604 ("WMS job wrapper must not reset umask"), waiting for an official patch, a WMS sysadmin could simply remove the following line

 
umask 022

Revision 12007-11-15 - FrancescoGiacomini

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WebHome"

How the Job Wrapper is Generated

The job wrapper is a shell script that wraps the user's job execution on the Worker Node.

In the LCG RB the full script generation was hard-coded in the RB itself.

In the gLite WMS, instead, the body of the shell script comes from a template file distributed with the WMS packages and installed in ${GLITE_LOCATION}/etc/templates/template.sh. For each job the template is augmented by the WMS with a few instructions in order to generate the final job wrapper. These instructions set a number of variables that are used inside the template. The value of these variables depends on the specific job.

The template has the following structure

#!/bin/sh
<empty line>
<body>

The WMS in correspondence of the empty line adds a header consisting in a number of instructions of the form:

<variable>=<job-specific value>

For example, the template contains the following excerpt of code:

if [ ${__job_type} -eq 0 ]; then # normal
  cmd_line="${__job} ${__arguments}"
elif [ ${__job_type} -eq 1 -o ${__job_type} -eq 2 ]; then # MPI LSF, PBS
  cmd_line="mpirun -np ${__nodes} -machinefile ${HOSTFILE} ${__job} ${__arguments}"
elif [ ${__job_type} -eq 3 ]; then # interactive
  cmd_line="./glite-wms-job-agent $BYPASS_SHADOW_HOST $BYPASS_SHADOW_PORT '${__job} ${__arguments}'"
fi

where a number of reserved variables are used: __job_type, __job, __arguments, __nodes. The setting of these variables is included by the WMS just after the line

#!/bin/sh

at the beginning of the script, so that they are well defined during the script execution.

Why the change?

We changed the approach on how to generate the job wrapper script because the complete generation from code is very inflexible: any change would require a change to the code, with all the obvious consequences (rebuilding other components depending on this one, testing, certification, etc.). With the current approach, instead, changing the way the job wrapper is generated most of the times requires only a change in the template file. For example, in order to fix bug #29604 ("WMS job wrapper must not reset umask"), waiting for an official patch, a WMS sysadmin could simply remove the following line

umask 022

from the template file.

-- FrancescoGiacomini - 15 Nov 2007

 
This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback