New CREAM-CE direct job submission metrics and WN probes

1. WN-softver probe - check middleware version on WN (via cream_jobSubmit.py)
2. WN-csh probe - check if WN has csh (via cream_jobSubmit.py)

1. 1. 1. 1. 1. 1. 1. 1. sec]

1. 1. 1. 1. 1. 1. 1. 1. sec]

1. 1. 1. 1. 1. 1. 1. 1. sec]

1. 1. 1. 1. 1. 1. 1. 1. sec]

1. 1. 1. 1. 1. 1. 1. 1. sec]

1. 1. 1. 1. 1. 1. 1. 1. sec]

WN-softver probe

$ ./cream_jobSubmit.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 -x /tmp/dteam.proxy -j ./WN-softver.jdl
DONE-OK: prod-wn-002 has EMI 1.11.0-1

$ cat WN-softver.jdl
[
Type="Job";
JobType="Normal";
Executable = "WN-softver.sh";
#Arguments = "a b c";
StdOutput = "std.out";
StdError = "std.err";
InputSandbox = {"WN-softver.sh"};
OutputSandbox = {"std.out","std.err"};
OutputSandboxBaseDestUri="gsiftp://localhost";
]

WN-softver.sh

The verbose option enabled gives the following output:

$ ./cream_jobSubmit.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 -x /tmp/dteam.proxy -j ./WN-softver.jdl --verbose
executing command: /usr/bin/voms-proxy-info -timeleft
executing command: /usr/bin/glite-ce-job-submit -d -a -r cream-41.pd.infn.it:8443/cream-lsf-creamtest1 ./WN-softver.jdl
['2014-01-31 13:05:05,802 DEBUG - Using certificate proxy file [/tmp/dteam.proxy]\n', '2014-01-31 13:05:05,823 DEBUG - VO from certificate=[dteam]\n', '2014-01-31 13:05:05,824 WARN - No configuration file suitable for loading. Using built-in configuration\n', '2014-01-31 13:05:05,824 DEBUG - Logfile is [/tmp/glite_cream_cli_logs/glite-ce-job-submit_CREAM_root_20140131-130505.log]\n', '2014-01-31 13:05:05,824 DEBUG - Processing file [WN-softver.sh]...\n', '2014-01-31 13:05:05,824 DEBUG - Adding absolute path [/root/WN-softver.sh]...\n', '2014-01-31 13:05:05,824 DEBUG - Inserting mangled InputSandbox in JDL: [{"/root/WN-softver.sh"}]...\n', '2014-01-31 13:05:05,827 INFO - certUtil::generateUniqueID() - Generated DelegationID: [b18342097b01b309cc9112a683d4c6bd15d35796]\n', '2014-01-31 13:05:07,517 DEBUG - Registering to [https://cream-41.pd.infn.it:8443/ce-cream/services/CREAM2] JDL=[ StdOutput = "std.out"; BatchSystem = "lsf"; QueueName = "creamtest1"; Executable = "WN-softver.sh"; Type = "Job"; JobType = "Normal"; OutputSandboxBaseDestUri = "gsiftp://localhost"; OutputSandbox = { "std.out","std.err" }; InputSandbox = { "/root/WN-softver.sh" }; StdError = "std.err" ] - JDL File=[./WN-softver.jdl]\n', '2014-01-31 13:05:07,870 DEBUG - JobID=[https://cream-41.pd.infn.it:8443/CREAM680089855]\n', '2014-01-31 13:05:07,871 DEBUG - UploadURL=[gsiftp://cream-41.pd.infn.it/var/glite/cream_sandbox/dteam/CN_Marco_Verlato_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam019/68/CREAM680089855/ISB]\n', '2014-01-31 13:05:07,873 INFO - Sending file [gsiftp://cream-41.pd.infn.it/var/glite/cream_sandbox/dteam/CN_Marco_Verlato_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam019/68/CREAM680089855/ISB/WN-softver.sh]\n', '2014-01-31 13:05:08,044 DEBUG - Will invoke JobStart for JobID [CREAM680089855]\n', 'https://cream-41.pd.infn.it:8443/CREAM680089855\n']
job id: https://cream-41.pd.infn.it:8443/CREAM680089855
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM680089855
['\n', '******  JobID=[https://cream-41.pd.infn.it:8443/CREAM680089855]\n', '\tStatus        = [DONE-OK]\n', '\tExitCode      = [0]\n', '\n', '\n']
exitCode=       ExitCode      = [0]

job status: DONE-OK
invoking getOutputSandbox
executing command: /usr/bin/glite-ce-job-output --noint https://cream-41.pd.infn.it:8443/CREAM680089855
output sandbox dir: ./cream-41.pd.infn.it_8443_CREAM680089855
invoking jobPurge
executing command: /usr/bin/glite-ce-job-purge --noint https://cream-41.pd.infn.it:8443/CREAM680089855
DONE-OK: prod-wn-002 has EMI 1.11.0-1

WN-csh probe

$ ./cream_jobSubmit.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 -x /tmp/dteam.proxy -j ./WN-csh.jdl
DONE-OK: prod-wn-002 has csh

$ cat WN-csh.jdl
[
Type="Job";
JobType="Normal";
Executable = "WN-csh.sh";
#Arguments = "a b c";
StdOutput = "std.out";
StdError = "std.err";
InputSandbox = {"WN-csh.sh"};
OutputSandbox = {"std.out","std.err"};
OutputSandboxBaseDestUri="gsiftp://localhost";
]

WN-csh.sh

Deployment example

$ ls -l /usr/libexec/grid-monitoring/probes/emi.cream/
total 48
-rwxr-xr-x 1 root root  1361 Jan 30 16:58 cream_allowedSubmission.py
-rwxr-xr-x 1 root root  2494 Jan 30 17:00 cream_jobCancel.py
-rwxr-xr-x 1 root root  2103 Jan 30 17:01 cream_jobPurge.py
-rwxr-xr-x 1 root root  2972 Jan 31 12:42 cream_jobSubmit.py
-rwxr-xr-x 1 root root 15527 Jan 30 16:29 cream.py
-rwxr-xr-x 1 root root  1416 Jan 31 12:42 cream_serviceInfo.py
-rw-r--r-- 1 root root   213 Jan 29 14:26 hostname.jdl
-rw-r--r-- 1 root root   129 Jan 30 16:21 sleep.jdl
drwxr-xr-x 2 root root  4096 Jan 31 11:34 wn

$ ls -l /usr/libexec/grid-monitoring/probes/emi.cream/wn
total 16
-rw-r--r-- 1 root root  292 Jan 31 11:34 WN-csh.jdl
-rwxr-xr-x 1 root root  603 Jan 31 11:34 WN-csh.sh
-rw-r--r-- 1 root root  300 Jan 31 11:34 WN-softver.jdl
-rwxr-xr-x 1 root root 1144 Jan 31 11:34 WN-softver.sh

and defined the new services adding in the file /etc/nagios/objects/services.cfg the following lines:

define service{
        use                             local-service
        host_name                       cream-41.pd.infn.it
        service_description             emi.cream.CEDIRECT-AllowedSubmission
        check_command                   ncg_check_native!/usr/libexec/grid-monit
oring/probes/emi.cream/cream_allowedSubmission.py!60!-x /tmp/dteam.proxy -p 8443
        normal_check_interval           6
        retry_check_interval            3
        max_check_attempts              2
        obsess_over_service             0
}
define service{
        use                             local-service
        host_name                       cream-41.pd.infn.it
        service_description             emi.cream.CEDIRECT-JobCancel
        check_command                   ncg_check_native!/usr/libexec/grid-monit
oring/probes/emi.cream/cream_jobCancel.py!60!-x /tmp/dteam.proxy -p 8443 -l lsf 
-q creamtest1 -j /usr/libexec/grid-monitoring/probes/emi.cream/sleep.jdl
        normal_check_interval           6
        retry_check_interval            3
        max_check_attempts              2
        obsess_over_service             0
}
define service{
        use                             local-service
        host_name                       cream-41.pd.infn.it
        service_description             emi.cream.CEDIRECT-JobPurge
        check_command                   ncg_check_native!/usr/libexec/grid-monit
oring/probes/emi.cream/cream_jobPurge.py!60!-x /tmp/dteam.proxy -p 8443 -l lsf -
q creamtest1 -j /usr/libexec/grid-monitoring/probes/emi.cream/hostname.jdl
        normal_check_interval           6
        retry_check_interval            3
        max_check_attempts              2
        obsess_over_service             0
}

define service{
        use                             local-service
        host_name                       cream-41.pd.infn.it
        service_description             emi.cream.CEDIRECT-ServiceInfo
        check_command                   ncg_check_native!/usr/libexec/grid-monit
oring/probes/emi.cream/cream_serviceInfo.py!60!-x /tmp/dteam.proxy -p 8443
        normal_check_interval           6
        retry_check_interval            3
        max_check_attempts              2
        obsess_over_service             0
}
define service{
        use                             local-service
        host_name                       cream-41.pd.infn.it
        service_description             emi.cream.CEDIRECT-JobSubmit
        check_command                   ncg_check_native!/usr/libexec/grid-monit
oring/probes/emi.cream/cream_jobSubmit.py!60!-x /tmp/dteam.proxy -p 8443 -l lsf 
-q creamtest1 -j /usr/libexec/grid-monitoring/probes/emi.cream/hostname.jdl --di
r /tmp 
        normal_check_interval           6
        retry_check_interval            3
        max_check_attempts              2
        obsess_over_service             0
}
define service{
        use                             local-service
        host_name                       cream-41.pd.infn.it
        service_description             emi.cream.WN-Softver
        check_command                   ncg_check_native!/usr/libexec/grid-monit
oring/probes/emi.cream/cream_jobSubmit.py!60!-x /tmp/dteam.proxy -p 8443 -l lsf 
-q creamtest1 -j /usr/libexec/grid-monitoring/probes/emi.cream/wn/WN-softver.jdl
 --dir /tmp
        normal_check_interval           6
        retry_check_interval            3
        max_check_attempts              2
        obsess_over_service             0
}
define service{
        use                             local-service
        host_name                       cream-41.pd.infn.it
        service_description             emi.cream.WN-Csh
        check_command                   ncg_check_native!/usr/libexec/grid-monit
oring/probes/emi.cream/cream_jobSubmit.py!60!-x /tmp/dteam.proxy -p 8443 -l lsf 
-q creamtest1 -j /usr/libexec/grid-monitoring/probes/emi.cream/wn/WN-csh.jdl --d
ir /tmp

        normal_check_interval           6
        retry_check_interval            3
        max_check_attempts              2
        obsess_over_service             0
}

The check_command ncg_check_native was defined in the file /etc/nagios/objects/commands.cfg as below:

define command{
        command_name                    ncg_check_native
        command_line                    $ARG1$ -H $HOSTNAME$ -t $ARG2$ $ARG3$
}

When the CREAM-CE is properly working the Nagios output is shown below:

If instead the CREAM-CE is not reachable (e.g. tomcat down in the CE), you will see:

and in the detail of the error message you will recognize that the connection was refused:

If instead e.g. the GRIDFTP server of the CREAM-CE is down, you will see:

i.e. only the probes not involving file transfers via GRIDFTP are still OK. Among the others, the ones waiting for the job reaching a terminal state will time out, while the WN-* will fail because these try to transfer the sandbox: in fact in the status information field you will recognize the GRIDFTP problem (ERROR - data_cb_read() - globus_xio: Unable to connect to cream-41.pd.infn.it:2811 )

-- LisaZangrando - 2014-01-16
-- MarcoVerlato - 2014-01-31

$ ls -la ./cream-41.pd.infn.it_8443_CREAM187996258
total 12
drwxr-xr-x   2 root root 4096 17 gen 16:20 .
dr-xr-x---. 23 root root 4096 17 gen 16:20 ..
-rw-------   1 root root    0 17 gen 16:20 std.err
-rw-------   1 root root   26 17 gen 16:20 std.out

1. cream_serviceInfo.py - get CREAM CE service info

1. cream_jobSubmit.py - submit a job directly to the selected CREAM CE

cream_serviceInfo.py

$ ./cream_serviceInfo.py --help
Usage: cream_serviceInfo.py [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname=HOSTNAME
                        The hostname of the CREAM service.
  -p PORT, --port=PORT  The port of the service. [default: none]
  -x PROXY, --proxy=PROXY
                        The proxy path
  -v, --verbose         verbose mode [default: False]
  -u URL, --url=URL     The status endpoint URL of the service. Example:
                        https://<host>[:<port>]

In order to get information about the CREAM service on the host https://cream-41.pd.infn.it:8443, use the following command:

$ ./cream_serviceInfo.py --url https://cream-41.pd.infn.it:8443
Interface Version  = [2.1]
Service Version    = [1.16.1 - EMI version: 3.5.1-1.el6]
Description        = [CREAM 2]
Started at         = [Thu Jan  2 19:06:51 2014]
Submission enabled = [YES]
Status             = [RUNNING]

or similary:

$ ./cream_serviceInfo.py --hostname cream-41.pd.infn.it --port 8443 
Interface Version  = [2.1]
Service Version    = [1.16.1 - EMI version: 3.5.1-1.el6]
Description        = [CREAM 2]
Started at         = [Thu Jan  2 19:06:51 2014]
Submission enabled = [YES]
Status             = [RUNNING]

cream_jobSubmit.py

$ ./cream_jobSubmit.py --help
Usage: cream_jobSubmit.py [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname=HOSTNAME
                        The hostname of the CREAM service.
  -p PORT, --port=PORT  The port of the service. [default: none]
  -u URL, --url=URL     The status endpoint URL of the service. Example:
                        https://<host>[:<port>]/cream-<lrms>-<queue>
  -x PROXY, --proxy=PROXY
                        The proxy path
  -v, --verbose         verbose mode [default: False]
  -l LRMS, --lrms=LRMS  The LRMS name (e.g.: 'lsf', 'pbs' etc)
  -q QUEUE, --queue=QUEUE
                        The queue name (e.g.: 'creamtest')
  -j JDL, --jdl=JDL     The jdl path

The --url (-u) directive must be used to target the probe to a specific CREAM CE identified by its identifier (i.e. CREAM CE ID). Alternatively is it possible to specify the CREAM CE identifier by using the --hostname , --port, --lrms and --queue options which are mutually exclusive with respect to the --url option.

Consider the JDL file hostname.jdl with the following content:

$ cat ./hostname.jdl
[
Type="Job";
JobType="Normal";
Executable = "/bin/hostname";
StdOutput = "std.out";
StdError = "std.err";
OutputSandbox = {"std.out","std.err"};
OutputSandboxBaseDestUri="gsiftp://localhost"
]

If verbose mode is disabled, the output should look like this:

$ ./cream_jobSubmit.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 --jdl ./hostname.jdl 
Job terminated with status DONE-OK

Notice: the use of the --url option is equivalent to specify both the options: --hostname, --port --lrms and --queue:

$ ./cream_jobSubmit.py --hostname cream-41.pd.infn.it --port 8443 --lrms lsf --queue creamtest1 --jdl ./hostname.jdl 
Job terminated with status DONE-OK

If the verbose mode is enabled, the output of the above command should be like to:

$ ./cream_jobSubmit.py --hostname cream-41.pd.infn.it --port 8443 --lrms lsf --queue creamtest1 --jdl ./hostname.jdl --verbose
executing command: /usr/bin/voms-proxy-info -timeleft
executing command: /usr/bin/glite-ce-job-submit -d -a -r cream-41.pd.infn.it:8443/cream-lsf-creamtest1 ./hostname.jdl
['2014-01-16 13:52:57,305 DEBUG - Using certificate proxy file [/tmp/x509up_u0]\n', '2014-01-16 13:52:57,324 DEBUG - VO from certificate=[dteam]\n', '2014-01-16 13:52:57,324 WARN - No configuration file suitable for loading. Using built-in configuration\n', '2014-01-16 13:52:57,324 DEBUG - Logfile is [/tmp/glite_cream_cli_logs/glite-ce-job-submit_CREAM_root_20140116-135257.log]\n', '2014-01-16 13:52:57,328 INFO - certUtil::generateUniqueID() - Generated DelegationID: [99b19aafc98e11bb7956cc58d901bd860697227d]\n', '2014-01-16 13:52:59,645 DEBUG - Registering to [https://cream-41.pd.infn.it:8443/ce-cream/services/CREAM2] JDL=[ StdOutput = "std.out"; BatchSystem = "lsf"; QueueName = "creamtest1"; Executable = "/bin/hostname"; Type = "Job"; JobType = "Normal"; OutputSandboxBaseDestUri = "gsiftp://localhost"; OutputSandbox = { "std.out","std.err" }; StdError = "std.err" ] - JDL File=[./hostname.jdl]\n', '2014-01-16 13:53:00,022 DEBUG - Will invoke JobStart for JobID [CREAM304354901]\n', 'https://cream-41.pd.infn.it:8443/CREAM304354901\n']
job id: https://cream-41.pd.infn.it:8443/CREAM304354901
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM304354901
job status: REALLY-RUNNING
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM304354901
job status: DONE-OK
invoking jobPurge
executing command: /usr/bin/glite-ce-job-purge --noint https://cream-41.pd.infn.it:8443/CREAM304354901
Job terminated with status DONE-OK

New CREAM-CE direct job submission metrics

The following metrics are a restructured version of the existing ones and try to provide a better approach for probing a CREAM CE.

1. cream_allowedSubmission.py - check if the submission to the selected CREAM CE is allowed
2. cream_jobCancel.py - cancel an active job.
3. cream_jobPurge.py - purge a terminted job.
4. cream_jobSubmit.py - submit a job directly to the selected CREAM CE
5. cream_serviceInfo.py - get CREAM CE service info

All of them have been implemented in python and are based on the cream-cli commands. They share the same logic structure and provide useful information about the their version, usage (i.e. help) including the options list and their meaning. For example:

$ ./cream_serviceInfo.py 
Usage: cream_serviceInfo.py [options]

cream_serviceInfo.py: error: Specify either option -u URL or option -H HOSTNAME (and -p PORT) or read the help (-h)

$ ./cream_serviceInfo.py --help
Usage: cream_serviceInfo.py [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname=HOSTNAME
                        The hostname of the CREAM service.
  -p PORT, --port=PORT  The port of the service. [default: none]
  -u URL, --url=URL     The status endpoint URL of the service. Example:
                        https://<host>[:<port>]
  -x PROXY, --proxy=PROXY
                        The proxy path
  -v, --verbose         verbose mode [default: False]

$ ./cream_serviceInfo.py --version
cream_serviceInfo v.1.0

The interaction with the CREAM CE requires the use of a valid VOMS proxy expressed by the X509_USER_PROXY env variable or through the --proxy option. All metrics check the existence of the proxy file and calculate the time left. In case of error, the related error message will be thrown:

$ ./cream_serviceInfo.py --hostname cream-41.pd.infn.it --port 8443 --proxy /tmp/x509up_u0 --verbose
Proxy file not found or not readable

The verbose mode (--verbose) could be enabled to each metric. It provides several details about the probe execution itself by highlighting the internal commands:

$ ./cream_serviceInfo.py --hostname cream-41.pd.infn.it --port 8443 --verbose
executing command: /usr/bin/voms-proxy-info -timeleft
invoking service info
executing command: /usr/bin/glite-ce-service-info cream-41.pd.infn.it:8443
Interface Version  = [2.1]
Service Version    = [1.16.1 - EMI version: 3.5.1-1.el6]
Description        = [CREAM 2]
Started at         = [Thu Jan  2 19:06:51 2014]
Submission enabled = [YES]
Status             = [RUNNING]

In case of mistakes on the selected options or on their values, the probe tries to explain what is wrong. For example the cream_serviceInfo doesn't support the --queue option:

$ ./cream_serviceInfo.py --hostname cream-41.pd.infn.it --port 8443 --queue creamtest1 --verbose
Usage: cream_serviceInfo.py [options]

cream_serviceInfo.py: error: no such option: --queue

In case of the errors in interacting with the CREAM CE, useful details will be provided about the failure:

$ ./cream_allowedSubmission.py --url https://cream-43.pd.infn.it:8443
command '/usr/bin/glite-ce-allowed-submission cream-43.pd.infn.it:8443' failed: return_code=1
details: ['2014-01-16 15:59:57,085 FATAL - Received NULL fault; the error is due to another cause: FaultString=[connection error] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] - FaultDetail=[Connection refused]\n']

cream_allowedSubmission

$ ./cream_allowedSubmission.py --help
Usage: cream_allowedSubmission.py [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname=HOSTNAME
                        The hostname of the CREAM service.
  -p PORT, --port=PORT  The port of the service. [default: none]
  -u URL, --url=URL     The status endpoint URL of the service. Example:
                        https://<host>[:<port>]
  -x PROXY, --proxy=PROXY
                        The proxy path
  -v, --verbose         verbose mode [default: False]

Notice: the use of the --url option is equivalent of specifying both the options: --hostname and --port:

$ ./cream_allowedSubmission.py --hostname cream-41.pd.infn.it --port 8443
ENABLED

$ ./cream_allowedSubmission.py --url https://cream-41.pd.infn.it:8443
ENABLED

The verbose mode highlights the internal commands:

$ ./cream_allowedSubmission.py --url https://cream-41.pd.infn.it:8443 --verbose
executing command: /usr/bin/voms-proxy-info -timeleft
invoking allowedSubmission
executing command: /usr/bin/glite-ce-allowed-submission cream-41.pd.infn.it:8443
ENABLED

cream_jobCancel.py

$ ./cream_jobCancel.py --help
Usage: cream_jobCancel.py [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname=HOSTNAME
                        The hostname of the CREAM service.
  -p PORT, --port=PORT  The port of the service. [default: none]
  -x PROXY, --proxy=PROXY
                        The proxy path
  -v, --verbose         verbose mode [default: False]
  -u URL, --url=URL     The status endpoint URL of the service. Example:
                        https://<host>[:<port>]/cream-<lrms>-<queue>
  -l LRMS, --lrms=LRMS  The LRMS name (e.g.: 'lsf', 'pbs' etc)
  -q QUEUE, --queue=QUEUE
                        The queue name (e.g.: 'creamtest')
  -j JDL, --jdl=JDL     The jdl path

$ ./cream_jobCancel.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 --jdl ./hostname.jdl 
job already terminated

$ cat ./sleep.jdl 
[
Type="Job";
JobType="Normal";
Executable = "/bin/sleep";
Arguments = "200";
StdOutput = "cream.out";
StdError = "cream.out";
]

$ ./cream_jobCancel.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 --jdl ./sleep.jdl 
job cancelled

$ ./cream_jobCancel.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 --jdl ./sleep.jdl --verbose
executing command: /usr/bin/voms-proxy-info -timeleft
executing command: /usr/bin/glite-ce-job-submit -d -a -r cream-41.pd.infn.it:8443/cream-lsf-creamtest1 ./sleep.jdl
['2014-01-16 17:22:49,728 DEBUG - Using certificate proxy file [/tmp/x509up_u0]\n', '2014-01-16 17:22:49,744 DEBUG - VO from certificate=[dteam]\n', '2014-01-16 17:22:49,745 WARN - No configuration file suitable for loading. Using built-in configuration\n', '2014-01-16 17:22:49,745 DEBUG - Logfile is [/tmp/glite_cream_cli_logs/glite-ce-job-submit_CREAM_root_20140116-172249.log]\n', '2014-01-16 17:22:49,747 INFO - certUtil::generateUniqueID() - Generated DelegationID: [69e3bdaf4e818e1f71f1b7ff442f74583c869b84]\n', '2014-01-16 17:22:52,165 DEBUG - Registering to [https://cream-41.pd.infn.it:8443/ce-cream/services/CREAM2] JDL=[ StdOutput = "cream.out"; BatchSystem = "lsf"; QueueName = "creamtest1"; Executable = "/bin/sleep"; Type = "Job"; Arguments = "200"; JobType = "Normal"; StdError = "cream.out" ] - JDL File=[./sleep.jdl]\n', '2014-01-16 17:22:52,513 DEBUG - Will invoke JobStart for JobID [CREAM437649288]\n', 'https://cream-41.pd.infn.it:8443/CREAM437649288\n']
job id: https://cream-41.pd.infn.it:8443/CREAM437649288
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM437649288
job status: REALLY-RUNNING
invoking jobCancel
executing command: /usr/bin/glite-ce-job-cancel --noint https://cream-41.pd.infn.it:8443/CREAM437649288
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM437649288
job status: REALLY-RUNNING
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM437649288
job status: CANCELLED
invoking jobPurge
executing command: /usr/bin/glite-ce-job-purge --noint https://cream-41.pd.infn.it:8443/CREAM437649288
job cancelled

cream_jobPurge.py

$ ./cream_jobPurge.py --help
Usage: cream_jobPurge.py [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname=HOSTNAME
                        The hostname of the CREAM service.
  -p PORT, --port=PORT  The port of the service. [default: none]
  -x PROXY, --proxy=PROXY
                        The proxy path
  -v, --verbose         verbose mode [default: False]
  -u URL, --url=URL     The status endpoint URL of the service. Example:
                        https://<host>[:<port>]/cream-<lrms>-<queue>
  -l LRMS, --lrms=LRMS  The LRMS name (e.g.: 'lsf', 'pbs' etc)
  -q QUEUE, --queue=QUEUE
                        The queue name (e.g.: 'creamtest')
  -j JDL, --jdl=JDL     The jdl path

$ ./cream_jobPurge.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 --jdl ./hostname.jdl 
job purged

$ ./cream_jobPurge.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 --jdl ./hostname.jdl --verbose
executing command: /usr/bin/voms-proxy-info -timeleft
executing command: /usr/bin/glite-ce-job-submit -d -a -r cream-41.pd.infn.it:8443/cream-lsf-creamtest1 ./hostname.jdl
['2014-01-16 14:02:48,282 DEBUG - Using certificate proxy file [/tmp/x509up_u0]\n', '2014-01-16 14:02:48,301 DEBUG - VO from certificate=[dteam]\n', '2014-01-16 14:02:48,301 WARN - No configuration file suitable for loading. Using built-in configuration\n', '2014-01-16 14:02:48,302 DEBUG - Logfile is [/tmp/glite_cream_cli_logs/glite-ce-job-submit_CREAM_root_20140116-140248.log]\n', '2014-01-16 14:02:48,305 INFO - certUtil::generateUniqueID() - Generated DelegationID: [85a80615fc8046baf6dce2a27b708fc82ecbce55]\n', '2014-01-16 14:02:51,105 DEBUG - Registering to [https://cream-41.pd.infn.it:8443/ce-cream/services/CREAM2] JDL=[ StdOutput = "std.out"; BatchSystem = "lsf"; QueueName = "creamtest1"; Executable = "/bin/hostname"; Type = "Job"; JobType = "Normal"; OutputSandboxBaseDestUri = "gsiftp://localhost"; OutputSandbox = { "std.out","std.err" }; StdError = "std.err" ] - JDL File=[./hostname.jdl]\n', '2014-01-16 14:02:51,504 DEBUG - Will invoke JobStart for JobID [CREAM691625071]\n', 'https://cream-41.pd.infn.it:8443/CREAM691625071\n']
job id: https://cream-41.pd.infn.it:8443/CREAM691625071
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM691625071
job status: DONE-OK
invoking jobPurge
executing command: /usr/bin/glite-ce-job-purge --noint https://cream-41.pd.infn.it:8443/CREAM691625071
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM691625071
job purged

cream_jobSubmit.py

$ ./cream_jobSubmit.py --help
Usage: cream_jobSubmit.py [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname=HOSTNAME
                        The hostname of the CREAM service.
  -p PORT, --port=PORT  The port of the service. [default: none]
  -u URL, --url=URL     The status endpoint URL of the service. Example:
                        https://<host>[:<port>]/cream-<lrms>-<queue>
  -x PROXY, --proxy=PROXY
                        The proxy path
  -v, --verbose         verbose mode [default: False]
  -l LRMS, --lrms=LRMS  The LRMS name (e.g.: 'lsf', 'pbs' etc)
  -q QUEUE, --queue=QUEUE
                        The queue name (e.g.: 'creamtest')
  -j JDL, --jdl=JDL     The jdl path

The JDL used for this example asks about the CREAM hostname:

$ cat ./hostname.jdl
[
Type="Job";
JobType="Normal";
Executable = "/bin/hostname";
StdOutput = "std.out";
StdError = "std.err";
OutputSandbox = {"std.out","std.err"};
OutputSandboxBaseDestUri="gsiftp://localhost"
]

The probe execution with the verbose mode disabled:

$ ./cream_jobSubmit.py --url https://cream-41.pd.infn.it:8443/cream-lsf-creamtest1 --jdl ./hostname.jdl 
Job terminated with status DONE-OK

Notice: the use of the --url option is equivalent of specifying both the options: --hostname, --port --lrms and --queue:

$ ./cream_jobSubmit.py --hostname cream-41.pd.infn.it --port 8443 --lrms lsf --queue creamtest1 --jdl ./hostname.jdl 
Job terminated with status DONE-OK

The execution of the above command with the verbose mode enabled:

$ ./cream_jobSubmit.py --hostname cream-41.pd.infn.it --port 8443 --lrms lsf --queue creamtest1 --jdl ./hostname.jdl --verbose
executing command: /usr/bin/voms-proxy-info -timeleft
executing command: /usr/bin/glite-ce-job-submit -d -a -r cream-41.pd.infn.it:8443/cream-lsf-creamtest1 ./hostname.jdl
['2014-01-16 13:52:57,305 DEBUG - Using certificate proxy file [/tmp/x509up_u0]\n', '2014-01-16 13:52:57,324 DEBUG - VO from certificate=[dteam]\n', '2014-01-16 13:52:57,324 WARN - No configuration file suitable for loading. Using built-in configuration\n', '2014-01-16 13:52:57,324 DEBUG - Logfile is [/tmp/glite_cream_cli_logs/glite-ce-job-submit_CREAM_root_20140116-135257.log]\n', '2014-01-16 13:52:57,328 INFO - certUtil::generateUniqueID() - Generated DelegationID: [99b19aafc98e11bb7956cc58d901bd860697227d]\n', '2014-01-16 13:52:59,645 DEBUG - Registering to [https://cream-41.pd.infn.it:8443/ce-cream/services/CREAM2] JDL=[ StdOutput = "std.out"; BatchSystem = "lsf"; QueueName = "creamtest1"; Executable = "/bin/hostname"; Type = "Job"; JobType = "Normal"; OutputSandboxBaseDestUri = "gsiftp://localhost"; OutputSandbox = { "std.out","std.err" }; StdError = "std.err" ] - JDL File=[./hostname.jdl]\n', '2014-01-16 13:53:00,022 DEBUG - Will invoke JobStart for JobID [CREAM304354901]\n', 'https://cream-41.pd.infn.it:8443/CREAM304354901\n']
job id: https://cream-41.pd.infn.it:8443/CREAM304354901
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM304354901
job status: REALLY-RUNNING
invoking jobStatus
executing command: /usr/bin/glite-ce-job-status https://cream-41.pd.infn.it:8443/CREAM304354901
job status: DONE-OK
invoking jobPurge
executing command: /usr/bin/glite-ce-job-purge --noint https://cream-41.pd.infn.it:8443/CREAM304354901
Job terminated with status DONE-OK

cream_serviceInfo.py

$ ./cream_serviceInfo.py --help
Usage: cream_serviceInfo.py [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname=HOSTNAME
                        The hostname of the CREAM service.
  -p PORT, --port=PORT  The port of the service. [default: none]
  -x PROXY, --proxy=PROXY
                        The proxy path
  -v, --verbose         verbose mode [default: False]
  -u URL, --url=URL     The status endpoint URL of the service. Example:
                        https://<host>[:<port>]

$ ./cream_serviceInfo.py --url https://cream-41.pd.infn.it:8443
Interface Version  = [2.1]
Service Version    = [1.16.1 - EMI version: 3.5.1-1.el6]
Description        = [CREAM 2]
Started at         = [Thu Jan  2 19:06:51 2014]
Submission enabled = [YES]
Status             = [RUNNING]

-- LisaZangrando - 2014-01-16