WMS Test Plan

WMS Test Plan

YAIM-WMS Configuration Testing

WMS Job Submission/GetOutput Testing

Registers the job into the wmproxy, delivers it to the WM that submits it to the grid. Test job submission with the following type of jobs/jdl:

Normal Job

to JC

[
Type = "Job";
JobType = "normal";
InputSandbox = { "file://$HOME/test.sh"};
VirtualOrganisation = "dteam";
Executable="test.sh";
Arguments="Hello ";
Requirements = ( !RegExp("/cream-",other.GlueCEUniqueID));
Rank = 0;
fuzzyrank = true;
StdOutput="message.txt";
StdError="err.log";
OutputSandbox={"message.txt","err.log",".BrokerInfo"};
RetryCount = 0;
usertags = [ jdl = "normal job" ];
ShallowRetryCount = 3;
]

to ICE

[
Type = "Job";
JobType = "normal";
InputSandbox = { "file:///home/emolinari/test.sh"};
VirtualOrganisation = "dteam";
Executable="test.sh";
Arguments="Hello ";
Requirements = ( RegExp("/cream-",other.GlueCEUniqueID));
Rank = 0;
fuzzyrank = true;
StdOutput="message.txt";
StdError="err.log";
OutputSandbox={"message.txt","err.log",".BrokerInfo"};
usertags = [ jdl = "normal job to ICE" ];
RetryCount = 0;
ShallowRetryCount = 3;
]

Perusal job

Job perusal is the ability to view output from a job while it is running. In order to use this feature the JDL attributes PerusalFileEnable and PerusalTimeInterval must be declared when the job is submitted for example:

PerusalFileEnable = true;
PerusalTimeInterval = 120;

This makes the WN upload, at regular time intervals (PerusalTimeInterval), a copy of the output files specified using the glite-wms-job-perusal --set command to the WMS machine.

Following an example on how to test a perusal job:

Given the jdl file:

[
Executable = "perusal.sh" ;
InputSandbox = {"file://$HOME/perusal.sh"};
Stdoutput = "perusal.out" ;
StdError =  "perusal.err" ;
OutputSandbox ={ "perusal.out" ,"perusal.err"} ;
usertags = [ jdl = "perusal" ];
PerusalFileEnable = true;
]

where perusal.sh is

#!/bin/bash

echo "Hi all, this is a perusal test program..."

for (( i=0 ; $i < 50 ; i = $(($i + 1)) )); do

        echo "Step number $i"
        date
        sleep 10

done

echo ""
echo "Bye"

Submit the job with

 glite-wms-job-submit --config glite_wms.conf -a perusal.jdl

Given the jobid of the previously submitted set the files that are to be perused with

glite-wms-job-perusal --set -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg

When the job starts running on the WN get the files as here:

glite-wms-job-perusal --get -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg

DAG job

to JC and IC as above specifying Requirements = ( RegExp("/cream-",other.GlueCEUniqueID)); or Requirements = ( RegExp("/cream-",other.GlueCEUniqueID)); respectively in the job jdl

DAG job jdls samples:

[
Type = "dag";
VirtualOrganisation = "dteam";
Max_nodes_running = 10;
InputSandbox = "test.sh";
FuzzyRank = true;
Nodes = [
nodeA = [
file= "test_dag.jdl";
];
nodeB = [
file= "test_dag.jdl";
];
nodeC = [
file= "test_dag.jdl";
];
nodeD = [
file= "test_dag.jdl";
];
nodeE = [
file= "test_dag.jdl";
];
nodeF= [
file= "test_dag.jdl";
];
];
Dependencies = {
{{nodeA, nodeB}, nodeC},{nodeD,nodeE.nodeF}
}
]

[
    type = "dag";
    DefaultNodeShallowRetryCount = 3;
        nodes = [
      nodeA = [
        node_type = "edg-jdl";
        file ="myjob.jdl" ;
      ];
        nodeB = [
                                node_type = "edg-jdl";
        file ="myjob.jdl" ;
       ];
         nodeC = [
         node_type = "edg-jdl";
         file ="myjob.jdl" ;
       ];
                        dependencies = {
        { nodeA, nodeB },
                                { nodeA, nodeC }
       }
     ];
  ]

Parametric Job

Submit it to ICE and JC. Following a sample jdl:

// Test parametric job.
// The standar Output and Error are parametrizied such as the
// enviroment variable: MYPATH_PARAM_.
 [
  JobType = "parametric";
  Executable = "/usr/bin/env";
  Environment = {"MYPATH_PARAM_=$PATH:/bin:/usr/bin:$HOME"};
  StdOutput = "echo_PARAM_.out";
  StdError = "echo_PARAM_.err";
  OutputSandbox = {"echo_PARAM_.out","echo_PARAM_.err"};
  Requirements = ( RegExp("/cream-",other.GlueCEUniqueID));
  Parameters =  5;
        usertags = [ jdl = "parametric cream" ];
 ]

Collection Job

WMS Job shallow and deep re-submission

For the shallow resubmission add the following attributes to your favourite jdl file:
```
Prologue = "/bin/false"; ShallowRetryCount = 3;
```
For the deep resubmission add the following attributes:
```
 Prologue = "/bin/false"; RetryCount = 3;
```

WMS Job List-match Testing

With data

Set these variables:

  export LCG_GFAL_INFOSYS=<BDII set in the wms conf file> (e.g. cert-bdii-04.cnaf.infn.it:2170)
  export LFC_HOST=<lfc host name> (e.g. lfcserver.cnaf.infn.it)
  export LFC_HOME=<lfc home directory> (e.g. lfcserver.cnaf.infn.it:/grid/infngrid)

  lcg-cr --vo <VO name> -d <SE host> -l lfn:<logical file name> file:<local file path>

Using the lcg-rep you can create some replicas:

  lcg-rep  --vo <VO name> -d <SE host> lfn:<logical file name>

lcg-lr is useful to see the list of replicas:

  lcg-lr --vo <VO name>  lfn:<logical file name>

Submit a list-match command with a jdl likes this one (as InputData put the lfn(s) registered before):

###########################################
#      JDL with Data Requirements         #
###########################################

Executable = "calc-pi.sh";
Arguments = "1000";
StdOutput = "std.out";
StdError = "std.err";
Prologue = "prologue.sh";
InputSandbox = {"calc-pi.sh", "fileA", "fileB","prologue.sh"};
OutputSandbox = {"std.out", "std.err","out-PI.txt","out-e.txt"};
Requirements = true;

DataRequirements = {
[
DataCatalogType = "DLI";
DataCatalog = "http://lfcserver.cnaf.infn.it:8085";
InputData = {"lfn:/grid/infngrid/cesini/PI_1M.txt","lfn:/grid/infngrid/cesini/e-2M.txt"};
]
};
DataAccessProtocol = "gsiftp";

You should obtain an output like this one:

[ale@cream-15 DataReq]$ glite-wms-job-list-match -a --rank --config ~/UI/etc/wmp_wms007.conf data-req.jdl
Connecting to the service https://wms007.cnaf.infn.it:7443/glite_wms_wmproxy_server

==========================================================================

         COMPUTING ELEMENT IDs LIST
 The following CE(s) matching your job requirements have been found:

  *CEId*                                                   *Rank*

 - cert-15.pd.infn.it:8443/cream-lsf-cert                           0
 - gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-cert           0
 - prod-ce-01.pd.infn.it:8443/cream-lsf-cert                        0
 - prod-ce-02.pd.infn.it:2119/jobmanager-lcglsf-cert                0
 - t2-ce-01.to.infn.it:8443/cream-pbs-cert                          0
 - t2-ce-01.to.infn.it:8443/cream-pbs-short                         0
 - t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-cert                  0
 - t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-short                 0
 - test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cert               0
 - test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-parallel           0
 - devce.cnaf.infn.it:8443/cream-pbs-cert                           -700634

==========================================================================

Finally you can check if these CEs are really closed to the SEs where you replicate your file(s)

First list the SEs:

[ale@cream-15 DataReq]$ lcg-lr --vo dteam  lfn:/grid/infngrid/cesini/PI_1M.txt

srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/infngrid/generated/2009-11-24/file0b6a613c-e1f1-4e53-a328-2152e44a8576
srm://gridit-se-01.cnaf.infn.it/dpm/cnaf.infn.it/home/infngrid/generated/2009-11-24/fileb9f3b972-806c-48a6-92fc-a9677901403a
srm://prod-se-01.pd.infn.it/infngrid/generated/2009-11-24/file10b5fe11-ad95-41e6-a3a3-14bc5cb9694b
srm://t2-se-00.to.infn.it/dpm/to.infn.it/home/infngrid/generated/2009-11-24/file1bb28727-1356-4fb1-bb73-5362f93e766f

then for each SEs in the list use this command to find the "Close" CEs:

ldapsearch -x -h <BDII set in the WMS conf file> -p 2170 -b mds-vo-name=local,o=grid "(&(objectclass=GlueCESEBindGroup)(GlueCESEBindGroupSEUniqueID=<SE HOST>))" | grep ^GlueCESEBindGroupCEUniqueID

GlueCESEBindGroupCEUniqueID: devce.cnaf.infn.it:8443/cream-pbs-cert
GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
GlueCESEBindGroupCEUniqueID: cert-08.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: cert-05.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-par
GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
GlueCESEBindGroupCEUniqueID: cert-07.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: cert-09.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
GlueCESEBindGroupCEUniqueID: cert-06.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: cert-04.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cer
GlueCESEBindGroupCEUniqueID: cert-13.cnaf.infn.it:8443/cream-lsf-pps

You can find more CEs because you should also filter using the "vo" attribute.

-- ElisabettaMolinari - 2010-02-24

Topic revision: r5 - 2010-03-02 - AlessioGianelle

Account
- Log In

~~Edit~~
~~Attach~~