WMS Test Plan
YAIM-WMS Configuration Testing
WMS Job Submission/GetOutput Testing
Registers the job into the wmproxy, delivers it to the WM that submits it to the grid.
Test job submission with the following type of jobs/jdl:
Normal Job
- to JC
[
Type = "Job";
JobType = "normal";
InputSandbox = { "file://$HOME/test.sh"};
VirtualOrganisation = "dteam";
Executable="test.sh";
Arguments="Hello ";
Requirements = ( !RegExp("/cream-",other.GlueCEUniqueID));
Rank = 0;
fuzzyrank = true;
StdOutput="message.txt";
StdError="err.log";
OutputSandbox={"message.txt","err.log",".BrokerInfo"};
RetryCount = 0;
usertags = [ jdl = "normal job" ];
ShallowRetryCount = 3;
]
- to ICE
[
Type = "Job";
JobType = "normal";
InputSandbox = { "file:///home/emolinari/test.sh"};
VirtualOrganisation = "dteam";
Executable="test.sh";
Arguments="Hello ";
Requirements = ( RegExp("/cream-",other.GlueCEUniqueID));
Rank = 0;
fuzzyrank = true;
StdOutput="message.txt";
StdError="err.log";
OutputSandbox={"message.txt","err.log",".BrokerInfo"};
usertags = [ jdl = "normal job to ICE" ];
RetryCount = 0;
ShallowRetryCount = 3;
]
Perusal job
Job perusal is the ability to view output from a job while it is running. In order to use this feature the JDL attributes
PerusalFileEnable and
PerusalTimeInterval must be declared when the job is submitted for example:
PerusalFileEnable = true;
PerusalTimeInterval = 120;
This makes the WN upload, at regular time intervals (
PerusalTimeInterval), a copy of the output files specified using the glite-wms-job-perusal --set command to the WMS machine.
- Following an example on how to test a perusal job:
- Given the jdl file:
[
Executable = "perusal.sh" ;
InputSandbox = {"file://$HOME/perusal.sh"};
Stdoutput = "perusal.out" ;
StdError = "perusal.err" ;
OutputSandbox ={ "perusal.out" ,"perusal.err"} ;
usertags = [ jdl = "perusal" ];
PerusalFileEnable = true;
]
where perusal.sh is #!/bin/bash
echo "Hi all, this is a perusal test program..."
for (( i=0 ; $i < 50 ; i = $(($i + 1)) )); do
echo "Step number $i"
date
sleep 10
done
echo ""
echo "Bye"
- Submit the job with
glite-wms-job-submit --config glite_wms.conf -a perusal.jdl
- Given the jobid of the previously submitted set the files that are to be perused with
glite-wms-job-perusal --set -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg
- When the job starts running on the WN get the files as here:
glite-wms-job-perusal --get -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg
DAG job
to JC and IC as above specifying Requirements = ( RegExp("/cream-",other.GlueCEUniqueID)); or Requirements = (
RegExp("/cream-",other.GlueCEUniqueID)); respectively in the job jdl
- DAG job jdls samples:
[
Type = "dag";
VirtualOrganisation = "dteam";
Max_nodes_running = 10;
InputSandbox = "test.sh";
FuzzyRank = true;
Nodes = [
nodeA = [
file= "test_dag.jdl";
];
nodeB = [
file= "test_dag.jdl";
];
nodeC = [
file= "test_dag.jdl";
];
nodeD = [
file= "test_dag.jdl";
];
nodeE = [
file= "test_dag.jdl";
];
nodeF= [
file= "test_dag.jdl";
];
];
Dependencies = {
{{nodeA, nodeB}, nodeC},{nodeD,nodeE.nodeF}
}
]
[
type = "dag";
DefaultNodeShallowRetryCount = 3;
nodes = [
nodeA = [
node_type = "edg-jdl";
file ="myjob.jdl" ;
];
nodeB = [
node_type = "edg-jdl";
file ="myjob.jdl" ;
];
nodeC = [
node_type = "edg-jdl";
file ="myjob.jdl" ;
];
dependencies = {
{ nodeA, nodeB },
{ nodeA, nodeC }
}
];
]
Parametric Job
Submit it to ICE and JC. Following a sample jdl:
// Test parametric job.
// The standar Output and Error are parametrizied such as the
// enviroment variable: MYPATH_PARAM_.
[
JobType = "parametric";
Executable = "/usr/bin/env";
Environment = {"MYPATH_PARAM_=$PATH:/bin:/usr/bin:$HOME"};
StdOutput = "echo_PARAM_.out";
StdError = "echo_PARAM_.err";
OutputSandbox = {"echo_PARAM_.out","echo_PARAM_.err"};
Requirements = ( RegExp("/cream-",other.GlueCEUniqueID));
Parameters = 5;
usertags = [ jdl = "parametric cream" ];
]
Collection Job
WMS Job shallow and deep re-submission
WMS Job List-match Testing
With data
- Set these variables:
export LCG_GFAL_INFOSYS=<BDII set in the wms conf file> (e.g. cert-bdii-04.cnaf.infn.it:2170)
export LFC_HOST=<lfc host name> (e.g. lfcserver.cnaf.infn.it)
export LFC_HOME=<lfc home directory> (e.g. lfcserver.cnaf.infn.it:/grid/infngrid)
- Register a file to a SE using lcg-cr command:
lcg-cr --vo <VO name> -d <SE host> -l lfn:<logical file name> file:<local file path>
- Using the lcg-rep you can create some replicas:
lcg-rep --vo <VO name> -d <SE host> lfn:<logical file name>
- lcg-lr is useful to see the list of replicas:
lcg-lr --vo <VO name> lfn:<logical file name>
- Submit a list-match command with a jdl likes this one (as InputData put the lfn(s) registered before):
###########################################
# JDL with Data Requirements #
###########################################
Executable = "calc-pi.sh";
Arguments = "1000";
StdOutput = "std.out";
StdError = "std.err";
Prologue = "prologue.sh";
InputSandbox = {"calc-pi.sh", "fileA", "fileB","prologue.sh"};
OutputSandbox = {"std.out", "std.err","out-PI.txt","out-e.txt"};
Requirements = true;
DataRequirements = {
[
DataCatalogType = "DLI";
DataCatalog = "http://lfcserver.cnaf.infn.it:8085";
InputData = {"lfn:/grid/infngrid/cesini/PI_1M.txt","lfn:/grid/infngrid/cesini/e-2M.txt"};
]
};
DataAccessProtocol = "gsiftp";
- You should obtain an output like this one:
[ale@cream-15 DataReq]$ glite-wms-job-list-match -a --rank --config ~/UI/etc/wmp_wms007.conf data-req.jdl
Connecting to the service https://wms007.cnaf.infn.it:7443/glite_wms_wmproxy_server
==========================================================================
COMPUTING ELEMENT IDs LIST
The following CE(s) matching your job requirements have been found:
*CEId* *Rank*
- cert-15.pd.infn.it:8443/cream-lsf-cert 0
- gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-cert 0
- prod-ce-01.pd.infn.it:8443/cream-lsf-cert 0
- prod-ce-02.pd.infn.it:2119/jobmanager-lcglsf-cert 0
- t2-ce-01.to.infn.it:8443/cream-pbs-cert 0
- t2-ce-01.to.infn.it:8443/cream-pbs-short 0
- t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-cert 0
- t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-short 0
- test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cert 0
- test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-parallel 0
- devce.cnaf.infn.it:8443/cream-pbs-cert -700634
==========================================================================
- Finally you can check if these CEs are really closed to the SEs where you replicate your file(s)
- First list the SEs:
[ale@cream-15 DataReq]$ lcg-lr --vo dteam lfn:/grid/infngrid/cesini/PI_1M.txt
srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/infngrid/generated/2009-11-24/file0b6a613c-e1f1-4e53-a328-2152e44a8576
srm://gridit-se-01.cnaf.infn.it/dpm/cnaf.infn.it/home/infngrid/generated/2009-11-24/fileb9f3b972-806c-48a6-92fc-a9677901403a
srm://prod-se-01.pd.infn.it/infngrid/generated/2009-11-24/file10b5fe11-ad95-41e6-a3a3-14bc5cb9694b
srm://t2-se-00.to.infn.it/dpm/to.infn.it/home/infngrid/generated/2009-11-24/file1bb28727-1356-4fb1-bb73-5362f93e766f
- then for each SEs in the list use this command to find the "Close" CEs:
ldapsearch -x -h <BDII set in the WMS conf file> -p 2170 -b mds-vo-name=local,o=grid "(&(objectclass=GlueCESEBindGroup)(GlueCESEBindGroupSEUniqueID=<SE HOST>))" | grep ^GlueCESEBindGroupCEUniqueID
GlueCESEBindGroupCEUniqueID: devce.cnaf.infn.it:8443/cream-pbs-cert
GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
GlueCESEBindGroupCEUniqueID: cert-08.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: cert-05.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-par
GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
GlueCESEBindGroupCEUniqueID: cert-07.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: cert-09.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
GlueCESEBindGroupCEUniqueID: cert-06.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: cert-04.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cer
GlueCESEBindGroupCEUniqueID: cert-13.cnaf.infn.it:8443/cream-lsf-pps
- You can find more CEs because you should also filter using the "vo" attribute.
--
ElisabettaMolinari - 2010-02-24