WMS Test Plan
YAIM-WMS Configuration Testing
WMS Job Submission/GetOutput Testing
Registers the job into the wmproxy, delivers it to the WM that submits it to the grid.
Test job submission with the following type of jobs/jdl:
Normal Job
- to JC
[
Type = "Job";
JobType = "normal";
InputSandbox = { "file://$HOME/test.sh"};
VirtualOrganisation = "dteam";
Executable="test.sh";
Arguments="Hello ";
Requirements = ( !RegExp("/cream-",other.GlueCEUniqueID));
Rank = 0;
fuzzyrank = true;
StdOutput="message.txt";
StdError="err.log";
OutputSandbox={"message.txt","err.log",".BrokerInfo"};
RetryCount = 0;
usertags = [ jdl = "normal job" ];
ShallowRetryCount = 3;
]
- to ICE
[
Type = "Job";
JobType = "normal";
InputSandbox = { "file:///home/emolinari/test.sh"};
VirtualOrganisation = "dteam";
Executable="test.sh";
Arguments="Hello ";
Requirements = ( RegExp("/cream-",other.GlueCEUniqueID));
Rank = 0;
fuzzyrank = true;
StdOutput="message.txt";
StdError="err.log";
OutputSandbox={"message.txt","err.log",".BrokerInfo"};
usertags = [ jdl = "normal job to ICE" ];
RetryCount = 0;
ShallowRetryCount = 3;
]
Perusal job
Job perusal is the ability to view output from a job while it is running. In order to use this feature the JDL attributes
PerusalFileEnable and
PerusalTimeInterval must be declared when the job is submitted for example:
PerusalFileEnable = true;
PerusalTimeInterval = 120;
This makes the WN upload, at regular time intervals (
PerusalTimeInterval), a copy of the output files specified using the glite-wms-job-perusal --set command to the WMS machine.
- Following an example on how to test a perusal job:
- Given the jdl file:
[
Executable = "perusal.sh" ;
InputSandbox = {"file://$HOME/perusal.sh"};
Stdoutput = "perusal.out" ;
StdError = "perusal.err" ;
OutputSandbox ={ "perusal.out" ,"perusal.err"} ;
usertags = [ jdl = "perusal" ];
PerusalFileEnable = true;
]
where perusal.sh is #!/bin/bash
echo "Hi all, this is a perusal test program..."
for (( i=0 ; $i < 50 ; i = $(($i + 1)) )); do
echo "Step number $i"
date
sleep 10
done
echo ""
echo "Bye"
- Submit the job with
glite-wms-job-submit --config glite_wms.conf -a perusal.jdl
- Given the jobid of the previously submitted set the files that are to be perused with
glite-wms-job-perusal --set -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg
- When the job starts running on the WN get the files as here:
glite-wms-job-perusal --get -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg
DAG job
- DAG job jdls samples:
[
Type = "dag";
VirtualOrganisation = "dteam";
Max_nodes_running = 10;
InputSandbox = "test.sh";
FuzzyRank = true;
Nodes = [
nodeA = [
file= "test_dag.jdl";
];
nodeB = [
file= "test_dag.jdl";
];
nodeC = [
file= "test_dag.jdl";
];
nodeD = [
file= "test_dag.jdl";
];
nodeE = [
file= "test_dag.jdl";
];
nodeF= [
file= "test_dag.jdl";
];
];
Dependencies = {
{{nodeA, nodeB}, nodeC},{nodeD,nodeE.nodeF}
}
]
[
type = "dag";
DefaultNodeShallowRetryCount = 3;
nodes = [
nodeA = [
node_type = "edg-jdl";
file ="myjob.jdl" ;
];
nodeB = [
node_type = "edg-jdl";
file ="myjob.jdl" ;
];
nodeC = [
node_type = "edg-jdl";
file ="myjob.jdl" ;
];
dependencies = {
{ nodeA, nodeB },
{ nodeA, nodeC }
}
];
]
Parametric Job
Submit it to ICE and JC. Following a sample jdl:
// Test parametric job.
// The standar Output and Error are parametrizied such as the
// enviroment variable: MYPATH_PARAM_.
[
JobType = "parametric";
Executable = "/usr/bin/env";
Environment = {"MYPATH_PARAM_=$PATH:/bin:/usr/bin:$HOME"};
StdOutput = "echo_PARAM_.out";
StdError = "echo_PARAM_.err";
OutputSandbox = {"echo_PARAM_.out","echo_PARAM_.err"};
Requirements = ( RegExp("/cream-",other.GlueCEUniqueID));
Parameters = 5;
usertags = [ jdl = "parametric cream" ];
]
Collection Job
WMS Job shallow and deep re-submission
WMS Job List-match Testing
With data
- Set these variables:
export LCG_GFAL_INFOSYS=<BDII set in the wms conf file> (e.g. cert-bdii-04.cnaf.infn.it:2170)
export LFC_HOST=<lfc host name> (e.g. lfcserver.cnaf.infn.it)
export LFC_HOME=<lfc home directory> (e.g. lfcserver.cnaf.infn.it:/grid/infngrid)
- Register a file to a SE using lcg-cr command:
lcg-cr --vo <VO name> -d <SE host> -l lfn:<logical file name> file:<local file path>
- Using the lcg-rep you can create some replicas:
lcg-rep --vo <VO name> -d <SE host> lfn:<logical file name>
- lcg-lr is useful to see the list of replicas:
lcg-lr --vo <VO name> lfn:<logical file name>
- Submit a list-match command with a jdl likes this one (as InputData put the lfn(s) registered before):
###########################################
# JDL with Data Requirements #
###########################################
Executable = "calc-pi.sh";
Arguments = "1000";
StdOutput = "std.out";
StdError = "std.err";
Prologue = "prologue.sh";
InputSandbox = {"calc-pi.sh", "fileA", "fileB","prologue.sh"};
OutputSandbox = {"std.out", "std.err","out-PI.txt","out-e.txt"};
Requirements = true;
DataRequirements = {
[
DataCatalogType = "DLI";
DataCatalog = "http://lfcserver.cnaf.infn.it:8085";
InputData = {"lfn:/grid/infngrid/cesini/PI_1M.txt","lfn:/grid/infngrid/cesini/e-2M.txt"};
]
};
DataAccessProtocol = "gsiftp";
- You should obtain an output like this one:
[ale@cream-15 DataReq]$ glite-wms-job-list-match -a --rank --config ~/UI/etc/wmp_wms007.conf data-req.jdl
Connecting to the service https://wms007.cnaf.infn.it:7443/glite_wms_wmproxy_server
==========================================================================
COMPUTING ELEMENT IDs LIST
The following CE(s) matching your job requirements have been found:
*CEId* *Rank*
- cert-15.pd.infn.it:8443/cream-lsf-cert 0
- gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-cert 0
- prod-ce-01.pd.infn.it:8443/cream-lsf-cert 0
- prod-ce-02.pd.infn.it:2119/jobmanager-lcglsf-cert 0
- t2-ce-01.to.infn.it:8443/cream-pbs-cert 0
- t2-ce-01.to.infn.it:8443/cream-pbs-short 0
- t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-cert 0
- t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-short 0
- test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cert 0
- test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-parallel 0
- devce.cnaf.infn.it:8443/cream-pbs-cert -700634
==========================================================================
- Finally you can check if these CEs are really closed to the SEs where you replicate your file(s)
- First list the SEs:
[ale@cream-15 DataReq]$ lcg-lr --vo dteam lfn:/grid/infngrid/cesini/PI_1M.txt
srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/infngrid/generated/2009-11-24/file0b6a613c-e1f1-4e53-a328-2152e44a8576
srm://gridit-se-01.cnaf.infn.it/dpm/cnaf.infn.it/home/infngrid/generated/2009-11-24/fileb9f3b972-806c-48a6-92fc-a9677901403a
srm://prod-se-01.pd.infn.it/infngrid/generated/2009-11-24/file10b5fe11-ad95-41e6-a3a3-14bc5cb9694b
srm://t2-se-00.to.infn.it/dpm/to.infn.it/home/infngrid/generated/2009-11-24/file1bb28727-1356-4fb1-bb73-5362f93e766f
- then for each SEs in the list use this command to find the "Close" CEs:
ldapsearch -x -h <BDII set in the WMS conf file> -p 2170 -b mds-vo-name=local,o=grid "(&(objectclass=GlueCESEBindGroup)(GlueCESEBindGroupSEUniqueID=<SE HOST>))" | grep ^GlueCESEBindGroupCEUniqueID
GlueCESEBindGroupCEUniqueID: devce.cnaf.infn.it:8443/cream-pbs-cert
GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
GlueCESEBindGroupCEUniqueID: cert-08.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: cert-05.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-par
GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
GlueCESEBindGroupCEUniqueID: cert-07.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: cert-09.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
GlueCESEBindGroupCEUniqueID: cert-06.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: cert-04.cnaf.infn.it:8443/cream-lsf-pps
GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cer
GlueCESEBindGroupCEUniqueID: cert-13.cnaf.infn.it:8443/cream-lsf-pps
- You can find more CEs because you should also filter using the "vo" attribute.
Others
BrokerInfo
- Use a jdl like this one to verify its creation:
###########################################
# JDL to test Brokerinfo Creation #
###########################################
Executable = "/bin/ls";
Arguments = "-la";
StdOutput = "std.out";
StdError = "std.err";
FuzzyRank = true;
InputSandbox = {"calc-pi.sh", "fileA", "fileB"};
OutputSandbox = {"std.out", "std.err",".BrokerInfo"};
Requirements = (other.GlueCEInfoHostName != "spacin-ce1.dma.unina.it") && !regexp("8443/cream", other.GlueCEUniqueID) ;
DataRequirements = {
[
DataCatalogType = "DLI";
DataCatalog = "http://lfcserver.cnaf.infn.it:8085";
InputData = {"lfn:/grid/infngrid/cesini/PI_1M.txt","lfn:/grid/infngrid/cesini/e-2M.txt"};
]
};
DataAccessProtocol = "gsiftp";
- After the retrieving of the output files you should see these files:
[ale@cream-15 DataReq]$ ls -la /tmp/ale_5XbbEhIAtjO5uC5wRyCmyA
total 1264
drwxr-xr-x 2 ale ale 4096 Mar 4 10:31 .
drwxrwxrwt 17 root root 1269760 Mar 4 10:31 ..
-rw-rw-r-- 1 ale ale 7804 Mar 4 10:31 .BrokerInfo
-rw-rw-r-- 1 ale ale 0 Mar 4 10:31 std.err
-rw-rw-r-- 1 ale ale 624 Mar 4 10:31 std.out
And also:
[ale@cream-15 DataReq]$ cat /tmp/ale_5XbbEhIAtjO5uC5wRyCmyA/std.out
total 32
drwxr-xr-x 2 dteam038 dteam 4096 Mar 4 10:27 .
drwx------ 5 dteam038 dteam 4096 Mar 4 10:27 ..
-rw-r--r-- 1 dteam038 dteam 7804 Mar 4 10:27 .BrokerInfo
-rw-r--r-- 1 dteam038 dteam 216 Mar 4 10:27 calc-pi.sh
-rw-r--r-- 1 dteam038 dteam 17 Mar 4 10:27 fileA
-rw-r--r-- 1 dteam038 dteam 17 Mar 4 10:27 fileB
-rw-r--r-- 1 dteam038 dteam 166 Mar 4 10:27 https_3a_2f_2fdevel17.cnaf.infn.it_3a9000_2f5XbbEhIAtjO5uC5wRyCmyA.output
-rw-r--r-- 1 dteam038 dteam 0 Mar 4 10:27 std.err
-rw-r--r-- 1 dteam038 dteam 0 Mar 4 10:27 std.out
-rw------- 1 dteam038 dteam 0 Mar 4 10:27 tmp.LesGC30313
- Check the goodness of the .BrokerInfo file generated:
- glite-brokerinfo getCE -f < .Brokerinfo file path >
- It should return the matched CE
- glite-brokerinfo getDataAccessProtocol -f < .Brokerinfo file path >
- It should return the DataAccessProtocol parameter as specified in the jdl
- glite-brokerinfo getInputData -f < .Brokerinfo file path >
- It should return the InputData files specified in the jdl
- glite-brokerinfo getSEs -f < .Brokerinfo file path >
- It should return all the SEs (seen by the BDII of the WMS) that store all the files specified in the InputData attribute (You can chek it using the command
lcg-lr --vo <VO name> <logical file name>
)
- glite-brokerinfo getCloseSEs -f < .Brokerinfo file path >
- It should return only the CloseSe to the matched CE (to see them use the command:
ldapsearch -x -h <specify the BDII host name> -p 2170 -b mds-vo-name=local,o=grid "(&(objectclass=GlueCESEBindGroup)(GlueCESEBindGroupCEUniqueID=<specify the CE name>))" | grep GlueCESEBindGroupSEUniqueID
)
- glite-brokerinfo getLFN2SFN <LFN of the files in InputData > -f < .Brokerinfo file path >
- It should return the output of the command
lcg-lr --vo <VO name> <logical file name>
- glite-brokerinfo getVirtualOrganization -f < .Brokerinfo file path >
Prologue and Epilogue jobs
- Submit a jdl like this one:
###########################################
# JDL with Prologue and Epilogue #
###########################################
Executable = "exe.sh";
Arguments = "1000";
StdOutput = "std.out";
StdError = "std.err";
Prologue = "prologue.sh";
Epilogue = "epilogue.sh";
FuzzyRank = true;
InputSandbox = {"exe/exe.sh", "data/pippo", "exe/epilogue.sh","exe/prologue.sh"};
OutputSandbox = {"std.out", "std.err", "prologue.out", "epilogue.out"};
#Requirements = regexp("8443/cream", other.GlueCEUniqueID);
#Requirements = regexp("2119/jobmanager", other.GlueCEUniqueID);
RetryCount = 1;
ShallowRetryCount = 2;
- Where:
[ale@cream-15 UI]$ cat exe/exe.sh
#!/bin/sh
date
echo "Hello world!"
echo "My prologue said:"
cat prologue.out
echo "My argument is $1"
[ale@cream-15 UI]$ cat exe/prologue.sh
#!/bin/sh
date > prologue.out
echo "I'm the prologue" >> prologue.out
echo "##########################" >> prologue.out
echo "This is the output of an ls command" >> prologue.out
ls >> prologue.out
echo "##########################" >> prologue.out
[ale@cream-15 UI]$ cat exe/epilogue.sh
#!/bin/sh
echo " This is the output of the job: `cat std.out`" > epilogue.out
echo "#########################" >> epilogue.out
echo "We finish at `date`" >> epilogue.out
echo "#########################" >> epilogue.out
echo "All the jokes are done!" >> epilogue.out
--
ElisabettaMolinari - 2010-02-24