Difference: WMSTask27731 (15 vs. 16)

Revision 162012-07-02 - FabioCapannini

Line: 1 to 1
 

WMS test report

Line: 1113 to 1113
 ===> Test log file is WMSService-TS_20120702105949.log
Added:
>
>

Testing forwarding parameters for parallel jobs

In order to test the passing of mpi parameters (wholenodes,smpgranularity,cpunumber,hostnumber) to the ce, an mpi jdl is submitted with varying values for the parameters, specifying a cream CE as destination. Then the cream jobid is retrieved via a glite-wms-job-logging-info -v 2 ("Dest jobid" field of a Transfer event).Finally the cream jdl is found via glite-ce-job-status -L 2 <cream-jobid>.

The following jdls were submitted:

1)

Executable = "openmpi-wrapper.sh";
Arguments="hello";
StdOutput  = "hello.out";
StdError   = "hello.err";
OutputSandbox = {"hello.out","hello.err"};
InputSandbox = {"/home/capannini/certification/mpi/openmpi-wrapper.sh","/home/capannini/certification/mpi/hello.c","/home/capannini/certification/mpi/Makefile"};
Requirements = Member ("OPENMPI",other.GlueHostApplicationSoftwareRunTimeEnvironment) && RegExp(".*cream.*", other.GlueCEUniqueID);
WholeNodes = true;
SMPGranularity = 2;
CpuNumber=2;
Hostnumber = 1;
Resulting cream JDL:
JDL            = [[ CEInfoHostName = "cert-ce-01.cnaf.infn.it"; OutputSandboxPath = "/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/output"; StdOutput = "hello.out"; BatchSystem = "pbs"; GlobusResourceContactString = "cert-ce-01.cnaf.infn.it:8443/cream-pbs"; QueueName = "cert"; ShallowRetryCount = 10; SignificantAttributes = { "Requirements","Rank","FuzzyRank" }; RetryCount = 3; Executable = "openmpi-wrapper.sh"; Type = "job"; LB_sequence_code = "UI=000000:NS=0000000004:WM=000005:BH=0000000000:JSS=000002:LM=000002:LRMS=000000:APP=000000:LBS=000000"; Arguments = "hello"; AllowZippedISB = true; maxOutputSandboxSize = -1.000000000000000E+00; VirtualOrganisation = "dteam"; JobType = "normal"; DefaultRank =  -other.GlueCEStateEstimatedResponseTime; ce_id = "cert-ce-01.cnaf.infn.it:8443/cream-pbs-cert"; ZippedISB = { "ISBfiles_sKVKJJgGMUeK6H6RwhcWBQ_0.tar.gz" }; OutputSandboxDestURI = { "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/output/hello.out","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/output/hello.err" }; OutputSandbox = { "hello.out","hello.err" }; CeRequirements = "( true && ( true && ( true && ( true && ( true && ( true && Member(\"OPENMPI\",other.GlueHostApplicationSoftwareRunTimeEnvironment) ) ) ) ) ) ) && ( true && ( ( ShortDeadlineJob is true ) ? false : true ) && ( EnableWmsFeedback is true ? true : true ) )"; InputSandbox = { "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/input/openmpi-wrapper.sh","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/input/hello.c","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/input/Makefile","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/input/.BrokerInfo" }; edg_jobid = "https://emi-demo12.cnaf.infn.it:9000/xtXTZHqgFsO-U5-LktmARA"; VOMS_FQAN = "/dteam/Role=NULL/Capability=NULL"; CertificateSubject = "/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Fabio Capannini"; StdError = "hello.err"; SMPGranularity = 2; InputSandboxPath = "/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/input"; CeApplicationDir = "/opt/exp_soft/dteam"; ReallyRunningToken = "gsiftp://emi-demo12.cnaf.infn.it/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/token.txt_0"; rank =  -other.GlueCEStateEstimatedResponseTime; WholeNodes = true; CpuNumber = 2; WMSHostname = "emi-demo12.cnaf.infn.it"; lrms_type = "pbs"; MyProxyServer = "myproxy.cnaf.infn.it"; X509UserProxy = "/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA/user.proxy"; InputSandboxDestFileName = { "openmpi-wrapper.sh","hello.c","Makefile" }; Hostnumber = 1; requirements = ( ( ( ( ( ( Member("OPENMPI",other.GlueHostApplicationSoftwareRunTimeEnvironment) && RegExp(".*cream.*",other.GlueCEUniqueID) ) && ( true ) ) && other.GlueCEInfoTotalCPUs >= 2 ) && other.GlueHostArchitectureSMPSize >= SMPGranularity ) && other.GlueCEInfoTotalCPUs >= 2 ) && other.GlueHostArchitectureSMPSize >= SMPGranularity ) && ( ( ( ShortDeadlineJob is true ) ? RegExp(".*sdj$",other.GlueCEUniqueID) :  !RegExp(".*sdj$",other.GlueCEUniqueID) ) && ( other.GlueCEPolicyMaxTotalJobs == 0 || other.GlueCEStateTotalJobs < other.GlueCEPolicyMaxTotalJobs ) && ( EnableWmsFeedback is true ? RegExp("cream",other.GlueCEImplementationName,"i") : true ) ); WMPInputSandboxBaseURI = "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/xt/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fxtXTZHqgFsO-U5-LktmARA" ]]
2)
Executable = "openmpi-wrapper.sh";
Arguments="hello";
StdOutput  = "hello.out";
StdError   = "hello.err";
OutputSandbox = {"hello.out","hello.err"};
InputSandbox = {"/home/capannini/certification/mpi/openmpi-wrapper.sh","/home/capannini/certification/mpi/hello.c","/home/capannini/certification/mpi/Makefile"};
Requirements = Member ("OPENMPI",other.GlueHostApplicationSoftwareRunTimeEnvironment) && RegExp(".*cream.*", other.GlueCEUniqueID);
WholeNodes = true;
SMPGranularity = 2;
Resulting ce JDL:
JDL            = [[ CEInfoHostName = "cex.grid.unipg.it"; OutputSandboxPath = "/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/output"; StdOutput = "hello.out"; BatchSystem = "pbs"; GlobusResourceContactString = "cex.grid.unipg.it:8443/cream-pbs"; QueueName = "cert"; ShallowRetryCount = 10; SignificantAttributes = { "Requirements","Rank","FuzzyRank" }; RetryCount = 3; Executable = "openmpi-wrapper.sh"; Type = "job"; LB_sequence_code = "UI=000000:NS=0000000004:WM=000017:BH=0000000000:JSS=000006:LM=000022:LRMS=000000:APP=000000:LBS=000000"; Arguments = "hello"; AllowZippedISB = true; maxOutputSandboxSize = -1.000000000000000E+00; VirtualOrganisation = "dteam"; JobType = "normal"; DefaultRank =  -other.GlueCEStateEstimatedResponseTime; ce_id = "cex.grid.unipg.it:8443/cream-pbs-cert"; ZippedISB = { "ISBfiles_4dXYGVKTKVRagB4f-hex8Q_0.tar.gz" }; OutputSandboxDestURI = { "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/output/hello.out","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/output/hello.err" }; OutputSandbox = { "hello.out","hello.err" }; CeRequirements = "( true && ( true && ( true && ( true && Member(\"OPENMPI\",other.GlueHostApplicationSoftwareRunTimeEnvironment) ) ) ) ) && ( true && ( ( ShortDeadlineJob is true ) ? false : true ) && ( EnableWmsFeedback is true ? true : true ) )"; InputSandbox = { "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/input/openmpi-wrapper.sh","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/input/hello.c","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/input/Makefile","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/input/.BrokerInfo" }; edg_jobid = "https://emi-demo12.cnaf.infn.it:9000/lYvOaWtj5auRglvI-uSZpw"; VOMS_FQAN = "/dteam/Role=NULL/Capability=NULL"; CertificateSubject = "/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Fabio Capannini"; StdError = "hello.err"; SMPGranularity = 2; InputSandboxPath = "/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/input"; CeApplicationDir = "/opt/exp_soft/dteam"; ReallyRunningToken = "gsiftp://emi-demo12.cnaf.infn.it/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/token.txt_0"; rank =  -other.GlueCEStateEstimatedResponseTime; WholeNodes = true; edg_previous_matches = { "grisuce.scope.unina.it","emi-ce01.scope.unina.it" }; ReplansCount = 0; WMSHostname = "emi-demo12.cnaf.infn.it"; lrms_type = "torque"; MyProxyServer = "myproxy.cnaf.infn.it"; edg_previous_matches_ex = { [ timestamp = 1341236596; ce_id = "grisuce.scope.unina.it" ],[ timestamp = 1341236597; ce_id = "emi-ce01.scope.unina.it" ] }; X509UserProxy = "/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw/user.proxy"; InputSandboxDestFileName = { "openmpi-wrapper.sh","hello.c","Makefile" }; requirements = ( ( ( ( Member("OPENMPI",other.GlueHostApplicationSoftwareRunTimeEnvironment) && RegExp(".*cream.*",other.GlueCEUniqueID) ) && ( true ) ) && other.GlueHostArchitectureSMPSize >= SMPGranularity ) && other.GlueHostArchitectureSMPSize >= SMPGranularity ) && ( ( ( ShortDeadlineJob is true ) ? RegExp(".*sdj$",other.GlueCEUniqueID) :  !RegExp(".*sdj$",other.GlueCEUniqueID) ) && ( other.GlueCEPolicyMaxTotalJobs == 0 || other.GlueCEStateTotalJobs < other.GlueCEPolicyMaxTotalJobs ) && ( EnableWmsFeedback is true ? RegExp("cream",other.GlueCEImplementationName,"i") : true ) ); WMPInputSandboxBaseURI = "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/lY/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2flYvOaWtj5auRglvI-uSZpw" ]]
3)
Executable = "openmpi-wrapper.sh";
Arguments="hello";
StdOutput  = "hello.out";
StdError   = "hello.err";
OutputSandbox = {"hello.out","hello.err"};
InputSandbox = {"/home/capannini/certification/mpi/openmpi-wrapper.sh","/home/capannini/certification/mpi/hello.c","/home/capannini/certification/mpi/Makefile"};
Requirements = Member ("OPENMPI",other.GlueHostApplicationSoftwareRunTimeEnvironment) && RegExp(".*cream.*", other.GlueCEUniqueID);
WholeNodes = false;
SMPGranularity = 3;
CpuNumber=3;
Hostnumber = 1;

Submission result:

[capannini@cert-17 mpi]$ glite-wms-job-submit -a -c ~/wms_emi-demo12.conf mpi.jdl 

Connecting to the service https://emi-demo12.cnaf.infn.it:7443/glite_wms_wmproxy_server


Error - 
SMPGranularity and HostNumber are mutually exclusive when WholeNodes allocation is not requested: wrong combination of values
4)
Executable = "openmpi-wrapper.sh";
Arguments="hello";
StdOutput  = "hello.out";
StdError   = "hello.err";
OutputSandbox = {"hello.out","hello.err"};
InputSandbox = {"/home/capannini/certification/mpi/openmpi-wrapper.sh","/home/capannini/certification/mpi/hello.c","/home/capannini/certification/mpi/Makefile"};
Requirements = Member ("OPENMPI",other.GlueHostApplicationSoftwareRunTimeEnvironment) && RegExp(".*cream.*", other.GlueCEUniqueID);
WholeNodes = false;
SMPGranularity = 3;
CpuNumber=3;
Resulting ce JDL:
JDL            = [[ CEInfoHostName = "ng-ce.grid.unipg.it"; OutputSandboxPath = "/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/output"; StdOutput = "hello.out"; BatchSystem = "pbs"; GlobusResourceContactString = "ng-ce.grid.unipg.it:8443/cream-pbs"; QueueName = "cert"; ShallowRetryCount = 10; SignificantAttributes = { "Requirements","Rank","FuzzyRank" }; RetryCount = 3; Executable = "openmpi-wrapper.sh"; Type = "job"; LB_sequence_code = "UI=000000:NS=0000000004:WM=000011:BH=0000000000:JSS=000004:LM=000012:LRMS=000000:APP=000000:LBS=000000"; Arguments = "hello"; AllowZippedISB = true; maxOutputSandboxSize = -1.000000000000000E+00; VirtualOrganisation = "dteam"; JobType = "normal"; DefaultRank =  -other.GlueCEStateEstimatedResponseTime; ce_id = "ng-ce.grid.unipg.it:8443/cream-pbs-cert"; ZippedISB = { "ISBfiles_IC2yIMsun757SF2rrFtzqA_0.tar.gz" }; OutputSandboxDestURI = { "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/output/hello.out","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/output/hello.err" }; OutputSandbox = { "hello.out","hello.err" }; CeRequirements = "( true && ( true && ( true && ( true && ( true && ( true && Member(\"OPENMPI\",other.GlueHostApplicationSoftwareRunTimeEnvironment) ) ) ) ) ) ) && ( true && ( ( ShortDeadlineJob is true ) ? false : true ) && ( EnableWmsFeedback is true ? true : true ) )"; InputSandbox = { "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/input/openmpi-wrapper.sh","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/input/hello.c","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/input/Makefile","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/input/.BrokerInfo" }; edg_jobid = "https://emi-demo12.cnaf.infn.it:9000/nM8Uj4qxVn2csciKW9eh2g"; VOMS_FQAN = "/dteam/Role=NULL/Capability=NULL"; CertificateSubject = "/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Fabio Capannini"; StdError = "hello.err"; SMPGranularity = 3; InputSandboxPath = "/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/input"; CeApplicationDir = "/opt/exp_soft/dteam"; ReallyRunningToken = "gsiftp://emi-demo12.cnaf.infn.it/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/token.txt_0"; rank =  -other.GlueCEStateEstimatedResponseTime; WholeNodes = false; CpuNumber = 3; edg_previous_matches = { "ce.scope.unina.it" }; ReplansCount = 0; WMSHostname = "emi-demo12.cnaf.infn.it"; lrms_type = "torque"; MyProxyServer = "myproxy.cnaf.infn.it"; edg_previous_matches_ex = { [ timestamp = 1341237438; ce_id = "ce.scope.unina.it" ] }; X509UserProxy = "/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g/user.proxy"; InputSandboxDestFileName = { "openmpi-wrapper.sh","hello.c","Makefile" }; requirements = ( ( ( ( ( ( Member("OPENMPI",other.GlueHostApplicationSoftwareRunTimeEnvironment) && RegExp(".*cream.*",other.GlueCEUniqueID) ) && ( true ) ) && other.GlueCEInfoTotalCPUs >= 3 ) && other.GlueHostArchitectureSMPSize >= SMPGranularity ) && other.GlueCEInfoTotalCPUs >= 3 ) && other.GlueHostArchitectureSMPSize >= SMPGranularity ) && ( ( ( ShortDeadlineJob is true ) ? RegExp(".*sdj$",other.GlueCEUniqueID) :  !RegExp(".*sdj$",other.GlueCEUniqueID) ) && ( other.GlueCEPolicyMaxTotalJobs == 0 || other.GlueCEStateTotalJobs < other.GlueCEPolicyMaxTotalJobs ) && ( EnableWmsFeedback is true ? RegExp("cream",other.GlueCEImplementationName,"i") : true ) ); WMPInputSandboxBaseURI = "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/nM/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fnM8Uj4qxVn2csciKW9eh2g" ]]
5)
Executable = "openmpi-wrapper.sh";
Arguments="hello";
StdOutput  = "hello.out";
StdError   = "hello.err";
OutputSandbox = {"hello.out","hello.err"};
InputSandbox = {"/home/capannini/certification/mpi/openmpi-wrapper.sh","/home/capannini/certification/mpi/hello.c","/home/capannini/certification/mpi/Makefile"};
Requirements = Member ("OPENMPI",other.GlueHostApplicationSoftwareRunTimeEnvironment) && RegExp(".*cream.*", other.GlueCEUniqueID);
WholeNodes = true;
SMPGranularity = 8;
CpuNumber=8;
Hostnumber = 2;
Resulting ce JDL:
JDL            = [[ CEInfoHostName = "gridsrv2-4.dir.garr.it"; OutputSandboxPath = "/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/output"; StdOutput = "hello.out"; BatchSystem = "pbs"; GlobusResourceContactString = "gridsrv2-4.dir.garr.it:8443/cream-pbs"; QueueName = "cert"; ShallowRetryCount = 10; SignificantAttributes = { "Requirements","Rank","FuzzyRank" }; RetryCount = 3; Executable = "openmpi-wrapper.sh"; Type = "job"; LB_sequence_code = "UI=000000:NS=0000000004:WM=000017:BH=0000000000:JSS=000006:LM=000022:LRMS=000000:APP=000000:LBS=000000"; Arguments = "hello"; AllowZippedISB = true; maxOutputSandboxSize = -1.000000000000000E+00; VirtualOrganisation = "dteam"; JobType = "normal"; DefaultRank =  -other.GlueCEStateEstimatedResponseTime; ce_id = "gridsrv2-4.dir.garr.it:8443/cream-pbs-cert"; ZippedISB = { "ISBfiles_vh0tUb1uOBxl3Nj8-svtgw_0.tar.gz" }; OutputSandboxDestURI = { "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/output/hello.out","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/output/hello.err" }; OutputSandbox = { "hello.out","hello.err" }; CeRequirements = "( true && ( true && ( true && ( true && ( true && ( true && Member(\"OPENMPI\",other.GlueHostApplicationSoftwareRunTimeEnvironment) ) ) ) ) ) ) && ( true && ( ( ShortDeadlineJob is true ) ? false : true ) && ( EnableWmsFeedback is true ? true : true ) )"; InputSandbox = { "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/input/openmpi-wrapper.sh","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/input/hello.c","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/input/Makefile","gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/input/.BrokerInfo" }; edg_jobid = "https://emi-demo12.cnaf.infn.it:9000/XH4DRPKTlCxGt5uL3NyVjA"; VOMS_FQAN = "/dteam/Role=NULL/Capability=NULL"; CertificateSubject = "/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Fabio Capannini"; StdError = "hello.err"; SMPGranularity = 8; InputSandboxPath = "/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/input"; CeApplicationDir = "/opt/exp_soft/dteam"; ReallyRunningToken = "gsiftp://emi-demo12.cnaf.infn.it/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/token.txt_0"; rank =  -other.GlueCEStateEstimatedResponseTime; WholeNodes = true; CpuNumber = 8; edg_previous_matches = { "ce.scope.unina.it","grisuce.scope.unina.it" }; ReplansCount = 0; WMSHostname = "emi-demo12.cnaf.infn.it"; lrms_type = "pbs"; MyProxyServer = "myproxy.cnaf.infn.it"; edg_previous_matches_ex = { [ timestamp = 1341238279; ce_id = "ce.scope.unina.it" ],[ timestamp = 1341238280; ce_id = "grisuce.scope.unina.it" ] }; X509UserProxy = "/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA/user.proxy"; InputSandboxDestFileName = { "openmpi-wrapper.sh","hello.c","Makefile" }; Hostnumber = 2; requirements = ( ( ( ( ( ( Member("OPENMPI",other.GlueHostApplicationSoftwareRunTimeEnvironment) && RegExp(".*cream.*",other.GlueCEUniqueID) ) && ( true ) ) && other.GlueCEInfoTotalCPUs >= 8 ) && other.GlueHostArchitectureSMPSize >= SMPGranularity ) && other.GlueCEInfoTotalCPUs >= 8 ) && other.GlueHostArchitectureSMPSize >= SMPGranularity ) && ( ( ( ShortDeadlineJob is true ) ? RegExp(".*sdj$",other.GlueCEUniqueID) :  !RegExp(".*sdj$",other.GlueCEUniqueID) ) && ( other.GlueCEPolicyMaxTotalJobs == 0 || other.GlueCEStateTotalJobs < other.GlueCEPolicyMaxTotalJobs ) && ( EnableWmsFeedback is true ? RegExp("cream",other.GlueCEImplementationName,"i") : true ) ); WMPInputSandboxBaseURI = "gsiftp://emi-demo12.cnaf.infn.it:2811/var/SandboxDir/XH/https_3a_2f_2femi-demo12.cnaf.infn.it_3a9000_2fXH4DRPKTlCxGt5uL3NyVjA" ]]
===> Test PASSED

Perusal

[capannini@cert-17 WMS-service]$ python WMS-job-cycle.py -c wms-command.conf -V dteam -d 3 -l -t 6 -i
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the WMS Service                       
===> + Description: Test a complete job cycle: from submission to get output 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

Enter the user proxy password:

 Set 6: Submit a perusal job 


 BEWARE default min perusal interval is 1000 secs, so this phase could take many minutes 

===> 
===> Test: WMS-job-cycle.py
===> WMS: emi-demo12.cnaf.infn.it
===> Started: 16:19:19
===> Ended  : 16:48:50
===> 
===>     === test PASSED === 
===> 
===> Test log file is WMSService-TS_20120702161919.log
[capannini@cert-17 WMS-service]$ python WMS-job-cycle.py -c wms-command.conf -V testers.eu-emi.eu  -d 3 -l -t 6 -i
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the WMS Service                       
===> + Description: Test a complete job cycle: from submission to get output 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

Enter the user proxy password:

 Set 6: Submit a perusal job 


 BEWARE default min perusal interval is 1000 secs, so this phase could take many minutes 

===> 
===> Test: WMS-job-cycle.py
===> WMS: emi-demo12.cnaf.infn.it
===> Started: 16:19:50
===> Ended  : 17:14:07
===> 
===>     === test PASSED === 
===> 
===> Test log file is WMSService-TS_20120702161950.log

Delegation

[capannini@cert-17 WMS-service]$ python WMS-delegation-test.py -c wms-command.conf -V dteam -d 3 -l -i
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the WMS Service                       
===> + Description: Test delegation operation 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

Enter the user proxy password:

 Test 1: Check --autm-delegation option 


 Test 2: Check --delegationid option 


 Test 3: Try to delegate with a short proxy and check the validity 


 Test 4: Works with expired proxy 

===> 
===> Test: WMS-delegation-test.py
===> WMS: emi-demo12.cnaf.infn.it
===> Started: 16:25:04
===> Ended  : 16:26:19
===> 
===>     === test PASSED === 
===> 
===> Test log file is WMSService-TS_20120702162504.log
[capannini@cert-17 WMS-service]$ python WMS-delegation-test.py -c wms-command.conf -V testers.eu-emi.eu -d 3 -l -i
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the WMS Service                       
===> + Description: Test delegation operation 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

Enter the user proxy password:

 Test 1: Check --autm-delegation option 


 Test 2: Check --delegationid option 


 Test 3: Try to delegate with a short proxy and check the validity 


 Test 4: Works with expired proxy 

===> 
===> Test: WMS-delegation-test.py
===> WMS: emi-demo12.cnaf.infn.it
===> Started: 16:25:25
===> Ended  : 16:26:37
===> 
===>     === test PASSED === 
===> 
===> Test log file is WMSService-TS_20120702162525.log
[capannini@cert-17 WMS-service]$ python WMS-delegation-test.py -c wms-command.conf -d 3 -l
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the WMS Service                       
===> + Description: Test delegation operation 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

 Test 1: Check --autm-delegation option 


 Test 2: Check --delegationid option 


 There are other two tests which require the user proxy password. Use -i option to enable them 

===> 
===> Test: WMS-delegation-test.py
===> WMS: emi-demo12.cnaf.infn.it
===> Started: 16:26:03
===> Ended  : 16:26:08
===> 
===>     === test PASSED === 
===> 
===> Test log file is WMSService-TS_20120702162603.log

Job list-match

without data

[capannini@cert-17 WMS-service]$ python WMS-listmatch-test.py -c wms-command.conf -V dteam -d 3 -l -i
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the WMS Service                       
===> + Description: WMS Job ListMatch Testing 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

Enter the user proxy password:

 Test 1: check if glite-wms-job-list-match command exists 


 Test 2: test a simple matching 


 Test 3: check --delegationid option 


 Test 4: exclude a CE from the match 


 Test 5: try a failure matching (Requirements == false) 


 Test 6: try a matching with EnableWMSFeedback attribute true 

===> 
===> Test: WMS-listmatch-test.py
===> WMS: emi-demo12.cnaf.infn.it
===> Started: 16:37:45
===> Ended  : 16:37:54
===> 
===>     === test PASSED === 
===> 
===> Test log file is WMSService-TS_20120702163745.log
[capannini@cert-17 WMS-service]$ python WMS-listmatch-test.py -c wms-command.conf -V testers.eu-emi.eu -d 3 -l -i
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the WMS Service                       
===> + Description: WMS Job ListMatch Testing 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

Enter the user proxy password:

 Test 1: check if glite-wms-job-list-match command exists 


 Test 2: test a simple matching 


 Test 3: check --delegationid option 


 Test 4: exclude a CE from the match 


 Test 5: try a failure matching (Requirements == false) 


 Test 6: try a matching with EnableWMSFeedback attribute true 

===> 
===> Test: WMS-listmatch-test.py
===> WMS: emi-demo12.cnaf.infn.it
===> Started: 16:38:08
===> Ended  : 16:38:20
===> 
===>     === test PASSED === 
===> 
===> Test log file is WMSService-TS_20120702163808.log
[capannini@cert-17 WMS-service]$ 
[capannini@cert-17 WMS-service]$ python WMS-listmatch-test.py -c wms-command.conf -d 3 -l
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the WMS Service                       
===> + Description: WMS Job ListMatch Testing 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

 Test 1: check if glite-wms-job-list-match command exists 


 Test 2: test a simple matching 


 Test 3: check --delegationid option 


 Test 4: exclude a CE from the match 


 Test 5: try a failure matching (Requirements == false) 


 Test 6: try a matching with EnableWMSFeedback attribute true 

===> 
===> Test: WMS-listmatch-test.py
===> WMS: emi-demo12.cnaf.infn.it
===> Started: 16:38:29
===> Ended  : 16:38:36
===> 
===>     === test PASSED === 
===> 
===> Test log file is WMSService-TS_20120702163829.log

with data

[capannini@cert-17 WMS-service]$ export LFC_HOST=prod-lfc-shared-central.cern.ch
[capannini@cert-17 WMS-service]$ lcg-lr --vo dteam lfn:/grid/dteam/fcapanni/PI_1M.txt
srm://gridsrm.pi.infn.it/dteam/fcapanni/PI_1M.txt
srm://srm-public.cern.ch/castor/cern.ch/grid/dteam/fcapanni/PI_1M.txt
The two SEs where the file can be found are gridsrm.pi.infn.it and srm-public.cern.ch.

[capannini@cert-17 WMS-service]$ cat JDLs/data-req.jdl 
###########################################
#      JDL with Data Requirements         #
###########################################

Executable = "/bin/ls";
Arguments = "-la";
StdOutput = "std.out";
StdError = "std.err";
Prologue = "prologue.sh";

FuzzyRank = true;
Environment={"LFC_HOST=prod-lfc-shared-central.cern.ch", "LFC_HOME=prod-lfc-shared-central.cern.ch:/grid/dteam/aleph"};
InputSandbox = {"JDLs/prologue.sh"};
OutputSandbox = {"std.out", "std.err","prologue.out", "PI_1M.txt"};

DataRequirements = {
[
DataCatalogType = "DLI";
DataCatalog ="http://prod-lfc-shared-central.cern.ch:8085/"; 
InputData = { "lfn:/grid/dteam/fcapanni/PI_1M.txt" };
]
};

DataAccessProtocol = "gsiftp";
AllowZippedISB=true;
RetryCount = 1;
ShallowRetryCount = 2;

[capannini@cert-17 WMS-service]$ glite-wms-job-list-match -a -c ~/wms_emi-demo12.conf JDLs/data-req.jdl

Connecting to the service https://emi-demo12.cnaf.infn.it:7443/glite_wms_wmproxy_server

==========================================================================

           COMPUTING ELEMENT IDs LIST 
 The following CE(s) matching your job requirements have been found:

   *CEId*
 - ce204.cern.ch:8443/cream-lsf-grid_2nh_dteam
 - ce204.cern.ch:8443/cream-lsf-grid_dteam
 - ce205.cern.ch:8443/cream-lsf-grid_2nh_dteam
 - ce205.cern.ch:8443/cream-lsf-grid_dteam
 - ce207.cern.ch:8443/cream-lsf-grid_2nh_dteam
 - ce207.cern.ch:8443/cream-lsf-grid_dteam
 - gridce4.pi.infn.it:8443/cream-lsf-certmpi
 - gridce0.pi.infn.it:8443/cream-lsf-cert
 - gridce1.pi.infn.it:2119/jobmanager-lcglsf-cert
 - gridce2.pi.infn.it:2119/jobmanager-lcglsf-cert
 - gridce3.pi.infn.it:8443/cream-lsf-certmpi

==========================================================================

In order to verify that these CEs are close to the above SEs:

[capannini@cert-17 WMS-service]$ lcg-infosites --vo dteam closeSE|grep -A 5 ce204
Name of the CE: ce204.cern.ch:8443/cream-lsf-grid_2nh_dteam
   srm-atlas.cern.ch
   srm-cms.cern.ch
   srm-lhcb.cern.ch
   srm-public.cern.ch

Name of the CE: ce204.cern.ch:8443/cream-lsf-grid_dteam
   srm-atlas.cern.ch
   srm-cms.cern.ch
   srm-lhcb.cern.ch
   srm-public.cern.ch

[capannini@cert-17 WMS-service]$ lcg-infosites --vo dteam closeSE|grep -A 5 ce205
Name of the CE: ce205.cern.ch:8443/cream-lsf-grid_2nh_dteam
   srm-atlas.cern.ch
   srm-cms.cern.ch
   srm-lhcb.cern.ch
   srm-public.cern.ch

Name of the CE: ce205.cern.ch:8443/cream-lsf-grid_dteam
   srm-atlas.cern.ch
   srm-cms.cern.ch
   srm-lhcb.cern.ch
   srm-public.cern.ch

[capannini@cert-17 WMS-service]$ lcg-infosites --vo dteam closeSE|grep -A 5 ce207
Name of the CE: ce207.cern.ch:8443/cream-lsf-grid_2nh_dteam
   srm-atlas.cern.ch
   srm-cms.cern.ch
   srm-lhcb.cern.ch
   srm-public.cern.ch

Name of the CE: ce207.cern.ch:8443/cream-lsf-grid_dteam
   srm-atlas.cern.ch
   srm-cms.cern.ch
   srm-lhcb.cern.ch
   srm-public.cern.ch

[capannini@cert-17 WMS-service]$ lcg-infosites --vo dteam closeSE|grep -A 5 gridce4.pi.infn.it
Name of the CE: gridce4.pi.infn.it:8443/cream-lsf-certmpi
   gridsrm.pi.infn.it

Name of the CE: gridgate.cp.dias.ie:2119/jobmanager-lcgpbs-test
   gridstore.cp.dias.ie

[capannini@cert-17 WMS-service]$ lcg-infosites --vo dteam closeSE|grep -A 5 gridce0.pi.infn.it
Name of the CE: gridce0.pi.infn.it:8443/cream-lsf-cert
   gridsrm.pi.infn.it

Name of the CE: gridce01.ifca.es:8443/cream-sge-sgm
   griddpm01.ifca.es
   storm.ifca.es
[capannini@cert-17 WMS-service]$ lcg-infosites --vo dteam closeSE|grep -A 5 gridce1.pi.infn.it
Name of the CE: gridce1.pi.infn.it:2119/jobmanager-lcglsf-cert
   gridsrm.pi.infn.it

Name of the CE: gridce2.pi.infn.it:2119/jobmanager-lcglsf-cert
   gridsrm.pi.infn.it

[capannini@cert-17 WMS-service]$ lcg-infosites --vo dteam closeSE|grep -A 5 gridce2.pi.infn.it
Name of the CE: gridce2.pi.infn.it:2119/jobmanager-lcglsf-cert
   gridsrm.pi.infn.it

Name of the CE: gridce3.pi.infn.it:8443/cream-lsf-certmpi
   gridsrm.pi.infn.it

[capannini@cert-17 WMS-service]$ lcg-infosites --vo dteam closeSE|grep -A 5 gridce3.pi.infn.it
Name of the CE: gridce3.pi.infn.it:8443/cream-lsf-certmpi
   gridsrm.pi.infn.it

Name of the CE: gridce4.pi.infn.it:8443/cream-lsf-certmpi
   gridsrm.pi.infn.it

===> Test PASSED

gangmatching

[capannini@cert-17 certification]$ cat gang.jdl 
[
executable="ls.sh";
myproxyserver="";
SErequirements = anyMatch(other.storage.CloseSEs, target.GlueSAStateAvailableSpace > 20);
//SErequirements = true;
CErequirements = other.GlueCEStateStatus == "Production" && other.GlueCEInfoTotalCPUs >= 2;
requirements = CErequirements && SErequirements;
inputsandbox={"ls.sh"};
stdoutput="out3.out";
stderror="err2.err";
outputsandbox={"out3.out", "err2.err", "ls.sh", "a"};
]

[capannini@cert-17 certification]$ glite-wms-job-list-match -a -c ~/wms_emi-demo12.conf gang.jdl

Connecting to the service https://emi-demo12.cnaf.infn.it:7443/glite_wms_wmproxy_server

==========================================================================

           COMPUTING ELEMENT IDs LIST 
 The following CE(s) matching your job requirements have been found:

   *CEId*
 - cccreamceli05.in2p3.fr:8443/cream-sge-long
 - cccreamceli05.in2p3.fr:8443/cream-sge-medium
 - cccreamceli05.in2p3.fr:8443/cream-sge-short
 - cccreamceli06.in2p3.fr:8443/cream-sge-long
 - cccreamceli06.in2p3.fr:8443/cream-sge-medium
 - cccreamceli06.in2p3.fr:8443/cream-sge-short
 - cccreamceli09.in2p3.fr:8443/cream-sge-long
 - cccreamceli09.in2p3.fr:8443/cream-sge-medium
 - cccreamceli09.in2p3.fr:8443/cream-sge-short
 - cmsce01.na.infn.it:2119/jobmanager-lcgpbs-cert
 - cmsrm-cream02.roma1.infn.it:8443/cream-lsf-cmsgcert
 - unict-dmi-ce-01.ct.pi2s2.it:2119/jobmanager-lcglsf-cert
 - unime-ce-01.me.pi2s2.it:2119/jobmanager-lcglsf-cert
 - virgo-ce.roma1.infn.it:2119/jobmanager-lcgpbs-cert
 - cmsrm-ce01.roma1.infn.it:2119/jobmanager-lcglsf-cmsgcert
 - cmsrm-ce02.roma1.infn.it:2119/jobmanager-lcglsf-cmsgcert
 - cmsrm-cream01.roma1.infn.it:8443/cream-lsf-cmsgcert
 - cccreamceli10.in2p3.fr:8443/cream-sge-long
 - cccreamceli10.in2p3.fr:8443/cream-sge-medium
 - cccreamceli10.in2p3.fr:8443/cream-sge-short
 - inaf-ce-01.ct.pi2s2.it:8443/cream-lsf-cert

==========================================================================

[capannini@cert-17 certification]$ lcg-infosites closeSE|grep -A 5 cccreamceli05
Name of the CE: cccreamceli05.in2p3.fr:8443/cream-sge-long
   ccsrm.in2p3.fr

Name of the CE: cccreamceli05.in2p3.fr:8443/cream-sge-medium
   ccsrm.in2p3.fr

Name of the CE: cccreamceli05.in2p3.fr:8443/cream-sge-short
   ccsrm.in2p3.fr

Name of the CE: cccreamceli06.in2p3.fr:8443/cream-sge-long
   ccsrm.in2p3.fr

[capannini@cert-17 certification]$ 
[capannini@cert-17 certification]$ lcg-infosites se |egrep 'Avail|ccsrm.in2p3.fr'
 Avail Space(kB)  Used Space(kB)  Type    SE
     12730125753      2302259783  SRM     ccsrm.in2p3.fr

[capannini@cert-17 certification]$ lcg-infosites closeSE|grep -A 5 cmsrm-ce01
Name of the CE: cmsrm-ce01.roma1.infn.it:2119/jobmanager-lcglsf-cmsgcert
   cmsrm-se01.roma1.infn.it

Name of the CE: cmsrm-ce02.roma1.infn.it:2119/jobmanager-lcglsf-cmsgcert
   cmsrm-se01.roma1.infn.it

[capannini@cert-17 certification]$ 
[capannini@cert-17 certification]$ lcg-infosites se |egrep 'Avail|cmsrm-se01.roma1.infn.it'
 Avail Space(kB)  Used Space(kB)  Type    SE
            1000          999999  SRM     cmsrm-se01.roma1.infn.it

[capannini@cert-17 certification]$ lcg-infosites se |egrep 'Avail|cmsrm-se01.roma1.infn.it'
 Avail Space(kB)  Used Space(kB)  Type    SE
            1000          999999  SRM     cmsrm-se01.roma1.infn.it
[capannini@cert-17 certification]$ 
[capannini@cert-17 certification]$ lcg-infosites closeSE|grep -A 5 unict-dmi-ce-01
Name of the CE: unict-dmi-ce-01.ct.pi2s2.it:2119/jobmanager-lcglsf-cert
   unict-dmi-se-01.ct.pi2s2.it

Name of the CE: unime-ce-01.me.pi2s2.it:2119/jobmanager-lcglsf-cert
   unime-se-01.me.pi2s2.it

[capannini@cert-17 certification]$ lcg-infosites se |egrep 'Avail|unict-dmi-se-01.ct.pi2s2.it'
 Avail Space(kB)  Used Space(kB)  Type    SE
     12512874596      6164874393  SRM     unict-dmi-se-01.ct.pi2s2.it
===> Test PASSED
 

Regression tests

Bugs attached to the task

Line: 1233 to 1809
 
META FILEATTACHMENT attachment="all.tgz" attr="" comment="" date="1340896901" name="all.tgz" path="all.tgz" size="11637" user="FabioCapannini" version="1"
META FILEATTACHMENT attachment="WMSService-TS_20120628172415.log" attr="" comment="" date="1340959307" name="WMSService-TS_20120628172415.log" path="WMSService-TS_20120628172415.log" size="53019" user="FabioCapannini" version="1"
META FILEATTACHMENT attachment="dag.tgz" attr="" comment="" date="1341225122" name="dag.tgz" path="dag.tgz" size="13734" user="FabioCapannini" version="1"
Added:
>
>
META FILEATTACHMENT attachment="deleg.tgz" attr="" comment="" date="1341239376" name="deleg.tgz" path="deleg.tgz" size="1972" user="FabioCapannini" version="1"
META FILEATTACHMENT attachment="listmatch.tgz" attr="" comment="" date="1341240179" name="listmatch.tgz" path="listmatch.tgz" size="1552" user="FabioCapannini" version="1"
META FILEATTACHMENT attachment="perusal.tgz" attr="" comment="" date="1341242877" name="perusal.tgz" path="perusal.tgz" size="3602" user="FabioCapannini" version="1"
 
This site is powered by the TWiki collaboration platformCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback