Tags:
,
view all tags
---+ WMS Test Plan %TOC% ---++ YAIM-WMS Configuration Testing ---++ WMS Job Submission/GetOutput Testing Registers the job into the wmproxy, delivers it to the WM that submits it to the grid. Test job submission with the following type of jobs/jdl: ---+++ Normal Job * to JC <verbatim>[ Type = "Job"; JobType = "normal"; InputSandbox = { "file://$HOME/test.sh"}; VirtualOrganisation = "dteam"; Executable="test.sh"; Arguments="Hello "; Requirements = ( !RegExp("/cream-",other.GlueCEUniqueID)); Rank = 0; fuzzyrank = true; StdOutput="message.txt"; StdError="err.log"; OutputSandbox={"message.txt","err.log",".BrokerInfo"}; RetryCount = 0; usertags = [ jdl = "normal job" ]; ShallowRetryCount = 3; ]</verbatim> * to ICE <verbatim>[ Type = "Job"; JobType = "normal"; InputSandbox = { "file:///home/emolinari/test.sh"}; VirtualOrganisation = "dteam"; Executable="test.sh"; Arguments="Hello "; Requirements = ( RegExp("/cream-",other.GlueCEUniqueID)); Rank = 0; fuzzyrank = true; StdOutput="message.txt"; StdError="err.log"; OutputSandbox={"message.txt","err.log",".BrokerInfo"}; usertags = [ jdl = "normal job to ICE" ]; RetryCount = 0; ShallowRetryCount = 3; ]</verbatim> ---+++ Perusal job Job perusal is the ability to view output from a job while it is running. In order to use this feature the JDL attributes PerusalFileEnable and PerusalTimeInterval must be declared when the job is submitted for example: <verbatim>PerusalFileEnable = true; PerusalTimeInterval = 120;</verbatim> This makes the WN upload, at regular time intervals (PerusalTimeInterval), a copy of the output files specified using the glite-wms-job-perusal --set command to the WMS machine. * Following an example on how to test a perusal job: * Given the jdl file: <verbatim>[ Executable = "perusal.sh" ; InputSandbox = {"file://$HOME/perusal.sh"}; Stdoutput = "perusal.out" ; StdError = "perusal.err" ; OutputSandbox ={ "perusal.out" ,"perusal.err"} ; usertags = [ jdl = "perusal" ]; PerusalFileEnable = true; ]</verbatim> where perusal.sh is <verbatim>#!/bin/bash echo "Hi all, this is a perusal test program..." for (( i=0 ; $i < 50 ; i = $(($i + 1)) )); do echo "Step number $i" date sleep 10 done echo "" echo "Bye"</verbatim> * Submit the job with <verbatim> glite-wms-job-submit --config glite_wms.conf -a perusal.jdl</verbatim> * Given the jobid of the previously submitted set the files that are to be perused with <verbatim>glite-wms-job-perusal --set -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg</verbatim> * When the job starts running on the WN get the files as here: <verbatim>glite-wms-job-perusal --get -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg</verbatim> ---+++ DAG job * DAG job jdls samples:<verbatim>[ Type = "dag"; VirtualOrganisation = "dteam"; Max_nodes_running = 10; InputSandbox = "test.sh"; FuzzyRank = true; Nodes = [ nodeA = [ file= "test_dag.jdl"; ]; nodeB = [ file= "test_dag.jdl"; ]; nodeC = [ file= "test_dag.jdl"; ]; nodeD = [ file= "test_dag.jdl"; ]; nodeE = [ file= "test_dag.jdl"; ]; nodeF= [ file= "test_dag.jdl"; ]; ]; Dependencies = { {{nodeA, nodeB}, nodeC},{nodeD,nodeE.nodeF} } ]</verbatim><verbatim>[ type = "dag"; DefaultNodeShallowRetryCount = 3; nodes = [ nodeA = [ node_type = "edg-jdl"; file ="myjob.jdl" ; ]; nodeB = [ node_type = "edg-jdl"; file ="myjob.jdl" ; ]; nodeC = [ node_type = "edg-jdl"; file ="myjob.jdl" ; ]; dependencies = { { nodeA, nodeB }, { nodeA, nodeC } } ]; ]</verbatim> ---+++ Parametric Job Submit it to ICE and JC. Following a sample jdl: <verbatim>// Test parametric job. // The standar Output and Error are parametrizied such as the // enviroment variable: MYPATH_PARAM_. [ JobType = "parametric"; Executable = "/usr/bin/env"; Environment = {"MYPATH_PARAM_=$PATH:/bin:/usr/bin:$HOME"}; StdOutput = "echo_PARAM_.out"; StdError = "echo_PARAM_.err"; OutputSandbox = {"echo_PARAM_.out","echo_PARAM_.err"}; Requirements = ( RegExp("/cream-",other.GlueCEUniqueID)); Parameters = 5; usertags = [ jdl = "parametric cream" ]; ]</verbatim> ---+++ Collection Job ---++ WMS Job shallow and deep re-submission * For the =shallow= resubmission submit the following jdl file: <verbatim> [ requirements = (other.GlueCEStateStatus == "Production"); Rank = -2 * other.GlueCEStateWaitingJobs; Executable = "/bin/ls"; prologue = "/bin/false"; shallowretrycount = 3; RetryCount = 0; usertags = [ exe = "shallowresub" ]; ]</verbatim> * The job should Aborted with message: *hit job shallow retry count (3)* * Check the logging-info (-v 2) of the job, you should find 3 events from the !WorkloadManager like this one:<verbatim> Event: Resubmission - Arrived = Fri Jun 11 13:11:22 2010 CEST - Host = devel18.cnaf.infn.it - Reason = token still exists - Result = SHALLOW - Source = WorkloadManager - Src instance = 13035 - Tag = /var/glite/SandboxDir/LW/https_3a_2f_2fdevel07.cnaf.infn.it_3a9000_2fLWVHkUN2cBOiEcWR7Rc6ew/token.txt - Timestamp = Fri Jun 11 13:11:22 2010 CEST</verbatim> * For the =deep= resubmission submit the following jdl:<verbatim> [ requirements = (other.GlueCEStateStatus == "Production"); Rank = -2 * other.GlueCEStateWaitingJobs; Executable = "/bin/ls"; prologue = "/bin/false"; shallowretrycount = -1; RetryCount = 3; usertags = [ exe = "deepresub" ]; ]</verbatim> * The job should Aborted with message: *hit job retry count (3)* * Check the logging-info (-v 2) of the job, you should find 3 events from the !WorkloadManager like this one:<verbatim> Event: Resubmission - Arrived = Fri Jun 11 14:15:19 2010 CEST - Host = devel18.cnaf.infn.it - Level = SYSTEM - Priority = synchronous - Reason = shallow resubmission is disabled - Result = WILLRESUB - Seqcode = UI=000000:NS=0000000005:WM=000016:BH=0000000000:JSS=000009:LM=000018:LRMS=000000:APP=000000:LBS=000000 - Source = WorkloadManager</verbatim> ---++ WMS Job List-match Testing ---+++ With data * Set these variables:<verbatim> export LCG_GFAL_INFOSYS=<BDII set in the wms conf file> (e.g. cert-bdii-04.cnaf.infn.it:2170) export LFC_HOST=<lfc host name> (e.g. lfcserver.cnaf.infn.it) export LFC_HOME=<lfc home directory> (e.g. lfcserver.cnaf.infn.it:/grid/infngrid)</verbatim> * Register a file to a SE using lcg-cr command: <verbatim> lcg-cr --vo <VO name> -d <SE host> -l lfn:<logical file name> file:<local file path></verbatim> * Using the lcg-rep you can create some replicas: <verbatim> lcg-rep --vo <VO name> -d <SE host> lfn:<logical file name></verbatim> * lcg-lr is useful to see the list of replicas: <verbatim> lcg-lr --vo <VO name> lfn:<logical file name></verbatim> * Submit a list-match command with a jdl likes this one (as !InputData put the lfn(s) registered before):<verbatim> ########################################### # JDL with Data Requirements # ########################################### Executable = "calc-pi.sh"; Arguments = "1000"; StdOutput = "std.out"; StdError = "std.err"; Prologue = "prologue.sh"; InputSandbox = {"calc-pi.sh", "fileA", "fileB","prologue.sh"}; OutputSandbox = {"std.out", "std.err","out-PI.txt","out-e.txt"}; Requirements = true; DataRequirements = { [ DataCatalogType = "DLI"; DataCatalog = "http://lfcserver.cnaf.infn.it:8085"; InputData = {"lfn:/grid/infngrid/cesini/PI_1M.txt","lfn:/grid/infngrid/cesini/e-2M.txt"}; ] }; DataAccessProtocol = "gsiftp";</verbatim> * You should obtain an output like this one:<verbatim> [ale@cream-15 DataReq]$ glite-wms-job-list-match -a --rank --config ~/UI/etc/wmp_wms007.conf data-req.jdl Connecting to the service https://wms007.cnaf.infn.it:7443/glite_wms_wmproxy_server ========================================================================== COMPUTING ELEMENT IDs LIST The following CE(s) matching your job requirements have been found: *CEId* *Rank* - cert-15.pd.infn.it:8443/cream-lsf-cert 0 - gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-cert 0 - prod-ce-01.pd.infn.it:8443/cream-lsf-cert 0 - prod-ce-02.pd.infn.it:2119/jobmanager-lcglsf-cert 0 - t2-ce-01.to.infn.it:8443/cream-pbs-cert 0 - t2-ce-01.to.infn.it:8443/cream-pbs-short 0 - t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-cert 0 - t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-short 0 - test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cert 0 - test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-parallel 0 - devce.cnaf.infn.it:8443/cream-pbs-cert -700634 ==========================================================================</verbatim> * Finally you can check if these CEs are really closed to the SEs where you replicate your file(s) * First list the SEs:<verbatim> [ale@cream-15 DataReq]$ lcg-lr --vo dteam lfn:/grid/infngrid/cesini/PI_1M.txt srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/infngrid/generated/2009-11-24/file0b6a613c-e1f1-4e53-a328-2152e44a8576 srm://gridit-se-01.cnaf.infn.it/dpm/cnaf.infn.it/home/infngrid/generated/2009-11-24/fileb9f3b972-806c-48a6-92fc-a9677901403a srm://prod-se-01.pd.infn.it/infngrid/generated/2009-11-24/file10b5fe11-ad95-41e6-a3a3-14bc5cb9694b srm://t2-se-00.to.infn.it/dpm/to.infn.it/home/infngrid/generated/2009-11-24/file1bb28727-1356-4fb1-bb73-5362f93e766f</verbatim> * then for each SEs in the list use this command to find the "Close" CEs:<verbatim> ldapsearch -x -h <BDII set in the WMS conf file> -p 2170 -b mds-vo-name=local,o=grid "(&(objectclass=GlueCESEBindGroup)(GlueCESEBindGroupSEUniqueID=<SE HOST>))" | grep ^GlueCESEBindGroupCEUniqueID GlueCESEBindGroupCEUniqueID: devce.cnaf.infn.it:8443/cream-pbs-cert GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs GlueCESEBindGroupCEUniqueID: cert-08.cnaf.infn.it:8443/cream-lsf-pps GlueCESEBindGroupCEUniqueID: cert-05.cnaf.infn.it:8443/cream-lsf-pps GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-par GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs GlueCESEBindGroupCEUniqueID: cert-07.cnaf.infn.it:8443/cream-lsf-pps GlueCESEBindGroupCEUniqueID: cert-09.cnaf.infn.it:8443/cream-lsf-pps GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs GlueCESEBindGroupCEUniqueID: cert-06.cnaf.infn.it:8443/cream-lsf-pps GlueCESEBindGroupCEUniqueID: cert-04.cnaf.infn.it:8443/cream-lsf-pps GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cer GlueCESEBindGroupCEUniqueID: cert-13.cnaf.infn.it:8443/cream-lsf-pps</verbatim> * You can find more CEs because you should also filter using the "vo" attribute. ---++ Others ---+++ !BrokerInfo * Use a jdl like this one to verify its creation:<verbatim> ########################################### # JDL to test Brokerinfo Creation # ########################################### Executable = "/bin/ls"; Arguments = "-la"; StdOutput = "std.out"; StdError = "std.err"; FuzzyRank = true; InputSandbox = {"calc-pi.sh", "fileA", "fileB"}; OutputSandbox = {"std.out", "std.err",".BrokerInfo"}; Requirements = (other.GlueCEInfoHostName != "spacin-ce1.dma.unina.it") && !regexp("8443/cream", other.GlueCEUniqueID) ; DataRequirements = { [ DataCatalogType = "DLI"; DataCatalog = "http://lfcserver.cnaf.infn.it:8085"; InputData = {"lfn:/grid/infngrid/cesini/PI_1M.txt","lfn:/grid/infngrid/cesini/e-2M.txt"}; ] }; DataAccessProtocol = "gsiftp"; </verbatim> * After the retrieving of the output files you should see these files:<verbatim> [ale@cream-15 DataReq]$ ls -la /tmp/ale_5XbbEhIAtjO5uC5wRyCmyA total 1264 drwxr-xr-x 2 ale ale 4096 Mar 4 10:31 . drwxrwxrwt 17 root root 1269760 Mar 4 10:31 .. -rw-rw-r-- 1 ale ale 7804 Mar 4 10:31 .BrokerInfo -rw-rw-r-- 1 ale ale 0 Mar 4 10:31 std.err -rw-rw-r-- 1 ale ale 624 Mar 4 10:31 std.out</verbatim> And also:<verbatim> [ale@cream-15 DataReq]$ cat /tmp/ale_5XbbEhIAtjO5uC5wRyCmyA/std.out total 32 drwxr-xr-x 2 dteam038 dteam 4096 Mar 4 10:27 . drwx------ 5 dteam038 dteam 4096 Mar 4 10:27 .. -rw-r--r-- 1 dteam038 dteam 7804 Mar 4 10:27 .BrokerInfo -rw-r--r-- 1 dteam038 dteam 216 Mar 4 10:27 calc-pi.sh -rw-r--r-- 1 dteam038 dteam 17 Mar 4 10:27 fileA -rw-r--r-- 1 dteam038 dteam 17 Mar 4 10:27 fileB -rw-r--r-- 1 dteam038 dteam 166 Mar 4 10:27 https_3a_2f_2fdevel17.cnaf.infn.it_3a9000_2f5XbbEhIAtjO5uC5wRyCmyA.output -rw-r--r-- 1 dteam038 dteam 0 Mar 4 10:27 std.err -rw-r--r-- 1 dteam038 dteam 0 Mar 4 10:27 std.out -rw------- 1 dteam038 dteam 0 Mar 4 10:27 tmp.LesGC30313 </verbatim> * Check the goodness of the .BrokerInfo file generated: * glite-brokerinfo getCE -f glite-brokerinfo getCE -f < .Brokerinfo file path > * It should return the matched CE * glite-brokerinfo getDataAccessProtocol -f < .Brokerinfo file path > * It should return the _DataAccessProtocol_ parameter as specified in the jdl * glite-brokerinfo getInputData -f < .Brokerinfo file path > * It should return the _InputData_ files specified in the jdl * glite-brokerinfo getSEs -f < .Brokerinfo file path > * It should return all the SEs (seen by the BDII of the WMS) that store all the files specified in the !InputData attribute (You can chek it using the command =lcg-lr --vo <VO name> <logical file name>=) * glite-brokerinfo getCloseSEs -f < .Brokerinfo file path > * It should return only the !CloseSe to the matched CE (to see them use the command: =ldapsearch -x -h <specify the BDII host name> -p 2170 -b mds-vo-name=local,o=grid "(&(objectclass=GlueCESEBindGroup)(!GlueCESEBindGroupCEUniqueID=<specify the CE name>))" | grep !GlueCESEBindGroupSEUniqueID=) * glite-brokerinfo getLFN2SFN <LFN of the files in !InputData > -f < .Brokerinfo file path > * It should return the output of the command =lcg-lr --vo <VO name> <logical file name>= * glite-brokerinfo getVirtualOrganization -f < .Brokerinfo file path > * It should return your VO ---+++ !Prologue and Epilogue jobs * Submit a jdl like this one:<verbatim> ########################################### # JDL with Prologue and Epilogue # ########################################### Executable = "exe.sh"; Arguments = "1000"; StdOutput = "std.out"; StdError = "std.err"; Prologue = "prologue.sh"; Epilogue = "epilogue.sh"; FuzzyRank = true; InputSandbox = {"exe/exe.sh", "data/pippo", "exe/epilogue.sh","exe/prologue.sh"}; OutputSandbox = {"std.out", "std.err", "prologue.out", "epilogue.out"}; #Requirements = regexp("8443/cream", other.GlueCEUniqueID); #Requirements = regexp("2119/jobmanager", other.GlueCEUniqueID); RetryCount = 1; ShallowRetryCount = 2;</verbatim> * Where:<verbatim> [ale@cream-15 UI]$ cat exe/exe.sh #!/bin/sh date echo "Hello world!" echo "My prologue said:" cat prologue.out echo "My argument is $1" [ale@cream-15 UI]$ cat exe/prologue.sh #!/bin/sh date > prologue.out echo "I'm the prologue" >> prologue.out echo "##########################" >> prologue.out echo "This is the output of an ls command" >> prologue.out ls >> prologue.out echo "##########################" >> prologue.out [ale@cream-15 UI]$ cat exe/epilogue.sh #!/bin/sh echo " This is the output of the job: `cat std.out`" > epilogue.out echo "#########################" >> epilogue.out echo "We finish at `date`" >> epilogue.out echo "#########################" >> epilogue.out echo "All the jokes are done!" >> epilogue.out</verbatim> * Use the requirements expression to choose an LCG or a CREAM ce * When the job is Done retrieve the output sandbox, it should contains these files: * epilogue.out * prologue.out * std.err and std.out * This should be the contenent of the files:<verbatim> [ale@cream-15 UI]$ cat prologue.out Fri Jun 11 10:24:15 BST 2010 I'm the prologue ########################## This is the output of an ls command epilogue.sh exe.sh pippo prologue.out prologue.sh ########################## [ale@cream-15 UI]$ cat std.out Fri Jun 11 10:24:15 BST 2010 Hello world! My prologue said: Fri Jun 11 10:24:15 BST 2010 I'm the prologue ########################## This is the output of an ls command epilogue.sh exe.sh pippo prologue.out prologue.sh ########################## My argument is 1000 [ale@cream-15 UI]$ cat epilogue.out This is the output of the job: Fri Jun 11 10:24:15 BST 2010 Hello world! My prologue said: Fri Jun 11 10:24:15 BST 2010 I'm the prologue ########################## This is the output of an ls command epilogue.sh exe.sh pippo prologue.out prologue.sh ########################## My argument is 1000 ######################### We finish at Fri Jun 11 10:24:15 BST 2010 ######################### All the jokes are done!</verbatim> -- Main.ElisabettaMolinari - 2010-02-24
Edit
|
Attach
|
PDF
|
H
istory
:
r30
|
r13
<
r12
<
r11
<
r10
|
B
acklinks
|
V
iew topic
|
More topic actions...
Topic revision: r11 - 2010-06-14
-
AlessioGianelle
Home
Site map
CEMon web
CREAM web
Cloud web
Cyclops web
DGAS web
EgeeJra1It web
Gows web
GridOversight web
IGIPortal web
IGIRelease web
MPI web
Main web
MarcheCloud web
MarcheCloudPilotaCNAF web
Middleware web
Operations web
Sandbox web
Security web
SiteAdminCorner web
TWiki web
Training web
UserSupport web
VOMS web
WMS web
WMSMonitor web
WeNMR web
EgeeJra1It Web
Create New Topic
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Account
Log In
Edit
Attach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback