• 355 collection of 25 nodes each (total: 8875 jobs)
• One collection every 60 seconds

• DONE OK: 8636 (97.3%)
• Aborted: 239 (proxy expired) (2.7%)

Performance test

Collection of 1000 nodes

• Collection ID: https://devel07.cnaf.infn.it:9000/0sqJfQjOX17cHiF5vONNDw
• Nodes are submitted to a production environment:
  • 320 are executed on a LCG CE
  • 680 are executed in a CREAM CE
• Final Status:
  • 1000 Done (Success)

Stress Test

Description:

• 720 collection of 25 nodes each (total: 18000 jobs)
• One collection every 60 seconds (=> 12 hours)
• two users
• Use LCG-CEs and CREAM-CEs from production enviroment (with different batch systems)
• Use automatic-delegation
• The job is a "sleep random(666)"
• Resubmission is enabled
• Enable proxy renewal

Normal, Collection, parametric, dag and parallel jobs

Perusal

===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the gLite-WMS command line interface  
===> + Description: test glite-wms-job-persual commmad 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

 Test 1 

===> >>> Check if command glite-wms-job-perusal exists

 Test 2 

===> >>> 
===> >>> Test --version option
===> >>> We are testing WMS User Interface version  3.3.2


 Test 3 

===> >>> 
===> >>> Test --set option

 Test 4 

===> >>> 
===> >>> Test --unset option

 Test 5 

===> >>> 
===> >>> Test --set and --config options

 Test 6 

===> >>> 
===> >>> Test --set and --output options

 Test 7 

===> >>> 
===> >>> Test --set and --logfile options

 Test 8 

===> >>> 
===> >>> Test --set --logfile and --debug option

 Test 9 

===> >>> 
===> >>> Test --set and --nodisplay options
===> >>> 
===> >>> Test --set option

 Test 10 

===> >>> 

 BEWARE default min perusal interval is 1000 secs, so this phase could take many minutes 

===> >>> Test --get option

 Test 11 

===> >>> 
===> >>> Test --get option with --dir

 Test 12 

===> >>> 
===> >>> Test --get option with --nodisplay and --all

 Test 13 

===> >>> 
===> 
===> Test: WMS-command_job-perusal.py
===> Started: 15:45:37
===> Ended  : 19:23:49
===> 
===>     === test PASSED === 

Test 1: Submit a normal job to an LCG-CE

Test 2: Submit a normal job to a CREAM CE

Test 3: Submit a normal job

Test 4: Submit a job collection to an LCG-CE

Test 5: Submit a job collection to CREAM CE

Test 6: Submit a job collection

Test 7: Submit a parametric job to an LCG-CE

Test 8: Submit a parametric job to CREAM CE

Test 9: Submit a parametric job

Test 10: Submit a DAG job

Test 11: Submit MPI job to an LCG-CE

Test 12: Submit MPI job to CREAM CE

Test 13: Submit MPI job

===> ===> Test: WMS-job-cycle.py ===> Started: 10:19:43 ===> Ended : 14:48:37 ===> ===> >>> TEST SUCCESS <<<

Test proxy renewal

===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the gLite-WMS command line interface  
===> + Description: WMS test proxy renewal operation 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

Enter the user proxy password:

 Test 1: Test proxy renewal with LCG CE 


 Test 2: Test proxy renewal with CREAM CE 


 Test 3: Test proxy renewal with LCG CE (without setting MYPROXYSERVER) 


 Test 4: Test proxy renewal with CREAM CE (without setting MYPROXYSERVER) 

===> 
===> Test: WMS-proxy-renewal.py
===> WMS: devel18.cnaf.infn.it
===> Started: 13:15:52
===> Ended  : 13:54:50
===> 
===>     === test PASSED === 

Shallow/Deep Resubmission

===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the gLite-WMS command line interface  
===> + Description: WMS Job Resubmission Testing 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

 Test 1: Try a shallow resubmission 

Prologue/Epilogue

===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the gLite-WMS command line interface  
===> + Description: WMS test prologue and epilogue attributes 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

 Test 1: Test prologue attribute 


 Test 2: Test epilogue attribute 


 Test 3: Test prologue and epilogue attributes 

===> 
===> Test: WMS-prologue-epilogue.py
===> WMS: cream-44.pd.infn.it
===> Started: 17:38:28
===> Ended  : 17:43:57
===> 
===>     === test PASSED === 

Cancel

===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the gLite-WMS command line interface  
===> + Description: WMS Job Cancel Testing 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

 Test 1: Try to cancel a normal job 


 Test 2: Try to cancel a DAG job 


 Test 3: Try to cancel one node of a DAG job 


 Test 4: Try to cancel a collection job 


 Test 5: Try to cancel one node of a collection job 


 Test 6: Try to cancel a parametric job 


 Test 7: Try to cancel one node of a parametric job 


 Test 8: Try to cancel a MPI job 

Delegation

===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the gLite-WMS command line interface  
===> + Description: test delegation operation 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

Enter the user proxy password:

 Test 1: Check --autm-delegation option 

Test 3: Try to delegate with a short proxy and check the validity

Test 4: Works with expired proxy

===> ===> Test: WMS-delegation-test.py ===> WMS: cream-44.pd.infn.it ===> Started: 14:00:15 ===> Ended : 14:01:38 ===> =>     test PASSED ==

Test Report

here

List Match

List match without data

===> +++++++++++++++++++++++++++++++++++++++++++++++++++++
===> + TestSuite of the gLite-WMS service                 
===> + Description: WMS Job List-match Testing 
===> +++++++++++++++++++++++++++++++++++++++++++++++++++++

 Test 1: check if command glite-wms-job-list-match exists 


 Test 2: test a simple matching 


 Test 3: check --delegationid option 


 Test 4: match only CREAM CE 


 Test 5: try a failure matching (Requirements == false) 

===> 
===> Test: WMS-listmatch-test.py
===> WMS: cream-44.pd.infn.it
===> Started: 15:42:43
===> Ended  : 15:42:55
===> 
===>     === test PASSED === 

List match with data

[ale@cream-12 DataReq]$ glite-wms-job-list-match -a -c ../etc/wmp_cream-44.conf data-req.jdl

Connecting to the service https://cream-44.pd.infn.it:7443/glite_wms_wmproxy_server

==========================================================================

           COMPUTING ELEMENT IDs LIST 
 The following CE(s) matching your job requirements have been found:

   *CEId*
 - cert-15.pd.infn.it:8443/cream-lsf-cert
 - cert-37.pd.infn.it:8443/cream-lsf-cert
 - cream-19.pd.infn.it:8443/cream-lsf-cert
 - cream-19.pd.infn.it:8443/cream-lsf-creamcert1
 - cream-19.pd.infn.it:8443/cream-lsf-creamcert2
 - cream-24.pd.infn.it:8443/cream-lsf-cert
 - cream-24.pd.infn.it:8443/cream-lsf-testbedB_1
 - cream-24.pd.infn.it:8443/cream-lsf-testbedB_2
 - cream-26:2119/jobmanager-lcgpbs-cert
 - cream-26:2119/jobmanager-lcgpbs-creamtest1
 - cream-26:2119/jobmanager-lcgpbs-creamtest2
 - cream-32.pd.infn.it:8443/cream-pbs-creamtest1
 - cream-32.pd.infn.it:8443/cream-pbs-creamtest2
 - grid010.ct.infn.it:2119/jobmanager-lcgpbs-cert
 - prod-ce-01.pd.infn.it:8443/cream-lsf-cert
 - prod-ce-02.pd.infn.it:2119/jobmanager-lcglsf-cert

==========================================================================

Submission

Normal, Collection, parametric and Dag jobs

• Starting from an sl4 we install the WMS:
  • wget http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.1/lcg-CA.repo -O /etc/yum.repos.d/lcg-CA.repo
  • yum install lcg-CA
  • wget http://etics-repository.cern.ch/repository/pm/registered/repomd/name/patch_4880_1/etics-registered-build-by-name.repo -O /etc/yum.repos.d/glite.repo
  • yum install glite-WMS
  • /opt/glite/yaim/bin/yaim -c -s site-info.def -n WMS

Certification report patch 4880

Author(s): Alessio Gianelle

Outcome: In certification

Information

Patch 4880	WMS 3.2.17
Build reports	http://etics-repository.cern.ch/repository/reports/id/f4f1f0a7-8d46-455a-a2e7-e9764f236ba2/slc4_ia32_gcc346/-/reports/index.html
Repository	http://etics-repository.cern.ch/repository/pm/registered/repomd/name/patch_4880_1/index.html

Clean installation

Upgrade from production

• Starting from a Production WMS:

[root@devel18 etc]# yum update
Loading "kernel-module" plugin
Loading "versionlock" plugin
Loading "installonlyn" plugin
Loading "changelog" plugin
Loading "tsflags" plugin
Loading "protectbase" plugin
Setting up Update Process
Setting up repositories
dag                       100% |=========================| 1.1 kB    00:00     
slc-update                100% |=========================|  951 B    00:00     
slc4-base                 100% |=========================|  951 B    00:00     
slc4-updates              100% |=========================|  951 B    00:00     
EGI-trustanchors          100% |=========================| 1.9 kB    00:00     
sl-base                   100% |=========================| 1.1 kB    00:00     
slc4-extras               100% |=========================|  951 B    00:00     
glite-WMS                 100% |=========================|  951 B    00:00     
slc-base                  100% |=========================| 1.1 kB    00:00     
CA                        100% |=========================|  951 B    00:00     
ETICS-name-patch_4880_1                                          764 B 00:00 
cnaf-local                100% |=========================|  951 B    00:00     
Reading repository metadata in from local files
primary.xml.gz            100% |=========================| 4.4 MB    00:00     
dag       : ################################################## 12410/12410
Added 12410 new packages, deleted 0 old in 50.16 seconds
primary.xml.gz            100% |=========================| 2.7 MB    00:00     
slc-update: ################################################## 8677/8677
Added 8677 new packages, deleted 0 old in 46.93 seconds
primary.xml.gz            100% |=========================| 707 kB    00:00     
slc4-base : ################################################## 1923/1923
Added 1923 new packages, deleted 0 old in 10.18 seconds
primary.xml.gz            100% |=========================| 3.2 MB    00:00     
slc4-updat: ################################################## 8573/8573
Added 8573 new packages, deleted 0 old in 45.44 seconds
primary.xml.gz            100% |=========================|  19 kB    00:00     
EGI-trusta: ################################################## 96/96
Added 96 new packages, deleted 0 old in 0.28 seconds
primary.xml.gz            100% |=========================| 599 kB    00:00     
sl-base   : ################################################## 1848/1848
Added 1848 new packages, deleted 0 old in 9.98 seconds
primary.xml.gz            100% |=========================| 233 kB    00:00     
slc4-extra: ################################################## 1442/1442
Added 1442 new packages, deleted 0 old in 3.17 seconds
primary.xml.gz            100% |=========================|  75 kB    00:00     
glite-WMS : ################################################## 279/279
Added 279 new packages, deleted 0 old in 1.70 seconds
primary.xml.gz            100% |=========================| 604 kB    00:00     
slc-base  : ################################################## 1923/1923
Added 1923 new packages, deleted 0 old in 8.46 seconds
primary.xml.gz            100% |=========================|  15 kB    00:00     
CA        : ################################################## 95/95
Added 95 new packages, deleted 0 old in 0.21 seconds
primary.xml.gz                                                   35 kB 00:00 
ETICS-name: ################################################## 94/94
Added 94 new packages, deleted 0 old in 0.48 seconds
primary.xml.gz            100% |=========================| 2.7 kB    00:00     
cnaf-local: ################################################## 17/17
Added 17 new packages, deleted 0 old in 0.03 seconds
Reading version lock configuration
108 packages excluded due to repository protections
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for glite-jdl-api-cpp to pack into transaction set.
glite-jdl-api-cpp-3.1.18- 100% |=========================| 5.3 kB    00:00     
---> Package glite-jdl-api-cpp.i386 0:3.1.18-0.slc4 set to be updated
---> Downloading header for glite-WMS to pack into transaction set.
glite-WMS-3.1.32-0.slc4.i 100% |=========================| 4.4 kB    00:00     
---> Package glite-WMS.i386 0:3.1.32-0.slc4 set to be updated
--> Running transaction check

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size 
=============================================================================
Updating:
 glite-WMS               i386       3.1.32-0.slc4    ETICS-name-patch_4880_1  5.3 k
 glite-jdl-api-cpp       i386       3.1.18-0.slc4    ETICS-name-patch_4880_1  2.6 M

Transaction Summary
=============================================================================
Install      0 Package(s)         
Update       2 Package(s)         
Remove       0 Package(s)         
Total download size: 2.6 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): glite-jdl-api-cpp-3.1.18-0.slc4.i386.rpm                 2.6 MB 00:00 
(2/2): glite-WMS-3.1.32-0 100% |=========================| 5.3 kB    00:00     
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating  : glite-jdl-api-cpp            ######################### [1/4] 
  Updating  : glite-WMS                    ######################### [2/4] 
  Cleanup   : glite-jdl-api-cpp            ######################### [3/4]
  Cleanup   : glite-WMS                    ######################### [4/4]

Updated: glite-WMS.i386 0:3.1.32-0.slc4 glite-jdl-api-cpp.i386 0:3.1.18-0.slc4
Complete!

Check bugs

Bugs #82654: URLs in ISB are sometimes converted to lowercase by WMproxy FIXED

[ale@cream-12 UI]$ cat 82654.jdl 
[
Type = "collection";
Nodes = {
[
NodeName   = "NodeName_0_0";
Executable = "submit.sh";
Arguments  = "cmsdataops_JH_gLite_test_110519_113545-Sandbox.tar.bz2 9352";
StdOutput  = "9352_0.stdout";
StdError   = "9352_0.stderr";
OutputSandboxBaseDestURI = "gsiftp://vocms37.cern.ch/data/wmagent/osb/JobCache/cmsdataops_JH_gLite_test_110519_113545/DataProcessing/JobCollection_26_8/job_9352";
OutputSandbox = {"Report.0.pkl",".BrokerInfo", "9352_0.stdout","9352_0.stderr"};
InputSandbox = {"gsiftp://vocms37.cern.ch/tmp/glite.submit.fakefile.empty","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/batch_9228-0/JobPackage.pkl",root.inputsandbox[0],root.inputsandbox[1],root.inputsandbox[2]};
],
[
NodeName   = "NodeName_1_0";
Executable = "submit.sh";
Arguments  = "cmsdataops_JH_gLite_test_110519_113545-Sandbox.tar.bz2 6133";
StdOutput  = "6133_0.stdout";
StdError   = "6133_0.stderr";
OutputSandboxBaseDestURI = "gsiftp://vocms37.cern.ch/data/wmagent/osb/JobCache/cmsdataops_JH_gLite_test_110519_113545/DataProcessing/JobCollection_26_5/job_6133";
OutputSandbox = {"Report.0.pkl",".BrokerInfo", "6133_0.stdout","6133_0.stderr"};
InputSandbox = {"gsiftp://vocms37.cern.ch/tmp/glite.submit.fakefile.empty","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/batch_6057-0/JobPackage.pkl",root.inputsandbox[0],root.inputsandbox[1],root.inputsandbox[2]};
]
};
InputSandbox = {"gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/cmsdataops_JH_gLite_test_110519_113545-Sandbox.tar.bz2","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/apps/wmcore/etc/submit.sh","gsiftp://vocms37.cern.ch/data/wmagent/isb/Unpacker.py"};
Requirements = Member("VO-cms-slc5_ia32_gcc434", other.GlueHostApplicationSoftwareRunTimeEnvironment) && (other.GlueHostNetworkAdapterOutboundIP) && other.GlueCEStateStatus == "Production" &&  other.GlueCEPolicyMaxCPUTime>=130  && ( Member("srm.ciemat.es", other.GlueCESEBindGroupSEUniqueID)) ;
MyProxyServer = "myproxy.cern.ch";
VirtualOrganization = "cms";
RetryCount = 0;
DefaultNodeRetryCount = 0;
ShallowRetryCount = -1;
DefaultNodeShallowRetryCount = -1;
SignificantAttributes = {"Requirements", "Rank", "FuzzyRank"};
]

[ale@cream-12 UI]$ glite-wms-job-submit -a -e https://devel18.cnaf.infn.it:7443/glite_wms_wmproxy_server 82654.jdl 

Connecting to the service https://devel18.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel07.cnaf.infn.it:9000/VbYQLwwE3f4LFWEv6-xb0Q

==========================================================================

Then verify the InputSandbox in the submitted jdl:

[ale@cream-12 UI]$ glite-wms-job-info --jdl https://devel07.cnaf.infn.it:9000/VbYQLwwE3f4LFWEv6-xb0Q | grep "InputSandbox ="
        InputSandbox = { "gsiftp://vocms37.cern.ch/tmp/glite.submit.fakefile.empty","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/batch_9228-0/JobPackage.pkl","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/cmsdataops_JH_gLite_test_110519_113545-Sandbox.tar.bz2","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/apps/wmcore/etc/submit.sh","gsiftp://vocms37.cern.ch/data/wmagent/isb/Unpacker.py" }; 
        InputSandbox = { "gsiftp://vocms37.cern.ch/tmp/glite.submit.fakefile.empty","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/batch_6057-0/JobPackage.pkl","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/cmsdataops_JH_gLite_test_110519_113545-Sandbox.tar.bz2","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/apps/wmcore/etc/submit.sh","gsiftp://vocms37.cern.ch/data/wmagent/isb/Unpacker.py" }; 
  InputSandbox = { "gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/cmsdataops_JH_gLite_test_110519_113545-Sandbox.tar.bz2","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/apps/wmcore/etc/submit.sh","gsiftp://vocms37.cern.ch/data/wmagent/isb/Unpacker.py","gsiftp://vocms37.cern.ch/tmp/glite.submit.fakefile.empty","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/batch_9228-0/JobPackage.pkl" }; 

Note, if you try in an old WMS (i.e. pre-patch) you should have:

[ale@cream-12 UI]$ glite-wms-job-info --jdl https://devel17.cnaf.infn.it:9000/Hh7KhY4cn49XIZHpgfLvpQ | grep "InputSandbox ="
        InputSandbox = { "gsiftp://vocms37.cern.ch/tmp/glite.submit.fakefile.empty","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/batch_9228-0/JobPackage.pkl","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/cmsdataops_JH_gLite_test_110519_113545-Sandbox.tar.bz2","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/apps/wmcore/etc/submit.sh","gsiftp://vocms37.cern.ch/data/wmagent/isb/Unpacker.py" }; 
        InputSandbox = { "gsiftp://vocms37.cern.ch/tmp/glite.submit.fakefile.empty","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/workqueuemanager/wf/cmsdataops_jh_glite_test_110519_113545/batch_6057-0/jobpackage.pkl","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/cmsdataops_JH_gLite_test_110519_113545-Sandbox.tar.bz2","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/apps/wmcore/etc/submit.sh","gsiftp://vocms37.cern.ch/data/wmagent/isb/Unpacker.py" }; 
  InputSandbox = { "gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/cmsdataops_JH_gLite_test_110519_113545-Sandbox.tar.bz2","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/apps/wmcore/etc/submit.sh","gsiftp://vocms37.cern.ch/data/wmagent/isb/Unpacker.py","gsiftp://vocms37.cern.ch/tmp/glite.submit.fakefile.empty","gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/WorkQueueManager/wf/cmsdataops_JH_gLite_test_110519_113545/batch_9228-0/JobPackage.pkl" }; 

As you can see the file gsiftp://vocms37.cern.ch/data/projects/v0.7.5/install/wmagent/workqueuemanager/wf/cmsdataops_jh_glite_test_110519_113545/batch_6057-0/jobpackage.pkl in the second entry is expressed in lower case

-- AlessioGianelle - 2011-06-16