Tags:
, view all tags

WMS 3.5 pre-certification report

Basic functionality testing

Testing instance at devel09.cnaf.infn.it (SL5.8). Deployed on Jan, 31th, 2013

[root@devel09 ~]# rpm -qa | grep glite-wms
glite-wms-purger-3.5.0-3.sl5
glite-wms-utils-classad-3.4.1-1.sl5
glite-wms-configuration-3.5.0-3.sl5
glite-wms-interface-3.5.0-3.sl5
glite-wms-core-3.5.0-5.sl5
glite-wms-common-3.5.0-3.sl5
glite-wms-jobsubmission-lib-3.5.0-3.sl5
glite-wms-utils-exception-3.4.1-1.sl5
glite-wms-ice-3.5.0-3.sl5
glite-wms-jobsubmission-3.5.0-3.sl5

job submission
[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server ls.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/ijBb9I_UnWn-SMueihQmXw

==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/ijBb9I_UnWn-SMueihQmXw


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/ijBb9I_UnWn-SMueihQmXw
Current Status:     Ready
Status Reason:      unavailable
Destination:        w-ce05.grid.sinica.edu.tw:2119/jobmanager-lcgpbs-dteam
Submitted:          Wed Jan 16 17:06:05 2013 CET
==========================================================================

list-match

[mcecchi@devel15 ~]$ glite-wms-job-list-match --rank -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server ls.jdl |tail
 - vgn003.hep.physik.uni-siegen.de:8443/cream-pbs-dteam             1
 - vm3.tier2.hep.manchester.ac.uk:8443/cream-pbs-long               1
 - vserv11.hep.phy.cam.ac.uk:8443/cream-pbs-express                 1
 - vserv11.hep.phy.cam.ac.uk:8443/cream-pbs-short                   1
 - wario.univ-lille1.fr:8443/cream-pbs-dteam                        1
 - wipp-crm.weizmann.ac.il:8443/cream-pbs-dteam                     1

==========================================================================

cancel

Cancel of a Running job:

======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ
Current Status:     Scheduled
Status Reason:      unavailable
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Mon Feb  4 16:00:08 2013 CET
==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-cancel https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ

Are you sure you want to remove specified job(s) [y/n]y : y

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


============================= glite-wms-job-cancel Success =============================

The cancellation request has been successfully submitted for the following job(s):

- https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ

========================================================================================


[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ
Current Status:     Cancelled
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Mon Feb  4 16:00:08 2013 CET
==========================================================================

Cancel of a Done job:

[mcecchi@devel15 ~]$ glite-wms-job-cancel https://devel09.cnaf.infn.it:9000/XbwZJuMcbwQ68XVvHm6ztQ

Are you sure you want to remove specified job(s) [y/n]y : y

Error - Cancel not allowed
Current Job Status is Done

delegation

[mcecchi@devel15 ~]$ glite-wms-job-delegate-proxy --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server -d mc

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


================== glite-wms-job-delegate-proxy Success ==================

Your proxy has been successfully delegated to the WMProxy(s):
https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server
with the delegation identifier: mc

==========================================================================

Check publication in EMIR

Put in your siteinfo.def something like:

BDII_EMIR_ENABLE=yes
BDII_EMIR_SERP_PERIOD=1
BDII_EMIR_SERP_VALIDITY=2
BDII_EMIR_HOST=http://emitbdsr1.cern.ch:9126

Then run:

[root@devel09 ~]#PATH=$PATH:/sbin /opt/glite/yaim/bin/yaim -r -f config_emir_serp -s siteinfo_/site-info.def

   WARNING: 
   WARNING: *****************************************************************************
   WARNING: Your siteinfo directory is world readable. This is generally a bad idea      
   WARNING: as configuration files may contain passwords and other sensitive information.
   WARNING: *****************************************************************************
   WARNING: 
   INFO: Configuring HOST: devel09.cnaf.infn.it
   INFO: Using site configuration file: siteinfo_/site-info.def
   INFO: 
         ###################################################################
         
         .             /'.-. ')
         .     yA,-"-,( ,m,:/ )   .oo.     oo    o      ooo  o.     .oo
         .    /      .-Y a  a Y-.     8. .8'    8'8.     8    8b   d'8
         .   /           ~ ~ /         8'    .8oo88.     8    8  8'  8
         . (_/         '===='          8    .8'     8.   8    8  Y   8
         .   Y,-''-,Yy,-.,/           o8o  o8o    o88o  o8o  o8o    o8o
         .    I_))_) I_))_)
         
         
         current working directory: /root
         site-info.def date: Jan 30 15:39 siteinfo_/site-info.def
         yaim command: -r -f config_emir_serp -s siteinfo_/site-info.def
         log file: /opt/glite/yaim/bin/../log/yaimlog
         Wed Jan 30 15:50:43 CET 2013 : /opt/glite/yaim/bin/yaim
         
         Installed YAIM versions:
         glite-lb-yaim 4.5.8-1
         glite-yaim-bdii 4.3.13-1
         glite-yaim-core 5.1.0-1
         
         ####################################################################
   INFO: The default location of the grid-env.(c)sh files will be: /usr/libexec
   INFO: Sourcing the utilities in /opt/glite/yaim/functions/utils
   INFO: Detecting environment
   INFO: Assuming the node types:  WMSLB
   INFO: Using hostname: devel09.cnaf.infn.it
   WARNING: config_emir_serp function is not supported by  WMSLB!
   WARNING: Anyway you can force this operation.
   WARNING: Would you really run config_emir_serp on  WMSLB? [y/]: 
   INFO: Executing function: config_emir_serp_check 
   INFO: Executing function: config_emir_serp 
EMIR-SERP startup                                          [  OK  ]
   INFO: YAIM terminated succesfully.

Check that the record has reached its destination

mcecchi@pccecchi ~ $ curl -v -H "Accept:application/xml" -X GET http://emitbdsr1.cern.ch:9126/services?Service_Endpoint_ID=devel09.cnaf.infn.it_wmproxy_2272271975_org.glite.wms.WMProxy_2272271975
* About to connect() to emitbdsr1.cern.ch port 9126 (#0)
*   Trying 128.142.136.66... connected
* Connected to emitbdsr1.cern.ch (128.142.136.66) port 9126 (#0)
> GET /services?Service_Endpoint_ID=devel09.cnaf.infn.it_wmproxy_2272271975_org.glite.wms.WMProxy_2272271975 HTTP/1.1
> User-Agent: curl/7.21.0 (i386-redhat-linux-gnu) libcurl/7.21.0 NSS/3.12.10.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.4
> Host: emitbdsr1.cern.ch:9126
> Accept:application/xml
> 
< HTTP/1.1 200 OK
< Content-Type: application/xml
< Vary: Accept-Encoding
< Content-Length: 981
< Server: Jetty(8.1.4.v20120524)
< 
* Connection #0 to host emitbdsr1.cern.ch left intact
* Closing connection #0
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><ns2:QueryResult xmlns="http://schemas.ogf.org/glue/2009/03/spec_2.0_r1" xmlns:ns2="http://www.eu-emi.eu/emiregistry" count="1"><Service BaseType="Entity" CreationTime="2013-01-30T15:22:41.000+01:00"><ID>devel09.cnaf.infn.it_wmproxy_2272271975</ID><Name>INFN-CNAF-WMProxy</Name><Extensions/><Type>org.glite.wms.WMProxy</Type><Location/><Endpoint CreationTime="2013-01-30T15:22:41.000+01:00"><ID>devel09.cnaf.infn.it_wmproxy_2272271975_org.glite.wms.WMProxy_2272271975</ID><URL>https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server</URL><Capability>executionmanagement.candidatesetgenerator</Capability><Capability>executionmanagement.jobdescription</Capability><Capability>executionmanagement.jobmanager</Capability><Technology>webservice</Technology><InterfaceName>org.glite.wms.WMProxy</InterfaceName><InterfaceVersion>1.0</InterfaceVersion><QualityLevel>production</QualityLevel></Endpoint></Service></ns2:QueryResult>mcecchi@pccecchi ~ $ 

Argus based authZ

see bug #99092 test report, below.

LIST OF BUGS

WMS stops accepting submissions when no swap is available (bug #96136) Yes / Done
Set asyncjobstart=false to avoid lost jobs (bug #97319) Yes / Done
Job perusal is broken in EMI-2 WMS (bug #98279) Yes / Done
WMS to limit ISB size as a whole, not only per file (bug #98280) No
EMI-2 WMS UI glite-wms-job-output creates files without checking VO membership first, also changing status (bug #98281) No
various issues with EMI-1/2 proxy cleanup (bug #98282) Yes / Done
classad plugin functions are broken (bug #100176) Yes / Done
glite-wms-job-output requires a valid delegated proxy on the WMS to enable output retrieval (bug #98524) Yes / Done
ICE removes its pid file at the start() instead of stop() (bug #98724) Yes / Done
wmproxy init script 'status' does not return error when service is not running (bug #99017) Yes / Done
WMS: fix various bugs on Argus authZ (bug #99092) Yes / Done

BUG VERIFICATION

BUG: WMS: stops accepting submissions when no swap is available (bug #96136). Certified by MC 31/12/2013
[root@devel09 ~]# /usr/sbin/glite_wms_wmproxy_load_monitor --oper jobSubmit|grep -i swap
Threshold for Swap Usage: 95 => Detected value for Swap Usage: 0.60%
[root@devel09 ~]# cat /proc/swaps 
Filename            Type      Size   Used   Priority
/dev/mapper/VolGroup00-LogVol01         partition   6094840   36808   -1
[root@devel09 ~]# swapoff /dev/mapper/VolGroup00-LogVol01
[root@devel09 ~]# cat /proc/swaps 
[root@devel09 ~]# /usr/sbin/glite_wms_wmproxy_load_monitor --oper jobSubmit|grep -i swap
Threshold for Swap Usage: 95 => Detected value for Swap Usage: 0.00%

BUG: Set asyncjobstart=false to avoid lost jobs (bug #97319). Certified by MC 30/12/2013

[root@devel09 ~]# grep -i asyncj /etc/glite-wms/glite_wms.conf
    AsyncJobStart  =  false;

BUG: classad plugin functions are broken (bug #100176). Certified by MC 30/12/2013

On a UI, list-match for this JDL:

[mcecchi@devel15 ~]$ cat fqanmember_emi2.jdl 
[
Executable = "/bin/echo";
Arguments = "Hello";
StdOutput = "out.log";
StdError = "err.log";
InputSandbox = {};
OutputSandbox = {"out.log", "err.log"};
leftVOMS_FQAN="/gridit/ansys/Role=NULL/Capability=NULL";
rightGlueCEACBR= { "VOMS:/gridit/ansys", "VOMS:/gridit/ansys/Role=SoftwareManager"};
requirements = fqanMember(strcat("VOMS:", leftVOMS_FQAN), rightGlueCEACBR) is true;
myproxyserver="";
ShallowRetryCount = -1;
]

if the number of returned queues if more than 0 then the bug is fixed (actually it should be high, because if the classad plugin function works requirements evaluates to true)

[mcecchi@devel15 ~]$ glite-wms-job-list-match --rank -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server fqanmember_emi2.jdl |wc -l
635

BUG: ICE removes its pid file at the start() instead of stop() (bug #98724). Certified by AD (31/01/2013, 11:30)

In the following log paste it is shown that after ICE stop the pid file /var/run/glite-wms-ice-safe.pid is correctly removed

[root@cream-01 siteinfo]# ps -ef|grep ice
root     23205  1918  0 11:31 pts/0    00:00:00 grep ice
[root@cream-01 siteinfo]# /etc/init.d/glite-wms-ice start
starting ICE... ok
[root@cream-01 siteinfo]# ll /var/run/glite-wms-ice-safe.pid
-rw-r--r-- 1 root root 6 Jan 31 11:31 /var/run/glite-wms-ice-safe.pid
[root@cream-01 siteinfo]# ps -ef|grep ice
glite    23232     1  0 11:31 ?        00:00:00 /usr/bin/glite-wms-ice-safe --conf glite_wms.conf --daemon /var/run/glite-wms-ice-safe.pid
glite    23237 23232  0 11:31 ?        00:00:00 sh -c /usr/bin/glite-wms-ice --conf glite_wms.conf /var/log/wms/ice_console.log 2>&1
glite    23238 23237  0 11:31 ?        00:00:00 /usr/bin/glite-wms-ice --conf glite_wms.conf /var/log/wms/ice_console.log
root     23275  1918  0 11:31 pts/0    00:00:00 grep ice
[root@cream-01 siteinfo]# /etc/init.d/glite-wms-ice stop
stopping ICE... ok
[root@cream-01 siteinfo]# ps -ef|grep ice
root     23309  1918  0 11:31 pts/0    00:00:00 grep ice
[root@cream-01 siteinfo]# ll /var/run/glite-wms-ice-safe.pid
ls: /var/run/glite-wms-ice-safe.pid: No such file or directory

BUG: various issues with EMI-1/2 proxy cleanup (bug #98282). Certified by MC 31/12/2013

Just perform these checks:

[root@devel09 ~]# ll /etc/cron.hourly
total 4
-rw-r--r-- 1 root root 351 Nov 22 12:26 purge_proxycache
[root@devel09 ~]# cat /etc/cron.hourly/purge_proxycache 
#!/bin/sh

exec >> /var/log/proxycache-cleanup.log 2>&1 < /dev/null

echo "===== `date` ====="

cd /var/proxycache/cache || exit

for i in *
do
(
test -d "$i" && cd "$i" || exit

echo "=== $i"

find . -maxdepth 1 -mindepth 1 -type d -mmin +5 -ls -exec \
sh -c '
k=$0/userkey.pem
[ -e "$k" ] && ls -l "$k" && rm "$k"
rmdir "$0"
' {} \;
)
done

echo ''

[root@devel09 ~]# cat /etc/cron.d/glite-wms-wmproxy-purge-proxycache.cron
HOME=/
MAILTO=wms-support@cnaf.infn.it

0 */2 * * * root . /usr/libexec/grid-env.sh ; /usr/bin/glite-wms-wmproxy-purge-proxycache /var/proxycache >> /var/log/wms/glite-wms-wmproxy-purge-proxycache.log 2>&1
[root@devel09 ~]# cat /etc/cron.d/glite-wms-wmproxy-purge-proxycache_keys.cron 
0 */2 * * * root sh /usr/bin/glite-wms-wmproxy-purge-proxycache_keys >> /var/log/wms/glite-wms-wmproxy-purge-proxycache-keys.log 2>&1 < /dev/null
[root@devel09 ~]#

[root@devel09 ~]# cat /usr/bin/glite-wms-wmproxy-purge-proxycache_keys
#!/bin/sh

log() {
  echo "`date`: $1"
}

log "proxy cache leftover private keys cleaner started"

cd /var/proxycache/cache || exit

for i in *
do
(
test -d "$i" && cd "$i" || exit

echo "=== $i"

find . -maxdepth 1 -mindepth 1 -type d -mmin +5 -ls -exec \
sh -c '
k=$0/userkey.pem
[ -e "$k" ] && ls -l "$k" && rm "$k"
rmdir "$0"
' {} \;
)
done

log "finished cleaning keys"
[root@devel09 ~]# sh /usr/bin/glite-wms-wmproxy-purge-proxycache_keys
Thu Jan 31 16:17:13 CET 2013: proxy cache leftover private keys cleaner started
Thu Jan 31 16:17:13 CET 2013: finished cleaning keys
[root@devel09 ~]# ll /var/proxycache/cache/
total 0
[root@devel09 ~]# 

On the UI:

[mcecchi@devel15 ~]$ voms-proxy-init --voms testers.eu-emi.eu
Enter GRID pass phrase:
Your identity: /C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Marco Cecchi
Creating temporary proxy ....................... Done
Contacting  emitestbed01.cnaf.infn.it:15002 [/C=IT/O=INFN/OU=Host/L=CNAF/CN=emitestbed01.cnaf.infn.it] "testers.eu-emi.eu" Done
Creating proxy ....................................... Done

Your proxy is valid until Thu Jan 31 22:26:29 2013

BUG: wmproxy init script 'status' does not return error when service is not running (bug #99017). Certified by MC 31/12/2013

[root@devel09 ~]# /etc/init.d/glite-wms-wmproxy stop
Stopping /usr/bin/glite_wms_wmproxy_server... ok
[root@devel09 ~]# /etc/init.d/glite-wms-wmproxy status
/usr/bin/glite_wms_wmproxy_server is not running
[root@devel09 ~]# echo $? 
1

BUG: glite-wms-job-output requires a valid delegated proxy on the WMS to enable output retrieval (bug #98524)

[mcecchi@devel15 ~]$ cat no_renewal.jdl 
[
Executable = "/bin/echo";
EnableZIppedISB=true;
Arguments = "Hello";
StdOutput = "out.log";
StdError = "err.log";
InputSandbox = {};
OutputSandbox = {"out.log", "err.log"};
requirements = true;
rank = 1;
AllowZippedISB = false;
myproxyserver="";
ShallowRetryCount = -1;
]

[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server no_renewal.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


Warning - Unable to submit the job to the service: https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server
Argus denied authorization on jobSubmit issued by DN: /C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Marco Cecchi
Error code: SOAP-ENV:Server


Error - Operation failed
Unable to find any endpoint where to perform service request

[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server no_renewal.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ
Current Status:     Waiting
Status Reason:      unavailable
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Thu Jan 31 12:50:34 2013 CET
==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ
Current Status:     Ready
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Thu Jan 31 12:50:34 2013 CET
==========================================================================

BUG: WMS: fix various bugs on Argus authZ (bug #99092). Certified by MC 31/12/2013

WMS:

[root@devel09 ~]# rpm -qa | grep glite-wms
glite-wms-utils-classad-3.4.1-1.sl5
glite-wms-purger-3.5.0-3.sl5
glite-wms-jobsubmission-3.5.0-3.sl5
glite-wms-core-devel-3.5.0-4.sl5
glite-wms-common-3.5.0-3.sl5
glite-wms-configuration-3.5.0-3.sl5
glite-wms-interface-3.5.0-3.sl5
glite-wms-ice-3.5.0-3.sl5
glite-wms-core-3.5.0-5.sl5
glite-wms-jobsubmission-lib-3.5.0-3.sl5
glite-wms-utils-exception-3.4.1-1.sl5

Check that siteinfo.def contains:

USE_ARGUS=yes
ARGUS_PEPD_ENDPOINTS="https://emitestbed30.cnaf.infn.it:8154/authz"

and run the configuration, if not already done. Submit a job a look at the wmp log:

WMS:

[root@devel09 SandboxDir]# tail -f /var/log/wms/wmproxy.log|grep -i argus
04 Feb, 15:44:29 -D- PID: 16297 - "WMPAuthorizer::authorize": Argus authZ and mapping
04 Feb, 15:44:29 -D- PID: 16297 - "argus_authZ": PEP version: argus-pep-api-c/2.2.0 (libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5)
04 Feb, 15:44:29 -I- PID: 16297 - "argus_authZ": creating XACML request for argus
04 Feb, 15:45:04 -D- PID: 16297 - "argus_autZ::get_response": argus decision permit
04 Feb, 15:45:04 -D- PID: 16297 - "argus_autZ::get_response": argus mapped user = tst27
04 Feb, 15:45:04 -D- PID: 16297 - "argus_autZ::get_response": argus mapped group = testers
04 Feb, 15:45:04 -I- PID: 16297 - "WMPAuthorizer::authorize": Argus returned PERMIT with mapping uid: 61027, gid: 6100
04 Feb, 15:45:04 -D- PID: 16297 - "WMPAuthorizer::authorize": Argus authZ and mapping
04 Feb, 15:45:04 -D- PID: 16297 - "argus_authZ": PEP version: argus-pep-api-c/2.2.0 (libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5)
04 Feb, 15:45:04 -I- PID: 16297 - "argus_authZ": creating XACML request for argus
04 Feb, 15:45:05 -D- PID: 16297 - "argus_autZ::get_response": argus decision permit
04 Feb, 15:45:05 -D- PID: 16297 - "argus_autZ::get_response": argus mapped user = tst27
04 Feb, 15:45:05 -D- PID: 16297 - "argus_autZ::get_response": argus mapped group = testers
04 Feb, 15:45:05 -I- PID: 16297 - "WMPAuthorizer::authorize": Argus returned PERMIT with mapping uid: 61027, gid: 6100

[root@devel09 SandboxDir]# ll aK/https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2faKh4KHvJF-Dzf69BDWCM7Q/
total 28
drwxrwx--- 2 tst27 glite 4096 Feb  4 15:45 input
-rw-r--r-- 1 glite glite  599 Feb  4 15:45 JDLOriginal
-rw-r--r-- 1 glite glite 2395 Feb  4 15:45 JDLStarted
drwxrwx--- 2 tst27 glite 4096 Feb  4 15:45 output
drwxrwx--- 2 tst27 glite 4096 Feb  4 15:45 peek
-rw------- 1 glite glite 6361 Feb  4 15:45 user.proxy

UI: submit a job with both ISB and OSB

[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server no_renewal.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/aKh4KHvJF-Dzf69BDWCM7Q

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/aKh4KHvJF-Dzf69BDWCM7Q


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/aKh4KHvJF-Dzf69BDWCM7Q
Current Status:     Done(Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Mon Feb  4 15:45:05 2013 CET
==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-output https://devel09.cnaf.infn.it:9000/XbwZJuMcbwQ68XVvHm6ztQ

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


Warning - JobPurging not allowed
 (Unable to copy file)


================================================================================

         JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://devel09.cnaf.infn.it:9000/XbwZJuMcbwQ68XVvHm6ztQ
have been successfully retrieved and stored in the directory:
/tmp/jobOutput/mcecchi_XbwZJuMcbwQ68XVvHm6ztQ

================================================================================


[mcecchi@devel15 ~]$ ll /tmp/jobOutput/mcecchi_XbwZJuMcbwQ68XVvHm6ztQ/out.log 
-rw-rw-r-- 1 mcecchi mcecchi 22 Feb  4 15:57 /tmp/jobOutput/mcecchi_XbwZJuMcbwQ68XVvHm6ztQ/out.log
[mcecchi@devel15 ~]$ cat /tmp/jobOutput/mcecchi_XbwZJuMcbwQ68XVvHm6ztQ/out.log 
a.out
err.log
out.log

BUG: WMS: Job perusal is broken in EMI-2 WMS (bug #98279). Certified by MC 02/02/2013

[mcecchi@devel15 ~]$ cat perusal.jdl 
Executable = "sleeper.sh";
requirements = true;
rank = 1;
EnableZippedIsb=true;
Arguments = "out.txt";
StdOutput = "std.out";
StdError = "std.err";
InputSandbox = {"sleeper.sh"};
OutputSandbox = {"std.out","std.err","out.txt"};
PerusalFileEnable = true;
[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server perusal.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ
Current Status:     Running
Status Reason:      unavailable
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Mon Feb  4 16:17:20 2013 CET
==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-perusal --set -f std.out https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-perusal Success ======================

File perusal has been successfully enabled for the job:
https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-perusal --get -f std.out https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-perusal Success ======================

No files to be retrieved for the job:
https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-perusal --get -f std.out https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-perusal Success ======================

The retrieved files have been successfully stored in:
/tmp/jobOutput/mcecchi_znM6WQNJQqVhMmAPSp4gFQ

==========================================================================


--------------------------------------------------------------------------
file 1/1: std.out-20130204153621_1-20130204153621_1
--------------------------------------------------------------------------

[mcecchi@devel15 ~]$ tail /tmp/jobOutput/mcecchi_znM6WQNJQqVhMmAPSp4gFQ/std.out-20130204153621_1-20130204153621_1 
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second

BUG: glite-wms-job-output requires a valid delegated proxy on the WMS to enable output retrieval (bug #98524)

To reproduce the bug submit a job producing at least one output sandbox file, to a WMS EMI2 server...

dorigoa@cream-51 18:52:15 ~>glite-wms-job-status https://devel09.cnaf.infn.it:9000/G5c9297pMa6nYcRR2WiAzA


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/G5c9297pMa6nYcRR2WiAzA
Current Status:     Done(Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        cream-47.pd.infn.it:8443/cream-lsf-creamtest1
Submitted:          Mon Feb  4 18:52:06 2013 CET
==========================================================================

Then, move to the WMS server and change directory to /var/SandboxDir/G5/https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2f/G5c9297pMa6nYcRR2WiAzA

and remove proxies

[root@cream-28 SandboxDir]# cd G5/https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fG5c9297pMa6nYcRR2WiAzA/
[root@cream-28 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fG5c9297pMa6nYcRR2WiAzA]# ll
total 24
drwxrwx--- 2 dteam042 glite 4096 Feb  4 18:52 input
-rw-r--r-- 1 glite    glite  708 Feb  4 18:52 JDLOriginal
-rw-r--r-- 1 glite    glite 2510 Feb  4 18:52 JDLToStart
drwxrwx--- 2 dteam042 glite 4096 Feb  4 18:52 output
drwxrwx--- 2 dteam042 glite 4096 Feb  4 18:52 peek
lrwxrwxrwx 1 glite    glite   64 Feb  4 18:52 user.proxy -> /var/glite/spool/glite-renewd/45a96bd6a16770e5fdc4c60bbae2646e.0
[root@cream-28 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fG5c9297pMa6nYcRR2WiAzA]# find . -iname "*proxy*"
./.user.proxy.bak
./user.proxy
[root@cream-28 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fG5c9297pMa6nYcRR2WiAzA]# \rm ./.user.proxy.bak ./user.proxy

Go back to the UI and try a glite-wms-job-output that must fail (because of proxies removal):

dorigoa@cream-51 18:53:53 ~>glite-wms-job-output https://devel09.cnaf.infn.it:9000/G5c9297pMa6nYcRR2WiAzA

Connecting to the service https://cream-28.pd.infn.it:7443/glite_wms_wmproxy_server


Error - getOutputFileList Error
 (Proxy exception: Unable to get Not Before date from Proxy)

Now, perform another submission using a EMI3 WMS server:

dorigoa@cream-51 18:59:30 ~>glite-wms-job-status https://devel09.cnaf.infn.it:9000/A1MWywIZXXW8-FMNd0pY9w


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/A1MWywIZXXW8-FMNd0pY9w
Current Status:     Done(Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        cream-47.pd.infn.it:8443/cream-lsf-creamtest1
Submitted:          Mon Feb  4 18:58:57 2013 CET
==========================================================================

Remove proxies from WMS's sandboxdir:

[root@cream-01 ~]# cd /var/SandboxDir/A1/https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fA1MWywIZXXW8-FMNd0pY9w/
[root@cream-01 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fA1MWywIZXXW8-FMNd0pY9w]# ls l
ls: l: No such file or directory
[root@cream-01 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fA1MWywIZXXW8-FMNd0pY9w]# find . -iname "*proxy*"
./.user.proxy.bak
./user.proxy
[root@cream-01 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fA1MWywIZXXW8-FMNd0pY9w]# \rm ./.user.proxy.bak ./user.proxy

Go back to UI and try another job-output, that should be successful:

dorigoa@cream-51 DING! ~>glite-wms-job-output https://devel09.cnaf.infn.it:9000/A1MWywIZXXW8-FMNd0pY9w

Connecting to the service https://cream-01.pd.infn.it:7443/glite_wms_wmproxy_server


Warning - JobPurging not allowed
 (Unable to copy file)


================================================================================

                        JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://devel09.cnaf.infn.it:9000/A1MWywIZXXW8-FMNd0pY9w
have been successfully retrieved and stored in the directory:
/tmp/jobOutput/dorigoa_A1MWywIZXXW8-FMNd0pY9w

================================================================================

dorigoa@cream-51 19:01:54 ~>ls -l /tmp/jobOutput/dorigoa_A1MWywIZXXW8-FMNd0pY9w
total 4
-rw-rw-r-- 1 dorigoa dorigoa 0 Feb  4 19:01 stderr
-rw-rw-r-- 1 dorigoa dorigoa 5 Feb  4 19:01 stdout

-- MarcoCecchi - 2013-01-16

Edit | Attach | PDF | History: r21 | r16 < r15 < r14 < r13 | Backlinks | Raw View | More topic actions...
Topic revision: r14 - 2013-02-04 - AlviseDorigo
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback