WMS36PreCertificationReport < WMS

Tags: , view all tags
---+ WMS 3.5 pre-certification report

---++++ Basic functionality testing

Testing instance at cream-08.pd.infn.it (SL6 x86_64). Deployed on Apr. 2013

<verbatim>
[root@cream-08 ~]# cat /etc/issue
Scientific Linux release 6.2 (Carbon)
Kernel \r on an \m

[root@cream-08 ~]# rpm -qa|grep wms
glite-wms-server-3.6.0-0.sl6.x86_64
glite-wms-utils-exception-3.4.1-1.sl6.x86_64
glite-wms-utils-classad-3.4.1-1.sl6.x86_64

[root@cream-08 ~]# ps -ef|grep wms
glite    19828     1  0 15:20 ?        00:00:00 /usr/bin/glite-wms-log_monitor -c glite_wms.conf
root     21084     1  0 15:24 ?        00:00:00 /usr/sbin/httpd -k start -f /etc/glite-wms/glite_wms_wmproxy_httpd.conf
glite    21115 21084  0 15:24 ?        00:00:00 /usr/sbin/httpd -k start -f /etc/glite-wms/glite_wms_wmproxy_httpd.conf
glite    21116 21084  0 15:24 ?        00:00:00 /usr/sbin/httpd -k start -f /etc/glite-wms/glite_wms_wmproxy_httpd.conf
glite    21117 21084  0 15:24 ?        00:00:00 /usr/sbin/httpd -k start -f /etc/glite-wms/glite_wms_wmproxy_httpd.conf
glite    21118 21084  0 15:24 ?        00:00:00 /usr/sbin/httpd -k start -f /etc/glite-wms/glite_wms_wmproxy_httpd.conf
glite    21119 21084  0 15:24 ?        00:00:00 /usr/sbin/httpd -k start -f /etc/glite-wms/glite_wms_wmproxy_httpd.conf
glite    21120 21084  0 15:24 ?        00:00:00 /usr/sbin/httpd -k start -f /etc/glite-wms/glite_wms_wmproxy_httpd.conf
glite    21121     1 90 15:24 ?        00:00:29 /usr/bin/glite-wms-workload_manager --conf glite_wms.conf --daemon
glite    21169     1  0 15:24 ?        00:00:00 /usr/bin/glite-wms-job_controller -c glite_wms.conf
glite    21207     1  0 15:24 ?        00:00:00 /usr/bin/glite-wms-ice-safe --conf glite_wms.conf --daemon /var/run/glite-wms-ice-safe.pid
glite    21212 21207  0 15:24 ?        00:00:00 sh -c /usr/bin/glite-wms-ice --conf glite_wms.conf /var/log/wms/ice_console.log 2>&1
glite    21213 21212  0 15:24 ?        00:00:00 /usr/bin/glite-wms-ice --conf glite_wms.conf /var/log/wms/ice_console.log

</verbatim>

---+++++ Job submission

----++++ Single job with Input Sandbox

CREAM CE

<verbatim>
dorigoa@cream-14 15:46:16 ~/JDL>glite-wms-job-delegate-proxy -e https://cream-08.pd.infn.it:7443/glite_wms_wmproxy_server -d pippo

Connecting to the service https://cream-08.pd.infn.it:7443/glite_wms_wmproxy_server


================== glite-wms-job-delegate-proxy Success ==================

Your proxy has been successfully delegated to the WMProxy(s):
https://cream-08.pd.infn.it:7443/glite_wms_wmproxy_server
with the delegation identifier: pippo

==========================================================================



dorigoa@cream-14 15:46:21 ~/JDL>cat wms_ISB_cream.jdl 
[
Executable = "/bin/sleep" ;
Arguments = "15";
InputSandbox =  {"/etc/group"};
InputSandbox = {"/home/dorigoa/my_sandbox_01.gz", "/home/dorigoa/my_sandbox_02.gz", "/home/dorigoa/my_sandbox_03.gz"};
requirements = RegExp("cream.*", other.GlueCEUniqueID);
RetryCount = 0;
ShallowRetryCount = 1;
MyProxyServer="";
]

dorigoa@cream-14 15:46:53 ~/JDL>glite-wms-job-submit -d pippo -e https://cream-08.pd.infn.it:7443/glite_wms_wmproxy_server wms_ISB_cream.jdl

Connecting to the service https://cream-08.pd.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/HgBwCcHn4QX-Ead14G3uVg

==========================================================================

dorigoa@cream-14 15:48:59 ~/JDL>glite-wms-job-status https://devel09.cnaf.infn.it:9000/HgBwCcHn4QX-Ead14G3uVg


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/HgBwCcHn4QX-Ead14G3uVg
Current Status:     Done(Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        cream-02.cnaf.infn.it:8443/cream-pbs-wnodes-sl6
Submitted:          Tue Apr 16 15:47:29 2013 CEST
==========================================================================


</verbatim>

LCG CE

<verbatim>
************** FROM HERE TODO TODO TODO
</verbatim>

ARC CE

Tested with Nordugrid people on korundi

</verbatim>
glite-wms-job-status https://devel09.cnaf.infn.it:9000/1C0TMnsCIC4SHG_K9wbuvg


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/1C0TMnsCIC4SHG_K9wbuvg
Current Status:     Cleared
Status Reason:      user retrieved output sandbox
Destination:        korundi.grid.helsinki.fi:2811/nordugrid-GE-mgrid
Submitted:          Wed Feb  6 15:48:26 2013 CET
==========================================================================
</verbatim>

----++++ Single job with Input Sandbox with automatic delegation

CREAM CE
<verbatim>
dorigoa@cream-14 15:33:37 ~/JDL>cat wms_OSB.jdl 
[
Executable = "/bin/echo";
Arguments = "ciao";
InputSandbox = {};
stdoutput="stdout";
stderror="stderr";
OutputSandbox = {"stdout","stderr"};
requirements = RegExp("cream.*", other.GlueCEUniqueID);
myproxyserver="cream-44.pd.infn.it";
RetryCount = 0;
ShallowRetryCount = 1;
]

dorigoa@cream-14 15:33:45 ~/JDL>glite-wms-job-submit -a -e https://cream-08.pd.infn.it:7443/glite_wms_wmproxy_server wms_OSB.jdl 

Connecting to the service https://cream-08.pd.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/xZqW-em3ZOHBTVtfv5_M_Q

==========================================================================

[...]

dorigoa@cream-14 15:33:59 ~/JDL>glite-wms-job-status https://devel09.cnaf.infn.it:9000/xZqW-em3ZOHBTVtfv5_M_Q


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/xZqW-em3ZOHBTVtfv5_M_Q
Current Status:     Done(Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        recasna-ce01.unina.it:8443/cream-pbs-recas-cert
Submitted:          Tue Apr 16 15:33:59 2013 CEST
==========================================================================

dorigoa@cream-14 15:35:17 ~/JDL>glite-wms-job-output https://devel09.cnaf.infn.it:9000/xZqW-em3ZOHBTVtfv5_M_Q

Connecting to the service https://cream-08.pd.infn.it:7443/glite_wms_wmproxy_server


================================================================================

                        JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://devel09.cnaf.infn.it:9000/xZqW-em3ZOHBTVtfv5_M_Q
have been successfully retrieved and stored in the directory:
/tmp/jobOutput/dorigoa_xZqW-em3ZOHBTVtfv5_M_Q

================================================================================

dorigoa@cream-14 15:35:27 ~/JDL>ll /tmp/jobOutput/dorigoa_xZqW-em3ZOHBTVtfv5_M_Q
total 4
-rw-rw-r-- 1 dorigoa dorigoa 0 Apr 16 15:35 stderr
-rw-rw-r-- 1 dorigoa dorigoa 5 Apr 16 15:35 stdout
dorigoa@cream-14 15:35:49 ~/JDL>cat /tmp/jobOutput/dorigoa_xZqW-em3ZOHBTVtfv5_M_Q/stdout 
ciao

</verbatim>


----++++ DAG job

<verbatim>
[mcecchi@devel15 ~]$ cat dag2.jdl 
[
	type = "dag";
	nodes = [
			nodeA = [file ="ls.jdl";];
			nodeB = [file ="ls.jdl";];
	Dependencies = {
		{nodeA, nodeB}
	}
  ];
]
</verbatim>

Submitted a DAG with one dependency. First job starts immediately.

<verbatim>
[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/a3SM8ZKG0L_IFDYtwhokGg


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/a3SM8ZKG0L_IFDYtwhokGg
Current Status:     Running
Destination:        dagman
Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/g9nVweK-mk2pTjDQjDxAPw
    Current Status:     Submitted
    Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/z6VCc1NOYD6s3ujNiy3tog
    Current Status:     Scheduled
    Status Reason:      unavailable
    Destination:        razi.biruni.upm.my:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================
</verbatim>

First job ends. 

<verbatim>
======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/a3SM8ZKG0L_IFDYtwhokGg
Current Status:     Running
Destination:        dagman
Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/g9nVweK-mk2pTjDQjDxAPw
    Current Status:     Submitted
    Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/z6VCc1NOYD6s3ujNiy3tog
    Current Status:     Done(Success)
    Logged Reason(s):
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        razi.biruni.upm.my:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================
</verbatim>
    
Wait a while (MatchRetryPeriod), before seeing the second job start. Notice that the first one was scheduled to a CREAM CE, while the second to a LCG-CE, which was a limitation of the previous design.

<verbatim>
======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/a3SM8ZKG0L_IFDYtwhokGg
Current Status:     Running
Destination:        dagman
Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/g9nVweK-mk2pTjDQjDxAPw
    Current Status:     Scheduled
    Status Reason:      unavailable
    Destination:        lcgce10.gridpp.rl.ac.uk:8443/cream-pbs-gridS
    Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/z6VCc1NOYD6s3ujNiy3tog
    Current Status:     Done(Success)
    Logged Reason(s):
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        razi.biruni.upm.my:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/a3SM8ZKG0L_IFDYtwhokGg


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/a3SM8ZKG0L_IFDYtwhokGg
Current Status:     Done(Success)
Exit code:          0
Status Reason:      DAG successfully completed
Destination:        dagman
Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/g9nVweK-mk2pTjDQjDxAPw
    Current Status:     Done(Success)
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        lcgce10.gridpp.rl.ac.uk:8443/cream-pbs-gridS
    Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/z6VCc1NOYD6s3ujNiy3tog
    Current Status:     Done(Success)
    Logged Reason(s):
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        razi.biruni.upm.my:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:10:48 2013 CET
==========================================================================
</verbatim>

----++++ Parametric

Beware of bug #79027. Work with jdl-api-cpp-3.4.1-3

<verbatim>
$ cat parametric.jdl 
[
ParameterStart = 0;
ParameterStep = 1;
Parameters = 3;
JobType = "parametric";
Executable = "/bin/echo";
Arguments = "_PARAM_";
Environment = {"MYPATH_PARAM_=$PATH:/bin:/usr/bin:$HOME"};
StdOutput = "echo_PARAM_.out";
StdError = "echo_PARAM_.err";
OutputSandbox = {"echo_PARAM_.out","echo_PARAM_.err"};
Retrycount=-1;
] 

$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server parametric.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/kf7upvOiNtOd7qAthY70lg

==========================================================================

$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/kf7upvOiNtOd7qAthY70lg


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/kf7upvOiNtOd7qAthY70lg
Current Status:     Done(Success)
Exit code:          0
Submitted:          Tue Feb  5 15:27:05 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/9eJkws7-aS5qcUKyZswydg
    Current Status:     Done(Success)
    Logged Reason(s):
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        cex.grid.unipg.it:8443/cream-pbs-cert
    Submitted:          Tue Feb  5 15:27:05 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/DQULXLoEiZ81BHomXV9nfw
    Current Status:     Done(Success)
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        grisuce.scope.unina.it:8443/cream-pbs-egeecert
    Submitted:          Tue Feb  5 15:27:05 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/LozDHI8Df9uz5GbAMesn_w
    Current Status:     Done(Success)
    Logged Reason(s):
        - BLAH error: submission command failed (exit code = 1) (stdout:) (stderr:T3_BO: User cannot use the queue. Job not submitted.-) N/A (jobId = CREAM450095355)
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        atlas-ce-01.roma1.infn.it:8443/cream-lsf-atlasgcert
    Submitted:          Tue Feb  5 15:27:05 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/varcOAEGkQ0QeIScEVd47w
    Current Status:     Done(Success)
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        gridce1.pi.infn.it:8443/cream-lsf-cert
    Submitted:          Tue Feb  5 15:27:05 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/ykG2-jocfC4vxt3jqPvQAA
    Current Status:     Done(Success)
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        grid012.ct.infn.it:8443/cream-lsf-cert
    Submitted:          Tue Feb  5 15:27:05 2013 CET
==========================================================================

</verbatim>


----++++ Collection

<verbatim>
dorigoa@cream-51 13:18:59 ~/JDLs/WMS>cat coll_small.jdl
  [
    type = "collection";
    myproxyserver="myproxy.cnaf.infn.it";
    nodes = {
      [
      file ="/home/dorigoa/JDLs/WMS/coll/job.jdl" ; 
      ],
      [
      file ="/home/dorigoa/JDLs/WMS/coll/job.jdl" ; 
      ]
    };
  ]

dorigoa@cream-51 13:19:06 ~/JDLs/WMS>cat coll/job.jdl
[
Executable = "/bin/ls" ;
Arguments = "/tmp" ;
RetryCount = 2 ;
Stdoutput = "std.out" ;
StdError =  "std.err" ;
OutputSandbox = { "std.out" ,"std.err"} ;
InputSandbox = { "data/pippo" };
rank = 1 ;
ShallowRetryCount = 2;
usertags = [ exe = "ls" ];
requirements = RegExp("cream.*", other.GlueCEUniqueID);
]


dorigoa@cream-51 13:19:08 ~/JDLs/WMS>\rm GIDS_coll ; glite-wms-job-submit -o GIDS_coll -d foo -e https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server coll_small.jdl

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/XImEEyQ7T7AX5x_KsquTJQ

The job identifier has been saved in the following file:
/home/dorigoa/JDLs/WMS/GIDS_coll

==========================================================================


dorigoa@cream-51 13:19:23 ~/JDLs/WMS>glite-wms-job-submit -o GIDS_coll -d foo -e https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server coll_small.jdl

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/rCZVB6E8FwUhON8TpfNf5w

The job identifier has been saved in the following file:
/home/dorigoa/JDLs/WMS/GIDS_coll

==========================================================================


dorigoa@cream-51 13:19:27 ~/JDLs/WMS>glite-wms-job-submit -o GIDS_coll -d foo -e https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server coll_small.jdl

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/EQkZUCGiH36RVpyORQFP6g

The job identifier has been saved in the following file:
/home/dorigoa/JDLs/WMS/GIDS_coll

==========================================================================

dorigoa@cream-51 13:19:29 ~/JDLs/WMS>glite-wms-job-status -i GIDS_coll

------------------------------------------------------------------
1 : https://devel09.cnaf.infn.it:9000/XImEEyQ7T7AX5x_KsquTJQ
2 : https://devel09.cnaf.infn.it:9000/rCZVB6E8FwUhON8TpfNf5w
3 : https://devel09.cnaf.infn.it:9000/EQkZUCGiH36RVpyORQFP6g
a : all
q : quit
------------------------------------------------------------------

Choose one or more jobId(s) in the list - [1-3]all:a



======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/XImEEyQ7T7AX5x_KsquTJQ
Current Status:     Waiting
Submitted:          Tue Feb  5 13:19:23 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/7pKKVeOJHzGB2Tu_WmHAeg
    Current Status:     Done(Success)
    Logged Reason(s):
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        atlas-creamce-02.roma1.infn.it:8443/cream-lsf-atlasgcert
    Submitted:          Tue Feb  5 13:19:23 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/GQ9TdSY-I9u9MuYIyzBpDg
    Current Status:     Scheduled
    Status Reason:      unavailable
    Destination:        cccreamceli08.in2p3.fr:8443/cream-sge-long
    Submitted:          Tue Feb  5 13:19:23 2013 CET
==========================================================================
    

======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/rCZVB6E8FwUhON8TpfNf5w
Current Status:     Waiting
Submitted:          Tue Feb  5 13:19:26 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/7Lc7o9--VsdQ97RgAnMWWQ
    Current Status:     Scheduled
    Status Reason:      unavailable
    Destination:        razi.biruni.upm.my:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:19:26 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/Q0GFvjB1euAdjyAh6gJApg
    Current Status:     Scheduled
    Status Reason:      unavailable
    Destination:        ce.cta.camk.edu.pl:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:19:26 2013 CET
==========================================================================

    

======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/EQkZUCGiH36RVpyORQFP6g
Current Status:     Waiting
Submitted:          Tue Feb  5 13:19:28 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/RCSl4p9ZaCf10wwhD3SYCQ
    Current Status:     Scheduled
    Status Reason:      unavailable
    Destination:        cccreamceli09.in2p3.fr:8443/cream-sge-short
    Submitted:          Tue Feb  5 13:19:28 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/jjRKDlVR7rmO0wXsfTR7IQ
    Current Status:     Scheduled
    Status Reason:      unavailable
    Destination:        bugaboo-hep.westgrid.ca:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:19:28 2013 CET
==========================================================================

dorigoa@cream-51 13:26:06 ~/JDLs/WMS>glite-wms-job-status -i GIDS_coll

------------------------------------------------------------------
1 : https://devel09.cnaf.infn.it:9000/XImEEyQ7T7AX5x_KsquTJQ
2 : https://devel09.cnaf.infn.it:9000/rCZVB6E8FwUhON8TpfNf5w
3 : https://devel09.cnaf.infn.it:9000/EQkZUCGiH36RVpyORQFP6g
a : all
q : quit
------------------------------------------------------------------

Choose one or more jobId(s) in the list - [1-3]all:a



======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/XImEEyQ7T7AX5x_KsquTJQ
Current Status:     Done(Success)
Exit code:          0
Submitted:          Tue Feb  5 13:19:23 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/7pKKVeOJHzGB2Tu_WmHAeg
    Current Status:     Done(Success)
    Logged Reason(s):
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        atlas-creamce-02.roma1.infn.it:8443/cream-lsf-atlasgcert
    Submitted:          Tue Feb  5 13:19:23 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/GQ9TdSY-I9u9MuYIyzBpDg
    Current Status:     Done(Success)
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        cccreamceli08.in2p3.fr:8443/cream-sge-long
    Submitted:          Tue Feb  5 13:19:23 2013 CET
==========================================================================
    

======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/rCZVB6E8FwUhON8TpfNf5w
Current Status:     Done(Success)
Exit code:          0
Submitted:          Tue Feb  5 13:19:26 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/7Lc7o9--VsdQ97RgAnMWWQ
    Current Status:     Done(Success)
    Logged Reason(s):
        - Transfer to CREAM failed due to exception: Failed to create a delegation id for job https://devel09.cnaf.infn.it:9000/7Lc7o9--VsdQ97RgAnMWWQ: reason is Authorization failure: Authorization error: Failed to get the local user id via glexec: glexec error: [gLExec]:  LCAS failed.           The reason can be found in the syslog.
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        razi.biruni.upm.my:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:19:26 2013 CET
==========================================================================
    Status info for the Job : https://devel09.cnaf.infn.it:9000/Q0GFvjB1euAdjyAh6gJApg
    Current Status:     Done(Success)
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        ce.cta.camk.edu.pl:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:19:26 2013 CET
==========================================================================
    

======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/EQkZUCGiH36RVpyORQFP6g
Current Status:     Done(Success)
Exit code:          0
Submitted:          Tue Feb  5 13:19:28 2013 CET
==========================================================================

- Nodes information for: 
    Status info for the Job : https://devel09.cnaf.infn.it:9000/RCSl4p9ZaCf10wwhD3SYCQ
    Current Status:     Done(Success)
    Logged Reason(s):
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        cccreamceli09.in2p3.fr:8443/cream-sge-short
    Submitted:          Tue Feb  5 13:19:28 2013 CET
==========================================================================
    
    Status info for the Job : https://devel09.cnaf.infn.it:9000/jjRKDlVR7rmO0wXsfTR7IQ
    Current Status:     Done(Success)
    Logged Reason(s):
        - job completed
        - Job Terminated Successfully
    Exit code:          0
    Status Reason:      Job Terminated Successfully
    Destination:        bugaboo-hep.westgrid.ca:8443/cream-pbs-dteam
    Submitted:          Tue Feb  5 13:19:28 2013 CET
==========================================================================
</verbatim>


---+++++ Job list-match

<verbatim>
[mcecchi@devel15 ~]$ glite-wms-job-list-match --rank -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server ls.jdl |tail
 - vgn003.hep.physik.uni-siegen.de:8443/cream-pbs-dteam             1
 - vm3.tier2.hep.manchester.ac.uk:8443/cream-pbs-long               1
 - vserv11.hep.phy.cam.ac.uk:8443/cream-pbs-express                 1
 - vserv11.hep.phy.cam.ac.uk:8443/cream-pbs-short                   1
 - wario.univ-lille1.fr:8443/cream-pbs-dteam                        1
 - wipp-crm.weizmann.ac.il:8443/cream-pbs-dteam                     1

==========================================================================
</verbatim>

---+++++ ISM purchasers

<verbatim>
[root@devel09 workload_manager]# cat dump 
[command="ism_dump"]
[root@devel09 workload_manager]# cp dump jobdir/new/
[root@devel09 workload_manager]# tail -f /var/log/wms/workload_manager_events.log
              ": OiBSMngxWlZOcGRHVlZibWx4ZFdWSlJEMUpURjlKVlVORFgwbEhJQT09",
              "GlueCEUniqueID=ngi-il-ce2.isragrid.org.il:8443/cream-sge-israelvo",
              "GlueCEUniqueID=ngi-il-ce2.isragrid.org.il:8443/cream-sge-dteam",
              "GlueCEUniqueID=ngi-il-ce2.isragrid.org.il:8443/cream-sge-ops"
           }; 
        GlueClusterName = "ngi-il-ce2.isragrid.org.il"; 
        GlueSchemaVersionMajor = 1; 
        GlueInformationServiceURL = "ldap://ngi-il-ce2.isragrid.org.il:2170/mds-vo-name=resource,o=grid"
    ]
05 Feb, 10:43:47 -I: [Info] operator()(../../src/dispatcher_utils.cpp:292): new ism_dump

[root@devel09 workload_manager]# ll
total 112244
-rw-r--r-- 1 root  root         21 Feb  5 10:43 dump
-rw-r--r-- 1 glite glite 114810221 Feb  5 10:43 ismdump.fl
drwxr-xr-x 5 glite glite      4096 Jan 30 13:10 jobdir
</verbatim>

---+++++ Glue 2.0 purchasers and match-making:

N.B.: it requires glite-wms-core-3.5.0-7

----+++++ Match-making.

makes sure that you don't inherit glue 1 defaults from your UI. Set WmsRequirements = true (or whatever valid glue2 expression) in the wms configuration.

<verbatim>
[mcecchi@devel15 ~]$ cat ls_g2.jdl 
[
Executable = "/bin/echo";
EnableZIppedISB=true;
Arguments = "Hello";
StdOutput = "out.log";
StdError = "err.log";
InputSandbox = {};
OutputSandbox = {"out.log", "err.log"};
requirements = other.GLUE2.Computing.Endpoint.QualityLevel == "production";
AllowZippedISB = true;
rank=1;
myproxyserver="";
RetryCount = -1;
ShallowRetryCount = -1;
]
[mcecchi@devel15 ~]$ glite-wms-job-list-match --debug -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server  ls_g2.jdl |wc -l

4238
</verbatim>

----+++++ Purchasing.

In the WM log:

<verbatim>
05 Feb, 17:11:45 -D: [Debug] fetch_bdii_ce_info_g2(../../../../src/ism/purchaser/ldap-utils-g2.cpp:1167): #98037 LDAP entries received in 169 seconds
06 Feb, 08:58:44 -D: [Debug] fetch_bdii_se_info_g2(../../../../src/ism/purchaser/ldap-utils-g2.cpp:1166): #18871 GLUE2StorageS
hare's ClassAd(s) generated in 5 seconds

</verbatim>

Create a dump request and check that the content contains GLUE2 descriptions


---+++++ Job cancel

Cancel of a Running job:

<verbatim>
======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ
Current Status:     Scheduled
Status Reason:      unavailable
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Mon Feb  4 16:00:08 2013 CET
==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-cancel https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ

Are you sure you want to remove specified job(s) [y/n]y : y

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


============================= glite-wms-job-cancel Success =============================

The cancellation request has been successfully submitted for the following job(s):

- https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ

========================================================================================


[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/HXnAN9vrdSsSi33YxEZmAQ
Current Status:     Cancelled
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Mon Feb  4 16:00:08 2013 CET
==========================================================================

</verbatim>

Cancel of a Done job:

<verbatim>
[mcecchi@devel15 ~]$ glite-wms-job-cancel https://devel09.cnaf.infn.it:9000/XbwZJuMcbwQ68XVvHm6ztQ

Are you sure you want to remove specified job(s) [y/n]y : y

Error - Cancel not allowed
Current Job Status is Done
</verbatim>

---+++++ Certificate delegation

<verbatim>
[mcecchi@devel15 ~]$ glite-wms-job-delegate-proxy --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server -d mc

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


================== glite-wms-job-delegate-proxy Success ==================

Your proxy has been successfully delegated to the WMProxy(s):
https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server
with the delegation identifier: mc

==========================================================================

</verbatim>


---+++++ Check publication in EMIR

Put in your siteinfo.def something like:

<verbatim>
BDII_EMIR_ENABLE=yes
BDII_EMIR_SERP_PERIOD=1
BDII_EMIR_SERP_VALIDITY=2
BDII_EMIR_HOST=http://emitbdsr1.cern.ch:9126
</verbatim>

Then run:

<verbatim>
[root@devel09 ~]#PATH=$PATH:/sbin /opt/glite/yaim/bin/yaim -r -f config_emir_serp -s siteinfo_/site-info.def

   WARNING: 
   WARNING: *****************************************************************************
   WARNING: Your siteinfo directory is world readable. This is generally a bad idea      
   WARNING: as configuration files may contain passwords and other sensitive information.
   WARNING: *****************************************************************************
   WARNING: 
   INFO: Configuring HOST: devel09.cnaf.infn.it
   INFO: Using site configuration file: siteinfo_/site-info.def
   INFO: 
         ###################################################################
         
         .             /'.-. ')
         .     yA,-"-,( ,m,:/ )   .oo.     oo    o      ooo  o.     .oo
         .    /      .-Y a  a Y-.     8. .8'    8'8.     8    8b   d'8
         .   /           ~ ~ /         8'    .8oo88.     8    8  8'  8
         . (_/         '===='          8    .8'     8.   8    8  Y   8
         .   Y,-''-,Yy,-.,/           o8o  o8o    o88o  o8o  o8o    o8o
         .    I_))_) I_))_)
         
         
         current working directory: /root
         site-info.def date: Jan 30 15:39 siteinfo_/site-info.def
         yaim command: -r -f config_emir_serp -s siteinfo_/site-info.def
         log file: /opt/glite/yaim/bin/../log/yaimlog
         Wed Jan 30 15:50:43 CET 2013 : /opt/glite/yaim/bin/yaim
         
         Installed YAIM versions:
         glite-lb-yaim 4.5.8-1
         glite-yaim-bdii 4.3.13-1
         glite-yaim-core 5.1.0-1
         
         ####################################################################
   INFO: The default location of the grid-env.(c)sh files will be: /usr/libexec
   INFO: Sourcing the utilities in /opt/glite/yaim/functions/utils
   INFO: Detecting environment
   INFO: Assuming the node types:  WMSLB
   INFO: Using hostname: devel09.cnaf.infn.it
   WARNING: config_emir_serp function is not supported by  WMSLB!
   WARNING: Anyway you can force this operation.
   WARNING: Would you really run config_emir_serp on  WMSLB? [y/]: 
   INFO: Executing function: config_emir_serp_check 
   INFO: Executing function: config_emir_serp 
EMIR-SERP startup                                          [  OK  ]
   INFO: YAIM terminated succesfully.
</verbatim>

Check that the record has reached its destination

<verbatim>
mcecchi@pccecchi ~ $ curl -v -H "Accept:application/xml" -X GET http://emitbdsr1.cern.ch:9126/services?Service_Endpoint_ID=devel09.cnaf.infn.it_wmproxy_2272271975_org.glite.wms.WMProxy_2272271975
* About to connect() to emitbdsr1.cern.ch port 9126 (#0)
*   Trying 128.142.136.66... connected
* Connected to emitbdsr1.cern.ch (128.142.136.66) port 9126 (#0)
> GET /services?Service_Endpoint_ID=devel09.cnaf.infn.it_wmproxy_2272271975_org.glite.wms.WMProxy_2272271975 HTTP/1.1
> User-Agent: curl/7.21.0 (i386-redhat-linux-gnu) libcurl/7.21.0 NSS/3.12.10.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.4
> Host: emitbdsr1.cern.ch:9126
> Accept:application/xml
> 
< HTTP/1.1 200 OK
< Content-Type: application/xml
< Vary: Accept-Encoding
< Content-Length: 981
< Server: Jetty(8.1.4.v20120524)
< 
* Connection #0 to host emitbdsr1.cern.ch left intact
* Closing connection #0
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><ns2:QueryResult xmlns="http://schemas.ogf.org/glue/2009/03/spec_2.0_r1" xmlns:ns2="http://www.eu-emi.eu/emiregistry" count="1"><Service BaseType="Entity" CreationTime="2013-01-30T15:22:41.000+01:00"><ID>devel09.cnaf.infn.it_wmproxy_2272271975</ID><Name>INFN-CNAF-WMProxy</Name><Extensions/><Type>org.glite.wms.WMProxy</Type><Location/><Endpoint CreationTime="2013-01-30T15:22:41.000+01:00"><ID>devel09.cnaf.infn.it_wmproxy_2272271975_org.glite.wms.WMProxy_2272271975</ID><URL>https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server</URL><Capability>executionmanagement.candidatesetgenerator</Capability><Capability>executionmanagement.jobdescription</Capability><Capability>executionmanagement.jobmanager</Capability><Technology>webservice</Technology><InterfaceName>org.glite.wms.WMProxy</InterfaceName><InterfaceVersion>1.0</InterfaceVersion><QualityLevel>production</QualityLevel></Endpoint></Service></ns2:QueryResult>mcecchi@pccecchi ~ $ 
</verbatim>


---+++++  Argus based authZ

see bug #99092 test report, below.


---++++++  NOTA BENE: access control with Argus is very slow, to the point that it sometimes causes a DOS. *NOT* suggested for use in production. In these circumstances, the error is the one reported below.

<verbatim>
[mcecchi@devel15 ~]$ glite-wms-job-list-match --rank -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server ls.jdl | grep lcg

Warning - Unable to perform the operation: Argus request on jobListMatch by DN: /C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Marco Cecchi failed for some reason
Error code: SOAP-ENV:Server
</verbatim>

---+++++  Support for RFC proxies

IMPORTANT: use glite-wms-interface>=3.5.0-6

<verbatim>
[mcecchi@devel15 ~]$ voms-proxy-init --voms testers.eu-emi.eu -rfcEnter GRID pass phrase:
Your identity: /C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Marco Cecchi
Creating temporary proxy .................................... Done
Contacting  emitestbed07.cnaf.infn.it:15002 [/C=IT/O=INFN/OU=Host/L=CNAF/CN=emitestbed07.cnaf.infn.it] "testers.eu-emi.eu" Done
Creating proxy ................................................................................ Done

Your proxy is valid until Sat Feb 16 04:26:01 2013

[mcecchi@devel15 ~]$ glite-wms-job-submit --register-only -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server cream.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully registered to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/JiEPmMuhp3M6AKxAD2vcWA

==========================================================================

To complete the operation, the following file containing the InputSandbox of the job needs to be transferred:
==========================================================================================================
ISB ZIP file : /tmp/ISBfiles_pt_Crq6aJiRrOa2614TmXg_0.tar.gz
Destination : gsiftp://devel09.cnaf.infn.it:2811/var/SandboxDir/Ji/https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fJiEPmMuhp3M6AKxAD2vcWA/input/ISBfiles_pt_Crq6aJiRrOa2614TmXg_0.tar.gz
-----------------------------------------------------------------------------

then start the job by issuing a submission with the option:
 --start https://devel09.cnaf.infn.it:9000/JiEPmMuhp3M6AKxAD2vcWA



[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server cream.jdl 
Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/FHNf0FyCVCm4_2zWv6RnyQ

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-cancel https://devel09.cnaf.infn.it:9000/FHNf0FyCVCm4_2zWv6RnyQ

Are you sure you want to remove specified job(s) [y/n]y : y

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


============================= glite-wms-job-cancel Success =============================

The cancellation request has been successfully submitted for the following job(s):

- https://devel09.cnaf.infn.it:9000/FHNf0FyCVCm4_2zWv6RnyQ

========================================================================================


[mcecchi@devel15 ~]$ glite-wms-job-list-match -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server cream.jdl |wc -l
14
</verbatim>


---++ LIST OF BUGS

| WMS stops accepting submissions when no swap is available  (bug #96136) |  %ICON{choice-yes}% |
| Set asyncjobstart=false to avoid lost jobs (bug #97319) |  %ICON{choice-yes}% |
| Job perusal is broken in EMI-2 WMS (bug #98279) |  %ICON{choice-yes}% |
| various issues with EMI-1/2 proxy cleanup (bug #98282) |  %ICON{choice-yes}% |
| classad plugin functions are broken (bug #100176) |  %ICON{choice-yes}% |
| glite-wms-job-output requires a valid delegated proxy on the WMS to enable output retrieval (bug #98524) |  %ICON{choice-yes}% |
| ICE removes its pid file at the start() instead of stop() (bug #98724) |  %ICON{choice-yes}% |
| wmproxy init script 'status' does not return error when service is not running (bug #99017) |  %ICON{choice-yes}% |
| WMS: fix various bugs on Argus authZ (bug #99092) |  %ICON{choice-yes}% |
| WM memory footprint bloats, especially on 64-bit architectures (bug #97288)  |  %ICON{choice-yes}% |

---++ BUG VERIFICATION

---+++++ BUG: WMS: stops accepting submissions when no swap is available  (bug #96136). Certified by MC 31/12/2013
<verbatim>
[root@devel09 ~]# /usr/sbin/glite_wms_wmproxy_load_monitor --oper jobSubmit|grep -i swap
Threshold for Swap Usage: 95 => Detected value for Swap Usage: 0.60%
[root@devel09 ~]# cat /proc/swaps 
Filename				Type		Size	Used	Priority
/dev/mapper/VolGroup00-LogVol01         partition	6094840	36808	-1
[root@devel09 ~]# swapoff /dev/mapper/VolGroup00-LogVol01
[root@devel09 ~]# cat /proc/swaps 
[root@devel09 ~]# /usr/sbin/glite_wms_wmproxy_load_monitor --oper jobSubmit|grep -i swap
Threshold for Swap Usage: 95 => Detected value for Swap Usage: 0.00%
</verbatim>

---+++++ BUG: Set asyncjobstart=false to avoid lost jobs (bug #97319). Certified by MC 30/12/2013

<verbatim>
[root@devel09 ~]# grep -i asyncj /etc/glite-wms/glite_wms.conf
    AsyncJobStart  =  false;
</verbatim>


---+++++ BUG: classad plugin functions are broken (bug #100176). Certified by MC 30/12/2013

On a UI, list-match for this JDL:

<verbatim>
[mcecchi@devel15 ~]$ cat fqanmember_emi2.jdl 
[
Executable = "/bin/echo";
Arguments = "Hello";
StdOutput = "out.log";
StdError = "err.log";
InputSandbox = {};
OutputSandbox = {"out.log", "err.log"};
leftVOMS_FQAN="/gridit/ansys/Role=NULL/Capability=NULL";
rightGlueCEACBR= { "VOMS:/gridit/ansys", "VOMS:/gridit/ansys/Role=SoftwareManager"};
requirements = fqanMember(strcat("VOMS:", leftVOMS_FQAN), rightGlueCEACBR) is true;
myproxyserver="";
ShallowRetryCount = -1;
]
</verbatim>

if the number of returned queues if more than 0 then the bug is fixed (actually it should be high, because if the classad plugin function works requirements evaluates to true)

<verbatim>
[mcecchi@devel15 ~]$ glite-wms-job-list-match --rank -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server fqanmember_emi2.jdl |wc -l
635
</verbatim>

---+++++BUG: ICE removes its pid file at the start() instead of stop() (bug #98724). Certified by AD (31/01/2013, 11:30)

In the following log paste it is shown that after ICE stop the pid file =/var/run/glite-wms-ice-safe.pid= is correctly removed
<verbatim>
[root@cream-01 siteinfo]# ps -ef|grep ice
root     23205  1918  0 11:31 pts/0    00:00:00 grep ice
[root@cream-01 siteinfo]# /etc/init.d/glite-wms-ice start
starting ICE... ok
[root@cream-01 siteinfo]# ll /var/run/glite-wms-ice-safe.pid
-rw-r--r-- 1 root root 6 Jan 31 11:31 /var/run/glite-wms-ice-safe.pid
[root@cream-01 siteinfo]# ps -ef|grep ice
glite    23232     1  0 11:31 ?        00:00:00 /usr/bin/glite-wms-ice-safe --conf glite_wms.conf --daemon /var/run/glite-wms-ice-safe.pid
glite    23237 23232  0 11:31 ?        00:00:00 sh -c /usr/bin/glite-wms-ice --conf glite_wms.conf /var/log/wms/ice_console.log 2>&1
glite    23238 23237  0 11:31 ?        00:00:00 /usr/bin/glite-wms-ice --conf glite_wms.conf /var/log/wms/ice_console.log
root     23275  1918  0 11:31 pts/0    00:00:00 grep ice
[root@cream-01 siteinfo]# /etc/init.d/glite-wms-ice stop
stopping ICE... ok
[root@cream-01 siteinfo]# ps -ef|grep ice
root     23309  1918  0 11:31 pts/0    00:00:00 grep ice
[root@cream-01 siteinfo]# ll /var/run/glite-wms-ice-safe.pid
ls: /var/run/glite-wms-ice-safe.pid: No such file or directory

</verbatim>

---+++++ BUG: various issues with EMI-1/2 proxy cleanup (bug #98282). Certified by MC 31/12/2013

Just perform these checks:

<verbatim>
[root@devel09 ~]# ll /etc/cron.hourly
total 4
-rw-r--r-- 1 root root 351 Nov 22 12:26 purge_proxycache
[root@devel09 ~]# cat /etc/cron.hourly/purge_proxycache 
#!/bin/sh

exec >> /var/log/proxycache-cleanup.log 2>&1 < /dev/null

echo "===== `date` ====="

cd /var/proxycache/cache || exit

for i in *
do
(
test -d "$i" && cd "$i" || exit

echo "=== $i"

find . -maxdepth 1 -mindepth 1 -type d -mmin +5 -ls -exec \
sh -c '
k=$0/userkey.pem
[ -e "$k" ] && ls -l "$k" && rm "$k"
rmdir "$0"
' {} \;
)
done

echo ''

[root@devel09 ~]# cat /etc/cron.d/glite-wms-wmproxy-purge-proxycache.cron
HOME=/
MAILTO=wms-support@cnaf.infn.it

0 */2 * * * root . /usr/libexec/grid-env.sh ; /usr/bin/glite-wms-wmproxy-purge-proxycache /var/proxycache >> /var/log/wms/glite-wms-wmproxy-purge-proxycache.log 2>&1
[root@devel09 ~]# cat /etc/cron.d/glite-wms-wmproxy-purge-proxycache_keys.cron 
0 */2 * * * root sh /usr/bin/glite-wms-wmproxy-purge-proxycache_keys >> /var/log/wms/glite-wms-wmproxy-purge-proxycache-keys.log 2>&1 < /dev/null
[root@devel09 ~]#

[root@devel09 ~]# cat /usr/bin/glite-wms-wmproxy-purge-proxycache_keys
#!/bin/sh

log() {
  echo "`date`: $1"
}

log "proxy cache leftover private keys cleaner started"

cd /var/proxycache/cache || exit

for i in *
do
(
test -d "$i" && cd "$i" || exit

echo "=== $i"

find . -maxdepth 1 -mindepth 1 -type d -mmin +5 -ls -exec \
sh -c '
k=$0/userkey.pem
[ -e "$k" ] && ls -l "$k" && rm "$k"
rmdir "$0"
' {} \;
)
done

log "finished cleaning keys"
[root@devel09 ~]# sh /usr/bin/glite-wms-wmproxy-purge-proxycache_keys
Thu Jan 31 16:17:13 CET 2013: proxy cache leftover private keys cleaner started
Thu Jan 31 16:17:13 CET 2013: finished cleaning keys
[root@devel09 ~]# ll /var/proxycache/cache/
total 0
[root@devel09 ~]# 

</verbatim>

---+++++ BUG: wmproxy init script 'status' does not return error when service is not running (bug #99017). Certified by MC 31/12/2013

<verbatim>
[root@devel09 ~]# /etc/init.d/glite-wms-wmproxy stop
Stopping /usr/bin/glite_wms_wmproxy_server... ok
[root@devel09 ~]# /etc/init.d/glite-wms-wmproxy status
/usr/bin/glite_wms_wmproxy_server is not running
[root@devel09 ~]# echo $? 
1
</verbatim>

---+++++ BUG: glite-wms-job-output requires a valid delegated proxy on the WMS to enable output retrieval (bug #98524)

<verbatim>
[mcecchi@devel15 ~]$ cat no_renewal.jdl 
[
Executable = "/bin/echo";
EnableZIppedISB=true;
Arguments = "Hello";
StdOutput = "out.log";
StdError = "err.log";
InputSandbox = {};
OutputSandbox = {"out.log", "err.log"};
requirements = true;
rank = 1;
AllowZippedISB = false;
myproxyserver="";
ShallowRetryCount = -1;
]

[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server no_renewal.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


Warning - Unable to submit the job to the service: https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server
Argus denied authorization on jobSubmit issued by DN: /C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Marco Cecchi
Error code: SOAP-ENV:Server


Error - Operation failed
Unable to find any endpoint where to perform service request

[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server no_renewal.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ
Current Status:     Waiting
Status Reason:      unavailable
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Thu Jan 31 12:50:34 2013 CET
==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/LR_KXTPtFJKWy2-33sWdqQ
Current Status:     Ready
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Thu Jan 31 12:50:34 2013 CET
==========================================================================

</verbatim>

---+++++ BUG: WMS: fix various bugs on Argus authZ (bug #99092). Certified by MC 31/12/2013

WMS:

<verbatim>
[root@devel09 ~]# rpm -qa | grep glite-wms
glite-wms-utils-classad-3.4.1-1.sl5
glite-wms-purger-3.5.0-3.sl5
glite-wms-jobsubmission-3.5.0-3.sl5
glite-wms-core-devel-3.5.0-4.sl5
glite-wms-common-3.5.0-3.sl5
glite-wms-configuration-3.5.0-3.sl5
glite-wms-interface-3.5.0-3.sl5
glite-wms-ice-3.5.0-3.sl5
glite-wms-core-3.5.0-5.sl5
glite-wms-jobsubmission-lib-3.5.0-3.sl5
glite-wms-utils-exception-3.4.1-1.sl5
</verbatim>

Check that siteinfo.def contains:
<verbatim>
USE_ARGUS=yes
ARGUS_PEPD_ENDPOINTS="https://emitestbed30.cnaf.infn.it:8154/authz"
</verbatim>

and run the configuration, if not already done. Submit a job a look at the wmp log:

WMS:
<verbatim>
[root@devel09 SandboxDir]# tail -f /var/log/wms/wmproxy.log|grep -i argus
04 Feb, 15:44:29 -D- PID: 16297 - "WMPAuthorizer::authorize": Argus authZ and mapping
04 Feb, 15:44:29 -D- PID: 16297 - "argus_authZ": PEP version: argus-pep-api-c/2.2.0 (libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5)
04 Feb, 15:44:29 -I- PID: 16297 - "argus_authZ": creating XACML request for argus
04 Feb, 15:45:04 -D- PID: 16297 - "argus_autZ::get_response": argus decision permit
04 Feb, 15:45:04 -D- PID: 16297 - "argus_autZ::get_response": argus mapped user = tst27
04 Feb, 15:45:04 -D- PID: 16297 - "argus_autZ::get_response": argus mapped group = testers
04 Feb, 15:45:04 -I- PID: 16297 - "WMPAuthorizer::authorize": Argus returned PERMIT with mapping uid: 61027, gid: 6100
04 Feb, 15:45:04 -D- PID: 16297 - "WMPAuthorizer::authorize": Argus authZ and mapping
04 Feb, 15:45:04 -D- PID: 16297 - "argus_authZ": PEP version: argus-pep-api-c/2.2.0 (libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5)
04 Feb, 15:45:04 -I- PID: 16297 - "argus_authZ": creating XACML request for argus
04 Feb, 15:45:05 -D- PID: 16297 - "argus_autZ::get_response": argus decision permit
04 Feb, 15:45:05 -D- PID: 16297 - "argus_autZ::get_response": argus mapped user = tst27
04 Feb, 15:45:05 -D- PID: 16297 - "argus_autZ::get_response": argus mapped group = testers
04 Feb, 15:45:05 -I- PID: 16297 - "WMPAuthorizer::authorize": Argus returned PERMIT with mapping uid: 61027, gid: 6100

[root@devel09 SandboxDir]# ll aK/https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2faKh4KHvJF-Dzf69BDWCM7Q/
total 28
drwxrwx--- 2 tst27 glite 4096 Feb  4 15:45 input
-rw-r--r-- 1 glite glite  599 Feb  4 15:45 JDLOriginal
-rw-r--r-- 1 glite glite 2395 Feb  4 15:45 JDLStarted
drwxrwx--- 2 tst27 glite 4096 Feb  4 15:45 output
drwxrwx--- 2 tst27 glite 4096 Feb  4 15:45 peek
-rw------- 1 glite glite 6361 Feb  4 15:45 user.proxy

</verbatim>

UI: submit a job with both ISB and OSB

<verbatim>
[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server no_renewal.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/aKh4KHvJF-Dzf69BDWCM7Q

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/aKh4KHvJF-Dzf69BDWCM7Q


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/aKh4KHvJF-Dzf69BDWCM7Q
Current Status:     Done(Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Mon Feb  4 15:45:05 2013 CET
==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-output https://devel09.cnaf.infn.it:9000/XbwZJuMcbwQ68XVvHm6ztQ

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


Warning - JobPurging not allowed
 (Unable to copy file)


================================================================================

			JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://devel09.cnaf.infn.it:9000/XbwZJuMcbwQ68XVvHm6ztQ
have been successfully retrieved and stored in the directory:
/tmp/jobOutput/mcecchi_XbwZJuMcbwQ68XVvHm6ztQ

================================================================================


[mcecchi@devel15 ~]$ ll /tmp/jobOutput/mcecchi_XbwZJuMcbwQ68XVvHm6ztQ/out.log 
-rw-rw-r-- 1 mcecchi mcecchi 22 Feb  4 15:57 /tmp/jobOutput/mcecchi_XbwZJuMcbwQ68XVvHm6ztQ/out.log
[mcecchi@devel15 ~]$ cat /tmp/jobOutput/mcecchi_XbwZJuMcbwQ68XVvHm6ztQ/out.log 
a.out
err.log
out.log
</verbatim>

---+++++ BUG: WMS: Job perusal is broken in EMI-2 WMS (bug #98279). Certified by MC 02/02/2013

<verbatim>
[mcecchi@devel15 ~]$ cat perusal.jdl 
Executable = "sleeper.sh";
requirements = true;
rank = 1;
EnableZippedIsb=true;
Arguments = "out.txt";
StdOutput = "std.out";
StdError = "std.err";
InputSandbox = {"sleeper.sh"};
OutputSandbox = {"std.out","std.err","out.txt"};
PerusalFileEnable = true;
[mcecchi@devel15 ~]$ glite-wms-job-submit -a --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server perusal.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-status https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ
Current Status:     Running
Status Reason:      unavailable
Destination:        emi002.grid.sinica.edu.tw:8443/cream-pbs-testers_q
Submitted:          Mon Feb  4 16:17:20 2013 CET
==========================================================================

[mcecchi@devel15 ~]$ glite-wms-job-perusal --set -f std.out https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-perusal Success ======================

File perusal has been successfully enabled for the job:
https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-perusal --get -f std.out https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-perusal Success ======================

No files to be retrieved for the job:
https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

==========================================================================


[mcecchi@devel15 ~]$ glite-wms-job-perusal --get -f std.out https://devel09.cnaf.infn.it:9000/znM6WQNJQqVhMmAPSp4gFQ

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-perusal Success ======================

The retrieved files have been successfully stored in:
/tmp/jobOutput/mcecchi_znM6WQNJQqVhMmAPSp4gFQ

==========================================================================


--------------------------------------------------------------------------
file 1/1: std.out-20130204153621_1-20130204153621_1
--------------------------------------------------------------------------

[mcecchi@devel15 ~]$ tail /tmp/jobOutput/mcecchi_znM6WQNJQqVhMmAPSp4gFQ/std.out-20130204153621_1-20130204153621_1 
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second
slept for 1 second

</verbatim>

---+++++ BUG: WM memory footprint bloats, especially on 64-bit architectures (bug #97288). HOPEFULLY FIXED

This bug cannot be checked in non production environments. A substantial change has been brought in the ldap purchasers, by removing legacy data structures and using a cleaner and unified approach. This needs to be tested in production


---+++++ BUG: glite-wms-job-output requires a valid delegated proxy on the WMS to enable output retrieval (bug #98524)

To reproduce the bug submit a job producing at least one output sandbox file, to a WMS EMI2 server...

<verbatim>
dorigoa@cream-51 18:52:15 ~>glite-wms-job-status https://devel09.cnaf.infn.it:9000/G5c9297pMa6nYcRR2WiAzA


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/G5c9297pMa6nYcRR2WiAzA
Current Status:     Done(Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        cream-47.pd.infn.it:8443/cream-lsf-creamtest1
Submitted:          Mon Feb  4 18:52:06 2013 CET
==========================================================================
</verbatim>

Then, move to the WMS server and change directory to =/var/SandboxDir/G5/https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2f/G5c9297pMa6nYcRR2WiAzA=

and remove proxies

<verbatim>
[root@cream-28 SandboxDir]# cd G5/https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fG5c9297pMa6nYcRR2WiAzA/
[root@cream-28 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fG5c9297pMa6nYcRR2WiAzA]# ll
total 24
drwxrwx--- 2 dteam042 glite 4096 Feb  4 18:52 input
-rw-r--r-- 1 glite    glite  708 Feb  4 18:52 JDLOriginal
-rw-r--r-- 1 glite    glite 2510 Feb  4 18:52 JDLToStart
drwxrwx--- 2 dteam042 glite 4096 Feb  4 18:52 output
drwxrwx--- 2 dteam042 glite 4096 Feb  4 18:52 peek
lrwxrwxrwx 1 glite    glite   64 Feb  4 18:52 user.proxy -> /var/glite/spool/glite-renewd/45a96bd6a16770e5fdc4c60bbae2646e.0
[root@cream-28 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fG5c9297pMa6nYcRR2WiAzA]# find . -iname "*proxy*"
./.user.proxy.bak
./user.proxy
[root@cream-28 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fG5c9297pMa6nYcRR2WiAzA]# \rm ./.user.proxy.bak ./user.proxy
</verbatim>

Go back to the UI and try a =glite-wms-job-output= that must fail (because of proxies removal):

<verbatim>
dorigoa@cream-51 18:53:53 ~>glite-wms-job-output https://devel09.cnaf.infn.it:9000/G5c9297pMa6nYcRR2WiAzA

Connecting to the service https://cream-28.pd.infn.it:7443/glite_wms_wmproxy_server


Error - getOutputFileList Error
 (Proxy exception: Unable to get Not Before date from Proxy)
</verbatim>

Now, perform another submission using a EMI3 WMS server:

<verbatim>
dorigoa@cream-51 18:59:30 ~>glite-wms-job-status https://devel09.cnaf.infn.it:9000/A1MWywIZXXW8-FMNd0pY9w


======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://devel09.cnaf.infn.it:9000/A1MWywIZXXW8-FMNd0pY9w
Current Status:     Done(Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        cream-47.pd.infn.it:8443/cream-lsf-creamtest1
Submitted:          Mon Feb  4 18:58:57 2013 CET
==========================================================================
</verbatim>

Remove proxies from WMS's sandboxdir:

<verbatim>
[root@cream-01 ~]# cd /var/SandboxDir/A1/https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fA1MWywIZXXW8-FMNd0pY9w/
[root@cream-01 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fA1MWywIZXXW8-FMNd0pY9w]# ls l
ls: l: No such file or directory
[root@cream-01 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fA1MWywIZXXW8-FMNd0pY9w]# find . -iname "*proxy*"
./.user.proxy.bak
./user.proxy
[root@cream-01 https_3a_2f_2fdevel09.cnaf.infn.it_3a9000_2fA1MWywIZXXW8-FMNd0pY9w]# \rm ./.user.proxy.bak ./user.proxy
</verbatim>

Go back to UI and try another job-output, that should be successful:

<verbatim>
dorigoa@cream-51 DING! ~>glite-wms-job-output https://devel09.cnaf.infn.it:9000/A1MWywIZXXW8-FMNd0pY9w

Connecting to the service https://cream-01.pd.infn.it:7443/glite_wms_wmproxy_server


Warning - JobPurging not allowed
 (Unable to copy file)


================================================================================

                        JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://devel09.cnaf.infn.it:9000/A1MWywIZXXW8-FMNd0pY9w
have been successfully retrieved and stored in the directory:
/tmp/jobOutput/dorigoa_A1MWywIZXXW8-FMNd0pY9w

================================================================================

dorigoa@cream-51 19:01:54 ~>ls -l /tmp/jobOutput/dorigoa_A1MWywIZXXW8-FMNd0pY9w
total 4
-rw-rw-r-- 1 dorigoa dorigoa 0 Feb  4 19:01 stderr
-rw-rw-r-- 1 dorigoa dorigoa 5 Feb  4 19:01 stdout

</verbatim>


-- Main.MarcoCecchi - 2013-01-16


-- Main.AlviseDorigo - 2013-04-16
Topic revision: r2 - 2013-04-16 - AlviseDorigo
Account
- Log In
Edit
Attach