(r72) WmsTestsP2459 < EgeeJra1It

Tags: , view all tags
---+++ Check bugs:

   * Bugs [[https://savannah.cern.ch/bugs/?39807][#39807]]:  In some circumstances, jobs which are killed by CREAM job wrapper might remain in ICE cache forever %GREEN%FIXED%ENDCOLOR%
      * Following the instructions reported in the bug's comments set: <verbatim>
start_listener = false;
start_subscription_updater = false;
poller_delay = 900;
poller_status_threshold_time = 60;</verbatim> in the Ice section of the configuration file (i.e. glite_wms.conf)
      * Submit a long job (i.e. a job that should run for more than 15 minutes), with a short proxy (i.e. a proxy with a lifetime of about 13 minutes).
      * Submit another job with a long proxy (i.e. more than an hour).
      * After about 17/18 minutes the original job should be ABORTED.

   * Bugs [[https://savannah.cern.ch/bugs/?42018][#42018]]: Missing exit on very severe error %GREEN%HOPEFULLY FIXED%ENDCOLOR%
      * Changes inside the code.

   * Bugs [[https://savannah.cern.ch/bugs/?42081][#42081]]: Exception not catched in ICE %GREEN%HOPEFULLY FIXED%ENDCOLOR%
      * Changes inside the code.

   * Bugs [[https://savannah.cern.ch/bugs/?42141][#42141]]: Calling the FileList::get_size() method should be mutex protected %GREEN%HOPEFULLY FIXED%ENDCOLOR%
      * Changes inside the code.

   * Bugs [[https://savannah.cern.ch/bugs/?44604][#44604]]: A bad handling of delegations slow down dramatically the submission rate of ICE %GREEN%HOPEFULLY FIXED%ENDCOLOR%
      * Show tests below.

   * Bugs [[https://savannah.cern.ch/bugs/?46116][#46116]]: MaxOutputSandboxSize value not sent to CREAM by ICE %GREEN%FIXED%ENDCOLOR%
      * Set the parameter MaxOutputSandboxSize? in the WorkloadManager? section of the configuration file /opt/glite/etc/glite_wms.conf on the WMS to 100 and restart the workload manager.
      * Submit to a cream CE a jdl like this:<verbatim>
[
Type = "Job";
Executable = "27215_exe.sh";
Arguments = "70";
StdOutput = "test.out";
StdError = "test.err";
InputSandbox = {"27215_exe.sh"};
OutputSandbox = {"test.err","test.out","out2", "out1"};
usertags = [ bug = "27215" ];
] </verbatim>where 27215_exe.sh contains <verbatim>
#!/bin/sh
MAX=$1
i=0
while [ $i -lt $MAX ]; do
                echo -n "1" >> out1
                echo -n "2" >> out2
    i=$[$i + 1]
done
</verbatim>
      * Take the _CreamJobID_ from the "Transfer Event" logged by the "LogMonitor" (i.e. The field _Dest jobid_)
      * Using the command of the client of the CE look inside the JDL sent to the ce: _glite-ce-job-status -L  2 &lt;CreamJobID&gt;_; you should find this parameter: *maxOutputSandboxSize = 1.000000000000000E+02;*
      * _Due to a bug in CREAM the output files are not truncated as expected._

   * Bugs [[https://savannah.cern.ch/bugs/?47389][#47389]]: There's a mem leak in ICE that raises in some very rare circumstances %GREEN%HOPEFULLY FIXED%ENDCOLOR%
      * Not easy to reproduce

   * Bugs [[https://savannah.cern.ch/bugs/?47509][#47509]]: ICE must be modified in order to be compliant with modification to CEMon C++ API %GREEN%FIXED%ENDCOLOR%
      * Verify if the subscription of ICE to the CE works well (you need to look inside the log file of ICE)

   * Bugs [[https://savannah.cern.ch/bugs/?47996][#47996]]: Apparent database corruption when ICE exits.%GREEN%FIXED%ENDCOLOR%
      * Submit a lot of jobs and restart ice daemon (i.e.: /opt/glite/etc/init.d/glite-wms-ice restart)

---+ TESTs on ICE

---++ 15)  Test starts on Thu Mar 19 at 12:00:57 CET 2009 (WMS: wms008)
 Description: 
   * 7200 collections each of 40 jobs
   * One collection every 60 seconds 
   * FIve users
   * max_ice_threads   =   20
   * Used all the CEs of testbedB (except cert-06.cnaf and cream-21.pd)
   * Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 313"
   * Resubmission is enabled
   * Lease mechanism is not used
   * Use rpms from patch [[https://savannah.cern.ch/patch/index.php?2459][#2459]]


---+++ Test finishes on  Tue Mar 24 at 11:56:59 CET 2009
   * 3750 collections submitted in 37283 seconds: 5/9/38 secs (min/avg/max)
      * 3450 submission fail due to _System load is too high_

---+++ Partial results taken on Fri Mar 24 at 13:44:29
   * Collections correctly submitted:  3750 (150000 jobs)     
      * DONE OK: %GREEN%  145210 (96.81%) %ENDCOLOR%
      * ABORTED: %RED% 611 (0.41%) %ENDCOLOR%
      * NotDone: %ORANGE% 4179 (2.78%) %ENDCOLOR%
      * Resubmitted: %BLUE% 1283 (0.86%) %ENDCOLOR%

   * Errors found (3191):
      * BLAH error _(201 times 6.3%)_
      * Cannot move ISB _(569 times 17.83%)_
      * Cannot move OSB _(148 times 4.64%)_
      * Transfer to CREAM failed _(*1895 times 59.39%*)_
      * Cannot take token _(50 times 1.57%)_
      * Proxy is expired _(315 times 9.87%)_
      * lsf_reason _(9 times 0.28%)_ 
      * blparser service is not alive _(3 times 0.09%)_ 
      * pbs_reason _(1 time 0.03%)_ 

   * All the aborted (and also the majority of the failures) are due to "proxy renewal" mechanism which doesn't work at the beginning of the test.
   * All the "NotDone" jobs are matched to a single ce (cert-05.cnaf.infn.it), which has some problems under investigation.

---++ 14)  Test starts on  Fri Mar 13 at 11:44:57 CET 2009 (WMS: wms008)
 Description: 
   * 7200 collections each of 40 jobs
   * One collection every 60 seconds 
   * Four users
   * max_ice_threads   =   20
   * Used all the CEs of testbedB (except cert-06.cnaf and cream-21.pd)
   * Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 313"
   * Resubmission is enabled
   * Lease mechanism is not used
   * Use rpms from patch [[https://savannah.cern.ch/patch/index.php?2459][#2459]]

---+++ Results (Mon Mar 16 17:21:55). Test interrupted due to a problem in the lsf server at Cnaf.
   * Collections correctly submitted:  2932 (117280 jobs)     
      * DONE OK: %GREEN%  113758 (97%) %ENDCOLOR%
      * ABORTED: %RED% 3 (0.003%) %ENDCOLOR%
      * NotDone: %ORANGE% 3519 (3%) %ENDCOLOR%
      * Resubmitted: %BLUE% 141 (0.12%) %ENDCOLOR%

   * Errors found (179):
      * Cannot move ISB _(14 times 7.82%)_
      * BLAH error _(*92 times 51.4%*)_
         * blah error: send command timeout _(28 times)_
         * submission command failed (exit code = -15) (stdout:) (stderr: <blah> exe_getouterr: 200 seconds timeout expired, killing child process.- <blah> killed by signal  15.-) N/A (jobId = [...]) _(43 times)_
         * submission command failed (exit code = 1) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...- <blah> exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (jobId = [...]) _(13 times)_
         * no jobId in submission script's output (stdout:) (stderr: <blah> exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (jobId = [...]) _(7 times)_
         *  submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) _(1 time)_
      * Transfer to CREAM failed _(56 times 31.28%)_
         *  due to exception: CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Fri 13 Mar 2009 11:50:35] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]" _(56 times)_
      * Cannot take token _(14 times 7.82%)_
      * Others  _(3 times 1.68%)_
         * The job cannot be submitted because the blparser service is not alive _(3 times)_


  

---++ 13)  Test starts on  Wed Mar  4 at 13:41:28 CET 2009 (WMS: wms007)
 Description: 
   * 120 collections each of 60 jobs
   * One collection every 60 seconds 
   * Four users
   * The job is a "sleep 313"
   * Resubmission is enabled
   * We use both CREAM and LCG CEs
   * Long proxy

---+++ Test finishes on Wed Mar  4 at 15:38:14 CET 2009
   * 120 collections submitted in 1108 seconds: 4/9/16 (min/avg/max)

---+++ Final results
   * Collections correctly submitted:  120 (7200 jobs)     
      * DONE OK: %GREEN%  7198 (99.97%) %ENDCOLOR%
         * CREAM: 2537
         * LCG: 4661
      * ABORTED: %RED% 0 (0%) %ENDCOLOR%
      * Not finished: %ORANGE% 2 (0.03%) %ENDCOLOR%
         * LCG: 2
      * Resubmitted: %BLUE% 182 (2.53%) %ENDCOLOR%

---++ 12)  Test starts on  Feb 24 at 10:29:07 CET 2009 (WMS: wms007)
 Description: 
   * 120 collections each of 60 jobs
   * One collection every 60 seconds 
   * Four users
   * The job is a "sleep 313"
   * Resubmission is enabled
   * We use both CREAM and LCG CEs
   * %PINK%Long proxy%ENDCOLOR%

---+++ Test finishes on Mon Feb 24 at 12:25:54 CET 2009
   * 92 collections submitted in 712 seconds: 4/7/13 (min/avg/max)
      * 28 submission(s) fail(s) (due to load limiter)

---+++ Final results
   * Collections correctly submitted:  91 (5460 jobs)     
      * DONE OK: %GREEN%  5460 (100%) %ENDCOLOR%
         * CREAM: 3400
         * LCG: 2060
      * ABORTED: %RED% 0 (0%) %ENDCOLOR%
      * Resubmitted: %BLUE% 163 (2.99%) %ENDCOLOR%

   * The submission of one collection failed due to:
<verbatim>
Status Reason:      LBProxy is enabled
Unable to query LB and LBProxy
edg_wll_QueryEvents[Proxy]
Exit code: 1413
LB[Proxy] Error: DNS resolver error
(edg_wll_gss_connect(): Unknown host)
</verbatim>

---++ 11)  Test starts on  Mon Feb 23  at  15:25:49CET 2009  (WMS: wms007)
 Description: 
   * 120 collections each of 60 jobs
   * One collection every 60 seconds 
   * Four users
   * The job is a "sleep 313"
   * Resubmission is enabled
   * %PINK%We use both CREAM and LCG CEs%ENDCOLOR%

---+++ Test finishes on Mon Feb 23 at 17:22:32 CET 2009
   * 120 collections submitted in 1092 seconds: 5/9/17 (min/avg/max)

---+++ Final results
   * Collections correctly submitted:  120 (7200 jobs)     
      * DONE OK: %GREEN%  6950 (96.53%) %ENDCOLOR%
         * CREAM: 2243
         * LCG: 4707
      * ABORTED: %RED% 249 (3.46%) %ENDCOLOR%
         * LCG: 249
      * Not finished: %ORANGE% 1 (0.01%) %ENDCOLOR%
         * LCG: 1
      * Resubmitted: %BLUE% 696 (9.7%) %ENDCOLOR%

   * All the jobs have been aborted for "proxy expired" because the job renewal daemon doesn't work. 

---++ 10)  Test starts on Tue Feb  5 at 12:41:35 CET 2009 (WMS: devel14)
 Description: 
   * 4320 collections each of 40 jobs
   * One collection every 60 seconds 
   * Five users
   * max_ice_threads   =   20
   * Used all the CEs of testbedB (except cert-06.cnaf) plus cream-04.pd.infn.it
   * Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 4242"
   * Resubmission is enabled
   * Lease mechanism is not used
%PINK%
   * Changes in the software wrt previous test:
      * ICE
         * Fix a problem with proxy renewal seen in the previous test
         * Removed useless check of proxy duration in subscriptionManager, which could result in performance problems
%ENDCOLOR%

---+++ Final results taken on Thu Feb 10 at 16:20:32 CET 2009
   * Collections correctly submitted:  1568 (62720 jobs)     
      * DONE OK: %GREEN%  58641 (93.5%) %ENDCOLOR%
      * ABORTED: %RED% 4079 (6.5%) %ENDCOLOR%
      * Resubmitted: %BLUE% 41601 (66.33%) %ENDCOLOR%

   * Errors found (82530):
      * Cannot move ISB _(*75625 times 91.63%*)_
      * Cannot move OSB _(115 times 0.14%)_
      * Proxy is expired _(5711 times 6.92%)_
      * pbs_reason _(702 times 0.85%)_ 
         * pbs_reason=1; [...] proxy expired _(477 times)_
         * pbs_reason=271  _(225 times)_
      * Transfer to CREAM failed _(187 times 0.23%)_
         * due to exception: CREAM Register raised std::exception Connection to service [...] failed: _(187 times)_
      * lsf_reason _(184 times 0.23%)_ 
         * lsf_reason=36608; Proxy expired: job killed Terminated Master process killed _(138 times)_
         * lsf_reason=256 _(43 times)_
         * lsf_reason=1603 _(3 times)_
      * Cannot take token _(4 times 0%)_
      * BLAH error _(2 times 0%)_
         * submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) _(2 times)_

   * Job Aborted (4079)
      * request expired _(792 times 19.42%)_
      * hit job shallow retry count (3) _(3263 times 80%)_ 
      * hit job retry count (2) _(24 times 0.58%)_

%ATTACHURL%/ice10.png

---++ 9) Test starts on Fri Jan 30 at 12:41:22 CET 2009 (WMS: devel14)
 Description: 
   * 4320 collections each of 40 jobs
   * One collection every 60 seconds 
   * Five users
   * max_ice_threads   =   20
   * Used all the CEs of testbedB (except cert-06.cnaf) plus cream-04.pd.infn.it
   * Used automatic-delegation and proxy renewal service (%PINK%MyProxyServer = "myproxy.cern.ch";%ENDCOLOR%)
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 4242"
   * Resubmission is enabled
   * Lease mechanism is not used
%PINK%
   * Changes in the software wrt previous test:
      * CEs: 
         * Fix for bug #45913 (only on cream-04.pd.infn.it)
         * Fix for bug #46283 (only on cream-04.pd.infn.it and cert-04.pd.infn.it)
      * ICE
         * Fix for bug  #46405
         * 5 sec. (instead of 60) of delay between two LB logging tries 
         * Error code is printed in the ICE log file when a log to LB fails
%ENDCOLOR%

---+++ Test finishes on Mon Feb  2 at 12:39:09 CET 2009
   * 1427 collections submitted in 29752 seconds: 5/20/90 (min/avg/max)
      * 2893 submissions fail due to load limiter

---+++ Results taken on Thu Feb 02 at 15:24:31 CET 2009
   * Collections correctly submitted:  1427 (57080 jobs)     
      * DONE OK: %GREEN%  28244 (49.48%) %ENDCOLOR%
      * ABORTED: %RED% 0 (0%) %ENDCOLOR%
      * Not finished: %ORANGE%  28836 (50.52%) %ENDCOLOR%
      * Resubmissions: %BLUE% 2012 (3.52%) %ENDCOLOR%

   * Errors found (2233):
      * BLAH error _(19 times 0.85%)_
         * submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-) N/A (jobId = [...]) _(18 times)_
         * submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)- <blah> exe_getouterr: poll() got an unknown event (stdout 0x0010 - stderr: 0x0000).-) N/A (jobId = [...]) _(1 time)_
      * Cannot move ISB _(*1872 times 83.84%*)_
      * Proxy is expired _(325 times 14.55%)_
      * lsf_reason _(1 time 0.04%)_ 
         * lsf_reason=36608 _(1 time)_
      * pbs_reason _(16 times 0.72%)_ 
         * pbs_reason=1; [...] proxy expired _(15 times)_
         * pbs_reason=271; Proxy expired: job killed Terminated Master process killed _(1 time)_

%ATTACHURL%/ice9.png

---++ 8) Test starts on Mon Jan 26 17:59:02 CET 2009 (WMS: devel14)
 Description: 
   * %PINK%4320%ENDCOLOR% collections each of 40 jobs
   * One collection every 60 seconds 
   * Five users
   * max_ice_threads   =   20
   * Used all the CEs of testbedB (except cert-06.cnaf)
   * Used automatic-delegation and proxy renewal service
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 4242"
   * Resubmission is able
   * Lease mechanism is not used
%PINK%
   * Changes in the software wrt previous test:
      * ICE: 
         * Fixed problem with proxy renewal seen in previous test
%ENDCOLOR%

*%PINK%Test interrupted for a problem in the proxy-renewal service daemon on Thu Jan 29 12:05:12 CET 2009%ENDCOLOR%*

---+++ Results taken on Thu Jan 29 at 18:35:31 CET 2009
   * Collections correctly submitted:  1433 (57320 jobs)     
      * DONE OK: %GREEN%  33566 (58.56%) %ENDCOLOR%
      * ABORTED: %RED% 8414 (14.68%) %ENDCOLOR%
      * Not finished: %ORANGE%  15340 (26.76%) %ENDCOLOR%
      * Resubmissions: %BLUE% 33507 (58.46%) %ENDCOLOR%

   * Errors found (33957):
      * BLAH error _(3 time 0.01%)_
         * submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) _(1 time)_
         * submission command failed (exit code = 106) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) _(2 times)_
      * Cannot move ISB _(4513 times 13.29%)_
      * Cannot move OSB _(73 times 0.21%)_
      * Transfer to CREAM failed _(28607 times 84.24%)_
         * due to exception: Authentication error: The proxy is EXPIRED! _(*28134 times*)_
         * due to exception: Authentication error: Unable to open the file [/var/glite/SandboxDir/[...]/user.proxy] : No such file or directory _(432 times)_
         * Failed to create a delegation id for job [...]: reason is Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] _(19 times)_
         * Failed to create a delegation id for job [...]: reason is Failed proxy validation - it has expired. _(4 times)_
         * Failed to create a delegation id for job [...]: reason is CreamProxy_Delegate::execute() - Coundl't open proxyfile [...]: The proxy is EXPIRED! _(1 time)_
         * CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] _(12 times)_
         * CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Wed 28 Jan 2009 17:42:42] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]" _(5 times)_
      * Proxy is expired _(688 times 2.03%)_
      * lsf_reason _(61 times 0.18%)_ 
         * lsf_reason=65280 _(32 times)_
         * lsf_reason=36608 _(22 times)_
         * lsf_reason=1603 _(1 time)_
         * lsf_reason=256 _(6 times)_
      * pbs_reason _(12 times 0.04%)_ 
         * pbs_reason=271; Proxy expired: job killed Terminated Master process killed _(12 times)_

%ATTACHURL%/ice8.png

---+++ BUGS:
   * CREAM
      * [[https://savannah.cern.ch/bugs/?46405][#46405]]: VOMSWrapper should try more than once to open a proxy file
   * BLAH
      * [[https://savannah.cern.ch/bugs/?46283][#46283]]: Possible memory leak in strtoken function for BLParser

---++ 7) Test starts on Fri Jan 23 12:28:01 CET 2009 (WMS: devel14)
 Description: 
   * 7200 collections each of 40 jobs
   * One collection every 60 seconds 
   * %PINK%Five users%ENDCOLOR%
   * max_ice_threads   =   20
   * Used all the CEs of testbedB (except cert-06.cnaf)
   * Used automatic-delegation and proxy renewal service
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * %PINK%The job is a "sleep 4242"%ENDCOLOR%
   * Resubmission is able
   * Lease mechanism is not used
%PINK%
   * Changes in the software wrt previous test:
      * ICE: 
         * Fixed problem seen in previous test
%ENDCOLOR%

*%PINK%Test interrupted on Mon Jan 26 17:05:12 CET 2009%ENDCOLOR%*

---+++ Results taken on Mon Jan 26 at 18:20:31 CET 2009
   * Collections correctly submitted:  1741 (69640 jobs)      
      * DONE OK: %GREEN% 51894 (74.52%) %ENDCOLOR%
      * ABORTED: %RED% 44 (0.06%) %ENDCOLOR%
      * Not finished: %ORANGE% 16186 (23.24%) %ENDCOLOR%
      * CANCELLED: %ORANGE% 1516 (2.18%) %ENDCOLOR%
      * Resubmissions: %BLUE% 18218 (26.16%) %ENDCOLOR%

   * Errors found (18218):
      * BLAH error _(2 times 0.01%)_
         * submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) _(1 time)_
         * submission command failed (exit code = 1) (stdout:) (stderr:Cannot resolve default server host 'cream-28.pd.infn.it' - check server_name file.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15008)-) N/A (jobId = [...]) _(1 time)_
      * Cannot take token _(77 times 0.42%)_
      * Cannot move ISB _(*14222 time 78.07%*)_
      * Cannot move OSB _(82 times 0.45%)_
      * Transfer to CREAM failed _(4 times 0.02%)_
         * due to exception: Authentication error: Unable to open the file [/var/glite/SandboxDir/[...]/user.proxy] : No such file or directory _(4 times)_
      * lsf_reason _(30 times 0.16%)_ 
         * lsf_reason=36608 _(17 times)_
         * lsf_reason=256 _(13 times)_
      * Proxy is expired _(3113 times 17.09%)_
      * pbs_reason _(688 times 3.78%)_
         * pbs_reason=-1 _(656 times)_
         * pbs_reason=271; Proxy expired: job killed Terminated Master process killed _(32 times)_

%ATTACHURL%/ice7.png

---++ 6) Test starts on Thu Jan 22 at 17:17:38 CET 2009 (WMS: devel14)
 Description: 
   * 7200 collections each of 40 jobs
   * One collection every 60 seconds 
   * %PINK%Five users%ENDCOLOR%
   * max_ice_threads   =   20
   * Used all the CEs of testbedB (except cert-06.cnaf)
   * Used automatic-delegation and proxy renewal service
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * %PINK%The job is a "sleep 4242"%ENDCOLOR%
   * Resubmission is able
   * Lease mechanism is not used
%PINK%
   * Changes in the software wrt previous test:
      * CEs: 
         * Fix for bug #45718
         * Fix for bug #45983
         * Fix for bug #46024
      * ICE 
         * Use of same delegationid if CREAM complains that it doesn't exist anymore 
%ENDCOLOR%
   
*%PINK%Test aborted on Fri Jan 23 10:30:12 CET 2009%ENDCOLOR%*

---++ 5) Test starts on Wed Jan 21 at 12:45:49 CET 2009 (WMS: devel14)
 Description: 
   * 300 collections each of %PINK%80 jobs%ENDCOLOR%
   * One collection every 60 seconds 
   * One user
   * %PINK%Set max_ice_threads   =   40;%ENDCOLOR%
   * Used the CEs of testbedB (only PD)
   * Used automatic-delegation and proxy renewal service
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 313"
   * Resubmission is able
   * Lease mechanism is not used

---+++ Test finishes on Wed Jan 21 at 17:44:58 CET 2009
   * 224 collections submitted in 7416 seconds: 64/33/6 (max/avg/min)
      * 76 submissions fail due to load limiter

   * Collections correctly submitted:  224 (17920 jobs)
      * DONE OK: %GREEN% 17850 (99.6%) %ENDCOLOR%
      * ABORTED: %RED% 0 (0.0%) %ENDCOLOR%
      * Not finished: %ORANGE% 70 (0.4%) %ENDCOLOR%
      * Resubmissions: %BLUE% 8 (0.04%) %ENDCOLOR%

   * Errors found (8):
      * Cannot move ISB _(3 times)_
      * lsf_reason=1603 _(3 times)_
      * BLAH error _(2 times)_

%ATTACHURL%/ice5.png

---++ 4) Test starts on Tue Jan 20 at 10:09:58 CET 2009 (WMS: devel14)
 Description: 
   * 300 collections each of %PINK%80 jobs%ENDCOLOR%
   * One collection every 60 seconds 
   * One user
   * Used the lsf CEs of testbedB (PD+CNAF) (cert-06 at cnaf is not considered)
   * Used automatic-delegation and proxy renewal service
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 313"
   * Resubmission is able
   * Lease mechanism is not used

---+++ Test finishes on Tue Jan 20 at 15:13:14 CET 2009
   * 197 collections submitted in 8499 seconds: 84/43/9 (max/avg/min)
      * 103 submissions fail due to load limiter

   * Collections correctly submitted:  197 (15760 jobs)
      * DONE OK: %GREEN% 15695 (99.6%) %ENDCOLOR%
      * ABORTED: %RED% 0 (0.0%) %ENDCOLOR%
      * Not finished: %ORANGE% 65 (0.4%) %ENDCOLOR%
      * Resubmissions: %BLUE% 3 (0.02%) %ENDCOLOR%

   * Errors found (3):
      * Cannot move OSB _(1 time)_
      * Cannot move ISB _(2 times)_

%ATTACHURL%/ice4.png

---++ 3) Test starts on Mon Jan 19 at 15:22:51 CET 2009 (WMS: devel14)
 Description: 
   * 2880 collections each of 40 jobs
   * One collection %PINK%every 30 seconds%ENDCOLOR%
   * One user
   * Used the lsf CEs of testbedB (PD+CNAF) 
   * Used automatic-delegation and proxy renewal service
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 313"
   * Resubmission is enabled
   * Lease mechanism is not used
   
Test has been modified on Mon Jan 19 at 17:03:41:
   * 1440 collections each of 80 jobs
   * One collection every 60 seconds

---+++  Test finishes on Tue Jan 20 at 01:53:01 CET 2009
 
   * Collections correctly submitted: 399 (24800 jobs)
      * DONE OK: %GREEN% 24702 (99.6%) %ENDCOLOR%
      * ABORTED: %RED% 0 (0.0%) %ENDCOLOR%
      * Not finished: %ORANGE% 98 (0.4%) %ENDCOLOR%
      * Resubmissions: %BLUE% 1176 (4.74%) %ENDCOLOR%

   * Errors found (1176):
      * Cannot take token _(1 time 0.08%)_
      * Cannot move ISB _(39 times 3.32%)_
      * Transfer to CREAM failed _(50 times 4.25%)_
         * Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client]  _(7 times)_
         * Transfer to CREAM failed due to exception: Authentication error: The proxy is EXPIRED!  _(43 times)_
      * lsf_reason=32512 _(*1085 times 92.27%*)_ 
      * lsf_reason=306 _(1 time 0.08%)_

%ATTACHURL%/ice3.png


---++ 2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14)
 Description: 
   * 7200 collections each of 40 jobs
   * One collection every 60 seconds 
   * One user
   * Used the CEs of testbedB (PD+CNAF) plus cream-04.pd.infn.it
   * Used automatic-delegation and proxy renewal service
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 313"
   * Resubmission is enabled
   * %PINK%Lease mechanism is not used%ENDCOLOR%
%PINK%
   * Changes in the software wrt previous test:
      * CEs: 
         * Fix for bug #45437
         * Fix for bug #45736 
      * ICE
         * Management of serialization error
         * Renewal done at 80 % of lifetime of proxy (or when there are only 20 minutes left)
%ENDCOLOR%

---+++  Test finishes on Sun Jan 18 at 15:42:28 CET 2009
   * 7180 collections submitted in 70789 seconds: 141/9/3 (max/avg/min)
      * 20 submissions fail due to load limiter

   * Collections correctly submitted:  7180 ( 287200 jobs)
      * DONE OK: %GREEN% 284838 (99.18%) %ENDCOLOR%
      * ABORTED: %RED% 0 (0.0%) %ENDCOLOR%
      * Not finished: %ORANGE% 2362 (0.82%) %ENDCOLOR%
      * Resubmissions: %BLUE% 4599 (1.60%) %ENDCOLOR%

   * Errors found (4599):
      * blparser service is not alive _(578 times 12.57%)_
      * BLAH error _(288 times 6.26%)_
         * no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = [...]) _(52 times)_
         * send command timeout _(2 times)_
         * submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) _(2 times)_
         * submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = [...]) _(219 times)_
         * submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) _(13 times)_
      * Cannot take token _(201 times 4.37%)_
      * Cannot move OSB _(1 time 0.02%)_
      * Cannot move ISB _(5 times 0.11%)_
      * Transfer to CREAM failed _(19 times 0.41%)_
         * FaultCause=[The problem seems to be related to glexec which reported: java.io.IOException: Too many open files]" _(10 times)_
         * CREAM Register raised std::exception Connection to service [https://cert-xx.cnaf.infn.it:8443/ce-cream/services/CREAM2] failed: _(9 times)_
      * lsf_reason=32512 _(*3505 times 76.22%*)_ 
      * Proxy is expired _(1 time 0.02%)_
      * lsf_reason=306 _(1 time 0.02%)_

   * Jobs not finished:

| *Schedul* | *Running* | *Tot.* | *Ce Name* |
|  0 |  6 |  6 | cert-04.cnaf.infn.it |
|  6 |  349 |  355 | cream-34.pd.infn.it |
|  0 |  3 |  3 | cream-26.pd.infn.it |
|  0 |  2 |  2 | cream-25.pd.infn.it |
|  5 |  332 |  337 | cream-28.pd.infn.it |
|  0 |  1 |  1 | cream-27.pd.infn.it |
|  0 |  2 |  2 | cream-22.pd.infn.it |
|  0 |  5 |  5 | cream-04.pd.infn.it |
|  0 |  1 |  1 | cream-23.pd.infn.it |
|  0 |  2 |  2 | cert-07.cnaf.infn.it |
|  1 |  0 |  1 | cert-13.cnaf.infn.it |
|  6 |  334 |  340 | cream-29.pd.infn.it |
|  9 |  327 |  336 | cream-33.pd.infn.it |
|  0 |  5 |  5 | cert-08.cnaf.infn.it |
|  0 |  2 |  2 | cert-05.cnaf.infn.it |
|  6 |  0 |  6 | cert-06.cnaf.infn.it |
|  6 |  307 |  313 | cream-31.pd.infn.it |
|  4 |  296 |  300 | cream-32.pd.infn.it |
|  8 |  337 |  345 | cream-30.pd.infn.it |
|  *51* |  *2311* |  *2362* | *Totals* |

---+++ BUGS:
   * CREAM
      * [[https://savannah.cern.ch/bugs/index.php?46024][#46024]]: Sometimes the AbstractJobExecutor throws the exception "Too many open files"
   * BLAH
      * [[https://savannah.cern.ch/bugs/index.php?45718][#45718]]: Some check on log lines should be added on BLParser code
      * [[https://savannah.cern.ch/bugs/index.php?45983][#45983]]: BLAH can leave children processes behind.

---++ 1) Test starts on Wed Jan  7 at 16:01:32 CET 2009 (WMS: devel18)
 Description: 
   * 7200 collections each of 40 jobs
   * One collection every 60 seconds 
   * One user
   * Used the CEs of testbedB (PD+CNAF) plus cream-12.pd.infn.it
   * Used automatic-delegation and proxy renewal service
   * Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
   * The job is a "sleep 313"
   * Resubmission is able

*%PINK%Test stopped on Monday Jan 12 for a _serialization error_ on ICE%ENDCOLOR%*

---+++ Results taken on Mon Jan 12 at 12:52:56 CET 2009
   * Collections correctly submitted: 3733 (149320 jobs)
      * DONE OK: %GREEN%144004 (96.44%) %ENDCOLOR%
      * ABORTED: %RED%446 (0.3%) %ENDCOLOR%
      * Not finished: %ORANGE%4870 (3.26%) %ENDCOLOR%

   * Errors found:
      * Transfer to CREAM failed due to exception:
         * FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Deadlock found when trying to get lock; try restarting transaction]"
         * Authentication error: Unable to open the file [...]: No such file or directory
         * Connection to service [...] failed:
         * FaultCause=[User [...] not authorized for operation JobRegister]
         * FaultCause=[The problem seems to be related to glexec which reported: java.io.IOException: Too many open files]"
         *  FaultCause=[org.glite.ce.common.db.DatabaseException: Server connection failure during transaction. Due to underlying exception: 'java.net.SocketException: Too many open files'.
         * FaultCause=[java.net.UnknownHostException: cream-31.pd.infn.it: cream-31.pd.infn.it]"
         * CREAM Start raised exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client]
         * Failed to get lease_id for job [...] Exception is Lease renew operation FAILED for lease ID [...] Exception is Connection to service [https://cream-29.pd.infn.it:8443/ce-cream/services/CREAM2] failed:
         * CREAM Start failed due to error MethodName=[JOB_START] Timestamp=[Wed 07 Jan 2009 22:10:43] ErrorCode=[2] Description=[the job has a status not compatible with the JOB_START command!] FaultCause=[N/A]
      * BLAH error:
         * submission command failed (exit code = -15) (stdout:) (stderr:/opt/glite/etc/blah.config: line 54: syntax error near unexpected token `('-/opt/glite/etc/blah.config: line 54: `//Added for test by Enrico Fattibene (07/01/2009)'--killed by signal 15-) N/A (jobId = CREAM251333253)
         * submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = CREAM550710004)
         * submission command failed (exit code = 1) (stdout:) (stderr:Cannot resolve default server host 'cream-28.pd.infn.it' - check server_name file.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15008)-) N/A (jobId = CREAM027575485)
         * submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = CREAM752590056)
         * no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = CREAM988027857)
      * DELEGATION_PROXY_CERT_SANDBOX_PATH not defined!
      * Cannot move ISB [...] The proxy credential [...] expired 0 minutes ago.
      * Proxy is expired; Proxy expired: job killed Terminated Master process killed
      * lsf_reason=32512 
      * Lease expired
      * The job cannot be submitted because the blparser service is not alive

---+++ BUGS:
   * CREAM
      * [[https://savannah.cern.ch/bugs/index.php?45914][#45914]]: glexec and proxy rotation
      * [[https://savannah.cern.ch/bugs/index.php?45913][#45913]]: Proxy renewal not done for CREAM jobs not yet in IDLE status
      * [[https://savannah.cern.ch/bugs/index.php?45736][#45736]]: Problems in case of resubmissions in the same CREAM CE 
      * [[https://savannah.cern.ch/bugs/index.php?45437][#45437]]: Sometimes the jobPurger throws the exception "Too many open files"
   * BLAH
      * [[https://savannah.cern.ch/bugs/index.php?45718][#45718]]: Some check on log lines should be added on BLParser code
      * [[https://savannah.cern.ch/bugs/index.php?45717][#45717]]: BLParserPBS should consider log lines like "unable to run job" 

-- Main.AlessioGianelle - 08 Jan 2009
Attachments
Topic attachments
I	Attachment	Action	Size	Date	Who	Comment
png	ice10.png	manage	5.6 K	2009-02-09 - 11:28	AlessioGianelle	Test 10 Ice submission rate
png	ice3.png	manage	4.5 K	2009-01-20 - 16:16	AlessioGianelle	Test 3 Ice submission rate
png	ice4.png	manage	6.4 K	2009-01-21 - 10:15	AlessioGianelle	Test4 Ice submission rate
png	ice5.png	manage	6.0 K	2009-01-22 - 08:41	AlessioGianelle	Test 5 Ice submission rate
png	ice7.png	manage	5.8 K	2009-01-26 - 17:08	AlessioGianelle	Test 7 Ice submission rate
png	ice8.png	manage	5.9 K	2009-01-28 - 11:06	AlessioGianelle	Test 8 Ice submission rate
png	ice9.png	manage	5.4 K	2009-02-02 - 11:33	AlessioGianelle	Test 9 Ice submission rate
Topic revision: r72 - 2009-03-24 - AlessioGianelle
Account
- Log In
~~Edit~~
~~Attach~~