Difference: WmsTestsP2459 (1 vs. 75)

Revision 752011-02-24 - AlessioGianelle

Line: 1 to 1
Changed:
<
<
META TOPICPARENT name="TestWokPlan"
>
>
META TOPICPARENT name="TestPage"
 

Check bugs:

  • Bugs #39807: In some circumstances, jobs which are killed by CREAM job wrapper might remain in ICE cache forever FIXED

Revision 742009-10-06 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 22 to 22
 
    • Changes inside the code.

  • Bugs #44604: A bad handling of delegations slow down dramatically the submission rate of ICE HOPEFULLY FIXED
Changed:
<
<
    • Show tests below.
>
>
    • See tests below.
 

Revision 732009-03-24 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 80 to 80
 
  • 3750 collections submitted in 37283 seconds: 5/9/38 secs (min/avg/max)
    • 3450 submission fail due to System load is too high
Changed:
<
<

Partial results taken on Fri Mar 24 at 13:44:29

>
>

Final results taken on Fri Mar 24 at 13:44:29

 
  • Collections correctly submitted: 3750 (150000 jobs)
    • DONE OK: 145210 (96.81%)
    • ABORTED: 611 (0.41%)

Revision 722009-03-24 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 80 to 80
 
  • 3750 collections submitted in 37283 seconds: 5/9/38 secs (min/avg/max)
    • 3450 submission fail due to System load is too high
Changed:
<
<

Partial results taken on Fri Mar 24 at 10:04:29

  • Collections correctly submitted: 3652 (146080 jobs)
    • DONE OK: 142379 (97.47%)
>
>

Partial results taken on Fri Mar 24 at 13:44:29

  • Collections correctly submitted: 3750 (150000 jobs)
    • DONE OK: 145210 (96.81%)
 
    • ABORTED: 611 (0.41%)
Changed:
<
<
    • NotDone: 3090 (2.12%)
    • Resubmitted: 1283 (0.88%)
>
>
    • NotDone: 4179 (2.78%)
    • Resubmitted: 1283 (0.86%)
 
  • Errors found (3191):
    • BLAH error (201 times 6.3%)

Revision 712009-03-24 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 75 to 75
 
  • Lease mechanism is not used
  • Use rpms from patch #2459
Changed:
<
<

Partial results taken on Fri Mar 23 at 10:04:29

  • Collections correctly submitted: 2974 (118960 jobs)
    • DONE OK: 115868 (97.4%)
    • ABORTED: 600 (0.5%)
    • NotDone: 2492 (2.1%)
    • Resubmitted: 1171 (0.98%)

  • Errors found (3069):
    • BLAH error (197 times 6.42%)
    • Cannot move ISB (560 times 18.25%)
    • Cannot move OSB (141 times 4.59%)
    • Transfer to CREAM failed (1861 times 60.64%)
    • Cannot take token (50 times 1.63%)
    • Proxy is expired (257 times 8.37%)
    • lsf_reason (3 time 0.1%)
>
>

Test finishes on Tue Mar 24 at 11:56:59 CET 2009

  • 3750 collections submitted in 37283 seconds: 5/9/38 secs (min/avg/max)
    • 3450 submission fail due to System load is too high

Partial results taken on Fri Mar 24 at 10:04:29

  • Collections correctly submitted: 3652 (146080 jobs)
    • DONE OK: 142379 (97.47%)
    • ABORTED: 611 (0.41%)
    • NotDone: 3090 (2.12%)
    • Resubmitted: 1283 (0.88%)

  • Errors found (3191):
    • BLAH error (201 times 6.3%)
    • Cannot move ISB (569 times 17.83%)
    • Cannot move OSB (148 times 4.64%)
    • Transfer to CREAM failed (1895 times 59.39%)
    • Cannot take token (50 times 1.57%)
    • Proxy is expired (315 times 9.87%)
    • lsf_reason (9 times 0.28%)
    • blparser service is not alive (3 times 0.09%)
    • pbs_reason (1 time 0.03%)
 
  • All the aborted (and also the majority of the failures) are due to "proxy renewal" mechanism which doesn't work at the beginning of the test.
Changed:
<
<
  • All the "NotDone" jobs are matched to a single ce, which has some problems under investigation.
>
>
  • All the "NotDone" jobs are matched to a single ce (cert-05.cnaf.infn.it), which has some problems under investigation.
 

14) Test starts on Fri Mar 13 at 11:44:57 CET 2009 (WMS: wms008)

Description:

Revision 702009-03-23 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 75 to 75
 
  • Lease mechanism is not used
  • Use rpms from patch #2459
Changed:
<
<

Partial results taken on Fri Mar 20 at 10:04:29

  • Collections correctly submitted: 827 (33080 jobs)
    • DONE OK: 31446 (95.06%)
    • ABORTED: 32 (0.10%)
    • NotDone: 1602 (4.84%)
    • Resubmitted: 124 (0.37%)
>
>

Partial results taken on Fri Mar 23 at 10:04:29

  • Collections correctly submitted: 2974 (118960 jobs)
    • DONE OK: 115868 (97.4%)
    • ABORTED: 600 (0.5%)
    • NotDone: 2492 (2.1%)
    • Resubmitted: 1171 (0.98%)

  • Errors found (3069):
    • BLAH error (197 times 6.42%)
    • Cannot move ISB (560 times 18.25%)
    • Cannot move OSB (141 times 4.59%)
    • Transfer to CREAM failed (1861 times 60.64%)
    • Cannot take token (50 times 1.63%)
    • Proxy is expired (257 times 8.37%)
    • lsf_reason (3 time 0.1%)

  • All the aborted (and also the majority of the failures) are due to "proxy renewal" mechanism which doesn't work at the beginning of the test.
  • All the "NotDone" jobs are matched to a single ce, which has some problems under investigation.
 

14) Test starts on Fri Mar 13 at 11:44:57 CET 2009 (WMS: wms008)

Description:

Revision 692009-03-20 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 75 to 75
 
  • Lease mechanism is not used
  • Use rpms from patch #2459
Added:
>
>

Partial results taken on Fri Mar 20 at 10:04:29

  • Collections correctly submitted: 827 (33080 jobs)
    • DONE OK: 31446 (95.06%)
    • ABORTED: 32 (0.10%)
    • NotDone: 1602 (4.84%)
    • Resubmitted: 124 (0.37%)
 

14) Test starts on Fri Mar 13 at 11:44:57 CET 2009 (WMS: wms008)

Description:

Revision 682009-03-19 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 60 to 60
 
    • Submit a lot of jobs and restart ice daemon (i.e.: /opt/glite/etc/init.d/glite-wms-ice restart)

TESTs on ICE

Added:
>
>

15) Test starts on Thu Mar 19 at 12:00:57 CET 2009 (WMS: wms008)

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
  • FIve users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf and cream-21.pd)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Use rpms from patch #2459
 

14) Test starts on Fri Mar 13 at 11:44:57 CET 2009 (WMS: wms008)

Description:
  • 7200 collections each of 40 jobs

Revision 672009-03-16 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 74 to 74
 
  • Lease mechanism is not used
  • Use rpms from patch #2459
Changed:
<
<

Partial Results (Mon Mar 16 10:05:55)

  • Collections correctly submitted: 2692 (107680 jobs)
    • DONE OK: 105122 (97.62%)
>
>

Results (Mon Mar 16 17:21:55). Test interrupted due to a problem in the lsf server at Cnaf.

  • Collections correctly submitted: 2932 (117280 jobs)
    • DONE OK: 113758 (97%)
 
    • ABORTED: 3 (0.003%)
Changed:
<
<
    • NotDone: 2555 (2.37%)
    • Resubmitted: 177 (0.16%)
>
>
    • NotDone: 3519 (3%)
    • Resubmitted: 141 (0.12%)

  • Errors found (179):
    • Cannot move ISB (14 times 7.82%)
    • BLAH error (92 times 51.4%)
      • blah error: send command timeout (28 times)
      • submission command failed (exit code = -15) (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.- killed by signal 15.-) N/A (jobId = [...]) (43 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...- exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (jobId = [...]) (13 times)
      • no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (jobId = [...]) (7 times)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) (1 time)
    • Transfer to CREAM failed (56 times 31.28%)
      • due to exception: CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Fri 13 Mar 2009 11:50:35] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]" (56 times)
    • Cannot take token (14 times 7.82%)
    • Others (3 times 1.68%)
      • The job cannot be submitted because the blparser service is not alive (3 times)

 

13) Test starts on Wed Mar 4 at 13:41:28 CET 2009 (WMS: wms007)

Description:

Revision 662009-03-16 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 74 to 74
 
  • Lease mechanism is not used
  • Use rpms from patch #2459
Added:
>
>

Partial Results (Mon Mar 16 10:05:55)

  • Collections correctly submitted: 2692 (107680 jobs)
    • DONE OK: 105122 (97.62%)
    • ABORTED: 3 (0.003%)
    • NotDone: 2555 (2.37%)
    • Resubmitted: 177 (0.16%)
 

13) Test starts on Wed Mar 4 at 13:41:28 CET 2009 (WMS: wms007)

Description:

Revision 652009-03-13 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 60 to 60
  * Submit a lot of jobs and restart ice daemon (i.e.: /opt/glite/etc/init.d/glite-wms-ice restart)

TESTs on ICE

Added:
>
>

14) Test starts on Fri Mar 13 at 11:44:57 CET 2009 (WMS: wms008)

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf and cream-21.pd)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Use rpms from patch #2459
 

13) Test starts on Wed Mar 4 at 13:41:28 CET 2009 (WMS: wms007)

Description:

Revision 642009-03-12 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 56 to 56
 
  • Bugs #47509: ICE must be modified in order to be compliant with modification to CEMon C++ API FIXED
    • Verify if the subscription of ICE to the CE works well (you need to look inside the log file of ICE)
Changed:
<
<
>
>
  • Bugs #47996: Apparent database corruption when ICE exits.FIXED * Submit a lot of jobs and restart ice daemon (i.e.: /opt/glite/etc/init.d/glite-wms-ice restart)
 

TESTs on ICE

Deleted:
<
<

14) Test starts on Wed Mar 11 at 11:19:16 CET 2009 (WMS: wms008)

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
  • Use rpms from patch #2459
 

13) Test starts on Wed Mar 4 at 13:41:28 CET 2009 (WMS: wms007)

Description:
  • 120 collections each of 60 jobs

Revision 632009-03-11 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 60 to 60
 

TESTs on ICE

Added:
>
>

14) Test starts on Wed Mar 11 at 11:19:16 CET 2009 (WMS: wms008)

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
  • Use rpms from patch #2459
 

13) Test starts on Wed Mar 4 at 13:41:28 CET 2009 (WMS: wms007)

Description:
  • 120 collections each of 60 jobs

Revision 622009-03-05 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 73 to 73
 

Test finishes on Wed Mar 4 at 15:38:14 CET 2009

  • 120 collections submitted in 1108 seconds: 4/9/16 (min/avg/max)
Changed:
<
<
>
>

Final results

  • Collections correctly submitted: 120 (7200 jobs)
    • DONE OK: 7198 (99.97%)
      • CREAM: 2537
      • LCG: 4661
    • ABORTED: 0 (0%)
    • Not finished: 2 (0.03%)
      • LCG: 2
    • Resubmitted: 182 (2.53%)
 

12) Test starts on Feb 24 at 10:29:07 CET 2009 (WMS: wms007)

Description:

Revision 612009-03-04 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 59 to 59
 

TESTs on ICE

Changed:
<
<

12) Test starts on Feb 24 at 10:29:07 (WMS: wms007)

>
>

13) Test starts on Wed Mar 4 at 13:41:28 CET 2009 (WMS: wms007)

  Description:
  • 120 collections each of 60 jobs
  • One collection every 60 seconds
Line: 69 to 70
 
  • We use both CREAM and LCG CEs
  • Long proxy
Added:
>
>

Test finishes on Wed Mar 4 at 15:38:14 CET 2009

  • 120 collections submitted in 1108 seconds: 4/9/16 (min/avg/max)

12) Test starts on Feb 24 at 10:29:07 CET 2009 (WMS: wms007)

Description:
  • 120 collections each of 60 jobs
  • One collection every 60 seconds
  • Four users
  • The job is a "sleep 313"
  • Resubmission is enabled
  • We use both CREAM and LCG CEs
  • Long proxy
 

Test finishes on Mon Feb 24 at 12:25:54 CET 2009

  • 92 collections submitted in 712 seconds: 4/7/13 (min/avg/max)
    • 28 submission(s) fail(s) (due to load limiter)

Revision 602009-03-04 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Changed:
<
<
  • Bugs #39807: In some circumstances, jobs which are killed by CREAM job wrapper might remain in ICE cache forever
>
>
  • Bugs #39807: In some circumstances, jobs which are killed by CREAM job wrapper might remain in ICE cache forever FIXED
    • Following the instructions reported in the bug's comments set:
      start_listener = false;
      start_subscription_updater = false;
      poller_delay = 900;
      poller_status_threshold_time = 60;
      in the Ice section of the configuration file (i.e. glite_wms.conf)
    • Submit a long job (i.e. a job that should run for more than 15 minutes), with a short proxy (i.e. a proxy with a lifetime of about 13 minutes).
    • Submit another job with a long proxy (i.e. more than an hour).
    • After about 17/18 minutes the original job should be ABORTED.
 
  • Bugs #42018: Missing exit on very severe error HOPEFULLY FIXED
    • Changes inside the code.

Revision 592009-02-27 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"
Added:
>
>

Check bugs:

  • Bugs #39807: In some circumstances, jobs which are killed by CREAM job wrapper might remain in ICE cache forever

  • Bugs #42018: Missing exit on very severe error HOPEFULLY FIXED
    • Changes inside the code.

  • Bugs #42081: Exception not catched in ICE HOPEFULLY FIXED
    • Changes inside the code.

  • Bugs #42141: Calling the FileList::get_size() method should be mutex protected HOPEFULLY FIXED
    • Changes inside the code.

  • Bugs #44604: A bad handling of delegations slow down dramatically the submission rate of ICE HOPEFULLY FIXED
    • Show tests below.

  • Bugs #46116: MaxOutputSandboxSize value not sent to CREAM by ICE FIXED
    • Set the parameter MaxOutputSandboxSize? in the WorkloadManager? section of the configuration file /opt/glite/etc/glite_wms.conf on the WMS to 100 and restart the workload manager.
    • Submit to a cream CE a jdl like this:
      [
      Type = "Job";
      Executable = "27215_exe.sh";
      Arguments = "70";
      StdOutput = "test.out";
      StdError = "test.err";
      InputSandbox = {"27215_exe.sh"};
      OutputSandbox = {"test.err","test.out","out2", "out1"};
      usertags = [ bug = "27215" ];
      ] 
      where 27215_exe.sh contains
      #!/bin/sh
      MAX=$1
      i=0
      while [ $i -lt $MAX ]; do
                      echo -n "1" >> out1
                      echo -n "2" >> out2
          i=$[$i + 1]
      done
      
    • Take the CreamJobID from the "Transfer Event" logged by the "LogMonitor" (i.e. The field Dest jobid)
    • Using the command of the client of the CE look inside the JDL sent to the ce: glite-ce-job-status -L 2 <CreamJobID>; you should find this parameter: maxOutputSandboxSize = 1.000000000000000E+02;
    • Due to a bug in CREAM the output files are not truncated as expected.

  • Bugs #47389: There's a mem leak in ICE that raises in some very rare circumstances HOPEFULLY FIXED
    • Not easy to reproduce

  • Bugs #47509: ICE must be modified in order to be compliant with modification to CEMon C++ API FIXED
    • Verify if the subscription of ICE to the CE works well (you need to look inside the log file of ICE)

 

TESTs on ICE

12) Test starts on Feb 24 at 10:29:07 (WMS: wms007)

Description:

Revision 582009-02-24 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

12) Test starts on Feb 24 at 10:29:07 (WMS: wms007)

Description:
  • 120 collections each of 60 jobs
  • One collection every 60 seconds
  • Four users
  • The job is a "sleep 313"
  • Resubmission is enabled
  • We use both CREAM and LCG CEs
  • Long proxy

Test finishes on Mon Feb 24 at 12:25:54 CET 2009

  • 92 collections submitted in 712 seconds: 4/7/13 (min/avg/max)
    • 28 submission(s) fail(s) (due to load limiter)

Final results

  • Collections correctly submitted: 91 (5460 jobs)
    • DONE OK: 5460 (100%)
      • CREAM: 3400
      • LCG: 2060
    • ABORTED: 0 (0%)
    • Resubmitted: 163 (2.99%)

  • The submission of one collection failed due to:
Status Reason:      LBProxy is enabled
Unable to query LB and LBProxy
edg_wll_QueryEvents[Proxy]
Exit code: 1413
LB[Proxy] Error: DNS resolver error
(edg_wll_gss_connect(): Unknown host)
 

11) Test starts on Mon Feb 23 at 15:25:49CET 2009 (WMS: wms007)

Description:

Revision 572009-02-24 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 14 to 14
 

Test finishes on Mon Feb 23 at 17:22:32 CET 2009

  • 120 collections submitted in 1092 seconds: 5/9/17 (min/avg/max)
Added:
>
>

Final results

  • Collections correctly submitted: 120 (7200 jobs)
    • DONE OK: 6950 (96.53%)
      • CREAM: 2243
      • LCG: 4707
    • ABORTED: 249 (3.46%)
      • LCG: 249
    • Not finished: 1 (0.01%)
      • LCG: 1
    • Resubmitted: 696 (9.7%)
 
Added:
>
>
  • All the jobs have been aborted for "proxy expired" because the job renewal daemon doesn't work.
 

10) Test starts on Tue Feb 5 at 12:41:35 CET 2009 (WMS: devel14)

Description:

Revision 562009-02-23 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 11 to 11
 
  • Resubmission is enabled
  • We use both CREAM and LCG CEs
Added:
>
>

Test finishes on Mon Feb 23 at 17:22:32 CET 2009

  • 120 collections submitted in 1092 seconds: 5/9/17 (min/avg/max)
 

Revision 552009-02-23 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

11) Test starts on Mon Feb 23 at 10:24:12 CET 2009 (WMS: wms007)

>
>

11) Test starts on Mon Feb 23 at 15:25:49CET 2009 (WMS: wms007)

  Description:
  • 120 collections each of 60 jobs
  • One collection every 60 seconds
Line: 11 to 11
 
  • Resubmission is enabled
  • We use both CREAM and LCG CEs
Changed:
<
<

Test finishes on Mon Feb 23 at 12:20:53 CET 2009

  • 120 collections submitted in 1000 seconds: 4/8/20 (min/avg/max)
>
>
 

10) Test starts on Tue Feb 5 at 12:41:35 CET 2009 (WMS: devel14)

Revision 542009-02-23 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 11 to 11
 
  • Resubmission is enabled
  • We use both CREAM and LCG CEs
Added:
>
>

Test finishes on Mon Feb 23 at 12:20:53 CET 2009

  • 120 collections submitted in 1000 seconds: 4/8/20 (min/avg/max)
 

10) Test starts on Tue Feb 5 at 12:41:35 CET 2009 (WMS: devel14)

Description:

Revision 532009-02-23 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

11) Test starts on Mon Feb 23 at 10:24:12 CET 2009 (WMS: wms007)

Description:
  • 120 collections each of 60 jobs
  • One collection every 60 seconds
  • Four users
  • The job is a "sleep 313"
  • Resubmission is enabled
  • We use both CREAM and LCG CEs
 

10) Test starts on Tue Feb 5 at 12:41:35 CET 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs

Revision 522009-02-10 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 21 to 21
 
      • Removed useless check of proxy duration in subscriptionManager, which could result in performance problems
Changed:
<
<

Partial results taken on Mon Feb 09 at 16:26:32 CET 2009

>
>

Final results taken on Thu Feb 10 at 16:20:32 CET 2009

 
  • Collections correctly submitted: 1568 (62720 jobs)
Changed:
<
<
    • DONE OK: 55351 (88.25%)
    • ABORTED: 4064 (6.48%)
    • Not finished: 3305 (5.27%)
    • Resubmissions: 39053 (62.27%)

  • Errors found (75677):
    • Cannot move ISB (69710 times 92.12%)
    • Cannot move OSB (100 times 0.13%)
    • Proxy is expired (4891 times 6.46%)
    • pbs_reason (628 times 0.83%)
      • pbs_reason=1; [...] proxy expired (437 times)
      • pbs_reason=271 (191 times)
    • Transfer to CREAM failed (186 times 0.25%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (186 times)
    • lsf_reason (157 times 0.21%)
      • lsf_reason=36608; Proxy expired: job killed Terminated Master process killed (117 times)
      • lsf_reason=256 (37 times)
>
>
    • DONE OK: 58641 (93.5%)
    • ABORTED: 4079 (6.5%)
    • Resubmitted: 41601 (66.33%)

  • Errors found (82530):
    • Cannot move ISB (75625 times 91.63%)
    • Cannot move OSB (115 times 0.14%)
    • Proxy is expired (5711 times 6.92%)
    • pbs_reason (702 times 0.85%)
      • pbs_reason=1; [...] proxy expired (477 times)
      • pbs_reason=271 (225 times)
    • Transfer to CREAM failed (187 times 0.23%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (187 times)
    • lsf_reason (184 times 0.23%)
      • lsf_reason=36608; Proxy expired: job killed Terminated Master process killed (138 times)
      • lsf_reason=256 (43 times)
 
      • lsf_reason=1603 (3 times)
Changed:
<
<
    • Cannot take token (3 times 0%)
>
>
    • Cannot take token (4 times 0%)
 
    • BLAH error (2 times 0%)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (2 times)
Changed:
<
<
  • Job Aborted (4064)
    • request expired (792 times 19.49%)
    • hit job shallow retry count (3) (3248 times 79.92%)
    • hit job retry count (2) (24 times 0.59%)
>
>
  • Job Aborted (4079)
    • request expired (792 times 19.42%)
    • hit job shallow retry count (3) (3263 times 80%)
    • hit job retry count (2) (24 times 0.58%)
  ice10.png

Revision 512009-02-09 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 21 to 21
 
      • Removed useless check of proxy duration in subscriptionManager, which could result in performance problems
Changed:
<
<

Partial results taken on Mon Feb 09 at 13:26:32 CET 2009

>
>

Partial results taken on Mon Feb 09 at 16:26:32 CET 2009

 
  • Collections correctly submitted: 1568 (62720 jobs)
Changed:
<
<
    • DONE OK: 53418 (85.17%)
>
>
    • DONE OK: 55351 (88.25%)
 
    • ABORTED: 4064 (6.48%)
Changed:
<
<
    • Not finished: 5238 (8.35%)
    • Resubmissions: 37120 (59.18%)
>
>
    • Not finished: 3305 (5.27%)
    • Resubmissions: 39053 (62.27%)
 
Changed:
<
<
  • Errors found (71149):
    • Cannot move ISB (65632 times 92.25%)
    • Cannot move OSB (90 times 0.13%)
    • Proxy is expired (4514 times 6.34%)
    • pbs_reason (590 times 0.83%)
      • pbs_reason=1; [...] proxy expired (416 times)
      • pbs_reason=271 (174 times)
    • Transfer to CREAM failed (184 times 0.26%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (184 times)
    • lsf_reason (134 times 0.19%)
      • lsf_reason=36608; Proxy expired: job killed Terminated Master process killed (96 times)
      • lsf_reason=256 (35 times)
>
>
  • Errors found (75677):
    • Cannot move ISB (69710 times 92.12%)
    • Cannot move OSB (100 times 0.13%)
    • Proxy is expired (4891 times 6.46%)
    • pbs_reason (628 times 0.83%)
      • pbs_reason=1; [...] proxy expired (437 times)
      • pbs_reason=271 (191 times)
    • Transfer to CREAM failed (186 times 0.25%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (186 times)
    • lsf_reason (157 times 0.21%)
      • lsf_reason=36608; Proxy expired: job killed Terminated Master process killed (117 times)
      • lsf_reason=256 (37 times)
 
      • lsf_reason=1603 (3 times)
    • Cannot take token (3 times 0%)
    • BLAH error (2 times 0%)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (2 times)
Added:
>
>
  • Job Aborted (4064)
    • request expired (792 times 19.49%)
    • hit job shallow retry count (3) (3248 times 79.92%)
    • hit job retry count (2) (24 times 0.59%)
 ice10.png

9) Test starts on Fri Jan 30 at 12:41:22 CET 2009 (WMS: devel14)

Revision 502009-02-09 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 21 to 21
 
      • Removed useless check of proxy duration in subscriptionManager, which could result in performance problems
Changed:
<
<

Partial results taken on Mon Feb 09 at 11:26:32 CET 2009

>
>

Partial results taken on Mon Feb 09 at 13:26:32 CET 2009

 
  • Collections correctly submitted: 1568 (62720 jobs)
Changed:
<
<
    • DONE OK: 37280 (59.44%)
    • ABORTED: 773 (1.23%)
    • Not finished: 24667 (39.33%)
    • Resubmissions: 20982 (33.45%)

  • Errors found (30097):
    • Cannot move ISB (27320 times 90.78%)
    • Cannot move OSB (60 times 0.2%)
    • Proxy is expired (2308 times 7.67%)
    • pbs_reason (241 times 0.8%)
      • pbs_reason=1; [...] proxy expired (180 times)
      • pbs_reason=271 (61 times)
    • Transfer to CREAM failed (103 times 1.97%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (103 times)
    • lsf_reason (64 times 0.21%)
      • lsf_reason=36608; Proxy expired: job killed Terminated Master process killed (42 times)
      • lsf_reason=256 (22 times)
    • Cannot take token (1 time 0%)
>
>
    • DONE OK: 53418 (85.17%)
    • ABORTED: 4064 (6.48%)
    • Not finished: 5238 (8.35%)
    • Resubmissions: 37120 (59.18%)

  • Errors found (71149):
    • Cannot move ISB (65632 times 92.25%)
    • Cannot move OSB (90 times 0.13%)
    • Proxy is expired (4514 times 6.34%)
    • pbs_reason (590 times 0.83%)
      • pbs_reason=1; [...] proxy expired (416 times)
      • pbs_reason=271 (174 times)
    • Transfer to CREAM failed (184 times 0.26%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (184 times)
    • lsf_reason (134 times 0.19%)
      • lsf_reason=36608; Proxy expired: job killed Terminated Master process killed (96 times)
      • lsf_reason=256 (35 times)
      • lsf_reason=1603 (3 times)
    • Cannot take token (3 times 0%)
    • BLAH error (2 times 0%)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (2 times)
  ice10.png
Line: 79 to 82
 
    • Resubmissions: 2012 (3.52%)

  • Errors found (2233):
Changed:
<
<
    • BLAH error (19 time 0.85%)
>
>
    • BLAH error (19 times 0.85%)
 
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-) N/A (jobId = [...]) (18 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)- exe_getouterr: poll() got an unknown event (stdout 0x0010 - stderr: 0x0000).-) N/A (jobId = [...]) (1 time)
    • Cannot move ISB (1872 times 83.84%)
Line: 454 to 457
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="ice10.png" attr="" comment="Test 10 Ice submission rate" date="1234178896" name="ice10.png" path="ice10.png" size="5733" stream="ice10.png" user="Main.AlessioGianelle" version="3"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Test 10 Ice submission rate" date="1234178896" name="ice10.png" path="ice10.png" size="5733" user="Main.AlessioGianelle" version="3"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 9 Ice submission rate" date="1233574400" name="ice9.png" path="ice9.png" size="5508" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 8 Ice submission rate" date="1233140819" name="ice8.png" path="ice8.png" size="6057" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"

Revision 492009-02-09 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 21 to 21
 
      • Removed useless check of proxy duration in subscriptionManager, which could result in performance problems
Changed:
<
<

Partial results taken on Thu Feb 06 at 15:26:32 CET 2009

  • Collections correctly submitted: 880 (35200 jobs)
    • DONE OK: 15475 (43.96%)
    • ABORTED: 4 (0.01%)
    • Not finished: 19721 (56.03%)
    • Resubmissions: 5128 (14.57%)

  • Errors found (5219):
    • Cannot move ISB (4554 times 87.26%)
    • Cannot move OSB (10 times 0.19%)
    • Proxy is expired (513 times 9.83%)
    • pbs_reason (37 times 0.71%)
      • pbs_reason=1; [...] proxy expired (31 times)
      • pbs_reason=271 (6 times)
>
>

Partial results taken on Mon Feb 09 at 11:26:32 CET 2009

  • Collections correctly submitted: 1568 (62720 jobs)
    • DONE OK: 37280 (59.44%)
    • ABORTED: 773 (1.23%)
    • Not finished: 24667 (39.33%)
    • Resubmissions: 20982 (33.45%)

  • Errors found (30097):
    • Cannot move ISB (27320 times 90.78%)
    • Cannot move OSB (60 times 0.2%)
    • Proxy is expired (2308 times 7.67%)
    • pbs_reason (241 times 0.8%)
      • pbs_reason=1; [...] proxy expired (180 times)
      • pbs_reason=271 (61 times)
 
    • Transfer to CREAM failed (103 times 1.97%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (103 times)
Changed:
<
<
    • lsf_reason (2 times 0.04%)
      • lsf_reason=36608; Proxy expired: job killed Terminated Master process killed (2 times)

  • All the jobs are aborted for "request expired"
>
>
    • lsf_reason (64 times 0.21%)
      • lsf_reason=36608; Proxy expired: job killed Terminated Master process killed (42 times)
      • lsf_reason=256 (22 times)
    • Cannot take token (1 time 0%)
  ice10.png
Line: 456 to 454
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Test 10 Ice submission rate" date="1233921923" name="ice10.png" path="ice10.png" size="4674" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attachment="ice10.png" attr="" comment="Test 10 Ice submission rate" date="1234178896" name="ice10.png" path="ice10.png" size="5733" stream="ice10.png" user="Main.AlessioGianelle" version="3"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 9 Ice submission rate" date="1233574400" name="ice9.png" path="ice9.png" size="5508" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 8 Ice submission rate" date="1233140819" name="ice8.png" path="ice8.png" size="6057" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"

Revision 482009-02-06 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 21 to 21
 
      • Removed useless check of proxy duration in subscriptionManager, which could result in performance problems
Changed:
<
<

Partial results taken on Thu Feb 06 at 10:26:32 CET 2009

>
>

Partial results taken on Thu Feb 06 at 15:26:32 CET 2009

 
  • Collections correctly submitted: 880 (35200 jobs)
Changed:
<
<
    • DONE OK: 12282 (34.89%)
    • ABORTED: 2 (0.006%)
    • Not finished: 22916 (65.1%)
    • Resubmissions: 1935 (5.5%)

  • Errors found (1974):
    • Cannot move ISB (1536 times 77.81%)
    • Cannot move OSB (8 times 0.41%)
    • Proxy is expired (320 times 16.21%)
    • pbs_reason (25 times 1.26%)
      • pbs_reason=1; [...] proxy expired (20 times)
      • pbs_reason=271 (5 times)
    • Transfer to CREAM failed (85 times 4.31%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (85 times)
>
>
    • DONE OK: 15475 (43.96%)
    • ABORTED: 4 (0.01%)
    • Not finished: 19721 (56.03%)
    • Resubmissions: 5128 (14.57%)

  • Errors found (5219):
    • Cannot move ISB (4554 times 87.26%)
    • Cannot move OSB (10 times 0.19%)
    • Proxy is expired (513 times 9.83%)
    • pbs_reason (37 times 0.71%)
      • pbs_reason=1; [...] proxy expired (31 times)
      • pbs_reason=271 (6 times)
    • Transfer to CREAM failed (103 times 1.97%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (103 times)
    • lsf_reason (2 times 0.04%)
      • lsf_reason=36608; Proxy expired: job killed Terminated Master process killed (2 times)
 
Changed:
<
<
  • The 2 jobs are aborted for "request expired"
>
>
  • All the jobs are aborted for "request expired"
 

Revision 472009-02-06 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 38 to 38
 
    • Transfer to CREAM failed (85 times 4.31%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (85 times)
Added:
>
>
  • The 2 jobs are aborted for "request expired"

 ice10.png

9) Test starts on Fri Jan 30 at 12:41:22 CET 2009 (WMS: devel14)

Line: 450 to 454
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="ice10.png" attr="" comment="Test 10 Ice submission rate" date="1233921923" name="ice10.png" path="ice10.png" size="4674" stream="ice10.png" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Test 10 Ice submission rate" date="1233921923" name="ice10.png" path="ice10.png" size="4674" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 9 Ice submission rate" date="1233574400" name="ice9.png" path="ice9.png" size="5508" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 8 Ice submission rate" date="1233140819" name="ice8.png" path="ice8.png" size="6057" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"

Revision 462009-02-06 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 450 to 450
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attachment="ice10.png" attr="" comment="Test 10 Ice submission rate" date="1233921923" name="ice10.png" path="ice10.png" size="4674" stream="ice10.png" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 9 Ice submission rate" date="1233574400" name="ice9.png" path="ice9.png" size="5508" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 8 Ice submission rate" date="1233140819" name="ice8.png" path="ice8.png" size="6057" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"
Deleted:
<
<
META FILEATTACHMENT attachment="ice10.png" attr="" comment="Test 10 Ice submission rate" date="1233914811" name="ice10.png" path="ice10.png" size="5259" stream="ice10.png" user="Main.AlessioGianelle" version="1"

Revision 452009-02-06 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 20 to 20
 
      • Fix a problem with proxy renewal seen in the previous test
      • Removed useless check of proxy duration in subscriptionManager, which could result in performance problems
Added:
>
>

Partial results taken on Thu Feb 06 at 10:26:32 CET 2009

  • Collections correctly submitted: 880 (35200 jobs)
    • DONE OK: 12282 (34.89%)
    • ABORTED: 2 (0.006%)
    • Not finished: 22916 (65.1%)
    • Resubmissions: 1935 (5.5%)

  • Errors found (1974):
    • Cannot move ISB (1536 times 77.81%)
    • Cannot move OSB (8 times 0.41%)
    • Proxy is expired (320 times 16.21%)
    • pbs_reason (25 times 1.26%)
      • pbs_reason=1; [...] proxy expired (20 times)
      • pbs_reason=271 (5 times)
    • Transfer to CREAM failed (85 times 4.31%)
      • due to exception: CREAM Register raised std::exception Connection to service [...] failed: (85 times)

ice10.png

 

9) Test starts on Fri Jan 30 at 12:41:22 CET 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs
Line: 47 to 67
 
  • 1427 collections submitted in 29752 seconds: 5/20/90 (min/avg/max)
    • 2893 submissions fail due to load limiter
Changed:
<
<

Partial results taken on Thu Feb 02 at 15:24:31 CET 2009

>
>

Results taken on Thu Feb 02 at 15:24:31 CET 2009

 
  • Collections correctly submitted: 1427 (57080 jobs)
    • DONE OK: 28244 (49.48%)
    • ABORTED: 0 (0%)
Line: 65 to 84
 
      • lsf_reason=36608 (1 time)
    • pbs_reason (16 times 0.72%)
      • pbs_reason=1; [...] proxy expired (15 times)
Changed:
<
<
      • pbs_reason=271; Proxy expired: job killed Terminated Master process killed (1 times)
>
>
      • pbs_reason=271; Proxy expired: job killed Terminated Master process killed (1 time)
  ice9.png
Line: 435 to 454
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 8 Ice submission rate" date="1233140819" name="ice8.png" path="ice8.png" size="6057" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attachment="ice10.png" attr="" comment="Test 10 Ice submission rate" date="1233914811" name="ice10.png" path="ice10.png" size="5259" stream="ice10.png" user="Main.AlessioGianelle" version="1"

Revision 442009-02-05 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

10) Test starts on Tue Feb 4 at 14:56:54 CET 2009 (WMS: devel14)

>
>

10) Test starts on Tue Feb 5 at 12:41:35 CET 2009 (WMS: devel14)

  Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds

Revision 432009-02-05 - MassimoSgaravatto

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 18 to 18
 
  • Changes in the software wrt previous test:
    • ICE
      • Fix a problem with proxy renewal seen in the previous test
Added:
>
>
      • Removed useless check of proxy duration in subscriptionManager, which could result in performance problems
 

9) Test starts on Fri Jan 30 at 12:41:22 CET 2009 (WMS: devel14)

Description:

Revision 422009-02-04 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

10) Test starts on Tue Feb 4 at 10:22:23 CET 2009 (WMS: devel14)

>
>

10) Test starts on Tue Feb 4 at 14:56:54 CET 2009 (WMS: devel14)

  Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds

Revision 412009-02-04 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

10) Test starts on Tue Feb 3 at 17:12:21 CET 2009 (WMS: devel14)

>
>

10) Test starts on Tue Feb 4 at 10:22:23 CET 2009 (WMS: devel14)

  Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds

Revision 402009-02-03 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

10) Test starts on Tue Feb 3 at 14:57:25 CET 2009 (WMS: devel14)

>
>

10) Test starts on Tue Feb 3 at 17:12:21 CET 2009 (WMS: devel14)

  Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds

Revision 392009-02-03 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

10) Test starts on Mon Feb 2 at 17:57:57 CET 2009 (WMS: devel14)

>
>

10) Test starts on Tue Feb 3 at 14:57:25 CET 2009 (WMS: devel14)

  Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds

Revision 382009-02-03 - MassimoSgaravatto

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 14 to 14
 
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used
Changed:
<
<
>
>
  • Changes in the software wrt previous test:
    • ICE
      • Fix a problem with proxy renewal seen in the previous test
 

9) Test starts on Fri Jan 30 at 12:41:22 CET 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs
Line: 35 to 39
 
    • ICE
      • Fix for bug #46405
      • 5 sec. (instead of 60) of delay between two LB logging tries
Added:
>
>
      • Error code is printed in the ICE log file when a log to LB fails
 

Test finishes on Mon Feb 2 at 12:39:09 CET 2009

Revision 372009-02-02 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

10) Test starts on Mon Feb 2 at 17:57:57 CET 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf) plus cream-04.pd.infn.it
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used
 

9) Test starts on Fri Jan 30 at 12:41:22 CET 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs

Revision 362009-02-02 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 28 to 28
 
  • 1427 collections submitted in 29752 seconds: 5/20/90 (min/avg/max)
    • 2893 submissions fail due to load limiter
Changed:
<
<

Partial results taken on Thu Feb 02 at 11:05:31 CET 2009

>
>

Partial results taken on Thu Feb 02 at 15:24:31 CET 2009

 
Changed:
<
<
  • Collections correctly submitted: 1402 (56080 jobs)
    • DONE OK: 26903 (47.97%)
>
>
  • Collections correctly submitted: 1427 (57080 jobs)
    • DONE OK: 28244 (49.48%)
 
    • ABORTED: 0 (0%)
Changed:
<
<
    • Not finished: 29177 (52.03%)
    • Resubmissions: 1775 (3.17%)
>
>
    • Not finished: 28836 (50.52%)
    • Resubmissions: 2012 (3.52%)
 
Changed:
<
<
  • Errors found (1966):
    • BLAH error (18 time 0.92%)
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-) N/A (jobId = [...]) (17 times)
>
>
  • Errors found (2233):
    • BLAH error (19 time 0.85%)
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-) N/A (jobId = [...]) (18 times)
 
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)- exe_getouterr: poll() got an unknown event (stdout 0x0010 - stderr: 0x0000).-) N/A (jobId = [...]) (1 time)
Changed:
<
<
    • Cannot move ISB (1662 times 84.54%)
    • Proxy is expired (270 times 13.73%)
    • lsf_reason (1 time 0.05%)
>
>
    • Cannot move ISB (1872 times 83.84%)
    • Proxy is expired (325 times 14.55%)
    • lsf_reason (1 time 0.04%)
 
      • lsf_reason=36608 (1 time)
Changed:
<
<
    • pbs_reason (15 times 0.76%)
      • pbs_reason=1; [...] proxy expired (14 times)
>
>
    • pbs_reason (16 times 0.72%)
      • pbs_reason=1; [...] proxy expired (15 times)
 
      • pbs_reason=271; Proxy expired: job killed Terminated Master process killed (1 times)

ice9.png

Revision 352009-02-02 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 14 to 14
 
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used
Added:
>
>
 
  • Changes in the software wrt previous test:
    • CEs:
      • Fix for bug #45913 (only on cream-04.pd.infn.it)
Line: 21 to 22
 
    • ICE
      • Fix for bug #46405
      • 5 sec. (instead of 60) of delay between two LB logging tries
Added:
>
>

Test finishes on Mon Feb 2 at 12:39:09 CET 2009

  • 1427 collections submitted in 29752 seconds: 5/20/90 (min/avg/max)
    • 2893 submissions fail due to load limiter

Partial results taken on Thu Feb 02 at 11:05:31 CET 2009

  • Collections correctly submitted: 1402 (56080 jobs)
    • DONE OK: 26903 (47.97%)
    • ABORTED: 0 (0%)
    • Not finished: 29177 (52.03%)
    • Resubmissions: 1775 (3.17%)

  • Errors found (1966):
    • BLAH error (18 time 0.92%)
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-) N/A (jobId = [...]) (17 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)- exe_getouterr: poll() got an unknown event (stdout 0x0010 - stderr: 0x0000).-) N/A (jobId = [...]) (1 time)
    • Cannot move ISB (1662 times 84.54%)
    • Proxy is expired (270 times 13.73%)
    • lsf_reason (1 time 0.05%)
      • lsf_reason=36608 (1 time)
    • pbs_reason (15 times 0.76%)
      • pbs_reason=1; [...] proxy expired (14 times)
      • pbs_reason=271; Proxy expired: job killed Terminated Master process killed (1 times)

ice9.png

 

8) Test starts on Mon Jan 26 17:59:02 CET 2009 (WMS: devel14)

Description:
Line: 34 to 62
 
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
 
  • Changes in the software wrt previous test:
    • ICE:
      • Fixed problem with proxy renewal seen in previous test
Added:
>
>
  Test interrupted for a problem in the proxy-renewal service daemon on Thu Jan 29 12:05:12 CET 2009
Line: 90 to 120
 
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
 
  • Changes in the software wrt previous test:
    • ICE:
      • Fixed problem seen in previous test
Added:
>
>
  Test interrupted on Mon Jan 26 17:05:12 CET 2009
Line: 109 to 141
 
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (1 time)
      • submission command failed (exit code = 1) (stdout:) (stderr:Cannot resolve default server host 'cream-28.pd.infn.it' - check server_name file.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15008)-) N/A (jobId = [...]) (1 time)
    • Cannot take token (77 times 0.42%)
Changed:
<
<
    • Cannot move ISB (14222 time 78.07%)
>
>
    • Cannot move ISB (14222 time 78.07%)
 
    • Cannot move OSB (82 times 0.45%)
    • Transfer to CREAM failed (4 times 0.02%)
      • due to exception: Authentication error: Unable to open the file [/var/glite/SandboxDir/[...]/user.proxy] : No such file or directory (4 times)
Line: 135 to 167
 
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
 
  • Changes in the software wrt previous test:
    • CEs:
      • Fix for bug #45718
Line: 142 to 175
 
      • Fix for bug #46024
    • ICE
      • Use of same delegationid if CREAM complains that it doesn't exist anymore
Changed:
<
<

>
>
  Test aborted on Fri Jan 23 10:30:12 CET 2009
Line: 160 to 192
 
  • Resubmission is able
  • Lease mechanism is not used
Changed:
<
<

Test finish on Wed Jan 21 at 17:44:58 CET 2009

>
>

Test finishes on Wed Jan 21 at 17:44:58 CET 2009

 
  • 224 collections submitted in 7416 seconds: 64/33/6 (max/avg/min)
Changed:
<
<
    • 76 submissions fails due to load limiter
>
>
    • 76 submissions fail due to load limiter
 
  • Collections correctly submitted: 224 (17920 jobs)
    • DONE OK: 17850 (99.6%)
Line: 190 to 221
 
  • Resubmission is able
  • Lease mechanism is not used
Changed:
<
<

Test finish on Tue Jan 20 at 15:13:14 CET 2009

>
>

Test finishes on Tue Jan 20 at 15:13:14 CET 2009

 
  • 197 collections submitted in 8499 seconds: 84/43/9 (max/avg/min)
Changed:
<
<
    • 103 submissions fails due to load limiter
>
>
    • 103 submissions fail due to load limiter
 
  • Collections correctly submitted: 197 (15760 jobs)
    • DONE OK: 15695 (99.6%)
Line: 218 to 249
 
  • Resubmission is enabled
  • Lease mechanism is not used

Deleted:
<
<
 Test has been modified on Mon Jan 19 at 17:03:41:
  • 1440 collections each of 80 jobs
  • One collection every 60 seconds
Line: 238 to 267
 
    • Transfer to CREAM failed (50 times 4.25%)
      • Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (7 times)
      • Transfer to CREAM failed due to exception: Authentication error: The proxy is EXPIRED! (43 times)
Changed:
<
<
    • lsf_reason=32512 (1085 times 92.27%)
>
>
    • lsf_reason=32512 (1085 times 92.27%)
 
    • lsf_reason=306 (1 time 0.08%)

ice3.png

Line: 255 to 284
 
  • The job is a "sleep 313"
  • Resubmission is enabled
  • Lease mechanism is not used
Added:
>
>
 
  • Changes in the software wrt previous test:
    • CEs:
      • Fix for bug #45437
Line: 262 to 292
 
    • ICE
      • Management of serialization error
      • Renewal done at 80 % of lifetime of proxy (or when there are only 20 minutes left)
Added:
>
>
 

Test finishes on Sun Jan 18 at 15:42:28 CET 2009

  • 7180 collections submitted in 70789 seconds: 141/9/3 (max/avg/min)
Changed:
<
<
    • 20 submissions fails due to load limiter
>
>
    • 20 submissions fail due to load limiter
 
  • Collections correctly submitted: 7180 ( 287200 jobs)
    • DONE OK: 284838 (99.18%)
Line: 287 to 318
 
    • Transfer to CREAM failed (19 times 0.41%)
      • FaultCause=[The problem seems to be related to glexec which reported: java.io.IOException: Too many open files]" (10 times)
      • CREAM Register raised std::exception Connection to service [https://cert-xx.cnaf.infn.it:8443/ce-cream/services/CREAM2] failed: (9 times)
Changed:
<
<
    • lsf_reason=32512 (3505 times 76.22%)
>
>
    • lsf_reason=32512 (3505 times 76.22%)
 
    • Proxy is expired (1 time 0.02%)
    • lsf_reason=306 (1 time 0.02%)
Line: 381 to 412
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Test 9 Ice submission rate" date="1233574400" name="ice9.png" path="ice9.png" size="5508" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 8 Ice submission rate" date="1233140819" name="ice8.png" path="ice8.png" size="6057" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"

Revision 342009-02-02 - MassimoSgaravatto

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 12 to 12
 
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch";)
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
Changed:
<
<
  • Resubmission is able
>
>
  • Resubmission is enabled
 
  • Lease mechanism is not used
Added:
>
>
  • Changes in the software wrt previous test:
    • CEs:
      • Fix for bug #45913 (only on cream-04.pd.infn.it)
      • Fix for bug #46283 (only on cream-04.pd.infn.it and cert-04.pd.infn.it)
    • ICE
      • Fix for bug #46405
      • 5 sec. (instead of 60) of delay between two LB logging tries
 

8) Test starts on Mon Jan 26 17:59:02 CET 2009 (WMS: devel14)

Description:
Line: 27 to 34
 
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
  • Changes in the software wrt previous test:
    • ICE:
      • Fixed problem with proxy renewal seen in previous test
  Test interrupted for a problem in the proxy-renewal service daemon on Thu Jan 29 12:05:12 CET 2009
Line: 80 to 90
 
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
  • Changes in the software wrt previous test:
    • ICE:
      • Fixed problem seen in previous test
  Test interrupted on Mon Jan 26 17:05:12 CET 2009
Line: 122 to 135
 
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
  • Changes in the software wrt previous test:
    • CEs:
      • Fix for bug #45718
      • Fix for bug #45983
      • Fix for bug #46024
    • ICE
      • Use of same delegationid if CREAM complains that it doesn't exist anymore

  Test aborted on Fri Jan 23 10:30:12 CET 2009
Line: 193 to 215
 
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
Changed:
<
<
  • Resubmission is able
>
>
  • Resubmission is enabled
 
  • Lease mechanism is not used
Added:
>
>
 Test has been modified on Mon Jan 19 at 17:03:41:
  • 1440 collections each of 80 jobs
  • One collection every 60 seconds
Line: 229 to 253
 
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
Changed:
<
<
  • Resubmission is able
>
>
  • Resubmission is enabled
 
  • Lease mechanism is not used
Added:
>
>
  • Changes in the software wrt previous test:
    • CEs:
      • Fix for bug #45437
      • Fix for bug #45736
    • ICE
      • Management of serialization error
      • Renewal done at 80 % of lifetime of proxy (or when there are only 20 minutes left)
 

Test finishes on Sun Jan 18 at 15:42:28 CET 2009

  • 7180 collections submitted in 70789 seconds: 141/9/3 (max/avg/min)

Revision 332009-01-30 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

9) Test starts on Fri Jan 30 at 12:41:22 CET 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf) plus cream-04.pd.infn.it
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch";)
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
 

8) Test starts on Mon Jan 26 17:59:02 CET 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs
Line: 15 to 28
 
  • Resubmission is able
  • Lease mechanism is not used
Changed:
<
<

Partial results taken on Wed Jan 28 at 12:20:31 CET 2009

  • Collections correctly submitted: 1400 (56000 jobs)
    • DONE OK: 30128 (53.8%)
    • ABORTED: 0 (0%)
    • Not finished: 25872 (46.2%)
    • Resubmissions: 30 (0.05%)
>
>
Test interrupted for a problem in the proxy-renewal service daemon on Thu Jan 29 12:05:12 CET 2009

Results taken on Thu Jan 29 at 18:35:31 CET 2009

  • Collections correctly submitted: 1433 (57320 jobs)
    • DONE OK: 33566 (58.56%)
    • ABORTED: 8414 (14.68%)
    • Not finished: 15340 (26.76%)
    • Resubmissions: 33507 (58.46%)
 
Changed:
<
<
  • Errors found (30):
    • BLAH error (1 time)
>
>
  • Errors found (33957):
    • BLAH error (3 time 0.01%)
 
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (1 time)
Changed:
<
<
    • Cannot move ISB (2 times)
    • lsf_reason (27 times)
      • lsf_reason=65280 (27 times)
>
>
      • submission command failed (exit code = 106) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) (2 times)
    • Cannot move ISB (4513 times 13.29%)
    • Cannot move OSB (73 times 0.21%)
    • Transfer to CREAM failed (28607 times 84.24%)
      • due to exception: Authentication error: The proxy is EXPIRED! (28134 times)
      • due to exception: Authentication error: Unable to open the file [/var/glite/SandboxDir/[...]/user.proxy] : No such file or directory (432 times)
      • Failed to create a delegation id for job [...]: reason is Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (19 times)
      • Failed to create a delegation id for job [...]: reason is Failed proxy validation - it has expired. (4 times)
      • Failed to create a delegation id for job [...]: reason is CreamProxy_Delegate::execute() - Coundl't open proxyfile [...]: The proxy is EXPIRED! (1 time)
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (12 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Wed 28 Jan 2009 17:42:42] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]" (5 times)
    • Proxy is expired (688 times 2.03%)
    • lsf_reason (61 times 0.18%)
      • lsf_reason=65280 (32 times)
      • lsf_reason=36608 (22 times)
      • lsf_reason=1603 (1 time)
      • lsf_reason=256 (6 times)
    • pbs_reason (12 times 0.04%)
      • pbs_reason=271; Proxy expired: job killed Terminated Master process killed (12 times)
  ice8.png
Added:
>
>

BUGS:

  • CREAM
    • #46405: VOMSWrapper should try more than once to open a proxy file
  • BLAH
    • #46283: Possible memory leak in strtoken function for BLParser
 

7) Test starts on Fri Jan 23 12:28:01 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs

Revision 322009-01-28 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 18 to 18
 

Partial results taken on Wed Jan 28 at 12:20:31 CET 2009

  • Collections correctly submitted: 1400 (56000 jobs)
    • DONE OK: 30128 (53.8%)
Changed:
<
<
    • ABORTED: 0 (0.06%)
>
>
    • ABORTED: 0 (0%)
 
    • Not finished: 25872 (46.2%)
    • Resubmissions: 30 (0.05%)

Revision 312009-01-28 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 15 to 15
 
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>

Partial results taken on Wed Jan 28 at 12:20:31 CET 2009

  • Collections correctly submitted: 1400 (56000 jobs)
    • DONE OK: 30128 (53.8%)
    • ABORTED: 0 (0.06%)
    • Not finished: 25872 (46.2%)
    • Resubmissions: 30 (0.05%)

  • Errors found (30):
    • BLAH error (1 time)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (1 time)
    • Cannot move ISB (2 times)
    • lsf_reason (27 times)
      • lsf_reason=65280 (27 times)
 ice8.png

7) Test starts on Fri Jan 23 12:28:01 CET 2009 (WMS: devel14)

Line: 299 to 313
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="ice8.png" attr="" comment="Test 8 Ice submission rate" date="1233140818" name="ice8.png" path="ice8.png" size="6057" stream="ice8.png" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Test 8 Ice submission rate" date="1233140819" name="ice8.png" path="ice8.png" size="6057" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"

Revision 302009-01-28 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 299 to 299
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attachment="ice8.png" attr="" comment="Test 8 Ice submission rate" date="1233140818" name="ice8.png" path="ice8.png" size="6057" stream="ice8.png" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"
Deleted:
<
<
META FILEATTACHMENT attachment="ice8.png" attr="" comment="Test 8 Ice submission rate" date="1233055916" name="ice8.png" path="ice8.png" size="6427" stream="ice8.png" user="Main.AlessioGianelle" version="1"

Revision 292009-01-27 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

8) Test starts on Mon Jan 26 17:59:02 CET 2009 (WMS: devel14)

Description:
Changed:
<
<
  • 4320 collections each of 40 jobs
>
>
  • 4320 collections each of 40 jobs
 
  • One collection every 60 seconds
Changed:
<
<
  • Five users
>
>
  • Five users
 
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
Changed:
<
<
  • The job is a "sleep 4242"
>
>
  • The job is a "sleep 4242"
 
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
ice8.png
 

7) Test starts on Fri Jan 23 12:28:01 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs
Line: 299 to 301
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attachment="ice8.png" attr="" comment="Test 8 Ice submission rate" date="1233055916" name="ice8.png" path="ice8.png" size="6427" stream="ice8.png" user="Main.AlessioGianelle" version="1"

Revision 282009-01-27 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 30 to 30
  Test interrupted on Mon Jan 26 17:05:12 CET 2009
Changed:
<
<

Partial Results taken on Mon Jan 26 at 11:24:31 CET 2009

>
>

Results taken on Mon Jan 26 at 18:20:31 CET 2009

 
  • Collections correctly submitted: 1741 (69640 jobs)
Changed:
<
<
    • DONE OK: 47459 (77.85%)
    • ABORTED: 44 (0.07%)
    • Not finished: 13457 (22.08%)
    • Resubmissions: 16139 (26.47%)
>
>
    • DONE OK: 51894 (74.52%)
    • ABORTED: 44 (0.06%)
    • Not finished: 16186 (23.24%)
    • CANCELLED: 1516 (2.18%)
    • Resubmissions: 18218 (26.16%)
 
Changed:
<
<
  • Errors found (16139):
>
>
  • Errors found (18218):
 
    • BLAH error (2 times 0.01%)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (1 time)
      • submission command failed (exit code = 1) (stdout:) (stderr:Cannot resolve default server host 'cream-28.pd.infn.it' - check server_name file.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15008)-) N/A (jobId = [...]) (1 time)
Changed:
<
<
    • Cannot take token (71 times 0.44%)
    • Cannot move ISB (12481 time 77.33%)
    • Cannot move OSB (69 times 0.43%)
    • Transfer to CREAM failed (4 times 0.03%)
>
>
    • Cannot take token (77 times 0.42%)
    • Cannot move ISB (14222 time 78.07%)
    • Cannot move OSB (82 times 0.45%)
    • Transfer to CREAM failed (4 times 0.02%)
 
      • due to exception: Authentication error: Unable to open the file [/var/glite/SandboxDir/[...]/user.proxy] : No such file or directory (4 times)
Changed:
<
<
    • lsf_reason (24 times 0.15%)
      • lsf_reason=36608 (13 times)
      • lsf_reason=256 (11 times)
    • Proxy is expired (2803 times 17.37%)
    • pbs_reason (685 times 4.24%)
      • pbs_reason=-1 (654 times)
      • pbs_reason=271; Proxy expired: job killed Terminated Master process killed (31 times)
>
>
    • lsf_reason (30 times 0.16%)
      • lsf_reason=36608 (17 times)
      • lsf_reason=256 (13 times)
    • Proxy is expired (3113 times 17.09%)
    • pbs_reason (688 times 3.78%)
      • pbs_reason=-1 (656 times)
      • pbs_reason=271; Proxy expired: job killed Terminated Master process killed (32 times)
  ice7.png

Revision 272009-01-26 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

8) Test starts on Mon Jan 26 17:59:02 CET 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
 

7) Test starts on Fri Jan 23 12:28:01 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs
Line: 15 to 28
 
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
Test interrupted on Mon Jan 26 17:05:12 CET 2009
 

Partial Results taken on Mon Jan 26 at 11:24:31 CET 2009

Changed:
<
<
  • Collections correctly submitted: 1524 (60960 jobs)
>
>
  • Collections correctly submitted: 1741 (69640 jobs)
 
    • DONE OK: 47459 (77.85%)
    • ABORTED: 44 (0.07%)
    • Not finished: 13457 (22.08%)
Line: 281 to 296
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Test 7 Ice submission rate" date="1232989691" name="ice7.png" path="ice7.png" size="5943" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"
Deleted:
<
<
META FILEATTACHMENT attachment="ice7.png" attr="" comment="Test 7 Ice submission rate" date="1232973750" name="ice7.png" path="ice7.png" size="6416" stream="ice7.png" user="Main.AlessioGianelle" version="1"

Revision 262009-01-26 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 15 to 15
 
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>

Partial Results taken on Mon Jan 26 at 11:24:31 CET 2009

  • Collections correctly submitted: 1524 (60960 jobs)
    • DONE OK: 47459 (77.85%)
    • ABORTED: 44 (0.07%)
    • Not finished: 13457 (22.08%)
    • Resubmissions: 16139 (26.47%)

  • Errors found (16139):
    • BLAH error (2 times 0.01%)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (1 time)
      • submission command failed (exit code = 1) (stdout:) (stderr:Cannot resolve default server host 'cream-28.pd.infn.it' - check server_name file.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15008)-) N/A (jobId = [...]) (1 time)
    • Cannot take token (71 times 0.44%)
    • Cannot move ISB (12481 time 77.33%)
    • Cannot move OSB (69 times 0.43%)
    • Transfer to CREAM failed (4 times 0.03%)
      • due to exception: Authentication error: Unable to open the file [/var/glite/SandboxDir/[...]/user.proxy] : No such file or directory (4 times)
    • lsf_reason (24 times 0.15%)
      • lsf_reason=36608 (13 times)
      • lsf_reason=256 (11 times)
    • Proxy is expired (2803 times 17.37%)
    • pbs_reason (685 times 4.24%)
      • pbs_reason=-1 (654 times)
      • pbs_reason=271; Proxy expired: job killed Terminated Master process killed (31 times)

ice7.png

 

6) Test starts on Thu Jan 22 at 17:17:38 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs
Line: 256 to 282
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attachment="ice7.png" attr="" comment="Test 7 Ice submission rate" date="1232973750" name="ice7.png" path="ice7.png" size="6416" stream="ice7.png" user="Main.AlessioGianelle" version="1"

Revision 252009-01-23 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

7) Test starts on Fri Jan 23 12:28:01 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
 

6) Test starts on Thu Jan 22 at 17:17:38 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs
Line: 15 to 28
 
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
Test aborted on Fri Jan 23 10:30:12 CET 2009
 

5) Test starts on Wed Jan 21 at 12:45:49 CET 2009 (WMS: devel14)

Description:
  • 300 collections each of 80 jobs

Revision 242009-01-22 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

6) Test starts on Thu Jan 22 at 17:17:38 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is able
  • Lease mechanism is not used
 

5) Test starts on Wed Jan 21 at 12:45:49 CET 2009 (WMS: devel14)

Description:
Changed:
<
<
  • 300 collections each of 80 jobs
>
>
  • 300 collections each of 80 jobs
 
  • One collection every 60 seconds
  • One user
  • Set max_ice_threads = 40;
Line: 225 to 238
 -- AlessioGianelle - 08 Jan 2009
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"
Deleted:
<
<
META FILEATTACHMENT attachment="ice5.png" attr="" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" stream="ice5.png" user="Main.AlessioGianelle" version="1"

Revision 232009-01-22 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 21 to 21
 
    • 76 submissions fails due to load limiter

  • Collections correctly submitted: 224 (17920 jobs)
Added:
>
>
    • DONE OK: 17850 (99.6%)
    • ABORTED: 0 (0.0%)
    • Not finished: 70 (0.4%)
    • Resubmissions: 8 (0.04%)

  • Errors found (8):
    • Cannot move ISB (3 times)
    • lsf_reason=1603 (3 times)
    • BLAH error (2 times)

ice5.png

 

4) Test starts on Tue Jan 20 at 10:09:58 CET 2009 (WMS: devel14)

Description:
Line: 216 to 227
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attachment="ice5.png" attr="" comment="Test 5 Ice submission rate" date="1232613712" name="ice5.png" path="ice5.png" size="6111" stream="ice5.png" user="Main.AlessioGianelle" version="1"

Revision 222009-01-21 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 15 to 15
 
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>

Test finish on Wed Jan 21 at 17:44:58 CET 2009

  • 224 collections submitted in 7416 seconds: 64/33/6 (max/avg/min)
    • 76 submissions fails due to load limiter

  • Collections correctly submitted: 224 (17920 jobs)
 

4) Test starts on Tue Jan 20 at 10:09:58 CET 2009 (WMS: devel14)

Description:
  • 300 collections each of 80 jobs

Revision 212009-01-21 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

4) Test starts on Tue Jan 20 at 10:09:58 CET 2009 (WMS: devel14)

>
>

5) Test starts on Wed Jan 21 at 12:45:49 CET 2009 (WMS: devel14)

  Description:
  • 300 collections each of 80 jobs
  • One collection every 60 seconds
  • One user
Added:
>
>
  • Set max_ice_threads = 40;
  • Used the CEs of testbedB (only PD)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
  • Resubmission is able
  • Lease mechanism is not used

4) Test starts on Tue Jan 20 at 10:09:58 CET 2009 (WMS: devel14)

Description:
  • 300 collections each of 80 jobs
  • One collection every 60 seconds
  • One user
 
  • Used the lsf CEs of testbedB (PD+CNAF) (cert-06 at cnaf is not considered)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
Line: 33 to 46
 

3) Test starts on Mon Jan 19 at 15:22:51 CET 2009 (WMS: devel14)

Description:
  • 2880 collections each of 40 jobs
Changed:
<
<
  • One collection every 30 seconds
>
>
  • One collection every 30 seconds
 
  • One user
  • Used the lsf CEs of testbedB (PD+CNAF)
  • Used automatic-delegation and proxy renewal service
Line: 76 to 89
 
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
  • Resubmission is able
Changed:
<
<
  • Lease mechanism is not used
>
>
  • Lease mechanism is not used
 

Test finishes on Sun Jan 18 at 15:42:28 CET 2009

  • 7180 collections submitted in 70789 seconds: 141/9/3 (max/avg/min)
Line: 194 to 207
 -- AlessioGianelle - 08 Jan 2009
Changed:
<
<
META FILEATTACHMENT attachment="ice4.png" attr="" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" stream="ice4.png" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"

Revision 202009-01-21 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 194 to 194
 -- AlessioGianelle - 08 Jan 2009
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232468058" name="ice4.png" path="ice4.png" size="4599" user="Main.AlessioGianelle" version="1"
>
>
META FILEATTACHMENT attachment="ice4.png" attr="" comment="Test4 Ice submission rate" date="1232532928" name="ice4.png" path="ice4.png" size="6540" stream="ice4.png" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"

Revision 192009-01-21 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 19 to 19
 
    • 103 submissions fails due to load limiter

  • Collections correctly submitted: 197 ( 15760 jobs)
Added:
>
>
    • DONE OK: 15695 (99.6%)
    • ABORTED: 0 (0.0%)
    • Not finished: 65 (0.4%)
    • Resubmissions: 3 (0.02%)

  • Errors found (3):
    • Cannot move OSB (1 time)
    • Cannot move ISB (2 times)
  ice4.png
Line: 187 to 195
 

META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232468058" name="ice4.png" path="ice4.png" size="4599" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="ice3.png" attr="" comment="Test 3 Ice submission rate" date="1232468217" name="ice3.png" path="ice3.png" size="4569" stream="ice3.png" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Test 3 Ice submission rate" date="1232468218" name="ice3.png" path="ice3.png" size="4569" user="Main.AlessioGianelle" version="2"

Revision 182009-01-20 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 20 to 20
 
  • Collections correctly submitted: 197 ( 15760 jobs)
Added:
>
>
ice4.png
 

3) Test starts on Mon Jan 19 at 15:22:51 CET 2009 (WMS: devel14)

Description:
  • 2880 collections each of 40 jobs
Line: 184 to 186
 -- AlessioGianelle - 08 Jan 2009
Changed:
<
<
META FILEATTACHMENT attachment="ice3.png" attr="" comment="Ice submission rate" date="1232453586" name="ice3.png" path="ice3.png" size="4182" stream="ice3.png" user="Main.AlessioGianelle" version="1"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Test4 Ice submission rate" date="1232468058" name="ice4.png" path="ice4.png" size="4599" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attachment="ice3.png" attr="" comment="Test 3 Ice submission rate" date="1232468217" name="ice3.png" path="ice3.png" size="4569" stream="ice3.png" user="Main.AlessioGianelle" version="2"

Revision 172009-01-20 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 14 to 14
 
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>

Test finish on Tue Jan 20 at 15:13:14 CET 2009

  • 197 collections submitted in 8499 seconds: 84/43/9 (max/avg/min)
    • 103 submissions fails due to load limiter

  • Collections correctly submitted: 197 ( 15760 jobs)
 

3) Test starts on Mon Jan 19 at 15:22:51 CET 2009 (WMS: devel14)

Description:
  • 2880 collections each of 40 jobs

Revision 162009-01-20 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 47 to 47
 
    • lsf_reason=32512 (1085 times 92.27%)
    • lsf_reason=306 (1 time 0.08%)
Added:
>
>
ice3.png
 

2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs
Line: 173 to 176
 

-- AlessioGianelle - 08 Jan 2009

Added:
>
>

META FILEATTACHMENT attachment="ice3.png" attr="" comment="Ice submission rate" date="1232453586" name="ice3.png" path="ice3.png" size="4182" stream="ice3.png" user="Main.AlessioGianelle" version="1"

Revision 152009-01-20 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

4) Test starts on Tue Jan 20 at 10:09:58 CET 2009 (WMS: devel14)

Description:
  • 300 collections each of 80 jobs
  • One collection every 60 seconds
  • One user
  • Used the lsf CEs of testbedB (PD+CNAF) (cert-06 at cnaf is not considered)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
  • Resubmission is able
  • Lease mechanism is not used
 

3) Test starts on Mon Jan 19 at 15:22:51 CET 2009 (WMS: devel14)

Description:
  • 2880 collections each of 40 jobs
Line: 18 to 30
 
  • 1440 collections each of 80 jobs
  • One collection every 60 seconds
Added:
>
>

Test finishes on Tue Jan 20 at 01:53:01 CET 2009

  • Collections correctly submitted: 399 (24800 jobs)
    • DONE OK: 24702 (99.6%)
    • ABORTED: 0 (0.0%)
    • Not finished: 98 (0.4%)
    • Resubmissions: 1176 (4.74%)
 
Added:
>
>
  • Errors found (1176):
    • Cannot take token (1 time 0.08%)
    • Cannot move ISB (39 times 3.32%)
    • Transfer to CREAM failed (50 times 4.25%)
      • Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (7 times)
      • Transfer to CREAM failed due to exception: Authentication error: The proxy is EXPIRED! (43 times)
    • lsf_reason=32512 (1085 times 92.27%)
    • lsf_reason=306 (1 time 0.08%)
 

2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14)

Description:

Revision 142009-01-19 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 14 to 14
 
  • Resubmission is able
  • Lease mechanism is not used
Added:
>
>
Test has been modified on Mon Jan 19 at 17:03:41:
  • 1440 collections each of 80 jobs
  • One collection every 60 seconds
 

2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14)

Description:

Revision 132009-01-19 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

3) Test starts on Mon Jan 19 at 15:22:51 CET 2009 (WMS: devel14)

Description:
  • 2880 collections each of 40 jobs
  • One collection every 30 seconds
  • One user
  • Used the lsf CEs of testbedB (PD+CNAF)
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
  • Resubmission is able
  • Lease mechanism is not used
 

2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs
Line: 66 to 79
 
8 337 345 cream-30.pd.infn.it
51 2311 2362 Totals
Added:
>
>

BUGS:

  • CREAM
  • BLAH
    • #45718: Some check on log lines should be added on BLParser code
    • #45983: BLAH can leave children processes behind.
 

1) Test starts on Wed Jan 7 at 16:01:32 CET 2009 (WMS: devel18)

Description:

Revision 122009-01-19 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14) ( in progress )

>
>

2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14)

  Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
Added:
>
>
  • One user
 
  • Used the CEs of testbedB (PD+CNAF) plus cream-04.pd.infn.it
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
Line: 13 to 14
 
  • Resubmission is able
  • Lease mechanism is not used
Changed:
<
<

Preliminary results taken on Fri Jan 16 2009

  • Collections correctly submitted: 4001 (160040 jobs)
    • DONE OK: 158990 (99.34%)
    • ABORETD: 0 (0.0%)
    • Not finished: 1050 (0.66%)
    • Resubmission: 1794 (1.12%)

  • Errors found:
    • blparser service is not alive (528 times)
    • BLAH error (256 times)
      • no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = [...]) (48 times)
>
>

Test finishes on Sun Jan 18 at 15:42:28 CET 2009

  • 7180 collections submitted in 70789 seconds: 141/9/3 (max/avg/min)
    • 20 submissions fails due to load limiter

  • Collections correctly submitted: 7180 ( 287200 jobs)
    • DONE OK: 284838 (99.18%)
    • ABORTED: 0 (0.0%)
    • Not finished: 2362 (0.82%)
    • Resubmissions: 4599 (1.60%)

  • Errors found (4599):
    • blparser service is not alive (578 times 12.57%)
    • BLAH error (288 times 6.26%)
      • no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = [...]) (52 times)
 
      • send command timeout (2 times)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) (2 times)
Changed:
<
<
      • submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = [...]) (195 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (9 times)
    • Cannot take token (183 times)
    • Cannot move OSB (1 time)
    • Cannot move ISB (2 times)
    • Transfer to CREAM failed (13 times)
>
>
      • submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = [...]) (219 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (13 times)
    • Cannot take token (201 times 4.37%)
    • Cannot move OSB (1 time 0.02%)
    • Cannot move ISB (5 times 0.11%)
    • Transfer to CREAM failed (19 times 0.41%)
 
      • FaultCause=[The problem seems to be related to glexec which reported: java.io.IOException: Too many open files]" (10 times)
Changed:
<
<
      • CREAM Register raised std::exception Connection to service [https://cert-06.cnaf.infn.it:8443/ce-cream/services/CREAM2] failed: (3 times)
    • lsf_reason=32512 (811 times)
>
>
      • CREAM Register raised std::exception Connection to service [https://cert-xx.cnaf.infn.it:8443/ce-cream/services/CREAM2] failed: (9 times)
    • lsf_reason=32512 (3505 times 76.22%)
    • Proxy is expired (1 time 0.02%)
    • lsf_reason=306 (1 time 0.02%)

  • Jobs not finished:

Schedul Running Tot. Ce Name
0 6 6 cert-04.cnaf.infn.it
6 349 355 cream-34.pd.infn.it
0 3 3 cream-26.pd.infn.it
0 2 2 cream-25.pd.infn.it
5 332 337 cream-28.pd.infn.it
0 1 1 cream-27.pd.infn.it
0 2 2 cream-22.pd.infn.it
0 5 5 cream-04.pd.infn.it
0 1 1 cream-23.pd.infn.it
0 2 2 cert-07.cnaf.infn.it
1 0 1 cert-13.cnaf.infn.it
6 334 340 cream-29.pd.infn.it
9 327 336 cream-33.pd.infn.it
0 5 5 cert-08.cnaf.infn.it
0 2 2 cert-05.cnaf.infn.it
6 0 6 cert-06.cnaf.infn.it
6 307 313 cream-31.pd.infn.it
4 296 300 cream-32.pd.infn.it
8 337 345 cream-30.pd.infn.it
51 2311 2362 Totals
 

1) Test starts on Wed Jan 7 at 16:01:32 CET 2009 (WMS: devel18)

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
Added:
>
>
  • One user
 
  • Used the CEs of testbedB (PD+CNAF) plus cream-12.pd.infn.it
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
Line: 51 to 83
 

Results taken on Mon Jan 12 at 12:52:56 CET 2009

  • Collections correctly submitted: 3733 (149320 jobs)
    • DONE OK: 144004 (96.44%)
Changed:
<
<
    • ABORETD: 446 (0.3%)
>
>
    • ABORTED: 446 (0.3%)
 
    • Not finished: 4870 (3.26%)

  • Errors found:

Revision 112009-01-16 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14)

>
>

2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14) ( in progress )

  Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
Line: 13 to 13
 
  • Resubmission is able
  • Lease mechanism is not used
Changed:
<
<

Partial results taken on Fri Jan 16 2009 ( Update )

>
>

Preliminary results taken on Fri Jan 16 2009

 
  • Collections correctly submitted: 4001 (160040 jobs)
    • DONE OK: 158990 (99.34%)
    • ABORETD: 0 (0.0%)
Line: 46 to 46
 
  • The job is a "sleep 313"
  • Resubmission is able
Changed:
<
<
Test stopped on Monday Jan 12 for a serialization error on ICE
>
>
Test stopped on Monday Jan 12 for a serialization error on ICE
 

Results taken on Mon Jan 12 at 12:52:56 CET 2009

  • Collections correctly submitted: 3733 (149320 jobs)

Revision 102009-01-16 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 13 to 13
 
  • Resubmission is able
  • Lease mechanism is not used
Changed:
<
<

Partial results taken on Thu Jan 15 2009 ( Update )

  • Collections correctly submitted: 2701 (108040 jobs)
    • DONE OK: 107399 (99.4%)
>
>

Partial results taken on Fri Jan 16 2009 ( Update )

  • Collections correctly submitted: 4001 (160040 jobs)
    • DONE OK: 158990 (99.34%)
 
    • ABORETD: 0 (0.0%)
Changed:
<
<
    • Not finished: 641 (0.6%)
    • Resubmission: 795 (0.73%)
>
>
    • Not finished: 1050 (0.66%)
    • Resubmission: 1794 (1.12%)
 
  • Errors found:
    • blparser service is not alive (528 times)
Changed:
<
<
    • BLAH error (153 times)
      • no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = [...]) (30 times)
>
>
    • BLAH error (256 times)
      • no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = [...]) (48 times)
 
      • send command timeout (2 times)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) (2 times)
Changed:
<
<
      • submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = [...]) (112 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (7 times)
    • Cannot take token (103 times)
>
>
      • submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = [...]) (195 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (9 times)
    • Cannot take token (183 times)
 
    • Cannot move OSB (1 time)
Changed:
<
<
    • Transfer to CREAM failed (10 times)
>
>
    • Cannot move ISB (2 times)
    • Transfer to CREAM failed (13 times)
 
      • FaultCause=[The problem seems to be related to glexec which reported: java.io.IOException: Too many open files]" (10 times)
Changed:
<
<
>
>
      • CREAM Register raised std::exception Connection to service [https://cert-06.cnaf.infn.it:8443/ce-cream/services/CREAM2] failed: (3 times)
    • lsf_reason=32512 (811 times)
 

1) Test starts on Wed Jan 7 at 16:01:32 CET 2009 (WMS: devel18)

Description:
Line: 79 to 81
 

BUGS:

  • CREAM
Added:
>
>
    • #45914: glexec and proxy rotation
 
    • #45913: Proxy renewal not done for CREAM jobs not yet in IDLE status
    • #45736: Problems in case of resubmissions in the same CREAM CE
    • #45437: Sometimes the jobPurger throws the exception "Too many open files"

Revision 92009-01-15 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 77 to 77
 
    • Lease expired
    • The job cannot be submitted because the blparser service is not alive
Changed:
<
<
>
>

BUGS:

  • CREAM
    • #45913: Proxy renewal not done for CREAM jobs not yet in IDLE status
    • #45736: Problems in case of resubmissions in the same CREAM CE
    • #45437: Sometimes the jobPurger throws the exception "Too many open files"
  • BLAH
    • #45718: Some check on log lines should be added on BLParser code
    • #45717: BLParserPBS should consider log lines like "unable to run job"
  -- AlessioGianelle - 08 Jan 2009

Revision 82009-01-15 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 11 to 11
 
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
  • Resubmission is able
Changed:
<
<
  • Lease mechanism not used
>
>
  • Lease mechanism is not used
 
Changed:
<
<

Partial results taken on Wed Jan 14 2009

  • Collections correctly submitted: 1401 (56040 jobs)
    • DONE OK: 55700 (99.4%)
>
>

Partial results taken on Thu Jan 15 2009 ( Update )

  • Collections correctly submitted: 2701 (108040 jobs)
    • DONE OK: 107399 (99.4%)
 
    • ABORETD: 0 (0.0%)
Changed:
<
<
    • Not finished: 340 (0.6%)
    • Resubmission: 620 (1.10%)
>
>
    • Not finished: 641 (0.6%)
    • Resubmission: 795 (0.73%)
 
  • Errors found:
    • blparser service is not alive (528 times)
Changed:
<
<
    • BLAH error (54 times)
      • no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = [...]) (15 times)
      • send command timeout (1 time)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) (1 time)
      • submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = [...]) (35 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (2 times)
    • Cannot take token (37 times)
>
>
    • BLAH error (153 times)
      • no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = [...]) (30 times)
      • send command timeout (2 times)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) (2 times)
      • submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = [...]) (112 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (7 times)
    • Cannot take token (103 times)
 
    • Cannot move OSB (1 time)
Added:
>
>
    • Transfer to CREAM failed (10 times)
      • FaultCause=[The problem seems to be related to glexec which reported: java.io.IOException: Too many open files]" (10 times)
 

1) Test starts on Wed Jan 7 at 16:01:32 CET 2009 (WMS: devel18)

Revision 72009-01-14 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

2) Test starts at Tue Jan 13 15:38:11 CET 2009 (WMS: devel14)

>
>

2) Test starts on Tue Jan 13 at 15:38:11 CET 2009 (WMS: devel14)

  Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
Line: 13 to 13
 
  • Resubmission is able
  • Lease mechanism not used
Added:
>
>

Partial results taken on Wed Jan 14 2009

  • Collections correctly submitted: 1401 (56040 jobs)
    • DONE OK: 55700 (99.4%)
    • ABORETD: 0 (0.0%)
    • Not finished: 340 (0.6%)
    • Resubmission: 620 (1.10%)
 
Changed:
<
<

1) Test starts at Wed Jan 7 16:01:32 CET 2009 (WMS: devel18)

>
>
  • Errors found:
    • blparser service is not alive (528 times)
    • BLAH error (54 times)
      • no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = [...]) (15 times)
      • send command timeout (1 time)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = [...]) (1 time)
      • submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = [...]) (35 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:qsub: Invalid credential-) N/A (jobId = [...]) (2 times)
    • Cannot take token (37 times)
    • Cannot move OSB (1 time)

1) Test starts on Wed Jan 7 at 16:01:32 CET 2009 (WMS: devel18)

  Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
Line: 24 to 42
 
  • The job is a "sleep 313"
  • Resubmission is able
Changed:
<
<

Results taken at Mon Jan 12 12:52:56 CET 2009

>
>
Test stopped on Monday Jan 12 for a serialization error on ICE

Results taken on Mon Jan 12 at 12:52:56 CET 2009

 
  • Collections correctly submitted: 3733 (149320 jobs)
    • DONE OK: 144004 (96.44%)
    • ABORETD: 446 (0.3%)

Revision 62009-01-14 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 10 to 10
 
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
Changed:
<
<
  • Resubmission is set to 3 (shallow)
>
>
  • Resubmission is able
 
  • Lease mechanism not used
Line: 22 to 22
 
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
Changed:
<
<
  • Resubmission is set to 3 (shallow)
>
>
  • Resubmission is able
 

Results taken at Mon Jan 12 12:52:56 CET 2009

  • Collections correctly submitted: 3733 (149320 jobs)

Revision 52009-01-13 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

1) Test starts on Wed Jan 7 16:01:32 CET 2009

>
>

2) Test starts at Tue Jan 13 15:38:11 CET 2009 (WMS: devel14)

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
  • Used the CEs of testbedB (PD+CNAF) plus cream-04.pd.infn.it
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
  • Resubmission is set to 3 (shallow)
  • Lease mechanism not used

1) Test starts at Wed Jan 7 16:01:32 CET 2009 (WMS: devel18)

  Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
Line: 12 to 24
 
  • The job is a "sleep 313"
  • Resubmission is set to 3 (shallow)
Changed:
<
<

Results taken on Mon Jan 12 12:52:56 CET 2009

>
>

Results taken at Mon Jan 12 12:52:56 CET 2009

 
  • Collections correctly submitted: 3733 (149320 jobs)
    • DONE OK: 144004 (96.44%)
    • ABORETD: 446 (0.3%)

Revision 42009-01-13 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 22 to 22
 
    • Transfer to CREAM failed due to exception:
      • FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Deadlock found when trying to get lock; try restarting transaction]"
      • Authentication error: Unable to open the file [...]: No such file or directory
Changed:
<
<
      • Connection to servic [...] failed:
>
>
      • Connection to service [...] failed:
 
      • FaultCause=[User [...] not authorized for operation JobRegister]
      • FaultCause=[The problem seems to be related to glexec which reported: java.io.IOException: Too many open files]"
      • FaultCause=[org.glite.ce.common.db.DatabaseException: Server connection failure during transaction. Due to underlying exception: 'java.net.SocketException: Too many open files'.

Revision 32009-01-13 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 41 to 41
 
    • Proxy is expired; Proxy expired: job killed Terminated Master process killed
    • lsf_reason=32512
    • Lease expired
Added:
>
>
    • The job cannot be submitted because the blparser service is not alive
 

-- AlessioGianelle - 08 Jan 2009

Revision 22009-01-12 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

Test starts at Wed Jan 7 16:01:32 CET 2009

>
>

1) Test starts on Wed Jan 7 16:01:32 CET 2009

  Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
Line: 12 to 12
 
  • The job is a "sleep 313"
  • Resubmission is set to 3 (shallow)
Added:
>
>

Results taken on Mon Jan 12 12:52:56 CET 2009

  • Collections correctly submitted: 3733 (149320 jobs)
    • DONE OK: 144004 (96.44%)
    • ABORETD: 446 (0.3%)
    • Not finished: 4870 (3.26%)

  • Errors found:
    • Transfer to CREAM failed due to exception:
      • FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Deadlock found when trying to get lock; try restarting transaction]"
      • Authentication error: Unable to open the file [...]: No such file or directory
      • Connection to servic [...] failed:
      • FaultCause=[User [...] not authorized for operation JobRegister]
      • FaultCause=[The problem seems to be related to glexec which reported: java.io.IOException: Too many open files]"
      • FaultCause=[org.glite.ce.common.db.DatabaseException: Server connection failure during transaction. Due to underlying exception: 'java.net.SocketException: Too many open files'.
      • FaultCause=[java.net.UnknownHostException: cream-31.pd.infn.it: cream-31.pd.infn.it]"
      • CREAM Start raised exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client]
      • Failed to get lease_id for job [...] Exception is Lease renew operation FAILED for lease ID [...] Exception is Connection to service [https://cream-29.pd.infn.it:8443/ce-cream/services/CREAM2] failed:
      • CREAM Start failed due to error MethodName=[JOB_START] Timestamp=[Wed 07 Jan 2009 22:10:43] ErrorCode=[2] Description=[the job has a status not compatible with the JOB_START command!] FaultCause=[N/A]
    • BLAH error:
      • submission command failed (exit code = -15) (stdout:) (stderr:/opt/glite/etc/blah.config: line 54: syntax error near unexpected token `('-/opt/glite/etc/blah.config: line 54: `//Added for test by Enrico Fattibene (07/01/2009)'--killed by signal 15-) N/A (jobId = CREAM251333253)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = CREAM550710004)
      • submission command failed (exit code = 1) (stdout:) (stderr:Cannot resolve default server host 'cream-28.pd.infn.it' - check server_name file.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15008)-) N/A (jobId = CREAM027575485)
      • submission command failed (exit code = -15) (stdout:) (stderr:-killed by signal 15-) N/A (jobId = CREAM752590056)
      • no jobId in submission script's output (stdout:) (stderr:) N/A (jobId = CREAM988027857)
    • DELEGATION_PROXY_CERT_SANDBOX_PATH not defined!
    • Cannot move ISB [...] The proxy credential [...] expired 0 minutes ago.
    • Proxy is expired; Proxy expired: job killed Terminated Master process killed
    • lsf_reason=32512
    • Lease expired
 

-- AlessioGianelle - 08 Jan 2009

Revision 12009-01-08 - AlessioGianelle

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Test starts at Wed Jan 7 16:01:32 CET 2009

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
  • Used the CEs of testbedB (PD+CNAF) plus cream-12.pd.infn.it
  • Used automatic-delegation and proxy renewal service
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 313"
  • Resubmission is set to 3 (shallow)

-- AlessioGianelle - 08 Jan 2009

 
This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback