Difference: WmsTestsP2862 (1 vs. 63)

Revision 632011-02-24 - AlessioGianelle

Line: 1 to 1
Changed:
<
<
META TOPICPARENT name="TestWokPlan"
>
>
META TOPICPARENT name="TestPage"
 

Check bugs:

  • BUG #47612: There's a mem leak in ICE in the handling the returned sequence code HOPEFULLY FIXED

Revision 622009-12-02 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 66 to 66
 
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used
Changed:
<
<
  • Notification mechanism is used: start_listener = false;
>
>
  • Notification mechanism is NOT used: start_listener = false;
 
  • Use "jobdir" input mechanism

Line: 122 to 122
 
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used
Changed:
<
<
  • Notification mechanism is used: start_listener = false;
>
>
  • Notification mechanism is NOT used: start_listener = false;
 
  • Use "jobdir" input mechanism

Submissions finish on Sat Jul 4 at 15:39:00 CEST 2009

Revision 612009-07-10 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 83 to 83
 
  • 2870 collections submitted in 17460 seconds: 3/6/40 (min/avg/max)
    • 10 submissions fail
Changed:
<
<

Partial results taken on Wed Jul 9 at 10:57:09 CEST 2009

>
>

Partial results taken on Fry Jul 10 at 10:57:09 CEST 2009

 
  • Collections correctly submitted: 2870 (114800 jobs)
Changed:
<
<
    • DONE OK: 55753 (48.57%)
    • NOT TERMINATED: 59047 (51.43%)
>
>
    • DONE OK: 96950 (84.45%)
    • NOT TERMINATED: 17850 (15.55%)
 
    • ABORTED: 0 (0%)
Changed:
<
<
    • Resubmitted: 57 (0.05%)
>
>
    • Resubmitted: 475 (0.41%)
 
Changed:
<
<
  • Errors found (58)
>
>
  • Errors found (478)
 
    • BLAH error (1 time)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = CREAM469174928)
Changed:
<
<
    • Cannot take token (1 time)
>
>
    • Cannot take token (2 times)
 
    • lsf_reason=1603 (1 time)
Changed:
<
<
    • Cannot move ISB [...] : proxy expired (33 times) *
    • Transfer to CREAM failed due to exception (5 times)
>
>
    • Cannot move ISB [...] : proxy expired (326 times) *
    • Transfer to CREAM failed due to exception (38 times)
 
      • Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-07-07T18:13:00.260Z0cannot write the authN proxy to file: nullcannot write the authN proxy to file: nullorg.glite.ce.faults.AuthenticationFaultcream-34.pd.infn.it] (1 time)
Changed:
<
<
      • CREAM Register raised std::exception Connection to service [https://cream-XX.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (3 times)
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (1 time)
    • Proxy is expired; Proxy expired: job killed Terminated Master process killed (7 times) *
    • pbs_reason=1 (10 times) *
>
>
      • CREAM Register raised std::exception Connection to service [https://cream-XX.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (33 times)
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times)
      • CREAM Start raised exception Connection to service [https://cream-33.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (1 time)
    • Proxy is expired; Proxy expired: job killed Terminated Master process killed (30 times) *
    • pbs_reason=1 (80 times) *
  ice16.png

NOTE

Added:
>
>
 

15) Test starts on Fri Jul 3 at 15:41:35 CEST 2009 (WMS: devel20)

Description:
Line: 777 to 779
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246629839" name="ice14.png" path="ice14.png" size="8697" user="Main.AlessioGianelle" version="5"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1246544708" name="icemem12.png" path="icemem12.png" size="7310" user="Main.AlessioGianelle" version="3"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="ice16.png" attr="" comment="Ice graph. Test 16" date="1247211609" name="ice16.png" path="ice16.png" size="7439" stream="ice16.png" user="Main.AlessioGianelle" version="4"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 16" date="1247211610" name="ice16.png" path="ice16.png" size="7439" user="Main.AlessioGianelle" version="4"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1246543739" name="ice13.png" path="ice13.png" size="7161" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 15" date="1246869548" name="ice15.png" path="ice15.png" size="6651" user="Main.AlessioGianelle" version="1"

Revision 602009-07-10 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 777 to 777
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246629839" name="ice14.png" path="ice14.png" size="8697" user="Main.AlessioGianelle" version="5"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1246544708" name="icemem12.png" path="icemem12.png" size="7310" user="Main.AlessioGianelle" version="3"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="ice16.png" attr="" comment="Ice graph. Test 16" date="1247150468" name="ice16.png" path="ice16.png" size="7270" stream="ice16.png" user="Main.AlessioGianelle" version="3"
>
>
META FILEATTACHMENT attachment="ice16.png" attr="" comment="Ice graph. Test 16" date="1247211609" name="ice16.png" path="ice16.png" size="7439" stream="ice16.png" user="Main.AlessioGianelle" version="4"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1246543739" name="ice13.png" path="ice13.png" size="7161" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 15" date="1246869548" name="ice15.png" path="ice15.png" size="6651" user="Main.AlessioGianelle" version="1"

Revision 592009-07-09 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 777 to 777
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246629839" name="ice14.png" path="ice14.png" size="8697" user="Main.AlessioGianelle" version="5"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1246544708" name="icemem12.png" path="icemem12.png" size="7310" user="Main.AlessioGianelle" version="3"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="ice16.png" attr="" comment="Ice graph. Test 16" date="1247131609" name="ice16.png" path="ice16.png" size="7154" stream="ice16.png" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attachment="ice16.png" attr="" comment="Ice graph. Test 16" date="1247150468" name="ice16.png" path="ice16.png" size="7270" stream="ice16.png" user="Main.AlessioGianelle" version="3"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1246543739" name="ice13.png" path="ice13.png" size="7161" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 15" date="1246869548" name="ice15.png" path="ice15.png" size="6651" user="Main.AlessioGianelle" version="1"

Revision 582009-07-09 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 79 to 79
  * Do not drop notification
Changed:
<
<

Partial results taken on Wed Jul 8 at 10:57:09 CEST 2009

  • Collections correctly submitted: 1450 (58000 jobs)
    • DONE OK: 35393 (--%)
    • NOT TERMINATED: 22607 (--%)
>
>

Submissions finish on Thu Jul 9 at 09:28:42 CEST 2009

  • 2870 collections submitted in 17460 seconds: 3/6/40 (min/avg/max)
    • 10 submissions fail

Partial results taken on Wed Jul 9 at 10:57:09 CEST 2009

  • Collections correctly submitted: 2870 (114800 jobs)
    • DONE OK: 55753 (48.57%)
    • NOT TERMINATED: 59047 (51.43%)
 
    • ABORTED: 0 (0%)
Changed:
<
<
    • Resubmitted: 2 (--%)
>
>
    • Resubmitted: 57 (0.05%)
 
Changed:
<
<
>
>
  • Errors found (58)
    • BLAH error (1 time)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = CREAM469174928)
    • Cannot take token (1 time)
    • lsf_reason=1603 (1 time)
    • Cannot move ISB [...] : proxy expired (33 times) *
    • Transfer to CREAM failed due to exception (5 times)
      • Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-07-07T18:13:00.260Z0cannot write the authN proxy to file: nullcannot write the authN proxy to file: nullorg.glite.ce.faults.AuthenticationFaultcream-34.pd.infn.it] (1 time)
      • CREAM Register raised std::exception Connection to service [https://cream-XX.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (3 times)
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (1 time)
    • Proxy is expired; Proxy expired: job killed Terminated Master process killed (7 times) *
    • pbs_reason=1 (10 times) *
  ice16.png
Added:
>
>

NOTE

 

15) Test starts on Fri Jul 3 at 15:41:35 CEST 2009 (WMS: devel20)

Description:
  • 1440 collections each of 40 jobs
Line: 763 to 777
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246629839" name="ice14.png" path="ice14.png" size="8697" user="Main.AlessioGianelle" version="5"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1246544708" name="icemem12.png" path="icemem12.png" size="7310" user="Main.AlessioGianelle" version="3"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 16" date="1247044127" name="ice16.png" path="ice16.png" size="6797" user="Main.AlessioGianelle" version="1"
>
>
META FILEATTACHMENT attachment="ice16.png" attr="" comment="Ice graph. Test 16" date="1247131609" name="ice16.png" path="ice16.png" size="7154" stream="ice16.png" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1246543739" name="ice13.png" path="ice13.png" size="7161" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 15" date="1246869548" name="ice15.png" path="ice15.png" size="6651" user="Main.AlessioGianelle" version="1"

Revision 572009-07-08 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 79 to 79
  * Do not drop notification
Added:
>
>

Partial results taken on Wed Jul 8 at 10:57:09 CEST 2009

  • Collections correctly submitted: 1450 (58000 jobs)
    • DONE OK: 35393 (--%)
    • NOT TERMINATED: 22607 (--%)
    • ABORTED: 0 (0%)
    • Resubmitted: 2 (--%)

ice16.png

 

15) Test starts on Fri Jul 3 at 15:41:35 CEST 2009 (WMS: devel20)

Description:
  • 1440 collections each of 40 jobs
Line: 748 to 763
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246629839" name="ice14.png" path="ice14.png" size="8697" user="Main.AlessioGianelle" version="5"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1246544708" name="icemem12.png" path="icemem12.png" size="7310" user="Main.AlessioGianelle" version="3"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 16" date="1247044127" name="ice16.png" path="ice16.png" size="6797" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1246543739" name="ice13.png" path="ice13.png" size="7161" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 15" date="1246869548" name="ice15.png" path="ice15.png" size="6651" user="Main.AlessioGianelle" version="1"

Revision 562009-07-07 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 48 to 48
 
    • Make sure that the file/directory pointed by the Input="..." attribute does not exis
    • Now, start ICE and see if the directory pointed by the Input="..." attribute gets created automatically.
Added:
>
>
  • BUG #52884: ICE uses too much memory HOPEFULLY FIXED
    • See tests below
 

TESTs on ICE

Added:
>
>

16) Test starts on Tue Jul at 7 09:27:28 CEST 2009 (WMS: devel20)

Description:
  • 2880 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 10
  • Used all the CEs of testbedB (except cert-07.cnaf and cream-25.pd)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Notification mechanism is used: start_listener = false;
  • Use "jobdir" input mechanism

  • Changes in the software wrt previous test:
    • ICE
      • Retrying some times on generic database exception before giving up
      • Fixed several SQL syntax error
      • Sleeping less time between a request fetch and subsequent... (useful when running with few threads)
      • Loading # of request from filelist/jobdir equal to # of threads * Do not drop notification
 

15) Test starts on Fri Jul 3 at 15:41:35 CEST 2009 (WMS: devel20)

Description:
  • 1440 collections each of 40 jobs

Revision 552009-07-06 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 66 to 66
 
  • Notification mechanism is used: start_listener = false;
  • Use "jobdir" input mechanism
Added:
>
>

Submissions finish on Sat Jul 4 at 15:39:00 CEST 2009

  • 902 collections submitted in 13717 seconds: 9/15/72 (min/avg/max)
    • 538 submissions fail

Final results taken on Fry Jul 6 at 09:42:38 CEST 2009

  • Collections correctly submitted: 902 (36080 jobs)
    • DONE OK: 36080 (100%)
    • NOT TERMINATED: 0 (0%)
    • ABORTED: 0 (0%)
    • Resubmitted: 3 (0.008%)

ice15.png

 

14) Test starts on Thu Jun 25 at 15:24:56 CEST 2009 (WMS: devel20)

Description:
Line: 701 to 722
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1246543739" name="ice13.png" path="ice13.png" size="7161" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 15" date="1246869548" name="ice15.png" path="ice15.png" size="6651" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243612371" name="icemem10.png" path="icemem10.png" size="6275" user="Main.AlessioGianelle" version="5"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"

Revision 542009-07-03 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 51 to 51
 

TESTs on ICE

Added:
>
>

15) Test starts on Fri Jul 3 at 15:41:35 CEST 2009 (WMS: devel20)

Description:
  • 1440 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 7
  • Used all the CEs of testbedB (except cert-07.cnaf and cream-25.pd)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Notification mechanism is used: start_listener = false;
  • Use "jobdir" input mechanism
 

14) Test starts on Thu Jun 25 at 15:24:56 CEST 2009 (WMS: devel20)

Description:
  • 7200 collections each of 40 jobs
Line: 68 to 84
 

Submissions finish on Wed Jul 1 at 05:11:15 CEST 2009

  • 4573 collections submitted in 124424 seconds: 6/27/505 (min/avg/max)
Changed:
<
<
    • 2627 submission(s) fail(s) due to load limiter
>
>
    • 2627 submissions fail due to load limiter
    • 10 submissions fail at wmproxy level due to a LB error
 
Changed:
<
<

Partial results taken on Fry Jul 3 at 10:42:38 CEST 2009

  • Collections correctly submitted: 4573 (182920 jobs)
    • DONE OK: 174033 (95.14%)
    • NOT TERMINATED: 8260 (4.52%)
>
>

Final results taken on Fry Jul 3 at 15:42:38 CEST 2009

  • Collections correctly submitted: 4533 (182520 jobs)
    • DONE OK: 181893 (99.66%)
    • NOT TERMINATED: 0 (0%)
 
    • ABORTED: 627 * (0.34%)
Changed:
<
<
    • Resubmitted: 1449 (0.79%)
>
>
    • Resubmitted: 3150 (1.73%)
 
Changed:
<
<
  • Errors found (3594)
    • BLAH error (16 times)
>
>
  • Errors found (5302)
    • BLAH error (17 times)
 
      • blah error: send command timeout (14 time)
Changed:
<
<
      • no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (2 times )
    • Cannot move ISB: (1084 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (211 times)
      • [...] : proxy expired (794 times)
    • Transfer to CREAM failed due to exception: (2125 times)
>
>
      • no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (3 times )
    • Cannot move ISB: (2540 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (298 times)
      • [...] : proxy expired (2160 times)
    • Transfer to CREAM failed due to exception: (2134 times)
 
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/I6/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fI6T7Z-FDo9uzJ_5f3GmAPXgw/user.proxy] : No such file or directory (2107 times) *
Changed:
<
<
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Tue 30 Jun 2009 14:16:53] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec [glexec reported = "glexec policy violation: see glexec log for more details"]]" (5 times)
>
>
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (4 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Tue 30 Jun 2009 14:16:53] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec [glexec reported = "glexec policy violation: see glexec log for more details"]]" (13 times)
 
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Sat 27 Jun 2009 13:51:57] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction] (10 times)
Changed:
<
<
    • Proxy is expired (260 times)
    • pbs_reason=1 (109 times)
>
>
    • Proxy is expired (312 times)
    • pbs_reason=1 (299 times)
 

NOTE

Line: 679 to 696
  -- AlessioGianelle - 05 Mar 2009
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246543755" name="ice14.png" path="ice14.png" size="8671" user="Main.AlessioGianelle" version="4"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246629839" name="ice14.png" path="ice14.png" size="8697" user="Main.AlessioGianelle" version="5"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1246544708" name="icemem12.png" path="icemem12.png" size="7310" user="Main.AlessioGianelle" version="3"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1246543739" name="ice13.png" path="ice13.png" size="7161" user="Main.AlessioGianelle" version="2"

Revision 532009-07-03 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 70 to 70
 
  • 4573 collections submitted in 124424 seconds: 6/27/505 (min/avg/max)
    • 2627 submission(s) fail(s) due to load limiter
Changed:
<
<

Partial results taken on Thu Jul 2 at 17:22:19 CEST 2009

>
>

Partial results taken on Fry Jul 3 at 10:42:38 CEST 2009

 
  • Collections correctly submitted: 4573 (182920 jobs)
Changed:
<
<
    • DONE OK: 140517 (76.82%)
    • NOT TERMINATED: 42376 (23.17%)
    • ABORTED: 27 (0.01%)
    • Resubmitted: 250 (0.14%)

  • Errors found (338)
    • blah error: send command timeout (4 time)
    • Cannot move ISB: (216 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (141 times)
      • [...] : proxy expired (50 times)
    • Transfer to CREAM failed due to exception: (94 times)
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/I6/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fI6T7Z-FDo9uzJ_5f3GmAPXgw/user.proxy] : No such file or directory (83 times)
>
>
    • DONE OK: 174033 (95.14%)
    • NOT TERMINATED: 8260 (4.52%)
    • ABORTED: 627 * (0.34%)
    • Resubmitted: 1449 (0.79%)

  • Errors found (3594)
    • BLAH error (16 times)
      • blah error: send command timeout (14 time)
      • no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (2 times )
    • Cannot move ISB: (1084 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (211 times)
      • [...] : proxy expired (794 times)
    • Transfer to CREAM failed due to exception: (2125 times)
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/I6/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fI6T7Z-FDo9uzJ_5f3GmAPXgw/user.proxy] : No such file or directory (2107 times) *
 
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times)
Changed:
<
<
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Sat 27 Jun 2009 13:51:57] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction] (8 times)
    • Proxy is expired (24 times)
>
>
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Tue 30 Jun 2009 14:16:53] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec [glexec reported = "glexec policy violation: see glexec log for more details"]]" (5 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Sat 27 Jun 2009 13:51:57] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction] (10 times)
    • Proxy is expired (260 times)
    • pbs_reason=1 (109 times)
 
Added:
>
>

NOTE

  ice14.png

Revision 522009-07-02 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 70 to 70
 
  • 4573 collections submitted in 124424 seconds: 6/27/505 (min/avg/max)
    • 2627 submission(s) fail(s) due to load limiter
Changed:
<
<

Partial results taken on Thu Jul 2 at 14:22:19 CEST 2009

>
>

Partial results taken on Thu Jul 2 at 17:22:19 CEST 2009

 
  • Collections correctly submitted: 4573 (182920 jobs)
Changed:
<
<
    • DONE OK: 131738 (72.02%)
    • NOT TERMINATED: 51155 (27.97%)
>
>
    • DONE OK: 140517 (76.82%)
    • NOT TERMINATED: 42376 (23.17%)
 
    • ABORTED: 27 (0.01%)
Changed:
<
<
    • Resubmitted: 248 (0.14%)
>
>
    • Resubmitted: 250 (0.14%)
 
  • Errors found (338)
    • blah error: send command timeout (4 time)
Line: 675 to 675
 -- AlessioGianelle - 05 Mar 2009

META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246543755" name="ice14.png" path="ice14.png" size="8671" user="Main.AlessioGianelle" version="4"
Changed:
<
<
META FILEATTACHMENT attachment="icemem12.png" attr="" comment="Ice graph. Test 12" date="1246544708" name="icemem12.png" path="ice12.png" size="7310" stream="ice12.png" user="Main.AlessioGianelle" version="3"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1246544708" name="icemem12.png" path="icemem12.png" size="7310" user="Main.AlessioGianelle" version="3"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1246543739" name="ice13.png" path="ice13.png" size="7161" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"

Revision 512009-07-02 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

Check bugs:

Line: 70 to 70
 
  • 4573 collections submitted in 124424 seconds: 6/27/505 (min/avg/max)
    • 2627 submission(s) fail(s) due to load limiter
Changed:
<
<

Partial results taken on Thu Jul 2 at 11:22:19 CEST 2009

>
>

Partial results taken on Thu Jul 2 at 14:22:19 CEST 2009

 
  • Collections correctly submitted: 4573 (182920 jobs)
Changed:
<
<
    • DONE OK: 120297 (65.77%)
    • NOT TERMINATED: 62596 (34.22%)
>
>
    • DONE OK: 131738 (72.02%)
    • NOT TERMINATED: 51155 (27.97%)
 
    • ABORTED: 27 (0.01%)
Changed:
<
<
    • Resubmitted: 242 (0.13%)
>
>
    • Resubmitted: 248 (0.14%)
 
Changed:
<
<
  • Errors found (332)
>
>
  • Errors found (338)
 
    • blah error: send command timeout (4 time)
Changed:
<
<
    • Cannot move ISB: (212 times)
>
>
    • Cannot move ISB: (216 times)
 
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (141 times)
      • [...] : proxy expired (50 times)
Changed:
<
<
    • Transfer to CREAM failed due to exception: (92 times)
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/I6/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fI6T7Z-FDo9uzJ_5f3GmAPXgw/user.proxy] : No such file or directory (81 times)
>
>
    • Transfer to CREAM failed due to exception: (94 times)
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/I6/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fI6T7Z-FDo9uzJ_5f3GmAPXgw/user.proxy] : No such file or directory (83 times)
 
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Sat 27 Jun 2009 13:51:57] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction] (8 times)
    • Proxy is expired (24 times)
Line: 674 to 674
  -- AlessioGianelle - 05 Mar 2009
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246444357" name="ice14.png" path="ice14.png" size="8347" user="Main.AlessioGianelle" version="3"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245077128" name="icemem12.png" path="icemem12.png" size="6439" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246543755" name="ice14.png" path="ice14.png" size="8671" user="Main.AlessioGianelle" version="4"
META FILEATTACHMENT attachment="icemem12.png" attr="" comment="Ice graph. Test 12" date="1246544708" name="icemem12.png" path="ice12.png" size="7310" stream="ice12.png" user="Main.AlessioGianelle" version="3"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1245774889" name="ice13.png" path="ice13.png" size="6219" user="Main.AlessioGianelle" version="1"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1246543739" name="ice13.png" path="ice13.png" size="7161" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243612371" name="icemem10.png" path="icemem10.png" size="6275" user="Main.AlessioGianelle" version="5"

Revision 502009-07-02 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"
Added:
>
>

Check bugs:

  • BUG #47612: There's a mem leak in ICE in the handling the returned sequence code HOPEFULLY FIXED
    • Not reproducible

  • BUG #47853: ICE should provide info to be used by WMSMonitor FIXED
    • There is a new tool: /opt/glite/bin/queryDb
      USAGE: queryDb --conf|-c <WMS CONFIGURATION FILE> [options]
      
      options: 
        --verbose|-v          Verbose output (print each db's record
        --status-filter|-s    Select only records in which the status column is one
                              of those specified as option argument; more states can
                              be ',' separated and they must be:
                              REGISTERED
                              PENDING
                              IDLE
                              RUNNING
                              REALLY_RUNNING
                              CANCELLED
                              HELD
                              ABORTED
                              DONE_OK
                              DONE_FAILED
                              UNKNOWN
                              PURGED
      
        --userdn|-u           Print the USERDN column of the job table
        --creamjobid|-C       Print the CREAM JOB ID column of the job table
        --gridjobid|-G        Print the GRID JOB ID column of the job table
        --userproxy|-p        Print the USER PROXY column of the job table
        --cream-url|-r        Print the CREAM URL column of the job table
        --myproxy-url|-m      Print the MYPROXY URL column of the job table
        --status|-S           Print the STATUS column of the job table
        --lease-id|-L         Print the LEASE-ID column of the job table
        --delegation-id|-D    Print the DELEGATION-ID column of the job table
        --proxy-exptime|-t    Print the PROXY EXPIRATION TIME column of the job table
        --worker-node|-w      Print the WORKER-NODE column of the job table
        --help|-h             Print this help

  • BUG #47911: Performance problems in ICE when there many (thousands) proxy files to manage HOPEFULLY FIXED
    • See tests below

  • BUG #48905: Failure in starting ICE if its output jobdir doesn't exist FIXED
    • In the WMS configuration file, section "WorkloadManager", set DispatcherType = "jobdir"
    • Make sure that the file/directory pointed by the Input="..." attribute does not exis
    • Now, start ICE and see if the directory pointed by the Input="..." attribute gets created automatically.
 

TESTs on ICE

14) Test starts on Thu Jun 25 at 15:24:56 CEST 2009 (WMS: devel20)

Line: 21 to 70
 
  • 4573 collections submitted in 124424 seconds: 6/27/505 (min/avg/max)
    • 2627 submission(s) fail(s) due to load limiter
Changed:
<
<

Partial results taken on Wed Jul 1 at 11:22:19 CEST 2009

>
>

Partial results taken on Thu Jul 2 at 11:22:19 CEST 2009

 
  • Collections correctly submitted: 4573 (182920 jobs)
Changed:
<
<
    • DONE OK: 72403 (--%)
    • NOT TERMINATED: 110501 (--%)
    • ABORTED: 16 (--%)
    • Resubmitted: 52 (--%)

  • Errors found (101)
    • blah error: send command timeout (1 time)
    • Cannot move ISB: (47 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (44 times)
    • Transfer to CREAM failed due to exception: (52 times)
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/I6/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fI6T7Z-FDo9uzJ_5f3GmAPXgw/user.proxy] : No such file or directory (48 times)
>
>
    • DONE OK: 120297 (65.77%)
    • NOT TERMINATED: 62596 (34.22%)
    • ABORTED: 27 (0.01%)
    • Resubmitted: 242 (0.13%)

  • Errors found (332)
    • blah error: send command timeout (4 time)
    • Cannot move ISB: (212 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (141 times)
      • [...] : proxy expired (50 times)
    • Transfer to CREAM failed due to exception: (92 times)
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/I6/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fI6T7Z-FDo9uzJ_5f3GmAPXgw/user.proxy] : No such file or directory (81 times)
 
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times)
Changed:
<
<
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Sat 27 Jun 2009 13:51:57] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction] (1 time)
    • Proxy is expired (1 time)
>
>
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Sat 27 Jun 2009 13:51:57] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction] (8 times)
    • Proxy is expired (24 times)
 

Revision 492009-07-01 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 21 to 21
 
  • 4573 collections submitted in 124424 seconds: 6/27/505 (min/avg/max)
    • 2627 submission(s) fail(s) due to load limiter
Changed:
<
<

Partial results taken on Tue Jun 30 at 10:02:17 CEST 2009

  • Collections correctly submitted: 4003 (160120 jobs)
    • DONE OK: 55595 (--%)
    • NOT TERMINATED: 104525 (--%)
    • ABORTED: 0 (0%)
    • Resubmitted: 20 (--%)

  • Errors found (20)
    • Cannot move ISB: (15 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (15 times)
    • Transfer to CREAM failed due to exception: (4 times)
>
>

Partial results taken on Wed Jul 1 at 11:22:19 CEST 2009

  • Collections correctly submitted: 4573 (182920 jobs)
    • DONE OK: 72403 (--%)
    • NOT TERMINATED: 110501 (--%)
    • ABORTED: 16 (--%)
    • Resubmitted: 52 (--%)

  • Errors found (101)
    • blah error: send command timeout (1 time)
    • Cannot move ISB: (47 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (44 times)
    • Transfer to CREAM failed due to exception: (52 times)
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/I6/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fI6T7Z-FDo9uzJ_5f3GmAPXgw/user.proxy] : No such file or directory (48 times)
 
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Sat 27 Jun 2009 13:51:57] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction] (1 time)
    • Proxy is expired (1 time)
Line: 622 to 624
  -- AlessioGianelle - 05 Mar 2009
Changed:
<
<
META FILEATTACHMENT attachment="ice14.png" attr="" comment="Ice graph. Test 14" date="1246444356" name="ice14.png" path="ice14.png" size="8347" stream="ice14.png" user="Main.AlessioGianelle" version="3"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246444357" name="ice14.png" path="ice14.png" size="8347" user="Main.AlessioGianelle" version="3"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245077128" name="icemem12.png" path="icemem12.png" size="6439" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1245774889" name="ice13.png" path="ice13.png" size="6219" user="Main.AlessioGianelle" version="1"

Revision 482009-07-01 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 17 to 17
 
  • Notification mechanism is used: start_listener = true;
  • Use "jobdir" input mechanism
Added:
>
>

Submissions finish on Wed Jul 1 at 05:11:15 CEST 2009

  • 4573 collections submitted in 124424 seconds: 6/27/505 (min/avg/max)
    • 2627 submission(s) fail(s) due to load limiter
 

Partial results taken on Tue Jun 30 at 10:02:17 CEST 2009

  • Collections correctly submitted: 4003 (160120 jobs)
Line: 619 to 622
  -- AlessioGianelle - 05 Mar 2009
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246350236" name="ice14.png" path="ice14.png" size="7922" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attachment="ice14.png" attr="" comment="Ice graph. Test 14" date="1246444356" name="ice14.png" path="ice14.png" size="8347" stream="ice14.png" user="Main.AlessioGianelle" version="3"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245077128" name="icemem12.png" path="icemem12.png" size="6439" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1245774889" name="ice13.png" path="ice13.png" size="6219" user="Main.AlessioGianelle" version="1"

Revision 472009-06-30 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 18 to 18
 
  • Use "jobdir" input mechanism
Changed:
<
<

Partial results taken on Mon Jun 29 at 10:02:17 CEST 2009

  • Collections correctly submitted: 3353 (134120 jobs)
    • DONE OK: 47941 (--%)
    • NOT TERMINATED: 86179 (--%)
>
>

Partial results taken on Tue Jun 30 at 10:02:17 CEST 2009

  • Collections correctly submitted: 4003 (160120 jobs)
    • DONE OK: 55595 (--%)
    • NOT TERMINATED: 104525 (--%)
 
    • ABORTED: 0 (0%)
Changed:
<
<
    • Resubmitted: 15 (--%)
>
>
    • Resubmitted: 20 (--%)
 
Changed:
<
<
  • Errors found (15)
    • Cannot move ISB: (11 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (11 times)
    • Transfer to CREAM failed due to exception: (3 times)
>
>
  • Errors found (20)
    • Cannot move ISB: (15 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (15 times)
    • Transfer to CREAM failed due to exception: (4 times)
 
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times)
Added:
>
>
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Sat 27 Jun 2009 13:51:57] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction] (1 time)
 
    • Proxy is expired (1 time)
Line: 618 to 619
  -- AlessioGianelle - 05 Mar 2009
Changed:
<
<
META FILEATTACHMENT attachment="ice14.png" attr="" comment="Ice graph. Test 14" date="1246350236" name="ice14.png" path="ice14.png" size="7922" stream="ice14.png" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246350236" name="ice14.png" path="ice14.png" size="7922" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245077128" name="icemem12.png" path="icemem12.png" size="6439" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1245774889" name="ice13.png" path="ice13.png" size="6219" user="Main.AlessioGianelle" version="1"

Revision 462009-06-30 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 618 to 618
  -- AlessioGianelle - 05 Mar 2009
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246263152" name="ice14.png" path="ice14.png" size="7473" user="Main.AlessioGianelle" version="1"
>
>
META FILEATTACHMENT attachment="ice14.png" attr="" comment="Ice graph. Test 14" date="1246350236" name="ice14.png" path="ice14.png" size="7922" stream="ice14.png" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245077128" name="icemem12.png" path="icemem12.png" size="6439" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1245774889" name="ice13.png" path="ice13.png" size="6219" user="Main.AlessioGianelle" version="1"

Revision 452009-06-29 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 17 to 17
 
  • Notification mechanism is used: start_listener = true;
  • Use "jobdir" input mechanism
Added:
>
>

Partial results taken on Mon Jun 29 at 10:02:17 CEST 2009

  • Collections correctly submitted: 3353 (134120 jobs)
    • DONE OK: 47941 (--%)
    • NOT TERMINATED: 86179 (--%)
    • ABORTED: 0 (0%)
    • Resubmitted: 15 (--%)

  • Errors found (15)
    • Cannot move ISB: (11 times)
      • [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (11 times)
    • Transfer to CREAM failed due to exception: (3 times)
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times)
    • Proxy is expired (1 time)

 ice14.png

13) Test starts on Jun 17 at 18:06:55 CEST 2009 (WMS: devel20)

Line: 601 to 618
  -- AlessioGianelle - 05 Mar 2009
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 14" date="1246263152" name="ice14.png" path="ice14.png" size="7473" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245077128" name="icemem12.png" path="icemem12.png" size="6439" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1245774889" name="ice13.png" path="ice13.png" size="6219" user="Main.AlessioGianelle" version="1"
Line: 611 to 629
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 11" date="1244023720" name="icemem11.png" path="icemem11.png" size="6434" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Deleted:
<
<
META FILEATTACHMENT attachment="ice14.png" attr="" comment="Ice graph. Test 14" date="1246263151" name="ice14.png" path="ice14.png" size="7473" stream="ice14.png" user="Main.AlessioGianelle" version="1"

Revision 442009-06-29 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 17 to 17
 
  • Notification mechanism is used: start_listener = true;
  • Use "jobdir" input mechanism
Added:
>
>
ice14.png
 

13) Test starts on Jun 17 at 18:06:55 CEST 2009 (WMS: devel20)

Description:
  • 7200 collections each of 40 jobs
Line: 609 to 611
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 11" date="1244023720" name="icemem11.png" path="icemem11.png" size="6434" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attachment="ice14.png" attr="" comment="Ice graph. Test 14" date="1246263151" name="ice14.png" path="ice14.png" size="7473" stream="ice14.png" user="Main.AlessioGianelle" version="1"

Revision 432009-06-25 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

14) Test starts on Thu Jun 25 at 15:24:56 CEST 2009 (WMS: devel20)

Description:
  • 7200 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-07.cnaf and cream-25.pd)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Notification mechanism is used: start_listener = true;
  • Use "jobdir" input mechanism
 

13) Test starts on Jun 17 at 18:06:55 CEST 2009 (WMS: devel20)

Description:
  • 7200 collections each of 40 jobs
Line: 30 to 45
 
  • 5220 collections submitted in 85950 seconds: 5/16/77 (min/avg/max)
    • 1980 submissions fail due to load limiter
Changed:
<
<
>
>

Final results

  • Collections correctly submitted: 5220 (208800 jobs)
    • DONE OK: 207628 (99.44%)
    • NOT TERMINATED: 910 (0.44%)
    • ABORTED: 262 (0.12%)
    • Resubmitted: 3398 (1.63%)

  • Errors found (4500)
    • Cannot move ISB: (322 times)
    • Transfer to CREAM failed due to exception: (3896 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Wed 17 Jun 2009 22:42:08] ErrorCode=[0] Description=[system error] FaultCause=[Batch System lsf not supported!]" (982 times)
      • CREAM Register raised std::exception The endpoint is blacklisted (1794)
      • CREAM Start raised exception The endpoint is blacklisted (30 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Mon 22 Jun 2009 09:22:49] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction]" (164 time)
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/XP/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fXPAysgaAaUdQrjs2h_5fXg1g/user.proxy] : No such file or directory (922 times)
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (4 times)
    • pbs_reason (16 times)
      • pbs_reason=1 (13 times)
      • pbs_reason=271 (3 times)
    • Proxy is expired (251 times)
    • BLAH error (15 times)
      • exit code = -15 (12 times)
      • exit code = 1 (2 times)
      • exit code = 120 (1 time)
 ice13.png
Added:
>
>

NOTE:

  • The 982 errors "Batch System lsf not supported" are due to a malconfigured CE wrongly attached to the BDII of the testbed
  • The 910 "non terminated" jobs are matched to cream-25.pd.infn.it which dyes (due to an hardware problem) during the test
  • The error "The endpoint is blacklisted" means that a timeout occurs during the CE connection
 

12) Test starts on Fri Jun 12 at 15:59:59 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs

Revision 422009-06-23 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 26 to 26
 
      • new indices on table jobs
Added:
>
>

Submissions finish on Mon Jun at 22 18:06:38 CEST 2009

  • 5220 collections submitted in 85950 seconds: 5/16/77 (min/avg/max)
    • 1980 submissions fail due to load limiter

ice13.png

 

12) Test starts on Fri Jun 12 at 15:59:59 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs
Line: 550 to 557
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245077128" name="icemem12.png" path="icemem12.png" size="6439" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 13" date="1245774889" name="ice13.png" path="ice13.png" size="6219" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243612371" name="icemem10.png" path="icemem10.png" size="6275" user="Main.AlessioGianelle" version="5"

Revision 412009-06-17 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

13) Test starts on Jun 17 at 14:06:45 CEST 2009 (WMS: devel20)

>
>

13) Test starts on Jun 17 at 18:06:55 CEST 2009 (WMS: devel20)

  Description:
Changed:
<
<
  • 7200 collections each of 50 jobs
>
>
  • 7200 collections each of 40 jobs
 
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-07.cnaf)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
Changed:
<
<
  • The job is a "sleep 2424"
>
>
  • The job is a "sleep 4242"
 
  • Resubmission is enabled
  • Lease mechanism is not used
  • Notification mechanism is not used: start_listener = false;
Line: 23 to 23
 
      • fixed a wrong sql query
      • Transaction to database is tried indefinitely (no more DbLockedException can be thrown)
      • new SQL commands to get the terminated jobs from the database
Added:
>
>
      • new indices on table jobs
 

12) Test starts on Fri Jun 12 at 15:59:59 CEST 2009 (WMS: devel20)

Revision 402009-06-17 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

13) Test starts on Tue Jun 16 at 17:20:06 CEST 2009 (WMS: devel20)

>
>

13) Test starts on Jun 17 at 14:06:45 CEST 2009 (WMS: devel20)

  Description:
Changed:
<
<
  • 2880 collections each of 50 jobs
>
>
  • 7200 collections each of 50 jobs
 
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20

Revision 392009-06-16 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

13) Test starts on Tue Jun 16 at 17:20:06 CEST 2009 (WMS: devel20)

Description:
  • 2880 collections each of 50 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-07.cnaf)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Notification mechanism is not used: start_listener = false;
  • Use "jobdir" input mechanism

  • Changes in the software wrt previous test:
    • ICE
      • fixed a wrong sql query
      • Transaction to database is tried indefinitely (no more DbLockedException can be thrown)
      • new SQL commands to get the terminated jobs from the database
 

12) Test starts on Fri Jun 12 at 15:59:59 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs

Revision 382009-06-16 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 33 to 33
 
    • 101 submissions fail due to load limiter
Changed:
<
<

Partial results taken on Mon Jun 15 at 17:18:25 CEST 2009

>
>

Final results

 
  • Collections correctly submitted: 4219 (168760 jobs)
Changed:
<
<
    • DONE OK: 106170 (62.91%)
    • NOT TERMINATED: 62590 (37.09%)
>
>
    • DONE OK: 168748 (99.99%)
    • NOT TERMINATED: 12 (0.01%)
 
    • ABORTED: 0 (0%)
Changed:
<
<
    • Resubmitted: 1293 (0.77%)
>
>
    • Resubmitted: 2414 (1.43%)
 
Changed:
<
<
  • Errors found (1307)
    • Cannot move ISB: (15 times)
    • Transfer to CREAM fails: (1289 times)
>
>
  • Errors found (2446)
    • Cannot move ISB: (18 times)
    • Transfer to CREAM fails: (1948 times)
 
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-06-13T06:23:42.151Z0cannot write the authN proxy to file: nullcannot write the authN proxy to file: nullorg.glite.ce.faults.AuthenticationFaultcream-34.pd.infn.it]
      • CREAM Start raised exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-06-14T07:00:11.781Z0USER_VO_LABEL not defined in msgContextUSER_VO_LABEL not defined in msgContextorg.glite.ce.faults.AuthenticationFaultcream-32.pd.infn.it]
Changed:
<
<
>
>
 
    • Cannot take token (3 times)
Added:
>
>
    • lsf_reason=65280
    • pbs_reason=271 (3 times)
    • Proxy is expired (473 times)
 

Revision 372009-06-15 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 27 to 27
 
      • Add some more logs to JobWrapper stdout for debugging purpose
Changed:
<
<

Partial results taken on Mon Jun 15 at 10:38:25 CEST 2009

  • Collections correctly submitted: 3780 (151200 jobs)
    • DONE OK: 101639 (...%)
    • NOT TERMINATED: 49561 (...%)
>
>

Submissions finish on Mon Jun 15 at 15:57:55 CEST 2009

  • 4219 collections submitted in 39736 seconds: 4/9/97 (min/avg/max)
    • 101 submissions fail due to load limiter

Partial results taken on Mon Jun 15 at 17:18:25 CEST 2009

  • Collections correctly submitted: 4219 (168760 jobs)
    • DONE OK: 106170 (62.91%)
    • NOT TERMINATED: 62590 (37.09%)
 
    • ABORTED: 0 (0%)
Changed:
<
<
    • Resubmitted: 1263 (...%)
>
>
    • Resubmitted: 1293 (0.77%)
 
Changed:
<
<
  • Errors found (1274)
    • Cannot move ISB: (15 times)
    • Transfer to CREAM fails: (1259 times)
>
>
  • Errors found (1307)
    • Cannot move ISB: (15 times)
    • Transfer to CREAM fails: (1289 times)
 
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-06-13T06:23:42.151Z0cannot write the authN proxy to file: nullcannot write the authN proxy to file: nullorg.glite.ce.faults.AuthenticationFaultcream-34.pd.infn.it]
      • CREAM Start raised exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-06-14T07:00:11.781Z0USER_VO_LABEL not defined in msgContextUSER_VO_LABEL not defined in msgContextorg.glite.ce.faults.AuthenticationFaultcream-32.pd.infn.it]
Changed:
<
<
>
>
 
Line: 514 to 521
  -- AlessioGianelle - 05 Mar 2009
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245055204" name="icemem12.png" path="icemem12.png" size="6622" user="Main.AlessioGianelle" version="1"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245077128" name="icemem12.png" path="icemem12.png" size="6439" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"

Revision 362009-06-15 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 28 to 28
 

Partial results taken on Mon Jun 15 at 10:38:25 CEST 2009

Added:
>
>
  • Collections correctly submitted: 3780 (151200 jobs)
    • DONE OK: 101639 (...%)
    • NOT TERMINATED: 49561 (...%)
    • ABORTED: 0 (0%)
    • Resubmitted: 1263 (...%)

  • Errors found (1274)
    • Cannot move ISB: (15 times)
    • Transfer to CREAM fails: (1259 times)
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-06-13T06:23:42.151Z0cannot write the authN proxy to file: nullcannot write the authN proxy to file: nullorg.glite.ce.faults.AuthenticationFaultcream-34.pd.infn.it]
      • CREAM Start raised exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-06-14T07:00:11.781Z0USER_VO_LABEL not defined in msgContextUSER_VO_LABEL not defined in msgContextorg.glite.ce.faults.AuthenticationFaultcream-32.pd.infn.it]
      • Failed to create a delegation id for job https://devel15.cnaf.infn.it:9000/00-d12pjYkJYAIz_eHUiCg: reason is Connection to service [https://cream-21.pd.infn.it:8443/ce-cream/services/gridsite-delegation] failed: (1257 times)
  icemem12.png
Line: 500 to 514
  -- AlessioGianelle - 05 Mar 2009
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 12" date="1245055204" name="icemem12.png" path="icemem12.png" size="6622" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
Line: 508 to 523
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 11" date="1244023720" name="icemem11.png" path="icemem11.png" size="6434" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Deleted:
<
<
META FILEATTACHMENT attachment="icemem12.png" attr="" comment="Ice graph. Test 12" date="1245055204" name="icemem12.png" path="icemem12.png" size="6622" stream="icemem12.png" user="Main.AlessioGianelle" version="1"

Revision 352009-06-15 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 27 to 27
 
      • Add some more logs to JobWrapper stdout for debugging purpose
Added:
>
>

Partial results taken on Mon Jun 15 at 10:38:25 CEST 2009

icemem12.png

 

11) Test starts on Fri May 29 at 18:20:13 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs
Line: 504 to 508
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 11" date="1244023720" name="icemem11.png" path="icemem11.png" size="6434" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attachment="icemem12.png" attr="" comment="Ice graph. Test 12" date="1245055204" name="icemem12.png" path="icemem12.png" size="6622" stream="icemem12.png" user="Main.AlessioGianelle" version="1"

Revision 342009-06-12 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

12) Test starts on Fri Jun 12 at 15:59:59 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-07.cnaf)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Notification mechanism is not used: start_listener = false;
  • Use "jobdir" input mechanism

  • Changes in the software wrt previous test:
    • ICE
      • fixed bad handling of decrement of the job counter for the better proxy
      • removing delegation if the better proxy is not there anymore
    • CE
      • Update BLAH rpm
      • Add some more logs to JobWrapper stdout for debugging purpose
 

11) Test starts on Fri May 29 at 18:20:13 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs

Revision 332009-06-03 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 21 to 21
 
  • 2788 collections submitted in 32565 seconds: 5/11/77 (min/avg/max)
    • 1532 submission(s) fail(s) due to load limiter
Changed:
<
<
>
>

Final results taken on Wed Jun 3 at 16:23:46 CEST 2009

  • Collections correctly submitted: 2788 (111520 jobs)
    • DONE OK: 110757 (99.318%)
    • NOT TERMINATED: 2 (0.002%)
    • ABORTED: 761* (0.68%)
    • Resubmitted: 5131 (4.6%)

(Note: the 760 jobs aborted are exactly 19 collections which stay in the ice queue too long, so their proxy expired)

  • Errors found (7620)
    • BLAH error: (1030 times)
      • blah error: send command timeout: (41 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:pps: Queue has been closed. Job not submitted.-TERM environment variable not set.-) N/A (887 times)
      • BLAH error: submission command failed (exit code = -15) (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.- killed by signal 15.-) N/A (102 times)
    • Cannot move ISB: (1550 times)
    • Transfer to CREAM fails: (4432 times)
      • Authentication error: Unable to open the file [/var/glite/SandboxDir/zZ/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fzZq20TYizStcltyqfblxwA/user.proxy] : No such file or directory (3358 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Mon 01 Jun 2009 01:18:50] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec [glexec reported = "glexec policy violation: see glexec log for more details"]]" (27 times)
      • Authentication error: The proxy is EXPIRED! (1041 times)
    • Proxy expired: (527 times)
    • Cannot take token: (78 times)
      • /opt/edg/libexec/edg-gridftp-base-rm: error globus_ftp_client: the server responded with an error 421 Service not available, closing control connection Cannot take token (30 times)
      • /opt/edg/libexec/edg-gridftp-base-rm: error globus_ftp_client: the server responded with an error 500 500-Command failed : System error in unlink: No such file or directory 500-A system call failed: No such file or directory 500 End. Cannot take token (3 times)
    • pbs_reason: (3 times)
  icemem11.png
Line: 43 to 66
 
  • 3872 collections submitted in 33435 seconds: 4/8/54 (min/avg/max)
    • 448 submissions fails due to load limiter
Changed:
<
<

Partial results taken on Thu May 28 at 12:04:48 CEST 2009

>
>

Final results taken on Thu May 28 at 12:04:48 CEST 2009

 
  • Collections correctly submitted: 3872 (154880 jobs)
    • DONE OK: 154873 (99.995%)
    • NOT TERMINATED: 7 (0.005%)
Line: 60 to 83
  icemem10.png
Added:
>
>

Open Bugs:

  • #50875: CREAM: reason for cancelled jobs should be reported
  • #50876: CREAM reports that the proxy expired even when the problem is in detecting the lifetime of the proxy
  • #51046: CREAM: DelegProxyInfo info sometimes is wrong
 

9) Test starts on Wed May 22 at 18:01:14 CEST 2009 (WMS: devel20)

Description:

Revision 322009-06-03 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 17 to 17
 
  • Notification mechanism is not used start_listener = false;
  • Use "jobdir" input mechanism
Added:
>
>

Submissions finish on Mon Jun 1 at 18:17:14 CEST 2009

  • 2788 collections submitted in 32565 seconds: 5/11/77 (min/avg/max)
    • 1532 submission(s) fail(s) due to load limiter
 
Added:
>
>
icemem11.png
 

10) Test starts on Tue May 26 at 15:56:53 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs
Line: 444 to 450
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243612371" name="icemem10.png" path="icemem10.png" size="6275" user="Main.AlessioGianelle" version="5"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 11" date="1244023720" name="icemem11.png" path="icemem11.png" size="6434" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"

Revision 312009-05-29 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

11) Test starts on Fri May 29 at 18:20:13 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-07.cnaf)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Notification mechanism is not used start_listener = false;
  • Use "jobdir" input mechanism

 

10) Test starts on Tue May 26 at 15:56:53 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs
Line: 19 to 37
 
  • 3872 collections submitted in 33435 seconds: 4/8/54 (min/avg/max)
    • 448 submissions fails due to load limiter
Deleted:
<
<
 

Partial results taken on Thu May 28 at 12:04:48 CEST 2009

Changed:
<
<
  • Collections correctly submitted: 2475 (99000 jobs)
    • DONE OK: 96188 (97.16%)
    • NOT TERMINATED: 2812 (2.84%)
>
>
  • Collections correctly submitted: 3872 (154880 jobs)
    • DONE OK: 154873 (99.995%)
    • NOT TERMINATED: 7 (0.005%)
 
    • ABORTED: 0 (0%)
Changed:
<
<
    • Resubmitted: 986 (1%)
>
>
    • Resubmitted: 1165 (0.75%)
 
Changed:
<
<
  • Errors found (986)
    • BLAH error: 26
    • Cannot move ISB: 84
    • Proxy expired: 872
>
>
  • Errors found (1165)
    • BLAH error: 185
    • Cannot move ISB: 86
    • Transfer to CREAM fails: 6
    • Proxy expired: 884
 
    • pbs_reason: 4
Line: 424 to 441
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243592423" name="icemem10.png" path="icemem10.png" size="6458" user="Main.AlessioGianelle" version="4"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243612371" name="icemem10.png" path="icemem10.png" size="6275" user="Main.AlessioGianelle" version="5"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"

Revision 302009-05-29 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

10) Test starts on Tue May 26 at 15:56:53 CEST 2009 (WMS: devel20)

Line: 15 to 15
 
  • Lease mechanism is not used
  • Use "jobdir" input mechanism

Added:
>
>

Submissions finish on Fri May 29 at 15:53:26 CEST 2009

  • 3872 collections submitted in 33435 seconds: 4/8/54 (min/avg/max)
    • 448 submissions fails due to load limiter
 

Partial results taken on Thu May 28 at 12:04:48 CEST 2009

  • Collections correctly submitted: 2475 (99000 jobs)
Line: 419 to 424
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="icemem10.png" attr="" comment="Ice graph. Test 10" date="1243592423" name="icemem10.png" path="icemem10.png" size="6458" stream="icemem10.png" user="Main.AlessioGianelle" version="4"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243592423" name="icemem10.png" path="icemem10.png" size="6458" user="Main.AlessioGianelle" version="4"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"

Revision 292009-05-29 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

10) Test starts on Tue May 26 at 15:56:53 CEST 2009 (WMS: devel20)

Line: 419 to 419
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243506253" name="icemem10.png" path="icemem10.png" size="5903" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attachment="icemem10.png" attr="" comment="Ice graph. Test 10" date="1243592423" name="icemem10.png" path="icemem10.png" size="6458" stream="icemem10.png" user="Main.AlessioGianelle" version="4"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"

Revision 282009-05-28 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

10) Test starts on Tue May 26 at 15:56:53 CEST 2009 (WMS: devel20)

Line: 16 to 16
 
  • Use "jobdir" input mechanism

Changed:
<
<

Partial results taken on Wed May 27 at 16:04:48 CEST 2009

  • Collections correctly submitted: 1385 (55400 jobs)
    • DONE OK: 53707 (96.94%)
    • NOT TERMINATED: 1693 (3.06%)
>
>

Partial results taken on Thu May 28 at 12:04:48 CEST 2009

  • Collections correctly submitted: 2475 (99000 jobs)
    • DONE OK: 96188 (97.16%)
    • NOT TERMINATED: 2812 (2.84%)
 
    • ABORTED: 0 (0%)
Changed:
<
<
    • Resubmitted: 983 (1.77%)
>
>
    • Resubmitted: 986 (1%)
 
Changed:
<
<
  • Errors found (983)
>
>
  • Errors found (986)
 
    • BLAH error: 26
Changed:
<
<
    • Cannot move ISB: 81
>
>
    • Cannot move ISB: 84
 
    • Proxy expired: 872
    • pbs_reason: 4
Line: 419 to 419
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243436510" name="icemem10.png" path="icemem10.png" size="6300" user="Main.AlessioGianelle" version="1"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243506253" name="icemem10.png" path="icemem10.png" size="5903" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"

Revision 272009-05-27 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

10) Test starts on Tue May 26 at 15:56:53 CEST 2009 (WMS: devel20)

Line: 16 to 16
 
  • Use "jobdir" input mechanism

Added:
>
>

Partial results taken on Wed May 27 at 16:04:48 CEST 2009

  • Collections correctly submitted: 1385 (55400 jobs)
    • DONE OK: 53707 (96.94%)
    • NOT TERMINATED: 1693 (3.06%)
    • ABORTED: 0 (0%)
    • Resubmitted: 983 (1.77%)

  • Errors found (983)
    • BLAH error: 26
    • Cannot move ISB: 81
    • Proxy expired: 872
    • pbs_reason: 4

icemem10.png

 

9) Test starts on Wed May 22 at 18:01:14 CEST 2009 (WMS: devel20)

Description:
Line: 401 to 417
 -- AlessioGianelle - 05 Mar 2009

META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="icemem9.png" attr="" comment="Ice graph. Test 9" date="1243347092" name="icemem9.png" path="icemem9.png" size="6416" stream="icemem9.png" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 9" date="1243347093" name="icemem9.png" path="icemem9.png" size="6416" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 10" date="1243436510" name="icemem10.png" path="icemem10.png" size="6300" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"

Revision 262009-05-26 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

10) Test starts on Tue May 26 at 15:56:53 CEST 2009 (WMS: devel20)

Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-07.cnaf)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Use "jobdir" input mechanism

 

9) Test starts on Wed May 22 at 18:01:14 CEST 2009 (WMS: devel20)

Description:
Line: 31 to 46
 
  • 2969 collecions submitted in 18907 seconds: 3/6/66 (min/avg/max)
    • 1351 submissions fails duo to load limiter
Changed:
<
<

Partial results taken on Mon May 25 at 17:58:16 CEST 2009

>
>

Final results taken on Tue May 26 at 10:58:16 CEST 2009

 
  • Collections correctly submitted: 2969 (118760 jobs)
Changed:
<
<
    • DONE OK: 110091 (92.7%)
    • NOT TERMINATED: 5145 (4.33%)
    • ABORTED: 3524 (2.97%)
    • Resubmitted: 20944 (17.64%)
>
>
    • DONE OK: 115175 (96.98%)
    • NOT TERMINATED: 0 (0%)
    • ABORTED: 3584 (3.02%)
    • Resubmitted: 22693 (19.11%)

  • Errors found (42042)
    • BLAH error: 26
    • Cannot move ISB: 1186
    • Transfer to CREAM failed due to exception: Authentication error: 20354
    • Proxy expired: 20327
    • Cannot take token: 103
    • lsf_reason: 4
    • pbs_reason: 42
 
Deleted:
<
<
  • Errors found (38457)
  icemem9.png

NOTE:

Changed:
<
<
  • All the errors (and so the aborted) are due to "proxy expired" problem, which happen for a misconfiguration in the proxy renewal service daemon
>
>
  • All the errors (and so the aborted) are due to a "proxy expired" problem, which happen for a misconfiguration in the proxy renewal service daemon.
 

8) Test starts on Mon May 11 at 11:44:15 CEST 2009 (WMS: devel14)

Line: 378 to 401
 -- AlessioGianelle - 05 Mar 2009

META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attachment="icemem9.png" attr="" comment="Ice graph. Test 9" date="1243347092" name="icemem9.png" path="icemem9.png" size="6416" stream="icemem9.png" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Deleted:
<
<
META FILEATTACHMENT attachment="icemem9.png" attr="" comment="Ice graph. Test 9" date="1243268494" name="icemem9.png" path="icemem9.png" size="5793" stream="icemem9.png" user="Main.AlessioGianelle" version="1"

Revision 252009-05-25 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 27 to 27
 
  • Redesign of the PBS section of the testbedB (1 pbs server per CE)
Added:
>
>

Submissions finish on Mon May 25 at 17:58:16 CEST 2009

  • 2969 collecions submitted in 18907 seconds: 3/6/66 (min/avg/max)
    • 1351 submissions fails duo to load limiter

Partial results taken on Mon May 25 at 17:58:16 CEST 2009

  • Collections correctly submitted: 2969 (118760 jobs)
    • DONE OK: 110091 (92.7%)
    • NOT TERMINATED: 5145 (4.33%)
    • ABORTED: 3524 (2.97%)
    • Resubmitted: 20944 (17.64%)
 
Added:
>
>
  • Errors found (38457)
 
Added:
>
>
icemem9.png

NOTE:

  • All the errors (and so the aborted) are due to "proxy expired" problem, which happen for a misconfiguration in the proxy renewal service daemon
  • In the last day we use as MyProxyServer: myproxy.cnaf.infn.it
 

8) Test starts on Mon May 11 at 11:44:15 CEST 2009 (WMS: devel14)

Description:
Line: 366 to 382
 
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attachment="icemem9.png" attr="" comment="Ice graph. Test 9" date="1243268494" name="icemem9.png" path="icemem9.png" size="5793" stream="icemem9.png" user="Main.AlessioGianelle" version="1"

Revision 242009-05-22 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

9) Test starts on Wed May 20 at 17:18:44 CEST 2009 (WMS: devel14)

>
>

9) Test starts on Wed May 22 at 18:01:14 CEST 2009 (WMS: devel20)

  Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds
Line: 14 to 14
 
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used
Added:
>
>
  • Use "jobdir" input mechanism
 
  • Changes in the software wrt previous test:
    • ICE
      • Limiting SELECT queries SQL to 100 records (for poller)
      • Setting on cache size of database to a low value in order to limit heap consumption
Added:
>
>
    • CE
      • Use version 1_11
  • Use a WMS 3.2 (patch 2597)
 
  • Redesign of the PBS section of the testbedB (1 pbs server per CE)

Revision 232009-05-21 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

9) Test starts on Wed May 20 at 17:18:44 CEST 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the CEs of testbedB
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used

  • Changes in the software wrt previous test:
    • ICE
      • Limiting SELECT queries SQL to 100 records (for poller)
      • Setting on cache size of database to a low value in order to limit heap consumption
  • Redesign of the PBS section of the testbedB (1 pbs server per CE)
 

8) Test starts on Mon May 11 at 11:44:15 CEST 2009 (WMS: devel14)

Description:
  • 4320 collections each of 20 jobs

Revision 222009-05-19 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 19 to 19
 

Submissions finish on Thu May 14 at 11:41:46 CEST 2009

  • 4320 collections submitted in 35444 seconds: 4/8/44 (min/avg/max)
Changed:
<
<

Partial results taken on Fri May 15 at 11:41:46

>
>

Final results taken on Tue May 19 at 10:11:16 CEST 2009

 
  • Collections correctly submitted: 4320 (86400 jobs)
Changed:
<
<
    • DONE OK: 58638 (67.87%)
    • NOT TERMINATED: 27450 (31.77%)
>
>
    • DONE OK: 72242 (83.61%)
    • NOT TERMINATED: 13846 (16.03%)
 
    • ABORTED: 312 (0.36%)
    • Resubmitted: 1389 (1.6%)
Line: 40 to 40
 
    • Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times 0.09%)
    • The job cannot be submitted because the blparser service is not alive (2 times 0.06%)
Added:
>
>

NOTE:

  • For the jobs in "not terminated" status, the notification of the "Done Ok" event has been lost.
  • Probably most of the "BLAH errors" are due to a malfunction in the LSF batch system at Cnaf
  icemem8.png

Revision 212009-05-15 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 16 to 16
 
  • Lease mechanism is not used
  • Poller mechanism is not used (start_poller = false;)
Changed:
<
<

Submissions finish on Thu May at 14 11:41:46 CEST 2009

>
>

Submissions finish on Thu May 14 at 11:41:46 CEST 2009

 
  • 4320 collections submitted in 35444 seconds: 4/8/44 (min/avg/max)
Changed:
<
<

Partial results taken on Thu May at 14 15:41:46

>
>

Partial results taken on Fri May 15 at 11:41:46

 
  • Collections correctly submitted: 4320 (86400 jobs)
Changed:
<
<
    • DONE OK: 48874 (56.57%)
    • NOT TERMINATED: 37214 (43.07%)
>
>
    • DONE OK: 58638 (67.87%)
    • NOT TERMINATED: 27450 (31.77%)
 
    • ABORTED: 312 (0.36%)
Changed:
<
<
    • Resubmitted: 1071 (1.24%)
>
>
    • Resubmitted: 1389 (1.6%)
 
Changed:
<
<
  • Errors found (3156)
    • BLAH error: (3150 times 99.81%)
>
>
  • Errors found (3146)
    • BLAH error: (3134 times 99.63%)
 
      • submission command failed (exit code = 1) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Failed in an LSF library call: Failed in sending/receiving a message: Connection reset by peer. Job not submitted.-TERM environment variable not set.-) N/A (jobId = [ ... ] ) (283 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot create job info file. Job not submitted.-TERM environment variable not set.-) N/A (jobId = [ ... ] ) ( times 1858)
      • no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (jobId = [ ... ]) (8 times)
      • submission command failed (exit code = -15) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-LSF is down. Please wait ...-LSF is down. Please wait ...-LSF is down. Please wait ...- exe_getouterr: 200 seconds timeout expired, killing child process.- killed by signal 15.-) N/A (jobId = [ ... ]) (8 times)
Changed:
<
<
      • send command timeout (993 times)
    • lsf reason (5 times 0.16%)
>
>
      • send command timeout (977 times)
    • lsf reason (6 times 0.19%)
 
      • lsf_reason=-1 (5 times)
Added:
>
>
      • lsf_reason=36608 (1 time)
 
    • Cannot take token (1 time 0.03%)
Added:
>
>
    • Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times 0.09%)
    • The job cannot be submitted because the blparser service is not alive (2 times 0.06%)
 

icemem8.png

Line: 330 to 333
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Deleted:
<
<
META FILEATTACHMENT attachment="icemem8.png" attr="" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" stream="icemem8.png" user="Main.AlessioGianelle" version="1"

Revision 202009-05-14 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 16 to 16
 
  • Lease mechanism is not used
  • Poller mechanism is not used (start_poller = false;)
Added:
>
>

Submissions finish on Thu May at 14 11:41:46 CEST 2009

  • 4320 collections submitted in 35444 seconds: 4/8/44 (min/avg/max)

Partial results taken on Thu May at 14 15:41:46

  • Collections correctly submitted: 4320 (86400 jobs)
    • DONE OK: 48874 (56.57%)
    • NOT TERMINATED: 37214 (43.07%)
    • ABORTED: 312 (0.36%)
    • Resubmitted: 1071 (1.24%)

  • Errors found (3156)
    • BLAH error: (3150 times 99.81%)
      • submission command failed (exit code = 1) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Failed in an LSF library call: Failed in sending/receiving a message: Connection reset by peer. Job not submitted.-TERM environment variable not set.-) N/A (jobId = [ ... ] ) (283 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot create job info file. Job not submitted.-TERM environment variable not set.-) N/A (jobId = [ ... ] ) ( times 1858)
      • no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (jobId = [ ... ]) (8 times)
      • submission command failed (exit code = -15) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-LSF is down. Please wait ...-LSF is down. Please wait ...-LSF is down. Please wait ...- exe_getouterr: 200 seconds timeout expired, killing child process.- killed by signal 15.-) N/A (jobId = [ ... ]) (8 times)
      • send command timeout (993 times)
    • lsf reason (5 times 0.16%)
      • lsf_reason=-1 (5 times)
    • Cannot take token (1 time 0.03%)

icemem8.png

 

7) Test starts on Mon May 4 at 15:47:37 CEST 2009 (WMS: devel18)

Description:
  • 4320 collections each of 20 jobs
Line: 307 to 331
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attachment="icemem8.png" attr="" comment="Ice graph. Test 8" date="1242314691" name="icemem8.png" path="icemem8.png" size="6580" stream="icemem8.png" user="Main.AlessioGianelle" version="1"

Revision 192009-05-11 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

8) Test starts on Mon May 11 at 11:44:15 CEST 2009 (WMS: devel14)

Description:
  • 4320 collections each of 20 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the lsf CEs of testbedB
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used
  • Poller mechanism is not used (start_poller = false;)

7) Test starts on Mon May 4 at 15:47:37 CEST 2009 (WMS: devel18)

Description:
  • 4320 collections each of 20 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • Used all the lsf CEs of testbedB
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used

  • Changes in the software wrt previous test:
    • ICE
      • Improvement in the memory management of the SQL queries

Submissions finish on Thu May 7 at 15:45:40 CEST 2009

  • 4317 collections submitted in 48130 seconds: 4/11/62 (min/avg/max)
    • 3 submission(s) fail(s) due to load limiter

Results:

  • Collections correctly submitted: 4317 (86340 jobs)
    • DONE OK: 64334 (74.51%)
    • NOT TERMINATED: 21997 (25.48%)
    • ABORTED: 9 (0.01%)
    • Resubmitted: 250 (0.29%)

  • Errors found (336)
    • BLAH error: (1 time 0.3%)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = CREAM539587168)
    • lsf reason (138 times 41.07%)
      • lsf_reason=1603 (2 times)
      • lsf_reason=-1 (136 times)
    • The job cannot be submitted because the blparser service is not alive (3 times 0.9%)
    • Transfer to CREAM failed due to exception: CREAM Register raised std::exception Connection to service [ ... ] failed: (194 times 57.73%)
 
Added:
>
>

NOTE:

  • Poller's call doesn't work due to a mistake in the code
 

6) Test starts on Thu Apr 23 13:23:35 CEST 2009 (WMS: devel14)

Description:
Line: 48 to 102
 
      • Authentication error: The proxy is EXPIRED! (1700 times)
      • Authentication error: Unable to open the file [ ... ] : No such file or directory (6736 times)
      • CREAM Register raised std::exception Cannot set credentials in the gsoap-plugin context (45 times)
Changed:
<
<
      • CREAM Register raised std::exception Connection to service [https://cream-28.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (11 time)
>
>
      • CREAM Register raised std::exception Connection to service [https://cream-28.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (11 times)
 
      • CREAM Register raised std::exception EOF detected during communication. Probably service closed connection or SOCKET TIMEOUT occurred. (11 times)
      • CREAM Register raised std::exception The endpoint is blacklisted (59 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Mon 27 Apr 2009 13:18:26] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]" (46 times)

Revision 182009-04-30 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 24 to 24
 
      • SQL queries fixed
Changed:
<
<

Partial results taken on Wed Apr 29 12:42:00 CEST 2009

>
>

Results taken on Wed Apr 30 12:42:00 CEST 2009

 
  • Collections correctly submitted: 5787 (231480 jobs)
Changed:
<
<
    • DONE OK: 81896 (35.38%)
    • CANCELLED: 23762 (10.27%)
    • NOT TERMINATED: 125704 (54.3%)
    • ABORTED: 118 (0.05%)
    • Resubmitted: 1160 (0.5%)

  • Errors found (2140)
    • BLAH error: (855 times)
      • submission command failed (exit code = 1) (827 times)
      • submission command failed (exit code = 106) (12 times)
>
>
    • DONE OK: 86978 (37.58%)
    • CANCELLED: 83568 (36.10%)
    • NOT TERMINATED: 59604 (25.75%)
    • ABORTED: 1330 (0.57%)
    • Resubmitted: 1908 (0.82%)

  • Errors found (12814)
    • BLAH error: (3943 times)
      • submission command failed (exit code = 1) (3839 times)
      • submission command failed (exit code = 106) (67 times)
 
      • submission command failed (exit code = 120) (2 times)
Changed:
<
<
      • submission command failed (exit code = -15) (2 times)
      • submission command failed (exit code = 107) (4 times)
>
>
      • submission command failed (exit code = -15) (9 times)
      • submission command failed (exit code = 107) (17 times)
 
      • no jobId in submission script's output (3 time)
Changed:
<
<
      • send command timeout (5 times)
>
>
      • send command timeout (6 times)
 
    • The job cannot be submitted because the blparser service is not alive (25 times)
    • lsf_reason=-1 (4 times)
Changed:
<
<
    • Cannot move ISB (12 times)
    • Transfer to CREAM failed due to exception (1241 times)
      • Authentication error: The proxy is EXPIRED! (419 times)
      • Authentication error: Unable to open the file [ ... ] : No such file or directory (763 times)
      • CREAM Register raised std::exception Cannot set credentials in the gsoap-plugin context (18 times)
      • CREAM Register raised std::exception Connection to service [https://cream-28.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (1 time)
      • CREAM Register raised std::exception EOF detected during communication. Probably service closed connection or SOCKET TIMEOUT occurred. (3 times)
      • CREAM Register raised std::exception The endpoint is blacklisted (26 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Mon 27 Apr 2009 13:18:26] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]" (10 times)
      • CREAM Start raised exception The endpoint is blacklisted (1 time)
    • Proxy is expired (3 times)
>
>
    • Cannot move ISB (171 times)
    • Transfer to CREAM failed due to exception (8613 times)
      • Authentication error: The proxy is EXPIRED! (1700 times)
      • Authentication error: Unable to open the file [ ... ] : No such file or directory (6736 times)
      • CREAM Register raised std::exception Cannot set credentials in the gsoap-plugin context (45 times)
      • CREAM Register raised std::exception Connection to service [https://cream-28.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (11 time)
      • CREAM Register raised std::exception EOF detected during communication. Probably service closed connection or SOCKET TIMEOUT occurred. (11 times)
      • CREAM Register raised std::exception The endpoint is blacklisted (59 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Mon 27 Apr 2009 13:18:26] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]" (46 times)
      • CREAM Start raised exception The endpoint is blacklisted (4 time)
      • CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[java.lang.NullPointerException] - FaultCode=[SOAP-ENV:Server.userException] - FaultSubCode=[SOAP-ENV:Server.userException] - FaultDetail=[cream-28.pd.infn.it] (1 time)
    • Proxy is expired (58 times)
  ice6.png
Added:
>
>

Problems

  • ProxyRenewal service daemon:
    Trying to renew proxy in f50d5ddd407f12c2cc55f102b7eb1f18.1411
    Error contacting MyProxy server for proxy f50d5ddd407f12c2cc55f102b7eb1f18.1411: ERROR from myproxy-server (myproxy.cern.ch):
    certificate chain verification failed
    X509_verify_cert() failed: certificate has expired
    authentication failed
  • Maui:
    04/27 09:59:45 WARNING:  job buffer overflow (cannot add job '1545422')
    04/27 09:59:45 ERROR:    job buffer is full  (ignoring job '1545422.cream-28.pd.infn.it')
 

5) Test starts on Tue Apr 15 at 11:40:01 CEST 2009 (WMS: devel14)

Description:
  • 3400 collections each of 40 jobs
Line: 236 to 251
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="ice6.png" attr="" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="icemem.png" size="8791" stream="icemem.png" user="Main.AlessioGianelle" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="ice6.png" size="8791" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"

Revision 172009-04-30 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 236 to 236
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1240588686" name="ice6.png" path="ice6.png" size="5890" user="Main.AlessioGianelle" version="1"
>
>
META FILEATTACHMENT attachment="ice6.png" attr="" comment="Ice graph. Test 6" date="1241093911" name="ice6.png" path="icemem.png" size="8791" stream="icemem.png" user="Main.AlessioGianelle" version="2"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"

Revision 162009-04-29 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 24 to 24
 
      • SQL queries fixed
Changed:
<
<

Partial results taken on Fri Apr 27 at 11:38:25 CEST 2009

  • Collections correctly submitted: 4704 (188160 jobs)
    • DONE OK: 66122 (35.14%)
    • NOT TERMINATED: 122038 (64.86%)
    • Resubmitted: 182 (0.1%)

  • Errors found (182)
    • BLAH error: (154 times)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = CREAM630084541) (1 time)
      • no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (jobId = CREAM259660302) (1 time)
      • send command timeout (2 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:Connection timed out-qsub: cannot connect to server cream-28.pd.infn.it (errno=110)-TERM environment variable not set.-) N/A (jobId = CREAM396241866) (4 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: Invalid credential MSG=cannot authenticate user-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-TERM environment variable not set.-) N/A (jobId = CREAM032001142) (7 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-TERM environment variable not set.-) N/A (jobId = CREAM003078012) (139 times)
    • The job cannot be submitted because the blparser service is not alive (20 times)
    • lsf_reason=-1 (2 times)
    • Cannot move ISB (6 times)
>
>

Partial results taken on Wed Apr 29 12:42:00 CEST 2009

  • Collections correctly submitted: 5787 (231480 jobs)
    • DONE OK: 81896 (35.38%)
    • CANCELLED: 23762 (10.27%)
    • NOT TERMINATED: 125704 (54.3%)
    • ABORTED: 118 (0.05%)
    • Resubmitted: 1160 (0.5%)

  • Errors found (2140)
    • BLAH error: (855 times)
      • submission command failed (exit code = 1) (827 times)
      • submission command failed (exit code = 106) (12 times)
      • submission command failed (exit code = 120) (2 times)
      • submission command failed (exit code = -15) (2 times)
      • submission command failed (exit code = 107) (4 times)
      • no jobId in submission script's output (3 time)
      • send command timeout (5 times)
    • The job cannot be submitted because the blparser service is not alive (25 times)
    • lsf_reason=-1 (4 times)
    • Cannot move ISB (12 times)
    • Transfer to CREAM failed due to exception (1241 times)
      • Authentication error: The proxy is EXPIRED! (419 times)
      • Authentication error: Unable to open the file [ ... ] : No such file or directory (763 times)
      • CREAM Register raised std::exception Cannot set credentials in the gsoap-plugin context (18 times)
      • CREAM Register raised std::exception Connection to service [https://cream-28.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (1 time)
      • CREAM Register raised std::exception EOF detected during communication. Probably service closed connection or SOCKET TIMEOUT occurred. (3 times)
      • CREAM Register raised std::exception The endpoint is blacklisted (26 times)
      • CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Mon 27 Apr 2009 13:18:26] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]" (10 times)
      • CREAM Start raised exception The endpoint is blacklisted (1 time)
    • Proxy is expired (3 times)
  ice6.png

5) Test starts on Tue Apr 15 at 11:40:01 CEST 2009 (WMS: devel14)

Revision 152009-04-27 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 24 to 24
 
      • SQL queries fixed
Changed:
<
<

Partial results taken on Fri Apr 24 at 15:38:25 CEST 2009

  • Collections correctly submitted: 1590 (63600 jobs)
    • DONE OK: 33303 (52.36%)
    • NOT TERMINATED: 30297 (47.64%)
    • Resubmitted: 5 (0.008%)

>
>

Partial results taken on Fri Apr 27 at 11:38:25 CEST 2009

  • Collections correctly submitted: 4704 (188160 jobs)
    • DONE OK: 66122 (35.14%)
    • NOT TERMINATED: 122038 (64.86%)
    • Resubmitted: 182 (0.1%)

  • Errors found (182)
    • BLAH error: (154 times)
      • submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = CREAM630084541) (1 time)
      • no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (jobId = CREAM259660302) (1 time)
      • send command timeout (2 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:Connection timed out-qsub: cannot connect to server cream-28.pd.infn.it (errno=110)-TERM environment variable not set.-) N/A (jobId = CREAM396241866) (4 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: Invalid credential MSG=cannot authenticate user-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-TERM environment variable not set.-) N/A (jobId = CREAM032001142) (7 times)
      • submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-TERM environment variable not set.-) N/A (jobId = CREAM003078012) (139 times)
    • The job cannot be submitted because the blparser service is not alive (20 times)
    • lsf_reason=-1 (2 times)
    • Cannot move ISB (6 times)
  ice6.png

5) Test starts on Tue Apr 15 at 11:40:01 CEST 2009 (WMS: devel14)

Line: 221 to 224
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Ice graph. Test 6" date="1240588686" name="ice6.png" path="ice6.png" size="5890" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Deleted:
<
<
META FILEATTACHMENT attachment="ice6.png" attr="" comment="Ice graph. Test 6" date="1240588686" name="ice6.png" path="ice6.png" size="5890" stream="ice6.png" user="Main.AlessioGianelle" version="1"

Revision 142009-04-24 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 24 to 24
 
      • SQL queries fixed
Added:
>
>

Partial results taken on Fri Apr 24 at 15:38:25 CEST 2009

  • Collections correctly submitted: 1590 (63600 jobs)
    • DONE OK: 33303 (52.36%)
    • NOT TERMINATED: 30297 (47.64%)
    • Resubmitted: 5 (0.008%)

ice6.png

 

5) Test starts on Tue Apr 15 at 11:40:01 CEST 2009 (WMS: devel14)

Description:
  • 3400 collections each of 40 jobs
Line: 205 to 222
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attachment="ice6.png" attr="" comment="Ice graph. Test 6" date="1240588686" name="ice6.png" path="ice6.png" size="5890" stream="ice6.png" user="Main.AlessioGianelle" version="1"

Revision 132009-04-23 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

6) Test starts on Mon Apr 20 at 12:22:42 CEST 2009 (WMS: devel14)

>
>

6) Test starts on Thu Apr 23 13:23:35 CEST 2009 (WMS: devel14)

  Description:
Changed:
<
<
  • 2880 collections each of 30 jobs
>
>
  • 7200 collections each of 40 jobs
 
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • max_ice_mem = 1048000;
Changed:
<
<
  • Used all the CEs of testbedB (except cert-06.cnaf and the LSF CEs in Padua)
>
>
  • Used all the CEs of testbedB (except cert-06.cnaf and cream-28.pd)
 
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"

Revision 122009-04-20 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

6) Test starts on Mon Apr 20 at 12:22:42 CEST 2009 (WMS: devel14)

Description:
  • 2880 collections each of 30 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • max_ice_mem = 1048000;
  • Used all the CEs of testbedB (except cert-06.cnaf and the LSF CEs in Padua)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used

  • Changes in the software wrt previous test:
    • ICE
      • SQL queries fixed
 

5) Test starts on Tue Apr 15 at 11:40:01 CEST 2009 (WMS: devel14)

Description:
  • 3400 collections each of 40 jobs
Line: 18 to 40
 
  • Changes in the software wrt previous test:
Added:
>
>
    • ICE:
 
    • Improvements in the memory management in database's queries
    • Fixed a bug in the management of notifications

Revision 112009-04-17 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 29 to 29
 

Partial results:

  • Collections correctly submitted: 1535 (61400 jobs)
Changed:
<
<
    • DONE OK: 21695 (35.33%)
    • NOT TERMINATED: 39705 (64.67%)
    • Resubmitted: 1 (0.002%)
>
>
    • DONE OK: 32000 (52.12%)
    • NOT TERMINATED: 29400 (47.88%)
    • Resubmitted: 7 (0.01%)
 
Changed:
<
<
  • Errors found (1):
    • BLAH error: submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-TERM environment variable not set.-) N/A (jobId = CREAM524685673)
>
>
  • Errors found (7):
    • BLAH error: submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-TERM environment variable not set.-) N/A (jobId = CREAM524685673) (7 times)
 

4) Test starts on Fri Apr 10 at 17:10:12 CEST 2009 (WMS: devel14)

Revision 102009-04-16 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 22 to 22
 
    • Fixed a bug in the management of notifications
Added:
>
>

Test interrupted on 2009-04-16 at 15:10:40

  • ICE has exited with this message:
    terminate called after throwing an instance of 'glite::wms::ice::db::DbOperationException'
      what():  Query [UPDATE jobs SET failure_reason='Transfer to CREAM failed due to exception: CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Thu 16 Apr 2009 13:32:10] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]"' WHERE gridjobid='https://devel15.cnaf.infn.it:9000/UV2KNfU-ypvg-tbr6GnrNQ';] failed due to error [near "s": syntax error]

Partial results:

  • Collections correctly submitted: 1535 (61400 jobs)
    • DONE OK: 21695 (35.33%)
    • NOT TERMINATED: 39705 (64.67%)
    • Resubmitted: 1 (0.002%)

  • Errors found (1):
    • BLAH error: submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-TERM environment variable not set.-) N/A (jobId = CREAM524685673)
 

4) Test starts on Fri Apr 10 at 17:10:12 CEST 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs

Revision 92009-04-15 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

5) Test starts on Tue Apr 14 at 15:48:52 CEST 2009 (WMS: devel14)

>
>

5) Test starts on Tue Apr 15 at 11:40:01 CEST 2009 (WMS: devel14)

  Description:
  • 3400 collections each of 40 jobs
  • One collection every 60 seconds
Line: 19 to 19
 
  • Changes in the software wrt previous test:
    • Improvements in the memory management in database's queries
Changed:
<
<
>
>
    • Fixed a bug in the management of notifications
 

4) Test starts on Fri Apr 10 at 17:10:12 CEST 2009 (WMS: devel14)

Revision 82009-04-14 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

5) Test starts on Tue Apr 14 at 15:48:52 CEST 2009 (WMS: devel14)

Description:
  • 3400 collections each of 40 jobs
  • One collection every 60 seconds
  • Four users
  • max_ice_threads = 20
  • max_ice_mem = 2800000;
  • Used all the CEs of testbedB (except cert-06.cnaf and cream-22.pd)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 2424"
  • Resubmission is enabled
  • Lease mechanism is not used

  • Changes in the software wrt previous test:
    • Improvements in the memory management in database's queries

 

4) Test starts on Fri Apr 10 at 17:10:12 CEST 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs
Line: 144 to 164
  -- AlessioGianelle - 05 Mar 2009
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" user="Main.AlessioGianelle" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Deleted:
<
<
META FILEATTACHMENT attachment="ice4.png" attr="" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" stream="ice4.png" user="Main.AlessioGianelle" version="1"

Revision 72009-04-14 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Line: 15 to 15
 
  • Resubmission is enabled
  • Lease mechanism is not used
Added:
>
>

Submissions finish on Mon Apr 13 at 20:35:45 CEST 2009

  • 3608 collections submitted in 118319 seconds: 5/32/213 (min/avg/max)
    • 712 submission(s) fail(s) due to load limiter

Test interrupted on 2009-04-13 at 07:55:40

  • ICE stopped work on the WMS:
    terminate called after throwing an instance of 'std::bad_alloc'
      what():  St9bad_alloc
    Aborted (core dumped)
  • MAUI stopped work on the torque-CEs:
    04/12 05:36:08 ERROR:  job hash table is FULL.  cannot add MJob[107] '1385803'
    04/12 05:36:08 ERROR:    job buffer is full  (ignoring job '1385803.cream-28.pd.infn.it')
    04/12 05:36:08 ERROR:  job hash table is FULL.  cannot add MJob[107] '1385804'
    04/12 05:36:08 ERROR:    job buffer is full  (ignoring job '1385804.cream-28.pd.infn.it')
    04/12 05:36:08 INFO:     35507 PBS jobs detected on RM base
    04/12 05:36:08 INFO:     jobs detected: 35507
    and
    04/12/2009 05:36:09;0080;PBS_Server;Req;dis_request_read;req header bad, dis error 7 (Premature end of message), type=Connect
    04/12/2009 05:36:09;0080;PBS_Server;Req;req_reject;Reject reply code=15056(Bad DIS based Request Protocol MSG=cannot decode message), aux=0, type=Connect, fr
    om @
    04/12/2009 05:36:09;0002;PBS_Server;Req;dis_reply_write;DIS reply failure, -1

Results:

  • Collections correctly submitted: 3608 (144320 jobs)
    • DONE OK: 31687 (21.96%)
    • NOT TERMINATED: 112633 (78.04%)
    • Resubmitted: 92 (0.06%)

ice4.png

 

3) Test starts on Fri Apr 10 at 12:11:20 CEST 2009 (WMS: devel14)

Description:
  • 150 collections each of 15 jobs
Line: 110 to 146
 
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"
Added:
>
>
META FILEATTACHMENT attachment="ice4.png" attr="" comment="Submission rate. Test 4" date="1239704412" name="ice4.png" path="ice4.png" size="5208" stream="ice4.png" user="Main.AlessioGianelle" version="1"

Revision 62009-04-10 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

4) Test starts on Fri Apr 10 at 17:10:12 CEST 2009 (WMS: devel14)

Description:
  • 4320 collections each of 40 jobs
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf.infn.it)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used

3) Test starts on Fri Apr 10 at 12:11:20 CEST 2009 (WMS: devel14)

Description:
  • 150 collections each of 15 jobs
  • Three collections every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf.infn.it)
  • Used automatic-delegation
  • The job is a "sleep 666"
  • Resubmission is enabled
  • Lease mechanism is not used

Submissions finish on Fri Apr 10 at 12:59:04 CEST 2009

  • 150 collections correctly submitted

Final results

  • Collections correctly submitted: 150 (2250 jobs)
    • DONE OK: 2250 (100%)
 

2) Test starts on Thu Apr 9 at 15:57:41 CEST 2009 (WMS: devel14)

Description:
Line: 22 to 59
 

Submissions finish on Thu Apr 9 20:04:05 CEST 2009

Changed:
<
<
  • 250 jobs submitted in 2213 seconds: 4/8/30 (min/avg/max)
>
>
  • 250 collections submitted in 2213 seconds: 4/8/30 (min/avg/max)
 

Final results

  • Collections correctly submitted: 250 (10000 jobs)

Revision 52009-04-10 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

2) Test starts on Thu Apr 9 at 15:57:41 CET 2009 (WMS: devel14)

>
>

2) Test starts on Thu Apr 9 at 15:57:41 CEST 2009 (WMS: devel14)

  Description:
  • 250 collections each of 40 jobs
  • One collection every 60 seconds
Line: 16 to 16
 
  • Resubmission is enabled
  • Lease mechanism is not used
Added:
>
>
  • Changes in the software wrt previous test:
    • fixed a bug in ICE about the decrement counter associated with the "super" proxy
 
Changed:
<
<

1) Test starts on Wed Apr 8 at 16:28:22 CET 2009 (WMS: devel14)

>
>

Submissions finish on Thu Apr 9 20:04:05 CEST 2009

  • 250 jobs submitted in 2213 seconds: 4/8/30 (min/avg/max)

Final results

  • Collections correctly submitted: 250 (10000 jobs)
    • DONE OK: 10000 (100%)
    • Resubmitted: 4 (0.04%)

ice2.png

1) Test starts on Wed Apr 8 at 16:28:22 CEST 2009 (WMS: devel14)

  Description:
  • 250 collections each of 40 jobs
  • One collection every 60 seconds
Line: 49 to 71
  -- AlessioGianelle - 05 Mar 2009
Changed:
<
<
META FILEATTACHMENT attachment="ice1.png" attr="" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" stream="ice1.png" user="Main.AlessioGianelle" version="1"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 2" date="1239350242" name="ice2.png" path="ice2.png" size="4811" user="Main.AlessioGianelle" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" user="Main.AlessioGianelle" version="1"

Revision 42009-04-09 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Added:
>
>

2) Test starts on Thu Apr 9 at 15:57:41 CET 2009 (WMS: devel14)

Description:
  • 250 collections each of 40 jobs
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB (except cert-06.cnaf.infn.it)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 666"
  • Resubmission is enabled
  • Lease mechanism is not used
 

1) Test starts on Wed Apr 8 at 16:28:22 CET 2009 (WMS: devel14)

Description:
  • 250 collections each of 40 jobs
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
Changed:
<
<
  • Used all the CEs of testbedB from Padua (plus cert-07.cnaf.infn.it)
>
>
  • Used all the CEs of testbedB (except cert-06.cnaf.infn.it)
 
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 616"

Revision 32009-04-09 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"

TESTs on ICE

Changed:
<
<

1) Test starts on Mon Apr 6 at 17:31:02 CET 2009 (WMS: devel14)

>
>

1) Test starts on Wed Apr 8 at 16:28:22 CET 2009 (WMS: devel14)

  Description:
Changed:
<
<
  • 500 collections each of 40 jobs
>
>
  • 250 collections each of 40 jobs
 
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB from Padua (plus cert-07.cnaf.infn.it)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
Changed:
<
<
  • The job is a "sleep 4242"
>
>
  • The job is a "sleep 616"
 
  • Resubmission is enabled
  • Lease mechanism is not used
Added:
>
>

Submissions finish on Wed Apr 8 at 20:34:44 CEST 2009

  • 250 collections submitted in 2030 seconds: 5/8/29 (min/avg/max)

Final results

  • Collections correctly submitted: 250 (10000 jobs)
    • DONE OK: 10000 (100%)
    • Resubmitted: 555 (5.55%)

  • Errors found (561):
    • Cannot move ISB (232 times 41.36%)
    • Cannot move OSB (79 times 14.08%)
    • Proxy is expired (216 times 38.5%)
    • lsf_reason=1603 (34 times 6.06%)

ice1.png

  -- AlessioGianelle - 05 Mar 2009 \ No newline at end of file
Added:
>
>
META FILEATTACHMENT attachment="ice1.png" attr="" comment="Submission rate. Test 1" date="1239267400" name="ice1.png" path="ice1.png" size="5273" stream="ice1.png" user="Main.AlessioGianelle" version="1"

Revision 22009-04-06 - AlessioGianelle

Line: 1 to 1
 
META TOPICPARENT name="TestWokPlan"
Added:
>
>

TESTs on ICE

 
Changed:
<
<
>
>

1) Test starts on Mon Apr 6 at 17:31:02 CET 2009 (WMS: devel14)

Description:
  • 500 collections each of 40 jobs
  • One collection every 60 seconds
  • Five users
  • max_ice_threads = 20
  • Used all the CEs of testbedB from Padua (plus cert-07.cnaf.infn.it)
  • Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
  • Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
  • The job is a "sleep 4242"
  • Resubmission is enabled
  • Lease mechanism is not used
 

-- AlessioGianelle - 05 Mar 2009 \ No newline at end of file

Revision 12009-03-05 - AlessioGianelle

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="TestWokPlan"

-- AlessioGianelle - 05 Mar 2009

 
This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback