Check bugs:

BUG #47612: There's a mem leak in ICE in the handling the returned sequence code HOPEFULLY FIXED
- Not reproducible

: ICE should provide info to be used by WMSMonitor FIXED

There is a new tool: /opt/glite/bin/queryDb

USAGE: queryDb --conf|-c <WMS CONFIGURATION FILE> [options]

options: 
  --verbose|-v          Verbose output (print each db's record
  --status-filter|-s    Select only records in which the status column is one
                        of those specified as option argument; more states can
                        be ',' separated and they must be:
                        REGISTERED
                        PENDING
                        IDLE
                        RUNNING
                        REALLY_RUNNING
                        CANCELLED
                        HELD
                        ABORTED
                        DONE_OK
                        DONE_FAILED
                        UNKNOWN
                        PURGED

  --userdn|-u           Print the USERDN column of the job table
  --creamjobid|-C       Print the CREAM JOB ID column of the job table
  --gridjobid|-G        Print the GRID JOB ID column of the job table
  --userproxy|-p        Print the USER PROXY column of the job table
  --cream-url|-r        Print the CREAM URL column of the job table
  --myproxy-url|-m      Print the MYPROXY URL column of the job table
  --status|-S           Print the STATUS column of the job table
  --lease-id|-L         Print the LEASE-ID column of the job table
  --delegation-id|-D    Print the DELEGATION-ID column of the job table
  --proxy-exptime|-t    Print the PROXY EXPIRATION TIME column of the job table
  --worker-node|-w      Print the WORKER-NODE column of the job table
  --help|-h             Print this help

BUG #47911: Performance problems in ICE when there many (thousands) proxy files to manage HOPEFULLY FIXED
- See tests below

BUG #48905: Failure in starting ICE if its output jobdir doesn't exist FIXED
- In the WMS configuration file, section "WorkloadManager", set DispatcherType = "jobdir"
- Make sure that the file/directory pointed by the Input="..." attribute does not exis
- Now, start ICE and see if the directory pointed by the Input="..." attribute gets created automatically.

BUG #52884: ICE uses too much memory HOPEFULLY FIXED
- See tests below

TESTs on ICE

16) Test starts on Tue Jul at 7 09:27:28 CEST 2009 (WMS: devel20)

Description:

2880 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 10
Used all the CEs of testbedB (except cert-07.cnaf and cream-25.pd)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 4242"
Resubmission is enabled
Lease mechanism is not used
Notification mechanism is NOT used: start_listener = false;
Use "jobdir" input mechanism

Changes in the software wrt previous test:
- ICE
  - Retrying some times on generic database exception before giving up
  - Fixed several SQL syntax error
  - Sleeping less time between a request fetch and subsequent... (useful when running with few threads)
  - Loading # of request from filelist/jobdir equal to # of threads * Do not drop notification

Submissions finish on Thu Jul 9 at 09:28:42 CEST 2009

2870 collections submitted in 17460 seconds: 3/6/40 (min/avg/max)
- 10 submissions fail

Partial results taken on Fry Jul 10 at 10:57:09 CEST 2009

Collections correctly submitted: 2870 (114800 jobs)
- DONE OK: 96950 (84.45%)
- NOT TERMINATED: 17850 (15.55%)
- ABORTED: 0 (0%)
- Resubmitted: 475 (0.41%)

Errors found (478)
- BLAH error (1 time)
  - submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = CREAM469174928)
- Cannot take token (2 times)
- lsf_reason=1603 (1 time)
- Cannot move ISB [...] : proxy expired (326 times) *
- Transfer to CREAM failed due to exception (38 times)
  - Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-07-07T18:13:00.260Z0cannot write the authN proxy to file: nullcannot write the authN proxy to file: nullorg.glite.ce.faults.AuthenticationFaultcream-34.pd.infn.it] (1 time)
  - CREAM Register raised std::exception Connection to service [https://cream-XX.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (33 times)
  - CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times)
  - CREAM Start raised exception Connection to service [https://cream-33.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (1 time)
- Proxy is expired; Proxy expired: job killed Terminated Master process killed (30 times) *
- pbs_reason=1 (80 times) *

NOTE

Open bug #52934 for *
Open bug #52937

15) Test starts on Fri Jul 3 at 15:41:35 CEST 2009 (WMS: devel20)

Description:

1440 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 7
Used all the CEs of testbedB (except cert-07.cnaf and cream-25.pd)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 4242"
Resubmission is enabled
Lease mechanism is not used
Notification mechanism is NOT used: start_listener = false;
Use "jobdir" input mechanism

Submissions finish on Sat Jul 4 at 15:39:00 CEST 2009

902 collections submitted in 13717 seconds: 9/15/72 (min/avg/max)
- 538 submissions fail

Final results taken on Fry Jul 6 at 09:42:38 CEST 2009

Collections correctly submitted: 902 (36080 jobs)
- DONE OK: 36080 (100%)
- NOT TERMINATED: 0 (0%)
- ABORTED: 0 (0%)
- Resubmitted: 3 (0.008%)

Errors found (3)
- Proxy is expired; Proxy expired: job killed Terminated Master process killed
  - https://devel15.cnaf.infn.it:9000/QKr_yraP3i9swoS8sfQlDw
- Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client]
  - https://devel15.cnaf.infn.it:9000/rCVZEFQRNAxcUp_qIx7uxg
- Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-07-03T19:59:40.008Z0USER_VO_LABEL not defined in msgContextUSER_VO_LABEL not defined in msgContextorg.glite.ce.faults.AuthenticationFaultcert-05.cnaf.infn.it]
  - https://devel15.cnaf.infn.it:9000/U2jU66zK2JqK7-b9Yl0FMg

14) Test starts on Thu Jun 25 at 15:24:56 CEST 2009 (WMS: devel20)

Description:

7200 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
Used all the CEs of testbedB (except cert-07.cnaf and cream-25.pd)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 4242"
Resubmission is enabled
Lease mechanism is not used
Notification mechanism is used: start_listener = true;
Use "jobdir" input mechanism

Submissions finish on Wed Jul 1 at 05:11:15 CEST 2009

4573 collections submitted in 124424 seconds: 6/27/505 (min/avg/max)
- 2627 submissions fail due to load limiter
- 10 submissions fail at wmproxy level due to a LB error

Final results taken on Fry Jul 3 at 15:42:38 CEST 2009

Collections correctly submitted: 4533 (182520 jobs)
- DONE OK: 181893 (99.66%)
- NOT TERMINATED: 0 (0%)
- ABORTED: 627 * (0.34%)
- Resubmitted: 3150 (1.73%)

Errors found (5302)
- BLAH error (17 times)
  - blah error: send command timeout (14 time)
  - no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (3 times )
- Cannot move ISB: (2540 times)
  - [...] Error with GSI credential globus_sysconfig: Could not find a valid trusted CA certificates directory globus_sysconfig: Error getting password entry for current user: Error occured for uid: 1656 [...] (298 times)
  - [...] : proxy expired (2160 times)
- Transfer to CREAM failed due to exception: (2134 times)
  - Authentication error: Unable to open the file [/var/glite/SandboxDir/I6/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fI6T7Z-FDo9uzJ_5f3GmAPXgw/user.proxy] : No such file or directory (2107 times) *
  - CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (4 times)
  - CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Tue 30 Jun 2009 14:16:53] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec [glexec reported = "glexec policy violation: see glexec log for more details"]]" (13 times)
  - CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Sat 27 Jun 2009 13:51:57] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction] (10 times)
- Proxy is expired (312 times)
- pbs_reason=1 (299 times)

NOTE

Open bug #52710 for *

13) Test starts on Jun 17 at 18:06:55 CEST 2009 (WMS: devel20)

Description:

7200 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
Used all the CEs of testbedB (except cert-07.cnaf)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 4242"
Resubmission is enabled
Lease mechanism is not used
Notification mechanism is not used: start_listener = false;
Use "jobdir" input mechanism

Changes in the software wrt previous test:
- ICE
  - fixed a wrong sql query
  - Transaction to database is tried indefinitely (no more DbLockedException can be thrown)
  - new SQL commands to get the terminated jobs from the database
  - new indices on table jobs

Submissions finish on Mon Jun at 22 18:06:38 CEST 2009

5220 collections submitted in 85950 seconds: 5/16/77 (min/avg/max)
- 1980 submissions fail due to load limiter

Final results

Collections correctly submitted: 5220 (208800 jobs)
- DONE OK: 207628 (99.44%)
- NOT TERMINATED: 910 (0.44%)
- ABORTED: 262 (0.12%)
- Resubmitted: 3398 (1.63%)

Errors found (4500)
- Cannot move ISB: (322 times)
- Transfer to CREAM failed due to exception: (3896 times)
  - CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Wed 17 Jun 2009 22:42:08] ErrorCode=[0] Description=[system error] FaultCause=[Batch System lsf not supported!]" (982 times)
  - CREAM Register raised std::exception The endpoint is blacklisted (1794)
  - CREAM Start raised exception The endpoint is blacklisted (30 times)
  - CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Mon 22 Jun 2009 09:22:49] ErrorCode=[0] Description=[system error] FaultCause=[org.glite.ce.common.db.DatabaseException: Rollback executed due to: Lock wait timeout exceeded; try restarting transaction]" (164 time)
  - Authentication error: Unable to open the file [/var/glite/SandboxDir/XP/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fXPAysgaAaUdQrjs2h_5fXg1g/user.proxy] : No such file or directory (922 times)
  - CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (4 times)
- pbs_reason (16 times)
  - pbs_reason=1 (13 times)
  - pbs_reason=271 (3 times)
- Proxy is expired (251 times)
- BLAH error (15 times)
  - exit code = -15 (12 times)
  - exit code = 1 (2 times)
  - exit code = 120 (1 time)

NOTE:

The 982 errors "Batch System lsf not supported" are due to a malconfigured CE wrongly attached to the BDII of the testbed
The 910 "non terminated" jobs are matched to cream-25.pd.infn.it which dyes (due to an hardware problem) during the test
The error "The endpoint is blacklisted" means that a timeout occurs during the CE connection

12) Test starts on Fri Jun 12 at 15:59:59 CEST 2009 (WMS: devel20)

Description:

4320 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
Used all the CEs of testbedB (except cert-07.cnaf)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 8 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 2424"
Resubmission is enabled
Lease mechanism is not used
Notification mechanism is not used: start_listener = false;
Use "jobdir" input mechanism

Changes in the software wrt previous test:
- ICE
  - fixed bad handling of decrement of the job counter for the better proxy
  - removing delegation if the better proxy is not there anymore
- CE
  - Update BLAH rpm
  - Add some more logs to JobWrapper stdout for debugging purpose

Submissions finish on Mon Jun 15 at 15:57:55 CEST 2009

4219 collections submitted in 39736 seconds: 4/9/97 (min/avg/max)
- 101 submissions fail due to load limiter

Final results

Collections correctly submitted: 4219 (168760 jobs)
- DONE OK: 168748 (99.99%)
- NOT TERMINATED: 12 (0.01%)
- ABORTED: 0 (0%)
- Resubmitted: 2414 (1.43%)

Errors found (2446)
- Cannot move ISB: (18 times)
- Transfer to CREAM fails: (1948 times)
  - CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-06-13T06:23:42.151Z0cannot write the authN proxy to file: nullcannot write the authN proxy to file: nullorg.glite.ce.faults.AuthenticationFaultcream-34.pd.infn.it]
  - CREAM Start raised exception Received NULL fault; the error is due to another cause: FaultString=[] - FaultCode=[SOAP-ENV:Server.generalException] - FaultSubCode=[SOAP-ENV:Server.generalException] - FaultDetail=[invoke2009-06-14T07:00:11.781Z0USER_VO_LABEL not defined in msgContextUSER_VO_LABEL not defined in msgContextorg.glite.ce.faults.AuthenticationFaultcream-32.pd.infn.it]
  - Failed to create a delegation id for job https://devel15.cnaf.infn.it:9000/00-d12pjYkJYAIz_eHUiCg: reason is Connection to service [https://cream-21.pd.infn.it:8443/ce-cream/services/gridsite-delegation] failed: (1946 times)
- Cannot take token (3 times)
- lsf_reason=65280
- pbs_reason=271 (3 times)
- Proxy is expired (473 times)

11) Test starts on Fri May 29 at 18:20:13 CEST 2009 (WMS: devel20)

Description:

4320 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
Used all the CEs of testbedB (except cert-07.cnaf)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 2424"
Resubmission is enabled
Lease mechanism is not used
Notification mechanism is not used start_listener = false;
Use "jobdir" input mechanism

Submissions finish on Mon Jun 1 at 18:17:14 CEST 2009

2788 collections submitted in 32565 seconds: 5/11/77 (min/avg/max)
- 1532 submission(s) fail(s) due to load limiter

Final results taken on Wed Jun 3 at 16:23:46 CEST 2009

Collections correctly submitted: 2788 (111520 jobs)
- DONE OK: 110757 (99.318%)
- NOT TERMINATED: 2 (0.002%)
- ABORTED: 761* (0.68%)
- Resubmitted: 5131 (4.6%)

(Note: the 760 jobs aborted are exactly 19 collections which stay in the ice queue too long, so their proxy expired)

Errors found (7620)
- BLAH error: (1030 times)
  - blah error: send command timeout: (41 times)
  - submission command failed (exit code = 1) (stdout:) (stderr:pps: Queue has been closed. Job not submitted.-TERM environment variable not set.-) N/A (887 times)
  - BLAH error: submission command failed (exit code = -15) (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.- killed by signal 15.-) N/A (102 times)
- Cannot move ISB: (1550 times)
- Transfer to CREAM fails: (4432 times)
  - Authentication error: Unable to open the file [/var/glite/SandboxDir/zZ/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fzZq20TYizStcltyqfblxwA/user.proxy] : No such file or directory (3358 times)
  - CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Mon 01 Jun 2009 01:18:50] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec [glexec reported = "glexec policy violation: see glexec log for more details"]]" (27 times)
  - Authentication error: The proxy is EXPIRED! (1041 times)
- Proxy expired: (527 times)
- Cannot take token: (78 times)
  - /opt/edg/libexec/edg-gridftp-base-rm: error globus_ftp_client: the server responded with an error 421 Service not available, closing control connection Cannot take token (30 times)
  - /opt/edg/libexec/edg-gridftp-base-rm: error globus_ftp_client: the server responded with an error 500 500-Command failed : System error in unlink: No such file or directory 500-A system call failed: No such file or directory 500 End. Cannot take token (3 times)
- pbs_reason: (3 times)

10) Test starts on Tue May 26 at 15:56:53 CEST 2009 (WMS: devel20)

Description:

4320 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
Used all the CEs of testbedB (except cert-07.cnaf)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 2424"
Resubmission is enabled
Lease mechanism is not used
Use "jobdir" input mechanism

Submissions finish on Fri May 29 at 15:53:26 CEST 2009

3872 collections submitted in 33435 seconds: 4/8/54 (min/avg/max)
- 448 submissions fails due to load limiter

Final results taken on Thu May 28 at 12:04:48 CEST 2009

Collections correctly submitted: 3872 (154880 jobs)
- DONE OK: 154873 (99.995%)
- NOT TERMINATED: 7 (0.005%)
- ABORTED: 0 (0%)
- Resubmitted: 1165 (0.75%)

Errors found (1165)
- BLAH error: 185
- Cannot move ISB: 86
- Transfer to CREAM fails: 6
- Proxy expired: 884
- pbs_reason: 4

Open Bugs:

#50875: CREAM: reason for cancelled jobs should be reported
#50876: CREAM reports that the proxy expired even when the problem is in detecting the lifetime of the proxy
#51046: CREAM: DelegProxyInfo info sometimes is wrong

9) Test starts on Wed May 22 at 18:01:14 CEST 2009 (WMS: devel20)

Description:

4320 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
Used all the CEs of testbedB
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 2424"
Resubmission is enabled
Lease mechanism is not used
Use "jobdir" input mechanism

Changes in the software wrt previous test:
- ICE
  - Limiting SELECT queries SQL to 100 records (for poller)
  - Setting on cache size of database to a low value in order to limit heap consumption
- CE
  - Use version 1_11
Use a WMS 3.2 (patch 2597)
Redesign of the PBS section of the testbedB (1 pbs server per CE)

Submissions finish on Mon May 25 at 17:58:16 CEST 2009

2969 collecions submitted in 18907 seconds: 3/6/66 (min/avg/max)
- 1351 submissions fails duo to load limiter

Final results taken on Tue May 26 at 10:58:16 CEST 2009

Collections correctly submitted: 2969 (118760 jobs)
- DONE OK: 115175 (96.98%)
- NOT TERMINATED: 0 (0%)
- ABORTED: 3584 (3.02%)
- Resubmitted: 22693 (19.11%)

Errors found (42042)
- BLAH error: 26
- Cannot move ISB: 1186
- Transfer to CREAM failed due to exception: Authentication error: 20354
- Proxy expired: 20327
- Cannot take token: 103
- lsf_reason: 4
- pbs_reason: 42

NOTE:

All the errors (and so the aborted) are due to a "proxy expired" problem, which happen for a misconfiguration in the proxy renewal service daemon.
In the last day we use as MyProxyServer: myproxy.cnaf.infn.it

8) Test starts on Mon May 11 at 11:44:15 CEST 2009 (WMS: devel14)

Description:

4320 collections each of 20 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
Used all the lsf CEs of testbedB
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 2424"
Resubmission is enabled
Lease mechanism is not used
Poller mechanism is not used (start_poller = false;)

Submissions finish on Thu May 14 at 11:41:46 CEST 2009

4320 collections submitted in 35444 seconds: 4/8/44 (min/avg/max)

Final results taken on Tue May 19 at 10:11:16 CEST 2009

Collections correctly submitted: 4320 (86400 jobs)
- DONE OK: 72242 (83.61%)
- NOT TERMINATED: 13846 (16.03%)
- ABORTED: 312 (0.36%)
- Resubmitted: 1389 (1.6%)

Errors found (3146)
- BLAH error: (3134 times 99.63%)
  - submission command failed (exit code = 1) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Failed in an LSF library call: Failed in sending/receiving a message: Connection reset by peer. Job not submitted.-TERM environment variable not set.-) N/A (jobId = [ ... ] ) (283 times)
  - submission command failed (exit code = 1) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot create job info file. Job not submitted.-TERM environment variable not set.-) N/A (jobId = [ ... ] ) ( times 1858)
  - no jobId in submission script's output (stdout:) (stderr: exe_getouterr: 200 seconds timeout expired, killing child process.-) N/A (jobId = [ ... ]) (8 times)
  - submission command failed (exit code = -15) (stdout:) (stderr:Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-Cannot connect to LSF. Please wait ...-LSF is down. Please wait ...-LSF is down. Please wait ...-LSF is down. Please wait ...- exe_getouterr: 200 seconds timeout expired, killing child process.- killed by signal 15.-) N/A (jobId = [ ... ]) (8 times)
  - send command timeout (977 times)
- lsf reason (6 times 0.19%)
  - lsf_reason=-1 (5 times)
  - lsf_reason=36608 (1 time)
- Cannot take token (1 time 0.03%)
- Transfer to CREAM failed due to exception: CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[Client fault] - FaultCode=[SOAP-ENV:Client] - FaultSubCode=[SOAP-ENV:Client] (3 times 0.09%)
- The job cannot be submitted because the blparser service is not alive (2 times 0.06%)

NOTE:

For the jobs in "not terminated" status, the notification of the "Done Ok" event has been lost.
Probably most of the "BLAH errors" are due to a malfunction in the LSF batch system at Cnaf

7) Test starts on Mon May 4 at 15:47:37 CEST 2009 (WMS: devel18)

Description:

4320 collections each of 20 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
Used all the lsf CEs of testbedB
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 2424"
Resubmission is enabled
Lease mechanism is not used

Changes in the software wrt previous test:
- ICE
  - Improvement in the memory management of the SQL queries

Submissions finish on Thu May 7 at 15:45:40 CEST 2009

4317 collections submitted in 48130 seconds: 4/11/62 (min/avg/max)
- 3 submission(s) fail(s) due to load limiter

Results:

Collections correctly submitted: 4317 (86340 jobs)
- DONE OK: 64334 (74.51%)
- NOT TERMINATED: 21997 (25.48%)
- ABORTED: 9 (0.01%)
- Resubmitted: 250 (0.29%)

Errors found (336)
- BLAH error: (1 time 0.3%)
  - submission command failed (exit code = 120) (stdout:) (stderr:glexec policy violation: see glexec log for more details-) N/A (jobId = CREAM539587168)
- lsf reason (138 times 41.07%)
  - lsf_reason=1603 (2 times)
  - lsf_reason=-1 (136 times)
- The job cannot be submitted because the blparser service is not alive (3 times 0.9%)
- Transfer to CREAM failed due to exception: CREAM Register raised std::exception Connection to service [ ... ] failed: (194 times 57.73%)

NOTE:

Poller's call doesn't work due to a mistake in the code

6) Test starts on Thu Apr 23 13:23:35 CEST 2009 (WMS: devel14)

Description:

7200 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
max_ice_mem = 1048000;
Used all the CEs of testbedB (except cert-06.cnaf and cream-28.pd)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 2424"
Resubmission is enabled
Lease mechanism is not used

Changes in the software wrt previous test:
- ICE
  - SQL queries fixed

Results taken on Wed Apr 30 12:42:00 CEST 2009

Collections correctly submitted: 5787 (231480 jobs)
- DONE OK: 86978 (37.58%)
- CANCELLED: 83568 (36.10%)
- NOT TERMINATED: 59604 (25.75%)
- ABORTED: 1330 (0.57%)
- Resubmitted: 1908 (0.82%)

Errors found (12814)
- BLAH error: (3943 times)
  - submission command failed (exit code = 1) (3839 times)
  - submission command failed (exit code = 106) (67 times)
  - submission command failed (exit code = 120) (2 times)
  - submission command failed (exit code = -15) (9 times)
  - submission command failed (exit code = 107) (17 times)
  - no jobId in submission script's output (3 time)
  - send command timeout (6 times)
- The job cannot be submitted because the blparser service is not alive (25 times)
- lsf_reason=-1 (4 times)
- Cannot move ISB (171 times)
- Transfer to CREAM failed due to exception (8613 times)
  - Authentication error: The proxy is EXPIRED! (1700 times)
  - Authentication error: Unable to open the file [ ... ] : No such file or directory (6736 times)
  - CREAM Register raised std::exception Cannot set credentials in the gsoap-plugin context (45 times)
  - CREAM Register raised std::exception Connection to service [https://cream-28.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (11 times)
  - CREAM Register raised std::exception EOF detected during communication. Probably service closed connection or SOCKET TIMEOUT occurred. (11 times)
  - CREAM Register raised std::exception The endpoint is blacklisted (59 times)
  - CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Mon 27 Apr 2009 13:18:26] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]" (46 times)
  - CREAM Start raised exception The endpoint is blacklisted (4 time)
  - CREAM Register raised std::exception Received NULL fault; the error is due to another cause: FaultString=[java.lang.NullPointerException] - FaultCode=[SOAP-ENV:Server.userException] - FaultSubCode=[SOAP-ENV:Server.userException] - FaultDetail=[cream-28.pd.infn.it] (1 time)
- Proxy is expired (58 times)

Problems

ProxyRenewal service daemon:

Trying to renew proxy in f50d5ddd407f12c2cc55f102b7eb1f18.1411
Error contacting MyProxy server for proxy f50d5ddd407f12c2cc55f102b7eb1f18.1411: ERROR from myproxy-server (myproxy.cern.ch):
certificate chain verification failed
X509_verify_cert() failed: certificate has expired
authentication failed

Maui:

04/27 09:59:45 WARNING:  job buffer overflow (cannot add job '1545422')
04/27 09:59:45 ERROR:    job buffer is full  (ignoring job '1545422.cream-28.pd.infn.it')

5) Test starts on Tue Apr 15 at 11:40:01 CEST 2009 (WMS: devel14)

Description:

3400 collections each of 40 jobs
One collection every 60 seconds
Four users
max_ice_threads = 20
max_ice_mem = 2800000;
Used all the CEs of testbedB (except cert-06.cnaf and cream-22.pd)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 2424"
Resubmission is enabled
Lease mechanism is not used

Changes in the software wrt previous test:
- ICE:
  - Improvements in the memory management in database's queries
  - Fixed a bug in the management of notifications

Test interrupted on 2009-04-16 at 15:10:40

ICE has exited with this message:

terminate called after throwing an instance of 'glite::wms::ice::db::DbOperationException'
  what():  Query [UPDATE jobs SET failure_reason='Transfer to CREAM failed due to exception: CREAM Register returned error "MethodName=[jobRegister] Timestamp=[Thu 16 Apr 2009 13:32:10] ErrorCode=[0] Description=[system error] FaultCause=[cannot create the job's working directory! The problem seems to be related to glexec]"' WHERE gridjobid='https://devel15.cnaf.infn.it:9000/UV2KNfU-ypvg-tbr6GnrNQ';] failed due to error [near "s": syntax error]

Partial results:

Collections correctly submitted: 1535 (61400 jobs)
- DONE OK: 32000 (52.12%)
- NOT TERMINATED: 29400 (47.88%)
- Resubmitted: 7 (0.01%)

Errors found (7):
- BLAH error: submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-TERM environment variable not set.-) N/A (jobId = CREAM524685673) (7 times)
  - https://devel15.cnaf.infn.it:9000/6fozSUIvPst6M79lNYtWjw

4) Test starts on Fri Apr 10 at 17:10:12 CEST 2009 (WMS: devel14)

Description:

4320 collections each of 40 jobs
One collection every 60 seconds
Five users
max_ice_threads = 20
Used all the CEs of testbedB (except cert-06.cnaf.infn.it)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 4242"
Resubmission is enabled
Lease mechanism is not used

Submissions finish on Mon Apr 13 at 20:35:45 CEST 2009

3608 collections submitted in 118319 seconds: 5/32/213 (min/avg/max)
- 712 submission(s) fail(s) due to load limiter

Test interrupted on 2009-04-13 at 07:55:40

ICE stopped work on the WMS:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  St9bad_alloc
Aborted (core dumped)

MAUI stopped work on the torque-CEs:

04/12 05:36:08 ERROR:  job hash table is FULL.  cannot add MJob[107] '1385803'
04/12 05:36:08 ERROR:    job buffer is full  (ignoring job '1385803.cream-28.pd.infn.it')
04/12 05:36:08 ERROR:  job hash table is FULL.  cannot add MJob[107] '1385804'
04/12 05:36:08 ERROR:    job buffer is full  (ignoring job '1385804.cream-28.pd.infn.it')
04/12 05:36:08 INFO:     35507 PBS jobs detected on RM base
04/12 05:36:08 INFO:     jobs detected: 35507

and

04/12/2009 05:36:09;0080;PBS_Server;Req;dis_request_read;req header bad, dis error 7 (Premature end of message), type=Connect
04/12/2009 05:36:09;0080;PBS_Server;Req;req_reject;Reject reply code=15056(Bad DIS based Request Protocol MSG=cannot decode message), aux=0, type=Connect, fr
om @
04/12/2009 05:36:09;0002;PBS_Server;Req;dis_reply_write;DIS reply failure, -1

Results:

Collections correctly submitted: 3608 (144320 jobs)
- DONE OK: 31687 (21.96%)
- NOT TERMINATED: 112633 (78.04%)
- Resubmitted: 92 (0.06%)

Errors found (92):
- BLAH error: submission command failed (exit code = 1) (stdout:) (stderr:pbs_iff: cannot read reply from pbs_server-No Permission.-qsub: cannot connect to server cream-28.pd.infn.it (errno=15007)-TERM environment variable not set.-) N/A (jobId = [...]) (91 times)
  - https://devel15.cnaf.infn.it:9000/XaxkR1YiqZbx3N0LifLufw
  - https://devel15.cnaf.infn.it:9000/5SZ3u6IOpnWmCqOc8KabGQ
- Transfer to CREAM failed due to exception: CREAM Register raised std::exception Connection to service [https://cream-32.pd.infn.it:8443/ce-cream/services/CREAM2] failed: (1 time)
  - https://devel15.cnaf.infn.it:9000/FJw7l76KUnMBhtWbcRgM4A

3) Test starts on Fri Apr 10 at 12:11:20 CEST 2009 (WMS: devel14)

Description:

150 collections each of 15 jobs
Three collections every 60 seconds
Five users
max_ice_threads = 20
Used all the CEs of testbedB (except cert-06.cnaf.infn.it)
Used automatic-delegation

Proxy renewal service:
- 50 collections use (MyProxyServer = "myproxy.cern.ch")
- 50 collections use (MyProxyServer = "myproxy.to.infn.it")
- 50 collections don`t set MyProxyServer attribute

The job is a "sleep 666"
Resubmission is enabled
Lease mechanism is not used

Submissions finish on Fri Apr 10 at 12:59:04 CEST 2009

150 collections correctly submitted

Final results

Collections correctly submitted: 150 (2250 jobs)
- DONE OK: 2250 (100%)

2) Test starts on Thu Apr 9 at 15:57:41 CEST 2009 (WMS: devel14)

Description:

250 collections each of 40 jobs
One collection every 60 seconds
Five users
max_ice_threads = 20
Used all the CEs of testbedB (except cert-06.cnaf.infn.it)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 666"
Resubmission is enabled
Lease mechanism is not used

Changes in the software wrt previous test:
- fixed a bug in ICE about the decrement counter associated with the "super" proxy

Submissions finish on Thu Apr 9 20:04:05 CEST 2009

250 collections submitted in 2213 seconds: 4/8/30 (min/avg/max)

Final results

Collections correctly submitted: 250 (10000 jobs)
- DONE OK: 10000 (100%)
- Resubmitted: 4 (0.04%)

Errors found (4):
- lsf_reason=65280 (3 times)
- lsf_reason=-1 (1 time)
  - https://devel15.cnaf.infn.it:9000/a9GWtJChtDzQyKujDzfjQw

1) Test starts on Wed Apr 8 at 16:28:22 CEST 2009 (WMS: devel14)

Description:

250 collections each of 40 jobs
One collection every 60 seconds
Five users
max_ice_threads = 20
Used all the CEs of testbedB (except cert-06.cnaf.infn.it)
Used automatic-delegation and proxy renewal service (MyProxyServer = "myproxy.cern.ch")
Proxy has 5 hours of lifetime (and it is renewed every 4 hours)
The job is a "sleep 616"
Resubmission is enabled
Lease mechanism is not used

Submissions finish on Wed Apr 8 at 20:34:44 CEST 2009

250 collections submitted in 2030 seconds: 5/8/29 (min/avg/max)

Final results

Collections correctly submitted: 250 (10000 jobs)
- DONE OK: 10000 (100%)
- Resubmitted: 555 (5.55%)

Errors found (561):
- Cannot move ISB (232 times 41.36%)
- Cannot move OSB (79 times 14.08%)
- Proxy is expired (216 times 38.5%)
- lsf_reason=1603 (34 times 6.06%)

-- AlessioGianelle - 05 Mar 2009

Attachments

Topic attachments
I	Attachment	Action	Size	Date	Who	Comment
png	ice1.png	manage	5.1 K	2009-04-09 - 08:56	AlessioGianelle	Submission rate. Test 1
png	ice13.png	manage	7.0 K	2009-07-02 - 14:08	AlessioGianelle	Ice graph. Test 13
png	ice14.png	manage	8.5 K	2009-07-03 - 14:03	AlessioGianelle	Ice graph. Test 14
png	ice15.png	manage	6.5 K	2009-07-06 - 08:39	AlessioGianelle	Ice graph. Test 15
png	ice16.png	manage	7.3 K	2009-07-10 - 07:40	AlessioGianelle	Ice graph. Test 16
png	ice2.png	manage	4.7 K	2009-04-10 - 07:57	AlessioGianelle	Submission rate. Test 2
png	ice4.png	manage	5.1 K	2009-04-14 - 10:20	AlessioGianelle	Submission rate. Test 4
png	ice6.png	manage	8.6 K	2009-04-30 - 12:18	AlessioGianelle	Ice graph. Test 6
png	icemem10.png	manage	6.1 K	2009-05-29 - 15:52	AlessioGianelle	Ice graph. Test 10
png	icemem11.png	manage	6.3 K	2009-06-03 - 10:08	AlessioGianelle	Ice graph. Test 11
png	icemem12.png	manage	7.1 K	2009-07-02 - 14:25	AlessioGianelle	Ice graph. Test 12
png	icemem8.png	manage	6.4 K	2009-05-14 - 15:24	AlessioGianelle	Ice graph. Test 8
png	icemem9.png	manage	6.3 K	2009-05-26 - 14:11	AlessioGianelle	Ice graph. Test 9

Topic revision: r63 - 2011-02-24 - AlessioGianelle

Account
- Log In

Edit
Attach