Task37715 < IGIRelease

Tags: , view all tags
---++Testing and Certification report savannah task #37715: BLAH v. 1.18.2  (EMI2)

%TOC%


---++ Summary

   * Product: *BLAH v. 1.18.2*
   * Release Task: [[https://savannah.cern.ch/task/?37715][Task #37715]]
   * ETICS Subsystem Configuration Name: *emi-blahp_R_1_18_2_1*
   * VCS Tag: glite-ce-blahp_R_1_18_2_1
   * EMI Major Release: EMI 2 (Matterhorn)
   * Platform: SL5 & SL6
   * Author: Sergio Traldi
   * Testing report: [[%ATTACHURL%/EMI_BLAH_1_18_2_test_report.txt][here]]
   * Certification report: [[%ATTACHURL%/EMI_BLAH_1_18_2_cert_report.txt][here]] 
   * Date: 20 Nov 2012
   * Outcome: %GREEN% Certified %ENDCOLOR%
 

---+++ Deployment tests 

---++++ Clean Installation 

 
   * EMI2 BLAH SL6 installation  from scratch log:  [[%ATTACHURL%/install-SL6-EMI2-CETorque.txt][install-SL6-EMI2-CETorque.txt]].
   * EMI2 BLAH SL6 configuration after an installation  from scratch log: [[%ATTACHURL%/configuration-SL6-EMI2-CETorque.txt][configuration-SL6-EMI2-CETorque.txt]].


---++++ Upgrade Installation

   * EMI2 BLAH SL5 update log: [[%ATTACHURL%/update-SL5-EMI2-CELSF.txt][update-SL5-EMI2-CELSF.txt]].
   * EMI2 BLAH SL5 configuration after update log: [[%ATTACHURL%/configuration-update-SL5-EMI2-CELSF.txt][configuration-update-SL5-EMI2-CELSF.txt]].


----+++ Unit Tests

   * SL5 Unit test report available [[http://etics-repository.cern.ch/repository/download/registered/emi/emi.cream-ce.blahp/1.18.2/sl5_x86_64_gcc412EPEL/glite-ce-blahp-1.18.2-1-reports.tar.gz][here]].
   * SL6 Unit test report available [[http://etics-repository.cern.ch/repository/download/registered/emi/emi.cream-ce.blahp/1.18.2/sl6_x86_64_gcc446EPEL/glite-ce-blahp-1.18.2-1-reports.tar.gz][here]].

---+++ System tests

---++++ Functionality tests 

---+++++ Submission tests SL5  %GREEN% PASSED %ENDCOLOR%
   * Submission tests using old parser:
      <verbatim>
       ./blah_exerciser.sh exiting: Submitted 1200 jobs. 1200 succeeded. 0 failed.
       Took 5228 seconds to submit.
        Took 8308 seconds to complete.
      </verbatim> 
    
     [[%ATTACHURL%/submission_test_SL5_OldParser_log.txt][Long log]].

   * Submission tests on SL5 new parser:
      <verbatim>
       ./blah_exerciser.sh exiting: Submitted 1200 jobs. 1200 succeeded. 0 failed.
       Took 5189 seconds to submit.
       Took 8409 seconds to complete.
      </verbatim> 

     [[%ATTACHURL%/submission_test_SL5_NewParser_log.txt][Long log]].

---+++++ Submission tests SL6  %GREEN% PASSED %ENDCOLOR%

   * Submission tests on SL6 old parser 
      <verbatim>
      ./blah_exerciser.sh exiting: Submitted 1000 jobs. 1000 succeeded. 0 failed.
      Took 3903 seconds to submit.
       Took 7460 seconds to complete.
     </verbatim>
     
     [[%ATTACHURL%/submission_test_SL6_OldParser_log.txt][Long log]]

   * Submission tests on SL6 new parser
      <verbatim>
      ./blah_exerciser.sh exiting: Submitted 1000 jobs. 1000 succeeded. 0 failed.
      Took 3888 seconds to submit.
      Took 6916 seconds to complete.
     </verbatim>

     [[%ATTACHURL%/submission_test_SL6_NewParser_log.txt][Long log]]


---+++++ Check Notifications SL5 - LSF New parser. %GREEN% PASSED %ENDCOLOR%

   * For normally finished jobs

<verbatim>
[traldi@cert-25 ~]$ pybot /opt/cream_blah_testsuites/testsuites/blah_testing/tests
==============================================================================
Tests                                                                         
==============================================================================
 batch_system = lsf
 ce_admin_user = root
 ce_admin_pass = omiss
 ce_host = cream-29.pd.infn.it
 ce_endpoint = cream-29.pd.infn.it:8443
 cream_queue = cream-lsf-cert
 vo = dteam
 proxy_pass = 
 log_level = TRACE
 tmp_dir = /tmp
 delete_files = False
 tomcat_service = tomcat5
 catalina_home = /var/lib/tomcat5
middleware_version = emi2
 ce_cream_xml = /etc/glite-ce-cream/cream-config.xml
 cream_config_xml = /etc/glite-ce-cream/cream-config.xml
 blah_config = /etc/blah.config
 site_info = /root/siteinfo/site-info.def
 services_glite_creamce = /root/siteinfo/services/glite-creamce
 blah_bupdater_loop_interval = 20
 blah_check_children_interval = 30
 blah_bupdater_use_bhist_for_killed = yes
 cream_sandbox_path = CREAM_SANDBOX_PATH
 cream_concurrency_level_yaim_name = CREAM_CONCURRENCY_LEVEL
 middleware_version = emi2
 cream_concurrency_level = commandworkerpoolsize
 ce = cream-29.pd.infn.it:8443/cream-lsf-cert
Command "/usr/bin/voms-proxy-info" output follows:
subject   : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
issuer    : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
identity  : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
type      : proxy
strength  : 1024 bits
path      : /tmp/x509up_u500
timeleft  : 11:59:53

Command's output printed.
 dn = /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
The files of this testsuite will be stored under: /tmp/tmpRRy05i.cream_testing/
Tests.Check Notifications For Cancelled Jobs :: Test that notifications are...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_cancelled_jobs                                | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Cancelled Jobs :: Test that notifica... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Normally Finished Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_normally_finished_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Normally Finished Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Suspended Resumed Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_suspended_resumed_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Suspended Resumed Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests                                                                 | PASS |
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
==============================================================================
Output:  /home/traldi/output.xml
Log:     /home/traldi/log.html
Report:  /home/traldi/report.html
</verbatim>

Long log and report of Check Notification [[%ATTACHURL%/CheckNotificationsSL5-LSFNewparser.tgz][CheckNotificationsSL5-LSFNewparser.tgz]]

---+++++ Check Notifications SL5 - LSF Old parser %GREEN% PASSED %ENDCOLOR%

<verbatim>
[traldi@cert-25 ~]$ pybot /opt/cream_blah_testsuites/testsuites/blah_testing/tests
==============================================================================
Tests                                                                         
==============================================================================
 batch_system = lsf
 ce_admin_user = root
 ce_admin_pass = omiss
 ce_host = cream-29.pd.infn.it
 ce_endpoint = cream-29.pd.infn.it:8443
 cream_queue = cream-lsf-cert
 vo = dteam
 proxy_pass = 
 log_level = TRACE
 tmp_dir = /tmp
 delete_files = False
 tomcat_service = tomcat5
 catalina_home = /var/lib/tomcat5
middleware_version = emi2
 ce_cream_xml = /etc/glite-ce-cream/cream-config.xml
 cream_config_xml = /etc/glite-ce-cream/cream-config.xml
 blah_config = /etc/blah.config
 site_info = /root/siteinfo/site-info.def
 services_glite_creamce = /root/siteinfo/services/glite-creamce
 blah_bupdater_loop_interval = 20
 blah_check_children_interval = 30
 blah_bupdater_use_bhist_for_killed = yes
 cream_sandbox_path = CREAM_SANDBOX_PATH
 cream_concurrency_level_yaim_name = CREAM_CONCURRENCY_LEVEL
 middleware_version = emi2
 cream_concurrency_level = commandworkerpoolsize
 ce = cream-29.pd.infn.it:8443/cream-lsf-cert
Command "/usr/bin/voms-proxy-info" output follows:
subject   : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
issuer    : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
identity  : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
type      : proxy
strength  : 1024 bits
path      : /tmp/x509up_u500
timeleft  : 11:37:50

Command's output printed.
 dn = /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
The files of this testsuite will be stored under: /tmp/tmpz5Mqf1.cream_testing/
Tests.Check Notifications For Cancelled Jobs :: Test that notifications are...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_cancelled_jobs                                | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Cancelled Jobs :: Test that notifica... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Normally Finished Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_normally_finished_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Normally Finished Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Suspended Resumed Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_suspended_resumed_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Suspended Resumed Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests                                                                 | PASS |
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
==============================================================================
Output:  /home/traldi/output.xml
Log:     /home/traldi/log.html
Report:  /home/traldi/report.html
</verbatim>

Long log and report of Check Notification [[%ATTACHURL%/CheckNotificationsSL5-LSFOldparser.tgz][CheckNotificationsSL5-LSFOldparser.tgz]]

---+++++ Check Notifications SL6 - LSF New parser %GREEN% PASSED %ENDCOLOR%

<verbatim>
[traldi@cert-25 ~]$ pybot /opt/cream_blah_testsuites/testsuites/blah_testing/tests
==============================================================================
Tests                                                                         
==============================================================================
 batch_system = pbs
 ce_admin_user = root
 ce_admin_pass = omiss
 ce_host = cert-42.pd.infn.it
 ce_endpoint = cert-42.pd.infn.it:8443
 cream_queue = cream-pbs-cert
 vo = dteam
 proxy_pass = 
 log_level = TRACE
 tmp_dir = /tmp
 delete_files = False
 tomcat_service = tomcat6
 catalina_home = /usr/share/tomcat6
middleware_version = emi2
 ce_cream_xml = /etc/glite-ce-cream/cream-config.xml
 cream_config_xml = /etc/glite-ce-cream/cream-config.xml
 blah_config = /etc/blah.config
 site_info = /root/siteinfo/site-info.def
 services_glite_creamce = /root/siteinfo/services/glite-creamce
 blah_bupdater_loop_interval = 20
 blah_check_children_interval = 30
 blah_bupdater_use_bhist_for_killed = yes
 cream_sandbox_path = CREAM_SANDBOX_PATH
 cream_concurrency_level_yaim_name = CREAM_CONCURRENCY_LEVEL
 middleware_version = emi2
 cream_concurrency_level = commandworkerpoolsize
 ce = cert-42.pd.infn.it:8443/cream-pbs-cert
Command "/usr/bin/voms-proxy-info" output follows:
subject   : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
issuer    : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
identity  : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
type      : proxy
strength  : 1024 bits
path      : /tmp/x509up_u500
timeleft  : 11:25:42

Command's output printed.
 dn = /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
The files of this testsuite will be stored under: /tmp/tmp4gYf8W.cream_testing/
Tests.Check Notifications For Cancelled Jobs :: Test that notifications are...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_cancelled_jobs                                | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Cancelled Jobs :: Test that notifica... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Normally Finished Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_normally_finished_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Normally Finished Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Suspended Resumed Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_suspended_resumed_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Suspended Resumed Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests                                                                 | PASS |
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
==============================================================================
Output:  /home/traldi/output.xml
Log:     /home/traldi/log.html
Report:  /home/traldi/report.html
</verbatim>

Long log and report of Check Notification [[%ATTACHURL%/CheckNotificationsSL6-LSFNewparser.tgz][CheckNotificationsSL6-LSFNewparser.tgz]]

---+++++ Check Notifications SL6 - LSF Old parser %GREEN% PASSED %ENDCOLOR%
<verbatim>
[traldi@cert-25 ~]$ pybot /opt/cream_blah_testsuites/testsuites/blah_testing/tests==============================================================================
Tests                                                                         
==============================================================================
 batch_system = pbs
 ce_admin_user = root
 ce_admin_pass = omiss
 ce_host = cert-42.pd.infn.it
 ce_endpoint = cert-42.pd.infn.it:8443
 cream_queue = cream-pbs-cert
 vo = dteam
 proxy_pass = 
 log_level = TRACE
 tmp_dir = /tmp
 delete_files = False
 tomcat_service = tomcat6
 catalina_home = /usr/share/tomcat6
middleware_version = emi2
 ce_cream_xml = /etc/glite-ce-cream/cream-config.xml
 cream_config_xml = /etc/glite-ce-cream/cream-config.xml
 blah_config = /etc/blah.config
 site_info = /root/siteinfo/site-info.def
 services_glite_creamce = /root/siteinfo/services/glite-creamce
 blah_bupdater_loop_interval = bupdater_loop_interval
 blah_check_children_interval = check_children_interval
 blah_bupdater_use_bhist_for_killed = bupdater_use_bhist_for_killed
 cream_sandbox_path = CREAM_SANDBOX_PATH
 cream_concurrency_level_yaim_name = CREAM_CONCURRENCY_LEVEL
 middleware_version = emi2
 cream_concurrency_level = commandworkerpoolsize
 ce = cert-42.pd.infn.it:8443/cream-pbs-cert
Command "/usr/bin/voms-proxy-info" output follows:
subject   : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
issuer    : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
identity  : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
type      : proxy
strength  : 1024 bits
path      : /tmp/x509up_u500
timeleft  : 10:24:56

Command's output printed.
 dn = /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
The files of this testsuite will be stored under: /tmp/tmpzu4sVs.cream_testing/
Tests.Check Notifications For Cancelled Jobs :: Test that notifications are...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_cancelled_jobs                                | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Cancelled Jobs :: Test that notifica... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Normally Finished Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_normally_finished_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Normally Finished Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Suspended Resumed Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_suspended_resumed_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Suspended Resumed Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests                                                                 | PASS |
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
==============================================================================
Output:  /home/traldi/output.xml
Log:     /home/traldi/log.html
Report:  /home/traldi/report.html
</verbatim>

Long log and report of Check Notification [[%ATTACHURL%/CheckNotificationsSL6-LSFOldparser.tgz][CheckNotificationsSL6-LSFOldparser.tgz]]

---++++ Regression tests 

---+++++ Verification attached bugs

---++++++ [[https://savannah.cern.ch/bugs/?97491][bug #97491]]:(BUpdaterLSF should not execute any bhist query if all the bhist related conf parameter are set to "no") %NAVY% FIXED, NOT CERTIFIED %ENDCOLOR%

---++++++ [[https://savannah.cern.ch/bugs/?95385][bug #95385]] (Misleading message when Cream SGE aborts jobs requesting more than one CPU) %NAVY% FIXED, NOT CERTIFIED %ENDCOLOR%

---++++++ [[https://savannah.cern.ch/bugs/?98855][bug #98855]] There is a memory leak in a blah function %GREEN% FIXED %ENDCOLOR%

*Before the bug was fixed:*

*FROM CE:*
top before submission:
<verbatim>
[root@cream-29 ~]# top

top - 18:01:21 up 138 days,  2:33,  2 users,  load average: 0.60, 0.44, 0.38
Tasks:  96 total,   2 running,  94 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  6.0%sy,  8.6%ni, 84.2%id,  0.3%wa,  0.0%hi,  0.9%si,  0.0%st
Mem:   2097152k total,  1966916k used,   130236k free,    97968k buffers
Swap:  2097144k total,    33640k used,  2063504k free,   766512k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30337 tomcat    36  19 2649m 769m  13m S  0.0 37.6   0:27.08 java
2606 mysql     35  19  324m  22m 3584 S  0.0  1.1 757:15.05 mysqld
22256 tomcat    34  19 78596  11m  376 S  0.7  0.6   0:07.84 BNotifier
23659 ldap      34  19  128m 9052 4704 S  0.0  0.4   0:00.26 slapd
31149 tomcat    34  19 37048 7524 7160 S  0.0  0.4   0:00.04 blahpd
25865 tomcat    34  19 37048 7300 7008 S  0.0  0.3   0:00.01 blahpd
22298 tomcat    39  19 15044 6082  420 S 39.4  0.3   4:22.97 BUpdaterLSF
25080 ntp       15   0 19212 4904 3800 S  0.0  0.2   0:00.37 ntpd
18445 ldap      36  19 69496 4456 1024 S  0.0  0.2   0:01.14 bdii-update
...
</verbatim>

*FROM UI:* 
(submitted 10.000 jobs having output DONE FAIL)
<verbatim>
[traldi@cert-25 ~]$ for i in {1..10000}; do glite-ce-job-submit -r cream-29.pd.infn.it:8443/cream-lsf-cert -a simpleCreamOutput.jdl; done
https://cream-29.pd.infn.it:8443/CREAM114333135
https://cream-29.pd.infn.it:8443/CREAM580535769
https://cream-29.pd.infn.it:8443/CREAM846997993
...

[traldi@cert-25 ~]$ glite-ce-job-status https://cream-29.pd.infn.it:8443/CREAM114333135

******  JobID=[https://cream-29.pd.infn.it:8443/CREAM114333135]
       Status        = [DONE-FAILED]
       ExitCode      = [W]
       FailureReason = [./pippoErrato.sh not found or unreadable]
</verbatim>

*FROM CE:*
[root@cream-29 ~]# top

top - 17:10:20 up 139 days,  2:40,  2 users,  load average: 0.14, 0.23, 1.21
Tasks: 149 total,   2 running, 147 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  4.1%sy,  6.9%ni, 88.2%id,  0.4%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   2097152k total,  2051612k used,    45540k free,   163372k buffers
Swap:  2097144k total,       60k used,  2097084k free,   749940k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30337 tomcat    39  19 2795m 703m  13m S  0.0 34.3 203:21.29 java
2606 mysql     34  19  324m  39m 5948 S  0.0  1.9 156:20.52 mysqld
22298 tomcat    35  19 30040  21m  612 S 19.0  1.1  27:55.49 BUpdaterLSF
23659 ldap      34  19  367m 9808 4760 S  0.0  0.5   0:08.37 slapd
22256 tomcat    34  19 54328 8020  508 S  0.0  0.4   0:38.12 BNotifier
1817 ldap      34  19 34032 7140 1736 S  4.3  0.3   0:00.13 info-dynamic-ls
25080 ntp       15   0 19212 4904 3800 S  0.0  0.2   0:00.24 ntpd
23716 ldap      35  19 69536 4380 1024 S  0.7  0.2   1:00.65 bdii-update
1769 ldap      36  19 69536 3840  484 S  0.0  0.2   0:00.00 bdii-update

BUpdaterLSF increase memory from 6082 to 21M after 10.000 submissions.

*After the bug was fixed:*

*CE Host:*
Before submission:
<verbatim>
[root@cream-29 ~]# top

top - 17:03:24 up 141 days,  2:33,  2 users,  load average: 0.60, 0.44, 0.38
Tasks:  96 total,   2 running,  94 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  6.0%sy,  8.6%ni, 84.2%id,  0.3%wa,  0.0%hi,  0.9%si,  0.0%st
Mem:   2097152k total,  1966916k used,   130236k free,    97968k buffers
Swap:  2097144k total,    33640k used,  2063504k free,   766512k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25235 tomcat    36  19 2649m 769m  13m S  0.0 37.6   0:27.08 java
2606 mysql     35  19  324m  22m 3584 S  0.0  1.1 757:15.05 mysqld
7638 tomcat    34  19 78596  11m  376 S  0.7  0.6   0:07.84 BNotifier
18432 ldap      34  19  128m 9052 4704 S  0.0  0.4   0:00.26 slapd
31149 tomcat    34  19 37048 7524 7160 S  0.0  0.4   0:00.04 blahpd
25865 tomcat    34  19 37048 7300 7008 S  0.0  0.3   0:00.01 blahpd
 7636 tomcat    39  19 15044 7008  420 S 39.4  0.3   4:22.97 BUpdaterLSF
.....
</verbatim>

After 50.000 submissions 35.000 DONE FAILD + 15.000 DONE OK

<verbatim>
[root@cream-29 ~]# top

top - 13:32:32 up 146 days, 23:02,  1 user,  load average: 0.00, 0.01, 0.00
Tasks:  95 total,   2 running,  93 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 92.2%id,  7.8%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2097152k total,  2056488k used,    40664k free,   132836k buffers
Swap:  2097144k total,    32364k used,  2064780k free,   873536k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25235 tomcat    36  19 2657m 775m  13m S  0.0 34.9  31:10.28 java
 2606 mysql     34  19  324m  22m 3584 S  0.0  1.1 773:55.00 mysqld
7638 tomcat    34  19 78596  11m  376 S  0.0  0.6  32:45.08 BNotifier
18432 ldap      34  19  367m 9784 4720 S  0.0  0.5   0:33.70 slapd
31149 tomcat    34  19 37048 7524 7160 S  0.0  0.4   0:00.06 blahpd
25865 tomcat    34  19 37048 7300 7008 S  0.0  0.3   0:00.03 blahpd
  7636 tomcat    35  19 16848 7920  472 S  0.0  0.3  35:25.62 BUpdaterLSF 
...
</verbatim>

The BUpdaterLSF memory did not increase so much


---+++++ Verification old bugs

---++++++ [[https://savannah.cern.ch/bugs/?81824][bug #81824]]: (yaim-cream-ce should manage the attribute bupdater_loop_interval) %GREEN% FIXED %ENDCOLOR%
---++++++ [[https://savannah.cern.ch/bugs/?84261][bug #84261]]: BNotifier on CREAM CE seems to not restart cleanly %GREEN% FIXED %ENDCOLOR%
---++++++ [[https://savannah.cern.ch/bugs/?86238][bug #86238]]: blahpd doesn't check the status of its daemons when idling %GREEN% FIXED %ENDCOLOR%
---++++++ [[https://savannah.cern.ch/bugs/?94519][bug #94519]]: ((Updater for LSF can misidentify killed jobs as finished) %GREEN% FIXED %ENDCOLOR%

<verbatim>
[traldi@cert-25 ~]$ pybot -i new_parser /opt/cream_blah_testsuites/testsuites/blah_regression/tests/bug_*.html
==============================================================================
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519                                 
==============================================================================
 batch_system = pbs
 ce_admin_user = root
 ce_admin_pass = omiss
 ce_host = cert-42.pd.infn.it
 ce_endpoint = cert-42.pd.infn.it:8443
 cream_queue = cream-pbs-cert
 vo = dteam
 proxy_pass = 
 log_level = TRACE
 tmp_dir = /tmp
 delete_files = False
 tomcat_service = tomcat6
 catalina_home = /usr/share/tomcat6
middleware_version = emi2
 ce_cream_xml = /etc/glite-ce-cream/cream-config.xml
 cream_config_xml = /etc/glite-ce-cream/cream-config.xml
 blah_config = /etc/blah.config
 site_info = /root/siteinfo/site-info.def
 services_glite_creamce = /root/siteinfo/services/glite-creamce
 blah_bupdater_loop_interval = bupdater_loop_interval
 blah_check_children_interval = check_children_interval
 blah_bupdater_use_bhist_for_killed = bupdater_use_bhist_for_killed
 cream_sandbox_path = CREAM_SANDBOX_PATH
 cream_concurrency_level_yaim_name = CREAM_CONCURRENCY_LEVEL
 middleware_version = emi2
 cream_concurrency_level = commandworkerpoolsize
 ce = cert-42.pd.infn.it:8443/cream-pbs-cert
Command "/usr/bin/voms-proxy-info" output follows:
subject   : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
issuer    : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
identity  : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
type      : proxy
strength  : 1024 bits
path      : /tmp/x509up_u500
timeleft  : 11:51:51

Command's output printed.
 dn = /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
The files of this testsuite will be stored under: /tmp/tmpa0V_zM.cream_testing/
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519.Bug 81824 :: Regression test ...
==============================================================================
bug_81824                                                             | PASS |
------------------------------------------------------------------------------
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519.Bug 81824 :: Regress... | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519.Bug 84261 :: Regression test ...
==============================================================================
bug_84261                                                             | PASS |
------------------------------------------------------------------------------
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519.Bug 84261 :: Regress... | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519.Bug 86238 :: Regression test ...
==============================================================================
bug_86238                                                             | PASS |
------------------------------------------------------------------------------
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519.Bug 86238 :: Regress... | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519.Bug 94519 :: Bug #94519 (Upda...
==============================================================================
bug_94519                                                             | PASS |
------------------------------------------------------------------------------
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519.Bug 94519 :: Bug #94... | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Bug 81824 & Bug 84261 & Bug 86238 & Bug 94519                         | PASS |
4 critical tests, 4 passed, 0 failed
4 tests total, 4 passed, 0 failed
==============================================================================
Output:  /home/traldi/output.xml
Log:     /home/traldi/log.html
Report:  /home/traldi/report.html
</verbatim>
Long log and report of Regression Test on SL6 New Parser [[%ATTACHURL%/RegressionSL6-TorqueNewParser.tgz][CheckNotificationsSL6-LSFNewparser.tgz]]

---++++++ [[https://savannah.cern.ch/bugs/?94414][bug #94414]]: (BLParserLSF could crash if a suspend on an idle job is done)  %GREEN% FIXED %ENDCOLOR%
The bug is already tested in check_notifications_for_suspended_resumed_jobs.html because the testsuite is designed to suspend the job when the job status is IDLE.


---++++++ [[https://savannah.cern.ch/bugs/?90085][bug #90085]]:  Suspend command doesn't work with old parser %GREEN% FIXED %ENDCOLOR%
---++++++ [[https://savannah.cern.ch/bugs/?94414][bug #94414]]:(BLParserLSF could crash if a suspend on an idle job is done)   %GREEN% FIXED %ENDCOLOR%
---++++++ [[https://savannah.cern.ch/bugs/?94414][bug #94414 Stronger]]: (BLParserLSF could crash if a suspend on an idle job is done) %GREEN% FIXED %ENDCOLOR%


<verbatim>
[traldi@cert-25 ~]$ pybot -i old_parser /opt/cream_blah_testsuites/testsuites/blah_regression/tests/bug_*.html
==============================================================================
Bug 90085 & Bug 94414 & Bug 94414 Stronger                                    
==============================================================================
 batch_system = lsf
 ce_admin_user = root
 ce_admin_pass = omiss
 ce_host = cream-29.pd.infn.it
 ce_endpoint = cream-29.pd.infn.it:8443
 cream_queue = cream-lsf-cert
 vo = dteam
 proxy_pass = 
 log_level = TRACE
 tmp_dir = /tmp
 delete_files = False
 tomcat_service = tomcat6
 catalina_home = /usr/share/tomcat6
middleware_version = emi2
 ce_cream_xml = /etc/glite-ce-cream/cream-config.xml
 cream_config_xml = /etc/glite-ce-cream/cream-config.xml
 blah_config = /etc/blah.config
 site_info = /root/siteinfo/site-info.def
 services_glite_creamce = /root/siteinfo/services/glite-creamce
 blah_bupdater_loop_interval = bupdater_loop_interval
 blah_check_children_interval = check_children_interval
 blah_bupdater_use_bhist_for_killed = bupdater_use_bhist_for_killed
 cream_sandbox_path = CREAM_SANDBOX_PATH
 cream_concurrency_level_yaim_name = CREAM_CONCURRENCY_LEVEL
 middleware_version = emi2
 cream_concurrency_level = commandworkerpoolsize
 ce = cream-29.pd.infn.it:8443/cream-lsf-cert
Command "/usr/bin/voms-proxy-info" output follows:
subject   : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
issuer    : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
identity  : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi
type      : proxy
strength  : 1024 bits
path      : /tmp/x509up_u500
timeleft  : 11:54:06

Command's output printed.
 dn = /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Sergio Traldi/CN=proxy
The files of this testsuite will be stored under: /tmp/tmpCF6h5w.cream_testing/
Bug 90085 & Bug 94414 & Bug 94414 Stronger.Bug 90085 :: Regression test of ...
==============================================================================
bug_90085                                                             | PASS |
------------------------------------------------------------------------------
Teardown                                                              | PASS |
------------------------------------------------------------------------------
Bug 90085 & Bug 94414 & Bug 94414 Stronger.Bug 90085 :: Regression... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Bug 90085 & Bug 94414 & Bug 94414 Stronger.Bug 94414 :: Regression test of ...
==============================================================================
bug_94414                                                             | PASS |
------------------------------------------------------------------------------
Bug 90085 & Bug 94414 & Bug 94414 Stronger.Bug 94414 :: Regression... | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Bug 90085 & Bug 94414 & Bug 94414 Stronger.Bug 94414 Stronger :: Regression...
==============================================================================
bug_94414_stronger                                                    | PASS |
------------------------------------------------------------------------------
Bug 90085 & Bug 94414 & Bug 94414 Stronger.Bug 94414 Stronger :: R... | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Bug 90085 & Bug 94414 & Bug 94414 Stronger                            | PASS |
4 critical tests, 4 passed, 0 failed
4 tests total, 4 passed, 0 failed
==============================================================================
Output:  /home/traldi/output.xml
Log:     /home/traldi/log.html
Report:  /home/traldi/report.html
</verbatim>
Long log and report of Regression Test on SL5 Old Parser [[%ATTACHURL%/RegressionSL5-LSFOldParser.tgz][CheckNotificationsSL6-LSFNewparser.tgz]]


-- Main.SergioTraldi - 2012-11-21
Attachments
Topic attachments
I	Attachment	Action	Size	Date	Who	Comment
txt	EMI_BLAH_1_18_2_cert_report.txt	manage	2.5 K	2012-12-07 - 09:56	CristinaAiftimiei	Test & Cert reports
txt	EMI_BLAH_1_18_2_test_report.txt	manage	2.5 K	2012-12-07 - 09:56	CristinaAiftimiei	Test & Cert reports
Topic revision: r11 - 2012-12-07 - CristinaAiftimiei
Account
- Log In
Edit
Attach