Tags:
, view all tags

Testing report: IGIRTC-56

Summary

  • Product: BLAH 1.16.5
  • Release Task: Task #27056
  • ETICS Subsystem Configuration Name: emi-cream-ce_R_1_13_8_2
  • VCS Tag: emi-blahp_R_1_16_5_2
  • EMI Major Release: EMI 1 (Kebnekaise)
  • Platform: SL5 epel
  • Author:
  • Testing report:
  • Certification report:
  • Outcome: In certification...

Deployment tests

Clean Installation

Upgrade Installation

LSF CE

PBS CE

Unit Tests

Not Available. The plan is to provide some unit tests starting with EMI-2.

System tests

Functionality tests

Test submission

  • Test result for LSF is available here PASSED
  • Test result for PBS is available here PASSED

BLParser test

Old BLParser

Tests.Check Notifications For Cancelled Jobs :: Test that notifications are...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_cancelled_jobs                                | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Cancelled Jobs :: Test that notifica... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Normally Finished Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_normally_finished_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Normally Finished Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Suspended Resumed Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_suspended_resumed_jobs                        | FAIL |
_error: Expected status should be in ['HELD'] for job https://cream-29.pd.infn.it:8443/CREAM705156955 was actually IDLE
------------------------------------------------------------------------------
Tests.Check Notifications For Suspended Resumed Jobs :: Test that ... | FAIL |
2 critical tests, 1 passed, 1 failed
2 tests total, 1 passed, 1 failed
==============================================================================
Tests                                                                 | FAIL |
6 critical tests, 5 passed, 1 failed
6 tests total, 5 passed, 1 failed
==============================================================================
Output:  /home/ale/blah/italiangrid-cream_blah_testsuites-09156c3_ver2/output.xml
Log:     /home/ale/blah/italiangrid-cream_blah_testsuites-09156c3_ver2/log.html
Report:  /home/ale/blah/italiangrid-cream_blah_testsuites-09156c3_ver2/report.html

Tests.Check Notifications For Cancelled Jobs :: Test that notifications are...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_cancelled_jobs                                | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Cancelled Jobs :: Test that notifica... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Normally Finished Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_normally_finished_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Normally Finished Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Suspended Resumed Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_suspended_resumed_jobs                        | FAIL |
_error: Expected status should be in ['HELD'] for job https://cream-41.pd.infn.it:8443/CREAM324312963 was actually DONE-OK
------------------------------------------------------------------------------
Tests.Check Notifications For Suspended Resumed Jobs :: Test that ... | FAIL |
2 critical tests, 1 passed, 1 failed
2 tests total, 1 passed, 1 failed
==============================================================================
Tests                                                                 | FAIL |
6 critical tests, 5 passed, 1 failed
6 tests total, 5 passed, 1 failed
==============================================================================
  • Job which is suspended and then resumed

New BLParser

Tests.Check Notifications For Cancelled Jobs :: Test that notifications are...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_cancelled_jobs                                | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Cancelled Jobs :: Test that notifica... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Normally Finished Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_normally_finished_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Normally Finished Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Suspended Resumed Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_suspended_resumed_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Suspended Resumed Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests                                                                 | PASS |
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
==============================================================================

Tests.Check Notifications For Cancelled Jobs :: Test that notifications are...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_cancelled_jobs                                | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Cancelled Jobs :: Test that notifica... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Normally Finished Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_normally_finished_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Normally Finished Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests.Check Notifications For Suspended Resumed Jobs :: Test that notificat...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
check_notifications_for_suspended_resumed_jobs                        | PASS |
------------------------------------------------------------------------------
Tests.Check Notifications For Suspended Resumed Jobs :: Test that ... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Tests                                                                 | PASS |
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
==============================================================================

Regression tests

Verification attached bugs

Bug #89527: BLAHP produced -W stage(in/out) directives are incompatible with Torque 2.5.8 FIXED

Content of file to check here.

Bug #91037: BUpdaterLSF should use bjobs to detect final job state FIXED

  • Change debug_level and restart the services:
[root@cream-29 ~]# sed -i 's/bupdater_debug_level=2/bupdater_debug_level=3/' /etc/blah.config 
[root@cream-29 ~]# mv /var/log/cream/glite-ce-bupdater.log /var/log/cream/glite-ce-bupdater.log.old
[root@cream-29 ~]# service gLite restart
STOPPING SERVICES
*** glite-ce-blahparser:
Shutting down BNotifier:                                   [  OK  ]
Shutting down BUpdaterLSF:                                 [  OK  ]

*** glite-lb-locallogger:
Stopping glite-lb-logd ... done
Stopping glite-lb-interlogd ... done

*** tomcat5:
Stopping tomcat5:                                          [  OK  ]

STARTING SERVICES
*** tomcat5:
Starting tomcat5:                                          [  OK  ]

*** glite-lb-locallogger:
Starting glite-lb-logd ...This is LocalLogger, part of Workload Management System in EU DataGrid & EGEE.
 done
Starting glite-lb-interlogd ... done

*** glite-ce-blahparser:
Starting BNotifier: /usr/bin/BNotifier: Error creating and binding socket: Address already in use
                                                           [FAILED]
Starting BUpdaterLSF:                                      [  OK  ]
  • Submit a job and wait for its completation:
[ale@cream-12 UI]$ glite-ce-job-submit -a -r cream-29.pd.infn.it:8443/cream-lsf-cert cream.jdl
https://cream-29.pd.infn.it:8443/CREAM239301025
[ale@cream-12 UI]$ glite-ce-job-status https://cream-29.pd.infn.it:8443/CREAM239301025

******  JobID=[https://cream-29.pd.infn.it:8443/CREAM239301025]
   Status        = [DONE-OK]
   ExitCode      = [0]

[root@cream-29 ~]# grep 239301025 /var/log/cream/glite-ce-bnotifier.log
2012-03-22 17:05:36 Sent for Cream:[BatchJobId="622199"; JobStatus=4; ChangeTime="2012-03-22 17:05:22"; JwExitCode=0; Reason="reason=0"; ClientJobId="239301025"; BlahJobName="cre29_239301025";]
  • Verify if bhist has been called:
[root@cream-29 ~]# grep bhist /var/log/cream/glite-ce-bupdater.log
2012-03-22 17:00:43 /usr/bin/BUpdaterLSF: key bupdater_use_bhist_time_constraint not found - using the default:no
2012-03-22 17:00:43 /usr/bin/BUpdaterLSF: key bupdater_use_bhist_for_killed not found - using the default:no
2012-03-22 17:01:26 /usr/bin/BUpdaterLSF: key bupdater_use_bhist_time_constraint not found - using the default:no
2012-03-22 17:01:26 /usr/bin/BUpdaterLSF: key bupdater_use_bhist_for_killed not found - using the default:no
2012-03-22 17:03:54 /usr/bin/BUpdaterLSF: key bupdater_use_bhist_time_constraint not found - using the default:no
2012-03-22 17:03:54 /usr/bin/BUpdaterLSF: key bupdater_use_bhist_for_killed not found - using the default:no
2012-03-22 17:04:51 /usr/bin/BUpdaterLSF: key bupdater_use_bhist_time_constraint not found - using the default:no
2012-03-22 17:04:51 /usr/bin/BUpdaterLSF: key bupdater_use_bhist_for_killed not found - using the default:no

Bug #92281: Purge of registry can cause registry corruption TBD

Bug #92774: BLParserLSF could crash searching in old logs TBD

Verification old bugs

Content of report file here.

==============================================================================
Bug 81824 :: Regression test of bug #81824 yaim-cream-ce should manage the ...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
bug_81824                                                             | PASS |
------------------------------------------------------------------------------
Bug 81824 :: Regression test of bug #81824 yaim-cream-ce should ma... | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================

Test PASSED

Submitted 5000 jobs to a CREAM CE configured using the new blparser, and with job_registry_use_mmap=yes.

Monitored the used RSS of the blahpd processes. At the end the maximum value between all the process is 4560.

Test PASSED

Configure /etc/blah.config:

[root@cream-29 ~]# tail -4 /etc/blah.config
# Verify fix for bug #77776
lsf_batch_caching_enabled=yes
batch_command_caching_filter=/usr/bin/runcmd.pl

Where runcmd.pl is:

#!/usr/bin/perl
#---------------------#
#  PROGRAM:  argv.pl  #
#---------------------#

$numArgs = $#ARGV + 1;
open (MYFILE, '>>/tmp/xyz');
foreach $argnum (0 .. $#ARGV) {
    print MYFILE "$ARGV[$argnum] ";
}
print MYFILE "\n";
close (MYFILE); 

Restart the services and submit 10 jobs to the CE.

[root@cream-29 cream]# cat /tmp/xyz 
/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/bjobs -u all -l -a 
/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/bhist -u all -d -l -n 10 
/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/bjobs -u all -l -a 
/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/bhist -u all -d -l -n 10 
/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/bjobs -u all -l -a 
/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/bhist -u all -d -l -n 10 
/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/bjobs -u all -l -a 
/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/bhist -u all -d -l -n 10 

Test PASSED

[root@cream-29 cream]# ls -l /var/blah
total 8
-rw-r--r-- 1 tomcat tomcat    0 Mar 23 15:49 blah_bnotifier.pid
-rw-r--r-- 1 tomcat tomcat    4 Mar 23 15:53 blah_bupdater.pid
drwxrwx--t 4 tomcat tomcat 4096 Mar 23 16:07 user_blah_job_registry.bjr
[root@cream-29 cream]#  ls -l /var/blah/user_blah_job_registry.bjr/
total 14388
-rw-rw-r-- 1 tomcat tomcat 11377096 Mar 23 17:04 registry
-rw-r--r-- 1 tomcat tomcat  3066960 Mar 23 16:07 registry.by_blah_index
-rw-rw-rw- 1 tomcat tomcat        0 Mar 23 17:04 registry.locktest
drwxrwx-wt 2 tomcat tomcat     4096 Mar 23 17:04 registry.npudir
drwxrwx-wt 2 tomcat tomcat   253952 Mar 23 17:04 registry.proxydir
-rw-r--r-- 1 tomcat tomcat       99 Mar 23 15:49 registry.subjectlist
[root@cream-29 cream]# ls -l /var/blah/user_blah_job_registry.bjr/registry.npudir
total 8
-rw-rw-r-- 1 dteam017 dteam 856 Mar 23 17:04 npu_d4X1Ao
-rw-rw-r-- 1 dteam017 dteam 856 Mar 23 17:04 npu_jZXJGF
[root@cream-29 cream]# ls -l /var/blah/user_blah_job_registry.bjr/registry.proxydir/ | head -10
total 2448
lrwxrwxrwx 1 dteam017 dteam 198 Mar 23 16:41 proxy_637551_RUGJ92 -> /var/glite/cream_sandbox/dteam/_C_IT_O_INFN_OU_Personal_Certificate_L_Padova_CN_Alessio_Gianelle_dteam_Role_NULL_Capability_NULL_dteam017/proxy/f9099a3af228b82c323e26c1f9f494aefdd1b43910396930026260
lrwxrwxrwx 1 dteam017 dteam 198 Mar 23 16:41 proxy_637552_LrF5tC -> /var/glite/cream_sandbox/dteam/_C_IT_O_INFN_OU_Personal_Certificate_L_Padova_CN_Alessio_Gianelle_dteam_Role_NULL_Capability_NULL_dteam017/proxy/23a5e3de27a7d82748cabae2d5dccec329853fc610396930026260
lrwxrwxrwx 1 dteam017 dteam 198 Mar 23 16:41 proxy_637553_WvebYm -> /var/glite/cream_sandbox/dteam/_C_IT_O_INFN_OU_Personal_Certificate_L_Padova_CN_Alessio_Gianelle_dteam_Role_NULL_Capability_NULL_dteam017/proxy/751c2ef4ec29986f4bf63a602fbe626d7d1b3cab10396930026260
lrwxrwxrwx 1 dteam017 dteam 198 Mar 23 16:41 proxy_637554_4vdiJs -> /var/glite/cream_sandbox/dteam/_C_IT_O_INFN_OU_Personal_Certificate_L_Padova_CN_Alessio_Gianelle_dteam_Role_NULL_Capability_NULL_dteam017/proxy/4966974580d533d4d42a4635158382566ec6c2f610396930026260
lrwxrwxrwx 1 dteam017 dteam 198 Mar 23 16:41 proxy_637555_QX5Bi6 -> /var/glite/cream_sandbox/dteam/_C_IT_O_INFN_OU_Personal_Certificate_L_Padova_CN_Alessio_Gianelle_dteam_Role_NULL_Capability_NULL_dteam017/proxy/785cbd414da9ee5e79b633477b59a86071dcf50f10396930026260
lrwxrwxrwx 1 dteam017 dteam 198 Mar 23 16:41 proxy_637556_jTHBm1 -> /var/glite/cream_sandbox/dteam/_C_IT_O_INFN_OU_Personal_Certificate_L_Padova_CN_Alessio_Gianelle_dteam_Role_NULL_Capability_NULL_dteam017/proxy/df2c50573906838f8894b361c2a14860c3e4f16210396930026260
lrwxrwxrwx 1 dteam017 dteam 198 Mar 23 16:41 proxy_637557_DbWTbB -> /var/glite/cream_sandbox/dteam/_C_IT_O_INFN_OU_Personal_Certificate_L_Padova_CN_Alessio_Gianelle_dteam_Role_NULL_Capability_NULL_dteam017/proxy/6fd59e85358187345a4635505c9af7fe32ec61a310396930026260
lrwxrwxrwx 1 dteam017 dteam 198 Mar 23 16:41 proxy_637558_hhAHKX -> /var/glite/cream_sandbox/dteam/_C_IT_O_INFN_OU_Personal_Certificate_L_Padova_CN_Alessio_Gianelle_dteam_Role_NULL_Capability_NULL_dteam017/proxy/ced55ce67190ef6b2d8c56907926aed2f1a4ddfd10396930026260
lrwxrwxrwx 1 dteam017 dteam 198 Mar 23 16:41 proxy_637559_WjeEfP -> /var/glite/cream_sandbox/dteam/_C_IT_O_INFN_OU_Personal_Certificate_L_Padova_CN_Alessio_Gianelle_dteam_Role_NULL_Capability_NULL_dteam017/proxy/1d197ae0e6d8681258517525f8fc568b56bdf17810396930026260

Test PASSED

[root@cream-29 cream]#  cat /etc/logrotate.d/blahp-logrotate | grep rotate
        rotate 365

Test PASSED

[root@cream-29 ~]# ps ax | grep BLParserLSF
  754 pts/1    S+     0:00 grep BLParserLSF
31468 ?        Sl     0:00 /usr/bin/BLParserLSF -d 1 -l /var/log/cream/glite-lsfparser.log -b /opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin -c /opt/lsf/conf -p 33333 -m 56565

Test PASSED

Topic attachments
I Attachment Action SizeSorted ascending Date Who Comment
Unknown file formatEXT cre40_591788054 manage 3.5 K 2012-03-23 - 16:27 SaraBertocco  
Unknown file formatlog pbs_update.log manage 3.6 K 2012-03-22 - 15:39 SaraBertocco  
Texttxt lsf_update.txt manage 3.9 K 2012-03-22 - 10:11 AlessioGianelle lsf update
Unknown file formatlog pbs_confnew_1.log manage 14.0 K 2012-03-22 - 16:55 SaraBertocco  
Unknown file formatlog pbs_confold.log manage 16.4 K 2012-03-22 - 16:50 SaraBertocco  
Texttxt lsf_update_confnew.txt manage 44.3 K 2012-03-22 - 10:17 AlessioGianelle Configuration log New blparser
Texttxt lsf_update_confold.txt manage 48.6 K 2012-03-22 - 10:47 AlessioGianelle Configuration log Old blparser
Compressed Zip archivetgz bug_81824_report.tgz manage 104.1 K 2012-03-23 - 16:46 SaraBertocco  
Compressed Zip archivetgz PBS_OldBlparser.tgz manage 117.9 K 2012-03-23 - 14:04 SaraBertocco  
Compressed Zip archivetgz PBS_NewBlparser.tgz manage 121.2 K 2012-03-23 - 14:03 SaraBertocco  
Compressed Zip archivetgz LSF_OLDBlparser.tgz manage 123.6 K 2012-03-22 - 11:28 AlessioGianelle Old blah parser tests
Compressed Zip archivetgz LSF_NewBlparser.tgz manage 143.7 K 2012-03-22 - 15:59 AlessioGianelle New blah parser tests
Unknown file formatlog lsfsubmission.log manage 1118.6 K 2012-03-22 - 14:01 AlessioGianelle LSf test
Unknown file formatlog newparser_blah_exerciser.sh.log manage 1215.3 K 2012-03-23 - 11:09 SaraBertocco  
Edit | Attach | PDF | History: r35 | r28 < r27 < r26 < r25 | Backlinks | Raw View | More topic actions...
Topic revision: r26 - 2012-03-26 - SaraBertocco
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback