Testing report: IGIRTC-146

Summary

  • Product: CREAM v. 1.15.2
  • Release Task: Task IGIRTC-146
  • ETICS Subsystem Configuration Name:
  • VCS Tag:
  • EMI Major Release: EMI 3 (Monte Bianco)
  • Platform: SL6 epel
  • Author: Sara Bertocco
  • Outcome: "In Certification*

Deployment tests

Upgrade from EMI2 to EMI3 cream 1.15.2

CREAM LSF

System tests

Functionality tests LSF

Simple submission

[bertocco@cream-12 ~]$ cat /home/bertocco/jdl/larocca_test.jdl
[
executable="/bin/sleep";
arguments="1";
]

[bertocco@cream-12 ~]$ cat /home/bertocco/jdl/env.jdl
[
executable="/bin/env";
stdoutput="xx2341";
stderror="bb2341";
outputsandbox={"xx2341", "bb2341"};
outputsandboxbasedesturi="gsiftp://localhost";
]

$  glite-ce-job-submit -a -r cert-41.pd.infn.it:8443/cream-lsf-cert /home/bertocco/jdl/larocca_test.jdl
https://cert-41.pd.infn.it:8443/CREAM478909634

[bertocco@cream-12 ~]$  glite-ce-job-submit -a -r cert-41.pd.infn.it:8443/cream-lsf-cert /home/bertocco/jdl/env.jdl
https://cert-41.pd.infn.it:8443/CREAM539735721

[bertocco@cream-12 ~]$ glite-ce-job-status https://cert-41.pd.infn.it:8443/CREAM478909634 https://cert-41.pd.infn.it:8443/CREAM539735721

******  JobID=[https://cert-41.pd.infn.it:8443/CREAM478909634]
   Status        = [DONE-OK]
   ExitCode      = [0]



******  JobID=[https://cert-41.pd.infn.it:8443/CREAM539735721]
   Status        = [DONE-OK]
   ExitCode      = [0]

Functionality testsuite (run by UI with /etc/emi-version 1.11.0-1)

$  /usr/bin/pybot /opt/cream_test/testsuite/cream_test_suite.html 
INFO: The files of this testsuite will be stored under: /tmp/tmpO9Ilzp.cream_testing/
INFO: Non-mandatory variable "sec_cert" has not been set.
INFO: Non-mandatory variable "sec_key" has not been set.
INFO: Non-mandatory variable "argus_host" has not been set.
INFO: Non-mandatory variable "sec_proxy_pass" has not been set.
INFO: Non-mandatory variable "argus_root_pass" has not been set.
==============================================================================
Cream Test Suite :: This is the main testing module,needed for testing the ...
==============================================================================
Set Log Level :: Set the log level used for the test suite. This c... | PASS |
------------------------------------------------------------------------------
Check Executables :: Check whether the needed executables for the ... | PASS |
------------------------------------------------------------------------------
Simple Submit :: Execute /bin/uname -a.Submit the jdl and wait for... | PASS |
------------------------------------------------------------------------------
ISB Base URI :: Execute a bash shell script.The script is stored i... | PASS |
------------------------------------------------------------------------------
ISB Client to CE :: Execute a bash shell script.The script is stor... | PASS |
------------------------------------------------------------------------------
ISB gsiftp to CE :: Execute a bash shell script.The script is stor... | PASS |
------------------------------------------------------------------------------
OSB localhost :: Execute /bin/uname -a.Store std out and error str... | PASS |
------------------------------------------------------------------------------
OSB gsiftp BaseDestURI :: Execute /bin/uname -a.Store std out and ... | PASS |
------------------------------------------------------------------------------
OSB gsiftp DestURI :: Execute /bin/uname -a.Store std out and erro... | PASS |
------------------------------------------------------------------------------
Environment :: Execute a bash shell script.The script is stored lo... | PASS |
------------------------------------------------------------------------------
Epilogue :: Execute two bash shell scripts,one for the job and the... | PASS |
------------------------------------------------------------------------------
Prologue :: Execute two bash shell scripts,one for the job and the... | PASS |
------------------------------------------------------------------------------
Explicit Delegation :: Test job submission with explicit delegatio... | PASS |
------------------------------------------------------------------------------
Job Cancel :: Test job cancellation.Two cases are tested:             | PASS |
------------------------------------------------------------------------------
Job List :: Test the job listing cli utility.Store the list of the... | PASS |
------------------------------------------------------------------------------
Job Suspend - Resume :: Submit a job and suspend it.Wait until it'... | PASS |
------------------------------------------------------------------------------
Manually Cancel Job :: Submit a job and then manually cancel it on... | PASS |
------------------------------------------------------------------------------
Verify CE service info :: This test case verifies the output of th... | PASS |
------------------------------------------------------------------------------
Enable - Disable Submission :: This test case tests the proper fun... | PASS |
------------------------------------------------------------------------------
Job Purge :: This test case verifies the proper functionality of t... | PASS |
------------------------------------------------------------------------------
Admin Manage Job :: This test case checks whether a user can manag... | FAIL |
Mandatory variable sec_proxy_pass is empty!
------------------------------------------------------------------------------
Limiter :: NOTE: This test case is about 25 minutes long!             | PASS |
------------------------------------------------------------------------------
Ban User :: This test case tests if the banning of user works. The... | PASS |
------------------------------------------------------------------------------
Sandbox Staging :: This test case tests the two different Sandbox ... | PASS |
------------------------------------------------------------------------------
Validate Glue Schema :: Run the glue validator on the cream host.     | PASS |
------------------------------------------------------------------------------
Automatic Proxy Purging :: This test case checks the automatic pro... | PASS |
------------------------------------------------------------------------------
Verify Job Status Output :: This test is checking the output of th... | PASS |
------------------------------------------------------------------------------
OutputData :: This test case tests the OutputData jdl attribute. T... | FAIL |
'[]' should not be empty
------------------------------------------------------------------------------
Automatic Job Purge :: This test case is testing the functionality... | PASS |
------------------------------------------------------------------------------
Proxy Renewal :: Submit a job lasting 20 minutes using a proxy val... | PASS |
------------------------------------------------------------------------------
CPU Allocation 1 :: Validate the contents of the batch job submiss... | PASS |
------------------------------------------------------------------------------
CPU Allocation 2 :: Validate the contents of the batch job submiss... | PASS |
------------------------------------------------------------------------------
CPU Allocation 3 :: Validate the contents of the batch job submiss... | PASS |
------------------------------------------------------------------------------
CPU Allocation 4 :: Validate the contents of the batch job submiss... | PASS |
------------------------------------------------------------------------------
CPU Allocation 5 :: Validate the contents of the batch job submiss... | PASS |
------------------------------------------------------------------------------
CPU Allocation 6 :: Validate the contents of the batch job submiss... | PASS |
------------------------------------------------------------------------------
Resource BDII :: Check if the resource bdii publishes all the rele... | PASS |
------------------------------------------------------------------------------
Logging and Bookkeeping :: Check if the logging and bookkeeping se... | FAIL |
_error: WMS Job status polling failed with return value: 1
Command reported: Traceback (most recent call last):
,  File "/usr/bin/glite-wms-job-status", line 53, in <module>
,    import wmsui_utils
,ImportError: No module named wmsui_utils
------------------------------------------------------------------------------
Old BLParser Restart :: Configure a CREAM CE using the old blah bl... | PASS |
------------------------------------------------------------------------------
Cream Test Suite :: This is the main testing module,needed for tes... | FAIL |
39 critical tests, 36 passed, 3 failed
39 tests total, 36 passed, 3 failed
==============================================================================
Output:  /home/bertocco/output.xml
Log:     /home/bertocco/log.html
Report:  /home/bertocco/report.html
  • Functionality testsuite log

Check of failed tests (run by UI with /etc/emi-version 2.8.0-1)

    • Admin Manage Job fails because the test requires to have two user proxyes and the tester has only one.
    • Logging and Bookkeeping failure was due to my glite-UI misconfiguration problem. From a different UI the test is successful
            $  /usr/bin/pybot  -i lb /opt/cream_test/testsuite/cream_test_suite.html 
            INFO: The files of this testsuite will be stored under: /tmp/tmp1epXyI.cream_testing/
            INFO: Non-mandatory variable "sec_cert" has not been set.
            INFO: Non-mandatory variable "sec_key" has not been set.
            INFO: Non-mandatory variable "argus_host" has not been set.
            INFO: Non-mandatory variable "sec_proxy_pass" has not been set.
            ==============================================================================
            Cream Test Suite :: This is the main testing module,needed for testing the ...
            ==============================================================================
            Logging and Bookkeeping :: Check if the logging and bookkeeping se... | PASS |
            ------------------------------------------------------------------------------
            Cream Test Suite :: This is the main testing module,needed for tes... | PASS |
            1 critical test, 1 passed, 0 failed
            1 test total, 1 passed, 0 failed
            ==============================================================================
            Output:  /home/bertocco/output.xml
            Log:     /home/bertocco/log.html
            Report:  /home/bertocco/report.html
            
      • Logging and Bookkeeping log
    • Outputdata test fails for a configuration problem
            
            $  /usr/bin/pybot  -i outputdata /opt/cream_test/testsuite/cream_test_suite.html 
            INFO: The files of this testsuite will be stored under: /tmp/tmpkkHrsW.cream_testing/
            INFO: Non-mandatory variable "sec_cert" has not been set.
            INFO: Non-mandatory variable "sec_key" has not been set.
            INFO: Non-mandatory variable "argus_host" has not been set.
            INFO: Non-mandatory variable "sec_proxy_pass" has not been set.
            ==============================================================================
            Cream Test Suite :: This is the main testing module,needed for testing the ...
            ==============================================================================
            OutputData :: This test case tests the OutputData jdl attribute. T... | FAIL |
            '[]' should not be empty
            ------------------------------------------------------------------------------
            Cream Test Suite :: This is the main testing module,needed for tes... | FAIL |
            1 critical test, 0 passed, 1 failed
            1 test total, 0 passed, 1 failed
            ==============================================================================
            Output:  /home/bertocco/output.xml
            Log:     /home/bertocco/log.html
            Report:  /home/bertocco/report.html
            
      • outputdata log
              $ /usr/bin/glite-ce-job-output -d -N -D /tmp/tmpMuznan.cream_testing/outputdata https://cert-41.pd.infn.it:8443/CREAM917243346
              2013-05-13 12:32:10,253 DEBUG - Using certificate proxy file [/tmp/x509up_u33337]
              2013-05-13 12:32:10,276 WARN - No configuration file suitable for loading. Using built-in configuration
              2013-05-13 12:32:10,276 DEBUG - Logfile is [/tmp/glite_cream_cli_logs/glite-ce-job-output_CREAM_bertocco_20130513-123210.log]
              2013-05-13 12:32:10,277 DEBUG - Processing job [https://cert-41.pd.infn.it:8443/CREAM917243346]...
        
              2013-05-13 12:32:10,277 DEBUG - Connecting to [https://cert-41.pd.infn.it:8443//ce-cream/services/CREAM2]...
              2013-05-13 12:32:10,277 DEBUG - Contacting service [https://cert-41.pd.infn.it:8443//ce-cream/services/CREAM2]
              2013-05-13 12:32:10,348 DEBUG - JobID [https://cert-41.pd.infn.it:8443/CREAM917243346]: Will retrieve output from [gsiftp://cert-41.pd.infn.it/var/glite/cream_sandbox/dteam     /CN_Sara_Bertocco_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam017/91/CREAM917243346/OSB]
              2013-05-13 12:32:10,349 INFO - For JobID [https://cert-41.pd.infn.it:8443/CREAM917243346] output will be stored in the dir /tmp/tmpMuznan.cream_testing/outputdata/cert-41.pd.infn.it_8443_CREAM917243346
              2013-05-13 12:32:10,350 DEBUG - Will execute command [/usr/bin/uberftp cert-41.pd.infn.it "parallel 1; lcd /tmp/tmpMuznan.cream_testing/outputdata/cert-41.pd.infn.it_8443_CREAM917243346; cd /var/glite/      cream_sandbox/dteam/CN_Sara_Bertocco_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam017/91/CREAM917243346/OSB; mget *; lcd /home/bertocco; quit"]
              [bertocco@cert-27 ~]$ ls /tmp/tmpMuznan.cream_testing/outputdata/cert-41.pd.infn.it_8443_CREAM917243346
        
              $ cat /tmp/tmpMuznan.cream_testing/outputdata/cert-41.pd.infn.it_8443_CREAM917243346/DSUpload_CREAM917243346.out
              #
              # Autogenerated by JobWrapper!
              #
              # The file contains the results of the upload and registration
              # process in the following format:
              # <outputfile> <lfn|guid|Error>
        
              script_output.txt    Error: [GFAL][get_se_types_and_endpoints][] [BDII][g1_sd_get_se_types_and_endpoints]: No available information
              lcg_cr: Invalid argument
              

Regression tests LSF

Verification attached bugs for CREAM LSF

Bug #101221 CREAM sends wrong authorization requests to Argus containing attributes with empty values. not reproducible
Bug #101108 Minor issues from INFN-T1 ???
# grep USE_ services/glite-creamce
USE_CEMON=false
USE_EMIES=false

/opt/glite/yaim/bin/yaim -c -s /usr/local/nfs1/jra1_test/cert-41/site-info.def  -n creamCE -n LSF_utils
..........
   INFO: YAIM terminated succesfully.

# grep tomcat6 /opt/glite/yaim/log/yaimlog
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]
Stopping tomcat6:                                          [  OK  ]
   INFO: Configuring /etc/tomcat6/server.xml...
Starting tomcat6:                                        [  OK  ]
yaim log

# ls -l /etc/tomcat6/Catalina/localhost
total 4
-rw-r--r-- 1 tomcat tomcat 56 May  9 17:00 ce-cream.xml


# cat /etc/glite-ce-cream/log4j.properties
log4j.rootLogger=debug, fileout
log4j.logger.httpclient.wire.content=off
log4j.logger.org.glite.voms.PKIStore=off
log4j.appender.fileout=org.apache.log4j.RollingFileAppender
log4j.appender.fileout.File=/var/log/cream/glite-ce-cream.log
log4j.appender.fileout.MaxFileSize=10MB
log4j.appender.fileout.MaxBackupIndex=20
log4j.appender.fileout.layout=org.apache.log4j.PatternLayout
log4j.appender.fileout.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %p %c - %m%n
# service tomcat6 restart
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]

Submit 3000 jobs using the stress testsuite:
# python2.6 /opt/cream_stress_test/bin/cream_stress_test.py  -d single -n 3000 -c 4 -s 5 -e cert-41.pd.infn.it:8443/cream-lsf-cert -t submit --jdl sleep --sleep 30

After all jobs are finished:
# ls -l /var/log/cream/glite-ce-cream.log
-rw-r--r-- 1 tomcat tomcat 408803 May 21 11:13 /var/log/cream/glite-ce-cream.log

Standard conformance tests

Glue 1 Compliance

  • glue-validator -t glue1 -h cert-41.pd.infn.it -p 2170 -b o=grid 
    log

Glue 2 Compliance

*
glue-validator -t glue2 -h cert-41.pd.infn.it -p 2170 -b o=glue 
log

Performance and scalability tests

  • Submit 3000 jobs (sleep 30) using 4 concurrent processes
# python2.6 /opt/cream_stress_test/bin/cream_stress_test.py  -d single -n 3000 -c 4 -s 5 -e cert-41.pd.infn.it:8443/cream-lsf-cert -t submit --jdl sleep --sleep 30 2>&1 |tee /dev/tty > stress_test.log 
stress_test.log

  • Submit and cancel 3000 jobs (sleep 30) using 4 concurrent processes
# python2.6 /opt/cream_stress_test/bin/cream_stress_test.py  -d single -n 3000 -c 4 -s 5 -e cert-41.pd.infn.it:8443/cream-lsf-cert -t cancel --jdl sleep --sleep 30 2>&1 |tee /dev/tty > cancel_stress_test.log 
cancel_stress_test.log

  • Submit and cancel 3000 jobs (sleep 50) using 12 concurrent processes
# python2.6 /opt/cream_stress_test/bin/cream_stress_test.py  -d single -n 3000 -c 12 -s 5 -e cert-41.pd.infn.it:8443/cream-lsf-cert -t cancel --jdl sleep --sleep 50 2>&1 |tee /dev/tty > cancel_stress_test_3.log
cancel_stress_test_3.log

-- SaraBertocco - 2013-05-09

Topic attachments
I Attachment ActionSorted ascending Size Date Who Comment
Unknown file formatlog CREAM_SL6_LSF_update.log manage 80.9 K 2013-05-10 - 14:29 SaraBertocco  
Unknown file formatlog cancel_stress_test_3.log manage 6354.0 K 2013-05-21 - 16:06 SaraBertocco  
Unknown file formatlog functionality_SL6.log manage 608.8 K 2013-05-13 - 09:23 SaraBertocco  
Unknown file formatlog lb.log manage 195.9 K 2013-05-13 - 10:43 SaraBertocco  
Unknown file formatlog outputdata.log manage 202.9 K 2013-05-13 - 10:43 SaraBertocco  
Unknown file formatlog stress_test.log manage 6077.5 K 2013-05-21 - 09:51 SaraBertocco  
Unknown file formatlog taskIGIRTC-146_conf_update_cream_lsf_sl6.log manage 12.7 K 2013-05-10 - 14:38 SaraBertocco  
Texttxt yaimlog.txt manage 12.7 K 2013-05-20 - 16:07 SaraBertocco  
Edit | Attach | PDF | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | More topic actions
Topic revision: r6 - 2013-05-21 - SaraBertocco
 
This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback