PATCH 4415

Automatic tests

• report #1:
  • CREAM UI version: 1.12.1; CREAM testsuite version: 1.0.7
  • Used direct polling for monitoring and BUpdater/BNotifier for status change detection
  • Batch system: TORQUE
  • All tests have been run (see the report from the testsuite) with the following results:
    • test #18 reports one submission failed due to network problems and one execution terminated by the batch system
    • all other tests have been correctly executed

Tested that the BDII is operational

ldapsearch -x -h cream-40.pd.infn.it -p 2170 -b o=infosys

Output is here

Checked the Glue 1.3 root entry:

ldapsearch -x -h cream-40.pd.infn.it -p 2170 -b mds-vo-name=resource,o=grid

Output is here

Checked the Glue 2.0 root entry:

ldapsearch -x -h cream-40.pd.infn.it -p 2170 -b o=glue

Output is here

Information content check:

$ gstat-validate-ce -H cream-40.pd.infn.it -p 2170 -b o=grid
OK - errors 0, warnings 0, info 0

Checked bugs

• Bug #54755: Could YAIM warn if a site tries to enable whole-VO FQANs? FIXED
  • Define a VO with the character "/" as a prefix in the variables [queue]_GROUP_ENABLE of the site-info.def
  • Run the YAIM configurator and verify that in the output the following warning appears: "Queue [vo name] was enabled for one or more FQANs which specify an entire VO"

• Bug #61396: CREAM and CEMon log4j.properties should take into account the value of GLITE_LOCATION_LOG FIXED
  • Specify the variable GLITE_LOCATION_LOG in the YAIM configuration (file services-glite-creamce), even with an nonexistent path, and reconfigure with YAIM.
  • Verify that the new path is correctly created and the log files for CREAM and CEMon are written in the new location

• Bug #62078: Missing glite-pbsparser.log file on standalone Torque-Server FIXED
  • Act as for bug #61396 and verify that the log file glite-pbsparser.log is correctly written inside the new location

• Bug #63007: CREAM sandbox dir should be relocatable FIXED
  • Define the variable CREAM_SANDBOX_PATH in the file services/glite-creamce
  • Reconfigure with YAIM, submit a job and verify that the sandbox of that job is placed into the new sandbox path

• Bug #63956: config_cream_gip may use wrong VO FIXED
  • patch verified by the submitter

• Bug #68225: In some cases the sudoers file created by yaim-cream-ce can be syntactically wrong FIXED
  • Consider a VO whose name contains a '-' character
  • Run the YAIM configurator and verify that the VO is filtered and does not appear in the sudoers file.

• Bug #69785: Use #include to modify /etc/sudoer for CREAM with sudo >= 1.7.1 FIXED
  • With sudo version >= 1.7.1 installed, reconfigure with YAIM
  • Verify that all CREAM directives are written into the file /etc/sudoers.forcream
  • Update from glite-CREAM version < 3.2.8, reconfigure with YAIM
  • Verify that the CREAM directives have been moved from the original /etc/sudoers file into the new /etc/sudoers.forcream

• Bug #70285: it should be possible to tell bhist and tracejob how many logs to read FIXED
  • Submit a bunch of jobs so that all the slots of the CE are busy for more than 24 hours, for example given 200 slots use the following test:

    cream-test-monitored-submit -r 30 -n 200 -m 200 -C 50 -j long.jdl -R [ce-resource-url]
    

    , where the jdl file contains:

    [executable="/bin/sleep";arguments="86400";]
    

  • Submit a new bunch of jobs so that they will be enqueued for more than 24 hours
  • Verify that all the jobs terminate correctly and that all the status changes are correctly logged into the Bnotifier log.

• Bug #70287: Problem with AssignFinalState if state is IDLE FIXED
  • Define a value short enough for the parameter "alldone_interval" in the blah.config file, for example 1200 seconds.
  • Submit a bunch of jobs so that all the slots of the CE are busy for more than 1200 seconds, for example given 200 slots use the following test:

    cream-test-monitored-submit -r 30 -n 200 -m 200 -C 50 -j long.jdl -R [ce-resource-url]
    

    , where the jdl file contains:

    [executable="/bin/sleep";arguments="1800";]
    

  • Submit a new bunch of jobs whose idle time and effective processing time are larger than 1200 seconds.
  • Verify that all the jobs terminate correctly and that all the status changes are correctly logged into the Bnotifier log.

• Bug #70808: PBS submission script creates wrong stagein/stageout directives with qsub's -W option FIXED
  • Save a backup of the BLAH submit wrapper, replacing the statement "rm -f $bls_tmp_file" with "cp $bls_tmp_file /tmp" in the script ${GLITE_LOCATION}/bin/blah_common_submit_functions.sh
  • Submit a job and verify that the -W directive, as specified in the BLAH submit wrapper, contains a list of item "stagein=" and "stageout="
  • Verify that all the file specified in the stagein or in the stageout directives are correctly moved between the CREAM and the worker node.

• Bug #71038: CE certificate publication error FIXED
  • Use a host certificate whose subject DN contains at least one blank character
  • Run the YAIM configurator and verify that the attribute GlueServiceDataValue of the item

    GlueServiceDataKey=DN,GlueServiceUniqueID=[unique ID for the CE],Mds-Vo-name=resource,o=grid
    

    contains the complete DN

• Bug #71055: LL_ and IL_ PIDFILE added more times in /opt/glite/etc/lb.conf FIXED
  • Run more times the YAIM configurator and verify that the file /opt/glite/etc/lb.conf does not change.

• Bug #71967: There should be in blah the option to set the temporary job work dir FIXED
  • Define the value of blah_wn_temporary_home_dir in the file ${GLITE_LOCATION}/etc/blah.config pointing to '/tmp'
  • Restart all gLite services on the node.
  • Submit the following jdl:

    [
    executable="/bin/pwd";
    stdoutput="out-pwd.txt";
    outputsandbox={
        "out-pwd.txt"
    };
    outputsandboxbasedesturi="gsiftp://xxxxx/tmp";
    epilogue="/bin/sleep";
    epiloguearguments="30";
    ]
    

    and verify that the file out-pwd.txt contains something like "/tmp/home_creXX_YYYYYYYYY/CREAMYYYYYYYYY

• Bug #72097: cream-config.xml and cemonitor-config.xml should use GLITE_LOCATION_LOG and GLITE_LOCATION_VAR FIXED
  • Define the variables GLITE_LOCATION_VAR=[new var directory] and CREAM_SANDBOX_PATH=[new sandbox dir] in the file services/glite-creamce
  • Run the YAIM configurator and verify that the new var directory is specified into the following attributes:
    • cream-config.xml: cream_scratch_dir, cream_sandbox_dir, delegation_storage, CREAM_SANDBOX_DIR, JOBS_SCRATCH_DIR
    • cemonitor-config.xml: backendLocation

• Bug #72508: variable CREAM_DB_PASSWORD missed in services/glite-creamce FIXED
  • Verify that the file ${GLITE_LOCATION}/yaim/examples/siteinfo/services/glite-creamce contains the variable CREAM_DB_PASSWORD

• Bug #72756: glite-ce-blahparser missed in gLiteservices FIXED
  • Stop all the gLite service on the CE
  • Remove the script glite-ce-blahparser from the gLiteservices list
  • Reconfigure with YAIM and verify that the glite-ce-blahparser has been re-inserted into the list

• Bug #73765: Performance problems with CREAM proxy renewal FIXED
  • Run the following test:

    cream-test-monitored-submit -r 30 -n 1000 -m 200 -C 50 -l log4py.conf --nopurge --sotimeout 60 -j 15m.jdl -R [ce-url]
    

    where the file 15m.jdl contains:

    [ executable="/bin/sleep"; arguments="900";]
    

  • Run a proxy renewal every 5 minutes specifying the delegation id as reported in log4py.conf
  • Keep the value of pending commands monitored, using the following script:

    for ((idx=0; idx < 240; idx++)); do /opt/glite/bin/glite_cream_load_monitor --show | grep pending; sleep 30; done
    

  • Verify that all the renewal requests are correctly processed and the number of pending commands does not increase

• Bug #73961: /tmp of CREAM CE full of glexec* proxies FIXED
  • Submit a job and verifiy that no proxy has been created in the temporary directory /tmp

• Bug #74107: ClientJobId missing in BNotifier classad output FIXED
  • Stop all gLite services on the CREAM node and run only the BNotifier/BUpdater (/opt/glite/etc/glite-ce-blahparser start)
  • Contact the BNotifier daemon via telnet and send the following set of commands:

    CREAMFILTER/[blah_prefix]
    PARSERVERSION/
    STARTNOTIFYJOBLIST/[blah_prefix][client_job_id]
    STARTNOTIFYJOBEND
    

    where
    • "[blah_prefix]" is the value of the parameter BLAH_JOBID_PREFIX of the cream configuration file
    • "[client_job_id]" is random 9-digits number so that the value "[blah_prefix][client_job_id]" is not contained into the BNotifier registry, this can be verified using the following command:

      strings $GLITE_LOCATION/var/blah/user_blah_job_registry.bjr/registry | grep [blah_prefix][client_job_id]
      

  • Verify that the classad returned by the BNotifier contains the parameter "ClientJobId"
  • Close the connection with BNotifier and restart all gLite services on the node

Clean installation

• Installation steps:

  wget http://etics-repository.cern.ch/repository/pm/registered/repomd/name/patch_4415_2/etics-registered-build-by-name.repo -O /etc/yum.repos.d/glite-CREAM.repo
  yum install xml-commons-apis
  yum install glite-CREAM
  wget http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.2/glite-TORQUE_utils.repo -O /etc/yum.repos.d/glite-TORQUE_utils.repo
  wget http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.2/glite-TORQUE_server.repo -O /etc/yum.repos.d/glite-TORQUE_server.repo
  yum install glite-TORQUE_utils glite-TORQUE_server
  /opt/glite/yaim/bin/yaim -c -s site-info.def -n creamCE -n TORQUE_server -n TORQUE_utils
  

• View the log of yum for a clean installation
• View the log of yaim for a clean installation (TORQUE is used)

Upgrade from production

• Upgrade steps:

  wget http://etics-repository.cern.ch/repository/pm/registered/repomd/name/patch_4415_2/etics-registered-build-by-name.repo -O /etc/yum.repos.d/glite-CREAM.repo
  yum update
  /opt/glite/yaim/bin/yaim -c -s site-info.def -n creamCE -n TORQUE_server -n TORQUE_utils
  

• View the log of yum for an upgrade
• View the log of yaim for an upgrade (TORQUE is used)