• Starting from a Production WMS we update it.

• yaim_conf_overwrite_true: yaim_conf_wms_overwrite

• Starting from a Production WMS we update it.

• test_upgrade_wms: test wms upgrade

• Bug #68891: ICE falls into an infinite loop when a job has expired proxy and has been submitted to a CREAM without EventQuery FIXED NOT CERTIFIED
  • It is not easy to reproduce the problem.

• copied registered repo for glite-wms_R_3_2_14_6 into '/etc/yum.repos.d':

  wget http://etics-repository.cern.ch/repository/pm/registered/repomd/id/5e0f9d1b-de35-48ec-ad71-3b32a55f2b46/slc4_ia32_gcc346/etics-registered-build-by-id.repo

• launched 'yum install glite-WMS', yum install log is here

• yaim configuration log file is here

• yum_install_wms: yum_install_wms

• yaim_conf_wms_log: yaim_conf_wms

• • test report here

• test_recovery: test recovery

• Bug #59339: [ICE] doesn't correctly handle request in jobdir/old when it is restarted FIXED

• • • test report here

• Prologue and Epilogue jobs

Stress test

Description:

• 1200 collections each of 20 jobs
• One collection every 60 seconds
• Four users
• No requirements (so randomly submitted to all the production CEs)
• Use automatic-delegation
• The job is a "sleep random(172)"
• Resubmission is enabled
• Use 2 proxy renewal services (myproxy.cern.ch and myproxy.cnaf.infn.it) plus jobs without specification of the MyproxyServer (equally distributed)
• Use a 3.2.0.12 LB Server (patch #4083)

Final results

• 1172 collections submitted in 6935 seconds: 3/6/125 (min/avg/max)
  • 28 submissions fails (due to LB error)
• Jobs correctly submitted: 23440 (90% to LCG CEs and 10% to CREAM CEs)
  • DONE OK: 23335 (99.52%)
  • ABORTED: 105 ( 0.48 %)
  • Resubmitted: 3067 ( 13.08 %)
• Aborted reasons:
  • Job proxy is expired: 42 times (40 %)
  • (LB query failed): 57 times (54.4 %)
  • Removal retries exceeded: 4 times (3.8%)
  • hit job shallow retry count (3): 1 time (0.9%)
  • Submission to condor failed: 1 time (0.9%)
• Failures (3377)
  • 7 an authentication operation failed (~63%)
  • File not available.Cannot read JobWrapper output, both from Condor and from Maradona. (~18%)
  • Transfer to CREAM failed due to exception (~12%)
  • Got a job held event (~5%)
  • ... others (~2%)

• • • test report here

• • • test report here

• • • test report here

• Bug #52937: ICE uses the wrong DN to log to LB Hopefully FIXED
  • Submit a job to a cream CE and the check with a glite-wms-job-logging-info -v 2:

    [ale@cream-15 UI]$ glite-wms-job-logging-info -v 2 https://devel15.cnaf.infn.it:9000/RITl2CsEH_KRk2nEusZVKw | grep User
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy
    - User                       =    /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle/CN=proxy/CN=proxy

• Bug #53297: [ yaim-wms ] glite_wms.conf hardcoded parameters FIXED

• Bug #66986: ICE must be able to print out on file the stack trace trapping SIGSEGV, SIGILL, SIGABRT etc. FIXED NOT CERTIFIED

• launched 'yum install glite-WMS', yum install log is here

• copied host certificate and key into '/etc/grid-security'

• yum_install_wms_log_good: yum_install_wms_log

• Starting from a Production WMS we update it.

• MPICH jobs:
  • • test report here

• copied registered repo for wms_3_2_14_5 into '/etc/yum.repos.d':

  wget http://etics-repository.cern.ch/repository/pm/registered/repomd/id/7c25a3e0-f9aa-4ad0-9c74-e52ed3166562/slc4_ia32_gcc346

• launched 'yum install glite-WMS', yum install log is here
• copied /opt/glite/yaim/examples/siteinfo/site-info.def into ~/siteinfo/site-info.def and /opt/glite/yaim/examples/siteinfo/services/glite-wms into ~/siteinfo/services/glite-wms
• launched 'yum install lcg-CA'
• launched yaim configuration '/opt/glite/yaim/bin/yaim -c -s site-info.def -n glite-WMS'
• yaim configuration log file is here

• Bug #53297: [ yaim-wms ] glite_wms.conf hardcoded parameters FIXED
  • tested by setting the parameter 'WMS_CONF_FILE_OVERWRITE' in the ~/siteinfo/services/glite-wms file
    • set the parameter 'WMS_CONF_FILE_OVERWRITE' to true: a backup copy of the glite_wms.conf file gets created in /opt/glite/etc/glite_wms.conf.bkp_20100608_101305 and the glite_wms.conf file gets overwritten
    • set the parameter 'WMS_CONF_FILE_OVERWRITE' to false: a new copy of the glite_wms.conf file gets created into /opt/glite/etc/glite_wms.conf.yaimnew_20100608_101633

• Bug #66721: Ineffective and never removed Job cancels FIXED
  • submitted a job with the option '--register-only' as in the following:

    glite-wms-job-submit --config ../glite_wms_devel14.conf --register-only -a ../myjob.jdl 

https://devel14.cnaf.infn.it:7443/glite_wms_wmproxy_server

================== glite-wms-job-submit Success ==================

The job has been successfully registered to the WMProxy Your job identifier is:

https://devel15.cnaf.infn.it:9000/mS8cXbg9szmcXsUETm8g2g

======================================================================

To complete the operation, the following file containing the InputSandbox of the job needs to be transferred: ====================================================================================================== ISB ZIP file : /tmp/ISBfiles_eCXKdK2egtqk7jZRlfFQpw_0.tar.gz Destination : gsiftp://devel14.cnaf.infn.it:2811/var/glite/SandboxDir/mS/https_3a_2f_2fdevel15.cnaf.infn.it_3a9000_2fmS8cXbg9szmcXsUETm8g2g/input/ISBfiles_eCXKdK2egtqk7jZRlfFQpw_0.tar.gz

https://devel15.cnaf.infn.it:9000/mS8cXbg9szmcXsUETm8g2g

• • cancel the previously submitted job as in the following:

    glite-wms-job-cancel https://devel15.cnaf.infn.it:9000/mS8cXbg9szmcXsUETm8g2g
    
    Are you sure you want to remove specified job(s) [y/n]y : y
    
    Connecting to the service https://devel14.cnaf.infn.it:7443/glite_wms_wmproxy_server
    
    
    ============================= glite-wms-job-cancel Success =============================
    
    The cancellation request has been successfully submitted for the following job(s):
    
    - https://devel15.cnaf.infn.it:9000/mS8cXbg9szmcXsUETm8g2g
    
    ========================================================================================
    

• Bug #66986: ICE must be able to print out on file the stack trace trapping SIGSEGV, SIGILL, SIGABRT etc. TO VERIFY

• Bug #67097: [yaim-wms] Removed lcg-condor-extra usage FIXED
  • checked NORDUGRID_GAHP is set to the right value:

    grep NORDUGRID_GAHP /opt/glite/yaim/functions/*
    /opt/glite/yaim/functions/config_condor_wms:   setValue NORDUGRID_GAHP "\$(SBIN)/nordugrid_gahp"

• Bug #52937: ICE uses the wrong DN to log to LB TO VERIFY

• Bug #53297: [ yaim-wms ] glite_wms.conf hardcoded parameters TO VERIFY

• Bug #66721: Ineffective and never removed Job cancels TO VERIFY

• Bug #66986: ICE must be able to print out on file the stack trace trapping SIGSEGV, SIGILL, SIGABRT etc. TO VERIFY

• Bug #67097: [yaim-wms] Removed lcg-condor-extra usage TO VERIFY

• • ICE creation test report here

• brokerinfo-ICE: brokerinfo-ICE

• • JC creation: test report here

• brokerinfo: brokerinfo

• • • bulk test report to ICE here

• bulk_jobs_toICE: bulk_jobs_toICE

• • • bulk test report to JC here

• bulk_jobs: bulk jobs to JC

• Bug #66721: Ineffective and never removed Job cancels TO VERIFY

• • test report here

• • • test result here

• • • test result here

• • • test report here

• • • test result here

• • • test result here

• cancel-dag: cancel-dag

• jobcancel-toICE: jobcancel-toICE

• jobcancel-toLcg: jobcancel-toLcg

• cancel-collection-toICE: cancel-collection-toICE

• cancel-collection-toJC: cancel-collection-toJC

• • • test report here

• perusal-toLcg: perusal-toLcg

Perusal jobs

• • • test report here

• perusal-toICE: perusal-toICE

Parametric jobs

• • • test report here

• • • test report here

• parametric-toICE: parametric-toICE

• parametric-toLcg: parametric-toLcg

Normal Jobs

DAG jobs

Collection jobs

• • • test report here

• • • test report here

• collection-toICE: collection-toICE

• collection-toLcg: collection-toLcg

• collection-toICE: collection-toICE

List match without data

• • tried with the following

     cat myjob-toICE.jdl
    [
    Type = "Job";
    JobType = "normal";
    InputSandbox = { "file:///home/emolinari/test.sh"};
    VirtualOrganisation = "dteam";
    Executable="test.sh";
    Arguments="Hello ";
    Requirements = ( RegExp("/cream-",other.GlueCEUniqueID));
    Rank = 0;
    fuzzyrank = true;
    StdOutput="message.txt";
    StdError="err.log";
    OutputSandbox={"message.txt","err.log",".BrokerInfo"};
    usertags = [ jdl = "normal job to ICE" ];
    RetryCount = 0;
    ShallowRetryCount = 3;
    ]
     glite-wms-job-list-match --config glite_wms_devel20.conf -a myjob-toICE.jdl
    
    Connecting to the service https://devel20.cnaf.infn.it:7443/glite_wms_wmproxy_server
    
    ==========================================================================
    
                         COMPUTING ELEMENT IDs LIST
     The following CE(s) matching your job requirements have been found:
    
            *CEId*
     - atlas-creamce-01.roma1.infn.it:8443/cream-lsf-atlasgcert
     - bocecream.bo.infn.it:8443/cream-pbs-cert
     - bocecream.bo.infn.it:8443/cream-pbs-certSL5
     - cccreamceli01.in2p3.fr:8443/cream-bqs-medium
     - cccreamceli01.in2p3.fr:8443/cream-bqs-short
     - ce01-lcg.cr.cnaf.infn.it:8443/cream-lsf-dteam
     - ce07-lcg.cr.cnaf.infn.it:8443/cream-lsf-dteam
     - ce201.cern.ch:8443/cream-lsf-grid_2nh_dteam
     - ce201.cern.ch:8443/cream-lsf-grid_dteam
     - ce202.cern.ch:8443/cream-lsf-grid_2nh_dteam
     - ce202.cern.ch:8443/cream-lsf-grid_dteam
     - cert-15.pd.infn.it:8443/cream-lsf-cert
     - cream-38.pd.infn.it:8443/cream-pbs-creamtest1
     - cream-38.pd.infn.it:8443/cream-pbs-creamtest2
     - cream-ce.ct.infn.it:8443/cream-lsf-cert
     - cream-ce.pr.infn.it:8443/cream-pbs-cert
     - cream-ce.research-infrastructures.eu:8443/cream-pbs-cert
     - devce.cnaf.infn.it:8443/cream-pbs-cert
     - gridce0.pi.infn.it:8443/cream-lsf-cert
     - prod-ce-01.pd.infn.it:8443/cream-lsf-cert
     - t2-ce-01.to.infn.it:8443/cream-pbs-cert
     - t2-ce-01.to.infn.it:8443/cream-pbs-short
     - t2-ce-05.lnl.infn.it:8443/cream-lsf-cert1
    
    ==========================================================================
    

  • tried substituting the Requirement with

    Requirements = ( !RegExp("/cream-",other.GlueCEUniqueID));
    glite-wms-job-list-match --config glite_wms_devel20.conf -a myjob-toLcg.jdl
    
    Connecting to the service https://devel20.cnaf.infn.it:7443/glite_wms_wmproxy_server
    
    ==========================================================================
    
                         COMPUTING ELEMENT IDs LIST
     The following CE(s) matching your job requirements have been found:
    
            *CEId*
     - argoce01.na.infn.it:2119/jobmanager-lcgpbs-cert
     - atlas-ce-01.roma1.infn.it:2119/jobmanager-lcglsf-atlasgcert
     - atlas-ce-02.roma1.infn.it:2119/jobmanager-lcglsf-atlasgcert
     - atlasce01.na.infn.it:2119/jobmanager-lcgpbs-cert
     - boalice3.bo.infn.it:2119/jobmanager-lcgpbs-cert
     - boalice3.bo.infn.it:2119/jobmanager-lcgpbs-certSL5
     - cclcgceli01.in2p3.fr:2119/jobmanager-bqs-long
     - cclcgceli01.in2p3.fr:2119/jobmanager-bqs-medium
     - cclcgceli01.in2p3.fr:2119/jobmanager-bqs-short
     - cclcgceli02.in2p3.fr:2119/jobmanager-bqs-long
     - cclcgceli02.in2p3.fr:2119/jobmanager-bqs-medium
     - cclcgceli02.in2p3.fr:2119/jobmanager-bqs-short
     - cclcgceli03.in2p3.fr:2119/jobmanager-bqs-long
     - cclcgceli03.in2p3.fr:2119/jobmanager-bqs-medium
     - cclcgceli03.in2p3.fr:2119/jobmanager-bqs-short
     - cclcgceli04.in2p3.fr:2119/jobmanager-bqs-long
     - cclcgceli04.in2p3.fr:2119/jobmanager-bqs-medium
     - cclcgceli04.in2p3.fr:2119/jobmanager-bqs-short
     - cclcgceli07.in2p3.fr:2119/jobmanager-bqs-long
     - cclcgceli07.in2p3.fr:2119/jobmanager-bqs-medium
     - cclcgceli07.in2p3.fr:2119/jobmanager-bqs-short
     - cclcgceli08.in2p3.fr:2119/jobmanager-bqs-long
     - cclcgceli08.in2p3.fr:2119/jobmanager-bqs-medium
     - cclcgceli08.in2p3.fr:2119/jobmanager-bqs-short
     - ce-01.grid.sissa.it:2119/jobmanager-lcgpbs-cert
     - ce-01.roma3.infn.it:2119/jobmanager-lcgpbs-cert
     - ce01-lhcb-t2.cr.cnaf.infn.it:2119/jobmanager-lcglsf-cert_t2
     - ce02-lhcb-t2.cr.cnaf.infn.it:2119/jobmanager-lcglsf-cert_t2
     - ce04-lcg.cr.cnaf.infn.it:2119/jobmanager-lcglsf-dteam
     - ce05-lcg.cr.cnaf.infn.it:2119/jobmanager-lcglsf-dteam
     - ce06-lcg.cr.cnaf.infn.it:2119/jobmanager-lcglsf-dteam
     - ce103.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce103.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce104.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce104.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce105.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce105.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce106.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce106.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce107.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce107.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce112.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce112.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce113.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce113.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce114.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce114.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce124.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce124.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce125.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce125.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce126.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce126.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce127.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce127.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce128.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce128.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce129.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce129.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce130.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce130.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce131.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce131.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce132.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce132.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - ce133.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
     - ce133.cern.ch:2119/jobmanager-lcglsf-grid_dteam
     - cmsce01.na.infn.it:2119/jobmanager-lcgpbs-cert
     - grid-ce-01.ba.infn.it:2119/jobmanager-lcgpbs-cert
     - grid-ce.lns.infn.it:2119/jobmanager-lcgpbs-cert
     - grid-ce.lns.infn.it:2119/jobmanager-lcgpbs-infinite
     - grid-ce.lns.infn.it:2119/jobmanager-lcgpbs-long
     - grid-ce.lns.infn.it:2119/jobmanager-lcgpbs-short
     - grid-ce2.pr.infn.it:2119/jobmanager-pbs-cert
     - grid-eo-engine04.esrin.esa.int:2119/jobmanager-lcgpbs-cert
     - grid0.fe.infn.it:2119/jobmanager-lcgpbs-cert
     - grid001.ts.infn.it:2119/jobmanager-lcglsf-cert
     - grid002.ca.infn.it:2119/jobmanager-lcglsf-cert
     - grid01.ge.infn.it:2119/jobmanager-lcglsf-cert
     - grid012.ct.infn.it:2119/jobmanager-lcglsf-cert
     - gridce.ilc.cnr.it:2119/jobmanager-lcgpbs-cert
     - gridce.pg.infn.it:2119/jobmanager-lcgpbs-cert
     - gridce.sns.it:2119/jobmanager-lcgpbs-cert
     - gridce1.pi.infn.it:2119/jobmanager-lcglsf-cert
     - gridce2.pi.infn.it:2119/jobmanager-lcglsf-cert
     - gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-cert
     - griditce01.na.infn.it:2119/jobmanager-lcgpbs-cert
     - lcg-ce.research-infrastructures.eu:2119/jobmanager-lcgpbs-cert
     - linucs-ce-01.cs.infn.it:2119/jobmanager-lcgpbs-atlasgcert
     - pamelace01.na.infn.it:2119/jobmanager-lcgpbs-cert
     - pbs-enmr.cerm.unifi.it:2119/jobmanager-lcgpbs-cert
     - prod-ce-02.pd.infn.it:2119/jobmanager-lcglsf-cert
     - t2-ce-01.lnl.infn.it:2119/jobmanager-lcglsf-cert1
     - t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-cert
     - t2-ce-02.lnl.infn.it:2119/jobmanager-lcglsf-cert1
     - t2-ce-02.mi.infn.it:2119/jobmanager-lcgcondor-cert
     - t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-cert
     - t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-short
     - t2-ce-03.lnl.infn.it:2119/jobmanager-lcglsf-cert1
     - t2-ce-04.lnl.infn.it:2119/jobmanager-lcglsf-cert1
     - t2-ce-06.lnl.infn.it:2119/jobmanager-lcglsf-cert1
     - test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cert
     - test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-parallel
     - virgo-ce.roma1.infn.it:2119/jobmanager-lcgpbs-cert
    
    

List match with data

• • • glite-wms-job-submit --config glite_wms_devel20.conf -a myjob-toICE.jdl
      
      Connecting to the service https://devel20.cnaf.infn.it:7443/glite_wms_wmproxy_server
      
      
      ====================== glite-wms-job-submit Success ======================
      
      The job has been successfully submitted to the WMProxy
      Your job identifier is:
      
      https://devel15.cnaf.infn.it:9000/Atu_PYr8SfD3C4VGCU_SjQ
      
      ==========================================================================
       glite-wms-job-status https://devel15.cnaf.infn.it:9000/Atu_PYr8SfD3C4VGCU_SjQ
      
      
      *************************************************************
      BOOKKEEPING INFORMATION:
      
      Status info for the Job : https://devel15.cnaf.infn.it:9000/Atu_PYr8SfD3C4VGCU_SjQ
      Current Status:     Done (Success)
      Exit code:          0
      Status Reason:      Job Terminated Successfully
      Destination:        ce202.cern.ch:8443/cream-lsf-grid_dteam
      Submitted:          Mon Apr 26 15:00:37 2010 CEST
      *************************************************************
      
       glite-wms-job-output https://devel15.cnaf.infn.it:9000/Atu_PYr8SfD3C4VGCU_SjQ
      
      Connecting to the service https://devel20.cnaf.infn.it:7443/glite_wms_wmproxy_server
      
      
      ================================================================================
      
                              JOB GET OUTPUT OUTCOME
      
      Output sandbox files for the job:
      https://devel15.cnaf.infn.it:9000/Atu_PYr8SfD3C4VGCU_SjQ
      have been successfully retrieved and stored in the directory:
      /tmp/jobOutput/emolinari_Atu_PYr8SfD3C4VGCU_SjQ
      --------------------------
      ls /tmp/jobOutput/emolinari_Atu_PYr8SfD3C4VGCU_SjQ/
      err.log  message.txt
      
      

• • • glite-wms-job-submit --config glite_wms_devel20.conf -a myjob-toLcg.jdl
      
      Connecting to the service https://devel20.cnaf.infn.it:7443/glite_wms_wmproxy_server
      
      
      ====================== glite-wms-job-submit Success ======================
      
      The job has been successfully submitted to the WMProxy
      Your job identifier is:
      
      https://devel15.cnaf.infn.it:9000/k0ue-prhJAvtAvjL2x_7Qg
      ----------------------------------------------------------------------
      glite-wms-job-status https://devel15.cnaf.infn.it:9000/k0ue-prhJAvtAvjL2x_7Qg
      
      
      *************************************************************
      BOOKKEEPING INFORMATION:
      
      Status info for the Job : https://devel15.cnaf.infn.it:9000/k0ue-prhJAvtAvjL2x_7Qg
      Current Status:     Done (Success)
      Logged Reason(s):
          -
          - Job terminated successfully
      Exit code:          0
      Status Reason:      Job terminated successfully
      Destination:        ce131.cern.ch:2119/jobmanager-lcglsf-grid_2nh_dteam
      Submitted:          Mon Apr 26 15:18:42 2010 CEST
      *************************************************************
      glite-wms-job-output https://devel15.cnaf.infn.it:9000/k0ue-prhJAvtAvjL2x_7Qg
      
      Connecting to the service https://devel20.cnaf.infn.it:7443/glite_wms_wmproxy_server
      
      
      ================================================================================
      
                              JOB GET OUTPUT OUTCOME
      
      Output sandbox files for the job:
      https://devel15.cnaf.infn.it:9000/k0ue-prhJAvtAvjL2x_7Qg
      have been successfully retrieved and stored in the directory:
      /tmp/jobOutput/emolinari_k0ue-prhJAvtAvjL2x_7Qg
      
      ================================================================================
      

• Bug #55452: CMS production struck by waves of "Globus error 10: data transfer to the server failed" FIXED NOT CERTIFIED

• Bug #59240: [ICE] abort reasons not always printed in its logfile FIXED NOT CERTIFIED

• Bug #42288: Problem in forwarding cerequirements to a CREAM CE FIXED

• Bug #55452: CMS production struck by waves of "Globus error 10: data transfer to the server failed" FIX NOT CERTIFIED

• Bug #59453: [ICE] polling needs to be improved FIXED NOT CERTIFIED
  • See also these tests

• Bug #42288: Problem in forwarding cerequirements to a CREAM CE FIXED
  • description of the problem --> "The parameters to be forwarded specified in the Requirements attribute of the .jdl classad are NOT considered and ICE does not send them to the CE, therefore the classad passed to BLAH does not contain them"
    • submitted the following .jdl via WMS:

       cat myjob_forwardReq.jdl
      [
      Type = "Job";
      JobType = "normal";
      InputSandbox = { "file:///home/emolinari/test.sh"};
      VirtualOrganisation = "dteam";
      Executable="test.sh";
      Arguments="Hello ";
      requirements = (other.GlueCEUniqueID == "cream-19.pd.infn.it:8443/cream-lsf-testbedB_1") && (other.GlueHostMainMemoryRAMSize >= 0) ;
      Rank = 0;
      myproxy = myproxy.cnaf.infn.it;
      fuzzyrank = true;
      StdOutput="message.txt";
      StdError="err.log";
      OutputSandbox={"message.txt","err.log",".BrokerInfo"};
      RetryCount = 0;
      ShallowRetryCount = 3;
      ]

    • checked in the ice log file on the WMS, /var/log/glite/ice.log, that the CERequirement field of the .jdl gets populated as in the following

       CeRequirements = "true && ( true && ( true && ( other.GlueHostMainMemoryRAMSize >= 0 ) ) )"; 

    • checked on the CE that blah generates the correct classad with the requirements to be forwarded, as in the following:

       cat /tmp/subfile
      #!/bin/bash
      # LSF job wrapper generated by lsf_submit.sh
      # on Wed Apr 14 19:13:10 CEST 2010
      #
      # LSF directives:
      #BSUB -L /bin/bash
      #BSUB -J cre19_725998524
      #BSUB -q testbedB_1
      #BSUB -R "select[mem>=0]"
      ......

• Bug #61405: [ICE] Missing proxy validity evaluation in ICE FIXED
  • Submit this jdl with a proxy of 30minutes NOT registered to the myproxy server (myproxy.cnaf.infn.it):

     [
      executable = "/bin/sleep"; 
      arguments = "2000"; 
      MyProxyServer = "myproxy.cnaf.infn.it"; 
      requirements = ( other.GlueCEStateStatus == "testbedb" ); 
      DefaultRank =  -other.GlueCEStateEstimatedResponseTime; 
     ]

  • After a while submit the same jdl with a fresh proxy and look in the ice's log if this new proxy is used to refresh the delegation of the previous job:
    • First it should try to renew the proxy contacting the myproxy server:

      2010-04-14 11:47:40,622 DEBUG - iceCommandDelegationRenewal::renewAllDelegations() - Contacting MyProxy server [myproxy.cnaf.infn.it] for user dn [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] with proxy certificate [/var/glite/ice/persist_dir/2A9DAF04C398C21D6ADF7E884BC192ED95AF554C.betterproxy] to renew it...
      2010-04-14 11:47:40,622 DEBUG - iceCommandDelegationRenewal::renewAllDelegations() - Executing command [export X509_USER_CERT=/var/glite/wms.proxy; export X509_USER_KEY=/var/glite/wms.proxy; /opt/glite/bin/glite-wms-ice-proxy-renew -s myproxy.cnaf.infn.it -p /var/glite/ice/persist_dir/2A9DAF04C398C21D6ADF7E884BC192ED95AF554C.betterproxy -o /var/glite/ice/persist_dir/2A9DAF04C398C21D6ADF7E884BC192ED95AF554C.betterproxy.renewed]...
      2010-04-14 11:47:40,783 DEBUG - iceCommandDelegationRenewal::renewAllDelegations() - Command output is [/opt/glite/bin/glite-wms-ice-proxy-renew: glite_renewal_core_renew() failed: Error contacting MyProxy server for proxy /var/glite/ice/persist_dir/2A9DAF04C398C21D6ADF7E884BC192ED95AF554C.betterproxy: ERROR from myproxy-server (myproxy.cnaf.infn.it):
      X509_verify_cert() failed: certificate has expired
      
      �]

    • Then it should use the proxy of the last arrived job to renew the delegation:

      2010-04-14 11:47:40,783 DEBUG - iceCommandDelegationRenewal::renewAllDelegations() - Looking for the better proxy for DN [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] MyProxy Server name [myproxy.cnaf.infn.it]...
      2010-04-14 11:47:40,783 INFO - iceCommandDelegationRenewal::renewAllDelegations() - Will Renew Delegation ID [12712381542E417936wms0072Ecnaf2Einfn2Eit] with BetterProxy [/var/glite/ice/persist_dir/2A9DAF04C398C21D6ADF7E884BC192ED95AF554C.betterproxy] that will expire on [Wed Apr 14 12:12:47 2010]
      2010-04-14 11:47:40,783 INFO - CreamProxy_DelegateRenew::execute() - Calling renewProxyReq to remote service [https://cream-39.pd.infn.it:8443/ce-cream/services/gridsite-delegation]

• Bug #58977: [ICE] Wrong database colum name in ICE SQL query FIXED
  • Submit some jobs to a ce (e.g. cream-25.pd.infn.it:8443/cream-lsf-testbedB_2):

    [root@wms007 20100409]# queryDb -v -C -G
    [https://cream-25.pd.infn.it:8443/CREAM881525184]  [https://devel17.cnaf.infn.it:9000/V2Lj0_-XWkrjRKMaf3f6ng]
    [https://cream-25.pd.infn.it:8443/CREAM425827870]  [https://devel17.cnaf.infn.it:9000/4TKQu1U_daCMMb2mRDR2cA]
    [https://cream-25.pd.infn.it:8443/CREAM543141647]  [https://devel17.cnaf.infn.it:9000/4wqozVNHEVUXWx5UzUaXqA]
    [https://cream-25.pd.infn.it:8443/CREAM769586568]  [https://devel17.cnaf.infn.it:9000/XLcaGE3kR3h8-oj8cMWE_A]
    [https://cream-25.pd.infn.it:8443/CREAM192029588]  [https://devel17.cnaf.infn.it:9000/PuOGoOxMf-pbfFu-wSkACw]
    [https://cream-25.pd.infn.it:8443/CREAM378177464]  [https://devel17.cnaf.infn.it:9000/T8ZKSu5zZPZY-Ee1gLRX5A]
    [https://cream-25.pd.infn.it:8443/CREAM299069473]  [https://devel17.cnaf.infn.it:9000/Xh1AMEor9hWOx4picngYkA]
    [https://cream-25.pd.infn.it:8443/CREAM012571708]  [https://devel17.cnaf.infn.it:9000/YjpCU6dfrLsDBs6wU_D3Hg]
    [https://cream-25.pd.infn.it:8443/CREAM561236418]  [https://devel17.cnaf.infn.it:9000/00Qc7RnutRORYVOd0ShIKg]
    [https://cream-25.pd.infn.it:8443/CREAM972351884]  [https://devel17.cnaf.infn.it:9000/ksz80OflJnDE_ynWHmKTwQ]
    [https://cream-25.pd.infn.it:8443/CREAM827240561]  [https://devel17.cnaf.infn.it:9000/WwfftKdV6_5lSgihPOUsaA]
    [https://cream-25.pd.infn.it:8443/CREAM573497695]  [https://devel17.cnaf.infn.it:9000/S5zbkyK72hv2LXwUD1vAFw]
    [https://cream-25.pd.infn.it:8443/CREAM735112819]  [https://devel17.cnaf.infn.it:9000/0J9nTy1tJxkcuRJ9oTZACw]
    [https://cream-25.pd.infn.it:8443/CREAM526570551]  [https://devel17.cnaf.infn.it:9000/Rcl0TypyUTXMwtLk86R3yA]
    [https://cream-25.pd.infn.it:8443/CREAM992848449]  [https://devel17.cnaf.infn.it:9000/xfH1fkIroQwNvlVBdn8N5A]
    [https://cream-25.pd.infn.it:8443/CREAM944698480]  [https://devel17.cnaf.infn.it:9000/xjiyHJo3rkUsXXHHe0s6yg]
    [https://cream-25.pd.infn.it:8443/CREAM729677007]  [https://devel17.cnaf.infn.it:9000/FIZA1Mjb4moUNel1N7UXvw]
    [https://cream-25.pd.infn.it:8443/CREAM589660323]  [https://devel17.cnaf.infn.it:9000/5DJlLG7M0v3C_-WMDKSdXQ]
    [https://cream-25.pd.infn.it:8443/CREAM994745139]  [https://devel17.cnaf.infn.it:9000/T_UdwnjC55dIVrPxJOVvmg]
    [https://cream-25.pd.infn.it:8443/CREAM228224655]  [https://devel17.cnaf.infn.it:9000/URw39mrv7jj-buJ3KDza8w]
    [https://cream-25.pd.infn.it:8443/CREAM397635733]  [https://devel17.cnaf.infn.it:9000/f3FOGwNoWpHyWkxO_87AIg]
    [https://cream-25.pd.infn.it:8443/CREAM510341828]  [https://devel17.cnaf.infn.it:9000/vEfH5j5_5R_7jFNrntEsog] 
    [https://cream-25.pd.infn.it:8443/CREAM788890890]  [https://devel17.cnaf.infn.it:9000/y0IVYbdR_UWbTmrXY5O8fA]
    
    ------------------------------------------------
    23 item(s) found

  • Check also the db_id registered in the ice's database

    [root@wms007 20100409]#  sqlite3 /var/glite/ice/persist_dir/ice.db "SELECT db_id, ceurl from ce_dbid;"
    1270820425000|https://cream-25.pd.infn.it:8443/ce-cream/services/CREAM2

  • Stop the cream CE. Drop its database. Create a new empty one. Restart the CE.
  • Check what happen in the ice log file:

    2010-04-09 16:16:53,953 WARN - iceCommandEventQuery::execute() -  TID=[150150560] *** CREAM HAS PROBABLY BEEN SCRATCHED. GOING TO ERASE ALL JOBS RELATED TO OLD DB_ID [1270820425000] ***

  • Check if there are jobs in the Ice's database:

    [root@wms007 persist_dir]# queryDb -v  -C -G
    
    ------------------------------------------------
    0 item(s) found

  • and if the db_id has been changed:

    [root@wms007 persist_dir]#  sqlite3 /var/glite/ice/persist_dir/ice.db "SELECT db_id, ceurl from ce_dbid;"
    1270822483000|https://cream-25.pd.infn.it:8443/ce-cream/services/CREAM2

  • Look at the status of a job that has been removed:

        Status info for the Job : https://devel17.cnaf.infn.it:9000/xjiyHJo3rkUsXXHHe0s6yg
        Current Status:     Aborted
        Logged Reason(s):
            - job completed
        Status Reason:      CREAM'S database has been scratched and all its jobs have been lost
        Destination:        cream-25.pd.infn.it:8443/cream-lsf-testbedB_2
        Submitted:          Fri Apr  9 16:11:34 2010 CEST

• • Submit a jdl with a file of more than 10Mb in the OutputSandbox parameter and set also the corresponding OutputSandboxDestURI parameter

• Bug #56636: [ICE] statistics counters for monitoring FIXED
  • Verify the command and its options:

    [root@wms007 persist_dir]#  queryStats -t "2010-04-08 00:00:00"
    JOB_REGISTERED=2
    JOB_IDLE=2
    JOB_RUNNING=2
    JOB_REALLY-RUNNING=2
    JOB_DONE-OK=2
    
    [root@wms007 persist_dir]#  queryStats -f "2010-04-08 00:00:01" -t "2010-04-09 11:00:00"
    JOB_REGISTERED=4
    JOB_IDLE=4
    JOB_RUNNING=4
    JOB_REALLY-RUNNING=4
    JOB_DONE-OK=1
    JOB_DONE-FAILED=3
    
    [root@wms007 persist_dir]#  queryStats -f "2010-04-09 11:00:01"
    JOB_REGISTERED=255
    JOB_IDLE=255
    JOB_RUNNING=193
    JOB_REALLY-RUNNING=204
    JOB_DONE-OK=191
    JOB_ABORTED=6
    
    [root@wms007 persist_dir]#  queryStats
    JOB_REGISTERED=261
    JOB_IDLE=261
    JOB_RUNNING=199
    JOB_REALLY-RUNNING=210
    JOB_DONE-OK=194
    JOB_DONE-FAILED=3
    JOB_ABORTED=6

• Bug #58099: WMS purger forces purge of jobs if LB cannot be reached FIXED
  • Stop the LBServer and then run the cron purger:

    07 Apr, 16:09:13 -E: [Error] query_job_status(purger.cpp:125): https://devel17.cnaf.infn.it:9000/yeoXs2eB1kvOaPp0Mtjthg:: edg_wll_JobStat [111] Connection refused(edg_wll_gss_connect())
    [glite@wms007 ~]$ 

  • Verify that the SandBox dir has not been removed:

     [glite@wms007 ~]$ ls -l /var/glite/SandboxDir/ye/https_3a_2f_2fdevel17.cnaf.infn.it_3a9000_2fyeoXs2eB1kvOaPp0Mtjthg/
    total 16
    drwxrwx---  2 dteam008 glite 4096 Apr  6 14:34 input
    drwxrwx---  2 dteam008 glite 4096 Apr  6 14:46 output
    drwxrwx---  2 dteam008 glite 4096 Apr  6 14:34 peek
    lrwxrwxrwx  1 glite    glite  102 Apr  6 14:34 user.proxy -> /var/glite/SandboxDir/Uo/https_3a_2f_2fdevel17.cnaf.infn.it_3a9000_2fUow8XY0NGbyumU3PPGMSng/user.proxy
    

  • Restart LBServer and verify that now the SBD of the job is purged:

    [glite@wms007 ~]$  /opt/glite/sbin/glite-wms-purgeStorage.sh  -p /var/glite/SandboxDir/ye  -t 10000
    07 Apr, 16:18:07 -I: [Info] operator()(purger.cpp:449): https://devel17.cnaf.infn.it:9000/yeoXs2eB1kvOaPp0Mtjthg: removed DONE job
    [glite@wms007 ~]$ ls -l /var/glite/SandboxDir/ye/https_3a_2f_2fdevel17.cnaf.infn.it_3a9000_2fyeoXs2eB1kvOaPp0Mtjthg/
    ls: /var/glite/SandboxDir/ye/https_3a_2f_2fdevel17.cnaf.infn.it_3a9000_2fyeoXs2eB1kvOaPp0Mtjthg/: No such file or directory

• Bug #64698: jobwrapper max osb limit should be considered only if the gridftp server is the wms FIXED only for LCG-CE
  • Set MaxOutputSandboxSize = 10000000; in section WorkloadManager of file glite_wms.conf
  • Submit a jdl with a file of more than 10Mb in the !OutputSandbox parameter and set also the corresponding OutputSandboxDestURI parameter
  • Check the output dir in the SE:

    [root@devel18 tmp]# ls -lh
    -rw-r--r--  1 dteam044 dteam  50M Apr  7 16:23 bigfile
    -rw-r--r--  1 dteam044 dteam  646 Apr  7 16:23 ls.out

  • If you don't set OutputSandboxDestURI in the jdl, than the SandBox dir in the WMS should contain a .tail file of less than 10Mb:

    [root@wms007 persist_dir]#  ls -lh /var/glite/SandboxDir/A1/https_3a_2f_2fdevel17.cnaf.infn.it_3a9000_2fA1cdkhzepvrCjiU_5fTaKbpg/output/
    total 9.6M
    -rw-r--r--  1 dteam008 dteam 9.6M Apr  7 16:26 bigfile.tail
    -rw-r--r--  1 dteam008 dteam  637 Apr  7 16:26 ls.out

Check bugs

• • JC work:

• • Note: collections stay in status 'waiting' when all the nodes are Done (Success) except for one that is 'Cancelled'

• Bug #42288: Problem in forwarding cerequirements to a CREAM CE NOT TESTED

• Bug #55452: CMS production struck by waves of "Globus error 10: data transfer to the server failed" NOT TESTED

• Bug #56636: [ICE] statistics counters for monitoring NOT TESTED

• Bug #58099: WMS purger forces purge of jobs if LB cannot be reached NOT TESTED

• Bug #58977: [ICE] Wrong database colum name in ICE SQL query NOT TESTED

• Bug #59240: [ICE] abort reasons not always printed in its logfile NOT TESTED

• Bug #59453: [ICE] polling needs to be improved NOT TESTED

• Bug #61405: [ICE] Missing proxy validity evaluation in ICE NOT TESTED

• Bug #64698: jobwrapper max osb limit should be considered only if the gridftp server is the wms NOT TESTED

• • Stopped gLite services and deleted the jobdir under '/var/glite/jobcontrol'

    [root@wms007 jobcontrol]# pwd
    /var/glite/jobcontrol
    [root@wms007 jobcontrol]# rm -rf jobdir
    [root@wms007 jobcontrol]# ls
    condorio  submit

    • re-started the JC service checking that the jobdir gets recreated

      [root@wms007 jobcontrol]# /opt/glite/etc/init.d/glite-wms-jc start JobController
      Starting JobController daemon(s)
         Starting JobController...                          [  OK  ]
      [root@wms007 jobcontrol]# ls
      condorio  jobdir  lock  submit
      [root@wms007 ice]# /opt/glite/etc/init.d/glite-wms-jc status JobController
      JobController running in pid: 3625

  • Stopped gLite services and deleted the jobdir under '/var/glite/ice'

    [root@wms007 ice]# pwd
    /var/glite/ice
    [root@wms007 ice]# ls
    jobdir  persist_dir
    [root@wms007 ice]# rm -rf jobdir/
    [root@wms007 ice]# ls
    persist_dir

    • re-started the ICE service checking that the jobdir gets recreated

      [root@wms007 ice]# /opt/glite/etc/init.d/glite-wms-ice start
      starting ICE... ok
      [root@wms007 ice]# ls
      jobdir  persist_dir
      [root@wms007 ice]# /opt/glite/etc/init.d/glite-wms-ice status
      /opt/glite/bin/glite-wms-ice-safe (pid 22783) is running...

  • Stopped gLite services and deleted all the jobdirs

    [root@wms007 glite]# ls workload_manager/ jobcontrol/ ice/
    ice/:
    persist_dir
    
    jobcontrol/:
    condorio  submit
    
    workload_manager/:
    ismdump.fl

    • re-started the WM service checking that all the jobdirs get recreated

      [root@wms007 glite]# /opt/glite/etc/init.d/glite-wms-wm start
      starting workload manager... ok
      [root@wms007 glite]# ls workload_manager/ jobcontrol/ ice/
      ice/:
      jobdir  persist_dir
      
      jobcontrol/:
      condorio  jobdir  submit
      
      workload_manager/:
      ismdump.fl  jobdir
      [root@wms007 glite]# /opt/glite/etc/init.d/glite-wms-wm status
      /opt/glite/bin/glite-wms-workload_manager (pid 23259) is running...

• • Stopped gLite services and deleted the jobdir under '/var/glite/workload_manager'

    [root@wms007 jobdir]# service gLite stop
    [...]
    [root@wms007 workload_manager]# pwd
    /var/glite/workload_manager
    [root@wms007 workload_manager]# ls
    ismdump.fl  jobdir
    [root@wms007 workload_manager]# rm -rf jobdir
    [root@wms007 workload_manager]# ls
    ismdump.fl 

  • re-started the LM service checking that the jobdir gets recreated

    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-lm start
    Starting LogMonitor...                                     [  OK  ]
    [root@wms007 workload_manager]# ls
    ismdump.fl  jobdir
    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-lm status
    Logmonitor running...

  • Comment Input/InputType parameter in wms conf file (Sections: ICE, WorkloadManager and JobController).
  • Try to start JobController:

    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-jc start JobController
    Starting !JobController daemon(s)
     Please set Input parameter in glite_wms.conf - JC section [FAILED]
    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-jc status JobController
    JobController stopped.

  • Try to start LogMonitor:

    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-lm start
    Starting LogMonitor...Please set Input parameter in glite_wms.conf - WM section
                                                               [FAILED]
    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-lm status
    LogMonitor stopped.

  • Try to start ICE:

    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-ice start
    starting ICE... failure
    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-ice status
    /opt/glite/bin/glite-wms-ice-safe is not running

  • Try to start WorkloadManager:

    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-wm start
    starting workload manager... Please set Input parameter in  - WM section
    Please set DispatcherType parameter in  - WM section
    Please set Input parameter in  - JC section
    Please set InputType parameter in  - JC section
    Please set Input parameter in  - ICE section
    Please set InputType parameter in  - ICE section
    failure
    [root@wms007 workload_manager]# /opt/glite/etc/init.d/glite-wms-wm status
    /opt/glite/bin/glite-wms-workload_manager is not running

• Bug #63989: [ICE] doesn't handle exception raised by jobDir::new_entries() FIXED
  • Change the permission of the new directory in jobdir:

    [root@wms007 jobdir]# chmod 111 new/
    [root@wms007 jobdir]# ls -l
    total 48
    d--x--x--x  2 glite glite 40960 Mar 24 16:13 new
    drwxr-xr-x  2 glite glite  4096 Mar 24 16:13 old
    drwxr-xr-x  2 glite glite  4096 Mar 24 16:13 tmp

  • Look in ICE's log:

    2010-03-25 09:45:39,545 ERROR - Request_source_jobdir::get_requests() - Error returned by method jobDir::new_entries(): boost::filesystem::directory_iterator constructor: "/var/glite/ice/jobdir/new": Permission denied
    2010-03-25 09:45:40,546 ERROR - Request_source_jobdir::get_requests() - Error returned by method jobDir::new_entries(): boost::filesystem::directory_iterator constructor: "/var/glite/ice/jobdir/new": Permission denied

• Bug #61748: [ICE] EventQuery/Polling must be done also to blacklisted CE FIXED
  • Submit some jobs to a CreamCEVerify
  • Trigger a socket timeout so that ICE blacklisted the CreamCE :

    2010-03-24 15:58:40,753 ERROR - CreamProxyMethod::execute() - Connection timed out to CREAM: "EOF detected during communicati
    on. Probably service closed connection or SOCKET TIMEOUT occurred." on try 3/3. Blacklisting endpoint and giving up.
    2010-03-24 15:58:40,753 DEBUG - CEBlackList::blacklist_endpoint() - Blacklisting CE https://cream-25.pd.infn.it:8443/ce-cream
    /services/gridsite-delegation until Wed Mar 24 16:08:40 2010

  • Verify that the QueryEvent commad is called in any case:

    2010-03-24 16:05:28,952 DEBUG - eventStatusPoller::body() - Adding EventQuery command for couple (/C=IT/O=INFN/OU=Personal Ce
    rtificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL, https://cream-25.pd.infn.it:8443/ce-cream/services/C
    REAM2) to the thread pool...

  • Instead a submission fails:

    2010-03-24 15:58:43,265 DEBUG - Delegation_manager::delegate() - Creating new delegation with delegation id [12694427232E2651
    16wms0072Ecnaf2Einfn2Eit] CREAM URL [https://cream-25.pd.infn.it:8443/ce-cream/services/CREAM2] Delegation URL [https://cream
    -25.pd.infn.it:8443/ce-cream/services/gridsite-delegation] user DN [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio 
    Gianelle-/dteam/Role=NULL/Capability=NULL] proxy hash [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dte
    am/Role=NULL/Capability=NULL] MyProxy Server [myproxy.cern.ch] Expiring on [Thu Mar 25 12:54:02 2010]
    2010-03-24 15:58:43,265 DEBUG - CEBlackList::is_blacklisted() - CE https://cream-25.pd.infn.it:8443/ce-cream/services/gridsit
    e-delegation is blacklisted until Wed Mar 24 16:08:40 2010
    2010-03-24 15:58:43,265 ERROR - Delegation_manager::delegate() - FAILED Creation of a new delegation with delegation id [1269
    4427232E265116wms0072Ecnaf2Einfn2Eit] CREAM URL [https://cream-25.pd.infn.it:8443/ce-cream/services/CREAM2] Delegation URL [h
    ttps://cream-25.pd.infn.it:8443/ce-cream/services/gridsite-delegation] user DN [/C=IT/O=INFN/OU=Personal Certificate/L=Padova
    /CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] proxy hash [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio G
    ianelle-/dteam/Role=NULL/Capability=NULL] MyProxy Server [myproxy.cern.ch] - ERROR is: [The endpoint is blacklisted]
    2010-03-24 15:58:43,265 ERROR - iceCommandSubmit::execute() -  TID=[159308760] Error during submission of jdl= Fatal Exceptio
    n is:Failed to create a delegation id for job https://devel17.cnaf.infn.it:9000/UoVsvjIj1CPluHb81xM_pQ: reason is The endpoin
    t is blacklisted

• Bug #60668: [ICE] does not respect LB server/proxy selection through the LBproxy attribute FIXED
  • Set LBProxy = false; in glite_wms.conf (section Common), restart ice and submit...

    mysql> select * from events where jobid="YFyqjw3FF-BO-0U5BxCOtA";
    +------------------------+-------+------+-----------------+---------------------+---------------------+----------------------------------+--------+-------+---------------------+
    | jobid                  | event | code | prog            | host                | time_stamp          | userid                           | usec   | level | arrived             |
    +------------------------+-------+------+-----------------+---------------------+---------------------+----------------------------------+--------+-------+---------------------+
    | YFyqjw3FF-BO-0U5BxCOtA |     0 |    5 | WorkloadManager | wms007.cnaf.infn.it | 2010-03-24 12:04:39 | bdd27610035bb0ec9287e2ecaa3da2eb | 394848 |     8 | 2010-03-24 12:04:39 |
    | YFyqjw3FF-BO-0U5BxCOtA |     1 |   15 | WorkloadManager | wms007.cnaf.infn.it | 2010-03-24 12:04:39 | bdd27610035bb0ec9287e2ecaa3da2eb | 548652 |     8 | 2010-03-24 12:04:39 |
    | YFyqjw3FF-BO-0U5BxCOtA |     2 |    4 | WorkloadManager | wms007.cnaf.infn.it | 2010-03-24 12:04:39 | bdd27610035bb0ec9287e2ecaa3da2eb | 608084 |     8 | 2010-03-24 12:04:39 |
    | YFyqjw3FF-BO-0U5BxCOtA |     3 |    4 | WorkloadManager | wms007.cnaf.infn.it | 2010-03-24 12:04:39 | bdd27610035bb0ec9287e2ecaa3da2eb | 657231 |     8 | 2010-03-24 12:04:39 |
    +------------------------+-------+------+-----------------+---------------------+---------------------+----------------------------------+--------+-------+---------------------+
    4 rows in set (0.00 sec)

  • * Set LBProxy = true; in glite_wms.conf (section Common), restart ice and submit...

    mysql> select * from events where jobid="SlKOGSnaW0oKO3TJqw9tbA";
    +------------------------+-------+------+-----------------+---------------------+---------------------+----------------------------------+--------+-------+---------------------+
    | jobid                  | event | code | prog            | host                | time_stamp          | userid                           | usec   | level | arrived             |
    +------------------------+-------+------+-----------------+---------------------+---------------------+----------------------------------+--------+-------+---------------------+
    | SlKOGSnaW0oKO3TJqw9tbA |     0 |   17 | NetworkServer   | wms007.cnaf.infn.it | 2010-03-24 12:09:53 | 3f82b966e8a77413044be1a9144a4af4 | 342720 |     8 | 2010-03-24 12:09:53 |
    | SlKOGSnaW0oKO3TJqw9tbA |     1 |   21 | NetworkServer   | wms007.cnaf.infn.it | 2010-03-24 12:09:53 | 3f82b966e8a77413044be1a9144a4af4 | 470416 |     8 | 2010-03-24 12:09:53 |
    | SlKOGSnaW0oKO3TJqw9tbA |     2 |   21 | NetworkServer   | wms007.cnaf.infn.it | 2010-03-24 12:09:53 | 3f82b966e8a77413044be1a9144a4af4 | 526402 |     8 | 2010-03-24 12:09:53 |
    | SlKOGSnaW0oKO3TJqw9tbA |     3 |    2 | NetworkServer   | wms007.cnaf.infn.it | 2010-03-24 12:09:54 | 3f82b966e8a77413044be1a9144a4af4 | 606511 |     8 | 2010-03-24 12:09:54 |
    | SlKOGSnaW0oKO3TJqw9tbA |     4 |    4 | NetworkServer   | wms007.cnaf.infn.it | 2010-03-24 12:09:54 | 3f82b966e8a77413044be1a9144a4af4 | 712100 |     8 | 2010-03-24 12:09:54 |
    | SlKOGSnaW0oKO3TJqw9tbA |     5 |    4 | NetworkServer   | wms007.cnaf.infn.it | 2010-03-24 12:09:55 | 3f82b966e8a77413044be1a9144a4af4 |  43631 |     8 | 2010-03-24 12:09:55 |
    | SlKOGSnaW0oKO3TJqw9tbA |     6 |    5 | WorkloadManager | wms007.cnaf.infn.it | 2010-03-24 12:09:55 | bdd27610035bb0ec9287e2ecaa3da2eb | 167414 |     8 | 2010-03-24 12:09:55 |
    | SlKOGSnaW0oKO3TJqw9tbA |     7 |   15 | WorkloadManager | wms007.cnaf.infn.it | 2010-03-24 12:09:55 | bdd27610035bb0ec9287e2ecaa3da2eb | 297333 |     8 | 2010-03-24 12:09:55 |
    | SlKOGSnaW0oKO3TJqw9tbA |     8 |    4 | WorkloadManager | wms007.cnaf.infn.it | 2010-03-24 12:09:55 | bdd27610035bb0ec9287e2ecaa3da2eb | 369636 |     8 | 2010-03-24 12:09:55 |
    | SlKOGSnaW0oKO3TJqw9tbA |     9 |    4 | WorkloadManager | wms007.cnaf.infn.it | 2010-03-24 12:09:55 | bdd27610035bb0ec9287e2ecaa3da2eb | 431565 |     8 | 2010-03-24 12:09:55 |
    | SlKOGSnaW0oKO3TJqw9tbA |    10 |    5 | JobController   | wms007.cnaf.infn.it | 2010-03-24 12:09:55 | bdd27610035bb0ec9287e2ecaa3da2eb | 745052 |     8 | 2010-03-24 12:09:55 |
    | SlKOGSnaW0oKO3TJqw9tbA |    11 |    1 | LogMonitor      | wms007.cnaf.infn.it | 2010-03-24 12:09:55 | bdd27610035bb0ec9287e2ecaa3da2eb | 846002 |     8 | 2010-03-24 12:09:55 |
    | SlKOGSnaW0oKO3TJqw9tbA |    12 |    1 | LogMonitor      | wms007.cnaf.infn.it | 2010-03-24 12:10:04 | bdd27610035bb0ec9287e2ecaa3da2eb | 869424 |     8 | 2010-03-24 12:10:04 |
    | SlKOGSnaW0oKO3TJqw9tbA |    13 |    8 | LogMonitor      | wms007.cnaf.infn.it | 2010-03-24 12:11:39 | bdd27610035bb0ec9287e2ecaa3da2eb |  94855 |     8 | 2010-03-24 12:11:39 |
    | SlKOGSnaW0oKO3TJqw9tbA |    14 |   25 | LogMonitor      | wms007.cnaf.infn.it | 2010-03-24 12:11:39 | bdd27610035bb0ec9287e2ecaa3da2eb | 181448 |     8 | 2010-03-24 12:11:39 |
    | SlKOGSnaW0oKO3TJqw9tbA |    15 |   10 | LogMonitor      | wms007.cnaf.infn.it | 2010-03-24 12:11:39 | bdd27610035bb0ec9287e2ecaa3da2eb | 250291 |     8 | 2010-03-24 12:11:39 |
    +------------------------+-------+------+-----------------+---------------------+---------------------+----------------------------------+--------+-------+---------------------+
    16 rows in set (0.00 sec)

• Bug #61312: [ICE] Error in handling user dn in ICE's poller FIXED
  • Submit 5 jobs to an old CreamCE (Cream 1.5) setting MyProxyServer attribute:

    2010-03-24 13:40:38,128 ERROR - iceCommandEventQuery::execute() -  TID=[159321352] Cannot query events for UserDN [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] CEUrl [https://cream-33.pd.infn.it:8443/ce-cream/services/CREAM2]. Exception Internal ex is [Received NULL fault; the error is due to another cause: FaultString=[No such operation 'QueryEventRequest'] - FaultCode=["http://xml.apache.org/axis/":Client] - FaultSubCode=["http://xml.apache.org/axis/":Client] - FaultDetail=[<ns2:hostname>cream-33.pd.infn.it</ns2:hostname>]]
    2010-03-24 13:40:38,128 WARN - iceCommandEventQuery::execute() -  TID=[159321352] Not present QueryEvent on CE [https://cream-33.pd.infn.it:8443/ce-cream/services/CREAM2]. Falling back to old-style StatusPoller.
    2010-03-24 13:40:38,128 INFO - iceCommandStatusPoller::execute() - Getting [100] jobs to poll for user [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] creamurl [https://cream-33.pd.infn.it:8443/ce-cream/services/CREAM2]
    2010-03-24 13:40:38,128 DEBUG - iceCommandStatusPoller::get_jobs_to_poll() - Collecting jobs to poll for userdn=[/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] creamurl=[https://cream-33.pd.infn.it:8443/ce-cream/services/CREAM2]. LIMIT set to [100]...
    2010-03-24 13:40:38,129 DEBUG - iceCommandStatusPoller::get_jobs_to_poll() - Finished collecting jobs to poll. [5] jobs are to poll
    [...]

• And so:

  [ale@cream-15 UI]$ glite-wms-job-status -v 0 -i testo --noint

Status info for the Job : https://devel17.cnaf.infn.it:9000/tt3GLYuIiHuwrmnl7fGVtA Current Status: Done (Success)

*********************************************************** BOOKKEEPING INFORMATION:

Status info for the Job : https://devel17.cnaf.infn.it:9000/lY9fdOgQk5RcaH99g23z5g Current Status: Done (Success)

*********************************************************** BOOKKEEPING INFORMATION:

Status info for the Job : https://devel17.cnaf.infn.it:9000/jta5KlBZEP-r2KbE0SB0Vw Current Status: Done (Success)

*********************************************************** BOOKKEEPING INFORMATION:

Status info for the Job : https://devel17.cnaf.infn.it:9000/TNqI_PbRyqgFAN3L52IpKQ Current Status: Done (Success)

*********************************************************** BOOKKEEPING INFORMATION:

Status info for the Job : https://devel17.cnaf.infn.it:9000/V7Pnv2yE47CdHKgQmRaIvQ Current Status: Done (Success) *************************************************************

• Bug #61413: [ICE] should not call EventQuery for a userDN if he/she doesn't have active jobs FIXED
  • Submit a job to a CreamCE and wait until it finished.
  • Submit another job to a different CreamCE, you should not see any query to the previous used CreamCE.

• Bug #59399: [ICE] doesn't correctly handle request in jobdir/old when it is restarted FIXED
  • Verify submitting a big collection to cream CEs, and then restarting ICE in the middle of the submit process:

    2010-03-23 15:55:43,604 DEBUG - iceCommandSubmit::try_to_submit() -  TID=[168434952] Going to START CreamJobID [https://cream
    -32.pd.infn.it:8443/CREAM036926381] related to GridJobID [https://devel17.cnaf.infn.it:9000/iM8C3YV12fwhvIG5mNip5Q]...

  • restarting ice...

    2010-03-23 15:55:45,760 DEBUG - ICE VersionID is [Fri Mar 19 13:53:17 CET 2010] ProcessID=[23579]
    2010-03-23 15:55:45,760 INFO - glite-wms-ice::main() - Host certificate is [/home/glite/.certs/hostcert.pem]
    2010-03-23 15:55:45,817 DEBUG - iceThreadPool::iceThreadPool(ICE Submission Pool) - Creating 10 worker threads
    2010-03-23 15:55:45,819 DEBUG - iceThreadPool::iceThreadPool(ICE Poller Pool) - Creating 5 worker threads
    [...]
    2010-03-23 15:55:48,967 INFO - iceCommandSubmit::execute() -  TID=[144321160] This request is a Submission...
    2010-03-23 15:55:48,968 INFO - iceCommandSubmit::try_to_submit() -  TID=[144321160] GridJobID [https://devel17.cnaf.infn.it:9
    000/iM8C3YV12fwhvIG5mNip5Q] has already been REGISTERED. Will only START it...
    2010-03-23 15:55:48,968 DEBUG - iceCommandSubmit::try_to_submit() -  TID=[144321160] Going to START CreamJobID [https://cream
    -32.pd.infn.it:8443/CREAM036926381] related to GridJobID [https://devel17.cnaf.infn.it:9000/iM8C3YV12fwhvIG5mNip5Q]...
    2010-03-23 15:55:49,154 INFO - iceLBContext::setLoggingJob - Setting log job to jobid=[https://devel17.cnaf.infn.it:9000/iM8C
    3YV12fwhvIG5mNip5Q] LB server=[devel17.cnaf.infn.it:9000] (port is not used, actually...)
    2010-03-23 15:55:49,155 INFO - iceLBLogger::logEvent() - Cream Transfer OK Event - [gridJobID="https://devel17.cnaf.infn.it:9
    000/iM8C3YV12fwhvIG5mNip5Q" CREAMJobID="https://cream-32.pd.infn.it:8443/CREAM036926381"]

• Bug #57596: [ICE] non resubmission if job failed for proxy expiration FIXED
  • Verify:

    2010-03-23 10:20:37,696 INFO - iceLBLogger::logEvent() - Job Done Failed Event, ExitCode=[0], FailureReason=[Proxy is expired; /opt/glite/bin/glite-lb-logevent: edg_wll_LogEvent*(): LB server (bkserver,lbproxy) store protocol error (edg_wll_LogEvent():  LB server (bkserver,lbproxy) store protocol error;; Logging library ERROR:  LB server (bkserver,lbproxy) store protocol error;; edg_wll_DoLogEvent(): edg_wll_log_connect error Transport endpoint is not connected;; edg_wll_gss_connect();; System Error: Connection refused) /opt/glite/bin/glite-lb-logevent: edg_wll_LogEvent*(): LB server (bkserver,lbproxy) store protocol error (edg_wll_LogEvent():  LB server (bkserver,lbproxy) store protocol error;; Logging library ERROR:  LB server (bkserver,lbproxy) store protocol error;; edg_wll_DoLogEvent(): edg_wll_log_connect error Transport endpoint is not connected;; edg_wll_gss_connect();; System Error: Connection refused) Proxy expired: job killed Terminated Master process killed] - [gridJobID="https://devel17.cnaf.infn.it:9000/jw2aeAy1skHY3mRJHCF8YA" CREAMJobID="https://ce202.cern.ch:8443/CREAM030114428"]
    2010-03-23 10:20:37,817 DEBUG - iceLBContext::testCode() - L&B call succeeded.
    2010-03-23 10:20:37,828 ERROR - Ice::resubmit_job() - Will NOT resubmit job [gridJobID="https://devel17.cnaf.infn.it:9000/jw2aeAy1skHY3mRJHCF8YA" CREAMJobID="https://ce202.cern.ch:8443/CREAM030114428"] because it's Input Sandbox proxy file is not valid: The proxy is EXPIRED!
    2010-03-23 10:20:37,828 INFO - iceLBContext::setLoggingJob - Setting log job to jobid=[https://devel17.cnaf.infn.it:9000/jw2aeAy1skHY3mRJHCF8YA] LB server=[devel17.cnaf.infn.it:9000] (port is not used, actually...)
    2010-03-23 10:20:37,828 INFO - iceLBLogger::logEvent() - Job Aborted Event, reason=[Input sandbox's proxy is missing. Cannot resubmit job] - [gridJobID="https://devel17.cnaf.infn.it:9000/jw2aeAy1skHY3mRJHCF8YA" CREAMJobID="https://ce202.cern.ch:8443/CREAM030114428"]

• Bug #58387: [ICE] should log a job aborted when it cannot resubmit the job for missing user proxy FIXED
  • Verify:

    *************************************************************
    BOOKKEEPING INFORMATION:

• Bug #57295: [ICE] queryDb tool may create empty DB as root FIXED
  • Verify:

    [root@wms007 ~]#  ll /var/glite/ice/persist_dir/ice.db 
    -rw-r--r--  1 glite glite 1280000 Mar 22 17:05 /var/glite/ice/persist_dir/ice.db
    [root@wms007 ~]#  /opt/glite/bin/queryDb -c glite_wms.conf -s RUNNING,REALLY_RUNNING 
    0 item(s) found
    [root@wms007 ~]#  ll /var/glite/ice/persist_dir/ice.db 
    -rw-r--r--  1 glite glite 1280000 Mar 22 17:05 /var/glite/ice/persist_dir/ice.db

• Bug #57579: [ICE] Occasionally the ICE's start/stop script doesn't kill the ICE process HOPEFULLY FIXED
  • Verify:

    [root@wms007 ~]# /opt/glite/etc/init.d/glite-wms-ice status
    /opt/glite/bin/glite-wms-ice-safe (pid 1433) is running...
    [root@wms007 ~]# /opt/glite/etc/init.d/glite-wms-ice stop
    stopping ICE... ok
    [root@wms007 ~]# ps ax | grep ice
    19866 pts/2    S+     0:00 grep ice
    [root@wms007 ~]# /opt/glite/etc/init.d/glite-wms-ice start
    starting ICE... ok
    [root@wms007 ~]# ps ax | grep ice
    19899 ?        Ss     0:00 /opt/glite/bin/glite-wms-ice-safe --conf glite_wms.conf --daemon /var/glite/glite-wms-ice-safe.pid
    19903 ?        S      0:00 sh -c /opt/glite/bin/glite-wms-ice --conf glite_wms.conf > /var/log/glite/ice_console.log 2>&1
    19904 ?        Sl     0:00 /opt/glite/bin/glite-wms-ice --conf glite_wms.conf
    19932 pts/2    S+     0:00 grep ice
    [root@wms007 ~]# /opt/glite/etc/init.d/glite-wms-ice stop
    stopping ICE... ok
    [root@wms007 ~]# ps ax | grep ice
    19978 pts/2    S+     0:00 grep ice
    [root@wms007 ~]# /opt/glite/etc/init.d/glite-wms-ice start
    starting ICE... ok
    [root@wms007 ~]# ps ax | grep ice
    20009 ?        Ss     0:00 /opt/glite/bin/glite-wms-ice-safe --conf glite_wms.conf --daemon /var/glite/glite-wms-ice-safe.pid
    20013 ?        S      0:00 sh -c /opt/glite/bin/glite-wms-ice --conf glite_wms.conf > /var/log/glite/ice_console.log 2>&1
    20014 ?        Sl     0:00 /opt/glite/bin/glite-wms-ice --conf glite_wms.conf
    20046 pts/2    S+     0:00 grep ice

• Bug #53460: [ICE] Detection of job status changes for CREAM jobs should be improved FIXED
  • Using a new CE (1.6) looking in ice's log there is:

    2010-03-22 16:47:50,496 INFO - scoped_timer iceCommandEventQuery::execute() - SOAP Connection for QueryEvent - TID=[150673032] 1269272870.288498 1269272870.496129 0.207631
    2010-03-22 16:47:50,496 DEBUG - iceCommandEventQuery::execute() -  TID=[150673032] There're [2] event(s) for the couple DN [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] CEUrl [https://cream-30.pd.infn.it:8443/ce-cream/services/CREAM2]
    2010-03-22 16:47:50,496 DEBUG - iceCommandEventQuery::execute() -  TID=[150673032] Database  ID=[1261041182000]
    2010-03-22 16:47:50,496 DEBUG - iceCommandEventQuery::execute() -  TID=[150673032] Exec time ID=[3]
    2010-03-22 16:47:50,496 DEBUG - iceCommandEventQuery::processEventsForJob() -  TID=[150673032] Processing [2] event(s) for Job [gridJobID="https://devel17.cnaf.infn.it:9000/uKbQNcbh7kIohBz6bDMNZQ" CREAMJobID="https://cream-30.pd.infn.it:8443/CREAM396193798"] userdn [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] and ce url [https://cream-30.pd.infn.it:8443/ce-cream/services/CREAM2].
    2010-03-22 16:47:50,496 DEBUG - iceCommandEventQuery::processEventsForJob() -  TID=[150673032] EventID [685143] timestsamp [1269272804]
    2010-03-22 16:47:50,496 INFO - scoped_timer iceCommandEventQuery::processSingleEvent - TID=[150673032] InsertStat 1269272870.496682 1269272870.496864 0.000182

  • Using an "old" CE instead the "poller" method is used:

    2010-03-22 16:55:55,397 INFO - scoped_timer iceCommandEventQuery::execute() - SOAP Connection for QueryEvent - TID=[150673032] 1269273355.242918 1269273355.397806 0.154888
    2010-03-22 16:55:55,397 ERROR - iceCommandEventQuery::execute() -  TID=[150673032] Cannot query events for UserDN [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] CEUrl [https://cream-34.pd.infn.it:8443/ce-cream/services/CREAM2]. Exception Internal ex is [Received NULL fault; the error is due to another cause: FaultString=[No such operation 'QueryEventRequest'] - FaultCode=["http://xml.apache.org/axis/":Client] - FaultSubCode=["http://xml.apache.org/axis/":Client] - FaultDetail=[<ns2:hostname>cream-34.pd.infn.it</ns2:hostname>]]
    2010-03-22 16:55:55,398 WARN - iceCommandEventQuery::execute() -  TID=[150673032] Not present QueryEvent on CE [https://cream-34.pd.infn.it:8443/ce-cream/services/CREAM2]. Falling back to old-style StatusPoller.
    2010-03-22 16:55:55,398 INFO - iceCommandStatusPoller::execute() - Getting [100] jobs to poll for user [/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] creamurl [https://cream-34.pd.infn.it:8443/ce-cream/services/CREAM2]
    2010-03-22 16:55:55,398 DEBUG - iceCommandStatusPoller::get_jobs_to_poll() - Collecting jobs to poll for userdn=[/C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle-/dteam/Role=NULL/Capability=NULL] creamurl=[https://cream-34.pd.infn.it:8443/ce-cream/services/CREAM2]. LIMIT set to [100]...

• Bug #55103: [ICE] ICE port 7010 not cleaned up properly FIXED
  • We try a stop/start/restart sequence

    [root@wms007 ~]# ps ax | grep ice
     1283 pts/2    S+     0:00 grep ice
    30985 ?        Ss     0:00 /opt/glite/bin/glite-wms-ice-safe --conf glite_wms.conf --daemon /var/glite/glite-wms-ice-safe.pid
    30989 ?        S      0:00 sh -c /opt/glite/bin/glite-wms-ice --conf glite_wms.conf > /var/log/glite/ice_console.log 2>&1
    30990 ?        Sl     0:15 /opt/glite/bin/glite-wms-ice --conf glite_wms.conf
    [root@wms007 ~]# /opt/glite/etc/init.d/glite-wms-ice stop
    stopping ICE... ok
    [root@wms007 ~]# ps ax | grep ice
     1321 pts/2    S+     0:00 grep ice
    [root@wms007 ~]# /opt/glite/etc/init.d/glite-wms-ice start
    starting ICE... ok
    [root@wms007 ~]# ps ax | grep ice
     1353 ?        Ss     0:00 /opt/glite/bin/glite-wms-ice-safe --conf glite_wms.conf --daemon /var/glite/glite-wms-ice-safe.pid
     1357 ?        S      0:00 sh -c /opt/glite/bin/glite-wms-ice --conf glite_wms.conf > /var/log/glite/ice_console.log 2>&1
     1358 ?        Sl     0:00 /opt/glite/bin/glite-wms-ice --conf glite_wms.conf
     1398 pts/2    S+     0:00 grep ice
    [root@wms007 ~]# /opt/glite/etc/init.d/glite-wms-ice restart
    stopping ICE... ok
    starting ICE... ok
    [root@wms007 ~]# ps ax | grep ice
     1433 ?        Ss     0:00 /opt/glite/bin/glite-wms-ice-safe --conf glite_wms.conf --daemon /var/glite/glite-wms-ice-safe.pid
     1437 ?        S      0:00 sh -c /opt/glite/bin/glite-wms-ice --conf glite_wms.conf > /var/log/glite/ice_console.log 2>&1
     1438 ?        Sl     0:00 /opt/glite/bin/glite-wms-ice --conf glite_wms.conf
     1470 pts/2    S+     0:00 grep ice

• • GridJobID: https://devel17.cnaf.infn.it:9000/dj8r_iFRd8tnWH4bThPNeg

• • • MyProxyServer = "myproxy.cern.ch";
  • GridJobID: https://devel17.cnaf.infn.it:9000/UNB2dHJNn7euaDP3FvJ3og

• • • MyProxyServer = "myproxy.cnaf.infn.it";

• Bug #52934: [ICE] Delegation in ICE doesn't refer to the myproxy server FIXED
  • glite-ce-job-status -L 2 https://cream-30.pd.infn.it:8443/CREAM504042437
    • Deleg Proxy ID = [12692524052E32526wms0072Ecnaf2Einfn2Eit]
    • Destination: cream-30.pd.infn.it:8443/cream-pbs-cream_B
    • Owner = /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle
  • glite-ce-job-status -L 2 https://cream-30.pd.infn.it:8443/CREAM456698107
    • Deleg Proxy ID = [12692523642E948823wms0072Ecnaf2Einfn2Eit]
    • Destination: cream-30.pd.infn.it:8443/cream-pbs-cream_B
    • Owner = /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Alessio Gianelle

• Bug #52934: [ICE] Delegation in ICE doesn't refer to the myproxy server

• Bug #55103: [ICE] ICE port 7010 not cleaned up properly

• Bug #53460: [ICE] Detection of job status changes for CREAM jobs should be improved

• Bug #56636: [ICE] statistics counters for monitoring

• Bug #57295: [ICE] queryDb tool may create empty DB as root

• Bug #57579: [ICE] Occasionally the ICE's start/stop script doesn't kill the ICE process

• Bug #57596: [ICE] non resubmission if job failed for proxy expiration

• Bug #58387: [ICE] should log a job aborted when it cannot resubmit the job for missing user proxy

• Bug #58977: [ICE] Wrong database colum name in ICE SQL query

• Bug #59240: [ICE] abort reasons not always printed in its logfile

• Bug #59399: [ICE] doesn't correctly handle request in jobdir/old when it is restarted

• Bug #59453: [ICE] polling needs to be improved

• Bug #60688: [ICE] does not respect LB server/proxy selection through the LBproxy attribute

• Bug #61312: [ICE] Error in handling user dn in ICE's poller

• Bug #61405: [ICE] Missing proxy validity evaluation in ICE

• Bug #61413: [ICE] should not call EventQuery for a userDN if he/she doesn't have active jobs

• Bug #61748: [ICE] EventQuery/Polling must be done also to blacklisted CE

• Bug #63989: [ICE] doesn't handle exception raised by jobDir::new_entries()

• Bug #48910: Failure starting LM if its output jobdir doesn't exist; unprotected chown in WM/LM/JC startup scripts FIXED
  • stopped gLite services
  • deleted the jobdir under '/var/glite/workload_manager'
  • re-started the LM service checking that the jobdir gets recreated

• • ICE creation
  • JC creation: