• wmproxy.log Used in case of authentication or submission error

• RuntimeMalloc: in the WM section, allows to use an alternative malloc library (i.e. nedmalloc, google performance tools and many more), run-time redirecting with LD_PRELOAD. Possible values are, for example, RuntimeMalloc? = "/usr/lib/libtcmalloc_minimal.so" if you use Google malloc.

• ${GLITE_LOCATION_VAR}/sandboxdir: it is where the job sandboxes are located, they are automatically purged when a get-output of a job is done. If a VO does not take care of getting the job output back from the WMS node SB dir can become a serious problem for the HD occupancy. Another situation in which the SB dir can become problematic is when a job (or a certain number of jobs) has a huge output sandbox and the control on the OSB is not enabled on the glite_wms.conf file (see WMProxy configuration parameter section). In any case it is a good habit to purge periodically the Sandbox Dir.
• ${GLITE_LOCATION_VAR}/workloadmanager/: it is where the input file list or the jobdir are located depending on the value of 'DispatcherType' attribute in the 'glite_wms.conf' file
• ${GLITE_LOCATION_VAR}/logmonitor/: this directory contains mainly condor log files
• ${GLITE_LOCATION_VAR}/jocontrol/

• ${GLITE_LOCATION_VAR}/.drain: this file is used to put the WMS in draining mode, so that it does not accept new submission request but allows any other operation as the output retrieval.

Cron Jobs

• /etc/cron.d/glite-wms-purger.cron: it periodically purges the sandbox dirs performing a check on the job status
• glite-wms-wmproxy-purge-proxycache.cron: expired proxies are purged by this cron job
• fetch-crl: cron job that retrieves the CRLs periodically

https://twiki.cnaf.infn.it/cgi-bin/twiki/view/EgeeJra1It/WMSConfFile

• ${GLITE_LOCATION}/etc/glite_wms_wmproxy.gacl: this file is used for authorization purposes in order to check for requesting user rights.
• ${GLITE_LOCATION}/etc/glite_wms_wmproxy_httpd.conf: this file is a WMProxy specific configuration file configuring the HTTP daemon and Fast CGI
• {GLITE_LOCATION}/etc/wmproxy_logrotate.conf: this file configures the logrotate tool, that performs the rotation of httpd-wmproxy-access and httpd-wmproxy-errors HTTPD daemon log files.

The Workload Management System Admin Guide

Introduction

Service Overview

• Workload Management – WM: Core component of the Workload Management, its purpose is to accept and satisfy requests for job management coming from its clients
• WMProxy: Web service interface to submit jobs to the WM
• Job Controller – JC: Acts as an interface to condor for the WM
• Log Monitor – LM: Directly connected to JC acts as a job monitoring tool parsing condor log files
• Local Logger: copy events to be sent to the LB server into a local disk file
• LBProxy: keeps a local view of the job state to be sent to the LB server
• Proxy Renewal: Service to renew the proxy of a long-running job
• ICE: (Interface to CREAM Environment) is the WMS service dealing when interacting with CREAM based CEs.

Installation and configuration

Hardware Requirements

• 4 GB RAM is the minimum suggested for memory
• quad-core processor is recommended to better handle parallel matchmaking and all the different sub-services running on a WMS
• Min disk space: depends on load and type of jobs submitted.
  • under '${GLITE_LOCATION_VAR}/sandboxdir' 30-40GB is the min on several wms used in production with the cron job to purge job sandboxes once a week enabled in order to accomodate submitted job sandbox dirs.

Install & Configure

• Install and configure OS and basic services according to the https://twiki.cern.ch/twiki/bin/view/LCG/GenericInstallGuide310
• Install the glite-WMS metapackage from the appropriate gLite software repository
• Configure the WMS node by running '/opt/glite/yaim/bin/yaim -c -s site-info.def -n WMS' Following a list of the WMS specific variables that can be set in the 'site-info.def' file:
  • $WMS_HOST -> the WMS hostname, ex. : 'egee-rb-01.$MY_DOMAIN'
  • $PX_HOST -> the hostname of a server myproxy, ex.: 'myproxy.$MY_DOMAIN'
  • $BDII_HOST -> the hostname of the site bdii to be used, ex: 'sitebdii.$MY_DOMAIN'
  • $LB_HOST -> the hostname of the LB server to be used, ex: 'lb-server.$MY_DOMAIN:9000' This variable is set as a service specific variable in the file services/glite-wms, located one directory below the one where the 'site-info.def' file is located

Daemons and services running

• glite-lb-locallogger:

   glite-lb-logd running
         glite-lb-interlogd running

• glite-lb-proxy:

   glite-lb-proxy running as 4137

• glite-proxy-renewald:

   glite-proxy-renewd running

• globus-gridftp:

   globus-gridftp-server (pid 3107) is running...

• glite-wms-jc:

   JobController running in pid: 10008
         CondorG master running in pid: 10063 10062
         CondorG schedd running in pid: 10070

• glite-wms-lm:

   Logmonitor running...

• glite-wms-wm:

   /opt/glite/bin/glite-wms-workload_manager (pid 9957) is running...

• glite-wms-wmproxy:

  WMProxy httpd listening on port 7443
  httpd (pid 22223 22222 22221 22220 22219 22218 22217) is running ....
  ===
  WMProxy Server running instances:
  UID        PID  PPID  C STIME TTY          TIME CMD

• glite-wms-ice:

  /opt/glite/bin/glite-wms-ice-safe (pid 10103) is running...

File Systems/Directories

Log files locations

• wmproxy.log Used in case of authentication or submisison error

• workload_manager_events.log Used to check the status of the matchmaking process (from Waiting to Ready status) and the query to the information system to fill in the InformationSuperMarket

• ice.log used to check jobs that matched a CREAM based CE and are sent to it via ICE

• jobcontoller_events.log Used to check the jobs events once arrived on condor

• httpd-wmproxy-errors.log Used in case of problems in contacting the WMProxy service

• httpd-wmproxy-access.log
• logmonitor_events.log Aggregate information about each job coming from various log files

• glite-wms-wmproxy-purge-proxycache.log
• lcmaps.log Used when there are problems in the mapping of remote users to local pool accounts

Other log files that can be useful in case of trouble are the condor log in:

• /var/local/condor/log/
• /var/glite/logmonitor/CondorG.log/

Select only specific VO resources

Troubleshooting

WMS Monitoring

• '/opt/glite/etc/glite-lb-index.conf' LB indexes configuration file. In order to set different indexes, modify the glite-lb-index.conf configuration file and then run '$/opt/glite/bin/glite-lb-bkindex -r /opt/glite/etc/glite-lb-index.conf', where -r mean really perform reindexing

• '/opt/glite/etc/lcmaps/lcmaps.db' and '/op/glite/etc/lcmaps/lcmaps.db.gridftp' are the lcmaps configuration files for wmproxy and gridftp respectively. Currently after running yaim configuration script, they have to be synchronized as explained in bug https://savannah.cern.ch/bugs/?35244 It will be fixed in the near future

Configuration files

• '/opt/glite/etc/glite_wms.conf' is divided in sub-sections, one common and one for each grid component the WMS is composed of, for a general description of this file and its configuration parameters, see here: https://twiki.cnaf.infn.it/cgi-bin/twiki/view/EgeeJra1It/WMS_configuration_file
• 'glite_wms_wmproxy.gacl' and 'glite_wms_wmproxy_httpd.conf' are the wmproxy configuration files, fora general description see here: http://trinity.datamat.it/projects/EGEE/wiki/wiki.php?n=GliteWmsWmproxyHttpd.Conf http://trinity.datamat.it/projects/EGEE/wiki/wiki.php?n=GliteWmsWmproxy.Gacl
• '/opt/glite/etc/lcmaps/lcmaps.db' and '/op/glite/etc/lcmaps/lcmaps.db.gridftp' are the lcmaps configuration files for wmproxy and gridftp respectively

Update to a more recent version/patch

• Set up the patch repo, like for example: http://grid-deployment.web.cern.ch/grid-deployment/glite/cert/3.1/patches/<patch#>/sl4/i386/
• Run 'yum install <patch#-glite-WMS>'

Select only specific VO resources

(|(objectclass=gluecesebind)(objectclass=gluecluster)(objectclass=gluesubcluster)(objectclass=gluevoview)(objectclass=gluece))

The idea is to supply system administrators with the possibility to specify an additional ldap clause which will be added in logical AND to the latest two clauses of the default filter in order to match gluece/gluevoview objectclasses specific attributes. To such an aim the configuration file supply users with:

• IsmIILDAPCEFilterExt for handling the additional search filter while purchasing information about CE from the BDII

As an example, by specifying the following:

IsmIILDAPCEFilterExt = "(|(GlueCEAccessControlBaseRule=VO:cms)(GlueCEAccessControlBaseRule=VOMS:/cms/*))"

the search filter during the purchasing would be:

(|(objectclass=gluecesebind)(objectclass=gluecluster)(objectclass=gluesubcluster) (&(|(objectclass=gluevoview)(objectclass=gluece)) (|(GlueCEAccessControlBaseRule=VO:cms)(GlueCEAccessControlBaseRule=VOMS:/cms/*))))

and thus the WMS would select only resources (i.e. CE/Views) belonging to CMS.

The Workload Management System Guide

Installation Guide

• Set up the CA repository. Get it from: http://grid-deployment.web.cern.ch/grid-deployment/yaim/repos/lcg-CA.repo
• Set up the latest certification repository file for the WMS under '/etc/yum.repos.d' Get it from here https://grid-deployment.web.cern.ch/grid-deployment/certification/repos/glite-WMS.repo
• Run 'yum update' , 'yum install cert-glite-WMS' and 'yum install lcg-CA'
• Run '/opt/glite/yaim/bin/yaim -c -s site-info.def -n WMS'

Configuration Guide

Tuning of some configuration parameters

https://twiki.cnaf.infn.it/cgi-bin/twiki/view/EgeeJra1It/WMS_configuration_file

• II_timeout: default by yaim is set to 30, increase it for low-memory machines. 4 GB is the minimum suggested for memory
• MatchRetryPeriod: once a job becomes pending, meaning that there are no resources available, this parameter represents the period between successive match-making attempts, in seconds. The default value is '1000', in order to decrease the number of periodic retries of unmatched jobs the value of this parameter should be increased. A suggested value (used on several production wms) is several hours, '14400'

Service Monitoring Guide

-- ElisabettaMolinari - 11 Apr 2008