Introduction to v2.1 Installation and Configuration

This installation guide is divided as follows:

• Installation of the sensors on the machine to be monitored (WMS/LB/WMSLB)
• Installation of the data_collector server

For the installation and configuration of other WMSMonitor modules (i.e. LoadBalancingArbiter and DBAnalyzer) please refer to the proper twiki session using the left bar.

This version of the server and of the sensor requires SL4 and snmp (installed by the installation scripts) on all the nodes.

Python, http, php and other packages are also needed on the collector machine - they will be installed automatically when the WMSMonitor packages are installed.

If sensors and/or data_collector are already running at your site, instruction on how to upgrade are given highlighted in green.

This version is installed via rpm and yum.

It is advisable, even if not required that the sensors are installed before the collector.

In case of problems during installation please contact wms-support<at>cnaf.infn.it

NOTE: In release 2.0 the database schema has changed! if you are running an update from a version older that 2.0 it will be automatically ported to the new schema by the configuration script.

If you run a version earlier to 2.0 and want to port the old data to the new VO statistics page please contact wms-support<at>cnaf.infn.it after the installation/configuration completes. If you are upgrading from 2.0 VO statistics will be automatically ported.

WMSMON Sensors installation on WMS/LB/WMSLB nodes

(do all the following as root on all your WMS, LB, WMSLB nodes - the procedure is independent on the node type)

• cd /etc/yum.repos.d
• Download this repo file containing the WMSMonitor repository contact string:

wget http://grid-it.cnaf.infn.it/certification/downloads/WMSMonitor.repo

• cd /root
  
• yum clean all
• yum install WMSMonitor.sensors

In release 2.1 the WMSMonitor rpms installs everything under /opt/WMSMonitor (in previous releases it was, by default, under /root/wmsmon).

• cd /opt/WMSMonitor/common

In /opt/WMSMonitor/common are installed all the files needed by all WMSMonitor components. One of this files is the wmsmon_site-info.def, it is now much easier to edit since many default parameters have been moved to the wmsmon_default.conf file.  The defaults file should usually not be changed.

• Edit the wmsmon_site-info.def

The wmsmon_site-info.def contains all the information needed to set up correctly all the monitor components. It is important to edit it carefully using a key = value notation.

Comments can be made using # at the beginning of the line.

Inline comments help you during the editing.

If this is an upgrade please re-edit the file in the new location, now it is much easier to edit. Of course use the same values used in the wmsmon_site-info.def used up to now.

The important variable for sensors installation are:

- WMSMON_HOST = 'Set here the name of the wmsmon data collector host'

- WMSMON_DB_PWD = Chose a root - password for mysql - if already set use it. (Even if not used by sensors please fiill in this field)
- SNMPPASSWD = 'The password you choose for snmp communication between data collector and wms instance !! NOTE: it must be the same on every wmsmon_site-info.def

- SERVER_MYSQL_PASSWORD = 'your_pass' (This is FUNDAMENTAL ON LB NODES)

- LB_PARA_HOST = Host publishing the LB_PARAMETER FILE that initializes lb queries. It is the procol://host:port of the http service running on the collector instances.

!!!(THIS IS FUNDAMENTAL ON EVRY HOST PLEASE READ CAREFULLY WHAT FOLLOWS)!!!

If you follow the instruction below for the server installation without modifying the httpd configuration it s

http://<WMSMMON_HOST>

# Other examples in case of modified httpd conf

#Examples:
# http://host.domain # https://host.domain # https://host.domain:8443

If this parameters are not set an error will be raised a t configuration time (only a warning for the mail address)

When setting a password please read the comments in the wmsmon_site-info.def file to know if the password must be enclosed between ' '

You can ignore the LOAD BALANCING PARAMETERS section if you are not installing a load balancing arbiter. 

• cd /opt/WMSMonitor/sensors/etc
  
• Run the configuration script

./WMSLB_wmsmon_configuration.py

(Alternatively if python is not present on the machine or you experience problems in modules importing you can run the python frozen code in setup_bins - cd setup_bins; ./WMSLB_wmsmon_configuration - )

This script will configure the /etc/snmp/snmpd.conf file in order to accept requests from the data collector. It is important that the snmp password in wmsmon_site-info.def is set correctly.  It willl not destroy lines not belonging to WMSMonitor in that file, in any cse a backup copy will be created in local directory.
If the node is an LB the script will add grants to for wmsmon user on DB lbserver20. It's important that the MYSQL root passwd is set corrrectly in wmsmon_site-info.def file.

The script will not remove old WMSMonitor path /root/wmsmon in order not to destroy user manual changes. That directory can be removed manually without creating problems to the sensors.

If at the end of the script you see the string: "WMSMONITOR SENSORS SUCCESSFULLY CONFIGURED!" it means that everything went fine .

In case of problems running the script please contact wms-support<at>cnaf.infn.it .

WMSMON Data Collector installation

(done all the following as root)

• Install an SL4 machine (Not to be done if this is an upgrade)

• If it is an upgrade it is safer to create a backup copy of the wmsmon database using an utility such as mysqldump (i.e. mysqldump -p wmsmon > wmsmon-dump.sql)
• IMPORTANT: If you manually changed files of the web interface under /var/www/html/wmsmon/ backup them because that directory will be overwritten.
  

• cd /etc/yum.repos.d
• Download this repo file containing the WMSMonitor repository contact string:

wget http://grid-it.cnaf.infn.it/certification/downloads/WMSMonitor.repo

• cd /root
  
• yum clean all
• yum install WMSMonitor.collector

In release 2.1 the WMSMonitor rpms installs everything under /opt/WMSMonitor (in previous releases it was, by default, under /root/wmsmon).

• cd /opt/WMSMonitor/common

In /opt/WMSMonitor/common are installed all the files needed by all WMSMonitor components. One of this files is the wmsmon_site-info.def, it is now much easier to edit since many default parameters have been moved to the wmsmon_default.conf file.  The defaults file should usually not be changed.

• Edit the wmsmon_site-info.def

The wmsmon_site-info.def contains all the information needed to set up correctly all the monitor components. It is important to edit it carefully using a key = value notation.

Comments can be made using # at the beginning of the line.

Inline comments help you during the editing.

If this is an upgrade please re-edit the file in the new location, now it is much easier to edit. Of course use the same values used in the wmsmon_site-info.def used up to now.

The important variable for a collector installation are:

- WMSMON_HOST = 'Set here the name of the wmsmon data collector host'

- WMSMON_DB_PWD = Chose a root - password for mysql - if already set use it.
- LEMONFLAG = 1 'If you do not have a lemon tool on the wms/lb instance set to 0'
- LEMONURL = 'Set to the machine lemon url if any'
- SNMPPASSWD = 'The password you choose for snmp communication between data collector and wms instance !! NOTE: it must be the same on every wmsmon_site-info.def

- WMSMON_SEVER_CONTACT_EMAIL = MAIL CONTACT that will appear in web pages error messages. Users will be invited to send email to this address.

- LB_PARA_HOST = Host publishing the LB_PARAMETER FILE that initializes lb queries. It is the procol://host:port of the http service running on the collector instances.

If you follow the instruction below for the server installation without modifying the httpd configuration it s

http://<WMSMMON_HOST>

# Other examples in case of modified httpd conf

#Examples:
# http://host.domain # https://host.domain # https://host.domain:8443

If this parameters are not set an error will be raised a t configuration time (only a warning for the mail address)

When setting a password please read the comments in the wmsmon_site-info.def file to know if the password must be enclosed between ' '

You can ignore the LOAD BALANCING PARAMETERS section if you are not installing a load balancing arbiter.

• cd /opt/WMSMonitor/collector/etc
• Edit the wmslist.conf

If this is an upgrade you can maintain the same wmslist.conf file used by the previous release, but note that it is no more called wmsmonlist.con (R1.4) but wmslist.conf.

This file contains the list of WMS present in your cluster (those were the wmsmon sensors are or will be installed).
It must be edited as shown in the template file in the following way:

wms1.your_domain lb1.your_domain vo1

....

wmsn.your_domain lbn.your_domain von

You should insert the wms/lb pairs monitored and a vo served by the pair. The vo indication will be used only to group the wms in the wmsmon web pages, and not for the job counting per vo.
If a pair serve more than one VO you can choose a word like multi or multiVO. You can also use this tag to group the wms by their role: PROD, DEVEL etc...

IMPORTANT: THE FILE CANNOT BE COMMENTED!

• Check that MySQL server is running (service mysqld status). If not, please start it (service mysqld start)
  
• Run the configuration script
  

./wmsmon_server_setup.py  

The script will look for older wmsmon installations. If none is found a brand new installation will start otherwise an upgrade process will be attempted.

The old cron files that trigger the collector execution are moved to /tmp directory and new crons will be installed.

The old $INSTALL_PATH/wmsmon directory (by default /root/wmsmon) will not be removed or moved in order not to destroy manual changes - if you want you can remove manually or leave untouched, it will not cause any problems.

Errors will be reported, please pay attention to them.

When the script completes the /opt/WMSMonitor directory should have been moved to the INSTALL_PATH defined and the /var/www/html/wmsmon directory should now be in /var/www/html/WEBDIR, where WEBDIR is defined in the wmsmon_default.conf file.

Two crons should be present in /etc/cron.d: wmsmon.cron and wmsmon_logrotate.conf.

The wmsmon.cron is the cron that launches the main data collector script. By default it is launched every 15 mins, if you want to change this frequency you should change the cron file by hand, and in this case in order to optimize performance you are advised to change also the STEPDATE value in the wmsmon_default.conf file to 2x(cron frequency).

By default the data collector script logs to the /var/log/WMSMONITOR.log file, but you can configure this on the default.def file. By default the wmsmon_logrotate cron creates 10 files 100MB big, to modify this parameter you should change by hand the /etc/wmsmon_logrotate.cron file.

The standard output of the data collector script is maintained in the /var/log/data_collector_main.log for debugging reasons, this file is rotated as the main log file, please keep it.

Look for errors. Configuration is complete only if a string like "WMSMonitor Successfully Configured" is displayed at the end.

In case of problems please contact wms-support<at>cnaf.infn.it

Post installation STEPS

• Increased php available memory

In order to increase php performance it is advisable that you increase the php allocable memory.
Modify the /etc/php.ini to have the folloving line:
memory_limit = 56M

• Optional http port change

The WMSMON web configuration does not modify the default port (80) used by httpd. To modify the port number edit /etc/httpd/conf/httpd.conf and edit the line:

Listen <port_number>

Then restart httpd ( service httpd restart)

• Optional secure http enabled

The WMSMON server needs a valid host certificate stored in a HOST_CERTIFICATE_DIR (i.e. /etc/grid-security)
Install mod_ssl package:
-Run: yum install mod_ssl

Install the accepted ca packages, i.e. you can execute the following:
- Create the /etc/yum.repos.d/lcg-ca.repo file containing:

    [CA]

    name=CAs

    baseurl=http://linuxsoft.cern.ch/LCG-CAs/current

    protect=1

- Run : yum install lcg_CA

Edit the /etc/httpd/conf/httpd.con f and add the following inside the <Directory /var/www/html> section the following lines :

         SSLRequireSSL

         SSLVerifyClient      require

         SSLVerifyDepth 10

Edit the /etc/httpd/conf.d/ssl.conf and:

- set the SSLCertificateFile variable to HOST_CERTIFICATE_DIR/hostcert.pem and comment any other line that set this variable.

- set the SSLCertificateKeyFile variable to HOST_CERTIFICATE_DIR/hostkey.pem and comment any other line that set this variable. _ _

- set the SSLCertificatePath variable to the name of the directory containing the CA file (i.e. /etc/grid-security/certificates if you installed the lcg_CA metapackage) and comment any other line that set this variable.

If you want to change the default https port (443) you should change in the /etc/httpd/conf.d/ssl.conf file the line:
Listen <port_number> (i.e. Listen 8443)

_Oprtional redirect - If you want to automatically redirect http requests to https pages you should add to the /etc/httpd/conf/httpd.conf file the following section (using the proper values for the variables_ SERVER_HOST_IP,SERVER_HOST_NAME and YOUR_DOMAIN):

<VirtualHost SERVER_HOST_IP:80>

   DocumentRoot /var/www/html

   ServerName SERVER_HOST_NAME.YOUR_DOMAIN

   RedirectMatch (/.*)$ https://SERVER_HOST_NAME.YOUR_DOMAIN/$1

</VirtualHost>

Restart httpd

• Unlock protected pages to specific certificate DNs

WMSMONitor reports a section with users activity on each WMS. Name and surname of each user is reported on some pages and these pages for privacy reasons are not exposed to all certificates

If you want to unlock those pages to some certificate DN you should enable https protocol as descrbed in the previous paragraph(Optional secure http enabled) and do the following:

- Edit the

/var/www/html/wmsmon/common/config.php file and modify the last line adding the list of DNs in the line

$config->dnEnabledList=array('DN1','DN2'....'DNn');

If you want to unlock the pages with sensible data to everyone (using either http or https) you have to change the value of the $config->protectedPage variable in /var/www/html/WEBDIR/common/config.php file (WEBDIR is defined in the site-info.def file) and set it to 0:

$config->protectedPages=0;

• Enable high port range communication for inter cluster monitoring

WMSMON uses the snmp standard port (in general the 183) for sensor-collector communications. If a firewall blocks the snmp port it is possible to enable the communication on a not-standard port. This can be useful when the data collector and the WMS cluster are not in the same computing centre.
If you are not in this case you can skip this section.
NOTE: this feature is not well tested, please report any problem and bug found to wms-support<at>cnaf.infn.it
To enable high port support you should modify the wmslist.conf file adding a fourth column indicating which is the port number to be used on that particular wms/lb pair:

wms1.your_domain lb1.your_domain vo1 port1

NOTE: it is not possible to specify 2 different port numbers for WMS and LB.

On WMS/LB sensor side you should set snmp in order to listen for request on the port you choose. This is accomplished by adding in the /etc/snmp.conf file the following line:

agentaddress <port_number>

and restarting snmp (service snmpd restart)

• Configuring WMS load balancing Arbiter

To configure the Arbiter parameters in the site_info.def file refer to:
https://twiki.cnaf.infn.it/cgi-bin/twiki/view/WMSMonitor/WMSLoadBalancingArbiter

The WMS instances load balancing arbiter is a python script wms_balancing_arbiter.py which must be executed periodically in a cron. It should be executed in cascade to the data_collector script in order to take advantage of latest available metric collected by the WMSMonitor from each WMS instance.
Note that also an utility is provided, external_wmsmon_metrics_collector.py, to allow the inclusion of WMS instances monitored by other WMSMonitor servers in the list of defined aliases. This is useful for ROC level definition of some WMS aliases pointing to WMS instances geographycally distributed and momitored by different WMSMonitor instances. To make this work you must edit the external_wmsmon_metrics_collector.py script directly adding the WMSMonitor urls of instances to monitor as in the example below:

urllist=['https://wmsmon.ct.infn.it:8443/wmsmon/details/details.php?wms=prod-wms-01.ct.infn.it', 'https://wmsmon.ct.infn.it:8443/wmsmon/details/details.php?wms=prod-wms-02.ct.infn.it']
and add its execution in the load balancing cron, which is something like:

5,20,35,50 * * * * root /PATH/external_wmsmon_metrics_collector.py; /PATH/wms_balancing_arbiter.py >> /var/www/html/loadbalancing_test.log 2>&1

Note that /var/www/html/loadbalancing_test.log file readable through the web displays the WMS IP list in the alias for all defined aliases.

It is advisable that you configure a logrotate for the arbiter log file. This is possiblle ading the following lines to the /etc/wmsmon_logrotate.conf file:

/var/www/html/loadbalancing_test.log {

        copytruncate
        rotate 100
        size = 1M
        missingok
        nomail

}