You are here: TWiki> WMSMonitor Web>WebDownload3 (2012-10-30, DaniloDongiovanni)

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

A completely new version of WMSMonitor is made available, for Pre-View testing.
For the moment being we only deploy sensors in a tarball, and you can use the WMSMonitor server Instance available at INFN-CNAF to visualize your instance data.
Notice that this new version can be installed on emiWMS/LB only on SL5/64 platform.

Here some of the novelties introduced with this new version:

  • New architecture
  • New web Interface with separate frames for WMS and LB instances
  • Data collection is now asynchronous, in fact a cron job executes sensors on monitored WMS/LB instances and send data to WMSMonitor server consumer
  • Messaging is now based on ActiveMQ
  • WMS job workload management rates are now derived exploiting LB api
  • Handling of several LB instances for each WMS
  • both globus and emi-cream JSS case is now handled
  • globus JSS error statistics available

WMSMonitor 3.0 ARCHITECTURE

architecture.png

WMSMonitor SENSORS INSTALLATION ON INSTANCES TO MONITOR

To be part of the preview, please write to wms-support asking for the tarball. Then follow instruction:

  1. Make sure you have your emi WMS / LB instance installed and you have root privileges on it
  2. $> cd /opt
  3. $> tar -xvzf WMSMonitor.tgz (--> untar the tarball we provided you with)
  4. $> cp /opt/WMSMonitor/sensors/bin/WMSMonitor.cron /etc/cron.d/
  5. Edit /etc/cron.d/WMSMonitor.cron uncommenting the WMS or LB line respectively (depending on the service hosted in your instance)
  6. IMPORTANT --> tell us the hostname of instance to be monitored
  7. Wait till first cron job to start and check whether a file like /tmp/WMSLOCKFILE_1312453801.45 exists
  8. go to INFN-CNAF WMSMonitor 3.0 monitor server with a valid user certificate and flash player in your browser and enjoy...

Note:

  1. WMSMonitor "user mapping" functionality requires wmproxy loglevel set a 6 in glite_wms.conf file
  2. Required package lsof.x86_64 installed on WMS/LB instance

WMSMonitor SERVER Installation & Configuration (Not Available for Download YET: WORK IN PROGRESS)

WMSMonitor Server Instance Installation

Requirements:

  1. Make sure you have your emi WMS / LB instances to monitor installed with new sensors as described in previous paragraph.
  2. By default the exploited ActiveMQ broker is the one provided by CERN for development purposes. If you use the default broker/topic you can ignore this step. To use your own ActiveMQ server you need to install one and setup a proper topic on it. This HOWTO does not cover the issue.
  3. A fresh installation of SL5/x64. Suggested HW: Virtual Machine with 3GB RAM, 30GB disk, 1 Core
  4. If a certificate based web access restriction is needed, an x509 host certificate is required
  5. Install php package:
    1. $> yum install php
  6. Install mod_ssl package:
    1. $> yum install mod_ssl
  7. Install mysql server package:
    1. $> yum install mysql-server
    2. $> chkconfig mysqld on
    3. $> service mysqld start
  8. Install apache server package:
    1. $> yum install httpd
    2. $> chkconfig httpd on
    3. $> service httpd start
  9. Install Mysql python library:
    1. $ yum install MySQL-python
  10. Install Mysql php library:
    1. $ yum install php-mysql

Then follow instruction:

  1. $> cd /opt
  2. $> tar -xvzf WMSMonitor_server_3.XX.tgz (--> untar the tarball we provided you with)
  3. $> cp /opt/WMSMonitor/deployment/*cron /etc/cron.d/

WMSMonitor Server Instance Configuration

  • Increased php available memory
In order to increase php performance it is advisable that you increase the php allocable memory.
Modify the /etc/php.ini to have the folloving line (at least 128) :
memory_limit = 128M

  • Secure http enabled (required for enabling user stats web access)
The WMSMON server needs a valid host certificate stored in a HOST_CERTIFICATE_DIR (i.e. /etc/grid-security)

Install the accepted ca packages, i.e. you can execute the following:

  1. $> wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/egi-trustanchors.repo -O /etc/yum.repos.d/egi-trustanchors.repo
  2. $> yum install ca-policy-egi-core

Edit the /etc/httpd/conf/httpd.conf and add the following at the end of the configuration (please modify according to your hostname where you find HOST.NAME.SOMETHING):

<Directory "/var/www/html">

    Options FollowSymLinks

    AllowOverride None

    Allow from all

    SSLRequireSSL
    SSLVerifyClient      require
    SSLVerifyDepth 10
</Directory>


<VirtualHost *:80>
        ServerName HOST.NAME.SOMETHING
        DocumentRoot /var/www/html/wmsmon
        RewriteEngine On
        RewriteCond %{HTTPS} !=on
#       RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 <https://%{server_name}/$1>[L,R,NC]
        RewriteRule ^/(.*) https://HOST.NAME.SOMETHING/$1
</VirtualHost>

Edit the /etc/httpd/conf.d/ssl.conf and:

- set the SSLCertificateFile variable to HOST_CERTIFICATE_DIR/hostcert.pem and comment any other line that set this variable.

- set the SSLCertificateKeyFile variable to HOST_CERTIFICATE_DIR/hostkey.pem and comment any other line that set this variable.

- set the SSLCACertificatePath variable to the name of the directory containing the CA file (i.e. /etc/grid-security/certificates if you installed the ca-policy-egi-core metapackage) and comment any other line that set this variable.

  1. $> restart Apache
      service httpd restart

  1. ENTER mysql console as root
    1. mysql> CREATE USER 'wmsmon'@'localhost' IDENTIFIED BY 'mypass';
    2. mysql> CREATE database wmsmon;
    3. mysql> grant ALL on wmsmon.* TO 'wmsmon'@'localhost';
    4. mysql> CREATE USER 'webwmsmon'@'localhost' IDENTIFIED BY 'myotherpass';
    5. mysql> grant SELECT on wmsmon.* TO 'webwmsmon'@'localhost';
    6. mysql> flush privileges;
    7. mysql> exit;
  2. Back on shell import the sql dump file provided
    1. $> mysql -u root -p wmsmon < /opt/WMSMonitor/deployment/wmsmon3.0_dumpfile.sql

  1. Edit file /opt/WMSMonitor/common/wmsmon_site-info.def filling following values:
    1. WMSMON_HOST = YOURSERVERWMSMonitor
    2. WMSMON_DB_PWD = yourwmsmon user passwd from above (eg. 'mypass') ----> NOTE: do not change USER!
    3. WMSMON_WEB_DB_PWD = yourwebwmsmon user passwd (eg. 'myotherpass' ) ----> NOTE: do not change USER!
    4. WMSMON_SERVER_CONTACT_EMAIL = siteemail@domain


WARNING: make sure you add valid strings in mysql, with no spaces before/after string !

  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:
    2. mysql$>call insertHostLabels(<WMS-HOSTANME-WITH-DOMAIN>, 'WMS', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1 (1 monitored and shown in the web page, 0 not monitored and hided from the web page), <HOST-SITENAME>);
      1. EX>>> call insertHostLabels('wms014.cnaf.infn.it','WMS','cms','MCproduction','production', 1,'cnaf');
    3. mysql$>call insertHostLabels(<LB-HOSTANME-WITH-DOMAIN>, 'LB', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1 (1 monitored and shown in the web page, 0 not monitored and hided from the web page), <HOST-SITENAME>);
      1. EX>>> call insertHostLabels('wms014.cnaf.infn.it','LB','cms','MCproduction','production', 1,'cnaf'); ---> NOTE do this also when wms/lb is cohosted
    4. shell$> python /opt/WMSMonitor/collector/bin/data_collector_daemon.py stop
    5. shell$> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start


WARNING: make sure you add valid strings in mysql, with no spaces before/after string !
  1. Set the dns contact in /opt/WMSMonitor/common/wmsmon_site-info.def
  2. Set the passwd for dns contact in /opt/WMSMonitor/common/wmsmon_site-info.def
  3. Define aliases in the dns contact
    1. For each given ALIAS and WMS instance associated:
      1. Enter Mysql database wmsmon
      2. ADD ALIAS: mysql$> call insert_loadbalancing_ALIAS(enable_flag #is the alias in use#, numout #number of instances to leave out of alias per period#, subtest_enable #IS a submission test to be used in status metric#, alias_name #without domain#) (to be executed only the first time a new alias is created);
        1. EX>>> call insert_loadbalancing_ALIAS( 1, 2, 0, 'wms-cms-prod');
      3. If needed: REMOVE ALIAS: mysql$> call remove_loadbalancing_ALIAS("alias_name")
        1. EX>>> call remove_loadbalancing_ALIAS( 'wms-prod'); Note that removing an alias will remove also wms association to that alias
      4. ADD HOST TO ALIAS mysql$> call add_wmshost_to_ALIAS(WMS_HOSTNAME_WITH_DOMAIN, ALIAS_NAME_WITHOUT_DOMAIN, SPARE_LABEL #1 if wms associated but not used#, test_urlIN #URL of the submission test used if enabled#);
        1. EX>>> call add_wmshost_to_ALIAS('wms002.cnaf.infn.it', 'wms-cms-prod', 0, '');
      5. If needed: REMOVE HOST FROM ALIAS mysql$> call remove_wmshost_from_ALIAS( "hostnamewms", "alias_name")

Due to privacy reasons statistics about WMS/LB service usage per USER are not public. Therefore access to this section is allowed to a configurable set of personal certificates identified by a subject DN. Authentication process exploits user personal certificate installed in the web browser.
WARNING: make sure you add valid strings in mysql, with no spaces before/after string !

  1. To grant access to a given user:
    1. Enter Mysql database wmsmon and for each user (identified by a DN) run:
    2. ADD DN: mysql$> insert into admin_web_users(dn, privileges) values('/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Danilo Nicola Dongiovanni','read');
  2. To revoke access privileges from a given user:
    1. Enter Mysql database wmsmon and for each user (identified by a DN) run:
    2. DELETE DN: mysql$> delete from admin_web_users where dn = '/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Danilo Nicola Dongiovanni' ;

-- DaniloDongiovanni - 2011-12-20

Topic attachments
I Attachment Action Size Date Who Comment
PNGpng architecture.png manage 621.0 K 2012-07-31 - 13:55 DaniloDongiovanni  
Topic revision: r20 - 2012-10-30 - DaniloDongiovanni
 
TWIKI.NET
This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback