Difference: WebDownload3 (1 vs. 20)

Revision 202012-10-30 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 213 to 213
 showimgright="/twiki/pub/TWiki/TWikiDocGraphics/toggleopen.gif" hideimgright="/twiki/pub/TWiki/TWikiDocGraphics/toggleclose.gif" }%
Added:
>
>

WARNING: make sure you add valid strings in mysql, with no spaces before/after string !
 
  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:
Line: 234 to 235
 showimgright="/twiki/pub/TWiki/TWikiDocGraphics/toggleopen.gif" hideimgright="/twiki/pub/TWiki/TWikiDocGraphics/toggleclose.gif" }%
Changed:
<
<
>
>

WARNING: make sure you add valid strings in mysql, with no spaces before/after string !
 
  1. Set the dns contact in /opt/WMSMonitor/common/wmsmon_site-info.def
  2. Set the passwd for dns contact in /opt/WMSMonitor/common/wmsmon_site-info.def
  3. Define aliases in the dns contact
Line: 260 to 261
 hideimgright="/twiki/pub/TWiki/TWikiDocGraphics/toggleclose.gif" }%
Changed:
<
<
Due to privacy reasons statistics about WMS/LB service usage per USER are not public. Therefore access to this section is allowed to a configurable set of personal certificates identified by a subject DN. Authentication process exploits user personal certificate installed in the web browser.
>
>
Due to privacy reasons statistics about WMS/LB service usage per USER are not public. Therefore access to this section is allowed to a configurable set of personal certificates identified by a subject DN. Authentication process exploits user personal certificate installed in the web browser.
WARNING: make sure you add valid strings in mysql, with no spaces before/after string !
 
  1. To grant access to a given user:
    1. Enter Mysql database wmsmon and for each user (identified by a DN) run:
    2. ADD DN: mysql$> insert into admin_web_users(dn, privileges) values('/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Danilo Nicola Dongiovanni','read');

Revision 192012-07-31 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 7 to 7
 Notice that this new version can be installed on emiWMS/LB only on SL5/64 platform.

Here some of the novelties introduced with this new version:

Added:
>
>
  • New architecture
 
  • New web Interface with separate frames for WMS and LB instances
  • Data collection is now asynchronous, in fact a cron job executes sensors on monitored WMS/LB instances and send data to WMSMonitor server consumer
  • Messaging is now based on ActiveMQ
Line: 16 to 17
 
  • globus JSS error statistics available

Added:
>
>

WMSMonitor 3.0 ARCHITECTURE

architecture.png

 

WMSMonitor SENSORS INSTALLATION ON INSTANCES TO MONITOR

Line: 266 to 270
  </>
<--/twistyPlugin-->
Added:
>
>
 -- DaniloDongiovanni - 2011-12-20
Added:
>
>

META FILEATTACHMENT attachment="architecture.png" attr="" comment="" date="1343742957" name="architecture.png" path="architecture.png" size="635864" user="DaniloDongiovanni" version="1"

Revision 182012-02-02 - AndreaCristofori

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 116 to 116
 
  1. $> wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/egi-trustanchors.repo -O /etc/yum.repos.d/egi-trustanchors.repo
  2. $> yum install ca-policy-egi-core
Changed:
<
<
Edit the /etc/httpd/conf/httpd.conf and add the following inside the <Directory /var/www/html> section the following lines :

>
>
Edit the /etc/httpd/conf/httpd.conf and add the following at the end of the configuration (please modify according to your hostname where you find HOST.NAME.SOMETHING):

<Directory "/var/www/html">

    Options FollowSymLinks

    AllowOverride None

    Allow from all
  SSLRequireSSL
Deleted:
<
<

  SSLVerifyClient require
Deleted:
<
<

  SSLVerifyDepth 10
Changed:
<
<
>
>
 
Deleted:
<
<
Edit the /etc/httpd/conf.d/ssl.conf and:
 
Changed:
<
<
- set the SSLCertificateFile variable to HOST_CERTIFICATE_DIR/hostcert.pem and comment any other line that set this variable.
>
>
ServerName HOST.NAME.SOMETHING DocumentRoot /var/www/html/wmsmon RewriteEngine On RewriteCond %{HTTPS} =on # RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 <https://%{server_name}/$1>[L,R,NC] RewriteRule ^/(.*) https://HOST.NAME.SOMETHING/$1
 
Deleted:
<
<
- set the SSLCertificateKeyFile variable to HOST_CERTIFICATE_DIR/hostkey.pem and comment any other line that set this variable. _ _
 
Deleted:
<
<
- set the SSLCACertificatePath variable to the name of the directory containing the CA file (i.e. /etc/grid-security/certificates if you installed the lcg_CA metapackage) and comment any other line that set this variable.
 
Changed:
<
<
\If you want to change the default https port (443) you should change in the /etc/httpd/conf.d/ssl.conf file the line:
Listen <port_number> (i.e. Listen 8443)
>
>
Edit the /etc/httpd/conf.d/ssl.conf and:
 
Changed:
<
<
_Oprtional redirect - If you want to automatically redirect http requests to https pages you should add to the /etc/httpd/conf/httpd.conf file the following section (using the proper values for the variables_ SERVER_HOST_IP,SERVER_HOST_NAME and YOUR_DOMAIN):
<VirtualHost SERVER_HOST_IP:80>
   DocumentRoot /var/www/html
   ServerName SERVER_HOST_NAME.YOUR_DOMAIN
   RedirectMatch (/.*)$ https://SERVER_HOST_NAME.YOUR_DOMAIN/$1
</VirtualHost>

>
>
- set the SSLCertificateFile variable to HOST_CERTIFICATE_DIR/hostcert.pem and comment any other line that set this variable.
 
Changed:
<
<
>
>
- set the SSLCertificateKeyFile variable to HOST_CERTIFICATE_DIR/hostkey.pem and comment any other line that set this variable.
 
Changed:
<
<
  1. $> service httpd restart
>
>
- set the SSLCACertificatePath variable to the name of the directory containing the CA file (i.e. /etc/grid-security/certificates if you installed the ca-policy-egi-core metapackage) and comment any other line that set this variable.

  1. $> restart Apache
      service httpd restart
 
Deleted:
<
<
  </>
<--/twistyPlugin-->

Revision 172012-01-25 - AndreaCristofori

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 220 to 220
 
      1. EX>>> call insertHostLabels('wms014.cnaf.infn.it','WMS','cms','MCproduction','production', 1,'cnaf');
    1. mysql$>call insertHostLabels(<LB-HOSTANME-WITH-DOMAIN>, 'LB', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1 (1 monitored and shown in the web page, 0 not monitored and hided from the web page), <HOST-SITENAME>);
      1. EX>>> call insertHostLabels('wms014.cnaf.infn.it','LB','cms','MCproduction','production', 1,'cnaf'); ---> NOTE do this also when wms/lb is cohosted
Deleted:
<
<
    1. mysql$> call insertLBHost();
      1. EX>>> call insertLBHost('lb013.cnaf.infn.it'); ---> NOTE do this only for LB is cohosted
 
    1. shell$> python /opt/WMSMonitor/collector/bin/data_collector_daemon.py stop
    2. shell$> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start

Revision 162012-01-20 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 19 to 19
 

WMSMonitor SENSORS INSTALLATION ON INSTANCES TO MONITOR

Added:
>
>
<--/twistyPlugin twikiMakeVisibleInline-->
 To be part of the preview, please write to wms-support asking for the tarball. Then follow instruction:
  1. Make sure you have your emi WMS / LB instance installed and you have root privileges on it
Line: 34 to 45
 
  1. WMSMonitor "user mapping" functionality requires wmproxy loglevel set a 6 in glite_wms.conf file
  2. Required package lsof.x86_64 installed on WMS/LB instance

Changed:
<
<

WMSMonitor SERVER (Not Available for Download YET: WORK IN PROGRESS)

>
>
<--/twistyPlugin-->

WMSMonitor SERVER Installation & Configuration (Not Available for Download YET: WORK IN PROGRESS)

 

WMSMonitor Server Instance Installation

Added:
>
>
<--/twistyPlugin twikiMakeVisibleInline-->
 Requirements:
  1. Make sure you have your emi WMS / LB instances to monitor installed with new sensors as described in previous paragraph.
  2. By default the exploited ActiveMQ broker is the one provided by CERN for development purposes. If you use the default broker/topic you can ignore this step. To use your own ActiveMQ server you need to install one and setup a proper topic on it. This HOWTO does not cover the issue.
Line: 66 to 87
 
  1. $> tar -xvzf WMSMonitor_server_3.XX.tgz (--> untar the tarball we provided you with)
  2. $> cp /opt/WMSMonitor/deployment/*cron /etc/cron.d/
Added:
>
>
<--/twistyPlugin-->
 

WMSMonitor Server Instance Configuration

Changed:
<
<

Web Access Configuration:

>
>
<--/twistyPlugin twikiMakeVisibleInline-->
 
Changed:
<
<
Increased php available memory
>
>
Increased php available memory
  In order to increase php performance it is advisable that you increase the php allocable memory.
Modify the /etc/php.ini to have the folloving line (at least 128) :
memory_limit = 128M

Changed:
<
<
Secure http enabled (required for enabling user stats web access)
>
>
Secure http enabled (required for enabling user stats web access)
  The WMSMON server needs a valid host certificate stored in a HOST_CERTIFICATE_DIR (i.e. /etc/grid-security)
Line: 133 to 163
 
Changed:
<
<

Database Initialization:

>
>
<--/twistyPlugin-->

<--/twistyPlugin twikiMakeVisibleInline-->
 
  1. ENTER mysql console as root
    1. mysql> CREATE USER 'wmsmon'@'localhost' IDENTIFIED BY 'mypass';
Line: 146 to 185
 
  1. Back on shell import the sql dump file provided
    1. $> mysql -u root -p wmsmon < /opt/WMSMonitor/deployment/wmsmon3.0_dumpfile.sql
Changed:
<
<

Sitedef editing:

>
>
<--/twistyPlugin-->

<--/twistyPlugin twikiMakeVisibleInline-->
 
  1. Edit file /opt/WMSMonitor/common/wmsmon_site-info.def filling following values:
    1. WMSMON_HOST = YOURSERVERWMSMonitor
    2. WMSMON_DB_PWD = yourwmsmon user passwd from above (eg. 'mypass') ----> NOTE: do not change USER!
Changed:
<
<
    1. WMSMON_SEVER_CONTACT_EMAIL = siteemail@domain
>
>
    1. WMSMON_WEB_DB_PWD = yourwebwmsmon user passwd (eg. 'myotherpass' ) ----> NOTE: do not change USER!
    2. WMSMON_SERVER_CONTACT_EMAIL = siteemail@domain

<--/twistyPlugin-->
 
Deleted:
<
<

Monitoring Instance Cluster Configuration:

 
Changed:
<
<
Adding instances to monitor to WMSMonitor server
>
>
<--/twistyPlugin twikiMakeVisibleInline-->
 
  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:
    2. mysql$>call insertHostLabels(<WMS-HOSTANME-WITH-DOMAIN>, 'WMS', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1 (1 monitored and shown in the web page, 0 not monitored and hided from the web page), <HOST-SITENAME>);
Line: 166 to 225
 
    1. shell$> python /opt/WMSMonitor/collector/bin/data_collector_daemon.py stop
    2. shell$> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start
Changed:
<
<
Configuring Load Balancing in WMSMonitor server
>
>
<--/twistyPlugin-->

<--/twistyPlugin twikiMakeVisibleInline-->
 
  1. Set the dns contact in /opt/WMSMonitor/common/wmsmon_site-info.def
  2. Set the passwd for dns contact in /opt/WMSMonitor/common/wmsmon_site-info.def
  3. Define aliases in the dns contact
Line: 181 to 251
 
      1. If needed: REMOVE HOST FROM ALIAS mysql$> call remove_wmshost_from_ALIAS( "hostnamewms", "alias_name")
Added:
>
>
<--/twistyPlugin-->

<--/twistyPlugin twikiMakeVisibleInline-->

Due to privacy reasons statistics about WMS/LB service usage per USER are not public. Therefore access to this section is allowed to a configurable set of personal certificates identified by a subject DN. Authentication process exploits user personal certificate installed in the web browser.

  1. To grant access to a given user:
    1. Enter Mysql database wmsmon and for each user (identified by a DN) run:
    2. ADD DN: mysql$> insert into admin_web_users(dn, privileges) values('/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Danilo Nicola Dongiovanni','read');
  2. To revoke access privileges from a given user:
    1. Enter Mysql database wmsmon and for each user (identified by a DN) run:
    2. DELETE DN: mysql$> delete from admin_web_users where dn = '/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=Danilo Nicola Dongiovanni' ;

<--/twistyPlugin-->
  -- DaniloDongiovanni - 2011-12-20

Revision 152012-01-19 - AndreaCristofori

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 173 to 173
 
    1. For each given ALIAS and WMS instance associated:
      1. Enter Mysql database wmsmon
      2. ADD ALIAS: mysql$> call insert_loadbalancing_ALIAS(enable_flag #is the alias in use#, numout #number of instances to leave out of alias per period#, subtest_enable #IS a submission test to be used in status metric#, alias_name #without domain#) (to be executed only the first time a new alias is created);
Changed:
<
<
        1. EX>>> call insert_loadbalancing_ALIAS( 1, 2, 0, 'wms-prod');
>
>
        1. EX>>> call insert_loadbalancing_ALIAS( 1, 2, 0, 'wms-cms-prod');
 
      1. If needed: REMOVE ALIAS: mysql$> call remove_loadbalancing_ALIAS("alias_name")
        1. EX>>> call remove_loadbalancing_ALIAS( 'wms-prod'); Note that removing an alias will remove also wms association to that alias
      2. ADD HOST TO ALIAS mysql$> call add_wmshost_to_ALIAS(WMS_HOSTNAME_WITH_DOMAIN, ALIAS_NAME_WITHOUT_DOMAIN, SPARE_LABEL #1 if wms associated but not used#, test_urlIN #URL of the submission test used if enabled#);
Changed:
<
<
        1. EX>>> call add_wmshost_to_ALIAS('wms002.cnaf.infn.it', 'wms-prod', 0, '');
>
>
        1. EX>>> call add_wmshost_to_ALIAS('wms002.cnaf.infn.it', 'wms-cms-prod', 0, '');
 
      1. If needed: REMOVE HOST FROM ALIAS mysql$> call remove_wmshost_from_ALIAS( "hostnamewms", "alias_name")

Revision 142012-01-19 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 136 to 136
 

Database Initialization:

  1. ENTER mysql console as root
Changed:
<
<
    1. mysql> CREATE USER 'wmsmon'@'%' IDENTIFIED BY 'mypass';
>
>
    1. mysql> CREATE USER 'wmsmon'@'localhost' IDENTIFIED BY 'mypass';
 
    1. mysql> CREATE database wmsmon;
Changed:
<
<
    1. mysql> grant ALL on wmsmon.* TO 'wmsmon'@'%';
>
>
    1. mysql> grant ALL on wmsmon.* TO 'wmsmon'@'localhost';
    2. mysql> CREATE USER 'webwmsmon'@'localhost' IDENTIFIED BY 'myotherpass';
    3. mysql> grant SELECT on wmsmon.* TO 'webwmsmon'@'localhost';
 
    1. mysql> flush privileges;
    2. mysql> exit;
  1. Back on shell import the sql dump file provided

Revision 132012-01-19 - AndreaCristofori

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 176 to 176
 
        1. EX>>> call remove_loadbalancing_ALIAS( 'wms-prod'); Note that removing an alias will remove also wms association to that alias
      1. ADD HOST TO ALIAS mysql$> call add_wmshost_to_ALIAS(WMS_HOSTNAME_WITH_DOMAIN, ALIAS_NAME_WITHOUT_DOMAIN, SPARE_LABEL #1 if wms associated but not used#, test_urlIN #URL of the submission test used if enabled#);
        1. EX>>> call add_wmshost_to_ALIAS('wms002.cnaf.infn.it', 'wms-prod', 0, '');
Changed:
<
<
      1. If needed: REMOVE HOST FROM ALIAS mysql$> remove_wmshost_from_ALIAS( "hostnamewms", "alias_name")
>
>
      1. If needed: REMOVE HOST FROM ALIAS mysql$> call remove_wmshost_from_ALIAS( "hostnamewms", "alias_name")
 

Revision 122012-01-16 - AndreaCristofori

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 9 to 9
 Here some of the novelties introduced with this new version:
  • New web Interface with separate frames for WMS and LB instances
  • Data collection is now asynchronous, in fact a cron job executes sensors on monitored WMS/LB instances and send data to WMSMonitor server consumer
Changed:
<
<
  • Messaging is now based on Active MQ
>
>
  • Messaging is now based on ActiveMQ
 
  • WMS job workload management rates are now derived exploiting LB api
  • Handling of several LB instances for each WMS
  • both globus and emi-cream JSS case is now handled
Line: 40 to 40
  Requirements:
  1. Make sure you have your emi WMS / LB instances to monitor installed with new sensors as described in previous paragraph.
Changed:
<
<
  1. By default the exploited ActiveMQ broker is the one provided by CERN for development purposes. If you use the default broker/topic you can ignore this step. To use your own ActiveMQ server you need to install one and setup a proper topic on it. This HOWTO does not cover the issue.
>
>
  1. By default the exploited ActiveMQ broker is the one provided by CERN for development purposes. If you use the default broker/topic you can ignore this step. To use your own ActiveMQ server you need to install one and setup a proper topic on it. This HOWTO does not cover the issue.
 
  1. A fresh installation of SL5/x64. Suggested HW: Virtual Machine with 3GB RAM, 30GB disk, 1 Core
  2. If a certificate based web access restriction is needed, an x509 host certificate is required
  3. Install php package:
Line: 56 to 56
 
    1. $> chkconfig httpd on
    2. $> service httpd start
  1. Install Mysql python library:
Changed:
<
<
    1. $ yum install MySQL-python
>
>
    1. $ yum install MySQL-python
  1. Install Mysql php library:
    1. $ yum install php-mysql
  Then follow instruction:
  1. $> cd /opt

Revision 112012-01-13 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 55 to 55
 
    1. $> yum install httpd
    2. $> chkconfig httpd on
    3. $> service httpd start
Added:
>
>
  1. Install Mysql python library:
    1. $ yum install MySQL-python
  Then follow instruction:
  1. $> cd /opt

Revision 102012-01-13 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 167 to 167
 
      1. Enter Mysql database wmsmon
      2. ADD ALIAS: mysql$> call insert_loadbalancing_ALIAS(enable_flag #is the alias in use#, numout #number of instances to leave out of alias per period#, subtest_enable #IS a submission test to be used in status metric#, alias_name #without domain#) (to be executed only the first time a new alias is created);
        1. EX>>> call insert_loadbalancing_ALIAS( 1, 2, 0, 'wms-prod');
Added:
>
>
      1. If needed: REMOVE ALIAS: mysql$> call remove_loadbalancing_ALIAS("alias_name")
        1. EX>>> call remove_loadbalancing_ALIAS( 'wms-prod'); Note that removing an alias will remove also wms association to that alias
 
      1. ADD HOST TO ALIAS mysql$> call add_wmshost_to_ALIAS(WMS_HOSTNAME_WITH_DOMAIN, ALIAS_NAME_WITHOUT_DOMAIN, SPARE_LABEL #1 if wms associated but not used#, test_urlIN #URL of the submission test used if enabled#);
        1. EX>>> call add_wmshost_to_ALIAS('wms002.cnaf.infn.it', 'wms-prod', 0, '');
Added:
>
>
      1. If needed: REMOVE HOST FROM ALIAS mysql$> remove_wmshost_from_ALIAS( "hostnamewms", "alias_name")

 -- DaniloDongiovanni - 2011-12-20

Revision 92012-01-13 - AndreaCristofori

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 81 to 81
 
  1. $> wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/egi-trustanchors.repo -O /etc/yum.repos.d/egi-trustanchors.repo
  2. $> yum install ca-policy-egi-core
Changed:
<
<
Edit the /etc/httpd/conf/httpd.con f and add the following inside the <Directory /var/www/html> section the following lines :
>
>
Edit the /etc/httpd/conf/httpd.conf and add the following inside the <Directory /var/www/html> section the following lines :
 
         SSLRequireSSL
Line: 160 to 160
 
    1. shell$> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start

Configuring Load Balancing in WMSMonitor server
Changed:
<
<
  1. Set the dns contact in site-info.def
  2. Set the passwd for dns contact in site-info.def
>
>
  1. Set the dns contact in /opt/WMSMonitor/common/wmsmon_site-info.def
  2. Set the passwd for dns contact in /opt/WMSMonitor/common/wmsmon_site-info.def
 
  1. Define aliases in the dns contact
    1. For each given ALIAS and WMS instance associated:
      1. Enter Mysql database wmsmon
Changed:
<
<
      1. ADD ALIAS: mysql$> call insert_loadbalancing_ALIAS(enable_flag #is the alias in use#, numout #number of instances to leave out of alias per period#, subtest_enable #IS a submission test to be used in status metric#, alias_name #without domain#);
      2. ADD HOST TO ALIAS mysql$> call add_wmshost_to_ALIAS(WMS_HOSTNAME_WITH_DOMAIN, ALIAS_NAME_WITHOUT_DOMAIN, SPARE_LABEL #1 if wms associated but not used#);
>
>
      1. ADD ALIAS: mysql$> call insert_loadbalancing_ALIAS(enable_flag #is the alias in use#, numout #number of instances to leave out of alias per period#, subtest_enable #IS a submission test to be used in status metric#, alias_name #without domain#) (to be executed only the first time a new alias is created);
        1. EX>>> call insert_loadbalancing_ALIAS( 1, 2, 0, 'wms-prod');
      2. ADD HOST TO ALIAS mysql$> call add_wmshost_to_ALIAS(WMS_HOSTNAME_WITH_DOMAIN, ALIAS_NAME_WITHOUT_DOMAIN, SPARE_LABEL #1 if wms associated but not used#, test_urlIN #URL of the submission test used if enabled#);
        1. EX>>> call add_wmshost_to_ALIAS('wms002.cnaf.infn.it', 'wms-prod', 0, '');
 -- DaniloDongiovanni - 2011-12-20 \ No newline at end of file

Revision 82012-01-12 - AndreaCristofori

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 150 to 150
 
Adding instances to monitor to WMSMonitor server
  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:
Changed:
<
<
    1. mysql$>call insertHostLabels(<WMS-HOSTANME-WITH-DOMAIN>, 'WMS', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
>
>
    1. mysql$>call insertHostLabels(<WMS-HOSTANME-WITH-DOMAIN>, 'WMS', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1 (1 monitored and shown in the web page, 0 not monitored and hided from the web page), <HOST-SITENAME>);
 
      1. EX>>> call insertHostLabels('wms014.cnaf.infn.it','WMS','cms','MCproduction','production', 1,'cnaf');
Changed:
<
<
    1. mysql$>call insertHostLabels(<LB-HOSTANME-WITH-DOMAIN>, 'LB', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
>
>
    1. mysql$>call insertHostLabels(<LB-HOSTANME-WITH-DOMAIN>, 'LB', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1 (1 monitored and shown in the web page, 0 not monitored and hided from the web page), <HOST-SITENAME>);
 
      1. EX>>> call insertHostLabels('wms014.cnaf.infn.it','LB','cms','MCproduction','production', 1,'cnaf'); ---> NOTE do this also when wms/lb is cohosted
    1. mysql$> call insertLBHost();
      1. EX>>> call insertLBHost('lb013.cnaf.infn.it'); ---> NOTE do this only for LB is cohosted

Revision 72012-01-11 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 42 to 42
 
  1. Make sure you have your emi WMS / LB instances to monitor installed with new sensors as described in previous paragraph.
  2. By default the exploited ActiveMQ broker is the one provided by CERN for development purposes. If you use the default broker/topic you can ignore this step. To use your own ActiveMQ server you need to install one and setup a proper topic on it. This HOWTO does not cover the issue.
  3. A fresh installation of SL5/x64. Suggested HW: Virtual Machine with 3GB RAM, 30GB disk, 1 Core
Added:
>
>
  1. If a certificate based web access restriction is needed, an x509 host certificate is required
 
  1. Install php package:
    1. $> yum install php
  2. Install mod_ssl package:
Line: 55 to 56
 
    1. $> chkconfig httpd on
    2. $> service httpd start
Deleted:
<
<
 Then follow instruction:
  1. $> cd /opt
Changed:
<
<
  1. $> tar -xvzf WMSMonitor-server.tgz (--> untar the tarball we provided you with)
  2. $> cp /opt/WMSMonitor/sensors/bin/WMSMonitor.cron /etc/cron.d/
>
>
  1. $> tar -xvzf WMSMonitor_server_3.XX.tgz (--> untar the tarball we provided you with)
  2. $> cp /opt/WMSMonitor/deployment/*cron /etc/cron.d/
 

WMSMonitor Server Instance Configuration

Web Access Configuration:

Line: 68 to 68
 
  • Increased php available memory
Changed:
<
<
In order to increase php performance it is advisable that you increase the php allocable memory.
Modify the /etc/php.ini to have the folloving line:
memory_limit = 56M
>
>
In order to increase php performance it is advisable that you increase the php allocable memory.
Modify the /etc/php.ini to have the folloving line (at least 128) :
memory_limit = 128M
 
Line: 77 to 77
  The WMSMON server needs a valid host certificate stored in a HOST_CERTIFICATE_DIR (i.e. /etc/grid-security)
Changed:
<
<
Install the accepted ca packages, i.e. you can execute the following:
- Create the /etc/yum.repos.d/lcg-ca.repo file containing:
    [CA]
    name=CAs
    baseurl=http://linuxsoft.cern.ch/LCG-CAs/current
    protect=1

- Run : yum install lcg_CA

>
>
Install the accepted ca packages, i.e. you can execute the following:
  1. $> wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/egi-trustanchors.repo -O /etc/yum.repos.d/egi-trustanchors.repo
  2. $> yum install ca-policy-egi-core
  Edit the /etc/httpd/conf/httpd.con f and add the following inside the <Directory /var/www/html> section the following lines :

Line: 112 to 100
  - set the SSLCACertificatePath variable to the name of the directory containing the CA file (i.e. /etc/grid-security/certificates if you installed the lcg_CA metapackage) and comment any other line that set this variable.
Changed:
<
<
If you want to change the default https port (443) you should change in the /etc/httpd/conf.d/ssl.conf file the line:
Listen <port_number> (i.e. Listen 8443)
>
>
\If you want to change the default https port (443) you should change in the /etc/httpd/conf.d/ssl.conf file the line:
Listen <port_number> (i.e. Listen 8443)
  _Oprtional redirect - If you want to automatically redirect http requests to https pages you should add to the
Line: 136 to 124
 
Changed:
<
<
Restart httpd
>
>
  1. $> service httpd restart
 
Added:
>
>

Database Initialization:

  1. ENTER mysql console as root
    1. mysql> CREATE USER 'wmsmon'@'%' IDENTIFIED BY 'mypass';
    2. mysql> CREATE database wmsmon;
    3. mysql> grant ALL on wmsmon.* TO 'wmsmon'@'%';
    4. mysql> flush privileges;
    5. mysql> exit;
  2. Back on shell import the sql dump file provided
    1. $> mysql -u root -p wmsmon < /opt/WMSMonitor/deployment/wmsmon3.0_dumpfile.sql

Sitedef editing:

  1. Edit file /opt/WMSMonitor/common/wmsmon_site-info.def filling following values:
    1. WMSMON_HOST = YOURSERVERWMSMonitor
    2. WMSMON_DB_PWD = yourwmsmon user passwd from above (eg. 'mypass') ----> NOTE: do not change USER!
    3. WMSMON_SEVER_CONTACT_EMAIL = siteemail@domain
 

Monitoring Instance Cluster Configuration:

Adding instances to monitor to WMSMonitor server
  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:
    2. mysql$>call insertHostLabels(<WMS-HOSTANME-WITH-DOMAIN>, 'WMS', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
Added:
>
>
      1. EX>>> call insertHostLabels('wms014.cnaf.infn.it','WMS','cms','MCproduction','production', 1,'cnaf');
 
    1. mysql$>call insertHostLabels(<LB-HOSTANME-WITH-DOMAIN>, 'LB', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
Added:
>
>
      1. EX>>> call insertHostLabels('wms014.cnaf.infn.it','LB','cms','MCproduction','production', 1,'cnaf'); ---> NOTE do this also when wms/lb is cohosted
 
    1. mysql$> call insertLBHost();
Added:
>
>
      1. EX>>> call insertLBHost('lb013.cnaf.infn.it'); ---> NOTE do this only for LB is cohosted
 
    1. shell$> python /opt/WMSMonitor/collector/bin/data_collector_daemon.py stop
    2. shell$> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start

Revision 62012-01-11 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 42 to 42
 
  1. Make sure you have your emi WMS / LB instances to monitor installed with new sensors as described in previous paragraph.
  2. By default the exploited ActiveMQ broker is the one provided by CERN for development purposes. If you use the default broker/topic you can ignore this step. To use your own ActiveMQ server you need to install one and setup a proper topic on it. This HOWTO does not cover the issue.
  3. A fresh installation of SL5/x64. Suggested HW: Virtual Machine with 3GB RAM, 30GB disk, 1 Core
Added:
>
>
  1. Install php package:
    1. $> yum install php
  2. Install mod_ssl package:
    1. $> yum install mod_ssl
 
  1. Install mysql server package:
    1. $> yum install mysql-server
    2. $> chkconfig mysqld on
Line: 57 to 63
 
  1. $> cp /opt/WMSMonitor/sensors/bin/WMSMonitor.cron /etc/cron.d/

WMSMonitor Server Instance Configuration

Changed:
<
<
  1. Apache configuration: ..........
>
>

Web Access Configuration:

  • Increased php available memory
In order to increase php performance it is advisable that you increase the php allocable memory.
Modify the /etc/php.ini to have the folloving line:
memory_limit = 56M

  • Secure http enabled (required for enabling user stats web access)
The WMSMON server needs a valid host certificate stored in a HOST_CERTIFICATE_DIR (i.e. /etc/grid-security)

Install the accepted ca packages, i.e. you can execute the following:
- Create the /etc/yum.repos.d/lcg-ca.repo file containing:

    [CA]
    name=CAs
    baseurl=http://linuxsoft.cern.ch/LCG-CAs/current
    protect=1

- Run : yum install lcg_CA

Edit the /etc/httpd/conf/httpd.con f and add the following inside the <Directory /var/www/html> section the following lines :

         SSLRequireSSL
         SSLVerifyClient      require
         SSLVerifyDepth 10

Edit the /etc/httpd/conf.d/ssl.conf and:

- set the SSLCertificateFile variable to HOST_CERTIFICATE_DIR/hostcert.pem and comment any other line that set this variable.

- set the SSLCertificateKeyFile variable to HOST_CERTIFICATE_DIR/hostkey.pem and comment any other line that set this variable. _ _

- set the SSLCACertificatePath variable to the name of the directory containing the CA file (i.e. /etc/grid-security/certificates if you installed the lcg_CA metapackage) and comment any other line that set this variable.

If you want to change the default https port (443) you should change in the /etc/httpd/conf.d/ssl.conf file the line:
Listen <port_number> (i.e. Listen 8443)

_Oprtional redirect - If you want to automatically redirect http requests to https pages you should add to the /etc/httpd/conf/httpd.conf file the following section (using the proper values for the variables_ SERVER_HOST_IP,SERVER_HOST_NAME and YOUR_DOMAIN):

<VirtualHost SERVER_HOST_IP:80>
   DocumentRoot /var/www/html
   ServerName SERVER_HOST_NAME.YOUR_DOMAIN
   RedirectMatch (/.*)$ https://SERVER_HOST_NAME.YOUR_DOMAIN/$1
</VirtualHost>

Restart httpd

  • Monitoring Instance Cluster Configuration:

 
Changed:
<
<

Adding instances to monitor to WMSMonitor server

>
>
Adding instances to monitor to WMSMonitor server
 
  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:
    2. mysql$>call insertHostLabels(<WMS-HOSTANME-WITH-DOMAIN>, 'WMS', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
Line: 68 to 150
 
    1. shell$> python /opt/WMSMonitor/collector/bin/data_collector_daemon.py stop
    2. shell$> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start
Changed:
<
<

Configuring Load Balancing in WMSMonitor server

>
>
Configuring Load Balancing in WMSMonitor server
 
  1. Set the dns contact in site-info.def
  2. Set the passwd for dns contact in site-info.def
  3. Define aliases in the dns contact

Revision 52012-01-11 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 17 to 17
 
Changed:
<
<

SENSORS INSTALLATION ON INSTANCES TO MONITOR

>
>

WMSMonitor SENSORS INSTALLATION ON INSTANCES TO MONITOR

  To be part of the preview, please write to wms-support asking for the tarball. Then follow instruction:
Line: 36 to 36
 

WMSMonitor SERVER (Not Available for Download YET: WORK IN PROGRESS)

Added:
>
>

WMSMonitor Server Instance Installation

Requirements:

  1. Make sure you have your emi WMS / LB instances to monitor installed with new sensors as described in previous paragraph.
  2. By default the exploited ActiveMQ broker is the one provided by CERN for development purposes. If you use the default broker/topic you can ignore this step. To use your own ActiveMQ server you need to install one and setup a proper topic on it. This HOWTO does not cover the issue.
  3. A fresh installation of SL5/x64. Suggested HW: Virtual Machine with 3GB RAM, 30GB disk, 1 Core
  4. Install mysql server package:
    1. $> yum install mysql-server
    2. $> chkconfig mysqld on
    3. $> service mysqld start
  5. Install apache server package:
    1. $> yum install httpd
    2. $> chkconfig httpd on
    3. $> service httpd start

Then follow instruction:

  1. $> cd /opt
  2. $> tar -xvzf WMSMonitor-server.tgz (--> untar the tarball we provided you with)
  3. $> cp /opt/WMSMonitor/sensors/bin/WMSMonitor.cron /etc/cron.d/

WMSMonitor Server Instance Configuration

  1. Apache configuration: ..........
 

Adding instances to monitor to WMSMonitor server

  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:

Revision 42011-12-21 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 30 to 30
 
  1. Wait till first cron job to start and check whether a file like /tmp/WMSLOCKFILE_1312453801.45 exists
  2. go to INFN-CNAF WMSMonitor 3.0 monitor server with a valid user certificate and flash player in your browser and enjoy...
Changed:
<
<
Note: WMSMonitor "user mapping" functionality requires wmproxy loglevel set a 6 in glite_wms.conf file
>
>
Note:
  1. WMSMonitor "user mapping" functionality requires wmproxy loglevel set a 6 in glite_wms.conf file
  2. Required package lsof.x86_64 installed on WMS/LB instance
 

WMSMonitor SERVER (Not Available for Download YET: WORK IN PROGRESS)

Revision 32011-12-20 - DaniloDongiovanni

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 37 to 37
 

Adding instances to monitor to WMSMonitor server

  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:
Changed:
<
<
    1. call insertHostLabels(<WMS-HOSTANME-WITH-DOMAIN>, 'WMS', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
    2. call insertHostLabels(<LB-HOSTANME-WITH-DOMAIN>, 'LB', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
    3. $> call insertLBHost();
    4. $> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py stop
    5. $> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start
>
>
    1. mysql$>call insertHostLabels(<WMS-HOSTANME-WITH-DOMAIN>, 'WMS', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
    2. mysql$>call insertHostLabels(<LB-HOSTANME-WITH-DOMAIN>, 'LB', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
    3. mysql$> call insertLBHost();
    4. shell$> python /opt/WMSMonitor/collector/bin/data_collector_daemon.py stop
    5. shell$> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start
 

Configuring Load Balancing in WMSMonitor server

Changed:
<
<
>
>
  1. Set the dns contact in site-info.def
  2. Set the passwd for dns contact in site-info.def
  3. Define aliases in the dns contact
    1. For each given ALIAS and WMS instance associated:
      1. Enter Mysql database wmsmon
      2. ADD ALIAS: mysql$> call insert_loadbalancing_ALIAS(enable_flag #is the alias in use#, numout #number of instances to leave out of alias per period#, subtest_enable #IS a submission test to be used in status metric#, alias_name #without domain#);
      3. ADD HOST TO ALIAS mysql$> call add_wmshost_to_ALIAS(WMS_HOSTNAME_WITH_DOMAIN, ALIAS_NAME_WITHOUT_DOMAIN, SPARE_LABEL #1 if wms associated but not used#);
  -- DaniloDongiovanni - 2011-12-20

Revision 22011-12-20 - EnricoFattibene

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

Line: 37 to 37
 

Adding instances to monitor to WMSMonitor server

  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:
Changed:
<
<
    1. $> call insertHost('LBHOSTNAME_WITH_DOMAIN');
    2. $> call insertHost('WMSHOSTNAME_WITH_DOMAIN');
    3. $> call insertLBHost('LBHOSTNAME_WITH_DOMAIN');
    4. $> insert into admin_host_labels(idhost,service,vo,vo_group,service_usage,active,host_owner) values ('TOGET_FROM_hosts_TABLE','WMS','VO->MULTI','','GROUP_USAGE(ex. PRODUCTION)','1','SITE_NAME');
    5. $> insert into admin_host_labels(idhost,service,vo,vo_group,service_usage,active,host_owner) values('TOGET_FROM_hosts_TABLE','LB','VO->MULTI','','GROUP_USAGE(ex. PRODUCTION)','1','SITE_NAME');
>
>
    1. call insertHostLabels(<WMS-HOSTANME-WITH-DOMAIN>, 'WMS', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
    2. call insertHostLabels(<LB-HOSTANME-WITH-DOMAIN>, 'LB', <VO(for multi-vo services set 'multi')>, <VO_GROUP(ex. 'analysis','mc_production',etc.)>, <SERVICE_USAGE(ex. 'production','test',etc.)>, 1, <HOST-SITENAME>);
    3. $> call insertLBHost();
 
    1. $> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py stop
    2. $> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start

Revision 12011-12-20 - DaniloDongiovanni

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WebHome"

NEW Installation for emi-WMS/LB, WMSMonitor 3.0

A completely new version of WMSMonitor is made available, for Pre-View testing.
For the moment being we only deploy sensors in a tarball, and you can use the WMSMonitor server Instance available at INFN-CNAF to visualize your instance data.
Notice that this new version can be installed on emiWMS/LB only on SL5/64 platform.

Here some of the novelties introduced with this new version:

  • New web Interface with separate frames for WMS and LB instances
  • Data collection is now asynchronous, in fact a cron job executes sensors on monitored WMS/LB instances and send data to WMSMonitor server consumer
  • Messaging is now based on Active MQ
  • WMS job workload management rates are now derived exploiting LB api
  • Handling of several LB instances for each WMS
  • both globus and emi-cream JSS case is now handled
  • globus JSS error statistics available

SENSORS INSTALLATION ON INSTANCES TO MONITOR

To be part of the preview, please write to wms-support asking for the tarball. Then follow instruction:

  1. Make sure you have your emi WMS / LB instance installed and you have root privileges on it
  2. $> cd /opt
  3. $> tar -xvzf WMSMonitor.tgz (--> untar the tarball we provided you with)
  4. $> cp /opt/WMSMonitor/sensors/bin/WMSMonitor.cron /etc/cron.d/
  5. Edit /etc/cron.d/WMSMonitor.cron uncommenting the WMS or LB line respectively (depending on the service hosted in your instance)
  6. IMPORTANT --> tell us the hostname of instance to be monitored
  7. Wait till first cron job to start and check whether a file like /tmp/WMSLOCKFILE_1312453801.45 exists
  8. go to INFN-CNAF WMSMonitor 3.0 monitor server with a valid user certificate and flash player in your browser and enjoy...

Note: WMSMonitor "user mapping" functionality requires wmproxy loglevel set a 6 in glite_wms.conf file

WMSMonitor SERVER (Not Available for Download YET: WORK IN PROGRESS)

Adding instances to monitor to WMSMonitor server

  1. Given an LB and a WMS instance to monitor:
    1. Enter Mysql database wmsmon:
    2. $> call insertHost('LBHOSTNAME_WITH_DOMAIN');
    3. $> call insertHost('WMSHOSTNAME_WITH_DOMAIN');
    4. $> call insertLBHost('LBHOSTNAME_WITH_DOMAIN');
    5. $> insert into admin_host_labels(idhost,service,vo,vo_group,service_usage,active,host_owner) values ('TOGET_FROM_hosts_TABLE','WMS','VO->MULTI','','GROUP_USAGE(ex. PRODUCTION)','1','SITE_NAME');
    6. $> insert into admin_host_labels(idhost,service,vo,vo_group,service_usage,active,host_owner) values('TOGET_FROM_hosts_TABLE','LB','VO->MULTI','','GROUP_USAGE(ex. PRODUCTION)','1','SITE_NAME');
    7. $> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py stop
    8. $> FROM SHELL: python /opt/WMSMonitor/collector/bin/data_collector_daemon.py start

Configuring Load Balancing in WMSMonitor server

-- DaniloDongiovanni - 2011-12-20

 
TWIKI.NET
This site is powered by the TWiki collaboration platformCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback