(r2) NotesAboutCreamWithoutTorqueWithMPI-EMI-2SL6 < SiteAdminCorner

Tags: , view all tags
---+!! Notes about Installation and Configuration of a CREAM Computing Element - EMI-2 - SL6 (external Torque, external Argus, MPI enabled)
   * *These notes are provided by site admins on a best effort base as a contribution to the IGI communities and MUST not be considered as a subsitute of the [[http://wiki.italiangrid.it/twiki/bin/view/IGIRelease/IgiEmi][Official IGI documentation]].*
   * This document is addressed to site administrators responsible for middleware installation and configuration.
   * The goal of this page is to provide some hints and examples on how to install and configure an EMI-2 *CREAM CE* service based on EMI middleware, in *no cluster mode*, with *TORQUE* as batch system installed on a different host, using an *external ARGUS server* for the users authorization and with MPI enabled.
| *CREAM* | *CLUSTER MODE* | *ARGUS* | *MPI* | *TORQUE* | *WNODES* |
| EMI-2 SL6 |  no | external server  | enabled | external server | TODO |

%TOC%

---++ References
   1. [[http://www.italiangrid.it/][About IGI - Italian Grid infrastructure]]
      1. [[http://wiki.italiangrid.it/twiki/bin/view/IGIRelease/WebHome][About IGI Release]]
      1. [[http://wiki.italiangrid.it/twiki/bin/view/IGIRelease/IgiEmi][IGI Official Installation and Configuration guide]]
   1. [[http://www.eu-emi.eu/emi-2-matterhorn][EMI-2 Release]]
      1.[[http://www.eu-emi.eu/emi-2-matterhorn-products/-/asset_publisher/B4Rk/content/cream-2][CREAM]]
      1. [[http://www.eu-emi.eu/emi-2-matterhorn-products/-/asset_publisher/B4Rk/content/cream-torque-module-1][CREAM TORQUE module]]
   1. [[https://twiki.cern.ch/twiki/bin/view/LCG/YaimGuide400][ Yaim Guide]]
      1. [[https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#site_info_def][site-info.def yaim variables]]
      1. [[https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#cream_CE][CREAM yaim variables]]
      1. [[https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#TORQUE][TORQUE Yaim variables]]
   1. [[https://wiki.egi.eu/wiki/Tools/Manuals/SiteProblemsFollowUp][Troubleshooting Guide for Operational Errors on EGI Sites]]
   1. [[https://wiki.egi.eu/wiki/Tools/Manuals/AdministrationFaq][Grid Administration FAQs page]]

---++ Service installation
---+++ O.S. and Repos 
   * Starts from a fresh installation of Scientific Linux 6.x (x86_64). 
<verbatim>
# cat /etc/redhat-release 
Scientific Linux release 6.2 (Carbon)
</verbatim>
  
   * Install the additional repositories: EPEL, Certification Authority, EMI-2

<verbatim>
# yum install yum-priorities yum-protectbase epel-release
# rpm -ivh http://emisoft.web.cern.ch/emisoft/dist/EMI/2/sl6/x86_64/base/emi-release-2.0.0-1.sl6.noarch.rpm

# cd /etc/yum.repos.d/
# wget http://repo-pd.italiangrid.it/mrepo/repos/egi-trustanchors.repo
</verbatim>

   * Be sure that SELINUX is disabled (or permissive). Details on how to disable SELINUX are [[http://fedoraproject.org/wiki/SELinux/setenforce][here]]:

<verbatim>
# getenforce 
Disabled
</verbatim>

---+++ yum install
<verbatim>
# yum clean all 

# yum install ca-policy-egi-core  emi-cream-ce emi-torque-utils glite-mpi 
</verbatim>

---++ Service configuration
You have to copy the configuration files in another path, for example root, and set them properly (see later):
<verbatim>
# cp -vr /opt/glite/yaim/examples/siteinfo .
</verbatim>

---+++ host certificate
<verbatim>
# ll /etc/grid-security/host*
-rw-r--r-- 1 root root 1440 Oct 18 09:31 /etc/grid-security/hostcert.pem
-r-------- 1 root root  887 Oct 18 09:31 /etc/grid-security/hostkey.pem
</verbatim>

---+++ vo.d directory
Create the directory =siteinfo/vo.d= and fill it with a file for each supported VO. You can download them from [[https://forge.cnaf.infn.it/plugins/scmsvn/viewcvs.php/branches/BRANCH-4_0_X/ig-yaim/examples/siteinfo/vo.d/?root=igrelease][HERE]] and [[VoDirContent][here]] an example for some VOs.
Information about the several VOs are available at the [[http://operations-portal.in2p3.fr/vo][CENTRAL OPERATIONS PORTAL]].

---+++ users and groups 
You can download them from [[https://forge.cnaf.infn.it/plugins/scmsvn/viewcvs.php/branches/BRANCH-4_0_X/ig-yaim/examples/?rev=6231&root=igrelease#dirlist][HERE]].

---+++ Munge 
Copy the key =/etc/munge/munge.key= from the Torque server to every host of your cluster, adjust the permissions and start the service
<verbatim>
# chown munge:munge /etc/munge/munge.key

# ls -ltr /etc/munge/
total 4
-r-------- 1 munge munge 1024 Jan 13 14:32 munge.key

# chkconfig munge on
# /etc/init.d/munge restart
</verbatim>

---+++ site-info.def
KISS: Keep it simple, stupid! For your convenience there is an explanation of each yaim variable. For more details look [[https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#TORQUE][HERE]]. 

SUGGESTION: use the same [[SiteDefCreamWN][site-info.def]] for CREAM and WNs: for this reason in this example file there are yaim variable used by CREAM, TORQUE or emi-WN.
<verbatim>
# cat site-info.def 
CE_HOST=cream-01.cnaf.infn.it
SITE_NAME=IGI-BOLOGNA


BATCH_SERVER=batch.cnaf.infn.it
BATCH_LOG_DIR=/var/torque

#BDII_HOST=egee-bdii.cnaf.infn.it

CE_BATCH_SYS=torque
JOB_MANAGER=pbs
BATCH_VERSION=torque-2.5.7
#CE_DATADIR=

CE_INBOUNDIP=FALSE
CE_OUTBOUNDIP=TRUE
CE_OS="ScientificSL"
CE_OS_RELEASE=6.2
CE_OS_VERSION="Carbon"

CE_RUNTIMEENV="IGI-BOLOGNA"

CE_PHYSCPU=8
CE_LOGCPU=16
CE_MINPHYSMEM=16000
CE_MINVIRTMEM=32000
CE_SMPSIZE=8
CE_CPU_MODEL=Xeon
CE_CPU_SPEED=2493
CE_CPU_VENDOR=intel
CE_CAPABILITY="CPUScalingReferenceSI00=1039 glexec"
CE_OTHERDESCR="Cores=1,Benchmark=4.156-HEP-SPEC06"
CE_SF00=951
CE_SI00=1039
CE_OS_ARCH=x86_64

CREAM_PEPC_RESOURCEID="http://cnaf.infn.it/cremino"

USERS_CONF=/root/siteinfo/ig-users.conf
GROUPS_CONF=/root/siteinfo/ig-users.conf

VOS="comput-er.it dteam igi.italiangrid.it infngrid ops gridit"
QUEUES="cert prod"
CERT_GROUP_ENABLE="dteam infngrid ops /dteam/ROLE=lcgadmin /dteam/ROLE=production /ops/ROLE=lcgadmin /ops/ROLE=pilot /infngrid/ROLE=SoftwareManager /infngrid/ROLE=pilot"
PROD_GROUP_ENABLE="comput-er.it gridit igi.italiangrid.it /comput-er.it/ROLE=SoftwareManager /gridit/ROLE=SoftwareManager /igi.italiangrid.it/ROLE=SoftwareManager"
VO_SW_DIR=/opt/exp_soft

WN_LIST="/root/siteinfo/wn-list.conf"
MUNGE_KEY_FILE=/etc/munge/munge.key
CONFIG_MAUI="no"

MYSQL_PASSWORD=*********************************
APEL_DB_PASSWORD=not_used
APEL_MYSQL_HOST=not_used
SE_LIST="darkstorm.cnaf.infn.it"
SE_MOUNT_INFO_LIST="none"
</verbatim>

---+++ WN list
Set in this file the WNs list, for example:
<verbatim>
# less /root/siteinfo/wn-list.conf
wn05.cnaf.infn.it
wn06.cnaf.infn.it
</verbatim>

---+++ services/glite-mpi_ce
<verbatim>
# cp /opt/glite/yaim/examples/siteinfo/services/glite-mpi_ce /root/siteinfo/services/

# cat services/glite-mpi_ce 
# Setup configuration variables that are common to both the CE and WN

if [ -r ${config_dir}/services/glite-mpi ]; then
 source ${config_dir}/services/glite-mpi
fi

# The MPI CE config function can create a submit filter for
# Torque to ensure that CPU allocation is performed correctly.
# Change this variable to "yes" to have YAIM create this filter.
# Warning: if you have an existing torque.cfg it will be modified.
MPI_SUBMIT_FILTER=${MPI_SUBMIT_FILTER:-"yes"}
</verbatim>

---+++ services/glite-creamce
<verbatim>
# cat /root/siteinfo/services/glite-creamce 
#
# YAIM creamCE specific variables
#

#
# CE-monitor host (by default CE-monitor is installed on the same machine as 
# cream-CE)
CEMON_HOST=$CE_HOST
#
# CREAM database user
CREAM_DB_USER=********************
CREAM_DB_PASSWORD=****************************

#
# Machine hosting the BLAH blparser.
# In this machine batch system logs must be accessible.
BLPARSER_HOST=$CE_HOST

# Value to be published as GlueCEStateStatus instead of Production
#CREAM_CE_STATE=Special
</verbatim>

---+++ services/dgas_sensors (not available yet)
TODO

---+++ yaim check
Verify to have set all the yaim variables by launching:
<verbatim>
# /opt/glite/yaim/bin/yaim -v -s /root/siteinfo/site-info.def -n creamCE -n TORQUE_utils 
</verbatim>

---+++ yaim config
<verbatim>
# /opt/glite/yaim/bin/yaim -c -s /root/siteinfo/site-info.def -n creamCE -n TORQUE_utils 
</verbatim>

---++ Service Checks
   * After service installation to have a look if all were installed in a proper way, you could have a look to [[http://wiki.italiangrid.it/twiki/bin/view/CREAM/ServiceReferenceCard][Service CREAM Reference Card]]
   * You can also perform some [[http://wiki.italiangrid.it/twiki/bin/view/CREAM/TroubleshootingGuide#1_Checks_to_be_done_after_instal][checks]] after the installation and configuration of your CREAM

---++ Revisions
| *Date* | *Comment* | *By* |
| 2012-05-25 | First draft | Paolo Veronesi |

-- Main.PaoloVeronesi - 2012-05-25
Topic revision: r2 - 2012-05-30 - PaoloVeronesi
Account
- Log In
~~Edit~~
~~Attach~~