Tags:
, view all tags

Tutorial StoRM

Pre-requisite checks

  • Verify the ntpd service
    ]# service ntpd status
    ntpd (pid  1713) is running...
    ]# chkconfig --list | grep ntpd
    ntpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
     

  • Verify the hostname output (Fully Qualified Domain Name (FQAN))
    ]# hostname -f
    tutor-storm.cnaf.infn.it
    ]# cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1               localhost.localdomain localhost
    ::1             localhost6.localdomain6 localhost6
    131.154.100.145         tutor-storm.cnaf.infn.it tutor-storm
     
    • In the case the hosts file does not contain the last line, please modify the file and restart network service (Foundamental check for having a gridftp service running properly)
  • Verify host file certificates on the StoRM node
    ]# ll /etc/grid-security/
    total 60
    drwxr-xr-x 2 root root 49152 Sep 25 13:07 certificates
    -rw-r--r-- 1 root root  1452 Oct  2 17:42 hostcert.pem
    -r-------- 1 root root   891 Oct  2 17:42 hostkey.pem
     
    • hostcert.pem must have permission 0644. If no, please change it
      ]# chmod 0644 hostcert.pem
       
    • hostkey.pem must have permission 0400. If no, please change it
      ]# chmod 0400 hostkey.pem
       
  • Verify acl support. StoRM uses the ACLs on files and directory to implement the security model by using the native access to the file system.
    • Verify acl CLIs
      ]# rpm -qa | grep acl
      ]# yum install acl
      Loaded plugins: downloadonly, kernel-module, priorities, protect-packages,
                   : protectbase, security, verify, versionlock
      521 packages excluded due to repository priority protections 
      0 packages excluded due to repository protections
      Reading version lock configuration
      Setting up Install Process
      Resolving Dependencies
      --> Running transaction check
      ---> Package acl.x86_64 0:2.2.39-8.el5 set to be updated
      --> Finished Dependency Resolution
      Beginning Kernel Module Plugin
      Finished Kernel Module Plugin
      
      Dependencies Resolved
      
      ================================================================================
       Package        Arch              Version                 Repository       Size
      ================================================================================
      Installing:
       acl            x86_64            2.2.39-8.el5            sl5x             69 k
      
      Transaction Summary
      ================================================================================
      Install       1 Package(s)
      Upgrade       0 Package(s)
      
      Total download size: 69 k
      Is this ok [y/N]: y
      Downloading Packages:
      acl-2.2.39-8.el5.x86_64.rpm                              |  69 kB     00:00
      Running rpm_check_debug
      Running Transaction Test
      Finished Transaction Test
      Transaction Test Succeeded
      Running Transaction
        Installing     : acl                                                      1/1
      
      Installed:
        acl.x86_64 0:2.2.39-8.el5
      
      Complete!
       
    • Verify extended ACL support by using the root user
      ]# echo test > testfile
      ]# setfacl -m u:root:rw testfile
      ]# getfacl  testfile
      # file: testfile
      # owner: root
      # group: root
      user::rw-
      user:root:rw
      group::r--
      mask::rw-
      other::r--
      ]# rm -rf testfile
       
  • Verify attr support. StoRM uses the extended attributes on files to store some metadata related to the file (e.g., checksum)
    • Verify attr CLIs
      ]# rpm -qa | grep attr
      ]# yum install attr
      Loaded plugins: downloadonly, kernel-module, priorities, protect-packages,
                    : protectbase, security, verify, versionlock
      521 packages excluded due to repository priority protections
      0 packages excluded due to repository protections
      Reading version lock configuration
      Setting up Install Process
      Resolving Dependencies
      --> Running transaction check
      ---> Package attr.x86_64 0:2.4.32-1.1 set to be updated
      --> Finished Dependency Resolution
      Beginning Kernel Module Plugin
      Finished Kernel Module Plugin
      
      Dependencies Resolved
      
      ================================================================================
       Package         Arch              Version                Repository       Size
      ================================================================================
      Installing:
       attr            x86_64            2.4.32-1.1             sl5x             52 k
      
      Transaction Summary
      ================================================================================
      Install       1 Package(s)
      Upgrade       0 Package(s)
      
      Total download size: 52 k
      Is this ok [y/N]: y
      Downloading Packages:
      attr-2.4.32-1.1.x86_64.rpm                               |  52 kB     00:00
      Running rpm_check_debug
      Running Transaction Test
      Finished Transaction Test
      Transaction Test Succeeded
      Running Transaction
        Installing     : attr                                                     1/1
      
      Installed:
        attr.x86_64 0:2.4.32-1.1
      
      Complete!
       
    • Verify extended attribute support by using
      ]# touch testfile
      ]# setfattr -n joda.test -v test testfile
      ]# getfattr -d testfile
      # file: testfile
      user.test="test"
      ]# rm testfile
       

  • Satisfy common repository settings
    ]# rpm -qa | grep epel
    epel-release-5-4
     

  • Satisfy EMI repository settings
    ]# mkdir localrpm
    ]# cd localrpm
    ]# wget http://emisoft.web.cern.ch/emisoft/dist/EMI/2/sl5/x86_64/base/emi-release-2.0.0-1.sl5.noarch.rpm
    --2012-10-02 18:46:32--  http://emisoft.web.cern.ch/emisoft/dist/EMI/2/sl5/x86_64/base/emi-release-2.0.0-1.sl5.noarch.rpm
    Resolving emisoft.web.cern.ch... 137.138.139.27
    Connecting to emisoft.web.cern.ch|137.138.139.27|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 5189 (5.1K) [application/x-rpm]
    Saving to: `emi-release-2.0.0-1.sl5.noarch.rpm'
    
    100%[======================================>] 5,189       --.-K/s   in 0s
    
    2012-10-02 18:46:32 (353 MB/s) - `emi-release-2.0.0-1.sl5.noarch.rpm' saved [5189/5189]
    ]# # yum localinstall --nogpgcheck emi-release-2.0.0-1.sl5.noarch.rpm
    Loaded plugins: downloadonly, kernel-module, priorities, protect-packages,
                  : protectbase, security, verify, versionlock
    Setting up Local Package Process
    Examining emi-release-2.0.0-1.sl5.noarch.rpm: emi-release-2.0.0-1.sl5.noarch
    Marking emi-release-2.0.0-1.sl5.noarch.rpm to be installed
    521 packages excluded due to repository priority protections
    0 packages excluded due to repository protections
    Reading version lock configuration
    Resolving Dependencies
    --> Running transaction check
    ---> Package emi-release.noarch 0:2.0.0-1.sl5 set to be updated
    --> Finished Dependency Resolution
    Beginning Kernel Module Plugin
    Finished Kernel Module Plugin
    
    Dependencies Resolved
    
    ================================================================================
     Package       Arch     Version         Repository                         Size
    ================================================================================
    Installing:
     emi-release   noarch   2.0.0-1.sl5     /emi-release-2.0.0-1.sl5.noarch   2.7 k
    
    Transaction Summary
    ================================================================================
    Install       1 Package(s)
    Upgrade       0 Package(s)
    
    Total size: 2.7 k
    Is this ok [y/N]: y
    Downloading Packages:
    Running rpm_check_debug
    Running Transaction Test
    Finished Transaction Test
    Transaction Test Succeeded
    Running Transaction
      Installing     : emi-release                                              1/1
    
    Installed:
      emi-release.noarch 0:2.0.0-1.sl5
    
    Complete!
     

  • Satisfy EGI repository settings
    ]# ls /etc/yum.repos.d/egi-trustanchors.repo
     

  • Verify the ca-policy-egi-core package
    ]# rpm -qa | grep ca-policy-egi-core
    ca-policy-egi-core-1.50-1
     

Install the StoRM services in a standalone deployment

  • Clean yum cache
    ]# yum clean all
    Loaded plugins: downloadonly, kernel-module, priorities, protect-packages,
                  : protectbase, security, verify, versionlock
    Cleaning up Everything
     

  • Install the main StoRM node metapackages:
    ]# yum install emi-storm-backend-mp emi-storm-frontend-mp emi-storm-globus-gridftp-mp emi-storm-gridhttps-mp
    ...
    EMI-2-base/gpgkey                                                                                                                     | 1.7 kB     00:00
    Importing GPG key 0xDF9E12EF "Doina Cristina Aiftimiei (EMI Release Manager) <aiftim@pd.infn.it>" from /etc/pki/rpm-gpg/RPM-GPG-KEY-emi
    Is this ok [y/N]: y
    ...
     

  • Install the StoRM pre-assembled configuration package
    ]# yum install storm-pre-assembled-configuration
    ...
    ]# rpm -ql  storm-pre-assembled-configuration
    /etc/storm
    /etc/storm/siteinfo
    /etc/storm/siteinfo/storm-groups.conf
    /etc/storm/siteinfo/storm-users.conf
    /etc/storm/siteinfo/storm-wn-list.conf
    /etc/storm/siteinfo/storm.def
    /etc/storm/siteinfo/storm.def.template
    /etc/storm/siteinfo/vo.d
    /etc/storm/siteinfo/vo.d/dteam
    /etc/storm/siteinfo/vo.d/infngrid
    /etc/storm/siteinfo/vo.d/ops
    /etc/storm/siteinfo/vo.d/testers.eu-emi.eu
    /usr/sbin/EMI_1_repos_install
    /usr/share/doc/storm-pre-assembled-configuration-1.0.0
    /usr/share/doc/storm-pre-assembled-configuration-1.0.0/AUTHORS
    /usr/share/doc/storm-pre-assembled-configuration-1.0.0/CREDITS
    /usr/share/doc/storm-pre-assembled-configuration-1.0.0/ChangeLog
    /usr/share/doc/storm-pre-assembled-configuration-1.0.0/LICENSE
    /usr/share/doc/storm-pre-assembled-configuration-1.0.0/README
     

Pre-configure checks

  • Configure the igi.italiangrid.it and infngrid VOs
    • Create vo.d/igi.italiangrid.it file inside the configuration /etc/storm/siteinfo/ directory
      ]# vi /etc/storm/siteinfo/vo.d/igi.italiangrid.it
      SW_DIR=/cvmfs/igi.italiangrid.it
      DEFAULT_SE=$SE_HOST
      STORAGE_DIR=$CLASSIC_STORAGE_DIR/igi
      VOMS_SERVERS="'vomss://vomsmania.cnaf.infn.it:8443/voms/igi.italiangrid.it?/igi.italiangrid.it'"
      VOMSES="'igi.italiangrid.it vomsmania.cnaf.infn.it 15003 /C=IT/O=INFN/OU=Host/L=CNAF/CN=vomsmania.cnaf.infn.it igi.italiangrid.it'"
      VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN CA'"
         
    • Create vo.d/infngrid file inside the configuration /etc/storm/siteinfo/ directory
      ]# vi /etc/storm/siteinfo/vo.d/infngrid
      SW_DIR=$VO_SW_DIR/infngrid
      DEFAULT_SE=$SE_HOST
      STORAGE_DIR=$CLASSIC_STORAGE_DIR/infngrid
      VOMS_SERVERS="'vomss://voms.cnaf.infn.it:8443/voms/infngrid?/infngrid' 'vomss://voms-01.pd.infn.it:8443/voms/infngrid?/infngrid'"
      VOMSES="'infngrid voms.cnaf.infn.it 15000 /C=IT/O=INFN/OU=Host/L=CNAF/CN=voms.cnaf.infn.it infngrid' 'infngrid voms-01.pd.infn.it 15000 /C=IT/O=INFN/OU=Host/L=Padova/CN=voms-01.pd.infn.it infngrid'"
      VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN CA' '/C=IT/O=INFN/CN=INFN CA'"
         
    • Modify the storm-users.conf file adding igi.italiagrid.it users
      ]# vi /etc/storm/siteinfo/storm-users.conf
      ...
      3601:igi001:7400:igi:igi.italiangrid.it::
      3602:igi002:7400:igi:igi.italiangrid.it::
      3603:igi003:7400:igi:igi.italiangrid.it::
      3604:igi004:7400:igi:igi.italiangrid.it::
      3605:igi005:7400:igi:igi.italiangrid.it::
      3606:igi006:7400:igi:igi.italiangrid.it::
      3607:igi007:7400:igi:igi.italiangrid.it::
      3608:igi008:7400:igi:igi.italiangrid.it::
      3609:igi009:7400:igi:igi.italiangrid.it::
      3610:igi010:7400:igi:igi.italiangrid.it::
      3611:igi011:7400:igi:igi.italiangrid.it::
      3612:igi012:7400:igi:igi.italiangrid.it::
      3613:igi013:7400:igi:igi.italiangrid.it::
      3614:igi014:7400:igi:igi.italiangrid.it::
      3615:igi015:7400:igi:igi.italiangrid.it::
      3616:igi016:7400:igi:igi.italiangrid.it::
      3617:igi017:7400:igi:igi.italiangrid.it::
      3618:igi018:7400:igi:igi.italiangrid.it::
      3619:igi019:7400:igi:igi.italiangrid.it::
      3620:igi020:7400:igi:igi.italiangrid.it::
      3621:igi021:7400:igi:igi.italiangrid.it::
      3622:igi022:7400:igi:igi.italiangrid.it::
      3623:igi023:7400:igi:igi.italiangrid.it::
      3624:igi024:7400:igi:igi.italiangigi:it::
      3625:igi025:7400:igi:igi.italiangrid.it::
      10161:sgmigi001:7450,7400:sgmigi,igi:igi.italiangrid.it:sgm:
      10162:sgmigi002:7450,7400:sgmigi,igi:igi.italiangrid.it:sgm:
      10163:sgmigi003:7450,7400:sgmigi,igi:igi.italiangrid.it:sgm:
      10164:sgmigi004:7450,7400:sgmigi,igi:igi.italiangrid.it:sgm:
      10165:sgmigi005:7450,7400:sgmigi,igi:igi.italiangrid.it:sgm:
      7106:sgminfngrid001:2504,2405:sgminfngrid,infngrid:infngrid:sgm:
      7206:sgminfngrid002:2504,2405:sgminfngrid,infngrid:infngrid:sgm:
      7306:sgminfngrid003:2504,2405:sgminfngrid,infngrid:infngrid:sgm:
      7401:pilinfngrid001:2604,2405:pilinfngrid,infngrid:infngrid:pilot:
      7402:pilinfngrid002:2604,2405:pilinfngrid,infngrid:infngrid:pilot:
      7403:pilinfngrid003:2604,2405:pilinfngrid,infngrid:infngrid:pilot:
      2451:infngrid001:2405:infngrid:infngrid::
      2452:infngrid002:2405:infngrid:infngrid::
      2453:infngrid003:2405:infngrid:infngrid::
      2454:infngrid004:2405:infngrid:infngrid::
      2455:infngrid005:2405:infngrid:infngrid::
      2456:infngrid006:2405:infngrid:infngrid::
      2457:infngrid007:2405:infngrid:infngrid::
      2458:infngrid008:2405:infngrid:infngrid::
      2459:infngrid009:2405:infngrid:infngrid::
      2460:infngrid010:2405:infngrid:infngrid::
      2461:infngrid011:2405:infngrid:infngrid::
      2462:infngrid012:2405:infngrid:infngrid::
      2463:infngrid013:2405:infngrid:infngrid::
      2464:infngrid014:2405:infngrid:infngrid::
      2465:infngrid015:2405:infngrid:infngrid::
      2466:infngrid016:2405:infngrid:infngrid::
      2467:infngrid017:2405:infngrid:infngrid::
      2468:infngrid018:2405:infngrid:infngrid::
      2469:infngrid019:2405:infngrid:infngrid::
      2470:infngrid020:2405:infngrid:infngrid::
         
    • Modify the storm-groups.conf file adding igi.italiagrid.it details
      ]# vi /etc/storm/siteinfo/storm-groups.conf
      ...
      "/igi.italiangrid.it/ROLE=SoftwareManager":::sgm:
      "/igi.italiangrid.it"::::
      "/igi.italiangrid.it/*"::::
      "/infngrid/ROLE=SoftwareManager":::sgm:
      "/infngrid/ROLE=pilot":::pilot:
      "/infngrid"::::
         
    • Modify the storm.def file adding the igi.italiangrid.it VO
      ]# vi /etc/storm/siteinfo/storm.def
      ...
      VOS="igi.italiangrid.it testers.eu-emi.eu dteam"
         

  • Configure the mysqld service
    • Modify the my.cnf file
      ]# cat /etc/my.cnf
      [mysqld]
      datadir=/var/lib/mysql
      socket=/var/lib/mysql/mysql.sock
      user=mysql
      # Default to using old password format for compatibility with mysql 3.x
      # clients (those using the mysqlclient10 compatibility package).
      old_passwords=1
      
      # Disabling symbolic-links is recommended to prevent assorted security risks;
      # to do so, uncomment this line:
      # symbolic-links=0
      
      [mysqld_safe]
      log-error=/var/log/mysqld.log
      pid-file=/var/run/mysqld/mysqld.pid
       
    • Add to the mysqld section the following options
      max_connections=2000
      wait_timeout=86400
      innodb_buffer_pool_size=256M
       

  • Modify the storm.def file
    • Modify top BDII hostname
      BDII_HOST=gridit-bdii-01.cnaf.infn.it
        
    • Modify MY_DOMAIN
      MY_DOMAIN="cnaf.infn.it"
        

Configure the StoRM services

  • Check DB information
    MYSQL_PASSWORD="storm"
    STORM_DB_PWD="bluemoon"
    STORM_DB_USER="storm"
     
  • Modify information useful for the Backend and Gridhttps services
    • Change STORM BACKEND HOST value
      STORM_BACKEND_HOST=tutor-storm.cnaf.infn.it
       
  • Modify information useful for the Backend and Frontend services
    STORM_DB_HOST=$STORM_BACKEND_HOST
    STORM_USER="storm"
     
  • Check Frontend information
    STORM_FE_BE_XMLRPC_HOST=$STORM_BACKEND_HOST
     
  • Check BDII information
    STORM_INFO_OVERWRITE=true
    SITE_NAME="storm-1.10-all-in-one"
    STORM_BDII_OVERWRITE="true"
    STORM_ENDPOINT_QUALITY_LEVEL=2
    STORM_ENDPOINT_SERVING_STATE=4
    SE_TYPE="disk"
    SE_LIST=$STORM_FRONTEND_PUBLIC_HOST
     
  • Check Backend Information
    • Set Frontend service host list
      STORM_FRONTEND_HOST_LIST=$STORM_BACKEND_HOST
       
      * Set Frontend service public host
      STORM_FRONTEND_PUBLIC_HOST=$STORM_BACKEND_HOST
       
      * Set Gridftp service host list
      STORM_GRIDFTP_POOL_LIST=${STORM_BACKEND_HOST}
       
      * Set Gridhttps service host list
      STORM_GRIDHTTPS_PLUGIN_CLASSNAME=it.grid.storm.https.GhttpsHTTPSPluginInterface
      STORM_GRIDHTTPS_SERVER_PORT=8088
      STORM_GRIDFTP_POOL_LIST=${STORM_BACKEND_HOST}
      STORM_GRIDHTTPS_ENABLED=true 
      STORM_INFO_HTTP_SUPPORT=true
      STORM_INFO_HTTPS_SUPPORT=true
      STORM_GRIDHTTPS_SERVER_USER_UID=91
      STORM_GRIDHTTPS_SERVER_GROUP_UID=91
      
    • Set default directory for Storage Area
      STORM_DEFAULT_ROOT="/storage"
       
      * Specify the list of the supported Storage Areas (Please remember that the values of the Storage Areas can be different to the VO name)
      STORM_STORAGEAREA_LIST="$VOS"
       
      * Set for each Storage Area at least the total size expressed in GB (Must be an integer value)
      STORM_DTEAM_ONLINE_SIZE=4
      STORM_TESTERSEUEMIEU_ONLINE_SIZE=4
      STORM_IGIITALIANGRIDIT_ONLINE_SIZE=4
        

  • Disable Checksum service
        
    STORM_CKSUM_SUPPORT=false
       

  • Set Gridhttps service
        
    STORM_GRIDHTTPS_SERVER_PORT=8088
       

  • Set Gridftp service to support
        
    GRIDFTP_WITH_DSI="yes"
    GRIDFTP_CONNECTIONS_MAX="2000"
     

  • Run yaim
        
    /opt/glite/yaim/bin/yaim -c -d 6 -s /etc/storm/siteinfo/storm.def -n se_storm_backend -n se_storm_frontend -n se_storm_gridftp -n se_storm_gridhttps
     

Verify the StoRM deployment

  • Check the StoRM services
        
    ]# service storm-backend-server status
    
    ]# service storm-frontend-server status
    storm-frontend-server (pid 13321) is running...
    ]# service storm-globus-gridftp status
    globus-gridftp-server (pid 13432) is running...
    service tomcat5 status
     
  • Check the StoRM log files
        
    tail -f /var/log/storm/storm-backend.log
    tail -f /var/log/storm/storm-frontend-server.log
     
  • BDII by using for example an LDAP browser
  • srmping
  • lcg-cp
  • lcg-del

References

  1. System Guide, http://storm.forge.cnaf.infn.it/_media/documentation/storm-1.10.0-sysadminguide.pdf?id=documentation%3Ahome&cache=cache
  2. Functionalities, http://storm.forge.cnaf.infn.it/_media/documentation/storm_guide_v0.5.pdf?id=documentation%3Ahome&cache=cache

-- Main.Elisabetta Ronchieri - 2012-10-2

Edit | Attach | PDF | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | More topic actions...
Topic revision: r8 - 2012-10-03 - RonchieriElisabetta
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platformCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback