Tags:
, view all tags

IGI CERTIFICATION TESTBED

This testbed is supposed to be used by the "Testing, Certification and Release" group people; it should be used mainly for functionality tests and for stress tests.

HOWTO, POLICIES & PROCEDURES

HOWTO Request a new instance

To request a new instance for certification please email grid-release mailing list specifying:
  • When the instance is needed for (>request day + 3 working days)
  • What is the instance needed for, and its reference in the inventory before (or whether it needs to be added)
  • HW requirements (DISK, RAM)
  • Virtual Machine deployment is default, please specify whether you need a physical server
  • Host certificate requirements
  • OS platform
  • Grid service to install
  • Life Time Estimate

Reasonable requests will be taken in charge and tracked as sub-task on Jira task TESTBED MANAGEMENT number 7. The requester will be added as "watcher" in the relative sub-task.

HOWTO Install Debian 6.0 on virtual KVM

  • Requirements:
    • dom0 server running KVM with Intel VIrtualization technology flag enabled (check with "cat /proc/cpuinfo |grep vmx")
    • host public IP + MAC address number
  • connect to your dom0: $ ssh -X root@dom0
  1. Installation from scratch
    • Download iso image: $wget http://linux.studenti.polito.it/linuxstudenti/debian-6.0.2.1-amd64-netinst.iso
    • NEW Virtual Image:
      • Start virt image, click on dom0 hostname, select NEW and enter following parameters: $ virt-image&
        • fully virtualized, x64, kvm
        • local install ISO, OS linux, Generic 2.6
        • select from browser the iso previously downloaded
        • select storage (file or partition depending on your choice) and size (>= 10GB)
        • shared physical bridge, (eth0/1), fixed MAC (choose a free one from NewEntropy)
        • from virt-image console enter parameters for installation
          • NETWORK: IP, gateway, nameserver
          • users (root passwd, one user)
          • US keyboard, ...
          • ssh server + graphycs if needed
    • Once finished, enter machine from console and run:
  2. Installation from previously created template image:
    • Shutdown template image to clone (ex. cert-31-Debian is the template image for which we have both .img file and xml conf file)
    • Clone image:
      • $virt-clone --original-xml=XMLFILEOFTEMPLATE --name=NEWmachineNAMEinVIRTMANAGER --mac=NEWMAC_address --file=DestinationImageFILE
      • Example: $virt-clone --original-xml=/etc/libvirt/qemu/cert-31-Debian.xml --name=cert-32-Debian --mac=00:16:3e:08:01:9b --file=/virtdisk1/cert-32-Debian.img
    • New image post configuration
      • NOTE The original image must be NOT running
      • start new machine from virt-manager and enter machine console:
        • CHANGE HOSTNAME: $sed -i 's/OLD_HOSTNAME/NEW_HOSTNAME/g' /etc/hostname
        • CHANGE NETWORK CONF:
          • $ifconfing > check which eth is used
          • $ifdown ethXX (from above)
          • editing files: /etc/resolv.con , /etc/network/interfaces (new IP there)
          • restart network: $ifup ethXX *$ ifconfig ----> to check it is fine.
    • restart both original and new machine.
    • remove puppet files and rerun the puppet configuration commands as in https://yam-server.cnaf.infn.it/EMIinstall/emi-post-install.sh script *$ apt-get remove puppet *$ rm /root/puppetfile?HOSTNAME *execute the following:
wget  -q http://doctorwho.cnaf.infn.it/cgi-bin/hostinfo.cgi?certname=$HOSTNAME
apt-get install -y puppet
touch /etc/puppet/namespaceauth.conf
TEST=`grep doctorwho /etc/puppet/puppet.conf|wc -l`
if [ $TEST -eq 0 ] ; then
        cat >> /etc/puppet/puppet.conf << EOF
           server = doctorwho.cnaf.infn.it
           report = true
           listen = true
           runinterval = 360
EOF
        update-rc.d puppet enable 5
        sed -i 's/START=no/START=yes/g' /etc/default/puppet
        /etc/init.d/puppet restart
fi

HOWTO Restore clean OS image disk in XEN

  • Requirements and checks:
    • Daemons stopped (or in @chkconfig on@) in server hosting your service
    • Backup File Image with clean OS
  1. Connect to dom0 server, as root
  2. get help on xen manager with: xm
  3. List running machines: $xm list
  4. Shutdown your machine with: =$xm shutdown NAME_OF_MACHINE_AS_FROM_XM_LIST_OUTPUT (ex.. emitestbed09-SL6X-64)
  5. Restore clean os image file from backup image: cp emitestbed18-SL5X-64_CLEANOS.img emitestbed18-SL5X-64.img
  6. N.B. The image file used for a given virtual server is stored in the xen configuration file. ex:
[root@emitest-dom01 ~]# cat /etc/xen/emitestbed18-SL5X-64
name = "emitestbed18-SL5X-64"
builder = "hvm"
memory = "1024"
disk = [ 'file:/domU/emitestbed18-SL5X-64.img,hda,w' ]                     <<<<<<<<<<<<<<<<<<<<<<<<<<< THE FILE<<<<<<<<<<<<<<<<
boot = "c"
vif = [ 'mac=00:16:3e:08:01:76, bridge=xenbr0' ]
vfb = ["type=vnc,vncunused=1"]
device_model = "/usr/lib64/xen/bin/qemu-dm"
kernel = "/usr/lib/xen/boot/hvmloader"
vcpus=1
on_reboot   = 'restart'
on_crash    = 'restart'

  1. START THE NEW MACHINE WITH: $>xm create emitestbed18-SL5X-64 ----------> same name as the conf file without path /etc/xen/emitestbed18-SL5X-64

PUPPET SETUP

AVAILABLE MODULES

  1. BASENODE: deployed on all IGI servers (production + R&D)
    1. fail2ban -> bans users aftern X wrong password attempts
    2. lcg-ca -> egi trustanchor repo + CA
    3. lemon -> lemon monitoring setup
    4. nrpe -> nagios probes
    5. nsca -> nagios probes
    6. ntp -> ntp server setup
    7. pakiti-> pakiti security monitoring setup
    8. puppet
    9. repo-updates
    10. resolv
    11. security-checks -> monitor existence of dangerous files
    12. security-updates -> monitor existence of dangerous files
    13. ssh -> ssh setup + some ssh keys
    14. yum -> repos
  2. EMI AND TESTBEDS: ON ALL EMITESTBED + IGI CERTIFICATION/RELEASE nodes
    1. emi-cerntb -> install cern utilities repo + BitFAce CA for Nagios tests
    2. emi-testers-vo -> creates following files /root/siteinfo/ + /root/siteinfo/groups.conf+users.conf + /root/siteinfo/vo.d/testers2.eu-emi.eu+testers.eu-emi.eu
    3. emi-generic-vo -> creates following files /root/siteinfo/ + /root/siteinfo/groups.conf+users.conf + /root/siteinfo/vo.d/testers2.eu-emi.eu+testers.eu-emi.eu+cms+dteam (other VO can be added if needed)
    4. emi-users -> creates users with password for pre-registered bastion users in IGI Middleware Unit
    5. emi-testwrapperscript -> creates file /root/siteinfo/script-wrapper.py
  3. EMI AND TESTBEDS: REPOSITORIES AVAILABLE ON DEMAND
    1. emi1-prodrepo -> install emi1 release package (on emi1 emitesbed servers)
    2. emi1-releaseupdatesrepo -> install emi1-deployment repo (on emi1 emitesbed servers)
    3. emi2-prodrepo -> install emi2 release package
    4. emi2-rcrepo -> install emi2 RC repo
  4. EMI TESTBEDS: AUTOMATED DEPLOYMENT MODULES
    1. ARGUS
      1. SCENARIO files to deploy an argus with emitestbed policy emi2-argus-deploy -> Provides template files for automated deployment test with /root/siteinfo/script-wrapper.py |-- files | |-- EMI2_SL5-64_ARGUS_deployement_command_list.txt | |-- EMI2_SL6-64_ARGUS_deployement_command_list.txt | |-- emitestbed_policy.txt | `-- site-info.def
    2. WMS
      1. SCENARIO files to deploy a WMS from testing repo and emitestbed setup. emi1-wmstesting-deploy -> Provides template files for automated deployment test with /root/siteinfo/script-wrapper.py |-- files |-- EMI1_TESTING_SL5-64_WMS_deployement_command_list.txt | `-- site-info.def
      2. SCENARIO files to deploy a WMS from production then update it to testing repo and emitestbed setup. emi1-wmsupdate-deploy -> Provides template files for automated deployment test with /root/siteinfo/script-wrapper.py |-- files |-- |-- EMI1_UPDATE_SL5-64_WMS_deployement_command_list.txt `-- site-info.def
  5. IGI CERTIFICATION: AUTOMATED DEPLOYMENT MODULES
    1. WMS
      1. SCENARIO: install a wms from testing repo with multi VO enabled and certification bdii etc. : emi1-wms-cert-install -> Provides template files for automated deployment test with /root/siteinfo/script-wrapper.py |-- files | |-- EMI1_CERT_INSTALL_SL5-64_WMS_deployement_command_list.txt | -- site-info.def
      2. SCENARIO: install a wms from production repo with multi VO enabled and certification bdii etc. then update it to testing: emi1-wms-cert-update -> Provides template files for automated deployment test with /root/siteinfo/script-wrapper.py |-- files | |-- | |-- EMI1_CERT_UPDATE_SL5-64_WMS_deployement_command_list.txt | -- site-info.def

TESTBED INVENTORY

Other services

HOSTNAME SO SERVICE State Note
cert-bdii-04.cnaf.infn.it SLC 4.8 BDII Yes / Done To be dismissed
cream-46.pd.infn.it SL 5.5 Epel EMI Argus server 1.3.0-4 Yes / Done  
cert-17.pd.infn.it SL 5.5 dag ig_BDII_site Yes / Done Used to test CE cert-34 with HLR Server cert-40
cert-42.cnaf.infn.it SL 5.5 BDII Yes / Done BDII di certificazione
cert-42.cnaf.infn.it DISCO SL 6 Epel EMI BDII   Certificazione WMS
cert-43.cnaf.infn.it DISCO SL 6 Epel EMI Nagios   Metapackage testing

User Interface

HOSTNAME SO SERVICE State Note
cream-12.pd.infn.it SL 5.5 Epel EMI UI 1.0.0 Yes / Done  
cream-03.pd.infn.it SL 5.5 GLITE UI 3.2.11 Yes / Done  
cert-17.cnaf.infn.it SL 5.5 Epel EMI UI 1.0.0 Yes / Done Virtual UI for certification
cert-02.pd.infn.it SL 5.5 Epel EMI UI 1.0.0 Yes / Done Used to test CE cert-34 with HLR Server cert-40
cert-31.cnaf.infn.it DISCO SL5 + SL6 + Debian 6 Epel cert-31.cnaf.infn.it EMI UI 2.0.0  
VIRTUAL UI EMI2 SL 6 Epel EMI UI 2.0.0 TBD  
VIRTUAL UI EMI2 DEBIAN Epel EMI UI 2.0.0 TBD  

Workload Management Service

HOSTNAME SO SERVICE State Note
cert-25.cnaf.infn.it SL 5.7 Epel WMS 3.3.5-2 Yes / Done EMI 1.11.2-1
cert-26.cnaf.infn.it SL 5.7 Epel WMS 3.3.5-2 Yes / Done EMI 1.11.2-1
emi-demo11.cnaf.infn.it SL 5.7 Epel WMS 3.3.5-2 Yes / Done EMI 1.11.2-1
emi-demo12.cnaf.infn.it SL 5.8 Epel WMS 3.3.8 Yes / Done EMI 1.12.0-1
devel20.cnaf.infn.it SL 5.7 Epel WMS 3.3.4 Yes / Done EMI 1.7.0-1
devel17.cnaf.infn.it SL 5.7 Epel LB 2.2.10 Yes / Done EMI 1.7.0-1
VIRTUAL LB EMI1 SL 5.5 Epel EMI LB 1.0.0 USE EMI INTEGRATION ????  
cert-27.cnaf.infn.it DISCO SL 6 Epel EMI WMS 2.0.0 Yes / Done  
cert-28.cnaf.infn.it DISCO SL 6 Epel EMI WMS 2.0.0 Yes / Done  
cert-32.cnaf.infn.it DISCO Debian 6 Epel EMI WMS 2.0.0 POWEROFF  

HLR Server

HOSTNAME SO SERVICE State Note
cert-13.pd.infn.it SL 5.7 HLR IGI Yes / Done used for the HLR Server certification

Storm

HOSTNAMESorted ascending SO SERVICE State Note
cert-41.cnaf.infn.it DISCO SL 6 Epel EMI STORM 2.0.0    

cert-44.cnaf.infn.it DISCO SL 6 Epel EMI STORM 2.0.0 TBD Front End

VOMS

HOSTNAME SO SERVICE State Note
emitestbed18.cnaf.infn.it DISCO SL 5 x32 Epel + DISCO SL 5 x64 Epel EMI VOMS 1.x / 2.0.0   dom0 = emitest-dom01, cleanos file available XEN
cert-30.cnaf.infn.it DISCO SL 6 x64 Epel EMI VOMS 1.x / 2.0.0   dom0 = cert-dom0-02 KVM
emi-demo14.cnaf.infn.it Availability Status (Restricted Access)    

Computing elements: GlueCEStateStatus: Testing

LCG CEs

HOSTNAME SO SERVICE State Note
cream-11.pd.infn.it SL 4.9 lcg CE 3.1.46 Yes / Done batch system: lcglsf - LSF 7.0.5.125007
cream-26.pd.infn.it SL 4.9 lcg CE 3.1.46 Yes / Done batch system: lcgpbs - TORQUE 2.3.6

gLite CEs

HOSTNAME SO SERVICE State Note
cream-24.pd.infn.it SL 5.5 Cream-CE 3.2.14-1 Yes / Done Cream LSF 1.12.6-0
cream-32.pd.infn.it SL 5.5 Cream-CE 3.2.14-1 Yes / Done Cream Torque 1.12.6-0

EMI CEs

LSF (7.0.5.125007) CEs
HOSTNAME SO SERVICE EMI Version State Note   CeMon Argus BLParser Cluster
cream-19.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13.4-1
BDII Site 1.0.0-1
1.9.0-1 Yes / Done     no yes new no
cream-20.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13-4-1 1.8.0-1 Yes / Done UIn use for EMI 2 CREAM 1.14.1   no yes new no
cream-22.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13.4-1 1.8.0-1 Yes / Done     no yes old no
cream-23.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13.4-1 1.8.0-1 Yes / Done     no yes new no
cream-29.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13.4-1 1.10.0-1 Yes / Done Update to BLAH 1.16.6 - using to certified cream 1.13.4-1   no yes new no
cert-41.pd.infn.it SL 6. Epel X86_64 CREAM 1.14.1 ..... Yes / Done In use for EMI 2 CREAM 1.14.1 certification   no yes new no
CREAM EMI2 SL 6 Epel EMI CREAM LSF 2.0.0 TBD   VIRTUAL Padova ?? + Usano i wn del pool LSF
CREAM EMI2 Debian 6 Epel EMI CREAM LSF 2.0.0 TBD   VIRTUAL Padova ?? + Usano i wn del pool LSF

Torque (2.5.7-2) CEs
HOSTNAME SO SERVICE EMI Version State Note   CeMon Argus BLParser Cluster
cream-30.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13.4.1 1.8.0-1 Yes / Done In use for EMI 2 CREAM 1.14.1   no no new no
cream-31.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13.4.1 1.8.0-1 Yes / Done     no no new no
cream-39.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13.4.1 1.8.0-1 Yes / Done Used to certify BLAH 1.16.5 - using to certified cream 1.13.4-1   no no new no
cream-40.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13.4-1 1.8.0-1 Yes / Done Used to certify BLAH 1.16.5   no no new no
cream-41.pd.infn.it SL 5.5 Epel X86_64 Cream 1.13.4-1 1.8.0-1 Yes / Done Used to certify BLAH 1.16.5   no no old no
cert-34.pd.infn.it SL 5.5 Epel Cream 1.13.4-1 + DGAS_sensors   Yes / Done  
cert-42.pd.infn.it SL 6. Epel X86_64 CREAM 1.14.1 ..... Yes / Done In use for EMI 2 CREAM 1.14.1 certification   no yes new no
cert-44.cnaf.infn.it SL 5 Epel EMI CREAM PBS 2.0.0   CREAM EMI2
cert-45.cnaf.infn.it SL 6 Epel EMI CREAM PBS 2.0.0   CREAM EMI2
WN SL5x 64bit emi-demo10.cnaf.infn.it Availability Status (Restricted Access)    
WN SL6/64bit emi-demo17.cnaf.infn.it Status GLiteJobManagement  
cert-46.cnaf.infn.it Debian 6 Epel EMI CREAM PBS 2.0.0 TBD CREAM EMI2
cert-47.cnaf.infn.it SL 6 Epel EMI WN 2.0.0 TBD WN EMI2
cert-48.cnaf.infn.it Debian 6 Epel EMI WN 2.0.0 TBD WN EMI2
WN EMI2 SL 6 Epel EMI WN 2.0.0 TBD  
WN EMI2 Debian 6 Epel EMI WN 2.0.0 TBD  

SGE CEs
CREAM EMI2 SL 6 Epel EMI CREAM SGE 2.0.0 TBD  
CREAM EMI2 SL 6 Epel EMI CREAM SGE 2.0.0 TBD