Porting ANSYS Software package to the Grid

Introduction

The work performed to implement a complex engineering simulation software usecase on distributed systems making use of the IGI web portal is here presented and discussed. The work has been carried out within a collaboration with the User Support Unit of the Italian Grid Initiative (IGI) and the INFN-Legnaro National Laboratories (INFN-LNL).

Ported Sw: ANSYS

Web interface

A web interface to run the simulation exploiting the production Grid services was provided through a dedicated portlet of the IGI Portal which is a powerful and easy to use gateway to distributed computing and storage resources. The web GUI was built in strong partnership with the user community interested in porting the application to gather their requirements

Web interface is reachable here: IGI Portal (user has to belong to the ANSYS group of the GRIDIT VO to access the customized interface).

Main features

In a typical usecase the user provides the initial input files and configuration parameters and waits for the results until the calculation is terminated. This process may take several hours, even days using the resources available to run this application. As the granted amount of cpu time is limited on Grid sites, special care was taken in handling the checkpointing of the calculation. Another crucial aspect of such long simulations is the evolution's audit of the calculations at runtime, so a facility that exploits Grid Storage Elements was created to make temporary files and partial output available for inspection at runtime. The present strategy enable the user to check the consistency of the output at runtime, evaluating possible strategies aimed at saving time, computing resources and at avoiding waste of license usage.

The bash compliant wrappers

The web interfae operates in collaboration with a a set of bash compliant wrappers that operate at WN level and are able to set the computational environment and monitor the application's execution.

ansys1srm_restart5.sh

Executable (successively called “the script”) in bash language able to control the ansys execution in the WN. The script is used for the first execution or resubmission after hang-up. Three variables come directly from the portal and are: output - specify the outputname userfolder - specify the user-folder used to get and put outputs url - specify the file to download in case of resubmission after hang-up

Action Query the PBS_NODEFILE variable to get the default walltime set for the queue

Action Define the binary and its path, the SE and the clientSRM command; set “rstauto” file value to FALSE by default

Function USERFOLDER Check if userfolder exists via clientSRM, if not create them. 4 retrying in creating the userfolder directory.

Function PREPARETOPUT Start the PreparetoPut function using the clientSRM functionalities to store a rewritable log file in the SE (file lifetime 24 hours)

Function PREPAREINPUT Prepare the input file needed for the run depending on the status (fist run or resubmission after hang-up) set by the content of $url variable

Function RUNNINGAPP Start the application, permit it to run for a predefined amount of CPU time, then kill it. The function has a check procedure enabled: at intervals of 1 hour the function control the status of the application, upload the log file to the SE by using the “check logs” function and upload the application’s CPU running time.

Function CHECKLOGS Check logs upload the log file to the SE by using the globus-url-copy command (4 retrying).

Function PREPAREOUTPUT Prepare output remove output_oldest file, if exists in the SE; move the output_last file to output_oldest in the SE; tar the output in the localhost and copy it as output_last in the SE (all the comands are performed by using the clientSRM functionalities; file lifetime 7 days).

Action Set the “rstauto” file content to FALSE or TRUE depending on the match of some variables used as control variables. The fiel will be sent to the portal for further submission evaluation.

Action Calling the described functions

ansys2srm_restart6.sh

Executable (successively called “the script”) in bash language able to control the ansys execution in the WN. The script is used for the first execution or resubmission after hang-up. Three variables come directly from the portal and are: output - specify the outputname userfolder - specify the user-folder used to get and put outputs url - specify the file to download in case of resubmission after hang-up

Action Query the PBS_NODEFILE variable to get the default walltime set for the queue

Action Define the binary and its path, the SE and the clientSRM command; set “rstauto” file value to FALSE by default

Function USERFOLDER Check if userfolder exists via clientSRM, if not the calculation is aborted.

Function PREPARETOPUT Start the PreparetoPut function using the clientSRM functionalities to store a rewritable log file in the SE (file lifetime 24 hours)

Function PREPAREINPUT Prepare the input file needed for the run by transferring the remote file stored in the SE and defined by the $url variable

Function RUNNINGAPP Start the application, permit it to run for a predefined amount of CPU time, then kill it. The function has a check procedure enabled: at intervals of 1 hour the function control the status of the application, upload the log file to the SE by using the “check logs” function and upload the application’s CPU running time.

Function CHECKLOGS Check logs upload the log file to the SE by using the globus-url-copy command (4 retrying).

Function PREPAREOUTPUT Prepare output remove output_oldest file, if exists in the SE; move the output_last file to output_oldest in the SE; tar the output in the localhost and copy it as output_last in the SE (all the comands are performed by using the clientSRM functionalities; file lifetime 7 days).

Action Set the “rstauto” file content to FALSE or TRUE depending on the match of some variables used as control variables. The fiel will be sent to the portal for further submission evaluation.

Action Calling the described functions

Installation&Configuration

ANSYS is an engineering simulation software (computer-aided engineering, or CAE) that offers a comprehensive range of engineering simulation solution sets providing access to virtually any field of engineering simulation that a design process requires. The next pages describe the procedure used for the installation and configuration. ANSYS version 13 has been downloaded from the website http://www.ansys.com/ after personal registration. ANSYS can be installed via GUI (recommended) or CLI. We describe both the procedires. No Server License Manager installation needed for INFN.

GUI (recommended)

1- Copy the file (ansys130_linx64_dvd.iso in our case) in the machine designed for the installation

2-

 #ssh –X < hostname > 

where < hostname > is the machine where ANSYS will be installed and configured. In this way the display export is enabled.

3- Mount the iso file (repeat if the iso files are more than one)

 #mkdir /mnt/iso 

 #mount -o loop -t iso9660 ansys130_linx64_dvd.iso /mnt/iso 

Additionally the iso files can be merged.

4- execute the installer and follow the instructions on the screen (the default option can be maintained)

 # ./INSTALL -debug -install_dir < ANSYS_DIR_PATH > 

Suggested value: ANSYS_DIR_PATH=/opt/exp_soft/gridit/ansys_inc

--Error1:

 Error:  The DISPLAY environment variable is not set. 
In order to run the GUI version of the installation, 
this must be set correctly. 
If you wish to use the silent install, please use the –silent 
install option. Please see documentation for usage instructions.

--Solution1: Set the DISPLAY variable as follow in the local_host

 DISPLAY=< DESTINATION_HOST >:0.0; export DISPLAY 

--Error2:

 Error while loading shared libraries: libXext.so.6: cannot open shared object file: No such file or directory 

--Solution2: Install libXext.so.6 32 bit and update LD_LIBRARY_PATH variable

5- Pro/ENGINEER window tick the boxes “Skip this step…” Do the same action for the Unigraphics NX (next window)

6- License Server Machine During the Installation the installer will ask you to set the license server(s) When requested add the following Hostname#1: ansys.infn.it Hostname#2: ansys2.infn.it Hostname#3: ansys3.infn.it

7- Probaly ANSYS will terminate with an error. Look at the file < ANSYS_DIR_PATH >/install.err If the error is something like the following, it is armless.

>>>--------------------------------------->
ANSYS installation: Mon Oct 1 10:19:57 2012
Revision: 13.0
    ERROR: /root/ansys_test/v130/aisol/.workbench  -cmd /root/ansys_test/v130/aisol/mw/bin-amd64_linux_optimized/regsvr32 -c -nodisplay /root/ansys_test/v130/aisol/dll/linx64/libans.cadint.pluginfemcom.so failed
>>>--------------------------------------->

CLI

1- Copy the file (ansys130_linx64_dvd.iso in our case) in the machine designed for the installation

2-

 #ssh < hostname > 

where < hostname > is the machine where ANSYS will be installed and configured.

3- Mount the iso file (repeat if the iso files are more than one)

 #mkdir /mnt/iso 

 #mount -o loop -t iso9660 ansys130_linx64_dvd.iso /mnt/iso 

NOTE: Merge the content if the iso files are more than one.

4- Install ANSYS

 # ./INSTALL -silent -debug ./INSTALL -silent -debug -licserverinfo 2325:1055:ansys.infn.it,ansys2.infn.it,ansys3.infn.it -install_dir < ANSYS_DIR_PATH > 

Suggested ANSYS_DIR_PATH=/opt/exp_soft/gridit/ansys_inc

--Error1: Error while loading shared libraries: libXext.so.6: cannot open shared object file: No such file or directory

--Solution1: Install libXext.so.6 32 bit and update LD_LIBRARY_PATH variable

4-Probaly ANSYS will terminate with an error. Look at the file < ANSYS_DIR_PATH >/install.err If the error is something like the following, it is armless.

>>>--------------------------------------->
ANSYS installation: Mon Oct 1 10:19:57 2012
Revision: 13.0
    ERROR: /root/ansys_test/v130/aisol/.workbench  -cmd /root/ansys_test/v130/aisol/mw/bin-amd64_linux_optimized/regsvr32 -c -nodisplay /root/ansys_test/v130/aisol/dll/linx64/libans.cadint.pluginfemcom.so failed
>>>--------------------------------------->

-- DanieleCesini - 2011-12-16 -- AlessandroCostantini - 2012-10-02