(r4) OpenFoam < UserSupport

Tags: , view all tags
---++ About this wiki

%TOC%

The following best practices document aims to provide some hints and examples on how to install and configure [[http://www.openfoam.com/][OpenFOAM-2.0.1]] released 04/08/11 on a grid based infrastructure.

---++ Application Description

OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package produced by a commercial company, OpenCFD Ltd.<br/>
It has a large user base across most areas of engineering and science, from both commercial and academic organizations. OpenFOAM has an extensive range of features<br/>
to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electro-magnetics.</br>
This wiki provides information on how to build OpenFOAM in a grid-based infrastructure.

---++ Pre-requisites

The instructions from this best practice have been tested on a:
   * Processor: Dual Core AMD Opteron(tm) Processor 275;
   * Memory: 4147300k
   * OS: Scientific Linux SL release 5.4 (Boron)
   * Compiler: GNU 4.4.0;
   * MPI: Open MPI 1.4.2.

---+++ Source Pack

The following tar-zipped gtgz source packs are available for download. 

| *Pack* | *File* | *md5sum* |
| OpenFOAM | [[http://downloads.sourceforge.net/foam/OpenFOAM-2.0.1.gtgz?use_mirror=mesh][OpenFOAM-2.0.1.gtgz]]  | 0a9dfa42282a5e629523b70e2544c773 |
| Third-Party  | [[http://downloads.sourceforge.net/foam/ThirdParty-2.0.1.gtgz?use_mirror=mesh][ThirdParty-2.0.1.gtgz]] | 4b91af77bdbd3a87d91eeccb0f596f59 |

---+++ Unpacking the source and installation

In a grid-based infrastructure we would recommend to install the software in the experiment software directory for the VO using sgm privileges and share this software<br/>
packages with other VO members. In this wiki we chosen to unpack this software under the <b>$VO_GRIDIT_SW_DIR</b> directory of the grid CE.
 
<PRE>
$ cd $VO_GRIDIT_SW_DIR
$ tar xzf OpenFOAM-2.0.1.gtgz
$ tar xzf ThirdParty-2.0.1.gtgz
$ chown -R root.root OpenFOAM-2.0.1/ ThirdParty-2.0.1/
</PRE>

The files unpack to produce directories OpenFOAM-2.0.1 and ThirdParty-2.0.1

<PRE>
$ ll OpenFOAM
total 12
drwxr-xr-x  11 sgmgridit001 sgmgridit   440 Nov 28 16:17 OpenFOAM-2.0.1
drwxr-xr-x  12 sgmgridit001 sgmgridit   768 Nov 28 15:57 ThirdParty-2.0.1
</PRE>

---+++ System Requirements

OpenFOAM-2.0.1 builds on many Linux distributions but the ParaView-3.10.1 version supplied in ThirdParty requires:<br/>
   * [[http://www.cmake.org/files/v2.8/cmake-2.8.6.tar.gz][cmake-2.8.6]] or higher
   * [[ftp://ftp.trolltech.com/qt/source/qt-everywhere-opensource-src-4.6.4.tar.gz][Qt-4.6.4]]
Both software packages can be freely downloaded for many Linux distributions.

Before to start building software, please check if the following packages have been installed on the server.

<PRE>
$ yum install bison 
$ yum install ncurses-devel 
$ yum install tix.x86_64 
$ yum install glibc-devel 
$ yum install flex 
$ yum install flex-devel 
$ yum install zlib-devel 
$ yum install libXt-devel 
$ yum install binutils-devel 
$ yum install gmp.x86_64 
$ yum install gmp-devel.x86_64 
$ yum install libstdc++44-devel.x86_64
</PRE>

Make sure you have GCC 4.4 installed on your server.

<PRE>
$ wget ftp://ftp.ntua.gr/pub/linux/scientificlinux/55/x86_64/SL/gcc44-4.4.0-6.el5.x86_64.rpm
$ rpm -ivh gcc44-4.4.0-6.el5.x86_64.rpm

$ wget ftp://ftp.ntua.gr/pub/linux/scientificlinux/55/x86_64/SL/gcc44-c++-4.4.0-6.el5.x86_64.rpm
$ rpm -ivh gcc44-c++-4.4.0-6.el5.x86_64.rpm
</PRE>

---++ Building Qt-4.6.4

First uncompress the archive in the preferred location, then unpack it:

<PRE>
$ cd /opt/exp_soft/gridit/OpenFOAM/ThirdParty-2.0.1
$ wget ftp://ftp.trolltech.com/qt/source/qt-everywhere-opensource-src-4.6.4.tar.gz
$ tar zxf qt-everywhere-opensource-src-4.6.4.tar.gz
$ chown -R root.root qt-everywhere-opensource-src-4.6.4
$ cd qt-everywhere-opensource-src-4.6.4
$ ./configure
$ gmake && gmake install
</PRE>

---++ Building cmake-2.8.6

<PRE>
$ cd /opt/exp_soft/gridit/OpenFOAM/ThirdParty-2.0.1
$ wget  http://www.cmake.org/files/v2.8/cmake-2.8.6.tar.gz
$ tar zxvf cmake-2.8.6.tar.gz
$ cd cmake-2.8.6
$ ./configure
$ gmake
</PRE>

---++ Makefile changes for GNU compilers

Once you have successfully installed GCC 4.4.0, then changes these files:

<PRE>
$ cd /opt/exp_soft/gridit/OpenFOAM/OpenFOAM-2.0.1/wmake/rules/linux64Gcc44/

$ cat wmake/rules/linux64Gcc44/c
.SUFFIXES: .c .h
cWARN        = -Wall
cc          = gcc44 -m64   <b><== Change Here!</b>
include $(RULES)/c$(WM_COMPILE_OPTION)
cFLAGS      = $(GFLAGS) $(cWARN) $(cOPT) $(cDBUG) $(LIB_HEADER_DIRS) -fPIC
ctoo        = $(WM_SCHEDULER) $(cc) $(cFLAGS) -c $$SOURCE -o $@
LINK_LIBS   = $(cDBUG)
LINKLIBSO   = $(cc) -shared
LINKEXE     = $(cc) -Xlinker --add-needed -Xlinker -z -Xlinker nodefs


$ cat wmake/rules/linux64Gcc44/c++
.SUFFIXES: .C .cxx .cc .cpp
c++WARN     = -Wall -Wextra -Wno-unused-parameter -Wold-style-cast
CC          = g++44 -m64   <b><== Change Here!</b>
include $(RULES)/c++$(WM_COMPILE_OPTION)
ptFLAGS     = -DNoRepository -ftemplate-depth-100
c++FLAGS    = $(GFLAGS) $(c++WARN) $(c++OPT) $(c++DBUG) $(ptFLAGS) $(LIB_HEADER_DIRS) -fPIC
Ctoo        = $(WM_SCHEDULER) $(CC) $(c++FLAGS) -c $$SOURCE -o $@
cxxtoo      = $(Ctoo)
cctoo       = $(Ctoo)
cpptoo      = $(Ctoo)
LINK_LIBS   = $(c++DBUG)
LINKLIBSO   = $(CC) $(c++FLAGS) -shared
LINKEXE     = $(CC) $(c++FLAGS) -Xlinker --add-needed
</PRE>

---++ Pre-setup requirements

<PRE>
$ cd /opt/exp_soft/gridit/OpenFOAM/OpenFOAM-2.0.1/etc
</PRE>

Configure the OpenFOAM <b>bashrc</b> file as follows:

   * # Location of the OpenFOAM installation
    <PRE>foamInstall=/opt/exp_soft/gridit/$WM_PROJECT</PRE>

   * #- Compiler:
     #   WM_COMPILER = Gcc | Gcc43 | Gcc44 | Gcc45 | Gcc46 | Clang | Icc (Intel icc)
    <PRE>export WM_COMPILER=Gcc44</PRE>

   * # Location of installation
     <PRE>export WM_PROJECT_INST_DIR=$FOAM_INST_DIR</PRE>
     <PRE>export WM_PROJECT_DIR=$WM_PROJECT_INST_DIR/$WM_PROJECT-$WM_PROJECT_VERSION</PRE>

   * # Location of third-party software
     <PRE>export WM_THIRD_PARTY_DIR=$WM_PROJECT_INST_DIR/ThirdParty-$WM_PROJECT_VERSION</PRE>

   * # Location of user file
     <PRE>export WM_PROJECT_USER_DIR=$WM_PROJECT_DIR</PRE>

---++ Checking the System

To check your system is ready to build the sources, execute the <b>foamSystemCheck</b> script (in the OpenFOAM-2.0.1/bin directory). <br/>
If any critical software is missing, or needs updating to a newer version, please contact the system administrator to install the required <br/>
software before proceeding to the build. 

<PRE>
$ cd /opt/exp_soft/gridit/OpenFOAM/OpenFOAM-2.0.1/bin
$ ./foamSystemCheck 

Checking basic system...
-----------------------------------------------------------------------
Shell:           /bin/bash
Host:            grid012.ct.infn.it
OS:              Linux version 2.6.9-89.0.11.ELsmp
User:            root

System check: PASS
==================
Continue OpenFOAM installation.
</PRE>

---++ Building Sources

Go to the top-level source directory $WM_PROJECT_DIR and execute the top-level build script <b>./Allwmake</b><br/>
In principle this will build everything, but if problems occur with the build order it may be necessary to update the environment variables and re-execute.

If you experience difficulties with building the source-pack, or your platform is not currently supported, please contact 
[[http://www.openfoam.com/support/software.php][software support]] to arrange a <br/> support contract and we will do the port and maintain it for future releases. 

<PRE>
$ cd /opt/exp_soft/gridit/OpenFOAM/OpenFOAM-2.0.1/
$ source  etc/bashrc
$ ./Allwmake >log 2>log &
</PRE>

---++ Compiling Paraview 3.10.1 and the PV3FoamReader Module

Paraview is the third-party software that we provide for graphical post-processing in OpenFOAM. It’s compilation is automated using a script called <b>makeParaView</b></br>
in the ThirdParty-2.0.1 directory. 
Installation of Paraview 3.10.1 requires a version of QT that is 3.6.2 or newer and cmake which is 2.8.2 or newer, so again make <br/> sure that this is on your system. 

<PRE>
$ cd $WM_THIRD_PARTY_DIR
</PRE>

Configure the _QMAKE_PATH_ and _CMAKE_PATH_ variable settings in the <b>makeParaView</b> file

   * # Set the path to the Qt-4.5 (or later) qmake if the system Qt is older
   <PRE>QMAKE_PATH="/opt/exp_soft/gridit/OpenFOAM/ThirdParty-2.0.1/qt-everywhere-opensource-src-4.6.4/bin/"</PRE>

   * # Set the path to cmake
   <PRE>CMAKE_PATH="/opt/exp_soft/gridit/OpenFOAM/ThirdParty-2.0.1/cmake-2.8.6/bin/"</PRE>

To install Paraview, execute the following: 

<PRE>
$ ./makeParaView
$ ./makeParaView >log_paraview 2>log_paraview &
</PRE>

The PV3blockMeshReader and the PV3FoamReader ParaView plugins are compiled as usual for OpenFOAM utilities: 

<PRE>
$ cd $FOAM_UTILITIES/postProcessing/graphics/PV3Readers
$ wmSET
$ ./Allwclean
$. /Allwmake 
</PRE>

---++ Testing the Installation

To check your installation setup, execute the <b>foamInstallationTest</b> script (in the OpenFOAM-2.0.1/bin directory). <br/>
If no problems are reported, proceed to getting started with OpenFOAM; otherwise, go back and check you have installed the software correctly<br/>
and/or contact your system administrator. 

---+++ Getting Started

Create a project directory within the $HOME/OpenFOAM directory named <USER>-2.0.1 and create a directory named run within it, e.g. by typing: 

<PRE>
$ mkdir -p $HOME/OpenFOAM/OpenFOAM-2.0.1/$FOAM_RUN 
</PRE>

Copy the tutorial examples directory in the OpenFOAM distribution to the run directory. If the OpenFOAM environment variables are set correctly, <br/>
then the following command will be correct: 

<PRE>
$ cp -r $FOAM_TUTORIALS $HOME/OpenFOAM/OpenFOAM-2.0.1/$FOAM_RUN 
</PRE>

Run the first example case of incompressible laminar flow in a cavity: 

<PRE>
$ cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity
$ blockMesh
$ icoFoam
$ paraFoam 
</PRE>

---++ Running OpenFOAM in Parallel

---+++ Create a new use case
The results from the previous example are generated using a fairly coarse mesh. 
In this new case we will demonstrate the parallel processing capability of OpenFOAM access to multiple processors.

<PRE>
$ cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity/system
</PRE>

<PRE>
$ cat decomposeParDict 
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.6                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 4; <b><== Change Here!</b>
method          scotch;
simpleCoeffs
{
    n               ( 2 2 1 );
    delta           0.001;
}
hierarchicalCoeffs
{
    n               ( 1 1 1 );
    delta           0.001;
    order           xyz;
}
scotchCoeffs
{
    processorWeights (
                        1 <b><== Add Here!</b>
                        1 <b><== Add Here!</b>
                        1 <b><== Add Here!</b>
                        1 <b><== Add Here!</b>
                     );
}
manualCoeffs
{
    dataFile        "";
}
distributed     no;
roots           ( );
// ************************************************************************* //
</PRE>

Create a new tar of the test case:
<PRE>
$ tar zcvf cavity.tar.gz cavity
cavity/
cavity/system/
cavity/system/fvSchemes
cavity/system/fvSolution
cavity/system/controlDict
cavity/system/decomposeParDict
cavity/constant/
cavity/constant/transportProperties
cavity/constant/polyMesh/
cavity/constant/polyMesh/boundary
cavity/constant/polyMesh/blockMeshDict
cavity/0/
cavity/0/U
cavity/0/p
</PRE>

---++ Testing OpenFOAM in a Grid Infrastructure

This section provides some hints for testing OpenFOAM job on the GRIDIT Infrastructure.

---++ Post-configuration on the LSF master node and client nodes

Add in the _/etc/bashrc_ file the following settings for sourcing the OpenFoam bashrc profile
<PRE>
. /opt/exp_soft/gridit/OpenFOAM/OpenFOAM-2.0.1/etc/bashrc
</PRE>

---++ Creation of a new MPI wrapper

Add in _/opt/i2g/etc/mpi-start/_ the definition of a new MPI wrapper for OpenFoam-2.0.1.<br/>
This wrapper has to be replicated in all the LSF client nodes.

<PRE>
$ cat /opt/i2g/etc/mpi-start/openmpi_openfoam.mpi 
#!/bin/sh
#
# Copyright (c) 2006-2007 High Performance Computing Center Stuttgart,
#                         University of Stuttgart.  All rights reserved.
#           (c) 2009      Instituto de Fisica de Cantabria - CSIC. 
#

# specifies where Open MPI is installed

export MPI_OPENMPI_PATH=/opt/exp_soft/gridit/OpenFOAM/ThirdParty-2.0.1/platforms/linux64Gcc44/openmpi-1.5.3/
export MPI_OPENMPI_MPIEXEC=/opt/exp_soft/gridit/OpenFOAM/ThirdParty-2.0.1/platforms/linux64Gcc44/openmpi-1.5.3/bin/mpiexec

if test "x$I2G_OPENMPI_PREFIX" = "x" ; then
    if test "x$MPI_OPENMPI_PATH" = "x" ; then
        if test "x$MPI_START_MPI_PREFIX" != "x"  ; then    
            export I2G_OPENMPI_PREFIX=$MPI_START_MPI_PREFIX
        else 
            export I2G_OPENMPI_PREFIX=/opt/i2g/openmpi
            debug_msg "use default installtion : $I2G_OPENMPI_PREFIX"
        fi
    else
        export I2G_OPENMPI_PREFIX=$MPI_OPENMPI_PATH
        debug_msg "use user provided prefix : $MPI_OPENMPI_PATH"
    fi
else 
    debug_msg "use user provided prefix : $I2G_OPENMPI_PREFIX"
fi

# activate MPI
mpi_start_activate_mpi $I2G_OPENMPI_PREFIX "$MPI_START_MPI_MODULE"

# add necessary PATH to the environment variables
#debug_msg "prepend Open MPI to PATH and LD_LIBRARY_PATH"
export PATH=$I2G_OPENMPI_PREFIX/bin:$PATH
export LD_LIBRARY_PATH=$I2G_OPENMPI_PREFIX/lib:$LD_LIBRARY_PATH

if test "x$I2G_MPI_TYPE" != "xopenmpi" ; then
    # we are not the primary MPI 
    # fall back to save settings that should work always
    debug_msg ""
    debug_msg "disable PBS, SGE"
    OPENMPI_PARAMS="-mca pls ^tm,gridengine -mca ras ^tm,gridengine  "
    #OPENMPI_PARAMS="$OPENMPI_PARAMS -x PACX_DEBUG_NODE=$PACX_DEBUG_NODE"
    debug_msg "export GLOBUS_TCP_PORT_RANGE : $GLOBUS_TCP_PORT_RANGE"
    OPENMPI_PARAMS="$OPENMPI_PARAMS -x GLOBUS_TCP_PORT_RANGE "
fi

#
# start an mpi job
#
mpi_exec () {

    #handle Open MPI 1.2.2 + PBS bug
    if test "x$PBS_NODEFILE" = "x" ; then
        debug_msg "found openmpi and a non-PBS batch system, set machinefile and np parameters"
        export I2G_MACHINEFILE_AND_NP="-machinefile $MPI_START_MACHINEFILE -np $I2G_MPI_NP"
    else
        debug_msg "found openmpi and PBS, don't set machinefile"
        export I2G_MACHINEFILE_AND_NP="-np $I2G_MPI_NP"
    fi

    #set the parameters to be always used with Open MPI:
    MPI_SPECIFIC_PARAMS="-wdir $PWD "

    #check for Marmot
    if test "x$I2G_USE_MARMOT" = "x1" ; then
        debug_msg "export LD_PRELOAD for Open MPI"
        MPI_SPECIFIC_PARAMS="$MPI_SPECIFIC_PARAMS -x LD_PRELOAD -x MARMOT_MAX_TIMEOUT_DEADLOCK -x MARMOT_LOGFILE_PATH"
    fi

    #if test "x$I2G_USE_MPITRACE" ="x1" ; then
        #MPI_SPECIFIC_PARAMS="-x MPITRACE_ON -x MPTRACE_DIR"
    #fi

    # check for user supplied mpiexec 
    MPIEXEC=`which mpiexec`
    if test "x$MPI_OPENMPI_MPIEXEC" != "x"  ; then
        MPIEXEC="$MPI_OPENMPI_MPIEXEC $I2G_MPI_MPIEXEC_PARAMS"
        debug_msg "using user supplied startup : '$MPIEXEC'"
        MPI_SPECIFIC_PARAMS="$MPI_SPECIFIC_PARAMS -x X509_USER_PROXY --prefix $I2G_OPENMPI_PREFIX $OPENMPI_PARAMS"
        . $MPI_START_PREFIX/../etc/mpi-start/generic_mpiexec.sh
        generic_mpiexec
    elif test "x$MPI_OPENMPI_MPIRUN" != "x" ; then
        MPIEXEC="$MPI_OPENMPI_MPIRUN $MPI_OPENMPI_MPIRUN_PARAMS"
        debug_msg "using user supplied startup : '$MPIEXEC'"
        MPI_SPECIFIC_PARAMS="$MPI_SPECIFIC_PARAMS -x X509_USER_PROXY --prefix $I2G_OPENMPI_PREFIX $OPENMPI_PARAMS"
        . $MPI_START_PREFIX/../etc/mpi-start/generic_mpiexec.sh
        generic_mpiexec
    else
        MPI_SPECIFIC_PARAMS="$MPI_SPECIFIC_PARAMS -x X509_USER_PROXY --prefix $I2G_OPENMPI_PREFIX $OPENMPI_PARAMS"
        . $MPI_START_PREFIX/../etc/mpi-start/generic_mpiexec.sh
        generic_mpiexec
    fi
    return $?
}

mpi_start () {
    . $MPI_START_PREFIX/../etc/mpi-start/generic_mpi_start.sh
    generic_mpi_start
    return $?
}
</PRE>

---++ JDL & script files (sequential mode)

This is an example of JDL file that can be used for testing OpenFoam in *sequential mode*:

<PRE>
[
     Type = "Job";
     JobType = "Normal";

     Executable = "/bin/bash";
     Arguments = "start_openfoam.sh";

     StdOutput = "log.out";
     StdError = "log.err";

     InputSandbox = {"start_openfoam.sh"};
     OutputSandbox = {"log.err","log.out","openfoam.log","openfoam.err"};
     Requirements = Member("<b>VO-gridit-OpenFoam-2.0.1</b>",other.GlueHostApplicationSoftwareRunTimeEnvironment);
     Rank = (other.GlueCEStateWaitingJobs == 0 ? other.GlueCEStateFreeCPUs : -other.GlueCEStateWaitingJobs);
]
</PRE>

This is the bash script sent in InputSandbox with the JDL file:

<PRE>
#!/bin/sh
echo "+ Running OpenFoam-2.0.1 on "`hostname -f` as `whoami`
echo;echo "+ Copying the OpenFoam example..."
cp -R $FOAM_TUTORIALS/incompressible/icoFoam/cavity $PWD
cd $PWD/cavity
echo; echo "+ Starting at "`date`

blockMesh
icoFoam >./openfoam.log 2>./openfoam.err
cp openfoam.* ../

# Testing the scratch area
cp openfoam.* $VO_GRIDIT_SW_DIR/scratch
chmod a+w $VO_GRIDIT_SW_DIR/scratch/openfoam.*
echo "+ Done at "`date`
</PRE>

---+++ Submission

<PRE>
$ glite-wms-job-submit -a -r grid012.ct.infn.it:2119/jobmanager-lcglsf-short 
                                        -e https://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server 
                                        -o jobID 
                                         openfoam-2.0.1.jdl 

Connecting to the service https://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server

===================== glite-wms-job-submit Success =====================
The job has been successfully submitted to the WMProxy
Your job identifier is:
https://lb-4.dir.garr.it:9000/vUAJk3Sq3xBnviahz7HSqg
The job identifier has been saved in the following file:
/home/larocca/OpenFoam-2.0.1/jobID
=====================================================================
</PRE>

---+++ Checking Status

<PRE>
$ glite-wms-job-status --noint -i jobID

====================== glite-wms-job-status Success ====================
BOOKKEEPING INFORMATION:
Status info for the Job : https://lb-4.dir.garr.it:9000/vUAJk3Sq3xBnviahz7HSqg
Current Status:     Done (Success)
Exit code:              0
Status Reason:     Job terminated successfully
Destination:           grid012.ct.infn.it:2119/jobmanager-lcglsf-short
Submitted:             Fri Dec  2 16:59:56 2011 CET
=====================================================================
</PRE>

---+++ Downloading Results

<PRE>

$ glite-wms-job-output --dir . --noint -i jobID

Connecting to the service https://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server
===============================================================================
			JOB GET OUTPUT OUTCOME
Output sandbox files for the job:
https://lb-4.dir.garr.it:9000/vUAJk3Sq3xBnviahz7HSqg
have been successfully retrieved and stored in the directory:
/home/larocca/OpenFoam-2.0.1/larocca_vUAJk3Sq3xBnviahz7HSqg
================================================================================
</PRE>

---++ JDL & script files (parallel mode)

This is an example of JDL file that can be used for testing OpenFoam in *parallel mode*:

<PRE>
[
JobType        = "NORMAL";
CPUNumber      = <b>4;</b>
Executable     = "mpi-start-wrapper.sh";
Arguments      = "icoFoam <b>OPENMPI_OPENFOAM</b>";
StdOutput      = "mpi-start.out";
StdError       = "mpi-start.err";
InputSandbox   = {"mpi-start-wrapper.sh","mpi-hooks.sh","cavity.tar.gz"};
OutputSandbox  = {"mpi-start.err","mpi-start.out","result"};
Requirements   = Member("<b>VO-gridit-OpenFoam-2.0.1</b>",other.GlueHostApplicationSoftwareRunTimeEnvironment);
]
</PRE>

MPI wrappers are shown below:

<PRE>
$ cat <b>mpi-start-wrapper.sh</b> 
#!/bin/bash

# Pull in the arguments.
MY_EXECUTABLE=$1
MPI_FLAVOR=$2

# Convert flavor to lowercase for passing to mpi-start.
MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'`

# Pull out the correct paths for the requested flavor.
eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH`

# Ensure the prefix is correctly set.  Don't rely on the defaults.
#eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH
#export I2G_${MPI_FLAVOR}_PREFIX

# Touch the executable.  It exist must for the shared file system check.
# If it does not, then mpi-start may try to distribute the executable
# when it shouldn't.
touch $MY_EXECUTABLE

# Setup for mpi-start.
export I2G_MPI_APPLICATION=$MY_EXECUTABLE
export I2G_MPI_APPLICATION_ARGS="-parallel"
export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER
export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh
export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh
export I2G_MPI_FILE_DIST="ssh"

# If these are set then you will get more debugging information.
export I2G_MPI_START_VERBOSE=1
export I2G_MPI_START_DEBUG=1

# Invoke mpi-start.
$I2G_MPI_START
</PRE>

<br/>
<PRE>
$ cat <b>mpi-hooks.sh</b> 
#!/bin/sh

#
# This function will be called before the MPI executable is started.
# You can, for example, compile the executable itself.
#
pre_run_hook () {

  # Compile the program.
  #echo "Compiling ${I2G_MPI_APPLICATION}"

  #debug_msg "LIST : $LSB_HOSTS"
  # Actually compile the program.

  #cmd="mpicc ${MPI_MPICC_OPTS} -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c"
  #echo $cmd
  #$cmd

  #if [ ! $? -eq 0 ]; then
  #  echo "Error compiling program.  Exiting..."
  #  exit 1
  #fi
  #chmod +x ${I2G_MPI_APPLICATION}

  # Everything's OK.
  #echo "Successfully compiled ${I2G_MPI_APPLICATION}"
  <b>
  # PRE OPENFOAM
  echo
  echo "========================================"
  echo "******This is my working directory******" 
  echo "========================================"
  tar zxvf cavity.tar.gz
  cd cavity
  blockMesh
  decomposePar
  echo "========================================"
  </b>
  return 0
}

#
# This function will be called before the MPI executable is finished.
# A typical case for this is to upload the results to a storage element.
#
post_run_hook () {
  echo "Executing post hook."
  echo "Finished the post hook."
  return 0
}
</PRE>


---++ References
[[http://www.openfoam.com/][OpenFOAM: The open source CFD toolbox]]