''' !!! GENERAL INSTRUCTION - NOT OPTIMIZED FOR PRODUCTION    !!! '''\\
For production runs please check the installed OpenFOAM on JURECA with
{{{ #!sh
module spider OpenFOAM
}}}

= OpenFOAM - Build Instructions for JURECA
[[PageOutline]]


'''OpenFOAM''' is free, open-source software for computational fluid dynamics.
* [http://www.openfoam.org OpenFOAM Foundation]
* [http://www.openfoam.com OpenCFD Ltd] - owner of the trademark OpenFOAM

OpenFOAM releases are scheduled every six months in June and December.
Releases designated OpenFOAM+ contain client-sponsored developments of which much has been transferred to, but not released in the OpenFOAM Foundation branch.

== 1. Preliminaries ==
* General build instructions can be found here:
  * http://www.openfoam.com/code/build-guide.php
  * http://www.openfoam.org/download/git.php

* There are, roughly, two components to OpenFOAM:
  * the actual CFD code FOAM -- definitely to be compiled with ICC
  * the GUI analysis tools, based on !ParaView (using the Qt GUI toolkit)

* All OpenFOAM applications do parallel communication through the Pstream library only.\\ So using any underlying message passing library (MPI, etc.) is just a matter of replacing Pstream. [https://openfoamwiki.net/index.php/Installation/HowTo_Pstream more...]

* Some of the !ThirdParty packages need to be built first. These third party sources are required, but are not bundled. The source tarballs could be expanded in the !ThirdParty-${VERSION} directory, but we use installed modules instead.
  * Boost - 1.61.0 works
  * CGAL - 4.8.2 works
    * GMP - 6.1.2 works
    * MPFR - 3.1.5 works
  * METIS - 5.1.0 is expected [http://glaros.dtc.umn.edu/gkhome/metis/metis/download METIS Official Website]

{{{ #!sh
### set env. variables we will use

VERSION=3.0.1

# $INST_DIR must NOT (!!) end with a '/', or build will fail with "touch [...] permission denied"
INST_DIR=<my_openfoam_install_dir> # e.g. ${HOME}/OpenFOAM_install

mkdir -p $INST_DIR
}}}

== 2. Download & Extract ==
Source packages are available from the download section of the OpenFOAM website:
     * http://www.openfoam.org/download/source.php
     * https://github.com/OpenFOAM

The source code must be unpacked into the top-level directory of the final installation.\\For this !HowTo this is $INST_DIR.

{{{ #!sh
### download OpenFOAM
cd $INST_DIR

wget https://downloads.sourceforge.net/project/foam/foam/${VERSION}/OpenFOAM-${VERSION}.tgz
wget https://downloads.sourceforge.net/project/foam/foam/${VERSION}/ThirdParty-${VERSION}.tgz

# extract OpenFOAM
tar -xzf OpenFOAM-${VERSION}.tgz -C $INST_DIR
tar -xzf ThirdParty-${VERSION}.tgz -C $INST_DIR
}}}

== 3. Setup OpenFOAM ==

=== Patch ===

Patch $INST_DIR/OpenFOAM-${VERSION}/etc/bashrc
{{{ #!sh
### bashrc

+ foamInstall=${INST_DIR}
+ : ${WM_COMPILER:=Gcc}; export WM_COMPILER
+ : ${WM_MPLIB:=SYSTEMOPENMPI}; export WM_MPLIB
}}}
* Instead of setting variable in bashrc, they can be set via environment variables before you call bashrc later.
  * {{{export FOAM_INST_DIR=...}}} overwrites foamInstall
  * {{{export WM_COMPILER=...}}}, default is Gcc, should be set to **Icc**
  * {{{export WM_MPLIB=...}}}, default is SYSTEMOPENMPI, should be set to **EASYBUILDMPI**

Patch $INST_DIR/OpenFOAM-${VERSION}/etc/config/settings.sh
{{{ #!sh
### settings.sh
Linux)
...
    x86_64)
...
        64)
            WM_ARCH=linux64
            export WM_COMPILER_LIB_ARCH=64
+           export WM_CC=${CC}                      # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_CXX=${CXX}                    # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_CFLAGS="${CFLAGS} -fPIC"      # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_CXXFLAGS="${CXXFLAGS} -fPIC"  # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_LDFLAGS=                      # for use in non-wmake (e.g. ThirdParty) compilations

}}}
{{{ #!sh
### settings.sh

+ #unset MPI_ARCH_PATH MPI_HOME FOAM_MPI_LIBBIN

+ EASYBUILDMPI)
+    export FOAM_MPI=mpi
+    ;;
}}}

Patch $INST_DIR/OpenFOAM-${VERSION}/wmake/rules
{{{ #!sh
### linux64Gcc/c, linux64Icc/c
+ cc        = ${MPICC}

### linux64Gcc/c++
+ CC        = ${MPICXX} -std=c++0x 

### linux64Icc/c++
+ CC        = ${MPICXX} -std=c++0x -fp-trap=common -fp-model precise

### linux64Gcc/cOpt, linux64Icc/cOpt
+ cOPT      = ${CFLAGS}
                                 
### linux64Gcc/c++Opt, linux64Icc/c++Opt
+ c++OPT    = ${CXXFLAGS}
}}}
* examples: https://github.com/OpenFOAM/OpenFOAM-dev/tree/master/wmake/rules
* MPI: ${MPICC} required to compile finiteVolume/lnInclude/fvPatchField.C

=== Set Environment ===

The environment must be set up according to where the source code directories are located.
{{{ #!sh
# set environment vars, if not set yet
# $INST_DIR must NOT (!!) end with a '/', or build will fail with "touch [...] permission denied"
#cd <my_openfoam_install_dir>
#export INST_DIR=$(pwd)
#export VERSION=3.0.1
cd $INST_DIR

# load modules
module --force purge
module use /usr/local/software/jureca/OtherStages
module load Stages/Devel
module load Intel/2017.0.098-GCC-5.4.0
module load IntelMPI/2017.0.098
module load flex/2.5.39 # flex 2.6.0 causes errors
module load Boost/1.61.0-Intel-2017.0.098
module load MPFR/3.1.5-Intel-2017.0.098
module load GMP/6.1.2-Intel-2017.0.098
module load CGAL/4.8.2-Intel-2017.0.098
module load METIS/5.1.0-Intel-2017.0.098

# configure environment
export FOAM_INST_DIR=$INST_DIR
export WM_COMPILER=Icc
export WM_MPLIB=EASYBUILDMPI
export MPICC=mpicc
export MPICXX=mpiicpc
export CC=icc
export CXX=icpc
export CFLAGS="-no-prec-div -O3 -xHost -fp-model precise -diag-disable 654,1125"
export CXXFLAGS="-O3 -xHost -fp-model precise -diag-disable 654,1125 -DMPICH_SKIP_MPICXX"

export WM_NCOMPPROCS=8  # parallel build
export WM_LABEL_SIZE=32 # 32bit for labels/indices - relevant for METIS (if 64 needed patch METIS first)

# load environment
source OpenFOAM-${VERSION}/etc/bashrc
foam
foamSystemCheck
}}}

* build with Intel 2017.1.138: [[OpenFOAM/config_Intel-2017.1.138]]
* build with Stage/2016a [[OpenFOAM/config_2016a]]
* build with Score-P [[OpenFOAM/config_ScoreP]]
* build for !Intel/Xeon Phi (Knights Landing) [[OpenFOAM/config_knl]]

== 4. Setup !ThirdParty ==

OpenFOAM has configs which set where it expects to find its "third party" packages.
There are in {{{$FOAM_ETC/config/}}} as various shell scripts.

The top-level ./Allwmake will build Scotch, but not CGAL, Boost, GMP, and MPFR.\\Those packages need to be built separately, and installed into the appropriate directory.

=== CGAL ===
Patch $INST_DIR/OpenFOAM-${VERSION}/etc/config/CGAL.sh
{{{ #!sh
### CGAL.sh

+ boost_version=$EBVERSIONBOOST
+ cgal_version=$EBVERSIONCGAL

+ export BOOST_ARCH_PATH=$EBROOTBOOST
+ export CGAL_ARCH_PATH=$EBROOTCGAL
}}}
* Boost -installed as module Boost/1.61.0-Intel-2017.0.098
* GMP - installed as module GMP/6.1.2-Intel-2017.0.098
* MPFR - installed as module MPFR/3.1.5-Intel-2017.0.098

=== METIS ===
Patch $INST_DIR/OpenFOAM-${VERSION}/etc/config/metis.sh
{{{ #!sh
# metis.sh

+ export METIS_ARCH_PATH=$EBROOTMETIS
}}}

Patch $WM_THIRD_PARTY_DIR/Allwmake
{{{ #!sh
# $WM_THIRD_PARTY_DIR/Allwmake

+ if [ -d "$METIS_ARCH_PATH" ] # if [ -d "$METIS_VERSION" ]
+ then
+    if [ -f $METIS_ARCH_PATH/include/metis.h \
+      -a -r $METIS_ARCH_PATH/lib/libmetis.so ]
+    then
+        cp $METIS_ARCH_PATH/lib/libmetis.so $FOAM_EXT_LIBBIN
+        echo "    metis header in $METIS_ARCH_PATH/include"
+        echo "    metis libs   in $FOAM_EXT_LIBBIN"
}}}
* ATTENTION: metis.h
  * METIS library's IDXTYPEWIDTH  must fit to $WM_LABEL_SIZE       (for METIS/5.1.0-Intel-2017.0.098: IDXTYPEWIDTH=32)
  * METIS library's REALTYPEWIDTH must fit to $WM_PRECISION_OPTION (for METIS/5.1.0-Intel-2017.0.098: REALTYPEWIDTH=64)


=== Scotch ===
Scotch decomposition which requires no geometric input from the user and attempts to minimise the number of processor boundaries.

Patch $WM_THIRD_PARTY_DIR/etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32
{{{ #!sh
# scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32
+ AR             = ${CC}
+ CCS            = ${CC}
+ CCP            = ${MPICC}
+ CCD            = ${MPICC}
+ CFLAGS         = -O3 -xHost -fp-model precise -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict
}}}


== 5. Build & Test OpenFOAM ==

=== Build OpenFOAM

Prepare for build
{{{ #!sh

# cd $WM_PROJECT_DIR
foam

# clean if this is a rebuild
wcleanAll ; wcleanLnIncludeAll ; wcleanMachine
$WM_THIRD_PARTY_DIR/Allclean
}}}

Copy third party source to required location.
{{{ #!sh

source $INST_DIR/OpenFOAM-${VERSION}/etc/config/scotch.sh 

mkdir -p $SCOTCH_ARCH_PATH
cp -a $WM_THIRD_PARTY_DIR/scotch_6.0.3/* $SCOTCH_ARCH_PATH
}}}

Build
{{{ #!sh
# build wmake
./wmake/makeWmake

# build OpenFOAM (except Qt & ParaView) 
./Allwmake
}}}

=== Test OpenFOAM
Validate the build by running

{{{ #!sh
# cd $WM_PROJECT_DIR
foam
foamInstallationTest

# create the user run directory
mkdir -p $FOAM_RUN

# test with a simple tutorial:
run
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily ./
cd pitzDaily
blockMesh
simpleFoam

# test with a more advanced tutorial
# details here: https://cfd.direct/openfoam/user-guide/cavity#x5-40002.1
run
cp -r $FOAM_TUTORIALS/incompressible/icoFoam/cavity ./
cd cavity
foamRunTutorials

#paraFoam 
}}}

* More detailed tutorials to test can be found here:
  * https://cfd.direct/openfoam/user-guide/tutorials/

== 6. Build Tools ==

=== ParaView ===
OpenFOAM is shipped with !ParaView for post-processing OpenFOAM field results.\\This requires a separate compilation which can be omitted if using other post-processing software.
{{{ #!sh
cd $WM_THIRD_PARTY_DIR
./makeParaView
}}}

== References ==
* flex 2.6.0
  * 2.6.0 causes errors: undefined reference to `yyFlexLexer::yywrap()'
  * https://www.cfd-online.com/Forums/openfoam-installation/163986-building-openfoam-3-0-archlinux.html
* -fp-model precise
  * Disables optimizations that are not value-safe on floating-point data.
  * https://www.nccs.nasa.gov/images/FloatingPoint_consistency.pdf
* -DMPICH_SKIP_MPICXX
  * https://www.cfd-online.com/Forums/openfoam/90965-errors-building-openfoam-2-0-0-icc-sgi-mpt-2-03-a.html#post317630 
  * https://sourceforge.net/p/openfoam-extend/foam-extend-3.1/ci/90bbdada8b322e227528311326783a80a1e26b35/#diff-4
* -diag-disable 654
  * hide warning !#654: 
* -diag-disable 1125
  * hide warning !#1125: function "..." is hidden by "..." -- virtual function override intended?