''' !!! WORK IN PROGRESS    !!! '''\\
Last Update 05.2017, Authors: Jens Henrik Göbbert, ...

== OpenFOAM Settings for use with Score-P

* This includes '''additional''' information on how to build OpenFOAM for use with Score-P.
  1. follow [https://trac.version.fz-juelich.de/vis/wiki/Software/OpenFOAM], but do NOT build OpenFOAM, yet.
  2. follow the additional instructions from this page 
  3. follow the build instructions from [https://trac.version.fz-juelich.de/vis/wiki/Software/OpenFOAM#a5.BuildTestOpenFOAM]

'''ATTENTION:'''\\
* Score-P can either support generating serial code or parallel/mpi code (default=parallel) - but '''never both at the same time'''.\\If you run an application instrumented by Score-P with parallel support in serial, Score-P will not be able to dump its results to disk but will report an error at application end: {{{SCOREP_IpcGroup_GetSize: Assertion ‘SCOREP_Status_IsMppInitialized()’ failed.”}}}\\ (this error also occurs, if MPI_finalize() is not called at the end of an MPI program instrumented with Score-P).\\ For an OpenFOAM installation for serial support uncomment the following in your environment setup script: {{{export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--mpp=none ${SCOREP_WRAPPER_INSTRUMENTER_FLAGS}"}}}
* OpenFOAM overwrites malloc.\\This clashes with the Score-P memory measurement system that  wraps malloc.\\ Hence, it is recommended to disable the memory wrapping of Score-P by adding {{{--nomemory}}} to the instrumenter flags.\\Furthermore, it is recommended to make sure the malloc function from OpenFOAM is not instrumented, e.g. automatically by compiler instrumentation.The particular source file is: {{{src/OSspecific/POSIX/signals/sigFpe.C}}}
* Use the GCC-compilers.\\GCC-compilers enable Score-P to distinguish between inlined and non-inlined functions.The GCC-compilers are free to inline any function and avoid to instrument these parts of the code. Especially for C++ code this should/must be considered.
* OpenFOAM communicates through the PStream library which hides the used MPI infrastructure.\\For some OpenFOAM apps it seems to be possible, that during linking the executable, neither the MPI library nor the MPI compiler wrapper appear at the link line. In these cases the Score-P MPI library wrapping fails even though a ldd analysis of the executable shows that the PStream library contains the Score-P MPI wrapper library as a dependency. How to get around this issue needs further investigation.

=== Set Environment ===

The environment must be set up according to where the source code directories are located.
{{{ #!sh
cd <my_openfoam_install_dir> 
export INST_DIR=$(pwd) # $INST_DIR must NOT (!!) end with a '/', or build will fail with "touch [...] permission denied"
cd $INST_DIR

# load modules (GCC)
module --force purge
module use /usr/local/software/jureca/OtherStages
module load Stages/Devel
module load gpsmpi/2016b
module load flex/2.5.39 # flex 2.6.0 causes errors
module load Boost/1.61.0
module load MPFR/3.1.3
module load GMP/6.1.1
module load CGAL/4.8.2
module load METIS/5.1.0-RTW64-IDX32

# configure Score-P
module load Score-P/3.0-p1
export SCOREP_WRAPPER_COMPILER_FLAGS=
export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--nomemory --verbose"

# Score-P can either support generating serial code or parallel/mpi code (default=parallel) - but never both at the same time.
# If you run an application instrumented by Score-P with parallel support in serial,
# Score-P will not be able to dump its results to disk but will report an error at application end.
# For an OpenFOAM installation for serial support uncomment the following.
#export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--mpp=none ${SCOREP_WRAPPER_INSTRUMENTER_FLAGS}"

# configure environment
export FOAM_INST_DIR=$INST_DIR
export WM_COMPILER=Gcc
export WM_MPLIB=EASYBUILDMPI
export MPICC="scorep-mpicc"
export MPICXX="scorep-mpicxx"
export MPICC_ALT="mpicc"
export MPICXX_ALT="mpicxx"
export CC="scorep-gcc"
export CXX="scorep-g++"
export CC_ALT="gcc"
export CXX_ALT="g++"
export CFLAGS="-O2 -fuse-ld=bfd"
export CXXFLAGS="-O2 -DMPICH_SKIP_MPICXX -fuse-ld=bfd"

export WM_NCOMPPROCS=8  # parallel build
export WM_LABEL_SIZE=32 # 32bit for labels/indices - relevant for METIS (if 64 needed patch METIS first)

# load environment
VERSION=3.0.1
source OpenFOAM-${VERSION}/etc/bashrc
foam
foamSystemCheck
}}}
* [wiki:/Software/OpenFOAM/config_ScoreP/config_Intel set environment configuration for Intel compiler] (does not perform well with Score-P - read info below)

=== etc/config/settings.sh

Patch $INST_DIR/OpenFOAM-${VERSION}/etc/config/settings.sh
{{{ #!sh
### settings.sh
Linux)
...
    x86_64)
...
        64)
            WM_ARCH=linux64
            export WM_COMPILER_LIB_ARCH=64
+           export WM_CC=${CC_ALT}                  # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_CXX=${CXX_ALT}                # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_CFLAGS="${CFLAGS} -fPIC"      # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_CXXFLAGS="${CXXFLAGS} -fPIC"  # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_LDFLAGS=                      # for use in non-wmake (e.g. ThirdParty) compilations

}}}

* You are using GCC (not Icc) - ensure you have 
 *  patched $INST_DIR/OpenFOAM-${VERSION}/wmake/rules/linux'''Gcc'''/* as described for the general OpenFOAM build. 

=== Scotch
Building of Scotch library builds and then runs an executable as part of the compilation which is called dummysizes. This must not be compiled with Score-P instrumentation. Hence, we compile it without the scorep wrapper.

Patch $WM_THIRD_PARTY_DIR/etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32
{{{ #!sh
# scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32
+ AR             = ${CC}
+ CCS            = ${CC}
+ CCP            = ${MPICC}
+ CCD            = ${MPICC}
+ CCD_ALT        = ${MPICC_ALT}
+ CFLAGS         = -O2 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict
}}}

Patch $WM_THIRD_PARTY_DIR/scotch_6.0.3/src/libscotch/Makefile
{{{ #!sh
# libscotch/Makefile
+ dummysizes$(EXE) : dummysizes.c                            \
[...]
                     $(CCD_ALT) $(CFLAGS) -DSCOTCH_VERSION=$(VERSION) -DSCOTCH_RELEASE=$(RELEASE) -DSCOTCH_PATCHLEVEL=$(PATCHLEVEL) $(<) -o $(@) $(LDFLAGS)

+ ptdummysizes$(EXE) : dummysizes.c                            \
[...]
+                     $(CCD_ALT) $(CFLAGS) -DSCOTCH_VERSION=$(VERSION) -DSCOTCH_RELEASE=$(RELEASE) -DSCOTCH_PATCHLEVEL=$(PATCHLEVEL) $(<) -o $(@) $(LDFLAGS)
}}}
* Ensure there is no generated file from a precious build left in $WM_THIRD_PARTY_DIR/platforms.
  * eg. $WM_THIRD_PARTY_DIR/platforms/linux64'''Gcc'''Int32/scotch_6.0.3/src/libscotch/Makefile
  * clean the installation first: $WM_THIRD_PARTY_DIR/Allwclean

=== Score-P
Fully instrumenting OpenFOAM with Score-P can easily result in a very slow and memory consuming application.\\With little effort this effect can be limited.

* '''Limiting the number of functions''', which are measured during runtime is required.\\This limiting can be done at compile- or runtime.
  1. Compile without any compiler instrumentation
    * {{{export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--nocompiler"}}} (for scorep-wrappers)
    * {{{scorep --nocompiler}}} (for direct call of scorep)
  2. Use sampling during compile-time:
    * Enable sampling\\{{{export SCOREP_ENABLE_UNWINDING=true}}}
    * To reduce overhead play with sampling period (<event>[@<period>])\\{{{export SCOREP_SAMPLING_EVENTS="perf_cycles@10000000"}}}
  3. Use filter file during runtime:
    * Create filter file and export filepath
{{{
SCOREP_REGION_NAMES_BEGIN
EXCLUDE
# here specify list of functions or bash-like wildcars
SCOREP_REGION_NAMES_END
}}}
{{{
export SCOREP_FILTERING_FILE=$PATH_TO_FILE/filename
}}}
    * You can filter out all functions, then you will see only MPI calls, but the overhead will be still big.
  4. In some cases manual instrumentation is very useful.
    * See Score-P documentation for this: https://silc.zih.tu-dresden.de/scorep-current/pdf/scorep.pdf