Software/OpenFOAM/config_ScoreP – vis

Context Navigation

Version 5 (modified by Jens Henrik Goebbert, 7 years ago) ( diff )
--

!!! WORK IN PROGRESS !!!
Last Update 02.2017, Authors: Jens Henrik Göbbert, ...

OpenFOAM Settings for use with Score-P

This includes additional information on how to build OpenFOAM for use with Score-P.
1. follow https://trac.version.fz-juelich.de/vis/wiki/Software/OpenFOAM, but do NOT build OpenFOAM, yet.
2. follow the additional instructions from this page
3. follow the build instructions from https://trac.version.fz-juelich.de/vis/wiki/Software/OpenFOAM#a5.BuildTestOpenFOAM

Set Environment

The environment must be set up according to where the source code directories are located.

cd <my_openfoam_install_dir> 
export INST_DIR=$(pwd) # $INST_DIR must NOT (!!) end with a '/', or build will fail with "touch [...] permission denied"
cd $INST_DIR

# load modules (GCC)
module --force purge
module use /usr/local/software/jureca/OtherStages
module load Stages/Devel
module load gpsmpi/2016b
module load flex/2.5.39 # flex 2.6.0 causes errors
module load Boost/1.61.0
module load MPFR/3.1.3
module load GMP/6.1.1
module load CGAL/4.8.2
module load METIS/5.1.0-RTW64-IDX32

# configure Score-P
module load Score-P/3.0-p1
export SCOREP_WRAPPER_COMPILER_FLAGS=
export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--nomemory --verbose"

# Score-P can either support generating serial code or parallel/mpi code (default=parallel) - but never both at the same time.
# If you run an application instrumented by Score-P with parallel support in serial,
# Score-P will not be able to dump its results to disk but will report an error at application end.
# For an OpenFOAM installation for serial support uncomment the following.
#export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--mpp=none ${SCOREP_WRAPPER_INSTRUMENTER_FLAGS}"

# configure environment
export FOAM_INST_DIR=$INST_DIR
export WM_COMPILER=Gcc
export WM_MPLIB=EASYBUILDMPI
export MPICC="scorep-mpicc -fuse-ld=bfd"
export MPICXX="scorep-mpicxx -fuse-ld=bfd"
export MPICC_ALT="mpicc -fuse-ld=bfd"
export MPICXX_ALT="mpicxx -fuse-ld=bfd"
export CC="scorep-gcc -fuse-ld=bfd"
export CXX="scorep-g++ -fuse-ld=bfd"
export CC_ALT="gcc -fuse-ld=bfd"
export CXX_ALT="g++ -fuse-ld=bfd"
export CFLAGS="-no-prec-div -O2 "
export CXXFLAGS="-O2 -DMPICH_SKIP_MPICXX"

export WM_NCOMPPROCS=8  # parallel build
export WM_LABEL_SIZE=32 # 32bit for labels/indices - relevant for METIS (if 64 needed patch METIS first)

# load environment
VERSION=3.0.1
source OpenFOAM-${VERSION}/etc/bashrc
foam
foamSystemCheck

set environment configuration for Intel compiler (does not perform well with Score-P - read info below)

etc/config/settings.sh

Patch $INST_DIR/OpenFOAM-${VERSION}/etc/config/settings.sh

### settings.sh
Linux)
...
    x86_64)
...
        64)
            WM_ARCH=linux64
            export WM_COMPILER_LIB_ARCH=64
+           export WM_CC=${CC_ALT}                  # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_CXX=${CXX_ALT}                # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_CFLAGS="${CFLAGS} -fPIC"      # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_CXXFLAGS="${CXXFLAGS} -fPIC"  # for use in non-wmake (e.g. ThirdParty) compilations
+           export WM_LDFLAGS=                      # for use in non-wmake (e.g. ThirdParty) compilations

You are using GCC (not Icc) - ensure you have
- patched $INST_DIR/OpenFOAM-${VERSION}/wmake/rules/linuxGcc/* as described for the general OpenFOAM build.

scotch

Building of Scotch library builds and then runs an executable as part of the compilation which is called dummysizes. This must not be compiled with Score-P instrumentation. Hence, we compile it without the scorep wrapper.

Patch $WM_THIRD_PARTY_DIR/etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32

# scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32
+ AR             = ${CC}
+ CCS            = ${CC}
+ CCP            = ${MPICC}
+ CCD            = ${MPICC}
+ CCD_ALT        = ${MPICC_ALT}
+ CFLAGS         = -O2 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict

Patch $WM_THIRD_PARTY_DIR/scotch_6.0.3/src/libscotch/Makefile

# libscotch/Makefile
+ dummysizes$(EXE) : dummysizes.c                            \
[...]
                     $(CCD_ALT) $(CFLAGS) -DSCOTCH_VERSION=$(VERSION) -DSCOTCH_RELEASE=$(RELEASE) -DSCOTCH_PATCHLEVEL=$(PATCHLEVEL) $(<) -o $(@) $(LDFLAGS)

+ ptdummysizes$(EXE) : dummysizes.c                            \
[...]
+                     $(CCD_ALT) $(CFLAGS) -DSCOTCH_VERSION=$(VERSION) -DSCOTCH_RELEASE=$(RELEASE) -DSCOTCH_PATCHLEVEL=$(PATCHLEVEL) $(<) -o $(@) $(LDFLAGS)

Ensure there is no generated file from a precious build left in $WM_THIRD_PARTY_DIR/platforms.
- eg. $WM_THIRD_PARTY_DIR/platforms/linux64GccInt32/scotch_6.0.3/src/libscotch/Makefile
- clean the installation first: $WM_THIRD_PARTY_DIR/Allwclean

Score-P

Fully instrumenting OpenFOAM with Score-P can easily result in a very slow and memory consuming application.
With little effort this effect can be limited.

Limiting the number of functions, which are measured during runtime is required.
This limiting can be done at compile- or runtime.
1. Compile without any compiler instrumentation
  - export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--nocompiler" (for scorep-wrappers)
  - scorep --nocompiler (for direct call of scorep)
2. Use sampling during compile-time:
  - Enable sampling
    export SCOREP_ENABLE_UNWINDING=true
  - To reduce overhead play with sampling period (<event>[@<period>])
    export SCOREP_SAMPLING_EVENTS="perf_cycles@10000000"
3. Use filter file during runtime:
  - Create filter file and export filepath
```
SCOREP_REGION_NAMES_BEGIN
EXCLUDE
# here specify list of functions or bash-like wildcars
SCOREP_REGION_NAMES_END
```
```
export SCOREP_FILTERING_FILE=$PATH_TO_FILE/filename
```
  - You can filter out all functions, then you will see only MPI calls, but the overhead will be still big.
4. In some cases manual instrumentation is very useful.
  - See Score-P documentation for this: https://silc.zih.tu-dresden.de/scorep-current/pdf/scorep.pdf

Limiting the measurements done for each function.
1. disables memory usage instrumentation
  - scorep --nomemory

more ...
- Switch to GCC compiler enables Score-P to distinguish between inlined and non-inlined functions.
  The compiler is now free to inline any function and stop these parts of the code to be instrumented.
  Especially for C++ code this should be considered.

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text