Version 7 (modified by 7 years ago) ( diff ) | ,
---|
!!! WORK IN PROGRESS !!!
Last Update 02.2017, Authors: Jens Henrik Göbbert, ...
OpenFOAM Settings for use with Score-P
- This includes additional information on how to build OpenFOAM for use with Score-P.
- follow https://trac.version.fz-juelich.de/vis/wiki/Software/OpenFOAM, but do NOT build OpenFOAM, yet.
- follow the additional instructions from this page
- follow the build instructions from https://trac.version.fz-juelich.de/vis/wiki/Software/OpenFOAM#a5.BuildTestOpenFOAM
Set Environment
The environment must be set up according to where the source code directories are located.
cd <my_openfoam_install_dir> export INST_DIR=$(pwd) # $INST_DIR must NOT (!!) end with a '/', or build will fail with "touch [...] permission denied" cd $INST_DIR # load modules (GCC) module --force purge module use /usr/local/software/jureca/OtherStages module load Stages/Devel module load gpsmpi/2016b module load flex/2.5.39 # flex 2.6.0 causes errors module load Boost/1.61.0 module load MPFR/3.1.3 module load GMP/6.1.1 module load CGAL/4.8.2 module load METIS/5.1.0-RTW64-IDX32 # configure Score-P module load Score-P/3.0-p1 export SCOREP_WRAPPER_COMPILER_FLAGS= export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--nomemory --verbose" # Score-P can either support generating serial code or parallel/mpi code (default=parallel) - but never both at the same time. # If you run an application instrumented by Score-P with parallel support in serial, # Score-P will not be able to dump its results to disk but will report an error at application end. # For an OpenFOAM installation for serial support uncomment the following. #export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--mpp=none ${SCOREP_WRAPPER_INSTRUMENTER_FLAGS}" # configure environment export FOAM_INST_DIR=$INST_DIR export WM_COMPILER=Gcc export WM_MPLIB=EASYBUILDMPI export MPICC="scorep-mpicc" export MPICXX="scorep-mpicxx" export MPICC_ALT="mpicc" export MPICXX_ALT="mpicxx" export CC="scorep-gcc" export CXX="scorep-g++" export CC_ALT="gcc" export CXX_ALT="g++" export CFLAGS="-O2 -fuse-ld=bfd" export CXXFLAGS="-O2 -DMPICH_SKIP_MPICXX -fuse-ld=bfd" export WM_NCOMPPROCS=8 # parallel build export WM_LABEL_SIZE=32 # 32bit for labels/indices - relevant for METIS (if 64 needed patch METIS first) # load environment VERSION=3.0.1 source OpenFOAM-${VERSION}/etc/bashrc foam foamSystemCheck
- set environment configuration for Intel compiler (does not perform well with Score-P - read info below)
etc/config/settings.sh
Patch $INST_DIR/OpenFOAM-${VERSION}/etc/config/settings.sh
### settings.sh Linux) ... x86_64) ... 64) WM_ARCH=linux64 export WM_COMPILER_LIB_ARCH=64 + export WM_CC=${CC_ALT} # for use in non-wmake (e.g. ThirdParty) compilations + export WM_CXX=${CXX_ALT} # for use in non-wmake (e.g. ThirdParty) compilations + export WM_CFLAGS="${CFLAGS} -fPIC" # for use in non-wmake (e.g. ThirdParty) compilations + export WM_CXXFLAGS="${CXXFLAGS} -fPIC" # for use in non-wmake (e.g. ThirdParty) compilations + export WM_LDFLAGS= # for use in non-wmake (e.g. ThirdParty) compilations
- You are using GCC (not Icc) - ensure you have
- patched $INST_DIR/OpenFOAM-${VERSION}/wmake/rules/linuxGcc/* as described for the general OpenFOAM build.
scotch
Building of Scotch library builds and then runs an executable as part of the compilation which is called dummysizes. This must not be compiled with Score-P instrumentation. Hence, we compile it without the scorep wrapper.
Patch $WM_THIRD_PARTY_DIR/etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32
# scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32 + AR = ${CC} + CCS = ${CC} + CCP = ${MPICC} + CCD = ${MPICC} + CCD_ALT = ${MPICC_ALT} + CFLAGS = -O2 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict
Patch $WM_THIRD_PARTY_DIR/scotch_6.0.3/src/libscotch/Makefile
# libscotch/Makefile + dummysizes$(EXE) : dummysizes.c \ [...] $(CCD_ALT) $(CFLAGS) -DSCOTCH_VERSION=$(VERSION) -DSCOTCH_RELEASE=$(RELEASE) -DSCOTCH_PATCHLEVEL=$(PATCHLEVEL) $(<) -o $(@) $(LDFLAGS) + ptdummysizes$(EXE) : dummysizes.c \ [...] + $(CCD_ALT) $(CFLAGS) -DSCOTCH_VERSION=$(VERSION) -DSCOTCH_RELEASE=$(RELEASE) -DSCOTCH_PATCHLEVEL=$(PATCHLEVEL) $(<) -o $(@) $(LDFLAGS)
- Ensure there is no generated file from a precious build left in $WM_THIRD_PARTY_DIR/platforms.
- eg. $WM_THIRD_PARTY_DIR/platforms/linux64GccInt32/scotch_6.0.3/src/libscotch/Makefile
- clean the installation first: $WM_THIRD_PARTY_DIR/Allwclean
Score-P
Fully instrumenting OpenFOAM with Score-P can easily result in a very slow and memory consuming application.
With little effort this effect can be limited.
- Limiting the number of functions, which are measured during runtime is required.
This limiting can be done at compile- or runtime.- Compile without any compiler instrumentation
export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--nocompiler"
(for scorep-wrappers)scorep --nocompiler
(for direct call of scorep)
- Use sampling during compile-time:
- Enable sampling
export SCOREP_ENABLE_UNWINDING=true
- To reduce overhead play with sampling period (<event>[@<period>])
export SCOREP_SAMPLING_EVENTS="perf_cycles@10000000"
- Enable sampling
- Use filter file during runtime:
- Create filter file and export filepath
SCOREP_REGION_NAMES_BEGIN EXCLUDE # here specify list of functions or bash-like wildcars SCOREP_REGION_NAMES_END
export SCOREP_FILTERING_FILE=$PATH_TO_FILE/filename
- You can filter out all functions, then you will see only MPI calls, but the overhead will be still big.
- Create filter file and export filepath
- In some cases manual instrumentation is very useful.
- See Score-P documentation for this: https://silc.zih.tu-dresden.de/scorep-current/pdf/scorep.pdf
- Compile without any compiler instrumentation
- Limiting the measurements done for each function.
- disables memory usage instrumentation
scorep --nomemory
- disables memory usage instrumentation
- more ...
- Switch to GCC compiler enables Score-P to distinguish between inlined and non-inlined functions.
The compiler is now free to inline any function and stop these parts of the code to be instrumented.
Especially for C++ code this should be considered.
- Switch to GCC compiler enables Score-P to distinguish between inlined and non-inlined functions.