== Building !VisIt on JUQUEEN [[PageOutline]] In addition to the normal ''build_visit script'', a special build script for JUQUEEN, called ''build_visit_BGQ'', ist needed. To build !VisIt one has to execute this script. build_visit_BGQ in turn calls the build_visit script with a special set of parameters to generate a static version of the libsim and other components which are necessary for a static linking to your simulation code. !VisIt versions tested for !BlueGene/G: * 2.9.2 (build successfull) * 2.10.0 (build FAILS, if HDF5 included: [https://visitbugs.ornl.gov/issues/2423 bugreport]) * 2.10.1 (build successfull) ==== 1) download scripts Before one can execute the build_visit_BGQ (and thereby build_visit) script, a couple of helper-scripts are also needed. These helper-scripts are contained in the ''svn_bin/bv_support'' folder of the visit source-tree. The two original build scripts and the bv_support folder can be downloaded as follows (for !VisIt version 2.10.1): {{{ #!ShellExample cd #download svn_bin directory svn co --non-interactive http://visit.ilight.com/svn/visit/tags/2.10.1/src/svn_bin/ #move important parts to . mv svn_bin/bv_support . mv svn_bin/build_visit . #use build_visit_BGQ, IF it is fixed ... mv svn_bin/build_visit_BGQ . #... if not, merge build_visit_BGQ with this one from here wget https://trac.version.fz-juelich.de/vis/raw-attachment/wiki/VisIt/build_Juqueen/build_visit_BGQ_2.10.1 mv build_visit_BGQ_2.10.1 build_visit_BGQ chmod +x build_visit_BGQ #delete non-required svn_bin chmod -R u+w svn_bin rm -r svn_bin }}} ==== 2) patch scripts As of Version 2.9.2 the original build_visit_BQG and build_visit scripts where not able to compile !VisIt on the JUQUEEN installation at JSC correctly. \\ We have made changes to the scripts, e.g. build_visit_BGQ was not able to handle multiple compiler arguments and the build_visit script generated a wrong host-configuration file.\\ For !VisIt 2.9.2 and 2.10.1 we fix these issues and now have corrected versions of the script build_visit_BGQ which should be used on the JUQUEEN at JSC: ===== patch build_visit_BGQ compare build_visit_BGQ with the following one and apply changes \\ (changes have been reported as [https://visitbugs.ornl.gov/issues/2412 bugreport]) * [attachment:build_visit_BGQ_2.9.2 build_visit_BGQ (for 2.9.2)] * [attachment:build_visit_BGQ_2.10.1 build_visit_BGQ (for 2.10.1)] ===== patch build_visit helper function In bv_support/helper_funcs.sh you will find a section with the following code \\ (if you have downloaded the build_visit including all functions of bv_support it is in build_visit directly) {{{ #!ShellExample vim bv_support/helper_funcs.sh [... close to line 1360 ...] echo "SET(BLUEGENEQ /bgsys/drivers/ppcfloor)" >> $HOSTCONF echo "VISIT_OPTION_DEFAULT(VISIT_PARALLEL ON TYPE BOOL)" >> $HOSTCONF echo "VISIT_OPTION_DEFAULT(VISIT_MPI_CXX_FLAGS \"-I\${BLUEGENEQ} -I\${BLUEGENEQ}/comm/include -I\${BLUEGENEQ}/spi/include -I\${BLUEGENEQ}/spi/include/kernel/cnk\" TYPE STRING)" >> $HOSTCONF echo "VISIT_OPTION_DEFAULT(VISIT_MPI_C_FLAGS \"-I\${BLUEGENEQ} -I\${BLUEGENEQ}/comm/include -I\${BLUEGENEQ}/spi/include -I\${BLUEGENEQ}/spi/include/kernel/cnk\" TYPE STRING)" >> $HOSTCONF echo "VISIT_OPTION_DEFAULT(VISIT_MPI_LD_FLAGS \"-L\${BLUEGENEQ}/spi/lib -L\${BLUEGENEQ}/comm/lib -R/opt/ibmcmp/lib64/bg/bglib64\" TYPE STRING)" >> $HOSTCONF echo "VISIT_OPTION_DEFAULT(VISIT_MPI_LIBS mpich-xl opa-xl mpl-xl pami-gcc SPI SPI_cnk rt pthread stdc++ pthread TYPE STRING)" >> $HOSTCONF [...] }}} Here the paths for MPI are set. You probably have to change them. Make sure they point to the correct directories. \\ For the current (15.10.2015) setup on JUQUEEN mpi.h can only be found if the paths include "${BLUEGENEQ}/spi/include". ===== patch !LaunchService.C (FIXED in 2.10.1) !LaunchService.C as to check for command line argument "-socket-bridge-host" and use the hostname provided instead of calling gethostname(). This is essential to make !VisIt run on the JUQUEEN. But this file cannot be patched right away, as you have to wait for the build_script to download and extract the !VisIt sources before. Best is to run the complete build procedure and than patch !LaunchService.C and rerun the build procedure again (it will skip a lot of steps). \\ \\ !LaunchService.C is located in * builds_static/src/common/comm/ * builds_static_BGQ/src/common/comm/ Here you find the patched file [attachment:LaunchService_socket-bridge-host.C LaunchService.C] for !VisIt 2.9.2. \\ Check the difference and adopt it to your version of !VisIt ([https://visitbugs.ornl.gov/issues/2429 bugreport]). ===== patch visitfortransimV2interface.c (FIXED in 2.10.1) Replace line 427 in function visitgetenv() ([https://visitbugs.ornl.gov/issues/2432 bugreport]). {{{ #!C++ sz = (len < *lenv) ? (sz - 1) : (*lenv - 1); }}} with {{{ #!C++ sz = (len < *lenv) ? (len) : (sz - 1); }}} ==== 3) add extras If you want visit to build additional I/O packages, e.g. boost, netcdf, hdf5, ..., needed by several !VisIt-readers (databases), you have to edit line 39 in the build_visit_BGQ script, e.g. {{{ #!ShellExample IO_PACKAGES="--boost --netcdf --szip --hdf5 --silo" }}} If you omit this line, visit tries to find system wide installations of the packages. But be aware that, in this case, several databases maybe cannot be build if the corresponding packages are not found. ==== 4) compile !VisIt The build_visit_BGQ script may be invoked e.g. with the following arguments: {{{ #!ShellExample ./build_visit_BGQ --svn-anonymous }}} if you do not want to change settings make in build_visit_BGQ, or {{{ #!ShellExample ./build_visit_BGQ \ --build-mode Release \ --database-plugins "BOV;Cale;CaleHDF5;Curve2D;EnSight;Miranda;PDB;PlainText;SAMRAI;Silo;VTK;Nek5000" \ --svn-anonymous \ \ --cc gcc \ --cxx g++ \ \ --bgcc bgxlc_r \ --bgcxx bgxlC_r \ --bgcflag "-qarch=qp -qtune=qp" \ --bgcxxflag "-qarch=qp -qtune=qp" \ \ --bggnucc /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-gcc \ --bggnucxx /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-g++ \ \ --bgmpicc mpixlc_r \ --bgmpicxx mpixlcxx_r \ --bgmpicflag "-qarch=qp -qtune=qp" \ --bgmpicxxflag "-qarch=qp -qtune=qp" }}} ---- If you want to instrument !VisIt with Score-P/Scalasca load the following modules {{{ #!ShellExample module load UNITE module load scalasca }}} and set the following compiler in build_visit_BGQ: {{{ #!ShellExample BG_CC=scorep-bgxlc_r BG_CXX=scorep-bgxlC_r BG_GNU_CC=/bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-gcc BG_GNU_CXX=/bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-g++ BG_MPI_GNU_CC=mpicc # not used BG_MPI_GNU_CXX=mpicxx BG_MPI_CC=scorep-mpixlc_r BG_MPI_CXX=scorep-mpixlcxx_r }}} and comment the following lines in builds_static_BGQ/src/CMakeLists.txt after it is downloaded and extracted: {{{ #!ShellExample ADD_SUBDIRECTORY(tools) ADD_SUBDIRECTORY(engine/main) }}} ---- The build sequence is as follows: * first, the compute-node version of !VisIt is build in the directory ''builds_static_BGQ'', * than the login-node version is build in the directory ''builds_static'' In the build process the following third party packages are installed in the directory ''thirdparty_static'': * mesa * zlib, szip * cmake * python (including python image library (PIL), pyparsing, seedme-python) * vtk * icet * I/O-Libs: boost, hdf5, netcdf, silo ==== 5) install !VisIt After the build process has successfully finished, !VisIt is installed in the directory ''linux-ppc64-BGQ''. \\ \\ The static libs required for code running on the compute nodes should now be collected in $LIBSIM_INST by the build_visit_BGQ script. \\ This way all static !VisIt libs your BGQ application needs for linking are in one common directory (''$LIBSIM_INST''). '''If''' build_visit_BGQ has not copied the additional libs, do it manually: {{{ #!ShellExample LIBSIM_INST=linux-ppc64-BGQ///libsim/V2/libBGQ/ cp builds_static_BGQ/src/lib/libsimV2runtime_par.a ${LIBSIM_INST} cp builds_static_BGQ/src/lib/libengine_par.a ${LIBSIM_INST} cp builds_static_BGQ/src/lib/libsimV2_static_par.a ${LIBSIM_INST} cp builds_static_BGQ/src/lib/libcognomen.a ${LIBSIM_INST} cp builds_static_BGQ/src/lib/libvisit_vtk.a ${LIBSIM_INST} }}} ===== install !VisIt at !VisIt for !BlueGene/Q is now installed in linux-ppc64-BGQ - just copy it to the destination path: {{{ #!ShellExample cp -a linux-ppc64-BGQ/* }}} * eg. cp -a linux-ppc64-BGQ/* /home/VisIt/ But LibsimV2 is compiled as a static library on JUQUEEN and has dependencies to multiple other libraries. \\ Hence, linking libsimV2 to a simulation results in a long link line. [wiki:VisIt/runon_Juqueen/linking more...] \\ We also need the static libs of VTK,IceT and MESA for !BlueGene/Q: {{{ #!ShellExample cp -a thirdparty_static // }}} Now the install path should look like this. Everything above the directory is version independent and backwards compatible. Hence, you can replace it with every new version copied to the . {{{ #!ShellExample ls 2.9.2 bin linux-ppc64 thirdparty_static 2.9.1 bin linux-ppc64 thirdparty_static 2.9.0 bin linux-ppc64 thirdparty_static bin current }}} ===== install !VisIt on JUQUEEN On JUQUEEN !VisIt is installed in two separate directories.\\ One is for !VisIt compiled for the login node and located at ''/usr/local/visit'' \\ and one is for !VisIt libs compiled for the compute node and located at ''/bgsys/local/visit''. Login Node: {{{ #!ShellExample VISIT_ROOT=/usr/local/visit cp -a linux-ppc64-BGQ/ ${VISIT_ROOT} cp -a linux-ppc64-BGQ/bin ${VISIT_ROOT} rm ${VISIT_ROOT}/current ln -s ${VISIT_ROOT}/ current }}} Compute Node: {{{ #!ShellExample VISIT_BGQ_ROOT=/bgsys/local/visit VISIT_BGQ_LIBSIM=${VISIT_BGQ_ROOT}//linux-ppc64_BGQ/libsim/V2/ mkdir -p ${VISIT_BGQ_LIBSIM} cp -a linux-ppc64-BGQ//linux-ppc64/libsim/V2/include/ ${VISIT_BGQ_LIBSIM}/ cp -a linux-ppc64-BGQ//linux-ppc64/libsim/V2/libBGQ ${VISIT_BGQ_LIBSIM}/lib cp -a thirdparty_static ${VISIT_BGQ_ROOT}// }}} Create new Module Files: {{{ #!ShellExample cd /usr/local/modulefiles/MISC/visit cp 2.9.2 vim vim .version }}} ==== 5) additional infos ===== launch process Make sure you understand the launch procedure of !VisIt by its python scripts: \\ * bin/visit -> bin/frontendlauncher * bin/fontendlauncher * makes sure Python settings are made * load frontendlauncher.py using !VisIt's python interpreter * bin/frontendlauncher.py * determine the !VisIt version to run * create Launcher object from python-file 'internallauncher' (eg. 2.10.1/bin/internallauncher) \\ or python file 'customlauncher' (if available) * /bin/internallauncher * run python-file internallauncher with launcher object [http://www.visitusers.org/index.php?title=VisIt_Launcher more] ---- any feedback welcomed - [mailto:h.zilken@fz.juelich.de], [mailto:j.goebbert@fz-juelich.de]