== In-situ Visualisation on JUQUEEN [[PageOutline]] \\ ---- === Steps On JUQUEEN === start simulation Example: [attachment:psOpen_VisIt.jobscript psOpen_VisIt.jobscript] ==== 1) prepare jobscript {{{ #!ShellExample export HOME= export VISIT=/usr/local/visit export PATH=${VISIT}/bin:${PATH} export LIBGL_ALWAYS_INDIRECT=1 NODE_RANKS=16 runjob --ranks-per-node ${NODE_RANKS} \ --exp-env HOME \ --exp-env PATH \ --exp-env LIBGL_ALWAYS_INDIRECT \ : }}} * Set $HOME or !VisIt segfaults ([https://visitbugs.ornl.gov/issues/2267 bug]) * Set $PATH so the 'visit' can be found * Set $LIBGL_ALWAYS_INDIRECT=1 to ensure off-screen rendering (flag not required, but good habit) ==== 2) submit job {{{ #!ShellExample llsubmit }}} ==== 3) wait until job has started and sim2 file is written {{{ #!ShellExample llq -u $USER }}} {{{ #!ShellExample cat ~/.visit/simulations/ host R63-IF-J02 port 5609 key 9fd458909abe0f26 comment My Simulation }}} \\ ---- === Steps On Local Machine (using Host profile) ==== 1) start !VisIt GUI {{{ #!ShellExample visit }}} * "-debug 5" \\ Executing !VisIt with "-debug 5" on the local machine will result in *.vlog files in $HOME on JUQUEEN written by !VisIt's metadata server. Additionally it writes log files of the GUI/Viewer to the current directory on your local machine, too. This can be helpful debugging server-side problems. But it slows down things A LOT! \\ Adding a 'd' character to the argument to the -debug command-line option (e.g. -debug 3d) will cause level 1 debug logs, and only level 1 debug logs, to be decorated with !__FILE!__ and !__LINE!__ (since 2.10.0). * "-nopty" \\ Executing !VisIt with "-nopty" on the local machine will avoid a hidden shell for ssh connection. Instead you can see all ssh-output on the terminal. ==== 2) create host profile [[Image(https://trac.version.fz-juelich.de/vis/raw-attachment/wiki/VisIt/runon_Juqueen/JUQUEEN_HostProfile-2.png, 400px, align=right, margin=10)]] [[Image(https://trac.version.fz-juelich.de/vis/raw-attachment/wiki/VisIt/runon_Juqueen/JUQUEEN_HostProfile-1.png, 400px, align=right, margin=10)]] ===== Setup a host profile Options -> Host profiles ... -> New Host * Host Profile * Host nickname: JSC JUQUEEN * Remote host name: juqueen1c1.zam.kfa-juelich.de * Host name aliases: juqueen1c1 * Path to !VisIt installation: /usr/local/visit * Username: * [x] Tunnel data connection through SSH * [x] Use gateway: juqueen1.fz-juelich.de * Launch Profile * New Profile: default * Additional arguments: -socket-bridge-host juqueen1c1.zam.kfa-juelich.de -fixed-buffer-sockets ===== Save host profile * Apply * Options -> Save Settings ===== Info: Why do we need to use a gateway [[Image(visit_connect.png, 380px, align=right, margin=10)]] The JUQUEEN login nodes have two IP addresses. One is valid for connections from outside the JUQUEEN network and one IP address is valid from connections inside the JUQUEEN network. \\ Compute nodes _cannot_ access the login node using the public IP. \\ ''public ip'' <------------> ''internal ip'' \\ 134.94.32.59 <- juqueen1/juqueen1c1 -> 134.94.141.13 \\ 134.94.32.60 <- juqueen2/juqueen2c1 -> 134.94.141.14 \\ 134.94.32.61 <- juqueen3/juqueen3c1 -> 134.94.141.15 \\ 134.94.32.62 <- juqueen4/juqueen4c1 -> 134.94.141.16 \\ ===== Info: Why do we need "-socket-bridge-host" !VisIt starts its !VisIt Component Launcher (VCL) on the host (here juqueen1c1) and tells the !VisIt engine on the cluster who this host is. On default VCL just uses gethostname() to get the name of the machine it is running on. But this function call on juqueen![1-4]c1 will answer juqueen![1-4] instead of juqueen![1-4]c1. juqueen![1-4] is not accessible from inside the cluster and the connection from the !VisIt engine will fail. \\ "-socket-bridge-host" will force VCL not to use gethostname() but just the name we pass by argument. ===== Info: Why do we need "-fixed-buffer-sockets" The IO nodes on the JUQUEEN collect messages and send them only if a certain number of bytes have been collected.\\ "-fixed-buffer-sockets" fills each message with zeros to reach that certain size. This makes sure, that the message is send without delay (which is essential). \\ \\ Alternatively you can also set the environment variable BG_SYSIODPOSIXMODE=1 => Run I/O operations with POSIX rules: * 0 == I/O operation that is initiated from a compute node can cause multiple I/O operations on the I/O node. * 1 == Each I/O operation that is initiated from a compute node completes atomically. {{{ #!ShellExample export BG_SYSIODPOSIXMODE=1 runjob --exp-env BG_SYSIODPOSIXMODE ... }}} ==== 3) connect with host profile Connecting from !VisIt GUI to a running simulation requires some preparations. \\ If you establish a connection for the first time to JUQUEEN please make sure: * check if you can connect with ssh directly to juqueen1.fz-juelich.de \\ (using public/private key authentification without passphrase) {{{ #!ShellExample client:> ssh @juqueen1.fz-juelich.de }}} * check if you can connect with ssh from juqueen1.fz-juelich.de to juqueen1c1.zam.kfa-juelich.de \\ (using public/private key authentification without passphrase) {{{ #!ShellExample juqueen1:> ssh @juqueen1c1.zam.kfa-juelich.de }}} Connect with host profile to JUQUEEN:\\ File -> Open file ... -> Host: -> JSC JUQUEEN ===== Passwordless SSH on Windows "!VisIt's ssh is based on PuTTY so it is compatible with PuTTY's pageant program. Pageant is an ssh key caching program that PuTTY programs can query for authentication purposes. Once you run pageant and load it with the public key for the remote computer you with to access, !VisIt will be able to connect to that remote computer without entering a password." [http://www.visitusers.org/index.php?title=SSH_on_Windows link] ==== 4) open sim2 file on with !VisIt GUI ~/.visit/simulations/ ---- === Steps On Local Machine (manual) If you do not want to run !VisIt's metadata server on the login node, you can connect to your simulation manually. \\ Do not try it, if you are not an expert (or have some spare time left). \\ [wiki:VisIt/runon_Juqueen/manual_connection more...] ---- any feedback welcomed - [mailto:h.zilken@fz.juelich.de], [mailto:j.goebbert@fz-juelich.de]