= How to use !VisIt on JURECA [[PageOutline]] == General Hardware Setup The following image illustrates the general JURECA hardware setup used in a distributed visualisation session.\\ The user workstation acts as a frontend to interactively setup the visualisation pipeline and to adjust parameters of the visualisation. Of course the rendered image is shown on the users workstation.\\ The right side of the image illustrates the major components of the JURECA cluster: login, compute and visualisation nodes and the GPFS storage system (JUST).\\ The user has access to the resources of JURECA via an ssh connection to a login node. In between is a firewall which only allows access to the system by ssh (port 22). All other network ports are blocked. Therefore all communication related to the distributed visualisation environment has to be tunneled over ssh.\\ [[Image(Hardware_Setup.png, 640px)]]\\ In general, a distributed visualisation package, like !VisIt, consists of a viewer/GUI component and a parallel server/engine component. These software components can be mapped onto the hardware components in different ways. Some of the most useful scenarios are discussed in the following sections. ---- == Using VNC and !VisIt on a vis login node The Virtual Network Computing (VNC, e.g. [http://www.tigervnc.org TigerVNC] or [http://www.turbovnc.org/ TurboVNC]) package is our recommendation for remote rendering purposes. See section [wiki:vnc3d] for a detailed description on how to setup a VNC session on a JURECA login node. The following image gives an overview about this setup:\\ [[Image(vis_login_node.png, 640px)]]\\ Once you have gained access to the remote desktop of a JURECA vis login node, you can start !VisIt. If you used Strudel together with the vis profile to open the VNC desktop, you can start !VisIt just by clicking on the !VisIt desktop icon. Otherwise you have to open a terminal on the remote desktop and start !VisIt by: {{{ #!ShellExample module use /usr/local/software/jureca/OtherStages module --force purge module load Stages/2016a module load Intel module load IntelMPI module load VisIt vglrun visit -hw-accel }}} As it is not allowed to spawn mpi processes on login nodes, only the serial, non parallel version of !VisIt can be used directly on the vis login node. This is usefull to visualize small or medium sized data. Nevertheless additional resources can be allocated (see next section). ==== How to start parallel !VisIt on vis login node It is possible to start parallel !VisIt engines from the GUI either on vis batch nodes (hardware rendering) or regular compute batch nodes (software rendering). The following image illustrates this scenario using vis batch nodes (the scenario for compute batch nodes is equivalent):\\ [[Image(vis_login_vis_parallel.png, 640px)]]\\ The following steps have to be performed to start parallel !VisIt from the vis login node: ===== a) Check ssh connectivity Make sure the you can establish an ssh connection between JURECA nodes. That means that you have to generate/copy a private ssh key in your JURECA home directory (in ~/.ssh) and that you add the corresponding public key to authorized_keys. ===== b) Create/import a host profile for JURECA To work in a distributed setup, !VisIt needs information about the cluster and about how to launch processes on this cluster. These settings are collected in a so called "Host profile" and can be adjusted in !VisIt under Options --> Host profiles.\\ To keep things simple, we have created a predefined host profile for JURECA\\ \\ '''The host profile for JURECA can be downloaded here: [raw-attachment:host_jureca.xml host_jureca.xml]'''. \\ \\ Just download this file, and, depending on your operating system, copy it onto your local machine to: * on a Linux system: ~/.visit/hosts * on a Windows system: C:\Users\\Documents\!VisIt\hosts After the successful installation of this file, start !VisIt and open the host profile panel (Options --> Host profile). You should find a host profile for JURECA now. Before you can connect to JURECA, you have to add in some account specific information to this host profile: * Under "Host Setting --> Username" you have to enter the correct username of your JURECA account. * If you use a specific ssh key to access JURECA, it is also necessary to enter this key under "SSH command", e.g. "ssh -i YOUR\SSH\KEY_FILE".\\ The host profile has a couple of launch profiles for launching the !VisIt servers either on vis batch nodes or on compute batch nodes. ===== c) Enable remote rendering Open the Options --> Rendering dialogue. Select the Advanced tab and set "use scalable rendering" to "Always". This way it is guaranteed that the image is rendered by the parallel engines. ===== d) Open a file Inside !VisIt click on "open file". A filebrowser pops up. Select JURECA as host. Now !VisIt tries to establish an ssh connection to another JURECA login nodes. This process may take some seconds, so be patient. If everything works fine, the filesystem is displayed in the filebrowser. Select the data file you want to visualize and click on "OK". ===== e) Start parallel !VisIt servers After the file is selected, !VisIt opens a panel where you can select a launch profile. * To use vis batch nodes, select either "VIS NODE 512 GB" (max 4 nodes) or "VIS NODE 1024 GB" (max 2 nodes) * To use regular compute batch nodes, select either "COMPUTE NODE 256 GB", "COMPUTE NODE 512 GB" or "COMPUTE NODE 1024 GB" You also can adjust the number of requested nodes (Num nodes) and the (total) number of mpi tasks (Num procs) here. As JURECA offers 24 cores per node, the maximum usable number of mpi processes is 24*number_of_nodes (without SMT). However, as many visualisation algorithms do not scale very well, you may want to spawn fewer processes than the maximal number possible. You should experiment with these values (Num nodes, Num procs) to find out a good solution for your specific data and visualisation setup. It is a good idea to start with a setup as simple as possible (1 node, 2 procs) and to increase the values step by step. '''PLEASE DO USE MORE THAN 1 MPI TASK (num procs)! WHEN ONLY 1 MPI TASK IS SELECTED, VISIT STARTS THE SERIAL VERSION IF IT'S ENGINE WHICH AT THE MOMENT CRASHES WHEN TRYING TO DO HARDWARE RENDERING! '''\\ Select a launch profile and click on "OK". Now !VisIt automatically creates a temporary batch script on the JURECA login node (under /tmp) and starts this script with the sbatch command. Depending on your selection of the launch profile, this batch job is either executed on the vis partition (sbatch --partition=vis) or the batch partition (sbatch --partition=batch).\\ The scheduling of the batch job may take a while, so be patient and remember that the job can only be started if there are free resources on the cluster. == Using VNC and !VisIt on a vis batch node The Virtual Network Computing (VNC, e.g. [http://www.tigervnc.org TigerVNC] or [http://www.turbovnc.org/ TurboVNC]) package is our recommendation for remote rendering purposes. See section [wiki:vnc3d] for a detailed description on how to setup a VNC session on one or up to four JURECA batch nodes. The following image gives an overview about this setup:\\ [[Image(vis_batch_node.png, 640px)]]\\ Once you have gained access to the remote desktop of the allocated JURECA vis batch node, you can start !VisIt. If you used Strudel together with the vis profile to open the VNC desktop, you can start !VisIt just by clicking on the !VisIt desktop icon. Otherwise you have to open a terminal on the remote desktop and start !VisIt by: {{{ #!ShellExample module use /usr/local/software/jureca/OtherStages module --force purge module load Stages/2016a module load Intel module load IntelMPI module load VisIt vglrun visit -hw-accel }}} ==== How to start parallel !VisIt on vis batch node When you work on a VNC desktop running on a vis batch node, this node (or more than one node) is already allocated for you. This means that mpi jobs can be started on this resource just by invoking srun, which is done by !VisIt internally. The following steps have to be performed to start parallel !VisIt from the vis batch node: ===== a) Create/import a host profile for JURECA Vis Batch Node To work in a distributed setup, !VisIt needs information about the cluster and about how to launch processes on this cluster. These settings are collected in a so called "Host profile" and can be adjusted in !VisIt under Options --> Host profiles.\\ To keep things simple, we have created a predefined host profile for JURECA Vis Batch Node\\ \\ '''The host profile for JURECA Vis Batch Node can be downloaded here: [raw-attachment:host_jureca_vis_batch_node.xml host_jureca_vis_batch_node.xml]'''. \\ \\ Just download this file, and, depending on your operating system, copy it onto your local machine to: * on a Linux system: ~/.visit/hosts * on a Windows system: C:\Users\\Documents\!VisIt\hosts After the successful installation of this file, start !VisIt (see above) and open the host profile panel (Options --> Host profile). You should find a host profile for "JURECA Vis Batch Node" now. ===== b) Enable remote rendering Open the Options --> Rendering dialogue. Select the Advanced tab and set "use scalable rendering" to "Always". This way it is guaranteed that the image is rendered by the parallel engines. ===== c) Open a file Inside !VisIt click on "open file". A filebrowser pops up. Select "JURECA Vis Batch Node" as host. If everything works fine, the filesystem is displayed in the filebrowser. Select the data file you want to visualize and click on "OK". ===== e) Start parallel !VisIt servers After the file is selected, !VisIt opens a panel where you can select a launch profile. Just select "localhost" there. You also can adjust the number of requested nodes (Num nodes) and the (total) number of mpi tasks (Num procs) here. As JURECA offers 24 cores per node, the maximum usable number of mpi processes is 24*number_of_nodes (without SMT). However, as many visualisation algorithms do not scale very well, you may want to spawn fewer processes than the maximal number possible. You should experiment with these values (Num nodes, Num procs) to find out a good solution for your specific data and visualisation setup. It is a good idea to start with a setup as simple as possible (1 node, 2 procs) and to increase the values step by step. '''PLEASE DO USE MORE THAN 1 MPI TASK (num procs)! WHEN ONLY 1 MPI TASK IS SELECTED, VISIT STARTS THE SERIAL VERSION IF IT'S ENGINE WHICH AT THE MOMENT CRASHES WHEN TRYING TO DO HARDWARE RENDERING! '''\\ Click on "OK". Now !VisIt automatically starts the corresponding number of parallel !VisIt engines (eingine_par) by invoking srun. ---- == Direct VisIt remote visualisation on JURECA nodes (vis node or compute node) In this scenario the !VisIt viewer (GUI) is running on the users workstation. The data processing and image generation is performed by one or more !VisIt engines (servers) running on one or more JURECA nodes, which can be nodes from the visualisation partition or from the compute (batch) partition. Like the compute nodes also the vis nodes have to be allocated by the batch system. That means that this visualisation scenario may perhaps not start instantly if all free resources are already taken by other users! In this case one has to wait for available nodes. At the moment, the reservation of vis nodes, which would guarantee the access to vis nodes at a certain point in time is not yet possible. This feature is under development and will be available in the future.\\ All network traffic (e.g. sending images from the !VisIt server to the viewer) runs across the login node and has to be tunneled. !VisIt has a build in functionality to connect to the login node and to start the necessary batch job from there, so the user does not need to login manually to JURECA and start visualisation related batch jobs for !VisIt by hand. Nevertheless !VisIt uses ssh under the hood. Therefore an important prerequesite to get this scenario running is a flawlessly working ssh connectivity between the users workstation and JURECA! If a Windows operating system is running on the users workstation, it may be necessary to use an ssh key agent like Pageant, which is a part of the [http://www.putty.org/ PuTTY] ssh client for Windows.\\ === Using vis nodes [[Image(Cluster_VisIt_3D.png, 640px)]]\\ JURECA is equipped with 12 vis nodes. 10 of them have 512GByte main memory, 2 have 1024GByte main memory. Two NVidia K40 GPU's are attached to each vis node. For visualisation purposes only one GPU can be used at the moment. Whenever a batch job is startet via "sbatch --gres gpu:2 --start-xserver", an X-Server (display :0.0) is startet on the vis node to enable the usage of hardware accelerated OpenGL rendering (these options are already included in the here provided host profile, see below). The figure above gives an overview of this setup.\\ \\ \\ === Using compute nodes [[Image(Cluster_VisIt_2D.png, 640px)]]\\ Although on compute nodes only software rendering (via OSMesa) is possible, it can make sense to use compute nodes for remote rendering for some reasons: * a really huge number of nodes can be used, e.g. to improve I/O or data processing speed (image rendering is not always the bottleneck) * in case that all vis nodes are occupied, using compute nodes could be a work-around * in-situ visualisation of a running simulation (not covered in this section) The figure above gives an overview about this setup.\\ \\ To setup one of the above described scenarios, the following steps have to be performed: === a) Check ssh connectivity Make sure that your workstation has flawless ssh connectivity to the login nodes of JURECA. On a Linux workstation just check that you can login to JURECA with the regular ssh command.\\ On a Windows system you may need to install an ssh client first, e.g. [http://www.putty.org/ PuTTY], together with the ssh key agent Pageant. Perhaps you also want to install PuTTYgen to generate new ssh keys. Start Pageant and load your private ssh key into Pageant. Pageant will ask you for your passphrase, which unlocks the private key. Now !VisIt will be able to connect to JURECA using the (unlocked) private key stored in Pageant.\\ In case you do not want to use Pageant, you can also modify the host profile (see below, step 3). Open the host settings for JURECA and set the entry "SSH command" to "qtssh -i YOUR\SSH\KEY_FILE". This way you tell qtssh what your private key file is. However in this case you always have to type in your passphrase by hand for every connection to JURECA. === b) Install (the correct version of) !VisIt on your local workstation If not already done, download [https://wci.llnl.gov/simulation/computer-codes/visit/downloads VisIt] and install it on your workstation. Your local !VisIt version must match the version installed on JURECA, which, as of February 2016, is version 2.10. So you have to install exactly !VisIt 2.10 to get the setup running. === c) Create/import a host profile for JURECA To work in a distributed setup, !VisIt needs information about the remote cluster and about how to launch processes on this cluster. These settings are collected in a so called "Host profile" and can be adjusted in !VisIt under Options --> Host profiles.\\ To keep things simple, we have created a predefined host profile for JURECA\\ \\ '''The host profile for JURECA can be downloaded here: [raw-attachment:host_jureca.xml host_jureca.xml]'''. \\ \\ Just download this file and, depending on your operating system, copy it onto your local machine to: * on a Linux system: ~/.visit/hosts * on a Windows system: C:\Users\\Documents\!VisIt\hosts After the successful installation of this file, start !VisIt and open the host profile panel (Options --> Host profile). You should find a host profile for JURECA now. Before you can connect to JURECA, you have to add in some account specific information to this host profile: * Under "Host Setting --> Username" you have to enter the correct username of your JURECA account. * If you use a specific ssh key to access JURECA, it is also necessary to enter this key under "SSH command", e.g. "ssh -i YOUR\SSH\KEY_FILE".\\ The host profile has a couple of launch profiles, one for launching the !VisIt server on a vis node and another one for launching it on compute nodes. === d) Enable remote rendering Open !VisIt on your local computer and open the Options --> Rendering dialogue. Select the Advanced tab and set "use scalable rendering" to "Always". This way the image is guaranteed to be rendered on JURECA. === e) Connect VisIt to a JURECA login node Inside !VisIt click on "open file". A filebrowser pops up. Select JURECA as host. Now !VisIt tries to establish an ssh connection to JURECA and to start some !VisIt components (visit component launcher vcl, media data server mdserver) on the login node. This process may take some seconds, so be patient. If everything works fine, the filesystem of JURECA is displayed in the filebrowser. Select the data file you want to visualise and click on "OK". === f) Start !VisIt server (engine_par) on the batch system After the file is selected, !VisIt opens a panel where you can select one of the two launch profiles (vis node or compute node). You also can adjust the number of requested nodes (Num nodes) and the (total) number of mpi tasks (Num procs) here. As JURECA offers 24 cores per node, the maximum usable number of mpi processes is 24*number_of_nodes (without SMT). However, as many visualisation algorithms do not scale very well, you may want to spawn fewer processes than the maximal number possible. You should experiment with these values (Num nodes, Num procs) to find out a good solution for your specific data and visualisation setup. It is a good idea to start with a setup as simple as possible (1 node, 2 procs) and to increase the values step by step. '''PLEASE DO USE MORE THAN 1 MPI TASK (num procs)! WHEN ONLY 1 MPI TASK IS SELECTED, VISIT STARTS THE SERIAL VERSION IF IT'S ENGINE WHICH AT THE MOMENT CRASHES WHEN TRYING TO DO HARDWARE RENDERING! '''\\ Select a launch profile and click on "OK". Now !VisIt automatically creates a temporary batch script on the JURECA login node (under /tmp) and starts this script with the sbatch command. Depending on your selection of the launch profile, this batch job is either executed on the vis partition (sbatch --partition=vis) or the batch partition (sbatch --partition=batch).\\ The scheduling of the batch job may take a while, so be patient and remember that the job can only be started if there are free resources on the cluster. === g) Tweaking the host profile You may want to adapt/optimise the predefined JURECA host profile to your needs for some reasons. Here are some ideas: * to allocate a vis node with 1024 GByte main memory, select the "vis node" launch profile, enter the "Parallel-->Advanced" tab and in "Lancher arguments" change the entry "--gres=mem512,gpu:2" to "--gres=mem1024,gpu:2" (BTW: the "launcher" on JURECA is just the sbatch command, so you can include all sbatch options here). * to restrict the number of mpi tasks per node to ''n'', just add "--ntasks-per-node=''n''" to the launcher arguments of either the "vis node" and/or "compute node" launch profile. * to set the wallclock time to, lets say, 2 hours, just add "--time=2:00:00" to the launcher arguments of either the "vis node" and/or "compute node" launch profile. The default wallclock time is 1 hour, the maximum is 24 hours.