Changes between Version 2 and Version 3 of Examples/Ear5Animating
- Timestamp:
- 07/09/18 17:56:13 (6 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
Examples/Ear5Animating
v2 v3 1 1 = Visualisation and Animation of ERA5 Reanalysis Climate Data = 2 == Prerequisites == 3 === NetCDF/HDF5 (on JURECA) === 2 == Dataformats and Tools == 3 === NetCDF and HDF5 on JURECA === 4 As HDF5 is the preferred dataformat for ParaView, maybe a conversion from netCDF-3 to netCDF-4 (which in fact is a HDF5 file) is necessary. So one has to deal with som netcdf and hdf5 tools. 5 4 6 How to find out what modules to load for NetCDF/HDF5: 5 7 {{{ … … 10 12 11 13 Load modules e.g. by: 12 13 14 {{{ 14 15 #!bash … … 18 19 }}} 19 20 20 Just some examples of ho tto use h5dump:21 Just some examples of how to use h5dump: 21 22 {{{ 22 23 #!bash … … 45 46 46 47 === Python on JURECA === 47 To use Python on JURECA, you have to load some modules first, e.g. 48 To use Python on JURECA, you have to load some modules first. Here are some useful commands: 49 {{{ 50 #!bash 51 module spider python 52 }}} 48 53 {{{ 49 54 #!bash … … 55 60 The files are stored at /data/slmet/slmet111/met_data/ecmwf/era5/netcdf4/2017/ 56 61 57 The Files cover a period of time for two month:June and August 2017 in 1 h steps.62 The files cover the period of time for the month June and August 2017 in 1 h steps. 58 63 Example of a filename: 2017061516_ml.nc (YYYYMMDDHH) 59 64 60 65 Some interesting variables stored in the *ml-files: 61 cc (1 x 137 x 601 x 1200): Fraction of cloud cover62 ciwc (1 x 137 x 601 x 1200): Specific cloud ice water content63 clwc (1 x 137 x 601 x 1200): Specific cloud liquid water content64 d (1 x 137 x 601 x 1200): divergence_of_wind65 o3 (1 x 137 x 601 x 1200): Ozone mass mixing ratio66 q (1 x 137 x 601 x 1200): Specific humidity67 t (1 x 137 x 601 x 1200): Temperature68 u (1 x 137 x 601 x 1200): U component of wind (eastward wind)69 v (1 x 137 x 601 x 1200): V component of wind (northward wind)70 w (1 x 137 x 601 x 1200): Vertical velocity71 vo (1 x 137 x 601 x 1200): Vorticity (relative)66 - cc (1 x 137 x 601 x 1200): Fraction of cloud cover 67 - ciwc (1 x 137 x 601 x 1200): Specific cloud ice water content 68 - clwc (1 x 137 x 601 x 1200): Specific cloud liquid water content 69 - d (1 x 137 x 601 x 1200): divergence_of_wind 70 - o3 (1 x 137 x 601 x 1200): Ozone mass mixing ratio 71 - q (1 x 137 x 601 x 1200): Specific humidity 72 - t (1 x 137 x 601 x 1200): Temperature 73 - u (1 x 137 x 601 x 1200): U component of wind (eastward wind) 74 - v (1 x 137 x 601 x 1200): V component of wind (northward wind) 75 - w (1 x 137 x 601 x 1200): Vertical velocity 76 - vo (1 x 137 x 601 x 1200): Vorticity (relative) 72 77 73 78 Variables related to coordinates:\\ 74 lat (601): latitude (degrees north), ranging from 90 to -9075 lon (1200): longitude (degrees east), ranging from 0 to 359.776 lev, lev_2 (137): hybrid_sigma_pressure, ranging from 1 to 137 (137 is ground level!)79 - lat (601): latitude (degrees north), ranging from 90 to -90 80 - lon (1200): longitude (degrees east), ranging from 0 to 359.7 81 - lev, lev_2 (137): hybrid_sigma_pressure, ranging from 1 to 137 (137 is ground level!) 77 82 78 83 Calculation of Coordinates:\\ 79 ParaView does not understand the original coordinates (lat, lon, lev_2) this way. Therefore, these must be converted into a "structured grid" data structure. See the "generate_coordinates.py" script.80 Here also aconversion to Cartesian coordinates takes place essentially via:84 !ParaView natively does not understand the original coordinates (lat, lon, lev_2). Therefore, these must be converted into a "structured grid" data structure. See the the "generate_coordinates.py" script. 85 In this script also the conversion to Cartesian coordinates takes place essentially via: 81 86 {{{ 82 87 #!python … … 87 92 }}} 88 93 The generated coordinates are stored in the new created file "coordinates.h5". 89 ATTENTION: ParaView can read this "structured grid", but cannot volume-render, as volume rendering only works good for image data. Therefore, the filter "Resample to Image" must be applied to the reader inParaView!94 ATTENTION: !ParaView can read this "structured grid", but cannot volume-render it, as volume rendering only works good for data of type "image data". Therefore, the filter "Resample to Image" must be applied to the reader in !ParaView! 90 95 91 Create XDMF file s:\\92 The hdf5 files are read via an xdmf reader. To enable this, an xdmf file must be created. The script "make_xdmf.py" does this. The script essentially scans the directory where the files are located and gets the names of all files for one month (the month is fixed in the script).93 Variables that can be read into ParaView later are noted in the script via:96 Create XDMF file:\\ 97 The hdf5 files are loaded via an xdmf reader. To enable this, an xdmf file must be created first. The script "make_xdmf.py" does this. The script essentially scans the directory where the files are located and gets the names of all files for one month (the month is fixed in the script). 98 Variables that can be later loaded into !ParaView are noted in the script in a python list: 94 99 {{{ 95 100 #!python … … 100 105 == !ParaView == 101 106 === Loading the necessary modules === 102 The modules can be found out with "module spider ParaView/5.5.0".107 The modules can be found out with "module spider !ParaView/5.5.0". 103 108 Load modules e.g. with: 104 109 {{{ … … 108 113 109 114 === ParaView GUI === 110 First load the modules , then start !ParaView GUI115 First load the modules as described above, then start !ParaView GUI 111 116 {{{ 112 117 #!bash 113 118 vglrun paraview 114 119 }}} 115 The GUI is well suited to prototype the scene , i.e. to define the visualization pipeline with its parameters, i.e. the filters, the color tables and the camera positions. However, for various reasons it makes sense to script the visualization in !ParaView:120 The GUI is well suited to prototype the scene. In the GUI one can define the visualization pipeline with its parameters, i.e. the readers and filters, the color tables and the camera positions. However, for various reasons it makes sense to script the visualization with paraview-python: 116 121 - In the script all parameters are recorded in text form 117 122 - Loading !ParaView GUI states sometimes does not work 118 - !ParaView has a memory leak, so after a few render steps you have to quit !ParaView and restart it at the aborted location. This can be automated using a script.123 - !ParaView has a memory leak, so after a number of render steps you have to quit !ParaView and restart it at the aborted location (otherwise !ParaView would crash). This can be automated using a script. 119 124 120 125 121 126 === From GUI to Script === 122 How -To transfer pipeline parameters from GUI to script:\\127 How To transfer pipeline parameters from GUI to script:\\ 123 128 In the !ParaView-GUI, start a Python trace by Tools->Start Trace. 124 Then create the pipeline you want. The corresponding Python commands are displayed in the trace. These can be transferred into a script with copy & paste.129 Then create the pipeline you want. (Most of) the corresponding Python commands are displayed in the trace. These can be transferred into a script with copy & paste. 125 130 126 How -To transfer colormaps from GUI to script:\\127 Once you have designed a good colormap , you can save it as a preset. This preset can then be renamed and saved to disk as*.json file.128 Since a different colormap makes sense for each variable, in the example the naming scheme for colormap files is "stein_''variable''.json", e.g. "stein_vo.json" for the vorticity. This naming scheme is expected in the Python scripts, which among other things load the color tables131 How To transfer colormaps from GUI to script:\\ 132 Once you have designed a good colormap in the GUI, you can save it there as a preset. This preset can then be renamed and saved to disk as a *.json file. 133 Since a different colormap makes sense for each variable, one will end up with mor then one colormap file. In this example the naming scheme for colormap files is "stein_''variable''.json", e.g. "stein_vo.json" for the vorticity. This naming scheme is expected in the Python scripts, which among other things load the color tables. 129 134 130 How -To transfer camera parameter from GUI to script:\\131 You can save four camera positions in !ParaView. Click on the camera icon ("Adjust Camera"), then "configure", then "Assign current view". The camera positions can be saved in an XML file via "export" and can later be read inand used in the Python script e.g. with:135 How To transfer camera parameter from GUI to script:\\ 136 You can save four (and more) camera positions in the !ParaView GUI. Click on the camera icon ("Adjust Camera"), then "configure", then "Assign current view". The camera positions can be saved in an XML file via "export" and can later be loaded and used in the Python script e.g. with: 132 137 {{{ 133 138 #!python … … 146 151 }}} 147 152 153 As you can see above, the script expects the naming convention "camera_''variable''.pvcvbc", e.g. "camera_vo.pvcvbc" for the vorticity. 148 154 155 === How-To start a !ParaView script === 156 Generally a !ParaView Python script is started with "pvpython script.py" (or, on a VNC-server, by "vglrun pvpython script.py"). On JURECA, however, the following approach is necessary: first start a pvserver on display :0.0 and let pvpython connect to this server. This can be done in one command line: 157 {{{ 158 #!bash 159 bash -c 'export DISPLAY=:0.0 && pvserver --disable-xdisplay-test' & sleep 2; see pvpython ./script.py 160 }}} 149 161 162 == Sample session for using the scripts: == 163 In the sample session, the typical usage of the scripts is demonstrated. All necessary files (scripts, colormaps, camera positions, texture) should be copied into the same directory. At the end, 10 images for the two variables "ciwc" and "clwc" should be generated. 164 165 === Prerequisites === 166 - Data is located as described above in /data/slmet/slmet111/met_data/ecmwf/era5/netcdf4/2017/ 167 - On JURECA, the modules for Python are loaded, e.g. with "module use otherstages && modules --force purge && module load !Stages/Devel-2017b GCC/7.2.0 ParaStationMPI/5.2.0-1 h5py/2.7.1-Python-2.7.14" 168 - The following scripts are installed: generate_coordinates.py, make_xdmf.py, make_movie_pvserver.sh, paraview_make_movie_pvserver.py 169 - Color tables for two attributes are installed: stein_ciwc.json and stein_clwc.json 170 - Camera positions for two attributes are installed: camera_ciwc.pvcvbc and camera_clwc.pvcvbc 171 - Texture earth_heller_transformed.jpg is installed 172 173 === Step 1: Create coordinates === 174 {{{ 175 #!bash 176 python ./generate_coordinates.py 177 }}} 178 As a result, the file "./coordinates.h5" should be generated. 179 180 === Step 2: Create XDMF file === 181 {{{ 182 #!bash 183 python ./make_xdmf.py 184 }}} 185 As a result, the file "./structuredgrid_201709.xdmf" should be created. 186 187 === Step 3: Render images === 188 {{{ 189 #!bash 190 ./make_movie_pvserver.sh 191 }}} 192 Rendering takes about 10 minutes. As a result, 10 images for the variables "ciwc" and "clwc" should be generated. The images can be viewed on JURECA with "/usr/local/jsc/etc/xdg/scripts/launch_gpicview.sh image*.jpg".
