Changes between Version 3 and Version 4 of Examples/Ear5Animating


Ignore:
Timestamp:
07/10/18 10:30:57 (6 years ago)
Author:
Herwig Zilken
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Examples/Ear5Animating

    v3 v4  
    22== Dataformats and Tools ==
    33=== NetCDF and HDF5 on JURECA ===
    4 As HDF5 is the preferred dataformat for ParaView, maybe a conversion from netCDF-3 to netCDF-4 (which in fact is a HDF5 file) is necessary. So one has to deal with som netcdf and hdf5 tools.
     4The typical data format in climate science is NetCDF. Though NetCDF files can be loaded in !ParaView, this approach is very unflexible, as !ParaViews NetCDF reader makes some fixed assumptions regarding some naming conventions of the variables.
     5Fortunately, NetCDF files geenrated with the newer NetCDF version 4 are based on HDF5 and can be loaded with !ParaViews [http://xdmf.org XDMF] reader.
     6In case one has older NetCDF files of version 3, a conversion from NetCDF-3 to NetCDF-4 is necessary. So one has to deal with som NetCDF and HDF5 related tools.
    57
    6 How to find out what modules to load for NetCDF/HDF5:
     8How to find out what modules to load for NetCDF/HDF5 on JURECA:
    79{{{
    810#!bash
    … …  
    1921}}}
    2022
    21 Just some examples of how to use h5dump:
     23Just as a reminder some examples of how to use h5dump:
    2224{{{
    2325#!bash
    … …  
    3234ncdump -k foo.nc
    3335}}}
    34 If ncdump returns netCDF-4, or netCDF-4  classic  model, then congratulations, you already have an HDF file, as netCDF-4 is the netCDF data model implemented using HDF5 as the storage layer. These files can be ready by the HDF library version 1.8 or later, and from what I can tell, pytables. If ncdump returns classic or 64-bit offset, then you are using netCDF-3 and will need to convert to netCDF-4 (and thus HDF). The good news is that if you have netCDF installed, you can do the conversion pretty easily:
     36If ncdump returns netCDF-4, or netCDF-4  classic  model, then congratulations, you already have an HDF file, as netCDF-4 is the netCDF data model implemented using HDF5 as the storage layer. These files can be ready by the HDF library version 1.8 or later, and from what I can tell, pytables. If ncdump returns classic or 64-bit offset, then you are using netCDF-3 and will need to convert to netCDF-4 (and thus HDF5). The good news is that if you have netCDF installed, you can do the conversion pretty easily:
    3537{{{
    3638#!bash
    … …  
    3840}}}
    3941
    40 Graphical hdf5 viewer:
     42Graphical HDF5 viewer:
    4143HDFView is a very handy tool to investigate hdf5 files. To open HDFView on JURECA, either click on the HDFView icon on the desktop. Or launch HDFView by
    4244{{{
    … …  
    4648
    4749=== Python on JURECA ===
    48 To use Python on JURECA, you have to load some modules first. Here are some useful commands:
     50To use python on JURECA, you have to load some modules first. Here are some useful commands:
    4951{{{
    5052#!bash
    … …  
    6062The files are stored at /data/slmet/slmet111/met_data/ecmwf/era5/netcdf4/2017/
    6163
    62 The files cover the period of time for the month June and August 2017 in 1 h steps.
    63 Example of a filename: 2017061516_ml.nc (YYYYMMDDHH)
     64The files cover the months June and August 2017 in 1 h steps.
     65For every file, its date is recorded in its filename. e.g. 2017061516_ml.nc (YYYYMMDDHH)
    6466
    65 Some interesting variables stored in the *ml-files:
     67Some interesting variables stored in the 3D *ml-files:
    6668- cc (1 x 137 x 601 x 1200): Fraction of cloud cover
    6769- ciwc (1 x 137 x 601 x 1200): Specific cloud ice water content
    … …  
    8183- lev, lev_2 (137): hybrid_sigma_pressure, ranging from 1 to 137 (137 is ground level!)
    8284
    83 Calculation of Coordinates:\\
    84 !ParaView natively does not understand the original coordinates (lat, lon, lev_2). Therefore, these must be converted into a "structured grid" data structure. See the the "generate_coordinates.py" script.
     85Calculation of coordinates:\\
     86!ParaView natively does not understand the original coordinates (lat, lon, lev_2). Therefore, these must be converted into a "structured grid" data structure, where every node of the 3D grid has its own 3D coordinate. This calculation is done in the "generate_coordinates.py" script.
    8587In this script also the conversion to Cartesian coordinates takes place essentially via:
    8688{{{
    … …  
    9294}}}
    9395The generated coordinates are stored in the new created file "coordinates.h5".
    94 ATTENTION: !ParaView can read this "structured grid", but cannot volume-render it, as volume rendering only works good for data of type "image data". Therefore, the filter "Resample to Image" must be applied to the reader in !ParaView!
     96ATTENTION: !ParaView can read this "structured grid", but cannot volume-render it, as volume rendering with [https://www.ospray.org/ OspRay] or GPU-based only works for data of type "image data", wich is a 3D rectiliniear grid. Therefore, the filter "Resample to Image" must be applied to the reader in !ParaView!
    9597
    9698Create XDMF file:\\
    97 The hdf5 files are loaded via an xdmf reader. To enable this, an xdmf file must be created first. The script "make_xdmf.py" does this. The script essentially scans the directory where the files are located and gets the names of all files for one month (the month is fixed in the script).
    98 Variables that can be later loaded into !ParaView are noted in the script in a python list:
     99The hdf5 files are loaded via an xdmf reader. To enable this, an xdmf file must be created first, containing information about alle time steps and all variables. The script "make_xdmf.py" does this. The script essentially scans the directory where the data files are located and gets the names of all files for one month (the definition of the month is fixed in the script).
     100Variables that can be later loaded into !ParaView are noted in the script in form of a python list:
    99101{{{
    100102#!python
    101  scalars=["cc", "ciwc", "clwc", "q", "d", "vo", "o3", "w"]]
     103 scalars=["cc", "ciwc", "clwc", "q", "d", "vo", "o3", "w"]
    102104}}}
    103105The name of the xdmf-output file is structuredgrid_201709.xdmf.
    … …  
    105107== !ParaView ==
    106108=== Loading the necessary modules ===
    107 The modules can be found out with "module spider !ParaView/5.5.0".
     109The needed modules can be found out with "module spider !ParaView/5.5.0".
    108110Load modules e.g. with:
    109111{{{
    … …  
    119121}}}
    120122The GUI is well suited to prototype the scene. In the GUI one can define the visualization pipeline with its parameters, i.e. the readers and filters, the color tables and the camera positions. However, for various reasons it makes sense to script the visualization with paraview-python:
    121 - In the script all parameters are recorded in text form
    122 - Loading !ParaView GUI states sometimes does not work
     123- In the script all parameters are recorded in text form, so one can look up these parameters later
     124- Loading !ParaView GUI state files sometimes does not work
    123125- !ParaView has a memory leak, so after a number of render steps you have to quit !ParaView and restart it at the aborted location (otherwise !ParaView would crash). This can be automated using a script.
    124126
    … …  
    127129How To transfer pipeline parameters from GUI to script:\\
    128130In the !ParaView-GUI, start a Python trace by Tools->Start Trace.
    129 Then create the pipeline you want. (Most of) the corresponding Python commands are displayed in the trace. These can be transferred into a script with copy & paste.
     131Then create the pipeline you want. (Most of) the corresponding Python commands are displayed in the trace. These can be transferred into a script with copy & paste and, if needed, modified there.
    130132
    131133How To transfer colormaps from GUI to script:\\
    132134Once you have designed a good colormap in the GUI, you can save it there as a preset. This preset can then be renamed and saved to disk as a *.json file.
    133 Since a different colormap makes sense for each variable, one will end up with mor then one colormap file. In this example the naming scheme for colormap files is "stein_''variable''.json", e.g. "stein_vo.json" for the vorticity. This naming scheme is expected in the Python scripts, which among other things load the color tables.
     135Since a different colormap makes sense for each variable, one will end up with more then one colormap file. In this example the naming scheme for colormap files is "stein_''variable''.json", e.g. "stein_vo.json" for the vorticity. This naming scheme is expected in the Python scripts, which among other things load the color tables.
    134136
    135 How To transfer camera parameter from GUI to script:\\
     137How To transfer camera parameters from GUI to script:\\
    136138You can save four (and more) camera positions in the !ParaView GUI. Click on the camera icon ("Adjust Camera"), then "configure", then "Assign current view". The camera positions can be saved in an XML file via "export" and can later be loaded and used in the Python script e.g. with:
    137139{{{
    … …  
    154156
    155157=== How-To start a !ParaView script ===
    156 Generally a !ParaView Python script is started with "pvpython script.py" (or, on a VNC-server, by "vglrun pvpython script.py"). On JURECA, however, the following approach is necessary: first start a pvserver on display :0.0 and let pvpython connect to this server. This can be done in one command line:
     158Generally a !ParaView python script is started with "pvpython script.py" (or, on a VNC-server, like JURECA, by "vglrun pvpython script.py"). On JURECA, however, the following approach is necessary: first start a pvserver on display :0.0 and let pvpython connect to this server. This can be done in one command line:
    157159{{{
    158160#!bash
    … …  
    165167=== Prerequisites ===
    166168- Data is located as described above in /data/slmet/slmet111/met_data/ecmwf/era5/netcdf4/2017/
    167 - On JURECA, the modules for Python are loaded, e.g. with "module use otherstages && modules --force purge && module load !Stages/Devel-2017b GCC/7.2.0 ParaStationMPI/5.2.0-1 h5py/2.7.1-Python-2.7.14"
    168 - The following scripts are installed: generate_coordinates.py, make_xdmf.py, make_movie_pvserver.sh, paraview_make_movie_pvserver.py
    169 - Color tables for two attributes are installed: stein_ciwc.json and stein_clwc.json
    170 - Camera positions for two attributes are installed: camera_ciwc.pvcvbc and camera_clwc.pvcvbc
    171 - Texture earth_heller_transformed.jpg is installed
     169- On JURECA, the modules for Python and H5PY are loaded, e.g. with "module use otherstages && modules --force purge && module load !Stages/Devel-2017b GCC/7.2.0 ParaStationMPI/5.2.0-1 h5py/2.7.1-Python-2.7.14"
     170- The following scripts are available: generate_coordinates.py, make_xdmf.py, make_movie_pvserver.sh, paraview_make_movie_pvserver.py
     171- Color tables for two attributes are available: stein_ciwc.json and stein_clwc.json
     172- Camera positions for two attributes are available: camera_ciwc.pvcvbc and camera_clwc.pvcvbc
     173- Texture earth_heller_transformed.jpg is available
    172174
    173175=== Step 1: Create coordinates ===