= JUMEL (JUROPA MEmory Logger) =

== General information ==
The JUROPA memory logger is intended for monitoring the memory usage of applications on NUMA architectures (especially JUROPA). It consists currently of two Python scripts:
  * `jumel` (JUROPA Memory Logger), the actual logger
  * `juman` (JUROPA Memory ANalyzer), a postprocessing tool

== Concept ==
The logger is started with `mpiexec` and subsequently starts the application to monitor. The monitoring is done per task and/or per node. In time steps information are gathered from files provided by the operating system or from commands that are issued by the logger itself. Currently, the following resources are used:

  * Monitoring by task:
    * file `/proc/<PID>/status` with keys
      * `VmExe`
      * `VmSt`
      * `VmData`
      * `VmSize`
      * `VmLck`
      * `VmLib`
      * `VmRSS`
  
  * Monitoring per node
    * command `vmstat` with keys
      * MFree
      * TWait
      * Idle
      * TDead
      * UsedUs
      * UsedKe

When monitoring per task each task writes the value for each key to the file `.memlog/task<MPI-rank>.log` and waits for the next time step. When monitoring by node the process running on core 0 of each node writes the value for each key to the file `node<node-name>_task<MPI-task>.log`.

After the run is finished the analyzer `juman` is run in the `PBS_O_WORKDIR`, either from within the same job script or afterwards on the login node to analyze the consumed resources. It will create graphs with the value of key (default: `VmSize`, => `key`, option `-k`) for each task at each time step (use `juman -k help` for a list of available keys), the process with the maximum value of the key at each time step and the total sum of all values of a key across all tasks at each time step (=> `statistics`, option `-s`). If `-i` is specified the graphs are displayed immediately.

== Usage ==
In order to get an overview of the valid options please use

{{{
jumel -u
juman -u
}}}

Defaults:
  * work directory: `.memlog`
  * delay: 10 seconds
  * all ranks log the memory consumption
  * resources logged: `/proc/<id>/status: VmSize, VmRSS, VmData, VmStk`

Suppose the application to monitor is started usually as follows:

{{{
mpiexec -np 32 -e APP_ROOT app.x -i app.inp > my.out
}}}

To start the application with jumel use


{{{
mpiexec -np 32 -e PBS_ID,APP_ROOT jumel -a "app.x -i app.inp" > my.out
}}}

The variable `PBS_ID` needs not to be specified, however it will be displayed in the `jumel` logfiles and eases the tracking of the runs afterwards (e.g. when looking for the job in the system logfiles). Once the run is finished run `juman` in the same directory:

{{{
juman -s all -i
}}}

The -s option switches on the statistics (currently: total consumption and maximum consumption per time step) and the `-i` option will start the graphical display of the results. Currently only a `gnuplot` interface is implemented and postscript of `xfig` files can be generated. An interface for visualization with Python is planned.

== Example: Namd ==
The following job script was used to monitor running the apoa1 benchmark with Namd:

{{{
#!/bin/bash 
#MSUB -l nodes=4:ppn=8
#MSUB -l walltime=00:15:00
#MSUB -v tpt=1 

module load namd/2.7

mpiexec -np 32 -e PBS_JOBID jumel -a "$NAMD_ROOT/bin/namd2 apoa1.namd"

}}}

The graphs were obtained afterwards on the login node with

{{{
juman -s all -i
}}}

== Results ==
Below the results for the Namd runs are shown for `VmSize` (default key). The values reported are in kB.[[BR]]

 
[[Image(source:/examples/Namd/VmSize.png,720px)]]
[[Image(source:/examples/Namd/VmSize_maxmin.png,720px)]]
[[Image(source:/examples/Namd/VmSize_total.png,720px)]]

== Example Mapt ==
{{{
#!/bin/bash 
#MSUB -l nodes=2:ppn=8
#MSUB -l walltime=00:05:00
#MSUB -v tpt=1 


mpiexec -np 16 jumel -d 2 -a "mapt.x" > mapt.out

}}}

The graphs were obtained afterwards on the login node with

{{{
juman -s all -i
}}}

== Results ==
Below the results for the Namd runs are shown for `VmSize` (default key). The values reported are in kB.[[BR]]

[[Image(source:/examples/Mapt/VmSize.png,720px)]]
[[Image(source:/examples/Mapt/VmSize_maxmin.png,720px)]]
[[Image(source:/examples/Mapt/VmSize_total.png,720px)]]

== SVN Access ==
{{{
svn list https://svn.version.fz-juelich.de/jumel
}}}