== Compiling and running BGAS applications ==
=== General information ===
JBRT (Juelich BGAS RunTime) is available on the BGAS system since 08.09.2014.

=== User accounts and access ===
In order to use the JBRT and log in to the BGAS nodes, a Juqueen account is required with a special extension. Please contact one of the system administrators or project cordinators for details.

=== Location of the JBRT components ===
The JBRT with header (include) files, executables and library files is located under the following path, which can be accessed from any of the NFS-enabled (front-end, ION, or BGAS) nodes:

{{{
/bgsys/local/bgas/jbrt
}}}

Because of the multi-platform nature of JBRT (i.e. parts of it run on the FN, CNs and IONs), for ease of use, the main JBRT directory has been divided into subdirectories corresponding to the targeted part of the system:

{{{
ls -l /bgsys/local/bgas/jbrt
total 64
drwxrwxr-x 4 bgas bgas   512 Sep  4 15:10 jbcn
drwxrwxr-x 3 bgas bgas   512 Sep  4 15:12 jbfe
drwxrwxr-x 5 bgas bgas 32768 Sep  8 14:51 jbion
}}}

==== Compute node components (JBCN) ====

The //jbcn// directory contains all the CN components:

{{{
ls -l /bgsys/local/bgas/jbrt/jbcn
total 0
drwxrwxr-x 2 bgas bgas 512 Sep  8 16:11 include
drwxrwxr-x 2 bgas bgas 512 Sep  4 15:10 lib
}}}

In the //include// directory of JBCN, all header files required to be included by the CN application utilizing JBRT can be found:

{{{
ls -l /bgsys/local/bgas/jbrt/jbcn/include 
total 64
-rwxrwxr-x 1 bgas    bgas 531 Sep  4 15:10 jbcnl.h
-rw-r--r-- 1 vandenb zam  959 Sep  8 16:11 jbcnl_impl.h
}}}

Please note that currently only the //jbcnl.h// file needs to be included in the user application. The //jbcnl_impl.h// file is an implementation internal file and should not be used. In order to include JBCN header files into user application, the ///bgsys/local/bgas/jbrt/jbcn/include// path should be used.

The //lib// subdirectory of the JBCN contains the //libjbcn.a//, JBCN library:

{{{
ls -l /bgsys/local/bgas/jbrt/jbcn/lib
total 160
-rwxrwxr-x 1 bgas bgas 140574 Sep  4 15:10 libjbcn.a
}}}

This library needs to be linked to the user application running on the CNs, using the ///bgsys/local/bgas/jbrt/jbcn/lib// path for linking.

==== Front-end components (JBFE) ====

The //jbfe// directory contains all the FE components:

{{{
ls -l /bgsys/local/bgas/jbrt/jbfe
total 0
drwxrwxr-x 2 bgas bgas 512 Sep  8 14:36 bin
}}}

In the //bin// directory, the //jbrunjob// command is located. This command is used in the LoadLeveller script in substitute of the regular //runjob// command, to launch processes which should be executed on the CNs, and preparing the environment on the CNs and IONs for the CN-side application to launch the user code on the IONs using dedicated function calls from the library.

The other file, //jbrunjob.x//, is the implementation file and should not be used.


The //jbion// directory contains all the ION components:

{{{
ls -l /bgsys/local/bgas/jbrt/jbion
total 64
drwxrwxr-x 2 bgas bgas   512 Sep  8 15:07 bin
drwxrwxr-x 2 bgas bgas   512 Sep  8 14:52 etc
drwxrwxr-x 2 bgas bgas 32768 Sep  8 15:50 log
}}}

From the above, the //bin// directory contains the following implementation files, which should not be used by the user:

{{{
ls -l /bgsys/local/bgas/jbrt/jbion/bin 
total 1888
-rwxrwxr-x 1 bgas bgas  313393 Sep 16 11:36 jblauncher.x
-rwxrwxr-x 1 bgas bgas  302008 Sep 16 11:36 jbmd.x
-rwxrwxr-x 1 bgas bgas 1267910 Sep 16 11:36 jbsd.x
}}}

The //etc// directory is a place where the config files are located. Currently, it contains only the //zlog// config, which in principle should not be modified by the user without consultation with the development team:

{{{
ls -l /bgsys/local/bgas/jbrt/jbion/etc 
total 0
-rwxrwxr-x 1 bgas bgas 103 Sep  4 15:17 zlog.config
}}}

The //log// directory is where the system logs will be placed. Like above, it is most useful for the developers.

=== Writing a BGAS program ===

A sample CN-side program can be written as follows:

{{{
  1 #include <stdio.h>
  2 #include "mpi.h"
  3 #include "jbcnl.h"
  4 
  5 #define JB_BUFFER_SIZE  1048576
  6 #define JB_BUFFER_COUNT 1
  7 
  8 #define I_COUNT 128
  9 #define J_COUNT 8
 10 
 11 int main (int argc, char** argv) {
 12   int i  = 0;
 13   int j  = 0;
 14   int ic = I_COUNT;
 15   int jc = J_COUNT;
 16   int rc = 0;
 17   int ec = 0;
 18 
 19   int mpiRank = 0;
 20   int mpiSize = 0;
 21 
 22   int tag[ic];
 23 
 24   char* execvArgs[3];
 25 
 26   //printf("# Test starting\n");
 27 
 28   MPI_Init(&argc, &argv);
 29 
 30   rc = jb_init(JB_BUFFER_SIZE, JB_BUFFER_COUNT);
 31 
 32   //printf("# jb_init: %d\n", rc);
 33 
 34   MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank);
 35   MPI_Comm_size(MPI_COMM_WORLD, &mpiSize);
 36 
 37   for (i = 0; i < ic; i += jc) {
 38     for (j = 0; j < jc; j++) {
 39       if (i + j < ic) {
 40         printf("# Sending request %d of %d\n", i + j + 1, ic);
 41 
 42         asprintf(&execvArgs[0], "%d", mpiRank);
 43         asprintf(&execvArgs[1], "%d", mpiSize);
 44         asprintf(&execvArgs[2], "%d", i);
 45 
 46         tag[j] = jb_execv("/path-to-the-ion-executable/ion-executable.x", execvArgs);
 47 
 48         //printf("# jb_execv[%d]: %d\n", i, tag[j]);
 49       }
 50     }
 51 
 52     for (j = 0; j < jc; j++) {
 53       if (i + j < ic) {
 54         rc = -1;
 55 
 56         printf("# MPI rank %d of %d waiting for request %d of %d\n", mpiRank, mpiSize, i + j + 1, ic);
 57 
 58         while (rc < 0) {
 59           rc = jb_execv_status(tag[j]);
 60 
 61           sleep(1);
 62         }
 63 
 64         if (rc != 0) ec++;
 65 
 66         //printf("# jb_execv_status[%d]: %d\n", i, rc);
 67       }
 68     }
 69   }
 70 
 71   jb_finalize();
 72 
 73   MPI_Finalize();
 74 
 75   printf("# Test done, errors: %d\n", ec);
 76 
 77   return 0;
 78 }
}}}

On the ION-side (BGAS node), a corresponding sample application can be written as in the following example:

{{{
  1 #include <stdio.h>
  2 #include <stdlib.h>
  3 #include <time.h>
  4 
  5 #define LOGS_DIRECTORY "/path-to-the-user-logs"
  6 
  7 int main(int argc, char** argv) {
  8   char hostName[256];
  9   char currentTime[128];
 10   char currentDate[128];
 11   char logFileName[512];
 12 
 13   FILE *LOGFILE;
 14 
 15   struct tm *tm;
 16 
 17   time_t timeT;
 18 
 19   timeT = time(NULL);
 20 
 21   tm = localtime(&timeT);
 22 
 23   strftime(currentDate, sizeof(currentDate), "%Y-%m-%d", tm);
 24   strftime(currentTime, sizeof(currentTime), "%H:%M:%S", tm);
 25 
 26   gethostname(hostName, 256);
 27 
 28   //snprintf(logFileName, sizeof(logFileName), "%s/%s-%s-%s.log", LOGS_DIRECTORY, hostName, currentDate, currentTime);
 29   snprintf(logFileName, sizeof(logFileName), "%s/%s.log", LOGS_DIRECTORY, hostName);
 30 
 31   LOGFILE = fopen(logFileName, "a+");
 32 
 33   if (LOGFILE== NULL) {
 34     printf("# Error opening the log file\n");
 35 
 36     return 1;
 37   }
 38 
 39   fprintf(LOGFILE, "%s %s Host name: %s, process rank %d of %d, loop %d\n", currentDate, currentTime, hostName, atoi(argv[1]), atoi(argv[2]), atoi(argv[3]));
 40 
 41   fclose(LOGFILE);
 42 
 43   return 0;
 44 }
}}}

=== Compiling a BGAS program ===

In order to compile a BGAS application, some understanding of the system architecture is required. The main point to note is the target architecture. Application running on the CN-side, and the application running on the ION-side should be compiled accordingly, using CN and ION header files and libraries.

Required //jbcn// header files can be found in ///bgsys/local/bgas/jbrt/jbcn/include//, while the CN library is located in ///bgsys/local/bgas/jbrt/jbcn/lib//. Those have to be used to make use of the remote process execution on IONs triggered by the CN application.

Currently, no specific including or linking on the ION-side is required.

A sample Makefile for the CN-side application can look as follows:

{{{
  1 MPICC = mpigcc
  2 
  3 CCFLAGS = -O3 -g
  4 LDFLAGS =
  5 
  6 JBRTDIR = /bgsys/local/bgas/jbrt
  7 
  8 INC = -I$(JBRTDIR)/jbcn/include
  9 LIB = -L$(JBRTDIR)/jbcn/lib -ljbcn
 10 
 11 TARGET = cn-executable.x
 12 
 13 SRC = $(wildcard *.c)
 14 OBJ = $(SRC:.c=.o)
 15 
 16 all = $(TARGET) $(SRC)
 17 
 18 .c.o:
 19   $(MPICC) $(CCFLAGS) $(INC) -c $< -o $@
 20 
 21 $(TARGET): $(OBJ)
 22   $(MPICC) $(LDFLAGS) $(OBJ) -o $@ $(LIB)
 23 
 24 clean:
 25   rm *.o
 26   rm *.x
}}}

An example Makefile for the ION-side application can look as follows:

{{{
 1 MPICC = mpigcc
  2 
  3 CCFLAGS = -O3 -g
  4 LDFLAGS =
  5 
  6 JBRTDIR = /bgsys/local/bgas/jbrt
  7 
  8 INC =
  9 LIB =
 10 
 11 TARGET = ion-executable.x
 12 
 13 SRC = $(wildcard *.c)
 14 OBJ = $(SRC:.c=.o)
 15 
 16 all = $(TARGET) $(SRC)
 17 
 18 .c.o:
 19   $(MPICC) $(CCFLAGS) $(INC) -c $< -o $@
 20 
 21 $(TARGET): $(OBJ)
 22   $(MPICC) $(LDFLAGS) $(OBJ) -o $@ $(LIB)
 23 
 24 clean:
 25   rm *.o
 26   rm *.x
}}}

=== Executing a BGAS program ===

To execute a BGAS program, the //jbrunjob// command has to be used in the LoadLeveller jobscript. This command is a special wrapper of the standard //runjob// command and supports the same set of arguments. The //jbrunjob// command can be found in the front-end part of the JBRT: ///bgsys/local/bgas/jbrt/jbfe//.

In the current setup, a special reservation is required, to ensure proper binding between CNs and BGAS nodes, and allow the system administrators to monitor the job's status. To request a reservation, contact one of the system administrators or project coordinators.

Below is an example of a LoadLeveller script:

{{{
  1 # @ job_name = BGAS_JOB_NAME
  2 # @ job_type = bluegene
  3 # @ comment = "BGL Job by Size"
  4 # @ error = $(job_name).$(jobid).out
  5 # @ output = $(job_name).$(jobid).out
  6 # @ environment = COPY_ALL;
  7 # @ wall_clock_limit = 00:10:00
  8 # @ notification = never
  9 # @ notify_user = your-email-address@your-domain
 10 # @ bg_size = 512
 11 # @ ll_res_id = your-reservation-id
 12 # @ class = system
 13 # @ queue
 14 
 15 /bgsys/local/bgas/jbrt/jbfe/bin/jbrunjob -n 8 -p 1 --envs BG_SHAREDMEMSIZE=32 : cn-executable.x
}}}

'''Addendum:''' It is now possible to use the JBRT on non-BGAS (i.e. standard production) IO nodes; however, no (HS4) flash memory is available and no ION-ION communication may be performed in this setup. To do so, comment out the "ll_res_id" and "class" lines (lines 11 and 12) in the example script above. No reservation is required in this case.

=== Troubleshooting ===

In case of any issues with the functionality of JBRT, contact **n.vandenbergen@fz-juelich.de**, **m.foszczynski@fz-juelich.de** or **m.stephan@fz-juelich.de**.