jb_init

Name

jb_init - initializes the JBRT environment.

Synopsis

.B #include <mpi.h>
.B #include "jbcnl.h"

int jb_init( size_t mr_segment_size, uint16_t mr_segments);

Include with -I/bgsys/local/bgas/jbrt/jbcn/include, link with -L/bgsys/local/bgas/jbrt/jbcn/lib -ljbcn.

Description

jb_init() prepares the calling application for use of other JBCNL functions. Furthermore, it triggers initialization of the JBSD (i.e. the IO node daemon launched by jbrunjob in addition to the CN application) and establishes an RDMA connection between the two, which is then used by other JBCNL functions.

The arguments to jb_init() pertain to the size of the memory buffer allocated by jb_init() for explicit data transfers to the BGAS IO nodes and/or the HS4 cards. To be precise, the buffer size allocated for explicit data transfers is "mr_segment_size"*"mr_segments". A smaller buffer of 40 bytes is allocated for data transfers from the IO nodes; this buffer is used in a read-only manner by certain JBCNL functions. Note that the buffer size is determined on a per-process (i.e. per-MPI-task) basis, that is, POSIX or OpenMP threads share the buffer allocated by jb_init().

For technical reasons, those buffers are actually allocated as POSIX shared memory. Therefore, their aggregate size on a given compute node (("mr_segment_size"*"mr_segments"+40)*"ranks-per-node" bytes, where "ranks-per-node" is the "ranks-per-node" argument of runjob, i.e. the number of MPI tasks - not threads - per compute node) must not exceed the amount of POSIX shared memory available to the compute node. This amount is controlled by the BG_SHAREDMEMSIZE environment variable. The latter is set by default to 32 by jbrunjob; this default may be overridden by adding "--envs BG_SHAREDMEMSIZE=X" to your runjob command, where X is the desired value. (Note that BG_SHAREDMEMSIZE is given in megabytes.)

Furthermore, jb_init() abuses the user buffer for the initial handshake with the JBSD, which means that it also needs to have a "minimal" size. A reasonable upper bound for this minimal size is 4096 bytes, thus any value of "mr_segment_size"*"mr_segments" over 4096 should be fine.

jb_init() must be successfully called before any other JBCNL functions are called; if the user fails to do so, the behaviour of the subsequent JBCNL function calls is undefined.

jb_init() must be called after MPI_Init() because it uses MPI operations internally. Furthermore, if those MPI operations fail, the behaviour of jb_init() (and subsequent JBCNL function calls) is undefined. By default, MPI errors are fatal, which is unproblematic; the warning about undefined behaviour is meant to imply that the user should not use non-default MPI error handlers in his code.

jb_init() internally uses collective MPI operations and global MPI barriers. Thus, jb_init() itself is a globally blocking operation.

jb_init() is not threadsafe; it must be called by one thread only (MPI_Init behaves similarly in this respect).

jb_init() may crash the application if either a fatal MPI error or a failure of the JBSD occurs. Any crashes during jb_init() are proof of bugs in the JBRT; please report them to n.vandenbergen@….

Return values

jb_init() returns 0 on success. On error, -1 is returned and "errno" is set.

Errors

EFBIG

"mr_segment_size"*"mr_segments" is too small.

EINVAL

BG_SHAREDMEMSIZE was explicitly set in the jbrunjob command and is not a sufficiently small positive integer.

ENOMEM

"mr_segment_size"*"mr_segments" is too big, resp. BG_SHAREDMEMSIZE is too small.

Other errors pertain to malloc failures, failures of POSIX shared memory operations or ZeroMQ errors on the ION side. Those are proof of bugs in the JBRT. If an error different from EFBIG, EINVAL and ENOMEM occurs, please report to n.vandenbergen@….