== Software Documentation == === Overview === This library is developed for connecting a master LDAP server (MLS) of the Human Brain Project (HBP) with the accounting data managed by the Juelich Supercomputing Centre (JSC). User data is filled into the MLS, queried with this library and finally integrated into JSC accounting data (JAD). The library is intended as one way synchronization. I.e. MLS contains the true data, which needs to be applied to the JSC data bases. New user accounts in MLS, new groups or any modifications of them should be added or accordingly modified in JAD. === Data flow === The data flow is depicted by the diagram DataFlow.dia. [[Image(DataFlow.png, 900px)]] The HBP LDAP server contains user and group data. The user data contains data relevant for d_pers and data relevant for the SC LDAP server. Thus, the user data from the HBP LDAP is split into d_pers data and SC LDAP data and then compared with the appropriate JSC data. HBP groups and HBP user accounts are prefixed with bp0. Account names follow the schema bp0xxxxx, with x=alphanumerical character with GID and UID in the range 1600000 to 1789999 For all accounts this data is provided by the HBP LDAP server. The group comparison is done by the library lib/GroupManagement.php, while any user data comparison is handled by lib/UserManagement.php (d_pers and SC LDAP data). There are no automatic changes possible for the JSC groups. Thus, only a mail is sent to the dispatch telling them about the required change. Groups are compared on the attributes GIDnumber, cn (group name) and hostnames (list of hosts, on which they are active). If any attribute value differs between HBP and JSC data, this difference is sent by mail. The same applies for new and removed groups. This cannot yet be done automatically and then a simple mail is sent. The data relevant for the d_pers table can be synchronized entirely automatically. New records can be created, existing modified and deleted. The user data stored in the SC LDAP can be partly updated automatically. For new user accounts an account application is generated and stored in the d_account_antrag table. This includes, that a user agreement is sent to the appropriate mail address. The dispatch can view the account application from https://dispatch.fz-juelich.de:8812/dispatch_account_req/DISPATCH_ZAM where all account applications are listed. As soon as the user agreement is signed and returned to the dispatch, the account can be joined. The record in d_account_antrag contains the login name, GID, UID and project name for the account. All these values are also available from the user agreement. Note, that one account application is created for each host (juqueen/jureca) and for each project. If user data in the SC LDAP needs to be modified or deleted, this needs to be done manually. So again a mail is sent to dispatch. === LDAP server structures === For a comparison of the structures of the HBP LDAP server and the JSC LDAP server have a look at the diagram /doc/LDAP_v2.dia. [[Image(LDAP_v2.png, 900px)]] It summarizes the branches of the LDAP servers and their functions. Since the HBP master LDAP server stores both personal and user account data, the structure is more complex than the JSC structure. The personal data for the JSC accounts is stored in the d_pers table. While the objectclass is used at JSC to decide, to which system an account is joined to, this is more complex for the HBP server. The manager attribute is replaced. To determine the sites, a user has access to, use the connection between users and groups via memberUid. The sites for each group can be detected by checking for systems, which have a hbpproject attribute with the group's name. The hbpsite attribute contains the site name, where each HPC system is located. === Account application from user perspective === See diagram UseCases.dia for an overview of steps for creating new accounts. [[Image(UseCases.png, 900px)]] A HBP user needs to apply for an account through the Collaboratory, a web portal for all HBP users, which offers access to all HBP services and platforms. The account application leads to an entry in the HBP LDAP server. The hbpaccounting application run in crontab detects the new account application and sends a user agreement to the user. He signs it and sends it back to the dispatch. Currently there is only one test project called bp0. Carsten Karbach is the lead of that project at JSC and needs to sign the agreement, too. With all the signatures the agreement is given to dispatch and the account is joined. Users are notified about the account activation by mail. Users can change their personal data through the portal. Changes are stored in the HBP LDAP server again. Then these changes are applied to the JSC data base either automatically (d_pers data) or manually. === Account application from dispatch perspective === The script notifyChanges.php (see Usage in production section) is run in a crontab to detect any updates in the HBP LDAP data. A mail with updates is sent to the dispatch. Currently, due to the early implementation state the script applying the changes needs to be run manually. Some changes can be applied automatically: d_pers, new account applications. Changes to groups, projects, existing SC LDAP accounts are not applied automatically. For those a mail is sent to dispatch informing exactly about the group/account and attributes, which need updating. New account applications will appear on https://dispatch.fz-juelich.de:8812/dispatch_account_req/DISPATCH_ZAM marked with ID-CLASS hbp. The workflow for joining users for HBP is identical with the normal workflow for all other account applications. As soon as the user agreement is received with all required signatures, the account can be joined. Note, that UID number, GID number and login name cannot be chosen freely but have to correspond with the data given in d_account_antrag for the automatically created account applications. The workflow for creating new projects is still under discussion. So far we have only established a single test project called bp0. It is not currently possible to have user accounts associated with multiple groups. See section "Usage in production" for instructions on how to use this application for data comparisons and automatic application of updates. === Content Structure === This application has the following folders with the corresponding functions listed next to them: - config | Configuration files, access credentials to LDAP and Oracle data bases, mail configuration - doc | Documentation for the software - gen | Parent folder for all artifacts generated by the library such as log files, user agreement forms - lib | Libraries written in PHP, which are used by the scripts folder, LDAP/Oracle access, comparison of MLS and JAD - actions | Implements actions for applying changes detected between MLS and JAD, modifies d_pers, creates user agreements and sends them - entitycompare | Generic comparison functions for user and group data, model for differences such as ToAdd and ToRemove - MAIL | Send signed mails with attachments - vendor | third party libraries needed for hbpaccounting, e.g. silex and swagger-generated code - publicapi | REST API for simple access to LDAP server content, see [/wiki/HBPHPCLDAP HBPHPCLDAP] for details - res | Resources used by the libraries, templates for mails and forms, dispatch certificate - scripts | top level scripts using the libraries in lib folder, run by crontab or via command line - production | scripts actually used in production, all other scripts only for test purposes - tests | PHPUnit tests for most of the library classes, show their functionality, check their function - LICENSE.txt | contains BSD-style license of the code developed by Forschungszentrum Jülich GmbH === Installation === To install this software, checkout all source files into a directory > svn checkout https://svn.version.fz-juelich.de/hbpaccounting/trunk accounting Make password files only readable for the user: > cd /config/mail > chmod 600 dispatch.zam.txt Afterwards, the installation can be tested > cd accounting > ./tests/phpunit.phar ./tests/ The last line of output should look like the following OK (XX tests, XXX assertions) Otherwise, an error occurred. The most likely explanation is a missing package in the installation. === Installation on Virtual Machine === At JSC a dedicated virtual machine is deployed for running these scripts. Documentation on the virtual machine and installation steps taken can be found at [https://trac.version.fz-juelich.de/hbpaccounting/wiki/serverconfig serverconfig] If access is granted one can access the machine with > ssh hbpadm@hbpacc This application is installed into /home/hbpadm/accounting The production scripts are placed at /home/hbpadm/accounting/scripts/production A crontab is going to be installed running the notifyChanges.php script in a reasonable interval (e.g. every day, or every 6 hours). === Usage in production === Only the scripts in /scripts/production/ are meant for usage in real production. They do not have any parameters. You can execute them like this > php notifyChanges.php Note, that the scripts starting with "apply" should be executed with great care. Check the changes, which are applied with the corresponding "showXXX" script. Here is a list of all scripts along with their overall functionality: - applyChanges.php | Apply changes to d_pers, SC LDAP user accounts, SC LDAP groups. This script applies the currently needed changes detected to d_pers and the account application tables. It also sends out user agreements for new account applications and notifies the dispatch about required changes in the SC LDAP server. - applyDPERSChanges.php | Only apply changes to the d_pers. This script only applies the changes needed for the d_pers table. Only differences in personal data are modified in d_pers, no account applications are generated here. - applyGroupLDAPChanges.php | Apply changes to the SC LDAP groups for HBP. This script checks for any changes in the group data by comparing all HBP groups from the master LDAP server with the JSC groups. If there is any difference a notification is sent to the dispatch. - applyUserLDAPChanges.php | Create new account applications for new user entries. This script applies changes to the SC LDAP server regarding user accounts. If new accounts need to be created, account applications are generated and stored in the d_account_antrag table. Corresponding user agreements are sent to the users. Upon return of the user agreements, the accounts can be created. - notifyChanges.php | This script checks for updates for d_pers and sc LDAP If there are updates required, a mail is sent to the logging address. - showDPERSChanges.php | List changes, which need to be applied to the d_pers table - showGroupLDAPChanges.php | List differences between HBP groups and JSC groups - showUserLDAPChanges.php | Show changes, which need to be applied to the JSC LDAP server regarding user accounts - showHBPDPERSData.php | Show data relevant for d_pers retrieved from HBP master server This script does not care about the data in the SC LDAP server. It simply lists all d_pers records retrieved from the HBP master server. As you can see, these scripts always come in pairs: notifyChanges.php - applyChanges.php, showDPERSChanges.php - applyDPERSChanges.php, showGroupLDAPChanges.php - applyGroupLDAPChanges.php, showUserLDAPChanges.php - applyUserLDAPChanges.php. The left partner of each pair only shows required changes. These changes are retrieved by comparing the data in the HBP master LDAP server with the corresponding JSC data. The right partner of a pair tries to apply the changes where possible. Except for more logging applyChanges.php runs all other applyXXX scripts (applyDPERSChanges.php, applyGroupLDAPChanges.php and applyUserLDAPChanges.php). The "notifyChanges.php" is meant to be run in a crontab in a given interval. It will send a mail to the dispatch telling them that changes occurred. Then the "applyChanges.php" should be run to apply the detected changes. All other scripts are used for partial comparisons except for showHBPDPERSData.php, which only shows d_pers relevant data found in the HBP master server. showDPERSChanges/applyDPERSChanges deals with the d_pers data showGroupLDAPChanges/applyGroupLDAPChanges deals with Unix groups in SC LDAP showUserLDAPChanges/applyUserLDAPChanges deals with user accounts To sum up, it should be sufficient to install notifyChanges.php run in a crontab. As soon as changes are sent, run applyChanges.php to apply them to the DB and LDAP servers. If the application of changes fails, the corresponding scripts for dpers, group or user LDAP data can be used for debugging. === Libraries in detail === This section gives an overview of the libraries located in /lib folder of the installation. - AccountStatus.php | Allows to check the status of account applications. - GenerateLoader.php | Helper script for generating class loaders for a folder of libraries - GroupManagement.php | Compares HBP LDAP groups with SC LDAP groups, allows to apply changes to the groups - LDAPLib.php | Connect to an LDAP server and read data concerning HBP accounts - Loader.php | class loader generated with GenerateLoader.php for /lib - Log.php | Provides functions for logging any kind of messages and logging mails sent by the system - MailLogging.php | Reads mail addresses from configuration for logging and allows log mail sending - OracleAttributeParser.php | Parses available attributes for a single table, access to attribute's data type, length and nullable - OracleDBLib.php | Library encapsulating access to oracle database - UserAgreementGenerator.php | Class for generating a user agreement PDF - UserManagement.php | Comparison of HBP LDAP server user data with existing users in the JSC LDAP, d_pers and LDAP handling - Utils.php | Collection of all utility function used at multiple places in the library - MAIL | Sending signed mails, borrowed from JARDS application, see https://trac.version.fz-juelich.de/antragssystem/ - SignedMail.php | Provides library access for sending signed mails with attachments. - sendsigned_mein.pl | library: send a mail with attachments, optionally signed - actions | Use model from entitycompare as differences, apply changes to different data sources based on the differences. - DPersHandler.php | This class handles all modifications to the d_pers database. - DiffHandler.php | Interface implemented by handlers, which can update the accounting data. - GroupdiffHandler.php | This handler deals with changes to the HBP groups. - LDAPHandler.php | Apply changes to the JSC LDAP server. Mainly this includes to add new entries to the d_account_antrag table. - entitycompare | generic comparison of data entities, entity is for instance a user, this library allows to compare entity lists. - AttributeDifference.php | Difference on attribute level, result of comparison of two entities. - Difference.php | Interface for set of differences, which are results of comparisons of entities. - Entity.php | Represents a single object from the database (LDAP or Oracle). - EntityList.php | List of entities, which can be compared with other lists of entities. - ToAdd.php | This indicates that an entire entity needs to be added. - ToRemove.php | This indicates that an entire entity needs to be removed. The entitycompare library allows to compare arbitrary data. Currently, it is used for comparing groups and users from two different data sources (HBP master vs. JSC data). An entity represents one data record, for instance a user or a group. With entitycompare entities can be compared on entity level and on entity list level, where two lists of users or groups are compared for any differences. Examples for usage of entitycompare can be found in the tests/lib/entitycompare directory. The actions library uses instances of "Difference" as input for applying any changes. There is one handler for each data source. which is updated. DPersHandler modifies the d_pers table, GroupdiffHandler modifies the UNIX groups and LDAPHandler modifies LDAP accounts for the users. Note that the d_pers can be updated with all data, new records can be created and old ones deleted. Thus, synchronizing the d_pers table can be conducted fully automatically. This is different for the GroupdiffHandler. There is no automatic way for inserting new groups. For all actions a mail is sent to dispatch, so that the change can be applied manually. Finally, LDAPHandler handles adding of new accounts automatically, but deletion and modification of accounts needs to be done manually. The MAIL library only encapsulates a library for sending signed mails. === Configurations === This section describes the function of all configuration files located at /config. Each configuration option is documented here. - AddressFieldsMaxLengths.txt | Configuration for address field as required by d_pers ORDER="INSTITUTION;STRASSE;POSTFACH;PLZ;ORT;LAND" defines order of atomic attributes stored in the address. All other key value pairs define the maximum length of each attribute. This file is used for LDAPLib::fixAddress. An arbitrary address read from the HBP master LDAP server is parsed and prepared for storing in the d_pers table. - DEFAULTS.txt | Contains default values as fallback. If corresponding values cannot be parsed from DB, these values are used. ORG used when no organisation is given by the HBP data DAA_PROJ_NR used as DAA_PROJ_NR when creating a new account application and no such number can be found for a project - HBPLDAPConfig.txt | LDAP connection configuration -- Configures how to access the HBP master LDAP server Stores the server's URL and configures how to find accounts relevant for JSC from the entire HBP database. HPCSYSTEMS: lists distinguished names in HBP master server for JSC supercomputers. With the ismemberof operator one can check, whether an account needs to be created for a given supercomputer. E.g. >ldapsearch -v -x -H ldaps://... -b "uid=bp000003,ou=users,dc=hbp,dc=eu" "ismemberof=cn=marenostrum,ou=groups,dc=hbp,dc=eu" checks, whether the given account must be active on marenostrum. This check is executed for all accounts in order to determine the HPC systems, for which the account needs to be created. There are mainly two groups of parameters in this file: user-related and group-related parameters. They include the attribute used as ID for generated entities, how to filter for JSC relevant groups and accounts as well as the base DN where to search for users and groups. - HBPTOJSCATTRIBUTEMAP.txt | Maps attribute names used in the HBP master server to d_pers and JSC LDAP attribute names This file contains two sections: ldap and d_pers. ldap maps attribute names of the HBP master LDAP server to the JSC attribute names in the user branch. d_pers maps LDAP attributes to attribute names in D_PERS table. The attribute names on the left hand side refer to the attribute names in the HBP master server in the branch ou=users,dc=hbp,dc=eu. The right hand side values in the ldap section refer to attributes in the SC LDAP server in the branch ou=users,ou=jsc,dc=fz-juelich,dc=de. For the d_pers section the right hand side values are column names in the d_pers table. This configuration is used to split the HBP LDAP data into data relevant for d_pers and the remaining data needed in the SCLDAP server. The ldap mapping is used in LDAPLib::getMasterHBPLDAPUserEntityList to generate entity lists for comparing to the SC LDAP data. The d_pers mapping is used in LDAPLib::getMasterHBPDPERSUserEntityList which is compared to the corresponding d_pers data. Note that projectnames, groupnames and hostnames do not directly exist in the LDAP servers. They are derived through corresponding functions: HBP LDAP: projectnames -> LDAPLib::getHBPProjectNamesForUID uses memberUid attribute of the groups groupnames = projectnames hostnames -> LDAPLib::getHBPHPCSystemsForUID uses the ismemberof operator of the HPC groups (e.g. at cn=juqueen,ou=groups,dc=hbp,dc=eu) JSC LDAP: projectnames -> LDAPLib::getJSCProjectNamesForUID uses memberUid attribute in JSC groups groupnames = projectnames hostnames -> LDAPLib::getJSCHPCSystemsForUID determines through the user's objectclass at which system he has access to. E.g. a user with objectclasses x-jsc-juropaAccount and x-jsc-juqueenAccount has access to juropa and juqueen. - HBPTOJSCGROUPATTRIBUTEMAP.txt | Similar function like HBPTOJSCATTRIBUTEMAP.txt but for groups. Currently it is assumed that Unix groups and projects will have the identical and connected names. I.e. the project bp0g1 will have the Unix group bp0g1 attached. This configuration file maps the HBP LDAP group attributes from the data in branch ou=groups,dc=hbp,dc=eu to the JSC groups found in the SC LDAP server in branch ou=groups,ou=jsc,dc=fz-juelich,dc=de. The hostnames attribute is not directly available in the group data. For HBP data it is retrieved with the function LDAPLib::getHBPHostsForProject, which uses the ismemberof operator. For JSC data the hostnames for a project are retrieved with LDAPLib::getJSCHostsForProject, which reads that information from the d_org_projekt table. - JSCOracle.txt | Configures how to connect to the JSC Oracle DB. Stores the server name and credentials for connecting to the server. Moreover, special tables are configured, where new account applications (d_account_antrag) and new personal data records (d_pers) are stored. These tables are made configurable to allow simple testing on copies of these tables, which is done in TestJSCOracle.txt file. This configuration is used in OracleDBLib::getDBConnection to start DB connections. - LOGGING.txt | Configures mail addresses for sending log mails and manual change requests, separated by white spaces. - SCLDAPConfig.txt | LDAP connection configuration -- Configures how to access the JSC LDAP server The function of this configuration file is similar to the HBPLDAPConfig.txt. It provides URL and credentials for accessing the JSC LDAP server. Moreover, some user and group IDs are listed here, which are ignored for comparisons. These were test groups at the JSC, which matched the HBP naming scheme, but were not available from the HBP server. - TestJSCOracle.txt | Test environment configuration for DB change tests. This configuration can be compared to JSCOracle.txt. It only provides different d_pers and d_account_antrag tables, which can be used for testing. This configuration is used in the unit tests. It allows to create new account applications and modify the d_pers in a safe test environment. - WEBLDAPConfig.txt | Configures access to JSC web services LDAP server. This configuration file is not really necessary for the production. It is only required for test reasons. - mail | Collection of configuration files for the mail library. - anhang_signed_mail.txt end of mail text sent to users - dispatch.zam.rc mail address shown in the from section of a sent mail - dispatch.zam.txt password for mail signature - INFO.txt Documentation on how to allow signed mail sending from web user