== Jupyter [[PageOutline]] [[span(style=color: #FF0000, '''Jupyter for JSC is in alpha phase''')]] [[Image(jupyterpreview.png, 400px, align=right, margin=10)]] [[Image(jupyter-logo.png,200px, align=left, margin=15)]] \\ The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more. \\ \\ [[Image(jupyterhub-logo.png, 200px, align=left, margin=15)]] \\ Project Jupyter created !JupyterHub to support many users. The Hub can offer notebook servers to a class of students, a corporate data science workgroup, a scientific research project, or a high performance computing group. With !JupyterHub a multi-user Hub which spawns, manages, and proxies multiple instances of the single-user Jupyter notebook (IPython notebook) server can be created. \\ \\ \\ \\ === Where are the sources? * Jupyter: [https://github.com/jupyter/jupyter] * Jupyter Notebook: [https://github.com/jupyter/notebook] * Jupyter Console: [https://github.com/jupyter/jupyter_console] * Jupyter NBViewer: [https://github.com/jupyter/nbviewer] * Jupyter Kernel * Jupyter IPyKernel: [https://github.com/ipython/ipykernel] * Jupyter IPyWidgets: [https://github.com/jupyter-widgets/ipywidgets] * other * Tornado Web Server: [https://github.com/tornadoweb/tornado] * !JupyterLab: [https://github.com/jupyterlab/jupyterlab] * !JupyterHub: [https://github.com/jupyterhub/jupyterhub] * Jupyter Http Proxy: [https://github.com/jupyterhub/configurable-http-proxy] === How does it work? ==== !JupyterHub !JupyterHub is a multi-user server that manages and proxies multiple instances of the single-user Jupyter notebook server. There are three basic processes involved: * multi-user Hub (!Python/Tornado) * configurable http proxy (node-http-proxy) * multiple single-user IPython notebook servers (Python/IPython/Tornado) The proxy is the only process that listens on a public interface. The Hub sits behind the proxy at /hub. Single-user servers sit behind the proxy at /user/[username]. more: https://jupyterhub.readthedocs.io/en/0.7.2/howitworks.html [[Image(JupyterHub-Structure.png, 400px, align=right, margin=10)]] ===== Callpath * start the Tornado event loop [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1638 app.py:1638] * initialize !JupyterHub and all its services [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1365 app.py:1365] * init handlers (connect urls with functions) [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L802 app.py:802] * add handlers [https://github.com/jupyterhub/jupyterhub/tree/master/jupyterhub/handlers handlers] * add apihandlers [https://github.com/jupyterhub/jupyterhub/tree/master/jupyterhub/apihandlers apihandlers] * init webserver with handlers [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1284 app.py:1284] * init Jinja2 template engine [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1291 app.py:1291] * static_path points to HTML/JS/JINJA2 files [https://github.com/jupyterhub/jupyterhub/tree/master/share/jupyter/hub/ webfiles] * start !JupyterHub and all its services [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1517 app.py:1517] * start webserver [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1532 app.py:1532] * start http proxy [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1542 app.py:1542] * start services [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1551 app.py:1551] ===== Websites The websites of !JupyterHub are stored in [https://github.com/jupyterhub/jupyterhub/tree/master/share/jupyter/hub/ share/jupyter/hub] \\ * They are build with the template engine [http://jinja.pocoo.org/ Jinja2] for Python for a dynamic generation of the html pages (a detailed description of the API can be found [http://jinja.pocoo.org/docs/2.9/api/ here]). * Components like buttons are used from [http://getbootstrap.com/ bootstrap]. It gets included via stylesheets [https://github.com/jupyterhub/jupyterhub/blob/master/share/jupyter/hub/templates/page.html#L39 here]. * [https://github.com/jupyterhub/jupyterhub/blob/master/share/jupyter/hub/templates/home.html home.html] * provides "block main" and "block script" required by [https://github.com/jupyterhub/jupyterhub/blob/master/share/jupyter/hub/templates/page.html page.html] * distinguages between "user.running" and "not user.running" * uses button class btn-danger and btn-success from [http://getbootstrap.com/docs/4.0/components/buttons/ bootstrap buttons] * "block script" includes !JavaScript file [https://github.com/jupyterhub/jupyterhub/blob/master/share/jupyter/hub/static/js/home.js home.js] * implements function if stop-button is clicked * for communication back to the server functions from the JupyterHubAPI in [https://github.com/jupyterhub/jupyterhub/blob/master/share/jupyter/hub/static/js/jhapi.js jhapi.js] are called * jhapi.js uses [https://jquery.com/ jquery.js] for ajax-functionality (send/recive commands between client/server) * values are send back to client with an an asynchronous HTTP (Ajax) request using jquery.ajax() [https://github.com/jupyterhub/jupyterhub/blob/master/share/jupyter/hub/static/js/jhapi.js#L43 here] \\ => this triggers a handler of the tornado web server (eg. [https://github.com/jupyterhub/jupyterhub/blob/master/share/jupyter/hub/static/js/jhapi.js#L46 start_server] sends command "POST" to url users//server, which triggers function [https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/apihandlers/users.py#L184 UserServerAPIHandler::post]) ===== Modifications !JupyterHub supports userdefined notebook spawners. At JSC our spawners for each partition of the supercomputer adds the functionality to start a jupyter notebook by submitting a SLURM job on the system. These additional parameters must be set by the user before he/she starts the server, which are * number of nodes * run time This is the minimum number of parameters, but more might be required in future. In the following the bits and pieces are shown that result in the modification, which adds this possibility to !JupyterHub: ====== HTML design [[Image(JupyterHub-Spawner-GUI.png, 400px, align=right, margin=10)]] !JupyterHub´s design is based on [http://getbootstrap.com/ bootstrap]. On the right you see a possible implementation of an interface for additional parameter of our custom spawner. \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ ---- any feedback welcomed - [mailto:h.zilken@fz.juelich.de], [mailto:j.goebbert@fz-juelich.de]