Jupyter

Jupyter for JSC is in alpha phase

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.

Project Jupyter created JupyterHub to support many users. The Hub can offer notebook servers to a class of students, a corporate data science workgroup, a scientific research project, or a high performance computing group. With JupyterHub a multi-user Hub which spawns, manages, and proxies multiple instances of the single-user Jupyter notebook (IPython notebook) server can be created.

Where are the sources?

• Jupyter: ​https://github.com/jupyter/jupyter
  • Jupyter Notebook: ​https://github.com/jupyter/notebook
    • Jupyter Console: ​https://github.com/jupyter/jupyter_console
    • Jupyter NBViewer: ​https://github.com/jupyter/nbviewer
  • Jupyter Kernel
    • Jupyter IPyKernel: ​https://github.com/ipython/ipykernel
    • Jupyter IPyWidgets: ​https://github.com/jupyter-widgets/ipywidgets
  • other
    • Tornado Web Server: ​https://github.com/tornadoweb/tornado

• JupyterLab: ​https://github.com/jupyterlab/jupyterlab

• JupyterHub: ​https://github.com/jupyterhub/jupyterhub
  • Jupyter Http Proxy: ​https://github.com/jupyterhub/configurable-http-proxy

How does it work?

JupyterHub

JupyterHub is a multi-user server that manages and proxies multiple instances of the single-user Jupyter notebook server.

There are three basic processes involved:

• multi-user Hub (Python/Tornado)
• configurable http proxy (node-http-proxy)
• multiple single-user IPython notebook servers (Python/IPython/Tornado)

The proxy is the only process that listens on a public interface. The Hub sits behind the proxy at /hub. Single-user servers sit behind the proxy at /user/[username].

more: ​https://jupyterhub.readthedocs.io/en/0.7.2/howitworks.html

Installation

JupyterHub is started from eg. /usr/local/bin/jupyterhub with

#!/usr/bin/python3

from jupyterhub.app import main
main()

• Hence, python files are installed in default location for python packages
  eg. /usr/local/lib/python3.5/dist-packages/jupyterhub
• HTML/JS/CSS files are installed in
  eg. /usr/local/share/jupyter/hub
  • static
    • components (external components downloaded on install of JupyterHub in setup.py)
    • ...
  • templates (for pythen template engine Jinja2)

Callpath

• start the Tornado event loop ​app.py:1638
• initialize JupyterHub and all its services ​app.py:1365
  • init handlers (connect urls with functions) ​app.py:802
    • add handlers ​handlers
    • add apihandlers ​apihandlers
  • init webserver with handlers ​app.py:1284
    • init Jinja2 template engine ​app.py:1291
    • static_path points to HTML/JS/JINJA2 files ​webfiles
• start JupyterHub and all its services ​app.py:1517
  • start webserver ​app.py:1532
  • start http proxy ​app.py:1542
  • start services ​app.py:1551

Websites

The websites of JupyterHub are stored in ​share/jupyter/hub

• They are build with the template engine ​Jinja2 for Python for a dynamic generation of the html pages (a detailed description of the API can be found ​here).
• Components like buttons are used from ​bootstrap. It gets included via stylesheets ​here.
  They are downloaded on installation of JupyterHub by ​setup.py to share/jupyter/hub/components/

• ​home.html
  • provides "block main" and "block script" required by ​page.html
  • distinguages between "user.running" and "not user.running"
  • uses button class btn-danger and btn-success from ​bootstrap buttons for stop- and start-server button
  • stop-server button
    • ​home.js implements function if stop-button is clicked
      • for communication back to the server functions from the JupyterHubAPI in ​jhapi.js are called
        • jhapi.js uses ​jquery.js for ajax-functionality (send/recive commands between client/server)
          • values are send back to client with an an asynchronous HTTP (Ajax) request using jquery.ajax() ​here
            => this triggers a handler of the tornado web server (eg. ​stop_server sends command "DELETE" to url users/<username>/server, which triggers function ​UserServerAPIHandler::delete)
  • start-server button
    • clicking the start-server button opens the webpage "http://<jupyterhub-ip>/hub/spawn", which is generated by Jinja2 from the template ​spawn.html
      • this page includes a <form> element, which includes the generated part "spawner_options_form":

         <form enctype="multipart/form-data" id="spawn_form" action="{{url}}" method="post" role="form">
              {{spawner_options_form | safe}}
              <br>
              <input type="submit" value="Spawn" class="btn btn-jupyter">
        </form>
        

      • "spawner_options_form" is defined in ​spawner.py in function options_form()

Modifications

JupyterHub supports userdefined notebook spawners (​examples).

At JSC our spawners for each partition of the supercomputer adds the functionality to start a jupyter notebook by submitting a SLURM job on the system.

These additional parameters must be set by the user before he/she starts the server, which are

• number of nodes
• run time

This is the minimum number of parameters, but more might be required in future. In the following the bits and pieces are shown that result in the modification, which adds this possibility to JupyterHub:

HTML design

JupyterHub´s design is based on ​bootstrap.

On the right you see a possible implementation of an interface for additional parameter of our custom spawner.
This prototype implementation can be downloaded here​

---

any feedback welcomed - h.zilken@…, j.goebbert@…