Man Group's newly open-sourced project, Notebooker, empowers users to publish their work with a few simple clicks. Users can transform the code within their Jupyter Notebooks into scheduled, parametrised, production-quality reports which can be browsed in a convenient webapp. Jupyter Notebooks have become a ubiquitous tool for python analysis; the Notebooker project puts even more power and reliability in the hands of our users.

The Notebooker logo

From research to reproducibility

Jupyter Notebooks are a fantastic tool for researchers, data scientists, and developers alike. Supporting a vast array of popular programming languages including Python, Ruby, and R, they allow for the super-fast iteration of ideas with code separated into “cells” and results presented inline, whether they be in a tabular format or in charts.

An example of a Jupyter Notebook

Despite their usefulness as a research tool, often the hard work put into the code within a Jupyter Notebook is lost. Either the notebook is converted into a scheduled report by a developer, or the notebook is never executed again: the results will be saved and become stale on the user’s hard drive. Additionally, if we wish to productionise the notebooks themselves, there are several properties which are traditionally difficult with Jupyter Notebooks, namely: code review, testability, re-usability, execution, and browsing of results. Here’s how Notebooker aims to resolve each:

The above report, but presented within Notebooker

Reviewability

When a change is made, one of the most critical steps before a report lands in production is peer review. In order to facilitate this in Notebooker, Notebooks are converted from .ipynb to .py using Jupytext. This turns the raw, difficult-to-review .ipynb into a much more user-friendly python file so that changes can be properly scrutinised. See this article for an example of how Jupytext works.

Testability

It’s important to ensure that once you’ve written a report that it will continue working in the future, preventing breakages either through external changes or future enhancements. Notebooker reports can be regression tested via a command-line tool which uses pytest, so that we can catch any errors before they happen in production.

Re-usability

By adding a parameters cell which can be consumed by papermill, Jupyter Notebooks can be used as a template for 10s or 100s of reports with the same or similar output when parametrised. For example, if you want to run a report per country, you only have to write one Jupyter Notebook and add a parameter for the country in question. You can then schedule a task to run the same report with different inputs, meaning you only had to write the code once. Within the webapp it’s also possible simply to type the parameters as if you are writing Python in a Jupyter cell. This allows for live experimentation with different parameters running in parallel on the webapp.

Executablility

Notebooker reports can either be executed by users on a webapp front-end, or via command-line. For example, a researcher may want to play around with values on the webapp and then schedule a cron job to run the report with those parameters every day. There’s also an option to email results and produce PDF output. Regardless of execution method, results are saved into the MongoDB database for later retrieval.

Browsability

Once we’ve run the report, we want to view it! To allow for this, the Notebooker homepage allows for searching and browsing of all results, with convenient ways to download rendered PDFs and view the reports online. When a notebook has been executed and the results are presented to the user, they may want to also share the results with colleagues. Notebooker URLs for results are unique and will reliably link the recipient to the intended report. The MongoDB database allows us to search for results in a variety of ways: for example, if a report has been run with the parameter ticker="$AAPL", we can find the most recent report with this parameter and present it to the user.

The Notebooker homepage

Deploying to Notebooker

Finally, when you’re happy with your Jupyter Notebook, deployment to Notebooker consists of the following steps:

  1. Convert your .ipynb to .py using Jupytext
  2. Commit the .py file into git
  3. Merge the .py file into the production branch

… and that’s it! The Notebook will appear on the deployed Notebooker instance as a template which can be executed on-demand and scheduled using an external job scheduler such as cron, Airflow, or Jenkins. A feature which we hope to add soon is the ability to schedule reports directly from the Notebooker webapp.

 

Final Thoughts

Since its inception in late 2018, Notebooker has been a successful addition to the internal toolkit within Man Group, with usages as varied as support dashboards, JIRA reporting, and portfolio drilldowns. We’re delighted to announce that we have open-sourced Notebooker and it is readily available at https://github.com/man-group/notebooker. We hope that you can make use of it too, so check it out on GitHub today!

Important information

Opinions expressed are those of the author and may not be shared by all personnel of Man Group plc (‘Man’). These opinions are subject to change without notice, are for information purposes only and do not constitute an offer or invitation to make an investment in any financial instrument or in any product to which the Company and/or its affiliates provides investment advisory or any other financial services. Any organisations, financial instrument or products described in this material are mentioned for reference purposes only which should not be considered a recommendation for their purchase or sale. Neither the Company nor the authors shall be liable to any person for any action taken on the basis of the information provided. Some statements contained in this material concerning goals, strategies, outlook or other non-historical matters may be forward-looking statements and are based on current indicators and expectations. These forward-looking statements speak only as of the date on which they are made, and the Company undertakes no obligation to update or revise any forward-looking statements. These forward-looking statements are subject to risks and uncertainties that may cause actual results to differ materially from those contained in the statements. The Company and/or its affiliates may or may not have a position in any financial instrument mentioned and may or may not be actively trading in any such securities. This material is proprietary information of the Company and its affiliates and may not be reproduced or otherwise disseminated in whole or in part without prior written consent from the Company. The Company believes the content to be accurate. However accuracy is not warranted or guaranteed. The Company does not assume any liability in the case of incorrectly reported or incomplete information. Unless stated otherwise all information is provided by the Company. Past performance is not indicative of future results.

Please update your browser

Unfortunately we no longer support Internet Explorer 8, 7 and older for security reasons.

Please update your browser to a later version and try to access our site again.

Many thanks.