Merge 2.1.1

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 2.1.0
+current_version = 2.1.1
 commit = True
 tag = True
 tag_name = {new_version}

CHANGELOG.md

@@ -1,3 +1,3 @@
 # Change Log
 
-See the [papermill documentation](https://papermill.readthedocs.io/changelog.html)
+See the [papermill documentation](https://papermill.readthedocs.io/en/latest/changelog.html)

CONTRIBUTING.md

@@ -1,4 +1,5 @@
 # So You Want to Contribute to Papermill!
+
 We welcome all contributions to Papermill both large and small. We encourage you to join our community.
 
 ## Our Community Values
@@ -12,35 +13,38 @@ All contributions are equally important. Documentation, answering questions, and
 Please read our entire code of conduct [here](https://github.com/nteract/nteract/blob/master/CODE_OF_CONDUCT.md). Also, check out the for the [Python](https://github.com/nteract/nteract/blob/master/CODE_OF_CONDUCT.md) code of conduct.
 
 ## Setting up Your Development Environment
+
 Following these instructions should give you an efficient path to opening your first pull-request.
 
 ### Cloning the Papermill Repository
+
 Fork the repository to your local Github account. Clone this repository to your local development machine.
+
 ```bash
 git clone https://github.com/<your_account>/papermill
 cd papermill
 ```
 
 ### Install an Editable Version
-We prefer to use [conda](https://conda.io/docs/user-guide/tasks/manage-environments.html) to manage the development environment.
-```bash
-conda create -n dev
-. activate dev
-```
-or use native venv capabilities if you prefer.
+
+We prefer to use native venv to manage the development environment.
+
 ```bash
 python3 -m venv dev
 source dev/bin/activate
 ```
 
 Install Papermill using:
+
 ```bash
 pip install -e '.[dev]'
 ```
 
-If you're using pip 19 or above, you should run
+or use conda if you prefer [conda](https://conda.io/docs/user-guide/tasks/manage-environments.html):
+
 ```bash
-pip install -e '.[dev]' --no-use-pep517
+conda create -n dev
+. activate dev
 ```
 
 _Note: When you are finished you can use `source deactivate` to go back to your base environment._
@@ -70,11 +74,9 @@ This will require python3.6, python3.8, and python3.7 to be installed. **Note**
 Alternavitely pytest can be used if you have an environment already setup which works or has custom packages not present in the tox build.
 
 ```bash
-pytest --pyargs papermill
+pytest
 ```
 
-The `pyargs` option allows `pytest` to interpret arguments as python package names. An advantage is that `pytest` will run in any directory, and this approach follows the `pytest` [best practices](https://docs.pytest.org/en/latest/goodpractices.html#tests-as-part-of-application-code).
-
 Now there should be a working and editable installation of Papermill to start making your own contributions.
 
 ### Building Documentation

README.md

@@ -7,7 +7,7 @@
 [![image](https://codecov.io/github/nteract/papermill/coverage.svg?branch=master)](https://codecov.io/github/nteract/papermill?branch=master)
 [![Documentation Status](https://readthedocs.org/projects/papermill/badge/?version=latest)](http://papermill.readthedocs.io/en/latest/?badge=latest)
 [![badge](https://tinyurl.com/ybwovtw2)](https://mybinder.org/v2/gh/nteract/papermill/master?filepath=binder%2Fprocess_highlight_dates.ipynb)
-[![badge](https://tinyurl.com/y7uz2eh9)](https://mybinder.org/v2/gh/nteract/papermill/master?
+[![badge](https://tinyurl.com/y7uz2eh9)](https://mybinder.org/v2/gh/nteract/papermill/master?)
 [![Python 3.6](https://img.shields.io/badge/python-3.6-blue.svg)](https://www.python.org/downloads/release/python-360/)
 [![Python 3.7](https://img.shields.io/badge/python-3.7-blue.svg)](https://www.python.org/downloads/release/python-370/)
 [![Python 3.8](https://img.shields.io/badge/python-3.8-blue.svg)](https://www.python.org/downloads/release/python-380/)

RELEASING.md

@@ -11,7 +11,7 @@ Change from patch to minor or major for appropriate version updates.
 
 ```bash
 bumpversion patch
-git push && git push --tags
+git push upstream && git push upstream --tags
 ```
 
 ## Push to PyPI

docs/changelog.md

@@ -1,5 +1,12 @@
 # Change Log
 
+## 2.1.1
+
+- DeadKernelExceptions, usually from OOM, now exit with a status code of 138 from the CLI.
+- Error cell at the top of failed notebook has been made better. It now also has a link to an injected cell where the error occurred.
+- Updated a deprecated function to the new function name for nbclient dependency.
+- Some development and documentation updates / fixes have also been made by a few different contributions (thank you!).
+
 ## 2.1.0
 
 - Support for python 3.5 has been dropped. Upstream library changes for async were causing process deadlocks with await commands. End-of-life is later this year for 3.5 anyway so we decided to also drop support here.

papermill/cli.py

@@ -4,6 +4,8 @@
 import os
 import sys
 from stat import S_ISFIFO
+import nbclient
+import traceback
 
 import base64
 import logging
@@ -216,24 +218,29 @@ def papermill(
     for name, value in parameters_raw or []:
         parameters_final[name] = value
 
-    execute_notebook(
-        input_path=notebook_path,
-        output_path=output_path,
-        parameters=parameters_final,
-        engine_name=engine,
-        request_save_on_cell_execute=request_save_on_cell_execute,
-        autosave_cell_every=autosave_cell_every,
-        prepare_only=prepare_only,
-        kernel_name=kernel,
-        progress_bar=progress_bar,
-        log_output=log_output,
-        stdout_file=stdout_file,
-        stderr_file=stderr_file,
-        start_timeout=start_timeout,
-        report_mode=report_mode,
-        cwd=cwd,
-        execution_timeout=execution_timeout,
-    )
+    try:
+        execute_notebook(
+            input_path=notebook_path,
+            output_path=output_path,
+            parameters=parameters_final,
+            engine_name=engine,
+            request_save_on_cell_execute=request_save_on_cell_execute,
+            autosave_cell_every=autosave_cell_every,
+            prepare_only=prepare_only,
+            kernel_name=kernel,
+            progress_bar=progress_bar,
+            log_output=log_output,
+            stdout_file=stdout_file,
+            stderr_file=stderr_file,
+            start_timeout=start_timeout,
+            report_mode=report_mode,
+            cwd=cwd,
+            execution_timeout=execution_timeout,
+        )
+    except nbclient.exceptions.DeadKernelError:
+        # Exiting with a special exit code for dead kernels
+        traceback.print_exc()
+        sys.exit(138)
 
 
 def _resolve_type(value):

papermill/clientwrap.py

@@ -36,7 +36,7 @@ def execute(self, **kwargs):
         with self.setup_kernel(**kwargs):
             self.log.info("Executing notebook with kernel: %s" % self.kernel_name)
             self.papermill_execute_cells()
-            info_msg = self._wait_for_reply(self.kc.kernel_info())
+            info_msg = self.wait_for_reply(self.kc.kernel_info())
             self.nb.metadata['language_info'] = info_msg['content']['language_info']
             self.set_widgets_metadata()
 

papermill/exceptions.py

@@ -22,7 +22,8 @@ class PapermillMissingParameterException(PapermillException):
 class PapermillExecutionError(PapermillException):
     """Raised when an exception is encountered in a notebook."""
 
-    def __init__(self, exec_count, source, ename, evalue, traceback):
+    def __init__(self, cell_index, exec_count, source, ename, evalue, traceback):
+        self.cell_index = cell_index
         self.exec_count = exec_count
         self.source = source
         self.ename = ename

papermill/execute.py

@@ -83,6 +83,8 @@ def execute_notebook(
             nb = parameterize_notebook(nb, parameters, report_mode)
 
         nb = prepare_notebook_metadata(nb, input_path, output_path, report_mode)
+        # clear out any existing error markers from previous papermill runs
+        nb = remove_error_markers(nb)
 
         if not prepare_only:
             # Fetch the kernel name if it's not supplied
@@ -144,13 +146,35 @@ def prepare_notebook_metadata(nb, input_path, output_path, report_mode=False):
     return nb
 
 
+ERROR_MARKER_TAG = "papermill-error-cell-tag"
+
+ERROR_STYLE = (
+    'style="color:red; font-family:Helvetica Neue, Helvetica, Arial, sans-serif; font-size:2em;"'
+)
+
 ERROR_MESSAGE_TEMPLATE = (
-    '<span style="color:red; font-family:Helvetica Neue, Helvetica, Arial, sans-serif; font-size:2em;">'
-    "An Exception was encountered at 'In [%s]'."
+    '<span ' + ERROR_STYLE + '>'
+    "An Exception was encountered at '<a href=\"#papermill-error-cell\">In [%s]</a>'."
+    '</span>'
+)
+
+ERROR_ANCHOR_MSG = (
+    '<span id="papermill-error-cell" ' + ERROR_STYLE + '>'
+    'Execution using papermill encountered an exception here and stopped:'
     '</span>'
 )
 
 
+def remove_error_markers(nb):
+    nb = copy.deepcopy(nb)
+    nb.cells = [
+        cell
+        for cell in nb.cells
+        if ERROR_MARKER_TAG not in cell.metadata.get("tags", [])
+    ]
+    return nb
+
+
 def raise_for_execution_errors(nb, output_path):
     """Assigned parameters into the appropriate place in the input notebook
 
@@ -162,7 +186,7 @@ def raise_for_execution_errors(nb, output_path):
        Path to write executed notebook
     """
     error = None
-    for cell in nb.cells:
+    for index, cell in enumerate(nb.cells):
         if cell.get("outputs") is None:
             continue
 
@@ -171,6 +195,7 @@ def raise_for_execution_errors(nb, output_path):
                 if output.ename == "SystemExit" and (output.evalue == "" or output.evalue == "0"):
                     continue
                 error = PapermillExecutionError(
+                    cell_index=index,
                     exec_count=cell.execution_count,
                     source=cell.source,
                     ename=output.ename,
@@ -180,9 +205,18 @@ def raise_for_execution_errors(nb, output_path):
                 break
 
     if error:
-        # Write notebook back out with the Error Message at the top of the Notebook.
+        # Write notebook back out with the Error Message at the top of the Notebook, and a link to
+        # the relevant cell (by adding a note just before the failure with an HTML anchor)
         error_msg = ERROR_MESSAGE_TEMPLATE % str(error.exec_count)
         error_msg_cell = nbformat.v4.new_markdown_cell(error_msg)
-        nb.cells = [error_msg_cell] + nb.cells
+        error_msg_cell.metadata['tags'] = [ERROR_MARKER_TAG]
+        error_anchor_cell = nbformat.v4.new_markdown_cell(ERROR_ANCHOR_MSG)
+        error_anchor_cell.metadata['tags'] = [ERROR_MARKER_TAG]
+
+        # put the anchor before the cell with the error, before all the indices change due to the
+        # heading-prepending
+        nb.cells.insert(error.cell_index, error_anchor_cell)
+        nb.cells.insert(0, error_msg_cell)
+
         write_ipynb(nb, output_path)
         raise error

papermill/tests/notebooks/broken1.ipynb

@@ -1,5 +1,34 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": [
+     "papermill-error-cell-tag"
+    ]
+   },
+   "source": [
+    "<span style=\"color:red; font-family:Helvetica Neue, Helvetica, Arial, sans-serif; font-size:2em;\">An Exception was encountered at '<a href=\"#papermill-error-cell\">In [1]</a>'.</span>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A markdown cell that makes the execution counts different to indices within the list of all cells."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": [
+     "papermill-error-cell-tag"
+    ]
+   },
+   "source": [
+    "<span id=\"papermill-error-cell\" style=\"color:red; font-family:Helvetica Neue, Helvetica, Arial, sans-serif; font-size:2em;\">Execution encountered an exception here and stopped:</span>"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -41,6 +70,20 @@
     "print(\"We're good.\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Another markdown cell"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A third one."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

papermill/tests/test_adl.py

@@ -44,7 +44,7 @@ def test_listdir_calls_ls_on_adl_adapter(self):
         self.ls.assert_called_once_with("path/to/directory")
 
     def test_read_opens_and_reads_file(self):
-        self.assertEquals(
+        self.assertEqual(
             self.adl.read("adl://foo_store.azuredatalakestore.net/path/to/file"), ["a", "b", "c"]
         )
         self.fakeFile.__iter__.assert_called_once_with()

papermill/tests/test_cli.py

@@ -6,6 +6,7 @@
 import sys
 import subprocess
 import uuid
+import nbclient
 
 import nbformat
 from jupyter_client import kernelspec
@@ -178,6 +179,16 @@ def test_parameters_yaml_override(self, execute_patch):
             )
         )
 
+    @patch(
+        cli.__name__ + '.execute_notebook', side_effect=nbclient.exceptions.DeadKernelError("Fake")
+    )
+    def test_parameters_dead_kernel(self, execute_patch):
+        result = self.runner.invoke(
+            papermill,
+            self.default_args + ['--parameters_yaml', '{"foo": "bar"}', '-y', '{"foo": ["baz"]}'],
+        )
+        assert result.exit_code == 138
+
     @patch(cli.__name__ + '.execute_notebook')
     def test_parameters_base64(self, execute_patch):
         self.runner.invoke(