Papermill: Papermill should be able to return different exit codes regarding exception raised

[axelborja]

Papermill seems to exit with code 1 in case of any error.

However, different kinds of errors can occur, and we should be able to distinguish them.

For example, in the case of technical errors like OOM (for example: nbconvert.preprocessors.execute.DeadKernelError: Kernel died), we could imagine to automatically ask for retrying the papermill execution with more resources.

2 possible solutions:
1 - Manage technical errors like OOM and return a different exit code.
2 - Let the user manage which exit code to return regarding the exception raised.

[radcheb]

👍

[MSeal]

We do capture sys.exit calls, but we just raise an exception with the information, letting python default the failure path. It'd probably be fairly easy to capture the sys.exit value in the CLI when an exception is raised and re-raise the appropriate exit code. For python use, I think raising the exception is still the correct pattern as we want users to be able to capture and handle failures.

As for OOM, that one is a lot trickier. Especially since an OOM might kill the papermill process itself, or the exit handlers may or may not run for the subprocess. See https://stackoverflow.com/questions/742413/return-code-when-os-kills-your-process for even more edge cases that mean the exit code is not guaranteed or even expected.

What we could do is if we see a DeadKernelError raise a particular exit code so it's easier to manage kernel death events, though usually you'd want to do this in a python process calling papermill rather than from a CLI call in which case you already have the exception available to act on.

[axelborja]

By using the papermill in python code instead of the CLI I have been able to choose the right exit code.

Thanks for the suggestion.

[MSeal]

FYI https://github.com/nteract/papermill/pull/485 has added the exit code 138 for cli commands that fail with a dead kernel error.

[MSeal]

I think this closes the issue and will be in the next release of papermill. If there's other types of failures we want to reserve codes for please create PRs for those patterns following Raman's link PR. Thanks!

[axelborja]

Thank You Matthew !! 🏅

[Philmod]

Hey @axelborja and @MSeal !

I would like the execute_notebook method to throw when the kernel crashes with OOM. Right now, the command is stuck "running".

I use papermill version 2.2.2 which should have this change.

My code:

import sys
import papermill as pm
pm.execute_notebook("/tmp/mod.ipynb", "/tmp/mod_out.ipynb", progress_bar=False, log_output=True, stdout_file=sys.stdout, stderr_file=sys.stderr, allow_errors=True)

The notebook is just filling up memory until OOM:

a = []
while True:
    a.append(' ' * 10**6)

Am I missing something?

Thanks!

[Philmod]

Hey @axelborja and @MSeal, any help on this ^ ?
Thank you so much!

[MSeal]

@Philmod exiting with a status code is only for calling from the CLI since it's above the programmatic interface. If you are calling from Python you have to catch the appropriate exception and handle it as you see fit. In this case you'd need to catch the nbclient.exceptions.DeadKernelError and do as you see fit about the OOM (or other terminal) state that would cause the exception.

Also apologies for being slow to respond on your initial ping

[Philmod]

Thanks @MSeal !