Papermill: How to connect papermill to a remote server?

[kebab-mai-haddi]

I want to run a .ipynb script via papermill on a remote server where I have installed certain dependencies. It should be noted that these dependencies are installed in an environment and that I have to provide as a kernel to my Python(Papermill) call for executing the script.

[kebab-mai-haddi]

Need help

I did read in the doc(https://buildmedia.readthedocs.org/media/pdf/papermill/latest/papermill.pdf) about creating a new engine. However, there is none API reference of specifying the jhub url or any such thing). It simply states:

Creating a new engine
Papermill engines need to inherit from the papermill.engines.Engine class.
In order to be used, the new class needs to implement the class method execute_managed_notebook. The call
signature should match that of the parent class:
class CustomEngine(papermill.engines.Engine):
@classmethod
execute_managed_notebook(cls, nb_man, kernel_name, **kwargs):
pass
nb_man is a nbformat.NotebookNode object, and kernel_name is a string. Your custom class then needs to
implement the execution of the notebook. For example, you could insert code that executes the notebook remotely on
a server, or executes the notebook many times to simulate different conditions.

@MSeal @willingc

[MSeal]

To note first I'll note that running the notebook on a remote server will have a number of potential pain points, but they're likely all manageable. It will also require a bit of code exploration to achieve as you'll need to learn some internals of nbconvert to achieve success.

That engine function described is called after the notebook document is loaded into memory and parameters have been applied. It captures the point at which papermill is about to execute the notebook and holds the responsibility to complete that task.

In your case you'll need to modify https://github.com/nteract/papermill/blob/master/papermill/preprocess.py#L10 class to run the connect it's km (kernel manager) / kc (kernel client) to your remote machine. The kn launches and cleans up the kc which runs the actual kernel communication. In this case you'll need to launch the kernel process on another machine where your dependencies are, and have the km monitor / cleanup when the task is done. This will likely mean reading the code in https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/manager.py to see how you can make a remote kernel manager. Then using that manager to launch the remote kernel. In this process you'll likely need to figure out how you'll be passing credentials to log into the remote server and to make sure you open up the appropriate ports. There's likely one or two other catches I'm not aware of that you might hit in the process.

[MSeal]

An easier use case may be to instead execute papermill entirely in the server that has the dependencies setup. You can more easily pull a notebook into a server than to run code remotely.

[kebab-mai-haddi]

@MSeal okay, so say I have a basic setup of my own software(Airflow) which offers running notebook as a service for my users(data scientists at my firm). They have their own dependencies and requirements which I do not want to install in my ecosystem. Also, I need some return values from the execution of their script(which they will write in their code, say they work on a pandas dataframe and return it).

What would be the way to achieve this?

[MSeal]

The way many production systems approach this is to isolate the execution from the scheduler by making the Airflow Node a remote execution task, though Kubernetes or Titus or Repo2Docker or some other container execution tools. You let users defines their dependencies within the container execution context and then your scheduler needs to know how to execute inside that system. This is no different from how one would want to schedule non-notebook executions with isolated dependencies, so there's a fair bit of information on the web on the topic.

For return values this can be tricky, but again falls into the same problem space as non-notebook cases. Even within the same execution context a notebook has a subprocess that can't directly return memory objects like dataframes to the parent process. Many times users will implement some sort of data write to their remote blob store system (S3, ABS, etc). The crudest version of this is to pickle the data you want and unpickle it in the downstream job that needs the data, but this can cause issues if there's any package differences. The cleanest way is to save complex data to a data table or other well known format via arrow or something else. Scrapbook has a WIP task which does enable this save pattern and direct support for pandas dataframes, but the interface is still being worked out: https://github.com/nteract/scrapbook/pull/37

Hope that helps!

[kebab-mai-haddi]

that really helped!
thanks a lot @MSeal !

[kebab-mai-haddi]

@MSeal , I am almost done with your method of remote execution. Here is how I am doing it:

• start the jhub pod(where the user's notebook and environment resides) if not already started using the REST API of JuPyterhub
• exec in the pod
• run the papermill command

While doing so, I am unable to exec in the pod as per the given kubernetes python client.
This is the question.
Since you were able to help me out in this scenario, I'd be indebted to your help in this last step as well.

[willingc]

Glad that this all worked out @avisrivastava254084.

[kebab-mai-haddi]

@willingc yeah :)
I owe it to @MSeal

Will write a blog post soon :)