Notebook: Long running jobs in the cloud

[biobenkj]

In terms of reproducible research, this is just spectacular! Often times, some next generation sequencing experiments (life sciences) will require large compute machines, resources, etc. so cloud computing is ideal. Incorporating the ability to leverage these large machines through notebooks is possible for short jobs, but not long jobs (>1 hr).

Is it possible to perform analysis as described above but "detach" from the machine instead of having to stay logged in the entire time (e.g. Amazon EC2)?

[minrk]

There isn't a 'detach' mechanism per se, since the document state is computed in the browser, so without a browser the transform of messages to document won't happen. We do plan to come up with a way to move document state to the server-side, which would alleviate this. For now, there are two ways that you can do an offline run of a notebook, however:

1. use jupyter nbconvert --execute --to notebook MYNOTEBOOK, which opens the notebook and runs it with a kernel, storing the output in a new notebook (add --inplace to overwrite the existing notebook instead of making a copy with the results). This isn't a perfect mirror of the in-browser run if there are javascript outputs or widgets that do certain things, but for more static outputs it should work fine.
2. script with a browser emulator, such as phantomjs. I don't have an example for such a script, but it would better match the experience of running a notebook in a real browser.

[biobenkj]

Thanks for the reply. I'll give the workaround a shot.

[ellisonbg]

Closing.

[divamgupta]

Any updates when background jobs will be officially supported?

[chogg]

I was wondering if this is possible too. When I turn off my browser, the calculations happen, but they are not updated in the browser when I turn it back on. Maybe there is a way to refresh the cells that I don't know about. Thanks @minrk for the tip.