Dvc.org: add "Jupyter notebook" article

[efiop]

[efiop]

@jurasan

[colllin]

If you have any quick tips here, I would appreciate them. I typically use notebooks for development and inline visualization, and I'm trying to migrate a project to dvc right now — my first dvc project 🎉. I'm thinking it might be best to develop and debug in the notebook as usual, then when I'm ready to run the notebook end-to-end, use e.g. dvc run -d train.ipynb -o training.html -o checkpoint.pt jupyter nbconvert --to html --execute train.ipynb.

[efiop]

Hi @colllin !

I see by the name of the notebook train.ipynb, that you are splitting your pipeline into separate steps, that you then plan to run using dvc. That is precisely what we usually recommend! You should be all set :tada: Please don't hesitate to share your experience, we would really appreciate it. :slightly_smiling_face:

[mlisovyi]

Any progress on such example?

[colllin]

@mlisovyi I’ve been using Jupyter in a pipeline with a command like:

jupyter nbconvert Train.ipynb --clear-output --inplace --execute --ExecutePreprocessor.timeout=-1

This executes the notebook and overwrites it in-place, as if I had opened it in Jupyter and ran the entire notebook manually and saved it. I then commit the resulting notebook to git. I also specify some outputs which are cached: a directory for model checkpoints and a directory for logs. For dependencies, I specify the notebook itself as well as a directory of supporting modules.

I believe the initial command to set it up looked something like

dvc run -d Train.ipynb -d src/ -o checkpoints/ -o logs/ jupyter nbconvert Train.ipynb --clear-output --inplace --execute --ExecutePreprocessor.timeout=-1

You might also need to specify a name for the pipeline step somewhere in that command — I used train.dvc, which I can then execute using dvc repro train.dvc.