nbconvert 6 doesn't run `preprocess_cell` with ExecuteProcessor

[jph00]

With the latest nbconvert from pypi, preprocess_cell in subclasses of ExecuteProcessor no longer run. Looking at the source, I can see that nothing is calling preprocess_cell any more.

I'm not sure if this is a bug, a missing feature, or a question... It at least appears to be some missing documentation.

Can you please advise how, in nbconvert v6 / nbclient, you recommend executing a notebook (without saving it) and preprocessing each cell before executing?

(More generally, I'm totally overwhelmed at all the things that have broken in nbdev with nbconvert v6, and I'm only finding the answers by looking at the source code for v6 and commit logs. Is there anywhere I could find an 'upgrade guide', and/or a list of API changes? I've read the release notes, but they don't show many of the changes.)

[MSeal]

The missing preprocess_cell looks like an oversight on my part when we moved the code to nbclient a long while back. So I'd classify this as a bug and I think I see what we need to do to fix it. Nbclient doesn't directly support cell manipulation just-in-time on it's own, and papermill persists notebooks as it executes, so there's not simple way to run a notebook and preprocess in the same way without the bug fixed. If you know what you want to change before executing any cells, you could run a custom preprocessor over a notebook and then run it through nbclient directly as an alternative pattern.

I'm sorry you feel frustrated with the change. In general most things should be behaving as they were with a few exceptions. And to be fair this is a major version bump on a large repo that's been sitting in alpha/beta releases for 6 months. We tried to test all usage patterns but it's a bit of a leaky boat with all the inheritance involved downstream, and a lot of expected behavior from nbconvert was/still is undocumented conventions. So apologies again that it's causing you friction, it wasn't intended to do so. There isn't a direct upgrade guide because of this intent, but I we can try to be responsive and update docs around things we missed or any unintentional changes in behavior.

[MSeal]

Issue is resolved. There's 1 more patch needed for an unrelated issue before release. I have a working branch of nbdev that uses nbconvert 6.0 with minimal changes I'll PR once nbconvert 6.0.2 is released.

[willingc]

Thanks @MSeal for your contributions to turn this around quickly. :sunny:

@jph00 FYI 6.0.2 has been released.

[jph00]

Gosh that was speedy.

And @MSeal even jumped into nbdev and fixed our code for us too! 🎉😍

[willingc]

@MSeal is awesome and really dedicated. 🏆

[MSeal]

I was especially keeping an eye on issues atm given the major release after over a year of no releases. With :baby: on the way I want to get things stable with nbconvert as there'll be one less active maintainer for a while.

[jph00]

Surely once you have 👶 that means you'll have an extra contributor?... ;)