Nbconvert: Docs haven't built for a month

[mpacer]

Somehow, the docs are first installing master, and then they are overwriting that with stable:

Installing collected packages: nbconvert
  Found existing installation: nbconvert 5.0.0.dev0
    Uninstalling nbconvert-5.0.0.dev0:
      Successfully uninstalled nbconvert-5.0.0.dev0
  Running setup.py develop for nbconvert
Successfully installed nbconvert-4.2.0

Not sure what changed.

https://readthedocs.org/projects/nbconvert/builds/4618074/

[mpacer]

Sphinx did a beta release on nov 6, which was when the failure started.

I don't think it's related, but it might be.

[mpacer]

@willingc Do you know of anything that's been happening in this space that might change how our installation commands are read?

[willingc]

@michaelpacer Hmm... I'll be up your way next week. I'll take a look this weekend and we can follow up next week. I didn't see anything in my quick scan about why that might be the case.

I've gone ahead and made the stable release the default in RTD and opened a PR to add the stable badge to the README #485.

[mpacer]

~Is it possible to similarly include a badge for the latest branch as well? It would be useful to know that any PRs are not breaking RTD in terms of the current version of master.~ Edit: it's already there.

I feel like we had a conversation about actually testing for RTD breakage as part of travis-CI and it was proving difficult… should we start up that conversation again?

Also, I just noticed that strangely it was getting nbconvert from conda not from pypi… maybe something changed in terms of how rtd handles conda envs?

[mpacer]

Command that breaks is

python /home/docs/checkouts/readthedocs.org/user_builds/nbconvert/conda/latest/bin/sphinx-build -T -E -b readthedocs -d _build/doctrees-readthedocs -D language=en . _build/html

Which isn't an install command…but a build command. Why is it installing something on a build at all?

[mpacer]

Aha…

From the sphinx docs for the command sphinx-build

-E
Don’t use a saved environment (the structure caching all cross-references), but rebuild it completely. The default is to only read and parse source files that are new or have changed since the last run.

I wonder if that's the cause…

[willingc]

Also, I just noticed that strangely it was getting nbconvert from conda not from pypi… maybe something changed in terms of how rtd handles conda envs?

Maybe. I tried pinning Sphinx to 1.4.8 but same issue so that's not the issue. If I find something out this weekend, I'll ping you :-)

[mpacer]

Sounds good…grazie!

[mpacer]

Any luck/thoughts?

[willingc]

Whelp. I spent a few hours yesterday. With frustratingly little success. When all was said and done, I'm not sure that RTD was clearing out my builds properly. A few thoughts and things I tried. I had the most success with changing to Python 3.4, pinning sphinx to 1.4.8, and changing the subprocess.run to subprocess.call. But I'm not happy with that so... We can pair on Tues or Wed and get it running.

[mpacer]

That's…odd (that it would depend on a python version difference)…

I'm trying to trace down any changes that might have occurred and I've narrowed it down to being sometime between Nov 1 and Nov 4 that the change occurred that broke our current doc building system.

[mpacer]

Ok some progress made and weirdness found.

I've been messing around on https://github.com/michaelpacer/nbconvert/tree/unbreak_docs (build report is here)

So if I remove nbsphinx, our custom install command, and dependencies on .ipynb, it builds ok, though with much less content.

However, when I add nbsphinx back in, things get weird. In particular we run into this breaking error:

File "/home/docs/checkouts/readthedocs.org/user_builds/nbconvert-mp/conda/unbreak_docs/lib/python3.5/site-packages/nbsphinx.py", line 1121, in depart_code_latex
assert lines[0] == ''

which… simply doesn't happen on my local sphinx runs.

Further investigation will ensue.

[mpacer]

Ok this isn't exactly the same but it does point out that sphinx 1.5 breaks nbsphinx https://github.com/spatialaudio/nbsphinx/issues/78

weirdly…it doesn't seem to do so locally. I'm not sure why.

[mpacer]

Ok, so even though sphinx released 1.5.1 yesterday…we're getting

"Error: Package missing in current linux-64 channels:

• sphinx >=1.5.1"

Which is…unfortunate since it seems to address that error (though I still can't reproduce the error even with 1.5.0.

Edit: Ah — it's a conda thing.

[willingc]

If you pip install sphinx=1.5.1 in the environment.yml file, the latest sphinx will be installed.

[mpacer]

Sorry — I stopped updating here, because it goes deeper and different.

There was a problem in the latest docutils build 0.13.1 that interacts with sphinx.

This somehow also ties in with some other changes that were made inside sphinx to make 1.5.0 incompatible with nbsphinx.

Fixes were merged into master on sphinx, but the release from yesterday (1.5.1) did not include the changes on master. And they don't release for another month. I'm going to pin our sphinx at 1.4.9.

[mpacer]

In the meantime I found an issue with cross referencing across document formats that is unrelated to breakage that i'm going to be posting on nbsphinx in a bit. The issue was https://github.com/spatialaudio/nbsphinx/issues/82

[mpacer]

So I spent the last few hours trying to get a local version of readthedocs running, and that didn't go super well. I hope it's not because I use conda rather than virtualenv, because from what I understood having those on the same machine is a recipe for a nightmare since they try to shadow each other's activate scripts (among other things).

Our api issues would be fixed if we would release 5.0, given that min asked for that on #482…I'm open to this idea, not because it's a real solution to the problem but because it gets us back to a state of successfully building docs, at least until some of these other pieces can get sorted out (e.g., the nbsphinx/sphinx/docutils stuff, conda in rtd, rtd inside conda, editable installs in conda environment.yml).

Does that seem like a "for now" ok solution?

[mpacer]

Released 5.0.0; forgot to merge my unbreak_docs branch. Started a 5.x branch on nbconvert, and am now pointing the docs to the 5.x.

I feel like this still doesn't count as taken care of though because it will break as soon as there is an API change.

[minrk]

@michaelpacer great, thanks! For reference, we don't generally need to create a 5.x branch until we start merging things that should be in 6.0 but not in 5.x. If this is likely to happen very soon, there's no cost to doing it immediately, but we often get to a few minor releases from master before we need to start the branch.

[willingc]

@michaelpacer I merged your PR :tada: I'll let you have the honor of closing this. I would recommend creating a new issue to revisit the nbsphinx and our build when sphinx 1.5.2 comes out.

[mpacer]

It also looks like https://github.com/spatialaudio/nbsphinx/pull/85 might deal with the 1.5.1 incompatibility (presumably going forward into 1.5.2). I think 1.5.0 will continue to be broken, because it has issues with docutils, but I'll still close this and create a new issue.