Evalml: RTD builds fail with numpy version RuntimeError

[dsherry]

Problem
All RTD builds fail with this error during ipynb execution:

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

Here's a recent failure

Marked as blocker because we can't release again until we can build docs.

[dsherry]

I was able to repro this using the RTD docker container.

Steps:

• docker pull readthedocs/build:6.0, from here
• docker run -it readthedocs/build:6.0 /bin/bash
• cd /home/docs/
• Copy-paste the commands from the RTD output.
  
  
  
  • I had to clone the repo using https instead of ssh
  
  
  • For pip installation, I had to wrap each clause which used !<>= operators for version matching in quotes.
  
  
  • Instead of running the final sphinx command, I ran:
  
  

source /home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/main/bin/activate
cd docs/source/
sphinx-build -T -E -W --keep-going -d _build/doctrees-readthedocs -D language=en . _build/html

I was able to get the same output as in RTD: "RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd"

[dsherry]

I followed this tip about incompatible C API versions for numpy (versions listed here), and I found that if I update numpy to 1.20.0rc1 the error goes away. But now I'm getting other errors in RTD:

reading sources... [  0%] demos/cost_benefit_matrix
[LightGBM] [Fatal] Check failed: (best_split_info.right_count) > (0) at /__w/1/s/python-package/compile/src/treelearner/serial_tree_learner.cpp, line 661 .

[LightGBM] [Fatal] Check failed: (best_split_info.right_count) > (0) at /__w/1/s/python-package/compile/src/treelearner/serial_tree_learner.cpp, line 661 .

[LightGBM] [Fatal] Check failed: (best_split_info.right_count) > (0) at /__w/1/s/python-package/compile/src/treelearner/serial_tree_learner.cpp, line 661 .

[LightGBM] [Fatal] Check failed: (best_split_info.right_count) > (0) at /__w/1/s/python-package/compile/src/treelearner/serial_tree_learner.cpp, line 661 .

...
nbsphinx.NotebookError: TemplateNotFound in demos/cost_benefit_matrix.ipynb:
rst.tpl
...
jinja2.exceptions.TemplateNotFound: rst.tpl

It took ~10min to get the error so I'm guessing the error itself isn't from that first demo ipynb.

[dsherry]

Hmm, I'm getting that error both when I base my branch on main, and also when I base my branch on main from last week before the breakages started.

I was able to repro the original error in the RTD docker container I pulled down; I'm not able to repro this error there... the docs build fine for me locally.

That plus the other warnings from the output (which I forgot to add earlier) are making me think this is a threading issue of some sort. My hunch is that its got something to do with the version of nbconvert relative to the other packages.

/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/ds_1520_rtd_numpy_failure/lib/python3.7/site-packages/joblib/externals/loky/backend/resource_tracker.py:304: UserWarning: resource_tracker: /tmp/joblib_memmapping_folder_271_c3607b3c57b84aabafecf05a68db4b7d_a0b5437a255241018fe1adfffc8c66bb/271-140365248875024-672b2943f8b34c7f9a1d4b66c749f0b6.pkl: FileNotFoundError(2, 'No such file or directory')
  'resource_tracker: %s: %r' % (name, e))

[dsherry]

Next on the list to try (thanks @freddyaboulton for chatting about it):

• Try python 3.8 instead of python 3.7
• Print out pip freeze on readthedocs and diff that with pip freeze in my local docker builder, and see what if anything is different in the envs
• Try installing dependencies using conda or setuptools instead of pip
• Perhaps there are other .readthedocs.yml parameters which can be tweaked.
• For attempts at local repro, try readthedocs/build:7.0 (latest) instead of readthedocs/build:6.0 (stable) which is what I was using
• Read up more on how to get sphinx-build to work with -b readthedocs (use RTD sphinx builder), because I haven't been able to get that to work yet in the docker container I've pulled down.

[dsherry]

Against all odds... I was able to get a green RTD build today! 🎊

Yesterday I learned a bit more about how RTD runs builds. Through some trial-and-error searching, I was able to get the build to pass locally in the RTD docker container, by overriding the version of setuptools the RTD install process uses.

I don't understand the root cause, why this happened now and why it didn't affect featuretools, compose or (seemingly) other open source projects. However I have the following observations to offer:

• I noticed that as soon as numpy put out their 1.20.0rc1 pre-release on Dec 3rd, our RTD build started failing... 🤔 more details here.
• I noticed that the RTD docker container includes several prebuilt pyenvs, for python 2.7 and 3.5-3.9. The dockerfile includes calls to install numpy in most of those pyenvs with the --only-binary flag passed to pip
• These three posts had the error messages which came up during my trial and error debugging. The root cause for all of those cases was that somehow there were two versions of numpy installed, and that removing the older version fixed the issue.
• I saw that the RTD build steps were installing an old version of setuptools (41.0.1) before installing our dependencies. That's confusing, because the dockerfile uses setuptools 45.1.0 to install numpy and other dependencies once the pyenvs are created.
• Through trial and error I was able to observe that upgrading the setuptools version to >=45.2.0 (45.1.0 would probably work, haven't confirmed) prior to installing our package's dependencies got rid of the errors sphinx-build was raising about a numpy/numpy-C-API version mismatch, and allowed RTD to complete successfully!!!

I'll get a PR up shortly with the fix, and will hold my breath until that's merged... 😆 I plan to file something with RTD as well.