Conda-forge.github.io: DOC matplotlib versus matplotlib-base

[beckermr]

We recently ran into this with pystan: https://github.com/conda-forge/pystan-feedstock/pull/27

Apparently, it is sometimes preferable to depend only on matplotlib-base. We might want to add a section to the special dependencies describing when and why.

@maresb has more details I think

[beckermr]

See also this issue: https://github.com/jupyter/docker-stacks/pull/896#issuecomment-515888808

[maresb]

Thanks @beckermr for bringing this up here. I don't have much to add beyond what's in the comment linked above.

Whenever I personally conda install a package which requires matplotlib instead of matplotlib-base, I have been submitting a pull request to the corresponding feedstock. I could imagine that in 99% of the cases where a package requires matplotlib it is more appropriate to require matplotlib-base. This makes me wonder if it would be sensible to somehow make a bulk pull request to all (active) feedstocks which require matplotlib to switch over to matplotlib-base. (Or simply to remove pyqt from matplotlib.) I'm relatively new here, so I don't understand the design decisions.

[beckermr]

So the only difference is whether or not a functional backend (pyqt) is installed too?

We can make bulk PRs via a special migration using the conda-forge bots if we think it is appropriate.

What happens in an env where only matplotlib-base is installed and someone tries to use matplotlib?

[maresb]

I use JupyterLab as my backend, and matplotlib (installed as matplotlib-base) runs perfectly fine without having qt/pyqt installed.

[beckermr]

OK. I did some testing on a linux box and on my mac. Without pyqt, everything appears to work out of the box.

I am confused on one thing though. On linux, it appears that matplotlib-base is shipping the tk backend already

Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.pyplot as plt
>>> plt.plot(list(range(10)))
[<matplotlib.lines.Line2D object at 0x7f4cc9754d30>]
>>> plt.show()
>>> plt.get_backend()
'TkAgg'
>>> exit()
(test-mpl) [beckermr@astro0010 ~]$ conda list
# packages in environment at /gpfs02/astro/workarea/beckermr/miniconda3/envs/test-mpl:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
bzip2                     1.0.8                h516909a_1    conda-forge
ca-certificates           2019.9.11            hecc5488_0    conda-forge
certifi                   2019.9.11                py37_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
freetype                  2.10.0               he983fc9_1    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
kiwisolver                1.1.0            py37hc9558a2_0    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_2    conda-forge
liblapack                 3.8.0               14_openblas    conda-forge
libopenblas               0.3.7                h6e990d7_3    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
matplotlib-base           3.1.2            py37h250f245_1    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.17.3           py37h95a1406_0    conda-forge
openssl                   1.1.1d               h516909a_0    conda-forge
pip                       19.3.1                   py37_0    conda-forge
pyparsing                 2.4.5                      py_0    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
setuptools                42.0.1                   py37_0    conda-forge
six                       1.13.0                   py37_0    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
tornado                   6.0.3            py37h516909a_0    conda-forge
wheel                     0.33.6                   py37_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

Does this need to be separated out like the qt backend?

@conda-forge/matplotlib for viz and insight

[dopplershift]

Technically speaking, the matplotlib-base package ships the code for all backends. The matplotlib package does not actually ship any files, all it does is depend on matplotlib-base and pyqt. Matplotlib as of v3.0 (and I think back-ported to 2.2.3) does only run-time detection of backends based on available libraries.

I'm not sure why matplotlib-base depends on tk and only on linux. As I understand it, though, that's effectively harmless as tk is required for Python on Mac and Linux.

[jjhelmus]

The python conda package depends on tk (except on Windows where it is vendored) as it is needed for the tkinter module in the standard library. Even if matplotlib-base dropped the dependency on tk it will still be installed unless one is using a custom built python package.

[beckermr]

OK. So blanket instructions that dependencies on matplotlib should be on matplotlib-base seems like it would be fine on all platforms?

[dopplershift]

I think that would be fair.

[beckermr]

I added docs. We might consider both a migrator and a PR on the linter. I will make issues for those.

[beckermr]

OK. This is 503 packages, so we def need a migrator if we want it.

[CJ-Wright]

how do you know if you need regular vs base? My understanding of the difference is if you just want the matplotlib "spec" or if you want the full batteries included plot making thing. IDK if many projects draw that distinction and may not explicitly depend on pyqt even though they want qt visualizations.

[beckermr]

It is pretty subtle I agree. I don't have a great hard and fast rule, except to say that if you need qt/pyqt in and of itself, you should not be relying on matplotlib to supply it given how the packages are structured.

I would say that if your code simply imports matplotlib with no references to a specific backend (namely qt) then it does not need a pyqt dep. If a user has that as their default backend, they are responsible for installing that backend.

[maresb]

Was there ever any justification for including pyqt with matplotlib in the first place? It seems to me like this was a mistake, but I could imagine it would be disastrous to suddenly remove pyqt from matplotlib due to implicit dependencies.

Wouldn't it be better to suggest never to use matplotlib? It's more explicit to require either matplotlib-base + pyqt or just matplotlib-base.

[beckermr]

I think the idea was to make sure that anyone who installed matplotlib would have a working qt backend out of the box so to speak. The principle of least surprise.

[dopplershift]

To be more specific, including pyqt is the only way to guarantee GUI functionality across all of the platforms.

[jakirkham]

Yeah this follows from how Anaconda was packaging it in the first place.

[beckermr]

frack what is this error

import: 'missingno'
Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/test_tmp/run_test.py", line 2, in <module>
    import missingno
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/site-packages/missingno/__init__.py", line 1, in <module>
    from .missingno import matrix
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/site-packages/missingno/missingno.py", line 4, in <module>
    import matplotlib.pyplot as plt
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/site-packages/matplotlib/pyplot.py", line 115, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/site-packages/matplotlib/backends/__init__.py", line 63, in pylab_setup
    [backend_name], 0)
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/site-packages/matplotlib/backends/backend_tkagg.py", line 4, in <module>
    from . import tkagg  # Paint image to Tk photo blitter extension.
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/site-packages/matplotlib/backends/tkagg.py", line 5, in <module>
    from six.moves import tkinter as Tk
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/site-packages/six.py", line 203, in load_module
    mod = mod._resolve()
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/site-packages/six.py", line 115, in _resolve
    return _import_module(self.mod)
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/site-packages/six.py", line 82, in _import_module
    __import__(name)
  File "/home/conda/feedstock_root/build_artifacts/missingno_1575567128173/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python2.7/lib-tk/Tkinter.py", line 39, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: libX11.so.6: cannot open shared object file: No such file or directory
Tests failed for missingno-0.4.2-py_1.tar.bz2 - moving package to /home/conda/feedstock_root/build_artifacts/broken
WARNING:conda_build.build:Tests failed for missingno-0.4.2-py_1.tar.bz2 - moving package to /home/conda/feedstock_root/build_artifacts/broken
WARNING conda_build.build:tests_failed(2231): Tests failed for missingno-0.4.2-py_1.tar.bz2 - moving package to /home/conda/feedstock_root/build_artifacts/broken
TESTS FAILED: missingno-0.4.2-py_1.tar.bz2

[beckermr]

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=101076&view=logs&jobId=1bf226d3-0e2f-52d8-fa93-7d9e633347b3

[beckermr]

I guess pyqt is pulling in this dep?

[CJ-Wright]

https://github.com/conda-forge/pyqt-feedstock/blob/master/recipe/meta.yaml#L45

[beckermr]

But that is a build cdt not a runtime dep from conda. We don’t ship most of x11 so how did this ever work?

[CJ-Wright]

I think there is a host cdt on mesa that might pull this in as well. I think we do ship large chunks of x11, there are 64 packages that start with xorg.

[beckermr]

We do ship this lib, https://github.com/conda-forge/xorg-libx11-feedstock/blob/master/recipe/meta.yaml

[beckermr]

But I still don't follow how this failure happens for mpl-base but not mpl when pulling pyqt in an env. Neither pyqt or qt have this library in them and none of the other deps I can find have it either.

[tacaswell]

@maresb tk does not support hi-dpi displays but Qt does so you get a significantly better user experience with Qt.

We pulled matplotlib-base out the back so that it would be possible to install matplotlib in situations where you know there will never be a GUI (ex docker containers running on a cluster or serving jupyter notebooks) to save the bits required to install Qt and all of its dependencies.

[dopplershift]

@beckermr It looks like it hits that error when trying to import tkinter. If you have pyqt installed, matplotlib will default to qt5agg backend over tkagg, so maybe it just never hits that in those cases?

[beckermr]

Right @dopplershift that makes sense.

hi @tacaswell long time no see since grad school! I hope all is well.

We had the idea of not having the dep for the same reason. It simply appears the import breaks without using some other backend. I wonder if we can force a default to agg in the CI so this doesn't happen?

[tacaswell]

You can set the env MPLBACKEND='Agg' which should override the default fallback behavior. Unsetting DISPLAY should also work.

@beckermr Sorry I missed you when you were at BNL for the gpu hackathon. At NSLS-II we have an active collaboration with the APS so I suspect I will end up visiting Argonne soon-ish.

[beckermr]

OK. I have submitted this PR: https://github.com/regro/cf-scripts/pull/710 to stop the migration until we can fix the CI env.

[beckermr]

@CJ-Wright for viz

[beckermr]

I took a pass at testing all of this locally. It would work fine if we injected 'Agg' in the env per comments above.

[dopplershift]

On the matplotlib feedstock, this is solved by yum_requirements.txt asking for mesa-libGL.

[beckermr]

Right. Adding a yum_req for every feedstock won't solve this on a mac though IIUIC. I think the backend var is cleaner, but I might be wrong here.

[dopplershift]

On macOS by default it will use the `MacOSX' backend, so it shouldn't be trying to import tkinter, which is why it's not needed for mac on the matplotlib feedstock.

[dopplershift]

To be clear, I'm not arguing for one approach or another. But IMO we should be consistent in solving this problem.

[beckermr]

Ohhhhhhhhhhhh. Well you know a ton more than me so your opinion is def useful! So @dopplershift Do you think it is better for conda smithy to automatically add this yum_requirement or automatically add the env var? The env var seems cleaner, but both might be problematic.

[tacaswell]

For Matplotlib we build the c-code required to talk to tk so I think we need to have all of the libraries available. For downstream libraries using the env to skip the requirements seems like the better option.

[dopplershift]

The problem is that the yum requirements only affects the build environment, it does nothing to ensure the library is present in the user's runtime environment. Of course, neither does setting the env variable. If there's a missing dependency then we should do something about it, but I'm not clear what would be missing based on what @CJ-Wright said above.

I'm not clear enough about what of X11 conda-forge ships and the rationale to know what the right solution is.

[beckermr]

So downstream users are expected to install the backend of their choice as opposed to us forcing them to download qt. I think the only issue is that the runtime env of the CI is not setup.

[dopplershift]

That's not...really true. matplotlib-base comes with tk and macosx backends already built--which is what makes having qt as optional not a horrific burden. If there's something else that should be in the deps somewhere to make tk work properly, that's IMO what should happen. I'm just not clear whether there is or isn't.

[beckermr]

Hmmmmmm. So to the extent that tk appears to require X11, you would say that conda should ship it?

Right now X11 is the wild west. Some packages use what is on conda, others do not.

[beckermr]

My guess is that when most users install the current packages on linux, it is picking up their local X11 install when linking.

[CJ-Wright]

@jjhelmus was there discussion about python's tk dependence?

[beckermr]

Is it python or matplotlib? I am so confused.

[CJ-Wright]

Python also has a tk dep https://docs.python.org/3/library/tk.html

[jjhelmus]

was there discussion about python's tk dependence?

Python depends on tk because it needs to build the tkinter package in the standard library, similar to the dependency on sqlite needed for the sqlite3 package in the standard library.

There has been some discussion on the tk packages dependency on the system X11 and if this should be changed. A good starting point is conda-forge/tk-feedstock#17

[beckermr]

Alright. It appears we are again blocked on where we think X11 should live in the stack. This is again a case where some sort of virtual package could be useful. It could detect if the user has X11 available locally and then we can have variants of packages that depend on x11 that get selected depending on this fact. However this work is well beyond the scope here and probably needs more discussion.

In the mean time, I think simply forcing the Agg backend via an env var in the CI is the simplest solution and will help solve the big qt package dependence issue as well.

What do you think @dopplershift?

[dopplershift]

Yeah, sounds like X11 in conda-forge is a bit of a mess. Whatever option is easiest to put in a migrator I suppose. I’m guessing there’s nothing we can put in matplotlib-base to solve this.

[beckermr]

https://github.com/conda-forge/staged-recipes/pull/10426 We are seeing a very weird error here. Can someone on this thread take a look? My understanding was that this should not be happening.

[beckermr]

@conda-forge/core @scopatz Can we reopen this issue until we have all of these errors sorted out?

[beckermr]

Thank you @ocefpaf !

[beckermr]

Alright, I am understanding more finally. It looks like we will need to suggest / add a yum requirements file to the feedstocks in the migrator. I think this will fix the import issues.

[beckermr]

fixed. closing