Nixpkgs: Ship usual Python packages with QGIS

[das-g]

TL;DR:

QGIS from NixPkgs seems to be missing the following Python packages, which may be relied on by QGIS plugins:

• future
• httplib2
• markupsafe
• plotly
• pyproj

Issue description

For non-Nix(OS) systems, QGIS usually ships with a base set of Python packages. Some of them are required by QGIS itself, while others aren't. But QGIS plugins or Python scripts of the user may still rely on them being accessible from within QGIS.

The QGIS in NixPkgs currently doesn't ship with the complete set of them:

QGIS 3.4 from NixOS release-19.03 currently has these python3Packages in its buildInputs:

• chardet
• dateutil
• gdal
• jinja2
• numpy
• owslib
• psycopg2
• pygments
• pyqt5
• pytz
• pyyaml
• qscintilla-qt5
• requests
• sip
• six
• urllib3

The Ubuntu package python3-qgis (from https://qgis.org/ubuntu-ltr) currently depends on these packages:

• python3-dateutil
• python3-future
• python3-httplib2
• python3-jinja2
• python3-markupsafe
• python3-owslib
• python3-plotly
• python3-psycopg2
• python3-pygments
• python3-pyproj
• python3-pyqt5
• python3-pyqt5.qsci
• python3-pyqt5.qtsql
• python3-pyqt5.qtsvg
• python3-pyqt5.qtwebkit
• python3-requests
• python3-sip
• python3-six
• python3-tz
• python3-yaml

So assuming that python3Packages.pytz corresponds to python3-tz and python3Packages.pyyaml to python3-yaml, the ones missing in NixOS QGIS are:

• future
• httplib2
• markupsafe
• plotly
• pyproj

I dunno yet whether what corresponds to

• python3-pyqt5.qsci
• python3-pyqt5.qtsql
• python3-pyqt5.qtsvg
• python3-pyqt5.qtwebkit

is already included in python3Packages.pyqt5.

Steps to reproduce

To see that some of these are indeed missing:

1. launch QGIS
2. in QGIS, click menu "Plugins" > "Python Console"
3. on the Python REPL that appears, enter
   python import future

Expected

module is successfully imported (no output)

Observed

Traceback (most recent call last):
  File "/nix/store/0n8slcq8p5x31kc9hncabsqq9y3fpkzp-python3-3.7.3/lib/python3.7/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/nix/store/xry2nyp8jmrdfrrp0ygq89fyw5vwpzm7-qgis-unwrapped-3.4.5/share/qgis/python/qgis/utils.py", line 672, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'future'

To see that some of these are indeed needed by some Plugins:

1. launch QGIS
2. in QGIS, click menu "Plugins" > "Manage and Install Plugins..."
3. In the plugin list, select "OSMDownloader" (you may use the search bar to find it)
4. click the "Install plugin" button

Expected

Plugin is successfully installed

Observed

Lengthy error message:

"ModuleNotFoundError: No module named 'future'" (click to expand)

Couldn't load plugin 'OSMDownloader' due to an error when calling its classFactory() method

ModuleNotFoundError: No module named 'future'

Traceback (most recent call last):
  File "/nix/store/xry2nyp8jmrdfrrp0ygq89fyw5vwpzm7-qgis-unwrapped-3.4.5/share/qgis/python/qgis/utils.py", line 335, in startPlugin
    plugins[packageName] = package.classFactory(iface)
  File "/home/das-g/.local/share/QGIS/QGIS3/profiles/default/python/plugins/OSMDownloader/__init__.py", line 34, in classFactory
    from .osmDownloader import OSMDownloader
  File "/nix/store/xry2nyp8jmrdfrrp0ygq89fyw5vwpzm7-qgis-unwrapped-3.4.5/share/qgis/python/qgis/utils.py", line 672, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "/home/das-g/.local/share/QGIS/QGIS3/profiles/default/python/plugins/OSMDownloader/osmDownloader.py", line 31, in 
    from .osmDownloader_dialog import OSMDownloaderDialog
  File "/nix/store/xry2nyp8jmrdfrrp0ygq89fyw5vwpzm7-qgis-unwrapped-3.4.5/share/qgis/python/qgis/utils.py", line 672, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "/home/das-g/.local/share/QGIS/QGIS3/profiles/default/python/plugins/OSMDownloader/osmDownloader_dialog.py", line 36, in 
    from .osm_downloader import OSMRequest
  File "/nix/store/xry2nyp8jmrdfrrp0ygq89fyw5vwpzm7-qgis-unwrapped-3.4.5/share/qgis/python/qgis/utils.py", line 672, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "/home/das-g/.local/share/QGIS/QGIS3/profiles/default/python/plugins/OSMDownloader/osm_downloader.py", line 25, in 
    from future import standard_library
  File "/nix/store/xry2nyp8jmrdfrrp0ygq89fyw5vwpzm7-qgis-unwrapped-3.4.5/share/qgis/python/qgis/utils.py", line 672, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'future'

Python version: 3.7.3 (default, Mar 25 2019, 20:59:09) [GCC 7.4.0]
QGIS version: 3.4.5-Madeira Madeira, exported

Python Path:

• /nix/store/xry2nyp8jmrdfrrp0ygq89fyw5vwpzm7-qgis-unwrapped-3.4.5/share/qgis/python
• /home/das-g/.local/share/QGIS/QGIS3/profiles/default/python
• /home/das-g/.local/share/QGIS/QGIS3/profiles/default/python/plugins
• /nix/store/xry2nyp8jmrdfrrp0ygq89fyw5vwpzm7-qgis-unwrapped-3.4.5/share/qgis/python/plugins
• /nix/store/8jkwfdn6hyw7sv464n1a6y5ddzjvzl15-python3.7-qscintilla-2.9.4/lib/python3.7/site-packages
• /nix/store/bn8xa5ww2p9pghjdgcix3w0alwlm00wh-python3.7-setuptools-40.8.0/lib/python3.7/site-packages
• /nix/store/agcwvy4pwb7m8w95x2jws18gy1jcgqlb-python3.7-PyQt-5.11.3/lib/python3.7/site-packages
• /nix/store/7ycqvb5q94l2474ll25cs2yx621qvhrx-gdal-2.4.0/lib/python3.7/site-packages
• /nix/store/ii1vfd7mkfvffvyhhkfxkp869lvkk58i-python3.7-Jinja2-2.10/lib/python3.7/site-packages
• /nix/store/fcjvf99dlfajbwzca2mysr3dxl2r8m9v-python3.7-MarkupSafe-1.1.0/lib/python3.7/site-packages
• /nix/store/i7vrd6sa32jji4brc5rbck2g2r765gdp-python3.7-numpy-1.16.1/lib/python3.7/site-packages
• /nix/store/xb52h37svdj68v72h0dnsszay538jy68-python3.7-psycopg2-2.7.7/lib/python3.7/site-packages
• /nix/store/50p861jrj4a15717kpnlgx2vi1s1jflf-python3.7-chardet-3.0.4/lib/python3.7/site-packages
• /nix/store/n5svn70dgs31qwc5jmfqs433kd864c4y-python3.7-python-dateutil-2.8.0/lib/python3.7/site-packages
• /nix/store/1kx85bwdyq7x7n9lzkli02hg1m0aj8d8-python3.7-six-1.12.0/lib/python3.7/site-packages
• /nix/store/j5awgm0fd0k74iindg9x6p63r8gcdyxx-python3.7-setuptools_scm-3.2.0/lib/python3.7/site-packages
• /nix/store/nn4vig3gcm1bn0900c7hyxjiv3ynj8kx-python3.7-PyYAML-3.13/lib/python3.7/site-packages
• /nix/store/a157w8x2jid5ybs5n65yhcd14qj70i9k-python3.7-pytz-2018.9/lib/python3.7/site-packages
• /nix/store/fqjiffd9g1xj33ajg75g54sh12fj6gm3-python3.7-requests-2.21.0/lib/python3.7/site-packages
• /nix/store/v81hggmfs9p22fv7qg9cw00m67c0r952-python3.7-urllib3-1.24.1/lib/python3.7/site-packages
• /nix/store/waqblxjkq9972igfzz7vyqyfri0pia89-python3.7-asn1crypto-0.24.0/lib/python3.7/site-packages
• /nix/store/hh2f7n57d6w37346ccv6ycnxh4fdsrmn-python3.7-packaging-19.0/lib/python3.7/site-packages
• /nix/store/s76v16hl3g0psk4ppblic3bq69xbg6i4-python3.7-pyparsing-2.3.1/lib/python3.7/site-packages
• /nix/store/iszz2a2jyzhanrjgxq41nxa60fj3nxl8-python3.7-pycparser-2.19/lib/python3.7/site-packages
• /nix/store/cwl7xyfnyn4bbnq3r0739vaw7spvv7bz-python3.7-cffi-1.12.1/lib/python3.7/site-packages
• /nix/store/zpfdn3090jvwx7hkjbq6lli3dmy3qln4-python3.7-cryptography-2.5/lib/python3.7/site-packages
• /nix/store/8ydh5516n1cmi99vlmag6w45vy4gk90x-python3.7-pyasn1-0.4.5/lib/python3.7/site-packages
• /nix/store/k69sv9c7gsba0k93v3277rmj9403gs2x-python3.7-idna-2.8/lib/python3.7/site-packages
• /nix/store/rpagf4q3x2an2qs3c8d7spg2kgiw5n9n-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages
• /nix/store/p63cylj7lin1g7lqxpx0yhqnh1wqpwlw-python3.7-certifi-2018.11.29/lib/python3.7/site-packages
• /nix/store/2vvrlp3hkz96mq49vrzmiirki5amg876-python3.7-pysocks-1.6.8/lib/python3.7/site-packages
• /nix/store/zw6mks59avymjjcpq4zr0sxn43z5sy23-python3.7-Pygments-2.3.1/lib/python3.7/site-packages
• /nix/store/7dnji817pa9015q01y6760898gijh2si-python3.7-docutils-0.14/lib/python3.7/site-packages
• /nix/store/szymii8h3m329m2cf1fr57n11490s781-python3.7-sip-4.19.13/lib/python3.7/site-packages
• /nix/store/hw8kqlhq7s01jpyfra61kw2vh82l6al8-python3.7-OWSLib-0.17.1/lib/python3.7/site-packages
• /nix/store/w51pqp1g0s20hrc65rln0flzp997g89v-python3.7-pyproj-unstable-2018-11-13/lib/python3.7/site-packages
• /nix/store/bn8xa5ww2p9pghjdgcix3w0alwlm00wh-python3.7-setuptools-40.8.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg
• /nix/store/0n8slcq8p5x31kc9hncabsqq9y3fpkzp-python3-3.7.3/lib/python37.zip
• /nix/store/0n8slcq8p5x31kc9hncabsqq9y3fpkzp-python3-3.7.3/lib/python3.7
• /nix/store/0n8slcq8p5x31kc9hncabsqq9y3fpkzp-python3-3.7.3/lib/python3.7/lib-dynload
• /nix/store/0n8slcq8p5x31kc9hncabsqq9y3fpkzp-python3-3.7.3/lib/python3.7/site-packages
• /home/das-g/.local/share/QGIS/QGIS3/profiles/default/python

screenshot (click to expand)

Technical details

• system: "x86_64-linux"
• host os: Linux 4.19.34, NixOS, 19.03.172205.ea497998e4b (Koi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.2
• channels(root): "nixos-19.03.172231.7b36963e7a7"
• channels(das-g): ""
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[FRidh]

future I would expect to be quite essential, as you've demonstrated. plotly I would be a bit more hesitant to. Anyway, it should be straightforward to extend it with optional Python packages.

[FRidh]

By the way, I suggest always cc'ing the maintainer of the expression.

[das-g]

By the way, I suggest always cc'ing the maintainer of the expression.

@lsix, thus?

[das-g]

I can't quite wrap my head around the description in https://nixos.org/nixpkgs/manual/#ssec-stdenv-dependencies. Which of these variables aught to be used for runtime-only dependencies? (As they're obviously not required for building QGIS, that's what the additional Python libraries would be, wouldn't they?)

[das-g]

future I would expect to be quite essential, as you've demonstrated. plotly I would be a bit more hesitant to. Anyway, it should be straightforward to extend it with optional Python packages.

QGIS, depending on QT, GDAL/OGR, GRASS GIS, NumPy and whatnot is already a very "fat" software, so I feel some python packages give-or-take shouldn't matter too much. Thus I'm tempted to add all of these python packages to its dependencies.

We can always make them optional later through an option, if the need for a (slightly) slimmer QGIS arises. (Or is there something particularly expensive about adding plotly, @FRidh?)

[lsix]

Hi,

I have added a extraPythonPackages to the qgis derivation to cover those use cases. If find that some required dependencies are missing, I will add them. If this is something that you need for custom plugins, I would recommend using a custom derivation:

$ nix-shell -p 'qgis.override { extraPythonPackages = (ps: [ ps.numpy ps.future ]);}'

We can for sure add a heavy version of qgis in all-packages.nix that will bring all the extra packages you would expect:

  qgis = …;
  qgisWithDeps = qgis.override { extraPythonPackages = (ps: [ ps.… ]); };

This will add nearly no load to hydra servers since qgis itsefl does not need to be rebuilt.

[das-g]

What do you mean by "custom plugins", @lsix?

I'd expect that if I install QGIS, I can install all plugins it offers me in its built-in plugin manager from the official QGIS plugin repository (https://plugins.qgis.org/plugins/), just like I can install arbitrary compatible Firefox plug-ins from addons.mozilla.org in Firefox.

The community-provided plugins are an integral part of the QGIS ecosystem and QGIS isn't a dev tool where we can expect the users to craft a dedicated shell.nix for every separate project, so having the variant with these packages in NixPkgs seems prudent. I do like your idea of using your extraPythonPackages mechanism for that, but I think the qgis attribute should then point to the variant _with_ those "default extra packages", e.g.:

{
  #…
  qgisMinimal = …;
  qgis = qgisMinimal.override { extraPythonPackages = (ps: [ ps.… ]); };
  #…
}

Unless there's a strong reason not to go that way, of course.

(I'm not quite happy with the name qgisMinimal, but it seems at least better than the qgisWithoutDeps I first thought of. Any suggestions for alternatives?)

[lsix]

qgisLight could be what you are looking for.

I have absolutely no reason to keep qgis as it is now with little dependencies. It is enough for my needs (or if not, I just use the extraPythonPackages), but I admin I do not rely on the plugins from QGIS.

I’ll make the modification probably somewhere next week (I’ll be mostly away from my computer for the long week end to come).

[das-g]

Cool, no hurry.

I may give it a try myself meanwhile.

[das-g]

I dunno yet whether what corresponds to

• python3-pyqt5.qsci
• python3-pyqt5.qtsql
• python3-pyqt5.qtsvg
• python3-pyqt5.qtwebkit

is already included in python3Packages.pyqt5.

Python modules PyQt5.Qsci, PyQt5.QtSql and PyQt5.QtSvg are already importable in QGIS and seem to be provided by /nix/store/8jkwfdn6hyw7sv464n1a6y5ddzjvzl15-python3.7-qscintilla-2.9.4.

I don't know yet whether Python module PyQt5.QtWebKit is available in NixPkgs at all.

[das-g]

I'm not quite sure how reliable my approach of looking at the dependencies of python3-qgis is for finding out what Python modules officially (for whatever value of "official") should be available to QGIS plugins. Thus, I figured I'd just ask on the QGIS-Developer mailing list:

mailing list message (click to expand)

On 23.04.19 13:51, Raphael Das Gupta wrote:

Heya all

What Python modules (beyond the "qgis" module provided by QGIS itself and the ones built into the Python language or included in its standard library) can QGIS plugins expect to be accessible?

Or asked the other way around: When packaging QGIS (e.g. for a Linux distribution), what Python modules do I have to make available in its Python, so that plugins may use it?

While one could probably determine this by looking at existing packaging efforts, e.g. the dependencies of Ubuntu package "python3-qgis" from https://qgis.org/ubuntu-ltr, I wonder whether this isn't already documented somewhere, as this should be of interest for both plugin authors and application packagers.

My context: I'm trying to improve the QGIS experience on NixOS and other NixPkgs-based installations and would like to do that "right" rather than basing my changes on guesswork. I've collected my current findings on https://github.com/NixOS/nixpkgs/issues/59842

Kind regards,
Raphael

[das-g]

Sources mentioned so far in that mailing list thread:

• dependencies of python3-qgis https://github.com/qgis/QGIS/blob/master/debian/control.in#L370
• setup.hint files of OSGeo4W's qgis-full: http://download.osgeo.org/osgeo4w/x86_64/release/qgis/
• packaged Python 3 modules of lutraconsulting's Mac OS installer: https://lutraconsulting.github.io/qgis-mac-packager/

Even taking into account that some of the latter might be build-only dependencies and that packaging granularity might be different, there seems to be quite some discrepancy:

| qgis Build-Depends: | python3-qgis Depends: | OSGeo4W's qgis-full/setup.hint | "Used Python3 modules" of lutraconsulting's MacOS packages of QGIS |
|-------------------------|--------------------------------|-------------------------|------------------------|
| python3-all-dev | | | |
| | | | asn1crypto 0.24.0 |
| | | | attr 19.1.0 |
| | | | attrs 19.1.0 |
| | | | cairo |
| | | | certifi 2019.3.9 |
| | | | cffi 1.12.2 |
| | | | chardet 3.0.4 |
| | | | coverage 4.5.3 |
| | | | cryptography 2.6.1 |
| | | | Cython 0.29.6 |
| | python3-dateutil | python3-python-dateutil | dateutil 2.8.0 |
| | | | decorator 4.4.0 |
| | | | descartes 1.1.0 |
| python3-dev | | | |
| | | | ExifRead 2.1.2 |
| | | python3-exifread | exifread 2.1.2 |
| | | | funcsigs 1.0.2 |
| python3-future | python3-future | python3-future | future 0.17.1 |
| python3-gdal | python3-gdal¹ | | GDAL 2.4.0 |
| | | | geos |
| | | | gi 3.32.0 |
| | | | git 2.1.11 |
| | | | gitdb 2.0.5 |
| | | | gitdb2 2.0.5 |
| | | | GitPython 2.1.11 |
| | | | h5py 2.9.0 |
| | python3-httplib2 | python3-httplib2 | httplib2 0.12.1 |
| | | | idna 2.8 |
| | | | ipython_genutils 0.2.0 |
| | | | Jinja2 2.10 |
| | python3-jinja2 | python3-jinja2 | jinja2 2.10 |
| | | | jsonschema 3.0.1 |
| | | | jupyter_core 4.4.0 |
| | | | libfuturize |
| | | | libpasteurize |
| | | | lxml 4.3.3 |
| | | | MarkupSafe 1.1.1 |
| | python3-markupsafe | python3-markupsafe | markupsafe 1.1.1 |
| | python3-matplotlib¹ | python3-matplotlib | matplotlib 3.0.3 |
| python3-mock | | python3-mock | mock 2.0.0 |
| | | | mpl_toolkits |
| | | | nbformat 4.4.0 |
| | | python3-networkx | networkx 2.2 |
| python3-nose2 | | python3-nose2 | nose2 0.9.0 |
| | | | numpy 1.16.2 |
| | | | oauthlib 3.0.1 |
| | | | OpenSSL 19.0.0 |
| | | | osgeo 2.4.1 |
| | | | OWSLib 0.17.1 |
| python3-owslib | python3-owslib | | owslib 0.17.1 |
| | | | palettable 3.1.1 |
| | | | pandas 0.24.2 |
| | | | past 0.17.1 |
| | | | pbr 5.1.3 |
| | | python3-pillow | |
| | | python3-pip | pip 19.0.3 |
| | | | pkg_resources |
| | python3-plotly | | plotly 3.7.1 |
| | | | _plotly_utils |
| | | | plotlywidget |
| | | | ply 3.11 |
| python3-psycopg2 | python3-psycopg2 | | psycopg2 2.7.7 |
| | | | pubsub 4.0.3 |
| | | | pycairo 1.18.0 |
| | | | pycparser 2.19 |
| | | | Pygments 2.3.1 |
| | python3-pygments | python3-pygments | pygments 2.3.1 |
| | | | pygtkcompat |
| | | python3-pyodbc | pyodbc 4.0.26 |
| | | | pyOpenSSL 19.0.0 |
| | | python3-pyparsing | |
| | | python3-pypiwin32 | |
| | python3-pyproj | | pyproj 2.1.2 |
| | | | Pypubsub 4.0.3 |
| python3-pyqt5 | python3-pyqt5 | | PyQt5 |
| python3-pyqt5.qsci | python3-pyqt5.qsci | | |
| python3-pyqt5.qtsql | python3-pyqt5.qtsql | | |
| python3-pyqt5.qtsvg | python3-pyqt5.qtsvg | | |
| | python3-pyqt5.qtwebkit | | |
| | python3-requests | | |
| | | | pyrsistent 0.14.11 |
| | | | pysal 2.0.0 |
| | | | python_dateutil 2.8.0 |
| | | | pytils 0.3 |
| | | | pytz 2018.9 |
| | | | pyximport |
| | | | PyYAML 5.1 |
| | | | requests 2.21.0 |
| | | | retrying 1.3.3 |
| | | python3-scipy | scipy 1.2.1 |
| | | | script |
| | | | seaborn 0.9.0 |
| | | | setuptools 40.8.0 |
| | | | Shapely 1.6.4.post2 |
| | | python3-shapely | shapely 1.6.4.post2 |
| | | python3-simplejson | simplejson 3.16.0 |
| python3-sip-dev | | | |
| python3-sip | python3-sip | | |
| | python3-six | | six 1.12.0 |
| | | | smmap 2.0.5 |
| | | | smmap2 2.0.5 |
| python3-termcolor | | | termcolor 1.1.0 |
| | | | test 2.3.4.5 |
| | | | tools 0.1.9 |
| | | | traitlets 4.3.2 |
| | python3-tz | python3-pytz | |
| | | | urllib3 1.24.1 |
| | | | wheel 0.33.1 |
| | | | xlrd 1.2.0 |
| | | | xlwt 1.3.0 |
| python3-yaml | python3-yaml | | yaml 5.1 |
| | | python3-xlrd | |
| | | python3-xlwt | |

---

¹ through python3-qgis-common

Maybe the list for the Mac OS installer includes transitive dependencies, too?

[das-g]

I don't know yet whether Python module PyQt5.QtWebKit is available in NixPkgs at all.

Looks like pyqt5_with_webkit would provide that, but I'm not sure how to use that, yet.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[das-g]

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me".

still important to me :hugs: