Nuitka: Lack of support for importlib_resources

[coldfix]

Hey,

While evaluating nuitka for a somewhat larger problem of mine, I hit a problem due to using importlib_resources. The problem seems to be that nuitka doesn't set module.__spec__.module_search_locations for packages to a list or string as requested by PEP451.

Consider the minimal example bar.py:

# bar.py
import importlib_resources

% python -m nuitka --follow-imports bar.py            
Nuitka:WARNING:/home/thomas/temp/gna/lib/python3.5/site-packages/importlib_resources/_py2.py:9: Cannot find 'pathlib2' in package 'importlib_resources' as absolute import (tried pathlib2).

% ./bar.bin                                            
ModuleSpec(name='importlib_resources', loader=<class '_nuitka_compiled_modules_loader'>)
Traceback (most recent call last):
  File "/home/thomas/temp/bar.py", line 1, in <module>
    import importlib_resources
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/home/thomas/temp/gna/lib/python3.5/site-packages/importlib_resources/__init__.py", line 36, in <module importlib_resources>
    __version__ = read_text('importlib_resources', 'version.txt').strip()
  File "/home/thomas/temp/gna/lib/python3.5/site-packages/importlib_resources/_py3.py", line 158, in read_text
    package = _get_package(package)
  File "/home/thomas/temp/gna/lib/python3.5/site-packages/importlib_resources/_py3.py", line 41, in _get_package
    raise TypeError('{!r} is not a package'.format(package))
TypeError: 'importlib_resources' is not a package

Checking the corresponding line, this error is raised due to module.__spec__.module_search_locations is None.

The problem is not restricted to importlib_resources but also appears when loading resources via importlib.resources on newer python. Grepping for module_search_locations, I can see that notably six assigns this as empty list in some locations, and refers to PEP451 which in turn says that this attribute must be set to a list or string whenever the object is a package.

I tried this in a virtualenv with both py3.5 and py3.7 with the stable version 0.6.0.6 from PyPI as well as 0.6.1rc9 from your download page:

% python -m nuitka --version 
0.6.0.6
Python: 3.5.5 (default, Sep  6 2018, 12:31:21) 
Executable: /home/thomas/temp/gna/bin/python
OS: Linux
Arch: x86_64

Best, Thomas

PS: thanks for all your work on this wonderful project:)

[kayhayen]

Seems this should be easy, I will look into this and see if I can include it for 0.6.1 still.

Nuitka starting having these __spec__ values a while ago, but apparently doesn't create them fully compatible. Seems it would be enough to assign the __path__ value to it.

[kayhayen]

So Nuitka uses ModuleSpec, but doesn't provide is_package, which is a keyword only argument and therefore a bit clumsy to do from C code. However, I made it so that I just set it to empty list after the fact, which is precisely what providing is_package would do.

I am not sure how aliases to __path__ the contents ought to be, it probably should. For that a test case would be nice, that outputs the values and checks things with is operator if its even the same list, which from the looks of it, won't be true though.

So my guess is these will falsely look like packages with no module search path, although __path__ will contain some and imports would actually work.

[kayhayen]

Factory instructions http://nuitka.net/doc/factory.html it would be nice to know if this improves anything for you. The package check itself should now work.

[coldfix]

Many thanks. As you say, the package check works now, however package resources still does not. On import importlib_resources (which does __version__ = read_text('importlib_resources', 'version.txt').strip(), I get now:

Traceback (most recent call last):
  File "/home/thomas/hit/dev/madgui/nuitka/foo.py", line 1, in <module>
    import importlib_resources
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/home/thomas/.local/share/virtualenvs/madgui/lib/python3.7/site-packages/importlib_resources-1.0.2-py3.7.egg/importlib_resources/__init__.py", line 36, in <module importlib_resources>
    __version__ = read_text('importlib_resources', 'version.txt').strip()
  File "/usr/lib/python3.7/importlib/resources.py", line 169, in read_text
    with open_text(package, resource, encoding, errors) as fp:
  File "/usr/lib/python3.7/importlib/resources.py", line 126, in open_text
    _check_location(package)
  File "/usr/lib/python3.7/importlib/resources.py", line 82, in _check_location
    raise FileNotFoundError(f'Package has no location {package!r}')
FileNotFoundError: Package has no location <module 'importlib_resources' from '/home/thomas/.local/share/virtualenvs/madgui/lib/python3.7/site-packages/importlib_resources-1.0.2-py3.7.egg/importlib_resources/__init__.py'>

This error is caused in the function _check_location in importlib.resources:

def _check_location(package):
    if package.__spec__.origin is None or not package.__spec__.has_location:
        raise FileNotFoundError(f'Package has no location {package!r}')

I changed importlib_resources to set origin = '' and has_location = True before reading the version. Now:

Traceback (most recent call last):
  File "/usr/lib/python3.7/importlib/resources.py", line 131, in open_text
    return open(full_path, mode='r', encoding=encoding, errors=errors)
FileNotFoundError: [Errno 2] No such file or directory: '/home/thomas/hit/dev/madgui/version.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/thomas/hit/dev/madgui/nuitka/foo.py", line 1, in <module>
    import importlib_resources
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/home/thomas/.local/share/virtualenvs/madgui/lib/python3.7/site-packages/importlib_resources-1.0.2-py3.7.egg/importlib_resources/__init__.py", line 41, in <module importlib_resources>
    __version__ = read_text('importlib_resources', 'version.txt').strip()
  File "/usr/lib/python3.7/importlib/resources.py", line 169, in read_text
    with open_text(package, resource, encoding, errors) as fp:
  File "/usr/lib/python3.7/importlib/resources.py", line 145, in open_text
    raise FileNotFoundError(message)
FileNotFoundError: 'version.txt' resource not found in 'importlib_resources'

This results from the following section in importlib.resources.open_binary:

        loader = cast(ResourceLoader, package.__spec__.loader)
        data = None
        if hasattr(package.__spec__.loader, 'get_data'):
            with suppress(OSError):
                data = loader.get_data(full_path)
        if data is None:
            package_name = package.__spec__.name
            message = '{!r} resource not found in {!r}'.format(
                resource, package_name)
            raise FileNotFoundError(message)

which seems to indicate that resource loader/resources are not implemented yet?

Best, Thomas

[kayhayen]

I think it is rather that get_data in implemented in the loader, but the __spec__ will not have the loader value set, but that is definitely something that might change easily. I will add that and let you know. It could also mean, that __path__ being incorrect has no impact, as it won't be used.

[coldfix]

So does nuitka somehow include package data files into the executable/shared objects/or some other directory/archive?

[kayhayen]

The first part is released now, I didn't deal with the other issue. The __spec__ not having the loader available is something I still need to fix, or maybe not. I could all be only a minor addition to the data files plugin.

I am a bit confused about your report. Can you give me a clean reproducer of the remaining issue, so I do not have do guess.

[coldfix]

Hey, sorry for being unclear.

I am still using the same minimal reproducer:

# bar.py:
import importlib_resources

% python -m nuitka --follow-imports bar.py            
% ./bar.bin                                            
[...]
FileNotFoundError: Package has no location <module 'importlib_resources' from '/home/thomas/.virtualenvs/nuitka/lib/python3.7/site-packages/importlib_resources/__init_│··
_.py'>

obtained with the most recent commit from the factory repository and branch.

The error is due to failing to read the version from a package data file. If nuitka does already include data files it could be a small addition.

Best, Thomas

[kayhayen]

I did have another error, with it accessing "__spec__.origin" as a pathname. I enhanced things to assign that from "__file__" during non-main module init. With that the crash is gone.

Loaders are implemented, there is get_data as a static method in the meta path based loader of Nuitka, but it appears you got another error somehow. Lets try this again. please update your factory branch install of nuitka and retry, ought to work now.

However, I am confused as to what changed in the first place, so it started crashing for me. Maybe the meta path based loader changes of recent made it use different code paths now. The importlib_resources and importlib in general are very much into guessing stuff.

Yours,
Kay

[coldfix]

Great! It now works like a charm!

[coldfix]

I'm sorry for the noise, it's not quite as complete as I initially thought..

The example above still fails on 3.7 with FileNotFoundError: Package has no location <module 'importlib_resources' from '/home/thomas/.virtualenvs/nuitka/lib/python3.7/site-packages/importlib_resources/__init_│·· _.py'> as written above.

On py3.4 - 3.6, the behaviour is as follows

python -m nuitka --follow-imports bar.py

# works fine at first
./bar.bin

# now uninstall importlib_resources, (or simply enter an environment without the package):
pip uninstall importlib_resources

./bar.bin
Traceback (most recent call last):
  File "/home/thomas/.virtualenvs/nuitka34/lib/python3.4/site-packages/importlib_resources-1.0.2-py3.4.egg/importlib_resources/_py3.py", line 120, in open_text
FileNotFoundError: [Errno 2] No such file or directory: '/home/thomas/.virtualenvs/nuitka34/lib/python3.4/site-packages/importlib_resources-1.0.2-py3.4.egg/importlib_resources/version.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/thomas/hit/dev/madgui/nuitka/src/bar.py", line 1, in <module>
    import foo
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "/home/thomas/.cache/Nuitka/egg-content/e44368ce805d0db87d22b7262fa5f8e4/foo/__init__.py", line 1, in <module foo>
    from importlib_resources import read_text
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "/home/thomas/.virtualenvs/nuitka34/lib/python3.4/site-packages/importlib_resources-1.0.2-py3.4.egg/importlib_resources/__init__.py", line 36, in <module importlib_resources>
  File "/home/thomas/.virtualenvs/nuitka34/lib/python3.4/site-packages/importlib_resources-1.0.2-py3.4.egg/importlib_resources/_py3.py", line 158, in read_text
  File "/home/thomas/.virtualenvs/nuitka34/lib/python3.4/site-packages/importlib_resources-1.0.2-py3.4.egg/importlib_resources/_py3.py", line 134, in open_text
FileNotFoundError: 'version.txt' resource not found in 'importlib_resources'

Note that the error is somewhat different than on py3.7.

So this almost looks like on py3.4-py3.6 the version.txt resource is not included into the bar.bin but looked up from the installed package at runtime (which wouldn't be so good for deployment purposes).

However this cannot be the full truth, as I was first testing the behaviour with a small dummy package that contained a single data file and prints its contents when imported. The confusing part is that in this case, the bar.bin binary worked even after fully removing the dummy package from the environment (but not anymore after removing importlib_resources).

[kayhayen]

I only tried it with python3.6 which is what you originally reported. From what you said, the python3.7 implementation is not a backport, and may therefore be radically different.

For accelerated mode, I am assuming it's fine now, but standalone indeed might require the data file, but this will be relatively easy to add. Standalone changes where the package thinks it lives, not in the original install, but where the binary lives in a package directory of course.

I will do it later. And from what I gather, you are saying with 3.7 the data file is not needed, is that correct?

[coldfix]

I believe on 3.7 importlib_resources is simple stub that defers to importlib.resources which in turn should be about the same implementation as backported importlib_resources on older versions.

I think on 3.7 the data file might not be needed since its always available as stdlib package (at least if you don't use standalone, I didn't try with that so far).

[kayhayen]

So for standalone, I added the "version.txt" to the data file plugin, so it's now automatically copied along to where it's needed.

However, python3.7 still gives me an error.

  File "/opt/python371/lib/python3.7/site-packages/importlib_resources/__init__.py", line 36, in <module importlib_resources>
    __version__ = read_text('importlib_resources', 'version.txt').strip()
  File "/opt/python371/lib/python3.7/importlib/resources.py", line 169, in read_text
    with open_text(package, resource, encoding, errors) as fp:
  File "/opt/python371/lib/python3.7/importlib/resources.py", line 126, in open_text
    _check_location(package)
  File "/opt/python371/lib/python3.7/importlib/resources.py", line 82, in _check_location
    raise FileNotFoundError(f'Package has no location {package!r}')
FileNotFoundError: Package has no location <module 'importlib_resources' from '/opt/python371/lib/python3.7/site-packages/importlib_resources/__init__.py'>

Not sure what it is trying to do that fails there, I will have to check that. But before Python 3.7 should work in all modes now, where as Python 3.7 doesn't work at all yet.

[kayhayen]

The code that handles python3.6 and earlier is on factory again.

[kayhayen]

There seems to be another attribute "has_location" to specs that is being checked for 3.7 and I need to find out what that is all about.

[kayhayen]

I pushed a fix for "has_location" to also be set. It seems to indicate that "origin" is really a path, but only 3.7 seems to actually check that property before using it that way.

The spec attribute also existed since 3.4, but obviously was never observed to have an effect so far, but these versions also changed that way now. I again pushed to factory.

This seems innocent enough to be part of the next release that I am constantly delaying for yet another fix.

[coldfix]

Keep up the great work!

So for standalone, I added the "version.txt" to the data file plugin, so it's now automatically copied along to where it's needed.

Sorry for my confusion, I somehow assumed that the default mode also includes data files for 3rdparty packages.

This at first seemed to be confirmed by the fact that I could create a test package foo with a data file in it and then run nuitka in default mode on a script import foo; print(importlib_resources.read_text('foo', 'data.txt')) that simply shows the data file. The resulting binary works just fine, even after uninstalling foo and removing the data file from disc - and only breaks on uninstalling importlib_resources.

Now I see that this appeared to work due to my package being installed as zip and nuitka therefore extracting the file somewhere in /home/thomas/.cache/Nuitka/egg-content/462fa8db6d8ae60a18d70bc7e87b6006/foo/data.txt, not due to including it in the binary. Furthermore the difference to importlib_resources is that this is not installed as zip. I will be sure to use standalone from now:)

Best and thanks a lot, Thomas

[kayhayen]

Ah @coldfix this .egg support is something added a few releases ago, and I rarely observed them. Good to see you benefit there.

The difference for standalone and accelerated mode is that in the later, the original location of the python source is used, whereas in standalone mode, a path relative to the binary is used.

And thanks for your support. I am very probably releasing over the weekend unless I find issues with the current factory branch on my buildbot farm before that, then this should be in stable and improve the compatibility of the __spec__ mechanism.

As for data files, I wish these ones only used to determine one thing from them were inlined, but that is asking too much. I might eventually make a plugin where it warns about non-python files not included for a package. But generally these are relatively rare and few, that are really needed, and I can hard code those exceptions once uncovered, otherwise it will probably become warning noise only.

[kayhayen]

Just released 0.6.1 with it.