Pyinstaller: Jinja2 NotImplementedError and pkg_resources support

[giumas]

What is the current status on Jinja2 support?

It looks like there is an hook file, but if I try to freeze an application that uses Jinja2 (using pyinstaller 3.1.1 from PyPi) I get at runtime:

 Traceback (most recent call last):
  [...]
  File "xxxxx\yyyy\generator\c\generator.py", line 180, in _init_files
    tmp = self.j2env.get_template('file_banner.j2')
  File "site-packages\jinja2\environment.py", line 812, in get_template
  File "site-packages\jinja2\environment.py", line 786, in _load_template
  File "site-packages\jinja2\loaders.py", line 113, in load
  File "site-packages\jinja2\loaders.py", line 234, in get_source
  File "site-packages\pkg_resources\__init__.py", line 1440, in has_resource
  File "site-packages\pkg_resources\__init__.py", line 1495, in _has
NotImplementedError: Can't perform this operation for unregistered loader type

This seems related to package_resources. From the FAQ:

pkg_resources is currently not supported by PyInstaller. This means that an application using a library which uses the the pkg_resources API will probably not work out of the box. The only situation in which it works is when it's being used on .egg files (see above). For details follow issue #183.

Well, that ticket is now closed. Does it mean that the issues with pkg_resources are somehow solved in development?

Possible related issues:

• https://github.com/SavinaRoja/PyUserInput/issues/29
• http://stackoverflow.com/questions/35811448/pyinstaller-jinja2-templatenotfound

[giumas]

UPDATE: A workaround is to avoid to hit the pkg_resources in Jinja2. So, after having substituted the get_template with from_string (and using the native load to read the template), it works!

[bjones1]

Yes, there's some support for pkg_resources via the copy_metadata function. See https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-logilab.py and https://htmlpreview.github.io/?https://github.com/pyinstaller/pyinstaller/blob/develop/doc/Manual.html#useful-items-in-pyinstaller-utils-hooks.

I don't understand your workaround -- did you modify the Jinja2 source, or the hook?

[giumas]

@bjones1 : I shamefully modified my app! Wherever I was calling the Jinja2.environment's method get_template('whatever.template') I now use from_string() (after having read the template file content using standard open('whatever.template') and read()).

In other words, Jinja2 uses pkg_resources to open a template file when get_template is called. It does not when the string content is directly passed with the from_string method.

So my workaround does not solve the general pyinstaller issue described in my ticket! However, I wrote it down since somebody can stop by this ticket googling around a similar problem.

[bjones1]

Without looking at Jinja2 at all, I wonder if adding the following to hook-jinja2.py would solve the problem generally. Would you mind testing?

from PyInstaller.utils.hooks import copy_metadata

datas = copy_metadata('Jinja2')

[giumas]

It does not work. Same exception. I have attempted both datas = copy_metadata('Jinja2') and datas = copy_metadata('jinja2').

[bjones1]

Thanks for looking. Would you post your code? I'd like to see if there are any easy fixes, or at least create some simple unit tests.

[giumas]

This is a minimal setup to replicate the error with a test package:

[test]
  | --> __init__.py
  | --> [templates]
             | --> test.j2 (empty file)

The content of __init__.py:

from jinja2 import Environment, PackageLoader
j2env = Environment(loader=PackageLoader('test', 'templates'))
j2env.get_template('test.j2')
print("done")

To replicate the error:

• create a jinja2_minimal.pyscript that just import the package
• execute pyinstaller jinja2_minimal.py

You should get something similar to this traceback when you try to execute the frozen app:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\users\xxxxxxx\appdata\local\temp\pip-build-gtjatb\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
  File "test\__init__.py", line 3, in <module>
    j2env.get_template('test.j2')
  File "site-packages\jinja2\environment.py", line 812, in get_template
  File "site-packages\jinja2\environment.py", line 786, in _load_template
  File "site-packages\jinja2\loaders.py", line 113, in load
  File "site-packages\jinja2\loaders.py", line 234, in get_source
  File "site-packages\pkg_resources\__init__.py", line 1440, in has_resource
  File "site-packages\pkg_resources\__init__.py", line 1495, in _has
NotImplementedError: Can't perform this operation for unregistered loader type
jinja2_minimal returned -1

[giumas]

And this is my workaround:

• modified __init__.py:

from jinja2 import Environment, PackageLoader
import os
j2env = Environment(loader=PackageLoader('test', 'templates'))
# j2env.get_template('test.j2')
j2env.from_string(open(os.path.join(os.path.dirname(__file__), 'templates', 'test.j2')).read())
print("done")

• modified jinja2_minimal.spec:

# -*- mode: python -*-
from PyInstaller.building.build_main import Analysis, PYZ, EXE, COLLECT, BUNDLE, TOC
from PyInstaller import is_darwin


def collect_pkg_data(package, include_py_files=False, subdir=None):
    import os
    from PyInstaller.utils.hooks import get_package_paths, remove_prefix, PY_IGNORE_EXTENSIONS

    # Accept only strings as packages.
    if type(package) is not str:
        raise ValueError

    pkg_base, pkg_dir = get_package_paths(package)
    if subdir:
        pkg_dir = os.path.join(pkg_dir, subdir)
    # Walk through all file in the given package, looking for data files.
    data_toc = TOC()
    for dir_path, dir_names, files in os.walk(pkg_dir):
        for f in files:
            extension = os.path.splitext(f)[1]
            if include_py_files or (extension not in PY_IGNORE_EXTENSIONS):
                source_file = os.path.join(dir_path, f)
                dest_folder = remove_prefix(dir_path, os.path.dirname(pkg_base) + os.sep)
                dest_file = os.path.join(dest_folder, f)
                data_toc.append((dest_file, source_file, 'DATA'))

    return data_toc

pkg_data = collect_pkg_data('test')
print(pkg_data)

block_cipher = None


a = Analysis(['jinja2_minimal.py'],
             pathex=['D:\\test_data\\jinja2'],
             binaries=None,
             datas=None,
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='jinja2_minimal',
          debug=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               pkg_data,
               strip=False,
               upx=True,
               name='jinja2_minimal')

[amouma]

Hit exactly the same problem. I'm trying to package a script that uses Bokeh, which in turn uses Jinja2. Any help appreciated.

[elypma]

Any success with bokeh?

[amouma]

Nope. I actually discarded Bokeh and went with matplotlib instead. this particular issue disappeared, but bumped into others. I'm exploring other methods to distribute my app. Pyinstaller seems to have too many issues.

[bjones1]

I can reproduce this bug. Some notes from my investigation: Jinja2 ends up calling provider = get_provider('test <or whatever the package containing jinja2 templates is called>'); has_resource('test.j2 <or whatever the template file is named'). This means it's relying a pkg_resources provider. PyInstaller only supports the NullProvider, based on pyi_rth_pkgres.py and a few additions in pyimod03_importers.py.

While editing pyi_rth_pkgres.py to use the DefaultProvider instead of the NullProvider works, it breaks egg-based tests (test_egg_zipped). A more complete fix would probably require writing a Provider for the FrozenImporter, which would probably take a fair amount of work.

However, a simple workaround is to exclude the package containing Jinja2 templates from PyInstaller, then use datas = collect_data_files('template_package', None, True) in a hook file (or manually copy the package). I've attached my working example (but used the manual copy process): jinja2.zip.

Here's a patch to add failing tests for has_resource:

diff --git a/tests/functional/scripts/pkg_resource_res_string.py b/tests/functional/scripts/pkg_resource_res_string.py
index c653d3e..54326cd 100644
--- a/tests/functional/scripts/pkg_resource_res_string.py
+++ b/tests/functional/scripts/pkg_resource_res_string.py
@@ -13,13 +13,22 @@ import pkg_resources as res
 import pkg3

 expected_data = 'This is data text for testing the packaging module data.'.encode('ascii')
+data_source = 'sample-data.txt'

 # With frozen app the resources is available in directory
 # os.path.join(sys._MEIPASS, 'pkg3/sample-data.txt')
-data = res.resource_string(pkg3.__name__, 'sample-data.txt')
+data = res.resource_string(pkg3.__name__, data_source)
 if data:
     data = data.strip()

 if data != expected_data:
     raise SystemExit('Error: Could not read data with pkg_resources module.')
 print('Okay: Resource data read.')
+
+# Test access through an `IResourceProvider <https://pythonhosted.org/setuptools/pkg_resources.html#iresourceprovider>`_,
+# which requires more methods to be implemented than those from the
+# `ResourceManager API <https://pythonhosted.org/setuptools/pkg_resources.html#resourcemanager-api>`_
+# used above.
+provider = res.get_provider(pkg3.__name__)
+assert provider.has_resource(data_source)
+assert provider.get_resource_string(res.ResourceManager, data_source).strip() == expected_data

[bjones1]

I'm closing this -- it the attached file doesn't work, please reopen it.

[jakko]

Bjones,

I am still getting the error while using jinja2. The problem is that I use Bokeh which is importing Jinja2 as a whole. I know many people are using bokeh but could only find this thread about a fix for the provider error.

How would you fix the error in case of using Bokeh? Where should I put the exclude statement and where to copy the jinja2 package? If it is possible at all.

Help would be appreciated!

[bjones1]

FWIW, the basic principle is to prevent Jinja2 templates from being place in pyinstaller's archive, instead running it from its source .py and data files. Since I'm unfamiliar with Bokeh, I can't provide much more help than that.

[severinsimmler]

@jakko, could you fix your bokeh issue?

[jakko]

@severinsimmler, Nope

[bryevdv]

This is pretty much the extent of Bokeh's use of Jinja:

from jinja2 import Environment, PackageLoader, Markup

_env = Environment(loader=PackageLoader('bokeh.core', '_templates'))
_env.filters['json'] = lambda obj: Markup(json.dumps(obj))

JS_RESOURCES = _env.get_template("js_resources.html")

# more of the same

Which seems fairly straightforward, but I'm not familiar at all with py2exe, so if there is some change we can make to work better with py2exe please advise.

[xinyuwei]

I also ran into the problem. Using FileSystemLoader instead of the PackageLoader seems resolved the problem.

Related answer here:
https://stackoverflow.com/questions/31259673/unable-to-include-jinja2-template-to-pyinstaller-distribution/45619374#45619374

[bryevdv]

That seems slightly unfortunate, or at least less clean than PackageLoader since it will require mucking with __file__, etc. explicitly. But if someone wants to

• verify that this actually solves the problem here, and
• submit at PR to Bokeh to make the change

I will commit to helping get it merged.

[runtel]

Try like this.

import sys
try:
    base_path = sys._MEIPASS
except:
    base_path = os.path.abspath(".")

templates_path = os.path.join(base_path, "docs_app", "templates")
# I'm use aiohttp)
# and replace PackageLoader to FileSystemLoader.
aiohttp_jinja2.setup(app, loader=jinja2.FileSystemLoader(templates_path))

spec file:

datas = [ 
('some_path/docs_app/static/', 'docs_app/static/'), 
('some_path/docs_app/templates/', 'docs_app/templates/')
]

FileSystemLoader not use ResourceManager from pkg_resources

[htgoebel]

You may want to read https://pyinstaller.readthedocs.io/en/latest/runtime-information.html

[DominicAntonacci]

@bryevdv, I can confirm that using https://stackoverflow.com/questions/31259673/unable-to-include-jinja2-template-to-pyinstaller-distribution/46628650#46628650 fixes the problem. I'll try to free up some time and get a PR out for bokeh.

There are some other missing file problems I ran into when I tried to start a bokeh server in Python, but got a successful workaround (only tested in Windows, but should work for others). Hopefully this helps others.

This setup will work for running a bokeh server in a multiprocessing Process, allowing one program to start a bokeh server process and data publishing thread inside a PyInstaller exe.

1. Modify bokeh/core/templates.py.
   Jinja doesn't play nicely, but this StackOverflow answer solved my problem: https://stackoverflow.com/questions/31259673/unable-to-include-jinja2-template-to-pyinstaller-distribution/46628650#46628650. I suspect @runtel's code also works, but I haven't tried it.

2. When calling bokeh serve in Python, I needed to mimic the command line usage

from bokeh.commands.subcommands.serve import Serve
import argparse
p = argparse.ArgumentParser()
s = Serve(p)
# Set up your command line arguments here by directly modifying args.
args = p.parse_args(args='')
args.show = True
args.files.append('your_bokeh_plot.py')
s.invoke(args)

1. Build using PyInstaller. I found I needed to add a hidden import and data file for my bokeh plotting code

pyinstaller run_full_system.py --hidden-import your_bokeh_plot.py --add-data your_bokeh_plot.py;.

1. Copy over all the bokeh files to the distribution directory. I found that the bokeh server was missing a bunch of js/css/html files. Rather than add a bunch of command line arguments, I found it easier just to copy it by hand. There's probably a better way to do this in PyInstaller.

(windows)
robocopy /path/to/bokeh/ /path/to/dist/run_full_system/bokeh /E

[bryevdv]

@DominicAntonacci happy to consider a PR, we should probably move Bokeh-specific discussion back to https://github.com/bokeh/bokeh/issues/799 (which we can re-open for a PR)

[jzybert]

@DominicAntonacci I'm in luck -- just encountered this bug at work while trying to create an executable using Bokeh. Could you post more of the code that you modified to get this to work? I attempted following your suggestions but for suggestion 1 I end up with a ModuleNotFoundError: No module named 'bokeh.core.templates' after making the changes and perhaps seeing run_full_system.py would also be helpful. Thanks.

[DominicAntonacci]

@jzybert I'll get a working minimal example posted in the next few days.

I haven't encountered that import error before, but it's possible bokeh isn't getting picked up by PyInstaller. Bokeh is only used in one file for me, which isn't imported by anything, so I've got to use the --hidden-import and --add-data pyinstaller flags to ensure that file is put in the proper place.

[DominicAntonacci]

@jzybert, I've put a working minimal example at https://github.com/DominicAntonacci/pyinstaller_bokeh_example. Hope that helps!

[axil]

@DominicAntonacci It works, but the _templates dir is expected to be in _MEIPASS. Do you know how to --add-data automatically?

[axil]

I've created a hook for bokeh which automates --add-data. It allows to write pyinstaller mybokehscript.py for a script which uses bokeh without any other command-line options.

In addition to bunding the _templates dir, it includes two more necessary dirs in a proper way.

With this hook @DominicAntonacci's minimal example works for me.

See my PR3607. It needs bokeh to be patched a bit: it's better to keep the _templates dir in its original locaiton, see my PR8038.

Before the hook is merged into the pyinstaller it can be used in a 'standalone' fashion: copy hook-bokeh.py file into the project dir root (say, where mybokehscript.py resides) and run

pyinstaller mybokehscript.py --additional-hooks-dir=.

[DominicAntonacci]

Looks like you have it all figured out, but I had successfully gotten bokeh to compile with a manual --add-data flag, passing in the whole bokeh install directory. I've only ever tried a directory build, so I don't know if it will all work out if you do a single file build.

If you have any other questions, I'll do my best, but I'm usually slow to respond on github.

[axil]

@DominicAntonacci Yes, it's pretty clear to me now. If you happen to freeze a bokeh project, try this approach, it's somewhat more straightforward, suggestions/fixes are welcome.

[axil]

@DominicAntonacci Single file build works fine for me: https://github.com/axil/frozen_sliders

[tobiz]

I encountered the original problem when using mplleaflet in my Github project OGN-Flogger. The run time trace back was pretty identical referring to jinja2 and pkg_resources. Has this issue been solved by the solutions for jinja2 or is there a specific solution needed for mplleaflet? Has a hook been created and included in the latest release of pyinstaller? BTW OGN-Flogger runs without issue when not built with pyinstaller (which I have to say is in general, an impressive piece of work)

[DominicAntonacci]

@tobiz I haven't been tracking this for multiple months, but I'll offer what I remember.
First, a file in Bokeh had to be modified to avoid some Jinja2 call. See https://stackoverflow.com/questions/31259673/unable-to-include-jinja2-template-to-pyinstaller-distribution/46628650#4662865 for more details. Then, a bokeh-specific Pyinstaller hook was created/updated (https://github.com/pyinstaller/pyinstaller/pull/3607).

I suspect you'll need to edit something in mplleaflet to avoid that function call and possibly need to make a hook for mplleaflet. My efforts getting Bokeh to work may be of use (https://github.com/DominicAntonacci/pyinstaller_bokeh_example), though others actually got merge requests in to fix everything.

[pythonwood]

gssapi same runtime error. (Ubuntu 18 + Python3.6 + PyInstaller 4.0 + pyfilesystem2)

fs.sshfs -> paramiko/ssh_gss.py -> gssapi/__init__.py ->gssapi/bindings/__init__.py

./a.bin
.......

  File "fs/sshfs/sshfs.py", line 14, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "pyvenv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "paramiko/__init__.py", line 22, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "pyvenv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "paramiko/transport.py", line 38, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "pyvenv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "paramiko/auth_handler.py", line 72, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "pyvenv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "paramiko/ssh_gss.py", line 49, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "pyvenv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "gssapi/__init__.py", line 4, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "pyvenv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "gssapi/bindings/__init__.py", line 230, in <module>
  File "gssapi/bindings/__init__.py", line 187, in _read_header
  File "pkg_resources/__init__.py", line 1133, in resource_exists
  File "pkg_resources/__init__.py", line 1404, in has_resource
  File "pkg_resources/__init__.py", line 1474, in _has
NotImplementedError: Can't perform this operation for unregistered loader type
[13996] Failed to execute script run

after uninstalling python-gssapi and build again, run it pass !

what ls in gssapi/bindings/:

autogenerated.cdef cffi_gssapi.cdef _cffi__x55c52808x13b739ff.cpython-36m-x86_64-linux-gnu.so __init__.py __pycache__