Pyinstaller: Best way to package app which uses pkg_resources.iter_entry_points

[tardyp]

My package is using pkg_resources.iter_entry_points to import its plugins.

I would like to package it into a single executable using pyinstaller.

I am looking for the best way to do it.

The code which imports the list of "plugins" is:

from pkg_resources import iter_entry_points

all_commands = {}
for ep in iter_entry_points("myapp.subcmds"):
  cmd = ep.load()()
  cmdname = ep.name
  cmd.NAME = cmdname
  all_commands[cmdname] = cmd

Which is 'standard' python way of distributing plugin-able apps.
It looks like pyinstaller do not populate the entry_points.txt in the egg, and iter_entry_points is not able to return anything after packing by pyinstaller.

I have only be able to find this info in the wiki:
https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Setuptools-Entry-Point

Looks like the only way is to add a hook to patch iter_entry_points and hardcode the entrypoints in the hook.
I'd like to know if this is the best way to implement that.

[tardyp]

I have good results with the following .spec:

Again not sure if this is the recommended best method.

# -*- mode: python -*-

block_cipher = None

from pkg_resources import iter_entry_points
import textwrap
all_commands = []
hiddenimports = []
for ep in iter_entry_points("xxx.subcmds"):
  all_commands.append("{} = {}:{}".format(ep.name, ep.module_name, ep.attrs[0]))
  hiddenimports.append(ep.module_name)

with open("hook.py", "w") as f:
  f.write(textwrap.dedent("""
  import pkg_resources
  eps = {}
  def iter_entry_points(group, name=None):
    for ep in eps:
      yield pkg_resources.EntryPoint.parse(ep)
  pkg_resources.iter_entry_points = iter_entry_points
  print("hooked!")
  """.format(all_commands)))

a = Analysis(['xxx.py'],
             pathex=[],
             binaries=[],
             datas=[],
             hiddenimports=hiddenimports,
             hookspath=[],
             runtime_hooks=["hook.py"],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='reposyncer',
          debug=False,
          strip=False,
          upx=False,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=False,
               name='xxx')

[htgoebel]

PyInstaller currently does not make the package meta-data (like entry points) available in the frozen app. Thus if your approach works for you, this is okay.

[Toilal]

If you application fails with a NoneType error on entry points, you may have to add a fake Distribution object on the parsed entry point.

  def iter_entry_points(group, name=None):
    for ep in eps:
      parsedEp = pkg_resources.EntryPoint.parse(ep)
      parsedEp.dist = pkg_resources.Distribution() # This can avoid some 'NoneType' errors.
      yield parsedEp

[Toilal]

For reference, here's a better spec file that should support any number of packages. (You may add generated/ to .gitignore)

# -*- mode: python -*-

block_cipher = None

import pkg_resources
import os

hook_ep_packages = dict()
hiddenimports = set()

# List of packages that should have there Distutils entrypoints included.
ep_packages = ["fs.opener"]

if ep_packages:
    for ep_package in ep_packages:
        for ep in pkg_resources.iter_entry_points(ep_package):
            if ep_package in hook_ep_packages:
                package_entry_point = hook_ep_packages[ep_package]
            else:
                package_entry_point = []
                hook_ep_packages[ep_package] = package_entry_point
            package_entry_point.append("{} = {}:{}".format(ep.name, ep.module_name, ep.attrs[0]))
            hiddenimports.add(ep.module_name)

    try:
        os.mkdir('./generated')
    except FileExistsError:
        pass

    with open("./generated/pkg_resources_hook.py", "w") as f:
        f.write("""# Runtime hook generated from spec file to support pkg_resources entrypoints.
ep_packages = {}

if ep_packages:
    import pkg_resources
    default_iter_entry_points = pkg_resources.iter_entry_points

    def hook_iter_entry_points(group, name=None):
        if group in ep_packages and ep_packages[group]:
            eps = ep_packages[group]
            for ep in eps:
                parsedEp = pkg_resources.EntryPoint.parse(ep)
                parsedEp.dist = pkg_resources.Distribution()
                yield parsedEp
        else:
            return default_iter_entry_points(group, name)

    pkg_resources.iter_entry_points = hook_iter_entry_points
""".format(hook_ep_packages))

a = Analysis(['bonobo_oison_v1v2/__main__.py'],
             pathex=['/c/devel/projects/AFB-oison/migration'],
             binaries=[],
             datas=[],
             hiddenimports=list(hiddenimports),
             hookspath=[],
             runtime_hooks=["./generated/pkg_resources_hook.py"],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
          cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='bonobo_oison_v1v2',
          debug=False,
          strip=False,
          upx=False,
          console=True)
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=False,
               name='bonobo_oison_v1v2')