Python-socketio: Invalid async_mode specified

[yousiku]

hello!
Everything was OK when I run my python file, but it raised error run after be packed with pyinstaller:

Traceback (most recent call last):

  File "ems\core\task.py", line 67, in add
  File "ems\ems_socket_service.py", line 26, in __init__
  File "site-packages\socketio\server.py", line 72, in __init__
  File "site-packages\engineio\server.py", line 100, in __init__
ValueError: Invalid async_mode specified

the code :self.socketio = socketio.Server(async_mode='gevent')

I tried self.socketio = socketio.Server(), it's also useless, and i have installed gevent.

[miguelgrinberg]

Can you share the output of pip freeze?

Edit: Sorry, I just realize this worked fine without pyinstaller. Did you look at the spec file to see if there is anything odd in there? I'm not super experienced with pyinstaller, but I can take a look if you want.

The error that you are getting indicates that when the system tries to import the gevent handling code there was something missing, so check for missing dependencies.

[yousiku]

here is my spec file:

# -*- mode: python -*-

block_cipher = None


a = Analysis(['ems\\ems_station_service.py'],
             pathex=['ems'],
             binaries=None,
             datas=[
                ('ems/core/rule_parser/*.py', 'core/rule_parser/'),
                ('ems/data/ems.conf', 'data'),
                ('ems/data/pid', 'data/pid'),
                ('ems/data/images', 'data/images'),
             ],
             hiddenimports=[
                'netifaces',
                'gevent.ssl',
                'gevent.builtins',
                'crc16',
                'ctypes',
             ],
             hookspath=None,
             runtime_hooks=None,
             excludes=None,
             win_no_prefer_redirects=None,
             win_private_assemblies=None,
             cipher=block_cipher)

a.binaries + [('my_lib.dll', 'ems\\data\\lib', 'BINARY')]

import platform
if platform.system().find("Windows")>= 0:
    a.datas = [i for i in a.datas if i[0].find('Include') < 0]

pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)

exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='ems_station_service',
          debug=False,
          strip=None,
          icon='ems\\data\\images\\logo.ico',
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=None,
               upx=True,
               name='ems_station_service')

I have no idea about this problem...
I am honored that you are willing take a look.

[miguelgrinberg]

I think it is a fairly common problem that pyinstaller missing some dependencies when those dependencies are optional (i.e. not explicitly listed as dependencies for other projects).

Take a look at how to explicitly include packages as hidden imports: https://pythonhosted.org/PyInstaller/when-things-go-wrong.html#listing-hidden-imports. I think you should add gevent and gevent-websocket as hidden imports. Let me know if that works.

[yousiku]

It works!!!
I add 'engineio.async_gevent' to hiddenimports in spec file.
very, very grateful for your help!! lol hahahahaha

[Popkultur]

I have the same problem, howevere I am not using gevent and not uwsgi, since I am using Python 3. I am using eventlet. Did someone manage to solve this issue with eventlet?

[miguelgrinberg]

@Popkultur You need to ensure engineio.async_eventlet is included in the installer package.

[espretto]

@Popkultur Would you mind sharing your pip freeze, python 3.x version ? I am trying to bundle the wsgi-example - the latency test - which either complains about wrong async_mode or fails to import name greenio. I am using python 3.4.2 and win32com build 220 64bit on a clean Win7sp1 Here is my latency.spec:

# -*- mode: python -*-
result = Analysis(
  ['latency.py'],
  hiddenimports=['engineio.async_eventlet'],
  hookspath=None,
  runtime_hooks=None
  )

def Datafiles(*filenames, **kw):
    [...]

templates = Datafiles('templates/latency.html', strip_path=False)
static = Datafiles('static/style.css', strip_path=False)

pyz = PYZ(result.pure)
exe = EXE(
  pyz,
  result.scripts,
  result.binaries,
  result.zipfiles,
  result.datas,
  templates,
  static,
  name='latency.exe',
  debug=True,
  strip=None,
  upx=False,
  console=True
  )

Any help is greatly appreciated.

[miguelgrinberg]

@espretto did you add eventlet to your list of hidden imports?

[espretto]

Yes, I tried adding 'eventlet' to the above hiddenimports but to no avail. Here is what I tried so far for python versions 2.7.13, 3.3.5, 3.4.2 and 3.5.3 on Win7sp1:

$ mkvirtualenv latency
(latency)$ pip install eventlet python-socketio Flask pyinstaller
(latency)$ pyinstaller --onefile latency.spec
(latency)$ .\dist\latency.exe

which gives me this:

Traceback (most recent call last):
  File "latency.py", line 9, in <module>
    sio = socketio.Server(async_mode=async_mode)
  File "site-packages\socketio\server.py", line 82, in __init__
  File "site-packages\engineio\server.py", line 114, in __init__
ValueError: Invalid async_mode specified

installing pyinstaller apparently pulls in the correct version of pywin32 itself. python latency.py runs just fine in every case.

update
apparently engineio uses the importlib module to dynamically import whichever module corresponds to the chosen async_mode. hardcoding that part now leaves me at these lines:

Traceback (most recent call last):
  File "latency.py", line 9, in <module>
    sio = socketio.Server(async_mode=async_mode)
  File "site-packages\socketio\server.py", line 82, in __init__
  File "site-packages\engineio\server.py", line 103, in __init__
  File "c:\users\test\envs\latency-2.7.13\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
    exec(bytecode, module.__dict__)
  File "site-packages\engineio\async_eventlet.py", line 4, in <module>
  File "c:\users\test\envs\latency-2.7.13\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
    exec(bytecode, module.__dict__)
  File "site-packages\eventlet\__init__.py", line 10, in <module>
  File "c:\users\test\envs\latency-2.7.13\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
    exec(bytecode, module.__dict__)
  File "site-packages\eventlet\convenience.py", line 3, in <module>
ImportError: cannot import name greenio

greenio as in eventlet.greenio, no the standalone pip module with the same name.

[Popkultur]

This is my working SPEC:

# -*- mode: python -*-

block_cipher = None


added_files = [ ( 'db/*.*', 'db' ),
         ( 'templates/*.*', 'templates' ),
         ( 'RECORD', 'RECORD' ),
         ( 'uploads', 'uploads' )
        ]

a = Analysis(['main.py'],
             pathex=['C:\Kamopticon'],
             binaries=[],
             datas=added_files,
             hiddenimports=['engineio.async_eventlet'],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries + [('msvcp120.dll', 'C:\\Windows\\System32\\msvcp120.dll', 'BINARY'),
                        ('msvcr120.dll', 'C:\\Windows\\System32\\msvcr120.dll', 'BINARY')],
          a.zipfiles,
          a.datas,
          Tree('static/', 'static'),
          name='main',
          debug=False,
          strip=False,
          upx=True,
          console=True )

And I used it within https://github.com/Popkultur/Kamopticon

[miguelgrinberg]

I can't really help you with specifics, as I don't use pyinstaller, but the invalid async mode error means that the file async_eventlet.py is missing, so whatever you are doing in your configuration to include this file does not seem to be working.

The second import error on eventlet.greenio suggests eventlet is also not being imported as a complete package. It seems eventlet.convenience is there, but then with this file tries to import eventlet.greenio it founds that this module is missing. You need to find a way to get the whole thing included in the package.

[espretto]

Thank you very much. I managed to get past the import errors although some pyinstaller warnings persist and now I've run into some python34.dll error. It would seem my Windows registry got caught up installing multiple versions of python and win23com (=pypiwin32 I think) - should have taken a snapshot beforehand. I'll get back to you as soon as make any progress on this. Might be a valuable addition to the example for bootstrapping web-based desktop apps.

[miguelgrinberg]

@espretto If you have a spec file that works well with regards to the imports, it would be awesome if you can post it here, so that people looking for the same answers will find it. Thanks!

[espretto]

Got it! eventlet==0.20.1 doesn't play well with pyinstaller==3.2.1. Here goes my setup - detail-peppered because obviously things break easily:

what's to achieve

find a pyinstaller-spec for the wsgi example to bundle in a dep-less executable file.

pyinstaller --onefile latency.spec
.\dist\latency.exe

systeminfo | findstr "OS"

OS Name:                   Microsoft Windows 7 Professional 
OS Version:                6.1.7601 Service Pack 1 Build 7601
[...]

python

Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 20:20:57) [MSC v.1600 64 bit (AMD64)] on win32

pip freeze (note eventlet version is not the latest 0.20.1)

appdirs==1.4.2
click==6.7
enum-compat==0.0.2
eventlet==0.19.0
Flask==0.12
future==0.16.0
greenlet==0.4.12
itsdangerous==0.24
Jinja2==2.9.5
MarkupSafe==0.23
packaging==16.8
PyInstaller==3.2.1
pyparsing==2.1.10
pypiwin32==219
python-engineio==1.2.3
python-socketio==1.7.1
six==1.10.0
Werkzeug==0.11.15

pywin32

although pyinstaller depends on pypiwin32 (confirm by pip freeze) it may still not work because that does not register any .dll-files as is (maybe?) required by pyinstaller. to this end, download the version corresponding to the one of your venv's (mind 32/64bit modes) from their sourceforge mirror and install as admin (they won't ask but report at the end instead).

pyinstaller

the pyinstaller latency.spec file (put next to latency.py):

# -*- mode: python -*-

result = Analysis(
  ['latency.py'],
  hiddenimports=['engineio.async_eventlet'],
  hookspath=None,
  runtime_hooks=None
  )

# from pyinstaller wiki
def Datafiles(*filenames, **kw):
    import os

    def datafile(path, strip_path=True):
        parts = path.split('/')
        path = name = os.path.join(*parts)
        if strip_path:
            name = os.path.basename(path)

        return name, path, 'DATA'

    strip_path = kw.get('strip_path', True)
    return TOC(
        datafile(filename, strip_path=strip_path)
        for filename in filenames
        if os.path.isfile(filename)
        )

templates = Datafiles('templates/latency.html', strip_path=False)
static = Datafiles('static/style.css', strip_path=False)

pyz = PYZ(result.pure)

exe = EXE(
  pyz,
  result.scripts,
  result.binaries,
  result.zipfiles,
  result.datas,
  templates,
  static,
  name='latency.exe',
  debug=False,
  strip=None,
  upx=True,
  console=True
  )

conclusion

The resulting executable does work, even on a clean machine with none of the prerequisites installed. the only one thing missing is static files. Somehow they still slip through but this would be a question for their board I guess.

Another issue is that disconnecting throws an error. That however may very well be a shortcoming of the minimal example implementation that doesn't explicitely handle the on_close event of engineio. Here is the output on closing the browser tab:

(2500) accepted ('127.0.0.1', 49697)
(2500) accepted ('127.0.0.1', 49698)
(2500) accepted ('127.0.0.1', 49699)
(2500) accepted ('127.0.0.1', 49700)
127.0.0.1 - - [24/Feb/2017 13:15:41] "GET /socket.io/?EIO=3&transport=websocket&
sid=e6fa88f2dfa94fa29cb31085c883fff9 HTTP/1.1" 200 0 12.054688
Traceback (most recent call last):
  File "site-packages\eventlet\greenpool.py", line 82, in _spawn_n_impl
  File "site-packages\eventlet\wsgi.py", line 719, in process_request
  File "socketserver.py", line 675, in __init__
  File "site-packages\eventlet\wsgi.py", line 636, in finish
  File "site-packages\eventlet\greenio\base.py", line 474, in shutdown_safe
OSError: [WinError 10038] An operation was attempted on something that is not a
socket

Update:
The missing static files were very well included as indicated by pyinstaller's .toc report. It was jinja2 that wasn't aware of being run from within a package. To make it so one can swap jinja2's loader:

# [...]
app = Flask(__name__)
# [...]
import sys, jinja2, os.path as ospath

class MeipassLoader(jinja2.BaseLoader):

    def __init__(self, path):
        self.path = ospath.join(sys._MEIPASS, path)

    def get_source(self, environment, template):
        path = ospath.join(self.path, template)
        if not ospath.exists(path):
            raise jinja2.TemplateNotFound(template)
        mtime = ospath.getmtime(path)
        with open(path) as f:
            source = f.read()
        return source, path, lambda: mtime == ospath.getmtime(path)

if hasattr(sys, '_MEIPASS'):
    app.jinja_loader = jinja2.ChoiceLoader([
      app.jinja_loader,
      MeipassLoader('templates')
    ])

todo: greenify the above loader.

credits:

• https://shanetully.com/2013/08/cross-platform-deployment-of-python-applications-with-pyinstaller/
• https://buxty.com/b/2012/05/custom-template-folders-with-flask/
• http://stackoverflow.com/a/13963480/1675431

[miguelgrinberg]

@espretto awesome, thanks for the super fine details!

Another issue is that disconnecting throws an error

Yeah, that is fairly common. Eventlet on Linux/Mac is a bit better, on Windows several people already reported this error. It's benign though, it does not affect the server.

[crobertsbmw]

I couldn't get around all the errors with eventlet & PyInstaller, so I uninstalled eventlet, and am trying to use the async_mode="threading". I also put import threading in my main script file. When I try to run after pyinstaller--onefile socket_app.py, I get a Invalid async_mode specified error. This comes from \site-packages\engineio\server.py __init__ method. I'm baffled. Any ideas?

[miguelgrinberg]

@crobertsbmw make sure the engineio/async_threading.py module is also imported, so that pyinstaller pulls it, along with all of its dependencies.

[crobertsbmw]

@miguelgrinberg Perfect. All I had to do was add from engineio import async_threading to the top of my script file to trick PyInstaller into pulling it, and that did the trick. Thanks so much!

[cs01]

eventlet==0.20.1 doesn't play well with pyinstaller==3.2.1.

Thanks @crobertsbmw, this helped me too. Or is it that pyinstaller doesn't play well with eventlet?Either way, falling back to threading is a good workaround.

[tonk777]

I've looked into why pyinstaller doesn't work with eventlet under flask-socketio. First off there are two hidden (run-time dynamic) imports required (first is part of flask-socketio; the second is from within eventlet):

engineio.async_eventlet
eventlet.support.dns

Unfortunately eventlet\support\greendns.py loads eventlet.support.dns by adding the full path to eventlet\support to sys.path (see code below) and tries to import the eventlet.support.dns module using just the identifier dns. This method (modifying sys.path at run-time) is incompatible with pyinstaller. eventlet\support\greendns.py needs to be modified to work with pyinstaller

The added ## comments are mine

def import_patched(module_name):
    # Import cycle note: it's crucial to use _socket_nodns here because
    # regular evenlet.green.socket imports *this* module and if we imported
    # it back we'd end with an import cycle (socket -> greendns -> socket).
    # We break this import cycle by providing a restricted socket module.
    # if (module_name + '.').startswith('dns.'):
    #     module_name = 'eventlet.support.' + module_name
    modules = {
        'select': select,
        'time': time,
        'os': os,
        'socket': _socket_nodns,
    }
    return patcher.import_patched(module_name, **modules)


sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))  ## This does not do the expected thing in a pyinstaller package
dns = import_patched('dns') ## Hence this is not found even after using --hidden-import=eventlet.support.dns
for pkg in dns.__all__:
    setattr(dns, pkg, import_patched('dns.' + pkg))
for pkg in dns.rdtypes.__all__:
    setattr(dns.rdtypes, pkg, import_patched('dns.rdtypes.' + pkg))
for pkg in dns.rdtypes.IN.__all__:
    setattr(dns.rdtypes.IN, pkg, import_patched('dns.rdtypes.IN.' + pkg))
for pkg in dns.rdtypes.ANY.__all__:
    setattr(dns.rdtypes.ANY, pkg, import_patched('dns.rdtypes.ANY.' + pkg))
del import_patched
sys.path.pop()

[nat-goodspeed]

@tonk777, I see that you quoted and commented the relevant passages from greendns.py, indicating what does not work.

Do you know what would work? How should greendns.py be modified?

[tonk777]

I did not investigate this particular issue any further because I switched to using an alternate library supported by flask-socketio which worked for me with pyinstaller (gevent). However I can point you towards a solution

The key for pyinstaller support is to avoid this -> os.path.dirname(__file__) as __file__ is not defined in the expected way in a "bundled" environment. You need to do a run-time check for if you are in a bundled environment and go from there. The relevant info that should allow a good fix is in this section of the pyinstaller docs.

http://pyinstaller.readthedocs.io/en/stable/runtime-information.html

It may just be that when in a "bundled" environment (sys.frozen exists) just a different path evaluation to the eventlet.support.dns module is needed (based off sys._MEIPASS and not __file__) and the sys.path temporary modification is still usable

Hope this helps

[nat-goodspeed]

fwiw - for closure - https://github.com/eventlet/eventlet/pull/486 seems to permit PyInstaller to pack a script that uses eventlet, given this hook-eventlet.support.greendns.py file in a PyInstaller --additional-hooks-dir directory:

# PyInstaller hook for dynamic import of dnspython

from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('dns')

[SelfDown]

It works!!!
I add 'engineio.async_gevent' to hiddenimports in spec file.
very, very grateful for your help!! lol hahahahaha

thanks a lot ,resolve my problem

[CpHarding]

It appears you now need

'engineio.async_drivers.eventlet'
not
'engineio.async_eventlet'

in hidden imports using the below version:

python-engineio 3.3.0
python-socketio 3.1.2
eventlet 0.24.1

[miguelgrinberg]

@CpHarding Yes, that is correct, the async drivers have been through a restructure in the v3 engineio package.

[TheAethereal]

I'm still struggling with this. I'm using the following command:

python -m PyInstaller -F signage.py --hidden-import=engineio.async_drivers.eventlet

My versions:

eventlet==0.24.1
PyInstaller==3.4
python-engineio==3.4.3
python-socketio==3.1.2

Error result:

Traceback (most recent call last):
  File "W:\Source\Sign\signage.py", line 21, in <module>
    socketio = SocketIO(app)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\flask_socketio\__init__.py", line 163, in __init__
    self.init_app(app, **kwargs)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\flask_socketio\__init__.py", line 221, in init_app
    self.server = socketio.Server(**self.server_options)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\socketio\server.py", line 89, in __init__
    self.eio = self._engineio_server_class()(**engineio_options)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\engineio\server.py", line 129, in __init__
    raise ValueError('Invalid async_mode specified')
ValueError: Invalid async_mode specified
[4936] Failed to execute script signage

[KITSMaheshChaudhari]

just importing gevent from engineio.async_drivers worked for me.

from engineio.async_drivers import gevent

[cschar]

I wrote a server in eventlet, came across all the nasty incompatibility issues listed above, switched to gevent, still no luck trying all proposed solutions in this thread.

PyInstaller develop branch (06f7da78) via pip install -e
Windows 10
Python 3.6.4

Went ahead and rewrote it using aiohttp.
used this as a template:
https://github.com/miguelgrinberg/python-socketio/blob/master/examples/server/aiohttp/app.py
Finally worked!

no hidden imports needed, no bajillion dns module required lol

[frankxiongzz]

I solved this issue by copy whole "engineio" directory to my dist/app/"engineio"
For example, copy your {{python installl dir}}/Lib/site-packages/engineio to /dist/{{your app}}/engineio. then run your app

[patarapolw]

just importing gevent from engineio.async_drivers worked for me.

from engineio.async_drivers import gevent

I also have to --hidden-import=gevent

[lysdyt]

from engineio.async_drivers import eventlet worked for me
socketio = SocketIO(app, async_mode='eventlet')
--hidden-import=engineio.async_eventlet

[minusplusminus]

Limiting to requirements inside an virtualenv fixed all issues for me. Somehow pyinstaller doesn't like that with the regular:

sudo python3 -m pip install --user virtualenv
python3 -m venv env
source env/bin/activate
pip3 install  --requirement requirements.txt

This reduced the file size for me too.

[fengerzh]

I use eventlet. None of the above solved my problem. Finally I got it run by these 3 steps:

Modify app.spec file:

hiddenimports=['engineio.async_drivers.eventlet'],

Modify app.py file

Add these lines at file head, it's useless to the python file, but it's necessary for pyinstaller packing.

from eventlet.hubs import epolls, kqueue, selects
from dns import dnssec, e164, hash, namedict, tsigkeyring, update, version, zone

Running pyinstaller

pyinstaller app.spec

Do NOT run pyinstaller app.py, it will overwrite app.spec.

[yingshaoxo]

Maybe the solution is just as simple as importing that module like this: from engineio.async_drivers import gevent

https://stackoverflow.com/a/55642042/8667243

[fengerzh]

Right! If you have to use multi-threading mode like me, you can add line as below and pyinstall it:

from engineio.async_drivers import threading

and also force it to use threading mode when create it:

        socketio = SocketIO(
            app,
            async_mode="threading"
        )

[Bongua]

just importing gevent from engineio.async_drivers worked for me.

from engineio.async_drivers import gevent

Thank you very much !!!!!!!!

[PSNAppz]

pip install eventlet did the trick for me

[lizhengtian]

Flask-SocketIO is Running under Werkzeug
hiddenimports=['engineio.async_drivers.threading']
I can package and run .exe file as normal,but the socketio is disabled.The front end cannot be automatically updated.
Can anyone give me some advice? Thank you