Nuitka: Importing not thread-save

[EnzoRondo]

Sorry for posting information without any additional data, I have lack of experience to provide you simple example. I hope current information will enough.

The problem was in pyinstaller https://github.com/pyinstaller/pyinstaller/issues/2371 and now I see absolutely same problem with Nuitka last factory version on windows 7. One thread works perfect, 2 threads or more shows:

AttributeError: module 'XXX' has no attribute 'YYY'

[kayhayen]

Yes, importing has a lock. Nuitka uses it when it calls "__import__" automatically. But for embedded module imports, it might not be acquired. Also not for attribute lookup forms of imports with Python3.

A test case would be needed though, otherwise this will have to be delayed.

[EnzoRondo]

Thanks for so fast reply, I will try to make test case

[EnzoRondo]

arguments:

nuitka _nb.py ^
--standalone ^
--recurse-all ^

test case (_nb.py)

import threading


def doit(nm):
    print(nm, 'started')
    import requests
    try:
        print(nm, requests.get('https://google.com'))
    finally:
        print(nm, requests)


t1 = threading.Thread(target=doit, args=('t1',))
t2 = threading.Thread(target=doit, args=('t2',))
t1.start()
t2.start()
doit('main')
t1.join()
print('t1 joined')
t2.join()
print('t2 joined')
print('finished.')

current output:

t1 started
t2 started
main started
t2 <module 'requests' from 'C:\\Users\\Administrator\\Desktop\\wow\\_nb.dist\\requests\\__init__.p
y'>
main <module 'requests' from 'C:\\Users\\Administrator\\Desktop\\wow\\_nb.dist\\requests\\__init__
.py'>
Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\_nb.py", line 17, in <module>
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\_nb.py", line 8, in doit
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\_nb.py", line 8, in doit
AttributeError: module 'requests' has no attribute 'get'

AttributeError: module 'requests' has no attribute 'get'
t1 <Response [200]>
t1 <module 'requests' from 'C:\\Users\\Administrator\\Desktop\\wow\\_nb.dist\\requests\\__init__.p
y'>

expected output:

t1 started
t2 started
main started
main <Response [200]>
main <module 'requests' from 'C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python36\\lib\\site-packages\\requests\\__init__.py'>
t1 <Response [200]>
t1 <module 'requests' from 'C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python36\\lib\\site-packages\\requests\\__init__.py'>
t1 joined
t2 <Response [200]>
t2 <module 'requests' from 'C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python36\\lib\\site-packages\\requests\\__init__.py'>
t2 joined
finished.

[kayhayen]

I don't currently have a Windows, but I am getting the idea there. Indeed, one thread loads the requests module, and before it is finished the other one uses it.

With Python2 I could not reproduce the issue at all, but with Python3 easily. Thanks for the reproducer. I am going to see what is going on there. Likely the properly using the lock for importng that I recall seeing in CPython code will do the trick already.

Yours,
Kay

[kayhayen]

For Python 3.3, there was "__initializing__" which seems to have moved into the "__spec__" of the module as "_initializing".

Nuitka assigns a "__spec__" when loading the compiled module, but definitely doesn't update it after module load is complete or set that attribute. I am changing this now.

The way CPython does this, uses a context manager that preserves the initial __spec__ value. My trial code won*t be as robust as that yet. But your example works immediately. I will try and get this sufficiently robust, so we can make a hotfix out of it.

Yours,
Kay

[kayhayen]

I pushed to factory a commit that addresses the issue, let me know if it works for you:

http://nuitka.net/doc/factory.html

[EnzoRondo]

Thanks a lot dear friend, you are extremely fast!

current output:

t1 started
t2 started
main started
t2 <module 'requests' from 'C:\\Users\\Administrator\\Desktop\\wow\\_nb.dist\\requests\\__init__.p
y'>
main <module 'requests' from 'C:\\Users\\Administrator\\Desktop\\wow\\_nb.dist\\requests\\__init__
.py'>
Traceback (most recent call last):
t1 <module 'requests' from 'C:\\Users\\Administrator\\Desktop\\wow\\_nb.dist\\requests\\__init__.p
y'>
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\_nb.py", line 17, in <module>
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\_nb.py", line 8, in doit
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\api.py", line 72, in get
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\api.py", line 58, in request
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\_nb.py", line 8, in doit
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\api.py", line 72, in get
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\api.py", line 58, in request
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\sessions.py", line 512, in request
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\sessions.py", line 622, in send
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\adapters.py", line 412, in send
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\adapters.py", line 226, in cert_verify

OSError: Could not find a suitable TLS CA certificate bundle, invalid path: C:\Users\Administrator\Desktop\wow\_nb.dist\certifi\cacert.pem
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\sessions.py", line 512, in request

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\_nb.py", line 8, in doit
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\api.py", line 72, in get
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\api.py", line 58, in request
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\sessions.py", line 512, in request
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\sessions.py", line 622, in send
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\adapters.py", line 412, in send
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\adapters.py", line 226, in cert_verify

OSError: Could not find a suitable TLS CA certificate bundle, invalid path: C:\Users\Administrator\Desktop\wow\_nb.dist\certifi\cacert.pem
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\sessions.py", line 622, in send

  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\adapters.py", line 412, in send
  File "C:\Users\Administrator\Desktop\wow\_nb.dist\requests\adapters.py", line 226, in cert_verify

OSError: Could not find a suitable TLS CA certificate bundle, invalid path: C:\Users\Administrator\Desktop\wow\_nb.dist\certifi\cacert.pem

added \certifi\cacert.pem and now works perfect:

t1 started
t2 started
main started
t2 <Response [200]>
t2 <module 'requests' from 'C:\\Users\\Administrator\\Desktop\\wow\\_nb.dist\\requests\\__init__.p
y'>
main <Response [200]>
main <module 'requests' from 'C:\\Users\\Administrator\\Desktop\\wow\\_nb.dist\\requests\\__init__
.py'>
t1 <Response [200]>
t1 <module 'requests' from 'C:\\Users\\Administrator\\Desktop\\wow\\_nb.dist\\requests\\__init__.p
y'>
t1 joined
t2 joined
finished.

that would be perfect to fix certifi issue

[kayhayen]

Given a reproducer, no issue, it was pretty obvious from the example code.

The certify issue is something I had been reported to before, but couldn't point a finger, if the ".pem" file is from the package or user provided. Can you create a new issue with a minimal producer for that too, and I will gladly fix it too if possible.

Yours,
Kay

[kayhayen]

I pushed this to the hotfix/0.5.32.8 branch, but I won't be able to actually release it until after my holiday I suspect, at least not to PyPI.

[EnzoRondo]

Thanks a lot dear friend, you are awesome!