Psycopg2: ImportError: DLL load failed while importing _psycopg: The specified module could not be found in python 3.9

[newETA]

Import failed while working with Python 3.9 experience this issue, But it's fine in 3.7 and 3.8.

Using psycopg2 2.8.6

[AntoninJcb]

I am having the same issue with python 3.9.
I could only make it work by reverting to 3.8.

[dvarrazzo]

Python 3.9 is not supported yet. Packages have not been released.

[hingston]

What is blocking the release?

This is only happening on Windows for me. Related #1099

[dvarrazzo]

Appveyor doesn't support Python 3.9 yet: https://ci.appveyor.com/project/psycopg/psycopg2-wheels/builds/35677049

[ahtik]

For reference, the Appveyor issue tracking Python 3.9 support, which seems to prevent the Python 3.9 Windows psycopg2-binary build.

There seems to be a way to manually install Python 3.9 for Appveyor CI, described in https://github.com/appveyor/ci/issues/3467#issuecomment-643534726. @dvarrazzo Could this be an option to get it built sooner?

[dvarrazzo]

@ahtik I don't have resources to work on this issue in this moment. If someone provides a patch I can release packages before Appveyor completes Python 3.9 support.

[ahtik]

@dvarrazzo Python 3.9.0 is now listed at https://github.com/appveyor/ci/issues/3541 and as of today 3.9.0 is listed at https://www.appveyor.com/docs/windows-images-software/#python.

Could you try running the https://ci.appveyor.com/project/psycopg/psycopg2-wheels build again to heal the PY_VER=39 section?

[dvarrazzo]

@ahtik Sure, trying a build.

[dvarrazzo]

@ahtik Still can't find the executable on the appveyor machine:

• https://ci.appveyor.com/project/psycopg/psycopg2/builds/35977860/job/u2ehf04x6l0wth03
• https://ci.appveyor.com/project/psycopg/psycopg2/builds/35977860/job/376li5550cfsjpcu

[dvarrazzo]

@ahtik Looks like Python 3.9 is not quite there yet. This is what is available on the Appveyor machine:

https://ci.appveyor.com/project/psycopg/psycopg2/build/job/igre2hjljb3ythao

I can try to re-run the job later or in a few days.

[ahtik]

@dvarrazzo Interesting, I just created a basic empty project where 3.9 seems to be available, https://ci.appveyor.com/project/ahtik/appv-39-exp/build/job/mdbt9env6vawk80q

https://github.com/psycopg/psycopg2/blob/master/.appveyor.yml doesn't have an image name specified, so maybe the default is not yet switched and adding smth like image: image: Visual Studio 2019 to appveyor.yml would help... Also found the release notes https://www.appveyor.com/updates/2020/10/26/

[dvarrazzo]

@ahtik I don't understand: your appveyor doesn't have an image either...

[ahtik]

@dvarrazzo My appveyor setup and the whole project was new, I speculated that the "old" projects without pinned env in .yml might be still using older images for slower rollout. To be sure I added image also to the conf and seems to work as well.

The Build worker image in Settings is still "Visual Studio 2019", not the "Previous...", right?

[dvarrazzo]

Seriously? A menu? The yaml file was too much a repeatable experience?

The image of the project was on VS 2015. I switched it to VS 2019, however it doesn't seem to have changed much (https://ci.appveyor.com/project/psycopg/psycopg2/builds/35986746/job/vpalli48lhh3c7e5).

[ahtik]

@dvarrazzo Apparently this menu item gets ignored when you have yaml file already in place, even without an image name. So the default image definition comes from wherever :) See https://github.com/appveyor/ci/issues/3541#issuecomment-717410972

[dvarrazzo]

@ahtik thank you: setting the image into appveyor worked indeed.

The build fails for other reasons that for us forward-slashed people are rather obscure: https://ci.appveyor.com/project/psycopg/psycopg2/builds/35989526/job/t9531ml0y4g6jrwl Would you be able to fix the build?

Thank you very much

[jerickso]

The VC 2019 image has a different version of perl causing that error https://www.appveyor.com/docs/windows-images-software/. A quick hack to add the Strawberry Perl path (C:\Strawberry\perl\bin) before the git perl path continues the build process, but I'm not far enough along to know what other issues this may cause.

The perl issue is not the only error in that log. VS 2019 default's location is different: C:\Program Files (x86)Microsoft Visual Studio\2019\Community\VCAuxiliaryBuild, so the compiler does not get setup.

I'm on another issue that I've come across with building the openssl libs:

Traceback (most recent call last):
  File "C:\Project\psycopg2\scripts\appveyor.py", line 900, in <module>
    sys.exit(main())
  File "C:\Project\psycopg2\scripts\appveyor.py", line 38, in main
    cmd()
  File "C:\Project\psycopg2\scripts\appveyor.py", line 167, in step_build_script
    build_openssl()
  File "C:\Project\psycopg2\scripts\appveyor.py", line 220, in build_openssl
    shutil.rmtree(sslbuild)
  File "C:\Python39\lib\shutil.py", line 737, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Python39\lib\shutil.py", line 615, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Python39\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
OSError: [WinError 87] The parameter is incorrect: 'C:\\Others\\32\\16.0\\Builds\\openssl-OpenSSL_1_1_1g\\NUL'

I've got to pause working on this for awhile, but I'll pick it up later this evening.

[dvarrazzo]

@jerickso Thank you, that would be amazing!

If you are looking into the Windows build, some time ago building the libpq started failing, so the version is being held back to 11.4. There are more details at 5e9572aff8c23f20d16587dd10baa38def6feca9. If you could restore build using libpq from Postgres 12 or 13 it would be amazing.

Thank you very much!

[jerickso]

I'll have to spend some more time getting it to build against a newer Postgres. My initial attempt was sending me down a path that would take some time to hammer out. I'll get it on my list to revisit it.

I'm still doing regression testing with the changes, but I did submit a pull requests for both psycopg2 and the wheels repository. I did not have any of the new appveyor config changes in the psycopg2 repository pull request.

[dvarrazzo]

@jerickso thank you for your work

I did not have any of the new appveyor config changes in the psycopg2 repository pull request.

Do you mean these? https://github.com/psycopg/psycopg2/commits/py39-appveyor

[jerickso]

Sorry I wasn't clearer. I didn't modify the appveyor yaml file in the main psycopg2 repository, just the psycopg2-wheels repository. In hind sight I should have just used the psycopg2 repository since there was a appveyor yaml file there, and it was more work testing out code updates in two repositories. I had the wheels repository already setup in my appveyor account so started my testing there. I'll see it didn't build so I'll get the main repository yaml file updated and do another pull request.

[liniribeiro]

I'm with sabe problem, but im using Python 3.8 on Windows.

[churongcon]

I have same problem, Python 3.9, Postgresql 13

[dvarrazzo]

Windows packages for Python 3.9 are now available: https://pypi.org/project/psycopg2/#files

[dvarrazzo]

Thank you very much to help with them, @jerickso :)

[domdinicola]

@dvarrazzo
when I try to build a docker image with python3.9 I get

File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 377, in execute File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 91, in populate File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 116, in create File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/usr/local/lib/python3.9/site-packages/django/contrib/postgres/apps.py", line 1, in <module> File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module> ImportError: Error relocating /usr/local/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-x86_64-linux-gnu.so: SSL_check_private_key: symbol not found

do you think those issues are related?
I don't get any problem with python3.8

[dvarrazzo]

@domdinicola no, it's not related: this is a windows ticket.