Psycopg2: Issue building against Python 3.8

[coleifer]

I recently ran into the following building psycopg2 2.7.7 on Python 3.8:

  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION=2.7.7 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110001 -DHAVE_LO64=1 -I/opt/python/3.8-dev/include/python3.8m -I. -I/usr/include/postgresql -I/usr/include/postgresql/11/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.8/psycopg/psycopgmodule.o -Wdeclaration-after-statement
  psycopg/psycopgmodule.c: In function ‘psyco_is_main_interp’:
  psycopg/psycopgmodule.c:689:18: error: dereferencing pointer to incomplete type ‘PyInterpreterState {aka struct _is}’
       while (interp->next)

It looks like in 3.8 the PyInterpreterState is moved into an internal header. From the changes doc:

The PyInterpreterState struct has been moved into the “internal” header files (specifically Include/internal/pycore_pystate.h). An opaque PyInterpreterState is still available as part of the public API (and stable ABI). The docs indicate that none of the struct’s fields are public, so we hope no one has been using them. However, if you do rely on one or more of those private fields and have no alternative then please open a BPO issue. We’ll work on helping you adjust (possibly including adding accessor functions to the public API). (See bpo-35886.)

Just a heads-up.

[dvarrazzo]

Fixed as of #854

[gotexis]

@dvarrazzo I saw commits that says 3.8 build succeeded. Have you built successfully? I can safely upgrade to py3.8?

Should I install from git master?

[dvarrazzo]

I built one-off with Py 3.8 from master. Py 3.8 has not been released yet and there is no support it Travis so it's not officially supported yet.

You can build on master, and it should build.

[tdakkota]

Python 3.8 has been released.

[dvarrazzo]

I'll make a release in a few days.

[MinchinWeb]

Just ran into this issue. Looking forward to this to the release.

[GauravRajSharma]

Waiting for the release.

[manuhortet]

Had to downgrade some stuff back to 3.7 for this. Can you prioritize it? @dvarrazzo

[jarekwg]

In case anyone here is still waiting (I didn't realise release was published two days ago. :D):
https://github.com/psycopg/psycopg2/releases/tag/2_8_4

[ShingenPizza]

I'm not sure it all works as intended.
On Windows 7 in Python 3.8 trying to install psycopg2 caused some errors about 10 days ago, but it doesn't do that anymore for whatever reason.
Installing psycopg2-binary instead worked, but trying to import it causes the following error

>>> import psycopg2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python38\lib\site-packages\psycopg2\__init__.py", line 50, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: DLL load failed while importing _psycopg: Nie można odnaleźć określonego modułu.

("Nie można odnaleźć określonego modułu." means "The specified module could not be found.")
A couple of days ago i tried installing regular psycopg2 again and it worked this time, but importing it causes the above error too.
When i tried to install it in Python 3.7.3 on the same machine both psycopg2 and psycopg2-binary install themselves properly (not at the same time obviously) AND cause no errors when imported.

I checked site-packages folders for both Python versions and i can see 2 differences:
Python 3.7: psycopg2/_psycopg.cp37-win_amd64.pyd file size is 2 373 632B
Python 3.8: psycopg2/_psycopg.cp38-win_amd64.pyd file size is only 255 488B
all other files in these folders are the same, except that 3.7.3 files' modification dates are of the installation moment, while 3.8 files' are all 2019-10-20 (except _psycopg.cp38-win_amd64.pyd that has the installation date).
Python 3.7 has a folder named psycopg2-2.8.4.dist-info or psycopg2_binary-2.8.4.dist-info
Python 3.8 has psycopg2-2.8.4-py3.8.egg-info or psycopg2_binary-2.8.4-py3.8.egg-info respectively.
Their content (between 3.7.3 and 3.8) is significantly different, i guess due to wheel vs egg formats.
Not sure if it means anything, but the installation on 3.8 seems to take a couple of times longer too.

Now just trying weird things, trying to install it in Python 3.7 with c:\python37\scripts\pip.exe install --no-cache-dir --no-binary :all: psycopg2 creates a psycopg2-2.8.4-py3.7.egg-info folder, and in the psycopg2 folder there's a _psycopg.cp37-win_amd64.pyd file that weights only 250 880B.
Importing works.

Any ideas what's wrong?

Also, while that isn't strictly my problem, i thought the 2.8+ psycopg2 was supposed to be always built from scratch. Why is it still installing a - to my understanding - prebuilt binary wheel version on Python 3.7 unless told otherwise?

[dvarrazzo]

@ShingenPizza see #990. Python 3.8 is supported but binary packages for Windows are not built because Appveyor doesn't support it yet.

[ShingenPizza]

Oh, okay, thanks for the information, and sorry for not searching for it further... I'll continue in #1006 that seems to be reporting the same issue.