Nuitka: factory openSUSE Urllib3Using.py on Python 2

[jayvdb]

When using branch factory Nuitka-0.6.3.1556202417, on openSUSE Python 2.7, .spec

[  774s] Consider output of recursively compiled program: Urllib3Using.py
[  774s] Comparing output of 'Urllib3Using.py' using '/usr/bin/python2' with flags silent, expect_success, standalone, remove_output ...
[  839s] --- /usr/bin/python2 (stderr)
[  839s] +++ nuitka (stderr)
[  839s] 
[  839s] @@ -1,5 +1,4 @@
[  839s] 
[  839s]  Traceback (most recent call last):
[  839s] -  File "/home/abuild/rpmbuild/BUILD/Nuitka-0.6.3.1556202417.e23ccdbb/tests/standalone/Urllib3Using.py", line 29, in <module>
[  839s] -    from http.server import BaseHTTPRequestHandler, HTTPServer
[  839s] +  File "/home/abuild/rpmbuild/BUILD/Nuitka-0.6.3.1556202417.e23ccdbb/tests/standalone/Urllib3Using.dist/Urllib3Using.py", line 29, in <module>
[  839s]  ImportError: No module named http.server
[  839s]  
[  839s] Error, outputs differed.
[  839s] Error exit! 1
[  839s] error: Bad exit status from /var/tmp/rpm-tmp.ysUnJG (%check)

[tommyli3318]

Looks like the issue comes from Urllib3Using.py trying to use the http.server import, which seems to be only in the standard library of python 3 and not python 2.

[tommyli3318]

@kayhayen would you like me to work on this issue?

[jayvdb]

Confirming this only occurs on Python 2. It passes on Python 3

[kayhayen]

@tommyli3318 Yes, please do so. I think the issue there is that http.server is a Python3 rename, and the Python2 module should be used as a fallback. I have on factory or develop, I forget, added a requirement on http.server as a stop gap measure the other day. But of course we would love to see this run on Python2 too.

[tommyli3318]

@jayvdb which version of Python2.7 are you using? I just ran this using Windows with python2.7.16 and I didn't get this error. Importing http.server seems to raise no issue.

[kayhayen]

@tommyli3318 you probably have a backport installed? This is clearly one of the one Python3 namespace cleanups

[jayvdb]

All issues with openSUSE in the title are using latest openSUSE rpms. In this case, for Python 2, the python and python-base rpms build according to recipes at https://build.opensuse.org/package/show/devel:languages:python:Factory/python . On the RHS are logs, which show the steps of the build.

As @kayhayen mentioned, you must have a backport, as http isnt valid in vanilla Python 2.

https://docs.python.org/3/library/http.html

vs

https://docs.python.org/2.7/search.html?q=http&check_keywords=yes&area=default

p.s. I'd love to know of a good backport of http for Python 2, as I could add that backport to my collection at https://build.opensuse.org/project/show/home:jayvdb:py-backports , which usually make it into the openSUSE main collection. At the end of the year, no more Python 2 stuff will be allowed into the main collection, and they will probably evict them then, but they should still be available in the released version of the OS, giving Python 2 a bit more life for those who need it.

[tommyli3318]

I see, I wasn't even aware that I have backports installed. It might've just came with my Python3 installation. Thanks for clarifications

[tommyli3318]

@jayvdb
I think https://docs.python.org/2.7/glossary.html#term-2to3 is the reason why my Python2 http import worked

[jayvdb]

2to3 is an installation tool. It doesnt help unless you install something which provides http and explicitly asked 2to3 to help (usually getting python2 stuff working under python3, not the other way around).

[jayvdb]

I think https://python-future.org/standard_library_imports.html may be where your http.server is coming from.

[jayvdb]

Yup. I installed python2-future, and the urllib3 test is now passing on Python 2.

It might be useful to move future to near the top of your list of PyPI packages to test, and then add it as a runtime dependency of Nuitka and derivatives on Python 2. It may solve a lot of heartache, as your tests will often have to copy the same voodoo that future has already built.

Soon this will become a loosing battle, as the moment any of the libraries you are testing need future, to properly test the others you need to isolate the test rig for the others, otherwise futures magic (and voodoo by similar libraries) will polute the testing environment. And libraries are also having the same problem -- increasingly future is (indirectly) being used by some part of their test environment, so they dont notice they even depend on its magic.

[tommyli3318]

When you say

add it [future] as a runtime dependency of Nuitka

do you mean make it so that Nuitka doesn't run unless future is present? Please elaborate if I'm wrong.
Sorry I'm new to big open-source projects like Nuitka so I'm not familiar with technical terms.

[jayvdb]

do you mean make it so that Nuitka doesn't run unless future is present?

Yes, for Python 2, and only for Python 2. For the things it silently does to make Python 2 much more similar to Python 3. But that is a big decision. Beyond the scope of this problem.

[kayhayen]

Nah, "Urllib3Using.py" is first and foremost a program tasked to test that "urllib3" is working correctly, with all the Pythons. Therefore It is supposed to be written portable, and not have too unusual dependencies.

All we there need is a small HTTP server, and I believe that https://docs.python.org/2/library/simplehttpserver.html fits the bill here simple enough, so that is what we are going to use.

Then, what the test shows, is that the working program continue to work when compiled with Nuitka in standalone mode. No more.

For nice and more complete test coverage, we are going to run the test suite of urllib3 against the compiled package, but that is not what this test is supposed to do. There we are going to have their test dependencies installed. For now, standalone tests are not really allowed to require lots of things that are not normally there.

[kayhayen]

This is fixed in the current factory and soon 0.6.4, I am closing it therefore.