Sphinx: Sphinx 1.8.0 fails build with syntax error unqualified exec is not allowed in function 'run'

[zxiiro]

Problem

• Build fails with syntax error SyntaxError: unqualified exec is not allowed in function 'run' it contains a nested function with free variables

Procedure to reproduce the problem

tox -e docs

This can be reproduced with docker too.

docker run -it --rm centos:7 /bin/bash

yum install git gcc python-virtualenv
virtualenv /tmp/venv
source /tmp/venv/bin/activate
pip install tox

git clone https://github.com/lfit/releng-lftools
cd releng-lftools
tox -e docs

Error logs / results

Running Sphinx v1.8.0

Exception occurred:
  File "/w/workspace/lf-infra-lftools-tox-verify-any/.tox/docs/lib/python2.7/site-packages/sphinx/highlighting.py", line 26, in <module>
    from sphinx.ext import doctest
SyntaxError: unqualified exec is not allowed in function 'run' it contains a nested function with free variables (doctest.py, line 97)
The full traceback has been saved in /tmp/sphinx-err-SIpkyY.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

• https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/lf-infra-lftools-tox-verify-any/338/console.log.gz

Expected results

Docs successfully builds as it did on the previous release.

Reproducible project / your project

https://github.com/lfit/releng-lftools

Environment info

• OS: Linux, Cent OS 7
• Python version: 2.7.5
• Sphinx version: 1.8.0

[zxiiro]

I posted the logs for one of our projects. But many of our projects that's using Sphinx started failing in our CI this morning with this issue after 1.8.0 was made public.

[tannewt]

This happens on Travis as well: https://travis-ci.org/adafruit/circuitpython/jobs/427888642#L1303

[kunicmarko20]

Same here https://travis-ci.org/FriendsOfSymfony/FOSCKEditorBundle/jobs/427947749

[tk0miya]

Sorry for inconvenience. This might be related with doctest_global_setup. But I can't find where do you set up config variables on your project. Could you let me know where it is used?

[tk0miya]

Note: The code was added in #5307.
@akaihola Please let me know about the error if you find the reason.

[elafontaine]

This seems to be caused by the way exec works;
https://stackoverflow.com/questions/4484872/why-doesnt-exec-work-in-a-function-with-a-subfunction

Maybe you guys want to specify the context to exec to avoid that issue. (I don't have the pretention to understand what it means or which context to apply) :) . Just trying to help as I'm impacted as well.

[gillesdouaire]

Right now we'll use Sphinx !=1.8.0 in our requirements to unblock-- so we'll use 1.7.9 for the time being, and then 1.8.1 as soon as it's out.

[akaihola]

@tk0miya, the failing tests above seem to have been run on Python 2.7.5. Sphinx 1.8.0 only supports Python >=3.5.

I tried this minimal test case to see differences in behavior between Python versions:

def func():
    context = {}
    exec('print("hi")', context)

    def inner():
        return None

func()

• with Python 2.6.9, this fails in the same way as the test mentioned above
• with Python 2.7.15, it succeeds (strangely enough)
• with Python 3.5.5, 3.6.5 and 3.7.0b4, it succeeds

I'll try to build Python 2.7.5 to see if there really is a difference between 2.7.5 and 2.7.15.

On Python 2.6, changing the exec line to:

exec 'print("hi")' in context

fixes the problem, but is not valid in Python 3.x.

What still baffles me is that the TestDirective.run() method doesn't actually have inner functions!

Anyway, if it's true that this doesn't fail on any Python version >=3.5, do we still want to try and fix compatibility on Python <3.5?

[tk0miya]

@akaihola No, Sphinx-1.8 has still supported python 2.7. So we should make a fix for both py27 and py34+

[tk0miya]

I confirmed with following Dockerfile:

FROM centos:7

RUN yum install -y git gcc epel-release
RUN yum install -y python-pip
RUN git clone https://github.com/lfit/releng-lftools
RUN pip install tox
WORKDIR /releng-lftools
RUN yum install -y python-devel
RUN tox -e docs --notest
RUN .tox/docs/bin/pip install -U Sphinx
RUN tox -e docs

[tk0miya]

Finally, I found the reason of the error. It seems list comprehension conflicts with exec under py2.7.5.

[root@0c85e272ba90 releng-lftools]# python -V
Python 2.7.5
[root@0c85e272ba90 releng-lftools]# cat test.py
def func():
    exec('print("h1")', {})
    (str(e) for e in range(10))
[root@0c85e272ba90 releng-lftools]# python test.py
  File "test.py", line 2
    exec('print("h1")', {})
SyntaxError: unqualified exec is not allowed in function 'func' it contains a nested function with free variables

And it is already resolved at latest python2.7.

$ python2.7 -V
Python 2.7.15
$ cat test.py
def func():
    exec('print("h1")', {})
    (str(e) for e in range(10))
$ python2.7 test.py
$

[tk0miya]

I confirmed #5443 fixes the problem:

FROM centos:7

RUN yum install -y git gcc epel-release
RUN yum install -y python-pip
RUN git clone https://github.com/lfit/releng-lftools
RUN pip install tox
WORKDIR /releng-lftools
RUN yum install -y python-devel
RUN tox -e docs --notest
RUN .tox/docs/bin/pip install -U git+https://github.com/tk0miya/sphinx@5417_crashed_with_py275
RUN tox -e docs

[elafontaine]

I think #5443 would be a simple fix to merge! The change is not that big and will resolve the issue :).

Thanks a lot @tk0miya for looking into this. It will make sphinx be able to be updated on old version of redhat as well.

[tk0miya]

Fixed by #5443.
Thank you for reporting.