Sphinx: [docutils 0.13.1] assert not self.context

[tardyp]

Following traceback is found when building buildbot documentation after upgrade to docutils 0.13.1

Traceback (most recent call last):
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/sphinx/cmdline.py", line 296, in main
    app.build(opts.force_all, filenames)
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/sphinx/application.py", line 333, in build
    self.builder.build_update()
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 251, in build_update
    'out of date' % len(to_build))
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 322, in build
    self.write(docnames, list(updated_docnames), method)
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 360, in write
    self._write_serial(sorted(docnames), warnings)
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 368, in _write_serial
    self.write_doc(docname, doctree)
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/sphinx/builders/html.py", line 452, in write_doc
    self.docwriter.write(doctree, destination)
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/docutils/writers/__init__.py", line 80, in write
    self.translate()
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/sphinx/writers/html.py", line 48, in translate
    self.document.walkabout(visitor)
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/docutils/nodes.py", line 187, in walkabout
    visitor.dispatch_departure(self)
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/docutils/nodes.py", line 1895, in dispatch_departure
    return method(node)
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
    assert not self.context, 'len(context) = %s' % len(self.context)
AssertionError: len(context) = 4

You can reproduce by downloading a copy of buildbot

git clone [email protected]:buildbot/buildbot.git
cd buildbot
pip install -e ./master[docs]'
make docs

[antnieszka]

Similar issue with travis/tox:
https://travis-ci.org/ant6/beprof/jobs/182614760

ant6% ➜ tox -e docs                    
GLOB sdist-make: /home/ant6/PycharmProjects/beprof/setup.py
docs installed: alabaster==0.7.9,apparmor==2.10.1,Babel==2.3.4,-e git+https://github.com/DataMedSci/beprof.git@55cc6fad7d1af630b4d3f5d77782335a86e79552#egg=beprof,bottle==0.12.8,Brlapi==0.6.3,certifi==2015.9.6.2,cffi==1.1.2,coverage==3.7.1,cupshelpers==1.0,cycler==0.10.0,docutils==0.13.1,ecdsa==0.13,imagesize==0.7.1,ipython==3.2.2,Jinja2==2.8,jsonschema==2.5.1,LibAppArmor==2.10.1,louis==2.5.4,MarkupSafe==0.23,matplotlib==1.5.3.post0,mistune==0.7,nose==1.3.7,numpy==1.11.2,paramiko==1.15.2,pexpect==4.1.0,ptyprocess==0.5.1,py==1.4.30,pycparser==2.14,pycrypto==2.6.1,pycups==1.9.72,pycurl==7.19.5.1,Pygments==2.0.2,pygobject==3.16.2,pymongo==3.0.3,pyparsing==2.0.3,pysmbc==1.0.15.4,python-dateutil==2.4.2,pytz==2015.6,pyzmq==14.7.0,requests==2.7.0,scipy==0.18.1,simplejson==3.8.0,six==1.9.0,snowballstemmer==1.2.1,Sphinx==1.5,tornado==4.2.1,versioneer==0.16
docs runtests: PYTHONHASHSEED='1692403530'
docs runtests: commands[0] | pip install -e .
Obtaining file:///home/ant6/PycharmProjects/beprof
Requirement already satisfied: numpy>=1.8.0 in /usr/lib64/python3.4/site-packages (from beprof==0.post0.dev2011481296902)
Installing collected packages: beprof
  Found existing installation: beprof 0.post0.dev201
    Uninstalling beprof-0.post0.dev201:
      Successfully uninstalled beprof-0.post0.dev201
  Running setup.py develop for beprof
Successfully installed beprof
docs runtests: commands[1] | sphinx-build -b html -d /home/ant6/PycharmProjects/beprof/.tox/docs/tmp/doctrees docs docs/build/_html
Running Sphinx v1.5
loading translations [en]... done
loading pickled environment... not yet created
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 4 source files that are out of date
updating environment: 4 added, 0 changed, 0 removed
reading sources... [ 25%] authors
reading sources... [ 50%] contributing
reading sources... [ 75%] index
reading sources... [100%] readme

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 25%] authors
writing output... [ 50%] contributing
writing output... [ 75%] index
writing output... [100%] readme

Exception occurred:
  File "/home/ant6/PycharmProjects/beprof/.tox/docs/lib/python3.4/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
    assert not self.context, 'len(context) = %s' % len(self.context)
AssertionError: len(context) = 3
The full traceback has been saved in /tmp/sphinx-err-2qj4g5b4.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
ERROR: InvocationError: '/home/ant6/PycharmProjects/beprof/.tox/docs/bin/sphinx-build -b html -d /home/ant6/PycharmProjects/beprof/.tox/docs/tmp/doctrees docs docs/build/_html'
_____________________________________________________________________________________________________________ summary ______________________________________________________________________________________________________________
ERROR:   docs: commands failed

[mtbc]

Same with https://travis-ci.org/openmicroscopy/ome-documentation/builds/182595860 -

     [exec] writing output... [ 12%] developers/Model/StructuredAnnotations
     [exec] writing output... [ 12%] developers/Model/Units
     [exec] 
     [exec] Exception occurred:
     [exec]   File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
     [exec]     assert not self.context, 'len(context) = %s' % len(self.context)
     [exec] AssertionError: len(context) = 2
     [exec] The full traceback has been saved in /tmp/sphinx-err-pe_H_o.log, if you want to report the issue to the developers.
     [exec] Please also report this if it was a user error, so that a better error message can be provided next time.
     [exec] A bug report can be filed in the tracker at <https://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks!
     [exec] writing output... [ 13%] developers/Model/XsltTransformations

In our .travis.yml we do pip install Sphinx==1.2.3 so could this new issue be in a dependency?

[tardyp]

It looks like it comes from docutils 0.13.1

pip install docutils==0.12   # fixes the issue.

After investigation:

• self.context contains 4 empty strings

• those empty strings are inserted in visit_image from html.py
  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/docutils/nodes.py", line 1882, in dispatch_visit
  return method(node)

  File "/Users/ptardy/dev/bb/buildbot/sandbox/lib/python2.7/site-packages/sphinx/writers/html.py", line 476, in visit_image
  self.context.append('')

[tardyp]

@gmilde @grubert the maintainers of docutils may have some ideas

[antnieszka]

docutils <0.13.1 worked for me, thanks 😄

[gmilde]

Sphinx overwrites visit_item() but not depart_item() in writer.html.

For other functions, it keeps the pair together:

# overwritten (but not changed) to keep pair of visit/depart_definition

This is the right thing to do:

Docutils html writer has a changed visit_item() function in version 0.13, which does not use the
heterogeneous "context" stack. Therefore, also the depart_item() was changed to just pass.

Copying the old depart_image() function to writers/html.py should solve the issue.

--- /usr/lib/python2.7/dist-packages/sphinx/writers/html.py 2016-10-01 17:14:37.000000000 +0200
+++ /tmp/html23031.4203.py  2016-12-09 22:19:36.815344886 +0100
@@ -498,6 +498,10 @@
                     node['height'] = str(size[1])
     BaseTranslator.visit_image(self, node)

+    # overwritten (but not changed) to keep pair of visit/depart_image
+    def depart_image(self, node):
+        self.body.append(self.context.pop())
+
 def visit_toctree(self, node):
     # this only happens when formatting a toc from env.tocs -- in this
     # case we don't want to include the subtree

``

[SylvainCorlay]

I am actually seeing the same error on RTD with docutils 0.12. (unless the version number displayed by RTD does not actually reflect the installed version of docutils ).

https://readthedocs.org/projects/ipywidgets/builds/4771583/

[tk0miya]

@SylvainCorlay Your case is similar to this, but not related issue.
could you file a new issue?

[tk0miya]

Merged #3217.

[tk0miya]

@SylvainCorlay Hmm? It seems your case is same as this issue.

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/ipywidgets/conda/latest/lib/python3.5/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
    assert not self.context, 'len(context) = %s' % len(self.context)
AssertionError: len(context) = 9

docutils/writers/_html_base.py is new file since docutils-0.13. It's not docutils-0.12.

At my local, it was also fixed by #3217.

[SylvainCorlay]

@tk0miya thanks. So it must be the combination of conda environment.yml and pip dependencies that results in docutils 0.13.1 being installed even when I specify <0.13.1. I will try to find the right incantations to get the version I need.

[jakirkham]

So it must be the combination of conda environment.yml and pip dependencies that results in docutils 0.13.1 being installed even when I specify <0.13.1. I will try to find the right incantations to get the version I need.

Had the same issue with RTD, @SylvainCorlay. Turns out they have a pip install step after the conda env step. In that step, docutils gets installed with version 0.13.1. So it is the same issue.

[return42]

I like to see this bug reopend. It is only fixed for a v1.5 patch level. A patch level for 1.3 and 1.4 is also needed.

To fix downwards, I recomend to fix the requirement docutils>=0.11 (https://github.com/sphinx-doc/sphinx/blob/1.4.9/setup.py#L47) and stick at docutils==0.11.

I also recomend to stick docutils for the future in a new sphinx 1.5 patch level to current docutils==0.13.1 (https://github.com/sphinx-doc/sphinx/blob/1.5.3/setup.py#L48)

Thanks!

[tk0miya]

Sorry, 1.4 and 1.5 are already outdated. We maintain only stable release.
Thanks,

[return42]

1.5 is outdated? ... I guess you mean 1.3 and 1.4 is outdated and you only maintain a HEAD version along your stable branch ... right?

[tk0miya]

Ah, sorry. 1.3 and 1.4 are outdate. 1.5 is still alive.