Readthedocs.org: Problems with project virtualenvs

[cornhundred]

Details

I am getting the following error on readthedocs when I try to manually build docs or when a build is triggered by a push to the GitHub repo:

 New python executable in /home/docs/checkouts/readthedocs.org/user_builds/clustergrammer/envs/latest/bin/python2.7
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 2328, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 925, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 1231, in install_python
    shutil.copyfile(executable, py_executable)
  File "/usr/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 40] Too many levels of symbolic links: '/home/docs/checkouts/readthedocs.org/user_builds/clustergrammer/envs/latest/bin/python2.7'

The builds seemed to spontaneously stop working and reverting to old versions of docs that previously worked does not help. I am running

sphinx-autobuild . _build_html
to build the docs.

• Project URL:
  http://clustergrammer.readthedocs.io/en/latest/
• Build URL (if applicable):
  https://github.com/MaayanLab/clustergrammer/tree/master/docs
• Read the Docs username (if applicable):
  corrnhundred
  

  Expected Result

I modified the documentation, successfully built with sphinx-autobuild, pushed to GitHub, but the build (automatic web hook) failed on readthedocs' site.
A description of what you wanted to happen
Documents to build on readthedocs' site

Actual Result

Build fail.
A description of what actually happened

[ppwwyyxx]

Also failed for me with the same error. My project is tensorpack.

[rameshthoomu]

Same error observed in my mkdocs build. Log output
Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 2328, in <module> main() File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 713, in main symlink=options.symlink) File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 925, in create_environment site_packages=site_packages, clear=clear, symlink=symlink)) File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 1231, in install_python shutil.copyfile(executable, py_executable) File "/usr/lib/python2.7/shutil.py", line 83, in copyfile with open(dst, 'wb') as fdst: IOError: [Errno 40] Too many levels of symbolic links: '/home/docs/checkouts/readthedocs.org/

This is blocking our documentation build process. Could you please check this let us know the workaround steps or please provide the fix ASAP.

[cornhundred]

I also tried re-building the docs from scratch using:

sphinx-quickstart

and deleting/remaking the project on readthedocs.io, but I am still getting the same error.

I also tried moving my docs to a new repo and building with the same project name, but this failed.

However, I am able to build using the same documentation (in a new repo) with a different project name

http://clustergrammer-docs.readthedocs.io/en/latest/.

So, I think the error is somehow tied to the url/project-name

http://clustergrammer.readthedocs.io/en/latest/

since I cannot get anything to build using that name, but I can get the same docs to build using another project name on readthedocs.io.

[agjohnson]

This is an issue stemming from upgrading the build container image, virtualenv does not like this change at all.

[agjohnson]

@cornhundred @ppwwyyxx I removed the virtualenvs for your project

@rameshthoomu I'm not sure what your project is, it looks like the output is scrubbed there.

[cornhundred]

@agjohnson Thanks, that worked - I just rebuilt it. Do you have any idea what could have caused that (e.g. the web hook) so I can try and prevent it?

[rameshbabu79]

@agjohnson : here are the project details hyperledger-fabric

Project link: http://hyperledger-fabric.readthedocs.io/en/latest/

[agjohnson]

This is a problem on our side, from upgraded build images. Not sure exactly what causes this scenario though, but perhaps it's a slight difference in the installed Python version between the two images.

[agjohnson]

@rameshthoomu this should be removed now

[rameshthoomu]

thanks @agjohnson.. that worked..

[timabbott]

I'm still seeing this issue now with the zulip project. E.g. https://readthedocs.org/projects/zulip/builds/5028881/

[agjohnson]

@timabbott wiped your envs from our builders and triggered a new build, seems to work

If anyone else has a project that is failing in a similar way, you have the power to wipe the project's versions in your version admin dashboard. Select the active builds you are having trouble with and select "Wipe".

Over the next two days, I'll be cycling each of our build servers to use the new build image. This will involve wiping everyone's virtualenv anyways. I started today with one build server, so your project may currently pass 25% of the time if it is failing. Otherwise, by the end of the week I will have all the servers cycled through. This is mostly a failsafe from blowing up everyone's builds at once, should anything be wrong with the new build images.

[psav]

I did the wipe, but now mine are failing with

 Collecting git+https://github.com/ManageIQ/manageiq-api-client-python.git (from -r docs/requirements.txt (line 3))
  Cloning https://github.com/ManageIQ/manageiq-api-client-python.git to /tmp/pip-IYPQRq-build
Obtaining file:///home/docs/checkouts/readthedocs.org/user_builds/cfme-tests/checkouts/master (from -r docs/requirements.txt (line 93))
No files/directories in /home/docs/checkouts/readthedocs.org/user_builds/cfme-tests/checkouts/master (from PKG-INFO)

[bogensberger]

We had the same problem, and since i did the wipe every second is passing the other fail with "An unexpected error occurred" (e.g.: https://readthedocs.org/projects/crate/builds/5034107/ ).

[agjohnson]

@bogensberger this looks separate. seems one of our build machines jumped to a readonly filesystem. Should be back to normal now.

@psav your project seems to have unrelated problems with it's dependencies

I wrapped up the last of the server changes yesterday evening, so all virtualenvs are now cleared and projects should all be using the new build image. If you notice any other problems, open a separate issue as it's likely unrelated.