Readthedocs.org: API to get active versions is slow, when injecting the sphinx context

[edi9999]

Details

• Read the Docs project URL: https://docxtemplater.readthedocs.io/en/latest/goals.html
• Build URL (if applicable): https://readthedocs.org/api/v2/build/10386322.txt
• Read the Docs username (if applicable): edi9999

Expected Result

I expect the build to succeed (or to fail with a clearer error message)

Actual Result

The build stopped after 40 seconds, just after doing following command :

/home/docs/checkouts/readthedocs.org/user_builds/docxtemplater/envs/latest/bin/python -m pip install --upgrade --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/docxtemplater/.cache/pip Pygments==2.3.1 setuptools==41.0.1 docutils==0.14 mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<1.1

I think this might be related to https://github.com/readthedocs/readthedocs.org/issues/6687, but I would like to keep older versions available in the doc if possible.

Also, if it can help, the last successful build for my project was on 2020-01-31T10:34:03.130141Z : https://readthedocs.org/api/v2/build/10360658.txt and the first failing build was on 2020-02-05T08:40:45.500491Z https://readthedocs.org/api/v2/build/10386322.txt.

The logs inside both these builds seem to be quite different, for example, the successful build does the following :

git remote set-url origin https://github.com/open-xml-templating/docxtemplater.git
git fetch origin --force --tags --prune --prune-tags --depth 50
git checkout --force origin/master

The failing build does the following :

git clone --no-single-branch --depth 50 https://github.com/open-xml-templating/docxtemplater.git .
git checkout --force origin/master
git clean -d -f -f

So I suspect there might have been a big change in readthedocs between those two dates (or something else happened which I don't understand).

[stsewd]

The logs inside both these builds seem to be quite different, for example, the successful build does the following

That isn't related, the first command is running a fresh clone, the second command is re-using a repo already on disk.

Also, thanks for the dates, we may have introduced something that makes the db operations slower or maybe we are just under high demand.

Also, I guess between those dates the number of versions of your project didn't increase? If they did, then the problem was just waiting to happen.

[edi9999]

Yes, there has been one version (new git tag) pushed between the failing and the succeeding build

[edi9999]

I have quite a few versions now indeed (146 in total), but I would expect readthedocs to work even with thousands of versions.

[stsewd]

Again, thanks for the dates. I think the problem is https://github.com/readthedocs/readthedocs.org/pull/6602/ (that was released on February 4th https://docs.readthedocs.io/en/stable/changelog.html?#version-3-11-6)

So, our API is taking more than 5 seconds to connect and retrieve the data from the server. Hitting this code

https://github.com/readthedocs/readthedocs.org/blob/2e100541ff5fdda1e0ddd213f499f14e2832b8b6/readthedocs/doc_builder/backends/sphinx.py#L124-L132

[stsewd]

Some quick things we can do are:

1) Add an option to the api to only return the fields we need (project is basically being returned for each version, and it's the same).
2) Don't hit that api at all. We inject these versions in the sphinx context and put them in the footer. But online docs use the footer api to get all the versions, so those are replaced.

If we do 2, we are breaking backwards compatibility. Some users may be using the context, we don't have an easy way of checking this. We could add a feature flag in case a user is using it.

[stsewd]

@edi9999 your project should be passing now. We still need to fix the real problem, but your project won't fail now.

https://readthedocs.org/projects/docxtemplater/builds/10552321/

[edi9999]

Thanks, I confirm that it works now !

[edi9999]

I am reopening since you say that the real problem is still not fixed, feel free to close it as for me all is good.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stsewd]

So, we do have some users that are using our context https://github.com/readthedocs/readthedocs.org/issues/6977 (downloads only)

[mgeier]

Apart from downloads, it would also be great if you could keep github_user, github_repo, display_github and the same thing for gitlab and bitbucket. And commit and conf_py_path, please.
This way, it is possible to create "Show Source" links that automagically link to the relevant Github/GitLab/Bitbucket repo.

And of course READTHEDOCS is very helpful, but this is just a constant, so it shouldn't take any time to obtain that information.