Readthedocs.org: ‘stable’ is built from branch instead of tag

[vfaronov]

Details

• Project URL: https://readthedocs.org/projects/httpolice/
• Build URL (if applicable): https://readthedocs.org/projects/httpolice/builds/6974496/
• Read the Docs username (if applicable): vfaronov

Expected Result

The ‘stable’ version on Read the Docs should be built from the latest version tag that is not pre-release. Currently, this is 0.7.0 (commit 0993a4f).

Actual Result

Instead, Read the Docs attempts to check out a branch named stable in the Git repo, which doesn’t exist and never did.

If I do push a stable branch to the repo, it is picked up, built and published correctly as the ‘stable’ version.

I used to have a problem which might be related: #2744 ‘stable’ was built from master instead of tag.

[stsewd]

I import your project on my local instance, the stable version point to the correct commit. But I can see on your versions list stable points to a branch...

Did you try wiping the environment and building again? (you can do this on versions, edit button).

[vfaronov]

Just tried; didn’t help: https://readthedocs.org/projects/httpolice/builds/6975726/

[vfaronov]

branch named stable in the Git repo, which doesn’t exist and never did

Correction: I did actually have a stable branch for at least one brief period in the past, but that was a long time ago, and there have since been successful builds of the ‘stable’ version from tag.

[stsewd]

Did you try doing the wipe on the latest version?

[stsewd]

Also, I going to see if I can replicate this by deleting a branch on my local instance, related to https://github.com/rtfd/readthedocs.org/issues/3714 https://github.com/rtfd/readthedocs.org/issues/3729 https://github.com/rtfd/readthedocs.org/issues/2032 https://github.com/rtfd/readthedocs.org/issues/3763

Note: The some users solve this by deleting the project and creating it again (the same name can be used)

[stsewd]

After playing around creating and deleting some branches/tags and resync the repo, I was able to reproduce the issue and ending with several unexisting versions and the stable version stuck on a previous commit.

My versions list

My repo

$ git show-ref        
c8f82e6559c78608a2d37c4d245e3e1923ff7a3d refs/heads/master
0797808b173d3d4a8188a37fc97fe8fc007863e1 refs/tags/2.0
c8f82e6559c78608a2d37c4d245e3e1923ff7a3d refs/tags/3.0
0af2b75f827bcc0a85bb093af8e3f86e88f0211e refs/tags/v01.1

Re-importing the same repo

Doesn't matter how many times wipe/disable/build the versions, the stable still stuck.

@vfaronov sorry, but I think this is a bug and the only solution, for now, is to recreate your project :/

@humitos was this behavior always present? I remember that the versions were deleted when the project was synchronized on the git clone step :thinking:

[vfaronov]

@stsewd Thank you for debugging. This is not a big issue for me — I’m unlikely to have a new stable release anytime soon, and then I can easily just create the stable branch — so I’ll live with it for now.

[humitos]

@humitos was this behavior always present? I remember that the versions were deleted when the project was synchronized on the git clone step thinking

I suppose that we never deleted versions. I think the only way to do this is by deleting the project and re-importing.

Then I found this line from 2013

https://github.com/rtfd/readthedocs.org/blob/7aa6f4d41acbc8b35b7f2b891496ae0ded9f68f8/readthedocs/api/base.py#L101

which now is always raising an exception because the _delete_versions method doesn't exist. So, it seems that at some point it was decided to not delete versions anymore and this line wasn't removed.

We should check the git log and understand why and what happened _and_ fix that bug :)

[humitos]

On the other hand, it seems there is an endpoint to delete non-existent versions in the repo: https://github.com/rtfd/readthedocs.org/blob/393e31ad3a9aafee297df64f1a654ffcda7ef04a/readthedocs/restapi/utils.py#L64-L85

[stsewd]

On the other hand, it seems there is an endpoint to delete non-existent versions in the repo:

Yeah, I think this is what I saw a time ago

[stsewd]

@humitos the _delete_versions was removed on a refactoring https://github.com/rtfd/readthedocs.org/commit/d11cc9f2a95054c5be3a53e9b47eb17d684f1d38#diff-aeef6dfbd37b07c9cc73482e8c2fb36eL95

[stsewd]

Well, I have a missing migration and that was giving an error when deleting the versions, but still are some versions remaining (and now my stable branch is updated) @humitos maybe there is something similar on the rtd servers?

[stsewd]

And as 2.0_a and 2.0_b has the same commit of 2.0 those aren't deleted. So the question is how they were created? Maybe something with my missing migration? I'm going to try to replicate the issue again.

[stsewd]

After applying the migrations I can't replicate the issue, but I learned that if some error happens here

https://github.com/rtfd/readthedocs.org/blob/393e31ad3a9aafee297df64f1a654ffcda7ef04a/readthedocs/restapi/views/model_views.py#L171

The database will end with inconsistencies, giving the above problems.

[humitos]

@stsewd I'm still a little confused here but I think that we are close to the solution.

Could you write a test case that makes this to fail? I will appreciate it :)

Removing that line of the api_utils.delete_versions will fix it and that test will pass?

[stsewd]

@humitos I will write the test, but all this is because if some problem happens while syncing the versions, the database ends with inconsistencies (like having to versions with the same commit). We can't rollback the db because this operation is executed from the API. So, I think a solution could be writing the code in such way that can detect and fix these inconsistencies while syncing the new versions.

[stsewd]

And I'm not sure why we are getting so many issues about this recently, maybe the servers (from the API) were down at this exact time?

[stsewd]

ok, I was able to replicate the issue with the repeated versions without raising any exceptions

• Create a tag and a branch with the same name (1.0)
• Build the docs on RTD
• Now we have 2 versions (1.0, 1.0_a)
• Recreate the tag with another commit (git tag -f 1.0)
• Build the docs on RTD
• Now we have 3 versions (1.0, 1.0_a, 1.0_b)

The stable version is still updated.
Doesn't matter if rebuild/wipe the versions, the duplicate versions still persist.
If I delete the branch, the duplicate tag still persists.
If I delete the tag (git tag -d 1.0), wiping and building remove the duplicate tag (both).

I haven't found a way for the stable version to get stuck (yet).

[stsewd]

I was able to replicate the issue about the stable versions, looks like it loses his machine created attribute at some point.

And it was more easy to replicate that the other one...

• Import the project
• Create a tag named stable
• Build the docs on RTD
• And we end with a stable version stuck with the commit of the previous tag (machine created == False)

[stsewd]

On the previous step, I had a tag 1.0. If the tag stable is the only created at the moment also works

[stsewd]

@vfaronov hey, the fix for this bug was already deployed, can you test if that really solves your problem? (you may need to wipe and rebuild another version).

You may need to do some additional steps if that doesn't work https://github.com/rtfd/readthedocs.org/pull/3913#issuecomment-389228733

[vfaronov]

@stsewd Thank you! It seems to work after the additional steps you mentioned.