Addons-server: GitError: old reference value does not match

[willdurand]

This is the second most frequent git error: https://sentry.prod.mozaws.net/operations/olympia-prod/issues/5526890/

GitError: old reference value does not match
(1 additional frame(s) were not displayed)
...
  File "celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "olympia/amo/decorators.py", line 108, in wrapper
    return f(*args, **kw)
  File "olympia/versions/tasks.py", line 130, in extract_version_to_git
    version=version, author=author, note=note)
  File "olympia/lib/git.py", line 343, in extract_and_commit_from_version
    branch=branch)
  File "olympia/lib/git.py", line 475, in _commit_through_worktree
    branch.set_target(commit.hex)

GitError: old reference value does not match

[willdurand]

See also #11431 (which is a tracker issue)

[willdurand]

So this comes from libgit2 and this is a very specific error that is only described here: https://github.com/libgit2/libgit2/pull/2042/files#r9511208

I managed to write a test case for it and I believe we can fix it. The idea would be to set the target on a more up-to-date branch by retrieving from the git repo instead of using the object passed to the _commit_through_worktree method.

[ioanarusiczki]

@willdurand If qa can test the issue, please add some STR.
Thanks!

[willdurand]

@ioanarusiczki it will be tricky to reproduce in -dev or stage I think. Maybe you could try to upload some versions of an add-on (or 2) and see if git-extraction works as intended.

[ioanarusiczki]

@willdurand

First test, I uploaded 9 versions one after the other (size of the addon is 5.73 MB) : https://reviewers.addons-dev.allizom.org/en-US/reviewers/review/noscriptcron1
In about 20 min all versions have been extracted so I encountered no errors.
The only thing I've noticed is that the admin queue doesn't really reflect immediately the state of the extractions meaning that from 9 entries only one showed the "In progress" green flag at page refresh and the other 8 entries kept being displayed after I could access all of them in code manager one by one without errors -> the queue is having a delay. I refreshed the page many time and then at one point they all disappeared.

The second test was an upload of 10 versions, for a 20MB addon (with many files):
https://reviewers.addons-dev.allizom.org/en-US/reviewers/review/ublocktest-origintest?page=1#history
The first 2 versions were extracted quite fast then 2 more versions were extracted.
After one hour the fifth version is still not extracted with all the following versions.

The queue looks like:

Once again the extracted versions, (from 1.22.7 to 1.23.0 ) are still visible in the queue even if I keep refreshing the queue it looks the same.

I tested this on dev with FF75(Win10).

[willdurand]

The only thing I've noticed is that the admin queue doesn't really reflect immediately the state of the extractions meaning that from 9 entries only one showed the "In progress" green flag at page refresh and the other 8 entries kept being displayed after I could access all of them in code manager one by one without errors -> the queue is having a delay. I refreshed the page many time and then at one point they all disappeared.

That's expected. We submit add-on IDs and not version IDs in this queue. The git-extraction task will take all the non-extracted versions into account for a given add-on and extract all of them. Each entry in this list gets processed by the task but when there is no version to extract, they are deleted directly (they should be green for a very short delay, though).

[willdurand]

The second test was an upload of 10 versions, for a 20MB addon (with many files):
reviewers.addons-dev.allizom.org/en-US/reviewers/review/ublocktest-origintest?page=1#history
The first 2 versions were extracted quite fast then 2 more versions were extracted.
After one hour the fifth version is still not extracted with all the following versions.

This means there was an error, which is likely due to a deployment related problem because I don't see any errors in the logs. Is it add-on 619668?

[ioanarusiczki]

@willdurand Yes, it is 619668. I still see them in the queue today.

[willdurand]

@willdurand Yes, it is 619668. I still see them in the queue today.

I just unlocked this add-on manually. The add-on being stuck was likely due to a new deployment happening in parallel. In -dev, this kills Celery workers and some of the existing tasks...

[ioanarusiczki]

@willdurand Yes, the queue is empty and all versions are available - they were extracted successfully.

So from qa point of view I can mark the issue verified.

One thing I forgot to mention is that for the addon with many versions extracted last week https://reviewers.addons-dev.allizom.org/en-US/reviewers/review/noscript?page=1#history
I don't receive a 404 but I could not see the content being loaded (I checked on the latest versions).

[willdurand]

I don't receive a 404 but I could not see the content being loaded (I checked on the latest versions).

Isn't it because of the validation/linter messages not being loaded?

[ioanarusiczki]

@willdurand yes, perhaps, because I see in the Network a 404 on https://addons-dev.allizom.org/en-US/developers/addon/noscript/file/245180/validation.json

[willdurand]

@willdurand yes, perhaps, because I see in the Network a 404 on addons-dev.allizom.org/en-US/developers/addon/noscript/file/245180/validation.json

Ok, so that's a problem with the linter messages, likely because the add-on is quite old. Not related to this issue.

[ioanarusiczki]

@willdurand Then perhaps I should file an issue in the codemanager repo. Old addon versions are available in prod but I can't tell if it works exactly as in dev. Still, maybe it's worth some further investigations.

[willdurand]

@willdurand Then perhaps I should file an issue in the codemanager repo. Old addon versions are available in prod but I can't tell if it works exactly as in dev. Still, maybe it's worth some further investigations.

Yes, maybe. I thought we had an issue for this problem already but maybe not.

[ioanarusiczki]

@willdurand Yes, the error was familiar and now I know why https://github.com/mozilla/addons-code-manager/issues/1436
I'll add a comment to it.