Pygithub: Recently random 401 Bad Credentials exceptions

[Borkason]

Hi,

PyGithub is a part of https://github.com/Borkason/google-code-issues-migrator, which is a fork by itself from an abandoned project, and recently I get random 401 Bad Credentials exceptions with that script. This exception did not show up in the past, that's why I think it might have to do with how PyGithub handles the Credentials. Maybe Github changed something in the past 1 or 2 months?

I could not find any pattern in the appearance of the exceptions, yet. I think it's random.

This is what I do
I authenticate once in __main__: https://github.com/Borkason/google-code-issues-migrator/blob/master/migrateissues.py#L493.
Then I start migrating all the issues. The script fetches all issues, does some things with the google code api and then creates new issues in github, including comments. Mostly during the issue creation I recieve the exceptions. Always at a random point (sometimes after adding 2 issues, sometimes after adding 100 issues ...

_I hope you can help in any way._

This is the exception

Traceback (most recent call last):
  File "migrateissues.py", line 529, in <module>
    process_gcode_issues(existing_issues)
  File "migrateissues.py", line 332, in process_gcode_issues
    add_comments_to_issue(github_issue, gid)
  File "migrateissues.py", line 236, in add_comments_to_issue
    existing_comments = [ comment.body for comment in github_issue.get_comments() ]
  File "/usr/local/lib/python2.7/dist-packages/PyGithub-1.12.2-py2.7.egg/github/PaginatedList.py", line 35, in __iter__
    newElements = self.__grow()
  File "/usr/local/lib/python2.7/dist-packages/PyGithub-1.12.2-py2.7.egg/github/PaginatedList.py", line 47, in __grow
    newElements = self._fetchNextPage()
  File "/usr/local/lib/python2.7/dist-packages/PyGithub-1.12.2-py2.7.egg/github/PaginatedList.py", line 104, in _fetchNextPage
    headers, data = self.__requester.requestJsonAndCheck("GET", self.__nextUrl, self.__nextParams, None)
  File "/usr/local/lib/python2.7/dist-packages/PyGithub-1.12.2-py2.7.egg/github/Requester.py", line 84, in requestJsonAndCheck
    return self.__check(*self.requestJson(verb, url, parameters, input))
  File "/usr/local/lib/python2.7/dist-packages/PyGithub-1.12.2-py2.7.egg/github/Requester.py", line 92, in __check
    raise GithubException.GithubException(status, output)
github.GithubException.GithubException: 401 {u'message': u'Bad credentials'}

[jacquev6]

Hello!

I've also experienced this issue in the past few days. It must be a bug on GIthub's side, it's not related to PyGithub. It can be reproduced with the following shell script (call it with your login and password as arguments):

#!/bin/sh

USER=$1
PASSWORD=$2

CONTINUE=1

while [ $CONTINUE == "1" ]
do
    OUTPUT=$(curl --include https://$USER:[email protected]/user 2>&1)

    if echo "$OUTPUT" | grep "200 OK" >/dev/null
    then
        echo OK
    else
        date
        echo "$OUTPUT"
        CONTINUE=0
    fi
done

It outputs a certain number of "OK"s and ends with the error message leading to the exception thrown by PyGithub:

[...]
OK
OK
OK
OK
Thu Mar 21 20:35:50 RST 2013
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    35  100    35    0     0     57      0 --:--:-- --:--:-- --:--:--    70HTTP/1.1 401 Unauthorized
Server: GitHub.com
Date: Thu, 21 Mar 2013 19:35:53 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 401 Unauthorized
X-GitHub-Media-Type: github.beta
Content-Length: 35

{
  "message": "Bad credentials"
}

You might want to submit a bug to Github directly.

[Borkason]

Yesterday I had no problems, today it's fully bugged again.

I did a quick and dirty hack to work around the Github issue in PaginatedList.py

def _fetchNextPage(self):
        myWorkaround = True
        while myWorkaround:
                try:
                        headers, data = self.__requester.requestJsonAndCheck("GET", self.__nextUrl, self.__nextParams, None)
                        myWorkaround = False
                except:
                        pass

        […]

[jacquev6]

Today I also experienced the issue on a simple script doing only a few (non-paginated) requests to the API. It is absolutely not critical for me, so I will not take the time to contact GitHub myself.

The issue is more general than pagination, so if you really need a workaround, you may prefer to do it in Requester.py:

Replace line 143

status, responseHeaders, output = self.__requestRaw(verb, url, requestHeaders, encoded_input)

by something like

status = 401
retries = 5
while retries > 0 and status == 401:
    status, responseHeaders, output = self.__requestRaw(verb, url, requestHeaders, encoded_input)
    retries -= 1

(I have not tested at all)

[Borkason]

Thanks for providing that snippet :+1:
I contacted Github, but they did not seem to be able to replicate that issue. But since I got a workaround, I won't invest any more time either :)

Cheers.