Google-cloud-python: Invalid JWT Token when using Service Account JSON

[dstufft]

1. OS type and version

macOS Sierra running a Debian Jessie Docker Container

1. Python version and virtual environment information python --version

CPython 3.5.0, no virtual environment

1. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze

google-cloud-bigquery==0.23.0

1. Stacktrace if available

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/app/warehouse/celery.py", line 53, in __call__
    return super().__call__(pyramid_env["request"], *args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/celery/app/task.py", line 379, in __call__
    return self.run(*args, **kwargs)
  File "/app/warehouse/packaging/tasks.py", line 58, in compute_trending
    query.run()
  File "/usr/local/lib/python3.5/site-packages/google/cloud/bigquery/query.py", line 364, in run
    method='POST', path=path, data=self._build_resource())
  File "/usr/local/lib/python3.5/site-packages/google/cloud/_http.py", line 299, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/local/lib/python3.5/site-packages/google/cloud/_http.py", line 193, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/local/lib/python3.5/site-packages/google/cloud/_http.py", line 223, in _do_request
    body=data)
  File "/usr/local/lib/python3.5/site-packages/google_auth_httplib2.py", line 187, in request
    self._request, method, uri, request_headers)
  File "/usr/local/lib/python3.5/site-packages/google/auth/credentials.py", line 116, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.5/site-packages/google/oauth2/service_account.py", line 318, in refresh
    request, self._token_uri, assertion)
  File "/usr/local/lib/python3.5/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/local/lib/python3.5/site-packages/google/oauth2/_client.py", line 109, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/usr/local/lib/python3.5/site-packages/google/oauth2/_client.py", line 59, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{\n  "error" : "invalid_grant",\n  "error_description" : "Invalid JWT Signature."\n}')

1. Steps to reproduce

Try to query anything in BigQuery using a service account with "Viewer" permissions and GOOGLE_APPLICATION_CREDENTIALS pointed to a JSON file downloaded when creating the service account.

1. Code example

bq = bigquery.Client()
query = bq.run_sync_query(
        """ SELECT project,
                   IF(
                        STDDEV(downloads) > 0,
                        (todays_downloads - AVG(downloads))/STDDEV(downloads),
                        NULL
                    ) as zscore
            FROM (
                SELECT project,
                       date,
                       downloads,
                       FIRST_VALUE(downloads) OVER (
                            PARTITION BY project
                            ORDER BY DATE DESC
                            ROWS BETWEEN UNBOUNDED PRECEDING
                                AND UNBOUNDED FOLLOWING
                        ) as todays_downloads
                FROM (
                    SELECT file.project as project,
                           DATE(timestamp) AS date,
                           COUNT(*) as downloads
                    FROM `the-psf.pypi.downloads*`
                    WHERE _TABLE_SUFFIX BETWEEN
                        FORMAT_DATE(
                            "%Y%m%d",
                            DATE_ADD(CURRENT_DATE(), INTERVAL -31 day))
                        AND
                        FORMAT_DATE(
                            "%Y%m%d",
                            DATE_ADD(CURRENT_DATE(), INTERVAL -1 day))
                    GROUP BY file.project, date
                )
            )
            GROUP BY project, todays_downloads
            HAVING SUM(downloads) >= 5000
            ORDER BY zscore DESC
        """
    )
query.use_legacy_sql = False
query.run()

[dhermes]

First, thanks for all the work you do on packaging @dstufft!

@jonparrott Any ideas? @dstufft This probably belongs in https://github.com/GoogleCloudPlatform/google-auth-library-python/, but let's discuss here first.

I've seen an issue where an invalid grant was caused by the clock on the local machine being out of sync by 4+ hours with the real world, but even being wrong by 65 minutes (in either direction) would make one of the "issued at" and "expires" timestamps in the JWT be impossible.

[theacodes]

Yeah this is a weird one. I'll try to reproduce.

On Mon, Mar 6, 2017, 8:38 AM Danny Hermes [e-mail] wrote:

First, thanks for all the work you do on packaging @dstufft
https://github.com/dstufft!

@jonparrott https://github.com/jonparrott Any ideas? @dstufft
https://github.com/dstufft This probably belongs in
https://github.com/GoogleCloudPlatform/google-auth-library-python/, but
let's discuss here first.

I've seen an issue where an invalid grant was caused by the clock on the
local machine being out of sync by 4+ hours with the real world, but even
being wrong by 65 minutes (in either direction) would make one of the
"issued at" and "expires" timestamps in the JWT be impossible.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3100#issuecomment-284453228,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUcwogK28lENp134BjxTWz6c0nW4czks5rjDaOgaJpZM4MTr5e
.

[theacodes]

@dstufft I can't reproduce. :(

I can't really debug any further without asking you for sensitive information. Do you mind reaching out to me via email or hangouts? I'm jonwayne at google.com.

[dstufft]

Sure.

[theacodes]

As per email conversation with @dstufft, closing for now.

[hakanak]

Hello,
I have the same problem. I am developing an application with C #.
How did you find a solution?
Thank you.

[liangjacky]

Having the same problem as well on the java library.

trying to do
credential.refreshToken();
String token = credential.getRefreshToken();
and it fails at the first line, giving the exact same error.

Running from the console with the command:
gcloud auth application-default print-access-token
gives the same error as well.

[theacodes]

@liangjacky can you check your system clock? This is a very strange error and is difficult to reproduce.

[liangjacky]

@jonparrott what about my system clock? Is there a specific command I should run? Sorry for being so layman, but I think the time on my macbook is correct?

[theacodes]

Basically just ensuring it's not too far off. Are you getting the exact same error? The code you posted in your comment appears to be java/javascript not Python.

[liangjacky]

I just synced my time with time.apple.com and and it didn't seem to change anything. Thanks for the suggestion though. To answer your question, I'm getting the exact same error.

invalid_grant + invalid JWT signature. Authenticating using the application-default, and a json key on disk.

[liangjacky]

Sorry, it was an issue with the key I was using. Please ignore my comments.

[aliannejadi]

I am getting the same error. I am running the following code on jupyter notebook with the version:
Python 3.6.1 :: Anaconda 4.4.0 (x86_64)

I checked my system time is synced with Apple server. Here is the code:

from google.cloud import storage
import google.auth

# creds = compute_engine.Credentials()
credentials, project = google.auth.default()
client = storage.Client(credentials=credentials)
# client = storage.Client()
bucket = client.list_buckets()
print(bucket)

for b in bucket:
    print(b)

And here is the stack trace:

RefreshError                              Traceback (most recent call last)
<ipython-input-35-9982667b4ccb> in <module>()
----> 1 for b in bucket:
      2     print(b)

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/cloud/iterator.py in _items_iter(self)
    216     def _items_iter(self):
    217         """Iterator for each item returned."""
--> 218         for page in self._page_iter(increment=False):
    219             for item in page:
    220                 self.num_results += 1

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/cloud/iterator.py in _page_iter(self, increment)
    245         Yields :class:`Page` instances.
    246         """
--> 247         page = self._next_page()
    248         while page is not None:
    249             self.page_number += 1

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/cloud/iterator.py in _next_page(self)
    345         """
    346         if self._has_next_page():
--> 347             response = self._get_next_page_response()
    348             items = response.get(self._items_key, ())
    349             page = Page(self, items, self._item_to_value)

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/cloud/iterator.py in _get_next_page_response(self)
    394                 method=self._HTTP_METHOD,
    395                 path=self.path,
--> 396                 query_params=params)
    397         elif self._HTTP_METHOD == 'POST':
    398             return self.client._connection.api_request(

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/cloud/_http.py in api_request(self, method, path, query_params, data, content_type, headers, api_base_url, api_version, expect_json, _target_object)
    297         response, content = self._make_request(
    298             method=method, url=url, data=data, content_type=content_type,
--> 299             headers=headers, target_object=_target_object)
    300 
    301         if not 200 <= response.status < 300:

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/cloud/_http.py in _make_request(self, method, url, data, content_type, headers, target_object)
    191         headers['User-Agent'] = self.USER_AGENT
    192 
--> 193         return self._do_request(method, url, headers, data, target_object)
    194 
    195     def _do_request(self, method, url, headers, data,

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/cloud/_http.py in _do_request(self, method, url, headers, data, target_object)
    221         """
    222         return self.http.request(uri=url, method=method, headers=headers,
--> 223                                  body=data)
    224 
    225     def api_request(self, method, path, query_params=None,

/Users/MOLi/anaconda/lib/python3.6/site-packages/google_auth_httplib2.py in request(self, uri, method, body, headers, **kwargs)
    185 
    186         self.credentials.before_request(
--> 187             self._request, method, uri, request_headers)
    188 
    189         # Check if the body is a file-like stream, and if so, save the body

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/auth/credentials.py in before_request(self, request, method, url, headers)
    119         # the http request.)
    120         if not self.valid:
--> 121             self.refresh(request)
    122         self.apply(headers)
    123 

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/oauth2/service_account.py in refresh(self, request)
    308         assertion = self._make_authorization_grant_assertion()
    309         access_token, expiry, _ = _client.jwt_grant(
--> 310             request, self._token_uri, assertion)
    311         self.token = access_token
    312         self.expiry = expiry

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/oauth2/_client.py in jwt_grant(request, token_uri, assertion)
    141     }
    142 
--> 143     response_data = _token_endpoint_request(request, token_uri, body)
    144 
    145     try:

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/oauth2/_client.py in _token_endpoint_request(request, token_uri, body)
    107 
    108     if response.status != http_client.OK:
--> 109         _handle_error_response(response_body)
    110 
    111     response_data = json.loads(response_body)

/Users/MOLi/anaconda/lib/python3.6/site-packages/google/oauth2/_client.py in _handle_error_response(response_body)
     57 
     58     raise exceptions.RefreshError(
---> 59         error_details, response_body)
     60 
     61 

RefreshError: ('invalid_grant: Invalid JWT Signature.', '{\n  "error" : "invalid_grant",\n  "error_description" : "Invalid JWT Signature."\n}')

[foresmac]

@jonparrott I'm getting the same error today using google-cloud-pubsub==0.29.4 and after updating to google-cloud-pubsub==0.30.1. Tried generating a new private key JSON file for my service account.

google.auth.exceptions.RefreshError: 
('invalid_grant: Invalid JWT: Token must be a short-lived token (60 minutes) and in a reasonable 
timeframe. Check your iat and exp values and use a clock with skew to account for clock differences 
between systems.', '{\n  "error" : "invalid_grant",\n  "error_description" : "Invalid JWT: Token must be a 
short-lived token (60 minutes) and in a reasonable timeframe. Check your iat and exp values and use a 
clock with skew to account for clock differences between systems."\n}')

My local system time and my phone time appear to be correct (we are CT so currently UTC-06:00):

This is code that has run fine for at least a month until today. It runs inside a Docker container, so there may be some issue there with setting time that I'm just not aware of. Any ideas would be greatly appreciated.

[foresmac]

Ok, I found as issue with Docker for Mac https://github.com/docker/for-mac/issues/2076 (a couple related issues actually, but this one pretty clearly covers the problem). The time of the VM that runs containers drifts from system time, and doesn't appear to reset itself properly. Stopping and restarting Docker for Mac fixed the issue for me.

[versex]

Just as a note, I was having the same issue on a windows machine. I was looking at the time on the clock and it was correct, but I was still receiving the error. It ended up being the timezone. Someone had set the clock but they had not changed the timezone to EST.

[johnbalvin]

I had the same problem in another programming language(golang) but think this can work on this language too.

I had that error because I was using a credential file that was deleted on my project, so I just create another credential file, that that solved it

[gauravshimpi]

I am facing same issue in python3.

from dags.util.GCloudStorage import GCloudStorage

client = GCloudStorage(
"/home/gaurav/airflow/dags/script/gcp_credentials/my_gcs_credentials.json", ""project_name)

client.create_bucket("test_bucket")

===============================================================
Here's the GCloudStorage.py looks like:
from google.cloud import storage
from google.oauth2 import service_account
from google.api_core import exceptions

class GCloudStorage:

def __init__(self, credential_file_path, project_id):
    """
    :param credential_file_path: credential File for Authentication.
    :param project_id: project ID
    """
    self.CREDENTIAL_FILE_PATH = credential_file_path
    self.PROJECT_ID = project_id
    self.DATASET_ID = None

def create_connection(self):
    """
    Creates Connection with Google Cloud Storage and returns client.
    :return client: Google Cloud Storage Client
    """
    google_credentials = service_account.Credentials.from_service_account_file(self.CREDENTIAL_FILE_PATH)
    # Construct a BigQuery client object.
    client = storage.Client(project=self.PROJECT_ID, credentials=google_credentials)
    return client

def create_bucket(self, bucket_name):
    """
    Creates a new empty bucket.
    :param bucket_name: name of the bucket.
    :return responseMsg: success or other message.
    """
    # Instantiates a client
    storage_client = self.create_connection()
    response = None
    try:
        bucket = storage_client.create_bucket(bucket_name)
        print(bucket)
        response = "Bucket {} created".format(bucket.name)
    except exceptions.Conflict as error:
        response = "Bucket Already Exist: {}".format(error.code)
    return {"response": response}

def upload_blob(self, bucket_name, source_file_name, destination_blob_name):
    """
    Uploads a file to the bucket.
    :param bucket_name: bucketName
    :param source_file_name: Source File Name
    :param destination_blob_name: Destination Blob Name.
    :return responseMsg: Success or failed response message.
    """
    storage_client = self.create_connection()
    bucket = storage_client.get_bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)
    print("BlobName ---> ", blob)
    blob.upload_from_filename(source_file_name)
    response = "File {} uploaded to {}".format(source_file_name, destination_blob_name)
    print(response)
    return {"response": response}

def delete_bucket(self, bucket_name):
    """
    Deletes a bucket. The bucket must be empty.
    :param bucket_name: bucket Name.
    :return responseMsg: success or failed response message.
    """
    # Instantiates a client
    storage_client = self.create_connection()
    bucket = storage_client.get_bucket(bucket_name)
    bucket.delete()
    response = "Bucket {} deleted".format(bucket.name)

return {"response": response}

when i run the above create bucket code it gives me below error:

Traceback (most recent call last):
File "/home/gaurav/airflow/dags/util/test cloud.py", line 7, in
client.create_bucket("test_bucket")
File "/home/gaurav/airflow/dags/util/GCloudStorage.py", line 37, in create_bucket
bucket = storage_client.create_bucket(bucket_name)
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/cloud/storage/client.py", line 436, in create_bucket
_target_object=bucket,
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/cloud/_http.py", line 417, in api_request
timeout=timeout,
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/cloud/_http.py", line 275, in _make_request
method, url, headers, data, target_object, timeout=timeout
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/cloud/_http.py", line 313, in _do_request
url=url, method=method, headers=headers, data=data, timeout=timeout
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/auth/transport/requests.py", line 277, in request
self.credentials.before_request(auth_request, method, url, request_headers)
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/auth/credentials.py", line 124, in before_request
self.refresh(request)
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/oauth2/service_account.py", line 334, in refresh
access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion)
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/oauth2/_client.py", line 153, in jwt_grant
response_data = _token_endpoint_request(request, token_uri, body)
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
_handle_error_response(response_body)
File "/home/gaurav/airflow/venv/lib/python3.6/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{\n "error": "invalid_grant",\n "error_description": "Invalid JWT Signature."\n}')

Process finished with exit code 1

I am newbie to GCP, so any silly mistakes will be helpful to me.
Thank you in advance.

[webcat12345]

Really a good catch!!!
https://github.com/googleapis/google-cloud-python/issues/3100#issuecomment-323210685

With some reason my system clock was 2 hours ahead and setting it back to correct time, all worked well.

[NathanaelZondo]

It was the time that was causing this error. I took my battery out and the laptop lost track of time. Now after reading your comments I just updated the time and everything worked well. Thank you guys.

[ggabriel96]

Ok, I found as issue with Docker for Mac docker/for-mac#2076 (a couple related issues actually, but this one pretty clearly covers the problem). The time of the VM that runs containers drifts from system time, and doesn't appear to reset itself properly. Stopping and restarting Docker for Mac fixed the issue for me.

I was having the same problem here on Linux and restarting the Docker service solved it for me. Too bad I didn't check the time inside my container before restarting Docker.

Thanks a lot :heart:

[abhishekjiitr]

First, thanks for all the work you do on packaging @dstufft!

@jonparrott Any ideas? @dstufft This probably belongs in https://github.com/GoogleCloudPlatform/google-auth-library-python/, but let's discuss here first.

I've seen an issue where an invalid grant was caused by the clock on the local machine being out of sync by 4+ hours with the real world, but even being wrong by 65 minutes (in either direction) would make one of the "issued at" and "expires" timestamps in the JWT be impossible.

I was having the same problem while using GCS, thanks a bunch!
The time of individual machines had somehow drifted, syncing via NTP fixed it.

[j3pic]

I'm running into this issue, and I know for certain that the time being used by the SDK is the correct time. I extracted the exact JWT that was being posted and verified that the exp being posted was in the future. The iat timestamp was set a few minutes in the past.

[kenyerman]

I have this issue but my code is deployed as a cloud function. What should i do?