Google-cloud-python: Unable to connect to datastore emulator

Created on 5 Sep 2017  路  7Comments  路  Source: googleapis/google-cloud-python

Hi, I've been trying to use the datastore emulator without success, here are the steps I've used:

  1. Start the datastore
$ gcloud beta emulators datastore start
WARNING: Reusing existing data in [/Users/patrick/.config/gcloud/emulators/datastore].
Executing: /Users/patrick/Documents/google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator start --host=::1 --port=8822 --store_on_disk=True --consistency=0.9 --allow_remote_shutdown /Users/patrick/.config/gcloud/emulators/datastore
[datastore] Sep 05, 2017 9:55:20 PM com.google.cloud.datastore.emulator.CloudDatastore$FakeDatastoreAction$7 apply
[datastore] INFO: Provided --allow_remote_shutdown to start command which is no longer necessary.
[datastore] Sep 05, 2017 9:55:20 PM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub <init>
[datastore] INFO: Local Datastore initialized:
[datastore]     Type: High Replication
[datastore]     Storage: /Users/patrick/.config/gcloud/emulators/datastore/WEB-INF/appengine-generated/local_db.bin
[datastore] Sep 05, 2017 9:55:20 PM io.grpc.internal.ManagedChannelImpl <init>
[datastore] INFO: [ManagedChannelImpl@cb0ed20] Created with target [::1]:8822
[datastore] Sep 05, 2017 9:55:20 PM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub load
[datastore] INFO: The backing store, /Users/patrick/.config/gcloud/emulators/datastore/WEB-INF/appengine-generated/local_db.bin, does not exist. It will be created.
[datastore] Sep 05, 2017 9:55:20 PM io.gapi.emulators.netty.NettyUtil applyJava7LongHostnameWorkaround
[datastore] INFO: Unable to apply Java 7 long hostname workaround.
[datastore] API endpoint: http://::1:8822
[datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:
[datastore]
[datastore]   export DATASTORE_EMULATOR_HOST=::1:8822
[datastore]
[datastore] Dev App Server is now running.
````
2. setting the environment variables

$ gcloud beta emulators datastore env-init
export DATASTORE_DATASET=blue-179012
export DATASTORE_EMULATOR_HOST=::1:8822
export DATASTORE_EMULATOR_HOST_PATH=::1:8822/datastore
export DATASTORE_HOST=http://::1:8822
export DATASTORE_PROJECT_ID=blue-179012

Then

$ $(gcloud beta emulators datastore env-init)

3. Run a simple query:

```python
from google.cloud import datastore
client = datastore.Client()
pdf = datastore.Entity(client.key('PDF'))
pdf.update({ 'path': 'asd' })
client.put(pdf)
  1. Error:
>>> client.put(pdf)
Traceback (most recent call last):
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/retry.py", line 121, in inner
    return to_call(*args)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/retry.py", line 68, in inner
    return a_func(*updated_args, **kwargs)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/grpc/_channel.py", line 492, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded)>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/_gax.py", line 74, in _catch_remap_gax_error
    yield
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/_gax.py", line 173, in commit
    return super(GAPICDatastoreAPI, self).commit(*args, **kwargs)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 345, in commit
    return self._commit(request, options)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/api_callable.py", line 452, in inner
    return api_caller(api_call, this_settings, request)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/api_callable.py", line 438, in base_caller
    return api_call(*args)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/api_callable.py", line 376, in inner
    return a_func(*args, **kwargs)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/retry.py", line 127, in inner
    ' classified as transient', exception)
google.gax.errors.RetryError: RetryError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded)>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/client.py", line 384, in put
    self.put_multi(entities=[entity])
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/client.py", line 411, in put_multi
    current.commit()
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/batch.py", line 273, in commit
    self._commit()
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/batch.py", line 249, in _commit
    self.project, mode, self._mutations, transaction=self._id)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/_gax.py", line 173, in commit
    return super(GAPICDatastoreAPI, self).commit(*args, **kwargs)
  File "/Users/patrick/.pyenv/versions/3.6.0/lib/python3.6/contextlib.py", line 100, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/_gax.py", line 82, in _catch_remap_gax_error
    six.reraise(error_class, new_exc, sys.exc_info()[2])
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/_gax.py", line 74, in _catch_remap_gax_error
    yield
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/datastore/_gax.py", line 173, in commit
    return super(GAPICDatastoreAPI, self).commit(*args, **kwargs)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 345, in commit
    return self._commit(request, options)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/api_callable.py", line 452, in inner
    return api_caller(api_call, this_settings, request)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/api_callable.py", line 438, in base_caller
    return api_call(*args)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/api_callable.py", line 376, in inner
    return a_func(*args, **kwargs)
  File "/Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages/google/gax/retry.py", line 127, in inner
    ' classified as transient', exception)
google.cloud.exceptions.GatewayTimeout: 504 Deadline Exceeded
>>>

Not sure if I'm doing something wrong or not. I've tried to update all the google cloud sdk components.
below are my system information.

os: macOS Sierra
python: python 3.6

$ pip show google-cloud
Name: google-cloud
Version: 0.27.0
Summary: API Client library for Google Cloud
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google Cloud Platform
Author-email: [email protected]
License: Apache 2.0
Location: /Users/patrick/.virtualenvs/blue/lib/python3.6/site-packages
Requires: google-cloud-pubsub, google-cloud-bigquery, google-cloud-dns, google-cloud-vision, google-cloud-runtimeconfig, google-cloud-language, google-cloud-spanner, google-cloud-core, google-cloud-translate, google-cloud-logging, google-cloud-storage, google-cloud-videointelligence, google-cloud-datastore, google-cloud-resource-manager, google-cloud-speech, google-cloud-bigtable, google-cloud-monitoring, google-cloud-error-reporting
bug datastore testing p2
Source
picture patrick91
👍1

Most helpful comment

It is probably a day or two of work; I am supportive in principle. :-)

picture lukesneeringer on 6 Sep 2017
👍4

All 7 comments

@patrick91 You'll want to use a custom credentials object in the constructor. I'm short on time right now, but have a look at https://github.com/GoogleCloudPlatform/google-cloud-python/blob/3caed41b88eb58673ee5c3396afa3f8fff97d4d4/datastore/tests/system/test_system.py#L53-L64 for an idea of how to do this.

The reason: by default your credentials will try to get a new token, but you don't need to do auth in the emulator.

dhermes picture dhermes on 5 Sep 2017

@dhermes thanks for the quick reply, unfortunately (unless I made a mistake) I'm getting the same error, this is the code I'm running now:

import os

import requests

import google.auth.credentials
from google.cloud import datastore


class EmulatorCreds(google.auth.credentials.Credentials):
    """A mock credential object.
    Used to avoid unnecessary token refreshing or reliance on the network
    while an emulator is running.
    """

    def __init__(self):  # pylint: disable=super-init-not-called
        self.token = b'seekrit'
        self.expiry = None

    @property
    def valid(self):
        """Would-be validity check of the credentials.
        Always is :data:`True`.
        """
        return True

    def refresh(self, unused_request):  # pylint: disable=unused-argument
        """Off-limits implementation for abstract method."""
        raise RuntimeError('Should never be refreshed.')


dataset = os.getenv('DATASTORE_DATASET')

assert dataset

credentials = EmulatorCreds()
http = requests.Session()

client = datastore.Client(
    project=dataset,
    namespace='ns_test',
    credentials=credentials,
    _http=http
)

pdf = datastore.Entity(client.key('PDF'))
pdf.update({'path': 'asd'})
client.put(pdf)

PS. here's a screen to double check the env configuration: http://s.patrick.wtf/tDbkbPgJxC

patrick91 picture patrick91 on 5 Sep 2017

Thanks for the update. It's possible that our emulator support has stopped working, we don't verify it on CI and I personally haven't tested it in some time.

@lukesneeringer How difficult would it be to get the gcloud SDK added to our Docker images so that we can start testing emulator support on CircleCI?

dhermes picture dhermes on 6 Sep 2017
👍2 ❤1

It is probably a day or two of work; I am supportive in principle. :-)

lukesneeringer picture lukesneeringer on 6 Sep 2017
👍4

Closing this for now in favor of #4840.

theacodes picture theacodes on 22 Feb 2018

Do I understand correct, that currently datastore emulator is not usable with Python client? Is that a Python-specific issue?

bereal picture bereal on 18 May 2018

The issue is still present in the latest SDK. Described it here: https://github.com/googleapis/google-cloud-python/issues/5738#issuecomment-468336989.

cmin764 picture cmin764 on 28 Feb 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

LMeinhardt picture LMeinhardt  路  4Comments
pongad picture pongad  路  4Comments
vrcs picture vrcs  路  3Comments
rmceoin picture rmceoin  路  3Comments
jlara310 picture jlara310  路  3Comments