Google-cloud-python: Datastore: Cannot connect the local datastore emulator with a Python 3 GAE app

[ramuta]

Hi,

I'm trying to connect a localhost GAE Python 3 web app with a localhost Datastore Emulator, but I'm receiving the credentials error:

google.api_core.exceptions.Unauthenticated: 401 Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential.

The Datastore Emulator is successfully running which I can confirm by going to the emulator localhost port (8081) via a browser and receiving the "Ok" response.

The env vars are set up via the env.yaml file, although there should be an alternative way to do it. The datastore.Client class should allow adding the Datastore Emulator path and thus avoiding the need of having env vars at all.

The issue is similar to this one for the BigQuery: https://github.com/googleapis/google-cloud-python/issues/6287

I'm using the Mock library for mock credentials as defined here and also used in the Google API Python client here.

Environment details

1. Datastore Emulator
2. MacOS
3. Python version: Python 3.7.1
4. google-api-core==1.14.2
   google-auth==1.6.3
   google-cloud-core==0.28.1
   google-cloud-datastore==1.7.0

Steps to reproduce

1. Run the Datastore emulator: gcloud beta emulators datastore start --no-legacy --data-dir=. --project test --host-port "127.0.0.1:8001"
2. Run the app code defined below with python main.py.

Code example

main.py:

import mock
from flask import Flask, render_template
from google.cloud import datastore
import google.auth.credentials

app = Flask(__name__)

credentials = mock.Mock(spec=google.auth.credentials.Credentials)
db = datastore.Client(project="test", credentials=credentials)


@app.route("/")
def index():
    # add object to db
    entity = datastore.Entity(key=db.key("Message", 1234))
    message = {"message": "hello world"}
    entity.update(message)
    db.put(entity)

    return render_template("index.html")


@app.route("/message")
def message():
    # query from db
    obj = db.get(db.key("Message", 1234))
    print(obj)

    return "Message for you: {}".format(obj["message"])


if __name__ == '__main__':
    app.run(port=8080, host="127.0.0.1")

env.yaml:

DATASTORE_DATASET: test
DATASTORE_EMULATOR_HOST: 127.0.0.1:8001
DATASTORE_EMULATOR_HOST_PATH: 127.0.0.1:8001/datastore
DATASTORE_HOST: http://127.0.0.1:8001
DATASTORE_PROJECT_ID: test

app.yaml:

runtime: python37

handlers:
- url: /static
  static_dir: static
- url: /.*
  script: auto

requirements.txt

google-cloud-datastore==1.7.0
mock
google-auth

Stack trace

 * Serving Flask app "main" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:8080/ (Press CTRL+C to quit)
[2019-08-24 10:01:44,053] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "venv/gae2-mock-example/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "venv/gae2-mock-example/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
    status = StatusCode.UNAUTHENTICATED
    details = "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."
    debug_error_string = "{"created":"@1566633704.052212000","description":"Error received from peer ipv4:172.217.16.106:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.","grpc_status":16}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "venv/gae2-mock-example/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "venv/gae2-mock-example/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "venv/gae2-mock-example/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "venv/gae2-mock-example/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "venv/gae2-mock-example/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "venv/gae2-mock-example/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/user/programming/gae2-mock-example/main.py", line 18, in index
    db.put(entity)
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/cloud/datastore/client.py", line 404, in put
    self.put_multi(entities=[entity])
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/cloud/datastore/client.py", line 431, in put_multi
    current.commit()
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/cloud/datastore/batch.py", line 273, in commit
    self._commit()
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/cloud/datastore/batch.py", line 249, in _commit
    self.project, mode, self._mutations, transaction=self._id)
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/cloud/datastore_v1/gapic/datastore_client.py", line 426, in commit
    request, retry=retry, timeout=timeout, metadata=metadata)
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/venv/gae2-mock-example/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "venv/gae2-mock-example/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.Unauthenticated: 401 Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
127.0.0.1 - - [24/Aug/2019 10:01:44] "GET / HTTP/1.1" 500 -

[alex-fer]

I had the same problem.
I solved it setting the
client.base_url = "http://localhost:8081"

The url is the DATASTORE_HOST variable returned in the emulator initialization

[HemangChothani]

@ramuta I have tested it , The problem is variables which you have defined in env.yaml file that you need set as a environment environment variables for that you need to parse your file and use os.environ[DATASTORE_HOST]=localhost:8081 or you need to use export DATASTORE_HOST=localhost:8081 or set define in/etc/environment.
file .

[ramuta]

Great, thanks guys! 🙂

I added this chunk of code just above the app = Flask(__name__) line:

import os

os.environ["DATASTORE_DATASET"] = "test"
os.environ["DATASTORE_EMULATOR_HOST"] = "127.0.0.1:8001"
os.environ["DATASTORE_EMULATOR_HOST_PATH"] = "127.0.0.1:8001/datastore"
os.environ["DATASTORE_HOST"] = "http://127.0.0.1:8001"
os.environ["DATASTORE_PROJECT_ID"] = "test"

I'll also create a blog post on this so in case someone else stumbles upon the same problem, they can know how to solve it. But it would be also great if Google would add a notice about this in their docs.