Getting-started-python: Import error on deploy only: No module named google.appengine.api

[aheyman11]

I've been following the tutorial, in particular section 6-pubsub. I have the project working locally just fine, but when I try to deploy, I get a timeout. See the stacktrace from the logs below. The import error is:

ImportError: No module named 'google.appengine.api'

and it occurs when I try to make a new logging Client. I've seen the same issue in another project when I try to make a new pubsub Client.

2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/home/vmagent/app/main.py", line 19, in <module>
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     app = bookshelf.create_app(config)
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/home/vmagent/app/bookshelf/__init__.py", line 40, in create_app
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     client = google.cloud.logging.Client(app.config['PROJECT_ID'])
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/env/lib/python3.4/site-packages/google/cloud/client.py", line 186, in __init__
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     Client.__init__(self, credentials=credentials, http=http)
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/env/lib/python3.4/site-packages/google/cloud/client.py", line 122, in __init__
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     credentials = get_credentials()
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/env/lib/python3.4/site-packages/google/cloud/credentials.py", line 87, in get_credentials
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     return client.GoogleCredentials.get_application_default()
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/env/lib/python3.4/site-packages/oauth2client/client.py", line 1288, in get_application_default
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     return GoogleCredentials._get_implicit_credentials()
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/env/lib/python3.4/site-packages/oauth2client/client.py", line 1273, in _get_implicit_credentials
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     credentials = checker()
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/env/lib/python3.4/site-packages/oauth2client/client.py", line 1191, in _implicit_credentials_from_gae
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     return _get_application_default_credential_GAE()
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/env/lib/python3.4/site-packages/oauth2client/client.py", line 1450, in _get_application_default_credential_GAE
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     from oauth2client.contrib.appengine import AppAssertionCredentials
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |   File "/env/lib/python3.4/site-packages/oauth2client/contrib/appengine.py", line 27, in <module>
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  |     from google.appengine.api import app_identity
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 worker.1  | ImportError: No module named 'google.appengine.api'
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 system    | worker.1 stopped (rc=1)
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 system    | sending SIGTERM to monitor.1 (pid 8)
2016-11-02 17:02:18 worker[20161102t123058]  17:02:18 system    | monitor.1 stopped (rc=-15)

[theacodes]

@aheyman11 wow, that should not happen. oauth2client is supposed to have import guards for this. Have you made any modifications?

[aheyman11]

Nope, no modifications at all to oauth2client or anything else in the sample code.

[theacodes]

Are you up for throwing some debug statements into your app?

import os
import sys
import oauth2client.client

print('oauth2client version', oauth2client.__version__)
print('SETTINGS.env_name 1', oauth2client.client.SETTINGS.env_name)
print('In GAE environment:', oauth2client.client._in_gae_environment())
print('SETTINGS.env_name 2', oauth2client.client.SETTINGS.env_name)
print('Server software', os.environ.get(oauth2client.client._SERVER_SOFTWARE, ''))
if 'google.appengine' in sys.modules:
    print('google.appengine ', sys.modules['google.appengine'].__file__)

[theacodes]

@aheyman11 hit submit too early - if you could, can you place those statements at the top of your main.py file and let me know what it spits out.

[aheyman11]

@jonparrott Sure, will do. Do you want to see the result after I run locally? Deploy the worker? Deploy the app?

[theacodes]

Not at the moment, since this is occurring in production I want to see what the live server says.

[theacodes]

@aheyman11 either should work, but if you're seeing this with the worker then do that one.

[aheyman11]

@jonparrott Sure thing, deploying the worker now

[aheyman11]

@jonparrott When I deployed the worker, I just got a timeout with nothing in the logs. When I deployed the app, I got the following output, followed by the same error.

2016-11-02 17:40:44 worker[20161102t132448]  17:40:44 worker.1  | oauth2client version 3.0.0
2016-11-02 17:40:44 worker[20161102t132448]  17:40:44 worker.1  | SETTINGS.env_name 1 None
2016-11-02 17:40:44 worker[20161102t132448]  17:40:44 worker.1  | In GAE environment: True
2016-11-02 17:40:44 worker[20161102t132448]  17:40:44 worker.1  | SETTINGS.env_name 2 GAE_PRODUCTION
2016-11-02 17:40:44 worker[20161102t132448]  17:40:44 worker.1  | Server software Google App Engine/1.9.46
2016-11-02 17:40:44 worker[20161102t132448]  17:40:44 worker.1  | google.appengine  /env/lib/python3.4/site-packages/google/appengine/__init__.py

[theacodes]

@aheyman11 can I see your app.yaml just for the sake of sanity?

[theacodes]

And your requirements.txt, as well, please.

[aheyman11]

@jonparrott Here you go (though I'm pretty sure they're the same as the sample code):

app.yaml:

``` .yaml

Copyright 2015 Google Inc.

#

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

#

http://www.apache.org/licenses/LICENSE-2.0

#

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

This file specifies your Python application's runtime configuration.

See https://cloud.google.com/appengine/docs/managed-vms/python/runtime

for details.

runtime: python
vm: true

[START entrypoint]

Instead of using gunicorn directly, we'll use Honcho. Honcho is a python port

of the Foreman process manager. For the default module, only the

frontend process is needed.

entrypoint: honcho start -f /app/procfile bookshelf

[END entrypoint]

runtime_config:
python_version: 3

beta_settings:
# If using Cloud SQL, uncomment and set this value to the Cloud SQL
# connection name, e.g.
# "project:region:cloudsql-instance"
# You must also update the values in config.py.
#
cloud_sql_instances: ""

requirements.txt:

``` .txt
Flask==0.11.1
google-cloud==0.19.0
gunicorn==19.6.0
oauth2client==3.0.0
Flask-SQLAlchemy==2.1
PyMySQL==0.7.9
Flask-PyMongo==0.4.1
PyMongo==3.3.0
six==1.10.0
requests[security]==2.11.1
honcho==0.7.1
psq==0.5.0

[theacodes]

Thank you. I'm trying to reproduce. This error is extremely bewildering.

[theacodes]

@aheyman11 can you give me the output of gcloud info? Feel free to elide your project ID and email.

[aheyman11]

@jonparrott From gcloud info:

Google Cloud SDK [133.0.0]

Platform: [Mac OS X, x86_64]
Python Version: [2.7.10 (default, Oct 23 2015, 19:19:21)  [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]]
Python Location: [/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python]
Site Packages: [Disabled]

Installation Root: [/Users/andreaheyman/Downloads/google-cloud-sdk]
Installed Components:
  core: [2016.11.01]
  core-nix: [2016.11.01]
  gcloud: []
  gsutil-nix: [4.22]
  gsutil: [4.22]
  bq: [2.0.24]
  bq-nix: [2.0.24]
System PATH: [/Users/andreaheyman/anaconda/envs/6-pubsub/bin:/Users/andreaheyman/Downloads/google-cloud-sdk/bin:/usr/local/mysql/bin:/Users/andreaheyman/anaconda/bin:/Users/andreaheyman/Library/Haskell/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin]
Cloud SDK on PATH: [True]

Installation Properties: [/Users/andreaheyman/Downloads/google-cloud-sdk/properties]
User Config Directory: [/Users/andreaheyman/.config/gcloud]
Active Configuration Name: [default]
Active Configuration Path: [/Users/andreaheyman/.config/gcloud/configurations/config_default]

Account: [******************]
Project: [******************]

Current Properties:
  [core]
    project: [******************]
    account: [******************]
    disable_usage_reporting: [False]

Logs Directory: [/Users/andreaheyman/.config/gcloud/logs]
Last Log File: [/Users/andreaheyman/.config/gcloud/logs/2016.11.02/13.40.52.650115.log]

[theacodes]

@aheyman11 thanks, I've tracked this down.

You should be able to add this to your requirements.txt and things (hopefully) should work:

googleapis-common-protos==1.3.5

[aheyman11]

@jonparrott Great, I'll try it out and let you know.

[aheyman11]

@jonparrott Yes, that resolved my issue! Thanks so much!

[theacodes]

@aheyman11 good to hear. FYI, a new release of googleapis-common-protos has been released so it should be safe to remove that from your requirements.txt

Thank you so much for answering all of my questions and helping me figure out this rather surprising bug. :)

[robincrlee]

@jonparrott

I ran into similar issue.

from oauth2client.contrib.appengine import AppAssertionCredentials

ImportError: No module named contrib.appengine

Then, I added googleapis-common-protos==1.3.5 in requirements.txt and solved the issue.

Is that still expected that I'd need to use common-protos to fix the bug?

[theacodes]

@robincrlee this shouldn't appear in any recent versions of any of these libraries. If you pip freeze before manually installing googleapis-common-protos, what version is installed?

[robincrlee]

@jonparrott

Here're what I have in requirements.txt

gsheets==0.1
Flask==0.12.1
sqlalchemy==1.1.11
gspread==0.6.2
google-auth==1.0.1
oauth2client==4.0.0
google-api-python-client
googleapis-common-protos==1.3.5

[MohammadHeydari]

import os
import sys
import oauth2client.client

print('oauth2client version', oauth2client.__version__)
print('SETTINGS.env_name 1', oauth2client.client.SETTINGS.env_name)
print('In GAE environment:', oauth2client.client._in_gae_environment())
print('SETTINGS.env_name 2', oauth2client.client.SETTINGS.env_name)
print('Server software', os.environ.get(oauth2client.client._SERVER_SOFTWARE, ''))
if 'google.appengine' in sys.modules:
    print('google.appengine ', sys.modules['google.appengine'].__file__)

Hi there.
here are my result:

oauth2client version 4.1.3
SETTINGS.env_name 1 None
In GAE environment: False
SETTINGS.env_name 2 None

I'm facing with no module named gae either.
Server software 

[SpeedyGonzalves]

Hi, I've got the same thing than @MohammadHeydari, and for the error its':
"No module named 'oauth2client.locked_file'" and "No module named 'google.appengine'"

[linsamtw]

@theacodes

please help me, i have same issue
No module named 'google.appengine'

my requirements.txt

Django==2.1.10
PyMySQL==0.9.3
numpy==1.17.0
plotly==4.0.0
pandas==0.25.0
requests==2.22.0

Flask==0.11.1
google-cloud==0.19.0
gunicorn==19.6.0
oauth2client==3.0.0
Flask-SQLAlchemy==2.1
Flask-PyMongo==0.4.1
PyMongo==3.3.0
six==1.10.0
honcho==0.7.1
psq==0.5.0
googleapis-common-protos==1.3.5

app.yam

# [START django_app]
runtime: python37

runtime_config:
  python_version: 3

handlers:
# This configures Google App Engine to serve the files in the app's static
# directory.
- url: /static
  static_dir: static/

# This handler routes all requests not caught above to your main app. It is
# required when static routes are defined, but can be omitted (along with
# the entire handlers section) when there are no static files defined.
- url: /.*
  script: auto
# [END django_app]

thank you