Getting-started-python: ImportError: No module named appengine.api (even after app-engine-python installation)

[MTHD486]

Hi,

I used a newly created CentOS7 VM in Google Cloud Platform to try to run the following Python code (Python version: 2.7.5):-

from google.appengine.api import urlfetch

But I got this error message:-

ImportError: No module named appengine.api

Previously, I already have already installed app-engine-python and app-engine-python-extras in gcloud. For example, if I now run this:-

yum install google-cloud-sdk-app-engine-python

I would get this to tell me that it is already installed:-

Package google-cloud-sdk-app-engine-python-194.0.0-1.noarch already installed and latest version Nothing to do

And if I now run this:-

gcloud components list

Again, I would get this to tell me that the app-engine components are already installed:-

┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┐
│                                                   Components                                                   
│
├──────────────────┬──────────────────────────────────────────────────────┬──────────────────────────┬───────────
┤
│      Status      │                         Name                         │            ID            │    Size   
│
├──────────────────┼──────────────────────────────────────────────────────┼──────────────────────────┼───────────
(...............................................................................)
(skipped showing other gcloud components)
(...............................................................................)
│ Installed        │ gcloud app Python Extensions                         │ app-engine-python        │   6.2 MiB 
│
│ Installed        │ gcloud app Python Extensions (Extra Libraries)       │ app-engine-python-extras │  27.8 MiB 
│
└──────────────────┴──────────────────────────────────────────────────────┴──────────────────────────┴───────────

I tried to download pip and use pip to install the stuff again, and I if I run pip freeze | grep ^google I would get this (without showing app-engine ?):-

google-api-core==0.1.4
google-auth==1.4.1
google-cloud-bigquery==0.28.0
google-cloud-bigquery-datatransfer==0.1.1
google-cloud-container==0.1.1
google-cloud-core==0.28.1
google-cloud-datastore==1.4.0
google-cloud-dns==0.28.0
google-cloud-firestore==0.28.0
google-cloud-language==1.0.1
google-cloud-monitoring==0.28.1
google-cloud-speech==0.30.0
google-cloud-trace==0.17.0
google-cloud-translate==1.3.1
google-cloud-vision==0.29.0
google-compute-engine==2.7.6
google-gax==0.15.16
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3

How can I get it solved? I just want to run from google.appengine.api import urlfetch successfully.

Thanks in advance!

[maxpaynestory]

Same problem here, i get error

No module named 'google.appengine'

[pronghorndev]

I have the same issue and even tried reinstalling everything and it didn't work. Did you figure it out?

[maximelebastard]

Same issue

[gogasca]

Same issue in MacOS using virtual environment
MacOS Sierra 10.13.6 (17G65)
Python 2.7

[mhdtouban]

You need to add the following to your ~/.bash_profile file
export GOOGLE_APP_ENGINE_DIR=/usr/local/google_appengine

[evanfir]

You need to add the following to your ~/.bash_profile file

export GOOGLE_APP_ENGINE_DIR=/usr/local/google_appengine

I tried this and I'm still getting the same error.
I don't use virtual environment.
MacOS Sierra 10.12.6 (16G1510)
Python 3.7.0

[mhdtouban]

Try to use miniconda conda.io/miniconda

On Thu, Sep 27, 2018, 9:57 PM Evan Fir [e-mail] wrote:

You need to add the following to your ~/.bash_profile file

export GOOGLE_APP_ENGINE_DIR=/usr/local/google_appengine

I tried this and I'm still getting the same error.
I don't use virtual environment.
MacOS Sierra 10.12.6 (16G1510)
Python 3.7.0

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/getting-started-python/issues/157#issuecomment-425305389,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACYvDw2hVWlB_1YQQ0ZoISz01oGTVKggks5ufZAkgaJpZM4S7ocQ
.

[karthigb]

Same problem here. People have been reporting this on Stackoverflow for a long time now. We should not have to install these packages in the magic order to get them working. Especially since there are so many different official guides on install the Cloud SDK.

[leonardoleenen]

I guys, I got the Same Issue but there is a clue: I have been upgrading my GCloud version (because I have been my OS to Mojave). So, all my artefacts have worked fine with an older version (gcloud and O.S). With new version of GCloud and Mojave I have this issue. Maybe I got a solution (monkey patch)... If I have success, came back ASAP.

[karthigb]

I got it working on my Ubuntu VM. You need to add the following line to your bashrc file.

export PYTHONPATH="${PYTHONPATH}:/PATH_TO_GC_SDK/google-cloud-sdk/platform/google_appengine

Don't forget to enter source ~/.bashrc afterwards.

You can verify whether the python interpreter can reference the package by importing _sys_, then printing sys.path. You should see the directory above listed in your path.

[brachipa]

I see the directory in sys path, but still, get the error when import.

[mfumagalli68]

Same exact problem. A nightmare

[StayFoolisj]

Same problem. Can't find a solution

[mfumagalli68]

For what it may be worth.
If I add to my PYTHONPATH this path:

export PYTHONPATH="${PYTHONPATH}:/PATH_TO_GC_SDK/google-cloud-sdk/platform/google_appengine/google

it works. Without the last google folder it doesn't work.

[SalamiArmy]

Sorry mfumagalli68, on Windows, where is the google_appengine folder? I can't find it after installing The Python Cloud SDK™. The furthest down your path I can get is E:\Google Cloud SDK\google-cloud-sdk\platform ??

[mmarquezv]

Same problem here export PYTHONPATH="${PYTHONPATH}:/PATH_TO_GC_SDK/google-cloud-sdk/platform/google_appengine/google" does not work

[andreshg112]

I'm totally frustrated with this error. I cannot install using pip, pip3, I cannot create a new virtual environment, the project is not working, I'm getting: ModuleNotFoundError: No module named 'google.appengine'

[wanquanw]

Same problem, tried adding PATH mentioned above but none of them work.
macOS Catalina
python 3.7.4

[tbro2020]

Same problem

[scythargon]

+1

[hmiguel]

Same here. :(

[gokul0801]

Tried different things.. Changing the PYTHONPATH , PATH, using virtualenv... nothing worked.

I removed app-engine-python installation and reinstalled and it worked.

sudo apt-get remove google-cloud-sdk-app-engine-python
sudo apt-get install google-cloud-sdk-app-engine-python

import google
print google.__path__

['/usr/lib/google-cloud-sdk/platform/google_appengine/google']

now i can import any module in google.appengine

from google.appengine.datastore import datastore_stub_util
from google.appengine.api import urlfetch

[ThiagoMusico]

Having the same error here, for me it seems the problem has to do with the fact that the site-packages folder already contains a folder named google, which was created when I installed google datastore package using pip.
python 2.7.17

[michaelawyu]

Hi everyone!

In most cases Cloud SDK or the package manager on your system installs packages app-engine-python and app-engine-python-extras inside Cloud SDK itself, i.e. if your Cloud SDK resides in /usr/lib64/google-cloud-sdk/, the packages will only be available in /usr/lib64/google-cloud-sdk/platform/google_appengine. Cloud SDK will NOT add this directory to your PYTHONPATH, thus Python will not search this location for said packages and will return the errors reported.

To fix this issue, simply add YOUR-CLOUD-SDK-INSTALLATION/platform/google_appengine to your PYTHONPATH. More specifically, run the command below:

export PYTHONPATH=${PYTHONPATH}:$(gcloud info --format="value(installation.sdk_root)")/platform/google_appengine

Additionally, note that Python will only search the first matched directory in the given paths without further instructions. For example, if you imported both google.mypackage and google.appengine.api in your script and these two packages live in two separate places, Python will report an error for either of them as it cannot find both in the same place. This should not be an issue for most Google Cloud Client Libraries, since their __init__.py is specified with __path__ = pkgutil.extend_path(__path__, __name__), which extends the path and tells Python to continue searching in the case of a mismatch. Should your error persists despite correct PYTHONPATH settings, check your site-packages folder and make sure that the __init__.py script under the google folder is corrected set.

michaelawyu

[linhnt2803]

I know this issue may be resolved in many ways. but with support from this article here, I encapsulate my solution to adding something like this to __init__.py file. Hope this helps!

import google
import os

# This fix issue with Google Cloud SDK - ImportError: No module named google.appengine
def get_sdk_appengine_path():
    command = 'gcloud info --format="value(installation.sdk_root)"'
    stream = os.popen(command)
    gae_root_path = stream.read().strip()
    if not bool(gae_root_path) or not gae_root_path.find('google') >= 0:
        raise Exception('Make sure App Engine APK is available!')
    return '{}/platform/google_appengine/google'.format(gae_root_path)

try:
    import google.appengine
except:
    # this value is something like 'C:\User\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google'
    gae_path = get_sdk_appengine_path()
    google.__path__.append(gae_path)

[NickFranceschina]

So for me (and I believe also @ThiagoMusico ) this is an issue inside of the Google Cloud Shell. So here are the steps to explain it clearly:

Go open a brand-spanking-new empty Google Shell:
https://shell.cloud.google.com/

type @michaelawyu 's command line here to see the path app engine SDK is installed at:

someuser@cloudshell:~$ echo $(gcloud info --format="value(installation.sdk_root)")/platform/google_appengine
/usr/lib/google-cloud-sdk/platform/google_appengine

NOTE that the SDK has a google_appengine/google folder... containing an appengine package:

someuser@cloudshell:~$ ls -la /usr/lib/google-cloud-sdk/platform/google_appengine/google
total 32
drwxr-xr-x  7 root root 4096 Nov 18 02:53 .
drwxr-xr-x  1 root root 4096 Nov 18 02:53 ..
drwxr-xr-x 12 root root 4096 Nov 18 02:53 appengine
-rw-r--r--  1 root root  601 Jan  1  1980 __init__.py
drwxr-xr-x  5 root root 4096 Nov 18 02:53 net
drwxr-xr-x  3 root root 4096 Nov 18 02:53 protobuf
drwxr-xr-x  2 root root 4096 Nov 18 02:54 __pycache__
drwxr-xr-x  3 root root 4096 Nov 18 02:53 pyglib

find the python site-packages location:

someuser@cloudshell:~$ python -c 'import site; print site.getsitepackages()'
['/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

NOTE: that one of those directories also has a google folder with _different_ packages:

nfranceschina@cloudshell:~$ ls -la /usr/local/lib/python2.7/dist-packages/google
total 60
drwxr-sr-x 15 root staff 4096 Nov 18 03:12 .
drwxrwsr-x  1 root staff 4096 Nov 18 03:40 ..
drwxr-sr-x  2 root staff 4096 Nov 18 03:11 api
drwxr-sr-x  5 root staff 4096 Nov 18 03:11 api_core
drwxr-sr-x  3 root staff 4096 Nov 18 03:11 _async_resumable_media
drwxr-sr-x  5 root staff 4096 Nov 18 03:11 auth
drwxr-sr-x 29 root staff 4096 Nov 18 03:12 cloud
drwxr-sr-x  3 root staff 4096 Nov 18 03:12 iam
drwxr-sr-x  3 root staff 4096 Nov 18 03:11 logging
drwxr-sr-x  2 root staff 4096 Nov 18 03:11 longrunning
drwxr-sr-x  2 root staff 4096 Nov 18 03:11 oauth2
drwxr-sr-x  6 root staff 4096 Nov 18 03:11 protobuf
drwxr-sr-x  3 root staff 4096 Nov 18 03:11 resumable_media
drwxr-sr-x  3 root staff 4096 Nov 18 03:11 rpc
drwxr-sr-x  2 root staff 4096 Nov 18 03:11 type

ok.. let's start python:

someuser@cloudshell:~$ python

Python 2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0] on linux2

# let's inspect the sys.path

>>> import sys

>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

# ^^ our SDK path isn't in there.  let's add it to the front of the list in the manner we're supposed to

>>> sys.path.insert(1, '/usr/lib/google-cloud-sdk/platform/google_appengine')

# lets look again

>>> sys.path
['', '/usr/lib/google-cloud-sdk/platform/google_appengine', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-pa
ckages', '/usr/lib/python2.7/dist-packages']

# ok, that should work... RIGHT?


>>> from google.appengine.api import urlfetch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named appengine.api

# NOPE... where's "appengine"???

>>> import google

# ummm....

>>> google.__path__
['/usr/local/lib/python2.7/dist-packages/google']

# ummmmmmmmmm.....

So there are a couple of issues and I don't understand how they relate or how to resolve:

• cloud shell has a google package that conflicts with the google package in the SDK
• why isn't the sys.path being honored anyways?

[TheRook]

Hi everyone!

In most cases Cloud SDK or the package manager on your system installs packages app-engine-python and app-engine-python-extras inside Cloud SDK itself, i.e. if your Cloud SDK resides in /usr/lib64/google-cloud-sdk/, the packages will only be available in /usr/lib64/google-cloud-sdk/platform/google_appengine. Cloud SDK will NOT add this directory to your PYTHONPATH, thus Python will not search this location for said packages and will return the errors reported.

To fix this issue, simply add YOUR-CLOUD-SDK-INSTALLATION/platform/google_appengine to your PYTHONPATH. More specifically, run the command below:

export PYTHONPATH=${PYTHONPATH}:$(gcloud info --format="value(installation.sdk_root)")/platform/google_appengine

Additionally, note that Python will only search the first matched directory in the given paths without further instructions. For example, if you imported both google.mypackage and google.appengine.api in your script and these two packages live in two separate places, Python will report an error for either of them as it cannot find both in the same place. This should not be an issue for most Google Cloud Client Libraries, since their __init__.py is specified with __path__ = pkgutil.extend_path(__path__, __name__), which extends the path and tells Python to continue searching in the case of a mismatch. Should your error persists despite correct PYTHONPATH settings, check your site-packages folder and make sure that the __init__.py script under the google folder is corrected set.

michaelawyu

So wait, gcloud knows the answer to the problem... and it chooses not to fix the strange paths it creates?

Why isn't google using this????????

[jubeira]

@NickFranceschina I had your exact same issue, described in https://github.com/GoogleCloudPlatform/getting-started-python/issues/157#issuecomment-737307802.

why isn't the sys.path being honored anyways?

You probably have google in sys.modules, which takes precedence over sys paths.
Try this to verify (assuming you have google-cloud-sdk in either PATH or PYTHONPATH):

>>> import google
>>> google.__path__
['/usr/local/lib/python2.7/dist-packages/google']   # This is the one that you don't want
>>> import sys
>>> del sys.modules['google']
>>> import google
>>> google.__path__
['/usr/lib/google-cloud-sdk/platform/google_appengine/google']