Google-cloud-python: Vision: document_text_detection no response on localhost

[kalravibhor]

Hi,

I am creating an application to which uses document_text_detection. I run into problems while testing the application on my machine (localhost), while the same runs perfectly well on the server (not GCP) where the application is deployed.

The application is server by gunicorn and the falcon framework and uses celery to process/annotate the images in an async fashion. Since the application itself is fairly complex, I won't be able to post a minimum working code of the same here.

While testing the application on localhost, I am unable to get a response for any image I upload through my async task queue (managed by celery). The workers in the task queue just wait for the response from Google Cloud Vision. I know I can use retry and timeout methods to counter this, however I still don't receive any response.

As a sidenote, I wanted to mention that I am sending out similar/same images almost all the time to test the application. Below are snippets of the code that I thought are important to have a look at to ascertain if I missing out on something.

os.environ['GOOGLE_APPLICATION_CREDENTIALS']=os.path.dirname(os.path.abspath(__file__)) + '/xxxx.json'
client = vision.ImageAnnotatorClient()
with io.open(imgFile, 'rb') as image_file:
    content = image_file.read()
image = types.Image(content=content)
response = client.document_text_detection(image=image)

I am using a service account key as reflected from the code above. However I also tried authenticating using gcloud auth application-default login and gcloud auth login with no avail.

Environment Details

OS X 10.13.6
Python 2.7.14
gunicorn (version 19.7.1)
falcon version 1.4.1
celery version 4.2.1

Name: google-cloud
Version: 0.34.0
Summary: API Client library for Google Cloud
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google Cloud Platform
Author-email: [email protected]
License: Apache 2.0
Location: /Users/Kronos/.virtualenvs/objdet/lib/python2.7/site-packages
Requires:
Required-by:

I really require the application to work well on my local machine to save time in testing and debugging. Any help will be greatly appreciated. TIA.

[tseaver]

Can you please verify that the snippet you showed above runs from your machine without all the gunicorn / celery stuff in place?

[kalravibhor]

@tseaver Yes, it does. Have tried it multiple times. What is perplexing is that this snippet works on any cloud sever without fail.

[tseaver]

Given that the snippet works without celery / gunicorn in play, I don't think we have an actual bug in the google-cloud-vision library, so I'm closing. Please feel free to keep discussing here, or reopen if you find something we could change to help your usecase.

One thing to check: compare the versions in the output of pip freeze in your two environments: perhaps you have a version mismatch?

Another thing to try is to trigger the API call using only celery or only gunicorn, to figure out which one is interacting badly with the API call. My suspicion is that gunicorn is the culprit, because it monkey patches lots of stdlib features.

[kalravibhor]

@tseaver : Looks like celery is interacting badly with the API call. Any idea on what might be causing this ? Looking for any help. Thanks !

[tseaver]

@kalravibhor I'm not positive that it would help, but I would try to avoid sharing the ImageAnnotatorClient across forks: can you arrange to create a new client instance as part of processing the task in each worker?

[kalravibhor]

@tseaver Yeah, I realized that the ImageAnnotatorClient is perhaps not fork safe. Found the same issue that somebody else was facing with the LanguageServiceClient on StackOverflow (link below).

My sense is that initializing the client for every worker might be slower than sending a request via the requests module in Python. I am able to successfully get the response inside a Celery worker process if I hit the Vision API endpoint using the same. Hope this works. Thanks !

https://stackoverflow.com/questions/51964541/call-to-google-cloud-api-in-celery-task-never-returns-hangs

[mcrasso]

I was suffering the same deadlock. As suggested by @tseaver the incompatibility seems to be with gnunicorn worker-class. I can reproduce the deadlock when using:
gunicorn --bind 0.0.0.0:80 --workers=1 --worker-class gevent <<cut>>

and works fine when using:
gunicorn --bind 0.0.0.0:80 --threads=4 --worker-class sync <<cut>>