Google-api-python-client: File caching for the oauth2client >= 4.0.0

[xmedeko]

@tmatsuo proposed in #110 to cache discovery.build(), however, #294 has stopped using cache outside of the GAE (appengine) environment (see also #299). E.g. the discovery.build() is not cached in the CGE environment. Is it possible to bring the file caching back, or provide some example of file cache (subclass of googleapiclient.discovery_cache.Cache)?

[theacodes]

Sure, mostly what needs to happen is that the current file cache needs to
be updated to not use oauth2client's locked file.

On Sun, Jan 8, 2017, 2:14 PM Ondrej Medek [e-mail] wrote:

@tmatsuo https://github.com/tmatsuo proposed in #110
https://github.com/google/google-api-python-client/issues/110 to cache
discovery.build(), however, #294
https://github.com/google/google-api-python-client/pull/294 has stopped
using cache outside of the GAE (appengine) environment (see also #299
https://github.com/google/google-api-python-client/issues/299). E.g.
the discovery.build() is not cached in the CGE environment. Is it
possible to bring the file caching back, or provide some example of file
cache (subclass of googleapiclient.discovery_cache.Cache)?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/google/google-api-python-client/issues/325, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAPUc2WtGDCjrz4H1QhftQxypDuOidnpks5rQV-6gaJpZM4Ld0fj
.

[Schweigi]

Is there a known workaround in the meantime?

[Schweigi]

I created a small workaround by using a in memory cache (see example below). Does anyone know if there should be a regular cache invalidation? Currently the memory cache is cleared on each deployment, I hope that's ok for the time being.

In memory cache:

class MemoryCache(Cache):
    _CACHE = {}

    def get(self, url):
        return MemoryCache._CACHE.get(url)

    def set(self, url, content):
        MemoryCache._CACHE[url] = content

Usage:

http = ...
service = discovery.build('calendar', 'v3', http=http, cache=MemoryCache())

[xmedeko]

AFAIK the Google production API should not change (the same version of API), so the you do not need to invalidate the cache. However, the discovery_cache/__init__.py defines the limit DISCOVERY_DOC_MAX_AGE as one day.

+1 for a neat, simple MemoryCache. IMO it's good when you have a single, long running process.

If you want to share the cache among processes, (like running scripts by the cron,) you may copy the old good oauth2client locked_file.py and then to plug it somehow with discovery_cache/file_cache.py. Beware, the locked_file.py is considered not 100% save, so you take the risk. (But works well in practice.) By "plugin somehow" I mean either copy & change the discovery_cache/file_cache.py to take your locked_file.py or to trick the Python to think your locked_file.py is from the oauth2client.locked_file package.

[xmedeko]

@jonparrott what about to change file lock to the multiprocessing.RLock() in the original locked_file.py? Would be it sufficient solution to accept back to the repo?

[theacodes]

I think it's likely possible to do this without any sort of locking.

On Wed, May 17, 2017, 12:02 AM Ondrej Medek [e-mail]
wrote:

@jonparrott https://github.com/jonparrott what about to change file
lock to the multiprocessing.RLock() in the original locked_file.py? Would
be it sufficient solution to accept back to the repo?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/google-api-python-client/issues/325#issuecomment-302004035,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUc-ne-ZS8p4rfQbV1qs3RtjXgc7Opks5r6puOgaJpZM4Ld0fj
.

[xmedeko]

You mean a file cache, shared among multiple processes, without any locking? How to prevent data corruption, when 2 process write to the same file?

[theacodes]

Likely using a folder containing a file for each cached discovery doc. No
locking needed. Worst case scenario two processes write the same file twice.

On Wed, May 17, 2017, 12:08 AM Ondrej Medek [e-mail]
wrote:

You mean a file cache, shared among multiple processes, without any
locking? How to prevent data corruption, when 2 process write to the same
file?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/google-api-python-client/issues/325#issuecomment-302005048,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUc9PWEHEBR_dWKW4KmNhmgV4lH8qiks5r6pzbgaJpZM4Ld0fj
.

[xmedeko]

FYI to use the locked_file from oauth2client 3.0.0 without that package (it's not required since 1.7.3), just use the gist https://gist.github.com/xmedeko/e5293707e075d18c9dde8a574d2670e5 and call the shim() once.

[Chronial]

Based on Schweigi's solution, I wrote a simple file based cache (unix-only) to also cache across processes.

import os.path
import hashlib
import tempfile

class DiscoveryCache:
    def filename(self, url):
        return os.path.join(
            tempfile.gettempdir(),
            'google_api_discovery_' + hashlib.md5(url.encode()).hexdigest())

    def get(self, url):
        try:
            with open(self.filename(url), 'rb') as f:
                return f.read().decode()
        except FileNotFoundError:
            return None

    def set(self, url, content):
        with tempfile.NamedTemporaryFile(delete=False) as f:
            f.write(content.encode())
            f.flush()
            os.fsync(f)
        os.rename(f.name, self.filename(url))

[busunkim96]

I'm opting to close this as it's unlikely we will implement it in the future.

These client libraries are officially supported by Google. However, the libraries are considered complete and are in maintenance mode. This means that we will address critical bugs and security issues but will not add any new features.

[aawilson]

I don't know if this changed at some point, but I had to import Cache from googleapiclient.discovery_cache.base (presumably googleapiclient.discovery_cache.file_cache also works fine, but inheriting from the Base class is almost certainly the better idea)