Django-storages: Why listing image urls with DRF using GCS is so slow?

[david-alejandro-reyes-milian]

I'm using google-cloud-storage==1.3.2 + django-storages==1.6.5. Currently, I'm able to upload images to my bucket using the settings provided in the documentation, get listing my images in a view is extremely slow.

I have something like:
```class Picture(models.Model):
created = models.DateTimeField(auto_now_add=True)
image = models.ImageField(upload_to='picture')

I'm using DRF with a serializer like this one:
```class PictureSerializer(serializers.ModelSerializer):
    class Meta:
        model = Picture
        fields = ('id', 'image')

Each time I try to get the list of my Pictures, which should only provide something like:

 {
        "id": 1,
        "image": "https://storage.googleapis.com/bucket/pic1.jpg"
    },
    {
        "id": 2,
                "image": "        "image": "https://storage.googleapis.com/bucket/pic1.jpg""
    },

it takes a lot of time. My assumption is that it needs to resolve the https route to my bucket, and this should be done using the gcloud-api, but it would be nice to have a fixed(fast) URL for it since it's always the same: "https://storage.googleapis.com/bucket/picture_id.jpg"

What should be the approach for solving this issue?
On my frontend, I have enabled lazy loading for the pictures, but this URL solving time-consuming issue is not helping.

[stinky2nine]

I would suggest doing some sort of memcaching so it doesn't need to make external API calls every single time.

[david-alejandro-reyes-milian]

Hello @stinky2nine, thanks for answering. Actually, I solved the issue by asking only for the picture name and manually building the storage URL to my image on the bucket. I know that maybe it's not the best approach but I honestly can't afford to have this time-consuming processes only to retrieve a picture URL :(

Regarding memcaching, well, it could be, but in my current environment, using stateless nodes I think I don't have a way to share cache between my nodes and also, it could make my life harder. What do you think?

BTW, could you explain how django-storages is actually retrieving the image urls? Cheers!

[feakuru]

Seeing this, too, in a project of mine. Would love some insights into speeding this up. :smiley:

[stinky2nine]

I am not a package maintainer so just sharing my two cents. I personally would rewrite the gcloud backend to accommodate top use cases, e.g. not requiring files to be public, better caching, etc.

The current code validates requested file exists on GCS before returning a public url. Since it also depends on google.auth library and it's using OAuth2 for authentication, it does take several RPCs to complete the operation especially if the OAuth2 access token expires or does not exist.

My pull request #423 caches the url for 1 day by using Django’s cache framework. You can use Memcached, runs as a daemon, to share the caches among the nodes or just use the defaults which is only available on the individual node. See https://docs.djangoproject.com/en/1.11/topics/cache/#setting-up-the-cache for more information.

I was thinking about increasing the timeout to say 7 days since static files you put on GCS aren't likely ever going to get deleted. But it's a different story for user uploads.

[sww314]

I do not think an extra cache is required. The issue is that the current code calls get_blob, that verifies the files exists on GCS. In most cases that is not required.

463 is a related issue.

[elnygren]

Google Cloud backend's def url(self, name): makes a round trip to Google's APIs and complete breaks all views and APIs that list several images. This is not what most Django devs would expect especially when considering how the default File storage in Django works (url is a matter of computing settings.MEDIA_URL and the file's name stored in database).

It's in fact quite common in Django to do something akin to:

# models.py

    @property
    def cover_photo_url(self):
        if self.cover_photo:
            return self.cover_photo.url
        return '' # or some default

# template
<img src={{foo.cover_photo_url}}/>

Now I do understand that .name exists and we can just use that to compute the URL ourselves. However, the documentation doesn't mention this at all which is what the real issue here is I suppose.

[tinhien11]

+1

[jschneier]

All of this discussion is now ongoing in #491.