Firebase-js-sdk: FR: Alternative to getDownloadURL that doesn't expose a public download URL

[jhuleatt]

[REQUIRED] Describe your environment

• Operating System version: [all]
• Firebase SDK version: 4.1.3
• Firebase Product: storage

[REQUIRED] Describe the problem

Currently, the only way to get an image is to call getDownloadUrl. The download URL is unguessable but not affected by security rules, which means that once someone has the download URL, they could theoretically use a bot to create a huge number of downloads and run up a big Cloud Storage bill or hit the project quota.

The Android SDK provides getFile, which downloads the file directly to the device. Each download involves a security rules check (source).

Request

Provide a function like getFile that checks security rules every time.

Alternatively, perhaps a function that returns a signed URL (for example, getSignedUrl(expireTime)) could accomplish a similar goal.

[sphippen]

We've been considering something like this for a while. The reason we decided to only expose a downloadURL on web is because of its versatility.

If we wanted to give you a way to download the file checked by security rules, the best we could do would probably be to give you an XMLHttpRequest you could use to download the object yourself. The problem with this is that if you're using an XHR the downloaded data ends up in memory, which means if it's a large file (say, more than a few gigabytes) it's difficult to do anything reasonable with the XHR.

On the other hand, with a download URL, many things are simpler (you can embed it directly in an img tag) and you can give the user the option to download the file (an <a> tag with the "download" attribute, tell them to right click > "Save file as...", etc.).

Of course, it's also easier for anybody to automate abusive behavior if they have a download URL (as you pointed out). At the same time, for a sufficiently dedicated malicious user there isn't a big difference between exposing the download URL vs not exposing it (any user capable of downloading the file via security rules can get a download URL and do anything they want with it). As a result, we don't consider this a high-priority change.

[dinvlad]

Could you clarify (apologies if these questions are better addressed elsewhere):

1) Does getDownloadUrl from FB JS SDK check security rules _before generating_ a link? In other words, is there a reasonable guarantee that a download URL, unless shared, can only be obtained by FB users who have access to the object via security rules? And what's the lifetime of this link?

2) Is it possible to use accessToken generated by FB (with Google auth provider) to access Cloud Storage directly (bypassing FB mechanism for getDownloadUrl), e.g. by using Google Cloud SDK for JS? In which case, do the FB rules still apply? Or do we then need to manage access manually through IAM rules on the affected buckets? In which case, does the accessToken represent their "normal" Google user identity (so we can just add their GSuite email to IAM rules)?

[sphippen]

1. Access to the download URL through the SDK is indeed controlled by security rules. If you don't have read access to the object, getDownloadURL will fail. The link lasts until it is revoked manually in the console.
2. The Firebase accessToken cannot be used to access GCS except through Firebase Storage (which we only officially support through the SDKs). Even if the Firebase accessToken was generated with a Google user login, you'll still have to use the more typical credentials: a service account key, etc., if you want to talk to GCS' APIs directly.

[dinvlad]

Thanks @sphippen, regarding (2) I don't quite understand: if I authenticate with FB and ask for https://www.googleapis.com/auth/devstorage.read_only scope (privacy/app verification issues aside), then I sure am able to access GCS APIs (https://www.googleapis.com/storage/v1/b/...), as I have just verified through Postman. What do you mean then by

The Firebase accessToken cannot be used to access GCS except through Firebase Storage

[sphippen]

Sorry, I misunderstood what you meant by accessToken. I thought you meant the Firebase Auth JWT, but I see you were referring to the accessToken associated with the OAuth credential.

So, if you log a Google user into your app with Firebase Auth and request the https://www.googleapis.com/auth/devstorage.read_only permission, you get the OAuth accessToken back, which you can use to read GCS resources that that user has access to. This access is controlled by the IAM policy (or GCS' legacy ACLs) on the bucket/object you attempt to access.

In which case, does the accessToken represent their "normal" Google user identity (so we can just add their GSuite email to IAM rules)?

Exactly correct.

To summarize, the Firebase Auth JWT lets you access buckets _associated with the project in which they were created_ through Firebase Storage. The Storage security rules are applied to these requests. (third-party auth)

If you log a Google user into your app, they'll click through a page granting you the permissions you request, and the OAuth accessToken will give you those permissions. You can use that token to make requests to GCS' normal API, but only to access resources the user already has access to through Cloud IAM or GCS' legacy ACL system. (first-party auth)

[dinvlad]

Thanks @sphippen! I've tested this approach and it works as you described. Hopefully it will help others as an alternative to getDownloadUrl().

That being said, requesting a devstorage scope is a more blanket permission and since last July Google requires verification of apps that use it. We may end up just using getDownloadUrl(), or (a 3rd alternative suggested by @jhuleatt) getSignedUrl() on the backend. The latter option is better for us because it enables limited-lifetime links. I just wish getDownloadUrl() contained an option to specify the lifetime.

[tigerpup]

I have a really basic question.
does getDownloadUrl give the same url on every call till the file has been reuploaded ?

[schmidt-sebastian]

In general, we return the same download URL. It is however possible to revoke an old URL in the Console, in which the next invocation will return a different URL.

[tborja]

Hello,
We use getDownloadUrl to give the firebase storage file link access to an authenticated user. The problem is the Url returned by getDownloadUrl has an access token attached to the link that grants access to the file even to unauthenticated users. How do we prevent this from happening?

[burtonator]

@tborja is right.. this is a major problem and I'm very very concerned that it hasn't been addressed yet.

Once this is handed out, there are NO security rules applied for reading that document.

This isn't really highlighted anywhere in the SDK docs and it's really really easy for a developer to shoot themselves in the foot here.

Specifically, I just did. I had assumed that getDownloadUrl would apply the storage security rules but that's not the case. I'm going to have to spend 1-2 weeks building an alternative.

Why can't there just be a proxy in between that evaluates the rules on every HTTP request and passes the rest on to cloud storage?

That's literally what I'm going to be doing next but Firebase should be doing this for me...

Maybe change this API call to getInsecureDownloadURL or getPublicDownloadURL or at least highlight in the API docs that there are NO security rules applied to this URL once created.

[burtonator]

Access to the download URL through the SDK is indeed controlled by security rules. If you don't have read access to the object, getDownloadURL will fail. The link lasts until it is revoked manually in the console.

Is there a way to revoke them via API and not manually?

I'm working on a feature for public sharing of documents and this way I could at least revoke the URL and generate a new one.

[dinvlad]

As an alternative, you can generate a signed URL with an expiration time using admin SDK inside a cloud function.

[burtonator]

I saw that but this doesn't really rectify the problem. Once the timed URL is issued there's no way to revoke it... This is back to the same issue. I need to hand out the URLs up to the moment the user revokes access.

This is why the Firebase rules would be ideal for this situation.

[dinvlad]

As a way around this problem, one could request a signed URL with a very short expiration time, upon user request. This way, the first time they request the URL through a cloud function, they obtain a link, but it will expire within seconds (or minutes maybe, I cannot recall what the lower limit is). Would that work for you?

[toddpi314]

@dinvlad What we could use is a method on the client SDK that has a fixed expiry on the download URL where the actual expiry term isn't defined in the user-agent. That would prevent people from just flat-out hacking the expiry value seeded into an API like the getSignedUrl and would allow us to create flash tokens within the AuthN/Firestore rule scope.

Forgive the inelegance, but something like getShortTermDownloadUrl().

The alternative currently is just to create an HTTPs Function that generates a short-term URL in the AuthN context for the Firebase user. This is a bummer, since calling getSignedUrl from the function against the Storage bucket relies on convention to enforce the security rules. That pretty much duplicates the storage AuthZ rules between the rule JSON in Storage and the functions that reference it.

[dinvlad]

IMO even having it as an option on the client would be good. For now, we just have to resort to getSignedUrl inside a CF.

[garyo]

I'm facing the same problem. My web app's users upload their private data to /users/UID/data/FILE, and I absolutely don't want there to be a public URL to that FILE -- I need to ensure that only that authenticated user can download it. A short-term public URL (seconds or a minute) would be OK.
I see @sphippen 's workaround above; I'm not sure how that would work since as far as I understand it, you can't set an IAM rule to only allow access to a subset of a bucket (users/UID in my case).

[seeiuu]

Hi @garyo,
you are not alone (like you could read in post https://github.com/googleapis/nodejs-storage/issues/697 as franklyn suggest above)

The only things is that this Firebase Storage Gotchas 😅 documents helped me. I hope it can help you too!

[frankyn]

Hi @schmidt-sebastian question that's related to this. We've received requests to add getDownloadURL() to the nodejs-storage library in (https://github.com/googleapis/nodejs-storage/issues/697) but this support is very specific to Storage for Firebase and not Cloud Storage API.

Has adding getDownloadURL() method been considered within the firebase-admin package? IIUC it's an alias to the nodejs-storage package so it might not be straight forward.

[nicoqh]

Adding getDownloadURL() to firebase-admin would definitely help a lot. Then we no longer need to store the download URLs in, say, Firestore in order to save the client from having to call getDownloadURL() all over the place and wait for those calls to resolve. Storing the URLs makes it a pain to revoke/regenerate tokens.

It would also allow us to use this Firebase functionality in Cloud Functions without storing tokens manually as metadata and bypassing getDownloadURL().

It doesn't solve the issue of download URLs being persistent and having no inherent security checks. But, creating a persistent, obscure URL that has no inherent security check (getDownloadURL()), is a valid use case, so I hope this functionality isn't removed.

I agree that having a "short-lived" option would be ideal, either as a new method or as an argument to getDownloadURL().

[schmidt-sebastian]

@frankyn We currently bundle the @google-cloud/storage SDK without modification. If we were to add our own getDownloadURL() method, we would have to monkey-patch your library. This poses a couple of challenges:

• We now have a "firebase-admin" version of GCS, but we don't control the actual library. Not only will we have to take on all aspects that come with maintaining a full library (documentation, support, releases), but this also creates a very strange discrepancy between these two libraries that our users will likely not understand.

• We don't have an endpoint to talk to to generate public storage URLs with Service Accounts. We have to build a completely separate request flow for this (the implications of which I am not at all familiar with).