Rails: ActiveStorage: Allow access to backing file from Service API

[jduff]

Steps to reproduce

Currently with the ActiveStorage::Service api you can only get a link through the url method which, for most services, gives back a public URL that expires in some timeframe. It would be very useful if there was a file method that returned the backing file object from the service so that you have more flexibility in how you can expose those files.

System configuration

Rails version: 5.2.0.beta2

[rafaelfranca]

Sounds good to me. @georgeclaghorn thoughts?

[georgeclaghorn]

I’m :-1: on the same method having a different signature in each Service class. It defeats the purpose of the Service abstraction.

[rafaelfranca]

Would a new file method require to have a different signature in each Service class?

[georgeclaghorn]

If I understand the proposal here correctly, it’d return a different type of object in every class. The file classes provided by the clients are different enough that any application using ActiveStorage::Service#file could not be indifferent to the configured service.

[rafaelfranca]

Ah yeah, got it. I understand your concerns but I think this doesn't defeats the purpose of the service abstraction. Take the Active Job adapter as example, each implementation return a different object each job with the same API. Or the Active Record connection adapters, where ActiveRecord::Base.connection can return similar objects but with slightly different API.

This proposal is not asking us to have different methods in the Service layer, it is asking to us to allow the inner implementation to be accessible via the common API in the Service layer. Of course as soon the application stats to use the information in the inner implementation it is now coupled to that provider, but at least now people can leverage the framework in order to build more complex cases that we don't support.

[georgeclaghorn]

I’m still :-1:. If you want, say, a Google Cloud Storage client, you don’t need to go through Active Storage to get one.

[rafaelfranca]

Of course not, in the same way if you want a MySQL connection you don't need to go through Active Record, but why make it harder if we can make it easy? Is not this the whole point of Rails?

[georgeclaghorn]

The point of Active Storage is to permit Rails applications to maintain a healthy indifference to various storage services. It was borne from a specific need to keep Basecamp at arm’s length from heterogenous storage backends.

I’ve already stated my objections on those grounds and would suggest that you find a different way to accomplish your still-unstated purpose. Nonetheless, it sounds like you’re going to proceed against my objections. It’s your call.

[rafaelfranca]

Nonetheless, it sounds like you’re going to proceed against my objections.

I'm not, this is why I asked your opinion.

I really understand your reasoning about this but I think we can make a compromise since if someone wants to couple their application to the storage it is their choice and I believe Rails could also help those users. Basecamp and other applications that users heterogenous backends would not need any change and should not care about this new method. But I can also understand your side of this being a possible sharp knife to hurt applications.

If you are still strong about it I'm totally fine with not exposing it.

[georgeclaghorn]

Sorry, I didn‘t mean to imply you’re not listening to me. I know you are and I appreciate it. :heart:

If you have a use for this, let’s add it. You’re right that we don’t have to use it.

[jduff]

I can work around this if I really need to, but I was thinking along the lines of @rafaelfranca when he said "why make it harder if we can make it easy".

The case I have for this is we have an internal application that is storing somewhat sensitive documents and we're hesitant around have a public url, even a random one that expires, available to the files. I already added a custom controller to redirect to the file so that I can authenticate the users from the application side before exposing the url, but they could still share that expiring url with someone that doesn't have access.

In this case the storage service can also build protected urls that will use the ACL we have set up, but I need the client and the file object to call those.

Again, I can patch around this, but it sounded like a case that others could run into as well. If there is another way to get the same result, or a different API that could be added that works too, I'm not married to the idea of exposing the file object. I would rather come up with the right API that fits with the overall goal of ActiveStorage than just the thing that meets our specific need.

Thanks for talking through this with us @georgeclaghorn ❤️

[meinac]

There is the download method and as far as I see from the implementation it returns the content of the file, doesn't it work for you?

[jduff]

That would be a possible solution, download the file and proxy it through the app. I was hoping to get a direct url to the file that would require google account authentication, but I could handle that part in the app and proxy the file as a work around.

[dwightwatson]

I had a crack at proxying files through the app in #30465. I would like to see a way of directly accessing certain attachments rather than being given an expiring link to them.

My use case is that some of the images we upload through ActiveStorage are intended to be public. We are seeing lag when it has to generate a new expiring link, and because they are expiring it's more difficult to cache.

In addition we're using CloudFront as a CDN which caches the redirect to the asset, not the end result of the asset itself. I don't know how other CDNs tackle this sort of thing, but it effectively makes CloudFront incompatible with ActiveStorage URLs.

[georgeclaghorn]

As with the OP’s case, I think there’s a general solution to that problem that doesn’t require apps to couple themselves to the underlying clients of the various services. (It might even be the same problem.) Please Do Investigate. :smile:

[matthewd]

I do think there's value, as a general principle, in providing an escape hatch to access the underlying layer: if someone can use the abstract ASt API for 95% of their needs, better we allow them to do so, without forcing them to choose between a fully custom no-ASt implementation, or manually reimplementing ASt's knowledge of how models map to entries in the store.

IMO, we lean pretty heavily to the pragmatic, rather than perfect, abstraction... an AR model will hand you the raw database connection; it'll also accept a backend-dialect-specific where condition. A goal of AR is certainly to allow an application to remain as arms-length as practical from the data store, but not so much that it obstructs people when a generic solution is unavailable.

All of that said, if we offered such a method, I think I'd want it named something slightly sharper-edged than file, more in line with AR's raw_connection -- just enough to make it sound like you're piercing a layer of abstraction.

[dhh]

(Not rendering an opinion on this specifically, just don't think it's a blocker for 5.2.0. I'm inclined too to open a syntactically vinegar'ed backdoor for people to do whatever they want.)

[koenpunt]

Not the same as what @jduff is asking for, but; I'm using Rails as a backend of a relatively high traffic website, with its main purpose of serving/displaying images. All these images are allowed to be public (and are configured like that on S3), so no need for signed URLs. And like mentioned by @dwightwatson, caching with signed URLs is an issue.
So for me using the url method is not a problem, but having the option to get it without signing parameters would be nice.

Of course this would be solvable with a custom implementation if the backing file would be exposed from the service.

Then related to this; how would one configure an CDN host for AS stored files (instead of the s3 urls)? Or is that currently not possible?

[wadestuart]

I have to say I am in the same boat as @koenpunt as it stands the architecture of AS really forces one into a very specific use case. There are large portions of apps out there that conflict entirely with the signed access/temp url endpoint use.

Is there a doc somewhere that goes over the reasoning behind some of these conventions? Perhaps stating the thought usage when the site has what seems like common cases of public assets, CDN fronts, static file delivery (without requests to rails per asset per user per time period).

IMHO the whole signed and managed url seems like it should be the optional behavior not the default.

[dhh]

The managed and signed URLs are indeed optional. We have a full explanation
in the docs about how you can make your own URLs that use a different
authentication scheme by using your own controllers. You're not forced to
use the signed URLs at all, but they're there if you want to.

On Mon, Jan 29, 2018 at 1:06 PM, wadestuart [e-mail]
wrote:

I have to say I am in the same boat as @koenpunt
https://github.com/koenpunt as it stands the architecture of AS really
forces one into a very specific use case. There are large portions of apps
out there that conflict entirely with the signed access/temp url endpoint
use.

Is there a doc somewhere that goes over the reasoning behind some of these
conventions? Perhaps stating the thought usage when the site has what seems
like common cases of public assets, CDN fronts, static file delivery
(without requests to rails per asset per user per time period).

IMHO the whole signed and managed url seems like it should be the optional
behavior not the default.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rails/rails/issues/31419#issuecomment-361386048, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAKtfvPghGHq1yR-_Cm9nAjy-pQDE5dks5tPjLIgaJpZM4Q_Z6H
.

[dhh]

You can see how little code there actually is in the default controllers here: https://github.com/rails/rails/blob/master/activestorage/app/controllers/active_storage/blobs_controller.rb

Adding your own controller that wraps this in Google Authentication or whatever scheme you please should be trivial.

[dhh]

@koenpunt I'd be happy to see a patch that generated a URL without signature if false was passed to expires_in: 👍

[koenpunt]

We have a full explanation in the docs about how you can make your own URLs that use a different authentication scheme by using your own controllers.

I seem to be unable to find that.. Can you point me in the right direction?

[dhh]

@koenpunt Here's the default controller: https://github.com/rails/rails/blob/master/activestorage/app/controllers/active_storage/blobs_controller.rb. You basically just do that, but with your own wrapping, and then you'll have your own URLs for it 👍

[koenpunt]

I'd be happy to see a patch that generated a URL without signature if false was passed to expires_in: 👍

I looked into this, and started with S3, but there the content disposition headers do not work for unsigned (public) urls, and thus you end up with a url like https://rails-as-test-1.s3.eu-central-1.amazonaws.com/rwFanLfBnQC831WjCaCMAEbh.
So unless content type and other headers are set when uploading the file, I doubt this is going to work.

It would be nice if the storage path could be configured, so that in the case of publicly accessible items, the service url can be used directly, instead of routing through Rails.

So the actual keys in the bucket would become something like:

• rwFanLfBnQC831WjCaCMAEbh/myfile.jpg
• variants/rwFanLfBnQC831WjCaCMAEbh/5ab8a8648821d31837ecfa2b3b5ae85f52c099af95308cd4f5c01f76882427b7/myfile.jpg

[koenpunt]

Here's the default controller:

I've seen that, but I expected there to be a more since you mentioned "a full explanation" 😅

[dhh]

The shorter the code, the fuller the explanation 😄

But yeah, if S3 doesn't actually support the use case, then you probably
just need to implement your own proxying controller and then hide that
behind a CDN.

On Wed, Jan 31, 2018 at 2:12 PM, Koen Punt [e-mail] wrote:

Here's the default controller:

I've seen that, but I expected there to be a more since you mentioned "a
full explanation" 😅

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rails/rails/issues/31419#issuecomment-362088443, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAKtRrXtZYiYa4DmNcJ9lBDJHcN7MDbks5tQOVagaJpZM4Q_Z6H
.

[stevenharman]

how would one configure an CDN host for AS stored files (instead of the s3 urls)? Or is that currently not possible?

@koenpunt Perhaps I'm misunderstanding your goal, but would Serving Private Content through CloudFront work?

[ipstatic]

I am in the same boat as @koenpunt, private application managing public assets in S3 that are served by a completed different app (static site). I have also tried to generate an unsigned URL and end up with the same issue (content type is not set, extension not present). Are there plans to add this kind of support? What kind of work needs to be done to support this? If someone with a better idea with how this should be coded lays out what needs to happen, I wouldn't mind taking a stab at a PR.

[koenpunt]

but would Serving Private Content through CloudFront work?

Technically maybe, but I'm not using CloudFront, and not planning to do so, since it's fairly expensive, especially compared to the flat-fee CDN Cloudflare has.

[themilkman]

In case I understood all this correctly (if not, sorry! ;-)), the problem is that there is no way to force a custom authorization for a specific file. In that case: I do also have this case. AS works really great but I do not want to risk that one could load a file without passing the given rules of my rails application - this is quite important for many use cases since there are no public files, only sensible ones.
I guess I could build my custom route which targets an inherited BlobsController Instance, but is this really the cleanest way to solve this?

Cheers and thanks for any feedback!

[georgeclaghorn]

@themilkman: No, that’s not related. If you want to authorize file access, you’ll need to implement a custom controller. The documentation says as much.

[gregblass]

EDIT: Alrighty, after consulting the ole rubber duck for a while, I've deleted the multiple comments I posted here and summarized to one.

For my use case, I've got a multi-tenant Rails app that I allow administrators to sign into and upload their company logo, which gets served through a GraphQL API to save app-wide settings on many mobile devices linked to that company account. So I need to just send along a public facing URL of the logo to save in some local settings on these mobile devices.

2ND EDIT: @dinatih's solution worked perfectly for me. I'd recommend anyone trying to expose public, no-expire URL's to use his code below!

[dinatih]

Update:
gist patch to add :acl (:public or :private) options
https://gist.github.com/dinatih/dbfdfd4e84faac4037448a06c9fdc016
(it use expire_in: false when public as proposed by @dhh in https://github.com/rails/rails/issues/31419#issuecomment-361411670)

class User < ActiveRecord::Base
  has_one_attached :avatar, acl: :public
  has_many_attached :documents, acl: :private
end

---

For those who want ActiveStorage with public acl and no expire time for S3 you can do this :

# config/initializers/active_storage.rb
Rails.application.config.to_prepare do
  if defined?(ActiveStorage::Service::S3Service)
    # Make ActiveStorage as public-read. @dinatih
    ActiveStorage::Service::S3Service.class_eval do
      def upload(key, io, checksum: nil)
        instrument :upload, key: key, checksum: checksum do
          begin
            object_for(key)
              .put(upload_options.merge(body: io, content_md5: checksum, acl: 'public-read'))
          rescue Aws::S3::Errors::BadDigest
            raise ActiveStorage::IntegrityError
          end
        end
      end

      def url(key, expires_in:, filename:, disposition:, content_type:)
        instrument :url, key: key do |payload|
          generated_url = object_for(key).public_url
          payload[:url] = generated_url
          generated_url
        end
      end
    end
  end
end

[gregblass]

Thanks @dinatih!!!

EDIT: This worked fantastic/perfect for me. Much appreciated!!

[MarkMurphy]

I landed on this thread looking for examples on how I might use AS with S3 and CloudFront. Currently, it doesn't look like there's any support for using a CDN.

I wonder if we could expose something similar to what Paperclip does:
https://stackoverflow.com/questions/32077746/rails-4-use-cloudfront-with-paperclip

Which is essentially to allow the S3 service to accept a host parameter for constructing the url or alternatively a callback method? IDK but the DiskService kind of already does this.

Thoughts?

[nukeproof]

To work with cloudfront, here's a hacky way you could add to the example that @dinatih provided above to generate public, unsigned urls with some extra code to replace the host and bucket portion of the urls with your cloudfront host.

# \config\initializers\activestorage.rb
module MyApp
  class Application < Rails::Application

    config.cloudfront_host = ENV["CLOUDFRONT_HOST"]

    config.after_initialize do
      if defined?(ActiveStorage::Service::S3Service)
        ActiveStorage::Service::S3Service.class_eval do
          def cloudfront_host
            @cloudflront_host ||= Rails.configuration.cloudfront_host
          end

          def proxy_url(url)
            return url unless cloudfront_host
            uri = URI(url)
            uri.host = cloudfront_host
            uri.path.gsub!("/#{bucket.name}","")
            uri.to_s
          end

          def upload(key, io, checksum: nil)
            instrument :upload, key: key, checksum: checksum do
              begin
                object_for(key).put(upload_options.merge(body: io, content_md5: checksum, acl: 'public-read'))
              rescue Aws::S3::Errors::BadDigest
                raise ActiveStorage::IntegrityError
              end
            end
          end

          def url(key, expires_in: nil, filename: nil, disposition: nil, content_type: nil)
            instrument :url, key: key do |payload|
              generated_url = proxy_url object_for(key).public_url
              payload[:url] = generated_url
              generated_url
            end
          end
        end
      end
    end
  end
end

[wbotelhos]

Hi @nukeproof ,

I tried your patch version, but in that moment the ActiveStorage::Service::S3Service still not defined. And to use to_prepare on initializers does not see the S3Service yet too.

Someone could point me to the right direction? I'm trying this steps:

1. Change s3.amazonaws.com/files.bucket.com to files.bucket.com.s3.amazonaws.com inside AS;
2. Create a CNAME on Cloudflare from files to files.bucket.com.s3.amazonaws.com;
3. Configure config.action_controller.asset_host = 'https://files.bucket.com';
4. Avoid expires on URL to avoid change the path and MISS the cache.

Has anyone managed or needed to do these steps to cache the uploaded files on a CDN?

Thanks! (:

[nukeproof]

@wbotelhos have you defined a configuration in config/storage.yml that uses the S3 service? Something like this:

# config/storage.yml
amazon:
  service: S3
  access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
  secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
  region: "ca-central-1"
  bucket: "your.bucketname"

And use that service in your config/environment/[development|test|production].rb like this:

 # config/environment/production.rb
 config.active_storage.service = :amazon

[wbotelhos]

Hi @nukeproof ,

Using it on production it worked.

For those who want to use Cloudflare:

• Attach a certificate on ALB;
• Configure SSL as Full, Full (strict) will raise 526;
• Creates a bucket with a sub domain name like: assets.example.com;
• Creates a CNAME to your bucket including the region like: assets -> assets.example.com.s3-sa-east-1.amazonaws.com;
• To use @nukeproof solution with assets.example.com as the CLOUDFRONT_HOST.

Thanks @nukeproof and @dinatih for the solution.

[collimarco]

+1 I don't like the choice to force the use of redirects. For many apps is not a viable solution.

_For example_, when you send web push notifications (W3C Push API), those redirects create a lot of extra load on servers and in any case you must provide a url that is valid for some weeks at least - you can't use a link that expires.

[m0n9oose]

I've found a pretty simple solution to generate permalinks without expired key, but it's still serve the files through rails controller. Hope this helps.

# config/routes.rb

get '/files/:key/*filename', to: 'files#show', as: :rails_public_blob
direct :public_blob do |blob, options|
  route_for(:rails_public_blob, blob.key, blob.filename, options)
end
resource :files, only: :show

# app/controllers/files_controller.rb

# frozen_string_literal: true

class FilesController < ActionController::API
  include ActionController::Live
  before_action :setup_response_headers

  def show
    disk_service.download(params[:key]) do |chunk|
      response.stream.write(chunk)
    end
  ensure
    response.stream.close
  end

  private

  def setup_response_headers
    response.headers['Content-Type'] = params[:content_type] || DEFAULT_SEND_FILE_TYPE
    response.headers['Content-Disposition'] = params[:disposition] || DEFAULT_SEND_FILE_DISPOSITION
  end

  def disk_service
    ActiveStorage::Blob.service
  end
end

And that's it. No redefines, no monkeypatching.

# view
json.link public_blob_url(model.file)

[pskl]

For public assets and using the answer from @georgeclaghorn in this post:

class ActiveStorage::Service::CloudfrontS3Service < ActiveStorage::Service::S3Service
  def url(key, **)
    uri = URI(super)
    uri.host = RunEnv.var!('S3_DISTRIBUTION')
    uri.to_s
  end
end

[programrails]

dinatih
I applied your code, but while (later) purging ActiveStorage attachments (made with your code), S3 (non-expiring) links remained alive (which is probably wrong).

[dinatih]

@programrails
I do not see why, it should be ok, I do not override the purge method...
Sorry I can't help you.

[kwent]

@dinatih Any idea how we can upload the content-type as well ? Cloudfront (or even direct link to S3) is resolving my image as content-type: binary/octet-stream but really it's should be image/png in my case.

[dinatih]

@kwent
You could pass the content_type in upload(key, io, checksum: nil)

object_for(key).put(
  upload_options.merge(
    body: io, 
    content_md5: checksum, 
    acl: 'public-read', 
    content_type: 'image/png')
)

No tested Hope it works

[kwent]

Of course but in this case the content type is fixed ... I have png / jpeg
/ gif / ...

On Wed, Jul 18, 2018 at 2:16 AM dinatih [e-mail] wrote:

@kwent https://github.com/kwent
You could pass the content_type in upload(key, io, checksum: nil)

object_for(key).put(
upload_options.merge(
body: io,
content_md5: checksum,
acl: 'public-read',
content_type: 'image/png')
)

No tested Hope it works

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rails/rails/issues/31419#issuecomment-405865986, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA1spKHJKWxOhIG7JTe4f5AMtnlJEHrvks5uHvzggaJpZM4Q_Z6H
.

[dinatih]

@kwent
That what I was thinking first 🙃. Take a look at my gist https://gist.github.com/dinatih/dbfdfd4e84faac4037448a06c9fdc016, see how my :acl option is passed through :

• Blob#upload(io)

service.upload(key, io, checksum: checksum, acl: metadata[:acl], content_type: content_type)

• Service::S3Service

def upload(key, io, checksum: nil, acl: 'private', content_type: 'image/png')
  # ...
  object_for(key).put(
    upload_options.merge(
      body: io, 
      content_md5: checksum,
      acl: acl == 'public' ? 'public-read' : 'private',
      content_type: content_type)
    )
  # ...
end

Hope it helps.

[kwent]

@dinatih That helps ! Here is a version of your gist where content type is passed along when uploading the file to s3. https://gist.github.com/kwent/0fb55eedb7eb45c9cea2676ab7760a6a/revisions

Regards

[akshaysharma096]

@dinatih will your gist: https://gist.github.com/dinatih/dbfdfd4e84faac4037448a06c9fdc016 be able to handle variants?

[dinatih]

@akshaysharma096 No, it is just a quick patch, sorry. You need to *class_eval Variant model too based on my override of Blob model

[yogodoshi]

I'm trying to find a workaround for my use case but as I think it's a pretty common one I'm sharing it here: applications that need to have image urls as meta tags, including variants.

Right now I can't find a way to have a non-expiring url for a variant to use in the og:image meta tag. And I need it to be a variant as I can't just link a 5mb image in the meta tag.

[jeffmcfadden]

@yogodoshi We created a VariantsController that handles requests specifically for variant images in cases like this, and proxies the data back to our CDN.

So the request path is Browser -> CDN -> Variants Controller -> Read File Data In S3, Return That Data -> CDN -> Browser

We return long-expiring headers to the CDN as well, which it passes on, to limit load on our app.

Our site is hosted on EC2; we've had no trouble with performance or load with this setup.

[ConfusedVorlon]

I'm a fairly unsophisticated Rails user, I though I'd add my perspective on the failure of active storage image variants to 'just work'

I just fired up Active Storage for the first time.
My index page is essentially a product list with a bunch of product thumbnails.

the active storage docs show how easy it is to
1) upload images
2) access variants

all this is true - except that the default implementation (for s3 at least)
a) can't cache image urls
b) adds an ~500ms delay for each image while AS checks if the variant has been processed

all this is great for secret attachments

for the 'just works' case of serving images though, it seems nuts.
(and the docs imply that ActiveStorage is a suitable solution for images and resized variants)

We don't have signing on images served through the asset pipeline - why do we have (required) signing on images uploaded through active storage?

I honestly didn't expect to be looking at monkeypatching, custom variants controllers, etc in order to show an image and a bunch of thumbnails sensibly. I expected Rails to have delivered the customary magic of 'just works'

for me, a syntax like

has_one_attached :hero_image, public: true
would make sense

and for variants, some solution whereby rails doesn't need to query s3 for each image
(at least for some defined sizes)

thanks as ever for the great work.

[xiobot]

Looks like I'll have to give up trying to get ActiveStorage to work for my rails app.

It just doesn't suit the context of what I need, specifically to load images behind a CDN. Carrierwave as it stands is a much simpler implementation and that clearly is not great news for what should be a nice out of the box rails feature.

I'll be switching to ActiveStorage when simple permalink functionality is enabled and this convoluted approach is deprecated (which it will be I hope)

[collimarco]

@xiobot I agree with you that ActiveStorage is not suited for all applications at the moment

[huacnlee]

I wrote an Active Storage use example with the Public URL generate.

https://github.com/huacnlee/rails-activestorage-example

[fleck]

I have a PR to add direct linking and proxying to active storage https://github.com/rails/rails/pull/34477 still needs work, but feedback would be appreciated.

[kylefox]

I think it would be useful to support multiple services within an application, instead of a single application-wide service.

There would still be a default (application-wide) storage service. But allowing different services _per attachment_ enables flexible behaviour like this:

class User < ApplicationRecord
  # Store profile photos on S3 publicly, serve from CDN, etc.
  has_one_attached :photo, storage: :public_s3

  # Store birth certificates on S3 privately, require signed URLs, etc.
  has_one_attached :birth_certificate, storage: :private_s3
end

Django has had a mechanism like this for years, and it's a delight to work with — it feels like the right balance of indirection and simplicity.

[dhh]

I’d be happy to see something like that 👍

On Dec 2, 2018, at 14:41, Kyle Fox [e-mail] wrote:

I think it would be useful to support multiple services within an application, instead of a single application-wide service.

There would still be a default (application-wide) storage service. But allowing different services per attachment enables flexible behaviour like this:

class User < ApplicationRecord
# Store profile photos on S3 publicly, serve from CDN, etc.
has_one_attached :photo, storage: :public_s3

# Store birth certificates on S3 privately, require signed URLs, etc.
has_one_attached :birth_certificate, storage: :private_s3
end
Django has had a mechanism like this for years, and it's a delight to work with — it feels like the right balance of indirection and simplicity.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[joeczucha]

Just to add another possible use-case for this...

My application has a number of user-submitted articles. An email is periodically generated and sent out to registered users showing the latest articles that have been added to the site; I've been including the avatar of the author with the article summary in the email by just linking to the asset on S3.

However, I can't think of a way that I can continue to do this if I migrate to ActiveStorage, given that there is no way to disable the expiry for certain assets.

[EnziinSystem]

@akshaysharma096 No, it is just a quick patch, sorry. You need to *class_eval Variant model too based on my override of Blob model

Sorry, but if you can not override Variant, then the patch on ActiveStorage::Service::S3Service not mean.
Because in fact, variants use very often.
If I use the line:
<%= image_tag @course.image.service_url %>
It's work, but if I use:
<%= image_tag @course.image.variant(resize: "850x480").service_url %>
It not working.

[sc0ttman]

I have a use-case where I would like all the originals to be private but variants to be public (essentially preview images of digital product).
I have taken what @georgeclaghorn has said here and subclassed the S3Service.
I have Overrode the def upload and def url methods to store and retrieve assets as public if if key.start_with?("variants").

[Manfred]

From my experience in working with applications with multiple storage providers I would like to add some observations to this issue which might be relevant.

When referencing an object in storage you generally need a storage provider (e.g. Amazon S3 or Google Cloud Storage), a region (availability regions are usually geographically tied), a container (e.g. bucket on S3 or container on Rackspace Cloud Files), and a path (e.g. key on S3 or path on Cloud Storage). These four are pretty consistent across providers.

A representation for an S3 object could be:

{
  service: 's3',
  region: 'eu-central-1',
  container: 'my-bucket-name',
  path: 'path/to/object.jpg
}

Storing metadata with an object reference is crucial for performance reasons, but we'll leave that aside for now.

Note that the first three are pretty static so we want to denormalize and store them centrally somewhere. Active Storage got this right and stores that information in storage.yml.

The only thing missing to reference objects on different storage providers is to add a service_name column to the blobs table and then implement ActiveStorage::Blob#service.

This allows for a great API to move or copy files between providers because the Blob always references both of them.

---

Code that generals URLs or paths (e.g. https://…, file://…, /path/to/file) almost operates like a view layer on the Blob. In 99% of the cases Blob#url is good enough, but are always one or two places in an application where you need to do something special.

URLs can have wildly different requirements because content delivery networks, asset servers, and everlasting URLs all use different additional information.

Everlasting URLs are a URL back to the website itself with some stable token that allows you to generate a URL which can be used in archived media like PDFs, emails, and chat.

• Serving private files through a CDN might require additional credentials or encryption keys to generate the URL.
• Asset servers can be dependent on the actual request (e.g. server pinning).
• Everlasting URLs might need to know the tenant domain to generate the URL (e.g. company-name.myproduct.com).

You never want to support all these options through one url method. I believe we've learnt through other Rails APIs that having methods with an incredible amount of options to make them completely unmaintainable.

Direct access to the ‘low-level’ storage objects can be useful for ops, but are not the ideal API when you want to generate URLs because it adds coupling to the underlying libs.

class Book < ApplicationRecord
  has_one_attached :cover_image
end

book.cover_image.storage_object #=> #<Aws::S3::Object>

if book.cover_image.on?('s3')
  book.cover_image.variant(
    resize_to_limit: [100, 100]
  ).storage_object.public_url #=> "https://…"
end

Active Storage could implement URL formatters which take a blob or variant, some configuration, and optional additional arguments to generate a URL.

For example, using this configuration (not sure where this would be stored).

{
  cdn: {
    service: 'CloudFront'
    origin: 's3:eu-central-1:my-bucket-name'
    domain_name: 'example.cloudfront.net'
    private_key: '…'
  }
}

You could implement plumbing similar to Service to fetch the configuration and initialize a formatter with static information.

formatter = ActiveStorage::UrlFormatter.configure(:cdn)
variant = book.cover_image.variant(
  resize_to_limit: [100, 100]
)
formatter.signed_url(key: variant.key)
formatter.public_url(key: variant.key)
formatter.url(key: variant.key)

Methods on a formatter don't have to conform to a specific interface, but it might be useful for all of them to expose a url method.

So if we go back to the book example, we would get the following model.

class Book < ApplicationRecord
  has_one_attached :cover_image

  def cover_image_url
    cdn_url_formatter.signed_url(
      key: cover_image_small_variant.key
    )
  end

  private

  def cover_image_small_variant
    cover_image.variant(
      resize_to_limit: [512, 512]
    )
  end

  def cdn_url_formatter
    ActiveStorage::UrlFormatter.configure(:cdn)
  end
end

Or in the case of a provider specific formatter for a publisher which requires request information for some reason.

class Book < ApplicationRecord
  has_one_attached :pdf_file

  def pdf_file_download_url(request_hostname)
    download_url_formatter(request_hostname).url
  end

  private

  def download_url_formatter(request_hostname)
    ActiveStorage::UrlFormatter.configure(
      :assets, request_hostname: request_hostname
    )
  end
end

[mengqing]

@akshaysharma096 No, it is just a quick patch, sorry. You need to *class_eval Variant model too based on my override of Blob model

Sorry, but if you can not override Variant, then the patch on ActiveStorage::Service::S3Service not mean.
Because in fact, variants use very often.
If I use the line:
<%= image_tag @course.image.service_url %>
It's work, but if I use:
<%= image_tag @course.image.variant(resize: "850x480").service_url %>
It not working.

I've commented in the gist with a version that works with variants / previews

https://gist.github.com/dinatih/dbfdfd4e84faac4037448a06c9fdc016#gistcomment-2940505

[dhh]

Your mind boggled that people who write and share software had different needs than you do? I’m not sure that open source is a safe environment for you then. It can’t be healthy to have your mind boggled so frequently 😄

Active Storage didn’t set up to be a replacement for anything. It, like every other major framework in Rails, was extracted from actual use in a real application. It wasn’t designed on a spec of what other gems or other users might offer or expect.

That’s the beauty of open source! We share what we built to serve our own needs, others then share their enhancements based on their needs, and together we get to enjoy the combined fruits of our labor.

I spoke about how I view that whole process at RailsConf this year. Including a diagnosis and prescription for the acute case of vendoritis you’re exhibiting. Feel free to self medicate for an hour or so: https://m.youtube.com/watch?v=VBwWbFpkltg

Then, hopefully endowed with a healthier perspective, you can come back and help us make software together. Free of the misconception that you’re a customer who bought something and is owed anything ❤️

[dhh]

You're right. Working PRs is not the place to pontificate on how your mind boggles that "basic features" aren't implemented as you would have done them. Thanks for your understanding ✌️

[8vius]

I tried @dinatih solution (thanks for posting it btw) but it doesn't seem to work for me. I'm still getting my image links expired. Anyone else having the same problem?

[yboulkaid]

This issue has been addressed in https://github.com/rails/rails/pull/36729 and will be available in rails 6.1

So we now can close this issue, thank @peterzhu2118 and marvel at the beauty of open source :)

[collimarco]

@yboulkaid @peterzhu2118 Wow, that's great news! Will It be possible to migrate a bucket from private to public?

[peterzhu2118]

@collimarco Yes, you should be able to change the access level of a bucket in S3, GCS, or Azure. See the Public access section on the edge guides that contains the instructions on how to change the access level for your bucket.

[collimarco]

Here's my final solution:
https://stackoverflow.com/a/59107484/51387

It works great for my use case, where we have public pages with many images.

[dhh]

Maybe add this to the ASt guide?

On Fri, Nov 29, 2019 at 7:32 AM Marco Colli [e-mail]
wrote:

Here's my final solution:
https://stackoverflow.com/a/59107484/51387

It works great for my use case, where we have public pages with many
images.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rails/rails/issues/31419?email_source=notifications&email_token=AAAAVNJKQBIOMPUWTWLKMCTQWEYY7A5CNFSM4EH5T2D2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFPD6UA#issuecomment-559824720,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAAVNMR3CQXPYEILDUXR6LQWEYY7ANCNFSM4EH5T2DQ
.

[collimarco]

Update: my solution above works, but I discovered an issue: currently you cannot block all public access to the bucket from s3 settings, because Rails tries to set public acl on uploads.

I am looking to build a similar solution that also allows to block all public access to bucket. In order to achieve that I need to find a solution to this question:

If I keep Rails private behavior / private bucket, is there any way to get a non-signed URL to a variant? e.g. Something like service_url(signed: false) would be great. Then you can add a CDN in front of the bucket, block all public access to the bucket, allow the IP ranges of the CDN (e.g. Cloudflare) in the bucket policy and finally use the non-signed URLs

[collimarco]

Solved! I can use the Rails default private bucket / private acl and simply allow the Cloudflare IP ranges in the bucket policy. Then when I wan to display the image from the CDN, without signatures, I use:

"https://storage.example.com/" + variant.key

storage.example.com points to Cloudflare, which proxies to the actual bucket on s3. I have updated my tutorial here.

[rails-bot[bot]]

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 6-0-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

[mingshenggan]

Did something similar for AWS Cloudfront.

Final approach: Same as @collimarco

Configure cloudfront to handle assets

1. Create distribution
2. Create origin group(s) - if you have 2 paths for cloudfront, you'll need 2 origin groups. For me, I did one for /assets and another for S3. For S3, I created an origin access identity so that nobody can directly access S3, instead, everything has to go through cloudfront and CF will query S3 to cache.
3. Modify behaviors. That's like a routing table, put the obvious routes first, then a default route.

Configure application to return url
This part is quite foolproof

1. Configure asset hosts, to what I have done so far:

# config/environments/production.rb
config.action_controller.asset_host = ENV["CLOUDFRONT_ENDPOINT"]
config.action_mailer.asset_host = ENV["CLOUDFRONT_ENDPOINT"]

1. For home-grown assets, use asset_path. For S3, use "https://#{ENV["CLOUDFRONT_ENDPOINT"]}/#{activestorage_image.key}"

Other thoughts

• I considered signing all my cf url, that is possible if you reference what activestorage did for s3. Basically using Aws::CloudFront directly. If I were to need this, I may build it as a separate service in activestorage. Pretty much overkill for me right now so I skipped it.