Shrine: S3 url method (with a provided host) strips bucket name from path in situations where it's required

[dkulchenko]

I noticed an issue when I was testing Shrine against my minio S3 endpoint.

Because force_path_style has to be enabled for minio to work, the bucket name is part of the path rather than part of the host. Also, my minio install sits within a VM, so it thinks public URLs for files are http://127.0.0.1:9000/mybucketname/etcetc but its external URL is actually http://192.168.3.1:9000/mybucketname/etcetc. So to make this work, I've set the host in Shrine's default_url_options to http://192.168.3.1:9000/mybucketname/.

The problem is that when I run .url on any uploaded files, the public URLs returned by Shrine are http://192.168.3.1:9000/myfile instead of http://192.168.3.1:9000/mybucketname/myfile.

I traced the issue back to line 325 in lib/shrine/storage/s3.rb - it seems to be stripping the bucket name from the path of the URL if the bucket name is not in the host, which it shouldn't be doing in this case. Changing the "unless" to an "if" on this line fixes this issue.

But here's the rub: this line was probably written the way it was for a situation where the public URL from the perspective of the endpoint is http://mys3endpoint.dev/mybucket/file but it should be publicly accessible from http://mycdnendpoint.dev/file.

So I'm not sure how to reconcile the two situations - maybe an additional configuration param that could be added to the URL options to let the url method know whether to strip the bucket from the URI or not?

[janko]

But here's the rub: this line was probably written the way it was for a situation where the public URL from the perspective of the endpoint is http://mys3endpoint.dev/mybucket/file but it should be publicly accessible from http://mycdnendpoint.dev/file.

Yes, that's almost correct. Most buckets will by default generate S3 URLs where bucket name is in the URL host, but buckets with dots (.) inside them have to fall back to the "path style" S3 URL (the aws-sdk-s3 gem handles this automatically). However, when a CDN is used, it will apparently handle both cases correctly, so the CDN URL should never include the bucket name in the path, even if it has dots inside it.

So I'm not sure how to reconcile the two situations - maybe an additional configuration param that could be added to the URL options to let the url method know whether to strip the bucket from the URI or not?

I think the correct behaviour would be to respect the :force_path_style option when it was explicitly given, and in that case not do the substitution. That would make sense for me, and shouldn't cause unexpected behaviour. I'll go ahead and make that change.

Note that if you're using Minio, you want to point the whole AWS S3 client to your Minio instance:

Shrine::Storage::S3.new(endpoint: "http://192.168.3.1:9000", **other_options)

Otherwise files would still get uploaded to AWS S3, and when you'd attempt to retrieve them from Minio they would be missing. The :endpoint option makes sure that the uploads, downloads, presigns etc. also all go through the Minio server.

[janko]

This change has been reverted in master, because it causes a regression for folks who are using an endpoint that requires :force_path_style, but need to generate object URLs with host without the bucket name in the URL path.

However, it's now possible do specify a :host with a path prefix, so this code should now work:

Shrine.plugin :default_url_options, store: { host: "http://192.168.3.1:9000/mybucketname/" }

This is still on master, it will be released in Shrine 2.13.0.