Openfoodnetwork: Amazon S3 Path Deprecation - URL for buckets have changed

[gweddle-nz]

Description

Amazon are changing the URL for buckets from https://s3.amazonaws.com/bucketname/ to https://bucketname.s3.amazon.com/

I'm not sure if this is a gradual roll out, as my bucket is brand new it may be that the old path-style URL has been disabled only for new accounts. The information I could find was that it is being deprecated by September 30 2020.

https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

Context

Any image uploaded to the S3 bucket is broken due to incorrect URL (might only be for new buckets).

Severity

bug-s2: a non-critical feature is broken, no workaround

Possible Fix

Change URL to new virtual-host style path or give option to switch to it.

[luisramos0]

hello! I think the workaround is to store the images on the server as some other instances are doing.

[lin-d-hop]

Downgrading to S3 as there is a workaround that is particularly well suited to new instances.

[Matt-Yorkley]

I think this could affect some instances, and we have 28 days to fix it before they explode, so bumping back to s2...

[Matt-Yorkley]

Thanks for the heads-up @gweddle-nz :+1:

[Matt-Yorkley]

Sorry, I misread it and thought it was 2019, not 2020. S3 it is.

[kristinalim]

France and I suppose also the other instances use the S3 region US East (N. Virginia). I investigated and found out that this region is the edge case, as in it uses a different _path-style URL_ from the other regions. Search for "US East" in this page for more info: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html @RachL

An easy solution to both the region issue and the deprecation warning is to go for _virtual-hosted–style URL_ already. I found out how to do this so I will take this issue.

@gweddle-nz Sorry this is late. I see you're not using S3 yet. My guess is that you tried to use a bucket in another region. So, the already supported setup is to use US East (N. Virginia). If you're not in a rush and want to use a different region, maybe you can wait for the release with the fix. Changing S3 bucket region is not very straightforward.

[luisramos0]

@RachL I am curious (because I dont understand the context) why is this a priority for FR?

[RachL]

@luisramos0 as far as I understood this is the first step to be able to locate our S3 buckets somewhere else than in the US.
There was a miscommunication when our new server was setup, and with Myriam we thought that everything was located at least in Europe. So we communicated to our users that yes it was Amazon, but at least under European law.
We discovered recently that it was not the case. If it would be only the images, I think it would be easier to handle, but the backups is something that does not sound good in our hubs privacy policies...

[luisramos0]

great, thanks for explaining!

[luisramos0]

I created an issue to upgrade the sdk we use to connect to aws: https://github.com/openfoodfoundation/openfoodnetwork/issues/6221
We may need to get this upgrade done to resolve this issue. I am not sure yet.

[luisramos0]

I read the doc again and Amazon has postponed this deprecation at least one more year (looks like because of buckets with dots in their names) to Sep 2021.

Anyway, there are two problems we want to solve here:

• NZ - new buckets dont work with path style URLs (the style OFN is able to use now)
• FR - we want to switch images and backups to a different region

I dont understand why @kristinalim says this second FR problem can be solved by changing the path style.
I wonder about @kristinalim's comment "I found out how to do this so I will take this issue."
It would probably be useful to get her feedback here for more info about what was the plan.

[sigmundpetersen]

We also have the oldest open issue in OFN which is about S3 signature version and not being able to use S3 in other regions: #511 😄 Maybe we could take that one into account here, or maybe best to split the two cases if they are independent?

[luisramos0]

nice reference @sigmundpetersen :+1:

[Matt-Yorkley]

I think we can use S3 in other regions already, so maybe we can close that issue.

[RachL]

I think we can use S3 in other regions already, so maybe we can close that issue.

@Matt-Yorkley are you sure? Then can we use this prioritization to make the change for FR? Last time I've asked, I've been told this issue was a blocker...

[luisramos0]

ok, I am giving this a try. Just setting up fr-staging to upload images to a bbucket in Europe.

First error I find is: [AWS S3 400 0.279163 0 retries] put_object(:acl=>:public_read,:bucket_name=>"ccs-ofn-staging",:cache_control=>"max-age=31557600",:content_length=>123111,:content_type=>"image/jpeg",:data=>Paperclip::UploadedFileAdapter: chard.jpg,:key=>"public/spree/products/129/original/chard.jpg") AWS::S3::Errors::InvalidRequest The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

From stack overflow: "You are mixing and matching elements from Signature V4 and Signature V2. They are incompatible algorithms. Your policy looks like V4, but your signature and the form you are posting are both V2. AWS4-SHA256-HMAC is Signature V4, and **the error message indicates that your bucket is in a region that supports only V4... so your code needs to all use V4 logic**.

Quite a lot of content in the aws support pages... I wonder if there's something quick we could do to make this work. I still wonder if the aws-sdk 1.11.1 we use, that was released in June 2013, is not blocking this signature v4 thing. Probably yes.

I am thinking we could maybe just dump aws-sdk all together and leap forward to aws-sdk v3 which is modularized and is now called for s3: aws-sdk-s3 v1.83.1 It should be straight forward to make this upgrade. I am investigating this possibility on my computer now.

[luisramos0]

ok, I have spent some time here. I am now convinced we need to move aws-sdk from v1.11.1 to something above v1.31.0 so we can set the parameter s3_signature_version to v4. That will make things work in all regions. We maybe need a new config param for the s3_images_region.
Currently locally, I am able to upload images to eu-west-3 on FR S3 with aws-sdk 1.67.0. I'll continue on this when I have more time :+1:

[luisramos0]

The AWS deprecation has just been postponed again... anyway, TLDR: I have fixed it now.

I have managed to make paperclip use virtualhost paths and also make the upload work for FR staging in the eu-west (this can be done by adding the region to the url, the virtual host path is not required for this).
So I am creating a PR that fixes both https://github.com/openfoodfoundation/openfoodnetwork/issues/511 and also this issue.

Afterwards, we still need to do the migration of the FR images to Europe.

The solution was to upgrade to aws-sdk 1.67 that uses v4 signature by default (fixes 511) and then switch paperclip config to use url :s3_alias_url as described here: https://www.rubydoc.info/github/thoughtbot/paperclip/Paperclip/Storage/S3
This way we change from https://s3.amazonaws.com/bucketname to https://bucketname.s3.eu-west-3.amazonaws.com/