Caddy: Are you interested in an s3 proxy directive?

[wolfeidau]

I have a pile of stuff in s3 I would like to serve using caddy and letsencrypt.

So the server will run under an IAM profile and proxy GET requests to an s3 bucket.

Cheers.

[mholt]

Sorry it took so long for me to respond. Is this something that can't be done with the proxy directive?

[alexandreteles]

@mholt said:
Is this something that can't be done with the proxy directive?

Yes, you can actually do that with the proxy directive. If you want, for example, to serve assets from a S3 bucket, it should be something like that:

example.com {
  proxy /assets https://your_bucket.s3.amazonaws.com/ {
    proxy_header       Host $s3_bucket
    proxy_header       Authorization ''
  }
}

It should be a bit messier for private files on S3 as Caddyfile doesn't support variables. Users will know that your site uses S3 as Caddy can't hide or ignore headers :disappointed:

To be honest, the proxy directive need some improvements to work better with S3. Introducing syntax for hiding and ignoring headers, to intercept errors and set resolvers. Regex for the path matching should be good if you want to serve some types of files across every directory too.

[wolfeidau]

Yes just to be clear I have a bucket I want to serve up content using caddy, but I don't want to make the bucket public to the world, hence this request.

outside <-letsencrypt-> caddy <-auth'd s3-> bucket

Just wanted to know what others thought of this.

Cheers

[mholt]

@wolfeidau and @alexandreteles - thanks for your comments! I'm going to close this since there's not much actionable development discussion happening in this issue, but I have linked to this issue from a thread in our community forum where we are aggregating ideas/requests for Caddy plugins.

If you want to continue discussing the plugin before development begins on it, feel free to start a new topic on the forum. :+1: Hope to see you participate with us there!

[jahands]

I would really like this. Posted my use case on the forum.

[afdecastro879]

Hello everyone!
I've been facing the same problem explain in this issue to proxy non private assets hosted in s3. If someone else interested in I have sort of an idea on how to work around it creating a new directive on the proxy plugin to allow this issue be solved. It's working!! but needs to be refactored.

I also have seen @jahands entry on the discussion forum in caddy is already close :'( so no idea is the problem have been solved or is still on discussion.

To complement the issue regarding s3 private hosting (that the proxied request needs to be signed), I propose to create the proxy directive to be able to sign upstream requests independently of the provider. Open to PR

Regards

[tobya]

We are always happy with a PR!

Could this be implemented as 3rd party plugin rather than a proxy subdirective, or does it have to be in the main code?

Start with a PR and we can take it from there.

[afdecastro879]

@tobya Thank you very much for your answer. In the specific case of signing s3 requests the request the reverse proxy is making needs to be modified directly. I don't really know if there's a way to intercept that call from another 3rd party plugin. Do you have any idea?

On the other hand, I'll organise and refactor the code I have to make a PR and start from there.

Cheers!

[mholt]

I'd prefer that if the proxy directive is modified, that the solution be generic -- not specific to S3.

Otherwise, I think this should be a plugin or it should be made to generalize.

[CMCDragonkai]

Was looking for this, found caddy s3browser, however that isn't a proper reverse proxy since it doesn't send authenticated/signed requests to S3.

[danlsgiga]

What I ended up doing was using Caddy as a proxy for S3 and limiting web access to it by applying the following S3 bucket policy to allow only requests from a specific vpc endpoint in my VPC.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Access-to-specific-VPCE-only",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::mybucket/*",
            "Condition": {
                "StringEquals": {
                    "aws:sourceVpce": "vpce-1234567890"
                }
            }
        }
    ]
}

This way, Caddy running inside my VPN is able to proxy to the S3 Webhosting endpoint.

The downside is the S3 endpoint is available only via http