Ingress-nginx: Use NGINX ingress controller with GCS bucket as a backend

[pragmasoft-ua]

FEATURE REQUEST

This is basically a duplicate of the same open issue in ingress-gce. We need the ability to serve static files from GCS buckets, and currently we cannot set up Ingress to proxy requests to GCS. Also, it would be good to be able to use GCS bucket as a default backend.

I searched for the following keywords: GCS, ingress, bucket, nginx
I've found this issue in ingress-gce https://github.com/kubernetes/ingress-gce/issues/33
And this actually is the same, except it is for Ingress NGINX

If there's some temporary workaround I'd appreciate for help.

[aledbf]

@pragmasoft-ua not sure if this will work. Please try:

kind: Service
apiVersion: v1
metadata:
  name: proxy-to-gcs
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ExternalName
  externalName: storage.googleapis.com

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: proxy-to-gcs
  annotations:
    # rewrite the path we send to gcs
    ingress.kubernetes.io/rewrite-target: /[BUCKET NAME]
    ingress.kubernetes.io/configuration-snippet: |
      # we cannot pass Host header
      set $h "";
      proxy_set_header Host                   $h;
spec:
  rules:
  - host: foo.bar
    http:
      paths:
      - path: /
        backend:
          serviceName: proxy-to-gcs
          servicePort: 80

[pragmasoft-ua]

Wouldn't this break other backends? I plan to proxy rest api through this ingress as well, thus I have more than one backend

[pragmasoft-ua]

Is it possible to configure rewrite rules per backend?

[aledbf]

The annotations only applies to one ingress rule. If you create multiple rules with the same host and different paths this works ootb

[aledbf]

Closing. Please reopen if you have more questions.

[pragmasoft-ua]

Thanks Manuel

I managed to get this working on this image only:

quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.16

later versions beta.19 and release do not work this way.

[pragmasoft-ua]

It seems I cannot reopen this issue myself

[pragmasoft-ua]

Also, I was unable to achieve, that rewrite-target rule was only applied to GCP backend - it mangles requests to all backends - I tried same yaml with two different host rules and two different ingresses - neither works.

[shaneog]

Any update on this? I need to serve some large, regularly recreated files and would prefer to do this from a storage bucket on Google Cloud Storage or Amazon S3.

[ops-ninja]

@aledbf could you post a snippet of what you mean? It seems people are reporting they can't replicate what you're suggesting.

[codepitbull]

I think the discussion in here is a little misleading as there is one crucial part missing.
First of all, using Ingress as a reverse proxy for an arbitrary HTTP service works with the following config:

kind: Service
apiVersion: v1
metadata:
  name: proxy-static
spec:
  type: ExternalName
  externalName: storage.googleapis.com

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: proxy-static
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/secure-backends: "true"
spec:
  rules:
  - http:
      paths:
      - path: /crm360-tester
        backend:
          serviceName: proxy-static
          servicePort: 443

As you might realize I use the GCS-URL as a backing storage. This only works if the content inside the buckets has been made public. You could put any other https based service in the above description (I tested with elastic search) and it would work.
The special part about GCS and hosting stuff from there is that you don't want GCS to be publicly readable.
That's our case. We want every access to the buckets to go through our auth-scheme which is handled by Ingress.
In that case you will have to go through IAM-based authentication to actually access GCS. That isn't implemented in Nginx-Ingress.
To sum it up:
Using GCS as a backing storage works if the content has been made public (=readable directly from the internet).
Using GCS without making it public can't work unless Nginx-Ingress learns to use the IAM-credentials to use the GCS-servide.

Does that sound correct?

[pragmasoft-ua]

The showstopper for me was that nginx.ingress.kubernetes.io/rewrite-target annotation is global per ingress, whereas I used it for both backend api and frontend, and only frontend needed to be rewritten.

[aledbf]

@pragmasoft-ua just create two ingress rules, one for the frontend and another for the backend api and only apply the annotation to the frontend rule

[pragmasoft-ua]

I tried this earlier and reported on this above:

Thanks Manuel I managed to get this working on this image only: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.16 later versions beta.19 and release do not work this way.

Will probably need to try again, that was quite a long ago

[codepitbull]

But how did you get around the GCS-authentication? Or did you just make the bucket contents public?

[pragmasoft-ua]

I just made them public

[codepitbull]

Thanks for clarifying.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[JessicaGreben]

Just thought I would add my two cents here in case its helpful to anyone in the future. Thanks @codepitbull's for your example and information about making the bucket public. That was very helpful.

For my case I needed to add this annotation to the ingress as well:

nginx.ingress.kubernetes.io/upstream-vhost: "storage.googleapis.com"

So that my ingress looked like this:

kind: Service
apiVersion: v1
metadata:
  name: google-storage-buckets
spec:
  type: ExternalName
  externalName: storage.googleapis.com

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: proxy-assets-ingress
  annotations:
    kubernetes.io/ingress.class: nginx-ingress
    nginx.ingress.kubernetes.io/rewrite-target: /[BUCKET_NAME]/[BUILD_SHA]
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/upstream-vhost: "storage.googleapis.com"
spec:
  rules:
  - http:
      paths:
      - path: /path/to/static/assets
        backend:
          serviceName: google-storage-buckets
          servicePort: 443

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[markfermor]

For anyone that comes here, the method for connecting to externalName resources with HTTPS has been changed from versions 0.20.0. Therefore the below was deprecated and removed

nginx.ingress.kubernetes.io/secure-backends: "true"
(removed in PR)

Instead you now use
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
(backend-protocol)

[ghost]

Trying to make exact same thing as @JessicaGreben did. I made following config:

kind: Service
apiVersion: v1
metadata:
  name: google-storage-buckets
spec:
  type: ExternalName
  externalName: storage.googleapis.com

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: website-static-ingress
  annotations:
    kubernetes.io/ingress.class: nginx-ingress
    nginx.ingress.kubernetes.io/rewrite-target: /my-bucket-name
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/upstream-vhost: "storage.googleapis.com"
spec:
  rules:
    - http:
        paths:
          - path: /*
            backend:
              serviceName: google-storage-buckets
              servicePort: 443

But it does not work
I really cannot get, what am I doing wrong
I have ingress w/o endpoint:

I would really appreciate the help!
Thank you in advance!

[Sergey80]

with nginx-ingress-controller:0.24.1

          - path: /
            backend:
              serviceName: google-storage-buckets-service
              servicePort: 443
          - path: /c
            backend:
              serviceName: hello-world-service
              servicePort: 8080

by reaching https://my.ip.add.ress/c - got both outputs: Hello world!bucket content.

https://stackoverflow.com/questions/55779048/kubernetes-external-loadbalancer-pointing-to-cloud-bucket

[CodyEakins]

What about having the ingress-controller pass a token while routing requests to services; is this possible? Can the service do this itself?

If this is possible, you can access resources of private buckets using the provided authorization token.

[Datamance]

@pragmasoft-ua just create two ingress rules, one for the frontend and another for the backend api and only apply the annotation to the frontend rule

The annotations only applies to one ingress rule. If you create multiple rules with the same host and different paths this works ootb

@aledbf you contradict that statement here: https://github.com/kubernetes/ingress-nginx/issues/3122#issuecomment-424055027

The annotations are applied to all the paths in the ingress. We are not going to change that rule. Main reason for this rule is makes everything more complex it should be (how we express to which path we need to apply the annotations, for instance).

When you say two ingress rules, I think you mean to say two ingress resources. "Rules" in this context implies path rules within same file - which is exactly what was confusing @pragmasoft-ua (and myself).

[aledbf]

@Datamance you are right, I mixed the definition of "rule"

The annotations only applies to one ingress rule.

That should be one Kubernetes Ingress resource definition.

We are not going to change that rule.

The context here means the way we process annotations defined in an Ingress. The annotations are applied to all the paths in the Ingress definition.

makes sense?

[Datamance]

@aledbf that does make sense, thank you for clarifying - does the rewrite extend to not just the paths but the default backend, as well?

also had a momentary scare looking at this documentation that says you have to have nginx plus to support externalname services, until I saw that ingress-nginx and kubernetes-ingress from nginx-inc were two different projects... so confusing!

[Datamance]

Also, if I'm splitting between two Resource definitions for the same host and using rewrite on one of them, it looks like nginx.ingress.kubernetes.io/use-regex: "true" becomes implicit for both. Makes sense. Also great that path priority is dealt with intelligently across multiple files!

[zepouet]

@aledbf that does make sense, thank you for clarifying - does the rewrite extend to not just the paths but the default backend, as well?

also had a momentary scare looking at this documentation that says you have to have nginx plus to support externalname services, until I saw that ingress-nginx and kubernetes-ingress from nginx-inc were two different projects... so confusing!

Hello,

Just to understand if it is only with commercial edition for Nginx named Nginx Plus ? Thanks

[aledbf]

@zepouet no, support for externalName services is available since 2017 https://github.com/kubernetes/ingress-nginx/pull/629
Just make sure you are using the ingress controller from this github repository :)

https://kubernetes.github.io/ingress-nginx/deploy/