Ingress-nginx: Multiple ingress definitions for the same host with rewrite-target lead to ordering problems

[ankon]

I'm trying to use kube-lego and the nginx-ingress-controller together with the rewrite-target annotation, but I'm running an issue: the way the location is build for the ingress gets picked always, and kube-lego's acme-challenge location is never selected.

nginx.conf snippets for illustration:
~~~~
server {
server_name media.DOMAIN;
listen [::]:80 proxy_protocol;
vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;

    location /.well-known/acme-challenge {
        set $proxy_upstream_name "kube-lego-nginx-8080";
        ....
    }

    location ~* / {
        ....
            rewrite /(.*) /media/$1 break;
            proxy_pass http://app-http;
    }

~~~~

This was generated by this ingress:
~~~~
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: media
annotations:
ingress.kubernetes.io/class: nginx
ingress.kubernetes.io/rewrite-target: /media
kubernetes.io/tls-acme: "true"
spec:
tls:

• hosts:
  
  
  
  • "media.DOMAIN"
    
    secretName: media-tls
    
    rules:
  
  
• host: "media.DOMAIN"
  http:
  paths:
  
  
  
  • path: /
    
    backend:
    
    serviceName: app
    
    servicePort: http
    
    ~~~~
  
  

Is there any way I can use "disable" the use of the "~*" operator, and still map from frontend / to backend /media/?

(FWIW: I'm trying right now to build a container from master, to try the app-root annotation)

[ankon]

app-root doesn't work, and I think the description in the readme is a bit misleading: app-root introduces a redirection only it seems.

For a test I nuked out most of the buildLocation function, and got my intended behavior:

diff --git a/controllers/nginx/pkg/template/template.go b/controllers/nginx/pkg/template/template.go
index 90ae4fe..cf19bc6 100644
--- a/controllers/nginx/pkg/template/template.go
+++ b/controllers/nginx/pkg/template/template.go
@@ -193,28 +193,14 @@ func buildSSLPassthroughUpstreams(b interface{}, sslb interface{}) string {
 }

 // buildLocation produces the location string, if the ingress has redirects
-// (specified through the ingress.kubernetes.io/rewrite-to annotation)
+// (specified through the ingress.kubernetes.io/rewrite-target annotation)
 func buildLocation(input interface{}) string {
    location, ok := input.(*ingress.Location)
    if !ok {
        return slash
    }

-   path := location.Path
-   if len(location.Redirect.Target) > 0 && location.Redirect.Target != path {
-       if path == slash {
-           return fmt.Sprintf("~* %s", path)
-       }
-       // baseuri regex will parse basename from the given location
-       baseuri := `(?<baseuri>.*)`
-       if !strings.HasSuffix(path, slash) {
-           // Not treat the slash after "location path" as a part of baseuri
-           baseuri = fmt.Sprintf(`\/?%s`, baseuri)
-       }
-       return fmt.Sprintf(`~* ^%s%s`, path, baseuri)
-   }
-
-   return path
+   return location.Path
 }

 func buildAuthLocation(input interface{}) string {

I cannot find a description for the various ingress-related annotations though, so I'm pretty sure I broke something else now.

What I think should happen here is that the controller should avoid any issues with the differences in matching order between regex locations and regular locations, and either use simple prefix matching consistently, or use regexes everywhere. Mixing things up leads to very confusing behavior, and it is extremely hard to describe and to reason about. I found https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms very helpful in understanding more about nginx's behavior here.

[ankon]

Here's a "self-contained" test for this issue:

1. Create echoserver deployment/service:
   ```yaml
   kind: Deployment
   apiVersion: extensions/v1beta1
   metadata:
   name: echoserver
   spec:
   replicas: 1
   template:
   metadata:
   labels:
   app: echoserver
   spec:
   containers:
   - name: echoserver
   image: gcr.io/google_containers/echoserver:1.4
   ports:
   - name: http
   containerPort: 8080

---

kind: Service
apiVersion: v1
metadata:
name: echoserver
spec:
ports:
- name: http
port: 8080
targetPort: http
selector:
app: echoserver
```

1. Create the ingress definitions:
   ```yaml
   kind: Ingress
   apiVersion: extensions/v1beta1
   metadata:
   name: root-ingress
   annotations:
   "ingress.kubernetes.io/rewrite-target": "/some-rewrite"
   spec:
   rules:
   
   
   
   • host: "ingress.example.com"
     
     http:
     
     paths:
     
     
     
     
     
     • path: "/"
       
       
       backend:
       
       
       serviceName: echoserver
       
       
       servicePort: http
     
     
     
   
   

---

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: path-ingress
spec:
rules:
- host: "ingress.example.com"
http:
paths:
- path: "/path"
backend:
serviceName: echoserver
servicePort: http
```

The expected outcome would be that querying for https://ingress.example.com/path/test.html would report that the path-ingress was used, i.e. in the output of the echoserver one would see "real path=/path/test.html". When querying for another location the root-ingress should be used -- in the echoserver output the "real path=/some-rewrite/...".

Output when testing this:
~~~~
$ curl -v --resolve ingress.example.com:80:192.168.99.100 http://ingress.example.com/path/test.html

• Added ingress.example.com:80:192.168.99.100 to DNS cache
• Hostname ingress.example.com was found in DNS cache
• Trying 192.168.99.100...
• TCP_NODELAY set
• Connected to ingress.example.com (192.168.99.100) port 80 (#0)

GET /path/test.html HTTP/1.1
Host: ingress.example.com
User-Agent: curl/7.51.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx/1.11.10
< Date: Tue, 11 Apr 2017 13:04:52 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
<
CLIENT VALUES:
client_address=172.17.0.1
command=GET
real path=/some-rewrite/path/test.html
query=nil
request_version=1.1
request_uri=http://ingress.example.com:8080/some-rewrite/path/test.html

SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001

HEADERS RECEIVED:
accept=/
connection=close
host=ingress.example.com
user-agent=curl/7.51.0
x-forwarded-for=::ffff:192.168.99.1
x-forwarded-host=ingress.example.com
x-forwarded-port=80
x-forwarded-proto=http
x-real-ip=::ffff:192.168.99.1
BODY:

• Curl_http_done: called premature == 0
• Connection #0 to host ingress.example.com left intact
  -no body in request-
  ~~~~

nginx.conf server section:
~~~~
server {
server_name ingress.example.com;
listen [::]:80;
vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;

    location /path {
        set $proxy_upstream_name "development-echoserver-http";

        port_in_redirect off;
        client_max_body_size                    "1m";

        proxy_set_header Host                   $host;

        # Pass Real IP
        proxy_set_header X-Real-IP              $remote_addr;

        # Allow websocket connections
        proxy_set_header                        Upgrade           $http_upgrade;
        proxy_set_header                        Connection        $connection_upgrade;

        proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host       $host;
        proxy_set_header X-Forwarded-Port       $pass_port;
        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

        # mitigate HTTPoxy Vulnerability
        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
        proxy_set_header Proxy                  "";

        # Custom headers

        proxy_connect_timeout                   15s;
        proxy_send_timeout                      60s;
        proxy_read_timeout                      600s;

        proxy_redirect                          off;
        proxy_buffering                         off;
        proxy_buffer_size                       "4k";

        proxy_http_version                      1.1;
        proxy_pass http://development-echoserver-http;
    }
    location ~* / {
        set $proxy_upstream_name "development-echoserver-http";

        port_in_redirect off;
        client_max_body_size                    "1m";

        proxy_set_header Host                   $host;

        # Pass Real IP
        proxy_set_header X-Real-IP              $remote_addr;

        # Allow websocket connections
        proxy_set_header                        Upgrade           $http_upgrade;
        proxy_set_header                        Connection        $connection_upgrade;

        proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host       $host;
        proxy_set_header X-Forwarded-Port       $pass_port;
        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

        # mitigate HTTPoxy Vulnerability
        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
        proxy_set_header Proxy                  "";

        # Custom headers

        proxy_connect_timeout                   15s;
        proxy_send_timeout                      60s;
        proxy_read_timeout                      600s;

        proxy_redirect                          off;
        proxy_buffering                         off;
        proxy_buffer_size                       "4k";

        proxy_http_version                      1.1;
rewrite /(.*) /some-rewrite/$1 break;
proxy_pass http://development-echoserver-http;

    }
}

~~~~

This behavior is reproducible for me with both &{NGINX 0.9.0-beta.3 git-3dd7461 [email protected]:ixdy/kubernetes-ingress.git} and &{NGINX 0.9.0-beta.2 git-7013a52 [email protected]:ixdy/kubernetes-ingress.git}, using this controller manifest:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    infrastructure: nginx-ingress-controller
  name: nginx-ingress-controller
  namespace: development
spec:
  replicas: 1
  template:
    metadata:
      labels:
        infrastructure: nginx-ingress-controller
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=$(POD_NAMESPACE)/nginx
        - --watch-namespace=$(POD_NAMESPACE)
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: nginx-ingress-controller
        ports:
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      hostNetwork: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 60

When I remove the root-ingress for testing everything starts working as soon as the nginx is reloaded:
~~~~
$ kubectl delete ing root-ingress; sleep 15; curl -v --resolve ingress.example.com:80:192.168.99.100 http://ingress.example.com/path/test.html
ingress "root-ingress" deleted

• Added ingress.example.com:80:192.168.99.100 to DNS cache
• Hostname ingress.example.com was found in DNS cache
• Trying 192.168.99.100...
• TCP_NODELAY set
• Connected to ingress.example.com (192.168.99.100) port 80 (#0)

GET /path/test.html HTTP/1.1
Host: ingress.example.com
User-Agent: curl/7.51.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx/1.11.10
< Date: Tue, 11 Apr 2017 13:15:54 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
<
CLIENT VALUES:
client_address=172.17.0.1
command=GET
real path=/path/test.html
query=nil
request_version=1.1
request_uri=http://ingress.example.com:8080/path/test.html

SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001

HEADERS RECEIVED:
accept=/
connection=close
host=ingress.example.com
user-agent=curl/7.51.0
x-forwarded-for=::ffff:192.168.99.1
x-forwarded-host=ingress.example.com
x-forwarded-port=80
x-forwarded-proto=http
x-original-uri=/path/test.html
x-real-ip=::ffff:192.168.99.1
x-scheme=http
BODY:

• Curl_http_done: called premature == 0
• Connection #0 to host ingress.example.com left intact
  -no body in request-
  ~~~~

[jurgenweber]

I am having the same sort of issue, I need the nginx.conf/host ingress locations to honour the order they are in, in the ingress configuration.

I have potentially overlapping ingress rules and I need first match to win just like in nginx. I am finding that all of the locations for one host are ordered in reverse alphabetical order.

for example:

kubectl exec -it ingress-nginx-grep location /etc/nginx/nginx.conf -n kube-ingress
        location ~* /ws/(.*)$ {
        location ~* /user/(.*)$ {
        location ~* /thumbnail/(.*)$ {
        location ~* /search/(.*)$ {
        location ~* /photos/(.*)$ {
        location ~* /find/(.*)$ {
        location ~* /(find|search)/(.*)/(a|b|c|d|e|f|g|h)/(.*)$ {

The order they appear in the ingress conf:

      - backend:
          serviceName: directory
          servicePort: 80
        path: /search/(.*)/(a|b|c|d|e|f|g|h)/(.*)$
      - backend:
          serviceName: find
          servicePort: 80
        path: /find/(.*)$
      - backend:
          serviceName: photos
          servicePort: 80
        path: /photos/(.*)$
      - backend:
          serviceName: search
          servicePort: 80
        path: /search/(.*)$
      - backend:
          serviceName: thumbnail
          servicePort: 80
        path: /thumbnail/(.*)$
      - backend:
          serviceName: user
          servicePort: 80
        path: /user/(.*)$
      - backend:
          serviceName: ws
          servicePort: 80

So I need serviceName: directory to be before find, but due to it reordering....... I am broken.

[aledbf]

I have potentially overlapping ingress rules and I need first match to win just like in nginx.

The first created Ingress rules wins. If you create the same rule (host/path) only the first one will be used

I am finding that all of the locations for one host are ordered in reverse alphabetical order.

NGINX process the request as defined in the cfg file. We use reverse alphabetical order to allow things like /foo/bar/baz and /foo/bar

[jurgenweber]

Yeah, I know.

I am finding that all of the locations for one host are ordered in reverse alphabetical order.

NGINX process the request as defined in the cfg file. We use reverse alphabetical order to allow things like /foo/bar/baz and /foo/bar

I know... How can i disable that? I don't think ingress should be thinking for us here and we should be able to set the order or atleast honour the order it was configured in.

[aledbf]

I know... How can i disable that?

This is not possible.

I don't think ingress should be thinking for us here and we should be able to set the order or atleast honour the order it was configured in.

We honour the order using the resource version of the ingress rule.

[jurgenweber]

I don't understand what that means. Looking at the code I see no order manipulation, etc but that could just be my lack of experience and understanding of go.

Also I did not note/forgot to mention that I created my own nginx.conf template and added to the location a '~*' which basically means every path in my ingress spec is now a regex (which is fine by me and suits my purposes). I know there are other tickets about implementing option regex on paths but nothing seems to have eventuated yet.

[ohmystack]

@aledbf The mandatory reverse alphabetical order might not be a good idea.

IMHO, the ordering issue is caused by allowing to define rules with the same "hostname" in multi ingress configurations.

e.g.
ingress1.yaml

  ...
  rules:
  - host: "ingress.example.com"
      - path: "/path1"
   ...

ingress2.yaml

  ...
  rules:
  - host: "ingress.example.com"
      - path: "/path2"
   ...

The "host" of ingress1 and ingress2 are the same. This causes the merge problem.
Ingress controller will be confused when merge these 2 configs, what the correct order should be.

One of the solution of this problem is to disable this. One "hostname" is allowed to be defined only in one ingress. Then, we can generate the ngnix.conf rules with the user defined order. No more need for "reverse alphabetical order" because there is no merge problem.

In the real world, this solution also sounds good. An ingress config can define rules for one or more hostnames, and a hostname resource should belong to one ingress configuration. In this case, the admin can easily get the whole picture of all the paths under a hostname by looking at that ingress config. If anyone wants to modify the rules under a hostname, he/she must have the permission of modifying that ingress.

I think the AWS ELB also doesn't allow duplicated hostnames among configs, but I haven't found any specific words in this docs.

@aledbf @jurgenweber @ankon
How do you think about this solution?

[aledbf]

@jurgenweber this are the two places where we sort and avoid to overwrite an existing path.
https://github.com/kubernetes/ingress/blob/master/core/pkg/ingress/controller/controller.go#L609
https://github.com/kubernetes/ingress/blob/master/core/pkg/ingress/controller/controller.go#L660

[aledbf]

@aledbf The mandatory reverse alphabetical order might not be a good idea.

Why you say that?

IMHO, the ordering issue is caused by allowing to define rules with the same "hostname" in multi ingress configurations.

This is not the issue here. You can have multiple ingress rules for the same hostname with different paths.

[aledbf]

In the real world, this solution also sounds good.

Please don't take this in a wrong way but this ingress controller provides a real world solution.
I use this controller with more than 700 rules with more than 250 in the same hostname.

[ohmystack]

@aledbf Sorry, I didn't realize the sort of ingressByRevision.
After sorting by (*extensions.Ingress).ResourceVersion, the list of Ingress objects is ordered from old to new. If many rules points to the same hostname in different Ingress objects, the admin has to update the one to make its order to the last.

e.g.
We define 2 Ingress like this

ingress1:
  ResourceVersion: 1
  host: "same.host"
  path: /a

ingress2:
  ResourceVersion: 2
  host: "same.host"
  path: /b

ingress1 is older than ingress2, so the nginx.conf order will be

"same.host"
  /a       <- 1
  /b       <- 2

Now, if the admin wants to swap the order, the admin has to update "ingress1" to make it newer than "ingress2",

ingress1:
  ResourceVersion: 3
  host: "same.host"
  path: /a

Then, the ngnix.conf will be

"same.host"
  /b       <- 2
  /a       <- 3

It is easy for just 2 Ingress.
But if there are 3 or more Ingress objects pointing to the same hostname, it will be hard to swap the order.

e.g.
From

"same.host"
  /a
  /b
  /c
  /d

To

"same.host"
  /b
  /a
  /c
  /d

I have to update 3 times (/a -> /c -> /d)
I'm not sure whether I understand the usage of ingressByRevision correctly this time.

[aledbf]

If many rules points to the same hostname in different Ingress objects, the admin has to update the one to make its order to the last.

Why? I don't understand what is the issue you are trying to explain or achieve controlling the order

"same.host"
  /b
  /a
  /c
  /d

If you define that in different rules it just works. The order is just to allow things like

"same.host"
  /b
  /a/another-app-inside-a
  /a
  /c
  /d/v2
  /d

Now, if the admin wants to swap the order, the admin has to update "ingress1" to make it newer than "ingress2",

Why you want do that?

[aledbf]

@ohmystack please keep in mind that the order using the revision is to avoid accidental overwrite of paths

[ohmystack]

@aledbf I was trying to solve @jurgenweber 's problem in this comment .
I guess he defined the rules in seperated Ingress objects. Nor, he wouldn't get the order issue.
Thanks for your intruduction, and let's wait for @jurgenweber 's reply.

[jurgenweber]

Hi

An interesting read but no, all of the configuration found in my previous comment here is one ingress under one hostname.

So kube version 1.6.6, nginx-ingress-controller:0.9.0-beta.10. I also have external-dns and kube-lego installed.

here is the top half of my config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: www-beta
  annotations:
    external-dns.alpha.kubernetes.io/controller: "dns-controller"
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/force-ssl-redirect: "true"
  namespace: engineering

spec:
  tls:
    - secretName: beta-www
      hosts:
        - www-beta.k8s.example.com.au

  rules:
     - host: www-beta.k8s.example.com.au
      http:
        paths:

this is the only ingress on my kube cluster as we are still getting ready for production.

I think the key take away here is @aledbf comment:

@jurgenweber this are the two places where we sort and avoid to overwrite an existing path.

that is the thing, I want the exact opposite behaviour. Not surprising the current result and competing interests. I now understand and appreciate why it is the way it is.

[nikitsrj]

@aledbf
I have created two different namespaces for different environment. one is devops-qa and another is devops-dev. I created two ingress in different namespaces. So while creating ingress of qa env in devops-qa namespace, the rules written inside ingress of qa is working fine. Means I am able to access the webpage of qa env. The moment I will create the ingress of dev env in devops-dev namespace, I will be able to access the webpage of dev env but wont be able to access the webpage of qa. And when I delete the dev ingress then again I will be able to access the qa env website

Below is the ingree of both dev and qa env.

Dev Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress-dev
  namespace: devops-dev
spec:
  tls:
  - hosts:
    - cafe.example.com
    secretName: default-token-jhtx3
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: miqpdev-svc
          servicePort: 80

QA Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress-qa
  namespace: devops-qa
spec:
  tls:
  - hosts:
    - cafe.example.com
    secretName: default-token-bjg3z
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /greentea
        backend:
          serviceName: greentea-svc
          servicePort: 80
      - path: /blackcoffee
        backend:
          serviceName: blackcoffee-svc
          servicePort: 80

The token mentioned in the ingress file is of each namespace. How can i run both the ingress and will be able to get all the websites deployed in both dev and qa env ? I am using Nginx Plus controller which is deployed right now in qa namespace but able to read the ingress of qa as well as dev namespace.Pls help me in this issue .. I am struggling from 1 whole day.

[jackzampolin]

@jurgenweber Did you find a solution for this?

[jurgenweber]

no... I ended up removing some edge cases, etc and found a sweet spot where the reverse alphabetical order worked for me.

I honestly feel I am not asking for anything uncommon, I think the ordering should be removed (in that the current behaviour should remain for backwards compatibility and annotations should be added to change the behaviour) or I have seen many requests/issues for the nginx controller supporting regex paths, those issues and resulting PR's should answer/solve the question of order(ing) as well.

[jackzampolin]

I've found out what I'm trying to do doesn't work on GCE. Womp womp

[MindFreeze]

I strongly believe one should be able to set the order of rules explicitly or at least disable regex matching . Consider this example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: main-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: example.com
    http:
      paths:
        - path: /a
          backend:
            serviceName: a-service
            servicePort: 80
        - path: /b
          backend:
            serviceName: b-service
            servicePort: 80

This creates rules like this:

• location ~* /b
• location ~* /a

Now suppose you have a route in your app like this /a/foo/b . This will first match the rule for b-service which just wrong.

UPDATE: As a very hacky workaround I am going to set /b to /0?b so it appers after /a but this is just ridiculous and even finding out about the ordering requires reading the ingress-nginx code.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[jurgenweber]

is there any motivation to pick this concept up?

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[wz185]

@ankon Have you eventually worked out the ingress file? Would switching the order of two definitions solve the issue? I am facing the same issue you had and interested in to know your solution.

[ankon]

@wz185 "worked out" -- In a way. We're using our own fork of the ingress controller with #565 applied.

[Bessonov]

/reopen

[k8s-ci-robot]

@Bessonov: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Bessonov]

/remove-lifecycle rotten