Ingress-nginx: Controller makes cluster level API calls with namespace isolation enabled

[jskeates]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):
namespace isolation
error getting node
node list

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report

NGINX Ingress controller version: 0.21.0

Kubernetes version (use kubectl version): 1.11.6

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): CoreOS
• Kernel (e.g. uname -a):
• Install tools: kops
• Others: Helm

What happened:
When running with the force-namespace-isolation flag, an error appears in the logs: Error getting node <node>: nodes "<node>" is forbidden: User "<serviceaccount>" cannot get nodes at the cluster scope

What you expected to happen:
No error appears, no Cluster level API calls are made.

How to reproduce it (as minimally and precisely as possible):
Deploy the controller with a ServiceAccount without any ClusterRole(Binding), but with the correct Role(Binding) to run inside the Namespace its deployed in. Also start with the --force-namespace-isolation and --watch-namespace= flags. The error appears quickly after startup and every minute or so while running.

Anything else we need to know:
The error seems to come from internal/k8s/main.go.

There is a related issue in Helm's Charts repository (https://github.com/helm/charts/issues/9636) which was fixed by adding a ClusterRole (https://github.com/helm/charts/pull/9637). Unfortunately this isn't suitable for us, as we run a cluster with a soft multi-tenancy setup and teams don't have permission to create Cluster level resources. A second issue was raised describing this problem (https://github.com/helm/charts/issues/11033).

[alexkursell]

I just looked into it. The culprit is https://github.com/kubernetes/ingress-nginx/blob/166dd1a41f7fce7a0c465c11b568dbbeeed31cea/internal/k8s/main.go#L43

The only place this is actually used is for finding the node IP/hostname so that it can be set in the loadBalancer status section on ingress resources. Unfortunately nodes aren't namespaced so a controller without cluster permissions can't get that info. I'll look into seeing if the same information can be gotten from the pod object

edit: It also seems like the --force-namespace-isolation flag does literally nothing. At the very least I'll make a PR deprecating it.

[harpratap]

@alexkursell How about fetching the hostIP from one of the pods deployed in the namespace you already have permission to? Unless you need the IP/hostname of EVERY node, this should work.

[ferantivero]

@alexkursell all this make sense for me, but is there any reason I could be still seeing this error with quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1?

User "system:serviceaccount:<my-namespace>:nginx-ingress" cannot get resource "nodes" in API group ""

I’m deploying using the helm chart nginx-ingress-1.6.14

helm stable/nginx-ingress --set rbac.create=true,controller.scope.enabled=true --namespace=<my-namespace>

partial manifest below in case it helps:

containers:
        - name: nginx-ingress-controller
          image: "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1"
          imagePullPolicy: "IfNotPresent"
          args:
            - /nginx-ingress-controller
            - --default-backend-service=<my-namespace>/nginx-ingress-default-backend
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=backend-dev/nginx-ingress-controller
            - --watch-namespace=<my-namespace>

downgrading to helm chart nginx-ingress-1.6.8 this error disappear and the reason behind is that back then it was introduce a kind of workaround from the helm chart that created a 'scoped' ClusterRole, recently removed >=1.6.9 for security reasons specially when tiller lacks of permissions to create ClusterRoles https://github.com/helm/charts/pull/13460

Am I missing something?

[ferantivero]

/auto-cc

[ferantivero]

/cc @aledbf

[aledbf]

Am I missing something?

Yes, without access to the node API (that requires cluster level access) is not possible to update the status field in the Ingresses.

You can avoid this with:

• disabling the status of the ingress field adding the flag --update-status
• or removing the flag --publish-service and setting a value manually --publish-status-address

[streof]

Same problem with version 0.25.1 (--update-status trick did not work for me).

I managed to solve it by following this CloudBees post and the solution of this issue.