Ingress-nginx: Advice: multiple nginx-ingress setups in the same cluster

[Stono]

Hey,
I know this might seem strange... but we have a requirement for multiple nginx ingress controllers. We have some ingress traffic coming from the public internet and some coming form internal networks so need to bind to both a 10.x ip as well as a public ip address.

At the moment, we're using ingress-nginx for our public internet traffic which is great as we can do things like WAF, External Auth and so on, and we're using the standard glbc ingress for internal traffic.

I'd like to throw glbc away and use ingress-nginx for both because the ingress-nginx is significantly more feature rich.

Any idea's how we could achieve this? Effectively two ingress-nginx deployments on the same cluster where one of them has an ingress-nginx LoadBalancer, and the other has an ingress-nginx LoadBalancer with the cloud.google.com/load-balancer-type: Internal annotation. We would then need to selectively bind certain services to certain controllers which makes me thing we'd need to be able to control the kubernetes.io/ingress.class value for each, perhaps the ability to set them as nginx and nginx-internal?

Totally open to ideas!

[pieterlange]

Using class for this is perfectly fine, i do the exact same thing for the same usecase.

[Stono]

@pieterlange could you point me to some doco? I use class to support glbc, istio and nginx together, but not to deploy multiple ingress-nginx setups on the same cluster as they'd both fight for nginx?

It would probably would if class was configurable, so I could do two deployments, one using nginx and the other using nginx-internal

[pieterlange]

https://github.com/kubernetes/ingress-nginx#annotation-ingressclass
https://github.com/kubernetes/ingress-nginx#running-multiple-ingress-controllers
https://github.com/kubernetes/contrib/issues/1463

[Stono]

@pieterlange that doesn't address my problem, that allows me to run nginx-ingerss along side another controller class such as glbc or istio, but i specifically want to run two nginx controllers, dealing with different traffic.

[pieterlange]

Try reading the linked documentation.

[Stono]

I have.... I may be overly tired as I have had a very long day, but I do
not see how those links help?

On 24 Jan 2018 11:07 pm, "Pieter Lange" [e-mail] wrote:

Try reading the linked documentation.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/ingress-nginx/issues/1976#issuecomment-360304959,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABaviYKMrqSBQIDDknhgeOarTCTMbuVYks5tN7eqgaJpZM4RsEAp
.

[aledbf]

@Stono it's really simple, the combination of the annotation kubernetes.io/ingress.class: "EXTERNAL|INTERNAL" and the flag --ingress-class=EXTERNAL|INTERNAL allows you to filter which Ingress rules should be picked by the nginx ingress controller.

[Stono]

Oh shit I see it. I'm sorry. As I say, exhausting day.

Thanks for your help folks

On 24 Jan 2018 11:08 pm, "Karl Stoney" [e-mail] wrote:

I have.... I may be overly tired as I have had a very long day, but I do
not see how those links help?

On 24 Jan 2018 11:07 pm, "Pieter Lange" [e-mail] wrote:

Try reading the linked documentation.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/ingress-nginx/issues/1976#issuecomment-360304959,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABaviYKMrqSBQIDDknhgeOarTCTMbuVYks5tN7eqgaJpZM4RsEAp
.

[aledbf]

@Stono keep in mind the value of the annotations are just strings so you can even use your ingress rules annotated with the glbc annotation just modifying the nginx ingress controller flag.
This allows you to test this scenario without irrupting the traffic :)

[aledbf]

@Stono can we close this issue?

[Stono]

Done :)
Thanks again

[Stono]

Sorry to re-open, any idea why this might be happening:

E0125 08:54:08.798343       7 leaderelection.go:258] Failed to update lock: configmaps "ingress-controller-leader-nginx-external" is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot update configmaps in the namespace "ingress-nginx": Unknown user "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount"

As you can see, it does exist:

❯ kn get serviceaccounts
NAME                           SECRETS   AGE
default                        1         23h
nginx-ingress-serviceaccount   1         23h

[Stono]

Never mind, found the problem!

[amlozano]

@Stono Could you tell what the problem was? It could be useful for other people hitting this post :)

[zhoulouzi]

@amlozano if you have two more nginx-ingress
change -election-id https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/

[medvednic]

@Stono Could you tell what the problem was? It could be useful for other people hitting this post :)

I've encountered the same issue, the solution is here:
https://github.com/kubeapps/kubeapps/issues/120#issuecomment-369922824

Added the name of the ConfigMap (nginx-configuration) to resources under kind: Role
And added "ingress-controller-leader-YOURCLASSNAME" to resourceNames of kind: Role

It solved the issue.

btw the class name should be lowercase