Charts: stable/nginx-ingress: In AKS, allow to assign static IP address from different resource group

[rnkhouse]

Is your feature request related to a problem? Please describe.
In Azure, every time when I create an AKS cluster I need to add the public IP address in the same resource group which is annoying having other operational issues. It would be nice if we can assign IP address from another resource group which is already created before.

Describe the solution you'd like
Azure supports to use pre-defined loadbalancer ip address from different resource group by giving right permissions to service principal as described in below link.

https://docs.microsoft.com/en-us/azure/aks/static-ip#use-a-static-ip-address-outside-of-the-node-resource-group

Describe alternatives you've considered
Right now, there is no alternative as we want to route all our requests from nginx-ingress controller.

This command is not working because LB_PUBLIC_IP is in a different resource group.

helm install stable/nginx-ingress --name nginx \
                        --set rbac.create=true \
                        --set controller.stats.enabled=true \
                        --set controller.metrics.enabled=true
                        --set controller.service.loadBalancerIP=${LB_PUBLIC_IP}

[diepes]

We just bumped into the same problem deploying kubernetes cluster on Azure, trying to use a existing public IP.
The deployment works without a error - even a bigger problem - and then at runtime the nginx-ingress-controller goes into a loop re-trying to bind to the ip.

$ kubectl describe service xxxxxx-nginx-ingress-controller
```

$ kubectl describe service xxxxxx-nginx-ingress-controller
Name: xxxxxx-nginx-ingress-controller
Namespace: kube-system
Labels: app=nginx-ingress
chart=nginx-ingress-1.4.0
component=controller
heritage=Tiller
release=xxxxxx
Annotations:
Selector: app=nginx-ingress,component=controller,release=xxxxxx
Type: LoadBalancer
IP: 10.0.35.124
IP: 52.xx.xx.xx
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31980/TCP
Endpoints: 10.244.0.16:80,10.244.0.17:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31826/TCP
Endpoints: 10.244.0.16:443,10.244.0.17:443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 2m3s (x11 over 27m) service-controller Ensuring load balancer
Warning CreatingLoadBalancerFailed 2m3s (x11 over 27m) service-controller Error creating load balancer (will retry): failed to ensure load balancer for service kube-system/xxxxxx-nginx-ingress-controller: user supplied IP Address 52.xx.xx.xx was not found in resource group MC_aks-xxxxxx

```

[matttabor]

Running into the same issues here where the public ip needs to be created in a different resource group. I eventually get a timeout error.

[matttabor]

@rnkhouse @diepes - were you able to find a work around to this issue?

[bkaid]

I figured this out from this issue. You need to set the controller service annotation service.beta.kubernetes.io/azure-load-balancer-resource-group to the public ip resource group name. I used the Terraform Helm provider and was able to use:

set {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-resource-group"
    value = "${azurerm_resource_group.resource_group.name}"
  }

which from command line is probably (give or take an escaping period slash):
--set controller.service.annotations.service\.beta\.kubernetes\.io/azure-load-balancer-resource-group=your-resource-group-name

[rnkhouse]

@bkaid I tried creating a new cluster with mentioned config but, it's not using pre-defined static IP from another resource group, instead, it creates a new IP address.

I used below command:

helm install stable/nginx-ingress --name nginx \
                    --set rbac.create=true \
                    --set controller.replicaCount=2 \
                    --set controller.stats.enabled=true \
                    --set controller.metrics.enabled=true \
                    --set controller.service.externalTrafficPolicy="Local" \
                    --set controller.service.loadBalancerIP=${LB_PUBLIC_IP} \
                    --set controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-resource-group="other-resource-group"

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[rnkhouse]

Please address this issue. It is not fixed yet.

[sahan-wimalasiri]

Please reopen this bug as this issue is currently affecting us too and hasn't been fixed. If it has could this ticket please be updated with instructions on how it is resolved? Thanks

[TimHoganPrinceton]

Also requesting that this be fixed as well. If the workaround explained by bkaid works, then we will use that but it is not a desirable approach.

[ankurkapoor]

I got it finally working by using following command:
helm install --name 'nginx-proxy' stable/nginx-ingress --namespace $namespaceName --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux
--set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux --set controller.service.loadBalancerIP="$staticIngressIP"
--set controller.service.annotations."service.beta.kubernetes.io/azure-load-balancer-resource-group"="$StaticIPRGname"

Hopefully this works for others as well.

[kwaazaar]

Specifying the annotation service.beta.kubernetes.io/azure-load-balancer-resource-group does work, but the service principal for the cluster must have Contributor-role in the resource group.

[afsilvasantos]

This did not work for me, when specifying the resource group where the Public IP is, it still says "Public IP not found in resource group X" despite specifiying in the command service.beta.kubernetes.io/azure-load-balancer-resource-group=resourcegroupY
Is this due what @kwaazaar reported?
In azure it is possible to assign any Public IP to the load balancer kubernetes deploys, so why is it not possible to choose one of the assigned ips to the load balancer, to the nginx service?

[snodnipper]

Hi folks, I too encountered challenges. Perhaps the following will be of help:

# Ensure that the Service Principal has the role "Owner" within the resource group
az role assignment create --assignee-object-id $SERVICE_PRINCIPAL_OBJECT_ID --role "Owner" --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP

# Create a static IP address for the cluster if it doesn't exist
if [ -z "$(az network public-ip list --query "[?name=='$STATIC_IP_NAME'].name | [0]" -o jsonc)" ]
then
echo "Static IP ($STATIC_IP_NAME) is undefined"
      echo "Creating static IP address"
      az network public-ip create --resource-group $RESOURCE_GROUP --name $STATIC_IP_NAME --sku Basic --allocation-method static --query publicIp.ipAddress -o tsv
else
      echo "Static IP ($STATIC_IP_NAME) is defined"
fi

# Print static IP address info
export STATIC_IP_ADDRESS=$(az network public-ip show --resource-group $RESOURCE_GROUP --name $STATIC_IP_NAME --query ipAddress --output tsv)
echo "The K8s static IP address for the LoadBalancer / Ingress is $STATIC_IP_ADDRESS"

# Ensure that K8s can use the static IP
# see: https://docs.microsoft.com/en-us/azure/aks/static-ip
# see also: https://pixelrobots.co.uk/2019/06/use-a-static-public-ip-address-outside-of-the-node-resource-group-with-the-azure-kubernetes-service-aks-load-balancer/
export CLIENT_ID=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query "servicePrincipalProfile.clientId" --output tsv)

if [ $CLIENT_ID != $SERVICE_PRINCIPAL ]; then
  echo "Client ID ($CLIENT_ID) should equal the service principal value ($SERVICE_PRINCIPAL).  Failure"
  exit 1
fi

az role assignment create\
    --assignee $CLIENT_ID \
    --role "Network Contributor" \
    --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP

# MS Documentation:
# https://docs.microsoft.com/en-us/azure/aks/ingress-tls
# https://docs.microsoft.com/en-us/azure/aks/ingress-static-ip
helm install nginx stable/nginx-ingress \
    --namespace default \
    --set controller.replicaCount=1 \
    --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set controller.service.loadBalancerIP="$STATIC_IP_ADDRESS" \
    --set controller.publishService.enabled=true \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-resource-group"="$RESOURCE_GROUP"

The above works for me.

In terms of debugging, the following commands may be a helpful starting point:

kubectl describe service nginx-nginx-ingress-controller

^ I had plenty of error feedback

kubectl --namespace default get services -o wide -w nginx-nginx-ingress-controller

^ note the external IP

[afsilvasantos]

I eventually made it work, i was pointing to the wrong resource group. the annotation gives a wrong understanding, its not the load balancer resource group you have to indicate, but the public ips resource group!
This worked with helm installation. But, i am now trying to deploy it with the Deployment YAML file, and i put the same annotation and it doesn't respect it. I set:
service.beta.kubernetes.io/azure-load-balancer-resource-group: "ResourceGroupX"
But it still says in the describe of the service that it cant find the IP in resource group Y

[mchudinov]

stable/nginx-ingress is deprecated!
I use ingress-nginx/ingress-nginx.
The question is the same: how to point to a custom resource group while installing ingress-nginx in AKS?
--set service.beta.kubernetes.io/azure-load-balancer-resource-group does NOT work!

helm install name ingress-nginx/ingress-nginx ??