Application-gateway-kubernetes-ingress: Cannot use private IP address as it is either allocated to a resource or being cleaned up.

[sbkg0002]

Describe the bug
Whenever we define a second ingress, or remove all of the ingresses and try to create a new ingress, we get the following error:

Log:

application-gateway-ingress-controller-ingress-azure-9bf4b95b8r ingress-azure E0430 17:37:43.416766 1 controller.go:132] Error mutating App Gateway config from k8s event. network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayFrontendIPStaticPrivateIPAddressNotAvailable" Message="FrontendIpConfiguration /subscriptions/SUBSCRIPTION_ID/resourceGroups/k8s/providers/Microsoft.Network/applicationGateways/k8s-appgateway/frontendIPConfigurations/AzureVnet-feip2 cannot use private IP address 10.161.134.5 as it is either allocated to a resource or being cleaned up." Details=[]

To Reproduce
Steps to reproduce the behavior:

1. Install controller
2. Create a AGW
3. Create the following ingress (this will result in a nice ingress):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/appgw-ssl-certificate: "mycert"
    appgw.ingress.kubernetes.io/use-private-ip: "true"
spec:
  rules:
  - host: guestbook.test-k8s.mydomain.com
    http:
      paths:
      - backend:
          serviceName: frontend
          servicePort: 80

1. Repeat step #3. The ingress will look normal, but the error (ApplicationGatewayFrontendIPStaticPrivateIPAddressNotAvailable) will appear.

Ingress Controller details
Version: 1.2.0-rc1
pod describe:

kdp application-gateway-ingress-controller-ingress-azure-9bf4b95b8r
Name:           application-gateway-ingress-controller-ingress-azure-9bf4b95b8r
Namespace:      cpt
Priority:       0
Node:           aks-cpt-34239724-vmss000000/10.161.132.4
Start Time:     Thu, 30 Apr 2020 17:17:01 +0200
Labels:         aadpodidbinding=application-gateway-ingress-controller-ingress-azure
                app=ingress-azure
                pod-template-hash=9bf4b78c7
                release=application-gateway-ingress-controller
Annotations:    prometheus.io/port: 8123
                prometheus.io/scrape: true
Status:         Running
IP:             10.161.132.7
IPs:            <none>
Controlled By:  ReplicaSet/application-gateway-ingress-controller-ingress-azure-9bf4b78c7
Containers:
  ingress-azure:
    Container ID:   docker://a6697e2d36a2a92f2c26a8c29444606fa20e69c77d49f1540910854ef89a39b6
    Image:          azure-application-gateway/kubernetes-ingress:1.2.0-rc1
    Image ID:       docker-pullable://azure-application-gateway/kubernetes-ingress@sha256:dd95b2feaf24e7ba6773452fb842d0eba5a6ea8a5d19bf22035fdcad78b18941
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 30 Apr 2020 17:17:03 +0200
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      application-gateway-ingress-controller-cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CONTEXT_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:           application-gateway-ingress-controller-ingress-azure-9bf4b95b8r (v1:metadata.name)
      AGIC_POD_NAMESPACE:      cpt (v1:metadata.namespace)
    Mounts:
      /etc/appgw/azure.json from azure (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from application-gateway-ingress-controller-sa-ingress-azure-tobqzbq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  application-gateway-ingress-controller-sa-ingress-azure-tobqzbq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  application-gateway-ingress-controller-sa-ingress-azure-tobqzbq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

[flmarins]

Hello,
I'm also facing the exact same error.

[sbkg0002]

What is your AGW config @flmarins ?

[flmarins]

What is your AGW config @flmarins ?
This is the template deployed for the AGW, WAF_V2

{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"variables": {
"appGwID": "[resourceId('Microsoft.Network/applicationGateways', 'egp-u-aks-agw')]"
},
"resources": [
{
"type": "Microsoft.Network/applicationGateways",
"apiVersion": "2019-04-01",
"name": "egp-u-aks-agw",
"location": "westeurope",
"dependsOn": [],
"tags": {},
"properties": {
"backendAddressPools": [
{
"name": "appGatewayBackendPool"
}
],
"backendHttpSettingsCollection": [
{
"name": "appGatewayBackendHttpSettings",
"properties": {
"Port": 80,
"Protocol": "Http",
"CookieBasedAffinity": "disabled",
"connectionDraining": {
"enabled": false,
"drainTimeoutInSec": 1
}
}
}
],
"frontendIPConfigurations": [
{
"name": "appGatewayFrontendIP",
"properties": {
"publicIPAddress": {
"id": "/subscriptions/SubId/resourceGroups/RgAks/providers/Microsoft.Network/publicIPAddresses/egp-u-aks-agw-pip"
}
}
}
],
"frontendPorts": [
{
"name": "appGatewayFrontendPort",
"properties": {
"Port": 80
}
}
],
"gatewayIPConfigurations": [
{
"name": "appGatewayFrontendIP",
"properties": {
"subnet": {
"id": "/subscriptions/SubId/resourceGroups/RgVnet/providers/Microsoft.Network/virtualNetworks/External_Network/subnets/External_Network_AKS_AGW"
}
}
}
],
"httpListeners": [
{
"name": "appGatewayHttpListener",
"properties": {
"FrontendIpConfiguration": {
"Id": "[concat(variables('appGwID'), '/frontendIPConfigurations/appGatewayFrontendIP')]"
},
"FrontendPort": {
"Id": "[concat(variables('appGwID'), '/frontendPorts/appGatewayFrontendPort')]"
},
"Protocol": "http",
"SslCertificate": null
}
}
],
"sku": {
"name": "WAF_v2",
"tier": "WAF_v2",
"capacity": 2
},
"requestRoutingRules": [
{
"Name": "rule1",
"properties": {
"RuleType": "Basic",
"httpListener": {
"id": "[concat(variables('appGwID'), '/httpListeners/appGatewayHttpListener')]"
},
"backendAddressPool": {
"id": "[concat(variables('appGwID'), '/backendAddressPools/appGatewayBackendPool')]"
},
"backendHttpSettings": {
"id": "[concat(variables('appGwID'), '/backendHttpSettingsCollection/appGatewayBackendHttpSettings')]"
}
}
}
]
}
}
],
"outputs": {
"applicationGateway": {
"type": "Object",
"value": "[reference('egp-u-aks-agw')]"
}
}
}

[amimof]

Having the same problem. I noticed today that I get this error when creating a private frontend ip using the Azure Portal. So might not be a AGIC specific issue.

[tiwood]

I'm also having this issue with a newly deployed AppGW, AKS Cluster & AGIC

error: network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayFrontendIPStaticPrivateIPAddressNotAvailable" Message="FrontendIpConfiguration /subscriptions/604665e1-18a1-4c68-b160-f1a48be5c014/resourceGroups/rg-contoso-dev-networking-appgateways/providers/Microsoft.Network/applicationGateways/appgw-7490e25be7cbd2a7/frontendIPConfigurations/appgw-7490e25be7cbd2a7-private-ipconf cannot use private IP address 172.28.1.229 as it is either allocated to a resource or being cleaned up.

Frontend config:

[
   {
      "PrivateIPAllocationMethod":"Dynamic",
      "PublicIPAddress":{
         "Id":"/subscriptions/604665e1-18a1-4c68-b160-f1a48be5c014/resourceGroups/rg-contoso-dev-networking-appgateways/providers/Microsoft.Network/publicIPAddresses/appgw-df0ddae6e805dbcc-pip"
      },
      "ProvisioningState":"Succeeded",
      "Type":"Microsoft.Network/applicationGateways/frontendIPConfigurations",
      "Name":"appgw-df0ddae6e805dbcc-public-ipconf",
      "Etag":"W/\"fb1ccc50-d7d5-41bd-90e4-e491222a9aa7\"",
      "Id":"/subscriptions/604665e1-18a1-4c68-b160-f1a48be5c014/resourceGroups/rg-contoso-dev-networking-appgateways/providers/Microsoft.Network/applicationGateways/appgw-df0ddae6e805dbcc/frontendIPConfigurations/appgw-df0ddae6e805dbcc-public-ipconf"
   },
   {
      "PrivateIPAddress":"172.28.1.230",
      "PrivateIPAllocationMethod":"Static",
      "Subnet":{
         "Id":"/subscriptions/604665e1-18a1-4c68-b160-f1a48be5c014/resourceGroups/rg-contoso-dev-networking/providers/Microsoft.Network/virtualNetworks/vnet-contoso-dev-01/subnets/AppGatewaySubnet"
      },
      "ProvisioningState":"Succeeded",
      "Type":"Microsoft.Network/applicationGateways/frontendIPConfigurations",
      "Name":"appgw-df0ddae6e805dbcc-private-ipconf",
      "Etag":"W/\"fb1ccc50-d7d5-41bd-90e4-e491222a9aa7\"",
      "Id":"/subscriptions/604665e1-18a1-4c68-b160-f1a48be5c014/resourceGroups/rg-contoso-dev-networking-appgateways/providers/Microsoft.Network/applicationGateways/appgw-df0ddae6e805dbcc/frontendIPConfigurations/appgw-df0ddae6e805dbcc-private-ipconf"
   }
]

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: guestbook
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/use-private-ip: "true"
    appgw.ingress.kubernetes.io/backend-protocol: http
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: frontend
          servicePort: 80

[mscatyao]

Hey all - this is a fix outside of AGIC that we've completed and are testing right now. We expect to roll out the fix within the next couple weeks, please check back on this thread for more updates. I'll close this thread when the fix is fully rolled out.

[anhlqn]

Is there any workaround at this time or do we have to use the public IP for any new sites published through the AppGW?

[flmarins]

Is there any workaround at this time or do we have to use the public IP for any new sites published through the AppGW?

I’m able to create multiple entries for the same private ip using different hostname, however if you delete all configurations using the private ip you will need to provision a new private ip in the AppGw to use again

[anhlqn]

Thanks. That is better than recreating the AppGW.

[mscatyao]

Wanted to update this thread to let everyone know that the fix for this has been rolled out; all new AppGWs should not encounter this issue and only a subset of existing AppGWs which were in a specific state previously would run into this issue. Please test your private IP scenarios; if you're still experiencing issues with using the private IP, try changing to a different private IP address. If that's not possible or you still run into issues, please create a support ticket through Azure Portal and we'll be able to fix things accordingly.

[sbkg0002]

I still run in this error, even in new VNET & AGW setups.

[mscatyao]

@sbkg0002 - please create a support ticket through Portal and a support engineer will be able to help you there.

[sbkg0002]

I did, it appears that you should use the first usable IP of the subnet as a static AGW IP.

[L480]

After changing the IP address of the App GW it was working for a few days, but now the issue appears again. I just changed the IP address again and afterwards it was back working. So it seems the issue is not fully fixed yet. Besides that, I cannot use several IP addresses from the VNet as App GW IP, since they are all allegedly "either allocated or being cleaned up", even though they are not in use at all.

[PPreamble]

This is still a problem, I've been encountering it for moths while developing Azure CLI to deploy environments that include an AGW. While creating script its often very typical to deploy an asset (such as AGW), delete the asset (or Resource Group), and then redeploy to test modifications.

The offending command is: az network application-gateway frontend-ip create

[bruun365]

Experiencing the same problems, seems that the issue hasn't been fixed yet. I've deployed the Application Gateway a few days ago.

If i remove the private IP adresses from the Application Gateway and try to re-add using the same IP address, this error message appears: _Private cannot use private IP address 10.248.63.5 as it is either allocated to a resource or being cleaned up._

Adding a different private IP address, for example 10.248.63.6, works

[Phrow]

Hi, I'm also having the same issue. I tried switching to another IP and that worked, however if I see that other people tried that before and encountered the same problem again after a while with the new IP, I'm afraid I cannot consider that a "proper solution".
And if otherwise the only resolution mentioned here is "ask a service engineer", I'm wondering what they can do?

[Gamecock]

@mscatyao , I also experienced. Resolved by changing front end address, but based on comments I expect to have to do this again. What does it take to get this ticket reopened (or a new issue) and a proper fix?