Azure-docs: The application rule collection causes API Server calls to fail for ingress controllers and other internal services

[toutougabi]

The rules mentionned in the "Adding firewall rules" talk about adding an Application Rule Collection to whitelist the https traffic going to the API Server. This rules will not work for internal kubernetes components.

Within kubernetes, in the default namespace a service is created automatically by AKS at the creation of the cluster to enable pods to talk to the API server. This service uses the IP address of the API server. Using this service, internal pods (ex. the NGINX ingress controller pods) make https request using the IP without specifying a host header. When the request goes through the firewall, the firewall does not match the traffic to the previously created application rule collection as the rule collection inspects the HTTP traffic and matches using the host header (the request does not contain a host header). So the firewall will log the following : Deny. Reason: SNI TLS extension was missing.

Therefore, this solution is not appropriate. What is needed is for the Azure Firewall team to create a Service Tag instead of an FQDN tag so we can use the network rules to whitelist the traffic. network rules do not inspect the https header and will not cause this problem.

In the meantime, the only solution is either to whitelist https traffic to the AzureCloud Service Tag or to your API Server's IP (From my analysis, it's actually static anyways.

Also, the solution to use the DNS proxy is also not appropriate as you are not guaranteed to have enough traffic for the firewall to keep the FQDN-IP match in memory since all the calls coming the within the cluster have no DNS request (again they are sent directly to the IP of the cluster)

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: a38439d0-0125-3445-1c0a-a03b940e315d
• Version Independent ID: c6436a35-94b9-6396-3b13-9291f0a17b21
• Content: Restrict egress traffic in Azure Kubernetes Service (AKS) - Azure Kubernetes Service
• Content Source: articles/aks/limit-egress-traffic.md
• Service: container-service
• GitHub Login: @palma21
• Microsoft Alias: jpalma

[palma21]

Hi @toutougabi

That scenario is covered in the network rules overview:
https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#required-outbound-network-rules-and-fqdns-for-aks-clusters

If you have an app or solution that needs to talk to the API server, you must add an additional network rule to allow TCP communication to port 443 of your API server's IP.

Also please note in that same section:

AKS uses an admission controller to inject the FQDN as an environment variable to all deployments under kube-system and gatekeeper-system, that ensures all system communication between nodes and API server uses the API server FQDN and not the API server IP.

So everything on kube-system will use the FQDN and not the IP. You may apply a similar option for additional namespaces like the ingress controller namespace.

Service Tags are not available for AKS at the moment as the IPs are dedicated per cluster, which is currently a blocker to offer service tags. But is being worked through.

[zr-msft]

Thank you @toutougabi for the feedback and @palma21 for following up.

please-close