Azure-docs: Confusing guidance on Network Address Ranges

[MarkTopping]

There are two bits of guidance in the Advanced Networking Deployment Parameters section which for me either contradict eachother, or require further elaboration to make more sense of the reason why:

Firstly, the documentation states: "Virtual network: The VNet into which you want to deploy the Kubernetes cluster."

Ok, that's straightforward enough, no confusion here.

But then:
"Kubernetes service address range: The Kubernetes service address range is the IP range from which addresses are assigned to Kubernetes services in your cluster"

"The Kubernetes service IP address range:

• Must not be within the VNet IP address range of your cluster"

Specifically this last point is what presents the confusion for me because it is telling me to use a range outside of my Vnet. Am I to understand that pods will be assigned an IP within range of the Vnet, but, Kubernetes Services must live outside of the VNET in some separate Address Range?

If so, please can the documentation elaborate as to why.
Or, is the documentation actually trying to say that the Kubernetes Service address range should not be within the address range of the Subnet (rather than Vnet) that the cluster is deployed into?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 22ad4873-c73d-4b90-2063-927a1060d2a2
• Version Independent ID: 8e6643ad-7333-a786-8712-3d17291e12a7
• Content: Network configuration in Azure Kubernetes Service (AKS)
• Content Source: articles/aks/networking-overview.md
• Service: container-service
• GitHub Login: @mmacy
• Microsoft Alias: marsma

[Karishma-Tiwari-MSFT]

Thanks for the feedback! We are currently investigating and will update you shortly.

[TravisCragg-MSFT]

@MarkTopping The Kubernetes Service Address Range is IP Addresses that Kubernetes exposes to use it's services. These are not the IP Addresses of the physical machines that Kubernetes creates.

the IP Address range should be within your VNET, but not within the ranges where your cluster can deploy. Using the description of the doc, you would need to deploy it in a different Subnet than where you deployed the AKS Cluster.

edit: Kubernetes Service Range cannot be in the same subnet as the AKS Cluster.

[MarkTopping]

Ok; how you've worded that makes more sense and clarifies the addressing requirements.

Hopefully someone could tweak the docs to be clearer as per your 'edit' line.

Thanks for responding promptly.

[mmacy]

@MarkTopping Travis is (EDIT: partially) correct: From a purely technical standpoint, you _could_ specify a K8s service range that's in a different subnet, but within same VNet as your cluster. However, it's then also possible to later create a second (or third or fourth or...) subnet in the VNet that stomps over the K8s service range, causing potential conflicts. This is because the Azure Networking side doesn't have visibility into the K8s cluster to know _not_ to allow that.

As such, that bullet's guidance is sound, and should be considered a best practice to avoid conflicts. Further, the K8s service range is not a routable range (within Azure, that is) and only K8s knows about it, routing packets from containers to a service within the cluster. To sum it up, this would be a valid config, and follows best practice guidance of using a service range outside the VNet:

VNet: 10.0.0.0/16
Subnet: 10.0.1.0/24
Kubernetes service range: 172.16.0.0/16

Cc: @aanandr

[MarkTopping]

Hi mmacy.

That also makes sense. I can work with that.

However, to say that Travis was right when he said:
"The IP Address range should be within your VNET, but not within the ranges where your cluster can deploy"

Well, you're guidance contrasts that by saying it is recommended for the IP Address Range to be outside of the VNET, not within.

And the documentation states it not as a recommendation, but uses the words 'must not'.

You're explanation here provides some useful and clear rational as to why its best to use a different address range - thank you.

Either way mind, if you use the Portal to provision a new cluster, the pre-create validation passes when you provision a cluster with the address range within or outside of the VNET; it might be better that validation did not pass when within. Anyhow, i'll proceed using your recommendation.

Thanks for your input.

[mmacy]

@MarkTopping Sorry for the confusion here. I've clarified my comment by adding the "partially correct" qualifier to Travis' statement.

I'll update the article with more clarification on this guidance.

in-progress

[MarkTopping]

Great. Thank you

[aanandr]

@mmacy - can we tweak the wordings a bit so that there is no confusion? For example we have to say that service IPs dont need to be created in a separate subnet. They are used only inside the cluster

@MarkTopping i didnt understand the last para of your response. Can you clarify

[MarkTopping]

@aanandr
Sure... assuming you mean this:
"Either way mind, if you use the Portal to provision a new cluster, the pre-create validation passes when you provision a cluster with the address range within or outside of the VNET; it might be better that validation did not pass when within. Anyhow, i'll proceed using your recommendation."

It has nothing to do with this documentation; I was just pointing out an extra observation...

While I use the ACS-Engine myself, this comment refers to the validation of the AKS configuration that automatically occurs when provision a new cluster through the Azure Portal. Basically if you work though the various tabs to create a new AKS instance, and then when you click on the Review + Create tab which is the final tab before you actually create the instance, then a validation step kicks off. That validation step will report everything is ok when you place your Kubernetes Service Range within the VNET which is contradictory to to the guidance being offered above and during this conversation.

That said, subsequent creation of the cluster still works when the Kubernetes Service Address Range is placed inside the VNET, and the resulting cluster is operational; although (probably unrelated) I have suffered CNI errors within that cluster which is why I'm re-reviewing the underlying network setup.

[aanandr]

Thanks Mark - when you specify a service IP range in the vnet space in the Networking tab you should have seen an error right there. I remember seeing it. Also, i dont think it lets you proceed further from that point. Let me know if this has been your observation.

Sent from Outlookhttp://aka.ms/weboutlook

---

From: Mark Topping [e-mail]
Sent: Monday, August 6, 2018 11:48 PM
To: MicrosoftDocs/azure-docs
Cc: Aanand Ramachandran; Mention
Subject: Re: [MicrosoftDocs/azure-docs] Confusing guidance on Network Address Ranges (#12929)

@aanandrhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faanandr&data=02%7C01%7Caanandr%40microsoft.com%7Cee1ec28efb3b446de8ed08d5fc31cbd3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636692213171849261&sdata=ushBOeLnms8zbdy9IDk5BKZFnnLGjEtdY3BYvrs5oUY%3D&reserved=0
Sure... assuming you mean this:
"Either way mind, if you use the Portal to provision a new cluster, the pre-create validation passes when you provision a cluster with the address range within or outside of the VNET; it might be better that validation did not pass when within. Anyhow, i'll proceed using your recommendation."

It has nothing to do with this documentation; I was just pointing out an extra observation...

While I use the ACS-Engine myself, this comment refers to the validation of the AKS configuration that automatically occurs when provision a new cluster through the Azure Portal. Basically if you work though the various tabs to create a new AKS instance, and then when you click on the Review + Create tab which is the final tab before you actually create the instance, then a validation step kicks off. That validation step will report everything is ok when you place your Kubernetes Service Range within the VNET which is contradictory to to the guidance being offered above and during this conversation.

That said, subsequent creation of the cluster still works when the Kubernetes Service Address Range is placed inside the VNET, and the resulting cluster is operational; although (probably unrelated) I have suffered CNI errors within that cluster which is why I'm re-reviewing the underlying network setup.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F12929%23issuecomment-410952064&data=02%7C01%7Caanandr%40microsoft.com%7Cee1ec28efb3b446de8ed08d5fc31cbd3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636692213171859271&sdata=Axw78IApFT55rG%2BZoPUS6nIIL83Rp%2Bny6X8vMwMqPpw%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAV3chYxNTnwo9O4S6amFMNXXyXb868beks5uOThBgaJpZM4VwU86&data=02%7C01%7Caanandr%40microsoft.com%7Cee1ec28efb3b446de8ed08d5fc31cbd3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636692213171869280&sdata=BhjwDTHf6jcxqrq7x%2BNJtWofSpppu0MYqKKRaBL4mps%3D&reserved=0.

[MarkTopping]

Hi AAnandr;
I'm afraid that's just not the case. Maybe when you try to specify a range in the Subnet space (which I've not tried), but you can definitely place it in the VNET Space.
i.e.
Vnet: 10.0.0.0/12
K8s Address Space: 10.0.0.0/16
Subnet: 10.1.0.0/16

Results in no warnings on neither the Networking tab or the Review + Create tab.

[mmacy]

@aanand I can confirm @MarkTopping's statement: no error is generated if you specify a subnet within the VNet for the K8s service range, as long as that subnet is _different_ from the subnet in which you place the cluster. If you attempt to place the K8s service range in the _same_ subnet as the cluster, then yes, an error is displayed.

[aanandr]

Ok got it. Thanks for the clarifications.
@Marsh [e-mail] - can we include a question in the faq that covers this? We can talk about what the service IP is and the impact of overlap. The content is what you already have.

Sent from Outlookhttp://aka.ms/weboutlook

---

From: Marsh Macy [e-mail]
Sent: Wednesday, August 8, 2018 7:57 AM
To: MicrosoftDocs/azure-docs
Cc: Aanand Ramachandran; Mention
Subject: Re: [MicrosoftDocs/azure-docs] Confusing guidance on Network Address Ranges (#12929)

@aanandhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faanand&data=02%7C01%7Caanandr%40microsoft.com%7C9f284d1118cd4c2d843908d5fd3f32f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636693370245612536&sdata=hf2ntwxywOMPBLeNqn6ZgVMqx5SiKVpWYwOev5v%2F3ik%3D&reserved=0 I can confirm @MarkToppinghttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMarkTopping&data=02%7C01%7Caanandr%40microsoft.com%7C9f284d1118cd4c2d843908d5fd3f32f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636693370245612536&sdata=mWqCyJqu5HBnadLYAm5mjwi1btte4PwXvOq9y0ZKf7M%3D&reserved=0's statement: no error is generated if you specify a subnet within the VNet for the K8s service range, as long as that subnet is different from the subnet in which you place the cluster. If you attempt to place the K8s service range in the same subnet as the cluster, then yes, an error is displayed.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F12929%23issuecomment-411436023&data=02%7C01%7Caanandr%40microsoft.com%7C9f284d1118cd4c2d843908d5fd3f32f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636693370245612536&sdata=kijkzZ%2F%2FfxVU%2FLEFsVMcZvD1K%2F4S5HWNQu0hUjSKvfE%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAV3chbhKWxeVa3Y-oOwvD0r_FgfMBw7kks5uOvw9gaJpZM4VwU86&data=02%7C01%7Caanandr%40microsoft.com%7C9f284d1118cd4c2d843908d5fd3f32f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636693370245612536&sdata=gN%2BiLz3PpfyVj%2FP20qVMxCVHOAoTIWVe2bKIjpmfTgc%3D&reserved=0.

[mmacy]

@aanandr Yup, already completed. See private repo PR 48525. The updated doc should go live at the 3PM publish. Once published live, this issue will be closed.

[jontreynes]

Somewhat off topic, but it would be helpful if the doc explained how to measure how much space you should need in the "Kubernetes service address range". Similar to how we calculate the space needed for the cluster subnet @mmacy

[mmacy]

Adding @mlearned who's the current article owner.

[jontreynes]

Also, it would help if the doc could explain how cbr0 and veth0 plays a part in AKS. A diagram of the network flow with these components would clear my confusion.