Azure-quickstart-templates: Can't redeploy Network Security Group for Databricks VNet injection

[edfreeman]

All-in-one Template for Databricks VNet Injected Workspaces

Issue Details

Hi there,

I'm using this "all-in-one" Datbricks/VNet/NSG template found here: https://github.com/Azure/azure-quickstart-templates/tree/master/101-databricks-all-in-one-template-for-vnet-injection/. It deploys fine the very first time, but subsequent deployments fail with a "Network Intent Policy" conflict.

Here's the full error:

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.","details":[{"code":"BadRequest","message":"{\r\n \"error\": {\r\n \"code\": \"ConflictWithNetworkIntentPolicy\",\r\n \"message\": \"Found conflicts with NetworkIntentPolicy. Details: Network Security Group cannot have resources which conflict with its subnets' network intent policies.\\r\\nNetwork Security Group: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkSecurityGroups/databricks-nsg conflicts with Network Intent Policy: adb-uksouth-<id>\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-worker-to-webapp, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-worker-to-webapp, Access: Allow, Direction: Outbound, Protocol: tcp, SourceAddressPrefix: VirtualNetwork, SourcePortRange: *, DestinationAddressPrefix: 51.140.204.4/32, DestinationPortRange: 443\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-worker-to-sql, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-worker-to-sql, Access: Allow, Direction: Outbound, Protocol: tcp, SourceAddressPrefix: VirtualNetwork, SourcePortRange: *, DestinationAddressPrefix: Sql, DestinationPortRange: 3306\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-worker-to-storage, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-worker-to-storage, Access: Allow, Direction: Outbound, Protocol: tcp, SourceAddressPrefix: VirtualNetwork, SourcePortRange: *, DestinationAddressPrefix: Storage, DestinationPortRange: 443\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-worker-to-eventhub, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-worker-to-eventhub, Access: Allow, Direction: Outbound, Protocol: tcp, SourceAddressPrefix: VirtualNetwork, SourcePortRange: *, DestinationAddressPrefix: EventHub, DestinationPortRange: 9093\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-control-plane-ssh, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-control-plane-ssh, Access: Allow, Direction: Inbound, Protocol: tcp, SourceAddressPrefix: 51.140.203.27/32, SourcePortRange: *, DestinationAddressPrefix: VirtualNetwork, DestinationPortRange: 22\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-control-plane-worker-proxy, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-control-plane-worker-proxy, Access: Allow, Direction: Inbound, Protocol: tcp, SourceAddressPrefix: 51.140.203.27/32, SourcePortRange: *, DestinationAddressPrefix: VirtualNetwork, DestinationPortRange: 5557\\r\\n ----\\r\\n---- ----\\r\\nNetwork Security Group: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkSecurityGroups/databricks-nsg conflicts with Network Intent Policy: adb-uksouth-<id>\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-worker-to-webapp, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-worker-to-webapp, Access: Allow, Direction: Outbound, Protocol: tcp, SourceAddressPrefix: VirtualNetwork, SourcePortRange: *, DestinationAddressPrefix: 51.140.204.4/32, DestinationPortRange: 443\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-worker-to-sql, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-worker-to-sql, Access: Allow, Direction: Outbound, Protocol: tcp, SourceAddressPrefix: VirtualNetwork, SourcePortRange: *, DestinationAddressPrefix: Sql, DestinationPortRange: 3306\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-worker-to-storage, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-worker-to-storage, Access: Allow, Direction: Outbound, Protocol: tcp, SourceAddressPrefix: VirtualNetwork, SourcePortRange: *, DestinationAddressPrefix: Storage, DestinationPortRange: 443\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-worker-to-eventhub, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-worker-to-eventhub, Access: Allow, Direction: Outbound, Protocol: tcp, SourceAddressPrefix: VirtualNetwork, SourcePortRange: *, DestinationAddressPrefix: EventHub, DestinationPortRange: 9093\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-control-plane-ssh, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-control-plane-ssh, Access: Allow, Direction: Inbound, Protocol: tcp, SourceAddressPrefix: 51.140.203.27/32, SourcePortRange: *, DestinationAddressPrefix: VirtualNetwork, DestinationPortRange: 22\\r\\n ----\\r\\n Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: databricks-control-plane-worker-proxy, Id: /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/networkIntentPolicies/adb-uksouth-<id>/securityRules/databricks-control-plane-worker-proxy, Access: Allow, Direction: Inbound, Protocol: tcp, SourceAddressPrefix: 51.140.203.27/32, SourcePortRange: *, DestinationAddressPrefix: VirtualNetwork, DestinationPortRange: 5557\\r\\n ----\\r\\n---- ----\",\r\n \"details\": []\r\n }\r\n}"}]}

Databricks VNet injection recently became GA, and we had to delegate our subnets to Microsoft.Databricks/workspaces on existing environments. I don't know whether this error is related teething problems, but this deployment used the updated ARM template above. Those security rules mentioned in the error seem to be the "old" security rules one had to define on the Databricks NSG, whilst the VNet injection feature was in preview.

Can anyone help?

Thanks,

Ed

Repro steps

1. Deploy the ARM template
2. Try to deploy it again, pointing towards the same resources

[azMantas]

It works fine as long as I deploy ARM template for the first time. Doesn't matter if I do make some changes in arm template or just try to redeploy template - get the same issue. Found conflicts with NetworkIntentPolicy ....

[MertSenel]

I'm having the same issue, I think it's a bug in the ARM deployment API rather than the template itself as there is a managed resource group component to it.

[edfreeman]

I think you're probably right.

In the absence of a permanent fix/solution, I've temporarily removed the NSG deployment from the ARM template and am handling it separately within my deployment process. The other resources seem to play ball.

[JamesQMurphy]

Same issue here... and same workaround as @edfreeman .

[marcin-vt]

The problem with the workaround is that it will not support complete deployment mode...

[edfreeman]

@marcin-vt True. I suppose you could create the NSG in a separate resource group, but this obviously isn't ideal. [Note: I haven't tested this, but I can't see a reason why it wouldn't work]

[marcin-vt]

Seems like another solution would be to create NSG already with all rules created by databricks (see below). However it sill won't work with complete deployment mode, because it will try to delete Network Intent Policy created by databricks and used by subnets...

{
"type": "Microsoft.Network/networkSecurityGroups",
"apiVersion": "2019-09-01",
"name": "[variables('networkSecurityGroup')]",
"location": "westeurope",
"properties": {
"securityRules": [
{
"name": "Microsoft.Databricks-workspaces_UseOnly_databricks-worker-to-worker-inbound",
"properties": {
"description": "Required for worker nodes communication within a cluster.",
"protocol": "",
"sourcePortRange": "",
"destinationPortRange": "",
"sourceAddressPrefix": "VirtualNetwork",
"destinationAddressPrefix": "VirtualNetwork",
"access": "Allow",
"priority": 100,
"direction": "Inbound",
"sourcePortRanges": [],
"destinationPortRanges": [],
"sourceAddressPrefixes": [],
"destinationAddressPrefixes": []
}
},
{
"name": "Microsoft.Databricks-workspaces_UseOnly_databricks-control-plane-ssh",
"properties": {
"description": "Required for Databricks control plane management of worker nodes.",
"protocol": "tcp",
"sourcePortRange": "",
"destinationPortRange": "22",
"sourceAddressPrefix": "23.100.0.135/32",
"destinationAddressPrefix": "VirtualNetwork",
"access": "Allow",
"priority": 101,
"direction": "Inbound",
"sourcePortRanges": [],
"destinationPortRanges": [],
"sourceAddressPrefixes": [],
"destinationAddressPrefixes": []
}
},
{
"name": "Microsoft.Databricks-workspaces_UseOnly_databricks-control-plane-worker-proxy",
"properties": {
"description": "Required for Databricks control plane communication with worker nodes.",
"protocol": "tcp",
"sourcePortRange": "",
"destinationPortRange": "5557",
"sourceAddressPrefix": "23.100.0.135/32",
"destinationAddressPrefix": "VirtualNetwork",
"access": "Allow",
"priority": 102,
"direction": "Inbound",
"sourcePortRanges": [],
"destinationPortRanges": [],
"sourceAddressPrefixes": [],
"destinationAddressPrefixes": []
}
},
{
"name": "Microsoft.Databricks-workspaces_UseOnly_databricks-worker-to-webapp",
"properties": {
"description": "Required for workers communication with Databricks Webapp.",
"protocol": "tcp",
"sourcePortRange": "",
"destinationPortRange": "443",
"sourceAddressPrefix": "VirtualNetwork",
"destinationAddressPrefix": "52.232.19.246/32",
"access": "Allow",
"priority": 100,
"direction": "Outbound",
"sourcePortRanges": [],
"destinationPortRanges": [],
"sourceAddressPrefixes": [],
"destinationAddressPrefixes": []
}
},
{
"name": "Microsoft.Databricks-workspaces_UseOnly_databricks-worker-to-sql",
"properties": {
"description": "Required for workers communication with Azure SQL services.",
"protocol": "tcp",
"sourcePortRange": "",
"destinationPortRange": "3306",
"sourceAddressPrefix": "VirtualNetwork",
"destinationAddressPrefix": "Sql",
"access": "Allow",
"priority": 101,
"direction": "Outbound",
"sourcePortRanges": [],
"destinationPortRanges": [],
"sourceAddressPrefixes": [],
"destinationAddressPrefixes": []
}
},
{
"name": "Microsoft.Databricks-workspaces_UseOnly_databricks-worker-to-storage",
"properties": {
"description": "Required for workers communication with Azure Storage services.",
"protocol": "tcp",
"sourcePortRange": "",
"destinationPortRange": "443",
"sourceAddressPrefix": "VirtualNetwork",
"destinationAddressPrefix": "Storage",
"access": "Allow",
"priority": 102,
"direction": "Outbound",
"sourcePortRanges": [],
"destinationPortRanges": [],
"sourceAddressPrefixes": [],
"destinationAddressPrefixes": []
}
},
{
"name": "Microsoft.Databricks-workspaces_UseOnly_databricks-worker-to-worker-outbound",
"properties": {
"description": "Required for worker nodes communication within a cluster.",
"protocol": "",
"sourcePortRange": "",
"destinationPortRange": "",
"sourceAddressPrefix": "VirtualNetwork",
"destinationAddressPrefix": "VirtualNetwork",
"access": "Allow",
"priority": 103,
"direction": "Outbound",
"sourcePortRanges": [],
"destinationPortRanges": [],
"sourceAddressPrefixes": [],
"destinationAddressPrefixes": []
}
},
{
"name": "Microsoft.Databricks-workspaces_UseOnly_databricks-worker-to-eventhub",
"properties": {
"description": "Required for worker communication with Azure Eventhub services.",
"protocol": "tcp",
"sourcePortRange": "",
"destinationPortRange": "9093",
"sourceAddressPrefix": "VirtualNetwork",
"destinationAddressPrefix": "EventHub",
"access": "Allow",
"priority": 104,
"direction": "Outbound",
"sourcePortRanges": [],
"destinationPortRanges": [],
"sourceAddressPrefixes": [],
"destinationAddressPrefixes": []
}
}
]
}
}

[Mlysiu]

This issue is really annoying. Any update on fixing it?

[Big0Bertha]

Same issue here. Any status updates?

[VOVELEE]

Actually you can redeploy it if you extend the ARM template with the rules required by networkIntentPolicies. See @marcin-vt suggestion

[MateusAttie]

Same issue here, any updates guys ?

[robfrut135]

Same issue here, any updates?

[robfrut135]

1 year open

[osapeliuk]

Same issue, are there any updates?

[rbnmk]

We really need a proper solution for this. Any update?

[PeterDowdy]

I'm having trouble with this, too. I can just pull the databricks deploy out of my combined pipeline, but that's not ideal.

[michalzima44]

What worked for us was to combine updating our templates based on https://github.com/Azure/azure-quickstart-templates/blob/master/101-databricks-all-in-one-template-for-vnet-injection/azuredeploy.json and using the solution by @marcin-vt to change the nsg part in this thread.

[rikedje]

Thank you @marcin-vt and @michalzima44 - works