External-dns: Azure Private DNS will replace Azure DNS Private Zones

[t0mmyt]

The current method of creating a private zone in Azure (a regular DNS zone set to type private) is going to be deprecated in coming months and the new method will be out of private preview in the next few weeks I am told.

I've had a quick look at what's involved and it will require a much newer version of the Azure SDK which in turn requires a new Azure rest client which in turn requires updating client-go.

I am happy to take on the Azure specific changes (updating the existing provider and adding a new one for private DNS) but updating to client-go v11.0.0 will require significantly more work than I have time to take on.

Is anyone able to get the client-go and k8s machinary up to date?

[bhicks329]

They’re going to be changing the top-level resource provider to separate it from the current DNS zones. At the moment they’re both covered under the same one.

https://azure.microsoft.com/en-gb/updates/announcing-preview-refresh-for-azure-dns-private-zones-2/

In a month or so, this functionality with Private DNS zones will break.

[lukasmrtvy]

@t0mmyt
I guess that --publish-internal-services will handle private zone, right ? ( I did not find any info, how to use external-dns with private zone ) ... seems that now its broken, because even with --publish-internal-services external-dns is publishing records to public zone.

[timja]

There needs to be code changes done to support it, private zone is a different top level resource and uses a different API and needs a new SDK version...

[szuecs]

Anyone that would like to work on this issue?
We have no Azure account and not the capacity to work on this.

[timja]

Happy to make the change, but ideally upgrading client-go and the azure-sdk too first would make this easier

I already tried to update it awhile ago, but I had trouble updating the dependencies

[lukasmrtvy]

@timja
What was wrong?

I used:

diff --git a/go.mod b/go.mod
index db42b9b9..c6e98b33 100644
--- a/go.mod
+++ b/go.mod
@@ -4,8 +4,8 @@ go 1.12

 require (
    cloud.google.com/go v0.37.4
-   github.com/Azure/azure-sdk-for-go v10.0.4-beta+incompatible
-   github.com/Azure/go-autorest v10.9.0+incompatible
+   github.com/Azure/azure-sdk-for-go v33.2.0+incompatible
+   github.com/Azure/go-autorest v13.0.1+incompatible
    github.com/alecthomas/assert v0.0.0-20170929043011-405dbfeb8e38 // indirect
    github.com/alecthomas/colour v0.0.0-20160524082231-60882d9e2721 // indirect
    github.com/alecthomas/kingpin v2.2.5+incompatible
diff --git a/provider/azure.go b/provider/azure.go
index 3f887e55..2fb19c2a 100644
--- a/provider/azure.go
+++ b/provider/azure.go
@@ -26,7 +26,7 @@ import (

    yaml "gopkg.in/yaml.v2"

-   "github.com/Azure/azure-sdk-for-go/arm/dns"
+        "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns"
    "github.com/Azure/go-autorest/autorest"
    "github.com/Azure/go-autorest/autorest/adal"
    "github.com/Azure/go-autorest/autorest/azure"
diff --git a/provider/azure_test.go b/provider/azure_test.go
index 36d69766..b1124e40 100644
--- a/provider/azure_test.go
+++ b/provider/azure_test.go
@@ -20,7 +20,7 @@ import (
    "context"
    "testing"

-   "github.com/Azure/azure-sdk-for-go/arm/dns"
+        "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns"
    "github.com/Azure/go-autorest/autorest"
    "github.com/Azure/go-autorest/autorest/azure"
    "github.com/Azure/go-autorest/autorest/to"

[timja]

It was awhile ago I'll try to have another look soon, could you create a PR with your change?

[timja]

Compilation fails on this with those changes:

provider/azure.go:122:3: cannot use zonesClient (type "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".ZonesClient) as type ZonesClient in field value:
    "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".ZonesClient does not implement ZonesClient (wrong type for ListByResourceGroup method)
        have ListByResourceGroup("context".Context, string, *int32) ("github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".ZoneListResultPage, error)
        want ListByResourceGroup(string, *int32) ("github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".ZoneListResult, error)
provider/azure.go:123:3: cannot use recordsClient (type "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".RecordSetsClient) as type RecordsClient in field value:
    "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".RecordSetsClient does not implement RecordsClient (wrong type for CreateOrUpdate method)
        have CreateOrUpdate("context".Context, string, string, string, "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".RecordType, "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".RecordSet, string, string) ("github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".RecordSet, error)
        want CreateOrUpdate(string, string, string, "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".RecordType, "github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".RecordSet, string, string) ("github.com/Azure/azure-sdk-for-go/services/dns/mgmt/2018-05-01/dns".RecordSet, error)
make: *** [build/external-dns] Error 2

Think I just went down a rabbit hole trying to update the tests last time, will try have another look

[timja]

Started on it in: https://github.com/kubernetes-incubator/external-dns/pull/1195
some client-go issues to sort out, updating the azure_test and fixing the pagination of dns zones + records sets

[saidst]

Great you started to work on it.

I see you are quite far. Can I help to complete it?

[timja]

See if you can figure out why it’s not compiling right now that would be much appreciated, I’ve got compatible Sdks working but the mock interface isn’t quite working, some go ism that I don’t understand...

[saidst]

FYI: I work on it. I hope I am able to push some first idea today. Compilation works.

I upgraded the azure dependencies once again.
Also, I figured, that authentication with Azure is not as MS wants it (https://github.com/Azure/azure-sdk-for-go#authentication). Thus, I was required to change that in order to keep code related to the _Authorizer_ simple.

Maybee (or very probably), we should about splitting those changes in various PRs.

[timja]

I wouldn't change the authentication in the PR unless it's required to make it work, can re-work it separately

[saidst]

Agreed. I will figure that out. But I am afraid, that code around ADAL/Manual token creation is at least deprecated if not more.

[timja]

@stsaid how're you getting on?

[saidst]

I didn't work on it the last two days. Maybee you can have a look on my changes? See the PR on the branch in your forked repo.
If this looks OK, I would continue. There is some stuff to do in the tests (in the Azure-Provider but also in others.).

[timja]

I took a look and managed to fix the tests, they now pass but need a CLA signature from you @stsaid as I merged your commit in.

[saidst]

I care for that today. Thanks for continuing work on the PR.

[saidst]

FYI: I've signed up at the Linux Foundation. So far, I haven't received the mail with the CLA Confirmation. Have filed a ticket requesting for support.

[saidst]

@timja, already thought how to handle the different types that come with Azure Private DNS?

So far, tests, services methods etc. are all bound to the specific types of Azure DNS.
An idea for that could be to create custom types for both private & public DNS which embed the Microsoft-ones, like e.g. a RecordSet. Additionally, those custom types implement "shared" interfaces which are independent from the semantics of public & private DNS.
Finally, adapt service methods etc. to use these interfaces and no longer bind to particular types of Azure DNS.

Alternatively, I tried to come up with an interlayer which does not embed Azure Types but exists separately. Example: Instead of Records()-function going straight to service methods of the Azure Clients, it invokes a "generic" function which again invokes both Azure DNS Client and Azure Private DNS client and returns "generic" types.
See: https://github.com/stsaid/external-dns/commit/4130ea194698f475b0705d39b6a2cfc3bbc4ea30

Lastly, one could even think of two distinct providers.

Opinions?

[timja]

From what I can remember from implementing support for it in terraform all the models and api calls are different and the api is different under the hood (less consistent at least on the zone side)

I would lean on the side of a separate provider side

[saidst]

I think so too....
The "interlayer"-approach makes the code a bit complicated.
Also, for AWS-there are already two different providers.

Regarding provider-names we could go like...
--provider azure-dns|azure --> current provider
--provider azure-private-dns --> new additional provider

I will start on it this week.

[saidst]

Please check this out: https://github.com/stsaid/external-dns/commit/07e46bc5605b6c1c415aabd97f2352175e5896e8

I copy&pasted the existing provider but changed the Authorizer.

In the meantime, I try to figure out the credentials issues.

[rnkhouse]

Please let us know if there is any update on this. Thanks!
@saidst

[saidst]

First version incl. tutorials have been completed 2-3 weeks ago.
We depend on #1195 to move on. If this is accepted, we know what to include in the PR for this issue.

[saidst]

1195 has been merged.

I will rebase my feature-branch on master and we can go from there.
@timja, have you also done anything on the actual private dns provider in the meantime?

[timja]

1195 has been merged.

I will rebase my feature-branch on master and we can go from there.
@timja, have you also done anything on the actual private dns provider in the meantime?

no I haven't, I'll test your changes out once you're able to open a PR and let me know if there's anything you're stuck on

[timja]

There's now a PR up (thanks @saidst)
Any testing or feedback on it would be welcomed:
https://github.com/kubernetes-sigs/external-dns/pull/1269

[timja]

Fixed via https://github.com/kubernetes-sigs/external-dns/pull/1195 and https://github.com/kubernetes-sigs/external-dns/pull/1269

Should be released soon (from slack):

muhaha 8:00 AM
Guys? :slightly_smiling_face: When will be released external-dns with Azure Private Zone support ? ( https://github.com/kubernetes-sigs/external-dns/pull/1269 and https://github.com/kubernetes-sigs/external-dns/pull/1195 ) (edited)

@Nick ^
Nick Jüttner 9:30 AM
@muhaha expect a release either this week or in coming week

/close

[k8s-ci-robot]

@timja: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

Fixed via https://github.com/kubernetes-sigs/external-dns/pull/1195 and https://github.com/kubernetes-sigs/external-dns/pull/1269

Should be released soon (from slack):

muhaha 8:00 AM
Guys? :slightly_smiling_face: When will be released external-dns with Azure Private Zone support ? ( https://github.com/kubernetes-sigs/external-dns/pull/1269 and https://github.com/kubernetes-sigs/external-dns/pull/1195 ) (edited)

@Nick ^
Nick Jüttner 9:30 AM
@muhaha expect a release either this week or in coming week

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dstampfli]

I'm doing some testing using in one of my AKS clusters using the Azure Private DNS code that was merged to master as PR 1269. To do this, I cloned the external-dns repo, built a Docker image locally and pushed it to an ACR instance that my cluster can pull from. I built locally because the latest exteranl-dns release is 0.5.17 which doesn't have the PR. I setup things up using this tutorial prepared by @timja and @saidst - https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure-private-dns.md. When I deploy my sample service, I see the following in the logs and no A record gets created in my private DNS zone. Any suggestions? Thanks in advance!

E1205 18:34:59.958913 1 round_trippers.go:324] CancelRequest not implemented by *instrumented_http.Transport
E1205 18:34:59.958990 1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
E1205 18:34:59.959150 1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
E1205 18:34:59.959169 1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)

[timja]

Those can be ignored

[timja]

Does someone want to close this?

[saidst]

@dstampfli, as timja says those logs are really irritating but probably not the cause.

1) Can you turn the application to debug-mode and send the logs after finishing the first synchronisation loop? Care for masking bearer-tokens etc. if there are any (I don't think so).
After the _update-interval_, external-dns will print which records is has found.
Maybee we find a hint into which direction we need to investigate,
```
cli-arg:
--log-level=debug

environment-variable:
AZURE_GO_SDK_LOG_LEVEL=debug
```

2) Can you confirm the SP used by external-dns to have the following role-assignments:
- Minimum 'Reader' on RG?
- Minimum 'Contributor' on Private DNS Zone?

[saidst]

Maybee, we close this ticket and you create a separate issue?

[dstampfli]

Appreciate the quick response. Troubleshooting now. On the ticket, I think it does make sense to close this issue and I create a new one if needed.

[dstampfli]

I added the debug flags and I see the following in the logs. What I see is a lot of "No endpoints could be generated from service..." messages which I suspect is my probelm but I'm not sure how to fix. The ingress controller is working and I can hit the services from the Internet.

This is how I deployed the ingress controller:
helm install stable/nginx-ingress \
--name nginx-ingress \
--set publish-service=default/nginx-ingress-controller \
--set controller.nodeSelector."beta.kubernetes.io/os"=linux \
--set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

Here's the debug log:
time="2019-12-06T13:25:43Z" level=info msg="config: {Master: KubeConfig: RequestTimeout:30s IstioIngressGatewayServices:[istio-system/istio-ingressgateway] ContourLoadBalancerService:heptio-contour/contour Sources:[service ingress] Namespace: AnnotationFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false Compatibility: PublishInternal:false PublishHostIP:false ConnectorSourceServer:localhost:8080 Provider:azure-private-dns GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[aksdemos.com] ExcludeDomains:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup:fido-rg03-poc AzureSubscriptionID:773b8e43-8ed2-4581-92ec-754830be7ba6 AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: Interval:1m0s Once:false DryRun:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false NS1Endpoint: NS1IgnoreSSL:false TransIPAccountName: TransIPPrivateKeyFile:}"
time="2019-12-06T13:25:43Z" level=info msg="Instantiating new Kubernetes client"
time="2019-12-06T13:25:43Z" level=debug msg="kubeMaster: "
time="2019-12-06T13:25:43Z" level=debug msg="kubeConfig: "
time="2019-12-06T13:25:43Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2019-12-06T13:25:43Z" level=info msg="Created Kubernetes client https://172.16.0.1:443"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="service added"
time="2019-12-06T13:25:43Z" level=debug msg="node added"
time="2019-12-06T13:25:43Z" level=debug msg="node added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:43Z" level=debug msg="pod added"
time="2019-12-06T13:25:45Z" level=debug msg="Retrieving Azure Private DNS zones for Resource Group 'fido-rg03-poc'"
(2019-12-06T13:25:45.6099902Z) INFO: REQUEST: GET https://management.azure.com/subscriptions/773b8e43-8ed2-4581-92ec-754830be7ba6/resourceGroups/fido-rg03-poc/providers/Microsoft.Network/privateDnsZones?api-version=2018-09-01
User-Agent: Go/go1.13.4 (amd64-linux) go-autorest/v13.0.1 Azure-SDK-For-Go/v36.0.0 privatedns/2018-09-01
Authorization: REDACTED
(2019-12-06T13:25:45.7756131Z) INFO: RESPONSE: 200 https://management.azure.com/subscriptions/773b8e43-8ed2-4581-92ec-754830be7ba6/resourceGroups/fido-rg03-poc/providers/Microsoft.Network/privateDnsZones?api-version=2018-09-01
Cache-Control: private
Content-Type: application/json; charset=utf-8
Vary: Accept-Encoding
X-Powered-By: ASP.NET
X-Ms-Routing-Request-Id: EASTUS2:20191206T132545Z:48d2524b-65f8-457a-a13b-5e5068f4f5d2
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: Microsoft-IIS/10.0
X-Ms-Request-Id: 171d360c-4466-4eb0-b6c7-d1b16cce8452
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 489
X-Ms-Ratelimit-Remaining-Subscription-Resource-Entities-Read: 59989
X-Aspnet-Version: 4.0.30319
X-Ms-Correlation-Request-Id: 48d2524b-65f8-457a-a13b-5e5068f4f5d2
Date: Fri, 06 Dec 2019 13:25:45 GMT
{"value":[{"id":"\/subscriptions\/773b8e43-8ed2-4581-92ec-754830be7ba6\/resourceGroups\/fido-rg03-poc\/providers\/Microsoft.Network\/privateDnsZones\/aksdemos.com","name":"aksdemos.com","type":"Microsoft.Network\/privateDnsZones","etag":"4612614a-47b1-49b7-bccf-9678a2a0c5c8","location":"global","properties":{"maxNumberOfRecordSets":25000,"maxNumberOfVirtualNetworkLinks":1000,"maxNumberOfVirtualNetworkLinksWithRegistration":100,"numberOfRecordSets":1,"numberOfVirtualNetworkLinks":0,"numberOfVirtualNetworkLinksWithRegistration":0,"provisioningState":"Succeeded"}}]}
time="2019-12-06T13:25:45Z" level=debug msg="Validating Zone: aksdemos.com"
time="2019-12-06T13:25:45Z" level=debug msg="Found 1 Azure Private DNS zone(s)."
time="2019-12-06T13:25:45Z" level=debug msg="Retrieving Azure Private DNS Records for resource group 'fido-rg03-poc'"
time="2019-12-06T13:25:45Z" level=debug msg="Retrieving Azure Private DNS Records for zone 'aksdemos.com'."
(2019-12-06T13:25:45.7760571Z) INFO: REQUEST: GET https://management.azure.com/subscriptions/773b8e43-8ed2-4581-92ec-754830be7ba6/resourceGroups/fido-rg03-poc/providers/Microsoft.Network/privateDnsZones/aksdemos.com/ALL?api-version=2018-09-01
User-Agent: Go/go1.13.4 (amd64-linux) go-autorest/v13.0.1 Azure-SDK-For-Go/v36.0.0 privatedns/2018-09-01
Authorization: REDACTED
(2019-12-06T13:25:46.0169183Z) INFO: RESPONSE: 200 https://management.azure.com/subscriptions/773b8e43-8ed2-4581-92ec-754830be7ba6/resourceGroups/fido-rg03-poc/providers/Microsoft.Network/privateDnsZones/aksdemos.com/ALL?api-version=2018-09-01
X-Ms-Correlation-Request-Id: 9b79d770-a919-4d09-a093-1007293a451a
X-Ms-Routing-Request-Id: EASTUS2:20191206T132546Z:9b79d770-a919-4d09-a093-1007293a451a
X-Content-Type-Options: nosniff
X-Powered-By: ASP.NET
X-Ms-Ratelimit-Remaining-Subscription-Resource-Entities-Read: 59994
Cache-Control: private
Server: Microsoft-IIS/10.0
Date: Fri, 06 Dec 2019 13:25:45 GMT
Content-Type: application/json; charset=utf-8
Vary: Accept-Encoding
X-Ms-Request-Id: a9323b2d-23f3-4bd5-b420-5268f6eb5d55
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 494
X-Aspnet-Version: 4.0.30319
{"value":[{"id":"\/subscriptions\/773b8e43-8ed2-4581-92ec-754830be7ba6\/resourceGroups\/fido-rg03-poc\/providers\/Microsoft.Network\/privateDnsZones\/aksdemos.com\/SOA\/@","name":"@","type":"Microsoft.Network\/privateDnsZones\/SOA","etag":"bcf06533-de30-4737-8aed-c67b85752293","properties":{"fqdn":"aksdemos.com.","ttl":3600,"soaRecord":{"email":"azureprivatedns-host.microsoft.com","expireTime":2419200,"host":"azureprivatedns.net","minimumTTL":null,"refreshTime":3600,"retryTime":300,"serialNumber":1,"minimumTtl":300},"isAutoRegistered":false}}]}
time="2019-12-06T13:25:46Z" level=debug msg="Failed to extract targets for 'aksdemos.com' with type 'SOA'."
time="2019-12-06T13:25:46Z" level=debug msg="Returning 0 Azure Private DNS Records for resource group 'fido-rg03-poc'"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service default/nginx"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service default/nginx2"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service default/nginx-ingress-default-backend"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service kube-system/kube-dns"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service default/nginx-ingress-controller"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service default/kubernetes"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service kube-system/kubernetes-dashboard"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service kube-system/metrics-server"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service kube-system/healthmodel-replicaset-service"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from service kube-system/tiller-deploy"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from ingress default/nginx"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from ingress default/nginx2"
time="2019-12-06T13:25:46Z" level=debug msg="Received 0 changes to process"
time="2019-12-06T13:25:46Z" level=debug msg="Retrieving Azure Private DNS zones for Resource Group 'fido-rg03-poc'"
(2019-12-06T13:25:46.0180711Z) INFO: REQUEST: GET https://management.azure.com/subscriptions/773b8e43-8ed2-4581-92ec-754830be7ba6/resourceGroups/fido-rg03-poc/providers/Microsoft.Network/privateDnsZones?api-version=2018-09-01
User-Agent: Go/go1.13.4 (amd64-linux) go-autorest/v13.0.1 Azure-SDK-For-Go/v36.0.0 privatedns/2018-09-01
Authorization: REDACTED
(2019-12-06T13:25:46.0594410Z) INFO: RESPONSE: 200 https://management.azure.com/subscriptions/773b8e43-8ed2-4581-92ec-754830be7ba6/resourceGroups/fido-rg03-poc/providers/Microsoft.Network/privateDnsZones?api-version=2018-09-01
X-Powered-By: ASP.NET
X-Ms-Routing-Request-Id: EASTUS2:20191206T132546Z:48acf85f-33f3-4621-980d-adc830cc0745
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
Content-Type: application/json; charset=utf-8
Vary: Accept-Encoding
X-Ms-Request-Id: 217e6628-7127-4e28-b072-7fb699e23bb5
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 488
X-Ms-Correlation-Request-Id: 48acf85f-33f3-4621-980d-adc830cc0745
Cache-Control: private
Server: Microsoft-IIS/10.0
X-Aspnet-Version: 4.0.30319
X-Ms-Ratelimit-Remaining-Subscription-Resource-Entities-Read: 59988
Date: Fri, 06 Dec 2019 13:25:45 GMT
{"value":[{"id":"\/subscriptions\/773b8e43-8ed2-4581-92ec-754830be7ba6\/resourceGroups\/fido-rg03-poc\/providers\/Microsoft.Network\/privateDnsZones\/aksdemos.com","name":"aksdemos.com","type":"Microsoft.Network\/privateDnsZones","etag":"4612614a-47b1-49b7-bccf-9678a2a0c5c8","location":"global","properties":{"maxNumberOfRecordSets":25000,"maxNumberOfVirtualNetworkLinks":1000,"maxNumberOfVirtualNetworkLinksWithRegistration":100,"numberOfRecordSets":1,"numberOfVirtualNetworkLinks":0,"numberOfVirtualNetworkLinksWithRegistration":0,"provisioningState":"Succeeded"}}]}
time="2019-12-06T13:25:46Z" level=debug msg="Validating Zone: aksdemos.com"
time="2019-12-06T13:25:46Z" level=debug msg="Found 1 Azure Private DNS zone(s)."
time="2019-12-06T13:25:46Z" level=debug msg="Records to be deleted: 0"
time="2019-12-06T13:25:46Z" level=debug msg="Records to be updated: 0"

[saidst]

As you obviously want to generate records from you ingress-resources, we need the manifests of those two:

time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from ingress default/nginx"
time="2019-12-06T13:25:46Z" level=debug msg="No endpoints could be generated from ingress default/nginx2"

In particular we need to check if

spec:
  rules:
  - host:

matches
DomainFilter:[aksdemos.com]

Can you provide the manifests or an excerpt of them?

[dstampfli]

Sure thing and thanks for your help.

Domain filter from external-dns deployment (partial):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: fidoacr03poc.azurecr.io/external-dns:v1
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=aksdemos.com
        - --provider=azure-private-dns
        - --azure-resource-group=fido-rg03-poc
        - --azure-subscription-id=<subscription id>
        - --log-level=debug

Here's one of the Ingress objects:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: sample.aksdemos.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80
        path: /

[saidst]

looks all good.

have you added this annotation to the ingress-controller?
external-dns.alpha.kubernetes.io/hostname: aksdemos.com.
see https://github.com/helm/charts/tree/master/stable/nginx-ingress#externaldns-service-configuration

you can use this parameter file for helm:

publish-service: default/nginx-ingress-controller
controller:
  nodeSelector:
    beta.kubernetes.io/os: linux
  service:
    annotations:
      external-dns.alpha.kubernetes.io/hostname: aksdemos.com.
defaultBackend:
  nodeSelector:
    beta.kubernetes.io/os: "linux"

[dstampfli]

Thanks @saidst. After adding the annotation and redeploying, an "A" and a "TXT" record were added to my private DNS zone. The A record is for the zone itself ("@") and has the correct IP for the load balancer.

Here are the contents of the TXT record:

"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/default/nginx-ingress-controller"

The bad news is that when I add my Ingress resources, A records are not being created for my services. I think the issue is that I don't see an address populated for the Ingress resource as you can see below. I am looking into this now but if you have any suggestions, I would appreciate it.

NAME     HOSTS                  ADDRESS   PORTS   AGE
nginx1   sample1.aksdemos.com             80      28m

[saidst]

Is there a way you can set up a lab on which you give me access?
You are with Microsoft, so I guess you might have possibilities for that.
On saturday I just created an AKS from scratch and built external-dns from master. I was able to make it work as you describe.

[dstampfli]

Yes, I can give you access to a lab environment. What's the best way to contact you via email? It doesn't look like I can do it directly on github.

[dstampfli]

I finally got this working. Here's the parameters file that I used to create the Nginx ingress controller. I needed to add the _controller.publishService.enabled: true_ parameter. After doing that and redeploying, Ingress resources are creating records in my private DNS zone. I've also included the sample service that I used for testing.

Helm parameters file:

name: nginx-ingress
publish-service: default/nginx-ingress-controller
controller:
  nodeSelector:
    beta.kubernetes.io/os: linux
  publishService:
    enabled: true
  service:
    annotations:
      external-dns.alpha.kubernetes.io/hostname: aksdemos.com  # https://github.com/helm/charts/tree/master/stable/nginx-ingress#externaldns-service-configuration
defaultBackend:
  nodeSelector:
    beta.kubernetes.io/os: linux

Sample service:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx1
spec:
  template:
    metadata:
      labels:
        app: nginx1
    spec:
      containers:
      - image: nginx
        name: nginx1
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx1
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx1
  type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx1
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: sample1.aksdemos.com
    http:
      paths:
      - backend:
          serviceName: nginx1
          servicePort: 80
        path: /

[saidst]

~Good and bad.~
~I _thought_ (!?) made it work without explicitly setting publishService to true.~

Sorry, I made a wrong suggestion because I overlooked my on scripts.

For ingresses we just need this:

publishService:
 enabled: true

Annotating the ingress-controller has no effect on the ingresses. I must have been too fast two days ago...

I think this page confirms it: https://github.com/kubernetes-sigs/external-dns/blob/9418e3acd83db8066d07efb80131c9c3ede03f82/docs/tutorials/aws.md#alias

[guitmz]

So is the new version of external-dns with azure private dns support being released anytime soon? PR was merged a couple of weeks ago, not sure if there is a release cycle in the project

[dstampfli]

No, I cloned master and built the Docker image locally and pushed it
to my container registry. Not sure when a new release is coming.

[lukasmrtvy]

From the Slack channel (2.12.2019):

sszuecs 9:37 AM
... we are working on it, but right now not possible until org change is done

[guitmz]

v0.5.18 was finally released yesterday containing this fix :)

[IbraheemAlSaady]

@guitmz does this work for the stable/external-dns helm chart as well? If yes, I'm not able to make it work for some reason, is there a documentation for the values.yml?

I'm doing:

provider: azure-private-dns

azure:
  resourceGroup:
  tenantId:
  subscriptionId:
  aadClientId:
  aadClientSecret:

Of course its filled with values. But I'm getting this error:

Failed to refresh the Token for request to https://management.azure.com/subscriptions//resourceGroups//providers/Microsoft.Network/privateDnsZones?api-version=2018-09-01

The subscription id and the resource group are not being picked.

I'm using this image:

image:
  registry: docker.io
  repository: bitnami/external-dns
  tag: 0.5.18

[guitmz]

@IbraheemAlSaady I'm currently updating my local chart with the changes of this new release and I should be able to test this week and let you know

[IbraheemAlSaady]

@guitmz
I have opened an issue and I have discovered a missing if statement in the deployment template, Check issue 20212

I'm gonna work on the PR today

[guitmz]

I have a weird issue when using

provider: azure-private-dns
azure:
  resourceGroup: "myGroup"

with this, the --azure-resource-group=myGroup is not added to the container arguments and I get this error:

Status=404 Code=\"SubscriptionNotFound\" Message=\"The subscription 'resourceGroups' could not be found.\""

If I do the same but with provider: azure, everything works well. I guess its the same as you mentioned in https://github.com/helm/charts/issues/20212#issuecomment-576190961 @IbraheemAlSaady

[saidst]

Have you cared for also setting the subscription via an cli-arg?

https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure-private-dns.md#manifest-for-clusters-without-rbac-enabled:

--azure-subscription-id=<use the id of your subscription>

[guitmz]

also, looks like theres more to it @IbraheemAlSaady .. I have added the missing if statement to my local chart and deployed again, the issue still persists even though now the --azure-resource-group=myGroup is there..

I think its related to the comment in your issue where it says that private dns is relying only on env-args at this moment, also just did like @saidst just said and worked, its also necessary in the chart

[IbraheemAlSaady]

@guitmz @saidst, yes indeed the subscription should also be added. There is also the environment variables like tenantId, clientId, and clientSecret, which are required for private-dns. Which is a behavior I’ve noticed. I’m hoping to push the PR today at night (CET) or tomorrow morning.

[guitmz]

I was able to make it work without env vars, just by adding the subscription argument and fixing the if statement as you mentioned in your issue @IbraheemAlSaady , just FYI

[saidst]

Can you paste the (created) manifest for the deployment (of course without credentials) ?

[IbraheemAlSaady]

@guitmz weird, it didn't work for me without the env, it was throwing a 403 error

[guitmz]

heres the templates/deployment.yaml of my local copy of the chart http://dpaste.com/142RY63

My modifications are:

• line 13 to include my azure managed identity annotation
• line 136 (like @IbraheemAlSaady proposed)
• line 184 (like @saidst proposed)

heres my values file for azure provider: http://dpaste.com/0P73S86
and for azure-private-dns provider: http://dpaste.com/2MZ0AR7

currently I have 2 deploys of external-dns, one for each provider since apparently theres no way of using multiple providers at once

rendered deployment for external-dns: http://dpaste.com/35D8MWQ
and for private-dns: http://dpaste.com/2M4M5S1 (named internal-dns just for differentiate both)

perhaps it works in my case because of pod-identity

[saidst]

Thx!

The args look OK (complete).

Do you use the managed-identity-extension for Azure Kubernetes Service (preview feature)?

If so, great. I've not tested that so far. But yes, in theory, the private-dns provider is able to pick up MSI. This is due to the used sdk's from microsoft.

I think you need to handle the case of not using msi. If not seen that while skimming through the code. In this case, you need to set the following env-vars:

AZURE_TENANT_ID (in which the SP exists)
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET

[guitmz]

@saidst yeah, created a managed-identity in azure, gave it DNS zone contributor permissions, used this https://github.com/Azure/aad-pod-identity to attach to my external-dns pods and it works..

I haven't tested without MSI because my end goal is to use it, but I gave a second look at the chart and yeah, I believe it will fail without MSI if the env-vars are not included. Handling it should probably be included of the PR that fixes the chart in upstream if possible

[IbraheemAlSaady]

Changes have been merged

Use version 2.15.1. Here is the PR

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.