Service-fabric: .net "No such host is known" using DNS Service on standalone cluster

[dealboy]

Hi,

using DNS Local SF Service on standalone cluster (6.0.216) from a .net application we get several times (but not constantly) "No such host is known" as shows at the attached screenshot.
The error does not appear when using normal nslookup.

Is this a know issue to you?
can you advise something? is there a workaround ?

kind regards
Yannis

[dealboy]

Here is also a network trace example with dns failing:

System.Net.Http Verbose: 0 : [4680] HttpClientHandler#58119780::SendAsync(HttpRequestMessage#50697503)
System.Net Verbose: 0 : [4680] HttpWebRequest#49442626::HttpWebRequest(https://monitoring.services.domain:30002/api/v1/trace#-2037844861)
System.Net Verbose: 0 : [4680] Exiting HttpWebRequest#49442626::HttpWebRequest() 
System.Net Verbose: 0 : [4680] HttpWebRequest#49442626::HttpWebRequest(uri: 'https://monitoring.services.domain:30002/api/v1/trace', connectionGroupName: '58119780')
System.Net Verbose: 0 : [4680] Exiting HttpWebRequest#49442626::HttpWebRequest() 
System.Net.Http Information: 0 : [4680] Associating HttpRequestMessage#50697503 with HttpWebRequest#49442626
System.Net Information: 0 : [4680] Associating HttpWebRequest#49442626 with ServicePoint#41572837
System.Net.Http Verbose: 0 : [4680] Exiting HttpClientHandler#58119780::SendAsync() -> Task`1#23636518
System.Net Verbose: 0 : [16064] HttpWebRequest#49442626::BeginGetRequestStream()
System.Net Information: 0 : [16064] Associating Connection#1046105 with HttpWebRequest#49442626
System.Net.Sockets Verbose: 0 : [16064] Socket#46758607::Socket(AddressFamily#2)
System.Net.Sockets Verbose: 0 : [16064] Exiting Socket#46758607::Socket() 
System.Net.Sockets Verbose: 0 : [16064] Socket#36175385::Socket(AddressFamily#23)
System.Net.Sockets Verbose: 0 : [16064] Exiting Socket#36175385::Socket() 
System.Net.Sockets Verbose: 0 : [16064] DNS::TryInternalResolve(monitoring.services.domain)
System.Net.Sockets Verbose: 0 : [16064] Socket#46758607::Dispose()
System.Net.Sockets Verbose: 0 : [16064] Socket#36175385::Dispose()
System.Net Error: 0 : [16064] Exception in HttpWebRequest#49442626:: - The remote name could not be resolved: 'monitoring.services.domain'.
at System.Net.ServicePoint.GetConnection(PooledStream PooledStream, Object owner, Boolean async, IPAddress& address, Socket& abortSocket, Socket& abortSocket6)
at System.Net.PooledStream.Activate(Object owningObject, Boolean async, GeneralAsyncDelegate asyncCallback)
at System.Net.Connection.CompleteStartConnection(Boolean async, HttpWebRequest httpWebRequest)
System.Net Verbose: 0 : [16064] HttpWebRequest#49442626::EndGetRequestStream()
System.Net Error: 0 : [16064] Exception in HttpWebRequest#49442626::EndGetRequestStream - The remote name could not be resolved: 'monitoring.services.domain'.
at System.Net.ServicePoint.GetConnection(PooledStream PooledStream, Object owner, Boolean async, IPAddress& address, Socket& abortSocket, Socket& abortSocket6)
at System.Net.PooledStream.Activate(Object owningObject, Boolean async, GeneralAsyncDelegate asyncCallback)
at System.Net.Connection.CompleteStartConnection(Boolean async, HttpWebRequest httpWebRequest)
System.Net.Http Error: 0 : [16064] Exception in HttpClientHandler#58119780::SendAsync - The remote name could not be resolved: 'monitoring.services.domain'.
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
System.Net Verbose: 0 : [16064] Exiting HttpWebRequest#49442626::BeginGetRequestStream() -> ContextAwareResult#17217266

[milvuko]

Hi @dealboy, my best guess is that this is happening due to response latency. Client libraries (Http, Dns) wait for response for some amount of time before the query is sent to the next DNS server in the chain. Based on your description, I would say that the response from DnsService wasn't received in timely fashion which caused the client library to send the request to the next DNS server which responded with "No such host is known".
Note that nslookup has a long timeout (~2sec) and that is probably the reason why it works.
DNS traces would have more information on if/when the request was received and if/when the response was sent back. Expected total time spent in DnsService for a single request should be a few milliseconds at most. An example of a DNS request-response trace looks like this:

2017-12-8 20:50:33.322  DNS.Trace         11152   31348   DnsExchangeOp activityId {ffc90de7-9445-4954-a968-4ffbfa1774a8} UDP read succeeded, status 0, bytesRead 31
2017-12-8 20:50:33.323  DNS.Trace         11152   31348   DnsExchangeOp activityId {ffc90de7-9445-4954-a968-4ffbfa1774a8}, processed query ID( 0x68c8 ) RC( 0 ) Flags ( 8580 ) my.dns Type ( 1 ) A( 10.0.0.4 10.0.0.6 10.0.0.7 10.0.0.11 10.0.0.13 10.0.0.8 10.0.0.10 10.0.0.9 ) SRV( ) TXT( ) 

It would be useful if you could find the trace for the request-response that failed from the client perspective, that would help clarify the issue.

[dealboy]

hi @milvuko , thanks for the response.
Are the entries logged by default of should I enable DNS.Trace ? how could I do that on a standalone SF cluster?

[dealboy]

hi @milvuko, is there any possibility that DnsService happens because of node traffic/load?

[milvuko]

Yes this is happening due to heavy load on the node. DnsService sends the response promptly (under 1ms) but the response gets stuck in the networking queue for more than 100ms. After approximately 30ms, .NET client sends the request to the next DNS in chain (the public DNS) which sends the unknown host response. We are looking into possible solutions.

[ninzavivek]

@dealboy, we are exploring ways to fix the issue. This involves a significant design change and testing effort.

We also haven't seen wide spread reports of this happening. If we see this happening often enough we will definitely prioritize the fix.

[hung-doan]

I have got the same problem. "No such host is known" while using DNS client C#.

In my Windows 10 machine, I tried to run "nslookup ip.application 127.0.0.1" and receive "connection timed out; no servers could be reached."

I wonder if DNS server in Fabric SDK didn't work properly ?!.

Is there any guidance to check if Fabric DNS Server is working as expected?

[vitalybibikov]

The same is for me. For production cluster. Appears rarely, but still.

Can't give any details, as I havn't found any reasons to fail.

[yustun]

@milvuko could you be able to find a solution for this issue. we are facing with the same issue

[chudsonwr]

Yes we are seeing it too in Production cluster.

[dario-ms]

Hi, can you please confirm that the issue happens intermittently while using standard (process) based SF applications (as opposed to container ones)?

[vitalybibikov]

Might be the same https://github.com/Azure/service-fabric-issues/issues/1421

[adrian-lopez-softtek]

Hi, can you please confirm that the issue happens intermittently while using standard (process) based SF applications (as opposed to container ones)?

Yeah, in our case it's an Asp.Net Core 2.2 Api running as stateless service.

[MonDeveloper]

Hi, same issue for us.
We are running the latest SF Version on a "on-prem" cluster: 6.4.658.9590 on Windows Server 2016

Issue occurs sporadically but it has a drastic effects, we noticed it for both .Net Services and guest applications.

Please prioritize it of find a viable workaround, please.

[dario-ms]

+Kayla

Thanks,

Dario

---

From: Marco Monducci [e-mail]
Sent: Thursday, May 9, 2019 6:32 AM
To: Azure/service-fabric-issues
Cc: Dario Bazan Bejarano; Assign
Subject: Re: [Azure/service-fabric-issues] .net "No such host is known" using DNS Service on standalone cluster (#683)

Hi, same issue for us.
We are running the latest SF Version on a "on-prem" cluster: 6.4.658.9590

Issue occurs sporadically but it has a drastic effects, please prioritize it of find a viable workaround, please.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fservice-fabric-issues%2Fissues%2F683%23issuecomment-490906294&data=01%7C01%7Cdariopb%40microsoft.com%7C28ad140ad6cd4473804408d6d482d866%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=IHCG%2FsA%2FTimff326EoLpm2FHhjuonMDgpwt6M1ear%2F0%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAINEJHWSY4RM3BSTMLPFAOTPUQRYPANCNFSM4EGDCWIA&data=01%7C01%7Cdariopb%40microsoft.com%7C28ad140ad6cd4473804408d6d482d866%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=TK%2F3oErpzdhvnJjJ3efMTmi9qZ%2Bn4%2FmPo790mvhlxp0%3D&reserved=0.

[reddogaw]

If @milvuko 's speculation is correct that it's timing out on the DNS resolution and failing over to the next DNS server (e.g. outside of the cluster), is it possible to configure the nodes to act as DNS backups for each other?

[mgrishkov]

We faced with the same problem.
Standalone cluster, AspNetCore 2.2. stateless API server.

[ysabih]

I'ma having the same problem, I'm working behing an enterprise proxy. I tried the same http client code in a standalone asp.net core web api and it worked. When running in the local cluster, I always get "No such host is know". I tried disabling dnsService but it didn't help.

[newpowera]

Got the same problem also. We're using the latest Service Fabric Cluster in Azure and this problem is still there. The "No such host is known" errors cluttered the error log when services are starting. Can someone from MS share the plan for this issue?

[andrewlewiskeoghs]

Still happening - Please resolve

[PlagueHO]

Hi @dario-ms - we're also seeing this in SF clusters in Azure as well. The clusters are upgraded to the latest version. We've raised the issue with support and have been advised to increase DnsService.InstanceCount setting, which has been done, but the problem is still occurring. It does seem like this is the underlying cause, but there does not seem to be any activity on resolving this issue.

Is it possible to get an update?