Jsdelivr: DNS Issue?

Created on 19 Sep 2018  ·  29Comments  ·  Source: jsdelivr/jsdelivr

I'm not sure this is on your end or on the Azure end of things but my "Availability" tests in Azure has started to fail with some errors from jsDelivr recently (started yesterday). You asked me to provide the resolved IPs but it doesn't seems like that is provided in my error.

Exception (subtype 'WebException') occured at 09/19/2018 07:33:21 (UTC) for Uri 'https://cdn.jsdelivr.net/autocomplete.js/0/autocomplete.min.js', step #1 with the error 'The remote name could not be resolved: 'cdn.jsdelivr.net'', exception text 'Microsoft.VisualStudio.TestTools.WebTesting.ConnectionFailedWebTestException: The following error occurred which may indicate you need to configure a proxy server in your Web test: The remote name could not be resolved: 'cdn.jsdelivr.net' ---> System.Net.WebException: The remote name could not be resolved: 'cdn.jsdelivr.net' at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at Microsoft.VisualStudio.TestTools.WebStress.WebTestTransaction.ResponseReceived(IAsyncResult result) --- End of inner exception stack trace ---', stack trace ' at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at Microsoft.VisualStudio.TestTools.WebStress.WebTestTransaction.ResponseReceived(IAsyncResult result)'.

I do not have much more info than this i'm afraid. I haven't heard any complaints from users and haven't noticed any issues myself. Maybe i should talk with Azure instead?

picture olaj

Most helpful comment

I believe everything is back to normal.
Right now everyone is hitting Cloudflare.
It seems like the issue was with Cedexis(Citrix). They silently started failing on some routes in Europe, mostly France.
It was impossible to detect because it impacted a very small amount of requests in very few locations, so none of our monitoring services detected any problems. Even those running tests from France.

I am really sorry for this issue and will try to make sure this never happens again. We were already looking into using an additional load-balancing services to complement or even replace Cedexis.

Let me know if anyone has any questions.

picture jimaek on 20 Sep 2018
👍5

All 29 comments

Do you know where these tests are running from? What country at least

jimaek picture jimaek on 19 Sep 2018

They are running from various places.

https://i.imgur.com/PwlchJ6.png

The red dots are failed and i do not see any patterns really, as you can see on the % there is about 10% failed on all of them.

olaj picture olaj on 19 Sep 2018

We removed one of our DNS providers in case that was causing the issue. Let me know if that helps.
Unfortunately we still have not gathered enough information to understand the problem

jimaek picture jimaek on 19 Sep 2018
👍2

Still see the errors. https://i.imgur.com/lHuGfcF.png (that graph is from 30 min ago)

olaj picture olaj on 19 Sep 2018

We disabled our load-balancing and switched all traffic to Cloudflare for now. We do not see any issues so its hard to find what is wrong for some of the users. Currently it seems to be limited to France.
Please check and let me know if it helps

jimaek picture jimaek on 19 Sep 2018

Hello There,
Issue still ongoing from France :/
Loading failed for the <script> with source “https://cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js”.

nicoladiaz picture nicoladiaz on 19 Sep 2018

@nicoladiaz it's possible either your local DNS cache or your provider still has the old records. Can you try clearing your DNS cache or switching to a different DNS provider (and maybe even switching back right after)?

MartinKolarik picture MartinKolarik on 19 Sep 2018

i switched to a VPN connection so wait and see :)

nicoladiaz picture nicoladiaz on 19 Sep 2018

Been all green now since about 6-7pm (Stockholm time), so seems resolved.

olaj picture olaj on 20 Sep 2018

I believe everything is back to normal.
Right now everyone is hitting Cloudflare.
It seems like the issue was with Cedexis(Citrix). They silently started failing on some routes in Europe, mostly France.
It was impossible to detect because it impacted a very small amount of requests in very few locations, so none of our monitoring services detected any problems. Even those running tests from France.

I am really sorry for this issue and will try to make sure this never happens again. We were already looking into using an additional load-balancing services to complement or even replace Cedexis.

Let me know if anyone has any questions.

jimaek picture jimaek on 20 Sep 2018
👍5

One of our admins (in France) just had problems loading some files. Started to work again after a while

olaj picture olaj on 1 Oct 2018

Can you please run all these commands on the computer having the issues and send me the full output?
Your IP address would also be helpful.

If Linux or Mac

dig +short cdn.jsdelivr.net
dig +trace cdn.jsdelivr.net
dig +short cdn.jsdelivr.net.cdn.cloudflare.net
dig +short jsdelivr3.dak.netdna-cdn.com
dig +short cdn.jsdelivr.net.mwcloudcdn.com
dig +short dualstack.f3.shared.global.fastly.net
dig +short 2-01-2cd3-000f.cdx.cedexis.net

If Windows

nslookup cdn.jsdelivr.net
nslookup cdn.jsdelivr.net.cdn.cloudflare.net
nslookup jsdelivr3.dak.netdna-cdn.com
nslookup cdn.jsdelivr.net.mwcloudcdn.com
nslookup dualstack.f3.shared.global.fastly.net
nslookup 2-01-2cd3-000f.cdx.cedexis.net
jimaek picture jimaek on 1 Oct 2018

Obviously i'm not the original poster. But i'm in France and me some others near me are having an issue that seem related (well in our case nothing related to Azure though).

Here is the output of the command you asked (all together, i can split it if need be), ip is 84.55.163.170

; <<>> DiG 9.8.3-P1 <<>> +trace cdn.jsdelivr.net
;; global options: +cmd
. 86218 IN NS g.root-servers.net.
. 86218 IN NS c.root-servers.net.
. 86218 IN NS l.root-servers.net.
. 86218 IN NS m.root-servers.net.
. 86218 IN NS a.root-servers.net.
. 86218 IN NS h.root-servers.net.
. 86218 IN NS j.root-servers.net.
. 86218 IN NS b.root-servers.net.
. 86218 IN NS e.root-servers.net.
. 86218 IN NS f.root-servers.net.
. 86218 IN NS d.root-servers.net.
. 86218 IN NS k.root-servers.net.
. 86218 IN NS i.root-servers.net.
;; Received 241 bytes from 213.244.0.15#53(213.244.0.15) in 11 ms

net. 172800 IN NS a.gtld-servers.net.
net. 172800 IN NS b.gtld-servers.net.
net. 172800 IN NS c.gtld-servers.net.
net. 172800 IN NS d.gtld-servers.net.
net. 172800 IN NS e.gtld-servers.net.
net. 172800 IN NS f.gtld-servers.net.
net. 172800 IN NS g.gtld-servers.net.
net. 172800 IN NS h.gtld-servers.net.
net. 172800 IN NS i.gtld-servers.net.
net. 172800 IN NS j.gtld-servers.net.
net. 172800 IN NS k.gtld-servers.net.
net. 172800 IN NS l.gtld-servers.net.
net. 172800 IN NS m.gtld-servers.net.
;; Received 491 bytes from 198.97.190.53#53(198.97.190.53) in 89 ms

jsdelivr.net. 172800 IN NS ns1.r4ns.com.
jsdelivr.net. 172800 IN NS dns1.p03.nsone.net.
jsdelivr.net. 172800 IN NS dns2.p03.nsone.net.
jsdelivr.net. 172800 IN NS dns3.p03.nsone.net.
jsdelivr.net. 172800 IN NS dns4.p03.nsone.net.
jsdelivr.net. 172800 IN NS ns2.r4ns.net.
;; Received 321 bytes from 192.48.79.30#53(192.48.79.30) in 84 ms

cdn.jsdelivr.net. 200 IN CNAME 2-01-2cd3-000f.cdx.cedexis.net.
;; Received 75 bytes from 198.51.44.67#53(198.51.44.67) in 5 ms

104.16.89.20
104.16.87.20
104.16.88.20
104.16.85.20
104.16.86.20
94.31.29.138
163.171.136.65
151.101.122.109

Edit: Things are back (after a bit of wait and then some DNS cache clearing). This command that was returning nothing before (or so it seems to me):

dig +short 2-01-2cd3-000f.cdx.cedexis.net

is now returning

cdn.jsdelivr.net.cdn.cloudflare.net.
104.16.85.20
104.16.87.20
104.16.88.20
104.16.86.20
104.16.89.20

Camalao picture Camalao on 1 Oct 2018

Got same error from France.
Signaled on Twitter (source)
Response : 

Sorry, the problem should be fixed now (may take a few minutes to propagate). 
We believe there's an issue with one of our providers (which we've removed for now) 
but it's limited to few users. 
Could you please email [email protected] to help us debug this?

It came back to normal after few minutes.
I need to flush my local DNS.

Nigui picture Nigui on 1 Oct 2018
👍1

https://github.com/jsdelivr/jsdelivr/issues/18093#issuecomment-425885386

@Camalao

104.16.89.20 104.16.87.20 104.16.88.20 104.16.85.20 104.16.86.20 belong to Cloudflare Anycast Network.
94.31.29.138 belongs to MaxCDN Anycast Network.
163.171.136.65 belongs to CDNetworks/Quantil, Vienna POP.
151.101.122.109 belongs to Fastly Network, France POP.

So it looks like your local dns is returning all A records of jsDelivr.

I think it will be helpful to run dig cdn.jsdelivr.net @8.8.8.8 +trace dig cdn.jsdelivr.net @208.67.222.222 +trace to see whether the wrong result is caused by your local dns or jsDelivr/Cedexis.

SukkaW picture SukkaW on 1 Oct 2018

https://github.com/jsdelivr/jsdelivr/issues/18093#issuecomment-425885386

Also, those IP are from different CNAME (cdn.jsdelivr.net.cdn.cloudflare.net jsdelivr3.dak.netdna-cdn.com cdn.jsdelivr.net.mwcloudcdn.com dualstack.f3.shared.global.fastly.net). How could be possible to return those A records from multi CNAME? There must be something wrong.

As Authoritative DNS will only be allowed and abled to return one CNAME at once, I personally think that it is fault of Local DNS,

SukkaW picture SukkaW on 1 Oct 2018

His output seems simply cut. It's not full

jimaek picture jimaek on 1 Oct 2018

One time out of two dig cdn.jsdelivr.net ends up without any A record and that's what is triggering errors for us. Then this resolution get cached and that's it jsDelivr will be broken for days BUT if you force DNS cache to be removed/resetted. Wonder what we could do

vvo picture vvo on 1 Oct 2018

Our TTL is very low which should force all DNS resolvers to update their caches very fast

jimaek picture jimaek on 1 Oct 2018

@vvo @jimaek Using shorter TTL is not enough.
I think it is necessary to use the global monitors to dig cdn.jsdelivr.net repeatedly from all over the world, prefetching the records. I think it could be a better solution.

SukkaW picture SukkaW on 1 Oct 2018

@jimaek

We have also some DNS issues right now:

christian-mb:~ christian$ dig cdn.jsdelivr.net

; <<>> DiG 9.10.6 <<>> cdn.jsdelivr.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 30776
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;cdn.jsdelivr.net. IN A

;; AUTHORITY SECTION:
jsdelivr.net. 1532 IN SOA ns1.r4ns.com. dak.prospectone.io. 1538395808 10800 3600 604800 3600

;; Query time: 0 msec
;; SERVER: 10.51.0.1#53(10.51.0.1)
;; WHEN: Mon Oct 01 16:58:23 CEST 2018
;; MSG SIZE rcvd: 111

christian-mb:~ christian$ curl -vv https://cdn.jsdelivr.net

  • Rebuilt URL to: https://cdn.jsdelivr.net/
  • Could not resolve host: cdn.jsdelivr.net
  • Closing connection 0
    curl: (6) Could not resolve host: cdn.jsdelivr.net
CEikermann picture CEikermann on 1 Oct 2018

I am also experiencing this problem at the moment. Here is my dig output:
$dig +short cdn.jsdelivr.net $

cortex picture cortex on 1 Oct 2018

Can you do dig +trace cdn.jsdelivr.net instead?

jimaek picture jimaek on 1 Oct 2018 
 $ dig +trace cdn.jsdelivr.net     

; <<>> DiG 9.13.3 <<>> +trace cdn.jsdelivr.net
;; global options: +cmd
.           42280   IN  NS  m.root-servers.net.
.           42280   IN  NS  b.root-servers.net.
.           42280   IN  NS  c.root-servers.net.
.           42280   IN  NS  d.root-servers.net.
.           42280   IN  NS  e.root-servers.net.
.           42280   IN  NS  f.root-servers.net.
.           42280   IN  NS  g.root-servers.net.
.           42280   IN  NS  h.root-servers.net.
.           42280   IN  NS  i.root-servers.net.
.           42280   IN  NS  j.root-servers.net.
.           42280   IN  NS  a.root-servers.net.
.           42280   IN  NS  k.root-servers.net.
.           42280   IN  NS  l.root-servers.net.
.           42280   IN  RRSIG   NS 8 0 518400 20181011170000 20180928160000 41656 . Pm4/nCxVqya8xqRcywKbLS59rHrwFZseFiSvZbsvi8rcovUEsFr3fVbV m4FxkhtT73vt4OCcZ0eoz8yBIfKyhVr/arEsmBqAVAjnVpH7AI8j17Z5 oiQD6fCPFrApkI/9B086MgIeZeuef+euISGGzMsCykpsGA8HL0L33NC1 YDkT97hgIJrr5iV12pqR4Eh5MkxHGtsCjciXkuktwxbuQAuBMz7gjqDg qVnnZ/fmIebv+NonNlqiR3bo2uBSIL+YyUFqS8IuTQ0nGhDzlsdN61DK 5svrCh4yFB53EgswxHOxRFf/JNCdQ+XWdU1nuTSRUYJ5OfIi31tSlRaj HaF1WQ==
;; Received 525 bytes from 192.168.1.1#53(192.168.1.1) in 8 ms

net.            172800  IN  NS  h.gtld-servers.net.
net.            172800  IN  NS  g.gtld-servers.net.
net.            172800  IN  NS  m.gtld-servers.net.
net.            172800  IN  NS  c.gtld-servers.net.
net.            172800  IN  NS  b.gtld-servers.net.
net.            172800  IN  NS  a.gtld-servers.net.
net.            172800  IN  NS  j.gtld-servers.net.
net.            172800  IN  NS  f.gtld-servers.net.
net.            172800  IN  NS  d.gtld-servers.net.
net.            172800  IN  NS  e.gtld-servers.net.
net.            172800  IN  NS  k.gtld-servers.net.
net.            172800  IN  NS  l.gtld-servers.net.
net.            172800  IN  NS  i.gtld-servers.net.
net.            86400   IN  DS  35886 8 2 7862B27F5F516EBE19680444D4CE5E762981931842C465F00236401D 8BD973EE
net.            86400   IN  RRSIG   DS 8 1 86400 20181014050000 20181001040000 2134 . rh18cRS7SH+OsPYDziMo0uvfpLYbw2EL0SUTJ15X97e5Gjn/Jn7SkwD7 vacI8aD1N6fwc5OPcDbfP96yUOJYRfuBRNMUNUWLWPIjXPqrRoo6hRcA xwzzFnxSpOea44+h64LlBWNm1eMOWVRCtR8SbxnUJrwpuuDNeDtnDcOC Q+aHMFH99myntawW9tPammjnNCUWQ8bF+hDUfYJ42THhtW4S81BTQUon 5OvuCdxA516PgWeVOjuABUwdoT36wznDNdmiq4NJQjbs7UdSsaY41hWA hZv5aiUUrcVaPzCNJXuR0vhDfnquoaa29XT2+6q7Nu8Za2A5OL3kG4nC 4oLUXA==
;; Received 1201 bytes from 192.112.36.4#53(g.root-servers.net) in 51 ms

jsdelivr.net.       172800  IN  NS  dns1.p03.nsone.net.
jsdelivr.net.       172800  IN  NS  dns2.p03.nsone.net.
jsdelivr.net.       172800  IN  NS  dns3.p03.nsone.net.
jsdelivr.net.       172800  IN  NS  dns4.p03.nsone.net.
A1RT98BS5QGC9NFI51S9HCI47ULJG6JH.net. 86400 IN NSEC3 1 1 0 - A1RUUFFJKCT2Q54P78F8EJGJ8JBK7I8B NS SOA RRSIG DNSKEY NSEC3PARAM
A1RT98BS5QGC9NFI51S9HCI47ULJG6JH.net. 86400 IN RRSIG NSEC3 8 2 86400 20181007052352 20180930041352 7934 net. Q8G4aTXZC6pkuNOf6Ah76WE5FTqRFwwiTHox8/uE3oSiK07sR4bvrfvh DE7TEj6rIokx25s5Gk4ZqkAnipVKaDRZUKWLq5AOiCE+0Bkx443c9xMR hqvU3NVopD5B90FdQNmrppUFQwzX1CNcBGcDOSI0svdrrIVKrhbiG9LQ V3E=
GBM9BCPKS21UM6V8SM1K0K42C7QMR2IC.net. 86400 IN NSEC3 1 1 0 - GBMKRB78QIII3C3NIFGFSK27G1IBHMM0 NS DS RRSIG
GBM9BCPKS21UM6V8SM1K0K42C7QMR2IC.net. 86400 IN RRSIG NSEC3 8 2 86400 20181008052222 20181001041222 7934 net. jLdRL1D8ZlRKpzypTGaDK8Ik8BDN6q2FnbRmyPgLEcQPQ7u37Lqwkvyy EjnIMyIHMqArqkDeRS6N+pYD/9SOwtgUzk3fNYwpREufhc/0PQ/Xp6Yh bRsMMzVzKsEhoWn9KAZ3me8G7euyb2b23Il66xOaanc9U2mVQ+JMXunE ADo=
;; Received 680 bytes from 192.31.80.30#53(d.gtld-servers.net) in 29 ms

cdn.jsdelivr.net.   200 IN  CNAME   cdn.jsdelivr.net.cdn.cloudflare.net.
;; Received 91 bytes from 198.51.44.3#53(dns1.p03.nsone.net) in 24 ms
cortex picture cortex on 1 Oct 2018

The log says you got a response. There should not be any issues

jimaek picture jimaek on 1 Oct 2018

So is there anyone know what is happening and where the issue is? The cedexis or local DNS?

SukkaW picture SukkaW on 3 Oct 2018

@omnjpa27516 I will try to do it tomorrow

jimaek picture jimaek on 8 Oct 2018

Everything should be back to normal one last time. If anyone sees any problems please email me at [email protected]

jimaek picture jimaek on 9 Oct 2018

Beijing back to normal now.

PING cdn.jsdelivr.net.mwcloudcdn.com (61.149.9.227): 56 data bytes
64 bytes from 61.149.9.227: icmp_seq=0 ttl=51 time=2.946 ms
64 bytes from 61.149.9.227: icmp_seq=1 ttl=51 time=5.117 ms
64 bytes from 61.149.9.227: icmp_seq=2 ttl=51 time=6.108 ms
64 bytes from 61.149.9.227: icmp_seq=3 ttl=51 time=6.260 ms
64 bytes from 61.149.9.227: icmp_seq=4 ttl=51 time=5.719 ms
64 bytes from 61.149.9.227: icmp_seq=5 ttl=51 time=3.896 ms
64 bytes from 61.149.9.227: icmp_seq=6 ttl=51 time=8.097 ms
64 bytes from 61.149.9.227: icmp_seq=7 ttl=51 time=3.256 ms
iflamed picture iflamed on 10 Oct 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

JosefJezek picture JosefJezek  ·  5Comments
bohanyang picture bohanyang  ·  3Comments
Drew233 picture Drew233  ·  8Comments
flopanzer picture flopanzer  ·  4Comments
wearhere picture wearhere  ·  4Comments