Home: Some servers of api.nuget.org are giving timeout

[NicolasDorier]

I noticed that api.nuget.org was making timeout on my CI server.

wget https://api.nuget.org/v3-flatcontainer/system.diagnostics.tracesource/index.json
--2018-12-10 13:26:59--  https://api.nuget.org/v3-flatcontainer/system.diagnostics.tracesource/index.json
Resolving api.nuget.org (api.nuget.org)... 152.199.4.33
Connecting to api.nuget.org (api.nuget.org)|152.199.4.33|:443... connected.
HTTP request sent, awaiting response... ^C

It was working on my machine which was resolving a different IP, so I tried to modify my /etc/hosts on the CI server.

echo "117.18.232.200 api.nuget.org" | sudo tee -a /etc/hosts

Then things start working again.

This problem started today.

[sherloqholmes]

I was seeing the same issue on our CI environment.
The problem seems to be resolved now.

[NicolasDorier]

Still had this problem 1H ago.

[skofman1]

Hi, We will need additional information in order to investigate. Please follow the instructions here: https://docs.microsoft.com/en-us/nuget/policies/nuget-faq#nugetorg-not-accessible to gather logs and send to [email protected]

[jugg1es]

You don't even need nuget to reproduce this. Just try to do a GET on https://api.nuget.org/v3-flatcontainer/system.diagnostics.tracesource/index.json

It times out.

[PeteBoyRocket]

The above link @jugg1es posted now works for me in the UK, but we are still seeing this issue on our AWS CI agents hosted in a US region

[skofman1]

@jugg1es , it's a CDN related issue, so it's not reproducible from any location (works from Redmond). To investigate we need logs from YOUR location.

[PeteBoyRocket]

ON it! Bear with me, takes a bit of faffing to get onto our CI nodes...

[jugg1es]

our build servers are linux, I can't run that tool. But the error is coming from AWS us-east-1 and my office which is near DC. Here is the nslookup from my office:
nslookup api.nuget.org
Server: ai-vmdc2.intranet.local
Address: 10.41.117.30

Non-authoritative answer:
Name: cs22.wpc.v0cdn.net
Address: 152.199.4.33
Aliases: api.nuget.org
nugetapiprod.trafficmanager.net
az320820.vo.msecnd.net

Here is the nslookup from the build server:

nslookup api.nuget.org
Server: 172.31.0.2
Address: 172.31.0.2#53

Non-authoritative answer:
api.nuget.org canonical name = nugetapiprod.trafficmanager.net.
nugetapiprod.trafficmanager.net canonical name = az320820.vo.msecnd.net.
az320820.vo.msecnd.net canonical name = cs22.wpc.v0cdn.net.
Name: cs22.wpc.v0cdn.net
Address: 152.199.4.33

[PeteBoyRocket]

Our CI agents that are failing are also in us-east-1 fyi

[skofman1]

Thanks for the info! I'm working with Verizon CDN on investigation.

[PeteBoyRocket]

Heres the WinMTR test results so far:

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                   No response from host -  100 |   56 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   56 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   56 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   56 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   56 |    0 |    0 |    0 |    0 |    0 |
|                           100.65.12.209 -    0 |  284 |  284 |    0 |    1 |   19 |   15 |
|                             52.93.28.99 -    0 |  284 |  284 |    0 |    0 |   21 |    1 |
|                   No response from host -  100 |   56 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   56 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   56 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   56 |    0 |    0 |    0 |    0 |    0 |
|                           52.93.114.122 -    0 |  284 |  284 |    0 |    4 |   30 |   11 |
|                          54.239.111.255 -    0 |  284 |  284 |    0 |    0 |   24 |    0 |
|               core1.dca.edgecastcdn.net -    1 |  276 |  274 |    1 |    2 |   26 |    1 |
|                          152.195.64.129 -    0 |  284 |  284 |    1 |    4 |   39 |    3 |
|                            152.199.4.33 -    0 |  284 |  284 |    0 |    0 |    0 |    0 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

I've also emailed it to [email protected]

[skofman1]

Is the issue still in progress? We need some additional details for investigation:

1. Source IP
2. HAR file (instructions here)

[jugg1es]

no it appears fixed. thank you!

[skofman1]

Is this fixed for everyone else as well or only @jugg1es ?

[NicolasDorier]

@skofman1 the source IP is in my logs 152.199.4.33, give it a try. With a request at https://api.nuget.org/v3-flatcontainer/system.diagnostics.tracesource/index.json. You can simulate this by changing your hosts to have api.nuget.org point to 152.199.4.33 and trying to fetch with wget.

I will try again today on my CI.

[PeteBoyRocket]

@skofman1 Working for me too now! thanks!
(Sorry for the late reply, I was asleep!)

[skofman1]

@NicolasDorier , are you still able to reproduce the issue?

[NicolasDorier]

I just removed the hack I added on my CI server to see if it restores.
Seems to work fine now thanks.

[eSPiYa]

Experiencing the same issue right now. Can't install any nuget package

WinMTR statistics

Host	%	Sent	Recv	Best	Avrg	Wrst	Last
192.168.100.1	0	182	182	0	1	10	1
203.160.163.145	0	182	182	2	6	39	3
10.1.100.1	1	178	177	2	16	272	6
116.93.32.49	4	158	152	2	16	228	5
ip-converge.0.127.121.in-addr.arpa	2	174	172	21	34	310	25
62.216.147.157	2	174	172	40	53	276	43
xe-3-2-1.0.cji02.hkg003.flagtel.com	2	174	172	40	55	273	41
15133.hkg.equinix.com	2	170	167	26	41	263	30
152.195.186.129	2	174	172	27	41	273	28
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0
No response from host	100	36	0	0	0	0	0

[skofman1]

Hi @eSPiYa ! Please follow instructions here: https://docs.microsoft.com/en-us/nuget/policies/nuget-faq#nugetorg-not-accessible and send logs to [email protected].
Please also specify the location of the machine (region) and if it's part of a cloud (like AWS). Usually it's a connectivity issue between you and our CDN and I recommend checking with your ISP as well as us.