Azure-pipelines-tasks: ERR:connect ECONNREFUSED with Nuget Tool Installer

[AHoran]

Environment

VSTS

Agent - Private:
Windows 2008 R2.
Current agent version: '2.123.0'

Issue Description

Have been using the Nuget Tool Installer to get nuget 4.3.0 installed for a build that requires it. Has been working fine, but this week it is having network issues of some sort. (ERR:connect ECONNREFUSED)
Machine seems to have full internet connectivity. We do have a proxy but that is configured.
Have reinstalled the agent.

Error logs

2017-10-30T03:44:01.4041864Z ##[section]Starting: Phase 1
2017-10-30T03:44:01.4041864Z Current agent version: '2.123.0'
2017-10-30T03:44:01.4041864Z Agent is running behind proxy server: 'http://1.2.3.4:port'
2017-10-30T03:44:02.8705394Z ##[section]Starting: Initialize Job
2017-10-30T03:44:02.8861389Z Prepare build directory.
2017-10-30T03:44:02.9329374Z Set build variables.
2017-10-30T03:44:02.9485369Z Download all required tasks.
2017-10-30T03:44:02.9485369Z Downloading task: NuGetToolInstaller
2017-10-30T03:44:05.2748624Z ##[section]Finishing: Initialize Job
2017-10-30T03:44:05.3060614Z ##[section]Starting: Get Sources
2017-10-30T03:44:05.3528599Z Syncing repository: {repoName} (TfsGit)
2017-10-30T03:44:05.3528599Z Prepending Path environment variable with directory containing 'git.exe'.
2017-10-30T03:44:05.3684594Z ##[command]git version
2017-10-30T03:44:05.4152579Z git version 2.12.2.windows.3
2017-10-30T03:44:05.4308574Z ##[command]git config --get remote.origin.url
2017-10-30T03:44:05.4620564Z ##[command]git config gc.auto 0
2017-10-30T03:44:05.4932554Z ##[command]git config --get-all http.https://xxxxxx
2017-10-30T03:44:05.5088549Z ##[command]git config --get-all http.proxy
2017-10-30T03:44:05.5400539Z ##[command]git -c http.extraheader="AUTHORIZATION: bearer **" -c http.proxy="http://1.2.3.4:port" fetch --tags --prune --progress --no-recurse-submodules origin
2017-10-30T03:44:05.9300414Z ##[command]git checkout --progress --force {guid}
2017-10-30T03:44:06.0236384Z HEAD is now at d91dd8e... added appsettings.Release.json/appsettings.Debug.json for transform purpose
2017-10-30T03:44:06.0392379Z ##[section]Finishing: Get Sources
2017-10-30T03:44:06.0392379Z ##[section]Starting: Use NuGet >= 4.3.0
2017-10-30T03:44:06.0548374Z ==============================================================================
2017-10-30T03:44:06.0548374Z Task : NuGet Tool Installer
2017-10-30T03:44:06.0548374Z Description : Acquires a specific version of NuGet from the internet or the tools cache and adds it to the PATH. Use this step to change the version of NuGet used in the NuGet steps.
2017-10-30T03:44:06.0548374Z Version : 0.1.5
2017-10-30T03:44:06.0548374Z Author : Microsoft Corporation
2017-10-30T03:44:06.0548374Z Help : More Information
2017-10-30T03:44:06.0548374Z ==============================================================================
2017-10-30T03:44:06.4136259Z You are using a query match on the version string. Behavior changes or breaking changes might occur as NuGet updates to a new version.
2017-10-30T03:44:07.4899914Z ERR:connect ECONNREFUSED 117.18.232.200:443
2017-10-30T03:44:07.4899914Z ##[section]Finishing: Use NuGet >= 4.3.0

[jotaylo]

Is that IP address by any chance the IP address of your proxy server?
The tool installer currently doesn't respect the proxy settings on the agent, I will see to it that that is fixed.

[AHoran]

The IP address in the error isn't one of mine it's external : _ERR:connect ECONNREFUSED 117.18.232.200:443_

Where i've substituted _http://1.2.3.4:port_ that is my internal proxy that the agent uses, no proxy auth required.

This line is to my VSTS : _2017-10-30T03:44:05.4932554Z ##[command]git config --get-all http.https://xxxxxx_

Note this was internal private build server connected to VSTS.

[jotaylo]

The IP address might be the nuget.org CDN. Can you please set the system.debug variable to true, run the build again, and post the log of just the tool installer task?

[markjsc]

I'm having the same issue with similar circumstances (proxy, works for other operations but fails for this one).

Here is the debug dump from the Use NuGet task:

2017-11-28T19:14:22.9973345Z ##[debug]Evaluating condition for step: 'Use NuGet 4.3.0'
2017-11-28T19:14:22.9973345Z ##[debug]Evaluating: succeeded()
2017-11-28T19:14:22.9973345Z ##[debug]Evaluating succeeded:
2017-11-28T19:14:22.9973345Z ##[debug]=> (Boolean) True
2017-11-28T19:14:22.9973345Z ##[debug]Expanded: True
2017-11-28T19:14:22.9973345Z ##[debug]Result: True
2017-11-28T19:14:22.9973345Z ##[section]Starting: Use NuGet 4.3.0
2017-11-28T19:14:23.0129597Z ==============================================================================
2017-11-28T19:14:23.0129597Z Task         : NuGet Tool Installer
2017-11-28T19:14:23.0129597Z Description  : Acquires a specific version of NuGet from the internet or the tools cache and adds it to the PATH. Use this step to change the version of NuGet used in the NuGet steps.
2017-11-28T19:14:23.0129597Z Version      : 0.1.5
2017-11-28T19:14:23.0129597Z Author       : Microsoft Corporation
2017-11-28T19:14:23.0129597Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=852538)
2017-11-28T19:14:23.0129597Z ==============================================================================
2017-11-28T19:14:23.4035896Z ##[debug]agent.workFolder=E:\TFSBuildAgent\_work
2017-11-28T19:14:23.5598409Z ##[debug]loading inputs and endpoints
2017-11-28T19:14:23.5598409Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2017-11-28T19:14:23.5598409Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2017-11-28T19:14:23.5598409Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2017-11-28T19:14:23.5598409Z ##[debug]loading INPUT_CHECKLATEST
2017-11-28T19:14:23.5598409Z ##[debug]loading INPUT_VERSIONSPEC
2017-11-28T19:14:23.5598409Z ##[debug]loaded 5
2017-11-28T19:14:23.5598409Z ##[debug]Agent.ProxyUrl=http://proxy.sscgp.dmz:3128
2017-11-28T19:14:23.5598409Z ##[debug]Agent.ProxyUsername=undefined
2017-11-28T19:14:23.5598409Z ##[debug]Agent.ProxyPassword=undefined
2017-11-28T19:14:23.5598409Z ##[debug]Agent.ProxyBypassList=undefined
2017-11-28T19:14:23.5598409Z ##[debug]check path : E:\TFSBuildAgent\_work\_tasks\NuGetToolInstaller_2c65196a-54fd-4a02-9be8-d9d1837b7c5d\0.1.5\node_modules\vsts-task-tool-lib\lib.json
2017-11-28T19:14:23.5598409Z ##[debug]set resource file to: E:\TFSBuildAgent\_work\_tasks\NuGetToolInstaller_2c65196a-54fd-4a02-9be8-d9d1837b7c5d\0.1.5\node_modules\vsts-task-tool-lib\lib.json
2017-11-28T19:14:23.5598409Z ##[debug]system.culture=en-US
2017-11-28T19:14:23.5598409Z ##[debug]check path : E:\TFSBuildAgent\_work\_tasks\NuGetToolInstaller_2c65196a-54fd-4a02-9be8-d9d1837b7c5d\0.1.5\task.json
2017-11-28T19:14:23.5598409Z ##[debug]set resource file to: E:\TFSBuildAgent\_work\_tasks\NuGetToolInstaller_2c65196a-54fd-4a02-9be8-d9d1837b7c5d\0.1.5\task.json
2017-11-28T19:14:23.5598409Z ##[debug]system.culture=en-US
2017-11-28T19:14:23.5598409Z ##[debug]versionSpec=4.3.0
2017-11-28T19:14:23.5598409Z ##[debug]checkLatest=false
2017-11-28T19:14:23.5598409Z ##[debug]isExplicit: 4.3.0
2017-11-28T19:14:23.5598409Z ##[debug]explicit? true
2017-11-28T19:14:23.5598409Z ##[debug]Exact match expected on version: 4.3.0
2017-11-28T19:14:23.5598409Z ##[debug]Trying to get tool from local cache
2017-11-28T19:14:23.5598409Z ##[debug]isExplicit: 4.3.0
2017-11-28T19:14:23.5598409Z ##[debug]explicit? true
2017-11-28T19:14:23.5598409Z ##[debug]Agent.Version=2.123.0
2017-11-28T19:14:23.5598409Z ##[debug]Agent.ToolsDirectory=E:\TFSBuildAgent\_work\_tool
2017-11-28T19:14:23.5598409Z ##[debug]checking cache: E:\TFSBuildAgent\_work\_tool\NuGet\4.3.0\x64
2017-11-28T19:14:23.5598409Z ##[debug]not found
2017-11-28T19:14:23.5598409Z ##[debug]Agent.Version=2.123.0
2017-11-28T19:14:23.5598409Z ##[debug]Agent.ToolsDirectory=E:\TFSBuildAgent\_work\_tool
2017-11-28T19:14:23.5598409Z ##[debug]isExplicit: 4.0.0
2017-11-28T19:14:23.5598409Z ##[debug]explicit? true
2017-11-28T19:14:23.5598409Z ##[debug]isExplicit: 4.1.0
2017-11-28T19:14:23.5598409Z ##[debug]explicit? true
2017-11-28T19:14:23.5598409Z ##[debug]evaluating 2 versions
2017-11-28T19:14:23.5598409Z ##[debug]match not found
2017-11-28T19:14:23.5598409Z ##[debug]Querying versions list
2017-11-28T19:14:24.6069123Z ERR:connect ECONNREFUSED 72.21.81.200:443
2017-11-28T19:14:24.6069123Z ##[debug]task result: Failed
2017-11-28T19:14:24.6069123Z ##[debug]Processed: ##vso[task.complete result=Failed;]
2017-11-28T19:14:24.6223570Z ##[section]Finishing: Use NuGet 4.3.0

The IP in question 72.21.81.200 is for https://dist.nuget.org/tools.json. I can reach that URL using a browser (with proxy) on my build agent, so it sounds like this task isn't picking up the proxy settings.

[AHoran]

Thanks for the debug output Mark, I've since worked around by using a build server that didn't need a proxy, and also by using a hosted agent, and so couldn't generate it.

I can confirm that the IP address looks to be a nuget CDN

C:\>nslookup dist.nuget.org 8.8.8.8
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Non-authoritative answer:
Name:    cs9.wpc.v0cdn.net
Addresses:  2606:2800:147:120f:30c:1ba0:fc6:265a
          117.18.232.200
Aliases:  dist.nuget.org
          db16.wpc.azureedge.net

[AHoran]

If anyone needs a workaround...

I spotted on here that you can set a http_proxy environment variable, and in my testing it seems to work. (restart agent after changing)

Set http_proxy to http://proxy:port see the linked post for details of authentication.

[markjsc]

@AHoran - The workaround worked for me! Good find.

I added the http_proxy (and also https_proxy, for good measure) variable as a VSTS Process Variable. No more ECONNREFUSED error!

Thanks for sharing the workaround.

[pabrams]

If I set the proxy variable, the "Checkout" task fails. The requested URL returned error: 400. The checkout task works fine if I don't set the proxy variable.