Runtime: Errors installing the SDK during builds

Created on 24 Mar 2020  ·  72Comments  ·  Source: dotnet/runtime

Getting an error from curl during download.

Downloading 'https://dot.net/v1/dotnet-install.sh'
Trying to run 'curl https://dot.net/v1/dotnet-install.sh -sSL --retry 10 --create-dirs -o /__w/1/s/.dotnet/dotnet-install.sh' for maximum of 5 attempts.
Ran 'curl https://dot.net/v1/dotnet-install.sh -sSL --retry 10 --create-dirs -o /__w/1/s/.dotnet/dotnet-install.sh' successfully.
dotnet_install: Warning: Unable to locate zlib. Probable prerequisite missing; install zlib.
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20168.11/dotnet-sdk-5.0.100-preview.3.20168.11-linux-x64.tar.gz
curl: (56) GnuTLS recv error (-9): A TLS packet with unexpected length was received.
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20168.11/dotnet-sdk-5.0.100-preview.3.20168.11-linux-x64.tar.gz
dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20168.11/dotnet-dev-ubuntu.16.04-x64.5.0.100-preview.3.20168.11.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20168.11/dotnet-dev-ubuntu.16.04-x64.5.0.100-preview.3.20168.11.tar.gz
dotnet_install: Error: Could not find/download: `.NET Core SDK` with version = 5.0.100-preview.3.20168.11
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
##[error](NETCORE_ENGINEERING_TELEMETRY=InitializeToolset) Failed to install dotnet SDK from public location (exit code '1').
/__w/1/s/eng/common/tools.sh: line 418: /__w/1/s/.dotnet/dotnet: No such file or directory
##[error](NETCORE_ENGINEERING_TELEMETRY=Build) Build failed (exit code '1').


Runfo Tracking Issue: Errors installing the SDK during builds
|Definition|Build|Kind|Job Name|
|---|---|---|---|
|runtime|909033|PR 42886|CoreCLR Product Build OSX x64 checked|
|runtime|900702|PR 45251|Build Browser wasm Release AllSubsets_Mono_RuntimeTests|
|runtime|891588|PR 44885|CoreCLR Product Build Linux_musl arm release|
|runtime|890567|PR 44847|CoreCLR Product Build OSX arm64 release|
|runtime|884978|PR 44490|CoreCLR Product Build OSX arm64 checked|
|runtime|884470|PR 44565|Installer Build and Test coreclr OSX_arm64 Release|
|runtime|882752|Rolling|Build OSX x64 Release AllSubsets_Mono_LLVMAOT|
|runtime|882681|PR 44529|Build iOS x64 Release AllSubsets_Mono|
|runtime|881713|PR 44495|Libraries Test Run checked coreclr OSX x64 Debug|

Build Result Summary
|Day Hit Count|Week Hit Count|Month Hit Count|
|---|---|---|
|0|1|9|

area-Infrastructure blocking-clean-ci blocking-official-build intermittent
Source
picture jaredpar

All 72 comments

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

Dotnet-GitSync-Bot picture Dotnet-GitSync-Bot on 24 Mar 2020

Data from the CI pipelines

|Build|Kind|Timeline Record|
|---|---|---|
|573152|PR https://github.com/dotnet/runtime/pull/34050|Override CoreCLR from live drop|
|573007|PR https://github.com/dotnet/runtime/pull/33749|Build|
|572108|PR https://github.com/dotnet/runtime/pull/33459|Build|
|572630|PR https://github.com/dotnet/runtime/pull/34071|Build|
|572630|PR https://github.com/dotnet/runtime/pull/34071|Build product|
|572124|PR https://github.com/dotnet/runtime/pull/33864|Build CoreCLR Runtime|
|572619|PR https://github.com/dotnet/runtime/pull/34070|Build product|
|572488|Rolling|Override CoreCLR from live drop|
|572416|PR https://github.com/dotnet/runtime/pull/34059|Override Mono from live drop|
|572412|PR https://github.com/dotnet/runtime/pull/34009|Build CoreCLR Runtime|
|572350|PR https://github.com/dotnet/runtime/pull/32592|Build CoreCLR Runtime|
|572194|Rolling|Build product|
|572061|Rolling|Build CoreCLR Runtime|
|572112|PR https://github.com/dotnet/runtime/pull/34029|Build CoreCLR Runtime|

Evaluated 100 builds
Impacted 13 bulids
Impacted 14 jobs

jaredpar picture jaredpar on 25 Mar 2020

https://github.com/dotnet/sdk/issues/11001

wli3 picture wli3 on 26 Mar 2020

@wli3, I think dotnet-install is working out fine for a long time and it seems like this issue is with the wrong version being picked up by these jobs. For example, it is also happening on Windows / PowerShell side:

link: https://dev.azure.com/dnceng/public/_build/results?buildId=574793&view=logs&jobId=fb000423-50a9-5897-1483-627612d50fb2&j=fb000423-50a9-5897-1483-627612d50fb2&t=0db499c2-6fe8-5392-f998-4ee23307441b

[2020/03/26 18:55:51][INFO] DotNet Install Path: 'F:\workspace\_work\1\s\Payload\performance\tools\dotnet\x64'
[2020/03/26 18:55:51][INFO] Downloading https://dot.net/v1/dotnet-install.ps1
[2020/03/26 18:55:52][INFO] $ pushd "F:\workspace\_work\1\s\Payload\performance"
[2020/03/26 18:55:52][INFO] $ powershell.exe -NoProfile -ExecutionPolicy Bypass F:\workspace\_work\1\s\Payload\performance\tools\dotnet\x64\dotnet-install.ps1 -InstallDir F:\workspace\_work\1\s\Payload\performance\tools\dotnet\x64 -Architecture x64 -Channel master
[2020/03/26 18:55:53][INFO] dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20176.3/dotnet-sdk-5.0.100-preview.3.20176.3-win-x64.zip
[2020/03/26 18:55:55][INFO] dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20176.3/dotnet-sdk-5.0.100-preview.3.20176.3-win-x64.zip
[2020/03/26 18:55:55][INFO] dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20176.3/dotnet-dev-win-x64.5.0.100-preview.3.20176.3.zip
[2020/03/26 18:55:57][INFO] dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20176.3/dotnet-dev-win-x64.5.0.100-preview.3.20176.3.zip
[2020/03/26 18:55:57][INFO] Could not find/download: ".NET Core SDK" with version = 5.0.100-preview.3.20176.3
[2020/03/26 18:55:57][INFO] Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
[2020/03/26 18:55:57][INFO] At F:\workspace\_work\1\s\Payload\performance\tools\dotnet\x64\dotnet-install.ps1:655 char:5
[2020/03/26 18:55:57][INFO] +     throw "Could not find/download: `"$assetName`" with version = $Sp ...
[2020/03/26 18:55:57][INFO] +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[2020/03/26 18:55:57][INFO]     + CategoryInfo          : OperationStopped: (Could not find/...ET Core support:String) [], RuntimeException
[2020/03/26 18:55:57][INFO]     + FullyQualifiedErrorId : Could not find/download: ".NET Core SDK" with version = 5.0.100-preview.3.20176.3
[2020/03/26 18:55:57][INFO] Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
[2020/03/26 18:55:57][INFO]
am11 picture am11 on 26 Mar 2020

Data from today

|Build|Kind|Timeline Record|
|---|---|---|
|583388|PR https://github.com/dotnet/runtime/pull/34297|Build product|
|583335|PR https://github.com/dotnet/runtime/pull/33553|Build CoreCLR Runtime|
|583335|PR https://github.com/dotnet/runtime/pull/33553|Build CoreCLR Runtime|
|583331|Rolling|Restore|
|583239|Rolling|Restore|
|583233|PR https://github.com/dotnet/runtime/pull/33553|Build CoreCLR Runtime|
|582153|Rolling|Build managed test components|
|582358|PR https://github.com/dotnet/runtime/pull/33604|Build CoreCLR Runtime|
|581971|PR https://github.com/dotnet/runtime/pull/33108|Build CoreCLR Runtime|
|580817|PR https://github.com/dotnet/runtime/pull/34314|Build CoreCLR Runtime|
|581803|Rolling|Override CoreCLR from live drop|
|581493|PR https://github.com/dotnet/runtime/pull/34125|Build product|
|581523|Rolling|Override CoreCLR from live drop|

Evaluated 100 builds
Impacted 12 bulids
Impacted 13 jobs

runfo search-timeline -d runtime -c 100 -pr -v "Failed to install dotnet" -m

jaredpar picture jaredpar on 1 Apr 2020

cc @janvorli, @safern, FWIW, I did some digging in my PRs and I thought I should share my findings:

In some cases, this error is due to wrong version being referenced, so it occurs on Linux, macOS and Windows build.

In other cases, from the top comment:

curl: (56) GnuTLS recv error (-9)

it occurs only on Linux machines, where GnuTLS (which has a bug or in other words has incompatible TLS handshake with some protocols/servers) is used as cURL backend, as opposed to OpenSSL based cURL.

Therefore, I think increasing the retries to even 100 in dotnet-install script in dotnet/sdk repo would not fix it. We can update the docker containers to use the desired cURL.

am11 picture am11 on 2 Apr 2020

New repro (Linux musl arm64 release library build):

https://dev.azure.com/dnceng/public/_build/results?buildId=584354&view=logs&jobId=a21a5f2c-b5cf-50e3-b899-59e91a62c520&j=a21a5f2c-b5cf-50e3-b899-59e91a62c520&t=8085b7db-84be-5349-c87f-28c18c6068a1

trylek picture trylek on 2 Apr 2020

@am11

In some cases, this error is due to wrong version being referenced, so it occurs on Linux, macOS and Windows build.

Can you elaborate on what is the wrong version here? Not quite sure which part of the stack your referring to.

it occurs only on Linux machines, where GnuTLS (which has a bug or in other words has incompatible TLS handshake with some protocols/servers) is used as cURL backend, as opposed to OpenSSL based cURL.

Do you know which images these are that we need to look into updating?

jaredpar picture jaredpar on 2 Apr 2020

Can you elaborate on what is the wrong version here?

https://github.com/dotnet/runtime/issues/34015#issuecomment-604729915, in this run, it failed on Linux and Windows, because that day, perf leg were failing because those were requiring the SDK version, which was really non-existing. I think it was later fixed.

Do you know which images these are that we need to look into updating?

I think it is bit more complicated as it does not always fail with the GnuTLS, but whenever it fails on Linux, as far as i can see, it is always with GnuTLS. At that time, you can quickly repro it with GnuTLS-backed curl locally, while wget and OpenSSL-backed curl will happily download the file.

Disclaimer, this information is empirical at best, so please treat it like so. However, it is maybe something to keep an eye on. :)

am11 picture am11 on 2 Apr 2020

@donJoseLuis who's team is driving the fixes here as there is some more context.

Disclaimer, this information is empirical at best, so please treat it like so. However, it is maybe something to keep an eye on. :)

Understood. Appreciate the incite.

We've also frankly been scratching our heads over this one too. It actually being two issues (general flakiness that can be improved with retry and fundamental issue with some images) would explain the data I've been seeing. Specifically that the persistent failures are concentrated in dotnet/runtime where we have a higher investment in docker and lots of Linux images but that the failure does also show up in other repos in batches.

jaredpar picture jaredpar on 2 Apr 2020

To confirm or rule out GnuTLS is a rootcause:

  1. one very quick fix could be to uninstall curl from docker containers, and install wget to satisfy this control flow: https://github.com/dotnet/sdk/blob/dc4d93adbc6d8893779a6e512f79dd43fa78925b/scripts/obtain/dotnet-install.sh#L231.
  2. or ensure all container images get OpenSSL based curl in PATH.
  3. another fix could be to switch these conditions, so wget gets precedence over curl.
am11 picture am11 on 2 Apr 2020
👍1

In some cases, this error is due to wrong version being referenced, so it occurs on Linux, macOS and Windows build.

This issue is where dev test builds of their official build was updating the information of the latest SDK in the specific channel, but since it was a build from a dev branch, it didn't publish any assets. I believe that is getting fixed already.

safern picture safern on 2 Apr 2020

I believe I encountered this (or at least a similar issue) on my PR (#33307):
https://dev.azure.com/dnceng/public/_build/results?buildId=586385&view=logs&j=00c243a4-9c16-5cfb-b528-2a0497049f82&t=41fca9fd-d018-57ef-b1d8-be9a93d2d9d3&l=47

CoreCLR Pri0 Test Run Linux x64 checked

...
dotnet_install: Warning: Unable to locate zlib. Probable prerequisite missing; install zlib.
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20170.6/dotnet-sdk-5.0.100-preview.3.20170.6-linux-x64.tar.gz
curl: (56) TCP connection reset by peer
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20170.6/dotnet-sdk-5.0.100-preview.3.20170.6-linux-x64.tar.gz
dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20170.6/dotnet-dev-centos-x64.5.0.100-preview.3.20170.6.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.3.20170.6/dotnet-dev-centos-x64.5.0.100-preview.3.20170.6.tar.gz
dotnet_install: Error: Could not find/download: `.NET Core SDK` with version = 5.0.100-preview.3.20170.6
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
Failed to install dotnet SDK from public location (exit code '1').
...
josalem picture josalem on 3 Apr 2020

I have also experienced the failure with attempt to download the SDK resulting in HTTP error 404. When that happened, I've tried to open the URL in my local browser on my Windows machine and I was also getting the HTTP error 404. So this had clearly no relation to GnuTLS.

janvorli picture janvorli on 3 Apr 2020

@donJoseLuis

wli3 picture wli3 on 3 Apr 2020

Hello @jaredpar , let us know if the situation is ameliorated, after https://github.com/dotnet/sdk/issues/11001.
Heads up @bozturkMSFT

donJoseLuis picture donJoseLuis on 8 Apr 2020

@donJoseLuis will do. The other TLS issues we had yesterday are muddying the data a bit. Have to probably wait another day to see if it's resolved. I will definitely be looking at the data though.

jaredpar picture jaredpar on 8 Apr 2020

@donJoseLuis it still appears to be a problem. Restricted my query to after 8PM PST on 4/7 and still see 10% failures installing the SDK

|Build|Kind|Timeline Record|Image|
|---|---|---|---|
|593126|PR https://github.com/dotnet/runtime/pull/34662|Build System.Private.CoreLib|ubuntu-16.04-cross-arm64|
|593105|PR https://github.com/dotnet/runtime/pull/32107|Build CoreCLR Runtime|ubuntu-16.04-cross|
|593053|Rolling|Build product|alpine-3.9-WithNode|
|592914|PR https://github.com/dotnet/runtime/pull/34683|Restore and Build Product|ubuntu-16.04-cross|
|592723|Rolling|Build product|ubuntu-16.04-cross|

Evaluated 42 builds
Impacted 5 builds
Impacted 5 jobs

jaredpar picture jaredpar on 9 Apr 2020

@donJoseLuis I dug through the build logs here. It seems the remaining SDK installation issues are specific to ARM. Guessing that means it has to do with the distro / image being used here.

jaredpar picture jaredpar on 9 Apr 2020
👍1

Another hit in this build which is not using the -cross images: https://dnceng.visualstudio.com/internal/_build/results?buildId=601678&view=logs&j=0f626836-ce8e-5186-f350-fec806664c15&t=b67d6b51-c793-5685-ca7c-c2fa3af942cb

Downloading 'https://dot.net/v1/dotnet-install.sh'
Trying to run 'curl https://dot.net/v1/dotnet-install.sh -sSL --retry 10 --create-dirs -o /__w/1/s/.dotnet/dotnet-install.sh' for maximum of 5 attempts.
Ran 'curl https://dot.net/v1/dotnet-install.sh -sSL --retry 10 --create-dirs -o /__w/1/s/.dotnet/dotnet-install.sh' successfully.
dotnet_install: Warning: Unable to locate zlib. Probable prerequisite missing; install zlib.
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.4.20202.8/dotnet-sdk-5.0.100-preview.4.20202.8-linux-x64.tar.gz
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.4.20202.8/dotnet-sdk-5.0.100-preview.4.20202.8-linux-x64.tar.gz
akoeplinger picture akoeplinger on 15 Apr 2020

Maybe we need curl 7.68.0 or above (as opposed to 7.47.0) to fix this TLS error: https://github.com/curl/curl/issues/4624#issuecomment-591483651.

am11 picture am11 on 15 Apr 2020

@bozturkMSFT a heads up on the status of this issue, in case we need to lend a hand.

donJoseLuis picture donJoseLuis on 15 Apr 2020

Yeah it does look like the curl fix didn't fix the issue. Here are the failures since 5PM yesterday which is well after @wfurt merged the fix in.

|Build|Kind|Timeline Record|
|---|---|---|
|602138|PR https://github.com/dotnet/runtime/pull/35010|Restore and Build Product|
|602057|PR https://github.com/dotnet/runtime/pull/35005|Build|
|602061|PR https://github.com/dotnet/runtime/pull/34899|Restore and Build Product|
|601901|PR https://github.com/dotnet/runtime/pull/34974|Prepare TestHost with runtime Mono|
|601903|PR https://github.com/dotnet/runtime/pull/34999|Build product|
|601590|Rolling|Build managed test components|
|601400|PR https://github.com/dotnet/runtime/pull/34822|Build|

Evaluated 48 builds
Impacted 7 builds
Impacted 7 jobs

runfo search-timeline -d runtime -c 100 -pr -v "Failed to install dotnet" -m -after "2020-04-14 5pm"

jaredpar picture jaredpar on 15 Apr 2020

Here is the error dump from the Android leg. Seems more informative than other ones

./build.sh -subset libs -rc release -configuration Release -ci -arch x64   -os Android   /p:RuntimeArtifactsPath=/__w/1/s/artifacts/transport/mono
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /__w/_temp/d3c06a02-32d4-4357-9b98-25f1121d7e6a.sh
__DistroRid: android-x64
__RuntimeId: android-x64
Downloading 'https://dot.net/v1/dotnet-install.sh'
Trying to run 'curl https://dot.net/v1/dotnet-install.sh -sSL --retry 10 --create-dirs -o /__w/1/s/.dotnet/dotnet-install.sh' for maximum of 5 attempts.
Ran 'curl https://dot.net/v1/dotnet-install.sh -sSL --retry 10 --create-dirs -o /__w/1/s/.dotnet/dotnet-install.sh' successfully.
dotnet_install: Warning: Unable to locate zlib. Probable prerequisite missing; install zlib.
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.4.20202.8/dotnet-sdk-5.0.100-preview.4.20202.8-linux-x64.tar.gz
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.4.20202.8/dotnet-sdk-5.0.100-preview.4.20202.8-linux-x64.tar.gz
dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.4.20202.8/dotnet-dev-ubuntu.18.04-x64.5.0.100-preview.4.20202.8.tar.gz
curl: (22) The requested URL returned error: 404 
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.4.20202.8/dotnet-dev-ubuntu.18.04-x64.5.0.100-preview.4.20202.8.tar.gz
dotnet_install: Error: Could not find/download: `.NET Core SDK` with version = 5.0.100-preview.4.20202.8
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
##[error](NETCORE_ENGINEERING_TELEMETRY=InitializeToolset) Failed to install dotnet SDK from public location (exit code '1').
/__w/1/s/eng/common/tools.sh: line 418: /__w/1/s/.dotnet/dotnet: No such file or directory
##[error](NETCORE_ENGINEERING_TELEMETRY=Build) Build failed (exit code '127').

##[error]Bash exited with code '127'.
##[section]Finishing: Restore and Build Product
jaredpar picture jaredpar on 15 Apr 2020

104 | ECONNRESET | Connection reset by peer
-- | -- | --

This looks like real networking problem. Either with the server or infrastructure in between.

wfurt picture wfurt on 16 Apr 2020 
       --retry <num>
              If  a  transient error is returned when curl tries to perform a transfer, it will retry this number of times before giving up. Setting the number to 0 makes curl do no retries
              (which is the default). Transient error means either: a timeout, an FTP 4xx response code or an HTTP 408 or 5xx response code.

              When curl is about to retry a transfer, it will first wait one second and then for all forthcoming retries it will double the waiting time until it reaches  10  minutes  which
              then  will  be  the delay between the rest of the retries.  By using --retry-delay you disable this exponential backoff algorithm. See also --retry-max-time to limit the total
              time allowed for retries.

              If this option is used several times, the last one will be used.

I don't think curl sees RST as transient error. There is certainly no sign of retry.
_If_ this is true, it may with of adding retry around dotnet-install.sh it self.

I will check the handshake to see if our servers send proper close and if the curl link @am11 posted may be relevant.

wfurt picture wfurt on 16 Apr 2020

If any changes are needed in dotnet-install.sh, I will be more than happy to help.

bozturkMSFT picture bozturkMSFT on 16 Apr 2020

We haven't been able to fix the issues with failing builds yet. Please track the progress of the current issue we are working on here: https://github.com/dotnet/sdk/issues/11320

Edit: Fixed the link. Thanks @wfurt

bozturkMSFT picture bozturkMSFT on 20 Apr 2020

The link seems wrong @bozturkMSFT. Different repo?

wfurt picture wfurt on 20 Apr 2020

I see many network-related issues, for example. dotnet-install: Installation finished successfully. but next step MSBuild.dll, error : Failed to download package because of HTTP request timed out.

My questions are:

  • How do you run the builds?
  • How many builds do you run in parallel?
  • If you use containers on Linux, do you run all of them on the same host? Can you get access to the system log of the host system? Maybe we will see some useful information there, e.g. too many interrupts from nic.
genalt picture genalt on 21 Apr 2020

From the get-go, the aspnetcore-ci and aspnetcore-official pipelines (100% unrelated pipelines) have occasionally hit this issue in our "Build: Linux Musl x64" job. These pipelines also use an Alpine docker image though it's somewhat hidden (ci.yml runs ./dockerbuild.sh ubuntu-alpine37). See https://dev.azure.com/dnceng/internal/_build/results?buildId=609519&view=logs&j=83d3b3f3-cac6-55e8-263d-4ea9bf3464dd&t=196c53e3-d592-5666-e468-be934c1ef156 for example.

dougbu picture dougbu on 21 Apr 2020

I'm getting a similar error in one of my PRs which is routinely failing the Linux x64 libraries build:

Trying to run 'curl https://dot.net/v1/dotnet-install.sh -sSL --retry 10 --create-dirs -o /__w/1/s/.dotnet/dotnet-install.sh' for maximum of 5 attempts.
Ran 'curl https://dot.net/v1/dotnet-install.sh -sSL --retry 10 --create-dirs -o /__w/1/s/.dotnet/dotnet-install.sh' successfully.
dotnet_install: Warning: Unable to locate zlib. Probable prerequisite missing; install zlib.
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.8.20362.3/dotnet-sdk-5.0.100-preview.8.20362.3-linux-x64.tar.gz
curl: (56) TCP connection reset by peer
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.8.20362.3/dotnet-sdk-5.0.100-preview.8.20362.3-linux-x64.tar.gz
dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.8.20362.3/dotnet-dev-centos-x64.5.0.100-preview.8.20362.3.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.8.20362.3/dotnet-dev-centos-x64.5.0.100-preview.8.20362.3.tar.gz
dotnet_install: Error: Could not find/download: `.NET Core SDK` with version = 5.0.100-preview.8.20362.3
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
##[error](NETCORE_ENGINEERING_TELEMETRY=InitializeToolset) Failed to install dotnet SDK from public location (exit code '1').
/__w/1/s/eng/common/tools.sh: line 436: /__w/1/s/.dotnet/dotnet: No such file or directory
##[error]Build failed (exit code '127').
##[error]Failed to restore the optimization data package.

https://dev.azure.com/dnceng/public/_build/results?buildId=729905&view=logs&jobId=423ae147-26a3-5a4d-a4d3-4b3fc1dd598e&j=86357178-b8b8-53d0-d37b-335a1e08c946&t=75d29754-3c44-5c16-fcdb-e0edbf440620

josalem picture josalem on 15 Jul 2020

I don't see how we can possibly improve this before we branch-off to 6.0 hence I'm moving this out of 5.0.

ViktorHofer picture ViktorHofer on 4 Aug 2020 
(NETCORE_ENGINEERING_TELEMETRY=InitializeToolset) Failed to install dotnet SDK from public location (exit code '1').

This makes me a sad panda. Seeing this SDK install failure on PR https://github.com/dotnet/runtime/pull/40422.

Coderrob picture Coderrob on 7 Aug 2020

Still happening, very occasionally

Private:
|Build|Kind|Job Name|
|---|---|---|
|826368|Rolling|Build Android arm64 release AllSubsets_Mono|

No public failures in the last 100 builds

directhex picture directhex on 28 Sep 2020

@directhex overall across the org it's happening about once a day

https://runfo.azurewebsites.net/search/timelines/?bq=started%3A%7E7&tq=Failed+to+install+dotnet+SDK

jaredpar picture jaredpar on 28 Sep 2020

Seems to only be happening on macOS. Does the sdk team know about that?

ViktorHofer picture ViktorHofer on 28 Sep 2020

I disagree with it only on osx? https://dev.azure.com/dnceng/internal/_build/results?buildId=826368&view=results is on linux

directhex picture directhex on 28 Sep 2020

Ok ok, I was only looking at Jared's data which listed macOS clients only.

ViktorHofer picture ViktorHofer on 28 Sep 2020

The difference here is the website doesn't have data on internal builds, only public ones. Point in time problem. I didn't want to deal with any authorization issues up front hence only put public data there. Now I have auth going, can add in internal builds (once I verify all the rules around it).

jaredpar picture jaredpar on 28 Sep 2020

Cool. @jaredpar who would be the right contact to restart the discussion?

ViktorHofer picture ViktorHofer on 1 Oct 2020

@marcpopMSFT is likely a good starting point here.

jaredpar picture jaredpar on 1 Oct 2020

@bozturkMSFT and @donJoseLuis who manage the install scripts. They've been adding telemetry to monitor the installs. I can't access Jared's link but from Jo's link, it was targeting a valid build build and reported an error in curl so I'll have to differ to the two of them.

We did fix an issue a few weeks back where if someone triggered a private build of installer, the install script would still pick it up and try to download it (and it wouldn't be available) but that's not the situation in that particular failure.

marcpopMSFT picture marcpopMSFT on 2 Oct 2020

Thanks @marcpopMSFT . I will check to see why it happens and how we can improve it.

bozturkMSFT picture bozturkMSFT on 15 Oct 2020

@bozturkMSFT

https://runfo.azurewebsites.net/search/timelines/?bq=started%3A%7E7&tq=Failed+to+install+dotnet+SDK

That will show you all the times it happened in the last ~7 days across the dotnet GitHub organization. Looks like ~10 total times across ~4,600 builds of which there are probably 10-20X as many jobs. So rough math would be that this is happening probably at most .015% of the time right now. IMHO that number is really low, seems like in the range of normal network errors where possibly we should be doing a better job of retrying.

On the other hand one interesting piece of data is the errors are pretty much exclusively on OSX. There are some Linux failures but predomiently it's OSX. Not sure if that's anything worth looking into.

jaredpar picture jaredpar on 15 Oct 2020
👍1

On the other hand one interesting piece of data is the errors are pretty much exclusively on OSX

Maybe related/connected to https://github.com/dotnet/core-eng/issues/9448

akoeplinger picture akoeplinger on 16 Oct 2020

@bozturkMSFT have you had a chance to take a look or do you have a tracking that I can subscribe to? Asking as this is still happening regularly in our builds.

ViktorHofer picture ViktorHofer on 19 Oct 2020

@ViktorHofer After Jared's response, I haven't taken a look since the issue seems rare enough to be just network related.
Can you send me links to a couple of your builds?

bozturkMSFT picture bozturkMSFT on 19 Oct 2020

@bozturkMSFT https://runfo.azurewebsites.net/search/timelines/?bq=started:~5&tq=Failed+to+install+dotnet+SDK+from+public+location.

ViktorHofer picture ViktorHofer on 19 Oct 2020

@ViktorHofer @jaredpar All the issues seem to be network related. However, we noticed that the retry logic for the bash script is not as good as what we have in the powershell script. I will create a PR in the coming days to improve it. Hopefully, this should resolve the issue.

bozturkMSFT picture bozturkMSFT on 22 Oct 2020

@bozturkMSFT are you sure about this? Isn't the 403 a symptom of something else going on?

dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-rc.2.20479.15/dotnet-sdk-5.0.100-rc.2.20479.15-osx-x64.tar.gz
curl: (7) Failed to connect to dotnetcli.azureedge.net port 443: Connection refused
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-rc.2.20479.15/dotnet-sdk-5.0.100-rc.2.20479.15-osx-x64.tar.gz
dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-rc.2.20479.15/dotnet-dev-osx-x64.5.0.100-rc.2.20479.15.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-rc.2.20479.15/dotnet-dev-osx-x64.5.0.100-rc.2.20479.15.tar.gz
dotnet_install: Error: Could not find/download: `.NET Core SDK` with version = 5.0.100-rc.2.20479.15
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
Failed to install dotnet SDK from public location (exit code '1').
ViktorHofer picture ViktorHofer on 22 Oct 2020

@ViktorHofer You are right, "network related" was a bad choice of words to explain the situation. What I meant is that the issue doesn't seem to be install-script-specific. It is possible that there is an Azure-CDN related issue.

bozturkMSFT picture bozturkMSFT on 22 Oct 2020

How can we know for sure? With whom do we need to follow-up regarding a potential Azure CDN issue?

ViktorHofer picture ViktorHofer on 22 Oct 2020

@MattGal Do you know who we can contact about this?

bozturkMSFT picture bozturkMSFT on 26 Oct 2020

Taking a look, but if the issue is with the Azure CDN or storage account, our only recourse is support ticket(s)

MattGal picture MattGal on 26 Oct 2020

@ViktorHofer / @bozturkMSFT I looked at the CDN and storage account a bit and we don't have any diagnostic logging turned on for dotnetcli.azureedge.net. As long as @leecow is OK with that I'm down to turn it on and see if it shows us anything helpful, but a lot of times "connection refused" or "connection timeout" doesn't actually show up on the destination side of logging, so this may or may not help.

If we can gather 5-10 examples that include timestamps (list of build URIs would be sufficient) feel free to open an issue in dotnet/core-eng and tag @ilyas1974 and myself, and we can try to talk to the support folks. Historically, this has not been a fruitful effort but if it's happening a lot maybe that will help our case.

MattGal picture MattGal on 26 Oct 2020

This happened again here:

https://dev.azure.com/dnceng/public/_build/results?buildId=873280&view=logs&j=f2a23cee-3431-50cb-c21a-1d5929055501&t=8aeffa89-f24d-58b0-d0c4-49f806ab2e85&l=33

https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/873280/logs/244

eerhardt picture eerhardt on 2 Nov 2020

As long as @leecow is OK with that I'm down to turn it on and see if it shows us anything helpful

@MattGal did you turn the diagnostics logging on?

ViktorHofer picture ViktorHofer on 3 Nov 2020

As long as @leecow is OK with that I'm down to turn it on and see if it shows us anything helpful

@MattGal did you turn the diagnostics logging on?

If you mean for the CLI storage account yes, classic diagnostics logging is on (and "modern" is not disable-able, I think?) and contains the last 7 days for the account. A lot of times issues like this result in the account never actually seeing the failed requests though. I'll take a quick peek at the timestamps around Eric's builds above and let you know if I see anything.

MattGal picture MattGal on 3 Nov 2020

@MattGal - note that the first error in the log is what caused my error. The real error gets hidden in the log because the dotnet-install script tries falling back to the "legacy" URL, which doesn't exist.

2020-11-02T17:47:25.7656940Z dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-rc.2.20479.15/dotnet-sdk-5.0.100-rc.2.20479.15-osx-x64.tar.gz
2020-11-02T17:48:35.0034950Z curl: (56) SSLRead() return error -9806

Do some mac machines have invalid certs installed or something?

eerhardt picture eerhardt on 3 Nov 2020

Unfortunately the logging doesn't have an error on that time range for any of the mac blobs. I found only one error anywhere nearby (about 9 minutes previous) accessing https://dotnetcli.blob.core.windows.net/dotnet/beta/Installers/Latest/dotnet-dev-osx-x64.latest.pkg. There was no log of anyone downloading dotnet-sdk-5.0.100-rc.2.20479.15-osx-x64.tar.gz in this time range in the log blobs.

It's absolutely possible for the macs to be partially messed up this way; we need as many data points as we can get to narrow it down. Providing as many builds as we can in the first URL format will help narrow it down and share with the team in case their machines are being misconfigured.

MattGal picture MattGal on 3 Nov 2020

Providing as many builds as we can in the first URL format will help narrow it down

Using the link @jaredpar posted above

https://runfo.azurewebsites.net/search/timelines/?bq=started%3A%7E7&tq=Failed+to+install+dotnet+SDK

All 3 of the occurrences (one of which was mine) have the same error message:

curl: (56) SSLRead() return error -9806

eerhardt picture eerhardt on 3 Nov 2020

Regarding the previous conversation about curl issue; if we switch the order of if/elif conditions here: https://github.com/dotnet/install-scripts/blob/ad55554a0b84244ff8c68579df8b84af30b9abfc/src/dotnet-install.sh#L711-L714 (wget-first) and these error disappear; then it would mean that all unix-y agents just need curl v7.47.0 or above. Note that the fixes that went into curl v7.47.0 applied to all TLS libraries (including libressl used by curl on macOS).

ps - dotnet-install script could anyway provide an option to select tool of choice (curl vs. wget) to avoid such code changes.

am11 picture am11 on 3 Nov 2020

@eerhardt I have a few things to chat about w/ the Mac MMS folks, but 3 is really just not a lot of data, if we can get more that'd be great.

MattGal picture MattGal on 3 Nov 2020

Converted this issue to a live tracking one (see top post). This still happens happens, but less often.

ViktorHofer picture ViktorHofer on 12 Nov 2020

Happened again in this official build: https://dev.azure.com/dnceng/internal/_build/results?buildId=895670&view=results. Unsure what's the current state of this issue. Is someone actively working on a mitigation? cc @marcpopMSFT

ViktorHofer picture ViktorHofer on 1 Dec 2020

@bozturkMSFT

wli3 picture wli3 on 1 Dec 2020

@donJoseLuis @bozturkMSFT are driving improvements into the install script at the moment. We should probably retry the initial link rather than falling back to the legacy link as that never seems to work.

marcpopMSFT picture marcpopMSFT on 1 Dec 2020

It would probably make sense to retry initial URL for network failures but not 404 and perhaps others. I'm not sure if we can get that granularity.
Is the legacy link completely broken or does it work only for certain .NET versions?
It does seems like it is _always_ for recent .NET versions.

wfurt picture wfurt on 1 Dec 2020

Is the legacy link completely broken or does it work only for certain .NET versions?

It only works for earlier versions of .NET. It is the “legacy” URL format. I think the switch happened sometime around 2.0

eerhardt picture eerhardt on 1 Dec 2020

so for 3.0+ it would make sense to only try the initial url, right? e.g. never fall-back for master and 5.0

wfurt picture wfurt on 1 Dec 2020

so for 3.0+ it would make sense to only try the initial url, right? e.g. never fall-back for master and 5.0

Yes, I think that would be fine. The URL naming change came in the summer of 2017 - https://github.com/dotnet/cli/pull/6764. For versions above that, it doesn't make sense to try the legacy URL format.

eerhardt picture eerhardt on 1 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Timovzl picture Timovzl  ·  3Comments
chunseoklee picture chunseoklee  ·  3Comments
v0l picture v0l  ·  3Comments
btecu picture btecu  ·  3Comments
matty-hall picture matty-hall  ·  3Comments