Azure-pipelines-tasks: Artifact tool not running on older Linux systems (libstdc++ too old)

[letmaik]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: UniversalPackages

Environment

• Server - Azure Pipelines or TFS on-premises? Azure Pipelines

https://dev.azure.com/WRF-CMake/WRF/_build/results?buildId=370&view=logs&jobId=f7ff90bd-af87-5777-db58-ce71fb90295c&taskId=adf83973-db35-58dc-a374-730505a4a34d&lineStart=14&lineEnd=25&colStart=1&colEnd=1

• Agent - Hosted or Private: Hosted

  • If using Hosted agent, provide agent queue name: ubuntu 16.04

Issue Description

I'm running a Container Job on a CentOS 6 image. It looks like the native artifact tool is built on a fairly recent system, meaning that certain Docker images can't execute it because they don't have a certain minimum libstdc++ version. The docs are silent about any version requirements and simply ask for glibc-based images, which CentOS is.

Being able to test software on older (still officially supported) Linux versions is important, so I consider this a bug. It can be fixed by re-building the artifact tool on an older Linux system to achieve widest compatibility.

(Also reported by another user for RHEL 6 in Developer Community)

Task logs

##[section]Starting: Download WPS outputs
==============================================================================
Task         : Universal Packages
Description  : Download or publish Universal Packages.
Version      : 0.151.1
Author       : Microsoft Corporation
Help         : [More Information](https://aka.ms/universalpackagesannounce)
==============================================================================
SYSTEMVSSCONNECTION exists true
Downloading: https://08wvsblobprodsu6weus73.vsblob.vsassets.io/artifacttool/artifacttool-linux-x64-Release_0.2.99.zip?sv=2017-04-17&sr=b&sig=wW0hSexA6JTcIF15wltu7uy58mPWmXC1kXLB5uwBVBM%3D&spr=https&se=2019-05-23T08%3A27%3A55Z&sp=r&P1=1558599775&P2=11&P3=2&P4=S76fJrVHYdKwsHmB6rGrO4ZhoNsT0ywn%2bysdfzi7Fno%3d
Caching tool: ArtifactTool 0.2.99 x64
SYSTEMVSSCONNECTION exists true
Downloading package: wps.ref, version: 0.0.2 using feed id: WATS
/__t/ArtifactTool/0.2.99/x64/artifacttool universal download --feed WATS --service https://dev.azure.com/WRF-CMake/ --package-name wps.ref --package-version 0.0.2 --path /__w/1/s/wats_wps_outputs/ --patvar UNIVERSAL_DOWNLOAD_PAT --verbosity None
/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool)
/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.17' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool)
/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.5' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool)
/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool)
/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool)
##[error]Error: An unexpected error occurred while trying to download the package. Exit code(1) and error(/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool)
/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.17' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool)
/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.5' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool)
/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool)
/__t/ArtifactTool/0.2.99/x64/artifacttool: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /__t/ArtifactTool/0.2.99/x64/artifacttool))
##[error]Packages failed to download
##[section]Finishing: Download WPS outputs

[zarenner]

Hi @letmaik,

Thanks for this bug report. ArtifactTool is shipped as a self-contained .NET Core executable. It looks like the portable 'linux-x64' runtime identifier does not support RHEL6/Centos6 due to its age, as you note.

I agree with you that since RHEL6 is not yet EOL and has usage we should try to support it. I've filed a bug internally for us to start shipping a a RHEL6 compatible ArtifactTool. I'll take a look at doing so after the holiday here - I'm hoping it'll be as simple as building for the 'rhel.6-x64' .NET Core RID.

[zarenner]

I tested ArtifactTool on CentOS 6, and it works... with the caveat that I couldn't get a compatible libicu working (following https://github.com/dotnet/core/commit/0b1a1631593d6d379fbdfe2b23597a5c25ea4fc9) and had to instead set System.Globalization.Invariant. Installing the libicu binaries linked there didn't seem to work, and for some reason the envvar CORECLR_GLOBAL_INVARIANT didn't work. Setting it in the runsettings did work, but before considering going that route I'd like to figure out if it's possible to actually get libicu working with .NET Core on CentOS 6.

Assuming we can figure that out, next steps are to start deploying the rhel.6-x64 variant and make a very small service change to serve it instead of the generic linux variant.

[letmaik]

How are you getting on? I'm really burning to give this a try.

[zarenner]

Haven't had a chance to get back to this - any chance you're able to see if https://github.com/dotnet/core/blob/master/Documentation/build-and-install-rhel6-prerequisites.md#icu works for you? I can share a private build of ArtifactTool with you.

Assuming you're able to get that working (not sure why it didn't for me, could have just been user error), it's not hard to get it deployed. I just didn't want to do that until the potential ICU issues are resolved.

[zarenner]

@letmaik I spent a good amount of time on this yesterday.

The main issue that we still need to solve is to modify the universalpackages task to send the distro information (like Azure CLI does). The problem is I couldn't find a great nodejs equivalent of python's 'distro' module. Ideally we want the node client to be sending as close as possible distro names between the CLI and the build task.

I found a couple of options - https://github.com/retrohacker/getos seems to be the largest, but takes an odd approach that makes it prone to being incorrect (reporting the _wrong_ distro rather than nothing, see its issues). That makes me a bit hesitant to use it, even though it does appear to be the most mature relevant library I could find. There's also https://github.com/bmacnaughton/linux-os-info, but it only looks at os-release (which centos6 lacks) and not the distro specific release files.

[letmaik]

@zarenner I think getos is probably good enough. I agree it's not ideal but it probably serves 99.9% of use cases for pipelines.

[aasim]

@salvmd @shubham90 - Zach had written this PR, can either of you get it merged to master and resolve this ticket?

[zarenner]

@aasim FYI this overall issue was also blocked on getting the rhel6 variant of artifacttool deployed. However, IIRC for some reason the rhel6 variant was hitting test failures for an unknown reason so that didn't progress.

[letmaik]

@zarenner Is this ever going to happen?

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days