Powershell: Build Generic linux-x64 rpm and deb packages

[TravisEz13]

Background: See https://github.com/PowerShell/PowerShell/issues/3721#issuecomment-306266883
tl;dr: This should build the assemblies in a way that they are portable to all the supported Linux x64 platforms for .NET, but requires that we package native binaries which dotnet publish will publish.

• [x] Update build so it can build and publish for the runtime linux-x64
• [x] Update packaging script to package native binaries published and remove unneeded dependencies
• [x] Work with Microsoft Repo owners to develop a package publishing story. @raghushantha and @travisez13 can get you started here.
• [x] Update documentation
• [x] Create .tar.gz package for Linux
• [x] Add documentation about Linux package dependencies
• [x] Update docker files
• [x] Update from 2.0.0-preview1-002106-00 to 2.0.0-preview2-25303-04 that includes https://github.com/dotnet/corefx/pull/19070 to support Ubuntu 17 and Debian 9 (_update: powershell has moved to official .NET Core 2.0 release_)

[janvorli]

@TravisEz13 I've located the very first version that supports Ubuntu 17 and Debian 9. It is 2.0.0-preview2-25303-04.

[borgdylan]

Has .NET Core 2.0 fixed SystemNative_SysConf?

[TravisEz13]

@borgdylan I see you just filed this issue, https://github.com/dotnet/corefx/issues/21386, about the same time you made this comment. It would be more appropriate to ask on that issue if it has been fixed. If you find something out from the .Net Core team that changes our decision to use portable Linux build, that would be a good time to cross-post back to PowerShell, with a link to the issue. Thanks

[borgdylan]

Please keep tracking https://github.com/dotnet/corefx/issues/21386 . Using 2.0 will keep powershell broken for now until corefx gets fixed. A workaround would be to not use Environment.ProcessorCount et al in the Linux build until this gets fixed.

[borgdylan]

The issues arose from a stale copy of System.Native installed system wide. After removing it, powershell starts normally. (tested against beta3)

[iSazonov]

@borgdylan Can you repo this? If so please open new Issue for discussion and maybe fix if system-wide assemblies overlap local ones..

[borgdylan]

The non starting issues are solved for power shell

[daxian-dbw]

@TravisEz13 I built powershell using Linux-x64 RID and Ubuntu.16.04-x64 respectively, and then compared the generated files in \publish folder. It turned out the files are exactly the same. So it looks like there are no additional native binaries published by using linux-x64 when building on Ubuntu.16.04.

Do you happen to know any docs about "many native libs will be included when you publish" when using linux-x64?

[janvorli]

.NET core starting from 2.0.0 is built in a portable way so that build for Linux-x64 RID works on most Linux distros that are based on glibc C library 2.14 or newer (so e.g. not Alpine which is based on MUSL C library). That's why the published components are equal.

[iSazonov]

@janvorli I guess tha same true for Windows RIDs too?

[daxian-dbw]

@janvorli In my case, I built PowerShell on a Ubuntu.16.04 machine with RID's Linux-x64 and Ubuntu.16.04-x64 respectively, and got the same number of published components. Does this mean that even when the Ubuntu.16.04-x64 RID is used, .NET CLI still builds it the same way as if the Linux-x64 RID is in use?

I was assuming that when Linux-x64 is in use, more components would be published comparing to using a more specific RID, because that seemed to be the way for being portable -- you bundle more dependencies with you so as to be less dependent on the individual distro. Is this not correct? Can you please point me to some docs/articles about how the Linux-x64 RID works for building portable apps?

[janvorli]

@chuanjiao10 .NET core now has initial support for redhat 6.x, but it is not supported by the portable build. The specific RID rhel.6-x64 needs to be used. See https://github.com/dotnet/core/blob/master/Documentation/build-and-install-rhel6-prerequisites.md for details. If you are publishing self-contained, you will need a nuget.config with an entry for our myget feed, just like: https://github.com/dotnet/core/blob/master/samples/nuget.config

[janvorli]

@daxian-dbw Linux-x64 isn't any different from Ubuntu.16.04-x64. The change we've made to enable the portable Linux-x64 rid was that we dynamically load OpenSSL and ICU libraries at runtime and we can also handle few missing APIs in various versions of CURL, so we can handle subtle differences between the distros. We don't build any distro specific binaries, so the publish publishes the same files.

[janvorli]

@chuanjiao10 from 2.0

[daxian-dbw]

@janvorli Thank you very much for the explanation!

I built on Ubuntu.16.04 and then played with the generated artifacts on CentOS7. It looks to me that our only obstacle is the native component libpsl-native.so (powershell owns it). We need to figure out a way to compile it in a distro-independent way (possible??) or bundle the native binaries that it depends on.

[janvorli]

@daxian-dbw if you build the libpsl-native.so on CentOS 7, then it should be as portable as the dotnet stuff (we build it on CentOS 7 to ensure that it doesn't depend on newer glibc).

[daxian-dbw]

@janvorli Many thanks! Yes, powershell works fine on Ubuntu.16.04 with the libpsl-native.so built from CentOS7. Maybe the way to go is to build libpsl-native.so on CentOS and then publish it as a nuget package which powershell build can depend on. I will try it out.