Powershell: Can we get a release that runs on Alpine?

[JasonMorgan]

Could we get an apk package for alpine linux? The main reason we don't use powershell as part of our ci/cd pipeline is that all our tasks run in alpine containers.

[SteveL-MSFT]

Are you able to use the AppImage?

[JasonMorgan]

I tried and couldn't get it to work. I don't see it in their list of supported os's.

[markekraus]

I took a stab at running the Appimage in Alpine. It looks like you could do it only if you have a full glibc chroot. The primary issue seems to be the musl libc that alpine uses and Appimage is made for glibc environments. It certainly increases the size of Alpine if you have to tack on a full glibc chroot to run the Appimage.

[JasonMorgan]

@SteveL-MSFT any thoughts on this? Adding an apk package would make it so I could start using it in most of my tools containers and I'm sure I'm not the only one.

[SteveL-MSFT]

@JasonMorgan Alpine isn't one of the official supported OS for corefx, so for us this isn't currently a priority. If a community PR makes it work, we'd be happy to take that change and start producing packages.

[SteveL-MSFT]

Looks like Alpine is sort of supported by CoreCLR, they have a build, but it doesn't appear to be tested regularly.

[markekraus]

dotnet/coreclr#917 It looks like they are still working the kinks out.

[markekraus]

dotnet/dotnet-docker#22 and dotnet/announcements#49 looks like they added an alpine docker for .NET Core. https://github.com/dotnet/dotnet-docker-samples/tree/master/dotnetapp-prod-alpine-preview so it should be theoretically possible to get a pwsh docker from that now.

[ghost]

How can we make this theory a fact? Something like:

docker run ...something?
apk install git
git clone [email protected]:PowerShell/PowerShell
./PowerShell/tools/install-powershell.sh

or are there any dependencies involved? Please publish an additional docker layer for PS sitting on top of that previewed docker from .NET Core.

[JasonMorgan]

@kasper3 I can confirm the install script doesn't handle alpine yet. I'm looking at building from source myself now. I'll let you know if I get it.

[JasonMorgan]

So looks like you need the dotnet sdk, not just dotnet to get powershell built from source. I'll wait for dotnet 2.1 to go GA, should be January according to the release notes before trying to go through the powershell build steps again.

Also just to address your question from before, ./PowerShell/tools/install-powershell.sh seems to just use the OS based package manager and doesn't support alpine yet.

[markekraus]

yea, for Alpine you would need to get the tool chain added manually and then build manually. at least that is my understanding.

[mikeschinkel]

Wow, it would be hugely helpful if Microsoft could offer a supported release of PowerShell on Alpine given that Alpine is the first-class choice for Docker containers.

[markekraus]

considering .NET Core was just made to work in Alpine and right before 6.0.0 GA, it's not surprising there is no alpine build yet... I might play with it in my spare time to see if I can get a build going. I've been meaning to learn more about the build internals anyway. This would be a good opportunity with that. no promises though. I have a ton on my plate already :)

[AceHack]

I would love an Alpine image

[JasonMorgan]

Just a heads up, still waiting on an instance of the 2.1 coreclr sdk on alpine. I checked the nightly builds repo and they still only list out the runtime.

[AceHack]

@JasonMorgan do you include the sdk when installing powershell? If so in which directory?

[JasonMorgan]

@AceHack unfortunately I'm still stuck at the, I can't install powershell on alpine yet, part. I tried building it from source with the coreclr runtime container but it needed the sdk to work. They haven't released the SDK on alpine yet so I'm still waiting. I think Ideally we'll do a multistage build so we can distribute as small a powershell image as possible.

If I'm being honest my goal here is just to make an apk package that has pwsh in it. I'll probably never use a straight alpine image provided by this team. Not that I have anything against them I just typically build a alpine base image for wherever I'm working.

[AceHack]

@JasonMorgan Please forgive my ignorance but is it possible to build an apk runtime package for alpine using a different OS as the SDK image? Thanks.

[JasonMorgan]

@AceHack I don't know, I'm only planning on trying with alpine. Unfortunately I don't understand enough about the various linux OS' to understand why some binaries work on one but not another.

[ghost]

@JasonMorgan, the difference between Alpine and "most of" other linux distribution is that Alpine uses musl-libc instead of glibc to avoid the codebloat. In headless containers scenarios, the lesser the OS footprints, the better.

Size of Linux Kernel ~2.7 MB
Size of Alpine Linux (base OS) ~5.5 MB
Size of TinyCore Linux (base OS) ~11.0 MB
Size of Ubuntu Linux (base OS) ~70 MB

TinyCore and Ubuntu both use glibc, Alpine is musl-libc.

@AceHack, all the abuild tooling is providing by the Alpine, for the Alpine, packaging on other OS is "possible" yes (lots of additional unnecessary porting of abuild/aports tooling code..) but I doubt it would be "feasible" or even worth the effort. In 8 lines of Dockerfile, you can fetch, build, test, pack and upload your binary to artifact hosting server, why bother building apk on other OS? BTW, to get PowerShell from aports, .NET Core first needs to have the official package publish to: https://github.com/alpinelinux/aports.

First lets build it on dev machine..

[massimonewsuk]

Is there any update on this issue?

[ghost]

@SteveL-MSFT, I read the building docs https://github.com/PowerShell/PowerShell/blob/master/docs/building/linux.md#toolchain-setup and found that it requires PowerShell to build PowerShell.. Could we please have a unix shell build script which would build the whole product from scratch on new platforms without this dogfooding? Does PowerShell scratch build require some closed source components?

This is what I did:

First in host (Windows 10 in my case with docker community edition installed):

docker run -it microsoft/dotnet:2.1-sdk-alpine

# install dependencies
apk update
apk add git cmake

# clone
git clone https://github.com/PowerShell/PowerShell
cd PowerShell

# build
./tools/install-powershell.sh

Error:

/bin/sh: ./tools/install-powershell.sh: not found

In Alpine Linux, default shell is ash, not bash. In NetBSD, default shell is tsh/csh not bash. It would be awesome if install-powershell.sh is written with /bin/sh (Bourne Shell / POSIX standard way) so we don't have to do this:

apk add bash

It's a bit more work to adhere to keep shell script POSIX compliant and neutral (with #!/bin/sh shebang), but it has advantages.

Anyway, then running the ./tools/install-powershell.sh, I got:

Get-PowerShell Core MASTER Installer Version 1.1.1
Installs PowerShell Core and Optional The Development Environment
  Original script is at: https://raw.githubusercontent.com/PowerShell/PowerShell/master/tools$gitscriptname
Arguments used:

Operating System Details:
  OS: linux
  DIST:
  DistroBasedOn:
  PSUEDONAME:
  REV:
  KERNEL: 4.9.60-linuxkit-aufs
  MACH: x86_64
Sorry, your operating system is based on  and is not supported by PowerShell Core or this installer at this time.

(something is missing between on and and from the message based on and)

[SteveL-MSFT]

Looks like dotnetcore currently only has an Alpine runtime and not an SDK available

[massimonewsuk]

@SteveL-MSFT Is this what you were referring to?
https://hub.docker.com/r/microsoft/dotnet/tags/ - search for 2.1-sdk-alpine
or
https://hub.docker.com/r/microsoft/dotnet-nightly/tags/ - search for 2.1-sdk-alpine

[ghost]

@SteveL-MSFT, from the example above:

docker run -it microsoft/dotnet:2.1-sdk-alpine

this is dotnet runtime with SDK. After hitting enter, when you get into the container:

dotnet new console -n test
cd test
dotnet run

# 'Hello World!' is printed

[SteveL-MSFT]

Thanks for the pointers, I might try to take a look at this on the weekend

[ghost]

@SteveL-MSFT, thanks! If there is a bootstrap-everything-powershell-from-souce-for-unknown-platform script in open, could you point it out, I maybe able to contribute here. I am trying to build CoreCLR on SmartOS (Solaris derivative from Joyent (node.js folks)), currently blocked on clang6 port in their package manager pkgsrc that is 4 weeks out, and once everything is bootstrapped on .NET Core side (that may even take ages), we will still be in same kind of situation on PowerShell side. Similar story is for FreeBSD and NetBSD. Would be cool to make those scripts more permissive rather than whitelisting the platforms (most of the platforms have similar kinds of APIs and they just work, all we need is a build recipe).

On dotnet side, things are moving towards building everything from source (even offline) direction: https://github.com/dotnet/source-build/blob/a1e063e/Documentation/boostrap-new-os.md

[SteveL-MSFT]

@kasper3 this project has evolved and created a dependency on PSCore6 itself to build the entire project. I believe you really need to just build two things on any new platform that .Net Core supports:

• libpsl-native is cmake based and you just need to perform the equivalent of these steps in build.psm1
• powershell itself you should be able to initially get away with dotnet restore, dotnet publish if the dotnet SDK is available

[ghost]

@SteveL-MSFT, thanks. I will give it a try. BTW, do you have plans to remove functionality from libpsl-native gradually and start depending on CoreFX when available? or will it stay there forever for backward compatibility? I think there are some symlink related proposals in CoreFX that are likely to be addressed soon.

[ghost]

Native part compiled (will send a PR), for managed dotnet restore was fine but got few errors with dotnet publish

  Restore completed in 5.16 ms for /powershell/src/Microsoft.WSMan.Management/Microsoft.WSMan.Management.csproj.
  Restore completed in 3.47 ms for /powershell/src/System.Management.Automation/System.Management.Automation.csproj.
  Restore completed in 8.86 ms for /powershell/src/powershell-win-core/powershell-win-core.csproj.
DotNetCode/Eventing/EventDescriptor.cs(41,61): error CS0103: The name 'DotNetEventingStrings' does not exist in the current context [/powershell/src/Microsoft.PowerShell.CoreCLR.Eventing/Microsoft.PowerShell.CoreCLR.Eventing.csproj]
DotNetCode/Eventing/EventDescriptor.cs(46,103): error CS0103: The name 'DotNetEventingStrings' does not exist in the current context [/powershell/src/Microsoft.PowerShell.CoreCLR.Eventing/Microsoft.PowerShell.CoreCLR.Eventing.csproj]
DotNetCode/Eventing/EventDescriptor.cs(58,63): error CS0103: The name 'DotNetEventingStrings' does not exist in the current context [/powershell/src/Microsoft.PowerShell.CoreCLR.Eventing/Microsoft.PowerShell.CoreCLR.Eventing.csproj]
DotNetCode/Eventing/EventDescriptor.cs(63,105): error CS0103: The name 'DotNetEventingStrings' does not exist in the current context [/powershell/src/Microsoft.PowerShell.CoreCLR.Eventing/Microsoft.PowerShell.CoreCLR.Eventing.csproj]
DotNetCode/Eventing/EventProviderTraceListener.cs(45,49): error CS0103: The name 'DotNetEventingStrings' does not exist in the current context [/powershell/src/Microsoft.PowerShell.CoreCLR.Eventing/Microsoft.PowerShell.CoreCLR.Eventing.csproj]
DotNetCode/Eventing/EventProviderTraceListener.cs(76,45): error CS0103: The name 'DotNetEventingStrings' does not exist in the current context [/powershell/src/Microsoft.PowerShell.CoreCLR.Eventing/Microsoft.PowerShell.CoreCLR.Eventing.csproj]
DotNetCode/Eventing/EventProvider.cs(627,75): error CS0103: The name 'DotNetEventingStrings' does not exist in the current context [/powershell/src/Microsoft.PowerShell.CoreCLR.Eventing/Microsoft.PowerShell.CoreCLR.Eventing.csproj]
DotNetCode/Eventing/EventProvider.cs(664,83): error CS0103: The name 'DotNetEventingStrings' does not exist in the current context [/powershell/src/Microsoft.PowerShell.CoreCLR.Eventing/Microsoft.PowerShell.CoreCLR.Eventing.csproj]

[ghost]

6367

[SteveL-MSFT]

@kasper3 removing functions from libpsl-native if CoreFx supports it makes sense, but it wouldn't be a high priority for my team. Need to ensure that there isn't a perf penalty moving away from the native code.

[massimonewsuk]

Hey guys. I wanted to ask if this is ready to use yet. We basically have a dockerfile that has this at the top:

FROM node:alpine

And I want to know if I can get powershell inside that docker image?

[SteveL-MSFT]

@massimonewsuk unfortunately this won't happen for Preview2, I'm planning on working on this next week

[ghost]

current status:

• there is a build script and docker that compiles powershell master branch on alpine
• libpsl-native.so and all the CoreCLR/CoreFX/CoreSetup/ASP.NET native binaries are linked with muscle libc
• powershell has dependency on other native libraries, that are linked with glibc, which makes the product not usable unless you install glibc family on alpine and defeat the primary purpose of using alpine.. for that we need some help on https://github.com/Microsoft/omi/issues/497 (making their unix script portable)

if you have time and expertise, please chime in and contribute to OMI, so we have a pure product without glibc-ism on alpine linux.

[mi-hol]

@SteveL-MSFT any news on this?

[SteveL-MSFT]

Was hoping to have an Alpine release along with Preview.4. Next week we'll be working on producing a docker image release.

[massimonewsuk]

@SteveL-MSFT Hey is there any update on this?

[SteveL-MSFT]

@TravisEz13 is looking at this, however, it's currently a lower priority than preparing for the GA release. Although we have updated the tooling to build PowerShell Core on Alpine, it wasn't complete as it didn't do a module restore and thus the resulting Alpine package didn't have any modules even though the engine/console worked. Travis has an idea on how to resolve this.

[TravisEz13]

I have a PR here https://github.com/PowerShell/PowerShell-Docker/pull/37 for the docker image.

There are two issues:

1. The version is 6.1.0-fixalpine . The code is the same as 6.1.0-rc.1 but there were some build issues, and this is what our build system does when you build from a private branch.
2. There are some rendering issues in the console (see image below.) You can avoid these issues by changing docker run -it to docker run -i, but you loose tab completion.

You can try my image: docker run -it travisez13/powershell:preview-alpine-3.8

Consider this highly experimental.

[SteveL-MSFT]

Open new issues for for specific problems

[TravisEz13]

fyi, the container is available via docker run -it mcr.microsoft.com/powershell:preview-alpine-3.8

[alexandair]

@TravisEz13 Container is available. but is practically unusable. You cannot type anything. :(

[TravisEz13]

@alexandair Yes, but we need something to bootstrap real development. It works for automation. I mentioned this issue here: https://github.com/PowerShell/PowerShell/issues/4605#issuecomment-419598695

We now need to update the build to use this image and to a PSModuleRestore.