Powershell: Failure to build from source on Arch Linux

[chungy]

Steps to reproduce

1. Use Arch Linux x86_64
2. Install gtest, and dotnet-cli from AUR
3. Try to build
4. "dotnet build --configuration Linux" results in:

Project Microsoft.PowerShell.CoreCLR.AssemblyLoadContext (.NETStandard,Version=v1.6) will be compiled because expected outputs are missing
Compiling Microsoft.PowerShell.CoreCLR.AssemblyLoadContext for .NETStandard,Version=v1.6
/opt/dotnet/dotnet compile-csc @/tmp/powershell/src/PowerShell-6.0.0-alpha.9/src/Microsoft.PowerShell.CoreCLR.AssemblyLoadContext/obj/Linux/netstandard1.6/dotnet-compile.rsp returned Exit Code 1
/tmp/powershell/src/PowerShell-6.0.0-alpha.9/src/Microsoft.PowerShell.CoreCLR.AssemblyLoadContext/CoreCLR/CorePsAssemblyLoadContext.cs(112,35): error CS0103: The name 'InitializeTypeCatalog' does not exist in the current context

Compilation failed.
    0 Warning(s)
    1 Error(s)

Time elapsed 00:00:03.4195229

The PKGBUILD I've constructed as a WIP is here: https://github.com/chungy/powershell.aur

Expected behavior

To build PowerShell successfully

Actual behavior

Failed to build

Environment data

> $PSVersionTable

N/A

[andschwa]

I'm sorry, our manual build steps are a bit out of date. We've recently switched to generating both the type catalog (Start-TypeGen) and resources (Start-ResGen) dynamically. The build.psm1 script is pretty much just a large set of convenience wrappers, to "do the right thing" for PowerShell developers working on supported platforms. When bootstrapping a new platform, a bit more work is involved (believe me, I know!).

Both the type catalog and resource generation should be pretty straight forward invocations of dotnet, the exact location and arguments encoded in the functions I referenced above (I'd link for you but am on mobile).

[willprice]

Here's the code from build.psm1:

function Start-TypeGen
{
    [CmdletBinding()]
    param()

    # Add .NET CLI tools to PATH
    Find-Dotnet

    Push-Location "$PSScriptRoot/src/TypeCatalogParser"
    try {
        dotnet run
    } finally {
        Pop-Location
    }

    Push-Location "$PSScriptRoot/src/TypeCatalogGen"
    try {
        dotnet run ../Microsoft.PowerShell.CoreCLR.AssemblyLoadContext/CorePsTypeCatalog.cs powershell.inc
    } finally {
        Pop-Location
    }
}

function Start-ResGen
{
    [CmdletBinding()]
    param()

    # Add .NET CLI tools to PATH
    Find-Dotnet

    Push-Location "$PSScriptRoot/src/ResGen"
    try {
        Start-NativeExecution { dotnet run } | Write-Verbose
    } finally {
        Pop-Location
    }
}

[qrwteyrutiyoup]

Last night I managed to build it on Arch, after taking a look at build.psm1. Here's a PKGBUILD: https://gist.github.com/qrwteyrutiyoup/1d06831e59728669632a55a33a690987

It become something like this, then:

pushd src/libpsl-native
cmake .
make
popd

dotnet restore

pushd src/ResGen
dotnet run -c Linux
popd

pushd src/TypeCatalogParser
dotnet run -c Linux
popd

pushd src/TypeCatalogGen
dotnet run -c Linux  ../Microsoft.PowerShell.CoreCLR.AssemblyLoadContext/CorePsTypeCatalog.cs powershell.inc
popd

cd src/powershell-unix
dotnet build -c Linux

However, it does seem to have some hardcoded checks for the runtime, which I was able to work around by using an Ubuntu-like _os-release_ file and _mount --bind_ it to _/etc/os-release_. Not pretty, but it did work.

I also needed icu55 from the AUR.

[andschwa]

Awesome work! And yes, the .NET CLI tools check the runtime too. I think it _might_ be possible to use a generic "linux" runtime, but haven't tried.

@brthor do you know?

[jaseg]

@qrwteyrutiyoup As a slightly less hacky alternative workaround to the os-release bind mount you could try using fakeroot to replace /etc/os-release.

[qrwteyrutiyoup]

@jaseg thanks, I will try looking into it. In the meantime, I have used proot, as someone in the AUR suggested.

PKGBUILD here https://github.com/qrwteyrutiyoup/powershell-git.aur

[nlowe]

c695d41 broke the proot workaround (according to a git bisect):

c695d41c47c8baa48db1a590fe7378641a9e0ab9 is the first bad commit
commit c695d41c47c8baa48db1a590fe7378641a9e0ab9
Author: Andrew Schwartzmeyer <[email protected]>
Date:   Thu Sep 1 16:24:11 2016 -0700

    Use lowercase name for package folders (#2162)

    .NET CLI build 3546 started placing packages in:

        ~/.nuget/packages/microsoft.codeanalysis.common

    instead of:

        ~/.nuget/packages/Microsoft.CodeAnalysis.Common

    Where the previous (capitalized) version corresponded directly to the
    package's name. Now they are all lower-cased.

:040000 040000 dbafd345af60c831d25551d9769a2e6b01481fe1 bfc7b206ad7c45a39ec25ac1d90862efa46e3a4e M  src

Even with proot, it's still throwing the error:

Unhandled Exception: System.IO.InvalidDataException: Reference assembly '/home/nathan/.nuget/packages/microsoft.codeanalysis.common/1.1.1/lib/portable-net45+win8/Microsoft.CodeAnalysis.dll' is declared in the reference list file '/home/nathan/.cache/pacaur/powershell-git/src/powershell/src/TypeCatalogGen/powershell.inc', but the assembly doesn't exist.
at Microsoft.PowerShell.CoreCLR.TypeCatalogGen.ResolveReferenceAssemblies(String path)
at Microsoft.PowerShell.CoreCLR.TypeCatalogGen.Main(String[] args)
~/.cache/pacaur/powershell-git/src/powershell
Telemetry is: Enabled
Project Microsoft.PowerShell.CoreCLR.AssemblyLoadContext (.NETStandard,Version=v1.6) will be compiled because expected outputs are missing
/home/nathan/.cache/pacaur/powershell-git/src/powershell/src/Microsoft.PowerShell.CoreCLR.AssemblyLoadContext/bin/Linux/netstandard1.6/Microsoft.PowerShell.CoreCLR.AssemblyLoadContext.dll
/home/nathan/.cache/pacaur/powershell-git/src/powershell/src/Microsoft.PowerShell.CoreCLR.AssemblyLoadContext/bin/Linux/netstandard1.6/Microsoft.PowerShell.CoreCLR.AssemblyLoadContext.pdb
Compiling Microsoft.PowerShell.CoreCLR.AssemblyLoadContext for .NETStandard,Version=v1.6
Running /opt/dotnet/dotnet /opt/dotnet/sdk/1.0.0-preview2-003121/csc.dll -noconfig @/home/nathan/.cache/pacaur/powershell-git/src/powershell/src/Microsoft.PowerShell.CoreCLR.AssemblyLoadContext/obj/Linux/netstandard1.6/dotnet-compile-csc.rsp
Process ID: 9389
/opt/dotnet/dotnet compile-csc @/home/nathan/.cache/pacaur/powershell-git/src/powershell/src/Microsoft.PowerShell.CoreCLR.AssemblyLoadContext/obj/Linux/netstandard1.6/dotnet-compile.rsp returned Exit Code 1
/home/nathan/.cache/pacaur/powershell-git/src/powershell/src/Microsoft.PowerShell.CoreCLR.AssemblyLoadContext/CoreCLR/CorePsAssemblyLoadContext.cs(112,35): error CS0103: The name 'InitializeTypeCatalog' does not exist in the current context

Compilation failed.
0 Warning(s)
1 Error(s)

Time elapsed 00:00:00.9760053

==> ERROR: A failure occurred in build().
Aborting...
:: failed to build powershell-git package(s)

I have a feeling that this is a case-sensitive filesystem issue, since if you don't run a git clean -fxd after checking out a working commit, master builds just fine.

Should this be its own issue?

[andschwa]

You need to update .NET CLI. It's latest versions changed the package layout.

[andschwa]

Proot might throw a kink in things since it sounds like it's grabbing unofficial packages and so has not yet picked up .NET's changes.

[nlowe]

@andschwa You mean v1.0.0-preview2.0.1 released 4 days ago? Or do I need a recent commit from master?

[andschwa]

The preview3 installed by Start-PSBootstrap; build 3546 or newer per my referenced commit above.

[andschwa]

https://github.com/dotnet/cli/issues/4141

[nlowe]

Huh, that's unfortunate. Thanks, I'll take a look. Unfortunately I think this means the AUR build is broken unless we get a dotnet-cli-git package made.

[andschwa]

Yeah, @nlowe this is entirely dependent not only on .NET Core supporting Arch Linux better, but also the build tool, .NET CLI. My guess is that it will continue to sometimes work; but until there's official support from both our language and build tool, there's not much we can do but point you in the right direction :/

Don't get me wrong, I _love_ Arch and wish I could develop routinely on it. It's my favorite distribution. For what it's worth, you can use Docker containers to accomplish the same thing; in fact, I have a script way, way back in the history that I wrote to ease the pain. IIRC it lets you dynamically build and run your local repo in the context of the container, preserving your host user permissions too.