Home: Executables restored from NuGet packages are no longer executable with .NET Core "2.0"

[eerhardt]

Details about Problem

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe):

Erics-MBP:cli eerhardt$ ./.dotnet_stage0/x64/dotnet --info
.NET Command Line Tools (2.0.0-alpha-004622)

Product Information:
 Version:            2.0.0-alpha-004622
 Commit SHA-1 hash:  1f44cabf4a

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.11-x64

Worked before? If so, with which NuGet version: Yes, with dotnet restore version 1.0.0-rc3-004530.

Detailed repro steps so we can see the same problem

1. Install a "2.0" CLI from https://github.com/dotnet/cli/tree/master#installers-and-binaries on a non-Windows machine.

2. Restore a package that contains a tool/executable. For example, Microsoft.NETCore.Runtime.CoreCLR version 1.2.0-beta-24904-03.

3. Try to execute the crossgen executable at .nuget/packages/runtime.osx.10.10-x64.microsoft.netcore.runtime.coreclr/1.2.0-beta-24904-03/tools/crossgen

Expected result:

With "1.0", the executable would run.

Actual result:

15:57:11.104     1>/Users/dotnet-bot/j/workspace/dotnet_cli/master/release_osx_x64_prtest/build/Microsoft.DotNet.Cli.Compile.targets(225,7): error MSB6003: The specified task executable "crossgen" could not be run. Operation not permitted [/Users/dotnet-bot/j/workspace/dotnet_cli/master/release_osx_x64_prtest/build.proj]

Notes

The regression is caused by System.IO.Compression in corefx. See https://github.com/dotnet/corefx/issues/15516. In .NET Core 2.0, the Zip APIs are no longer setting the "x" mod on files unzipped.

This is going to be an issue for all tooling using NuGet to restore executables on non-Windows platforms. We shouldn't expect every place that is executing programs from NuGet packages to have to ensure the file has an "x" permission on it.

[rrelyea]

Any reason this is a shipstopper for rtm?

[eerhardt]

Not a shipstopper. This only affects "2.0". "1.0" works fine.

[eerhardt]

@rrelyea - when VS 2017 is out the door, can we get this issue triaged? This is going to affect a lot of "2.0" scenarios, so I'd like to get it fixed early in "2.0".

[eerhardt]

For example, this is breaking the dotnet cache feature on non-Windows machines in 2.0 because the crossgen executable is not marked with the "x" permission.

@gkhanna79 @ramarag

[eerhardt]

Shell scripts need to be marked executable as well.

/cc @livarcocc

[eerhardt]

@rrelyea - as we are getting more people using .NET Core 2.0, this is becoming more of an issue. See https://github.com/dotnet/cli/issues/6286. Can we get this looked into?

[eerhardt]

/cc @livarcocc

[Eilon]

@rrelyea - this issue blocks ASP.NET Core (and presumably other) apps from running after running the publish operation. Do we have an idea when we might see a fix for this?

[muratg]

:eyes:

[rrelyea]

From thread with NetCore team...i think they are addressing this.
However, Eric said in that thread:
I kind of think this will have to get resolved in NuGet regardless. The reasoning being that we already have a bunch of .nupkgs that were built with 1.0 that don’t have permission information inside of them.

[eerhardt]

From thread with NetCore team...i think they are addressing this.

The NetCore team is not addressing this. It is an intentional breaking change from netcoreapp1.x to netcoreapp2.0 in the System.IO.Compression area.

[tmds]

In 1.x, files created by .NET Core had the x-bit set. In 2.x, files created by .NET Core no longer have the x-bit set. This change is not specific to System.IO.Compression.

A change made in ZipArchive is making the permission bits accessible (https://github.com/dotnet/corefx/pull/18565) but no-one is using those yet.

Packages created so far, don't have the x-bit set for their files. Extracting packages, resulted in the x-bit being set for all files on 1.x and for no files on 2.x. This means executable files will no longer be executable when extracting with 2.x.

When support is added for maintaining permissions when packing and unpacking, there is still an issue for existing packages that don't have the x-bit set and relied on the 1.x behavior to set it.

Most packages contain only dlls and so files, they seem to work fine without the x-bit. Packages containing executables&scripts are affected.

[tmds]

@emgarten @eerhardt is this in the 2.0.0-preview1? I am still missing the x-bit...

[eerhardt]

No it isn't in 2.0.0-preview1. And I just verified this morning that this fix isn't in the latest 2.0.0-preview2 builds either. So it looks like there is no version of the .NET Core SDK that has this fix in it yet.

@emgarten @rohit21agrawal - any idea when a new version of NuGet will make its way into dotnet/cli?

[rohit21agrawal]

Hoping to make an insertion before the end of this week.

[kichalla]

Has this been fixed, I am using the following version of CLI and I am still able to reproduce it.

dotnet --info:

Repro steps:

• mkdir testconsole
• cd testconsole
• Add a NuGet.config with the following content:
  xml <?xml version="1.0" encoding="utf-8"?> <configuration> <packageSources> <clear /> <add key="DotNetCore" value="https://dotnet.myget.org/F/dotnet-core/api/v3/index.json" /> <add key="NuGet" value="https://api.nuget.org/v3/index.json" /> </packageSources> </configuration>
• dotnet new console
• Open the testconsole.csproj file and add the property <RuntimeIdentifier>linux-x64</RuntimeIdentifier> and save the file
• dotnet restore
• dotnet publish -r linux-x64
• cd to the published directory
• Run ./testconsole

Expected:
Hello World!

Actual:
bash: ./testconsole: Permission denied

[kichalla]

cc @muratg

[muratg]

@emgarten @rrelyea @anangaur @eerhardt

Guys, was this fixed in Preview2? Please see above @kichalla's report that this is still repro'ing.

[tmds]

@muratg I don't think the fix is included.
The commit https://github.com/NuGet/NuGet.Client/commit/91023394890b1458ec0c5940128da73e04089869 is in the 4.3.0-preview3-4123 and 4.3.0-preview2-4087 tags.
While the daily cli build (2.0.0-preview2-006127) uses 4.3.0-preview1-* NuGet packages.

[muratg]

@livarcocc are you planning to ingest this version of the nuget?

[livarcocc]

We are already at 4.3.0-preview3-4146.

[tmds]

@livarcocc this is why I think the package is still at preview1

$ wget https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-dev-linux-x64.latest.tar.gz
$ tar xvf dotnet-dev-linux-x64.latest.tar.gz
$ strings sdk/2.0.0-preview2-006127/NuGet.*.dll | grep 4.3 | grep preview
4.3.0-preview1-2500
4.3.0-preview1-2500
4.3.0-preview1-2500
4.3.0-preview1-2500
4.3.0-preview1-2500
4.3.0-preview1-2500
4.3.0-preview1-3937
4.3.0-preview1-2500
4.3.0-preview1-2500
4.3.0-preview1-3937
4.3.0-preview1-2500
4.3.0-preview1-2500
4.3.0-preview1-2500
4.3.0-preview1-3937

[kichalla]

Yeah, I am still able to repro this issue with the latest Preview2 CLI : 2.0.0-preview2-006391

[tmds]

@kichalla @livarcocc can you check the version in the tar.gz is the one you expect it to be?

[kichalla]

This is the url from where I downloaded the latest version of CLI:
https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.0-preview2-006391/dotnet-sdk-2.0.0-preview2-006391-linux-x64.tar.gz

and I am seeing that it does have NuGet version that @livarcocc had mentioned earlier:

[tmds]

Ah, ok. The 2.1.0-preview1 daily builds are using the older NuGet package. And the 2.0.0-preview2 is then using the newer version.
For my information, where did you find that url?

[kichalla]

We (ASP.NET) maintain a copy of the scripts from this location and use it in our build process to install the latest CLI and shared runtime. Very recently these scripts were updated:

[tmds]

Using the 2.0.0-preview2 link of @kichalla, the execute bit is set!
@kichalla did you clear the nuget cache before publishing? The bit gets set when the package is extracted. So if you still had the package in the cache, it will not change.

[kichalla]

@tmds Hmm, did you test with the repro scenario I mentioned here? I did clear the nuget cache doing dotnet nuget locals all --clear and followed the repro steps in the link above, but I still see the execute permission not set. Just want to make sure if we both are testing the same scenario.

[tmds]

@kichalla the quest continues. The x-bit is not set on the application. However, it is set in the package folder upon extracting!

$ ls -lah ~/.nuget/packages/runtime.linux-x64.microsoft.netcore.dotnetapphost/2.0.0-preview2-25407-01/runtimes/linux-x64/native/apphost 
-rwxrw-rw-. 1 tmds tmds 96K Jun  7 13:04 /home/tmds/.nuget/packages/runtime.linux-x64.microsoft.netcore.dotnetapphost/2.0.0-preview2-25407-01/runtimes/linux-x64/native/apphost

[tmds]

@kichalla If I followed the breadcrumbs correctly, the execute bit gets lost here: https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/EmbedAppNameInHost.cs#L62.

[gkhanna79]

Is this onky happening for standalone scenario' s entryloint executable only?

Adding @eerhardt to chime in as well.

Thanks,
Gaurav

-------- Original message --------
From: Tom Deseyn [e-mail]
Date: 6/9/17 12:08 PM (GMT-08:00)
To: NuGet/Home [e-mail]
Cc: "Gaurav Khanna (CLR)" [e-mail], Mention [e-mail]
Subject: Re: [NuGet/Home] Executables restored from NuGet packages are no longer executable with .NET Core "2.0" (#4424)

@kichallahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkichalla&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C557ead296d2946b63ac608d4af6add56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326320893010305&sdata=6zPycgrC5U2emf%2FAL5b4eqTF5lOzygnOl%2FlWRPO9Q0A%3D&reserved=0 If I followed the breadcrumbs correctly, the execute bit gets lost here: https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/EmbedAppNameInHost.cs#L62https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fsdk%2Fblob%2Fmaster%2Fsrc%2FTasks%2FMicrosoft.NET.Build.Tasks%2FEmbedAppNameInHost.cs%23L62&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C557ead296d2946b63ac608d4af6add56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326320893010305&sdata=%2BMJG0kUCrQYrOluYFWABVfbL2XnT5bqVYcNVW9OPtkM%3D&reserved=0.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNuGet%2FHome%2Fissues%2F4424%23issuecomment-307474563&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C557ead296d2946b63ac608d4af6add56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326320893020313&sdata=blMj6MjR39iXzB4N88Dei98JZqCKFFCOxX55Hypqs7c%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJA3wWStReHWkMAmPN5Sr1sBhlt25Yvpks5sCZgVgaJpZM4LwlX3&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C557ead296d2946b63ac608d4af6add56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326320893020313&sdata=8izC6ZlJm090nxhYa9y6drs7N8%2BHR7xrExudSmZKwqc%3D&reserved=0.

[gkhanna79]

If the possible cause identified by @tmds is correct, then this issue has been present for standalone publish in Preview1 as well and fix would be to set the X bit once the modified binary is published.

@steveharter can you PTAL?

[gkhanna79]

@kichalla Can you please file an issue in SDK for your repro above and assign it to @eerhardt and @steveharter?

[gkhanna79]

@steveharter Possbile fix needs to be like https://github.com/dotnet/sdk/blob/ddfd75cacfd5fb94bebc442441c75852ca4b1ee2/test/Microsoft.NET.TestFramework/CommandExtensions.cs#L16.

[tmds]

or alternatively, the apphost can be placed in the publish folder using File.Copy and renamed using File.Move.

[gkhanna79]

Apphost needs to be modified as part of publish operation so simply copying is not going to work.

[tmds]

You can open the copied file

[gkhanna79]

@livarcocc When CLI published a standalone app, does it ever mark its entrypoint executable with the "X" bit on non-Windows?

[steveharter]

Taking a look now

[steveharter]

My understanding:

• Only the modified apphost executable needs the +x; other executables are now properly restored from nuget.
• This only occurs (the x is lost) during publish of a standalone app, when copying the source apphost to the destination\modified apphost

I am still going through the repro steps; clearing the nuget cache and then restore is being very slow...

[gkhanna79]

Yes, that sounds about right @steveharter.

Thanks,
Gaurav

-------- Original message --------
From: Steve Harter [e-mail]
Date: 6/9/17 2:37 PM (GMT-08:00)
To: NuGet/Home [e-mail]
Cc: "Gaurav Khanna (CLR)" [e-mail], Mention [e-mail]
Subject: Re: [NuGet/Home] Executables restored from NuGet packages are no longer executable with .NET Core "2.0" (#4424)

My understanding:

• Only the modified apphost executable needs the +x; other executables are now properly restored from nuget.
• This only occurs (the x is lost) during publish of a standalone app, when copying the source apphost to the destination\modified apphost

I am still going through the repro steps; clearing the nuget cache and then restore is being very slow...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNuGet%2FHome%2Fissues%2F4424%23issuecomment-307505749&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C52edb22c45164e2ebadb08d4af7fba44%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326410504448082&sdata=5ChvKqVNNmIjiHrPkR5WGWPGxUAWMIpz8ANuqr%2FVi5Q%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJA3wQelNjndMwnP3VtbuWha2Yw2VAVwks5sCbsKgaJpZM4LwlX3&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C52edb22c45164e2ebadb08d4af7fba44%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326410504448082&sdata=ABN5C0qZ3hWegSovCaRerK2bvkSti1pFETezgrqT0lo%3D&reserved=0.

[livarcocc]

If the file is signed in the NuGet package, why is it losing that permission? It seems like these permissions should not be lost when copying/moving the file. And explicitly setting it in the SDK seems like a workaround for an issue somewhere else.

[kichalla]

@kichalla Can you please file an issue in SDK for your repro above and assign it to @eerhardt and @steveharter?

https://github.com/dotnet/sdk/issues/1331

[tmds]

If the file is signed in the NuGet package, why is it losing that permission? It seems like these permissions should not be lost when copying/moving the file.

@livarcocc This is the code which is 'copying' the app host: https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/EmbedAppNameInHost.cs#L62. It is creating a new file from bytes in memory.