Runtime: Build failure on WSL due to : PackagingException: File not found: '/home/dan/git/corefx/LICENSE.TXT'

[danmosemsft]

On WSL Ubuntu. And the file does exist:

dan@danmose3:~/git/corefx$ ls /home/dan/git/corefx/LICENSE.TXT
/home/dan/git/corefx/LICENSE.TXT

/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error : Error when creating nuget lib package from /home/dan/git/corefx/artifacts/packages/Debug/specs/Microsoft.NETCore.Targets.nuspec. NuGet.Packaging.Core.PackagingException: File not found: '/home/dan/git/corefx/LICENSE.TXT'. [/home/dan/git/corefx/pkg/Microsoft.NETCore.Targets/Microsoft.NETCore.Targets.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.AddFiles(String basePath, String source, String destination, String exclude) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Targets/Microsoft.NETCore.Targets.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.PopulateFiles(String basePath, IEnumerable`1 files) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Targets/Microsoft.NETCore.Targets.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at Microsoft.DotNet.Build.Tasks.Packaging.NuGetPack.Pack(String nuspecPath, String nupkgPath, Manifest manifest, Boolean packSymbols) in /_/src/Microsoft.DotNet.Build.Tasks.Packaging/src/NuGetPack.cs:line 256 [/home/dan/git/corefx/pkg/Microsoft.NETCore.Targets/Microsoft.NETCore.Targets.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error : Error when creating nuget packed package from /home/dan/git/corefx/artifacts/packages/Debug/specs/Microsoft.NETCore.Targets.nuspec. NuGet.Packaging.Core.PackagingException: File not found: '/home/dan/git/corefx/LICENSE.TXT'. [/home/dan/git/corefx/pkg/Microsoft.NETCore.Targets/Microsoft.NETCore.Targets.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.AddFiles(String basePath, String source, String destination, String exclude) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Targets/Microsoft.NETCore.Targets.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.PopulateFiles(String basePath, IEnumerable`1 files) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Targets/Microsoft.NETCore.Targets.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at Microsoft.DotNet.Build.Tasks.Packaging.NuGetPack.Pack(String nuspecPath, String nupkgPath, Manifest manifest, Boolean packSymbols) in /_/src/Microsoft.DotNet.Build.Tasks.Packaging/src/NuGetPack.cs:line 256 [/home/dan/git/corefx/pkg/Microsoft.NETCore.Targets/Microsoft.NETCore.Targets.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error : Error when creating nuget lib package from /home/dan/git/corefx/artifacts/packages/Debug/specs/Microsoft.NETCore.Platforms.nuspec. NuGet.Packaging.Core.PackagingException: File not found: '/home/dan/git/corefx/LICENSE.TXT'. [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms/Microsoft.NETCore.Platforms.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.AddFiles(String basePath, String source, String destination, String exclude) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms/Microsoft.NETCore.Platforms.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.PopulateFiles(String basePath, IEnumerable`1 files) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms/Microsoft.NETCore.Platforms.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at Microsoft.DotNet.Build.Tasks.Packaging.NuGetPack.Pack(String nuspecPath, String nupkgPath, Manifest manifest, Boolean packSymbols) in /_/src/Microsoft.DotNet.Build.Tasks.Packaging/src/NuGetPack.cs:line 256 [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms/Microsoft.NETCore.Platforms.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error : Error when creating nuget packed package from /home/dan/git/corefx/artifacts/packages/Debug/specs/Microsoft.NETCore.Platforms.nuspec. NuGet.Packaging.Core.PackagingException: File not found: '/home/dan/git/corefx/LICENSE.TXT'. [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms/Microsoft.NETCore.Platforms.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.AddFiles(String basePath, String source, String destination, String exclude) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms/Microsoft.NETCore.Platforms.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.PopulateFiles(String basePath, IEnumerable`1 files) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms/Microsoft.NETCore.Platforms.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at Microsoft.DotNet.Build.Tasks.Packaging.NuGetPack.Pack(String nuspecPath, String nupkgPath, Manifest manifest, Boolean packSymbols) in /_/src/Microsoft.DotNet.Build.Tasks.Packaging/src/NuGetPack.cs:line 256 [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms/Microsoft.NETCore.Platforms.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error : Error when creating nuget lib package from /home/dan/git/corefx/artifacts/packages/Debug/specs/Microsoft.NETCore.Platforms.Future.nuspec. NuGet.Packaging.Core.PackagingException: File not found: '/home/dan/git/corefx/LICENSE.TXT'. [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms.Future/Microsoft.NETCore.Platforms.Future.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.AddFiles(String basePath, String source, String destination, String exclude) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms.Future/Microsoft.NETCore.Platforms.Future.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.PopulateFiles(String basePath, IEnumerable`1 files) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms.Future/Microsoft.NETCore.Platforms.Future.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at Microsoft.DotNet.Build.Tasks.Packaging.NuGetPack.Pack(String nuspecPath, String nupkgPath, Manifest manifest, Boolean packSymbols) in /_/src/Microsoft.DotNet.Build.Tasks.Packaging/src/NuGetPack.cs:line 256 [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms.Future/Microsoft.NETCore.Platforms.Future.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error : Error when creating nuget packed package from /home/dan/git/corefx/artifacts/packages/Debug/specs/Microsoft.NETCore.Platforms.Future.nuspec. NuGet.Packaging.Core.PackagingException: File not found: '/home/dan/git/corefx/LICENSE.TXT'. [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms.Future/Microsoft.NETCore.Platforms.Future.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.AddFiles(String basePath, String source, String destination, String exclude) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms.Future/Microsoft.NETCore.Platforms.Future.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at NuGet.Packaging.PackageBuilder.PopulateFiles(String basePath, IEnumerable`1 files) [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms.Future/Microsoft.NETCore.Platforms.Future.pkgproj]
/home/dan/.nuget/packages/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19315.2/build/Packaging.targets(1258,5): error :    at Microsoft.DotNet.Build.Tasks.Packaging.NuGetPack.Pack(String nuspecPath, String nupkgPath, Manifest manifest, Boolean packSymbols) in /_/src/Microsoft.DotNet.Build.Tasks.Packaging/src/NuGetPack.cs:line 256 [/home/dan/git/corefx/pkg/Microsoft.NETCore.Platforms.Future/Microsoft.NETCore.Platforms.Future.pkgproj]
    0 Warning(s)
    6 Error(s)

I remember @MarcoRossignoli had this same issue. @tarekgh could this be related to how string comparisons are done - I seem to remember you mentioning some issue related to Nuget?

[tarekgh]

@danmosemsft This not look familiar to me. my guess would be casing of the file path is not matching the casing in the nuspec.

[stephentoub]

I've had this issue in the past when my repo was in an inconsistent state. Doing a clean and then rebuild fixed it for me.

[tarekgh]

I have tried that with a clean repo and I can repro the same issue. Will try to look more to understand what could be the problem.

[danmosemsft]

Interesting. If this isn't breaking the official builds, I wonder whether it needs a specific filesystem to repro, or distro.

[tarekgh]

I can build fine on native Ubuntu VMs. so this is scoped to WSL only I think. looking if I can find interesting thing there.

[ericstj]

Keeping this in 3.0 mainly because I'm concerned it could come back to an issue in IO or Globalization. Let's root cause and then decide what to do next. @joperezr I think you were building a private NuGet and debug with @tarekgh?

[tarekgh]

@joperezr Jose Perez Rodriguez FTE I think you were building a private NuGet and debug with @tarekgh Tarek Mahmoud Sayed FTE?

Yes, Jose built it and we'll continue debugging this later today.

[tarekgh]

Thanks @joperezr for the help you have provided compiling the needed assemblies with the embedded PDBs. That was very helpful.

The problem was, WSL is setting the default culture to "C" (which is translated to en-US_posix) and that culture doesn't support case insensitive comparisons. NuGet is using Regex for comparing the nuspec file list with the file system names with case insensitive option which fail to match under "C" culture.

To fix this, just reset the default culture to something like en-US.

export LANG=en_US.UTF-8

I am seriously considering changing the behavior of "C" in .NET core to map it to invariant (or even en-US) as the POSIX culture is really useless and not helping at all. I am not sure if I do such change will be ok for 3.0 or should we do it in 5.0.

@jkotas @stephentoub I like to hear your thoughts about what I am suggesting.

[tarekgh]

https://github.com/dotnet/corefx/issues/32250

[stephentoub]

I like to hear your thoughts about what I am suggesting.

Is there any situation in which the current behavior is useful / valuable / expected? If no, and assuming the change for this is as trivial as I'm expecting it to be, I suggest we do it for 3.0 Preview 7. If there's any situation in which it might be desirable, we should also consider a switch. This comes up so frequently as a pain point / bad first-run experience.

Also, will this be impacted at all by whether ICU is available? We carry our own invariant data, right?

[tarekgh]

Is there any situation in which the current behavior is useful / valuable / expected?

Only for scenario when users want to always compare strings as a binary bits. Maybe this is a security scenario but in general I am not seeing any other benefit of using this behavior. And still if anyone want this scenario we can ask them to explicitly use en-us_posix locale.

If all agree, I can try get this soon in P7.

[danmosemsft]

Hmm, on my 1903 machine at home, Ubuntu WSL starts up with LANG of en_US.UTF-8.

[jkotas]

Ubuntu WSL starts up with LANG of en_US.UTF-8.

Same for me.

[jkotas]

NuGet is using Regex for comparing the nuspec file list with the file system names with case insensitive option which fail to match under "C" culture.

This sounds like a bug in NuGet to me. You pretty much never want to use current culture for any kind of system level thing. This should be fixed in NuGet by using InvariantCulture.

[stephentoub]

Ubuntu WSL starts up with LANG of en_US.UTF-8.

For me with Version 10.0.18362 Build 18362 I get:

~$ printenv LANG
C.UTF-8

[danmosemsft]

@stephentoub's 18362 is 1903, where I'm getting en_US.UTF-8 so there is not a simple relationship.

My other 1809 machine incidentally has C.UTF-8.

[danmosemsft]

I have no idea which regex in NuGet we are looking at, but in NuGetClient repo there are several uses of RegexOptions.IgnoreCase that are clearly operating on file system paths and none of them also have RegexOptions.CultureInvariant. That's wrong, I guess?

[ericstj]

I have no idea which regex in NuGet we are looking at

Here: https://github.com/NuGet/NuGet.Client/blob/686dd6b060639a0fb401eceb4c3a4029c1482023/src/NuGet.Core/NuGet.Common/PathUtil/PathResolver.cs#L90

I've filed https://github.com/NuGet/Home/issues/8246

I suspect that this has something to do with the image for WSL that comes from the windows app store. Perhaps that's missing proper LANG setting. For folks that repro'ed: did you install on WSL from Windows App Store?

[MarcoRossignoli]

For folks that repro'ed: did you install on WSL from Windows App Store

Yes for me, in all my installations

[stephentoub]

for folks that repro'ed: did you install on WSL from Windows App Store?

Ubuntu came from the store. I didn't know there was an alternative?

[MarcoRossignoli]

There is a way to manually download https://docs.microsoft.com/en-us/windows/wsl/install-win10#install-your-linux-distribution-of-choice
BTW I don't know if it's the same as store one.

[tarekgh]

here is the response we got from the WSL team:

It is likely specific to the distribution.  We respect whatever is in /etc/default/locale.  In the very early days of WSL we used to attempt to do a locale mapping on the behalf of the user, but our users didn’t like that.

[danmosemsft]

dan@danmose3:~$ printenv LANG
C.UTF-8
dan@danmose3:~$ cat /env/default/locale
cat: /env/default/locale: No such file or directory

vs

dan@danmose00:~$ printenv LANG
en_US.UTF-8
dan@danmose00:~$ cat /etc/default/locale
#  File generated by update-locale

so what is "update-locale" ...

[danmosemsft]

update-locale is in /usr/sbin/update-locale and exists on both machines. If I run it on the "bad" machine it runs but does not seem to write locale though.

[tarekgh]

in my WSL, I m seeing

tarek@tarekms-dev:~$ cat /etc/default/locale
LANG=C.UTF-8

[danmosemsft]

That's interesting, you have a 3rd mode then.

Randomly trying relevant seeming commands:

dan@danmose3:~$ sudo locale-gen
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
dan@danmose3:~$ cat /env/default/locale
cat: /env/default/locale: No such file or directory
dan@danmose3:~$ sudo update-locale
dan@danmose3:~$ cat /env/default/locale
cat: /env/default/locale: No such file or directory

[tarekgh]

just wondering, did you install ICU in your WSL?

[danmosemsft]

Yes

dan@danmose3:~$ sudo apt-get install  libicu60
Reading package lists... Done
Building dependency tree
Reading state information... Done
libicu60 is already the newest version (60.2-3ubuntu3).
0 upgraded, 0 newly installed, 0 to remove and 250 not upgraded.

[tarekgh]

interesting, could you double check which Ubuntu version you are using? I am just trying to understand what is the difference.

tarek@tarekms-dev:/etc$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

[danmosemsft]

dan@danmose3:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic
dan@danmose3:~$ uname -a
Linux danmose3 4.4.0-17763-Microsoft dotnet/corefx#379-Microsoft Wed Mar 06 19:16:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

and for the "good" machine

dan@danmose00:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial
dan@danmose00:~$ uname -a
Linux danmose00 4.4.0-17763-Microsoft dotnet/corefx#379-Microsoft Wed Mar 06 19:16:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

The store has "Ubuntu" (which is what is on danmose00, and says it installs latest LTS) as well as "Ubuntu 16.04 LTS" and "Ubuntu 18.04 LTS". I'll try explicitly installing 18.04

[danmosemsft]

Installed 18.04 WSL (it's side by side) on the "good" machine and it has C.UTF-8 whereas 16.04 on this machine was en-US.UTF8.

dan@danmose00:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic
dan@danmose00:~$ uname -a
Linux danmose00 4.4.0-17763-Microsoft dotnet/corefx#379-Microsoft Wed Mar 06 19:16:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux
dan@danmose00:~$ printenv LANG
C.UTF-8

So it appears that the difference is the version of Ubuntu. And also that if you don't install an explicit version, just installing "Ubuntu" from the store can give either.

And as confirmation I put 1604 on the "bad" machine and it is en-us.UTF8 also.