Extensions: Packaging error: Invalid nuget package with 2 identical files created

[Flohack74]

Describe the bug

See https://github.com/fsprojects/Paket/issues/3743 - Currently one or many projects that were pushed in the last 24hrs seem to have lliegal content in the ZIP file, creating issues with compression libs. Specifically, the packaged icon is duplicated. See screenshot

To Reproduce

1. Just download nupkg file and inspect visually
2. See error

Screenshots

Additional context

While the paket guys implemented a fix for that, this should be investigated thoroughly, as it seems to hit multiple packaging jobs not connected to the same namespace.

[slang25]

I've confirmed by checking out tag v3.1.0 and running:

./build.sh
unzip -l ./artifacts/packages/Debug/Shipping/Microsoft.Extensions.DependencyInjection.Abstractions.3.1.0.zip

you'll see 2 x packageIcon.png.

[forki]

There are multiple packages in the 3.1 release cycle witu this issue. It seems to be a systematic issue somewhere in packaging.

[rrelyea]

From internal email thread:
This doesn’t seem to reproduce the problem of duplicate embedded icons:

Md paket-2019-12-break
Cd paket-2019-12-break
Dotnet new classlib
Create packageicon.jpg and packageicon2.jpg

Modify project file to be:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <PackageIcon>packageicon.jpg</PackageIcon>
  </PropertyGroup>
  <ItemGroup>
    <None Include="packageicon.jpg" Pack="true" PackagePath="\" />
    <None Include="packageicon2.jpg" Pack="true" PackagePath="\packageicon.jpg" />
  </ItemGroup>
</Project>

Dotnet pack

actual behavior: only one packageicon.jpg entry (as viewed via 7-zip)
UPdate: looks like we raise a warning during pack:
C:\Program Files\dotnetsdk\3.1.100\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): warning NU5118: File 'c:\temp\paket-2019-12-break\packageicon2.jpg' is not added because the package already contains file '\packageicon.jpg' [c:\temp\paket-2019-12-break\paket-2019-12-break.csproj]

would love to understand if we (nuget client) have an issue to address here.

[dougbu]

• @aspnet/build

@rrelyea we got into this situation in two steps

1. Update all ASP.NET Core repos to manually include embedded icons with the correct NuGet metadata
2. Upgrade the Arcade SDK to a version which centralized all the logic, adding overlapping metadata and icon files

@wtgodbe didn't you fix all of our repos up to use _only_ the Arcade SDK logic? Did merges forward into our 3.1 branches just miss the boat on the GA release?

[wtgodbe]

@dougbu I did fix this, but it didn't get merged forward for 3.1.0. The fixes are in 3.1.1 and master now, though.

[dougbu]

@rrelyea @Pilchie is 3.1.1 soon enough from your perspectives?

For one thing, how many customers do we have using these packages with 3rd party clients?

[rrelyea]

Given that 3.1.0 package already shipped. We need to understand if having a 3.1.1 package with the fix will help fix the issue for most people.
That depends on the scenario of how people decide to download 3.1.0.

If they pick it because it is hardcoded into a template, then the fix won't be too helpful.
If most people pick it because an update to their current 3.0 package via NuGet Update tab in VS, then fixing it now would help people avoid it.

We also should understand which scenarios are hurt by it. Just Paket? Other?

Paket? all Versions? How often do paket users upgrade to latest, etc... - @forki and team can likely help us understand impact of their user base and importance of fixed packages - now or in about a month?

[forki]

From Paket perspective: workaround is applied and a new Paket version is available. Usually this means the problem is gone next day because our user base is mostly running on the latest Paket. So if the problem is identified by MS then I don't think 3.1.1 packages are really needed to be pushed immediately. At least not from our perspective. But I think nuget team should add an additional check for duplicates in package verification. This would prevent similar issues in the future

[rrelyea]

By the way, if somebody is using a nuspec, it is possible to have duplicate items in a nupkg: https://github.com/NuGet/Home/issues/6941

[Fido789]

I would prefer 3.1.1 versions of affected packages ASAP as this issue block us from pushing them to private NuGet server. The server is returning status 406 (not accepted).

[forki]

@Fido789 which private NuGet server implementation do you use?

[Fido789]

@forki I have tried it with https://www.nuget.org/packages/NuGet.Server/3.1.2 and https://www.nuget.org/packages/NuGet.Server/3.4.1

[forki]

Wow. So private installation of nuget is rejecting the packages but the public server did not?

[rrelyea]

NuGetGallery and NuGet.Server are two different codebases. Not surprising that their behavior differs a bit here. @joelverhagen

[ajcvickers]

@Pilchie Should this issue be triaged into a milestone and assigned to someone so that people can track the status?

[forki]

If nuget's own private server implementation is rejecting the packages and Paket did, then chances are other 3rd-party servers and tools will do as well.

[Pilchie]

I believe this is already fixed in 3.1.1. @wtgodbe can you confirm?

[wtgodbe]

@Pilchie that's correct, this is fixed for 3.1.1.

[forki]

@Pilchie are those packages already available? Will the broken ones be unlisted? And then there is the problem of nuget's behavior with unlisted packages. If it's a transitive dependency then nuget will continue to install 3.1.0 because of it's minimum matching version resolver. Even if that package is unlisted.
So you'd would need to roll out the full wave of 3.1.1. Even with packages that are not broken but depend on a broken package.

[Pilchie]

No, the packages are not available yet, we're still working through issues with preparing 3.1.1 all-up. I don't expect they will be released before the new year unfortunately.

[forki]

But why can't you push a wave of packages that just fix this specific issue? As more and more people are moving to 3.1 this will be getting more serious.

[Pilchie]

That's just not the way our infrastructure is set up unfortunately.

[willsmith9182]

Any news on this or an eta for release?

I'm allergic to build warnings when restoring packages

[dougbu]

The next patch release (3.1.1) includes a fix for this issue. We're working to get that release out soon.

[daconglee]

Verbose: File system changed. File: microsoft.extensions.configuration.3.1.0.nupkg - Change: Changed
Info: Start adding packages from drop folder.
Error: An exception occurred while handling a file system event: System.ArgumentException: Item has already been added. Key in dictionary: 'packageIcon.png' Key being added: 'packageIcon.png'
at System.Collections.Hashtable.Insert(Object key, Object nvalue, Boolean add)