Nugetgallery: NuGet.Server error "File contains corrupted data" on PUT package on remote server only

[Ubloobok]

Hi!

NuGet.Server (method PackageService.CreatePackage) throws error "File contains corrupted data" on PUT package on remote server only, in IIS hosted application (created by instruction http://nugetserver.net/), but locally all works correctly. It reproducing for any package.
Server: Windows Server 2008 R2, .NET 4.5, IIS 7.5, configured application pool with .NET Framework v4.0.30319, managed pipeline mode Integrated.
NuGet.Server v2.11.1.0, nuget.exe version 3.4.3.855.
Exception:

System.IO.FileFormatException: File contains corrupted data.
at MS.Internal.IO.Zip.ZipIOEndOfCentralDirectoryBlock.FindPosition(Stream archiveStream)
at MS.Internal.IO.Zip.ZipIOBlockManager.LoadEndOfCentralDirectoryBlock()
at MS.Internal.IO.Zip.ZipArchive.OpenOnStream(Stream stream, FileMode mode, FileAccess access, Boolean streaming)
at System.IO.Packaging.ZipPackage..ctor(Stream s, FileMode mode, FileAccess access, Boolean streaming)
at System.IO.Packaging.Package.Open(Stream stream, FileMode packageMode, FileAccess packageAccess, Boolean streaming)
at System.IO.Packaging.Package.Open(Stream stream)
at NuGet.OptimizedZipPackage.EnsureManifest()
at NuGet.Server.Publishing.PackageService.CreatePackage(HttpContextBase context)
at NuGetServer.NuGetRoutes.PushPackage(RequestContext context)

Research shown that problem can be with content type "multipart/form-data", but we cannot recognize - problem in server, or in RouteMagic+NuGet.Server, or in both.
On local machine/devIIS in debug HttpContext.Request.Files has 1 file with package, therefore CreatePackage works as expected.
On server HttpContext.Request.Files are empty, but InputStream contains all neccessary headers + data.
Requests headers/data on local and server - identical:

PUT /api/package/ HTTP/1.1
Connection: Keep-Alive
Content-Length: 128432
Content-Type: multipart/form-data; boundary="ec6ee113-c37c-4fdc-8f02-2ab7fd2de007"
Accept-Encoding: gzip, deflate
Expect: 100-continue
Host: test.com
User-Agent: NuGet Command Line/3.4.3.855 (Microsoft Windows NT 6.1.7601 Service Pack 1)
X-NuGet-ApiKey: test

--ec6ee113-c37c-4fdc-8f02-2ab7fd2de007
Content-Type: application/octet-stream
Content-Disposition: form-data; name=package; filename=package.nupkg; filename*=utf-8''package.nupkg

PK    „yєHqFу§ ... ZIP package... [Content_Types].xmlPK
     „yєH–Э­ѓб
  н  Q             =п
 package/services/metadata/core-properties/e4bb51182ca14486b1a306374401d6fb.psmdcpPK    
 )  Ќс
   
--ec6ee113-c37c-4fdc-8f02-2ab7fd2de007--

Were suspected case described here http://forums.iis.net/t/1229356.aspx?Windows+Update+breaks+multipart+form+data, but update KB3104002 are not installed on the server.
We will be very grateful if you help to understand because of what it is a problem. Perhaps the reason only properly configured server, but we already no have any ideas.
Thank you in advance!

[maartenba]

Can you try capturing a Fiddler trace f the push operation to your production server? (scroll to bottom of https://docs.nuget.org/consume/nuget-faq#managing-packages-in-nuget.org)

Feel free to send it over to support at NuGet org (mention my GitHub username :))

[Ubloobok]

@maartenba , ok, I did send it to support email, thanks!

[maartenba]

@Ubloobok can you resend? We haven't received a Fiddler trace so far.

[Ubloobok]

@maartenba , I resend email "[NuGet.Server] Error "File contains corrupted data" on PUT package on remote server only" to [email protected] and your email with ZIP-archive.

[raphaelSch]

The same here with the following configuration:

Server: 2012 R2
IIS: 8.5
NuGet Server: 2.11.1.0
NuGet.exe: 3.4.3.855

[maartenba]

@raphaelSch Out of curiosity, could you try using a newer NuGet client? https://dist.nuget.org/win-x86-commandline/v3.5.0-beta/NuGet.exe

Also, could you capture a Fiddler trace (https://docs.nuget.org/consume/nuget-faq#managing-packages-in-nuget.org, make sure to File > Save > All sessions as a .saz file) and send it over to support at nuget org?

[stpatrick2016]

Had same issue with latest client. Also tried the beta 3.5 client - same result. Downgrading to client 2.7.41115.310 solved the problem though.
Inspecting the zip packages on the server - there is extra contend added to each package, which breaks it. The extra content shown by Ubloobok in his first post here, it is:

Content-Type: application/octet-stream
Content-Disposition: form-data; name=package; filename=package.nupkg; filename*=utf-8''package.nupkg

Older version doesn't add it.

[maartenba]

Thanks! Any chance you can capture that Fiddler trace?

[maartenba]

@stpatrick2016 @raphaelSch @Ubloobok Do you have any insights into the .NET version on your server? Also, can you replace the system.web and system.webserver elements in Web.config with these:

  <system.web>
    <httpRuntime maxRequestLength="31457280" />
    <compilation debug="true" />
  </system.web>
  <system.serviceModel>
    <serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
  </system.serviceModel>
  <system.webServer>
    <staticContent>
      <mimeMap fileExtension=".nupkg" mimeType="application/zip" />
    </staticContent>
    <modules runAllManagedModulesForAllRequests="true">
      <remove name="WebDAVModule" />
    </modules>
    <handlers>
      <remove name="WebDAV" />

      <remove name="ExtensionlessUrlHandler-Integrated-4.0" />
      <add name="ExtensionlessUrlHandler-Integrated-4.0" path="*." verb="GET,HEAD,POST,PUT,DEBUG" type="System.Web.Handlers.TransferRequestHandler" preCondition="integratedMode,runtimeVersionv4.0" />
    </handlers>
    <security>
      <requestFiltering>
        <requestLimits maxAllowedContentLength="31457280" />
      </requestFiltering>
    </security>
  </system.webServer>

We suspect the WebDAV module from interfering with the content, this config update may fix the issue. If not, we'll have to build a fallback for extracting the multipart content on our end.

[stpatrick2016]

.NET is 4.6.1.
I have uploaded the fiddler logs and sent through support, maybe they will help.

Will try tomorrow or today later to alter the web.config and report results. One thing I already have is the to fix the issue when push returned 405 (Method not allowed).

[stpatrick2016]

Tried replacing the section - got the same result.

[maartenba]

@stpatrick2016 @raphaelSch @Ubloobok Any chance Windows Update with KB3104002 is on your server? Could you try this hotfix? https://support.microsoft.com/en-us/kb/3125446

[stpatrick2016]

Checked - that KB is not installed. A bit problematic for me to install the update though - the production server has no internet connection and the update is not on our wsus server either. And I couldn't find offline installation package for it :(

[maartenba]

Can you capture a failed request trace as well? http://www.iis.net/learn/troubleshoot/using-failed-request-tracing/troubleshooting-failed-requests-using-tracing-in-iis

[stpatrick2016]

Sure, sent you via that support thread.

[maartenba]

Thanks all for your help!

Seems NuGet 3.3 and higher touches a server-side feature that is only available in .NET 4.6. Please upgrade your server .NET framework to use .NET 4.6 or higher.

We will be fixing this in the NuGet.Server package as well - https://github.com/NuGet/NuGetGallery/issues/3063