Xamarin-android: VS 16.2.0 regression: CopyToAsync from DeflateStream to GzipStream throws NotImplementedException

[mfeingol]

Steps to Reproduce

1. Use the ZipArchive and ZipArchiveEntry classes to open a stream to an entry in a zip file
2. Set up a GZipStream compressing to a MemoryStream
3. Copy the zip entry stream to the GZipStream
4. Observe the NotImplementedException

Repro project: https://github.com/mfeingol/repros/tree/master/Xamarin.Android-3397-CopyToAsync

Expected Behavior

Stream copy is successful, as it was in prior versions of the Mono runtime.

Actual Behavior

NotImplementedException:

  at System.IO.Compression.DeflateStream.WriteAsyncMemory (System.ReadOnlyMemory`1[T] source, System.Threading.CancellationToken cancellationToken) [0x00000] in <ca7419b40e504a6dbe088f6fe95d09aa>:0 
  at (wrapper remoting-invoke-with-check) System.IO.Compression.DeflateStream.WriteAsyncMemory(System.ReadOnlyMemory`1<byte>,System.Threading.CancellationToken)
  at System.IO.Compression.GZipStream.WriteAsync (System.ReadOnlyMemory`1[T] buffer, System.Threading.CancellationToken cancellationToken) [0x00026] in <ca7419b40e504a6dbe088f6fe95d09aa>:0 
  at System.IO.Stream.CopyToAsyncInternal (System.IO.Stream destination, System.Int32 bufferSize, System.Threading.CancellationToken cancellationToken) [0x000c7] in <ff07eae8184a40a08e79049bbcb31a0e>:0 
  at CopyToAsync.MainPage.Repro (System.Object sender, System.EventArgs args) [0x0008a] in C:\Users\mfeingol\source\Repos\CopyToAsync\CopyToAsync\CopyToAsync\MainPage.xaml.cs:35 

Example code:

ZipArchive archive = new ZipArchive(ZipFile, ZipArchiveMode.Read, true, Encoding.UTF8);
ZipArchiveEntry entry = archive.GetEntry("a4.jpg");
Stream src = entry.Open();

MemoryStream dest = new MemoryStream();

using (GZipStream gzip = new GZipStream(dest, CompressionLevel.Optimal, true))
    await src.CopyToAsync(gzip);

byte[] compressed = dest.ToArray();

Version Information

VS 16.2.0, fresh project from new project wizard, latest Xamarin.Forms Nuget packages (not that it's relevant)

Log File

log.txt

[jonathanpeppers]

When I look at the code in Mono for this: https://github.com/mono/mono/blob/95f8a68c56e3312945b28b0523d2551d90c1797c/mcs/class/System/System.IO.Compression/DeflateStream.cs#L176-L179

It seems it has been this way for > 2 years, did this code work in 16.1 for sure?

[mfeingol]

I may have one machine that still has 16.1.x on it, so I may be able to test on that later.

However, the relevant stream copy is called from a pretty core scenario in my app, and that scenario was definitely working with previous versions of VS/Mono.

I should note that I temporarily worked around the issue by allocating a buffer and reading to/writing from that buffer to copy between streams, and that works fine. Synchronous CopyTo works fine as well. So presumably the issue is that the CopyToAsync path has changed to call that not-implemented method.

[jonathanpeppers]

@baulig do you know if something changed between mono 2018-08 and 2019-02?

I don't really see anything here to cause this: https://github.com/mono/mono/commit/5068c7cc13661f7f09d962596a33e82fdb276741

[mfeingol]

I just tried this on my tablet with 16.1.5 and I can confirm it works. No exception is thrown.

You're able to repro the problem on 16.2.0, right?

Aside: the exception does not repro if a memory stream is used as the source stream, or as the dest stream. It only occurs when DeflateStream copies into GZipStream.

[jonathanpeppers]

@mfeingol yes I can reproduce it, 😞 I just don't have an explanation.

I think we'll need someone on the Mono team to investigate.

[jwosty]

This appears not to be restricted to Xamarin Android. I'm using Suave and Xamarin.Mac, which presumably calls these async methods internally (I never do it myself directly). It broke as soon as I upgraded to the latest VSfM (version 8.2.2 build 21).

My (incomplete) stacktrace looks like this, but duplicated many times over:

[19:35:34 WRN] TCP request processing failed
System.AggregateException: One or more errors occurred. (The method or operation is not implemented.) ---> System.NotImplementedException: The method or operation is not implemented.
  at System.IO.Compression.DeflateStream.WriteAsyncMemory (System.ReadOnlyMemory`1[T] source, System.Threading.CancellationToken cancellationToken) [0x00000] in /Library/Frameworks/Xamarin.Mac.framework/Versions/5.14.0.114/src/Xamarin.Mac/mcs/class/System/System.IO.Compression/DeflateStream.cs:178 
  at (wrapper remoting-invoke-with-check) System.IO.Compression.DeflateStream.WriteAsyncMemory(System.ReadOnlyMemory`1<byte>,System.Threading.CancellationToken)
  at System.IO.Compression.GZipStream.WriteAsync (System.ReadOnlyMemory`1[T] buffer, System.Threading.CancellationToken cancellationToken) [0x00026] in /Library/Frameworks/Xamarin.Mac.framework/Versions/5.14.0.114/src/Xamarin.Mac/external/corefx/src/System.IO.Compression/src/System/IO/Compression/GZipStream.cs:195 
  at System.IO.Stream.CopyToAsyncInternal (System.IO.Stream destination, System.Int32 bufferSize, System.Threading.CancellationToken cancellationToken) [0x000b4] in /Library/Frameworks/Xamarin.Mac.framework/Versions/5.14.0.114/src/Xamarin.Mac/external/corert/src/System.Private.CoreLib/shared/System/IO/Stream.cs:152 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.NotImplementedException: The method or operation is not implemented.
  at System.IO.Compression.DeflateStream.WriteAsyncMemory (System.ReadOnlyMemory`1[T] source, System.Threading.CancellationToken cancellationToken) [0x00000] in /Library/Frameworks/Xamarin.Mac.framework/Versions/5.14.0.114/src/Xamarin.Mac/mcs/class/System/System.IO.Compression/DeflateStream.cs:178 
  at (wrapper remoting-invoke-with-check) System.IO.Compression.DeflateStream.WriteAsyncMemory(System.ReadOnlyMemory`1<byte>,System.Threading.CancellationToken)
  at System.IO.Compression.GZipStream.WriteAsync (System.ReadOnlyMemory`1[T] buffer, System.Threading.CancellationToken cancellationToken) [0x00026] in /Library/Frameworks/Xamarin.Mac.framework/Versions/5.14.0.114/src/Xamarin.Mac/external/corefx/src/System.IO.Compression/src/System/IO/Compression/GZipStream.cs:195 
  at System.IO.Stream.CopyToAsyncInternal (System.IO.Stream destination, System.Int32 bufferSize, System.Threading.CancellationToken cancellationToken) [0x000b4] in /Library/Frameworks/Xamarin.Mac.framework/Versions/5.14.0.114/src/Xamarin.Mac/external/corert/src/System.Private.CoreLib/shared/System/IO/Stream.cs:152 <---

[mfeingol]

@jonathanpeppers: should we file a GHI on the Mono team?

[steveisok]

So, there is a slight difference that changes the code path to call DeflateStream.WriteAsyncMemory.

In 5.18 (2018-08), Mono was using the referencesource version of Stream, which opened its own buffer and called this WriteAsync method in GZipStream:

https://github.com/mono/corefx/blob/c976905c1de06786292a6fb01656b43650f262c9/src/System.IO.Compression/src/System/IO/Compression/GZipStream.cs#L177-L181

In 6.0 (2019-02), we changed changed to the corefx one via this commit:

https://github.com/mono/mono/commit/46a28303e219bf3ede81bc5989b0add250212860

And now, that code rents a shared buffer and calls this method in GZipStream:

https://github.com/mono/corefx/blob/c976905c1de06786292a6fb01656b43650f262c9/src/System.IO.Compression/src/System/IO/Compression/GZipStream.cs#L183-L197

This is a regression. To fix, I'm not sure if there's anything stopping us from taking the corefx version, but I'm sure we can at the very least take the WriteAsyncMemory piece.

[czuvich]

I've been bitten by this bug as well. Are there workarounds besides not upgrading? We use GZip for our backend web calls.

[steveisok]

@czuvich I haven't tried it, but I think if you derive your own GZipStream, it'll call the old WriteAsync. That's because typeof should return your class as opposed to GZipStream.

Note:
https://github.com/mono/corefx/blob/c976905c1de06786292a6fb01656b43650f262c9/src/System.IO.Compression/src/System/IO/Compression/GZipStream.cs#L183-L197

[czuvich]

@steveisok That did the trick!

// https://github.com/xamarin/xamarin-android/issues/3397#issuecomment-521224235
internal class HackGZipStream : GZipStream {
        public HackGZipStream(Stream stream, CompressionMode mode) : base(stream, mode) {

        }

        public HackGZipStream(Stream stream, CompressionMode mode, bool leaveOpen) : base(stream, mode, leaveOpen) {

        }

        public HackGZipStream(Stream stream, CompressionLevel level) : base(stream, level) {

        }

        public HackGZipStream(Stream stream, CompressionLevel level, bool leaveOpen) : base(stream, level, leaveOpen) { 

        }
    }

[baulig]

Maybe we should wrap that check in some #if MONO conditional as a quick work-around?

We could also just implement those new ReadOnlySpan<T> based API.

[AbeniMatteo]

WriteAsyncMemory was "implemented" here directly in mono.

I'm having almost the same problem but with WriteCore (using VS 16.3.0 preview 2.0).

System.NotImplementedException: The method or operation is not implemented.
  at System.IO.Compression.DeflateStream.WriteCore (System.ReadOnlySpan`1[T] source) [0x00000] in <379743d8665e4975a273a646d2b450ba>:0 
  at (wrapper remoting-invoke-with-check) System.IO.Compression.DeflateStream.WriteCore(System.ReadOnlySpan`1<byte>)
  at System.IO.Compression.GZipStream.Write (System.ReadOnlySpan`1[T] buffer) [0x00025] in <379743d8665e4975a273a646d2b450ba>:0 
  ...

@steveisok will WriteCore also be implemented?

[steveisok]

Ah, thanks for pointing that out. Sorry :-). I'll try to get a PR up sometime this week.

[brendanzagaeski]

A new Preview version has now been published that includes a fix for this item. The fix is not yet included in a Release version. I will update this item again when a Release version is available that includes the fix.

Fix included in Xamarin.Android 10.0.0.43.

Fix included on Windows in Visual Studio 2019 version 16.3 Preview 4. To try the Preview version that includes the fix, check for the latest updates in Visual Studio Preview.

Fix included on macOS in Visual Studio 2019 for Mac version 8.3 Preview 5. To try the Preview version that includes the fix, check for the latest updates on the Preview updater channel.

[AbeniMatteo]

Just checked, WriteCore still throws NotImplementedException with VS 16.3 Preview 4.
Check here for details.

[brendanzagaeski]

Ah! Thanks for checking on WriteCore(). WriteCore() was not yet updated in the commit https://github.com/mono/mono/commit/fd75f0f487999d88052ba356203b897ce532f659 with the fix for this item.

I'll submit a new issue to track the WriteCore() behavior and comment back here with the link.

[brendanzagaeski]

I have submitted the new issue for WriteCore() in https://github.com/mono/mono/issues/16950.

[brendanzagaeski]

A new Release version has now been published that includes the fix for this item (not yet including the follow-up fix for WriteCore()).

Fix included in Xamarin.Android 10.0.0.43

Fix included on Windows in Visual Studio 2019 version 16.3. To get the new version that includes the fix, check for the latest updates or install the latest version from https://visualstudio.microsoft.com/downloads/.

Fix included on macOS in Visual Studio 2019 for Mac version 8.3. To get the new version that includes the fix, check for the latest updates on the Stable updater channel.

I did a quick local check with these versions to make sure I could still reproduce the original problem before the update and could no longer reproduce the problem after the update.

I will accordingly close this item. Thanks again for reporting it!

I will update https://github.com/mono/mono/issues/16950 for WriteCore() with similar Xamarin.Android release information in the future when it becomes available.