Runtime: System.IO.Compression: ZipArchive does not respect CP437 encoding when reading files

[qmfrederik]

The ZipArchive class uses ASCII encoding instead of IBM 437 (CP 437) (since dotnet/corefx#9004) encoding when reading and processing file names of zip entries which do not have the language encoding flag (bit 11 in the general purpose bit flag) set.

The zip file specification appendix D specifies how encoding should be handled:

The ZIP format has historically supported only the original IBM PC character
encoding set, commonly referred to as IBM Code Page 437. This limits storing
file name characters to only those within the original MS-DOS range of values
and does not properly support file names in other character encodings, or
languages. To address this limitation, this specification will support the
following change.

If general purpose bit 11 is unset, the file name and comment should conform
to the original ZIP character encoding. If general purpose bit 11 is set, the
filename and comment must support The Unicode Standard, Version 4.1.0 or
greater using the character encoding form defined by the UTF-8 storage
specification.

This creates a problem mainly for European languages: they tend to use a lot of the characters which are present in CP437 but not in ASCII, so you may get inconsistent behavior there.

Windows 10 still defaults to CP437 where possible: you can try this by creating a file called Über.txt and zipping it using File Explorer. The filename will be encoded using CP437; but the ZipArchive class decodes it as �ber.txt.

Because the zip archive re-creates the individual zip entries when the file is saved in Update mode, this results in data loss when opening & closing the archive, even if that entry is not modified.

This appears to be a side-effect of removing CP437 support from .NET Core, although it is still available in the System.Text.Encoding.Codepages package (see dotnet/runtime#17849 for more information).

A simple fix would be to have the System.IO.Compression package take a dependency on System.Text.Encoding.Codepages, but not sure if this is desirable.

/cc @ianhays

[karelz]

@ianhays says that this is reasonable solution: Have System.IO.Compression package take a dependency on System.Text.Encoding.Codepages.
Anyone wants to grab it?

[clarkis117]

@karelz I'm thinking about grabbing this, though not sure how project references are supposed to work with both a csproj and project.json. In addition, I did manage to replicate the described behavior by adding an additional zip archive containing a text file, Über.txt, to the existing unit tests.

[karelz]

Cool. @ianhays will be able to provide further guidance (I don't know the answer myself, sorry).

I sent you repo Collaborator invite. Once you accept, we can assign it to you. Assigning to @ianhays for now.

[clarkis117]

@karelz invitation accepted

[karelz]

Assignment updated ...

[ianhays]

@clarkis117 you'll want to add the reference to CodePages in the project.json under the "dependencies" section.

[karelz]

@clarkis117 just curious if you were able to make any progress ... or if you need any help

[clarkis117]

@karelz Yeah, I seem to be running into issues where the APIs for System.Text.Encoding.CodePages are not available on netstandard1.7. I will post the project.json files I'm using and example code that reproduces the issue, in a more detailed write up soon.

[clarkis117]

Example code:

```c#

if netstandard17

Encoding readEntryNameEncoding = System.Text.CodePagesEncodingProvider.Instance.GetEncoding(437);

endif

project.json for ref System.IO.Compression

```json
{
  "dependencies": {
    "System.Runtime": "4.4.0-beta-24715-03",
    "System.IO": "4.4.0-beta-24715-03",
    "System.Text.Encoding": "4.4.0-beta-24715-03",
    "System.Text.Encoding.CodePages": "4.4.0-beta-24715-03",
    "System.Threading.Tasks": "4.4.0-beta-24715-03"
  },
  "frameworks": {
    "netstandard1.7": {}
  }
}

project.json for src System.IO.Compression

{
  "frameworks": {
    "netstandard1.6": {
      "dependencies": {
        "Microsoft.NETCore.Platforms": "1.0.1",
        "System.Buffers": "4.0.0",
        "System.Collections": "4.0.11",
        "System.Diagnostics.Contracts": "4.0.1",
        "System.Diagnostics.Debug": "4.0.11",
        "System.Diagnostics.Tools": "4.0.1",
        "System.IO": "4.1.0",
        "System.Resources.ResourceManager": "4.0.1",
        "System.Runtime": "4.1.0",
        "System.Runtime.Extensions": "4.1.0",
        "System.Runtime.Handles": "4.0.1",
        "System.Runtime.InteropServices": "4.1.0",
        "System.Text.Encoding": "4.0.11",
        "System.Text.Encoding.CodePages": "4.0.1",
        "System.Threading": "4.0.11",
        "System.Threading.Tasks": "4.0.11"
      }
    },
    "netstandard1.7": {
      "dependencies": {
        "Microsoft.NETCore.Platforms": "1.2.0-beta-24715-03",
        "System.Buffers": "4.0.0",
        "System.Collections": "4.4.0-beta-24715-03",
        "System.Diagnostics.Contracts": "4.4.0-beta-24715-03",
        "System.Diagnostics.Debug": "4.4.0-beta-24715-03",
        "System.Diagnostics.Tools": "4.4.0-beta-24715-03",
        "System.IO": "4.4.0-beta-24715-03",
        "System.Resources.ResourceManager": "4.4.0-beta-24715-03",
        "System.Runtime": "4.4.0-beta-24715-03",
        "System.Text.Encoding": "4.4.0-beta-24715-03",
        "System.Text.Encoding.CodePages": "4.4.0-beta-24715-03",
        "System.Runtime.Extensions": "4.4.0-beta-24715-03",
        "System.Runtime.Handles": "4.4.0-beta-24715-03",
        "System.Runtime.InteropServices": "4.4.0-beta-24715-03",
        "System.Threading": "4.4.0-beta-24715-03",
        "System.Threading.Tasks": "4.4.0-beta-24715-03"
      }
    },
    "netstandard1.3": {
      "dependencies": {
        "Microsoft.NETCore.Platforms": "1.0.1",
        "System.Buffers": "4.0.0",
        "System.Collections": "4.0.11",
        "System.Diagnostics.Contracts": "4.0.1",
        "System.Diagnostics.Debug": "4.0.11",
        "System.Diagnostics.Tools": "4.0.1",
        "System.IO": "4.1.0",
        "System.Resources.ResourceManager": "4.0.1",
        "System.Runtime": "4.1.0",
        "System.Runtime.Extensions": "4.1.0",
        "System.Runtime.Handles": "4.0.1",
        "System.Runtime.InteropServices": "4.1.0",
        "System.Text.Encoding": "4.0.11",
            "System.Text.Encoding.CodePages": "4.0.1",
        "System.Threading": "4.0.11",
        "System.Threading.Tasks": "4.0.11"
      }
    },
    "net46": {
      "dependencies": {
        "Microsoft.TargetingPack.NETFramework.v4.6": "1.0.1"
      }
    },
    "net463": {
      "dependencies": {
        "Microsoft.TargetingPack.NETFramework.v4.6": "1.0.1"
      }
    }
  }
}

Related build warnings and errors:

C:\Users\clark\Source\Repos\corefx\src\System.IO.Compression\src\System.IO.Compression.nuget.targets(4,5): warning : Pa
ckages containing MSBuild targets and props files cannot be fully installed in projects targeting multiple frameworks.
The MSBuild targets and props files have been ignored. [C:\Users\clark\Source\Repos\corefx\src\System.IO.Compression\sr
c\System.IO.Compression.csproj]
C:\Users\clark\Source\Repos\corefx\src\System.IO.Compression\src\System.IO.Compression.nuget.targets(4,5): warning : Pa
ckages containing MSBuild targets and props files cannot be fully installed in projects targeting multiple frameworks.
The MSBuild targets and props files have been ignored. [C:\Users\clark\Source\Repos\corefx\src\System.IO.Compression\sr
c\System.IO.Compression.csproj]
System\IO\Compression\ZipArchiveEntry.cs(456,44): error CS0234: The type or namespace name 'CodePagesEncodingProvider'
does not exist in the namespace 'System.Text' (are you missing an assembly reference?) [C:\Users\clark\Source\Repos\cor
efx\src\System.IO.Compression\src\System.IO.Compression.csproj]
Command execution failed with exit code 1.

[ianhays]

how are you calling msbuild? You shouldn't be passing a TargetGroup.

If you post your branch I can take a look and see if anything stands out to me.

[clarkis117]

@ianhays Currently I'm using the scripts in the root directory to build corefx:

sync
build

branch: https://github.com/clarkis117/corefx

[jnm2]

On net462 I'm getting šber.txt instead of Über.txt. I've actually run up against this issue in production. Please port this back to full framework.

[karelz]

@jnm2
We can consider it when we will be porting changes over to full Framework (no ETA at this moment), if the fix is is low risk, reasonable value and there is no reasonable workaround.
If you need faster turnaround, I would recommend to use Microsoft Support.

[jnm2]

Thanks @karelz.

[clarkis117]

@ianhays Is there preprocessor symbol for distinguishing between coreclr and full framework? I’m trying to distinguish between the CodePages package and the CP437 support in full framework.

[ianhays]

Is there preprocessor symbol for distinguishing between coreclr and full framework? I’m trying to distinguish between the CodePages package and the CP437 support in full framework.

You could use FEATURE_ZLIB or add your own to be more specific. If the code ends up being especially complex then I would suggest splitting the file, make the class partial, and conditionally including based on architecture e.g. ZipArchive.net46 and ZipArchive.Windows.

[karelz]

@clarkis117 any update on your progress?

[karelz]

Triage: No activity for a long time, not enough upvotes, closing.