Deno: panic on non-UTF8 file

[kesupile]

Hi, first off let me say: great job with Deno! Its a truly exciting development in the JS ecosystem...

My Deno script works just fine when I run deno run src/cli.ts. However if I bundle the project and try to run the output .js file, I get the following error:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 0, error_len: Some(1) }', cli\file_fetcher.rs:622:41
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This happens when I run it locally, or refer to the remote url.

Repo
https://github.com/kesupile/ts-percentage

Operating System
Windows 10
using the powershell install

[bartlomieju]

Are there any non-UTF-8 characters in the file?

[kesupile]

Nope

[KaranGauswami]

I have checked that file , Its encoded in UTF-16 Little-endian Unicode text.

[Spoonbender]

I think the first thing to do is fix the panic, so we'll get a graceful error handling behavior.

Next, consider how we'd like to address the issue, assuming Deno is supposed to support non-UTF8 files.
The core difficulty is that there is no guaranteed way of knowing which encoding a file uses.
Some non-mutually-exclusive approaches:

1. Let the caller specify the encoding of the file
2. If the file is fetched over HTTP, and the Content-Type header specifies the encoding, then use the encoding specified
3. Use a file encoding guessing functionality which will work in most cases (which is the approach taken by browsers, as well as apps such as Notepad++, Beyond Compare and various IDEs)

[Spoonbender]

FYI, I tested this again on master and it no longer panics - now we just get a error: invalid utf-8 sequence of 1 bytes from index 0.
It does panic on 1.0.0 as @kesupile specified.
So, I guess it only boils down to:

1. adding support for non-utf8 files
2. adding a test to ensure non-regression (that we don't panic in a future version)

[Spoonbender]

I'm starting to work on this