Mbed-cli: utf8 error by importing mercurial repository

[toyowata]

When importing repository in the os.mbed.com (Mercurial), I got following problem in my PC (Japanese Windows 7).

PS C:\Users\toyom> mbed --version
1.7.3
PS C:\Users\toyom> mbed import https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-blinky/
[mbed] Importing program "mbed-os-example-blinky" from "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-blinky" at latest revision in the current branch
[mbed] ERROR: Unknown Error: 'utf8' codec can't decode byte 0x91 in position 0: invalid start byte
---

There is no problem importing GitHub repositiry.

PS C:\Users\toyom> mbed import https://github.com/ARMmbed/mbed-os-example-blinky
[mbed] Importing program "mbed-os-example-blinky" from "https://github.com/ARMmbed/mbed-os-example-blinky" at latest revision in the current branch
[mbed] Adding library "mbed-os" from "https://github.com/ARMmbed/mbed-os" at rev #03196b244ee0
PS C:\Users\toyom>

Also, mbed-cli v1.6.0 did not have this problem.

PS C:\Users\toyom> mbed --version
1.6.0
PS C:\Users\toyom> mbed import https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-blinky
[mbed] Importing program "mbed-os-example-blinky" from "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-blinky" at latest revision in the current branch
[mbed] Adding library "mbed-os" from "https://github.com/ARMmbed/mbed-os" at rev #03196b244ee0
PS C:\Users\toyom>

[screamerbg]

@toyowata Thanks for the bug report. Which version of python do you use?

Cc @cmonr
[Mirrored to Jira]

[toyowata]

The version of python is v2.7.15.
[Mirrored to Jira]

[screamerbg]

@toyowata Can you rerun the command with -vv and paste the log here?
[Mirrored to Jira]

[cmonr]

@toyowata Would you also mind running and sharing the outputs of python --version and pip list --verbose?
[Mirrored to Jira]

[toyowata]

@screamerbg @cmonr

Interestingly, after I put -vv option for the Mbed-CLI, I could import the repository successfully. I can now import any mercurial repos without any problem. This is strange.

Imported log file is here:
mbed_import.txt

[Python version]

PS C:\test> python --version
Python 2.7.15

[pip list]

PS C:\test> pip list --verbose
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
appdirs (1.4.3)
astroid (1.6.3)
backports.functools-lru-cache (1.5)
beautifulsoup4 (4.6.0)
certifi (2018.4.16)
chardet (3.0.4)
colorama (0.3.9)
configparser (3.5.0)
enum34 (1.1.6)
fasteners (0.14.1)
funcsigs (1.0.2)
functools32 (3.2.3.post2)
future (0.16.0)
futures (3.2.0)
fuzzywuzzy (0.16.0)
idna (2.6)
intelhex (2.2.1)
intervaltree (2.1.0)
isort (4.3.4)
Jinja2 (2.10)
jsonschema (2.6.0)
junit-xml (1.8)
lazy-object-proxy (1.3.1)
lockfile (0.12.2)
MarkupSafe (1.0)
mbed-cli (1.7.3)
mbed-greentea (1.4.0)
mbed-host-tests (1.3.1)
mbed-ls (1.4.6)
mccabe (0.6.1)
mock (2.0.0)
monotonic (1.5)
pbr (4.0.4)
pip (9.0.3)
prettytable (0.7.2)
project-generator (0.9.11)
project-generator-definitions (0.2.35)
pyelftools (0.24)
pylint (1.8.4)
pyOCD (0.10.0)
pyserial (3.4)
pyusb (1.0.2)
pywinusb (0.4.2)
PyYAML (3.12)
requests (2.18.4)
setuptools (39.0.1)
singledispatch (3.4.0.3)
six (1.11.0)
sortedcontainers (2.0.4)
urllib3 (1.22)
websocket-client (0.48.0)
wrapt (1.10.11)
xmltodict (0.11.0)
You are using pip version 9.0.3, however version 10.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

[Mirrored to Jira]

[screamerbg]

@toyowata Can I ask you to modify your local Mbed CLI installation and change if very_verbose: to if very_verbose or 1: (around line 3023) and try again?
[Mirrored to Jira]

[toyowata]

@screamerbg

I used latest version of the Mbed-CLI in the Github master branch and I added or 1 for the if statement around line 3052:

    except Exception as e:
        if very_verbose or 1:
            traceback.print_exc(file=sys.stdout)
        error("Unknown Error: %s" % e, 255)
    sys.exit(status or 0)

I can import the mbed-os-blinky example here without any problem:

PS C:\Users\toyom> mbed -vv import https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-blinky/
[mbed] Importing program "mbed-os-example-blinky" from "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-blinky" at latest revision in the current branch
[mbed] Adding library "mbed-os" from "https://github.com/ARMmbed/mbed-os" at rev #50bd61a4a723

However, I got import error for the repository below:

PS C:\Users\toyom> mbed -vv import https://os.mbed.com/users/MACRUM/code/mbed-cloud-example-ESP8266/
[mbed] Importing program "mbed-cloud-example-ESP8266" from "https://os.mbed.com/users/MACRUM/code/mbed-cloud-example-ESP8266" at latest revision in the current branch
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\mbed\mbed.py", line 3034, in main
    status = pargs.command(pargs)
  File "c:\python27\lib\site-packages\mbed\mbed.py", line 1856, in thunk
    return command(**argv)
  File "c:\python27\lib\site-packages\mbed\mbed.py", line 2001, in import_
    if repo.clone(repo.url, repo.path, rev=repo.rev, depth=depth, protocol=protocol, offline=offline):
  File "c:\python27\lib\site-packages\mbed\mbed.py", line 1244, in clone
    scm.clone(url, path, depth=depth, protocol=protocol, **kwargs)
  File "c:\python27\lib\site-packages\mbed\mbed.py", line 474, in clone
    pquery([hg_cmd, 'clone', '--config', 'progress.assume-tty=true', formaturl(url, protocol), name], output_callback=Hg.action_progress)
  File "c:\python27\lib\site-packages\mbed\mbed.py", line 273, in pquery
    return stdout.decode("utf-8")
  File "c:\python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x91 in position 0: invalid start byte
[mbed] ERROR: Unknown Error: 'utf8' codec can't decode byte 0x91 in position 0: invalid start byte
---
PS C:\Users\toyom>

[Mirrored to Jira]

[theotherjimmy]

@screamerbg We could resolve this issue by changing the return value from a decoded string into a raw bytes object. Then we would have to use bytes regex to process it. We could also use a replacing decoder, but this looks like it's related to cloning, so that may not work as expected.
[Mirrored to Jira]