Mbed-cli: Traceback error when using v1.7.2 + Python 3

[MarceloSalazar]

Mbed CLI v1.7.2 supports Python 3
This is expected to work well when using Mbed OS 5.9+

However, for those users of Mbed OS 5.8 and earlier versions, there is a really bad traceback error when attempting to compile.

C:\mbed\mbed-os-example-wifi>mbed compile -t GCC_ARM -m K64F -c
Traceback (most recent call last):
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\make.py", line 45, in <module>
    from tools.options import get_default_options_parser
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\options.py", line 24, in <module>
    from .toolchains import TOOLCHAINS
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\toolchains\__init__.py", line 40, in <module>
    from ..memap import MemapParser
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\memap.py", line 80, in <module>
    class _GccParser(_Parser):
  File "C:\mbed\mbed-os-example-wifi\mbed-os\tools\memap.py", line 82, in _GccParser
    RE_LIBRARY_OBJECT = re.compile(r'^.+' + sep + r'lib((.+\.a)\((.+\.o)\))$')
  File "c:\users\ms\appdata\local\programs\python\python36-32\lib\re.py", line 233, in compile
    return _compile(pattern, flags)
  File "c:\users\ms\appdata\local\programs\python\python36-32\lib\re.py", line 301, in _compile
    p = sre_compile.compile(pattern, flags)
  File "c:\users\ms\appdata\local\programs\python\python36-32\lib\sre_compile.py", line 562, in compile
    p = sre_parse.parse(p, flags)
  File "c:\users\ms\appdata\local\programs\python\python36-32\lib\sre_parse.py", line 855, in parse
    p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
  File "c:\users\ms\appdata\local\programs\python\python36-32\lib\sre_parse.py", line 416, in _parse_sub
    not nested and not items))
  File "c:\users\ms\appdata\local\programs\python\python36-32\lib\sre_parse.py", line 502, in _parse
    code = _escape(source, this, state)
  File "c:\users\ms\appdata\local\programs\python\python36-32\lib\sre_parse.py", line 401, in _escape
    raise source.error("bad escape %s" % escape, len(escape))
sre_constants.error: bad escape \l at position 3
[mbed] ERROR: "c:\users\ms\appdata\local\programs\python\python36-32\python.exe" returned error code 1.
[mbed] ERROR: Command "c:\users\ms\appdata\local\programs\python\python36-32\python.exe -u C:\mbed\mbed-os-example-wifi\mbed-os\tools\make.py -t GCC_ARM -m K64F --source . --build .\BUILD\K64F\GCC_ARM -c" in "C:\mbed\mbed-os-example-wifi"
---

This is a serious problem for workshops and training events where we continue to use Mbed OS <= 5.8.

The proposed solution : catch the error and display a message that Python 3 is supported from 5.9+
It may also require a modification in the OS tools to detect the version of Mbed OS (TBC).

FYI @theotherjimmy @cmonr @screamerbg

[cmonr]

@MarceloSalazar Thanks for opening up an issue to discuss the technical aspects of the fix.

That's an interesting solution. I'm not a fan of unconditionally blaming Python 3 because it creates the implication that all cli errors are caused by the Python 3 backwards compatibility issues. _However_, I would be open to printing a message _asking_ user verify their version of Mbed OS is Python 3 compatible, _if_ they're running Python 3, before submitting an issue.

Alternatives I'd also like to propose:

• Detect the version of Python and Mbed OS at runtime, and print a warning.
  
  
  
  • As noted in the Handbook PR, the worst case is not simple. It would involve determining the Mbed OS base version by an arbitrary SHA hash. I'm sure there's a single git command that could do magic, but figuring that out would probably take time we don't have.
  
  
• Unconditionally printing a warning when running Mbed CLI with Python 3 about backwards compatibility (aka, the lack of it).
  
  
  
  • I don't like this since imo, users should be reading documentation for this information, _but_ it's a <15 min update.
  
  

A more general question. By adding a block of code to check versions in real time, don't you think we're setting an expectation that users don't need to read the documentation?

[screamerbg]

@cmonr Agreed with you concerns and suggested approaches. I thought that you added warning when mbed CLI is running with python 3?

[MarceloSalazar]

I think one of the simple checks that Mbed CLI could potentially do when running Python 3 is whether the Mbed OS version is 5.9+ or not.

I don't want to get involved in discussing actual implementations - but these lines contain useful information:
https://github.com/ARMmbed/mbed-os/blob/mbed-os-5.8/mbed.h#L23-L24

I would be open to printing a message asking user verify their version of Mbed OS is Python 3 compatible

This should be ok as quick/initial fix. My main interest is to catch the traceback error and display a message.

A more general question. By adding a block of code to check versions in real time, don't you think we're setting an expectation that users don't need to read the documentation?

The docs contain lots of useful information but the reality is first-time users shouldn't need to spend (much) time with it. The tools should help to guide them and ensure a great UX :)

[cmonr]

@screamerbg That's the weird thing. I very clearly recall adding a warning, but looking at my personal branch's commit history, it looks like it didn't get committed.

@MarceloSalazar AH! Thanks for the link. That's the bit that I was missing. I thought they only way to read out Mbed OS' version was through git.

On Monday, I'll look into doing one or the other, either reading the file, or catching any exception to present a Python 3 warning.

[screamerbg]

@cmonr Let's not get ahead of ourselves. Firstly, mixing code, build system and CLI to determine python 3 compatibility sounds wrong. There are other ways to achieve this. Also this functionality was originally planned and the fact that it didn't land in mbed CLI 1.7 is the main reason for this bug report / feature request to exist. Let's fix that first.

[theotherjimmy]

Several thoughts:

Version detection of mbed OS it hard @MarceloSalazar : mbed.h is all 0's on master; versions are commits that don't exists on master.

Let's not do version detection when it might work, only when ~it~ something has already failed. Even then, I'm not sure we should be doing version detection.

@screamerbg Most of this work was about getting python 3 support in Mbed OS and it's dependencies. That makes it a Mbed OS 5.9 feature, more than anything else. This bug report exists because:

for those users of Mbed OS 5.8 and earlier versions, there is a really bad traceback

So we're quite on topic.

[cmonr]

Fyi: https://github.com/ARMmbed/mbed-cli/pull/689

[cmonr]

Moving the conversation back here about the second part of the original suggestion.

But I believe we'll still hit the error with <=5.8, unless we include a fix in CLI.
Is there anything we could do in CLI to catch the trace back for this particular use case?

... _the error_ ...

Specifically which error? The exception that causes the traceback, or the usability issue? If a user were to use the latest Mbed CLI version (assuming #689 is merged and released) and a user uses Mbed OS v5.8 (for example), the traceback _would_ happen, but the user _would_ receive a Python 3 warning.

... _catch the traceback for this particular use case?_ ...

I'm incredibly hesitant to add code into the project that is only run for a single edge case. This isn't the only way that Python 3 compatibility can be broken when running an older version of Mbed OS, and I'd like to avoid going down the path of printing custom error messages and doing custom error detection for each known failure case, especially since it's always possible for more error cases to be found (but we always hope for the opposite).

[MarceloSalazar]

@cmonr I get your point and understand the interest not to add specific code to catch the traceback.

Considering that the warning will be printed before the traceback, it should be OK to give the user a hint on where the problem is coming from.

Apart from detecting the version of Mbed OS (and @theotherjimmy already commented the inconveniences) I can't think of anything else.

I'll consider this ticket resolved once https://github.com/ARMmbed/mbed-cli/pull/689 is released. Thanks!

[ciarmcom]

ARM Internal Ref: MBOTRIAGE-441

[cmonr]

https://github.com/ARMmbed/mbed-cli/releases/tag/1.7.3