Mbed-os: GCC Toolchain Broken with Python 3

[tracernz]

Description

In this case compiling for NUCLEO_L476RG (but that seems irrelevant), GCC_ARM toolchain (GCC 8.2.0), mbed-cli 1.7.5, Python 3.7.0. The problem occurs with current master HEAD (2a824a1cebd4b363deb322e0fac5ba52b306923d), replace master in the git bisect command with 2a824a1cebd4b363deb322e0fac5ba52b306923d to repro exactly. The bug was introduced in 6f54a8fdd74e0f7526295a3cd8d8758e61a144a4.

Repro:

$ cd mbed-project-that-is-ready-to-build/mbed-os
$ git bisect start
$ git bisect bad master
$ git bisect good mbed_lib_rev162
$ git bisect run mbed compile -v
....snip...
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[6f54a8fdd74e0f7526295a3cd8d8758e61a144a4] Check version of Mbed CLI compile
running mbed compile --target NUCLEO_L476RG -v
[mbed] WARNING: If you're using Python 3 with Mbed OS 5.8 and earlier versions, Python errors will occur when compiling, testing and exporting
---
[mbed] Working path "/home/mike/dev/mbed/mbed-ws2812/mbed-os" (library)
[mbed] Exec "/usr/bin/python -u /home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/make.py -t GCC_ARM -m NUCLEO_L476RG --source .. --build ../BUILD/NUCLEO_L476RG/GCC_ARM -v" in "/home/mike/dev/mbed/mbed-ws2812/mbed-os"
Traceback (most recent call last):
  File "/home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/make.py", line 290, in <module>
    ignore=options.ignore)
  File "/home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/build_api.py", line 555, in build_project
    toolchain.version_check()
  File "/home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/toolchains/gcc.py", line 119, in version_check
    match = self.GCC_VERSION_RE.match(word)
TypeError: cannot use a string pattern on a bytes-like object
[mbed] ERROR: "/usr/bin/python" returned error.
       Code: 1
       Path: "/home/mike/dev/mbed/mbed-ws2812/mbed-os"
       Command: "/usr/bin/python -u /home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/make.py -t GCC_ARM -m NUCLEO_L476RG --source .. --build ../BUILD/NUCLEO_L476RG/GCC_ARM -v"
       Tip: You could retry the last command with "-v" flag for verbose output
---
6f54a8fdd74e0f7526295a3cd8d8758e61a144a4 is the first bad commit
commit 6f54a8fdd74e0f7526295a3cd8d8758e61a144a4
Author: Jimmy Brisson <[email protected]>
Date:   Mon Jun 18 14:03:09 2018 -0500

    Check version of Mbed CLI compile

Issue request type

[ ] Question
[ ] Enhancement
[X] Bug

[amq]

Related: https://github.com/ARMmbed/mbed-os/issues/7026
[Mirrored to Jira]

[cmonr]

@tracernz Would you mind seeing if this still breaks with Python 3.6? We're not quite testing our codebase with 3.7 yet.
[Mirrored to Jira]

[tracernz]

@cmonr Same result building with Python 3.6.6. This is a Python 2 -> 3 issue, where bytes-like objects are not interchangeable with string-like objects anymore. With Python 3.5 it fails with #7026. Do you have a Python 3 build in CI?

Python 3.6.6:

[mbed] Exec "/home/mike/dev/mbed/mbed-ws2812/mbed-os/.venv/bin/python3.6 -u /home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/make.py -t GCC_ARM -m NUCLEO_L476RG --source .. --build ../BUILD/NUCLEO_L476RG/GCC_ARM -v" in "/home/mike/dev/mbed/mbed-ws2812/mbed-os"
Traceback (most recent call last):
  File "/home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/make.py", line 290, in <module>
    ignore=options.ignore)
  File "/home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/build_api.py", line 555, in build_project
    toolchain.version_check()
  File "/home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/toolchains/gcc.py", line 119, in version_check
    match = self.GCC_VERSION_RE.match(word)
TypeError: cannot use a string pattern on a bytes-like object
[mbed] ERROR: "/home/mike/dev/mbed/mbed-ws2812/mbed-os/.venv/bin/python3.6" returned error.
       Code: 1
       Path: "/home/mike/dev/mbed/mbed-ws2812/mbed-os"
       Command: "/home/mike/dev/mbed/mbed-ws2812/mbed-os/.venv/bin/python3.6 -u /home/mike/dev/mbed/mbed-ws2812/mbed-os/tools/make.py -t GCC_ARM -m NUCLEO_L476RG --source .. --build ../BUILD/NUCLEO_L476RG/GCC_ARM -v"
       Tip: You could retry the last command with "-v" flag for verbose output
---
6f54a8fdd74e0f7526295a3cd8d8758e61a144a4 is the first bad commit
commit 6f54a8fdd74e0f7526295a3cd8d8758e61a144a4
Author: Jimmy Brisson <[email protected]>
Date:   Mon Jun 18 14:03:09 2018 -0500

    Check version of Mbed CLI compile

[Mirrored to Jira]

[cmonr]

Thanks for testing again. Not yet for Mbed OS, but it's being worked on.
[Mirrored to Jira]

[cmonr]

Also, a reminder that the supported GCC version for Mbed OS is GCC 6 (we're working on bumping versions as well).

https://os.mbed.com/docs/latest/tools/index.html
[Mirrored to Jira]

[adbridge]

Internal Jira reference: https://jira.arm.com/browse/IOTCORE-366

[cmonr]

@tracernz Would you mind taking another look at this?

We've since (_finally_) added multiple Py3 versions within CI along with fixes that were needed.

[tracernz]

@cmonr Great to hear! I haven't used mbed since this issue so the project was sitting in the same state. First I repro'd as per my initial report. Then I fetched and fast-forward merged the latest mbed-os master. Unfortunately I ran into a different problem. It looks like a driver was moved in mbed sources. I'll try to find some time to figure that out (I forgot how mbed libraries work etc! :P), and test this again.

tools.config.ConfigException: Library name 'atmel-rf' is not unique (defined in '/home/mike/dev/mbed/mbed-ws2812/easy-connect/atmel-rf-driver/mbed_lib.json' and '/home/mike/dev/mbed/mbed-ws2812/mbed-os/components/802.15.4_RF/atmel-rf-driver/mbed_lib.json')

[tracernz]

@cmonr Okay, I just deleted the easy connect library (+ commented associated code in my app), and all is good with e03b3b68c1fde585989a22134193152fb5e6988d and Python 3.7.2. I can confirm this bug is fixed. :+1: