Mbed-os: Compile failing when project path contains non-ascii character

[xymopen]

Description

• Type: Bug

---

Bug

platform:
(Windows 10 64 Bit)

Python version:
(2.7.13 64 Bit)

mbed-cli version:
(1.2.0)

mbed-os sha:
(692d905)

Steps to reproduce

# 示例 means example
mkdir 示例
cd 示例
mbed import mbed-os
cd mbed-os
mbed compile -m DISCO_F407VG -t GCC_ARM

Actual behavior

Building project mbed-os (DISCO_F407VG, GCC_ARM)
Scan: .
Scan: env
Scan: mbed
Compile [  0.3%]: AnalogIn.cpp
Compile [  0.7%]: BusInOut.cpp
Compile [  1.0%]: BusIn.cpp
Compile [  1.3%]: BusOut.cpp
Compile [  1.6%]: FlashIAP.cpp
Compile [  2.0%]: CAN.cpp
Compile [  2.3%]: Ethernet.cpp
Compile [  2.6%]: I2CSlave.cpp
Compile [  2.9%]: I2C.cpp
Compile [  3.3%]: InterruptIn.cpp
Compile [  3.6%]: InterruptManager.cpp
Compile [  3.9%]: RawSerial.cpp
Compile [  4.2%]: SPISlave.cpp
Compile [  4.6%]: SPI.cpp
Compile [  4.9%]: Serial.cpp
Compile [  5.2%]: SerialBase.cpp
Compile [  5.5%]: Ticker.cpp
Compile [  5.9%]: Timeout.cpp
Compile [  6.2%]: TimerEvent.cpp
Compile [  6.5%]: Timer.cpp
Compile [  6.8%]: UARTSerial.cpp
Compile [  7.2%]: equeue_posix.c
Compile [  7.5%]: EventQueue.cpp
Compile [  7.8%]: equeue_mbed.cpp
Compile [  8.1%]: equeue.c
Compile [  8.5%]: Dir.cpp
Compile [  8.8%]: File.cpp
Compile [  9.1%]: FileSystem.cpp
Compile [  9.4%]: ChainingBlockDevice.cpp
Compile [  9.8%]: ccsbcs.cpp
Compile [ 10.1%]: HeapBlockDevice.cpp
Compile [ 10.4%]: SlicingBlockDevice.cpp
Compile [ 10.7%]: MBRBlockDevice.cpp
Compile [ 11.1%]: ff.cpp
Compile [ 11.4%]: greentea_metrics.cpp
Compile [ 11.7%]: greentea_serial.cpp
Compile [ 12.1%]: FATFileSystem.cpp
Compile [ 12.4%]: greentea_test_env.cpp
Compile [ 12.7%]: unity.c
Compile [ 13.0%]: mbed-utest-shim.cpp
Compile [ 13.4%]: unity_handler.cpp
Compile [ 13.7%]: utest_case.cpp
Compile [ 14.0%]: utest_default_handlers.cpp
Compile [ 14.3%]: utest_stack_trace.cpp
Compile [ 14.7%]: utest_greentea_handlers.cpp
Compile [ 15.0%]: mbed_trng.c
Compile [ 15.3%]: utest_harness.cpp
Compile [ 15.6%]: aesni.c
Compile [ 16.0%]: utest_shim.cpp
Compile [ 16.3%]: arc4.c
Compile [ 16.6%]: utest_types.cpp
Compile [ 16.9%]: aes.c
Compile [ 17.3%]: asn1parse.c
Compile [ 17.6%]: asn1write.c
Compile [ 17.9%]: base64.c
Compile [ 18.2%]: blowfish.c
Compile [ 18.6%]: camellia.c
Compile [ 18.9%]: certs.c
Compile [ 19.2%]: ccm.c
Compile [ 19.5%]: cipher_wrap.c
Compile [ 19.9%]: cipher.c
Compile [ 20.2%]: cmac.c
Compile [ 20.5%]: ctr_drbg.c
Compile [ 20.8%]: bignum.c
Compile [ 21.2%]: debug.c
Compile [ 21.5%]: des.c
Compile [ 21.8%]: dhm.c
Compile [ 22.1%]: ecdh.c
Compile [ 22.5%]: ecjpake.c
Compile [ 22.8%]: ecdsa.c
Compile [ 23.1%]: entropy.c
Compile [ 23.5%]: entropy_poll.c
Compile [ 23.8%]: ecp_curves.c
Compile [ 24.1%]: havege.c
Compile [ 24.4%]: error.c
Compile [ 24.8%]: gcm.c
Compile [ 25.1%]: hmac_drbg.c
Compile [ 25.4%]: md2.c
Compile [ 25.7%]: ecp.c
Compile [ 26.1%]: md4.c
Compile [ 26.4%]: md5.c
Compile [ 26.7%]: md.c
Compile [ 27.0%]: net_sockets.c
Compile [ 27.4%]: memory_buffer_alloc.c
Compile [ 27.7%]: md_wrap.c
Compile [ 28.0%]: padlock.c
Compile [ 28.3%]: oid.c
Compile [ 28.7%]: pk.c
Compile [ 29.0%]: pem.c
Compile [ 29.3%]: pkcs11.c
Compile [ 29.6%]: pkcs12.c
Compile [ 30.0%]: pkcs5.c
Compile [ 30.3%]: pk_wrap.c
Compile [ 30.6%]: ripemd160.c
Compile [ 30.9%]: platform.c
Compile [ 31.3%]: pkwrite.c
Compile [ 31.6%]: pkparse.c
Compile [ 31.9%]: sha1.c
Compile [ 32.2%]: ssl_cache.c
Compile [ 32.6%]: ssl_ciphersuites.c
Compile [ 32.9%]: sha256.c
Compile [ 33.2%]: sha512.c
Compile [ 33.6%]: ssl_cli.c
Compile [ 33.9%]: rsa.c
Compile [ 34.2%]: ssl_cookie.c
Compile [ 34.5%]: ssl_srv.c
Compile [ 34.9%]: ssl_ticket.c
Compile [ 35.2%]: ssl_tls.c
Compile [ 35.5%]: threading.c
Compile [ 35.8%]: timing.c
Compile [ 36.2%]: version.c
Compile [ 36.5%]: version_features.c
Compile [ 36.8%]: x509_create.c
Compile [ 37.1%]: x509_csr.c
Compile [ 37.5%]: x509.c
Compile [ 37.8%]: x509_crl.c
Compile [ 38.1%]: x509write_crt.c
Compile [ 38.4%]: x509write_csr.c
Compile [ 38.8%]: xtea.c
Compile [ 39.1%]: aes_alt.c
Compile [ 39.4%]: sha1_alt.c
Compile [ 39.7%]: sha256_alt.c
Compile [ 40.1%]: NetworkInterface.cpp
Compile [ 40.4%]: x509_crt.c
Compile [ 40.7%]: NetworkStack.cpp
Compile [ 41.0%]: Socket.cpp
Compile [ 41.4%]: TCPServer.cpp
Compile [ 41.7%]: SocketAddress.cpp
Compile [ 42.0%]: OnboardCellularInterface.cpp
Compile [ 42.3%]: WiFiAccessPoint.cpp
Compile [ 42.7%]: TCPSocket.cpp
Compile [ 43.0%]: UDPSocket.cpp
Compile [ 43.3%]: mbed_flash_api.c
Compile [ 43.6%]: PPPCellularInterface.cpp
Compile [ 44.0%]: UARTCellularInterface.cpp
Compile [ 44.3%]: nsapi_dns.cpp
Compile [ 44.6%]: mbed_gpio.c
Compile [ 45.0%]: mbed_lp_ticker_api.c
Compile [ 45.3%]: mbed_pinmap_common.c
Compile [ 45.6%]: mbed_ticker_api.c
Compile [ 45.9%]: mbed_us_ticker_api.c
Compile [ 46.3%]: CallChain.cpp
Compile [ 46.6%]: FileHandle.cpp
Compile [ 46.9%]: FileBase.cpp
Compile [ 47.2%]: ATCmdParser.cpp
Compile [ 47.6%]: FilePath.cpp
Compile [ 47.9%]: LocalFileSystem.cpp
Compile [ 48.2%]: FileSystemHandle.cpp
Compile [ 48.5%]: mbed_application.c
Compile [ 48.9%]: mbed_alloc_wrappers.cpp
Compile [ 49.2%]: Stream.cpp
Compile [ 49.5%]: mbed_assert.c
Compile [ 49.8%]: mbed_board.c
Compile [ 50.2%]: mbed_critical.c
Compile [ 50.5%]: mbed_error.c
Compile [ 50.8%]: mbed_mem_trace.c
Compile [ 51.1%]: mbed_mktime.c
Compile [ 51.5%]: mbed_interface.c
Compile [ 51.8%]: mbed_sdk_boot.c
Compile [ 52.1%]: mbed_poll.cpp
Compile [ 52.4%]: mbed_rtc_time.cpp
Compile [ 52.8%]: mbed_stats.c
Compile [ 53.1%]: mbed_wait_api_no_rtos.c
Compile [ 53.4%]: mbed_semihost_api.c
Compile [ 53.7%]: mbed_retarget.cpp
Compile [ 54.1%]: Mutex.cpp
Compile [ 54.4%]: Semaphore.cpp
Compile [ 54.7%]: mbed_wait_api_rtos.cpp
Compile [ 55.0%]: RtosTimer.cpp
Compile [ 55.4%]: mbed_boot.c
Compile [ 55.7%]: RTX_Config.c
Compile [ 56.0%]: rtos_idle.c
Compile [ 56.4%]: cmsis_os1.c
Compile [ 56.7%]: irq_cm4f.S
Compile [ 57.0%]: rt_OsEventObserver.c
Compile [ 57.3%]: Thread.cpp
Compile [ 57.7%]: rtx_delay.c
Compile [ 58.0%]: rtx_evflags.c
Compile [ 58.3%]: rtx_lib.c
Compile [ 58.6%]: rtx_evr.c
Compile [ 59.0%]: rtx_memory.c
Compile [ 59.3%]: rtx_kernel.c
Compile [ 59.6%]: rtx_mempool.c
Compile [ 59.9%]: rtx_msgqueue.c
Compile [ 60.3%]: rtx_mutex.c
Compile [ 60.6%]: rtx_semaphore.c
Compile [ 60.9%]: rtx_system.c
Compile [ 61.2%]: mbed_rtx_handlers.c
Compile [ 61.6%]: rtx_timer.c
Compile [ 61.9%]: PeripheralPins.c
Compile [ 62.2%]: startup_stm32f407xx.S
Compile [ 62.5%]: system_clock.c
Compile [ 62.9%]: rtx_thread.c
Compile [ 63.2%]: analogin_api.c
Compile [ 63.5%]: hal_init_pre.c
Compile [ 63.8%]: analogout_device.c
Compile [ 64.2%]: stm32f4xx_hal.c
Compile [ 64.5%]: stm32f4xx_hal_adc_ex.c
Compile [ 64.8%]: stm32f4xx_hal_cec.c
Compile [ 65.1%]: stm32f4xx_hal_adc.c
Compile [ 65.5%]: stm32f4xx_hal_can.c
Compile [ 65.8%]: stm32f4xx_hal_cryp.c
Compile [ 66.1%]: stm32f4xx_hal_crc.c
Compile [ 66.4%]: stm32f4xx_hal_cortex.c
Compile [ 66.8%]: stm32f4xx_hal_cryp_ex.c
Compile [ 67.1%]: stm32f4xx_hal_dac_ex.c
Compile [ 67.4%]: stm32f4xx_hal_dac.c
Compile [ 67.8%]: stm32f4xx_hal_dcmi_ex.c
Compile [ 68.1%]: stm32f4xx_hal_dcmi.c
Compile [ 68.4%]: stm32f4xx_hal_dfsdm.c
Compile [ 68.7%]: stm32f4xx_hal_dma2d.c
Compile [ 69.1%]: stm32f4xx_hal_dma.c
Compile [ 69.4%]: stm32f4xx_hal_dma_ex.c
Compile [ 69.7%]: stm32f4xx_hal_dsi.c
Compile [ 70.0%]: stm32f4xx_hal_flash.c
Compile [ 70.4%]: stm32f4xx_hal_eth.c
Compile [ 70.7%]: stm32f4xx_hal_flash_ex.c
Compile [ 71.0%]: stm32f4xx_hal_flash_ramfunc.c
Compile [ 71.3%]: stm32f4xx_hal_fmpi2c_ex.c
Compile [ 71.7%]: stm32f4xx_hal_fmpi2c.c
Compile [ 72.0%]: stm32f4xx_hal_hash.c
Compile [ 72.3%]: stm32f4xx_hal_gpio.c
Compile [ 72.6%]: stm32f4xx_hal_hash_ex.c
Compile [ 73.0%]: stm32f4xx_hal_i2c_ex.c
Compile [ 73.3%]: stm32f4xx_hal_hcd.c
Compile [ 73.6%]: stm32f4xx_hal_i2s.c
Compile [ 73.9%]: stm32f4xx_hal_i2s_ex.c
Compile [ 74.3%]: stm32f4xx_hal_irda.c
Compile [ 74.6%]: stm32f4xx_hal_iwdg.c
Compile [ 74.9%]: stm32f4xx_hal_i2c.c
Compile [ 75.2%]: stm32f4xx_hal_lptim.c
Compile [ 75.6%]: stm32f4xx_hal_ltdc.c
Compile [ 75.9%]: stm32f4xx_hal_ltdc_ex.c
Compile [ 76.2%]: stm32f4xx_hal_msp_template.c
Compile [ 76.5%]: stm32f4xx_hal_mmc.c
[Warning] stm32f4xx_hal_mmc.c@1319,30: suggest parentheses around comparison in operand of '&' [-Wparentheses]
Compile [ 76.9%]: stm32f4xx_hal_nor.c
Compile [ 77.2%]: stm32f4xx_hal_pccard.c
Compile [ 77.5%]: stm32f4xx_hal_nand.c
Compile [ 77.9%]: stm32f4xx_hal_pcd_ex.c
Compile [ 78.2%]: stm32f4xx_hal_pwr.c
Compile [ 78.5%]: stm32f4xx_hal_pwr_ex.c
Compile [ 78.8%]: stm32f4xx_hal_pcd.c
[Warning] stm32f4xx_hal_pcd.c@1285,11: comparison between signed and unsigned integer expressions [-Wsign-compare]
[Warning] stm32f4xx_hal_pcd.c@1300,13: comparison between signed and unsigned integer expressions [-Wsign-compare]
[Warning] stm32f4xx_hal_pcd.c@1312,10: comparison between signed and unsigned integer expressions [-Wsign-compare]
Compile [ 79.2%]: stm32f4xx_hal_qspi.c
Compile [ 79.5%]: stm32f4xx_hal_rcc.c
Compile [ 79.8%]: stm32f4xx_hal_rcc_ex.c
Compile [ 80.1%]: stm32f4xx_hal_rng.c
Compile [ 80.5%]: stm32f4xx_hal_rtc.c
Compile [ 80.8%]: stm32f4xx_hal_sai.c
Compile [ 81.1%]: stm32f4xx_hal_sai_ex.c
Compile [ 81.4%]: stm32f4xx_hal_rtc_ex.c
Compile [ 81.8%]: stm32f4xx_hal_sdram.c
Compile [ 82.1%]: stm32f4xx_hal_spdifrx.c
Compile [ 82.4%]: stm32f4xx_hal_smartcard.c
Compile [ 82.7%]: stm32f4xx_hal_sd.c
Compile [ 83.1%]: stm32f4xx_hal_sram.c
Compile [ 83.4%]: stm32f4xx_hal_spi.c
Compile [ 83.7%]: stm32f4xx_hal_tim_ex.c
Compile [ 84.0%]: stm32f4xx_hal_uart.c
Compile [ 84.4%]: stm32f4xx_hal_tim.c
Compile [ 84.7%]: stm32f4xx_ll_adc.c
Compile [ 85.0%]: stm32f4xx_hal_wwdg.c
Compile [ 85.3%]: stm32f4xx_ll_crc.c
Compile [ 85.7%]: stm32f4xx_ll_dac.c
Compile [ 86.0%]: stm32f4xx_ll_dma.c
Compile [ 86.3%]: stm32f4xx_ll_dma2d.c
Compile [ 86.6%]: stm32f4xx_hal_usart.c
Compile [ 87.0%]: stm32f4xx_ll_exti.c
Compile [ 87.3%]: stm32f4xx_ll_gpio.c
Compile [ 87.6%]: stm32f4xx_ll_i2c.c
Compile [ 87.9%]: stm32f4xx_ll_fmc.c
Compile [ 88.3%]: stm32f4xx_ll_pwr.c
Compile [ 88.6%]: stm32f4xx_ll_lptim.c
Compile [ 88.9%]: stm32f4xx_ll_rcc.c
Compile [ 89.3%]: stm32f4xx_ll_fsmc.c
Compile [ 89.6%]: stm32f4xx_ll_rng.c
Compile [ 89.9%]: stm32f4xx_ll_rtc.c
Compile [ 90.2%]: stm32f4xx_ll_spi.c
Compile [ 90.6%]: stm32f4xx_ll_tim.c
Compile [ 90.9%]: stm32f4xx_ll_usart.c
Compile [ 91.2%]: stm32f4xx_ll_sdmmc.c
Compile [ 91.5%]: system_stm32f4xx.c
Compile [ 91.9%]: stm32f4xx_ll_utils.c
Compile [ 92.2%]: gpio_irq_device.c
Compile [ 92.5%]: stm32f4xx_ll_usb.c
Compile [ 92.8%]: pwmout_device.c
Compile [ 93.2%]: spi_api.c
Compile [ 93.5%]: serial_device.c
[Warning] serial_device.c@290,35: unused variable 'tmpval' [-Wunused-variable]
Compile [ 93.8%]: analogout_api.c
Compile [ 94.1%]: can_api.c
Compile [ 94.5%]: gpio_api.c
Compile [ 94.8%]: hal_tick_16b.c
Compile [ 95.1%]: gpio_irq_api.c
Compile [ 95.4%]: hal_tick_32b.c
Compile [ 95.8%]: lp_ticker.c
Compile [ 96.1%]: mbed_overrides.c
Compile [ 96.4%]: pinmap.c
Compile [ 96.7%]: i2c_api.c
Compile [ 97.1%]: port_api.c
Compile [ 97.4%]: pwmout_api.c
Compile [ 97.7%]: rtc_api.c
Compile [ 98.0%]: serial_api.c
Compile [ 98.4%]: trng_api.c
Compile [ 98.7%]: sleep.c
[ERROR] 'ascii' codec can't decode byte 0xcf in position 0: ordinal not in range(128)
[mbed] ERROR: "c:\program files\python\2.x\python.exe" returned error code 1.
[mbed] ERROR: Command "c:\program files\python\2.x\python.exe -u E:\项目\示例\mbed-os\tools\make.py -t GCC_ARM -m DISCO_F407VG --source . --build .\BUILD\DISCO_F407VG\GCC_ARM" in "E:\项目\示例\mbed-os"
---

Removing the non-ascii characters in project path resolves the issue.

[geky]

cc @theotherjimmy
[Mirrored to Jira]

[Nodraak]

Is a Python 3 migration of the mbed-os tools planned? I believe that it would solve a lot of utf-8-related issues.
[Mirrored to Jira]

[theotherjimmy]

@Nodraak Yes it would solve this immediate problem. We have a few dependencies on other packages that are python 2 only, so we're waiting on them. Further, if we upgrade to python 3, we would have to update mbed CLI to call python 3, which would break old versions of the tools...
[Mirrored to Jira]

[Nodraak]

I understand. Is there an issue or something that keeps track of which packages are not supported by py3 yet? Concerning mbed cli, maybe it would be feasable to support both python version, or otherwise find a way for mbed cli to detect which python version should be used (based on the shebang? It's a bit hacky, but could be a solution).
[Mirrored to Jira]

[theotherjimmy]

based on the shebang? It's a bit hacky, but could be a solution

I was going to suggest that. and default to python 2 when no shebang is present should be backwards compatible.
[Mirrored to Jira]

[theotherjimmy]

Anyway. @xymopen I'm going to get this working for you soon. I'll mention you on the PR when I put it up.
[Mirrored to Jira]

[xymopen]

@theotherjimmy Thanks.
[Mirrored to Jira]

[theotherjimmy]

I think that the upgrade to python 3 (and 2) we did recently fixed this bug. @cmonr Could you comment on this?
[Mirrored to Jira]

[cmonr]

Interesting. Definitely looks like an issue. Will start looking into how to solve this, with some new tests to come.
[Mirrored to Jira]

[ciarmcom]

Thank you for raising this detailed GitHub issue. I am now notifying our internal issue triagers.
Internal Jira reference: https://jira.arm.com/browse/IOTOSM-2153