Platformio-core: Problem with Python 3.8+ on Windows when network drive is used

[Hofei90]

Temporary solution

1. Please uninstall all Pythons from the system and install the latest Python 3.7 from https://www.python.org/downloads/ using https://docs.platformio.org/en/latest/faq.html#install-python-interpreter
2. Move your project to local disk (for example, C:)

---

Configuration

Operating system:
Windows_NT x64 10.0.17763

PlatformIO Version (platformio --version):
4.2.2b1
and
4.2.1

Python Version
3.8 32bit

Description of problem

During compilation I get the Python Traceback UnicodeDecodeError: 'charmap' codec can't decode byte.

Full Message:

> Executing task in folder IR_Sender_WB: C:\Users\***\.platformio\penv\Scripts\platformio.exe run <

Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 2.3.3 > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
- framework-arduinoespressif8266 3.20603.200130 (2.6.3)
- tool-esptool 1.413.0 (4.13)
- tool-esptoolpy 1.20800.0 (2.8.0)
- toolchain-xtensa 2.40802.191122 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 31 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <IRremoteESP8266> 2.7.4
Building in release mode
Compiling .pio\build\nodemcuv2\src\main.cpp.o
Generating LD script .pio\build\nodemcuv2\ld\local.eagle.app.v6.common.ld
Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 46, in run
    self.do_reading()
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in do_reading
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in <lambda>
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 167: character maps to <undefined>
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRac.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRrecv.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRsend.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRtext.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRtimer.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRutils.cpp.o
======================================================================================== [FAILED] Took 5.42 seconds ========================================================================================
The terminal process terminated with exit code: 1

Terminal will be reused by tasks, press any key to close it.

If problems with PlatformIO Build System:

The content of platformio.ini:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env:nodemcuv2]
platform = espressif8266
board = nodemcuv2
framework = arduino
monitor_speed = 115200

Additional info

On another PC with Python version 3.7 there are no problems compiling.
Here are the data of the PC where there is no problem:

Version: 1.43.0 (user setup)
Commit: 78a4c91400152c0f27ba4d363eb56d2835f9903a
Date: 2020-03-09T19:47:57.235Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Windows_NT x64 10.0.17763

PlatformIO: Core 4.2.1, Home 3.1.1

[ivankravets]

1. Does both machines use the same user name?
2. Do you have non-ASCII chars in the user name on a machine that fails?
3. Do you use the same locale?

[Hofei90]

1. Does both machines use the same user name?

no the machines have different user names
2. Do you have non-ASCII chars in the user name on a machine that fails?
no, the names contain only characters from [A-Z] [a-z] and char \ from the domain Name. But Domainname is same on both maschines.
3. Do you use the same locale?
Do you mean the character encoding setting like UTF-8? In Visual Studio Code is it UTF-8 or you mean on the OS?

[ivankravets]

Where did you get this Python 3.8? Could you try to uninstall it and install again the latest Python 3? Also, please remove .platformio/penv folder form user home folder. Restart VSCode.

[Hofei90]

Hello,
all Python versions were downloaded from the official website. (https://www.python.org/downloads/)

What did I do:

1. uninstalled Python 3.8 and installed the last release with version 3.8.2 and deleted the penv folder and restarted VS Code
   Result: Problem still exists
2. platformIO completely deleted from VS Code and reinstalled (Version 4.2.1)
   Result Problem still exists
3. uninstall Python 3.8.2 and install Python 3.7.3, so that the versions are exactly the same as the working PC. delete the penv folder and restart VS Code.
   Result: Compilation works!

Conclusion:
Python 3.8 causes the error posted above
In Python 3.7 these problems do not yet exist

Translated with www.DeepL.com/Translator (free version)

[ivankravets]

@Hofei90 could you provide a simple project ot reproduce this issue?

[Hofei90]

This is the project where the problems were.
Just removed the docs folder, .vscode, .git and .pio.
IR_Sender_WB.zip

[ivankravets]

I moved this issue to ESP8266 dev-platform. I tried in my VM and cannot reproduce it.

---

@Hofei90 , could you open PlatformIO IDE Terminal and change directory to this project and run:

pio run --jobs 1

Could you provide a new output? It will show where it fails.

[Hofei90]

I tried in my VM and cannot reproduce it.

Did you use Python 3.8.2 32bit too?
I just installed Python 3.8.2 on the other PC where 3.7 was previously installed and it worked.
After upgrading to Python 3.8.2, compiling no longer works.

Here is the output with
pio run --jobs 1

PS U:\gitea\IR_Sender_WB> pio run --jobs 1
Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 2.3.3 > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20603.200130 (2.6.3)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - toolchain-xtensa 2.40802.191122 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 30 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <IRremoteESP8266> 2.7.4
Building in release mode
Compiling .pio\build\nodemcuv2\src\main.cpp.o
Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 46, in run
    self.do_reading()
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in do_reading
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in <lambda>
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 166: character maps to <undefined>
======================================================================================== [FAILED] Took 5.49 seconds ========================================================================================
PS U:\gitea\IR_Sender_WB>

EDIT, add Output with verbose Option:

PS U:\gitea\IR_Sender_WB> pio run --jobs 1 --verbose
Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino; monitor_speed: 115200)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 2.3.3 > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20603.200130 (2.6.3)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - toolchain-xtensa 2.40802.191122 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 30 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <IRremoteESP8266> 2.7.4 (\\***\***\***\gitea\IR_Sender_WB\lib\IRremoteESP8266)
Building in release mode
xtensa-lx106-elf-g++ -o .pio\build\nodemcuv2\src\main.cpp.o -c -fno-rtti -std=c++11 -Os -mlongcalls -mtext-section-literals -falign-functions=4 -U__STRICT_ANSI__ -ffunction-sections -fdata-sections -fno-exceptions -Wall -DPLATFORMIO=40201 -DESP8266 -DARDUINO_ARCH_ESP8266 -DARDUINO_ESP8266_NODEMCU -DF_CPU=80000000L -D__ets__ -DICACHE_FLASH -DARDUINO=10805 -DARDUINO_BOARD=\"PLATFORMIO_NODEMCUV2\" -DFLASHMODE_DIO -DLWIP_OPEN_SRC -DNONOSDK22x_190703=1 -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DVTABLES_IN_FLASH -Iinclude -Isrc -Ilib\IRremoteESP8266\src -IC:\Users\***\.platformio\Exception in thread Thread-3:
pTraceback (most recent call last):
a  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
ck    self.run()
a  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
ges\fra    self._target(*self._args, **self._kwargs)
m  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 46, in run
ewo    self.do_reading()
r  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in do_reading
k-    for byte in iter(lambda: self._pipe_reader.read(1), ""):
a  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in <lambda>
rdui    for byte in iter(lambda: self._pipe_reader.read(1), ""):
n  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
oes    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
pUnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 166: character maps to <undefined>
ressif8266\tools\sdk\include -IC:\Users\***\.platformio\packages\framework-arduinoespressif8266\tools\sdk\libc\xtensa-lx106-elf\include -IC:\Users\***\.platformio\packages\framework-arduinoespressif8266\cores\esp8266 -IC:\Users\***\.platformio\packages\framework-arduinoespressif8266\tools\sdk\lwip2\include -IC:\Users\***\.platformio\packages\framework-arduinoespressif8266\variants\nodemcu
src\main.cpp
======================================================================================== [FAILED] Took 5.44 seconds ========================================================================================

[ivankravets]

\**\userdaten

Could you try to move this project to C:\test-project. Just for test.

[Hofei90]

Good eye, can you please make the path in your post unrecognizable and also remove it from the history

If the folder is on C: it works.
Funny though that under Python 3.7 it also works on the network folder

[ivankravets]

can you please make the path in your post unrecognizable

Sorry, done!

---

So, I moved back this issue to PIO Core because it looks like a bug. What is a type of this drive? How can I create it?

[Hofei90]

Hello,
this is a personal network share within the local network. Further details, which hardware and/or software makes this available, I cannot say at the moment. I will try to find out more details

[Hofei90]

Hello, the network drive is provided by a QNAP NAS. Unfortunately I cannot provide any further information about this

[draeron]

i had this issue on windows (local disk). I solved this by changing the following line:

https://github.com/platformio/platformio-core/blob/develop/platformio/proc.py#L33

from :

self._pipe_reader = os.fdopen(self._fd_read)

to :

self._pipe_reader = os.fdopen(self._fd_read, encoding='utf-8')

there are a couple of stackoverflow on this.

[ivankravets]

@Hofei90 @draeron could you re-test with this PIO Core dev? Please run pio upgrade --dev

Does it work now?

[draeron]

@ivankravets look like it solved the issue i had

[ivankravets]

@draeron thanks!

@Hofei90 could you confirm?

[ivankravets]

I merged into the development branch. Let see what people report.

[ivankravets]

We reverted all changes, the fix for this issue broke PlatformIO for Windows when non-Latin chars are used in project path https://github.com/platformio/platformio-core/issues/3481

We will think about a better solution.

[huexpub]

mark same error and changes files work...

[ivankravets]

@huexpub do you also use network drive?

[huexpub]

@huexpub do you also use network drive?

local disk c:/administrators, python3.8

[ivankravets]

Could you try 1 solution? I saw in our forum https://community.platformio.org/ people downgrading Python to 3.7 and it helps them. Could you try this for the test? Uninstall Python 3.8 and install 3.7. See https://docs.platformio.org/en/latest/faq.html#install-python-interpreter

[zekageri]

Reverted back to python 3.7 on windows 10, not solved my problem. Changing the proc.py file like drareon suggested does help.

But i think i had some comment like this:

/** etc etc **
etc etc
etc
/** etc **/

And that broke everything

[ivankravets]

@zekageri do you also use network drive?

[zekageri]

Nop. Local GitHub folder

[ivankravets]

Please re-run pio upgrade --dev and provide info from

pio system info

[zekageri]

--------------------------  ----------------------------------------------------------
PlatformIO Core             4.4.0a4
Python                      3.7.0-final.0
System Type                 windows_amd64
Platform                    Windows-10
File System Encoding        utf-8
Locale Encoding             cp1250
PlatformIO Core Directory   C:\Users\Dr.Random\.platformio
PlatformIO Core Executable  C:\Users\Dr.Random\.platformio\penv\Scripts\platformio.exe
Python Executable           c:\users\dr.random\.platformio\penv\scripts\python.exe
Global Libraries            0
Development Platforms       2
Package Tools               19
--------------------------  ----------------------------------------------------------

[stisa]

Hello, I'm having similar issues trying to compiler marlin, using vscode and platformio ide.

Traceback (most recent call last):
  File "C:\Users\user.name\.platformio\python3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\user.name\.platformio\python3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\user.name\.platformio\penv\lib\site-packages\platformio\proc.py", line 47, in run
    self.do_reading()
  File "c:\users\user.name\.platformio\penv\lib\site-packages\platformio\proc.py", line 70, in do_reading
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "c:\users\user.name\.platformio\penv\lib\site-packages\platformio\proc.py", line 70, in <lambda>
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "C:\Users\user.name\.platformio\python3\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 170: character maps to <undefined>

Output from pio system info:

--------------------------  --------------------------------------------------------------
PlatformIO Core             5.0.5a1
Python                      3.7.7-final.0
System Type                 windows_amd64
Platform                    Windows-10
File System Encoding        utf-8
Locale Encoding             cp1252
PlatformIO Core Directory   C:\Users\user.name\.platformio
PlatformIO Core Executable  C:\Users\user.name\.platformio\penv\Scripts\platformio.exe
Python Executable           c:\users\user.name\.platformio\penv\scripts\python.exe
Global Libraries            0
Development Platforms       3
Tools & Toolchains          9
--------------------------  --------------------------------------------------------------

Was this ever resolved?

[ivankravets]

Do you have non-ascii chars in your account name?

[stisa]

No, only a-z and a dot, like name.surname

[ivankravets]

Does your FULL project path contain NON-ASCII chars?

[stisa]

No, the full path should be: L:\name.surname\Documents\Marlin-2.0.x, where L is a networked drive, but no non-ascii chars in that either.

[ivankravets]

Could you try to move a project to C: disk? What is a type of L: drive?

[stisa]

Moving the project to C: works, thanks!

I'm not sure what's the exact type of L:, it's a folder on a server, which then get mapped to act as my home folder, but the configuration is done by another department so I don't know more.

The only difference I can think of is that sometimes the path can become \\{some ip address}\name.surname\Documents\Marlin-2.0.x , but I doubt the \\ at the beginning would cause this issue, right?

[ivankravets]

Please re-test with the latest development version of PIO Core

pio upgrade --dev

[stisa]

did a quick test, seems to work, thanks!