Mbed-os: Mbed doesn't work with Python 3.8 if pip install -U mbed-cli is done

[greglandry]

Description of defect

If I install Mbed CLI and Python 3.8, I can compile Mbed projects (using Mbed OS 5.15.1).

However, if I run "pip install -U mbed-cli" to update the Mbed CLI, the compile then starts to fail with the following error:

The Mbed OS tools in this program require the following Python modules: pyyaml, jsonschema, mbed_cloud_sdk, jinja2, mbed_ls, mbed_host_tests, mbed_greentea, pyelftools, manifest_tool, icetea, pycryptodome, pyusb, hidapi, cmsis_pack_manager, pywin32, wmi, psutil, cryptography, click
You can install all missing modules by running "pip install -r requirements.txt" in "C:\Users\Greg\Documents\Mbed\mbed-os-example-blinky\mbed-os"

Running "pip install -r requirements.txt" from mbed-os as recommended gives another error message:

ERROR: Could not find a version that satisfies the requirement pywin32==224 (from -r requirements.txt (line 26)) (from versions: 225, 226, 227)
ERROR: No matching distribution found for pywin32==224 (from -r requirements.txt (line 26))

The same set of operations using Python 3.7.7 works with no errors.

Target(s) affected by this defect ?

Tested on CY8CKIT_062_WIFI_BT but should affect others.

Toolchain(s) (name and version) displaying this defect ?

GCC_ARM

What version of Mbed-os are you using (tag or sha) ?

mbed-os-5.15.1

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

mbed-cli v0.4.10
python 3.8.2
Visual C++ build tools

How is this defect reproduced ?

On a Windows 10 machine:
Install mbed cli 0.4.10
Install Python 3.8.2 and add to the Windows path before other Python versions (including Scripts folder)
Install Visual C++ build tools if not already installed
Run "mbed import mbed-os-example-blinky"
Update mbed-os ("mbed update mbed-os-5.15.1")
Compile project ("mbed compile -m CY8CKIT_062_WIFI_BT -t GCC_ARM") (no error message)
Run "pip install -U mbed-cli"
Compile project (mbed compile -m CY8CKIT_062_WIFI_BT -t GCC_ARM) (error message)
Run "pip install -r requirements.txt" from mbed-os folder (error message)

[0xc0170]

cc @ARMmbed/mbed-os-tools

[mark-edgeworth]

Installing the right set of non-conflicting python requirements is a real nightmare I'm afraid. We are using Python 3.7 and strongly recommend the use of pipenv to create a virtual environment to run the mbed CLI tools.

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOTRIAGE-2599

[0xc0170]

Installing the right set of non-conflicting python requirements is a real nightmare I'm afraid. We are using Python 3.7 and strongly recommend the use of pipenv to create a virtual environment to run the mbed CLI tools.

Is that how to resolve this, and ask for a closure of this issue?

[mark-edgeworth]

That is certainly the only sensible advice that I can offer at the moment, so yes.

[ladislas]

for people not familiar with python, does the documentation explain how to set the pipenv?

[mark-edgeworth]

I thought there was, but I have been unable to find it so far, so I'm probably wrong. I'm looking to get this updated through our docs team now.

[mark-edgeworth]

Summary is 'install pipenv for python 3.7; drop into the pipenv shell; run pip install mbed-cli'. mbed commands should then be available inside that shell, and the installation won't pollute the external python installation (nor will it conflict).

[greglandry]

Since the recommended version of python is 3.7, are there any plans to update the Mbed CLI installer to include Python 3.7 instead of 2.7?

[mark-edgeworth]

We are working on a potential replacement for the mbed OS build system for the forthcoming mbed 6 release. I expect there will be a replacement installer for this.

[ladislas]

We are working on a potential replacement for the mbed OS build system for the forthcoming mbed 6 release.

@mark-edgeworth that sounds really exciting! can we learn more? can we help? can we beta test?

[greglandry]

If I create a new pipenv shell (on a Windows machine) and run "pip install mbed-cli", it doesn't install mercurial or git. Is that expected behavior? Do they need to be installed separately?

[madchutney]

Yes, pipenv only handles the Python part of the install. There is some help in the following page, but please don't follow section _2. Install Mbed CLI_ as this is covered by the pipenv install you have already completed.

https://os.mbed.com/docs/mbed-os/v5.15/tools/manual-installation.html

[0xc0170]

I believe this can be closed. Using python 3.7 for now. We will update to a newer version (will require an update in requirements as well).