Arduino: Error building on MacOS due to python3 not found

[harveydobson]

Basic Infos

• [x] This issue complies with the issue POLICY doc.
• [x] I have read the documentation at readthedocs and the issue is not addressed there.
• [x] I have tested that the issue is present in current master branch (aka latest git).
• [x] I have searched the issue tracker for a similar issue.
• [x] If there is a stack dump, I have decoded it.
• [x] I have filled out all fields below.

Platform

• Hardware: [ESP8266-12e]
• Core Version: [21/12/2019 12:10pm GMT]
• Development Env: [Arduino IDE]
• Operating System: [MacOS]

Settings in IDE

Problem Description

Edit by @devyte:
The problem seems to be that the build process doesn't find python3 on MacOS. Duplicate WiFi libs warning is unrelated.

Original Problem Description follows.
--------------------------------------------------------------------------------------------------------------------

When including the library and selecting the board, any code i use whether it is the wifi search or the code I was looking to use in my project, I get the following error:

Arduino: 1.8.10 (Mac OS X), Board: "NodeMCU 0.9 (ESP-12 Module), 80 MHz, Flash, Legacy (new can return nullptr), All SSL ciphers (most compatible), 4MB (FS:2MB OTA:~1019KB), v2 Lower Memory, Disabled, None, Only Sketch, 115200"

Multiple libraries were found for "ESP8266WiFi.h"
 Used: /Users/xxxx/Library/Arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266WiFi
fork/exec /Users/xxxx/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3: no such file or directory
Error compiling for board NodeMCU 1.0 (ESP-12E Module).

This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

If i remove the library it finds nothing. This is a new installation on my mac so there wouldn't be any conflicts. Not sure if anyone has experienced the same..? Seems likely unless i'm doing something very wrong, but looking at the current issue list I cannot see anything like this.

Thanks

MCVE Sketch

/*
    This sketch demonstrates how to scan WiFi networks.
    The API is almost the same as with the WiFi Shield library,
    the most obvious difference bei›‹‹›ng the different file you need to include:
*/

#include "ESP8266WiFi.h"

void setup() {
  Serial.begin(115200);

  // Set WiFi to station mode and disconnect from an AP if it was previously connected
  WiFi.mode(WIFI_STA);
  WiFi.disconnect();
  delay(100);

  Serial.println("Setup done");
}

void loop() {
  Serial.println("scan start");

  // WiFi.scanNetworks will return the number of networks found
  int n = WiFi.scanNetworks();
  Serial.println("scan done");
  if (n == 0) {
    Serial.println("no networks found");
  } else {
    Serial.print(n);
    Serial.println(" networks found");
    for (int i = 0; i < n; ++i) {
      // Print SSID and RSSI for each network found
      Serial.print(i + 1);
      Serial.print(": ");
      Serial.print(WiFi.SSID(i));
      Serial.print(" (");
      Serial.print(WiFi.RSSI(i));
      Serial.print(")");
      Serial.println((WiFi.encryptionType(i) == ENC_TYPE_NONE) ? " " : "*");
      delay(10);
    }
  }
  Serial.println("");

  // Wait a bit before scanning again
  delay(5000);
}

Debug Messages

Arduino: 1.8.10 (Mac OS X), Board: "NodeMCU 0.9 (ESP-12 Module), 80 MHz, Flash, Legacy (new can return nullptr), All SSL ciphers (most compatible), 4MB (FS:2MB OTA:~1019KB), v2 Lower Memory, Disabled, None, Only Sketch, 115200"

Multiple libraries were found for "ESP8266WiFi.h"
 Used: /Users/xxxx/Library/Arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266WiFi
fork/exec /Users/xxxx/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3: no such file or directory
Error compiling for board NodeMCU 1.0 (ESP-12E Module).

This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

[jandrassy]

the "Multiple libraries were found for " is a bug in Arduino IDE (builder). it reports it at any error.
your true error is tools/python3/3.7.2-post1/python3: no such file or directory

[harveydobson]

the "Multiple libraries were found for " is a bug in Arduino IDE (builder). it reports it at any error.
your true error is tools/python3/3.7.2-post1/python3: no such file or directory

Thanks, I will investigate further!

[jandrassy]

on arduino forum was discussed that the python3 link is wrong on Mac and should be set to correct path
https://forum.arduino.cc/index.php?topic=649739.msg4401959#msg4401959

@earlephilhower, do you know about the problem?

[earlephilhower]

@jandrassy I think the problem is that there are now 2 spots where python3 may be on the Mac. We're actually building on MacOS as part of CI, now, and when homebrew installs python3, it's in the directory linked in the tarball.

Maybe there was an OS update/release recently where python3 is already installed? MacOS in CI (sorry, no idea what rev but a Mac guy might be able to look at any build log and tell from any PR) only had the obsolete Python2 installed under /usr/bin/python and no Python3 at all (hence the homebrew installation). Can anyone with a Mac confirm?

[liebman]

I have an up to date Mac 10.15.2 and there is no python3 in /usr/bin. Mine is installed via brew and its located in /usr/local/bin.

[earlephilhower]

Thanks, @liebman . So it looks like on an updated Mac the link in the tarball is correct (at least when brew is used).

@harveydobson, how did you install Python3 on your Mac? brew or something else?

[tonywhitfort]

I get the same error (no such file or directory)
MacOS 10.15.2
ESP V2.6.3
python3 is in /usr/bin
I have updated the alias to point to this folder and it now works

[earlephilhower]

@tonywhitfort , how did you get Python3 on your Mac? Is it now part of the standard distro, did you download a standalone installer, use brew, or something else?

[tonywhitfort]

I didn’t install Python3 myself it was possibly installed by one of these

Eclipse IDE (which I installed earlier this year for some Arduino projects)
Platform IO and also Xcode (which I installed earlier this year but haven’t used since)

Python3 version is 3.7.3 dated 15 Nov 2019, so something is keeping it up to date.

I installed MacOS Catalina as an upgrade, I haven’t done a clean install for years

On 29 Dec 2019, at 5:27 am, Earle F. Philhower, III [e-mail] wrote:

@tonywhitfort https://github.com/tonywhitfort , how did you get Python3 on your Mac? Is it now part of the standard distro, did you download a standalone installer, use brew, or something else?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/esp8266/Arduino/issues/6931?email_source=notifications&email_token=AE3IZCW5XHWRME2SRWA3MWLQ26LCTA5CNFSM4J6G5DOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHYPTLQ#issuecomment-569440686, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3IZCVAUPJDXSVQE7HTIN3Q26LCTANCNFSM4J6G5DOA.

[farthinder]

Just chiming in, I have this problem as well on my Mac OS.
I cant honestly say where I got my python installation from but likely it came with MacOs or I got it via brew.

Currently /Users/USER/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3 points at /usr/local/bin/python3

But it should point at /usr/bin/python3
However, creating such a symlink seems problematic: https://stackoverflow.com/questions/36730549/cannot-create-a-symlink-inside-of-usr-bin-even-as-sudo

[farthinder]

My workaround that seems to work:

cd ~/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1
sudo unlink python3
sudo ln -s /usr/bin/python3 /usr/local/bin/python3
sudo ln -s /usr/local/bin/python3 python3

[earlephilhower]

<rant>What is Apple thinking, keeping only Python 2 in the latest OS</rant>

Anyway, does MacOS have /usr/bin/env and does it work with brew installed P3 as well as P3 direct from python.org? I'm thinking a wrapper script could be written using /usr/bin/env python3 and letting the OS handle disambiguation.

[liebman]

I have an up to date Mac 10.15.2 and there is no python3 in /usr/bin. Mine is installed via brew and its located in /usr/local/bin.

UPDATE: I've updated xcode and now I have a python3 in /usr/bin. (and one from brew in /usr/local/bin)

@earlephilhower - yes /usr/bin/env exists and is functional on MacOS.

[earlephilhower]

@farthinder, @tonywhitfort , others with /usr/bin/python3 : Can you please try #6960 and report?

You'll need to re-run the get.py to retrive the new python3 redirect stuff?

You'll want to remove any symlinks you made in /usr/bin/local to make the old version work, of course. This new one uses the env command to find the proper python3 to call and should work on Mac and Linux as well.

[farthinder]

@earlephilhower I happened to be setting up a new mac so I gave this a try to the best of my abilities.

First of, on a clean mac when running python3 you get asked if you want to install "command line dev tools" which I declined at first. Then when cloning your pullrequest #6960 I got the same dialogbox when running git so I allowed the "command line dev tools" to get installed.
I cloned the repo, fetched the pull request and initilized the submodules then when running python3 get.py I get a cert warning:

esp8266 % git fetch origin pull/6960/head:pythonfix
esp8266 % git submodule update --init
esp8266 % git checkout pythonfix
esp8266 % cd tools
tools % python3 get.py
Platform: x86_64-apple-darwin
Downloading python3-macosx-placeholder.tar.gz
Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
server_hostname=server_hostname)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
session=session
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create
self.do_handshake()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "get.py", line 131, in
main()
File "get.py", line 128, in main
get_tool(tool)
File "get.py", line 80, in get_tool
urlretrieve(url, local_path, report_progress)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 247, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError:

[earlephilhower]

That's a MacOS python problem, unfortunately, and not much we can do. When Python3 is installed (any way) it doesn't register the SSL certificates needed to actually do any internet access and dumps the error you're seeing. You'll have the same problem running any other Python3 wget-like thing that needs HTTPS.

The solution is included in the Python3 readme, or should be if it's been repackaged.

( cd "/Applications/Python 3.7/" && sudo "./Install Certificates.command" )

You're going to have to adjust the CD to wherever you've got the Python3 directory installed. That script from the python team does the registration of those certs. After that, https connections in Python should work.

[fabiradi]

My workaround that seems to work:

cd ~/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1
sudo unlink python3
sudo ln -s /usr/bin/python3 /usr/local/bin/python3
sudo ln -s /usr/local/bin/python3 python3

Workaround without admin permissions (i.e. in a corporate environment):

cd ~/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1
unlink python3
ln -s /usr/bin/python3 python3

The symlink can be created without the need for sudo.

[tohqecom]

the first you must run command to identify your current python:
python --version => Python 3.7.7
which python
and ressult like this:

/usr/local/opt/python/libexec/bin/python
cd ~/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1
unlink python3
ln -s /usr/local/opt/python/libexec/bin/python ~/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3
Just that!

[michelcmorel]

cd ~/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1
sudo unlink python3
sudo ln -s /usr/bin/python3 /usr/local/bin/python3
sudo ln -s /usr/local/bin/python3 python3

After 12 hrs of headaches trying to figure out how to bypass iMac SIP, so to get python3 path to work with my Wemos d1.. This worked like a charm! Thank you so much.

[davecramer]

I get the same error and when I run
/Users/davec/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3
it runs fine

[danbates2]

This seems to be still an issue?

Latest versions of all IDE and ESP8266 core except I run High Sierra.

cd ~/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1
sudo unlink python3
sudo ln -s /usr/bin/python3 /usr/local/bin/python3
sudo ln -s /usr/local/bin/python3 python3

this works

[michelcmorel]

This seems to be still an issue?

Latest versions of all IDE and ESP8266 core except I run High Sierra.

cd ~/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1
sudo unlink python3
sudo ln -s /usr/bin/python3 /usr/local/bin/python3
sudo ln -s /usr/local/bin/python3 python3

this works

No, it's no longer an issue for me at least.
Farthinder's answer worked, I'm grateful for his post.

[danbates2]

Yes but I updated the esp core from 2.7.0 to 2.7.1 and the problem resurfaced.
No big deal for most I'm sure.

[davecramer]

Mine was an update as well. I had to remove something else to get it to work. Sad that macos is still shipping python 2.

[danbates2]

High Sierra was released September 25, 2017.
Catalina, October 7, 2019, I read now has the placeholder for the python3 bins, so it's easier to integrate using command line tools.
Yeah, it's unfortunate it's not been included by default in recent releases.

I don't know if it's worth accommodating the brew installed python3 package. They're pretty common. The symlink is at /usr/local/bin/python3 right.

[blazoncek]

I have python3 installed by HomeBrew and the issue resurfaces every time I update ESP8266 board.
To fix it I run:
cd ~/Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1
unlink python3
ln -s /usr/local/bin/python3 python3

[hellvesper]

I propose to change default python3 location to /usr/bin/python3 as this is system python3 location