Pyenv: Outdated tips for macOS 10.15 or later

[smorimoto]

When running Mojave or higher (10.14+) you will also need to install the additional SDK headers by downloading them from Apple Developers. You can also check under /Library/Developer/CommandLineTools/Packages/ as some versions of Mac OS will have the pkg locally.

https://github.com/pyenv/pyenv/wiki/Common-build-problems

See this comment: https://github.com/Homebrew/homebrew-core/pull/45304#issuecomment-541637112

[joshfriend]

That's not correct, installing the legacy SDK headers is not a good solution since that package will likely be removed soon. The correct fix is to properly set the include paths to what xcrun tells you, as described to you earlier by this comment on the PR you linked: https://github.com/Homebrew/homebrew-core/pull/45304#issuecomment-541635570

The current version of pyenv uses xcrun to find this SDK path already.

[smorimoto]

@joshfriend Oh, I wanted to say that this guide should be deleted because it only causes confusion. I salute pyenv for addressing this issue early, unlike other tools. but, still failed to build on Catalina. Do you have any ideas?

Installing Python-3.8.0...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 10.15 using python-build 1.2.14-3-g8bf5d9d3)

Inspect or clean up the working tree at /var/folders/sm/qls397w14hgb7rw38_vrlv_c0000gn/T/python-build.20191017002218.45755
Results logged to /var/folders/sm/qls397w14hgb7rw38_vrlv_c0000gn/T/python-build.20191017002218.45755.log

Last 10 log lines:
checking size of float... 0
checking size of double... 0
checking size of fpos_t... 0
checking size of size_t... 0
checking size of pid_t... 0
checking size of uintptr_t... 0
checking for long double... yes
checking size of long double... configure: error: in `/var/folders/sm/qls397w14hgb7rw38_vrlv_c0000gn/T/python-build.20191017002218.45755/Python-3.8.0':
configure: error: cannot compute sizeof (long double)
See `config.log' for more details

(config.log was empty...)

[smorimoto]

Solved with this.

SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk"

CFLAGS="-I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include" 

CPPFLAGS="-I/usr/local/opt/zlib/include"

LDFLAGS="-L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib"

[greatvovan]

I am experiencing the same error, the parameters from @imbsky don't help. Any other ideas?

[smorimoto]

@greatvovan I don't have any information about your system environment, so I'm wondering how to advise you.

[smorimoto]

Have you installed these packages?

brew install readline xz openssl zlib

[smorimoto]

I fixed the comment because there was a strange line break.

[smorimoto]

env SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" CFLAGS="-I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include" CPPFLAGS="-I/usr/local/opt/zlib/include" LDFLAGS="-L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib" pyenv install 3.8.0

[greatvovan]

@imbsky sorry, forgot to say that the environment is similar to yours, that's why I am here.
I recently upgraded OS to Catalina, installed command-line tools with xcode-select --install, installed dependent packages with brew (zlib is available in xcode sdk).

But... I just tested it again with you variables in it worked! So did Python 3.7.5 (that was failing with another error). Probably previous time I improperly copied them to the terminal...

Now I am wondering what is the best way to store these variables as the default. Is it for something suitable for ~/.bash_profile or there's a better way?

[smorimoto]

If you know what is CFLAGS, CPPFLAGS, and LDFLAGS, you know that it's not good to store these variables in a place like .bash_profile. But, in my opinion, there is no problem that store SDKROOT variable.

[smorimoto]

@greatvovan I don't know what error did you get so I don't know what to do but I may be able to help if you can show them.

[greatvovan]

@imbsky for Python 3.8 the error was the same as yours: checking size of long double... configure: error: in '/var/folders/.../Python-3.8.0': configure: error: cannot compute sizeof (long double). For Python 3.7.* the output was:

$ pyenv install 3.7.5
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.5.tar.xz...
-> https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tar.xz
Installing Python-3.7.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 10.15.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/4f/bbj3wtw95b9_d92xxtgl1v_m0000gn/T/python-build.20191209102358.47012
Results logged to /var/folders/4f/bbj3wtw95b9_d92xxtgl1v_m0000gn/T/python-build.20191209102358.47012.log

Last 10 log lines:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h:540:6: warning: macro expansion producing 'defined' has undefined behavior [-Wexpansion-to-defined]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h:200:34: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
        defined(SWIFT_CLASS_EXTRA) && (!defined(SWIFT_SDK_OVERLAY_PTHREAD_EPOCH) || (SWIFT_SDK_OVERLAY_PTHREAD_EPOCH < 1))
                                        ^
218 warnings and 10 errors generated.
218 warnings and 10 errors generated.
make: *** [Parser/listnode.o] Error 1
make: *** [Programs/python.o] Error 1
218 warnings and 10 errors generated.
make: *** [Parser/grammar1.o] Error 1

But adding your env vars helped as well.

I can guess that these flags control compilation parameters and namely -I appends additional include paths, but I am not aware why it is bad to put it in Bash profile and what the best practices are. I just want my build tools to work in all standard cases.

[fr0der1c]

I'm running pyenv 1.2.15 and encountered this problem. Is the env virable the best solution we can get?

[smorimoto]

It's not a common definition, so it shouldn't be in a profile. You should either fix the Makefile or configure file or create a shell script for the build that defines flags. (Well, I don't know how python-build works.) and this is only for Python build, in my opinion it's sufficient to use the env command to set the environment variables temporarily.

[saadali5997]

env SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" CFLAGS="-I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include" CPPFLAGS="-I/usr/local/opt/zlib/include" LDFLAGS="-L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib" pyenv install 3.8.0

It's not working for me. :( I can see ssl installed at that path but it still says missing.
I am trying to install python 3.5.2. MacOS Catalina.

[jpmelos]

I also can't install Python with pyenv in my system, even following all instructions in this issue. I get the same errors as other people, more specifically:

ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib?

Please consult to the Wiki page to fix the problem.
https://github.com/pyenv/pyenv/wiki/Common-build-problems


BUILD FAILED (OS X 10.15.2 using python-build 1.2.16-5-g7097f820)

Inspect or clean up the working tree at /var/folders/3v/ql_4tqq17c728pk0nj0c_dvh0000gn/T/python-build.20200127113220.27139
Results logged to /var/folders/3v/ql_4tqq17c728pk0nj0c_dvh0000gn/T/python-build.20200127113220.27139.log

Last 10 log lines:
rm -f /Users/joao.sampaio/.pyenv/versions/2.7.12/share/man/man1/python.1
(cd /Users/joao.sampaio/.pyenv/versions/2.7.12/share/man/man1; ln -s python2.1 python.1)
if test "xno" != "xno"  ; then \
        case no in \
            upgrade) ensurepip="--upgrade" ;; \
            install|*) ensurepip="" ;; \
        esac; \
         ./python.exe -E -m ensurepip \
            $ensurepip --root=/ ; \
    fi

Does anyone know what the actual solution looks like?

[kimsean]

I am also having this problem. on Catalina 10.15

BUILD FAILED (OS X 10.15.3 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/34/9v1b_7zd6fb14kx36fggp6940000gn/T/python-build.20200204011011.13508
Results logged to /var/folders/34/9v1b_7zd6fb14kx36fggp6940000gn/T/python-build.20200204011011.13508.log

Last 10 log lines:
(cd /Users/devkim/.pyenv/versions/3.5.1/share/man/man1; ln -s python3.5.1 python3.1)
if test "xupgrade" != "xno"  ; then \
        case upgrade in \
            upgrade) ensurepip="--upgrade" ;; \
            install|*) ensurepip="" ;; \
        esac; \
         ./python.exe -E -m ensurepip \
            $ensurepip --root=/ ; \
    fi

[hcaadmin]

@jpmelos / @kimsean - This was a major pain.

Basically, I believe that the issue is that only certain versions of Python have OpenSSL 1.1, while others have OpenSSL 1.0.2.

And Homebrew installs now OpenSSL 1.1.

So, what I found is the solutions above might work for the version of Python you want to install, or they might not, depends on the minor version of python. For example, 2.7.13 will work with the brew version of OpenSSL following the solution written above, but 2.7.10 will not.

I infer the root problem from several posts online:
This GitHub bug report: https://github.com/pyenv/pyenv/issues/945

OpenSSL 1.1.0 support in the ssl module was only added to Python 2.7.13, 3.5.3 and 3.6.0 ...

and

https://bugs.python.org/issue26470

To get pyenv to install 2.7.10 (legacy project I inherited), I had to download and compile OpenSSL 1.0.2 (https://stackoverflow.com/questions/38670295/homebrew-refusing-to-link-openssl - go to the Manually install and symlink answer).

Here is are the steps that I did (basically copy of that text with out the symlink):

Manually install and symlink

cd /usr/local/src
If you're getting "No such file or directory", make it:

cd /usr/local && mkdir src && cd src
Download openssl:

curl --remote-name https://www.openssl.org/source/openssl-1.0.2h.tar.gz
Extract and cd in:

tar -xzvf openssl-1.0.2h.tar.gz
cd openssl-1.0.2h

Compile and install:

./configure darwin64-x86_64-cc --prefix=/usr/local/openssl-1.0.2h shared
make depend
make
make install

Then I ended up needing to uninstall brew's openssl to get the pyenv to see it
brew uninstall --ignore-dependencies openssl

Then I set the following:

export CFLAGS="-I/usr/local/openssl-1.0.2h/include -I/usr/local/openssl-1.0.2h/include/openssl $CFLAGS"
export CPPFLAGS="-I/usr/local/openssl-1.0.2h/include -I/usr/local/openssl-1.0.2h/include/openssl $CPPFLAGS"
export LDFLAGS="-L/usr/local/openssl-1.0.2h/lib $LDFLAGS"
export PATH=/usr/local/openssl-1.0.2h/bin:$PATH
export PKG_CONFIG_PATH=/usr/local/openssl-1.0.2h/lib/pkgconfig
export LD_LIBRARY_PATH="/usr/local/openssl-1.0.2h/lib"

then ran
PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install -v 2.7.10

That built and install v 2.7.10

> pyenv shell 2.7.10
> python -V
Python 2.7.10
j> python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.0.2k  26 Jan 2017

Then I reinstalled openssl with brew and switched the CFLAGS,CPPFLAGS and LDFLAGS back to the brew OpenSSL 1.1 paths and was able to install python 3.8.0:

> ~ % pyenv shell 3.8.0
> ~ % python -V
Python 3.8.0
> ~ % python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.1.1d  10 Sep 2019

Hope that helps!

[hcaadmin]

I am able to confirm that was able to get pyenv install -v 2.7.13 to build and install on MacOSX 10.15.3 with the following:

SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" \
CFLAGS="-I$(brew --prefix openssl)/include/include -I$(brew --prefix openssl)/include/openssl -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include" \
CPPFLAGS="-I$(brew --prefix openssl)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib" \
pyenv install -v 2.7.13
......
......
if test "xno" != "xno"  ; then \
        case no in \
            upgrade) ensurepip="--upgrade" ;; \
            install|*) ensurepip="" ;; \
        esac; \
         ./python.exe -E -m ensurepip \
            $ensurepip --root=/ ; \
    fi
/var/folders/0v/gdh0w7m941s_nr399b7rfkl00000gn/T/python-build.20200204195918.21852 ~
~
Installed Python-2.7.13 to /Users/******/.pyenv/versions/2.7.13

Here is the output of the test

> ~ % pyenv shell 2.7.13
> ~ % python -V         
Python 2.7.13
> ~ % python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.1.1d  10 Sep 2019

[joshfriend]

Of the numerous versions of python2.7 i have installed, 2.7.15 and higher were build with openssl 1.1. Is there something preventing you from updating to one of those versions that might be compatible? Once you get past the SSL changes in 2.7.9, I don't know of any dealbreakers you would encounter in newer patch versions

At this point, python2 is dying. While I completely understand that you might still need to use it, I'm also not interested in trying to keep old patch versions of a legacy python version on life support.

[hcaadmin]

Of the numerous versions of python2.7 i have installed, 2.7.15 and higher were build with openssl 1.1. Is there something preventing you from updating to one of those versions that might be compatible? Once you get past the SSL changes in 2.7.9, I don't know of any dealbreakers you would encounter in newer patch versions

Thanks @joshfriend for the response and thank you for pyenv, really appreciate your work there.

To be candid, I don't know yet.

I just basically inherited some 7+ year old code (developed circa 2010, but some additions, no real update since 2016). Since I've never actually, ever developed in python (20 years of C/C++, Java, JavaScript, ObjC, Swift, etc, etc) so I'm not sure what version of python I needed for the project to work. I'm testing today with 2.7.13 and to see if that works with the old code.

At this point, python2 is dying. While I completely understand that you might still need to use it, I'm also not interested in trying to keep old patch versions of a legacy python version on life support.

Completely understandable, and I agree with you 100%. My ignorance of the python ecosystem is probably showing a bit with me trying to force 2.7.10 instead of upgrading to a more supported 2.7.x.

Thought I would put that comment up there so that people who come across this might benefit from understanding that specific versions of Python will have trouble installing on 10.13.x and others will not.

Thank you again!

[stefanitsky]

@hcaadmin thank you so much! It works for <= 3.5.0 also.

[oleksii-haponiuk]

@hcaadmin
macOS 10.15.4
When I try to create folder in /usr/local, even with sudo, I get this. I definitely have admin permission and there's no src folder already.

➜ sudo cd /usr/local && mkdir src && cd src
Password:
mkdir: src: Permission denied

/usr/local  took 2s

By the way, about different python version -- I need 3.4.x, I don't care which one. I tried 3.4.7, 3.4.10, maybe something more, it's all the same. Should I try some other version, like 3.4.1, 3.4.2 and so on?

[xingfanxia]

@Sarasei sudo doesn't chain to ands; sudo su and run the commands

[lylel]

@jpmelos / @kimsean - This was a major pain.

Basically, I believe that the issue is that only certain versions of Python have OpenSSL 1.1, while others have OpenSSL 1.0.2.

And Homebrew installs now OpenSSL 1.1.

So, what I found is the solutions above might work for the version of Python you want to install, or they might not, depends on the minor version of python. For example, 2.7.13 will work with the brew version of OpenSSL following the solution written above, but 2.7.10 will not.

I infer the root problem from several posts online:
This GitHub bug report: #945

OpenSSL 1.1.0 support in the ssl module was only added to Python 2.7.13, 3.5.3 and 3.6.0 ...

and

https://bugs.python.org/issue26470

To get pyenv to install 2.7.10 (legacy project I inherited), I had to download and compile OpenSSL 1.0.2 (https://stackoverflow.com/questions/38670295/homebrew-refusing-to-link-openssl - go to the Manually install and symlink answer).

Here is are the steps that I did (basically copy of that text with out the symlink):

Manually install and symlink

cd /usr/local/src
If you're getting "No such file or directory", make it:

cd /usr/local && mkdir src && cd src
Download openssl:

curl --remote-name https://www.openssl.org/source/openssl-1.0.2h.tar.gz
Extract and cd in:

tar -xzvf openssl-1.0.2h.tar.gz
cd openssl-1.0.2h

Compile and install:

./configure darwin64-x86_64-cc --prefix=/usr/local/openssl-1.0.2h shared
make depend
make
make install

Then I ended up needing to uninstall brew's openssl to get the pyenv to see it
brew uninstall --ignore-dependencies openssl

Then I set the following:

export CFLAGS="-I/usr/local/openssl-1.0.2h/include -I/usr/local/openssl-1.0.2h/include/openssl $CFLAGS"
export CPPFLAGS="-I/usr/local/openssl-1.0.2h/include -I/usr/local/openssl-1.0.2h/include/openssl $CPPFLAGS"
export LDFLAGS="-L/usr/local/openssl-1.0.2h/lib $LDFLAGS"
export PATH=/usr/local/openssl-1.0.2h/bin:$PATH
export PKG_CONFIG_PATH=/usr/local/openssl-1.0.2h/lib/pkgconfig
export LD_LIBRARY_PATH="/usr/local/openssl-1.0.2h/lib"

then ran
PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install -v 2.7.10

That built and install v 2.7.10

> pyenv shell 2.7.10
> python -V
Python 2.7.10
j> python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.0.2k  26 Jan 2017

Then I reinstalled openssl with brew and switched the CFLAGS,CPPFLAGS and LDFLAGS back to the brew OpenSSL 1.1 paths and was able to install python 3.8.0:

> ~ % pyenv shell 3.8.0
> ~ % python -V
Python 3.8.0
> ~ % python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.1.1d  10 Sep 2019

Hope that helps!

Thank you so much, this worked perfectly for python 3.5.2.

[QYQSDTC]

Does anyone know what's this issue? I am lacking a module '_scproxy'

[joshfriend]

@QYQSDTC please search the issues list, there have been several existing threads about _scproxy specifically.

https://github.com/pyenv/pyenv/issues/521
https://github.com/pyenv/pyenv/issues/1107

[masonlouchart]

Nothing worked for me. I did the following to solve this issue:

• installed macports
• sudo port -v selfupdate
• sudo port install openssl
• sudo port install cctools

Then I could install Python 2.7.17 on macOS 10.15.4 with pyenv and create a virtualenv for my project and install all requirements of this one.

Be sure to have this variable /opt/local/bin:/opt/local/sbin: in your $PATH. See the The Postflight Script section of the MacPorts Guide.

[ryanb8]

Tacking on to @imbsky 's answer, I added this function to my bash profile so that I can run pyenv install like normal and it will auto-set the env vars for my builds to compile:

pyenv() {
    if [[ $1 == "install" ]]; then
        command env SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" CFLAGS="-I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include" CPPFLAGS="-I/usr/local/opt/zlib/include" LDFLAGS="-L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib" pyenv install "${@:2}"
    else
        command pyenv "$@"
    fi
}