It doesn't build with Xcode 11.5 either.

bjackson on 27 Jun 2020

@bjackson on macOS Catalina or the Big Sur beta?

Sjors on 27 Jun 2020

Big Sur beta

bjackson on 27 Jun 2020

Same here - Big Sur beta, Xcode 12.

pyenv install 3.8.3 fails as above.

Tried setting SDKROOT to 10.15 and 10.16, neither was successful.

./configure && make are successful (with cpython source from python.og) but the readline module does not get built. Wonder if this might have to do with readline?? (Def not an expert ;-)

Will update if I get it working.

bryanmobrien on 28 Jun 2020

So, still struggling with pyenv on BIg Sur (obv not a pressing issue for me). However, to add to this thread, I attempted to install python from homebrew. With:

brew install python --HEAD

I got the following error:

==> Pouring pkg-config-0.29.2_3.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/pkg-config/0.29.2_3: 11 files, 623.7KB
==> Installing python --HEAD
Error: An exception occurred within a child process:
  Utils::InreplaceError: inreplace failed
setup.py:
  expected replacement of "do_readline = self.compiler.find_library_file(lib_dirs, 'readline')" with "do_readline = '/usr/local/opt/readline/lib/libhistory.dylib'"

I tried to install vanilla python from homebrew (sans --HEAD) and homebrew python installed without error.

```==> Pouring python-3.7.7.catalina.bottle.tar.gz
==> /usr/local/Cellar/python/3.7.7/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/
==> /usr/local/Cellar/python/3.7.7/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/
==> /usr/local/Cellar/python/3.7.7/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/
==> Caveats
Python has been installed as
/usr/local/bin/python3

Unversioned symlinks python, python-config, pip etc. pointing to
python3, python3-config, pip3 etc., respectively, have been installed into
/usr/local/opt/python/libexec/bin

You can install Python packages with
pip3 install
They will install into the site-package directory
/usr/local/lib/python3.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺 /usr/local/Cellar/python/3.7.7: 4,006 files, 61.2MB
```
I really think (since homebrew install with --HEAD option and pyenv install both try to install from source, and both fail with readline error messages) there might be an issue with the readline module and Big Sur. I don't have the technical skills to absolutely determine this, but it is fishy. Wondering, @Sjors , if you are able to install python via homebrew (first attempt with --HEAD) and then attempt without --HEAD.

bryanmobrien on 1 Jul 2020

I tried the homebrew binary back then and it worked. IIRC having Homebrew build from source resulted in the same problem as PyEnv had. We should probably keep an eye on https://github.com/Homebrew/brew/issues/7857

Sjors on 1 Jul 2020

I added #undef _POSIX_C_SOURCE right before #include <sys/socket.h> in Modules/posixmodule.c to get things compiling.

palmerc on 1 Jul 2020

@palmerc , I'm showing my ignorance here. How to you add #undef to posixmodule.c if pyenv install downloads and compiles python in one step? I'm missing how to interrupt pyenv before it starts compiling the downloaded source code.

bryanmobrien on 2 Jul 2020

@palmerc , I'm showing my ignorance here. How to you add #undef to posixmodule.c if pyenv install downloads and compiles python in one step? I'm missing how to interrupt pyenv _before_ it starts compiling the downloaded source code.

No ignorance - this is obviously me hacking until I could get it to compile. Yes, you'll have to apply a patch to the source after it is installed. I'm not suggesting this is a legitimate solution, simply that the problem is related to the _POSIX_C_SOURCE #define.

What I did to get python on my beta build machine was to build it on a Catalina machine with Xcode 11 and then copy the whole .pyenv directory over (symlinks and all).

palmerc on 2 Jul 2020

I am also facing similar issue with posixmodule
I have tried following options unsuccessfully :(
Please note though I want to install Python 3.8.0, I have tried all 3.8 releases with same result

Setting SDKROOT and MACOSX_DEPLOYMENT_TARGET

SDKROOT=/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
MACOSX_DEPLOYMENT_TARGET=10.16
PYTHON_CONFIGURE_OPTS="--enable-framework"
pyenv install 3.8.0
Setting CFLAGS AND LDFLAGS

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib" pyenv install 3.8.0

% sw_vers 
ProductName:    macOS
ProductVersion: 10.16
BuildVersion:   20A4300b

% clang --version
Apple clang version 12.0.0 (clang-1200.0.22.7)
Target: x86_64-apple-darwin20.0.0
Thread model: posix
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

% pyenv install 3.8.0
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Installing Python-3.8.0...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 10.16 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/xw/srzcqq0j1lqfsm1f23wqp6380000gn/T/python-build.20200708180922.77980
Results logged to /var/folders/xw/srzcqq0j1lqfsm1f23wqp6380000gn/T/python-build.20200708180922.77980.log

Last 10 log lines:
           ^
./Modules/posixmodule.c:9084:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/pritam/.pyenv/versions/3.8.0/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/pritam/.pyenv/versions/3.8.0/include   -DPy_BUILD_CORE_BUILTIN  -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/pritam/.pyenv/versions/3.8.0/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/pritam/.pyenv/versions/3.8.0/include   -DPy_BUILD_CORE_BUILTIN  -c ./Modules/_sre.c -o Modules/_sre.o
2 errors generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
1 warning generated.

pyenv_3_8_0_install.log

PritamDutt on 8 Jul 2020

I would like to share that ... I tried install Python using Homebrew it installed without any errors .. only downside in my case was that it I did not get 3.8.0..

Sharing the output of my session

% brew install [email protected]
Updating Homebrew...
==> Auto-updated Homebrew!
Updated Homebrew from b763d29df to d5123743b.
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
omake
==> Updated Formulae
exploitdb                                                   fastlane                                                    semgrep                                                     wildfly-as
==> Updated Casks
alt-tab                                                                         mos                                                                             qt-creator

Warning: You are using macOS 10.16.
We do not provide support for this pre-release version.
You will encounter build failures with some formulae.
Please create pull requests instead of asking for help on Homebrew's GitHub,
Discourse, Twitter or IRC. You are responsible for resolving any issues you
experience while you are running this pre-release version.

Warning: Building [email protected] from source:
  The bottle needs the Apple Command Line Tools to be installed.
  You can install them, if desired, with:
    xcode-select --install

==> Downloading https://github.com/python/cpython/commit/8ea6353.patch?full_index=1
######################################################################## 100.0%
==> Downloading https://files.pythonhosted.org/packages/df/ed/bea598a87a8f7e21ac5bbf464102077c7102557c07db9ff4e207bd9f7806/setuptools-46.0.0.zip
######################################################################## 100.0%
==> Downloading https://files.pythonhosted.org/packages/8e/76/66066b7bc71817238924c7e4b448abdb17eb0c92d645769c223f9ace478f/pip-20.0.2.tar.gz
######################################################################## 100.0%
==> Downloading https://files.pythonhosted.org/packages/75/28/521c6dc7fef23a68368efefdcd682f5b3d1d58c2b90b06dc1d0b805b51ae/wheel-0.34.2.tar.gz
######################################################################## 100.0%
==> Downloading https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
######################################################################## 100.0%
==> Patching
==> Applying 8ea6353.patch
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3398 (offset -28 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
==> ./configure --prefix=/usr/local/Cellar/[email protected]/3.8.3_1 --enable-ipv6 --datarootdir=/usr/local/Cellar/[email protected]/3.8.3_1/share --datadir=/usr/local/Cellar/[email protected]/3.8.3_1/share --enable-framework=/usr/local/Cellar/[email protected]/3.8.
==> make
==> make install PYTHONAPPSDIR=/usr/local/Cellar/[email protected]/3.8.3_1
==> make frameworkinstallextras PYTHONAPPSDIR=/usr/local/Cellar/[email protected]/3.8.3_1/share/[email protected]
==> /usr/local/Cellar/[email protected]/3.8.3_1/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/[email protected]/3.8.3_1/bin --install-lib=/usr/local/lib/python3.8/site-packages --single-version-exte
==> /usr/local/Cellar/[email protected]/3.8.3_1/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/[email protected]/3.8.3_1/bin --install-lib=/usr/local/lib/python3.8/site-packages --single-version-exte
==> /usr/local/Cellar/[email protected]/3.8.3_1/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/[email protected]/3.8.3_1/bin --install-lib=/usr/local/lib/python3.8/site-packages --single-version-exte
==> Caveats
Python has been installed as
  /usr/local/opt/[email protected]/bin/python3

You can install Python packages with
  /usr/local/opt/[email protected]/bin/pip3 install <package>
They will install into the site-package directory
  /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/[email protected]/3.8.3_1: 8,827 files, 125.1MB, built in 3 minutes 1 second
% brew info [email protected]
[email protected]: stable 3.8.3 (bottled)
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/[email protected]/3.8.3_1 (8,827 files, 125.1MB) *
  Built from source on 2020-07-08 at 20:59:27
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/[email protected]
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, [email protected] ✔, readline ✔, sqlite ✔, xz ✔
==> Caveats
Python has been installed as
  /usr/local/opt/[email protected]/bin/python3

You can install Python packages with
  /usr/local/opt/[email protected]/bin/pip3 install <package>
They will install into the site-package directory
  /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Analytics
install: 372,347 (30 days), 1,149,638 (90 days), 1,723,074 (365 days)
install-on-request: 12,656 (30 days), 34,243 (90 days), 57,756 (365 days)
build-error: 0 (30 days)
% /usr/local/opt/[email protected]/bin/python3
Python 3.8.3 (default, Jul  8 2020, 20:58:19)
[Clang 12.0.0 (clang-1200.0.22.7)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

PritamDutt on 8 Jul 2020

Thank you HomeBrew team for all the hard work you did to find the patch that python team is brewing. I am now able to install Python 3.8.0 using pyenv, and this is how I did it.

I have tested installing versions 3.8.0 and 3.8.3. As

% CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.0 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Installing Python-3.8.0...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3394 (offset -32 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.0 to /Users/pritam/.pyenv/versions/3.8.0

% CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.3.tar.xz...
-> https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
Installing Python-3.8.3...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3398 (offset -28 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.3 to /Users/pritam/.pyenv/versions/3.8.3

% pyenv versions
* system (set by /Users/pritam/.pyenv/version)
  3.6.8
  3.6.9
  3.8.0
  3.8.3

PritamDutt on 8 Jul 2020

👍14 ❤4

@PritamDutt , thank you.
Trying your suggestions gives me the following warning (for 3.8.0):

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.0 to /Users/bobrien/.pyenv/versions/3.8.0

When I tried 3.8.3, again, pasting your script above:

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.3 to /Users/bobrien/.pyenv/versions/3.8.3

Checked homebrew, and it is installed, but not linked.

brew link bzip2

Had no effect. I removed 3.8.3 and reattempted

pyenv install 3.8.3

and got the same warning.

At least, I have python installed on Big Sur via pyenv.

bryanmobrien on 10 Jul 2020

It took a while to get "pyenv install ..." to work under macOS Big Sur with Xcode 12 beta, but I have now build 3.8.3, 3.8-dev, 3.9.0b4 and 3.9-dev. I have also been able to build python via home-brew (i.e. brew install [email protected])

Here is the relevant sections I have added to my ~/.zshrc and ~/.bash_profile:

# HOMEBREW Environment Variables
export HOMEBREW_INSTALL_CLEANUP=TRUE
export HOMEBREW_PREFIX=$(brew --prefix)

export PATH="$HOMEBREW_PREFIX/opt/tcl-tk/bin:$PATH"

# Use PyEnv to set Python Environment
export PYENV_SHELL=zsh
export PYENV_ROOT=$(pyenv root)
export PYENV_VERSION=$(pyenv version-name)
export PYTHONPATH=$PYENV_ROOT/shims

# PyEnv & HOMEBREW Build variables
PYTHON_CONFIGURE_OPTS="--with-tcltk-includes='-I$HOMEBREW_PREFIX/opt/tcl-tk/include'"
export PYTHON_CONFIGURE_OPTS="$PYTHON_CONFIGURE_OPTS --with-tcltk-libs='-L$HOMEBREW_PREFIX/opt/tcl-tk/lib -ltcl8.6 -ltk8.6'"
export CFLAGS="-O2 -I$HOMEBREW_PREFIX/include"
CPPFLAGS="-I$HOMEBREW_PREFIX/opt/sqlite/include -I$HOMEBREW_PREFIX/opt/tcl-tk/include"
CPPFLAGS="$CPPFLAGS -I$HOMEBREW_PREFIX/opt/zlib/include"
CPPFLAGS="$CPPFLAGS -I$HOMEBREW_PREFIX/opt/bzip2/include"
export CPPFLAGS="$CPPFLAGS -I$HOMEBREW_PREFIX/opt/[email protected]/include"
LDFLAGS="-L$HOMEBREW_PREFIX/opt/sqlite/lib -L$HOMEBREW_PREFIX/opt/tcl-tk/lib"
LDFLAGS="$LDFLAGS -L$HOMEBREW_PREFIX/opt/zlib/lib"
LDFLAGS="$LDFLAGS -L$HOMEBREW_PREFIX/opt/bzip2/lib"
export LDFLAGS="$LDFLAGS -L$HOMEBREW_PREFIX/opt/[email protected]/lib -L$HOMEBREW_PREFIX/opt/readline/lib"
PKG_CONFIG_PATH="$HOMEBREW_PREFIX/opt/sqlite/lib/pkgconfig:$HOMEBREW_PREFIX/opt/tcl-tk/lib/pkgconfig"
PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$HOMEBREW_PREFIX/opt/zlib/lib/pkgconfig"
PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$HOMEBREW_PREFIX/opt/bzip2/lib/pkgconfig"
PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$HOMEBREW_PREFIX/opt/[email protected]/lib/pkgconfig"
export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$PYENV_ROOT/versions/$PYENV_VERSION/lib/pkgconfig"

if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi

Al-Ferguson on 14 Jul 2020

👍1

I tried building using those inputs to my .zshrc and still got an error:

andytriboletti@andys-mac ~ % CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.3.tar.xz...
-> https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
Installing Python-3.8.3...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3398 (offset -28 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.0 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/1r/bdbwv9r53ls_80hpk1t6fvlm0000gn/T/python-build.20200715094722.77820
Results logged to /var/folders/1r/bdbwv9r53ls_80hpk1t6fvlm0000gn/T/python-build.20200715094722.77820.log

Last 10 log lines:
checking size of _Bool... 1
checking size of off_t... 8
checking whether to enable large file support... no
checking size of time_t... 8
checking for pthread_t... yes
checking size of pthread_t... 8
checking size of pthread_key_t... 8
checking whether pthread_key_t is compatible with int... no
configure: error: Unexpected output of 'arch' on OSX
make: *** No targets specified and no makefile found.  Stop.
andytriboletti@andys-mac ~ % arch
arm64

andytriboletti on 15 Jul 2020

I compiled it on i7 - i386 architecture.

Does brew install work for you?

PritamDutt on 15 Jul 2020

@PritamDutt , thank you.
Trying your suggestions gives me the following warning (for 3.8.0):
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.0 to /Users/bobrien/.pyenv/versions/3.8.0
When I tried 3.8.3, again, pasting your script above:
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.3 to /Users/bobrien/.pyenv/versions/3.8.3
Checked homebrew, and it is installed, but not linked.
brew link bzip2
Had no effect. I removed 3.8.3 and reattempted
pyenv install 3.8.3
and got the same warning.

At least, I have python installed on Big Sur via pyenv.

Here is the output of brew info bzip2 from my system...

Hope this can help!

% brew info bzip2
bzip2: stable 1.0.8 (bottled) [keg-only]
Freely available high-quality data compressor
https://sourceware.org/bzip2/
/usr/local/Cellar/bzip2/1.0.8 (26 files, 400.6KB)
  Poured from bottle on 2020-07-09 at 00:34:34
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/bzip2.rb
==> Caveats
bzip2 is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have bzip2 first in your PATH run:
  echo 'export PATH="/usr/local/opt/bzip2/bin:$PATH"' >> ~/.zshrc

For compilers to find bzip2 you may need to set:
  export LDFLAGS="-L/usr/local/opt/bzip2/lib"
  export CPPFLAGS="-I/usr/local/opt/bzip2/include"

==> Analytics
install: 1,045 (30 days), 3,612 (90 days), 16,828 (365 days)
install-on-request: 946 (30 days), 3,014 (90 days), 13,350 (365 days)
build-error: 0 (30 days)

PritamDutt on 15 Jul 2020

👍1

I saw something somewhere on make/cmake having issues under Big Sur & Xcode 12, so I did do a homebrew reinstall from source:
brew reinstall -s make cmake
As homebrew is does not yet support Big Sur binaries, this may make a difference.

FYI: I was just able to install Python 3.8.4 with my configuration (i.e. "pyenv install 3.8.4") under MacOS 11 & Xcode 12. And with home-brew's 3.8.4 roll out (via [email protected]), that build successfully as well.

If you could not tell from my ~/.zshrc above, I have installed the following key homebrew components before building python via "pyenv install ...":

[email protected]
bzip2
readline
tcl-tk
zlib
sqlite

Also, while not having the following installed (and added to my ~/.zshrc) did not produce an error; output from the compile indicated the following could not be found, so support for them was not included in the python build:

_curses & _curses_panel
ossaudiodev
spwd

So if you need support for those, you will need to install them and add the to the ~/.zshrc.

Al-Ferguson on 15 Jul 2020

@PritamDutt , thank you.
Trying your suggestions gives me the following warning (for 3.8.0):
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.0 to /Users/bobrien/.pyenv/versions/3.8.0
When I tried 3.8.3, again, pasting your script above:
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-3.8.3 to /Users/bobrien/.pyenv/versions/3.8.3
Checked homebrew, and it is installed, but not linked.
brew link bzip2
Had no effect. I removed 3.8.3 and reattempted
pyenv install 3.8.3
and got the same warning.
At least, I have python installed on Big Sur via pyenv.

Here is the output of brew info bzip2 from my system...

Hope this can help!

% brew info bzip2
bzip2: stable 1.0.8 (bottled) [keg-only]
Freely available high-quality data compressor
https://sourceware.org/bzip2/
/usr/local/Cellar/bzip2/1.0.8 (26 files, 400.6KB)
  Poured from bottle on 2020-07-09 at 00:34:34
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/bzip2.rb
==> Caveats
bzip2 is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have bzip2 first in your PATH run:
  echo 'export PATH="/usr/local/opt/bzip2/bin:$PATH"' >> ~/.zshrc

For compilers to find bzip2 you may need to set:
  export LDFLAGS="-L/usr/local/opt/bzip2/lib"
  export CPPFLAGS="-I/usr/local/opt/bzip2/include"

==> Analytics
install: 1,045 (30 days), 3,612 (90 days), 16,828 (365 days)
install-on-request: 946 (30 days), 3,014 (90 days), 13,350 (365 days)
build-error: 0 (30 days)

@PritamDutt ,
Thanks, stupid me - I did not have bzip2 from homebrew in my $PATH.

Thank you, again!

bryanmobrien on 22 Jul 2020

👍1

This slightly-modified version of @PritamDutt's invocation allowed me to install a couple of different versions of Python (2.7.16, 3.7.7, and 3.8.3) without having to to brew link bzip2 or zlib:

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(xcrun --show-sdk-path)/usr/lib" \
pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

I was also able to install Pythons 3.8.4 and 3.8.5 by omitting the patch, like this:

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(xcrun --show-sdk-path)/usr/lib" \
pyenv install 3.8.5

htp on 31 Aug 2020

🎉8 👍4

Thanks @htp! I managed to get this working for installing Python 3.5 using this script:

#!/bin/bash

set -ex

export CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include"
export LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(xcrun --show-sdk-path)/usr/lib -L/usr/local/opt/zlib/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include"
export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"

pyenv install --patch 3.5.9 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

I also did brew reinstall -s make cmake as someone else suggested earlier, not sure if that helped or not!

aaron-trout on 1 Sep 2020

🎉2 👍2

Give this a try:

Download and install the latest Xcode 12 beta
Go to > Preferences > Locations > Command Line Tools > Choose Xcode 12.0.

Then set the following environment variables in your shell session. I created a shell script that does it all.

#!/bin/bash
echo "Installing Dependencies"
brew install zlib
brew install sqlite
brew install bzip2
brew install libiconv
brew install libzip
echo "Done!"
echo
echo -e "Setting Environment Variables"
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/zlib/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/zlib/include"
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/sqlite/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/sqlite/include"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/zlib/lib/pkgconfig"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/sqlite/lib/pkgconfig"
echo "Done!"
echo
echo "Installing Python 3.8.5"
pyenv install 3.8.5
echo "Done!"
exit 0

Then I was able to install Python. Script output below:

# bash fix.sh

Installing Dependencies
## BREW OUTPUT REMOVED ##
Done!

Setting Environment Variables
Done!

Installing Python 3.8.5
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.5.tar.xz...
-> https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz
Installing Python-3.8.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.5 to ~/.pyenv/versions/3.8.5
Done!

rdbreak on 5 Sep 2020

👍19 ❤10 🎉9 🚀7

Does anyone know if this issue is still present on the newest beta (macOS Big Sur beta 7)?

henrycodebreaker on 19 Sep 2020

Does anyone know if this issue is still present on the newest beta (macOS Big Sur beta 7)?

I used this fix on the Big Sur beta and it should work for all iterations of the beta.

rdbreak on 19 Sep 2020

A unique issue may occur if one has installed and set the latest XCode SDK. pyenv fails on run due to invalid clang version:

  λ sudo xcode-select --switch /Applications/Xcode.app/
  λ clang --version
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
  λ ./pyenv_install_rdbreak_original_script.sh

Where ./pyenv_install_rdbreak_original_script.sh is the script posted above (here).

BUILD FAILED (OS X 11.0 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/4m/1v3nfl_x32z07msw_m6y_1m40000gn/T/python-build.20200919155415.61074
Results logged to /var/folders/4m/1v3nfl_x32z07msw_m6y_1m40000gn/T/python-build.20200919155415.61074.log

Last 10 log lines:
checking for python3.8... python3.8
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... "darwin"
checking for gcc... clang
checking whether the C compiler works... no
configure: error: in `/var/folders/4m/1v3nfl_x32z07msw_m6y_1m40000gn/T/python-build.20200919155415.61074/Python-3.8.5':
configure: error: C compiler cannot create executables
See `config.log' for more details
make: *** No targets specified and no makefile found.  Stop.
Done!

The issue relies on the fact that Big Sur is now serving _both_ versions 11.x and 10.x by default (for backwards compatibility) with automatic context switch (source, and a good article with a decent sitrep):

rfXPY

There are two viable solutions:

First Solution

Set your default SDK to the latest beta, which will automatically use context switching and utilize SDK 10.x with pyenv:

Download Command Line Tools for Xcode 12.2 beta 2 and then do the followowing:

  λ sudo xcode-select --switch /Library/Developer/CommandLineTools
  λ clang --version
Apple clang version 12.0.0 (clang-1200.0.32.2)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Second Solution

The second solution involves simply updating @rdbreak's script above to include the new SYSTEM_VERSION_COMPAT=1 flag:

#!/bin/bash
echo "Installing Dependencies"
brew install zlib
brew install sqlite
brew install bzip2
brew install libiconv
brew install libzip
echo "Done!"
echo
echo -e "Setting Environment Variables"
export SYSTEM_VERSION_COMPAT=1
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/zlib/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/zlib/include"
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/sqlite/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/sqlite/include"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/zlib/lib/pkgconfig"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/sqlite/lib/pkgconfig"
echo "Done!"
echo
echo "Installing Python 3.8.5"
pyenv install 3.8.5
echo "Done!"
exit 0

Either solution will work in the meantime.

Possible Solution (?)

~Because of all the dependencies already installed on my computer, I am unable to test this from scratch to ensure it's working as expected -- however, it seems that:~

SYSTEM_VERSION_COMPAT=1 pyenv install 3.8.5

~Works just fine as long as brew dependencies are also installed correctly (without the need to export majority of the flags in the script above). If someone is willing to test this out, it would be appreciated -- my current assumption is that as long as zlib and readline is installed and exported correctly via brew install and brew link (following the instructions), the above one liner should work.~

synchronizing on 19 Sep 2020

😄2

Hello everyone, I'm trying to install python 3.6.x using @synchronizing script however this is the error I ran into

Last 10 log lines:
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include -I/Users/henrychea/.pyenv/versions/3.6.9/include  -I/usr/local/opt/zlib/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include -I/Users/henrychea/.pyenv/versions/3.6.9/include  -I/usr/local/opt/zlib/include -I/usr/local/opt/sqlite/include  -DPy_BUILD_CORE  -c ./Modules/errnomodule.c -o Modules/errnomodule.o
./Modules/posixmodule.c:8210:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
./Modules/posixmodule.c:10432:5: warning: code will never be executed [-Wunreachable-code]
    Py_FatalError("abort() called from Python code didn't abort!");
    ^~~~~~~~~~~~~
1 warning and 1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
Done!

Currently I'm on macOS 11.00 (20A5374i) Beta 8. This issue isn't present when installing python 3.8.x or python 3.7.x

henrycodebreaker on 25 Sep 2020

Currently I'm on macOS 11.00 (20A5374i) Beta 8. This issue isn't present when installing python 3.8.x or python 3.7.x

I was shown the available update yesterday, to beta 8, but I haven't gotten the chance to update yet. Would you mind reporting what your clang --version is? Also, are you installing all of the brew packages as mentioned in the script?

By the way, the script is originally from @rdbreak, so give his a shot as well and see if it helps.

synchronizing on 25 Sep 2020

I was shown the available update yesterday, to beta 8, but I haven't gotten the chance to update yet. Would you mind reporting what your clang --version is? Also, are you installing all of the brew packages as mentioned in the script?

By the way, the script is originally from @rdbreak, so give his a shot as well and see if it helps.

I've given @rdbreak 's script a try as well but to no avail. I have all the dependencies from brew. Here is my clang version

Apple clang version 12.0.0 (clang-1200.0.32.4)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I can run 3.7.x or higher without issue but this is just to notify anyone using versions 3.6.x

henrycodebreaker on 26 Sep 2020

I can run 3.7.x or higher without issue but this is just to notify anyone using versions 3.6.x

Updated to beta 8. Updating SDK to Command Line Tools for Xcode 12.2 beta 2 and running

sudo xcode-select --switch /Library/Developer/CommandLineTools
export SYSTEM_VERSION_COMPAT=1
pyenv install 3.6.0

fixed my issues.

synchronizing on 30 Sep 2020

👍1

libz is not present in /usr/lib on macOS 11.

❯ ls /usr/lib
cron       python2.7  updaters              libMTLCapture.dylib       libiodbcinst.2.dylib
dtrace     rpcsvc     xpc                   libffi-trampolines.dylib  libiodbcinst.dylib
groff      ruby       zsh                   libgmalloc.dylib          libobjc-trampolines.dylib
log        sasl2      charset.alias         libhunspell-1.2.0.dylib   libpython.dylib
pam        sqlite3    dsc_extractor.bundle  libhunspell-1.2.dylib     libpython2.7.dylib
php        swift      dyld                  libiodbc.2.dylib          libstdc++.6.dylib
pkgconfig  system     libLeaksAtExit.dylib  libiodbc.dylib            libstdc++.dylib

From Big Sur 11 Beta Release notes:

New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

This is a straightforward linker error and can be fixed by modifying LDFLAGS.

python-build (and pyenv) already have logic to add "-I$(xcrun --show-sdk-path)/usr/include" to CFLAGS, but they do not appear to touch LDFLAGS.

The dynamic library exists under $(xcrun --show-sdk-path)/usr/lib on the Big Sur beta once you install the CLT.

❯ ls $(xcrun --show-sdk-path)/usr/lib/libz*
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libz.1.1.3.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libz.1.2.11.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libz.1.2.5.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libz.1.2.8.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libz.1.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libz.tbd

The simplest interim fix is:

LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.8.6

Long-term, python-build should automatically detect this condition and add to LDFLAGS. It would also be great to get the equivalent PYTHON_LDFLAGS (doesn't currently exist) companion of PYTHON_CFLAGS (detailed here).

lambda-9 on 5 Oct 2020

👍4

@lambda-9 LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.8.6 this is working for me

Zheaoli on 6 Oct 2020

🎉5

Created PR https://github.com/pyenv/pyenv/pull/1711 to address the LDFLAGS issues. To build on Big Sur in general, you also need the Xcode 12.2 beta installed which includes the macOS 11 SDK. This does not come bundled with the macOS beta. You can download the Xcode beta from Apple here by selecting the Command Line Tools for Xcode 12.2 beta 2.

lambda-9 on 7 Oct 2020

I'm hoping that some kind soul will help an old Aspie out here.

I assume I must be misreading, but I'm at a loss as to what set of instructional pieces from the long thread above is 'cannon' now? Forgive my rather broad line of inquiry, but I have a combination that will work as long as I use a bare zsh shell, but if I use anything else (my focus being getting my fish shell to work) I get either a linker or a clang error.

Big Sur Beta 20A5395g
Xcode 12.2 beta 3
Command Line tools for Beta 3
clang --version "Apple clang version 12.0.0 (clang-1200.0.32.27)"
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Should we be using --HEAD for either/both of brew/pyenv?

Should we get rid of any/all LDFLAGS settings we had previously (including openssl)?

Forgive me if I'm being a time waster, I've done my best to RTFM.

ehagerty on 19 Oct 2020

Forgive me if I'm being a time waster, I've done my best to RTFM.

@ehagerty - it never hurts to ask!

I don't think this is _quite_ the right place to figure out the issue(s) you're running into, but shoot me an email (it's in my profile) and I'd be happy to help you out. If we learn anything interesting along the way, we can post back here so other folks can benefit.

htp on 20 Oct 2020

❤1

@htp thank you for the kind offer. If it is of any use to anyone, here is an update to the script from @synchronizing which itself was kindly provided by @rdbreak

Now I only wish I could figure out why grpcio fails clang builds too...

`#!/usr/local/bin/fish

echo "<< UPGRADING BREW >>"
echo
brew update
brew upgrade
brew cleanup
brew doctor
echo "<< reinstalling pyenv compile requirements >>"
echo
brew reinstall zlib
brew reinstall sqlite
brew reinstall libiconv
brew reinstall libzip
brew reinstall bzip2
echo "<< setting Big Sur BETA flags >>"
echo

sudo xcode-select --switch /Library/Developer/CommandLineTools

set SYSTEM_VERSION_COMPAT "1"
echo "<< resetting LDFLAGS, CFLAGS and PKG_CONFIG_PATH to force pyenv to use them >>"
echo
set -g LDFLAGS "-L/usr/local/opt/zlib/lib"
set -g CPPFLAGS "-I/usr/local/opt/zlib/include"
set -g LDFLAGS "-L/usr/local/opt/sqlite/lib" $LDFLAGS
set -g CPPFLAGS "-I/usr/local/opt/sqlite/include" $CPPFLAGS
set -g LDFLAGS "-L/usr/local/opt/bzip2/lib" $LDFLAGS
set -g CPPFLAGS "-I/usr/local/opt/bzip2/include" $CPPFLAGS
set -g PKG_CONFIG_PATH "/usr/local/opt/zlib/lib/pkgconfig"
set -g PKG_CONFIG_PATH "/usr/local/opt/sqlite/lib/pkgconfig" $PKG_CONFIG_PATH
set -g PKG_CONFIG_PATH "/usr/local/opt/bzip2/lib/pkgconfig" $PKG_CONFIG_PATH
set -g fish_user_paths "/usr/local/opt/bzip2/bin" $fish_user_paths
echo
echo
read -l -P "What version do you want installed? " install_version
switch $install_version
case ''
echo "fine, i will install 3.8.5 then..."
echo
set $install_version 3.8.5
end
echo "<< Installing Python $install_version >>"
echo
pyenv install $install_version
echo
echo "I HATE BETAS!"
echo`

ehagerty on 21 Oct 2020

Thanks to the author of this article. It saved my day.
https://medium.com/@koji_kanao/install-python-3-8-0-via-pyenv-on-bigsur-b4246987a548
Mine is MacOs Big Sur Beta 11.0 Version
XCode 12.2
CommandLineTool 12.2

IamGabrielWu on 23 Oct 2020

So I have a weird situation pyenv will install some python versions but not all. See below. I am on Big Sur Beta also. For the errors for the failed install I get:
error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = sendfile(in, out, offset, &sbytes, &sf, flags);
^
1 error generated.
make: * [Modules/posixmodule.o] Error 1
make: * Waiting for unfinished jobs....
1 warning generated.

For example:
pyenv install 3.6.x - Fails
pyenv install 3.7.x - Fails
pyenv install 3.8.0-3 - Fails
pyenv install 3.8.4-6 - Success
pyenv install 3.9.0 - Success

usmcfiredog on 24 Oct 2020

👍2

can't build 3.6.X too with

Last 10 log lines: ./Modules/posixmodule.c:8210:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ret = sendfile(in, out, offset, &sbytes, &sf, flags); ^ ./Modules/posixmodule.c:10432:5: warning: code will never be executed [-Wunreachable-code] Py_FatalError("abort() called from Python code didn't abort!"); ^~~~~~~~~~~~~

upper versions looks fine.
but 3.6 won't work as installed with installer, wont build from python.org src, won't build with pyenv... =(

evilsneer on 12 Nov 2020

The following worked for me on 3.8.6

Aside - I have found homebrew pyenv to be problematic so I installed it manually. Set up according to the instructions on the GitHub page

Install Xcode 12.2 Beta 4
in Xcode Preferences Locations update the command-line tools location to point at the 12.2
brew install openssl zlib readline

#!/usr/bin/env bash

BREW_PACKAGES=(openssl zlib readline)
SYSROOT=$(xcrun --sdk macosx --show-sdk-path)

INCLUDES=""
LIBRARIES=""
for INDEX in ${!BREW_PACKAGES[@]}; do
    BREW_PACKAGE=${BREW_PACKAGES[${INDEX}]}
    PATH_PREFIX=$(brew --prefix ${BREW_PACKAGE})
    INCLUDES="-I${PATH_PREFIX}/include ${INCLUDES}"
    LIBRARIES="-L${PATH_PREFIX}/lib ${LIBRARIES}"
done
INCLUDES="${INCLUDES} -I${SYSROOT}/usr/include"
LIBRARIES="${LIBRARIES} -L${SYSROOT}/usr/lib"

CFLAGS="-isysroot ${SYSROOT} ${INCLUDES}"
LDFLAGS="${LIBRARIES}"

CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} pyenv install 3.8.6

### Older versions need to be patched
### CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} pyenv install --patch 3.8.3 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

palmerc on 12 Nov 2020

👍1

@palmerc thank you very much.
With the BigSur version I was not able to install any versions of python through pyenv.
With your script I installed python.

mett96 on 14 Nov 2020

@palmerc I was able to install 3.9.0 on MacOS Big Sur 11.0.1 with your script. Thank you!

I manually installed pyenv then used this shell script:

#!/usr/bin/env bash

BREW_PACKAGES=(openssl zlib readline)
SYSROOT=$(xcrun --sdk macosx --show-sdk-path)

INCLUDES=""
LIBRARIES=""
for INDEX in ${!BREW_PACKAGES[@]}; do
    BREW_PACKAGE=${BREW_PACKAGES[${INDEX}]}
    PATH_PREFIX=$(brew --prefix ${BREW_PACKAGE})
    INCLUDES="-I${PATH_PREFIX}/include ${INCLUDES}"
    LIBRARIES="-L${PATH_PREFIX}/lib ${LIBRARIES}"
done
INCLUDES="${INCLUDES} -I${SYSROOT}/usr/include"
LIBRARIES="${LIBRARIES} -L${SYSROOT}/usr/lib"

CFLAGS="-isysroot ${SYSROOT} ${INCLUDES}"
LDFLAGS="${LIBRARIES}"

CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} pyenv install 3.9.0

### Older versions need to be patched
### CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} pyenv install --patch 3.9.0 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

royimous on 15 Nov 2020

👍1

There's at least a couple problems in this ticket so I'll try to summarize what worked and didn't for me upgrading from working versions of the same Pythons in Catalina.

The following worked for 2.7.18, 3.7.9, 3.8.6, and 3.9.0. It's worth nothing this looks like it's been fixed in #1711:

$ LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.6.8

However, on 3.7.6 there's different errors based on whether you use SYSTEM_VERSION_COMPAT, showing just the default here:

$ LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.7.6
... snip ...
Last 10 log lines:
./Modules/posixmodule.c:8436:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);

3.6.12:

$ LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.6.12
... snip ...
Last 10 log lines:
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
./Modules/posixmodule.c:10432:5: warning: code will never be executed [-Wunreachable-code]
    Py_FatalError("abort() called from Python code didn't abort!");
    ^~~~~~~~~~~~~

3.5.10:

$ LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.5.10
... snip ...
Last 10 log lines:
    ^~~~~~~~~~~~~
./Modules/posixmodule.c:11345:9: warning: code will never be executed [-Wunreachable-code]
        posix_error();
        ^~~~~~~~~~~

mmerickel on 15 Nov 2020

I can confirm installing pyenv manually from sources as of today (Nov 16th) in this manner works out of the box on Big Sur:
https://github.com/pyenv/pyenv#basic-github-checkout

davidmarquis on 16 Nov 2020

I can confirm installing pyenv manually from sources as of today (Nov 16th) in this manner works out of the box on Big Sur:
pyenv/pyenv#basic-github-checkout

@davidmarquis I am still having this issue 😔 . Just installed the official Big Sur release two days ago.

jdtzmn on 16 Nov 2020

I cloned the PyEnv master branch and was able to build 3.9.0 on macOS Big Sur 11.0.1 (the Homebrew version hasn't been updated yet).

Sjors on 16 Nov 2020

@Sjors Oh I didn't realize the Homebrew version wasn't updated yet. I'll try cloning the master branch and will update this comment accordingly. Thank you!

Update: Python 3.7.3 is still failing:

error: implicit declaration of function 'sendfile' is invalid in C99

jdtzmn on 16 Nov 2020

It does always warn me: WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?

Python 3.8.3 still results in the same error as before, but Python 3.8.6 does build so that's good enough for me.

More annoying is that Python 3.6.12 doesn't build:

% pyenv install 3.6.12
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.6.12.tar.xz...
-> https://www.python.org/ftp/python/3.6.12/Python-3.6.12.tar.xz
Installing Python-3.6.12...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.0.1 using python-build 1.2.21-1-g943015eb)

Inspect or clean up the working tree at /var/folders/dq/pp76yn792zn8hpdcsv5t4b040000gn/T/python-build.20201116174329.85347
Results logged to /var/folders/dq/pp76yn792zn8hpdcsv5t4b040000gn/T/python-build.20201116174329.85347.log

Last 10 log lines:
./Modules/posixmodule.c:8210:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include -I/Users/sjors/.pyenv/versions/3.6.12/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include -I/Users/sjors/.pyenv/versions/3.6.12/include   -DPy_BUILD_CORE  -c ./Modules/errnomodule.c -o Modules/errnomodule.o
./Modules/posixmodule.c:10432:5: warning: code will never be executed [-Wunreachable-code]
    Py_FatalError("abort() called from Python code didn't abort!");
    ^~~~~~~~~~~~~
1 warning and 1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....

Sjors on 16 Nov 2020

Waiting for a new Xcode version (or downloading a beta, which IIUC is when you need step 2) seems reasonable to me. Looks like I forgot to update the command line tools this time. I'm still on Xcode 12.2 so will wait a bit.

Sjors on 16 Nov 2020

Waiting for a new Xcode version (or downloading a beta, which IIUC is when you need step 2) seems reasonable to me. Looks like I forgot to update the command line tools this time. I'm still on Xcode 12.2 so will wait a bit.

Indeed. Bad design by Apple, with almost no documentation outside of a random Tweet I found by an Apple employee that had _some_ context clues. Horribly done on their end. I'm sure this is resulting in major bugs in thousands of softwares and frameworks around the web on Big Sur. Hopefully it gets fixed soon -- but then again, Winter is Coming, and their ARM chip will be bringing along millions of new problems; I'll just sit back with my Intel chip and watch as carnage begins to happen... this will be magnitude worst than the dev community in 2008 when Apple said good-bye to Flash. But then again, progress requires sacrifices.

synchronizing on 16 Nov 2020

I reinstalled everything (I hope) but it is not working for me.

macOS 11.0.1
Developertools for XCode 12.3 beta
python 2.7.16 (system)
pip 20.2.4 (istalled in system's python)

sudo xcode-select --switch /Library/Developer/CommandLineTools

Both commands
pyenv install 3.9.0
SYSTEM_VERSION_COMPAT=1 pyenv install 3.9.0

result in:

python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.0.tar.xz...
-> https://www.python.org/ftp/python/3.9.0/Python-3.9.0.tar.xz
Installing Python-3.9.0...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 10.16 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116182443.71956
Results logged to /var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116182443.71956.log

Last 10 log lines:
  File "/private/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116182443.71956/Python-3.9.0/Lib/ensurepip/__init__.py", line 210, in _main
    return _bootstrap(
  File "/private/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116182443.71956/Python-3.9.0/Lib/ensurepip/__init__.py", line 129, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/private/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116182443.71956/Python-3.9.0/Lib/ensurepip/__init__.py", line 38, in _run_pip
    return subprocess.run([sys.executable, "-c", code], check=True).returncode
  File "/private/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116182443.71956/Python-3.9.0/Lib/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116182443.71956/Python-3.9.0/python.exe', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/tmpux9blbc2/setuptools-49.2.1-py3-none-any.whl\', \'/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/tmpux9blbc2/pip-20.2.3-py2.py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/tmpux9blbc2\', \'--root\', \'/\', \'--upgrade\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.
make: *** [install] Error 1

wolfgang-werner on 16 Nov 2020

👀1

I reinstalled everything (I hope) but it is not working for me.

Check my response above. I added perm fix for cpython via the following patch. You will be looking at doing the following now:

pyenv install --patch 3.8.0 <<(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

Hope it helps!

synchronizing on 16 Nov 2020

❤1

Hi Felipe,

pyenv install --patch 3.8.0 <<(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch?full_index=1)

thanks for this but - in my case - this seams not to be the problem. I can do the build with the patch, but the error is then:

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L/usr/local/lib -L$(brew --prefix bzip2)/lib" pyenv install --patch 3.8.0 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.0.tar.xz...
-> https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tar.xz
Installing Python-3.8.0...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3394 (offset -32 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.0.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116195357.15726
Results logged to /var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116195357.15726.log

Last 10 log lines:
    return _bootstrap(
  File "/private/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116195357.15726/Python-3.8.0/Lib/ensurepip/__init__.py", line 119, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/private/var/folders/0b/spth3nrn4cb2zvnmppd5wmdw0000gn/T/python-build.20201116195357.15726/Python-3.8.0/Lib/ensurepip/__init__.py", line 27, in _run_pip
    import pip._internal
  File "<frozen zipimport>", line 241, in load_module
  File "<frozen zipimport>", line 709, in _get_module_code
  File "<frozen zipimport>", line 570, in _get_data
zipimport.ZipImportError: can't decompress data; zlib not available

May it be a problem with the zlib? I installed that but it is not linked, if i try to link it (brew linlk zlib) the answer is: Will not link system shadow...

I tried to search for libz and give different paths to the command, bitz did not succeed...

find /usr -name "libz*"
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libzip.rb
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libzt.rb
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libzzip.rb
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libzdb.rb
/usr/local/var/homebrew/linked/libzip
/usr/local/lib/libzip.dylib
/usr/local/lib/pkgconfig/libzip.pc
/usr/local/lib/libzip.5.dylib
/usr/local/lib/cmake/libzip
/usr/local/lib/libzip.5.3.dylib
/usr/local/opt/libzip
/usr/local/Cellar/libzip
/usr/local/Cellar/libzip/1.7.3/.brew/libzip.rb
/usr/local/Cellar/libzip/1.7.3/lib/libzip.dylib
/usr/local/Cellar/libzip/1.7.3/lib/pkgconfig/libzip.pc
/usr/local/Cellar/libzip/1.7.3/lib/libzip.5.dylib
/usr/local/Cellar/libzip/1.7.3/lib/cmake/libzip
/usr/local/Cellar/libzip/1.7.3/lib/cmake/libzip/libzip-config-version.cmake
/usr/local/Cellar/libzip/1.7.3/lib/cmake/libzip/libzip-targets-release.cmake
/usr/local/Cellar/libzip/1.7.3/lib/cmake/libzip/libzip-config.cmake
/usr/local/Cellar/libzip/1.7.3/lib/cmake/libzip/libzip-targets.cmake
/usr/local/Cellar/libzip/1.7.3/lib/libzip.5.3.dylib
/usr/local/Cellar/libzip/1.7.3/share/man/man3/libzip.3
/usr/local/share/man/man3/libzip.3

I tried:

-L/usr/local/lib
-L/usr/local/Cellar/libzip/1.7.3/lib

but no luck.

Any suggestions? All the best!

wolfgang-werner on 16 Nov 2020

@wolfgang-werner

zipimport.ZipImportError: can't decompress data; zlib not available

Man, Big Sur is a big fuckfest at this point- hopefully things get fixed. Try the following:

brew install sqlite bzip2 zlib

Then:

PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" \
CFLAGS="-I$(brew --prefix zlib)/include -I$(brew --prefix sqlite)/include -I$(brew --prefix bzip2)/include" \
LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix sqlite)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.0 <<(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

Let me know if it works so I can update the post above as a "just in case". The solution above ensures you have all the dependencies required - which might not _all_ be needed, but it's hard to tell on my end without a fresh Big Sur install and trial & error and source code hunting - but will hopefully fix error for you.

synchronizing on 16 Nov 2020

👍3

I merged in a Big Sur fix yesterday (#1711) that sets LDFLAGS as required, hope that helps somewhat.

joshfriend on 16 Nov 2020

❤1 👍1

[...] Try the following:
brew install sqlite bzip2 zlib
Then:
PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" \
CFLAGS="-I$(brew --prefix zlib)/include -I$(brew --prefix sqlite)/include -I$(brew --prefix bzip2)/include" \
LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix sqlite)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.0 <<(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
Let me know if it works so I can update the post above as a "just in case". The solution above ensures you have all the dependencies required - which might not _all_ be needed, but it's hard to tell on my end without a fresh Big Sur install and trial & error and source code hunting - but will hopefully fix error for you.

In my case the zlib was damaged - maybe from earlier installations or Mac Port.... (I updated an old machine).

I had to delete an empty directory for zlib and then it worked!

Edited terminal output:

~ brew install zlib
[...]
Error: /usr/local/opt/zlib is not a valid keg
~ rm -rf /usr/local/opt/zlib
~ brew install zlib
[...]
==> Downloading https://homebrew.bintray.com/bottles/zlib-1.2.11.big_sur.bottle.tar.gz
######################################################################## 100.0%
==> Pouring zlib-1.2.11.big_sur.bottle.tar.gz
==> Summary
🍺  /usr/local/Cellar/zlib/1.2.11: 12 files, 413.4KB

~ PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" \
CFLAGS="-I$(brew --prefix zlib)/include -I$(brew --prefix sqlite)/include -I$(brew --prefix bzip2)/include" \
LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix sqlite)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.0 <<(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.0.tar.xz...
-> https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tar.xz
Installing Python-3.8.0...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3394 (offset -32 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.0 to /Users/wwerner/.pyenv/versions/3.8.0

Thanks a lot!

wolfgang-werner on 16 Nov 2020

❤2 👍2

One mote thing - can you provide the fix for the newer versions? E.g. 3.9.0?

PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" \
CFLAGS="-I$(brew --prefix zlib)/include -I$(brew --prefix sqlite)/include -I$(brew --prefix bzip2)/include" \
LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix sqlite)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.9.0 <<(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.0.tar.xz...
-> https://www.python.org/ftp/python/3.9.0/Python-3.9.0.tar.xz
Installing Python-3.9.0...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 FAILED at 3426.
1 out of 1 hunk FAILED -- saving rejects to file configure.rej
patching file configure.ac
Hunk #1 FAILED at 510.
1 out of 1 hunk FAILED -- saving rejects to file configure.ac.rej

wolfgang-werner on 16 Nov 2020

If you are trying to build 3.9.0 (and many other versions), the merge that @joshfriend completed the other day should allow it to build without issue. Just update pyenv to use the master branch and pull the latest changes. I am not sure if these changes are integrated in homebrew yet. 3.9.0, 3.8.6, and 3.7.9 all work without issue after pulling the changes.

Python 3.8.0's build makes some wrong assumptions that do not work with Big Sur in its current form. For 3.8.0, you have two options: install 3.8.6 (which should be compatible and builds fine) or use the patch that @synchronizing supplied to cpython itself. Both will resolve the sendfile() issues.

lambda-9 on 17 Nov 2020

👍1

Vvbk

Android için Outlookhttps://aka.ms/ghei36'u edinin

From: network-shark notifications@github.com
Sent: Sunday, November 15, 2020 11:55:02 PM
To: pyenv/pyenv pyenv@noreply.github.com
Cc: Subscribed subscribed@noreply.github.com
Subject: Re: [pyenv/pyenv] Unable to build Python on macOS Big Sur with Xcode 12 beta (#1643)

@mmerickelhttps://github.com/mmerickel Are you able to import openssl ? I successfully compiled 3.8.5 , but importing openssl stillt does not work .

╰─ pip list | grep 'pyOpenSSL'

pyOpenSSL 19.0.0

─ python3

Python 3.8.5 (default, Nov 14 2020, 14:32:21)

[Clang 12.0.0 (clang-1200.0.32.27)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

import openssl

Traceback (most recent call last):

File "", line 1, in

ModuleNotFoundError: No module named 'openssl'

import pyopenssl

Traceback (most recent call last):

File "", line 1, in

ModuleNotFoundError: No module named 'pyopenssl'

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/pyenv/pyenv/issues/1643#issuecomment-727635170, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQ4EYF2ZMHJK6Q2R7I5DG3LSQA52NANCNFSM4OJNZUQQ.

gokhan275 on 17 Nov 2020

If you are trying to build 3.9.0 (and many other versions), the merge that @joshfriend completed the other day should allow it to build without issue. Just update pyenv to use the master branch and pull the latest changes. I am not sure if these changes are integrated in homebrew yet. 3.9.0, 3.8.6, and 3.7.9 all work without issue after pulling the changes.

Python 3.8.0's build makes some wrong assumptions that do not work with Big Sur in its current form. For 3.8.0, you have two options: install 3.8.6 (which should be compatible and builds fine) or use the patch that @synchronizing supplied to cpython itself. Both will resolve the sendfile() issues.

When do you plan to create a new release so we can update the homebrew formula?

flpStrri on 17 Nov 2020

👍1

before you shuold intalled: readline xz zlib

brew install readline xz zlib

and add readline xz zlib environment to .zshrc
I solved this problem by use:

rm -rvf ~/.pyenv
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
brew upgrade pyenv

then I succeed install 3.8.6 and 3.9.0 and so on

this is my Detailed process png

if you con't see the png, can go to there:
https://github.com/LearningChanging/fix-Unable-to-build-Python-on-macOS-Big-Sur-with-Xcode-12-beta/blob/main/README.md

this my .zshrc config for readline zx zlib

#If you need to have [email protected] first
export PATH="/usr/local/opt/[email protected]/bin:$PATH"

# For compilers to find [email protected]
export LDFLAGS="-L/usr/local/opt/[email protected]/lib"
export CPPFLAGS="-I/usr/local/opt/[email protected]/include"

# For pkg-config to find [email protected]
export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig"

# For compilers to find readline
export LDFLAGS="-L/usr/local/opt/readline/lib"
export CPPFLAGS="-I/usr/local/opt/readline/include"

# For pkg-config to find readline
export PKG_CONFIG_PATH="/usr/local/opt/readline/lib/pkgconfig"


# For compilers to find zlib
export LDFLAGS="-L/usr/local/opt/zlib/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include"

# For pkg-config to find zlib
export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"

export PATH="/usr/local/opt/sqlite/bin:$PATH"

# For compilers to find sqlite
export LDFLAGS="-L/usr/local/opt/sqlite/lib"
export CPPFLAGS="-I/usr/local/opt/sqlite/include"

# For pkg-config to find sqlite
export PKG_CONFIG_PATH="/usr/local/opt/sqlite/lib/pkgconfig"

LearningChanging on 17 Nov 2020

❤3 🎉3 👍3

I was able to install the following Python versions:

# 2.7.18
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 2.7.18
# 3.6.12
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install --patch --force  3.6.12 <<(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
# 3.7.9
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.7.9
# 3.8.6
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.8.6
# 3.9.0
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.9.0

Note that the arch here is mainly so that I can build this stuff on the Apple Silicon M1, since trying to build natively does not seem to work yet, and it'll be a while before that stuff is backported.

This is using Xcode 12.2 from the App Store. pyenv from Homebrew.

bertjwregeer on 18 Nov 2020

🎉12 ❤3

This warning keeps showing
what is the reason?

～ pyenv install 2.7.15
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-2.7.15.tar.xz...
-> https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tar.xz
Installing Python-2.7.15...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-2.7.15 to /Users/xuanleung/.pyenv/versions/2.7.15

～ pyenv versions
* system (set by /Users/xuanleung/.pyenv/version)
  2.7.15
  3.8.5

iexxk on 18 Nov 2020

@iexxk

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?

brew install bzip2

synchronizing on 18 Nov 2020

This warning keeps showing
what is the reason?

～ pyenv install 2.7.15
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-2.7.15.tar.xz...
-> https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tar.xz
Installing Python-2.7.15...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
Installed Python-2.7.15 to /Users/xuanleung/.pyenv/versions/2.7.15

～ pyenv versions
* system (set by /Users/xuanleung/.pyenv/version)
  2.7.15
  3.8.5

you can ignore the WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?.
it does not affect the use of Python and pyenv.

LearningChanging on 18 Nov 2020

@iexxk

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?

brew install bzip2

No. Bzip2 is in Xcode. If you correctly set the -L and -I flags and sysroot, all will be fine. I wrote a short bash script to deal with this earlier in the thread.

https://github.com/pyenv/pyenv/issues/1643#issuecomment-726051337

palmerc on 18 Nov 2020

Very useless comment... I fixed it and i have no idea how

svenzaimovic on 18 Nov 2020

😄5 👀1

Very useless comment... I fixed it and i have no idea how

I'd say it is less a useless comment and more of a... you don't compile a lot of software with weird non-standard dependencies.
When it says you are missing bzip2 it means it did not find the headers, or lack some other prereq. A number of libraries are provided in Xcode, Apple has for most intents and purposes removed system headers, and brew does not want to install conflicting (shadow) libraries by default.

You'll find the tbd for libbz2 under:

`xcrun -sdk macosx -show-sdk-path`/usr/lib

You'll find the bzip2.h under:

`xcrun -sdk macosx -show-sdk-path`/usr/include

-I is used to add a search path for includes
-L is used to add a search path for libraries
-l is used to specify a library to link against
-isysroot tells the compiler explicitly the search root for the SDK you intend to use, Xcode ships with several, under sysroot you'll find /usr/{include, lib, bin}

so...

SYSROOT=`xcrun --sdk macosx --show-sdk-path`
CFLAGS="-isysroot ${SYSROOT} -I/path/to/include"
LDFLAGS="-L/path/to/lib".
CPPFLAGS="${CFLAGS}"

CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} pyenv install 3.8.6

I set CFLAGS and CPPFLAGS for good measure.
FYI - in this context CPP -> C Pre-Processor _not_ C++

So when you call pyenv install it calls the plugin pyenv-build and it tries to detect where things are. When it fails at detecting, you can help the compiler along by passing the correct search paths for the libraries and their headers via environment variables. You can always look inside pyenv-build and find out what condition needs to be satisfied. I just wrote that shell script to try and construct programatically the missing -L and -I and add the -isysroot. I also recall seeing that somewhere in the python source it will check for isysroot in CFLAGS and use that rather than guessing, so I go ahead and add that to CFLAGS.

https://github.com/pyenv/pyenv/issues/1643#issuecomment-726051337

palmerc on 19 Nov 2020

🚀1 ❤1 👎1

Very useless comment... I fixed it and i have no idea how

I'd say it is less a useless comment and more of a... you don't compile a lot of software with weird non-standard dependencies.
When it says you are missing bzip2 it means it did not find the headers, or lack some other prereq. A number of libraries are provided in Xcode, Apple has for most intents and purposes removed system headers, and brew does not want to install conflicting (shadow) libraries by default.

You'll find the tbd for libbz2 under:
`xcrun -sdk macosx -show-sdk-path`/usr/lib
You'll find the bzip2.h under:
`xcrun -sdk macosx -show-sdk-path`/usr/include
-I is used to add a search path for includes
-L is used to add a search path for libraries
-l is used to specify a library to link against
-isysroot tells the compiler explicitly the search root for the SDK you intend to use, Xcode ships with several, under sysroot you'll find /usr/{include, lib, bin}

so...
SYSROOT=`xcrun --sdk macosx --show-sdk-path`
CFLAGS="-isysroot ${SYSROOT} -I/path/to/include"
LDFLAGS="-L/path/to/lib".
CPPFLAGS="${CFLAGS}"

CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} pyenv install 3.8.6
I set CFLAGS and CPPFLAGS for good measure.
FYI - in this context CPP -> C Pre-Processor _not_ C++

So when you call pyenv install it calls the plugin pyenv-build and it tries to detect where things are. When it fails at detecting, you can help the compiler along by passing the correct search paths for the libraries and their headers via environment variables. You can always look inside pyenv-build and find out what condition needs to be satisfied. I just wrote that shell script to try and construct programatically the missing -L and -I and add the -isysroot. I also recall seeing that somewhere in the python source it will check for isysroot in CFLAGS and use that rather than guessing, so I go ahead and add that to CFLAGS.

#1643 (comment)

Oh no, you got me wrong. My comment, where i said that i dont know how i fixed it was useless. I guess i worded it poorly

svenzaimovic on 19 Nov 2020

😄1 👍1

Reporting in with a similar issue trying to install 3.4.3. I've tired several of the suggested fixes including setting the XCode SDK, trying several of the suggested bash scripts and setting CFLAGS and LFLAGS, to no avail. Running the install command provides the following error:

➜ pyenv install 3.4.3
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.4.3.tar.xz...
-> https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tar.xz
Installing Python-3.4.3...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.0.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/fd/d66fbjt52158hypywt3hqrxh0000gn/T/python-build.20201120135412.16826
Results logged to /var/folders/fd/d66fbjt52158hypywt3hqrxh0000gn/T/python-build.20201120135412.16826.log

Last 10 log lines:
        -o Python/sysmodule.o ./Python/sysmodule.c
        int res = getentropy(buffer, len);
                  ^
Python/random.c:81:19: note: did you mean 'py_getentropy'?
Python/random.c:77:1: note: 'py_getentropy' declared here
py_getentropy(unsigned char *buffer, Py_ssize_t size, int fatal)
^
1 error generated.
make: *** [Python/random.o] Error 1
make: *** Waiting for unfinished jobs....

Unfortunately Python 3.4.3 is a hard dependency for me at work, so if I'm unable to install it I may have to downgrade to Catalina.

oreid01 on 20 Nov 2020

It is not you, 3.4.3 does not compile cleanly on Big Sur and this isn't strictly a pyenv problem.

This is a problem because pyconfig.h says your system has a C function getentropy() that cannot be found at compile-time. In fact, if you clear this error it will have the same problem with sendfile().

The short answer to compiling is to undefine (comment out) HAVE_GETENTROPY and HAVE_SENDFILE in pyconfig.h.

Have you considered copying a working 3.4.3 over from Catalina?

palmerc on 20 Nov 2020

I was able to install the following Python versions:
# 2.7.18
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 2.7.18
# 3.6.12
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install --patch --force  3.6.12 <<(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
# 3.7.9
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.7.9
# 3.8.6
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.8.6
# 3.9.0
arch -arch x86_64 env LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib" pyenv install 3.9.0
Note that the arch here is mainly so that I can build this stuff on the Apple Silicon M1, since trying to build natively does not seem to work yet, and it'll be a while before that stuff is backported.

This is using Xcode 12.2 from the App Store. pyenv from Homebrew.

Confirmed! Only workaround that worked for so far for me (macOS 11.0.1, Xcode 12.2). Would be great if someone could find a way to get 3.7.4 too.

tbrodbeck on 21 Nov 2020

👍5 🎉1

before you shuold intalled: readline xz zlib

brew install readline xz zlib

and add readline xz zlib environment to .zshrc
I solved this problem by use:

rm -rvf ~/.pyenv
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
brew upgrade pyenv

then I succeed install 3.8.6 and 3.9.0 and so on

this is my Detailed process png

if you con't see the png, can go to there:
https://github.com/LearningChanging/fix-Unable-to-build-Python-on-macOS-Big-Sur-with-Xcode-12-beta/blob/main/README.md

this my .zshrc config for readline zx zlib

#If you need to have [email protected] first
export PATH="/usr/local/opt/[email protected]/bin:$PATH"

# For compilers to find [email protected]
export LDFLAGS="-L/usr/local/opt/[email protected]/lib"
export CPPFLAGS="-I/usr/local/opt/[email protected]/include"

# For pkg-config to find [email protected]
export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig"

# For compilers to find readline
export LDFLAGS="-L/usr/local/opt/readline/lib"
export CPPFLAGS="-I/usr/local/opt/readline/include"

# For pkg-config to find readline
export PKG_CONFIG_PATH="/usr/local/opt/readline/lib/pkgconfig"


# For compilers to find zlib
export LDFLAGS="-L/usr/local/opt/zlib/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include"

# For pkg-config to find zlib
export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"

export PATH="/usr/local/opt/sqlite/bin:$PATH"

# For compilers to find sqlite
export LDFLAGS="-L/usr/local/opt/sqlite/lib"
export CPPFLAGS="-I/usr/local/opt/sqlite/include"

# For pkg-config to find sqlite
export PKG_CONFIG_PATH="/usr/local/opt/sqlite/lib/pkgconfig"

above code only can succeed install 3.7.9、3.8.4-3.9.0

i use the following command code succeed install 3.8.0、3.8.2、3.8.3

PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" \
CFLAGS="-I$(brew --prefix zlib)/include -I$(brew --prefix sqlite)/include -I$(brew --prefix bzip2)/include" \
LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix sqlite)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.3 <<(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

cmd out

python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.3.tar.xz...
-> https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tar.xz
Installing Python-3.8.3...
patching file Misc/NEWS.d/next/macOS/2020-06-24-13-51-57.bpo-41100.mcHdc5.rst
patching file configure
Hunk #1 succeeded at 3398 (offset -28 lines).
patching file configure.ac
Hunk #1 succeeded at 498 (offset -12 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Installed Python-3.8.3 to /Users/justdoit/.pyenv/versions/3.8.3

LearningChanging on 21 Nov 2020

The best solution is to abandon pyenv, Anaconda3 is great. 😏

LearningChanging on 21 Nov 2020

👎14

After upgrading macOS to Big Sur, Python build using pyenv did not work properly, such as readline library not being compiled. Thankfully, GitHub allows me to build code and upload artifacts on macOS Catalina.

The installation guide was written down in README.md at the link below.
https://github.com/youngminz/macos-pyenv-binary

youngminz on 22 Nov 2020

👍1

@oreid01
Here's a patch to help install 3.4.3 with pyenv: https://gist.github.com/orip/00142a498cf6d0c536b5d38771ad6249

orip on 23 Nov 2020

After struggling with the same error message, what did work for me was reinstalling homebrew and pyenv

For homebrew I just run the installation command

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

And regarding pyenv, I deleted .pyenv folder in my home directory

rm -rf ~/.pyenv

and then reinstalled it

brew reinstall pyenv

hecasanova57 on 24 Nov 2020

brew remove binutils

Fixed my issue where I was getting link error:

Last 10 log lines:
  "__PyRuntime", referenced from:
      _check_preinit_isolated_config in _testembed.o
  "__Py_Dealloc", referenced from:
      _test_open_code_hook in _testembed.o
      __audit_hook_run in _testembed.o
  "__Py_InitializeMain", referenced from:
      _test_init_main in _testembed.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Programs/_testembed] Error 1

rterbush on 24 Nov 2020

@oreid01
Here's a patch to help install 3.4.3 with pyenv: https://gist.github.com/orip/00142a498cf6d0c536b5d38771ad6249

@orip Life saver! Thanks for the help.

Unfortunately, I got your message as the Catalina installer was running to fresh install macOS on my machine, so I can't verify that the fix works for Big Sur, however I can say that I had the same issue when I tried installing on macOS Catalina and your patch fixed the issue there.

Cheers!

oreid01 on 24 Nov 2020

The solution mentioned in the first answer of Stackoverflow Question worked for installing Python 3.6.0.
Adding the command I used here as well:
CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" pyenv install --patch 3.6.0 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

pardeep-singh on 25 Nov 2020

The above solution does work , however, the module bz2 is missing when pandas is imported into the script

samho93 on 25 Nov 2020

Pyenv: Unable to build Python on macOS Big Sur with Xcode 12 beta

Most helpful comment

All 80 comments

First Solution

Second Solution

Possible Solution (?)

sudo xcode-select --switch /Library/Developer/CommandLineTools

this is my Detailed process png

this my .zshrc config for readline zx zlib

this is my Detailed process png

this my .zshrc config for readline zx zlib

Related issues