Pyenv: Python 3.7 needs libffi headers to build on Linux

[legnaleurc]

I'm building on Raspbian, but this could apply to all Linux distro.

Original issue: https://bugs.python.org/issue31652

They bundled libffi prior to 3.7, now they removed it from Linux build.
Mac OS X and Windows has not been affected yet but it's in their plan.

Probably a good thing to add to pyenv-doctor or common build problem.

[knodir]

Applies to Ubuntu 16.04. pyenv install 3.7.0 failed with

$ pyenv install 3.7.0
Downloading Python-3.7.0.tar.xz...
-> https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
Installing Python-3.7.0...

BUILD FAILED (Ubuntu 16.04 using python-build 1.2.5-4-gc057a80)

Inspect or clean up the working tree at /tmp/python-build.20180704120702.24198
Results logged to /tmp/python-build.20180704120702.24198.log

Last 10 log lines:
  File "/tmp/tmp4933xm_8/pip-10.0.1-py2.py3-none-any.whl/pip/_internal/__init__.py", line 42, in <module>
  File "/tmp/tmp4933xm_8/pip-10.0.1-py2.py3-none-any.whl/pip/_internal/cmdoptions.py", line 16, in <module>
  File "/tmp/tmp4933xm_8/pip-10.0.1-py2.py3-none-any.whl/pip/_internal/index.py", line 25, in <module>
  File "/tmp/tmp4933xm_8/pip-10.0.1-py2.py3-none-any.whl/pip/_internal/download.py", line 39, in <module>
  File "/tmp/tmp4933xm_8/pip-10.0.1-py2.py3-none-any.whl/pip/_internal/utils/glibc.py", line 3, in <module>
  File "/tmp/python-build.20180704120702.24198/Python-3.7.0/Lib/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
Makefile:1122: recipe for target 'install' failed
make: *** [install] Error 1

Installation succeeded after running sudo apt-get install libffi-dev (as suggested here). I did not try with libffi itself though; could work.

[hjarvard]

I've got libffi6 (there are no libffi package) installed on debian 9.5 system and I've got the same error message. After installing libffi-dev package eveerything worked. The required packaged are listed in the Common build problems page though.

[aoindustries]

We confirm the same required for CentOS 7 builds:

sudo yum install libffi-devel

[joshfriend]

The wiki suggests installing libffi on linux, but it was only mentioned under the CentOS/Fedora section. I added it to the list of packages to install on Ubuntu as well. If something is sill missing, please edit the wiki to include it

[zachliu]

Confirmed @knodir on Linux Mint 19 with kernel 4.15.0-33-generic:

sudo apt install libffi-dev

[dgspahn]

Thank you. That module _ctypes was giving me fits.

[mskoenz]

Hi all!
Same issue, but regrettably no sudo environment to just install libffi. I manually installed libffi to my ~/.local folder and added the corresponding paths for CFLAGS, LDFLAGS, CPLUS_INCLUDE_PATH, C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, LD_LIBRARY_PATH

running pyenv install 3.7.0 fails, and the log shows this as first error:

/usr/bin/install -c python-config.py /users/mskoenz/.pyenv/versions/3.7.0/lib/python3.7/config-3.7m-x86_64-linux-gnu/python-config.py
/usr/bin/install -c python-config /users/mskoenz/.pyenv/versions/3.7.0/bin/python3.7m-config
./python -E ./setup.py install \
    --prefix=/users/mskoenz/.pyenv/versions/3.7.0 \
    --install-scripts=/users/mskoenz/.pyenv/versions/3.7.0/bin \
    --install-platlib=/users/mskoenz/.pyenv/versions/3.7.0/lib/python3.7/lib-dynload \
    --root=/
running install
running build
running build_ext
INFO: Could not locate ffi libs and/or headers

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Failed to build these modules:
_ctypes

grep ffi on the log yields:

checking for suffix of executables...
checking for suffix of object files... o
checking for --with-suffix...
checking for --with-system-ffi... yes
checking for sched_setaffinity... yes
INFO: Could not locate ffi libs and/or headers
INFO: Could not locate ffi libs and/or headers
INFO: Could not locate ffi libs and/or headers

I tried CONFIGURE_OPTS="--with-ffi=..." python install 3.7.0 in the CONFIGURATION_OPTS but that was not recognized...

Is there a way to teach pyenv where the libffi headers are?

[b1tstreamalpha]

Hi all!
Same issue, but regrettably no sudo environment to just install libffi. I manually installed libffi to my ~/.local folder and added the corresponding paths for CFLAGS, LDFLAGS, CPLUS_INCLUDE_PATH, C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, LD_LIBRARY_PATH
...
Is there a way to teach pyenv where the libffi headers are?

Try setting your PKG_CONFIG_PATH variable to include the path where libffi.pc was installed. It will be something like /your_ffi_install_path/lib/pkgconfig/libffi.pc. This worked for me, although I am building python3 from a source tarball. (In my case I can see that ./configure is using pkg-config to locate the libffi headers).

[DarkStar1]

I also have this problem with trying to install pip for Python 3.7.1rc1. I installed python 3.7.1rc1 from the deadsnakes repo as described here on SO (because the build instructions from the official python page does not succeed for the same reasons), so far installing libffi-dev and installing all packages listed in the common build problems wiki page as pointed to by @hjarvard in an earlier comment do not work.
I am installing on ubuntu 16.04.5 LTS. WOuld apprecate any pointers thanks.

[dgspahn]

Skip it and install Linux Mint 19.

[joshfriend]

@dgspahn just so you know, your reply included your phone number, which I edited out just to be safe. Probably has something to do with replying to the issue via email and your mail client tacked it on.

[ragreddy]

@b1tstreamalpha Even after adding package config path, I am seeing the issue.
Do you happen to have commands for configure
and
any variables (pkg_config_path) you exported?
I am trying to build 3.7.1 from source tar ball.

[Azulinho]

similar problem on archlinux, for some reason I was missing the pkgconf package.
sudo pacman -S pkgconf libffi and pyenv install 3.7.x worked

[haroldduan]

Confirmed,on Linux Mint 19 with Kernel Linux 4.15.0-46-generic x86_64;

sudo apt install libffi-dev

[GRRedWings]

@DarkStar1 did you ever figure out your issue? I'm having the same issue on Ubunto 16.04LTS and trying to figure out what to try next.

[DarkStar1]

@GRRedWings Actually we dumped 16.04 and switched our VMs to the next LTS (18.04) instead. We didn't have enough time to spend investigating and resolving the issue.

[GRRedWings]

@DarkStar1 Since that cleaned it up, I'll do the same.

[gchamon]

@Azulinho installing pkgconf seems to have fixed the issue we were having compiling python in archlinux/base-based docker image. May I ask how did you come across the answer?

[Azulinho]

sorry @gchamon I can barely remember what happened last Friday,
as of this from 5 months ago I can only say I switched it off an on again

[gchamon]

thats ok, I was really curious because lots of people mentioned libffi but you were the only one to come up with pkgconf

[alvations]

Maybe adding this information on sudo apt install libffi-dev on the README.md might be good ;P

[makslevental]

anyone have some advice on this? i'm on a system where i don't have sudo so i can't yum install. i downloaded and unpacked the libff-devel rpm and tried to run

CFLAGS=-I/home/me/libffi/include \
LDFLAGS=-L/home/me/libffi/lib64 \
pyenv install -v 3.7.4

and i still get the same missing ctypes error.

[JohnWSteill]

Yeah I just got it after like 12h of struggle!

I had to install the libffi, and make sure i linked to both the libraries and the libffi.pc file.

./configure --prefix=$PY_HOME/python LDFLAGS=-L$WKSP/lib64 PKG_CONFIG_PATH=$WKSP/lib/pkgconfig --enable-shared

entire script:

export WKSP=/tmp/steill_vim_py374_install
export PY_HOME=$WKSP/P374
export VIM_HOME=$WKSP/vim
export LD_LIBRARY_PATH=$WKSP/lib:$WKSP/lib64:$PY_HOME/python/lib:/isitools/gcc4.9.4/lib64
rm -rf $WKSP # DANGER! Make sure $WKSP isn't something silly
mkdir $WKSP
mkdir $PY_HOME
mkdir $VIM_HOME

0.5) Install libffi (Didn't work)

cd $WKSP
wget ftp://sourceware.org/pub/libffi/libffi-3.2.tar.gz
tar -xzf libffi-3.2.tar.gz
cd libffi-3.2/
./configure --prefix=$WKSP
make && make install

1) Install Python

cd $PY_HOME
wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
tar zxfv Python-3.7.4.tgz
find . -type d | xargs chmod 0755
cd Python-3.7.4
./configure --prefix=$PY_HOME/python \
LDFLAGS=-L$WKSP/lib64 \
PKG_CONFIG_PATH=$WKSP/lib/pkgconfig \
--enable-shared

--enable-shared req'd to get python loading in vim working

--without-ensurepip

--without-ensurepip needed to work around "missing _ctypesi module" - no libffi

https://bugs.python.org/issue35170#msg329384

make && make install

2) Install vim

We need to use python for the vim installation:

export PATH=$PY_HOME/python/bin/:$PATH
export PYTHONPATH=$PY_HOME/python/bin/

cd $VIM_HOME
git clone https://github.com/vim/vim.git
cd vim
./configure LDFLAGS="-L$PY_HOME/python/lib/" --enable-gui=no --prefix=$VIM_HOME \
--with-features=huge --disable-pythoninterp --enable-python3interp=yes \
--with-python3-command=python3.7 \
--with-python3-config-dir=$PY_HOME/python/lib/python3.7/config-3.7m-x86_64-linux-gnu \
--enable-fail-if-missing --enable-multibyte
make && make install

3) Setup environment

echo "PATH=$PY_HOME/python/bin/:$PATH" >> $HOME/bin/Bash_Profile/bash_profile
echo "export PYTHONPATH=$PY_HOME/python/bin/" >> $HOME/bin/Bash_Profile/bash_profile
echo "export LD_LIBRARY_PATH=$WKSP/lib:$WKSP/lib64:$PY_HOME/python/lib:/isitools/gcc4.9.4/lib64" \

$HOME/bin/Bash_Profile/bash_profile
alias vi=$VIM_HOME/bin/vim
echo "alias vi=$VIM_HOME/bin/vim" >> $HOME/bin/Bash_Profile/alias_and_env_profile

[wincle]

@JohnWSteill Thanks! it help me a lot!

[tommyjcarpenter]

@gchamon I agree with your comment RE @Azulinho exactly. Even after installing libffi on Arch, and reinstalling python 3.7.6 on Arch, it still bombed. However, installing both pkgconf AND libffi, then reinstalling, worked!

[mjsteinbaugh]

@tommyjcarpenter Thanks for the heads up on pkgconf with Arch...I had trouble figuring this one out too.

[kevinfreeman1101]

Using pyenv, I struggled with this for hours, but finally had success after switching from pyenv to conda:

For brevity, I only included output from installing psycopg2 using "dev_html" as my virtualenv name
pyenv deactivate
conda create -n dev_html
conda activate dev_html
conda install django
conda install libffi
conda install psycopg2

Collecting package metadata (current_repodata.json): done
Solving environment: done

Package Plan

environment location: /home/kevin/anaconda3/envs/dev_html

added / updated specs:
- psycopg2

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
certifi-2019.11.28         |           py38_0         153 KB
libpq-11.2                 |       h20c2e04_0         2.0 MB
pip-20.0.2                 |           py38_1         1.7 MB
psycopg2-2.8.4             |   py38h1ba5d50_0         171 KB
python-3.8.1               |       h0371630_1        49.5 MB
setuptools-46.0.0          |           py38_0         522 KB
wheel-0.34.2               |           py38_0          51 KB
------------------------------------------------------------
                                       Total:        54.0 MB

The following NEW packages will be INSTALLED:

ca-certificates pkgs/main/linux-64::ca-certificates-2020.1.1-0
certifi pkgs/main/linux-64::certifi-2019.11.28-py38_0
krb5 pkgs/main/linux-64::krb5-1.16.4-h173b8e3_0
ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
libedit pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
libpq pkgs/main/linux-64::libpq-11.2-h20c2e04_0
ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_0
openssl pkgs/main/linux-64::openssl-1.1.1d-h7b6447c_4
pip pkgs/main/linux-64::pip-20.0.2-py38_1
psycopg2 pkgs/main/linux-64::psycopg2-2.8.4-py38h1ba5d50_0
python pkgs/main/linux-64::python-3.8.1-h0371630_1
readline pkgs/main/linux-64::readline-7.0-h7b6447c_5
setuptools pkgs/main/linux-64::setuptools-46.0.0-py38_0
sqlite pkgs/main/linux-64::sqlite-3.31.1-h7b6447c_0
tk pkgs/main/linux-64::tk-8.6.8-hbc83047_0
wheel pkgs/main/linux-64::wheel-0.34.2-py38_0
xz pkgs/main/linux-64::xz-5.2.4-h14c3975_4
zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3

Proceed ([y]/n)? y

Downloading and Extracting Packages
certifi-2019.11.28 | 153 KB | ################################################################################################# | 100%
setuptools-46.0.0 | 522 KB | ################################################################################################# | 100%
pip-20.0.2 | 1.7 MB | ################################################################################################# | 100%
wheel-0.34.2 | 51 KB | ################################################################################################# | 100%
psycopg2-2.8.4 | 171 KB | ################################################################################################# | 100%
libpq-11.2 | 2.0 MB | ################################################################################################# | 100%
python-3.8.1 | 49.5 MB | ################################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

[slowkoni]

If you are doing something nobody here will listen you about because "you're doing it the wrong way", but you have to do it "the wrong way" for reasons too asinine (for instance, in my case it quickly degrades into foul words about somebody on devops team overweight mother), you need to first:

Get libffi and install it into your user install area the usual way.

git clone https://github.com/libffi/libffi.git
cd libffi
./configure --prefix=path/to/your/install/root
make
make install

Then go back to your Python 3 source and find this part of the code in setup.py at the top level of the python source directory

        ffi_inc = [sysconfig.get_config_var("LIBFFI_INCLUDEDIR")]
        if not ffi_inc or ffi_inc[0] == '':
            ffi_inc = find_file('ffi.h', [], inc_dirs)
        if ffi_inc is not None:
            ffi_h = ffi_inc[0] + '/ffi.h'
            if not os.path.exists(ffi_h):
                ffi_inc = None
                print('Header file {} does not exist'.format(ffi_h))
        ffi_lib = None
        if ffi_inc is not None:
            for lib_name in ('ffi', 'ffi_pic'):
                if (self.compiler.find_library_file(lib_dirs, lib_name)):
                    ffi_lib = lib_name
                    break

        ffi_lib="ffi"  # --- AND INSERT THIS LINE HERE THAT DOES NOT APPEAR ---
        if ffi_inc and ffi_lib:
            ext.include_dirs.extend(ffi_inc)
            ext.libraries.append(ffi_lib)
            self.use_system_libffi = True

and add the line I have marked above with the comment. Why it is necessary, and why there is no way to get configure to respect '--without-system-ffi` on Linux platforms, perhaps I will find out why that is "unsupported" in the next couple of hours, but everything has worked ever since. Otherwise, best of luck... YMMV.

WHAT IT DOES: just overrides the logic there and causes the compiler linking command to add "-lffi" which is all that it really needs. If you have the library user-installed, it is probably detecting the headers fine as long as your PKG_CONFIG_PATH includes path/to/your/install/root/lib/pkgconfig.

[learnedtocode]

Here is what worked for me, I don't have sudo access either, and I can't use conda because I need Python 3.8

Go to https://rpmfind.net/linux/rpm2html/search.php?query=libffi.so&submit=Search+...&system=&arch= and pick your platform and architecture.
For CentOS 7 x86_64: https://rpmfind.net/linux/centos/7.8.2003/os/x86_64/Packages/libffi-devel-3.0.13-19.el7.i686.rpm
The link is slightly wrong. Editing it manually to say x86_64 in the filename works: https://rpmfind.net/linux/centos/7.8.2003/os/x86_64/Packages/libffi-devel-3.0.13-19.el7.x86_64.rpm

Unpack this package to, $HOME/lib/libffi-devel-3.0.13-19.el7.x86_64
These two files should exist now:

$HOME/lib/libffi-devel-3.0.13-19.el7.x86_64/usr/include/ffi.h
$HOME/lib/libffi-devel-3.0.13-19.el7.x86_64/usr/lib64/libffi.so

But there is a problem, $HOME/lib/libffi-devel-3.0.13-19.el7.x86_64/usr/lib64/libffi.so is a link to a missing file. This fixes it for me since libffi was already installed, just not the development files

ln -s /lib64/libffi.so.6.0.1 $HOME/lib/libffi-devel-3.0.13-19.el7.x86_64/usr/lib64/

Then in the shell

pyenv uninstall 3.8.3
LDFLAGS=-L$HOME/lib/libffi-devel-3.0.13-19.el7.x86_64/usr/lib64 CPPFLAGS=-I$HOME/lib/libffi-devel-3.0.13-19.el7.x86_64/usr/include pyenv install -v 3.8.3

This works because of https://github.com/python/cpython/blob/007bb06a2de9e64fa978f5dd9131d0100227b4cf/setup.py#L723-L740

What was failing for me before was creating and activating a virtualenv, then running pip install PyYAML==5.3.1 (actually a package that depends on it, but same difference). This now succeeds.