Cibuildwheel: auditwheel: error: cannot repair "...cp37m-linux_x86_64.whl" to "manylinux2010_x86_64"

[horta]

The relevant bit:

    + rm -rf /tmp/cibuildwheel/built_wheel
    + mkdir -p /tmp/cibuildwheel/built_wheel
    + pip wheel /project -w /tmp/cibuildwheel/built_wheel --no-deps
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/opt/_internal/cpython-3.7.7/bin/python3.7 -m pip install --upgrade pip' command.
Processing /project
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: bgen-reader
  Building wheel for bgen-reader (PEP 517): started
  Building wheel for bgen-reader (PEP 517): finished with status 'done'
  Created wheel for bgen-reader: filename=bgen_reader-4.1.0-cp37-cp37m-linux_x86_64.whl size=103211 sha256=fee846b9b912bee0451c8e6804a0f2e984e5a44cea8398559ab4ed0b348f9e7b
  Stored in directory: /tmp/pip-ephem-wheel-cache-dqhl06it/wheels/84/ce/ef/3768e7d143f298e3f79dda4c505f7ffa005f429720ee7533a3
Successfully built bgen-reader
    + /opt/python/cp38-cp38/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)'
    + rm -rf /tmp/cibuildwheel/repaired_wheel
    + mkdir -p /tmp/cibuildwheel/repaired_wheel
    + sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/bgen_reader-4.1.0-cp37-cp37m-linux_x86_64.whl'
INFO:auditwheel.main_repair:Repairing bgen_reader-4.1.0-cp37-cp37m-linux_x86_64.whl
usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: cannot repair "/tmp/cibuildwheel/built_wheel/bgen_reader-4.1.0-cp37-cp37m-linux_x86_64.whl" to "manylinux2010_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.
cibuildwheel-6341f69c-b8bf-4da6-9ff8-9b87f7e381eb
Command ['sh', '-c', 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/bgen_reader-4.1.0-cp37-cp37m-linux_x86_64.whl'] failed with code 2. 
The command "python3 -m cibuildwheel --output-dir wheelhouse" exited with 1.

Link: https://travis-ci.com/github/limix/bgen-reader-py/jobs/370485778

It seems to have happened after I added pyproject.toml to my project.

[horta]

Related to issue #247

[horta]

Sorry for this issue. The problem was a C file leftover created by cffi when testing my package on the travis machine, then going to manylinux container without cleaning that up first.

[YannickJadoul]

Huh, OK, I though we had a check to print a better error message in such a case, somewhere.
Do you know what exactly was left behind, and why it wasn't manylinux-compatible (I'd think a C file would be fine, since it still needs to be compiled) ?

Thanks for reporting the solution, btw! :-)

[horta]

Hi @YannickJadoul . Thanks a lot for the looking into it. That is what happened.

Steps to the error

My package has C extension built using CFFI python package: https://github.com/limix/bgen-reader-py/blob/master/build_ext.py

Before the usual .travis.yml lines:

script:
  - python3 -m cibuildwheel --output-dir wheelhouse
after_success:
  - |
    if [ "${TRAVIS_TAG:-}" != "" ]; then
      python3 -m pip install twine -U
      python3 -m twine upload --skip-existing wheelhouse/*
    fi

I do some tests on my package, which involves in build_ext'ng it. By doing it, CFFI produces a C file inside my package gluing all the things together, which is based on the current the environment (i.e., outside of the manylinux images).

When then entering into the cibuildwheel pipeline, build_ext is ran again but this time CFFI believes that the gluing-C file is up-to-date, instead of generating a new one based on the current manylinux environment. This must have caused the error:

auditwheel: error: cannot repair "/tmp/cibuildwheel/built_wheel/bgen_reader-4.1.0-cp37-cp37m-linux_x86_64.whl" to "manylinux2010_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

Solution

My solution was simply to issue an git clean -xdfq before entering into the cibuildwheel pipeline.

I'm not sure if there is something that cibuildwheel could do about it.

[YannickJadoul]

Hi, @horta. Thanks for the extra info!
I'm still surprised a C file can be compiled inside the manylinux but not be manylinux-compatible. I'd expect it would be a stale object file/shared library remaining, rather than a source file causing trouble. There's not enough output to see from Travis, but could it be that it's actually picking up on the previously generated libbgen.so?

@joerick Does the linux.py troubleshoot function still work after #386? I have a sneaking suspicion that this second part of if (isinstance(error, subprocess.CalledProcessError) and 'exec' in error.cmd): might not work anymore, but you're the expert on the DockerContainer functionality.

(reopening until this last part is figured out)

[horta]

I see what you mean. Here is the verbose=3 travis output: https://travis-ci.com/github/limix/bgen-reader-py/jobs/370550804

Have a look at the line https://travis-ci.com/github/limix/bgen-reader-py/jobs/370550804#L3649 onward:

running build_ext
generating cffi module 'build/temp.linux-x86_64-3.7/bgen_reader._ffi.c'
already up-to-date
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/bgen_reader
copying build/lib.linux-x86_64-3.7/bgen_reader/_testit.py -> build/bdist.linux-x86_64/wheel/bgen_reader
(...)
copying build/lib.linux-x86_64-3.7/bgen_reader/_ffi.abi3.so -> build/bdist.linux-x86_64/wheel/bgen_reader

What the above tells me is that CFFI checked that (inside manylinux) bgen_reader._ffi.c is up-to-date (this is CFFI-generated file I told you about) and therefore skipped the actual building of the _ffi.abi3.so file. That .so file was compiled outside the manylinux environment (and even linked to libpython3.7m.so.1.0, which was later not found in the manylinux environment and then causing an error).

Do you think this is more an user problem, CFFI or cibuildwheel?

It feels to me that is more an user problem but I have to admit that I spent around 4 hours trying to figure it out.

[YannickJadoul]

Thanks for checking that! :-) So it is a combination of effects, indeed, but there is a .so file compiled outside of the manylinux image.

Do you think this is more an user problem, CFFI or cibuildwheel?

CFFI is definitely not to blame, I believe. You're intuition is right: it's a combination of the user and cibuildwheel. The thing is, we had some functionality where cibuildwheel would warn you, after #139 (i.e., PR #263). But now that you've confirmed that there was a .so file, I'm even more convinced that #386 broke this warning. So let's share blame ;-)

[Czaki]

I think that the problem comes from using same job for test and wheel build. Best option is to define separate job for this. Then building wheel starts on clean project.

[horta]

That actually makes a lot of sense, @Czaki . It will prevent further problems in the future, thanks!

[YannickJadoul]

Yes, @horta found a solution. But we used to have a warning if it goes wrong, and that seems to be broken now. That's why I reopened and asked @joerick.

[joerick]

Thanks for the prod @YannickJadoul - troubleshoot() was indeed broken, but has now been fixed, and a test will ensure it doesn't happen again :)

[YannickJadoul]

Thanks for sticking along and helping debugging, @horta! :-)