Docker-alpine: Problem using PyInstaller on Alpine (musl)

[tomdee]

On project calico, we currently use a ubuntu base image but I'd love to swithc to alpine. The current stumbling block is with PyInstaller which doesn't seem to be compatible with libc.musl

The output I'm seeing from PyInstaller is below, and can also be seen at https://semaphoreci.com/calico/calico-docker--5/branches/alpine-build/builds/1

5 INFO: wrote /code/calicoctl.spec
18 INFO: UPX is not available.
30 INFO: Processing hook hook-os
97 INFO: Processing hook hook-time
98 INFO: Processing hook hook-cPickle
155 INFO: Processing hook hook-_sre
246 INFO: Processing hook hook-cStringIO
304 INFO: Processing hook hook-encodings
328 INFO: Processing hook hook-codecs
580 INFO: Extending PYTHONPATH with /code
581 INFO: checking Analysis
581 INFO: building Analysis because out00-Analysis.toc non existent
581 INFO: running Analysis out00-Analysis.toc
585 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/libpython2.7.so.1.0)
619 INFO: Analyzing /usr/lib/python2.7/site-packages/PyInstaller/loader/_pyi_bootstrap.py
627 INFO: Processing hook hook-os
636 INFO: Processing hook hook-site
645 INFO: Processing hook hook-encodings
704 INFO: Processing hook hook-time
706 INFO: Processing hook hook-cPickle
770 INFO: Processing hook hook-_sre
857 INFO: Processing hook hook-cStringIO
965 INFO: Processing hook hook-codecs
1323 INFO: Processing hook hook-pydoc
1419 INFO: Processing hook hook-email
1467 INFO: Processing hook hook-httplib
1489 INFO: Processing hook hook-email.message
1526 INFO: Analyzing /usr/lib/python2.7/site-packages/PyInstaller/loader/pyi_importers.py
1578 INFO: Analyzing /usr/lib/python2.7/site-packages/PyInstaller/loader/pyi_archive.py
1604 INFO: Analyzing /usr/lib/python2.7/site-packages/PyInstaller/loader/pyi_carchive.py
1630 INFO: Analyzing /usr/lib/python2.7/site-packages/PyInstaller/loader/pyi_os_path.py
1633 INFO: Analyzing calico_containers/calicoctl.py
1907 INFO: Processing hook hook-xml
1936 INFO: Processing hook hook-xml.sax
1991 INFO: Processing hook hook-pyexpat
2444 INFO: Processing hook hook-distutils
2671 INFO: Looking for run-time hooks
2676 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/datetime.so)
2678 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/site-packages/greenlet.so)
2681 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/site-packages/gevent/ares.so)
2683 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/site-packages/gevent/_util.so)
2686 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_codecs_tw.so)
2689 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/select.so)
2692 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_heapq.so)
2694 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/binascii.so)
2697 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/cPickle.so)
2701 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/unicodedata.so)
2710 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_bisect.so)
2712 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/strop.so)
2714 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_codecs_iso2022.so)
2717 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/site-packages/gevent/_semaphore.so)
2720 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/cStringIO.so)
2722 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/math.so)
2725 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_locale.so)
2728 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_collections.so)
2730 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/array.so)
2734 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_csv.so)
2737 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_codecs_hk.so)
2741 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_hashlib.so)
2744 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/bz2.so)
2747 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_ssl.so)
2752 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_io.so)
2754 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_json.so)
2757 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/resource.so)
2760 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/fcntl.so)
2764 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_ctypes.so)
2767 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/itertools.so)
2771 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/termios.so)
2774 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_codecs_kr.so)
2777 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/zlib.so)
2780 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/pyexpat.so)
2782 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/audioop.so)
2785 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/site-packages/gevent/core.so)
2787 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_functools.so)
2791 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_multibytecodec.so)
2794 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/operator.so)
2798 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_codecs_jp.so)
2801 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_socket.so)
2804 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_codecs_cn.so)
2808 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_struct.so)
2811 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/_random.so)
2814 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/grp.so)
2817 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/readline.so)
2819 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/python2.7/lib-dynload/time.so)
2821 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /lib/libz.so.1)
2833 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /lib/libcrypto.so.1.0.0)
2835 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/libbz2.so.1)
2838 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /lib/libssl.so.1.0.0)
2842 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/libffi.so.6)
2844 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/libexpat.so.1)
2847 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/libreadline.so.6)
2849 ERROR: Can not find libc.musl-x86_64.so.1 in path ldd (needed by /usr/lib/libncurses.so.5)
2851 INFO: Using Python library /usr/lib/libpython2.7.so.1.0
2875 INFO: Warnings written to /code/build/calicoctl/warncalicoctl.txt
2884 INFO: checking PYZ
2884 INFO: rebuilding out00-PYZ.toc because out00-PYZ.pyz is missing
2884 INFO: building PYZ (ZlibArchive) out00-PYZ.toc
6214 INFO: checking PKG
6214 INFO: rebuilding out00-PKG.toc because out00-PKG.pkg is missing
6214 INFO: building PKG (CArchive) out00-PKG.pkg
9318 INFO: checking EXE
9318 INFO: building because out00-EXE.toc missing or bad
9318 INFO: building EXE from out00-EXE.toc
9319 INFO: Appending archive to EXE /code/dist/calicoctl

You can repro by cloning calico-docker, checking out the alpine-build branch and typing make binary

[andyshinn]

Hi Tom, greating talking with you at DockerCon.

For this to work we might just need to symlink ld-musl-x86_64.so to /bin/ldd. Check out http://wiki.musl-libc.org/wiki/FAQ#Q:_where_is_ldd_.3F.

I was going to test this. But I can't actually find the alpine-build branch. Is https://github.com/Metaswitch/calico-docker the correct repository?

[tomdee]

Ah, it's on my own repo - here https://github.com/tomdee/calico-docker/tree/alpine-build

I'll try your fix when I get a chance, but I now realize that the binary that Pyinstaller builds will be dynamically linked with the musl libc and I want to run the binary on other linuxes.

So my hope of bulding in an Alpine linux container is probably a non-starter.

[tomdee]

Symlinking ld-musl-x86_64.so to /bin/ldd didn't work. I might try a glibc version of alpine

[tomdee]

After reading through https://github.com/gliderlabs/docker-alpine/issues/11 I see that trying to use glibc would be the wrong approach - I would still be bundling a python binary that depends on musl.

FTR - the problem can be repro-ed by checking out the above code (https://github.com/tomdee/calico-docker/tree/alpine-build) and running make binary

[tomdee]

I've just another quick look at this and I suspect it could be an issue with the Pyinstaller code.

Possibly the way they are looking at libs is incompatible with the dynamic behaviour of ldd on alpine

The Pytinstaller code I've looked at is here https://github.com/pyinstaller/pyinstaller/blob/dd1c17e7874e7b066d44b9fe4eaca80cb1baf96f/PyInstaller/depend/bindepend.py#L552

[tomdee]

Ah yes - so the problem is ldd producing output like this

ldd /lib/libssl.so.1.0.0
    ldd (0x7fce901fe000)
    libcrypto.so.1.0.0 => /lib/libcrypto.so.1.0.0 (0x7fce8fb75000)
    libc.musl-x86_64.so.1 => ldd (0x7fce901fe000)
    libz.so.1 => /lib/libz.so.1 (0x7fce8f95f000)

Pyinstaller is trying to find the path to libc.musl-x86_64.so.1 and finding that ldd isn't a path that exists

[tomdee]

It seems I can fix it with ln -s /lib/libc.musl-x86_64.so.1 ldd. The binary that gets produced still doesn't run though as it tries to find some libs in /lib64. This is resolved with ln -s /lib /lib64

[andyshinn]

Interesting. Thanks for the workarounds. Does Pyinstaller pull in binaries that are compiled against glibc and looking in /lib64? If Pyinstaller is looking at ldd output and making assumptions based upon it, maybe filing a bug upstream to support musl output as well might be a good idea. But if binaries are being pulled in that expect libs in certain places at /lib64 then we may just have to deal with the symlink workaround.

[andyshinn]

I'm going to close this issue as there sounds like a workaround. Let me know if otherwise and I'm happy to try and debug some more.

[tomdee]

The workaround isn't pretty, but does seem to work - FTR I'm using it here https://github.com/projectcalico/k8s-policy/pull/10

[six8]

I found a good solution.

• Build a pyinstaller bootloader with Alpine -- This gets rid of all need to symlink lib64 and ld-linux-x86-64.so.2
• Wrap ldd to output "/lib/libc.musl-x86_64.so.1" instead of "ldd" -- This gets rid of need for ldd symlink
• Use official Python Alpine Docker image -- Using alpine:3.3 and then installing python with apk had issues with segfaulting. I don't think the python apk has all the correct compiler options.

The end result is I can create single file Python apps that work on other Alpine hosts (even non-Docker ones).

I packaged all this up for easy explanation and re-use https://github.com/six8/pyinstaller-alpine

[jessesuen]

@six8 than you for that workaround! I don't think I would have been able to figure that one out myself. BTW, I don't know if something was fixed in alpine:3.4, but so far, my pyinstaller bundled app does not segfault when python is installed via APK vs. using the official python docker image as you suggested. It might be worth retrying in alpine:3.4. This lets me have both python 3 and python 2 installed in a single pyinstaller container and decide which version of python to bundle the app with at runtime.

[mattrasband]

If you use setup.py (instead of requirements.txt) and/or python 3.5 - I am pushing a version for that nerdwaller/pyinstaller-alpine:py3

It still defaults to requirements.txt, but if it is not found it looks for setup.py and does pip install .