Python: [3-alpine] shutil.copytree fail to copy a tree with a broken symbolic link

[fruch]

When trying to use pre-commit hock from this repo:
https://github.com/pre-commit/pre-commit-hooks

pip failed when using the shutil.copytree

I've narrowed it to this example, it passed on 2.7-alpine, but fails on 3.5-alpine and 3.6-alpine

FROM python:3.6-alpine

RUN apk update
RUN apk add git

RUN mkdir /bug

RUN cd /bug && ln -s /broken_path/to_nowhere broken
RUN python -c "import shutil; shutil.copytree('/bug', '/temp', symlinks=True)"

EDIT:
After playing around, I've nailed it to this command failing

>>> os.chmod('/bug/broken', 511, follow_symlinks=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 95] Not supported: '/bug/broken'

shutil was expecting NotImplementedError

this seems related
https://bugs.python.org/issue17076

[kevin-brown]

PYTHON-17076 seems to indicate that the bug should have been fixed in Python 3.4, so either it regressed or you're running into a new, but still similar, bug with Alpine.

[fruch]

I've opened a different one, it seems to fail on a different call, https://bugs.python.org/issue28627

[AnalogJ]

I just ran into this issue as well when using pyenv and tox :/

Does anyone have a workaround?

[fruch]

unfortunately no, I've stopped using alpine for python3
(other workaround is to remove broken symlinks)

I didn't had time to try to suggest a patch... i'm not sure I've identified the exact origin of the problem

[zancas]

I've also hit this issue with alpine/python3.

[fruch]

@zancas you were using yelp precommit ?

[zancas]

@fruch
I was not using yelp precommit.
I was cloning the ansible repo into my container, and running pip install . inside the top level directory (the distribution root) for ansible.

[zancas]

OK @fruch I've recappred this much of your experiment:

docker run -it python:3.6.1-alpine3.6 sh
/ # ln -s /nowhere badlink
/ # python -c 'import os; os.chmod("badlink", 511, follow_symlinks=False)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OSError: [Errno 95] Not supported: 'badlink'

[asottile]

fwiw, I've been making some headway here: https://github.com/pre-commit/pre-commit/issues/655

[asottile]

I've submitted a patch to cpython: https://github.com/python/cpython/pull/4267

[fruch]

looks good, I've kinda left it in the middle...

[asottile]

@fruch if the patch looks correct, feel free to review it, this seems to expedite the cpython process even if the reviewer is an outsider

[fruch]

tried, still need a core-dev to review

[asottile]

indeed, but adding an outside review changes the tags to [awaiting core reviewer] which seems to be slightly better 🤷‍♂️

[jmagnusson]

@asottile's new patch https://github.com/python/cpython/pull/4783 seems to fix tox inside python alpine, yay!

Here's a Dockerfile which recreates this error.

FROM python:3.6.3-alpine3.6
WORKDIR '/app'
ENTRYPOINT ["tox"]
RUN pip install tox
RUN mkdir myapp && echo 'print(1)' > myapp/__init__.py
RUN mkdir tests && echo $'def test():\n\
    assert True' > tests/test_myapp.py

RUN echo $'[tox]\n\
[testenv]\n\
commands = pytest tests\n\
deps = .' > tox.ini
RUN echo $'from setuptools import setup, find_packages\n\
setup(\n\
    name="myapp",\n\
    install_requires=["pytest"],\n\
    packages=find_packages()\n\
)' > setup.py

Running above Dockerfile confirms that we get the copytree error:

docker build -t mvce
docker run -it --rm mvce
# ...
#  File "/app/.tox/python/lib/python3.6/shutil.py", line 359, in copytree
#    raise Error(errors)
# shutil.Error: [('/app/.tox/python/lib/python3.6/imp.py',
# ...

Now if we change the FROM in above dockerfile to jacobsvante/python:3.7.0a3-alpine3.6-with-bpo-31940-lchown-fix the error doesn't appear. Thanks @asottile!

[tianon]

Closing, given that this is an issue with Python itself (not the image), and that it appears to be solved. :+1:

[asottile]

Unfortunately it hasn't been merged yet but I do agree with it being an upstream issue :)