Dvc: ImportError on `dvc pull` in Alpine env

[AlexJoz]

dvc pull fails in Docker based on Alpine 3.10 with latest dvc (0.53.1)

Traceback (most recent call last):
  File "/usr/bin/dvc", line 6, in <module>
    from dvc.main import main
  File "/usr/lib/python3.7/site-packages/dvc/__init__.py", line 11, in <module>
    import dvc.logger
  File "/usr/lib/python3.7/site-packages/dvc/logger.py", line 5, in <module>
    from dvc.utils.compat import str, StringIO
  File "/usr/lib/python3.7/site-packages/dvc/utils/__init__.py", line 29, in <module>
    from ruamel.yaml import YAML
ImportError: cannot import name 'YAML' from 'ruamel.yaml' (/usr/lib/python3.7/site-packages/ruamel/yaml/__init__.py)

Dockerfile to reproduce:

FROM alpine:3.10

RUN apk add --update build-base python3 python3-dev zlib-dev jpeg-dev linux-headers git bash

RUN pip3 install --no-cache-dir --upgrade pip
RUN pip3 install dvc[gs]
RUN dvc pull my.dvc

Going through history, found commit after which this error has appeared.
Basically, using ruamel.yaml ver. 0.15.9 fixes the issue.

Latest working tag:
pip3 install dvc[gs]==0.40.0

[efiop]

Hi @AlexJoz ! Thanks for reporting this! This is pretty weird, I am able to reproduce this with your dockerfile, but it works fine for me locally with the same yaml version. Looking into it.

[efiop]

Looks like the issue is not on our side, more tailored reproducer

FROM alpine:3.10

RUN apk add --update build-base python3 python3-dev zlib-dev jpeg-dev linux-headers git bash

RUN pip3 install --no-cache-dir --upgrade pip
RUN pip3 install ruamel.yaml
RUN python3 -c 'from ruamel.yaml import YAML'

shows that there is something wrong with ruamel itself. Mind opening an issue in ruamel too please?

[efiop]

For the record ruamel.yaml==0.15.100 works fine.

[AlexJoz]

@efiop, ok!
Thanks for a quick response =)

[ghost]

@efiop , could it be related to alpine using musl libc? :thinking:

[ghost]

@efiop, check tout he following: https://bitbucket.org/ruamel/yaml/commits/tag/0.16.0
split off C library into separate package

it doesn't seem like a relevant update.

@AlexJoz , a quick workaround would be to install ruamel.yaml before dvc, like the following:

FROM python:alpine

RUN apk add --update build-base python3 python3-dev zlib-dev jpeg-dev linux-headers git bash

RUN pip3 install --no-cache-dir --upgrade pip
RUN pip3 install 'ruamel.yaml==0.15.100'
RUN pip3 install dvc[gs]
RUN python3 -c 'from ruamel.yaml import YAML'

If you create an issue in the ruamel.yaml repo, please let us know :)
And, indeed, thanks for reporting this!

[AvdN]

This almost certainly has to do with ruamel.yaml having been split it up and the second package (ruamel.yaml.clib which contains the C loader/dumper) installing an __init__.py and .pth file (which it shouldn't).
It might be a few days before I can look into this, you should pin to "ruamel.yaml<0.16" until this is resolved.

[AlexJoz]

@AvdN Cool, thanx!

@mroutis
seems a bit late, but lets link: https://bitbucket.org/ruamel/yaml/issues/299 =))

[efiop]

@AvdN Thank you! :slightly_smiling_face:

@AlexJoz Btw, maybe you'd like to submit a quick fix for ruamel.yaml<0.16 in our setup.py? :wink: I'll release a new dvc version with it right after that.

[efiop]

@AlexJoz Big thanks for the fix! I'll trigger the release in a moment. Let's keep this issue open until ruamel.yaml is fixed, so we don't forget to update requirements in dvc :slightly_smiling_face:

[efiop]

@AlexJoz FYI: 0.53.2 is out. Please feel free to upgrade! :slightly_smiling_face:

[AlexJoz]

@efiop already done =) Works fine! Thanks

[AvdN]

I set the issue on bitbucket to resolved, as I think this got fixed by releasing a new version of ruamel.yaml.clib (you should use ruamel.yaml.clib>=0.1.2).
Unless you have pinned the previous version of ruamel.yaml.clib, clean rebuilding should now get a working version.

[efiop]

@AvdN Great news! Thank you so much!

[efiop]

It would be best to wait for the next ruamel.yaml release, which, I suppose, will have updated requirements for clib. But if it works for you @AlexJoz already, we could relax the requirements right away without waiting for the next release.

[AvdN]

@efiop There is a new ruamel.yaml. (0.16.1) which has that requirement

[efiop]

@AvdN Oh, that is perfect! Thank you!