Dvc: Adding file to dvc fails with Unknown argument(s): {'lock_args': (False,)}

[nbeuchat]

When adding a file with dvc add myfile or dvc push, I get the following error:

ERROR: unexpected error - "Unknown argument(s): {'lock_args': (False,)}"

It used to work perfectly with the same version but we started getting this error today. Any idea what this could be?

Config

['remote "livebucket"']
url = s3://****/***
[core]
remote = livebucket

Please provide information about your setup
DVC version: 0.80.0
Platform: Ubuntu
Install method: pip install in conda environment

DVC version(i.e. dvc --version), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))

[efiop]

Hi @nbeuchat ! Could you please show verbose log (e.g. dvc add -v)?

Also, 0.80 is quite old, could you please upgrade to the latest version of dvc and see if the issue persists?

[efiop]

Another user getting this same error https://discordapp.com/channels/485586884165107732/485596304961962003/691786988235391028

[efiop]

@nbeuchat Looks like you simply need to upgrade dvc (or tqdm): https://discordapp.com/channels/485586884165107732/485596304961962003/691794435029205093 . Closing.

[efiop]

@casperdcl Could you take a look please?

[casperdcl]

@george-gca in your trace:

...
File "/snap/dvc/388/lib/python3.6/site-packages/dvc/progress.py", line 90, in init
**kwargs
File "/home/gca/.local/lib/python3.6/site-packages/tqdm/_tqdm.py", line 860, in init
...

This makes no sense. Your isolated snap (/snap/dvc/388/...) is then using your system's tqdm (/home/...)? Really? If that's the case seems like a major upstream bug with snap.

Could you try snap remove dvc && snap install dvc?

In any case /home/gca/.local/lib/python3.6/site-packages/tqdm/_tqdm.py", line 860 is definitely an old version of tqdm<4.36.0.

[george-gca]

@casperdcl I don't really know if it is a snap bug since to install dvc through snap I need to enable the --classic flag. I did the snap remove dvc && snap install dvc but to no avail.

I figured out that the tqdm lib that I updated was actually inside a virtualenv. So I deactivated the virtualenv, tried to update it again, but it said it was already updated. Then I found out that the dvc uses python3.6, but my system default is python3.7, so I had to update the tqdm lib for the specific python version with python3.6 -m pip install --user --upgrade tqdm. That fixed it.

Nevertheless, I agree that the snap version shouldn't rely on a lib outside its scope, specifically if it is not my system's default.

[casperdcl]

thanks @george-gca would be great if you could reproduce this on a different system. There are two issues, I'm more concerned with the second:

1. Virtualenvs interfering with each other/misconfigured
   
   
   
   • not too worried about this; it's just a standard pain we all have
   
   
2. Snap packages trying to import modules outside the snap directory
   
   
   
   • really not ok. Even with --classic.
   
   

[george-gca]

@casperdcl by different system you mean distro? I recently installed Ubuntu Desktop 20.04 on my laptop, installed dvc package via snap, and everything worked fine. But it is part of my default setting to install the tqdm package in default python version since I use it in some of my scripts. Don't know if if I haven't done this it would work though.

The thing is, the other system where I installed dvc is my deep learning machine that is remote, and probably was configured a while ago. Likewise, all the packages (like tqdm and on) were outdated. Maybe in a "fresh install" of Ubuntu 18.04 things will work fine. Would you like me to test this in a virtual machine?

[casperdcl]

No by different system I mean different physical computer (or reinstall the OS on the same computer) - i.e. something that may be reproducible. My feeling is this issue is not reproducible and just exists on one really broken system.

AFAIK snap packages should not be looking for import locations on the system. I'm sure the snap team would be very interested if this was reproducible.

To be clear, if you can install 20.04 from scratch, pip install tqdm<4 (or any other outdated dvc dependency), and then snap install dvc and somehow find dvc commands complaining about outdated system pacakges - that's a big problem.

[george-gca]

Ok, since I can't format the remote pc I will test this in a virtual machine and report back.

[efiop]

Another user running into this issue in snap
https://discordapp.com/channels/485586884165107732/485596304961962003/713255811794731048 . Escalating the priority...

[casperdcl]

is it a similar stack trace to the one posted here? Is it reproducible in a clean env? If so could open an issue with snapcraft...

[casperdcl]

Finally got on the Discord bandwagon. Posting stack trace here:

$ dvc get https://github.com/iterative/dataset-registry get-started/data.xml
...
ERROR: unexpected error - "Unknown argument(s): {'lock_args': (False,)}"
Exception ignored in: <object repr() failed>
$ dvc version
...
ERROR: unexpected error - Version mismatch: this is the 'cffi' package version 1.14.0, located in '/snap/dvc/429/lib/python3.6/site-packages/cffi/api.py'.  When we import the top-level '_cffi_backend' extension module, we get version 1.12.2, located in '/home/utkarsh/.local/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so'.  The two versions should be equal; check your installation.

Looks like it may be something to alert snapcraft about? Or it could be that their python plugin is broken. Even for --classic confinement, it really should default to looking for internal libraries first. On my machine, $PATH contains conda/virtualenvs first before /snap/bin yet I've never encountered this issue.

[casperdcl]

Reported here: https://bugs.launchpad.net/dvc/+bug/1880319, feel free to bump

[sergiusens]

Are there any measures to take https://github.com/iterative/dvc/blob/master/bin/dvc into consideration? This should be running $SNAP/usr/bin/python3 instead of just python from PATH unless PATH is preset to prefer python in $SNAP/usr/bin, but you may or may not want to set that depending on if you want to call system binaries or not.

[casperdcl]

@sergiusens I'm not sure that's the problem. A plugin: python snap should not be trying to use the system's PATH over its own internal python interpreter. Also:

$ which python
/home/user/miniconda/bin/python
$ python --version
Python 2.7.15
$ python -m dvc version
/home/user/miniconda/bin/python: No module named dvc
$ which dvc
/snap/bin/dvc
$ dvc version
DVC version: 1.0.0a5
Python version: 3.6.9
...
$ echo $PATH
...:/home/user/miniconda/bin:...:/snap/bin:...

clearly indeed the snap is picking up its own internal python and not the system's python even though the user's PATH has the system python first.

Also from https://github.com/iterative/dvc/blob/2fecc66cb96442a8bf5d296e71a72766341d6e35/bin/dvc#L3 note that $DVC_HOME should be blank inside the snap.

[sergiusens]

Oh, thanks for the clarification, I took a closer look, and the snap issue in the thread, it is not system python that is getting in the way, but user site packages (tqdm https://github.com/iterative/dvc/issues/3514#issuecomment-625538574). We cannot do this centrally from the python plugin as there may be legitimate use cases for this and AFAIK there is no easy toggle in python to change this but to add a line to sitecustomize, we have this https://github.com/snapcore/snapcraft/blob/master/tools/snapcraft-override-build.sh#L18 for the Snapcraft snap.

[casperdcl]

Thanks @sergiusens! Really looks like that should be in the docs somewhere!

[casperdcl]

I'm still not convinced about this issue though @sergiusens - just tried and can't reproduce this.

$ snap refresh --revision=388 dvc  # unpatched revision
$ conda create -n test python=3.6 cffi=1.12.2 tqdm=4.35.0  # should make dvc fail
$ conda activate test
$ dvc add -v my_folder  # still works without errors
$ echo $PATH
/home/user/miniconda2/bin:/home/user/.local/bin:/home/user/miniconda2/envs/test/bin:/home/user/miniconda2/condabin:...:/snap/bin

[casperdcl]

@george-gca could you try snap refresh --channel=stable/3878 dvc and see if your issues are fixed on a previously broken setup?