Dvc: pip install errors: moto/boto3, awscli/colorama, requests?

Created on 4 Oct 2019  路  15Comments  路  Source: iterative/dvc

As noted in https://discordapp.com/channels/485586884165107732/565699007037571084/629449391266070563

I already have the repo cloned. I fetch and pull the latest master branch (to this moment) and freshly regenerate my virtualenv in .env, activate it and run pip install -e ".[all,tests]" (as instructed in https://dvc.org/doc/user-guide/contributing). It succeeds but I get the following errors:

Building wheels for collected packages: psutil
...
ERROR: moto 1.3.14.dev302 has requirement boto3>=1.9.201, but you'll have boto3 1.9.115 which is incompatible.
ERROR: awscli 1.16.252 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
bug c3-small-fix p2-medium question

Most helpful comment

@edisongustavo , by the way, it felt a bit weird that I submitted the PR after your comment. It looks discouraging, to some extend, maybe you had the intention to submit a PR for it :slightly_smiling_face: I just had the commit laying right there a few hours before but forgot to create the PR :see_no_evil: https://github.com/iterative/dvc/issues/2570#ref-commit-4fd7013
Thanks again for your comment, appreciate your research! :slightly_smiling_face:

All 15 comments

@jorgeorpinel
What can you give some more details about your env? I am unable to reproduce on fresh Python 3.7.4 environment,
tried:
pip install -e ."[all,tests]"
and
pip install -e ."[all,tests]" --no-cache

OK so I pull the latest version of master (449f43f3 rn), I make sure no venv is active and remove my previous .env dir. Then:

$ virtualenv --python python3 .env
...
$ source .env/bin/activate
...
$ pip install -e ".[all,tests]"  

Obtaining file:///.../dvc
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting requests>=2.22.0 (from dvc==0.61.2+449f43)
...
Collecting jsonpointer>=1.9 (from jsonpatch->cfn-lint>=0.4.0->moto>=1.3.14.dev55; extra == "tests"->dvc==0.61.2+449f43) ...
Building wheels for collected packages: pathspec, aws-sam-translator
  Building wheel for pathspec (setup.py) ... done
  ...
Successfully built pathspec aws-sam-translator
ERROR: awscli 1.16.252 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
ERROR: moto 1.3.14.dev312 has requirement boto3>=1.9.201, but you'll have boto3 1.9.115 which is incompatible.
talling collected packages: certifi, ...
$ which dvc
/.../dvc/.env/bin/dvc
$ dvc version             
DVC version: 0.60.1
Python version: 3.7.3
Platform: Darwin-18.7.0-x86_64-i386-64bit
Binary: False
Cache: reflink - True, hardlink - True, symlink - True
Filesystem type (cache directory): ('apfs', '/dev/disk1s1')
Filesystem type (workspace): ('apfs', '/dev/disk1s1')

See full output for the pip install command in https://pastebin.com/f7c59V4n

BTW I just upgraded my system DVC with pip3 install dvc --upgrade and it worked but with 2 red lines:

docker-compose 1.23.2 has requirement requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1, but you'll have requests 2.22.0 which is incompatible.
awsebcli 3.15.3 has requirement requests<2.21,>=2.20.1, but you'll have requests 2.22.0 which is incompatible.

which seems like a similar thing.
dvc version after update:

DVC version: 0.61.2
Python version: 3.7.3
Platform: Darwin-18.7.0-x86_64-i386-64bit
Binary: False
Cache: reflink - True, hardlink - True, symlink - True
Filesystem type (cache directory): ('apfs', '/dev/disk1s1')
Filesystem type (workspace): ('apfs', '/dev/disk1s1')

BTW I just upgraded my system DVC with pip3 install dvc --upgrade and it worked but with 2 red lines:

This one is about your previously installed packages, that require something else than what dvc requires. There is nothing we can do about it on our side.

And these

ERROR: awscli 1.16.252 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
ERROR: moto 1.3.14.dev312 has requirement boto3>=1.9.201, but you'll have boto3 1.9.115 which is incompatible.

are valid and are our fault, as our all reqs conflict with test reqs, which doesn't affect users, but does affect developers in a minor way. We should fix that, for sure.

This one is about your previously installed packages, that require something else than what dvc requires. There is nothing we can do about it on our side.

How is it possible if it was installed in the clean virtualenv?

@shcheklein Apparently it wasn't clean, but rather inherited from the global env. docker-compose and awsebcli are definitely not our dependencies.

How is it possible if it was installed in the clean virtualenv?

Ruslan is right about that one @shcheklein. In that https://github.com/iterative/dvc/issues/2570#issuecomment-538624781 I was referring to updating my system DVC installed as a package with pip, not to the development version installed as a git repo. Sorry for the confusion, we can safely ignore that comment and focus on https://github.com/iterative/dvc/issues/2570#issuecomment-538645844:

are valid and are our fault, as our all reqs conflict with test reqs, which doesn't affect users, but does affect developers in a minor way. We should fix that, for sure.

p.s. do you mind if I hide https://github.com/iterative/dvc/issues/2570#issuecomment-538624781 and all its responses, off-topic and now resolved? 馃構

p.s. do you mind if I hide #2570 (comment) and all its responses, off-topic and now resolved?

@jorgeorpinel Please don't hide comments. These are related to the topic.

is this still relevant?

@mroutis Yes, it is, I've seen it myself not that long ago.

Hello, I am also affected by this issue.

I think the root cause of the problem is that DVC requires a specific version of boto3:

s3 = ["boto3==1.9.115"]

The problem is that this makes it impossible to install DVC together with other libraries that also depend on boto3 that don't fit this specific requirement.

My proposal is to depend on a version of boto3 that is newer than 1.9.115:

  1. boto3>=1.9.115,<2 (pin to major version)
  2. boto3>=1.9.115,<1.10 (pin to minor version).

I think option 1 is the best. I don't really believe that boto3 will break compatibility in the future 1.* versions. I wouldn't recommend option 2 because it requires bumping DVC's version when an upgrade to boto3 is required.

Thanks, @edisongustavo :) ! There's a PR already that should fix this https://github.com/iterative/dvc/pull/2902

@edisongustavo , by the way, it felt a bit weird that I submitted the PR after your comment. It looks discouraging, to some extend, maybe you had the intention to submit a PR for it :slightly_smiling_face: I just had the commit laying right there a few hours before but forgot to create the PR :see_no_evil: https://github.com/iterative/dvc/issues/2570#ref-commit-4fd7013
Thanks again for your comment, appreciate your research! :slightly_smiling_face:

@mroutis hahaha, no no, not at all :)

I just wanted to give a signal that this is also important for me :)

Was this page helpful?
0 / 5 - 0 ratings

Related issues

shcheklein picture shcheklein  路  3Comments

prihoda picture prihoda  路  3Comments

TezRomacH picture TezRomacH  路  3Comments

dnabanita7 picture dnabanita7  路  3Comments

shcheklein picture shcheklein  路  3Comments