Aws-sam-cli: Dependencies too strict

[HonzaRepresent]

Description

We use aws-sam-cli to deploy our lambdas. However, the dependencies it requires are pinned quite tightly:

six~=1.11.0
chevron~=0.12
click~=7.0
enum34~=1.1.6; python_version<"3.4"
Flask~=1.0.2
boto3~=1.9, >=1.9.56
PyYAML~=5.1
cookiecutter~=1.6.0
aws-sam-translator==1.14.0
docker~=4.0
dateparser~=0.7
python-dateutil~=2.6
pathlib2~=2.3.2; python_version<"3.4"
requests==2.22.0
serverlessrepo==0.1.9
aws_lambda_builders==0.4.0

This makes it harder to work with other libraries. In my specific case, pylint-quotes, which requires six==1.12.

Anecdotally, it seems aws-sam-cli works fine with six==1.12. For this reason, and also to make co-existence with other libs easier, I'd like to encourage you to provide wider version ranges where/if possible.

I understand there might be reasons behind this that I don't see, so perhaps at least explaining those would help me come to terms with this.

Some reasoning

I install aws-sam-cli side-by-side with other libs to have the deploy scripts which depend on aws-sam-cli in the same repository as the rest of my Python project, which requires the other libs. pipenv won't let me easily work around this, e.g. by letting me create another virtual environment for just the deploy scripts without non-trivial changes to the directory structure (can elaborate if needed). I also still believe this is a different issue and the version constraints should be relaxed, if possible.

Steps to reproduce

pipenv --three
pipenv install --dev aws-sam-cli
pipenv install --dev pylint-quotes

with the last command failing due to a conflict.

Observed result

Cannot install both libs, as they require six~=1.11.0 and six==1.12. I assume similar results for other approaches.

Expected result

Wider version ranges that would allow me to install these libs side by side. See above.

Workaround

One can force their dependency management system to ignore the conflict. For pipenv, installing aws-sam-cli as a --dev dependency and pylint-quotes as a default/normal package. This gives priority to the default packages requirements and installs six==1.12.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS 10.14.6 Mojave
2. sam --version: SAM CLI, version 0.22.0

[awood45]

The CLI dependencies are kept strict to make builds predictable, and since it's a CLI, the downstream impacts of tight dependencies aren't considered as critical. Are you trying to extend the CLI? How are you running in to this problem?

[HonzaRepresent]

Hello Alex, thanks for getting back to me

The CLI dependencies are kept strict to make builds predictable

Agreed. If you understand aws-sam-cli to be an app, rather than a lib, that makes sense.

How are you running in to this problem?

As I've outlined above, I have a python project that uses pipenv. In this project, I have a set of deploy scripts that I can run manually or reference from a CI build script and do the deploys automatically. These deploy scripts depend on aws-sam-cli.

I have aws-sam-cli as a dependency of the whole project so that it gets installed with the rest of the project. Also, since I use pipenv, I anticipate it to be quite difficult to cleanly install aws-sam-cli in one virtual env and the rest of the project into another virtual env, especially when running the deploy script on our CI provider. Although I do admit I have not looked into that too much. Additionally, I guess a lot of this is also cause by pipenv and its quirks.

I think I always saw aws-sam-cli as sort of a library to provide our deploy scripts with functionality. With your above explanation, I see better why trying to install aws-sam-cli in a separate virtual env would make sense, although I'm afraid the results are going to be quite messy.

I'd be happy to hear anything else you have to say on the subject. Otherwise, consider this resolved for me (though if you do decide to view aws-sam-cli as a library and relax the dependencies and make your own life more difficult in the process, I'm sure I'm not the only one who'd appreciate it); will reopen if I end up having something else to add in the future.

[jfuss]

@HonzaRepresent We do not view SAM CLI as a library and why we keep our dependencies more locked down. As @awood45 stated, this is to make the CLI reproducible. If we start relaxing all the dependencies, than you might be installing something was that untested and would break the CLI. We know many people use PIP but we really don't recommend it unless you really need to, it a very fragile way to install and requires the correct python version and to use a venv on your part (to get the isolation we are after).

My recommendation is to create a Makefile that will install SAM CLI into an venv on the system, if PIP is really want you want to use. You can then use that locally and in your CI environment, assuming the correct python versions we support are installed.

Closing this but feel free to re-open, if there are further questions on the topic.

[awood45]

I'd just add that the recommended installation methods are covered in our developer guide and avoid pip/pipenv installation difficulties. Our behaviors are tested as calls through the CLI, not patching directly into our code as a dependency.

[diablodale]

pylint + aws-sam-cli in a python 3.7 pipenv fails

pylint demands ==1.12
aws-sam-cli demands ~=1.11.0

Many of the dependencies of aws-sam-cli use ~=1.11 which is a workable solution.
However when aws-sam-cli uses ~=1.11.0 is unworkable as it allows nothing but 1.11.* versions.

Setup

• ubuntu 18.04.3 x64
• python 3.7.3 installed from official ubuntu repos using apt

Repo

mkdir sam1467
cd sam1467
pipenv --python 3.7
pipenv install -d pylint
pipenv install -d aws-sam-cli

Result

Installing aws-sam-cli…
Adding aws-sam-cli to Pipfile's [dev-packages]…
✔ Installation Succeeded
Pipfile.lock (3e63f5) out of date, updating to (36d424)…
Locking [dev-packages] dependencies…
✘ Locking Failed!
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches six==1.12,>=1.4.0,>=1.5,~=1.11,~=1.11.0
[pipenv.exceptions.ResolutionFailure]:       Tried: 0.9.0, 0.9.1, 0.9.2, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.5.2, 1.5.2, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.0, 1.11.0, 1.12.0, 1.12.0
[pipenv.exceptions.ResolutionFailure]:       Skipped pre-versions: 1.0b1
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches six==1.12,>=1.4.0,>=1.5,~=1.11,~=1.11.0
Tried: 0.9.0, 0.9.1, 0.9.2, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.5.2, 1.5.2, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.0, 1.11.0, 1.12.0, 1.12.0
Skipped pre-versions: 1.0b1
There are incompatible versions in the resolved dependencies.
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches six==1.12,>=1.4.0,>=1.5,~=1.11,~=1.11.0
[pipenv.exceptions.ResolutionFailure]:       Tried: 0.9.0, 0.9.1, 0.9.2, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.5.2, 1.5.2, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.0, 1.11.0, 1.12.0, 1.12.0
[pipenv.exceptions.ResolutionFailure]:       Skipped pre-versions: 1.0b1
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches six==1.12,>=1.4.0,>=1.5,~=1.11,~=1.11.0
Tried: 0.9.0, 0.9.1, 0.9.2, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.5.2, 1.5.2, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.0, 1.11.0, 1.12.0, 1.12.0
Skipped pre-versions: 1.0b1
There are incompatible versions in the resolved dependencies.