Poetry: 1.1.2: regression with poetry export when using python version constraints.

Created on 9 Oct 2020  ·  26Comments  ·  Source: python-poetry/poetry

  • [x] I am on the latest Poetry version.
  • [x] I have searched the issues of this repo and believe that this is not a duplicate.
  • [ ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
  • OS version and name: MacOS 10.15.5
  • Poetry version: 1.1.2
  • Link of a Gist with the contents of your pyproject.toml file: N/A

Issue

I have notice a regression compared to poetry 1.0.10 in the following case:

  • I add package dependency with a python version constraint ("typing-extensions")
  • I add a development dependency having the same sub-dependency with no python version constraint ("black").

The command poetry export give a wrong result with poetry 1.1.2 (it's OK with 1.0.10)

How to reproduce:

poetry new example
cd example
sed -i '/^python = /s/3.8/3.7/' pyproject.toml
poetry add --python="<3.8" typing-extensions
poetry export -f requirements.txt | grep typing
poetry add --dev black
poetry export -f requirements.txt | grep typing

Output for Poetry 1.0.10 (Python 3.8.6):

...
typing-extensions==3.7.4.3; python_version < "3.8" \
...
typing-extensions==3.7.4.3 \

Output for Poetry 1.1.2 (Python 3.8.6):

...
typing-extensions==3.7.4.3; python_version < "3.8" \
...
typing-extensions==3.7.4.3; python_version < "3.8" \
Bug Triage
Source
picture oncleben31
👍5

Most helpful comment

@abn @finswimmer Could you have a look at this? This was a regression in Poetry 1.1.3 that is still present in 1.1.4. It affects every project that has pre-commit in its development dependencies and supports Python 3.6. Thank you!

I have narrowed the repro down a little further.

Create this pyproject.toml:

[tool.poetry]
name = "app"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.6"
importlib-resources = {version = "^3.3.0", python = "<3.7"}

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Then export the requirements:

$ poetry export --dev --format=requirements.txt | grep zipp
zipp==3.4.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.7" or python_full_version >= "3.6.0" and python_version < "3.7" and python_version >= "3.6" \

If you look at the output, you can see that the python constraint <3.7 on importlib-resources affects its dependency zipp, but it shouldn't because zipp is also a dependency of pytest (via importlib-metadata), which doesn't have a marker.

Update: This also reproduces with Poetry's master. As a side note, reproducing it on master required running in a Python 3.7 environment. With Python 3.9 the dependency on importlib-metadata (and, by extension, on zipp) did not appear at all, even with poetry show --tree. But that seems to be an unrelated issue.

picture cjolowicz on 4 Nov 2020
👍3

All 26 comments

I have noticed similar behavior with platform-specific dependencies, which may be related:

I have a package (my-package) which lists a subdependency in pyproject.toml:

[tool.poetry.dependencies]
...
requests-kerberos = { version = "^0.12.0", markers = "sys_platform == 'win32'"}
...

This package is built as a wheel and is then a dependency of another application, which lists it in pyproject.toml:

[tool.poetry.dependencies]
...
my-package = "^1.0.3"
...

When using poetry export -f requirements.txt, I noticed different output between poetry versions, where the sys_platform == "win32" is not included in poetry==1.1.2.

poetry 1.0.9:

requests-kerberos==0.12.0; sys_platform == "win32"

poetry 1.1.2

requests-kerberos==0.12.0

I'm not entirely sure these are related, but they seem pretty similar.

mattkram picture mattkram on 9 Oct 2020

@mattkram can you please try the fix at #3147.

Using pipx

pipx install --suffix=@3147 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/3147/head'

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -e
python -m pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/3147/head
python -m poetry new foobar
pushd foobar
sed -i /pytest/d pyproject.toml
# your code snippet
EOF
abn picture abn on 10 Oct 2020

@abn Looks like the output is identical for me between poetry==1.1.2 and #3147 .

I created this gist to minimally reproduce, with output requirement.txt files for three versions: https://gist.github.com/mattkram/74ac2ac2de2065815ddfb379e235a7b6

The requests-kerberos setup.py lists two Windows-only dependencies (pykerberos & winkerberos), which are not listed correctly in the output requirements.txt file but which do contain the correct markers in the poetry.lock file.

Also, please note I had previously noticed this issue on Windows 10, and now have the same behavior on MacOS.

Please let me know if there is anything else I can do to help.

mattkram picture mattkram on 11 Oct 2020

@mattkram can you try that again; you might need to regenerate the lock file.

@oncleben31 your issue should also be fixed with #3147. Can you please confirm?

abn picture abn on 12 Oct 2020

For what it's worth: I encountered a similar issue when running poetry 1.1.2 in our CI/CD pipeline while running poetry 1.0.9 locally (which created the .lock file). In that case the solution is to update the local poetry to 1.1.2 as well to make the lock file compatible. The constraint information seems to be stored in another place in the lock file than it was before, which causes this information to disappear silently from the generated requirements.txt file.

gwdekker picture gwdekker on 12 Oct 2020

The lock file format has undergone some changes from 1.0.z to 1.1.z. This meant that the logic used in 1.0.z for export no longer worked. This along with various improvements required meant a new logic for export was introduced, one which relied more so on the lock file itself. This means that older lock files may not have the required information to generate exports using newer poetry versions.

@gwdekker thank you for pointing that out.

abn picture abn on 12 Oct 2020

No same error by installing poetry fromgit+https://github.com/abn/poetry.git@issue/3129 and redoing the instructions in first post.

oncleben31 picture oncleben31 on 12 Oct 2020

@oncleben31 apologies, I referenced the wrong pull request for your case. See #3119.

Using pipx

pipx install --suffix=@3119 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/3119/head'

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
python -m pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/3119/head
python -m poetry new foobar
pushd foobar
sed -i /pytest/d pyproject.toml
sed -i '/^python = /s/3.8/3.7/' pyproject.toml
poetry add --python="<3.8" typing-extensions
poetry export -f requirements.txt | grep typing
poetry add --dev black
poetry export -f requirements.txt | grep typing
poetry export -f requirements.txt --dev | grep typing
EOF
abn picture abn on 13 Oct 2020

Too bad same result. But perhaps it's on my side. I did have poetry errors after using your pipx command.
I've tried by installing poetry with pipx install git+https://github.com/abn/poetry.git@cleanup/exports

Have I done something wrong ?

oncleben31 picture oncleben31 on 13 Oct 2020

You need to do a pipx install --force i think if you are reinstalling because it is already installed before. Your command above would replace your current poetry binary., hence why I recommended you use the suffix.

abn picture abn on 13 Oct 2020

OK I've uninstall poetry before my command and with yours I have the following error and I don't know if it's important:

test-fix2 pipx install --suffix=@3119 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/3119/head' 
  WARNING: Did not find branch or tag 'refs/pull/3119/head', assuming revision or ref.ll/3119/head'
⚠️  Note: poetry was already on your PATH at /Users/benoit/.local/bin/poetry
  installed package poetry 1.1.2, Python 3.8.5
  These apps are now globally available
    - chardetect
    - keyring
    - pkginfo
    - poetry@3119
    - virtualenv
done! ✨ 🌟 ✨
➜  test-fix2 pipx list
venvs are in /Users/benoit/.local/pipx/venvs
apps are exposed on your $PATH at /Users/benoit/.local/bin
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/usr/local/Cellar/pipx/0.15.5.1/libexec/lib/python3.8/site-packages/pipx/commands/common.py", line 151, in get_package_summary
    package_metadata = venv.package_metadata[package]
KeyError: 'poetry@3119'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/Cellar/pipx/0.15.5.1/libexec/bin/pipx", line 33, in <module>
    sys.exit(load_entry_point('pipx==0.15.5.1', 'console_scripts', 'pipx')())
  File "/usr/local/Cellar/pipx/0.15.5.1/libexec/lib/python3.8/site-packages/pipx/main.py", line 591, in cli
    return run_pipx_command(parsed_pipx_args)
  File "/usr/local/Cellar/pipx/0.15.5.1/libexec/lib/python3.8/site-packages/pipx/main.py", line 184, in run_pipx_command
    return commands.list_packages(venv_container, args.include_injected)
  File "/usr/local/Cellar/pipx/0.15.5.1/libexec/lib/python3.8/site-packages/pipx/commands/list_packages.py", line 32, in list_packages
    for package_summary in p.map(
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
KeyError: 'poetry@3119'
➜  test-fix2 poetry#3119

I can run the command the with the suffix with the same results:

poetry@3119 export -f requirements.txt | grep typing
typing-extensions==3.7.4.3; python_version < "3.8" \
oncleben31 picture oncleben31 on 13 Oct 2020

The issue you are seeing with list is a pipx issue: https://github.com/pipxproject/pipx/pull/491

As for the results, you need --dev for the right results. The ouput above is correct since your pyproject.toml specifies that constraint. With --dev is when the nested dependencies of black gets activated causing two seperate entries. The latter is because black depends on this without constraints.

abn picture abn on 13 Oct 2020

So new and different results: 

$ poetry@3119 export -f requirements.txt --dev | grep typing
typing-extensions==3.7.4.3 \
typing-extensions==3.7.4.3; python_version < "3.8" \

Is it OK to have 2 lines and not the union?

oncleben31 picture oncleben31 on 13 Oct 2020

@oncleben31 yes; it is upto the PEP 517 front-end to decide what to do with that. We can create unions as well; but that might cause more problems down the lane. Also, this was the case in1.0.10 too correct?

abn picture abn on 13 Oct 2020

Thank you for taking time for your explanation.
In 1.0.10 without the --dev flag, export function was doing the union (as shown in the example of the first post of this issue).
In need to reinstall poetry 1.0.10 to make the test and to confirm. I will do it later today.

oncleben31 picture oncleben31 on 13 Oct 2020

With poetry 1.0.10 the results seems to be the union with or without the --dev flag:

➜  test-fix3 poetry@1010 export -f requirements.txt | grep typing     
typing-extensions==3.7.4.3 \
➜  test-fix3 poetry@1010 export -f requirements.txt --dev| grep typing
typing-extensions==3.7.4.3 \
oncleben31 picture oncleben31 on 13 Oct 2020

@oncleben31 I have added a fix to the same branch so that we emulate the same behaviour as 1.0.10.

$ poetry@local export --without-hashes 
typing-extensions==3.7.4.3; python_version < "3.8"
$ poetry@local export --without-hashes --dev
appdirs==1.4.4
black==20.8b1; python_version >= "3.6"
click==7.1.2; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
dataclasses==0.7; python_version >= "3.6" and python_version < "3.7"
mypy-extensions==0.4.3
pathspec==0.8.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
regex==2020.10.11
toml==0.10.1
typed-ast==1.4.1
typing-extensions==3.7.4.3
abn picture abn on 13 Oct 2020
👍2

@abn Initial problem is solve with poetry 1.1.3, but another pop up and is related to export feature.

In my repo https://github.com/hacf-fr/meteofrance-api, the command nox -s tests will be OK for Python 3.8 and 3.6 but not for 3.7 with this error:

nox > Command pip install --constraint=/var/folders/v5/wtydb2rx72b74xds_1hc2xr80000gq/T/tmpne54j6j5/requirements.txt coverage[toml] pytest requests-mock failed with exit code 1:
Ignoring atomicwrites: markers 'python_version >= "2.7" and python_full_version < "3.0.0" and sys_platform == "win32" or python_full_version >= "3.4.0" and sys_platform == "win32"' don't match your environment
Ignoring colorama: markers 'python_version >= "2.7" and python_full_version < "3.0.0" and sys_platform == "win32" or python_version >= "2.7" and python_full_version < "3.0.0" and sys_platform == "win32" and platform_system == "Windows" or sys_platform == "win32" and python_full_version >= "3.5.0" or sys_platform == "win32" and python_full_version >= "3.5.0" and platform_system == "Windows" or python_version >= "2.7" and python_full_version < "3.0.0" and platform_system == "Windows" or platform_system == "Windows" and python_full_version >= "3.5.0"' don't match your environment
Ignoring dataclasses: markers 'python_version < "3.7"' don't match your environment
Ignoring importlib-resources: markers 'python_version >= "2.7" and python_full_version < "3.0.0" and python_version < "3.7" or python_full_version >= "3.5.0" and python_version < "3.7"' don't match your environment
Ignoring zipp: markers 'python_version < "3.7" and python_version >= "3.6"' don't match your environment
Collecting coverage==5.3
  Using cached coverage-5.3-cp37-cp37m-macosx_10_13_x86_64.whl (205 kB)
Collecting pytest==6.1.1
  Using cached pytest-6.1.1-py3-none-any.whl (272 kB)
Collecting requests-mock==1.8.0
  Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: requests==2.24.0 in ./.nox/tests-3-7/lib/python3.7/site-packages (from -c /var/folders/v5/wtydb2rx72b74xds_1hc2xr80000gq/T/tmpne54j6j5/requirements.txt (line 287)) (2.24.0)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting toml==0.10.1
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Requirement already satisfied: urllib3==1.25.10 in ./.nox/tests-3-7/lib/python3.7/site-packages (from -c /var/folders/v5/wtydb2rx72b74xds_1hc2xr80000gq/T/tmpne54j6j5/requirements.txt (line 394)) (1.25.10)
Collecting pluggy==0.13.1
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting importlib-metadata==1.7.0
  Using cached importlib_metadata-1.7.0-py2.py3-none-any.whl (31 kB)
Collecting py==1.9.0
  Using cached py-1.9.0-py2.py3-none-any.whl (99 kB)
Collecting packaging==20.4
  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting attrs==19.3.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting iniconfig==1.0.1
  Using cached iniconfig-1.0.1-py3-none-any.whl (4.2 kB)
Requirement already satisfied: idna==2.10 in ./.nox/tests-3-7/lib/python3.7/site-packages (from -c /var/folders/v5/wtydb2rx72b74xds_1hc2xr80000gq/T/tmpne54j6j5/requirements.txt (line 122)) (2.10)
Requirement already satisfied: chardet==3.0.4 in ./.nox/tests-3-7/lib/python3.7/site-packages (from -c /var/folders/v5/wtydb2rx72b74xds_1hc2xr80000gq/T/tmpne54j6j5/requirements.txt (line 33)) (3.0.4)
Requirement already satisfied: certifi==2020.6.20 in ./.nox/tests-3-7/lib/python3.7/site-packages (from -c /var/folders/v5/wtydb2rx72b74xds_1hc2xr80000gq/T/tmpne54j6j5/requirements.txt (line 27)) (2020.6.20)
Collecting zipp>=0.5
Collecting pyparsing==2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    zipp>=0.5 from https://files.pythonhosted.org/packages/64/f2/0aa52375e14863b377bc9eb58fde6ec48e0a7f73fdf91c9876669559fe45/zipp-3.3.1-py3-none-any.whl#sha256=16522f69653f0d67be90e8baa4a46d66389145b734345d68a257da53df670903 (from importlib-metadata==1.7.0->-c /var/folders/v5/wtydb2rx72b74xds_1hc2xr80000gq/T/tmpne54j6j5/requirements.txt (line 128))

In the Python 3.7 venv the command poetry export --format=requirements.txt --without-hashes --dev | grep zipp return:

  • zipp==3.1.0; python_version < "3.7" and python_version >= "3.6" with poetry v1.1.3
  • zipp==3.1.0; python_version < "3.8" with poetry v1.0.10

I didn't succeed to create a simple poetry project to quickly reproduce the issue yet.

oncleben31 picture oncleben31 on 18 Oct 2020

I found sequence to reproduce:

poetry new debug-poetry-export
cd debug-poetry-export

Check you have the following setting for python in pyproject.toml:

[tool.poetry.dependencies]
python = "^3.6.1"

poetry add --dev pre-commit
poetry export --format=requirements.txt --without-hashes --dev | grep zipp

The last command will return:

  • zipp==3.1.0; python_version < "3.7" and python_version >= "3.6" with poetry v1.1.3
  • zipp==3.1.0; python_version < "3.8" with poetry v1.0.10

Do we continue here or do you need I open a new issue ?

oncleben31 picture oncleben31 on 21 Oct 2020

Problem not solved with poetry 1.1.4. After a poetry lock --no-update:

poetry export --format=requirements.txt --without-hashes --dev | grep zipp return zipp==3.3.1; python_version < "3.7" and python_version >= "3.6" and python_full_version >= "3.6.1"

oncleben31 picture oncleben31 on 26 Oct 2020

@abn @finswimmer Could you have a look at this? This was a regression in Poetry 1.1.3 that is still present in 1.1.4. It affects every project that has pre-commit in its development dependencies and supports Python 3.6. Thank you!

I have narrowed the repro down a little further.

Create this pyproject.toml:

[tool.poetry]
name = "app"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.6"
importlib-resources = {version = "^3.3.0", python = "<3.7"}

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Then export the requirements:

$ poetry export --dev --format=requirements.txt | grep zipp
zipp==3.4.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.7" or python_full_version >= "3.6.0" and python_version < "3.7" and python_version >= "3.6" \

If you look at the output, you can see that the python constraint <3.7 on importlib-resources affects its dependency zipp, but it shouldn't because zipp is also a dependency of pytest (via importlib-metadata), which doesn't have a marker.

Update: This also reproduces with Poetry's master. As a side note, reproducing it on master required running in a Python 3.7 environment. With Python 3.9 the dependency on importlib-metadata (and, by extension, on zipp) did not appear at all, even with poetry show --tree. But that seems to be an unrelated issue.

cjolowicz picture cjolowicz on 4 Nov 2020
👍3

Does anyone have this work on one platform (linux for us in our CI/CD pipelines) and fail on another (macos local dev environment)?

Also, I'm getting the error on macos but I don't understand why since the hashes are in the requirements.txt file?

$ poetry export --ansi --dev --format=requirements.txt | grep -A 25 cryptography
cryptography==3.2.1; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "linux" or sys_platform == "linux" and python_version >= "3.6" and python_full_version >= "3.5.0" \
    --hash=sha256:6dc59630ecce8c1f558277ceb212c751d6730bd12c80ea96b4ac65637c4f55e7 \
    --hash=sha256:75e8e6684cf0034f6bf2a97095cb95f81537b12b36a8fedf06e73050bb171c2d \
    --hash=sha256:4e7268a0ca14536fecfdf2b00297d4e407da904718658c1ff1961c713f90fd33 \
    --hash=sha256:7117319b44ed1842c617d0a452383a5a052ec6aa726dfbaffa8b94c910444297 \
    --hash=sha256:a733671100cd26d816eed39507e585c156e4498293a907029969234e5e634bc4 \
    --hash=sha256:a75f306a16d9f9afebfbedc41c8c2351d8e61e818ba6b4c40815e2b5740bb6b8 \
    --hash=sha256:5849d59358547bf789ee7e0d7a9036b2d29e9a4ddf1ce5e06bb45634f995c53e \
    --hash=sha256:bd717aa029217b8ef94a7d21632a3bb5a4e7218a4513d2521c2a2fd63011e98b \
    --hash=sha256:efe15aca4f64f3a7ea0c09c87826490e50ed166ce67368a68f315ea0807a20df \
    --hash=sha256:32434673d8505b42c0de4de86da8c1620651abd24afe91ae0335597683ed1b77 \
    --hash=sha256:7b8d9d8d3a9bd240f453342981f765346c87ade811519f98664519696f8e6ab7 \
    --hash=sha256:d3545829ab42a66b84a9aaabf216a4dce7f16dbc76eb69be5c302ed6b8f4a29b \
    --hash=sha256:a4e27ed0b2504195f855b52052eadcc9795c59909c9d84314c5408687f933fc7 \
    --hash=sha256:13b88a0bd044b4eae1ef40e265d006e34dbcde0c2f1e15eb9896501b2d8f6c6f \
    --hash=sha256:07ca431b788249af92764e3be9a488aa1d39a0bc3be313d826bbec690417e538 \
    --hash=sha256:a035a10686532b0587d58a606004aa20ad895c60c4d029afa245802347fab57b \
    --hash=sha256:d26a2557d8f9122f9bf445fc7034242f4375bd4e95ecda007667540270965b13 \
    --hash=sha256:545a8550782dda68f8cdc75a6e3bf252017aa8f75f19f5a9ca940772fc0cb56e \
    --hash=sha256:55d0b896631412b6f0c7de56e12eb3e261ac347fbaa5d5e705291a9016e5f8cb \
    --hash=sha256:3cd75a683b15576cfc822c7c5742b3276e50b21a06672dc3a800a2d5da4ecd1b \
    --hash=sha256:d25cecbac20713a7c3bc544372d42d8eafa89799f492a43b79e1dfd650484851 \
    --hash=sha256:d3d5e10be0cf2a12214ddee45c6bd203dab435e3d83b4560c03066eda600bfe3

The error message:

$ pip install --constraint=/var/folders/qh/sgf65ggd7z91nhh8hx8fffyw0000gp/T/tmplc8hu_tn/requirements.txt coverage[toml] pytest boto3 moto
...
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    cryptography>=2.3.0 from https://files.pythonhosted.org/packages/c4/78/6c28c899181c395d8e07778110caff21248ba97774e567e7f7895951d92e/cryptography-3.2.1-cp35-abi3-macosx_10_10_x86_64.whl#sha256=bd717aa029217b8ef94a7d21632a3bb5a4e7218a4513d2521c2a2fd63011e98b (from moto==1.3.14->-c /var/folders/qh/sgf65ggd7z91nhh8hx8fffyw0000gp/T/tmplc8hu_tn/requirements.txt (line 497))
alexifm picture alexifm on 12 Nov 2020

I'm getting the error on macos but I don't understand why since the hashes are in the requirements.txt file?

@alexifm Your _pip install_ command clearly mixes hash-pinned requirements and non-hash-pinned requirements. I believe that is what _pip_ is complaining about, it is not enough to have hashes for all requirements in the file, they all have to be hashed, even those from the command line. Are you running the exact same command on both platforms?

sinoroc picture sinoroc on 12 Nov 2020

Are you running the exact same command on both platforms?

Yup.

Here are the relevant pins for each of coverage, pytest, boto3, and moto.

[tool.poetry.dependencies]
...
boto3 = "~1.14"
...

[tool.poetry.dev-dependencies]
...
coverage = "^5.0"
pytest = "^5.4"
moto = "1.3.14"
...

I also inspected the requirements file and everything in there has a hash. The only thing that's different is that it doesn't work on macos. Can someone deduce what is being output by Poetry here? To my untrained eye it looks funky but maybe it passes the logic of pip?

cryptography==3.2.1; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "linux" or sys_platform == "linux" and python_version >= "3.6" and python_full_version >= "3.5.0" \
alexifm picture alexifm on 12 Nov 2020

@alexifm

Here are the relevant pins for each of coverage, pytest, boto3, and moto.

Those are not pinned version specifiers. Those are ranges. The pins are usually in the lockfile poetry.lock and look like something==1.23 (the double equal is the mark of a pinned version specifier). Anyway I do not think they are relevant now, or at least this is not where I would look for now.

I also inspected the requirements file and everything in there has a hash.

This might well be the case, but if you have pip install -r requirements.txt something, then the something requirement is not pinned and without a hash. So as far as I can remember this should fail.

cryptography==3.2.1; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "linux" or sys_platform == "linux" and python_version >= "3.6" and python_full_version >= "3.5.0" \

This definitely looks "_funky_" :D
For one this bit python_version >= "3.6" and python_full_version < "3.0.0" is impossible.

Not sure if it is a lead, but... we have to start somewhere:
Could you give us some details about the environments you are trying? For example I would be particularly interested in the _pip_ version strings (Python as well).

sinoroc picture sinoroc on 12 Nov 2020

Here are the steps to replicate (on macos 10.15.7):

  1. create a new package with poetry new
  2. Update the pyproject file with these deps. I know most of these are irrelevant but moto alone takes forever for poetry to resolve so I added the deps from our other package to help it go faster.
[tool.poetry.dependencies]
python = "~3.8"
boto3 = "~1.14"
botocore = "~1.17.63"
idna = "<2.9"

[tool.poetry.dev-dependencies]
black = "^19.10b0"
commitizen = "^2.4.1"
colorama = "<0.5.0,>=0.4.1"
coverage = "^5.0"
isort = "^4.3"
seed-isort-config = "^2.1"
mypy = "^0.770"
lxml = "^4.5"
pytest = "^5.4"
moto = "1.3.14"
python-semantic-release = "^6.4"
flakehell = "^0.5"
pep8-naming = "^0.10"
flake8-docstrings = "^1.5.0"
flake8-rst-docstrings = "^0.0.13"
flake8-bugbear = "^20.1"
darglint = "^1.4"
flake8-bandit = "^2.1"
typeguard = "^2.9"
# build system dependencies
Cython = "^0.29"
numpy = "~1.18.0"
setuptools = ">=47.1.0"
poetry_core = ">=1.0.0"
  1. Create a virtualenv with pyenv using Python 3.8.5 with pip version 20.1.1
  2. Run poetry update and then poetry export --ansi --dev --format=requirements.txt --output=requirements.txt
  3. Create a new identical clean virtual environment (replicating a pipeline setup that uses nox or tox) and run pip install --constraint=requirements.txt moto

Here's more detail on those last two steps:

poetry.lock:

[[package]]
name = "cryptography"
version = "3.2.1"
description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
category = "dev"
optional = false
python-versions = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*"

...

cryptography = [
    {file = "cryptography-3.2.1-cp27-cp27m-macosx_10_10_x86_64.whl", hash = "sha256:6dc59630ecce8c1f558277ceb212c751d6730bd12c80ea96b4ac65637c4f55e7"},
    {file = "cryptography-3.2.1-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:75e8e6684cf0034f6bf2a97095cb95f81537b12b36a8fedf06e73050bb171c2d"},
    {file = "cryptography-3.2.1-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:4e7268a0ca14536fecfdf2b00297d4e407da904718658c1ff1961c713f90fd33"},
    {file = "cryptography-3.2.1-cp27-cp27m-win32.whl", hash = "sha256:7117319b44ed1842c617d0a452383a5a052ec6aa726dfbaffa8b94c910444297"},
    {file = "cryptography-3.2.1-cp27-cp27m-win_amd64.whl", hash = "sha256:a733671100cd26d816eed39507e585c156e4498293a907029969234e5e634bc4"},
    {file = "cryptography-3.2.1-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:a75f306a16d9f9afebfbedc41c8c2351d8e61e818ba6b4c40815e2b5740bb6b8"},
    {file = "cryptography-3.2.1-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:5849d59358547bf789ee7e0d7a9036b2d29e9a4ddf1ce5e06bb45634f995c53e"},
    {file = "cryptography-3.2.1-cp35-abi3-macosx_10_10_x86_64.whl", hash = "sha256:bd717aa029217b8ef94a7d21632a3bb5a4e7218a4513d2521c2a2fd63011e98b"},
    {file = "cryptography-3.2.1-cp35-abi3-manylinux1_x86_64.whl", hash = "sha256:efe15aca4f64f3a7ea0c09c87826490e50ed166ce67368a68f315ea0807a20df"},
    {file = "cryptography-3.2.1-cp35-abi3-manylinux2010_x86_64.whl", hash = "sha256:32434673d8505b42c0de4de86da8c1620651abd24afe91ae0335597683ed1b77"},
    {file = "cryptography-3.2.1-cp35-abi3-manylinux2014_aarch64.whl", hash = "sha256:7b8d9d8d3a9bd240f453342981f765346c87ade811519f98664519696f8e6ab7"},
    {file = "cryptography-3.2.1-cp35-cp35m-win32.whl", hash = "sha256:d3545829ab42a66b84a9aaabf216a4dce7f16dbc76eb69be5c302ed6b8f4a29b"},
    {file = "cryptography-3.2.1-cp35-cp35m-win_amd64.whl", hash = "sha256:a4e27ed0b2504195f855b52052eadcc9795c59909c9d84314c5408687f933fc7"},
    {file = "cryptography-3.2.1-cp36-abi3-win32.whl", hash = "sha256:13b88a0bd044b4eae1ef40e265d006e34dbcde0c2f1e15eb9896501b2d8f6c6f"},
    {file = "cryptography-3.2.1-cp36-abi3-win_amd64.whl", hash = "sha256:07ca431b788249af92764e3be9a488aa1d39a0bc3be313d826bbec690417e538"},
    {file = "cryptography-3.2.1-cp36-cp36m-win32.whl", hash = "sha256:a035a10686532b0587d58a606004aa20ad895c60c4d029afa245802347fab57b"},
    {file = "cryptography-3.2.1-cp36-cp36m-win_amd64.whl", hash = "sha256:d26a2557d8f9122f9bf445fc7034242f4375bd4e95ecda007667540270965b13"},
    {file = "cryptography-3.2.1-cp37-cp37m-win32.whl", hash = "sha256:545a8550782dda68f8cdc75a6e3bf252017aa8f75f19f5a9ca940772fc0cb56e"},
    {file = "cryptography-3.2.1-cp37-cp37m-win_amd64.whl", hash = "sha256:55d0b896631412b6f0c7de56e12eb3e261ac347fbaa5d5e705291a9016e5f8cb"},
    {file = "cryptography-3.2.1-cp38-cp38-win32.whl", hash = "sha256:3cd75a683b15576cfc822c7c5742b3276e50b21a06672dc3a800a2d5da4ecd1b"},
    {file = "cryptography-3.2.1-cp38-cp38-win_amd64.whl", hash = "sha256:d25cecbac20713a7c3bc544372d42d8eafa89799f492a43b79e1dfd650484851"},
    {file = "cryptography-3.2.1.tar.gz", hash = "sha256:d3d5e10be0cf2a12214ddee45c6bd203dab435e3d83b4560c03066eda600bfe3"},
]

Output from the pip install command:

$ pip install --constraint=requirements.txt moto
Ignoring atomicwrites: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.4.0"' don't match your environment
Ignoring cryptography: markers 'python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "linux" or sys_platform == "linux" and python_version >= "3.6" and python_full_version >= "3.5.0"' don't match your environment
Ignoring jeepney: markers 'sys_platform == "linux" and python_version >= "3.6"' don't match your environment
Ignoring pywin32-ctypes: markers 'sys_platform == "win32" and python_version >= "3.6"' don't match your environment
Ignoring pywin32: markers 'python_version >= "2.7" and python_full_version < "3.0.0" and sys_platform == "win32" or python_full_version >= "3.5.0" and sys_platform == "win32"' don't match your environment
Ignoring secretstorage: markers 'sys_platform == "linux" and python_version >= "3.6"' don't match your environment
Looking in indexes: https://pypi.org/simple, https://gitlab-pypi-deploy:****@gitlab.com/api/v4/projects/20895990/packages/pypi/simple
Collecting moto==1.3.14
  Using cached moto-1.3.14-py2.py3-none-any.whl (730 kB)
...
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    cryptography>=2.3.0 from https://files.pythonhosted.org/packages/c4/78/6c28c899181c395d8e07778110caff21248ba97774e567e7f7895951d92e/cryptography-3.2.1-cp35-abi3-macosx_10_10_x86_64.whl#sha256=bd717aa029217b8ef94a7d21632a3bb5a4e7218a4513d2521c2a2fd63011e98b (from moto==1.3.14->-c requirements.txt (line 387))

It's trying to install cryptography-3.2.1-cp35-abi3-macosx_10_10_x86_64.whl which appears to me to be a Python 3.5 wheel but I'm not sure if those can be made compatible to later versions.

Finally, our CI pipelines run on linux and have no problem doing these exact same tasks.

Let me know if you need any other information.

alexifm picture alexifm on 12 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

probablykasper picture probablykasper  ·  3Comments
kierun picture kierun  ·  3Comments
AWegnerGitHub picture AWegnerGitHub  ·  3Comments
ambv picture ambv  ·  3Comments
ghost picture ghost  ·  3Comments