poetry tries to reinstall from git, even if it's already installed

[itajaja]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• OS version and name: MacOS, but also docker (ubuntu)
• Poetry version:0.12.10

Issue

to reproduce, use the following pyproject:

[tool.poetry]
name = "test"
version = "0.0.0"
authors = ["buh"]

[tool.poetry.dependencies]
python = "^3.6"
Tqp = { git = "git://github.com/4Catalyzer/tqp.git", tag="v0.1.5" }

run poetry install, disable wifi, run poetry install again. the second time it will complain because

[CalledProcessError]
Command '['git', 'clone', 'ssh://[email protected]/4Catalyzer/tqp.git', '/var/folders/b2/4b21t4_90pg6rwkbpgdr_l540000gn/T/pypoetry-git-tqpq4geocan']' returned non-zero exit status 128.

this segment of the lockfile:

[metadata.hashes]
boto3 = ["0a0c0f0859a2be56b23823f8c1d50abf3c89d7d4d054019f24de69eeee9ad75c", "b429d48b8e99a9fdd18c3aef68370f779e0aa76cfe275a55e1adff427d44ca9a"]
botocore = ["6b9edd1e18436bce8b28b95b3dec582588080a90b5636abc1b3a795f5a0e6cf3", "9dac7753d81e8a725b9a169fd63b43d2a3caeceb51de3fafd5e5bd10e25589cb"]
docutils = ["02aec4bd92ab067f6ff27a38a38a41173bf01bed8f89157768c1573f53e474a6", "51e64ef2ebfb29cae1faa133b3710143496eca21c530f3f71424d77687764274", "7a4bd47eaf6596e1295ecb11361139febe29b084a87bf005bf899f9a42edc3c6"]
jmespath = ["6a81d4c9aa62caf061cb517b4d9ad1dd300374cd4706997aff9cd6aedd61fc64", "f11b4461f425740a1d908e9a3f7365c3d2e569f6ca68a2ff8bc5bcd9676edd63"]
python-dateutil = ["063df5763652e21de43de7d9e00ccf239f953a832941e37be541614732cdfc93", "88f9287c0174266bb0d8cedd395cfba9c58e87e5ad86b2ce58859bc11be3cf02"]
s3transfer = ["90dc18e028989c609146e241ea153250be451e05ecc0c2832565231dacdf59c1", "c7a9ec356982d5e9ab2d4b46391a7d6a950e2b04c472419f5fdec70cc0ada72f"]
six = ["70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9", "832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"]
tqp = []
urllib3 = ["61bf29cada3fc2fbefad4fdf059ea4bd1b4a86d2b6d15e1c7c0b582b9752fe39", "de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22"]

where tqp doesn't have a hash, could be the culript?

[itajaja]

I pinned down the problem. The issue is that pip freeze always stores the full commit hash as reference, but the lock file stores whatever reference is set in pyproject (tag/branch/reference).

Instead the reference should always be normalized to the full commit hash (that's also what yarn does), so that regardless of what you specify (tag, branch, etc) the lock only stores the resolved full hash of that reference

[itajaja]

actually, regardless of that, which still holds true, doing poetry install will always do a git clone, even if it's not strictly necessary

[sdispater]

but the lock file stores whatever reference is set in pyproject (tag/branch/reference).

If that's actually the case it's a bug since Poetry should store the exact commit after resolving.

[itajaja]

cool, thanks for looking into that. it doesn't afaik. but the problem is that it still tries to clone anyway, even if it stores the right reference

[hedgepigdaniel]

Is there any workaround for this problem? I tried using the rev argument to specify an exact full git SHA, but that did not fix it.

[bibz]

Through @itajaja's analysis, two bugs can be pointed out:

1. A git dependency will always be cloned again (see version 0.12.17: poetry/installation/pip_installer.py#L215, also valid for current master @ f4803e91bca440e6f4af6ad54e396d01c024a76f). This is unrelated to any version constraints.
2. The git dependency does not get locked properly if a constraint is explicitly set. Whatever is defined in the branch, rev, or tag arguments gets copied as-is as the package.source.reference instead of the full commit SHA-1.

The first bug is well described in OP, but I believe the second one should have its own space.

[bibz]

I cannot reproduce the behaviour since 1.0.0b4 (tested up to and including 1.0.0b9), I think it has been fixed with bf2d515a.

@itajaja Can you still reproduce?

[itajaja]

nice, I'll try. I am still using <1 but I can temporarily switch to beta

[sdispater]

It shoud be fixed in now that version 1.0.0 is released. Closing.

[jrmlhermitte]

This is still happening for me. When I run poetry install and run it a second time, all my git repositories are cloned.
For repositories for example:

[tool.poetry.dependencies]
python = "^3.7"
A = { git = "https://github.com/XXX/XXX.git", tag="v0.1.5" }
(5 more)

I keep seeing

Package operations: 0 installs, 6 updates, 0 removals

where the 6 updates are the git repositories.

With git, I am using "insteadOf" global option:

git config --global url."https://SECRET_KEY:@github".insteadOf https://github

I am not sure if that could be a culprit.
Has this failed for anyone else?

(Thanks for the wonderful library as well, well done! :-) )

[bibz]

I cannot reproduce anymore (just tried again with 1.0.2).

Can you please give a minimum reproducible example?
What version of Poetry are you using?

[jrmlhermitte]

I will try to send one a bit later today.
I am using 1.0.0 as well, so I will try to upgrade.

I do have one question. Is the github url used to compute the hash as well?
Because in my toml file I have a different git url from what actually gets downloaded (since I have setup git to add my API key in the url).

Anyway, I'll respond later today. Thanks so much for the quick response!

[jrmlhermitte]

ok, here's a working example:

[build-system]
requires = [ "poetry>=1.0.0",]
build-backend = "poetry.masonry.api"

[tool.poetry]
name = "test_lib"
version = "0.0.1"
description = "Test Lib"
authors = [ "foo [email protected]>",]
readme = "README.md"
repository = "https://github.com/foo/foo"
homepage = "https://github.com/foo/foo"

[tool.semantic_release]
version_variable = "version.py:version"
version_source = "commit"
upload_to_pypi = false
commit_message = "semantic release version change commit"

[tool.poetry.dependencies]
python = "^3.7"
feedparser = { git = "https://github.com/kurtmckee/feedparser", tag = "v5.1.3" }


[tool.poetry.dev-dependencies]

[tool.poetry.scripts]

Then calling twice:

XXXXXXX$ poetry install
Creating virtualenv test-lib-1FOm0thz-py3.7 in /XXXXX/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (5.4s)

Writing lock file


Package operations: 1 install, 0 updates, 0 removals

  - Installing feedparser (5.1.3 73f60e6)
XXXX$ poetry install
Installing dependencies from lock file


Package operations: 0 installs, 1 update, 0 removals

  - Updating feedparser (5.1.3 4056bbb -> 5.1.3 73f60e6)

my poetry version:

$ poetry version
Poetry version 1.0.2

Do you encounter the same issues? Thanks so much for the response!

[bibz]

Thanks for the MRE.

I can reproduce your example now.
You found a different bug than we had, due to how git handled ~signed~ annotated tags.

To summarise:

• The (annotated) tag v5.1.3 itself is a git object with the short sha 73f60e6
• The tree pointed to by the tag has the short sha 4056bbb

The lock file refers to the tag object, because it is what v5.1.3 resolves to directly.
Since the checked-out tree has a different sha, poetry thinks it is not up-to-date (the references are different) and then re-install it.

I think we need a new issue for this.

[jrmlhermitte]

Ah that is great, thank you so much for the investigation! I have created issue #1916.