poetry add git+https:// will fail with CalledProcessError if package name includes illegal file name characters

[tanj]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• Microsoft Windows 10 Pro: Version 10.0.18363 Build 18363:
• Poetry version: 1.1.4:
• Link of a Gist with the contents of your pyproject.toml file:

Issue

I was starting a new project and adding some dependencies and came across a bug in the git clone process.

 % poetry add git+https://github.com/tanj/swissarmy.git

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing python-dateutil (2.8.1)
  • Installing swiss army: untility package (0.1 7e67114)

  CalledProcessError

  Command '['git', 'clone', '--recurse-submodules', 'https://github.com/tanj/swissarmy.git', 'C:\\Users\\REDACTED\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\gkeep-organizer-mzsur4FB-py3.8\\src\\swiss army: untility package']' returned non-zero exit status 128.

  at ~\.poetry\lib\poetry\utils\_compat.py:217 in run
      213│                 process.wait()
      214│                 raise
      215│             retcode = process.poll()
      216│             if check and retcode:
    → 217│                 raise CalledProcessError(
      218│                     retcode, process.args, output=stdout, stderr=stderr
      219│                 )
      220│         finally:
      221│             # None because our context manager __exit__ does not use them.


Failed to add packages, reverting the pyproject.toml file to its original content.

The issue is that my swissarmy package has a : in the package name and that is an illegal character for a windows file path.

The offending code is located at https://github.com/python-poetry/poetry/blob/639d5e05773ed8b91b80ec03c2ee64f500534c10/poetry/installation/executor.py#L562

I realize that naming my package like that probably is breaking some PEP, but python itself has no issue installing and using the package...

pip freeze output for that package
-e git+https://github.com/tanj/swissarmy.git@7e671142565cb2b2aa04292ec0775c024bf47547#egg=Swiss_Army_Untility_Package

[sinoroc]

My first impression, is that I do not think it is fair to classify this as a bug in _poetry_. But...

One would need to check exactly what the standard (if any) says for such a case. Maybe indeed _poetry_ is supposed to apply some name normalization somewhere and it does not currently...

@tanj Have you managed to identify the corresponding text of a standard for project names? What is allowed and what is not? Should _poetry_ do some name normalization or should it reject such a project from installation?

python itself has no issue installing

@tanj You mean _pip_? Or do you mean python setup.py install?

[sinoroc]

The best I could find in a short time is:

The name of the distribution. The name field is the primary identifier for a distribution. A valid name consists only of ASCII letters and numbers, period, underscore and hyphen. It must start and end with a letter or number. Distribution names are limited to those which match the following regex (run with re.IGNORECASE):

^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$

-- https://packaging.python.org/specifications/core-metadata/#name

See also: https://www.python.org/dev/peps/pep-0508/#names

And indeed:

$ python -c "import re; print(re.match(r'^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$', 'Swiss Army: Untility Package', re.IGNORECASE))"
None

[uda]

Update: this comment is irrelevant to the issue

I have the same issue, but I am trying to install https://github.com/ageekinside/fastapi-jinja.git.

• Tried pip install git+https://github.com/ageekinside/fastapi-jinja.git and it wroked just fine
• Added the dependency to pyproject.toml as fastapi-jinja = { git = "https://github.com/ageekinside/fastapi-jinja.git", branch = "master" } and ran poetry lock and still failed

What did work? I cloned the repo, Ran python setup.py sdist bdist_wheel, Copied both tar.gz and whl files into the project and ran poetry add ./filename and no errors were given.

What I think is that the git (or maybe any vcs) installation is inconsistent with the other installation paths.

[sinoroc]

@uda But there are no illegal characters in the name of the project you are trying to install, are there? Maybe you would need to add your voice to a different ticket. Maybe this one: https://github.com/python-poetry/poetry/issues/3366, or just simply correct your pyproject.toml to read branch = 'main', since it seems to be your actual issue.

[uda]

Oy, thanks @singulared , I didn't even realize that that's the issue...

[tanj]

python itself has no issue installing

@tanj You mean _pip_? Or do you mean python setup.py install?

python setup.py install

It appears that when installing in that way setuptools strips the offending characters. I did go looking briefly before posting this bug report but didn't find PEP508 or the page you found.

Maybe I should report this to setuptools? I would think a warning should be issued if they are modifying the package name to fit PEP508.

[sinoroc]

python itself has no issue installing

@tanj You mean _pip_? Or do you mean python setup.py install?

python setup.py install

OK, thanks for the confirmation.

It appears that when installing in that way setuptools strips the offending characters. I did go looking briefly before posting this bug report but didn't find PEP508 or the page you found.

No worries, it is a labyrinth... :D

Maybe I should report this to setuptools? I would think a warning should be issued if they are modifying the package name to fit PEP508.

I guess, it is worth asking them the question. Yes a warning sounds good to me.

_[I do not know if it can be considered a bug at this point. _setuptools_ (and _distutils_ before that) are so old now, that by now it must be considered a feature for some of their users. And removing this "_feature_" from _setuptools_ would only cause trouble. :D ]_

Maybe _poetry_ as well could be more careful here. I do not know if it is possible to check the metadata before creating the directory (with a name containing a :).