Poetry: Poetry cannot properly parse URL with Gitlab [deploy tokens]

[xinbinhuang]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04
• Poetry version: 1.0.3
• Link of a Gist with the contents of your pyproject.toml file:

Issue

Poetry cannot properly parse URL with gitlab deploy tokens. The project is hosted on a internal hosted Gitlab server.

The same git URL worked before, but I am not sure since when it is failing.

Command I ran:

poetry add "git+https://<token-name>:<token-key>@<my-org-self-hosted-gitlab-url>/<repo-path>/<repo-name>.git" -vvv

Output

[ValueError]
Invalid git url "git+https://<token-name>:<token-key>@<my-org-self-hosted-gitlab-url>/<repo-path>/<repo-name>.git"

Traceback (most recent call last):
  File "/home/binbin/.poetry/lib/poetry/_vendor/py3.7/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/home/binbin/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/home/binbin/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/home/binbin/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/home/binbin/.poetry/lib/poetry/console/commands/add.py", line 89, in handle
    packages, allow_prereleases=self.option('allow-prereleases')
  File "/home/binbin/.poetry/lib/poetry/console/commands/init.py", line 294, in _determine_requirements
    requires = self._parse_requirements(requires)
  File "/home/binbin/.poetry/lib/poetry/console/commands/init.py", line 371, in _parse_requirements
    parsed = ParsedUrl.parse(requirement)
  File "/home/binbin/.poetry/lib/poetry/vcs/git.py", line 118, in parse
    raise ValueError('Invalid git url "{}"'.format(url))

[xinbinhuang]

Just checked that it works when poetry is <=1.0.2. Maybe there is an incompatible change introduced in 1.0.3?

[finswimmer]

Hello @xinbinhuang,

thanks a lot for reporting. The git url parsing has changed significant within the last release to be more consistent. As I was not aware of something like deploy tokens, the regex will probably not match anymore.

I will take a closer look at this. Could you give me please an example how <token-name>:<token-key> looks like?

fin swimmer

[xinbinhuang]

Hi @finswimmer ,

Thanks for the quick response.

The Gitlab's doc is here.

I tried to generate a few different tokens until hitting an error.

# This is Gitlab's format requirements: 
# Here Username == <token-name>
Username can contain only letters, digits, '_', '-', '+', and '.'

So something like this [\w-\+\.] should work for <token-name>

For the <token-key>, it is auto generated. So far, I have only seen alphanumeric (letters + digits) characters, but it may also contains the same pattern as the <token-name>.

Let me know if you would need extra information or help from me.

Thanks
Bin

[jedie]

I am also running into this issue :( Any update or work-a-round here?

[xinbinhuang]

@jedie , while waiting for the bug to be fixed. You can pin your poetry version in your pyproject.toml

For example

[tool.poetry]
name = "new-package"
version = "0.1.0"
description = ""

...

[build-system]
requires = ["poetry<=0.12"]                 # this line: pin poetry to be <= 1.0.2
build-backend = "poetry.masonry.api"

[abn]

@xinbinhuang would making use of gitcredentials be the better apporach here? This would work both in your development and deployment scenarios. As an additional benifit, would allow you to ensure seperation of privilleges using different credentials.

As for respecting the git url spec, I agree we should ensure that a url of the form https://user:[email protected]/repo.git is accepted. However, I would not consider using these urls when adding dependencies the best practice. This is because that would mean that the credentials will end up in plain text in both your peotry.lock and pyproject.toml files. We should also considering logging warnings once this is fixed and a password is matched/detected.

A reasonable approach might be extend the URL validation to also include a new group "password". The user should stil consider : invalid.

[xinbinhuang]

@abn Thanks for the suggestion, I have not used gitcredentials before.

While I think your suggestion is valid in common scenarios for user passwords, it is a different case for Gitlab Deploy Tokens. These deploy tokens are not user credentials and designed specifically by Gitlab with limited permission scopes (similar to service account). So in the context of Gitalb Deploy Tokens, I believe it's valid.

While security suggestion by warnings is good (with optional disabling it), I don't think poetry as a package management tool should by any means enforce this validation and should leave this as a choice for the users.

I haven't used Gitlab for a while now, so it doesn't affect me that much. I like poetry and hope it will be successful in the future, so I hope poetry can make good decisions along the way.