Poetry: It is possible to add a dependency of a private git repo?

[Hammond95]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Question

It is possible to add a dependency of a private git repo?

I've just stated to use poetry, I quite like some of the features, but i'm now stuck since installing a private repo doesn't seem to work...

toml file extract

[tool.poetry.dependencies]
python = "~3.6"  # Compatible python versions must be declared here
toml = "^0.9"

# Git dependencies
myprivaterepo = { git = "[email protected]:myorganization/myprivaterepo.git", branch = "master" }

I've also tried with different git url forms:

• https://github.com/myorganization/myprivaterepo.git
• git+ssh://[email protected]/myorganization/myprivaterepo.git

the corresponding pip command i have used till now is:

pip install git+ssh://[email protected]/myorganization/myprivaterepo.git

This is the error i get:

Updating dependencies
Resolving dependencies... (0.0s)

[AssertionError]  

Exception trace:
 /Users/mdeluca/.poetry/lib/poetry/_vendor/py3.6/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /Users/mdeluca/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /Users/mdeluca/.poetry/lib/poetry/_vendor/py3.6/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /Users/mdeluca/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /Users/mdeluca/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /Users/mdeluca/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /Users/mdeluca/.poetry/lib/poetry/console/commands/update.py in handle() at line 41
   return installer.run()
 /Users/mdeluca/.poetry/lib/poetry/installation/installer.py in run() at line 76
   self._do_install(local_repo)
 /Users/mdeluca/.poetry/lib/poetry/installation/installer.py in _do_install() at line 158
   ops = solver.solve(use_latest=self._whitelist)
 /Users/mdeluca/.poetry/lib/poetry/puzzle/solver.py in solve() at line 38
   packages, depths = self._solve(use_latest=use_latest)
 /Users/mdeluca/.poetry/lib/poetry/puzzle/solver.py in _solve() at line 171
   self._package, self._provider, locked=locked, use_latest=use_latest
 /Users/mdeluca/.poetry/lib/poetry/mixology/__init__.py in resolve_version() at line 7
   return solver.solve()
 /Users/mdeluca/.poetry/lib/poetry/mixology/version_solver.py in solve() at line 79
   next = self._choose_package_version()
 /Users/mdeluca/.poetry/lib/poetry/mixology/version_solver.py in _choose_package_version() at line 380
   for incompatibility in self._provider.incompatibilities_for(version):
 /Users/mdeluca/.poetry/lib/poetry/puzzle/provider.py in incompatibilities_for() at line 447
   for dep in dependencies
 /Users/mdeluca/.poetry/lib/poetry/puzzle/provider.py in <listcomp>() at line 447
   for dep in dependencies
 /Users/mdeluca/.poetry/lib/poetry/mixology/incompatibility.py in __init__() at line 59
   assert by_ref[ref] is not None

[jasongi-actu]

You need to specify the protocol:

myprivaterepo = { git = "ssh://[email protected]/myorganization/myprivaterepo.git", branch = "master" }

Or just add using the --git flag

poetry add myprivaterepo --git ssh://[email protected]/myorganization/myprivaterepo.git

See here for more info on the add command https://poetry.eustace.io/docs/cli/#add

[sdispater]

@Hammond95 Is your issue fixed now?

[Peque]

@jasongi-actu 's solution did not fix the issue for me. I get this error:

[RuntimeError]
Unable to retrieve the package version for /tmp/pypoetry-git-mypackagekg2xNw

Dependency is declared as:

mypackage = { git = "ssh://[email protected]:12345/organization/mypackage.git", tag = "0.1.2" }

Update

Using Poetry version 1.0.0a2. Tried with branch = "master" instead of tag = "x.x.x", without success.

[Peque]

@jasongi-actu Did you try that command you suggested? Did it work for you? I even tried on a public repository I have in GitHub (only specifying the ssh:// protocol and it did not work. The documentation you linked seems to not mention anything about SSH protocol (assumes HTTPS, with no SSH authentication).

[jasongi-actu]

Yes I use that command all the time in multiple public/private github repos. I think the next step in solving this would be to get a reproducible test case.

Also, are you running PIP <= 18.1? There are issues with git dependencies and poetry using pip 19.xx

[Peque]

@jasongi-actu I am indeed running pip==18.1:

$ poetry run pip --version
pip 18.1 from [...]

I can reproduce the error by trying to install a public repo of my own:

[tool.poetry.dependencies]
markdownreveal = { git = "ssh://[email protected]/Peque/markdownreveal.git", branch = "master" }

Whereas I am able to:

git clone ssh://[email protected]/Peque/markdownreveal.git`

I guess if you fork it and are able to install from your own fork using the SSH protocol, then it is related to differences between our systems/configurations/environments, not depending on the project that it is to be installed. In which case I am not sure how to proceed to debug.

[Peque]

@jasongi-actu Can you reproduce the issue when forking markdownreveal and setting this dependency?

[tool.poetry.dependencies]
markdownreveal = { git = "ssh://[email protected]/jasongi-actu/markdownreveal.git", branch = "master" }

Alternatively, could you provide a repo that works fine for you so that I can fork it and try it myself?

[dsevero]

@Peque I ran poetry add <repo> --git ssh://[email protected]/<org>/<repo>.git -vvv and it worked like a charm!

======

 * Version: 0.12.12
 * Python:  3.7.1


Virtualenv
==========

 * Python:         3.7.1
 * Implementation: CPython
 * Path:           /home/severo/.cache/pypoetry/virtualenvs/<repo>-py3.7
 * Valid:          True


System
======

 * Platform: linux
 * OS:       posix
 * Python:   /home/severo/miniconda3/envs/<repo>

pip 19.1 from /home/severo/.cache/pypoetry/virtualenvs/<repo>-py3.7/lib/python3.7/site-packages/pip (python 3.7)

hope it helps!

[Hammond95]

Sorry for the late reply, tried @jasongi-actu proposed solution (adding the ssh protocol), but this still doesn't work...

This is the error I get:

[CalledProcessError]
Command '['git', 'clone', 'ssh://[email protected]:Organization/myRepo.git', '/var/folders/d8/0yv7fd_j5rn0cjrsk5dp09cc0000
gp/T/pypoetry-git-myproject243mqkhk']' returned non-zero exit status 128.

Running manually the corresponding command

git clone ssh://[email protected]:Organization/myRepo.git /var/folders/d8/0yv7fd_j5rn0cjrsk5dp09cc0000gp/T/pypoetry-git-myproject243mqkhk

results into the following error:

Cloning into '/var/folders/d8/0yv7fd_j5rn0cjrsk5dp09cc0000gp/T/pypoetry-git-myproject243mqkhk'...
ssh: Could not resolve hostname github.com:Organization: nodename nor servname provided, or not known
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

running manually the same command but without the ssh:// in front

git clone [email protected]:Organization/myRepo.git /var/folders/d8/0yv7fd_j5rn0cjrsk5dp09cc0000gp/T/pypoetry-git-myproject243mqkhk

works fine.

I get the same errors with the poetry add command.

PS. I am using poetry version 0.12.11

[Hammond95]

Updating poetry to 0.12.14 partially solved the issue, it manage to get to the resolving dependencies phase, but then it fails because it can't find pip.

Installing dependencies from lock file


Package operations: 55 installs, 0 updates, 0 removals

  - Installing pycparser (2.19)

[EnvCommandError]
Command ['/Users/mdeluca/Library/Caches/pypoetry/virtualenvs/mypackage-py3.6/bin/python', '-m', 'pip', 'install', '
--no-deps', 'pycparser==2.19'] errored with the following output:
/Users/mdeluca/Library/Caches/pypoetry/virtualenvs/mypackage-py3.6/bin/python: No module named pip

I didn't created this env (maybe poetry did?); I am using a conda env to make poetry use a python3.6 interpreter.

[thedrow]

There's an infinite loop somewhere.
When I run poetry add dependencies --git [email protected]:dry-python/dependencies.git -vvv I get the following repeated log message:

   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5

It runs forever and reaches 100% CPU.
The repository is public.

Is this the same bug or a different one?

[ekhaydarov]

ummmm the --git option does not exist anymore?

[NoSuchOptionException]
The "--git" option does not exist.

you can just put it directly into toml file and run poetry update

[stephsamson]

@Hammond95 I will close this issue now since your particular issue seems to be resolved and @dsevero has also offered an alternative solution. Furthermore, you can add git dependencies since version 1 using just poetry add without any options.

@thedrow that looks like a different bug and the --git option does not exist since version 1.0 as @ekhaydarov has said.

[darakian]

@stephsamson

Can I request a reopen of this issue? I'm using poetry 1.0.0 withpip 19.2.3 and get the

[RuntimeError]
Unable to retrieve the package version for /var/folders/3g/fpzgglls03vbzcvbwvgw0r540000gn/T/pypoetry-git-myrepo-fooGTw158

error.

Digging into the code it looks like lines 340-348 of poetry/puzzle/provider.py are my issue.

                if not result["version"]:
                    # The version could not be determined
                    # so we raise an error since it is mandatory
                    print("Result {}".format(result))
                    raise RuntimeError(
                        "Unable to retrieve the package version for {}".format(
                            directory
                        )
                    )

I added a simple print out of the result dict and get

Result {'install_requires': [], 'extras_require': {}, 'version': None, 'python_requires': None, 'name': u'myrepofoo'}

In my case I'm matching on a tag which is actually how we're versioning our dependency. In our setup.py file we have a more or less static version set, but if there's a method that poetry would accept then I'd happily add that. I've attempted using the branch directive as well and get the same behavior.

Thanks.

[finswimmer]

Hey @darakian ,

please open a new issue, where you describe your problem by showing your pyproject.toml, what steps need to be done to reproduce your problem, what's your expected behavior and what happens instead.

Thanks!

fin swimmer

[darakian]

Hey @finswimmer

I ended up realizing that it was an error on my end (sorry). Turns out that our code was using distutils rather than setuptools and the lines
https://github.com/python-poetry/poetry/blob/636ce8b0eba7dfa390b3fd961d1b9fb533d5d033/poetry/puzzle/provider.py#L324-L332
were borking out as the egg_info command was failing. A one line change of

from distutils.core import setup

to

from setuptools import setup

in our setup.py file fixed up the poetry install.

[dazza-codes]

poetry 1.x

[NoSuchOptionException]
The "--git" option does not exist.

Not enough information in the help too:

$ poetry --version
Poetry version 1.0.2

$ poetry help add
USAGE
  poetry add [-D] [-E <...>] [--optional] [--python <...>] [--platform <...>] [--allow-prereleases] [--dry-run] <name1> ... [<nameN>]

ARGUMENTS
  <name>                 The packages to add.

OPTIONS
  -D (--dev)             Add as a development dependency.
  -E (--extras)          Extras to activate for the dependency. (multiple values allowed)
  --optional             Add as an optional dependency.
  --python               Python version for which the dependency must be installed.
  --platform             Platforms for which the dependency must be installed.
  --allow-prereleases    Accept prereleases.
  --dry-run              Output the operations but do not execute anything (implicitly enables --verbose).

GLOBAL OPTIONS
  -h (--help)            Display this help message
  -q (--quiet)           Do not output any message
  -v (--verbose)         Increase the verbosity of messages: "-v" for normal output, "-vv" for more verbose output and "-vvv" for debug
  -V (--version)         Display this application version
  --ansi                 Force ANSI output
  --no-ansi              Disable ANSI output
  -n (--no-interaction)  Do not ask any interactive question

DESCRIPTION
  The add command adds required packages to your pyproject.toml and installs them.

  If you do not specify a version constraint, poetry will choose a suitable one based on the available package versions.

$ poetry add ssh://[email protected]/org/private-repo.git

[ValueError]
Could not parse version constraint: //[email protected]/org/private-repo.git

😞

[finswimmer]

@dazza-codes: The correct syntax would be:

$ poetry add git+ssh://[email protected]/org/private-repo.git

fin swimmer