Poetry: Private registries not taken into consideration in local dependencies

[jonapich]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04

• Poetry version: 1.0.5

Issue

When you reference a local path as a dependency, and that reference has a registry setup, it's not taken into account during poetry install:

in repo/bigproject/pyproject.toml:

[tool.poetry.dependencies]
local-dep = { path = "../local-dep" }

in repo/local-dep/pyproject.toml:

[tool.poetry.dependencies]
private-lib = "*"

[tool.poetry.sources]
...  # private repo setup here

• I can lock from repo/local-dep just fine (it checks the private repo :+1:)
• I cannot lock from repo/bigproject; it will not attempt to use the private registry

Workaround: Add the repo information in repo/bigproject/pyproject.toml even though it doesn't technically needs it.

Sidenote: Overall the local dependencies feature seems to have opened a can of worms. For instance:

• multiple references will create oddities in some lock files such as this-lib = 'somepath/../someotherpath/../this-lib/ where it should just be this-lib/.
• you can't build a sdist, because the source gets copied in a temporary folder without the local references. the paths are then evaluated from the temp folder and not from the original pyproject.toml file location and it will eventually fail saying the local package cannot be found.
• private repositories need to be copied everywhere (what the current issue is about)

[jonapich]

possibly related issues:

2078

2178

(now I wish I had 2278 :laughing: )

[jonapich]

Today, I tried a much simpler scenario, and I'm still stuck. I'm not sure what poetry does, that breaks pip...

Given the following pyproject.toml:

[tool.poetry.dependencies]
my-lib = { version = "*", source = "my.pypi" }

[[tool.poetry.source]]
name = "my.pypi"
url = "https://my.pypi.com/simple"

With this, poetry.lock will list all dependencies like before, and only the my-lib dependency will have a source defined, which is really what is expected here (although, I'm not sure what would happen if the dependency also had a dependency on my private pypi...)

Unfortunately:

$ poetry install
Creating virtualenv... (new venv created)
Installing dependencies from lock file

Package operations: 9 installs, 0 updates, 0 removals

  - Installing certifi (2019.11.28)
  - Installing chardet (3.0.4)
  - Installing idna (2.9)
  - Installing inflection (0.3.1)
  - Installing typing-extensions (3.7.4.1)
  - Installing urllib3 (1.22)
  - Installing attrs (19.3.0)
  - Installing my-lib (0.0.1)

[EnvCommandError]
Command ['/path/to/.virtualenvs/kVqTLuGo-py3.6/bin/pip', 'install', '--no-deps', '--index-url', 'https://my.pypi.com/simple', '--extra-index-url', 'https://pypi.org/simple/', 'my-lib==0.0.1'] errored with the following return code 2, and output: 
Collecting my-lib==0.0.1
Exception:
Traceback (most recent call last):
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/path/to/.virtualenvs/kVqTLuGo-py3.6/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/my-lib/

However, if I run the exact same thing (outside of a virtual environment), it works:

$ /path/to/.virtualenvs/kVqTLuGo-py3.6/bin/pip install --no-deps --index-url https://my.pypi.com/simple --extra-index-url https://pypi.org/simple/ --no-cache my-lib==0.0.1
Collecting my-lib==0.0.1
  Downloading https://my.pypi.com/api/package/my-lib/my_lib-0.0.1-py3-none-any.whl
Installing collected packages: my-lib
Successfully installed my-lib-0.0.1

I'm thinking there may be an environment variable to blame here :thinking:

[jonapich]

I found a way to fix it... I can't explain why though.

https://github.com/python-poetry/poetry/blob/master/poetry/utils/env.py#L889

Adding shell=True to that subprocess.check_output() call will fix the issue for me and correctly install my-lib from my private pypi.

(should I write a separate issue?)

[snejus]

@jonapich I had exactly the same issue and I found that my virtual environment used pip= ^9.0.0(even though v20 is available) and thought that this could have been related to the issue. I upgraded it with poetry run pip install pip==19.3.1 and the issue was gone. I use poetry 1.1.0a1 if that helps by the way.

It looks like when a virtual environment gets created, it installs pip version that was bundled together with the required python release (python 3.6.9 in my case (I see you're using py3.6 too), which was bundled with pip 9). Currently, it seems that there isn't a way to override this behaviour through poetry, so we'll have to keep upgrading pip manually before this is fixed (WIP here: #1971). See #732 , #1661 , #1962 and #1651 for similar issues.