Pipenv: Pipenv doesn't support browser auth flow for Azure artifacts-keyring

Created on 19 Dec 2019 · 7Comments · Source: pypa/pipenv

Background

Trying to install a closed-source python package from my company's private package feed hosted on Azure DevOps, following these instructions.

The instructions, exactly as presented in that link, work fine using pip install, within a virtual environment (which I entered via pipenv shell). For readers unfamiliar with artifacts-keyring, it causes the pip install process to produce a message on stdout (well, I _assume_ it's stdout) with a URL to follow and a one-time code to copy, and then wait. The human has to manually browse to the URL, paste the one-time code, then log in to their Microsoft account, then close the browser tab to allow the pip installation process to continue.

But I want to use pipenv install directly if possible!

Notable here as well is that artifacts-keyring is currently in pre-release, which means #3947 is relevant, but I believe I've controlled for it below.

Issue description

This would not be directly reproducible for anyone else copy-pasting exactly from below, because the private package feed in question _is private_. You'll have to set up your own Azure DevOps Artifacts package feed if you want to completely replicate, I'm afraid. I have anonymized the organization name, package name etc as a precaution, since it wouldn't work either way.

Start in a virtual environment, with artifacts-keyring already successfully installed by pipenv (using --pre switch because it is a pre-release package).

Try to install my private package:

$ pipenv install mypackage --pre --verbose --skip-lock --pypi-mirror https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/

Including --skip-lock works around #3947. Without that flag, I am just reproducing #3947.
I don't think --pre is really necessary here but I threw it in just in case.

Expected result

Successful install. This would have to invoke the browser authentication flow, same as Pip does (provided one uses a totally fresh virtual env for this test; the user access token Pip obtained hangs around within a virtual env, I believe). After doing that I'd expect to get the usual happy messages from pipenv, and to see my Pipfile updated. (But not Pipfile.lock in this case.)

Actual result

The installation process churns for an unusually long time, mostly staying on the [== ] Installing... progress monitor. I don't see any authentication prompts. Then eventually it spews a lot more output and ends, all in quick succession.

The complete output is attached: pipenv traceback.txt.

Notably, this output clearly shows that multiple attempts were made to carry out the interactive browser authentication flow. But since I couldn't see them until too late, I couldn't do them, and there is apparently a 90-second timeout.

So pipenv needs to pass the inner output back up to the user's stdout much sooner, instead of presumably caching it?

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: 'c:\\python38\\lib\\site-packages\\pipenv'

Python location: 'c:\\python38\\python.exe'

Python installations found:

3.8.0: C:\Python38\python.exe
3.7.4: C:\Python37\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.0',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.18362',
 'python_full_version': '3.8.0',
 'python_version': '3.8',
 'sys_platform': 'win32'}

System environment variables:

ALLUSERSPROFILE
APPDATA
CHOCOLATEYINSTALL
COMMONPROGRAMFILES
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
COMPUTERNAME
COMSPEC
DRIVERDATA
FPS_BROWSER_APP_PROFILE_STRING
FPS_BROWSER_USER_PROFILE_STRING
HOMEDRIVE
HOMEPATH
LOCALAPPDATA
LOGONSERVER
NUMBER_OF_PROCESSORS
ONEDRIVE
ONEDRIVECOMMERCIAL
OS
PATH
PATHEXT
PIPENV_ACTIVE
PIP_DISABLE_PIP_VERSION_CHECK
PIP_PYTHON_PATH
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMDATA
PROGRAMFILES
PROGRAMFILES(X86)
PROGRAMW6432
PROMPT
PSMODULEPATH
PUBLIC
PYTHONDONTWRITEBYTECODE
SESSIONNAME
SNC_LIB
SYSTEMDRIVE
SYSTEMROOT
TEMP
TMP
UATDATA
USERDNSDOMAIN
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
VIRTUAL_ENV
WINDIR
PIP_SHIMS_BASE_MODULE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv├╗specific environment variables:

PIPENV_ACTIVE: 1

Contents of Pipfile ('C:\src\python\testcreds\Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
url = "https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/"
verify_ssl = true
name = "drpypi"

[pipenv]
allow_prereleases = true

[dev-packages]

[packages]
artifacts-keyring = "*"
mypackage = "*"

[requires]
python_version = "3.8"

Contents of Pipfile.lock ('C:\src\python\testcreds\Pipfile.lock'):
Omitted, not relevant, lockfile is outdated due to other issue.

triage

Source

philosophicles

👍8

All 7 comments

One potential workaround is to run this beforehand:

$ pipenv run keyring get -b artifacts_keyring.ArtifactsKeyringBackend https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/ VssSessionToken

It triggers the auth flow manually, and then caches the details so that pipenv install can use them later. This does mean requiring the venv to already exist, with artifacts-keyring installed unfortunately.

I'm currently scripting it up to

$ pipenv --python 3.7
$ pipenv run keyring get -b artifacts_keyring.ArtifactsKeyringBackend https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/ VssSessionToken
$ pipenv install --dev

It at least gets me a step further. Now I'm stuck at pipenv trying to resolve packages and not being able to find/auth to the repo.

keyz@mac-mini test % pipenv install --dev --verbose --pre
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✘ Locking Failed!
Using pip: -i https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/
Using pip: -i https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/
Using pip: -i https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/

                          ROUND 1
Current constraints:
  mypackage

Finding the best candidates:
Traceback (most recent call last):
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/utils.py", line 385, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/resolver.py", line 198, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/resolver.py", line 198, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/resolver.py", line 263, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 175, in find_best_match
    raise NoCandidateFound(ireq, all_candidates, self.finder)
pipenv.patched.piptools.exceptions.NoCandidateFound: Could not find a version that matches mypackage
No versions found
Was https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/ reachable?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/resolver.py", line 119, in main
    parsed.requirements_dir, parsed.packages)
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/resolver.py", line 85, in _main
    requirements_dir=requirements_dir,
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/resolver.py", line 69, in resolve
    req_dir=requirements_dir
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/utils.py", line 726, in resolve_deps
    req_dir=req_dir,
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
    resolved_tree = resolver.resolve()
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/utils.py", line 395, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches mypackage
No versions found
Was https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/ reachable?
Using pip: -i https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/
Using pip: -i https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/

                          ROUND 1
Current constraints:
  mypackage

Finding the best candidates:
Traceback (most recent call last):
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/utils.py", line 385, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/resolver.py", line 198, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/resolver.py", line 198, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/resolver.py", line 263, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 175, in find_best_match
    raise NoCandidateFound(ireq, all_candidates, self.finder)
pipenv.patched.piptools.exceptions.NoCandidateFound: Could not find a version that matches mypackage
No versions found
Was https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/ reachable?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/resolver.py", line 119, in main
    parsed.requirements_dir, parsed.packages)
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/resolver.py", line 85, in _main
    requirements_dir=requirements_dir,
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/resolver.py", line 69, in resolve
    req_dir=requirements_dir
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/utils.py", line 726, in resolve_deps
    req_dir=req_dir,
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
    resolved_tree = resolver.resolve()
  File "/Users/keyz/Library/Python/3.7/lib/python/site-packages/pipenv/utils.py", line 395, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches mypackage
No versions found
Was https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/ reachable?

keyz182 on 7 Mar 2020

Have you guys seen this post? https://github.com/microsoft/artifacts-keyring/issues/8

It was working for me until early this week. Perhaps Azure updated the artifacts authentication.

rinman24 on 10 Mar 2020

👍1

I wasn't aware that the artifacts-keyring project was also tracking this, no.

From the initial issue description:

I think it has to do with Pipenv running pip in subprocesses and hiding the stdout/stderr streams. This broke our development workflow.

That sounds about right to me, and I still think this is pipenv's issue to fix, not the responsibility of the artifacts-keyring project.

@rinman24 Do you mean the workaround described in that issue was working and has stopped? Or do you mean that pipenv was working more generally with artifacts auth for you until last week?

philosophicles on 16 Mar 2020

👍1

@philosophicles I mean that the workaround described in the issue using the PAT has stopped working. I agree that this is pipenv's issue to fix.

rinman24 on 24 Mar 2020

From what I can understand and see, pipenv simply does not support keyring by itself.
It cannot proceed with the Locking... phase, which means it never moves on to pip install which includes keyring support...

Correct me if I am wrong, but I don't see any reference to keyring in pipenv code.

ggirard07 on 9 Apr 2020

Is there any news on this issue? It is a rather important issue for working with ADO artifacts in python.

sheigel on 21 Aug 2020

combining work around here and in the other related issue, i got this to work:

# get a TOKEN
$ export ADO_ARTIFACTS_TOKEN=$(pipenv run keyring get -b artifacts_keyring.ArtifactsKeyringBackend https://pkgs.dev.azure.com/COMPANY/PROJECT/_packaging/FEED/pypi/simple/ VssSessionToken 2> /dev/null)

# Pipfile: set up the index URL to use the token
[[source]]
name = "pypi-ado"
url = "https://azure:${ADO_ARTIFACTS_TOKEN}@pkgs.dev.azure.com/COMPANY/PROJECT/_packaging/FEED/pypi/simple"
verify_ssl = true

# Install the package
$ pipenv install PACKAGE[==VERSION] -i pypi-ado [--verbose]