Poetry: setting up gemfury is not obvious

Created on 21 Feb 2019  ·  12Comments  ·  Source: python-poetry/poetry

  • [x] I am on the latest Poetry version.
  • [x] I have searched the issues of this repo and believe that this is not a duplicate.
  • [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
  • OS version and name: macOS 10.14.3 (18D109)
  • Poetry version: 0.12.11

Issue

I tried following the instructions at https://poetry.eustace.io/docs/repositories/ but it doesn't seem to be working.

❯ poetry config repositories.REDACTED https://pypi.fury.io/REDACTED/

❯ poetry config http-basic.dialogue jasonkuhrt

Password:


❯ poetry config --list
settings.virtualenvs.create = true
settings.virtualenvs.in-project = false
settings.virtualenvs.path = "/Users/jasonkuhrt/Library/Caches/pypoetry/virtualenvs"
repositories.dialogue.url = "https://pypi.fury.io/REDACTED/"

❯ head pyproject.toml
REDACTED


[[tool.poetry.source]]
name = "REDACTED"
url = "https://pypi.fury.io/REDACTED/"

❯ poetry add REDACTED

[PackageNotFound]
Package [REDACTED] not found.

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...
picture jasonkuhrt
👀6

Most helpful comment

These are the instructions we currently have at Dialogue internally. Hope its useful to anyone else. I would be happy to make a PR to the official docs but not sure if these instructions are flawed in some way.

Publishing Packages

Steps

  • Register the dialogue package repository

    poetry config repositories.dialogue https://pypi.fury.io/dialogue

  • Setup auth to dialogue repo

    poetry config http-basic.dialogue $GEMFURY_TOKEN $GEMFURY_TOKEN

  • Specify dialogue repo when publishing. Do not specify -p or -u (auth done in previous step)

    poetry publish --build --repository dialogue

Example

❯ poetry publish -r dialogue --build

Building foobar (0.1.2)
 - Building sdist
 - Built foobar-0.1.2.tar.gz

 - Building wheel
 - Built foobar-0.1.2-py3-none-any.whl

Publishing foobar (0.1.2) to dialogue
 - Uploading foobar-0.1.2-py3-none-any.whl 100%
 - Uploading foobar-0.1.2.tar.gz 100%

Using Dialogue gemfury private repository

Steps:

  • Add in pyproject.toml:

    [[tool.poetry.source]]
name = "dialogue"
url = "https://pypi.fury.io/dialogue"

  • To poetry add or poetry remove , setup poetry with gemfury token for user and pass like so:

    $ poetry config http-basic.dialogue $GEMFURY_TOKEN $GEMFURY_TOKEN

    In CI, since changes are not done, http-basic config is not needed, just the env var which is present in contect via org-global-v2. However, as you can see in publishing setup, this step is always needed for that.

  • In order to do poetry install with private packages listed in lock file, must have PIP_EXTRA_INDEX_URL set globally like so:

    $ echo $PIP_EXTRA_INDEX_URL
https://pypi.fury.io/$GEMFURY_TOKEN/dialogue/
picture jasonkuhrt on 19 Apr 2019
👍11

All 12 comments

What is the error message? Have you tried to curl the simple endpoint manually?

cenzhe picture cenzhe on 6 Mar 2019

Hey @cenzhe,

We discovered that we had not configured things properly.

In order to make changes like poetry add or poetry remove with private packages, we must have setup gemfury token for user and pass like so:

$ poetry config http-basic.dialogue $GEMFURY_TOKEN $GEMFURY_TOKEN

In order to do poetry install with private packages listed in lock file, must have PIP_EXTRA_INDEX_URL set globally like so:

$ echo $PIP_EXTRA_INDEX_URL
https://pypi.fury.io/$GEMFURY_TOKEN/dialogue/

Observations:

  1. This wasn't obvious from either the poetry or gemfury docs
  2. there were no particularly helpful error messges from poetry

I realize this may be/is an integration issue, but I think it could only help poetry to take it upon itself to have a doc section for various private PyPI registries.

We were _very_ close to delaying adoption of poetry until one team member cracked the issue.

I've renamed the issue title to reflect this new information

jasonkuhrt picture jasonkuhrt on 6 Mar 2019
👍6

Agree - the docs have placeholder values in their examples, but don't explain what they mean! An actual worked example would be really helpful.

robertlagrant picture robertlagrant on 6 Mar 2019
👍1

These are the instructions we currently have at Dialogue internally. Hope its useful to anyone else. I would be happy to make a PR to the official docs but not sure if these instructions are flawed in some way.

Publishing Packages

Steps

  • Register the dialogue package repository

    poetry config repositories.dialogue https://pypi.fury.io/dialogue

  • Setup auth to dialogue repo

    poetry config http-basic.dialogue $GEMFURY_TOKEN $GEMFURY_TOKEN

  • Specify dialogue repo when publishing. Do not specify -p or -u (auth done in previous step)

    poetry publish --build --repository dialogue

Example

❯ poetry publish -r dialogue --build

Building foobar (0.1.2)
 - Building sdist
 - Built foobar-0.1.2.tar.gz

 - Building wheel
 - Built foobar-0.1.2-py3-none-any.whl

Publishing foobar (0.1.2) to dialogue
 - Uploading foobar-0.1.2-py3-none-any.whl 100%
 - Uploading foobar-0.1.2.tar.gz 100%

Using Dialogue gemfury private repository

Steps:

  • Add in pyproject.toml:

    [[tool.poetry.source]]
name = "dialogue"
url = "https://pypi.fury.io/dialogue"

  • To poetry add or poetry remove , setup poetry with gemfury token for user and pass like so:

    $ poetry config http-basic.dialogue $GEMFURY_TOKEN $GEMFURY_TOKEN

    In CI, since changes are not done, http-basic config is not needed, just the env var which is present in contect via org-global-v2. However, as you can see in publishing setup, this step is always needed for that.

  • In order to do poetry install with private packages listed in lock file, must have PIP_EXTRA_INDEX_URL set globally like so:

    $ echo $PIP_EXTRA_INDEX_URL
https://pypi.fury.io/$GEMFURY_TOKEN/dialogue/
jasonkuhrt picture jasonkuhrt on 19 Apr 2019
👍11

I'm not sure if specifying $GEMFURY_TOKEN as the password is strictly necessary. I use this command just fine across several Dockerfiles:

poetry config http-basic.repo $GEMFURY_TOKEN ""

We use this command on our local development machines too, to set up our private Gemfury repository across all of our projects. The repository for each package installed is recorded in poetry.lock so we don't need the PIP_EXTRA_INDEX_URL environment variable.

connorbrinton picture connorbrinton on 18 Oct 2019
👍2

We still have the problem from time to time as soon as the cache gets bumped and also when we push to gemfury. yes the above solution works but since poetry 1.0.0 it is getting clunky.

[TypeError]
quote_from_bytes() expected bytes

Traceback (most recent call last):
  File "/home/circleci/.poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/home/circleci/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/home/circleci/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/home/circleci/.poetry/lib/poetry/_vendor/py3.8/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/home/circleci/.poetry/lib/poetry/console/commands/install.py", line 63, in handle
    return_code = installer.run()
  File "/home/circleci/.poetry/lib/poetry/installation/installer.py", line 74, in run
    self._do_install(local_repo)
  File "/home/circleci/.poetry/lib/poetry/installation/installer.py", line 286, in _do_install
    self._execute(op)
  File "/home/circleci/.poetry/lib/poetry/installation/installer.py", line 302, in _execute
    getattr(self, "_execute_{}".format(method))(operation)
  File "/home/circleci/.poetry/lib/poetry/installation/installer.py", line 327, in _execute_install
    self._installer.install(operation.package)
  File "/home/circleci/.poetry/lib/poetry/installation/pip_installer.py", line 63, in install
    index_url = repository.authenticated_url
  File "/home/circleci/.poetry/lib/poetry/repositories/legacy_repository.py", line 224, in authenticated_url
    password=quote(self._auth.auth.password),
  File "/usr/local/lib/python3.8/urllib/parse.py", line 839, in quote
    return quote_from_bytes(string, safe)
  File "/usr/local/lib/python3.8/urllib/parse.py", line 864, in quote_from_bytes
    raise TypeError("quote_from_bytes() expected bytes")
mbelang picture mbelang on 18 Dec 2019

Couple quick comments since I happened upon this issue while looking for another:

  1. The quote_from_bytes issue mentioned above is actually fixed in later versions of Poetry (1.0.2 I think?)
  2. We use Gemfury for private dependencies at my work and don't have to set the global PIP_EXTRA_INDEX_URL like you have mentioned. What's the error you're getting if you don't set that?
dbanty picture dbanty on 2 Feb 2020

@dbanty I notice that I do not see the error anymore since at least 1.0.2 update. So this is probably fixed. I think we could close the issue. In fact it is probably fixed by 1.0.1 as 1.0.2 is a revert commit of a fix from 1.0.1.

mbelang picture mbelang on 2 Feb 2020

i have documented here how to do it with env vars:
https://gist.github.com/sloev/357be5288a6baad78747c801464fb9f1

it says:

a response to
https://github.com/python-poetry/poetry/issues/910

$ cat pyproject.toml

[[tool.poetry.source]]
name = "gemfury"
url = "https://pypi.fury.io/GEM_FURY_USERNAME"

[tool.poetry.dependencies]
python = "^3.7"
secret_package = {version = "^2.0.0", source = "gemfury"}

Then you write the locally checked in non-dangerous poetry.toml file in the project directory.

$ cat poetry.toml
[repositories]
[repositories.gemfury]
url = "https://pypi.fury.io/GEM_FURY_USERNAME"

then you can install with:

POETRY_HTTP_BASIC_GEMFURY_PASSWORD=GEMFURY_TOKEN \
  POETRY_HTTP_BASIC_GEMFURY_USERNAME=GEMFURY_TOKEN \
  poetry install

in this way the token is not cached during ci or anything

sloev picture sloev on 17 Mar 2020

@sloev I just got the problem again and tried your solution and it works but I do not understand why it only started to happen with poetry 1.1.0.

our config before:

cat pyproject.toml

[[tool.poetry.source]]
name = "gemfury"
url = "https://pypi.fury.io/GEM_FURY_USERNAME"

### Env setup
export PIP_EXTRA_INDEX_URL-=https://pypi.fury.io/${GEMFURY_TOKEN}/${GEM_FURY_USERNAME}/

This worked for a while now. Since poetry 1.0.2 and it also worked with poetry 0.12.x up until poetry 1.0.10.

So why this has changed in 1.1.0?

mbelang picture mbelang on 2 Oct 2020

So why this has changed in 1.1.0?

With 1.1.0 there has been a rewrite of our installer. The reason why your old config worked, I believe is because of your repository configuration in PIP_EXTRA_INDEX_URL. Prior to 1.1.0 Poetry used pip to download artifacts to install. This is now managed by Poetry itself, thereby your pip global configuration will not be used at all.

https://python-poetry.org/blog/announcing-poetry-1-1-0.html#brand-new-installer

abn picture abn on 2 Oct 2020

Ohhh that makes a lot of sens but this is very odd as I have the problem only inside docker build and not on my local machine :thinking:

mbelang picture mbelang on 2 Oct 2020
👀1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

kierun picture kierun  ·  3Comments
ghost picture ghost  ·  3Comments
EdgyEdgemond picture EdgyEdgemond  ·  3Comments
jeremy886 picture jeremy886  ·  3Comments
alexlatchford picture alexlatchford  ·  3Comments