Renovate: Python private package index support for poetry

[janjagusch]

What would you like Renovate to be able to do?

I would like Renovate to be able to update poetry.lock files that contain dependencies from a private package index.

Similar to #4487, you can add a hostRule, such as

{
      'baseUrl': 'https://my-private-pypi.com/',
      'username': 'username',
      'password': 'password'
}

to the config.js file. This allows Renovate to find outdated dependencies, but not update them. The result is that the pyproject.toml file gets updated, whereas the poetry.lock file does not.

From the log files, you can extract this error message:

Creating virtualenv venv in /home/ubuntu/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

[SolverProblemError]
Because X depends on Y (^0.Z.0) which doesn't match any versions, version solving failed.

I believe this has to do with Renovate using the renovate/python Docker image when trying to update artifacts. For poetry to be able to update dependencies from a private package index, it first needs to be configured, as described here.

Describe the solution you'd like

There should be a way to configure poetry from the config.js so that you can specify private package indices.

Describe alternatives you've considered

You should be able to use the global pre-configured poetry on your system, instead of the renovate/python Docker image.

[viceice]

looks like it can be configured with environment variables, so should be easy to implement. PR's welcome

[rarkins]

This should be pretty similar to what we do with Bundler artifacts updating. In that case we query for all hostRules matching hostType=bundler and then add an environment variable for each one found.

@janjagusch would a few commands like the following satisfy your needs?

export POETRY_PYPI_TOKEN_PYPI=my-token
export POETRY_HTTP_BASIC_PYPI_USERNAME=username
export POETRY_HTTP_BASIC_PYPI_PASSWORD=password

My main concern is that those commands aren't host-based - i.e. it seems like they're be sent to every PyPi host, including the public one? Or is your Poetry config set up to query your private registry exclusively?

[viceice]

@rarkins We can do it fgor all configured repos:

[viceice]

maybe we need to read the toml

[[tool.poetry.source]]
url = "https://XXXXX/nexus/repository/pypi-central/simple"
name = "my_repository"

so we can match the url to host rule, to get the matching env var names

[janjagusch]

thanks for your lightning fast feedback, @viceice and @rarkins! :zap:

configuring poetry through environment variables seems like a good idea. :+1:
there seems to be an issue with processing environment variables for authentication in poetry, though.

i will see today whether i can make the environment variable configuration work. i will keep you posted.

[viceice]

there is also a workaround for the bug https://github.com/python-poetry/poetry/issues/1871#issuecomment-600816090

[janjagusch]

i think this issue has been resolved in the 1.09 release.

updating seemed to have resolved the issue for me.

[viceice]

Ok, but why the issue is still open?

[janjagusch]

okay, this seems to work for me.

from a clean poetry v1.0.9 installation, have a pyproject.toml like:

[tool.poetry]
# ...

[[tool.poetry.source]]
name = "mysource"
url = "https://mysource.com"

[tool.poetry.dependencies]
python = "^3.7"
mydep= {source="mysource", version="^0.1.0"}

[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"
#

after setting POETRY_HTTP_BASIC_MYSOURCE_USERNAME and POETRY_HTTP_BASIC_MYSOURCE_PASSWORD, running poetry install will succeed.

[janjagusch]

Ok, but why the issue is still open?

I am not sure, I will leave a comment there.

[rarkins]

Are you running self-hosted? And passing env variables through worked?

[janjagusch]

Are you running self-hosted? And passing env variables through worked?

For now I can only confirm that configuring poetry works in a clean python docker container. I cannot say anything yet about how it integrates with renovate.

(But once I test this with renovate, it will be running in a container on my personal machine)

[viceice]

@janjagusch No, we need to pass the env explicitly to docker, so need to change renovate.

You can try to install petry globaly, configure auth env before starting renovate with --trust-level high. This way renovate will pass the current env to sub process.

[janjagusch]

Can confirm that setting the POETRY_HTTP_BASIC_MYSOURCE_USERNAME and POETRY_HTTP_BASIC_MYSOURCE_PASSWORD and setting trust-level="high" solved this issue for me. Thanks, @viceice and @rarkins!

It would still be a great feature, if renovate set those poetry env vars for me, based on the information provided in the hostRules.

What do you think?

[viceice]

Yes we like to support this.

We are always happy to review pr's which solves this or other issues 🙃

[janjagusch]

@viceice, I can try to put something together this weekend.
it will be my first PR in this repo and my first PR with JS in general - so your review will be much appreciated. :)

[janjagusch]

Sorry guys, I tried to add the feature but didn't succeed! :disappointed:

I believe it would be better if someone worked on this who knows JS/TS.

[renovate-release]

:tada: This issue has been resolved in version 22.21.0 :tada:

The release is available on:

• GitHub release
• 22.21.0

Your semantic-release bot :package::rocket: