Poetry: Add support for environment variables at pyproject.toml

[loop0]

It would be handy to have support for using values from environment vars on pyproject.toml
The use case I have is that I need to use different repository urls depending on my environment.
Something like:

[[tool.poetry.source]]                                                          
name = "private"                                                          
url = "${PRIVATE_REPO_URL}"

I'm still new to poetry, but I could help implementing the feature with some guidance.

[jordanhamill]

I too would like to not check in a private repo url. Perhaps in this case, the value from config.toml could be used for the repository?

The docs already mention adding a repository url for pushing. If url was missing from source you could check for the following key:

f"repositories.{source.name}.url"

I pushed up a spike 290e8863da8b8596d82f60484dcc08534abb3b3d to see if it would work and it looks like it does, I'm not sure if it is the correct solution though.

My suggestion becomes a little cumbersome when building docker images. Currently I'm doing everything on one line so nothing sensitive is cached in a layer:

ARG PYPI_USERNAME
ARG PYPI_PASSWORD
RUN poetry config http-basic.my-repo ${PYPI_USERNAME} ${PYPI_PASSWORD} && poetry install ${POETRY_INSTALL_OPTIONS} && poetry config --unset http-basic.my-repo

[kofron]

@loop0 did you take a stab at this? I'd love to see this implemented and have some ideas around how to do it.

[loop0]

Not yet, I was planning to tackle this next week as I am starting to use poetry again. Do you mind sharing your ideas?

[kofron]

Awesome, great to hear - let me know if I can help at all.

Basically what I was imagining is a new function (or class) that lives in poetry.util.helpers called environment_expander or similar.

The basic idea is that after pyproject.toml is read from disk, and after we know that it exists, we take a transformation pass over the TomlFile object, and applying a function to all of the values in the object that evaluate expressions of the form ${SOME_ENV_VARIABLE} by calling os.env.

I would want to apply this pass as early as often - at poetry/puzzle/provider.py:161 for example.

Not sure if you would want to raise if SOME_ENV_VARIABLE didn't exist...

Thoughts?

[loop0]

Sounds a great idea! And of course you can help, I will try to make some time tomorrow to write the first code for it. If you feel you want to write it first just mark me on the PR and I can also help you later.

[dimkirt]

Any update on this one?

[kofron]

@dimkirt https://github.com/sdispater/poetry/pull/481 - was closed a couple of days ago by the author, who doesn't want to add this feature.

[Peque]

Pipenv implements this feature. As they mention, it is "quite useful if you need to authenticate to a private PyPI". That is my case too.

@sdispater Being ranked as the 11th most popular issue in this repository (out of 400+), would you reconsider accepting pull-requests for this? :innocent:

[metrofun]

For us it's also a blocker to finally migrate from Pipenv

[loop0]

We need this feature to migrate to poetry from pipenv as well, that's why I initially open this issue. Unfortunately I didn't have much time to work in the solution, but as far as I remember the code, it shouldn't be hard. We only need the bless from poetry's developers.

[kofron]

I did actually do this work in this PR: https://github.com/sdispater/poetry/pull/481

However, @sdispater did not want to add this feature to the project.

[Peque]

Let us hope they eventually change their mind and reconsider adding this to Poetry. :innocent:

[absassi]

Although this solution would solve the problem, and it's the same solution as Pipenv, it's only good for CI, but terrible for humans. Once you put environment variable names in pyproject.toml, every developer must somehow export the variables to use Poetry, they can't just use poetry config http-basic anymore.

It would be much better if Poetry recognized specific environment variables as credentials (such as POETRY_USERNAME and POETRY_PASSWORD or maybe POETRY_REPONAME_USERNAME, to support different credentials for different repositories), like twine does, for instance. Then the environment variables are needed only for CI and humans can do whatever they want in their machines.

But this is not something you can do with an external program, it's something that needs to be implemented in Poetry.

[delphyne]

@absassi I disagree. Any large enterprise already has a set of consistent environment variables every developer must export. These are used across many languages and toolsets (pipenv, cake, gradle, maven, ant, node, yarn, docker, terraform, etc.). Adding a new set of environment variables is just as bad as using whatever people want.

Maybe just compromise and do both. look for some poetry-specific variable, but also allow environment variables to be interpolated like just about every other build tool that exists today.

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[Peque]

Still relevant.

Maybe @loop0 has some updates on this? :blush:

[loop0]

No updates, sorry. We still want to migrate from pipenv to poetry here, but this issue is a blocker for most projects.

[loop0]

For instance, there is a way to use poetry with gemfury's repository, but the documentation is not clear, here are the steps:

poetry config repositories.your_repo https://pypi.fury.io/your_repo/
# at the auth configuration you just repeat your token twice
poetry config http-basic.your_repo YOUR_TOKEN YOUR_TOKEN

And then you add the following to your pyproject.toml (after the [tool.poetry] block)

[[tool.poetry.source]]
name = "your_repo"
url = "https://pypi.fury.io/your_repo/"

So with some scripting it is possible to have this running on your CI

[adawalli]

For instance, there is a way to use poetry with gemfury's repository, but the documentation is not clear, here are the steps:

poetry config repositories.your_repo https://pypi.fury.io/your_repo/
# at the auth configuration you just repeat your token twice
poetry config http-basic.your_repo YOUR_TOKEN YOUR_TOKEN

And then you add the following to your pyproject.toml (after the [tool.poetry] block)

[[tool.poetry.source]]
name = "your_repo"
url = "https://pypi.fury.io/your_repo/"

So with some scripting it is possible to have this running on your CI

While this is true, if you are using docker, you run the risk of caching these credentials, which is obviously not good. As it stands, it's not really possible (without jumping through some serious hoops) to write a 12 Factor App (https://12factor.net/) with Poetry + Private Repos without better Environment Variable support.

[sandroden]

I want o add another reason for using variable interpolation in pyproject.toml. When you add a dependency on locally stored projects as in:

[tool.poetry.dev-dependencies]
thx-sphinx = {path = "../thx-sphinx"}

you're making strong assumptions on where people place their files that is definitely not a nice behaviour. Much better to set a default that users can overwrite. A .env is a typical solution in docker environment and not only (ie: autoenv). When you enter a directory the .env is read and configured correctly. This would make poetry flexible enought for normal workflow.

[papagala]

This would be incredibly convenient for CI/CD pipelines and for reusing already available private repository urls in the form of environment variables.

[absassi]

@sandroden, depending on what you want to accomplish, you might be interested also in the feature request #1168, which can be an alternative to path dependencies.

[bubthegreat]

@sdispater - Any reconsideration here? This is difficult when I want to test anything that needs environment variables but I can't set and unset them without editing the virtualenv activate file directly, which is a huge hassle. It makes it hard to test in multiple packages or environments that might need environment variables set, like DB_HOST, etc.

[funkyfuture]

the system administrator in me likes that. but apart from it, it would lead to a situation where poetry.lock wouldn't be reproducible without sharing all environment variables. and then, what would be the point?

besides, i'm missing defaults being mentioned in this thread for which Docker-Compose has a good syntax, as an example.

[bubthegreat]

I'm doing my own workarounds by having an .env file and pulling them from there, but my ideal world would be setting them locally in the virtualenv and adding them as checks in poetry.lock with a warning if they're not set.

[jacebrowning]

Assuming the following environment variables are set:

export EXAMPLE_PYPI_URL=https://pypi.example.com/pypi/
export EXAMPLE_PYPI_USERNAME=example
export EXAMPLE_PYPI_PASSWORD=example

This is how I handle uploads to a private PyPI server:

poetry build
poetry config repositories.example ${EXAMPLE_PYPI_URL}
poetry config http-basic.example ${EXAMPLE_PYPI_USERNAME} ${EXAMPLE_PYPI_PASSWORD}
poetry publish --repository example

I can see why adding environment variable expansion to pyproject.toml could be too much of a scope creep for Poetry.

[SpicySyntax]

What should we do in the case that the url for the private repo contains private username and password? I.E.

[[tool.poetry.source]]                                                          
name = "private"                                                          
url = "https://${ARTIFACTORY_USER}:${ARTIFACTORY_APIKEY}@artifactory.corperation.com/artifactory/api/pypi/pip-local/simple"

[yousefissa]

+1

What should we do in the case that the url for the private repo contains private username and password? I.E.

[[tool.poetry.source]]                                                          
name = "private"                                                          
url = "https://${ARTIFACTORY_USER}:${ARTIFACTORY_APIKEY}@artifactory.corperation.com/artifactory/api/pypi/pip-local/simple"

+1 on this. Any tips for poetry with private user/pass?

[gghildyal]

With AWS codeartifact repository, the authentication token is a part of the URL, so this becomes even more important to get a smooth CI experience. Currently I am having to add git commits every time the token expires which is every 12 hours :(

[pawel-ch]

What should we do in the case that the url for the private repo contains private username and password? I.E.

[[tool.poetry.source]]                                                          
name = "private"                                                          
url = "https://${ARTIFACTORY_USER}:${ARTIFACTORY_APIKEY}@artifactory.corperation.com/artifactory/api/pypi/pip-local/simple"

Can't do poetry update with url set like this and depend on packages from our repository in local development. And yes – we would like CI to work too.

[absassi]

@SpicySyntax , @yousefissa , @pawel-ch , see comment above from @loop0 on how this can be scripted:

https://github.com/python-poetry/poetry/issues/208#issuecomment-553480601

[pawel-ch]

@absassi ~That doesn't solve the problem of storing credentials from URL in repository (both pyproject.toml and poetry.lock). Ie. for GitLab it looks like this in pyproject.toml~ (not true, see edit below):

[[tool.poetry.source]]
name = "gitlab"
url = "https://__token__:[email protected]/api/v4/projects/42/packages/pypi/simple"
secondary = true

And like this in poetry.lock:

[[package]]
name = "my-super-package"
version = "0.1.1"
description = ""
category = "main"
optional = false
python-versions = ">=3.8,<4.0"

[package.dependencies]
dj-database-url = ">=0.5.0,<0.6.0"
Django = ">=3.0.0,<3.1.0"
djangorestframework = ">=3.11.0,<4.0.0"

[package.source]
type = "legacy"
url = "https://__token__:[email protected]/api/v4/projects/42/packages/pypi/simple"
reference = "gitlab"

Edit: OK, tried to change url to https://gitlab.com/api/v4/projects/42/packages/pypi/simple and then storing credentials via poetry config and it looks like it worked.

[abn]

@pawel-ch you can also use the environment variables POETRY_HTTP_BASIC_GITLAB_USERNAME and POETRY_HTTP_BASIC_GITLAB_PASSWORD if you want to use environment variables instead of the config.

[GuillaumeLegoy]

With AWS codeartifact repository, the authentication token is a part of the URL, so this becomes even more important to get a smooth CI experience. Currently I am having to add git commits every time the token expires which is every 12 hours :(

Hello @gghildyal and others. I'm working with AWS CodeArtifact and managed to make it work after struggling a little bit, here is how:

1. In Poetry configure your private repository link as follow (note that in AWS CodeArtifact the below link is available when you click "Connection Instructions" and choose "pip"). Make sure to remove the aws:$CODEARTIFACT_AUTH_TOKEN part before the @:

poetry config repositories.<your_private_repository_name> https://<your_private_aws_repository_account>.d.codeartifact.<your_private_aws_repository_region>.amazonaws.com/pypi/<your_private_package_name>/simple/

1. Export the $CODEARTIFACT_AUTH_TOKEN environment variable. Again AWS CA gives us the command when we check the instructions, as follow:

export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain <your_private_repository_domain> --domain-owner <your_domain_id> --query authorizationToken --output text`

1. Configure the authentication in Poetry for the repository we configured in step 1 (make sure to use the same repo name). Here you can see the aws and $CODEARTIFACT_AUTH_TOKEN constitutes the part we removed from the AWS CA link before the @:

poetry config http-basic.<your_private_repository_name> aws $CODEARTIFACT_AUTH_TOKEN

1. Add your private repository to pyproject.toml to be able to install your custom packages (again use the same name we setup in 1.):

[[tool.poetry.source]]
name = "<your_private_repository_name>"
url = "https://<your_private_aws_repository_account>.d.codeartifact.<your_private_aws_repository_region>.amazonaws.com/pypi/<your_private_package_name>/simple/"
secondary = true

1. You can now run poetry add <your_custom_package> in your project and it should work.

[gghildyal]

@GuillaumeLegoy Thanks for the writeup! This is exactly what I did for local non-dockerized development. Getting AWS credentials to run aws codeartifact to get the token inside of a docker container at image build time is a bit trickier though.