Poetry: Support for .env files

[ptink]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

First off I really like poetry- I'd actually only recently migrated over to pipenv but I already see the advantages of poetry and would ideally like to use it full time.

I primarily work with Django and other web frameworks that can require a large amount of configuration which- when following 12 factor app practices- involves lots of environment variables that necessitates the use of a .env file.

Is there a possibility of adding in .env file support, specifically for poetry's run command so that the resulting command runs both in the virtualenv and with the environment as configured in the .env file?

[sdispater]

I don't think this is in the scope of Poetry.

Since Poetry is not configured via environment variables there are no reasons for it to load a .env file.

If the libraries you use need it, it's their responsability to load it not Poetry's.

[jacebrowning]

I've always liked using direnv for this. It supports loading .env files automatically.

[ptink]

@jacebrowning direnv isn't cross platform unfortunately. When the environments I worked in were solely on Linux I used direnv in conjunction with vex to achieve functionality similar to pipenv run <command> for example. This workflow doesn't work on Windows which was a major reason for my discovery and subsequent switch to pipenv when I realised it could do the same thing.

I agree that poetry doesn't need .env files for it's own configuration (although it's not a bad idea to use them to allow for directory-specific configuration that may differ from the globals defined in the config file) but I don't think it's out of scope for poetry to load .env files for run and shell when you consider that they are both attempts to isolate the given command in an environment that is specific to the current project.

I understand the concerns about scope creep, especially when the PR adds a new dependency. If you're exclusively writing libraries then this particular use case may not occur very often but for 12 factor apps it's pretty vital to have tooling to support it- and it always seemed like a neat solution to me to have the virtualenvironment be in control of it.

[AlJohri]

I've used https://github.com/jezdez/envdir/ for something similar.

it uses an envs folder instead of a .env file.

https://github.com/toddbluhm/env-cmd is also good if you want to stick to the .env file

it should be simple to use an alias that chains the two commands together

[ptink]

Again, I'm aware other external libraries exist that do this but I'm of the belief that this particular functionality would be useful to have included in Poetry itself. It's a vital part of tooling required for 12 factor apps and it would be helpful to people who are moving towards adopting an environment variables-based approach in their project to have support for it in the tooling they (ideally) are already using for their project. The goal being to reduce the barrier to entry for managing python projects so that we don't end up back in the "oh just install pip and virtualenv and virtualenvwrapper, easy!" mess that we're finally moving away from :)

If @sdispater would prefer that this isn't included within the project then my next solution would be to modify the entry-point to my Django applications so that they read the required env vars using python-dotenv. That's obviously less ideal but still better than relying on more external libraries that drop in and out of maintanence.

[Pluckerpluck]

I'd like to voice my support of this. The ability to automatically load .env files when running poetry run or poetry shell makes a lot of sense to me. (There's a reason pipenv does it)

Environmental variables are generally designed to be read from the shell (i.e. not a file) and so it makes sense for this to be within the scope of poetry, given that it creates a sub shell for isolation.

In short: Environmental variables should be part of Poetry's virtual environment.

[pmav99]

What's the problem with having direnv or aactivator or whatever handle this? To be honest, I also use them to handle the virtualenv creation/activation but that's purely optional. I prefer those dedicated projects, because I can also use them on non-python projects (e.g. to set up credentials for databases, environmental variables etc).

I can understand that it could be convenient to have this implemented in poetry itself, but it feels like it is kind of out of scope + there are projects that do exactly this thing and IMHO quite well so.

[pawamoy]

Could be a candidate for a plugin once they are a thing.

[Pluckerpluck]

@pmav99 Me being on Windows is really the biggest problem here.

@pawamoy As long as plugins could work with this type of thing (likely, but never guaranteed) I'd be fine with that.

[pmav99]

The fact that setting up an environment on Widows is hard does not answer why setting up the environment becomes poetry's responsibility though.

[Pluckerpluck]

@pmav99 I've already stated why I think setting up the environment in poetry's virtual environment makes sense. Poetry uses virtual environments to provide isolation. That's the point of virtual environments. How does it not makes sense that within that environment, environmental variables could be set?

This is particularly true for the poetry shell command which literally spawns a sub shell for you to access directly. For example, I regularly use poetry shell followed by setting FLASK_ENV to development in order to run my HTTP endpoints locally while debugging.

You simply stated why I couldn't use other pieces of software, and I answered. Windows is not supported by them. Poetry, on the other hand, claims to support Windows.

[ptink]

If the upcoming plugin system has hooks for injecting env vars then I agree that this would be an ideal candidate for a plugin

[ekhaydarov]

oh boy, coming from pipenv, which has an array of issues, poetry seems to have issues which pipenv has solved. why dont they just marry already

[sdispater]

Like I said above this is beyond the scope of Poetry so I am closing this issue.

[marcosfelt]

So, I know this issue is closed, but can anyone give instructions on how to set the environmental variables on a poetry run command?

[orn688]

I use a custom shell function, so I can do withenv <env-file-name> <command-to-run>. This spawns a subshell, sources the .env file into that subshell, and then runs the command in the subshell.

For example, withenv .env poetry run python script.py.

Bash:

withenv () {
  env_file="$1"
  cmd="${@:2}"
  bash -c "source $env_file && $cmd"
}

Fish:

# Source: http://lewandowski.io/2016/10/fish-env/
function posix-source -d "Source a POSIX-formatted .env file"
  for i in (cat $argv)
    set arr (echo $i |tr = \n)
    set -gx $arr[1] $arr[2]
  end
end

function withenv -d "Run a command in a subshell with a .env file sourced"
  set env_file $argv[1]
  set cmd $argv[2..-1]
  fish -c "posix-source $env_file; and $cmd"
end

[Keda87]

For example, withenv .env poetry run python script.py.

Bash:

withenv () {
  env_file="$1"
  cmd="${@:2}"
  bash -c "source $env_file && $cmd"
}

Fish:

# Source: http://lewandowski.io/2016/10/fish-env/
function posix-source -d "Source a POSIX-formatted .env file"
  for i in (cat $argv)
    set arr (echo $i |tr = \n)
    set -gx $arr[1] $arr[2]
  end
end

function withenv -d "Run a command in a subshell with a .env file sourced"
  set env_file $argv[1]
  set cmd $argv[2..-1]
  fish -c "posix-source $env_file; and $cmd"
end

Hi @orn688 is it compatible with zsh ?

[orn688]

Hi @orn688 is it compatible with zsh ?

It should be, just substitute zsh -c for bash -c from the bash version.

[golyalpha]

I feel like the community is in strong disagreement about this issue...

[tumido]

I'll also voice my meaningless disagreement with this decision. I came to poetry from pipenv because I wanted to simplify the toolkit I use - to spend less time maintaining and more time creating. This issue goes against the flow and poetry actually creates a problem that was already solved with pipenv.

To paraphrase @sdispater:
Now we can't have single tool to manage my Python projects from start to finish. You wanted something reliable and intuitive that the community could use and enjoy. Sadly I'm not enjoying this anymore. And it's not intuitive anymore.

This issue was not closed because of a technical reason.

[rajasimon]

Coming from pipenv and uninstall poetry immediately after realizing it doesn't support .env 👋

[sdispater]

@tumido

This issue was not closed because of a technical reason.

You're right. The reason for closing it was not technical but because this feature does not align with the vision I have for Poetry. Believe me, I try to analyze every feature request made here to see if it matches the direction I want the project to go in and, if not, I reject the idea. It's as simple as that.

I understand that this feature could be useful for some people, I do, however it's not the purpose of a dependency and package manager to manage the .env file and environment variables.

That being said, my idea has always been for Poetry to be extensible that's why I am working on a plugin system so that people can tailor it to there needs if they feel like what Poetry is doing by default is not enough.

@rajasimon Sorry to see you go! I hope you'll change your mind in the future.

[flyte]

I am working on a plugin system so that people can tailor it to there needs

This would be great, thank you! I'm also of the opinion that if you're offering automated shell creation with poetry shell and poetry run then it makes sense to be able to set some environment vars in the shell you're creating. Of course I'll be happy with a plugin to do it though.

[mludvig]

however it's not the purpose of a dependency and package manager to manage the .env file and environment variables.

No one expects Poetry to manage the .env file. What we are calling for is a simple hook in poetry shell and poetry run that will source the .env file if it exists. Simple as that.

Having a vision is indeed a noble thing but also I assume that you are developing Poetry for people. When so many of us have a need for a particular feature there may be perhaps a genuine, logical reason?

[gormster]

@sdispater Were poetry shell, poetry run and poetry env part of your vision? If so, surely you understood that if you are taking on the responsibility of managing a virtual environment, you're also taking on the responsibility of modifying that environment?

[sdispater]

@gormster I am failing to see your point:

• poetry env is a command to create, delete and get info on virtual environments. This has nothing to do with environment variables.
• poetry run is a command to execute a command inside a virtual environment. This has nothing to do with environment variables.
• poetry shell is a commodity command to activate a virtual environment managed by Poetry. This has nothing to do with environment variables.

So, to be clear, I don't understand what virtual environments have to do with environment variables.

And to add to that, none of the package/dependency managers out there that I know of (beside pipenv, but then again Poetry won't replicate every feature of pipenv) do this automatic reading of environment variables because it's just not their role.

So, let me ask you a question: before Poetry (and pipenv) how did you manage your environment variables?

I will also reiterate what I said above:

If the libraries you use need it, it's their responsibility to load it not Poetry's.

If you want to go the 12-factor apps route, it's your responsibility to make your application load and use the environment variables it needs. Poetry has nothing to do with that.

[rajasimon]

When I'm deploying the application I don't work with Virtualenv so simply pip install -r will do the work. And I will add environment variable in systemd control.

But in local development I'm using this simple single line to update the environment before doning python app.py.

# This was before pipenv era
export $(grep -v '^#' .env | xargs -0)

[brianfinkelstein]

While working with one of my super clever coworkers this week, I happily learned how to work around this "issue" with very minimal overhead.

I decided to try and share my education with the community by putting together a quick boilerplate repo.

It is a really simple flask app setup with poetry and python-dotenv. (Be sure to check out the Justfile!)

[kmmbvnr]

@brianfinkelstein so you sugges to use just (https://github.com/casey/just#environment-variables-1) to read .env file

It's cool, but it's still 3d party program with the custom installation procedure on every os. It's not even in Ubuntu repositories :(

[miguendes]

For those interested in a solution that doesn't involve third party tools, on my zsh I overrode poetry to behave like pipenv.

function poetry() {
    # if POETRY_DONT_LOAD_ENV is *not* set, then load .env if it exists
    if [[ -z "$POETRY_DONT_LOAD_ENV" && -f .env ]]; then
        echo 'Loading .env environment variables…'
        export $(grep -v '^#' .env | tr -d ' ' | xargs)
        command poetry "$@"
        unset $(grep -v '^#' .env | sed -E 's/(.*)=.*/\1/' | xargs)
    else
        command poetry "$@"
    fi
}

It will load your .env ignoring comments and allowing spaces like MY_VAR = 3. After running poetry it unset them.

You can add it to your ~/.zshrc or ~/.bashrc and then source it: . ~/.zshrc

It even works for cases where you need to pass a shell var that is not in the .env:

➜ cat .env                                                               
ENV_VAR1 = 23
#LCTIME=en_UK.UTF-8
ENV_VAR2=hello

```bash
➜ ENV_VAR3=45 poetry run python test_env.py
Loading .env environment variables…
ENV_VAR3: 45
ENV_VAR1: 23
LCTIME: pt_BR.UTF-8
ENV_VAR2: hello

```bash
POETRY_DONT_LOAD_ENV=1 poetry run python test_env.py
ENV_VAR3:  None
ENV_VAR1:  None
LCTIME:  pt_BR.UTF-8
ENV_VAR2:  None

Edit: if there's no .env or POETRY_DONT_LOAD_ENV is set then just run poetry

[brianfinkelstein]

@brianfinkelstein so you sugges to use just (https://github.com/casey/just#environment-variables-1) to read .env file

It's cool, but it's still 3d party program with the custom installation procedure on every os. It's not even in Ubuntu repositories :(

@kmmbvnr the code I shared does not actually depend on Just in any way (unless you want to use Justfile for shortcuts on the command line while developing). I did not actually taking advantage of the tool's environment variable support in any way...(AFAIK)

That being said, it does depend on Flask and python-dotenv

You are welcome to type out poetry run flask run in your terminal and the .env file will be loaded just the same 😅

[mrchrisadams]

Hey folks, I found myself missing the handling of .env files when I came to Poetry, and this is the approach I've settled on for now.

Rather than relying on Poetry to include the handling of .env files as part of it's functionality, if you're only using python scripts, it turns out not to be too much work to use python-dotenv to wrap calls to your original binary with python dotenv's CLI commands:

You add python-dotenv as a dependency, and then instead of call your original file like so:

poetry run python ./example.py

You can run it like so:

poetry run dotenv run python ./example.py

I've put together a sample project demonstrating this in use, with further examples for using different .env files, and changing env vars without needing to reload the virtual environment each time.

https://github.com/mrchrisadams/poetry-with-dotenvs

If you have a project already using Poetry, I think in many cases, adapting a project can be as easy as calling poetry add "python-dotenv[cli]", to add the necessary python dependency, and then adding dotenv runbefore the command you want to run.

Hope this helps - it might seem obvious and I know the original poster, @ptink alludes to this in his comments. The other examples in the comments seem to assume the existence of other languages (env cmd uses node), or control over a specific shell on a server, or even the file structure, in the case of things like direnv, whereas the approach I've outlined at least stays in Python, and doesn't many more assumptions, than 'you can install python packages into the virtualenv'.

Note: there was some mention of plugin hooks before, and the code to support .env files seems smaller enough to add as a plugin system.

To anyone familiar the the plans for the plugin hooks for Poetry approach: I'd be up for working with you to implement this as a plugin.

[abn]

A tool independent option might be to use direnv.

[Vozf]

Quite strange to see the issue closed when obviously community wants this feature and one person decides that "it is not in scope"

[golyalpha]

@Vozf I see your point, and am on your side, but @sdispater is the maintainer and creator of the project, so, for better or worse, it's up to him.

[tumido]

Hey! Just one more reason to impatiently wait for or (!) help out with the plugin system. :smile: https://github.com/python-poetry/poetry/pull/1237 That will be finally a truly community-driven functionality. :slightly_smiling_face:

[kajuberdut]

First of all, thank you @sdispater for creating this free and open tool.

However, I too took an evening to convert from Pipenv to Poetry and am going to discard these changes without committing and roll back to Pipenv.

There just doesn't seem to be a good tool to make this happen on both Windows and Linux like Pipenv does and it's entirely too useful to live without.

I would also add that besides Pipenv there are many other common tools similar to Poetry that do handle this for users. The ubiquitous Webpack, also the new Vite by the wonderful Evan You (Author of Vue.js), Buildozer from the Kivy project.

[golyalpha]

Hey! Just one more reason to impatiently wait for or (!) help out with the plugin system. :smile: https://github.com/python-poetry/poetry/pull/1237 That will be finally a truly community-driven functionality. :slightly_smiling_face:

Plugins are cool, but shouldn't be an excuse to not implement something the community wants into the core tool. Plugins are meant to serve a niche that wants a feature, that no one else might use.

[jonfreedman]

I also feel this should be supported out of the box, if a 3rd party library decides to use environment variables there's not a lot that can be done about it, and the "hello world" examples of poetry run pytest end up doing unexpected things - personally I don't expect to set an environment variable, run a command and then have that environment variable ignored.

[ekhaydarov]

the strange thing is its in the name. poetry creates a new virtual _environment_. Part of that new env should be _environment variables_. its in the name. it should not be too difficult to implement this either. shame the maintainer does not want to support such a simple yet complementary feature

[Vozf]

@ekhaydarov This was already implemented and pr was declined by maintainer https://github.com/python-poetry/poetry/pull/339

[ekhaydarov]

maybe we can just fork and set up a ci to auto pull from upstream just to have this feature enabled? to be honest tools like this dont need to be updated often at all. a good server host is only required for production workflows where ci needs to download this package.

[Vozf]

This feature should be available with custom plugin when plugin system is finished and that is expected for 1.2. So for me it's easier to wait.

[golyalpha]

@Vozf My issue is that this feature was already implemented, and the only reason it's not implemented as of yet in Poetry itself is because @sdispater doesn't want it.

[adonig]

If you want this feature for django, you can use django-environ. Otherwise you can also use direnv.

EDIT: Oh and python-dotenv looks really nice.

[kajuberdut]

If you want this feature for django, you can use django-environ. Otherwise you can also use direnv.

These, along with all of the other suggestions that I have seen here do not apply to the 50-80% of computer users who are on windows (https://en.wikipedia.org/wiki/Usage_share_of_operating_systems#Desktop_and_laptop_computers). From my research the only established library that makes loading .env painless and automatic for Windows users of Python is Pipenv.