Poetry: How to handle production only dependencies. psycopg2

[autoferrit]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Question

One issue with poetry, which also is a problem in something like Pipenv, are dependencies that should only be installed on production. In my case, this is psycopg2. When doing development/testing, it is easy enough to install psycopg2-binary under [tool.poetry.dev-dependencies] but the problem is, that the normal dependencies are not "production only". They are "all environemnts". So if I have:

[tool.poetry.dependencies]
psycopg2 = "*"

[tool.poetry.dev-dependencies]
psycopg2-binary = "*"

I still get the following error when installing on my local machine, where postgres is not installed.

$ poetry install
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file


Package operations: 2 installs, 0 updates, 0 removals

  - Installing psycopg2 (2.8.3)

[EnvCommandError]                                                                                                                               
Command ['/home/shawn/.local/share/virtualenvs/spedsa-nMjnt-5Z/bin/python', '-m', 'pip', 'install', '--no-deps', 'psycopg2==2.8.3'] errored with the follow        
ing return code 1, and output:                                                                                                                               
Collecting psycopg2==2.8.3                                                                                                                                   
  Using cached https://files.pythonhosted.org/packages/5c/1c/6997288da181277a0c29bc39a5f9143ff20b8c99f2a7d059cfb55163e165/psycopg2-2.8.3.tar.gz              
    ERROR: Command errored out with exit status 1:                                                                                                           
     command: /home/shawn/.local/share/virtualenvs/spedsa-nMjnt-5Z/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cz2                
8dqk0/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-cz28dqk0/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.rea  
d().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info                          
         cwd: /tmp/pip-install-cz28dqk0/psycopg2/                                                                                                            
    Complete output (23 lines):                                                                                                                              
    running egg_info                                                                                                                                         
    creating pip-egg-info/psycopg2.egg-info                                                                                                                  
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO                                                                                                          
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt                                                                          
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt                                                                                  
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'                                                                                       

    Error: pg_config executable not found.                                                                                                                   

    pg_config is required to build psycopg2 from source.  Please add the directory                                                                           
    containing pg_config to the $PATH or specify the full executable path with the                                                                           
    option:                                                                                                                                                  

        python setup.py build_ext --pg-config /path/to/pg_config build ...                                                                                   

    or with the pg_config option in 'setup.cfg'.                                                                                                             

    If you prefer to avoid building psycopg2 from source, please install the PyPI                                                                            
    'psycopg2-binary' package instead.                                                                                                                       

    For further information please check the 'doc/src/install.rst' file (also at                                                                             
    <http://initd.org/psycopg/docs/install.html>).                                                                                                           

    ----------------------------------------                                                                                                                 
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                              

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

So i can't leave psycopg2 to only be installed on production and the binary version otherwise. How should I handle this? Others have asked to have a prod-dependencies or something similar. But it doesnt seem like that will ever happen. Nor in Pipenv.

How should I make this work? Assuming both environments are on linux.

[abn]

@autoferrit you could make use of extras as a potential solution for your scenario.

poetry add --dev psycopg2-binary
poetry add --extras=production psycopg2

And when installing in your production environment, you can do the following.

# if installing from a package index
pip install somepackge[production]

# if installing from source
poetry install --no-dev --extras=production somepackage

However, this means that if installed without extras your package/application may not work as expected. An operation solution might be better for your particular case of using psycopg2. For example, using a private index (nexus etc.) that has a securely built psycopg2-binary wheel; or you can consider pre-creating a virtual environment and installing psycopg2-binary prior to using poetry install. Hope this helps.

EDIT: Just realised that this might not solve your particular issue as poetry installs optional packages too when installing in editable mode.

[autoferrit]

Yea that doesn't really solve the issue. Really, normal deps would only be installed in production mode. and dev deps would only be installed in development. Dev deps would also have everything in the production deps as well.

[dmontagu]

@autoferrit It's not clear to me -- are you just trying to find a solution that will work, or are you asking for a "clean" solution? Given the lack of support for prod-dependencies, and the fact that extras are installed in development mode, I think the only hope for a truly "clean" approach might be to just require postgres to be installed locally, and just use psycopg2 as a dev depedency.

Some less "clean" approaches that could also work if you are willing to go off the beaten path a little:

• Create a script/docker image for building the psycopg2 wheel, then add it as a path dependency.
• Create a script that manages converting psycopg2<->psycopg2-binary in the pyproject.toml (as a function of the desired dev/prod environment), and include it as part of your CI to ensure the prod wheel always uses the non-binary dependency. (It would also be easy to call manually in development if necessary.)
  
  
  
  • (This would be a one-liner using sed; if that rubs you the wrong way you could write it a little more carefully using the toml python package and it should still be pretty easy.)
  
  
• Don't use poetry/pyproject.toml to manage the psycopg2/psycopg2-binary dependency, and instead just install using pip install (this would mean a built wheel was missing the psycopg2 dependency though)

[autoferrit]

For now as a workaround, I am using the extras method mentioned by @abn above. both are marked optional and locally I do poetry install -E development and in prod poetry install -E production. It works locally. Haven't tried deploying it yet

[justinmayer]

This is essentially a duplicate of https://github.com/sdispater/poetry/issues/1007, and for the sake of maintaining some semblance of sanity in this repository, I humbly recommend that this issue be closed. 😊

[autoferrit]

1007 is about having _more_ environments. I still just want to use the ones i have. As it stands, passing the -E flag in the cli is the ONLY way I can get around this and its not very intuitive and isn't explained the best in the docs if this is what that is meant for. Ideally I should always assume production unless the environment it is installed in has some flag stating otherwise. Like NODE_ENV for node applications.

If I can specify a ENV var to specify which -E flags to use when installing, I would be very happy with that. since then I can still just do poetry install and it would see APP_ENV=development,wsl for example and run it like poetry install -E development -E wsl

[blaggacao]

gomod AFAIK provides a mechanism to define alternate sources or even completely alternate (alias) packages (which however provide the same namespace api).

Here is yet another use case:

[tool.poetry.dependencies]
...
dodoo = "^2.0"  # Use from package mgt

[tool.poetry.dev-dependencies]
...
dodoo = {path = "../dodoo"}

From a UX perspectve, maybe the install --develop=DEVELOP has almost a 90% overlap in semantics with the requested feature. It would be an acceptable restriction to current functionality, if:

• install --develop=DEVELOP always uses the alternate dev dep -- if specified -- _in addition_ to install said dep in development mode

Please no workarounds, this is a genuine and valid use case of it's own!

[autoferrit]

Closing this for now. there are other issues tracking the preferred methods. And for now using the -E flag, while not as intuitive as I had hoped, does work.

[bittner]

I hit the same issue, albeit for a different use case. (Or maybe it's really the same use case, but we describe it differently.) – Here it goes:

For local development I use docker-compose, hence my dependencies are isolated in a Docker image (via the project's Dockerfile). My developer machine is minimal, my system dependencies for psycopg2 (or for mysql-connector, depending on the database engine) are not installed locally, but only in the Dockerfile.

This is a perfect setup for working with tools like pip-compile or pipenv if I want to generate or update my project dependencies "from outside", i.e. running one of those commands on my developer machine locally (to create, say, a requirements.txt file). Those tools resolve the dependencies without requiring me to have runtime dependencies of the packages installed.

Poetry seems to work differently. It fails, like above, because I don't have the database client libraries (and headers) installed. This makes it impossible for me to use Poetry. It doesn't support my usual development workflow. – And it's not an unusual workflow, is it?