Poetry: Any plans to make it work with anaconda python?

[tkossak]

Hi,
Anaconda doesn't have ensurepip so I get error as below. Are there any plans to make it work with anaconda?
I would install it under different python, but I saw a thread it doesn't work (yet) with pyenv, which I use to manage my python versions.

Thank you.

> python -VV                                                                                                                            
Python 3.6.2 | packaged by conda-forge | (default, Jul 23 2017, 22:59:30) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

> python get-poetry.py                                                                                                                  
Retrieving metadata

Installing version: 0.10.3
  - Getting dependencies
  - Vendorizing dependencies
  - Installing poetry

poetry (0.10.3) successfully installed!

> poetry new test                                                                                                                       
Created package test in test

> cd test                                                                                                                               
> poetry install                                                                                                                 
Creating virtualenv test-py3.6 in /home/kossak/.cache/pypoetry/virtualenvs

[CalledProcessError]                                                                                                                           
Command '['/home/kossak/.cache/pypoetry/virtualenvs/test-py3.6/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit stat  
us 127.                                                                                                                                                     

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

> python -m ensurepip
/home/kossak/anaconda3/bin/python: No module named ensurepip

[sdispater]

To be honest I am not familiar with anaconda, so it's not high on the priority list right now. However, if someone that have more knowledge than me on the matter wants to give it a try and make a PR I would gladly review it.

but I saw a thread it doesn't work (yet) with pyenv

Where did you see that? I can confirm that it works fine with pyenv, at least on my end.

[tkossak]

Where did you see that? I can confirm that it works fine with pyenv, at least on my end.

here on github: https://github.com/sdispater/poetry/issues/172

In that thread I posted what doesn't work for me.

[kiwi0fruit]

@sdispater Anconda uses another package manager - conda. And also can install pip packages via pip on top of conda packages in a compatible way. I guess if poetry would provide a unified interface for both pip based and conda based environments it would be a "+1" consideration to use it instead of pipenv.

Conda already has a nice dependency resolver. Pip that is installed on top - does not.

I've investigated the abilities of conda to reliably reproduce the installation - and it's not there yet. But I foresee that it's not that complex to add. I've collected issues about it here: https://github.com/kiwi0fruit/misc/issues/4

[kiwi0fruit]

By the way. @sdispater I guess if at some point you would be interested in adding conda support to poetry you can talk with @mingwandroid It might help.

[mingwandroid]

I'm not sure if poetry targets the same use cases as conda, isn't poetry a Python-only thing?

[kiwi0fruit]

Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right stack everywhere.

So if you need to make pinned snapshots of packages for some reason and want an easy way to keep installation updated then poetry helps.

This task can be with any package manager. And I thought that it might be a good idea to have a unified interface for both pip and conda so it would decrease python package managers fragmentation - in a way :)

[shadowrylander]

Er, quick note: I've used ensurepip with Anaconda (miniconda3) before, both within an environment, as well as (I think) in the base environment. Maybe via command as python -m ensurepip?

[gagarine]

Anaconda is a distribution of software for datascience and conda is the package manager of this distribution. Conda can install different kind of package, not only python package. For example you can install R with conda: https://anaconda.org/r/r . Conda is closer to brew than pip.

I don't think poetry should provide an unified way to install pip and conda package: conda is already a unified way to install python (pip) package and other kind of lib packaged as "conda package".

Python doesn't have a good package manager like npm or composer, so let's first fix that.

[mingwandroid]

I agree that poetry should not get involved with conda packages, unless it was going to deal exclusively with them), but for additional reasons.

Python doesn't have a good package manager like npm or composer, so let's first fix that.

Software languages do not necessarily need their own package managers and in some regards they can be harmful; software ecosystems need package managers. The harm I speak of is that you cannot share code across languages and end up with massive redundancy in various dimensions (human effort, memory usage and bug surface area being the most important ones).

Fix software ecosystem packaging and by definition you fix language packaging for nearly all of them, including some not yet invented. Yes there can be rough edges but given that in general Linux package managers are also whole-ecosystem ones there's lots of momentum for such things to "just work".

This is a large part of the value proposition of conda (the others being reproducability and multiple environments).

I would love for people to think beyond their languages silos more and actually contribute to conda, conda-build and conda-forge but I appreciate that since I work for Anaconda, Inc. I may be considered biased.

[gagarine]

software ecosystems need package managers

I 100% agree with that: https://twitter.com/gagarine/status/895388059327508481

[goldslope]

I have made a tool on behalf of Eigen Technologies that integrates poetry into conda environments.

It sidesteps the virtualenv and installs the poetry packages directly into the conda environment.

We are using it to create, package, and deploy environments to clients. Any non-binary pip packages are compiled with conda's cross-compilers for maximum compatibility across OS versions/distros.

You don't even need a conda environment file, its optional; using this tool you can zip up Poetry environments into a portable cross-distro/cross-version tarball

It works for Linux and macOS only.

https://github.com/eigentechnologies/cpip

Check it out! Let me know if you have any suggestions.

[mingwandroid]

Let me know if you have any suggestions.

How does this compare to conda-pack? I know that does work on Windows at least and is maintained by an Anaconda, Inc. employee.

[goldslope]

it uses conda-pack underneath, sorry for not clarifying

2 things keeping it from being used with Windows:

• lack of Anaconda cross-compilers for Windows
• I wrote it in Bash :(

It handle cross-compiling and setups of the environment and uses conda-pack at the end to package the environment.

The script is somewhat big in comparison, but the fundamental steps are quite simple.

[kiwi0fruit]

lack of Anaconda cross-compilers for Windows
I wrote it in Bash :(

I guess just Windows only compiling on Windows _may_ be enough. Does Anaconda cross-compilers on Unix are capable of compiling for Windows? If not then if cpip compiles on Windows just for Windows and on Unix for all Unix -- it may be good enough.

If you didn't use something special using Git Bash on Windows _may_ be a good option (but there are high chances that something will not work - then Msys2 has a higher chances to work).

[goldslope]

Based on what it says here,

https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html

compiler’s “build platform” is the platform where the compiler runs and builds the code.

A compiler’s “host platform” is the platform where the built code will finally be hosted and run.

Notice that all of these package names end in a platform identifier which specifies the host platform. All compiler packages are specific to both the build platform and the host platform.

Im not sure the Unix cross-compilers will work for Windows.

I don't have Windows available to me, and haven't developed on Windows in a long time, so I really am out of my wheelhouse here.

As for the Bash thing, I think really the ultimate fix, is if necessary to migrate the code to Python, something I just don't have the time for now.

[leycec]

As this issue now appears to host the collective discussion for conda + poetry integration, I have a large-scale query for the resident wizards in the room: is conda-forge known to play nicely with poetry?

Like all involved, I have an abiding hatred for the festering cesspit that is setuptools + pip + pipenv. Like all involved, I also like things that work – and conda-forge feedstocks work out-of-the-box with that cesspit. I'm concerned that transitioning wholesale to poetry will compromise our ability to deliver timely updates to conda-forge feedstocks... or perhaps our ability to develop conda-forge feedstocks at all. There probably be undocumented dragons here that will burn us all horrifyingly, aren't there?

Are there any existing conda-forge feedstocks leveraging a pure-poetry installation pathway (i.e., defining only a single top-level pyproject.toml file)? My dim suspicion is "Nope." Still, my hatred is sufficiently abiding that we'll probably take the poetry plunge for our next large-scale project, cross many fingers, chew much bubblegum, and dream wildly of sane tooling in a galaxy far, far away.

[leycec]

It must also be stated that cpip looks like super hotness. Bravo, @goldslope. :snowman:

[goldslope]

@leycec thanks! Let me know if you have any issues using it. I will be glad to help.

[brycedrennan]

As @gagarine pointed out, conda installs more than just python packages Additionally packages in conda and pypi may have the same name but be different things entirely. Trying to make this work would introduce an immense amount of new complexity.

As such I'd vote to close this issue as wontfix

[stephsamson]

I agree with @brycedrennan and will close this issue.

If people insist on using conda to install Python packages and manage their virtualenvs but poetry as a package management tool, then #1432 should take care of this for you.

[dojeda]

For anyone interested: I wrote a small script to convert poetry to conda environments: https://github.com/dojeda/poetry2conda

[mingwandroid]

@dojeda, the naming issue you mention with conda packages is because we cover more than what's available on pypi so we have namespace clashes not present on pypi.

[mingwandroid]

(i.e. other languages)

[dojeda]

@mingwandroid I see... and I understand why name clashes are unavoidable in that case. Perhaps a dependency level by language could have solve that problem... but that's a discussion out of the scope of this thread...