Poetry: enum34 tripping up Python 3.7.3

[dycw]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

enum34 has long caused packaging problems, for example:

• poetry/#881 regarding tox,
• poetry/#844 regarding netmiko,
• poetry/#1049 regarding numba,
• tensorflow/#15136 regarding tensorflow.

(Gosh, even a compatibility library solely for it exists.)

My issue is with luigi. Relevant history I can find is:

• luigi/#2603 the use of environment markers are acknowledged,
• luigi/#2623 ultimately reversed in this commit.

Now I have two machines, with not-identical but out-of-my-control settings:

>>> uname -r
3.10.0-862.el7.x86_64

>>> poetry debug:info

Poetry
======

 * Version: 0.12.14
 * Python:  3.7.2


Virtualenv
==========

 * Python:         3.7.2
 * Implementation: CPython
 * Path:           /home/wander/test_enum34/.venv
 * Valid:          True


System
======

 * Platform: linux
 * OS:       posix
 * Python:   /export/Python-3.7.2

and machine 2:

>>> uname -r
3.10.0-862.3.3.el7.x86_64

>>> poetry debug:info

Poetry
======

 * Version: 0.12.16
 * Python:  3.7.3


Virtualenv
==========

 * Python:         3.7.3
 * Implementation: CPython
 * Path:           /home/wander/test_enum34/.venv
 * Valid:          True


System
======

 * Platform: linux
 * OS:       posix
 * Python:   /export/Python-3.7.3

On the first machine, I can create a project which explicitly installs enum34 (as luigi does) and then a second one which depends on it:

#!/usr/bin/env bash

cd "$(dirname "${BASH_SOURCE[0]}")"
rm -rf project1 project2

poetry new project1 --name=project1 --no-interaction
cd project1/
sed -i 's/pytest = "^3.0"//g' pyproject.toml
poetry add enum34
cd ..

poetry new project2 --name=project2 --no-interaction
cd project2/
sed -i 's/pytest = "^3.0"//g' pyproject.toml
poetry add project1 --path ../project1 -vvv

Running on the first machine:

Created package project1 in project1
Creating virtualenv project1-py3.7 in /home/wander/test_enum34/project1/.venv
Using version ^1.1 for enum34

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file


Package operations: 1 install, 0 updates, 0 removals

  - Installing enum34 (1.1.6)
Created package project2 in project2
Creating virtualenv project2-py3.7 in /home/wander/test_enum34/project2/.venv
Using virtualenv: /home/wander/test_enum34/project2/.venv

Updating dependencies
Resolving dependencies...
   1: fact: project2 is 0.1.0
   1: derived: project2
   1: fact: project2 depends on project1 (*)
   1: selecting project2 (0.1.0)
   1: derived: project1 (*)
   1: fact: project1 (0.1.0) depends on enum34 (^1.1)
   1: selecting project1 (0.1.0 ../project1)
   1: derived: enum34 (^1.1)
PyPI: 7 packages found for enum34 >=1.1,<2.0
   1: selecting enum34 (1.1.6)
   1: Version solving took 0.036 seconds.
   1: Tried 1 solutions.

Writing lock file


Package operations: 2 installs, 0 updates, 0 removals

  - Installing enum34 (1.1.6)
  - Installing project1 (0.1.0 ../project1)

...no problem. Running on the second machine:

Created package project1 in project1
Creating virtualenv project1-py3.7 in /local/data/export/test_enum34/project1/.venv
Using version ^1.1 for enum34

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file


Package operations: 1 install, 0 updates, 0 removals

  - Installing enum34 (1.1.6)
Created package project2 in project2
Creating virtualenv project2-py3.7 in /local/data/export/test_enum34/project2/.venv
Using virtualenv: /local/data/export/test_enum34/project2/.venv

Updating dependencies
Resolving dependencies...
   1: fact: project2 is 0.1.0
   1: derived: project2
   1: fact: project2 depends on project1 (*)
   1: selecting project2 (0.1.0)
   1: derived: project1 (*)
   1: fact: project1 (0.1.0) depends on enum34 (^1.1)
   1: selecting project1 (0.1.0 ../project1)
   1: derived: enum34 (^1.1)
PyPI: 7 packages found for enum34 >=1.1,<2.0
   1: selecting enum34 (1.1.6)
   1: Version solving took 0.020 seconds.
   1: Tried 1 solutions.

Writing lock file


Package operations: 2 installs, 0 updates, 0 removals

  - Installing enum34 (1.1.6)
  - Installing project1 (0.1.0 ../project1)

  [EnvCommandError]
Command ['/local/data/export/test_enum34/project2/.venv/bin/python', '-m', 'pip', 'install', '--no-deps', '-U', '-e', '/local/data/export/test_enum34/project1'] errored with the following output:

Obtaining file:///local/data/export/test_enum34/project1
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /local/data/export/test_enum34/project2/.venv/bin/python /local/data/export/test_enum34/project2/.venv/lib/python3.7/site-packages/pip install --ignore-installed --no-user -
-prefix /tmp/pip-build-env-gi5ep75t/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12:
  Traceback (most recent call last):
    File "/local/data/export/longhor/Python-3.7.3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/local/data/export/longhor/Python-3.7.3/lib/python3.7/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/local/data/export/test_enum34/project2/.venv/lib/python3.7/site-packages/pip/__main__.py", line 16, in <module>
      from pip._internal import main as _main  # isort:skip # noqa
    File "/local/data/export/test_enum34/project2/.venv/lib/python3.7/site-packages/pip/_internal/__init__.py", line 4, in <module>
      import locale
    File "/local/data/export/longhor/Python-3.7.3/lib/python3.7/locale.py", line 16, in <module>
      import re
    File "/local/data/export/longhor/Python-3.7.3/lib/python3.7/re.py", line 143, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'

  ----------------------------------------
Command "/local/data/export/test_enum34/project2/.venv/bin/python /local/data/export/test_enum34/project2/.venv/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build
-env-gi5ep75t/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12" failed with error code 1 in None
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Exception trace:
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/console/commands/add.py in handle() at line 139
   status = installer.run()
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/installation/installer.py in run() at line 73
   self._do_install(local_repo)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/installation/installer.py in _do_install() at line 290
   self._execute(op)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/installation/installer.py in _execute() at line 306
   getattr(self, "_execute_{}".format(method))(operation)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/installation/installer.py in _execute_install() at line 331
   self._installer.install(operation.package)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/installation/pip_installer.py in install() at line 30
   self.install_directory(package)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/installation/pip_installer.py in install_directory() at line 205
   return self.run(*args)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/installation/pip_installer.py in run() at line 112
   return self._env.run("python", "-m", "pip", *args, **kwargs)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/utils/env.py in run() at line 536
   return super(VirtualEnv, self).run(bin, *args, **kwargs)
 /local/data/export/longhor/Python-3.7.3/with_poetry/lib/python3.7/site-packages/poetry/utils/env.py in run() at line 385
   raise EnvCommandError(e)

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

Two different machines, two different Python versions, and two different poetry versions - don't know where to start.

[agoose77]

I have the same problem installing a dependency using --path, for my own custom library instead of enum_34.

poetry add texat --path TexAT/analysis/texat

Traceback:

Updating dependencies
Resolving dependencies... (1.4s)


Package operations: 2 installs, 1 update, 0 removals

  - Updating bqplot (0.0.0 dc8a6e5 -> 0.11.2)
  - Installing dephell-discover (0.2.5)
  - Installing texat (0.1.2 TexAT/analysis/texat)

[EnvCommandError]                                                
Command ['/home/angus/.pyenv/versions/miniconda3-latest/envs/nuclear-phd/bin        
/python', '-m', 'pip', 'install', '--no-deps', '-U', '-e', '/home/angus/Git/  
nuclear-phd/TexAT/analysis/texat'] errored with the following output:         
Obtaining file:///home/angus/Git/nuclear-phd/TexAT/analysis/texat             
  Installing build dependencies: started                                      
  Installing build dependencies: finished with status 'error'                 
  ERROR: Complete output from command /home/angus/.pyenv/versions/miniconda3  
-latest/envs/nuclear-phd/bin/python /home/angus/.pyenv/versions/miniconda3-l  
atest/envs/nuclear-phd/lib/python3.7/site-packages/pip install --ignore-inst  
alled --no-user --prefix /tmp/pip-build-env-fbuiifpr/overlay --no-warn-scrip  
t-location --no-binary :none: --only-binary :none: -i https://pypi.org/simpl  
e -- 'poetry>=0.12':                                                          
  ERROR: Traceback (most recent call last):                                   
    File "/home/angus/.pyenv/versions/miniconda3-latest/envs/nuclear-phd/lib  
/python3.7/runpy.py", line 193, in _run_module_as_main                        
      "__main__", mod_spec)                                                   
    File "/home/angus/.pyenv/versions/miniconda3-latest/envs/nuclear-phd/lib  
/python3.7/runpy.py", line 85, in _run_code                                   
      exec(code, run_globals)                                                 
    File "/home/angus/.pyenv/versions/miniconda3-latest/envs/nuclear-phd/lib  
/python3.7/site-packages/pip/__main__.py", line 16, in <module>               
      from pip._internal import main as _main  # isort:skip # noqa            
    File "/home/angus/.pyenv/versions/miniconda3-latest/envs/nuclear-phd/lib  
/python3.7/site-packages/pip/_internal/__init__.py", line 4, in <module>      
      import locale                                                           
    File "/home/angus/.pyenv/versions/miniconda3-latest/envs/nuclear-phd/lib  
/python3.7/locale.py", line 16, in <module>                                   
      import re                                                               
    File "/home/angus/.pyenv/versions/miniconda3-latest/envs/nuclear-phd/lib  
/python3.7/re.py", line 143, in <module>                                      
      class RegexFlag(enum.IntFlag):                                          
  AttributeError: module 'enum' has no attribute 'IntFlag'                    
  ----------------------------------------                                    
ERROR: Command "/home/angus/.pyenv/versions/miniconda3-latest/envs/nuclear-p  
hd/bin/python /home/angus/.pyenv/versions/miniconda3-latest/envs/nuclear-phd  
/lib/python3.7/site-packages/pip install --ignore-installed --no-user --pref  
ix /tmp/pip-build-env-fbuiifpr/overlay --no-warn-script-location --no-binary  
 :none: --only-binary :none: -i https://pypi.org/simple -- 'poetry>=0.12'" f  
ailed with error code 1 in None                                               

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

The thing is, this doesn't seem to be anything to do with Poetry. The blame here lies with enum34 having an out of date interface. I'm not sure which package brought it into my site-packages directory, but I've just removed it manually for now.

[mohelm]

@agoose77 I have this problem as well. Can you explain a bit more what it means "... I've just removed it manually for now"? Thx.

[agoose77]

I just removed it using pip, which is obviously not ideal :p

[mohelm]

Thanks @agoose77 . So this means that the poetry machine is not available right? I have a use case where one of my projects, a library, is hosted on Github and I want to install it into some other project.

I would like to do it with poetry install and a corresponding pyproject.toml, but this gives the aforementioned error.

The workaround I am using currently is to build the library locally and then install it via pip install path/to/dist/library.tar.gz.

[dazza-codes]

• also applies to python 3.6 on poetry 0.12.17

$ pip freeze | grep enum
enum34==1.1.6

In [1]: import enum                                                                                                                                  
In [2]: enum                                                                                                                                         
Out[2]: <module 'enum' from '/home/joe/miniconda3/envs/py3.6/lib/python3.6/enum.py'>

[EnvCommandError]                                                                                                                     
Command ['/home/joe/.cache/pypoetry/virtualenvs/project-py3.6/bin/python', '-m', 'pip', 'install', '--no-deps', '-U', '-e', '/home/joe/.c        
ache/pypoetry/virtualenvs/project-py3.6/src/priv-log'] errored with the following return code 1, and output:                                    
Obtaining file:///home/joe/.cache/pypoetry/virtualenvs/project-py3.6/src/priv-log                                                           
  Installing build dependencies: started                                                                                                           
  Installing build dependencies: finished with status 'error'                                                                                      
  ERROR: Command errored out with exit status 1:                                                                                                   
   command: /home/joe/.cache/pypoetry/virtualenvs/project-py3.6/bin/python /home/joe/.cache/pypoetry/virtualenvs/project-py3.6/lib/python  
3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k0fam8cf/overlay --no-warn-script-location --no-binary :no  
ne: --only-binary :none: -i https://pypi.org/simple -- 'poetry>=0.12'                                                                              
       cwd: None                                                                                                                                   
  Complete output (14 lines):                                                                                                                      
  Traceback (most recent call last):                                                                                                               
    File "/home/joe/miniconda3/envs/py3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main                                            
      "__main__", mod_spec)                                                                                                                        
    File "/home/joe/miniconda3/envs/py3.6/lib/python3.6/runpy.py", line 85, in _run_code                                                       
      exec(code, run_globals)                                                                                                                      
    File "/home/joe/.cache/pypoetry/virtualenvs/project-py3.6/lib/python3.6/site-packages/pip/__main__.py", line 16, in <module>               
      from pip._internal import main as _main  # isort:skip # noqa                                                                                 
    File "/home/joe/.cache/pypoetry/virtualenvs/project-py3.6/lib/python3.6/site-packages/pip/_internal/__init__.py", line 4, in <module>      
      import locale                                                                                                                                
    File "/home/joe/miniconda3/envs/py3.6/lib/python3.6/locale.py", line 16, in <module>                                                       
      import re                                                                                                                                    
    File "/home/joe/miniconda3/envs/py3.6/lib/python3.6/re.py", line 142, in <module>                                                          
      class RegexFlag(enum.IntFlag):                                                                                                               
  AttributeError: module 'enum' has no attribute 'IntFlag'                                                                                         
  ----------------------------------------                                                                                                         
ERROR: Command errored out with exit status 1: /home/joe/.cache/pypoetry/virtualenvs/project-py3.6/bin/python /home/joe/.cache/pypoetry/v  
irtualenvs/project-py3.6/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k0fam8cf/overlay --no-w  
arn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'poetry>=0.12' Check the logs for full command output.   

Exception trace:
 /home/joe/.poetry/lib/poetry/_vendor/py3.6/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /home/joe/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /home/joe/.poetry/lib/poetry/_vendor/py3.6/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /home/joe/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /home/joe/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /home/joe/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /home/joe/.poetry/lib/poetry/console/commands/install.py in handle() at line 55
   return_code = installer.run()
 /home/joe/.poetry/lib/poetry/installation/installer.py in run() at line 73
   self._do_install(local_repo)
 /home/joe/.poetry/lib/poetry/installation/installer.py in _do_install() at line 290
   self._execute(op)
 /home/joe/.poetry/lib/poetry/installation/installer.py in _execute() at line 306
   getattr(self, "_execute_{}".format(method))(operation)
 /home/joe/.poetry/lib/poetry/installation/installer.py in _execute_install() at line 331
   self._installer.install(operation.package)
 /home/joe/.poetry/lib/poetry/installation/pip_installer.py in install() at line 35
   self.install_git(package)
 /home/joe/.poetry/lib/poetry/installation/pip_installer.py in install_git() at line 230
   self.install_directory(pkg)
 /home/joe/.poetry/lib/poetry/installation/pip_installer.py in install_directory() at line 205
   return self.run(*args)
 /home/joe/.poetry/lib/poetry/installation/pip_installer.py in run() at line 112
   return self._env.run("python", "-m", "pip", *args, **kwargs)
 /home/joe/.poetry/lib/poetry/utils/env.py in run() at line 539
   return super(VirtualEnv, self).run(bin, *args, **kwargs)
 /home/joe/.poetry/lib/poetry/utils/env.py in run() at line 388
   raise EnvCommandError(e, input=input_)

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

[dazza-codes]

pipdeptree shows that enum34 is:

• not in the conda base environment that poetry uses to derive a virtualenv
• it is not a dependency of any other dependencies in the project
• the project does not explicitly include that dependency

But, when it's uninstalled, poetry reinstalls it. Can't figure out why (yet).

$ pipdeptree --reverse --packages enum34
enum34==1.1.6

$ pip uninstall -y enum34
Uninstalling enum34-1.1.6:
  Successfully uninstalled enum34-1.1.6

$ poetry install
Installing dependencies from lock file

Package operations: 2 installs, 0 updates, 1 removal

  - Installing enum34 (1.1.6)
  - Installing wheel (0.31.1)

$ rm poetry.lock
$ pip uninstall -y enum34
Uninstalling enum34-1.1.6:
  Successfully uninstalled enum34-1.1.6

$ poetry install
Updating dependencies
Resolving dependencies... (1.4s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  - Installing enum34 (1.1.6)
  - Installing wheel (0.31.1)
  - Installing gis-app (0.1.2)

Where the hec is enum34 coming from?

It's not packaged in poetry itself (at least not in the develop branch), i.e.

$ git remote -v
upstream    [email protected]:sdispater/poetry.git (fetch)
upstream    [email protected]:sdispater/poetry.git (push)
$ git fetch -ap upstream
$ git checkout develop
$ git status
On branch develop
nothing to commit, working tree clean

$ git pull upstream develop 
$ git ll
*   25f02c5 - (HEAD -> develop, upstream/develop) Merge branch 'master' into develop (25 hours ago) <Sébastien Eustace>

$ poetry install
Installing dependencies from lock file

Nothing to install or update

  - Installing poetry (1.0.0b1)

$ poetry shell
Virtual environment already activated: /home/joe/.cache/pypoetry/virtualenvs/poetry-py3.6
$ pipdeptree | grep enum34  # nada, good!

$ python
Python 3.6.7 | packaged by conda-forge | (default, Jul  2 2019, 02:18:42) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import enum
>>> enum
<module 'enum' from '/home/joe/miniconda3/envs/py3.6/lib/python3.6/enum.py'>
>>> dir(enum)
['DynamicClassAttribute', 'Enum', 'EnumMeta', 'Flag', 'IntEnum', 'IntFlag', 'MappingProxyType', 'OrderedDict', '_EnumDict', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_auto_null', '_decompose', '_high_bit', '_is_descriptor', '_is_dunder', '_is_sunder', '_make_class_unpicklable', '_or_', '_power_of_two', '_reduce_ex_by_name', 'auto', 'reduce', 'sys', 'unique']
>>> 

So, poetry 1.x (poetry-develop) will not encounter this bug in the development process, AFAICT, at least on py ^3.6 but maybe it's present in earlier py-ver builds/CI?

[dazza-codes]

• enum34 does not come from a bare-poetry project, i.e.

$ poetry --version
Poetry 0.12.17
$ mkdir tmp-project
$ cd tmp-project/
$ poetry init

... snipped ...

Generated file

[tool.poetry]
name = "tmp-project"
version = "0.1.0"
description = ""
authors = ["Darren <snipped>"]

[tool.poetry.dependencies]
python = "^3.6"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

$ poetry install
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Nothing to install or update

$ ls -1
poetry.lock
pyproject.toml

$ poetry shell
Spawning shell within /home/joe/.cache/pypoetry/virtualenvs/tmp-project-py3.6

$ pipdeptree | grep enum  # nada, good!

$ python
Python 3.6.7 | packaged by conda-forge | (default, Jul  2 2019, 02:18:42) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import enum
>>> enum
<module 'enum' from '/home/joe/miniconda3/envs/py3.6/lib/python3.6/enum.py'>
>>> dir(enum)
['DynamicClassAttribute', 'Enum', 'EnumMeta', 'Flag', 'IntEnum', 'IntFlag', 'MappingProxyType', 'OrderedDict', '_EnumDict', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_auto_null', '_decompose', '_high_bit', '_is_descriptor', '_is_dunder', '_is_sunder', '_make_class_unpicklable', '_or_', '_power_of_two', '_reduce_ex_by_name', 'auto', 'reduce', 'sys', 'unique']

So, with poetry 0.12.x on py3.6, this is not something introduced by poetry itself (nor is it introduced by miniconda3 when it is used to create a base py3.6 env).

[dazza-codes]

• enum34 was _not_ introduced by any of the following:

[tool.poetry.dependencies]
python = "^3.6"
boto3 = "^1.9"
dataclasses = "^0.6.0"
requests = "^2.22"

s3fs = "^0.3.4"
dask = {version = "^2.3", extras = ["delayed"]}
fastparquet = "^0.3.2"

pycrs = "^1.0"

[tool.poetry.dev-dependencies]
wheel = "^0.31"
moto = "^1.3"
black = {version = "^18.3-alpha.0", allows-prereleases = true}
autopep8 = "^1.4"
flake8 = "^3.7"
ipdb = "^0.12.2"
invoke-release = "^4.1"
mypy = "^0.720.0"
pipdeptree = "^0.13.2"
pylint = "^2.3"
pytest = "^5.1"
pytest-cov = "^2.7"
pytest-mock = "^1.10"
requests-mock = "^1.7"
twine = "^1.14"

It _was_ introduced by rasterio 1.x

[tool.poetry.dependencies]
python = "^3.6"
rasterio = "^1.0"

$ poetry add rasterio
Using version ^1.0 for rasterio

Updating dependencies
Resolving dependencies... (1.0s)

Writing lock file


Package operations: 7 installs, 0 updates, 0 removals

  - Installing affine (2.3.0)
  - Installing click-plugins (1.1.1)
  - Installing cligj (0.5.0)
  - Installing enum34 (1.1.6)
  - Installing snuggs (1.4.6)
  - Installing wheel (0.31.1)
  - Installing rasterio (1.0.28)

But, even so, the enum appears to behave as expected, e.g.

$ rio insp {any_tif_file}
Rasterio 1.0.28 Interactive Inspector (Python 3.6.7)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> import enum
>>> enum
<module 'enum' from '/home/joe/miniconda3/envs/py3.6/lib/python3.6/enum.py'>
>>> dir(enum)
['DynamicClassAttribute', 'Enum', 'EnumMeta', 'Flag', 'IntEnum', 'IntFlag', 'MappingProxyType', 'OrderedDict', '_EnumDict', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_auto_null', '_decompose', '_high_bit', '_is_descriptor', '_is_dunder', '_is_sunder', '_make_class_unpicklable', '_or_', '_power_of_two', '_reduce_ex_by_name', 'auto', 'reduce', 'sys', 'unique']

The enum34 is still in the dependencies, but somehow not loaded/imported in that case?

$ pipdeptree | grep enum
enum34==1.1.6

# But it is lost in the dep-graph for rasterio
$ pipdeptree | grep -A45 rasterio | grep enum34  # nada?  why?

https://github.com/mapbox/rasterio/blob/master/requirements.txt#L6 has an explicit dependency on enum34 for py<3.4 - is something messed up in this syntax (or how it is used by setup.py)?

affine>=1.3.0
attrs>=16.0.0
boto3>=1.2.4
click==7.0
cligj>=0.5
enum34; python_version<'3.4'
numpy>=1.10
snuggs>=1.4.1
setuptools>=0.9.8

@sgillies ?

• linked issue https://github.com/mapbox/rasterio/issues/1777

https://stackoverflow.com/questions/19559247/requirements-txt-depending-on-python-version

# subtle parsing problems with:

enum34; python_version<'3.4'

# vs e.g.

enum34 >= 1.0; python_version < '3.4'

Or maybe something to do with extra options not being explicit about the python-version restriction for enum34?

• https://github.com/mapbox/rasterio/blob/master/setup.py#L200

On the poetry side, parsing the environment-markers is the feature that should control this

• https://www.python.org/dev/peps/pep-0508/#environment-markers

According to PEP-0508, the restriction example they give looks like

argparse;python_version<"2.7"

Translating that to rasterio/enum looks almost identical to what is already there

enum34;python_version<"3.4"

(only a whitespace and single->double quote change).

[dazza-codes]

The following "hack" does not solve the problem from the poetry perspective by using an explicit dependency to exclude enum34 like so

poetry add enum34 --python="<3.4"

[tool.poetry.dependencies]
python = "^3.6"
rasterio = "^1.0"
enum34 = {version = "^1.1", python = "<3.4"}

It still installs the enum34.

$ poetry install
Creating virtualenv tmp-project-py3.6 in /home/dlweber/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file


Package operations: 10 installs, 0 updates, 0 removals

  - Installing click (7.0)
  - Installing numpy (1.17.2)
  - Installing pyparsing (2.4.2)
  - Installing affine (2.3.0)
  - Installing attrs (19.1.0)
  - Installing click-plugins (1.1.1)
  - Installing cligj (0.5.0)
  - Installing enum34 (1.1.6)
  - Installing snuggs (1.4.6)
  - Installing rasterio (1.0.28)

Even a version incompatibility did not avoid installing enum34 because rasterio has no version restraints on it and poetry installed version 1.1.6 anyway (which is the really weird bit).

[tool.poetry.dependencies]
python = "^3.6"
rasterio = "^1.0"
enum34 = {version = "<1.0", python = "<3.4"}

With enum34 at version > 1.0, things seemed to work, but with enum34 at 1.0, the bug is reproduced

[tool.poetry.dependencies]
python = "^3.6"
enum34 = "<=1.0"
rasterio = "^1.0"

[EnvCommandError]                                                                                                                    
Command ['/home/joe/.cache/pypoetry/virtualenvs/tmp-project-py3.6/bin/python', '-m', 'pip', 'install', '--no-deps', 'enum34==1.0'] errored with the following return code 1, and output:                                                                                                       
Collecting enum34==1.0                                                                                                                            
  Downloading https://files.pythonhosted.org/packages/2f/63/6c3fa92c2f1c6c4fb09d3b2b04e38c036c17de4698d20a85192ef4a6cabb/enum34-1.0.tar.gz        
    Complete output from command python setup.py egg_info:                                                                                        
    Traceback (most recent call last):                                                                                                            
      File "<string>", line 1, in <module>                                                                                                        
      File "/home/joe/.cache/pypoetry/virtualenvs/tmp-project-py3.6/lib/python3.6/site-packages/setuptools/__init__.py", line 5, in <module>  
        import distutils.core                                                                                                                     
      File "/home/joe/miniconda3/envs/py3.6/lib/python3.6/distutils/core.py", line 16, in <module>                                            
        from distutils.dist import Distribution                                                                                                   
      File "/home/joe/miniconda3/envs/py3.6/lib/python3.6/distutils/dist.py", line 9, in <module>                                             
        import re                                                                                                                                 
      File "/home/joe/miniconda3/envs/py3.6/lib/python3.6/re.py", line 142, in <module>                                                       
        class RegexFlag(enum.IntFlag):                                                                                                            
    AttributeError: module 'enum' has no attribute 'IntFlag'

[dazza-codes]

Sanity check:

[tool.poetry.dependencies]
python = "^3.6"
enum34 = {version = "^1.0", python = "<3.4"}

# wipe out the virtualenv
$ rm poetry.lock
$ poetry install
Creating virtualenv tmp-project-py3.6 in /home/joe/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Nothing to install or update

OK, good.

[alex-lechner]

Having the same issue with enum34... Is there still no fix for this? I'm currently using poetry 1.0.0b3 on MacOS 10.15

My workaround is the following:

1. Track down which dependency relies on enum34. In my case it's geopandas.
2. First, install all libraries first that DON'T rely on enum34
3. When everything is installed, install the remaining libs that use enum34

It's a very suboptimal solution because if you want to build a Docker container you have to make sure to copy the poetry.lock-file without the enum34 dependent libraries and install the enum34 dependent libs (e.g. geopandas) separately in the Docker build process because - unfortunately - poetry does not install the packages in pyproject.toml sequentially.

Here is an example of my very weird hack to overcome this issue during Docker build:

FROM python:3.7.3-slim AS compile
ARG USERNAME 
ARG PASSWORD
RUN apt-get update && apt-get install -y --no-install-recommends build-essential gcc

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" \
    POETRY_VERSION=1.0.0b3

WORKDIR /my-app-folder

COPY pyproject.toml poetry.lock /my-app-folder/

RUN sed -i '/geopandas/d' pyproject.toml && \ # Remove libs with enum34 dependencies from pyproject.toml (e.g. geopandas)
    sed -i -e "s/localhost/host.docker.internal/g" * && \ # Replace occurences of 'localhost' with 'host.docker.internal' to resolve the IP address of the host machine instead of the docker container during build
    pip install poetry==$POETRY_VERSION && \ 
    poetry config virtualenvs.create false && \
    poetry config repositories.foo http://host.docker.internal:1000/my-private-pypi/simple && \ # set private repository
    poetry config http-basic.foo $USERNAME $PASSWORD && \ # set auth credentials
    poetry install --no-dev --no-interaction --no-ansi && \
    pip install geopandas # Install libs with enum34 dependencies here at last step (geopandas needs to be installed with pip because poetry raises ModuleNotFoundError: No module named 'attr')

COPY . /my-app-folder

FROM python:3.7.3-slim
COPY --from=compile /opt/venv /opt/venv
COPY --from=compile /my-app-folder /my-app-folder

WORKDIR /my-app-folder

ENV PATH="/opt/venv/bin:$PATH"
CMD ["python", "app.py"]

If anyone has a better solution please post!

[Granitosaurus]

[tool.poetry.dependencies]
python = "^3.6"
enum34 = {version = "^1.0", python = "<3.4"}

This doesn't seem to work; I can't replicate it in any way, shape or form. poetry seems to always install enum34, ignoring python versioning.

Seems like Poetry fails to interpret setuptools python_version keyword and blindly considers everything under install_requires as a dependency. For example:

# setup.py
install_requires = [
    "enum34; python_version < '3.5'
]

If this project is added to pyproject.toml installation will break even if enum34 is explicitly stated to be skipped:

[tool.poetry.dependencies]
python = "^3.6"
enum34 = {version = "^1.0", python = "<3.4"}

I'm encountering this issue with couchbase package from pypi.

There's definitely something wrong with poetry's dependency resolution here.

[magnusja]

Hey @Granitosaurus

did you uninstall enum34? It will fail if enum34 is still installed from a previous run I think.

Worst case go to the lock file after you did a poetry update (which fails) and then remove all enum34 references (should be 3 in total) and then do a poetry install.

[Granitosaurus]

did you uninstall enum34? It will fail if enum34 is still installed from a previous run I think.

I did:

$ rm poetry.lock
$ rm ~/.cache/pypoetry/virtualenvs/myproject-py3.7 -r  
$ poetry install -vv
...
AttributeError: module 'enum' has no attribute 'IntFlag'

No luck resolving this other than manually editing lock file which is really awkward and kinda defeats the whole purpose of poetry.
I'm not really versed enough in Poetry's dependency resolution but as far as I understand enum34 in this case shouldn't even be added as a dependency as two sources are indicating that it shouldn't: couchbase project's setup.py indicates python <3.5 and pyproject.toml has explicit <3.4 - so why is poetry picking it up?

[magnusja]

@Granitosaurus I dont know.. I am annoyed by this myself. I have the problem with tensorflow, but adding

[tool.poetry.dependencies]
python = "^3.6"
enum34 = {version = "^1.0", python = "<3.4"}

to my project toml and uninstalling enum34 / deleting the env works for me. I am using python 3.7.4 though

[sdispater]

@Granitosaurus This is not on Poetry's end. If you look at the PyPI API here: https://pypi.org/pypi/couchbase/json. The requires_dist section specifies enum34 without environment markers so Poetry considers it a mandatory dependency. It does not matter if you put a restriction on it in your pyproject.toml file because Poetry will always include it due to the fact that it sees that it's required by couchbase. Since it seems that the setup.py of the couchbase package properly declares the marker I think there is an issue in the way they are uploading the package to PyPI. This should be reported to them.

Every reported case in this issue is a variation of this. For instance, it's the same thing for rasterio (see https://pypi.org/pypi/rasterio/json).

In each case it should be reported upstream.

[agoose77]

@sdispater
I'm not hugely knowledgeable on any of this, but frequently it seems Poetry is falling foul of poorly written packages. It's unfortunate that this is the case, but really it's not entirely fair to expect the tooling (poetry, etc) to handle this when it should be sorted upstream. enum34 is just one rather awkward example of this. Thanks for all the good work! I really love using Poetry.

[alex-lechner]

@sdispater Would it be possible to at least orchestrate the installation order of packages? Let's say I have a package like couchbase that requires enum34 but messes up the installation of other packages, I will install couchbase at the very end.

Something like:

[tool.poetry.dependencies]
python = "^3.7"
numpy = "^1.17.3"
matplotlib = "^3.1.1"
couchbase = {version = "^2.5.9", order = "-1"}

Or simply execute the installation per package consecutively as specified in pyproject.toml.
What are your thoughts on this?

[Granitosaurus]

Really poetry should have some sort of resolution order and priority - this will be an endless cat and mouse game. You either:

• fix upstream which might take a long time.
• edit poetry.lock file by hand which is a complete hack.

I should be able to specify explictly that this dependency should be ignored.

[sixers]

It would be great if poetry had a way to get those broken packages under control.
I think poetry should include a mechanism for enforcing specific versions / overwriting them.

If I depend on a package which includes enum34 without python version marker, adding

enum34 = {version = "^1.1.6", markers = "python_version ~= '2.7'"}

Should fix the issue. Right now the python version is ignored. I guess it's on purpose, so there could be additional flag, to enforce or overwrite this dependency

enum34 = {version = "^1.1.6", markers = "python_version ~= '2.7'", overwrite = true}

And this should pin the correct marker / version regardless of the rest of the dependency tree.

It could be named force as well.

WDYT?

It might also be useful in other cases, e.g. if we want to force our own fork of a particular dependency, without changing the dependent library.

[Granitosaurus]

I'd generally be heavily in favor of force flag.
You might argue that it's a _slippery slope_ but at the end of the day the pypi dep resolution just isn't 100% trustworthy and manual adjustments are necessary.
As an illustration the enum example is just filled with pitfalls here. The couchbase has been listing it as optional but for some reason pypi's json output failed to parse it:

That's weird, it (enum) is and has always been a conditional dependency:
Not sure why the dependency isn't listed conditionally in the JSON file that PyPi produces.
https://issues.couchbase.com/browse/PYCBC-688

Chasing upsream just doesn't seem to be sustainable as a single strategy imho.

[davidkelly]

When I curl https://pypi.org/pypi/couchbase/json, I see
requires_dist":["enum34 ; python_version < \"3.5\"","typing ; python_version < \"3.7\""]

So - unsure what the issue is. I've installed this version (using pip3) with python 3.6 in the past without issues, as well. There are embedded quotes, but I'm not sure if that is an issue. LMK what I'm missing here and I'll see if we can fix it.

[agoose77]

The version specifier was only just added in the latest release (December):
https://pypi.org/pypi/couchbase/2.5.9/json

[griels]

Echoing https://github.com/python-poetry/poetry/issues/1122#issuecomment-530567802:

Interesting, we had already noticed that at least some implementations of Environmental Markers from PEP508, require double rather than single quotes around marker_expr strings - the "enum34; python_version<'3.5'" marker was there since enum34 was added - we simply changed the single to double quotes and that has seemed to satisfy PyPi and the remaining PEP508 implementations. It looks like perhaps whatever Poetry uses for PEP508 is the same. It does look like this sort of quoting is indicated in the examples, though I can't see it clearly in the grammar here: https://www.python.org/dev/peps/pep-0508/#grammar ... In fact, in the 'tests' at https://www.python.org/dev/peps/pep-0508/#id27 there is a line as follows:

"name[quux, strange];python_version<'2.7' and platform_version=='2'"

Has our change in couchbase==2.5.9 resolved the issue with Poetry, at least?

[mickeykkim]

I was able to fix this by adding enum34 = "==1.1.8" to pyproject.toml. Apparently enum34 had a feature in v1.1.8 that avoided this error, but this regressed in v1.1.9+. This is just a workaround though. The better solution would be for packages to use environment markers so you don't have to install enum34 at all unless needed.

[ojomio]

@mickeykkim thanks a lot, your workaround works! :tada:

[curtiscook]

I was able to fix this by adding enum34 = "==1.1.8" to pyproject.toml. Apparently enum34 had a feature in v1.1.8 that avoided this error, but this regressed in v1.1.9+. This is just a workaround though. The better solution would be for packages to use environment markers so you don't have to install enum34 at all unless needed.

This isn't working for me because a package is conservatively being interpreted as having an enum34 1.1.9 requirement

This is definitely an enum34 issue, but seeing as it's a backport for older versions of enum -- can't there be a way that poetry interprets the requirement as met if we're running python34+? Pipenv finally had a release and it seems like this might be what they are doing.

Alternatively, I think the only real workaround is to not include the package in the pyproject.toml and then pip install (which kind of defeats the purposes of a package manager)

[levrik]

I've suddenly starting to experience this.
poetry.lock wasn't changed. It's passing on CI where I can also see that enum34 package is installed through some dependency of my project.
I just don't understand why this suddenly starts to pop up. Also for a colleague it's working completely fine.
Already cleared all caches, re-installed Poetry, upgrade Pip etc etc
Will try the solutions posted here now but I'm just confused why this suddenly pops up while I didn't change anything on my setup and the poetry.lock.

[levrik]

Okay. I was able to fix this by upgrading enum34 from 1.1.6 to 1.1.10 inside poetry.lock.
Achieved this by doing poetry add enum34==1.1.10, followed by poetry remove enum34.
Maybe a poetry update enum34 would also have worked. No idea.