Distributed: Click dependency problem with latest version (8.0.0)

[nmerket]

What happened: Error when calling dask-scheduler or dask-worker

What you expected to happen: Not an error

Minimal Complete Verifiable Example:

# Start from a blank python 3.8 environment
$ pip install distributed
# installs click 8.0.0 as a dependency

$ dask-scheduler --help
Traceback (most recent call last):
  File "/Users/nmerket/opt/anaconda3/envs/tempdask/bin/dask-scheduler", line 5, in <module>
    from distributed.cli.dask_scheduler import go
  File "/Users/nmerket/opt/anaconda3/envs/tempdask/lib/python3.8/site-packages/distributed/cli/dask_scheduler.py", line 120, in <module>
    def main(
  File "/Users/nmerket/opt/anaconda3/envs/tempdask/lib/python3.8/site-packages/click/decorators.py", line 247, in decorator
    _param_memo(f, OptionClass(param_decls, **option_attrs))
  File "/Users/nmerket/opt/anaconda3/envs/tempdask/lib/python3.8/site-packages/click/core.py", line 2467, in __init__
    super().__init__(param_decls, type=type, multiple=multiple, **attrs)
  File "/Users/nmerket/opt/anaconda3/envs/tempdask/lib/python3.8/site-packages/click/core.py", line 2106, in __init__
    raise ValueError(
ValueError: 'default' must be a list when 'multiple' is true.

$ dask-worker --help
Traceback (most recent call last):
  File "/Users/nmerket/opt/anaconda3/envs/tempdask/bin/dask-worker", line 8, in <module>
    sys.exit(go())
  File "/Users/nmerket/opt/anaconda3/envs/tempdask/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 461, in go
    check_python_3()
  File "/Users/nmerket/opt/anaconda3/envs/tempdask/lib/python3.8/site-packages/distributed/cli/utils.py", line 32, in check_python_3
    _unicodefun._verify_python3_env()
AttributeError: module 'click._unicodefun' has no attribute '_verify_python3_env'

Anything else we need to know?:

If I downgrade click to 7.1.2, it works as expected.

Environment:

• Dask version: 2021.4.1
• Python version: 3.7, 3.8
• Operating System: Mac, CentOS
• Install method (conda, pip, source): pip or conda (conda-forge)

certifi==2020.12.5
click==8.0.0
cloudpickle==1.6.0
dask==2021.4.1
distributed==2021.4.1
fsspec==2021.4.0
HeapDict==1.0.1
locket==0.2.1
msgpack==1.0.2
partd==1.2.0
psutil==5.8.0
PyYAML==5.4.1
sortedcontainers==2.3.0
tblib==1.7.0
toolz==0.11.1
tornado==6.1
zict==2.0.0

[jrbourbeau]

Thanks for reporting this issue @nmerket! There's a new release of distributed coming out tomorrow which includes updates to support click >= 8.0.0 (xref https://github.com/dask/distributed/pull/4810).

Additionally, we've recently patched all existing distributed releases on conda-forge to pin click < 8. If you're using pip to install distributed from PyPI, could you please manually pin click < 8 until the distributed release tomorrow. Thanks!

[jrbourbeau]

I'm also pinning this issue in the issue tracker to (hopefully) increase its visibility for other who may also be running into the same issue

[nmerket]

Wow, thanks for the quick response!

[fredms]

Seeing the same issue. Tried to install the latest package of distributed 2021.5.0, but got error:
ResolvePackageNotFound:

• distributed==2021.5.0

[jrbourbeau]

@fredms how are you attempting to install distributed? distributed==2021.5.0 should be on PyPI, but isn't up on conda-forge yet

[fredms]

Just tried pip install distributed==2021.5.0
ERROR: Could not find a version that satisfies the requirement distributed==2021.5.0
ERROR: No matching distribution found for distributed==2021.5.0

[jrbourbeau]

Hmm, unfortunately I'm not able to reproduce. distributed==2021.5.0 is definitely on PyPI. I did the following locally on my mac:

# Create a fresh conda environment with Python
$ conda create -n test python=3
# Activate that environment
$ conda activate test
# Install `distributed==2021.5.0` with `pip`
$ python -m pip install distributed==2021.5.0

which ran successfully for me (output shown below). I'm not totally sure how to help debug further. By chance to do you have pip configured to point to some non-default index?

Details:

(test) ➜  ~ python -m pip install distributed==2021.5.0
Collecting distributed==2021.5.0
  Using cached distributed-2021.5.0-py3-none-any.whl (699 kB)
Collecting psutil>=5.0
  Using cached psutil-5.8.0-cp39-cp39-macosx_10_9_x86_64.whl (236 kB)
Collecting tblib>=1.6.0
  Using cached tblib-1.7.0-py2.py3-none-any.whl (12 kB)
Collecting toolz>=0.8.2
  Using cached toolz-0.11.1-py3-none-any.whl (55 kB)
Collecting msgpack>=0.6.0
  Using cached msgpack-1.0.2-cp39-cp39-macosx_10_14_x86_64.whl (74 kB)
Collecting tornado>=6.0.3
  Using cached tornado-6.1-cp39-cp39-macosx_10_9_x86_64.whl (416 kB)
Collecting sortedcontainers!=2.0.0,!=2.0.1
  Using cached sortedcontainers-2.3.0-py2.py3-none-any.whl (29 kB)
Collecting pyyaml
  Using cached PyYAML-5.4.1-cp39-cp39-macosx_10_9_x86_64.whl (259 kB)
Collecting click>=6.6
  Using cached click-8.0.0-py3-none-any.whl (96 kB)
Requirement already satisfied: setuptools in ./miniforge3/envs/test/lib/python3.9/site-packages (from distributed==2021.5.0) (49.6.0.post20210108)
Collecting cloudpickle>=1.5.0
  Using cached cloudpickle-1.6.0-py3-none-any.whl (23 kB)
Collecting zict>=0.1.3
  Using cached zict-2.0.0-py3-none-any.whl (10 kB)
Collecting dask==2021.05.0
  Using cached dask-2021.5.0-py3-none-any.whl (960 kB)
Collecting partd>=0.3.10
  Using cached partd-1.2.0-py3-none-any.whl (19 kB)
Collecting fsspec>=0.6.0
  Using cached fsspec-2021.5.0-py3-none-any.whl (111 kB)
Collecting locket
  Using cached locket-0.2.1-py2.py3-none-any.whl (4.1 kB)
Collecting heapdict
  Using cached HeapDict-1.0.1-py3-none-any.whl (3.9 kB)
Installing collected packages: toolz, locket, pyyaml, partd, heapdict, fsspec, cloudpickle, zict, tornado, tblib, sortedcontainers, psutil, msgpack, dask, click, distributed
Successfully installed click-8.0.0 cloudpickle-1.6.0 dask-2021.5.0 distributed-2021.5.0 fsspec-2021.5.0 heapdict-1.0.1 locket-0.2.1 msgpack-1.0.2 partd-1.2.0 psutil-5.8.0 pyyaml-5.4.1 sortedcontainers-2.3.0 tblib-1.7.0 toolz-0.11.1 tornado-6.1 zict-2.0.0

[fredms]

It works now. I tried a different machine.

[jakirkham]

Closing as this should be fixed in the latest release