Distributed: TypeError: catching classes that do not inherit from BaseException is not allowed

Created on 21 Mar 2020  路  6Comments  路  Source: dask/distributed

I upgraded my python3 version on my MacBook pro to 3.7.7, and now distributed is not able to serialize/deserialize. 

distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 101, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 193, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.core - ERROR - unpackb() got an unexpected keyword argument 'strict_map_key'
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/core.py", line 346, in handle_comm
    msg = await comm.read()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/tcp.py", line 212, in read
    frames, deserialize=self.deserialize, deserializers=deserializers
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py", line 75, in from_frames
    res = _from_frames()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    frames, deserialize=deserialize, deserializers=deserializers
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 101, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 193, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 101, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 193, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
Traceback (most recent call last):
  File "cu_training.py", line 51, in <module>
    cluster = LocalCluster()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/local.py", line 211, in __init__
    security=security,
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/spec.py", line 256, in __init__
    self.sync(self._start)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/cluster.py", line 161, in sync
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/utils.py", line 348, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/utils.py", line 332, in f
    result[0] = yield future
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/spec.py", line 289, in _start
    await super()._start()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/cluster.py", line 60, in _start
    self.scheduler_info = await comm.read()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/tcp.py", line 212, in read
    frames, deserialize=self.deserialize, deserializers=deserializers
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py", line 75, in from_frames
    res = _from_frames()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    frames, deserialize=deserialize, deserializers=deserializers
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 101, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 193, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/utils.py", line 200, in ignoring
    yield
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/spec.py", line 607, in close_clusters
    cluster.close(timeout=10)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/cluster.py", line 82, in close
    return self.sync(self._close, callback_timeout=timeout)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/cluster.py", line 161, in sync
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/utils.py", line 348, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/utils.py", line 332, in f
    result[0] = yield future
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/spec.py", line 380, in _close
    self.scale(0)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/deploy/spec.py", line 444, in scale
    v["name"] for v in self.scheduler_info["workers"].values()
KeyError: 'workers'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/utils.py", line 201, in ignoring
    except exceptions as e:
TypeError: catching classes that do not inherit from BaseException is not allowed

Have any of you experienced the same situation?

picture muammar

Most helpful comment

Thanks for raising an issue @muammar. This error: TypeError: unpackb() got an unexpected keyword argument 'strict_map_key' is coming from msgpack. What version of msgpack do you have installed? We recently bumped the minimum supported version to msgpack==0.6.0, which should have the 'strict_map_key' keyword argument

picture jrbourbeau on 21 Mar 2020
👍2

All 6 comments

Thanks for raising an issue @muammar. This error: TypeError: unpackb() got an unexpected keyword argument 'strict_map_key' is coming from msgpack. What version of msgpack do you have installed? We recently bumped the minimum supported version to msgpack==0.6.0, which should have the 'strict_map_key' keyword argument

jrbourbeau picture jrbourbeau on 21 Mar 2020
👍2

You are correct. That solved it. I should close the report.

muammar picture muammar on 21 Mar 2020

Glad to hear that fixed things

jrbourbeau picture jrbourbeau on 21 Mar 2020
❤1

I have this bug.

distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:38591
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:38591
distributed.worker - INFO -          dashboard at:            127.0.0.1:38859
distributed.worker - INFO - Waiting to connect to:       tcp://127.0.0.1:8786
distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:40635
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:40635
distributed.worker - INFO -          dashboard at:            127.0.0.1:45265
distributed.worker - INFO - Waiting to connect to:       tcp://127.0.0.1:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          2
distributed.worker - INFO -                Memory:                    8.36 GB
distributed.worker - INFO -       Local Directory: /data/matcha.doodle.backend/dask-worker-space/worker-0p7c8zw4
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          2
distributed.worker - INFO -                Memory:                    8.36 GB
distributed.worker - INFO -       Local Directory: /data/matcha.doodle.backend/dask-worker-space/worker-ek57dqzs
distributed.worker - INFO - -------------------------------------------------
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 107, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 199, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 713, in run
    await worker
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/worker.py", line 1060, in start
    await self._register_with_scheduler()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/worker.py", line 844, in _register_with_scheduler
    response = await future
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/tcp.py", line 212, in read
    frames, deserialize=self.deserialize, deserializers=deserializers
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/utils.py", line 75, in from_frames
    res = _from_frames()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    frames, deserialize=deserialize, deserializers=deserializers
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 107, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 199, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 107, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 199, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 713, in run
    await worker
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/worker.py", line 1060, in start
    await self._register_with_scheduler()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/worker.py", line 844, in _register_with_scheduler
    response = await future
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/tcp.py", line 212, in read
    frames, deserialize=self.deserialize, deserializers=deserializers
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/utils.py", line 75, in from_frames
    res = _from_frames()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    frames, deserialize=deserialize, deserializers=deserializers
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 107, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 199, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP  local=tcp://127.0.0.1:34912 remote=tcp://127.0.0.1:8786>
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP  local=tcp://127.0.0.1:34914 remote=tcp://127.0.0.1:8786>
distributed.nanny - ERROR - Failed while trying to start worker process: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.nanny - ERROR - Failed while trying to start worker process: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.dask_worker - INFO - End worker
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/bin/dask-worker", line 11, in <module>
    sys.exit(go())
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 436, in go
    main()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 422, in main
    loop.run_sync(run)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/ioloop.py", line 532, in run_sync
    return future_cell[0].result()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 416, in run
    await asyncio.gather(*nannies)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 528, in start
    msg = await self._wait_until_connected(uid)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 643, in _wait_until_connected
    raise msg
TypeError: exceptions must derive from BaseException
rjurney picture rjurney on 10 Jun 2020

The comment above was then raised in another issue. It turned out to be an installation issue with msgpack.

mrocklin picture mrocklin on 10 Jun 2020

FYI: I had to do pip uninstall msgpack-python and pip install msgpack to get this to work for me, per the discussion on upgrading to 1.0 here.

richardzhu picture richardzhu on 8 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lostmygithubaccount picture lostmygithubaccount  路  4Comments
fjetter picture fjetter  路  6Comments
mrocklin picture mrocklin  路  4Comments
wmlba picture wmlba  路  4Comments
DPeterK picture DPeterK  路  3Comments