Distributed: Cannot persist dask.dataframes

Created on 8 Feb 2021 · 15Comments · Source: dask/distributed

What happened:

DataFrame collections like dask dataframes or dask-cudf cannot be persisted after release 2021.2.0. @wphicks triaged that after the merge of this PR the issue started to present: https://github.com/dask/distributed/pull/4406

What you expected to happen:

Persist to work (see reproducer)

Minimal Complete Verifiable Example:

import dask.dataframe as dd
import numpy as np
import pandas as pd

from dask.distributed import Client
from dask.distributed import LocalCluster


def persist_across_workers(client, objects, workers=None):
    if workers is None:
        # Default to all workers
        workers = client.has_what().keys()
    return client.persist(objects, workers={o: workers for o in objects})


if __name__ == "__main__":

    cluster = LocalCluster()
    client = Client(cluster)

    X = np.ones((10000, 20))

    X_df = pd.DataFrame(X)
    X_dist = dd.from_pandas(X_df, npartitions=2)

    X_f = persist_across_workers(client, X_dist)

Output:

distributed.protocol.core - CRITICAL - Failed to Serialize
Traceback (most recent call last):
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/protocol/core.py", line 39, in dumps
    small_header, small_payload = dumps_msgpack(msg, **compress_opts)
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/protocol/core.py", line 184, in dumps_msgpack
    payload = msgpack.dumps(msg, default=msgpack_encode_default, use_bin_type=True)
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'dict_keys' object
distributed.comm.utils - ERROR - can not serialize 'dict_keys' object
Traceback (most recent call last):
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/comm/utils.py", line 32, in _to_frames
    protocol.dumps(
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/protocol/core.py", line 39, in dumps
    small_header, small_payload = dumps_msgpack(msg, **compress_opts)
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/protocol/core.py", line 184, in dumps_msgpack
    payload = msgpack.dumps(msg, default=msgpack_encode_default, use_bin_type=True)
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'dict_keys' object
distributed.batched - ERROR - Error in batched write
Traceback (most recent call last):
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/batched.py", line 93, in _background_send
    nbytes = yield self.comm.write(
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/comm/tcp.py", line 230, in write
    frames = await to_frames(
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/comm/utils.py", line 52, in to_frames
    return _to_frames()
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/comm/utils.py", line 32, in _to_frames
    protocol.dumps(
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/protocol/core.py", line 39, in dumps
    small_header, small_payload = dumps_msgpack(msg, **compress_opts)
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/distributed/protocol/core.py", line 184, in dumps_msgpack
    payload = msgpack.dumps(msg, default=msgpack_encode_default, use_bin_type=True)
  File "/home/galahad/miniconda3/envs/ns0208/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'dict_keys' object

Environment:

Dask version: 2021.2.0
Distributed version: 2021.2.0 from conda and built from master after https://github.com/dask/distributed/pull/4406
Python version: 3.7 and 3.8
Operating System: Linux / AMD64
Install method (conda, pip, source): conda and from source

cc @jakirkham @pentschev @madsbk @wphicks

Source

dantegd

Most helpful comment

@trivialfis you might be interested in this -- I think xgboost maybe does similar things ?

quasiben on 9 Feb 2021

👍3

All 15 comments

cc @ian-r-rose

jakirkham on 8 Feb 2021

Thanks for the nice example @dantegd. In https://github.com/dask/distributed/pull/4406 we removed the ability to pass Dask collections to priority=, workers= keywords as they were broken in many cases. Instead you can now use Dask's new dask.annotate machinery in these cases, which should hopefully be more robust.

For the above example, this means changing

    client.persist(objects, workers={o: workers for o in objects})

    with dask.annotate(workers=set(workers)):
        return client.persist(objects)

Details:

import dask.dataframe as dd
import numpy as np
import pandas as pd

import dask
from dask.distributed import Client
from dask.distributed import LocalCluster


def persist_across_workers(client, objects, workers=None):
    if workers is None:
        # Default to all workers
        workers = client.has_what().keys()
    with dask.annotate(workers=set(workers)):
        return client.persist(objects)


if __name__ == "__main__":

    cluster = LocalCluster()
    client = Client(cluster)

    X = np.ones((10000, 20))

    X_df = pd.DataFrame(X)
    X_dist = dd.from_pandas(X_df, npartitions=2)

    X_f = persist_across_workers(client, X_dist)

jrbourbeau on 8 Feb 2021

❤3

Should we improve the error message here then? Maybe drop this flag? Something else? As it is, this is not obviously unsupported based on the error message given (MsgPack unable to serialize something)

jakirkham on 9 Feb 2021

Yes, that's right @jrbourbeau .

@jakirkham I agree that an improved error message would be helpful. At the very least, we could do a better job ensuring that the shape of the priority/workers/etc makes sense (i.e., iterable for workers, number for priority, error if dict-of-collections)

ian-r-rose on 9 Feb 2021

👍3

@trivialfis you might be interested in this -- I think xgboost maybe does similar things ?

quasiben on 9 Feb 2021

👍3

cc @hcho3 (from xgboost as well)

jakirkham on 9 Feb 2021

I think xgboost maybe does similar things ?

Good to know. Briefly looking through xgboost, they specify individual worker addresses in workers= without any Dask collections, so that shouldn't be impacted by any of the recent changes

jrbourbeau on 9 Feb 2021

@hcho3 is out for a bit, but it sounds like we're good on the xgboost front? @trivialfis, please feel free to hit me up if you need extra hands/eyes on that.

wphicks on 9 Feb 2021

Good to know. Briefly looking through xgboost, they specify individual worker addresses in workers= without any Dask collections, so that shouldn't be impacted by any of the recent changes

Thanks for checking @jrbourbeau

quasiben on 9 Feb 2021

Just checking in here, was there anything else we still need to do or is this ok to close now?

jakirkham on 18 Mar 2021

It was proposed we could improve the error message which is raised (https://github.com/dask/distributed/issues/4492#issuecomment-775523990). But otherwise I think things have already been fixed upstream in dask-cudf

jrbourbeau on 18 Mar 2021

I'm sorry for completely missing this thread until I tried to dig up some old emails...

Good to know. Briefly looking through xgboost, they specify individual worker addresses in workers= without any Dask collections, so that shouldn't be impacted by any of the recent changes

We do have a line that might be related https://github.com/dmlc/xgboost/blob/905fdd3e08d91077aada776346c7e49e4ff69334/python-package/xgboost/dask.py#L335 , copying it to here: