Libertem: Executing the "basic.py" example fails

[uellue]

This doesn't occur with the equivalent RAW file, i.e. seems to be something specific to HDF5.

Error message:

(libertem) C:\Users\weber\Documents\LiberTEM\LiberTEM>python examples\basic.py
tornado.application - ERROR - Exception in Future <Future cancelled> after timeout
Traceback (most recent call last):
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\tornado\gen.py", line 970, in error_callback
    future.result()
concurrent.futures._base.CancelledError
Traceback (most recent call last):
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\base.py", line 643, in normalize_function
    return function_cache[func]
KeyError: <exception str() failed>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\base.py", line 643, in normalize_function
    return function_cache[func]
KeyError: <exception str() failed>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\copyreg.py", line 69, in _reduce_ex
    getstate = self.__getstate__
AttributeError: 'Shape' object has no attribute '__getstate__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\base.py", line 681, in _normalize_function
    return cloudpickle.dumps(func, protocol=0)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\cloudpickle\cloudpickle.py", line 931, in dumps
    cp.dump(obj)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\cloudpickle\cloudpickle.py", line 284, in dump
    return Pickler.dump(self, obj)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 409, in dump
    self.save(obj)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\cloudpickle\cloudpickle.py", line 703, in save_instancemethod
    self.save_reduce(types.MethodType, (obj.__func__, obj.__self__), obj=obj)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 610, in save_reduce
    save(args)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 751, in save_tuple
    save(element)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 835, in _batch_setitems
    save(v)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\pickle.py", line 496, in save
    rv = reduce(self.proto)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\copyreg.py", line 72, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "examples\basic.py", line 31, in <module>
    tileshape=(1, 8, 128, 128)
  File "c:\users\weber\documents\libertem\libertem\src\libertem\api.py", line 65, in load
    ds = self.executor.run_function(ds.initialize)
  File "c:\users\weber\documents\libertem\libertem\src\libertem\executor\dask.py", line 124, in run_function
    future = self.client.submit(functools.partial(fn, *args, **kwargs), priority=1)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\distributed\client.py", line 1246, in submit
    key = funcname(func) + '-' + tokenize(func, kwargs, *args)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\base.py", line 600, in tokenize
    return md5(str(tuple(map(normalize_token, args))).encode()).hexdigest()
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\utils.py", line 413, in __call__
    return meth(arg, *args, **kwargs)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\base.py", line 634, in normalize_object
    return normalize_function(o) if callable(o) else uuid.uuid4().hex
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\base.py", line 645, in normalize_function
    result = _normalize_function(func)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\base.py", line 671, in _normalize_function
    return (normalize_function(func.func), args, kws)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\base.py", line 645, in normalize_function
    result = _normalize_function(func)
  File "C:\Users\weber\AppData\Local\conda\conda\envs\libertem\lib\site-packages\dask\base.py", line 683, in _normalize_function
    return str(func)
  File "c:\users\weber\documents\libertem\libertem\src\libertem\io\dataset\hdf5.py", line 129, in __repr__
    return "<H5DataSet of %s shape=%s>" % (self.dtype, self.shape)
  File "c:\users\weber\documents\libertem\libertem\src\libertem\io\dataset\hdf5.py", line 82, in dtype
    raise RuntimeError("please call initialize")
RuntimeError: please call initialize

[sk1p]

The crucial parts of the error:

AttributeError: 'Shape' object has no attribute '__getstate__'
[...]
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

So, need to implement __getstate__ on classes to make them pickleable again.

[sk1p]

Huh, cloudpickle claims to use the latest pickle protocol which is supposed to support __slots__ without __getstate__... weird.

[uellue]

I've confirmed that I'm using cloudpickle 0.6.1, the latest version from pip.

[sk1p]

Ah, it's right in the traceback: return cloudpickle.dumps(func, protocol=0) - see the corresponding code in dask.base.

I didn't yet get to the root cause, but I think this was caused by something calling repr() on the H5DataSet before it was fully initialized. I committed a workaround for that.

Serialization of objects with __slots__ in cloudpickle seems to be broken anyways:

class Bar(object):
    __slots__ = ["a"]

    def __init__(self):
        self.a = 42

# works:
barpickle = pickle.dumps(Bar())
pickle.loads(barpickle)

# works:
barpickle = pickle.dumps(Bar())
cloudpickle.loads(barpickle)

# crashes: 
barpickle = cloudpickle.dumps(Bar())
cloudpickle.loads(barpickle)

# also crashes: 
barpickle = cloudpickle.dumps(Bar())
pickle.loads(barpickle)

Crashes raise an exception like this one:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-13-2febb7e9bc6e> in <module>()
      1 barpickle = cloudpickle.dumps(Bar())
----> 2 cloudpickle.loads(barpickle)

AttributeError: type object 'Bar' has no attribute 'a'

[sk1p]

Reported upstream: https://github.com/cloudpipe/cloudpickle/issues/225

[uellue]

@sk1p thank you for the fix! Since the basic.py execution doesn't fail anymore, should we close this issue and leave it be, or do we have loose ends somewhere that we should take care of?