Probability: Incompatibility with cloudpickle==1.5.0

[ltetrel]

Hi all,

Due to a new update, it is not possible to import tensorflow_probability anymore.
Using cloudpickle <= 1.4.1 fixed the issue

>>> import tensorflow_probability as tfp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_probability/__init__.py", line 76, in <module>
    from tensorflow_probability.python import *  # pylint: disable=wildcard-import
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_probability/python/__init__.py", line 23, in <module>
    from tensorflow_probability.python import distributions
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_probability/python/distributions/__init__.py", line 88, in <module>
    from tensorflow_probability.python.distributions.pixel_cnn import PixelCNN
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_probability/python/distributions/pixel_cnn.py", line 37, in <module>
    from tensorflow_probability.python.layers import weight_norm
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_probability/python/layers/__init__.py", line 31, in <module>
    from tensorflow_probability.python.layers.distribution_layer import CategoricalMixtureOfOneHotCategorical
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_probability/python/layers/distribution_layer.py", line 28, in <module>
    from cloudpickle.cloudpickle import CloudPickler
ImportError: cannot import name 'CloudPickler'

[mli]

+1 got the same issue

[terrytangyuan]

@jburnim Is https://github.com/tensorflow/probability/commit/5cc832b9d28ed5562961385a3b30ad242d76aac1 a temporary workaround for this or some other issues? Is there any particular reason to pin this specific version of cloudpickle?

[matthewfeickert]

Hi. I opened up an Issue on cloudpickle for this, but we're also observing this in pyhf for the same reasons.

[jburnim]

Thank you for the report, @ltetrel !

It looks like we pinned the CloudPickle dependency to 1.3 in 5cc832b9d28ed5562961385a3b30ad242d76aac1 because of compatibility issues with CloudPickle and some versions of Python 3.5 . (It also looks like CloudPickle has since fixed these issues in https://github.com/cloudpipe/cloudpickle/pull/359 and https://github.com/cloudpipe/cloudpickle/pull/361 .)

As a quick fix, we are considering a TFP 0.10.1 release that is just TFP 0.10.0 but requiring CloudPickle 1.3. (If this would cause problems for you -- e.g., you're using TFP 0.10 and a higher version of CloudPickle -- please comment on this issue to let us know.)

We are also investigating further and taking a look at the fix https://github.com/tensorflow/probability/pull/993 from @matthewfeickert .

[terrytangyuan]

@jburnim Sounds great. We have no issue requiring that specific version. Thanks for the prompt response!

[matthewfeickert]

As a quick fix, we are considering a TFP 0.10.1 release that is just TFP 0.10.0 but requiring CloudPickle 1.3

For the library I work on (pyhf) this would work for the near term. cc @lukasheinrich @kratsg

Though if is possible to have releases that don't explicitly pin dependencies to a single version number I think that's nicer.

We are also investigating further and taking a look at the fix #993

Cool. Let me know if there is anything you need me to iterate on. I haven't taken the time to debug what the one test that is failing in CI is due to (given that all of CI fails at the moment).

[emilyfertig]

Update: We've now released TFP 0.10.1, which pins the CloudPickle version to 1.3, and are still looking into #993 .

[matthewfeickert]

Thank you for fixing this @jburnim! :bow:

[StevenSong]

this is an issue again with the latest version of tensorflow 2.3.1 (security patch from 4 days ago) which has cloudpickle dependency >= 1.5.0 - meanwhile tensorflow-probability 0.11.0 still has cloudpickle == 1.3.0

[matthewfeickert]

this is an issue again with the latest version of tensorflow 2.3.1 (security patch from 4 days ago) which has cloudpickle dependency >= 1.5.0 - meanwhile tensorflow-probability 0.11.0 still has cloudpickle == 1.3.0

The good news is that this was already resolved in TFP https://github.com/tensorflow/probability/commit/7601ef6d6aee014a5566937858974177e4f53122. So the next TFP release _should_ have this all taken care of. I'm not sure what the release schedule for TFP is though.

[csuter]

@matthewfeickert we generally build a new stable release whenever TF does, since in general we end up depending on new TF features in between their (and hence our) stable releases. We could increase our (TFP's) release cadence, so long as we either a) don't have such deps on not-yet-in-stable-TF features, or b) can easily hack around such issues on the release branch.

[matthewfeickert]

Thanks for that info @csuter. :+1: I wasn't meaning to complain about not knowing (it isn't super important to me and I know full well that having people ask about release schedules can be a tiresome discussion point), but I do appreciate you offering up this information here as that probably helps people (though I'm sure had I searched harder I would have already found this information in another Issue).

[csuter]

Definitely didn't detect any complaint! And I could talk release schedules all day! 😁

I think we (TFP) could probably do a bit better at communicating these processes, so folks don't have to go digging in Issues to find the info. Then again Google is a really good search engine, so maybe it's fine to have these bits buried in here 😅

[csuter]

Quick update (h/t to @brianwa84 for pointing out to me the actual context here, which I overlooked) -- TFP should actually release a patch to go with the TF 2.3.1 patch here. We'll look into it ASAP.

[csuter]

TFP 0.11.1 is up on pypi now, and should work fine with TF 2.3.1 and newer cloudpickles. Please let us know if you run into further issues!