Arviz: error with pymc3.compareplot

[drbenvincent]

When trying to plot results of model comparison with PyMC3's pm.compareplot I get an error...

KeyError                                  Traceback (most recent call last)
<ipython-input-26-a737fed9e2d4> in <module>
----> 1 ax = pm.compareplot(df_comp_WAIC)

/anaconda3/lib/python3.7/site-packages/pymc3/plots/__init__.py in compareplot(*args, **kwargs)
     81     else:
     82         args[0] = comp_df
---> 83     return az.plot_compare(*args, **kwargs)
     84 
     85 from .posteriorplot import plot_posterior_predictive_glm

/anaconda3/lib/python3.7/site-packages/arviz/plots/compareplot.py in plot_compare(comp_df, insample_dev, plot_standard_error, plot_ic_diff, order_by_rank, figsize, textsize, plot_kwargs, ax)
    107 
    108     if order_by_rank:
--> 109         comp_df.sort_values(by="rank", inplace=True)
    110 
    111     if plot_ic_diff:

/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py in sort_values(self, by, axis, ascending, inplace, kind, na_position)
   5006 
   5007             by = by[0]
-> 5008             k = self._get_label_or_level_values(by, axis=axis)
   5009 
   5010             if isinstance(ascending, (tuple, list)):

/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py in _get_label_or_level_values(self, key, axis)
   1772             values = self.axes[axis].get_level_values(key)._values
   1773         else:
-> 1774             raise KeyError(key)
   1775 
   1776         # Check for duplicates

KeyError: 'rank'

Any empty plot figure is returned, but the expected compare plot figure is not produced.

Am using arviz 0.5.1

[aloctavodia]

Thanks for reporting this issue. As a workaround could you see if using az.compare() and az.plot_compare() work as expected.

[ahartikainen]

Hi, thanks for the bug report.

Does this work?

import arviz as az
idata = az.from_pymc3(trace)
idata2 = az.from_pymc3(trace2)
comp_df = az.compare({'model_1' : idata, 'model_2' : idata2})
az.plot_compare(comp_df)

edit. @aloctavodia was faster :)
edit2. Missing 3

[drbenvincent]

Responding to @aloctavodia... Thanks for the quick reply. Yes. That does work, although the model names are not reflected in the y-axis labels (see pic).

@ahartikainen Tried your code snippet, but AttributeError: module 'arviz' has no attribute 'from_pymc'

[aloctavodia]

try something like this

model_dict = dict(zip(['model_0_name', 'model_1_name''], traces))
comp = az.compare(model_dict)

[ahartikainen]

Oh, there was a missing 3

[aloctavodia]

The problem I think is that the current stable version of PyMC3 uses arviz by default for the plots but not for other functions, creating potential inconsistencies like the one you reported. This have been solved on PyMC3's master, and it will be solved for the next release (that hopefully is almost here).

My recommendation is to explicitly use ArviZ functions, and keep the pm.() alias just as backward compatible solution.

[drbenvincent]

@aloctavodia That works...

model_dict = dict(zip(['model_0_name', 'model_1_name'], 
                      [model_0.posterior_samples, model_1.posterior_samples]))
comp = az.compare(model_dict)
az.plot_compare(comp)

So maybe it's just something problematic in how pm.compareplot() calls arviz ?
Either way, this solves my immediate problem. Thanks!

[aloctavodia]

Yeah, see my previous comment. Sorry for the inconvenience and glad you are back on track!

[drbenvincent]

Does this work?

import arviz as az
idata = az.from_pymc3(trace)
idata2 = az.from_pymc3(trace2)
comp_df = az.compare({'model_1' : idata, 'model_2' : idata2})
az.plot_compare(comp_df)

edit. @aloctavodia was faster :)
edit2. Missing 3

This did not work...

h_free_trace = az.from_pymc3(h_free.posterior_samples)
mr_free_trace = az.from_pymc3(mr_free.posterior_samples)
comp_df = az.compare({'hyperbolic' : h_free_trace, 'modified Rachlin' : mr_free})
az.plot_compare(comp_df)

with error...

ValueError                                Traceback (most recent call last)
<ipython-input-40-4ac9fd4fe01c> in <module>
      1 h_free_trace = az.from_pymc3(h_free.posterior_samples)
      2 mr_free_trace = az.from_pymc3(mr_free.posterior_samples)
----> 3 comp_df = az.compare({'hyperbolic' : h_free_trace, 'modified Rachlin' : mr_free})
      4 az.plot_compare(comp_df)

/anaconda3/lib/python3.7/site-packages/arviz/stats/stats.py in compare(dataset_dict, ic, method, b_samples, alpha, seed, scale)
    196     for name, dataset in dataset_dict.items():
    197         names.append(name)
--> 198         ics = ics.append([ic_func(dataset, pointwise=True, scale=scale)])
    199     ics.index = names
    200     ics.sort_values(by=ic, inplace=True, ascending=ascending)

/anaconda3/lib/python3.7/site-packages/arviz/stats/stats.py in waic(data, pointwise, scale)
   1085     `deviance` scale, the `log` (and `negative_log`) correspond to ``elpd`` (and ``-elpd``)
   1086     """
-> 1087     inference_data = convert_to_inference_data(data)
   1088     for group in ("sample_stats",):
   1089         if not hasattr(inference_data, group):

/anaconda3/lib/python3.7/site-packages/arviz/data/converters.py in convert_to_inference_data(obj, group, coords, dims, **kwargs)
    118         raise ValueError(
    119             "Can only convert {} to InferenceData, not {}".format(
--> 120                 ", ".join(allowable_types), obj.__class__.__name__
    121             )
    122         )

ValueError: Can only convert xarray dataset, dict, netcdf file, numpy array, pystan fit, pymc3 trace, emcee fit, pyro mcmc fit, numpyro mcmc fit, cmdstan fit csv, cmdstanpy fit to InferenceData, not ModifiedRachlinFreeSlope

[ahartikainen]

Thanks, yeah I think we have a bug in there (then @aloctavodia code should work (or moving the dict from az.compare to az.plot_compare)

[OriolAbril]

@drbenvincent could the issue be in this line? Looks like a typo:

comp_df = az.compare({'hyperbolic' : h_free_trace, 'modified Rachlin' : mr_free})

Have you tried mr_free_trace? What is failing is actually convert_to_inference_data even though you have called from_pymc3 so tge objects should already be of inference data type.