Arviz: to_netcdf error for InferenceData loaded with from_cmdstan

[teddygroves]

Describe the bug
The method to_netcdf fails on my computer for InferenceData objects generated using arviz.from_cmdstan, with this error:

TypeError: illegal data type for attribute b'diagnostic_file', must be one of dict_keys(['S1', 'i1', 'u1', 'i2', 'u2', 'i4', 'u4', 'i8', 'u8', 'f4', 'f8']), got O

To Reproduce
Here's a python script which produces the error on my computer:

import arviz
import cmdstanpy

STAN_PROGRAM = """
data {
  int<lower=0> N;
  int<lower=0,upper=1> y[N];
}
parameters {
  real<lower=0,upper=1> theta;
}
model {
  theta ~ beta(1,1);  // uniform prior on interval 0,1
  y ~ bernoulli(theta);
}
"""
DATA = {"N": 2, "y": [0, 1]}
STAN_FILE = "bern.stan"
NCDF_FILE = "bern.ncdf"

def main():
    print("arviz version: " + arviz.__version__)
    print("cmdstanpy version: " + cmdstanpy.__version__)
    print("cmdstan path: " + cmdstanpy.cmdstan_path())
    with open(STAN_FILE, "w") as f:
        f.write(STAN_PROGRAM)
    model = cmdstanpy.CmdStanModel(stan_file=STAN_FILE)
    mcmc = model.sample(data=DATA, chains=2)
    infd = arviz.from_cmdstan(mcmc.runset.csv_files)
    infd.to_netcdf(NCDF_FILE)

if __name__ == "__main__":
    main()

This gives the following output:

$ python netcdf_arviz_bad.py 
arviz version: 0.11.1
cmdstanpy version: 0.9.67
cmdstan path: /Users/tedgro/.cmdstanpy/cmdstan-2.26.0
INFO:cmdstanpy:compiling stan program, exe file: /Users/tedgro/Code/pyhack/bern
INFO:cmdstanpy:compiler options: stanc_options=None, cpp_options=None
INFO:cmdstanpy:compiled model file: /Users/tedgro/Code/pyhack/bern
INFO:cmdstanpy:start chain 1
INFO:cmdstanpy:start chain 2
INFO:cmdstanpy:finish chain 1
INFO:cmdstanpy:finish chain 2
/Users/tedgro/.pyenv/versions/3.7.2/lib/python3.7/site-packages/arviz/data/base.py:174: UserWarning: More chains (2) than draws (0). Passed array should have shape (chains, draws, *shape)
  UserWarning,
Traceback (most recent call last):
  File "netcdf_arviz_bad.py", line 33, in <module>
    main()
  File "netcdf_arviz_bad.py", line 30, in main
    infd.to_netcdf(NCDF_FILE)
  File "/Users/tedgro/.pyenv/versions/3.7.2/lib/python3.7/site-packages/arviz/data/inference_data.py", line 373, in to_netcdf
    data.to_netcdf(filename, mode=mode, group=group, **kwargs)
  File "/Users/tedgro/.pyenv/versions/3.7.2/lib/python3.7/site-packages/xarray/core/dataset.py", line 1567, in to_netcdf
    invalid_netcdf=invalid_netcdf,
  File "/Users/tedgro/.pyenv/versions/3.7.2/lib/python3.7/site-packages/xarray/backends/api.py", line 1082, in to_netcdf
    dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
  File "/Users/tedgro/.pyenv/versions/3.7.2/lib/python3.7/site-packages/xarray/backends/api.py", line 1128, in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)  File "/Users/tedgro/.pyenv/versions/3.7.2/lib/python3.7/site-packages/xarray/backends/common.py", line 252, in store
    self.set_attributes(attributes)
  File "/Users/tedgro/.pyenv/versions/3.7.2/lib/python3.7/site-packages/xarray/backends/common.py", line 269, in set_attributes
    self.set_attribute(k, v)
  File "/Users/tedgro/.pyenv/versions/3.7.2/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 430, in set_attribute
    self.ds.setncattr(key, value)
  File "netCDF4/_netCDF4.pyx", line 2883, in netCDF4._netCDF4.Dataset.setncattr
  File "netCDF4/_netCDF4.pyx", line 1652, in netCDF4._netCDF4._set_att
TypeError: illegal data type for attribute b'diagnostic_file', must be one of dict_keys(['S1', 'i1', 'u1', 'i2', 'u2', 'i4', 'u4', 'i8', 'u8', 'f4', 'f8']), got O
deleting tmpfiles dir: /var/folders/nw/nbg0t9fx0hvdhwlgwk2kb48wqkndw3/T/tmpg7047mzx
done

Expected behavior
When I use this line to load an InferenceData object

    infd = arviz.from_cmdstanpy(mcmc)

Everything seems to work as normal:

$ python netcdf_arviz_good.py 
arviz version: 0.11.1
cmdstanpy version: 0.9.67
cmdstan path: /Users/tedgro/.cmdstanpy/cmdstan-2.26.0
INFO:cmdstanpy:compiling stan program, exe file: /Users/tedgro/Code/pyhack/bern
INFO:cmdstanpy:compiler options: stanc_options=None, cpp_options=None
INFO:cmdstanpy:compiled model file: /Users/tedgro/Code/pyhack/bern
INFO:cmdstanpy:start chain 1
INFO:cmdstanpy:start chain 2
INFO:cmdstanpy:finish chain 1
INFO:cmdstanpy:finish chain 2
deleting tmpfiles dir: /var/folders/nw/nbg0t9fx0hvdhwlgwk2kb48wqkndw3/T/tmpssaboq8j
done

Additional context

• arviz version: 0.11.1
• cmdstanpy version: 0.9.67
• cmdstan version: 2.26.0
• python version: 3.7.2
• operating system: macos Mojave

[ahartikainen]

Hi, thanks for the info. I think there is missing encode("utf-8") somewhere.

[ahartikainen]

Nope, apparently there is a limitation what attrs can handle. (No None, no list of lists, no dictionaries?).

Easiest way to save to netCDF is to delete the attrs (if you don't need it)

infd.posterior.attrs = {}
infd.sample_stats.attrs = {}

cc. @OriolAbril

[OriolAbril]

I was thinking that we could add some attrs_to_str and attrs_from_str methods to inferencedata using json, nothing too fancy. Maybe there already is something like this in xarray :thinking: Ideally even a try except to catch this error and advise on how to solve it.

[ahartikainen]

Currently everything in cmdstan attrs is a string. (Temp fix)