Pandas-profiling: TypeError: unhashable type: 'list' for categorical variable

[DerpMind]

Describe the bug

Hi!
Since the new release the ProfileReport function crashes when I include DataFrame columns of a categorical type.

To Reproduce

import numpy as np
import pandas as pd
from pandas_profiling import ProfileReport

df = pd.DataFrame(
np.random.rand(100, 5),
columns=['a', 'b', 'c', 'd', 'e']
)
# make one column categorical
df["a"] = df["a"].multiply(5).astype("int").astype("category")

profile = ProfileReport(df, title='Pandas Profiling Report', html={'style':{'full_width':True}})

Version information:

• _Python version_: 3.7
• _Environment_: JupyterNotebook

Error message

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-f6fabd598922> in <module>
----> 1 profile = ProfileReport(df, title='Pandas Profiling Report', html={'style':{'full_width':True}})

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas_profiling\__init__.py in __init__(self, df, minimal, config_file, **kwargs)
     67 
     68         # Get dataset statistics
---> 69         description_set = describe_df(df)
     70 
     71         # Build report structure

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas_profiling\model\describe.py in describe(df)
    524         with multiprocessing.pool.ThreadPool(pool_size) as executor:
    525             series_description = {}
--> 526             results = executor.starmap(multiprocess_1d, df.iteritems())
    527             for col, description in results:
    528                 series_description[col] = description

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\multiprocessing\pool.py in starmap(self, func, iterable, chunksize)
    274         `func` and (a, b) becomes func(a, b).
    275         '''
--> 276         return self._map_async(func, iterable, starmapstar, chunksize).get()
    277 
    278     def starmap_async(self, func, iterable, chunksize=None, callback=None,

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\multiprocessing\pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658 
    659     def _set(self, i, obj):

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\multiprocessing\pool.py in worker(inqueue, outqueue, initializer, initargs, maxtasks, wrap_exception)
    119         job, i, func, args, kwds = task
    120         try:
--> 121             result = (True, func(*args, **kwds))
    122         except Exception as e:
    123             if wrap_exception and func is not _helper_reraises_exception:

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\multiprocessing\pool.py in starmapstar(args)
     45 
     46 def starmapstar(args):
---> 47     return list(itertools.starmap(args[0], args[1]))
     48 
     49 #

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas_profiling\model\describe.py in multiprocess_1d(column, series)
    376         A tuple with column and the series description.
    377     """
--> 378     return column, describe_1d(series)
    379 
    380 

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas_profiling\model\describe.py in describe_1d(series)
    335 
    336     # Replace infinite values with NaNs to avoid issues with histograms later.
--> 337     series.replace(to_replace=[np.inf, np.NINF, np.PINF], value=np.nan, inplace=True)
    338 
    339     # Infer variable types

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas\core\series.py in replace(self, to_replace, value, inplace, limit, regex, method)
   4177             limit=limit,
   4178             regex=regex,
-> 4179             method=method,
   4180         )
   4181 

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas\core\generic.py in replace(self, to_replace, value, inplace, limit, regex, method)
   6701                 else:  # [NA, ''] -> 0
   6702                     new_data = self._data.replace(
-> 6703                         to_replace=to_replace, value=value, inplace=inplace, regex=regex
   6704                     )
   6705             elif to_replace is None:

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas\core\internals\managers.py in replace(self, value, **kwargs)
    587     def replace(self, value, **kwargs):
    588         assert np.ndim(value) == 0, value
--> 589         return self.apply("replace", value=value, **kwargs)
    590 
    591     def replace_list(self, src_list, dest_list, inplace=False, regex=False):

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas\core\internals\managers.py in apply(self, f, filter, **kwargs)
    440                 applied = b.apply(f, **kwargs)
    441             else:
--> 442                 applied = getattr(b, f)(**kwargs)
    443             result_blocks = _extend_blocks(applied, result_blocks)
    444 

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas\core\internals\blocks.py in replace(self, to_replace, value, inplace, filter, regex, convert)
   2943         result = self if inplace else self.copy()
   2944         if filter is None:  # replace was called on a series
-> 2945             result.values.replace(to_replace, value, inplace=True)
   2946             if convert:
   2947                 return result.convert(numeric=False, copy=not inplace)

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas\core\arrays\categorical.py in replace(self, to_replace, value, inplace)
   2440         inplace = validate_bool_kwarg(inplace, "inplace")
   2441         cat = self if inplace else self.copy()
-> 2442         if to_replace in cat.categories:
   2443             if isna(value):
   2444                 cat.remove_categories(to_replace, inplace=True)

~\AppData\Local\Continuum\anaconda3\envs\UDB\lib\site-packages\pandas\core\indexes\numeric.py in __contains__(self, key)
    230         Check if key is a float and has a decimal. If it has, return False.
    231         """
--> 232         hash(key)
    233         try:
    234             if is_float(key) and int(key) != key:

TypeError: unhashable type: 'list'

[sbrugman]

The code below does not reproduce the error. Let me know if how this test differs from your situation if the error has not been resolved in the latest version.

"""
Test for issue 353:
https://github.com/pandas-profiling/pandas-profiling/issues/353
"""
import numpy as np
import pandas as pd

from pandas_profiling import ProfileReport


def test_issue353():
    df = pd.DataFrame(
        np.random.rand(100, 5),
        columns=['a', 'b', 'c', 'd', 'e']
    )
    # make one column categorical
    df["a"] = df["a"].multiply(5).astype("int").astype("category")

    profile = ProfileReport(df, title='Pandas Profiling Report', html={'style': {'full_width': True}})

[IloBe]

Hi all,
FYI
today, using jupyter notebook with Python 3.7, I have the same error message "TypeError: unhashable type: 'list'!" after installing statsmodels library V0.11.0;
the commandline conda update installation changed other libraries as well, so, for the prediction project with UCI bike sharing Washington DC dataset (using day.csv as dataframe having converted some columns to categorical dtypes and calling profile_report() for it), the following setting has been available:
h5py=>2.9.0
matplotlib=>3.1.2
seaborn=>0.9.0
numpy=>1.18.1
pandas=>1.0.0
pandas_profiling=>2.4.0
scipy=>1.3.2
scikit-learn=>0.21.2
ipykernel=>5.1.1
ipywidgets=>7.5.0
sqlalchemy=>1.3.13
statsmodels=>0.11.0
Everything is implemented in projects environment. It seems as if the the libraries are inconsistent.

The function worked fine before installation of statsmodels, using lower versions of

• pandas: V0.24.2
• pandas_profiling: V2.3.0 (with this version error appeared as well, after installation)
• numpy: V1.16.4
• scipy: V1.2.1

Working again with the older pandas version 0.24.2, everything was fine again.
In other words root-cause has been the new pandas version 1.0.0.

[sbrugman]

@IloBe Thank you for figuring out that pandas-profiling is currently incompatible with pandas 1.0.0.

The next version should be compatible with pandas 1.0.0 or fix the pandas version if that is highly nontrivial. The current workaround then is to install the lastest pre-v1 version of pandas:

pip install "pandas<1.0.0"

[DerpMind]

@sbrugman another workaround is to set categories to strings. This, of course, can be quite time and memory consuming for larger datasets.

[mw0]

As a reference to others arriving here, pandas-profiling v2.6.0 does not work with pandas v1.0.1, but does work for pandas v1.0.3.