Handson-ml: Chapter 2 - TypeError: object of type 'CategoricalDtype' has no len()

[Rahul954]

While running below code

from sklearn.model_selection import StratifiedShuffleSplit
split=StratifiedShuffleSplit(n_splits=1,test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing["income_cat"]):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]

it gives below error

if hasattr(array, "dtypes") and len(array.dtypes):
TypeError: object of type 'CategoricalDtype' has no len()

[Zram]

I have the same error =(

[Rahul954]

Hello Zram,

If you come across the solution. Please post it.

Else we can wait for someone who has done it successfully to post it.

I believe there will be hundreds who would have done it correctly.

I will also try hard on it.

Regards,
Rahul

[ageron]

Hi @Rahul954 ,
I tried to reproduce the problem, but I couldn't, everything works fine for me. My guess is that it's a problem with the version of Pandas or NumPy you are using. Could you please upgrade NumPy and Pandas to the latest version and try again?
Also, could you please paste the full stacktrace, not just the two lines, it would help understand where the bug comes from.
Thanks!

[Rahul954]

Hello Aurelien,

I am feeling excited that I got a reply from you. I really appreciate your work from the bottom of my heart that you took out some time from your precious time.

I have fixed it. These are two codes:

___________Not Working _______________________________________________

housing["income_cat"] = pd.cut(housing["median_income"],
                               bins=[0., 1.5, 3.0, 4.5, 6., np.inf],
                               labels=[1, 2, 3, 4, 5])

____________Working ___________________________________________________

housing["income_cat"]=np.ceil(housing["median_income"]/1.5)
housing["income_cat"].where(housing["income_cat"] < 5,5.0,inplace=True)

The first code which is not working is on github. Then I read the book again and made modifications according to the book which worked perfectly.

I will really appreciate it that since you are extending your both arms to help learners like us to post the code which is working perfectly for you for all the chapters. It will really help a lot to save time. Though after making mistakes I learn some new things.

Regards,
Rahul Rathore

[Zram]

housing["income_cat"]=np.ceil(housing["median_income"]/1.5)
housing["income_cat"].where(housing["income_cat"] < 5,5.0,inplace=True)

housing["income_cat"] = pd.cut(housing["median_income"],

Wow Thanx mate it works!

[ageron]

Hi @Rahul954 ,

Thanks for your feedback, and for your very kind words! :) I'm glad you found the solution to the problem.

I recently replaced the code to compute the income_cat column, it worked for me, but I'll double check and I'll add a comment. Perhaps it does not work with all versions of Pandas? Did you upgrade to the latest version of Pandas? If not, what version are you using?

Also, when I asked for the full stacktrace, I meant the exception stacktrace, not the full code (thanks @Zram for identifying the delta, I wasn't sure what the difference was between the non-working code and the working code). For example, if I define a simple function like this:

def f(x):
    return 10 / x

Then a stacktrace would look like this:

>>> f(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in f
ZeroDivisionError: division by zero

Note that in github (Markdown format), you can surround code examples with ``` . You can also add the language at the beginning to get syntax highlighting:

```python
your code
goes here
```

Hope this helps,
Aurélien

[ageron]

I can confirm that everything works fine for me using Pandas 0.24.1 and Scikit-Learn 0.20.3.
I also tried with Pandas 0.22.2 and 0.23.0 and Scikit-Learn 0.18 and 0.19.0, and everything worked fine.
Could you please tell me which versions of these packages you have? Just type this:

import sklearn
import pandas as pd

print(sklearn.__version__)
print(pd.__version__)

Here is the test I ran:

import pandas as pd
import numpy as np
housing = pd.DataFrame({"median_income": np.random.rand(100)*10})
housing["income_cat"] = pd.cut(housing["median_income"],
                               bins=[0., 1.5, 3.0, 4.5, 6., np.inf],
                               labels=[1, 2, 3, 4, 5])

from sklearn.model_selection import StratifiedShuffleSplit
split=StratifiedShuffleSplit(n_splits=1,test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing["income_cat"]):
   strat_train_set = housing.loc[train_index]
   strat_test_set = housing.loc[test_index]

[ageron]

Perhaps it's a Python 2 issue? Which version of Python are you using? Please run this command:

import sys

print(sys.version)

If you are using Python 2, then you should really migrate to Python 3 (if possible 3.7, and at least 3.5). Python 2 is going do end its life on January 1st 2020, so it's best to migrate to Python 3 as soon as possible.

[Zram]

Perhaps it's a Python 2 issue? Which version of Python are you using? Please run this command:

import sys

print(sys.version)

If you are using Python 2, then you should really migrate to Python 3 (if possible 3.7, and at least 3.5). Python 2 is going do end its life on January 1st 2020, so it's best to migrate to Python 3 as soon as possible.

I'm using Python 3.7.1
The scikit-learn version is 0.20.1.
The pandas version is 0.23.4.
The numpy version is 1.15.4.

Error looks like

---

TypeError                                 Traceback (most recent call last)
<ipython-input-38-43d32e4685ef> in <module>
      5 
      6 split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
----> 7 for train_index, test_index in split.split(housing, housing["income_cat"]):
      8     strat_train_set = housing.loc[train_index]
      9     strat_test_set = housing.loc[test_index]

~\Anaconda3\lib\site-packages\sklearn\model_selection\_split.py in split(self, X, y, groups)
   1771         to an integer.
   1772         """
-> 1773         y = check_array(y, ensure_2d=False, dtype=None)
   1774         return super(StratifiedShuffleSplit, self).split(X, y, groups)
   1775 

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    478     # DataFrame), and store them. If not, store None.
    479     dtypes_orig = None
--> 480     if hasattr(array, "dtypes") and len(array.dtypes):
    481         dtypes_orig = np.array(array.dtypes)
    482 

TypeError: object of type 'CategoricalDtype' has no len()

[Rahul954]

Hello @ageron ,

I am using below version.

print(sys.version)
3.7.1 (default, Dec 14 2018, 19:28:38)
[GCC 7.3.0]

Python3.7

pd.__version__
Out[6]: '0.23.4'

np.version.version
Out[9]: '1.15.4'

print(sklearn.__version__)
0.20.1

Regards,
Rahul

[Zram]

I just tested the code with scikit-learn version 0.19.0, and it works. The problem seems to be in the new version of scikit-learn

[ageron]

Aha! Indeed, @Zram , I managed to reproduce the bug with Scikit-Learn 0.20.1, and only with that version. I tried 0.20.0, 0.20.2, 0.20.3 and 0.19.0, they all worked fine.

So the bottom line is: you can either use the pd.where() solution like in the book, or you can use the pd.cut() version as long as you are not using Scikit-Learn 0.20.1, it has a bug.

To upgrade:

pip3 install -U scikit-learn

Thanks for your help in fixing this issue! Please close this issue if the problem is fixed.

[Rahul954]

Hello,

Thanks for your help.

I am closing it.

Regards,
Rahul

[choym92]

Hello Aurelien,

Love your work so much!! I am digging through it. Thanks Rahul for this post as well :)
I came across same exact issue and was discouraged moving on from CH2 due to not being able to debug.

As above, scikit 20.1 -> 20.3 solved it.

Appreciate both of you.