Handson-ml: I m not understanding some lines of code. Sir if you could explain these 3 lines elaborately, it would be of great help... Cheers.

[Akash0305]

for train_index, test_index in split.split(housing, housing["income_cat"]):
        strat_train_set = housing.loc[train_index]
        strat_test_set = housing.loc[test_index]

page no. 52

Thank you in advance.

[ageron]

Hi @Akash0305 ,

Good question. In this section, we are assuming that we know the income category is very important feature for predicting the housing prices. So we want to make sure our training set and our test set contain exactly the right proportion of each income category. For example, if the housing dataset contains 35.05% instances of income_cat equal to 3, then we want the training set and the test set to have very close to 35.05% instances of that income_cat. If we just pick 80% of the instances randomly and put them in the training set, and the remaining 20% go in the test set, then by pure chance the exact proportion of income_cat 3 may be slightly off (e.g., 35.85%). We want to get rid of this sampling noise.

Similarly, suppose you toss a coin 10 times, you might get 6 heads and 4 tails, even though the real probability of heads is 50%, not 60%. People who run surveys go to great lengths to reduce the sampling noise. One of the strategies is called stratified sampling where they estimate the actual proportion of some variable of interest in the population (e.g., proportion of female), because it is believed to be an important factor in whatever the survey is about, and then they make sure their sample contains the same proportion. For example, if they find that 51% of all people in the population are female, and they want to ask questions to 1,000 people, then they will make sure there are 510 female and 490 male. They won't just leave the proportion to chance, because if they are unlucky they will end up with (say) 480 female and 520 male (by pure chance), and their sample will be biased towards men's opinions.

Coming back to the code, split is an instance of the StratifiedShuffleSplit class. This class is designed to split a dataset while preserving some proportions. In this case, we are asking it to split the housing dataset randomly, while ensuring that the proportions of each income_cat are as respected as closely as possible.

The train_index and test_index arrays contain the indices of the instances that were selected by the split instance. So when we write housing.loc[train_index], we get back all the instances selected by the splitter for the training set. Similarly, housing.loc[test_index] returns all the instances selected by the splitter for the test set.

A common way to use the StratifiedShuffleSplit class is to ensure that the training set and the test set contain the same proportion of each class (in a classification problem). In this case, you would write something like for train_index, test_index in split.split(X_train, y_train).

Hope this helps!

[Akash0305]

Thanks for replying . It was helpful... But my qs specifically is how does split() works ? What it returns ? Why looping is done ?

Does subsetting done in every iteration ? How many iterations ?

Strat_train_set and strat_test_set are dataframes, right ?

[ageron]

My pleasure @Akash0305!
Any splitter's split() method returns an iterator over various train/test splits of the same dataset. Having multiple different splits is useful for cross-validation, but in our case we just need one split, so we set n_splits=1 when creating the StratifiedShuffleSplit. This means the loop will only run once.
The iterator returns two arrays of indices: one for the training set, and the other for the test set.
Since housing is a DataFrame, computing housing.loc[train_index] will also be a DataFrame.

Here's a simple example:

import numpy as np
from sklearn.model_selection import StratifiedShuffleSplit
split = StratifiedShuffleSplit(n_splits=3)
X = np.arange(20) * 10.
y = (X < 100).astype(np.int32)
print("X =", X)
print("y =", y)
for train_index, test_index in split.split(X, y):
    print(train_index, test_index)

This outputs:

X = [  0.  10.  20.  30.  40.  50.  60.  70.  80.  90. 100. 110. 120. 130.
 140. 150. 160. 170. 180. 190.]
y = [1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0]
[17 19 16  0  8 14  9 11  4 12 15  7  6  2 10  5 13  1] [18  3]
[ 8  2 10 19  7 12 14  3  4 11  6 18  5  9 17 13  0 16] [ 1 15]
[15  6  1  0 13 19  8 12 11  7  2  3 17 10 18  9 16  4] [ 5 14]

As you can see, the splitter provides the train & test indices for 3 train/test splits. Each split defines a train set with 18 instances and a test set with 2 instances (that's a 90%/10% split). Since y has ten 0s and ten 1s, the splitter tries to preserve this 50/50 proportion in both the train set and the test set. That's why you will find exactly 9 indices lower than 10 (<=> X < 100), and 9 indices greater than or equal to 10 (<=> X ≥ 100) in the 3 training sets, and one of each in the test set.

Hope this helps.

[Akash0305]

Ageron Sir, I m deeply touched by your concern to help me... I have never came across a teacher like you.. wishing you very best for your life... Best book that I have read till date !

[ageron]

Hi @Akash0305 , I'm really happy that you found my answers useful, and I thank you for your kind words! I hope you'll build amazing things with ML! :)