Handson-ml: Question about code in Chapter 2

[ssilverac]

Hi Aurelien

Im loving the practicality of your book so far. very good examples and very informative.

I am having trouble understanding a block of code in chapter 2 and I was wondering if youd be able to walk me through what is going on in the code. I understand the basic idea of why its there but I dont understand what is actually happening.
Any insight you may be able to provide would be greatly appreciated

this is the code:

from sklearn.base import BaseEstimator, TransformerMixin

rooms_ix, bedrooms_ix, population_ix, household_ix = [
    list(housing.columns).index(col)
    for col in ("total_rooms", "total_bedrooms", "population", "households")]

class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
    def __init__(self, add_bedrooms_per_room = True): # no *args or **kwargs
        self.add_bedrooms_per_room = add_bedrooms_per_room
    def fit(self, X, y=None):
        return self  # nothing else to do
    def transform(self, X, y=None):
        rooms_per_household = X[:, rooms_ix] / X[:, household_ix]
        population_per_household = X[:, population_ix] / X[:, household_ix]
        if self.add_bedrooms_per_room:
            bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
            return np.c_[X, rooms_per_household, population_per_household,
                         bedrooms_per_room]
        else:
             return np.c_[X, rooms_per_household, population_per_household]

attr_adder = CombinedAttributesAdder(add_bedrooms_per_room=False)
housing_extra_attribs = attr_adder.transform(housing.values)

thanks again!

[ageron]

Hi @sebassilverac ,

I'm glad you're finding the book useful! :)

In this code, we're creating a custom transformer that will add combined features to the dataset, including the number of rooms per household, the population per household, and optionally the number of bedrooms per room (this is determined by the constructor argument add_bedrooms_per_room).

Let's look at each part of the code:

from sklearn.base import BaseEstimator, TransformerMixin

Importing these classes is not compulsory (since Scikit-Learn uses duck-typing), but it simplifies creating a custom transformer.

rooms_ix, bedrooms_ix, population_ix, household_ix = [
    list(housing.columns).index(col)
    for col in ("total_rooms", "total_bedrooms", "population", "households")]

We get the indices of the columns we're interested in.

class CombinedAttributesAdder(BaseEstimator, TransformerMixin):

We create the custom class, extending the BaseEstimator and TransformerMixin classes. The first one provides the get_params() and set_params() methods (based on introspecting the arguments of the constructor), and the second adds the fit_transform() method which just calls fit() then transform().

    def __init__(self, add_bedrooms_per_room = True): # no *args or **kwargs
        self.add_bedrooms_per_room = add_bedrooms_per_room

The constructor should not have *args or **kwargs, or else it breaks the get_params() and set_params() methods from BaseEstimator. Here we define an add_bedrooms_per_room argument, so we set it as an instance attribute.

    def fit(self, X, y=None):
        return self  # nothing else to do

This transformer does not learn from the data, so its fit() method does not do anything, other than returning self to respect Scikit-Learns API.

    def transform(self, X, y=None):
        rooms_per_household = X[:, rooms_ix] / X[:, household_ix]

We create the transform() method. It must take input features X, and optionally labels y. Even though we don't use y, it's useful to add it to the method signature, or else the transformer will not be usable in a pipeline (which expects the transform method to take both X and y arguments).

        rooms_per_household = X[:, rooms_ix] / X[:, household_ix]

We compute the number of rooms per household for every instance in X. Note that X is 2-dimensional: its shape is [batch size, number of features]. We want to compute the number of rooms per household for each and every instance, hence the colon : in the equation, for the first dimension: X[:, rooms_ix]. The result is a one-dimensional NumPy array containing the number of rooms per household for each and every instance in X.

        population_per_household = X[:, population_ix] / X[:, household_ix]

Compute the population per household.

        if self.add_bedrooms_per_room:
            bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
            return np.c_[X, rooms_per_household, population_per_household,
                         bedrooms_per_room]

Compute the number of bedrooms per room, if it was requested, and return a concatenation of X and the newly created combined features, including the bedrooms_per_room.
Note that np.c_[a,b,c] concatenates NumPy arrays a, b and c along axis=1 (features). In a matrix, that corresponds to appending columns.
If we used np.r_[a,b,c] instead, we would be appending along axis=0 (instances), which is not what we want.

        else:
             return np.c_[X, rooms_per_household, population_per_household]

If add_bedrooms_per_room is False, then we just return X concatenated with the two combined features rooms_per_household and population_per_household.

attr_adder = CombinedAttributesAdder(add_bedrooms_per_room=False)
housing_extra_attribs = attr_adder.transform(housing.values)

We create an instance of the custom transformer we just defined, setting add_bedrooms_per_room=False, and we use it to transform the housing dataset. Note that housing.values converts a Pandas DataFrame to a NumPy array.

Hope this helps!

[ageron]

ps: to include multiple lines of Python code in a Github comment, you should use this syntax:

```python
print("hello")
print("world")
```

which will give you nice syntax highlighting:

print("hello")
print("world")

[ssilverac]

Hi Aurelien
Thank you for that in depth explanation, I really appreciate the help. Its great that you are willing to take the time to help people out with questions. I am very new to machine learning and was quite overwhelmed at first but knowing that I can ask questions when I am confused is a great motivator to keep learning.
Cheers

[ssilverac]

Hi @ageron
I was just going over your explanation in a little more depth and just had a few follow up questions if you dont mind.

1)

def __init__(self, add_bedrooms_per_room = True): # no *args or **kwargs
        self.add_bedrooms_per_room = add_bedrooms_per_room

in this line, this is where we define the hyperparameters is that correct? so if we wanted to experiment with other hyperparameters included here?

2) def fit(self, X, y=None

here you said that this transformer does not learn from the data, could you just explain what you mean by that?

and finally, why did we set add_bedrooms_per_rooms to False at the end there?

Thanks again for your help

[ageron]

Hi @AyEnEss ,

Sure!

1. Indeed, if you wanted to add more hyperparameters, you would just have to add them to the arguments of the constructor and set corresponding instance variables, then use these hyperparameters in the fit() or transform() method (or both).

2. When you call the fit() method on most Scikit-Learn estimators, some value is computed based on the data X and/or the labels y. For example, the StandardScaler transformer computes the scale_ and mean_ instance attributes. These are called "learned parameters", since they are computed based on the data. By convention, their name ends with an underscore, so you can get the list of learned parameters by running the following code:

[attr for attr in dir(scaler) if attr.endswith("_") and not attr.startswith("_")]

Note that these learned parameters usually don't exist until you call the fit() method.

Now our CombinedAttributesAdder custom transformer does not need to compute anything when we call the fit() method. Indeed, the transform() method can directly compute the new features for each instance without requiring any knowledge of the other instances. So the fit() method does not need to do anything.

In contrast, in order to standardize the features of an instance, the StandardScaler transformer needs to know what the mean and scale of each feature is.

1. Finally, we set add_bedrooms_per_rooms=False arbitrarily. We can run everything, measure the validation error, and then start again setting add_bedrooms_per_rooms=True and measure the validation error again. This way, we would know whether the extra features helps or not. That's the benefit of making it easy to include the feature or not.

Hope this helps.

[ssilverac]

That actually makes a lot of sense thank you for that