Handson-ml: chapter 2:End-to-End Machine Learning Project

[aishwaryashinde6]

I have a doubt related to Custom Transformers, I am new to coding and I didn't under why are we using the customized transformers and how does it help in the data cleaning process?

[ageron]

Hello @aishwaryashinde6 ,

You need custom transformers whenever you want to do some preprocessing for which no standard transformer exists. For example, say you want to find the 5 most used values of a category and remove the rest, there's no standard transformer to do that, so you would have to write your own.

In the book, I used a custom transformer to select a subset of the columns so that they can be preprocessed differently than the rest. The basic idea was this:

• Build a Pipeline to preprocess all numerical columns
• Build another Pipeline to preprocess all categorical columns
• Merge their outputs using a FeatureUnion
• Since a FeatureUnion feeds the same data to both pipelines, at the beginning of each pipeline I used a custom transformer to select the appropriate columns (either numerical or categorical).

However, Scikit-Learn 0.20 introduced the ColumnTransformer class, which makes this much easier to do this:

• Create a ColumnTransformer and tell which transformer to use for each column.

Hope this helps!

[ageron]

I think perhaps you were asking why you need custom transformers as opposed to just executing the preprocessing steps directly on the data. There are two reasons:

• Writing a custom transformer makes your code more standard and easier to reuse, in particular you can use transformers in Scikit-Learn pipelines.
• Transformers can have learned parameters. For example, a StandardScaler learns the mean and standard deviation of the training set, and it can then use these learned parameters when scaling the validation set, the test set, and new instances. If you just called a standardization function that makes your data zero-centered with standard deviation equal to 1, then it would use the mean and stddev of whatever data you give it. But once your code is in production, you may need to make predictions for one instance at a time, so it would not make sense to use this single instance's "mean" (and the stddev would be undefined). The only mean and stddev that make sense then are the training set mean and stddev. So this means you need to learn these values on the training set and use them to scale all data. It's just convenient to have an object that both learns these values, holds on to them and uses them when requested.

[aishwaryashinde6]

from sklearn.preprocessing import FunctionTransformer

def add_extra_features(X, add_bedrooms_per_room=True):
    rooms_per_household = X[:, rooms_ix] / X[:, household_ix]
    population_per_household = X[:, population_ix] / X[:, household_ix]
    if add_bedrooms_per_room:
        bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
        return np.c_[X, rooms_per_household, population_per_household,
                     bedrooms_per_room]
    else:
        return np.c_[X, rooms_per_household, population_per_household]

attr_adder = FunctionTransformer(add_extra_features, validate=False,
                                 kw_args={"add_bedrooms_per_room": False})
housing_extra_attribs = attr_adder.fit_transform(housing.values)

why did u set add_bedrooms_per_rooms as your hyperparameter, and I also didn't understand the if else condition in the code

[ageron]

Hi @aishwaryashinde6 ,
The add_bedrooms_per_room hyperparameter is just there to illustrate the fact that you can treat some of the preprocessing decisions as hyperparameters that can be explored automatically. Perhaps adding this extra feature helps the model, or perhaps it hurts it, it's hard to know without actually trying both options. Rather than do it manually, you can write a transformer with a hyperparameter that controls that, and later in the chapter, when we explore automatic hyperparameter search, this will be one of the parameters that can be searched automatically to see which option is best.

Hope this helps!

[aishwaryashinde6]

Thank you, Sir your book is amazing:)

On Wed, Mar 6, 2019 at 4:42 PM Aurélien Geron [e-mail]
wrote:

Hi @aishwaryashinde6 https://github.com/aishwaryashinde6 ,
The add_bedrooms_per_room hyperparameter is just there to illustrate the
fact that you can treat some of the preprocessing decisions as
hyperparameters that can be explored automatically. Perhaps adding this
extra feature helps the model, or perhaps it hurts it, it's hard to know
without actually trying both options. Rather than do it manually, you can
write a transformer with a hyperparameter that controls that, and later in
the chapter, when we explore automatic hyperparameter search, this will be
one of the parameters that can be searched automatically to see which
option is best.

Hope this helps!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ageron/handson-ml/issues/368#issuecomment-470068442,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AqtbTO-NMs1Kedlm2ehKYRi3l2SONxm1ks5vT6K5gaJpZM4be7vC
.