Handson-ml: chapter 3 titanic Question

[nitml]

hi,
In titanic question in jupyter notebook DataFrameSelector code was:

#cell 110
class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return X[self.attribute_names]

but should not the last line be :

return X[self.attribute_names].values

if it is X[self.attribute_names] then retured object will still be a pandas.DataFrame but we want numpy.array so that it can be used in sklearn.

and also should not we be dropping Survived column for X_train before:

#cell 118
X_train = preprocess_pipeline.fit_transform(train_data)

[ageron]

Hi @nitml ,

Good questions. Scikit-Learn estimators actually accept DataFrames as inputs: they just use the values automatically.
Regarding dropping the Survived column before calling fit_transform(), it might indeed be safer but in this particular case the pipeline uses the DataFrameSelector transformer to select a subset of the columns, and this does not include the Survived column, so we're fine.

For example:

cat_pipeline = Pipeline([
        ("select_cat", DataFrameSelector(["Pclass", "Sex", "Embarked"])),
        ("imputer", MostFrequentImputer()),
        ("cat_encoder", OneHotEncoder(sparse=False)),
    ])

[nitml]

Thanks for your time to clear doubt ,
wasn't the whole point of writing DataFrameSelector class was to make dataframe into numpy ?
but if sklearn can handle dataframe why do we need DataFrameSelector class ?

[ageron]

My pleasure :)

The point of the DataFrameSelector is to select the appropriate columns in the DataFrame. The numerical pipeline selects the numerical columns, while the categorical pipeline selects the categorical columns.
If you just passed the original DataFrame to a Scikit-Learn estimator, it would just gets its values, but this would contain a mix of numerical and categorical columns (not to mention the Survived column!), and it would fail.
Hope this helps

[nitml]

Thanks Great explanation !!