Handson-ml: chapter 2: What is the correct way of creating a pipeline

[akjain90]

I really appreciate your work and the efforts that you took by creating this whole series of notebooks.
My issue is regarding the transformation pipeline which you are creating in cell 73 as follows:

num_attribs = list(housing_num)
cat_attribs = ["ocean_proximity"]

num_pipeline = Pipeline([
        ('selector', DataFrameSelector(num_attribs)),
        ('imputer', Imputer(strategy="median")),
        ('attribs_adder', CombinedAttributesAdder()),
        ('std_scaler', StandardScaler()),
    ])

cat_pipeline = Pipeline([
        ('selector', DataFrameSelector(cat_attribs)),
        ('cat_encoder', CategoricalEncoder(encoding="onehot-dense")),
    ])

The first component of the numerical pipeline is DataframeSelector which returns a dataframe without a categorical entry ie ocean_proximity. Further operations are Imputer and CombinedAttributesAdder.

Just before the definition of CombinedAttributesAdder class in cell 68, you define the column index that needs to be combined, as follows:

rooms_ix, bedrooms_ix, population_ix, household_ix = 3, 4, 5, 6

class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
    def __init__(self, add_bedrooms_per_room = True): # no *args or **kargs
        self.add_bedrooms_per_room = add_bedrooms_per_room
    def fit(self, X, y=None):
        return self  # nothing else to do
    def transform(self, X, y=None):
        rooms_per_household = X[:, rooms_ix] / X[:, household_ix]
        population_per_household = X[:, population_ix] / X[:, household_ix]
        if self.add_bedrooms_per_room:
            bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
            return np.c_[X, rooms_per_household, population_per_household,
                         bedrooms_per_room]
        else:
            return np.c_[X, rooms_per_household, population_per_household]

In the present case ocean_proximity is the last column and that's why our column index doesn't change even after calling the DataframeSelector. But in case ocean_proximity is not the last feature we need to calculate the updated column index before hand which could be very tedious work in case we have a huge set of features.

So what I wanted to ask (This could be a silly doubt ) is, is there a standard way of doing the above-mentioned procedure?

Thank you.

[ageron]

Hi @akjain90 ,
Thanks for your kind words and for your question. Unfortunately, Scikit-Learn is not integrated with Pandas. Sure, you can feed a DataFrame to any Scikit-Learn transformer, but it will output a NumPy array, so you lose the index and column information. Similarly, predictors lose the index information when they make predictions.

One solution is to create DataFrames based on the NumPy arrays returned by Scikit-Learn, and restore column and index information, for example like this:

>>> import pandas as pd
>>> from sklearn.linear_model import LinearRegression
>>> X = pd.DataFrame({"age":[20,30,25], "weight":[120, 130, 135]},
...                  index=["joe", "jane", "jack"])
...
>>> y = pd.Series([180, 170, 175], index=["joe", "jane", "jack"], name="height")
>>> lin_reg = LinearRegression()
>>> lin_reg.fit(X, y)
LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)
>>> X_test = pd.DataFrame({"age":[22,24], "weight":[133, 140]}, index=["alice", "bob"])
>>> y_pred = lin_reg.predict(X_test)
>>> y_pred
array([ 178.,  176.])
>>> y_pred = pd.DataFrame(y_pred, columns=["height"], index=X_test.index)
>>> y_pred
       height
alice   178.0
bob     176.0

It's a bit annoying to have to do this manually all the time (which is why I didn't do it in the book), and I just kept track of column index.

I guess another option would be to create a DataFrameEstimatorWrapper class that would wrap any Scikit-Learn estimator (transformer, classifier, regressor...), and it would just add back the appropriate index and column names to the outputs of predictions and transformations. Something like this:

>>> import pandas as pd
>>> from sklearn.linear_model import LinearRegression
>>> from dataframe_estimator_wrapper import DataFrameEstimatorWrapper # to be implemented
>>> X = pd.DataFrame({"age":[20,30,25], "weight":[120, 130, 135]},
...                  index=["joe", "jane", "jack"])
...
>>> y = pd.Series([180, 170, 175], index=["joe", "jane", "jack"], name="height")
>>> lin_reg = DataFrameEstimatorWrapper(LinearRegression())
>>> lin_reg.fit(X, y)
>>> y_pred = lin_reg.predict(X_test)
>>> y_pred
       height
alice   178.0
bob     176.0

Seems a bit too magical to me, I'm afraid there would be tons of edge cases to handle, and you would have to wrap every estimator.

Otherwise, in Pull Request #9012, there's a ColumnTransformer class that could help, so if it is merged one day, things will be simpler, since it will be easy to do different transformations for each column. In the meantime, you can copy it in your code if you find it useful.

Another option is to run pip3 install sklearn-pandas to get a DataFrameMapper class with a similar objective.

Hope this helps,
Aurélien

[akjain90]

Thank you for the inputs. Will check all these possibilities.

[YiRongZhuang]

CategoricalEncoder briefly existed in 0.20dev. Its functionality has been rolled into the OneHotEncoder and OrdinalEncoder. This stub will be removed in version 0.21.

[VicktorKrum]

Hi,
Can you kindly explain how to use CombinedAttributesAdder() directly from library? I have done the code from the book where it is first used and am getting error that it is not defined.

Kindly help me with how to fix it.
Thank You.