Koalas: pivot_table function with renaming columns

[thoo]

Here is the example from the documentation.

df = ks.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo",
                          "bar", "bar", "bar", "bar"],
                    "C": ["small", "large", "large", "small",
                          "small", "large", "small", "small",
                          "large"],
                    "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]},
                   columns=['A',  'C',  'E'])
df.pivot_table(values='E', index=['A'],
                        columns='C', aggfunc='sum')

The above code works as expected. Then I rename the columns like

df.columns =['New_A','New_C','New_E']
print(df.columns)

This works fine with applymap and other functions like to_datetime but as soon as I use pivot_table , I got an error.

df.pivot_table(values='New_E', index=['New_A'],
                        columns='New_C', aggfunc='sum')

The above code will give the following error:

KeyError: 'No StructField named New_E'

Is it not allowed to change the column name right now?? Thanks.

[harupy]

Seems like df.columns = new_columns doesn't change the columns in the internal spark dataframe. pivot_table tries to access the column New_E which doesn't exist and raise an error.

>>> df._sdf.show()
+-----------------+---+-----+---+
|__index_level_0__|  A|    C|  E|
+-----------------+---+-----+---+
|                0|foo|small|  2|
|                1|foo|large|  4|
|                2|foo|large|  5|
|                3|foo|small|  5|
|                4|foo|small|  6|
|                5|bar|large|  6|
|                6|bar|small|  8|
|                7|bar|small|  9|
|                8|bar|large|  9|
+-----------------+---+-----+---+

[ueshin]

@harupy Actually the root cause is the pivot_table doesn't handle metadata (_InternalFrame) properly. This is the same as merge case at #782. So far fixing columns is easier way, though, but it doesn't fix the case multi-index columns are used.

[HyukjinKwon]

BTW, thanks for reporting this @thoo. the issue looks very easy to understand and reproduce.

[ueshin]

Shall we fix the columns setter to rename the Spark columns for now?
@itholic @harupy Do you want to fix it?

I'll handle merge and pivot_table soon.

[harupy]

I'll work on this.

[ueshin]

@harupy Great, thanks!

[itholic]

oh It seems pretty interesting,
thanks for suggestion @ueshin ,
and go for it @harupy ! 😃