Koalas: Duplicated key when groupby -> apply is used

[syonekura]

Hello, I opened this thread in SO, but viewing other issues reported here ([1], [2]), I'm suspecting that this could also be a bug.

I'm not sure if I'm missing something about the groupby - apply when moving from pandas to koalas, but the only requirement that we have is that the function to be applied cannot be type annotated since it will be used in different contexts, with different dataframes, so we don't know all the columns in advance. We're aware that koalas must run the function to infer the schema before applying the function, and we're ok with that overhead

[syonekura]

Hi again, I encountered this closed issue in pandas, it seems that the grouping column is kept when one performs apply, and it is explicitly tested for (so it's a feature of pandas)

The problem arises then koala's apply has to infer the schema, runs the pandas groupby - apply and then initiates a koalas.DataFrame with the result (that after groupby-apply has index and a column with the same name):

def apply(self, func):
    ...
    if should_infer_schema:
        # Here we execute with the first 1000 to get the return type.
        limit = get_option("compute.shortcut_limit")
        pdf = self._kdf.head(limit)._to_internal_pandas()
        pdf = pdf.groupby(input_groupnames).apply(func)  # <- pandas apply receive the key as column and index
        kdf = DataFrame(pdf) #  <- ValueError: cannot insert "key", already exists
        return_schema = kdf._sdf.drop(*HIDDEN_COLUMNS).schema

for now I can drop the key column inside func, I'm not sure if koalas should handle the logic for deduplicating index/column names before the initialization of ks.DataFrame

[itholic]

Sincerely appreciate for the report an issue & detailed investigation 👍

I think we also should handle this logic and add the test explicitly like pandas.

@ueshin , @HyukjinKwon , WDYT about this?

Or, maybe are you interested in creating a PR to fix this, @syonekura ?

[itholic]

@ueshin , @HyukjinKwon

For saving your time, the summary of this issue is like the below.

• in pandas.

>>> df = pd.DataFrame({'d': [1.0, 1.0, 1.0, 2.0, 2.0, 2.0], 'v': [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
>>> df
     d    v
0  1.0  1.0
1  1.0  2.0
2  1.0  3.0
3  2.0  4.0
4  2.0  5.0
5  2.0  6.0

>>> df.groupby('d').sum()
        v
d
1.0   6.0
2.0  15.0

>>> df.groupby('d').apply(sum)
       d     v
d
1.0  3.0   6.0
2.0  6.0  15.0

• but in koalas

>>> ks.from_pandas(df).groupby('d').sum()
        v
d
1.0   6.0
2.0  15.0

>>> ks.from_pandas(df).groupby('d').apply(sum)
...
ValueError: cannot insert d, already exists

[HyukjinKwon]

Thanks, yeah. It would be great to fix to match at this moment.

In Spark, it supports the duplicated columns in some contexts. In some contexts, it doesn't.
So, as a future plan, we're thinking about disallowing duplicated column & index names in general as described in the best practice due to the limitation of Spark.

[syonekura]

@itholic I would love to submit a PR for this, I'll give it a try during this weekend

[itholic]

@syonekura Sure, let's try :smile_cat: