Koalas: ks.groupby won't drop groups with one or more null keys

[pandareen]

This is basically where the pandas groupby adopts its behaviour from r.
I can submit a pr if this is not on anyones roadmap

[harupy]

pyspark doesn't drop null groups. I didn't know this.

>>> sdf.show()
+----+
|   x|
+----+
|null|
| 1.0|
| 1.0|
+----+

>>> sdf.groupBy('x').count().show()
+----+-----+
|   x|count|
+----+-----+
|null|    1|
| 1.0|    2|
+----+-----+

>>> sdf.toPandas().groupby('x').agg({'x': 'count'})
     x
x   
1.0  2

[pandareen]

@harupy correct. That is to keep it consistent with r library

[HyukjinKwon]

Yeah, I noticed this behaviour as well lately. Please go ahead for a PR!

[pandareen]

Okay thanks, I'm on it

[LucasG0]

Hi, pandas 1.1.0 recently added a dropna parameter in order to allow groupby on null keys.
By default (True), null keys are dropped, and it has pyspark behavior when set to False.
I think it would be interesting to add it to koalas dataframe groupby method.
May i work on it ?

[ueshin]

@LucasG0 Awesome, please go ahead! Thanks!
cc @itholic as working on pandas 1.1 support

[itholic]

Thanks, @LucasG0 .
And thanks for the notice, @ueshin